Only retrieve updated query statement metrics

[sethsamuel]

What does this PR do?

Updates MySQL statement collection to only gather statements that have executed since the last collection. This functionality is behind a flag to allow for slow rollout and testing before hopefully making it standard.

Four variant builds were compared:

• latest, a control with current live behavior
• digest-only, query for the same records as latest but only get the digest, then perform a second query for the digest text if it is not already cached
• last-seen, query only for digests of statements that have executed since the last collection, then query for digest text if not cached
• last-seen-with-text, query for statements executed since last collection but with digest text.

Testing was performed on orders app with 1 minute bursts of high cardinality queries every 5 minutes.

Somewhat surprisingly, querying for the digest and then for digest text made little difference in performance, even with large random statements (~1kb) executed during bursts. When querying for only updated rows, the much smaller row count allows for fetching the text as well, avoiding a second database query.

CPU usage is greatly reduced during normal load, and somewhat reduced during high cardinality bursts.

Query time and overall statement metrics collection time (note that y-axis is in seconds, not ms) are greatly reduced.

Motivation

Customers have complained about CPU usage by the MySQL check. This change uses a similar technique to the other database integrations to minimize the amount of unnecessary data retrieved for each statement collection.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

[codecov]

Codecov Report

Attention: Patch coverage is 94.44444% with 2 lines in your changes missing coverage. Please review.

Project coverage is 87.65%. Comparing base (aa9bd4e) to head (2600722).
Report is 12 commits behind head on master.

Additional details and impacted files

Flag	Coverage Δ
activemq	?
cassandra	?
hive	?
hivemq	?
ignite	?
jboss_wildfly	?
kafka	?
mysql	89.51% <94.44%> (-0.01%)	⬇️
presto	?
solr	?

Flags with carried forward coverage won't be shown. Click here to find out more.

[lu-zhengda]

Is there a reason to filter the rows after retrieving from the database? can you still leverage WHERE digest_textNOT LIKE 'EXPLAIN %' ORdigest_text IS NULL with last_seen filter?

[sethsamuel]

We could but it slows down the query and since we're not limiting anymore it's simple enough to filter them on the client. In practice it seems like EXPLAINs are never a substantial number of rows.