Optimize queries used in Snowflake lineage connector

[mars-lan]

🤔 Why?

It takes an exceedingly long time (> 1.5 hours) to run the Snowflake lineage crawler against large Snowflake.AccountUnage.QUERY_HISTORY views.

Turns out that similar to ACCESS_HISTORY, we must also specify a filter against START_TIME in order to query QUERY_HISTORY efficiently.

🤓 What?

• Specify the START_TIME filter for QUERY_HISTORY and use inner JOIN when joining with ACCESS_HISTORY in Snowflake lineage connector
• Qualify the filters in the base Snowflake connector to avoid confusion.

🧪 Tested?

Verified that the MCEs before & after the changes are identical. Observed an order of magnitude improvement in performance:

Before:

Ended running with RunStatus.SUCCESS at 2024-03-14 08:54:28.949943, fetched 78 entities, took 217.1s

After:

Ended running with RunStatus.SUCCESS at 2024-03-14 08:47:54.532785, fetched 78 entities, took 27.6s

☑️ Checks

• My PR contains actual code changes, and I have updated the version number in pyproject.toml.

[shortcut-integration]

This pull request has been linked to Shortcut Story #25106: Snowflake lineage crawler timing out due to inefficient queries.

[alyiwang]

LGTM

[github-actions]

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines	Covered	Coverage	Threshold	Status
16051	14848	93%	85%	🟢

New Files

No new covered files...

Modified Files

File	Coverage	Status
metaphor/snowflake/extractor.py	86%	🟢
metaphor/snowflake/lineage/extractor.py	68%	🟢
TOTAL	77%	🟢

updated for commit: 61d2a7a by action🐍