[sc-26213] Lower max query text length limit to 100K

[usefulalgorithm]

🤔 Why?

The Snowflake crawler parses query logs from the QUERY_HISTORY view, whose query_text column is potentially truncated if the value is too long: https://docs.snowflake.com/en/sql-reference/account-usage/query_history

(The link says the limit is 100K characters, but in our Snowflake instance we can see a query is parsed that's longer than 100K:

)

From the lineage parser's perspective a truncated expression is an invalid expression. To make lineage parsing possible, just drop the sql queries that are longer than 100K characters long, since it's very likely to be a INSERT INTO expression.

🤓 What?

Lower max query text length limit to 100_000.

🧪 Tested?

• Crawler run locally against metaphor's Snowflake instance.

☑️ Checks

• My PR contains actual code changes, and I have updated the version number in pyproject.toml.

[github-actions]

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines	Covered	Coverage	Threshold	Status
18906	17752	94%	85%	🟢

New Files

No new covered files...

Modified Files

File	Coverage	Status
metaphor/snowflake/config.py	100%	🟢
metaphor/snowflake/extractor.py	85%	🟢
TOTAL	92%	🟢

updated for commit: 0a7957f by action🐍

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.89%. Comparing base (ec69369) to head (0a7957f).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #859      +/-   ##
==========================================
+ Coverage   93.71%   93.89%   +0.18%     
==========================================
  Files         203      161      -42     
  Lines       19074    18906     -168     
==========================================
- Hits        17875    17752     -123     
+ Misses       1199     1154      -45     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mars-lan]

I'm confused, the example you gave shows that a query_text of length 144k chars got truncated. Isn't that the correct expectation?

[usefulalgorithm]

I'm confused, the example you gave shows that a query_text of length 144k chars got truncated. Isn't that the correct expectation?

ack I saw wrong, wait wait

[usefulalgorithm]

@mars-lan can you take another look? updated the pr description and some names in the logic

[mars-lan]

@mars-lan can you take another look? updated the pr description and some names in the logic

Got it. In that case, let's just ignore queries longer than 100k w/o doing any regex magic?

[mars-lan]

Otherwise LGTM