[sc-24431] Replace FileSink with StreamSink

[usefulalgorithm]

🤔 Why?

In order to reduce crawler memory usage, instead of having all parsed MCE hogging up memory it's perhaps more memory efficient to pipe a MCE to the file once it's been parsed. In order to do that, the sink has to support writing a single MCE at a time.

The new StreamSink has the exact same chunking mechanism as FileSink, but it has to be context managed:

with StreamSink(config, metadata) as sink:
    for event in events: # TODO: in the near future `events` will be replaced by `extractor.run_async()`, which is a iterator that yields a MCE once at a time.
        sink.write_event(event)

When the context ends, the sink finalizes the last batch file, and writes the execution logs and metadata.

🤓 What?

• Implemented new StreamSink class to support piped MCEs from crawler classes.
• Removed Sink ABC and FileSink.
• Modified unit test.

🧪 Tested?

Tested on personal dev env with Snowflake crawler:

crawler output:

ingestion logs:
live-tail-results.csv

All 427 MCEs are ingested successfully.

☑️ Checks

• My PR contains actual code changes, and I have updated the version number in pyproject.toml.

[shortcut-integration]

This pull request has been linked to Shortcut Story #24431: Refactor BaseExtractor for memory efficiency.

[github-actions]

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines	Covered	Coverage	Threshold	Status
15794	14569	92%	85%	🟢

New Files

No new covered files...

Modified Files

File	Coverage	Status
metaphor/common/base_config.py	100%	🟢
metaphor/common/cli.py	100%	🟢
metaphor/common/runner.py	80%	🟢
metaphor/common/sink.py	97%	🟢
metaphor/common/storage.py	72%	🟢
TOTAL	90%	🟢

updated for commit: 94f0937 by action🐍

[codecov]

Codecov Report

Attention: 15 lines in your changes are missing coverage. Please review.

Comparison is base (568318a) 92.10% compared to head (94f0937) 92.24%.

Files	Patch %	Lines
metaphor/common/storage.py	54.54%	10 Missing ⚠️
metaphor/common/runner.py	70.00%	3 Missing ⚠️
metaphor/common/sink.py	98.16%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #776      +/-   ##
==========================================
+ Coverage   92.10%   92.24%   +0.14%     
==========================================
  Files         194      153      -41     
  Lines       15918    15794     -124     
==========================================
- Hits        14661    14569      -92     
+ Misses       1257     1225      -32     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[usefulalgorithm]

looks like S3 simply does not allow append:
https://stackoverflow.com/questions/41783903/append-data-to-an-s3-object

[usefulalgorithm]

figured out a new way to do this, closing this PR