Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stream 1088/near live view udtfs #99

Closed
wants to merge 15 commits into from

Conversation

ShahNewazKhan
Copy link

@ShahNewazKhan ShahNewazKhan commented Nov 13, 2024

  • Adds make directive to tmp deploy and rm dbt logs
  • Adds config_near_high_level_abstractions macro for Near Live View UDTF's
    • Adds tf_get_block_data Python UDTF
      • Given an array of external stage S3 urls, retrieves file contents
    • Adds fact_blocks UDTF
      • Uses udf_api to retrieve chainhead and generate spine
      • Uses tf_get_block_data to retrieve Near block data from the Near Lake from spine range

Deployed to livequery_dev, test TF_GET_BLOCK_DATA & `TF_FACT_BLOCKS:

-- Test with multiple files
SELECT block_data 
FROM TABLE(get_block_data_bulk(ARRAY_CONSTRUCT(
    BUILD_SCOPED_FILE_URL(@streamline.bronze.near_lake_data_mainnet, '000132121137/block.json'),
    BUILD_SCOPED_FILE_URL(@streamline.bronze.near_lake_data_mainnet, '000132121138/block.json'),
    BUILD_SCOPED_FILE_URL(@streamline.bronze.near_lake_data_mainnet, '000132121139/block.json')
)));
SELECT * FROM TABLE(LIVEQUERY_DEV.NEAR_MAINNET.TF_FACT_BLOCKS(132121139,FALSE));

@ShahNewazKhan ShahNewazKhan marked this pull request as ready for review November 13, 2024 23:28
@ShahNewazKhan ShahNewazKhan marked this pull request as draft November 14, 2024 18:47
@ShahNewazKhan
Copy link
Author

Closing in favor of #102

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant