Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MAINTENANCE] Simplify logic around data context ID creation #10404

Closed
wants to merge 5 commits into from

Conversation

cdkini
Copy link
Member

@cdkini cdkini commented Sep 16, 2024

Legacy behavior here has depended on the expectations store with some pretty convoluted logic

  • Description of PR changes above includes a link to an existing GitHub issue
  • PR title is prefixed with one of: [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], [CONTRIB]
  • Code is linted - run invoke lint (uses ruff format + ruff check)
  • Appropriate tests and docs have been updated

For more information about contributing, see Contribute.

After you submit your PR, keep the page open and monitor the statuses of the various checks made by our continuous integration process at the bottom of the page. Please fix any issues that come up and reach out on Slack if you need help. Thanks for contributing!

Copy link

netlify bot commented Sep 16, 2024

Deploy Preview for niobium-lead-7998 canceled.

Name Link
🔨 Latest commit b3c9c56
🔍 Latest deploy log https://app.netlify.com/sites/niobium-lead-7998/deploys/66edb4368422670008091067

Copy link

codecov bot commented Sep 16, 2024

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
27821 2 27819 5010
View the top 2 failed tests by shortest run time
tests.datasource.fluent.test_viral_snippets test_file_context_add_and_save_fluent_datasource
Stack Traces | 0.204s run time
file_dc_config_dir_init = PosixPath('.../pytest-0/test_file_context_add_and_save0/gx')
fluent_only_config = GxConfig(fluent_datasources=[PostgresDatasource(type='postgres', name='my_pg_ds', id=UUID('dae0accb-fec0-410b-9baa-dd7...0, multiindex_hierarchies=False, mode='xmla')], workspace=None, dataset=UUID('0993ddd1-f01d-4c8f-a64e-9c4e5b8905e8'))])
sqlite_database_path = PosixPath('.../taxi_yellow_tripdata_samples/sqlite/yellow_tripdata.db')

    @pytest.mark.filesystem
    def test_file_context_add_and_save_fluent_datasource(
        file_dc_config_dir_init: pathlib.Path,
        fluent_only_config: GxConfig,
        sqlite_database_path: pathlib.Path,
    ):
        datasource_name = "save_ds_test"
        config_file = file_dc_config_dir_init / FileDataContext.GX_YML
    
        initial_yaml = config_file.read_text()
        assert datasource_name not in initial_yaml
    
        context: FileDataContext = get_context(context_root_dir=config_file.parent, cloud_mode=False)
    
        ds = context.data_sources.add_sqlite(
            name=datasource_name, connection_string=f"sqlite:///{sqlite_database_path}"
        )
    
        final_yaml = config_file.read_text()
        diff = difflib.ndiff(initial_yaml.splitlines(), final_yaml.splitlines())
    
        print("\n".join(diff))
    
        assert datasource_name == ds.name
        assert datasource_name in final_yaml
        # ensure comments preserved
>       assert "# Welcome to Great Expectations!" in final_yaml
E       AssertionError: assert '# Welcome to Great Expectations!' in 'config_version: 4.0\nconfig_variables_file_path: uncommitted/config_variables.yml\nplugins_directory: plugins/\nstores:\n  expectations_store:\n    class_name: ExpectationsStore\n    store_backend:\n      class_name: TupleFilesystemStoreBackend\n      base_directory: expectations/\n  validation_results_store:\n    class_name: ValidationResultsStore\n    store_backend:\n      class_name: TupleFilesystemStoreBackend\n      base_directory: uncommitted/validations/\n  checkpoint_store:\n    class_name: CheckpointStore\n    store_backend:\n      class_name: TupleFilesystemStoreBackend\n      suppress_store_backend_id: true\n      base_directory: checkpoints/\n  validation_definition_store:\n    class_name: ValidationDefinitionStore\n    store_backend:\n      class_name: TupleFilesystemStoreBackend\n      base_directory: validation_definitions/\nexpectations_store_name: expectations_store\nvalidation_results_store_name: validation_results_store\ncheckpoint_store_name: checkpoint_store\ndata_docs_sites:\n  local_site:\n    class_name: SiteBuilder\n    show_how_to_buttons: true\n    store_backend:\n      class_name: TupleFilesystemStoreBackend\n      base_directory: uncommitted/data_docs/local_site/\n    site_index_builder:\n      class_name: DefaultSiteIndexBuilder\nanalytics_enabled: true\ndata_context_id: 37c7f6e4-60f0-4411-b0a0-38f236a05c2d\nfluent_datasources:\n  save_ds_test:\n    type: sqlite\n    id: 33b2db74-b0e6-44ca-8109-1040538ca2c6\n    connection_string: \n      sqlite:///.../taxi_yellow_tripdata_samples/sqlite/yellow_tripdata.db\n'

.../datasource/fluent/test_viral_snippets.py:143: AssertionError
tests.datasource.fluent.test_viral_snippets test_file_context_add_and_save_fluent_datasource
Stack Traces | 0.248s run time
file_dc_config_dir_init = PosixPath('.../pytest-0/test_file_context_add_and_save0/gx')
fluent_only_config = GxConfig(fluent_datasources=[PostgresDatasource(type='postgres', name='my_pg_ds', id=UUID('dae0accb-fec0-410b-9baa-dd7...0, multiindex_hierarchies=False, mode='xmla')], workspace=None, dataset=UUID('0993ddd1-f01d-4c8f-a64e-9c4e5b8905e8'))])
sqlite_database_path = PosixPath('.../taxi_yellow_tripdata_samples/sqlite/yellow_tripdata.db')

    @pytest.mark.filesystem
    def test_file_context_add_and_save_fluent_datasource(
        file_dc_config_dir_init: pathlib.Path,
        fluent_only_config: GxConfig,
        sqlite_database_path: pathlib.Path,
    ):
        datasource_name = "save_ds_test"
        config_file = file_dc_config_dir_init / FileDataContext.GX_YML
    
        initial_yaml = config_file.read_text()
        assert datasource_name not in initial_yaml
    
        context: FileDataContext = get_context(context_root_dir=config_file.parent, cloud_mode=False)
    
        ds = context.data_sources.add_sqlite(
            name=datasource_name, connection_string=f"sqlite:///{sqlite_database_path}"
        )
    
        final_yaml = config_file.read_text()
        diff = difflib.ndiff(initial_yaml.splitlines(), final_yaml.splitlines())
    
        print("\n".join(diff))
    
        assert datasource_name == ds.name
        assert datasource_name in final_yaml
        # ensure comments preserved
>       assert "# Welcome to Great Expectations!" in final_yaml
E       AssertionError: assert '# Welcome to Great Expectations!' in 'config_version: 4.0\nconfig_variables_file_path: uncommitted/config_variables.yml\nplugins_directory: plugins/\nstores:\n  expectations_store:\n    class_name: ExpectationsStore\n    store_backend:\n      class_name: TupleFilesystemStoreBackend\n      base_directory: expectations/\n  validation_results_store:\n    class_name: ValidationResultsStore\n    store_backend:\n      class_name: TupleFilesystemStoreBackend\n      base_directory: uncommitted/validations/\n  checkpoint_store:\n    class_name: CheckpointStore\n    store_backend:\n      class_name: TupleFilesystemStoreBackend\n      suppress_store_backend_id: true\n      base_directory: checkpoints/\n  validation_definition_store:\n    class_name: ValidationDefinitionStore\n    store_backend:\n      class_name: TupleFilesystemStoreBackend\n      base_directory: validation_definitions/\nexpectations_store_name: expectations_store\nvalidation_results_store_name: validation_results_store\ncheckpoint_store_name: checkpoint_store\ndata_docs_sites:\n  local_site:\n    class_name: SiteBuilder\n    show_how_to_buttons: true\n    store_backend:\n      class_name: TupleFilesystemStoreBackend\n      base_directory: uncommitted/data_docs/local_site/\n    site_index_builder:\n      class_name: DefaultSiteIndexBuilder\nanalytics_enabled: true\ndata_context_id: c16cd4b8-63ff-4ecb-baa6-68d119cd9808\nfluent_datasources:\n  save_ds_test:\n    type: sqlite\n    id: 737649f5-5c7b-4b42-b30a-9c3b8cf3f709\n    connection_string: \n      sqlite:///.../taxi_yellow_tripdata_samples/sqlite/yellow_tripdata.db\n'

.../datasource/fluent/test_viral_snippets.py:143: AssertionError

To view individual test run time comparison to the main branch, go to the Test Analytics Dashboard

Comment on lines -122 to -125
@pytest.mark.aws_deps
@mock_s3
def test_DataContext_construct_data_context_id_uses_id_of_currently_configured_expectations_store(
aws_credentials,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We no longer have this conditional logic based on expectations store - we don't need these tests

@cdkini cdkini closed this Oct 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants