[MAINTENANCE] Remove version-specific logic from GXCloudStoreBackend #10173

cdkini · 2024-08-06T18:51:55Z

Description of PR changes above includes a link to an existing GitHub issue
PR title is prefixed with one of: [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], [CONTRIB]
Code is linted - run invoke lint (uses ruff format + ruff check)
Appropriate tests and docs have been updated

For more information about contributing, see Contribute.

After you submit your PR, keep the page open and monitor the statuses of the various checks made by our continuous integration process at the bottom of the page. Please fix any issues that come up and reach out on Slack if you need help. Thanks for contributing!

netlify · 2024-08-06T18:52:10Z

✅ Deploy Preview for niobium-lead-7998 canceled.

Name	Link
🔨 Latest commit	`90f61e7`
🔍 Latest deploy log	https://app.netlify.com/sites/niobium-lead-7998/deploys/66fc0f08047035000830c25e

codecov · 2024-08-06T18:56:50Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
27847	1	27846	5010

View the top 1 failed tests by shortest run time

tests.analytics.test_analytics test_cloud_context_init

Stack Traces | 0.034s run time

cloud_api_fake = &lt;responses.RequestsMock object at 0x7f86ffbb0340&gt;
cloud_details = CloudDetails(base_url='https://app.greatexpectations.fake.io/', org_id='12345678-1234-5678-1234-567812345678', access_...yJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c')
monkeypatch = &lt;_pytest.monkeypatch.MonkeyPatch object at 0x7f86ff526d00&gt;

    @pytest.mark.cloud
    def test_cloud_context_init(cloud_api_fake, cloud_details, monkeypatch):
        monkeypatch.setattr(ENV_CONFIG, "gx_analytics_enabled", True)  # Enable usage stats
    
        with mock.patch(
            "great_expectations.data_context.data_context.cloud_data_context.init_analytics"
        ) as mock_init, mock.patch("posthog.capture") as mock_submit:
&gt;           _ = gx.get_context(
                cloud_access_token=cloud_details.access_token,
                cloud_organization_id=cloud_details.org_id,
                cloud_base_url=cloud_details.base_url,
                cloud_mode=True,
            )

tests/analytics/test_analytics.py:113: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../hostedtoolcache/Python/3.8.18.................................................../x64/lib/python3.8.../data_context/data_context/context_factory.py:531: in get_context
    return project_manager.get_project(
.../hostedtoolcache/Python/3.8.18.................................................../x64/lib/python3.8.../data_context/data_context/context_factory.py:73: in get_project
    self.__project = self._build_context(
.../hostedtoolcache/Python/3.8.18.................................................../x64/lib/python3.8.../data_context/data_context/context_factory.py:224: in _build_context
    context = context_fn(**kwargs)
.../hostedtoolcache/Python/3.8.18.................................................../x64/lib/python3.8.../data_context/data_context/context_factory.py:249: in _get_default_context
    cloud_context = self._get_cloud_context(
.../hostedtoolcache/Python/3.8.18.................................................../x64/lib/python3.8.../data_context/data_context/context_factory.py:316: in _get_cloud_context
    return CloudDataContext(
.../hostedtoolcache/Python/3.8.18.................................................../x64/lib/python3.8.../data_context/data_context/cloud_data_context.py:124: in __init__
    super().__init__(
.../hostedtoolcache/Python/3.8.18.................................................../x64/lib/python3.8.../data_context/data_context/serializable_data_context.py:66: in __init__
    super().__init__(runtime_environment=runtime_environment)
.../hostedtoolcache/Python/3.8.18.................................................../x64/lib/python3.8.../data_context/data_context/abstract_data_context.py:243: in __init__
    self._init_analytics()
.../hostedtoolcache/Python/3.8.18.................................................../x64/lib/python3.8.../data_context/data_context/cloud_data_context.py:138: in _init_analytics
    user_id=self._get_cloud_user_id(),
.../hostedtoolcache/Python/3.8.18.................................................../x64/lib/python3.8.../data_context/data_context/cloud_data_context.py:149: in _get_cloud_user_id
    response = self._request_cloud_backend(
.../hostedtoolcache/Python/3.8.18.................................................../x64/lib/python3.8.../data_context/data_context/cloud_data_context.py:342: in _request_cloud_backend
    response = session.get(url)
.../hostedtoolcache/Python/3.8.18.................................................../x64/lib/python3.8........./site-packages/requests/sessions.py:602: in get
    return self.request("GET", url, **kwargs)
.../hostedtoolcache/Python/3.8.18.................................................../x64/lib/python3.8........./site-packages/requests/sessions.py:589: in request
    resp = self.send(prep, **send_kwargs)
.../hostedtoolcache/Python/3.8.18.................................................../x64/lib/python3.8........./site-packages/requests/sessions.py:703: in send
    r = adapter.send(request, **kwargs)
.../hostedtoolcache/Python/3.8.18.................................................../x64/lib/python3.8.../great_expectations/core/http.py:39: in send
    return super().send(request, **kwargs)
.../hostedtoolcache/Python/3.8.18.................................................../x64/lib/python3.8....../site-packages/responses/__init__.py:1180: in send
    return self._on_request(adapter, request, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = &lt;responses.RequestsMock object at 0x7f86ffbb0340&gt;
adapter = &lt;great_expectations.core.http._TimeoutHTTPAdapter object at 0x7f86f2582af0&gt;
request = &lt;PreparedRequest [GET]&gt;, retries = None
kwargs = {'cert': None, 'proxies': OrderedDict(), 'stream': False, 'timeout': None, ...}
match = None
match_failed_reasons = ['URL does not match', 'URL does not match', 'URL does not match', 'Method does not match', 'URL does not match', 'Method does not match', ...]
resp_callback = None
error_msg = "Connection refused by Responses - the call doesn't match any registered mock.\n\nRequest: \n- GET https://app.greatex....../organizations/12345678-1234-5678-1234-567812345678/validation-definitions/[a-f0-9-]{36}') URL does not match\n"
i = 20, m = &lt;responses.CallbackResponse object at 0x7f8700aaee50&gt;

    def _on_request(
        self,
        adapter: "HTTPAdapter",
        request: "PreparedRequest",
        *,
        retries: Optional["_Retry"] = None,
        **kwargs: Any,
    ) -&gt; "models.Response":
        # add attributes params and req_kwargs to 'request' object for further match comparison
        # original request object does not have these attributes
        request.params = self._parse_request_params(request.path_url)  # type: ignore[attr-defined]
        request.req_kwargs = kwargs  # type: ignore[attr-defined]
        request_url = str(request.url)
    
        match, match_failed_reasons = self._find_match(request)
        resp_callback = self.response_callback
    
        if match is None:
            if any(
                [
                    p.match(request_url)
                    if isinstance(p, Pattern)
                    else request_url.startswith(p)
                    for p in self.passthru_prefixes
                ]
            ):
                logger.info("request.allowed-passthru", extra={"url": request_url})
                return self._real_send(adapter, request, **kwargs)  # type: ignore
    
            error_msg = (
                "Connection refused by Responses - the call doesn't "
                "match any registered mock.\n\n"
                "Request: \n"
                f"- {request.method} {request_url}\n\n"
                "Available matches:\n"
            )
            for i, m in enumerate(self.registered()):
                error_msg += "- {} {} {}\n".format(
                    m.method, m.url, match_failed_reasons[i]
                )
    
            if self.passthru_prefixes:
                error_msg += "Passthru prefixes:\n"
                for p in self.passthru_prefixes:
                    error_msg += f"- {p}\n"
    
            response = ConnectionError(error_msg)
            response.request = request
    
            self._calls.add(request, response)
&gt;           raise response
E           requests.exceptions.ConnectionError: Connection refused by Responses - the call doesn't match any registered mock.
E           
E           Request: 
E           - GET https://app.greatexpectations.fake..../12345678-1234-5678-1234-567812345678/accounts/me
E           
E           Available matches:
E           - GET https://app.greatexpectations.fake..../12345678-1234-5678-1234-567812345678/accounts/me URL does not match
E           - GET https://app.greatexpectations.fake..../organizations/12345678-1234-5678-1234-567812345678/data-context-configuration URL does not match
E           - GET https://app.greatexpectations.fake..../organizations/12345678-1234-5678-1234-567812345678/datasources URL does not match
E           - POST https://app.greatexpectations.fake..../organizations/12345678-1234-5678-1234-567812345678/datasources Method does not match
E           - GET re.compile('https://app.greatexpectations.fake..../organizations/12345678-1234-5678-1234-567812345678/datasources/[a-f0-9-]{36}') URL does not match
E           - DELETE re.compile('https://app.greatexpectations.fake..../organizations/12345678-1234-5678-1234-567812345678/datasources/[a-f0-9-]{36}') Method does not match
E           - PUT re.compile('https://app.greatexpectations.fake..../organizations/12345678-1234-5678-1234-567812345678/datasources/[a-f0-9-]{36}') Method does not match
E           - DELETE re.compile('https://app.greatexpectations.fake..../organizations/12345678-1234-5678-1234-567812345678/data-assets/[a-f0-9-]{36}') Method does not match
E           - GET https://app.greatexpectations.fake..../organizations/12345678-1234-5678-1234-567812345678/expectation-suites URL does not match
E           - GET re.compile('https://app.greatexpectations.fake..../organizations/12345678-1234-5678-1234-567812345678/expectation-suites/[a-f0-9-]{36}') URL does not match
E           - POST https://app.greatexpectations.fake..../organizations/12345678-1234-5678-1234-567812345678/expectation-suites Method does not match
E           - PUT re.compile('https://app.greatexpectations.fake..../organizations/12345678-1234-5678-1234-567812345678/expectation-suites/[a-f0-9-]{36}') Method does not match
E           - DELETE re.compile('https://app.greatexpectations.fake..../organizations/12345678-1234-5678-1234-567812345678/expectation-suites/[a-f0-9-]{36}') Method does not match
E           - GET https://app.greatexpectations.fake..../organizations/12345678-1234-5678-1234-567812345678/checkpoints URL does not match
E           - POST https://app.greatexpectations.fake..../organizations/12345678-1234-5678-1234-567812345678/checkpoints Method does not match
E           - DELETE re.compile('https://app.greatexpectations.fake..../organizations/12345678-1234-5678-1234-567812345678/checkpoints/[a-f0-9-]{36}') Method does not match
E           - DELETE https://app.greatexpectations.fake..../organizations/12345678-1234-5678-1234-567812345678/checkpoints Method does not match
E           - GET re.compile('https://app.greatexpectations.fake..../organizations/12345678-1234-5678-1234-567812345678/checkpoints/[a-f0-9-]{36}') URL does not match
E           - POST https://app.greatexpectations.fake.io.../organizations/12345678-1234-5678-1234-567812345678/validation-results Method does not match
E           - POST https://app.greatexpectations.fake.io.../organizations/12345678-1234-5678-1234-567812345678/validation-definitions Method does not match
E           - GET re.compile('https://app.greatexpectations.fake.io.../organizations/12345678-1234-5678-1234-567812345678/validation-definitions/[a-f0-9-]{36}') URL does not match

.../hostedtoolcache/Python/3.8.18.................................................../x64/lib/python3.8....../site-packages/responses/__init__.py:1107: ConnectionError

To view individual test run time comparison to the main branch, go to the Test Analytics Dashboard

billdirks

This looks great, thanks for the cleanup! One small comment about adding a new noqa.

billdirks · 2024-08-07T22:39:12Z

great_expectations/data_context/store/gx_cloud_store_backend.py

-                attributes_value=attributes_value,
-                resource_id=resource_id,
-                **kwargs,
+            raise TypeError(  # noqa: TRY003


We should avoid adding more noqa's to the code.

…_expectations into m/_/clean_up_gx_cloud_store_backend

clean up gx cloud store backend

6e35cfe

mypy

1722597

cdkini force-pushed the m/_/clean_up_gx_cloud_store_backend branch from e7fdb63 to 1722597 Compare August 7, 2024 22:09

revert

f710c0c

billdirks approved these changes Aug 7, 2024

View reviewed changes

Merge branch 'develop' of https://github.com/great-expectations/great…

a488829

…_expectations into m/_/clean_up_gx_cloud_store_backend

cdkini enabled auto-merge August 27, 2024 14:30

Merge branch 'develop' into m/_/clean_up_gx_cloud_store_backend

90f61e7

cdkini closed this Oct 1, 2024

auto-merge was automatically disabled October 1, 2024 16:18
Pull request was closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MAINTENANCE] Remove version-specific logic from GXCloudStoreBackend #10173

[MAINTENANCE] Remove version-specific logic from GXCloudStoreBackend #10173

cdkini commented Aug 6, 2024

netlify bot commented Aug 6, 2024 •

edited

Loading

codecov bot commented Aug 6, 2024 •

edited

Loading

billdirks left a comment

billdirks Aug 7, 2024

[MAINTENANCE] Remove version-specific logic from GXCloudStoreBackend #10173

[MAINTENANCE] Remove version-specific logic from GXCloudStoreBackend #10173

Conversation

cdkini commented Aug 6, 2024

netlify bot commented Aug 6, 2024 • edited Loading

✅ Deploy Preview for niobium-lead-7998 canceled.

codecov bot commented Aug 6, 2024 • edited Loading

❌ 1 Tests Failed:

billdirks left a comment

Choose a reason for hiding this comment

billdirks Aug 7, 2024

Choose a reason for hiding this comment

netlify bot commented Aug 6, 2024 •

edited

Loading

codecov bot commented Aug 6, 2024 •

edited

Loading