Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNOW-1418543 make local testing thread safe #2185

Merged
merged 17 commits into from
Sep 11, 2024

Conversation

sfc-gh-aalam
Copy link
Contributor

@sfc-gh-aalam sfc-gh-aalam commented Aug 28, 2024

  1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

    Fixes SNOW-1418543

  2. Fill out the following pre-review checklist:

    • I am adding a new automated test(s) to verify correctness of my new code
      • If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
    • I am adding new logging messages
    • I am adding a new telemetry message
    • I am adding new credentials
    • I am adding a new dependency
    • If this is a new feature/behavior, I'm adding the Local Testing parity changes.
  3. Please describe how your code solves the related issue.

    Note: For easy reviewing, enable hide whitespace when viewing the gitdiff.

    This PR adds locks to connection, oob_telemetry and MockFunctionRegistry to ensure updates to local testing objects are thread-safe. A list of updates that are now protected are

  • _connection.py:
    • MockServerConnection
      • _get_client_side_session_parameter, _get_current_parameter
      • close
    • TabularEntityRegistry
      • methods to check if table/view exists
      • read/write/drop operations to table/views
      • add new methods read_view_if_exists and read_table_if_exists to atomically read table/view if they exist
  • _functions.py:
    • MockedFunctionRegistry
      • create a single instance of class
      • get/register/unregister are protected
  • _plan.py
    • execute_mock_plan
      • use atomic operations to check and read table/view from entity registry
      • read/update/merge/delete table atomically
    • handle_udf_expression: udf and imports are read atomically
    • handle_function_expression
  • _stage_registry.py
    • StageEntityRegistry
      • get/put read and upload are protected.
      • StageEntity class is not updated
  • _stored_procedure.py and _udf.py
    • registration, adding/clearing imports and importing files are lock protected
    • calling stored proc is also lock protected
  • _telemetry.py
    • add, flush and export_queue_to_string are lock protected
  • session.py
    • updating connection attributes like _active_schema, _active_database are lock protected

Sorry, something went wrong.

@github-actions github-actions bot added the local testing Local Testing issues/PRs label Aug 28, 2024
@sfc-gh-aalam sfc-gh-aalam changed the title Aalam snow 1418543 make local testing thread safe SNOW-1418543 make local testing thread safe Aug 28, 2024
@sfc-gh-aalam sfc-gh-aalam added the NO-CHANGELOG-UPDATES This pull request does not need to update CHANGELOG.md label Aug 30, 2024
@sfc-gh-aalam sfc-gh-aalam marked this pull request as ready for review September 3, 2024 20:35
@sfc-gh-aalam sfc-gh-aalam requested a review from a team as a code owner September 3, 2024 20:35
Copy link
Contributor

@sfc-gh-jrose sfc-gh-jrose left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you've hit all the places that I would expect to have a lock. I'm not a fan of the analyzer.session._conn._lock syntax used in a few places. Maybe have connections expose there lock so you don't have to access a private variable.

Copy link
Contributor

@sfc-gh-aling sfc-gh-aling left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great job, thanks for updating the local testing part

@sfc-gh-aalam sfc-gh-aalam merged commit 7d54d20 into main Sep 11, 2024
34 checks passed
@sfc-gh-aalam sfc-gh-aalam deleted the aalam-SNOW-1418543-make-local-testing-thread-safe branch September 11, 2024 18:07
@github-actions github-actions bot locked and limited conversation to collaborators Sep 11, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
local testing Local Testing issues/PRs NO-CHANGELOG-UPDATES This pull request does not need to update CHANGELOG.md
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants