Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
add resource telemetry to pipes client resources (#17111)
## Summary & Motivation This PR adds telemetry tracking to the following resourcers: - PipesSubprocessClient - PipesK8sClient - PipesDockerClient - PipesDatabricksClient These resources are implemented in the `ResourceParam[TheBarePythonObject]` pattern such as [here](https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/subprocess.py#L113), which goes through the hardcoded_resource path in core. So this PR threads the info through hardcoded_resource into external resource data, in order to pass in the correct `dagster_maintained` value and the correct original resource type. I'm entirely not sure if this is a reasonable implementation to check in. Open to more reasonable paths! ## How I Tested These Changes in local: ```python defs = Definitions( assets=[my_asset], resources={ "pipes_subprocess_client": PipesSubprocessClient(), "pipes_k8s_client": PipesK8sClient(), "pipes_docker_client": PipesDockerClient(), "pipes_databricks_client": PipesDatabricksClient(client=MagicMock()), }, ) ``` writes the following telemetry: ```bash { 'action': 'update_repo_stats', 'client_time': '2023-10-09 22:28:46.098683', 'event_id': '18b2fdc3-8b80-407b-879e-010f8b05b8f5', 'elapsed_time': '', 'instance_id': '9ed76fef-a574-4c90-9356-05e0edaa0345', 'metadata': { 'dagster_resources': [ { 'module_name': 'dagster_databricks', 'class_name': '_PipesDatabricksClient' }, { 'module_name': 'dagster_docker', 'class_name': '_PipesDockerClient' }, { 'module_name': 'dagster_k8s', 'class_name': '_PipesK8sClient' }, { 'module_name': 'dagster', 'class_name': '_PipesSubprocess' } ], 'has_custom_resources': 'False', 'num_pipelines_in_repo': '1', 'num_schedules_in_repo': '0', 'num_sensors_in_repo': '0', 'num_assets_in_repo': '1', 'num_source_assets_in_repo': '0', 'num_partitioned_assets_in_repo': '0', 'num_dynamic_partitioned_assets_in_repo': '0', 'num_multi_partitioned_assets_in_repo': '0', 'num_assets_with_freshness_policies_in_repo': '0', 'num_assets_with_eager_auto_materialize_policies_in_repo': '0', 'num_assets_with_lazy_auto_materialize_policies_in_repo': '0', 'num_assets_with_single_run_backfill_policies_in_repo': '0', 'num_assets_with_multi_run_backfill_policies_in_repo': '0', 'num_observable_source_assets_in_repo': '0', 'num_dbt_assets_in_repo': '0', 'num_assets_with_code_versions_in_repo': '0', 'num_asset_reconciliation_sensors_in_repo': '0', 'num_asset_checks': '1', 'source': 'dagit', 'pipeline_name_hash': '', 'repo_hash': 'f17e9128abe12b4ff329425c469a7c5abc06bace32a2237848bc3a71cf9ef808', 'location_name_hash': '04af437226cf0d37c15f17e4082824fe53867e3ba8ffc35f46ad78a33cda2160' }, 'python_version': '3.9.16', 'dagster_version': '1!0+dev', 'os_desc': 'macOS-13.5-arm64-arm-64bit', 'os_platform': 'Darwin', 'run_storage_id': '', 'is_known_ci_env': 'False' } ```
- Loading branch information