Tons of errors with "FileNotFoundError: [Errno 2] No such file or directory"

[hebo-yang]

I installed the tool with %pip install git+https:///github.com/amesar/mlflow-export-import/#egg=mlflow-export-import to a Databricks notebook and ran export_experiments("all", "dbfs:/mnt/databricks-common-assets/ml_platform/e2_migration/mlflow_export/", True, notebook_formats="DBC", use_threads=True).

However it ran into so many "No such file or directory" error which is rather weird as fs.mkdirs(notebook_dir) was called before it.

Is there some problem with DBFS or S3 mounts that could have caused this?

Exporting run 111: f2754468f148478a8e8553a3884f32ec
Wrote 3241 bytes.
Wrote 8605 bytes.
ERROR: run_id: f2754468f148478a8e8553a3884f32ec Exception: [Errno 2] No such file or directory: 'dbfs:/mnt/databricks-common-assets/ml_platform/e2_migration/mlflow_export/1036469/f2754468f148478a8e8553a3884f32ec/artifacts/notebooks/manifest.json'
Traceback (most recent call last):
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-8ebc49b2-997d-4303-aa17-8b98b91cded1/lib/python3.8/site-packages/mlflow_export_import/run/export_run.py", line 76, in export_run
self.export_notebook(output_dir, notebook, run.data.tags, fs)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-8ebc49b2-997d-4303-aa17-8b98b91cded1/lib/python3.8/site-packages/mlflow_export_import/run/export_run.py", line 96, in export_notebook
with open(path, "w") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'dbfs:/mnt/databricks-common-assets/ml_platform/e2_migration/mlflow_export/1036469/f2754468f148478a8e8553a3884f32ec/artifacts/notebooks/manifest.json'

[amesar]

Some recent updates broke the notebooks. Will look at them shortly. In the meantime, you can export to your external node (laptop) and then import into destination workspace.

[hebo-yang]

Thanks Andre!

[amesar]

Checked in the fix. Refresh your notebooks from github too.

[hebo-yang]

@amesar Thanks for the quick fix! A few follow-ups:

1. I am running export_models("all", "/dbfs/mnt/databricks-common-assets/ml_platform/e2_migration/mlflow_export/2022_03_21", notebook_formats="DBC", stages="", export_all_runs=False, use_threads=True). It seems to only imported top 100 models. This might need to pass in a really large max_results

2. When I am importing with import_all("/dbfs/mnt/databricks-common-assets/ml_platform/e2_migration/mlflow_export/2022_03_21", delete_model=False, use_src_user_id=False, import_metadata_tags=True, verbose=False, use_threads=True)

There are lots of errors while creating models:
2022/03/22 04:23:22 ERROR mlflow.utils.rest_utils: API request to https://oregon.cloud.databricks.com/api/2.0/preview/mlflow/registered-models/create failed with code 500 != 200, retrying up to 2 more times. API response body: {"error_code":"INTERNAL_ERROR","message":"Got unexpected exception for resource 'Registered Model', key ac_bias_adj_model_test_v2_11"}

And also this final error:

/databricks/python/lib/python3.8/site-packages/mlflow_export_import/bulk/import_models.py in import_all(input_dir, delete_model, use_src_user_id, import_metadata_tags, verbose, use_threads)
     76     dct = { "duration": duration, "experiment_import": exp_res[1], "model_import": model_res }
     77     fs = _filesystem.get_filesystem(".")
---> 78     utils.write_json_file(fs, "import_report.json", dct)
     79     print("\nImport report:")
     80     print(json.dumps(dct,indent=2)+"\n")

/databricks/python/lib/python3.8/site-packages/mlflow_export_import/utils.py in write_json_file(fs, path, dct)
     68 
     69 def write_json_file(fs, path, dct):
---> 70     fs.write(path, json.dumps(dct,indent=2)+"\n")
     71 
     72 def write_file(path, content):

/usr/lib/python3.8/json/__init__.py in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
    232     if cls is None:
    233         cls = JSONEncoder
--> 234     return cls(
    235         skipkeys=skipkeys, ensure_ascii=ensure_ascii,
    236         check_circular=check_circular, allow_nan=allow_nan, indent=indent,

/usr/lib/python3.8/json/encoder.py in encode(self, o)
    199         chunks = self.iterencode(o, _one_shot=True)
    200         if not isinstance(chunks, (list, tuple)):
--> 201             chunks = list(chunks)
    202         return ''.join(chunks)
    203 

/usr/lib/python3.8/json/encoder.py in _iterencode(o, _current_indent_level)
    429             yield from _iterencode_list(o, _current_indent_level)
    430         elif isinstance(o, dict):
--> 431             yield from _iterencode_dict(o, _current_indent_level)
    432         else:
    433             if markers is not None:

/usr/lib/python3.8/json/encoder.py in _iterencode_dict(dct, _current_indent_level)
    403                 else:
    404                     chunks = _iterencode(value, _current_indent_level)
--> 405                 yield from chunks
    406         if newline_indent is not None:
    407             _current_indent_level -= 1

/usr/lib/python3.8/json/encoder.py in _iterencode_dict(dct, _current_indent_level)
    403                 else:
    404                     chunks = _iterencode(value, _current_indent_level)
--> 405                 yield from chunks
    406         if newline_indent is not None:
    407             _current_indent_level -= 1

/usr/lib/python3.8/json/encoder.py in _iterencode_list(lst, _current_indent_level)
    323                 else:
    324                     chunks = _iterencode(value, _current_indent_level)
--> 325                 yield from chunks
    326         if newline_indent is not None:
    327             _current_indent_level -= 1

/usr/lib/python3.8/json/encoder.py in _iterencode(o, _current_indent_level)
    436                     raise ValueError("Circular reference detected")
    437                 markers[markerid] = o
--> 438             o = _default(o)
    439             yield from _iterencode(o, _current_indent_level)
    440             if markers is not None:

/usr/lib/python3.8/json/encoder.py in default(self, o)
    177 
    178         """
--> 179         raise TypeError(f'Object of type {o.__class__.__name__} '
    180                         f'is not JSON serializable')
    181 

TypeError: Object of type MlflowExportImportException is not JSON serializable

[amesar]

It seems to only imported top 100 models. This might need to pass in a really large max_results

Good catch. I'll look into it.

[amesar]

I checked in a fix with issue #51. Try it again.

[hebo-yang]

Thanks for the quick fix! Wondering if you know what this error is about please?
Should it be calling "https://oregon.cloud.databricks.com/api/2.0/workspace/mkdirs" instead of our own databricks host?
I had export DATABRICKS_HOST=https://{my-comanpy}.cloud.databricks.com and also set the same env in python

Creating Databricks workspace directory '/Users/[email protected]/Mx'
Traceback (most recent call last):
  File "/databricks/python/lib/python3.8/site-packages/mlflow_export_import/bulk/import_models.py", line 39, in import_experiments
    _run_info_map = importer.import_experiment( exp["name"], exp_input_dir)
  File "/databricks/python/lib/python3.8/site-packages/mlflow_export_import/experiment/import_experiment.py", line 36, in import_experiment
    mlflow_utils.set_experiment(self.dbx_client, exp_name)
  File "/databricks/python/lib/python3.8/site-packages/mlflow_export_import/common/mlflow_utils.py", line 57, in set_experiment
    create_workspace_dir(dbx_client, os.path.dirname(exp_name))
  File "/databricks/python/lib/python3.8/site-packages/mlflow_export_import/common/mlflow_utils.py", line 48, in create_workspace_dir
    dbx_client.post("workspace/mkdirs", { "path": workspace_dir })
  File "/databricks/python/lib/python3.8/site-packages/mlflow_export_import/common/http_client.py", line 50, in post
    return json.loads(self._post(resource, data).text)
  File "/databricks/python/lib/python3.8/site-packages/mlflow_export_import/common/http_client.py", line 46, in _post
    self._check_response(rsp,uri)
  File "/databricks/python/lib/python3.8/site-packages/mlflow_export_import/common/http_client.py", line 63, in _check_response
    raise MlflowExportImportException(f"HTTP status code: {rsp.status_code}. Reason: {rsp.reason}. URI: {uri}. Params: {params}.")
mlflow_export_import.common.MlflowExportImportException: HTTP status code: 400. Reason: Bad Request. URI: https://oregon.cloud.databricks.com/api/2.0/workspace/mkdirs. Params: None.

[amesar]

Not quite clear what the issue is. To externally access Databricks MLflow see https://github.com/amesar/mlflow-resources/blob/master/MLflow_FAQ.md#how-do-i-access-databricks-mlflow-from-outside-databricks

[hebo-yang]

I ran everything inside the databricks cluster & notebook. Not sure why it is trying to connect to "https://oregon.cloud.databricks.com/api/2.0/workspace/mkdirs"

[amesar]

https://docs.databricks.com/dev-tools/api/latest/workspace.html#mkdirs is legitimate endpoint. Needs to create the target folder if it doesn't exist.

[hebo-yang]

Yes, the endpoint is legitimate but I thought it should be made to our databricks host instead of https://oregon.cloud.databricks.com?

[hebo-yang]

Hi @amesar, why is this closed please?

[amesar]

Best to send me a zip file of your exported directory: andre at databricks.