From 60855565e03b70d88562c50efbb8f0e6c3a4ab48 Mon Sep 17 00:00:00 2001 From: ZePing Guo Date: Thu, 2 Jan 2025 12:19:08 +0800 Subject: [PATCH 1/9] doc --- docs/source/examples/managed-jobs.rst | 34 ++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/docs/source/examples/managed-jobs.rst b/docs/source/examples/managed-jobs.rst index 99fa461249d..379a8aaaeaa 100644 --- a/docs/source/examples/managed-jobs.rst +++ b/docs/source/examples/managed-jobs.rst @@ -152,6 +152,39 @@ The :code:`MOUNT` mode in :ref:`SkyPilot bucket mounting ` ensures Note that the application code should save program checkpoints periodically and reload those states when the job is restarted. This is typically achieved by reloading the latest checkpoint at the beginning of your program. + +.. _intermediate-bucket: + +Intermediate Bucket +~~~~~~~~~~~~~~~~~~~~ + +For managed jobs, SkyPilot requires a bucket to store intermediate files, such as local file mounts, temporary files, and the workdir. +If you do not configure a bucket, SkyPilot will automatically create a default temporary bucket named :code:`skypilot-filemounts-{username}-{run_id}` for each job launch. +Alternatively, you can specify a persistent bucket using :code:`jobs.bucket` in :code:`~/.sky/config.yaml`. If you choose to specify a bucket, ensure that the bucket already exists and that you have the necessary permissions. + +.. code-block:: yaml + + # ~/.sky/config.yaml + jobs: + bucket: s3://my-bucket # Supports s3://, gs://, https://.blob.core.windows.net/, r2://, cos:/// + +When using :code:`jobs.bucket`, SkyPilot organizes files in the following structure: + +.. code-block:: text + + # cloud bucket, s3://my-bucket/ for example + my-bucket/ + ├── job-15891b25/ # Job-specific directory + │ ├── local-file-mounts/ # Files from local file mounts + │ ├── tmp-files/ # Temporary files + │ └── workdir/ # Files from workdir + └── job-cae228be/ # Another job's directory + ├── local-file-mounts/ + ├── tmp-files/ + └── workdir/ + +SkyPilot automatically deletes the entire bucket after the job completes if no bucket is specified. When using a custom bucket (:code:`jobs.bucket`), only the job-specific directories (e.g., :code:`job-15891b25/`) are cleaned up after job completion, while the bucket itself persists. + .. _spot-jobs-end-to-end: An End-to-End Example @@ -505,4 +538,3 @@ The :code:`resources` field has the same spec as a normal SkyPilot job; see `her These settings will not take effect if you have an existing controller (either stopped or live). For them to take effect, tear down the existing controller first, which requires all in-progress jobs to finish or be canceled. - From 7dc615c58bd5a37861856423d78c424374ea10a9 Mon Sep 17 00:00:00 2001 From: zpoint Date: Thu, 2 Jan 2025 14:58:12 +0800 Subject: [PATCH 2/9] Update docs/source/examples/managed-jobs.rst Co-authored-by: Romil Bhardwaj --- docs/source/examples/managed-jobs.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/source/examples/managed-jobs.rst b/docs/source/examples/managed-jobs.rst index 379a8aaaeaa..2fce7ae5939 100644 --- a/docs/source/examples/managed-jobs.rst +++ b/docs/source/examples/managed-jobs.rst @@ -160,7 +160,8 @@ Intermediate Bucket For managed jobs, SkyPilot requires a bucket to store intermediate files, such as local file mounts, temporary files, and the workdir. If you do not configure a bucket, SkyPilot will automatically create a default temporary bucket named :code:`skypilot-filemounts-{username}-{run_id}` for each job launch. -Alternatively, you can specify a persistent bucket using :code:`jobs.bucket` in :code:`~/.sky/config.yaml`. If you choose to specify a bucket, ensure that the bucket already exists and that you have the necessary permissions. + +Alternatively, you can pre-provision a bucket and use it as an intermediate for storing file by setting :code:`jobs.bucket` in :code:`~/.sky/config.yaml`. If you choose to specify a bucket, ensure that the bucket already exists and that you have the necessary permissions. .. code-block:: yaml From 78d460016d6bb0a418802266948e1604cfb93326 Mon Sep 17 00:00:00 2001 From: zpoint Date: Thu, 2 Jan 2025 14:58:38 +0800 Subject: [PATCH 3/9] Update docs/source/examples/managed-jobs.rst Co-authored-by: Romil Bhardwaj --- docs/source/examples/managed-jobs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/examples/managed-jobs.rst b/docs/source/examples/managed-jobs.rst index 2fce7ae5939..bcb049a5d05 100644 --- a/docs/source/examples/managed-jobs.rst +++ b/docs/source/examples/managed-jobs.rst @@ -155,7 +155,7 @@ This is typically achieved by reloading the latest checkpoint at the beginning o .. _intermediate-bucket: -Intermediate Bucket +Intermediate storage for files ~~~~~~~~~~~~~~~~~~~~ For managed jobs, SkyPilot requires a bucket to store intermediate files, such as local file mounts, temporary files, and the workdir. From fb28ffb7087d4749c39270e90f8fafa4eb90be6d Mon Sep 17 00:00:00 2001 From: zpoint Date: Thu, 2 Jan 2025 14:58:59 +0800 Subject: [PATCH 4/9] Update docs/source/examples/managed-jobs.rst Co-authored-by: Romil Bhardwaj --- docs/source/examples/managed-jobs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/examples/managed-jobs.rst b/docs/source/examples/managed-jobs.rst index bcb049a5d05..852d1260c2d 100644 --- a/docs/source/examples/managed-jobs.rst +++ b/docs/source/examples/managed-jobs.rst @@ -158,7 +158,7 @@ This is typically achieved by reloading the latest checkpoint at the beginning o Intermediate storage for files ~~~~~~~~~~~~~~~~~~~~ -For managed jobs, SkyPilot requires a bucket to store intermediate files, such as local file mounts, temporary files, and the workdir. +For managed jobs, SkyPilot requires an intermediate bucket to store files used in the task, such as local file mounts, temporary files, and the workdir. If you do not configure a bucket, SkyPilot will automatically create a default temporary bucket named :code:`skypilot-filemounts-{username}-{run_id}` for each job launch. Alternatively, you can pre-provision a bucket and use it as an intermediate for storing file by setting :code:`jobs.bucket` in :code:`~/.sky/config.yaml`. If you choose to specify a bucket, ensure that the bucket already exists and that you have the necessary permissions. From e6eca837604fceffd5f136dd337b2f72c089a4eb Mon Sep 17 00:00:00 2001 From: zpoint Date: Thu, 2 Jan 2025 14:59:16 +0800 Subject: [PATCH 5/9] Update docs/source/examples/managed-jobs.rst Co-authored-by: Romil Bhardwaj --- docs/source/examples/managed-jobs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/examples/managed-jobs.rst b/docs/source/examples/managed-jobs.rst index 852d1260c2d..50ee7407e42 100644 --- a/docs/source/examples/managed-jobs.rst +++ b/docs/source/examples/managed-jobs.rst @@ -184,7 +184,7 @@ When using :code:`jobs.bucket`, SkyPilot organizes files in the following struct ├── tmp-files/ └── workdir/ -SkyPilot automatically deletes the entire bucket after the job completes if no bucket is specified. When using a custom bucket (:code:`jobs.bucket`), only the job-specific directories (e.g., :code:`job-15891b25/`) are cleaned up after job completion, while the bucket itself persists. +When using a custom bucket (:code:`jobs.bucket`), the job-specific directories (e.g., :code:`job-15891b25/`) created by SkyPilot are removed when the job completes. .. _spot-jobs-end-to-end: From ea1c18983a407e715299cafe7af78b283ea29e82 Mon Sep 17 00:00:00 2001 From: zpoint Date: Thu, 2 Jan 2025 14:59:45 +0800 Subject: [PATCH 6/9] Update docs/source/examples/managed-jobs.rst Co-authored-by: Romil Bhardwaj --- docs/source/examples/managed-jobs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/examples/managed-jobs.rst b/docs/source/examples/managed-jobs.rst index 50ee7407e42..374a06e21f0 100644 --- a/docs/source/examples/managed-jobs.rst +++ b/docs/source/examples/managed-jobs.rst @@ -169,7 +169,7 @@ Alternatively, you can pre-provision a bucket and use it as an intermediate for jobs: bucket: s3://my-bucket # Supports s3://, gs://, https://.blob.core.windows.net/, r2://, cos:/// -When using :code:`jobs.bucket`, SkyPilot organizes files in the following structure: +When using a pre-provisioned intermediate bucket with :code:`jobs.bucket`, SkyPilot creates job-specific directories under the bucket root to store files. They are organized in the following structure: .. code-block:: text From 81253ca2f625e7d688fe6b703537391bcb9a47c2 Mon Sep 17 00:00:00 2001 From: zpoint Date: Thu, 2 Jan 2025 15:11:11 +0800 Subject: [PATCH 7/9] Update docs/source/examples/managed-jobs.rst Co-authored-by: Romil Bhardwaj --- docs/source/examples/managed-jobs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/examples/managed-jobs.rst b/docs/source/examples/managed-jobs.rst index 374a06e21f0..4000696cc13 100644 --- a/docs/source/examples/managed-jobs.rst +++ b/docs/source/examples/managed-jobs.rst @@ -159,7 +159,7 @@ Intermediate storage for files ~~~~~~~~~~~~~~~~~~~~ For managed jobs, SkyPilot requires an intermediate bucket to store files used in the task, such as local file mounts, temporary files, and the workdir. -If you do not configure a bucket, SkyPilot will automatically create a default temporary bucket named :code:`skypilot-filemounts-{username}-{run_id}` for each job launch. +If you do not configure a bucket, SkyPilot will automatically create a temporary bucket named :code:`skypilot-filemounts-{username}-{run_id}` for each job launch. SkyPilot automatically deletes the bucket after the job completes. Alternatively, you can pre-provision a bucket and use it as an intermediate for storing file by setting :code:`jobs.bucket` in :code:`~/.sky/config.yaml`. If you choose to specify a bucket, ensure that the bucket already exists and that you have the necessary permissions. From e9eee76d59721aa9be4c2b0635589848fc896252 Mon Sep 17 00:00:00 2001 From: ZePing Guo Date: Thu, 2 Jan 2025 15:18:26 +0800 Subject: [PATCH 8/9] add tip --- docs/source/examples/managed-jobs.rst | 70 ++++++++++++++------------- 1 file changed, 37 insertions(+), 33 deletions(-) diff --git a/docs/source/examples/managed-jobs.rst b/docs/source/examples/managed-jobs.rst index 4000696cc13..b357f00434a 100644 --- a/docs/source/examples/managed-jobs.rst +++ b/docs/source/examples/managed-jobs.rst @@ -153,39 +153,6 @@ Note that the application code should save program checkpoints periodically and This is typically achieved by reloading the latest checkpoint at the beginning of your program. -.. _intermediate-bucket: - -Intermediate storage for files -~~~~~~~~~~~~~~~~~~~~ - -For managed jobs, SkyPilot requires an intermediate bucket to store files used in the task, such as local file mounts, temporary files, and the workdir. -If you do not configure a bucket, SkyPilot will automatically create a temporary bucket named :code:`skypilot-filemounts-{username}-{run_id}` for each job launch. SkyPilot automatically deletes the bucket after the job completes. - -Alternatively, you can pre-provision a bucket and use it as an intermediate for storing file by setting :code:`jobs.bucket` in :code:`~/.sky/config.yaml`. If you choose to specify a bucket, ensure that the bucket already exists and that you have the necessary permissions. - -.. code-block:: yaml - - # ~/.sky/config.yaml - jobs: - bucket: s3://my-bucket # Supports s3://, gs://, https://.blob.core.windows.net/, r2://, cos:/// - -When using a pre-provisioned intermediate bucket with :code:`jobs.bucket`, SkyPilot creates job-specific directories under the bucket root to store files. They are organized in the following structure: - -.. code-block:: text - - # cloud bucket, s3://my-bucket/ for example - my-bucket/ - ├── job-15891b25/ # Job-specific directory - │ ├── local-file-mounts/ # Files from local file mounts - │ ├── tmp-files/ # Temporary files - │ └── workdir/ # Files from workdir - └── job-cae228be/ # Another job's directory - ├── local-file-mounts/ - ├── tmp-files/ - └── workdir/ - -When using a custom bucket (:code:`jobs.bucket`), the job-specific directories (e.g., :code:`job-15891b25/`) created by SkyPilot are removed when the job completes. - .. _spot-jobs-end-to-end: An End-to-End Example @@ -489,6 +456,43 @@ especially useful when there are many in-progress jobs to monitor, which the terminal-based CLI may need more than one page to display. +.. _intermediate-bucket: + +Intermediate storage for files +------------------------------ + +For managed jobs, SkyPilot requires an intermediate bucket to store files used in the task, such as local file mounts, temporary files, and the workdir. +If you do not configure a bucket, SkyPilot will automatically create a temporary bucket named :code:`skypilot-filemounts-{username}-{run_id}` for each job launch. SkyPilot automatically deletes the bucket after the job completes. + +Alternatively, you can pre-provision a bucket and use it as an intermediate for storing file by setting :code:`jobs.bucket` in :code:`~/.sky/config.yaml`. If you choose to specify a bucket, ensure that the bucket already exists and that you have the necessary permissions. + +.. code-block:: yaml + + # ~/.sky/config.yaml + jobs: + bucket: s3://my-bucket # Supports s3://, gs://, https://.blob.core.windows.net/, r2://, cos:/// + +When using a pre-provisioned intermediate bucket with :code:`jobs.bucket`, SkyPilot creates job-specific directories under the bucket root to store files. They are organized in the following structure: + +.. code-block:: text + + # cloud bucket, s3://my-bucket/ for example + my-bucket/ + ├── job-15891b25/ # Job-specific directory + │ ├── local-file-mounts/ # Files from local file mounts + │ ├── tmp-files/ # Temporary files + │ └── workdir/ # Files from workdir + └── job-cae228be/ # Another job's directory + ├── local-file-mounts/ + ├── tmp-files/ + └── workdir/ + +When using a custom bucket (:code:`jobs.bucket`), the job-specific directories (e.g., :code:`job-15891b25/`) created by SkyPilot are removed when the job completes. + +.. tip:: + Multiple users can share the same intermediate bucket. Each user's jobs will have their own unique job-specific directories, ensuring that files are kept separate and organized. + + Concept: Jobs Controller ------------------------ From d14e1a51d5d5ce5b052863dd6f5823f9b6588bf2 Mon Sep 17 00:00:00 2001 From: Romil Bhardwaj Date: Fri, 3 Jan 2025 07:25:08 -0800 Subject: [PATCH 9/9] minor changes --- docs/source/examples/managed-jobs.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/source/examples/managed-jobs.rst b/docs/source/examples/managed-jobs.rst index b357f00434a..2cd99b6c24b 100644 --- a/docs/source/examples/managed-jobs.rst +++ b/docs/source/examples/managed-jobs.rst @@ -464,7 +464,7 @@ Intermediate storage for files For managed jobs, SkyPilot requires an intermediate bucket to store files used in the task, such as local file mounts, temporary files, and the workdir. If you do not configure a bucket, SkyPilot will automatically create a temporary bucket named :code:`skypilot-filemounts-{username}-{run_id}` for each job launch. SkyPilot automatically deletes the bucket after the job completes. -Alternatively, you can pre-provision a bucket and use it as an intermediate for storing file by setting :code:`jobs.bucket` in :code:`~/.sky/config.yaml`. If you choose to specify a bucket, ensure that the bucket already exists and that you have the necessary permissions. +Alternatively, you can pre-provision a bucket and use it as an intermediate for storing file by setting :code:`jobs.bucket` in :code:`~/.sky/config.yaml`: .. code-block:: yaml @@ -472,6 +472,9 @@ Alternatively, you can pre-provision a bucket and use it as an intermediate for jobs: bucket: s3://my-bucket # Supports s3://, gs://, https://.blob.core.windows.net/, r2://, cos:/// + +If you choose to specify a bucket, ensure that the bucket already exists and that you have the necessary permissions. + When using a pre-provisioned intermediate bucket with :code:`jobs.bucket`, SkyPilot creates job-specific directories under the bucket root to store files. They are organized in the following structure: .. code-block:: text