Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Moved import and export jobs into their own section in croud docs. #468

Merged
merged 2 commits into from
Oct 31, 2023
Merged

Moved import and export jobs into their own section in croud docs. #468

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0ace7a3
Moved import and export jobs into their own section in croud docs.
SStorm Oct 30, 2023
0f9da73
Removed the bytes imported counter (no longer used)
SStorm Oct 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGES.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ Unreleased

- Added support for showing import job progress.

- Added the ``--transformations`` argument for import jobs.

- Moved import and export jobs into their own section in croud docs.

1.7.0 - 2023/09/11
==================

Expand Down
10 changes: 10 additions & 0 deletions croud/__main__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,16 @@
help="Whether the table should be created automatically"
" if it does not exist.",
),
Argument(
"--transformations",
type=str,
required=False,
help="The transformations to apply when fetching data. This is the SELECT "
"statement from an SQL query that is executed on the internal DuckDB "
"database that the data is loaded to before inserting into CrateDB. "
"This can be used to apply arbitrary SQL functions on your data before "
"inserting into CrateDB, i.e. `UNNEST()`, `SUM()` and similar.",
),
]

# fmt: off
Expand Down
18 changes: 8 additions & 10 deletions croud/clusters/commands.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,9 @@ def import_jobs_create(args: Namespace, extra_payload: Dict[str, Any]) -> None:
if args.create_table is not None:
body["destination"]["create_table"] = args.create_table

if args.transformations:
body["schema"] = {"select": args.transformations}

if extra_payload:
body.update(extra_payload)

Expand Down Expand Up @@ -845,11 +848,6 @@ def _get_import_job_operation_status(
return status, msg, feedback_data


def _get_formatted_size(feedback: dict) -> str:
num_bytes = feedback.get("progress", {}).get("bytes")
return bitmath.Byte(num_bytes).best_prefix().format("{value:.2f} {unit}")


def _get_formatted_records_normalized(feedback: dict) -> str:
num_records = feedback.get("progress", {}).get("records")

Expand All @@ -865,13 +863,13 @@ def _get_formatted_records_normalized(feedback: dict) -> str:

def _data_job_feedback_func(status: str, feedback: dict, job_type: str):
records_normalized = _get_formatted_records_normalized(feedback)
size = _get_formatted_size(feedback)
percent = "{:.2f}".format(feedback.get("progress", {}).get("percent", 0))

if status == "SUCCEEDED":
print_info(f"Done {job_type}ing {records_normalized} records and {size}.")
print_info(f"Done {job_type}ing {records_normalized} records")
else:
print_info(
f"{job_type}ing... {records_normalized} records and {size} {job_type}ed "
f"{job_type}ing... {records_normalized} ({percent}%) records {job_type}ed "
"so far."
)

Expand Down Expand Up @@ -975,8 +973,8 @@ def _wait_for_completed_operation(

# Call for custom feedback if function available and there is status to report.
if status in ["IN_PROGRESS", "SUCCEEDED"] and feedback_func:
(feedback_func, feedback_args) = feedback_func
feedback_func(status, feedback, *feedback_args)
(feedback_f, feedback_args) = feedback_func
feedback_f(status, feedback, *feedback_args)

# Final statuses
if status == "SUCCEEDED":
Expand Down
270 changes: 0 additions & 270 deletions docs/commands/clusters.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -586,274 +586,4 @@ Example
to organization and project admins.


``clusters import-jobs``
========================

.. argparse::
:module: croud.__main__
:func: get_parser
:prog: croud
:path: clusters import-jobs
:nosubcommands:



``clusters import-jobs create``
===============================

.. argparse::
:module: croud.__main__
:func: get_parser
:prog: croud
:path: clusters import-jobs create
:nosubcommands:


``clusters import-jobs create from-url``
========================================

.. argparse::
:module: croud.__main__
:func: get_parser
:prog: croud
:path: clusters import-jobs create from-url

Example
-------

.. code-block:: console

sh$ croud clusters import-jobs create from-url --cluster-id e1e38d92-a650-48f1-8a70-8133f2d5c400 \
--file-format csv --table my_table_name --url https://s3.amazonaws.com/my.import.data.gz --compression gzip
+--------------------------------------+--------------------------------------+------------+
| id | cluster_id | status |
|--------------------------------------+--------------------------------------+------------|
| dca4986d-f7c8-4121-af81-863cca1dab0f | e1e38d92-a650-48f1-8a70-8133f2d5c400 | REGISTERED |
+--------------------------------------+--------------------------------------+------------+
==> Info: Status: REGISTERED (Your import job was received and is pending processing.)
==> Info: Done importing 3 records and 36 Bytes.
==> Success: Operation completed.


``clusters import-jobs create from-file``
=========================================

.. argparse::
:module: croud.__main__
:func: get_parser
:prog: croud
:path: clusters import-jobs create from-file

.. code-block:: console

sh$ croud clusters import-jobs create from-file --cluster-id e1e38d92-a650-48f1-8a70-8133f2d5c400 \
--file-format csv --table my_table_name --file-id 2e71e5a6-a21a-4e99-ae58-705a1f15635c
+--------------------------------------+--------------------------------------+------------+
| id | cluster_id | status |
|--------------------------------------+--------------------------------------+------------|
| 9164f886-ae37-4a1b-b3fe-53f9e1897e7d | e1e38d92-a650-48f1-8a70-8133f2d5c400 | REGISTERED |
+--------------------------------------+--------------------------------------+------------+
==> Info: Status: REGISTERED (Your import job was received and is pending processing.)
==> Info: Done importing 3 records and 36 Bytes.
==> Success: Operation completed.


``clusters import-jobs create from-s3``
=======================================

.. argparse::
:module: croud.__main__
:func: get_parser
:prog: croud
:path: clusters import-jobs create from-s3

.. code-block:: console

sh$ croud clusters import-jobs create from-s3 --cluster-id e1e38d92-a650-48f1-8a70-8133f2d5c400 \
--secret-id 71e7c5da-51fa-44f2-b178-d95052cbe620 --bucket cratedbtestbucket \
--file-path myfiles/cratedbimporttest.csv --file-format csv --table my_table_name
+--------------------------------------+--------------------------------------+------------+
| id | cluster_id | status |
|--------------------------------------+--------------------------------------+------------|
| f29fdc02-edd0-4ad9-8839-9616fccf752b | e1e38d92-a650-48f1-8a70-8133f2d5c400 | REGISTERED |
+--------------------------------------+--------------------------------------+------------+
==> Info: Status: REGISTERED (Your import job was received and is pending processing.)
==> Info: Done importing 3 records and 36 Bytes.
==> Success: Operation completed.


``clusters import-jobs list``
=============================

.. argparse::
:module: croud.__main__
:func: get_parser
:prog: croud
:path: clusters import-jobs list

Example
-------

.. code-block:: console

sh$ ❯ croud clusters import-jobs list --cluster-id e1e38d92-a650-48f1-8a70-8133f2d5c400
+--------------------------------------+--------------------------------------+-----------+--------+-------------------+
| id | cluster_id | status | type | destination |
|--------------------------------------+--------------------------------------+-----------+--------+-------------------|
| dca4986d-f7c8-4121-af81-863cca1dab0f | e1e38d92-a650-48f1-8a70-8133f2d5c400 | SUCCEEDED | url | my_table_name |
| 00de6048-3af6-41da-bfaa-661199d1c106 | e1e38d92-a650-48f1-8a70-8133f2d5c400 | SUCCEEDED | s3 | my_table_name |
| 035f5ec1-ba9e-4a5c-9ce1-44e9a9cab6c1 | e1e38d92-a650-48f1-8a70-8133f2d5c400 | SUCCEEDED | file | my_table_name |
+--------------------------------------+--------------------------------------+-----------+--------+-------------------+


``clusters import-jobs delete``
===============================

.. argparse::
:module: croud.__main__
:func: get_parser
:prog: croud
:path: clusters import-jobs delete

Example
-------

.. code-block:: console

sh$ ❯ croud clusters import-jobs delete \
--cluster-id e1e38d92-a650-48f1-8a70-8133f2d5c400 \
--import-job-id 00de6048-3af6-41da-bfaa-661199d1c106
==> Success: Success.


``clusters import-jobs progress``
=================================

.. argparse::
:module: croud.__main__
:func: get_parser
:prog: croud
:path: clusters import-jobs progress

Examples
--------

.. code-block:: console

sh$ ❯ croud clusters import-jobs progress \
--cluster-id e1e38d92-a650-48f1-8a70-8133f2d5c400 \
--import-job-id 00de6048-3af6-41da-bfaa-661199d1c106 \
--summary true
+-----------+-----------+------------------+-----------------+---------------+
| percent | records | failed_records | total_records | total_files |
|-----------+-----------+------------------+-----------------+---------------+
| 100 | 891 | 0 | 891 | 2 |
+-----------+-----------+------------------+-----------------+---------------+


.. code-block:: console

sh$ ❯ croud clusters import-jobs progress \
--cluster-id e1e38d92-a650-48f1-8a70-8133f2d5c400 \
--import-job-id 00de6048-3af6-41da-bfaa-661199d1c106 \
--limit ALL
--offset 0
+-----------+-----------+-----------+------------------+-----------------+
| name | percent | records | failed_records | total_records |
|-----------+-----------+-----------+------------------+-----------------|
| file1.csv | 100 | 800 | 0 | 800 |
| file2.csv | 100 | 91 | 0 | 91 |
+-----------+-----------+-----------+------------------+-----------------+


``clusters export-jobs``
========================

.. argparse::
:module: croud.__main__
:func: get_parser
:prog: croud
:path: clusters export-jobs
:nosubcommands:


``clusters export-jobs create``
===============================

.. argparse::
:module: croud.__main__
:func: get_parser
:prog: croud
:path: clusters export-jobs create

Example
-------

.. code-block:: console

sh$ ❯ croud clusters export-jobs create --cluster-id f6c39580-5719-431d-a508-0cee4f9e8209 \
--table nyc_taxi --file-format csv
+--------------------------------------+--------------------------------------+------------+
| id | cluster_id | status |
|--------------------------------------+--------------------------------------+------------|
| 85dc0024-b049-4b9d-b100-4bf850881692 | f6c39580-5719-431d-a508-0cee4f9e8209 | REGISTERED |
+--------------------------------------+--------------------------------------+------------+
==> Info: Status: SENT (Your creation request was sent to the region.)
==> Info: Status: IN_PROGRESS (Export in progress)
==> Info: Exporting... 2.00 K records and 19.53 KiB exported so far.
==> Info: Exporting... 4.00 K records and 39.06 KiB exported so far.
==> Info: Done exporting 6.00 K records and 58.59 KiB.
==> Success: Download URL: https://cratedb-file-uploads.s3.amazonaws.com/some/download
==> Success: Operation completed.


.. NOTE::

This command will wait for the operation to finish or fail. It is only available
to organization admins.


``clusters export-jobs list``
=============================

.. argparse::
:module: croud.__main__
:func: get_parser
:prog: croud
:path: clusters export-jobs list

Example
-------

.. code-block:: console

sh$ ❯ croud clusters export-jobs list \
--cluster-id f6c39580-5719-431d-a508-0cee4f9e8209
+--------------------------------------+--------------------------------------+-----------+---------------------+-----------------------------------------------+
| id | cluster_id | status | source | destination |
|--------------------------------------+--------------------------------------+-----------+---------------------+-----------------------------------------------|
| b311ba9d-9cb4-404a-b58d-c442ae251dbf | f6c39580-5719-431d-a508-0cee4f9e8209 | SUCCEEDED | nyc_taxi | Format: csv |
| | | | | File ID: 327ad0e6-607f-4f99-a4cc-c1e98bf28e4d |
+--------------------------------------+--------------------------------------+-----------+---------------------+-----------------------------------------------+


``clusters export-jobs delete``
===============================

.. argparse::
:module: croud.__main__
:func: get_parser
:prog: croud
:path: clusters export-jobs delete

Example
-------

.. code-block:: console

sh$ ❯ croud clusters export-jobs delete \
--cluster-id f6c39580-5719-431d-a508-0cee4f9e8209 \
--export-job-id 3b311ba9d-9cb4-404a-b58d-c442ae251dbf
==> Success: Success.

.. _here: https://hub.docker.com/r/crate/crate/tags
Loading
Loading