- Accept WAIT_ON_QUEUES as valid ingest job status.
- Exit out if upload queue no longer exists (non-existence indicates job complete or cancelled).
- Added an optional
--ramp_seconds/-rflag to delay the creation of parallel processes. A ramp value of 10 will wait ten seconds in between creating new processes. This is helpful when the source of the ingest data needs time to scale (e.g. Google Cloud Storage). - Add an intern chunk processor for ingesting large cutouts.
Dropping support for Python 2
- New ingest complete functionality implemented.
- Removed delayed wait when requesting job credentials.
- Added exponential backoff on retries when a client is requesting job status of an ingest job.
- Added exponential backoff on retries when a client is trying to join an ingest job.
- Stop client from creating tile index queue when doing volumetric ingest as that is handled by endpoint.
- Added ability to volumetric ingest jobs using Princeton's CloudVolume data format.
- Ingest-job will not be completed unless the upload-queue has zero messages in it. If an error occurs job will remain now and not close.
- Added no_cache option to the intern plugin, this will increase the download speed be avoiding the cache.
- Added x and y tile sizes to metadata of s3 tiles. This allows the ingest backend to create blank tiles when corrupt images are received.
- Added in prep_ingest.py which will become a new script prep_ingest on pip install. It will create Collection, Experiment, Coordinate Frame, and Channel for you and add them to your ingest-client json config file.
- Adjusted requirements.txt versions.
- Added ability to wait for server to verify job has all tiles uploaded before completing.
- updated requirements.txt to be compatible with boss-manage and intern by removing pinned versions of libraries.
- Fixed an off by one error while looping for credentials renewal
- Updated to catch AccessDenied and InvalidAccessKeyId errors and then request new credentials when this occurs.
- The try-catch in Engine.run() was just around the ingest client’s IO to S3, but user developed plugins can be doing IO to anything and have random, transient errors as well. Moved the try statement to the top of the loop.
- Added automatic completion of ingest jobs. When all tiles have been successfully uploaded the job is marked as
Completeand resources are cleaned up
- Fixed error that printed the wrong tile count estimates when the Ingest Job extents contained an offset
- PR5: Allow injection of Configuration objects into Engine - Merged with minor modifications
- Makes it easier for users to programmatically run the ingest client if many configuration files or parameter changes are needed
- Add the ability to instantiate an Engine instance directly with a Configuration instance that can be easily manipulated
- Added CHANGELOG.md
- IMPORTANT Refactored client for pip installation capability
- refactored
ingestlibrary toingestclient - refactored
client.pytoboss-ingest - Note: existing ingest job configuration files that were loading built-in plugins from the boss-ingest package will need to be updated from
ingesttoingestclient
- refactored
- Added pip install capability (
pip install boss-ingest) - Added
--force-fflag to automatically accept all prompts (useful for launching on remote nodes) - Added initial feedback for job progress
- Added
-log_level-vflag to indicate desired log level. Defaults towarning - Added additional feedback on the job that is about to be created when starting a new job
- Added small delay as worker processes spin up to ramp Lambda gracefully
- Reduced excessive printing