This is a list of user-visible changes worth mentioning. In each new release, items are added to the top of the file and identify the version they pertain to.
- Terminal log output is now colorized according to message level! (Thanks to Adrian Etter for suggesting this feature.)
- Old-style sessions are not supported any more. (This should not be a problem, as they have been automatically converted to "new-style" since years now. In the unlikely case you still have an old-style session directory on disk, just run any session command from version 2.4 and it will convert the format automatically.)
- The environment variable
GC3PIE_RESOURCE_INIT_ERRORS_ARE_FATAL
can be set toyes
or1
to cause GC3Pie to abort if any errors occur while initializing the configured resources. The default behavior of GC3Pie is instead to keep running until there is at least one resource that can be used. - A resource is now automatically disabled if an unrecoverable error occurs during its use.
The ARC backends and supporting code have been removed: it is no longer possible to use GC3Pie to submit tasks to an ARC job manager.
The environment variable
GC3PIE_NO_CATCH_ERRORS
now can specify a list of patterns to selectively unignore unexpected/generic errors in the code. As this feature should only be used in debugging code, we allow ourselves to break backwards compatibility.The cloud and mathematics libraries are no longer installed by default with
pip install gc3pie
-- please use:pip install gc3pie[openstack,ec2,optimizer]
to install support for all optional backends and libraries.
The gc3libs.utils.ifelse function was removed in favor of Python's ternary operator.
- New task collection `DependentTaskCollection`:class: to run a collection of tasks with given pre/post dependencies across them.
- GC3Pie will now parse and obey the
Port
,Identity
,User
,ConnectionTimeout
, andProxyCommand
options from the SSH config file. Location of an alternate configuration file to use with GC3Pie can be set in any[auth/*]
section of type SSH; see the `configuration`:ref: section for details. Thanks to Niko Eherenfeuchter and Karandash8 for feature requests and preliminary implementations. - Application prologue and epilogue scripts can now be embedded in the GC3Pie configuration file, or referenced by file name.
- New selection options have been added to the `gselect`:ref: command.
- gc3libs.Configuration will now raise different exceptions depending on whether no files could be read (`NoAccessibleConfigurationFile`:class:) or could not be parsed (`NoValidConfigurationFile`:class:).
- Shell metacharacters are now allowed in Application arguments. Each argument string is now properly quoted before passing it to the execution layer.
- LSF backend updated to work with both bjobs and bacct for accounting, or to parse information provided in the final output file as a last resort.
- All backends should now set a Task's returncode and exitcode values according to the documented meaning. Thanks to Y. Yakimovitch for reporting the issue.
- New
openstack
backend for running jobs on ephemeral VMs on OpenStack-compatible IaaS cloud systems. This is preferred over the OpenStack EC2 compatibility layer. - New configurable scheduler for GC3Pie's `Engine`:class:
- Session-based scripts can now snapshot the output of RUNNING jobs at every cycle.
- ARC backends are now deprecated: they will be removed in the next major version of GC3Pie.
- The
pbs
backend can now handle also Altair's PBSPro.
gget
: New option -A to download output files of all tasks in a session.gget
: New option-c
/--changed-only
to only download files that have apparently changed remotely.- The GC3Apps collection has been enriched with several new applications.
- Working directory for remote jobs using the
shellcmd
backend is now stored in/var/tmp
instead of/tmp
, which should allow results to be retrieved even after a reboot of the remote machine.
- GC3Pie now requires Python 2.6 or above to run.
- New
ec2
backend for running jobs on ephemeral VMs on EC2-compatible IaaS cloud systems. - New package gc3libs.optimizer to find local optima of functions that can be computed through a job. Currently only implements the "Differential Evolution" algorithm, but the framework is generic enough to plug any genetic algorithm.
- New configuration options
prolog_content
andepilog_content
, to allow execute oneliners before or after the command without having to create an auxiliary file. - New
resourcedir
option forshellcmd
resources. This is used to modify the default value for the directory containing job informations.
- New command
gcloud
to interface with cloud-based VMs that were spawned by GC3Pie to run jobs. - Table output now uses a different formatting (we use Python's prettytable package instead of the texttable package that we were using before, due to Py3 compatibility).
- GC3Pie can now run on MacOSX.
- A session now has a configurable storage location, which can be a directory on the filesystem (FilesystemStore, the default so far) or can be a table in an SQL database (of any kind supported by SQLAlchemy).
- New ARC1 backend to use ARC resources through the new NorduGrid 1.x library API.
- New backend "subprocess": execute applications as local processes.
- New backends for running on various batch-queueing systems: SLURM, LSF, PBS.
- Implement recursive upload and download of directories if they are specified in an Application's input or output attribute.
- New execution state TERMINATING: task objects are in this state when execution is finished remotely, but the task output has not yet been retrieved.
- Reorganize documentation and move it to http://gc3pie.readthedocs.org/
- Script logging is now controlled by a single configuration file .gc3/gc3utils.log.conf
- Session-based scripts now print WARNING messages to STDERR by default (previously, only ERROR messages were logged).
- Add caching to ARC backends, to reduce the number of network queries.
- Use GNU ".~NUMBER~" format for backup directories.
- Rename ARC0 resource type to arc0
- New gsession command to manage sessions.
- The
glist
command was renamed togservers
- The
gsub
andgnotify
commands were removed. - The
PATH
tag no longer gets any special treatment in session-based scripts--output
processing. ginfo
: New option--tabular
to print information in table format.gkill
: New option -A/--all to remove all jobs in a session.- Use the rungms script to execute GAMESS.
- Module
gc3libs.dag
has been renamed togc3libs.workflow
. - API changes in `gc3libs.cmdline.SessionBasedScript`:class: allow new_tasks() in SessionBasedScript instances to return Task instances instead of quadruples.
- Interpret Application.requested_memory as the total memory for the job.
- the
Resource
andLRMS
objects were merged - the
gc3libs.scheduler
module has been removed; its functionality is now incorporated in theApplication
class. - configuration-related code moved into gc3libs.config module
- removed the application registry.
- New package gc3libs.compat to provide 3rd-party functionality that is not present in all supported versions of Python.
- Implement gc3libs.ANY_OUTPUT to retrieve the full contents of the output directory, whatever it is.
- New RetryableTask class to wrap a task and re-submit it on failure until some specified condition is met.
- Renamed configuration file to
gc3pie.conf
: the filegc3utils.conf
will no longer be read! - SGE clusters must now have
type = sge
in the configuration file (instead oftype = ssh-sge
) - All computational resource must have an
architecture = ...
line; see the ConfigurationFile wiki page for details - Probably more changes than it's worth to list here: check your configuration against the `configuration`:ref: page!
- GC3Utils and GC3Apps (
grosetta
/ggamess
/etc.) now all accept a-s
/--session
option for locating the job storage directory: this allows grouping jobs into folders instead of shoveling them all into~/.gc3/jobs
. - GC3Apps: replaced option
-t
/--table
with-l
/--states
. The new option prints a table of submitted jobs in addition to the summary stats; if a comma-separated list of job states follows the option, only job in those states are printed. - Command
gstat
will now print a summary of the job states if the list is too long to fit on screen; use the -v option to get the full job listing regardless of its length. - Command
gstat
can now print information on jobs in a certain state only; see help text for option--state
- Removed
-l
option from ginfo; use-v
instead. - GC3Utils: all commands accepting multiple job IDs on the command line, now exit with the number of errors/failures occurred. Since exit codes are practically limited to 7 bits, exit code 126 means that more than 125 failures happened.
- First release for public use outside of GC3