Releases: InstituteforDiseaseModeling/idmtools
Releases · InstituteforDiseaseModeling/idmtools
v2.0.2
Release notes for idmtools 2.0.2
The release of idmtools 2.0.2 represents a feature release, focusing on MPI support in Platforms and some bug fixes.
Highlights:
Key features and changes in this release include:
-
MPI support in several idmtools platforms:
SlurmPlatform
ContainerPlatform
ProcessPlatform -
Fixed assets copy issue with relative symlinks.
-
Fixed major performance in Container and Slurm platforms.
-
Fixed idmtools cache issue.
-
Removed direct srun call in sbatch.
-
Changed the default 'max_running_jobs' in SlurmPlatform from 1 to 100
-
Bug fixes
Change log:
Bugs
- #2391 - Random fail with large number of simulations with NoneType of platform
- #2393 - Calibra may fail with ContainerPlatform
- #2395 - Slurm: Do not need to delete suite if job_directory is not exist or suite not exist anymore
- #2399 - docker container can become unusable in linux
Documentation
- #2386 - jupyter lab version fix
Feature Request
- #2377 - Need to change github actions to only build container docker image when there are related changes
- #2172 - Adjust the default 'max_running_jobs' in SlurmPlatform
- #2379 - Container Platform needs to support multiple processes
- #2396 - Assets duplication
Platforms
- #2388 - Performance issue with get_item in slurm platform when there are lot of items in job directory
- #2389 - Performance issue with get_item in container platform when there are lot of items in job directory
- #2247 - Investigation: run each simulation on SLURM with multi cores
Release/Packaging
- #2400 - Add script for generating changelog by project id and release version
User Experience
- #2387 - Investigation multiple warnings for binding in NU cluster for singularity
- #2145 - slurm platform -- unable to run from compute nodes as expected
All commits from 2.0.1 to 2.0.2: #2411
v2.0.1
v2.0.0
Release Notes for idmtools 2.0.0
The release of idmtools 2.0.0 represents a major update, featuring several important enhancements such as a newly configurable directory structure for Suites, Experiments, and Simulations, a restructured Platform Factory, and enhanced logging capabilities.
New Directory Structure
The newly organized Suite/Experiment/Simulation directory layout helps users identify their experiments more efficiently. This structure can now be set to use either named folders or UUID-based folders.
Platform Factory Refactoring
The Platform Factory has been simplified by eliminating the confusing missing_ok parameter and ensuring uniformity of parameters among INI files, platform aliases, and user inputs. Moreover, logging enhancements contribute to a clearer and more user-friendly interface.
Highlights
Key features and changes in this release include:
- Configurable Directory Structure: Users now have the option to select either named folders or UUIDs for Suite/Experiment/Simulation directories.
- Simplified API: The
runmethod has been streamlined by removing thewait_on_doneparameter, leaving onlywait_until_donefor job execution management. - Platform Type Case-Insensitivity: Platform types are now insensitive to case variations.
- Platform Creation: Platforms can be generated via an INI file, pre-defined platform Alias, or Platform Factory. Additionally, users can create individual concrete platform classes to leverage the availability of input parameters.
Change log
Platform
- #1655 - Support 'missing_ok' argument to Platform class without generating warnings
- #2313 - idmtools.ini 'type' parameter should not be case sensitive
- #2327 - When submit experiment with specific platform(instead of factory platform), it requires pass platform to run function
- #2171 - Experiment is missing add_simulation and add_simulations methods
- #2028 - iplatform non-descriptive error message: invalid type
- #2357 - Need to make sure assets exist when retrieving assets
- #2366 - Add max folder/files path length check for Windows
Bugs
- #2355 - Cache issue when deleting suite
- #2342 - idmtools cli still contains some command for Local Platform
- #2325 - If not use factory platform, it will throw warning
Feature requests
- #2143 - Readable/meaningful suite/experiment/simulation output directory names
- #2352 - Refactor SlurmPlatform item path (user readable)
- #2037 - experiment should only need one wait_on_done method
Documentation
- #2345 - Improve document (add more details to Container Platform)
v1.7.11
Release notes for idmtools 1.7.11
The idmtools 1.7.11 release is a feature update that introduces a new container platform and maintains compatibility across other platforms with no breaking changes.
The ContainerPlatform is a new platform designed to streamline the execution of experiments and simulations within Docker containers. It offers a fully equipped environment with all the necessary tools and dependencies, ensuring smooth integration and execution of computational tasks.
Highlights
Highlights of this release include:
-
New features:
- Introduced ContainerPlatform for running experiments and simulations within Docker containers.
- Provided default Docker images based on RockyLinux 9.2 and Debian 12, with automatic pulling of these images when using ContainerPlatform.
- Enabled support for custom Docker images, allowing users to run experiments and simulations with their own configurations.
- Added CLI commands for managing and monitoring experiments and simulations on ContainerPlatform:
idmtools container jobs [<container-id>] [-l <limit>] [-n <next>]: List running jobs.idmtools container status <item-id> [-c <container_id>] [-l <limit>] [--verbose/--no-verbose]: Check the status of an experiment or simulation.idmtools container cancel <item-id> [-c <container_id>]: Cancel a job.idmtools container history [<container-id>] [-l <limit>] [-n <next>]: View experiment history.idmtools container --all --help: View all container platform CLI commands.
- Added
containerandslurmpip install groups streamlined installation of related packages. - Implemented the ability to add extra Python packages to the container platform on the fly or via CLI commands.
- Facilitated viewing experiment/simulation results either within the container or on the host machine.
- Provided default and custom binding mounts to the container.
- Included examples/tests to demonstrate running experiments with ContainerPlatform.
-
idmtools improvements:
- Removed support for Python 3.7 in idmtools.
- Added a new platform alias for Nibbler.
- Resolved the scheduling issue on the COMPS platform.
- Corrected default values for the COMPSPlatform's environment and URL fields.
-
Documentation improvements:
- Introduced new documentation for ContainerPlatform.
- Added new docstrings for the platform factory and platform classes.
- Fixed the broken build issue on Windows and GitHub actions.
- Updated existing documentation with improved structure and more detailed information.
Change log
Platform
- #2253 - ContainerPlatform: Design Container Platform
- #2254 - ContainerPlatform: Task list
- #2266 - ContainerPlatform: Platform implementation (basic class)
- #2267 - ContainerPlatform: Handle Platform alias
- #2268 - ContainerPlatform: Handle file directory binding
- #2269 - ContainerPlatform: Modify required to File/Process Platform
- #2270 - ContainerPlatform: Container Utilities
- #2271 - ContainerPlatform: Handle Windows/Linux differences
- #2272 - ContainerPlatform: Implement CLI Status check
- #2273 - ContainerPlatform: Run AnalyzeManager
- #2274 - ContainerPlatform: Consider Calibra
- #2275 - ContainerPlatform: Consider Container Cleanup
- #2276 - ContainerPlatform: Consider Cancel experiment/simulation
- #2277 - ContainerPlatform: Consider Pre and Post run (ep4)
- #2278 - ContainerPlatform: Install extra packages required for ep4
- #2279 - ContainerPlatform: Collect EMOD requirements
- #2280 - ContainerPlatform: Build docker image
- #2281 - ContainerPlatform: Upload image to hub (idm artifactory)
- #2320 - ContainerPlatform: user select container to use
- #2331 - Container Platform: Code cleanup
Bugs
- #2318 - add_schdule_config should not hard code for num_cores and node_group
- #2306 - idmtoolscore experiment run can wait long time until timeout in certain case
- #2298 - idmtools package should not include the tests folder
- #2251 - local doc build (make html) errors out and does not complete
- #2169 - For dry run, we better not to display 'check status message
- #2208 - SlurmPlatform: experiment's metadata.json file has suite_id is null
Feature Request
- #2261- Consider changes to new COMPS scheduling support
- #2328 - COMPSPlatform should set default values for endpoint and environment set to production instead of staging
- #2326 - SlurmPlatform should have default mode to 'Local' instead of None
- #2319 - Add new alias for COMPS new environment: Nibbler
- #2317 - idmtools should stop support python3.7
- #2337 - Add container, slurm options in idmtools setup for customized installation
Developer/Test
- #2282 - ContainerPlatform: Test image
- #2283 - ContainerPlatform: Test platform on docker container
- #2284 - ContainerPlatform: Test utilities
- #2285 - ContainerPlatform: Test Windows/Linux
- #2286 - ContainerPlatform: Test CLI
- #2287 - ContainerPlatform: Test AnalyzeManager
- #2288 - ContainerPlatform: Test Calibration (Calibra)
- #2289 - ContainerPlatform: Write unit tests
Documentation
v1.7.10
What's Changed
- Fix pygit2 by @shchen-idmod in #2158
- Assetization workflow for generating SIF image should track using Asset ID instead of AC ID by @shchen-idmod
#2032 - Remove local platform from idmtools repo by @shchen-idmod in #2160
- Improved Simulation builders (fixed several issues) by @ZDu-IDM in #2184
- Refactor idmtools Simulation Builders structure. by @ZDu-IDM in #2203
- Refactor Simulation Builders to remove duplicated code by @ZDu-IDM in #2196
- Combine two add sweeping functions into one @ZDu-IDM in #2195
- Sweeping function allow parameters with default values @ZDu-IDM #2194
- Builder function add_multiple_parameter_sweep_definition doesn't support function with single dictionary parameter @ZDu-IDM #2192
- SimulationBuilder assume input values object has len defined @ZDu-IDM #2183
- Refactor ArmSimulationBuilder and move general functionality to a base class @ZDu-IDM #2168
- ArmSimulationBuilder needs rasie exception for call to add_sweep_definition and add_multiple_parameter_sweep_definition @ZDu-IDM #2167
- SweepArm should support add_multiple_parameter_sweep_definition @ZDu-IDM #2166
- ArmSimulationBuilder display simulation count incorrect @ZDu-IDM #2165
- Give deprecating warning for duplicate wait_on_done function by @shchen-idmod in #2173
- Fix circular dependency for core and cli tests and move comps related test in core to comps test by @shchen-idmod in #2175
- Add download asset and use asset id in generate singularity builder instead of assetcollection by @shchen-idmod in #2210
- Fix bug for save_as for content type and add unittest by @shchen-idmod in #2213
- Update idmtools to work with python 3.12 by @shchen-idmod in #2215
- unregister plugin in test_hooks.py to avoid affecting other test in p… by @shchen-idmod in #2226
- Update idm-buildtools requirement from ~=1.0.3 to ~=1.0.5 by @dependabot in #2227
- Update junitparser requirement from ~=3.1.1 to ~=3.1.2 by @dependabot in #2228
- Update coverage requirement from <6.6,>=5.3 to >=5.3,<7.5 by @dependabot in #2229
- Update flake8 requirement from ~=6.0.0 to ~=7.0.0 by @dependabot in #2230
- Update pytest-xdist requirement from ~=3.3 to ~=3.5 by @dependabot in #2231
- fix hook run with serial mark by @shchen-idmod in #2235
- Update docker requirement from <6.1.0,>=4.3.1 to >=4.3.1,<7.1.0 by @dependabot in #2177
- Update junitparser requirement from ~=2.8.0 to ~=3.1.1 by @dependabot in #2178
- Update jinja2 requirement from ~=3.1.2 to ~=3.1.3 by @dependabot in #2176
- Update packaging requirement from <22.0,>=20.4 to >=20.4,<24.0 by @dependabot in #2181
- Update readthedocs-sphinx-search requirement from ~=0.3.1 to ~=0.3.2 by @dependabot in #2182
- Update click requirement from ~=8.1.3 to ~=8.1.7 by @dependabot in #2180
- Update pytest requirement from ~=7.2.0 to ~=8.0.0 by @dependabot in #2179
- Revert "Update pytest requirement from ~=7.2.0 to ~=8.0.0" by @shchen-idmod in #2185
- Update pytest-timeout requirement from ~=2.1.0 to ~=2.3.1 by @dependabot in #2204
- Update cookiecutter requirement from ~=2.1.1 to ~=2.6.0 by @dependabot in #2199
- Update jupyterlab requirement from ~=4.0.2 to ~=4.1.5 by @dependabot in #2212
- update examples for platform from Belegost to CALCULON by @shchen-idmod in #2225
- Update nbsphinx requirement from ~=0.9.2 to ~=0.9.3 by @dependabot in #2221
- Update pygithub requirement from ~=1.57 to ~=2.3 by @dependabot in #2223
- Update packaging requirement from ~=23.2 to ~=24.0 by @dependabot in #2222
- Update pluggy requirement from ~=1.2 to ~=1.4 by @dependabot in #2220
- Update yaspin requirement from <2.4.0,>=1.2.0 to >=1.2.0,<3.1.0 by @dependabot in #2189
Full Changelog: v1.7.9...v1.7.10
Contributors
ZDu-IDM, dependabot, and shchen-idmod
Assets 2
v1.7.9
## What's Changed
- First PR to fix some bugs by @ZDu-IDM in #2149
- Add hook tests to comps, file, slurm platforms by @shchen-idmod in #2150
- Remove Bayesian tests by @shchen-idmod in #2134
- Read the Docs Sphinx updates by @JSchripsema-IDM in #2142
- docs/gitignores by @bryanressler-idmod in #2141
- changed default search behavior to search only across this doc project by @JSchripsema-IDM in #2151
- add metadata for python3.11 to all setup.py package by @shchen-idmod in #2108
- Bump json5 from 1.0.1 to 1.0.2 in /idmtools_platform_local/idmtools_webui by @dependabot in #1985
- Bump markupsafe from 2.1.1 to 2.1.3 by @dependabot in #2076
- Update pytest-xdist requirement from ~=3.1 to ~=3.3 by @dependabot in #2072
- Update natsort requirement from ~=8.2.0 to ~=8.4.0 by @dependabot in #2085
- Bump crypto-js from 3.1.9-1 to 3.3.0 in /idmtools_platform_local/idmtools_webui by @dependabot in #2081
- Bump pipreqs from 0.4.11 to 0.4.12 in /examples by @dependabot in #2099
- Bump @hapi/hoek from 8.5.0 to 8.5.1 in /idmtools_platform_local/idmtools_webui by @dependabot in #2046
- Update sqlalchemy requirement from ~=1.4.45 to ~=2.0.18 by @dependabot in #2111
- Bump sphinx-copybutton from 0.5.1 to 0.5.2 by @dependabot in #2117
- Update psycopg2-binary requirement from ~=2.9.5 to ~=2.9.6 by @dependabot in #2115
- Update more-itertools requirement from ~=9.0.0 to ~=9.1.0 by @dependabot in #2113
- Update flask requirement from ~=2.2.2 to ~=2.3.2 by @dependabot in #2116
- Update pluggy requirement from ~=1.0.0 to ~=1.2.0 by @dependabot in #2114
- change branch related from master to main by @shchen-idmod in #2123
- Added Jenkinsfile to main branch by @shchen-idmod in #2127
- Merge Jenkins and main branch change to release branch by @shchen-idmod in #2128
- Merge Jenkins and main branch changes to dev branch by @shchen-idmod in #2129
- Update allure-pytest requirement from <2.13,>=2.8.34 to >=2.8.34,<2.14 by @dependabot in #2118
- bump version to 1.7.9 by @shchen-idmod in #2153
New Contributors
- @bryanressler-idmod made their first contribution in #2141
Full Changelog: v1.7.8...v1.7.9
Contributors
ZDu-IDM, JSchripsema-IDM, and 3 other contributors
Assets 2
v1.7.8
What's Changed
=====
1.7.8
Additional Changes
#2100 <https://github.com/InstituteforDiseaseModeling/idmtools/issues/2100>_ - Setup.py does not conform to newest pip in python requires#2101 <https://github.com/InstituteforDiseaseModeling/idmtools/issues/2101>_ - Deprecate 3.6 references from idmtools#2102 <https://github.com/InstituteforDiseaseModeling/idmtools/issues/2102>_ - Doc fix
Bugs
#2083 <https://github.com/InstituteforDiseaseModeling/idmtools/issues/2083>_ - python11 issue with dataclasses
Full Changelog: v1.7.7...v1.7.8
v1.7.7
=====
1.7.7
Bugs
#2084 <https://github.com/InstituteforDiseaseModeling/idmtools/issues/2084>_ - Potential issue with mismatch version of pandas and matplotlib
Dependencies
#2013 <https://github.com/InstituteforDiseaseModeling/idmtools/issues/2013>_ - Update yaspin requirement from <2.3.0,>=1.2.0 to >=1.2.0,<2.4.0#2024 <https://github.com/InstituteforDiseaseModeling/idmtools/issues/2024>_ - Update coverage requirement from <6.6,>=5.3 to >=5.3,<7.3
Documentation
#2000 <https://github.com/InstituteforDiseaseModeling/idmtools/issues/2000>_ - slurm commission take too much memory which can exceeds head node's max memory#2042 <https://github.com/InstituteforDiseaseModeling/idmtools/issues/2042>_ - Write doc: run main script as SLURM job
Feature Request
#1998 <https://github.com/InstituteforDiseaseModeling/idmtools/issues/1998>_ - Potential issue with max count of simulations in slurm platform#2043 <https://github.com/InstituteforDiseaseModeling/idmtools/issues/2043>_ - Write Python utility to run main script as SLURM job#2041 <https://github.com/InstituteforDiseaseModeling/idmtools/issues/2041>_ - Write workaround steps: run main script as SLURM job#2095 <https://github.com/InstituteforDiseaseModeling/idmtools/issues/2095>_ - Add singularity bind experiment by default for slurm#2096 <https://github.com/InstituteforDiseaseModeling/idmtools/issues/2096>_ - Add few more COMPS server aliases
v1.7.6
What's Changed
Summary
idmtools release 1.7.6
- new package idmtools_platform_general which contains
. FilePlatform
. ProcessPlatform - ID Generator
- Fixed deepcopy issue
Details
- Implement SlurmPlatform Status utility by @ZDu-IDM in #2004
- Support alternate id generators by @emilykclaps in #1825
- Add example for ssmt with extra packages based on Clinton's example by @shchen-idmod in #2020
- Add unittests for idmtools_platform_file and add/update github actions by @shchen-idmod in #2023
- File platform: implemented workflow (folders and files are generated). by @ZDu-IDM in #2019
- Fix test in File platform by @shchen-idmod in #2026
- File platform: implemented experiment execution (batch and status, etc.) by @ZDu-IDM in #2025
- Add file platform cli tests by @shchen-idmod in #2039
- File platform: implementation of CLI utility by @ZDu-IDM in #2038
- Add unittests and examples for file and process platforms by @shchen-idmod in #2045
- Platform-General: implementation of ProcessPlatform by @ZDu-IDM in #2044
Contributors
ZDu-IDM, shchen-idmod, and emilydriano