diff --git a/.github/workflows/_publish.yaml_ b/.github/workflows/publish-docs.yml similarity index 89% rename from .github/workflows/_publish.yaml_ rename to .github/workflows/publish-docs.yml index 97ef2a866..7ea7d218c 100644 --- a/.github/workflows/_publish.yaml_ +++ b/.github/workflows/publish-docs.yml @@ -1,13 +1,15 @@ name: Publish documentation site - on: release: types: [published] push: branches: - - main + - master - docs + paths: + - docs/** + - mkdocs.yml jobs: diff --git a/cm/CHANGES.md b/cm/CHANGES.md index 492d9a92f..c3c12206f 100644 --- a/cm/CHANGES.md +++ b/cm/CHANGES.md @@ -1,3 +1,8 @@ +## V3.4.2.1 + - fixed pyproject.toml + - fixed docs generation + - improved cm pull repo with branch and SHA (#1355) + ## V3.4.2 - added utils.flatten_dict - added utils.safe_int diff --git a/cm/MANIFEST.in b/cm/MANIFEST.in new file mode 100644 index 000000000..f08d39c54 --- /dev/null +++ b/cm/MANIFEST.in @@ -0,0 +1,3 @@ +include CHANGES.md +include CONTRIBUTING.md +include COPYRIGHT.txt diff --git a/cm/cmind/__init__.py b/cm/cmind/__init__.py index d63eb4ef4..f67ba6269 100644 --- a/cm/cmind/__init__.py +++ b/cm/cmind/__init__.py @@ -2,7 +2,7 @@ # # Written by Grigori Fursin -__version__ = "3.4.2" +__version__ = "3.4.2.1" from cmind.core import access from cmind.core import x diff --git a/cm/pyproject.toml b/cm/pyproject.toml index 7159530d3..f94832481 100644 --- a/cm/pyproject.toml +++ b/cm/pyproject.toml @@ -1,5 +1,3 @@ -# Developer(s): Grigori Fursin - [build-system] build-backend = "setuptools.build_meta" requires = ["setuptools"] @@ -62,6 +60,7 @@ version = {attr = "cmind.__version__"} [tool.setuptools] zip-safe = false +include-package-data = true [tool.setuptools.package-data] cmind = ["repo/*", diff --git a/docs/cm/README.md b/docs/cm/README.md new file mode 100644 index 000000000..e9b96edd0 --- /dev/null +++ b/docs/cm/README.md @@ -0,0 +1,88 @@ +[![PyPI version](https://badge.fury.io/py/cmind.svg)](https://pepy.tech/project/cmind) +[![Python Version](https://img.shields.io/badge/python-3+-blue.svg)](https://github.com/mlcommons/ck/tree/master/cm/cmind) +[![License](https://img.shields.io/badge/License-Apache%202.0-green)](LICENSE.md) +[![Downloads](https://static.pepy.tech/badge/cmind)](https://pepy.tech/project/cmind) + +[![arXiv](https://img.shields.io/badge/arXiv-2406.16791-b31b1b.svg)](https://arxiv.org/abs/2406.16791) +[![CM test](https://github.com/mlcommons/ck/actions/workflows/test-cm.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cm.yml) +[![CM script automation features test](https://github.com/mlcommons/ck/actions/workflows/test-cm-script-features.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cm-script-features.yml) + +### About + +Collective Mind (CM) is a small, modular, cross-platform and decentralized workflow automation framework +with a human-friendly interface to make it easier to build, run, benchmark and optimize applications +across diverse models, data sets, software and hardware. + +CM is a part of [Collective Knowledge (CK)](https://github.com/mlcommons/ck) - +an educational community project to learn how to run emerging workloads +in the most efficient and cost-effective way across diverse +and continuously changing systems. + +CM includes a collection of portable, extensible and technology-agnostic automation recipes +with a common API and CLI (aka CM scripts) to unify and automate different steps +required to compose, run, benchmark and optimize complex ML/AI applications +on any platform with any software and hardware. + +CM scripts extend the concept of `cmake` with simple Python automations, native scripts +and JSON/YAML meta descriptions. They require Python 3.7+ with minimal dependencies and are +[continuously extended by the community and MLCommons members](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md) +to run natively on Ubuntu, MacOS, Windows, RHEL, Debian, Amazon Linux +and any other operating system, in a cloud or inside automatically generated containers +while keeping backward compatibility. + +CM scripts were originally developed based on the following requirements from the +[MLCommons members](https://mlcommons.org) +to help them automatically compose and optimize complex MLPerf benchmarks, applications and systems +across diverse and continuously changing models, data sets, software and hardware +from Nvidia, Intel, AMD, Google, Qualcomm, Amazon and other vendors: +* must work out of the box with the default options and without the need to edit some paths, environment variables and configuration files; +* must be non-intrusive, easy to debug and must reuse existing + user scripts and automation tools (such as cmake, make, ML workflows, + python poetry and containers) rather than substituting them; +* must have a very simple and human-friendly command line with a Python API and minimal dependencies; +* must require minimal or zero learning curve by using plain Python, native scripts, environment variables + and simple JSON/YAML descriptions instead of inventing new workflow languages; +* must have the same interface to run all automations natively, in a cloud or inside containers. + +### Maintainers + +* [Grigori Fursin](https://cKnowledge.org/gfursin) + +### Resources + +* CM v2.x (2022-cur) (stable): [installation on Linux, Windows, MacOS](https://access.cknowledge.org/playground/?action=install) ; + [docs](https://docs.mlcommons.org/ck) ; [popular commands](https://github.com/mlcommons/ck/tree/master/cm/docs/demos/some-cm-commands.md) ; + [getting started guide](https://github.com/mlcommons/ck/blob/master/docs/getting-started.md) +* CM v3.x aka CMX (2024-cur) (stable): [docs](https://github.com/orgs/mlcommons/projects/46) +* MLPerf inference benchmark automated via CM + * [Run MLPerf for submissions](https://docs.mlcommons.org/inference) + * [Run MLPerf at the Student Cluster Competition'24](https://docs.mlcommons.org/inference/benchmarks/text_to_image/reproducibility/scc24) +* Examples of modular containers and GitHub actions with CM commands: + * [GitHub action with CM commands to test MLPerf inference benchmark](https://github.com/mlcommons/inference/blob/master/.github/workflows/test-bert.yml) + * [Dockerfile to run MLPerf inference benchmark via CM](https://github.com/mlcommons/ck/blob/master/cm-mlops/script/app-mlperf-inference/dockerfiles/bert-99.9/ubuntu_22.04_python_onnxruntime_cpu.Dockerfile) + +### License + +[Apache 2.0](LICENSE.md) + +### Citing CM project + +If you found CM automations useful, please cite this article: +[ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ]. + +You can learn more about the motivation behind these projects from the following presentations: + +* "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ] +* ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ] +* ACM TechTalk'21 about Collective Knowledge project: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ] + +### Acknowledgments + +The Collective Mind (CM) automation framework was originally +developed by [Grigori Fursin](https://cKnowledge.org/gfursin), +as a part of the [Collective Knowledge educational initiative](https://cKnowledge.org), +sponsored by [cTuning.org](https://cTuning.org) and [cKnowledge.org](https://cKnowledge.org), +and contributed to MLCommons for the benefit of all. +This open-source technology, including CM4MLOps/CM4MLPerf, CM4ABTF, CM4Research, and more, +is a collaborative community-driven project made possible by our +[amazing volunteers, collaborators, and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)!