v1.2.1-rc3

What's Changed

• Bump github.com/jedib0t/go-pretty/v6 from 6.4.7 to 6.4.9 by @dependabot in #3091
• Remove building Bacalhau as part of Docker image build by @simonwo in #3092
• Docs migration by @simonwo in #3070
• Bump github.com/go-git/go-git/v5 from 5.8.1 to 5.11.0 by @dependabot in #3097
• Bump github.com/hashicorp/go-hclog from 1.5.0 to 1.6.2 by @dependabot in #3105
• Custom job type docker images by @rossjones in #3093
• Fix error message when missing plugin path by @rossjones in #3109
• fix: require teststack connected before test by @frrist in #3111
• fix: helpful error when IPFS fails to get content by @frrist in #3101
• refactor: allow node info store ttl configuration by @frrist in #3102
• Fix GPU labels for nodes with multiple GPU vendors by @simonwo in #3116
• Speed up CI a little bit by @simonwo in #3119
• Don't auto-generate Go-based docs using Swagger anymore by @simonwo in #3115
• Bump github.com/labstack/echo/v4 from 4.11.1 to 4.11.3 by @dependabot in #3117
• Fix non-working test to save 30s by @rossjones in #3124
• Speed up the failing ipfs downloader test by @rossjones in #3123
• Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace from 1.18.0 to 1.21.0 by @dependabot in #3126
• feat: further configuration of embedded ipfs daemon by @frrist in #3096
• Bump golang.org/x/crypto from 0.16.0 to 0.17.0 by @dependabot in #3129
• Webui fix job name and by @olgibbons in #3098
• Run code generation by @rossjones in #3133
• Bump golang.org/x/time from 0.3.0 to 0.5.0 by @dependabot in #3134
• Appeasing linter by @rossjones in #3122
• Provide flag to set webui port by @rossjones in #3132
• Translation converting custom task types to docker tasks by @rossjones in #3108
• Build and release on tags by @simonwo in #3136

Full Changelog: v1.2.0...v1.2.1-rc2

v1.2.1-rc1

What's Changed

• Bump github.com/jedib0t/go-pretty/v6 from 6.4.7 to 6.4.9 by @dependabot in #3091
• Remove building Bacalhau as part of Docker image build by @simonwo in #3092
• Docs migration by @simonwo in #3070
• Bump github.com/go-git/go-git/v5 from 5.8.1 to 5.11.0 by @dependabot in #3097
• Bump github.com/hashicorp/go-hclog from 1.5.0 to 1.6.2 by @dependabot in #3105
• Custom job type docker images by @rossjones in #3093
• Fix error message when missing plugin path by @rossjones in #3109
• fix: require teststack connected before test by @frrist in #3111
• fix: helpful error when IPFS fails to get content by @frrist in #3101
• refactor: allow node info store ttl configuration by @frrist in #3102
• Fix GPU labels for nodes with multiple GPU vendors by @simonwo in #3116
• Speed up CI a little bit by @simonwo in #3119
• Don't auto-generate Go-based docs using Swagger anymore by @simonwo in #3115
• Bump github.com/labstack/echo/v4 from 4.11.1 to 4.11.3 by @dependabot in #3117
• Fix non-working test to save 30s by @rossjones in #3124
• Speed up the failing ipfs downloader test by @rossjones in #3123
• Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace from 1.18.0 to 1.21.0 by @dependabot in #3126
• feat: further configuration of embedded ipfs daemon by @frrist in #3096
• Bump golang.org/x/crypto from 0.16.0 to 0.17.0 by @dependabot in #3129
• Webui fix job name and by @olgibbons in #3098
• Run code generation by @rossjones in #3133
• Bump golang.org/x/time from 0.3.0 to 0.5.0 by @dependabot in #3134
• Appeasing linter by @rossjones in #3122
• Provide flag to set webui port by @rossjones in #3132
• Translation converting custom task types to docker tasks by @rossjones in #3108

Full Changelog: v1.2.0...v1.2.1-rc1

v1.2.0

Announcing Bacalhau v1.2

📢 We are extremely proud to announce the release of Bacalhau 1.2! Since we announced the release of Bacalhau 1.1 in September 2023, we have been the community has stepped forward to release a number of novel ways to use Bacalhau. For example:

• Save $2.5M Per Year by Managing Logs the AWS Way
• Edge-based Machine Learning Inference
• Distributed Warehousing Made Easy With Bacalhau
• How to Solve Edge Container Orchestration

We are also excited to announce that the U.S. Navy Chose Bacalhau to Manage Predictive Maintenance Workloads!

But that’s just the start - Bacalhau 1.2 is packed with new features, enhancements, and bug fixes to improve your workflow and overall user experience.

Read on to learn more about these exciting updates! Or install straight away.

Job templates

Users can now create and customize job templates to streamline their workflow by creating a large number of similar jobs more easily.

A job template contains named placeholders:

Name: docker job
Type: batch
Count: 1
Tasks:
  - Name: main
    Engine:
      Type: docker
      Params:
        Image: ubuntu:latest
        Entrypoint:
          - /bin/bash
        Parameters:
          - -c
          - echo {{.greeting}} {{.name}} 

The placeholders will be filled in during a call to bacalhau job run with environment variables or command-line flags:

export greeting=Hello
bacalhau job run job.yaml --template-vars "name=World" --template-envs "*"

The templating functionality is based on the Go text/template package. This robust library offers a wide range of features for manipulating and formatting text based on template definitions and input variables. For more information about the Go text/template library and its syntax, you can refer to the official documentation: Go text/template Package.

Telemetry from inside WASM jobs with Dylibso integration

This release introduces the ability to collect telemetry data from within WebAssembly (WASM) jobs via integration with the Dylibso Observe SDK.

Now, WebAssembly modules that have been automatically or manually instrumented will pass
tracing information to any OTEL endpoints configured for Bacalhau to use. This happens
automatically if OTEL is configured and the WASM is instrumented. Uninstrumented WASM continues to be run as normal with no required changes.

This allows WASM jobs to pass telemetry into Jaeger or any other configured OTEL client:

Simplified node bootstrapping

Bacalhau nodes now no longer require peers to be supplied with a LibP2P key, which simplifies configuration of private Bacalhau clusters. Now, Bacalhau only requires an IP address and port number to establish peering with other nodes.

Whereas previously, Bacalhau required a public key to be specified with peer information:

/ip4/35.245.161.250/tcp/1235/p2p/QmbxGSsM6saCTyKkiWSxhJCt6Fgj7M9cns1vzYtfDbB5Ws

A simpler version with automatic key lookup is now supported:

/ip4/35.245.161.250/tcp/1235/http

This means in private environments, node set-up can be considerably simplified, as only the IP address and port need to be known in advance of establishing the cluster.

New browser-based dashboard

We have introduced a modern and intuitive browser-based dashboard, providing web users with a fresh experience. Users can now see both job execution lists and node information:

For the demonstration network, you can now visit http://bootstrap.production.bacalhau.org to see the dashboard in action.

It is simple to add the web UI to your own private cluster – just pass --web-ui to your bacalhau serve command or set Node.WebUI: yes in your config file. Once set Bacalhau will serve the web UI on port 80 automatically with no further configuration required.

Support for AMD and Intel GPUs

Users can now take advantage of the power of AMD and Intel GPUs for their computational tasks, in addition to our existing support for Nvidia GPUs.

Now, if a compute node has the AMD utlity rocm-smi or Intel utility xpu-smi installed, Bacalhau will automatically detect their GPUs at boot and make them available for use in Docker jobs.

Further, details on available GPUs are now available from the Nodes API. Calls to /api/v1/orchestrator/nodes now return extended GPU information that identifies the GPU’s vendor, available VRAM, and model name.

Support for multiple GPUs in Docker jobs

We now support the utilization of multiple GPUs in Docker jobs, enabling workload splitting and parallel processing.

Users can take advantage of multiple GPUs by passing the --gpu=... flag to bacalhau docker run or by specifying the number of GPUs in the Resources section of the job spec. The job will wait for the requested number of GPUs to become available at which point the job will have exclusive use of the GPUs for as long as it is executing.

Results download from S3-compatible buckets

Users can now download results directly from S3-compatible buckets using bacalhau get, simplifying the data retrieval process.

The S3 Publisher will now generate pre-signed URLs on-demand for use by Bacalhau clients downloading job results without needing to provide credentials to the bucket itself. This requires the requester node to have appropriate IAM permissions for reading buckets.

Support for Google Cloud buckets

Users can now seamlessly integrate Google Cloud Storage (GCS) buckets into their jobs, allowing for easy storage and retrieval of data.

To use GCS buckets, request data from a GCS storage endpoint or configure an S3 publisher using a [storage.googleapis.com](http://storage.googleapis.com) endpoint.

Programmatic API for manipulating config

Developers can now programmatically manipulate configuration settings using the new bacalhau config command, providing more flexibility and automation in managing Bacalhau nodes.

• Use bacalhau config list to show the current state of configuration variables including defaults.
• Use bacalhau config set <key> <value> to permanently set a configuration option.
• Use bacalhau config default to generate a Bacalhau configuration file in YAML format without any user-specific overrides in place.
• Use bacalhau config auto-resources to detect available system capacity (CPU, RAM, Disk, GPUs, etc.) and write the capacity into the config file. Users can specify a number from 0-100 to limit the capacity to that percentage of total capacity. A number over 100 can also be used to overcommit on capacity, which is useful mainly for setting queue capacities.

Improvements and Bug Fixes

• Sanity checking of WebAssembly resource usage: we now check that WebAssembly jobs have requested...

v1.1.7-rc1

What's Changed

• Layout of plugins directory by @rossjones in #3026
• Fix exec probe rejection message. by @simonwo in #3035
• Small Repo Cleanup by @rossjones in #3037
• support GCP buckets by @wdbaruni in #3040
• WebUI with ability to spin up using the serve command by @XandraMcC in #3044
• Ensure commands using the Client APIs are cancellable by @simonwo in #3048
• Run Web UI in deployed Bacalhau clusters. by @simonwo in #3047
• Temporarily hide "View" button until Job Detail and Node Detail pages are available by @XandraMcC in #3051
• Web UI: design updates and tweaks by @simonwo in #3052
• CSS more accurately represents design document by @XandraMcC in #3056
• Remove PNGs and replace with SVGs. by @simonwo in #3054
• feat: add a command to generate a default config and list keys by @frrist in #3012
• Integrate observe-sdk to get tracing information during Wasm execution by @zshipko in #2971
• Support multiple AMD and Nvidia GPUs for Docker jobs by @simonwo in #3061
• [circleci] Build swagger reference - this is an automatic commit [skip ci] by @enricorotundo in #3064
• More gracefully handle bad tool output by @simonwo in #3065
• Webui fix job labels to display properly by @olgibbons in #3067
• Tell StorageProviders where to store data by @rossjones in #3053
• fix: correctly parse resource config by @frrist in #3072
• Bulk update OpenTelemetry dependencies. by @simonwo in #3069
• Webui job detail by @XandraMcC in #3066
• Copy large inline sources to IPFS by @rossjones in #3073

New Contributors

• @zshipko made their first contribution in #2971

Full Changelog: v1.1.6...v1.1.7-rc1

v1.1.6

What's Changed

• Allow IPFS to run in devstack when ./plugins exists by @rossjones in #3024
• Add Templating Support to bacalhau job run by @wdbaruni in #3020
• s3 storage pattern filtering by @wdbaruni in #3031

Full Changelog: v1.1.5...v1.1.6

v1.1.6-rc1

What's Changed

• Allow IPFS to run in devstack when ./plugins exists by @rossjones in #3024
• Add Templating Support to bacalhau job run by @wdbaruni in #3020
• s3 storage pattern filtering by @wdbaruni in #3031

Full Changelog: v1.1.5...v1.1.6-rc1

v1.1.5

What's Changed

• change to orchestrator endoint file to allow cross origin comms by @XandraMcC in #2998
• Improve bid strategy error messages by @simonwo in #3006
• Show node ranking failures in the CLI (again) by @simonwo in #3008
• Fix Python linting and ensure packages are built on PRs. by @simonwo in #3007
• Bugfix: invalid wasm memory request by @Nasfame in #2992
• refactor: no longer write a config file by default by @frrist in #3009

Full Changelog: v1.1.4...v1.1.5

v1.1.5-rc1

What's Changed

• change to orchestrator endoint file to allow cross origin comms by @XandraMcC in #2998
• Improve bid strategy error messages by @simonwo in #3006
• Show node ranking failures in the CLI (again) by @simonwo in #3008
• Fix Python linting and ensure packages are built on PRs. by @simonwo in #3007
• Bugfix: invalid wasm memory request by @Nasfame in #2992
• refactor: no longer write a config file by default by @frrist in #3009

Full Changelog: v1.1.4...v1.1.5-rc1

v1.1.4

What's Changed

• allow describing nodes with their short IDs by @wdbaruni in #2986
• support shortIDs for GetHistory and GetExecutions by @wdbaruni in #2990
• fixes: image manifest cache by @Nasfame in #2993
• Choose better log stream addresses internally by @rossjones in #2996
• enable downloading s3 results by @wdbaruni in #2995
• change prod bootstrapping peers by @wdbaruni in #2999

Full Changelog: v1.1.3...v1.1.4

v1.1.4-rc2

What's Changed

• allow describing nodes with their short IDs by @wdbaruni in #2986
• support shortIDs for GetHistory and GetExecutions by @wdbaruni in #2990
• fixes: image manifest cache by @Nasfame in #2993
• Choose better log stream addresses internally by @rossjones in #2996
• enable downloading s3 results by @wdbaruni in #2995
• change prod bootstrapping peers by @wdbaruni in #2999

Full Changelog: v1.1.3...v1.1.4-rc2