Skip to content

Pre Built Packages

Jump to bottom
Adam Lindberg edited this page May 7, 2018 · 2 revisions

Version 1.0

Creating Packages

Format

The OTP installation is compressed into a single .tar.gz archive. The following will be stripped out before creating the archive:

  • erts-*/bin/beam (since we use beam.bin) peer: I would keep that since it contains the debug symbols if needed
  • erts-*/{include,lib,src,man,doc} folders
  • usr folder
  • bin/{erl,start_erl,start,epmd} files
  • lib/*/{src,c_src,examples} folders (unclear if they're needed to build the release, but probably not)

The following can be investigated to further minimize the size of packages:

  • compression of lib/* app folders into .ez archives (could speed up booting of VM as well) peer: would do that later after we verified it really speeds up booting)

The file name should be grisp_otp_build_<VERSION>_<HASH>.tar.gz.

  • <VERSION> is the OTP version string used in rebar.config (see Rebar Configuration)
  • <HASH> is the compound hash of all overridden files in the build

Hashing

To get a unique package ID, we use the combination of the OTP version and a compound hash of all modified or added source files. The procedure to collect all files, compute the hash and copy them into the build is as follows:

  1. Create the definitive list of source and destination paths, together with their (rendered) content
  2. Hash the content using SHA256 (crypto:hash(sha256, Bin))
  3. Create a newline delimited list of checksums and file name pairs separated by a space. Terminate the list with a newline. E.g.: 
    d315a259c467c67dc51ffa5187043a3518be549cc9d1a94d304936b201e88fec path/to/file/a
91da537e28c457a93bbd2555ca718a903f78d6a28a32bb24081e525c8a7f3be4 another/path/to/file/b
  4. Hash this to create the compound hash for the OTP build (the <HASH> component)
  5. Use this hash
    1. When getting a package, use it to build the URL
    2. When building from source, place the file list and compound hash in the files GRISP_PACKAGE_FILES and GRISP_PACKAGE_CHECKSUM respectively in the built OTP install directory

Using Packages

Downloading

Archives will be put on an DigitalOcean Spaces bucket and be publicly available via HTTP/HTTPS. The address for a packages will be https://endpoint.example.com/packages/otp/build/grisp_otp_build_<VERSION>_<HASH>.tar.gz.

Rebar Configuration

The Rebar configuration semantics are changed to allow for package downloads to be the default. The toolchain path is moved into a build key, and only if this key exists a source build will be performed. If it doesn't exist, the version configured for OTP will be used to download the right package.

Example:

{grisp, [
    {otp, [{version, "20.2"}]},
    {deploy, [{destination, "/Volumes/GRISP"}]},
    {build, [ % Source only built if this exists
        {toolchain, [{directory, "PATH/TO/TOOLCHAIN"}]}
    ]}
]}.

Caching

Once the packages is downloaded, it is to be placed in $HOME/.cache/grisp/packages/otp/build/grisp_otp_build_<VERSION>_<HASH>.tar.gz. If a package already exists when downloading, an MD5 sum should be calculated for the existing cached package and used in the ETag header when requesting the same package version from the repository. If the packages is newer in the repository, a new version should be downloaded and replace the current cached one.

Extraction

Packages should be extracted from the cached archive to the path $HOME/.cache/grisp/packages/otp/build/grisp_otp_build_<VERSION>_<HASH> folder. If the folder already exists, it should only be replaced with a new extraction if the folder modification time is older than the cached archive.

The deploy task should be updated to figure out if OTP is built from source or from package and use the correct folder when creating the release.

Deployment

The deploy task should be enhanced to automatically download the archived OTP build. If building from source is configured, the deploy plugin should fail if there is no OTP install in _grisp/otp/<VERSION>/install and require running rebar3 grisp build manually before.

Clone this wiki locally