Operating cdimage
=================

cdimage is a collection of Python scripts wrapped around the debian-cd
package.  Originally, it was just the few things you had to type to make
debian-cd build CDs; since then it has evolved to handle generating package
lists using germinate, live CD building, and CD publication for multiple
projects.  The following is a quick operator's guide.

cdimage requires Python >= 2.7.  It should also work with Python >= 3.1 (not
3.0), but at the moment this has only been confirmed by unit-testing and not
by field experience.

Running the cdimage test suite also requires python-mock.


Day-to-day operation
--------------------

etc/crontab lists the jobs that are run regularly from cron.  These include
the following scripts, which may be called by hand:

  * cron.daily

    Build install CDs for all relevant Ubuntu architectures.

  * cron.daily-live

    Build live CDs for all relevant Ubuntu architectures.

  * cron.dvd

    Build DVDs for all relevant Ubuntu architectures.  These are normally
    combined install/live images.

  * for-project

    Execute a command in the context of a certain project: run for-project
    without arguments to see the available list.  At the moment, this just
    sets the PROJECT and CAPPROJECT environment variables and then runs the
    command (many parts of cdimage and of the Ubuntu branch of debian-cd pay
    attention to these variables), but you should always use the for-project
    command to do this in case this changes.

    The project context affects a number of parameters of image building and
    publication, such as the set of seeds passed to germinate, the location
    of live filesystem images, the preseed files installed on the CD, the
    place where images are published in the WWW output tree, and the
    frequency with which old published images are purged.

Each of these scripts sets a few variables and then calls the main
underlying script, which is the core of cdimage:

  * build-image-set

This calls lots of other scripts to build a set of images.  For day-to-day
operation it is not necessary to be intimately familiar with all of these
scripts, but fixing bugs may require some level of knowledge of them.
Here's a quick run-down of the steps:

  * anonftpsync

    Sync cdimage's mirror of the Ubuntu archive.

  * update-local-indices

    Update index files for the local override archive; packages in this
    archive will be used in preference to packages in the Ubuntu archive
    proper.  Nowadays, of course, this local archive is rarely used, but it
    is still occasionally useful for specialised custom builds.

  * extract-debootstrap

    Extract debootstrap scripts from debootstrap-udeb packages in the
    archive.  debian-cd uses these to make sure that it has a complete base
    system on the CD.  (This is largely unnecessary as of Ubuntu 5.10, since
    debootstrap now calculates the base system based on the Packages file.)

  * run-germinate

    Run germinate for each architecture to determine the list of packages in
    each seed output (desktop, installer, etc.).  See
    https://wiki.ubuntu.com/SeedManagement and the germinate documentation
    for more details.

  * germinate-to-tasks

    Translate germinate output into debian-cd task lists.

  * update-tasks

    Install new debian-cd task lists.  This can optionally mail the
    differences versus the last set of task lists to an operator, which can
    be useful for spotting problems.

  * download-live-filesystems

    For live CD builds, fetch the live filesystem image from a build daemon.

At this stage, debian-cd's build_all.sh script is invoked to build the
images.

  * check-installable

    For install CD builds, generate a report on uninstallable packages on
    the image.

  * publish-daily

    Publish the image to the WWW output tree.

  * purge-old-images

    Delete images older than a certain number of days (configured in
    etc/purge-days) from the WWW output tree.

  * sync-mirrors

    Trigger mirrors to update from the WWW output tree, thereby making the
    images available to the world.


Releasing images
----------------

From time to time, an operator will need to tag a set of images as a release
of some kind.  The script responsible for this (which is always run by hand,
never automatically) is:

  * publish-release

    This is heavily geared to the requirements of Ubuntu and its
    derivatives, although it's possible that other users could adapt it to
    their needs.  It publishes images from a daily build (as published by
    publish-daily) to a special release tree.  An OFFICIAL argument governs
    the layout of the result.  Run publish-release without arguments to see
    a usage message.

There are in fact three important subtrees of the release tree.  The
'simple' tree is intended for smaller mirrors and for ease of use by naïve
end users.  It contains a pool of images and a tree per release of symlinks
into that pool with filenames that include the status of the image (e.g.
preview, sounder-9, release).

The 'full' tree contains everything except the releases that are in the
simple tree (so in practice it contains alpha/beta releases), and has a more
complicated structure that ordinary users ultimately shouldn't have to pay
too much attention to.

The 'torrent' tree is mirrored by the BitTorrent tracker so that it knows
what it's supposed to be tracking.  It is not otherwise visible to the
world.

It is best to give a few examples here.  The Breezy Colony 5 release was
published using the following commands:

  for-project ubuntu publish-release daily 20050923.3 install no colony-5
  for-project ubuntu publish-release daily-live 20050923.1 live no colony-5
  sync-mirrors

As you can see, it is possible to publish two sets of images into the same
release directory, provided that they have different types (the "install"
and "live" arguments).

Note that it is always necessary to run sync-mirrors after publish-release;
publish-release does not do so automatically, in order to give the operator
a chance to make sure that the release tree is correct before publishing it
to the world with sync-mirrors.

The Edubuntu 5.10 Preview release (which consisted of only an install CD)
was initially published using the following commands:

  for-project edubuntu publish-release daily 20050909.4 install poolonly preview
  sync-mirrors

The "poolonly" OFFICIAL argument instructs publish-release to place the
images only in the .pool directory, which is generally not visible in
directory listings on mirrors.  This is polite to mirrors by giving them a
little time to catch up before the images are made visible to everyone (and
generally announced shortly afterwards) using the following commands:

  for-project edubuntu publish-release daily 20050909.4 install yes preview
  sync-mirrors

When the Edubuntu 5.10 release is later published, publish-release will
automatically remove the preview images.

HTML indices
------------

A requirement common to both daily and release publication is the generation
of HTML pages describing the images that are available.  This is mostly
handled automatically, but the script is full of special cases and sadly the
output is not always perfect, so it is sometimes necessary to correct it and
run it by hand.

  * make-web-indices

    Generate HEADER.html, FOOTER.html, and .htaccess files instructing
    Apache to emit directory listings with added descriptions of what images
    are available.

The HEADER.html, FOOTER.html, and .htaccess files at the top of the simple
tree and in each project's subdirectory are currently generated by hand, as
are the version number symlinks; because they only change every six months,
there has not been much impetus to automate this task.  When publishing a
release to the simple tree for the first time, the operator will need to
edit these files manually.