Skip to content

How to build gdal and ogr2ogr2

Jump to bottom
Rafa de la Torre edited this page Sep 30, 2015 · 9 revisions

Introduction

I'm writing this guide to let others reproduce the process and add patches to important packages we use in production.

gdal is one of the most important libraries in CartoDB. Many things depend on it:

  • import process (ogr2ogr binary)
  • mapnik (at least at build time)
  • its python bindings
  • postgis

At the time of writing this lines, we are using gdal 1.11.0.

But at some point we wanted to deploy new features present in gdal trunk but not in the latest stable version, namely the csv type guessing. In order to minimize risk, we decided to do so by packaging the ogr2ogr binary statically and make it coexist with gdal. This is the raison d'être of the ogr2ogr2 package.

Both packages are built and hosted in launchpad

Generating a new version of ogr2ogr2

Setup source repo

There's a project in github in order to simplify the generation of new packages. My recommendation is to clone from the upstream repository (actually a mirror), then fetch the references from the helper repository:

git clone https://github.com/OSGeo/gdal
cd gdal
git remote add cartodb https://github.com/cartodb/gdal
git fetch cartodb

There are two important branches in that repository:

  • upstream pointing to the upstream commit from which we generate the package
  • ogr2ogr2 pointing to the source package (all the debian files, see below for more on this).

This is done so that we can simply merge a branch to update the source package while keeping the debian files kind of separated from the upstream changes.

Update with changes in trunk

So, in order to generate a new version from trunk:

  1. Update trunk from the upstream repo:
git checkout trunk
git pull origin trunk
  1. Update the upstream branch to make it point to most recent trunk:
git checkout upstream
git merge --ff-only origin/trunk

(the merge should always be a fast-forward)

  1. Update the branch that contains the debian packages with upstream:
git checkout ogr2ogr2
git merge upstream
  1. Modify the gdal/debian/changelog file. Beware debian tools are pretty picky with the format of the package changelog.
git show -s --format=%B upstream # Take note of the svn version it is based on.
cd gdal
dch -i
  1. Re-generate the .tar.gz with the upstream code:
cd ..
git rm ogr2ogr2_*+svn.*.orig.tar.gz
SVN_REV=$(git show -s --format=%B upstream | grep git-svn-id | sed -r 's/.*trunk@([0-9]+).*/\1/')
VERSION=$(cat gdal/VERSION)
# make sure the source dir is clean before generating the .tar.gz
tar -h --exclude='./ogr2ogr2/debian' -czf ogr2ogr2_${VERSION}+svn.${SVN_REV}.orig.tar.gz ogr2ogr2/

Generate and upload the source package

So now you should have the package sources updated locally.

To build the binary package:

cd ogr2ogr2
debuild -us -uc

or to build in parallel:

DEB_BUILD_OPTIONS="parallel=4" debuild -us -uc

To build the source package (you need a gpg key to sign it. See launchpad docs for more info):

# E.g: debuild -S -sa -k'Rafa de la Torre <[email protected]>'
debuild -S -sa -k<GPGKEY>

To upload to launchpad and get it built there:

cd ..
dput ppa:cartodb/gis-testing ogr2ogr2_${VERSION}+svn.28306-precise1ubuntu1_source.changes

Commit and push the changes

git add gdal/debian/changelog
git add ogr2ogr2_${VERSION}+svn.${SVN_REV}.orig.tar.gz
git commit -m "New version of ogr2ogr2 based on svn repo @${SVN_REV}"
git push cartodb ogr2ogr2 upstream

Tag the new version

git tag -a ogr2ogr2_${VERSION}+svn.${SVN_REV}-precise1 -m "Sources for ogr2ogr2-static-bin_${VERSION}+svn.${SVN_REV}-precise1_amd64.deb"
git push --tags cartodb ogr2ogr2_${VERSION}+svn.${SVN_REV}-precise1

Debian packaging crash course

Essential tools

sudo apt-get install build-essential fakeroot devscripts

Get the sources

If you followed the cartodb setup guide (add PPA's) this will pick the source packages from http://ppa.launchpad.net/cartodb/gis/ubuntu/

apt-get source gdal
apt-get source ogr2ogr2-static-bin

note you can also pick the sources from their repositories. But debian source packages are meant to let you reproduce a build of a binary package.

Get the build dependencies

sudo apt-get build-dep gdal
sudo apt-get build-dep ogr2ogr2-static-bin

Building

cd gdal-1.10.1/
fakeroot debian/rules binary # a priori this does not clean before building

Patching

Create a patch for the package

First, setup everything, including .quiltrc file. See https://wiki.debian.org/UsingQuilt

quilt push -a # ensure all patches are applied
quilt new patch_name
quilt add filename.c # the file you want to modify
vim filename.c # make changes

Update changelog

dch -n

Build the modified package

fakeroot debian/rules binary

or

DEB_BUILD_OPTIONS=nostrip,noopt fakeroot debian/rules binary

the nostrip means keep the debugging symbols. Stripping is the process of removing them. The noopt means "do not apply optimization options" when building, which is needed in some circumstances to debug the code. Otherwise the debugger can return something like "optimized output" when trying to inspect values, which basically means "I don't know how to display it, because the compiler applied some optimizations".

These options are pretty dependent on the source package.

Make the patch

quilt refresh # if more changes made to the file being patched
quilt pop -a  # revert the file(s) to the download state and add the changes to the patch file

Build the source package

debuild -us -uc

Get the differences with the previous source package

debdiff package-blabla.dsc package-blabla.1.dsc > another_debdiff.diff

Links

Clone this wiki locally