3d-forge

Read/Write quantized-mesh tiles

Dependecies

$ sudo apt-get install postgresql postgresql-contrib postgis python-pip gdal-bin python-gdal mono-complete

Set up the required variables in your .bashrc

export PGPASS=xxx
export DBHOST=xxx
export BUCKETNAME=xxx
export PROFILENAME=xxx
export LOGFILEFOLDER=xxx

Getting started

$ make all

Create the database, user and import the shapefiles in the DB

$ make create populate

Test a tile generated in S3

$ wget --header="Accept-Encoding: gzip,deflate" http://3d.geo.admin.ch/1.0.0/ch.swisstopo.terrain.3d_water/default/20151231/4326/12/4268/3110.terrain -O 12_43268_3110.terrain.gz
$ gzip -d 12_43268_3110.terrain.gz

Interactive programming

$ source venv/bin/activate
$ ipython
$ run {yourScript}.py

Enter debug mode

> %debug

Get the latest traceback

> %tb

Styling

Check styling

  make lint

Fix styling (only pep8 errors)

  make autolint

Copy a file to S3 from command line

  aws --profile $PROFILENAME s3 cp layer.json  s3://$BUCKETNAME/xxx/layer.json

Create 3d-tiling instance

• Adapt your configurations in .bashrc

• start DB manually

  su postrgres
  /usr/lib/postgresql/9.4/bin/pg_ctl -D /var/lib/postgresql/9.4/main/ -l start # 'stop' to stop
  

• connect to db host

• sudo su postgres

  • change password of superuser to xxx with the following sql query:

    ALTER ROLE pgkogis WITH PASSWORD 'xxxxxx';

  • postgres=# \q and then $ exit

• add credentials to .boto file or .aws/config or .aws/credentials

• mount EFS

• get project with git clone --recursive https://github.com/geoadmin/3d-forge

• install with make install

• create db with make createdb, adapt configs/terrain/database.cfg for different data sets

• create tiles with tmspyramid, adapt configs/terrain/tms.cfg for different tiler configurations

• you can also deactivate stats collection by setting track_activities to off in /etc/postgresql/9.1/main/postgresql.conf

• choose the appropriate logging level for postgres client_min_messages = error and log_min_messages = error and log_min_error_statement = error

• if you don't have tileforge user make sure you're using md5 mode in pg_hba.conf (local all all md5)

Remove logs

rm -f /var/log/tileforge/*.log

sudo su postgres

rm -r /var/log/postgresql/*

SQL Helper Functions

During the installation some sql functions are installed on the target database.

meta.sql

This script is creating a view called public.meta to the database. This view contains useful information (table size, index size, row count) for the monitoring of database growth during import. note: the rowcount is coming from the statistics collector and is only containing values during import.

$ psql -d forge -c "SELECT * FROM meta;"
         Table         | Size Total | External Size | rowcount
-----------------------+------------+---------------+----------
 break_lines_0_25m     | 94 GB      | 27 GB         |        0
 break_lines_0_5m      | 38 GB      | 11 GB         |        0
 break_lines_1m        | 19 GB      | 5631 MB       |        0
 break_lines_2m        | 10078 MB   | 2968 MB       |        0
 break_lines_4m        | 1967 MB    | 575 MB        |        0
 break_lines_8m        | 904 MB     | 265 MB        |        0
 break_lines_16m       | 653 MB     | 194 MB        |        0
 break_lines_32m       | 458 MB     | 136 MB        |        0
 mnt25_simplified_100m | 390 MB     | 113 MB        |        0
 break_lines_64m       | 330 MB     | 96 MB         |        0
 spatial_ref_sys       | 3360 kB    | 176 kB        |        0
 topology              | 24 kB      | 24 kB         |        0
 layer                 | 24 kB      | 24 kB         |        0
 zoomlevel_6           | 16 kB      | 8192 bytes    |        1
 zoomlevel_4           | 16 kB      | 8192 bytes    |        1
 zoomlevel_3           | 16 kB      | 8192 bytes    |        1
 zoomlevel_8           | 16 kB      | 8192 bytes    |        1
 zoomlevel_2           | 16 kB      | 8192 bytes    |        1
 zoomlevel_7           | 16 kB      | 8192 bytes    |        1
 zoomlevel_5           | 16 kB      | 8192 bytes    |        1
 zoomlevel_0           | 16 kB      | 8192 bytes    |        1
 table_z17_extent      | 16 kB      | 8192 bytes    |        0
 zoomlevel_1           | 16 kB      | 8192 bytes    |        1

visualize_tms_grid.sql

This script is installing the following sql functions to the database:

• public.bgdi_global_geodetic_recursive(geom geometry,maxlevels integer)
• public.bgdi_lonlat2tile(lon double precision DEFAULT 7.0, lat double precision DEFAULT 46.0, zoom integer DEFAULT 5)

bgdi_global_geodetic_recursive(geom geometry,maxlevels integer) can be used to visualize tile geometries and label them with their tms address or google address. You have to pass an input geometry and the maximal depth / zoomlevel to the function. Of course these parameters should be set with caution. Generating all the tiles for a big geometry until zoomlevel 20 will take hours!

$ psql -c "select id,zoomlevel,x,y,st_astext(the_geom),label_tsm,label_google FROM bgdi_global_geodetic_recursive(st_setsrid('BOX(8.15974 46.75665,8.16465 46.75913)'::box2d::geometry, 4326),5);" -d forge
 id | zoomlevel | x  | y  |                            st_astext                            | label_tsm | label_google
----+-----------+----+----+-----------------------------------------------------------------+-----------+--------------
  0 |         0 |  1 |  0 | POLYGON((0 -90,0 90,180 90,180 -90,0 -90))                      | 0/1/0     | 0/1/0
  1 |         1 |  2 |  0 | POLYGON((0 0,0 90,90 90,90 0,0 0))                              | 1/2/0     | 1/2/1
  2 |         2 |  4 |  0 | POLYGON((0 45,0 90,45 90,45 45,0 45))                           | 2/4/0     | 2/4/3
  3 |         3 |  8 |  2 | POLYGON((0 45,0 67.5,22.5 67.5,22.5 45,0 45))                   | 3/8/1     | 3/8/6
  4 |         4 | 16 |  5 | POLYGON((0 45,0 56.25,11.25 56.25,11.25 45,0 45))               | 4/16/3    | 4/16/12
  5 |         5 | 32 | 11 | POLYGON((5.625 45,5.625 50.625,11.25 50.625,11.25 45,5.625 45)) | 5/33/7    | 5/33/24

This function can be used for the visualization in qgis:

The function public.bgdi_lonlat2tile(lon double precision DEFAULT 7.0, lat double precision DEFAULT 46.0, zoom integer DEFAULT 5) can be used to calculate the tile address at a given zoomlevel. The function returns a point geomtry:

• x-coordinate -> X Tile Address
• y-coordinate -> Y Tile Address a la google
• z-coordinate -> Y Tile Address global geodetic (TSM)

$ psql -c "select st_astext(bgdi_lonlat2tile(8.159259,46.758162,17));" -d forge
          st_astext
------------------------------
 POINT Z (137013 99584 31487)
(1 Zeile)

watermask.sql

This script is installing the following sql functions to the database:

• bgdi_watermask_rasterize(geometry, integer, integer, regclass, text)

bgdi_watermask_rasterize(geometry, integer, integer, regclass, text) can be used to create the watermask for a given tile geometry. All the lake intersections within this tile will be rasterized, the raster dimension (pixel width and height) and the lake geometry table has to be set within the function call.

The input parameters of the function are:

bgdi_watermask_rasterize(
bbox geometry                 -- geometry object with the tile geometry
, width integer               -- tile width in pixels
, height integer              -- tile height in pixels
, watermask_table regclass    -- schema.table containing lake geometry
, watermask_geom_column text  -- geometry column
)

The returned raster is of type '1BB' 1-bit boolean [1]:

1 -> water
0 -> land

Example Query:

SELECT bgdi_watermask_rasterize(st_setsrid('POLYGON((6.865 46.379, 6.865 46.383 , 6.869 46.383 , 6.869 46.379, 6.865 46.379 ))'::geometry,4326), 256,256,'v25_pri25_a_seeflaechen'::regclass,'the_geom'::text)as geom;

The current implementation is rasterizing the vector features with the GDAL ALL_TOUCHED=TRUE option.

[1] http://postgis.net/docs/RT_ST_BandPixelType.html

TODO

• Fix normal vectors computation at tile borders
• Add comprehensive doc