Quantized-mesh output support

[ahuarte47]

Provides some new features:

• New quantized-mesh format ouput (-f Mesh), optionally you could add VertexNormals with -N option.
• New metadata file output.
• New cesium-friendly option to automatically create missing root tiles (level 0).
• Consider nodata values when creating tiles.
• Catch GDAL 'Integer overflow' error when creating tiles of low levels.
• Refactoring of code to easily write new output formats.

Details:
https://www.linkedin.com/pulse/fast-cesium-terrain-rendering-new-quantized-mesh-output-alvaro-huarte/

[kvuorine]

Seems very good so far during my tests, well done.

Performance of Cesium took a leap forward compared to heightmap terrain tiles.

There is one problem, when I tried to play with --mesh-qfactor option, ctb-tile segfaulted.

[ahuarte47]

Thanks @kvuorine. About --mesh-qfactor error, may I have your settings or data to try fix it?

[kvuorine]

With commands below you can get some of the data I am working with, I tested that this dataset produces
--mesh-qfactor error too

wget http://kartat.kapsi.fi/files/korkeusmalli/hila_2m/etrs-tm35fin-n2000/L4/L41/L4141E.tif http://kartat.kapsi.fi/files/korkeusmalli/hila_2m/etrs-tm35fin-n2000/L4/L41/L4141G.tif http://kartat.kapsi.fi/files/korkeusmalli/hila_2m/etrs-tm35fin-n2000/L4/L41/L4141F.tif http://kartat.kapsi.fi/files/korkeusmalli/hila_2m/etrs-tm35fin-n2000/L4/L41/L4141H.tif
gdalbuildvrt merged.vrt L4141?.tif
mkdir terrain
ctb-tile -o terrain -f Mesh -g 1.1 merged.vrt

[ahuarte47]

ok, I'll review this as soon as possible

[ahuarte47]

Hi @kvuorine, I have fixed this option. Thanks for your notice!

[Rik79]

Hello @ahuarte47 ! I tried to use this code to generate quantized mesh for Cesium and it works very well.
Thank you! 👍 🥇

I had only an issue when i tried to generate tiles from two different adjacent tiff images (from SRTM dataset) : on the seam between these tiles there is a "canyon", a crack. I am sure that the tiffs data are correct, they are perfectly adjacent.

I launched "ctb-tile" 2 times, each with a tile and on the seam this is the result...

do you have some advice? should be the two tiles concatenated before launch ctb-tile?

The result...

[ahuarte47]

Hi, I think you need an unique MDT that covers all area in order to execute the ctb-tile only once. You could create a virtual raster with "gdalbuildvrt" to virtually create an unique source from many individual files.

You can see a previous comment to how to use or read about "gdalbuildvrt" tool:

gdalbuildvrt merged.vrt L4141?.tif
mkdir terrain
ctb-tile -o terrain -f Mesh -g 1.1 merged.vrt

[Rik79]

Ok thank you very much, i will try this command! (Probably this is the solution, but i am asking the need of memory for the entire world tileset!)

Looking at the image is interesting to note that the first generated tileset on the right is "cutted", the second on the left is "seamed" to globe. Futhermore looking far where the tiles are loaded with minor zoom level the canyon appers to be larger. These can be clues to uderstand.
I am wondering if "ctb-tile" has an option not to seam to the ground level the tileset border.

About the generation of the "layout.json" file, also this works very well, but again in my case with two tiles generated it is not very usefull. Could be interesting a tool that analize the directory tree and from this generate the layout.json.

All this things let me understand that my approach is not correct. :)

[Rik79]

Ok, as not said. :)

"gdalbuildvrt" solves perfectly the problems, also the "layer.json" one. Thank you a lot! :)

[ediebold]

Is there a technical reason as to why the new "-l" option is separate from the actual terrain generation? For large datasets, generating the layer.json file still takes quite a while.

[ahuarte47]

Hi @ediebold, there is none technical reason. But I thought it seems better two separate tasks, I can be wrong but it looks more versatile. You could create the layer.json of existing old terrain sets, or update only a specific level without overwriting an existing json file (You could have edited it with your own settings). If you update a level, only the limits of it are calculated.

[kvuorine]

How about creating layer.json if there is none when generating tiles, but update layer.json only with command line option selected?

I have been testing --mesh-qfactor option which now doesn't crash the program, but I don't quite get the purpose of this option because tileset size on disk doesn't seem to vary at all when testing different values. @ahuarte47 could you explain it a bit?

[ahuarte47]

Thanks everybody for your advices, I will try to integrate the -l option as complementary to -f option, but it will require me to do many changes in the original base code, but indeed it seems more logic.

About --mesh-qfactor option, it is supposed larger values should mean minor amount of triangles with minor quality. As my little tutorial said, the process takes care of an input geometric error to safely merge each pair of triangles with its parent one if there is not a visible loss of quality. The factor multiplies the default error value, larger values should mean minor quality.

Did not you notice any change? You can set "wireframe" rendering mode in Cesium to compare between different factors

[ahuarte47]

Hi @kvuorine, I changed the code, now CTB creates the layer.json if there is none when generating tiles. Thanks everybody!

[kvuorine]

@ahuarte47 Thanks for your support.

I did notice difference in terrain. I just thought simplified mesh would take less space on disk and be quicker to serve over net. I guess simple mesh still reduces rendering workload.

I tried to read your docs/tutorials earlier but that registration barrier was just too much for me.

[epifanio]

How to serve the generated tiles? @ahuarte47 I guess storing them on a simple apache directory is not an option, right? I tried to copy some tiles here temp-tiles (they are using the same files used by @kvuorine )
sand-castle example returns:

An error occurred in "CesiumTerrainProvider": An error occurred while accessing https://epinux.com/epinux_data/shared/data/terrain2/layer.json.

[ahuarte47]

Set as URL the root directory:
https://epinux.com/epinux_data/shared/data/terrain2/

without the json file.

[ahuarte47]

Also, You have clone the missing level-0 folder from the other existing one (You can google about it, you will find this trick).

#1 or #2

UPDATE: You could read this, it notices some steps to take care (Search... "So I got 65 000 terrain tiles").

[kvuorine]

@epifanio , I also have following apache directives for the directory

AddType application/octet-stream terrain
AddEncoding gzip terrain

Not sure if quantized-mesh requires them, but I had problems serving heightmap tiles without those.

[muratkendir]

Hi,
I finally rendered our first city-level quantized-mesh terrain with help of Alvaro's CTB. Thanks for all kind of useful information.

[epifanio]

Hi, I'm still having troubles with the missing tiles issue.
I tried several time to debug the issue but I can't figure it uout ...
I still have the missing 0.terrain isue. In my last attmpt I'm serving the tiles directly from the /root/Apps/ Cesium forlder using nodejs server.js this is exactly what I did:

• download and unzip cesium.js src code:

mkdir Cesium
wget -O Cesium/Cesium-1.43.zip https://github.com/AnalyticalGraphicsInc/cesium/releases/download/1.43/Cesium-1.43.zip
cd Cesium
unzip Cesium-1.43.zip

• install Cesium
  npm install

• cd into cesium/Apps

cd Apps

• add a new HTML file with the following code in it (add your BingMapsApi):

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, minimum-scale=1, user-scalable=no">
  <title>Hello World!</title>
  <script src="../Build/Cesium/Cesium.js"></script>
  <style>
      @import url(../Build/Cesium/Widgets/widgets.css);
      html, body, #cesiumContainer {
          width: 100%; height: 100%; margin: 0; padding: 0; overflow: hidden;
      }
  </style>
</head>
<body>
  <div id="cesiumContainer"></div>
  <script>
    Cesium.BingMapsApi.defaultKey = '------------------------'
    var viewer = new Cesium.Viewer('cesiumContainer');
    var terrainProvider = new Cesium.CesiumTerrainProvider({
      url: 'tiles/'
    });
    viewer.terrainProvider = terrainProvider;
    viewer.scene.globe.enableLighting = true;
  </script>
</body>
</html>

• Download and merge the terrain sample GeoTiff in a gdal VRT file:

wget http://kartat.kapsi.fi/files/korkeusmalli/hila_2m/etrs-tm35fin-n2000/L4/L41/L4141E.tif \
http://kartat.kapsi.fi/files/korkeusmalli/hila_2m/etrs-tm35fin-n2000/L4/L41/L4141G.tif \
http://kartat.kapsi.fi/files/korkeusmalli/hila_2m/etrs-tm35fin-n2000/L4/L41/L4141F.tif \
http://kartat.kapsi.fi/files/korkeusmalli/hila_2m/etrs-tm35fin-n2000/L4/L41/L4141H.tif 
gdalbuildvrt merged.vrt L4141?.tif

• Generate the terrain tiles using ctb-tile:

mkdir tiles
ctb-tile -o tiles -f Mesh -g 1.1 merged.vrt

• Download the blank terrain tile"

wget -O 0.terrain https://raw.github.com/geo-data/cesium-terrain-builder/master/data/smallterrain-blank.terrain

• Add the blank tile where needed (here I maybe do something wrong)

mkdir -p tiles/0/0/
cp 0.terrain tiles/0/0/

• Serve the app using nodejs:

cd ..
nodejs server.js --port 8000 --public true

On OSX, using safari as the browser, I got the following in the JS console:

[Log] An error occurred in "w": Failed to obtain terrain tile X: 0 Y: 0 Level: 0. (Cesium.js, line 477)
[Log] An error occurred in "w": Failed to obtain terrain tile X: 1 Y: 0 Level: 0. (Cesium.js, line 477)

Can you help me to understand what I'm missing/doing wrong?

[CosmosEcho]

Hi @ahuarte47, I got a error in my building. Please help me!
error C2662: 'char **CPLStringList::List(void)' : cannot convert 'this' pointer from 'const CPLStringList' to 'CPLStringList &'
Win10/gdal2.2.4/visual studio 2015

[ahuarte47]

Hi @CosmosEcho, I succesfully built with VS2017, but I did a change

[CosmosEcho]

@ahuarte47 Which gdal version did you used？

[ahuarte47]

I use : GDAL 2.3.0dev, released 2017/99/99
Can you build CTB?

[CosmosEcho]

I can not build. The problem is not solved. I think that I need change my env as yours, but I could not found GDAL 2.3.0dev https://trac.osgeo.org/gdal/wiki/DownloadSource

[ahuarte47]

Is the problem here?

[CosmosEcho]

Yes this is.

[ahuarte47]

It is weird, the signature of this function has not changed for a long time, it is equal in 2.0...2.2

Edit:
If you are only going to create Terrain or Mesh Tiles (No GDAL raster outputs, tiff, png,...), you can disable this line to build, this function will not be called.

[CosmosEcho]

@ahuarte47 OK, thank you！I will try it again.

[CosmosEcho]

Is that true? "GDALCreateOverviewDataset missing since gdal-2.2"

https://github.com/OSGeo/gdal/issues/413

[CosmosEcho]

Hi @ahuarte47, I still can't build completely. Can you give me a builded exe? I want to test my data.

[markerikson]

I did some experimenting on Friday with re-implementing the MBTiles changes on top of the latest version of this branch, and generated some test terrain that included the vertex normals data. At first I didn't see any difference visually between quantized-mesh + vertex normals, and quantized-mesh without normals. I finally figured out that it only matters when you've got viewer.scene.globe.enableLighting = true, which changes from a solid light everywhere to simulating globe lighting based on the position of the sun.

For our use case we don't care about time of day lighting, so I'm going to skip trying to regenerate that terrain set using vertex normals.

I will try to finish re-implementing the MBTiles and missing tiles logic on top of this branch, but it'll have to wait until I have time to finish it outside of work, which may be a while.

[ahuarte47]

Hi @CosmosEcho; where could I copy you a zip file (31mb) with CTB+GDAL for Windows x64?

[CosmosEcho]

@ahuarte47 It's very nice about a zip file. You can send it to my email: [email protected]. Many thanks!

[CosmosEcho]

@ahuarte47 if my data problem.

[bjpirt]

Hello,
Firstly thanks for adding this feature, it will be great to generate terrain files from GDAL files. Unfortunately I'm having an issue building this branch on Mac OS X and I wondered if you might be able to point me in the right direction to get it compiling. I've installed the GDAL framework (v2.2) that came with QGIS. I'm currently getting this error:

/cesium-terrain-builder/src/CTBFileTileSerializer.cpp:117:77: error: member function 'List' not viable: 'this' argument has type
      'const CPLStringList', but function is not marked const
  poDstDS = driver->CreateCopy(temp_filename.c_str(), tile->dataset, FALSE, creationOptions.List(), NULL, NULL);
                                                                            ^~~~~~~~~~~~~~~
/Library/Frameworks/GDAL.framework/Headers/cpl_string.h:522:12: note: 'List' declared here
    char **List() { return papszList; }
           ^
1 error generated.
make[2]: *** [src/CMakeFiles/ctb.dir/CTBFileTileSerializer.cpp.o] Error 1
make[1]: *** [src/CMakeFiles/ctb.dir/all] Error 2
make: *** [all] Error 2

Any clues?
Incidentally, the original branch seemed to compile without issues.

[ahuarte47]

Hello @bjpirt, I have pushed a minor change. Could you test it? thanks!

[ahuarte47]

Ups! sorry I did a mesh with git, I accidentally closed this pull, I reopend it as #64

[Crare]

I keep getting this error:

An error occurred in "T": Failed to obtain terrain tile X: 1 Y: 0 Level: 0 Cesium.js:420

and

An error occurred in "CesiumTerrainProvider": Failed to obtain terrain tile X: 1 Y: 0 Level: 0. Cesium.js:36877

I've added the files to /0/0/0.terrain and /0/1/0.terrain and I have generated layer.json and tried to add existing layer.json too. I can access the files via http http://localhost/data/tilesets/terrain/test/

And I have same location in CesiumTerrainProvider:

let customTerrainProvider = new Cesium.CesiumTerrainProvider({
    url : 'http://localhost/data/tilesets/terrain/test'
});
viewer.scene.terrainProvider = customTerrainProvider;

Where is this web.config you want to edit?
where do you put that .htaccess file exactly?
I've tried gzipping the test-folder that has numbered folders and layer-json in it.

I've tried multiple types of tif images and even converting xyz-files but all they get stuck at that same point where I try to show them in Cesium.js.

I'm using docker versions of ctb and terrain-server, are they up to date?

[BWibo]

@Crare I can confirm that the tumgis/ctb-quantized-mesh Docker image is up-to-date.
It is based on this commit: ahuarte47@4bb78ba

If you want to make sure you are using the latest commit, you can simply build the image yourself.
The lastest commit from ahuarte47/cesium-terrain-builder/tree/master-quantized-mesh is pulled during the build.

git clone https://github.com/tum-gis/cesium-terrain-builder-docker
cd cesium-terrain-builder-docker
docker build -t ctb-quantized-mesh .