Add MBTiles output and support for skipping empty tiles #61
Conversation
Provides two new features: - New quantized-mesh format ouput - New metadata file output
|
Hi, thanks for this effort!
|
|
@epifanio : I definitely don't have time to try out something like that myself. My task at work was specifically to generate a new Cesium terrain dataset we can self-host, and doing that efficiently is what allowed me to spend some time on making these changes to CTB. Now that the tool works enough to generate what I want, I have to move on to other tasks. |
|
I did something of refactoring in #55 to more easly write new output formats (MBTiles, ...). I hope this helps when anyone wants to merge this o whatever |
|
@markerikson how did your project go creating the global qmesh from ALO30? I'm about to start doing the same thing and was curious if you had any key takeaways? |
|
@bneigher : it ultimately worked out pretty well. I was able to generate terrain up through zoom level 13, although that got pretty sizeable. I did have to do a whole bunch of preprocessing for the ALOS30 dataset. I have been wanting to write up a blog post about the process of generating terrain and hosting it, but it's unfortunately been very low priority on my todo list. One problem I ran into was that there's clearly some kind of bug in this branch. The output works, but when generating the last couple of zoom levels, sometimes the If you've got specific questions, feel free to ask! |
That's an issue I am l am looking into as we speak. I can recreate the segmentation fault with a relatively small dsm - so perhaps I can look into what the bug is if I have some free cycles. Yea - I was worried that this bug would also occur during the ALOS30 dataset processing that I'm going to do. It's also bummer you have to download each region one by one.. Anyway - cheers. |
|
@bneigher : the ALOS30 data is actually accessible via an FTP site. They want you to register to get the address, but ultimately it's just there on an FTP site. I wound up writing a Python script to automatically download all the 5x5 .tar.gz files . If you can figure out what the segfault issue is, please let me know! |
|
I am working on merging this PR with ahuerte47 pull/64. I just got the two diffed & compiled. I need to test and I can issue (yet another) PR. Is there anyone maintaining these PRs? |
|
No, sorry, I don't have any further time to look at this myself. Happy to answer any questions you have on what I did, though! |
|
I am happy to answer any questions you have on what I did on #64 too |
|
@markerikson it doesn't look like you write the layer Metadata to the mbtiles db, although you have the sql statements for it. Is that correct? |
|
@a4chet : that sounds right. I know the MBTiles spec has a metadata table, but I ultimately didn't need it for my task. |
|
I have an initial merge of these 2 baselines in my repo here https://github.com/a4chet/cesium-terrain-builder. The layer.json for mbTiles is writing verbatim to the metadata table as a clob. I will issue a pull request after I've verified i've got what I needed into github. Thank you @markerikson & @ahuarte47 for your contributions! I was able to make a world coverage mbTile database @ 7.5 arc seconds for our application. It clocks in at about ~1GB - so not shabby at all! |
|
@a4chet Your branch does not compile on Linux (e.g. your code: |
Fixed & pushed. |
|
I came across this thread looking to do something similar to what I expect everyone else here is doing. However, when I try to compile @a4chet's fork, It's telling me that I can't seem to find where MbTilesDb.cpp/hpp come from originally, so I thought I'd ask for some help here. |
|
To follow up from my previous post, it appears you need to I've been working with both @a4chet and @markerikson's forks for a bit; I still haven't gotten @a4chet's to compile completely, and I think the problem is just things that VSVC lets one get away with that GCC does not, but I haven't been able to suss out all the problems. The current error I'm running into is the following: /home/ubuntu/Hackery/cesium-terrain-builder/tools/ctb-tile.cpp: In function ‘void checkCreateBaseTiles(TerrainBuild*, std::shared_ptr<TerrainSerialize>&, ctb::Grid&)’:
/home/ubuntu/Hackery/cesium-terrain-builder/tools/ctb-tile.cpp:882:61: error: cannot bind non-const lvalue reference of type ‘std::shared_ptr<TerrainMetadata>&’ to an rvalue of type ‘std::shared_ptr<TerrainMetadata>’
runTiler(missingTileName.c_str(), command, grid, std::shared_ptr<TerrainMetadata>(NULL), serializer);If I make things a bit more strict in ctb-tile.cpp (which I might be doing wrong, my C++ is very rusty) with the following (line ~882): missingTileName = createEmptyRootElevationFile(missingTileName, grid, missingTileCoord);
std::shared_ptr<TerrainMetadata> emptyTM = std::shared_ptr<TerrainMetadata>(NULL);
runTiler(missingTileName.c_str(), command, grid, emptyTM, serializer);
VSIUnlink(missingTileName.c_str());the error changes to the much more intimidating: In file included from /home/ubuntu/Hackery/cesium-terrain-builder/tools/ctb-tile.cpp:42:0:
/usr/include/c++/7/thread: In instantiation of ‘struct std::thread::_Invoker<std::tuple<std::packaged_task<int(const char*, TerrainBuild*, const ctb::Grid&, std::shared_ptr<TerrainMetadata>&, std::shared_ptr<TerrainSerialize>&)>, const char*, TerrainBuild*, ctb::Grid, std::shared_ptr<TerrainMetadata>, std::shared_ptr<TerrainSerialize> > >’:
/usr/include/c++/7/thread:127:22: required from ‘std::thread::thread(_Callable&&, _Args&& ...) [with _Callable = std::packaged_task<int(const char*, TerrainBuild*, const ctb::Grid&, std::shared_ptr<TerrainMetadata>&, std::shared_ptr<TerrainSerialize>&)>; _Args = {const char*, TerrainBuild*, ctb::Grid&, std::shared_ptr<TerrainMetadata>&, std::shared_ptr<TerrainSerialize>&}]’
/home/ubuntu/Hackery/cesium-terrain-builder/tools/ctb-tile.cpp:991:88: required from here
/usr/include/c++/7/thread:240:2: error: no matching function for call to ‘std::thread::_Invoker<std::tuple<std::packaged_task<int(const char*, TerrainBuild*, const ctb::Grid&, std::shared_ptr<TerrainMetadata>&, std::shared_ptr<TerrainSerialize>&)>, const char*, TerrainBuild*, ctb::Grid, std::shared_ptr<TerrainMetadata>, std::shared_ptr<TerrainSerialize> > >::_M_invoke(std::thread::_Invoker<std::tuple<std::packaged_task<int(const char*, TerrainBuild*, const ctb::Grid&, std::shared_ptr<TerrainMetadata>&, std::shared_ptr<TerrainSerialize>&)>, const char*, TerrainBuild*, ctb::Grid, std::shared_ptr<TerrainMetadata>, std::shared_ptr<TerrainSerialize> > >::_Indices)’
operator()()
^~~~~~~~I'm not sure what to make of that. @markerikson's fork has been much more forthcoming. I'm able to generate quantized mesh terrain from the NASA ASTER GDEM for zoom levels 0-5 with mixed results (tiles vanish from certain angles of the globe in Cesium, for example) and output it to an MBTiles archive, albeit one with no metadata, and a seemingly incomplete layer.json I've been writing into the metadata table by hand. It's not perfect but it's getting there. |
|
Glad someone's been able to make some use of that code :) I wish I'd been able to get it into a slightly more useable shape, but I just needed to get enough implemented to actually generate our own dataset and move on. |
|
Oh, one more note about @a4chet's fork, before I forget; If you're trying to run this on Ubuntu, you'll need to add the ubuntugis:ubuntugis-unstable PPA and upgrade your libgdal-dev and gdal-bin packages; @markerikson Thank you for checking in! I'm gonna keep banging my head against it, as if I can solve these small problems and implement a means to pass a latitude/longitude bounding box in the options, I'm going to be quite happy. |
|
I'm having a problem I can't make heads or tails of at the moment, and I hope that someone might be able to help. I needed to update @markerikson's fork to the latest quantized-mesh stuff from @ahuarte47's fork, and found myself basically repeating the work @a4chet must've been doing to get his build. I've worked out most of the kinks of migrating MBTiles code into @ahuarte47's branch in its current state, but I'm getting a very similar error to that which I get with @a4chet's fork, so I ended up just copying over his work on supporting multiple filetypes into @ahuarte47's branch and have been compiler-bashing my way through all the problems until I got to this: The code to which it refers is the following: std::shared_ptr<TerrainSerialize> serializer(new TerrainSerialize(command.fileFormat));
// ...
Grid grid;
// ...
std::shared_ptr<TerrainMetadata> metadata = command.metadata || !fileExists(metadataFileName) || (command.fileFormat == TilerFileFormat::MBTiles) ?
std::shared_ptr<TerrainMetadata>(new TerrainMetadata()) :
std::shared_ptr<TerrainMetadata>(NULL);
// Instantiate the threads using futures from a packaged_task
for (int i = 0; i < threadCount ; ++i) {
packaged_task<int(const char *, TerrainBuild *, Grid &, std::shared_ptr<TerrainMetadata>&, std::shared_ptr<TerrainSerialize>&)> task(runTiler); // wrap the function
tasks.push_back(task.get_future()); // get a future
thread(move(task), command.getInputFilename(), &command, grid, metadata, serializer).detach(); // launch on a thread
}And the signature of the function (based on @a4chet's changes) for runTiler is this: static int
runTiler(const char *inputFilename, TerrainBuild *command, Grid &grid, std::shared_ptr<TerrainMetadata> &metadata, std::shared_ptr<TerrainSerialize> &serializer) {I'm truly at a loss; the error, as I read it, doesn't seem like there should be any problem. It finds candidates but fails to convert and I just don't know why. Googling certainly hasn't been of much help yet. The problem appears to be that the runTiler tasks I'm attempting to generate just can't seem to properly match the function signature needed. But nothing I do seems to appease it; everything else is unhandleable; trying to pass it as a *value-of reports that there is no * operator for std::shared_ptr. Trying to pass &address-of complains as well. I really hope I'm just missing something obvious. |
|
Holy crap I think I fixed it. I wrapped my grid, metadata, and serializer variables with |
|
@sevvie : glad to hear it :) I wouldn't have a clue what was causing that error. This is why I'm happy to primarily be a Javascript/Python dev :) |
|
@markerikson A few questions, if you can spare the time:
|
|
@sevvie : "size" as far as file size, geographic area, number of input pixels, ... ? I did a bunch of conversion work to turn the ALOS30 dataset into a better prepared input format, but ultimately it was about feeding a bunch of geotiffs into CTB/GDAL. |
@sevvie Can you please give me your solution, or what you exactly did? I have no idea how to get a successfull build. |
Implements #56 .
This PR is based off an older version of @ahuarte47 's quantized-mesh branch. Since it looks like that branch has been force-pushed a few times, the history in this branch has diverged noticeably. It's probably best if someone were to cherry-pick the commits I've made on top of the latest version of the q-m branch, and integrate things that way. I unfortunately don't have time to tackle that myself, but I wanted to make this code public and available for use.
I've implemented support for writing tiles directly to an MBTiles container, which is much more efficient for transporting large tilesets. It's also faster to write to.
I've also added logic to skip generating tiles if the source terrain for that tile is empty / invalid. Note that this requires GDAL 2.2+, as it's based on the the APIs added in the GDAL "sparse datasets" RFC.
I'm almost done generating a whole-world quantized-mesh terrain dataset. I've used the ALOS30 terrain as a source, and set up a VRT with +75 to -60 degrees latitude coverage. The 30m dataset gives me terrain up to zoom level 14, and I'm generally seeing 2/3 of the tiles per level skipped due to the terrain being "empty" (ie, water cells that had no corresponding source terrain). This is a big deal because of the output size - zoom 11 is about 3GB, zoom 12 is about 12GB, zoom 13 is 50GB, and zoom level 14 looks to be close to 200GB. For comparison, the source dataset is about 90GB.
Unfortunately, it appears that because quantized-mesh encodes ECEF coordinates, a basically empty mesh tile is still unique, whereas with heightmaps they could easily be identical. So, deduplicating quantized-mesh tiles is pointless.
I'd love to see all this functionality get combined in with the latest changes in the original quantized-mesh branch, and would be happy to answer questions about what I've hacked together here.