Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tile statistics to mbtiles #986

Merged
merged 30 commits into from
Nov 13, 2023
Merged
Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
3016d16
Add stats command
sharkAndshark Nov 1, 2023
99ffdc2
Inline page_size query
sharkAndshark Nov 8, 2023
d68f612
Use MbtType directly
sharkAndshark Nov 8, 2023
8b7ff70
Use bounds and assert_yaml_snapshot
sharkAndshark Nov 9, 2023
21b4180
fmt and clippy
sharkAndshark Nov 9, 2023
ff5fd34
Update expected stats.txt
sharkAndshark Nov 9, 2023
b9668d0
Use size-format to prettify output
sharkAndshark Nov 9, 2023
25f07b9
Fmt and clippy
sharkAndshark Nov 9, 2023
9a7e5b8
Use epsg 4326
sharkAndshark Nov 9, 2023
a0c27e7
Update statistics structure
sharkAndshark Nov 9, 2023
fb878ca
Refactor
sharkAndshark Nov 9, 2023
8a33a60
Add document about mbtiles stats
sharkAndshark Nov 10, 2023
549bfaa
Update doc link
sharkAndshark Nov 10, 2023
7befd50
Code style update
sharkAndshark Nov 10, 2023
5daea72
Update test for coord transformation
sharkAndshark Nov 10, 2023
7c3b83c
Update naming
sharkAndshark Nov 10, 2023
1030ece
Inlining
sharkAndshark Nov 10, 2023
f56193f
Update test for meter_to_lnglat
sharkAndshark Nov 10, 2023
b45285f
Update crate importing
sharkAndshark Nov 10, 2023
9d5e950
Right align numbers
sharkAndshark Nov 11, 2023
2083448
Add error msg for displaying statistics
sharkAndshark Nov 12, 2023
8865786
Make table title upcase
sharkAndshark Nov 12, 2023
7a0c9e5
Add epmty file test
sharkAndshark Nov 12, 2023
b73684a
Formar SQL
sharkAndshark Nov 13, 2023
6c63f51
Cleanup display implementation
sharkAndshark Nov 13, 2023
eefaf72
Cleanup cargo.toml
sharkAndshark Nov 13, 2023
005f035
Keep links still in document
sharkAndshark Nov 13, 2023
1e76b8a
Update stats document
sharkAndshark Nov 13, 2023
ea979b3
Cleanup
sharkAndshark Nov 13, 2023
a0f2079
Merge branch 'main' into mbstats
nyurik Nov 13, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ actix-http = "3"
actix-rt = "2"
actix-web = "4"
anyhow = "1.0"
approx = "0.5.1"
async-trait = "0.1"
bit-set = "0.5.3"
brotli = "3"
Expand Down Expand Up @@ -49,6 +50,7 @@ rustls-native-certs = "0.6"
rustls-pemfile = "1"
semver = "1"
serde = { version = "1", features = ["derive"] }
size_format = "1.0.2"
sharkAndshark marked this conversation as resolved.
Show resolved Hide resolved
serde_json = "1"
serde_yaml = "0.9"
spreet = { version = "0.9", default-features = false }
Expand Down
20 changes: 20 additions & 0 deletions docs/src/52-mbtiles-stats.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Get a tile statistics from MBTiles file

For the concern of efficiency, you could figure out the page size and file size, tile size and bounds(represented as WGS 84 latitude and longitude values) of covered area with `mbtiles stats` command.

```shell
mbtiles stats /path/to/my_file.mbtiles
File: ./tests/fixtures/mbtiles/world_cities.mbtiles
FileSize: 48.00KiB
Schema: flat
Page size: 4.00KiB
| zoom | count |smallest | largest | average | bbox |
sharkAndshark marked this conversation as resolved.
Show resolved Hide resolved
| 0 | 1 | 1.08KiB | 1.08KiB | 1107B |-179.99999997494382,-85.05112877764508,180.00000015460688,85.05112879314403|
| 1 | 4 | 160B | 650B | 366B |-179.99999997494382,-85.05112877764508,180.00000015460688,85.05112879314403|
| 2 | 7 | 137B | 495B | 239B |-179.99999997494382,-66.51326042021836,180.00000015460688,66.51326049182072|
| 3 | 17 | 67B | 246B | 134B |-134.99999995874995,-40.9798980140281,180.00000015460688,66.51326049182072|
| 4 | 38 | 64B | 175B | 86B |-134.99999995874995,-40.9798980140281,180.00000015460688,66.51326049182072|
| 5 | 57 | 64B | 107B | 72B |-123.74999995470151,-40.9798980140281,180.00000015460688,61.60639642757953|
| 6 | 72 | 64B | 97B | 68B |-123.74999995470151,-40.9798980140281,180.00000015460688,61.60639642757953|
| all | 196 |64B|1.0KiB|96B|
```
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ The `validate` command will run `PRAGMA integrity_check` on the file, and will f
The `validate` command will verify that the `tiles` table/view exists, and that it has the expected columns and indexes. It will also verify that the `metadata` table/view exists, and that it has the expected columns and indexes.

## Per-tile validation
If the `.mbtiles` file uses [flat_with_hash](54-mbtiles-schema.md#flat-with-hash) or [normalized](54-mbtiles-schema.md#normalized) schema, the `validate` command will verify that the MD5 hash of the `tile_data` column matches the `tile_hash` or `tile_id` columns (depending on the schema).
If the `.mbtiles` file uses [flat_with_hash](55-mbtiles-schema.md#flat-with-hash) or [normalized](55-mbtiles-schema.md#normalized) schema, the `validate` command will verify that the MD5 hash of the `tile_data` column matches the `tile_hash` or `tile_id` columns (depending on the schema).
sharkAndshark marked this conversation as resolved.
Show resolved Hide resolved

A typical Normalized schema generated by tools like [tilelive-copy](https://github.com/mapbox/TileLive#bintilelive-copy) use MD5 hash in the `tile_id` column. The Martin's `mbtiles` tool can use this hash to verify the content of each tile. We also define a new [flat-with-hash](54-mbtiles-schema.md#flat-with-hash) schema that stores the hash and tile data in the same table, allowing per-tile validation without the multiple table layout.
A typical Normalized schema generated by tools like [tilelive-copy](https://github.com/mapbox/TileLive#bintilelive-copy) use MD5 hash in the `tile_id` column. The Martin's `mbtiles` tool can use this hash to verify the content of each tile. We also define a new [flat-with-hash](55-mbtiles-schema.md#flat-with-hash) schema that stores the hash and tile data in the same table, allowing per-tile validation without the multiple table layout.

Per-tile validation is not available for the `flat` schema, and will be skipped.

Expand Down
File renamed without changes.
7 changes: 4 additions & 3 deletions docs/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,8 @@
- [Recipes](45-recipes.md)
- [Tools](50-tools.md)
- [MBTiles Metadata](51-mbtiles-meta.md)
- [MBTiles Copying / Diffing](52-mbtiles-copy.md)
- [MBTiles Validation](53-mbtiles-validation.md)
- [MBTiles Schemas](54-mbtiles-schema.md)
- [MBTiles statistics](52-mbtiles-stats.md)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't want to break all internet links every time we add a new docs page... we ought to figure out a way not to do that. For now, lets just add thing at the end.

- [MBTiles Copying / Diffing](53-mbtiles-copy.md)
- [MBTiles Validation](54-mbtiles-validation.md)
- [MBTiles Schemas](55-mbtiles-schema.md)
- [Development](60-development.md)

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion mbtiles/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,16 @@ anyhow = { workspace = true, optional = true }
clap = { workspace = true, optional = true }
env_logger = { workspace = true, optional = true }
serde_yaml = { workspace = true, optional = true }
size_format.workspace = true
sharkAndshark marked this conversation as resolved.
Show resolved Hide resolved
tokio = { workspace = true, features = ["rt-multi-thread"], optional = true }

[dev-dependencies]
# For testing, might as well use the same async framework as the Martin itself
actix-rt.workspace = true
approx.workspace = true
ctor.workspace = true
env_logger.workspace = true
insta = { workspace = true, features = ["toml"] }
insta = { workspace = true, features = ["toml", "yaml"] }
pretty_assertions.workspace = true
rstest.workspace = true

Expand Down
10 changes: 10 additions & 0 deletions mbtiles/src/bin/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@ enum Commands {
/// MBTiles file to read from
file: PathBuf,
},
/// Gets tile statistics from MBTiels file
#[command(name = "stats")]
Stats { file: PathBuf },
/// Gets a single value from the MBTiles metadata table.
#[command(name = "meta-get")]
MetaGetValue {
Expand Down Expand Up @@ -114,6 +117,13 @@ async fn main_int() -> anyhow::Result<()> {
let mbt = Mbtiles::new(file.as_path())?;
mbt.validate(integrity_check, update_agg_tiles_hash).await?;
}
Commands::Stats { file } => {
let mbt = Mbtiles::new(file.as_path())?;
let mut conn = mbt.open_readonly().await?;

let statistics = mbt.statistics(&mut conn).await?;
println!("{statistics}");
}
}

Ok(())
Expand Down
Loading