Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Polygon increase label grid limit(Beta version) #996

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

CrazyBug-11
Copy link

@CrazyBug-11 CrazyBug-11 commented Aug 24, 2024

The current feature is incomplete, and the existing code does not achieve the expected results! #993

Copy link

github-actions bot commented Aug 24, 2024

This Branch e8e211e Base cf49b86
0:01:05 DEB [archive] - Tile stats:
0:01:05 DEB [archive] - Biggest tiles (gzipped)
1. 14/4942/6092 (156k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.40015 (poi:84k)
2. 9/154/190 (149k) https://onthegomap.github.io/planetiler-demo/#9.5/41.77078/-71.36719 (landcover:85k)
3. 10/308/380 (138k) https://onthegomap.github.io/planetiler-demo/#10.5/41.90214/-71.54297 (landcover:66k)
4. 10/308/381 (137k) https://onthegomap.github.io/planetiler-demo/#10.5/41.63994/-71.54297 (landcover:72k)
5. 14/4941/6092 (113k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.42212 (poi:65k)
6. 14/4941/6093 (111k) https://onthegomap.github.io/planetiler-demo/#14.5/41.81227/-71.42212 (building:62k)
7. 14/4940/6092 (100k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.44409 (building:92k)
8. 11/616/762 (99k) https://onthegomap.github.io/planetiler-demo/#11.5/41.7057/-71.63086 (landcover:71k)
9. 14/4942/6091 (96k) https://onthegomap.github.io/planetiler-demo/#14.5/41.84501/-71.40015 (building:79k)
10. 11/616/761 (96k) https://onthegomap.github.io/planetiler-demo/#11.5/41.83679/-71.63086 (landcover:72k)
0:01:05 DEB [archive] - Max tile sizes
                      z0    z1    z2    z3    z4    z5    z6    z7    z8    z9   z10   z11   z12   z13   z14   all
           boundary  155   375   444   584   939   341   435   550   775  1.6k  2.1k  7.2k  6.4k  5.8k  4.5k  7.2k
              water 7.7k  3.7k  8.6k  5.5k  2.6k  5.1k   15k   18k   16k   26k   15k   13k   17k   15k   12k   26k
              place    0     0   441   441   441   640   714    1k  1.6k  3.1k  5.7k  3.3k  1.7k   803   948  5.7k
            landuse    0     0     0     0   549   695  1.6k  6.8k   17k   44k   59k   50k   38k   19k   12k   59k
     transportation    0     0     0     0   314   850  1.2k    6k    8k   24k   17k   19k   65k   49k   34k   65k
           waterway    0     0     0     0   112   119     0     0     0  3.1k  2.3k  2.1k  2.1k  4.9k  2.4k  4.9k
               park    0     0     0     0     0     0  1.2k    4k  9.7k   19k   13k  8.2k  4.3k  3.4k  4.4k   19k
transportation_name    0     0     0     0     0     0   369   464  1.2k  1.8k  5.5k  4.7k  3.9k  3.4k   18k   18k
          landcover    0     0     0     0     0     0     0  9.5k   29k   85k   72k   81k   53k   30k   24k   85k
      mountain_peak    0     0     0     0     0     0     0  1.1k  1.8k  3.4k  4.3k  2.8k  1.4k  1.4k   869  4.3k
         water_name    0     0     0     0     0     0     0     0     0   486   461   433   452  1.2k  1.5k  1.5k
    aerodrome_label    0     0     0     0     0     0     0     0     0     0   666   328   273   221   221   666
            aeroway    0     0     0     0     0     0     0     0     0     0  1.6k  2.1k    3k  3.4k  2.8k  3.4k
                poi    0     0     0     0     0     0     0     0     0     0     0     0   506   503   84k   84k
           building    0     0     0     0     0     0     0     0     0     0     0     0     0   59k   92k   92k
        housenumber    0     0     0     0     0     0     0     0     0     0     0     0     0     0   35k   35k
          full tile 7.9k    4k  9.5k  6.5k  3.8k  6.1k   20k   42k   85k  203k  185k  135k  114k  129k  246k  246k
            gzipped 6.2k  3.6k  7.1k  5.2k  3.1k  4.9k   14k   29k   60k  149k  138k   99k   84k   92k  156k  156k
0:01:05 DEB [archive] -    Max tile: 246k (gzipped: 156k)
0:01:05 DEB [archive] -    Avg tile: 5.4k (gzipped: 4.1k) using weighted average based on OSM traffic
0:01:05 DEB [archive] -     # tiles: 4,115,029
0:01:05 DEB [archive] -  # features: 5,492,804
0:01:05 INF [archive] - Finished in 18s cpu:1m8s avg:3.7
0:01:05 INF [archive] -   read    1x(3% 0.5s wait:17s done:1s)
0:01:05 INF [archive] -   encode  4x(56% 10s wait:2s done:1s)
0:01:05 INF [archive] -   write   1x(22% 4s wait:12s done:1s)
0:01:05 INF [archive] - Finished in 1m5s cpu:3m33s gc:1s avg:3.3
0:01:05 INF [archive] - FINISHED!
0:01:05 INF [archive] - 
0:01:05 INF [archive] - ----------------------------------------
0:01:05 INF [archive] - data errors:
0:01:05 INF [archive] - 	render_snap_fix_input	16,671
0:01:05 INF [archive] - 	osm_multipolygon_missing_way	360
0:01:05 INF [archive] - 	osm_boundary_missing_way	73
0:01:05 INF [archive] - 	merge_snap_fix_input	12
0:01:05 INF [archive] - 	osm_boundary_duplicate_member	2
0:01:05 INF [archive] - 	feature_centroid_if_convex_osm_invalid_multipolygon_empty_after_fix	2
0:01:05 INF [archive] - 	omt_fix_water_before_ne_intersect	1
0:01:05 INF [archive] - 	feature_polygon_osm_invalid_multipolygon_empty_after_fix	1
0:01:05 INF [archive] - 	feature_point_on_surface_osm_invalid_multipolygon_empty_after_fix	1
0:01:05 INF [archive] - ----------------------------------------
0:01:05 INF [archive] - 	overall          1m5s cpu:3m33s gc:1s avg:3.3
0:01:05 INF [archive] - 	lake_centerlines 3s cpu:6s avg:2.4
0:01:05 INF [archive] - 	  read     1x(19% 0.5s done:2s)
0:01:05 INF [archive] - 	  process  4x(0% 0s done:2s)
0:01:05 INF [archive] - 	  write    1x(0% 0s done:2s)
0:01:05 INF [archive] - 	water_polygons   15s cpu:42s avg:2.8
0:01:05 INF [archive] - 	  read     1x(39% 6s done:7s)
0:01:05 INF [archive] - 	  process  4x(28% 4s wait:4s done:5s)
0:01:05 INF [archive] - 	  write    1x(4% 0.6s wait:10s done:5s)
0:01:05 INF [archive] - 	natural_earth    6s cpu:13s avg:2
0:01:05 INF [archive] - 	  read     1x(96% 6s)
0:01:05 INF [archive] - 	  process  4x(13% 0.8s wait:6s)
0:01:05 INF [archive] - 	  write    1x(0% 0s wait:6s)
0:01:05 INF [archive] - 	osm_pass1        2s cpu:6s avg:3.3
0:01:05 INF [archive] - 	  read     1x(2% 0s wait:2s)
0:01:05 INF [archive] - 	  parse    4x(32% 0.6s)
0:01:05 INF [archive] - 	  process  1x(69% 1s)
0:01:05 INF [archive] - 	osm_pass2        18s cpu:1m12s avg:3.9
0:01:05 INF [archive] - 	  read     1x(0% 0s wait:11s done:8s)
0:01:05 INF [archive] - 	  process  4x(75% 14s)
0:01:05 INF [archive] - 	  write    1x(2% 0.4s wait:18s)
0:01:05 INF [archive] - 	ne_lakes         0s cpu:0s avg:0
0:01:05 INF [archive] - 	boundaries       0s cpu:0s avg:1.3
0:01:05 INF [archive] - 	agg_stop         0s cpu:0s avg:16.5
0:01:05 INF [archive] - 	sort             1s cpu:4s avg:2.8
0:01:05 INF [archive] - 	  worker  1x(51% 0.7s)
0:01:05 INF [archive] - 	archive          18s cpu:1m8s avg:3.7
0:01:05 INF [archive] - 	  read    1x(3% 0.5s wait:17s done:1s)
0:01:05 INF [archive] - 	  encode  4x(56% 10s wait:2s done:1s)
0:01:05 INF [archive] - 	  write   1x(22% 4s wait:12s done:1s)
0:01:05 INF [archive] - ----------------------------------------
0:01:05 INF [archive] - 	archive	108MB
0:01:05 INF [archive] - 	features	290MB
-rw-r--r-- 1 runner docker 85M Aug 27 09:35 run.jar
0:01:04 DEB [archive] - Tile stats:
0:01:04 DEB [archive] - Biggest tiles (gzipped)
1. 14/4942/6092 (156k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.40015 (poi:84k)
2. 9/154/190 (149k) https://onthegomap.github.io/planetiler-demo/#9.5/41.77078/-71.36719 (landcover:85k)
3. 10/308/380 (138k) https://onthegomap.github.io/planetiler-demo/#10.5/41.90214/-71.54297 (landcover:66k)
4. 10/308/381 (137k) https://onthegomap.github.io/planetiler-demo/#10.5/41.63994/-71.54297 (landcover:72k)
5. 14/4941/6092 (113k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.42212 (poi:65k)
6. 14/4941/6093 (111k) https://onthegomap.github.io/planetiler-demo/#14.5/41.81227/-71.42212 (building:62k)
7. 14/4940/6092 (100k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.44409 (building:92k)
8. 11/616/762 (99k) https://onthegomap.github.io/planetiler-demo/#11.5/41.7057/-71.63086 (landcover:71k)
9. 14/4942/6091 (96k) https://onthegomap.github.io/planetiler-demo/#14.5/41.84501/-71.40015 (building:79k)
10. 11/616/761 (96k) https://onthegomap.github.io/planetiler-demo/#11.5/41.83679/-71.63086 (landcover:72k)
0:01:04 DEB [archive] - Max tile sizes
                      z0    z1    z2    z3    z4    z5    z6    z7    z8    z9   z10   z11   z12   z13   z14   all
           boundary  155   375   444   584   939   341   435   550   775  1.6k  2.1k  7.2k  6.4k  5.8k  4.5k  7.2k
              water 7.7k  3.7k  8.6k  5.5k  2.6k  5.1k   15k   18k   16k   26k   15k   13k   17k   15k   12k   26k
              place    0     0   441   441   441   640   714    1k  1.6k  3.1k  5.7k  3.3k  1.7k   803   948  5.7k
            landuse    0     0     0     0   549   695  1.6k  6.8k   17k   44k   59k   50k   38k   19k   12k   59k
     transportation    0     0     0     0   314   850  1.2k    6k    8k   24k   17k   19k   65k   49k   34k   65k
           waterway    0     0     0     0   112   119     0     0     0  3.1k  2.3k  2.1k  2.1k  4.9k  2.4k  4.9k
               park    0     0     0     0     0     0  1.2k    4k  9.7k   19k   13k  8.2k  4.3k  3.4k  4.4k   19k
transportation_name    0     0     0     0     0     0   369   464  1.2k  1.8k  5.5k  4.7k  3.9k  3.4k   18k   18k
          landcover    0     0     0     0     0     0     0  9.5k   29k   85k   72k   81k   53k   30k   24k   85k
      mountain_peak    0     0     0     0     0     0     0  1.1k  1.8k  3.4k  4.3k  2.8k  1.4k  1.4k   869  4.3k
         water_name    0     0     0     0     0     0     0     0     0   486   461   433   452  1.2k  1.5k  1.5k
    aerodrome_label    0     0     0     0     0     0     0     0     0     0   666   328   273   221   221   666
            aeroway    0     0     0     0     0     0     0     0     0     0  1.6k  2.1k    3k  3.4k  2.8k  3.4k
                poi    0     0     0     0     0     0     0     0     0     0     0     0   506   503   84k   84k
           building    0     0     0     0     0     0     0     0     0     0     0     0     0   59k   92k   92k
        housenumber    0     0     0     0     0     0     0     0     0     0     0     0     0     0   35k   35k
          full tile 7.9k    4k  9.5k  6.5k  3.8k  6.1k   20k   42k   85k  203k  185k  135k  114k  129k  246k  246k
            gzipped 6.2k  3.6k  7.1k  5.2k  3.1k  4.9k   14k   29k   60k  149k  138k   99k   84k   92k  156k  156k
0:01:04 DEB [archive] -    Max tile: 246k (gzipped: 156k)
0:01:04 DEB [archive] -    Avg tile: 5.4k (gzipped: 4.1k) using weighted average based on OSM traffic
0:01:04 DEB [archive] -     # tiles: 4,115,029
0:01:04 DEB [archive] -  # features: 5,492,804
0:01:04 INF [archive] - Finished in 19s cpu:1m8s avg:3.7
0:01:04 INF [archive] -   read    1x(3% 0.5s wait:17s done:1s)
0:01:04 INF [archive] -   encode  4x(55% 10s wait:2s done:1s)
0:01:04 INF [archive] -   write   1x(21% 4s wait:13s done:1s)
0:01:04 INF [archive] - Finished in 1m4s cpu:3m32s gc:1s avg:3.3
0:01:04 INF [archive] - FINISHED!
0:01:04 INF [archive] - 
0:01:04 INF [archive] - ----------------------------------------
0:01:04 INF [archive] - data errors:
0:01:04 INF [archive] - 	render_snap_fix_input	16,671
0:01:04 INF [archive] - 	osm_multipolygon_missing_way	360
0:01:04 INF [archive] - 	osm_boundary_missing_way	73
0:01:04 INF [archive] - 	merge_snap_fix_input	12
0:01:04 INF [archive] - 	osm_boundary_duplicate_member	2
0:01:04 INF [archive] - 	feature_centroid_if_convex_osm_invalid_multipolygon_empty_after_fix	2
0:01:04 INF [archive] - 	omt_fix_water_before_ne_intersect	1
0:01:04 INF [archive] - 	feature_polygon_osm_invalid_multipolygon_empty_after_fix	1
0:01:04 INF [archive] - 	feature_point_on_surface_osm_invalid_multipolygon_empty_after_fix	1
0:01:04 INF [archive] - ----------------------------------------
0:01:04 INF [archive] - 	overall          1m4s cpu:3m32s gc:1s avg:3.3
0:01:04 INF [archive] - 	lake_centerlines 2s cpu:5s avg:2.4
0:01:04 INF [archive] - 	  read     1x(21% 0.5s done:2s)
0:01:04 INF [archive] - 	  process  4x(0% 0s done:2s)
0:01:04 INF [archive] - 	  write    1x(0% 0s done:2s)
0:01:04 INF [archive] - 	water_polygons   15s cpu:41s avg:2.8
0:01:04 INF [archive] - 	  read     1x(40% 6s done:7s)
0:01:04 INF [archive] - 	  process  4x(27% 4s wait:4s done:5s)
0:01:04 INF [archive] - 	  write    1x(4% 0.6s wait:10s done:5s)
0:01:04 INF [archive] - 	natural_earth    7s cpu:13s avg:2.1
0:01:04 INF [archive] - 	  read     1x(95% 6s)
0:01:04 INF [archive] - 	  process  4x(12% 0.8s wait:6s)
0:01:04 INF [archive] - 	  write    1x(0% 0s wait:6s)
0:01:04 INF [archive] - 	osm_pass1        2s cpu:6s avg:3.4
0:01:04 INF [archive] - 	  read     1x(2% 0s wait:2s)
0:01:04 INF [archive] - 	  parse    4x(34% 0.6s)
0:01:04 INF [archive] - 	  process  1x(72% 1s)
0:01:04 INF [archive] - 	osm_pass2        18s cpu:1m11s avg:3.9
0:01:04 INF [archive] - 	  read     1x(0% 0s wait:11s done:7s)
0:01:04 INF [archive] - 	  process  4x(76% 14s)
0:01:04 INF [archive] - 	  write    1x(2% 0.4s wait:18s)
0:01:04 INF [archive] - 	ne_lakes         0s cpu:0s avg:0
0:01:04 INF [archive] - 	boundaries       0s cpu:0s avg:1.3
0:01:04 INF [archive] - 	agg_stop         0s cpu:0s avg:19.8
0:01:04 INF [archive] - 	sort             1s cpu:4s avg:2.7
0:01:04 INF [archive] - 	  worker  1x(49% 0.7s)
0:01:04 INF [archive] - 	archive          19s cpu:1m8s avg:3.7
0:01:04 INF [archive] - 	  read    1x(3% 0.5s wait:17s done:1s)
0:01:04 INF [archive] - 	  encode  4x(55% 10s wait:2s done:1s)
0:01:04 INF [archive] - 	  write   1x(21% 4s wait:13s done:1s)
0:01:04 INF [archive] - ----------------------------------------
0:01:04 INF [archive] - 	archive	108MB
0:01:04 INF [archive] - 	features	290MB
-rw-r--r-- 1 runner docker 85M Aug 27 09:36 run.jar

Full logs: https://github.com/onthegomap/planetiler/actions/runs/10575539878

@msbarry
Copy link
Contributor

msbarry commented Aug 26, 2024

Thanks @CrazyBug-11 ! Would you mind giving me access to push to your fork?

@CrazyBug-11
Copy link
Author

Certainly, I've already invited you.

@msbarry
Copy link
Contributor

msbarry commented Aug 26, 2024

Thanks, but I'm pretty sure you just need to click the "allow edits from maintainers" button on the PR

@CrazyBug-11
Copy link
Author

CrazyBug-11 commented Aug 26, 2024

Haha, got it, I've already click the allow edits by maintainers

@msbarry
Copy link
Contributor

msbarry commented Aug 26, 2024

OK thanks! Pushed some end to end tests that should pass once this is working.

@CrazyBug-11
Copy link
Author

Thank you for your support. @msbarry
would like to know why the first point of the feature is used. Based on my understanding, at the same zoom level, the first point of a feature should only be present in one tile. However, after executing sliceIntoTiles, the feature is split into multiple tiles. Why isn't the number of points limited by taking the vertices of each smaller feature after the split?

Copy link

sonarcloud bot commented Aug 27, 2024

@msbarry
Copy link
Contributor

msbarry commented Aug 27, 2024

We want to show/hide an entire feature, not individual parts of a feature that may appear on different tiles otherwise it could appear broken. It could either use the centroid for the whole feature, the first point, an interior point, etc. to compute the label grid coordinate. In practice this should only get used on very small features so it probably shouldn't make much of a difference although I could see switching to centroid as well.

Can you try this out and see how it works for your dataset?

@CrazyBug-11
Copy link
Author

Thank you very much for your patient guidance.
I noticed that FeatureGroup.TileFeatures.add applies certain restrictions to features, but I couldn’t locate the logic that verifies if the current tile information matches the group information.For example, if a large feature is split across (1,0,0) and (1,1,0), is the group calculation performed twice within the same tile grid, or is it handled separately for each tile?
If you could point me to the specific part of the code where this is managed, that would be really helpful. I’m planning to carefully review the workflow in this section.

@msbarry
Copy link
Contributor

msbarry commented Aug 28, 2024

Yes exactly, the features get grouped into tiles then when reading all the features for a tile, FeatureGroup.TileFeatures.add starts ignoring features in the current tile once a group ID (in that tile) has too many features in it. Since a polygon's center or first point always falls into one tile it doesn't matter if the entire large feature is split across many tiles. The results might look strange though, so in practice you'd only want to use this on small polygons/lines.

In the case where the tile size is not evenly divisible by the group size, you need to take care to make the tile buffer large enough that the same set of features show up in each group when that group is duplicated between adjacent tiles. But for 16, 32, 64, 128 that's not a concern.

@msbarry
Copy link
Contributor

msbarry commented Aug 28, 2024

Ahh actually now that I think more about it, this could be problematic since if part of a large feature appears in this tile then the center of it might be far away, and there could be lots of other small features around the center point that would cause it to get hidden on the tile that contains all of those features, but this tile doesn't have visibility into them.

The right way to do this would be to turn it into a 2-step process: emit all raw features along with their group ID, sort/group/filter by the group ID, then continue processing all of those features. There are other operations that could benefit from this "global grouping" step so might make sense to happen along with them.

But for very small features (smaller than the tile buffer) I think that the limited implementation here could still work?

@CrazyBug-11
Copy link
Author

I have a dataset of over 300 million land-use polygons, the data is about 200g, and after testing, the results seem a bit strange. There are noticeable connection boundaries between tiles, and at certain zoom levels, you can clearly see the four borders of each tile. After further testing, I've realized that even if this functionality is implemented, it doesn't meet my current needs. Please forgive me for initially thinking in the wrong direction. I thought there would be a large boundary feature encompassing all the data, so even if some small features didn't hit, the boundary would serve as the background. However, in reality, there isn't such a large boundary feature, but rather many small polygons. When features are limited, it leads to many small areas being left blank.

What I actually need is to achieve a display where detailed polygon outlines are shown only at higher zoom levels, such as 11-14.
image
However, at lower zoom levels (0-10), the entire area should still be covered, achieving a similar effect to the following:
image

To ensure full coverage while keeping the tile size within approximately 1MB, my idea is to pixelate the polygons before executing encodeAndEmitFeature. For example, with a tile resolution of 256x256, I would calculate how many pixels each geom to be output occupies, and then replace the original feature with small squares for each pixel. The goal is to minimize the size of each feature while ensuring that the area is fully covered. Could you help me assess whether this approach is feasible? I really appreciate your help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants