Extra coordinate precision; feature ordering; compression improvements

* Add an option to retain extra coordinate precision at maxzoom

* Make sure not to shift away the extra detail from coordinates

* Add an option to convert double-precision attributes to single

* Sort attribute values in tiles to make them compress a little better

* Slightly improve polygon simplification

By choosing a point that would be retained after simplification
to be the start/end point that always gets retained

* I regret making all of these tests involve polygons

* Add an option to specify the size of tiny polygons

* Fix accidental requiring of argument for --single-precision

* Guard against duplicate points when generating "sizes" for them

* Restore the intended behavior that tiny polygons don't get simplified

* Make the extra detail settable rather than always maximizing it

* Revert "Improve maxzoom guessing for tightly-clustered point data sources (#4)"

This reverts commit fec5e83.

* Add an option to prevent choosing a base zoom higher than the maxzoom

* Keep the drop rate high enough when the basezoom gets constrained

* Revert "Revert "Improve maxzoom guessing for tightly-clustered point data sources (#4)""

This reverts commit db6bc27.

* Add --order-by and --order-descending options

* Accept multiple --order-by and --order-descending-by sort keys

CHANGELOG.md

@@ -1,3 +1,14 @@
+## 2.5.0
+
+* Add an option to add extra detail at maxzoom that does not factor into guessing
+* Restore the intended behavior that tiny polygons don't get further simplified
+* Add an option to use single-precision floating point in tiles
+* Improve polygon simplification by choosing a better start/end point
+* Sort attribute values in tiles to make them compress a little better
+* Fix dropping of "largest" points when there are duplicate points
+* Add an option to prevent guessing a basezoom higher than the maxzoom
+* Add --order-by and --order-descending options
+
 ## 2.4.1
 
 * Accept tilestats limiting options in tile-join, not just tippecanoe

README.md

@@ -383,6 +383,7 @@ zoom level | precision (ft) | precision (m) | map scale
  * `-d` _detail_ or `--full-detail=`_detail_: Detail at max zoom level (default 12, for tile resolution of 2^12=4096)
  * `-D` _detail_ or `--low-detail=`_detail_: Detail at lower zoom levels (default 12, for tile resolution of 2^12=4096)
  * `-m` _detail_ or `--minimum-detail=`_detail_: Minimum detail that it will try if tiles are too big at regular detail (default 7)
+ * `--extra-detail=`_detail_: Generate tiles with even more detail than the "full" detail at the max zoom level, to maximize location precision. These tiles may not work with some rendering software that internally limits detail to 12 or 13. The extra detail does not affect the choice of maxzoom guessing, the amount of simplification, or the "tiny polygon" threshold as `--full-detail` does. The tiles should look the same as they did without it, except that they will be more precise when overzoomed.
 
 All internal math is done in terms of a 32-bit tile coordinate system, so 1/(2^32) of the size of Earth,
 or about 1cm, is the smallest distinguishable distance. If _maxzoom_ + _detail_ > 32, no additional
@@ -410,6 +411,7 @@ be reduced to the maximum that can be used with the specified _maxzoom_.
  * `-pe` or `--empty-csv-columns-are-null`: Treat empty CSV columns as nulls rather than as empty strings.
  * `-aI` or `--convert-stringified-ids-to-numbers`: If a feature ID is the string representation of a number, convert it to a plain number to use as the feature ID.
  * `--use-attribute-for-id=`*name*: Use the attribute with the specified *name* as if it were specified as the feature ID. (If this attribute is a stringified number, you must also use `-aI` to convert it to a number.)
+ * `-pN` or `--single-precision`: Write double-precision numeric attribute values to tiles as single-precision to reduce tile size.
 
 ### Filtering features by attributes
 
@@ -449,6 +451,7 @@ the same layer, enclose them in an `all` expression so they will all be evaluate
    If you use `-Bg`, it will guess a zoom level that will keep at most 50,000 features in the densest tile.
    You can also specify a marker-width with `-Bg`*width* to allow fewer features in the densest tile to
    compensate for the larger marker, or `-Bf`*number* to allow at most *number* features in the densest tile.
+ * `--limit-base-zoom-to-maximum-zoom` or `-Pb`: Limit the guessed base zoom not to exceed the maxzoom, even if this would put more than the requested number of features in a base zoom tile.
  * `-al` or `--drop-lines`: Let "dot" dropping at lower zooms apply to lines too
  * `-ap` or `--drop-polygons`: Let "dot" dropping at lower zooms apply to polygons too
  * `-K` _distance_ or `--cluster-distance=`_distance_: Cluster points (as with `--cluster-densest-as-needed`, but without the experimental discovery process) that are approximately within _distance_ of each other. The units are tile coordinates within a nominally 256-pixel tile, so the maximum value of 255 allows only one feature per tile. Values around 10 are probably appropriate for typical marker sizes. See `--cluster-densest-as-needed` below for behavior.
@@ -477,6 +480,7 @@ the same layer, enclose them in an `all` expression so they will all be evaluate
  * `-pS` or `--simplify-only-low-zooms`: Don't simplify lines and polygons at maxzoom (but do simplify at lower zooms)
  * `-pn` or `--no-simplification-of-shared-nodes`: Don't simplify away nodes that appear in more than one feature or are used multiple times within the same feature, so that the intersection node will not be lost from intersecting roads. (This will not be effective if you also use `--coalesce` or `--detect-shared-borders`.)
  * `-pt` or `--no-tiny-polygon-reduction`: Don't combine the area of very small polygons into small squares that represent their combined area.
+ * `--tiny-polygon-size=`_size_: Use the specified _size_ for tiny polygons instead of the default 2. Anything above 6 or so will lead to visible artifacts with the default tile detail.
 
 ### Attempts to improve shared polygon boundaries
 
@@ -496,6 +500,8 @@ the same layer, enclose them in an `all` expression so they will all be evaluate
  * `-ao` or `--reorder`: Reorder features to put ones with the same attributes in sequence (instead of ones that are approximately spatially adjacent), to try to get them to coalesce. You probably want to use this if you use `--coalesce`.
  * `-ar` or `--reverse`: Try reversing the directions of lines to make them coalesce and compress better. You probably don't want to use this.
  * `-ah` or `--hilbert`: Put features in Hilbert Curve order instead of the usual Z-Order. This improves the odds that spatially adjacent features will be sequentially adjacent, and should improve density calculations and spatial coalescing. It should be the default eventually.
+ * `--order-by=`_attribute_: Order features by the specified _attribute_, in alphabetical or numerical order. Multiple `--order-by` and `--order-descending-by` options may be specified, the first being the primary sort key.
+ * `--order-descending-by=`_attribute_: Order features by the specified _attribute_, in reverse alphabetical or numerical order. Multiple `--order-by` and `--order-descending-by` options may be specified, the first being the primary sort key.
 
 ### Adding calculated attributes
 

geometry.cpp

@@ -517,7 +517,7 @@ drawvec simple_clip_poly(drawvec &geom, int z, int buffer) {
 
 drawvec reduce_tiny_poly(drawvec &geom, int z, int detail, bool *reduced, double *accum_area) {
 	drawvec out;
-	long long pixel = (1 << (32 - detail - z)) * 2;
+	const long long pixel = (1 << (32 - detail - z)) * tiny_polygon_size;
 
 	*reduced = true;
 	bool included_last_outer = false;
@@ -544,6 +544,11 @@ drawvec reduce_tiny_poly(drawvec &geom, int z, int detail, bool *reduced, double
 				// inner rings must just have their area de-accumulated rather
 				// than being drawn since we don't really know where they are.
 
+				// i.e., this ring (inner or outer) is small enough that we are including it
+				// in a tiny polygon rather than letting it represent itself,
+				// OR it is an inner ring and we haven't output an outer ring for it to be
+				// cut out of, so we are just subtracting its area from the tiny polygon
+				// rather than trying to deal with it geometrically
 				if (std::fabs(area) <= pixel * pixel || (area < 0 && !included_last_outer)) {
 					// printf("area is only %f vs %lld so using square\n", area, pixel * pixel);
 
@@ -552,9 +557,9 @@ drawvec reduce_tiny_poly(drawvec &geom, int z, int detail, bool *reduced, double
 						// XXX use centroid;
 
 						out.push_back(draw(VT_MOVETO, geom[i].x - pixel / 2, geom[i].y - pixel / 2));
-						out.push_back(draw(VT_LINETO, geom[i].x + pixel / 2, geom[i].y - pixel / 2));
-						out.push_back(draw(VT_LINETO, geom[i].x + pixel / 2, geom[i].y + pixel / 2));
-						out.push_back(draw(VT_LINETO, geom[i].x - pixel / 2, geom[i].y + pixel / 2));
+						out.push_back(draw(VT_LINETO, geom[i].x - pixel / 2 + pixel, geom[i].y - pixel / 2));
+						out.push_back(draw(VT_LINETO, geom[i].x - pixel / 2 + pixel, geom[i].y - pixel / 2 + pixel));
+						out.push_back(draw(VT_LINETO, geom[i].x - pixel / 2, geom[i].y - pixel / 2 + pixel));
 						out.push_back(draw(VT_LINETO, geom[i].x - pixel / 2, geom[i].y - pixel / 2));
 
 						*accum_area -= pixel * pixel;
@@ -563,13 +568,17 @@ drawvec reduce_tiny_poly(drawvec &geom, int z, int detail, bool *reduced, double
 					if (area > 0) {
 						included_last_outer = false;
 					}
-				} else {
+				}
+				// i.e., this ring is large enough that it gets to represent itself
+				else {
 					// printf("area is %f so keeping instead of %lld\n", area, pixel * pixel);
 
 					for (size_t k = i; k <= j && k < geom.size(); k++) {
 						out.push_back(geom[k]);
 					}
 
+					// which means that the overall polygon has a real geometry,
+					// which means that it gets to be simplified.
 					*reduced = false;
 
 					if (area > 0) {
@@ -950,14 +959,60 @@ drawvec fix_polygon(drawvec &geom) {
 				ring = tmp;
 			}
 
+			// calculate centroid
+			// a + 1 < size() because point 0 is duplicated at the end
+			long long xtotal = 0;
+			long long ytotal = 0;
+			long long count = 0;
+			for (size_t a = 0; a + 1 < ring.size(); a++) {
+				xtotal += ring[a].x;
+				ytotal += ring[a].y;
+				count++;
+			}
+			xtotal /= count;
+			ytotal /= count;
+
+			// figure out which point is furthest from the centroid
+			long long dist2 = 0;
+			long long furthest = 0;
+			for (size_t a = 0; a + 1 < ring.size(); a++) {
+				long long xd = ring[a].x - xtotal;
+				long long yd = ring[a].y - ytotal;
+				long long d2 = xd * xd + yd * yd;
+				if (d2 > dist2) {
+					dist2 = d2;
+					furthest = a;
+				}
+			}
+
+			// then figure out which point is furthest from *that*
+			long long dist2b = 0;
+			long long furthestb = 0;
+			for (size_t a = 0; a + 1 < ring.size(); a++) {
+				long long xd = ring[a].x - ring[furthest].x;
+				long long yd = ring[a].y - ring[furthest].y;
+				long long d2 = xd * xd + yd * yd;
+				if (d2 > dist2b) {
+					dist2b = d2;
+					furthestb = a;
+				}
+			}
+
+			// rotate ring so the furthest point is the duplicated one.
+			// the idea is that simplification will then be more efficient,
+			// never wasting the start and end points, which are always retained,
+			// on a point that has little impact on the shape.
+
 			// Copy ring into output, fixing the moveto/lineto ops if necessary because of
 			// reversal or closing
 
 			for (size_t a = 0; a < ring.size(); a++) {
+				size_t a2 = (a + furthestb) % (ring.size() - 1);
+
 				if (a == 0) {
-					out.push_back(draw(VT_MOVETO, ring[a].x, ring[a].y));
+					out.push_back(draw(VT_MOVETO, ring[a2].x, ring[a2].y));
 				} else {
-					out.push_back(draw(VT_LINETO, ring[a].x, ring[a].y));
+					out.push_back(draw(VT_LINETO, ring[a2].x, ring[a2].y));
 				}
 			}
 

main.cpp

@@ -66,6 +66,7 @@
 static int low_detail = 12;
 static int full_detail = -1;
 static int min_detail = 7;
+int extra_detail = -1;
 
 int quiet = 0;
 int quiet_progress = 0;
@@ -77,9 +78,13 @@ double simplification = 1;
 size_t max_tile_size = 500000;
 size_t max_tile_features = 200000;
 int cluster_distance = 0;
+int tiny_polygon_size = 2;
 long justx = -1, justy = -1;
 std::string attribute_for_id = "";
 
+std::vector<order_field> order_by;
+bool order_reverse;
+
 int prevent[256];
 int additional[256];
 
@@ -2250,6 +2255,10 @@ int read_input(std::vector<source> &sources, char *fname, int maxzoom, int minzo
 			}
 		}
 
+		if (obasezoom < 0 && basezoom > maxzoom && prevent[P_BASEZOOM_ABOVE_MAXZOOM]) {
+			basezoom = maxzoom;
+		}
+
 		if (obasezoom < 0 && basezoom > maxzoom) {
 			fprintf(stderr, "Couldn't find a suitable base zoom. Working from the other direction.\n");
 			if (gamma == 0) {
@@ -2653,6 +2662,7 @@ int main(int argc, char **argv) {
 		{"full-detail", required_argument, 0, 'd'},
 		{"low-detail", required_argument, 0, 'D'},
 		{"minimum-detail", required_argument, 0, 'm'},
+		{"extra-detail", required_argument, 0, '~'},
 
 		{"Filtering feature attributes", 0, 0, 0},
 		{"exclude", required_argument, 0, 'x'},
@@ -2666,6 +2676,7 @@ int main(int argc, char **argv) {
 		{"empty-csv-columns-are-null", no_argument, &prevent[P_EMPTY_CSV_COLUMNS], 1},
 		{"convert-stringified-ids-to-numbers", no_argument, &additional[A_CONVERT_NUMERIC_IDS], 1},
 		{"use-attribute-for-id", required_argument, 0, '~'},
+		{"single-precision", no_argument, &prevent[P_SINGLE_PRECISION], 1},
 
 		{"Filtering features by attributes", 0, 0, 0},
 		{"feature-filter-file", required_argument, 0, 'J'},
@@ -2674,6 +2685,7 @@ int main(int argc, char **argv) {
 		{"Dropping a fixed fraction of features by zoom level", 0, 0, 0},
 		{"drop-rate", required_argument, 0, 'r'},
 		{"base-zoom", required_argument, 0, 'B'},
+		{"limit-base-zoom-to-maximum-zoom", no_argument, &prevent[P_BASEZOOM_ABOVE_MAXZOOM], 1},
 		{"drop-lines", no_argument, &additional[A_LINE_DROP], 1},
 		{"drop-polygons", no_argument, &additional[A_POLYGON_DROP], 1},
 		{"cluster-distance", required_argument, 0, 'K'},
@@ -2697,6 +2709,7 @@ int main(int argc, char **argv) {
 		{"no-line-simplification", no_argument, &prevent[P_SIMPLIFY], 1},
 		{"simplify-only-low-zooms", no_argument, &prevent[P_SIMPLIFY_LOW], 1},
 		{"no-tiny-polygon-reduction", no_argument, &prevent[P_TINY_POLYGON_REDUCTION], 1},
+		{"tiny-polygon-size", required_argument, 0, '~'},
 		{"no-simplification-of-shared-nodes", no_argument, &prevent[P_SIMPLIFY_SHARED_NODES], 1},
 
 		{"Attempts to improve shared polygon boundaries", 0, 0, 0},
@@ -2714,6 +2727,8 @@ int main(int argc, char **argv) {
 		{"coalesce", no_argument, &additional[A_COALESCE], 1},
 		{"reverse", no_argument, &additional[A_REVERSE], 1},
 		{"hilbert", no_argument, &additional[A_HILBERT], 1},
+		{"order-by", required_argument, 0, '~'},
+		{"order-descending-by", required_argument, 0, '~'},
 
 		{"Adding calculated attributes", 0, 0, 0},
 		{"calculate-feature-density", no_argument, &additional[A_CALCULATE_FEATURE_DENSITY], 1},
@@ -2834,6 +2849,21 @@ int main(int argc, char **argv) {
 					fprintf(stderr, "%s: %s: minimum maxzoom can be at most %d\n", argv[0], optarg, MAX_ZOOM);
 					exit(EXIT_FAILURE);
 				}
+			} else if (strcmp(opt, "tiny-polygon-size") == 0) {
+				tiny_polygon_size = atoi(optarg);
+			} else if (strcmp(opt, "extra-detail") == 0) {
+				extra_detail = atoi_require(optarg, "Extra detail");
+				if (extra_detail > 30) {
+					// So the maximum geometry delta of just under 2 tile extents
+					// is less than 2^31
+
+					fprintf(stderr, "%s: --extra-detail can be at most 30\n", argv[0]);
+					exit(EXIT_FAILURE);
+				}
+			} else if (strcmp(opt, "order-by") == 0) {
+				order_by.push_back(order_field(optarg, false));
+			} else if (strcmp(opt, "order-descending-by") == 0) {
+				order_by.push_back(order_field(optarg, true));
 			} else {
 				fprintf(stderr, "%s: Unrecognized option --%s\n", argv[0], opt);
 				exit(EXIT_FAILURE);
@@ -3265,12 +3295,16 @@ int main(int argc, char **argv) {
 		}
 	}
 
-	geometry_scale = 32 - (full_detail + maxzoom);
-	if (geometry_scale < 0) {
+	if (extra_detail >= 0) {
 		geometry_scale = 0;
-		if (!guess_maxzoom) {
-			// This shouldn't be able to happen any more. Can it still?
-			fprintf(stderr, "Full detail + maxzoom > 32, so you are asking for more detail than is available.\n");
+	} else {
+		geometry_scale = 32 - (full_detail + maxzoom);
+		if (geometry_scale < 0) {
+			geometry_scale = 0;
+			if (!guess_maxzoom) {
+				// This shouldn't be able to happen any more. Can it still?
+				fprintf(stderr, "Full detail + maxzoom > 32, so you are asking for more detail than is available.\n");
+			}
 		}
 	}
 

main.hpp

@@ -43,6 +43,7 @@ extern int quiet_progress;
 extern json_logger logger;
 extern double progress_interval;
 extern std::atomic<double> last_progress;
+extern int extra_detail;
 
 extern size_t CPUS;
 extern size_t TEMP_FILES;
@@ -51,6 +52,19 @@ extern size_t max_tile_size;
 extern size_t max_tile_features;
 extern int cluster_distance;
 extern std::string attribute_for_id;
+extern int tiny_polygon_size;
+
+struct order_field {
+	std::string name;
+	bool descending;
+
+	order_field(std::string _name, bool _descending)
+	    : name(_name),
+	      descending(_descending) {
+	}
+};
+
+extern std::vector<order_field> order_by;
 
 int mkstemp_cloexec(char *name);
 FILE *fopen_oflag(const char *name, const char *mode, int oflag);