Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce bounding box column definition #191

Merged
merged 15 commits into from
Mar 11, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified examples/example.parquet
Binary file not shown.
8 changes: 8 additions & 0 deletions examples/example_metadata.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,14 @@
180.0,
83.6451
],
"covering": {
"bbox": {
"xmax": "bbox.xmax",
"xmin": "bbox.xmin",
"ymax": "bbox.ymax",
"ymin": "bbox.ymin"
}
},
"crs": {
"$schema": "https://proj.org/schemas/v0.6/projjson.schema.json",
"area": "World.",
Expand Down
32 changes: 32 additions & 0 deletions format-specs/geoparquet.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,8 @@ Each geometry column in the dataset MUST be included in the `columns` field abov
| edges | string | Name of the coordinate system for the edges. Must be one of `"planar"` or `"spherical"`. The default value is `"planar"`. |
| bbox | \[number] | Bounding Box of the geometries in the file, formatted according to [RFC 7946, section 5](https://tools.ietf.org/html/rfc7946#section-5). |
| epoch | number | Coordinate epoch in case of a dynamic CRS, expressed as a decimal year. |
| covering | object | Object containing information like bounding boxes to help accelerate spatial data retrieval |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we phrase this more clearly that this "information" is pointing to columns in the file, and doesn't itself has the covering data? Because as it is written now, it sounds like that this object stores a bounding box itself.

(don't directly have a concrete suggestion, though, sorry! It's difficult to put this into words succinctly)

Copy link
Collaborator Author

@jwass jwass Jan 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call. I rewrote as "Object containing bounding box column names to help accelerate spatial data retrieval". Thoughts?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we expand to other types beyond just the bbox then we can rephrase to something more generic.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good!



#### crs

Expand Down Expand Up @@ -134,6 +136,36 @@ For non-geographic coordinate reference systems, the items in the bbox are minim

The bbox values are in the same coordinate reference system as the geometry.

#### covering

The covering field specifies optional simplified representations of each geometry. The keys of the "covering" object MUST be a supported encoding. Currently the only supported encoding is "bbox" which specifies the names of [bounding box columns](#bounding-box-columns)

Example:
```
"covering": {
"bbox": {
"xmin": "bbox.xmin",
"ymin": "bbox.ymin",
"xmax": "bbox.xmax",
"ymax": "bbox.ymax"
}
}
```

##### bbox covering encoding

Including a per-row bounding box can be useful for accelerating spatial queries by allowing consumers to inspect row group bounding box summary statistics. Furthermore a bounding box may be used to avoid complex spatial operations by first checking for bounding box overlaps. This field captures the name of a column containing the bounding box of the geometry for every row.

The format of `bbox` encoding is `{"xmin": "column_name.xmin", "ymin": "column_name.ymin", "xmax": "column_name.xmax", "ymax": "column_name.ymax"}` where `column_name` MUST exist in the Parquet file and meet the criteria in the [Bounding Box Column](#bounding-box-columns) definition. The values MUST end in `.xmin`, `.ymin`, etc. All values MUST use the same column name.

Note: the value specified in this field should not be confused with the top-level [`bbox`](#bbox) field which contains the single bounding box of this geometry over the whole GeoParquet file.

### Bounding Box Columns

A bounding box column MUST be a Parquet group field with 4 child fields named `xmin`, `xmax`, `ymin`, and `ymax`. For three dimensions the additional fields `zmin` and `zmax` MAY be present but are not required. The fields MUST be of Parquet type `FLOAT` or `DOUBLE`. The repetition of a bounding box column MUST match the geometry column's [repetition](#repetition). A row MUST contain a bounding box value if and only if the row contains a geometry value. In cases where the geometry is optional and a row not contain a geometry value, the row MUST NOT contain a bounding box value.
jwass marked this conversation as resolved.
Show resolved Hide resolved

The bounding box column MUST be at the root of the schema. The bounding box column MUST NOT be nested in a group.

### Additional information

#### Feature identifiers
Expand Down
28 changes: 28 additions & 0 deletions format-specs/schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,34 @@
},
"epoch": {
"type": "number"
},
"covering": {
"type": "object",
"minProperties": 1,
"properties": {
"bbox": {
"type": "object",
"required": ["xmin", "xmax", "ymin", "ymax"],
"properties": {
"xmin": {
"type": "string",
"pattern": "\\.xmin$"
},
"xmax": {
"type": "string",
"pattern": "\\.xmax$"
},
"ymin": {
"type": "string",
"pattern": "\\.ymin$"
},
"ymax": {
"type": "string",
"pattern": "\\.ymax$"
}
}
}
}
}
}
}
Expand Down
22 changes: 18 additions & 4 deletions scripts/generate_example.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
>>> import json, pprint, pyarrow.parquet as pq
>>> pprint.pprint(json.loads(pq.read_schema("example.parquet").metadata[b"geo"]))
"""
from collections import OrderedDict
import json
import pathlib

Expand All @@ -19,6 +20,14 @@

df = geopandas.read_file(geopandas.datasets.get_path("naturalearth_lowres"))
df = df.to_crs("ogc:84")

geometry_bbox = df.bounds.rename(
OrderedDict(
[("minx", "xmin"), ("miny", "ymin"), ("maxx", "xmax"), ("maxy", "ymax")]
),
axis=1,
)
df["bbox"] = geometry_bbox.to_dict("records")
table = pa.Table.from_pandas(df.head().to_wkb())


Expand All @@ -39,14 +48,19 @@ def get_version() -> str:
"crs": json.loads(df.crs.to_json()),
"edges": "planar",
"bbox": [round(x, 4) for x in df.total_bounds],
"covering": {
"bbox": {
"xmin": "bbox.xmin",
"ymin": "bbox.ymin",
"xmax": "bbox.xmax",
"ymax": "bbox.ymax",
},
},
},
},
}

schema = (
table.schema
.with_metadata({"geo": json.dumps(metadata)})
)
schema = table.schema.with_metadata({"geo": json.dumps(metadata)})
table = table.cast(schema)

pq.write_table(table, HERE / "../examples/example.parquet")
61 changes: 61 additions & 0 deletions scripts/test_json_schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,14 @@ def get_version() -> str:
"geometry": {
"encoding": "WKB",
"geometry_types": [],
"covering": {
"bbox": {
"xmin": "bbox.xmin",
"ymin": "bbox.ymin",
"xmax": "bbox.xmax",
"ymax": "bbox.ymax",
},
},
jwass marked this conversation as resolved.
Show resolved Hide resolved
},
},
}
Expand Down Expand Up @@ -210,6 +218,59 @@ def get_version() -> str:
metadata["columns"]["geometry"]["epoch"] = "2015.1"
invalid_cases["epoch_string"] = metadata

# Geometry Bbox

# Allow "any_column.xmin" etc.
metadata = copy.deepcopy(metadata_template)
metadata["columns"]["geometry"]["covering"]["bbox"] = {
"xmin": "any_column.xmin",
"ymin": "any_column.ymin",
"xmax": "any_column.xmax",
"ymax": "any_column.ymax",
}
valid_cases["valid_but_not_bbox_struct_name"] = metadata

metadata = copy.deepcopy(metadata_template)
metadata["columns"]["geometry"]["covering"].pop("bbox")
invalid_cases["empty_geometry_bbox"] = metadata


metadata = copy.deepcopy(metadata_template)
metadata["columns"]["geometry"]["covering"]["bbox"] = {}
invalid_cases["empty_geometry_bbox_missing_fields"] = metadata

metadata = copy.deepcopy(metadata_template)
metadata["columns"]["geometry"]["covering"]["bbox"].pop("xmin")
invalid_cases["covering_bbox_missing_xmin"] = metadata

metadata = copy.deepcopy(metadata_template)
metadata["columns"]["geometry"]["covering"]["bbox"].pop("ymin")
invalid_cases["covering_bbox_missing_ymin"] = metadata

metadata = copy.deepcopy(metadata_template)
metadata["columns"]["geometry"]["covering"]["bbox"].pop("xmax")
invalid_cases["covering_bbox_missing_xmax"] = metadata

metadata = copy.deepcopy(metadata_template)
metadata["columns"]["geometry"]["covering"]["bbox"].pop("ymax")
invalid_cases["covering_bbox_missing_ymax"] = metadata

## Invalid bbox xmin/xmax/ymin/ymax values
metadata = copy.deepcopy(metadata_template)
metadata["columns"]["geometry"]["covering"]["bbox"]["xmin"] = "not_bbox_dot_xmin"
invalid_cases["covering_bbox_invalid_xmin"] = metadata

metadata = copy.deepcopy(metadata_template)
metadata["columns"]["geometry"]["covering"]["bbox"]["xmax"] = "not_bbox_dot_xmax"
invalid_cases["covering_bbox_invalid_xmax"] = metadata

metadata = copy.deepcopy(metadata_template)
metadata["columns"]["geometry"]["covering"]["bbox"]["ymin"] = "not_bbox_dot_ymin"
invalid_cases["covering_bbox_invalid_ymin"] = metadata

metadata = copy.deepcopy(metadata_template)
metadata["columns"]["geometry"]["covering"]["bbox"]["ymax"] = "not_bbox_dot_ymax"
invalid_cases["covering_bbox_invalid_ymax"] = metadata

# # Tests

Expand Down
Loading