Add validation command #29

JayKickliter · 2023-04-03T21:42:12Z

This PR adds a command to check batch h3idx files for overlapping cells. It dedups works, so for M input files, it does (M*M - M)/2 file comparisons.

Example output:

$ time ./target/release/lw-generator index overlaps local/*.h3idz
2023-04-04T22:23:13.668Z DEBUG [lw_generator::index] Comparing 'local/AS923-1.res7.h3idz' against 'local/AS923-1B.res7.h3idz'
2023-04-04T22:23:14.312Z DEBUG [lw_generator::index] Comparing 'local/AS923-1C.res7.h3idz' against 'local/AS923-1.res7.h3idz'
2023-04-04T22:23:17.853Z DEBUG [lw_generator::index] Comparing 'local/AS923-2.res7.h3idz' against 'local/AS923-1.res7.h3idz'
2023-04-04T22:23:20.366Z DEBUG [lw_generator::index] Comparing 'local/AS923-3.res7.h3idz' against 'local/AS923-1.res7.h3idz'
2023-04-04T22:23:22.650Z DEBUG [lw_generator::index] Comparing 'local/AS923-1.res7.h3idz' against 'local/AS923-4.res7.h3idz'
2023-04-04T22:23:22.777Z DEBUG [lw_generator::index] Comparing 'local/AU915-SB1.res7.h3idz' against 'local/AS923-1.res7.h3idz'
2023-04-04T22:23:25.008Z DEBUG [lw_generator::index] Comparing 'local/AU915.res7.h3idz' against 'local/AS923-1.res7.h3idz'
2023-04-04T22:23:30.107Z DEBUG [lw_generator::index] Comparing 'local/AS923-1.res7.h3idz' against 'local/CD900-1A.res7.h3idz'
2023-04-04T22:23:30.139Z DEBUG [lw_generator::index] Comparing 'local/CN470.res7.h3idz' against 'local/AS923-1.res7.h3idz'
2023-04-04T22:23:33.021Z DEBUG [lw_generator::index] Comparing 'local/EU433.res7.h3idz' against 'local/AS923-1.res7.h3idz'
2023-04-04T22:23:36.905Z DEBUG [lw_generator::index] Comparing 'local/EU868.res7.h3idz' against 'local/AS923-1.res7.h3idz'
2023-04-04T22:23:54.044Z DEBUG [lw_generator::index] Comparing 'local/IN865.res7.h3idz' against 'local/AS923-1.res7.h3idz'
2023-04-04T22:23:58.169Z DEBUG [lw_generator::index] Comparing 'local/AS923-1.res7.h3idz' against 'local/KR920.res7.h3idz'
2023-04-04T22:23:58.462Z DEBUG [lw_generator::index] Comparing 'local/RU864.res7.h3idz' against 'local/AS923-1.res7.h3idz'
2023-04-04T22:24:04.126Z DEBUG [lw_generator::index] Comparing 'local/AS923-1C.res7.h3idz' against 'local/AS923-1B.res7.h3idz'
2023-04-04T22:24:04.443Z DEBUG [lw_generator::index] Comparing 'local/AS923-2.res7.h3idz' against 'local/AS923-1B.res7.h3idz'
2023-04-04T22:24:04.642Z DEBUG [lw_generator::index] Comparing 'local/AS923-3.res7.h3idz' against 'local/AS923-1B.res7.h3idz'
2023-04-04T22:24:04.807Z DEBUG [lw_generator::index] Comparing 'local/AS923-1B.res7.h3idz' against 'local/AS923-4.res7.h3idz'
2023-04-04T22:24:04.818Z DEBUG [lw_generator::index] Comparing 'local/AU915-SB1.res7.h3idz' against 'local/AS923-1B.res7.h3idz'
2023-04-04T22:24:05.010Z DEBUG [lw_generator::index] Comparing 'local/AU915.res7.h3idz' against 'local/AS923-1B.res7.h3idz'
2023-04-04T22:24:05.435Z DEBUG [lw_generator::index] Comparing 'local/AS923-1B.res7.h3idz' against 'local/CD900-1A.res7.h3idz'
2023-04-04T22:24:05.438Z DEBUG [lw_generator::index] Comparing 'local/CN470.res7.h3idz' against 'local/AS923-1B.res7.h3idz'
2023-04-04T22:24:05.711Z DEBUG [lw_generator::index] Comparing 'local/EU433.res7.h3idz' against 'local/AS923-1B.res7.h3idz'
2023-04-04T22:24:06.037Z DEBUG [lw_generator::index] Comparing 'local/EU868.res7.h3idz' against 'local/AS923-1B.res7.h3idz'
2023-04-04T22:24:07.477Z DEBUG [lw_generator::index] Comparing 'local/IN865.res7.h3idz' against 'local/AS923-1B.res7.h3idz'
2023-04-04T22:24:07.820Z DEBUG [lw_generator::index] Comparing 'local/KR920.res7.h3idz' against 'local/AS923-1B.res7.h3idz'
2023-04-04T22:24:07.845Z DEBUG [lw_generator::index] Comparing 'local/RU864.res7.h3idz' against 'local/AS923-1B.res7.h3idz'
2023-04-04T22:24:08.370Z DEBUG [lw_generator::index] Comparing 'local/AS923-2.res7.h3idz' against 'local/AS923-1C.res7.h3idz'
2023-04-04T22:24:09.519Z DEBUG [lw_generator::index] Comparing 'local/AS923-1C.res7.h3idz' against 'local/AS923-3.res7.h3idz'
2023-04-04T22:24:10.418Z DEBUG [lw_generator::index] Comparing 'local/AS923-1C.res7.h3idz' against 'local/AS923-4.res7.h3idz'
2023-04-04T22:24:10.473Z DEBUG [lw_generator::index] Comparing 'local/AU915-SB1.res7.h3idz' against 'local/AS923-1C.res7.h3idz'
2023-04-04T22:24:11.496Z DEBUG [lw_generator::index] Comparing 'local/AS923-1C.res7.h3idz' against 'local/AU915.res7.h3idz'
2023-04-04T22:24:13.951Z DEBUG [lw_generator::index] Comparing 'local/AS923-1C.res7.h3idz' against 'local/CD900-1A.res7.h3idz'
2023-04-04T22:24:13.965Z DEBUG [lw_generator::index] Comparing 'local/CN470.res7.h3idz' against 'local/AS923-1C.res7.h3idz'
2023-04-04T22:24:15.385Z DEBUG [lw_generator::index] Comparing 'local/AS923-1C.res7.h3idz' against 'local/EU433.res7.h3idz'
2023-04-04T22:24:17.246Z DEBUG [lw_generator::index] Comparing 'local/AS923-1C.res7.h3idz' against 'local/EU868.res7.h3idz'
2023-04-04T22:24:26.234Z DEBUG [lw_generator::index] Comparing 'local/AS923-1C.res7.h3idz' against 'local/IN865.res7.h3idz'
2023-04-04T22:24:28.322Z DEBUG [lw_generator::index] Comparing 'local/AS923-1C.res7.h3idz' against 'local/KR920.res7.h3idz'
2023-04-04T22:24:28.461Z DEBUG [lw_generator::index] Comparing 'local/RU864.res7.h3idz' against 'local/AS923-1C.res7.h3idz'
2023-04-04T22:24:31.595Z DEBUG [lw_generator::index] Comparing 'local/AS923-2.res7.h3idz' against 'local/AS923-3.res7.h3idz'
2023-04-04T22:24:32.182Z DEBUG [lw_generator::index] Comparing 'local/AS923-2.res7.h3idz' against 'local/AS923-4.res7.h3idz'
2023-04-04T22:24:32.218Z DEBUG [lw_generator::index] Comparing 'local/AU915-SB1.res7.h3idz' against 'local/AS923-2.res7.h3idz'
2023-04-04T22:24:32.891Z DEBUG [lw_generator::index] Comparing 'local/AS923-2.res7.h3idz' against 'local/AU915.res7.h3idz'
2023-04-04T22:24:34.471Z DEBUG [lw_generator::index] Comparing 'local/AS923-2.res7.h3idz' against 'local/CD900-1A.res7.h3idz'
2023-04-04T22:24:34.480Z DEBUG [lw_generator::index] Comparing 'local/CN470.res7.h3idz' against 'local/AS923-2.res7.h3idz'
2023-04-04T22:24:35.352Z DEBUG [lw_generator::index] Comparing 'local/AS923-2.res7.h3idz' against 'local/EU433.res7.h3idz'
2023-04-04T22:24:36.547Z DEBUG [lw_generator::index] Comparing 'local/AS923-2.res7.h3idz' against 'local/EU868.res7.h3idz'
2023-04-04T22:24:41.712Z DEBUG [lw_generator::index] Comparing 'local/AS923-2.res7.h3idz' against 'local/IN865.res7.h3idz'
2023-04-04T22:24:42.949Z DEBUG [lw_generator::index] Comparing 'local/AS923-2.res7.h3idz' against 'local/KR920.res7.h3idz'
2023-04-04T22:24:43.041Z DEBUG [lw_generator::index] Comparing 'local/RU864.res7.h3idz' against 'local/AS923-2.res7.h3idz'
2023-04-04T22:24:44.782Z DEBUG [lw_generator::index] Comparing 'local/AS923-3.res7.h3idz' against 'local/AS923-4.res7.h3idz'
2023-04-04T22:24:44.813Z DEBUG [lw_generator::index] Comparing 'local/AU915-SB1.res7.h3idz' against 'local/AS923-3.res7.h3idz'
2023-04-04T22:24:45.333Z DEBUG [lw_generator::index] Comparing 'local/AU915.res7.h3idz' against 'local/AS923-3.res7.h3idz'
2023-04-04T22:24:46.562Z DEBUG [lw_generator::index] Comparing 'local/AS923-3.res7.h3idz' against 'local/CD900-1A.res7.h3idz'
2023-04-04T22:24:46.569Z DEBUG [lw_generator::index] Comparing 'local/CN470.res7.h3idz' against 'local/AS923-3.res7.h3idz'
2023-04-04T22:24:47.273Z DEBUG [lw_generator::index] Comparing 'local/AS923-3.res7.h3idz' against 'local/EU433.res7.h3idz'
2023-04-04T22:24:48.240Z DEBUG [lw_generator::index] Comparing 'local/EU868.res7.h3idz' against 'local/AS923-3.res7.h3idz'
2023-04-04T22:24:52.496Z DEBUG [lw_generator::index] Comparing 'local/IN865.res7.h3idz' against 'local/AS923-3.res7.h3idz'
2023-04-04T22:24:53.633Z DEBUG [lw_generator::index] Comparing 'local/AS923-3.res7.h3idz' against 'local/KR920.res7.h3idz'
2023-04-04T22:24:53.724Z DEBUG [lw_generator::index] Comparing 'local/RU864.res7.h3idz' against 'local/AS923-3.res7.h3idz'
2023-04-04T22:24:55.394Z DEBUG [lw_generator::index] Comparing 'local/AU915-SB1.res7.h3idz' against 'local/AS923-4.res7.h3idz'
2023-04-04T22:24:55.442Z DEBUG [lw_generator::index] Comparing 'local/AU915.res7.h3idz' against 'local/AS923-4.res7.h3idz'
2023-04-04T22:24:55.530Z DEBUG [lw_generator::index] Comparing 'local/AS923-4.res7.h3idz' against 'local/CD900-1A.res7.h3idz'
2023-04-04T22:24:55.531Z DEBUG [lw_generator::index] Comparing 'local/CN470.res7.h3idz' against 'local/AS923-4.res7.h3idz'
2023-04-04T22:24:55.591Z DEBUG [lw_generator::index] Comparing 'local/EU433.res7.h3idz' against 'local/AS923-4.res7.h3idz'
2023-04-04T22:24:55.670Z DEBUG [lw_generator::index] Comparing 'local/EU868.res7.h3idz' against 'local/AS923-4.res7.h3idz'
2023-04-04T22:24:55.996Z DEBUG [lw_generator::index] Comparing 'local/IN865.res7.h3idz' against 'local/AS923-4.res7.h3idz'
2023-04-04T22:24:56.072Z DEBUG [lw_generator::index] Comparing 'local/KR920.res7.h3idz' against 'local/AS923-4.res7.h3idz'
2023-04-04T22:24:56.079Z DEBUG [lw_generator::index] Comparing 'local/RU864.res7.h3idz' against 'local/AS923-4.res7.h3idz'
2023-04-04T22:24:56.197Z DEBUG [lw_generator::index] Comparing 'local/AU915-SB1.res7.h3idz' against 'local/AU915.res7.h3idz'
2023-04-04T22:24:57.794Z DEBUG [lw_generator::index] Comparing 'local/AU915-SB1.res7.h3idz' against 'local/CD900-1A.res7.h3idz'
2023-04-04T22:24:57.802Z DEBUG [lw_generator::index] Comparing 'local/CN470.res7.h3idz' against 'local/AU915-SB1.res7.h3idz'
2023-04-04T22:24:58.798Z DEBUG [lw_generator::index] Comparing 'local/AU915-SB1.res7.h3idz' against 'local/EU433.res7.h3idz'
2023-04-04T22:25:00.008Z DEBUG [lw_generator::index] Comparing 'local/AU915-SB1.res7.h3idz' against 'local/EU868.res7.h3idz'
2023-04-04T22:25:05.019Z DEBUG [lw_generator::index] Comparing 'local/AU915-SB1.res7.h3idz' against 'local/IN865.res7.h3idz'
2023-04-04T22:25:06.202Z DEBUG [lw_generator::index] Comparing 'local/AU915-SB1.res7.h3idz' against 'local/KR920.res7.h3idz'
2023-04-04T22:25:06.283Z DEBUG [lw_generator::index] Comparing 'local/RU864.res7.h3idz' against 'local/AU915-SB1.res7.h3idz'
2023-04-04T22:25:07.864Z DEBUG [lw_generator::index] Comparing 'local/AU915.res7.h3idz' against 'local/CD900-1A.res7.h3idz'
2023-04-04T22:25:07.884Z DEBUG [lw_generator::index] Comparing 'local/CN470.res7.h3idz' against 'local/AU915.res7.h3idz'
2023-04-04T22:25:09.753Z DEBUG [lw_generator::index] Comparing 'local/AU915.res7.h3idz' against 'local/EU433.res7.h3idz'
2023-04-04T22:25:12.269Z DEBUG [lw_generator::index] Comparing 'local/AU915.res7.h3idz' against 'local/EU868.res7.h3idz'
2023-04-04T22:25:23.418Z DEBUG [lw_generator::index] Comparing 'local/AU915.res7.h3idz' against 'local/IN865.res7.h3idz'
2023-04-04T22:25:26.039Z DEBUG [lw_generator::index] Comparing 'local/AU915.res7.h3idz' against 'local/KR920.res7.h3idz'
2023-04-04T22:25:26.231Z DEBUG [lw_generator::index] Comparing 'local/RU864.res7.h3idz' against 'local/AU915.res7.h3idz'
2023-04-04T22:25:29.887Z DEBUG [lw_generator::index] Comparing 'local/CN470.res7.h3idz' against 'local/CD900-1A.res7.h3idz'
2023-04-04T22:25:29.899Z DEBUG [lw_generator::index] Comparing 'local/EU433.res7.h3idz' against 'local/CD900-1A.res7.h3idz'
2023-04-04T22:25:29.916Z DEBUG [lw_generator::index] Comparing 'local/EU868.res7.h3idz' against 'local/CD900-1A.res7.h3idz'
2023-04-04T22:25:29.984Z DEBUG [lw_generator::index] Comparing 'local/IN865.res7.h3idz' against 'local/CD900-1A.res7.h3idz'
2023-04-04T22:25:30.000Z DEBUG [lw_generator::index] Comparing 'local/KR920.res7.h3idz' against 'local/CD900-1A.res7.h3idz'
2023-04-04T22:25:30.002Z DEBUG [lw_generator::index] Comparing 'local/RU864.res7.h3idz' against 'local/CD900-1A.res7.h3idz'
2023-04-04T22:25:30.027Z DEBUG [lw_generator::index] Comparing 'local/CN470.res7.h3idz' against 'local/EU433.res7.h3idz'
2023-04-04T22:25:31.431Z DEBUG [lw_generator::index] Comparing 'local/CN470.res7.h3idz' against 'local/EU868.res7.h3idz'
2023-04-04T22:25:37.589Z DEBUG [lw_generator::index] Comparing 'local/CN470.res7.h3idz' against 'local/IN865.res7.h3idz'
2023-04-04T22:25:39.145Z DEBUG [lw_generator::index] Comparing 'local/CN470.res7.h3idz' against 'local/KR920.res7.h3idz'
2023-04-04T22:25:39.256Z DEBUG [lw_generator::index] Comparing 'local/RU864.res7.h3idz' against 'local/CN470.res7.h3idz'
2023-04-04T22:25:41.453Z DEBUG [lw_generator::index] Comparing 'local/EU868.res7.h3idz' against 'local/EU433.res7.h3idz'
2023-04-04T22:25:49.927Z DEBUG [lw_generator::index] Comparing 'local/IN865.res7.h3idz' against 'local/EU433.res7.h3idz'
2023-04-04T22:25:52.185Z DEBUG [lw_generator::index] Comparing 'local/EU433.res7.h3idz' against 'local/KR920.res7.h3idz'
2023-04-04T22:25:52.330Z DEBUG [lw_generator::index] Comparing 'local/RU864.res7.h3idz' against 'local/EU433.res7.h3idz'
2023-04-04T22:25:55.236Z DEBUG [lw_generator::index] Comparing 'local/EU868.res7.h3idz' against 'local/IN865.res7.h3idz'
2023-04-04T22:26:04.395Z DEBUG [lw_generator::index] Comparing 'local/EU868.res7.h3idz' against 'local/KR920.res7.h3idz'
2023-04-04T22:26:05.053Z DEBUG [lw_generator::index] Comparing 'local/RU864.res7.h3idz' against 'local/EU868.res7.h3idz'
2023-04-04T22:26:17.679Z DEBUG [lw_generator::index] Comparing 'local/IN865.res7.h3idz' against 'local/KR920.res7.h3idz'
2023-04-04T22:26:17.837Z DEBUG [lw_generator::index] Comparing 'local/RU864.res7.h3idz' against 'local/IN865.res7.h3idz'
2023-04-04T22:26:21.337Z DEBUG [lw_generator::index] Comparing 'local/RU864.res7.h3idz' against 'local/KR920.res7.h3idz'
{
  "local/AU915-SB1.res7.h3idz": {
    "local/AS923-1.res7.h3idz": [
      [
        610436957087989759,
        610436957087989759
      ]
    ]
  }
}

real	3m7.939s
user	22m59.275s
sys	0m2.877s

Remaining work

move validate to index subcommand
remove development prints
decide on how to report overlaps
either configure default log-level to not show DEBUG or separate log/report to stderr/stout
🚲🏠 names

'index overlaps <FILES>' returns a non-0 exit code if any index file has overlapping cells with any other.

JayKickliter force-pushed the jsk/check-overlaps branch 3 times, most recently from 5f46f09 to 0ba2133 Compare April 4, 2023 20:17

JayKickliter mentioned this pull request Apr 4, 2023

Add index check command #28

Merged

JayKickliter marked this pull request as ready for review April 4, 2023 22:28

JayKickliter requested a review from madninja April 4, 2023 22:28

JayKickliter and others added 8 commits April 5, 2023 21:05

Add 'index overlaps' subcommand

30a99d8

'index overlaps <FILES>' returns a non-0 exit code if any index file has overlapping cells with any other.

Add a proof-of-concept overlap report

1274caf

Document loop logic

81c9927

Generate json report

d5a102d

Use full path

f365792

Reduce cloning

bccc2b7

Simplify dedup logic

6ff678a

Add debug flag to show debug prints

c55eee6

madninja force-pushed the jsk/check-overlaps branch from 7df38df to c55eee6 Compare April 6, 2023 02:06

Use h3cell only for find

07efa29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add validation command #29

Add validation command #29

JayKickliter commented Apr 3, 2023 •

edited

Loading

Add validation command #29

Are you sure you want to change the base?

Add validation command #29

Conversation

JayKickliter commented Apr 3, 2023 • edited Loading

Remaining work

JayKickliter commented Apr 3, 2023 •

edited

Loading