add generated_data to repo to facilitate CI #59
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…unning the tests in CI (since generating the data requires to equip the test runners with pyspark, which at least mother prefers not to) - the data is minimized to 5MB, by setting the scale factor 10x smaller, and from expected_results/ only keeping last/ this motivates rewriting some testcases that used expected_results/q08 into expected_results/last (which is the same) - removed one testcase, which tests if we get an error when parquet is not loaded (in MotherDuck, when we run iceberg in so-called "remote_only" mode, this will actually not raise an error) it was kind of a trivial test, so I hope this is OK..
samansmink
reviewed
Jun 28, 2024
Makefile
Outdated
| @@ -9,11 +9,11 @@ include extension-ci-tools/makefiles/duckdb_extension.Makefile | |||
|
|
|||
| # Custom makefile targets | |||
| data: data_clean | |||
| python3 scripts/test_data_generator/generate_iceberg.py 0.01 data/iceberg/generated_spec1_0_01 1 | |||
| python3 scripts/test_data_generator/generate_iceberg.py 0.01 data/iceberg/generated_spec2_0_01 2 | |||
| python3 scripts/test_data_generator/generate_iceberg.py 0.001 data/iceberg/generated_spec1_0_01 1 | |||
There was a problem hiding this comment.
nit: the name of the directory is now no longer correct which could be a little confusing
|
windows CI is failing due to some upstream issue, however the tests also seem to fail due to only the json metadata files being added and not the avro and parquet files |
- also in Makefile and in the tests (name, and paths used)
apologies for that messup. I have made the name change and made sure all files are pushed now. |
|
Thanks for the changes! looks good! |
mike-luabase
pushed a commit
to definite-app/duckdb_iceberg
that referenced
this pull request
Oct 27, 2024
add generated_data to repo to facilitate CI
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Commit the generated data into the repo, because this facilitates running the tests in CI
(since generating the data requires to equip the test runners with pyspark, which at least motherduck prefers not to).
The data is minimized to 5MB, by setting the scale factor 10x smaller, and from expected_results/ only keeping last/.
This motivates rewriting some testcases that used expected_results/q08 into expected_results/last (which is the same)
Removed one testcase, which tests if we get an error when parquet is not loaded (in MotherDuck, when we run iceberg in so-called "remote_only" mode, this will actually not raise an error).
It was kind of a trivial test, so I hope this is OK..