Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CORE] Remove TPC-H test dataset from module backends-velox, use the one from gluten-core uniformly #8004

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

[CORE] Remove TPC-H test dataset from module backends-velox, use the one from gluten-core uniformly #8004

wants to merge 3 commits into from

Conversation

zhztheplayer
Copy link
Member

@zhztheplayer zhztheplayer commented Nov 20, 2024 •
edited
Loading

Use gluten-core/test/resources/tpch-data-sf100 uniformly in tests where tpch dataset is needed.

@github-actions GitHub Actions
Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI on x86

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI on x86

@zhztheplayer
Copy link
Member Author

zhztheplayer commented Nov 20, 2024 •
edited
Loading

@zzcclp This is the attempt to use Parquet data in gluten-core from backend-velox's test code. But it seems that the resource files in core jar cannot be accessed from other modules.

I am thinking to introduce an env var like GLUTEN_HOME to pass to test suites to avoid specifying relevant paths in other modules. otherwise the code will rely on relative path between modules which is fragile and tricky to maintain.

zzcclp
zzcclp reviewed Nov 21, 2024
View reviewed changes
...nds-clickhouse/src/test/scala/org/apache/gluten/execution/GlutenClickHouseDecimalSuite.scala
@@ -38,7 +38,7 @@ class GlutenClickHouseDecimalSuite

override protected val needCopyParquetToTablePath = true

override protected val tablesPath: String = basePath + "/tpch-data"
override protected val tablesPath: String = basePath + "/tpch-data-sf100"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the sf of this tpch data is 1G, no 100G

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes this was my mistake.

It's likely SF 0.1 but I am not sure, will confirm before merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zzcclp zzcclp zzcclp left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
CLICKHOUSE CORE works for Gluten Core DATA_LAKE DOCS RSS VELOX
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@zhztheplayer @zzcclp