Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](hive docker)Exit on creating table failed #47390

Merged
merged 9 commits into from
Jan 26, 2025
Merged

[opt](hive docker)Exit on creating table failed #47390

merged 9 commits into from
Jan 26, 2025
+24 −16

Conversation

Thearas
Copy link
Contributor

@Thearas Thearas commented Jan 24, 2025 •
edited
Loading

What problem does this PR solve?

Problem Summary:

There're some table not found error when querying external hive catalog in CI, like #614159 and #613945.
image

The create table script should exit once the sql failed, so we can easily to see which script failed.

Failed scripts

Total 7 ~ 10+ failures, you can search FAILED: in hive metastore docker logs.

  1. Create table when hive metastore is not ready , like student:

    2025-01-24T04:30:01.794131928Z FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

  2. HQL grammar wrong, like /mnt/scripts/data/regression/crdmm_data/run.sh :

    FAILED: ParseException line 3:12 extraneous input 'crdmm_data' expecting EOF near '<EOF>'

  3. Error at msck repair table partition_location_2;, need set hive.msck.path.validation=ignore;, like /mnt/scripts/data/multi_catalog/partition_location_2/run.sh:

    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
[opt](hive docker)Exit on creating table failed
61860fe 
Change-Id: Ic5547208db9f56257d0ab1c6981c2cb180e70ed2
zgxme
zgxme approved these changes Jan 24, 2025
View reviewed changes
Copy link
Contributor

@zgxme zgxme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zgxme
Copy link
Contributor

zgxme commented Jan 24, 2025

run buildall

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@Thearas
fix some load
f89b48b 
Change-Id: I212395f220cd855b53b4cc937a0d531ee25b84ee
@Thearas Thearas force-pushed the hive-create-table-exit branch from af0fb88 to f89b48b Compare January 24, 2025 07:28
@Thearas
export HMS_PORT into metastore container
ef2fe5d 
Change-Id: I6263a2762f0302399e5b1a01cd28de6a7a768f0f
@Thearas
Copy link
Contributor Author

Thearas commented Jan 24, 2025

run buildall

@Thearas
auto restart hive2 namenode and datanode
b486340 
Change-Id: Idbe6e43e9b3d060dd8be77aef92bf8ff426ee069
@Thearas
auto restart hive3 namenode and datanode
abaf119 
Change-Id: I15d26cd548305ca4f862ea31b9cedda584adce83
@Thearas Thearas force-pushed the hive-create-table-exit branch from 665d09a to 41348ff Compare January 24, 2025 13:53
@Thearas
lock file after start hive metastore
bc33a15 
Change-Id: Id84568dd98d30e3ea8284bd014c518563171cc84
@Thearas Thearas force-pushed the hive-create-table-exit branch from 41348ff to bc33a15 Compare January 24, 2025 15:00
@Thearas
fix partition_location_1/partition_location_2 msck repair table failed
9ca3acd 
Change-Id: Id5c9a35819d176f374c440358e3c1a66abfa263d
@Thearas
add set hive.msck.path.validation=ignore;
dc02de3 
Change-Id: I38489b0c5539033364845c1f87b5da03539f5f3a
@Thearas
Copy link
Contributor Author

Thearas commented Jan 25, 2025

run external

@Thearas
fix scripts
ba61150 
Change-Id: I66928ac1cec624d5af6084bf79dfec448123dbef
@Thearas
Copy link
Contributor Author

Thearas commented Jan 25, 2025

run buildall

@Thearas
Copy link
Contributor Author

Thearas commented Jan 26, 2025

run cloud_p0

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 26, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

zgxme
zgxme approved these changes Jan 26, 2025
View reviewed changes
Copy link
Contributor

@zgxme zgxme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 15acf46 into apache:master Jan 26, 2025
28 checks passed
github-actions bot pushed a commit that referenced this pull request Jan 26, 2025
@Thearas
[opt](hive docker)Exit on creating table failed (#47390)
bdba152 
### What problem does this PR solve?

Problem Summary:

There're some table not found error when querying external hive catalog
in CI, like
[#614159](http://43.132.222.7:8111/buildConfiguration/Doris_External_Regression/614159?buildTab=overview&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildTestsSection=true&expandBuildChangesSection=true&expandBuildDeploymentsSection=false)
and
[#613945](http://43.132.222.7:8111/buildConfiguration/Doris_External_Regression/613945?buildTab=tests&status=failed&expandedTest=build%3A%28id%3A613945%29%2Cid%3A2000000035).
<img width="782" alt="image"
src="https://github.com/user-attachments/assets/74ce8b6f-3116-4aca-a5a5-3d7f3845a1c9"
/>

The create table script should exit once the sql failed, so we can
easily to see which script failed.

#### Failed scripts

Total 7 ~ 10+ failures, you can search `FAILED: ` in [hive metastore
docker
logs](https://github.com/user-attachments/files/18532083/logs.txt).

1. Create table when hive metastore is not ready , like `student`:
    ```bash
2025-01-24T04:30:01.794131928Z FAILED: SemanticException
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    ```
2. HQL grammar wrong, like
`/mnt/scripts/data/regression/crdmm_data/run.sh` :
    ```bash
FAILED: ParseException line 3:12 extraneous input 'crdmm_data' expecting
EOF near '<EOF>'
    ```

3. Error at `msck repair table partition_location_2;`, need `set
hive.msck.path.validation=ignore;`, like
`/mnt/scripts/data/multi_catalog/partition_location_2/run.sh`:
    ```bash
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
    ```
@github-actions github-actions bot mentioned this pull request Jan 26, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Jan 26, 2025
@Thearas
[opt](hive docker)Exit on creating table failed (#47390)
9bd29b4 
### What problem does this PR solve?

Problem Summary:

There're some table not found error when querying external hive catalog
in CI, like
[#614159](http://43.132.222.7:8111/buildConfiguration/Doris_External_Regression/614159?buildTab=overview&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildTestsSection=true&expandBuildChangesSection=true&expandBuildDeploymentsSection=false)
and
[#613945](http://43.132.222.7:8111/buildConfiguration/Doris_External_Regression/613945?buildTab=tests&status=failed&expandedTest=build%3A%28id%3A613945%29%2Cid%3A2000000035).
<img width="782" alt="image"
src="https://github.com/user-attachments/assets/74ce8b6f-3116-4aca-a5a5-3d7f3845a1c9"
/>

The create table script should exit once the sql failed, so we can
easily to see which script failed.

#### Failed scripts

Total 7 ~ 10+ failures, you can search `FAILED: ` in [hive metastore
docker
logs](https://github.com/user-attachments/files/18532083/logs.txt).

1. Create table when hive metastore is not ready , like `student`:
    ```bash
2025-01-24T04:30:01.794131928Z FAILED: SemanticException
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    ```
2. HQL grammar wrong, like
`/mnt/scripts/data/regression/crdmm_data/run.sh` :
    ```bash
FAILED: ParseException line 3:12 extraneous input 'crdmm_data' expecting
EOF near '<EOF>'
    ```

3. Error at `msck repair table partition_location_2;`, need `set
hive.msck.path.validation=ignore;`, like
`/mnt/scripts/data/multi_catalog/partition_location_2/run.sh`:
    ```bash
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
    ```
@github-actions github-actions bot mentioned this pull request Jan 26, 2025
Merged
morningman pushed a commit that referenced this pull request Jan 26, 2025
@github-actions @Thearas
branch-3.0: [opt](hive docker)Exit on creating table failed #47390 (#…
1f8faaa 
…47452)

Cherry-picked from #47390

Co-authored-by: Thearas <[email protected]>
morningman pushed a commit that referenced this pull request Jan 26, 2025
@github-actions
branch-2.1: [opt](hive docker)Exit on creating table failed #47390 (#…
af55eba 
…47453)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zgxme zgxme zgxme approved these changes

@morningman morningman morningman approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.9-merged dev/3.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Thearas @zgxme @morningman