Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue 26: Prepare for 0.3.0 release #27

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Issue 26: Prepare for 0.3.0 release #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6b3be2e
bump version to 0.3.0
derekm Dec 19, 2022
07c0a08
upgrade parquet-tools to parquet-cli
derekm Dec 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,8 +118,8 @@ If you would rather use a more automated deployment method, skip to the next sec
3. Upload the artifact:
- group: io.pravega
- artifact: flink-tools
- version: 0.2.0
- file: flink-tools/build/libs/pravega-flink-tools-0.2.0.jar
- version: 0.3.0
- file: flink-tools/build/libs/pravega-flink-tools-0.3.0.jar

4. Create Flink Cluster.
- Name: stream-to-file
Expand All @@ -130,7 +130,7 @@ If you would rather use a more automated deployment method, skip to the next sec
5. Create New App.
- Name: stream-to-file
- Main Application File Type: maven
- Main Application File: io.pravega:fliink-tools:0.2.0
- Main Application File: io.pravega:flink-tools:0.3.0
- Main Class: io.pravega.flinktools.StreamToFileJob
- Cluster Selectors: name: stream-to-file
- Parallelism: 1
Expand Down Expand Up @@ -267,7 +267,7 @@ If the Parquet file is located on a standard Linux file system (including NFS),
you can use a command similar to the following to view the content.

```shell script
scripts/parquet-tools.sh cat /tmp/sample1-parquet/2020-08-19--03/part-0-887
scripts/parquet-cli.sh cat /tmp/sample1-parquet/2020-08-19--03/part-0-887
```

If the Parquet file is located on an HDFS cluster in Kubernetes,
Expand All @@ -276,8 +276,8 @@ you can use commands similar to the following to view the content.
```shell script
scripts/hadoop-bash.sh
root@hadoop-8c428aa0-76c0-4f42-8bea-2fc1e8300f78:~#
wget https://repo1.maven.org/maven2/org/apache/parquet/parquet-tools/1.11.1/parquet-tools-1.11.1.jar
hadoop jar parquet-tools-1.11.1.jar cat hdfs://hadoop-hadoop-hdfs-nn.examples.svc.cluster.local:9000/tmp/sample1-parquet/2020-08-19--03/part-0-887
wget https://repo1.maven.org/maven2/org/apache/parquet/parquet-cli/1.12.2/parquet-cli-1.12.2-runtime.jar
hadoop jar parquet-cli-1.12.2-runtime.jar org.apache.parquet.cli.Main cat hdfs://hadoop-hadoop-hdfs-nn.examples.svc.cluster.local:9000/tmp/sample1-parquet/2020-08-19--03/part-0-887
```

## Stream-to-CSV-File: Continuously copying a Pravega stream to CSV files
Expand Down Expand Up @@ -401,8 +401,8 @@ Below shows how to deploy this Flink job using the SDP UI.
2. Upload the artifact:
- group: io.pravega
- artifact: flink-tools
- version: 0.2.0
- file: flink-tools/build/libs/pravega-flink-tools-0.2.0.jar
- version: 0.3.0
- file: flink-tools/build/libs/pravega-flink-tools-0.3.0.jar

3. Create Flink Cluster.
- Name: sample-data-generator-job
Expand All @@ -416,7 +416,7 @@ Below shows how to deploy this Flink job using the SDP UI.
4. Create New App.
- Name: sample-data-generator-job
- Main Application File Type: maven
- Main Application File: io.pravega:fliink-tools:0.2.0
- Main Application File: io.pravega:flink-tools:0.3.0
- Main Class: io.pravega.flinktools.SampleDataGeneratorJob
- Cluster Selectors: name: sample-data-generator-job
- Parallelism: 1
Expand Down Expand Up @@ -517,8 +517,8 @@ Follow these steps to release a new version of Flink Tools to Github Releases.
git push --tags
````

3. Monitor Travis CI at
https://travis-ci.com/github/pravega/flink-tools.
3. Monitor GitHub Actions CI at
https://github.com/pravega/flink-tools/actions.

4. When complete, the installation archive will be available at
https://github.com/pravega/flink-tools/releases.
Expand Down
2 changes: 1 addition & 1 deletion scripts/env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ APP_NAME=flink-tools
APP_GROUP_ID=${APP_GROUP_ID:-io.pravega}
APP_ARTIFACT_ID=${APP_ARTIFACT_ID:-flink-tools}
# Change line below to bump the application version.
APP_VERSION=${APP_VERSION:-0.2.15}
APP_VERSION=${APP_VERSION:-0.3.0}
GRADLE_OPTIONS="${GRADLE_OPTIONS:-"-Pversion=${APP_VERSION}"}"
FLINK_IMAGE_TAG="1.15.2-2.12-1.3.1-20-f59a7da"
NEW_IMAGE_TAG="${FLINK_IMAGE_TAG}-hadoop2.8.3"
Expand Down
6 changes: 3 additions & 3 deletions scripts/parquet-tools.sh → scripts/parquet-cli.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,6 @@
#
# http://www.apache.org/licenses/LICENSE-2.0
set -ex
JAR_FILE=/tmp/parquet-tools-1.11.1.jar
[ -f ${JAR_FILE} ] || wget -O ${JAR_FILE} https://repo1.maven.org/maven2/org/apache/parquet/parquet-tools/1.11.1/parquet-tools-1.11.1.jar
docker run --rm -it -v /tmp:/tmp -v /desdp:/desdp danisla/hadoop:2.9.0 hadoop jar /tmp/parquet-tools-1.11.1.jar $*
JAR_FILE=/tmp/parquet-cli-1.12.2-runtime.jar
[ -f ${JAR_FILE} ] || wget -O ${JAR_FILE} https://repo1.maven.org/maven2/org/apache/parquet/parquet-cli/1.12.2/parquet-cli-1.12.2-runtime.jar
docker run --rm -it -v /tmp:/tmp -v /desdp:/desdp danisla/hadoop:2.9.0 hadoop jar /tmp/parquet-cli-1.12.2-runtime.jar org.apache.parquet.cli.Main $*