Skip to content

Commit

Permalink
add Pravega, Flink and Scala versions
Browse files Browse the repository at this point in the history
Signed-off-by: Derek Moore <[email protected]>
  • Loading branch information
derekm committed Nov 16, 2021
1 parent e5b9c0d commit 96098eb
Show file tree
Hide file tree
Showing 2 changed files with 29 additions and 26 deletions.
39 changes: 20 additions & 19 deletions documentation/src/docs/dev-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
import versions from '@site/versions';

# Flink Connector - Dev Guide

Expand All @@ -26,7 +27,7 @@ Learn how to build your own applications that using Flink connector for Pravega.

To complete this guide, you need:

* JDK 8 or 11 installed with JAVA_HOME configured appropriately
* JDK 8 or 11 installed with `JAVA_HOME` configured appropriately
* Pravega running(Check [here](https://pravega.io/docs/latest/getting-started/) to get started with Pravega)
* Use Gradle or Maven

Expand All @@ -38,12 +39,12 @@ However, you can go straight to the completed example at [flink-connector-exampl

# Starting Flink

Download Flink release and un-tar it. We use Flink 1.11.2 here.
<p>Download Flink release and un-tar it. We use Flink {versions.flink} here.</p>

```bash
$ tar -xzf flink-1.11.2-bin-scala_2.12.tgz
$ cd flink-1.11.2-bin-scala_2.12
```
<pre><code {...{ "className": "language-bash"}}>
{`$ tar -xzf flink-${versions.flink}-bin-scala_${versions.scala}.tgz
$ cd flink-${versions.flink}-bin-scala_${versions.scala}`}
</code></pre>

Start a cluster

Expand All @@ -70,11 +71,11 @@ You can follow [here](https://ci.apache.org/projects/flink/flink-docs-stable/dev

Add the below snippet to dependencies section of build.gradle in the app directory, connector dependencies should be part of the shadow jar. For flink connector dependency, we need to choose the connector which aligns the Flink major version and Scala version if you use Scala, along with the same Pravega version you run.

```groovy
compile group 'org.apache.flink', name: 'flink-streaming-java_2.12', version: '1.11.2'
<pre><code {...{ "className": "language-groovy" }}>
{`compile group 'org.apache.flink', name: 'flink-streaming-java_${versions.scala}', version: '${versions.flink}'

flinkShadowJar group: 'io.pravega', name: 'pravega-connectors-flink-1.11_2.12', version: '0.9.0'
```
flinkShadowJar group: 'io.pravega', name: 'pravega-connectors-flink-${versions['flink-minor']}_${versions.scala}', version: '${versions['flink-connectors']}'`}
</code></pre>

Define custom configurations `flinkShadowJar`

Expand All @@ -99,20 +100,20 @@ You can check [maven-quickstart](https://ci.apache.org/projects/flink/flink-docs

Add below dependencies into Maven POM, these dependencies should be part of the shadow jar

```xml
<dependency>
<pre><code {...{ "className": "language-xml" }}>
{`<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.12</artifactId>
<version>1.11.2</version>
<artifactId>flink-streaming-java_${versions.scala}</artifactId>
<version>${versions['flink']}</version>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>io.pravega</groupId>
<artifactId>pravega-connectors-flink-1.11_2.12</artifactId>
<version>0.9.0</version>
</dependency>
```
<artifactId>pravega-connectors-flink-${versions['flink-minor']}_${versions.scala}</artifactId>
<version>${versions['flink-connectors']}</version>
</dependency>`}
</code></pre>

Invoke `mvn clean package` to build/package your project. You will find a JAR file that contains your application, plus connectors and libraries that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.

Expand Down Expand Up @@ -220,4 +221,4 @@ flink run -c <classname> ${your-app}.jar --controller <pravega-controller-uri>

# What’s next?

This guide covered the creation of a application that uses Flink connector to read and wirte from a pravega stream. However, there is much more. We recommend continuing the journey by going through [flink connector documents](https://pravega.io/docs/latest/connectors/flink-connector/) and check other examples on [flink-connector-examples](https://github.com/pravega/pravega-samples/tree/master/flink-connector-examples).
This guide covered the creation of a application that uses Flink connector to read and wirte from a pravega stream. However, there is much more. We recommend continuing the journey by going through [flink connector documents](overview) and check other examples on [flink-connector-examples](https://github.com/pravega/pravega-samples/tree/master/flink-connector-examples).
16 changes: 9 additions & 7 deletions documentation/src/docs/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
import versions from '@site/versions';

## Creating a Flink Stream Processing Project

Expand All @@ -33,8 +34,8 @@ Once after the set up, please follow the below instructions to add the **Flink P

To add the Pravega connector dependencies to your project, add the following entry to your project file: (For example, `pom.xml` for Maven)

```xml
<!-- Before Pravega 0.6 -->
<pre><code {...{ "className": "language-xml" }}>
{`<!-- Before Pravega 0.6 -->
<dependency>
<groupId>io.pravega</groupId>
<artifactId>pravega-connectors-flink_2.12</artifactId>
Expand All @@ -44,12 +45,13 @@ To add the Pravega connector dependencies to your project, add the following ent
<!-- Pravega 0.6 and After -->
<dependency>
<groupId>io.pravega</groupId>
<artifactId>pravega-connectors-flink-1.9_2.12</artifactId>
<version>0.6.0</version>
</dependency>
```
<artifactId>pravega-connectors-flink-${versions['flink-minor']}_${versions.scala}</artifactId>
<version>${versions['flink-connectors']}</version>
</dependency>`}
</code></pre>

<p>Use appropriate version as necessary. <code>{versions['flink-minor']}</code> is the Flink Major-Minor version. <code>{versions.scala}</code> is the Scala version. <code>{versions.pravega}</code> is the Pravega version.</p>

Use appropriate version as necessary. `1.9` is the Flink Major-Minor version. `2.12` is the Scala version. `0.6.0` is the Pravega version.
The snapshot versions are published to [`GitHub Packages`](https://github.com/orgs/pravega/packages) repository and the release artifacts are available in [`Maven Central`](https://mvnrepository.com/artifact/io.pravega/pravega-connectors-flink) repository.

Alternatively, we could build and publish the connector artifacts to local maven repository by executing the following command and make use of that version as your application dependency.
Expand Down

0 comments on commit 96098eb

Please sign in to comment.