Skip to content

Commit

Permalink
Preparing SHC release v1.1.1-2.1
Browse files Browse the repository at this point in the history
  • Loading branch information
weiqingy committed May 18, 2017
1 parent 64b98c2 commit c2d1309
Show file tree
Hide file tree
Showing 4 changed files with 14 additions and 14 deletions.
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,15 +44,15 @@ Run indiviudal test

Run SHC examples

./bin/spark-submit --verbose --class org.apache.spark.sql.execution.datasources.hbase.examples.HBaseSource --master yarn-cluster --packages com.hortonworks:shc-core:1.1.0-2.1-s_2.11 --repositories http://repo.hortonworks.com/content/groups/public/ --files /usr/hdp/current/hbase-client/conf/hbase-site.xml shc-examples-1.1.0-2.1-s_2.11-SNAPSHOT.jar
./bin/spark-submit --verbose --class org.apache.spark.sql.execution.datasources.hbase.examples.HBaseSource --master yarn-cluster --packages com.hortonworks:shc-core:1.1.1-2.1-s_2.11 --repositories http://repo.hortonworks.com/content/groups/public/ --files /usr/hdp/current/hbase-client/conf/hbase-site.xml shc-examples-1.1.1-2.1-s_2.11-SNAPSHOT.jar

The following illustrates how to run your application in real hbase cluster. You need to provide the hbase-site.xml. It may subject to change based on your specific cluster configuration.

./bin/spark-submit --class your.application.class --master yarn-client --packages com.hortonworks:shc-core:1.1.0-2.1-s_2.11 --repositories http://repo.hortonworks.com/content/groups/public/ --files /etc/hbase/conf/hbase-site.xml /To/your/application/jar
./bin/spark-submit --class your.application.class --master yarn-client --packages com.hortonworks:shc-core:1.1.1-2.1-s_2.11 --repositories http://repo.hortonworks.com/content/groups/public/ --files /etc/hbase/conf/hbase-site.xml /To/your/application/jar

Running Spark applications with this connector, HBase jars of version 1.1.2 will be pulled by default. If Phoenix is enabled on HBase cluster, you need to use "--jars" to pass "phoenix-server.jar". For example:

./bin/spark-submit --class your.application.class --master yarn-client --packages com.hortonworks:shc-core:1.1.0-2.1-s_2.11 --repositories http://repo.hortonworks.com/content/groups/public/ --jars /usr/hdp/current/phoenix-client/phoenix-server.jar --files /etc/hbase/conf/hbase-site.xml /To/your/application/jar
./bin/spark-submit --class your.application.class --master yarn-client --packages com.hortonworks:shc-core:1.1.1-2.1-s_2.11 --repositories http://repo.hortonworks.com/content/groups/public/ --jars /usr/hdp/current/phoenix-client/phoenix-server.jar --files /etc/hbase/conf/hbase-site.xml /To/your/application/jar

## Application Usage
The following illustrates the basic procedure on how to use the connector. For more details and advanced use case, such as Avro and composite key support, please refer to the examples in the repository.
Expand Down Expand Up @@ -120,15 +120,15 @@ Given a DataFrame with specified schema, above will create an HBase table with 5
## Configuring Spark-package
Users can use the Spark-on-HBase connector as a standard Spark package. To include the package in your Spark application use:

_**Note**: com.hortonworks:shc-core:1.1.0-2.1-s_2.11 has not been uploaded to [spark-packages.org](https://spark-packages.org/package/hortonworks-spark/shc), but will be there soon._
_**Note**: com.hortonworks:shc-core:1.1.1-2.1-s_2.11 has not been uploaded to [spark-packages.org](https://spark-packages.org/package/hortonworks-spark/shc), but will be there soon._

spark-shell, pyspark, or spark-submit

$SPARK_HOME/bin/spark-shell --packages com.hortonworks:shc-core:1.1.0-2.1-s_2.11
$SPARK_HOME/bin/spark-shell --packages com.hortonworks:shc-core:1.1.1-2.1-s_2.11

Users can include the package as the dependency in your SBT file as well. The format is the spark-package-name:version in build.sbt file.

libraryDependencies += “com.hortonworks/shc-core:1.1.0-2.1-s_2.11”
libraryDependencies += “com.hortonworks/shc-core:1.1.1-2.1-s_2.11”

## Running in secure cluster

Expand All @@ -142,9 +142,9 @@ Suppose hrt_qa is a headless account, user can use following command for kinit:

kinit -k -t /tmp/hrt_qa.headless.keytab hrt_qa

/usr/hdp/current/spark-client/bin/spark-submit --class your.application.class --master yarn-client --files /etc/hbase/conf/hbase-site.xml --packages com.hortonworks:shc-core:1.1.0-2.1-s_2.11 --repositories http://repo.hortonworks.com/content/groups/public/ /To/your/application/jar
/usr/hdp/current/spark-client/bin/spark-submit --class your.application.class --master yarn-client --files /etc/hbase/conf/hbase-site.xml --packages com.hortonworks:shc-core:1.1.1-2.1-s_2.11 --repositories http://repo.hortonworks.com/content/groups/public/ /To/your/application/jar

/usr/hdp/current/spark-client/bin/spark-submit --class your.application.class --master yarn-cluster --files /etc/hbase/conf/hbase-site.xml --packages com.hortonworks:shc-core:1.1.0-2.1-s_2.11 --repositories http://repo.hortonworks.com/content/groups/public/ /To/your/application/jar
/usr/hdp/current/spark-client/bin/spark-submit --class your.application.class --master yarn-cluster --files /etc/hbase/conf/hbase-site.xml --packages com.hortonworks:shc-core:1.1.1-2.1-s_2.11 --repositories http://repo.hortonworks.com/content/groups/public/ /To/your/application/jar

If the solution above does not work and you encounter errors like :

Expand Down
4 changes: 2 additions & 2 deletions core/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,12 @@
<parent>
<groupId>com.hortonworks</groupId>
<artifactId>shc</artifactId>
<version>1.1.1-2.1-s_2.11-SNAPSHOT</version>
<version>1.1.1-2.1-s_2.11</version>
<relativePath>../pom.xml</relativePath>
</parent>

<artifactId>shc-core</artifactId>
<version>1.1.1-2.1-s_2.11-SNAPSHOT</version>
<version>1.1.1-2.1-s_2.11</version>
<packaging>jar</packaging>
<name>HBase Spark Connector Project Core</name>

Expand Down
4 changes: 2 additions & 2 deletions examples/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,12 @@
<parent>
<groupId>com.hortonworks</groupId>
<artifactId>shc</artifactId>
<version>1.1.1-2.1-s_2.11-SNAPSHOT</version>
<version>1.1.1-2.1-s_2.11</version>
<relativePath>../pom.xml</relativePath>
</parent>

<artifactId>shc-examples</artifactId>
<version>1.1.1-2.1-s_2.11-SNAPSHOT</version>
<version>1.1.1-2.1-s_2.11</version>
<packaging>jar</packaging>
<name>HBase Spark Connector Project Examples</name>

Expand Down
4 changes: 2 additions & 2 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
</parent>
<groupId>com.hortonworks</groupId>
<artifactId>shc</artifactId>
<version>1.1.1-2.1-s_2.11-SNAPSHOT</version>
<version>1.1.1-2.1-s_2.11</version>
<packaging>pom</packaging>
<name>HBase Spark Connector Project Parent POM</name>
<url>https://github.com/hortonworks-spark/shc</url>
Expand All @@ -42,7 +42,7 @@
</distributionManagement>

<properties>
<spark.version>2.1.0</spark.version>
<spark.version>2.1.1</spark.version>
<hbase.version>1.1.2</hbase.version>
<phoenix.version>4.9.0-HBase-1.1</phoenix.version>
<test_classpath_file>${project.build.directory}/spark-test-classpath.txt</test_classpath_file>
Expand Down

0 comments on commit c2d1309

Please sign in to comment.