Home

Table of Contents Introduction How to add the plugin to your project Minimal project Build Support for Apache Pig Modification of standard JAR file Working with Pig's scripts EASY way to work with Pig's scripts COMPLEX way to work with Pig's scripts

Introduction

Oozie Maven Plugin (OMP) was created to simplify the creation of packages (tar.gz files) containing workflow definition of Apache Oozie as well as all other files needed to run a workflow (configuration files, libraries, etc.). In addition, this plugin enables workflow's reusability -- generated packages can be uploaded to a Maven's repository, added as a dependency to other workflows and reused as subworkflows. The plugin defines a new type of Maven's artifact called oozie, but it uses standard build lifecycles.

How to add the plugin to your project

The sources of OMP are available at https://github.com/CeON/oozie-maven-plugin.

The binaries are available in the Maven's repository of ICM. To use it, you need to add the following sections in your pom.xml:

    <build>
        <plugins>
            <plugin>
                <groupId>pl.edu.icm.maven</groupid>
                <artifactId>oozie-maven-plugin</artifactid>
                <version>current_version_number</version>
                <extensions>true</extensions>
            </plugin>
        </plugins>
    </build>

and

    <pluginRepositories>
        <pluginRepository>
            <id>yadda</id>
            <name>YADDA project repository</name>
            <url>http://maven.icm.edu.pl/artifactory/repo</url>
        </pluginrepository>
    </pluginrepositories>

Minimal project

Minimal project that uses OMP needs to contain the following files:

pom.xml

    <groupId>my-project-groupId</groupid>
    <artifactId>my-project-artifactId</artifactid>
    <version>VERSION_NUMBER</version>
    <packaging>oozie</packaging>
    
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceencoding>
    </properties>
    
    <build>
        <plugins>
            <plugin>
                <groupId>pl.edu.icm.maven</groupid>
                <artifactId>oozie-maven-plugin</artifactid>
                <version>1.1</version>
                <extensions>true</extensions>
            </plugin>
        </plugins>
    </build>
    <dependencies>
        <dependency>
            <groupId>pl.edu.icm.oozie</groupid>
            <artifactId>oozie-runner</artifactid>
            <version>1.2-SNAPSHOT</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <repositories>
        <repository>
            <id>yadda</id>
            <name>YADDA project repository</name>
            <url>http://maven.icm.edu.pl/artifactory/repo</url>
        </repository>
    </repositories>
    <pluginRepositories>
        <pluginRepository>
            <id>yadda</id>
            <name>YADDA project repository</name>
            <url>http://maven.icm.edu.pl/artifactory/repo</url>
        </pluginrepository>
    </pluginrepositories>
 </project>

src/main/oozie/workflow.xml

You can generate these files with https://github.com/CeON/oozie-maven-archetype:

 mvn archetype:generate -DarchetypeArtifactId=oozie-maven-archetype \
   -DarchetypeGroupId=pl.edu.icm.maven.archetypes -DarchetypeVersion=1.0-SNAPSHOT \
   -DinteractiveMode=false -DgroupId=my-project-groupId -DartifactId=my-project-artifactId \
   -Dversion=VERSION_NUMBER -DarchetypeRepository=http://maven.icm.edu.pl/artifactory/repo

Build

You can build the project by calling

 mvn install

This call creates the package that can be uploaded to a Maven's repository and used in other projects. That package does not contain any dependencies (subworkflows, libraries), that should also be stored in the Maven's repository. The file created in this procedure is named artifactId-version-oozie-wf.tar.gz.

If you want to build a package intended for run on an Oozie server, you need to call

 mvn install -DjobPackage

The file created (artifactId-version-oozie-job.tar.gz) contains everything that is necessary to run a given workflow.

Support for Apache Pig

Oozie Maven Plugin (OMP) supports scripts written in PigLatin.

Modification of standard JAR file

OMP allows to use Pig's scripts from dependent modules. Such a module (containing Java classes such as UDFs used by Pig's scripts) should be added to a your project as direct dependency. A proper resource management in pom.xml file is necessary to ensure that a given dependent module contains Pig's scripts. For example, the following inset in pom.xml should guarantee that requirement:

        <build>
                <resources>
                        <resource>
                                <directory>src/main/pig</directory>
                                <filtering>false</filtering>
                                <includes>
                                        <include>**/*.pig</include>
                                </includes>
                                <excludes>
                                        <exclude>**/AUXIL*.pig</excludes>
                                </excludes>
                                <targetPath>${project.build.directory}/classes/pig</targetpath>
                        </resource>
                </resources>
        </build>

Once the above inset is added to pom.xml, the instruction mvn install will add to a generated JAR a directory pig with files *.pig copied from src/main/pig.

Example 1: file src/main/pig/lorem/ipsum/dolor/sit.pig will appear in the JAR file as pig/lorem/ipsum/dolor/sit.pig.
Example 2: file src/main/pig/lorem/ipsum/dolor/AUXIL_sit.pig will not be added to the JAR file, because it was excluded.

Working with Pig's scripts

The generation of workflows that utilize Pig's scripts is described here. In short: Pig's scripts are indicated in tags <script></script> , while imported scripts (i.e. macros) should be included in tags <file></file>. This in NOT the way OMP works.
OMP allows to work with Pig's scripts in two ways:

EASY
COMPLEX

EASY way to work with Pig's scripts

In this strategy, the POM file needs to contain the description of all used Pig's scripts in the folder pig (see section "Modification of standard JAR file").
This strategy is very easy and it is a highly recommended way of working with OMP.
OMP automatically manages macro files:
- in the JAR file, macro files have to be in a different path than /pig<code>, e.g. <code>/pig/macros<code>

*** it is recommended to put regular scripts in <code>src/main/pig, while macros should go into src/main/pig/macros

- you CANNOT use tags <file></file> to store macro files (it can cause errors).

COMPLEX way to work with Pig's scripts

When you have already implemented a module with large number of scripts, then adjusting the code to the EASY way of working with Pig's scripts may be tedious. In that case, one can use COMPLEX strategy that has two disadvantages:
file pom.xml will be longer
additional descriptor files are necessary

To use COMPLEX strategy you need to:

Place executable scripts in pig-scripts and macros in pig-macros, e.g. pom.xml
Place information about the localization of a descriptor in workflow's pom.xml - only the first descriptor will be used e.g. pom.xml
Add the descriptor, e.g. descriptor.xml, conforming the following XSD schema:

 <?xml version="1.0" encoding="UTF-8"?>
 <xs:schema version="1.0"
 		xmlns:xs="http://www.w3.org/2001/XMLSchema" 
 		elementFormDefault="qualified"
 		attributeFormDefault="unqualified"
 		>
     <xs:complexType name="includesType">
     	<xs:sequence>
     		<xs:element name="include" minOccurs="0" maxOccurs="unbounded" type="xs:string"/>
     	</xs:sequence>
     </xs:complextype>    	
 	<xs:complexType name="excludesType">
 		<xs:sequence>
     		<xs:element name="exclude" minOccurs="0" maxOccurs="unbounded" type="xs:string"/>
     	</xs:sequence>
 	</xs:complextype>
 	<xs:complexType name="scriptHandlingType">
 		<xs:sequence>
 			<xs:element name="root" maxOccurs="1" type="xs:string" default="pig" />
 			<xs:element name="preserve" maxOccurs="1" type="xs:boolean" default="true"/>
 			<xs:element name="target" maxOccurs="1" type="xs:string" default="/"/>
 			<xs:element name="includes" minOccurs="1" maxOccurs="1" type="includesType"/>
 			<xs:element name="excludes" minOccurs="0" maxOccurs="1" type="excludesType"/>
 		</xs:sequence>
 	</xs:complextype>	
 	<xs:complexType name="mainProjectPigType">
 		<xs:sequence>
 			<xs:element name="scripts" type="scriptHandlingType"/>
 			<xs:element name="macros" type="scriptHandlingType"/>
 		</xs:sequence>
 	</xs:complextype>
 	<xs:complexType name="depsProjectPigType">
 		<xs:sequence>
 			<xs:element name="all-scripts" type="scriptHandlingType"/>
 		</xs:sequence>
 	</xs:complextype>
 	<xs:complexType name="oozieMavenPluginType">
 		<xs:sequence>
 			<xs:element name="main-project-pig" type="mainProjectPigType"/>
 			<xs:element name="deps-project-pig" type="depsProjectPigType"/>
 		</xs:sequence>
 	</xs:complextype>
 	<xs:element  name="oozie-maven-plugin" type="oozieMavenPluginType"/>
 </xs:schema>

In a descriptor file, in <build><resources> tag, one needs to specify whether a given script is a main script or a macro. Rules for inclusion/exclusion of main scripts are described in tags <main-project-pig><scripts> and <main-project-pig><macros>.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly