Configuring Naiad to run on Azure HDInsight

You can run any Naiad program on the worker nodes of an Azure HDInsight 3.0 cluster. First follow the instructions (coming soon) for setting up a cluster. Let's assume that your cluster name is myCluster and you associated it with the default storage account clusterAccount with key clusterKey and the default storage container clusterContainer.

Next fetch the submission program from NuGet. This is a bit messy, since NuGet is designed for supplying libraries to use when building projects rather than supplying complete binaries.

1. Create a new dummy Visual Studio solution (Visual Studio 2012 or later) called DummyNaiadSubmission.

2. Right click on the DummyNaiadSubmission project, and select 'Manage NuGet packages...'.

3. Make sure the dropdown at the top says 'Include Prerelease' not 'Stable only'.

4. Select 'Online/nuget.org' from the left hand pane.

5. Type 'naiad' into the 'Search Online box'.

6. Select 'Naiad - Azure submission tool' and install it.

7. Build the project. This will place a lot of binaries into the 'bin/Debug' directory of your solution.

8. Copy the contents of %mySolution%/bin/Debug/ to a location of your choice, e.g. c:\NaiadAzureSubmission.

9. If you want to use a default cluster, edit NaiadSubmission.exe.config. Find the <appSettings> XML tag and add the following new tags inside it:

    <appSettings>
      <add key="clustername" value="myCluster" />
      <add key="storageaccount" value="clusterAccount" />
      <add key="storagekey" value="clusterKey" />
      <add key="container" value="clusterContainer" />
    </appSettings>
   

   where myCluster, clusterAccount, clusterKey and clusterContainer are replaced with your cluster's details.

10. Add c:\NaiadAzureSubmission to your PATH environment variable.

11. If you don't want to set a default cluster, or you want to use multiple clusters, you can override the entries in NaiadSubmission.exe.config from the command line using arguments listed by typing AzureSubmission.exe -help.