Meetup11-Lab2-BatchAI-MultiNode-Horovod.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LAB 2 - Batch AI with Horovod (Multi-GPU, Multi-node)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This recipe shows how to run Horovod distributed training framework using Batch AI.\n",
    "Currently Batch AI has no native support for Horovod framework, but it's easy to run it using customtoolkit and job preparation command line..\n",
    "\n",
    "## Details\n",
    "\n",
    "- Standard Horovod tensorflow_mnist.py example will be used;\n",
    "- tensorflow_mnist.py downloads training data on its own during execution;\n",
    "- The job will be run on standard tensorflow container tensorflow/tensorflow:1.1.0-gpu;\n",
    "- Horovod framework will be installed in the container using job preparation command line. Note, you can build your own docker image containing tensorflow and horovod instead.\n",
    "- Standard output of the job will be stored on Azure File Share."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instructions\n",
    "\n",
    "### Create in Azure the Resource Groups and Storage Accounts needed.\n",
    "```\n",
    "> ssh sshuser@YOUR.VM.IP.ADDRESS\n",
    "> az login\n",
    "> az group create --name batchai_rg  --location eastus\n",
    "> az storage account create --location eastus --name batchaipablo --resource-group batchai_rg --sku Standard_LRS\n",
    "> az storage account keys list --account-name batchaipablo --resource-group batchai_rg -o table\n",
    "> az ad sp create-for-rbac --name MyAppSvcPppl --password Passw0rd\n",
    "> az storage account keys list --account-name batchaipablo --resource-group batchai_rg\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read Configuration and Create Batch AI client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "nbpresent": {
     "id": "bfa11f00-8866-4051-bbfe-a9646e004910"
    }
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "from datetime import datetime\n",
    "import os\n",
    "import sys\n",
    "import zipfile\n",
    "\n",
    "from azure.storage.file import FileService\n",
    "import azure.mgmt.batchai.models as models\n",
    "\n",
    "# utilities.py contains helper functions\n",
    "import utilities\n",
    "\n",
    "# Resource Group\n",
    "location = 'eastus'\n",
    "resource_group = 'batchai_rg'\n",
    "\n",
    "# credentials used for authentication\n",
    "client_id = 'ec0640c7-61fa-4662-bce4-8a3e931939ac'\n",
    "secret = 'Passw0rd'\n",
    "token_uri = 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/token'\n",
    "subscription_id = 'b1395605-1fe9-4af4-b3ff-82a4725a3791'\n",
    "\n",
    "# credentials used for storage\n",
    "storage_account_name = 'batchaipablo'\n",
    "storage_account_key = 'y59heteYEbw5nTLBB/b7rj3jUphvs2Iwslg4AsXFSb4G7ZLgJUep4AuccSmST7I3E8Zw4BaUloebK+VyKmGpog=='\n",
    "\n",
    "# specify the credentials used to remote login your GPU node\n",
    "admin_user_name = 'sshuser'\n",
    "admin_user_password = 'Passw0rd.1!!'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from azure.common.credentials import ServicePrincipalCredentials\n",
    "import azure.mgmt.batchai as batchai\n",
    "import azure.mgmt.batchai.models as models\n",
    "\n",
    "creds = ServicePrincipalCredentials(client_id=client_id, secret=secret, token_uri=token_uri)\n",
    "\n",
    "client = batchai.BatchAIManagementClient(credentials=creds,subscription_id=subscription_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create File Share\n",
    "\n",
    "For this example we will create a new File Share with name `batchaisample` under your storage account.\n",
    "\n",
    "**Note** You don't need to create new file share for every cluster. We are doing this in this sample to simplify resource management for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "azure_file_share_name = 'batchailab2'\n",
    "service = FileService(storage_account_name, storage_account_key)\n",
    "service.create_share(azure_file_share_name, fail_on_exist=False)\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Compute Cluster\n",
    "\n",
    "- For this example we will use a gpu cluster of STANDARD_NC6 nodes. Number of nodes in the cluster is configured with nodes_count variable;\n",
    "- We will mount file share at folder with name external. Full path of this folder on a computer node will be $AZ_BATCHAI_MOUNT_ROOT/external;\n",
    "- We will call the cluster nc6.\n",
    "\n",
    "So, the cluster will have the following parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "azure_file_share = 'external'\n",
    "nodes_count = 2\n",
    "cluster_name = 'nc6'\n",
    "vmsize = \"Standard_NC6\" \n",
    "\n",
    "volumes = models.MountVolumes(\n",
    "    azure_file_shares=[\n",
    "        models.AzureFileShareReference(\n",
    "            account_name=storage_account_name,\n",
    "            credentials=models.AzureStorageCredentialsInfo(\n",
    "                account_key=storage_account_key),\n",
    "            azure_file_url = 'https://{0}.file.core.windows.net/{1}'.format(\n",
    "                storage_account_name, azure_file_share_name),\n",
    "            relative_mount_path=azure_file_share)\n",
    "    ]\n",
    ")\n",
    "\n",
    "parameters = models.ClusterCreateParameters(\n",
    "    location=location,\n",
    "    vm_size=vmsize,\n",
    "    virtual_machine_configuration=models.VirtualMachineConfiguration(\n",
    "        image_reference=models.ImageReference(\n",
    "            publisher=\"microsoft-ads\",\n",
    "            offer=\"linux-data-science-vm-ubuntu\",\n",
    "            sku=\"linuxdsvmubuntu\",\n",
    "            version=\"latest\")),    \n",
    "    user_account_settings=models.UserAccountSettings(\n",
    "        admin_user_name=admin_user_name,\n",
    "        admin_user_password=admin_user_password),\n",
    "    scale_settings=models.ScaleSettings(\n",
    "        manual=models.ManualScaleSettings(target_node_count=nodes_count)\n",
    "    ),\n",
    "    node_setup=models.NodeSetup(\n",
    "        mount_volumes=volumes,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Compute Cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cluster = client.clusters.create(resource_group, cluster_name, parameters).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitor Cluster Creation\n",
    "\n",
    "Monitor the just created cluster. utilities.py contains a helper function to print out detail status of the cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cluster state: AllocationState.resizing Target: 2; Allocated: 0; Idle: 0; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n"
     ]
    }
   ],
   "source": [
    "cluster = client.clusters.get(resource_group, cluster_name)\n",
    "utilities.print_cluster_status(cluster)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Deploy Sample Script and Configure the Input Directories\n",
    "\n",
    "- Download original sample script:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading https://raw.githubusercontent.com/uber/horovod/v0.9.10/examples/tensorflow_mnist.py ...Done\n"
     ]
    }
   ],
   "source": [
    "sample_script_url = 'https://raw.githubusercontent.com/uber/horovod/v0.9.10/examples/tensorflow_mnist.py'\n",
    "utilities.download_file(sample_script_url, 'tensorflow_mnist.py')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Create a folder in the file share and upload the sample script to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "samples_dir = \"horovod_samples\"\n",
    "service = FileService(storage_account_name, storage_account_key)\n",
    "service.create_directory(\n",
    "    azure_file_share_name, samples_dir, fail_on_exist=False)\n",
    "service.create_file_from_path(\n",
    "    azure_file_share_name, samples_dir, 'tensorflow_mnist.py', 'tensorflow_mnist.py')\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The job needs to know where to find train_mnist.py script (the script will download MNIST dataset on its own). So, we will configure an input directory for the script:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "input_directories = [\n",
    "    models.InputDirectory(\n",
    "        id='SCRIPTS',\n",
    "        path='$AZ_BATCHAI_MOUNT_ROOT/{0}/{1}'.format(azure_file_share, samples_dir))\n",
    "]\n",
    "output_directories = [\n",
    "    models.OutputDirectory(\n",
    "        id='MODEL',\n",
    "        path_prefix='$AZ_BATCHAI_MOUNT_ROOT/{0}'.format(azure_file_share),\n",
    "        path_suffix=\"Models\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Output Directories\n",
    "We will store standard and error output of the job in File Share:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "std_output_path_prefix = \"$AZ_BATCHAI_MOUNT_ROOT/{0}\".format(azure_file_share)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are multiple ways to create folders and upload files into Azure File Share - you can use [Azure Portal](https://ms.portal.azure.com), [Storage Explorer](http://storageexplorer.com/), [Azure CLI2](/azure-cli-extension) or Azure SDK for your preferable programming language.\n",
    "In this example we will use Azure SDK for python to copy files into file share."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The job will be able to reference those directories using $AZ_BATCHAI_INPUT_SCRIPTS environment variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Job\n",
    "- Will use configured previously input and output directories;\n",
    "- We will use custom toolkit job to run tensorflow_mnist.py on multiple nodes (use node_count parameter to specify number of nodes). Note, Batch AI will create a hostfile for the job, it can be found via $AZ_BATCHAI_MPI_HOST_FILE environment variable;\n",
    "- Horovod framework will be installed by job preparation command line;\n",
    "- Will output standard output and error streams to file share.\n",
    "\n",
    "You can delete container_settings from the job definition to run the job directly on host DSVM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "job_name = datetime.utcnow().strftime(\"horovod_%m_%d_%Y_%H%M%S\")\n",
    "parameters = models.job_create_parameters.JobCreateParameters(\n",
    "     location=location,\n",
    "     cluster=models.ResourceId(cluster.id),\n",
    "     node_count=nodes_count,\n",
    "     input_directories=input_directories,\n",
    "     output_directories=output_directories,\n",
    "     std_out_err_path_prefix=std_output_path_prefix,\n",
    "     container_settings=models.ContainerSettings(\n",
    "         models.ImageSourceRegistry(image='tensorflow/tensorflow:1.1.0-gpu')),\n",
    "     job_preparation=models.JobPreparation(\n",
    "         command_line=\"apt update; apt install mpi-default-dev mpi-default-bin -y; pip install horovod\"),\n",
    "     custom_toolkit_settings = models.CustomToolkitSettings(\n",
    "         command_line='mpirun -mca btl_tcp_if_exclude docker0,lo --allow-run-as-root --hostfile $AZ_BATCHAI_MPI_HOST_FILE python $AZ_BATCHAI_INPUT_SCRIPTS/tensorflow_mnist.py'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a training Job and wait for Job completion\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created Job: horovod_02_13_2018_223056\n"
     ]
    }
   ],
   "source": [
    "job = client.jobs.create(resource_group, job_name, parameters).result()\n",
    "print('Created Job: {}'.format(job_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wait for Job to Finish\n",
    "The job will start running when the cluster will have enought idle nodes. The following code waits for job to start running printing the cluster state. During job run, the code prints current content of stdout.\n",
    "\n",
    "**Note** Execution may take several minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 1; Unusable: 0; Running: 0; Preparing: 1; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 1; Unusable: 0; Running: 0; Preparing: 1; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 1; Unusable: 0; Running: 0; Preparing: 1; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 1; Unusable: 0; Running: 0; Preparing: 1; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 1; Unusable: 0; Running: 0; Preparing: 1; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: queued ExitCode: None\n",
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 2; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: running ExitCode: None\n",
      "Waiting for job output to become available...\n",
      "Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/data/docker/overlay2/l/JWRO22M657HIKRYNOVZQVNMJRZ:/data/docker/overlay2/l/VXD4X2JAELFF5XLV42LLWDA5SC:/data/docker/overlay2/l/C4TDOTNL4FTYJUOCWD55FERE5I:/data/docker/overlay2/l/GBVIOLTFXGU4LAKXFRPHLY7TWA:/data/docker/overlay2/l/PQL47OVRNTD6KQDC5ZI5IXQ3PT:/data/docker/overlay2/l/KPAVOCA7MQZVFLU3XPDET3TNUE:/data/docker/overlay2/l/VNEZRIPWGTTKPJ7MHMQFDA22CY:/data/docker/overlay2/l/AU2YBICSLERAEPTGZPUOSOTYIC:/data/docker/overlay2/l/C7IS6SZJLL3HZGBJUJVFMZG23Y:/data/docker/'\n",
      "Unexpected end of /proc/mounts line `overlay2/l/52FJSGI27RQQGKLNX76HFCHT7N:/data/docker/overlay2/l/JKVOBSOFDI4FWKGRCVVGZH77ZF:/data/docker/overlay2/l/FRRTB4XW67U7KRR34IRTWH7VWX:/data/docker/overlay2/l/7H4ZF4ZJQJ3IVNGT4RWC432IOL:/data/docker/overlay2/l/6R22AEXKZ4BJTMW2DENI2YCMS7:/data/docker/overlay2/l/CT3WV7M4SV7JDGDPVU5PNSOETQ:/data/docker/overlay2/l/UXCWHKPE3H3UQFMROXYQZYQYX7:/data/docker/overlay2/l/MAVRUDOFQUMKB3OYJYJ5YETCGB:/data/docker/overlay2/l/RC25PVGIYJL3OVPOH4B3LZEYV6:/data/docker/overlay2/l/4GVHWTBEHVF2TNUJP5O3PHHASI:/data/docker/o'\n",
      "Warning: Permanently added '[10.0.0.5]:23' (ECDSA) to the list of known hosts.\n",
      "Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/data/docker/overlay2/l/XT7OFFONR6D34FJCKLYPEWIEW5:/data/docker/overlay2/l/EVGIK7TOAYP2F6IEDUQDBYMVJQ:/data/docker/overlay2/l/Z33S555XJFTFIJNDFZ5N5625M5:/data/docker/overlay2/l/AZZXMLPDFF73ZRSGFKQ2SVTJVT:/data/docker/overlay2/l/GBHCB7T27R74QFBVGO3ZNG2Q6B:/data/docker/overlay2/l/FUOHITAO75WFFBJXJNFRHCC44J:/data/docker/overlay2/l/5UTZM3FNN6AVFL56IBZJNUU2IM:/data/docker/overlay2/l/2BWVN56TGI3VKYX4IOKSL6UV2N:/data/docker/overlay2/l/HGU4SOUT2EWXNJ4YQH4PF23P7S:/data/docker/'\n",
      "Unexpected end of /proc/mounts line `overlay2/l/SFUVEPGTMLSKVJZONBCTNLKUMZ:/data/docker/overlay2/l/MGMHRNRSOMLCWKNDMW2R63BXWX:/data/docker/overlay2/l/6SGJ3PIXC4KCBZEX5PMLEABGFH:/data/docker/overlay2/l/HMZAV7TM4UGNWS3J4A7GH4BNKR:/data/docker/overlay2/l/E4QOT74ZLH44A3CWGS2AVVDYMB:/data/docker/overlay2/l/RSBHGDUPMA6AASONXRE75UYMTX:/data/docker/overlay2/l/TVOHPRGEVQXPCN7VAXTBGL3V3I:/data/docker/overlay2/l/ROI335IOT7R2FV5UQ5TV3I5IEH:/data/docker/overlay2/l/GL7J4DXWBKFYV2MIU27EGGAYKM:/data/docker/overlay2/l/EKA43NLEYC5PYIMVUIBDEEDJNZ:/data/docker/o'\n",
      "2018-02-13 22:36:16.690219: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 22:36:16.690262: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 22:36:16.690275: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 22:36:16.690284: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 22:36:16.690293: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 22:36:16.785414: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 22:36:16.785454: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 22:36:16.785466: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 22:36:16.785475: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 22:36:16.785485: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 22:36:17.040690: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 22:36:17.040741: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 22:36:17.040756: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 22:36:17.040768: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 22:36:17.040779: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-02-13 22:36:17.110423: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties: \n",
      "name: Tesla K80\n",
      "major: 3 minor: 7 memoryClockRate (GHz) 0.8235\n",
      "pciBusID f568:00:00.0\n",
      "Total memory: 11.17GiB\n",
      "Free memory: 11.10GiB\n",
      "2018-02-13 22:36:17.110465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 1 \n",
      "2018-02-13 22:36:17.110478: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 1:   Y \n",
      "2018-02-13 22:36:17.110497: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 1, name: Tesla K80, pci bus id: f568:00:00.0)\n",
      "2018-02-13 22:36:17.112201: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 22:36:17.112248: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 22:36:17.112263: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 22:36:17.112273: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 22:36:17.112283: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 22:36:17.431139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties: \n",
      "name: Tesla K80\n",
      "major: 3 minor: 7 memoryClockRate (GHz) 0.8235\n",
      "pciBusID de16:00:00.0\n",
      "Total memory: 11.17GiB\n",
      "Free memory: 11.10GiB\n",
      "2018-02-13 22:36:17.431179: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0 \n",
      "2018-02-13 22:36:17.431192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0:   Y \n",
      "2018-02-13 22:36:17.431206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K80, pci bus id: de16:00:00.0)\n",
      "2018-02-13 22:36:17.633667: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties: \n",
      "name: Tesla K80\n",
      "major: 3 minor: 7 memoryClockRate (GHz) 0.8235\n",
      "pciBusID cb24:00:00.0\n",
      "Total memory: 11.17GiB\n",
      "Free memory: 11.10GiB\n",
      "2018-02-13 22:36:17.633715: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0 \n",
      "2018-02-13 22:36:17.633730: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0:   Y \n",
      "2018-02-13 22:36:17.633745: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K80, pci bus id: cb24:00:00.0)\n",
      "2018-02-13 22:36:17.896361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties: \n",
      "name: Tesla K80\n",
      "major: 3 minor: 7 memoryClockRate (GHz) 0.8235\n",
      "pciBusID e9ca:00:00.0\n",
      "Total memory: 11.17GiB\n",
      "Free memory: 11.10GiB\n",
      "2018-02-13 22:36:17.896412: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 1 \n",
      "2018-02-13 22:36:17.896427: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 1:   Y \n",
      "2018-02-13 22:36:17.896442: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 1, name: Tesla K80, pci bus id: e9ca:00:00.0)\n",
      "INFO:tensorflow:loss = 2.31104, step = 1\n",
      "INFO:tensorflow:loss = 2.30567, step = 1\n",
      "INFO:tensorflow:loss = 2.30738, step = 1\n",
      "INFO:tensorflow:loss = 2.29781, step = 1\n",
      "[a23eec6aaccb4e239966103c13735a14000000:01201] 3 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics\n",
      "[a23eec6aaccb4e239966103c13735a14000000:01201] Set MCA parameter \"orte_base_help_aggregate\" to 0 to see all help / error messages\n",
      "INFO:tensorflow:loss = 2.27969, step = 11 (1.118 sec)\n",
      "INFO:tensorflow:loss = 2.29065, step = 11 (1.118 sec)\n",
      "INFO:tensorflow:loss = 2.28894, step = 11 (1.118 sec)\n",
      "INFO:tensorflow:loss = 2.27034, step = 11 (1.118 sec)\n",
      "INFO:tensorflow:loss = 2.23895, step = 21 (1.264 sec)\n",
      "INFO:tensorflow:loss = 2.24862, step = 21 (1.264 sec)\n",
      "INFO:tensorflow:loss = 2.2543, step = 21 (1.263 sec)\n",
      "INFO:tensorflow:loss = 2.25494, step = 21 (1.265 sec)\n",
      "INFO:tensorflow:loss = 2.13104, step = 31 (1.191 sec)\n",
      "INFO:tensorflow:loss = 2.15726, step = 31 (1.192 sec)\n",
      "INFO:tensorflow:loss = 2.12664, step = 31 (1.191 sec)\n",
      "INFO:tensorflow:loss = 2.11923, step = 31 (1.190 sec)\n",
      "INFO:tensorflow:loss = 1.52892, step = 41 (1.017 sec)\n",
      "INFO:tensorflow:loss = 1.57253, step = 41 (1.016 sec)\n",
      "INFO:tensorflow:loss = 1.61064, step = 41 (1.017 sec)\n",
      "INFO:tensorflow:loss = 1.54809, step = 41 (1.019 sec)\n",
      "INFO:tensorflow:loss = 1.60446, step = 51 (1.046 sec)\n",
      "INFO:tensorflow:loss = 1.59581, step = 51 (1.046 sec)\n",
      "INFO:tensorflow:loss = 1.55591, step = 51 (1.046 sec)\n",
      "INFO:tensorflow:loss = 1.46641, step = 51 (1.044 sec)\n",
      "INFO:tensorflow:loss = 1.32951, step = 61 (1.041 sec)\n",
      "INFO:tensorflow:loss = 1.1142, step = 61 (1.041 sec)\n",
      "INFO:tensorflow:loss = 1.24239, step = 61 (1.041 sec)\n",
      "INFO:tensorflow:loss = 1.29762, step = 61 (1.041 sec)\n",
      "INFO:tensorflow:loss = 0.607636, step = 71 (1.040 sec)\n",
      "INFO:tensorflow:loss = 0.761951, step = 71 (1.041 sec)\n",
      "INFO:tensorflow:loss = 0.684534, step = 71 (1.040 sec)\n",
      "INFO:tensorflow:loss = 0.630192, step = 71 (1.040 sec)\n",
      "INFO:tensorflow:loss = 0.567172, step = 81 (1.288 sec)\n",
      "INFO:tensorflow:loss = 0.708766, step = 81 (1.287 sec)\n",
      "INFO:tensorflow:loss = 0.558971, step = 81 (1.288 sec)\n",
      "INFO:tensorflow:loss = 0.609399, step = 81 (1.289 sec)\n",
      "INFO:tensorflow:loss = 0.396271, step = 91 (1.099 sec)\n",
      "INFO:tensorflow:loss = 0.380076, step = 91 (1.096 sec)\n",
      "INFO:tensorflow:loss = 0.568736, step = 91 (1.098 sec)\n",
      "INFO:tensorflow:loss = 0.338829, step = 91 (1.099 sec)\n",
      "Job state: succeeded ExitCode: 0\n"
     ]
    }
   ],
   "source": [
    "utilities.wait_for_job_completion(client, resource_group, job_name, cluster_name, 'stdouterr', 'stderr.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Download stdout.txt and stderr.txt files for the Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading https://batchaipablo.file.core.windows.net/batchailab1/b1395605-1fe9-4af4-b3ff-82a4725a3791/batchai_rg/jobs/horovod_02_13_2018_223056/534590b5-bd2b-4c05-b733-0c6abdecdb4f/stderr-job_prep-tvm-3657382398_1-20180213t222810z.txt?sv=2016-05-31&sr=f&sig=yWqR8oZ3%2B4b7WtgTB4cqsVgRovIAY3KkuLLfCQd4g00%3D&se=2018-02-13T23%3A37%3A43Z&sp=rl ...Done\n",
      "Downloading https://batchaipablo.file.core.windows.net/batchailab1/b1395605-1fe9-4af4-b3ff-82a4725a3791/batchai_rg/jobs/horovod_02_13_2018_223056/534590b5-bd2b-4c05-b733-0c6abdecdb4f/stderr-job_prep-tvm-3657382398_2-20180213t222810z.txt?sv=2016-05-31&sr=f&sig=LlRXFakQabcEcHHylUB4yR30aKEzsI%2BXskkD8t77Yus%3D&se=2018-02-13T23%3A37%3A43Z&sp=rl ...Done\n",
      "Downloading https://batchaipablo.file.core.windows.net/batchailab1/b1395605-1fe9-4af4-b3ff-82a4725a3791/batchai_rg/jobs/horovod_02_13_2018_223056/534590b5-bd2b-4c05-b733-0c6abdecdb4f/stderr.txt?sv=2016-05-31&sr=f&sig=cDsMwOQvjIr%2BV3hCsvW5vxoL8VDHRWztMTuJM45M6%2B4%3D&se=2018-02-13T23%3A37%3A43Z&sp=rl ...Done\n",
      "Downloading https://batchaipablo.file.core.windows.net/batchailab1/b1395605-1fe9-4af4-b3ff-82a4725a3791/batchai_rg/jobs/horovod_02_13_2018_223056/534590b5-bd2b-4c05-b733-0c6abdecdb4f/stdout-job_prep-tvm-3657382398_1-20180213t222810z.txt?sv=2016-05-31&sr=f&sig=1frbBF6dvecYTQHM%2B5VNsf0Vkzkg166Ifc66BmrYfp4%3D&se=2018-02-13T23%3A37%3A43Z&sp=rl ...Done\n",
      "Downloading https://batchaipablo.file.core.windows.net/batchailab1/b1395605-1fe9-4af4-b3ff-82a4725a3791/batchai_rg/jobs/horovod_02_13_2018_223056/534590b5-bd2b-4c05-b733-0c6abdecdb4f/stdout-job_prep-tvm-3657382398_2-20180213t222810z.txt?sv=2016-05-31&sr=f&sig=%2BAOyWTG29wP8GoZtUyS5Qe%2BBub%2BT3NNqUJhSyUUJQhA%3D&se=2018-02-13T23%3A37%3A43Z&sp=rl ...Done\n",
      "Downloading https://batchaipablo.file.core.windows.net/batchailab1/b1395605-1fe9-4af4-b3ff-82a4725a3791/batchai_rg/jobs/horovod_02_13_2018_223056/534590b5-bd2b-4c05-b733-0c6abdecdb4f/stdout.txt?sv=2016-05-31&sr=f&sig=S%2BASdxC84JUR2ymWBtiqSC6nSe8bAY46nE2mLv81FCM%3D&se=2018-02-13T23%3A37%3A43Z&sp=rl ...Done\n",
      "All files downloaded\n"
     ]
    }
   ],
   "source": [
    "files = client.jobs.list_output_files(resource_group, job_name, models.JobsListOutputFilesOptions(\"stdOuterr\")) \n",
    "for f in list(files):\n",
    "    utilities.download_file(f.download_url, f.name)\n",
    "print(\"All files downloaded\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "_ = client.jobs.delete(resource_group, job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the Cluster\n",
    "When you are finished with the sample and don't want to submit any more jobs you can delete the cluster using the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "_ = client.clusters.delete(resource_group, cluster_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Delete File Share\n",
    "When you are finished with the sample and don't want to submit any more jobs you can delete the file share completely with all files using the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "service = FileService(storage_account_name, storage_account_key)\n",
    "service.delete_share(azure_file_share_name)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}