Flows-Example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Globus Automate Introduction\n",
    "## Tyler-edited version for Xtract. \n",
    "\n",
    "This notebook goes over using Automate to create a simple flow and monitor its status.\n",
    "\n",
    "To use the notebook you will need to both install the SDK:\n",
    "\n",
    "- pip install globus-automate-client\n",
    "\n",
    "And be added to the Automate whitelist group to be allowed to create and run flows. Please email rchard@anl.gov to get access to this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "import time\n",
    "import json\n",
    "sys.path.append(\".\")\n",
    "os.environ['JUPYTER_PATH'] = '.'\n",
    "CLIENT_ID = \"e6c75d97-532a-4c88-b031-8584a319fa3e\"\n",
    "\n",
    "from globus_automate_client import  (get_access_token_for_scope, create_action_client, \n",
    "                                    create_flows_client)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Globus Automate: Flows and Actions\n",
    "\n",
    "## Flow Definition\n",
    "\n",
    "* Flows are composed of *Action* invocations\n",
    "* Each Action invocation reads from and contributes back to the *Flow State* which is referenced in Flow steps using the `InputPath` and `ResultPath` properties of an Action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "flow_definition = {\n",
    "  \"Comment\": \"Xtract: Crawl and Extract\",\n",
    "  \"StartAt\": \"Crawl_Xtract_v1\",\n",
    "  \"States\": {\n",
    "    \"Crawl_Xtract_v1\": {\n",
    "      \"Comment\": \"Crawl and Extraction of Data stored at user-defined Globus or Google Drive directory path\",\n",
    "      \"Type\": \"Action\",\n",
    "        \"ActionUrl\": \"https://xtract.materialsdatafacility.org\",\n",
    "        \"ActionScope\": \"https://auth.globus.org/scopes/34284fb1-2eea-4532-a04a-9c8ad1702856/xtract_crawl_and_extract\",\n",
    "      \"InputPath\": \"$.Xtract1Input\",\n",
    "      \"ResultPath\": \"$.Xtract1Result\",\n",
    "      \"WaitTime\": 3600,\n",
    "      \"End\": True\n",
    "    }\n",
    "  }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* This flow composes two transfers into a single logical operation\n",
    "  * Suitable, for example, for doing a two stage transfer between a local campus endpoint, a DMZ data transfer endpoint, and a dataset repository.\n",
    "  * Each step in the Flow uses the same Action: Transfer which is referenced by URL\n",
    "  * Globus Auth Scope information is required to authenticate operations to the Action.\n",
    "  * Source and destination information for the Transfer state are given in `InputPath` and `ResultPath`\n",
    "    * Format of the input is Action dependent (see below)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GlobusHTTPResponse({'action_url': 'http://flows.automate.globus.org/flows/d7b97c9e-8473-4c1e-92d3-5ccb7707d51a', 'administered_by': [], 'api_version': '1.0', 'definition': {'Comment': 'Xtract: Crawl and Extract', 'StartAt': 'Crawl_Xtract_v1', 'States': {'Crawl_Xtract_v1': {'ActionScope': 'https://auth.globus.org/scopes/34284fb1-2eea-4532-a04a-9c8ad1702856/xtract_crawl_and_extract', 'ActionUrl': 'https://xtract.materialsdatafacility.org', 'Comment': 'Crawl and Extraction of Data stored at user-defined Globus or Google Drive directory path', 'End': True, 'InputPath': '$.Xtract1Input', 'ResultPath': '$.Xtract1Result', 'Type': 'Action', 'WaitTime': 3600}}}, 'description': '', 'globus_auth_scope': 'https://auth.globus.org/scopes/d7b97c9e-8473-4c1e-92d3-5ccb7707d51a/flow_d7b97c9e_8473_4c1e_92d3_5ccb7707d51a_user', 'globus_auth_username': 'd7b97c9e-8473-4c1e-92d3-5ccb7707d51a@clients.auth.globus.org', 'id': 'd7b97c9e-8473-4c1e-92d3-5ccb7707d51a', 'keywords': [], 'log_supported': True, 'principal_urn': 'urn:globus:auth:identity:d7b97c9e-8473-4c1e-92d3-5ccb7707d51a', 'runnable_by': [], 'subtitle': '', 'synchronous': False, 'title': 'test-two-hop', 'types': ['Action'], 'visible_to': []})\n",
      "Newly created flow with id:\n",
      "d7b97c9e-8473-4c1e-92d3-5ccb7707d51a\n",
      "and scope:\n",
      "https://auth.globus.org/scopes/d7b97c9e-8473-4c1e-92d3-5ccb7707d51a/flow_d7b97c9e_8473_4c1e_92d3_5ccb7707d51a_user\n"
     ]
    }
   ],
   "source": [
    "flows_client = create_flows_client(CLIENT_ID)\n",
    "flow = flows_client.deploy_flow(flow_definition, title='test-two-hop')\n",
    "flow_id = flow['id']\n",
    "print(flow)\n",
    "flow_scope = flow['globus_auth_scope']\n",
    "print(f'Newly created flow with id:\\n{flow_id}\\nand scope:\\n{flow_scope}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Login Here:\n",
      "\n",
      "https://auth.globus.org/v2/oauth2/authorize?client_id=e6c75d97-532a-4c88-b031-8584a319fa3e&redirect_uri=https%3A%2F%2Fauth.globus.org%2Fv2%2Fweb%2Fauth-code&scope=https%3A%2F%2Fauth.globus.org%2Fscopes%2Fd7b97c9e-8473-4c1e-92d3-5ccb7707d51a%2Fflow_d7b97c9e_8473_4c1e_92d3_5ccb7707d51a_user&state=_default&response_type=code&code_challenge=jxl0cyQ6U0jPW7H-kxu7j_dY_0ggRwfEExSL2t9yK_U&code_challenge_method=S256&access_type=offline&prefill_named_grant=Globus+Automate+Client\n",
      "\n",
      "\n",
      "Note that this link can only be used once! If login or a later step in the flow fails, you must restart it.\n",
      "Enter resulting code:NMU8S56zoPA9PQAC2qP9KdK7UR8wS4\n",
      "GlobusHTTPResponse({'action_id': '4759c190-81fb-4ebc-ad75-ff9f50aafe2a', 'completion_time': 'None', 'created_by': 'urn:globus:auth:identity:f12d8fb3-5502-4895-ac33-e59581f9f03f', 'details': {'code': 'ActionStarted', 'description': 'State Crawl_Xtract_v1 of type Action started', 'details': {'input': {'dataset_mdata': {'test1': 'test2'}, 'dest_eid': '4f99675c-ac1f-11ea-bee8-0e716405a293', 'dest_path': '/experiments/test', 'dir_path': '/mdf_open/tsopanidis_wha_amaps_v1.1/Activation Maps/Activation Maps for the WHA Dataset', 'eid': '82f1b5c6-6e9b-11e5-ba47-22000b92c6ec', 'grouper': 'matio', 'mapping': 'match', 'metadata_storage_ep': 'foobar', 'validator_params': {'schema_branch': 'master', 'validation_info': {'test1': 'test2'}}}, 'state_name': 'Crawl_Xtract_v1', 'state_type': 'Action'}, 'time': '2020-09-11T20:59:41.145000+00:00'}, 'start_time': '2020-09-11T20:59:41.040000+00:00', 'status': 'ACTIVE'})\n",
      "Flow action started with id: 4759c190-81fb-4ebc-ad75-ff9f50aafe2a\n",
      "Flow status: ACTIVE\n",
      "Flow status: ACTIVE\n",
      "Flow status: ACTIVE\n",
      "Flow status: ACTIVE\n",
      "Flow status: ACTIVE\n",
      "Flow status: ACTIVE\n",
      "Flow status: ACTIVE\n",
      "Flow status: ACTIVE\n",
      "Flow status: ACTIVE\n",
      "Flow status: ACTIVE\n",
      "Flow status: ACTIVE\n",
      "Flow status: ACTIVE\n",
      "Flow status: FAILED\n",
      "GlobusHTTPResponse({'action_id': '4759c190-81fb-4ebc-ad75-ff9f50aafe2a', 'completion_time': '2020-09-11T21:00:16.292000+00:00', 'created_by': 'urn:globus:auth:identity:f12d8fb3-5502-4895-ac33-e59581f9f03f', 'details': {'code': 'FlowFailed', 'details': {'cause': 'The cause could not be determined because Lambda did not return an error type.'}, 'time': '2020-09-11T21:00:16.292000+00:00'}, 'start_time': '2020-09-11T20:59:41.040000+00:00', 'status': 'FAILED'})\n"
     ]
    }
   ],
   "source": [
    "flow_input = {\n",
    "  \"Xtract1Input\": {\n",
    "    \"metadata_storage_ep\": \"foobar\",\n",
    "    \"eid\": \"82f1b5c6-6e9b-11e5-ba47-22000b92c6ec\",\n",
    "    \"dir_path\": \"/mdf_open/tsopanidis_wha_amaps_v1.1/Activation Maps/Activation Maps for the WHA Dataset\",\n",
    "    \n",
    "    \"mapping\": \"match\",\n",
    "    \"dataset_mdata\": {\"test1\": \"test2\"},\n",
    "    \"validator_params\": {\"schema_branch\": \"master\", \"validation_info\": {\"test1\": \"test2\"}},\n",
    "    \"grouper\": \"matio\",  # options are 'directory/matio'\n",
    "    \"dest_eid\": \"4f99675c-ac1f-11ea-bee8-0e716405a293\", \n",
    "    \"dest_path\": \"/experiments/test\"\n",
    "  }\n",
    "}\n",
    "\n",
    "flow_action = flows_client.run_flow(flow_id, flow_scope, flow_input)\n",
    "print(flow_action)\n",
    "\n",
    "flow_action_id = flow_action['action_id']\n",
    "flow_status = flow_action['status']\n",
    "print(f'Flow action started with id: {flow_action_id}')\n",
    "while flow_status == 'ACTIVE':\n",
    "    time.sleep(2)\n",
    "    flow_action = flows_client.flow_action_status(flow_id, flow_scope, flow_action_id)\n",
    "    flow_status = flow_action['status']\n",
    "    print(f'Flow status: {flow_status}')\n",
    "print (flow_action)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Attachments",
  "kernelspec": {
   "display_name": "Python (plz)",
   "language": "python",
   "name": "plz"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}