scicat_workshop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "45c72961-dc84-4884-bedd-eaa54959fca3",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# SciCat workshop\n",
    "### Prepared by: Max Novelli and Jan-Lukas Wynen\n",
    "### Version 2023/03/21"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab2cdf17-1dcf-4902-8fbb-7087a5ebc871",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Preface\n",
    "This content is available on [github](http://github.com) in the following repository:\n",
    "- SciCat Workshop: https://github.com/SciCatProject/scicat-workshop"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0aa34b7c-b4af-4d0a-97d2-a0695b07e483",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Introduction\n",
    "\n",
    "SciCat is a Scientific Data Catalogue.  \n",
    "There are at least 5 facilities contributing to the project and more using it.   \n",
    "ESS is one of the main contributors, at the moment.\n",
    "\n",
    "- Project on github: https://github.com/SciCatProject/  \n",
    "- Documentation: https://scicatproject.github.io/documentation/ (Really outdated)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cd12821-1379-4e65-9621-af69fb561de3",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "SciCat has two main components: \n",
    "- the __backend__, which provides the API and the logic to store and access the metadata related to the datasets,\n",
    "- the __frontend__, which provides a web UI to the backend.\n",
    "\n",
    "SciCat is in the middle of backend migration, from v3.x to v4.x.  \n",
    "The project aims to make the migration completely transparent to the user."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b964660-beef-454d-9246-741fcfcdf2c3",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "At ESS, there are two main instances of SciCat:\n",
    "- __production__\n",
    "  the official instance where ESS users will login to retrieve their data. This instance runs an officially released version of SciCat.\n",
    "- __staging__\n",
    "  runs the production code and it is used for testing data ingestion and data curation. It shares the same storage and data transfer services as the production instance.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6fea53e-ad71-420d-98a9-3b708b900124",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "To access SciCat at ESS, please follow the following links:\n",
    "- production: https://scicat.ess.eu\n",
    "- staging : https://staging.scicat.ess.eu\n",
    "\n",
    "You should use your ESS login credentials to login."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64ba0c8e-b3fb-454b-887f-0999a907cf12",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## UI Overview\n",
    "\n",
    "Let's explore SciCat web UI.  \n",
    "Open the ESS staging instance SciCat at https://staging.scicat.ess.eu  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be477e28-9ebf-44ec-8675-1b8f7aca6c54",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Public datasets\n",
    "If you are not logged in, you should be presented with the list of datasets that are _publicly accessible_.  \n",
    "\n",
    "![Landing page, Unauthenticated users](./images/PublicDatasets.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03f7c9b8-3150-4ef4-be74-2fc6ffb23d1e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Login\n",
    "Click on the icon on the top right corner of the screen.  \n",
    "Login with your ESS credentials\n",
    "\n",
    "![Login page](images/Login.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73457095-e456-4b37-adef-9e9b2b0b22d7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### My datasets\n",
    "Once you are logged in, you can see the list of datasets that you have access to.\n",
    "You can check that you are logged in, by verifying the you name is shown at the top right corner of the page.  \n",
    "\n",
    "![My datasets](images/Datasets.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc59e424-7448-417d-bd6c-438981ab8ef5",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Dataset Details\n",
    "\n",
    "![Dataset details](images/DatasetDetails.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d212e968-43a5-4e82-ba7b-b7a131ed7ec4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Dataset's Files\n",
    "\n",
    "![DatasetFiles](images/DatasetFiles.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d5fa270-67ed-4ebc-893d-2d1db0934b56",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## SciCat UI Exercises\n",
    "\n",
    "Interactive play time with the SciCat web UI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3898e32-bb79-4b55-b075-7f6af7753c8a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Exercise #1\n",
    "Find all the datasets acquired at YMIR.  \n",
    "\n",
    "Suggestion: Use the __location__ field in the search form"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6952bd32-21bf-4a8a-85a3-4a4ae6eea67b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Exercise #2\n",
    "Find all the tomography datasets acquired at YMIR.\n",
    "\n",
    "Suggestion: Use the __keywords__ field in the search form"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2536f79b-a00c-4a9d-b85f-f1b767b2e960",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Exercise #3\n",
    "Retrieve the derived dataset related to the tomography dataset acquired on YMIR in run 3931\n",
    "\n",
    "Suggestion: Use the __type__ and __text search__ fields in the search form"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da6946a1-c06a-4250-95b4-6700b97d8a81",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Exercise #4 (1)\n",
    "Retrieve the dataset derived from raw data acquired on YMIR where __run number__ is 3931, __extra entry instrument flir camera image key maximum value__ is equal to 3\n",
    "\n",
    "Suggestions: \n",
    "- Use the __type__ field and __add condition__ feature in the search form to retrieve the raw dataset\n",
    "- in __add condition__, convert the name of the metadata/property to lower case snake case\n",
    "    - run number => run_number\n",
    "    - extra entry instrument flir camera image key maximum value => extra_entry_instrument_flir_camera_image_key_maximum_value\n",
    "- find and save the pid of the raw dataset\n",
    "- clear the search form\n",
    "- use the __type__ and __txt search__ fields"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "407c820e-a797-4e43-8c6a-87b0b1a8f80a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Exercise #4 (2)\n",
    "- Select and copy the dataset pid\n",
    "    - _849c591c-a42e-462b-b5b4-6345e9d2200b_\n",
    "- Go back to the dataset list\n",
    "- Clear the search form\n",
    "- Paste the pid in the __Text Search__ field\n",
    "- Select ___derived___ in the __Type__ field"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f70287cb-356b-40f2-a67b-f94df0483563",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Python Exercises\n",
    "\n",
    "Exploring how to leverage the python libraries to use the information in SciCat in any workflow in Python and/or Jupyter notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78ce94e3-caed-4267-aabc-9a96735622d1",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Set up\n",
    "\n",
    "In order to be able to interact with SciCat API from Python, you can install two different libraries:\n",
    "- [pySciCat](https://github.com/SciCatProject/pyscicat): lower level library porting SciCat endpoints to python\n",
    "- [Scitacean](https://github.com/SciCatProject/scitacean): higher level library which aggregates functionalities in user friendly concepts\n",
    "\n",
    "You can install the previous libraries using the following pip commands\n",
    "```bash\n",
    "pip install pyscicat\n",
    "pip install scitacean\n",
    "```\n",
    "Scitacean is also available through Conda Forge.\n",
    "\n",
    "Keep in mind that both libraries are work-in-progress.\n",
    "The documentation can be found here:\n",
    "- [pySciCat](https://scicatproject.github.io/pyscicat/)\n",
    "- [Scitacean](https://scicatproject.github.io/scitacean/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "917e950e-264e-43e7-990a-db5fc264bd06",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Set up: python environment\n",
    "\n",
    "We suggest to use conda and create an environment from the specification file provided in this repository.  \n",
    "The files needed to create a conda environment or to install the packages in a simple python environment using pip, are available in the folder:\n",
    "```bash\n",
    "environment\n",
    "```\n",
    "\n",
    "Detailed instructions on how to create such environments are explained in details in the [setup](./environment/SETUP.md) file.  \n",
    "In case you want to get going quickly, here is the summary:  \n",
    "\n",
    "#### Conda\n",
    "```sh\n",
    "conda env create -f conda-scicat-workshop.yml\n",
    "conda activate scicat-workshop\n",
    "```\n",
    "\n",
    "### Pip\n",
    "```sh\n",
    "python -m pip install -r pip-scicat-workshop.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9940269a-e88b-4a1f-bdcd-389cd142777c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Set up: ssh passwordless login\n",
    "\n",
    "To make the process smooth, we suggest to create a passowrdless login with ssh.   \n",
    "Here are the steps:\n",
    "- set up an SSH-key for `login.esss.dk` (Please check this [guide](https://linuxize.com/post/how-to-setup-passwordless-ssh-login/) if you need to help in this step)\n",
    "- add it to your ssh-agent (https://www.ssh.com/academy/ssh/agent)\n",
    "- define an alias in `~/.ssh/config`:\n",
    "\n",
    "```\n",
    "Host login.esss.dk\n",
    "    HostName login.esss.dk\n",
    "    User <your-username>\n",
    "    IdentityFile <private-key-file>\n",
    "```\n",
    "\n",
    "Please read the detailed instruction provided in the [setup](./environment/SETUP.md) file.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b189b7a-49fc-4fb8-b1b1-d25c8dea958a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Retrieve your user token\n",
    "\n",
    "In order to facilitate accessing SciCat and improve security,  \n",
    "we can instruct the python libraries to use the authentication token that we already acquired when logging in into the SciCat web UI.  \n",
    "Here is how we can retrieve it:\n",
    "- Click on the user icon (top right where the user name is shown)\n",
    "- Select __Settings__\n",
    "- In the tile named __User Information__, the last item is the token that we need to copy.\n",
    "- Click on the __Copy__ icon to the right of the string\n",
    "- Paste the string in the Jupyter notebook, where it says __\\<YOUR_CURRENT_SCICAT_TOKEN\\>__   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9501183-0fd9-408c-ab4f-ea14899e381e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "![Autehntication Toen](images/User_Information.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39c9e2f6-6039-4752-8c24-2e4db32c5a4e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Exercise #6\n",
    "\n",
    "This exercise walks you through downloading a dataset and data files from SciCat and uploading processed data to SciCat.\n",
    "It uses a basic, contrived workflow to process the data using [Scipp](https://scipp.github.io/).  \n",
    "Please use the notebook template for this [exercise](./exercises/exercise_6.ipynb) provided under the exercises folder  \n",
    "\n",
    "Here is the overview of this exercise:\n",
    "1. Find one of the datasets specifically prepared for this workshop:\n",
    "    - acquired at location `ess/dmsc/PeakMeister`\n",
    "    - acquired on instrument `PeakMeister`\n",
    "    - using sample `Dummy sample for the SciCat Workshop at DMSC in spring 2023`\n",
    "    - acquired under proposal `DMSC SciCat Workshop 2023` with id `276577`\n",
    "    - tagged with keywords `SciCat workshop` and `PeakMeister`\n",
    "2. Copy the selected dataset PID\n",
    "3. Start the notebook\n",
    "4. Insert your scicat token\n",
    "5. Insert your dataset PID\n",
    "6. Load the dataset with Scitacean\n",
    "7. Perform the magic analysis\n",
    "8. Create a derived dataset\n",
    "9. Upload the new dataset to scicat"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a0ae894-7630-489c-b644-7f0a1f06b9bf",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Exercise #7 (Extra)\n",
    "Use Scitacean to retrieve the dataset from scicat  \n",
    "You can use the exercise #7 template notebook as a start  \n",
    "\n",
    "1. Retrieve a tomography dataset (Select one of the datasets that you found in Exercise #2)\n",
    "2. Print the dataset information\n",
    "3. Print the dataset metadata\n",
    "4. Retrieve the data file\n",
    "5. Explore the file data structure"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69e76494-578b-442c-8ac2-53a026e58552",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Exercise #8 (Extra)\n",
    "\n",
    "Use pyScicat to query and search datasets, select one and than use Scitacean to load the dataset, download the file and print the structure of the nexus file.  \n",
    "A template is not provided at this time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b225aef-005b-4557-b576-a451d5f978c3",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Thank you\n",
    "\n",
    "Thank you for participating.  \n",
    "Please share your thoughts... kindly!!! We are sensitive human beings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27835677-3185-4e57-bf2a-3abdcb274c24",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}