Skip to content

Commit

Permalink
Merge pull request #70 from sennetconsortium/libpitt/67-assay-docs
Browse files Browse the repository at this point in the history 
Fixing various links and updating APIs page
  • Loading branch information
@maxsibilla
maxsibilla authored Sep 12, 2024
2 parents f3ef3f8 + 36a75fd commit 77f96ba
Show file tree
Hide file tree
Showing 4 changed files with 80 additions and 21 deletions.
67 changes: 63 additions & 4 deletions docs/apis/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,78 @@ layout: default
---

# SenNet APIs
Below is a list of API services which can be used to search and retrieve information about SenNet data.

The following APIs, available as RESTful web services, support data ingest, querying, and delivery of metadata.
Data delivery is available via the [Globus Transfer Service](https://www.globus.org/data-transfer)
and [Globus Transfer API](https://docs.globus.org/api/transfer/). See the [SenNet IDs](#identifiers-used-in-sennet-and-the-apis) section below for
information about how the IDs are used by the SenNet APIs.

## Entity API
Exposes endpoints for creating, updating, and reading standard SenNet provenance graph entities.
The Entity API returns information about SenNet data entities.
- [GitHub](https://github.com/sennetconsortium/entity-api)
- [Smart API](https://smart-api.info/ui/7d838c9dee0caa2f8fe57173282c5812)

## Ingest API
Exposes endpoints for establishing authentication, database connections, and validation.

The Ingest API supports writing data and metadata to SenNet. Tissue Mapping Centers (TMCs) use it to contribute data to
SenNet which is then used to deposit derived data resulting from the execution of pipelines.

- [GitHub](https://github.com/sennetconsortium/ingest-api)
- [Smart API](https://smart-api.info/ui/6d853628150eb1883bacf98690c890d4)

## Search API
Exposes endpoints for indexing and reindexing data in the backend Elasticsearch.

The Search API supports searching and reindexing of SenNet metadata and data. The `/search` endpoint returns sets of data
entities matching specified queries for Sources, Tissue Samples, and Datasets. The `/reindex` endpoint is used internally
to index new and changed entities, this endpoint is not accessible externally, but only from other APIs that create,
update, or delete entities.
<br/><br/>
For a quick guide to query and data download access via the APIs please see
the [Data Query and Download example](/param-search/data-query-download-example.html) and the detailed guide to using
the [Search API Parameter Search feature](/param-search).

- [GitHub](https://github.com/sennetconsortium/search-api)
- [Smart API](https://smart-api.info/ui/10ed9b5eb8ff960d4431befc591ed842)

### UUID API
The UUID API supports donor and tissue sample registration, submission of data, and collection of provenance information via the Ingest UI. The Ingest UI is a web user interface used by the Tissue Mapping Centers (TMCs) when contributing raw and derived data which result from the execution of pipelines.
- [GitHub](https://github.com/x-atlas-consortia/uuid-api)

### Ontology/UBKG API

The Ontology API accesses an instance of a **Unified Biomedical Knowledge Graph** (UBKG), a neo4j knowledge graph that
links infomation from a variety of biomedical vocabulary systems. The SenNet instance of UBKG includes SenNet's *
*application ontology** that represents the SenNet operational model.

- [UBKG](https://ubkg.docs.xconsortia.org/), including links to relevant GitHub repositories
- [Smart API](https://smart-api.info/ui/96e5b5c0b0efeef5b93ea98ac2794837)


# Identifiers used in SenNet and the APIs

SenNet uses three different kinds of identifiers:

### SenNet ID

SenNet IDs are "human readable" identifiers that are used when displaying information about SenNet entities such as Donors, Tissue Samples, Datasets, and Collections.

- Example: `SNT123.ABCD.456`
- Used for identification of SenNet entities and referencing in SenNet context, e.g. in the portal UI, slides, human-human communication, etc.
- These identifiers can be used in the APIs to query portal UI and APIs
- There is a one-to-one mapping between SenNet IDs and UUIDs, with all SenNet IDs guaranteed to having a corresponding UUID, though not all UUIDs have a corresponding SenNet ID.

### UUID
SenNet UUIDs are intended for use internally in software and data storage. They are intended for use by software and systems only and not human readable.

- These ids are 32 digit hexadecimal numbers. Example: `0123456789abcdef0123456789abcdef`
- Used for software implementation and data storage.
- These identifiers can be used to query the APIs and data portal.

### DOI
Digital Object Identifiers (DOIs) are generated for published SenNet data and allow for permanent references outside of SenNet.

- Example: `10.1234/SNT123.ABCD.456`
- Used for referencing outside SenNet context, in particular for use as references in publications.
- Displayed as: `doi:10.1234/SNT.123.ABCD.456`
- Linked to: `https://doi.org/10.1234/SNT.123.ABCD.456`
- Not all SenNet IDs are registered as DOIs, primarily published (public) Datasets and Collections of datasets have DOIs.
8 changes: 4 additions & 4 deletions docs/param-search/data-query-download-example.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ The following query will return all Histology (`dataset_type=Histology`) Dataset
GET https://search.api.sennetconsortium.org/param-search/datasets?dataset_type=Histology&ingest_metadata.metadata.acquisition_instrument_model=BZ-X800&origin_sample.organ=HT
```

As is, if this query is submitted via HTTP GET it will produce a json Response with an array of dataset objects which match the query. Adding the `produce-clt-manifest=true` option to this query will instead produce a list of Dataset IDs pointing to the Datasets that match this query in a format that will be directly usable by the [SenNet Command Line Transfer Tool](../clt/index.html).
As is, if this query is submitted via HTTP GET it will produce a json Response with an array of dataset objects which match the query. Adding the `produce-clt-manifest=true` option to this query will instead produce a list of Dataset IDs pointing to the Datasets that match this query in a format that will be directly usable by the [SenNet Command Line Transfer Tool](/libraries/clt/).

To run this from the command line and save the results to a file run:
```
Expand All @@ -30,13 +30,13 @@ This results in a file that looks like:

```
SNT948.QRZW.946 /
SNT976.WLTL.469 /
SNT976.WLTL.469 /f
...
```

To use the SenNet CLT tool to download the data from these datasets:

- Install the Globus Connect Personal client and the SenNet CLT per the [SenNet CLT Setup Instructions](../clt/install-sennet-clt.html)
- Install the Globus Connect Personal client and the SenNet CLT per the [SenNet CLT Setup Instructions](/libraries/clt/)
- Python 3.9 or greater is required for the SenNet CLT, install from the [Python Downloads page](https://www.python.org/downloads/)
- Setup Note: A common issue arises between the configuration of the GCP client and SenNet CLT. By default, SenNet CLT stores files in the user's home directory under a directory called `sennet-downloads`, so make sure to configure the GCP client by going to "Preferences"-->"Access" and adding the `sennet-downloads` directory in the user's home like (Example shown is Mac OS X):<br/>
<img src="/imgs/globus-properties.jpg" alt="SenNet Provenance" width="400"/>
Expand All @@ -50,4 +50,4 @@ To use the SenNet CLT tool to download the data from these datasets:
sennet-clt transfer dataset-manifest-for-download.out
```

Futher instructions on the usage of the SenNet CLT are available on the main [SenNet Command Line Transfer Tool page](../clt/index.html)
Futher instructions on the usage of the SenNet CLT are available on the main [SenNet Command Line Transfer Tool page](/libraries/clt/)
4 changes: 2 additions & 2 deletions docs/param-search/example-data-query-and-download-jupyter-notebook.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,15 +32,15 @@
"\n",
"### Example Query and Download\n",
"\n",
"The following query will return all Histology (`dataset_type=Histology`) Datasets run on a Keyence BZ-X800 machine (`ingest_metadata.metadata.acquisition_instrument_model=BZ-X800`) where tissue from a heart was used (`origin_sample.organ=HT`). See the [RESTful parameterized search page](https://docs.sennetconsortium.org/index.html) for further information on querying dataset, organ (`origin_sample.organ` represents the organ in the query and `HT` is the organ code (organ code list available [here](https://docs.sennetconsortium.org/schema-sample.html#organ-attribute-values)) and dataset metadata fields.\n",
"The following query will return all Histology (`dataset_type=Histology`) Datasets run on a Keyence BZ-X800 machine (`ingest_metadata.metadata.acquisition_instrument_model=BZ-X800`) where tissue from a heart was used (`origin_sample.organ=HT`). See the [RESTful parameterized search page](https://docs.sennetconsortium.org/index.html) for further information on querying dataset, organ (`origin_sample.organ`) represents the organ in the query and `HT` is the organ code (organ code list available [here](https://docs.sennetconsortium.org/param-search/schema-sample.html#organ-attribute-values)) and dataset metadata fields.\n",
"\n",
"```\n",
"GET https://search.api.sennetconsortium.org/param-search/datasets?dataset_type=Histology&ingest_metadata.metadata.acquisition_instrument_model=BZ-X800&origin_sample.organ=HT\n",
"```\n",
"\n",
"#### Producing a CLT manifest file\n",
"\n",
"As is, if this query is submitted via HTTP GET it will produce a json Response with an array of dataset objects which match the query. Adding the `produce-clt-manifest=true` option to this query will instead prduce a list of Dataset IDs pointing to the Datasets that match this query in a format that will be directly usable by the [SenNet Command Line Transfer Tool](.https://docs.sennetconsortium.org/libraries/clt).\n",
"As is, if this query is submitted via HTTP GET it will produce a json Response with an array of dataset objects which match the query. Adding the `produce-clt-manifest=true` option to this query will instead prduce a list of Dataset IDs pointing to the Datasets that match this query in a format that will be directly usable by the [SenNet Command Line Transfer Tool](https://docs.sennetconsortium.org/libraries/clt/).\n",
"\n",
"```\n",
"GET https://search.api.sennetconsortium.org/param-search/datasets?dataset_type=Histology&ingest_metadata.metadata.acquisition_instrument_model=BZ-X800&origin_sample.organ=HT&produce-clt-manifest=true\n",
Expand Down
22 changes: 11 additions & 11 deletions docs/param-search/schema-sample.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,21 +15,21 @@ GET https://search.api.sennetconsortium.org/param-search/samples?sample_category
## Description:
A query string is built by combining schema elements documented below with matching values. Each "term" of the query is combined using the & character, and the entire query is attached to the base URL after a ? character, per web standards.

Query terms may be composited from attributes deeper in the schema type of an attribute. For example, the Sample Schema attribute ```metadata``` is has a type of [Sample Metadata Schema](libraries/ingest-validation-tools/schemas/), and Sample `Block` Metadata Schema has an attribute ```preparation_condition```. Querying Samples supports a term to search for sample blocks with a preparation condition, such as ```metadata.preparation_condition=Frozen%20on%20ice```, and a query like:
Query terms may be composited from attributes deeper in the schema type of an attribute. For example, the Sample Schema attribute ```metadata``` is has a type of [Sample Metadata Schema](/libraries/ingest-validation-tools/schemas/), and Sample `Block` Metadata Schema has an attribute ```preparation_condition```. Querying Samples supports a term to search for sample blocks with a preparation condition, such as ```metadata.preparation_condition=Frozen%20on%20ice```, and a query like:
```
GET https://search.api.sennetconsortium.org/param-search/samples?metadata.preparation_condition=Frozen%20on%20ice&origin_sample.organ=HT
```

| Attribute | Type | Description |
|-----------------------------|-----------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| uuid | string | The SenNet unique identifier, intended for internal software use only. This is a 32 digit hexadecimal uuid e.g. 461bbfdc353a2673e381f632510b0f17 |
| sennet_id | string | A SenNet Consortium wide unique identifier randomly generated in the format SNT###.ABCD.### for every entity. |
| protocol_url | string | The protocols.io doi url pointing the protocol under wich the sample was obtained and/or prepared. |
| group_name | string | The name of the SenNet data provider group which registered the sample. See the [SenNet Data Provider Groups](data-provider-groups.html) for a list of valid group names. |
| data_access_level | string from [`data_access_level` attribute values](#data_access_level-attribute-values) | One of the values: public, consortium |
| sample_category | string from [`sample_category` attribute values](#sample_category-attribute-values) | A code representing the category of the specimen. Must be one of organ, block, section , suspension. This is a required field. If set to organ, the organ property must be provided as well. |
| organ | string from [`organ` attribute values](#organ-attribute-values) | Organ code specifier, only set if sample_category == organ. Valid values found in the `rui_code` attribute in: [organ types](https://ontology.api.hubmapconsortium.org/organs?application_context=SENNET) |
| metadata | JSON-encoded string for a supported [`Sample Metadata Schema`](/libraries/ingest-validation-tools/schemas/) | The sample specific metadata derived from the uploaded sample_metadata.tsv file. Returned as a json object. |
| Attribute | Type | Description |
|-----------------------------|-----------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| uuid | string | The SenNet unique identifier, intended for internal software use only. This is a 32 digit hexadecimal uuid e.g. 461bbfdc353a2673e381f632510b0f17 |
| sennet_id | string | A SenNet Consortium wide unique identifier randomly generated in the format SNT###.ABCD.### for every entity. |
| protocol_url | string | The protocols.io doi url pointing the protocol under wich the sample was obtained and/or prepared. |
| group_name | string | The name of the SenNet data provider group which registered the sample. See the [SenNet Data Provider Groups](data-provider-groups.html) for a list of valid group names. |
| data_access_level | string from [`data_access_level` attribute values](#data_access_level-attribute-values) | One of the values: public, consortium |
| sample_category | string from [`sample_category` attribute values](#sample_category-attribute-values) | A code representing the category of the specimen. Must be one of organ, block, section , suspension. This is a required field. If set to organ, the organ property must be provided as well. |
| organ | string from [`organ` attribute values](#organ-attribute-values) | Organ code specifier, only set if sample_category == Organ. Valid values found in the `rui_code` attribute in: [organ types](https://ontology.api.hubmapconsortium.org/organs?application_context=SENNET) |
| metadata | JSON-encoded string for a supported [`Sample Metadata Schema`](/libraries/ingest-validation-tools/schemas/) | The sample specific metadata derived from the uploaded sample_metadata.tsv file. Returned as a json object. |

### `data_access_level` attribute values
The data_access_level of the `Sample Schema` is one of the values following enumerated values:
Expand Down

0 comments on commit 77f96ba

Please sign in to comment.