diff --git a/bycon/config/entity_defaults.yaml b/bycon/config/entity_defaults.yaml index 69c4c7d18..31264a158 100644 --- a/bycon/config/entity_defaults.yaml +++ b/bycon/config/entity_defaults.yaml @@ -125,7 +125,7 @@ individual: The type of response used for the endpoint depends on the requested and granted `responseGranularity`, _i.e._ while a "record" level `responseGranularity` has to be served through a `beaconResultsetsResponse` other granularities - might employ aggregate response types. + might employ summary response types. request_entity_path_id: individuals response_entity_id: individual collection: individuals diff --git a/docs/API-beacon.md b/docs/API-beacon.md new file mode 100644 index 000000000..20576cfa4 --- /dev/null +++ b/docs/API-beacon.md @@ -0,0 +1,20 @@ +# API: Beacon Responses + +The following is a list of standard Beacon responses supported by the `bycon` package. +Responses for individual entities or endpoints are grouped by their Beacon framework +response classes (e.g. `beaconResultsetsResponse` for `biosamples`, `g_variants` etc.).\n\n +Please be reminded about the general syntax used in Beacon: A **path element** such +as `/biosamples` corresponds to an entity (here `biosample`). Below these relations +are indicated by the `@` symbol.\n\n + +!!! info "API Parameters" + A complete list of parameters accepted by the API is provided on the [_Web and Command Line Parameters_](API-parameters.md) page. + +#### Schemas **{S}**, Tests **{T}** and Examples **{E}** +Tests, examples and schemas are run from the server defined in this site's build instructions +(see the `reference_server_url` entry in `mkdocs.yaml` file in the repository's root). + +{% + include-markdown "generated/beacon-responses.md" +%} + diff --git a/docs/API-parameters.md b/docs/API-parameters.md new file mode 100644 index 000000000..0d916646a --- /dev/null +++ b/docs/API-parameters.md @@ -0,0 +1,32 @@ +# API: Web and Command Line Parameters + +## General API Parameters + +The `bycon` package supports number of parameters for filtering and querying the data which includes and extends +standard Beacon parameters and can - depending on the parameter and the current scope - in general be +invoked through HTTP requests and as command line arguments. + +Only parameters sefined in `config/argument_definitions.yaml` will be +interpreted. + +Parameters are listed in `snake_case` format although for command line arguments +(and also optionally web requests) `camelCase` versions are required (see the `cmdFlags`). + + +{% + include-markdown "generated/argument_definitions.md" +%} + +## Services: Plot Parameters + +The `byconServices` package inside `bycon` provides a number of plotting functions which can be used to visualize the data in the database. Generally +plot functionality is focussed on generating CNV visualizations for per-sample and +aggregated CNV data (e.g. frequencyplots). Additionally some geographic map projectins are provided e.g. for samples and metadata. + +More information can be found on these pages: + +* plot documentation on [this page](plotting.md) + +{% + include-markdown "generated/plot_defaults.md" +%} diff --git a/docs/API-services.md b/docs/API-services.md new file mode 100644 index 000000000..cccba65b6 --- /dev/null +++ b/docs/API-services.md @@ -0,0 +1,58 @@ +# API: Beacon Services + +The _bycon_ environment provides a number of data services beyond typical Beacon +functionality. These services can be loosely grouped into two major types: + +* services which extend Beacon formats, _i.e._ use Beacon concepts and query + parameters but provide non-standard output + - plotting functions + - tabular and text output + - aggregated data such as CNV frequencies (see [_intervalFrequencies_](services/intervalFrequencies.md)) +* services which make use of utility function and data existing primarily for the + support of Beacon functionality + - ontology term cross mapping + - geneId to location lookup + - geographic location mapping and map projections (see [_geolocations_](services/geolocations.md)) + - ISCN cytogenetic band mapping + - publication data for cancer genome screening publications + +!!! info "API Parameters" + A complete list of parameters accepted by the API is provided on the [_Web and Command Line Parameters_](API-parameters.md) page. + +## Response formats + +Standard responses are provided as `Content-Type: application/json`. The wrapper +format is based on the Beacon v2 response format, with the data returned in the +`results` array: + +``` +meta: + api_version: ... + returned_schemas: [ ] +response: + exists: true | false + info: { } + resultSets: [ ] +``` + +This (incomplete) example response may help with understanding the general +format. Here, the data is a dictionary/object with a single key (`genes`): + +#### Request Example + +* + +## Plotting + +The `byconServices` package inside `bycon` provides a number of plotting functions which can be used to visualize the data in the database. Generally +plot functionality is focussed on generating CNV visualizations for per-sample and +aggregated CNV data (e.g. frequencyplots). Additionally some geographic map projectins are provided e.g. for samples and metadata. + +More information can be found in the plot documentation on [this page](plotting.md). + +## List of Services + +{% + include-markdown "generated/beacon-services.md" +%} + diff --git a/docs/API.md b/docs/API.md index 69b602f99..28557a8a2 100644 --- a/docs/API.md +++ b/docs/API.md @@ -1,9 +1,10 @@ # API Documentation -## Beacon Responses [:link:](/generated/beacon-responses) +{% + include-markdown "generated/beacon-responses.md" +%} - -## Service Endpoints [:link:](/generated/services) +## Service Endpoints [:link:](/generated/services.md) Additionally to the standard Beacon endpoints provided for the Beacon framework and default data model `bycon` also has a number of service endpoints. Some of them are used to provide direct Beacon support (e.g. `/endpoints` or `/schemas` to implement endpoints accessed from Beacon responses), while others @@ -11,9 +12,9 @@ provide more specialized functionality (e.g. `/cytomapper`). More information: -* [list of bycon services](/generated/services) +* [list of bycon services](/generated/services.md) -## Parameters [... more](/generated/argument_definitions) +## Parameters [... more](/generated/argument_definitions.md) The `bycon` package supports number of parameters for filtering and querying the data which includes and extends standard Beacon parameters and can - depending on the parameter and the current scope - in general be @@ -22,7 +23,7 @@ invoked through HTTP requests and as command line arguments. Only parameters sefined in `config/argument_definitions.yaml` will be interpreted. Those parameters are listed in [this document](/generated/argument_definitions). -## Endpoint Tests [... more](/tests) +## Endpoint Tests [... more](/tests.md) ==TBD== @@ -34,5 +35,5 @@ aggregated CNV data (e.g. frequencyplots). Additionally some geographic map proj More information can be found on these pages: -* plot documentation on [this page](/plotting) -* plot [parameters and defaults](/generated/plot_defaults) +* plot documentation on [this page](./plotting.md) +* plot [parameters and defaults](./generated/plot_defaults.md) diff --git a/docs/generated/argument_definitions.md b/docs/generated/argument_definitions.md index 9baa8c69c..b0996dc12 100644 --- a/docs/generated/argument_definitions.md +++ b/docs/generated/argument_definitions.md @@ -1,31 +1,26 @@ -# `bycon` Arguments and Parameters -The following is a list of arguments and parameters used in the `bycon` package -as well as the `byconaut` tools. -Parameters are listed in `snake_case` format although for command line arguments -(and also optionally web requests) `camelCase` versions are required (see the `cmdFlags`). -## Definitions -### `user_name` +### Definitions +#### `user_name` **type:** string **pattern:** `^\w+$` **cmdFlags:** `--userName` **description:** faking a user name -### `test_mode` +#### `test_mode` **type:** boolean **cmdFlags:** `-t,--testMode` **description:** test setting, i.e. returning some random documents **default:** `False` -### `skip` +#### `skip` **type:** integer **cmdFlags:** `--skip` **description:** pages to be skipped **default:** `0` -### `limit` +#### `limit` **type:** integer **cmdFlags:** `-l,--limit` **description:** @@ -33,7 +28,7 @@ limit number of documents; a value of 0 sets to unlimited **default:** `200` **local:** 0 -### `requested_granularity` +#### `requested_granularity` **type:** string **pattern:** `^\w+$` **description:** @@ -41,7 +36,7 @@ The requested granularity of the beacon **cmdFlags:** `--requestedGranularity` **default:** `record` -### `request_entity_path_id` +#### `request_entity_path_id` **type:** string **cmdFlags:** `--requestEntityPathId` **description:** @@ -50,7 +45,7 @@ The requested granularity of the beacon * this only is used for command-line tests instead of the REST path value seen by the stack in web server mode -### `response_entity_path_id` +#### `response_entity_path_id` **type:** string **cmdFlags:** `--responseEntityPathId` **description:** @@ -60,7 +55,7 @@ The requested granularity of the beacon a special response entity in byconaut services (e.g. `indviduals` in `/sampletable/`) -### `include_resultset_responses` +#### `include_resultset_responses` **type:** string **cmdFlags:** `--includeResultsetResponses` **description:** @@ -70,7 +65,7 @@ The requested granularity of the beacon some use in networks **default:** `HIT` -### `dataset_ids` +#### `dataset_ids` **type:** array **items:** - `type`: `string` @@ -78,7 +73,7 @@ The requested granularity of the beacon **description:** dataset ids -### `cohort_ids` +#### `cohort_ids` **type:** array **items:** - `type`: `string` @@ -86,7 +81,7 @@ dataset ids **description:** cohort ids -### `filters` +#### `filters` **type:** array **items:** - `type`: `string` @@ -94,28 +89,28 @@ cohort ids **description:** prefixed filter values, comma concatenated; or objects in POST -### `filter_precision` +#### `filter_precision` **type:** string **cmdFlags:** `--filterPrecision` **description:** either `start` or `exact` for matching filter values **default:** `exact` -### `filter_logic` +#### `filter_logic` **type:** string **cmdFlags:** `--filterLogic` **description:** Global for either OR or AND (translated to the MongoDB $and etc.). The Beacon protocol only knows AND. **default:** `AND` -### `include_descendant_terms` +#### `include_descendant_terms` **type:** boolean **cmdFlags:** `--includeDescendantTerms` **description:** global treatment of descendant terms **default:** `True` -### `assembly_id` +#### `assembly_id` **type:** string **pattern:** `^\w+?[\w\-\.]*?\w*?$` **db_key:** assembly_id @@ -123,7 +118,7 @@ global treatment of descendant terms **description:** assembly id; currently not used in bycon's version -### `reference_name` +#### `reference_name` **type:** string **db_key:** location.sequence_id **pattern:** `^\w+.*?\w?$` @@ -131,7 +126,7 @@ assembly id; currently not used in bycon's version **description:** chromosome -### `mate_name` +#### `mate_name` **type:** string **db_key:** adjoined_sequences.sequence_id **pattern:** `^\w+.*?\w?$` @@ -139,7 +134,7 @@ chromosome **description:** chromosome -### `reference_bases` +#### `reference_bases` **type:** string **db_key:** reference_sequence **pattern:** `^[ACGTN]+$` @@ -147,7 +142,7 @@ chromosome **description:** reference bases -### `alternate_bases` +#### `alternate_bases` **type:** string **db_key:** sequence **pattern:** `^[ACGTN]+$` @@ -155,7 +150,7 @@ reference bases **description:** alternate bases -### `variant_type` +#### `variant_type` **type:** string **db_key:** variant_state.id **pattern:** `^\w+[\w \-\:]\w+?$` @@ -163,7 +158,7 @@ alternate bases **description:** variant type, e.g. DUP -### `start` +#### `start` **type:** array **db_key:** location.start **items:** @@ -173,7 +168,7 @@ variant type, e.g. DUP **description:** genomic start position -### `end` +#### `end` **type:** array **db_key:** location.end **items:** @@ -183,7 +178,7 @@ genomic start position **description:** genomic end position -### `mate_start` +#### `mate_start` **type:** integer **db_key:** adjoined_sequences.start **pattern:** `^\d+?$` @@ -191,7 +186,7 @@ genomic end position **description:** genomic start position of fusion partner breakpoint region -### `mate_end` +#### `mate_end` **type:** integer **db_key:** adjoined_sequences.end **pattern:** `^\d+?$` @@ -199,7 +194,7 @@ genomic start position of fusion partner breakpoint region **description:** genomic end position of fusion partner breakpoint region -### `variant_min_length` +#### `variant_min_length` **type:** integer **db_key:** info.var_length **pattern:** `^\d+?$` @@ -207,7 +202,7 @@ genomic end position of fusion partner breakpoint region **description:** The minimal variant length in bases e.g. for CNV queries. -### `variant_max_length` +#### `variant_max_length` **type:** integer **db_key:** info.var_length **pattern:** `^\d+?$` @@ -215,7 +210,7 @@ The minimal variant length in bases e.g. for CNV queries. **description:** The maximum variant length in bases e.g. for CNV queries. -### `gene_id` +#### `gene_id` **type:** array **items:** - `type`: `string` @@ -225,7 +220,7 @@ The maximum variant length in bases e.g. for CNV queries. **description:** gene id -### `aminoacid_change` +#### `aminoacid_change` **type:** string **db_key:** molecular_attributes.aminoacid_changes **pattern:** `^\w+?$` @@ -234,7 +229,7 @@ gene id **description:** Aminoacid alteration in 1 letter format -### `genomic_allele_short_form` +#### `genomic_allele_short_form` **type:** string **db_key:** identifiers.genomicHGVS_id **pattern:** `^\w+.*\w$` @@ -243,7 +238,7 @@ Aminoacid alteration in 1 letter format **description:** Genomic HGVSId descriptor -### `variant_query_digests` +#### `variant_query_digests` **type:** array **db_key:** None **items:** @@ -254,7 +249,7 @@ Genomic HGVSId descriptor **description:** EXPERIMENTAL Variant query digest-style short form -### `variant_multi_pars` +#### `variant_multi_pars` **type:** array **db_key:** None **items:** @@ -263,7 +258,7 @@ EXPERIMENTAL Variant query digest-style short form **description:** EXPERIMENTAL List of multiple variant queries, for POST -### `variant_internal_id` +#### `variant_internal_id` **type:** string **db_key:** variant_internal_id **pattern:** `^\w[\w\:\-\,]+?\w$` @@ -272,7 +267,7 @@ EXPERIMENTAL List of multiple variant queries, for POST **description:** An id value used for all variant instances of the same composition; a kind of `digest` -### `accessid` +#### `accessid` **type:** string **db_key:** id **pattern:** `^\w[\w\-]+?\w$` @@ -281,7 +276,7 @@ An id value used for all variant instances of the same composition; a kind of `d **description:** An accessid for retrieving handovers etc. -### `file_id` +#### `file_id` **type:** string **pattern:** `^\w[\w\-]+?\w$` **examples:** `90e19951-1443-4fa8-8e0b-6b5d8c5e45cc` @@ -289,7 +284,7 @@ An accessid for retrieving handovers etc. **description:** A file id e.g. as generated by the uploader service -### `id` +#### `id` **type:** string **db_key:** id **pattern:** `^\w[\w\:\-\,]+?\w$` @@ -298,7 +293,7 @@ A file id e.g. as generated by the uploader service **description:** An id; this parameter only makes sense for specific REST entry types -### `ids` +#### `ids` **type:** array **items:** - `type`: `string` @@ -307,7 +302,7 @@ An id; this parameter only makes sense for specific REST entry types **description:** One or more ids; this parameter only makes sense for specific REST entry types -### `biosample_ids` +#### `biosample_ids` **type:** array **items:** - `type`: `string` @@ -317,7 +312,7 @@ One or more ids; this parameter only makes sense for specific REST entry types **description:** biosample ids -### `analysis_ids` +#### `analysis_ids` **type:** array **items:** - `type`: `string` @@ -327,7 +322,7 @@ biosample ids **description:** analysis ids -### `individual_ids` +#### `individual_ids` **type:** array **items:** - `type`: `string` @@ -337,7 +332,7 @@ analysis ids **description:** subject ids -### `variant_ids` +#### `variant_ids` **type:** array **items:** - `type`: `string` @@ -347,28 +342,28 @@ subject ids **description:** variant ids -### `debug_mode` +#### `debug_mode` **type:** boolean **cmdFlags:** `--debugMode` **description:** debug setting **default:** `False` -### `show_help` +#### `show_help` **type:** boolean **cmdFlags:** `--showHelp` **description:** specific help display **default:** `False` -### `test_mode_count` +#### `test_mode_count` **type:** integer **cmdFlags:** `--testModeCount` **description:** setting the number of documents reurned in test mode **default:** `5` -### `output` +#### `output` **type:** string **cmdFlags:** `--output` **description:** @@ -379,73 +374,73 @@ For defining a special output format, mostly for `byconaut` services use. Exampl * `text`, for some services to deliver a text table instead of JSON * in byconaut for the target database when copying... -### `include_handovers` +#### `include_handovers` **type:** boolean **default:** `True` **cmdFlags:** `--includeHandovers` **description:** only used for web requests & testing -### `method` +#### `method` **type:** string **cmdFlags:** `--method` **description:** special method **default:** `None` -### `group_by` +#### `group_by` **type:** string **cmdFlags:** `--groupBy` **description:** group parameter e.g. for subset splitting **default:** `text` -### `mode` +#### `mode` **type:** string **cmdFlags:** `-m,--mode` **description:** mode, e.g. file type -### `update` +#### `update` **type:** boolean **cmdFlags:** `-u,--update` **description:** update existing records - might be deprecated; only used for publications **default:** `False` -### `force` +#### `force` **type:** boolean **cmdFlags:** `--force` **description:** force mode, e.g. for update or insert (cmd line) **default:** `False` -### `inputfile` +#### `inputfile` **type:** string **cmdFlags:** `-i,--inputfile` **description:** a custom file to specify input data, usually tab-delimited with special header -### `outputdir` +#### `outputdir` **type:** string **cmdFlags:** `--outputdir` **description:** output directory where supported (cmd line) -### `outputfile` +#### `outputfile` **type:** string **cmdFlags:** `-o,--outputfile` **description:** output file where supported (cmd line) -### `min_number` +#### `min_number` **type:** integer **cmdFlags:** `--minNumber` **description:** minimal number, e.g. for collations, where supported **default:** `0` -### `delivery_keys` +#### `delivery_keys` **type:** array **items:** - `type`: `string` @@ -453,7 +448,7 @@ minimal number, e.g. for collations, where supported **description:** delivery keys to force only some parameters in custom exporters -### `collation_types` +#### `collation_types` **type:** array **items:** - `type`: `string` @@ -461,19 +456,19 @@ delivery keys to force only some parameters in custom exporters **description:** selected collation types, e.g. "EFO" -### `selected_beacons` +#### `selected_beacons` **type:** array **items:** - `type`: `string` -### `genome_binning` +#### `genome_binning` **type:** string **default:** `1Mb` **cmdFlags:** `--genomeBinning` **description:** one of the predefined genome binning keys - default 1Mb -### `cyto_bands` +#### `cyto_bands` **type:** string **pattern:** `^(?:chro?)?([12]?[\dXY])([pq](?:(?:ter)|(?:cen)|(?:[1-4](?:\d(?:\.\d\d*?)?)?)?))?\-?([pq](?:(?:cen)|(?:ter)|(?:[1-4](?:\d(?:\.\d\d*?)?)?)?))?$` **db_key:** None @@ -481,7 +476,7 @@ one of the predefined genome binning keys - default 1Mb **description:** cytobands, e.g. 8q21q24.1 -### `chro_bases` +#### `chro_bases` **type:** string **pattern:** `^(chro?)?([12]?[\dXY])\:(\d+?)(\-(\d+?))?$` **db_key:** None @@ -489,37 +484,37 @@ cytobands, e.g. 8q21q24.1 **description:** only for the cytoband converter ... e.g. 8:0-120000000 -### `city` +#### `city` **type:** string **cmdFlags:** `-c,--city` **description:** only for the geolocations... -### `geo_latitude` +#### `geo_latitude` **type:** number **cmdFlags:** `--geoLatitude` **description:** only for the geolocations... -### `geo_longitude` +#### `geo_longitude` **type:** number **cmdFlags:** `--geoLongitude` **description:** only for the geolocations... -### `geo_distance` +#### `geo_distance` **type:** integer **cmdFlags:** `--geoDistance` **description:** only for the geolocations... -### `plot_pars` +#### `plot_pars` **type:** string **cmdFlags:** `--plotPars` **description:** plot parameters in form `par=value` concatenated by `::` -### `plot_type` +#### `plot_type` **type:** string **cmdFlags:** `--plotType` **description:** diff --git a/docs/generated/beacon-responses.md b/docs/generated/beacon-responses.md index 325f4dda4..0d19b27c9 100644 --- a/docs/generated/beacon-responses.md +++ b/docs/generated/beacon-responses.md @@ -1,20 +1,3 @@ -# Beacon Responses - -The following is a list of standard Beacon responses supported by the `bycon` package. -Responses for individual entities or endpoints are grouped by their Beacon framework -response classes (e.g. `beaconResultsetsResponse` for `biosamples`, `g_variants` etc.). - - -Please be reminded about the general syntax used in Beacon: A **path element** such -as `/biosamples` corresponds to an entity (here `biosample`). Below these relations -are indicated by the `@` symbol. - - -#### Schemas **{S}**, Tests **{T}** and Examples **{E}** -Tests, examples and schemas are run from the server defined in this site's build instructions -(see the `reference_server_url` entry in `mkdocs.yaml` file in the repository's root). - - ## beaconBooleanResponse Complete definition for a minimal response that provides *only* an aggregate Boolean `"exists": true` or `"exists": false` answer to the query. @@ -116,7 +99,7 @@ The type of response used for the endpoint depends on the requested and granted None -The type of response used for the endpoint depends on the requested and granted `responseGranularity`, _i.e._ while a "record" level `responseGranularity` has to be served through a `beaconResultsetsResponse` other granularities might employ aggregate response types. +The type of response used for the endpoint depends on the requested and granted `responseGranularity`, _i.e._ while a "record" level `responseGranularity` has to be served through a `beaconResultsetsResponse` other granularities might employ summary response types. diff --git a/docs/generated/beacon-services.md b/docs/generated/beacon-services.md new file mode 100644 index 000000000..59f18a427 --- /dev/null +++ b/docs/generated/beacon-services.md @@ -0,0 +1,268 @@ +### `/byconschemas` + +This helper service reads and serves local schema definition files. The name of +the schema (corresponding to the file name minus extension) is provided either +as an `id` query parameter or as the first part of the path after `schemas/`. + +* + + +### `/cnvstats` + +==TBD== + + +### `/collationplots` + +The `collationplots` function is a service to provide plots for CNV data aggregated +for samples matching individual filter values such as diagnostic codes or experimental +series id values. The default response is an SVG histogram ("histoplot"). Please refer +to the plot parameters documentation and the `ByconPlot` class for other options. + +For a single plot one can provide the entity id as path id value. + +#### Examples (using the Progenetix resource as endpoint): + +* https://progenetix.org/services/collationplots/pgx:cohort-TCGAcancers +* https://progenetix.org/services/collationplots/?filters=NCIT:C7376,PMID:22824167,pgx:icdom-85003 +* https://progenetix.org/services/collationplots/?filters=NCIT:C7376,PMID:22824167&plotType=histoheatplot +* https://progenetix.org/services/collationplots/?collationTypes=icdom&minNumber=1000&plotType=histoheatplot + + +### `/collations` + +The `collations` service provides access to information about data "subsets" +in the project databases. Collations typically are aggregations of samples +sharing an ontology code (e.g. NCIT) or external identifier (e.g. PMID). Therefore, +in the context of Beacon the collations in `bycon` provide the `filtering_terms` +available through Beacon queries, but also additional information e.g. about +child terms and statistics related to the terms. + +In the case of the web projects the main purpose of the `services/collations/ +endpoin is in providing the child terms and path relations for generating ontology +trees in the UI. + +### Parameters + +* `collationTypes=...` +* `includeDescendantTerms=false` + - only delivers data about codes with direct matches, i.e. excluding such + where only a child term had a direct match + - this is especially useful for e.g. getting a fast overview about mappings + of deeply nested coding systems like `NCIT` +* `deliveryKeys=...` + +### Examples + +* +* +* + + +### `/cytomapper` + +The `cytomapper` function provides a JSON response with cytoband information +such as matched cytobands and the genome coordinates of their extend. + +There is **currently only support for GRCh38**. + +#### Response Schema + +* + +#### Parameters + +* `cytoBands` (path default) + - a properly formatted cytoband annotation + - "8", "9p11q21", "8q", "1p12qter" +* or `chroBases` + - `7:23028447-45000000` + - `X:99202660` + +#### Examples (using the Progenetix resource as endpoint): + +* https://progenetix.org/services/cytomapper/8q21q24 +* https://progenetix.org/services/cytomapper/13q +* https://progenetix.org/services/cytomapper?chroBases=12:10000000-45000000 + + +### `/dbstats` + +This service endpoint provides statistic information about the resource's +datasets. + +#### Examples + +* +* + + +### `/endpoints` + +The service provides the schemas for the `BeaconMap` OpenAPI endpoints. + +#### Examples (using the Progenetix resource as endpoint): + +* +* + + +### `/genespans` + +The `genespans` function provides a JSON response with the coordinates of +matching gene IDs. + +#### Examples (using the Progenetix resource as endpoint): + +* https://progenetix.test/services/genespans/MYC +* https://progenetix.test/services/genespans/?geneId=MYC + + +### `/geolocations` + +==TBD== + +#### Examples + +* +* +* +* +* + + +### `/ids` + +The `ids` service forwards compatible, prefixed ids (see `config/ids.yaml`) to specific +website endpoints. There is no check if the id exists; this is left to the web +page handling itself. + +Stacking with the "pgx:" prefix is allowed. + +#### Examples (using the Progenetix resource as endpoint): + +* +* +* + + +### `/intervalFrequencies` + +None + + +### `/ontologymaps` + +#### Examples + +* + + +### `/pgxsegvariants` + +None + + +### `/publications` + +The _publications_ service provides API functionality for accessing the +Progenetix [publications](http://progenetix.org/publications/) collection, which +represents curated information about several thousand articles reporting +genome-wide screening experiments in cancer. + +#### Examples + +* +* +* +* + + +### `/samplemap` + +==TBD== + + +### `/samplematrix` + +The service uses the standard bycon data retrieval pipeline with `analysis` +as entity type. Therefore, all standard Beacon query parameters work and also +the path is interpreted for an biosample `id` value if there is an entry at +`.../biosamples/{id}` + + +### `/sampleplots` + +The plot service uses the standard bycon data retrieval pipeline with `biosample` +as entity type. Therefore, all standard Beacon query parameters work and also +the path is interpreted for an biosample `id` value if there is an entry at +`.../sampleplots/{id}` + +The plot type can be set with `plotType=samplesplot` (or `histoplot` but that is +the fallback). Plot options are available as usual. + +#### Examples (using the Progenetix resource as endpoint): + +* http://progenetix.org/services/sampleplots/pgxbs-kftvjv8w +* http://progenetix.org/services/sampleplots/pgxbs-kftvjv8w?plotType=samplesplot&datasetIds=cellz +* http://progenetix.org/services/sampleplots?plotType=samplesplot&datasetIds=cellz&filters=cellosaurus:CVCL_0030 +* http://progenetix.org/services/sampleplots?filters=pgx:icdom-81703 +* http://progenetix.org/services/sampleplots/?testMode=true&plotType=samplesplot +* http://progenetix.org/services/sampleplots?filters=pgx:icdom-81703&plotType=histoplot&plotPars=plot_chro_height=0::plot_title_font_size=0::plot_area_height=18::plot_margins=0::plot_axislab_y_width=0::plot_grid_stroke=0::plot_footer_font_size=0::plot_width=400 +* http://progenetix.org/services/sampleplots?datasetIds=progenetix&plotMinLength=1000&plotMaxLength=3000000&geneId=CDKN2A&variantType=EFO:0020073&plotPars=plotChros=9::plotGeneSymbols=CDKN2A::plotWidth=300&plotType=histoplot + + +### `/sampletable` + +The service uses the standard bycon data retrieval pipeline with `biosample` +as entity type. Therefore, all standard Beacon query parameters work and also +the path is interpreted for an biosample `id` value if there is an entry at +`.../sampletable/{id}` + +The table type can be changed with `tableType=individuals` (or `analyses`). + +#### Examples + +* http://progenetix.org/services/sampletable/pgxbs-kftvjv8w +* http://progenetix.org/services/sampletable?datasetIds=cellz&filters=cellosaurus:CVCL_0030 +* http://progenetix.org/services/sampletable?filters=pgx:icdom-81703 + + +### `/services` + +The `services` application deparses a request URI and calls the respective +script. The functionality is combined with the correct configuration of a +rewrite in the server configuration for creation of canonical URLs. + + +### `/uploader` + +This service is used by UI implementations to upload user provided `.pgxseg` files +for visualization of the variants using the packages plotting functions. + +As exception to the general rule the `uploader` service does not make use of standard +argument parsing but directly uses `cgi.FieldStorage()` and `....file.read()`. + + +### `/variantsbedfile` + +The `variantsbedfile` function provides a BED file with the matched genomic +variants from a Beacon query or a sample id. + +#### Examples + +* http://progenetix.org/services/variantsbedfile/pgxbs-kftvjv8w + + +### `/vcfvariants` + +The VCF service uses the standard bycon data retrieval pipeline with `biosample` +as entity type. Therefore, all standard Beacon query parameters work and also +the path is interpreted for an biosample `id` value if there is an entry at +`.../vcfvariants/{id}` + +#### Examples + +* http://progenetix.org/services/vcfvariants/pgxbs-kftvjv8w + + diff --git a/docs/generated/plot_defaults.md b/docs/generated/plot_defaults.md index e4e65a715..aefb2e188 100644 --- a/docs/generated/plot_defaults.md +++ b/docs/generated/plot_defaults.md @@ -1,18 +1,17 @@ -# Plot Parameters and Information -## Plot Types -### `histoplot` +### Plot Types +#### `histoplot` **description:** The default option, used to plot histograms of the CNV frequencies per data collection ("collation") or aggregated sample data. **data_key:** interval_frequencies_bundles **data_type:** collations -### `histoheatplot` +#### `histoheatplot` **description:** A "heatmap" style version of the histogram plot, where a single gain/loss frequency result is transformed into a small heat color strip. **data_key:** interval_frequencies_bundles **data_type:** collations -### `histosparkplot` +#### `histosparkplot` **description:** A version of the histogram with predefined parameters for representing a small and unlabeled plot, e.g. for use in hover previews. As in the normal histogram parameters can be overridden. **data_key:** interval_frequencies_bundles @@ -32,7 +31,7 @@ A version of the histogram with predefined parameters for representing a small a - `plot_axis_y_max`: `80` **modded:** histoplot -### `histocircleplot` +#### `histocircleplot` **description:** A version circular of the histogram. **data_key:** interval_frequencies_bundles @@ -45,50 +44,50 @@ A version circular of the histogram. - `plot_dendrogram_width`: `0` - `plot_labelcol_width`: `0` -### `samplesplot` +#### `samplesplot` **description:** A plot of the called CNV segments per sample, with the samples ordered by their clustering (_i.e._ similarity of binned CNV data). **data_key:** analyses_variants_bundles **data_type:** samples -### `geomapplot` +#### `geomapplot` **description:** A leaflet based plot of geolocations. **data_key:** geolocs_list **data_type:** geolocs -## Plot Parameters -### `plot_id` +### Plot Parameters +#### `plot_id` **default:** `genomeplot` -### `plot_title` +#### `plot_title` **description:** title above the plot -### `plot_group_by` +#### `plot_group_by` **description:** group samples in histograms by a filter type (NCIT, PMID...) **default:** `` -### `plot_filter_empty_samples` +#### `plot_filter_empty_samples` **description:** By setting to `true` samples w/o data can be removed e.g. from sample plots **type:** boolean **default:** `False` -### `force_empty_plot` +#### `force_empty_plot` **description:** By setting to `true` a plot strip will be forced even if there are no CNV samples **type:** boolean **default:** `False` -### `plot_cluster_results` +#### `plot_cluster_results` **description:** By setting to `false` clustering can be suppressed **type:** boolean **default:** `True` -### `plot_samples_cluster_type` +#### `plot_samples_cluster_type` **description:** Selection of which measurees are used to generate the clustering matrix @@ -98,34 +97,34 @@ Selection of which measurees are used to generate the clustering matrix **default:** `intcoverage` **oneOf:** `chrostats,intcoverage` -### `plot_cluster_metric` +#### `plot_cluster_metric` **default:** `ward` **oneOf:** `average,centroid,complete,median,single,ward,weighted` -### `plot_dup_color` +#### `plot_dup_color` **default:** `#FFC633` -### `plot_hldup_color` +#### `plot_hldup_color` **default:** `#FF6600` -### `plot_del_color` +#### `plot_del_color` **default:** `#33A0FF` -### `plot_hldel_color` +#### `plot_hldel_color` **default:** `#0033CC` -### `plot_loh_color` +#### `plot_loh_color` **default:** `#0066FF` -### `plot_snv_color` +#### `plot_snv_color` **default:** `#FF3300` -### `plot_chros` +#### `plot_chros` **type:** array **items:** string **default:** `1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22` -### `plot_width` +#### `plot_width` **description:** * width of the plot image, in px @@ -135,13 +134,13 @@ Selection of which measurees are used to generate the clustering matrix **type:** integer **default:** `1024` -### `plot_area_height` +#### `plot_area_height` **description:** height of the plot area (applies only to histogram plots) **type:** integer **default:** `100` -### `plot_axis_y_max` +#### `plot_axis_y_max` **description:** * frequency value the maximum of the Y-axis corresponds to @@ -149,19 +148,19 @@ height of the plot area (applies only to histogram plots) **type:** integer **default:** `100` -### `plot_samplestrip_height` +#### `plot_samplestrip_height` **description:** height of a single sample strip **type:** integer **default:** `12` -### `plot_margins` +#### `plot_margins` **description:** outer plot margins, in px **type:** integer **default:** `25` -### `plot_labelcol_width` +#### `plot_labelcol_width` **description:** * width of the space for left text labels (e.g. sample ids, collation @@ -171,13 +170,13 @@ outer plot margins, in px **type:** integer **default:** `220` -### `plot_axislab_y_width` +#### `plot_axislab_y_width` **description:** width of the space for histogram percentage markers **type:** integer **default:** `30` -### `plot_dendrogram_width` +#### `plot_dendrogram_width` **description:** * width of the cluster tree @@ -185,135 +184,135 @@ width of the space for histogram percentage markers **type:** integer **default:** `50` -### `plot_dendrogram_color` +#### `plot_dendrogram_color` **description:** color of the cluster tree stroke **default:** `#333333` -### `plot_dendrogram_stroke` +#### `plot_dendrogram_stroke` **description:** thickness of the cluster tree stroke **type:** number **default:** `0.5` -### `plot_chro_height` +#### `plot_chro_height` **description:** height (well, width...) of the chromosomes in the ideogram strip **type:** integer **default:** `14` -### `plot_region_gap_width` +#### `plot_region_gap_width` **type:** integer **default:** `3` -### `plot_canvas_color` +#### `plot_canvas_color` **description:** color of the document background **default:** `#ffffff` -### `plot_area_color` +#### `plot_area_color` **description:** color of the plot area background **default:** `#eef6ff` -### `plot_area_opacity` +#### `plot_area_opacity` **description:** opacity of the plot background **type:** number **default:** `0.8` -### `plot_heat_intensity` +#### `plot_heat_intensity` **description:** factor for frequency heatmap value brightness **type:** number **default:** `1` -### `plot_grid_stroke` +#### `plot_grid_stroke` **type:** integer **default:** `1` -### `plot_grid_color` +#### `plot_grid_color` **description:** color of grid lines **default:** `#c0e3ee` -### `plot_grid_opacity` +#### `plot_grid_opacity` **type:** float **default:** `0.8` -### `plot_font_color` +#### `plot_font_color` **default:** `#000000` -### `plot_font_size` +#### `plot_font_size` **description:** font size, in px **type:** integer **default:** `10` -### `plot_title_font_size` +#### `plot_title_font_size` **description:** title font size, in px **type:** integer **default:** `16` -### `plot_labelcol_font_size` +#### `plot_labelcol_font_size` **description:** label font size (left column), in px **type:** integer **default:** `12` -### `plot_label_y_font_size` +#### `plot_label_y_font_size` **description:** font size for Y-axis labels (percents ...) **type:** integer **default:** `8` -### `plot_label_y_font_color` +#### `plot_label_y_font_color` **description:** font color for Y-axis labels (percents ...) **default:** `#666666` -### `plot_label_y_values` +#### `plot_label_y_values` **type:** array **items:** integer **default:** `25,50,75` -### `plot_label_y_unit` +#### `plot_label_y_unit` **type:** string **default:** `%` -### `circ_start_gap` +#### `circ_start_gap` **description:** in degrees; top (usually) gap providing separation & space for labels **type:** integer **default:** `20` -### `circ_start_angle` +#### `circ_start_angle` **description:** in degrees; start of plot circle from 12 o'clock position **type:** integer **default:** `0` -### `plot_probe_y_factor` +#### `plot_probe_y_factor` **description:** relative y-scaling of the probes in array-/probeplots **type:** integer **default:** `1` -### `plot_probe_label_y_values` +#### `plot_probe_label_y_values` **type:** array **items:** number **default:** `1,2,3,4,5,6,7,8,9` -### `plot_probedot_size` +#### `plot_probedot_size` **type:** integer **default:** `1` -### `plot_probedot_opacity` +#### `plot_probedot_opacity` **type:** integer **default:** `222` -### `plot_region_labels` +#### `plot_region_labels` **description:** * placeholder for markers / labels in the @@ -322,53 +321,53 @@ relative y-scaling of the probes in array-/probeplots * label is optional **type:** array -### `plot_regionlabel_color` +#### `plot_regionlabel_color` **default:** `#ddceff` -### `plot_gene_symbols` +#### `plot_gene_symbols` **description:** * label a gene's position by its symbol (CDKN2A, MYC, ERBB2...) * comma-concatenation for multiple values **type:** array -### `plot_cytoregion_labels` +#### `plot_cytoregion_labels` **description:** * label a cytoband's position (8q24, 1p12p11, 17q...) * comma-concatenation for multiple values **type:** array -### `plot_cytoregion_color` +#### `plot_cytoregion_color` **default:** `#ffe3ee` -### `plot_marker_font_color` +#### `plot_marker_font_color` **description:** font color for gene and region markers **default:** `#dd3333` -### `plot_marker_font_size` +#### `plot_marker_font_size` **type:** integer **default:** `10` -### `plot_marker_label_padding` +#### `plot_marker_label_padding` **description:** text padding of markers versus background/box **type:** integer **default:** `4` -### `plot_marker_lane_padding` +#### `plot_marker_lane_padding` **type:** integer **default:** `2` -### `plot_footer_font_size` +#### `plot_footer_font_size` **type:** integer **default:** `10` -### `plot_footer_font_color` +#### `plot_footer_font_color` **default:** `#999999` -### `cytoband_shades` +#### `cytoband_shades` **type:** object **default:** - `gpos100`: `{'0%': 'rgb(39,39,39)', '100%': 'rgb(0,0,0)'}` @@ -380,7 +379,7 @@ text padding of markers versus background/box - `stalk`: `{'0%': 'rgb(39,39,39)', '100%': 'rgb(0,0,0)'}` - `acen`: `{'0%': 'rgb(163,55,247)', '100%': 'rgb(138,43,226)'}` -### `histoval_directions` +#### `histoval_directions` **type:** object **default:** - `gain_frequency`: `1` @@ -388,7 +387,7 @@ text padding of markers versus background/box - `loss_frequency`: `-1` - `loss_hlfrequency`: `-1` -### `histoval_colorkeys` +#### `histoval_colorkeys` **type:** object **default:** - `gain_frequency`: `plot_dup_color` @@ -396,53 +395,53 @@ text padding of markers versus background/box - `loss_frequency`: `plot_del_color` - `loss_hlfrequency`: `plot_hldel_color` -### `tiles_source` +#### `tiles_source` **default:** `https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png` -### `attribution` +#### `attribution` **default:** `Map data © OpenStreetMap contributors, CC-BY-SA` -### `init_latitude` +#### `init_latitude` **default:** `30` -### `init_longitude` +#### `init_longitude` **default:** `9` -### `zoom` +#### `zoom` **default:** `1` -### `head` +#### `head` **default:** ` ` -### `map_w_px` +#### `map_w_px` **default:** `800` -### `map_h_px` +#### `map_h_px` **default:** `512` -### `marker_type` +#### `marker_type` **default:** `marker` -### `bubble_stroke_color` +#### `bubble_stroke_color` **default:** `#dd6633` -### `bubble_stroke_weight` +#### `bubble_stroke_weight` **default:** `1` -### `bubble_fill_color` +#### `bubble_fill_color` **default:** `#cc9966` -### `bubble_opacity` +#### `bubble_opacity` **default:** `0.4` -### `marker_scale` +#### `marker_scale` **default:** `2` -### `marker_max_r` +#### `marker_max_r` **default:** `1000` -### `zoom_min` +#### `zoom_min` **default:** `2` -### `zoom_max` +#### `zoom_max` **default:** `14` diff --git a/docs/installation.md b/docs/installation.md index 9fe623896..53846356b 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -434,7 +434,22 @@ or from the corresponding one in the web cgi directory, if installed. ``` Of note the `--requestEntityPathId biosamples` etc. here simulates -the corresponding REST path following the `/beacon/` component. + +## `services.py` and URL Mapping + +Bycon web services are called through the `services.py` app which is installed +at the bycon server root. The system path for `services.py` is + +``` +{bycon_install_dir}/services/services.py +``` + +... where `bycon_install_dir` has to be user defined inside the `local/local_paths.yaml` +configuration file (see [Installation](installation.md)). The service URL format `progenetix.org/services/__service-name__?parameter=value` +is based on the remapping of the `services.py` script to the `/services` path and +then extraction of the service name as the path parameter following `/services/`. + + [^1]: Metadata in biomedical genomics is "everything but the sequence variation" diff --git a/docs/services/intervalFrequencies.md b/docs/services/intervalFrequencies.md index 6f11f998c..3d3b92d56 100644 --- a/docs/services/intervalFrequencies.md +++ b/docs/services/intervalFrequencies.md @@ -65,7 +65,7 @@ direct access to the frequecy list at `response.results[0].interval_frequencies` ##### `id` * standard parameter to retrieve a frequency set by its `id` -* available values can be looked up using the [`collations`](collations.md) +* available values can be looked up using the `collations` service: - * an `id` value will override any given `filters` diff --git a/markdowner.py b/markdowner.py index 73b8efd8a..49193cf0b 100755 --- a/markdowner.py +++ b/markdowner.py @@ -24,7 +24,7 @@ def main(): p = Path(path.join( dir_path, "byconServices" )) s_p_s = [ f for f in p.rglob("*.py") if f.is_file() ] - pp_f = path.join(generated_docs_path, f"services.md") + pp_f = path.join(generated_docs_path, f"beacon-services.md") pp_fh = open(pp_f, "w") md = [] for f in sorted(s_p_s): @@ -33,7 +33,7 @@ def main(): continue # print(f'from {fn} import {fn}') - pp_fh.write(f'## `/{fn}`\n\n') + pp_fh.write(f'### `/{fn}`\n\n') func = getattr(byconServices, fn) pp_fh.write(f'{inspect.getdoc(func)}\n\n\n') @@ -91,18 +91,7 @@ def main(): "endpoints": [], } } - pp_fh.write("""# Beacon Responses - -The following is a list of standard Beacon responses supported by the `bycon` package. -Responses for individual entities or endpoints are grouped by their Beacon framework -response classes (e.g. `beaconResultsetsResponse` for `biosamples`, `g_variants` etc.).\n\n -Please be reminded about the general syntax used in Beacon: A **path element** such -as `/biosamples` corresponds to an entity (here `biosample`). Below these relations -are indicated by the `@` symbol.\n\n -#### Schemas **{S}**, Tests **{T}** and Examples **{E}** -Tests, examples and schemas are run from the server defined in this site's build instructions -(see the `reference_server_url` entry in `mkdocs.yaml` file in the repository's root). -\n\n""") + # pp_fh.write("""""") for e, e_d in BYC["entity_defaults"].items(): if not (e_d.get("is_beacon_entity", False)): @@ -146,7 +135,6 @@ def main(): pp_fh.write(f'* **{{E}}** <{e_url}{e}>\n\n') pp_fh.write('\n\n') pp_fh.write(f'{beacon_ets[r_s].get("postscript", "")}\n\n') - pp_fh.close() @@ -158,19 +146,8 @@ def main(): "plot_type_defs": "Plot Types", "plot_parameters": "Plot Parameters" }, - "title": "Plot Parameters and Information" }, - "argument_definitions": { - "title": "`bycon` Arguments and Parameters", - "preamble": """ -The following is a list of arguments and parameters used in the `bycon` package -as well as the `byconaut` tools. - - -Parameters are listed in `snake_case` format although for command line arguments -(and also optionally web requests) `camelCase` versions are required (see the `cmdFlags`). -""" - } + "argument_definitions": {} } for d_k, d_v in file_pars.items(): @@ -179,8 +156,7 @@ def main(): pp_f = path.join(generated_docs_path, f"{d_k}.md") - ls = [f'# {d_v.get("title")}'] - ls.append(f'{d_v.get("preamble", "")}') + ls = [] if not "chapters" in d_v: d_v = { @@ -193,9 +169,9 @@ def main(): pp = BYC[d_k] else: pp = BYC[d_k].get(chapter, {}) - ls.append(f'## {title}') + ls.append(f'### {title}') for pk, pi in pp.items(): - ls.append(f'### `{pk}` \n') + ls.append(f'#### `{pk}` \n') for pik, piv in pi.items(): if type(piv) is dict: diff --git a/mkdocs.yaml b/mkdocs.yaml index c4294a50e..4a49259c3 100644 --- a/mkdocs.yaml +++ b/mkdocs.yaml @@ -10,20 +10,23 @@ reference_server_url: https://progenetix.org nav: - Documentation Home: / - - Installation: installation - - Importing Data: importers - - Maintenance: housekeepers - - Beacon API: API - - Bycon Services: services - - Recent Changes: changes - - Upcoming & To Do: future - - Progenetix Site: http://progenetix.org + - Setup & Maintainance: + - Installation: installation.md + - Importing Data: importers/index.md + - Housekeeping: housekeepers/index.md + - Beacon API: + - Beacon API: API-beacon.md + - Services API: API-services.md + - API Parameters: API-parameters.md + - Recent Changes: changes.md + # - Upcoming & To Do: future.md - Code Repositories: - bycon: https://github.com/progenetix/bycon - byconaut: https://github.com/progenetix/byconaut - Front End: https://github.com/progenetix/beaconplus-web - Progenetix Front End: https://github.com/progenetix/progenetix-web - More Info: + - Progenetix Site: http://progenetix.org - baudisgroup@UZH: https://info.baudisgroup.org - Beacon Documentation: https://docs.genomebeacons.org @@ -69,12 +72,14 @@ theme: - content.tabs.link - search.highlight - search.share + - navigation.expand plugins: - search - macros - mermaid2 - mkdocstrings + - include-markdown extra: excerpt_separator: