{% if show_full_path %}{{ class.path }}{% else %}{{ class.name }}{% endif %}
+ {% endif %}
+
+ {% with labels = class.labels %}
+ {% include "labels.html" with context %}
+ {% endwith %}
+
+ {% endfilter %}
+
+ {% if config.separate_signature and config.merge_init_into_class %}
+ {% if "__init__" in class.members %}
+ {% with function = class.members["__init__"] %}
+ {% filter format_signature(function, config.line_length, crossrefs=config.signature_crossrefs) %}
+ {% if show_full_path %}{{ class.path }}{% else %}{{ class.name }}{% endif %}
+ {% endfilter %}
+ {% endwith %}
+ {% endif %}
+ {% endif %}
+
+ {% else %}
+ {% if config.show_root_toc_entry %}
+ {% filter heading(heading_level,
+ role="class",
+ id=html_id,
+ toc_label=class.path if config.show_root_full_path else class.name,
+ hidden=True) %}
+ {% endfilter %}
+ {% endif %}
+ {% set heading_level = heading_level - 1 %}
+ {% endif %}
+
+
+ Bases: {% for expression in class.bases -%}
+ {% include "expression.html" with context %}{% if not loop.last %}, {% endif %}
+ {% endfor -%}
+
{{ class.relative_filepath }}{{ class.relative_filepath }}{{ section.title or "Parameters:" }}
diff --git a/docs/index.md b/docs/index.md index 5600aebe8..3c883f305 100644 --- a/docs/index.md +++ b/docs/index.md @@ -33,7 +33,7 @@ Once you've installed the library, let's begin with a very simple example that e ```python import spacy -nlp = spacy.blank("fr") # (1) +nlp = spacy.blank("eds") # (1) terms = dict( covid=["covid", "coronavirus"], # (2) @@ -67,60 +67,7 @@ This example is complete, it should run as-is. Check out the [spaCy 101 page](tu ## Available pipeline components -=== "Core" - - | Pipeline | Description | - | ------------------------ | ----------------------------------------------- | - | `eds.normalizer` | Non-destructive input text normalisation | - | `eds.sentences` | Better sentence boundary detection | - | `eds.matcher` | A simple yet powerful entity extractor | - | `eds.terminology` | A simple yet powerful terminology matcher | - | `eds.contextual-matcher` | A conditional entity extractor | - | `eds.endlines` | An unsupervised model to classify each end line | - -=== "Qualifiers" - - | Pipeline | Description | - | --------------------- | ------------------------------------ | - | `eds.negation` | Rule-based negation detection | - | `eds.family` | Rule-based family context detection | - | `eds.hypothesis` | Rule-based speculation detection | - | `eds.reported_speech` | Rule-based reported speech detection | - | `eds.history` | Rule-based medical history detection | - -=== "Miscellaneous" - - | Pipeline | Description | - | ------------------------ | ------------------------------------------- | - | `eds.dates` | Date extraction and normalisation | - | `eds.consultation_dates` | Identify consultation dates | - | `eds.measurements` | Measure extraction and normalisation | - | `eds.sections` | Section detection | - | `eds.reason` | Rule-based hospitalisation reason detection | - | `eds.tables` | Tables detection | - -=== "NER" - - | Pipeline | Description | - | ------------------------ | --------------------------- | - | `eds.covid` | A COVID mentions detector | - | `eds.charlson` | A Charlson score extractor | - | `eds.sofa` | A SOFA score extractor | - | `eds.emergency.priority` | A priority score extractor | - | `eds.emergency.ccmu` | A CCMU score extractor | - | `eds.emergency.gemsa` | A GEMSA score extractor | - | `eds.TNM` | A TNM score extractor | - | `eds.cim10` | A CIM10 terminology matcher | - | `eds.drugs` | A Drug mentions extractor | - | `eds.adicap` | A ADICAP codes extractor | - | `eds.umls` | A UMLS terminology matcher | - -=== "Trainable" - - | Pipeline | Description | - | -------------------- | -------------------------------------------------------------------- | - | `eds.nested-ner` | Nested and overlapping named entity recogntion | - | `eds.span-qualifier` | A trainable component for multi-class multi-label span qualification | +--8<-- "docs/pipelines/overview.md:components" ## Disclaimer @@ -142,5 +89,3 @@ If you use EDS-NLP, please cite us as below. url = {http://aphp.github.io/edsnlp} } ``` - -\bibliography diff --git a/docs/pipelines/core/contextual-matcher.md b/docs/pipelines/core/contextual-matcher.md index 3d8c6f72d..439ca5489 100644 --- a/docs/pipelines/core/contextual-matcher.md +++ b/docs/pipelines/core/contextual-matcher.md @@ -21,7 +21,7 @@ Let us see step by step how to build such a list using the example stated just a To do this, we can build either a set of `terms` or a set of `regex`. `terms` will be used to search for exact matches in the text. While less flexible, it is faster than using regex. In our case we could use the following lists (which are of course absolutely not exhaustives): -```python3 +```python terms = [ "cancer", "tumeur", @@ -36,7 +36,7 @@ regex = [ Maybe we want to exclude mentions of benign cancers: -```python3 +```python benign = "benign|benin" ``` @@ -44,7 +44,7 @@ benign = "benign|benin" For this we will forge a RegEx with one capturing group (basically a pattern enclosed in parentheses): -```python3 +```python stage = "stade (I{1,3}V?|[1234])" ``` @@ -52,7 +52,7 @@ This will extract stage between 1 and 4 We can add a second regex to try to capture if the cancer is in a metastasis stage or not: -```python3 +```python metastase = "(metasta)" ``` @@ -60,8 +60,7 @@ metastase = "(metasta)" We can now put everything together: -```python3 - +```python cancer = dict( source="Cancer solide", regex=regex, @@ -75,8 +74,8 @@ cancer = dict( dict( name="stage", regex=stage, - window=(-10,10), - replace_entity=True, + window=(-10, 10), + replace_entity=False, reduce_mode=None, ), dict( @@ -86,7 +85,7 @@ cancer = dict( replace_entity=False, reduce_mode="keep_last", ), - ] + ], ) ``` @@ -108,7 +107,7 @@ lymphome = dict( In this case, the configuration can be concatenated in a list: -```python3 +```python patterns = [cancer, lymphome] ``` @@ -143,12 +142,12 @@ This parameter can be se to `True` **only for a single assign key per dictionary **Please notice** that with `replace_entity` set to True, if the correponding assign key matches nothing, the entity will be discarded. -## Usage +## Examples ```python import spacy -nlp = spacy.blank("fr") +nlp = spacy.blank("eds") nlp.add_pipe("sentences") nlp.add_pipe("normalizer") @@ -158,6 +157,7 @@ nlp.add_pipe( name="Cancer", config=dict( patterns=patterns, + label="cancer", ), ) ``` @@ -167,15 +167,13 @@ Let us see what we can get from this pipeline with a few examples === "Simple match" - - - ```python3 + ```python txt = "Le patient a eu un cancer il y a 5 ans" doc = nlp(txt) ent = doc.ents[0] ent.label_ - # Out: Cancer + # Out: cancer ent._.source # Out: Cancer solide @@ -188,9 +186,7 @@ Let us see what we can get from this pipeline with a few examples Let us check that when a *benign* mention is present, the extraction is excluded: - - - ```python3 + ```python txt = "Le patient a eu un cancer relativement bénin il y a 5 ans" doc = nlp(txt) @@ -200,12 +196,10 @@ Let us see what we can get from this pipeline with a few examples === "Extracting additional infos" - - All informations extracted from the provided `assign` configuration can be found in the `assigned` attribute under the form of a dictionary: - ```python3 + ```python txt = "Le patient a eu un cancer de stade 3." doc = nlp(txt) @@ -213,14 +207,6 @@ Let us see what we can get from this pipeline with a few examples # Out: {'stage': '3'} ``` -## Configuration - -The pipeline can be configured using the following parameters : - -::: edsnlp.pipelines.core.contextual_matcher.factory.create_component - options: - only_parameters: true - However, most of the configuration is provided in the `patterns` key, as a **pattern dictionary** or a **list of pattern dictionaries** ## The pattern dictionary @@ -293,7 +279,7 @@ A patterr is a nested dictionary with the following keys: ### A full pattern dictionary example -```python3 +```python dict( source="AVC", regex=[ @@ -336,10 +322,13 @@ dict( expand_entity=False, window=-3, ), - ] + ], ) ``` +::: edsnlp.pipelines.core.contextual_matcher.factory.create_component + options: + only_parameters: true ## Authors and citation diff --git a/docs/pipelines/core/endlines.md b/docs/pipelines/core/endlines.md index 965b05e5f..3845cd0e9 100644 --- a/docs/pipelines/core/endlines.md +++ b/docs/pipelines/core/endlines.md @@ -1,95 +1,8 @@ -# Endlines - -The `eds.endlines` pipeline classifies newline characters as actual end of lines or mere spaces. In the latter case, the token is removed from the normalised document. - -Behind the scenes, it uses a `endlinesmodel` instance, which is an unsupervised algorithm based on the work of Zweigenbaum et al[@zweigenbaum2016]. - -## Usage - -The following example shows a simple usage. - -### Training - -```python -import spacy -from edsnlp.pipelines.core.endlines.endlinesmodel import EndLinesModel - -nlp = spacy.blank("fr") - -texts = [ - """Le patient est arrivé hier soir. -Il est accompagné par son fils - -ANTECEDENTS -Il a fait une TS en 2010; -Fumeur, il est arreté il a 5 mois -Chirurgie de coeur en 2011 -CONCLUSION -Il doit prendre -le medicament indiqué 3 fois par jour. Revoir médecin -dans 1 mois. -DIAGNOSTIC : - -Antecedents Familiaux: -- 1. Père avec diabete - -""", - """J'aime le \nfromage...\n""", -] - -docs = list(nlp.pipe(texts)) - -# Train and predict an EndLinesModel -endlines = EndLinesModel(nlp=nlp) - -df = endlines.fit_and_predict(docs) -df.head() - -PATH = "/tmp/path_to_save" -endlines.save(PATH) -``` - -### Inference - -```python -import spacy -from spacy.tokens import Span -from spacy import displacy - -nlp = spacy.blank("fr") - -PATH = "/tmp/path_to_save" -nlp.add_pipe("eds.endlines", config=dict(model_path=PATH)) - -docs = list(nlp.pipe(texts)) - -doc_exemple = docs[1] - -doc_exemple.ents = tuple( - Span(doc_exemple, token.i, token.i + 1, "excluded") - for token in doc_exemple - if token.tag_ == "EXCLUDED" -) - -displacy.render(doc_exemple, style="ent", options={"colors": {"space": "red"}}) -``` - -## Configuration - -The pipeline can be configured using the following parameters : - -| Parameter | Explanation | Default | -| ------------ | -------------------------------- | -------- | -| `model_path` | Path to the pre-trained pipeline | Required | - -## Declared extensions - -The `eds.endlines` pipeline declares one [spaCy extensions](https://spacy.io/usage/processing-pipelines#custom-components-attributes), on both `Span` and `Token` objects. The `end_line` attribute is a boolean, set to `True` if the pipeline predicts that the new line is an end line character. Otherwise, it is set to `False` if the new line is classified as a space. - -The pipeline also sets the `excluded` custom attribute on newlines that are classified as spaces. It lets downstream matchers skip excluded tokens (see [normalisation](./normalisation.md)) for more detail. - -## Authors and citation - -The `eds.endlines` pipeline was developed by AP-HP's Data Science team based on the work of Zweigenbaum et al[@zweigenbaum2016]. - -\bibliography +# Endlines {: #edsnlp.pipelines.core.endlines.factory.create_component } + +::: edsnlp.pipelines.core.endlines.factory.create_component + options: + heading_level: 2 + show_bases: false + show_source: false + only_class_level: true diff --git a/docs/pipelines/core/index.md b/docs/pipelines/core/index.md deleted file mode 100644 index ceae7f535..000000000 --- a/docs/pipelines/core/index.md +++ /dev/null @@ -1,7 +0,0 @@ -# Core Pipelines - -This section deals with "core" functionalities offered by EDS-NLP: - -- Matching a terminology -- Normalising a text -- Detecting sentence boundaries diff --git a/docs/pipelines/core/matcher.md b/docs/pipelines/core/matcher.md index cb08afbb2..984541ec4 100644 --- a/docs/pipelines/core/matcher.md +++ b/docs/pipelines/core/matcher.md @@ -1,54 +1,8 @@ -# Matcher - -EDS-NLP simplifies the matching process by exposing a `eds.matcher` pipeline -that can match on terms or regular expressions. - -## Usage - -Let us redefine the pipeline : - -```python -import spacy - -nlp = spacy.blank("fr") - -terms = dict( - covid=["coronavirus", "covid19"], # (1) - patient="patient", # (2) -) - -regex = dict( - covid=r"coronavirus|covid[-\s]?19|sars[-\s]cov[-\s]2", # (3) -) - -nlp.add_pipe( - "eds.matcher", - config=dict( - terms=terms, - regex=regex, - attr="LOWER", - term_matcher="exact", - term_matcher_config={}, - ), -) -``` - -1. Every key in the `terms` dictionary is mapped to a concept. -2. The `eds.matcher` pipeline expects a list of expressions, or a single expression. -3. We can also define regular expression patterns. - -This snippet is complete, and should run as is. - -## Configuration - -The pipeline can be configured using the following parameters : +# Matcher {: #edsnlp.pipelines.core.matcher.factory.create_component } ::: edsnlp.pipelines.core.matcher.factory.create_component options: - only_parameters: true - -Patterns, be they `terms` or `regex`, are defined as dictionaries where keys become the label of the extracted entities. Dictionary values are a either a single expression or a list of expressions that match the concept (see [example](#usage)). - -## Authors and citation - -The `eds.matcher` pipeline was developed by AP-HP's Data Science team. + heading_level: 2 + show_bases: false + show_source: false + only_class_level: true diff --git a/docs/pipelines/core/normalisation.md b/docs/pipelines/core/normalizer.md similarity index 97% rename from docs/pipelines/core/normalisation.md rename to docs/pipelines/core/normalizer.md index bd7c0ebfd..5c6b7223c 100644 --- a/docs/pipelines/core/normalisation.md +++ b/docs/pipelines/core/normalizer.md @@ -1,10 +1,8 @@ -# Normalisation +# Normalisation {: #edsnlp.pipelines.core.normalizer.factory.create_component } The normalisation scheme used by EDS-NLP adheres to the non-destructive doctrine. In other words, - - -```python +```{ .python .no-check } nlp(text).text == text ``` @@ -39,7 +37,7 @@ The normalisation is handled by the single `eds.normalizer` pipeline. The follow import spacy from edsnlp.matchers.utils import get_text -nlp = spacy.blank("fr") +nlp = spacy.blank("eds") nlp.add_pipe("eds.normalizer") # Notice the special character used for the apostrophe and the quotes @@ -87,7 +85,7 @@ config = dict( pollution=False, ) -nlp = spacy.blank("fr") +nlp = spacy.blank("eds") nlp.add_pipe("eds.normalizer", config=config) text = "Pneumopathie à NBNbWbWbNbWbNBNbNbWbW `coronavirus'" @@ -118,7 +116,7 @@ config = dict( pollution=False, ) -nlp = spacy.blank("fr") +nlp = spacy.blank("eds") nlp.add_pipe("eds.normalizer", config=config) text = "Pneumopathie à NBNbWbWbNbWbNBNbNbWbW `coronavirus'" @@ -148,7 +146,7 @@ config = dict( pollution=False, ) -nlp = spacy.blank("fr") +nlp = spacy.blank("eds") nlp.add_pipe("eds.normalizer", config=config) text = "Pneumopathie à NBNbWbWbNbWbNBNbNbWbW `coronavirus'" @@ -207,7 +205,7 @@ config = dict( pollution=True, ) -nlp = spacy.blank("fr") +nlp = spacy.blank("eds") nlp.add_pipe("eds.normalizer", config=config) text = "Pneumopathie à NBNbWbWbNbWbNBNbNbWbW `coronavirus'" @@ -235,7 +233,7 @@ For instance, if we consider biology tables as pollution, we only need to instan ```python import spacy -nlp = spacy.blank("fr") +nlp = spacy.blank("eds") nlp.add_pipe( "eds.normalizer", config=dict( @@ -264,7 +262,7 @@ For instance, to consider text between "AAA" and "ZZZ" as pollution you might us ```python import spacy -nlp = spacy.blank("fr") +nlp = spacy.blank("eds") nlp.add_pipe( "eds.normalizer", config=dict( diff --git a/docs/pipelines/core/overview.md b/docs/pipelines/core/overview.md new file mode 100644 index 000000000..e9d71307e --- /dev/null +++ b/docs/pipelines/core/overview.md @@ -0,0 +1,22 @@ +# Core Components + +This section deals with "core" functionalities offered by EDS-NLP: + +- Generic matchers against regular expressions and list of terms +- Text cleaning +- Sentence boundaries detection + +## Available components + + + +| Component | Description | +|----------------------------------------------------------------|-------------------------------------------------| +| [`eds.normalizer`](/pipelines/core/normalizer) | Non-destructive input text normalisation | +| [`eds.sentences`](/pipelines/core/sentences) | Better sentence boundary detection | +| [`eds.matcher`](/pipelines/core/matcher) | A simple yet powerful entity extractor | +| [`eds.terminology`](/pipelines/core/terminology) | A simple yet powerful terminology matcher | +| [`eds.contextual_matcher`](/pipelines/core/contextual-matcher) | A conditional entity extractor | +| [`eds.endlines`](/pipelines/core/endlines) | An unsupervised model to classify each end line | + + diff --git a/docs/pipelines/core/sentences.md b/docs/pipelines/core/sentences.md index 9bd34a166..0d1cfde54 100644 --- a/docs/pipelines/core/sentences.md +++ b/docs/pipelines/core/sentences.md @@ -1,71 +1,8 @@ -# Sentences - -The `eds.sentences` pipeline provides an alternative to spaCy's default `sentencizer`, aiming to overcome some of its limitations. - -Indeed, the `sentencizer` merely looks at period characters to detect the end of a sentence, a strategy that often fails in a clinical note settings. Our `sentences` component also classifies end-of-lines as sentence boundaries if the subsequent token begins with an uppercase character, leading to slightly better performances. - -Moreover, the `eds.sentences` pipeline can use the output of the `eds.normalizer` pipeline, and more specifically the end-of-line classification. This is activated by default. - -## Usage - -=== "EDS-NLP" - - - - ```python - import spacy - - nlp = spacy.blank("fr") - nlp.add_pipe("eds.sentences") - - text = ( - "Le patient est admis le 23 août 2021 pour une douleur à l'estomac\n" - "Il lui était arrivé la même chose il y a deux ans." - ) - - doc = nlp(text) - - for sentence in doc.sents: - print("