+
+
+ ' + + '' + + _("Hide Search Matches") + + "
" + ) + ); + }, + + /** + * helper function to hide the search marks again + */ + hideSearchWords: () => { + document + .querySelectorAll("#searchbox .highlight-link") + .forEach((el) => el.remove()); + document + .querySelectorAll("span.highlighted") + .forEach((el) => el.classList.remove("highlighted")); + localStorage.removeItem("sphinx_highlight_terms") + }, + + initEscapeListener: () => { + // only install a listener if it is really needed + if (!DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS) return; + + document.addEventListener("keydown", (event) => { + // bail for input elements + if (BLACKLISTED_KEY_CONTROL_ELEMENTS.has(document.activeElement.tagName)) return; + // bail with special keys + if (event.shiftKey || event.altKey || event.ctrlKey || event.metaKey) return; + if (DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS && (event.key === "Escape")) { + SphinxHighlight.hideSearchWords(); + event.preventDefault(); + } + }); + }, +}; + +_ready(() => { + /* Do not call highlightSearchWords() when we are on the search page. + * It will highlight words from the *previous* search query. + */ + if (typeof Search === "undefined") SphinxHighlight.highlightSearchWords(); + SphinxHighlight.initEscapeListener(); +}); diff --git a/cran.html b/cran.html new file mode 100644 index 0000000..fa468ea --- /dev/null +++ b/cran.html @@ -0,0 +1,844 @@ + + + + + + +
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/genindex.html b/genindex.html
new file mode 100644
index 0000000..9cbd39c
--- /dev/null
+++ b/genindex.html
@@ -0,0 +1,573 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ CRAN module
+This module imports R packages published at the Comprehensive R Archive Network (CRAN).
+Specifically, it reads the table of packages ordered by date of publication. +This table contains for each R package the package name, title and date of publication. Based on the package name, +each package url can be accessed from:
+https://cran.r-project.org/web/packages/<package_name>/index.html
Several attributes are listed for each package. Among them, the following attributes are imported, when present, to the MaRDI knowledge graph:
+-
+
- Version: +
Version of the package.
+
+- Depends: +
Software and package dependencies, including other R packages.
+
+- Published: +
Date of publication.
+
+- Author: +
Authors of the package are to be indicated according to the CRAN Repository Policy with the abbreviation [aut]. Given that this guideline is not always implemented, it is not always +possible to properly parse the authors.
+When no abbrevations describing the role of each individual are included, just the first listed author is imported.
+
+- License: +
- + +
- Maintainer: +
Software maintainer (Generally one of the authors).
+
+
mardi_importer.cran.CRANSource class
+-
+
- +class mardi_importer.cran.CRANSource.CRANSource[source] +
Bases:
+ADataSourceProcesses data from the Comprehensive R Archive Network.
+Metadata for each R package is scrapped from the CRAN Repository. Each +Wikibase item corresponding to each R package is subsequently updated +or created, in case of a new package.
+-
+
- +packages +
Dataframe with package name, title and date of publication for +each package in CRAN.
+-
+
- Type: +
Pandas dataframe
+
+
-
+
- +pull()[source] +
Reads date, package name and title from the CRAN Repository URL.
+The result is saved as a pandas dataframe in the attribute packages.
+-
+
- Returns: +
Attribute
+packages
+- Return type: +
Pandas dataframe
+
+- Raises: +
ImporterException – If table at the CRAN url cannot be accessed or read.
+
+
-
+
- +push()[source] +
Updates the MaRDI Wikibase entities corresponding to R packages.
+For each package name in the attribute packages checks +if the date in CRAN coincides with the date in the MaRDI +knowledge graph. If not, the package is updated. If the package +is not found in the MaRDI knowledge graph, the corresponding +item is created.
+It creates a
+mardi_importer.cran.RPackageinstance +for each package.
mardi_importer.cran.RPackage class
+-
+
- +class mardi_importer.cran.RPackage.RPackage(date: str, label: str, description: str, api: ~mardi_importer.integrator.MardiIntegrator.MardiIntegrator, long_description: str = '', url: str = '', version: str = '', versions: ~typing.List[~typing.Tuple[str, str]] = <factory>, authors: ~typing.List[~mardi_importer.publications.Author.Author] = <factory>, license_data: ~typing.List[~typing.Tuple[str, str]] = <factory>, dependencies: ~typing.List[~typing.Tuple[str, str]] = <factory>, imports: ~typing.List[~typing.Tuple[str, str]] = <factory>, maintainer: str = '', author_pool: ~typing.List[~mardi_importer.publications.Author.Author] = <factory>, crossref_publications: ~typing.List[~mardi_importer.publications.CrossrefPublication.CrossrefPublication] = <factory>, arxiv_publications: ~typing.List[~mardi_importer.publications.ArxivPublication.ArxivPublication] = <factory>, zenodo_resources: ~typing.List[~mardi_importer.publications.ZenodoResource.ZenodoResource] = <factory>, _QID: str = '', _item: ~mardi_importer.integrator.MardiEntities.MardiItemEntity = None)[source] +
Bases:
+objectClass to manage R package items in the local Wikibase instance.
+-
+
- +date +
Date of publication
+-
+
- Type: +
str
+
+
-
+
- +label +
Package name
+-
+
- Type: +
str
+
+
-
+
- +description +
Title of the R package
+-
+
- Type: +
str
+
+
-
+
- +long_description +
Detailed description of the R package
+-
+
- Type: +
str
+
+
-
+
- +url +
URL to the CRAN repository
+-
+
- Type: +
str
+
+
-
+
- +version +
Version of the R package
+-
+
- Type: +
str
+
+
-
+
- +versions +
Previous published versions
+-
+
- Type: +
List[Tuple[str, str]]
+
+
-
+
- +author +
Author(s) of the package
+
-
+
- +license +
Software license
+
-
+
- +dependency +
Dependencies to R and other packages
+
-
+
- +imports +
Imported R packages
+-
+
- Type: +
List[Tuple[str, str]]
+
+
-
+
- +maintainer +
Software maintainer
+-
+
- Type: +
str
+
+
-
+
- +_QID +
Package QID
+-
+
- Type: +
str
+
+
-
+
- +integrator +
API to MaRDI integrator
+
-
+
- +date: str +
-
+
- +label: str +
-
+
- +description: str +
-
+
- +api: MardiIntegrator +
-
+
- +long_description: str = '' +
-
+
- +url: str = '' +
-
+
- +version: str = '' +
-
+
- +versions: List[Tuple[str, str]] +
-
+
- +authors: List[Author] +
-
+
- +license_data: List[Tuple[str, str]] +
-
+
- +dependencies: List[Tuple[str, str]] +
-
+
- +imports: List[Tuple[str, str]] +
-
+
- +maintainer: str = '' +
-
+
- +author_pool: List[Author] +
-
+
- +crossref_publications: List[CrossrefPublication] +
-
+
- +arxiv_publications: List[ArxivPublication] +
-
+
- +zenodo_resources: List[ZenodoResource] +
-
+
- +property QID: str +
Return the QID of the R package in the knowledge graph.
+Searches for an item with the package label in the Wikibase +SQL tables and returns the QID if a matching result is found.
+-
+
- Returns: +
The entity QID representing the R package.
+
+- Return type: +
str
+
+
-
+
- +property item: MardiItemEntity +
Return the integrator Item representing the R package.
+Adds also the label and description of the package.
+-
+
- Returns: +
Integrator item
+
+- Return type: +
- + +
-
+
- +exists() str[source] +
Checks if an item corresponding to the R package already exists.
+-
+
- Returns: +
Entity ID
+
+- Return type: +
str
+
+
-
+
- +is_updated() bool[source] +
Checks if the Item corresponding to the R package is up to date.
+Compares the last update property in the local knowledge graph with +the publication date imported from CRAN.
+-
+
- Returns: +
True if both dates coincide, False otherwise.
+
+- Return type: +
bool
+
+
-
+
- +pull()[source] +
Imports metadata from CRAN corresponding to the R package.
+Imports Version, Dependencies, Imports**m **Authors, +Maintainer and License and saves them as instance +attributes.
+
-
+
- +create() None[source] +
Create a package in the Wikibase instance.
+This function pulls the package, inserts its claims, and writes +it to the Wikibase instance.
+-
+
- Returns: +
None
+
+
-
+
- +write() Dict[str, str] | None[source] +
Write the package item to the Wikibase instance.
+If the item has claims, it will be written to the Wikibase instance. +If the item is successfully written, a dictionary with the QID of the +item will be returned.
+-
+
- Returns: +
A dictionary with the QID of the written item if successful, +or None otherwise.
+
+- Return type: +
Optional[Dict[str, str]]
+
+
-
+
- +update()[source] +
Updates existing WB item with the imported metadata from CRAN.
+The metadata corresponding to the package is first pulled from CRAN and +saved as instance attributes through
+pull(). The statements that +do not coincide with the locally saved information are updated or +subsituted with the updated information.Uses
+mardi_importer.wikibase.WBItemto update the item +corresponding to the R package.-
+
- Returns: +
ID of the updated R package.
+
+- Return type: +
str
+
+
-
+
- +parse_publications(description)[source] +
Extracts the DOI identification of related publications.
+Identifies the DOI of publications that are mentioned using the +format doi: or arXiv: in the long description of the +R package.
+-
+
- Returns: +
List containing the wikibase IDs of mentioned publications.
+
+- Return type: +
List
+
+
-
+
- +get_last_update()[source] +
Returns the package last update date saved in the Wikibase instance.
+-
+
- Returns: +
Last update date in format DD-MM-YYYY.
+
+- Return type: +
str
+
+
-
+
- +clean_package_list(table_html)[source] +
Processes raw imported data from CRAN to enable the creation of items.
+-
+
Package dependencies are splitted at the comma position.
+License information is processed using the
parse_license()method.
+Author information is processed using the
parse_authors()method.
+Maintainer information is processed using the
parse_maintainer()method.
+
-
+
- Parameters: +
table_html – HTML code obtained with BeautifulSoup corresponding to the table +containing the metadata of the R package imported from CRAN.
+
+- Returns: +
Dataframe with processed data from a single R package including columns: +Version, Author, License, Depends, Imports +and Maintainer.
+
+- Return type: +
(Pandas dataframe)
+
+
-
+
- +parse_software(software_str: str) List[Tuple[str, str]][source] +
Processes the dependency and import information of each R package.
+This includes: +- Extracting the version information of each dependency/import if provided. +- Providing the Item QID given the dependency/import label. +- Creating a new Item if the dependency/import is not found in the
++
+local knowledge graph.
+-
+
- Returns: +
List of tuples including software QID and version.
+
+- Return type: +
List[Tuple[str, str]]
+
+
-
+
- +parse_license(x: str) List[Tuple[str, str]][source] +
Splits string of licenses.
+Takes into account that licenses are often not uniformly listed. +Characters |, + and , are used to separate licenses. Further +details on each license are often included in square brackets.
+The concrete License is identified and linked to the corresponding +item that has previously been imported from Wikidata. Further license +information, when provided between round or square brackets, is added +as a qualifier.
+If a file license is mentioned, the linked to the file license +in CRAN is added as a qualifier.
+-
+
- Parameters: +
x (str) – String imported from CRAN representing license +information.
+
+- Returns: +
List of license tuples. Each tuple contains the license QID +as the first element and the license qualifier as the +second element.
+
+- Return type: +
List[Tuple[str, str]]
+
+
-
+
- +parse_authors(x)[source] +
Splits the string corresponding to the authors into a dictionary.
+Author information in CRAN is not registered uniformly. This function +parses the imported string and returns just the names of the individuals +that can be unequivocally identified as authors (i.e. they are followed +by the [aut] abbreviation).
+Generally, authors in CRAN are indicated with the abbreviation [aut]. +When no abbreviations are included, only the first individual is imported +to Wikibase (otherwise it can often not be established whether +information after the first author refers to another individual, +an institution, a funder, etc.)
+-
+
- Parameters: +
x (String) – String imported from CRAN representing author +information.
+
+- Returns: +
Dictionary of authors and corresponding ORCID ID, if provided.
+
+- Return type: +
(Dict)
+
+
-
+
- +parse_maintainer(name: str) str[source] +
Remove unnecessary information from maintainer string.
+-
+
- Parameters: +
x (str) – String imported from CRAN which may contain e-mail +address and comments within brackets
+
+- Returns: +
Name of the maintainer
+
+- Return type: +
(str)
+
+
-
+
- +get_license_QID(license_str: str) str[source] +
Returns the Wikidata item ID corresponding to a software license.
+The same license is often denominated in CRAN using differents names. +This function returns the wikidata item ID corresponding to a single +unique license that is referenced in CRAN under different names (e.g. +Artistic-2.0 and Artistic License 2.0 both refer to the same +license, corresponding to item Q14624826).
+-
+
- Parameters: +
license_str (str) – String corresponding to a license imported from CRAN.
+
+- Returns: +
Wikidata item ID.
+
+- Return type: +
(str)
+
+
-
+
- +get_wikidata_QID() str | None[source] +
Get the Wikidata QID for the R package.
+Searches for the R package in Wikidata using its label. Retrieves +the QID of matching entities and checks if there is an instance of +an R package. If so, returns the QID.
+-
+
- Returns: +
The Wikidata QID of the R package if found, or None otherwise.
+
+- Return type: +
Optional[str]
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/importer.html b/importer.html
new file mode 100644
index 0000000..2baf599
--- /dev/null
+++ b/importer.html
@@ -0,0 +1,215 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Index
+ +
+ _
+ | A
+ | C
+ | D
+ | E
+ | G
+ | I
+ | L
+ | M
+ | N
+ | O
+ | P
+ | Q
+ | R
+ | S
+ | U
+ | V
+ | W
+ | Z
+
+
+_
+| + |
A
+| + | + |
C
+D
+| + | + |
E
+| + |
G
+I
+L
+| + | + |
M
+
|
+
|
+
N
+| + |
O
+| + |
P
+Q
+| + | + |
R
+| + |
S
+| + |
U
+| + | + |
V
+| + | + |
W
+| + | + |
Z
+| + | + |
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/index.html b/index.html
new file mode 100644
index 0000000..9ee794e
--- /dev/null
+++ b/index.html
@@ -0,0 +1,179 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ mardi_importer.importer module
+mardi_importer.importer.Importer module
+Created on Thu Feb 17 18:53:53 2022
+@author: alvaro
+-
+
- +class mardi_importer.importer.Importer.AConfigParser[source] +
Bases:
+objectAbstract base class for parsing config files
+ + +
-
+
- +class mardi_importer.importer.Importer.ADataSource[source] +
Bases:
+objectAbstract base class for reading data from external sources.
+ + + + + + + + +
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/integrator.html b/integrator.html
new file mode 100644
index 0000000..2a6e978
--- /dev/null
+++ b/integrator.html
@@ -0,0 +1,584 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ MaRDI Importer Documentation
+This is the documentation of docker-importer.
+Table of contents
+
+
+Manual
+ +
+
+API Documentation
+ +Indices and tables
+-
+
- +
- +
- +
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/main.html b/main.html
new file mode 100644
index 0000000..00e2ad7
--- /dev/null
+++ b/main.html
@@ -0,0 +1,166 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ mardi_importer.integrator module
+mardi_importer.integrator.MardiIntegrator module
+-
+
- +class mardi_importer.integrator.MardiIntegrator.MardiIntegrator(languages=['en', 'de'])[source] +
Bases:
+MardiClient-
+
- +config()[source] +
Sets up initial configuration for the integrator
+-
+
- Returns: +
Clientlogin object
+
+
-
+
- +convert_claim_ids(entity)[source] +
Function for in-place conversion of wikidata +ids found in claims into local ids
+-
+
- Parameters: +
entity –
+
+- Returns: +
None
+
+
-
+
- +convert_entity_links(snak)[source] +
Function for in-place conversion of unit for quantity +and globe for globecoordinate to a link to the local entity +instead of a link to the wikidata entity.
+-
+
- Parameters: +
snak – a wikibaseintegrator snak
+
+- Returns: +
None
+
+
-
+
- +create_db_table()[source] +
Check if db table for id mapping is there; if not, create.
+-
+
- Parameters: +
None –
+
+- Returns: +
None
+
+
-
+
- +create_id_list_from_file(file)[source] +
Function for creating a list of ids +from a while where each id is in a new line
+-
+
- Parameters: +
file – path to file
+
+
Returns: list of ids
+
-
+
- +get_local_id_by_label(entity_str, entity_type)[source] +
Check if entity with a given label or wikidata PID/QID +exists in the local wikibase instance.
+-
+
- Parameters: +
-
+
entity_str (str) – It can be a string label or a wikidata ID, +specified with the prefix wdt: for properties and wd: +for items.
+entity_type (str) – Either ‘property’ or ‘item’ to specify +which type of entity to look for.
+
+- Returns: +
Local ID of the entity, if found.
+
+- Return type: +
str
+
+
-
+
- +get_qualifiers(claim)[source] +
Function for creating qualifiers from wikidata qualifiers +and in place adding them to the claim
+-
+
- Parameters: +
claim – a wikibaseintegrator claim
+
+- Returns: +
Qualifiers object, can also be an empty object
+
+
-
+
- +get_references(claim)[source] +
Function for creating references from wikidata references +and in place adding them to the claim
+-
+
- Parameters: +
claim – a wikibaseintegrator claim
+
+- Returns: +
List with references, can also be an empty list
+
+
-
+
- +get_wikidata_information(wikidata_id, recurse=False)[source] +
Function for pulling wikidata information
+-
+
- Parameters: +
-
+
wikidata_id (str) – wikidata id of the desired entity
+recurse (Bool) – if claims should also be imported
+
+
Returns: wikibase integrator entity or None, if the entity has no labels
+
-
+
- +import_claim_entities(wikidata_id)[source] +
Function for importing entities that are mentioned +in claims from wikidata to the local wikibase instance
+-
+
- Parameters: +
wikidata_id (str) – id of the entity to be imported
+
+- Returns: +
local id or None, if the entity had no labels
+
+
-
+
- +import_entities(id_list=None, filename='', recurse=True)[source] +
Function for importing entities from wikidata +into the local instance.
+It can accept a single id, a list of ids or a file containing +a the ids to be imported.
+-
+
- Parameters: +
-
+
id_list – Single string or list of strings of wikidata +entity ids. Lexemes not supported.
+filename – Filename containing list of entities to be +imported.
+recurse – Whether to import claims for the entities in +id_list
+
+- Returns: +
Dictionary containing the local ids of +all the imported entities.
+
+- Return type: +
Imported entities (Dict)
+
+
-
+
- +import_from_label(label)[source] +
Imports an entity from Wikidata just from a label
+-
+
- Parameters: +
label (str) – label to be imported from wikidata
+
+- Returns: +
local id for the imported entity
+
+- Return type: +
local_id (str)
+
+
-
+
- +init_wikidata_PID()[source] +
Searches the wikidata PID property ID to link +properties to its ID in wikidata. When not found, +it creates the property.
+-
+
- Returns
wikidata_PID (str): wikidata PID property ID
+
+
-
+
- +init_wikidata_QID()[source] +
Searches the wikidata QID property ID to link +items to its ID in wikidata. When not found, +it creates the property.
+-
+
- Returns
wikidata_QID (str): wikidata QID property ID
+
+
-
+
- +insert_id_in_db(wikidata_id, local_id, has_all_claims)[source] +
Insert wikidata_id, local_id and has_all_claims into mapping table.
+-
+
- Parameters: +
-
+
wikidata_id – Wikidata id
+local_id – local Wikibase id
+has_all_claims – Boolean indicating whether the entity has been +imported with all claims or no claims (i.e. no recurse)
+
+- Returns: +
None
+
+
-
+
- +overwrite_entity(wikidata_id, local_id)[source] +
Function for completing an already existing local entity +with its statements from wikidata.
+-
+
- Parameters: +
-
+
wikidata_id – Wikidata entity ID to be imported.
+local_id – Local id of the existing entity that needs to +be completed with further statements.
+
+- Returns: +
Local entity ID
+
+- Return type: +
local_id
+
+
-
+
- +query(parameter, wikidata_id)[source] +
Query the wb_id_mapping db table for a given parameter.
+The two important parameters are the local_id and whether the +entity has already been imported with all claims
+-
+
- Parameters: +
-
+
parameter (str) – Either local_id or has_all_claims
+wikidata_id (str) – Wikidata ID
+
+- Returns: +
-
+
- for local_id returns the local ID if it exists,
otherwise None. For has_all_claims, a boolean is returned.
+
+
+- Return type: +
str or boolean
+
+
-
+
- +query_with_local_id(parameter, local_id)[source] +
Query the wb_id_mapping db table for a given parameter.
+The two important parameters are the wikidata_id and whether the +entity has already been imported with all claims
+-
+
- Parameters: +
-
+
parameter (str) – Either wikidata_id or has_all_claims
+local_id (str) – local ID
+
+- Returns: +
-
+
- for wikidata_id returns the wikidata ID if it exists,
otherwise None. For has_all_claims, a boolean is returned.
+
+
+- Return type: +
str or boolean
+
+
mardi_importer.integrator.MardiEntities module
+-
+
- +class mardi_importer.integrator.MardiEntities.MardiItemEntity(labels: Labels | None = None, descriptions: Descriptions | None = None, aliases: Aliases | None = None, sitelinks: Sitelinks | None = None, **kwargs: Any)[source] +
Bases:
+MardiItem-
+
- +get(entity_id, **kwargs)[source] +
Request the MediaWiki API to get data for the entity specified in argument.
+-
+
- Parameters: +
-
+
entity_id – The entity_id of the Item entity you want. Must start with a ‘Q’.
+kwargs –
+
+- Returns: +
an ItemEntity instance
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/objects.inv b/objects.inv
new file mode 100644
index 0000000..0fdda28
Binary files /dev/null and b/objects.inv differ
diff --git a/polydb.html b/polydb.html
new file mode 100644
index 0000000..c26efaa
--- /dev/null
+++ b/polydb.html
@@ -0,0 +1,129 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Introduction
+What is this about?
+Installation
+Without Docker
+Install the python package of mardi-importer by first installing the
+requirements from requirements.txt,
pip install -r requirements.txt
+Then install the packages via
+pip install -U -e .
+-U enforces reinstalling the package, with -e modifications in
+the source files are automatically taken into account.
Note: for convenience, local installations not using docker can be placed within +virtual environments by first calling
+python3 -m venv env
+source env/bin/activate
+With Docker
+TODO
+Build documentation
+In docs/, run make html to generate the documentation for a
+local installation. The modules have to be installed and findable by import
+module. To view the docs, open the file docs/_build/html/index.html.
Usage
+TODO
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/py-modindex.html b/py-modindex.html
new file mode 100644
index 0000000..2d1552e
--- /dev/null
+++ b/py-modindex.html
@@ -0,0 +1,174 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/scripts.html b/scripts.html
new file mode 100644
index 0000000..fec89ab
--- /dev/null
+++ b/scripts.html
@@ -0,0 +1,143 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Python Module Index
+ +
+ m
+
+
+ | + m | ||
| + |
+ mardi_importer | + |
| + |
+ mardi_importer.cran.CRANSource | + |
| + |
+ mardi_importer.cran.RPackage | + |
| + |
+ mardi_importer.importer.Importer | + |
| + |
+ mardi_importer.integrator.MardiEntities | + |
| + |
+ mardi_importer.integrator.MardiIntegrator | + |
| + |
+ mardi_importer.polydb | + |
| + |
+ mardi_importer.zbmath.misc | + |
| + |
+ mardi_importer.zbmath.ZBMathSource | + |
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/search.html b/search.html
new file mode 100644
index 0000000..3a76551
--- /dev/null
+++ b/search.html
@@ -0,0 +1,134 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/searchindex.js b/searchindex.js
new file mode 100644
index 0000000..65eabf6
--- /dev/null
+++ b/searchindex.js
@@ -0,0 +1 @@
+Search.setIndex({"docnames": ["cran", "importer", "index", "integrator", "main", "polydb", "scripts", "tech", "zbmath"], "filenames": ["cran.rst", "importer.rst", "index.rst", "integrator.rst", "main.rst", "polydb.rst", "scripts.rst", "tech.rst", "zbmath.rst"], "titles": ["CRAN module", "mardi_importer.importer module", "MaRDI Importer Documentation", "mardi_importer.integrator module", "Introduction", "mardi_importer.polydb module", "Launcher script", "Importer diagram", "mardi_importer.zbmath module"], "terms": {"thi": [0, 2, 4, 8], "import": [0, 3, 4, 6], "r": [0, 4], "packag": [0, 4], "publish": 0, "comprehens": 0, "archiv": 0, "network": 0, "specif": [0, 8], "read": [0, 1, 8], "tabl": [0, 3], "order": 0, "date": [0, 8], "public": [0, 8], "contain": [0, 1, 3], "each": [0, 3, 8], "name": [0, 2, 8], "titl": [0, 8], "base": [0, 1, 3, 8], "url": [0, 8], "can": [0, 3, 4], "access": 0, "from": [0, 1, 3, 4, 8], "http": 0, "project": 0, "org": 0, "web": 0, "package_nam": 0, "index": [0, 2, 4], "html": [0, 4], "sever": 0, "attribut": 0, "ar": [0, 3, 4, 8], "list": [0, 3], "among": 0, "them": [0, 3], "follow": 0, "when": [0, 3], "present": 0, "mardi": [0, 1, 4, 8], "knowledg": [0, 1], "graph": [0, 1], "version": 0, "depend": 0, "softwar": 0, "includ": 0, "other": 0, "author": [0, 1, 8], "indic": [0, 3], "accord": 0, "repositori": 0, "polici": 0, "abbrevi": 0, "aut": 0, "given": [0, 3], "guidelin": 0, "i": [0, 2, 3, 4], "alwai": 0, "implement": 0, "possibl": [0, 6], "properli": 0, "pars": [0, 1, 8], "abbrev": 0, "describ": 0, "role": 0, "individu": 0, "just": [0, 3], "first": [0, 4], "licens": 0, "accept": [0, 3], "maintain": 0, "gener": [0, 4], "one": 0, "sourc": [0, 1, 3, 4, 8], "adatasourc": [0, 1, 8], "process": [0, 1, 8], "data": [0, 1, 3, 8], "metadata": 0, "scrap": 0, "wikibas": [0, 1, 3, 8], "item": [0, 3], "correspond": [0, 8], "subsequ": 0, "updat": [0, 3, 8], "creat": [0, 1, 3, 8], "case": 0, "new": [0, 3], "datafram": 0, "type": [0, 3, 8], "panda": 0, "setup": [0, 8], "all": [0, 3, 8], "necessari": [0, 8], "properti": [0, 3, 8], "entiti": [0, 3, 8], "create_local_ent": [0, 8], "pull": [0, 1, 3, 8], "The": [0, 3, 4], "result": 0, "save": [0, 8], "return": [0, 1, 3, 8], "rais": 0, "importerexcept": [0, 1], "If": 0, "cannot": 0, "push": [0, 1, 8], "For": [0, 3], "check": [0, 3], "coincid": 0, "found": [0, 3, 8], "It": [0, 3, 8], "instanc": [0, 3, 8], "str": [0, 3], "label": [0, 3], "descript": [0, 3], "api": [0, 1, 3, 8], "integr": [0, 2], "mardiintegr": [0, 2], "long_descript": 0, "tupl": 0, "factori": 0, "license_data": 0, "author_pool": 0, "crossref_publ": 0, "crossrefpubl": 0, "arxiv_publ": 0, "arxivpubl": 0, "zenodo_resourc": 0, "zenodoresourc": 0, "_qid": 0, "_item": 0, "mardient": [0, 2], "mardiitement": [0, 3], "none": [0, 3, 8], "object": [0, 1, 3], "manag": [0, 1], "local": [0, 1, 3, 4], "detail": 0, "previou": 0, "": 0, "qid": [0, 3], "search": [0, 2, 3], "an": [0, 1, 3], "sql": 0, "match": 0, "repres": 0, "add": 0, "also": [0, 3], "exist": [0, 3], "alreadi": [0, 3], "id": [0, 3, 8], "is_upd": 0, "bool": [0, 3], "up": [0, 3], "compar": 0, "last": 0, "true": [0, 1, 3], "both": 0, "fals": [0, 3], "otherwis": [0, 3], "m": [0, 4], "function": [0, 3, 8], "insert": [0, 3], "its": [0, 3], "claim": [0, 3], "write": [0, 1], "dict": [0, 3, 8], "ha": [0, 3], "written": 0, "successfulli": 0, "dictionari": [0, 1, 3], "A": 0, "success": 0, "option": [0, 8], "insert_claim": 0, "wb": 0, "through": 0, "statement": [0, 3], "do": 0, "inform": [0, 3, 8], "subsitut": 0, "us": [0, 4], "wbitem": 0, "process_claim": 0, "prop_nr": 0, "qualifier_nr": 0, "parse_publ": 0, "extract": [0, 8], "doi": [0, 8], "identif": 0, "relat": 0, "identifi": 0, "mention": [0, 3], "format": 0, "arxiv": 0, "long": 0, "get_last_upd": 0, "dd": 0, "mm": 0, "yyyi": 0, "clean_package_list": 0, "table_html": 0, "raw": [0, 8], "enabl": 0, "creation": [0, 8], "split": 0, "comma": 0, "posit": 0, "parse_licens": 0, "method": [0, 8], "parse_author": 0, "parse_maintain": 0, "paramet": [0, 3, 8], "code": 0, "obtain": 0, "beautifulsoup": 0, "singl": [0, 3], "column": 0, "parse_softwar": 0, "software_str": 0, "provid": 0, "x": 0, "string": [0, 3, 8], "take": 0, "account": [0, 4], "often": 0, "uniformli": 0, "charact": 0, "separ": 0, "further": [0, 3], "squar": 0, "bracket": 0, "concret": 0, "link": [0, 3], "previous": 0, "been": [0, 3], "wikidata": [0, 3], "between": [0, 8], "round": 0, "ad": [0, 3, 8], "qualifi": [0, 3, 8], "file": [0, 1, 3, 4], "element": [0, 8], "second": 0, "regist": 0, "unequivoc": 0, "e": [0, 3, 4, 8], "thei": 0, "onli": [0, 8], "establish": 0, "whether": [0, 3], "after": 0, "refer": [0, 3], "anoth": 0, "institut": 0, "funder": 0, "etc": 0, "orcid": 0, "remov": 0, "unnecessari": 0, "which": [0, 3], "mai": 0, "mail": 0, "address": 0, "comment": 0, "within": [0, 4], "get_license_qid": 0, "license_str": 0, "same": [0, 3], "denomin": 0, "differ": 0, "uniqu": 0, "referenc": 0, "under": 0, "g": [0, 8], "artist": 0, "2": 0, "0": 0, "q14624826": 0, "get_wikidata_qid": 0, "get": [0, 3, 8], "retriev": 0, "so": 0, "get_vers": 0, "thu": 1, "feb": 1, "17": 1, "18": 1, "53": 1, "2022": 1, "alvaro": 1, "class": [1, 2, 3, 8], "aconfigpars": 1, "abstract": [1, 8], "config": [1, 3], "parse_config": 1, "valu": [1, 8], "rtype": 1, "extern": 1, "process_data": [1, 8], "dump": [1, 8], "datasourc": [1, 8], "write_data_dump": [1, 8], "control": 1, "import_al": 1, "except": 1, "fail": 1, "oper": 1, "docker": 2, "introduct": 2, "instal": 2, "without": 2, "With": 2, "build": 2, "usag": [2, 6], "diagram": 2, "launcher": 2, "script": 2, "argument": [2, 3], "cran": [2, 6], "modul": [2, 4], "mardi_import": 2, "cransourc": 2, "rpackag": 2, "polydb": [2, 6], "polydbsourc": 2, "zbmath": [2, 6], "zbmathsourc": 2, "misc": 2, "page": 2, "languag": 3, "en": 3, "de": 3, "mardicli": 3, "set": 3, "initi": 3, "configur": 3, "clientlogin": 3, "convert_claim_id": 3, "place": [3, 4], "convers": 3, "convert_entity_link": 3, "snak": 3, "unit": 3, "quantiti": 3, "globe": 3, "globecoordin": 3, "instead": 3, "wikibaseintegr": 3, "create_db_t": 3, "db": 3, "map": 3, "create_engin": 3, "sqlalchemi": 3, "engin": 3, "create_id_list_from_fil": 3, "while": 3, "where": 3, "line": 3, "path": 3, "get_local_id_by_label": 3, "entity_str": 3, "entity_typ": 3, "pid": 3, "specifi": 3, "prefix": 3, "wdt": 3, "wd": 3, "either": 3, "look": 3, "get_qualifi": 3, "empti": 3, "get_refer": 3, "get_wikidata_inform": 3, "wikidata_id": 3, "recurs": 3, "desir": 3, "should": 3, "import_claim_ent": 3, "had": 3, "import_ent": 3, "id_list": 3, "filenam": 3, "lexem": 3, "support": 3, "import_from_label": 3, "local_id": 3, "init_wikidata_pid": 3, "wikidata_pid": 3, "init_wikidata_qid": 3, "wikidata_qid": 3, "insert_id_in_db": 3, "has_all_claim": 3, "boolean": 3, "overwrite_ent": 3, "complet": 3, "need": 3, "queri": [3, 8], "wb_id_map": 3, "two": 3, "query_with_local_id": 3, "update_has_all_claim": 3, "alias": 3, "sitelink": 3, "kwarg": 3, "ani": 3, "mardiitem": 3, "entity_id": 3, "request": 3, "mediawiki": 3, "you": 3, "want": 3, "must": 3, "start": 3, "q": 3, "itement": 3, "get_qid": 3, "alia": 3, "mardipropertyent": 3, "datatyp": 3, "wikibasedatatyp": 3, "mardiproperti": 3, "get_pid": 3, "what": 4, "about": 4, "python": 4, "requir": 4, "txt": 4, "pip": 4, "Then": 4, "via": 4, "u": 4, "enforc": 4, "reinstal": 4, "modif": 4, "automat": 4, "taken": 4, "note": 4, "conveni": 4, "virtual": 4, "environ": 4, "call": 4, "python3": 4, "venv": 4, "env": 4, "bin": 4, "activ": 4, "todo": 4, "In": 4, "doc": 4, "run": 4, "make": 4, "have": 4, "findabl": 4, "To": 4, "view": 4, "open": 4, "_build": 4, "py": 6, "h": 6, "mode": 6, "openml": 6, "conf_path": 6, "wikidata_id_file_path": 6, "choic": 6, "out_dir": 8, "tag": 8, "from_dat": 8, "until_d": 8, "raw_dump_path": 8, "processed_dump_path": 8, "split_id": 8, "zb": 8, "math": 8, "get_creation_d": 8, "xml_record": 8, "xml": 8, "record": 8, "get_de_numb": 8, "parse_record": 8, "pair": 8, "overrid": 8, "csv": 8, "zbmathpubl": 8, "journal": 8, "well": 8, "get_info_from_doi": 8, "kei": 8, "crossref": 8, "document_titl": 8, "now": 8, "document": 8, "get_tag": 8, "tag_nam": 8, "namespac": 8, "fulli": 8, "parse_doi_info": 8, "val": 8, "work_info": 8, "respons": 8}, "objects": {"mardi_importer.cran": [[0, 0, 0, "-", "CRANSource"], [0, 0, 0, "-", "RPackage"]], "mardi_importer.cran.CRANSource": [[0, 1, 1, "", "CRANSource"]], "mardi_importer.cran.CRANSource.CRANSource": [[0, 2, 1, "", "create_local_entities"], [0, 3, 1, "", "packages"], [0, 2, 1, "", "pull"], [0, 2, 1, "", "push"], [0, 2, 1, "", "setup"]], "mardi_importer.cran.RPackage": [[0, 1, 1, "", "RPackage"]], "mardi_importer.cran.RPackage.RPackage": [[0, 4, 1, "", "QID"], [0, 3, 1, "", "_QID"], [0, 3, 1, "", "api"], [0, 3, 1, "", "arxiv_publications"], [0, 3, 1, "", "author"], [0, 3, 1, "", "author_pool"], [0, 3, 1, "", "authors"], [0, 2, 1, "", "clean_package_list"], [0, 2, 1, "", "create"], [0, 3, 1, "", "crossref_publications"], [0, 3, 1, "id0", "date"], [0, 3, 1, "", "dependencies"], [0, 3, 1, "", "dependency"], [0, 3, 1, "id2", "description"], [0, 2, 1, "", "exists"], [0, 2, 1, "", "get_last_update"], [0, 2, 1, "", "get_license_QID"], [0, 2, 1, "", "get_versions"], [0, 2, 1, "", "get_wikidata_QID"], [0, 3, 1, "id7", "imports"], [0, 2, 1, "", "insert_claims"], [0, 3, 1, "", "integrator"], [0, 2, 1, "", "is_updated"], [0, 4, 1, "", "item"], [0, 3, 1, "id1", "label"], [0, 3, 1, "", "license"], [0, 3, 1, "", "license_data"], [0, 3, 1, "id3", "long_description"], [0, 3, 1, "id8", "maintainer"], [0, 2, 1, "", "parse_authors"], [0, 2, 1, "", "parse_license"], [0, 2, 1, "", "parse_maintainer"], [0, 2, 1, "", "parse_publications"], [0, 2, 1, "", "parse_software"], [0, 2, 1, "", "process_claims"], [0, 2, 1, "", "pull"], [0, 2, 1, "", "update"], [0, 3, 1, "id4", "url"], [0, 3, 1, "id5", "version"], [0, 3, 1, "id6", "versions"], [0, 2, 1, "", "write"], [0, 3, 1, "", "zenodo_resources"]], "mardi_importer.importer": [[1, 0, 0, "-", "Importer"]], "mardi_importer.importer.Importer": [[1, 1, 1, "", "AConfigParser"], [1, 1, 1, "", "ADataSource"], [1, 1, 1, "", "Importer"], [1, 5, 1, "", "ImporterException"]], "mardi_importer.importer.Importer.AConfigParser": [[1, 2, 1, "", "parse_config"]], "mardi_importer.importer.Importer.ADataSource": [[1, 2, 1, "", "process_data"], [1, 2, 1, "", "pull"], [1, 2, 1, "", "push"], [1, 2, 1, "", "write_data_dump"]], "mardi_importer.importer.Importer.Importer": [[1, 2, 1, "", "import_all"]], "mardi_importer.integrator": [[3, 0, 0, "-", "MardiEntities"], [3, 0, 0, "-", "MardiIntegrator"]], "mardi_importer.integrator.MardiEntities": [[3, 1, 1, "", "MardiItemEntity"], [3, 1, 1, "", "MardiPropertyEntity"]], "mardi_importer.integrator.MardiEntities.MardiItemEntity": [[3, 2, 1, "", "get"], [3, 2, 1, "", "get_QID"], [3, 2, 1, "", "new"]], "mardi_importer.integrator.MardiEntities.MardiPropertyEntity": [[3, 2, 1, "", "get"], [3, 2, 1, "", "get_PID"], [3, 2, 1, "", "new"]], "mardi_importer.integrator.MardiIntegrator": [[3, 1, 1, "", "MardiIntegrator"]], "mardi_importer.integrator.MardiIntegrator.MardiIntegrator": [[3, 2, 1, "", "config"], [3, 2, 1, "", "convert_claim_ids"], [3, 2, 1, "", "convert_entity_links"], [3, 2, 1, "", "create_db_table"], [3, 2, 1, "", "create_engine"], [3, 2, 1, "", "create_id_list_from_file"], [3, 2, 1, "", "get_local_id_by_label"], [3, 2, 1, "", "get_qualifiers"], [3, 2, 1, "", "get_references"], [3, 2, 1, "", "get_wikidata_information"], [3, 2, 1, "", "import_claim_entities"], [3, 2, 1, "", "import_entities"], [3, 2, 1, "", "import_from_label"], [3, 2, 1, "", "init_wikidata_PID"], [3, 2, 1, "", "init_wikidata_QID"], [3, 2, 1, "", "insert_id_in_db"], [3, 2, 1, "", "overwrite_entity"], [3, 2, 1, "", "query"], [3, 2, 1, "", "query_with_local_id"], [3, 2, 1, "", "update_has_all_claims"]], "mardi_importer": [[5, 0, 0, "-", "polydb"]], "mardi_importer.zbmath": [[8, 0, 0, "-", "ZBMathSource"], [8, 0, 0, "-", "misc"]], "mardi_importer.zbmath.ZBMathSource": [[8, 1, 1, "", "ZBMathSource"]], "mardi_importer.zbmath.ZBMathSource.ZBMathSource": [[8, 2, 1, "", "create_local_entities"], [8, 2, 1, "", "get_creation_date"], [8, 2, 1, "", "get_de_number"], [8, 2, 1, "", "parse_record"], [8, 2, 1, "", "process_data"], [8, 2, 1, "", "pull"], [8, 2, 1, "", "push"], [8, 2, 1, "", "setup"], [8, 2, 1, "", "write_data_dump"]], "mardi_importer.zbmath.misc": [[8, 6, 1, "", "get_info_from_doi"], [8, 6, 1, "", "get_tag"], [8, 6, 1, "", "parse_doi_info"]]}, "objtypes": {"0": "py:module", "1": "py:class", "2": "py:method", "3": "py:attribute", "4": "py:property", "5": "py:exception", "6": "py:function"}, "objnames": {"0": ["py", "module", "Python module"], "1": ["py", "class", "Python class"], "2": ["py", "method", "Python method"], "3": ["py", "attribute", "Python attribute"], "4": ["py", "property", "Python property"], "5": ["py", "exception", "Python exception"], "6": ["py", "function", "Python function"]}, "titleterms": {"cran": 0, "modul": [0, 1, 3, 5, 8], "mardi_import": [0, 1, 3, 5, 8], "cransourc": 0, "class": 0, "rpackag": 0, "import": [1, 2, 7], "mardi": 2, "document": [2, 4], "tabl": 2, "content": 2, "manual": 2, "api": 2, "indic": 2, "integr": 3, "mardiintegr": 3, "mardient": 3, "introduct": 4, "instal": 4, "without": 4, "docker": 4, "With": 4, "build": 4, "usag": 4, "polydb": 5, "polydbsourc": 5, "launcher": 6, "script": 6, "name": 6, "argument": 6, "diagram": 7, "zbmath": 8, "zbmathsourc": 8, "misc": 8}, "envversion": {"sphinx.domains.c": 3, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 9, "sphinx.domains.index": 1, "sphinx.domains.javascript": 3, "sphinx.domains.math": 2, "sphinx.domains.python": 4, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.viewcode": 1, "sphinx.ext.todo": 2, "sphinx": 60}, "alltitles": {"CRAN module": [[0, "cran-module"]], "mardi_importer.cran.CRANSource class": [[0, "module-mardi_importer.cran.CRANSource"]], "mardi_importer.cran.RPackage class": [[0, "module-mardi_importer.cran.RPackage"]], "mardi_importer.importer module": [[1, "mardi-importer-importer-module"]], "mardi_importer.importer.Importer module": [[1, "module-mardi_importer.importer.Importer"]], "MaRDI Importer Documentation": [[2, "mardi-importer-documentation"]], "Table of contents": [[2, "table-of-contents"]], "Manual": [[2, null]], "API Documentation": [[2, null]], "Indices and tables": [[2, "indices-and-tables"]], "mardi_importer.integrator module": [[3, "mardi-importer-integrator-module"]], "mardi_importer.integrator.MardiIntegrator module": [[3, "module-mardi_importer.integrator.MardiIntegrator"]], "mardi_importer.integrator.MardiEntities module": [[3, "module-mardi_importer.integrator.MardiEntities"]], "Introduction": [[4, "introduction"]], "Installation": [[4, "installation"]], "Without Docker": [[4, "without-docker"]], "With Docker": [[4, "with-docker"]], "Build documentation": [[4, "build-documentation"]], "Usage": [[4, "usage"]], "mardi_importer.polydb module": [[5, "mardi-importer-polydb-module"]], "mardi_importer.polydb.PolyDBSource module": [[5, "module-mardi_importer.polydb"]], "Launcher script": [[6, "launcher-script"]], "Named Arguments": [[6, "named-arguments"]], "Importer diagram": [[7, "importer-diagram"]], "mardi_importer.zbmath module": [[8, "mardi-importer-zbmath-module"]], "mardi_importer.zbmath.ZBMathSource module": [[8, "module-mardi_importer.zbmath.ZBMathSource"]], "mardi_importer.zbmath.misc module": [[8, "module-mardi_importer.zbmath.misc"]]}, "indexentries": {"cransource (class in mardi_importer.cran.cransource)": [[0, "mardi_importer.cran.CRANSource.CRANSource"]], "qid (mardi_importer.cran.rpackage.rpackage property)": [[0, "mardi_importer.cran.RPackage.RPackage.QID"]], "rpackage (class in mardi_importer.cran.rpackage)": [[0, "mardi_importer.cran.RPackage.RPackage"]], "_qid (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "mardi_importer.cran.RPackage.RPackage._QID"]], "api (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "mardi_importer.cran.RPackage.RPackage.api"]], "arxiv_publications (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "mardi_importer.cran.RPackage.RPackage.arxiv_publications"]], "author (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "mardi_importer.cran.RPackage.RPackage.author"]], "author_pool (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "mardi_importer.cran.RPackage.RPackage.author_pool"]], "authors (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "mardi_importer.cran.RPackage.RPackage.authors"]], "clean_package_list() (mardi_importer.cran.rpackage.rpackage method)": [[0, "mardi_importer.cran.RPackage.RPackage.clean_package_list"]], "create() (mardi_importer.cran.rpackage.rpackage method)": [[0, "mardi_importer.cran.RPackage.RPackage.create"]], "create_local_entities() (mardi_importer.cran.cransource.cransource method)": [[0, "mardi_importer.cran.CRANSource.CRANSource.create_local_entities"]], "crossref_publications (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "mardi_importer.cran.RPackage.RPackage.crossref_publications"]], "date (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "id0"], [0, "mardi_importer.cran.RPackage.RPackage.date"]], "dependencies (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "mardi_importer.cran.RPackage.RPackage.dependencies"]], "dependency (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "mardi_importer.cran.RPackage.RPackage.dependency"]], "description (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "id2"], [0, "mardi_importer.cran.RPackage.RPackage.description"]], "exists() (mardi_importer.cran.rpackage.rpackage method)": [[0, "mardi_importer.cran.RPackage.RPackage.exists"]], "get_last_update() (mardi_importer.cran.rpackage.rpackage method)": [[0, "mardi_importer.cran.RPackage.RPackage.get_last_update"]], "get_license_qid() (mardi_importer.cran.rpackage.rpackage method)": [[0, "mardi_importer.cran.RPackage.RPackage.get_license_QID"]], "get_versions() (mardi_importer.cran.rpackage.rpackage method)": [[0, "mardi_importer.cran.RPackage.RPackage.get_versions"]], "get_wikidata_qid() (mardi_importer.cran.rpackage.rpackage method)": [[0, "mardi_importer.cran.RPackage.RPackage.get_wikidata_QID"]], "imports (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "id7"], [0, "mardi_importer.cran.RPackage.RPackage.imports"]], "insert_claims() (mardi_importer.cran.rpackage.rpackage method)": [[0, "mardi_importer.cran.RPackage.RPackage.insert_claims"]], "integrator (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "mardi_importer.cran.RPackage.RPackage.integrator"]], "is_updated() (mardi_importer.cran.rpackage.rpackage method)": [[0, "mardi_importer.cran.RPackage.RPackage.is_updated"]], "item (mardi_importer.cran.rpackage.rpackage property)": [[0, "mardi_importer.cran.RPackage.RPackage.item"]], "label (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "id1"], [0, "mardi_importer.cran.RPackage.RPackage.label"]], "license (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "mardi_importer.cran.RPackage.RPackage.license"]], "license_data (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "mardi_importer.cran.RPackage.RPackage.license_data"]], "long_description (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "id3"], [0, "mardi_importer.cran.RPackage.RPackage.long_description"]], "maintainer (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "id8"], [0, "mardi_importer.cran.RPackage.RPackage.maintainer"]], "mardi_importer.cran.cransource": [[0, "module-mardi_importer.cran.CRANSource"]], "mardi_importer.cran.rpackage": [[0, "module-mardi_importer.cran.RPackage"]], "module": [[0, "module-mardi_importer.cran.CRANSource"], [0, "module-mardi_importer.cran.RPackage"], [1, "module-mardi_importer.importer.Importer"], [3, "module-mardi_importer.integrator.MardiEntities"], [3, "module-mardi_importer.integrator.MardiIntegrator"], [5, "module-mardi_importer.polydb"], [8, "module-mardi_importer.zbmath.ZBMathSource"], [8, "module-mardi_importer.zbmath.misc"]], "packages (mardi_importer.cran.cransource.cransource attribute)": [[0, "mardi_importer.cran.CRANSource.CRANSource.packages"]], "parse_authors() (mardi_importer.cran.rpackage.rpackage method)": [[0, "mardi_importer.cran.RPackage.RPackage.parse_authors"]], "parse_license() (mardi_importer.cran.rpackage.rpackage method)": [[0, "mardi_importer.cran.RPackage.RPackage.parse_license"]], "parse_maintainer() (mardi_importer.cran.rpackage.rpackage method)": [[0, "mardi_importer.cran.RPackage.RPackage.parse_maintainer"]], "parse_publications() (mardi_importer.cran.rpackage.rpackage method)": [[0, "mardi_importer.cran.RPackage.RPackage.parse_publications"]], "parse_software() (mardi_importer.cran.rpackage.rpackage method)": [[0, "mardi_importer.cran.RPackage.RPackage.parse_software"]], "process_claims() (mardi_importer.cran.rpackage.rpackage method)": [[0, "mardi_importer.cran.RPackage.RPackage.process_claims"]], "pull() (mardi_importer.cran.cransource.cransource method)": [[0, "mardi_importer.cran.CRANSource.CRANSource.pull"]], "pull() (mardi_importer.cran.rpackage.rpackage method)": [[0, "mardi_importer.cran.RPackage.RPackage.pull"]], "push() (mardi_importer.cran.cransource.cransource method)": [[0, "mardi_importer.cran.CRANSource.CRANSource.push"]], "setup() (mardi_importer.cran.cransource.cransource method)": [[0, "mardi_importer.cran.CRANSource.CRANSource.setup"]], "update() (mardi_importer.cran.rpackage.rpackage method)": [[0, "mardi_importer.cran.RPackage.RPackage.update"]], "url (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "id4"], [0, "mardi_importer.cran.RPackage.RPackage.url"]], "version (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "id5"], [0, "mardi_importer.cran.RPackage.RPackage.version"]], "versions (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "id6"], [0, "mardi_importer.cran.RPackage.RPackage.versions"]], "write() (mardi_importer.cran.rpackage.rpackage method)": [[0, "mardi_importer.cran.RPackage.RPackage.write"]], "zenodo_resources (mardi_importer.cran.rpackage.rpackage attribute)": [[0, "mardi_importer.cran.RPackage.RPackage.zenodo_resources"]], "aconfigparser (class in mardi_importer.importer.importer)": [[1, "mardi_importer.importer.Importer.AConfigParser"]], "adatasource (class in mardi_importer.importer.importer)": [[1, "mardi_importer.importer.Importer.ADataSource"]], "importer (class in mardi_importer.importer.importer)": [[1, "mardi_importer.importer.Importer.Importer"]], "importerexception": [[1, "mardi_importer.importer.Importer.ImporterException"]], "import_all() (mardi_importer.importer.importer.importer method)": [[1, "mardi_importer.importer.Importer.Importer.import_all"]], "mardi_importer.importer.importer": [[1, "module-mardi_importer.importer.Importer"]], "parse_config() (mardi_importer.importer.importer.aconfigparser method)": [[1, "mardi_importer.importer.Importer.AConfigParser.parse_config"]], "process_data() (mardi_importer.importer.importer.adatasource method)": [[1, "mardi_importer.importer.Importer.ADataSource.process_data"]], "pull() (mardi_importer.importer.importer.adatasource method)": [[1, "mardi_importer.importer.Importer.ADataSource.pull"]], "push() (mardi_importer.importer.importer.adatasource method)": [[1, "mardi_importer.importer.Importer.ADataSource.push"]], "write_data_dump() (mardi_importer.importer.importer.adatasource method)": [[1, "mardi_importer.importer.Importer.ADataSource.write_data_dump"]], "mardiintegrator (class in mardi_importer.integrator.mardiintegrator)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator"]], "mardiitementity (class in mardi_importer.integrator.mardientities)": [[3, "mardi_importer.integrator.MardiEntities.MardiItemEntity"]], "mardipropertyentity (class in mardi_importer.integrator.mardientities)": [[3, "mardi_importer.integrator.MardiEntities.MardiPropertyEntity"]], "config() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.config"]], "convert_claim_ids() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.convert_claim_ids"]], "convert_entity_links() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.convert_entity_links"]], "create_db_table() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.create_db_table"]], "create_engine() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.create_engine"]], "create_id_list_from_file() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.create_id_list_from_file"]], "get() (mardi_importer.integrator.mardientities.mardiitementity method)": [[3, "mardi_importer.integrator.MardiEntities.MardiItemEntity.get"]], "get() (mardi_importer.integrator.mardientities.mardipropertyentity method)": [[3, "mardi_importer.integrator.MardiEntities.MardiPropertyEntity.get"]], "get_pid() (mardi_importer.integrator.mardientities.mardipropertyentity method)": [[3, "mardi_importer.integrator.MardiEntities.MardiPropertyEntity.get_PID"]], "get_qid() (mardi_importer.integrator.mardientities.mardiitementity method)": [[3, "mardi_importer.integrator.MardiEntities.MardiItemEntity.get_QID"]], "get_local_id_by_label() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.get_local_id_by_label"]], "get_qualifiers() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.get_qualifiers"]], "get_references() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.get_references"]], "get_wikidata_information() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.get_wikidata_information"]], "import_claim_entities() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.import_claim_entities"]], "import_entities() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.import_entities"]], "import_from_label() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.import_from_label"]], "init_wikidata_pid() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.init_wikidata_PID"]], "init_wikidata_qid() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.init_wikidata_QID"]], "insert_id_in_db() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.insert_id_in_db"]], "mardi_importer.integrator.mardientities": [[3, "module-mardi_importer.integrator.MardiEntities"]], "mardi_importer.integrator.mardiintegrator": [[3, "module-mardi_importer.integrator.MardiIntegrator"]], "new() (mardi_importer.integrator.mardientities.mardiitementity method)": [[3, "mardi_importer.integrator.MardiEntities.MardiItemEntity.new"]], "new() (mardi_importer.integrator.mardientities.mardipropertyentity method)": [[3, "mardi_importer.integrator.MardiEntities.MardiPropertyEntity.new"]], "overwrite_entity() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.overwrite_entity"]], "query() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.query"]], "query_with_local_id() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.query_with_local_id"]], "update_has_all_claims() (mardi_importer.integrator.mardiintegrator.mardiintegrator method)": [[3, "mardi_importer.integrator.MardiIntegrator.MardiIntegrator.update_has_all_claims"]], "mardi_importer.polydb": [[5, "module-mardi_importer.polydb"]], "zbmathsource (class in mardi_importer.zbmath.zbmathsource)": [[8, "mardi_importer.zbmath.ZBMathSource.ZBMathSource"]], "create_local_entities() (mardi_importer.zbmath.zbmathsource.zbmathsource method)": [[8, "mardi_importer.zbmath.ZBMathSource.ZBMathSource.create_local_entities"]], "get_creation_date() (mardi_importer.zbmath.zbmathsource.zbmathsource method)": [[8, "mardi_importer.zbmath.ZBMathSource.ZBMathSource.get_creation_date"]], "get_de_number() (mardi_importer.zbmath.zbmathsource.zbmathsource method)": [[8, "mardi_importer.zbmath.ZBMathSource.ZBMathSource.get_de_number"]], "get_info_from_doi() (in module mardi_importer.zbmath.misc)": [[8, "mardi_importer.zbmath.misc.get_info_from_doi"]], "get_tag() (in module mardi_importer.zbmath.misc)": [[8, "mardi_importer.zbmath.misc.get_tag"]], "mardi_importer.zbmath.zbmathsource": [[8, "module-mardi_importer.zbmath.ZBMathSource"]], "mardi_importer.zbmath.misc": [[8, "module-mardi_importer.zbmath.misc"]], "parse_doi_info() (in module mardi_importer.zbmath.misc)": [[8, "mardi_importer.zbmath.misc.parse_doi_info"]], "parse_record() (mardi_importer.zbmath.zbmathsource.zbmathsource method)": [[8, "mardi_importer.zbmath.ZBMathSource.ZBMathSource.parse_record"]], "process_data() (mardi_importer.zbmath.zbmathsource.zbmathsource method)": [[8, "mardi_importer.zbmath.ZBMathSource.ZBMathSource.process_data"]], "pull() (mardi_importer.zbmath.zbmathsource.zbmathsource method)": [[8, "mardi_importer.zbmath.ZBMathSource.ZBMathSource.pull"]], "push() (mardi_importer.zbmath.zbmathsource.zbmathsource method)": [[8, "mardi_importer.zbmath.ZBMathSource.ZBMathSource.push"]], "setup() (mardi_importer.zbmath.zbmathsource.zbmathsource method)": [[8, "mardi_importer.zbmath.ZBMathSource.ZBMathSource.setup"]], "write_data_dump() (mardi_importer.zbmath.zbmathsource.zbmathsource method)": [[8, "mardi_importer.zbmath.ZBMathSource.ZBMathSource.write_data_dump"]]}})
\ No newline at end of file
diff --git a/tech.html b/tech.html
new file mode 100644
index 0000000..e3c9601
--- /dev/null
+++ b/tech.html
@@ -0,0 +1,123 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/zbmath.html b/zbmath.html
new file mode 100644
index 0000000..db696e8
--- /dev/null
+++ b/zbmath.html
@@ -0,0 +1,302 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Importer diagram
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ mardi_importer.zbmath module
+mardi_importer.zbmath.ZBMathSource module
+-
+
- +class mardi_importer.zbmath.ZBMathSource.ZBMathSource(out_dir, tags, from_date=None, until_date=None, raw_dump_path=None, processed_dump_path=None, split_id=None)[source] +
Bases:
+ADataSourceReads data from zb math API.
+ + +-
+
- +get_creation_date(xml_record)[source] +
Get creation date from xml record.
+-
+
- Parameters: +
xml_record (xml element) – record returned by zbMath API
+
+- Returns: +
creation date
+
+- Return type: +
string
+
+
-
+
- +get_de_number(xml_record)[source] +
Get zbMath id from xml record.
+-
+
- Parameters: +
xml_record (xml element) – record returned by zbMath API
+
+- Returns: +
zbMath ID
+
+- Return type: +
string
+
+
-
+
- +parse_record(xml_record)[source] +
Parse xml record from zbMath API.
+-
+
- Parameters: +
xml_record (xml element) – record returned by zbMath API
+
+- Returns: +
dict of (tag,value) pairs extracted from xml_record
+
+- Return type: +
dict
+
+
-
+
- +process_data()[source] +
Overrides abstract method. +Reads a raw zbMath data dump and processes it, then saves it as a csv.
+
mardi_importer.zbmath.misc module
+-
+
- +mardi_importer.zbmath.misc.get_info_from_doi(doi, key)[source] +
Query crossref API for DOI information.
+-
+
- Parameters: +
-
+
doi – doi
+key – document_title only for now
+
+- Returns: +
document title
+
+- Return type: +
title
+
+
-
+
- +mardi_importer.zbmath.misc.get_tag(tag_name, namespace)[source] +
Returns a fully qualified tag name.
+-
+
- Parameters: +
-
+
tag_name (string) – name of tag, e.g. author
+namespace (string) – namespace URL of a namespace
+
+
-
+
- +mardi_importer.zbmath.misc.parse_doi_info(val, work_info)[source] +
Function to extract information returned by a doi query for a specific tag.
+-
+
- Parameters: +
-
+
val (string) – tag, e.g. author
+work_info (dict) – information from doi query response
+
+- Returns: +
information for specific tag, None if not found
+
+- Return type: +
string
+
+