Skip to content

Commit

Permalink
Add files via upload
Browse files Browse the repository at this point in the history
  • Loading branch information
@ssp24
ssp24 authored Aug 26, 2024
1 parent 23fc9d5 commit 1a34323
Showing 1 changed file with 346 additions and 0 deletions.
346 changes: 346 additions & 0 deletions workshops/SRU_get_data.ipynb
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,346 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "47549bfd-d4a1-4c0a-b044-cb526c1fa50b",
"metadata": {},
"source": [
"# Datenbezug über SRU und Umwandlung der Inhalte in eine Tabelle:"
]
},
{
"cell_type": "markdown",
"id": "9abe312c-b515-4330-b7cf-881a104ef632",
"metadata": {},
"source": [
"## Grundlagen zur SRU-Schnittstelle der DNB:\n",
"\n",
"Die DNB bietet Ihre Daten auf drei verschiedene \"Kataloge\" gesplittet an, von denen immer einer für eine Abfrage ausgewählt werden muss. Dies geschieht über eine Erweiterung der Base-URL. Zur Verfügung stehen folgende Kataloge:\n",
"\n",
"- Katalog der Deutschen Nationalbibliothek (DNB) - hierin befinden sich die Titeldaten\n",
"- Katalog des Deutschen Musikarchivs (DMA) - Datensätze des Deutschen Musikarchivs\n",
"- Katalog der Gemeinsamen Normdatei (GND) - hierin befinden sich die Normdaten\n",
"\n",
"Die erweiterungen für die URL sind folgende:\n",
"\n",
"- DNB: https://services.dnb.de/sru/dnb\n",
"- DMA: https://services.dnb.de/sru/dnb.dma\n",
"- GND: https://services.dnb.de/sru/authorities\n",
"\n",
"Werden die jeweiligen Bereiche ohne weitere Spezifikationen abgefragt, senden sie eine \"Explain-Response\" in XML zurück. \n",
"Mit Hilfe der Bibliothek BeautifulSoup kann die Antwort direkt in XML umgewandelt werden.\n",
"\n",
"[Weitere Informationen und Dokumentation zur SRU-Schnittstelle](https://www.dnb.de/sru)\n",
"\n",
"### Laden benötigter Zusatzbibliotheken: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "51bb5a4b-e64e-41e7-ad2a-b96b1de4a11e",
"metadata": {},
"outputs": [],
"source": [
"import requests\n",
"from bs4 import BeautifulSoup as soup\n",
"import unicodedata\n",
"from lxml import etree\n",
"import pandas as pd"
]
},
{
"cell_type": "markdown",
"id": "87404a94-7132-4f99-8279-2a25e179a0e0",
"metadata": {},
"source": [
"## SRU-Anfrage"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1ee29b90-36ed-4a73-b52f-9bc54d04d5ff",
"metadata": {},
"outputs": [],
"source": [
"#Base-URL für den Katalog der DNB (Titeldaten):\n",
"dnb = requests.get(\"https://services.dnb.de/sru/dnb\")\n",
"\n",
"#Einfache Amfrage an die Schnittstelle ohne Suche \n",
"response = soup(dnb.content, features=\"xml\") #Parsen der Schnittstellenantwort als xml\n",
"print(response.prettify()[0:500]) #Einschränken der Ausgabe auf die ersten 500 Zeichen"
]
},
{
"cell_type": "markdown",
"id": "ffc1a87a-e58f-4fca-9931-f6cf4ef2e015",
"metadata": {},
"source": [
"Für eine Suchanfrage an die Daten der DNB wird nun zunächst über die Wahl der URL der Katalog definiert. Mit Hilfe der Variable parameter werdem dann alle weiteren Parameter, die die SRU-Schnittstelle benötigt, übergeben."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c01637fc-a150-4895-81ed-c9eca69bff89",
"metadata": {},
"outputs": [],
"source": [
"#Base-URL:\n",
"dnb_url = \"https://services.dnb.de/sru/dnb\"\n",
"\n",
"#Parameter, die mit der Anfrage übergeben werden sollen: \n",
"parameter = {'version' : '1.1' , \n",
" 'operation' : 'searchRetrieve' , \n",
" 'query' : 'Klimawandel and jhr=2010', \n",
" 'recordSchema' : 'MARC21-xml'} \n",
"\n",
"#Abfrage der Schnittstelle:\n",
"r = requests.get(dnb_url, params = parameter)\n",
"#Parsen der Antwort in xml:\n",
"response = soup(r.content, features=\"xml\")\n",
"#Überführen der einzelnen, in der Antwort enthaltenen Datensätze/Treffer ('Records') in eine Liste:\n",
"result = response.find_all('record', {'type':'Bibliographic'})\n",
"print(response.prettify()[0:750])"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e013ff23-cc54-4c41-8d17-d60c872239c1",
"metadata": {},
"outputs": [],
"source": [
"number = response.find('numberOfRecords')\n",
"print(number.text, 'Ergebnisse')\n",
"print(len(result))"
]
},
{
"cell_type": "markdown",
"id": "4e071f81-62bb-4c6e-bf4b-d14cfe11eb7b",
"metadata": {},
"source": [
"Wichtig: Die SRU-Schnittstelle liefert per default zunächst immer nur 10 Datensätze auf einmal aus, maximal 100. "
]
},
{
"cell_type": "markdown",
"id": "45fd84cb-8cef-4108-a54b-75cb0b3a9fa1",
"metadata": {},
"source": [
"### Umwandlung in eine Funktion:"
]
},
{
"cell_type": "markdown",
"id": "ad50c950-7c48-4d0b-a262-6bac2f1f7f88",
"metadata": {},
"source": [
"Funktion, die dank while-Schleife alle Datensätze der Anfrage sammelt: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c09f414b-76e6-4715-9180-9d4d0a11fda3",
"metadata": {},
"outputs": [],
"source": [
"def dnb_sru(query):\n",
" \n",
" base_url = \"https://services.dnb.de/sru/dnb\"\n",
" params = {'recordSchema' : 'MARC21-xml',\n",
" 'operation': 'searchRetrieve',\n",
" 'version': '1.1',\n",
" 'maximumRecords': '100',\n",
" 'query': query\n",
" }\n",
" \n",
" i = 0 \n",
" records = []\n",
"\n",
" while i % 100 == 0: \n",
" params.update({'startRecord': i}) \n",
" r = requests.get(base_url, params=params)\n",
" xml = soup(r.content, features=\"xml\")\n",
" result = xml.find_all('record', {'type':'Bibliographic'})\n",
" records+=result\n",
" num_results = len(result)\n",
" if num_results != 100:\n",
" break\n",
" else: \n",
" i += 100 \n",
" \n",
" return records"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8f7f4b07-e684-4e0a-84dc-abc7cb0ad1e1",
"metadata": {},
"outputs": [],
"source": [
"records = dnb_sru('tit=Klimawandel and jhr=2010')\n",
"print(len(records), 'Ergebnisse')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7446fb3e-eaf3-494f-a571-8b6e301b6b55",
"metadata": {},
"outputs": [],
"source": [
"print(records[5])"
]
},
{
"cell_type": "markdown",
"id": "72b8f418-4d6d-494f-bbf0-d1b402134aab",
"metadata": {},
"source": [
"## Inhalte aus dem XML extrahieren: "
]
},
{
"cell_type": "markdown",
"id": "b7b92abd-1f48-4c57-8068-40ee6ec0e311",
"metadata": {},
"source": [
"### Simple Version: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5a28cefc-851d-42a2-9806-0aa7e7ec3cb3",
"metadata": {},
"outputs": [],
"source": [
"def parse_record(record):\n",
" \n",
" ns = {\"marc\":\"http://www.loc.gov/MARC21/slim\"}\n",
" xml = etree.fromstring(unicodedata.normalize(\"NFC\", str(record)))\n",
" \n",
" #idn\n",
" idn = xml.xpath(\"marc:controlfield[@tag = '001']\", namespaces=ns)\n",
" try:\n",
" idn = idn[0].text\n",
" except:\n",
" idn = 'fail'\n",
" \n",
" # titel\n",
" titel = xml.xpath(\"marc:datafield[@tag = '245']/marc:subfield[@code = 'a']\", namespaces=ns)\n",
" try:\n",
" titel = titel[0].text\n",
" except:\n",
" titel = \"unkown\"\n",
" \n",
" meta_dict = {\"idn\":idn, \"titel\":titel}\n",
" \n",
" return meta_dict"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "177ad51e-5f79-40d5-a9f4-9085f44d2264",
"metadata": {},
"outputs": [],
"source": [
"output = [parse_record(record) for record in records]\n",
"df = pd.DataFrame(output)\n",
"df"
]
},
{
"cell_type": "markdown",
"id": "f2a1cd93-0481-44a8-8e5d-4ee173237963",
"metadata": {},
"source": [
"### Alternative: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b1c8e1f1-c9c5-4eab-b070-6f85e53365b1",
"metadata": {},
"outputs": [],
"source": [
"def alternative_parsing(record):\n",
" ns = {\"marc\": \"http://www.loc.gov/MARC21/slim\"}\n",
" xml = etree.fromstring(unicodedata.normalize(\"NFC\", str(record)))\n",
" \n",
" def extract_text(xpath_query):\n",
" fields = xml.xpath(xpath_query, namespaces=ns)\n",
" if fields:\n",
" return \"; \".join(field.text for field in fields if field.text)\n",
" return \"unknown\"\n",
"\n",
" idn = extract_text(\"marc:controlfield[@tag='001']\")\n",
" titel = extract_text(\"marc:datafield[@tag='245']/marc:subfield[@code='a']\")\n",
" #author = extract_text(\"marc:datafield[@tag='100']/marc:subfield[@code='a']\")\n",
" #author_rela = extract_text(\"marc:datafield[@tag='100']/marc:subfield[@code='e']\")\n",
" #add_author = extract_text(\"marc:datafield[@tag='700']/marc:subfield[@code='a']\")\n",
" #add_author_rela = extract_text(\"marc:datafield[@tag='700']/marc:subfield[@code='e']\")\n",
" \n",
" return {\"idn\": idn, \"titel\": titel, \"author\": author, \"rela\":author_rela, \"additional author\":add_author, \"added_rela\": add_author_rela}"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1a7c5e05-1648-4fa9-88ae-fabec82c5ec8",
"metadata": {},
"outputs": [],
"source": [
"result = [alternative_parsing(record) for record in records]\n",
"df2 = pd.DataFrame(result)\n",
"df2"
]
},
{
"cell_type": "markdown",
"id": "8a4035fd-5e5b-4cfb-b514-c694aac1f6c2",
"metadata": {},
"source": [
"## Speichern: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bcc6a9a9-1b6b-43fd-af2e-1665082eb2cc",
"metadata": {},
"outputs": [],
"source": [
"df.to_csv(\"SRU_Titel.csv\", index=False)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.4"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

0 comments on commit 1a34323

Please sign in to comment.