-
Notifications
You must be signed in to change notification settings - Fork 4
/
datasetd.1.html
246 lines (244 loc) · 8.01 KB
/
datasetd.1.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
<!DOCTYPE html>
<html>
<head>
<title>Dataset Project</title>
<link href='https://fonts.googleapis.com/css?family=Open+Sans' rel='stylesheet' type='text/css'>
<link rel="stylesheet" href="https://caltechlibrary.github.io/css/site.css">
</head>
<body>
<header>
<a href="http://library.caltech.edu" title="link to Caltech Library Homepage"><img src="https://caltechlibrary.github.io/assets/liblogo.gif" alt="Caltech Library logo"></a>
</header>
<nav>
<ul>
<li><a href="/">Home</a></li>
<li><a href="index.html">README</a></li>
<li><a href="LICENSE">LICENSE</a></li>
<li><a href="install.html">INSTALL</a></li>
<li><a href="user_manual.html">User Manual</a></li>
<li><a href="about.html">About</a></li>
<li><a href="search.html">Search</a></li>
<li><a href="https://github.com/caltechlibrary/dataset">GitHub</a></li>
</ul>
</nav>
<section>
<h1 id="name">NAME</h1>
<p>datasetd</p>
<h1 id="synopsis">SYNOPSIS</h1>
<p>datasetd <a href="#options">OPTIONS</a> SETTINGS_FILE</p>
<h1 id="description">DESCRIPTION</h1>
<p>datasetd provides a web service for one or more dataset collections.
Requires the collections to exist (e.g. created previously with the
dataset cli). It requires a settings JSON or YAML file that decribes the
web service configuration and permissions per collection that are
available via the web service.</p>
<h1 id="options">OPTIONS</h1>
<dl>
<dt>-help</dt>
<dd>
display detailed help
</dd>
<dt>-license</dt>
<dd>
display license
</dd>
<dt>-version</dt>
<dd>
display version
</dd>
<dt>-debug</dt>
<dd>
log debug information
</dd>
</dl>
<h1 id="settings_file">SETTINGS_FILE</h1>
<p>The settings files provides datasetd with the configuration of the
service web service and associated dataset collection(s).</p>
<p>It can be writen as either a JSON or YAML file. If it is a YAML file
you should use the “.yaml” extension so that datasetd will correctly
parse the YAML.</p>
<p>The top level YAML attributes are</p>
<dl>
<dt>host</dt>
<dd>
(required) hostname a port for the web service to listen on,
e.g. localhost:8485
</dd>
<dt>htdocs</dt>
<dd>
(optional) if set static content will be serviced based on this path.
This is a good place to implement a browser side UI in HTML, CSS and
JavaScript.
</dd>
<dt>collections</dt>
<dd>
(required) A list of dataset collections that will be supported with
this web service. The dataset collections can be pairtrees or SQL
stored. The latter is preferred for web access to avoid problems of
write collisions.
</dd>
</dl>
<p>The collections object is a list of configuration objects. The
configuration attributes you should supply are as follows.</p>
<dl>
<dt>dataset</dt>
<dd>
(required) The path to the dataset collection you are providing a web
API to.
</dd>
<dt>query</dt>
<dd>
(optional) is map of query name to SQL statement. A POST is used to
access the query (i.e. a POST To the path
“<code>/api/<COLLECTION_NAME>/query/<QUERY_NAME></code>”)
The parameters submitted in the post are passed to the SQL statement.
NOTE: Only dataset collections using a SQL store are supported. The SQL
needs to conform the SQL dialect of the store being used (e.g. MySQL,
Postgres, SQLite3). The SQL statement functions with the same contraints
of dsquery SQL statements. The SQL statement is defined as a YAML text
blog.
</dd>
</dl>
<h2 id="api-permissions">API Permissions</h2>
<p>The following are permissioning attributes for the collection. These
are global to the collection and by default are set to false. A read
only API would normally only include “keys” and “read” attributes set to
true.</p>
<dl>
<dt>keys</dt>
<dd>
(optional, default false) allow object keys to be listed
</dd>
<dt>create</dt>
<dd>
(optional, default false) allow object creation through a POST to the
web API
</dd>
<dt>read</dt>
<dd>
(optional, default false) allow object to be read through a GET from the
web API
</dd>
<dt>update</dt>
<dd>
(optional, default false) allow object updates through a PUT to the web
API.
</dd>
<dt>delete</dt>
<dd>
(optional, default false) allow object deletion through a DELETE to the
web API.
</dd>
<dt>attachments</dt>
<dd>
(optional, default false) list object attachments through a GET to the
web API.
</dd>
<dt>attach</dt>
<dd>
(optional, default false) Allow adding attachments through a POST to the
web API.
</dd>
<dt>retrieve</dt>
<dd>
(optional, default false) Allow retrieving attachments through a GET to
the web API.
</dd>
<dt>prune</dt>
<dd>
(optional, default false) Allow removing attachments through a DELETE to
the web API.
</dd>
<dt>frame_read</dt>
<dd>
(optional, default false) Allow reading frames from a GET to the web
API.
</dd>
<dt>frame_write</dt>
<dd>
(optional, default false) Allow writing new frames through a POST to the
web API.
</dd>
<dt>versions</dt>
<dd>
(optional, default false) Allow setting versioning of attachments via
POST to the web API.
</dd>
</dl>
<h1 id="examples">EXAMPLES</h1>
<p>Starting up the web service</p>
<pre><code> datasetd settings.yaml</code></pre>
<p>In this example we cover a short life cycle of a collection called
“t1.ds”. We need to create a “settings.json” file and an empty dataset
collection. Once ready you can run the datasetd service to interact with
the collection via cURL.</p>
<p>To create the dataset collection we use the “dataset” command and the
“vi” text edit (use can use your favorite text editor instead of
vi).</p>
<pre><code> createdb t1
dataset init t1.ds \
"postgres://$PGUSER:$PGPASSWORD@/t1?sslmode=disable"
vi settings.yaml</code></pre>
<p>You can create the “settings.yaml” with this Bash script. I’ve
created an htdocs directory to hold the static content to interact with
the dataset web service.</p>
<pre><code>mkdir htdocs
cat <<EOT >settings.yaml
host: localhost:8485
htdocs: htdocs
collections:
# Each collection is an object. The path prefix is
# /api/<dataset_name>/...
- dataset: t1.ds
# What follows are object level permissions
keys: true
create: true
read: true
update: true
delete: true
# These are attachment related permissions
attachments: true
attach: true
retrieve: true
prune: true
# These are frame level permissions
frame_read: true
frame_write: true
# This sets versioning behavior
versions: true
EOT</code></pre>
<p>Now we can run datasetd and make the dataset collection available via
HTTP.</p>
<pre><code> datasetd settings.yaml</code></pre>
<p>You should now see the start up message and any log information
display to the console. You should open a new shell sessions and try the
following.</p>
<p>We can now use cURL to post the document to the
“api//t1.ds/object/one” end point.</p>
<pre><code> curl -X POST http://localhost:8485/api/t1.ds/object/one \
-d '{"one": 1}'</code></pre>
<p>Now we can list the keys available in our collection.</p>
<pre><code> curl http://localhost:8485/api/t1.ds/keys</code></pre>
<p>We should see “one” in the response. If so we can try reading it.</p>
<pre><code> curl http://localhost:8485/api/t1.ds/read/one</code></pre>
<p>That should display our JSON document. Let’s try updating (replacing)
it.</p>
<pre><code> curl -X POST http://localhost:8485/api/t1.ds/object/one \
-d '{"one": 1, "two": 2}'</code></pre>
<p>If you read it back you should see the updated record. Now lets try
deleting it.</p>
<pre><code> curl http://localhost:8485/api/t1.ds/object/one</code></pre>
<p>List the keys and you should see that “one” is not longer there.</p>
<pre><code> curl http://localhost:8485/api/t1.ds/keys</code></pre>
<p>In the shell session where datasetd is running press “ctr-C” to
terminate the service.</p>
<p>datasetd 2.1.23</p>
</section>
<footer>
<span>© 2022 <a href="https://www.library.caltech.edu/copyright">Caltech Library</a></span>
<address>1200 E California Blvd, Mail Code 1-32, Pasadena, CA 91125-3200</address>
<span><a href="mailto:[email protected]">Email Us</a></span>
<span>Phone: <a href="tel:+1-626-395-3405">(626)395-3405</a></span>
</footer>
</body>
</html>