Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Links for Hugo #5531

Closed
wants to merge 5 commits into from
Closed

Fix Links for Hugo #5531

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
876679d
Remove the .md suffix from docs
jameskerr Dec 12, 2024
ea6e211
Remove .md extension
jameskerr Dec 12, 2024
77ad42e
Remove .md
jameskerr Dec 12, 2024
65c79f4
Remove .md
jameskerr Dec 12, 2024
e53670d
remove _index
jameskerr Dec 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 20 additions & 20 deletions docs/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,17 @@ heading: SuperDB
---

SuperDB offers a new approach that makes it easier to manipulate and manage
your data. With its [super-structured data model](formats/_index.md#2-a-super-structured-pattern),
your data. With its [super-structured data model](formats#2-a-super-structured-pattern),
messy JSON data can easily be given the fully-typed precision of relational tables
without giving up JSON's uncanny ability to represent eclectic data.

## Getting Started

Trying out SuperDB is easy: just [install](install.md) the command-line tool
[`super`](commands/super.md) and run through the [tutorial](tutorials/zq.md).
Trying out SuperDB is easy: just [install](install) the command-line tool
[`super`](commands/super) and run through the [tutorial](tutorials/zq).

Compared to putting JSON data in a relational column, the
[super-structured data model](formats/zed.md) makes it really easy to
[super-structured data model](formats/zed) makes it really easy to
mash up JSON with your relational tables. The `super` command is a little
like [DuckDB](https://duckdb.org/) and a little like
[`jq`](https://stedolan.github.io/jq/) but super-structured data ties the
Expand All @@ -25,44 +25,44 @@ For a non-technical user, SuperDB is as easy to use as web search
while for a technical user, SuperDB exposes its technical underpinnings
in a gradual slope, providing as much detail as desired,
packaged up in the easy-to-understand
[Super JSON data format](formats/jsup.md) and
[SuperPipe language](language/_index.md).
[Super JSON data format](formats/jsup) and
[SuperPipe language](language).

While `super` and its accompanying data formats are production quality, the project's
[SuperDB data lake](commands/super-db.md) is a bit earlier in development.
[SuperDB data lake](commands/super-db) is a bit earlier in development.

## Terminology

"Super" is an umbrella term that describes
a number of different elements of the system:
* The [super data model](formats/zed.md) is the abstract definition of the data types and semantics
* The [super data model](formats/zed) is the abstract definition of the data types and semantics
that underlie the super-structured data formats.
* The [super data formats](formats/_index.md) are a family of
[human-readable (Super JSON, JSUP)](formats/jsup.md),
[sequential (Super Binary, BSUP)](formats/bsup.md), and
[columnar (Super Columnar, CSUP)](formats/csup.md) formats that all adhere to the
* The [super data formats](formats) are a family of
[human-readable (Super JSON, JSUP)](formats/jsup),
[sequential (Super Binary, BSUP)](formats/bsup), and
[columnar (Super Columnar, CSUP)](formats/csup) formats that all adhere to the
same abstract super data model.
* The [SuperPipe language](language/_index.md) is the system's pipeline language for performing
* The [SuperPipe language](language) is the system's pipeline language for performing
queries, searches, analytics, transformations, or any of the above combined together.
* A [SuperPipe query](language/overview.md) is a script that performs
* A [SuperPipe query](language/overview) is a script that performs
search and/or analytics.
* A [SuperPipe shaper](language/shaping.md) is a script that performs
* A [SuperPipe shaper](language/shaping) is a script that performs
data transformation to _shape_
the input data into the desired set of organizing super-structured data types called "shapes",
which are traditionally called _schemas_ in relational systems but are
much more flexible in SuperDB.
* A [SuperDB data lake](commands/super-db.md) is a collection of super-structured data stored
across one or more [data pools](commands/super-db.md#data-pools) with ACID commit semantics and
* A [SuperDB data lake](commands/super-db) is a collection of super-structured data stored
across one or more [data pools](commands/super-db#data-pools) with ACID commit semantics and
accessed via a [Git](https://git-scm.com/)-like API.

## Digging Deeper

The [SuperPipe language documentation](language/_index.md)
The [SuperPipe language documentation](language)
is the best way to learn about `super` in depth.
All of its examples use `super` commands run on the command line.
Run `super -h` for a list of command options and online help.

The [`super db` documentation](commands/super-db.md)
The [`super db` documentation](commands/super-db)
is the best way to learn about the SuperDB data lake.
All of its examples use `super db` commands run on the command line.
Run `super db -h` or `-h` with any subcommand for a list of command options
Expand Down Expand Up @@ -92,7 +92,7 @@ or other third-party services to interpret the lake data.
Once copied, a new service can be instantiated by pointing a `super db serve`
at the copy of the lake.

Functionality like [data compaction](commands/super-db.md#manage) and retention are all API-driven.
Functionality like [data compaction](commands/super-db#manage) and retention are all API-driven.

Bite-sized components are unified by the super-structured data, usually in the SUPZ format:
* All lake meta-data is available via meta-queries.
Expand Down
10 changes: 5 additions & 5 deletions docs/commands/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,15 @@ title: Commands
weight: 2
---

The [`super` command](super.md) is used to execute command-line queries on
inputs from files, HTTP URLs, or [S3](../integrations/amazon-s3.md).
The [`super` command](super) is used to execute command-line queries on
inputs from files, HTTP URLs, or [S3](../integrations/amazon-s3).

The [`super db` sub-commands](super-db.md) are for creating, configuring, ingesting
The [`super db` sub-commands](super-db) are for creating, configuring, ingesting
into, querying, and orchestrating SuperDB data lakes. These sub-commands are
organized into further subcommands like the familiar command patterns
of `docker` or `kubectl`.

All operations with these commands utilize the [super data model](../formats/_index.md)
and provide querying via [SuperSQL](../language/_index.md).
All operations with these commands utilize the [super data model](../formats)
and provide querying via [SuperSQL](../language).

Built-in help for `super` and all sub-commands is always accessible with the `-h` flag.
58 changes: 29 additions & 29 deletions docs/commands/super-db.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ title: super db

> **TL;DR** `super db` is a sub-command of `super` to manage and query SuperDB data lakes.
> You can import data from a variety of formats and it will automatically
> be committed in [super-structured](../formats/_index.md)
> be committed in [super-structured](../formats)
> format, providing full fidelity of the original format and the ability
> to reconstruct the original data without loss of information.
>
Expand All @@ -16,13 +16,13 @@ title: super db
<p id="status"></p>

:::tip Status
While [`super`](super.md) and its accompanying [formats](../formats/_index.md)
While [`super`](super) and its accompanying [formats](../formats)
are production quality, the SuperDB data lake is still fairly early in development
and alpha quality.
That said, SuperDB data lakes can be utilized quite effectively at small scale,
or at larger scales when scripted automation
is deployed to manage the lake's data layout via the
[lake API](../lake/api.md).
[lake API](../lake/api).

Enhanced scalability with self-tuning configuration is under development.
:::
Expand All @@ -32,7 +32,7 @@ Enhanced scalability with self-tuning configuration is under development.
A SuperDB data lake is a cloud-native arrangement of data, optimized for search,
analytics, ETL, data discovery, and data preparation
at scale based on data represented in accordance
with the [super data model](../formats/zed.md).
with the [super data model](../formats/zed).

A lake is organized into a collection of data pools forming a single
administrative domain. The current implementation supports
Expand Down Expand Up @@ -70,7 +70,7 @@ bite-sized chunks for learning how the system works and how the different
components come together.

While the CLI-first approach provides these benefits,
all of the functionality is also exposed through [an API](../lake/api.md) to
all of the functionality is also exposed through [an API](../lake/api) to
a lake service. Many use cases involve an application like
[SuperDB Desktop](https://zui.brimdata.io/) or a
programming environment like Python/Pandas interacting
Expand Down Expand Up @@ -115,8 +115,8 @@ replication easy to support and deploy.

The cloud objects that comprise a lake, e.g., data objects,
commit history, transaction journals, partial aggregations, etc.,
are stored as super-structured data, i.e., either as [row-based Super Binary](../formats/bsup.md)
or [Super Columnar](../formats/csup.md).
are stored as super-structured data, i.e., either as [row-based Super Binary](../formats/bsup)
or [Super Columnar](../formats/csup).
This makes introspection of the lake structure straightforward as many key
lake data structures can be queried with metadata queries and presented
to a client for further processing by downstream tooling.
Expand Down Expand Up @@ -261,11 +261,11 @@ which is the sort key for all data stored in the lake. Different data pools
can have different pool keys but all of the data in a pool must have the same
pool key.

As pool data is often comprised of [records](../formats/zed.md#21-record) (analogous to JSON objects),
As pool data is often comprised of [records](../formats/zed#21-record) (analogous to JSON objects),
the pool key is typically a field of the stored records.
When pool data is not structured as records/objects (e.g., scalar or arrays or other
non-record types), then the pool key would typically be configured
as the [special value `this`](../language/pipeline-model.md#the-special-value-this).
as the [special value `this`](../language/pipeline-model#the-special-value-this).

Data can be efficiently scanned if a query has a filter operating on the pool
key. For example, on a pool with pool key `ts`, the query `ts == 100`
Expand Down Expand Up @@ -294,7 +294,7 @@ optimize scans over such data is impaired.

Because commits are transactional and immutable, a query
sees its entire data scan as a fixed "snapshot" with respect to the
commit history. In fact, the [`from` operator](../language/operators/from.md)
commit history. In fact, the [`from` operator](../language/operators/from)
allows a commit object to be specified with the `@` suffix to a
pool reference, e.g.,
```
Expand Down Expand Up @@ -331,7 +331,7 @@ Time travel using timestamps is a forthcoming feature.

## `super db` Commands

While `super db` is itself a sub-command of [`super`](super.md), it invokes
While `super db` is itself a sub-command of [`super`](super), it invokes
a large number of interrelated sub-commands, similar to the
[`docker`](https://docs.docker.com/engine/reference/commandline/cli/)
or [`kubectl`](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)
Expand All @@ -347,16 +347,16 @@ for that sub-command.
sub-command and so forth.

By default, commands that display lake metadata (e.g., [`log`](#log) or
[`ls`](#ls)) use the human-readable [lake metadata output](super.md#superdb-data-lake-metadata-output)
[`ls`](#ls)) use the human-readable [lake metadata output](super#superdb-data-lake-metadata-output)
format. However, the `-f` option can be used to specify any supported
[output format](super.md#output-formats).
[output format](super#output-formats).

### Auth
```
super db auth login|logout|method|verify
```
Access to a lake can be secured with [Auth0 authentication](https://auth0.com/).
A [guide](../integrations/zed-lake-auth.md) is available with example configurations.
A [guide](../integrations/zed-lake-auth) is available with example configurations.
Please reach out to us on our [community Slack](https://www.brimdata.io/join-slack/)
if you have feedback on your experience or need additional help.

Expand Down Expand Up @@ -403,7 +403,7 @@ The `-orderby` option indicates the [pool key](#pool-key) that is used to sort
the data in lake, which may be in ascending or descending order.

If a pool key is not specified, then it defaults to
the [special value `this`](../language/pipeline-model.md#the-special-value-this).
the [special value `this`](../language/pipeline-model#the-special-value-this).

A newly created pool is initialized with a branch called `main`.

Expand Down Expand Up @@ -468,12 +468,12 @@ a "table" as all super-structured data is _self describing_ and can be queried i
schema-agnostic fashion. Data of any _shape_ can be stored in any pool
and arbitrary data _shapes_ can coexist side by side.

As with [`super`](super.md),
the [input arguments](super.md#usage) can be in
any [supported format](super.md#input-formats) and
As with [`super`](super),
the [input arguments](super#usage) can be in
any [supported format](super#input-formats) and
the input format is auto-detected if `-i` is not provided. Likewise,
the inputs may be URLs, in which case, the `load` command streams
the data from a Web server or [S3](../integrations/amazon-s3.md) and into the lake.
the data from a Web server or [S3](../integrations/amazon-s3) and into the lake.

When data is loaded, it is broken up into objects of a target size determined
by the pool's `threshold` parameter (which defaults to 500MiB but can be configured
Expand Down Expand Up @@ -538,7 +538,7 @@ The `date` field here is used by the lake system to do [time travel](#time-trave
through the branch and pool history, allowing you to see the state of
branches at any time in their commit history.

Arbitrary metadata expressed as any [Super JSON value](../formats/jsup.md)
Arbitrary metadata expressed as any [Super JSON value](../formats/jsup)
may be attached to a commit via the `-meta` flag. This allows an application
or user to transactionally commit metadata alongside committed data for any
purpose. This approach allows external applications to implement arbitrary
Expand All @@ -547,7 +547,7 @@ commit history.

Since commit objects are stored as super-structured data, the metadata can easily be
queried by running the `log -f bsup` to retrieve the log in Super Binary format,
for example, and using [`super`](super.md) to pull the metadata out
for example, and using [`super`](super) to pull the metadata out
as in:
```
super db log -f bsup | super -c 'has(meta) | yield {id,meta}' -
Expand Down Expand Up @@ -580,7 +580,7 @@ A commit object includes
an optional author and message, along with a required timestamp,
that is stored in the commit journal for reference. These values may
be specified as options to the [`load`](#load) command, and are also available in the
[lake API](../lake/api.md) for automation.
[lake API](../lake/api) for automation.

:::tip note
The branchlog meta-query source is not yet implemented.
Expand Down Expand Up @@ -612,7 +612,7 @@ If the `-monitor` option is specified and the lake is [located](#locating-the-la
via network connection, `super db manage` will run continuously and perform updates
as needed. By default a check is performed once per minute to determine if
updates are necessary. The `-interval` option may be used to specify an
alternate check frequency in [duration format](../formats/jsup.md#23-primitive-values).
alternate check frequency in [duration format](../formats/jsup#23-primitive-values).

If `-monitor` is not specified, a single maintenance pass is performed on the
lake.
Expand Down Expand Up @@ -657,13 +657,13 @@ according to configured policies and logic.
```
super db query [options] <query>
```
The `query` command runs a [SuperSQL](../language/_index.md) query with data from a lake as input.
A query typically begins with a [`from` operator](../language/operators/from.md)
The `query` command runs a [SuperSQL](../language) query with data from a lake as input.
A query typically begins with a [`from` operator](../language/operators/from)
indicating the pool and branch to use as input.

The pool/branch names are specified with `from` in the query.

As with [`super`](super.md), the default output format is Super JSON for
As with [`super`](super), the default output format is Super JSON for
terminals and Super Binary otherwise, though this can be overridden with
`-f` to specify one of the various supported output formats.

Expand All @@ -685,7 +685,7 @@ Filters on pool keys are efficiently implemented as the data is laid out
according to the pool key and seek indexes keyed by the pool key
are computed for each data object.

When querying data to the [Super Binary](../formats/bsup.md) output format,
When querying data to the [Super Binary](../formats/bsup) output format,
output from a pool can be easily piped to other commands like `super`, e.g.,
```
super db query -f bsup 'from logs' | super -f table -c 'count() by field' -
Expand All @@ -699,7 +699,7 @@ By default, the `query` command scans pool data in pool-key order though
the query optimizer may, in general, reorder the scan to optimize searches,
aggregations, and joins.
An order hint can be supplied to the `query` command to indicate to
the optimizer the desired processing order, but in general, [`sort` operators](../language/operators/sort.md)
the optimizer the desired processing order, but in general, [`sort` operators](../language/operators/sort)
should be used to guarantee any particular sort order.

Arbitrarily complex queries can be executed over the lake in this fashion
Expand Down Expand Up @@ -773,7 +773,7 @@ super db serve [options]
```
The `serve` command implements the [server personality](#command-personalities) to service requests
from instances of the client personality.
It listens for [lake API](../lake/api.md) requests on the interface and port
It listens for [lake API](../lake/api) requests on the interface and port
specified by the `-l` option, executes the requests, and returns results.

The `-log.level` option controls log verbosity. Available levels, ordered
Expand Down
Loading
Loading