Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: AIP-159 – Reading across collections #36

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 100 additions & 0 deletions aip/general/0159/aip.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Reading across collections

Sometimes, it is useful for a user to be able to retrieve resources across

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Sometimes, it is useful for a user to be able to retrieve resources across
Sometimes it is useful for a user to be able to retrieve resources across

multiple collections, or retrieve a single resource without needing to know
what collection it is in.

## Guidance

There are two potential approaches for reading resources across multiple
collections. Groups of related services **should**, as a unit, select a
preferred approach and apply it consistently.

### Promoting Resources

One approach is to evade the problem entirely by promoting resources above
their erstwhile parents in the event that reading across collections is likely
to be a requirement.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • @hudlow Under this approach, a service may represent the book in multiple places; e.g. top-level collection and a subcollection under publishers.
    • This is usually done when it is many-to-many though and this is one-to-many, so we would not do it here.
    • @lukesneeringer Is this a fundamentally different pattern if it is many-to-many?
    • @hudlow The other pattern is we would say that there must be a canonical path (you can not have it return a different href in the body no matter what).
    • @lukesneeringer Probably we should say that you may have multiple paths but must have a single canonical path, and move the many-to-many case @hudlow mentions to AIP-144.
    • @mkistler The spirit of the AIP is "how do I get to all members of a subcollection regardless of parent"; I feel uncomfortable saying "just promote it". I believe they should not be removed as subcollections.
      • @lukesneeringer Does this violate the "only one way to do it" principle?
      • @mkistler Not for reading across collection.
      • @lukesneeringer I just realized an answer to my own objection: What if you can only GET the virtual collection, but not the individual resources?
        • @hudlow 404 seems a bit rude, maybe 308?
        • @lukesneeringer If you 404, you can allow IDs that are unique to the scope of the collection rather than globally unique. Maybe say that you may do this and 404, or may have gloablly-unique IDs and 308?
        • @hudlow This still does the thing @Alfus hates though.
  • @lukesneeringer Note: The book IDs must be unique across the collections if this approach is used.
  • @hudlow It is not obvious to me how y'all (Google) decide how to make things into subcollections. Surely sometimes you come across a situation where something has a strong parent-child relationship with multiple parents.
    • @lukesneeringer We do, but every resource remains a child of a single parent. A book that is a child of an author is not a child of a publisher.

Consensus:

  • The top-level collection is called a "virtual collection".
    • For individual resources, you may 308 the non-canonical single-resource GET if IDs are globally unique, or 404 the non-canonical single-resource GET if they are unique only in the scope of the collection.
    • The resource must provide the parent ID in the body of the resource somewhere.

In this approach, the parent becomes a field on the resource:

```typescript
interface Book {
// The ID of the book's publisher
publisherId: string;

// Other fields...
}
```

Similarly, the parent is removed from the URI:

```
GET /v1/books?filter...
```

### Wildcards

The other approach is to use a wildcard character. Services **may** support
reading resources across multiple collections by allowing users to specify a
`-` (the hyphen or dash character) as a wildcard character in a standard
[`List`][aip-132] operation:

```
GET /v1/publishers/-/books?filter=...
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • @mkistler Maybe suggest both the Google and IBM approach, with guidance to pick one.
    • Maybe suggest the rationale behind both.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I forgot what the IBM approach was. :-(

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The IBM approach is creating a new top-level collection.

```

- The URI pattern **must** still accept a parameter to permit the collection to
be specified; a URI pattern **must not** hard-code the `-` character.
- The operation **must** explicitly document that this behavior is supported.
- Responses **must** populate the canonical parent ID rather than `-` if the
lukesneeringer marked this conversation as resolved.
Show resolved Hide resolved
parent ID is normally provided in the resource in some way.
- If resources include the parent ID within the name of the resource, the
resources provided in the response **must** use the canonical name of the
resource, with the actual parent collection identifiers (instead of `-`).
- If resources include a field indicating the ID of the parent, resources
provided in the response **must** populate the canonical parent ID (instead
of `-`).
- Services **may** support reading across collections on `List` requests
regardless of whether the identifiers of the child resources are guaranteed
to be unique.
- However, services **must not** support reading across collections on `Get`
requests if the child resources might have a collision.
- Cross-parent requests **may** support `order_by`. However, if reading across
collections requires consulting multiple backends, databases, or locations,
cross-parent requests **should not** support `order_by`, and if they do, the
field **must** document that it is best effort. This is because cross-parent
requests introduce ambiguity around ordering, especially if there is
difficulty reaching a parent (see AIP-217).

**Important:** If listing across multiple collections introduces the
possibility of partial failures due to unreachable parents (such as when
listing across locations), the operation **must** indicate this following the
guidance in AIP-217.

### Unique resource lookup

Sometimes, a resource within a sub-collection has an identifier that is unique
across parent collections. In this case, it may be useful to allow a
[`Get`][aip-131] operation to retrieve that resource without knowing which
parent collection contains it. In such cases, APIs **may** allow users to
specify the wildcard collection ID `-` (the hyphen or dash character) to
represent any parent collection:

```
GET https://example.googleapis.com/v1/publishers/-/books/{book}
```

- The URI pattern **must** still be specified with `*` and permit the
collection to be specified; a URI pattern **must not** hard-code the `-`
character.
- The operation **must** explicitly document that this behavior is supported.
Comment on lines +64 to +67

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these lines can be deleted as they seem to simply restate lines 23-25

Suggested change
- The URI pattern **must** still be specified with `*` and permit the
collection to be specified; a URI pattern **must not** hard-code the `-`
character.
- The operation **must** explicitly document that this behavior is supported.

- The resource name in the response **must** use the canonical name of the
resource, with actual parent collection identifiers (instead of `-`). For
example, the request above returns a resource with a name like
`publishers/123/books/456`, _not_ `publishers/-/books/456`.
- The resource ID **must** be unique within parent collections.

## Further reading

- For partial failures due to unreachable resources, see AIP-217.
7 changes: 7 additions & 0 deletions aip/general/0159/aip.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
id: 159
state: approved
created: 2019-07-26
placement:
category: design-patterns
order: 70