Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: redefine union request types when c8 REST docs are generated #4363

Merged
merged 7 commits into from
Oct 3, 2024

Conversation

pepopowitz
Copy link
Collaborator

@pepopowitz pepopowitz commented Sep 27, 2024

Description

Follow-up to #4308 (comment).

Modifies the C8 REST api doc generation strategy to:

  • Redefine two union-type requests, so that the schema and request explorers describe valid requests.
    • Previously, these requests were defined as accepting both an ID and a key in the request, which is an invalid request. We were relying on descriptions to dissuade users from submitting both, but that was easy to overlook.
    • Now, the requests are defined more accurately for valid request bodies.
    • These changes are not able to be made upstream, see Update camunda REST API doc #4308 (comment).
    • If one of the requests appears to have been rewritten already in our spec, it will not attempt to rewrite a second time.
  • Remove a description hack that is no longer needed.

Risks

This does introduce drift between the upstream source and the spec in these docs. I personally believe this drift is warranted, because of the improvement to the UX of the generated docs.

The biggest risk with these changes is that the request for these endpoints changes, and the pre-generation steps either fail, or rewrite the spec in a way that doesn't make sense. It'll be me that has to fix things if this happens, and I accept that responsibility.

Other implementations considered

You'll see that the rewriting happens through regular expressions, effectively doing a string.replace() to the source spec. This results in some tricky code, which I've hopefully commented enough to explain clearly.

I did consider using the js-yaml package to update the spec instead of string replacement via regular expression, however it resulted in a lot of churn in the spec due to formatting differences, and worse, it rewrote the description properties of many endpoints to line-break in undesired locations. I could live with the churn because once it's committed, it won't re-churn. But the description properties are often intentionally line-broken, at the end of a sentence, because the generator uses the first line for meta description tags, and we don't want those to end mid-sentence.

I would reconsider if people think this code is too hard to read or maintain.

Some screenshots of the updated requests

Create process instance, by key:

image

Create process instance, by ID:

image

An improvement I wasn't expecting -- previously the the default "example" for these requests was invalid because it contained both key and ID; now it contains only the key, and is valid:

image

Evaluate decision, by key:

image

Evaluate decition, by ID:

image

When should this change go live?

  • This is a bug fix, security concern, or something that needs urgent release support.
  • This is already available but undocumented and should be released within a week.
  • This on a specific schedule and the assignee will coordinate a release with the DevEx team. (apply hold label or convert to draft PR)
  • This is part of a scheduled alpha or minor. (apply alpha or minor label)
  • There is no urgency with this change and can be released at any time.

PR Checklist

  • My changes are for an already released minor and are in /versioned_docs directory.
  • My changes are for the next minor and are in /docs directory (aka /next/).

@pepopowitz pepopowitz added the deploy Stand up a temporary docs site with this PR label Sep 27, 2024
@pepopowitz pepopowitz self-assigned this Sep 27, 2024
@pepopowitz pepopowitz requested a review from tmetzke September 27, 2024 22:02
Copy link
Contributor

👋 🤖 🤔 Hello! Did you make your changes in all the right places?

These files were changed only in docs/. You might want to duplicate these changes in versioned_docs/version-8.5/.

  • docs/apis-tools/camunda-api-rest/specifications/activate-jobs.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/assign-user-task.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/broadcast-signal.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/cancel-process-instance.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/complete-job.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/complete-user-task.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/correlate-a-message.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/create-document-link-alpha.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/create-process-instance.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/delete-document-alpha.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/delete-resource.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/deploy-resources.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/download-document-alpha.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/evaluate-decision.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/fail-job.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/find-all-users.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/get-cluster-topology.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/get-decision-definition-xml-alpha.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/get-incident-by-key-alpha.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/get-status-of-camunda-license.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/migrate-process-instance.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/modify-process-instance.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/pin-internal-clock-alpha.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/publish-a-message.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/query-decision-definitions-alpha.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/query-decision-instances-alpha.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/query-decision-requirements-alpha.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/query-flow-node-instances-alpha.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/query-incidents-alpha.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/query-process-instances-alpha.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/query-user-tasks-alpha.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/report-error-for-job.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/reset-internal-clock-alpha.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/resolve-incident.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/unassign-user-task.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/update-a-job.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/update-element-instance-variables.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/update-user-task.api.mdx
  • docs/apis-tools/camunda-api-rest/specifications/upload-document-alpha.api.mdx

You may have done this intentionally, but we wanted to point it out in case you didn't. You can read more about the versioning within our docs in our documentation guidelines.

@pepopowitz pepopowitz added theme:api-streamline Issues related to the theme of streamlining APIs epic:api-interactivity labels Sep 27, 2024
@github-actions github-actions bot temporarily deployed to camunda-docs September 27, 2024 22:20 Destroyed
@tmetzke
Copy link
Member

tmetzke commented Sep 30, 2024

Thanks for providing this, @pepopowitz 👍

While I agree that the documentation becomes a bit clearer, I struggle with the potential longer-term maintenance costs of this change. The code to introduce this necessarily has to do some text parsing to determine if the change is already present. This in itself usually becomes error-prone quite quickly 😅

Plus, the original spec and the one used and displayed in our reference docs differ quite a bit now. I could imagine that users might be confused if our reference docs look a lot different than a document generated from our public OpenAPI spec.

I am wondering if a clear documentation of the endpoint with good descriptions is not good enough in our case and simply creates less maintenance burden to think and take care of.

@pepopowitz
Copy link
Collaborator Author

pepopowitz commented Sep 30, 2024

This in itself usually becomes error-prone quite quickly

I think it depends on how much we anticipate these two endpoints changing. If they are relatively stable APIs, I don't think it would be very risky. If the request shape is still evolving, then I agree that it is risky and fragile. The schema modifications are built in a way that they assume the mutually-exclusive keys are listed first in the schema definition, and that anything below them is shared across the mutually-exclusive request shapes. If the request changes to add more non-key fields, after the key fields, then the modifications should (theoretically) continue to work properly. If the keys themselves change, then yes, the modifications might cause incorrect results.

Do you have any thoughts @tmetzke on how stable these specific request schemas are?

I'm also open to other suggestions on how to provide this more-accurate schema definition in our docs, because I do think it's important. We've already had two people internally mention that the current definition confused them; I'd expect the confusion to be more wide-spread when the public has access to it.

@jwulf
Copy link
Member

jwulf commented Sep 30, 2024

Can we get to the source of this issue?

From this comment:

One aspect of the oneOf mechanism that OpenAPI provides is that it poses an issue with the Java models we generate from it. There are multiple issues with this polymorphism mechanism in the code generators and not only for Java, if I remember correctly. The generators struggle to create code that works in an API setup where data needs to be (de-)serialized, leading to custom post-processing effort required by generator users (us, in this case) to make it work.

It sounds as though the core of the issue at the moment is:

  1. The programmatic spec is not precise and accurate because the code generation from the spec to the implementation has an issue.
  2. Changing the spec to make it precise is possible, but has impacts on the code generation pipelines, and we don't have time budget for this additional complexity before the release.

This is just me thinking out loud: if I understand the issue, then it seems that the best way to resolve this is to address the complexity at the code generation point, rather than moving it up the stack to the documentation layer, with a complex post-processing patch, or the documentation / support layer by addressing user queries proactively (documentation) and reactively (support).

We may not be able to do that before the 8.6.0 release, but could we do that work immediately after?

@pepopowitz
Copy link
Collaborator Author

I've heard people's feedback, and I appreciate it, but I would like to move forward with this PR prior to the upcoming release.

Re: differences in the spec

Plus, the original spec and the one used and displayed in our reference docs differ quite a bit now. I could imagine that users might be confused if our reference docs look a lot different than a document generated from our public OpenAPI spec.

  1. I've added a step to the generation strategy for this API to add a disclaimer to the beginning of the spec. I don't think many people will look at this spec, but if they do, this will cue them to the fact that we've modified this spec a bit for the sake of the docs.
  2. The differences introduced in this change do not create a scenario where a user could generate from the docs a request that doesn't function with the API itself. In fact, it's the opposite -- the docs will now generate only examples that function with the API. If the changes introduced created a scenario where users could create requests that didn't function, I would be more inclined to agree that this drift is bad. But as it is, this is patching an issue in the original spec, and I think it's valuable and important.

Re: introducing risk/fragility

There is certainly risk and fragility introduced by the parsing/editing in this PR --- but the person who will be affected by this code breaking is me. I am willing to accept that risk, because I think this eliminates more friction than it introduces.


Having said all that, I can't merge this without an approval from someone. Well...technically I can, but I would prefer not to.

@github-actions github-actions bot temporarily deployed to camunda-docs October 2, 2024 22:19 Destroyed
Copy link
Member

@jwulf jwulf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good polyfill for user experience until we can resolve it at the source.

@pepopowitz pepopowitz merged commit e7bbf27 into main Oct 3, 2024
7 checks passed
@pepopowitz pepopowitz deleted the pepopowitz/oneof-requests-schema branch October 3, 2024 16:57
Copy link
Contributor

github-actions bot commented Oct 3, 2024

🧹 Preview environment for this PR has been torn down.

christinaausley pushed a commit that referenced this pull request Oct 7, 2024
…4363)

* chore: remove unused hack

* fix: add missing import

* feat: redefine createProcessInstanceRequest to express union type

* chore: regenerate docs

* feat: redefine evaluateDecisionRequest to express union type

* chore: regenerate docs

* feat: add a disclaimer to the top of the c8 spec so that people know it doesn't match the source exactly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
deploy Stand up a temporary docs site with this PR epic:api-interactivity theme:api-streamline Issues related to the theme of streamlining APIs
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants