adding support for large yaml files as input #409
Conversation
| @@ -18,7 +18,7 @@ ARG QUARKUS_EXTENSIONS=org.kie:kogito-addons-quarkus-jobs-knative-eventing:9.100 | |||
|
|
|||
| # Additional java/mvn arguments to pass to the builder. | |||
| # This are is conventient to pass sonataflow and quarkus build time properties. | |||
| ARG MAVEN_ARGS_APPEND="-Dkogito.persistence.type=jdbc -Dquarkus.datasource.db-kind=postgresql -Dkogito.persistence.proto.marshaller=false" | |||
| ARG MAVEN_ARGS_APPEND="-DmaxYamlCodePoints=99999999 -Dkogito.persistence.type=jdbc -Dquarkus.datasource.db-kind=postgresql -Dkogito.persistence.proto.marshaller=false" | |||
There was a problem hiding this comment.
which size does this value represent? could there be a need for a higher value?
pls make sure also to update https://www.parodos.dev/blog/extracting-openapi-documents/ to include this option.
There was a problem hiding this comment.
the codePoints param refers to Unicode points, which are basically the number of characters in the input file (as we're not using special chars like emojis in the yaml input files).
The maximum code point number is the integer max, 2^31-1.
the value 99999999 can represent around 190MB size input file,
I can bump it to 99999999, which is closer to the max , or 2^31-1, of course.
There was a problem hiding this comment.
no need for that high value, but pls include a comment in the docker file for this property, and the current value it represents.
There was a problem hiding this comment.
Sure. So any preferred benchmark for input size? 100MB?
For reference, the full github-openapi spec is 10MB
There was a problem hiding this comment.
OCP openapi-v2-4.14.yaml is 23M, let's go with 30M.
This change adds a build time param for the serverless workflow dockerfile that allows passing YAML input files to a workflow with an unlimited size. This change comes in handy when we might use large openAPI spec files.
This change has been tested using a custom image built using this suggested change, and validated that the workflow does accept the arge input.
A reproducer for this fix can be seen here.
For full discretion, using large input files with this parameter may cause the workflow execution time to be slower.