ci: add a script and CI job to validate spec examples #1046

AnimeshKumar923 · 2024-03-19T09:04:50Z

This PR adds a script and CI to validate the AsyncAPI documents in the /examples directory.

The embedded examples in spec are to be validated through it.
Additionally, if any incoming PR is changing the spec examples, it should check that also,

Tasks

Write a script to validate the existing AsyncAPI documents examples under the /examples directory.
Develop a YAML file for the CI. (WIP)
Write a script to validate the AsyncAPI documents containing embedded examples. --> ~~Will be done through a separate PR (context) (phase-2)~~ ~~Being done through~~ --> achieved through ci: script to validate embedded examples in the asyncapi.md file #1059

cc: @smoya PTAL

Related issue(s):
#957 (partially resolves)

Changes: - added initial version of the workflow file - will be worked and moved forward with this file with the required modifications

Changes: - updated the checkout action version - change the main branch to master

Changes: - added the both YAML file extension to validate all files present in the examples directory and further sub-directories

Changes: - updated code to validate the example folder and sub-folders - add dev dependecies for the script

Changes: - removed the 'test' script - improved the 'validate-examples' workflow file code

Changes: - migrated package.json to root of the directory - changes package.json accordingly

Changes: - update script accordingly to the newly migrated package.json file - update workflow yaml with workflow_dispatch events

smoya · 2024-03-22T11:48:10Z

scripts/validation/validate-examples.js

+const glob = require('glob');
+
+// Use glob to find all YAML files in the examples directory
+const files = glob.sync('./examples/**/*.{yml,yaml}');


I believe there are some files you won't be able to validate since they are not AsyncAPI documents such as the files located in https://github.com/asyncapi/spec/tree/master/examples/social-media/common. Not sure if there are more, you will need to check.

Yes, it is like that.
You can see the output:

So, what can be done about it? Exclude these? Or something else?

Exclude them, yes.

Exclude them, yes.

Got it!

keeping this conversation unresolved for the time being, to serve as a reminder

scripts/validation/validate-examples.js

smoya · 2024-03-22T12:04:34Z

scripts/validation/validate-examples.js

+  filesCount++;
+  try {
+    console.log(`\nValidating: ${file}`);
+    execSync(`npx asyncapi validate ${file}`, { stdio: 'inherit' });


This is gonna be pretty slow. Can we run it async somehow and compare performance? 🤔

This is gonna be pretty slow. Can we run it async somehow and compare performance? 🤔

Okay, let me try doing it asynchronously...

Applied suggestion through baa7c12

Performance Report

Before

When the script was executed synchronously:

Job link

Screenshot

After

When the script was executed asynchronously:

Job link

Screenshot

Conclusion: The execution time here has been reduced by 1m 10s with the use of asynchronous way of JavsScript.

Thanks for the suggestion! @smoya 👍

Execution time for the script itself

Before

After

I think we can go further and skip using a JS script here and find a simple online terminal command that do the job.

I believe by using find which also allows excluding paths + piping to asyncapi command (CLI) through xargs would end up being a really simple oneliner, not complicated and simple to read and maintain.

I think we can go further and skip using a JS script here and find a simple online terminal command that do the job.

I believe by using find which also allows excluding paths + piping to asyncapi command (CLI) through xargs would end up being a really simple oneliner, not complicated and simple to read and maintain.

Let me look into that...

@smoya The code has been successfully executed. Please have a look at this workflow job detail

Attaching screenshot for more details:

Job run details

Link for reference

Changes: - applied suggestion from: https://github.com/asyncapi/spec/pull/1046/files?diff=unified&w=0#r1535456030

Changes: - removed the mistakenly placed code block from the line

Changes: - made the script asynchronous in nature as suggested by Sergio to make it's performance more better - applied suggestion from here: https://github.com/asyncapi/spec/pull/1046/files#r1535471235

smoya · 2024-04-08T11:45:49Z

.github/workflows/validate-examples.yml

+        run: npm install -g @asyncapi/cli
+      - name: Validate AsyncAPI documents
+        run: |
+          find examples/ \( -path 'examples/social-media/*' -prune \) -o -type f \( -name "*.yml" -o -name "*.yaml" \) -exec asyncapi validate {} \;


Nice oneliner 👏 👏

There is only a couple of issues here:

The path to skip should be 'examples/social-media/common.

I don't find prune option to be very readable. Instead, you can use the -not operator. E.g. -not -path 'examples/social-media/common'

With -exec option you can't parallelise the work, so everything will run sequentially.
Instead, try piping the find output to xargs and the -P arg.

Applying that, will look like:

find examples -type f \( -name "*.yml" -o -name "*.yaml" \) -not -path 'examples/social-media/common' | xargs -P 10 -L 1 asyncapi validate

Additionally, please add a comment on top of the command explaining why we are skipping that dir.

Nice oneliner 👏 👏

Thank you! 😁 To be honest, I'm still learning bash so somethings may take time to understand 😅 and many things are new like the usage of xargs. Will learn along the way, thanks to you... 😊

There is only a couple of issues here:

1. The path to skip should be `'examples/social-media/common`. 2. I don't find `prune` option to be very readable. Instead, you can use the `-not` operator. E.g. `-not -path 'examples/social-media/common'` 3. With `-exec` option you can't parallelise the work, so everything will run sequentially. Instead, try piping the find output to `xargs` and the `-P` arg.

Applying that, will look like:

find examples -type f \( -name "*.yml" -o -name "*.yaml" \) -not -path 'examples/social-media/common' | xargs -P 10 -L 1 asyncapi validate

Thank you for the suggestions, will try to understand and implement the same. 🙏

Additionally, please add a comment on top of the command explaining why we are skipping that dir.

Sure, I'll do that.

applied through cb7d070

Changes: - applied Sergio's suggestion from: https://github.com/asyncapi/spec/pull/1046/files#r1555684920 - added comment as to why we're skipping the mentioned directory

smoya

On my side, this has my 👍 (pending to switch the branch from script-ci-spec-validation to master in the workflow).
Additinoally, if we want to merge this PR before continuing the work for Write a script to validate the AsyncAPI documents containing embedded examples. which I believe we should, please @AnimeshKumar923 remove the sentence Fixes https://github.com/asyncapi/spec/issues/957 from the description so the issue remains open after merging.

Can you @derberg do a quick check on the workflow just in case I'm missing something critical? 🙏

AnimeshKumar923 · 2024-04-10T12:13:28Z

On my side, this has my 👍 (pending to switch the branch from script-ci-spec-validation to master in the workflow).

@smoya

Good to hear that! Also, want to ask that should we run this on the incoming PRs and remove the push to master? Because originally, this was proposed in the issue description by Jonas. 👇

We need to make sure all the examples are complete and valid according to our spec. This should of course be checked for each PR that change the examples.

Or both the ways can be achieved at the same time with the current state of the YAML file?
What would be the better approach?

Additinoally, if we want to merge this PR before continuing the work for Write a script to validate the AsyncAPI documents containing embedded examples. which I believe we should, please @AnimeshKumar923 remove the sentence Fixes https://github.com/asyncapi/spec/issues/957 from the description so the issue remains open after merging.

Yes, I'll remove the Fixes from the description. 👍

derberg

I changed PR prefix cause otherwise we would release new spec version 😁

also, one question, if we use AsyncAPI CLI on each PR - what if we do a PR that adds new features for new release, and updates examples 🤔

.github/workflows/validate-examples.yml

derberg · 2024-04-10T19:46:35Z

.github/workflows/validate-examples.yml

+        run: npm install -g @asyncapi/cli
+      - name: Validate AsyncAPI documents
+        run: |
+          find examples -type f \( -name "*.yml" -o -name "*.yaml" \) -not -path 'examples/social-media/common/*' | xargs -P 10 -L 1 asyncapi validate 


omg, bash on steroids

luckily we have chat gpt to translate it for me when needed 😁

omg, bash on steroids

luckily we have chat gpt to translate it for me when needed 😁

😆

Come on! are self explanatory 😆 Let me save your time

find examples -type f \( -name "*.yml" -o -name "*.yaml" \) -not -path 'examples/social-media/common/*'

This is completely human readable: find in examples dir "things" of type file (f) with name suffixes yml or yaml and do not include the path 'examples/social-media/common/*' . The output is a list of paths to those files.

xargs -P 10 -L 1 asyncapi validate

Here maybe the trickiest part: xargs meaning do "x" with the piped arguments (which is the list of files), the -L 1 means take each new line as one execution, -P 10 means parallellize in 10 runs at the same time, and for each of those arguments (paths), execute asyncapi validate command.

derberg · 2024-04-10T19:48:43Z

.github/workflows/validate-examples.yml

+        with:
+          node-version: '20'
+      - name: Install AsyncAPI CLI
+        run: npm install -g @asyncapi/cli


bo it uses latest CLI - advantage is that all latest features will be used but obvious disadvantage is randomly failing CI cause of some breaking change in validation, or some new validation enabled.

yeah, not so obvious in this case 😕

so should be use a specific version of the CLI? 👀

If I can give my opinion here, I think using latest is a good option. We are always testing on the last version, supporting last spec schemas, etc + if something is wrong with the CLI, we are gonna be one of the first who notice.

ping @derberg @AnimeshKumar923

Ping @derberg 👋

no strong opinion really, there are pros and cons for using latest and not using latest. We just need to be aware that if we decide to use latest - then on PRs we can have random fails from time to time and we need to support contributors

Agree. I suggest then we can try with this but opened to change if needed at any time.

aeworxet · 2024-04-15T07:51:27Z

@asyncapi/bounty_team

AnimeshKumar923 · 2024-04-18T16:31:25Z

UPDATE

The 1st phase of the issue is complete, just the merging of this PR remains and we're good with it.
For the 2nd phase of the issue, another PR will be raised addressing that.
I have my final end semester exams going on which will end on 25th May, due to which I'm not able to dedicate much time on the issue recently, and won't be able to focus properly on it during this time period. Hence, I request an extension of final PR submission date to be extended and changed to 7th June.
A decent amount of time will be required after my exams to get myself familiar with the issue again, so keeping that in mind I've proposed the specified date above.

Thank you! 🙏
ping @smoya @aeworxet

sonarcloud · 2024-04-29T10:30:20Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

smoya

LGTM!

smoya · 2024-04-29T14:04:50Z

/rtm

AnimeshKumar923 added 15 commits March 8, 2024 00:24

feat: add new workflow to validate example

178cf39

Changes: - added initial version of the workflow file - will be worked and moved forward with this file with the required modifications

chore: update checkout action version

ce9b567

Changes: - updated the checkout action version - change the main branch to master

fix: add YAML extension support

883b13c

Changes: - added the both YAML file extension to validate all files present in the examples directory and further sub-directories

chore: add current testing branch in workflow

7fe9564

chore: update workflow version

1e22e40

chore: update workflow version

f5a1a98

fix: update code to match workflow

b7da489

fix: update code to match workflow

fb898a4

chore: update code

65c8f15

fix: update code, script in working condition

dacaf85

Changes: - updated code to validate the example folder and sub-folders - add dev dependecies for the script

fix: update package.json, workflow file

d3427b8

Changes: - removed the 'test' script - improved the 'validate-examples' workflow file code

chore: migrate package.json to root

37ba41b

Changes: - migrated package.json to root of the directory - changes package.json accordingly

fix: update script and yaml file

9e7f840

Changes: - update script accordingly to the newly migrated package.json file - update workflow yaml with workflow_dispatch events

chore: add \n for better readability

8a34862

fix: update script, add error files information

ba7a2a0

AnimeshKumar923 requested review from fmvilas, derberg, dalelane, smoya, char0n, asyncapi-bot-eve and GreenRover as code owners March 19, 2024 09:04

smoya reviewed Mar 22, 2024

View reviewed changes

scripts/validation/validate-examples.js Outdated Show resolved Hide resolved

smoya reviewed Mar 22, 2024

View reviewed changes

AnimeshKumar923 added 4 commits March 24, 2024 19:10

Merge branch 'master' into script-ci-spec-validation

ccc2e93

fix: insert process.exit(1) at correct place

abbda0d

Changes: - applied suggestion from: https://github.com/asyncapi/spec/pull/1046/files?diff=unified&w=0#r1535456030

chore: remove process.exit(1)

d2e81b1

Changes: - removed the mistakenly placed code block from the line

fix: made the script asynchronous

baa7c12

Changes: - made the script asynchronous in nature as suggested by Sergio to make it's performance more better - applied suggestion from here: https://github.com/asyncapi/spec/pull/1046/files#r1535471235

AnimeshKumar923 requested a review from smoya March 24, 2024 15:17

AnimeshKumar923 added 3 commits April 5, 2024 23:53

Delete package-lock.json

1edbb60

chore: delete package.json

3a47a66

chore: delete scripts/validation/validate-examples.js

8a18479

smoya reviewed Apr 8, 2024

View reviewed changes

AnimeshKumar923 added 8 commits April 9, 2024 01:12

fix: update bash script, pipe xargs

cb7d070

Changes: - applied Sergio's suggestion from: https://github.com/asyncapi/spec/pull/1046/files#r1555684920 - added comment as to why we're skipping the mentioned directory

chore: fixing errors

aef6961

chore: update code, fixing errors

1e80279

chore: update code, fixing errors

0752a62

chore: update code, fixing errors

c3fbc0f

chore: bump action versions

a73faaa

chore: testing different arguments

6ff12cf

chore: not much difference in the modified argument, reverting back

41ab956

smoya reviewed Apr 10, 2024

View reviewed changes

derberg changed the title ~~feat: add a script and CI job to validate spec examples~~ ci: add a script and CI job to validate spec examples Apr 10, 2024

derberg reviewed Apr 10, 2024

View reviewed changes

AnimeshKumar923 added 2 commits April 11, 2024 11:56

Merge branch 'master' into script-ci-spec-validation

296f776

fix: remove on:push:master

b49f4a1

asyncapi-bot added the bounty AsyncAPI Bounty label Apr 15, 2024

aeworxet mentioned this pull request Apr 19, 2024

Add a script and CI job to validate spec examples #957

Closed

Merge branch 'master' into script-ci-spec-validation

adec980

smoya approved these changes Apr 29, 2024

View reviewed changes

asyncapi-bot added the ready-to-merge label Apr 29, 2024

asyncapi-bot merged commit eda8937 into asyncapi:master Apr 29, 2024
7 checks passed

AnimeshKumar923 mentioned this pull request Jun 14, 2024

Update README.md AnimeshKumar923/college-stuff#5

Merged

asyncapi-bot removed the ready-to-merge label Jun 14, 2024

ci: add a script and CI job to validate spec examples #1046

ci: add a script and CI job to validate spec examples #1046

Conversation

AnimeshKumar923 commented Mar 19, 2024 • edited Loading

Tasks

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnimeshKumar923 Apr 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Performance Report

Before

After

AnimeshKumar923 Mar 24, 2024 • edited Loading

Choose a reason for hiding this comment

Execution time for the script itself

Before

After

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Attaching screenshot for more details:

Job run details

smoya Apr 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnimeshKumar923 Apr 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smoya left a comment • edited Loading

Choose a reason for hiding this comment

AnimeshKumar923 commented Apr 10, 2024 • edited Loading

derberg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aeworxet commented Apr 15, 2024

AnimeshKumar923 commented Apr 18, 2024 • edited Loading

UPDATE

sonarcloud bot commented Apr 29, 2024

Quality Gate passed

smoya left a comment

Choose a reason for hiding this comment

smoya commented Apr 29, 2024

AnimeshKumar923 commented Mar 19, 2024 •

edited

Loading

AnimeshKumar923 Apr 3, 2024 •

edited

Loading

AnimeshKumar923 Mar 24, 2024 •

edited

Loading

smoya Apr 8, 2024 •

edited

Loading

AnimeshKumar923 Apr 8, 2024 •

edited

Loading

smoya left a comment •

edited

Loading

AnimeshKumar923 commented Apr 10, 2024 •

edited

Loading

AnimeshKumar923 commented Apr 18, 2024 •

edited

Loading