Generate PDF report for cypress + environment management improvements

[roicarrera]

added mochawesome
added pdf-generator script file
read baseurl on config maps based on environment
added skip for prod

[cschweikert]

Hi @roicarrera, since this PR is also reworking parts of the respective configs, I would propose to use cypress.config.ts as base configuration that is then mixed into the other configs (import the base config and using the spread operator). This would prevent/remove a lot of code duplication and put more focus on the actual difference between those files.

[cschweikert]

Thanks for making all the changes. Really improved things around the configs! Only little things left.

[cschweikert]

🚀

[metmajer]

Hello @roicarrera,

• We need a clear concept for generating reports, screenshots, and videos from our testing Quickstarters. I cannot see clearly how their creation and storing does not affect Jenkins. The only place to store potentially larger files is Nexus.
• Disabling test execution in PROD in the way you did seriously messes with regulations. A solid design involving the core platform team needs to be done to prevent potential damage.

As a general rule, I also suggest creating one PR for one purpose, and not mixing many different things. This will help increase clarity during code reviews.

[cschweikert]

* We need a clear concept for generating reports, screenshots, and videos from our testing Quickstarters. I cannot see clearly how their creation and storing does not affect Jenkins. The only place to store potentially larger files is Nexus.

Build artifacts like screenshots and videos also fit the purpose of analyzing problems in case of broken tests. Yes, you should assure that e2e tests are successful BEFORE you commit. This could be done by running them on a locally running representation of your target system. However, the situation on the cluster is different. In rare cases you are even facing things like flaky tests that are hard to reproduce even on the cluster. In those situations as a developer you are thankful for every piece of information you could get.

In this context they fit where they are. However, we should discuss about the right amount of "daysToKeep". Also it would be great to understand, if storing screenshots and videos temporarily even creates a problem regarding storage sizes. Maybe Jenkins already takes care of this.

[cschweikert]

I continued thinking about and like to come back to my previous comment regarding the purpose of the videos:

Build artifacts like screenshots and videos also fit the purpose of analyzing problems ...

I still don't see the need for videos being part of any documentation, if there are things like test protocols and screenshots that can be linked to the responsible source code that defines the test scenarios. I'd recommend to keep videos as build artifacts with Jenkins and rather restrict the numToKeep to 1. This would require Jenkins to store one video ZIP in total. With this the risk of any storage issues is quite low.

More important is the following: Imagine every little e2e test run in the cluster leads to an upload of a big video file into Nexus. This will cause additional load in the network, puts pressure on the Nexus instance and consumes a good amount of storage capacity. I'm not aware of how long the concrete documents will be stored in Nexus. If there is no limited data retention, this will cause ever increasing resource allocation, which I don't see to be a sustainable option.

@roicarrera @metmajer What are your thoughts about this?

[metmajer]

@cschweikert

Build artifacts like screenshots and videos also fit the purpose of analyzing problems in case of broken tests.

There is nothing wrong with providing test execution results, screenshots, and videos. As a former software engineer, I can relate to what you say. Since Jenkins is a SPoF, we need to align before using Jenkins as a file store, even for short-term storage. Currently, Jenkin's file system is only 50 MB in size and would exhaust after only a few pipeline runs. As a result, Jenkins would stop working and would refuse to start until disk space has been freed up. Also, our support team would be busy handling support tickets. Additionally, I want to ensure that additional test reports align with the SLC documents we already produce to ensure that the benefit of either type of report is understood internally, and, by our users.

I still don't see the need for videos being part of any documentation

I agree. I believe that we should have a solid reasoning for videos. Whatever we put into our Quickstarters should appeal to 80% of our users, and not be an edge case. Capable users will be able to integrate support for videos should they need it.

FYI @roicarrera

[metmajer]

I recently shared the following feedback with @roicarrera. Sharing it here for transparency and record. FYI @cschweikert

Those are not only integration tests

ods-quickstarters/e2e-cypress/Jenkinsfile.template

Line 71 in 3f51c8a

stage('Integration Test') {

--
How do we make users understand what they have to do here? - Why not leave them commented in and simply educate that the config map must be populated / fail the pipeline if they are not set?

ods-quickstarters/e2e-cypress/Jenkinsfile.template

Lines 74 to 79 in 3f51c8a

	def baseUrls = [
	: // remove this line once you have defined the config map and uncommented the next two lines, it's only here to make the example default case work
	// dev: sh(returnStdout: true, script:"oc get configmaps cypress-config -o jsonpath='{.data.DEV_BASE_URL}'").trim(),
	// test: sh(returnStdout: true, script:"oc get configmaps cypress-config -o jsonpath='{.data.TEST_BASE_URL}'").trim()
	// prod: sh(returnStdout: true, script:"oc get configmaps cypress-config -o jsonpath='{.data.PROD_BASE_URL}'").trim()
	]

--
You need to use a credentials secret in OpenShift and use Jenkins' withCredentials to safely use the secret in transit.

ods-quickstarters/e2e-cypress/Jenkinsfile.template

Lines 86 to 87 in 3f51c8a

	// cypressUser = sh(returnStdout: true, script:"oc get secret e2euser -o jsonpath='{.data.${context.environment}_username}' | base64 -d")
	// cypressPassword = sh(returnStdout: true, script:"oc get secret e2euser -o jsonpath='{.data.${context.environment}_password}' | base64 -d")

--
These are a lot of commented lines; why not build better logic around them? Quickstarters should be quick to use, not require a deeper understanding of what's going on inside them.

ods-quickstarters/e2e-cypress/Jenkinsfile.template

Lines 83 to 88 in 3f51c8a

	// Example for loading environment variables for Azure SSO; please adapt variable names to your OpenShift config,
	// making sure to precede the variable names with the environment name in lowercase (e.g., dev_username, dev_password,
	// test_username, test_password, etc.)
	// cypressUser = sh(returnStdout: true, script:"oc get secret e2euser -o jsonpath='{.data.${context.environment}_username}' | base64 -d")
	// cypressPassword = sh(returnStdout: true, script:"oc get secret e2euser -o jsonpath='{.data.${context.environment}_password}' | base64 -d")
	// authenticatorOTPSecret = sh(returnStdout: true, script:"oc get secret azure -o jsonpath='{.data.OTP_SECRET}' | base64 -d")

and

ods-quickstarters/e2e-cypress/Jenkinsfile.template

Lines 99 to 101 in 3f51c8a

	// string(credentialsId: cypressUser, variable: 'CYPRESS_USERNAME'),
	// string(credentialsId: cypressPassword, variable: 'CYPRESS_PASSWORD'),
	// string(credentialsId: authenticatorOTPSecret, variable: 'OTP_SECRET')

--
Stashes are for communicating data to the Release Manager. We should only use them if needed. For now, there is no contract defined for exchanging screenshots between e2e-test components and the RM. Let's do this only once the contract has been defined.

ods-quickstarters/e2e-cypress/Jenkinsfile.template

Line 143 in 3f51c8a

stash(name: "acceptance-test-screenshots-${context.componentId}-${context.buildNumber}", includes: 'cypress/screenshots.zip', allowEmpty: true)

--
We need to avoid storing files in Jenkins. Quickstarters can too quickly fill up Jenkins and Jenkins is only an executor, not a file store. For this we have Nexus.

ods-quickstarters/e2e-cypress/Jenkinsfile.template

Line 144 in 3f51c8a

archiveArtifacts artifacts: 'cypress/screenshots.zip', fingerprint: true, daysToKeep: 2, numToKeep: 3

--
The return statement does not do anything. Please align with the behavior of Spock/Geb. A failed attempt to execute tests must fail the component. See https://github.com/opendevstack/ods-quickstarters/blob/master/e2e-spock-geb/Jenkinsfile.template#L32-L34 for reference.

ods-quickstarters/e2e-cypress/Jenkinsfile.template

Line 147 in 3f51c8a

return status

[cschweikert]

Currently, Jenkin's file system is only 50 MB in size and would exhaust after only a few pipeline runs.

Hm, I wonder if "50 MB" is a typo. When looking into the different *-cd OC projects I saw volume claims of 5 GB or 10 GB.

I did some quick experiments regarding filesize of those video files. A test file that does full page reloads every 2 seconds showing different websites in the resolution configured in this QS. Extrapolated to a 60 minutes test run the filesize would be about 0.5 GB. This is about the size of the node_modules folder of an average SPA frontend project.

Capable users will be able to integrate support for videos should they need it.

I like the idea. I would recommend to remove any record-specifc scripts (e.g. e2e:jenkins:record). This would also make the documentation a bit simpler. At the same time, I would suggest to keep the following code:

if (fileExists('cypress/videos')) {
  zip zipFile: 'cypress/videos.zip', archive: false, dir: 'cypress/videos'
  archiveArtifacts artifacts: 'cypress/videos.zip', fingerprint: true, daysToKeep: 1, numToKeep: 1
}

It will only trigger when the user enables the recording feature. Having it in place is helpful and would also reflect our "opinion" regarding a good setting for things like numToKeep.

[roicarrera]

I like the idea. I would recommend to remove any record-specifc scripts (e.g. e2e:jenkins:record). This would also make the documentation a bit simpler. At the same time, I would suggest to keep the following code:

Regarding this, the --record flag, and subsequently the e2e:jenkins:record command are for using Cypress Cloud, not for recording videos. The video feature is enabled in the cypress config file. I agree with leaving the video disabled by default tho, and having the users enable it if wanted, but the record command is a separate feature that does not affect Jenkins space

[metmajer]

@roicarrera @cschweikert I can see that the most critical feedback has been addressed. Points left for potential improvement are:

• commenting/uncommenting of lines; can you think of a better approach that transparently uses configmaps/secrets once defined by the user?
• daysToKeep and numToKeep; looking at https://github.com/jenkinsci/jenkins/blob/e2999799a9f983b302c63d7aaf7856f4da62e7d4/core/src/main/java/hudson/tasks/ArtifactArchiver.java#L391, I believe that the named properties are ignored completely? They are declared only in https://github.com/jenkinsci/jenkins/blob/e2999799a9f983b302c63d7aaf7856f4da62e7d4/core/src/main/java/hudson/tasks/LogRotator.java#L269, but class ArtifactArchiver neither inherits nor is composed of a LogRotator. Googling also didn't show any examples of archiveArtifact in combination with these properties. Thoughts?

[cschweikert]

LGTM 😊

Thanks for bringing in all those changes together!

@metmajer Is there anything open from your end?

[metmajer]

Well done @roicarrera and thanks @cschweikert for the support -- LGTM at last :)

Please squash-merge into a single commit, thanks