From 77f94430726baa5947318fbf39ae8ba1405d37e6 Mon Sep 17 00:00:00 2001 From: Dima May Date: Wed, 14 Aug 2024 14:25:39 +0000 Subject: [PATCH] #287: updated contribution guide and develoment guide for skills-service:3.0.0 release --- docs/contribution/architecture.md | 9 +- docs/contribution/devEnv.md | 385 ++++-------------------------- 2 files changed, 52 insertions(+), 342 deletions(-) diff --git a/docs/contribution/architecture.md b/docs/contribution/architecture.md index 20abf5d5..36c0b39b 100644 --- a/docs/contribution/architecture.md +++ b/docs/contribution/architecture.md @@ -17,10 +17,9 @@ SkillTree's skills-service is then configured to use PostgeSQL and Stomp Brokers ![Dashboard with Integrated Application](./diagrams/SkillsServiceArchitecture.jpg) Integrated clients utilize skills-client libs posted on the public NPM repositories. -These libraries are very thin wrappers around an iFrame tag, they simply retrieve the Skills Display app and its associated data from the skills-service. -What that means is that the Skills Display is served as its own web-based application from the skills-service and then inserted into an iFrame tag on the client's browser. +These libraries are very thin wrappers around an iFrame tag, they simply retrieve the Skills Display views and its associated data from the skills-service application. +What that means is that the Skills Display is served as its own dedicated url from the Dashboard application of the skills-service and then inserted into an iFrame tag on the client's browser. Of course all of these details are 100% hidden from the skills-client library users. -The skills-service also serves the dashboard as a separate application and that is what is returned by default when users arrive at the "/" url. The skills-client libraries also enable integrators to report skill events. Skill reporting utilities will call the [Report Skill Event Endpoint](/skills-client/endpoints.html#report-skill-event-endpoint) which is exposed via the skills-service. @@ -32,8 +31,8 @@ External clients may report skill events directly via the [Report Skill Event En ## SkillTree Repositories -1. [skills-service](https://github.com/NationalSecurityAgency/skills-service): has code for the skills service, dashboard and client display. This is where the majority of code changes occur. -1. [skills-client](https://github.com/NationalSecurityAgency/skills-client): client JS libraries that provide skill event reporting utilities and a thin iFrame-based wrapper for the client display. +1. [skills-service](https://github.com/NationalSecurityAgency/skills-service): has code for the skills service and dashboard. This is where the majority of code changes occur. +1. [skills-client](https://github.com/NationalSecurityAgency/skills-client): client JS libraries that provide skill event reporting utilities and a thin iFrame-based wrapper for the Skills Display. 1. [skills-docs](https://github.com/NationalSecurityAgency/skills-docs): Documentation, you are reading this now! 1. [skills-stress-test](https://github.com/NationalSecurityAgency/skills-stress-test): Web-based application that facilitates stress tests against the SkillTree service. 1. [call-stack-profiler](https://github.com/NationalSecurityAgency/call-stack-profiler): Groovy annotation-driven in-code profiling utility used by the services. diff --git a/docs/contribution/devEnv.md b/docs/contribution/devEnv.md index 093f02cd..63989c0a 100644 --- a/docs/contribution/devEnv.md +++ b/docs/contribution/devEnv.md @@ -5,24 +5,14 @@ We recommend developing SkillTree in a *nix environment. ## Prerequisites - modern *nix environment -- JDK 17+, we suggest [Open JDK](https://openjdk.java.net/) -- [Git](https://git-scm.com/) version 2.23+ -- [Node.js](https://nodejs.org/en/) v12+ and [npm](https://www.npmjs.com/) 6+ -- [Maven](https://maven.apache.org/) 3.6.2+ -- [PostgreSQL](https://www.postgresql.org/) 12+ +- JDK 19+ +- [Git](https://git-scm.com/) version 2.34+ +- [Node.js](https://nodejs.org/en/) v20+ and [npm](https://www.npmjs.com/) 10+ +- [Maven](https://maven.apache.org/) 3.9+ +- [PostgreSQL](https://www.postgresql.org/) 14+ ## Development Overview -The following sections will cover: -1. skills-service project development -1. skills-client project development - -::: tip -More than likely your development will be scoped to the skills-service project but, although rare, there are situations when you'll need to modify the skills-client as well. -If you are making improvements to the SkillTree dashboard or its Skills Display views then your development will be scoped within the skills-service project. -If you have not yet, please take few minutes to review the [architecture](/contribution/architecture.html) section. -::: - The tech stack is large but to get started you will need to be familiar (but more likely experienced) with these core technologies: - Java and Groovy @@ -32,13 +22,7 @@ The tech stack is large but to get started you will need to be familiar (but mor - [cypress.io](https://www.cypress.io/) - Build tools: maven, npm & webpack -Each sub-section follows the same flow: -1. Get, build, and test -1. [cypress.io](https://www.cypress.io/) end-to-end tests -1. Day-to-day development -1. Test checklist - -## Skills-service development +## Development The skills-service project encapsulates code for the SkillTree dashboard, REST apis and Skills Display views. @@ -68,10 +52,6 @@ skills-service | │ pom.xml | │ package.json | | .... -└───client-display -| │ pom.xml -| │ package.json -| | .... └───e2e-tests | │ package.json | | .... @@ -81,10 +61,8 @@ skills-service The runtime artifact is a [spring boot](https://spring.io/projects/spring-boot) application and will be created in ``service/skills-service-.jar``; we ran ``mvn install`` to generate this artifact and the following sequence of steps were performed: - Build dashboard web application: ``npm run build`` in dashboard project -- Build client display web application: ``npm run build`` in client-display project - Compile Java and Groovy classes in the ``service`` project: ``mvn compile`` - Copy the built dashboard app to ``service/src/main/resources/public`` so the spring boot app can host the dashboard web-application -- Copy the built client-display app to ``service/src/main/resources/public/static/clientPortal`` so the spring boot app can host the client-display web-application - Generate runtime artifact: ``mvn package`` in the service project Of course this doesn't cover the entire build cycle so please familiarize yourself with all of the pom.xml and package.json files. @@ -108,7 +86,7 @@ spring.datasource.password= ### Cypress.io end-to-end tests -SkillTree utilizes the [cypress.io](https://www.cypress.io/) framework to perform end-to-end tests and specifically verify features of the dashboard and client-display web applications. +SkillTree utilizes the [cypress.io](https://www.cypress.io/) framework to perform end-to-end tests and specifically verify features of the dashboard web application. Cypress tests are located in the ``e2e-tests`` project: ``` markdown @@ -123,7 +101,7 @@ skills-service | | | | _spec.js ``` -The end-to-end tests stand-up the skills-service, the dashboard, and the client display web applications and then execute numerous tests against these web-applications to mimic users' actions. +The end-to-end tests stand-up the skills-service and the dashboard applications and then execute numerous tests against these web-applications to mimic users' actions. To run cypress end-to-end tests (assumes that you have built the skills-service.jar via the previous step): @@ -139,16 +117,21 @@ npm run cy:run npm run cyServices:kill ``` -Buckle down as these tests will take a while to run, and depending on your system specs you should expect anywhere from 5 to 20 minutes. +::: tip +Please note that you may need to clear DB tables if they were populated by the previous installation. +Cypress tests purge all the user data except the dashboard root user, so if one was already created it +will need to be manually deleted. The easiest way is to just delete all rows from the `user_attrs` table. +::: + +Buckle down as these tests will take a while to run. Generally only select tests are executed locally and all tests are performed in the CI lifecycle, in parallel. ::: tip -Please note that ports 8080, 8083, 1026, and 1081 have to be available on your system +Please note that ports 8080, 1025, and 1080 have to be available on your system ::: ``npm run cyServices:start`` command starts: - The programmatic service and the dashboard by utilizing the already built jar; will run on port 8080 -- the client-display using webpack dev server; will run on port 8083 -- A mock smtp server; will run on ports 1026 and 1081 +- A mock smtp server; will run on ports 1025 and 1080 Now that you can build and run integration tests let's discuss day-to-day development setup for the skills-service project. @@ -158,13 +141,13 @@ Most IDEs (Intellij, Eclipse, etc...) provide first-class support for maven proj Please note that code additions will generally fall into these categories: 1. Enhancing the skills-service programmatic REST API -1. Making changes to the web-based dashboard application or the client-display web-based application +1. Making changes to the web-based dashboard application ###### skills-service programmatic REST API The code for the API can be found under ``skills-service/service`` which follows standard maven conventions. -The programmatic API tests reside in ```service/src/test/java``` and you can run all of the tests via +The programmatic API tests reside in ```service/src/test/java``` and you can run all the service tests via ```bash cd skills-service/service mvn test @@ -175,7 +158,7 @@ You can also run any of the tests using your favorite IDE. SkillTree's overall testing strategy is to implement black-box integration tests and supplement with unit tests whenever an integration test is not possible. ::: -Generally, programmatic API service development is facilitated via the integration tests . +Generally, programmatic API service development is facilitated via the integration/service tests . Skills Service integration tests stand-up the skills-service application and then execute various endpoints, validating the result. These integration tests reside in the ``skills.intTests`` package and extend the ``DefaultIntSpec.groovy`` class. @@ -194,27 +177,36 @@ A few things to note: There are hundreds of tests in the ``skills.intTests`` package, please feel free to explore. -###### Web-Based Dashboard and/or Client Display +###### Web-Based Dashboard Steps to develop the web-based dashboard are: 1. Stand up the service (programmatic API) -1. Bring up the webpack dev server +1. Bring up the vite dev server 1. Use browser and cypress tests to drive development To stand up the service you can execute ``skills.SpringBootApp`` in the ``service`` project from your IDE. + If that's not an option you can always build a jar and run it from the command line: ```bash java -jar service/target/skills-service-.jar ``` + +At a minimum will need to set DB connection properties: +```properties +spring.datasource.url=jdbc:postgresql://localhost:5432/skills +spring.datasource.username= +spring.datasource.password= +``` + The service will run on port 8080. -Next is to start the webpack dev server in the ``dashboard`` project: +Next is to start the vite dev server in the ``dashboard`` project: ```bash cd dashboard -npm run serve +npm run dev ``` -This webpack dev server will run on port 8082 and will make requests for data to the service running on port 8080. +This vite dev server will run on port 5173 and will make requests for data to the service running on port 8080. Generally development is facilitated by writing cypress tests: @@ -224,13 +216,19 @@ npm run cy:open:dev ``` You can then start adding tests under e2e-tests/cypress/integration to an existing file or by creating a new file. -If you are making changes to the client display OR want to run all of the cypress integration tests, then you will need to start the client-display webpack dev server: +###### Skills Display +[Skills Display](/skills-client/js.html#skills-display) components provides a comprehensive visualization of a user's skill and progress profile! +The code for the Skills Display is mostly encapsulated under `dashboard/src/skills-display` directory with an entry point of `SkillsDisplay.vue` Vue component. -```bash -cd client-display -npm run serve -``` -The dev server will run on port 8083 and will make requests for data to the service running on port 8080. +`SkillsDisplay.vue` is used in various scenarios: +- to show user's progress within a single project on the Progress & Ranking pages +- served as a dedicated url to power [skills-client](https://github.com/NationalSecurityAgency/skills-client) libraries +- server under several urls for testing purposes + +There are three places that Sills Display need to be tested: +- `/test-skills-display/` - most of the cypress tests utilize this url; Skills Display is served natively and is customized for testing +- `/test-skills-client/` - Skills Display is served within an iframe simulating [skills-client](https://github.com/NationalSecurityAgency/skills-client)'s usage; see cypress tests for examples +- `/progress-and-rankings/projects/` - Dashboard's usage of skills-display to show user's progress and rankings for a single project ### Test Checklist @@ -239,9 +237,7 @@ SkillTree's overall testing strategy is to implement black-box integration tests ::: We like tests (especially integration tests), so please make sure you thoroughly test your code. -Prior to making a Pull Request make sure that ALL tests pass: -1. Run all service tests -1. Run all Cypress tests +Prior to making a Pull Request make sure that ALL tests pass, once you push your branch GitHub Actions CI will execute all the service and Cypress tests. Provide the following properties to run the service tests against PostgreSQL: ```properties @@ -265,289 +261,4 @@ npm run cyServices:start npm run cy:run # kill background servers npm run cyServices:kill -``` - -## Skills-client development - -Skills-client JS libraries provide skills event reporting utilities and a thin iFrame-based wrapper for the Client Display. - -::: tip Is this the right project for me? - -Please keep in mind that the majority of the SkillTree features are encapsulated in the [skills-service](/contribution/devEnv.html#skills-service-development) project. - -If you are making improvements to the SkillTree dashboard or its Skills Display views then your development will be scoped within the skills-service project. I -f you have not yet, please take few minutes to review the [architecture](/contribution/architecture.html#skilltree-repositories) section. - -If you are not sure whether you need to make skill-client changes, contact the SkillTree core development team. -::: - - -### Get, build and test - -First things first, **fork** and checkout the code (please see the [Contribution Steps](/contribution/#contribution-steps)), build it and run all of the unit and integration tests. Then we'll discuss the development and testing steps. - -```bash -git clone :skills-client.git -cd skills-client -``` - -skills-client is an assembly of projects that build JS client libraries AND integration tests that enable comprehensive testing of these libraries: - -- **skills-client-js:** client app implementation, please see [Pure JS Integration Guide](/skills-client/js.html#pure-js-integration) to understand how these artifacts are utilized by a SkillTree integrator - - Majority of client code resides here, framework specific libraries are minimal wrappers around this client implementation - - Provides support for [Skill Event Reporting](/skills-client/js.html#skills-event-reporting), [Skills Display](/skills-client/js.html#skills-display) as well as [Skills Configuration](/skills-client/js.html#skills-configuration) -- **skills-client-integration:** integration tests that enable comprehensive testing of skills-client-js and all of the framework specific libraries. - - A set of web-applications that integrate and exercise these skills client libraries - - Cypress tests that utilize these applications to end-to-end test functionality of the skills client libraries - -SkillTree uses [maven](https://maven.apache.org/) and [npm](https://www.npmjs.com/) for its dependency management and to facilitate the build lifecycle. Let's get familiar with the projects layout: -``` markdown -skills-client -└───skills-client-js -| │ package.json -| | .... -└───skills-client-integration -│ │ pom.xml -``` - -To build client libraries and run their unit tests: - -```bash -cd skills-client-js -npm install -npm run build -``` - -To build integration test artifacts: - -```bash -cd skills-client-integration -mvn install -``` -Several actions were performed in the ``mvn install`` command: -- client libraries were built -- client libraries' unit tests were executed -- integration apps were built (these apps utilize client libs) -- This step does not execute end-to-end tests. - -End-to-end [cypress.io](https://www.cypress.io/) tests then exercise these integration applications in order to thoroughly test our client libraries. -The next section will break this process down in detail. - -### Cypress.io end-to-end tests - -There are a number of challenges when integration testing client libraries. -These JS libraries themselves don't present an executable artifact since the purposes of them is to be utilized by the external web applications. - -To enable end-to-end testing of skills-client-js, we developed a set of web-applications that integrate and exercise these libraries. -These applications mimic real world integration and usage scenarios. -[cypress.io](https://www.cypress.io/) is then used to execute numerous end-to-end tests against these web applications, ensuring that these libraries properly work in an integrated environment. - -The code for the web applications that mimics real world usage and end-to-end [cypress.io](https://www.cypress.io/) tests resides in the ``skills-client/skills-client-integration`` project. -Let's take a look at its directory structure: - -``` markdown -skills-client -└───skills-client-integration -│ │ pom.xml -| └───skills-int-client-js -| │ │ package.json -| │ │ pom.xml -| | └───target -| └───skills-int-e2e-test -| │ │ package.json -| │ │ pom.xml -| | └───target -| | | skills-int-e2e-test-.jar -| └───skills-int-service -| │ │ pom.xml -| | └───target -| | | skills-int-service-.jar -``` - -Here is an explanation for each project: -- **skills-int-client-js**: Web application that depends on the skills-client-js artifact and mimics its real world usage -- **skills-int-service**: Pulls together the web application (skills-int-client-js) and exposes a web server to serve these applications. [Cypress.io](https://www.cypress.io/) tests execute tests against this service. -- **skills-int-e2e-test**: [Cypress.io](https://www.cypress.io/) tests that utilize skills-int-service to perform thorough end-to-end tests of the integrated libraries. - -To run [cypress.io](https://www.cypress.io/) tests you will need to -1. Start backend service - - skills-service: REST data endpoints - - serves Skills Display application -1. Start integration web applications (skills-int-service) -1. Run cypress tests - -Please visit the [architecture](/contribution/architecture.html#skilltree-repositories) section to further explore the relationship between these services. - -We are going to assume that you are versed in skills-service installation and development, -but if not please visit the [skills-service development](/contribution/devEnv.html#skills-service-development) section. -Commands below assume: -- ``skills-service-XXX.jar`` was generated and that skills-service and skills-client projects reside in the same parent directory. -- ``skills-int-service--XXX.jar`` was generated via earlier steps. - -To run cypress end-to-end tests: -```bash -cd skills-client/skills-client-integration/skills-int-e2e-test -npm install - -# will start skills-service and skills-int-service processes -npm run cyServices:start - -# run cypress integration tests -npm run cy:run - -# kill background servers (skills-service and skills-int-service) -npm run cyServices:kill -``` -Buckle down as these tests will take a while to run, and depending on your system specs you should expect anywhere from 2 to 15 minutes. - -::: tip -Please note that ports 8080 and 8090 have to be available on your system -::: - -Now that you can build and run the integration tests let's discuss day-to-day development setup for the skill-client project. - -### Day-to-day development -Most IDEs (Intellij, Eclipse, etc...) provide first-class support for maven projects so the very first step would be to import the skills-client project into your favorite IDE. - -Let's first understand JS dependency chain. - -![JS Dependencies](./diagrams/JsDeps.jpg) - -The Diagram above depicts JS dependencies that can be traced using package.json files in each project. -Because we are authoring code across multiple projects in the dependency chain, a traditional [npm-publish](https://docs.npmjs.com/cli/publish) approach falls short. -With the [npm-publish](https://docs.npmjs.com/cli/publish) approach, in order to test the change, the artifact would have to be built and pushed to the npm repository. -Then all of the projects that depend on this resource would have to be updated to point to it (package.json). -And then any follow-on projects that depend on the updated projects need to be modified as well. -That is a large amount of overhead to test a potentially trivial change. - -SkillsTree opted to utilize the [npm link](https://docs.npmjs.com/cli/link) mechanism to work around this issue. -``Npm link`` enables dependency on the local version of the code and will not require a new artifact version to be generated and published. -This way we can make changes to several projects in the chain then build and test them locally before committing. -It also reduces the number of versions we publish and reduces the risk of publishing JS libs with critical bugs. - -We've created a script that facilitates ``npm links`` for all the connections depicted in the above diagram: -```bash -cd skills-client/skills-client-integration/skills-int-e2e-test -npm run dev:setupNpmLinks -``` - -Now that links are established you will be able to make changes to multiple JS projects then build them and see those changes persist. -For example ``skills-int-client-js`` depends on ``skills-client-js``, we can make a code change and see it right away: - -Terminal 1: -```bash -cd skills-client/skills-client-integration/skills-int-client-js -npm run serve -# view dev server on http://localhost:8091/ -``` -Then make a code change in ``skills-client/skills-client-js``, followed by in - -Terminal 2: -```bash -cd skills-client/skills-client-js -npm run build -``` - -The change is automatically reflected in the dev server hosted on ``http://localhost:8091/`` - -What happened is that ``npm run build`` in ``skills-client-js`` created js artifacts in its ``dist`` directory and -``skills-int-client-js`` points to that directory using the ``npm-link`` mechanism. - -Here is the summary of the client lib and its corresponding integration app. - -| client lib | corresponding integration app | -| ------------- | ----------- | -| skills-client-js | skills-int-client-js | - -::: tip -Please note that you can start the development server from an integration app by executing ``npm run serve``. -::: - -::: danger Exception to the Rule -``skills-int-client-js`` has a unique setup and requires that ``npm install`` be run for its dependencies to be picked up! - -This integration apps mimics a pure JS application without the use of webpack (uses ``