Skip to content

Latest commit

 

History

History
370 lines (268 loc) · 14.5 KB

README.md

File metadata and controls

370 lines (268 loc) · 14.5 KB
Raw

navigator.business.nj.gov

This is the development repository for the work-in-progress business navigator from the New Jersey Office of Innovation. For info on the existing Business.NJ.gov site, please see the bottom of this document.

CircleCI build status tested with jest semantic-release

Architecture

Everything is written in TypeScript and runs on Node.js.

The frontend is React via Next.js and deployed in Docker containers on an AWS Elastic Container Service cluster.

The backend is an Express app deployed as an AWS Lambda function using Serverless Framework. It connects to an AWS DynamoDB instance that is also configured through Terraform.

The app uses AWS Cognito (through AWS Amplify) to handle authentication for registered users. Unregistered users are able to browse the site in guest mode without saving their data to the DynamoDB database.

We deploy using CircleCI for CI/CD.

Development

You will need Node.js (with Yarn installed via npm or corepack) installed for primary development. Additionally, for running the server in local development mode, you will need a Java runtime (for serverless-dynamodb) and Python (for the AWS CLI and some of our scripts) installed (details below).

We recommend using WSL2 if developing on Windows.

For pair programming, we recommend Visual Studio Code with the Live Share extension.

Software requirements

Windows (non-WSL) only:

Pre-flight

You will then setup your AWS credentials:

aws configure

Clone the code and navigate to the root of this repository. There is an install script that will install all yarn packages for both the frontend and backend. It will also set up serverless's local DynamoDB.

./scripts/install.sh

Local env

Before you can run locally, you will need to:

  • create a ./web/.env that includes all the values laid out in the ./web/.env-template file.
  • create a ./api/.env that includes all the values laid out in the ./api/.env-template file.
  • create a .venv virtual environment and install requirements if working on Python:
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Run tests

We use Jest for our TypeScript-based unit tests across our projects. Run all tests with:

yarn test

We use Cypress for end-to-end (e2e) testing. You can run these tests locally with:

./scripts/local-feature-tests.sh

To run all tests with code coverage (locally):

yarn test:coverage

Some of the Cypress tests only run in the CI environment. When using the local-feature-tests script, make sure to have a locally running instance of the application.

We use Python's unittest for our Python tests. Run all Python unit tests with:

yarn test:python

Running locally

Start the services:

yarn start:dev

Troubleshooting

If you get an error from serverless that looks like Inaccessible host: localhost at port 8000, this is likely a permissions error. To solve, grant the ./api/.dynamodb folder write permissions on your machine.

Deploying

Use ship-it to run prettier, linting, and tests before pushing:

./scripts/ship-it.sh

The CircleCI CI/CD (which is configured in .circleci/config.yml) will pick up the job and deploy to the development environment for commits to the main branch.

Frontend deep-dive

The frontend code lives in ./web.

When running locally, the ./scripts/start-web.sh will execute npm run dev for local hot-refreshing of Next.js. In production, we run npm run build which will execute next build and next export. This allows Next.js to build all our pages and pre-render what it can. The export puts the final files in ./web/out directory, which is served via a Docker Container.

Styles

We depend on the compiled version of NJ Web Design Standards, which is included via npm and then imports these styles into our code (which is configured via a line in _app.tsx).

Environment variables

For Next.js, environment variables are inserted at build time. In ./web/next.config.js, the environment variables that the code will have access to are set up for the Next.js framework by pulling them in from the .env file. This works because the system that is building the Next.js code (via CircleCI workflow) has access to these variables via environment variables set during that build step.

Important: This means that any time you build the app, the system building it (your local terminal, for example) needs to have these environment variables set as well, or else they will not get set for the build app.

For Jest testing, the value that will be used for the environment variables is set in ./web/setupTests.js.

For running (and testing) the app locally in development mode, it needs environment variables locally as well. These should be provided via a ./web/.env file which should never be checked into source control. There exists a file ./web/.env-template which provides a blank template to show what variables should be included. This is checked into source control and should be updated any time new frontend variables are added.

Roadmaps

In the ./content/src/roadmaps directory, we have JSON files containing the static content defining tasks and steps for different roadmaps. Each industry has its own roadmap definition, located in ./content/src/roadmaps/industries.

Authentication

Amazon Cognito has a way of tightly coupling itself to code dependencies, which is not great. It's somewhat isolated right now:

  • /signin page uses the AWS Cognito components to render a sign-in page.
  • AuthButton makes use of Cognito's sign-out feature
  • `sessionHelper' is a wrapper around most Cognito Auth features.

For the rest of the app, we use our domain representation of authentication to abstract Cognito away from this:

  • AuthContext provides the current user state to any component that requires it, and is the canonical representation of whether the app is signed-in or signed-out.
  • signinHelper is an abstraction of the business logic for what happens when a user signs in (redirects, etc) and should be triggered after auth actions regardless of authentication provider.

We also configure Cognito authentication programmatically in the Cypress loginByCognitoApi command.

useSWR

We use the useSWR hook for handling userData fetching from the API. This is nice because it has a built-in caching and revalidating system that fetches data when it needs to, and serves cached data when we don't need new data. It acts as a form of state management for React, and any component that needs to access or update the userData state from storage should make use of the useUserData wrapper around this hook.

Backend deep-dive

The backend code lives in ./api. It uses Serverless Framework for handling the integration with AWS Lambdas.

We use Serverless Framework to deploy the backend app. If you do this locally, your local serverless CLI needs to be configured with AWS credentials.

Locally, it uses serverless-offline and serverless-dynamodb to run and simulate the AWS environment. Everything AWS and serverless is configured in ./api/serverless.ts.

The backend app itself is defined in src/functions/migrate.ts and is mostly a regular Express app, except it wraps its export in serverless to become a handler. Then, src/functions/index.ts defines the config structure that proxies all routes through to be handled by the Express routing system.

The rest of the code is regular hexagonal Express structure. The src/api folder contains route definitions and depends on abstractions and types defined in src/domain. The src/db folder contains the DynamoDB-specific logical implementation of the interfaces in the domain. The top-level Express app is responsible for wiring the DynamoDB implementation into the router or anything that has a database dependency. We use jest-dynalite as a testing helper to mock out DynamoDB for testing.

Database schema and migrations

DynamoDB isn't strongly schema'd, but we do expect objects of a certain structure to be returned to the frontend when we request user data. Sometimes we'll be changing the data structure and we need the database to be able to account for and understand this.

We solve this by using document versioning and running migrations on individual documents as we retrieve them from storage. The documents are stored with a schema version number (which is stripped before sending to the frontend). On a get request for a document, if its version is out-of-date with the most recent, we run a series of migrations on it to map the data to the current structure. We then save that new document in the current version, and return it to the frontend.

Reasons for this approach:

  • zero-downtime for the database anytime we change document schema
  • managed only in code, no need to handle AWS lambdas and streams to make a migration happen

Notes about this approach:

  • in the database itself, various documents will be structured differently, as they will only be migrated when they are accessed. This isn't a concern if the documents are only ever accessed through the DB client layer, which performs the migrations as it accesses them

Adding a new migration

If you want to change the structure of the UserData object, here's how:

  1. Create a new file in ./api/src/db/migrations and name it v{X}_descriptionHere.ts where {X} is replaced by the next successive version.

  2. Create a new type in the file and name it v{X}UserData that defines the new structure of your new UserData type.

  3. Create a migration function in the file with type signature (v{X-1}UserData) => v{X}UserData and in here, define the way that the previous version of the object should be mapped to the new structure. Test it.

  4. Add the migration function to the list of functions in ./api/src/db/migrations/migrations.ts. Make sure it's in order at its proper index. Do NOT skip versions because the index of this array must match the index that it is migrating from. ie, migrate_v4_to_v5 must be at index 4 of this array.

  5. Change the types in types.ts for UserData (and factories and anywhere else needed) to reflect the newest version of the type to the rest of the code.

Ports

service local dev & CI feature tests local feature tests unit tests
Next.js frontend 3000 3001
Serverless backend 5002 5001
DynamoDB 8000 8001
Lambda port 5050 5051
Dynalite local 8002

Business.NJ.gov

Business.NJ.gov is being developed by the New Jersey State Office of Innovation, New Jersey Department of State's Business Action Center, and the New Jersey Economic Development Authority with support from departments and agencies across the State.

This site and the Governor's Business First Stop initiative are intended to simplify and streamline access to the information, resources, and services that aspiring entrepreneurs and business owners need to start, operate, and grow their business in the Garden State.

We are launching Business.NJ.gov in beta and using real-time user input to iteratively improve the site and services. The site is also being developed with the support and input of business communities throughout the state.

Built as open source

Business.NJ.gov is an open source project that is meant to serve as a base for anyone who is looking to create an online resource for their own business community. You can find full license details at Business.NJ.gov/license.

Launch a copy of the website

We manage the website through Webflow. To launch your own copy of the site, visit the Webflow showcase and click "Open in Webflow": https://webflow.com/website/State-of-NJ-Business-One-Stop

Repositories and code

A static export of the Business.NJ.gov website source code can be found at the NewJersey/open-business-portal Github Repository.

Thank you

Business.NJ.gov is made possible by building on the work and creativity of the Cities of San Francisco and Los Angeles as well as their technology partners, whose Business Portal content and code base served as a foundation for the site.

Suggestions, feedback, and contributions

Please reach out or leave feedback for us at Business.NJ.gov/feedback. You can also open a GitHub pull request or issue. If you want to get in touch with the Office of Innovation team, please email us at [email protected].

Join the Office of Innovation!

If you are excited to design and deliver modern policies and services to improve the lives of all New Jerseyans, you should join us!