This repository is an initial set of Argo-CD-based deployments of AppStudio components to a cluster and a kcp workspace, plus a script to bootstrap Argo CD onto that cluster (to drive these Argo-CD-based deployments, via OpenShift GitOps) and configure AppStudio kcp workspaces.
This repository is structured as a GitOps monorepo (e.g. the repository contains the K8s resources for multiple applications), using Kustomize.
The contents of this repository are not owned by any single individual, and should instead be collectively managed and maintained through PRs by individual teams.
You may use the has component as an example for how to add your own component. Here has refers to the HAS service team's K8s resources.
These are the steps to add your own component:
- Create a new directory for your team's components, under
components/(team-name). - Add a
kustomization.yamlfile under that directory, which points to the individual K8s YAML resources you wish to deploy.- You may also structure your deployment into directories and files. See the Kustomize documentation for more information, and/or examples below.
- See
components/has/stagingfor an example of this.
- Create an Argo CD
ApplicationSetresource inargo-cd-apps/base/(team-name).yaml).- See
has.yamlfor a template of how this should look. - The
.spec.template.spec.source.pathvalue should point to the directory you created in previous step. - The
.spec.template.spec.destination.namespaceshould match the target namespace you wish to deploy your resources to. - The suffix of the
.spec.template.metadata.nameshould correspond to your team name, but keep the{{kcp-name}}-prefix for proper templating:{{kcp-name}}-(team-name).
- See
- Add a reference to your new
(team-name).yamlfile, toargo-cd-apps/base/kustomization.yaml(the reference to your YAML file should be in theresources:list field). - Run
kustomize build (repo root)/argo-cd-apps/overlays/stagingand ensure it passes, and outputs your new Argo CD Application CR. - Add an entry in
argo-cd-apps/overlays/development/repo-overlay.yamlfor your new component so you can use the preview mode for testing. - Open a PR for all of the above.
More examples of using Kustomize to drive deployments using GitOps can be found here.
Simply update the files under components/(team-name), and open a PR with the changes.
TIP: For development purposes, you can use kustomize build . to output the K8s resources that are being generated for your folder.
The prerequisites are:
- You must have
kubectl(> v1.24),oc,jq,openssl,realpathandyqinstalled. - kubectl-kcp plugin
- kubectl-oidc_login plugin (only when connecting to CPS)
- You must have
kubectlandocpointing to an existing OpenShift cluster, that you wish to deploy to. Alternatively, you can configure a local CodeReady Containers VM to deploy to. - You must have another
kubeconfigpointing to an existing kcp instance, that you wish to deploy to. You can use either a CPS or a local kcp instance. - The script
./hack/setup/install-pre-req.shwill install these prerequisites for you, if they're not already installed.
If you don't already have a test OpenShift cluster available, CodeReady Containers is a popular option. It runs a small OpenShift cluster in a single VM on your local workstation.
- Create or log in using your free Red Hat account, and install CodeReady Containers (CRC).
- Make sure you have the latest version of CRC:
crc version - Run
./hack/setup/prepare-crc.shto configure CodeReady Containers with the recommended minimum memory (16 GiB) and CPUs (6) for App Studio. The script has optional parameters for customizingmemoryandcpuallowance. It also supportsforce deleteof existing cluster. Run./hack/setup/prepare-crc.sh --helpto see the options. The script will also enable cluster monitoring and log you in as the cluster administrator.
If you are using Quicklab to provision your development cluster, you will need to setup storage prior to running the bootstrap script.
To boostrap AppStudio run:
./hack/bootstrap.sh -kk [kubeconfig-pointing-to-kcp] -ck [kubeconfig-pointing-to-openshift] -rw [workspace-to-be-used-as-root] -m [mode|upstream,dev,preview-ckcp,preview-cps]which will:
- Bootstrap Argo CD (using OpenShift GitOps) - it will output the Argo CD Web UI route when it's finished.
- Create kcp workspaces and SyncTarget pointing to the OpenShift cluster.
- Create an ArgoCD representation of a cluster pointing to
redhat-appstudioworkspace (as a secret in ArgoCD format). - Setup the Argo CD
Application/ApplicationSetCustom Resources (CRs) for each component.
upstream(default) mode will expect access to both CPS instanceskcp-stableandkcp-unstable- each of them should be represented by a kubeconfig context having the same name as the CPS instance.devmode will use one kcp instance as the deployment target - it can be any instance (either CPS or local kcp). The current kubeconfig context should point to it.preview-[ckcp,cps]mode will enable preview mode used for development and testing on non-production clusters using the same deployment target asdevmode. See Preview mode for your clusters.
If -rw | --root-workspace parameter is not specified, then by default, all workspaces are automatically created under the root workspace.
There are two workspaces created per kcp instance:
redhat-appstudio-internal-compute- This is the workspace where the SyncTarget for the OpenShift workload cluster is configured. If the root workspace is different fromroot, then the name of the workspace is set tocomputeto work around this issue. (The name of the workspace can be overridden by setting theCOMPUTE_WORKSPACEvariable)redhat-appstudio- In this workspace ArgoCD deploys all kcp-related manifests from the infra-deployments repository. It's the place where all AppStudio components run. (The name of the workspace can be overridden by setting theAPPSTUDIO_WORKSPACEvariable)
If you decide to run the upstream mode, then the bootstrap.sh script tries to configure two instances of kcp: kcp-stable and kcp-unstable. However, kcp-stable instance may require different version of kubectl kcp plugin than the kcp-unstable one. This makes running the bootstrap script impossible for the upstream mode, because you cannot use two versions of the plugin at the same time.
To work around the issue, you can skip the configuration of the kcp part by using -sk | --skip-kcp parameter <true/false>:
./hack/bootstrap.sh -sk true ...The bootstrap.sh script then configures only ArgoCD (including the Application/ApplicationsSets) in the workload cluster and doesn't do anything for kcp.
To configure the kcp instances separately use the configure-kcp.sh script:
./hack/configure-kcp.sh -kk [kubeconfig-pointing-to-kcp] -ck [kubeconfig-pointing-to-openshift] -rw [workspace-to-be-used-as-root] -kn [kcp-name|kcp-stable,kcp-unstable,dev]which takes care of creation of the workspaces, SyncTarget, and the representation of the cluster in ArgoCD for the given kcp instance. So to fully finish the upstream configuration run the script for both kcp instances using the parameter -kn kcp-stable and -kn kcp-unstable.
Open the Argo CD Web UI to see the status of your deployments. You can use the route from the previous step and login using your OpenShift credentials (using the 'Login with OpenShift' button), or login to the OpenShift Console and navigate to Argo CD using the OpenShift Gitops menu in the Applications pulldown.
If your deployment was successful, you should see several applications running, such as "all-components", "has", and so on.
Even with 6 CPU cores, you will need to reduce the CPU resource requests for each App Studio application. Either run ./hack/reduce-gitops-cpu-requests.sh which will set resources.requests.cpu values to 50m or use kubectl edit argocd/openshift-gitops -n openshift-gitops to reduce the values to some other value. More details are in the FAQ below.
Once you bootstrap your environment without -m preview-ckcp or -m preview-cps, the root ArgoCD Application and all of the component applications will each point to the upstream repository. Or you can bootstrap cluster directly in mode which you need.
To enable development for a team or individual to test changes on your own cluster, you need to replace the references to https://github.com/redhat-appstudio/infra-deployments.git with references to your own fork.
There are a set of scripts that help with this, and minimize the changes needed in your forks.
There is a development configuration in argo-cd-apps/overlays/development which includes a kustomize overlay that can redirect the default components individual repositories to your fork.
The script also supports branches automatically. If you work in a checked out branch, each of the components in the overlays will mapped to that branch by setting targetRevision:.
Preview mode works in a feature branch, apply script which creates new preview branch and create additional commit with customization.
Two preview modes are offered:
preview-ckcpdeploys containerized KCP directly into your cluster and connects the cluster.preview-cpsconnects to CPS instance. Before the mode is used the kubeconfig file must be created manually based on 'CSP onboarding document' and setCPS_KUBECONFIGvariable inhack/preview.env
Usage:
./hack/bootstrap.sh -m preview-ckcp
or
./hack/bootstrap.sh -m preview-cps
Steps:
- Copy
hack/preview-template.envtohack/preview.envand update new file based on instructions. Filehack/preview.envshould never be included in commit. - Work on your changes in a feature branch
- Commit your changes
- Run
./hack/preview.sh, which will do: a) New branch is created from your current branch, the name of new branch ispreview-<name-of-current-branch>b) Commit with changes related to your environment is added into preview branch c) Preview branch is pushed into your fork d) ArgoCD is set to point to your fork and the preview branch e) User is switched back to feature branch to create additional changes
If you want to reset your environment you can run the hack/util-update-app-of-apps.sh https://github.com/redhat-appstudio/infra-deployments.git staging main to reset everything including your cluster to https://github.com/redhat-appstudio/infra-deployments.git and match the upstream config.
Note running these scripts in a clone repo will have no effect as the repo will remain https://github.com/redhat-appstudio/infra-deployments.git
Authorization is managed by components/authorization. Authorization is disabled in dev and preview mode.
For access to OpenShift Stage cluster the github user has to be part of stage team in redhat-appstudio-sre organization.
Authorization in root:redhat-appstudio workspace in CPS is managed by components/authorization/kcp/members-view.yaml.
Other questions? Ask on #wg-developer-appstudio.
Ultimately, as a team, we should decide on a resource deployment strategy going forward: whether we want every team's K8s resources to be defined within this repository (as a GitOps monorepo), or within individual team's Git repositories. IMHO it is easiest to coordinate deployments within a single Git repository (such as this one), rather than multiple independent repositories.
However, if one or more services want to split off their K8s resource into independent repositories owned by those teams, they can modify the Argo CD ApplicationSet CR for their service (created in 'How to add your own component' step 3, above) to point to their new repository.
Q: When Argo CD attempts to synchronize my Argo CD Application, I am seeing 'the server could not find the requested resource' sync error on my custom resources. How can I fix this?
Before Argo CD attempts a synchronize operation (syncing your Git repository with the K8s cluster), it performs a dry-run to ensure that all the K8s resources in your Git repository are valid. If your repository contains custom resources which are not yet defined, it will refuse to begin the synchronize operation.
The easiest way to solve this is to add this annotation to your custom resources operands: argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true.
For example, we would add this annotation to all of our Pipeline CRs:
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
# Add this annotation to your CRs:
annotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=trueIf you are using Kustomize, you can place the following in your kustomization.yaml file to automatically add it to all resources:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- # (your resources)
# Add these lines to your kustomization.yaml:
commonAnnotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=trueSee the Argo CD docs for more on this sync option. See the redhat-cop/gitops-catalog for examples of this option used with Kustomize and OLM-installed operators.
If you stopped and started your cluster, a timing issue might cause the following message to appear
when you try logging into ArgoCD with Log In Via OpenShift:
Failed to query provider "https://openshift-gitops-server-openshift-gitops.apps.myserver.mydomain.com/api/dex": Get "http://openshift-gitops-dex-server.openshift-gitops.svc.cluster.local:5556/api/dex/.well-known/openid-configuration": dial tcp 1##.###.###.###:5556: connect: connection refused
To correct this problem:
- Delete the openshift-gitops-dex-server-* pod:
oc delete pods -n openshift-gitops -l app.kubernetes.io/name=openshift-gitops-dex-server
- The pod will automatically restart and ArgoCD
Log In Via OpenShiftshould be working again.
Q: What is the recommended memory and CPU allocation for CodeReady Containers for development purposes?
We recommend 7+ cores and 24+ GiB (24576 MiB) of memory.
Q: When using CodeReady Containers for development purposes, I am getting an error message similar to: 0/1 nodes available: insufficient memory.
The default worker node memory allocation of 8192 MiB insufficient to run App Studio. Increase the memory to 16 MiB using crc config set memory 16384 and then create a new CRC VM to apply your changes, using crc delete and crc start. Finally, repeat the cluster bootstrapping process.
See the CodeReady Containers docs for more on this configuration option.
Q: When using CodeReady Containers for development purposes, I am getting an error message similar to: 0/1 nodes available: insufficient cpu.
The default 4-CPU allocation will not be sufficient for the CPU resource requests in this repo. Increase number of cores, for example, crc config set cpus 6 if your hardware supports it, and then create a new CRC VM to apply your changes, using crc delete and crc start. Finally, repeat the cluster bootstrapping process.
See the CodeReady Containers docs for more on this configuration option.
Even with 6 CPU cores, you will need to reduce the CPU resource requests for each App Studio application. Using kubectl edit argocd/openshift-gitops -n openshift-gitops, reduce the resources.requests.cpu values from 250m to 100m or less. For example, change each line with
requests:
cpu: 250m
to
requests:
cpu: 100m
Then save and exit the editor. The updates will be applied to the cluster immediately, and the App Studio deployment should complete within a few minutes.
There is an OWNERS file present in each component folder like this, people mentioned in the file have the respective access to approve/review PR's.
To add yourself change the OWNERS file present in your component folder and Raise a pull request, if you want to be a Approver for the entire repo please change the OWNERS file present in the root level of this repository
Difference Between Reviewers and Approvers
More about code review using OWNERS