Agile like kanban approach to deployment [Draft]
See also: Release Plan, Continuous Delivery
We previously had a couple of attempts for a deployment pipeline, we had criticisms along the direction of we need faster feed back from the users in the deployment pipeline, and also a test driven approach doesn't scale to small projects without a dedicated test team. It was also commented that our milestones have become very large and we need a way to do intermediate releases while we work towards a milestone.
An Agile-like approach was proposed, where we do small pushes towards small goals one step at a time. Agile needs to be tweaked to suit opensource development where developers work in their own time. The goal is to have SCRUM like small goals which we push towards, but since the developers work in their own pace, we work on these small goals through a Kanban process. To help with such quick feedback, we also need a "focus group", which serves as a sample of our userbase.
The Focus group will consist of users who serve as a representative sample of the userbase. Their purpose would be to check the application before release, and help decide if the application would be in a "releasable" state. This sometimes involves playing a shortturn game specifically designed to test some features, daily driving the potential release for longturn games, and mostly being active on the community discord to provide feedback to the developers. The focus group is not meant to be technically driven unlike the testers, but rather approve the "feel" of the application.
A key principle of scrum is the dual recognition that customers will change the scope of what is wanted (often called requirements volatility). The general pipeline of scrum is, there exists a product backlog which is a list of things you want in the product. Then a sprint backlog is created, which lists the things which need to be done in a particular timeframe (called a sprint), At the end of the sprint, we need to have a releasable software and feedback which goes into the product backlog and the next sprints. In our case, the focus group will act as a scrum team, which gives immediate feedback before a release is made and also helps deciding whether the product is in a releasable state.
In our case, each release R-1, R-2, R-3 would act as a sprint. Not with a particular time constraint but with respect to the process.
The Kanban board is located at Release. The board would have 7 sections. To-do, Design, Code Draft, Testing, Merge, Focus Group, Release. As with the principles of Kanban, each section has an upper limit and is prioritized for the release. This helps us keep track of the backlogs more clearly. The Kanban board also logs the time when the issue is moved across sections to keep track of the time spent in particular sections. This helps tweak the pipeline in the next iterations. The process involved within each section is described below.
The todo board is populated with issues which go into the next release, next to next release and so on. (Say, we look ahead to 3 releases for now. R-1, R-2, R-3. These would be tagged in the corresponding issues/pull requests. Developer feedback is used to keep each release possible within a timeframe (say 2 weeks). Following the Agile philosophy, each issue needs to be small and incremental so that we get immediate feedback from the pipeline.
These are issues, where if needed, design of the solution is discussed. Wireframes for UI, infrastructure of the code etc. It is at this phase, the issues are sometimes split into further smaller issues and split across releases R1, R2, R3 or kept for later milestones. In the case of pull requests where the code is directly available we can skip this step and go ahead to the next step.
Pull requests are submitted for a particular issue. If they are at an draft stage feedback is provided by the team to the contributors. If it is a direct pull request, it is recommended that the contributors keep the drafts small so that the team agrees on the direction the solution is taking before working further.
Once a draft is ready for testing, we test the code for expected behavior and the testing pipeline ensures that the code is ready for release. This involves autogames, games with ASAN enabled, some tests with save states etc.
This step involves integrating the pull-request into master. This step spots flaws/conflicts across multiple PRs and takes time to solve these issues/creates further issues for next releases depending on the priority. Once the PRs has been integrated into master and tested by developers, the master is sent to the focus group as a potential candidate for release.
We would have a sample group of users, who would test the code at this stage. This doesnt need to happen at every issue, a bunch of commits put together for a release, are tested by the focus group, who give general opinions about the direction of the project, additional feature requests which go as issues for further releases. They also vote on the priority of things which need be focused by the developers. The main task of the focus group is to approve that the client is in a "releasable" state.
Once a bunch of commits, are approved as being "releasable", a new build is released. These builds are incremental and regular and work towards a minor release according to the milestone.