Skip to content

Commit

Permalink
[E2E Test] Utilities for State Sync Test (#874)
Browse files Browse the repository at this point in the history
## Description

<!-- reviewpad:summarize:start -->
### Summary generated by Reviewpad on 03 Aug 23 20:11 UTC
This pull request introduces several changes across multiple files. 

Here is a summary of the changes:

1. `go.mod`:
   - The `golang.org/x/text` dependency is now listed explicitly instead of being an indirect dependency.

2. `scenario_test.go`:
   - Added a new feature called "State Sync Namespace" that includes various commands and waits for specific amounts of time.
   - Code improvements and TODO comments have been added.

3. `FAQ.md`:
   - Updated an issue related to starting LocalNet with SELinux on an operating system. Replaced the command `make compose_and_watch` with `make lightweight_localnet` to avoid permission denied errors.

4. `.gitignore`:
   - Removed the entry "main" from the list of ignored files.
   - Removed the entries "rpc/server.gen.go" and "rpc/client.gen.go" from the list of ignored files.
   - Added the entry "**/gomock_reflect_*/" to ignore mock temporary files.

5. `e2e/README.md`:
   - Added a new section on `Keywords`.
   - Modified scenario descriptions and code examples to replace instances of "Validator" with "Node".
   - Included a flowchart depicting the E2E scenarios with updated terminology.

6. Consensus module files:
   - Added a new logging statement in the `HandleDebugMessage` function.
   - Simplified the handling of the `DEBUG_CONSENSUS_RESET_TO_GENESIS` action.

7. `.tiltignore`:
   - Removed the entry "main" from the list of ignored files.
   - Removed the entries "rpc/server.gen.go" and "rpc/client.gen.go" from the list of ignored files.

8. `CHANGELOG.md`:
   - Updated build commands and added a new section on `Keywords`.

9. `persistence/docs/CHANGELOG.md`:
   - Several changes related to deprecation, addition, and fixing of functions and issues.

10. `validator.feature`:
    - Renamed file from "valdator.feature" to "validator.feature".
    - Updated scenario titles and step descriptions to use more descriptive terminology.
    - Replaced references to "validator" with "node".

11. `tilt_helpers.go`:
    - Added a new file containing functions related to syncing network configuration and checking package installation.

12. `debug.go`:
    - Added new debug commands and subcommands.
    - Updated existing functions and added new functions for debug actions.
    
13. `account.feature`:
    - Added a new file containing scenarios for testing node account functionalities.

14. `README.md` files:
    - Updated sections, titles, dependencies, and instructions in various README.md files.

15. `build/config/README.md`:
    - Updated usage instructions, changing the command `make compose_and_watch` to `make lightweight_localnet`.

16. `iteration_3_end_to_end_tx.md`:
    - Updated commands to start LocalNet and consensus debugger.

17. Deleted files:
   - `validator.go`
   - `watch_build.sh`

18. New files added:
   - `debug.feature`
   - `tilt_helpers.go`
   - `account.feature`

Please let me know if you need more information about any specific change.
<!-- reviewpad:summarize:end -->

## Issue

Fixes par of #579

## Type of change

Please mark the relevant option(s):

- [x] New feature, functionality or library
- [ ] Bug fix
- [ ] Code health or cleanup
- [ ] Major breaking change
- [ ] Documentation
- [ ] Other <!-- add details here if it a different type of change -->

## List of changes

- `s/compose_and_watch/lightweight_localnet` and all related helpers
- `s/validator/node` in e2e tests for clarity
- Add fire-and-forget `Debug` CLI w/ several useful initial subcommands
- Add `keywords` to the `e2e` document
- Add an `e2e debug` test to trigger views and track the blockchain increasing
- Avoid rebuilding the actors if the CLI changes
- Small miscellaneous improvements & code cleanup

## Testing

- [x] `make develop_test`; if any code changes were made
- [ ] `make test_e2e` on [k8s LocalNet](https://github.com/pokt-network/pocket/blob/main/build/localnet/README.md); if any code changes were made
- [ ] `e2e-devnet-test` passes tests on [DevNet](https://pocketnetwork.notion.site/How-to-DevNet-ff1598f27efe44c09f34e2aa0051f0dd); if any code was changed
- [ ] [Docker Compose LocalNet](https://github.com/pokt-network/pocket/blob/main/docs/development/README.md); if any major functionality was changed or introduced
- [ ] [k8s LocalNet](https://github.com/pokt-network/pocket/blob/main/build/localnet/README.md); if any infrastructure or configuration changes were made

## Required Checklist

- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added, or updated, [`godoc` format comments](https://go.dev/blog/godoc) on touched members (see: [tip.golang.org/doc/comment](https://tip.golang.org/doc/comment))
- [ ] I have tested my changes using the available tooling
- [ ] I have updated the corresponding CHANGELOG

### If Applicable Checklist

- [x] I have updated the corresponding README(s); local and/or global
- [x] I have added tests that prove my fix is effective or that my feature works
- [ ] I have added, or updated, [mermaid.js](https://mermaid-js.github.io) diagrams in the corresponding README(s)
- [ ] I have added, or updated, documentation and [mermaid.js](https://mermaid-js.github.io) diagrams in `shared/docs/*` if I updated `shared/*`README(s)

---

Co-authored-by: d7t <[email protected]>
  • Loading branch information
2 people authored and red-0ne committed Aug 4, 2023
1 parent 92ece19 commit fef4217
Show file tree
Hide file tree
Showing 32 changed files with 715 additions and 303 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -197,4 +197,4 @@ jobs:
- id: "run-e2e-tests"
run: |
./argo-linux-amd64 submit --wait --log --namespace devnet-issue-${{ github.event.pull_request.number }} --from 'wftmpl/dev-e2e-tests' --parameter gitsha="${{ github.event.pull_request.head.sha }}"
./argo-linux-amd64 submit --wait --log --namespace devnet-issue-${{ github.event.pull_request.number }} --from 'wftmpl/dev-e2e-tests' --parameter tags="~@skip_in_ci" --parameter gitsha="${{ github.event.pull_request.head.sha }}"
6 changes: 3 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,6 @@ temp_test.go
test_results.json
coverage.out

# Output of `make build_and_watch`
main

# generated RPC server and client from openapi.yaml
rpc/server.gen.go
rpc/client.gen.go
Expand Down Expand Up @@ -90,3 +87,6 @@ tools/wiki

# ggshield
.cache_ggshield

# mock temporary files
**/gomock_reflect_*/
3 changes: 0 additions & 3 deletions .tiltignore
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,6 @@ temp_test.go
test_results.json
coverage.out

# Output of `make build_and_watch`
main

# generated RPC server and client from openapi.yaml
rpc/server.gen.go
rpc/client.gen.go
Expand Down
51 changes: 29 additions & 22 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ help:
docker_check:
{ \
if ( ! ( command -v docker >/dev/null && (docker compose version >/dev/null || command -v docker-compose >/dev/null) )); then \
echo "Seems like you don't have Docker or docker-compose installed. Make sure you review docs/development/README.md before continuing"; \
echo "Seems like you don't have Docker or docker-compose installed. Make sure you review build/localnet/README.md and docs/development/README.md before continuing"; \
exit 1; \
fi; \
}
Expand All @@ -47,11 +47,21 @@ docker_check:
kubectl_check:
{ \
if ( ! ( command -v kubectl >/dev/null )); then \
echo "Seems like you don't have Kubectl installed. Make sure you review docs/development/README.md before continuing"; \
echo "Seems like you don't have Kubectl installed. Make sure you review build/localnet/README.md and docs/development/README.md before continuing"; \
exit 1; \
fi; \
}

# Internal helper target - check if rsync is installed.
rsync_check:
{ \
if ( ! ( command -v kubectl >/dev/null )); then \
echo "Seems like you don't have rsync installed. Make sure you review build/localnet/README.md and docs/development/README.md before continuing"; \
exit 1; \
fi; \
}


.PHONY: trigger_ci
trigger_ci: ## Trigger the CI pipeline by submitting an empty commit; See https://github.com/pokt-network/pocket/issues/900 for details
git commit --allow-empty -m "Empty commit"
Expand Down Expand Up @@ -133,6 +143,9 @@ go_imports: ## Group imports using rinchsan/gosimports
go_fmt: ## Format all the .go files in the project in place.
gofmt -w -s .

# TODO(#964): add `rsync_check`, `kubectl_check`, `docker_check` as a validation in `install_cli_deps`; https://github.com/pokt-network/pocket/assets/1892194/a7a24a11-f54d-46e2-a73e-9e8ea7d06726
# .PHONY: install_cli_deps
# install_cli_deps: rsync_check kubectl_check docker_check ## Installs `helm`, `tilt` and the underlying `ci_deps`
.PHONY: install_cli_deps
install_cli_deps: ## Installs `helm`, `tilt` and the underlying `ci_deps`
make install_ci_deps
Expand Down Expand Up @@ -163,33 +176,27 @@ develop_test: docker_check ## Run all of the make commands necessary to develop
make develop_start && \
make test_all

.PHONY: client_start
client_start: docker_check ## Run a client daemon which is only used for debugging purposes
.PHONY: lightweight_localnet_client
lightweight_localnet_client: docker_check ## Run a client daemon which is only used for debugging purposes
# Add `--build` to rebuild the client
${docker-compose} up -d client

.PHONY: rebuild_client_start
rebuild_client_start: docker_check ## Rebuild and run a client daemon which is only used for debugging purposes
${docker-compose} up -d --build client

.PHONY: client_connect
client_connect: docker_check ## Connect to the running client debugging daemon
.PHONY: lightweight_localnet_client_debug
lightweight_localnet_client_debug: docker_check ## Connect to the running client debugging daemon
docker exec -it client /bin/bash -c "go run -tags=debug app/client/*.go DebugUI"

.PHONY: build_and_watch
build_and_watch: ## Continous build Pocket's main entrypoint as files change
/bin/sh ${PWD}/build/scripts/watch_build.sh
# IMPROVE: Avoid building the binary on every shell execution and sync it from local instead
.PHONY: lightweight_localnet_shell
lightweight_localnet_shell: docker_check ## Connect to the running client debugging daemon
docker exec -it client /bin/bash -c "go build -tags=debug -o p1 ./app/client/*.go && chmod +x p1 && mv p1 /usr/bin && echo \"Finished building a new p1 binary\" && /bin/bash"

# TODO(olshansky): Need to think of a Pocket related name for `compose_and_watch`, maybe just `pocket_watch`?
.PHONY: compose_and_watch
compose_and_watch: docker_check db_start monitoring_start ## Run a localnet composed of 4 consensus validators w/ hot reload & debugging
.PHONY: lightweight_localnet
lightweight_localnet: docker_check db_start monitoring_start ## Run a lightweight localnet composed of 4 validators w/ hot reload & debugging
# Add `--build` to rebuild the client
${docker-compose} up --force-recreate validator1 validator2 validator3 validator4 servicer1 fisherman1

.PHONY: rebuild_and_compose_and_watch
rebuild_and_compose_and_watch: docker_check db_start monitoring_start ## Rebuilds the container from scratch and launches compose_and_watch
${docker-compose} up --build --force-recreate validator1 validator2 validator3 validator4 servicer1 fisherman1

.PHONY: db_start
db_start: docker_check ## Start a detached local postgres and admin instance; compose_and_watch is responsible for instantiating the actual schemas
db_start: docker_check ## Start a detached local postgres and admin instance; lightweight_localnet is responsible for instantiating the actual schemas
${docker-compose} up --no-recreate -d db pgadmin

.PHONY: db_cli
Expand Down Expand Up @@ -245,7 +252,7 @@ docker_wipe_nodes: docker_check prompt_user db_drop ## [WARNING] Remove all the
docker ps -a -q --filter="name=node*" | xargs -r -I {} docker rm {}

.PHONY: monitoring_start
monitoring_start: docker_check ## Start grafana, metrics and logging system (this is auto-triggered by compose_and_watch)
monitoring_start: docker_check ## Start grafana, metrics and logging system (this is auto-triggered by lightweight_localnet)
${docker-compose} up --no-recreate -d grafana loki vm

.PHONY: docker_loki_install
Expand Down
137 changes: 114 additions & 23 deletions app/client/cli/debug.go
Original file line number Diff line number Diff line change
@@ -1,11 +1,15 @@
package cli

import (
"fmt"
"log"
"os"
"os/exec"
"time"

"github.com/manifoldco/promptui"
"github.com/spf13/cobra"
"golang.org/x/exp/slices"
"google.golang.org/protobuf/types/known/anypb"

"github.com/pokt-network/pocket/app/client/cli/helpers"
Expand Down Expand Up @@ -35,46 +39,123 @@ var items = []string{
}

func init() {
dbg := newDebugCommand()
dbg.AddCommand(newDebugSubCommands()...)
rootCmd.AddCommand(dbg)

dbgUI := newDebugUICommand()
dbgUI.AddCommand(newDebugUISubCommands()...)
rootCmd.AddCommand(dbgUI)
}

// newDebugUISubCommands builds out the list of debug subcommands by matching the
// handleSelect dispatch to the appropriate command.
// * To add a debug subcommand, you must add it to the `items` array and then
// write a function handler to match for it in `handleSelect`.
func newDebugUISubCommands() []*cobra.Command {
commands := make([]*cobra.Command, len(items))
for idx, promptItem := range items {
commands[idx] = &cobra.Command{
Use: promptItem,
// newDebugCommand returns the cobra CLI for the Debug command.
func newDebugCommand() *cobra.Command {
return &cobra.Command{
Use: "Debug",
Aliases: []string{"d"},
Short: "Debug utility for rapid development",
Long: "Debug utility to send fire-and-forget messages to the network for development purposes",
Args: cobra.MaximumNArgs(1),
}
}

// newDebugSubCommands is a list of commands that can be "fired & forgotten" (no selection necessary)
func newDebugSubCommands() []*cobra.Command {
cmds := []*cobra.Command{
{
Use: "PrintNodeState",
Aliases: []string{"print", "state"},
Short: "Prints the node state",
Long: "Sends a message to all visible nodes to log the current state of their consensus",
Args: cobra.ExactArgs(0),
PersistentPreRunE: helpers.P2PDependenciesPreRunE,
Run: func(cmd *cobra.Command, _ []string) {
// TECHDEBT(#874): this is a magic number, but an alternative would be to have the p2p module wait until connections are open and to flush the message correctly
time.Sleep(500 * time.Millisecond) // give p2p module time to start
handleSelect(cmd, cmd.Use)
time.Sleep(500 * time.Millisecond) // give p2p module time to broadcast
Run: func(cmd *cobra.Command, args []string) {
runWithSleep(func() {
handleSelect(cmd, PromptPrintNodeState)
})
},
ValidArgs: items,
}
},
{
Use: "ResetToGenesis",
Aliases: []string{"reset", "genesis"},
Short: "Reset to genesis",
Long: "Broadcast a message to all visible nodes to reset the state to genesis",
Args: cobra.ExactArgs(0),
PersistentPreRunE: helpers.P2PDependenciesPreRunE,
Run: func(cmd *cobra.Command, args []string) {
runWithSleep(func() {
handleSelect(cmd, PromptResetToGenesis)
})
},
},
{
Use: "TriggerView",
Aliases: []string{"next", "trigger", "view"},
Short: "Trigger the next view in consensus",
Long: "Sends a message to all visible nodes on the network to start the next view (height/step/round) in consensus",
Args: cobra.ExactArgs(0),
PersistentPreRunE: helpers.P2PDependenciesPreRunE,
Run: func(cmd *cobra.Command, args []string) {
runWithSleep(func() {
handleSelect(cmd, PromptTriggerNextView)
})
},
},
{
Use: "TogglePacemakerMode",
Aliases: []string{"toggle", "pcm"},
Short: "Toggle the pacemaker",
Long: "Toggle the consensus pacemaker either on or off so the chain progresses on its own or loses liveness",
Args: cobra.ExactArgs(0),
PersistentPreRunE: helpers.P2PDependenciesPreRunE,
Run: func(cmd *cobra.Command, args []string) {
runWithSleep(func() {
handleSelect(cmd, PromptTogglePacemakerMode)
})
},
},
{
Use: "ScaleActor",
Aliases: []string{"scale"},
Short: "Scales the number of actors up or down",
Long: "Scales the type of actor specified to the number provided",
Args: cobra.ExactArgs(2),
PersistentPreRunE: helpers.P2PDependenciesPreRunE,
Run: func(cmd *cobra.Command, args []string) {
actor := args[0]
numActors := args[1]
validActors := []string{"fishermen", "full_nodes", "servicers", "validators"}
if !slices.Contains(validActors, actor) {
logger.Global.Fatal().Msg("Invalid actor type provided")
}
sedReplaceCmd := fmt.Sprintf("/%s:/,/count:/ s/count: [0-9]*/count: %s/", actor, numActors)
sedCmd := exec.Command("sed", "-i", sedReplaceCmd, "/usr/local/localnet_config.yaml")
if err := sedCmd.Run(); err != nil {
log.Fatal(err)
}
},
},
}
return commands
return cmds
}

// newDebugUICommand returns the cobra CLI for the Debug UI interface.
func newDebugUICommand() *cobra.Command {
return &cobra.Command{
Aliases: []string{"dui", "debug"},
Use: "DebugUI",
Short: "Debug selection ui for rapid development",
Short: "Debug utility with an interactive UI for development purposes",
Long: "Opens a shell-driven selection UI to view and select from a list of debug actions for development purposes",
Args: cobra.MaximumNArgs(0),
PersistentPreRunE: helpers.P2PDependenciesPreRunE,
RunE: runDebug,
RunE: selectDebugCommand,
}
}

func runDebug(cmd *cobra.Command, _ []string) (err error) {
// selectDebugCommand builds out the list of debug subcommands by matching the
// handleSelect dispatch to the appropriate command.
// - To add a debug subcommand, you must add it to the `items` array and then
// write a function handler to match for it in `handleSelect`.
func selectDebugCommand(cmd *cobra.Command, _ []string) error {
for {
if selection, err := promptGetInput(); err == nil {
handleSelect(cmd, selection)
Expand Down Expand Up @@ -162,7 +243,17 @@ func handleSelect(cmd *cobra.Command, selection string) {
}
}

// Broadcast to the entire network.
// HACK: Because of how the p2p module works, we need to surround it with sleep both BEFORE and AFTER the task.
// - Starting the task too early after the debug client initializes results in a lack of visibility of the nodes in the network
// - Ending the task too early before the debug client completes its task results in a lack of propagation of the message or retrieval of the result
// TECHDEBT: There is likely an event based solution to this but it would require a lot more refactoring of the p2p module.
func runWithSleep(task func()) {
time.Sleep(1000 * time.Millisecond)
task()
time.Sleep(1000 * time.Millisecond)
}

// broadcastDebugMessage broadcasts the debug message to the entire visible network.
func broadcastDebugMessage(cmd *cobra.Command, debugMsg *messaging.DebugMessage) {
anyProto, err := anypb.New(debugMsg)
if err != nil {
Expand All @@ -178,7 +269,7 @@ func broadcastDebugMessage(cmd *cobra.Command, debugMsg *messaging.DebugMessage)
}
}

// Send to just a single (i.e. first) validator in the set
// sendDebugMessage sends the debug message to just a single (i.e. first) node visible
func sendDebugMessage(cmd *cobra.Command, debugMsg *messaging.DebugMessage) {
anyProto, err := anypb.New(debugMsg)
if err != nil {
Expand Down
2 changes: 1 addition & 1 deletion build/config/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ It is not recommended at this time to build infrastructure components that rely

## Origin Document

Currently, the Genesis and Configuration generator is necessary to create development `localnet` environments for iterating on V1. A current example (as of 09/2022) of this is the `make compose_and_watch` debug utility that generates a `localnet` using `docker-compose` by injecting the appropriate `config.json` and `genesis.json` files.
Currently, the Genesis and Configuration generator is necessary to create development `localnet` environments for iterating on V1. A current example (as of 09/2022) of this is the `make lightweight_localnet` debug utility that generates a `localnet` using `docker-compose` by injecting the appropriate `config.json` and `genesis.json` files.

## Usage

Expand Down
2 changes: 1 addition & 1 deletion build/docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [0.0.0.1] - 2022-12-29

- Updated all `config*.json` files with the missing `max_mempool_count` value
- Added `is_client_only` to `config1.json` so Viper knows it can be overridden. The config override is done in the Makefile's `client_connect` target. Setting this can be avoided if we merge the changes in https://github.com/pokt-network/pocket/compare/main...issue/cli-viper-environment-vars-fix
- Added `is_client_only` to `config1.json` so Viper knows it can be overridden. The config override is done in the Makefile's `lightweight_localnet_client_debug` target. Setting this can be avoided if we merge the changes in https://github.com/pokt-network/pocket/compare/main...issue/cli-viper-environment-vars-fix

## [0.0.0.0] - 2022-12-22

Expand Down
10 changes: 7 additions & 3 deletions build/localnet/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

This guide shows how to deploy a LocalNet using [pocket-operator](https://github.com/pokt-network/pocket-operator).

- [TLDR](#tldr)
- [TL;DR](#tldr)
- [Dependencies](#dependencies)
- [Choosing Kubernetes Distribution](#choosing-kubernetes-distribution)
- [How to create Kind Kubernetes cluster](#how-to-create-kind-kubernetes-cluster)
Expand All @@ -16,6 +16,8 @@ This guide shows how to deploy a LocalNet using [pocket-operator](https://github
- [Interacting w/ LocalNet](#interacting-w-localnet)
- [Make Targets](#make-targets)
- [Addresses and keys on LocalNet](#addresses-and-keys-on-localnet)
- [Applications staked on LocalNet](#applications-staked-on-localnet)
- [Servicers staked on LocalNet](#servicers-staked-on-localnet)
- [How to change configuration files](#how-to-change-configuration-files)
- [Overriding default values for localnet with Tilt](#overriding-default-values-for-localnet-with-tilt)
- [How does it work?](#how-does-it-work)
Expand All @@ -26,7 +28,7 @@ This guide shows how to deploy a LocalNet using [pocket-operator](https://github
- [Full Cleanup](#full-cleanup)
- [Code Structure](#code-structure)

## TLDR
## TL;DR

If you feel adventurous, and you know what you're doing, here is a rapid guide to start LocalNet:

Expand All @@ -46,6 +48,7 @@ All necessary dependencies, except Docker and Kubernetes cluster, are installed
3. `Kubernetes cluster`: refer to [Choosing Kubernetes Distribution](#choosing-kubernetes-distribution) section for more details.
4. `kubectl`: CLI is required and should be configured to access the cluster. This should happen automatically if using Docker Desktop, Rancher Desktop, k3s, k3d, minikube, etc.
5. [helm](https://helm.sh/docs/intro/install): required to template the YAML manifests for the dependencies (e.g., Postgres, Grafana). Installation instructions available.
6. [rsync](https://www.hostinger.com/tutorials/how-to-use-rsync): required to for some extensions used with `Tilt`; https://github.com/tilt-dev/tilt-extensions/tree/master/syncback#usage

### Choosing Kubernetes Distribution

Expand Down Expand Up @@ -149,8 +152,8 @@ For example:
- `0010297b55fc9278e4be4f1bcfe52bf9bd0443f8` is a servicer #001.
- `314019dbb7faf8390c1f0cf4976ef1215c90b7e4` is an application #314.


#### Applications staked on LocalNet

Applications with the following addresses are staked on LocalNet, through the [applications field of the genesis.json in the LocalNet configuration](https://github.com/pokt-network/pocket/blob/main/build/localnet/manifests/configs.yaml#L4088)

- `00001fff518b1cdddd74c197d76ba5b5dedc0301`
Expand All @@ -159,6 +162,7 @@ Applications with the following addresses are staked on LocalNet, through the [a
These addresses can be used for e.g. testing the CLI.

#### Servicers staked on LocalNet

Servicers with the following addresses are staked on LocalNet, through the [servicers field of the genesis.json in the LocalNet configuration](https://github.com/pokt-network/pocket/blob/main/build/localnet/manifests/configs.yaml#L4120)

- `00002b8cea1bcc3dadc72ebecf95564ceb9c2e2a`
Expand Down
Loading

0 comments on commit fef4217

Please sign in to comment.