Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add spec.ordinals.start. #448

Merged
merged 44 commits into from
Dec 14, 2024
Merged
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
3d50513
feat: Add staringOrdinal
vimystic Nov 11, 2024
212e049
fix lint error
vimystic Nov 11, 2024
6c4f498
update build pods test to reflect startingordinals usage
vimystic Nov 12, 2024
8c55a1f
add retries for trivy to overcome the throttling
vimystic Nov 14, 2024
9dfdca0
Merge branch 'main' into vimystic/feat/445-add-starting-ordinals
vimystic Nov 18, 2024
62db355
Update: reorg so that we use a similar structure to stateful set inst…
vimystic Nov 19, 2024
b9ccbfa
optimize & use starting ordinal for pvcs too
vimystic Nov 20, 2024
804e076
Update to use starting ordinal in appropriate files
vimystic Nov 20, 2024
75dcd8a
Optimize tests
vimystic Nov 20, 2024
bfe1a51
fix test
vimystic Nov 20, 2024
e088656
fix test
vimystic Nov 20, 2024
0a52ff4
Use Ordinal within spec (duh))
vimystic Nov 20, 2024
91bded5
Remove redundant parameters
vimystic Nov 20, 2024
6bb42dd
Note: this is incorrect since PVC allocations start from 0, only the …
vimystic Nov 20, 2024
a8cc9e0
Attempt to find root cause of mapping issue crash
vimystic Nov 20, 2024
be87c7d
Adding spec.ordinal.start usage back to configmap_builder.go
vimystic Nov 20, 2024
e62a9a9
Adding spec.ordinal.start usage back to node_keybuilder.go
vimystic Nov 20, 2024
5905c9b
Adding spec.ordinal.start usage back to peer_collector.go
vimystic Nov 20, 2024
3b27cf1
Add spec.ordinal.start usage back to service_builder.go
vimystic Nov 20, 2024
65c1d33
service_builder.go
vimystic Nov 20, 2024
51747d4
Update pvc to use name from ordinals , but resource from 0
vimystic Nov 20, 2024
df5a242
revert previous commit
vimystic Nov 20, 2024
8be6661
Update pvc_contro.go to use ordinal start
vimystic Nov 20, 2024
dc23a15
name/label update
vimystic Nov 20, 2024
060d6cf
Attempting range check
vimystic Nov 20, 2024
4a11667
Revert previous
vimystic Nov 20, 2024
3c8739e
Update configmap , key peer and service builders to use correct ordin…
vimystic Nov 20, 2024
fcbcf8a
Merge branch 'main' into vimystic/feat/445-add-starting-ordinals
vimystic Nov 21, 2024
b124b07
Allow instanceoverrides for pvcs (example: changing storage) to wor…
vimystic Nov 27, 2024
ae29f8a
Use 'Ordinals' instead of 'Ordinals' as per review
vimystic Dec 3, 2024
212fd65
Update autogenerated yaml file with 'ordinals'
vimystic Dec 3, 2024
8a24b65
match the description for start with that of the stateful set
vimystic Dec 4, 2024
b1163e0
add happy path test with a non 0 starting ordinal
vimystic Dec 4, 2024
f27d4a3
add node keybuilder test
vimystic Dec 4, 2024
94afe93
test ofr peer collector
vimystic Dec 4, 2024
57019ed
pvc and service builder tests
vimystic Dec 4, 2024
e0afa9b
Ensure correct tags for goloang json
vimystic Dec 4, 2024
4595139
make for loops consistent across all builders
vimystic Dec 4, 2024
79e5c85
fix service builder loop. seperate index from ordinal name since both…
vimystic Dec 4, 2024
c8ba8ce
potential lint error fix (new lines between functions)
vimystic Dec 4, 2024
0bc73a8
remove unnecessary new line
vimystic Dec 4, 2024
ffdaafa
Merge branch 'main' into vimystic/feat/445-add-starting-ordinals
vimystic Dec 4, 2024
2749a79
Update the Ordinals description to match language similar to be simil…
vimystic Dec 5, 2024
1c27636
update with no starting ordinal defined. i.e: Default value 0
vimystic Dec 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 11 additions & 8 deletions .github/workflows/release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,14 @@ jobs:
build-args: VERSION=${{ steps.meta.outputs.version }}

- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@master
with:
image-ref: '${{ fromJSON(steps.meta.outputs.json).tags[0] }}'
format: 'table'
exit-code: '1'
ignore-unfixed: true
vuln-type: 'os,library'
severity: 'CRITICAL,HIGH'
run: |
for i in {1..3}; do
if docker run --rm aquasec/trivy:latest image --exit-code 0 --severity CRITICAL,HIGH --ignore-unfixed ${{ fromJSON(steps.meta.outputs.json).tags[0] }}; then
break
elif [ $i -lt 3 ]; then
echo "Retrying in 60 seconds..."
sleep 60
else
exit 1
fi
done
10 changes: 10 additions & 0 deletions api/v1/cosmosfullnode_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,21 @@ const CosmosFullNodeController = "CosmosFullNode"
// EDIT THIS FILE! THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required. Any new fields you add must have json tags for the fields to be serialized.

// Ordinal specifies the configuration for pod ordinal numbers
type Ordinal struct {
// Start specifies the initial ordinal number for pod naming
// +kubebuilder:validation:Minimum:=0
Start int32 `json:"start,omitempty"`
vimystic marked this conversation as resolved.
Show resolved Hide resolved
}

// FullNodeSpec defines the desired state of CosmosFullNode
type FullNodeSpec struct {
// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
// Important: Run "make" to regenerate code after modifying this file

// Ordinal specifies the configuration for pod ordinal numbers
vimystic marked this conversation as resolved.
Show resolved Hide resolved
Ordinal Ordinal `json:"ordinal,omitempty"`

// Number of replicas to create.
// Individual replicas have a consistent identity.
// +kubebuilder:validation:Minimum:=0
Expand Down
16 changes: 16 additions & 0 deletions api/v1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 10 additions & 0 deletions config/crd/bases/cosmos.strange.love_cosmosfullnodes.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -555,6 +555,16 @@ spec:
Example: cosmos-1
Used for debugging.
type: object
ordinal:
vimystic marked this conversation as resolved.
Show resolved Hide resolved
description: Ordinal specifies the configuration for pod ordinal numbers
properties:
start:
description: Start specifies the initial ordinal number for pod
naming
format: int32
minimum: 0
type: integer
type: object
podTemplate:
description: |-
Template applied to all pods.
Expand Down
2 changes: 1 addition & 1 deletion internal/fullnode/build_pods.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ func BuildPods(crd *cosmosv1.CosmosFullNode, cksums ConfigChecksums) ([]diff.Res
pods []diff.Resource[*corev1.Pod]
)
candidates := podCandidates(crd)
for i := int32(0); i < crd.Spec.Replicas; i++ {
for i := crd.Spec.Ordinal.Start; i < crd.Spec.Ordinal.Start+crd.Spec.Replicas; i++ {
pod, err := builder.WithOrdinal(i).Build()
if err != nil {
return nil, err
Expand Down
38 changes: 23 additions & 15 deletions internal/fullnode/build_pods_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ import (
func TestBuildPods(t *testing.T) {
t.Parallel()

t.Run("happy path", func(t *testing.T) {
t.Run("happy path with starting ordinal", func(t *testing.T) {
crd := &cosmosv1.CosmosFullNode{
ObjectMeta: metav1.ObjectMeta{
Name: "agoric",
Expand All @@ -30,40 +30,44 @@ func TestBuildPods(t *testing.T) {
Image: "busybox:latest",
},
InstanceOverrides: nil,
Ordinal: cosmosv1.Ordinal{
Start: 2,
},
},
}

cksums := make(ConfigChecksums)
for i := 0; i < int(crd.Spec.Replicas); i++ {
cksums[client.ObjectKey{Namespace: crd.Namespace, Name: fmt.Sprintf("agoric-%d", i)}] = strconv.Itoa(i)
cksums[client.ObjectKey{Namespace: crd.Namespace, Name: fmt.Sprintf("agoric-%d", i+int(crd.Spec.Ordinal.Start))}] = strconv.Itoa(i + int(crd.Spec.Ordinal.Start))
}

pods, err := BuildPods(crd, cksums)
require.NoError(t, err)
require.Equal(t, 5, len(pods))

for i, r := range pods {
require.Equal(t, int64(i), r.Ordinal(), i)
expectedOrdinal := crd.Spec.Ordinal.Start + int32(i)
require.Equal(t, int64(expectedOrdinal), r.Ordinal(), i)
require.NotEmpty(t, r.Revision(), i)
require.Equal(t, strconv.Itoa(i), r.Object().Annotations["cosmos.strange.love/config-checksum"])
require.Equal(t, strconv.Itoa(int(expectedOrdinal)), r.Object().Annotations["cosmos.strange.love/config-checksum"])
}

want := lo.Map([]int{0, 1, 2, 3, 4}, func(_ int, i int) string {
want := lo.Map([]int{2, 3, 4, 5, 6}, func(i int, _ int) string {
return fmt.Sprintf("agoric-%d", i)
})
got := lo.Map(pods, func(pod diff.Resource[*corev1.Pod], _ int) string { return pod.Object().Name })
require.Equal(t, want, got)

pod, err := NewPodBuilder(crd).WithOrdinal(0).Build()
pod, err := NewPodBuilder(crd).WithOrdinal(crd.Spec.Ordinal.Start).Build()
require.NoError(t, err)
require.Equal(t, pod.Spec, pods[0].Object().Spec)
})

t.Run("instance overrides", func(t *testing.T) {
t.Run("instance overrides with starting ordinal", func(t *testing.T) {
const (
image = "agoric:latest"
overrideImage = "some_image:custom"
overridePod = "agoric-5"
overridePod = "agoric-7"
)
crd := &cosmosv1.CosmosFullNode{
ObjectMeta: metav1.ObjectMeta{
Expand All @@ -75,18 +79,21 @@ func TestBuildPods(t *testing.T) {
Image: image,
},
InstanceOverrides: map[string]cosmosv1.InstanceOverridesSpec{
"agoric-2": {DisableStrategy: ptr(cosmosv1.DisablePod)},
"agoric-4": {DisableStrategy: ptr(cosmosv1.DisableAll)},
"agoric-4": {DisableStrategy: ptr(cosmosv1.DisablePod)},
"agoric-6": {DisableStrategy: ptr(cosmosv1.DisableAll)},
overridePod: {Image: overrideImage},
},
Ordinal: cosmosv1.Ordinal{
Start: 2,
},
},
}

pods, err := BuildPods(crd, nil)
require.NoError(t, err)
require.Equal(t, 4, len(pods))

want := lo.Map([]int{0, 1, 3, 5}, func(i int, _ int) string {
want := lo.Map([]int{2, 3, 5, 7}, func(i int, _ int) string {
return fmt.Sprintf("agoric-%d", i)
})
got := lo.Map(pods, func(pod diff.Resource[*corev1.Pod], _ int) string { return pod.Object().Name })
Expand All @@ -101,18 +108,19 @@ func TestBuildPods(t *testing.T) {
}
})

t.Run("scheduled volume snapshot pod candidate", func(t *testing.T) {
t.Run("scheduled volume snapshot pod candidate with starting ordinal", func(t *testing.T) {
crd := &cosmosv1.CosmosFullNode{
ObjectMeta: metav1.ObjectMeta{
Name: "agoric",
},
Spec: cosmosv1.FullNodeSpec{
Replicas: 6,
Ordinal: cosmosv1.Ordinal{Start: 2},
},
Status: cosmosv1.FullNodeStatus{
ScheduledSnapshotStatus: map[string]cosmosv1.FullNodeSnapshotStatus{
"some.scheduled.snapshot.1": {PodCandidate: "agoric-1"},
"some.scheduled.snapshot.2": {PodCandidate: "agoric-2"},
"some.scheduled.snapshot.1": {PodCandidate: "agoric-3"},
"some.scheduled.snapshot.2": {PodCandidate: "agoric-4"},
"some.scheduled.snapshot.ignored": {PodCandidate: "agoric-99"},
},
},
Expand All @@ -122,7 +130,7 @@ func TestBuildPods(t *testing.T) {
require.NoError(t, err)
require.Equal(t, 4, len(pods))

want := lo.Map([]int{0, 3, 4, 5}, func(i int, _ int) string {
want := lo.Map([]int{2, 5, 6, 7}, func(i int, _ int) string {
return fmt.Sprintf("agoric-%d", i)
})
got := lo.Map(pods, func(pod diff.Resource[*corev1.Pod], _ int) string { return pod.Object().Name })
Expand Down
7 changes: 4 additions & 3 deletions internal/fullnode/configmap_builder.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,13 @@ const (
func BuildConfigMaps(crd *cosmosv1.CosmosFullNode, peers Peers) ([]diff.Resource[*corev1.ConfigMap], error) {
var (
buf = bufPool.Get().(*bytes.Buffer)
cms = make([]diff.Resource[*corev1.ConfigMap], crd.Spec.Replicas)
cms = make([]diff.Resource[*corev1.ConfigMap], 0, crd.Spec.Replicas)
)
defer bufPool.Put(buf)
defer buf.Reset()
startOrdinal := crd.Spec.Ordinal.Start

for i := int32(0); i < crd.Spec.Replicas; i++ {
for i := startOrdinal; i < startOrdinal+crd.Spec.Replicas; i++ {
data := make(map[string]string)
instance := instanceName(crd, i)
if err := addConfigToml(buf, data, crd, instance, peers); err != nil {
Expand Down Expand Up @@ -75,7 +76,7 @@ func BuildConfigMaps(crd *cosmosv1.CosmosFullNode, peers Peers) ([]diff.Resource
)
cm.Data = data
kube.NormalizeMetadata(&cm.ObjectMeta)
cms[i] = diff.Adapt(&cm, i)
cms = append(cms, diff.Adapt(&cm, int(i-startOrdinal)))
}

return cms, nil
Expand Down
8 changes: 5 additions & 3 deletions internal/fullnode/node_key_builder.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,10 @@ const nodeKeyFile = "node_key.json"
// If the secret already has a node key, it is reused.
// Returns an error if a new node key cannot be serialized. (Should never happen.)
func BuildNodeKeySecrets(existing []*corev1.Secret, crd *cosmosv1.CosmosFullNode) ([]diff.Resource[*corev1.Secret], error) {
secrets := make([]diff.Resource[*corev1.Secret], crd.Spec.Replicas)
for i := int32(0); i < crd.Spec.Replicas; i++ {
secrets := make([]diff.Resource[*corev1.Secret], 0, crd.Spec.Replicas)
startOrdinal := crd.Spec.Ordinal.Start

for i := startOrdinal; i < startOrdinal+crd.Spec.Replicas; i++ {
var s corev1.Secret
s.Name = nodeKeySecretName(crd, i)
s.Namespace = crd.Namespace
Expand Down Expand Up @@ -52,7 +54,7 @@ func BuildNodeKeySecrets(existing []*corev1.Secret, crd *cosmosv1.CosmosFullNode
}
}

secrets[i] = diff.Adapt(&secret, i)
secrets = append(secrets, diff.Adapt(&secret, int(i-startOrdinal)))
}
return secrets, nil
}
Expand Down
4 changes: 3 additions & 1 deletion internal/fullnode/peer_collector.go
Original file line number Diff line number Diff line change
Expand Up @@ -105,12 +105,14 @@ func NewPeerCollector(client Getter) *PeerCollector {
// Collect peer information given the crd.
func (c PeerCollector) Collect(ctx context.Context, crd *cosmosv1.CosmosFullNode) (Peers, kube.ReconcileError) {
peers := make(Peers)
startOrdinal := crd.Spec.Ordinal.Start

clusterDomain := "cluster.local"
if crd.Spec.Service.ClusterDomain != nil {
clusterDomain = *crd.Spec.Service.ClusterDomain
}
for i := int32(0); i < crd.Spec.Replicas; i++ {

for i := startOrdinal; i < startOrdinal+crd.Spec.Replicas; i++ {
secretName := nodeKeySecretName(crd, i)
var secret corev1.Secret
// Hoping the caching layer kubebuilder prevents API errors or rate limits. Simplifies logic to use a Get here
Expand Down
36 changes: 19 additions & 17 deletions internal/fullnode/pvc_builder.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,14 +40,16 @@ func BuildPVCs(

var pvcs []diff.Resource[*corev1.PersistentVolumeClaim]
for i := int32(0); i < crd.Spec.Replicas; i++ {
if pvcDisabled(crd, i) {
ordinal := i + crd.Spec.Ordinal.Start
if pvcDisabled(crd, ordinal) {
continue
}

pvc := base.DeepCopy()
name := pvcName(crd, i)
name := pvcName(crd, ordinal)
pvc.Name = name
pvc.Labels[kube.InstanceLabel] = instanceName(crd, i)
podName := instanceName(crd, ordinal)
pvc.Labels[kube.InstanceLabel] = podName

var dataSource *corev1.TypedLocalObjectReference
var existingSize resource.Quantity
Expand All @@ -65,15 +67,15 @@ func BuildPVCs(
}

tpl := crd.Spec.VolumeClaimTemplate
if override, ok := crd.Spec.InstanceOverrides[instanceName(crd, i)]; ok {
if override, ok := crd.Spec.InstanceOverrides[podName]; ok {
if overrideTpl := override.VolumeClaimTemplate; overrideTpl != nil {
tpl = *overrideTpl
}
}

pvc.Spec = corev1.PersistentVolumeClaimSpec{
AccessModes: sliceOrDefault(tpl.AccessModes, defaultAccessModes),
Resources: pvcResources(crd, name, dataSources[i], existingSize),
Resources: pvcResources(crd, name, dataSources[i], existingSize, tpl.Resources),
StorageClassName: ptr(tpl.StorageClassName),
VolumeMode: valOrDefault(tpl.VolumeMode, ptr(corev1.PersistentVolumeFilesystem)),
}
Expand All @@ -88,24 +90,14 @@ func BuildPVCs(
return pvcs
}

func pvcDisabled(crd *cosmosv1.CosmosFullNode, ordinal int32) bool {
name := instanceName(crd, ordinal)
disable := crd.Spec.InstanceOverrides[name].DisableStrategy
return disable != nil && *disable == cosmosv1.DisableAll
}

func pvcName(crd *cosmosv1.CosmosFullNode, ordinal int32) string {
name := fmt.Sprintf("pvc-%s-%d", appName(crd), ordinal)
return kube.ToName(name)
}

func pvcResources(
crd *cosmosv1.CosmosFullNode,
name string,
dataSource *dataSource,
existingSize resource.Quantity,
tplResources corev1.ResourceRequirements,
) corev1.ResourceRequirements {
var reqs = crd.Spec.VolumeClaimTemplate.Resources.DeepCopy()
reqs := tplResources.DeepCopy()

if dataSource != nil {
reqs.Requests[corev1.ResourceStorage] = dataSource.size
Expand All @@ -129,3 +121,13 @@ func pvcResources(

return *reqs
}
func pvcDisabled(crd *cosmosv1.CosmosFullNode, ordinal int32) bool {
name := instanceName(crd, ordinal)
disable := crd.Spec.InstanceOverrides[name].DisableStrategy
return disable != nil && *disable == cosmosv1.DisableAll
}

func pvcName(crd *cosmosv1.CosmosFullNode, ordinal int32) string {
name := fmt.Sprintf("pvc-%s-%d", appName(crd), ordinal)
return kube.ToName(name)
}
8 changes: 5 additions & 3 deletions internal/fullnode/pvc_control.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,8 @@ func (control PVCControl) Reconcile(ctx context.Context, reporter kube.Reporter,
dataSources := make(map[int32]*dataSource)
if len(currentPVCs) < int(crd.Spec.Replicas) {
for i := int32(0); i < crd.Spec.Replicas; i++ {
name := pvcName(crd, i)
ordinal := i + crd.Spec.Ordinal.Start
name := pvcName(crd, ordinal)
found := false
for _, pvc := range currentPVCs {
if pvc.Name == name {
Expand All @@ -61,7 +62,7 @@ func (control PVCControl) Reconcile(ctx context.Context, reporter kube.Reporter,
}
}
if !found {
ds := control.findDataSource(ctx, reporter, crd, i)
ds := control.findDataSource(ctx, reporter, crd, ordinal)
if ds == nil {
ds = &dataSource{
size: crd.Spec.VolumeClaimTemplate.Resources.Requests[corev1.ResourceStorage],
Expand Down Expand Up @@ -160,7 +161,8 @@ type dataSource struct {
}

func (control PVCControl) findDataSource(ctx context.Context, reporter kube.Reporter, crd *cosmosv1.CosmosFullNode, ordinal int32) *dataSource {
if override, ok := crd.Spec.InstanceOverrides[instanceName(crd, ordinal)]; ok {
podName := instanceName(crd, ordinal)
if override, ok := crd.Spec.InstanceOverrides[podName]; ok {
if overrideTpl := override.VolumeClaimTemplate; overrideTpl != nil {
return control.findDataSourceWithPvcSpec(ctx, reporter, crd, *overrideTpl, ordinal)
}
Expand Down
Loading
Loading