Persist Automated Rules Across Pod Restarts

[anguzo]

Description:

Hello,

I've encountered an issue with Cryostat deployed via the Helm chart. I created some automated rules for continuous monitoring, but noticed they were lost after a pod restart caused by resource eviction.

Here's the scenario:

1. Deployment: Cryostat was deployed using the Helm chart.

2. Setup: Configured automated rules for specific monitoring tasks.

3. Observation: After some time, the Cryostat pod was evicted due to insufficient memory on the node. A new pod was created to replace it.

   kubectl get pod -n cryostat
   NAME                       READY   STATUS    RESTARTS       AGE
   cryostat-cc4785448-cgxkq   0/6     Error     12 (18d ago)   18d
   cryostat-cc4785448-qqvvc   6/6     Running   2 (10h ago)    38h

4. Eviction Reason:

   kubectl describe pod cryostat-cc4785448-cgxkq -n cryostat
   Status:           Failed
   Reason:           Evicted
   Message:          The node was low on resource: memory. Threshold quantity: 100Mi, available: 102004Ki. Container cryostat-db was using 31116Ki, request is 0, has larger consumption of memory. Container cryostat-storage was using 19292Ki, request is 0, has larger consumption of memory. ...

5. Problem: After the new pod was created, all the previously configured automated rules were lost.

---

Questions:

1. Is there currently any way to persist automated rules? Perhaps a configuration option I'm missing?
2. If not, are there plans to implement persistence for automated rules in future releases?

I also think that it would have been possible to use Init Container (if cryostat-helm supported it) for this. By the way, support for Init Container might also possibly make "declarative config" possible in a sense?

[andrewazores]

Automated Rules are stored in the database. In 3.0 the database is part of the same Deployment but is still a separate container in the Pod, whereas in upcoming 4.0 it has its own Deployment. In any case, it sounds like your database is not configured for persistent storage and whatever was in the database filesystem got dropped when the Pod was evicted. Please check the pvc.* configuration parameters.

I'm not sure what you mean about init containers - what container are you trying to run, for what purpose?

[anguzo]

Thank you for the quick answer. I will check the pvc.* configuration parameters.

What I meant about init containers is, if deployment supported it maybe like so:

spec:
  {{- if .Values.initContainers }}
  initContainers:
    {{- toYaml .Values.initContainers | nindent 8 }}
  {{- end }}

then I guess I could just specify it as configuration parameter for example like so:

initContainers:
  - name: create-automated-rules
    image: curlimages/curl:latest
    command: ["/bin/sh", "-c"]
    args:
      - |
        echo "Creating automated rules in Cryostat"
        curl -X POST "http://localhost:8181/api/v2/rules" \
          -H "Content-Type: application/json" \
          -d '{
            "name": "MyAutomatedRule",
            "description": "Automated rule description",
            "targetAlias": "target-jvm",
            "eventSpecifier": "template=Continuous,type=TARGET",
            "archivalPeriodSeconds": 3600,
            "preservedArchives": 5,
            "enabled": true
          }'

Then I wouldn't care about persisting the automated rules as they can be automatically created on container startup.

[andrewazores]

But if it's an init container then it runs before the other containers, so Cryostat is not yet up and ready to receive this request, I think. If the container is added to a separate Pod then there is the difference in Pod lifecycle and timing to think about.

[andrewazores]

I suppose if this was a sidecar container instead of an init container, then the sidecar could start, poll for Cryostat to be ready, and then submit the API request to create an Automated Rule (or other resource) once Cryostat is ready. I would still prefer to expose the declarative configurations by ConfigMaps and Secrets however, rather than as sidecar container definitions.