Signed-off-by: Annaraya Narasagond <[email protected]>
Signed-off-by: Annaraya Narasagond <[email protected]> Annaraya Narasagond <[email protected]>

Signed-off-by: Annaraya Narasagond <[email protected]>
Signed-off-by: Annaraya Narasagond <[email protected]> Annaraya Narasagond <[email protected]>

Signed-off-by: Annaraya Narasagond <[email protected]>
Signed-off-by: Annaraya Narasagond <[email protected]> Annaraya Narasagond <[email protected]>

Previously we used VolumeReplication name and namespace for updating PVC conditions due to the fact
that we had one-to-one relationship between VolumeReplication and PVC. Now when we are going to use
consistency group this handling will not be right anymore, because several PVCs can be related to
one VolumeReplication. So, we need to use PVC name and namespace for updating PVC conditions.

Signed-off-by: Elena Gershkovich <[email protected]>

Signed-off-by: rakeshgm <[email protected]>

Using one item per line to make future changes easier.

Signed-off-by: Nir Soffer <[email protected]>

Current python on Fedora 39 and macOS is 3.12, and 3.13 development
version is available on github actions for a while. Add the versions so
we can ensure compatibility with current and next python version.

We keep python3.9 for compatibility with system using old python like
RHEL 9. At some point we will want to drop old version and consume
newer features in current python, but we don't have anything requiring
this yet, so let try to be compatible.

Signed-off-by: Nir Soffer <[email protected]>

For ramen, the capabilities for hub and
cluster bundles currently states "Basic Install",
Updating it to "Seamless Upgrades".

Fixes: [Bug-2303823](https://bugzilla.redhat.com/show_bug.cgi?id=2303823)
Signed-off-by: Abhijeet Shakya <[email protected]>

Signed-off-by: Abhijeet Shakya <[email protected]>

The changeset includes a new DeleteNamespaceManifestWork() func,
which first checks if the mw.Spec has delete option or if it already
has a DeletionTimestamp. Accordingly, it proceeds to delete the
namespace manifestwork.
It also updates the namespace manifestwork with the deleteOption and
propogationPolicy of type orphan, whenever the createOrUpdateNamespaceManifest()
func is called.

Fixes: [Bug 2059669](https://bugzilla.redhat.com/show_bug.cgi?id=2059669)
Signed-off-by: Abhijeet Shakya <[email protected]>

Signed-off-by: Abhijeet Shakya <[email protected]>

Signed-off-by: Raghavendra Talur <[email protected]>

Signed-off-by: Elena Gershkovich <[email protected]>

In newer controller-runtime version 0.19.0 the check for unique controller names.
In our tests we are registering the same controller (drcluster controller) several times - in
suite_test and in drcluster_mmode_test and in drcluster_drcconfig_tests. As a temporaty solution
I added a flag for skipping the unique controller name validation. Another solution can be
adding a name as a parameter for SetupWithManager function.

Signed-off-by: Elena Gershkovich <[email protected]>

The issue where the "velero.io/exclude-from-backup" label was not applied
originates from our deployment workflow. Initially, we deploy the workload,
followed by enabling DR. During the first deployment, the Placement Operator
creates the "PlacementDecision" with default labels. However, when "Ramen" tries
to add the "velero.io/exclude-from-backup" label during DR setup, it skips because
the "PlacementDecision" already exists. Consequently, "Velero" backs up the
"PlacementDecision". And during hub recovery, it is restored without its status,
leading to the unintended deletion of the workload. This situation only occurs
when the current state wasn't updated before hub recovery was applied.

The fix in this PR does not address the scenario where the workload is deployed,
a hub backup is taken, DR is enabled, and then the hub is recovered before another
backup is created.

Fixes bug: 2308801

Signed-off-by: Benamar Mekhissi <[email protected]>

Fixing this error seen in the CI:

    drenv.commands.Error: Command failed:
       command: ('addons/argocd/test', 'rdr-hub', 'rdr-dr1', 'rdr-dr2')
       exitcode: 1
       error:
          Traceback (most recent call last):
            ...
          drenv.commands.Error: Command failed:
             command: ('kubectl', 'delete', '--context', 'rdr-dr2', 'namespace', 'argocd-test', '--wait=false')
             exitcode: 1
             error:
                Error from server (NotFound): namespaces "argocd-test" not found

Signed-off-by: Nir Soffer <[email protected]>

This commit allows PVs to be resized by enabling the
'AllowVolumeExpansion' flag in the StorageClass.
When this flag is set to true, users can dynamically
adjust the size of PVCs by specifying the desired
capacity in the PVC specification.

Signed-off-by: rakeshgm <[email protected]>

Signed-off-by: rakeshgm <[email protected]>

We updated controller SDK to the 0.19 version, moving the
envtest to the same version for required testing.

This does not fix any test or such, it is just a hygine change.

Signed-off-by: Shyamsundar Ranganathan <[email protected]>

Developers can enable it by copying the file from the hack directory
instead of copying and pasting the code from docs/devel-quick-start.md
and making the hook executable.

Signed-off-by: Nir Soffer <[email protected]>

k8s.io/component-base should have been a direct dependency.
Fixes the go mod error.

Signed-off-by: Raghavendra Talur <[email protected]>

- Integrated the latest CRD from external-snapshotter.
- Updated dependencies and relevant configuration files.
- Changed VolumeSnapshotRefList in VolumeGroupSnapshotStatus to PVCVolumeSnapshotRefList,
which allows to map a VolumeSnapshot to the application PVC.

Signed-off-by: Benamar Mekhissi <[email protected]>

Signed-off-by: Benamar Mekhissi <[email protected]>

This is mainly to ensure that the VRG on the primary and secondary are in sync
in regards to labels and annotations

Signed-off-by: Benamar Mekhissi <[email protected]>

* Allow unprivileged pods to access block devices

We need this for testing volsync with block devices with minikube
clusters.

For minikube clusters this is done via the environment file which is
nicer, but it requires configuring containerd after the cluster has
started, which can cause failures in addons scripts.

We need to upstream this change to minikube later.

Signed-off-by: Nir Soffer <[email protected]>

* Log replication source status in yaml

It is easier to read and works better for viewing the replication logs
in the status.

Signed-off-by: Nir Soffer <[email protected]>

* Improve volsync test teardown

- Delete the replication source before unexporting the volsync service
  using it.
- Log every every teardown step to make debugging easier.

Signed-off-by: Nir Soffer <[email protected]>

* Fail if deleting destination namespace get stuck

Replace delete used for waiting for waiting on the deleted namespace
with a timeout. If deletion get stuck, the test will fail instead of
blocking forever, breaking stress test.

When delete get suck we can inspect the resources in the test gather
directory:

    % tree out/013.gather/dr2/namespaces/busybox
    out/013.gather/dr2/namespaces/busybox
    └── snapshot.storage.k8s.io
        └── volumesnapshots
            └── volsync-busybox-dst-dst-20240914203905.yaml

    % cat out/013.gather/dr2/namespaces/busybox/snapshot.storage.k8s.io/volumesnapshots/volsync-busybox-dst-dst-20240914203905.yaml
    apiVersion: snapshot.storage.k8s.io/v1
    kind: VolumeSnapshot
    metadata:
      creationTimestamp: "2024-09-14T20:39:05Z"
      deletionGracePeriodSeconds: 0
      deletionTimestamp: "2024-09-14T20:39:05Z"
      finalizers:
      - snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
      generation: 2

This looks like an external-snapshotter bug since volsync deleted the
snapshot and removed its finalizers.

Signed-off-by: Nir Soffer <[email protected]>

---------

Signed-off-by: Nir Soffer <[email protected]>

To make it possible to debug platform and machine detection in github
and or in developer environment.

Signed-off-by: Nir Soffer <[email protected]>

All the public functions in the minikube module accept a profile, but
this is actually a profile name. We want to pass a profile dict to
start(). Use `name` for functions accepting a profile names.

Signed-off-by: Nir Soffer <[email protected]>

The minikube module is mostly a thin wrapper for the minikube command,
and we have higher level helpers in __main__.py. Since we wan to have
multiple providers (e.g. lima, external), move all the helpers to the
minikube module.

Signed-off-by: Nir Soffer <[email protected]>

This step is not part of creating vnev, and will be more complicated to
do as part of creating the venv when adding providers. This must be run
manually as we do in the CI.

Signed-off-by: Nir Soffer <[email protected]>

The envfile can have now a "provider" property, defaults to "$provider",
which expands to the platform default provider. The first provider is
minikube.

The setup and cleanup commands requires now an env file, since they need
to get the provider to setup.

Signed-off-by: Nir Soffer <[email protected]>

We want to minimize the provider interface to make it easier to create
new providers.

Signed-off-by: Nir Soffer <[email protected]>

These commands are not very portable, they work only on Linux when using
minikube kvm2 driver.

Signed-off-by: Nir Soffer <[email protected]>

All functions used in drenv/__main__.py pass now the profile dict
instead of the name. This is required for some functions like suspend
and resume, since these operations are available only on certain profile
driver.

The load_files() function was renamed to configure(). The function also
accepts the profile so the provider can configure the cluster based on
the cluster configuration. This will be useful to configure containerd
later.

The setup_files() and cleanup_files() were renamed to setup() and
cleanup(). They do not accept a profile since they are called once per
provider.

Functions are grouped by type: provider scope, cluster scope, and
private helpers.

Signed-off-by: Nir Soffer <[email protected]>

This makes the start flow more generic, and allow every provider to do
the right thing for the profile and cluster status.

Signed-off-by: Nir Soffer <[email protected]>

This is a specific minikube workaround - when starting an existing
cluster, kubernetes reports stale state for a while. The wait is not
needed for external cluster which we never restart. If this will be
needed for other provider we can extract a common helper later.

Signed-off-by: Nir Soffer <[email protected]>

Some commands (like drenv) log to stderr without writing anything to
stdout. When we watch such commands we want to watch stderr instead of
stdout. Since the command do not write anything to stdout, we can
redirect the command stderr to stdout.

When a command fails, we cannot report the error message since it was
already yielded to the code watching the command. This is the issue with
logging everything to stderr, but we don't control the commands we run.

This change add an option to redirect stderr to commands.watch() and
and test the behavior.

Signed-off-by: Nir Soffer <[email protected]>

Like limactl, drenv logs only to stderr, so when running it in tests
with:

    commands.run("drenv", "start", ...)

we don't see anything in the test logs. If the test is blocking for long
time, we have no way to debug this. The helpful log lines are buffered
in the command error buffer.

Use the new stderr= argument to watch and log the command output, and
add helpers for drenv in a consistent way.

Signed-off-by: Nir Soffer <[email protected]>

We used `kubectl version` as a proxy for cluster readynes, checking for
server info in the response. Replace the check with lower level check if
API server /readyz endpoint.

Signed-off-by: Nir Soffer <[email protected]>

Before this change we had only one provider, so this was internal
implementation detail. This change adds the second provider, allowing
users to configure the provider in the environment file.

Replace the `external: true` option with `provider: external`. With this
we can remove the special handling or external cluster with calls to the
external provider which does the right thing.

The external provider basically does nothing, since we do not manage
this cluster. However in start() we ensure that the cluster exists and
then wait until the cluster is ready. This helps to debug issues with
external cluster and reduces log noise.

Signed-off-by: Nir Soffer <[email protected]>

…RamenDR#1485)

* Add logger to DRClusterConfig reconciler

Also, cleanup some scaffolding comments.

Signed-off-by: Shyamsundar Ranganathan <[email protected]>

* Add initial reconcile for DRClusterConfig

- Add finalizer to resource being reconciled
- Remove on delete
- Update reconciler to rate limit max exponential backoff to
5 minutes

Signed-off-by: Shyamsundar Ranganathan <[email protected]>

* Add roles for various storage classes and cluster claims

Signed-off-by: Shyamsundar Ranganathan <[email protected]>

* Add StorageClass listing and dummy functions for claim creation

Building the scaffold for the overall functionality.

Signed-off-by: Shyamsundar Ranganathan <[email protected]>

* Add ClusterClaims for detected StorageClasses

Signed-off-by: Shyamsundar Ranganathan <[email protected]>

* Implement pruning of ClusterClaims

For classes listed, those that do not need a ClusterClaim
any longer are deleted.

Added a StorageClass watcher as well to the reconcile on
changes to StorageClasses.

Signed-off-by: Shyamsundar Ranganathan <[email protected]>

* Implement CLassClaims for VRClass and VSClass

Signed-off-by: Shyamsundar Ranganathan <[email protected]>

---------

Signed-off-by: Shyamsundar Ranganathan <[email protected]>

Working with lima yaml revealed an issue with pyyaml, not preserving
multiline strings when reading and writing yaml. This breaks blocks
using the "|" or "|>" yaml format.

Add a wrapper module configuring pyyaml to preserve multiline
strings. The module provide the minimal interface we use.

The module also enforce good defaults:
- Use only safe load functions
- Don't sort keys to preserve the original object order

All existing users are using the new wrapper now.

Signed-off-by: Nir Soffer <[email protected]>

The module provides:
- merge(): merge one or more source kubeconfigs into a target
  kubeconfig.
- remove(): remove context, cluster, and user matching a name.

We will use this to import lima clusters kubeconfigs after clusters are
started, and remove the kubeconfigs when clusters are stopped or
deleted, matching minikube behavior.

To use kubectl for merging configs, added an env argument for
kubectl.config(). This is required since since the --kubeconfig
parameter access only single kubeconfig.

The module use a lockfile compatible with `kubectl config` or other
programs using the same client-go library. This ensures that concurrent
use of `kubctl config` and multiple drenv threads will not conflict when
modifying the default kubeconfig.

Signed-off-by: Nir Soffer <[email protected]>

This release improves errors handling, build, and testing.

Signed-off-by: Nir Soffer <[email protected]>

On macOS setup is much simpler since most of the tools are available via
the brew package manager.

For creating vms we use the lima project. We need to build upstream
version to consume important fixes and enhancements. We would be able to
use next lima release around October.

For shared network we use socket_vmnet. It should be installed as root
and the easiest way to do this is building from source.

Signed-off-by: Nir Soffer <[email protected]>

We use the "vz" VM type, using Apple virtualization framework, providing
very good performance. A critical feature is Rosetta 2 integration,
supporting running amd64 container images if an arm64 image is not
available. This is required to run OCM since it does not provide arm64
images for all components.

Lima provisioning is very easy to customize, based on cloud-init and
yaml configuration, so unlike Minikube, we can easily configure
everything we need when creating a cluster. However, provisioning is
very slow since it does not support preloading content like Minikube. We
can improve this later by creating our own template disk images with
pre-installed and configured Kubernetes.

For networking we use shared network via socket_vmnet. This is user
based network providing host to VM and VM to VM access. Performance is
poor (1 Gbps, 30 times slower than native networking) but it should be
fast enough for our use case.

Lima is the default provider for Apple silicon machines.  On Linux
minikube is faster and integrated better.

The provider configuration is derived from lima-vm k8s.yaml example,
using only the stable arm64 image and reformatted to fix yamllint
warnings.

Signed-off-by: Nir Soffer <[email protected]>

This is a copy of regional-dr.yaml disabling components that do not work
yet. When we fix all the issues, we can delete this environment and use
the standard one.

- rook-ceph, requires kubernetes-csi/external-snapshotter
- submariner: need to label nodes
- volsync: needs submarinner and cephfs
- argocd: argocd tool fails with PEM error

Signed-off-by: Nir Soffer <[email protected]>

Similar to minikube, starting a stopped cluster is more flaky. Even when
k8s reports that everything is ready, some components are not ready and
running the start hooks can fail randomly.

Example failure:

    Error from server (InternalError): Internal error occurred: failed
    calling webhook "managedclustermutators.admission.cluster.open-cluster-management.io":
    failed to call webhook:
    Post "https://cluster-manager-registration-webhook.open-cluster-management-hub.svc:9443/
    mutate-cluster-open-cluster-management-io-v1-managedcluster?timeout=10s":
    dial tcp 10.110.203.24:9443: connect: no route to host

Try to avoid this by adding a short delay after starting a stopped
cluster, before we start to run the hooks. This change affects only
developers that stop the environment and start it again.

In minikube we added delay in configure(), but for lima is better done
in start(), since there we can tell if this is a start of a stopped
cluster.

Signed-off-by: Nir Soffer <[email protected]>

limactl logs everything to stderr, and we watch stderr to consume the
logs. Since we drop limactl logs when running in normal log level, when
limactl fail we don't have any info on the error, and the only way to
debug limactl error is run with verbose mode.

With this change we extract limactl log level and log errors as drenv
errors, so the last log before the error provide some info on the error.

Tested by uninstalling socket_vmnet:

    sudo make uninstall.launchd

With this limaclt fails to connect to the vmnet socket:

    % drenv start envs/vm.yaml
    2024-09-09 22:30:07,354 INFO    [vm] Starting environment
    2024-09-09 22:30:07,376 INFO    [cluster] Starting lima cluster
    2024-09-09 22:30:26,490 ERROR   [cluster] exiting, status={Running:false Degraded:false
        Exiting:true Errors:[] SSHLocalPort:0} (hint: see "/Users/nsoffer/.lima/cluster/ha.stderr.log")
    2024-09-09 22:30:26,492 ERROR   Command failed
    Traceback (most recent call last):
       ...
    drenv.commands.Error: Command failed:
       command: ('limactl', '--log-format=json', 'start', 'cluster')
       exitcode: 1
       error:

The lima error message is not very useful but this is what we have. This
should be improved in lima.

If we inspect the log file mentioned we can see the actual error:

    % tail -3 ~/.lima/cluster/ha.stderr.log
    {"level":"debug","msg":"Start tcp DNS listening on: 127.0.0.1:51618","time":"2024-09-09T22:30:26+03:00"}
    {"level":"info","msg":"new connection from  to ","time":"2024-09-09T22:30:26+03:00"}
    {"level":"fatal","msg":"dial unix /var/run/socket_vmnet: connect: connection refused","time":"2024-09-09T22:30:26+03:00"}

Signed-off-by: Nir Soffer <[email protected]>

Usually this is an early usage error, written before the logger is
configured to json format, so we get a text log instead o message. Log
this line as is in debug level to allow debugging the issue.

Example error when using older lima version not supporting --log-format:

    2024-09-12 22:25:55,637 DEBUG   [drenv-test-cluster]
    time="2024-09-12T22:25:55Z" level=fatal msg="unknown flag:
    --log-format"

Without this change, this error is dropped and we don't have a clue what
went wrong.

Signed-off-by: Nir Soffer <[email protected]>

We access the cluster via the IP address on the shared network.  Port
forwarding cannot work for multiple clusters since same port from
multiple clusters is mapped to the same host port.

Signed-off-by: Nir Soffer <[email protected]>

Without this the API server will listen on the user network which is not
accessible from the host. Lima try to mitigate this by changing the
address to 127.0.0.1, but this does not work for multiple clusters.

With this change we can access all clusters from the host.

Signed-off-by: Nir Soffer <[email protected]>

Without this configuration the rook-ceph pods are listening on the user
network (192.168.5.0/24) instead of the shared network (192.168.105.0/24),
and rbd-mirror is broken.

With this change we can run the rook environment.

Thanks: Raghavendra Talur <[email protected]>
Signed-off-by: Nir Soffer <[email protected]>

Previously configured as minikube --extra-config.

Signed-off-by: Nir Soffer <[email protected]>

With minikube this is set in the profile, and configured via
--feature-gates flag. With lima we can configure this directly in
KubeletConfiguration. Currently the feature gates are hard coded in the
configuration for all cluster. We can configure based on the profile
later if needed.

Signed-off-by: Nir Soffer <[email protected]>

Currently we have:

    $ sysctl fs.inotify
    fs.inotify.max_queued_events = 16384
    fs.inotify.max_user_instances = 128
    fs.inotify.max_user_watches = 45827

And we see errors like this on managed clusters even with trivial
busybox workloads:

    failed to create fsnotify watcher: too many open files

We use OpenShift worker defaults, already used for minikube[1].

[1] kubernetes/minikube#18832

Signed-off-by: Nir Soffer <[email protected]>

We used 6 month old release, time to upgrade.

Signed-off-by: Nir Soffer <[email protected]>

This makes it work in lima cluster without deploying a csi-hostpath
driver. We can add such driver later if there is a real need.

With this change we can run the minio environment.

Signed-off-by: Nir Soffer <[email protected]>

This allows using commands.run() and commands.watch() with an open file
connected to the child process stdin. We will use this to load images
into lima cluster.

Signed-off-by: Nir Soffer <[email protected]>

Some commands like drenv must run in specific location. Add cwd argument
allowing this when using commands.run() and commands.watch(). We will
use this to run `drenv load` in `ramenctl deploy`, which may run in any
directory.

Signed-off-by: Nir Soffer <[email protected]>

This command loads an image in tar format into all clusters. This will
be used in ramenctl to load images into the clusters, and can also be
used manually. The environment may use one or more provider, and each
one will use the right command to load the image.

External provider do not support loading images. Pushing the ramen image
to a registry will work.

Usage:

    % drenv load -h
    usage: drenv load [-h] [-v] [--name-prefix PREFIX] --image IMAGE filename

    positional arguments:
      filename              path to environment file

    options:
      -h, --help            show this help message and exit
      -v, --verbose         be more verbose
      --name-prefix PREFIX  prefix profile names
      --image IMAGE         image to load into the cluster in tar format

Example run:

    % drenv load --image /tmp/image.tar envs/regional-dr.yaml
    2024-09-10 22:33:30,896 INFO    [rdr] Loading image '/tmp/image.tar'
    2024-09-10 22:33:30,902 INFO    [dr1] Loading image
    2024-09-10 22:33:30,902 INFO    [dr2] Loading image
    2024-09-10 22:33:30,902 INFO    [hub] Loading image
    2024-09-10 22:33:33,314 INFO    [dr1] Image loaded in 2.41 seconds
    2024-09-10 22:33:33,407 INFO    [dr2] Image loaded in 2.50 seconds
    2024-09-10 22:33:33,628 INFO    [hub] Image loaded in 2.73 seconds
    2024-09-10 22:33:33,628 INFO    [rdr] Image loaded in 2.73 seconds

Signed-off-by: Nir Soffer <[email protected]>

With this ramenctl can deploy ramen on any cluster type without knowing
anything about the cluster provider.

Example run:

    % ramenctl deploy --source-dir .. envs/regional-dr-lima.yaml
    2024-09-09 00:52:14,231 INFO    [ramenctl] Starting deploy
    2024-09-09 00:52:14,234 INFO    [ramenctl] Preparing resources
    2024-09-09 00:52:18,192 INFO    [ramenctl] Loading image 'quay.io/ramendr/ramen-operator:latest'
    2024-09-09 00:52:22,023 INFO    [ramenctl] Deploying ramen operator in cluster 'hub'
    2024-09-09 00:52:22,023 INFO    [ramenctl] Deploying ramen operator in cluster 'dr1'
    2024-09-09 00:52:22,025 INFO    [ramenctl] Deploying ramen operator in cluster 'dr2'
    2024-09-09 00:52:22,600 INFO    [ramenctl] Waiting until 'ramen-hub-operator' is rolled out in cluster 'hub'
    2024-09-09 00:52:22,687 INFO    [ramenctl] Waiting until 'ramen-dr-cluster-operator' is rolled out in cluster 'dr1'
    2024-09-09 00:52:22,697 INFO    [ramenctl] Waiting until 'ramen-dr-cluster-operator' is rolled out in cluster 'dr2'
    2024-09-09 00:52:29,893 INFO    [ramenctl] Finished deploy in 15.65 seconds

Signed-off-by: Nir Soffer <[email protected]>

There may be a better way, but for testing setup we could not care less
about certificates checks. We can try to improve this later if we think
that drenv will be used on real clusters.

Thanks: Raghavendra Talur <[email protected]>
Signed-off-by: Nir Soffer <[email protected]>

On lima cluster submariner use the public IP of the host (the address
assigned by your ISP) as the public IP of the clusters, and all
clusters get the same IP:

    % subctl show connections --context dr1
     ✓ Showing Connections
    GATEWAY    CLUSTER   REMOTE IP        NAT   CABLE DRIVER   SUBNETS        STATUS      RTT avg.
    lima-dr2   dr2       93.172.220.134   yes   vxlan          242.1.0.0/16   connected

    % subctl show endpoints --context dr1
     ✓ Showing Endpoints
    CLUSTER   ENDPOINT IP    PUBLIC IP        CABLE DRIVER   TYPE
    dr1       192.168.5.15   93.172.220.134   vxlan          local
    dr2       192.168.5.15   93.172.220.134   vxlan          remote

With this change it uses the actual IP address of the cluster in the
vmnet network:

    % subctl show connections --context dr1
     ✓ Showing Connections
    GATEWAY    CLUSTER   REMOTE IP        NAT   CABLE DRIVER   SUBNETS        STATUS      RTT avg.
    lima-dr2   dr2       192.168.105.10   yes   vxlan          242.1.0.0/16   connected

    % subctl show endpoints --context dr1
     ✓ Showing Endpoints
    CLUSTER   ENDPOINT IP    PUBLIC IP        CABLE DRIVER   TYPE
    dr1       192.168.5.15   192.168.105.11   vxlan          local
    dr2       192.168.5.15   192.168.105.10   vxlan          remote

Thanks: Raghavendra Talur <[email protected]>
Signed-off-by: Nir Soffer <[email protected]>

After provisioning a lima vm we have 2 default routes:

    % limactl shell dr1 ip route show default
    default via 192.168.5.2 dev eth0 proto dhcp src 192.168.5.15 metric 100
    default via 192.168.105.1 dev lima0 proto dhcp src 192.168.105.11 metric 100

192.168.5.0/24 is the special user network used by lima to bootstrap the
VM. All vms use have the same IP address (192.168.5.15) so this network
cannot be used to access the vm from the host.

192.168.105.0/24 is the vmnet shared network, providing access from host
to vm and from vm to vm. We wan to use only this network.

Without this change submariner uses the special user network
(192.168.5.0/24) for the endpoints, which cannot work for accessing the
other clusters:

    % subctl show connections --context dr1
     ✓ Showing Connections
    GATEWAY    CLUSTER   REMOTE IP        NAT   CABLE DRIVER   SUBNETS        STATUS      RTT avg.
    lima-dr2   dr2       192.168.105.10   yes   vxlan          242.1.0.0/16   connected

    % subctl show endpoints --context dr1
     ✓ Showing Endpoints
    CLUSTER   ENDPOINT IP    PUBLIC IP        CABLE DRIVER   TYPE
    dr1       192.168.5.15   192.168.105.11   vxlan          local
    dr2       192.168.5.15   192.168.105.10   vxlan          remote

I tried to fix this issue by deleting the default route via 192.168.5.2.
This works for deploying submariner, but this route is recreated later,
and this breaks submariner gateway and connectivity between the
clusters.

Changing the order of the default routes seem to work, both for
deploying submariner, and for running tests on the running clusters. We
do this by modifying the metric of the preferred route so it becomes
first:

    % limactl shell dr1 ip route show default
    default via 192.168.105.1 dev lima0 proto dhcp src 192.168.105.11 metric 1
    default via 192.168.5.2 dev eth0 proto dhcp src 192.168.5.15 metric 100

With this change the endpoint listen on the public ip (in the vmnet
network), allowing access to other clusters:

    % subctl show connections --context dr1
     ✓ Showing Connections
    GATEWAY    CLUSTER   REMOTE IP        NAT   CABLE DRIVER   SUBNETS        STATUS      RTT avg.
    lima-dr2   dr2       192.168.105.10   no    vxlan          242.1.0.0/16   connected

    % subctl show endpoints --context dr1
     ✓ Showing Endpoints
    CLUSTER   ENDPOINT IP      PUBLIC IP        CABLE DRIVER   TYPE
    dr1       192.168.105.11   192.168.105.11   vxlan          local
    dr2       192.168.105.10   192.168.105.10   vxlan          remote

Signed-off-by: Nir Soffer <[email protected]>

With 0.17.0 and 0.17.2 the globalnet pod fails with:

    2024-09-08T15:42:25.498Z FTL ../gateway_monitor.go:286 Globalnet
    Error starting the controllers error="error creating the Node
    controller: error retrieving local Node \"lima-dr1\": nodes \"lima-dr1\"
    is forbidden: User
    \"system:serviceaccount:submariner-operator:submariner-globalnet\"
    cannot get resource \"nodes\" in API group \"\" at the cluster scope"

This worked with minikube clusters, so maybe this related to the some
difference in the way the cluster is deployed, but we want to upgrade to
latest submariner anyway to detect regressions early.

Signed-off-by: Nir Soffer <[email protected]>

When testing the small submariner environment, we may start deploying
one cluster before the other cluster is ready. This fail randomly with
lima clusters when submariner use the wrong interface.

This may happen if we install submariner before flannel is ready.

Signed-off-by: Nir Soffer <[email protected]>

Remove the nslookup step, since is problematic:
- nslookuop and curl use different DNS resolvers, so when nslookup
  succeeds it does not mean that curl will succeed.
- nslookup sometimes return zero exit code with a message that the
  lookup failed! Then we try to access the DNS name with curl with a
  short timeout (60 seconds) and fail.

Simply to check only with curl, increasing the timeout to 300 seconds.

Signed-off-by: Nir Soffer <[email protected]>

It was configured to exit after 300 seconds, which makes it hard to test
when it takes lot of time to wait for connectivity.

Signed-off-by: Nir Soffer <[email protected]>

Submariner works now so we can enable in the regional-dr-lima.yaml.

Signed-off-by: Nir Soffer <[email protected]>

This addons replaces the minikube volumesnapshot addon, and is needed
for cephfs, volsync, and for testing volume replication of snapshot and
pvcs created from snapshots.

Signed-off-by: Nir Soffer <[email protected]>

This addon works on both minikube and lima clusters. It is used by the
cephfs and volsync and will be used for testing DR for workloads using
rbd pvc restored from snapshot.

To use snapshot with rbd storage a snapshot class was added based on
rook 1.15 example.

With this change we can enable cephfs and volsync in
regional-dr-lima.yaml.

Signed-off-by: Nir Soffer <[email protected]>

We did not flatten the config since it is not needed in minikube, using
path to the certificate. But in lima we get the actual certificate from
the guest, and without flattening we get:

    clusters:
    - name: drenv-test-cluster
      cluster:
        server: https://192.168.105.45:6443
        certificate-authority-data: DATA+OMITTED
    users:
    - name: drenv-test-cluster
      user:
        client-certificate-data: DATA+OMITTED
        client-key-data: DATA+OMITTED
    ...

`DATA-OMITTED` is not a valid certificate, so argocd fail to parse it.

With this change argocd works, and we can use regional-dr.yaml on macOS.

Signed-off-by: Nir Soffer <[email protected]>

Limactl is racy, trying to access files in other clusters directories
and failing when files were deleted. Until this issue is fixed in lima,
ensure that only single vm can be deleted at the same time.

Example failure:

    % drenv delete envs/regional-dr.yaml
    2024-09-13 05:59:57,159 INFO    [rdr] Deleting environment
    2024-09-13 05:59:57,169 INFO    [dr1] Deleting lima cluster
    2024-09-13 05:59:57,169 INFO    [dr2] Deleting lima cluster
    2024-09-13 05:59:57,169 INFO    [hub] Deleting lima cluster
    2024-09-13 05:59:57,255 WARNING [dr2] no such process
    2024-09-13 05:59:57,265 WARNING [dr2] remove /Users/nsoffer/.lima/dr2/ssh.sock: no such file or directory
    2024-09-13 05:59:57,265 WARNING [hub] remove /Users/nsoffer/.lima/hub/ssh.sock: no such file or directory
    2024-09-13 05:59:57,297 ERROR   [dr1] open /Users/nsoffer/.lima/dr2/lima.yaml: no such file or directory
    2024-09-13 05:59:57,297 ERROR   [hub] open /Users/nsoffer/.lima/dr2/lima.yaml: no such file or directory
    2024-09-13 05:59:57,298 ERROR   Command failed
    Traceback (most recent call last):
      ...
    drenv.commands.Error: Command failed:
       command: ('limactl', '--log-format=json', 'delete', '--force', 'dr1')
       exitcode: 1
       error:

Note how delete command for "dr1" and "hub" are failing to read lima.yaml
of cluster "dr2":

    2024-09-13 05:59:57,297 ERROR   [dr1] open /Users/nsoffer/.lima/dr2/lima.yaml: no such file or directory
    2024-09-13 05:59:57,297 ERROR   [hub] open /Users/nsoffer/.lima/dr2/lima.yaml: no such file or directory

With the lock, we run single limactl process at a time, so it cannot
race with other clusters.

Signed-off-by: Nir Soffer <[email protected]>

…menDR#1565)

* Ensure deleted DRPolicies do not add schedules to DRClusterConfig

Signed-off-by: Shyamsundar Ranganathan <[email protected]>

* Correct external snapshotter dependency to 0.8

Was introduced as part of the commit 91e5a5b

Signed-off-by: Shyamsundar Ranganathan <[email protected]>

* Fetch external-snapshotter resources using raw URL

Older scheme was using a git clone URL, this is more efficient as
it fetches required resources direcly and avoids the clone.

Ref.: https://github.com/kubernetes-sigs/kustomize/blob/master/examples/remoteBuild.md#remote-directories

Signed-off-by: Shyamsundar Ranganathan <[email protected]>

---------

Signed-off-by: Shyamsundar Ranganathan <[email protected]>

This brings the new Validated condition needed for fixing disable dr
when a VR precondition has failed. With this change we can use the new
feature in ramen for testing the fix.

Signed-off-by: Nir Soffer <[email protected]>

Done using:

    % go get github.com/csi-addons/[email protected]
    go: upgraded github.com/csi-addons/kubernetes-csi-addons v0.9.1 => v0.10.0
    go: upgraded github.com/google/pprof v0.0.0-20240727154555-813a5fbdbec8 => v0.0.0-20240827171923-fa2c70bbbfe5
    go: upgraded github.com/onsi/ginkgo/v2 v2.20.0 => v2.20.2
    go: upgraded github.com/onsi/gomega v1.34.1 => v1.34.2
    go: upgraded golang.org/x/sys v0.23.0 => v0.24.0
    go: upgraded k8s.io/api v0.31.0 => v0.31.1
    go: upgraded k8s.io/apimachinery v0.31.0 => v0.31.1

    % go mod tidy

Signed-off-by: Nir Soffer <[email protected]>

Signed-off-by: Elena Gershkovich <[email protected]>

use lower version as the hard dependency and
use required hard dependency in toolchain as
not all the system will have the required golang
z version installed

Signed-off-by: Madhu Rajanna <[email protected]>

Command run: `go get -u github.com/ramendr/recipe`

Signed-off-by: Raghavendra Talur <[email protected]>

Signed-off-by: Raghavendra Talur <[email protected]>
Co-Authored-by: Annaraya Narasagond <[email protected]>

Changes are:
1. Find the backup and restore workflow by name as CaptureWorkflow and
   RecoverWorkflow don't exist anymore.
2. Change timeout to int
3. Change command to string type

Signed-off-by: Raghavendra Talur <[email protected]>
Co-Authored-by: Annaraya Narasagond <[email protected]>

Command run: `go get -u github.com/ramendr/ramen/api/v1alpha1`

Signed-off-by: Raghavendra Talur <[email protected]>

This is required as the dr-cluster DRClusterConfig reconciler
manages cluster claims.

This was missed out in commit 91e5a5b

Signed-off-by: Shyamsundar Ranganathan <[email protected]>

Signed-off-by: Elena Gershkovich <[email protected]>

We want to be disable or enable consistency groups support for testing with drenv.
The enabling/disabling flag can be added in envs files:

ramen:
hub: hub
clusters: [dr1, dr2]
topology: regional-dr
features:
volsync: true
consistency_groups: true

When flag is not present, consistency groups are disabled by default

Signed-off-by: Elena Gershkovich <[email protected]>

…rpc watcher

- Moved all resource-watching related functions from drplacementcontrol_controller.go
to a new file drplacementcontrol_watcher.go.
- No functional changes introduced, purely a structural refactor.

Signed-off-by: Benamar Mekhissi <[email protected]>

- Added functionality to watch for changes in DRPolicy resources to trigger DRPC
reconciliation when necessary.

Signed-off-by: Benamar Mekhissi <[email protected]>

Was forgetten when we replaced minikube volumesnapshot addons.

Signed-off-by: Nir Soffer <[email protected]>

The comment is still unclear, but at least easier to read.

Signed-off-by: Nir Soffer <[email protected]>

- Replace "VR under deletion" with "deleted VR", matching the
  terminology used in the code.
- Replace "detected as missing or deleted" with "is missing" when we
  know the VR does not exist.

Signed-off-by: Nir Soffer <[email protected]>

Moved from validateVRStatus() to make room from checking VR VAlidated
status. This also make the function easier to understand, keeping the
same level of abstraction and getting rid of the uninteresting details.

Signed-off-by: Nir Soffer <[email protected]>

Rename msg to errorMsg and document that this is an error message,
if the we could not get the condition value, because it is missing,
stale, or unknown.

Signed-off-by: Nir Soffer <[email protected]>

Log after important changes to the system in delete VR flow to make it
easier to understand what the system is doing, and how ramen changed the
system.

New logs:
- delete the VR resource
- remove annotations from PV

Improve logs:
- remove annotations, labels, and finalizers from PVC

Signed-off-by: Nir Soffer <[email protected]>

When deleting a primary VRG, we wait until the VR Completed condition is
met. However if a VR precondition failed, for example using a drpolicy
without flattening enabled when the PVC needs flattening, the VR will
never complete and the vrg and drpc deletion will never complete.

Since csi-addons 0.10.0 we have a new Validated VR condition, set to
true if pre conditions are met, and false if not. VR is can be deleted
safely in this state, since mirroring was not enabled.

This changes modifies deleted VRG processing to check the new VR
Validated status. If the condition exist and the condition status is
false, validateVRStatus() return true, signaling that the VR is in the
desired state, and ramen completes the delete flow.

If the VR does not report the Validated condition (e.g. old csi-addon
version) or the condition status is true (mirroring in progress), we
continue in the normal flow. The VR will be deleted only when the
Completed condition status is true.

Tested with discovered deployment and vm using a pvc created from a
volume snapshot.

Signed-off-by: Nir Soffer <[email protected]>

Signed-off-by: Abhijeet Shakya <[email protected]>