Microk8s - MountVolume.SetUp failed

[fornof]

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.]

#235 this is about what I'm running into, but I didn't see how to solve it in the ticket:

   Warning  FailedMount  86s (x12 over 9m40s)  kubelet  MountVolume.SetUp failed for volume "pvc-3da01b7f-3eb3-427d-943d-42758527c8f2" : applyFSGroup failed for vol pvc-3da01b7f-3eb3-427d-943d-42758527c8f2: lstat /var/snap/microk8s/common/var/lib/kubelet/pods/b803ceff-d82e-4fe2-be34-bb05f6bee84d/volumes/kubernetes.io~csi/pvc-3da01b7f-3eb3-427d-943d-42758527c8f2/mount: no such file or directory                                               

I tried again using a deployment mount and I get:

persistent volume:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM
            STORAGECLASS           VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-2e49cd25-4b11-4618-817f-3095de775145   20Gi       RWO            Delete           Released   minio-operator/0-microk8s-ss-0-0   microk8s-hostpath      <unset>                          46d
pvc-3a97c089-e659-4d9e-91a6-62bae21f89f4   76Gi       RWO            Delete           Bound      postgres/postgres-fornof-zfs-pvc   openebs-zfspv-fornof   <unset>                          11h
pvc-3da01b7f-3eb3-427d-943d-42758527c8f2   76Gi       RWO            Delete           Bound      db/postgres-fornof-zfs-pvc         openebs-zfspv-fornof   <unset>                          29m
persistent volume claims : 

{
"apiVersion": "v1",
"items": [
{
"apiVersion": "v1",
"kind": "PersistentVolumeClaim",
"metadata": {
"creationTimestamp": "2024-06-22T16:59:42Z",
"finalizers": [
"kubernetes.io/pvc-protection"
],
"labels": {
"app.kubernetes.io/component": "primary",
"app.kubernetes.io/instance": "postgres-bitnami",
"app.kubernetes.io/name": "postgresql"
},
"name": "data-postgres-bitnami-postgresql-0",
"namespace": "db",
"resourceVersion": "17057302",
"uid": "9a5bb71b-76d0-4a91-a84b-78eb5f9790ee"
},
"spec": {
"accessModes": [
"ReadWriteOnce"
],
"resources": {
"requests": {
"storage": "8Gi"
}
},
"volumeMode": "Filesystem"
},
"status": {
"phase": "Pending"
}
},
{
"apiVersion": "v1",
"kind": "PersistentVolumeClaim",
"metadata": {
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"postgres-fornof-zfs-pvc","namespace":"db"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"76Gi"}},"storageClassName":"openebs-zfspv-fornof"}}\n",
"pv.kubernetes.io/bind-completed": "yes",
"pv.kubernetes.io/bound-by-controller": "yes",
"volume.beta.kubernetes.io/storage-provisioner": "zfs.csi.openebs.io",
"volume.kubernetes.io/storage-provisioner": "zfs.csi.openebs.io"
},
"creationTimestamp": "2024-06-22T17:14:01Z",
"finalizers": [
"kubernetes.io/pvc-protection"
],
"name": "postgres-fornof-zfs-pvc",
"namespace": "db",
"resourceVersion": "17059608",
"uid": "3da01b7f-3eb3-427d-943d-42758527c8f2"
},
"spec": {
"accessModes": [
"ReadWriteOnce"
],
"resources": {
"requests": {
"storage": "76Gi"
}
},
"storageClassName": "openebs-zfspv-fornof",
"volumeMode": "Filesystem",
"volumeName": "pvc-3da01b7f-3eb3-427d-943d-42758527c8f2"
},
"status": {
"accessModes": [
"ReadWriteOnce"
],
"capacity": {
"storage": "76Gi"
},
"phase": "Bound"
}
}
],
"kind": "List",
"metadata": {
"resourceVersion": ""
}
}

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
data-postgres-bitnami-postgresql-0 Pending
44m
postgres-fornof-zfs-pvc Bound pvc-3da01b7f-3eb3-427d-943d-42758527c8f2 76Gi RWO openebs-zfspv-fornof 30m

storage classes: 

NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
openebs-hostpath openebs.io/local Delete WaitForFirstConsumer false 77m
openebs-zfspv-fornof zfs.csi.openebs.io Delete Immediate false 15h

openebs pods: 

NAME READY STATUS RESTARTS AGE
openebs-localpv-provisioner-7cd9f85f8f-6d7cp 1/1 Running 0 76m
openebs-zfs-localpv-controller-7fdcd7f65-74l55 5/5 Running 0 76m
openebs-zfs-localpv-node-6dn8t 2/2 Running 0 76m

rob@media:/mnt$ zfs list
NAME                                                    USED  AVAIL     REFER  MOUNTPOINT
robin-makeup                                            299M  1.76T      283M  /mnt/robin-makeup-zfs
robin-makeup/pvc-3da01b7f-3eb3-427d-943d-42758527c8f2    24K  76.0G       24K  legacy
**What did you expect to happen:**
expected volume to mount

**The output of the following commands will help us better understand what's going on**:
(Pasting long output into a [GitHub gist](https://gist.github.com) or other [Pastebin](https://pastebin.com/) is fine.)

* `kubectl logs -f openebs-zfs-controller-f78f7467c-blr7q -n openebs -c openebs-zfs-plugin`
https://nopaste.net/B3aBBGpU1q
* `kubectl logs -f openebs-zfs-node-[xxxx] -n openebs -c openebs-zfs-plugin`
* https://nopaste.net/yjl29wBl0F
* `kubectl get pods -n openebs`

NAME READY STATUS RESTARTS AGE
openebs-localpv-provisioner-7cd9f85f8f-6d7cp 1/1 Running 0 88m
openebs-zfs-localpv-controller-7fdcd7f65-74l55 5/5 Running 0 88m
openebs-zfs-localpv-node-6dn8t 2/2 Running 0 88m

* `kubectl get zv -A -o yaml`

kubectl get zv -A -o yaml
apiVersion: v1
items:

• apiVersion: zfs.openebs.io/v1
  kind: ZFSVolume
  metadata:
  creationTimestamp: "2024-06-22T17:14:01Z"
  finalizers:
  • zfs.openebs.io/finalizer
    generation: 2
    labels:
    kubernetes.io/nodename: media
    name: pvc-3da01b7f-3eb3-427d-943d-42758527c8f2
    namespace: openebs
    resourceVersion: "17059601"
    uid: 685593db-552a-4daa-838b-4b8137a30c1a
    spec:
    capacity: "81604378624"
    compression: "off"
    dedup: "off"
    fsType: zfs
    ownerNodeID: media
    poolName: robin-makeup
    recordsize: 128k
    volumeType: DATASET
    status:
    state: Ready
    kind: List
    metadata:
    resourceVersion: ""

**Anything else you would like to add:**
I'm trying to get postgresql up and going on kubernetes and having a hard time mounting the persistent volume. 
[Miscellaneous information that will assist in solving the issue.]
I'm using bitnami postgresql with persistence set to true and storage set to true as the same volume claim in the helm chart values. 

**Environment:**
- LocalPV-ZFS version
- openebs.io/version=4.0.0   

- Kubernetes version (use `kubectl version`):

Client Version: v1.30.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.1

- Kubernetes installer & version:
- MicroK8s v1.30.1 revision 6876

- Cloud provider or hardware configuration:
Ubuntu 2TB mirrored storage ZFS pool. 
- OS (e.g. from `/etc/os-release`):
```PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

[Abhinandan-Purkait]

@fornof Did you change the kubelet dir for the microk8s while installing zfs?

--set zfs-localpv.zfsNode.kubeletDir=/var/snap/microk8s/common/var/lib/kubelet/

[avishnu]

@fornof can you provide the information sought by @Abhinandan-Purkait here? #547 (comment)

[avishnu]

Meanwhile, we could fix documentation to cover these steps: https://github.com/openebs/zfs-localpv?tab=readme-ov-file#installation

[fornof]

This can be closed as I am no longer working on getting zfs mounts for microk8s, I ended up just using docker compose for persistent storage/ host storage .
But no , i did not change the mount point

[Abhinandan-Purkait]

Sure @fornof Feel free to reopen this if you come by this again. Thanks

[avishnu]

Can we keep this opened to track the doc changes?

[Abhinandan-Purkait]

@avishnu We already have this documented in the repo readme. https://github.com/openebs/zfs-localpv?tab=readme-ov-file#installation

We can reopen this if we feel it's needed to be added in the docs website. Probably an issue to track this on the website repo would be better place wdyt?

[avishnu]

Yes, it's not updated in the documentation clearly.

[Abhinandan-Purkait]

Have reopened this. cc @balaharish7

[balaharish7]

This issue is documented in #499.