Unable to create HDFS clusters in EKS and Rosa

[rajarshp]

Hi Team,

We are using AWS EKS and doing our configuration.
We are using this Stackable HDFS operator for the first time. We did follow the instruction and installed the HDFS, listener operator.

Post this I was trying to install the HDFS cluster as per the - https://docs.stackable.tech/home/stable/hdfs/getting_started/first_steps

we want to use the service type as Load-balancer, so we did use the listener class as external-stable. But when we apply the HDFS file getting this error : -

Events:
Type     Reason                  Age                From                     Message
----     ------                  ----               ----                     -------
Normal   Scheduled               46s                default-scheduler        Successfully assigned test/test-service-hdfs-namenode-default-0 to ip-xxxxxxxxx
Normal   SuccessfulAttachVolume  43s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-193640e0-e0c9-4eb5-8635-7dxxxxxx"
Warning  FailedMount             13s (x7 over 45s)  kubelet                  MountVolume.SetUp failed for volume "pvc-XXXXXXX-27a4-4855-a0fb-7155XXXXXXX" : rpc error: code = Unavailable desc = listener has no addresses yet

we have no idea how we can fix this ?

Datanode was also having same error but node moved to running stage after a while.

Can you someone let me know how I can fix this ?

[sbernauer]

Hi @rajarshp is the problem solved by now? In the message I can see the error only is 45 seconds old. The start of the namenode is delayed until the service of type LoadBalancer is provisioned by AWS, which can take longer than 45 seconds.

In case the problem still persists, can you please provide the output of the kubectl get service command? Maybe the LoadBalander is still pending.

[rajarshp]

thanks for the direction. I checked that the load balancer was not up and running. I fixed and now it running

[rajarshp]

Hi @sbernauer
I am working on this for the first time.

So,now trying to configure them in aws Rosa. I applied the files but little but confused how I can get the load balancer url which I can use to access it from outside of Rosa or from EKS

Also, when I was creating in EKS I saw among the 2 node listeners one took LoadBalancer but another took NodePort, though the listener class I given as external-stable

listener-test-hdfs-namenode-default-0 LoadBalancer
listener-test-hdfs-namenode-default-1 NodePort

When I am applying the same file in AWS Rosa I can see the both listener type is taking as NodePort

  clusterIPs:
    - x.x.x.x
  type: NodePort
  ipFamilyPolicy: SingleStack
  sessionAffinity: None
  selector:
    listener.stackable.tech/mnt.ab69a6ed1d5eae02118f673f: listener-test-hdfs-namenode-default-1
status:
  loadBalancer: {}

[sbernauer]

Did you change the listenerClass after the HDFS cluster was deployed? The reason I'm asking is that the namenodes tries to maintain a stable address (even when using NodePorts). For this to work the listener-operator creates PVCs to "keep track" of that address.
What you could try is to delete all pvcs of your HdfsCluster having "listener" in their name and restart all HDFS Pods. This should cause listener-op to hand out new addresses and switch NameNode 0 to NodePort

[NickLarsenNZ]

Hi @rajarshp,

I applied this in our custom listener class but it seems this annotation is not taking in to effect, the service is still in pending state. do we need to apply any other annotation to make it work or expose it externally ?

Are you able to share the Service manifests created by the Listener Operator for both the working one (I believe that was the EKS one), and the not working one (Rosa)?

This seems a bug because when we are applying it manually to the listener service yaml (from Rosa service console) it is able to create it and assign LB urls.

Would you be able to add another annotation to the ListenerClass.spec.serviceAnnotations to see if and when that appears in the service?

[rajarshp]

Hi @NickLarsenNZ
sorry I cannot share those files, but here is the small snippet of them. I did add another annotation and deployed in Rosa but it seems none of them are applied

listener class:

spec:
  serviceType: LoadBalancer
  serviceAnnotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    service.beta.kubernetes.io/aws-load-balancer-type: internal

Listener Service yaml post deployment:

kind: Service
apiVersion: v1
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: red-hat-managed=true
  resourceVersion: '212799487'
  name: listener-test-hdfs-namenode-default-0

[sbernauer]

Hi @rajarshp many thanks for the detailed report and the patience!
I did give it a try and you are absolutely right! We straight up forgot to propagate the annotations, sorry for that!
I created stackabletech/listener-operator#234, which should solve the problem. After the merge it will be in the 0.0.0-dev version and will be included in the 24.11 release

[sbernauer]

@rajarshp the fix has been merged and is available in the 0.0.0-dev version. You can switch to that version using stackablectl op un listener and stackablectl op in listener.
Happy about feedback if that solves your problems!

[rajarshp]

Thanks @sbernauer for the update. We cannot use this dev version on our env, so we will check after the 24.11 is released

[rajarshp]

Hi @sbernauer, @NickLarsenNZ
Could you please let me know what annotation I need to add in order to expose this LB as NLB ?
it seems when I am exposing them as LB with the above annotations it is exposing it as classic LB

[NickLarsenNZ]

Hi @rajarshp, if you are using the latest AWS Load Balancer Controller, I believe it should be deploying an NLB instead of a classic load balancer.

Here is a link to the available annotations: https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/annotations/#annotations

If you are using the latest AWS Load Balancer Controller (2.10), I would suggest opening an issue with them.

[rajarshp]

Thanks will check this.
But another problem we are seeing nd trying to find out why this is happening.

We followed this https://docs.stackable.tech/home/stable/hdfs/getting_started/first_steps and put a test file in HDFS which is configured in Rosa and exposed via a LB. We can put a file using these steps but not able to do it the same using the utility page

We created a Route which pointing to the namenode listener service. If we access this we can utility page and see an option to upload/view file.

if I am trying to upload a file form this page it is giving me an error - Couldn't upload the file

not sure why this error is coming and how to fix it

[rajarshp]

@NickLarsenNZ , @sbernauer any idea on this issue ?

[NickLarsenNZ]

Hi @rajarshp, I'm unsure what could be causing the issue.

Do you see anything suspicious in the HDFS logs? Or you notice a difference on your EKS cluster?

[rajarshp]

Hi @NickLarsenNZ , @sbernauer
Sorry for the late response. We are using it in Rosa for now. But I am not from where I can get the exact log. We have installed Stackable operator, but cannot see any error in the HDFS namenode pod log, neither I can see any potential error in the datanode log.

so not sure where exactly I can see the logs for this ?

[rajarshp]

also, we are using Stackable operator for this, so is there a way that we can change the rpc port, we want to use 9000 instead of 8020 ?

and if we dont want to use this operator and deploy it using helm chart, which HDFS image we can use if you can share the link for the same ?

[sbernauer]

Sadly you can not change the port being used, sorry!
One image you could use is docker.stackable.tech/stackable/hadoop:3.4.0-stackable24.7.0 or docker.stackable.tech/stackable/hadoop:3.4.0-stackable0.0.0-dev for the development version (more recent but not stable)

[rajarshp]

Thanks @sbernauer for the details.

[rajarshp]

@NickLarsenNZ , @sbernauer
Could you please let me know for these above issues ?