Events are indexed with wrong timezone

[ChristophHerd]

Hi!

I have timezone issues with splunk-otel-collector-0.62.1 chart .
My Kubernetes host has CET (UTC+1) set, and it seems that otel collector is sending the CET timestamp as UTC to splunk because to find my data in splunk enterprise I need to search it 1 hour in future (UTC+2).

I´m using fluentd log collection.

Is there any possibility to configure otel collector to use the host timezone or an way to override the otel collector timezone?

Thank you!
br,
Christoph

[ChristophHerd]

After some more digging into this issue I´m pretty sure that only container logs are affected (sourcetype=kube:container:*).

In the yaml template configmap-fluentd.yaml there is "localtime false" set for containers.log on line 79 - this seems to me related to my issue, can someone please confirm this or give some advice on how to deal with this?

[ChristophHerd]

Setting the UI timezone helps for the container logs - unfortunately everyting else is then incorrect. I.e. the Metrics exported by the same otel collector are one hour in the past then...
Changing the UI timezone because of one "wrong" ingested sourcetype does not seem to be an proper solution for me.

[matthewmodestino]

Hmm, yeah that's not right. I would do whatever works, but I guess I'm not clear on what the root cause of your timezone discrepancy actually is. Is it just the server setting the timezone? Or your splunk server interpreting on index?

I can try and replicate if you shoot me some replication setup steps.

What kind of k8s host?

Can you show a sample log by navigating to the file path on the k8s node and retrieving a raw docker or cri-o log line or two?

[ChristophHerd]

I´m unfortunately not 100% sure where exactly the issue is located - for me it seems that the local CET timezone is ignored by fluentd and sent to the Indexer as UTC.

Initially this was identified in an Kubernetes Environment - I was easily able to reproduce it on an test server with microk8s (on an Alma Linux 8.6 host).

For the installation I followed the helm install guide on github.
Attached you can find the yaml that I used.
System timezone is CET.

This are some logs of the local metallb container (they have the same time as the server - CET), in splunk I see them with time 16:57:xx (one hour in the future):

2022-12-15T15:57:00.902100295+01:00 stdout F {"caller":"service_controller.go:60","controller":"ServiceReconciler","level":"info","start reconcile":"kube-system/microk8s.io-hostpath","ts":"2022-12-15T14:57:00Z"}
2022-12-15T15:57:00.902129943+01:00 stdout F {"caller":"service_controller.go:103","controller":"ServiceReconciler","end reconcile":"kube-system/microk8s.io-hostpath","level":"info","ts":"2022-12-15T14:57:00Z"}
2022-12-15T15:57:01.969094446+01:00 stdout F {"caller":"service_controller.go:60","controller":"ServiceReconciler","level":"info","start reconcile":"openebs/openebs.io-local","ts":"2022-12-15T14:57:01Z"}
2022-12-15T15:57:01.969119859+01:00 stdout F {"caller":"service_controller.go:103","controller":"ServiceReconciler","end reconcile":"openebs/openebs.io-local","level":"info","ts":"2022-12-15T14:57:01Z"}
2022-12-15T15:57:02.913121067+01:00 stdout F {"caller":"service_controller.go:60","controller":"ServiceReconciler","level":"info","start reconcile":"kube-system/microk8s.io-hostpath","ts":"2022-12-15T14:57:02Z"}
2022-12-15T15:57:02.913151317+01:00 stdout F {"caller":"service_controller.go:103","controller":"ServiceReconciler","end reconcile":"kube-system/microk8s.io-hostpath","level":"info","ts":"2022-12-15T14:57:02Z"}
2022-12-15T15:57:03.980537339+01:00 stdout F {"caller":"service_controller.go:60","controller":"ServiceReconciler","level":"info","start reconcile":"openebs/openebs.io-local","ts":"2022-12-15T14:57:03Z"}
2022-12-15T15:57:03.980638869+01:00 stdout F {"caller":"service_controller.go:103","controller":"ServiceReconciler","end reconcile":"openebs/openebs.io-local","level":"info","ts":"2022-12-15T14:57:03Z"}
2022-12-15T15:57:04.937424665+01:00 stdout F {"caller":"service_controller.go:60","controller":"ServiceReconciler","level":"info","start reconcile":"kube-system/microk8s.io-hostpath","ts":"2022-12-15T14:57:04Z"}
2022-12-15T15:57:04.93745389+01:00 stdout F {"caller":"service_controller.go:103","controller":"ServiceReconciler","end reconcile":"kube-system/microk8s.io-hostpath","level":"info","ts":"2022-12-15T14:57:04Z"}

I used an single node splunk installation on another testserver for my tests, timezone on the server is also CET.

Thanks for your help and please let me know in case you need more details.

otel_collector_values.yaml.txt

[matthewmodestino]

Yeah i think I see it, the containerd timestamp has the local time, but the app log has Zulu, aka UTC.

Will review closer, i believe you may be able to solve this a couple ways. Need to test, back soon.

[atoulme]

You can manipulate timestamps with https://github.com/signalfx/splunk-otel-collector/tree/main/pkg/processor/timestamp.