inputs.cisco_telemetry_mdt giving panic: runtime error: index out of range [0] with length 0 when using with TCP protocol #13789
Comments
|
@Rajat0312 are you sure you provided whole I don't see |
|
[[outputs.prometheus_client]] yeah . It was also there and logs I have provided is whole |
|
I'm not sure if provided
Can you run |
|
/ # cat /etc/telegraf/telegraf.conf [[inputs.cisco_telemetry_mdt]] [[outputs.file]] [[outputs.kafka]] ssl_ca = "ca.pem"ssl_key = "elvarx1.key"[[outputs.prometheus_client]] Path to publish the metrics on, defaults to /metricspath = "/metrics" Expiration interval for each metric. 0 == no expirationexpiration_interval = "60s" Send string metrics as Prometheus labels.Unless set to false all string metrics will be sent as labels.string_as_label = true |
|
Just theoretically (don't have environment to reproduce, and it is hard to say without full stack trace), there can be a problem here: Or here: Or here: @powersj What do you think? |
|
Hi, @zak-pawel thanks for taking a look!
@Rajat0312 - whenever there is a panic there are additional log messages below it that say where the panic occurred. I would prefer not to guess and see what that message actually said please. It should look something like: |
|
Hi @powersj , I am running telegraf via kubernetes and it is the last log before pod gets terminated. Is there any workaround by which I can get these logs? |
telegraf-tiger
bot
removed
the
waiting for response
Based on your logs it happens right away, which is good. You can log to a file maybe and see if it collects it In any case I really want to see the full trace please. |
|
I am trying to get whole logs till can you suggest any workaround to run this new image 1.27.3 |
telegraf-tiger
bot
removed
the
waiting for response
|
@powersj @zak-pawel , I have tried to get full stacktrace but its only showing logs till panic runtime error , I have also tried to get logs in a file but I am getting below error in it , please look into this part how I can use this new image for tcp. 2023-08-21T06:01:01Z I! Using config file: /etc/telegraf/telegraf.conf Config= |
telegraf-tiger
bot
removed
the
waiting for response
This is not a valid configuration option. Remove |
|
@powersj I have also tried without loglevel but no logfile is created. Then I tried to change in source code. I have just added a if condition in this for loop and it’s working now but I need to confirm that is it right way ? Any other functionality will work properly? |
telegraf-tiger
bot
removed
the
waiting for response
Does this path exist? If you have the ability to build and run a custom telegraf, then do you have the ability to jump into one of these containers and run telegraf by hand? This should not require building a custom version of telegraf. All we are trying to do is get the complete log message.
You are guessing as to the cause. It might be the case, but without the full trace we cannot be certain and I would rather fix this with certainty and confidence, rather than play guess and check with you. |
|
So, I used different path too like logfile = "/etc/telegraf/telegraf.log" but files are not being created. I also want to check full stack trace. Yah ! I can access telegraf in containers but I can’t do changes there in code. |
telegraf-tiger
bot
removed
the
waiting for response
|
I am not asking you to do changes to the code. I am asking you to run telegraf by hand: telegraf --debug --config <your config>Can you reproduce the issue that way? |
|
/ # telegraf --debug --config /etc/telegraf/telegraf.conf Inside the kubernetes pod can not able to reproduce that way |
telegraf-tiger
bot
removed
the
waiting for response
|
2023-08-21T14:12:24Z E! Unable to open /etc/telegraf/databus/telegraf.log (open /etc/telegraf/databus/telegraf.log: read-only file system), using stderr / # tail -f tmp/telegraf.log |
|
Take a look at your error messages. In the first case:
Probably because you had telegraf running already?
That is usually out of memory error in a pod. I have put up #13813 with your suggested fix. I would much rather see an actual trace than do this, but if you could give that a try. |
|
Thanks @powersj , i will try to get this trace |
telegraf-tiger
bot
removed
the
waiting for response
|
Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Forums or provide additional details in this issue and reqeust that it be re-opened. Thank you! |
Relevant telegraf.conf
Logs from Telegraf
System info
Telegraf 1.27.3 alpine 3.18.2
Docker
FROM alpine:3.18
RUN echo 'hosts: files dns' >> /etc/nsswitch.conf
RUN apk add --no-cache iputils ca-certificates net-snmp-tools procps lm_sensors tzdata su-exec libcap &&
update-ca-certificates
ENV TELEGRAF_VERSION 1.27.3
RUN ARCH= &&
case "$(apk --print-arch)" in
x86_64) ARCH='amd64';;
aarch64) ARCH='arm64';;
) echo "Unsupported architecture: $(apk --print-arch)"; exit 1;;
esac &&
set -ex &&
mkdir ~/.gnupg;
echo "disable-ipv6" >> ~/.gnupg/dirmngr.conf;
apk add --no-cache --virtual .build-deps wget gnupg tar &&
for key in
9D539D90D3328DC7D6C8D3B9D8FF8E1F7DF8B07E ;
do
gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys "$key" ;
done &&
wget --no-verbose https://dl.influxdata.com/telegraf/releases/telegraf-${TELEGRAF_VERSION}_linux_${ARCH}.tar.gz.asc &&
wget --no-verbose https://dl.influxdata.com/telegraf/releases/telegraf-${TELEGRAF_VERSION}_linux_${ARCH}.tar.gz &&
gpg --batch --verify telegraf-${TELEGRAF_VERSION}linux${ARCH}.tar.gz.asc telegraf-${TELEGRAF_VERSION}linux${ARCH}.tar.gz &&
mkdir -p /usr/src /etc/telegraf &&
tar -C /usr/src -xzf telegraf-${TELEGRAF_VERSION}linux${ARCH}.tar.gz &&
mv /usr/src/telegraf/etc/telegraf/telegraf.conf /etc/telegraf/ &&
mkdir /etc/telegraf/telegraf.d &&
cp -a /usr/src/telegraf*/usr/bin/telegraf /usr/bin/ &&
gpgconf --kill all &&
rm -rf .tar.gz /usr/src /root/.gnupg &&
apk del .build-deps &&
addgroup -S telegraf &&
adduser -S telegraf -G telegraf &&
chown -R telegraf:telegraf /etc/telegraf
EXPOSE 8125/udp 8092/udp 8094
COPY entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
CMD ["telegraf"]
Steps to reproduce
1.start telegraf with tcp and grpc both cisco mdt plugin
2.input data from source -{"fields":{"threshold":346,"speed":327},"name":"cisco","tags":{"host":"telegraf","name":"1","path":"cisco-path","source":"router-1","subscription":"1"},"timestamp":1631191178}
3.it was working in telegraf 1.23.3
...
Expected behavior
It should be running with no error . Right now it gets down
Actual behavior
It gets down with error panic: runtime error: index out of range [0] with length 0
Additional info
It is running in telegraf 1.23.3 but not in latest versions
The text was updated successfully, but these errors were encountered: