Support long running queries in timestream

[sarahzinger]

Discussed in https://github.com/grafana/timestream-datasource/discussions/248

^{Originally posted by dleonard-nasuni October 6, 2023}
I am using the timestream-datasource, and have a query that takes a particularly large amount of time. I occasionally receive a 504 server error when I hover over the panel that has the query that is timing out, and each instance of the timeout is right around the minute mark. The logs in /var/log/grafana/grafana.log seem to indicate that this timeout is on the timestream-datasource plugin side due to cancelled context. Is there a way to configure this timeout?

Here are the /var/log/grafana/grafana.log logs:

logger=context userId=4 orgId=1 uname=<redacted> t=2023-10-06T20:05:45.450493766Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Canceled desc = context canceled" remote_addr=<redacted> traceID=logger=context userId=4 orgId=1 uname=<redacted> t=2023-10-06T20:05:45.450601507Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=<redacted> time_ms=60003 duration=1m0.003610261s size=116 referer="https://<redacted>" handler=/api/ds/query

logger=context userId=4 orgId=1 uname=<redacted> t=2023-10-06T20:05:45.450601507Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=<redacted> time_ms=60003 duration=1m0.003610261s size=116 referer="https://<redacted>" handler=/api/ds/query
```</div>

[sarahzinger]

@dleonard-nasuni just a heads up we're moving from discussions to issues so we can more easily track user feedback, so I'm converting your discussion into an issue. I'm curious are you still having trouble with timestream?

[dleonard-nasuni]

Hi @sarahzinger , thank you for opening this as an issue. I am still experiencing timeouts around the 1 minute mark for some panels that are data-intensive. I did try setting timeout in timestream.yaml in the jsonData section of my datasource configuration (after trying to figure out where timeouts are picked up in code), but timeouts still occurred after 1 minute after setting this field to 30 seconds.

[iwysiu]

I took a look at this and this sounds like a long running query issue. This has been handled in some of the other datasources (cloudwatch logs, athena, and redshift) by having the frontend query the backend periodically for updates, which we can and probably should do here.

Implementing that would be a decently large feature/refactor, so I'm going to move this into our backlog for now and our team will prioritize this against other work.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last year. It will be closed in 30 days if no further activity occurs. Please feel free to leave a comment if you believe the issue is still relevant. Thank you for your contributions!

[dleonard-nasuni]

Commenting to avoid the issue being closed