Make opensearch traces limit settable

[zoesyc]

Investigate the opensearch span limits and why traces are currently capped at a limit of 'the first 1000 spans' in the trace

Use Case: Teams can monitor complex systems which often result in traces that include over 1000 spans. What is the reason for the current cap?

[idastambuk]

So as far as I can remember, there wasn't a particular reason, apart from that this is the limit in OpenSearch dashboards as well, and it cannot be overridden there either.
However, I think it makes sense for us to give the user the ability to override this limit of 1000, by providing an input for Span limit. Tempo datasource also has this.
They also have a separate input for number of traces returned. We could also provide the user that as well, though I'm not sure if there is need for this currently (it makes sense to me that there is greater need to not limit spans within a trace).
Adding this to backlog.

[DWebb0]

I'm registering my interest here. cc @annelaurefroment

[iwysiu]

Investigation results: Based on both elasticsearch and opensearch docs, it seems like the actual limit is 10,000. From what I could tell, there isn't really any reason for our 1000 limit besides performance.
We can make it a configurable field and then just use the current 1000 as the default, I'll update the issue to reflect that.