Upgrading the ELK stack

[4OH4]

Great project, thanks for sharing - and supporting for so long!

I ran into a few problems running the ELK stack - the Elasticsearch container kept restarting with java.lang.IllegalStateException
docker-elk-logs.txt

I couldn't find the root cause for this, but in the end switched to using a later version of the ELK stack - v7.10 - which gave good results, and used Filebeat rather than Logstash as there seemed to be more documentation around this use-case. Not sure if this is a change you wanted to make to the project, but have my files on a branch here - happy to submit a pull request if you think that it might be useful:
https://github.com/4OH4/scrapy-cluster/tree/elk-update

Haven't managed to properly import the Kibana dashboard configuration from export.json though - I guess a few things have changed between the different versions of Kibana.

Cheers

[madisonb]

If you've got Filebeat and the latest ELK stack going (with the json logs parsed correctly into the index) I would 100% accept a PR - it's been on my todo list to move the project over to it.

As a bonus, I would also prefer we switch over all the logging to stdout and pull directly from the container logs as that's the better practice nowadays vs logstash.

[4OH4]

Ok - great. I'll take another look at the parsing of the log files - at the moment I am using an index pattern of filebeat-* so its not being parsed in quite the same way as before, although the JSON key/value pairs are being stored correctly in Elasticsearch.