Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Process is not killing itself on fatal error (docker) #690

Open
ozbarshalom opened this issue Mar 30, 2017 · 5 comments
Open

Process is not killing itself on fatal error (docker) #690

ozbarshalom opened this issue Mar 30, 2017 · 5 comments

Comments

@ozbarshalom
Copy link

ozbarshalom commented Mar 30, 2017

I am using docker (via OpenShift) with Mongo Connector and elastic2-doc-manager in their latest versions.
Everything is ok until I am getting for any reason ReadTimeoutError because one of the requests to the Elastic took more than 10sec.
In this case, the mongo-connector should send a SIGNAL to kill itself, It will help the Docker restart automatically (recover) and try to continue from the last oplog entry.

In my case, I noticed that the connector was "stuck" in this error for about 20 hours, And when I restarted it manually the connector failed to trace in time the changes happened in the MongoDB (there are about 20,000 updated per hour).

@ShaneHarvey
Copy link
Contributor

ShaneHarvey commented Mar 30, 2017

Can you post the trackback you're seeing in the logs? Also what version of mongo-connector and elastic-doc-manager are you running?

Seems related to yougov/elastic2-doc-manager#18

@ozbarshalom
Copy link
Author

I am using elastic2-doc-manager version 0.3.0 and mongo-connector`s version from the latest commit on master branch.
Looks like a missing exception in the wrap_exceptions for ConnectionTimeout.
I just created a related issue mongodb-labs/elastic2-doc-manager#44 and opened a pull request.

Should handle this.

@ShaneHarvey
Copy link
Contributor

Mapping the error to OperationFailed would cause mongo-connector to ignore the error and not exit. The real issue is that the elastic doc manager does not handle timeouts on its Elasticsearch client. That error should currently be uncaught and mongo-connector should exit. It should not be "stuck". Can you post the ReadTimeoutError error message or stacktrace that you're seeing?

@ozbarshalom
Copy link
Author

I can`t export the traceback but, The error is: ReadTimeoutError: HTTPConnectionPool(host='hostname', port=9200): Read Time out. (read timeout=10)

Maybe it is not exit because I used --continue-on-error ? But it even did not continue... just got stuck

@nzapponi
Copy link

Have there been any updates on this? I'm still seeing this issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants