Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for user to reset token mid-crawl #10

Open
tskluzac opened this issue Jan 15, 2020 · 1 comment
Open

Allow for user to reset token mid-crawl #10

tskluzac opened this issue Jan 15, 2020 · 1 comment
Assignees

Comments

@tskluzac
Copy link
Collaborator

See error:
2020-01-15 20:13:25,231 - Caught error : (401, 'AuthenticationFailed', 'Token is not active', 'E4MtgqmLj')

Tens of thousands of files into a crawl, we have to throw it away and start over because we happened to launch the crawl AFTER the token expired. What we should do is either

(A) Find a way to implement refresh tokens? (not sure if helpful here--need to follow up).
(B) Allow user to re-submit their credential.

@tskluzac tskluzac self-assigned this Jan 15, 2020
@tskluzac
Copy link
Collaborator Author

This will be best accomplished by catching the API Auth errors, saving all current state to S3, and then creating a 'resume' mode on the crawl id (with new 'headers')

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant