Skip to content

Commit

Permalink
Made sure image loading from ComicVine also obeys Comic Vines new speed
Browse files Browse the repository at this point in the history
limit.   In fact, adjust the speed limit so that the app doesn't access
Comic Vine more than once every 2 seconds.
  • Loading branch information
cbanack committed Nov 20, 2015
1 parent a12b536 commit 95679be
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 3 deletions.
6 changes: 3 additions & 3 deletions src/py/database/comicvine/cvconnection.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
__next_query_time_ms = 0

# the amount of time to wait between queries
__QUERY_DELAY_MS = 1500
__QUERY_DELAY_MS = 2000

# =============================================================================
def _query_series_ids_dom(API_KEY, searchterm_s, page_n=1):
Expand Down Expand Up @@ -223,7 +223,7 @@ def __get_page(url):
may be thrown. If the exception is a DatabaseConnectionError, that
represents an problem connecting to the Comic Vine database.
'''
__wait_until_ready() # throttle to keep use from going too fast
wait_until_ready() # throttle request speed to make ComicVine happy

try:
return utils.get_html_string(url)
Expand Down Expand Up @@ -255,7 +255,7 @@ def is_valid_xml(c):
return xml

# =============================================================================
def __wait_until_ready():
def wait_until_ready():
'''
Waits until a fixed amount of time has passed since this function was
last called. Returns immediately if that much time has already passed.
Expand Down
1 change: 1 addition & 0 deletions src/py/database/comicvine/cvdb.py
Original file line number Diff line number Diff line change
Expand Up @@ -424,6 +424,7 @@ def _query_image( ref, lasttry = False ):
response = None
response_stream = None
try:
cvconnection.wait_until_ready() # throttle our request speed
request = WebRequest.Create(image_url_s)
response = request.GetResponse()
response_stream = response.GetResponseStream()
Expand Down

2 comments on commit 95679be

@RogueBurger
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If anyone's looking for a number that works as of today, based on my testing 4000 ms works for me, 3000 ms does not. I have not been able to test any more precise increments yet due to being locked out of my account now from all the testing.

Hopefully they fix their system to actually only need a 1000 ms delay, but until then I hope this helps anyone who wants to get their scrape on.

@cbanack
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran a test for one of the Comic Vine developers related to this yesterday. As far as I know, they are aware of the problem and working on it.

Please sign in to comment.