constant scanning of same directory #2

JohnFawkes · 2023-03-11T06:04:41Z

So I opened this issue Cloudbox/autoscan#202 up on autoscan thinking it was autoscan, however disabling autoscan-adaptor in my config gets rid of the internal 500 error and fixes the issue. I believe it has something to do with spaces in the url and the update that was pushed a few days ago as thats when my issue started

Request failed request_url=https://<redacted>/triggers/manual?dir=%2Fhome%2Fjohnfawkes%2FMergerFS%2Fmovies-4k%2FThe+Inspection+%282022%29 response_status=500

the directory here is supposed to be /home/johnfawkes/MergerFS/The Inspection (2022) This is using TRASH naming scheme. Went back to version 0.0.4 using the tag in docker for now until solved. Thanks for making adapter. Loving it for bazarr and to trigger intro detection when autoscan scans items in plex

The text was updated successfully, but these errors were encountered:

drgnfyre · 2023-03-16T23:05:40Z

I am running into a similar problem with the return code of 500. For me, it's for any directory that a-train picks up but autoscan is unfamiliar with.
Normally it says something like unknown directory, and moves on to the next object.

However, when pointed to autoscan-adapter, it gets hung in an endless loop of "not all targets are available, retrying in 15 seconds" and then "500 internal server error"

Here is what autoscan is reporting

Mar 14 12:48:06 INF Scan stats processed=13 remaining=2
Mar 14 12:48:33 WRN No target libraries found error="/dbbackup/2022: failed determining libraries" target=plex url=http://plex:32400
Mar 14 12:48:33 ERR Not all targets are available, retrying in 15 seconds... error="scan: 500 INTERNAL SERVER ERROR: target unavailable"

That last line repeats every 15 seconds. If another item comes down, autoscan never gets to it because adapter keeps giving it 500 errors

Mar 14 13:02:02 INF Scan moved to processor event=Download id= method=POST path=/mnt/unionfs/tv/Last.of.Us,.The.(2023)_{tvdb-392256}/Season.01 url=/triggers/sonarr
Mar 14 13:18:06 INF Scan stats processed=13 remaining=5

It will then go into the previous loop of retrying / error 500.

Here is what autoscan-adapter is showing

03/15/2023 4:23:06 PM
[2023-03-15 16:23:06,119] WARNING in app: Starting directory scan of: /plex-backup/2022
03/15/2023 4:23:06 PM
[2023-03-15 16:23:06,127] ERROR in app: Exception on /triggers/manual [POST]
03/15/2023 4:23:06 PM
Traceback (most recent call last):
03/15/2023 4:23:06 PM
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 2525, in wsgi_app
03/15/2023 4:23:06 PM
    response = self.full_dispatch_request()
03/15/2023 4:23:06 PM
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
03/15/2023 4:23:06 PM
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1822, in full_dispatch_request
03/15/2023 4:23:06 PM
    rv = self.handle_user_exception(e)
03/15/2023 4:23:06 PM
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
03/15/2023 4:23:06 PM
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1820, in full_dispatch_request
03/15/2023 4:23:06 PM
    rv = self.dispatch_request()
03/15/2023 4:23:06 PM
         ^^^^^^^^^^^^^^^^^^^^^^^
03/15/2023 4:23:06 PM
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1796, in dispatch_request
03/15/2023 4:23:06 PM
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
03/15/2023 4:23:06 PM
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
03/15/2023 4:23:06 PM
  File "/app/app.py", line 31, in trigger
03/15/2023 4:23:06 PM
    files_refreshed = plex_api.refresh_metadata(metadata_files)
03/15/2023 4:23:06 PM
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
03/15/2023 4:23:06 PM
  File "/app/plexapihandler.py", line 72, in refresh_metadata
03/15/2023 4:23:06 PM
    for element in metadata_files:
03/15/2023 4:23:06 PM
TypeError: 'NoneType' object is not iterable
03/15/2023 4:23:06 PM
172.18.0.8 - - [15/Mar/2023 16:23:06] "POST /triggers/manual?dir=%2Fplex-backup%2F2022 HTTP/1.1" 500 -
03/15/2023 4:23:25 PM
172.18.0.8 - - [15/Mar/2023 16:23:25] "HEAD /triggers/manual HTTP/1.1" 200 -

If I go back to 0.0.4, autoscan-adapter reports similar findings, but autoscan just keeps going to the next item.
autoscan-

Mar 16 04:47:23 INF Scan moved to processor id=cg9ebauo0l6sjbeds330 method=POST path=/plex-backup url=/triggers/a-train/0AL_HNZgN1Xu7Uk9PVA
Mar 16 05:05:22 WRN No target libraries found error="/plex-backup: failed determining libraries" target=plex url=http://plex:32400
Mar 16 05:09:35 INF Scan moved to target path=/plex-backup target=autoscan url=http://autoscan-adapter:5000

autoscan-adapter

[2023-03-16 04:20:30,163] WARNING in app: Starting directory scan of: ['/plex-backup']
[2023-03-16 04:20:35,234] ERROR in app: Exception on /triggers/manual [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Try again
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.10/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.10/http/client.py", line 975, in send
    self.connect()
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f012c7901f0>: Failed to establish a new connection: [Errno -3] Try again
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='plex', port=32400): Max retries exceeded with url: /library/sections (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f012c7901f0>: Failed to establish a new connection: [Errno -3] Try again'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2525, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1822, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1820, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1796, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/app/app.py", line 20, in manual_trigger
    metadata_files = plex_api.find_metadata_from_dirs(
  File "/app/plexapihandler.py", line 15, in find_metadata_from_dirs
    sections = self.plex.library.sections()
  File "/usr/local/lib/python3.10/site-packages/plexapi/library.py", line 53, in sections
    self._loadSections()
  File "/usr/local/lib/python3.10/site-packages/plexapi/library.py", line 41, in _loadSections
    for elem in self._server.query(key):
  File "/usr/local/lib/python3.10/site-packages/plexapi/server.py", line 697, in query
    response = method(url, headers=headers, timeout=timeout, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='plex', port=32400): Max retries exceeded with url: /library/sections (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f012c7901f0>: Failed to establish a new connection: [Errno -3] Try again'))
172.18.0.8 - - [16/Mar/2023 04:20:36] "POST /triggers/manual?dir=%2Fplex-backup HTTP/1.1" 500 -

since A-train doesn't currently, and probably won't ever give us exclusion options, it would be nice if autoscan or autoscan-adapter did.

Thanks for writing this - I don't know of anyone else that had come up with a solution for Plex not refreshing the metadata on new files.

JohnFawkes · 2023-03-17T08:26:23Z

so what i did is just roll back to the previous version of 0.0.4 until someone from autoscan-adaptor replies with a fix. if using docker-compose you can do
image: dantebarba/autoscan-adapter:0.0.4
for your image
edit: I see that you did that but get an error still. When I went back to 0.0.4 I had zero issues
edit2: I am also not using a-train. I use sonarr and radarr connections to trigger autoscan

zelik888 · 2023-03-28T15:56:54Z

I'm getting this issue only with Lidarr. When Lidarr connects to autoscan the errors start. If I disconnect lidarr from autoscan then everything is fine. From my rudimentary debugging skills I'm assuming Lidarr is not supported when autoscan-adapter is enabled

dantebarba · 2023-05-25T21:20:28Z

Hi all,

Looks like the refactoring I've made following some recommendations triggered this issue. I didn't have time to debug it but I'll do it eventually. For now just continue using 0.0.4

Also

However, when pointed to autoscan-adapter, it gets hung in an endless loop of "not all targets are available, retrying in 15 seconds" and then "500 internal server error"

That last line repeats every 15 seconds. If another item comes down, autoscan never gets to it because adapter keeps giving it 500 errors

This could be a bug in autoscan, which doesn't have a max retry option when failure occurs.

Null pointer was rasied when scanning for a directory that didn't exist in Plex Fixed by handling the null pointer and returning an OK response (200)

dantebarba · 2023-05-27T00:53:44Z

The issue should be solved in https://github.com/dantebarba/autoscan-adapter/releases/tag/0.1.1. You can use the 0.1.1 tag in docker to download the new image. Let me know if it worked.

I'd recommend using the 0.1.1 because it has performance improvements over 0.0.4

JohnFawkes · 2023-05-27T01:11:25Z

Awesome! Going to test this out now. Thanks!

JohnFawkes · 2023-06-01T19:10:49Z

So far everything seems to working however I am seeing a bunch of this in logs

87.236.176.232 - - [28/May/2023 20:24:16] "GET / HTTP/1.1" 200 -
87.251.75.108 - - [28/May/2023 20:54:34] code 400, message Bad HTTP/0.9 request type ("\x03\x00\x00,'à\x00\x00\x00\x00\x00Cookie:")
87.251.75.108 - - [28/May/2023 20:54:34] "\x03\x00\x00,'à\x00\x00\x00\x00\x00Cookie: mstshash=Domain" 400 -
87.251.75.108 - - [28/May/2023 20:54:36] code 400, message Bad HTTP/0.9 request type ("\x03\x00\x00,'à\x00\x00\x00\x00\x00Cookie:")
87.251.75.108 - - [28/May/2023 20:54:36] "\x03\x00\x00,'à\x00\x00\x00\x00\x00Cookie: mstshash=Domain" 400 -
87.251.75.108 - - [28/May/2023 20:54:38] code 400, message Bad HTTP/0.9 request type ("\x03\x00\x00,'à\x00\x00\x00\x00\x00Cookie:")
87.251.75.108 - - [28/May/2023 20:54:38] "\x03\x00\x00,'à\x00\x00\x00\x00\x00Cookie: mstshash=Domain" 400 -

Not sure where the IP's are coming from but they are different IP's everytime

EDIT: setup portscan blocker and now the IP's are gone. Logs were just showing random bots running port scans on online servers

dantebarba · 2023-06-03T03:20:10Z

Ok closing!

dantebarba added a commit that referenced this issue May 26, 2023

Fixed constant scanning of same directory #2

ab01b41

Null pointer was rasied when scanning for a directory that didn't exist in Plex Fixed by handling the null pointer and returning an OK response (200)

dantebarba self-assigned this May 27, 2023

dantebarba added the bug Something isn't working label May 27, 2023

dantebarba closed this as completed Jun 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

constant scanning of same directory #2

constant scanning of same directory #2

JohnFawkes commented Mar 11, 2023 •

edited

Loading

drgnfyre commented Mar 16, 2023 •

edited

Loading

JohnFawkes commented Mar 17, 2023 •

edited

Loading

zelik888 commented Mar 28, 2023

dantebarba commented May 25, 2023 •

edited

Loading

dantebarba commented May 27, 2023 •

edited

Loading

JohnFawkes commented May 27, 2023

JohnFawkes commented Jun 1, 2023 •

edited

Loading

dantebarba commented Jun 3, 2023

constant scanning of same directory #2

constant scanning of same directory #2

Comments

JohnFawkes commented Mar 11, 2023 • edited Loading

drgnfyre commented Mar 16, 2023 • edited Loading

JohnFawkes commented Mar 17, 2023 • edited Loading

zelik888 commented Mar 28, 2023

dantebarba commented May 25, 2023 • edited Loading

dantebarba commented May 27, 2023 • edited Loading

JohnFawkes commented May 27, 2023

JohnFawkes commented Jun 1, 2023 • edited Loading

dantebarba commented Jun 3, 2023

JohnFawkes commented Mar 11, 2023 •

edited

Loading

drgnfyre commented Mar 16, 2023 •

edited

Loading

JohnFawkes commented Mar 17, 2023 •

edited

Loading

dantebarba commented May 25, 2023 •

edited

Loading

dantebarba commented May 27, 2023 •

edited

Loading

JohnFawkes commented Jun 1, 2023 •

edited

Loading