You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Tests - specifically, RDS-related tests - fail due to Spotipy requiring an auth code for fetching data.
However, tests pass after the load_history DAG is ran at least once.
This is a strange failure because Spotipy already has my app's client ID and secret. However, it seems to use an auth code to retrieve the data instead of the app's ID and secret.
Data is properly fetched after the load_history DAG runs because it caches - what I assume is - the last used auth code of one of my users. We will have to dig into this...
Failure logs
d96963131133
*** Found local files:
*** * /opt/airflow/logs/dag_id=test_rds_load_artist/run_id=manual__2023-07-11T17:47:15.820691+00:00/task_id=fetch_artists_data/attempt=1.log
[2023-07-11, 17:47:20 UTC] {taskinstance.py:1103} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: test_rds_load_artist.fetch_artists_data manual__2023-07-11T17:47:15.820691+00:00 [queued]>
[2023-07-11, 17:47:20 UTC] {taskinstance.py:1103} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: test_rds_load_artist.fetch_artists_data manual__2023-07-11T17:47:15.820691+00:00 [queued]>
[2023-07-11, 17:47:20 UTC] {taskinstance.py:1308} INFO - Starting attempt 1 of 1
[2023-07-11, 17:47:20 UTC] {taskinstance.py:1327} INFO - Executing <Task(PythonOperator): fetch_artists_data> on 2023-07-11 17:47:15.820691+00:00
[2023-07-11, 17:47:20 UTC] {standard_task_runner.py:57} INFO - Started process 1107 to run task
[2023-07-11, 17:47:20 UTC] {standard_task_runner.py:84} INFO - Running: ['***', 'tasks', 'run', 'test_rds_load_artist', 'fetch_artists_data', 'manual__2023-07-11T17:47:15.820691+00:00', '--job-id', '29', '--raw', '--subdir', 'DAGS_FOLDER/tests/test_rds_load_artist.py', '--cfg-path', '/tmp/tmpvhdgn0mg']
[2023-07-11, 17:47:20 UTC] {standard_task_runner.py:85} INFO - Job 29: Subtask fetch_artists_data
[2023-07-11, 17:47:20 UTC] {task_command.py:410} INFO - Running <TaskInstance: test_rds_load_artist.fetch_artists_data manual__2023-07-11T17:47:15.820691+00:00 [running]> on host d96963131133
[2023-07-11, 17:47:20 UTC] {taskinstance.py:1547} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='***' AIRFLOW_CTX_DAG_ID='test_rds_load_artist' AIRFLOW_CTX_TASK_ID='fetch_artists_data' AIRFLOW_CTX_EXECUTION_DATE='2023-07-11T17:47:15.820691+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2023-07-11T17:47:15.820691+00:00'
[2023-07-11, 17:47:20 UTC] {artist_utils.py:98} INFO - Fetching artists data for 2 artists.
[2023-07-11, 17:47:20 UTC] {artist_utils.py:110} INFO - Fetching chunk 1/1
[2023-07-11, 17:47:20 UTC] {oauth2.py:471} INFO - User authentication requires interaction with your web browser. Once you enter your credentials and give authorization, you will be redirected to a url. Paste that url you were directed to to complete the authorization.
[2023-07-11, 17:47:20 UTC] {logging_mixin.py:149} INFO - Go to the following URL: https://accounts.spotify.com/authorize?client_id=a7e0021432e845fc8e22b0c4e404c545&response_type=code&redirect_uri=http%3A%2F%2Flocalhost%3A5000%2Fspotify%2Fauthorize
Enter the URL you were redirected to:
[2023-07-11, 17:47:20 UTC] {taskinstance.py:1824} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/spotipy/oauth2.py", line 115, in _get_user_input
return raw_input(prompt)
NameError: name 'raw_input' is not defined
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 181, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 198, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/opt/airflow/dags/src/utils/artist_utils.py", line 111, in fetch_artists_data_with_dates
artists = fetch_artists([artist_id for artist_id, _ in chunk])["artists"]
File "/opt/airflow/dags/src/utils/spotify_utils.py", line 222, in fetch_artists
artists_data = sp.artists(artist_ids)
File "/home/airflow/.local/lib/python3.7/site-packages/spotipy/client.py", line 402, in artists
return self._get("artists/?ids=" + ",".join(tlist))
File "/home/airflow/.local/lib/python3.7/site-packages/spotipy/client.py", line 323, in _get
return self._internal_call("GET", url, payload, kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/spotipy/client.py", line 247, in _internal_call
headers = self._auth_headers()
File "/home/airflow/.local/lib/python3.7/site-packages/spotipy/client.py", line 238, in _auth_headers
token = self.auth_manager.get_access_token(as_dict=False)
File "/home/airflow/.local/lib/python3.7/site-packages/spotipy/oauth2.py", line 535, in get_access_token
"code": code or self.get_auth_response(),
File "/home/airflow/.local/lib/python3.7/site-packages/spotipy/oauth2.py", line 499, in get_auth_response
return self._get_auth_response_interactive(open_browser=open_browser)
File "/home/airflow/.local/lib/python3.7/site-packages/spotipy/oauth2.py", line 450, in _get_auth_response_interactive
response = self._get_user_input(prompt)
File "/home/airflow/.local/lib/python3.7/site-packages/spotipy/oauth2.py", line 117, in _get_user_input
return input(prompt)
EOFError: EOF when reading a line
[2023-07-11, 17:47:20 UTC] {taskinstance.py:1350} INFO - Marking task as FAILED. dag_id=test_rds_load_artist, task_id=fetch_artists_data, execution_date=20230711T174715, start_date=20230711T174720, end_date=20230711T174720
[2023-07-11, 17:47:20 UTC] {standard_task_runner.py:109} ERROR - Failed to execute job 29 for task fetch_artists_data (EOF when reading a line; 1107)
[2023-07-11, 17:47:20 UTC] {local_task_job_runner.py:225} INFO - Task exited with return code 1
[2023-07-11, 17:47:20 UTC] {taskinstance.py:2651} INFO - 0 downstream tasks scheduled from follow-on schedule check
The text was updated successfully, but these errors were encountered:
Description
Tests - specifically, RDS-related tests - fail due to
Spotipy
requiring an auth code for fetching data.However, tests pass after the
load_history
DAG is ran at least once.This is a strange failure because Spotipy already has my app's client ID and secret. However, it seems to use an auth code to retrieve the data instead of the app's ID and secret.
Data is properly fetched after the
load_history
DAG runs because it caches - what I assume is - the last used auth code of one of my users. We will have to dig into this...Failure logs
The text was updated successfully, but these errors were encountered: