Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests initially fail due to Spotipy requiring authentication #4

Open
nosas opened this issue Jul 11, 2023 · 0 comments
Open

Tests initially fail due to Spotipy requiring authentication #4

nosas opened this issue Jul 11, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@nosas
Copy link
Member

nosas commented Jul 11, 2023

Description

Tests - specifically, RDS-related tests - fail due to Spotipy requiring an auth code for fetching data.
However, tests pass after the load_history DAG is ran at least once.

This is a strange failure because Spotipy already has my app's client ID and secret. However, it seems to use an auth code to retrieve the data instead of the app's ID and secret.

Data is properly fetched after the load_history DAG runs because it caches - what I assume is - the last used auth code of one of my users. We will have to dig into this...

Failure logs

d96963131133
*** Found local files:
***   * /opt/airflow/logs/dag_id=test_rds_load_artist/run_id=manual__2023-07-11T17:47:15.820691+00:00/task_id=fetch_artists_data/attempt=1.log
[2023-07-11, 17:47:20 UTC] {taskinstance.py:1103} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: test_rds_load_artist.fetch_artists_data manual__2023-07-11T17:47:15.820691+00:00 [queued]>
[2023-07-11, 17:47:20 UTC] {taskinstance.py:1103} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: test_rds_load_artist.fetch_artists_data manual__2023-07-11T17:47:15.820691+00:00 [queued]>
[2023-07-11, 17:47:20 UTC] {taskinstance.py:1308} INFO - Starting attempt 1 of 1
[2023-07-11, 17:47:20 UTC] {taskinstance.py:1327} INFO - Executing <Task(PythonOperator): fetch_artists_data> on 2023-07-11 17:47:15.820691+00:00
[2023-07-11, 17:47:20 UTC] {standard_task_runner.py:57} INFO - Started process 1107 to run task
[2023-07-11, 17:47:20 UTC] {standard_task_runner.py:84} INFO - Running: ['***', 'tasks', 'run', 'test_rds_load_artist', 'fetch_artists_data', 'manual__2023-07-11T17:47:15.820691+00:00', '--job-id', '29', '--raw', '--subdir', 'DAGS_FOLDER/tests/test_rds_load_artist.py', '--cfg-path', '/tmp/tmpvhdgn0mg']
[2023-07-11, 17:47:20 UTC] {standard_task_runner.py:85} INFO - Job 29: Subtask fetch_artists_data
[2023-07-11, 17:47:20 UTC] {task_command.py:410} INFO - Running <TaskInstance: test_rds_load_artist.fetch_artists_data manual__2023-07-11T17:47:15.820691+00:00 [running]> on host d96963131133
[2023-07-11, 17:47:20 UTC] {taskinstance.py:1547} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='***' AIRFLOW_CTX_DAG_ID='test_rds_load_artist' AIRFLOW_CTX_TASK_ID='fetch_artists_data' AIRFLOW_CTX_EXECUTION_DATE='2023-07-11T17:47:15.820691+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2023-07-11T17:47:15.820691+00:00'
[2023-07-11, 17:47:20 UTC] {artist_utils.py:98} INFO - Fetching artists data for 2 artists.
[2023-07-11, 17:47:20 UTC] {artist_utils.py:110} INFO - Fetching chunk 1/1
[2023-07-11, 17:47:20 UTC] {oauth2.py:471} INFO - User authentication requires interaction with your web browser. Once you enter your credentials and give authorization, you will be redirected to a url.  Paste that url you were directed to to complete the authorization.
[2023-07-11, 17:47:20 UTC] {logging_mixin.py:149} INFO - Go to the following URL: https://accounts.spotify.com/authorize?client_id=a7e0021432e845fc8e22b0c4e404c545&response_type=code&redirect_uri=http%3A%2F%2Flocalhost%3A5000%2Fspotify%2Fauthorize
Enter the URL you were redirected to: 
[2023-07-11, 17:47:20 UTC] {taskinstance.py:1824} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/spotipy/oauth2.py", line 115, in _get_user_input
    return raw_input(prompt)
NameError: name 'raw_input' is not defined
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 181, in execute
    return_value = self.execute_callable()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 198, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/opt/airflow/dags/src/utils/artist_utils.py", line 111, in fetch_artists_data_with_dates
    artists = fetch_artists([artist_id for artist_id, _ in chunk])["artists"]
  File "/opt/airflow/dags/src/utils/spotify_utils.py", line 222, in fetch_artists
    artists_data = sp.artists(artist_ids)
  File "/home/airflow/.local/lib/python3.7/site-packages/spotipy/client.py", line 402, in artists
    return self._get("artists/?ids=" + ",".join(tlist))
  File "/home/airflow/.local/lib/python3.7/site-packages/spotipy/client.py", line 323, in _get
    return self._internal_call("GET", url, payload, kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/spotipy/client.py", line 247, in _internal_call
    headers = self._auth_headers()
  File "/home/airflow/.local/lib/python3.7/site-packages/spotipy/client.py", line 238, in _auth_headers
    token = self.auth_manager.get_access_token(as_dict=False)
  File "/home/airflow/.local/lib/python3.7/site-packages/spotipy/oauth2.py", line 535, in get_access_token
    "code": code or self.get_auth_response(),
  File "/home/airflow/.local/lib/python3.7/site-packages/spotipy/oauth2.py", line 499, in get_auth_response
    return self._get_auth_response_interactive(open_browser=open_browser)
  File "/home/airflow/.local/lib/python3.7/site-packages/spotipy/oauth2.py", line 450, in _get_auth_response_interactive
    response = self._get_user_input(prompt)
  File "/home/airflow/.local/lib/python3.7/site-packages/spotipy/oauth2.py", line 117, in _get_user_input
    return input(prompt)
EOFError: EOF when reading a line
[2023-07-11, 17:47:20 UTC] {taskinstance.py:1350} INFO - Marking task as FAILED. dag_id=test_rds_load_artist, task_id=fetch_artists_data, execution_date=20230711T174715, start_date=20230711T174720, end_date=20230711T174720
[2023-07-11, 17:47:20 UTC] {standard_task_runner.py:109} ERROR - Failed to execute job 29 for task fetch_artists_data (EOF when reading a line; 1107)
[2023-07-11, 17:47:20 UTC] {local_task_job_runner.py:225} INFO - Task exited with return code 1
[2023-07-11, 17:47:20 UTC] {taskinstance.py:2651} INFO - 0 downstream tasks scheduled from follow-on schedule check
@nosas nosas added the bug Something isn't working label Jul 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant