Skip to content

a-coding-Kat/rest-api

BranchesTags

Repository files navigation

RESTful API, WDB Module Mini-Challenge

The goal of this repository is to implement a RESTful API that offers CRUD access to Spotify tracks obtained from Kaggle. It also allows for filtering and pagination of songs by any attribute in the database.

To interact with this server, you can use your own HTTP client, or use the provided Python client.

The tracks are accessible in the data/ folder in the form of a CSV file. By running import_data.py from the root of this project you can generate a sqlite3 database file that is later used by the Flask server.

Getting started

This project requires Python 3.6 or higher.

Clone repository:

git clone https://github.com/a-coding-Kat/rest-api.git
cd rest-api

Running the server in Docker

To build and run the container execute from the root of this project:

sh build_docker.sh
sh run_docker.sh

Alternatively, if you do not have bash, you can execute the build and run command manually:

docker build -t wdb_rest .
docker run -it -p 5000:5000 wdb_rest

The web service will be accessible on the URL http://127.0.0.1:5000.

Running the server manually

Create environment:

python -m venv wdb
# On Windows: .\wdb\Scripts\activate
source wdb/bin/activate
pip install -r requirements.txt

To create the database, execute the import_data.py from the root of the project:

python import_data.py

Start the REST server from the root of the project.

python wdb_rest/server.py

Running the tests

Before running the tests you must initialize the database by running import_data.py. The tests will modify the database. Rerun import_data.py before starting the server. To use the tests on a test database, the database connection string in server.py needs to be adjusted to go to "database-test.db" instead of database.db. When starting the server, the test database will then be used. We tried to implement an automatic connection to the test database while testing, but this didn't work and it is on the TODO-list.

Unit tests

From the root folder of this project:

python -m unittest test_data.py

Integration tests

Before running integration tests you must run the server. You can do this via docker or with the instructions above. Integration tests also modify the database, so afterwards re-run import_data.py.

From the root folder of this project:

python -m unittest test_api.py

Using the client

Import and initialize the client:

from wdb_rest.client import TrackClient

# Change the URL to the address you are running this server on.
url = "http://127.0.0.1:5000/api/"

client = TrackClient(url)

Available methods and which endpoints they access:

Client method Endpoint HTTP verb Description
create_track Track PUT Create a new track.
get_tracks TrackList GET Get list of tracks by some criteria.
get_track Track GET Get single track by id.
update_track Track POST Update single track by id.
delete_track Track DELETE Delete single track by id.

More details on these methods are available in the form of doc strings in the client itself.

All methods return a tuple in the format: (server_response_json, HTTP return code). For more information on the return JSON and method parameters see the REST API chapter in this document.

Project dependencies

The following libraries need to be installed on the machine:

  • Flask - Application server
  • Flask_RESTful - RESTful extension for Flask
  • Flask_SQLAlchemy - Object relationship mapping tool for saving and loading Python objects form a database
  • pandas - Import data from CSV to sqlite
  • requests - Generate requests from the client

All requirements are added to the requirements.txt file.

REST API

The REST API offers 3 endpoints for accessing individual tracks and a list of tracks (based on the selection criteria).

  • TrackList endpoint /api/tracks/: Get a list of songs based on some criteria

    • GET /api/tracks/: Return the requested list of Tracks

      This endpoints supports 5 URL parameters:

      • page (default = 1) - which page of the paginated result you want to read.
      • sort_field (default = 'id') - by which field to order the result.
      • sort_order (default = 'asc') - in which direction ('asc' or 'desc') to order the results.
      • filter_field (optional) - field on which to filter.
      • filter_value (optional) - field value on which to filter.

      This endpoints always return results paginated with page_size = 10. The page URL parameters define which page you would like to receive back. If not specified otherwise, the sorting is always done in ascending order on the id field. The filtering is by default disabled, but if you specify the filter_field and filter_value your results will be filtered accordingly:

      • If the filter_field is a string, the service supports exact matches as well as '%LIKE%' SQL style matches.
      • If the filter_field is a number, we support only exact match.

      TODO: DESCRIBE THIS ENTIRE RETURN FORMAT.

      Example:

      # Retrieve the first page of songs whose artist contains the string "%Lana%", sorted by id in ascending order.
curl -s -X GET http://localhost:5000/api/tracks/?page=1&sort_field=id&sort_order=asc&filter_field=artist&filter_value=%Lana%

      Positive response (HTTP code 200):

      {
    "page": 1,
    "has_next": true,
    "has_prev": false,
    "tracks_iter": [1,2,3,4,5,null,4109,4110],
    "next_num": 2,
    "items": [],
    "prev_num": null
}

  • Track endpoint /api/track/<int:track_id>: CRUD access to Track records in database.

    • GET /api/track/<int:track_id> - retrieve single Track by id.

      Get returns you the requested Track if successful and a message telling you what is wrong otherwise.

      Example:

      # Retrieve track with id=1
curl -s -X GET http://localhost:5000/api/track/1

      Positive response (HTTP code 200):

      {
    "id": 1,
    "track": "Jealous Kind Of Fella",
    "artist": "Garland Green",
    "danceability": 0.417,
    "key": 3,
    "instrumentalness": 0.0,
    "tempo": 185.655,
    "duration_ms": 173533.0,
    "popularity": 1,
    "decade": "60s"
}

      Negative response (HTTP code 500):

      {
  "msg": "Cannot get track, track_id does not exist."
}

    • PUT /api/track/<int:track_id> - Create new Track record

      Creates a new Track. All fields are mandatory. Check the chapter import_data.py below for more details on the attributes and their types. You must specify all attributes in the database except id, because the database will assign it to the new Track.

      If successful you will get the newly created Track back, otherwise you will get a message telling you what is wrong.

      Example:

      # Create a new track. Note: you must pass 0 as the track_id in the URL when doing this request.
curl -s -X PUT http://localhost:5000/api/track/0 -H "Content-Type: application/json" -d '
{
    "track": "Jealous Kind Of Fella",
    "artist": "Garland Green",
    "danceability": 0.417,
    "key": 3,
    "instrumentalness": 0.0,
    "tempo": 185.655,
    "duration_ms": 173533.0,
    "popularity": 1,
    "decade": "60s"
}'

      Positive response (HTTP code 201):

      {
    "id": 41100,
    "track": "Jealous Kind Of Fella",
    "artist": "Garland Green",
    "danceability": 0.417,
    "key": 3,
    "instrumentalness": 0.0,
    "tempo": 185.655,
    "duration_ms": 173533.0,
    "popularity": 1,
    "decade": "60s"
}

      Negative response (HTTP code 500) (example if request was missing "decade" attribute):

       {"message": { "decade": "Decade is required." }}

    • PATCH /api/track/<int:track_id> - Update existing track by id

      Updates an existing Track. All fields are mandatory. Check the chapter import_data.py below for more details on the attributes and their types.

      If successful you will get the newly updated Track back, otherwise you will get a message telling you what is wrong.

      Example:

      # Update track with id=1
curl -s -X PATCH http://localhost:5000/api/track/1 -H "Content-Type: application/json" -d '
{
    "id": 1,
    "track": "Jealous Kind Of Fella",
    "artist": "Garland Green",
    "danceability": 0.417,
    "key": 3,
    "instrumentalness": 0.0,
    "tempo": 185.655,
    "duration_ms": 173533.0,
    "popularity": 1,
    "decade": "60s"
}'

      Postive response (HTTP code 200):

      {
    "id": 1,
    "track": "Jealous Kind Of Fella",
    "artist": "Garland Green",
    "danceability": 0.417,
    "key": 3,
    "instrumentalness": 0.0,
    "tempo": 185.655,
    "duration_ms": 173533.0,
    "popularity": 1,
    "decade": "60s"
}

      Negative response (HTTP code 500):

      {
  "msg": "Cannot update track, track_id does not exist."
}

    • DELETE /api/track/<int:track_id> - Delete existing track by id

      Delete the requested Track. If successful an empty JSON will be returned, otherwise a message telling you what is wrong.

      Example:

      # Delete track with id=1
curl -s -X DELETE http://localhost:5000/api/track/1

      Positive response (HTTP code 200):

      {}

      Negative response (HTTP code 500):

      {
  "msg": "Cannot delete track, track_id = 1 does not exist."
}

  • Track endpoint /api/recommendation/<int:track_id>/<int:how_many_recommendations>: Get a list of recommendations for a song.

    • GET /api/track/<int:track_id>/<int:how_many_recommendations>' - Return a selected list of recommendations based on a song.

      Get returns you a list of tracks similar to a track based on its ID. The similarity is calculated using cosine similarity and uses the following track attributes for recommendation:

      • danceability
      • key
      • instrumentalness
      • tempo
      • duration_ms
      • popularity
      • decade

      This endpoint supports the following parameters: how_many_recommendations (default = 10) - How many recommendations should the api return. Minimum is 1, maximum is 100.

      In a further step, there will be the possibility to retrieve recommendations by weighing the attributes above differently. For example [10, 1, 1, 1, 1, 1, 1] would mean recommending songs mostly based on their "danceability" similarity.

      Example:

      # Retrieve recommendations for track with id=1
curl -s -X GET http://localhost:5000/api/recommendation/1/?how_many_recommendations=10

      Positive response (HTTP code 200):

      [
    {
        "id":3,
        "track":"Melody Twist",
        "artist":"Lord Melody",
        "danceability":0.657,
        "decade":"60s",
        "duration_ms":223960.0,
        "instrumentalness":4.42e-06,
        "key":5,
        "popularity":0,
        "tempo":115.94
    },
    {
        "id":1647,
        "track":"Who Is Gonna Love Me?",
        "artist":"Dionne Warwick",
        "danceability":0.353,
        "decade":"60s",
        "duration_ms":192573.0,
        "instrumentalness":0.0,
        "key":0,"popularity":1,
        "tempo":94.655
    }
]

      Negative response (HTTP code 500):

      {
  "msg": "Invalid track id or track not found."
}

Project architecture

The REST api is built on top of Flask and uses Flask_RESTful for creating rest endpoints. The communication with the database is done via Flask_SQLAlchemy ORM tool in order to avoid writing SQL queries manually. The database is created and populated with Spotify songs and their attributes. To communicate with the server we offer a Python client which implements access to all available methods.

import_data.py

This script needs to be run before we start using the REST server, otherwise there will be no database to connect to. It first creates a new table called track_model in an sqlite3 database (creates a new file called database.db).

Table columns:

Column name Type Description
id INTEGER Primary index of records. This field is auto-incremented by the database.
track TEXT Name of the track.
artist TEXT Name of the artist performing the track.
danceability REAL Danceability score assigned by Spotify.
key INTEGER Key the track is performed in.
instrumentalness REAL Instrumentalness score assigned by Spotify.
tempo REAL Tempo of the track.
duration_ms INTEGER Track duration in milliseconds.
popularity INTEGER Track popularity on Spotify.
decade TEXT Decade in which the track was created.

After the table has been created, the script reads the CSV file with the Spotify data (stored in data/spotify_dataset.csv) into a Pandas Dataframe. The Dataframe allows for exporting its content into a database with the function to_sql. A copy of the database is created for testing. It holds the same but with the suffix "-test".

wdb_rest/server.py

RESTful server implementation in Flask. The server uses SQLAlchemy to communicate with the sqlite database created by the import_data.py script. It declares one model TrackModel and 2 Resources (endpoints):

  • Track - CRUD access to the data in the track_model table in the database.
  • TrackList - allows to get a filtered, sorted and paginated list of songs based on some search and sorting criteria.

The server validates all create and update requests on the Track resource - all fields in the Track model are mandatory.

The TrackList endpoint can filter and sort based on the fields in the track_model database. It also offers pagination of the results. For string fields it allows for like matching: searching for artist "%Lana%" returns all artist that have "Lana" somewhere in their name. For numeric fields it allows for exact matching only.

wdb_rest/data.py

Data access object that communicates with the database and implements all the data operations needed by server.py. With this file we separate the API definition (in server.py) and the database access code (in data.py), making it easier to change the data storage if needed.

wdb_rest/client.py

To communicate with the server you can use the provided Python client. It uses the requests module for generating HTTP requests and implements all the endpoints the server provides. This is a thin client in the sense that no validation or processing happens on the client itself - this is all done on the server. The client just forwards the user requests and returns the server responses.

About

RESTful API in Python

Resources

Readme
Activity

Stars

0 stars

Watchers

2 watching

Forks

2 forks
Report repository

Releases

No releases published

Packages

No packages published

Contributors 2

  •  
  •  

Languages