Recommender.ipynb

{"cells":[{"cell_type":"markdown","source":["## Collaborative Filtering\nCollaborative filtering is a machine learning technique that predicts ratings awarded to items by users.\n\n### Import the ALS class\nIn this exercise, you will use the Alternating Least Squares collaborative filtering algorithm to creater a recommender."],"metadata":{}},{"cell_type":"code","source":["from pyspark.ml.recommendation import ALS"],"metadata":{"scrolled":false,"collapsed":false},"outputs":[],"execution_count":2},{"cell_type":"markdown","source":["### Load Source Data\nThe source data for the recommender is in two files - one containing numeric IDs for movies and users, along with user ratings; and the other containing details of the movies."],"metadata":{}},{"cell_type":"code","source":["ratings = spark.read.csv('wasb://spark@<YOUR_ACCOUNT>.blob.core.windows.net/data/ratings.csv', inferSchema=True, header=True)\nmovies = spark.read.csv('wasb://spark@<YOUR_ACCOUNT>.blob.core.windows.net/data/movies.csv', inferSchema=True, header=True)\nratings.join(movies, \"movieId\").show()"],"metadata":{"scrolled":false,"collapsed":false},"outputs":[],"execution_count":4},{"cell_type":"markdown","source":["### Prepare the Data\nTo prepare the data, split it into a training set and a test set."],"metadata":{}},{"cell_type":"code","source":["data = ratings.select(\"userId\", \"movieId\", \"rating\")\nsplits = data.randomSplit([0.7, 0.3])\ntrain = splits[0].withColumnRenamed(\"rating\", \"label\")\ntest = splits[1].withColumnRenamed(\"rating\", \"trueLabel\")\ntrain_rows = train.count()\ntest_rows = test.count()\nprint (\"Training Rows:\", train_rows, \" Testing Rows:\", test_rows)"],"metadata":{"collapsed":false},"outputs":[],"execution_count":6},{"cell_type":"markdown","source":["### Build the Recommender\nThe ALS class is an estimator, so you can use its **fit** method to traing a model, or you can include it in a pipeline. Rather than specifying a feature vector and as label, the ALS algorithm requries a numeric user ID, item ID, and rating."],"metadata":{}},{"cell_type":"code","source":["als = ALS(maxIter=5, regParam=0.01, userCol=\"userId\", itemCol=\"movieId\", ratingCol=\"label\")\nmodel = als.fit(train)"],"metadata":{"scrolled":false,"collapsed":false},"outputs":[],"execution_count":8},{"cell_type":"markdown","source":["### Test the Recommender\nNow that you've trained the recommender, you can see how accurately it predicts known ratings in the test set."],"metadata":{}},{"cell_type":"code","source":["prediction = model.transform(test)\nprediction.join(movies, \"movieId\").select(\"userId\", \"title\", \"prediction\", \"trueLabel\").show(100, truncate=False)"],"metadata":{"scrolled":false,"collapsed":false},"outputs":[],"execution_count":10},{"cell_type":"markdown","source":["The data used in this exercise describes 5-star rating activity from [MovieLens](http://movielens.org), a movie recommendation service. It was created by GroupLens, a research group in the Department of Computer Science and Engineering at the University of Minnesota, and is used here with permission.\n\nThis dataset and other GroupLens data sets are publicly available for download at <http://grouplens.org/datasets/>.\n\nFor more information, see F. Maxwell Harper and Joseph A. Konstan. 2015. [The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems (TiiS) 5, 4, Article 19 (December 2015)](http://dx.doi.org/10.1145/2827872)"],"metadata":{}}],"metadata":{"kernelspec":{"display_name":"PySpark","name":"pysparkkernel","language":""},"language_info":{"mimetype":"text/x-python","pygments_lexer":"python2","name":"pyspark","codemirror_mode":{"version":2,"name":"python"}},"name":"Python Recommender","notebookId":3378903555804655},"nbformat":4,"nbformat_minor":0}