How Recommendation Engines Work: Collaborative and Content-Based Filtering

Ever wondered how Netflix or Spotify manages to guess exactly what you want to watch or listen to next? The secret lies in recommendation algorithms. A recommender system is a machine learning system that predicts which items a user will prefer, based on past ratings or item features, and surfaces the highest-scoring ones.

Whether they are suggesting movies, songs, or the next book to read, these systems generally rely on two core approaches: collaborative filtering (finding people with similar tastes) and content-based filtering (finding items similar to what you already like).

In this post, we will look under the hood of these recommendation systems, exploring the mathematical theory behind them and implementing them from scratch using Python. Let’s go!

The Rating Prediction Task in Recommender Systems

When we create Recommender Systems (RS), we consider that we have a set of users and items which are recommended to these users. In practice, we have a prior history of user ratings. This history is used to create suggestions or recommendations.

Consider a movie recommender as a widely given example of a recommender system. For instance, users watch Netflix content and rate movies they watch. Netflix has knowledge of preferred movies and recommends movies not yet seen that will be possibly liked by users (ideally :)

Basic RS uses matrices to store user ratings, such as :

[Users\Movies]  | User 1 | User 2 | User 3 | ... | User N |
----------------|-----------------------------------------|
Movie 1         |   10   |    4   |   6    | ... |   9    |
----------------------------------------------------------|
Movie 2         |   ?    |    7   |   9    | ... |   7    |
----------------------------------------------------------|   
Movie 3         |   7    |    9   |   6    | ... |   ?    |
----------------------------------------------------------|
Movie 4         |   ?    |    ?   |   9    | ... |   7    |
----------------------------------------------------------|  

Notice that User 1 did not watch Movie 2 (we have a “?” question mark in the cell of the rating table), and User N did not watch Movie 3.

We have to predict the missing user ratings. This task is called rating prediction. We can recommend movies with the highest predicted ratings when we predict all the missing values or movie ratings.

Indeed, not all recommenders use this matrix format in practice. Data structures and algorithms must be optimised for effective resource management and reduced computation time.

Recommendations should be created quickly with scalability in mind. This is particularly important when dealing with big data in production settings. However, let’s keep it simple and consider the rating matrix structure.

Collaborative Filtering: Recommending by User and Item Similarity

🔒 Subscribe to keep reading.

Content-Based Filtering: Recommending by Item Features

🔒 Subscribe to keep reading.

Summary: Collaborative vs Content-Based Filtering

🔒 Subscribe to keep reading.

Conclusion

🔒 Subscribe to keep reading.

References

🔒 Subscribe to keep reading.