Build a Hybrid Recommender System in Python using LightFM

Project Overview

A hybrid recommendation system built on collaborative filtering and content-based filtering is developed in this project to deliver product recommendations to customers. The primary goal this project aims to achieve is to build a personalized recommendation model suggesting products based on customer purchase history and product features like customer segments and product attributes.

One of the things the project makes use of is the LightFM Model, a popular library for building recommendation systems that can work with both types of filtering very efficiently. The system integrates two main sources of information:

Collaborative Filtering: User item-based (e.g. purchase history) interactions are used to recommend products similar to the user's preferences using this method.

Content-Based Filtering: This method is recommended for product features (it considers the product features, the customer segments associated with each product, characteristics, etc.).

Prerequisites

Python Programming: A basic acquaintance with Python and some libraries such as NumPy, Pandas, and SciPy.
Recommendation Systems: The collaborative and content-based filtering methods need to be known.
Data Preprocessing: The capability of merging datasets and making interaction matrices.
LightFM Library: Practical experience in the use of LightFM for making recommendation models.
Machine Learning Basics: Familiarity with model training for AUC and other metrics.
Sparse Matrices: Sparse matrices for large datasets and their application.

Approach:

In this project, collaborative filtering is combined with content-based to form a formal recommendation system. First, the merge of customer, product, and order data is done to create interaction matrices. The matrices in these cases are user-item interactions, both recording a purchase and item-feature interactions that link products to segments and features. The model is trained on these matrices using the LightFM library to predict customer preferences for the products, given: a) products' characteristics and b) customers' behavior. Finally, one evaluates the model using the AUC (Area of curve) as the metric; this model generates personalized product recommendations for users based on interactions of these two aspects; the features of these items and the user's past purchases.

Workflow and Methodology

Workflow

Data collection: Gather data on customers, products, and orders using the above datasets.
Data Merge: Merge this data to create a complete survey database for analysis and modeling purposes.
Data Preprocessing: Clean and convert all data into user-item interaction matrices and item features interaction matrices.
Train-Test Split: Separate training and test data to evaluate the model's performance.
Model Building: Train the hybrid recommendation model using the LightFM library with collaborative and content-based filtering.
Model Evaluation: The model will be evaluated with the help of AUC.
Recommendation Generation: A personalized recommendation will be generated for the customer using the trained model.

Methodology

Collaborative Filtering: Utilize user-item interaction data to recommend products based on how the user wants them.
Content-based Filtering: Use product characteristics (customer segmentation, age, gender, etc.) to recommend items according to those characteristics.
Hybrid Model: Present both filtering combined in fact-based and collaborative ways to make better recommendations.
LightFM: Implement and train the hybrid recommendation model using LightFM. The AUC metric measures: It ranks accurately for all items and assesses the ability of the model.
Sparse matrices: Constructing sparse matrices enables efficient storage and computation in working with large datasets.

Data Collection and Preparation

Data Collection:

In this project, we collected the dataset from a public repository. If you are looking to work on a real-world problem, you can get these kinds of datasets from publicly available repositories such as Kaggle, UCI Machine Learning Repository, or company-specific data. We will provide the dataset in this project so that you can work on the same dataset.

Data Preparation Workflow:

Loaded the Excel file to a dataframe of customer, product, and order data.
Merged the datasets based on the CustomerID and ProductName so that there was one dataset.
Aggregated how much each customer would purchase the quantity of products provided.
The user-item interaction matrix was created for training and test datasets.
Created an item-feature interaction matrix that maps products to customer segments.
Split the data into training and testing sets (67:33 ratio).
Assigned integer indices to mapped users, products, and features so that they can be used in our LightFM model.

Code Explanation

STEP 1:

Mounting of Google Drive

This code mounts your Google Drive into the Colab environment so that you can access files stored in your drive. Your Google Drive is made accessible under /content/drive path.

from google.colab import drive
drive.mount('/content/drive')

Required Libraries Installation

This code installs the required libraries for building and managing the recommendation system.

!pip install lightfm
!pip install numpy
!pip install pandas
!pip install scipy

Import Necessary Libraries

This code imports necessary libraries pandas for data manipulation, numpy for math computation, coo_matrix for constructing sparse matrix for evaluation auc_score, and LightFM model for model building.

import pandas as pd # pandas for data manipulation
import numpy as np # numpy for sure
from scipy.sparse import coo_matrix # for constructing sparse matrix
# lightfm
from lightfm import LightFM # model
from lightfm.evaluation import auc_score
# timing
import time

STEP 2:

Loading Datasets

This code loads three datasets and stores them in order, customer, and product variables respectively. This loads the dataset from Excel to DataFrame for analysis.

# import the data
order=pd.read_excel('/content/drive/MyDrive/New 90 Projects/Project_16/Dataset/Rec_sys_data.xlsx','order')
customer=pd.read_excel('/content/drive/MyDrive/New 90 Projects/Project_16/Dataset/Rec_sys_data.xlsx','customer')
product=pd.read_excel('/content/drive/MyDrive/New 90 Projects/Project_16/Dataset/Rec_sys_data.xlsx','product')

Merging Dataset

This code merges the three datasets in one DataFrame called full_table using CustomerID and StockCode as keys.

# merge the data
full_table=pd.merge(order,customer,left_on=['CustomerID'], right_on=['CustomerID'], how='left')
full_table=pd.merge(full_table,product,left_on=['StockCode'], right_on=['StockCode'], how='left')

Previewing Order Data

This code displays the order dataset's first few rows for a quick overview.

# check for first 5 rows for order data
order.head()

Previewing Customer Data

This code displays the customer dataset's first few rows for a quick overview.

# check for first 5 rows for customer data
customer.head()

Previewing Product Data

This code displays the product dataset's first few rows for a quick overview.

# check for the first 5 rows for product data
product.head()

STEP 3:

Visualize Unit Price

This code creates a line graph of the Unit Price column in the Product DataFrame.

from matplotlib import pyplot as plt
product['Unit Price'].plot(kind='line', figsize=(8, 4), title='Unit Price')
plt.gca().spines[['top', 'right']].set_visible(False)

Preparing Unique Data and Mappings:

Unique Users and Items: Sort unique user and item IDs through the functions unique_users and unique_items based on specified columns.
Feature Aggregation: features_to_add will combine and surpass duplicates from three columns in the customer DataFrame.
ID Mappings: mapping creates bi-directional mappings for users, items, and features for recommendation model processing, in which ID-to-index and vice versa.

# Creating the list of unique users
def unique_users(data, column):
    return np.sort(data[column].unique())
# Creating the list of unique produts
def unique_items(data, column):
    item_list = data[column].unique()
    return item_list
def features_to_add(customer, column1,column2,column3):
    customer1 = customer[column1]
    customer2 = customer[column2]
    customer3 = customer[column3]
    return pd.concat([customer1,customer3,customer2], ignore_index = True).unique()
# Create id mappings to convert user_id, item_id, and feature_id
def mapping(users, items, features):
    user_to_index_mapping = {}
    index_to_user_mapping = {}
    for user_index, user_id in enumerate(users):
        user_to_index_mapping[user_id] = user_index
        index_to_user_mapping[user_index] = user_id
    item_to_index_mapping = {}
    index_to_item_mapping = {}
    for item_index, item_id in enumerate(items):
        item_to_index_mapping[item_id] = item_index
        index_to_item_mapping[item_index] = item_id
    feature_to_index_mapping = {}
    index_to_feature_mapping = {}
    for feature_index, feature_id in enumerate(features):
        feature_to_index_mapping[feature_id] = feature_index
        index_to_feature_mapping[feature_index] = feature_id
    return user_to_index_mapping, index_to_user_mapping, \
           item_to_index_mapping, index_to_item_mapping, \
           feature_to_index_mapping, index_to_feature_mapping

Extracting Users, Items, and Features

The purpose of this code is to extract unique users, items, and features from raw data into a form suitable for a recommendation system. It does this by finding unique CustomerID values from the order DataFrame, unique product names from the product DataFrame, and extracting customer segments, ages, and genders from the customer DataFrame into one deduplicated list of features.

# create the user, item, feature lists
users = unique_users(order, "CustomerID")
items = unique_items(product, "Product Name")
features = features_to_add(customer,'Customer Segment',"Age","Gender")

The users variable contains a sorted list of unique CustomerID values extracted from the order DataFrame, representing all distinct users.

users

The items variable holds a list of unique product names extracted from the product DataFrame, representing all distinct items available.

items

The features variable combines and deduplicates respective values from customer Segment, Age, and Gender columns in the customer DataFrame, where attributes like unique IDs of the customer.

features

Generating mapping for LightFM

This code builds integer-based mappings for users, items, and features, rendering them usable by the LightFM library, which requires models with numerical indices.

# generate mapping, LightFM library can't read other than (integer) index
user_to_index_mapping, index_to_user_mapping, \
           item_to_index_mapping, index_to_item_mapping, \
           feature_to_index_mapping, index_to_feature_mapping = mapping(users, items, features)

Previewing Data

This code displays the full_table DataFrame’s first few rows for a quick overview.

full_table.head()

STEP 4:

Generating the Training Data

The user_to_product_rating_train variable extracts CustomerID, Product Name, and Quantity columns from the full_table DataFrame required for preparing user-item interaction data to train.

user_to_product_rating_train=full_table[['CustomerID','Product Name','Quantity']]

Mapping Products to Features

The product_to_feature variable extracts the columns for Product Names, Customer Segments, and Quantities from the full_table DataFrame and links products with the feature associated with their quantity.

product_to_feature=full_table[['Product Name','Customer Segment','Quantity']]

Aggregating User-Product Ratings

This code aggregates user_to_product_rating_train by CustomerID and Product Name to compute total interaction (rating) based on the sum of the Quantity values for each user-product pair. Reset the index for a clean DataFrame after that.

user_to_product_rating_train=user_to_product_rating_train.groupby(['CustomerID','Product Name']).agg({'Quantity':'sum'}).reset_index()

Display the Last Few Rows of User-Product Ratings

The tail() function returns the last rows of the user_to_product_rating_train DataFrame. In this way, the user-product interaction data can quickly be viewed after aggregation.

user_to_product_rating_train.tail()

Importing Train-Test Split

This code imports the train-test-split function from sklearn.model_selection which is one of the built-in functions in Python that allows users to split a particular programmatically defined data set into training and test sets for purposes of evaluation of its performance.

from sklearn.model_selection import train_test_split

Data Split into Train and Test Set

Here, the DataFrame user_to_product_rating_train is split into training (67%) and test (33%) datasets through the use of the train_test_split method while ensuring the benefit of reproducibility through a random_state value of 42.

# perform train test split - 67:33 percent
user_to_product_rating_train,user_to_product_rating_test = train_test_split(user_to_product_rating_train,test_size=0.33, random_state=42)

Checking the Dimension of Training Data

The shape property shows the number of rows and columns of the user_to_product_rating_train DataFrame, which corresponds to the dimension of the training dataset.

# check the shape of train data
user_to_product_rating_train.shape

Checking the Dimension of Test Data

Checking the Dimension of Test Data. The shape property shows the number of rows and columns of the user_to_product_rating_test DataFrame, which corresponds to the dimension of the test dataset.

# check the shape of the test data
user_to_product_rating_test.shape

Grouping Data with Aggregation

This block of code groups the product_to_feature DataFrame by Product Name and Customer Segment; and then aggregates to total the Quantity over all transactions of an individual group. Finally resetting the index to give it a more clear table format.

# perform groupby
product_to_feature=product_to_feature.groupby(['Product Name','Customer Segment']).agg({'Quantity':'sum'}).reset_index()

Previewing Data

This code displays the product_to_feature DataFrame’s first few rows for a quick overview.

product_to_feature.head()

STEP 5:

Creating Sparse Matrix for the Interactions

The interactions function takes data and populates rows, columns, and values using provided mappings. It generates a sparse matrix (coo_matrix) that efficiently represents interactions among rows and columns with their corresponding values.

# create a function for interactions
def interactions(data, row, col, value, row_map, col_map):
    row = data[row].apply(lambda x: row_map[x]).values
    col = data[col].apply(lambda x: col_map[x]).values
    value = data[value].values
    return coo_matrix((value, (row, col)), shape = (len(row_map), len(col_map)))

Generate Interaction Matrices

The code creates sparse matrices to map customer-product interactions and product-feature associations based on quantities.

# generate user_item_interaction_matrix for train data
user_to_product_interaction_train = interactions(user_to_product_rating_train, "CustomerID",
                                                    "Product Name", "Quantity", user_to_index_mapping, item_to_index_mapping)
# generate item_to_feature interaction
product_to_feature_interaction = interactions(product_to_feature, "Product Name", "Customer Segment","Quantity",
                                                        item_to_index_mapping, feature_to_index_mapping)

Generate Test Interaction Matrix

This code creates a sparse matrix mapping customer-product interactions in the test data based on quantities.

# generate user_item_interaction_matrix for test data
user_to_product_interaction_test = interactions(user_to_product_rating_test, "CustomerID",
                                                    "Product Name", "Quantity", user_to_index_mapping, item_to_index_mapping)

A sparse matrix representing customer-product interactions in the training data, based on quantities purchased.

user_to_product_interaction_train

A sparse matrix representing customer-product interactions in the test data, based on quantities purchased.

user_to_product_interaction_test

A sparse matrix linking products to their features, using quantities as interaction values.

product_to_feature_interaction

STEP 6:

Model Initialization and Training

The LightFM model with the use of "warp" loss is firstly initialized. Next, it combines collaborative and content filtering to train the user product to product features train for 100 epochs with 4 threads. Time has been measured and printed.

# initialising model with warp loss function
model_with_features = LightFM(loss = "warp")
# fitting the model with hybrid collaborative filtering + content based (product + features)
start = time.time()
#===================
model_with_features.fit_partial(user_to_product_interaction_train,
          user_features=None,
          item_features=product_to_feature_interaction,
          sample_weight=None,
          epochs=100,
          num_threads=4,
          verbose=False)
#===================
end = time.time()
print("time taken = {0:.{1}f} seconds".format(end - start, 2))

AUC Scoring for Model Assessment

This code checks the model's performance relative to the AUC score based on tests of user-to-product interaction test and train (user_to_product_interaction_test vs. user_to_product_interaction_train) in conjunction with item features (product_to_feature_interaction). It measures the time taken for evaluation against the average AUC score to evaluate the model's predictive accuracy.

start = time.time()
#===================
auc_with_features = auc_score(model = model_with_features,
                        test_interactions = user_to_product_interaction_test,
                        train_interactions = user_to_product_interaction_train,
                        item_features = product_to_feature_interaction,
                        num_threads = 4, check_intersections=False)
#===================
end = time.time()
print("time taken = {0:.{1}f} seconds".format(end - start, 2))
print("average AUC without adding item-feature interaction = {0:.{1}f}".format(auc_with_features.mean(), 2))

Model Initialization and Training with Logistic Loss

It initializes the LightFM model with "logistic" loss function and trains that on hybrid collaborative and content-based approaches. It runs this for 100 epochs via the user_to_product_interaction_train and product_to_feature_interaction datasets for further interaction. Finally, the elapsed time for training is captured and printed.

# initialising model with warp loss function
model_with_features = LightFM(loss = "logistic")
# fitting the model with hybrid collaborative filtering + content based (product + features)
start = time.time()
#===================
model_with_features.fit_partial(user_to_product_interaction_train,
          user_features=None,
          item_features=product_to_feature_interaction,
          sample_weight=None,
          epochs=100,
          num_threads=4,
          verbose=False)
#===================
end = time.time()
print("time taken = {0:.{1}f} seconds".format(end - start, 2))

Model Evaluation with AUC Score (Logistic Loss)

Evaluate the test (user_to_product_interaction_test) and training (user_to_product_interaction_train) data using the AUC score to assess the performance of the model, along with item features (product_to_feature_interaction). The period for evaluation is taken, and the average AUC score is printed to account for the model's effectiveness.

start = time.time()
#===================
auc_with_features = auc_score(model = model_with_features,
                        test_interactions = user_to_product_interaction_test,
                        train_interactions = user_to_product_interaction_train,
                        item_features = product_to_feature_interaction,
                        num_threads = 4, check_intersections=False)
#===================
end = time.time()
print("time taken = {0:.{1}f} seconds".format(end - start, 2))
print("average AUC without adding item-feature interaction = {0:.{1}f}".format(auc_with_features.mean(), 2))

Training Model Using BPR Loss and More Threads

Initialization of the LightFM model through a bpr loss function which then trains with the help of collaborative filtering as well as content-based filtering. The model makes use of user-product interactions and product features for training, along with the print time taken to train the model.

# initialising model with warp loss function
model_with_features = LightFM(loss = "bpr")
# fitting the model with hybrid collaborative filtering + content based (product + features)
start = time.time()
#===================
model_with_features.fit_partial(user_to_product_interaction_train,
          user_features=None,
          item_features=product_to_feature_interaction,
          sample_weight=None,
          epochs=100,
          num_threads=4,
          verbose=False)
#===================
end = time.time()
print("time taken = {0:.{1}f} seconds".format(end - start, 2))

Model Evaluation with AUC Score (BPR Loss)

The code assesses the model using its AUC score with two evaluation criteria that compare the item features of training and test data interaction. Also, it keeps track of the evaluation time, which it prints alongside the average AUC score representation of model accuracy.

start = time.time()
#===================
auc_with_features = auc_score(model = model_with_features,
                        test_interactions = user_to_product_interaction_test,
                        train_interactions = user_to_product_interaction_train,
                        item_features = product_to_feature_interaction,
                        num_threads = 4, check_intersections=False)
#===================
end = time.time()
print("time taken = {0:.{1}f} seconds".format(end - start, 2))
print("average AUC without adding item-feature interaction = {0:.{1}f}".format(auc_with_features.mean(), 2))

Training Model Using Logistic Loss and More Threads

Initialization of the LightFM model through a logistic loss function, which then trains with the help of collaborative filtering as well as content-based filtering. The model makes use of user-product interactions and product features for training, along with the print time taken to train the model.

model_with_features = LightFM(loss = "logistic")
# fitting the model with hybrid collaborative filtering + content based (product + features)
start = time.time()
#===================
model_with_features.fit_partial(user_to_product_interaction_train,
          user_features=None,
          item_features=product_to_feature_interaction,
          sample_weight=None,
          epochs=100,
          num_threads=20,
          verbose=False)
#===================
end = time.time()
print("time taken = {0:.{1}f} seconds".format(end - start, 2))

Model Evaluation with AUC Score (Logistic Loss)

start = time.time()
#===================
auc_with_features = auc_score(model = model_with_features,
                        test_interactions = user_to_product_interaction_test,
                        train_interactions = user_to_product_interaction_train,
                        item_features = product_to_feature_interaction,
                        num_threads = 4, check_intersections=False)
#===================
end = time.time()
print("time taken = {0:.{1}f} seconds".format(end - start, 2))
print("average AUC without adding item-feature interaction = {0:.{1}f}".format(auc_with_features.mean(), 2))

STEP 7:

Function to Merge Training and Testing Data

The train_test_merge function merges the training and testing data into a single data set. First, it stores training data in a dictionary and then adds testing data to the dictionary replacing values if there are any. Then it merges the data into a merged data back into a sparse matrix (coo_matrix) for efficient storage and computation.

def train_test_merge(training_data, testing_data):
    # initialising train dict
    train_dict = {}
    for row, col, data in zip(training_data.row, training_data.col, training_data.data):
        train_dict[(row, col)] = data
    # replacing with the test set
    for row, col, data in zip(testing_data.row, testing_data.col, testing_data.data):
        train_dict[(row, col)] = max(data, train_dict.get((row, col), 0))
    # converting to the row
    row_list = []
    col_list = []
    data_list = []
    for row, col in train_dict:
        row_list.append(row)
        col_list.append(col)
        data_list.append(train_dict[(row, col)])
    # converting to np array
    row_list = np.array(row_list)
    col_list = np.array(col_list)
    data_list = np.array(data_list)
    return coo_matrix((data_list, (row_list, col_list)), shape = (training_data.shape[0], training_data.shape[1]))

Merging Training and Testing Data

The train_test_merge combines the training and testing user-product interaction matrices into one data set, updating training data with test data when applicable.

user_to_product_interaction = train_test_merge(user_to_product_interaction_train,
                                                 user_to_product_interaction_test)

This is the merged user-product interaction matrix that combines both training and testing data.

user_to_product_interaction

Retraining the Final Model

We retrain the LightFM model with a user_to_product_interaction data set using a combined dataset, with the “logistic” loss function and 30 components. Then this is trained for 1000 epochs 20 threads and the time taken for training is printed.

# retraining the final model with combined dataset
final_model = LightFM(loss = "logistic",no_components=30)
# fitting to combined dataset
start = time.time()
#===================
final_model.fit(user_to_product_interaction,
          user_features=None,
          item_features=product_to_feature_interaction,
          sample_weight=None,
          epochs=1000,
          num_threads=20,
          verbose=False)
#===================
end = time.time()
print("time taken = {0:.{1}f} seconds".format(end - start, 2))

Function to Get Recommendations

The function get_recommendations creates recommendations for a given user for products. The first thing the function does is retrieve the user's index and then retrieves products that the user has previously bought. The model predicts scores for all items and then selects the top recommended items based on the highest scores. The next step is for the system to print out the list of already known products and the recommended products for the user.

def get_recommendations(model,user,items,user_to_product_interaction_matrix,user2index_map,product_to_feature_interaction_matrix):
    # getting the userindex
        userindex = user2index_map.get(user, None)
        if userindex == None:
            return None
        users = userindex
        # products already bought
        known_positives = items[user_to_product_interaction_matrix.tocsr()[userindex].indices]
        print('User index =',users)
        # scores from model prediction
        scores = model.predict(user_ids = users, item_ids = np.arange(user_to_product_interaction_matrix.shape[1]),item_features=product_to_feature_interaction_matrix)
        # top items
        top_items = items[np.argsort(-scores)]
        # printing out the result
        print("User %s" % user)
        print("     Known positives:")  # already known products
        for x in known_positives[:10]:
            print("                  %s" % x)
        print("     Recommended:")  # products that are reccommended to the user
        for x in top_items[:10]:
            print("                  %s" % x)

Recommendation Verification for User

The get_recommendations function gets called for the userID-17017 using final_model, which has been trained. This retrieves and prints the list of products already bought by the user and the top recommended products according to the predictions of the model.

# check for the reccomendation
get_recommendations(final_model,17017,items,user_to_product_interaction,user_to_index_mapping,product_to_feature_interaction)

Recommendation Verification for User

The get_recommendations function gets called for the userID-18287 using final_model, which has been trained. This retrieves and prints the list of products already bought by the user and the top recommended products according to the predictions of the model.

get_recommendations(final_model,18287,items,user_to_product_interaction,user_to_index_mapping,product_to_feature_interaction)

Recommendation Verification for User

The get_recommendations function gets called for the userID-13933 using final_model which has been trained. This retrieves and prints the list of products already bought by the user and the top recommended products according to the predictions of the model.

get_recommendations(final_model,13933,items,user_to_product_interaction,user_to_index_mapping,product_to_feature_interaction)

Conclusion

This project successfully built a hybrid recommendation system that combines collaborative filtering and content-based filtering methods using the LightFM library. The method is a combination of customer, product and order data into user-item and item-feature interaction matrices. Using these matrices, a recommendation model is constructed which predicts personalized product recommendations according to customer behavior and product features. The model evaluation has been conducted using the metric AUC demonstrating the effectiveness of ranking items. The hybrid system provides a better experience for user-relevant recommendations than traditional means. Here, we hand-pick the importance of synchronizing data preprocessing, feature engineering, and evaluation of the model into building robust recommendation systems.

Challenges New Coders Might Face

Challenge: Handling noisy or unstructured text data.
Solution: Utilize text cleaning methods, which may include the exclusion of special symbols, figures, and extra spaces.
Challenge: Preprocessing Large Text Data
Solution: Enhance text cleaning processes by employing better libraries such as NLTK and adopting batch processing for the data.
Challenge: Curse of Dimensionality in the high-dimensional text datasets affecting clustering and classification results.
Solution: Use TF-IDF vectorization and reduction techniques like (PCA) to control dimensionality.
Challenge: Cold Start Problem
Solution: Content-based filtering can be applied to new items and users. Demographic information such as age, gender, etc., can be utilized for new users, and for new products, product features are more heavily weighted than customer segments.
Challenge: Data Sparsity
Solution: Matrix factorization can be applied or hybrid approaches (like content-based filtering) can be used to minimize the effects of sparseness. LightFM also supports hybrid models because it can be both collaborative and content-based filtering.

Frequently Asked Questions (FAQs)

Question 1: What is a hybrid recommendation system?
Answer: A Hybrid recommendation system uses multiple techniques, such as collaborative filtering, content-based filtering, and so on. This system helps improve the accuracy of the recommendation, thereby making it more diverse for products. User interactions and product features are strongly combined traits that give all users personalized suggestions.

Question 2: How Collaborative Filtering is done in a Recommendation System?
Answer: The user's interactive information for example, the history of the items purchased by the people is treated and then those patterns are found in user behavioral interaction and the resultant recommendation is produced whose basis is similar to the preferences among users.

Question 3: What is content-based filtering in recommendation systems?
Answer: Content-based filtering is an item recommendation with reference to attributes of the product like features, categories and customer segment. Again, It will take characteristics of an item, but it won't consider the user behavior at all during the recommendation.

Question 4: Why is LightFM used to build hybrid recommendation models?
Answer: LightFM is the one powerful Python library designed for the hybrid recommendation system. The use of this library makes it efficient for collaborative and content-based filtering, sparsely populated matrices, and ranking them in an optimized fashion for creating personalized recommendations.

Question 5: How do you address data sparsity in collaborative filtering?
Answer: Data sparsity limits collaborative filtering as a result of the sparsity in terms of very few interactions by the users with the items. Remedies to this include using matrix factorization techniques, hybrid models (content-based filtering), and algorithms that fit well in sparse data analysis.

Build a Hybrid Recommender System in Python using LightFM

Project Overview

Prerequisites

Python Programming: A basic acquaintance with Python and some libraries such as NumPy, Pandas, and SciPy.

Recommendation Systems: The collaborative and content-based filtering methods need to be known.

Data Preprocessing: The capability of merging datasets and making interaction matrices.

LightFM Library: Practical experience in the use of LightFM for making recommendation models.

Machine Learning Basics: Familiarity with model training for AUC and other metrics.

Sparse Matrices: Sparse matrices for large datasets and their application.

Approach:

Workflow and Methodology

Workflow

Methodology

Data Collection and Preparation

Data Collection:

Data Preparation Workflow:

Code Explanation

STEP 1:

Mounting of Google Drive

Required Libraries Installation

Import Necessary Libraries

STEP 2:

Loading Datasets

Merging Dataset

Previewing Order Data

Previewing Customer Data

Previewing Product Data

STEP 3:

Visualize Unit Price

Extracting Users, Items, and Features

Generating mapping for LightFM

Previewing Data

STEP 4:

Generating the Training Data

Mapping Products to Features

Aggregating User-Product Ratings

Display the Last Few Rows of User-Product Ratings

Data Split into Train and Test Set

Checking the Dimension of Training Data

Checking the Dimension of Test Data

Grouping Data with Aggregation

Previewing Data

STEP 5:

Creating Sparse Matrix for the Interactions

Generate Interaction Matrices

Generate Test Interaction Matrix

STEP 6:

Model Initialization and Training

AUC Scoring for Model Assessment

Model Initialization and Training with Logistic Loss

Model Evaluation with AUC Score (Logistic Loss)

Training Model Using BPR Loss and More Threads

Model Evaluation with AUC Score (BPR Loss)

Training Model Using Logistic Loss and More Threads

Model Evaluation with AUC Score (Logistic Loss)

STEP 7:

Function to Merge Training and Testing Data

Merging Training and Testing Data

Retraining the Final Model

Function to Get Recommendations

Recommendation Verification for User

Recommendation Verification for User

Recommendation Verification for User

Conclusion

Challenges New Coders Might Face

Frequently Asked Questions (FAQs)