Learn to Build a Polynomial Regression Model from Scratch

Project Overview

In this project, you will be learning how to create your polynomial regression model from scratch. You will realize how to mold a basic linear model to your advantage. First, let’s collect data and clean it. After that, we will discuss how polynomial regression is different from simple linear regression. Last, we will proceed with coding step by step while explaining it in detail. By the end of this project, you will have a working model that can handle non-linear trends perfectly. Ready to start? Let’s dive in!

Prerequisites

• Basic Python Skills: Be able to write loops, functions, and variables.
• Understanding of Linear Regression: Familiar with drawing a straight line to fit data.
• Basic Math Knowledge: Knowledge in simple algebra, power, and exponentiation.
• Libraries: NumPy and Matplotlib: Familiarity with calculations, data manipulations, and data visualization.

With these basics, you’re ready to start the project!

Approach

Our approach starts by setting up the dataset. After that, we'll use the dataset to train and test our model. We start with a quick review of linear regression to see where polynomial regression can add value. Next, we’ll transform our data by adding polynomial features. This allows our model to capture curves instead of just straight lines. Once the data is prepared, we’ll code the polynomial regression model step-by-step. We will employ the Python programming language and its NumPy library for numerical calculations and transformations. After training, we'll present the results using Matplotlib to assess the data-fitting ability of the model.

In the end, we will do a comparison of these two models and perform a more advanced analysis to show the clear benefit of using this model in practical applications dealing with databases.

Workflow

1. Data Collection: First, we gathered the datasets made of the images of healthy and infected leaf samples.
2. Data preprocessing: Once we gathered samples, we handled missing values. Then the images were normalized. After that, it was divided into data for training and others for validation.
3. Feature Engineering
   Add polynomial features to the dataset, which will allow our model to fit curves.
4. Model Building: Implementing polynomial regression from scratch using Python and NumPy to handle the math.
5. Training the Model: The model is then trained on the training dataset to learn from the patterns.
6. Evaluation: Carrying out model testing and its performance assessment.
7. Comparison with Linear Regression
   Compare results with a basic linear regression model to highlight the power of polynomial regression.

Methodology

To construct a polynomial regression model, we start by increasing the dimension of our features by adding polynomial terms of the original features like squares and cubes of the original respectively. This enables the model to fit higher-order trends. We then use least squares estimation to train the model and this is just a concept of getting closer to the real values. By mapping each step, from feature engineering up to training and evaluation, we shall not only know how it works but also find out why the model suits non-linear data. We can visualize and compare the effects of polynomial regression, thus making a practical and useful methodology!

Data Collection

First, we collected a public dataset that shows a non-linear pattern. This will allow us to see the benefits of polynomial regression. You can also use real-world data like housing prices or stock trends. In this dataset, values fluctuate in complex ways.

If you're a beginner, try to create artificial data with a curved pattern. This way at least you will be in control of the data. This makes it easier to know whether the model we are using is functioning as required.

Data Preparation

Once the dataset has been collected, it is then time to prepare the data set. We will start with the missing values so that it doesn’t mess up our data set. Following this, we will normalize our features to ensure all of them are in the same range. This is especially important in polynomial regression. Also, the last step to perform is transforming (squaring, cubing, or taking to higher powers) the original built features to make new features. This transformation enables our model to fit curves rather than mere straight lines. So that more relationships can be captured.

Data Preparation Workflow

• Handle Missing Values: Impute missing values so that it does not reduce the total number of features in a dataset on which the training is going to be done.
• Feature Scaling: Scale features to the same scale for training an accurate model.
• Generate Polynomial Features: To incorporate non-linear relationships, add polynomial features.
• Split the Data: Divide the available data into training and validation data.
  Final Check: Check all features for modeling with the conditions that no outliers are influencing the results and no need for scale transformations.

STEP 1:

Code explanation

Here’s what is happening under the hood. Let’s go through it step by step:

Mount Google Drive

Mount your Google Drive to access and save datasets, models, and other resources.

Suppress Warnings

It excludes non-critical warnings from the output, producing a cleaner view of the results.

Install Required Libraries

This installs the LightGBM and Scikit-Learn libraries. LightGBM is often used for gradient boosting in machine learning, while Scikit-Learn provides essential tools for building and evaluating models.

Import Libraries

This section imports the necessary library for data manipulation (pandas, numpy), graph plotting (seaborn, plotly, matplotlib), statistical operations (scipy), and machine learning models (sklearn).

STEP 2:

Load the Dataset

The code employs the pandas library to retrieve the dataset from Google Drive. The head() method shows the first few records within the dataset to have an overview of the dataset.

Check the Dataset Dimensions

This statement displays the dimension of the dataset in terms of how many rows and columns it has. For example, if it says (317, 7), this means that the dataset has 317 rows and 7 columns, thereby helping to verify that the correct amount of data was indeed loaded.

Display Column Names

The code is designed to print the names of the dataset's columns. This helps to quickly understand the features present in the data to be analyzed.

Dataset Overview

Before starting the data cleaning or the analysis process, it is wise to understand the structure of the data and look for missing values. This command is very useful in undertaking this task, as it lets the user easily perceive the characteristics of the data.

Renamed Columns for Easy Analysis

The code alters the column names of the data DataFrame to make simpler names and produces a new DataFrame df with such column names. It then uses df.head(10) to present the first 10 rows to check the modifications made.

STEP 3:

Visualizing Data Distribution and Outliers for Deeper Insights into Points

This code creates three visualizations for the Points variable: These include a density plot, a transformed density plot (using the square root of Points), and a boxplot. For detail, each of the plots assists us in identifying different aspects of data distribution such as spread, skewness, and outliers. The layout is changed to make it easier to view the plots, and the plots are shown beside each other for contrast.

Viewing the Last 100 Rows of the Dataset

The code displays the last 100 rows of the df DataFrame.

Creating Violin and Box Plots for Each Variable by Team

The function plotting_box_violin_plots() is constructed to create violin and box plots for any given variable against the ‘Team’ column to better visualize the spread and the central tendency of the data. This function is subsequently called in a loop to draw these plots for each variable (‘WL’, ‘Yoga’, ‘Laps’, ‘WI’, ‘PAFS’) and allow for comparison for each of them against the ‘Team’ label.

STEP 4:

Identifying Outliers in Selected Columns

The function find_outliers() is employed to determine outliers in a given column by computing the Interquartile Range (IQR) of that column. In a for-loop manner, this function is executed column-wise on the columns (‘WL’, ‘Yoga’, ‘Laps’, ‘WI’, ‘PAFS’) and any values that are beyond the composed range are displayed, making it possible to identify data points that need additional attention.

Eliminating Certain Rows for Data Cleaning Purposes

The code constructs a revised DataFrame df_clean such that the rows with index numbers 142, 143, and 144 in df are excluded. In addition, the command df_clean.shape is employed to provide information regarding the size of the cleaned DataFrame.

Replacing Invalid Values with NaN in the Cleaned Data

The specified code eliminates all the instances of the value 1111111.0 present in the 'WL' column of df_clean by substituting it with NaN denoting it as missing data. On displaying df_clean['WL’] the modified column is presented which helps in identifying the records that will require additional treatment during the analysis.

STEP 5:

Calculating and Displaying Missing Data Proportion

The dataframe ncounts display the amount of missing data per column of the dataframe df_clean. It resorts to df_clean.isna().mean() to get the count of the NaN values as a percentage and the column is then transposed and given the name data_missing for ease of reference.

Visualizing Missing Data Proportion

The purpose of this code is to generate a horizontal bar graph showing the proportion of missing values against each of the columns in ncounts. Here, ncounts.plot(kind='barh', title='% of missing values across each column’) gives an intuitive understanding of the missing data in the columns and assists in determining which sections may need data winnowing. Lastly, the output has been generated using plt.show().

Comparing Data Shapes Before and After Dropping Missing Values

This comparison helps you understand the impact of dropping rows or columns with missing data on the dataset’s size.

Getting an Overview of the Cleaned Data

This summary provides insight into the arrangement of the prepared dataset before carrying out any evaluation.

Filling Missing Values in the 'WL' Column

The code replaces all NaN values in the 'WL' column with -1.

STEP 6:

Visualizing the Effect of Filling Missing Values with Mean and Median

The presented code illustrates the effect on the distribution of the WL column when its missing values are replaced with the mean or the median. The placing of these two plots next to each other facilitates the comparison between the two imputation approaches and gives an idea of which method is better at retaining the original characteristics of the data.

Calculating Mean 'WL' per Team

This piece of code computes the average of the WL column for every distinct Team in df_clean, utilizing groupby(), and outputs the result in the form of a dictionary.

Replacing NaN Values in the 'WL' Column on a Team-Wise Basis

This code goes through every row of df_clean. In cases where the WL value is missing within a specific column, the void is filled with the mean WL value, which is associated with the team from the mean_WL dictionary.

Depicting the Distribution of 'WL', Once the NA's are Filled

Here’s a code that is aimed at creating a distribution plot for the column WL in a cleaned data frame where missing data has been filled with mean scores corresponding to each of the teams.

Using Simple Imputer to Handle Missing Values

This snippet fills up the missing values in the defined columns of the data frame by employing a SimpleImputer using the mean strategy. A new data frame, named si_impt_df is created, which consists of all the values present in the original data frame without any missing values, as each column has been filled with the mean of that particular column.

Applying Iterative Imputer in Handling Missing Data

This script executes an IterativeImputer with 10 max_iter to handle missing values in the specified columns. It predictively improves any missing values using other features and generates thus imputed data in a new DataFrame called ITI_impt_df. This imputation is complex compared to filling values using a mean.

Using KNN Imputer for Missing Value Imputation

This code employs a KNNImputation with n_neighbors=3 that fills in missing values by averaging the values of the three closest neighbors. The imputed data is returned in a new DataFrame, KNN_impt_df that provides instance-based filling of missing values in the other entries of the dataset.

STEP 7:

Handle Missing Value Imputation through Iterative Imputer with LightGBM

This code defines an IterativeImputer object, the estimator of which is a LightGBMRegressor used to perform the missing value imputation in the given columns. The imputed data set is stored in a new DataFrame called lgbm_impt_df, making highly accurate imputations with the help of LightGBM capability for prediction.

Inspecting and Renaming Columns in the Imputed Data

This section of the code assigns the lgbm_impt_df DataFrame filled with LightGBM-imputed data and creates a new DataFrame called df_new.

Visualizing Distributions and Boxplots of Transformed Features

The function creates the grid of plots per number of features in df_new that examines their distributions and boxplots. It also transformed the distributions (square root and cube root) and created boxplots for WL, Yoga, and PAFS to enable a clear comparison of how the data is spaced out as well as the possible outliers.

Adding 'Points' and 'Team' Columns to the Imputed Data

This code copies the 'Points' and 'Team' columns from df_clean into the df_new FrameData.

Calculating the Correlation Matrix for Numerical Columns

This piece of code filters out numerical columns from the df. It also computes the correlation matrix with the help of numerical_df.corr(). The correlation matrix correlation_matrix outlines how numerical attributes are related helping in spotting out strong relationships that may be positive or negative for further investigation.

Using a Heatmap to Illustrate the Correlation Matrix

The code provided here uses sns.heatmap() to construct a correlation matrix as a heatmap and adds the values of correlation on the map (annot=True). This is utilized to show the existence of positive and negative correlations, making strong relationships between variables apparent.

Creating a Pair Plot to Visualize Relationships Between Features

This code creates scatter plots of all feature pairs in df_new, with points colored by the Team column. This helps to visually explore relationships and patterns between variables across different teams.

Creating a Pair Plot to Visualize Relationships Between Features

This code creates scatter plots of all feature pairs in df_new. This helps to visually explore relationships and patterns between features.

Calculating Custom Correlation Using Ranked Data

The function aionlinecourse_corr() calculates a custom correlation coefficient between two variables, x, and y, in the DataFrame df.

Calculating Custom Correlation Between Variables

This piece of code implements a function called aionlinecourse_corr(), which computes a certain correlation coefficient which is based on rankings. It is then used to find out how well Points correlate with the rest of the variables (WI, Laps, PAFS, Yoga) and helps see how strong the associations are towards those variables.

STEP 8:

Creating a Sample DataFrame and Grouping by Team

This snippet of code draws a random sample containing 50 rows of the df_new data frame with emphasis on the 'Team' and 'Points' columns. The sample is then divided into groups regarding 'Teams' to assess how many times each one occurs effectively summarizing how the teams are spread out in the sample.

Creating Probability Plots per Each Distinct Team

This script generates probability plots for the distribution of Points for each distinct team contained in sample_df and arranges them in a grid.

Calculating the Ratio of Maximum to Minimum Standard Deviation Across Teams

This piece of code computes the standard deviation of Points for each of the teams in sample_df and afterward calculates and returns the value of the maximum standard deviation for all the teams to the smallest standard deviation. This helps to understand the differences in variability in the Points among the teams and how far apart or clustered the team's scores are from each other.

Creating an ANOVA Table for an Analysis of Variance for Groups

The code creates an empty ANOVA table called anova_table and has specific sections for different sources of variation along with their corresponding Sum of Squares (SS), degrees of freedom (df), Mean Square (MS), F value, P value and F critical values for between-groups, within groups and total.

The Sum of Squares Between Groups (SSTR) is obtained as the difference between the mean Points of each team to the overall mean (x_bar) to capture the extent of variation among team means. The SSTR calculated is then populated into the ANOVA table under the “Between Groups” section in the “SS” column. This is important in the analysis of the research question that seeks to determine whether there are significant differences within teams.

Computation of the Between-Groups Sum of Squares (SSE)

SSE is computed in the code by determining the variance within each of the teams (clusters) and multiplying that by the sample size of each team subtracting one. This measure captures the variability in Points to the Scores within the Team Showing how much each score varies from the average score of the team.

Updating ANOVA Table with Sum of Squares Within Groups (SSE)

The code adds the total SSE for Points to the "SS" column, which is listed as "Within Groups" in the anova_table. This update helps to finalize the accurate sum of squares and prepares the ANOVA table for subsequent operations.

Calculating and Adding the Total Sum of Squares to ANOVA Table

This section of code calculated the Total Sum of Squares (SST) by aggregating the two components, SSE and SSTR for Points, and it considered the overall variations present in the data set. The total is then inserted into the “SS” row of the anova_table in the column labeled ‘Total’, thereby finalizing the sum of squares section in the ANOVA table.

Calculating and Adding Degrees of Freedom to the ANOVA Table

This code calculates and adds the degrees of freedom for "Between Groups," "Within Groups," and "Total" to the anova_table.

Computation of Mean Squares, F Value, and P Value for ANOVA Table

This code calculates the Mean Square (MS) for each variation category by dividing the Sum of Squares (SS) against the degrees of freedom (df). For the “Between Groups” effect, the F value is obtained from the ratio of MS of “Between Groups” to the MS of “Within Groups.” The P value for “Between Groups” is further computed based on the cumulative distribution function of the F-distributed random variable. These values are appended to anova_table facilitating analysis of the statistical significance of differences in groups.

Calculating F Critical Value for Hypothesis Testing

The present code calculates the F critical value for “Between Groups” in a two-tailed test by modifying the significant level alpha and incorporating the F-distribution. This value is also incorporated within anova_table to conduct the final hypothesis testing.

STEP 9:

Interpreting the P-Value and Drawing a Conclusion

This snippet of code explains the results obtained from the ANOVA output by calculating the P value for the “Between Groups” and comparing it with alpha. In the case where the P value is less than alpha, this leads to the rejection of the null hypothesis, more so indicating that there is a difference between the groups. If not, as a result, it states that the null hypothesis is not tenable. It then proceeds to display the F value, the P value, and the summary for further understanding.

Interpreting F Critical Value for Hypothesis Testing

This code compares the F value to the F critical value. If the F value exceeds the F critical value, it concludes "Null hypothesis is rejected"; otherwise, it states "Failed to reject the null hypothesis".

Applying One-Hot Encoding to Categorical Column

The following code implements one-hot encoding on the Team column in df_clean and constructs a new DataFrame one_hot_df with all teams represented in binary format as separate columns for each team.

STEP 10:

Training Data Preparation and Train/Test Split

The code initializes the feature matrix X using selected columns of one_hot_df, which consist of the one-hot coded team columns, and sets y as the dependent variables (Points). The data is further divided into training and testing sets in an 80/20 ratio (i.e. using train_test_split) and respective dimensions of each split are provided for validation purposes. This arrangement of data is done to facilitate the training and testing of the model.

Imputing Missing Values in Training Data Using LightGBM

This code employs IterativeImputer with LightGBMRegressor as an estimator to perform missing value treatment on X_train. Predicting missing values from other features is performed in a loop, which is set by max_iter=15. Then, the filled matrix is reconstructed to a DataFrame, X_train_clean, with original column names to provide a full and clear training set for the next step of modeling.

Fitting an Ordinary Least Squares (OLS) Model on Cleaned Training Data

In this code, an OLS regression model is fitted using statsmodels to study the relationship between the target variable y_train and the features X_train_clean. The indices of y_train and X_train_clean are reset to avoid misalignment during modeling. Model statistics in the result which include coefficients, p-values, and goodness of fit measurements are shown in result.summary, hence giving information about the importance of the features and the effectiveness of the model.

Adding a Constant Term and Fitting an OLS Model

This function modifies the variable X_train_clean by adding a constant term using sm.add_constant(), creating X_const in the process. The OLS regression model is then re-estimated on the new data, which includes X_const and y_train and result.summary() returns detailed information including coefficients, p-values, and fit statistics for a detailed assessment of the model and its features.

Imputing Missing Values in the Test Set

This code applies the previously trained imputer to fill in missing values in X_test, creating a new DataFrame test, with the imputed values.

Preparing Test Data for Prediction

This piece of code adjusts the column names of the test set test under the training set X_train_clean, allowing model predictions to be made without differences in the naming of the attributes. Then a constant term is also added to the test, and the fitted result model is used to make predictions based on provided features, keeping the forecasts in res. This action brings the evaluation of the model on the test set to a final.

Evaluating Model Performance Using Regression Metrics

The code determines the accuracy of the prediction of the model with the use of four metrics: MSE (mean square error), RMSE (a measure of how different from the real values are the predicted ones), R-squared (a metric of how much variation is explained by the regression), and MAE (mean absolute error). Such evaluation metrics give an overall assessment of the effectiveness of the model when used on the test data.

STEP 11:

Visualization of Polynomial-fitting Data by Graphing Quadratic and Cubic Data.

This code runs and presents two different forms of data points: a quadratic (y \= x^2) and a cubic (y_cubic \= x^3 + 2x^2), both of which are appended with noise to mimic the real-world scenario. It also plots these data points and provides appropriate legends to identify the quadratic and cubic graphs. The legends, the titles of the axes, and the title of the graph help in understanding the polynomial relationship, making it a good case for polynomial regression.

Creating a Plot to Compare Linear, Quadratic, and Cubic Polynomial Fits

The function serves the purpose of visualizing a polynomial fit by plotting the linear, quadratic, and cubic models over the data points. For each polynomial degree, Convinces all over the data, embeds the fitting lines in the appropriate places on the Graph, and gives an appropriate title so that one can see how each of the models fits the trend in the data.

This code call uses the create_polynomial_plot() function to compare polynomial fits of the WI feature from X_train_clean against the target variable y_train.

Using Polynomial Feature Transformation and Applying to the Training Set

The presented code generates second-degree polynomial features for a given train matrix X_train_clean using the PolynomialFeatures class. This transformation (fit_transform) increases the dimension of X_train_clean by squaring feature values and adding interaction features into the matrix, which is saved as X_poly. By applying poly to X_poly and y_train the model is capable of capturing any effectively present non-linearity aiding the model in making the right polynomic predictions given the data.

Training a Linear Regression Model on Polynomial Features

This code initializes a linear regression model (lm) and fits it to the polynomial-transformed training data (X_poly) and target variable (y_train)

Evaluating the Polynomial Regression Model with R² Score

This code generates predictions on the test data (test) by transforming it with the same polynomial feature expansion as used in training. The R² score is then calculated to assess how well the model's predictions match the actual values in y_test.

Calculating Root Mean Squared Error (RMSE) for Model Evaluation

This code computes the Root Mean Squared Error (RMSE) between the actual values (y_test) and the model’s predictions.

Turning Predictions into Binarized Classes and Computing Classification Metrics

The following code applies a threshold (0.5), which allows continuous predictions to be rolled into binary classes: predictions greater than or equal to 0.5 are classified as 1, and those below as 0. From this transformation, predictions_classes is generated, which is the output that is used in the classification_report against y_test. The report includes precision, recall, and F1-score measures which are used to analyze the performance of the model in classification problems within binary classes.

Conclusion

In this project, we were able to utilize the polynomial regression method in establishing as well as predicting non-linear relations within a dataset. Extent polynomial features helped to capture quadratic and other curves that are not detectable by simple linear models. To take care of missing data we used the LightGBM method for imputation. Lastly, we have complete data for training the model. Parameters like model accuracy or the generalization capabilities of the model with a view to Root Mean Squared Error (RMSE) and classification report were used to justify the accuracy of the proposed model.

Thus we considered polynomial regression as an answer to the problem of enhancing predictive accuracy on non-linear data and discussed its possible applications among the industries where accurate predictions are crucial including finance, healthcare, and engineering. The flexibility of the method used in this project to analyze structured data made it ideal to build a stronger case using the basic principles of regression analysis. In this way, polynomial regression can be prepared as an effective objective for data scientists who are looking for accurate and comprehensible analytical models when examining sophisticated data.

Challenges and Solution

Problem: Missing Data Handling

• Solution: Implement advanced imputation techniques like Iterative Imputer with LightGBM to fill in missing values. This maintains data integrity and improves model performance.

Problem: Overfitting with High-Degree Polynomials

• Solution: When the degree is high, there is a tendency to fit noise and not interesting phenomena, thus the model tends to overfit. Experiment with different degrees of polynomial features and apply cross-validation to choose the number of terms that introduce the least variance.

Problem: Unbalanced Data for Classification

• Solution: If there are more instances of one class than the other, it can skew model performance. Apply resampling techniques or use evaluation metrics like precision, recall, and F1-score to assess model performance fairly.

Problem: Large Prediction Errors on New Data

• Solution: If the model is too specialized for the training data, it may not generalize well to unseen data. Follow the test data track Root Mean Squared Error (RMSE) and see if you can fit the data using either Ridge or Lasso regression to minimize prediction errors.

Problem: Polynomial Features Transformation of Complex Data

• Solution: However, creating polynomial features can be very computationally expensive and would create a huge feature set leading to slower training. The feature size should be limited by the expansion of polynomials to relevant features or the usage of dimensionality reduction techniques such as Principal Component Analysis (PCA) to reduce feature size and improve the model’s efficiency.

FAQ

1. What is polynomial regression and why do we use it?

Polynomial regression is one of the regression analysis methods, where the regression equation is expressed as an nth degree polynomial in the input variables. This model deals with non-linear data. Thus, it applies in cases where the data cannot be fitted into a straight line, examples include complicated financial patterns and even biological growth cycles.

2. What do I do when I have missing data in a polynomial regression project?

Imputation techniques make missing data workable. For this, you’d just use an Iterative Imputer with LightGBM, predict, and fill the empty values. It keeps data quality and improves model accuracy by using existing patterns present in the data.

3. How to determine the ideal degree for polynomial regression?

The choice of a suitable polynomial degree depends also on data and model performance. The higher degrees will have more complex patterns to capture that will more likely lead to overfitting. Test different polynomial degrees with the use of cross-validation and select a degree that maximizes accuracy and minimizes generalizability.

4, What is the way to look at the performance of a polynomial regression model?

Evaluation metrics for the polynomial regression model include R² score (explained variance), Root Mean Squared Error (RMSE) (average prediction error), and Mean Absolute Error (MAE). This helps to know how good the model is in fitting the data and also how well the model predicts new data points.

5. Can polynomial regression be used to solve classification problems?

It is possible to turn polynomial regression into classification by applying a threshold to the predictions that will transform them into binary classes. Next, after generating the predictions, use a threshold (e.g., 0.5) to evaluate the model using classification metrics such as precision, recall, and F1-score to evaluate the model’s classification performance.