Skip to content

Latest commit

 

History

History
34 lines (29 loc) · 3.04 KB

README.md

File metadata and controls

34 lines (29 loc) · 3.04 KB

This repository about Linear Regression and Stochastic Gradient Descent

Motivation of Experiment

  • Further understand of linear regression ，closed-form solution and Stochastic gradient descent.
  • Conduct some experiments under small scale dataset.
  • Realize the process of optimization and adjusting parameters.

Dataset

Linear Regression uses Housing in LIBSVM Data, including 506 samples and each sample has 13 features. You are expected to download scaled edition. After downloading, you are supposed to divide it into training set, validation set.

Environment for Experiment

Python3, at least including following python package: sklearn，numpy，jupyter，matplotlib It is recommended to install anaconda3 directly, which has built-in python package above.

Experiment Step

closed-form solution of Linear Regression

  • Load the experiment data. You can use load_svmlight_file function in sklearn library.
  • Devide dataset. You should divide dataset into training set and validation set using train_test_split function. Test set is not required in this experiment.
  • Select a Loss function.
  • Get the formula of the closed-form solution, the process details the courseware ppt.
  • Get the value of parameter W by the closed-form solution, and update the parameter W.
  • Get the Loss, loss_train under the training set and loss_val by validating under validation set.
  • Output the value of Loss, loss_train and loss_val.

Linear Regression and Stochastic Gradient Descent

  • Load the experiment data. You can use load_svmlight_file function in sklearn library.
  • Devide dataset. You should divide dataset into training set and validation set using train_test_split function. Test set is not required in this experiment.
  • Initialize linear model parameters. You can choose to set all parameter into zero, initialize it randomly or with normal distribution.
  • Choose loss function and derivation: Find more detail in PPT.
  • Calculate G toward loss function from each sample.
  • Denote the opposite direction of gradient G as D.
  • Update model: Wt=Wt−1+ηD. η. is learning rate, a hyper-parameter that we can adjust.
  • Get the loss_train under the training set and loss_val by validating under validation set.
  • Repeate step 5 to 8 for several times, andand output the value of loss_train as well as loss_val.