ISLR on Amit Rajan

Unsupervised Learning: Applied Exercises

Thu, 19 Jul 2018 06:25:52 +0100

Applied

Q7. In the chapter, we mentioned the use of correlation-based distance and Euclidean distance as dissimilarity measures for hierarchical clustering. It turns out that these two measures are almost equivalent: if each observation has been centered to have mean zero and standard deviation one, and if we let \(r_{ij}\) denote the correlation between the ith and jth observations, then the quantity \(1−r _{ij}\) is proportional to the squared Euclidean distance between the ith and jth observations. On the USArrests data, show that this proportionality holds.

Unsupervised Learning: Conceptual Exercises

Sun, 15 Jul 2018 03:51:27 +0100

10.7 Exercises

Conceptual

Q1. This problem involves the K-means clustering algorithm.

(a) Prove (10.12).

Sol: Equation 10.12 is:

\[\frac{1}{|C_k|} \sum _{i, i^{'} \in C_k} \sum _{j=1}^{p} (x _{ij} - x _{i^{'}j})^2 = 2\sum _{i \in C_k} \sum _{j=1}^{p} (x _{ij} - \bar{x} _{kj})^2\]

where \(\bar{x} _{kj} = \frac{1}{|C_k|} \sum _{i \in C_k} x _{ij}\), is the mean of feature \(j\) in cluster \(C_k\). Expanding LHS, we get

\[\frac{1}{|C_k|} \sum _{i, i^{'} \in C_k} \sum _{j=1}^{p} (x _{ij} - x _{i^{'}j})^2 = \frac{1}{|C_k|} \sum _{i, i^{'} \in C_k} \sum _{j=1}^{p} x _{ij}^2 + \frac{1}{|C_k|} \sum _{i, i^{'} \in C_k} \sum _{j=1}^{p} x _{i^{'}j}^2 - \frac{2}{|C_k|} \sum _{i, i^{'} \in C_k} \sum _{j=1}^{p} x _{ij}x _{i^{'}j} \\ = 2 \sum _{i \in C_k} \sum _{j=1}^{p} x _{ij}^2 - \frac{2}{|C_k|} \sum _{i, i^{'} \in C_k} \sum _{j=1}^{p} x _{ij}x _{i^{'}j} \]

Expanding RHS and substituting the value of \(\bar{x} _{kj}\), we get

Hierarchical Clustering

Thu, 12 Jul 2018 13:01:01 +0100

10.3.2 Hierarchical Clustering

K-means clustering has a disadvantage that there is a need to pre-specify the number of clusters \(K\). Hierarchical clutsring is an alternative approach which is free from this problem which results in an altarnative tree-based representation of the observations, called as dendrogram.

The most common technique used for hierarchical clustering is bottom-up or agglomerative clustering. It is based on the fact that the dendrogram (generally depicted as an upside-down tree) is built starting from leaves and combining the clusters up to the trunk.

K-Means Clustering

Mon, 09 Jul 2018 17:09:41 +0100

10.3 Clustering Methods

Clustering is a technique for finding subgroups or clusters in a data set based on similarity between individual observations. For clustering, we need to define the measure of similarity which depends on the knowledge of the data set. Two best known clustering methods are K-means clustering and hierarchical clustering. In K-means clustering, we partition the observations into a pre-defined number of clusters. In hierarchical clustering, the number of clusters is unknown and the results of clustering is represented as a dendrogram, which is a tree-like visualization technique that allows us to view the clustering results for various number of clusters (from 1 to \(n\)).

Principal Components Analysis: More on PCA

Fri, 06 Jul 2018 01:19:52 +0100

10.2.3 More on PCA

Scaling the Variables

The results of PCA also depend on the fact that whether the variables are individually scaled or not. If we perform PCA on the unscaled variables, the variables with higher variance will have very large loading. As it is undesirable for the principal components obtained to depend on the scale of the variables, we scale each variables to have the standard deviation 1 before performing PCA. If the individual variables are measured in the same unit, the scaling need not to be done.

Principal Components Analysis

Wed, 04 Jul 2018 03:09:02 +0100

Unsupervised Learning

10.1 The Challenge of Unsupervised Learning

Unsupervised learning is often performed as a a part of an exploratory data analysis. In unsupervised learning there is no practical way to assess the performance of the model as we do not have the true answer.

10.2 Principal Components Analysis

In the case of a large set of correlated variables, principal component analysis helps in summarizing the data set with a small number of representative variables that explains most of the variability in the data. The principal component directions are the directions in the feature space along which the original data is highly variable. Principal component analysis (PCA) refers to the process by which principal components are computed.

Support Vector Machines: Applied Exercises

Fri, 29 Jun 2018 06:19:12 +0100

Applied

Q4. Generate a simulated two-class data set with 100 observations and two features in which there is a visible but non-linear separation between the two classes. Show that in this setting, a support vector machine with a polynomial kernel (with degree greater than 1) or a radial kernel will outperform a support vector classifier on the training data. Which technique performs best on the test data? Make plots and report training and test error rates in order to back up your assertions.

Support Vector Machines: Conceptual Exercises

Tue, 26 Jun 2018 03:29:17 +0100

9.7 Exercises

Conceptual

Q1. This problem involves hyperplanes in two dimensions.

(a) Sketch the hyperplane 1 + 3X1 − X2 = 0. Indicate the set of points for which 1 + 3X1 − X2 > 0, as well as the set of points for which 1 + 3X1 − X2 < 0.

(b) On the same plot, sketch the hyperplane −2 + X1 + 2X2 = 0. Indicate the set of points for which −2+ X1 +2X2 > 0, as well as the set of points for which −2+ X1 + 2X2 < 0.

Support Vector Machines and Kernels

Sat, 23 Jun 2018 04:19:37 +0100

9.3 Support Vector Machines

9.3.1 Classification with Non-linear Decision Boundaries

Support vector classifiers, which are designed to work in the setting of linear decision boundary, can be extended to handle the case of non-linear decision boundary by enlarging the feature space using polynomial transformation of the predictors. For example, if we have a \(p\)-dimensional feature space given as: \(X_1, X_2, ..., X_p\), we could instead fit a support vector classifier using \(2p\) features: \(X_1, X_1^2, X_2, X_2^2, ..., X_p, X_p^2\). In the transformed feature space, the decision boundary is still linear, but if we consider the original feature space, the decison boundary will be non-linear. There are many ways to enlarge the feature space, such as using higher order polynomials or using interaction terms.

Support Vector Classifiers

Wed, 20 Jun 2018 02:24:24 +0100

9.2 Support Vector Classifiers

9.2.1 Overview of the Support Vector Classifier

The maximal margin classifiers can be sensitive to individual observations. Sometimes, adding a single observation in the data set, can lead to dramatic change in the separating hyperplane. The sensitivity and the low margin for a maximal margin classifier may suggest that the maximal margin classifier has overfit the training data. So, sometimes we may be willing to consider a classifier based on a hyperplane that does not perfectly separate the two classes, and hence, will be more robust (or less sensitive to individual observations) and will give better results for the unseen data points.

Maximal Margin Classifier

Tue, 19 Jun 2018 12:14:23 +0100

Support Vector Machines

Support vector machine is a generalization of a simple and intutive classifier called the maximal margin classifier. Maximal margin classifier has a limitation that it can be only applied to a data set whose classes are seperated by a linaer boundary.

9.1 Maximal Margin Classifier

9.1.1 What Is a Hyperplane?

In a \(p\)-dimensional space, a hyperplane is a flat affine subspace of \(p-1\) dimensions. In two dimensions, the hyperplane is a line and can be given as:

Tree-Based Methods: Applied Exercises

Sat, 16 Jun 2018 01:34:53 +0100

Applied

Q 7. In the lab, we applied random forests to the Boston data using mtry=6 and using ntree=25 and ntree=500. Create a plot displaying the test error resulting from random forests on this data set for a more comprehensive range of values for mtry and ntree. You can model your plot after Figure 8.10. Describe the results obtained.

import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from math import sqrt

boston = pd.read_csv("data/Boston.csv")
boston.dropna(inplace=True)

def random_forest_ntree(X_train, Y_train, X_test, Y_test, nTrees, max_feature):
 test_MSE = {}
 for nTree in nTrees:
 regr = RandomForestRegressor(max_features=max_feature, n_estimators=nTree)
 regr.fit(X_train, Y_train)
 p = regr.predict(X_test)
 test_MSE[nTree] = mean_squared_error(p, Y_test)
 return test_MSE

np.random.seed(5)
predictors = 13

X_train, X_test, y_train, y_test = train_test_split(boston.loc[:, boston.columns != 'medv'],
 boston[['medv']], test_size=0.1)

test_MSE_p = random_forest_ntree(X_train, y_train.values.ravel(), X_test, y_test.values.ravel(), np.arange(1,300,5)
 , predictors)
test_MSE_pby2 = random_forest_ntree(X_train, y_train.values.ravel(), X_test, y_test.values.ravel(), np.arange(1,300,5)
 , int(predictors/2))
test_MSE_psqrt = random_forest_ntree(X_train, y_train.values.ravel(), X_test, y_test.values.ravel(), np.arange(1,300,5)
 , int(sqrt(predictors)))

fig = plt.figure(figsize=(15,8))
ax = fig.add_subplot(111)

lists = sorted(test_MSE_p.items())
x, y = zip(*lists)
plt.plot(x, y, color='r', label='m = p')

lists = sorted(test_MSE_pby2.items())
x, y = zip(*lists)
plt.plot(x, y, color='g', label='m = p/2')

lists = sorted(test_MSE_psqrt.items())
x, y = zip(*lists)
plt.plot(x, y, color='b', label='m = sqrt(p)')

ax.set_xlabel('Number of Trees')
ax.set_ylabel('Test MSE')
ax.set_title('Test MSE vs Number of Trees')

plt.grid(b=True)
plt.legend()
plt.show()

Q 8. In the lab, a classification tree was applied to the Carseats data set after converting Sales into a qualitative response variable. Now we will seek to predict Sales using regression trees and related approaches, treating the response as a quantitative variable.

Tree-Based Methods: Conceptual Exercises

Thu, 14 Jun 2018 07:14:13 +0100

8.4 Exercises

Conceptual

Q2. It is mentioned in Section 8.2.3 that boosting using depth-one trees (or stumps) leads to an additive model: that is, a model of the form

\[f(X) = \sum_{j=1}^{p} f_j(X_j)\]

Explain why this is the case. You can begin with (8.12) in Algorithm 8.2.

Sol: As for depth-one trees, value of \(d\) is 1. Each tree is generated by splitting the data on only one predictor and the final model is formed by adding the shrunken version of them repeatedly. Hence, in the final model:

Bagging, Random Forests, Boosting

Wed, 13 Jun 2018 13:04:28 +0100

8.2 Bagging, Random Forests, Boosting

8.2.1 Bagging

The decision trees discussed above suffers from a problem of high variance. Bootstrap aggregation or bagging is a procedure that reduces the variance of a statistical learning method.

Give a set of \(n\) independent observation sets \(Z_1, Z_2, ..., Z_n\), each with variance \(\sigma^2\), the variance of the mean \(\bar{Z}\) is given by \(\sigma^2/n\), i.e. averaging a set of observations reduces variance. Hence, a natural way to reduce the variance of a statistical model is to take many training samples from the population, fit individual models on them, and give the average of them as the final model. Hence, we can calculate \(\widehat{f}^1(x), \widehat{f}^2(x), ..., \widehat{f}^B(x)\) using \(B\) seperate training sets, and finally obtain the low variance statistical model as:

Decision Trees

Mon, 11 Jun 2018 17:07:13 +0100

Tree-Based Methods

Tree-based methods stratify or segment the predictor space into a number of simple regions. To make a prediction, we use the mean or the mode of the training observations in the region in which the observation to be predicted belongs. The set of splitting rules can be summarized via a tree, these methods are also known as decision tree methods. Bagging, random forests and boosting produce multiple trees and then combine them in a single model to make the prediction. They provide improved accuracy at the cost of interpretability.

Moving Beyond Linearity: Applied Exercises

Fri, 08 Jun 2018 03:17:36 +0100

Applied

Q6. In this exercise, you will further analyze the Wage data set considered throughout this chapter.

(a) Perform polynomial regression to predict wage using age. Use cross-validation to select the optimal degree d for the polynomial. What degree was chosen, and how does this compare to the results of hypothesis testing using ANOVA? Make a plot of the resulting polynomial fit to the data.

Sol: The optimal degree of polynomial selected from cross-validation is 4. From the ANOVA of models of degree 1 to 5, it is found that the models of degree 2 and 3 are significant. The test MSE shows the lowest value for degree 3 model as well.

Moving Beyond Linearity: Conceptual Exercises

Wed, 06 Jun 2018 06:27:22 +0100

7.9 Exercises

Conceptual

Q1. It was mentioned in the chapter that a cubic regression spline with one knot at ξ can be obtained using a basis of the form \(x, x^2, x^3, (x − ξ)^3 _+\), where \((x − ξ)^3 _+ = (x − ξ)^3\) if x > ξ and equals 0 otherwise. We will now show that a function of the form \(f(x) = β_0 + β_1x + β_2x^2 + β_3x^3 + β_4(x − ξ)^3 _+\) is indeed a cubic regression spline, regardless of the values of \(β_0, β_1, β_2, β_3, β_4\).

Local Regression, Generalized Additive Models

Mon, 04 Jun 2018 16:12:19 +0100

7.6 Local Regression

Local regression comutes the fit at a target point \(x_0\) using only the nearby training observstions. The algorithm for local regression is as follows:

Gather the \(k\) points closest to \(x_0\).
Assign a weight \(K_{i0} = K(x_i, x_0)\) to all the points in the neighborhood such that the points that are farthest have lower weights. All the points except from these \(k\) nearest neighbors have weigth 0.
Fit a weighted least squares regression of the aformentioned points using weights, by finding \(\beta_0, \beta_1\) that minimize

\[\sum _{i=1}^{n}K _{i0}(y_i - \beta_0 - \beta_1 x_i)^2\]

The fitted value at \(x_0\) is given as \(\widehat{f}(x_0) = \widehat{\beta_0} + \widehat{\beta_1} x_0\).

The span s of a local regression is defined as \(s = \frac{k}{n}\), where \(n\) is total number of training samples. It plays the role of controling the flexibility of the non-linera fit. Smaller the value of \(s\), the more local or wiggly is the fit. For larger values of \(s\), we obtain a global fit. An appropriate value of \(s\) can be chosen by cross-validation.

Smoothing Splines

Wed, 30 May 2018 06:02:06 +0100

7.5 Smoothing Splines

7.5.1 An Overview of Smoothing Splines

Regression splines are created by specifying a set of knots, producing a sequence of basis functions and then estimate spline coefficients using least squares.

To fit a smooth curve to a data set, we need to find a function \(g(x)\) such that \(RSS = \sum_{i=1}^{n}(y_i - g(x_i))^2\) is minimum. If we do not put any constraint on \(g(x)\), we can always find a function \(g(x)\), which will make RSS 0. This function will be too flexible and will overfit the data. Hence, we need to find a function \(g\) which makes RSS small and which is smooth as well.

Regression Splines

Mon, 28 May 2018 11:12:18 +0100

7.4 Regression Splines

Regression splines are flixible class of basis functions that extend upon polynomial and piecewise constant regression approaches.

7.4.1 Piecewise Polynomials

Piecewise polynomial regression fits separate low-degree polynomials over different regions of \(X\). For example, a piecewise squared polynomial fits squared regression model of the form

\[y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \epsilon_i\]

where the coefficients \(\beta_0, \beta_1, \beta_2\) differs in different parts of the range of \(X\). The points where the coefficients change are called knots. Each of the polynomial functions can be fit using least square methods. Increasing the number of knots will give a more flexible piecewise polynomial.

Polynomial Regression, Step Functions, Basis Functions

Mon, 28 May 2018 04:22:38 +0100

Moving Beyond Linearity

Lineaer models have its limitations in terms of predictive power. Linear models can be extended simply as:

Polynomial regression extends linear regression by adding extra higher order predictors (predictors rasied to higher order powers).
Step functions cut the range of a variable into \(K\) distinct regions in order to produce a qualitative variable.
Regression splines is the extension of polynomial regression and step functions. It divides the range of predictor \(X\) into \(K\) distinct regions and within each region a polynomial function is fit to the data.
Smoothing splines
Local regression
Generalized additive models

Linear Model Selection and Regularization: Applied Exercises

Sat, 26 May 2018 11:08:14 +0100

Applied

Q8. In this exercise, we will generate simulated data, and will then use this data to perform best subset selection.

(a) Use the rnorm() function to generate a predictor X of length n = 100, as well as a noise vector of length n = 100.

import numpy as np

np.random.seed(5)
X = np.random.normal(0, 1, 100)
e = np.random.normal(0, 1, 100)

(b) Generate a response vector Y of length n = 100 according to the model \(Y = β_0 + β_1X + β_2X^2 + β_3X^3 + \epsilon\), where \(β_0, β_1, β_2,\) and \(β_3\) are constants of your choice.

Linear Model Selection and Regularization: Conceptual Exercises

Fri, 25 May 2018 06:18:38 +0100

6.8 Exercises

Conceptual

Q1. We perform best subset, forward stepwise, and backward stepwise selection on a single data set. For each approach, we obtain p + 1 models, containing 0, 1, 2, . . . , p predictors. Explain your answers:

(a) Which of the three models with k predictors has the smallest training RSS?

Sol: Training RSS is minimum for best subset selection.

(b) Which of the three models with k predictors has the smallest test RSS?

Dimension Reduction Methods

Thu, 24 May 2018 01:06:16 +0100

6.3 Dimension Reduction Methods

Instead of performing a least squares regression on all the \(p\) predictors, we can transform the predictors and then fit a least squares model on the transformed variables. Let the transformed variables be \(Z_1, Z_2, ..., Z_M\), where \(M < p\), where each of \(Z_m\)s is a linear combination of predictors \(X_1, X_2, ..., X_p\).i.e.

\[Z_m = \sum _{j=1}^{p} \phi _{jm}X_j\]

We can then fit the least squares regression model as \(Z_m\)s as the predictors:

Shrinkage Methods

Tue, 22 May 2018 21:16:20 +0100

6.2 Shrinkage Methods

As an alternative to subset selection methods, a model containing all the \(p\) predictors can be fit using a technique that constrains or regularizes the coefficient estimates (or shrinks the coefficeint estimates towards 0). Two best known techniques for shrinking the coefficient estimates towards 0 are: ridge regression and the lasso.

6.2.1 Ridge Regression

In a least squares fitting, the parameters of the model is estimated by minimizing

Subset Selection

Mon, 21 May 2018 13:08:40 +0100

Linear Model Selection and Regularization

Befor moving to non-linear models, there are certain other fitting procedures through which a plain linear model can be improved. These alternate fitting procedures can yield better prediction accuracy and model interpretability.

Prediction Accuracy: Provided that the relationship between predictors and response is linear, the least square estimates will have low bias. If n » p, this model will have low variance as well. If n is not much larger than p, the model will have a lot of variability, resulting in overfitting and poor prediction accuracy. If p > n, the variance is almost infinite and hence the plain least square method can not be used at all.
Model Interpretability: There may be the case that some of the predictors used in the linear model are not significant. Including such irrelevant variables may lead to unnecessary complexity in the model. These variables can be excluded from the model by feature selection or variable selection and hence a better model interpretability.

Resampling Methods: Applied Exercises

Sat, 19 May 2018 13:08:40 +0100

Applied

Q5. In Chapter 4, we used logistic regression to predict the probability of default using income and balance on the Default data set. We will now estimate the test error of this logistic regression model using the validation set approach. Do not forget to set a random seed before beginning your analysis.

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
import statsmodels.api as sm

np.random.seed(1)
default = pd.read_excel("data/Default.xlsx")
default['student'] = default['student'].map({'Yes': 1, 'No': 0})
default['default'] = default['default'].map({'Yes': 1, 'No': 0})

(a) Fit a logistic regression model that uses income and balance to predict default.

Resampling Methods: Conceptual Exercises

Fri, 18 May 2018 07:18:30 +0100

5.4 Exercises

Conceptual

Q1. Using basic statistical properties of the variance, as well as singlevariable calculus, derive that the value of \(\alpha\) which minimizes \(Var(\alpha X + (1 - \alpha) Y)\) is:

\[\alpha = \frac{\sigma_Y^2 - \sigma _{XY}}{\sigma_X^2 + \sigma_Y^2 - 2\sigma _{XY}}\]

Sol: As we know that \(Var(aX + bY) = a^2 Var(X) + b^2 Var(Y) + 2abCov(X, Y)\), the above quantity (that needs to be minimized) can be transformed as:

The Bootstrap

Fri, 18 May 2018 02:26:28 +0100

5.2 The Bootstrap

Bootstrap can be used to to quantify the uncertainty associated with a given statistical model. For example, bootstrap can be used to estimate standard errors (which measures the uncertainty) of the coefficients from a linear regression fit. Bootstrap can be applied to a wide range of statistical learning methods. The method of bootstrap is explained below via an example:

Suppose we wish to invest money in two financial assests which yield returns of \(X\) and \(Y\). We will invest fraction \(\alpha\) of our money in \(X\) and \(1-\alpha\) in \(Y\). As there is a variablility associated with these returns, we wish to choose \(\alpha\) which minimizes the total risk or variance of our investment. i.e. We want to minimize \(Var(\alpha X + (1-\alpha) Y)\). The value that minimizes this is given as:

Cross-Validation

Thu, 17 May 2018 05:22:19 +0100

Resampling Methods

Resampling methods involve repeatedly drawing samples from training data and refitting a model on them. Resampling approaches can be computationally expensive. Cross-validation and bootstrap are two of the most commonly used resampling methods. Cross-validation can be used to estimate the test error rate associated with a given model in order to evaluate its performance. The process of evaluating a model’s performance is called model assessment. The process of selecting proper level of flexibility for a model is known as model selection.

Classification: Applied Exercises

Wed, 16 May 2018 10:20:15 +0100

Applied

Q10. This question should be answered using the Weekly data set.

(a) Produce some numerical and graphical summaries of the Weekly data. Do there appear to be any patterns?

import seaborn as sns
weekly = pd.read_csv("data/Weekly.csv")

sns.pairplot(weekly, vars=['Lag1', 'Lag2', 'Lag3', 'Lag4', 'Lag5', 'Volume'], hue='Direction')

(b) Use the full data set to perform a logistic regression with Direction as the response and the five lag variables plus Volume as predictors. Use the summary function to print the results. Do any of the predictors appear to be statistically significant? If so, which ones?

Classification: Conceptual Exercises

Tue, 15 May 2018 09:02:08 +0100

4.7 Exercises

Conceptual

Q1. Using a little bit of algebra, prove that the logistic function representation and logit representation for the logistic regression model are equivalent.

Sol: Logistic function representation is given as:

\[p(X) = \frac{e^{\beta_0 + \beta_1X}}{1 + e^{\beta_0 + \beta_1X}}\]

then \(1-p(X) = \frac{1}{1 + e^{\beta_0 + \beta_1X}}\), Taking the ratio of these two and then taking the log, we get

\[log\bigg( \frac{p(X)}{1-p(X)} \bigg) = \beta_0 + \beta_1X\]

Q2. Under the assumption that the observations in the \(k\)th class is drawn from a normal distribution, the Bayes’ classifier assign an observation to the class for which the discriminant function is maximized.

Linear Discriminant Analysis

Mon, 14 May 2018 11:12:28 +0100

4.4 Linear Discriminant Analysis

In logistic regression, we model the the conditional distribution of response \(Y\) given the predictors \(X\). As an alternative approach, we model the distribution of predictors X seperately for each of the response classe. We then use Bayes’ Theorem to flip these around into estimates for Pr(Y = k | X = x). When these distributions are assumed to be normal, this model is very similar to logistic regression. The advantage of LDA is as follows:

Logistic Regression

Sun, 13 May 2018 02:17:58 +0100

Classification

A process for predicting qualitative or categorical variables is called as Classification.

4.1 An Overview of Classification

The dataset used in this chapter will be Default dataset. We will predict that whether an individual will default on his/her credit card payment on the basis of annual income and monthly credit card balance. The data is displayed below:

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

default = pd.read_excel("data/Default.xlsx")

markers = {"Yes": "x", "No": "+"}
palette = {"Yes": "red", "No": "green"}
fig = plt.figure(figsize=(15,8))
ax = fig.add_subplot(111)
sns.scatterplot(x="balance", y="income", hue="default", style="default", markers=markers, palette=palette,
 alpha=0.6, data=default)
ax.set_xlabel('Balance')
ax.set_ylabel('Income')
ax.set_title('Scatter Plot')

fig = plt.figure(figsize=(15,8))

ax = fig.add_subplot(121)
sns.boxplot(x="default", y="balance", data=default)
ax.set_xlabel('Default')
ax.set_ylabel('Balance')
ax.set_title('Box Plot')

ax = fig.add_subplot(122)
sns.boxplot(x="default", y="income", data=default)
ax.set_xlabel('Default')
ax.set_ylabel('Income')
ax.set_title('Box Plot')


plt.show()

4.2 Why Not Linear Regression?

In general, there is no natural way to convert a qualitative response variable with more than two levels into a quantitative response that is ready for linear regression. For the binary qualitative response, we can simply encode the variables as 0 and 1 and predict the values taking 0.5 as threshold.

Linear Regression: Applied Exercises

Fri, 11 May 2018 05:18:18 +0100

Applied

Solution 8:

(a) Perform linear regression on auto data with mpg as response and horsepower as the predictor and display the summary results.

import statsmodels.api as sm

X = sm.add_constant(auto[['horsepower']], prepend=True)
model = sm.OLS(auto['mpg'], X)
result = model.fit()
print(result.summary())
print("Prediction for horsepower 98: " +str(result.predict([1, 98])))
print("95% CI: " +str(result.conf_int(alpha=0.05, cols=None)))

 OLS Regression Results
==============================================================================
Dep. Variable: mpg R-squared: 0.606
Model: OLS Adj. R-squared: 0.605
Method: Least Squares F-statistic: 599.7
Date: Thu, 06 Sep 2018 Prob (F-statistic): 7.03e-81
Time: 21:37:43 Log-Likelihood: -1178.7
No. Observations: 392 AIC: 2361.
Df Residuals: 390 BIC: 2369.
Df Model: 1
Covariance Type: nonrobust
==============================================================================
 coef std err t P>|t| [0.025 0.975]
------------------------------------------------------------------------------
const 39.9359 0.717 55.660 0.000 38.525 41.347
horsepower -0.1578 0.006 -24.489 0.000 -0.171 -0.145
==============================================================================
Omnibus: 16.432 Durbin-Watson: 0.920
Prob(Omnibus): 0.000 Jarque-Bera (JB): 17.305
Skew: 0.492 Prob(JB): 0.000175
Kurtosis: 3.299 Cond. No. 322.
==============================================================================

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
Prediction for horsepower 98: [24.46707715]
95% CI: 0 1
const 38.525212 41.346510
horsepower -0.170517 -0.145172

i. Is there a relationship between the predictor and the response?

Linear Regression: Conceptual Exercises

Thu, 10 May 2018 10:28:18 +0100

3.7 Exercises

Conceptual

Solution: The linear fit can be given as:

\[50 + (20 \times GPA) + (0.07 \times IQ) + (35 \times GENDER) + (0.01 \times GPA \times IQ) - (10 \times GPA \times GENDER)\]

(a) For a fixed value of IQ and GPA, the average salary for male will be \(50 + 20 \times GPA\) and for the female it will be \(85 + 20 \times GPA - 10 \times GPA\). For the salary of male to be higher, \(50 + 20GPA > 85 + 10GPA\), i.e. \(GPA > 3.5\).

Other Considerations in the Regression Model

Wed, 09 May 2018 11:11:48 +0100

3.3 Other Considerations in the Regression Model

3.3.1 Qualitative Predictors

There can be a case when predictor variables can be qualitative.

Predictors with Only Two Levels

For the predictors with only two values, we can create an indicator or dummy variable with values 0 and 1 and use it in the regression model. The final prediction will not depend on the coding scheme. Only difference will be in the model coefficients and the way they are interpreted.

Multiple Linear Regression

Tue, 08 May 2018 06:12:03 +0100

3.2 Multiple Linear Regression

In general, suppose we have \(p\) distinct predictors, the multiple linear regression takes the form:

\[Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + ... + \beta_p X_p + \epsilon\]

where \(\beta_j\) can be interpreted as the average effect on \(Y\) of a one unit increase in \(X_j\), holding all other predictors fixed.

3.2.1 Estimating the Regression Coefficients

Given the estimates, \(\widehat{\beta_0}, \widehat{\beta_1},..., \widehat{\beta_p}\), predictions can be done as:

Simple Linear Regression

Tue, 08 May 2018 02:18:09 +0100

Linear Regression

Linear Regression is a useful tool for predicting a quantitative response.

3.1 Simple Linear Regression

Simple linear regression is a straightforward approach for predicting a quantitative response on the basis of a single predictor variable. Mathematically it can be written as:

\[Y \approx \beta_0 + \beta_1 X\]

\(\beta_0\) and \(\beta_1\) represent intercept and slope and are called as model coefficients or parameters. The estimated equation is given as:

\[\widehat{y} \approx \widehat{\beta_0} + \widehat{\beta_1} x\]

where hat symbol is used to denote the estimated values.

Statistical Learning: Applied Exercises

Sun, 06 May 2018 07:24:34 +0100

Applied

Q8. This exercise relates to the College data set, which can be found in the file College.csv. It contains a number of variables for 777 different universities and colleges in the US.

import pandas as pd

college = pd.read_csv("data/College.csv")
college.set_index('Unnamed: 0', drop=True, inplace=True)
college.index.names = ['Name']
college.describe()

	Apps	Accept	Enroll	Top10perc	Top25perc	F.Undergrad	P.Undergrad	Outstate	Room.Board	Books	Personal	PhD	Terminal	S.F.Ratio	perc.alumni	Expend	Grad.Rate
count	777.000000	777.000000	777.000000	777.000000	777.000000	777.000000	777.000000	777.000000	777.000000	777.000000	777.000000	777.000000	777.000000	777.000000	777.000000	777.000000	777.00000
mean	3001.638353	2018.804376	779.972973	27.558559	55.796654	3699.907336	855.298584	10440.669241	4357.526384	549.380952	1340.642214	72.660232	79.702703	14.089704	22.743887	9660.171171	65.46332
std	3870.201484	2451.113971	929.176190	17.640364	19.804778	4850.420531	1522.431887	4023.016484	1096.696416	165.105360	677.071454	16.328155	14.722359	3.958349	12.391801	5221.768440	17.17771
min	81.000000	72.000000	35.000000	1.000000	9.000000	139.000000	1.000000	2340.000000	1780.000000	96.000000	250.000000	8.000000	24.000000	2.500000	0.000000	3186.000000	10.00000
25%	776.000000	604.000000	242.000000	15.000000	41.000000	992.000000	95.000000	7320.000000	3597.000000	470.000000	850.000000	62.000000	71.000000	11.500000	13.000000	6751.000000	53.00000
50%	1558.000000	1110.000000	434.000000	23.000000	54.000000	1707.000000	353.000000	9990.000000	4200.000000	500.000000	1200.000000	75.000000	82.000000	13.600000	21.000000	8377.000000	65.00000
75%	3624.000000	2424.000000	902.000000	35.000000	69.000000	4005.000000	967.000000	12925.000000	5050.000000	600.000000	1700.000000	85.000000	92.000000	16.500000	31.000000	10830.000000	78.00000
max	48094.000000	26330.000000	6392.000000	96.000000	100.000000	31643.000000	21836.000000	21700.000000	8124.000000	2340.000000	6800.000000	103.000000	100.000000	39.800000	64.000000	56233.000000	118.00000

Statistical Learning: Conceptual Exercises

Sun, 06 May 2018 01:38:54 +0100

2.4 Exercises

Conceptual

Q1. For each of parts (a) through (d), indicate whether we would generally expect the performance of a flexible statistical learning method to be better or worse than an inflexible method. Justify your answer.

(a) The sample size n is extremely large, and the number of predic- tors p is small.

Sol: Better

(b) The number of predictors p is extremely large, and the number of observations n is small.

Assessing Model Accuracy

Sat, 05 May 2018 07:11:10 +0100

2.2 Assessing Model Accuracy

No one statistical learning method dominates all other over all possible data sets. Hence it is an important task to decide that for any given data set which model fits best.

2.2.1 Measuring the Quality of Fit

In the regression setting, the most commonly used measure for quality of fit is mean squared error (MSE), which is given as:

\[MSE = \frac{1}{n} \sum_{i=1}^{n} (y_i - \widehat{f}(x_i))^{2}\]

where \(\widehat{f}(x_i)\) is the prediction for the \(i\)th observation. MSE will be small if the predicted responses are very close to the true responses. While training a model, training MSE is of lesser significance. We should be more interested in test MSE, which is the MSE for the previously unseen test observation not used to train the model. When the test data is available, we can simply compute test MSE and select the model which has the lowest test MSE. In the absence of test data, the basic approach is to simply select a model with the lowest training MSE. Below figure shows the MSE of test and train data for a model.

What Is Statistical Learning?

Fri, 04 May 2018 09:32:21 +0100

Statistical Learning

2.1 What Is Statistical Learning?

The Advertising data set consists of the sales of that product in 200 different markets, along with advertising budgets for the product in each of those markets for three different media: TV, radio, and newspaper. The plot of data is shown below. Our goal is to develop an accurate model that can be used to predict sales on the basis of the three media budgets.

Introduction to Statistical Learning

Wed, 02 May 2018 11:14:32 +0100

Chapter 1: Introduction

Wage Data

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

wage = pd.read_csv("data/Wage.csv")
wage.loc[df['education'] == '1. < HS Grad', 'education'] = 1
wage.loc[df['education'] == '2. HS Grad', 'education'] = 2
wage.loc[df['education'] == '3. Some College', 'education'] = 3
wage.loc[df['education'] == '4. College Grad', 'education'] = 4
wage.loc[df['education'] == '5. Advanced Degree', 'education'] = 5
wage.head()

fig = plt.figure(figsize=(15,8))

ax = fig.add_subplot(131)
sns.regplot(x="age", y="wage", color='#D7DBDD', data=wage, order=2, scatter_kws={'alpha':0.5},
 line_kws={'color':'g', 'alpha':0.7})
ax.set_xlabel('Age')
ax.set_ylabel('Wage')
ax.set_title('Wage vs Age')

ax = fig.add_subplot(132)
sns.regplot(x="year", y="wage", color='#D7DBDD', data=wage, order=1, scatter_kws={'alpha':0.5},
 line_kws={'color':'g', 'alpha':0.7})
ax.set_xlabel('Year')
ax.set_ylabel('Wage')
ax.set_title('Wage vs Year')

ax = fig.add_subplot(133)
sns.boxplot(x="education", y="wage", data=wage)
ax.set_xlabel('Education')
ax.set_ylabel('Wage')
ax.set_title('Wage vs Education')

plt.show()

The Wage dataset consists of data containing income, age, year and education level of different individuals. As shown in the above figure, it can be interpreted that wage increases with increase in age upto 60 years and decreases afterwards (but deviation is significant). Wage increases linearly with the year (but deviation is significant). The people with higher education level tends to have higher wage. So ideally if we want to predict wage, we should consider the non-linear relationship between wage and age and take into consideration the effect of year and education level as well. This process involves predicting a continuous or quantitative value and is often referred to as a regression problem.