Kernel: Python [conda env:py37]
In [1]:
Supervised Learning
Classification and Regression
Generalization, Overfitting, and Underfitting
Relation of Model Complexity to Dataset Size
Supervised Machine Learning Algorithms
Some Sample Datasets
In [2]:
X.shape: (26, 2)
Invalid PDF output
In [3]:
Text(0, 0.5, 'Target')
Invalid PDF output
In [4]:
cancer.keys():
dict_keys(['data', 'target', 'target_names', 'DESCR', 'feature_names', 'filename'])
In [5]:
Shape of cancer data: (569, 30)
In [6]:
Sample counts per class:
{'malignant': 212, 'benign': 357}
In [7]:
Feature names:
['mean radius' 'mean texture' 'mean perimeter' 'mean area'
'mean smoothness' 'mean compactness' 'mean concavity'
'mean concave points' 'mean symmetry' 'mean fractal dimension'
'radius error' 'texture error' 'perimeter error' 'area error'
'smoothness error' 'compactness error' 'concavity error'
'concave points error' 'symmetry error' 'fractal dimension error'
'worst radius' 'worst texture' 'worst perimeter' 'worst area'
'worst smoothness' 'worst compactness' 'worst concavity'
'worst concave points' 'worst symmetry' 'worst fractal dimension']
In [8]:
Data shape: (506, 13)
In [9]:
X.shape: (506, 104)
k-Nearest Neighbors
k-Neighbors classification
In [10]:
Invalid PDF output
In [11]:
Invalid PDF output
In [12]:
In [13]:
In [14]:
KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
metric_params=None, n_jobs=None, n_neighbors=3, p=2,
weights='uniform')
In [15]:
Test set predictions: [1 0 1 0 1 0 0]
In [16]:
Test set accuracy: 0.857
Analyzing KNeighborsClassifier
In [17]:
<matplotlib.legend.Legend at 0x7f9df3d1e908>
Invalid PDF output
In [18]:
<matplotlib.legend.Legend at 0x7f9df3c61550>
Invalid PDF output
k-neighbors regression
In [19]:
Invalid PDF output
In [20]:
Invalid PDF output
In [21]:
KNeighborsRegressor(algorithm='auto', leaf_size=30, metric='minkowski',
metric_params=None, n_jobs=None, n_neighbors=3, p=2,
weights='uniform')
In [22]:
Test set predictions:
[-0.054 0.357 1.137 -1.894 -1.139 -1.631 0.357 0.912 -0.447 -1.139]
In [23]:
Test set R^2: 0.83
Analyzing KNeighborsRegressor
In [24]:
<matplotlib.legend.Legend at 0x7f9df3fa74e0>
Invalid PDF output
Strengths, weaknesses, and parameters
Linear Models
Linear models for regression
In [25]:
w[0]: 0.393906 b: -0.031804
Invalid PDF output
Linear regression aka ordinary least squares
In [26]:
In [27]:
lr.coef_: [0.394]
lr.intercept_: -0.031804343026759746
In [28]:
Training set score: 0.67
Test set score: 0.66
In [29]:
In [30]:
Training set score: 0.95
Test set score: 0.61
Ridge regression
In [31]:
Training set score: 0.89
Test set score: 0.75
In [32]:
Training set score: 0.79
Test set score: 0.64
In [33]:
Training set score: 0.93
Test set score: 0.77
In [34]:
<matplotlib.legend.Legend at 0x7f9df22abd30>
Invalid PDF output
In [35]:
Invalid PDF output
Lasso
In [36]:
Training set score: 0.29
Test set score: 0.21
Number of features used: 4
In [37]:
Training set score: 0.90
Test set score: 0.77
Number of features used: 33
In [38]:
Training set score: 0.95
Test set score: 0.64
Number of features used: 96
In [39]:
Text(0, 0.5, 'Coefficient magnitude')
Invalid PDF output
Linear models for classification
In [40]:
/home/andy/checkout/scikit-learn/sklearn/svm/base.py:922: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
"the number of iterations.", ConvergenceWarning)
<matplotlib.legend.Legend at 0x7f9df213ecf8>
Invalid PDF output
In [41]:
Invalid PDF output
In [42]:
Training set score: 0.953
Test set score: 0.958
In [43]:
Training set score: 0.972
Test set score: 0.965
In [44]:
Training set score: 0.934
Test set score: 0.930
In [45]:
<matplotlib.legend.Legend at 0x7f9df1f88dd8>
Invalid PDF output
In [46]:
Training accuracy of l1 logreg with C=0.001: 0.91
Test accuracy of l1 logreg with C=0.001: 0.92
/home/andy/checkout/scikit-learn/sklearn/svm/base.py:922: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
"the number of iterations.", ConvergenceWarning)
Training accuracy of l1 logreg with C=1.000: 0.96
Test accuracy of l1 logreg with C=1.000: 0.96
Training accuracy of l1 logreg with C=100.000: 0.99
Test accuracy of l1 logreg with C=100.000: 0.98
<matplotlib.legend.Legend at 0x7f9df1f34048>
Invalid PDF output
Linear models for multiclass classification
In [47]:
<matplotlib.legend.Legend at 0x7f9df2173d30>
Invalid PDF output
In [48]:
Coefficient shape: (3, 2)
Intercept shape: (3,)
In [49]:
<matplotlib.legend.Legend at 0x7f9df3ea9550>
Invalid PDF output
In [50]:
Text(0, 0.5, 'Feature 1')
Invalid PDF output
Strengths, weaknesses and parameters
In [51]:
In [52]:
In [53]:
Naive Bayes Classifiers
In [54]:
In [55]:
Feature counts:
{0: array([0, 1, 0, 2]), 1: array([2, 0, 2, 1])}
Strengths, weaknesses and parameters
Decision trees
In [56]:
Invalid PDF output
Building decision trees
In [57]:
Invalid PDF output
Invalid PDF output
Invalid PDF output
Invalid PDF output
Controlling complexity of decision trees
In [58]:
Accuracy on training set: 1.000
Accuracy on test set: 0.937
In [59]:
Accuracy on training set: 0.988
Accuracy on test set: 0.951
Analyzing Decision Trees
In [60]:
In [61]:
Feature Importance in trees
In [62]:
Feature importances:
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.01 0.048
0. 0. 0.002 0. 0. 0. 0. 0. 0.727 0.046 0. 0.
0.014 0. 0.018 0.122 0.012 0. ]
In [63]:
Invalid PDF output
In [64]:
Feature importances: [0. 1.]
Invalid PDF output
In [65]:
Text(0, 0.5, 'Price in $/Mbyte')
Invalid PDF output
In [66]:
In [67]:
<matplotlib.legend.Legend at 0x7f9df1e9d5c0>
Invalid PDF output
Strengths, weaknesses and parameters
Ensembles of Decision Trees
Random forests
Building random forests
Analyzing random forests
In [68]:
RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
max_depth=None, max_features='auto', max_leaf_nodes=None,
min_impurity_decrease=0.0, min_impurity_split=None,
min_samples_leaf=1, min_samples_split=2,
min_weight_fraction_leaf=0.0, n_estimators=5, n_jobs=None,
oob_score=False, random_state=2, verbose=0, warm_start=False)
In [69]:
[<matplotlib.lines.Line2D at 0x7f9df1e78c50>,
<matplotlib.lines.Line2D at 0x7f9df1e78080>]
Invalid PDF output
In [70]:
Accuracy on training set: 1.000
Accuracy on test set: 0.972
In [71]:
Invalid PDF output
Strengths, weaknesses, and parameters
Gradient Boosted Regression Trees (Gradient Boosting Machines)
In [72]:
Accuracy on training set: 1.000
Accuracy on test set: 0.958
In [73]:
Accuracy on training set: 0.991
Accuracy on test set: 0.972
In [74]:
Accuracy on training set: 0.988
Accuracy on test set: 0.965
In [75]:
Invalid PDF output
Strengths, weaknesses and parameters
Kernelized Support Vector Machines
Linear Models and Non-linear Features
In [76]:
Text(0, 0.5, 'Feature 1')
Invalid PDF output
In [77]:
/home/andy/checkout/scikit-learn/sklearn/svm/base.py:922: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
"the number of iterations.", ConvergenceWarning)
Text(0, 0.5, 'Feature 1')
Invalid PDF output
In [78]:
Text(0.5, 0, 'feature1 ** 2')
Invalid PDF output
In [79]:
/home/andy/checkout/scikit-learn/sklearn/svm/base.py:922: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
"the number of iterations.", ConvergenceWarning)
Text(0.5, 0, 'feature1 ** 2')
Invalid PDF output
In [80]:
Text(0, 0.5, 'Feature 1')
Invalid PDF output
The Kernel Trick
Understanding SVMs
In [81]:
Text(0, 0.5, 'Feature 1')
Invalid PDF output
Tuning SVM parameters
In [82]:
<matplotlib.legend.Legend at 0x7f9df14a4d68>
Invalid PDF output
In [83]:
Accuracy on training set: 1.00
Accuracy on test set: 0.63
In [84]:
Text(0, 0.5, 'Feature magnitude')
Invalid PDF output
Preprocessing data for SVMs
In [85]:
Minimum for each feature
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.]
Maximum for each feature
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.
1. 1. 1. 1. 1. 1.]
In [86]:
In [87]:
Accuracy on training set: 0.948
Accuracy on test set: 0.951
In [88]:
Accuracy on training set: 0.988
Accuracy on test set: 0.972
Strengths, weaknesses and parameters
Neural Networks (Deep Learning)
The Neural Network Model
In [89]:
In [90]:
In [91]:
Text(0, 0.5, 'relu(x), tanh(x)')
Invalid PDF output
In [92]:
Tuning Neural Networks
In [93]:
Text(0, 0.5, 'Feature 1')
Invalid PDF output
In [94]:
Text(0, 0.5, 'Feature 1')
Invalid PDF output
In [95]:
Text(0, 0.5, 'Feature 1')
Invalid PDF output
In [96]:
Text(0, 0.5, 'Feature 1')
Invalid PDF output
In [97]:
Invalid PDF output
In [98]:
Invalid PDF output
In [99]:
Cancer data per-feature maxima:
[ 28.11 39.28 188.5 2501. 0.163 0.345 0.427 0.201
0.304 0.097 2.873 4.885 21.98 542.2 0.031 0.135
0.396 0.053 0.079 0.03 36.04 49.54 251.2 4254.
0.223 1.058 1.252 0.291 0.664 0.207]
In [100]:
Accuracy on training set: 0.94
Accuracy on test set: 0.92
In [101]:
Accuracy on training set: 0.991
Accuracy on test set: 0.965
/home/andy/checkout/scikit-learn/sklearn/neural_network/multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
% self.max_iter, ConvergenceWarning)
In [102]:
Accuracy on training set: 1.000
Accuracy on test set: 0.972
In [103]:
Accuracy on training set: 0.988
Accuracy on test set: 0.972
In [104]:
<matplotlib.colorbar.Colorbar at 0x7f9df09cd710>
Invalid PDF output
Strengths, weaknesses and parameters
Estimating complexity in neural networks
Uncertainty estimates from classifiers
In [105]:
GradientBoostingClassifier(criterion='friedman_mse', init=None,
learning_rate=0.1, loss='deviance', max_depth=3,
max_features=None, max_leaf_nodes=None,
min_impurity_decrease=0.0, min_impurity_split=None,
min_samples_leaf=1, min_samples_split=2,
min_weight_fraction_leaf=0.0, n_estimators=100,
n_iter_no_change=None, presort='auto', random_state=0,
subsample=1.0, tol=0.0001, validation_fraction=0.1,
verbose=0, warm_start=False)
The Decision Function
In [106]:
X_test.shape: (25, 2)
Decision function shape: (25,)
In [107]:
Decision function: [ 4.136 -1.702 -3.951 -3.626 4.29 3.662]
In [108]:
Thresholded decision function:
[ True False False False True True False True True True False True
True False True False False False True True True True True False
False]
Predictions:
['red' 'blue' 'blue' 'blue' 'red' 'red' 'blue' 'red' 'red' 'red' 'blue'
'red' 'red' 'blue' 'red' 'blue' 'blue' 'blue' 'red' 'red' 'red' 'red'
'red' 'blue' 'blue']
In [109]:
pred is equal to predictions: True
In [110]:
Decision function minimum: -7.69 maximum: 4.29
In [111]:
<matplotlib.legend.Legend at 0x7f9df09848d0>
Invalid PDF output
Predicting Probabilities
In [112]:
Shape of probabilities: (25, 2)
In [113]:
Predicted probabilities:
[[0.016 0.984]
[0.846 0.154]
[0.981 0.019]
[0.974 0.026]
[0.014 0.986]
[0.025 0.975]]
In [114]:
<matplotlib.legend.Legend at 0x7f9df0868358>
Invalid PDF output
Uncertainty in multiclass classification
In [115]:
GradientBoostingClassifier(criterion='friedman_mse', init=None,
learning_rate=0.01, loss='deviance', max_depth=3,
max_features=None, max_leaf_nodes=None,
min_impurity_decrease=0.0, min_impurity_split=None,
min_samples_leaf=1, min_samples_split=2,
min_weight_fraction_leaf=0.0, n_estimators=100,
n_iter_no_change=None, presort='auto', random_state=0,
subsample=1.0, tol=0.0001, validation_fraction=0.1,
verbose=0, warm_start=False)
In [116]:
Decision function shape: (38, 3)
Decision function:
[[-0.529 1.466 -0.504]
[ 1.512 -0.496 -0.503]
[-0.524 -0.468 1.52 ]
[-0.529 1.466 -0.504]
[-0.531 1.282 0.215]
[ 1.512 -0.496 -0.503]]
In [117]:
Argmax of decision function:
[1 0 2 1 1 0 1 2 1 1 2 0 0 0 0 1 2 1 1 2 0 2 0 2 2 2 2 2 0 0 0 0 1 0 0 2 1
0]
Predictions:
[1 0 2 1 1 0 1 2 1 1 2 0 0 0 0 1 2 1 1 2 0 2 0 2 2 2 2 2 0 0 0 0 1 0 0 2 1
0]
In [118]:
Predicted probabilities:
[[0.107 0.784 0.109]
[0.789 0.106 0.105]
[0.102 0.108 0.789]
[0.107 0.784 0.109]
[0.108 0.663 0.228]
[0.789 0.106 0.105]]
Sums: [1. 1. 1. 1. 1. 1.]
In [119]:
Argmax of predicted probabilities:
[1 0 2 1 1 0 1 2 1 1 2 0 0 0 0 1 2 1 1 2 0 2 0 2 2 2 2 2 0 0 0 0 1 0 0 2 1
0]
Predictions:
[1 0 2 1 1 0 1 2 1 1 2 0 0 0 0 1 2 1 1 2 0 2 0 2 2 2 2 2 0 0 0 0 1 0 0 2 1
0]
In [120]:
unique classes in training data: ['setosa' 'versicolor' 'virginica']
predictions: ['versicolor' 'setosa' 'virginica' 'versicolor' 'versicolor' 'setosa'
'versicolor' 'virginica' 'versicolor' 'versicolor']
argmax of decision function: [1 0 2 1 1 0 1 2 1 1]
argmax combined with classes_: ['versicolor' 'setosa' 'virginica' 'versicolor' 'versicolor' 'setosa'
'versicolor' 'virginica' 'versicolor' 'versicolor']