3 - What makes a model?

Practical Machine Learning with tidymodels

Your turn

How do you fit a linear model in R?

How many different ways can you think of?

• lm for linear model

• glmnet for regularized regression

• keras for regression using TensorFlow

• stan for Bayesian regression

• spark for large data sets

• brulee for regression using torch

To specify a model

• Choose a model type
• Specify an engine
• Set the mode

To specify a model

logistic_reg()
#> Logistic Regression Model Specification (classification)
#> 
#> Computational engine: glm

Models have default engines

To specify a model

• Choose a model type
• Specify an engine
• Set the mode

To specify a model

logistic_reg() |>
  set_engine("glmnet")
#> Logistic Regression Model Specification (classification)
#> 
#> Computational engine: glmnet

To specify a model

logistic_reg() |>
  set_engine("stan")
#> Logistic Regression Model Specification (classification)
#> 
#> Computational engine: stan

To specify a model

• Choose a model type
• Specify an engine
• Set the mode

To specify a model

decision_tree()
#> Decision Tree Model Specification (unknown mode)
#> 
#> Computational engine: rpart

Some models have a default mode

To specify a model

decision_tree() |> 
  set_mode("classification")
#> Decision Tree Model Specification (classification)
#> 
#> Computational engine: rpart

All available models are listed at https://www.tidymodels.org/find/parsnip/

To specify a model

• Choose a model type
• Specify an engine
• Set the mode

Your turn

Run the tree_spec chunk in your .qmd.

Edit this code to use a logistic regression model.

All available models are listed at https://www.tidymodels.org/find/parsnip/

Extension/Challenge: Edit this code to use a different model. For example, try using a conditional inference tree as implemented in the partykit package by changing the engine - or try an entirely different model type!

Models we’ll be using today

• Logistic regression
• Decision trees

Logistic regression

Logistic regression

Logistic regression

• Logit of outcome probability modeled as linear combination of predictors:

\(log(\frac{p}{1 - p}) = \beta_0 + \beta_1\cdot \text{A}\)

• Find a sigmoid line that separates the two classes

Decision trees

Decision trees

• Series of splits or if/then statements based on predictors

• First the tree grows until some condition is met (maximum depth, no more data)

• Then the tree is pruned to reduce its complexity

Decision trees

All models are wrong, but some are useful!

Logistic regression

Decision tree

In two dimensions

Logistic regression

Decision tree

A model workflow

Workflows bind preprocessors and models

Explain that PCA that is a preprocessor / dimensionality reduction, used to decorrelate data

What is wrong with this?

The source of data leakage here is preprocessing on the whole source data set rather than on the whole training set. So, the risk is “the values from the test set affect the mean and standard deviations used in the preprocessing.” That discussion shouldn’t mention metrics, as they haven’t been introduced yet.

While centering + scaling are easy to explain the general idea with, it’s not necessarily the biggest danger in practice. The more elaborate the estimation in your preprocessing, the higher the risk of overfitting coming back to bite you. An example of a more elaborate preprocessing estimation is principal components.

Why a workflow()?

• Workflows handle new data better than base R tools in terms of new factor levels

• You can use other preprocessors besides formulas

• They can help organize your work when working with multiple models

• Most importantly, a workflow captures the entire modeling process: fit() and predict() apply to the pre/post processing steps in addition to the actual model fit

Two ways workflows handle levels better than base R:

• Enforces that new levels are not allowed at prediction time (this is an optional check that can be turned off)

• Restores missing levels that were present at fit time, but happen to be missing at prediction time (like, if your “new” data just doesn’t have an instance of that level)

A model workflow

tree_spec <-
  decision_tree() |> 
  set_mode("classification")

tree_spec |> 
  fit(class ~ ., data = cls_train) 
#> parsnip model object
#> 
#> n= 800 
#> 
#> node), split, n, loss, yval, (yprob)
#>       * denotes terminal node
#> 
#>  1) root 800 266 class_1 (0.66750000 0.33250000)  
#>    2) pred_2< 0.3297983 510  30 class_1 (0.94117647 0.05882353) *
#>    3) pred_2>=0.3297983 290  54 class_2 (0.18620690 0.81379310)  
#>      6) pred_1>=1.236161 71  28 class_1 (0.60563380 0.39436620)  
#>       12) pred_2< 1.476946 42   7 class_1 (0.83333333 0.16666667) *
#>       13) pred_2>=1.476946 29   8 class_2 (0.27586207 0.72413793)  
#>         26) pred_1>=2.070158 7   2 class_1 (0.71428571 0.28571429) *
#>         27) pred_1< 2.070158 22   3 class_2 (0.13636364 0.86363636) *
#>      7) pred_1< 1.236161 219  11 class_2 (0.05022831 0.94977169) *

A model workflow

tree_spec <-
  decision_tree() |> 
  set_mode("classification")

workflow() |>
  add_formula(class ~ .) |>
  add_model(tree_spec) |>
  fit(data = cls_train) 
#> ══ Workflow [trained] ════════════════════════════════════════════════
#> Preprocessor: Formula
#> Model: decision_tree()
#> 
#> ── Preprocessor ──────────────────────────────────────────────────────
#> class ~ .
#> 
#> ── Model ─────────────────────────────────────────────────────────────
#> n= 800 
#> 
#> node), split, n, loss, yval, (yprob)
#>       * denotes terminal node
#> 
#>  1) root 800 266 class_1 (0.66750000 0.33250000)  
#>    2) pred_2< 0.3297983 510  30 class_1 (0.94117647 0.05882353) *
#>    3) pred_2>=0.3297983 290  54 class_2 (0.18620690 0.81379310)  
#>      6) pred_1>=1.236161 71  28 class_1 (0.60563380 0.39436620)  
#>       12) pred_2< 1.476946 42   7 class_1 (0.83333333 0.16666667) *
#>       13) pred_2>=1.476946 29   8 class_2 (0.27586207 0.72413793)  
#>         26) pred_1>=2.070158 7   2 class_1 (0.71428571 0.28571429) *
#>         27) pred_1< 2.070158 22   3 class_2 (0.13636364 0.86363636) *
#>      7) pred_1< 1.236161 219  11 class_2 (0.05022831 0.94977169) *

A model workflow

tree_spec <-
  decision_tree() |> 
  set_mode("classification")

workflow(class ~ ., tree_spec) |> 
  fit(data = cls_train) 
#> ══ Workflow [trained] ════════════════════════════════════════════════
#> Preprocessor: Formula
#> Model: decision_tree()
#> 
#> ── Preprocessor ──────────────────────────────────────────────────────
#> class ~ .
#> 
#> ── Model ─────────────────────────────────────────────────────────────
#> n= 800 
#> 
#> node), split, n, loss, yval, (yprob)
#>       * denotes terminal node
#> 
#>  1) root 800 266 class_1 (0.66750000 0.33250000)  
#>    2) pred_2< 0.3297983 510  30 class_1 (0.94117647 0.05882353) *
#>    3) pred_2>=0.3297983 290  54 class_2 (0.18620690 0.81379310)  
#>      6) pred_1>=1.236161 71  28 class_1 (0.60563380 0.39436620)  
#>       12) pred_2< 1.476946 42   7 class_1 (0.83333333 0.16666667) *
#>       13) pred_2>=1.476946 29   8 class_2 (0.27586207 0.72413793)  
#>         26) pred_1>=2.070158 7   2 class_1 (0.71428571 0.28571429) *
#>         27) pred_1< 2.070158 22   3 class_2 (0.13636364 0.86363636) *
#>      7) pred_1< 1.236161 219  11 class_2 (0.05022831 0.94977169) *

Your turn

Run the tree_wflow chunk in your intro-03-classwork.qmd.

tree_spec <- decision_tree() |> 
  set_mode("classification")

tree_wflow <- workflow() |>
  add_formula(class ~ .) |>
  add_model(tree_spec)

tree_wflow

Edit this code to make a workflow with your own model of choice.

Extension/Challenge: Other than formulas, what kinds of preprocessors are supported?

Predict with your model

How do you use your new tree_fit model?

tree_spec <-
  decision_tree() |> 
  set_mode("classification")

tree_fit <-
  workflow(class ~ ., tree_spec) |> 
  fit(data = cls_train)

Your turn

Run:

predict(tree_fit, new_data = cls_test)

What do you notice about the structure of the result?

Your turn

Run:

augment(tree_fit, new_data = cls_test)

How does the output compare to the output from predict()?

The tidymodels prediction guarantee!

• The predictions will always be inside a tibble
• The column names and types are unsurprising and predictable
• The number of rows in new_data and the output are the same

Understand your model

How do you understand your new tree_fit model?

Understand your model

How do you understand your new tree_fit model?

library(rpart.plot)
tree_fit |>
  extract_fit_engine() |>
  rpart.plot()

You can extract_*() several components of your fitted workflow.

⚠️ Never predict() with any extracted components!

roundint = FALSE is only to quiet a warning

Understand your model

How do you understand your new tree_fit model?

You can use your fitted workflow for model and/or prediction explanations:

• overall variable importance, such as with the vip package

• flexible model explainers, such as with the DALEXtra package

Learn more at https://www.tmwr.org/explain.html

Your turn

Extract the model engine object from your fitted workflow and check it out.

Afterward, ask what kind of object people got from the extraction, and what they did with it (e.g. give it to summary(), plot(), broom::tidy() ). Live code along

The whole game - status update

Stress that fitting a model on the entire training set was only for illustrating how to fit a model