Advanced tidymodels

We might want to modify our predictors columns for a few reasons:

- The model requires them in a different format (e.g. dummy variables for linear regression).
- The model needs certain data qualities (e.g. same units for K-NN).
- The outcome is better predicted when one or more columns are transformed in some way (a.k.a “feature engineering”).

The first two reasons are fairly predictable (next page).

The last one depends on your modeling problem.

Think of a feature as some *representation* of a predictor that will be used in a model.

Example representations:

- Interactions
- Polynomial expansions/splines
- Principal component analysis (PCA) feature extraction

There are a lot of examples in *Feature Engineering and Selection* (FES) and *Applied Machine Learning for Tabular Data* (aml4td).

How can we represent date columns for our model?

When we use a date column in its native format, most models in R convert it to an integer.

We can re-engineer it as:

- Days since a reference date
- Day of the week
- Month
- Year
- Indicators for holidays

*Data preprocessing*steps allow your model to fit.*Feature engineering*steps help the model do the least work to predict the outcome as well as possible.

The recipes package can handle both!

We’ll go from here and create a set of resamples to use for model assessments.

We’ll use simple 10-fold cross-validation (stratified sampling):

```
set.seed(472)
hotel_rs <- vfold_cv(hotel_train, strata = avg_price_per_room)
hotel_rs
#> # 10-fold cross-validation using stratification
#> # A tibble: 10 × 2
#> splits id
#> <list> <chr>
#> 1 <split [3372/377]> Fold01
#> 2 <split [3373/376]> Fold02
#> 3 <split [3373/376]> Fold03
#> 4 <split [3373/376]> Fold04
#> 5 <split [3373/376]> Fold05
#> 6 <split [3374/375]> Fold06
#> 7 <split [3375/374]> Fold07
#> 8 <split [3376/373]> Fold08
#> 9 <split [3376/373]> Fold09
#> 10 <split [3376/373]> Fold10
```

- The recipes package is an extensible framework for pipeable sequences of preprocessing and feature engineering steps.

- Statistical parameters for the steps can be
*estimated*from an initial data set and then*applied*to other data sets.

- The resulting processed output can be used as inputs for statistical or machine learning models.

- The
`recipe()`

function assigns columns to roles of “outcome” or “predictor” using the formula

```
summary(hotel_rec)
#> # A tibble: 27 × 4
#> variable type role source
#> <chr> <list> <chr> <chr>
#> 1 lead_time <chr [2]> predictor original
#> 2 stays_in_weekend_nights <chr [2]> predictor original
#> 3 stays_in_week_nights <chr [2]> predictor original
#> 4 adults <chr [2]> predictor original
#> 5 children <chr [2]> predictor original
#> 6 babies <chr [2]> predictor original
#> 7 meal <chr [3]> predictor original
#> 8 country <chr [3]> predictor original
#> 9 market_segment <chr [3]> predictor original
#> 10 distribution_channel <chr [3]> predictor original
#> # ℹ 17 more rows
```

The `type`

column contains information on the variables

What do you think are in the `type`

vectors for the `lead_time`

and `country`

columns?

`02:00`

For any factor or character predictors, make binary indicators.

There are

*many*recipe steps that can convert categorical predictors to numeric columns.`step_dummy()`

records the levels of the categorical predictors in the training set.

In case there is a factor level that was never observed in the training data (resulting in a column of all `0`

s), we can delete any *zero-variance* predictors that have a single unique value.

This centers and scales the numeric predictors.

The recipe will use the

*training*set to estimate the means and standard deviations of the data.

- All data the recipe is applied to will be normalized using those statistics (there is no re-estimation).

To deal with highly correlated predictors, find the minimum set of predictor columns that make the pairwise correlations less than the threshold.

PCA feature extraction…

A fancy machine learning supervised dimension reduction technique called UMAP…

Nonlinear transforms like natural splines, and so on!

*Create a recipe() for the hotel data to:*

*use a Yeo-Johnson (YJ) transformation on*`lead_time`

*convert factors to indicator variables**remove zero-variance variables**add the spline technique shown above*

`03:00`

We’ll compute two measures: mean absolute error and the coefficient of determination (a.k.a \(R^2\)).

\[\begin{align} MAE &= \frac{1}{n}\sum_{i=1}^n |y_i - \hat{y}_i| \notag \\ R^2 &= cor(y_i, \hat{y}_i)^2 \end{align}\]

The focus will be on MAE for parameter optimization. We’ll use a metric set to compute these:

```
set.seed(9)
hotel_lm_wflow <-
workflow() %>%
add_recipe(hotel_indicators) %>%
add_model(linear_reg())
ctrl <- control_resamples(save_pred = TRUE)
hotel_lm_res <-
hotel_lm_wflow %>%
fit_resamples(hotel_rs, control = ctrl, metrics = reg_metrics)
collect_metrics(hotel_lm_res)
#> # A tibble: 2 × 6
#> .metric .estimator mean n std_err .config
#> <chr> <chr> <dbl> <int> <dbl> <chr>
#> 1 mae standard 16.6 10 0.214 Preprocessor1_Model1
#> 2 rsq standard 0.884 10 0.00339 Preprocessor1_Model1
```

*Use fit_resamples() to fit your workflow with a recipe.*

*Collect the predictions from the results.*

`05:00`

```
# Since we used `save_pred = TRUE`
lm_cv_pred <- collect_predictions(hotel_lm_res)
lm_cv_pred %>% print(n = 7)
#> # A tibble: 3,749 × 5
#> .pred id .row avg_price_per_room .config
#> <dbl> <chr> <int> <dbl> <chr>
#> 1 75.1 Fold01 20 40 Preprocessor1_Model1
#> 2 49.3 Fold01 28 54 Preprocessor1_Model1
#> 3 64.9 Fold01 45 50 Preprocessor1_Model1
#> 4 52.8 Fold01 49 42 Preprocessor1_Model1
#> 5 48.6 Fold01 61 49 Preprocessor1_Model1
#> 6 29.8 Fold01 66 40 Preprocessor1_Model1
#> 7 36.9 Fold01 88 49 Preprocessor1_Model1
#> # ℹ 3,742 more rows
```

There are 98 unique agent values and 100 unique companies in our training set. How can we include this information in our model?

We could:

make the full set of indicator variables 😳

lump agents and companies that rarely occur into an “other” group

use feature hashing to create a smaller set of indicator variables

use effect encoding to replace the

`agent`

and`company`

columns with the estimated effect of that predictor (in the extra materials)

There is a recipe step that will redefine factor levels based on their frequency in the training set:

Using this code, 34 agents (out of 98) were collapsed into “other” based on the training set.

We *could* try to optimize the threshold for collapsing (see the next set of slides on model tuning).

```
hotel_other_wflow <-
hotel_lm_wflow %>%
update_recipe(hotel_other_rec)
hotel_other_res <-
hotel_other_wflow %>%
fit_resamples(hotel_rs, control = ctrl, metrics = reg_metrics)
collect_metrics(hotel_other_res)
#> # A tibble: 2 × 6
#> .metric .estimator mean n std_err .config
#> <chr> <chr> <dbl> <int> <dbl> <chr>
#> 1 mae standard 16.7 10 0.213 Preprocessor1_Model1
#> 2 rsq standard 0.884 10 0.00341 Preprocessor1_Model1
```

About the same MAE and much faster to complete.

Now let’s look at a more sophisticated tool called effect feature hashing.

Between `agent`

and `company`

, simple dummy variables would create 198 new columns (that are mostly zeros).

Another option is to have a binary indicator that combines some levels of these variables.

Feature hashing (for more see *FES*, *SMLTAR*, *TMwR*, and *aml4td*):

- uses the character values of the levels
- converts them to integer hash values
- uses the integers to assign them to a specific indicator column.

Suppose we want to use 32 indicator variables for `agent`

.

For a agent with value “`Max_Kuhn`

”, a hashing function converts it to an integer (say 210397726).

To assign it to one of the 32 columns, we would use modular arithmetic to assign it to a column:

Hash functions are meant to *emulate* randomness.

- The procedure will automatically work on new values of the predictors.
- It is fast.
- “Signed” hashes add a sign to help avoid aliasing.

- There is no real logic behind which factor levels are combined.
- We don’t know how many columns to add (more in the next section).
- Some columns may have all zeros.
- If a indicator column is important to the model, we can’t easily determine why.

The textrecipes package has a step that can be added to the recipe:

```
library(textrecipes)
hash_rec <-
recipe(avg_price_per_room ~ ., data = hotel_train) %>%
step_YeoJohnson(lead_time) %>%
# Defaults to 32 signed indicator columns
step_dummy_hash(agent) %>%
step_dummy_hash(company) %>%
# Regular indicators for the others
step_dummy(all_nominal_predictors()) %>%
step_zv(all_predictors()) %>%
step_spline_natural(arrival_date_num, deg_free = 10)
hotel_hash_wflow <-
hotel_lm_wflow %>%
update_recipe(hash_rec)
```

```
hotel_hash_res <-
hotel_hash_wflow %>%
fit_resamples(hotel_rs, control = ctrl, metrics = reg_metrics)
collect_metrics(hotel_hash_res)
#> # A tibble: 2 × 6
#> .metric .estimator mean n std_err .config
#> <chr> <chr> <dbl> <int> <dbl> <chr>
#> 1 mae standard 16.7 10 0.239 Preprocessor1_Model1
#> 2 rsq standard 0.884 10 0.00324 Preprocessor1_Model1
```

About the same performance but now we can handle new values.

- Typically, you will want to use a workflow to estimate and apply a recipe.

- If you have an error and need to debug your recipe, the original recipe object (e.g.
`hash_rec`

) can be estimated manually with a function called`prep()`

. It is analogous to`fit()`

. See TMwR section 16.4

- Another function (
`bake()`

) is analogous to`predict()`

, and gives you the processed data back.

- The
`tidy()`

function can be used to get specific results from the recipe.

- Once
`fit()`

is called on a workflow, changing the model does not re-fit the recipe.

- A list of all known steps is at https://www.tidymodels.org/find/recipes/.

- Some steps can be skipped when using
`predict()`

.

- The order of the steps matters.