Annotations

Introduction

👀

This page contains annotations for selected slides.

There’s a lot that we want to tell you. We don’t want people to have to frantically scribble down things that we say that are not on the slides.

We’ve added sections to this document with longer explanations and links to other resources.

Data Budget

Data splitting and spending

More about the initial data split can be found in Chapter 3 of Applied Machine Learning for Tabular Data (AML4TD).

In particular, a three-way split into training, validation, and testing set can be done via

set.seed(123)
initial_validation_split(cls_data_2026, prop = c(0.6, 0.2))
#> <Training/Validation/Testing/Total>
#> <600/200/200/1000>

What is set.seed()?

What does set.seed() do?

We’ll use pseudo-random numbers (PRN) to partition the data into training and testing. PRN are numbers that emulate truly random numbers (but really are not truly random).

Think of PRN as a box that takes a starting value (the “seed”) that produces random numbers using that starting value as an input into its process.

If we know a seed value, we can reproduce our “random” numbers. To use a different set of random numbers, choose a different seed value.

For example:

set.seed(1)
runif(3)
#> [1] 0.2655087 0.3721239 0.5728534

# Get a new set of random numbers:
set.seed(2)
runif(3)
#> [1] 0.1848823 0.7023740 0.5733263

# We can reproduce the old ones with the same seed
set.seed(1)
runif(3)
#> [1] 0.2655087 0.3721239 0.5728534

If we don’t set the seed, R uses the clock time and the process ID to create a seed. This isn’t reproducible.

Since we want our code to be reproducible, we set the seeds before random numbers are used.

In theory, you can set the seed once at the start of a script. However, if we do interactive data analysis, we might unwittingly use random numbers while coding. In that case, the stream is not the same, and we don’t get reproducible results.

The value of the seed is an integer and really has no meaning. Max has a script to generate random integers to use as seeds to “spread the randomness around”. It is basically:

cat(paste0("set.seed(", sample.int(10000, 5), ")", collapse = "\n"))
#> set.seed(9725)
#> set.seed(8462)
#> set.seed(4050)
#> set.seed(8789)
#> set.seed(1301)

What Makes A Model?

What is wrong with this?

If we treat the preprocessing as a separate task, it raises the risk that we might accidentally overfit to the data at hand.

For example, someone might estimate something from the entire data set (such as the principal components) and treat that data as if it were known (and not estimated). Depending on what was done with the data, the consequences of doing that could be:

  • Your performance metrics are slightly-to-moderately optimistic (e.g., you might think your accuracy is 85% when it is actually 75%)
  • A consequential component of the analysis is not right, and the model just doesn’t work.

The big issue here is that you won’t be able to figure this out until you get a new piece of data, such as the test set.

A really good example of this is in ‘Selection bias in gene extraction on the basis of microarray gene-expression data’. The authors re-analyze a previous publication and show that the original researchers did not include feature selection in the workflow. Because of that, their performance statistics were extremely optimistic. In one case, they could do the original analysis on complete noise and still achieve zero errors.

Generally speaking, this problem is referred to as data leakage. Some other references:

Understand your model

On the next slide, we’ve abbreviated the code necessary to make this plot. If you want to replicate the plot exactly, you’ll need:

library(rpart.plot)
split_fun <- function(x, labs, digits, varlen, faclen) {
  for (i in 1:length(labs)) {
    if (grepl(",", labs[i])) {
      parts <- strsplit(labs[i], ",")[[1]]
      parts <- trimws(parts)
      if (length(parts) > 5) {
        parts <- c(parts[1:5], "...")
      }
      labs[i] <- paste(parts, collapse = ", ")
    }
    labs[i] <- paste(strwrap(labs[i], width = 15), collapse = "\n")
  }
  labs
}

tree_fit |>
  extract_fit_engine() |>
  rpart.plot(
    roundint = FALSE,
    type = 3,
    clip.right.labs = FALSE,
    split.fun = split_fun
  )

Evaluating Models

Brier score

The Brier score measures how close a model probability estimate is to its best possible value (i.e., zero or one).

In the best case, the model is perfect, and every prediction equals 0.0 or 1.0 (depending on the true class). In this case, the Brier score is zero.

When the model is uninformative and there are two classes, the worst-case values range from 0.25 to about 0.50. Imagine that the model predicts the same noninformative prediction of 50% (basically “¯\(ツ)/¯”). In that case, every prediction is either \((0.00 - 0.50)^2\) or \((1.00 - 0.50)^2\). The average of those is 0.25.

There are many different ways a model can be bad, though, and some of these will produce Brier scores between 0.25 and 0.50.

Where are the fitted models?

The primary purpose of resampling is to estimate model performance. The models are almost never needed again.

Also, if the data set is large, the model object may require a lot of memory to save so, by default, we don’t keep them.

For more advanced use cases, you can extract and save them. See:

Tuning models

Different types of grids

More on space-filling designs in Chapters 4 and 5 of Surrogates: Gaussian process modeling, design, and optimization for the applied sciences.

These designs also scale well as the number of tuning parameters grows. They are designed to fill the predictor space efficiently.

Running in parallel

We usually leave one or two cores available to work with when we run in parallel.

If you are using an HPC system, be careful to use only the cores allocated to you.

Also, memory is duplicated for each worker. If your main R session is using 1GB of memory, using 10 workers requires 11GB.

Comparing these methods:

  • They usually perform about the same.
  • future is more mature and is geared towards users and developers.
  • mirai contains its own queuing system and can be more efficient in allocating work.

Feature Engineering: dummies and embeddings

Isomap with recipes

The results from Isomap, unlike UMAP, cannot be distorted in the same way. Here, the number of neighbors changes between the charts. Each of the charts is an application of Isomap on the same completely random data. The points are in different locations, but none of the charts appear to show any patterns.

Extras - Case Study on Transportation

A recipe - handle correlations

In this code chunk, what’s the story with !!stations?

chi_pca_rec <- 
  chi_rec |> 
  step_normalize(all_of(!!stations)) |> 
  step_pca(all_of(!!stations), num_comp = tune())

stations is a vector of names of 20 columns that we want to use in the steps. If the list were shorter, we could type them in (e.g., c("col1", "col2"), etc.).

The vector lives in our global workspace, and if we are in parallel, the worker processes might not have access to stations. The !! (frequently said as “bang bang”) inserts the actual contents of the vector into the all_of() calls so that it looks like you just typed it in.

This means that the parallel process workers have a copy of the data in their reach, and the code will run without error.

Extras - Iterative Search

Gaussian Processes and Optimization

Some other references for GP’s:

Acquisition Functions

More references: