1 Introduction

Quarto makes it surprisingly easy to build a blog.

You write your content, render it, and publish it. Everything works—until it doesn’t.

Quarto has made it remarkably easy to create modern technical websites, blogs, books, and reports from plain text files. A typical Quarto website can combine narrative text, executable code, figures, tables, references, and multiple output formats in a single reproducible publishing workflow. In that sense, Quarto is not only a writing tool; it is also a publishing system designed especially for computational and data-driven content. The official Quarto documentation describes websites as projects that can be rendered and published to several destinations, including GitHub Pages, Netlify, Posit Connect, and other static hosting services (Posit PBC 2026a, 2026b).

For someone writing about R, statistics, or data science, this is very attractive. You can write a blog post in .qmd, run your R code inside the document, generate plots and tables, render the site locally, and then publish the resulting static files. At first glance, the workflow looks almost linear:

1. write the content,
2. render the site,
3. deploy it,
4. share the link.

Many introductory tutorials understandably focus on this smooth path. They explain how to create a Quarto website, configure the _quarto.yml file, add posts, render the project, and publish the site. These steps are necessary, but they do not fully describe what happens when a Quarto blog becomes a living project rather than a one-time demo.

The real questions usually appear later. What happens when the site grows? What happens when posts include code, external data sources, generated images, downloadable files, or multiple output formats? What happens when the website builds successfully on your own computer but fails in the deployment environment? At that point, publishing is no longer just about pushing HTML files to the web. It becomes a question of reproducibility, dependency management, build strategy, and platform choice.

This article reflects on that second stage: the stage where a Quarto blog moves from a local project to a maintained public website. More specifically, it discusses the practical lessons learned while moving a Quarto-based blog from Netlify to GitHub Pages. The aim is not to provide another “click here, then click there” tutorial. Instead, the goal is to discuss the kinds of issues that are often invisible at the beginning: build limits, environment differences, hidden dependencies, external services, file paths, output formats, and the trade-offs between convenience and control.

In short, this is a real-world deployment story. Not because the technical details are unique, but because the pattern is common: a tool works beautifully in local development, then the publishing pipeline reveals the assumptions we did not know we were making.

2 When Things Start to Break

Like many users, I initially chose Netlify as my deployment platform. It is fast, easy to configure, and works very well for traditional static websites. With minimal setup, it is possible to connect a repository, trigger automatic builds, and publish a site within minutes. For simple blogs and documentation pages, this model is both convenient and efficient.

For a while, everything worked smoothly.

However, as the project evolved, the nature of the website also started to change. What initially looked like a static blog gradually became a more dynamic, computation-driven project. Posts were no longer just text; they included code execution, data processing, and generated outputs such as figures, tables, and downloadable files.

At this point, some structural limitations of build-based deployment started to become more visible.

First, every deployment is essentially a full rebuild. Even small changes may trigger a complete build process, depending on the configuration. While this is not an issue for lightweight static content, it becomes more significant for projects that rely on computation.

Second, data-driven Quarto projects are inherently heavier than typical static sites. Rendering a post may involve running R code, loading libraries, generating plots, or even accessing external data sources. These steps increase both build time and resource usage.

Third, frequent updates amplify the effect. A workflow that feels fast at the beginning can become noticeably slower as the number of posts grows and the project becomes more complex. Over time, this can translate into longer build durations and increased consumption of available resources.

None of these are “failures” in the strict sense. They are natural consequences of using a system designed primarily for static content in a context that increasingly behaves like a computational workflow.

At this stage, the central question was no longer:

How do I deploy this site?

but rather:

Is this deployment model sustainable for a data-driven Quarto project in the long run?

3 Moving to GitHub Pages

At this point, the decision to explore alternatives was not driven by a single failure, but by a growing mismatch between the project’s needs and the deployment model.

GitHub Pages emerged as a natural alternative.

Unlike platforms that rely on external build services, GitHub Pages is closely integrated with the repository itself. This creates a different workflow: instead of delegating the entire process to a managed service, the developer has more direct control over how the site is built and deployed.

This shift might seem subtle, but it changes the way you think about publishing.

In a repository-driven approach, the website is no longer just an output. It becomes part of a controlled pipeline:

• the source files are versioned,
• the build process is explicitly defined,
• and the output is reproducible under the same conditions.

This level of control is particularly important for projects that include code execution and data processing. When rendering depends on computations, it becomes essential to understand how and where those computations are performed.

Another important difference is transparency. Build logs, dependency resolution, and execution steps are visible and traceable. While this may introduce additional complexity at first, it also makes debugging and long-term maintenance significantly easier.

Of course, this approach comes with a trade-off.

Compared to Netlify, GitHub Pages requires a bit more effort to set up and maintain. It is less “plug-and-play” and more “build-your-own-pipeline.” However, for projects that go beyond simple static content, this added responsibility often translates into greater flexibility.

In that sense, the transition was not just about switching platforms. It was about moving from a convenience-oriented model to a control-oriented one.

And that shift becomes especially meaningful once the project starts to grow.

4 What You Don’t See in Tutorials

Most tutorials focus on the ideal path: everything works, the site renders, and deployment succeeds. While this is useful for getting started, it often hides an important reality.

As soon as a project moves beyond a simple example, a different set of challenges begins to emerge—challenges that are rarely discussed in introductory guides.

5 Lessons Learned

After going through this transition, it became clear that the real challenge is not learning a tool, but understanding the system behind it. What initially looked like a simple publishing workflow turned out to involve multiple layers—each with its own assumptions, constraints, and trade-offs.

Several key lessons emerged from this process.

6 Netlify vs GitHub Pages

After working with both platforms, the differences become clearer when viewed from a practical perspective rather than a purely technical one.

Both Netlify and GitHub Pages are capable solutions for publishing Quarto websites. However, they are built around different assumptions, and those assumptions become more visible as a project grows.

Netlify excels in simplicity. For lightweight static sites, documentation pages, or personal blogs with minimal computation, it provides a smooth and efficient experience. The setup is fast, and the platform handles most of the deployment process automatically.

GitHub Pages, on the other hand, offers greater control. While it may require more initial effort, it provides a clearer view of the build process and allows more flexibility in handling dependencies, workflows, and project structure.

The difference becomes especially important for Quarto projects that include code execution, data processing, or multiple outputs. In such cases, having visibility and control over the pipeline can make a significant difference in both stability and maintainability.

7 Which One Should You Choose?

There is no single correct answer, but there is a practical way to think about the choice.

• If your project is a simple static blog with minimal computation, Netlify is often the most convenient option.
• If your project involves data processing, code execution, or a more complex workflow, GitHub Pages tends to offer a more sustainable solution.

Ultimately, the decision is less about the platform itself and more about the nature of the project.

8 Final Thoughts

Publishing a Quarto blog is easy. Maintaining it as a real-world project is not. As soon as a project moves beyond a simple example, deployment becomes part of the system design. It requires thinking about environments, dependencies, workflows, and long-term sustainability. The tools themselves are not the challenge. The challenge is understanding how they interact. Once that becomes clear, the process becomes not only manageable, but also much more intentional. In that sense, deployment is no longer just a final step. It is part of the architecture.

1 A model can run and still be fundamentally wrong

Many time series models fail before they even begin. Not because the software crashes. Not because the code is wrong. But because the data entering the model violate one of the most important assumptions in time series analysis: stationarity.

This is where many analyses quietly go off the rails. A model is estimated, forecasts are produced, coefficients look serious, and the graphs appear convincing. But the model may be chasing a moving target rather than learning a stable data-generating mechanism.

In this post, we will work with a real macroeconomic series rather than a toy example. The data come from the Consumer Price Index for All Urban Consumers: All Items (CPIAUCSL), published by the U.S. Bureau of Labor Statistics and distributed through FRED. FRED describes CPIAUCSL as a monthly, seasonally adjusted price index and notes that percent changes in the index are commonly used to measure inflation.

Because live API access may fail in some institutional or offline environments, this workflow uses a locally downloaded CSV file instead of fetching the series on the fly. You can download the file directly from the CPIAUCSL page on FRED.

The goal is simple: show why raw time series levels often mislead us, what stationarity really means, and why transformations such as differencing and log-differencing are not cosmetic tricks but conceptual necessities.

2 What stationarity really means

In informal language, a stationary series is one whose behavior does not drift in a systematic way over time. More formally, a weakly stationary process () satisfies three conditions:

The first condition says the mean does not change over time. The second says the variance is constant. The third says the covariance between observations depends only on the lag (k), not on calendar time itself.

This matters because a large part of classical time series modeling is built on the idea that the stochastic structure is stable. When that structure is drifting, many familiar tools become unreliable or at least much harder to interpret. A trending series can generate strong autocorrelation even when the underlying dynamic structure is weak. A persistent upward path can trick the analyst into seeing “model fit” where the model is merely inheriting inertia from the level of the series.

Put differently: without stationarity, a model may explain movement without actually explaining the mechanism.

3 Load the CPI data from a CSV file

Download the CSV file for CPIAUCSL from the official FRED series page and save it in your working directory with the name CPIAUCSL.csv. The file typically includes the columns observation_date and CPIAUCSL. FRED is the distribution platform, while the source agency for the series is the U.S. Bureau of Labor Statistics.

library(readr)
library(dplyr)
library(ggplot2)
library(tibble)
library(zoo)
library(scales)
library(patchwork)
library(tseries)

cpi_tbl <- read_csv("CPIAUCSL.csv", show_col_types = FALSE) %>%
  transmute(
    date = as.Date(observation_date),
    cpi  = as.numeric(CPIAUCSL)
  ) %>%
  arrange(date) %>%
  filter(!is.na(date), !is.na(cpi))

cpi_tbl %>% slice_head(n = 5)

# A tibble: 5 × 2
  date         cpi
  <date>     <dbl>
1 1947-01-01  21.5
2 1947-02-01  21.6
3 1947-03-01  22  
4 1947-04-01  22  
5 1947-05-01  22.0

The line filter(!is.na(date), !is.na(cpi)) is important. If your CSV has an NA for a month such as October 2025, that observation is safely excluded from the analysis instead of silently breaking the workflow.

4 Start with the visual story, not the test statistic

In time series analysis, the first serious diagnostic is often visual rather than formal. That is not because tests are unimportant. It is because plots let us see the basic character of the data before we start compressing everything into a p-value.

If a series has a visible trend, changing volatility, sudden level shifts, or unusual gaps, that already tells us something about whether a stationary model is likely to behave well.

p_level <- ggplot(cpi_tbl, aes(x = date, y = cpi)) +
  geom_line(linewidth = 0.9, color = "#1B4965") +
  labs(
    title = "U.S. CPI (CPIAUCSL): level series",
    subtitle = "Monthly, seasonally adjusted index from FRED",
    x = NULL,
    y = "Index"
  ) +
  scale_y_continuous(labels = label_number()) +
  theme_minimal(base_size = 12)

p_level

cpi_roll <- cpi_tbl %>%
  mutate(
    roll_mean_24 = zoo::rollmean(cpi, k = 24, fill = NA, align = "right"),
    roll_sd_24   = zoo::rollapply(cpi, width = 24, FUN = sd, fill = NA, align = "right")
  )

p_roll_mean <- ggplot(cpi_roll, aes(date, roll_mean_24)) +
  geom_line(linewidth = 0.9, color = "#2A9D8F") +
  labs(
    title = "24-month rolling mean of CPI",
    x = NULL,
    y = "Rolling mean"
  ) +
  theme_minimal(base_size = 12)

p_roll_sd <- ggplot(cpi_roll, aes(date, roll_sd_24)) +
  geom_line(linewidth = 0.9, color = "#E76F51") +
  labs(
    title = "24-month rolling standard deviation of CPI",
    x = NULL,
    y = "Rolling SD"
  ) +
  theme_minimal(base_size = 12)

p_roll_mean / p_roll_sd

5 Why raw CPI levels are a good example

CPI is ideal for illustrating this problem because the level series typically trends upward over time. That is not a defect in the data; it is what a price index often does. But from a modeling perspective, it creates trouble.

If the level keeps drifting upward, then the mean is not constant. If the size of movements changes as the level rises, the variance may also appear unstable. In such a setting, fitting a model directly to the raw series can mix long-run inflationary drift with short-run dynamic behavior.

Economically, analysts are usually not interested in the index level itself as much as they are interested in inflation, that is, the rate at which the price level changes. Statistically, this is convenient too, because transforming the series from levels to changes often brings it closer to stationarity.

6 A statistical check: the Augmented Dickey-Fuller test

Visual diagnosis matters, but it is usually not enough. A commonly used statistical tool is the Augmented Dickey-Fuller (ADF) test, which tests for the presence of a unit root. In practical terms, the test is often used to assess whether a series behaves like a non-stationary process with persistent stochastic trend.

The null hypothesis of the ADF test is that the series has a unit root. That means the burden of proof is asymmetric:

• a large p-value means we do not have strong evidence against non-stationarity,
• a small p-value means the data are more consistent with stationarity.

That distinction is easy to say and easy to misuse. Failing to reject the null is not the same thing as proving a series is non-stationary beyond all doubt. It simply means the test did not find enough evidence against the unit-root view.

Let us start with the raw CPI level.

adf_level <- tseries::adf.test(cpi_tbl$cpi)
adf_level

    Augmented Dickey-Fuller Test

data:  cpi_tbl$cpi
Dickey-Fuller = -0.1813, Lag order = 9, p-value = 0.99
alternative hypothesis: stationary

The Augmented Dickey–Fuller (ADF) test provides a formal way to assess whether the series contains a unit root. The null hypothesis of the test is that the series is non-stationary (i.e., it has a unit root), while the alternative hypothesis is stationarity.

In this case, the p-value is extremely high (p ≈ 0.99), meaning that we fail to reject the null hypothesis. In other words, there is no statistical evidence to support that the CPI level series is stationary.

However, this result should not be interpreted in isolation. Statistical tests and visual diagnostics should complement each other. The high p-value is entirely consistent with what we observed earlier: the series exhibits a strong upward trend and does not fluctuate around a constant mean.

Taken together, both the visual evidence and the ADF test point to the same conclusion — the raw CPI level behaves more like a drifting (unit root) process than a stationary one. This reinforces the need for transforming the series before attempting any meaningful modeling.

7 The first rescue: differencing

One of the oldest and most important ideas in time series analysis is that differencing can remove certain forms of trend. The first difference is

This transformation asks a different question. Instead of modeling the level, we model the change from one period to the next.

cpi_diff_tbl <- cpi_tbl %>%
  mutate(diff_cpi = c(NA, diff(cpi))) %>%
  filter(!is.na(diff_cpi))

p_diff <- ggplot(cpi_diff_tbl, aes(x = date, y = diff_cpi)) +
  geom_line(linewidth = 0.8, color = "#6D597A") +
  labs(
    title = "First difference of CPI",
    subtitle = "Absolute month-to-month change in the index",
    x = NULL,
    y = expression(Delta*CPI)
  ) +
  theme_minimal(base_size = 12)

p_diff

Taking the first difference removes a large part of the visible trend in the series. Compared to the raw CPI level, the differenced series fluctuates much more around a relatively stable center, which is an encouraging sign from a modeling perspective.

However, differencing does not fully solve the problem. While it helps stabilize the mean, the variability of the series still appears to change over time, particularly in more recent periods where larger fluctuations are observed. This suggests that the series may still violate the constant variance assumption.

There is also a more subtle but important issue: interpretation. The first difference represents absolute changes in the index, not relative ones. In macroeconomic data, a one-point increase in CPI does not carry the same meaning when the index is around 100 versus when it exceeds 300. As the scale of the series grows, the same absolute change reflects a smaller proportional movement.

In other words, differencing improves the statistical properties of the series, but it does not yet provide a fully consistent or interpretable measure of change. This is why we often go one step further and consider transformations based on relative (percentage) changes.

8 The more meaningful rescue: log differences

This is where the log transformation becomes more than a technical detail. Consider

For moderate changes, this is approximately the proportional growth rate. In the CPI context, it moves us from the language of index levels toward the language of inflation.

That shift is both statistical and economic.

cpi_log_tbl <- cpi_tbl %>%
  mutate(
    log_cpi = log(cpi),
    dlog_cpi = c(NA, diff(log_cpi)),
    annualized_inflation_pct = 1200 * dlog_cpi,
    yoy_inflation_pct = 100 * (cpi / lag(cpi, 12) - 1)
  )

p_dlog <- cpi_log_tbl %>%
  filter(!is.na(annualized_inflation_pct)) %>%
  ggplot(aes(x = date, y = annualized_inflation_pct)) +
  geom_line(linewidth = 0.8, color = "#D62828") +
  labs(
    title = "Monthly log-difference of CPI (annualized)",
    subtitle = "A close cousin of short-run inflation",
    x = NULL,
    y = "Percent"
  ) +
  theme_minimal(base_size = 12)

p_yoy <- cpi_log_tbl %>%
  filter(!is.na(yoy_inflation_pct)) %>%
  ggplot(aes(x = date, y = yoy_inflation_pct)) +
  geom_line(linewidth = 0.8, color = "#F4A261") +
  labs(
    title = "Year-over-year CPI inflation",
    subtitle = "A slower-moving inflation measure",
    x = NULL,
    y = "Percent"
  ) +
  theme_minimal(base_size = 12)

p_dlog / p_yoy

Two key insights emerge from these transformations.

First, moving from levels to rates of change fundamentally improves interpretability. The log-difference series represents approximate percentage changes — in this context, a close proxy for short-run inflation. This is the quantity economists actually care about. A 1% increase has the same meaning regardless of whether the index is at 100 or 300, making comparisons over time much more meaningful.

Second, the transformation has a clear impact on the statistical properties of the series. Compared to the raw level and even the first difference, the log-differenced series fluctuates more consistently around a stable mean. While it still exhibits volatility spikes and occasional outliers, the overall behavior is much closer to what we would expect from a stationary process.

The comparison between the two plots is also instructive. The monthly log-difference captures short-term fluctuations and reacts quickly to shocks, while the year-over-year inflation series smooths out this noise and highlights longer-term inflation dynamics. Both are useful, but they answer different questions.

To put it bluntly: you did not just transform the data — you changed the question.

9 Re-test after transformation

Let us apply the ADF test again, this time to the log-differenced series.

adf_dlog <- cpi_log_tbl %>%
  filter(!is.na(dlog_cpi)) %>%
  pull(dlog_cpi) %>%
  tseries::adf.test()

adf_dlog

    Augmented Dickey-Fuller Test

data:  .
Dickey-Fuller = -4.3862, Lag order = 9, p-value = 0.01
alternative hypothesis: stationary

The contrast between the two ADF test results is striking and highly informative.

For the raw CPI level, we failed to reject the null hypothesis of a unit root, indicating that the series behaves as a non-stationary process. In contrast, for the log-differenced series, the p-value drops to around 0.01, allowing us to reject the null hypothesis and conclude that the transformed series is consistent with stationarity.

This shift is not just a technical detail — it reflects a fundamental change in how the data behaves. The transformation has effectively removed the persistent trend component and brought the series closer to a stable statistical structure.

That said, the test result should always be interpreted alongside the visual evidence. The ADF test provides formal confirmation, but the intuition comes from the plots. What we saw visually — a drifting level series versus a mean-reverting transformed series — is now supported by statistical testing.

In essence, the workflow comes full circle:
we start with a problematic series, diagnose the issue visually, apply a transformation, and then verify the improvement formally.

This is the core of time series thinking.

10 A subtle but crucial point: transformation changes interpretation

This is the point where many explanations remain superficial.

When you difference a series, you are not merely “cleaning” it — you are redefining the object of analysis.

• Modeling CPI levels asks how the price index evolves over time.
• Modeling first differences asks how much the index changes from one period to the next.
• Modeling log differences asks about proportional change, which is directly linked to inflation.

These are not equivalent statistical questions, and they are certainly not equivalent economic questions.

This is why time series preprocessing should never be treated as a mechanical step. Every transformation involves a trade-off: it improves certain statistical properties while simultaneously altering the meaning of the data.

Understanding that trade-off is not optional — it is central to sound time series analysis.

11 Why this matters for ARIMA-style modeling

ARIMA models are often presented as if the workflow were mechanical: inspect the series, difference if needed, identify orders, estimate parameters, check residuals, and forecast. While this workflow is useful, it can create the misleading impression that differencing is simply a procedural step — a box to tick.

It is not.

Differencing is a deliberate modeling choice. Its purpose is to separate persistent, trend-like behavior from shorter-run dynamics. If you skip it when it is needed, your model may inherit non-stationarity and produce unreliable or misleading inference. If you apply it excessively, you risk removing meaningful structure and end up modeling noise.

The real question, therefore, is not “Should I difference?” but rather:
What feature of the data am I trying to stabilize, and what question do I want the model to answer?

12 A compact comparison

13 Common mistakes

Most mistakes in time series analysis are not computational — they are conceptual.

Mistake 1: fitting models directly to raw levels because the plot “looks smooth.”
Smoothness is not stationarity. A strong trend can produce visually smooth series that are statistically problematic.

Mistake 2: treating differencing as a harmless default.
Differencing changes the meaning of the data. It may improve statistical properties while quietly reducing interpretability if applied without care.

Mistake 3: relying on a single test result as final truth.
The ADF test is useful, but it is only one piece of evidence. Visual inspection, domain knowledge, structural breaks, and alternative tests all matter.

Mistake 4: forgetting the economics.
In the case of CPI, the focus is typically on inflation, not the index level itself. A good transformation is one that improves statistical validity while remaining aligned with the economic question.

Taken together, these mistakes point to a simple lesson:
time series analysis is not about applying steps — it is about making informed choices.

14 Final thoughts

Most time series models do not fail because we cannot estimate them. They fail because we model the wrong object.

The raw CPI series is a clear reminder that not every observed series is ready for modeling. A trending index is rarely an appropriate input for a stationary model. Once we difference — and especially log-difference — the data, the series becomes more interpretable, more stable, and much closer to the type of process that classical time series methods are designed to handle.

So before asking whether your model is sophisticated enough, ask a more fundamental question:

Am I modeling a stable process — or just chasing drift?

In many cases, the answer to this question matters far more than whether you choose AR(1), ARIMA(1,1,1), or any other fashionable specification.

15 References and further reading

1 Introduction – Why This Topic Matters

A model that performs exceptionally well on a test set is not necessarily a good model; in many cases, it is a warning sign. High accuracy or low error metrics are meaningful only if we understand how they were obtained. In real-world settings, models rarely encounter data generated under the same conditions as the training phase: data arrive sequentially, delays occur, missingness patterns change, and measurement errors accumulate. Under such conditions, impressive validation metrics can quickly lose their relevance.

A common scenario in applied data science is deceptively familiar. During development, the model looks flawless: cross-validation results are stable, performance metrics are strong, and diagnostic plots inspire confidence. Once deployed, however, performance deteriorates—sometimes rapidly. Forecasts drift, classification decisions become unreliable, and stakeholders begin to question the entire modeling pipeline. While this failure is often attributed to distributional shift or concept drift, a more fundamental issue is frequently overlooked: the model was exposed, directly or indirectly, to information it would not have access to at prediction time.

This phenomenon is known as data leakage. Importantly, data leakage is rarely the result of an obvious coding mistake. More often, it emerges from subtle flaws in experimental design, preprocessing order, or feature construction decisions made well before the model is fitted. As a result, leakage can silently inflate performance metrics, creating models that appear robust on paper but collapse in practice.
> “A model that performs perfectly on paper but fails miserably in practice is often a victim of data leakage.”

In this article, we examine data leakage not as a technical curiosity, but as a structural threat to valid statistical modeling. We begin by clarifying what data leakage is—and what it is not—before demonstrating, using a real dataset and R-based workflows, how seemingly reasonable preprocessing choices can contaminate model evaluation. We then reconstruct the same analysis using a leakage-free pipeline, highlighting the practical and conceptual differences through numerical results and carefully designed visualizations.

2 What Is Data Leakage?

At its core, data leakage occurs when information that would not be available at prediction time is inadvertently used during model training or evaluation. This information can enter the modeling pipeline in subtle ways—often long before a model is fitted—leading to overly optimistic performance estimates. The critical issue is not that the model “cheats,” but that the experimental setup allows future or target-related information to influence learning.

Formally, consider a supervised learning problem where we aim to estimate a function:

using a training set and evaluate it on a test set . A valid evaluation assumes that is generated independently of and that no function of influences the training process. Data leakage violates this assumption by introducing a dependency—direct or indirect—between training and test information.

3 Common Sources of Data Leakage in Practice

Data leakage rarely appears as an obvious error. In practice, it is often the result of reasonable-looking preprocessing decisions applied in the wrong order or under incorrect assumptions. This section outlines the most common sources of leakage encountered in applied statistical modeling and machine learning workflows, with a particular focus on preprocessing stages that precede model fitting.

4 Dataset Description: Airbnb Listings Data

To demonstrate how data leakage arises in practice, we use a real-world dataset derived from Airbnb listings. The dataset is obtained from the publicly available Inside Airbnb project, which provides detailed, regularly updated information on short-term rental listings for major cities worldwide. In this study, we focus on the Istanbul listings, which offer a rich combination of numerical and categorical variables and exhibit common data quality issues encountered in applied modeling tasks.

The Inside Airbnb project aims to support research, policy analysis, and public discussion by making scraped Airbnb data openly accessible. The dataset includes listing-level attributes such as pricing information, accommodation characteristics, host-related variables, and geographic identifiers. Due to its size, heterogeneity, and real-world imperfections, it provides an ideal setting for illustrating preprocessing pitfalls and evaluation errors.

5 A Naive Preprocessing Pipeline (And Why It Is Wrong)

At first glance, many preprocessing pipelines appear perfectly reasonable. Data are cleaned, missing values are handled, variables are scaled, and only then is the dataset split into training and test sets. This workflow is intuitive, easy to implement, and—most importantly—widely used. Unfortunately, it is also fundamentally flawed.

In this section, we deliberately construct such a naive pipeline to illustrate how data leakage can arise without any obvious warning signs.

library(tidyverse)
library(rsample)

# Load data (example assumes listings.csv from Inside Airbnb)
airbnb <- read_csv("listings.csv")

airbnb_model <- airbnb %>%
  select(
    price,
    accommodates,
    bedrooms,
    bathrooms,
    minimum_nights,
    availability_365
  ) %>%
  mutate(
    price = as.numeric(str_remove_all(price, "[$,]"))
  )

airbnb_preprocessed <- airbnb_model %>%
  mutate(across(
    .cols = -price,
    .fns  = ~ ifelse(is.na(.x), mean(.x, na.rm = TRUE), .x)
  )) %>%
  mutate(across(
    .cols = -price,
    .fns  = scale
  ))

set.seed(123)

split <- initial_split(airbnb_preprocessed, prop = 0.8)
train_data <- training(split)
test_data  <- testing(split)

# Fit a linear regression model on the training data
model_naive <- lm(price ~ ., data = train_data)

# Generate predictions for the test set
pred_test <- predict(model_naive, newdata = test_data)

# Create an evaluation dataset with observed targets only
eval_df <- test_data %>%
  transmute(
    price = price,
    pred  = pred_test
  ) %>%
  filter(!is.na(price), !is.na(pred))

# Root Mean Squared Error
rmse_naive <- sqrt(mean((eval_df$price - eval_df$pred)^2))
rmse_naive

[1] 21213.69

6 Detecting Data Leakage: Repeated Splits and Performance Distributions

A single train–test split provides only a point estimate of model performance. To assess whether the suspiciously favorable evaluation observed earlier is a coincidence or a structural issue, we repeat the naive preprocessing and evaluation procedure across multiple random splits of the data. This allows us to examine the distribution of performance metrics rather than relying on a single value.

set.seed(123)

n_repeats <- 30
rmse_values <- numeric(n_repeats)

for (i in seq_len(n_repeats)) {
  
  split_i <- initial_split(airbnb_preprocessed, prop = 0.8)
  train_i <- training(split_i)
  test_i  <- testing(split_i)
  
  model_i <- lm(price ~ ., data = train_i)
  pred_i  <- predict(model_i, newdata = test_i)
  
  eval_i <- tibble(
    price = test_i$price,
    pred  = pred_i
  ) %>%
    filter(!is.na(price), !is.na(pred))
  
  rmse_values[i] <- sqrt(mean((eval_i$price - eval_i$pred)^2))
}

rmse_df <- tibble(
  iteration = seq_len(n_repeats),
  rmse      = rmse_values
)

library(ggplot2)

ggplot(rmse_df, aes(x = rmse)) +
  geom_histogram(bins = 15, fill = "#4C72B0", color = "white") +
  geom_vline(xintercept = mean(rmse_df$rmse), 
             linetype = "dashed", 
             linewidth = 1) +
  labs(
    title = "RMSE Distribution Under Naive Preprocessing",
    subtitle = "Repeated random train–test splits",
    x = "RMSE",
    y = "Count"
  ) +
  theme_minimal(base_size = 12)

7 A Leakage-Free Preprocessing Pipeline

To assess whether the previously observed behavior is driven by improper preprocessing, we now reconstruct the entire workflow using a leakage-free pipeline. The key principle is simple but fundamental: any transformation that learns from the data must be estimated using the training set only and then applied to the test set without re-estimation.

set.seed(123)

n_repeats <- 30
rmse_correct <- numeric(n_repeats)

for (i in seq_len(n_repeats)) {
  
  # Split first
  split_i <- initial_split(airbnb_model, prop = 0.8)
  train_raw <- training(split_i)
  test_raw  <- testing(split_i)
  
  # Estimate preprocessing parameters on training data only
  train_processed <- train_raw %>%
    mutate(across(
      .cols = -price,
      .fns  = ~ ifelse(is.na(.x), mean(.x, na.rm = TRUE), .x)
    ))
  
  scaling_params <- train_processed %>%
    summarise(across(-price, list(mean = mean, sd = sd), na.rm = TRUE))
  
  scale_train <- function(x, m, s) {
    ifelse(s > 0, (x - m) / s, 0)
  }
  
  for (v in names(train_processed)[names(train_processed) != "price"]) {
    m <- scaling_params[[paste0(v, "_mean")]]
    s <- scaling_params[[paste0(v, "_sd")]]
    
    train_processed[[v]] <- scale_train(train_processed[[v]], m, s)
  }
  
  # Apply the same transformations to the test set
  test_processed <- test_raw %>%
    mutate(across(
      .cols = -price,
      .fns  = ~ ifelse(is.na(.x), mean(train_raw[[cur_column()]], na.rm = TRUE), .x)
    ))
  
  for (v in names(test_processed)[names(test_processed) != "price"]) {
    m <- scaling_params[[paste0(v, "_mean")]]
    s <- scaling_params[[paste0(v, "_sd")]]
    
    test_processed[[v]] <- scale_train(test_processed[[v]], m, s)
  }
  
  # Fit model
  model_i <- lm(price ~ ., data = train_processed)
  pred_i  <- predict(model_i, newdata = test_processed)
  
  # Evaluate where target is observed
  eval_i <- tibble(
    price = test_processed$price,
    pred  = pred_i
  ) %>%
    filter(!is.na(price), !is.na(pred))
  
  rmse_correct[i] <- sqrt(mean((eval_i$price - eval_i$pred)^2))
}

rmse_correct_df <- tibble(
  iteration = seq_len(n_repeats),
  rmse      = rmse_correct
)

rmse_compare <- bind_rows(
  rmse_df %>% mutate(pipeline = "Naive preprocessing"),
  rmse_correct_df %>% mutate(pipeline = "Leakage-free preprocessing")
)

ggplot(rmse_compare, aes(x = rmse, fill = pipeline)) +
  geom_histogram(position = "identity", alpha = 0.6, bins = 15) +
  labs(
    title = "RMSE Distributions Under Different Preprocessing Pipelines",
    subtitle = "Naive vs leakage-free evaluation",
    x = "RMSE",
    y = "Count"
  ) +
  theme_minimal(base_size = 12)

8 Conclusion

This article set out with a seemingly straightforward question: can data leakage lead to misleadingly strong model performance? The empirical results presented here suggest a more nuanced answer. In the examined setting—using a simple linear model and a highly heterogeneous real-world dataset—improper preprocessing did not result in dramatic or easily detectable performance inflation. Naive and leakage-free pipelines produced nearly identical error distributions.

However, this outcome does not diminish the importance of data leakage. On the contrary, it highlights its most insidious characteristic: data leakage is dangerous precisely because it does not always announce itself through obvious performance gains. Evaluation metrics may remain unchanged, stable, or even reasonable, while the underlying logic of the evaluation has already been violated.

The central lesson is therefore not about performance optimization, but about validity. Correct model evaluation is a matter of respecting information boundaries—temporal, logical, and structural—regardless of whether immediate numerical consequences are visible. Relying on empirically convenient shortcuts simply because they “seem to work” risks building pipelines that fail silently when transferred to new data, different models, or operational settings.

Ultimately, data leakage should be treated as a methodological error, not a performance issue. Thinking carefully about preprocessing order, information flow, and evaluation design is not optional; it is a prerequisite for trustworthy statistical modeling.

9 References

• Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer.
  https://doi.org/10.1007/978-0-387-84858-7

• Kuhn, M., & Johnson, K. (2013). Applied Predictive Modeling. Springer.
  https://doi.org/10.1007/978-1-4614-6849-3

• Kuhn, M., & Silge, J. (2022). Tidy Modeling with R. O’Reilly Media.
  https://www.tmwr.org/

• Scikit-learn documentation. (n.d.). Common pitfalls in machine learning.
  https://scikit-learn.org/stable/common_pitfalls.html

• Inside Airbnb. (n.d.). Get the data.
  https://insideairbnb.com/get-the-data/

1 Introduction

This article is part of a broader series on data preprocessing in R. In earlier posts, we focused on two problems that quietly ruin analyses long before modeling begins: missing data and outliers. Both topics shared a common theme: preprocessing choices are not cosmetic; they change what the model is allowed to learn. In this installment, we move to the next decision point in the same pipeline: normalization (scaling)—often treated as “just a quick step,” but in practice a decisive modeling choice.

Related posts in this preprocessing series

• Handling Missing Data in R: A Comprehensive Guide
  https://medium.com/r-evolution/handling-missing-data-in-r-a-comprehensive-guide-eca195eaead3

• Outliers in Data Analysis: Detecting Extreme Values Before Modeling in R
  https://medium.com/r-evolution/outliers-in-data-analysis-detecting-extreme-values-before-modeling-in-r-with-i%CC%87stanbul-airbnb-data-3b37e9ee989e

Normalization (or more broadly, scaling) is frequently presented as a minor technical adjustment—something to apply quickly and forget. In practice, scaling is not a technical detail but a modeling decision. When the same dataset is processed using different scaling strategies, the behavior of many models changes substantially. Distances, similarity measures, penalty terms, and optimization paths are all affected. As a result, the nearest neighbors selected by KNN, the clusters formed by K-means, the principal components identified by PCA, and even the coefficients chosen by Ridge or Lasso regression can differ. Scaling does not merely “prepare” the data; it actively shapes how a model interprets importance and structure.

More importantly, scaling is not universally beneficial. Applied in the wrong context, it can degrade model performance or—worse—introduce subtle forms of data leakage that contaminate evaluation. A common example is learning scaling parameters (such as means and standard deviations) from the entire dataset before splitting into training and test sets. This procedure allows information from the test distribution to leak into the training process, producing performance estimates that cannot be trusted. In such cases, the issue is not the scaling method itself, but when and how it is applied. Knowing how to call scale() in R is trivial; understanding what to scale, when to scale it, and why is not.

In this article, normalization is treated as an integral part of the modeling strategy rather than a routine preprocessing step. We will address, step by step, the following questions:

• Why is normalization necessary?
• Should it always be applied?
• At what stage should it be performed—before or after the train–test split?
• Which scaling methods are commonly used, and in which contexts do they make sense?
• Should different data types be treated differently?
• Is scaling appropriate for all variables, including the target variable?

By combining conceptual discussion with practical R implementations, this guide aims to provide clear and principled answers to each of these questions.

2 Normalization vs. Standardization: Clearing Up the Terminology

In both academic writing and everyday practice, the terms normalization and standardization are frequently used interchangeably. This loose usage is one of the main sources of confusion in data preprocessing. In reality, these terms refer to different scaling strategies, each with distinct assumptions, effects, and use cases. Before discussing when and how scaling should be applied, it is therefore essential to clarify what is actually meant by each approach.

Standardization, often referred to as z-score scaling, rescales a variable so that it has a mean of zero and a standard deviation of one. Formally, each observation is transformed by subtracting the sample mean and dividing by the sample standard deviation. In the R ecosystem, this logic is implemented in preprocessing tools such as step_normalize() from the recipes package. Standardization preserves the shape of the original distribution while putting variables on a comparable scale. It is particularly useful for models that are sensitive to the relative magnitude of predictors, such as linear models with regularization, support vector machines, and neural networks.

Normalization, in a stricter sense, often refers to min–max scaling. This approach rescales variables to lie within a fixed interval, most commonly [0,1]. Each value is transformed based on the minimum and maximum observed in the training data. Min–max scaling is easy to interpret and is frequently used in algorithms where bounded inputs are desirable. However, it is also more sensitive to extreme values, since a single outlier can heavily influence the scaling range.

A third commonly used approach is robust scaling, which relies on the median and the interquartile range (IQR) instead of the mean and standard deviation. By construction, this method is less affected by outliers and heavy-tailed distributions. Robust scaling is especially useful in real-world datasets where extreme values are not errors but genuine observations. At the same time, it is not a universal solution; in some data structures, robust measures may become unstable or uninformative.

The reason terminology becomes blurred in practice is simple: many practitioners use the word normalization as a generic label for “any kind of scaling.” As a result, two people may both say they normalized their data while having applied entirely different transformations. Throughout this article, we will avoid this ambiguity by explicitly stating which scaling method is used and why. This distinction is not pedantic—it is essential for understanding how scaling choices influence model behavior.

3 Why Is Normalization Necessary?

The necessity of normalization becomes clear once we recognize that many modeling techniques do not operate on raw variable values directly, but on relationships derived from them—such as distances, similarities, penalties, or variance directions. When predictors are measured on different scales, these derived quantities can be dominated by variables with larger numerical ranges, regardless of their substantive importance. In such cases, the model does not learn from the data structure itself, but from arbitrary measurement units.

This issue is most apparent in distance-based methods such as k-nearest neighbors (KNN) and K-means clustering. These algorithms rely explicitly on distance calculations, typically Euclidean distance. If one variable ranges between 0 and 1 while another ranges between 0 and 10,000, the latter will dominate the distance computation almost entirely. As a result, proximity is determined not by overall similarity but by the scale of a single variable. Normalization ensures that each predictor contributes to the distance metric in a balanced and interpretable way, allowing the algorithm to reflect genuine similarity rather than numerical magnitude.

Normalization is equally critical in models that incorporate regularization, such as Ridge and Lasso regression. In these models, coefficients are penalized to control model complexity. However, the penalty term is directly tied to the scale of the predictors. If variables are not on comparable scales, the regularization mechanism will shrink coefficients unevenly, effectively penalizing some predictors more than others for reasons unrelated to their predictive relevance. Scaling aligns the predictors so that regularization operates as intended: as a constraint on model complexity rather than an artifact of measurement units.

Other widely used techniques—including support vector machines (SVMs), neural networks, and principal component analysis (PCA)—are also highly sensitive to scaling. In SVMs and neural networks, optimization procedures depend on gradients that are influenced by feature magnitudes, affecting both convergence speed and stability. In PCA, the directions of maximum variance are determined by the scale of the variables; without normalization, components may simply reflect variables with the largest variances rather than the most informative underlying structure. In all these cases, scaling is not an optional refinement but a prerequisite for meaningful model behavior.

By contrast, tree-based models such as decision trees, random forests, and gradient boosting machines are generally invariant to monotonic transformations of individual predictors. Since splits are based on ordering rather than distance or magnitude, scaling is often unnecessary for these methods. Nevertheless, this does not imply that normalization is universally irrelevant in tree-based pipelines. Hybrid workflows—where tree-based models are combined with distance-based components, rule-based similarity measures, or downstream models sensitive to scale—may still require careful consideration of scaling choices. The key point is not that normalization should always be applied, but that it should be applied with respect to the assumptions of the modeling approach.

From a broader perspective, normalization plays a central role in modern predictive modeling workflows. As emphasized in the predictive modeling literature, preprocessing steps are not independent of the model; they are part of the modeling strategy itself. Scaling decisions shape how information is represented and, ultimately, how learning takes place. Understanding why normalization is necessary is therefore a prerequisite for deciding when and how it should be applied—a topic we address next.

4 Should Normalization Always Be Applied?

A natural question at this point is whether normalization should be applied by default in every modeling task. The short answer is no. Normalization is not a universally beneficial preprocessing step; its usefulness depends on the assumptions and internal mechanics of the chosen model. Applying scaling blindly can be as problematic as ignoring it altogether. What is needed is a decision framework that links model characteristics to preprocessing choices.

For a large class of models, normalization is strongly recommended. This group includes distance-based methods such as k-nearest neighbors (KNN) and K-means clustering, as well as techniques like principal component analysis (PCA), support vector machines (SVMs), neural networks, and penalized regression models (Ridge, Lasso, Elastic Net). In all these cases, either distances, inner products, variance directions, or penalty terms play a central role. Without scaling, these mechanisms are dominated by variables with larger numerical ranges, leading to distorted learning behavior. For such models, normalization is not a refinement but a prerequisite for meaningful results.

By contrast, normalization is generally unnecessary for tree-based models such as decision trees, random forests, and gradient boosting machines (e.g., XGBoost, GBM). These models rely on recursive binary splits based on variable ordering rather than on distances or magnitudes. Since monotonic transformations do not affect the relative ordering of values, scaling typically has no impact on model performance. As a result, normalization is often omitted in purely tree-based pipelines without any loss of effectiveness.

Between these two extremes lies a set of models for which normalization is context-dependent. Ordinary linear regression, for example, does not require scaling for estimation itself, but normalization may still be useful for numerical stability, interpretability of coefficients, or comparability across predictors. Similarly, Naive Bayes models may or may not benefit from scaling depending on the assumed feature distributions and the types of variables involved. In these cases, the decision to normalize should be guided by the modeling objective rather than by a fixed rule.

The key takeaway is that normalization should be applied with respect to the model’s assumptions, not as a default preprocessing habit. To make this decision explicit, Table 1 summarizes common modeling approaches and whether normalization is typically required.

5 When Should Normalization Be Applied? Before or After the Train–Test Split?

This is the most critical question in the entire preprocessing workflow—and the point at which many otherwise sound analyses quietly go wrong. The issue is not whether normalization should be applied, but when it should be applied. At the center of this question lies a fundamental concept in predictive modeling: data leakage.

Data leakage occurs when information from outside the training set is used, directly or indirectly, during model training. In the context of normalization, leakage typically arises when scaling parameters—such as means and standard deviations (for standardization) or minimum and maximum values (for min–max scaling)—are estimated using the full dataset before splitting into training and test sets. Although this may appear harmless, it allows information from the test set to influence the preprocessing step, leading to overly optimistic performance estimates.

The correct principle is straightforward but non-negotiable:
scaling parameters must be learned exclusively from the training data.
Once learned, the same transformation—with fixed parameters—must be applied to the test set and to any future, unseen data. This ensures that the test set truly represents new information and that model evaluation reflects genuine generalization rather than procedural artifacts.

This principle is central to modern modeling frameworks. In the tidymodels/recipes philosophy, preprocessing steps are trained on the training data and then applied consistently to all other datasets. Similarly, in the caret framework, preprocessing transformations are estimated from the training set and reused when predicting on new data. In both cases, preprocessing is treated as part of the model training process—not as an independent, preliminary operation.

To see why this distinction matters, consider the following conceptual comparison.

6 Common Normalization Methods and When to Use Them

Normalization is not a single technique but a family of transformations, each designed to address a specific modeling concern. Choosing an appropriate method requires understanding what problem the transformation is solving and which assumptions it implicitly makes. In this section, we review the most commonly used scaling approaches, discuss their strengths and limitations, and clarify when each method is appropriate.

7 Do Different Data Types Require Different Scaling Strategies?

Normalization decisions should never be made independently of data types. Different variable types carry different semantic meanings, and applying the same scaling strategy indiscriminately can lead to misleading representations or unnecessary transformations. A principled preprocessing workflow therefore begins by distinguishing between variable types and understanding how each interacts with scaling.

8 Should All Variables Be Scaled?

A common mistake in preprocessing workflows is to treat normalization as a blanket operation applied to every variable in the dataset. In reality, not all variables should be scaled, and doing so indiscriminately can reduce interpretability or even introduce unintended distortions. Scaling decisions must therefore be made at the variable level, guided by both statistical and semantic considerations.

9 Application Plan in R: Data and Modeling Scenario

To demonstrate the practical implications of normalization decisions, we use the Ames Housing dataset, a well-known benchmark dataset designed for predictive modeling. The dataset contains 2,930 observations and a rich set of predictors describing residential properties in Ames, Iowa. These predictors span multiple data types, including continuous numeric variables, discrete counts, ordinal ratings, and categorical features. This diversity makes the dataset particularly suitable for illustrating how scaling interacts with different variable types.

The Ames Housing dataset is distributed within the modeldata package in the tidymodels ecosystem. It was explicitly curated for teaching and methodological demonstrations, ensuring a realistic but well-documented structure. The presence of variables measured on vastly different scales—such as living area, lot size, and quality scores—provides a natural setting for exploring the effects of normalization.

10 Implementation in R: Split, Baseline, and the Cost of Doing It Wrong

In this section, we operationalize the key principle introduced earlier:

Split → fit preprocessing on train → apply to train/test

We use the Ames Housing dataset from the modeldata package (no external download required) and compare three pipelines using the same model:

1. Baseline (no scaling)
2. Incorrect scaling (data leakage): scaling parameters learned from the full dataset
3. Correct scaling: scaling parameters learned from the training set only

The goal is not to build the best possible model but to isolate the effect of scaling decisions.

library(tidymodels)
library(modeldata)

data(ames, package = "modeldata")

set.seed(2026)

ames_small <- ames %>%
  dplyr::select(
    Sale_Price,
    Gr_Liv_Area,
    Lot_Area,
    Year_Built,
    Overall_Cond,
    Latitude,
    Longitude
  )

# Missing-value check within the selected columns
ames_small %>%
  summarise(across(everything(), ~ sum(is.na(.)))) %>%
  tidyr::pivot_longer(everything(), names_to = "variable", values_to = "n_missing")

# A tibble: 7 × 2
  variable     n_missing
  <chr>            <int>
1 Sale_Price           0
2 Gr_Liv_Area          0
3 Lot_Area             0
4 Year_Built           0
5 Overall_Cond         0
6 Latitude             0
7 Longitude            0

set.seed(2026)

split_obj <- initial_split(ames_small, prop = 0.80, strata = Sale_Price)

train_data <- training(split_obj)
test_data  <- testing(split_obj)

nrow(train_data)

[1] 2342

nrow(test_data)

[1] 588

-    training: 2,342 rows

-    test: 588 rows

bind_rows(
  train_data %>% mutate(split = "train"),
  test_data  %>% mutate(split = "test")
) %>%
  ggplot(aes(x = Sale_Price, fill = split)) +
  geom_histogram(bins = 40, alpha = 0.7, color = "white") +
  facet_wrap(~ split, scales = "free_y") +
  scale_fill_manual(
    values = c(train = "#1f77b4", test = "#ff7f0e")
  ) +
  labs(
    title = "Sale_Price distribution after train–test split",
    x = "Sale_Price",
    y = "Count",
    fill = "Data split"
  ) +
  theme_minimal()

train_summary <- train_data %>%
summarise(
split = "train",
n = n(),
mean = mean(Sale_Price),
median = median(Sale_Price),
sd = sd(Sale_Price),
min = min(Sale_Price),
max = max(Sale_Price)
)

test_summary <- test_data %>%
summarise(
split = "test",
n = n(),
mean = mean(Sale_Price),
median = median(Sale_Price),
sd = sd(Sale_Price),
min = min(Sale_Price),
max = max(Sale_Price)
)

bind_rows(train_summary, test_summary)

# A tibble: 2 × 7
  split     n    mean median     sd   min    max
  <chr> <int>   <dbl>  <dbl>  <dbl> <int>  <int>
1 train  2342 180447. 160000 79157. 12789 755000
2 test    588 182185. 160500 82784. 35311 625000

knn_spec <- nearest_neighbor(
  neighbors = 15,
  weight_func = "rectangular"
) %>%
  set_engine("kknn") %>%
  set_mode("regression")

rec_none <- recipe(Sale_Price ~ ., data = train_data)

wf_none <- workflow() %>%
add_recipe(rec_none) %>%
add_model(knn_spec)

fit_none <- fit(wf_none, data = train_data)

pred_none <- predict(fit_none, test_data) %>%
bind_cols(test_data %>% dplyr::select(Sale_Price))

metrics_none <- yardstick::metrics(
pred_none,
truth = Sale_Price,
estimate = .pred
)

metrics_none

# A tibble: 3 × 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 rmse    standard   35643.   
2 rsq     standard       0.816
3 mae     standard   23726.   

rec_leak <- recipe(Sale_Price ~ ., data = ames_small) %>%
  step_normalize(all_numeric_predictors())

# WRONG on purpose: prepping on full data (leakage), but now type-safe
prep_leak <- prep(rec_leak, training = ames_small)

train_leak <- bake(prep_leak, new_data = train_data)
test_leak  <- bake(prep_leak, new_data = test_data)

wf_leak <- workflow() %>%
  add_model(knn_spec) %>%
  add_formula(Sale_Price ~ .)

fit_leak <- fit(wf_leak, data = train_leak)

pred_leak <- predict(fit_leak, test_leak) %>%
  bind_cols(test_leak %>% dplyr::select(Sale_Price))

metrics_leak <- yardstick::metrics(pred_leak, truth = Sale_Price, estimate = .pred)
metrics_leak

# A tibble: 3 × 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 rmse    standard   37036.   
2 rsq     standard       0.801
3 mae     standard   24411.   

rec_ok <- recipe(Sale_Price ~ ., data = train_data) %>%
  step_normalize(all_numeric_predictors())

wf_ok <- workflow() %>%
  add_recipe(rec_ok) %>%
  add_model(knn_spec)

fit_ok <- fit(wf_ok, data = train_data)

pred_ok <- predict(fit_ok, test_data) %>%
  bind_cols(test_data %>% dplyr::select(Sale_Price))

metrics_ok <- yardstick::metrics(pred_ok, truth = Sale_Price, estimate = .pred)

metrics_ok

# A tibble: 3 × 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 rmse    standard   35643.   
2 rsq     standard       0.816
3 mae     standard   23726.   

results_tbl <- dplyr::bind_rows(
  metrics_none %>% mutate(scenario = "A — No Scaling"),
  metrics_leak %>% mutate(scenario = "B — Incorrect Scaling (Leakage)"),
  metrics_ok   %>% mutate(scenario = "C — Correct Scaling (Train-Only)")
) %>%
  dplyr::select(scenario, .metric, .estimate) %>%
  tidyr::pivot_wider(
    names_from = .metric,
    values_from = .estimate
  )

results_tbl

# A tibble: 3 × 4
  scenario                           rmse   rsq    mae
  <chr>                             <dbl> <dbl>  <dbl>
1 A — No Scaling                   35643. 0.816 23726.
2 B — Incorrect Scaling (Leakage)  37036. 0.801 24411.
3 C — Correct Scaling (Train-Only) 35643. 0.816 23726.

results_tbl %>%
ggplot(aes(x = scenario, y = rmse, fill = scenario)) +
geom_col(alpha = 0.8) +
scale_fill_manual(
values = c(
"A — No Scaling" = "#1f77b4",
"B — Incorrect Scaling (Leakage)" = "#d62728",
"C — Correct Scaling (Train-Only)" = "#2ca02c"
)
) +
labs(
title = "RMSE comparison across preprocessing scenarios",
x = "Preprocessing scenario",
y = "RMSE"
) +
theme_minimal() +
theme(legend.position = "none")

11 Discussion and Conclusion

Normalization is often introduced as a routine preprocessing step, applied almost reflexively before modeling. This article has argued—and demonstrated—that such a view is incomplete. Normalization is not a purely technical adjustment; it is a modeling decision whose consequences depend on the interaction between data, model assumptions, and evaluation design.

From a theoretical perspective, scaling matters because many learning algorithms are sensitive to the relative magnitudes of predictors. Distance-based methods, regularized models, kernel methods, and optimization-driven algorithms implicitly encode assumptions about scale. Ignoring these assumptions can distort model behavior, while respecting them can improve stability and interpretability. At the same time, scaling does not create new information. It reshapes how existing information is represented.

The empirical application using the Ames Housing dataset reinforced these points. By holding the model and data split constant and varying only the preprocessing strategy, we isolated the effect of normalization decisions. Three key findings emerged.

First, normalization does not guarantee performance improvements. In the correct workflow, scaling reproduced the baseline results exactly. This confirms that normalization should not be expected to “fix” a model by itself. Its role is conditional and context-dependent.

Second, incorrect normalization compromises validity. Learning scaling parameters from the full dataset—thereby introducing data leakage—altered model behavior and degraded performance in this example. More importantly, even if the metrics had improved, the evaluation would have been invalid. Leakage undermines the fundamental purpose of a test set: to approximate unseen data.

Third, the timing of preprocessing is as important as the method chosen. The difference between valid and invalid evaluation hinged not on whether scaling was applied, but on when its parameters were learned. This distinction is often overlooked in practice, yet it is central to trustworthy modeling.

Taken together, these results support a broader principle: preprocessing steps should be treated as integral components of the modeling pipeline, not as detached technical preliminaries. Decisions about normalization should be guided by model assumptions, data characteristics, and evaluation design—not by habit or generic checklists.

In practical terms, this leads to a simple but robust rule:

Split the data first. Learn preprocessing parameters from the training set only. Apply the same transformations to all future data.

Normalization, when used deliberately and correctly, is a powerful tool. When applied mechanically or at the wrong stage, it can mislead. Understanding this distinction is essential for building models that are not only accurate, but also scientifically defensible.

12 References

• Hastie, T., Tibshirani, R., & Friedman, J. (2009).
  The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer.

• Kuhn, M., & Johnson, K. (2013).
  Applied Predictive Modeling. Springer.

• Kuhn, M., & Wickham, H. (2023).
  Tidymodels: A Collection of Packages for Modeling and Machine Learning Using Tidyverse Principles.
  https://www.tidymodels.org/

• Tidymodels Recipes Documentation.
  https://recipes.tidymodels.org/

• Kuhn, M. (Caret package documentation).
  https://topepo.github.io/caret/

• Modeldata package documentation (Ames Housing dataset).
  https://modeldata.tidymodels.org/reference/ames.html

1 Introduction

When learning R, most people focus on functions, models, and visualizations. However, many real-world problems start much earlier — at the data import stage — and end much later — with exporting results.

If data is read incorrectly, no statistical method can save the analysis.

In this post, we focus on the logic of data import and export in R, using CSV and Excel files. Rather than memorizing functions, we build a mental model for how R interacts with files.

2 Why Data Import and Export Matters

Data analysis is a workflow:

Data source → Import → Analysis → Results → Export → Sharing

Errors often occur at the import stage:

• wrong delimiters,

• incorrect decimal separators,

• incorrect file paths,

• silently converted data types.

The result?
A model that runs perfectly — on the wrong data.

3 CSV vs Excel: Not a Competition

Before touching R, we should clarify the difference between file formats.

total_bill,tip,sex
16.99,1.01,Female

4 Working Directory: Where R Actually Looks

One of the most common beginner mistakes has nothing to do with R syntax.

R does not search your entire computer for files. It only looks inside its working directory.

getwd()

This command shows where R is currently looking.

If a file exists on your computer but not in this directory, R behaves as if the file does not exist.

This is why errors like:

cannot open the connection

usually indicate a path problem, not a coding problem.

5 The Example Dataset: tips

Throughout this post, we use a single dataset: tips.

• Restaurant tipping data

• Small and easy to understand

• Contains numeric and categorical variables

• Ideal for demonstrating import/export logic

Data source:
https://raw.githubusercontent.com/mwaskom/seaborn-data/master/tips.csv

6 Reading CSV Files: The Core Logic

When R reads a CSV file, it needs answers to four questions:

1. How are columns separated?

2. Is the first row a header?

3. What is the decimal separator?

4. How should text be interpreted?

These answers are provided via function arguments.

7 read.table(): The Foundation

All CSV-reading functions in base R are built on read.table().

tips <- read.table(
  file = "tips.csv",
  header = TRUE,
  sep = ",",
  dec = ".",
  stringsAsFactors = FALSE
)

Understanding this function means understanding CSV import in R.

8 read.csv() and Its Assumptions

read.csv() is simply a shortcut for a common case:

• Columns separated by commas

• Decimal separator is a dot

tips <- read.csv("tips.csv")

This works perfectly — if the assumptions match the file.

The dangerous part? R may not throw an error even if the assumptions are wrong.

The most dangerous errors are silent ones.

9 read.csv2() and Regional Differences

In many European datasets:

• Columns are separated by semicolons

• Decimals use commas

total_bill;tip;sex
16,99;1,01;Female

For this structure, read.csv2() is designed.

tips2 <- read.csv2("tips_semicolon.csv")

Important nuance:
Even if decimals use dots, read.csv2() may still work in some cases — but this is not guaranteed.

Correct approach:

Always inspect the file structure before choosing the function.

10 Writing CSV Files from R

Data analysis rarely ends in R. Results are shared as files.

write.csv(tips, "tips_comma.csv", row.names = FALSE)

write.csv2(tips, "tips_semicolon.csv", row.names = FALSE)

11 Why We Still Need Excel

CSV is technically superior in many ways. Yet Excel remains dominant in practice.

Why?

• Multiple tables in one file

• Familiar interface for non-technical users

• Common reporting format

Excel is not an analysis tool — but it is a powerful delivery tool.

12 Working with Excel in R: openxlsx

The openxlsx package allows Excel operations without requiring Excel itself.

library(openxlsx)

write.xlsx(tips, "tips.xlsx", sheetName = "tips")

tips_excel <- read.xlsx("tips.xlsx", sheet = 1)

13 Multiple Sheets: A Mini Report

Excel shines when organizing related tables.

summary_tips <- aggregate(tip ~ day, data = tips, mean)

wb <- createWorkbook()

addWorksheet(wb, "Raw Data")
writeData(wb, "Raw Data", tips)

addWorksheet(wb, "Summary")
writeData(wb, "Summary", summary_tips)

saveWorkbook(wb, "tips_report.xlsx", overwrite = TRUE)

One file.

Multiple views.

Clean structure.

14 Common Mistakes to Watch For

Most errors are not caused by R, but by assumptions:

• Incorrect working directory

• Wrong delimiter (sep)

• Wrong decimal separator (dec)

• Reading the wrong Excel sheet

• Overwriting files unintentionally

A healthy habit after every import:

head(data)
str(data)
summary(data)

15 Final Thoughts

If you can:

• read data correctly,

• write data consciously,

• choose file formats intentionally,

you have already crossed one of the most important thresholds in data analysis.

For a complementary discussion, you may also find this article useful:
https://medium.com/p/e730f4a84b3b

Extended version on Medium:
https://medium.com/@Fatih.Tuzen/understanding-data-import-and-export-in-r-working-with-csv-and-excel-files-6322e61049b2

1 Introduction

Data preprocessing is often presented as a sequence of technical steps. However, each preprocessing decision implicitly embeds a statistical assumption.

In a previous article, I discussed how missing observations can bias analysis if they are ignored or handled improperly:

Handling Missing Data in R: A Comprehensive Guide

This article continues that discussion by focusing on outliers. Unlike missing values, outliers are observed data points. The challenge is not their absence, but their extremeness.

Understanding whether an extreme value is informative or misleading is a crucial step before any modeling effort.

2 Why Outliers Matter

Outliers can affect statistical analysis in several fundamental ways:

• They distort summary statistics such as the mean and standard deviation
• They can dominate parameter estimates in regression models
• They influence distance-based methods such as clustering

More importantly, outliers force analysts to confront a key question:

Are we observing rare but valid behavior, or a deviation from the assumed data-generating process?

3 What Is an Outlier?

Informally, an outlier is an observation that appears unusually large or small relative to the rest of the data. Formally, an outlier is an observation that is inconsistent with the bulk of the data under a given statistical model. Outliers are therefore not absolute objects. They depend on assumptions about distribution, scale, and structure.

4 The Dataset: Inside Airbnb Listings (Istanbul)

To demonstrate outlier detection methods, we will use Inside Airbnb listings data. Inside Airbnb is a mission-driven project that publishes datasets scraped from publicly available Airbnb listing pages and provides city-level downloads for research and analysis.

In this article, we will work with the detailed listings file:

• listings.csv.gz (detailed listing-level data; typically rich and feature-complete)

You can download the dataset from the Inside Airbnb “Get the Data” page (choose a city and download Detailed Listings data).

library(dplyr)
library(readr)
library(stringr)

# Example URL (Istanbul). You can get the latest link from Inside Airbnb "Get the Data".
# The URL structure typically follows:
# https://data.insideairbnb.com/turkey/marmara/istanbul/2025-09-29/data/listings.csv.gz

listings_raw <- read_csv(
  "listings.csv.gz",
  show_col_types = FALSE
)

vars_keep <- c(
  "id", "name",
  "price", "minimum_nights", "number_of_reviews",
  "room_type", "neighbourhood_cleansed"
)

listings_small <- listings_raw %>%
  select(any_of(vars_keep))

glimpse(listings_small)

Rows: 30,051
Columns: 7
$ id                     <dbl> 1.342043e+18, 1.342082e+18, 1.342211e+18, 1.342…
$ name                   <chr> "Отдельная квартира на Фатих(Балат).", "Blue st…
$ price                  <chr> "$2,290.00", "$1,101.00", "$3,430.00", "$3,178.…
$ minimum_nights         <dbl> 5, 7, 2, 100, 1, 100, 1, 5, 2, 100, 100, 2, 100…
$ number_of_reviews      <dbl> 4, 4, 26, 1, 2, 0, 41, 0, 19, 0, 0, 26, 0, 0, 1…
$ room_type              <chr> "Entire home/apt", "Private room", "Entire home…
$ neighbourhood_cleansed <chr> "Fatih", "Beyoglu", "Beyoglu", "Sisli", "Sisli"…

listings_small <- listings_small %>%
  mutate(price_num = price %>%
           str_replace_all("[^0-9.]", "") %>%
           as.numeric())

summary(listings_small$price_num)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
     80    1644    2538    5084    4108 4437598    4803 

5 Visualizing Price Distributions and Potential Outliers

Before applying any formal outlier detection rule, it is good practice to explore the distribution of the variable visually. Visualization helps us understand the shape, spread, and asymmetry of the data, and often reveals extreme values immediately.

In this section, we focus on the numeric price variable price_num.

library(ggplot2)
library(scales)

ggplot(listings_small, aes(x = price_num)) +
  geom_histogram(bins = 40, fill = "#B0B0B0", color = "white") +
  scale_x_continuous(labels = label_number(big.mark = ",")) +
  labs(
    title = "Distribution of Nightly Prices (raw scale)",
    x = "Nightly price",
    y = "Count"
  ) +
  theme_minimal(base_size = 13)

listings_small %>%
  count(room_type, sort = TRUE)

# A tibble: 4 × 2
  room_type           n
  <chr>           <int>
1 Entire home/apt 20243
2 Private room     9494
3 Hotel room        157
4 Shared room       157

ggplot(listings_small, aes(x = price_num)) +
geom_histogram(bins = 35, fill = "#4C72B0", color = "white", alpha = 0.9) +
scale_x_log10(
breaks = log_breaks(n = 6),
labels = label_number(big.mark = ",")
) +
facet_wrap(~ room_type, scales = "free_y") +
labs(
title = "Nightly Price Distributions by Room Type (log scale)",
subtitle = "Log scale improves readability in heavily right-skewed price data",
x = "Nightly price (log scale)",
y = "Count"
) +
theme_minimal(base_size = 13)

ggplot(listings_small, aes(x = room_type, y = price_num)) +
  geom_boxplot(outlier.alpha = 0.35, fill = "#DDDDDD") +
  scale_y_log10(labels = label_number(big.mark = ",")) +
  labs(
    title = "Nightly Prices by Room Type (boxplot, log scale)",
    subtitle = "Potential outliers are assessed within each room type",
    x = "Room type",
    y = "Nightly price (log scale)"
  ) +
  theme_minimal(base_size = 13) +
  theme(axis.text.x = element_text(angle = 20, hjust = 1))

6 Formal Outlier Detection Within Room Type

Visual exploration suggested that nightly prices exhibit strong right-skewness and that extreme values should be interpreted within the context of room_type. In this section, we formalize that intuition using statistical outlier detection rules.

Our goal is not to mechanically remove observations, but to identify and examine listings whose prices are unusually high relative to their own room type.

outliers_iqr <- listings_small %>%
  group_by(room_type) %>%
  mutate(
    Q1 = quantile(price_num, 0.25, na.rm = TRUE),
    Q3 = quantile(price_num, 0.75, na.rm = TRUE),
    IQR_value = Q3 - Q1,
    lower_bound = Q1 - 1.5 * IQR_value,
    upper_bound = Q3 + 1.5 * IQR_value,
    outlier_iqr = price_num < lower_bound | price_num > upper_bound
  ) %>%
  ungroup()

outliers_iqr %>%
  count(room_type, outlier_iqr) %>%
  arrange(room_type, desc(outlier_iqr))

# A tibble: 12 × 3
   room_type       outlier_iqr     n
   <chr>           <lgl>       <int>
 1 Entire home/apt TRUE         1458
 2 Entire home/apt FALSE       16616
 3 Entire home/apt NA           2169
 4 Hotel room      TRUE           17
 5 Hotel room      FALSE         106
 6 Hotel room      NA             34
 7 Private room    TRUE          441
 8 Private room    FALSE        6470
 9 Private room    NA           2583
10 Shared room     TRUE            3
11 Shared room     FALSE         137
12 Shared room     NA             17

top_price_outliers <- outliers_iqr %>%
  filter(outlier_iqr) %>%
  group_by(room_type) %>%
  arrange(desc(price_num)) %>%
  slice_head(n = 5) %>%
  ungroup() %>%
  select(
    room_type,
    price,
    price_num,
    minimum_nights,
    number_of_reviews,
    neighbourhood_cleansed,
    name
  )

top_price_outliers

# A tibble: 18 × 7
   room_type       price         price_num minimum_nights number_of_reviews
   <chr>           <chr>             <dbl>          <dbl>             <dbl>
 1 Entire home/apt $4,437,598.00   4437598            100                14
 2 Entire home/apt $2,658,600.00   2658600            100                 3
 3 Entire home/apt $2,109,690.00   2109690            100                 0
 4 Entire home/apt $2,000,000.00   2000000            100                 0
 5 Entire home/apt $1,250,008.00   1250008            100                 0
 6 Hotel room      $2,439,497.00   2439497              1                 0
 7 Hotel room      $2,439,497.00   2439497              1                 0
 8 Hotel room      $2,439,497.00   2439497              1                 0
 9 Hotel room      $2,439,497.00   2439497              1                 0
10 Hotel room      $2,433,427.00   2433427              1                 0
11 Private room    $390,271.00      390271            365                 1
12 Private room    $390,271.00      390271            365                 0
13 Private room    $390,271.00      390271            365                 0
14 Private room    $390,271.00      390271            100                 0
15 Private room    $390,271.00      390271            100                 0
16 Shared room     $7,221.00          7221              1                 0
17 Shared room     $6,086.00          6086              1                 2
18 Shared room     $5,841.00          5841            100                 0
# ℹ 2 more variables: neighbourhood_cleansed <chr>, name <chr>

outliers_z <- listings_small %>%
  group_by(room_type) %>%
  mutate(
    mean_price = mean(price_num, na.rm = TRUE),
    sd_price   = sd(price_num, na.rm = TRUE),
    z_price    = (price_num - mean_price) / sd_price,
    outlier_z  = abs(z_price) > 3
  ) %>%
  ungroup()

outliers_z %>%
  count(room_type, outlier_z) %>%
  arrange(room_type, desc(outlier_z))

# A tibble: 12 × 3
   room_type       outlier_z     n
   <chr>           <lgl>     <int>
 1 Entire home/apt TRUE         24
 2 Entire home/apt FALSE     18050
 3 Entire home/apt NA         2169
 4 Hotel room      TRUE          5
 5 Hotel room      FALSE       118
 6 Hotel room      NA           34
 7 Private room    TRUE         26
 8 Private room    FALSE      6885
 9 Private room    NA         2583
10 Shared room     TRUE          1
11 Shared room     FALSE       139
12 Shared room     NA           17

top_z_outliers <- outliers_z %>%
  filter(outlier_z) %>%
  group_by(room_type) %>%
  arrange(desc(abs(z_price))) %>%
  slice_head(n = 5) %>%
  ungroup() %>%
  select(
    room_type,
    price,
    price_num,
    z_price,
    minimum_nights,
    number_of_reviews,
    neighbourhood_cleansed,
    name
  )

top_z_outliers

# A tibble: 16 × 8
   room_type       price      price_num z_price minimum_nights number_of_reviews
   <chr>           <chr>          <dbl>   <dbl>          <dbl>             <dbl>
 1 Entire home/apt $4,437,59…   4437598   92.9             100                14
 2 Entire home/apt $2,658,60…   2658600   55.6             100                 3
 3 Entire home/apt $2,109,69…   2109690   44.1             100                 0
 4 Entire home/apt $2,000,00…   2000000   41.8             100                 0
 5 Entire home/apt $1,250,00…   1250008   26.1             100                 0
 6 Hotel room      $2,439,49…   2439497    4.84              1                 0
 7 Hotel room      $2,439,49…   2439497    4.84              1                 0
 8 Hotel room      $2,439,49…   2439497    4.84              1                 0
 9 Hotel room      $2,439,49…   2439497    4.84              1                 0
10 Hotel room      $2,433,42…   2433427    4.83              1                 0
11 Private room    $390,271.…    390271   31.3             365                 1
12 Private room    $390,271.…    390271   31.3             365                 0
13 Private room    $390,271.…    390271   31.3             365                 0
14 Private room    $390,271.…    390271   31.3             100                 0
15 Private room    $390,271.…    390271   31.3             100                 0
16 Shared room     $7,221.00       7221    3.65              1                 0
# ℹ 2 more variables: neighbourhood_cleansed <chr>, name <chr>

comparison_summary <- outliers_iqr %>%
  select(id, room_type, outlier_iqr) %>%
  left_join(
    outliers_z %>% select(id, outlier_z),
    by = "id"
  ) %>%
  count(outlier_iqr, outlier_z)

comparison_summary

# A tibble: 4 × 3
  outlier_iqr outlier_z     n
  <lgl>       <lgl>     <int>
1 FALSE       FALSE     23329
2 TRUE        FALSE      1863
3 TRUE        TRUE         56
4 NA          NA         4803

7 Should Outliers Be Removed?

Detecting outliers does not imply that they should be automatically removed. Outlier detection is a diagnostic step, not a cleaning instruction.

In the context of Airbnb price data, many extreme values correspond to luxury properties, large homes, or special accommodation types. Blindly removing such observations may erase precisely the information that makes the data interesting.

Instead, several alternative strategies should be considered.

8 Final Remarks

Outlier detection is a natural step in the data preprocessing workflow. It typically follows missing data analysis and precedes scaling, transformation, or model fitting.

In this article, the focus was not on eliminating extreme values, but on understanding why they occur. Through visual exploration, context-aware analysis using room_type, and formal detection rules such as the IQR method and Z-scores, we demonstrated that extreme values are not noise by default.

In many real-world datasets, especially those involving prices or economic behavior, extreme values reflect structural heterogeneity rather than data quality issues. Treating them blindly as errors risks discarding meaningful information.

Outliers should therefore be approached as questions posed by the data: Why is this observation extreme? Does it represent a different regime, a rare event, or a distinct subgroup?

Answering these questions requires a combination of statistical tools, domain knowledge, and clear analytical goals. When handled thoughtfully, outlier analysis enhances both the robustness and the interpretability of downstream models.

9 References and Further Reading

• Tukey, J. W. (1977). Exploratory Data Analysis. Addison-Wesley.
  (Foundational reference for boxplots, IQR, and exploratory thinking.)

• Hastie, T., Tibshirani, R., Friedman, J. (2009). The Elements of Statistical Learning. Springer.
  (Statistical foundations and the role of robust methods in modeling.)

• James, G., Witten, D., Hastie, T., Tibshirani, R. (2021). An Introduction to Statistical Learning. Springer.
  (Accessible discussion of preprocessing, transformations, and practical modeling considerations.)

• NIST/SEMATECH e-Handbook of Statistical Methods – Outliers
  https://www.itl.nist.gov/div898/handbook/eda/section3/eda35h.htm
  (Authoritative overview of outlier concepts and detection methods.)

• Wickham, H., Grolemund, G. (2017). R for Data Science. O’Reilly Media.
  (Practical guidance on data exploration, visualization, and preprocessing workflows in R.)

• Inside Airbnb – Get the Data
  https://insideairbnb.com/get-the-data/
  (Data source used in this article; city-level Airbnb listings data.)

• Wickham, H. (2016). ggplot2: Elegant Graphics for Data Analysis. Springer.
  (Principles of layered graphics and effective visualization used throughout the article.)

1 Introduction

Data preprocessing is a cornerstone of any data analysis or machine learning pipeline. Raw data rarely comes in a form ready for direct analysis — it often requires cleaning, transformation, normalization, and careful handling of anomalies. Among these preprocessing tasks, dealing with missing data stands out as one of the most critical and unavoidable challenges.

Missing values appear in virtually every domain: surveys may have skipped questions, administrative registers might contain incomplete records, and clinical trials can suffer from dropout patients. Ignoring these gaps or handling them naively does not just reduce the amount of usable information; it can also introduce bias, decrease statistical power, and ultimately compromise the validity of conclusions. In other words, missing data is not just an inconvenience — it is a methodological problem that demands rigorous attention.

In statistical practice, missingness is often represented as NA (Not Available) in R. However, not all missing values are created equal. Some are missing completely at random, others depend on observed variables, and in some cases, the missingness itself carries meaningful information. Understanding these mechanisms is essential before deciding how to address them. This makes missing data imputation a fundamental part of the broader data preprocessing workflow, alongside tasks such as outlier detection, data normalization, and feature engineering.

In this article, we will cover:

• The theoretical foundations of missing data mechanisms (MCAR, MAR, MNAR).
• How to detect and visualize missing values in R.
• Different strategies for handling missingness, from simple imputation to advanced multiple imputation techniques.
• A practical workflow using the NHANES dataset, widely used in health research, to demonstrate methods in R.
• Best practices, pitfalls, and recommendations for applied data science.

We will use several R packages throughout this tutorial:

• tidyverse: Data wrangling and visualization
• naniar and VIM: Tools for exploring and visualizing missing data
• mice: Multiple imputation by chained equations
• missForest: Random forest–based imputation for nonlinear data

By integrating missing data handling into the larger context of preprocessing, this structured approach will not only help you manage incomplete datasets effectively but also ensure that your entire analytical workflow remains robust, transparent, and reliable.

2 NHANES Dataset

In this section, we will work with the NHANES dataset, which comes from the US National Health and Nutrition Examination Survey.
The dataset includes demographic, examination, and laboratory data collected from thousands of individuals.
Since the full dataset is quite large, we will focus only on a subset of variables that are relevant for preprocessing examples.

Here are the variables we will use:

• ID: Unique identifier for each participant
• Age: Age of the participant
• Gender: Biological sex (male or female)
• BMI: Body Mass Index
• BPSysAve: Average systolic blood pressure
• Diabetes: Whether the participant has been diagnosed with diabetes

Before diving into preprocessing, let’s take a quick look at the structure of these selected variables:

library(NHANES)
library(dplyr)

data("NHANES")

# Select relevant variables
nhanes_sub <- NHANES |> 
  select(ID, Age, Gender, BMI, BPSysAve, Diabetes)

glimpse(nhanes_sub)

Rows: 10,000
Columns: 6
$ ID       <int> 51624, 51624, 51624, 51625, 51630, 51638, 51646, 51647, 51647…
$ Age      <int> 34, 34, 34, 4, 49, 9, 8, 45, 45, 45, 66, 58, 54, 10, 58, 50, …
$ Gender   <fct> male, male, male, male, female, male, male, female, female, f…
$ BMI      <dbl> 32.22, 32.22, 32.22, 15.30, 30.57, 16.82, 20.64, 27.24, 27.24…
$ BPSysAve <int> 113, 113, 113, NA, 112, 86, 107, 118, 118, 118, 111, 104, 134…
$ Diabetes <fct> No, No, No, No, No, No, No, No, No, No, No, No, No, No, No, N…

3 Why Missingness Matters

Missing data is not just an inconvenience — it can distort the statistical conclusions we draw from a dataset.
There are several critical reasons why handling missingness properly is essential:

• Biased results: If the missing values are not random, analyses may systematically misrepresent the population.
• Reduced sample size: Complete-case analysis (simply dropping missing rows) reduces data availability, weakening statistical power.
• Model incompatibility: Many modeling techniques in R (e.g., lm(), glm()) require complete data, and will automatically drop cases with missing values, sometimes silently.

# How many missing in BMI?
sum(is.na(nhanes_sub$BMI))

[1] 366

# Complete-case dataset (dropping missing BMI)
nhanes_complete <- nhanes_sub |> 
  filter(!is.na(BMI))

# Compare sample sizes
nrow(nhanes_sub)     # original sample size

[1] 10000

nrow(nhanes_complete) # after dropping missing BMI

[1] 9634

# Fit regression with complete cases only
model_complete <- lm(BPSysAve ~ BMI + Age + Gender, data = nhanes_complete)
summary(model_complete)

Call:
lm(formula = BPSysAve ~ BMI + Age + Gender, data = nhanes_complete)

Residuals:
    Min      1Q  Median      3Q     Max 
-56.281  -8.652  -0.955   7.560 102.790 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) 90.016503   0.677618  132.84   <2e-16 ***
BMI          0.328076   0.023228   14.12   <2e-16 ***
Age          0.412758   0.008076   51.11   <2e-16 ***
Gendermale   4.346847   0.313476   13.87   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 14.43 on 8483 degrees of freedom
  (1147 observations deleted due to missingness)
Multiple R-squared:  0.2969,    Adjusted R-squared:  0.2966 
F-statistic:  1194 on 3 and 8483 DF,  p-value: < 2.2e-16

4 Missing Data Mechanisms

One of the most crucial aspects of handling missing data is to understand why the data are missing.
The mechanism behind missingness determines whether our chosen method will yield unbiased and efficient estimates.

# Packages we already use
library(dplyr)
library(knitr)

# 1) Overall BMI missingness
nhanes_sub |>
  summarise(pct_missing_BMI = mean(is.na(BMI)) * 100) |>
  mutate(pct_missing_BMI = round(pct_missing_BMI, 1)) |>
  kable(caption = "Overall BMI missingness (%)")

# 2) Does BMI missingness vary by observed variables? (MAR hint)
#    - By Gender
by_gender <- nhanes_sub |>
  group_by(Gender) |>
  summarise(pct_miss_BMI = mean(is.na(BMI)) * 100,
            n = n(), .groups = "drop") |>
  mutate(pct_miss_BMI = round(pct_miss_BMI, 1))

#    - By Age groups (bins)
by_age <- nhanes_sub |>
  mutate(AgeBand = cut(Age, breaks = c(0, 30, 45, 60, Inf),
                       labels = c("<=30", "31–45", "46–60", "60+"),
                       right = FALSE)) |>
  group_by(AgeBand) |>
  summarise(pct_miss_BMI = mean(is.na(BMI)) * 100,
            n = n(), .groups = "drop") |>
  mutate(pct_miss_BMI = round(pct_miss_BMI, 1))

# Show summaries nicely
kable(by_gender, caption = "BMI missingness by Gender (%)")

kable(by_age,    caption = "BMI missingness by Age band (%)")

5 Detecting Missing Data

Before applying any imputation or modeling technique, it is essential to explore the extent and structure of missingness in the dataset. The nhanes_sub data frame, derived from the NHANES dataset, will be used for illustration.

# Count missing values for each variable
nhanes_sub %>%
  summarise(across(everything(), ~ sum(is.na(.)))) 

# A tibble: 1 × 6
     ID   Age Gender   BMI BPSysAve Diabetes
  <int> <int>  <int> <int>    <int>    <int>
1     0     0      0   366     1449      142

sum(complete.cases(nhanes_sub))       # number of complete rows

[1] 8482

sum(!complete.cases(nhanes_sub))      # number of incomplete rows

[1] 1518

library(naniar)
library(ggplot2)

# Visualize missing values by variable
gg_miss_var(nhanes_sub, show_pct = TRUE) +
  labs(title = "Missing Values by Variable in NHANES Subset",
       x = "Variables",
       y = "Proportion of Missing Values")

library(VIM)

aggr(nhanes_sub, numbers = TRUE, prop = FALSE, sortVar = TRUE)

 Variables sorted by number of missings: 
 Variable Count
 BPSysAve  1449
      BMI   366
 Diabetes   142
       ID     0
      Age     0
   Gender     0

library(visdat)

vis_dat(nhanes_sub)

6 Handling Missing Data — Methods

In this section we review the main families of methods, show when each is appropriate, and demonstrate them on nhanes_sub. We will explicitly call out the trade-offs so readers can choose deliberately—not by habit.

# How many rows would we lose if we required complete cases for these variables?
n_total <- nrow(nhanes_sub)
n_cc    <- nhanes_sub |> stats::complete.cases() |> sum()

cbind(
  total_rows    = n_total,
  complete_cases= n_cc,
  lost_rows     = n_total - n_cc,
  lost_pct      = round((n_total - n_cc) / n_total * 100, 1)
)

     total_rows complete_cases lost_rows lost_pct
[1,]      10000           8482      1518     15.2

set.seed(2025)

# Create a median-imputed BMI for illustration (only if BMI is missing)
nh_med <- nhanes_sub |>
  mutate(
    BMI_med = ifelse(is.na(BMI), stats::median(BMI, na.rm = TRUE), BMI)
  )

# Compare how many BMI were imputed
sum(is.na(nhanes_sub$BMI))           # original missing BMI count

[1] 366

sum(is.na(nh_med$BMI_med))           # should be 0

[1] 0

library(ggplot2)

# Compare BMI distribution: complete-case vs median-imputed
p_cc  <- nhanes_sub |>
  filter(!is.na(BMI)) |>
  ggplot(aes(x = BMI)) +
  geom_density() +
  labs(title = "BMI density — complete cases")

p_med <- nh_med |>
  ggplot(aes(x = BMI_med)) +
  geom_density() +
  labs(title = "BMI density — median-imputed")

p_cc; p_med

# kNN imputation with VIM::kNN (works on data frames; chooses donors by similarity)
library(VIM)

# We impute only BMI here; set k=5 as a reasonable starting point.
nh_knn <- nhanes_sub |>
  select(Age, Gender, BMI, BPSysAve, Diabetes) |>
  VIM::kNN(k = 5, imp_var = FALSE)  # imp_var=FALSE avoids extra *_imp columns

# Check imputation effect
sum(is.na(nhanes_sub$BMI))   # original missing BMI

[1] 366

sum(is.na(nh_knn$BMI))       # after kNN (should be 0)

[1] 0

library(mice)

# Create imputations
imp <- mice(nhanes_sub, m = 3, seed = 123)

 iter imp variable
  1   1  BMI  BPSysAve  Diabetes
  1   2  BMI  BPSysAve  Diabetes
  1   3  BMI  BPSysAve  Diabetes
  2   1  BMI  BPSysAve  Diabetes
  2   2  BMI  BPSysAve  Diabetes
  2   3  BMI  BPSysAve  Diabetes
  3   1  BMI  BPSysAve  Diabetes
  3   2  BMI  BPSysAve  Diabetes
  3   3  BMI  BPSysAve  Diabetes
  4   1  BMI  BPSysAve  Diabetes
  4   2  BMI  BPSysAve  Diabetes
  4   3  BMI  BPSysAve  Diabetes
  5   1  BMI  BPSysAve  Diabetes
  5   2  BMI  BPSysAve  Diabetes
  5   3  BMI  BPSysAve  Diabetes

# Look at a summary
imp

Class: mids
Number of multiple imputations:  3 
Imputation methods:
      ID      Age   Gender      BMI BPSysAve Diabetes 
      ""       ""       ""    "pmm"    "pmm" "logreg" 
PredictorMatrix:
         ID Age Gender BMI BPSysAve Diabetes
ID        0   1      1   1        1        1
Age       1   0      1   1        1        1
Gender    1   1      0   1        1        1
BMI       1   1      1   0        1        1
BPSysAve  1   1      1   1        0        1
Diabetes  1   1      1   1        1        0

# Inspect first few imputations for BMI
head(imp$imp$BMI)

        1     2     3
61  43.00 17.70 12.90
161 16.70 13.50 18.59
210 16.28 24.00 23.10
309 24.00 37.32 26.20
310 24.00 28.55 26.30
320 25.10 32.25 29.20

# Extract the first imputed dataset
nhanes_completed <- complete(imp, 1)

head(nhanes_completed)

     ID Age Gender   BMI BPSysAve Diabetes
1 51624  34   male 32.22      113       No
2 51624  34   male 32.22      113       No
3 51624  34   male 32.22      113       No
4 51625   4   male 15.30       92       No
5 51630  49 female 30.57      112       No
6 51638   9   male 16.82       86       No

library(dplyr)
library(missForest)

# Start from the existing subset:
# nhanes_sub <- NHANES |> select(ID, Age, Gender, BMI, BPSysAve, Diabetes)

# 1) Keep only model-relevant columns (drop pure identifier)
# 2) Convert character variables to factors (missForest expects factors, not raw character)
# 3) Coerce to base data.frame to avoid tibble-related method dispatch issues
mf_input <- nhanes_sub |>
  select(Age, Gender, BMI, BPSysAve, Diabetes) |>
  mutate(across(where(is.character), as.factor)) |>
  as.data.frame()

set.seed(123)
mf_fit <- missForest(
  mf_input,
  ntree   = 200,    # more trees -> stabler imputations
  maxiter = 5,      # outer iterations (default 10; 5 is fine for demo)
  verbose = FALSE
)

# Completed data and OOB error
mf_imputed <- mf_fit$ximp
mf_oob     <- mf_fit$OOBerror

# Quick checks
sum(is.na(mf_input$BMI))

[1] 366

sum(is.na(mf_imputed$BMI))   # should go to 0

[1] 0

mf_oob

     NRMSE        PFC 
0.17890307 0.02667884 

7 Single vs. Multiple Imputation

One critical distinction in handling missing data is single imputation vs. multiple imputation (MI).

• Single imputation (mean, median, regression, etc.) fills each missing value once. While simple, it ignores uncertainty, treating imputed values as if they were observed.

• Multiple imputation generates several plausible versions of the dataset (e.g., 5–10). Each dataset is analyzed separately, and results are then combined (pooled). This approach accounts for variability due to missingness and produces more reliable inferences.

Let’s illustrate with our nhanes_sub dataset:

# Complete-case analysis (ignores missing data)
lm_cc <- lm(BMI ~ Age + Gender + BPSysAve + Diabetes,
            data = nhanes_sub, na.action = na.omit)

# Single imputation (mean imputation for BMI)
nhanes_single <- nhanes_sub |> 
  mutate(BMI = ifelse(is.na(BMI), mean(BMI, na.rm = TRUE), BMI))

lm_si <- lm(BMI ~ Age + Gender + BPSysAve + Diabetes,
            data = nhanes_single)

# Multiple imputation with mice
imp <- mice(nhanes_sub, m = 5, method = "pmm", seed = 123)

 iter imp variable
  1   1  BMI  BPSysAve  Diabetes
  1   2  BMI  BPSysAve  Diabetes
  1   3  BMI  BPSysAve  Diabetes
  1   4  BMI  BPSysAve  Diabetes
  1   5  BMI  BPSysAve  Diabetes
  2   1  BMI  BPSysAve  Diabetes
  2   2  BMI  BPSysAve  Diabetes
  2   3  BMI  BPSysAve  Diabetes
  2   4  BMI  BPSysAve  Diabetes
  2   5  BMI  BPSysAve  Diabetes
  3   1  BMI  BPSysAve  Diabetes
  3   2  BMI  BPSysAve  Diabetes
  3   3  BMI  BPSysAve  Diabetes
  3   4  BMI  BPSysAve  Diabetes
  3   5  BMI  BPSysAve  Diabetes
  4   1  BMI  BPSysAve  Diabetes
  4   2  BMI  BPSysAve  Diabetes
  4   3  BMI  BPSysAve  Diabetes
  4   4  BMI  BPSysAve  Diabetes
  4   5  BMI  BPSysAve  Diabetes
  5   1  BMI  BPSysAve  Diabetes
  5   2  BMI  BPSysAve  Diabetes
  5   3  BMI  BPSysAve  Diabetes
  5   4  BMI  BPSysAve  Diabetes
  5   5  BMI  BPSysAve  Diabetes

lm_mi <- with(imp, lm(BMI ~ Age + Gender + BPSysAve + Diabetes))
pooled <- pool(lm_mi)

summary(lm_cc)

Call:
lm(formula = BMI ~ Age + Gender + BPSysAve + Diabetes, data = nhanes_sub, 
    na.action = na.omit)

Residuals:
    Min      1Q  Median      3Q     Max 
-18.771  -4.640  -1.053   3.553  53.003 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) 17.488597   0.513225  34.076   <2e-16 ***
Age          0.047930   0.004273  11.217   <2e-16 ***
Gendermale  -0.278756   0.144799  -1.925   0.0542 .  
BPSysAve     0.067504   0.004903  13.767   <2e-16 ***
DiabetesYes  3.789606   0.265240  14.287   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 6.588 on 8477 degrees of freedom
  (1518 observations deleted due to missingness)
Multiple R-squared:  0.1136,    Adjusted R-squared:  0.1132 
F-statistic: 271.5 on 4 and 8477 DF,  p-value: < 2.2e-16

summary(lm_si)

Call:
lm(formula = BMI ~ Age + Gender + BPSysAve + Diabetes, data = nhanes_single)

Residuals:
    Min      1Q  Median      3Q     Max 
-18.657  -4.634  -1.029   3.533  53.004 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) 17.600406   0.508667  34.601   <2e-16 ***
Age          0.047159   0.004237  11.129   <2e-16 ***
Gendermale  -0.287221   0.143820  -1.997   0.0458 *  
BPSysAve     0.066791   0.004858  13.748   <2e-16 ***
DiabetesYes  3.725941   0.262781  14.179   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 6.57 on 8541 degrees of freedom
  (1454 observations deleted due to missingness)
Multiple R-squared:  0.1118,    Adjusted R-squared:  0.1114 
F-statistic: 268.8 on 4 and 8541 DF,  p-value: < 2.2e-16

summary(pooled)

         term    estimate   std.error statistic       df       p.value
1 (Intercept) 14.66559186 0.487686304 30.071773 1253.210 5.075329e-150
2         Age  0.10367759 0.003748806 27.656164 3916.798 6.073157e-154
3  Gendermale -0.30521870 0.134707691 -2.265785 3727.382  2.352165e-02
4    BPSysAve  0.06754332 0.004761820 14.184349 1160.000  3.121286e-42
5 DiabetesYes  3.23549109 0.260299313 12.429887 8866.664  3.529334e-35

We applied three different approaches to handle missing BMI values in the nhanes_sub dataset, modeling BMI ~ Age + Gender + BPSysAve + Diabetes. Here is what we found:

1. Complete-Case Analysis (CCA)

• What we did: We dropped all observations with missing values (na.omit).

• Result:

  • Coefficients: Age (0.048), BPSysAve (0.068), DiabetesYes (+3.79), Gender slightly negative.

  • Standard errors: Relatively large because ~1500 observations were discarded.

  • R²: 0.114 — fairly low.

• Takeaway: CCA wastes data and may bias estimates if missingness is not MCAR (Missing Completely at Random).

2. Single Imputation (Mean Substitution for BMI)

• What we did: Replaced missing BMI values with the mean BMI.

• Result:

  • Coefficients: Very close to CCA (Age 0.047, BPSysAve 0.067, DiabetesYes +3.73).

  • Gender effect became just significant (p = 0.045).

  • Residual SE decreased slightly (6.57).

• Takeaway: Looks “better” because all observations are retained, but this approach ignores imputation uncertainty and artificially stabilizes estimates. Standard errors are underestimated, leading to overconfidence.

3. Multiple Imputation (MI with mice, m = 5, method = “pmm”)

• What we did: Generated 5 imputed datasets using Predictive Mean Matching (PMM), fit the same model in each, and pooled results.

• Result:

  • Coefficients: Age effect doubled (0.104), intercept dropped (14.7 vs. ~17.5), Diabetes effect slightly smaller (+3.24), Gender effect remained modest but significant (p = 0.023).

  • Standard errors: Properly adjusted upwards — reflecting real uncertainty in imputed BMI values.

  • Inference: Despite differences in point estimates, the conclusions are more statistically honest.

• Takeaway: MI balances efficiency (uses all data) and validity (acknowledges missingness uncertainty).

🔑 Overall Comparison

Interpretation:

• Complete-case drops too much data and risks bias.

• Single imputation keeps the data but gives too much confidence in results.

• Multiple imputation changes some coefficients (notably Age) and reports more realistic uncertaint

👉 Lesson: If your goal is valid inference, especially in epidemiological or social science settings, multiple imputation is the gold standard.

8 Comparison of Common Imputation Methods

No single imputation method is universally optimal—each comes with trade-offs between simplicity, accuracy, and interpretability. For instance, listwise deletion is tempting for its ease but can heavily bias results if missingness is not random. Simple mean or median imputation keeps the dataset intact but artificially reduces variability and masks true correlations. More advanced techniques such as MICE, missForest, and EM provide statistically sound imputations that preserve uncertainty and relationships, but they demand more computational resources and methodological expertise.

In practice:

• Exploratory analysis often starts with simple methods (e.g., median replacement) to get a sense of the data.

• Time series data may rely on LOCF or interpolation.

• Complex survey or clinical datasets typically benefit from advanced approaches like MICE or missForest, which better respect the multivariate nature of the data.

Ultimately, the choice depends on the data structure, missingness mechanism (MCAR, MAR, MNAR), and analytical goals.

9 Conclusion

There is no one-size-fits-all solution for missing data. The right approach depends on your goal (prediction vs. inference), the missingness mechanism (MCAR/MAR/MNAR), your data structure (cross-sectional vs. longitudinal), and practical constraints (time, compute, expertise).

10 References

• Allison, P. D. (2001). Missing Data. Sage Publications.

• Enders, C. K. (2010). Applied Missing Data Analysis. The Guilford Press.

• Little, R. J. A., & Rubin, D. B. (2002). Statistical Analysis with Missing Data (2nd ed.). Wiley.

• van Buuren, S. (2018). Flexible Imputation of Missing Data (2nd ed.). Chapman & Hall/CRC.

• Stekhoven, D. J., & Bühlmann, P. (2012). “MissForest—Nonparametric Missing Value Imputation for Mixed-Type Data.” Bioinformatics, 28(1), 112–118. https://doi.org/10.1093/bioinformatics/btr597

• Rubin, D. B. (1987). Multiple Imputation for Nonresponse in Surveys. Wiley.

• Schafer, J. L. (1997). Analysis of Incomplete Multivariate Data. Chapman & Hall/CRC.

• R Documentation: mice package

• R Documentation: missForest package

1 Introduction: Why This Confusion Still Matters

In the world of data analysis and statistics, standard deviation (SD) and standard error (SE) are two concepts that are often misunderstood or—worse—used interchangeably. This confusion isn’t just academic: misinterpreting these two measures can lead to poor conclusions, misleading visualizations, and incorrect inferences, especially in reports intended for non-technical audiences.

Think about this: you read a news article stating that “the average income of a sample group is $3,000 with a standard error of $500.” But then another article says “the same average income with a standard deviation of $500.” Should your level of confidence change? Absolutely—because they tell two fundamentally different stories.

This article aims to:

• Define and differentiate standard deviation and standard error,
• Explore their mathematical foundations,
• Demonstrate their practical implications with real R code and visuals,
• Warn about common pitfalls and interpretation mistakes.

By the end of this post, you’ll not only understand the difference but also know exactly when and why each metric matters.

2 Definitions and Mathematical Foundation

Understanding the difference between standard deviation and standard error requires going beyond surface-level definitions. While they are mathematically related, they answer fundamentally different questions.

3 Visualizing the Difference with R: Simulation and Interpretation

Let’s use R to visualize and truly understand the difference between standard deviation and standard error.

We’ll start by generating a single random sample from a known population and examining the spread of individual values. Then, we’ll simulate multiple samples to show how the sample means vary—and how that variation reflects the standard error.

set.seed(42)
sample_data <- rnorm(50, mean = 100, sd = 15)

library(ggplot2)

ggplot(data.frame(x = sample_data), aes(x = x)) +
  geom_histogram(aes(y = ..density..), binwidth = 5, fill = "steelblue", color = "white", alpha = 0.6) +
  geom_density(color = "black", linewidth = 1.2, linetype = "solid") +
  geom_vline(aes(xintercept = mean(x)), color = "red", linetype = "dashed", linewidth = 1) +
  labs(
    title = "Standard Deviation: Spread of Individual Values",
    x = "Value", y = "Density"
  )

sample_means <- replicate(1000, mean(rnorm(50, mean = 100, sd = 15)))

ggplot(data.frame(mean = sample_means), aes(x = mean)) +
  geom_histogram(aes(y = ..density..), binwidth = 1, fill = "darkorange", color = "white", alpha = 0.7) +
  geom_density(color = "black", linewidth = 1.2, linetype = "solid") +
  geom_vline(aes(xintercept = mean(mean)), color = "red", linetype = "dashed", linewidth = 1) +
  labs(
    title = "Standard Error: Variability of Sample Means",
    x = "Sample Mean", y = "Density"
  )

se_estimate <- sd(sample_means)
se_estimate

[1] 2.113943

3.3 Interpretation: Two Types of Spread, Two Different Questions

Let’s pause and reflect on what we’ve seen so far.

Although standard deviation and standard error are both measures of “spread,” they describe very different things, answer different questions, and are used in different contexts.

Concept	What it Measures	Based on…	Changes with Sample Size ()
Standard Deviation	Spread of individual data values	Single sample	❌ No
Standard Error	Spread of sample means across repeated samples	Sampling distribution	✅ Yes

---

3.3.1 Summary of Interpretation

• Standard deviation (SD) tells us:
  > “How much do individual values differ from the average within a sample?”

• Standard error (SE) tells us:
  > “How much would the sample average vary if we repeated the sampling?”

In other words:

• SD measures natural variability among individuals (or observations).
• SE measures the statistical uncertainty of an estimate, usually the sample mean.

This difference is not just semantic—it has critical consequences for data interpretation:

• You use SD when describing the spread of your sample or population.
• You use SE when making inferences, estimating confidence intervals, or assessing how trustworthy your sample statistic is.

3.3.2 The Mathematical Connection

As we saw earlier, the standard error is mathematically derived from the standard deviation:

This formula reveals a fundamental principle in statistics:

• The more data you collect (larger ), the more stable your sample mean becomes.
• However, the variability within the sample (standard deviation ) may remain roughly the same—because it depends on the population, not on how many observations you took.

🧠 Key insight:
Standard deviation reflects the reality of your data.
Standard error reflects your uncertainty about the mean.

4 Common Mistakes and Misinterpretations

Despite their differences, standard deviation and standard error are frequently confused—even in academic papers, business reports, and media articles. Below are some of the most common mistakes and why they matter.

5 A Real-World Example: Monthly Spending Survey in USD

Let’s now apply what we’ve learned in a more realistic, international scenario.

Imagine a survey conducted in a mid-sized city where 40 individuals are asked:

“How much money do you spend per month (in US Dollars)?”

We simulate responses centered around $2,000, with a standard deviation of $500.

set.seed(123)
n <- 40
monthly_spending <- round(rnorm(n, mean = 2000, sd = 500), 0)

head(monthly_spending)

[1] 1720 1885 2779 2035 2065 2858

mean_spending <- mean(monthly_spending)
sd_spending <- sd(monthly_spending)
se_spending <- sd_spending / sqrt(n)

mean_spending

[1] 2022.6

sd_spending

[1] 448.8549

se_spending

[1] 70.9702

library(ggplot2)

ggplot(data.frame(spending = monthly_spending), aes(x = spending)) +
  geom_histogram(aes(y = ..density..), binwidth = 250, fill = "skyblue", color = "white", alpha = 0.7) +
  geom_density(color = "darkblue", linewidth = 1.2) +
  geom_vline(aes(xintercept = mean_spending), color = "red", linetype = "dashed", linewidth = 1) +
  labs(
    title = "Distribution of Monthly Spending",
    x = "Monthly Spending (USD)", y = "Density"
  )

lower <- mean_spending - 1.96 * se_spending
upper <- mean_spending + 1.96 * se_spending

c(lower, upper)

[1] 1883.498 2161.702

6 Conclusion

Standard deviation and standard error are often mentioned in the same breath, but they serve very different purposes in data analysis and statistical reasoning.

• Standard deviation reflects the natural variability in a dataset. It tells us how different individuals are from one another.
• Standard error quantifies the precision of a sample estimate, such as the mean. It tells us how much we can trust our estimate of the population parameter.

While they are mathematically related, confusing one for the other can lead to serious misinterpretations—especially in scientific communication, data journalism, or policymaking.

Here are some final takeaways:

• Use standard deviation when describing the data you have.
• Use standard error when making inferences about the population from your sample.
• Always label your charts and error bars clearly, and report sample size to give proper context.
• Don’t mistake low standard error for low variability—it only means your estimate is more precise, not that your data is more uniform.

🎯 In short:
Standard deviation tells you about your data.
Standard error tells you how much you can trust your mean.

Understanding this distinction is more than just a statistical nuance—it’s a sign of analytical maturity.

7 References

• James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An Introduction to Statistical Learning with Applications in R. Springer. https://www.statlearning.com

• Moore, D. S., McCabe, G. P., & Craig, B. A. (2017). Introduction to the Practice of Statistics. W.H. Freeman.

• R Core Team. (2024). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.r-project.org

• Wickham, H., Çetinkaya-Rundel, M., & Grolemund, G. (2023). R for Data Science (2e). https://r4ds.hadley.nz

• Navarro, D. (2019). Learning Statistics with R: A tutorial for psychology students and other beginners. https://learningstatisticswithr.com

1 Introduction

“Correlation is not causation” – it’s a refrain we hear often, yet the distinction between these concepts is deceptively easy to overlook. Correlation refers to a statistical association: when one variable changes, another tends to change as well. Causation, on the other hand, means a change in one variable directly produces a change in another. In other words, there is a cause-and-effect relationship. A crucial insight (sometimes phrased as “causation implies correlation (but not vice versa)”) is that while causation always entails some correlation, observing a correlation by itself does not prove causation. This article will explore the theoretical basis of correlation and causation, illustrate the difference with real-world examples in economics and healthcare, and demonstrate with R code how misleading correlations can arise – and how we can attempt to control for confounding factors. Along the way, we’ll dispel common misconceptions and share expert insights to encourage critical thinking about causal claims in data.

Judea Pearl, a pioneer of modern causal inference, put it succinctly: “Correlation is not causation; merely observing a relationship between variables does not imply a causal connection”. In practical terms, correlation is a symmetric relationship – X and Y vary together – whereas causation is directional: X produces Y. If two things are correlated, there are several possibilities: X causes Y, Y causes X, or some other factor influences both (or it could even be a chance coincidence). As statistician David Freedman warned, “Misinterpreting correlation as causation can lead to erroneous conclusions and misguided actions”. To use data responsibly, we must dig deeper than surface-level associations.

2 Theoretical Background: Correlation in a Nutshell

In statistics, correlation is often measured by the Pearson correlation coefficient (usually notated r). Mathematically, for variables X and Y, this coefficient is defined as:

where Cov(X,Y) is the covariance and σ denotes standard deviations. This value ranges from –1 (perfect negative correlation) to +1 (perfect positive correlation). An r near 0 indicates no linear relationship. Correlation captures how closely two variables move in sync. For example, if higher values of X tend to coincide with higher values of Y (and lower with lower), the correlation is positive. If one tends to go up when the other goes down, the correlation is negative. Crucially, correlation is a descriptive statistic – it quantifies an association, but it does not explain why the variables are related.

Correlation alone is silent on mechanism. It answers “Are X and Y related?” not “Does X change Y?”. To establish causation, we usually rely on theory, controlled experiments, or advanced observational study designs. In the language of causality, we think about interventions: if we do something to X, does Y change as a result? This is fundamentally a different question than observing X and Y moving together. Empirically, evidence of causation typically requires satisfying conditions such as temporal precedence (cause precedes effect), a credible mechanism linking X to Y, consistency with other evidence, and ruling out alternative explanations (confounders).

3 Why Correlation ≠ Causation: Confounders, Coincidences, and Reverse Causality

If correlation doesn’t imply causation, what might be going on when two variables track together? There are a few common scenarios:

• Confounding (Third Variables): A hidden factor influences both variables. This lurking variable makes X and Y move together, creating an illusion that they’re directly linked. Classic example: children’s shoe sizes are strongly correlated with their reading ability. Obviously, bigger shoes don’t make kids read better. The confounder is age: older children have larger feet and also read more proficiently – age drives both. Once age is taken into account, the shoe size–reading correlation disappears. As Pearl humorously noted, “The third variable problem highlights the danger of assuming causation based solely on correlation”. We’ll demonstrate a confounding example with R shortly.

• Pure Coincidence (Spurious Correlations): With enough data, you’re bound to find some weird correlations by chance alone. In fact, whole websites are devoted to absurd correlations. Tyler Vigen’s famous collection of spurious correlations highlights gems like a 0.95 correlation between U.S. per capita cheese consumption and the number of people who died by becoming tangled in their bedsheets. It’s a comical reminder that with countless variables in the world, some will line up in sync purely by accident. High correlation can occur in entirely unrelated data — a cautionary tale for data miners. We should always ask: Is there a plausible reason for this correlation, or could it be random?

• Reverse Causation (Directionality): Sometimes X and Y truly are causally related, but not in the direction one assumes. For example, suppose data show a correlation between depression and low vitamin D levels. Does lack of vitamin D cause depression, or do depressed individuals tend to get less sunlight and thus have lower vitamin D? The data alone can’t tell us the direction. Another example: cities with more police officers tend to have higher crime rates. This doesn’t mean police cause crime; rather, high-crime areas hire more police. In economic contexts, we’ll see debates like “Do higher interest rates reduce inflation, or is it that rising inflation prompts central banks to raise rates?” – in such cases, cause and effect can be easily confused if we only look at correlations.

• Selection Bias and Other Pitfalls: In observational data, how samples are collected can also create misleading correlations. For instance, a medical study might find that patients on a certain medication have higher survival rates – but if those patients were also healthier or younger on average (selection bias), the medication’s effect is confounded. Correlation can even vanish or flip sign when data is disaggregated, a phenomenon known as Simpson’s Paradox. The aggregate data might show one trend, while each subgroup shows the opposite trend. This often indicates a confounding variable at play.

# Simulate data for two groups: 'Young' and 'Old'
set.seed(42)
AgeGroup <- rep(c("Young", "Old"), each = 50)
# For Young group, generate foot_size and reading_score with no true correlation
foot_size <- c(rnorm(50, mean = 20, sd = 2),    # Young have smaller feet on average
               rnorm(50, mean = 25, sd = 2))    # Old have larger feet on average
reading_score <- c(rnorm(50, mean = 50, sd = 5),# Young have lower reading scores on avg
                   rnorm(50, mean = 80, sd = 5))# Old have higher reading scores on avg

# Check correlations
cor(foot_size, reading_score)                        # overall correlation

[1] 0.7597618

cor(foot_size[AgeGroup=="Young"], reading_score[AgeGroup=="Young"])  # within Young group

[1] 0.1043372

cor(foot_size[AgeGroup=="Old"], reading_score[AgeGroup=="Old"])      # within Old group

[1] -0.07429122

# Plot the data, coloring by group, and add regression lines
library(ggplot2)
df <- data.frame(AgeGroup, foot_size, reading_score)
ggplot(df, aes(x = foot_size, y = reading_score, color = AgeGroup)) +
  geom_point(size = 2, alpha = 0.7) +
  geom_smooth(method = "lm", se = FALSE) + 
  geom_smooth(aes(group = 1), method = "lm", se = FALSE, color = "black", linetype = "dashed") +
  labs(x = "Foot size (cm)", y = "Reading score",
       title = "Spurious correlation: Foot size vs Reading score",
       subtitle = "Colored by AgeGroup. Solid lines = separate group fits (no correlation); dashed line = overall fit.")

4 Real-World Example 1: Interest Rates and Inflation

One arena where correlation vs causation debates rage is macroeconomics. Consider interest rates and inflation – two metrics that often move in tandem. Central banks (like the U.S. Federal Reserve or Bank of England) adjust interest rates as a policy tool, aiming to control inflation. Intuition says raising interest rates should cause inflation to decrease (by cooling off spending and investment). Indeed, periods of tight monetary policy often coincide with inflation coming down. But does that correlation mean the rate hikes caused the relief in inflation? Not necessarily. As one economics blogger noted during the 2022–2023 inflation surge: “One might be tempted to draw a direct line between higher interest rates and lower inflation rates. But correlation does not necessarily imply causation.” In that episode, global inflation started easing after its 2022 peak, at the same time central banks were aggressively raising rates. However, careful analysis suggested much of the inflation decline was due to resolving supply chain issues and falling commodity prices – factors largely independent of interest rate moves. In other words, inflation would have started abating on its own as pandemic-era supply shocks faded, even if interest rates had not been hiked so sharply. The overlap in timing was a correlation, not a definitive proof of causation.

Economists have to untangle these relationships with statistical tools and historical data. One approach is to look at lead-lag relationships: if interest rate changes truly cause lower inflation, we’d expect to see inflation consistently drop a few quarters after rate hikes. If instead we observe that inflation spikes often precede rate hikes (as central banks react to rising inflation), that indicates reverse causation – inflation causing interest rate changes. Studies of the UK economy, for instance, found that in the short run, raising interest rates sometimes correlated with higher inflation in subsequent quarters. This counter-intuitive positive correlation could mean that initial rate hikes were implemented when inflation was already rising (so inflation kept climbing shortly after), or that rate hikes had supply-side effects (e.g. raising business costs) that temporarily stoked inflation. Only after a longer time lag did the correlation turn negative as expected (inflation easing modestly) – and even then, the effect was statistically weak in some analyses. An outside observer summed up the mixed evidence wryly: “If correlation means causality then possibly not. [Rate hikes] may have an effect, but the effect might be weak on inflation and brutal on society”. In other words, simply correlating past interest increases with inflation outcomes can be misleading; it takes careful modeling to isolate the causal impact (and it might be smaller than popularly assumed).

This example highlights two key points: First, directionality matters – are we seeing X→Y or Y→X or both? (In economics, feedback loops are common: inflation could prompt rate changes, which in turn influence future inflation, a two-way causality.) Second, confounding variables abound – other factors like global supply conditions, fiscal policy, or consumer expectations can drive inflation, obscuring the effect of interest rates. Analysts tackle these challenges with techniques such as Vector Autoregression (VAR) models, instrumental variables, or by “clustering” data to compare similar periods or countries. A commenter on an economics forum pointed out that failing to control for such factors is akin to falling for Simpson’s Paradox: “Plotting inflation vs interest rates can be misleading unless you cluster to avoid confounding variables”. The lesson: even in highly data-driven fields like economics, correlation alone can support multiple stories, and solid conclusions require digging into the causal structure of the problem.

5 Real-World Example 2: Vaccines and Disease Prevalence

Few areas demonstrate the difference between correlation and causation as starkly as healthcare and epidemiology. Let’s examine vaccines and disease rates. Vaccines are designed based on a known causal mechanism (they induce immunity, which prevents disease), and countless studies and trials have validated their efficacy. Thus, when a vaccine is introduced, we expect disease incidence to drop as a causal result. Conversely, if vaccination rates fall, diseases can surge. Both correlations have been observed in reality – one led to a life-saving public health success, the other to a dangerous resurgence of disease – and they underline why understanding causality is critical.

Correlation used as evidence of causation (correctly): In the 1950s, polio was a dreaded disease paralyzing tens of thousands each year. In 1955, the Salk polio vaccine was introduced. Within just a few years, polio cases plummeted. In the United States, annual polio cases dropped from ~58,000 to about 5,600 by 1957, and only 161 cases by 1961. The timing and magnitude of this drop, alongside laboratory and clinical evidence, provided convincing proof that the vaccine caused the decline in polio. Here the correlation (vaccine rollout followed by disease collapse) was no coincidence – it was a predicted outcome based on a causal understanding of immunity. As another example, when the HPV vaccine was introduced, health officials observed sharp declines in HPV infections and related cancers in subsequent years, consistent with the expected causal effect of vaccinating adolescents. In such cases, correlation was a strong hint that led scientists to conclude causation, bolstered by controlled trials and biological plausibility.

Correlation misinterpreted as causation (incorrectly): Not all observed links are what they seem. A notorious case was the now-debunked claim that the MMR (measles, mumps, rubella) vaccine caused autism in children. This idea stemmed from a 1998 study (later found fraudulent) and the anecdotal observation that autism diagnoses were often made around the same age children receive lots of vaccines. In truth, the apparent correlation was driven by coincidental timing and increased awareness/diagnosis of autism in the 1990s – not by vaccines. Extensive research over decades showed no causal link: “Research over the past 15 years has shown that childhood vaccines don’t cause autism”. Unfortunately, the fear incited by the false correlation led many parents to avoid the MMR vaccine. The result? Measles, once near-eliminated, came roaring back. Great Britain, for instance, experienced a measles epidemic in the 2000s as vaccination rates fell. Public health officials directly attributed this to the drop in vaccinations after the autism scare: “Great Britain is in the midst of a measles epidemic, one that public health officials say is the result of parents refusing to vaccinate their children after a safety scare that was later proved to be fraudulent”. In regions where MMR vaccination rates fell below about 80%, measles cases spiked dramatically. One commentator lamented, “This is the legacy of the Wakefield scare”. The correlation here – lower vaccination accompanied by higher disease incidence – reflected a causal relationship, but in the opposite direction of the original false claim. Vaccines prevent measles, so when vaccination dropped, measles returned. It’s a sobering example of how a misunderstood correlation (vaccines and autism) led to behaviors that revealed a very real causation (lack of vaccines causing disease outbreaks).

In summary, the vaccine story teaches us that we must have external evidence and domain knowledge to distinguish meaningful correlations from spurious ones. When strong theory and additional evidence support a correlation (as with vaccines preventing disease), we can infer causation with confidence. But when a correlation flies in the face of established knowledge or lacks a plausible mechanism (as with vaccines causing autism), it demands deep skepticism and further investigation. Correlation may open the door to a hypothesis, but only rigorous science can confirm causality.

6 From Correlation to Causation: How Can We Tell?

So, if correlation alone isn’t enough, how do scientists and statisticians actually establish causation? This is the realm of causal inference, and entire textbooks (and careers) are devoted to it. Here we’ll outline a few key principles and methods:

• Controlled Experiments: The gold standard for testing causality is the randomized controlled trial (RCT). By randomly assigning subjects to a treatment (X) or control, we ensure no systematic confounders differ between groups. Any difference in outcomes (Y) can then be attributed to X (within known statistical error). As statistician Paul Rosenbaum emphasizes, “Experimental design is crucial for establishing causal relationships and overcoming confounding factors”. In fields like medicine, RCTs are required to claim a drug causes an effect. In more complex domains (economics, social sciences) where RCTs may be infeasible or unethical, researchers look for natural experiments or instrumental variables to approximate that level of control.

• Temporal Checks: Ensure the cause precedes the effect. Sounds obvious, but it’s a simple way to weed out some mistaken causal interpretations. If Y happens before X, X cannot be the cause. Sometimes lagged correlations or time-series analyses (like Granger causality tests in economics) are used to see if changes in X consistently come before changes in Y. In our interest rate example, analysts examined whether inflation tended to drop after interest rate hikes (and found mixed results, indicating caution in the causal claim).

• Controlling for Confounders: In observational studies, a common strategy is to measure possible confounding variables and include them in a regression or stratify the analysis. For instance, if we suspect age is a confounder in our earlier example, we can compare individuals of similar age (or include age in a multiple regression model) to see if foot size still correlates with reading ability within those strata. If the correlation vanishes after controlling for the third variable, it was likely spurious. Techniques like multiple regression, matching, propensity score adjustment, and difference-in-differences analysis are all about simulating a “ceteris paribus” condition – i.e. comparing like with like, so that the effect of interest can be isolated. In R, one might use lm() (linear modeling) to adjust for confounders. For example, lm(reading_score ~ foot_size + Age, data=df) would tell us if foot_size still has any predictive power for reading_score once Age is accounted for. (In our simulated data, it would show foot_size is not significant when Age is included, reinforcing that foot size itself wasn’t causing better reading.)

• Multiple Studies and Triangulation: We gain confidence in causation when multiple independent studies, using different methods, consistently point to the same conclusion. If correlational evidence is supported by lab experiments, longitudinal studies, and perhaps natural experiments, the case for causality strengthens. In the smoking and lung cancer example: early on, skeptics said “correlation is not causation” – maybe smokers had other habits causing cancer. But over time, mountains of evidence (animal experiments, biological mechanisms, epidemiological studies controlling for diet, etc.) converged to establish that smoking does cause cancer.

• Plausibility and Mechanism: A correlation accompanied by a plausible mechanism is more convincing. If we can explain how X could influence Y (through physics, biology, or logic), we are more likely to consider X a potential cause of Y. In contrast, if no one can conceive a realistic way that X would affect Y, we suspect a lurking variable or coincidence. (For instance, it’s hard to imagine how eating more cheese would directly cause strangulation by bedsheets – more likely, as one humorous analysis noted, it’s just an “accidental, misleading pattern” or related to a confounder like time or lifestyle).

• Causal Graphs and Models: Modern data science sometimes employs causal graphs or Bayesian networks (à la Judea Pearl’s do-calculus) to formally model assumptions about causation and test if the observed correlations fit a causal structure. While beyond the scope of this article, these tools provide a framework to encode “X causes Y” assumptions and see what observational patterns should emerge if that’s true. They also help identify what additional data or experiments are needed to distinguish between competing causal hypotheses.

In practice, determining causation is often like solving a puzzle. We marshal all available evidence, use critical thinking, and sometimes still end up with uncertainty. However, the effort is worthwhile because acting on false causal assumptions can be costly. Misattributing causation can lead to bad policy, ineffective or harmful interventions, or simply wasting resources chasing the wrong problem. As we’ve seen, data should be approached with a skeptical eye. Correlations can be tantalizing – they can indeed be hints to causal relationships – but we must verify those hints. By combining statistical rigor with domain expertise, we improve our chances of getting the causation right.

7 Conclusion

Understanding the difference between correlation and causation is essential for anyone who consumes data-driven information (which these days is all of us). We’ve covered how correlation is a mathematical relationship that can flag interesting connections but can also mislead us through confounding, coincidence, or reversed cause-and-effect. We explored examples from economics and healthcare where these distinctions have real-world consequences – from guiding central bank policies to informing public health decisions. The key takeaway is to think critically: when you hear that X is linked to Y, ask why and how. Look for evidence that goes beyond the raw correlation. As the saying goes (often attributed to many scientists), “Correlation is not causation, but it sure is a hint.” Use the hint to investigate further, not to jump to conclusions.

In the words of statistician David Freedman, misinterpreting correlation as causation is not just an academic error but one that can lead to “misguided actions”. By staying curious and skeptical – and by leveraging tools like R to analyze data properly – we can uncover the true stories our data are telling us. Correlation can open the door to discovery, but only rigorous analysis and critical thinking will reveal what’s inside.

8 References & Further Reading

• Bandholz, H. “Correlation does not imply causation: higher interest and lower inflation rates” – Macro to Go blog (2023)

• Eddy-OJB. “Do Interest Rates Tackle Inflation?” – Medium (2023)

• Statsig Team. Examples of misleading correlations (2024)

• Matthews, R. “Storks Deliver Babies (p = 0.008)” – Teaching Statistics via Priceonomics

• Scribbr – Correlation vs Causation (2021)

• NPR – Measles epidemic after vaccine scare (2013)

• WHO – History of polio vaccination (2021)

1 Introduction

You trust R². Should you?
You proudly present a model with R² = 0.95. Everyone applauds.
But what if your model fails miserably on the next new data?

When building a statistical model, one of the first numbers analysts and data scientists often cite is the R², or coefficient of determination. It’s widely reported in research, academic theses, and industry reports — and yet, frequently misunderstood or misused.

Does a high R² mean your model is good? Is it enough to evaluate model performance? What about its adjusted or predictive counterparts?

This article will explore in depth: - What R², Adjusted R², and Predicted R² actually mean - Why relying solely on R² can mislead you - How to evaluate models using both explanatory and predictive power - Real-life implementation using the {tidymodels} framework in R

We’ll also discuss best practices and common pitfalls, and equip you with a mindset to look beyond surface-level model summaries.

2 Theoretical Background

3 Dataset Overview

We’ll use the classic Boston Housing Dataset to demonstrate. It includes:

• Socio-economic and housing variables for 506 Boston suburbs

• Target: medv (median value of owner-occupied homes in $1000s)

Below are the key variables:

• crim: per capita crime rate by town
• zn: proportion of residential land zoned for large lots
• indus: proportion of non-retail business acres
• chas: Charles River dummy variable (1 = tract bounds river; 0 = otherwise)
• nox: nitric oxides concentration (parts per 10 million)
• rm: average number of rooms per dwelling
• age: proportion of owner-occupied units built before 1940
• dis: weighted distance to employment centers
• rad: index of accessibility to radial highways
• tax: property-tax rate per $10,000
• ptratio: pupil-teacher ratio by town
• black: 1000(Bk - 0.63)^2 where Bk is the proportion of Black residents
• lstat: percentage of lower status of the population
• medv: target — median value of owner-occupied homes (in $1000s)

This regression problem mimics common real estate or socio-economic modeling use cases. Let’s first examine the dataset’s summary statistics.

library(tidymodels)
library(MASS)
library(ggplot2)
library(corrr)
library(skimr)
library(patchwork)


boston <- MASS::Boston
skim(boston)

Commentary:

• Variables like crim, tax, and lstat exhibit high variability and potential skewness.

• chas is binary and acts like a categorical indicator.

• The target variable medv ranges from $5,000 to $50,000 (capped).

• rm (average number of rooms) and lstat (lower status population) show notable spread and will likely play strong roles in the model.

Next, we examine correlations with medv:

boston %>% correlate() %>% corrr::focus(medv) %>% arrange(desc(medv))

# A tibble: 13 × 2
   term      medv
   <chr>    <dbl>
 1 rm       0.695
 2 zn       0.360
 3 black    0.333
 4 dis      0.250
 5 chas     0.175
 6 age     -0.377
 7 rad     -0.382
 8 crim    -0.388
 9 nox     -0.427
10 tax     -0.469
11 indus   -0.484
12 ptratio -0.508
13 lstat   -0.738

Interpretation of Correlations:

• rm shows a strong positive correlation with medv — more rooms generally imply higher value.

• lstat and crim have strong negative correlations — as lower status or crime increases, housing values drop.

• nox, age, and ptratio also show negative correlations with price, hinting at socio-environmental effects.

These insights will guide us in building and evaluating our model.

4 Exploratory Data Analysis

Let’s visualize some of the most influential variables in relation to medv, our target variable. These exploratory graphs help reveal potential linear or nonlinear relationships, outliers, or the need for transformation.

# Define individual plots with improved formatting for Quarto rendering
p1 <- ggplot(boston, aes(rm, medv)) +
  geom_point(alpha = 0.5, color = "#2c7fb8") +
  geom_smooth(method = "lm", se = FALSE, color = "black") +
  labs(
    title = "Rooms\nvs. Median Value",
    x = "Average Number of Rooms (rm)",
    y = "Median Value of Homes ($1000s)"
  ) +
  theme_minimal() +
  theme(plot.title = element_text(size = 11, lineheight = 1.1))

p2 <- ggplot(boston, aes(lstat, medv)) +
  geom_point(alpha = 0.5, color = "#de2d26") +
  geom_smooth(method = "loess", se = FALSE, color = "black") +
  labs(
    title = "Lower Status %\nvs. Median Value",
    x = "% Lower Status Population (lstat)",
    y = "Median Value of Homes ($1000s)"
  ) +
  theme_minimal() +
  theme(plot.title = element_text(size = 11, lineheight = 1.1))

p3 <- ggplot(boston, aes(nox, medv)) +
  geom_point(alpha = 0.5, color = "#31a354") +
  geom_smooth(method = "loess", se = FALSE, color = "black") +
  labs(
    title = "NOx Concentration\nvs. Median Value",
    x = "NOx concentration (ppm)",
    y = "Median Value of Homes ($1000s)"
  ) +
  theme_minimal() +
  theme(plot.title = element_text(size = 11, lineheight = 1.1))

p4 <- ggplot(boston, aes(age, medv)) +
  geom_point(alpha = 0.5, color = "#ff7f00") +
  geom_smooth(method = "loess", se = FALSE, color = "black") +
  labs(
    title = "Old Homes %\nvs. Median Value",
    x = "% Homes Built Before 1940 (age)",
    y = "Median Value of Homes ($1000s)"
  ) +
  theme_minimal() +
  theme(plot.title = element_text(size = 11, lineheight = 1.1))

p5 <- ggplot(boston, aes(tax, medv)) +
  geom_point(alpha = 0.5, color = "#6a3d9a") +
  geom_smooth(method = "loess", se = FALSE, color = "black") +
  labs(
    title = "Tax Rate\nvs. Median Value",
    x = "Tax Rate (per $10,000)",
    y = "Median Value of Homes ($1000s)"
  ) +
  theme_minimal() +
  theme(plot.title = element_text(size = 11, lineheight = 1.1))

p6 <- ggplot(boston, aes(dis, medv)) +
  geom_point(alpha = 0.5, color = "#1f78b4") +
  geom_smooth(method = "loess", se = FALSE, color = "black") +
  labs(
    title = "Distance to Jobs\nvs. Median Value",
    x = "Weighted Distance to Employment Centers (dis)",
    y = "Median Value of Homes ($1000s)"
  ) +
  theme_minimal() +
  theme(plot.title = element_text(size = 11, lineheight = 1.1))