If you haven’t seen our previous post about the exciting new features in dplyr 1.2.0, you’ll want to go check that out first!

Here’s a before-and-after benchmark with if_else() :

# Using https://github.com/DavisVaughan/cross
cross::bench_versions(pkgs = c("tidyverse/dplyr@v1.1.4", "tidyverse/dplyr"), {
  library(dplyr)
  set.seed(123)

  condition <- sample(c(TRUE, FALSE, NA), size = 1e7, replace = TRUE)
  x <- sample(10, size = 1e7, replace = TRUE)
  y <- sample(10, size = 1e7, replace = TRUE)
  z <- sample(10, size = 1e7, replace = TRUE)

  bench::mark(if_else = if_else(condition, x, y, missing = z))
})

1
2
3
4
5

#> # A tibble: 2 × 6
#>   pkg                    expression      min   median `itr/sec` mem_alloc
#>   <chr>                  <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>
#> 1 tidyverse/dplyr@v1.1.4 if_else    248.25ms 249.25ms      4.02   381.6MB
#> 2 tidyverse/dplyr        if_else      7.27ms   7.51ms    132.      38.2MB

And with case_when() :

cross::bench_versions(pkgs = c("tidyverse/dplyr@v1.1.4", "tidyverse/dplyr"), {
  library(dplyr)
  set.seed(123)

  column <- sample(100, size = 1e7, replace = TRUE)

  x_condition <- column < 20
  y_condition <- column < 50
  z_condition <- column < 80

  x <- sample(10, size = 1e7, replace = TRUE)
  y <- sample(10, size = 1e7, replace = TRUE)
  z <- sample(10, size = 1e7, replace = TRUE)

  bench::mark(
    case_when = case_when(
      x_condition ~ x,
      y_condition ~ y,
      z_condition ~ z
    )
  )
})

1
2
3
4
5

#> # A tibble: 2 × 6
#>   pkg                    expression      min   median `itr/sec` mem_alloc
#>   <chr>                  <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>
#> 1 tidyverse/dplyr@v1.1.4 case_when   228.3ms  231.2ms      4.33   419.9MB
#> 2 tidyverse/dplyr        case_when    15.5ms   15.8ms     62.8     38.3MB

So a 33x speed improvement for if_else() , a 15x speed improvement for case_when() , and a 10x improvement in memory usage for both! In the rest of this post, we’ll explain how we’ve achieved these numbers.

library(dplyr)

Let’s talk memory

We’ll start with case_when() , because if_else() is actually just a small variant of that.

The most important place to start is with the memory usage. Memory usage and raw speed are often related, as allocating memory takes time. Let’s look at the memory usage of case_when() in dplyr 1.1.4:

set.seed(123)

column <- sample(100, size = 1e7, replace = TRUE)

x_condition <- column < 20
y_condition <- column < 50
z_condition <- column < 80

x <- sample(10, size = 1e7, replace = TRUE)
y <- sample(10, size = 1e7, replace = TRUE)
z <- sample(10, size = 1e7, replace = TRUE)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

profmem::profmem(
  threshold = 1000,
  case_when(
    x_condition ~ x,
    y_condition ~ y,
    z_condition ~ z
  )
)
#> Rprofmem memory profiling of:
#> case_when(x_condition ~ x, y_condition ~ y, z_condition ~ z)
#>
#> Memory allocations (>= 1000 bytes):
#>        what     bytes                                           calls
#> 1     alloc  40000048     case_when() -> vec_case_when() -> vec_rep()
#> 2     alloc  40000048                  case_when() -> vec_case_when()
#> 3     alloc  40000056       case_when() -> vec_case_when() -> which()
#> 4     alloc   7600664       case_when() -> vec_case_when() -> which()
#> 5     alloc  40000048                  case_when() -> vec_case_when()
#> 6     alloc  40000056       case_when() -> vec_case_when() -> which()
#> 7     alloc  12003312       case_when() -> vec_case_when() -> which()
#> 8     alloc  40000048                  case_when() -> vec_case_when()
#> 9     alloc  40000056       case_when() -> vec_case_when() -> which()
#> 10    alloc  11996112       case_when() -> vec_case_when() -> which()
#> 11    alloc  40000056       case_when() -> vec_case_when() -> which()
#> 12    alloc   8400112       case_when() -> vec_case_when() -> which()
#> 13    alloc   7600664   case_when() -> vec_case_when() -> vec_slice()
#> 14    alloc  12003312   case_when() -> vec_case_when() -> vec_slice()
#> 15    alloc  11996112   case_when() -> vec_case_when() -> vec_slice()
#> 16    alloc   8400112 case_when() -> vec_case_when() -> vec_recycle()
#> 17    alloc  40000048 case_when() -> vec_case_when() -> list_unchop()
#> total       440000864

That’s a lot of allocations! And it’s pretty hard to understand where they are coming from without a bit more explanation. For that, we’re actually going to “manually” implement an underpowered version of case_when() for this example.

Here’s a diagram of what we need to accomplish:

In bullets:

• x_condition selects the blue elements of x
• y_condition selects the red elements of y
• z_condition selects the green elements of z
• A default is built around the unused locations
• We combine all of the pieces into out

The trickiest part about case_when() is handling places where x_condition and y_condition overlap. In the image, even though both x and y are selected at location 5, only the value of x is retained since it is hit “first”. This forces us to have to modify y_condition to avoid already “used” locations.

An R implementation that computes these modified locations might look like:

n <- length(x_condition)

unused <- rep(TRUE, times = n) # 1

x_loc <- unused & x_condition # 2
x_loc <- which(x_loc) # 3,4
unused[x_loc] <- FALSE

y_loc <- unused & y_condition # 5
y_loc <- which(y_loc) # 6,7
unused[y_loc] <- FALSE

z_loc <- unused & z_condition # 8
z_loc <- which(z_loc) # 9,10
unused[z_loc] <- FALSE

Anything that is still unused falls through to the default:

default <- NA_integer_
default_loc <- which(unused) # 11,12

With x_loc, y_loc, z_loc, and default_loc in hand, we can build the output from the pieces:

out <- vector("integer", length = n) # 17

out[x_loc] <- x[x_loc] # 13
out[y_loc] <- y[y_loc] # 14
out[z_loc] <- z[z_loc] # 15

out[default_loc] <- rep(default, times = length(default_loc)) # 16

And sure enough, this is identical to case_when() :

identical(
  out,
  case_when(
    x_condition ~ x,
    y_condition ~ y,
    z_condition ~ z
  )
)
#> [1] TRUE

You might be wondering what all of the comments with numbers beside them mean. Those actually map 1:1 with the allocations that case_when() was emitting. In fact, we can now split up those allocations into their respective role:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

#> # Tracking `unused` locations
#> 1     alloc  40000048     case_when() -> vec_case_when() -> vec_rep()

#> # Computing `x_loc`, `y_loc`, and `z_loc`
#> 2     alloc  40000048                  case_when() -> vec_case_when()
#> 3     alloc  40000056       case_when() -> vec_case_when() -> which()
#> 4     alloc   7600664       case_when() -> vec_case_when() -> which()
#> 5     alloc  40000048                  case_when() -> vec_case_when()
#> 6     alloc  40000056       case_when() -> vec_case_when() -> which()
#> 7     alloc  12003312       case_when() -> vec_case_when() -> which()
#> 8     alloc  40000048                  case_when() -> vec_case_when()
#> 9     alloc  40000056       case_when() -> vec_case_when() -> which()
#> 10    alloc  11996112       case_when() -> vec_case_when() -> which()

#> # Computing `default_loc`
#> 11    alloc  40000056       case_when() -> vec_case_when() -> which()
#> 12    alloc   8400112       case_when() -> vec_case_when() -> which()

#> # Slicing `x`, `y`, and `z` to align with `x_loc`, `y_loc`, and `z_loc`
#> 13    alloc   7600664   case_when() -> vec_case_when() -> vec_slice()
#> 14    alloc  12003312   case_when() -> vec_case_when() -> vec_slice()
#> 15    alloc  11996112   case_when() -> vec_case_when() -> vec_slice()

#> # Recycling `default` of `NA` to align with `default_loc`
#> 16    alloc   8400112 case_when() -> vec_case_when() -> vec_recycle()

#> # Final output container, which we assign `x`, `y`, `z`, and `default` into
#> # at locations `x_loc`, `y_loc`, and `z_loc`
#> 17    alloc  40000048 case_when() -> vec_case_when() -> list_unchop()

We sought to remove every one of these allocations except for the last one, which is the final output container that is returned to the user. In other words, we were after this, which is the actual profmem result of this case_when() call in dplyr 1.2.0:

1
2
3
4
5
6
7

#> Rprofmem memory profiling of:
#> case_when(x_condition ~ x, y_condition ~ y, z_condition ~ z)
#>
#> Memory allocations (>= 1000 bytes):
#>        what    bytes                          calls
#> 1     alloc 40000048 case_when() -> vec_case_when()
#> total       40000048

Sliced assignment

To work towards this, let’s focus on what happens to x throughout this process:

We had a hypothesis that we could cut out the intermediate work here. Ideally, we’d take the logical LHS x_condition and the RHS x and map that straight into the output, with no extra allocations:

But this just wasn’t possible with the way that assignment typically works in R!

x <- c("a", "b", "c", "d", "e", "f", "g")
x_condition <- c(FALSE, TRUE, FALSE, FALSE, TRUE, FALSE, FALSE)
out <- vector("character", length = length(x_condition))

out[x_condition] <- x
#> Warning in out[x_condition] <- x: number of items to replace is not a multiple of replacement length

Instead, you must pre-slice x to a length that matches the locations that x_condition points to in out, i.e.:

out[x_condition] <- x[x_condition]

Now, in case_when() we don’t actually use [<- for assignment or [ for slicing. Instead, we use tools from vctrs , a low level package for building consistent tidyverse functions. In this case, we’d use vctrs::vec_assign() and vctrs::vec_slice():

out <- vctrs::vec_assign(out, x_condition, vctrs::vec_slice(x, x_condition))

But vec_assign() had the same problem!

To solve this, we’ve added a new boolean argument to vec_assign() called slice_value. You use it like this:

out <- vctrs::vec_assign(out, x_condition, x, slice_value = TRUE)

With slice_value = TRUE, vec_assign() assumes that both out and x are the same length and that x_condition applies to both of these. Internally, rather than materializing x[x_condition], we instead just loop over both out and x at the same time (at C level) and copy over values from x whenever x_condition is TRUE.

This is huge! It means that allocations 13-15 from above related to slicing x, y, and z all disappear.

Logical indices

You might have noticed that we’ve been using which() quite a bit in the above algorithm. This turns a logical vector of TRUE and FALSE into an integer vector of locations pointing to where the logical vector was TRUE:

x_condition <- c(TRUE, FALSE, TRUE, FALSE, FALSE, TRUE)
x_loc <- which(x_condition)
x_loc
#> [1] 1 3 6

We perform this conversion up front due to how the following works at C level:

out[x_condition] <- x[x_condition]

Both [ and [<- will convert a logical x_condition into the integer x_loc form before proceeding with the assignment, meaning that which() gets called twice if we don’t do it once up front. And vctrs is the same way! Both vec_assign() and vec_slice() here would convert x_condition to an integer vector.

vctrs::vec_assign(out, x_condition, vctrs::vec_slice(x, x_condition))

Now, with the previous optimization we’ve already seen that we can reduce this to:

vctrs::vec_assign(out, x_condition, x, slice_value = TRUE)

But vec_assign() still converts a logical x_condition to integer locations internally before doing the assignment. So now it doesn’t matter whether we do this conversion up front via which() or if we let vec_assign() do it, it still happens once per input. But we’d like to avoid it entirely!

The solution here wasn’t too magical, it just involved a good bit of grunt work. We’ve added a path in vec_assign()’s C code that can handle logical indices like x_condition directly, rather than forcing them to be converted to integer locations first.

But this is a huge win, because it means that allocations 1-10, which were all related to which() , can now be removed. vec_assign() will just handle that optimally for us without any extra allocations.

The nice part about an optimization like this is that any other existing code that is using vec_assign() with a logical index will also benefit from this without having to change a thing!

default handling

The remaining allocations are 11-12 and 16, which all have to do with the implied default. Allocations 11-12 were about figuring out where to put default, and allocation 16 was about recycling a typed size 1 default to the right size before assigning it into out.

As it turns out, we don’t need any of this!

In vctrs, when we initialize any output container, we use vec_init():

vctrs::vec_init(integer(), n = 5)
#> [1] NA NA NA NA NA

vctrs::vec_init(tibble(x = integer(), y = character()), n = 5)
#> # A tibble: 5 × 2
#>       x y    
#>   <int> <chr>
#> 1    NA NA   
#> 2    NA NA   
#> 3    NA NA   
#> 4    NA NA   
#> 5    NA NA

This already has the implied default assigned to every location. We then overwrite this with x, y, and z at the appropriate locations, but anything left untouched by those is still set to the default, so we’re done!

For cases where the user supplies their own default, things are slightly more complicated. We actually do have to compute a default_loc implied from x_condition, y_condition, and z_condition, but internally we do so using a C vector of bool (even more efficient than R’s logical vector type), so the memory footprint is as small as it can be.

The “first wins” conundrum

One thing we’ve skipped over is the “first wins” behavior of case_when() mentioned earlier. Now that we’ve removed x_loc, y_loc, and z_loc, which is where that was being handled, how do we keep this behavior without slowing things down?

To be explicit, we are talking about this feature of case_when() where only the first hit is kept when you have overlapping logical indices:

x <- c("x1", "x2", "x3")
y <- c("y1", "y2", "y3")
z <- c("z1", "z2", "z3")

x_condition <- c(TRUE, FALSE, TRUE)
y_condition <- c(TRUE, TRUE, FALSE)
z_condition <- c(FALSE, TRUE, TRUE)

case_when(
  x_condition ~ x,
  y_condition ~ y,
  z_condition ~ z
)
#> [1] "x1" "y2" "x3"

A naive approach doesn’t work, as you end up with “last wins” behavior:

out <- vctrs::vec_init(character(), n = 3)

out <- vctrs::vec_assign(out, x_condition, x, slice_value = TRUE)
out <- vctrs::vec_assign(out, y_condition, y, slice_value = TRUE)
out <- vctrs::vec_assign(out, z_condition, z, slice_value = TRUE)

# This is wrong!
out
#> [1] "y1" "z2" "z3"

identical(
  out,
  case_when(
    x_condition ~ x,
    y_condition ~ y,
    z_condition ~ z
  )
)
#> [1] FALSE

Instead, case_when() just iterates in reverse, assigning z, then y, then x:

out <- vctrs::vec_init(character(), n = 3)

out <- vctrs::vec_assign(out, z_condition, z, slice_value = TRUE)
out <- vctrs::vec_assign(out, y_condition, y, slice_value = TRUE)
out <- vctrs::vec_assign(out, x_condition, x, slice_value = TRUE)

identical(
  out,
  case_when(
    x_condition ~ x,
    y_condition ~ y,
    z_condition ~ z
  )
)
#> [1] TRUE

This diagram demonstrates how that works:

Optimizing speed?

Now that we’ve optimized the memory usage of case_when() , you might be wondering if we did anything else to specifically optimize its speed. Not really! We have moved everything from R to C, but focusing our efforts on reducing memory also resulted in some pretty performant code, and there wasn’t much left to optimize after that.

if_else()

if_else() can actually be written as a form of case_when() :

if_else(condition, true, false, missing)

case_when(
  condition ~ true,
  !condition ~ false,
  is.na(condition) ~ missing
)

In our actual C implementation of if_else() , for simple types like integer, character, or numeric vectors we have an extremely fast path that’s even more optimized than this, but for anything with a class we pretty much use this exact case_when() approach.

For package developers

If you’re a package developer, you’ll be happy to know that vctrs itself now exposes low dependency versions of if_else() and case_when() , here’s the full family:

• vec_if_else()
• vec_case_when()
• vec_replace_when()
• vec_recode_values()
• vec_replace_values()

dplyr::if_else() and friends are now just very thin wrappers over these. Feel free to use the vctrs versions in your package if you need the consistency of the tidyverse without the heavy-ish dependency of dplyr.

At the deepest level, list_combine()

At the deepest level of all of this is one final new vctrs function, list_combine(). This is a flexible way to combine multiple vectors together at locations specified by indices.

list_combine() powers all of vec_case_when(), vec_replace_when(), vec_recode_values(), vec_replace_values(), vec_if_else(), and even vec_c(), the tidyverse version of c() .

set.seed(123)

column <- sample(100, size = 1e7, replace = TRUE)

x_condition <- column < 20
y_condition <- column < 50
z_condition <- column < 80

x <- sample(10, size = 1e7, replace = TRUE)
y <- sample(10, size = 1e7, replace = TRUE)
z <- sample(10, size = 1e7, replace = TRUE)

out <- vctrs::list_combine(
  x = list(x, y, z),

  # `indices` are allowed to be logical and aren't forced to integer
  indices = list(x_condition, y_condition, z_condition),
  size = length(x_condition),

  # When there are overlaps, take the "first"
  multiple = "first",

  # Same as `slice_value` from `vec_assign()`
  slice_x = TRUE
)

identical(
  out,
  case_when(
    x_condition ~ x,
    y_condition ~ y,
    z_condition ~ z
  )
)
#> [1] TRUE

• filter_out() , the missing complement to filter() , and accompanying when_any() and when_all() helpers.

• recode_values() , replace_values() , and replace_when() , three new functions that join case_when() to create a cohesive family of powerful tools for recoding and replacing.

Both of these sets of features are backed by successful tidyups, the tidyverse’s community facing proposal process (filtering , recoding ). We really enjoyed having the community weigh in on these features!

You can install dplyr 1.2.0 from CRAN with:

install.packages("dplyr")

You can see a full list of changes in the release notes .

library(dplyr)

Expanding the filter() family

filter() has been a core dplyr verb since the very beginning, but over the years we’ve isolated a few key issues with it:

• The name filter() is ambiguous, are you keeping or dropping rows? i.e., are you filtering for rows or filtering out rows?

• filter() is optimized for the case of keeping rows, but you are just as likely to try and use it for dropping rows. Using filter() to drop rows quickly forces you to confront complex boolean logic and explicitly handle missing values, which is difficult to teach, error prone to write, and hard to understand when you come back to it in the future.

• filter() combines comma separated conditions with & because this covers the majority of the cases. But if you’d like to combine conditions with |, then you have to introduce parentheses around your conditions and combine them into one large condition separated by |, reducing readability.

In the next few sections, we’ll motivate these issues and discuss how some new features in dplyr can simplify things dramatically!

Filtering…out!

Take a look at this patients data. Our task with this data is:

Filter out rows where the patient is deceased and the year was before 2012.

patients <- tibble(
  name = c("Anne", "Mark", "Sarah", "Davis", "Max", "Derek", "Tina"),
  deceased = c(FALSE, TRUE, NA, TRUE, NA, FALSE, TRUE),
  date = c(2005, 2010, NA, 2020, 2010, NA, NA)
)

patients
#> # A tibble: 7 × 3
#>   name  deceased  date
#>   <chr> <lgl>    <dbl>
#> 1 Anne  FALSE     2005
#> 2 Mark  TRUE      2010
#> 3 Sarah NA          NA
#> 4 Davis TRUE      2020
#> 5 Max   NA        2010
#> 6 Derek FALSE       NA
#> 7 Tina  TRUE        NA

With filter() , you’d probably start by translating “patient is deceased and the year was before 2012” into deceased & date < 2012, and then inverting that with !(<expression>) to drop rows:

patients |>
  filter(!(deceased & date < 2012))
#> # A tibble: 3 × 3
#>   name  deceased  date
#>   <chr> <lgl>    <dbl>
#> 1 Anne  FALSE     2005
#> 2 Davis TRUE      2020
#> 3 Derek FALSE       NA

That seems to have worked, let’s use an anti_join() to check with rows have been dropped from patients:

# These rows were dropped
anti_join(
  patients,
  patients |> filter(!(deceased & date < 2012)),
  join_by(name, deceased, date)
)
#> # A tibble: 4 × 3
#>   name  deceased  date
#>   <chr> <lgl>    <dbl>
#> 1 Mark  TRUE      2010
#> 2 Sarah NA          NA
#> 3 Max   NA        2010
#> 4 Tina  TRUE        NA

This is subtly wrong! We only wanted to drop rows where we know that the patient was deceased before 2012. If a missing value is present, we don’t want to drop that row because we aren’t sure about the condition. In this case, we were hoping to only drop Mark! It seems like filter() has unexpectedly dropped more rows than we expected.

Here’s what a technically correct filter() call might look like:

patients |>
  filter(
    !((deceased & !is.na(deceased)) &
      (date < 2012 & !is.na(date)))
  )
#> # A tibble: 6 × 3
#>   name  deceased  date
#>   <chr> <lgl>    <dbl>
#> 1 Anne  FALSE     2005
#> 2 Sarah NA          NA
#> 3 Davis TRUE      2020
#> 4 Max   NA        2010
#> 5 Derek FALSE       NA
#> 6 Tina  TRUE        NA

That’s horrible! You’ll likely look back on this in a year wondering what you were even trying to do here.

This phenomenon is rather confusing, but is due to the fact that filter() is designed around the idea that you’re going to tell it which rows to keep. With that design in mind, dropping NAs makes sense, i.e. if you don’t know that you want to keep that row (because an NA is ambiguous), then you probably don’t want to keep it.

This works well until you try to use filter() as a way to filter out rows, at which point this behavior works against you. At this point, most people (reasonably!) reach for & !is.na() and you end up with the mess from above.

We took a close look at many examples like this one, and eventually realized that the core issue is:

• filter() is designed around supplying which rows to keep
• We are missing a verb designed around supplying which rows to drop

filter_out() fills that gap:

patients |>
  filter_out(deceased, date < 2012)
#> # A tibble: 6 × 3
#>   name  deceased  date
#>   <chr> <lgl>    <dbl>
#> 1 Anne  FALSE     2005
#> 2 Sarah NA          NA
#> 3 Davis TRUE      2020
#> 4 Max   NA        2010
#> 5 Derek FALSE       NA
#> 6 Tina  TRUE        NA

Just like with filter() , filter_out() treats NA values as FALSE. The difference is that filter_out() expects that you are going to tell it which rows to drop (rather than which rows to keep), so the default behavior of treating NA like FALSE works with you rather than against you. It’s also much easier to understand when you look back on it a year from now!

In general, our advice is that if you find yourself using “negative” operators like != or ! or reaching for the !is.na() pattern to manually handle missing values, try reaching for filter_out() instead.

Personally, I’ve always been pretty jealous of Stata here because they had both keep if and drop if , allowing them to write drop if deceased & date < 2012. In my first job, I translated a bunch of Stata code over to R and still remember being frustrated by NA handling every time I had to translate a drop if to a filter() . With filter_out() , it feels like I can finally let go of a long term grudge I’ve held over the past 6 years 🙂.

Combining with OR rather than AND

So far, we’ve talked a lot about dropping rows, but dplyr 1.2.0 also has a new feature to help with keeping rows using conditions combined with | - when_any() .

Our goal here is:

Filter for rows where “US” and “CA” have a score between 200-300, or rows where “PR” and “RU” have a score between 100-200.

countries <- tibble(
  name = c("US", "CA", "PR", "RU", "US", NA, "CA", "PR"),
  score = c(200, 100, 150, NA, 50, 100, 300, 250)
)

countries
#> # A tibble: 8 × 2
#>   name  score
#>   <chr> <dbl>
#> 1 US      200
#> 2 CA      100
#> 3 PR      150
#> 4 RU       NA
#> 5 US       50
#> 6 NA      100
#> 7 CA      300
#> 8 PR      250

Here’s a filter() solution, note how we lose the ability to specify comma separated conditions, and in the process we’ve introduced 3 operators, &, |, and (), decreasing readability and increasing the mental gymnastics required to understand it:

countries |>
  filter(
    (name %in% c("US", "CA") & between(score, 200, 300)) |
      (name %in% c("PR", "RU") & between(score, 100, 200))
  )
#> # A tibble: 3 × 2
#>   name  score
#>   <chr> <dbl>
#> 1 US      200
#> 2 PR      150
#> 3 CA      300

With when_any() , you specify comma separated conditions like you’re used to, but they get combined with | rather than &. This allows us to reduce the amount of operators introduced down to just &, and it remains very readable:

countries |>
  filter(when_any(
    name %in% c("US", "CA") & between(score, 200, 300),
    name %in% c("PR", "RU") & between(score, 100, 200)
  ))
#> # A tibble: 3 × 2
#>   name  score
#>   <chr> <dbl>
#> 1 US      200
#> 2 PR      150
#> 3 CA      300

when_any() and its counterpart when_all() aren’t restricted to filter() . They are normal vector functions that can be used anywhere. And if you’re a package author, you might be interested in vctrs::vec_pany() and vctrs::vec_pall() , the underlying low dependency functions that power the dplyr variants.

Reaching recoding nirvana

Over the years, we’ve experimented with various ways of recoding columns and replacing values within them, including:

• plyr::mapvalues()
• plyr::revalue()
• dplyr::recode()
• dplyr::case_when()

Despite all of our improvements, it’s felt like there have always been holes in our solutions. Most recently, this came to the forefront in a post about recoding using a lookup table , which is almost impossible to do with case_when() , and had people resorting to confusing solutions using the superseded dplyr::recode() combined with !!! to splice in a lookup table.

After seeing this, we took a step back and were finally able to isolate the issues with our current solutions. The result of our analysis is three new functions that join case_when() to form a powerful recoding and replacing family.

It’ll be helpful to define exactly what we mean by recoding vs replacing:

• Recoding a column creates an entirely new column using values from an existing column. The new column may have a different type from the original column.

• Replacing values within a column partially updates an existing column with new values. The result has the same type as the original column.

The family of functions can be summarized by the following table:

We’ve written a new vignette that expands on all of these from first principles, and in the next few sections we’ll look at some examples.

recode_values()

The goal of the post from above was to recode a numeric column of Likert scale scores into their string counterparts.

likert <- tibble(
  score = c(1, 2, 3, 4, 5, 2, 3, 1, 4)
)

We could certainly try case_when() :

likert |>
  mutate(
    category = case_when(
      score == 1 ~ "Strongly disagree",
      score == 2 ~ "Disagree",
      score == 3 ~ "Neutral",
      score == 4 ~ "Agree",
      score == 5 ~ "Strongly agree"
    )
  )
#> # A tibble: 9 × 2
#>   score category         
#>   <dbl> <chr>            
#> 1     1 Strongly disagree
#> 2     2 Disagree         
#> 3     3 Neutral          
#> 4     4 Agree            
#> 5     5 Strongly agree   
#> 6     2 Disagree         
#> 7     3 Neutral          
#> 8     1 Strongly disagree
#> 9     4 Agree

But score == is repeated so many times! When you find yourself using == in this way, recognize that what you’re really doing is matching on the values of a single column. In cases like these, you’ll want to switch to recode_values() . Rather than taking logical vectors, recode_values() takes values on the left-hand side to match against a single input that you’ll provide as the first argument.

likert |>
  mutate(
    category = score |>
      recode_values(
        1 ~ "Strongly disagree",
        2 ~ "Disagree",
        3 ~ "Neutral",
        4 ~ "Agree",
        5 ~ "Strongly agree"
      )
  )
#> # A tibble: 9 × 2
#>   score category         
#>   <dbl> <chr>            
#> 1     1 Strongly disagree
#> 2     2 Disagree         
#> 3     3 Neutral          
#> 4     4 Agree            
#> 5     5 Strongly agree   
#> 6     2 Disagree         
#> 7     3 Neutral          
#> 8     1 Strongly disagree
#> 9     4 Agree

This removes all of the repetition, allowing you to focus on the mapping. And it should feel pretty familiar! This is the same formula interface of case_when() .

If you squint, the mapping should look roughly like a lookup table between the numeric value and the Likert encoding. One of the novel features of recode_values() is that it has an alternate interface that allows you to make this lookup table more explicit. Using a tribble() , we can extract out the lookup table into its own standalone data frame.

lookup <- tribble(
  ~from , ~to                 ,
      1 , "Strongly disagree" ,
      2 , "Disagree"          ,
      3 , "Neutral"           ,
      4 , "Agree"             ,
      5 , "Strongly agree"    ,
)

We can then utilize the alternative from and to arguments of recode_values() rather than supplying formulas to specify how the values should be recoded:

likert |>
  mutate(category = recode_values(score, from = lookup$from, to = lookup$to))
#> # A tibble: 9 × 2
#>   score category         
#>   <dbl> <chr>            
#> 1     1 Strongly disagree
#> 2     2 Disagree         
#> 3     3 Neutral          
#> 4     4 Agree            
#> 5     5 Strongly agree   
#> 6     2 Disagree         
#> 7     3 Neutral          
#> 8     1 Strongly disagree
#> 9     4 Agree

Lifting the lookup table out to the top of the file is particularly nice when you have a long pipe chain. The details of the mapping get some room to breathe, and in the pipe chain you can focus on the actual data manipulations.

It’s also very common for your lookup table to exist in a CSV file that you have to read in separately. In that case, you can replace the tribble() call with:

lookup <- readr::read_csv("lookup.csv")

Then everything else works the same.

Unmatched cases

If you are confident that you’ve captured every case during the recoding process, you can now supply unmatched = "error" as an alternative to default. recode_values() will error if that assertion doesn’t hold. This is great for defensive programming!

# Notice the `0` that we don't have a mapping for!
likert <- tibble(
  score = c(0, 1, 2, 2, 4, 5, 2, 3, 1, 4)
)

likert |>
  mutate(
    score = score |>
      recode_values(
        from = lookup$from,
        to = lookup$to,
        unmatched = "error"
      )
  )
#> Error in `mutate()`:
#> ℹ In argument: `score = recode_values(score, from = lookup$from, to = lookup$to, unmatched = "error")`.
#> Caused by error in `recode_values()`:
#> ! Each location must be matched.
#> ✖ Location 1 is unmatched.

Note that missing values must be explicitly handled when setting unmatched = "error", even if that’s just setting NA ~ NA, otherwise they will trigger the unmatched error. This forces you to explicitly opt in to expecting missing values.

Similar to recode_values() , case_when() has also gained the .unmatched argument.

replace_values()

Out of all of the new things introduced in dplyr 1.2.0, I think I’m most excited about replace_values() .

While recode_values() is great for creating an entirely new column (possibly with a new type), if you just need to replace a few rows of an existing column, then replace_values() is the best tool for the job!

Imagine we’d like to collapse some, but not all, of these school names into common buckets:

schools <- tibble(
  name = c(
    "UNC",
    "Chapel Hill",
    NA,
    "Duke",
    "Duke University",
    "UNC",
    "NC State",
    "ECU"
  )
)

We could use case_when() or recode_values() :

schools |>
  mutate(
    name = case_when(
      name %in% c("UNC", "Chapel Hill") ~ "UNC Chapel Hill",
      name %in% c("Duke", "Duke University") ~ "Duke",
      .default = name
    )
  )
#> # A tibble: 8 × 1
#>   name           
#>   <chr>          
#> 1 UNC Chapel Hill
#> 2 UNC Chapel Hill
#> 3 NA             
#> 4 Duke           
#> 5 Duke           
#> 6 UNC Chapel Hill
#> 7 NC State       
#> 8 ECU

schools |>
  mutate(
    name = recode_values(
      name,
      c("UNC", "Chapel Hill") ~ "UNC Chapel Hill",
      c("Duke", "Duke University") ~ "Duke",
      default = name
    )
  )
#> # A tibble: 8 × 1
#>   name           
#>   <chr>          
#> 1 UNC Chapel Hill
#> 2 UNC Chapel Hill
#> 3 NA             
#> 4 Duke           
#> 5 Duke           
#> 6 UNC Chapel Hill
#> 7 NC State       
#> 8 ECU

But this “partial update” operation is so common that it really deserves its own name that doesn’t require you to specify default and is type stable on the input. For that, we have replace_values() :

schools |>
  mutate(
    name = name |>
      replace_values(
        c("UNC", "Chapel Hill") ~ "UNC Chapel Hill",
        c("Duke", "Duke University") ~ "Duke"
      )
  )
#> # A tibble: 8 × 1
#>   name           
#>   <chr>          
#> 1 UNC Chapel Hill
#> 2 UNC Chapel Hill
#> 3 NA             
#> 4 Duke           
#> 5 Duke           
#> 6 UNC Chapel Hill
#> 7 NC State       
#> 8 ECU

Notice how pipe friendly replace_values() is! The first input is your “primary” input, and you can expect the output to have the same type and size as that input.

Like recode_values() , replace_values() has an alternative from and to API that works well with lookup tables and allows you to move your mapping out of the pipe chain:

lookup <- tribble(
  ~from             , ~to               ,
  "UNC"             , "UNC Chapel Hill" ,
  "Chapel Hill"     , "UNC Chapel Hill" ,
  "Duke"            , "Duke"            ,
  "Duke University" , "Duke"            ,
)

schools |>
  mutate(name = replace_values(name, from = lookup$from, to = lookup$to))
#> # A tibble: 8 × 1
#>   name           
#>   <chr>          
#> 1 UNC Chapel Hill
#> 2 UNC Chapel Hill
#> 3 NA             
#> 4 Duke           
#> 5 Duke           
#> 6 UNC Chapel Hill
#> 7 NC State       
#> 8 ECU

An extremely neat feature of the from and to API is that they also take lists of vectors that describe the mapping, which has been designed to work elegantly with the fact that tribble() can create list columns, allowing you to further collapse this lookup table:

# Condensed lookup table with a `many:1` mapping per row
lookup <- tribble(
  ~from                        , ~to               ,
  c("UNC", "Chapel Hill")      , "UNC Chapel Hill" ,
  c("Duke", "Duke University") , "Duke"            ,
)

# Note that `from` is a list column
lookup
#> # A tibble: 2 × 2
#>   from      to             
#>   <list>    <chr>          
#> 1 <chr [2]> UNC Chapel Hill
#> 2 <chr [2]> Duke

lookup$from
#> [[1]]
#> [1] "UNC"         "Chapel Hill"
#> 
#> [[2]]
#> [1] "Duke"            "Duke University"

# Works the same as before
schools |>
  mutate(name = replace_values(name, from = lookup$from, to = lookup$to))
#> # A tibble: 8 × 1
#>   name           
#>   <chr>          
#> 1 UNC Chapel Hill
#> 2 UNC Chapel Hill
#> 3 NA             
#> 4 Duke           
#> 5 Duke           
#> 6 UNC Chapel Hill
#> 7 NC State       
#> 8 ECU

The formula interface of replace_values() is a bit of a Swiss Army knife for all manner of scenarios where you might have previously reached for dplyr::coalesce() , dplyr::na_if() , or tidyr::replace_na() :

state <- c("NC", "NY", "CA", NA, "NY", "Unknown", NA)

# Replace missing values with a constant
replace_values(state, NA ~ "Unknown")
#> [1] "NC"      "NY"      "CA"      "Unknown" "NY"      "Unknown" "Unknown"

# Replace missing values with the corresponding value from another column
region <- c("South", "North", "West", "East", "North", "Unknown", "West")
replace_values(state, NA ~ region)
#> [1] "NC"      "NY"      "CA"      "East"    "NY"      "Unknown" "West"

# Replace problematic values with a missing value
replace_values(state, "Unknown" ~ NA)
#> [1] "NC" "NY" "CA" NA   "NY" NA   NA

# Standardize multiple issues at once
replace_values(state, c(NA, "Unknown") ~ "<missing>")
#> [1] "NC"        "NY"        "CA"        "<missing>" "NY"        "<missing>"
#> [7] "<missing>"

We also think it better expresses intent than if_else() or case_when() when performing a partial update:

# - Type stable on `x`
# - Intent of "partially updating" `state` is clear
# - Pipe friendly
state |> replace_values(NA ~ "Unknown")

# Historically this has been "the way" to do a partial update,
# but it's odd that the "primary" input is at the end!
if_else(is.na(state), "Unknown", state)
case_when(is.na(state) ~ "Unknown", .default = state)

If you’re a package author, you’ll probably also be interested in vctrs::vec_recode_values() and vctrs::vec_replace_values() , which are low dependency functions that power the dplyr variants.

What about case_match()?

We’ve soft-deprecated case_match() in favor of recode_values() , which is a drop in replacement.

case_match() was an incremental step towards this recoding family, but:

• It has a pretty confusing name compared with recode_values() .
• It lacked a way to work with lookup tables, like from and to.
• It lacks a replacement variant, like replace_values() .

Rather than keeping case_match() around indefinitely, we’ve decided to initiate the process of its removal since it was only introduced in dplyr 1.1.0.

Deprecations

dplyr 1.2.0 advances the lifecycle stage of many deprecated functions. These deprecations have been in the works for many years now, due to our slow and very deliberate deprecation process via the lifecycle package . We’ll cover the highlights, and you can find the full list here .

For any packages that we broke via these deprecations, we provided a pull request (or at least an issue, for complex cases) and some advance warning. We semi-automated some of this process using Claude Code, which you can read about here .

• All underscored verbs have moved from deprecated to defunct, such as mutate_() and arrange_() . These have been deprecated since dplyr 0.7.0 back in 2017 (yes, 2017!!). Use the non-underscored versions, see vignette("programming") for details.

• mutate_each() and summarise_each() have moved from deprecated to defunct. These were also deprecated in dplyr 0.7.0. Use mutate() and summarise() with across() instead.

• Returning more or less than 1 row per group in summarise() has moved from deprecated to defunct. This was deprecated in dplyr 1.1.0 in 2023 after we realized that this was an unsafe feature for summarise() , which you should always expect to return exactly 1 row per group. reframe() is a drop in replacement when you need this.

• In case_when() , supplying all size 1 LHS inputs along with a size >1 RHS input is now soft-deprecated. This is an improper usage of case_when() that should instead be a series of if statements, like:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

  # Scalars! code <- 1L flavor <- "vanilla" # Improper usage: case_when( code == 1L && flavor == "chocolate" ~ x, code == 1L && flavor == "vanilla" ~ y, code == 2L && flavor == "vanilla" ~ z, .default = default ) # Recommended: if (code == 1L && flavor == "chocolate") { x } else if (code == 1L && flavor == "vanilla") { y } else if (code == 2L && flavor == "vanilla") { z } else { default }

  The recycling behavior that allows this style of case_when() to work is unsafe, and can result in silent bugs that we’d like to guard against with an error in the future. See this issue for context.

• The dplyr.legacy_locale global option is soft-deprecated. If you used this to affect the ordering of arrange() , use arrange(.locale =) instead. If you used this to affect the ordering of group_by() |> summarise(), follow up with an additional call to arrange(.locale =) instead.

• if_else() no longer allows condition to be a logical array. It must be a logical vector with no dim attribute.

• We’ve removed a number of previously defunct functions, shrinking the footprint of dplyr’s API:

  • id()
  • failwith()
  • select_vars() and select_vars_()
  • rename_vars() and rename_vars_()
  • select_var()
  • current_vars()
  • bench_tbls(), compare_tbls(), compare_tbls2(), eval_tbls(), and eval_tbls2()
  • location() and changes()

Acknowledgements

We’d like to thank all 177 contributors who help in someway, whether it was filing issues or contributing code and documentation: @abalter , @abichat , @adupaix , @AlexBainton , @alexmcsw , @AltfunsMA , @AmeliaMN , @antdurrant , @AnthonyEbert , @apalacio9502 , @apeterson91 , @arnaudgallou , @awpsoras , @bakaburg1 , @barnabasharris , @BHII-KSC , @bholtemeyer , @billdenney , @bounlu , @brendensm , @bridroberts1 , @brookslogan , @catalamarti , @cboettig , @cbrnr , @ccani007 , @charliejhadley , @ChrisHIV , @ChristianRohde , @cobac , @conig , @const-ae , @Copilot , @d-morrison , @DanChaltiel , @daniel-simeone , @DanielBraddock , @david-romano , @davidrsch , @davidss101 , @DavisVaughan , @dcaud , @deschen1 , @DesiQuintans , @devster31 , @dkutner , @dmuenz , @ds-jim , @eitsupi , @EmilHvitfeldt , @etiennebacher , @eutwt , @EvertonTLima , @ferreirafm , @gaborcsardi , @GabryS3 , @Gastonia02 , @GBarnsley , @gevro , @ggrothendieck , @GischD , @gks281263 , @gracehartley , @graphdr , @hadley , @heliconone , @Hzanib , @ilovemane , @ja-ortiz-uniandes , @jack-davison , @james-kilgour , @JamesHWade , @jaymicro , @JBrandenburg02 , @jc-usda , @jennybc , @jeroenjanssens , @jestover , @jl5000 , @jmbarbone , @john-b-edwards , @jordanmross , @joshua-theisen , @jrwinget , @juliaapolonio , @jxu , @KaiAragaki , @kiki830621 , @KittJonathan , @kleinerChemiker , @kletts , @krlmlr , @ks8997 , @kylebutts , @larsentom , @latot , @lboller-pwbm , @lionel- , @Longfei2 , @lschneiderbauer , @LukasTang , @lukebandy , @maciekbanas , @maelle , @marcuslehr , @Mark-AP , @markwestcott34 , @maskegger , @matiasandina , @matthewjnield , @mbcann01 , @Meghansaha , @metanoid , @MichaelChirico , @MikeJohnPage , @MilesMcBain , @mine-cetinkaya-rundel , @MohsenSoltanifar , @moodymudskipper , @Moohan , @mp8 , @mpsturbo , @mr-c , @muschellij2 , @musvaage , @Mzhuk7 , @nalimilan , @nathanhaigh , @nirguk , @nmercadeb , @olivermagnanimous , @olivroy , @orgadish , @pangchaoran , @paschatz , @prubin73 , @PStaus , @psychelzh , @py9mrg , @Raesu , @randyzwitch , @Raoul-Kima , @ReedMerrill , @RodDalBen , @RodrigoZepeda , @rossholmberg , @RoyalTS , @ryandward , @sbanville-delfi , @ScientiaFelis , @shirdekel , @slager , @sschooler , @steffen-stell , @szimmer , @TheClownBongo , @thomasjwood , @TimTaylor , @tlyons253 , @tomalrussell , @tomwagstaff-opml , @torfason , @Tyrrx , @Unaimend , @VisruthSK , @vorpalvorpal , @walkerjameschris , @wbvguo , @wbzyl , @wkumler , @yaboody , @yjunechoe , @ynsec37 , @ywhcuhk , @ZHBHSMILE , @zhjx19 , and @ZIBOWANGKANGYU .

We’re thrilled to announce the release of purrr 1.2.0! purrr enhances R’s functional programming toolkit with a complete and consistent set of tools for working with functions and vectors.

You can install it from CRAN with:

install.packages("purrr")

Overall, this is a pretty unexciting release since it primarily focuses on removing long-deprecated functions. It does, however, include a couple of small performance improvements to predicate functions and a brand new getting started vignette. We also require a newer version of the carrier package for in_parallel() so that it’s easier to use. You can see a full list of changes in the release notes .

library(purrr)

Lifecycle changes

• All functions and arguments that were deprecated in purrr 0.3.0 have now been removed after being deprecated for over 5 years. These include:

  • %@%
  • accumulate_right()
  • at_depth()
  • cross_d()
  • cross_n()
  • reduce2_right()
  • reduce_right()

• All functions that were soft-deprecated in purrr 1.0.0 are now fully deprecated. They will continue to work but will generate a deprecation warning, and will be removed in a future release. This includes:

  • invoke_*() functions
  • lift_*() functions
  • cross*() functions (use tidyr::expand_grid() instead)
  • prepend()
  • splice()
  • rbernoulli()
  • rdunif()
  • when()
  • update_list()
  • *_raw() functions
  • vec_depth()

  These deprecations help keep purrr focused on its core purpose: facilitating functional programming in R.

• map_chr() no longer automatically coerces logical, integer, or double values to strings. Previously, this coercion happened silently, which could mask bugs in your code. Of the four CRAN packages that required fixes due to this change, two of them (50%) were bugs.

• The predicate functions every() , some() , and none() now require that the predicate function .p returns a logical scalar: TRUE, FALSE, or NA. Previously, NA values of other types (like NA_integer_ or NA_character_) were allowed.

Minor improvements

Apart from all the breaking changes, there were a couple of small improvements:

• every() , some() , and none() have been optimized and are now significantly faster. They’re now as fast as or faster than the equivalent any(map_lgl()) or all(map_lgl()) calls, making them the preferred choice for checking predicates across lists.

• purrr (finally) has a “getting started” vignette at vignette("purrr").

Easier in_parallel()

In purrr 1.1.0, we introduced in_parallel() for parallel processing and we’ve had great feedback from the community so far. But it was clear that we hadn’t made it easy enough to include helper functions or other variables required by your map functions. We’ve updated this behaviour in carrier 0.3.0, which is now required by purrr. Now the following (in your global environment) will work as you expect:

fn <- function(x) helper_fn(x) * 2
helper_fn <- function(x) x + 1
1:5 |> map(in_parallel(\(x) fn(x), fn = fn, helper_fn = helper_fn))

Whereas previously, fn() would have been unable to find helper_fn(), this is solved by all functions passed to in_parallel() now sharing the same environment.

Acknowledgements

We’d like to thank everyone who contributed to this release by filing issues and submitting pull requests. Your feedback and contributions help make purrr better for everyone! @feinleib , @filipemsc , @fwimp , @hadley , @its-gazza , @jcolt45 , @jeroenjanssens , @jrwinget , @khusmann , @luisDVA , @MarkPaulin , @Meghansaha , @mtcarsalot , @og2293 , @padpadpadpad , @PMassicotte , @shikokuchuo , @steffen-stell , and @wahalulu .

We’re delighted to announce the release of stringr 1.6.0! stringr provides a cohesive set of functions designed to make working with strings as easy as possible. You can install it from CRAN with:

install.packages("stringr")

This release includes some lifecycle changes, improvements to sql_like()/sql_ilike(), and a handful of other useful features. You can see a full list of changes in the release notes .

library(stringr)

Lifecycle changes

This release includes two lifecycle changes that may affect existing code.

First, the ignore_case argument to str_like() is now deprecated. Previously, it defaulted to ignore_case = TRUE. This was a mistake because it doesn’t align with the conventions of the SQL LIKE operator, which is always case sensitive. Going forward, str_like() will always be case sensitive, so if you need case-insensitive matching, use the new str_ilike() function.

Second, if you use str_replace_all() with a function for the replacement argument, that function now receives all values in a single character vector instead of being called once for each individual string. This change dramatically improves performance, so I decided it was worth it despite its impact on existing code. It shouldn’t affect too much existing code in the wild, and it only broke 11 of the 2,381 CRAN packages that use stringr (which I fixed via PR). If you haven’t used this functionality before, it’s pretty cool, because it allows you to supply a function that can transform matches in any way you can imagine. For example, this code replaces color names with their corresponding hex values:

colours <- str_c("\\b", colors(), "\\b", collapse = "|")
col2hex <- function(col) {
  rgb <- col2rgb(col)
  rgb(rgb["red", ], rgb["green", ], rgb["blue", ], maxColorValue = 255)
}

# by Claude Code
poem <- "Azure skies meet honeydew light,
While coral blooms kiss morning's face.
Gold spills across the wheat-field bright,
As lavender shadows lose their place.
Turquoise waters catch the sun,
Where salmon leap through chartreuse streams.
Orchid petals, one by one,
Fall like plum-soft summer dreams.
"
cat(str_replace_all(poem, regex(colours, ignore_case = TRUE), col2hex))
#> #F0FFFF skies meet #F0FFF0 light,
#> While #FF7F50 blooms kiss morning's face.
#> #FFD700 spills across the #F5DEB3-field bright,
#> As #E6E6FA shadows lose their place.
#> #40E0D0 waters catch the sun,
#> Where #FA8072 leap through #7FFF00 streams.
#> #DA70D6 petals, one by one,
#> Fall like #DDA0DD-soft summer dreams.

Which version do you think is catchier? 🤣

Other improvements

This release includes several other useful enhancements:

• Three new functions make it easy to convert between different programming case conventions:

  x <- "quick brown fox"
  
  str_to_camel(x)
  #> [1] "quickBrownFox"
  str_to_camel(x, first_upper = TRUE)
  #> [1] "QuickBrownFox"
  str_to_snake(x)
  #> [1] "quick_brown_fox"
  str_to_kebab(x)
  #> [1] "quick-brown-fox"
  

• All relevant stringr functions now preserve names. This means if your input vector has names, those names will be preserved in the output:

  x <- c(first = "apple", second = "banana")
  str_to_upper(x)
  #>    first   second 
  #>  "APPLE" "BANANA"
  str_detect(x, "b")
  #>  first second 
  #>  FALSE   TRUE
  

• A new vignette("locale-sensitive") provides detailed information about locale-sensitive functions in stringr, helping you understand how different locales affect string operations like sorting and case conversion.

Acknowledgements

A big thank you to everyone who contributed to this release! @alexanderbeatson , @allenbaron , @Anaherasm , @arnaudgallou , @AustinFournierKL , @brownj31 , @ChristelSwift , @DanChaltiel , @davidciani , @davidhodge931 , @Edgar-Zamora , @edward-burn , @gaborcsardi , @hadley , @jack-davison , @jdonland , @jeroenjanssens , @JFormoso , @jonovik , @KimLopezGuell , @krlmlr , @kylieainslie , @librill , @Longfei2 , @LouisMPenrod , @mararva , @mgacc0 , @MiguelCos , @nash-delcamp-slp , @ning-y , @Rekyt , @salim-b , @shaggycamel , @SoyAndrea , @tamimart , @tvedebrink , @VisruthSK , @warnes , and @wright13 .

httr2 1.2.0

We’re delighted to announce the release of httr2 1.2.0! httr2 (pronounced “hitter2”) is a comprehensive HTTP client that provides a modern, pipeable API for working with web APIs. It builds on top of {curl} to provide features like explicit request objects, built-in rate limiting & retry tooling, comprehensive OAuth support, and secure handling of secrets and credentials.

You can install it from CRAN with:

1

install.packages("httr2")

This blog post will walk you through the most important changes in 1.2.0: lifecycle updates, improved security for redacted headers, URL handling improvements, improved debugging tools, and a handful of other quality of life improvements. You can see a full list of changes in the release notes

library(httr2)

Lifecycle changes

Part of httr2’s continued evolution is phasing out features that we now believe were mistakes. In this release:

• req_perform_stream() has been soft deprecated (not just superseded) in favour of req_perform_connection() .

• Deprecated functions multi_req_perform(), req_stream(), with_mock(), and local_mock() have now been removed. Please use their modern replacements, req_perform_parallel() , req_perform_stream() , with_mocked_responses() , and local_mocked_responses() , instead.

• Deprecated arguments req_perform_parallel(pool), req_oauth_auth_code(host_name, host_ip, port), and oauth_flow_auth_code(host_name, host_ip, port) have been removed. Please use req_perform_parallel(max_active) and req_oauth_auth_code(redirect_url)/ oauth_flow_auth_code(redirect_url) instead.

Enhanced security for redacted headers

One of the most important improvements in this release improves the security of redacted headers. Redacted headers are used to conceal secrets, like API keys or passwords, that you don’t want to accidentally reveal. For a long time, httr2 has automatically hidden these headers when you print() or str() them, ensuring that they don’t accidentally end up in log files. You can see this in action with the Authorization header, which httr2 now automatically redacts:

req <- request("http://example.com") |>
  req_auth_basic("username", "password")

req
#> <httr2_request>
#> GET http://example.com
#> Headers:
#> * Authorization: <REDACTED>
#> Body: empty
str(req$headers)
#>  <httr2_headers>
#>  $ Authorization: <REDACTED>
req |> req_dry_run()
#> GET / HTTP/1.1
#> accept: */*
#> accept-encoding: deflate, gzip
#> authorization: <REDACTED>
#> host: example.com
#> user-agent: httr2/1.2.0 r-curl/6.4.0 libcurl/8.14.1

(If for you do really need to see the redacted values you can get with a bit of extra effort: call the new req_get_headers() function with redacted = "reveal".)

In httr2 1.2.0, we’ve gone one step further, and prevented redacted headers from being saved to disk. Now if you save and reload a request, you’ll notice that the redacted headers are no longer present:

path <- tempfile()
saveRDS(req, path)

req2 <- readRDS(path)
req2 |> req_dry_run()
#> GET / HTTP/1.1
#> accept: */*
#> accept-encoding: deflate, gzip
#> host: example.com
#> user-agent: httr2/1.2.0 r-curl/6.4.0 libcurl/8.14.1

This protects you from accidentally revealing your credentials if you save a request to disk. This is easier to do than you might expect because httr2 includes the request object in every response (since this makes debugging much easier). That means if you’re caching a slow response, it’s very easy to accidentally store a secret, potentially leaking secure values. (Don’t ask me how I discovered this!)

URL handling improvements

URL construction is now powered by curl::curl_modify_url() , which correctly escapes the path component:

req <- request("https://api.example.com")
req |> req_url_path("/users/john doe/profile") |> req_get_url()
#> [1] "https://api.example.com/users/john%20doe/profile"

This means that req_url_path() can now only affect the path component of the URL, not the query parameters. If you previously relied on this behaviour, you’ll need to switch to req_url_query() :

# won't work any more:
req |> 
  req_url_path("/users?name=john-doe") |> 
  req_get_url()
#> [1] "https://api.example.com/users%3Fname%3Djohn-doe"
# so now do this:
req |> 
  req_url_path("/users") |> 
  req_url_query(name = "john-doe") |> 
  req_get_url()
#> [1] "https://api.example.com/users?name=john-doe"

Improved debugging tools

The vast majority of modern APIs use JSON, so httr2 now includes a few features to make debugging those APIs a little easier:

• last_request() and last_response() are now paired with last_request_json() and last_response_json() which pretty-print the JSON bodies of the last request and response.
• req_dry_run() and req_verbose() automatically pretty print JSON bodies (turn this off by setting options(httr2_pretty_json = FALSE)).

We’ve also included a few general tools to make it easier to control httr2’s default verbosity. You can now control the default via the HTTR2_VERBOSITY environment variable and there’s a new local_verbosity() function to match the existing with_verbosity() .

Quality of life improvements

This release also includes a bunch of few smaller quality of life improvements:

• req_perform_parallel() now lifts many of its restrictions. It now supports simplified versions of req_throttle() and req_retry() , it can refresh OAuth tokens, and it checks the cache before each request.

• req_get_url() , req_get_method() , req_get_headers() , req_body_get_type(), and req_get_body() allow you to introspect request objects.

• req_throttle() now uses a “leaky bucket ”. This maintains the same average rate limit as before, while allowing bursts of higher rates.

• resp_timing() exposes detailed timing information for a response.

Acknowledgements

A big thanks to everyone who contributed to this release through issues, pull requests, and discussions on GitHub: @Aariq , @annalena13 , @apsteinmetz , @arcresu , @arnaudgallou , @atheriel , @DavidRLovell , @dfalbel , @Eli-Berkow , @fwimp , @hadley , @jansim , @jcheng5 , @jeffreyzuber , @jjesusfilho , @jonthegeek , @Kevanness , @m-muecke , @maelle , @mayeulk , @mdsumner , @noamross , @omuelle , @pedrobtz , @plietar , @ramiromagno , @salim-b , @sckott , @shikokuchuo , @vibalre , @vladimirobucina , and @ZheFrench .

We’re thrilled to announce the release of purrr 1.1.0, bringing a game-changing feature to this cornerstone of the tidyverse: parallel processing.

For the first time in purrr’s history, you can now scale your map() operations across multiple cores and even distributed systems, all while maintaining the elegant, functional programming style you know and love.

This milestone represents more than just a performance boost—it’s a fundamental shift that makes purrr suitable for production-scale data processing tasks without sacrificing the clarity and composability that make it such a joy to use.

Get started by installing purrr 1.1.0 today:

1

install.packages("purrr")

The parallel processing functionality requires the mirai and carrier packages. You will be prompted to install them when you first call in_parallel().

Ready to supercharge your functional programming workflows? Parallel purrr is here, and it’s remarkably simple to use.

The power of in_parallel()

The magic happens through a shiny new function: in_parallel(). This purrr adverb wraps your functions to signal that they should run in parallel, powered by the venerable mirai package.

Here’s how simple it is to transform your sequential operations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

library(purrr)
library(mirai)

# Set up parallel processing (6 background processes)
daemons(6)

# Sequential version
mtcars |> map_dbl(\(x) mean(x))
#>    mpg    cyl   disp     hp   drat     wt   qsec     vs     am   gear   carb 
#>  20.09   6.19 230.72 146.69   3.60   3.22  17.85   0.44   0.41   3.69   2.81

# Parallel version - just wrap your function with in_parallel()
mtcars |> map_dbl(in_parallel(\(x) mean(x)))
#>    mpg    cyl   disp     hp   drat     wt   qsec     vs     am   gear   carb 
#>  20.09   6.19 230.72 146.69   3.60   3.22  17.85   0.44   0.41   3.69   2.81

# Don't forget to clean up when done
daemons(0)

The results are identical, but the second version distributes the work across multiple CPU cores. For computationally intensive tasks, the performance gains can be dramatic.

The beauty of using an adverb is that in_parallel() works not just with map(), but across the entire purrr ecosystem:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

daemons(6, output = TRUE)

# Works with all map variants
1:4 |> map_int(in_parallel(\(x) x^2))
1:4 |> map_chr(in_parallel(\(x) paste("Number", x)))

# Works with map2 and pmap
map2_dbl(1:3, 4:6, in_parallel(\(x, y) x + y))

list(a = 1:3, b = 4:6, c = 7:9) |>
  pmap_dbl(in_parallel(\(a, b, c) mean(c(a, b, c))))

# Even works with walk for side effects
1:3 |> walk(in_parallel(\(x) cat("Processing", x, "\n")))

daemons(0)

If you use in_parallel() but don’t set daemons(), then the map will just proceed sequentially, so you don’t need to worry about having two separate code paths for parallel vs non-parallel execution.

Real-world example: parallel model fitting

Let’s look at a more realistic scenario where parallel processing truly shines—fitting multiple models:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

library(purrr)
library(mirai)

# Set up 4 parallel processes
daemons(4)

# Define a slow model fitting function
slow_lm <- function(formula, data) {
  Sys.sleep(0.1)  # Simulate computational complexity
  lm(formula, data)
}

# Fit models to different subsets of data in parallel
models <- mtcars |>
  split(mtcars$cyl) |>
  map(in_parallel(\(df) slow_lm(mpg ~ wt + hp, data = df), slow_lm = slow_lm))

# Extract R-squared values
models |> 
  map(summary) |> 
  map_dbl("r.squared")
#>         4         6         8 
#> 0.6807065 0.5889239 0.4970692

daemons(0)

Notice how we pass the slow_lm function as an argument to in_parallel()—this ensures our custom function is available in the parallel processes.

Production-ready with mirai

The choice of mirai as the parallel backend wasn’t arbitrary. mirai is a production-grade async evaluation framework that brings several key advantages:

• Minimal overhead: Built on modern networking and concurrency principles
• Reliable scheduling: Leveraging fast inter-process communications locally
• Scalable architecture: From multi-process to distributed computing on HPC clusters
• Security: Offers zero-configuration TLS over TCP for additional assurance

This means your parallel purrr code isn’t just fast—it’s production-ready.

Compared to the furrr package:

• Much lower overhead means you can get a performance boost even for relatively fast functions
• More linear scaling means you get the same benefits whether you’re running on 2 or 200 cores

We’ve learned a lot from our work on furrr, and from Henrik Bengtsson ’s excellent work on the futureverse ecosystem. purrr doesn’t use future as the underlying engine for parallelism because we’ve made some design decisions that differ at a fundamental level, but Henrik’s entire ecosystem deserves credit for pushing the boundaries of parallelism in R farther than many thought possible.

Creating self-contained functions

One of the key concepts when using in_parallel() is creating self-contained functions. Since your function gets serialized and sent to parallel processes, it needs to be completely standalone:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# ❌ This won't work - external dependencies not declared
my_data <- c(1, 2, 3)
map(1:3, in_parallel(\(x) mean(my_data)))

# ✅ This works - dependencies explicitly provided
my_data <- c(1, 2, 3)
map(1:3, in_parallel(\(x) mean(my_data), my_data = my_data))

# ✅ Package functions need explicit namespacing
map(1:3, in_parallel(\(x) vctrs::vec_init(integer(), x)))

# ✅ Or load packages within the function
map(1:3, in_parallel(\(x) {
  library(vctrs)
  vec_init(integer(), x)
}))

This explicit dependency management might seem verbose, but it ensures your parallel code is reliable and predictable—crucial for production environments.

It also removes the danger of accidentally shipping large objects to parallel processes—often a source of performance degradation.

When to use parallel processing

Not every map() operation benefits from parallelization. The overhead of setting up parallel tasks and communicating between processes can outweigh the benefits for simple operations. As a rule of thumb, consider parallel processing when:

• Each iteration takes at least 100 microseconds to 1 millisecond
• You’re performing CPU-intensive computations
• You’re working with I/O-bound operations that can benefit from concurrency
• The data being passed between processes isn’t excessively large

For quick operations like simple arithmetic, sequential processing will often be faster.

If you’re a package developer, use in_parallel() where you see fit, but please be mindful not to call daemons() within your package code. How to set mirai daemons should be always be for the end user to decide.

Distributed computing made simple

Want to scale beyond your local machine? mirai’s networking capabilities make distributed computing surprisingly straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

library(mirai)

# Set up remote daemons on a Slurm HPC cluster
daemons(
  n = 100,
  url = host_url(),
  remote = cluster_config(command = "sbatch")
)

# Your purrr code remains exactly the same!
results <- big_dataset |>
  split(big_dataset$group) |>
  map(in_parallel(\(df) complex_analysis(df), complex_analysis = complex_analysis))

daemons(0)

The same in_parallel() syntax that works locally scales seamlessly to distributed systems.

Please refer to the mirai documentation on remote daemons and launching remote daemons for more details. This mirai blog post will also be useful if you’re working with High-Performance Computing (HPC) clusters.

Looking forward

The addition of parallel processing to purrr 1.1.0 represents a significant evolution in the package’s capabilities. It maintains purrr’s core philosophy of functional programming while opening doors to high-performance computing scenarios that were previously challenging to achieve with such clean, readable code.

This feature is currently marked as experimental as we gather feedback from the community, but the underlying mirai infrastructure is production-proven and battle-tested. We encourage you to try it out and let us know about your experiences.

Whether you’re processing large datasets, fitting complex models, or running simulations, purrr 1.1.0’s parallel processing capabilities can help you scale your R workflows without sacrificing code clarity or reliability.

Acknowledgements

A big thanks to all those who posted issues and contributed PRs since our last release! @ar-puuk , @DanChaltiel , @davidrsch , @ErdaradunGaztea , @h-a-graham , @hadley , @HenningLorenzen-ext-bayer , @krivit , @MarceloRTonon , @MarkPaulin , @salim-b , @ScientiaFelis , @shikokuchuo , and @sierrajohnson .

install.packages("duckplyr")

This article shows how duckplyr can be used instead of dplyr, explain how you can help improve the package, and share a selection of further resources.

A drop-in replacement for dplyr

Imagine you have to wrangle a huge dataset, like this one from the TPC-H benchmark , a famous database benchmarking dataset.

lineitem_tbl <- duckdb:::sql("INSTALL tpch; LOAD tpch; CALL dbgen(sf=1); FROM lineitem;")
lineitem_tbl <- tibble::as_tibble(lineitem_tbl)
dplyr::glimpse(lineitem_tbl)
#> Rows: 6,001,215
#> Columns: 16
#> $ l_orderkey      <dbl> 1, 1, 1, 1, 1, 1, 2, 3, 3, 3, 3, 3, 3, 4, 5, 5, 5, 6, …
#> $ l_partkey       <dbl> 155190, 67310, 63700, 2132, 24027, 15635, 106170, 4297…
#> $ l_suppkey       <dbl> 7706, 7311, 3701, 4633, 1534, 638, 1191, 1798, 6540, 3…
#> $ l_linenumber    <dbl> 1, 2, 3, 4, 5, 6, 1, 1, 2, 3, 4, 5, 6, 1, 1, 2, 3, 1, …
#> $ l_quantity      <dbl> 17, 36, 8, 28, 24, 32, 38, 45, 49, 27, 2, 28, 26, 30, …
#> $ l_extendedprice <dbl> 21168.23, 45983.16, 13309.60, 28955.64, 22824.48, 4962…
#> $ l_discount      <dbl> 0.04, 0.09, 0.10, 0.09, 0.10, 0.07, 0.00, 0.06, 0.10, …
#> $ l_tax           <dbl> 0.02, 0.06, 0.02, 0.06, 0.04, 0.02, 0.05, 0.00, 0.00, …
#> $ l_returnflag    <chr> "N", "N", "N", "N", "N", "N", "N", "R", "R", "A", "A",…
#> $ l_linestatus    <chr> "O", "O", "O", "O", "O", "O", "O", "F", "F", "F", "F",…
#> $ l_shipdate      <date> 1996-03-13, 1996-04-12, 1996-01-29, 1996-04-21, 1996-…
#> $ l_commitdate    <date> 1996-02-12, 1996-02-28, 1996-03-05, 1996-03-30, 1996-…
#> $ l_receiptdate   <date> 1996-03-22, 1996-04-20, 1996-01-31, 1996-05-16, 1996-…
#> $ l_shipinstruct  <chr> "DELIVER IN PERSON", "TAKE BACK RETURN", "TAKE BACK RE…
#> $ l_shipmode      <chr> "TRUCK", "MAIL", "REG AIR", "AIR", "FOB", "MAIL", "RAI…
#> $ l_comment       <chr> "to beans x-ray carefull", " according to the final fo…

To work with this in duckplyr instead of dplyr, all you need to do is load duckplyr:

library(duckplyr)
#> Loading required package: dplyr
#> The duckplyr package is configured to fall back to dplyr when it encounters an incompatibility.
#> Fallback events can be collected and uploaded for analysis to guide future development. By
#> default, data will be collected but no data will be uploaded.
#> ℹ Automatic fallback uploading is not controlled and therefore disabled, see
#>   `?duckplyr::fallback()`.
#> ✔ Number of reports ready for upload: 4.
#> → Review with `duckplyr::fallback_review()`, upload with `duckplyr::fallback_upload()`.
#> ℹ Configure automatic uploading with `duckplyr::fallback_config()`.
#> ✔ Overwriting dplyr methods with duckplyr methods.
#> ℹ Turn off with `duckplyr::methods_restore()`.

Now we can express the well-known (at least in the database community!) “TPC-H benchmark query 1” in dplyr syntax and execute it in DuckDB via duckplyr.

tpch_dplyr <- function(lineitem) {
  lineitem |>
    filter(l_shipdate <= !!as.Date("1998-09-02")) |>
    summarise(
      sum_qty = sum(l_quantity),
      sum_base_price = sum(l_extendedprice),
      sum_disc_price = sum(l_extendedprice * (1 - l_discount)),
      sum_charge = sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)),
      avg_qty = mean(l_quantity),
      avg_price = mean(l_extendedprice),
      avg_disc = mean(l_discount),
      count_order = n(),
      .by = c(l_returnflag, l_linestatus)
    ) |>
    arrange(l_returnflag, l_linestatus)
}

tpch_dplyr(lineitem_tbl)
#> # A tibble: 4 × 10
#>   l_returnflag l_linestatus  sum_qty sum_base_price sum_disc_price    sum_charge
#>   <chr>        <chr>           <dbl>          <dbl>          <dbl>         <dbl>
#> 1 A            F            37734107   56586554401.   53758257135.  55909065223.
#> 2 N            F              991417    1487504710.    1413082168.   1469649223.
#> 3 N            O            74476040  111701729698.  106118230308. 110367043872.
#> 4 R            F            37719753   56568041381.   53741292685.  55889619120.
#> # ℹ 4 more variables: avg_qty <dbl>, avg_price <dbl>, avg_disc <dbl>,
#> #   count_order <int>

Like other dplyr backends such as dtplyr and dbplyr, duckplyr gives you higher performance without learning a different syntax. Unlike other dplyr backends, duckplyr does not require you to change existing code or learn specific idiosyncrasies. Not only is the syntax the same, the semantics are too! If an operation cannot be carried out with DuckDB, it is automatically outsourced to dplyr. Over time, we expect fewer and fewer fallbacks to dplyr to be needed.

How to use duckplyr

There are two ways to use duckplyr:

• As above, you can library(duckplyr) , and replace all existing dplyr methods. This is safe because duckplyr is guaranteed to give the exactly same the results as dplyr, unlike other backends.

• Create individual “duck frames” using conversion functions like duckdplyr::duckdb_tibble() or duckdplyr::as_duckdb_tibble(), or ingestion functions like duckdplyr::read_csv_duckdb().

Here’s an example of the second form:

out <- lineitem_tbl |>
  duckplyr::as_duckdb_tibble() |>
  tpch_dplyr()

out
#> # A duckplyr data frame: 10 variables
#>   l_returnflag l_linestatus  sum_qty sum_base_price sum_disc_price    sum_charge
#>   <chr>        <chr>           <dbl>          <dbl>          <dbl>         <dbl>
#> 1 A            F            37734107   56586554401.   53758257135.  55909065223.
#> 2 N            F              991417    1487504710.    1413082168.   1469649223.
#> 3 N            O            74476040  111701729698.  106118230308. 110367043872.
#> 4 R            F            37719753   56568041381.   53741292685.  55889619120.
#> # ℹ 4 more variables: avg_qty <dbl>, avg_price <dbl>, avg_disc <dbl>,
#> #   count_order <int>

Note that the resulting object is indistinguishable from a regular tibble, except for the additional class.

typeof(out)
#> [1] "list"
class(out)
#> [1] "duckplyr_df" "tbl_df"      "tbl"         "data.frame"
out$count_order
#> [1] 1478493   38854 2920374 1478870

Operations not yet supported by duckplyr are automatically outsourced to dplyr. For instance, filtering on grouped data is not supported, but it still works thanks to the fallback mechanism. By default, the fallback is silent, but you can make it visible by setting an environment variable. This is useful if you want to better understanding what’s making your code slow.

Sys.setenv(DUCKPLYR_FALLBACK_INFO = TRUE)

lineitem_tbl |>
  duckplyr::as_duckdb_tibble() |>
  filter(l_quantity == max(l_quantity), .by = c(l_returnflag, l_linestatus))
#> Cannot process duckplyr query with DuckDB, falling back to dplyr.
#> ℹ `filter(.by = ...)` not implemented, try `mutate(.by = ...)` followed by a simple `filter()`.
#> # A duckplyr data frame: 16 variables
#>    l_orderkey l_partkey l_suppkey l_linenumber l_quantity l_extendedprice
#>         <dbl>     <dbl>     <dbl>        <dbl>      <dbl>           <dbl>
#>  1          5     37531        35            3         50          73426.
#>  2        131     44255      9264            2         50          59962.
#>  3        199    132072      9612            1         50          55204.
#>  4        231    198124       644            3         50          61106 
#>  5        260    155887      5888            1         50          97144 
#>  6        263    142891       434            3         50          96694.
#>  7        323    163628      1177            1         50          84581 
#>  8        354     58125      8126            3         50          54156 
#>  9        484    183351      5870            3         50          71718.
#> 10        485    149523      9524            1         50          78626 
#> # ℹ more rows
#> # ℹ 10 more variables: l_discount <dbl>, l_tax <dbl>, l_returnflag <chr>,
#> #   l_linestatus <chr>, l_shipdate <date>, l_commitdate <date>,
#> #   l_receiptdate <date>, l_shipinstruct <chr>, l_shipmode <chr>,
#> #   l_comment <chr>

You can also directly use DuckDB functions with the dd$ qualifier. Functions with this prefix will not be translated at all and passed through directly to DuckDB. For example, the following code uses DuckDB’s internal implementation of Levenstein distance :

tibble(a = "dbplyr", b = "duckplyr") %>%
  mutate(c = dd$levenshtein(a, b))
#> # A tibble: 1 × 3
#>   a      b            c
#>   <chr>  <chr>    <dbl>
#> 1 dbplyr duckplyr     3

See vignette("duckdb") for more information on these features.

If you’re working with dbplyr too, you can use as_tbl() you to convert a duckplyr tibble to a dbplyr lazy table. This allows you to seamlessly interact with existing code that might use inline SQL or other dbplyr functionality. With as_duckdb_tibble() , you can convert a dbplyr lazy table to a duckplyr tibble. Both operations work without intermediate materialization.

Benchmark

duckplyr is often much faster than dplyr. The comparison below is done in a fresh R session where dplyr is attached but duckplyr is not.

We use tpch_dplyr() as defined above to run the query with dplyr. The function that runs it with duckplyr only wraps the input data in a duck frame and forwards it to the dplyr function. The collect() at the end is required only for this benchmark to ensure fairness.²

tpch_duckplyr <- function(lineitem) {
  lineitem |>
    duckplyr::as_duckdb_tibble() |>
    tpch_dplyr() |>
    collect()
}

And now we compare the two:

bench::mark(
  tpch_dplyr(lineitem_tbl),
  tpch_duckplyr(lineitem_tbl),
  check = ~ all.equal(.x, .y, tolerance = 1e-10)
)
#> Warning: Some expressions had a GC in every iteration; so filtering is disabled.
#> # A tibble: 2 × 6
#>   expression                       min   median `itr/sec` mem_alloc `gc/sec`
#>   <bch:expr>                  <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
#> 1 tpch_dplyr(lineitem_tbl)     611.6ms  611.6ms      1.64    1.25GB     1.64
#> 2 tpch_duckplyr(lineitem_tbl)   71.4ms   72.3ms     13.8   314.38KB     0

In this example, duckplyr is a lot faster than dplyr. It also appears to use much less memory, but this is misleading: DuckDB manages the memory, not R, so the memory usage is not visible to bench::mark() .

Out-of-memory data

As well as improved speed with in-memory datasets, duckplyr makes it easy to work with datasets that are too big to fit in memory. In this case, you want:

1. To work with data stored in modern formats designed for large data (e.g. Parquet).
2. To be able to store large intermediate results on disk, keeping them out of memory.
3. Fast computation!

duckdplyr provides each of these features:

1. You can read data from disk with functions like read_parquet_duckdb() .
2. You can save intermediate results to disk with compute_parquet() and compute_csv() .
3. duckdplyr takes advantage of DuckDB’s query planner which considers your entire pipeline holistically to figure out the most efficient way to get the data you need.

See vignette("large") for a walkthrough and more details.

Help us improve duckplyr!

Our goals for future development of duckplyr include:

• Enabling users to provide custom translations of dplyr functionality;
• Making it easier to contribute code to duckplyr;
• Supporting more dplyr and tidyr functionality natively in DuckDB.

You can help!

• Please report any issues, especially regarding unknown incompabilities. See vignette("limits") .
• Contribute to the codebase after reading duckplyr’s contributing guide .
• Turn on telemetry to help us hear about the most frequent fallbacks so we can prioritize working on the corresponding missing dplyr translation. See vignette("telemetry") and duckplyr::fallback_sitrep() .

Additional resources

Eager to learn more about duckplyr – beside by trying it out yourself? The duckplyr website features several articles . Furthermore, the blog post “duckplyr: dplyr Powered by DuckDB” by Hannes Mühleisen provides some context on duckplyr including its inner workings, as also seen in a section of the R-hub blog post “Lazy introduction to laziness in R” by Maëlle Salmon, Athanasia Mo Mowinckel and Hannah Frick.

Acknowledgements

A big thanks to all folks who filed issues, created PRs and generally helped to improve duckplyr and its workhorse duckdb !

@adamschwing , @alejandrohagan , @andreranza , @apalacio9502 , @apsteinmetz , @barracuda156 , @beniaminogreen , @bob-rietveld , @brichards920 , @cboettig , @davidjayjackson , @DavisVaughan , @Ed2uiz , @eitsupi , @era127 , @etiennebacher , @eutwt , @fmichonneau , @hadley , @hannes , @hawkfish , @IndrajeetPatil , @JanSulavik , @JavOrraca , @jeroen , @jhk0530 , @joakimlinde , @JosiahParry , @kevbaer , @larry77 , @lnkuiper , @lorenzwalthert , @lschneiderbauer , @luisDVA , @math-mcshane , @meersel , @multimeric , @mytarmail , @nicki-dese , @PMassicotte , @prasundutta87 , @rafapereirabr , @Robinlovelace , @romainfrancois , @sparrow925 , @stefanlinner , @szarnyasg , @thomasp85 , @TimTaylor , @Tmonster , @toppyy , @wibeasley , @yjunechoe , @ywhcuhk , @zhjx19 , @ablack3 , @actuarial-lonewolf , @ajdamico , @amirmazmi , @anderson461123 , @andrewGhazi , @Antonov548 , @appiehappie999 , @ArthurAndrews , @arthurgailes , @babaknaimi , @bcaradima , @bdforbes , @bergest , @bill-ash , @BorgeJorge , @brianmsm , @chainsawriot , @ckarnes , @clementlefevre , @cregouby , @cy-james-lee , @daranzolin , @david-cortes , @DavZim , @denis-or , @developertest1234 , @dicorynia , @dsolito , @e-kotov , @EAVWing , @eddelbuettel , @edward-burn , @elefeint , @eli-daniels , @elysabethpc , @erikvona , @florisvdh , @gaborcsardi , @ggrothendieck , @hdmm3 , @hope-data-science , @IoannaNika , @jabrown-aepenergy , @JamesLMacAulay , @jangorecki , @javierlenzi , @Joe-Heffer-Shef , @kalibera , @lboller-pwbm , @lgaborini , @m-muecke , @meztez , @mgirlich , @mtmorgan , @nassuphis , @nbc , @olivroy , @pdet , @phdjsep , @pierre-lamarche , @r2evans , @ran-codes , @rplsmn , @Saarialho , @SimonCoulombe , @tau31 , @thohan88 , @ThomasSoeiro , @timothygmitchell , @vincentarelbundock , @VincentGuyader , @wlangera , @xbasics , @xiaodaigh , @xtimbeau , @yng-me , @Yousuf28 , @yutannihilation , and @zcatav

Special thanks to Joe Thorley (@joethorley ) for help with choosing the right words.

Data validation is a very important part of any data pipeline. And with Polars gaining popularity as a superfast and feature-packed DataFrame library, developers need validation tools that work seamlessly with it. But here’s the thing: not all validation libraries are created equal, and choosing the wrong one can lead to frustration, technical debt, or validation gaps that could bite you later.

In this survey (conducted halfway through 2025) we’ll explore five Python validation libraries that support Polars DataFrames, each bringing distinct strengths to different validation challenges.

Recommendations

Here are the unique strengths for each library:

Based on these strengths, here are my recommendations for which libraries to use according to use case:

Setup

We are going to run through examples with Pandera, Patito, Pointblank, Validoopsie, and Dataframely, using this Polars DataFrame as our test case:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

import polars as pl

# Standard dataset for all validation examples
user_data = pl.DataFrame({
    "user_id": [1, 2, 3, 4, 5],
    "age": [25, 30, 22, 45, 95],  # <- includes a very high age
    "email": [
        "user1@example.com", "user2@example.com", "invalid-email",  # <- has an invalid email
        "user4@example.com", "user5@example.com"
    ],
    "score": [85.5, 92.0, 78.3, 88.7, 95.2]
})

We’ll try to run the same data validation across the surveyed libraries, so we’ll check:

• schema validation (correct column types)
• user_id values greater than 0
• age values between 18 and 80 (inclusive)
• email strings matching a basic email regex pattern
• score values between 0 and 100 (inclusive)

Now let’s dive into each library, starting with the statistically-focused Pandera.

1. Pandera: Schema-First Validation with Statistical Checks

Pandera is a statistical data validation toolkit designed to provide a flexible and expressive API for performing data validation on dataframe-like objects. The library centers on schema-centric validation, where you define the expected structure and constraints of your data upfront. You can enable both runtime validation and static type checking integration. Pandera added Polars support in version 0.19.0 (early 2024).

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

import pandera.polars as pa

# Define schema using our standard dataset
schema = pa.DataFrameSchema({
    "user_id": pa.Column(pl.Int64, checks=pa.Check.gt(0)),
    "age": pa.Column(pl.Int64, checks=[pa.Check.ge(18), pa.Check.le(80)]),
    "email": pa.Column(pl.Utf8, checks=pa.Check.str_matches(r"^[^@]+@[^@]+\.[^@]+$")),
    "score": pa.Column(pl.Float64, checks=pa.Check.in_range(0, 100))
})

# Validate the schema
try:
    validated_data = schema.validate(user_data)
    print("Validation successful!")
except pa.errors.SchemaError as e:
    print(f"Validation failed: {e}")

Validation failed: Column 'age' failed validator number 1: <Check less_than_or_equal_to: less_than_or_equal_to(80)> failure case examples: [{'age': 95}]

This example demonstrates Pandera’s declarative approach, where you define what your data should look like rather than writing imperative validation logic. The schema acts as both documentation and as a validation contract. Notice how multiple checks can be applied to a single column (here, the age column receives two checks), and the validation either succeeds completely or provides error information about what failed.

Comparisons

Both Pandera and Patito use declarative, schema-centric approaches, but differ in their design philosophies:

• Pandera uses a dictionary-like schema structure with Column objects for defining validation rules
• Patito uses Pydantic model classes with familiar Field syntax for validation constraints
• Pandera focuses heavily on statistical validation capabilities like hypothesis testing
• Patito emphasizes integration with existing Pydantic workflows and object modeling
• a key behavioral difference: Patito reports all validation errors in a single pass, while Pandera stops at the first failure

The choice between them often comes down to whether you prefer Pandera’s statistical focus or Patito’s Pydantic integration.

Unlike Pointblank’s step-by-step validation reporting, Pandera validates the entire schema at once. Compared to Patito’s model-based approach, Pandera focuses more on statistical validation capabilities. Unlike Validoopsie’s and Pointblank’s method chaining style, Pandera uses a more declarative, schema-centric approach.

Unique Strengths and When to Use

Here are some of stand-out features that Pandera has:

• type-safe schema definitions with mypy integration
• statistical hypothesis testing for data distributions: perform t-tests, chi-square tests, and custom statistical tests directly in your validation schema
• excellent integration with Pandas, Polars, and Arrow support
• declarative schema syntax that serves as documentation
• built-in support for data coercion and transformation

This statistical validation capability goes beyond basic type and range checking to test actual data relationships and distributional assumptions. For example, you can validate that the mean height of group "M" is significantly greater than group "F" using a two-sample t-test, or test whether a column follows a normal distribution. This makes Pandera uniquely powerful for data science workflows where the statistical properties of your data are as important as individual data points meeting basic constraints.

Data practitioners should choose Pandera when building type-safe data pipelines where schema validation is critical, especially in data science workflows that require statistical validation. It’s ideal for users that value static type checking, need to validate statistical properties of their data, or want schemas that double as documentation.

Pandera also excels in environments where data contracts between teams are important and where the statistical properties of data matter as much as basic type checking.

2. Patito: Pydantic-Style Data Models for DataFrames

Patito brings Pydantic’s well-received model-based validation approach to DataFrame validation, creating a bridge between Pydantic-style data validation and DataFrame processing. The library’s primary goal is to provide a familiar, Pydantic-style interface for defining and validating DataFrame schemas, making it particularly appealing to developers already using Pydantic in their applications.

Patito launched with Polars support from the beginning (in late 2022). Native Polars integration is touted as one of its core features, reflecting the growing adoption of Polars in the Python ecosystem.

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

import patito as pt
from typing import Annotated

class UserModel(pt.Model):
    user_id: int = pt.Field(gt=0)
    age: Annotated[int, pt.Field(ge=18, le=80)]
    email: str = pt.Field(pattern=r"^[^@]+@[^@]+\.[^@]+$")
    score: float = pt.Field(ge=0.0, le=100.0)

# Validate using the model
try:
    UserModel.validate(user_data)
    print("Validation successful!")
except pt.exceptions.DataFrameValidationError as e:
    print(f"Validation failed: {e}")

Validation failed: 2 validation errors for UserModel
age
  1 row with out of bound values. (type=value_error.rowvalue)
email
  1 row with out of bound values. (type=value_error.rowvalue)

This example showcases Patito’s model-centric approach where validation rules are embedded in class definitions. The use of Python’s type hints and Pydantic’s Field syntax makes the validation rules self-documenting. Notably, Patito reports all validation errors at once, providing a fairly comprehensive view of data quality issues, whereas other libraries (e.g., Pandera) stop at the first failure.

Column Validation Approaches: Pandera vs Patito

Pandera offers a much more extensive and flexible system for column validation compared to Patito’s field-based approach. While Patito provides a solid set of built-in field constraints (like gt, le, regex, unique, etc.) that cover common validation scenarios, Pandera’s Check system is designed for both simple and highly sophisticated validation logic.

The key architectural difference seems to lie in extensibility and complexity. Pandera’s Check objects accept arbitrary functions, allowing you to write custom validation logic that can be as simple as lambda s: s > 0 or as complex as statistical hypothesis tests using scipy. You can create vectorized checks that operate on entire Series objects for performance, element-wise checks for atomic validation, and even grouped checks that validate subsets of data based on other columns. Patito’s Field constraints, while clean and declarative, are more limited to the predefined validation types that Pydantic and Patito provide.

Pandera also supports advanced validation patterns that Patito doesn’t directly offer, such as wide-form data checks (validating relationships across multiple columns), grouped validation (where checks are applied to subsets of data based on grouping columns), and the ability to raise warnings instead of errors for non-critical validation failures. While Patito does support custom constraints through Polars expressions via the constraints parameter, this requires knowledge of Polars expression syntax and, depending on where you’re coming from, could be less intuitive than Pandera’s function-based approach.

For most common validation scenarios, Patito’s field-based validation is simpler and more readable, especially for teams already familiar with Pydantic. However, for complex data validation requirements, statistical validation, or when you need maximum flexibility in defining validation logic, Pandera’s Check system provides significantly more power and extensibility.

Unique Strengths and When to Use

• Pydantic-style model definitions with familiar syntax for Pydantic users
• rich type system integration with Python’s typing system
• model inheritance and composition for complex data structures
• seamless integration with existing Pydantic-based applications
• row-level object modeling for converting DataFrame rows to Python objects with methods
• mock data generation for testing with .examples() method

People should choose Patito when they’re already using Pydantic in their applications and want consistent validation patterns across data processing and application logic. It’s great when you need to validate DataFrames and then work with individual rows as rich Python objects with embedded business logic and methods (e.g., a Product row that has a .url property or .calculate_discount() method). Patito is also good when you need to generate realistic test data and want object-oriented interfaces for their data models.

3. Pointblank: Comprehensive Validation with Beautiful Reports

Pointblank is a comprehensive data validation framework designed to make data quality assessment both thorough and accessible to stakeholders. Originally inspired by the R package of the same name, Pointblank’s primary mission is to provide validation workflows that generate beautiful, interactive reports that can be shared with both technical and non-technical team members.

Pointblank launched with Polars support as a core feature from its initial Python release in late 2024, built on top of the Narwhals and Ibis compatibility layers to provide consistent DataFrame operations across multiple backends including Polars, Pandas, and database connections.

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

import pointblank as pb

schema = pb.Schema(
    columns=[("user_id", "Int64"), ("age", "Int64"), ("email", "String"), ("score", "Float64")]
)

validation = (
    pb.Validate(data=user_data, label="An example.", tbl_name="users", thresholds=(0.1, 0.2, 0.3))
    .col_vals_gt(columns="user_id", value=0)
    .col_vals_between(columns="age", left=18, right=80)
    .col_vals_regex(columns="email", pattern=r"^[^@]+@[^@]+\.[^@]+$")
    .col_vals_between(columns="score", left=0, right=100)
    .col_schema_match(schema=schema)
    .interrogate()
)

validation