Software by Charlie Gao#
Events attended by Charlie Gao#
Posts and resources by Charlie Gao#
Purrrfectly parallel, purrrfectly distributed (Charlie Gao, Posit) | posit::conf(2025)
Purrrfectly parallel, purrrfectly distributed
Speaker(s): Charlie Gao
Abstract:
purrr is a powerful functional programming toolkit that has long been a cornerstone of the tidyverse. In 2025, it receives a modernization that means you can use it to harness the power of all computing cores on your machine, dramatically speeding up map operations.
More excitingly, it opens up the doors to distributed computing. Through the mirai framework used by purrr, this is made embarrassingly simple. For those in small businesses, or even large ones – every case where there is a spare server in your network, you can now put to good use in simple, straightforward steps.
Let us show you how distributed computing is no longer the preserve of those with access to high performance compute clusters.
Materials - https://shikokuchuo-posit2025.share.connect.posit.cloud/ posit::conf(2025) Subscribe to posit::conf updates: https://posit.co/about/subscription-management/
{mirai} and {crew}: next-generation async to supercharge {promises}, Plumber, Shiny, and {targets}
{mirai} is a minimalist, futuristic, and reliable way to parallelise computations – either on the local machine, or across the network. It combines the latest scheduling technologies with fast, secure connection types. With built-in integration to {promises}, {mirai} provides a simple and efficient asynchronous back-end for Shiny and Plumber apps. The {crew} package extends {mirai} to batch computing environments for massively parallel statistical pipelines, e.g. Bayesian modeling, simulations, and machine learning. It consolidates tasks in a central {R6} controller, auto-scales workers, and helps users create plug-ins for platforms like SLURM and AWS Batch. It is the new workhorse powering high-performance computing in {targets}.
Talk by Charlie Gao and Will Landau
Slides: https://wlandau.github.io/posit2024
GitHub Repo: https://github.com/wlandau/posit2024
mirai: https://shikokuchuo.net/mirai/
crew: https://wlandau.github.io/crew/