sparklyr and friends have been getting some important updates in the past few months, here are some highlights:

• spark_apply() now works on Databricks Connect v2

• sparkxgb is coming back to life

• Support for Spark 2.3 and below has ended

pysparklyr 0.1.4

spark_apply() now works on Databricks Connect v2. The latest pysparklyr release uses the rpy2 Python library as the backbone of the integration.

Databricks Connect v2, is based on Spark Connect. At this time, it supports Python user-defined functions (UDFs), but not R user-defined functions. Using rpy2 circumvents this limitation. As shown in the diagram, sparklyr sends the the R code to the locally installed rpy2, which in turn sends it to Spark. Then the rpy2 installed in the remote Databricks cluster will run the R code.

A big advantage of this approach, is that rpy2 supports Arrow. In fact it is the recommended Python library to use when integrating Spark, Arrow and R . This means that the data exchange between the three environments will be much faster!

As in its original implementation, schema inferring works, and as with the original implementation, it has a performance cost. But unlike the original, this implementation will return a ‘columns’ specification that you can use for the next time you run the call.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

spark_apply(
  tbl_mtcars,
  nrow,
  group_by = "am"
)

#> To increase performance, use the following schema:
#> columns = "am double, x long"

#> # Source:   table<`sparklyr_tmp_table_b84460ea_b1d3_471b_9cef_b13f339819b6`> [2 x 2]
#> # Database: spark_connection
#>      am     x
#>   <dbl> <dbl>
#> 1     0    19
#> 2     1    13

A full article about this new capability is available here: Run R inside Databricks Connect

sparkxgb

The sparkxgb is an extension of sparklyr. It enables integration with XGBoost . The current CRAN release does not support the latest versions of XGBoost. This limitation has recently prompted a full refresh of sparkxgb. Here is a summary of the improvements, which are currently in the development version of the package :

• The xgboost_classifier() and xgboost_regressor() functions no longer pass values of two arguments. These were deprecated by XGBoost and cause an error if used. In the R function, the arguments will remain for backwards compatibility, but will generate an informative error if not left NULL:

  • sketch_eps - As of XGBoost version 1.6.0 sketch_eps was replaced by max_bins
  • timeout_request_workers - Removed in XGBoost version 1.7.0 because it was no longer needed when XGBoost added barrier support

• Updates the JVM version used during the Spark session. It now uses xgboost4j-spark version 2.0.3 , instead of 0.8.1. This gives us access to XGboost’s most recent Spark code.

• Updates code that used deprecated functions from upstream R dependencies. It also stops using an un-maintained package as a dependency (forge). This eliminated all of the warnings that were happening when fitting a model.

• Major improvements to package testing. Unit tests were updated and expanded, the way sparkxgb automatically starts and stops the Spark session for testing was modernized, and the continuous integration tests were restored. This will ensure the package’s health going forward.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

remotes::install_github("rstudio/sparkxgb")

library(sparkxgb)
library(sparklyr)

sc <- spark_connect(master = "local")
iris_tbl <- copy_to(sc, iris)

xgb_model <- xgboost_classifier(
  iris_tbl,
  Species ~ .,
  num_class = 3,
  num_round = 50,
  max_depth = 4
)

xgb_model %>% 
  ml_predict(iris_tbl) %>% 
  select(Species, predicted_label, starts_with("probability_")) %>% 
  dplyr::glimpse()
#> Rows: ??
#> Columns: 5
#> Database: spark_connection
#> $ Species                <chr> "setosa", "setosa", "setosa", "setosa", "setosa…
#> $ predicted_label        <chr> "setosa", "setosa", "setosa", "setosa", "setosa…
#> $ probability_setosa     <dbl> 0.9971547, 0.9948581, 0.9968392, 0.9968392, 0.9…
#> $ probability_versicolor <dbl> 0.002097376, 0.003301427, 0.002284616, 0.002284…
#> $ probability_virginica  <dbl> 0.0007479066, 0.0018403779, 0.0008762418, 0.000…

sparklyr 1.8.5

The new version of sparklyr does not have user facing improvements. But internally, it has crossed an important milestone. Support for Spark version 2.3 and below has effectively ended. The Scala code needed to do so is no longer part of the package. As per Spark’s versioning policy, found here , Spark 2.3 was ’end-of-life’ in 2018.

This is part of a larger, and ongoing effort to make the immense code-base of sparklyr a little easier to maintain, and hence reduce the risk of failures. As part of the same effort, the number of upstream packages that sparklyr depends on have been reduced. This has been happening across multiple CRAN releases, and in this latest release tibble, and rappdirs are no longer imported by sparklyr.

You can install it from CRAN with:

1

install.packages("bundle")

Let’s walk through what bundle does, and when you might need to use it.

Saving things is hard

We often think of a trained model as a self-contained R object. The model exists in memory in R and if we have some new data, the model object can generate predictions on its own:

In reality, model objects sometimes also make use of references to generate predictions. A reference is a piece of information that a model object refers to that isn’t part of the object itself; this could be something like a connection to a server, a file on disk, or an internal function in the package used to train the model. When we call predict(), model objects know where to look to retrieve that information:

Saving model objects can sometimes disrupt those references. Thus, if we want to train a model, save it, re-load it in a production setting, and generate predictions with it, we may run into issues:

We need some way to preserve access to those references. This new package provides a consistent interface for bundling model objects with their references so that they can be safely saved and re-loaded in production:

When to bundle your model

Let’s walk through building a couple of models using data on cell body segmentation .

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

library(tidymodels)
data(cells, package = "modeldata")

set.seed(123)
cell_split <- cells %>% 
  select(-case) %>%
  initial_split(strata = class)

cell_train <- training(cell_split)
cell_test  <- testing(cell_split)

First, let’s train a logistic regression model:

1

glm_fit <- glm(class ~ ., family = "binomial", data = cell_train)

If we’re satisfied with this model and think it is ready for production, we might want to deploy it somewhere, maybe as a REST API or as a Shiny app. A typical approach would be to:

• save our model object
• start up a new R session
• load the model object into the new session
• predict on new data with the loaded model object

The callr package is helpful for demonstrating this kind of situation; it allows us to start up a fresh R session and pass a few objects in.

We’ll just make use of two of the arguments to the function r():

• func: A function that, given a model object and some new data, will generate predictions, and
• args: A named list, giving the arguments to the above function.

Let’s save our model object to a temporary file and pass it to a fresh R session for prediction, like if we had deployed the model somewhere.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

library(callr)

temp_file <- tempfile()
saveRDS(glm_fit, file = temp_file)

r(
  function(temp_file, new_data) {
    model_object <- readRDS(file = temp_file)
    predict(model_object, new_data)
  },
  args = list(
    temp_file = temp_file,
    new_data = head(cell_test)
  )
)

##          1          2          3          4          5          6 
## -4.8706401 -1.8143956  2.3386470 -1.2735249 -0.3586448  2.7865270

Nice! 😀

What if instead we wanted to train a neural network using tidymodels, with keras as the modeling engine?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

cell_rec <- 
  recipe(class ~ ., data = cell_train) %>%
  step_YeoJohnson(all_numeric_predictors()) %>%
  step_normalize(all_numeric_predictors())

keras_spec <- 
  mlp(penalty = 0, epochs = 10) %>% 
  set_mode("classification") %>% 
  set_engine("keras", verbose = 0) 

keras_fit <- 
  workflow(cell_rec, keras_spec) %>%
  fit(data = cell_train)

Let’s try to save this to disk and then reload it in a new session.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

temp_file <- tempfile()
saveRDS(keras_fit, file = temp_file)

r(
  function(temp_file, new_data) {
    library(workflows)
    model_object <- readRDS(file = temp_file)
    predict(model_object, new_data)
  },
  args = list(
    temp_file = temp_file,
    new_data = head(cell_test)
  )
)

## Error: ! error in callr subprocess
## Caused by error in `do.call(object$predict, args)`:
## ! 'what' must be a function or character string

Oh no! 😱

It turns out that keras models need to be saved in a special way . This is true of a handful of models, like XGBoost, and even some preprocessing steps, like UMAP. These special ways to save objects, like the ones that keras provide, are often referred to as native serialization. Methods for native serialization know which references need to be brought along in order for an object to effectively do its thing in a new environment, but they are different for each model.

The bundle package provides a consistent way to deal with all these kinds of special serialization. The package provides two functions, bundle() and unbundle(), that take care of all of the minutae of preparing to save and load R objects effectively. You bundle() your model before you save it:

1
2
3
4

library(bundle)
temp_file <- tempfile()
keras_bundle <- bundle(keras_fit)
saveRDS(keras_bundle, file = temp_file)

And then you unbundle() after you read the object in a new session:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

r(
  function(temp_file, new_data) {
    library(bundle)
    library(workflows)
    model_bundle <- readRDS(file = temp_file)
    model_object <- unbundle(model_bundle)
    predict(model_object, new_data)
  },
  args = list(
    temp_file = temp_file,
    new_data = head(cell_test)
  )
)

## # A tibble: 6 × 1
##   .pred_class
##   <fct>      
## 1 PS         
## 2 PS         
## 3 WS         
## 4 PS         
## 5 PS         
## 6 WS

Hooray! 🎉

We have support in bundle for a wide variety of models that require (or sometimes require) special handling for serialization, from H2O to torch luz models . Soon bundle will be integrated into vetiver , for better and more robust deployment options. If you use a model that needs special serialization and is not yet supported, let us know in an issue.

Acknowledgements

Thank you so much to everyone who contributed to this first release: @dfalbel , @juliasilge , @qiushiyan , and @simonpcouch . I would especially like to highlight Simon’s contributions, which have been central to bundle getting off the ground!

We are thrilled to announce the release of vetiver , a framework for MLOps tasks in R and Python! The goal of vetiver is to provide fluent tooling to version, share, deploy, and monitor a trained model. If you like perfume or candles, you may recognize this name; vetiver, also known as the “oil of tranquility”, is used as a stabilizing ingredient in perfumery to preserve more volatile fragrances.

You can install the released version of vetiver for R from CRAN :

1

install.packages("vetiver")

You can install the released version of vetiver for Python from PyPI :

1

pip install vetiver

We are sharing more about what vetiver is and how it works over on the RStudio blog so check that out, but we want to share here as well!

Train a model

For this example, let’s work with data on everyone’s favorite dataset on fuel efficiency for cars to predict miles per gallon. In R, we can train a decision tree model to predict miles per gallon using a tidymodels workflow:

1
2
3
4
5

library(tidymodels)

car_mod <-
    workflow(mpg ~ ., decision_tree(mode = "regression")) %>%
    fit(mtcars)

In Python, we can train the same kind of model using scikit-learn :

1
2
3

from vetiver.data import mtcars
from sklearn import tree
car_mod = tree.DecisionTreeRegressor().fit(mtcars, mtcars["mpg"])

For both R and Python, the car_mod object is a fitted model, with parameters estimated using our training data mtcars.

Create a vetiver model

We can create a vetiver_model() in R or VetiverModel() in Python from the trained model; a vetiver model object collects the information needed to store, version, and deploy a trained model.

1
2
3
4
5
6

library(vetiver)
v <- vetiver_model(car_mod, "cars_mpg")
v
#> 
#> ── cars_mpg ─ <butchered_workflow> model for deployment 
#> A rpart regression modeling workflow using 10 features

1
2
3
4
5

from vetiver import VetiverModel
v = VetiverModel(car_mod, model_name = "cars_mpg", 
                 save_ptype = True, ptype_data = mtcars)
v.description
#> "Scikit-learn <class 'sklearn.tree._classes.DecisionTreeRegressor'> model"

See our documentation for how to use these deployable model objects and:

• publish and version your model
• deploy your model as a REST API

Be sure to also read more on the RStudio blog .

Acknowledgements

We’d like to extend our thanks to all of the contributors who helped make these initial releases of vetiver for R and Python possible!

• R package: @cderv , @ggpinto , @isabelizimm , @juliasilge , and @mfansler

• Python package: @has2k1 , and @isabelizimm

With the plumbertableau package (and its corresponding Python package, fastapitableau), you can use functions or models created in R or Python from Tableau through an API. These packages allow you to showcase cutting-edge data science results in your organization’s preferred dashboard tool.

While this post mentions R, anything possible with R and plumbertableau is also doable with Python and fastapitableau.

Foster Data Analytics Capabilities With plumbertableau

With plumbertableau, you can fully develop your model with code-first data science. The package uses plumber to create an API directly from your code. Since your model is fully developed in your data science editor, it can use all the packages and complex calculations it needs.

You can extract the best data science results using R’s capabilities as your model will not be constrained by Tableau’s environment.

Improve Data Quality With APIs for Continuous Use

Seamless integration between analytic platforms prevents issues like using outdated, inaccurate, or incomplete data. Rather than depending on a manual process, data scientists can depend on their data pipelines to ensure data integrity.

With plumbertableau, your tools are integrated through an API. The Tableau dashboard displays results without any intermediate manipulation like copy-and-pasting code or uploading datasets. You can work in confidence knowing your results are synchronized, accurate, and reproducible.

Increase Deliverability by Streamlining Data Pipelines

If your model has many dependencies or versioning requirements, it can be difficult to handle them outside of the development environment. Debugging is even more time-consuming when you need to work in separate environments to figure out what went wrong.

With RStudio Connect, you can publish directly plumbertableau extensions directly from the RStudio IDE. RStudio Connect automatically manages your API’s dependent packages and files to recreate an environment closely mimicking your local development environment. And since all your R code remains in R, you can use your usual data science techniques to efficiently resolve issues.

Read more on the Hosting page of the plumber package.

How to Use plumbertableau: XGBoost with Dynamic Model Output Example

In this walkthrough, we will be using data from the Seattle Open Data Portal to predict the paid parking occupancy percentage in various areas around the city. We will run an XGBoost model in RStudio, create a plumbertableau extension to embed into Tableau, and visualize and interact with the model in a Tableau dashboard. The code is here for reproducibility purposes; however, it will require an RStudio Connect account to complete.

The plumbertableau and fastapi packages have wonderful documentation. Be sure to read them for more information on:

• The anatomy of the extensions
• Details on setting up RStudio Connect and Tableau
• Other examples to try out in your Tableau dashboards

1. Build the model

First, we need to build a model. This walkthrough won’t be covering how to create, tune, or validate a model. If you’d like to learn more on models and machine learning, check out the tidymodels website and Julia Silge’s fantastic screencasts and tutorials.

Load Libraries

library(tidyverse)
library(RSocrata)
library(lubridate)
library(usemodels)
library(tidymodels)

Download and Clean Data

The Seattle Open Data Portal uses Socrata, a data management tool, for its APIs. We can use the RSocrata package to download the data.

parking_data <-
  RSocrata::read.socrata(
    "https://data.seattle.gov/resource/rke9-rsvs.json?$where=sourceelementkey <= 1020"
  )

parking_id <-
  parking_data %>%
  group_by(blockfacename, location.coordinates) %>%
  mutate(id = cur_group_id()) %>%
  ungroup()

parking_clean <-
  parking_id %>%
  mutate(across(c(parkingspacecount, paidoccupancy), as.numeric),
         occupancy_pct = paidoccupancy / parkingspacecount) %>%
  group_by(
    id = id,
    hour = as.numeric(hour(occupancydatetime)),
    month = as.numeric(month(occupancydatetime)),
    dow = as.numeric(wday(occupancydatetime)),
    date = date(occupancydatetime)
  ) %>%
  summarize(occupancy_pct = mean(occupancy_pct, na.rm = TRUE)) %>%
  drop_na() %>%
  ungroup()

We will also need information on the city blocks, so let’s create that dataset.

parking_information <-
  parking_id %>%
  mutate(loc = location.coordinates) %>%
  select(id, blockfacename, loc) %>%
  distinct(id, blockfacename, loc) %>%
  unnest_wider(loc, c('loc1', 'loc2'))

Create Training Data

Now, let’s create the training set from our original data.

parking_split <-
  parking_clean %>%
  arrange(date) %>%
  select(-date) %>%
  initial_time_split(prop = 0.75)

Train and Tune the Model

Here, we train and tune the model. We select the model with the best RSME to use in our dashboard.

xgboost_recipe <-
  recipe(formula = occupancy_pct ~ ., data = parking_clean) %>%
  step_zv(all_predictors())  %>%
  prep()

xgboost_folds <-
  recipes::bake(xgboost_recipe,
                new_data = training(parking_split)) %>%
  rsample::vfold_cv(v = 5)

xgboost_model <-
  boost_tree(
    mode = "regression",
    trees = 1000,
    min_n = tune(),
    tree_depth = tune(),
    learn_rate = tune(),
    loss_reduction = tune()
  ) %>%
  set_engine("xgboost", objective = "reg:squarederror")

xgboost_params <-
  parameters(min_n(),
             tree_depth(),
             learn_rate(),
             loss_reduction())

xgboost_grid <-
  grid_max_entropy(xgboost_params,
                   size = 5)

xgboost_wf <-
  workflows::workflow() %>%
  add_model(xgboost_model) %>%
  add_formula(occupancy_pct ~ .)

xgboost_tuned <- tune::tune_grid(
  object = xgboost_wf,
  resamples = xgboost_folds,
  grid = xgboost_grid,
  metrics = yardstick::metric_set(rmse, rsq, mae),
  control = tune::control_grid(verbose = TRUE)
)

xgboost_best <-
  xgboost_tuned %>%
  tune::select_best("rmse")

xgboost_final <-
  xgboost_model %>%
  finalize_model(xgboost_best)

We bundle the recipe and fitted model in an object so we can use it later.

train_processed <-
  bake(xgboost_recipe, new_data = training(parking_split))

prediction_fit <-
  xgboost_final %>%
  fit(formula = occupancy_pct ~ .,
      data    = train_processed)

model_details <- list(model = xgboost_final,
                      recipe = xgboost_recipe,
                      prediction_fit = prediction_fit)

Save Objects for the plumbertableau Extension

We’ll want to save our data and our model so that we can use them in the extension. If you have an RStudio Connect account, the pins package is a great choice for saving these objects.

rsc <-
  pins::board_rsconnect(server = Sys.getenv("CONNECT_SERVER"),
                        key = Sys.getenv("CONNECT_API_KEY"))

pins::pin_write(
  board = rsc,
  x = model_details,
  name = "seattle_parking_model",
  description = "Seattle Occupancy Percentage XGBoost Model",
  type = "rds"
)

pins::pin_write(
  board = rsc,
  x = parking_information,
  name = "seattle_parking_info",
  description = "Seattle Parking Information",
  type = "rds"
)

2. Create a plumbertableau Extension

Next, we will use our model to create a plumbertableau extension. As noted previously, the plumbertableau extension is a Plumber API with some special annotations.

Create an R script called plumber.R. At the top, we list the libraries we’ll need.

library(plumber)
library(pins)
library(tibble)
library(xgboost)
library(lubridate)
library(dplyr)
library(tidyr)
library(tidymodels)
library(plumbertableau)

We want to bring in our model details and our data. If you pinned your data, you’ll change the name of the pin below.

rsc <-
  pins::board_rsconnect(
    server = Sys.getenv("CONNECT_SERVER"),
    key = Sys.getenv("CONNECT_API_KEY")
  )

xgboost_model <-
  pins::pin_read("isabella.velasquez/seattle_parking_model", board = rsc)

Now, we add our annotations. Note that we use plumbertableau annotations, which are slightly different than the ones from plumber.

• We use tableauArg rather than params.
• We specify what is returned to Tableau with tableauReturn.
• We must use post for what is being returned.

#* @apiTitle Seattle Parking Occupancy Percentage Prediction API
#* @apiDescription Return the predicted occupancy percentage at various Seattle locations

#* @tableauArg block_id:integer numeric block ID
#* @tableauArg ndays:integer number of days in the future for the prediction

#* @tableauReturn [numeric] Predicted occupancy rate
#* @post /pred

Now, we create our function with the arguments station_id and ndays. These will have corresponding arguments in Tableau. The function will output our predicted occupancy percentage, which will be what we visualize and interact with in the dashboard.

This function takes the city block and number of days in the future to give us the predicted occupancy percentage at that time.

function(block_id, ndays) {
  times <- Sys.time() + lubridate::ddays(ndays)
  
  current_time <-
    tibble::tibble(times = times,
                   id = block_id)
  
  current_prediction  <-
    current_time %>%
    transmute(
      id = id,
      hour = hour(times),
      month = month(times),
      dow = wday(times),
      occupancy_pct = NA
    ) %>%
    bake(xgboost_model$recipe, .)
  
  parking_prediction <-
    xgboost_model$prediction_fit %>%
    predict(new_data = current_prediction)
  
  predictions <-
    parking_prediction$.pred
  
  predictions[[1]]
  
}

Finally, we finish off our script with the extension footer needed for plumbertableau extensions.

#* @plumber
tableau_extension

Here is the full plumber.R script:

library(plumber)
library(pins)
library(tibble)
library(xgboost)
library(lubridate)
library(dplyr)
library(tidyr)
library(tidymodels)
library(plumbertableau)

rsc <-
  pins::board_rsconnect(server = Sys.getenv("CONNECT_SERVER"),
                        key = Sys.getenv("CONNECT_API_KEY"))

xgboost_model <-
  pins::pin_read("isabella.velasquez/seattle_parking_model", board = rsc)

#* @apiTitle Seattle Parking Occupancy Percentage Prediction API
#* @apiDescription Return the predicted occupancy percentage at various Seattle locations

#* @tableauArg block_id:integer numeric block ID
#* @tableauArg ndays:integer number of days in the future for the prediction

#* @tableauReturn [numeric] Predicted occupancy rate
#* @post /pred

function(block_id, ndays) {
  times <- Sys.time() + lubridate::ddays(ndays)
  
  current_time <-
    tibble::tibble(times = times,
                   id = block_id)
  
  current_prediction  <-
    current_time %>%
    transmute(
      id = id,
      hour = hour(times),
      month = month(times),
      dow = wday(times),
      occupancy_pct = NA
    ) %>%
    bake(xgboost_model$recipe, .)
  
  parking_prediction <-
    xgboost_model$prediction_fit %>%
    predict(new_data = current_prediction)
  
  predictions <-
    parking_prediction$.pred
  
  predictions[[1]]
  
}

#* @plumber
tableau_extension

3. Host your API

We have to host our API so that it can be accessed in Tableau. In our case, we publish it to RStudio Connect.

Once hosted, plumbertableau automatically generates a documentation page. Notice that the SCRIPT_* value is not R code. This is a Tableau command that we will use to connect our extension and Tableau.

4. Create a calculated field in Tableau

There are a few steps you need to take so that Tableau can use your plumbertableau extension. If you are using RStudio Connect, read the documentation on how to configure RStudio Connect as an analytic extension.

Create a new workbook and upload the station_information file. Under Analysis, turn off Aggregate Measures. Drop Lat into Rows and Lon into Columns, which will create a map. Save the workbook.

Make sure your workbook knows to connect to RStudio Connect by going to Analysis > Manage Analytic Extensions Connection > Choose a Connection. Then, select your Connect account.

Drag Id into the “Detail” mark. Create a parameter called “Days in the Future”. We’re using our model to predict parking occupancy percentage for that date. Show the parameter on the worksheet.

Create a calculated field using the SCRIPT from the plumbertableau documentation page:

SCRIPT_REAL("/plumbertableau-xgboost-example/pred", block_id, ndays) 

For each tableauArg we have listed in the extension, we will replace it with its corresponding Tableau value. If you’re following along, this means block_id will become ATTR([Id]) and ndays will become ATTR([Days in the Future]).

SCRIPT_REAL("/plumbertableau-xgboost-example/pred", ATTR([Id]), ATTR([Days in the Future]))

5. Run model and visualize results in Tableau

That’s it! Once you embed your extension in Tableau’s calculated fields, you can use your model’s results in your Tableau dashboard like any other measure or dimension.

We can change the ndays argument to get new predictions from our XGBoost model and display them on our Tableau dashboard.

You can style your Tableau dashboard and then provide your users something that is not only aesthetically pleasing, but is dynamically calculating predictions based on a model you have created in R.

Conclusion

With plumbertableau, you can showcase sophisticated model results that are easy to integrate, debug, and reproduce. Your work will be at the forefront of data science while being visualized in Tableau’s easy, point-and-click interface.

Learn More

Watch James Blair showcase plumbertableau in Leveraging R & Python in Tableau with RStudio Connect:

More on how RStudio supports interoperability across tools can be found on our BI and Data Science Overview Page.

Teams often need access to key data to do their work, but have you ever opened your coworker’s script to see:

dat <-  
 read_csv("C://Users/someone_else/data/dataset.csv")

more_dat <- 
 read_csv("S://Path_to_mapped_drive_that_you_dont_have/dataset.csv")

Yikes! How will you get these files? Let’s hope you can reach your coworker before they’ve logged off for the day.

How can your code be reproducible if you have to manually change the file paths? Shudder.

What if you need to make edits to the data, will you have to keep copying CSVs and emailing files forever? Double shudder.

What if your coworker accidentally forwards your email to someone who is not supposed to have access? Oh no.

We can struggle to share data assets easily and safely, relying on emailed files to keep our analyses up to date. This makes it difficult to keep current or know what version of the data we’re using. If you’ve ever experienced any of the scenarios above, consider pins as a solution that can help you share your data assets.

What is a pin, anyway?

Pins, from the R package of the same name, are a versatile way to publish R objects on a virtual corkboard so you can share them across projects and people.

Good pins are data or assets that are a few hundred megabytes or smaller. You can pin just about any object: data, models, JSON files, feather files from the Arrow package, and more. One of the most frequent use cases is pinning small data sets — often ephemeral data or reference tables that don’t quite merit being in a database, but seemingly don’t have a good home elsewhere (until now).

Pins get published to a board, which can be an RStudio Connect server, an AWS S3 bucket or Azure Blob Storage, a shared drive like Dropbox or Sharepoint, or a variety of other options. Try it out for yourself — read in this data set we’ve pinned for you on RStudio Connect!

# Install the latest pins from CRAN
install.packages("pins")

library(pins)

# Identify the board
board <-
  board_url(c("penguins" = "https://colorado.rstudio.com/rsc/example_pin/"))

# Read the shared data
board %>%
  pin_read("penguins")

In short, if you’ve ever wondered where to put an R object that you or your colleague will need to use again, you might just want to pin it.

Pins for Sharing Across Projects and Teams

One of the greatest strengths of pins is how your pin becomes accessible directly from your R scripts and the R scripts of anyone else to whom you’ve given access. Different projects can include code that reads the same pin without creating more copies of the data:

It’s easier (and safer) to share a pin across multiple projects or people than to email files around. Pins respect the access controls of the board. Say you’ve pinned to RStudio Connect: you can control who gets to use the pin, just like any other piece of content.

Pins for Updating and Versioning

You may be wondering why use pins if you already have a shared drive with your teammates. But what happens if you need to replace the dataset with a new one? Do you email everybody to let them know? Is it dataFINALv2.csv? Or dataFINALfinal.csv?

The pins package retrieves the newest version of the pin by default. That means pin users never have to worry about getting a stale version of the pin. If you need to update your pin regularly, a scheduled R Markdown on RStudio Connect can handle this task for you, so your pin stays fresh.

But you’re not locked into losing old versions of a pin. You can version pins so that writing to an existing pin adds a new copy rather than replacing the existing data.

Here’s what versioning looks like using a temporary board:

library(pins)


board2 <- board_temp(versioned = TRUE)

board2 %>% pin_write(1:5, name = "x", type = "rds")
#> Creating new version '20210304T050607Z-ab444'
#> Writing to pin 'x'

board2 %>% pin_write(2:6, name = "x", type = "rds")
#> Creating new version '20210304T050607Z-a077a'
#> Writing to pin 'x'

board2 %>% pin_write(3:7, name = "x", type = "rds")
#> Creating new version '20210304T050607Z-0a284'
#> Writing to pin 'x'

# see all versions
board2 %>% pin_versions("x")
#> # A tibble: 3 × 3
#>   version                created             hash 
#>   <chr>                  <dttm>              <chr>
#> 1 20210304T050607Z-0a284 2021-03-04 05:06:00 0a284
#> 2 20210304T050607Z-a077a 2021-03-04 05:06:00 a077a
#> 3 20210304T050607Z-ab444 2021-03-04 05:06:00 ab444

Learn More

With pins, you and your teammates can know where your important data assets are, how to access them, and whether they are the correct version. You can work with confidence knowing you’re using the right asset, your work is reproducible, and you’re following good practices for data management.

There’s more to explore with pins. We’re excited to share how you can adopt them into your workflow.

Learn more about how and when to use pins:

• The pins package documentation
• RStudio Pro Tips: Creating Efficient Workflows with pins and RStudio Connect

See pins in action:

• Pins can pull intensive ETL processes out of your apps, improve performance, and save you the hassle of redeploying whenever the underlying data changes.
  • Watch: Deploying End-To-End Data Science with Shiny, Plumber, and Pins
• Pins can play a key role in MLOps, publishing versioned models, and monitoring model metrics.
  • Read: Model Monitoring with R Markdown, pins, and RStudio Connect

I’m delighted to announce that pins 1.0.0 is now available on CRAN. The pins package publishes data, models, and other R objects, making it easy to share them across projects and with your colleagues. You can pin objects to a variety of pin boards, including folders (to share on a networked drive or with services like DropBox), RStudio Connect, Amazon S3, and Azure blob storage. Pins can be versioned, making it straightforward to track changes, re-run analyses on historical data, and undo mistakes. Our users have found numerous ways to use this ability to fluently share and version data and other objects, such as automating ETL for a Shiny app.

You can install pins with:

install.packages("pins")

pins 1.0.0 includes a major overhaul of the API. The legacy API (pin(), pin_get(), board_register(), and friends) will continue to work, but new features will only be implemented with the new API, so we encourage you to switch to the modern API as quickly as possible. If you’re an existing pins user, you can learn more about the changes and how to update you code in vignette("pins-update").

library(pins)

board <- board_temp()
board
#> Pin board <pins_board_folder>
#> Path: '/tmp/RtmpLu2Bkx/pins-114af466104ab'
#> Cache size: 0

board %>% pin_write(head(mtcars), "mtcars")
#> Guessing `type = 'rds'`
#> Creating new version '20211004T155644Z-f8797'
#> Writing to pin 'mtcars'

board %>% pin_read("mtcars")
#>                    mpg cyl disp  hp drat    wt  qsec vs am gear carb
#> Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
#> Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
#> Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
#> Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
#> Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
#> Valiant           18.1   6  225 105 2.76 3.460 20.22  1  0    3    1

board <- board_rsconnect()
#> Connecting to RSC 1.9.0.1 at <https://connect.rstudioservices.com>
board %>% pin_write(tidy_sales_data, "sales-summary", type = "rds")
#> Writing to pin 'hadley/sales-summary'

board <- board_rsconnect()
board %>% pin_read("hadley/sales-summary")

To install sparklyr.sedona from GitHub using the remotes package ¹, run

1

remotes::install_github(repo = "apache/incubator-sedona", subdir = "R/sparklyr.sedona")

In this blog post, we will provide a quick introduction to sparklyr.sedona, outlining the motivation behind this sparklyr extension, and presenting some example sparklyr.sedona use cases involving Spark spatial RDDs, Spark dataframes, and visualizations.

Motivation for sparklyr.sedona

A suggestion from the mlverse survey results earlier this year mentioned the need for up-to-date R interfaces for Spark-based GIS frameworks. While looking into this suggestion, we learned about Apache Sedona , a geospatial data system powered by Spark that is modern, efficient, and easy to use. We also realized that while our friends from the Spark open-source community had developed a sparklyr extension for GeoSpark, the predecessor of Apache Sedona, there was no similar extension making more recent Sedona functionalities easily accessible from R yet. We therefore decided to work on sparklyr.sedona, which aims to bridge the gap between Sedona and R.

The lay of the land²

We hope you are ready for a quick tour through some of the RDD-based and Spark-dataframe-based functionalities in sparklyr.sedona, and also, some bedazzling visualizations derived from geospatial data in Spark.

In Apache Sedona, Spatial Resilient Distributed Datasets (SRDDs) are basic building blocks of distributed spatial data encapsulating “vanilla” RDD s of geometrical objects and indexes. SRDDs support low-level operations such as Coordinate Reference System (CRS) transformations, spatial partitioning, and spatial indexing. For example, with sparklyr.sedona, SRDD-based operations we can perform include the following:

• Importing some external data source into a SRDD:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

library(sparklyr)
library(sparklyr.sedona)

sedona_git_repo <- normalizePath("~/incubator-sedona")
data_dir <- file.path(sedona_git_repo, "core", "src", "test", "resources")

sc <- spark_connect(master = "local")

pt_rdd <- sedona_read_dsv_to_typed_rdd(
  sc,
  location = file.path(data_dir, "arealm.csv"),
  type = "point"
)

• Applying spatial partitioning to all data points:

1

sedona_apply_spatial_partitioner(pt_rdd, partitioner = "kdbtree")

• Building spatial index on each partition:

1

sedona_build_index(pt_rdd, type = "quadtree")

• Joining one spatial data set with another using “contain” or “overlap” as the join predicate:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

polygon_rdd <- sedona_read_dsv_to_typed_rdd(
  sc,
  location = file.path(data_dir, "primaryroads-polygon.csv"),
  type = "polygon"
)

pts_per_region_rdd <- sedona_spatial_join_count_by_key(
  pt_rdd,
  polygon_rdd,
  join_type = "contain",
  partitioner = "kdbtree"
)

It is worth mentioning that sedona_spatial_join() will perform spatial partitioning and indexing on the inputs using the partitioner and index_type only if the inputs are not partitioned or indexed as specified already.

From the examples above, one can see that SRDDs are great for spatial operations requiring fine-grained control, e.g., for ensuring a spatial join query is executed as efficiently as possible with the right types of spatial partitioning and indexing.

Finally, we can try visualizing the join result above, using a choropleth map:

1
2
3
4
5
6
7
8

sedona_render_choropleth_map(
  pts_per_region_rdd,
  resolution_x = 1000,
  resolution_y = 600,
  output_location = tempfile("choropleth-map-"),
  boundary = c(-126.790180, -64.630926, 24.863836, 50.000),
  base_color = c(63, 127, 255)
)

which gives us the following:

Wait, but something seems amiss. To make the visualization above look nicer, we can overlay it with the contour of each polygonal region:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

contours <- sedona_render_scatter_plot(
  polygon_rdd,
  resolution_x = 1000,
  resolution_y = 600,
  output_location = tempfile("scatter-plot-"),
  boundary = c(-126.790180, -64.630926, 24.863836, 50.000),
  base_color = c(255, 0, 0),
  browse = FALSE
)

sedona_render_choropleth_map(
  pts_per_region_rdd,
  resolution_x = 1000,
  resolution_y = 600,
  output_location = tempfile("choropleth-map-"),
  boundary = c(-126.790180, -64.630926, 24.863836, 50.000),
  base_color = c(63, 127, 255),
  overlay = contours
)

which gives us the following:

With some low-level spatial operations taken care of using the SRDD API and the right spatial partitioning and indexing data structures, we can then import the results from SRDDs to Spark dataframes. When working with spatial objects within Spark dataframes, we can write high-level, declarative queries on these objects using dplyr verbs in conjunction with Sedona spatial UDFs , e.g. ³ , the following query tells us whether each of the 8 nearest polygons to the query point contains that point, and also, the convex hull of each polygon.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

tbl <- DBI::dbGetQuery(
  sc, "SELECT ST_GeomFromText(\"POINT(-66.3 18)\") AS `pt`"
)
pt <- tbl$pt[[1]]
knn_rdd <- sedona_knn_query(
  polygon_rdd, x = pt, k = 8, index_type = "rtree"
)

knn_sdf <- knn_rdd %>%
  sdf_register() %>%
  dplyr::mutate(
    contains_pt = ST_contains(geometry, ST_Point(-66.3, 18)),
    convex_hull = ST_ConvexHull(geometry)
  )

knn_sdf %>% print()

# Source: spark<?> [?? x 3]
  geometry                         contains_pt convex_hull
  <list>                           <lgl>       <list>
1 <POLYGON ((-66.335674 17.986328… TRUE        <POLYGON ((-66.335674 17.986328,…
2 <POLYGON ((-66.335432 17.986626… TRUE        <POLYGON ((-66.335432 17.986626,…
3 <POLYGON ((-66.335432 17.986626… TRUE        <POLYGON ((-66.335432 17.986626,…
4 <POLYGON ((-66.335674 17.986328… TRUE        <POLYGON ((-66.335674 17.986328,…
5 <POLYGON ((-66.242489 17.988637… FALSE       <POLYGON ((-66.242489 17.988637,…
6 <POLYGON ((-66.242489 17.988637… FALSE       <POLYGON ((-66.242489 17.988637,…
7 <POLYGON ((-66.24221 17.988799,… FALSE       <POLYGON ((-66.24221 17.988799, …
8 <POLYGON ((-66.24221 17.988799,… FALSE       <POLYGON ((-66.24221 17.988799, …

Acknowledgements

The author of this blog post would like to thank Jia Yu , the creator of Apache Sedona, and Lorenz Walthert for their suggestion to contribute sparklyr.sedona to the upstream incubator-sedona repository. Jia has provided extensive code-review feedback to ensure sparklyr.sedona complies with coding standards and best practices of the Apache Sedona project, and has also been very helpful in the instrumentation of CI workflows verifying sparklyr.sedona works as expected with snapshot versions of Sedona libraries from development branches.

The author is also grateful for his colleague Sigrid Keydana for valuable editorial suggestions on this blog post.

That’s all. Thank you for reading!

Photo by NASA on Unsplash

To install sparklyr 1.7 from CRAN, run

1

install.packages("sparklyr")

In this blog post, we wish to present the following highlights from the sparklyr 1.7 release:

• Image and binary data sources
• New spark_apply() capabilities
• Better integration with sparklyr extensions
• Other exciting news

Image and binary data sources

As a unified analytics engine for large-scale data processing, Apache Spark is well-known for its ability to tackle challenges associated with the volume, velocity, and last but not least, the variety of big data. Therefore it is hardly surprising to see that – in response to recent advances in deep learning frameworks – Apache Spark has introduced built-in support for image data sources and binary data sources (in releases 2.4 and 3.0, respectively). The corresponding R interfaces for both data sources, namely, spark_read_image() and spark_read_binary() , were shipped recently as part of sparklyr 1.7.

The usefulness of data source functionalities such as spark_read_image() is perhaps best illustrated by a quick demo below, where spark_read_image(), through the standard Apache Spark ImageSchema , helps connecting raw image inputs to a sophisticated feature extractor and a classifier, forming a powerful Spark application for image classifications.

The demo

In this demo, we shall construct a scalable Spark ML pipeline capable of classifying images of cats and dogs accurately and efficiently, using spark_read_image() and a pre-trained convolutional neural network code-named Inception (Szegedy et al. (2015)).

The first step to building such a demo with maximum portability and repeatability is to create a sparklyr extension that accomplishes the following:

• Specifying the required MVN dependencies of this demo (namely, the Spark Deep Learning library (Databricks, Inc. (2019)), which contains an Inception-V3-based image feature extractor accessible through the Spark ML Transformer interface )
• Bundling with itself two randomly selected ¹ and disjoint subsets of the dogs-vs-cats dataset (Elson et al. (2007)) as train and test data, which are stored in the extdata/{train,test} sub directories of the package)

A reference implementation of such a sparklyr extension can be found in here .

The second step, of course, is to make use of the above-mentioned sparklyr extension to perform some feature engineering. We will see very high-level features being extracted intelligently from each cat/dog image based on what the pre-built Inception-V3 convolutional neural network has already learned from classifying a much broader collection of images:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

library(sparklyr)
library(sparklyr.deeperer)

# NOTE: the correct spark_home path to use depends on the configuration of the
# Spark cluster you are working with.
spark_home <- "/usr/lib/spark"
sc <- spark_connect(master = "yarn", spark_home = spark_home)

data_dir <- copy_images_to_hdfs()

# extract features from train- and test-data
image_data <- list()
for (x in c("train", "test")) {
  # import
  image_data[[x]] <- c("dogs", "cats") %>%
    lapply(
      function(label) {
        numeric_label <- ifelse(identical(label, "dogs"), 1L, 0L)
        spark_read_image(
          sc, dir = file.path(data_dir, x, label, fsep = "/")
        ) %>%
          dplyr::mutate(label = numeric_label)
      }
    ) %>%
      do.call(sdf_bind_rows, .)

  dl_featurizer <- invoke_new(
    sc,
    "com.databricks.sparkdl.DeepImageFeaturizer",
    random_string("dl_featurizer") # uid
  ) %>%
    invoke("setModelName", "InceptionV3") %>%
    invoke("setInputCol", "image") %>%
    invoke("setOutputCol", "features")
  image_data[[x]] <-
    dl_featurizer %>%
    invoke("transform", spark_dataframe(image_data[[x]])) %>%
    sdf_register()
}

Third step: equipped with features that summarize the content of each image well, we can build a Spark ML pipeline that recognizes cats and dogs using only logistic regression ²

1
2
3
4
5
6
7
8
9

label_col <- "label"
prediction_col <- "prediction"
pipeline <- ml_pipeline(sc) %>%
  ml_logistic_regression(
    features_col = "features",
    label_col = label_col,
    prediction_col = prediction_col
  )
model <- pipeline %>% ml_fit(image_data$train)

Finally, we can evaluate the accuracy of this model on the test images:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

predictions <- model %>%
  ml_transform(image_data$test) %>%
  dplyr::compute()

cat("Predictions vs. labels:\n")
predictions %>%
  dplyr::select(!!label_col, !!prediction_col) %>%
  print(n = sdf_nrow(predictions))

cat("\nAccuracy of predictions:\n")
predictions %>%
  ml_multiclass_classification_evaluator(
    label_col = label_col,
    prediction_col = prediction_col,
    metric_name = "accuracy"
  ) %>%
    print()

## Predictions vs. labels:
## # Source: spark<?> [?? x 2]
##    label prediction
##    <int>      <dbl>
##  1     1          1
##  2     1          1
##  3     1          1
##  4     1          1
##  5     1          1
##  6     1          1
##  7     1          1
##  8     1          1
##  9     1          1
## 10     1          1
## 11     0          0
## 12     0          0
## 13     0          0
## 14     0          0
## 15     0          0
## 16     0          0
## 17     0          0
## 18     0          0
## 19     0          0
## 20     0          0
##
## Accuracy of predictions:
## [1] 1

New spark_apply() capabilities

Optimizations & custom serializers

Many sparklyr users who have tried to run spark_apply() or doSpark to parallelize R computations among Spark workers have probably encountered some challenges arising from the serialization of R closures. In some scenarios, the serialized size of the R closure can become too large, often due to the size of the enclosing R environment required by the closure. In other scenarios, the serialization itself may take too much time, partially offsetting the performance gain from parallelization. Recently, multiple optimizations went into sparklyr to address those challenges. One of the optimizations was to make good use of the broadcast variable construct in Apache Spark to reduce the overhead of distributing shared and immutable task states across all Spark workers. In sparklyr 1.7, there is also support for custom spark_apply() serializers, which offers more fine-grained control over the trade-off between speed and compression level of serialization algorithms. For example, one can specify

1

options(sparklyr.spark_apply.serializer = "qs")

,

which will apply the default options of qs::qserialize() to achieve a high compression level, or

1
2

options(sparklyr.spark_apply.serializer = function(x) qs::qserialize(x, preset = "fast"))
options(sparklyr.spark_apply.deserializer = function(x) qs::qdeserialize(x))

,

which will aim for faster serialization speed with less compression.

Inferring dependencies automatically

In sparklyr 1.7, spark_apply() also provides the experimental auto_deps = TRUE option. With auto_deps enabled, spark_apply() will examine the R closure being applied, infer the list of required R packages, and only copy the required R packages and their transitive dependencies to Spark workers. In many scenarios, the auto_deps = TRUE option will be a significantly better alternative compared to the default packages = TRUE behavior, which is to ship everything within .libPaths() to Spark worker nodes, or the advanced packages = <package config> option, which requires users to supply the list of required R packages or manually create a spark_apply() bundle.

Better integration with sparklyr extensions

Substantial effort went into sparklyr 1.7 to make lives easier for sparklyr extension authors. Experience suggests two areas where any sparklyr extension can go through a frictional and non-straightforward path integrating with sparklyr are the following:

• The dbplyr SQL translation environment
• Invocation of Java/Scala functions from R

We will elaborate on recent progress in both areas in the sub-sections below.

Customizing the dbplyr SQL translation environment

sparklyr extensions can now customize sparklyr’s dbplyr SQL translations through the spark_dependency() specification returned from spark_dependencies() callbacks. This type of flexibility becomes useful, for instance, in scenarios where a sparklyr extension needs to insert type casts for inputs to custom Spark UDFs. We can find a concrete example of this in sparklyr.sedona , a sparklyr extension to facilitate geo-spatial analyses using Apache Sedona . Geo-spatial UDFs supported by Apache Sedona such as ST_Point() and ST_PolygonFromEnvelope() require all inputs to be DECIMAL(24, 20) quantities rather than DOUBLEs. Without any customization to sparklyr’s dbplyr SQL variant, the only way for a dplyr query involving ST_Point() to actually work in sparklyr would be to explicitly implement any type cast needed by the query using dplyr::sql(), e.g.,

1
2
3
4
5
6

my_geospatial_sdf <- my_geospatial_sdf %>%
  dplyr::mutate(
    x = dplyr::sql("CAST(`x` AS DECIMAL(24, 20))"),
    y = dplyr::sql("CAST(`y` AS DECIMAL(24, 20))")
  ) %>%
  dplyr::mutate(pt = ST_Point(x, y))

.

This would, to some extent, be antithetical to dplyr’s goal of freeing R users from laboriously spelling out SQL queries. Whereas by customizing sparklyr’s dplyr SQL translations (as implemented in here and here ), sparklyr.sedona allows users to simply write

1

my_geospatial_sdf <- my_geospatial_sdf %>% dplyr::mutate(pt = ST_Point(x, y))

instead, and the required Spark SQL type casts are generated automatically.

Improved interface for invoking Java/Scala functions

In sparklyr 1.7, the R interface for Java/Scala invocations saw a number of improvements.

With previous versions of sparklyr, many sparklyr extension authors would run into trouble when attempting to invoke Java/Scala functions accepting an Array[T] as one of their parameters, where T is any type bound more specific than java.lang.Object / AnyRef. This was because any array of objects passed through sparklyr’s Java/Scala invocation interface will be interpreted as simply an array of java.lang.Objects in absence of additional type information. For this reason, a helper function jarray() was implemented as part of sparklyr 1.7 as a way to overcome the aforementioned problem. For example, executing

1
2
3
4
5
6
7

sc <- spark_connect(...)

arr <- jarray(
  sc,
  seq(5) %>% lapply(function(x) invoke_new(sc, "MyClass", x)),
  element_type = "MyClass"
)

will assign to arr a reference to an Array[MyClass] of length 5, rather than an Array[AnyRef]. Subsequently, arr becomes suitable to be passed as a parameter to functions accepting only Array[MyClass]s as inputs. Previously, some possible workarounds of this sparklyr limitation included changing function signatures to accept Array[AnyRef]s instead of Array[MyClass]s, or implementing a “wrapped” version of each function accepting Array[AnyRef] inputs and converting them to Array[MyClass] before the actual invocation. None of such workarounds was an ideal solution to the problem.

Another similar hurdle that was addressed in sparklyr 1.7 as well involves function parameters that must be single-precision floating point numbers or arrays of single-precision floating point numbers. For those scenarios, jfloat() and jfloat_array() are the helper functions that allow numeric quantities in R to be passed to sparklyr’s Java/Scala invocation interface as parameters with desired types.

In addition, while previous verisons of sparklyr failed to serialize parameters with NaN values correctly, sparklyr 1.7 preserves NaNs as expected in its Java/Scala invocation interface.

Other exciting news

There are numerous other new features, enhancements, and bug fixes made to sparklyr 1.7, all listed in the NEWS.md file of the sparklyr repo and documented in sparklyr’s HTML reference pages. In the interest of brevity, we will not describe all of them in great detail within this blog post.

Acknowledgement

In chronological order, we would like to thank the following individuals who have authored or co-authored pull requests that were part of the sparklyr 1.7 release:

• @yitao-li
• @mzorko
• @jozefhajnala
• @lresende

We’re also extremely grateful to everyone who has submitted feature requests or bug reports, many of which have been tremendously helpful in shaping sparklyr into what it is today.

Furthermore, the author of this blog post is indebted to @skeydan for her awesome editorial suggestions. Without her insights about good writing and story-telling, expositions like this one would have been less readable.

If you wish to learn more about sparklyr, we recommend visiting sparklyr.ai , spark.rstudio.com , and also reading some previous sparklyr release posts such as sparklyr 1.6 and sparklyr 1.5 .

That is all. Thanks for reading!

Databricks, Inc. 2019. Deep Learning Pipelines for Apache Spark. V. 1.5.0. Released January 25. https://spark-packages.org/package/databricks/spark-deep-learning .

Elson, Jeremy, John (JD) Douceur, Jon Howell, and Jared Saul. 2007. “Asirra: A CAPTCHA That Exploits Interest-Aligned Manual Image Categorization.” Proceedings of 14th ACM Conference on Computer and Communications Security (CCS), Proceedings of 14th ACM Conference on Computer and Communications Security (CCS) Editions. https://www.microsoft.com/en-us/research/publication/asirra-a-captcha-that-exploits-interest-aligned-manual-image-categorization/ .

Szegedy, Christian, Wei Liu, Yangqing Jia, et al. 2015. “Going Deeper with Convolutions.” Computer Vision and Pattern Recognition (CVPR). http://arxiv.org/abs/1409.4842 .

To install sparklyr 1.6 from CRAN, run

1

install.packages("sparklyr")

In this blog post, we shall highlight the following features and enhancements from sparklyr 1.6:

• Weighted quantile summaries
• Power iteration clustering
• spark_write_rds() + collect_from_rds()
• Dplyr-related improvements

Weighted quantile summaries

Apache Spark is well-known for supporting approximate algorithms that trade off marginal amounts of accuracy for greater speed and parallelism. Such algorithms are particularly beneficial for performing preliminary data explorations at scale, as they enable users to quickly query certain estimated statistics within a predefined error margin, while avoiding the high cost of exact computations. One example is the Greenwald-Khanna algorithm for on-line computation of quantile summaries, as described in Greenwald and Khanna (2001). This algorithm was originally designed for efficient $\epsilon$- approximation of quantiles within a large dataset without the notion of data points carrying different weights, and the unweighted version of it has been implemented as approxQuantile() since Spark 2.0. However, the same algorithm can be generalized to handle weighted inputs, and as sparklyr user @Zhuk66 mentioned in this issue , a weighted version of this algorithm makes for a useful sparklyr feature.

To properly explain what weighted-quantile means, we must clarify what the weight of each data point signifies. For example, if we have a sequence of observations $(1, 1, 1, 1, 0, 2, -1, -1)$, and would like to approximate the median of all data points, then we have the following two options:

• Either run the unweighted version of approxQuantile() in Spark to scan through all 8 data points

• Or alternatively, “compress” the data into 4 tuples of (value, weight): $(1, 0.5), (0, 0.125), (2, 0.125), (-1, 0.25)$, where the second component of each tuple represents how often a value occurs relative to the rest of the observed values, and then find the median by scanning through the 4 tuples using the weighted version of the Greenwald-Khanna algorithm

We can also run through a contrived example involving the standard normal distribution to illustrate the power of weighted quantile estimation in sparklyr 1.6. Suppose we cannot simply run qnorm() in R to evaluate the quantile function of the standard normal distribution at $p = 0.25$ and $p = 0.75$, how can we get some vague idea about the 1st and 3rd quantiles of this distribution? One way is to sample a large number of data points from this distribution, and then apply the Greenwald-Khanna algorithm to our unweighted samples, as shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

library(sparklyr)

sc <- spark_connect(master = "local")

num_samples <- 1e6
samples <- data.frame(x = rnorm(num_samples))

samples_sdf <- copy_to(sc, samples, name = random_string())

samples_sdf %>%
  sdf_quantile(
    column = "x",
    probabilities = c(0.25, 0.75),
    relative.error = 0.01
  ) %>%
  print()

##        25%        75%
## -0.6629242  0.6874939

Notice that because we are working with an approximate algorithm, and have specified relative.error = 0.01, the estimated value of $-0.6629242$ from above could be anywhere between the 24th and the 26th percentile of all samples. In fact, it falls in the $25.36896$-th percentile:

1

pnorm(-0.6629242)

## [1] 0.2536896

Now how can we make use of weighted quantile estimation from sparklyr 1.6 to obtain similar results? Simple! We can sample a large number of $x$ values uniformly randomly from $(-\infty, \infty)$ (or alternatively, just select a large number of values evenly spaced between $(-M, M)$ where $M$ is approximately $\infty$), and assign each $x$ value a weight of $\displaystyle \frac{1}{\sqrt{2 \pi}}e^{-\frac{x^2}{2}}$, the standard normal distribution’s probability density at $x$. Finally, we run the weighted version of sdf_quantile() from sparklyr 1.6, as shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

library(sparklyr)

sc <- spark_connect(master = "local")

num_samples <- 1e6
M <- 1000
samples <- tibble::tibble(
  x = M * seq(-num_samples / 2 + 1, num_samples / 2) / num_samples,
  weight = dnorm(x)
)

samples_sdf <- copy_to(sc, samples, name = random_string())

samples_sdf %>%
  sdf_quantile(
    column = "x",
    weight.column = "weight",
    probabilities = c(0.25, 0.75),
    relative.error = 0.01
  ) %>%
  print()

##    25%    75%
## -0.696  0.662

Voilà! The estimates are not too far off from the 25th and 75th percentiles (in relation to our abovementioned maximum permissible error of $0.01$):

1

pnorm(-0.696)

## [1] 0.2432144

1

pnorm(0.662)

## [1] 0.7460144

Power iteration clustering

Power iteration clustering (PIC), a simple and scalable graph clustering method presented in Lin and Cohen (2010), first finds a low-dimensional embedding of a dataset, using truncated power iteration on a normalized pairwise-similarity matrix of all data points, and then uses this embedding as the “cluster indicator”, an intermediate representation of the dataset that leads to fast convergence when used as input to k-means clustering. This process is very well illustrated in figure 1 of Lin and Cohen (2010) (reproduced below)

in which the leftmost image is the visualization of a dataset consisting of 3 circles, with points colored in red, green, and blue indicating clustering results, and the subsequent images show the power iteration process gradually transforming the original set of points into what appears to be three disjoint line segments, an intermediate representation that can be rapidly separated into 3 clusters using k-means clustering with $k = 3$.

In sparklyr 1.6, ml_power_iteration() was implemented to make the PIC functionality in Spark accessible from R. It expects as input a 3-column Spark dataframe that represents a pairwise-similarity matrix of all data points. Two of the columns in this dataframe should contain 0-based row and column indices, and the third column should hold the corresponding similarity measure. In the example below, we will see a dataset consisting of two circles being easily separated into two clusters by ml_power_iteration(), with the Gaussian kernel being used as the similarity measure between any 2 points:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

gen_similarity_matrix <- function() {
  # Guassian similarity measure
  guassian_similarity <- function(pt1, pt2) {
    exp(-sum((pt2 - pt1) ^ 2) / 2)
  }
  # generate evenly distributed points on a circle centered at the origin
  gen_circle <- function(radius, num_pts) {
    seq(0, num_pts - 1) %>%
      purrr::map_dfr(
        function(idx) {
          theta <- 2 * pi * idx / num_pts
          radius * c(x = cos(theta), y = sin(theta))
        })
  }
  # generate points on both circles
  pts <- rbind(
    gen_circle(radius = 1, num_pts = 80),
    gen_circle(radius = 4, num_pts = 80)
  )
  # populate the pairwise similarity matrix (stored as a 3-column dataframe)
  similarity_matrix <- data.frame()
  for (i in seq(2, nrow(pts)))
    similarity_matrix <- similarity_matrix %>%
      rbind(seq(i - 1L) %>%
        purrr::map_dfr(~ list(
          src = i - 1L, dst = .x - 1L,
          similarity = guassian_similarity(pts[i,], pts[.x,])
        ))
      )

  similarity_matrix
}

library(sparklyr)

sc <- spark_connect(master = "local")
sdf <- copy_to(sc, gen_similarity_matrix())
clusters <- ml_power_iteration(
  sdf, k = 2, max_iter = 10, init_mode = "degree",
  src_col = "src", dst_col = "dst", weight_col = "similarity"
)

clusters %>% print(n = 160)

## # A tibble: 160 x 2
##        id cluster
##     <dbl>   <int>
##   1     0       1
##   2     1       1
##   3     2       1
##   4     3       1
##   5     4       1
##   ...
##   157   156       0
##   158   157       0
##   159   158       0
##   160   159       0

The output shows points from the two circles being assigned to separate clusters, as expected, after only a small number of PIC iterations.

spark_write_rds() + collect_from_rds()

spark_write_rds() and collect_from_rds() are implemented as a less memory- consuming alternative to collect(). Unlike collect(), which retrieves all elements of a Spark dataframe through the Spark driver node, hence potentially causing slowness or out-of-memory failures when collecting large amounts of data, spark_write_rds(), when used in conjunction with collect_from_rds(), can retrieve all partitions of a Spark dataframe directly from Spark workers, rather than through the Spark driver node. First, spark_write_rds() will distribute the tasks of serializing Spark dataframe partitions in RDS version 2 format among Spark workers. Spark workers can then process multiple partitions in parallel, each handling one partition at a time and persisting the RDS output directly to disk, rather than sending dataframe partitions to the Spark driver node. Finally, the RDS outputs can be re-assembled to R dataframes using collect_from_rds().

Shown below is an example of spark_write_rds() + collect_from_rds() usage, where RDS outputs are first saved to HDFS, then downloaded to the local filesystem with hadoop fs -get, and finally, post-processed with collect_from_rds():

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

library(sparklyr)
library(nycflights13)

num_partitions <- 10L
sc <- spark_connect(master = "yarn", spark_home = "/usr/lib/spark")
flights_sdf <- copy_to(sc, flights, repartition = num_partitions)

# Spark workers serialize all partition in RDS format in parallel and write RDS
# outputs to HDFS
spark_write_rds(
  flights_sdf,
  dest_uri = "hdfs://<namenode>:8020/flights-part-{partitionId}.rds"
)

# Run `hadoop fs -get` to download RDS files from HDFS to local file system
for (partition in seq(num_partitions) - 1)
  system2(
    "hadoop",
    c("fs", "-get", sprintf("hdfs://<namenode>:8020/flights-part-%d.rds", partition))
  )

# Post-process RDS outputs
partitions <- seq(num_partitions) - 1 %>%
  lapply(function(partition) collect_from_rds(sprintf("flights-part-%d.rds", partition)))

# Optionally, call `rbind()` to combine data from all partitions into a single R dataframe
flights_df <- do.call(rbind, partitions)

Dplyr-related improvements

Similar to other recent sparklyr releases, sparklyr 1.6 comes with a number of dplyr-related improvements, such as

• Support for where() predicate within select() and summarize(across(...)) operations on Spark dataframes
• Addition of if_all() and if_any() functions
• Full compatibility with dbplyr 2.0 backend API

select(where(...)) and summarize(across(where(...)))

The dplyr where(...) construct is useful for applying a selection or aggregation function to multiple columns that satisfy some boolean predicate. For example,

1
2
3

library(dplyr)

iris %>% select(where(is.numeric))

returns all numeric columns from the iris dataset, and

1
2
3

library(dplyr)

iris %>% summarize(across(where(is.numeric), mean))

computes the average of each numeric column.

In sparklyr 1.6, both types of operations can be applied to Spark dataframes, e.g.,

1
2
3
4
5
6
7
8
9

library(dplyr)
library(sparklyr)

sc <- spark_connect(master = "local")
iris_sdf <- copy_to(sc, iris, name = random_string())

iris_sdf %>% select(where(is.numeric))

iris %>% summarize(across(where(is.numeric), mean))

if_all() and if_any()

if_all() and if_any() are two convenience functions from dplyr 1.0.4 (see here for more details) that effectively ¹ combine the results of applying a boolean predicate to a tidy selection of columns using the logical and/or operators.

Starting from sparklyr 1.6, if_all() and if_any() can also be applied to Spark dataframes, .e.g.,

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

library(dplyr)
library(sparklyr)

sc <- spark_connect(master = "local")
iris_sdf <- copy_to(sc, iris, name = random_string())

# Select all records with Petal.Width > 2 and Petal.Length > 2
iris_sdf %>% filter(if_all(starts_with("Petal"), ~ .x > 2))

# Select all records with Petal.Width > 5 or Petal.Length > 5
iris_sdf %>% filter(if_any(starts_with("Petal"), ~ .x > 5))

Compatibility with dbplyr 2.0 backend API

Sparklyr 1.6 is fully compatible with the newer dbplyr 2.0 backend API (by implementing all interface changes recommended in here ), while still maintaining backward compatibility with the previous edition of dbplyr API, so that sparklyr users will not be forced to switch to any particular version of dbplyr.

This should be a mostly non-user-visible change as of now. In fact, the only discernible behavior change will be the following code

1
2
3
4
5
6

library(dbplyr)
library(sparklyr)

sc <- spark_connect(master = "local")

print(dbplyr_edition(sc))

outputting

[1] 2

if sparklyr is working with dbplyr 2.0+, and

[1] 1

if otherwise.

Acknowledgements

In chronological order, we would like to thank the following contributors for making sparklyr 1.6 awesome:

• @yitao-li
• @pgramme
• @javierluraschi
• @andrew-christianson
• @jozefhajnala
• @nathaneastwood
• @mzorko

We would also like to give a big shout-out to the wonderful open-source community behind sparklyr, without whom we would not have benefitted from numerous sparklyr-related bug reports and feature suggestions.

Finally, the author of this blog post also very much appreciates the highly valuable editorial suggestions from @skeydan .

If you wish to learn more about sparklyr, we recommend checking out sparklyr.ai , spark.rstudio.com , and also some previous sparklyr release posts such as sparklyr 1.5 and sparklyr 1.4 .

That is all. Thanks for reading!

Greenwald, Michael, and Sanjeev Khanna. 2001. “Space-Efficient Online Computation of Quantile Summaries.” SIGMOD Rec. (New York, NY, USA) 30 (2): 58–66. https://doi.org/10.1145/376284.375670 .

Lin, Frank, and William Cohen. 2010. “Power Iteration Clustering.” August, 655–62.

To install sparklyr 1.5 from CRAN, run

1

install.packages("sparklyr")

In this blog post, we will highlight the following aspects of sparklyr 1.5:

• Better dplyr interface
• 4 useful additions to the sdf_* family of functions
• New RDS-based serialization routines along with several serialization-related improvements and bug fixes

Better dplyr interface

A large fraction of pull requests that went into the sparklyr 1.5 release were focused on making Spark dataframes work with various dplyr verbs in the same way that R dataframes do. The full list of dplyr-related bugs and feature requests that were resolved in sparklyr 1.5 can be found in here .

In this section, we will showcase three new dplyr functionalities that were shipped with sparklyr 1.5.

Stratified sampling

Stratified sampling on an R dataframe can be accomplished with a combination of dplyr::group_by() followed by dplyr::sample_n() or dplyr::sample_frac(), where the grouping variables specified in the dplyr::group_by() step are the ones that define each stratum. For instance, the following query will group mtcars by number of cylinders and return a weighted random sample of size two from each group, without replacement, and weighted by the mpg column:

1
2
3
4

mtcars %>%
  dplyr::group_by(cyl) %>%
  dplyr::sample_n(size = 2, weight = mpg, replace = FALSE) %>%
  print()

## # A tibble: 6 x 11
## # Groups:   cyl [3]
##     mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
##   <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1  33.9     4  71.1    65  4.22  1.84  19.9     1     1     4     1
## 2  22.8     4 108      93  3.85  2.32  18.6     1     1     4     1
## 3  21.4     6 258     110  3.08  3.22  19.4     1     0     3     1
## 4  21       6 160     110  3.9   2.62  16.5     0     1     4     4
## 5  15.5     8 318     150  2.76  3.52  16.9     0     0     3     2
## 6  19.2     8 400     175  3.08  3.84  17.0     0     0     3     2

Starting from sparklyr 1.5, the same can also be done for Spark dataframes with Spark 3.0 or above, e.g.,:

1
2
3
4
5
6
7
8
9

library(sparklyr)

sc <- spark_connect(master = "local", version = "3.0.0")
mtcars_sdf <- copy_to(sc, mtcars, replace = TRUE, repartition = 3)

mtcars_sdf %>%
  dplyr::group_by(cyl) %>%
  dplyr::sample_n(size = 2, weight = mpg, replace = FALSE) %>%
  print()

# Source: spark<?> [?? x 11]
# Groups: cyl
    mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
  <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1  21       6 160     110  3.9   2.62  16.5     0     1     4     4
2  21.4     6 258     110  3.08  3.22  19.4     1     0     3     1
3  27.3     4  79      66  4.08  1.94  18.9     1     1     4     1
4  32.4     4  78.7    66  4.08  2.2   19.5     1     1     4     1
5  16.4     8 276.    180  3.07  4.07  17.4     0     0     3     3
6  18.7     8 360     175  3.15  3.44  17.0     0     0     3     2

or

1
2
3
4

mtcars_sdf %>%
  dplyr::group_by(cyl) %>%
  dplyr::sample_frac(size = 0.2, weight = mpg, replace = FALSE) %>%
  print()

## # Source: spark<?> [?? x 11]
## # Groups: cyl
##     mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
##   <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1  21       6 160     110  3.9   2.62  16.5     0     1     4     4
## 2  21.4     6 258     110  3.08  3.22  19.4     1     0     3     1
## 3  22.8     4 141.     95  3.92  3.15  22.9     1     0     4     2
## 4  33.9     4  71.1    65  4.22  1.84  19.9     1     1     4     1
## 5  30.4     4  95.1   113  3.77  1.51  16.9     1     1     5     2
## 6  15.5     8 318     150  2.76  3.52  16.9     0     0     3     2
## 7  18.7     8 360     175  3.15  3.44  17.0     0     0     3     2
## 8  16.4     8 276.    180  3.07  4.07  17.4     0     0     3     3

Row sums

The rowSums() functionality offered by dplyr is handy when one needs to sum up a large number of columns within an R dataframe that are impractical to be enumerated individually. For example, here we have a six-column dataframe of random real numbers, where the partial_sum column in the result contains the sum of columns b through d within each row:

1
2
3
4
5
6
7
8

ncols <- 6
nums <- seq(ncols) %>% lapply(function(x) runif(5))
names(nums) <- letters[1:ncols]
tbl <- tibble::as_tibble(nums)

tbl %>%
  dplyr::mutate(partial_sum = rowSums(.[2:5])) %>%
  print()

## # A tibble: 5 x 7
##         a     b     c      d     e      f partial_sum
##     <dbl> <dbl> <dbl>  <dbl> <dbl>  <dbl>       <dbl>
## 1 0.781   0.801 0.157 0.0293 0.169 0.0978        1.16
## 2 0.696   0.412 0.221 0.941  0.697 0.675         2.27
## 3 0.802   0.410 0.516 0.923  0.190 0.904         2.04
## 4 0.200   0.590 0.755 0.494  0.273 0.807         2.11
## 5 0.00149 0.711 0.286 0.297  0.107 0.425         1.40

Beginning with sparklyr 1.5, the same operation can be performed with Spark dataframes:

1
2
3
4
5
6
7
8

library(sparklyr)

sc <- spark_connect(master = "local")
sdf <- copy_to(sc, tbl, overwrite = TRUE)

sdf %>%
  dplyr::mutate(partial_sum = rowSums(.[2:5])) %>%
  print()

## # Source: spark<?> [?? x 7]
##         a     b     c      d     e      f partial_sum
##     <dbl> <dbl> <dbl>  <dbl> <dbl>  <dbl>       <dbl>
## 1 0.781   0.801 0.157 0.0293 0.169 0.0978        1.16
## 2 0.696   0.412 0.221 0.941  0.697 0.675         2.27
## 3 0.802   0.410 0.516 0.923  0.190 0.904         2.04
## 4 0.200   0.590 0.755 0.494  0.273 0.807         2.11
## 5 0.00149 0.711 0.286 0.297  0.107 0.425         1.40

As a bonus from implementing the rowSums feature for Spark dataframes, sparklyr 1.5 now also offers limited support for the column-subsetting operator on Spark dataframes. For example, all code snippets below will return some subset of columns from the dataframe named sdf:

1
2

# select columns `b` through `e`
sdf[2:5]

1
2

# select columns `b` and `c`
sdf[c("b", "c")]

1
2

# drop the first and third columns and return the rest
sdf[c(-1, -3)]

Weighted-mean summarizer

Similar to the two dplyr functions mentioned above, the weighted.mean() summarizer is another useful function that has become part of the dplyr interface for Spark dataframes in sparklyr 1.5. One can see it in action by, for example, comparing the output from the following

1
2
3
4
5
6
7
8
9

library(sparklyr)

sc <- spark_connect(master = "local")

mtcars_sdf <- copy_to(sc, mtcars, replace = TRUE)
mtcars_sdf %>%
  dplyr::group_by(cyl) %>%
  dplyr::summarize(mpg_wm = weighted.mean(mpg, wt)) %>%
  print()

with output from the equivalent operation on mtcars in R:

1
2
3
4

mtcars %>%
  dplyr::group_by(cyl) %>%
  dplyr::summarize(mpg_wm = weighted.mean(mpg, wt)) %>%
  print()

both of them should evaluate to the following:

##     cyl mpg_wm
##   <dbl>  <dbl>
## 1     4   25.9
## 2     6   19.6
## 3     8   14.8

New additions to the sdf_* family of functions

sparklyr provides a large number of convenience functions for working with Spark dataframes, and all of them have names starting with the sdf_ prefix.

In this section we will briefly mention four new additions and show some example scenarios in which those functions are useful.

sdf_expand_grid()

As the name suggests, sdf_expand_grid() is simply the Spark equivalent of expand.grid(). Rather than running expand.grid() in R and importing the resulting R dataframe to Spark, one can now run sdf_expand_grid(), which accepts both R vectors and Spark dataframes and supports hints for broadcast hash joins. The example below shows sdf_expand_grid() creating a 100-by-100-by-10-by-10 grid in Spark over 1000 Spark partitions, with broadcast hash join hints on variables with small cardinalities:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

library(sparklyr)

sc <- spark_connect(master = "local")

grid_sdf <- sdf_expand_grid(
  sc,
  var1 = seq(100),
  var2 = seq(100),
  var3 = seq(10),
  var4 = seq(10),
  broadcast_vars = c(var3, var4),
  repartition = 1000
)

grid_sdf %>% sdf_nrow() %>% print()

## [1] 1e+06

sdf_partition_sizes()

As sparklyr user @sbottelli suggested here , one thing that would be great to have in sparklyr is an efficient way to query partition sizes of a Spark dataframe. In sparklyr 1.5, sdf_partition_sizes() does exactly that:

1
2
3
4
5
6
7

library(sparklyr)

sc <- spark_connect(master = "local")

sdf_len(sc, 1000, repartition = 5) %>%
  sdf_partition_sizes() %>%
  print(row.names = FALSE)

##  partition_index partition_size
##                0            200
##                1            200
##                2            200
##                3            200
##                4            200

sdf_unnest_longer() and sdf_unnest_wider()

sdf_unnest_longer() and sdf_unnest_wider() are the equivalents of tidyr::unnest_longer() and tidyr::unnest_wider() for Spark dataframes. sdf_unnest_longer() expands all elements in a struct column into multiple rows, and sdf_unnest_wider() expands them into multiple columns. As illustrated with an example dataframe below,

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

library(sparklyr)

sc <- spark_connect(master = "local")
sdf <- copy_to(
  sc,
  tibble::tibble(
    id = seq(3),
    attribute = list(
      list(name = "Alice", grade = "A"),
      list(name = "Bob", grade = "B"),
      list(name = "Carol", grade = "C")
    )
  )
)

1
2
3

sdf %>%
  sdf_unnest_longer(col = record, indices_to = "key", values_to = "value") %>%
  print()

evaluates to

## # Source: spark<?> [?? x 3]
##      id value key
##   <int> <chr> <chr>
## 1     1 A     grade
## 2     1 Alice name
## 3     2 B     grade
## 4     2 Bob   name
## 5     3 C     grade
## 6     3 Carol name

whereas

1
2
3

sdf %>%
  sdf_unnest_wider(col = record) %>%
  print()

evaluates to

## # Source: spark<?> [?? x 3]
##      id grade name
##   <int> <chr> <chr>
## 1     1 A     Alice
## 2     2 B     Bob
## 3     3 C     Carol

RDS-based serialization routines

Some readers must be wondering why a brand new serialization format would need to be implemented in sparklyr at all. Long story short, the reason is that RDS serialization is a strictly better replacement for its CSV predecessor. It possesses all desirable attributes the CSV format has, while avoiding a number of disadvantages that are common among text-based data formats.

In this section, we will briefly outline why sparklyr should support at least one serialization format other than arrow, deep-dive into issues with CSV-based serialization, and then show how the new RDS-based serialization is free from those issues.

Why arrow is not for everyone?

To transfer data between Spark and R correctly and efficiently, sparklyr must rely on some data serialization format that is well-supported by both Spark and R. Unfortunately, not many serialization formats satisfy this requirement, and among the ones that do are text-based formats such as CSV and JSON, and binary formats such as Apache Arrow, Protobuf, and as of recent, a small subset of RDS version 2. Further complicating the matter is the additional consideration that sparklyr should support at least one serialization format whose implementation can be fully self-contained within the sparklyr code base, i.e., such serialization should not depend on any external R package or system library, so that it can accommodate users who want to use sparklyr but who do not necessarily have the required C++ compiler tool chain and other system dependencies for setting up R packages such as arrow or protolite . Prior to sparklyr 1.5, CSV-based serialization was the default alternative to fallback to when users do not have the arrow package installed or when the type of data being transported from R to Spark is unsupported by the version of arrow available.

Why is the CSV format not ideal?

There are at least three reasons to believe CSV format is not the best choice when it comes to exporting data from R to Spark.

One reason is efficiency. For example, a double-precision floating point number such as .Machine$double.eps needs to be expressed as "2.22044604925031e-16" in CSV format in order to not incur any loss of precision, thus taking up 20 bytes rather than 8 bytes.

But more important than efficiency are correctness concerns. In a R dataframe, one can store both NA_real_ and NaN in a column of floating point numbers. NA_real_ should ideally translate to null within a Spark dataframe, whereas NaN should continue to be NaN when transported from R to Spark. Unfortunately, NA_real_ in R becomes indistinguishable from NaN once serialized in CSV format, as evident from a quick demo shown below:

1
2

original_df <- data.frame(x = c(NA_real_, NaN))
original_df %>% dplyr::mutate(is_nan = is.nan(x)) %>% print()

##     x is_nan
## 1  NA  FALSE
## 2 NaN   TRUE

1
2
3
4

csv_file <- "/tmp/data.csv"
write.csv(original_df, file = csv_file, row.names = FALSE)
deserialized_df <- read.csv(csv_file)
deserialized_df %>% dplyr::mutate(is_nan = is.nan(x)) %>% print()

##    x is_nan
## 1 NA  FALSE
## 2 NA  FALSE

Another correctness issue very much similar to the one above was the fact that "NA" and NA within a string column of an R dataframe become indistinguishable once serialized in CSV format, as correctly pointed out in this Github issue by @caewok and others.

RDS to the rescue!

RDS format is one of the most widely used binary formats for serializing R objects. It is described in some detail in chapter 1, section 8 of this document . Among advantages of the RDS format are efficiency and accuracy: it has a reasonably efficient implementation in base R, and supports all R data types.

Also worth noticing is the fact that when an R dataframe containing only data types with sensible equivalents in Apache Spark (e.g., RAWSXP, LGLSXP, CHARSXP, REALSXP, etc) is saved using RDS version 2, (e.g., serialize(mtcars, connection = NULL, version = 2L, xdr = TRUE)), only a tiny subset of the RDS format will be involved in the serialization process, and implementing deserialization routines in Scala capable of decoding such a restricted subset of RDS constructs is in fact a reasonably simple and straightforward task (as shown in here ).

Last but not least, because RDS is a binary format, it allows NA_character_, "NA", NA_real_, and NaN to all be encoded in an unambiguous manner, hence allowing sparklyr 1.5 to avoid all correctness issues detailed above in non-arrow serialization use cases.

Other benefits of RDS serialization

In addition to correctness guarantees, RDS format also offers quite a few other advantages.

One advantage is of course performance: for example, importing a non-trivially-sized dataset such as nycflights13::flights from R to Spark using the RDS format in sparklyr 1.5 is roughly 40%-50% faster compared to CSV-based serialization in sparklyr 1.4. The current RDS-based implementation is still nowhere as fast as arrow-based serialization though (arrow is about 3-4x faster), so for performance-sensitive tasks involving heavy serialization, arrow should still be the top choice.

Another advantage is that with RDS serialization, sparklyr can import R dataframes containing raw columns directly into binary columns in Spark. Thus, use cases such as the one below will work in sparklyr 1.5

1
2
3
4
5
6
7
8

library(sparklyr)

sc <- spark_connect(master = "local")

tbl <- tibble::tibble(
  x = list(serialize("sparklyr", NULL), serialize(c(123456, 789), NULL))
)
sdf <- copy_to(sc, tbl)

While most sparklyr users probably won’t find this capability of importing binary columns to Spark immediately useful in their typical sparklyr::copy_to() or sparklyr::collect() usages, it does play a crucial role in reducing serialization overheads in the Spark-based foreach parallel backend that was first introduced in sparklyr 1.2. This is because Spark workers can directly fetch the serialized R closures to be computed from a binary Spark column instead of extracting those serialized bytes from intermediate representations such as base64-encoded strings. Similarly, the R results from executing worker closures will be directly available in RDS format which can be efficiently deserialized in R, rather than being delivered in other less efficient formats.

Acknowledgement

In chronological order, we would like to thank the following contributors for making their pull requests part of sparklyr 1.5:

• @wkdavis
• @yitao-li
• @falaki
• @nathaneastwood
• @pgramme

We would also like to express our gratitude towards numerous bug reports and feature requests for sparklyr from a fantastic open-source community.

Finally, the author of this blog post is indebted to @javierluraschi , @batpigandme , and @skeydan for their valuable editorial inputs.

If you wish to learn more about sparklyr, check out sparklyr.ai , spark.rstudio.com , and some of the previous release posts such as sparklyr 1.4 and sparklyr 1.3 .

Thanks for reading!

1

install.packages("sparklyr")

In this blog post, we will showcase the following much-anticipated new functionalities from the sparklyr 1.4 release:

• Parallelized Weighted Sampling with Spark
• Support for Tidyr Verbs on Spark Dataframes
• ft_robust_scaler as the R interface for RobustScaler from Spark 3.0
• Option for enabling RAPIDS GPU acceleration plugin in spark_connect()
• Higher-order functions and dplyr-related improvements

Parallelized Weighted Sampling

Readers familiar with dplyr::sample_n() and dplyr::sample_frac() functions may have noticed that both of them support weighted-sampling use cases on R dataframes, e.g.,

1

dplyr::sample_n(mtcars, size = 3, weight = mpg, replace = FALSE)

               mpg cyl  disp  hp drat    wt  qsec vs am gear carb
Fiat 128      32.4   4  78.7  66 4.08 2.200 19.47  1  1    4    1
Merc 280C     17.8   6 167.6 123 3.92 3.440 18.90  1  0    4    4
Mazda RX4 Wag 21.0   6 160.0 110 3.90 2.875 17.02  0  1    4    4

and

1

dplyr::sample_frac(mtcars, size = 0.1, weight = mpg, replace = FALSE)

             mpg cyl  disp  hp drat    wt  qsec vs am gear carb
Honda Civic 30.4   4  75.7  52 4.93 1.615 18.52  1  1    4    2
Merc 450SE  16.4   8 275.8 180 3.07 4.070 17.40  0  0    3    3
Fiat X1-9   27.3   4  79.0  66 4.08 1.935 18.90  1  1    4    1

will select some random subset of mtcars using the mpg attribute as the sampling weight for each row. If replace = FALSE is set, then a row is removed from the sampling population once it gets selected, whereas when setting replace = TRUE, each row will always stay in the sampling population and can be selected multiple times.

Now the exact same use cases are supported for Spark dataframes in sparklyr 1.4! For example:

1
2
3
4
5
6

library(sparklyr)

sc <- spark_connect(master = "local")
mtcars_sdf <- copy_to(sc, mtcars, repartition = 4L)

dplyr::sample_n(mtcars_sdf, size = 5, weight = mpg, replace = FALSE)

will return a random subset of size 5 from the Spark dataframe mtcars_sdf.

More importantly, the sampling algorithm implemented in sparklyr 1.4 is something that fits perfectly into the MapReduce paradigm: as we have split our mtcars data into 4 partitions of mtcars_sdf by specifying repartition = 4L, the algorithm will first process each partition independently and in parallel, selecting a sample set of size up to 5 from each, and then reduce all 4 sample sets into a final sample set of size 5 by choosing records having the top 5 highest sampling priorities among all.

How is such parallelization possible, especially for the sampling without replacement scenario, where the desired result is defined as the outcome of a sequential process? A detailed answer to this question is in this blog post , which includes a definition of the problem (in particular, the exact meaning of sampling weights in term of probabilities), a high-level explanation of the current solution and the motivation behind it, and also, some mathematical details all hidden in one link to a PDF file, so that non-math-oriented readers can get the gist of everything else without getting scared away, while math-oriented readers can enjoy working out all the integrals themselves before peeking at the answer.

Tidyr Verbs

The specialized implementations of the following tidyr verbs that work efficiently with Spark dataframes were included as part of sparklyr 1.4:

• tidyr::fill
• tidyr::nest
• tidyr::unnest
• tidyr::pivot_wider
• tidyr::pivot_longer
• tidyr::separate
• tidyr::unite

We can demonstrate how those verbs are useful for tidying data through some examples.

Let’s say we are given mtcars_sdf, a Spark dataframe containing all rows from mtcars plus the name of each row:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

library(sparklyr)

sc <- spark_connect(master = "local")
mtcars_sdf <- cbind(
  data.frame(model = rownames(mtcars)),
  data.frame(mtcars, row.names = NULL)
) %>%
  copy_to(sc, ., repartition = 4L)

print(mtcars_sdf, n = 5)

# Source: spark<?> [?? x 12]
  model          mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
  <chr>        <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 Mazda RX4     21       6   160   110  3.9   2.62  16.5     0     1     4     4
2 Mazda RX4 W…  21       6   160   110  3.9   2.88  17.0     0     1     4     4
3 Datsun 710    22.8     4   108    93  3.85  2.32  18.6     1     1     4     1
4 Hornet 4 Dr…  21.4     6   258   110  3.08  3.22  19.4     1     0     3     1
5 Hornet Spor…  18.7     8   360   175  3.15  3.44  17.0     0     0     3     2
# … with more rows

and we would like to turn all numeric attributes in mtcar_sdf (in other words, all columns other than the model column) into key-value pairs stored in 2 columns, with the key column storing the name of each attribute, and the value column storing each attribute’s numeric value. One way to accomplish that with tidyr is by utilizing the tidyr::pivot_longer functionality:

1
2
3

mtcars_kv_sdf <- mtcars_sdf %>%
  tidyr::pivot_longer(cols = -model, names_to = "key", values_to = "value")
print(mtcars_kv_sdf, n = 5)

# Source: spark<?> [?? x 3]
  model     key   value
  <chr>     <chr> <dbl>
1 Mazda RX4 am      1
2 Mazda RX4 carb    4
3 Mazda RX4 cyl     6
4 Mazda RX4 disp  160
5 Mazda RX4 drat    3.9
# … with more rows

To undo the effect of tidyr::pivot_longer, we can apply tidyr::pivot_wider to our mtcars_kv_sdf Spark dataframe, and get back the original data that was present in mtcars_sdf:

1
2
3

tbl <- mtcars_kv_sdf %>%
  tidyr::pivot_wider(names_from = key, values_from = value)
print(tbl, n = 5)

# Source: spark<?> [?? x 12]
  model         carb   cyl  drat    hp   mpg    vs    wt    am  disp  gear  qsec
  <chr>        <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 Mazda RX4        4     6  3.9    110  21       0  2.62     1  160      4  16.5
2 Hornet 4 Dr…     1     6  3.08   110  21.4     1  3.22     0  258      3  19.4
3 Hornet Spor…     2     8  3.15   175  18.7     0  3.44     0  360      3  17.0
4 Merc 280C        4     6  3.92   123  17.8     1  3.44     0  168.     4  18.9
5 Merc 450SLC      3     8  3.07   180  15.2     0  3.78     0  276.     3  18
# … with more rows

Another way to reduce many columns into fewer ones is by using tidyr::nest to move some columns into nested tables. For instance, we can create a nested table perf encapsulating all performance-related attributes from mtcars (namely, hp, mpg, disp, and qsec). However, unlike R dataframes, Spark Dataframes do not have the concept of nested tables, and the closest to nested tables we can get is a perf column containing named structs with hp, mpg, disp, and qsec attributes:

1
2

mtcars_nested_sdf <- mtcars_sdf %>%
  tidyr::nest(perf = c(hp, mpg, disp, qsec))

We can then inspect the type of perf column in mtcars_nested_sdf:

1

sdf_schema(mtcars_nested_sdf)$perf$type

[1] "ArrayType(StructType(StructField(hp,DoubleType,true), StructField(mpg,DoubleType,true), StructField(disp,DoubleType,true), StructField(qsec,DoubleType,true)),true)"

and inspect individual struct elements within perf:

1
2

perf <- mtcars_nested_sdf %>% dplyr::pull(perf)
unlist(perf[[1]])

    hp    mpg   disp   qsec
110.00  21.00 160.00  16.46

Finally, we can also use tidyr::unnest to undo the effects of tidyr::nest:

1
2
3

mtcars_unnested_sdf <- mtcars_nested_sdf %>%
  tidyr::unnest(col = perf)
print(mtcars_unnested_sdf, n = 5)

# Source: spark<?> [?? x 12]
  model          cyl  drat    wt    vs    am  gear  carb    hp   mpg  disp  qsec
  <chr>        <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 Mazda RX4        6  3.9   2.62     0     1     4     4   110  21    160   16.5
2 Hornet 4 Dr…     6  3.08  3.22     1     0     3     1   110  21.4  258   19.4
3 Duster 360       8  3.21  3.57     0     0     3     4   245  14.3  360   15.8
4 Merc 280         6  3.92  3.44     1     0     4     4   123  19.2  168.  18.3
5 Lincoln Con…     8  3     5.42     0     0     3     4   215  10.4  460   17.8
# … with more rows

Robust Scaler

RobustScaler is a new functionality introduced in Spark 3.0 (SPARK-28399 ). Thanks to a pull request by @zero323 , an R interface for RobustScaler, namely, the ft_robust_scaler() function, is now part of sparklyr.

It is often observed that many machine learning algorithms perform better on numeric inputs that are standardized. Many of us have learned in stats 101 that given a random variable $X$, we can compute its mean $\mu = E[X]$, standard deviation $\sigma = \sqrt{E[X^2] - (E[X])^2}$, and then obtain a standard score $z = \frac{X - \mu}{\sigma}$ which has mean of 0 and standard deviation of 1.

However, notice both $E[X]$ and $E[X^2]$ from above are quantities that can be easily skewed by extreme outliers in $X$, causing distortions in $z$. A particular bad case of it would be if all non-outliers among $X$ are very close to $0$, hence making $E[X]$ close to $0$, while extreme outliers are all far in the negative direction, hence dragging down $E[X]$ while skewing $E[X^2]$ upwards.

An alternative way of standardizing $X$ based on its median, 1st quartile, and 3rd quartile values, all of which are robust against outliers, would be the following:

$\displaystyle z = \frac{X - \text{Median}(X)}{\text{P75}(X) - \text{P25}(X)}$

and this is precisely what RobustScaler offers.

To see ft_robust_scaler() in action and demonstrate its usefulness, we can go through a contrived example consisting of the following steps:

• Draw 500 random samples from the standard normal distribution

1
2

sample_values <- rnorm(500)
print(sample_values)

  [1] -0.626453811  0.183643324 -0.835628612  1.595280802  0.329507772
  [6] -0.820468384  0.487429052  0.738324705  0.575781352 -0.305388387
  ...

• Inspect the minimal and maximal values among the $500$ random samples:

1

print(min(sample_values))

  [1] -3.008049

1

print(max(sample_values))

  [1] 3.810277

• Now create $10$ other values that are extreme outliers compared to the $500$ random samples above. Given that we know all $500$ samples are within the range of $(-4, 4)$, we can choose $-501, -502, \ldots, -509, -510$ as our $10$ outliers:

1

outliers <- -500L - seq(10)

• Copy all $510$ values into a Spark dataframe named sdf

1
2
3
4

library(sparklyr)

sc <- spark_connect(master = "local", version = "3.0.0")
sdf <- copy_to(sc, data.frame(value = c(sample_values, outliers)))

• We can then apply ft_robust_scaler() to obtain the standardized value for each input:

1
2
3
4
5

scaled <- sdf %>%
  ft_vector_assembler("value", "input") %>%
  ft_robust_scaler("input", "scaled") %>%
  dplyr::pull(scaled) %>%
  unlist()

• Plotting the result shows the non-outlier data points being scaled to values that still more or less form a bell-shaped distribution centered around $0$, as expected, so the scaling is robust against influence of the outliers:

1
2
3
4
5

library(ggplot2)

ggplot(data.frame(scaled = scaled), aes(x = scaled)) +
  xlim(-7, 7) +
  geom_histogram(binwidth = 0.2)

• Finally, we can compare the distribution of the scaled values above with the distribution of z-scores of all input values, and notice how scaling the input with only mean and standard deviation would have caused noticeable skewness – which the robust scaler has successfully avoided:

1
2
3
4
5

all_values <- c(sample_values, outliers)
z_scores <- (all_values - mean(all_values)) / sd(all_values)
ggplot(data.frame(scaled = z_scores), aes(x = scaled)) +
  xlim(-0.05, 0.2) +
  geom_histogram(binwidth = 0.005)

• From the 2 plots above, one can observe while both standardization processes produced some distributions that were still bell-shaped, the one produced by ft_robust_scaler() is centered around $0$, correctly indicating the average among all non-outlier values, while the z-score distribution is clearly not centered around $0$ as its center has been noticeably shifted by the $10$ outlier values.

RAPIDS

Readers following Apache Spark releases closely probably have noticed the recent addition of RAPIDS GPU acceleration support in Spark 3.0. Catching up with this recent development, an option to enable RAPIDS in Spark connections was also created in sparklyr and shipped in sparklyr 1.4. On a host with RAPIDS-capable hardware (e.g., an Amazon EC2 instance of type ‘p3.2xlarge’), one can install sparklyr 1.4 and observe RAPIDS hardware acceleration being reflected in Spark SQL physical query plans:

1
2
3
4

library(sparklyr)

sc <- spark_connect(master = "local", version = "3.0.0", packages = "rapids")
dplyr::db_explain(sc, "SELECT 4")

== Physical Plan ==
*(2) GpuColumnarToRow false
+- GpuProject [4 AS 4#45]
   +- GpuRowToColumnar TargetSize(2147483647)
      +- *(1) Scan OneRowRelation[]

Higher-Order Functions and dplyr-Related Improvements

All newly introduced higher-order functions from Spark 3.0, such as array_sort() with custom comparator, transform_keys(), transform_values(), and map_zip_with(), are supported by sparklyr 1.4.

In addition, all higher-order functions can now be accessed directly through dplyr rather than their hof_* counterparts in sparklyr. This means, for example, that we can run the following dplyr queries to calculate the square of all array elements in column x of sdf, and then sort them in descending order:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

library(sparklyr)

sc <- spark_connect(master = "local", version = "3.0.0")
sdf <- copy_to(sc, tibble::tibble(x = list(c(-3, -2, 1, 5), c(6, -7, 5, 8))))

sq_desc <- sdf %>%
  dplyr::mutate(x = transform(x, ~ .x * .x)) %>%
  dplyr::mutate(x = array_sort(x, ~ as.integer(sign(.y - .x)))) %>%
  dplyr::pull(x)

print(sq_desc)

[[1]]
[1] 25  9  4  1

[[2]]
[1] 64 49 36 25

Acknowledgement

In chronological order, we would like to thank the following individuals for their contributions to sparklyr 1.4:

• @javierluraschi
• @nealrichardson
• @yitao-li
• @wkdavis
• @Loquats
• @zero323

We also appreciate bug reports, feature requests, and valuable other feedback about sparklyr from our awesome open-source community (e.g., the weighted sampling feature in sparklyr 1.4 was largely motivated by this Github issue filed by @ajing , and some dplyr-related bug fixes in this release were initiated in #2648 and completed with this pull request by @wkdavis ).

Last but not least, the author of this blog post is extremely grateful for fantastic editorial suggestions from @javierluraschi , @batpigandme , and @skeydan .

If you wish to learn more about sparklyr, we recommend checking out sparklyr.ai , spark.rstudio.com , and also some of the previous release posts such as sparklyr 1.3 and sparklyr 1.2 .

Thanks for reading!

Now, in order to process ImageNet, we will first have to divide and conquer, partitioning the dataset into several manageable subsets. Afterwards, we will train ImageNet using AlexNet across multiple GPUs and compute instances. Preprocessing ImageNet and distributed training are the two topics that this post will present and discuss, starting with preprocessing ImageNet.

Preprocessing ImageNet

When dealing with large datasets, even simple tasks like downloading or reading a dataset can be much harder than what you would expect. For instance, since ImageNet is roughly 300GB in size, you will need to make sure to have at least 600GB of free space to leave some room for download and decompression. But no worries, you can always borrow computers with huge disk drives from your favorite cloud provider. While you are at it, you should also request compute instances with multiple GPUs, Solid State Drives (SSDs), and a reasonable amount of CPUs and memory. If you want to use the exact configuration we used, take a look at the mlverse/imagenet repo, which contains a Docker image and configuration commands required to provision reasonable computing resources for this task. In summary, make sure you have access to sufficient compute resources.

Now that we have resources capable of working with ImageNet, we need to find a place to download ImageNet from. The easiest way is to use a variation of ImageNet used in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) , which contains a subset of about 250GB of data and can be easily downloaded from many Kaggle competitions, like the ImageNet Object Localization Challenge .

If you’ve read some of our previous posts, you might be already thinking of using the pins package, which you can use to: cache, discover and share resources from many services, including Kaggle. You can learn more about data retrieval from Kaggle in the Using Kaggle Boards article; in the meantime, let’s assume you are already familiar with this package.

All we need to do now is register the Kaggle board, retrieve ImageNet as a pin, and decompress this file. Warning, the following code requires you to stare at a progress bar for, potentially, over an hour.

1
2
3
4
5

library(pins)
board_register("kaggle", token = "kaggle.json")

pin_get("c/imagenet-object-localization-challenge", board = "kaggle")[1] %>%
  untar(exdir = "/localssd/imagenet/")

If we are going to be training this model over and over using multiple GPUs and even multiple compute instances, we want to make sure we don’t waste too much time downloading ImageNet every single time.

The first improvement to consider is getting a faster hard drive. In our case, we locally-mounted an array of SSDs into the /localssd path. We then used /localssd to extract ImageNet and configured R’s temp path and pins cache to use the SSDs as well. Consult your cloud provider’s documentation to configure SSDs, or take a look at mlverse/imagenet .

Next, a well-known approach we can follow is to partition ImageNet into chunks that can be individually downloaded to perform distributed training later on.

In addition, it is also faster to download ImageNet from a nearby location, ideally from a URL stored within the same data center where our cloud instance is located. For this, we can also use pins to register a board with our cloud provider and then re-upload each partition. Since ImageNet is already partitioned by category, we can easily split ImageNet into multiple zip files and re-upload to our closest data center as follows. Make sure the storage bucket is created in the same region as your computing instances.

1
2
3
4
5
6
7

board_register("<board>", name = "imagenet", bucket = "r-imagenet")

train_path <- "/localssd/imagenet/ILSVRC/Data/CLS-LOC/train/"
for (path in dir(train_path, full.names = TRUE)) {
  dir(path, full.names = TRUE) %>%
    pin(name = basename(path), board = "imagenet", zip = TRUE)
}

We can now retrieve a subset of ImageNet quite efficiently. If you are motivated to do so and have about one gigabyte to spare, feel free to follow along executing this code. Notice that ImageNet contains lots of JPEG images for each WordNet category.

1
2
3
4
5

board_register("https://storage.googleapis.com/r-imagenet/", "imagenet")

categories <- pin_get("categories", board = "imagenet")
pin_get(categories$id[1], board = "imagenet", extract = TRUE) %>%
  tibble::as_tibble()

# A tibble: 1,300 x 1
   value                                                           
   <chr>                                                           
 1 /localssd/pins/storage/n01440764/n01440764_10026.JPEG
 2 /localssd/pins/storage/n01440764/n01440764_10027.JPEG
 3 /localssd/pins/storage/n01440764/n01440764_10029.JPEG
 4 /localssd/pins/storage/n01440764/n01440764_10040.JPEG
 5 /localssd/pins/storage/n01440764/n01440764_10042.JPEG
 6 /localssd/pins/storage/n01440764/n01440764_10043.JPEG
 7 /localssd/pins/storage/n01440764/n01440764_10048.JPEG
 8 /localssd/pins/storage/n01440764/n01440764_10066.JPEG
 9 /localssd/pins/storage/n01440764/n01440764_10074.JPEG
10 /localssd/pins/storage/n01440764/n01440764_1009.JPEG 
# … with 1,290 more rows

When doing distributed training over ImageNet, we can now let a single compute instance process a partition of ImageNet with ease. Say, 1/16 of ImageNet can be retrieved and extracted, in under a minute, using parallel downloads with the callr package:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

categories <- pin_get("categories", board = "imagenet")
categories <- categories$id[1:(length(categories$id) / 16)]

procs <- lapply(categories, function(cat)
  callr::r_bg(function(cat) {
    library(pins)
    board_register("https://storage.googleapis.com/r-imagenet/", "imagenet")
    
    pin_get(cat, board = "imagenet", extract = TRUE)
  }, args = list(cat))
)
  
while (any(sapply(procs, function(p) p$is_alive()))) Sys.sleep(1)

We can wrap this up partition in a list containing a map of images and categories, which we will later use in our AlexNet model through tfdatasets .

1
2
3
4
5
6
7
8
9

data <- list(
    image = unlist(lapply(categories, function(cat) {
        pin_get(cat, board = "imagenet", download = FALSE)
    })),
    category = unlist(lapply(categories, function(cat) {
        rep(cat, length(pin_get(cat, board = "imagenet", download = FALSE)))
    })),
    categories = categories
)

Great! We are halfway there training ImageNet. The next section will focus on introducing distributed training using multiple GPUs.

Distributed Training

Now that we have broken down ImageNet into manageable parts, we can forget for a second about the size of ImageNet and focus on training a deep learning model for this dataset. However, any model we choose is likely to require a GPU, even for a 1/16 subset of ImageNet. So make sure your GPUs are properly configured by running is_gpu_available(). If you need help getting a GPU configured, the Using GPUs with TensorFlow and Docker video can help you get up to speed.

1
2

library(tensorflow)
tf$test$is_gpu_available()

[1] TRUE

We can now decide which deep learning model would best be suited for ImageNet classification tasks. Instead, for this post, we will go back in time to the glory days of AlexNet and use the r-tensorflow/alexnet repo instead. This repo contains a port of AlexNet to R, but please notice that this port has not been tested and is not ready for any real use cases. In fact, we would appreciate PRs to improve it if someone feels inclined to do so. Regardless, the focus of this post is on workflows and tools, not about achieving state-of-the-art image classification scores. So by all means, feel free to use more appropriate models.

Once we’ve chosen a model, we will want to me make sure that it properly trains on a subset of ImageNet:

1
2

remotes::install_github("r-tensorflow/alexnet")
alexnet::alexnet_train(data = data)

Epoch 1/2
 103/2269 [>...............] - ETA: 5:52 - loss: 72306.4531 - accuracy: 0.9748

So far so good! However, this post is about enabling large-scale training across multiple GPUs, so we want to make sure we are using as many as we can. Unfortunately, running nvidia-smi will show that only one GPU currently being used:

1

nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.152.00   Driver Version: 418.152.00   CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:05.0 Off |                    0 |
| N/A   48C    P0    89W / 149W |  10935MiB / 11441MiB |     28%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           Off  | 00000000:00:06.0 Off |                    0 |
| N/A   74C    P0    74W / 149W |     71MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

In order to train across multiple GPUs, we need to define a distributed-processing strategy. If this is a new concept, it might be a good time to take a look at the Distributed Training with Keras tutorial and the distributed training with TensorFlow docs. Or, if you allow us to oversimplify the process, all you have to do is define and compile your model under the right scope. A step-by-step explanation is available in the Distributed Deep Learning with TensorFlow and R video. In this case, the alexnet model already supports a strategy parameter, so all we have to do is pass it along.

1
2
3
4
5

library(tensorflow)
strategy <- tf$distribute$MirroredStrategy(
  cross_device_ops = tf$distribute$ReductionToOneDevice())

alexnet::alexnet_train(data = data, strategy = strategy, parallel = 6)