e e c r i è t t s s u i n J i a M l e

d Headless Chr me Automation with

THE CRRRI PACKAGE

Romain Lesur

Deputy Head of the Statistical Service

Retrouvez-nous sur justice.gouv.fr

A web browser is like a shadow puppet theater

Suyash Dwivedi CC BY-SA 4.0 via Wikimedia Commons

Ministère crrri package — Headless Automation with p. 2 de la Justice Behind the scenes

The puppet masters

Mr.Niwat Tantayanusorn, Ph.D. CC BY-SA 4.0 via Wikimedia Commons

Ministère crrri package — Headless Automation with p. 3 de la Justice What is a ?

Turn off the light: no visual interface

Be the stage director… in the dark!

Kent Wang from London, United Kingdom CC BY-SA 2.0 via Wikimedia Commons

Ministère crrri package — Headless Automation with p. 4 de la Justice Some use cases

Responsible (with JavaScript generated content)

Webpages screenshots

PDF generation

Testing (or Shiny apps)

Ministère crrri package — Headless Automation with p. 5 de la Justice Related packages

{RSelenium} client for WebDriver, requires a Selenium server Headless browser is an old (Java). topic {webshot}, {webdriver} relies on the abandoned PhantomJS library.

{hrbrmstr/} uses the HtmlUnit Java library.

{hrbrmstr/splashr} uses the Splash python library.

{hrbrmstr/decapitated} uses headless Chrome command-line instructions or the Node.js gepetto module (built-on top of the puppeteer Node.js module)

Ministère crrri package — Headless Automation with p. 6 de la Justice Headless Chr me

Basic tasks can be executed using command-line instructions Since Chrome 59 Offers the possibility to have the full control of Chrome using Node.js modules puppeteer, chrome-remote-interface…

Have the full control of from without Java, Node or any server The crrri package developed with Low-level API inspired by the chrome-remote-interface JS module Christophe Dervieux WIP give access to 500+ functions to control Chrome github.com/RLesur/crrri Dedicated to advanced uses / R packages developers

Compatible with , EdgeHtml and

Ministère crrri package — Headless Automation with p. 7 de la Justice Technical explanations

Headless Chrome can be controlled using the Chrome DevTools Protocol (CDP)

1. Launch Chrome in headless mode Steps to interact with headless Chrome 2. Connect R to Chrome through

3. Build an asynchronous function that sends CDP commands to Chrome listen to CDP events from Chrome

4. Execute this async flow with R

The goal of {crrri} is to ease these steps.

Ministère crrri package — Headless Automation with p. 8 de la Justice Chrome DevTools Protocol

Program actions usually done with Chrome DevTools

Ministère crrri package — Headless Automation with p. 9 de la Justice Playing with headless Chr me in RStudio 1.2

remotes::install_github("rlesur/crrri") client is a connection library(crrri) object chrome <- Chrome$new() # launch headless Chrome client <- chrome$connect(callback = ~.x$inspect())

Inspect headless Chrome in RStudio

Ministère crrri package — Headless Automation with p. 10 de la Justice Chrome DevTools Protocol commands: an example

Page <- client$Page # extract a domain A domain is a set of Page$navigate(url = "://urosconf.org") commands and events #> listeners

Ministère crrri package — Headless Automation with p. 11 de la Justice An API similar to JavaScript

Page$navigate(url = "https://urosconf.org") All the functions are #> asynchronous An object of class Promise from the {promises} package.

Page$navigate(url = "https://urosconf.org") %...>% Chain with the appropriate print() pipe! #> $frameId #> [1] "D1660E2ECC76A8356F78820F410BAA8C"

#> $loaderId #> [1] "18180FE5BE9D9A60CC37F01610227729"

Ministère crrri package — Headless Automation with p. 12 de la Justice Chaining commands and events listeners

# ask Chrome to send Page domain events To receive events from Page$enable() %...>% { Chrome most of domains # send the 'Page.navigate' command need to be enabled Page$navigate(url = "https://urosconf.org") } %...>% { cat('Navigation starts in frame', .$frameId, '\n') # wait the event 'Page.frameStoppedLoading' # fires for the main frame Page$frameStoppedLoading(frameId = .$frameId) } %...>% { cat('Main frame loaded.\n') }

Chrome DevTools Protocol documentation

chromedevtools.github.io/devtools-protocol

Ministère crrri package — Headless Automation with p. 13 de la Justice Building higher level functions

print_pdf <- function(client) { Write an asynchronous Page <- client$Page remote flow Page$enable() %...>% { Page$navigate(url = "https://r-project.org/") Page$loadEventFired() # await the load event } %...>% { Page$printToPDF() } %...>% # await PDF reception write_base64("r_project.pdf") }

Modify this script depending on the page content (JS libraries…)

perform_with_chrome(print_pdf) Perform this flow synchronously in R

Ministère crrri package — Headless Automation with p. 14 de la Justice Headless Chrome features

More than 40 domains, 400+ commands, 100+ events Frequent updates Each release of Chrome brings new features. No need to update {crrri}: commands are fetched from Chrome.

DOM/CSS manipulation, extraction Features JavaScript Runtime (more than V8) Inspect/intercept network traffic Emulate devices Set JS bindings between Chrome and R PDF generation Screenshots Screencast…

Ministère crrri package — Headless Automation with p. 15 de la Justice Example: emulate a device

iPhone8 <- function(client) { Screenshot your website Emulation <- client$Emulation with different devices Page <- client$Page Emulation$setDeviceMetricsOverride( width = 375, height = 667, mobile = TRUE, deviceScaleFactor = 2 ) %...>% { Page$enable() } %...>% { Page$navigate("https://rlesur.github.io/crrri") } %...>% { Page$loadEventFired() } %>% wait(3) %...>% { Page$captureScreenshot() } %...>% write_base64("iphone8.png") }

perform_with_chrome(iPhone8)

Ministère crrri package — Headless Automation with p. 16 de la Justice Example: screencast

Navigate in RStudio 1.2 and record screencast see on Youtube

Ministère crrri package — Headless Automation with p. 17 de la Justice Example: web scraping

dump_DOM <- function(client) { Page <- client$Page Runtime <- client$Runtime Page$enable() %...>% { Page$navigate(url = 'https://github.com') } %...>% { Page$loadEventFired() } %>% wait(3) %...>% { Runtime$evaluate( expression = 'document.documentElement.outerHTML' ) } %...>% { writeLines(.$result$value, "gh.") } }

perform_with_chrome(dump_DOM)

Ministère crrri package — Headless Automation with p. 18 de la Justice Conclusion

{crrri} package

only one dependency: Chrome Pros Java, Node or a server are not required easy to use with Travis or Docker integrates well with RStudio 1.2 just update Chrome to get the latest features flexible: define your own flow compatible with Shiny (because of the {promises} package) orchestrate headless Chrome in your Shiny app!

low-level interface: Chrome DevTools Protocol is highly technical Cons mostly dedicated to R developers/hackers

Ministère crrri package — Headless Automation with p. 19 de la Justice Credits

Miles McBain for chradle Thanks to Bob Rudis for decapitated

Andrea Cardaci for chrome-remote-interface

Marvelapp for devices.

Questions?

This deck was made with pagedown and is licensed under

Ministère crrri package — Headless Automation with p. 20 de la Justice