The goal of this document is to introduce the user to the seleniumPipes
package and to illustrate the basic operations/functions in the package.
WebDriver is a remote control interface that enables introspection and control of user agents. It provides a platform- and language-neutral wire protocol as a way for out-of-process programs to remotely instruct the behaviour of web browsers.
seleniumPipes
implements the w3c specification for webdrivers.
The seleniumPipes
package provides functions to connect to and communicate with a server which accepts w3c webdriver compliant requests. The seleniumPipes
package does not provide functions or methods to manage such a server. An appropriate running server is assummed throughout.
In this short section we will highlight some ways a server maybe started.
On a given machine a Selenium Server can be started using the stand-alone binary available from the Selenium Project. This link contains version of the Selenium Server. A stand-alone binary is released with each version. Once downloaded it can be started via the commandline in its most basic form as:
> java -jar selenium-server-standalone-3.0.0-beta2.jar
This assumes the file selenium-server-standalone-3.0.0-beta2.jar
is in our path or we are issuing the command from the files directory. There are many additional options which can be passed via the commadline so of which are listed using -help
:
> java -jar selenium-server-standalone-3.0.0-beta2.jar -help
Other options are passed as system properties to the JVM:
> java -DpropertyName=value -jar selenium-server-standalone-3.0.0-beta2.jar -port 4445
The Selenium Project maintains a number of Docker images that can be used to run a Docker container with a Selenium Server. A introduction to running a Selenium Server via Docker containers can be found in the RSelenium package.
The user can start and manage a Selenium server via an appropriate R script. An example of sourcing and starting a Selenium Server is given in the RSelenium package
in the serverUtils
directory full path file.path(find.package("RSelenium"), "examples/serverUtils")
.
In what follows I am running my code on Ubuntu 16.04. The same machine has a Selenium Server running in a docker container. This Docker image can be found here. I am starting my Selenium server with the following docker
command:
docker run -d -p 4445:4444 -p 5901:5900 selenium/standalone-chrome-debug:2.53.1
So I am running the standalone chrome debug image from the Selenium project at Docker Hub. I am running the version of this image with tag 2.53.1
. On my host machine the Selenium Server is exposed on port 4445 and the browser can be viewed via VNC on port 5901 on my ubuntu 16.04 host.
The above is only for the users information and allows them to replicate my setup. It is not necessary and the user may use any Selenium Server they have access to.
To connect to the Selenium Server seleniumPipes
provides a remoteDr
function which takes a number of arguments and returns an object of class rDriver
which can be used for further interaction with the Server.
library(seleniumPipes)
remDr <- remoteDr(browserName = "chrome", port = 4445L)
> remDr
Remote Ip Address: http://localhost:4445/wd/hub
Remote sessionId: 640ad436-74e1-4ce8-afc3-898c0dee095d
> class(remDr)
[1] "rDriver"
Running the above should connect to the running Selenium Server which has default ip address (localhost) and is running on port 4445 (the default is 4444). It asks to open a chrome
browser (the default is firefox). A browser should be opened:
To close the browser and remove the session from the Selenium Server we use the deleteSession
function.
remDr %>% deleteSession
The browser on the machine running Selenium Server should now be closed and the session removed from the Selenium Server.
There are a number of functions in seleniumPipes
that find elements in the DOM of the remote browser. They return objects of class wElement
and can be used for further chaining.
The findElement
function allows the user to search for an element in the remote browser DOM.
remDr <- remoteDr(browserName = "chrome", port = 4445L)
remDr %>% go("http://www.google.com/ncr")
# find the search form query box and search for "R project"
webElem <- remDr %>% findElement("name", "q")
If the selector returns a match an object of class wElement
is returned:
> webElem
ElementId: 0
Remote Driver:
Remote Ip Address: http://localhost:4445/wd/hub
Remote sessionId: 53f7a733-0e0e-4f73-87f2-e141ed530ebe
> class(webElem)
[1] "wElement"
There are a number of different ways to search for an element. xpath
, css
, id
are all examples of how a search can be carried out. Look at the help files for more info (?findElement
)
There are a number of functions which allow you to interact with the element in the DOM. As an example ?elementSendKeys
. Looking at the help file we see other functions in the elementInteraction
group. As an example we will send some text to the google search box, clear it then send different text and submit it by pressing the search button.
queryBox <- remDr %>% findElement("name", "q")
# send text to the query box
queryBox %>% elementSendKeys("Some ", "text")
# clear the query box
queryBox %>% elementClear
# get the search button
searchBtn <- remDr %>% findElement("name", "btnG")
# send text to query box
queryBox %>% elementSendKeys("R project")
# click the search button
searchBtn %>% elementClick
Using seleniumPipes
we can inject JavaScript into the remote browser. There are two functions executeScript and executeAsyncScript which execute JavaScript synchronously and asynchronously respectively.
Synchronous execution executes the script and returns any values. We can pass arguments to our script as follows:
> remDr %>% executeScript("return 1+1;")
[1] 2
> remDr %>% executeScript("return arguments[0]+1;", list(1))
[1] 2
> remDr %>% executeScript("return arguments[0]+arguments[1];", list(1,2))
[1] 3
We can pass a web Element as an argument to the execute
functions and also have web Elements returned.
As an example we navigate to the Google home page. We select the search box in the DOM. We then pass the resulting web element back in our script and change the search box color:
searchElem <- remDr %>% go("http://www.google.com/ncr") %>%
findElement("name", "q")
blueScript <- "arguments[0].style.backgroundColor = 'blue';"
remDr %>% executeScript(blueScript, list(searchElem))
We can return web Elements from our script also. When web Elements are detected they are converted. This option is converned by thereplace
argument. By default elements in the DOM are replaced with web Elements.
As an example lets refresh our page and return the google logo:
logoScript <- "return document.getElementById('hplogo');"
logoElem <- remDr %>% refresh %>%
executeScript(script = logoScript, args = list())
We can check that we have a web Element returned. We can also pass it back and change the visibility of the logo:
#> logoElem
#ElementId: 1
#Remote Driver:
#Remote Ip Address: http://localhost:4445/wd/hub
#Remote sessionId: 5e322e1a-c06e-43c3-bebe-fe4465e0df7a
remDr %>% executeScript("arguments[0].hidden = true;", args = list(logoElem))
# and turn the visibility back on
remDr %>% executeScript("arguments[0].hidden = false;", args = list(logoElem))
Finally we show a slightly more complex object being returned with replace
as TRUE
and FALSE
respectively
# Return a web Element in a more complex object
script <- "var test ={num:1, str:'a', el:document.getElementById('hplogo')};return test;"
remDr %>% executeScript(script = script
, args = list())
# $str
# [1] "a"
#
# $num
# [1] 1
#
# $el
# ElementId: 0
# Remote Driver:
# Remote Ip Address: http://localhost:4444/wd/hub
# Remote sessionId: 9a83672a-d72b-4873-aa7d-96f7f1f80fa0
# Run with replace = FALSE
remDr %>% executeScript(script = script
, args = list(), replace = FALSE)
# $str
# [1] "a"
#
# $num
# [1] 1
#
# $el
# $el$ELEMENT
# [1] "0"
By default the timeout on scripts is effectively set to zero. Asynchronous execution basically means our function waits for the script to finish execution. An appropriate value should be set for script timeouts. Here we choose the default value of 10 seconds (10000 millisconds).
A callback function is passed as the last argument and we call
this function to signal ouyr script is finished.
Here is an example illustrating the difference between sync calls and async calls:
remDr %>% setTimeout("script") # set default 10 secs for script timeout
asScript <- "cb = arguments[0];setTimeout(function(){cb('DONE');},5000); "
system.time(test1 <- remDr %>% executeAsyncScript(asScript, args = list()))
sScript <- "setTimeout(function(){},5000); return 'DONE';"
system.time(test2 <- remDr %>% executeScript(sScript, args = list()))
As a slightly more involved example:
script <- "
var cb = arguments[ arguments.length - 1 ];
secondValue = arguments[0];
setTimeout(function () {
cb({
firstValue: 1,
secondValue: secondValue
});
}, 3000);
"
res <- remDr %>% executeAsyncScript(script = script, list("Super 2nd value"))
#> system.time(res <- remDr %>% executeAsyncScript(script = script, list("Super 2nd value")))
# user system elapsed
# 0.032 0.000 3.103
#> res
#$secondValue
#[1] "Super 2nd value"
#
#$firstValue
#[1] 1
remDr %>% deleteSession()
Notice we can return a JavaScript object.
By default there is a built-in retry
function in the seleniumPipes
codebase. If a call to the server returns an error the call is tried a number of times. See ?retry
for more details.
By default functions are tried three times with a 5 second delay between tries. These values can be changed see the package options ?seleniumPipes
.
As an example of how the retry
functionality works consider the following
remDr <- remoteDr(browserName = "chrome", port = 4445L)
webElem <- remDr %>% go("http://www.google.com/ncr") %>%
findElement("name", "q")
# change the name of q with an 8 second delay
myscript <- "var myElem = arguments[0]; window.setTimeout(function(){
myElem.setAttribute('name','funkyname');
}, 8000);"
remDr %>% executeScript(myscript, args = list(webElem))
newWebElem <- remDr %>% findElement("name", "funkyname")
# > newWebElem <- remDr %>% findElement("name", "funkyname")
#
# Calling findElement - Try no: 1 of 3
#
# Calling findElement - Try no: 2 of 3
newWebElem %>% getElementAttribute("name")
In the above we navigate to the google homepage. We select an element in the DOM with name q
. We change the name of the element but set an 8 second delay on the change. Then we try to find the element. The first two attempts fail. The third attempt successfully finds the element as our 8 second delay is over and the time between function calls is a default 5 seconds.
We can effectively turn retry
off by setting appropriate options e.g.
# options(seleniumPipes_no_try = 1L)
# options(seleniumPipes_no_try_delay = 50L)
which would limit calls to a given function to 1 try and a 50 millisecond delay. We can also pass a retry argument in all the package functions
remDr %>% findElement("id", "i am not here", retry = list(noTry = 5, delay = 10))
remDr %>% findElement("id", "i am not here", retry = FALSE)
remDr %>% deleteSession
Sometimes websites can be composed using frames. These are in effect seperate webpages which are brought together in a frameset. A webdriver needs to switch from frame to frame. The easiest way to explain this is by way of example.
The r-project
web-page at CRAN
conviently uses frames so we will interact with it:
remDr <- remoteDr(browserName = "chrome", port = 4445L)
remDr %>% go("https://cran.r-project.org/") %>%
getPageSource %>%
xml_find_all("//frame") %>%
xml_attr("name")
# [1] "logo" "contents" "banner"
So we have navigated to the CRAN
page, returned the page source, passed it to some functions in xml2
which have returned the frame names
.
We can have a look at the structure of the html:
remDr %>% getPageSource %>%
html_structure
#<html [xmlns]>
# <head>
# {text}
# <title>
# {text}
# {text}
# <meta [http-equiv, content]>
# {text}
# <link [rel, href, type]>
# {text}
# <link [rel, href, type]>
# {text}
# <link [rel, type, href]>
# {text}
# {text}
# <frameset [cols, style]>
# {text}
# <frameset [rows]>
# {text}
# <frame [src, name, frameborder]>
# {text}
# <frame [src, name, frameborder]>
# {text}
# {text}
# <frame [src, name, frameborder]>
# {text}
# <noframes>
# {text}
# {text}
# {text}
Looking at the structure we can see the three frames we found. The logo frame contains simple the logo. The contents frame is the side menu on the left and the banner frame is the main section on the right. Lets inject some JavaScript to make this clearer:
frames <- remDr %>% findElements("css", "frame")
script <- "arguments[0].setAttribute(arguments[1], arguments[2]);"
style <- paste("border: 5px solid", c("yellow", "red", "blue"))
remDr %>% {lapply(seq(3), function(x){
executeScript(., script, list(frames[[x]], "style", style[x]))
})
}
The above code colors the frames yellow (logo), red(content) and blue(banner).