Using selenium on NodeJS

 

In this post we will review how to use selenium on NodeJS to automate browser usage. This is usually used for automatic site testing. We will make some changes which are specifically useful for such scenario.

First, lets install the chormedriver from this link. Make sure to download the zip file which match you chrome browser version, unzip it, and move the chormedriver to a folder which is in the path, for example:

sudo mv chromedriver /usr/local/bin/

Now for the project: create the package.json file:

{
  "name": "demo",
  "version": "1.0.0",
  "description": "",
  "author": "",
  "license": "ISC",
  "scripts": {
    "demo": "node main.js"
  },
  "dependencies": {
    "chrome-modheader": "^1.0.6",
    "selenium-webdriver": "^4.0.0-beta.3"
  }
}

We use the selenium driver, and add the chrome-modheader, which allows us to set headers on the requests. Setting header can be used for A/B testing, and for setting the XFF header to simulate a source IP.

The general code structure is as follows:

const {Builder, until, By, logging} = require('selenium-webdriver')
const chrome = require('selenium-webdriver/chrome')
const {getExtension, getAddHeaderUrl} = require('chrome-modheader')

main()

async function main() {
  // our code here
}

We start our code by starting a browser, and setting a random IP in the XFF header.

const preferences = new logging.Preferences()
preferences.setLevel(logging.Type.BROWSER, logging.Level.ALL)

const options = new chrome.Options()
options.setLoggingPrefs(preferences)
options.addArguments('--ignore-certificate-errors')
options.addArguments('--no-sandbox')
options.addExtensions(getExtension())


const driver = await new Builder()
  .forBrowser('chrome')
  .setChromeOptions(options)
  .build()

function getRandomIpSection() {
  return Math.ceil(Math.random() * 256)
}

const ip = `${getRandomIpSection()}.${getRandomIpSection()}.${getRandomIpSection()}.${getRandomIpSection()}`
console.log(`random IP is ${ip}`)
await driver.get(getAddHeaderUrl('X-Forwarded-For', ip))

Now we open our site, and cleanup cookies for a fresh session:

await driver.get('http://my.site.com/')
await driver.manage().deleteAllCookies()

We can locate elements by CSS and by Xpath:

const BUTTON_SELECTOR = '#features a.read-more:first-of-type'
const button = await driver.wait(until.elementLocated(By.css(BUTTON_SELECTOR)), 10000)

const TEXT_SELECTOR = '//div/span[contains(@style,\'color: green\')]'
const text = await driver.wait(until.elementLocated(By.xpath(TEXT_SELECTOR)), 10000)

We can click on items. In this case we click on the element that we have just located.

button.click()

We can run our own javacript code in the page:

await driver.executeScript(`
  window.enableMyDebugLog=true
`)

We can sleep, waiting for something:

await driver.sleep(2000)

and we can scan the console logs:

const logs = await driver.manage().logs().get(logging.Type.BROWSER)
for (const log of logs) {
  const message = log.message
  if (message.includes("my-log-data")) {
    console.log(message)
  }
}

Eventually to close the browser, use quit:

await driver.quit()