What Is Web Scraping?


Web scraping is an approach for extracting data and information from websites that do not have an API. Web scraping automatically extracts data and presents it in a format a user can easily make sense of.

This tutorial will show you how to scrape websites with Ruby and Headless Chrome, using Selenium WebDriver.

Prerequisites

Selenium WebDriver

Open the command prompt on your operating system and execute the command below.

gem install selenium-webdriver

(If you already have the gem installed make sure you have version 3.4.1 or higher.)

ChromeDriver

Download and install the latest version of ChromeDriver

Unzip Chromedriver and add the executable to /usr/local/bin

Or if using Homebrew.

brew install chromedriver

Below is a list of finders used with Ruby and Selenium WebDriver. For the examples, I will mostly be using :class but any of the finders can be used if they are present in the websites HTML you are scraping.

FINDERS =

  :class             => 'ClassName',
  :class_name        => 'ClassName',
  :css               => 'CssSelector',
  :id                => 'Id',
  :link              => 'LinkText',
  :link_text         => 'LinkText',
  :name              => 'Name',
  :partial_link_text => 'PartialLinkText',
  :tag_name          => 'TagName',
  :xpath             => 'Xpath'

Extracting a Web Page Title and URL

require 'selenium-webdriver'

options = Selenium::WebDriver::Chrome::Options.new(args: ['headless'])

driver = Selenium::WebDriver.for(:chrome, options: options)

driver.get('http://www.bbc.co.uk/')

puts driver.title

puts driver.current_url

driver.quit 

The first example script demonstrates the fundamentals of web scraping. Navigating to the website and extracting data from the HTML Using the #title and #current_url methods to return the page title ‘BBC - Home’ and the URL ‘https://www.bbc.co.uk/’

Copy and paste the code and try changing the URL to a different website. To see other titles and the URL.

Extracting Data

This next example scrapes the title of the most recent post on the Guardian Football League blog.

require 'selenium-webdriver'

options = Selenium::WebDriver::Chrome::Options.new(args: ['headless'])

driver = Selenium::WebDriver.for(:chrome, options: options)

driver.get('https://www.theguardian.com/football/football-league-blog')

element = driver.find_element(:class, 'js-headline-text')

puts element.text.strip

driver.quit

The #find_element method selects a single element with the given class. In the example above this syntax is written as

element = driver.find_element(:class, 'js-headline-text')

The #text method returns the text content inside the element, in this case the title of the latest blog entry.

But what if we wanted to return all blog entries on the page with the class name of ‘js-headline-text’

require 'selenium-webdriver'

options = Selenium::WebDriver::Chrome::Options.new(args: ['headless'])

driver = Selenium::WebDriver.for(:chrome, options: options)

driver.get('https://www.theguardian.com/football/football-league-blog')

element = driver.find_elements(:class, 'js-headline-text')

element .each do |articles| 

puts articles.text.strip

end

driver.quit

The #find_elements method selects all elements on the page with the given class. In the example above this syntax is written as.

element = driver.find_elements(:class, 'js-headline-text')

Next we need to use the each iterator to loop through the page and return all elements with the class name of ‘js-headline-text’

element .each do |articles| 

puts articles.text.strip

Filling in a form

The next example shows how to fill out a search box and click a link. This script will search the imdb.com website for the film The Shawshank Redemption and return a list of the films cast and the character that they play.

require 'selenium-webdriver'

options = Selenium::WebDriver::Chrome::Options.new(args: ['headless'])

driver = Selenium::WebDriver.for(:chrome, options: options)

driver.get('http://www.imdb.com')

element = driver.find_element(name: 'q')

element.send_keys('shawshank redemption')

element.submit

driver.find_element(link_text: 'The Shawshank Redemption').click

puts driver.title

cast = driver.find_element(:class, 'cast_list')

puts cast.text.strip

driver.quit

More Than One Element

What if we needed to scrape more than one element and also group them together?

The next example shows how to handle more than one element and also merge them together using the .zip method.

The script will scrape the title and description of trophies for the Playstation 4 game The Witcher 3: Wild Hunt - Game Of The Year Edition and merge them together in a new array called data_sets using the .zip method

require 'selenium-webdriver'
options = Selenium::WebDriver::Chrome::Options.new(args: ['headless'])

driver = Selenium::WebDriver.for(:chrome, options: options)

driver.get('https://www.playstationtrophies.org/game/the-witcher-3-goty/trophies/')

puts driver.title

trophies = driver.find_elements(:class, 'link_ach')
descriptions = driver.find_elements(:class, 'ac3')
data_sets = trophies.zip(descriptions)

data_sets.each do |trophy, description|

puts trophy.text.strip, description.text.strip
end

driver.quit

Try it yourself

Give it a try. Write your own scripts using Ruby and Selenium WebDriver. Scraping is a great and easy way to learn coding skills.


Russell Morley

Test Lead | Software Developer In Test | Automation Enthusiast