Since I’ve moved to England, I started to use ebay much more. Super useful platform and a great platform for me to extend my web scraping portfolio on to. Besides obviously data, I love sports, cycling in particular. But since the weather in England is not particularly inspirational for cycling most of the time, I sometimes struggle to go out but then at the same time I truly miss it. So, in part II of web scraping, I will scrape data from ebay with ‘requests’, ‘BeautifulSoup’ & ‘selenium’. I will also use ‘time’ to show how you can control your web scraping.
As always, I start with importing the libraries I need in order to scrape the data from ebay. I also import ‘pandas’ so that I’m able to build a dataframe with the scraped data.
from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd
import time
In order to use selenium you need to download the webdriver. More information about how to do that can be found on: https://selenium-python.readthedocs.io/installation.html
driver = webdriver.Chrome()
Sidenote I: install selenium and test
To make sure it works, you can test your webdriver by run a small script where you open your driver and close it (with time)
import time
from selenium import webdriver
driver = webdriver.Chrome()
driver.get(‘https://www.google.com’)
time.sleep(5)
driver.quit()
The (5) is needed to actually see the google page starting up and loading before it quits.
So if you know that your driver works you can continue building your web scraper.
driver = webdriver.Chrome()
driver.get("copy-paste-your-ebay-URL")
time.sleep(5)
Build a for loop to web scrape the items that you want. I want to have the products, prices and links per product.
products = []
prices = []
link = []
content = driver.page_source
soup = BeautifulSoup(content, "lxml")
product = soup.findAll('h3', attrs={'class': 's-item__title'})
for a in product:
product = a.text
products.append(product)
price = soup.findAll('span', attrs={'class' : 's-item__price'})
for b in price:
price = b.text
prices.append(price)
for c in soup.findAll('a',{'class':'s-item__link'}):
links = c.attrs.get("href")
link.append(links)
In case you get empty values, which can be an issue transforming it into a data set, set a default value:
default_value = ""
When you have done that, you can use pd (Pandas) to transform your data into a dataframe, so you can actually work with it.
Because the amount of data per item was different in my case, I decided to put the data into two dataframes to find out what’s going on.
df3 = pd.DataFrame({'Product Name': products, 'Links': link})
df4 = pd.DataFrame({'Price' : prices})
df3.apply(lambda x: x if x is not None else default_value)
df4.apply(lambda x: x if x is not None else default_value)
df3.to_csv('/Users/XXX/XXX/products1.csv', index=False, encoding='utf-8')
df4.to_csv('/Users/XXX/XXX/products2.csv', index=False, encoding='utf-8')
Sidenote II:
In my case there was a price popping up in my dataset that was not in the initial results list on ebay. I deleted that one and merged two datasets into one.
Use time
The library time is a multifunctional library that not only can be used for testing whether your webdriver works.
In order to scrape the webpage for example twice a day, you can set up a timer with time. Sixty means 60 seconds, so you just need to calculate the amount of seconds you want in between scraping your page. To make sure the timer stops, include a timer when you want it to end.
#add time
time.sleep(60) #it runs every 60 seconds
end_timer = time.time() + 60 * 2
while time.time() < end_timer:
soup()
Once you have saved your data into a csv, you can basically do everything with it you would to normally. Running this script provides you with your data needed. But also realize that you don’t get the data if you don’t run it. This brings me on to the next topic, automation. More about that can be found here.
Cheers,
Laura