eBay is an online marketplace that enables users to buy and sell various goods and services. eBay has been operating since 1995, and It is one of the first companies to offer online auctions.
What is the use of eBay listing data?
eBay is one of the largest online marketplaces in the US and EU regions. It has around 187 Million users worldwide and sold worth $27.5 Billion dollars of items in 2021.
You can use the data of eBay listing to monitor the used market trends and price intelligence that can help you make your purchasing decision. There are many uses of this data for individuals to businesses.
Install the script and run the Python code:
1: Download the scraper code from our git project and put it inside the folder.
Git URL:
2: Run the command. It will install all the required dependencies for the scraper.
pip3 install -r requirments.txt 3: Now run the command:
python3 scraper.pyIt asks you enter the product you want to search for. It prints the scraped data in the terminal in JSON format and outputs in the file named listing.json as well in the same folder.
How to scrape eBay listing data using Python with Proxies
You can use many ways to crawl the data from the organic results of eBay. Fetch use any paid scraper API is limited to the quota, which is an expensive option.
Other way, Around is to use the private proxies to crawl the data from the eBay search results.
This option I use private proxies from Proxy-Hub.com and the fixed the cost of the proxy, and you can scrape tons of data without have to think about any specific limits. Without further delay, Let’s dive into the actual task.
from bs4 import BeautifulSoup
import json
import cloudscraper
import os
## Input from the user in the terminal
product_serach = input('Enter Product name you want to get data==>> ')
## This will fetch the current path of this script that we will use to write the json file
current_path = os.path.dirname(os.path.realpath(__file__))
#print(current_path)
# Proxy list defined
proxy = {
"http": "http://USERNAME:PASSWORD@IP:PORT",
"https": "http://USERNAME:PASSWORD@IP:PORT"}
# Created the cloudscraper instance
scraper = cloudscraper.create_scraper()
# Testing and printing the proxy it use in this run
proxy_test = scraper.get("http://ip.seeip.org/jsonip?",proxies=proxy).text
print(proxy_test)
def ebay_serach_result_pages_result():
ebay_listing_html = scraper.get("https://www.ebay.com/sch/i.html?_nkw="+str(product_serach),proxies=proxy).text
soup = BeautifulSoup(ebay_listing_html)
data = []
for item in soup.select('.s-item__wrapper.clearfix'):
title = item.select_one('.s-item__title').text
link = item.select_one('.s-item__link')['href']
try:
condition = item.select_one('.SECONDARY_INFO').text
except:
condition = None
try:
shipping = item.select_one('.s-item__logisticsCost').text
except:
shipping = None
try:
location = item.select_one('.s-item__itemLocation').text
except:
location = None
try:
watchers_sold = item.select_one('.NEGATIVE').text
except:
watchers_sold = None
if item.select_one('.s-item__etrs-badge-seller') is not None:
top_rated = True
else:
top_rated = False
try:
bid_count = item.select_one('.s-item__bidCount').text
except:
bid_count = None
try:
bid_time_left = item.select_one('.s-item__time-left').text
except:
bid_time_left = None
try:
reviews = item.select_one('.s-item__reviews-count span').text.split(' ')[0]
except:
reviews = None
try:
exctention_buy_now = item.select_one('.s-item__purchase-options-with-icon').text
except:
exctention_buy_now = None
try:
price = item.select_one('.s-item__price').text
except:
price = None
data.append({
'item': {'title': title, 'link': link, 'price': price},
'condition': condition,
'top_rated': top_rated,
'reviews': reviews,
'watchers_or_sold': watchers_sold,
'buy_now_extention': exctention_buy_now,
'delivery': {'shipping': shipping, 'location': location},
'bids': {'count': bid_count, 'time_left': bid_time_left},
})
# This will print the output in the terminal if you don't want that commit this line below
print(json.dumps(data, indent = 2, ensure_ascii = False))
# Output the data in json file
with open('listing.json', 'w') as file_obj:
json.dump(data,file_obj)
if __name__ == "__main__":
ebay_serach_result_pages_result()
eBay Scraper Run Results
Here is the example run for “iPhone 13 Pro” product search and i have just listed couple products response only, otherwise , it will be too long.
{
"item": {
"title": "Shop on eBay",
"link": "https://ebay.com/itm/123456?hash=item28caef0a3a:g:E3kAAOSwlGJiMikD&amdata=enc%3AAQAHAAAAsJoWXGf0hxNZspTmhb8%2FTJCCurAWCHuXJ2Xi3S9cwXL6BX04zSEiVaDMCvsUbApftgXEAHGJU1ZGugZO%2FnW1U7Gb6vgoL%2BmXlqCbLkwoZfF3AUAK8YvJ5B4%2BnhFA7ID4dxpYs4jjExEnN5SR2g1mQe7QtLkmGt%2FZ%2FbH2W62cXPuKbf550ExbnBPO2QJyZTXYCuw5KVkMdFMDuoB4p3FwJKcSPzez5kyQyVjyiIq6PB2q%7Ctkp%3ABlBMULq7kqyXYA",
"price": "$20.00"
},
"condition": "Brand New",
"top_rated": false,
"reviews": null,
"watchers_or_sold": null,
"buy_now_extention": null,
"delivery": {
"shipping": null,
"location": null
},
"bids": {
"count": null,
"time_left": null
}
},
{
"item": {
"title": "NEW SEAL Apple iPhone 13 Pro Max 256GB Graphite/Blue (AT&T/CRICKET)",
"link": "https://www.ebay.com/itm/334542602686?epid=24049280232&hash=item4de44b01be:g:0P0AAOSw-uVi2chs&amdata=enc%3AAQAHAAAA4Nlb7WH6GsfvrCRTkhx%2FseyFEpa3bY1NcuruqI8Hl%2BMDIzWwQXbP86gdc1b0okY1878Mt8ILpPvbwOkRJeZrBH3e5%2B6eO4OO01vcqLiwT29jZIECDX56vksJUNn1LP5n3R0YosI58tRB1mkto9aEMsmuT56pZBCE%2B37TPXpQO8mdE1iEKqPeZQgcexQ%2BECDiy8EZLV6IMjQIhNYTk8XS6W1futhXpWBquoCoddSXuuRbayJs5QQFEjSazt0VAjWSiegESxTH59IkVwU457A54nKuF4i9gV2508nmVWf4ruaO%7Ctkp%3ABFBMjPrlofRg",
"price": "$999.00"
},
"condition": "Brand New",
"top_rated": false,
"reviews": "(3)",
"watchers_or_sold": null,
"buy_now_extention": null,
"delivery": {
"shipping": "+$33.44 shipping estimate",
"location": "from United States"
},
"bids": {
"count": null,
"time_left": null
}
},
{
"item": {
"title": "UNLOCK SERVICE all AT&T Apple iPhone 13 12 11 pro max ProMax xs x s mini 8 7 6 5",
"link": "https://www.ebay.com/itm/295213557948?hash=item44bc1968bc:g:I3EAAOSw6k5jHldc&amdata=enc%3AAQAHAAAA4Ob43v1DpNOU%2Fh96eHUyYCC9C6LQHFRoPFzQ71CfBVSh47UfkigHpsj5nTmQCwBV5Mlu1hMRaZluFlqwoiWpXESwdZatqoRYqm%2BnLAQD0haWozSGhZosIEv469QjLYRbYH9GDt0nAeVn81fDwefWg8HWD2imnO4mp0IeAwWRRlBp0wQ%2BU1vRoMVyEG%2B3UpPcdr3sJY4lMJV%2BTxSL105sNagE%2FnF%2BvDmgHE8EpwmJZfPgeKE8VxcGZYnVyhY8Qehljyx%2Frik0jyMmuBYqUBUz6hOIm1R1TwM4CE%2Fz583O67Er%7Ctkp%3ABFBMjPrlofRg",
"price": "$6.99"
},
"condition": "Brand New",
"top_rated": false,
"reviews": null,
"watchers_or_sold": null,
"buy_now_extention": null,
"delivery": {
"shipping": "Free International Shipping",
"location": "from United States"
},
"bids": {
"count": null,
"time_left": null
}
},Get eBay listing data in the JSON format
This python code fetch and put the data in the JSON file format Because this is easily readable and can be imported to any application or visualization software.
The data points it will provide you are there:
Title: It is the title of the product
Link: Direct link to the product listed
Condition: It shows you the product condition check more here.
Top-Rated: If it is top rated, then True else, False
Reviews: Number of reviews on the product listing
watchers_or_sold: Number of people watching and sold.
buy_now_extention: It shows two options: Buy It Now or Null
delivery:
shipping: Shipping fee estimated
location: Where the product seller is located
bids
count: Number of bidders
time_left: Time left for the product auction
Wrap Up:
In this article, we covered how to scrape eBay by using Python with Private Proxies. We also looked at how to use the BeautifulSoup library to parse HTML pages and extract data from them. We provide you a ebay proxies solution as well reach us at sales@proxy-hub.com or live support.
