Scraping Website Using Python

5 library recommended for scraping website,

https://elitedatascience.com/python-web-scraping-libraries

• The Farm: Requests
• The Stew: Beautiful Soup 4
• The Salad: lxml
• The Restaurant: Selenium
• The Chef: Scrapy

Resources

• Requests Quickstart Guide – Official documentation. Covers practical topics like passing parameters, handling responses, and configuring headers.

• Beautiful Soup Documentation – Includes convenient quickstart guide.
• Really Short Example – Short example of using Beautiful Soup and Requests together.

• lxml Documentation – Official documentation.
• HTML Scraping with lxml and Requests – Short and sweet tutorial on pulling a webpage with Requests and then using XPath selectors to mine the desired data. This is more beginner-friendly than the official documentation.

• Selenium with Python – Documentation for Selenium’s Python bindings.
• Webscraping with Selenium – Excellent, thorough 3-part tutorial for scraping websites with Selenium.
• Scraping Hotel Prices – Code snippet for scraping hotel prices using Selenium and lxml.

• Scrapy Documentation - Official site with links to many other resources.
• Extracting data from websites with Scrapy - Detailed tutorial for scraping an e-commerce site using Scrapy.
• Scrapinghub - Cloud-based crawling service by the creators of Scrapy. The first cloud unit is free.