diff --git a/.ipynb_checkpoints/lab-web-scraping-checkpoint.ipynb b/.ipynb_checkpoints/lab-web-scraping-checkpoint.ipynb new file mode 100644 index 0000000..13f5d0b --- /dev/null +++ b/.ipynb_checkpoints/lab-web-scraping-checkpoint.ipynb @@ -0,0 +1,708 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "7e7a1ab8-2599-417d-9a65-25ef07f3a786", + "metadata": { + "id": "7e7a1ab8-2599-417d-9a65-25ef07f3a786" + }, + "source": [ + "# Lab | Web Scraping" + ] + }, + { + "cell_type": "markdown", + "id": "ce8882fc-4815-4567-92fa-b4816358ba7d", + "metadata": { + "id": "ce8882fc-4815-4567-92fa-b4816358ba7d" + }, + "source": [ + "Welcome to the \"Books to Scrape\" Web Scraping Adventure Lab!\n", + "\n", + "**Objective**\n", + "\n", + "In this lab, we will embark on a mission to unearth valuable insights from the data available on Books to Scrape, an online platform showcasing a wide variety of books. As data analyst, you have been tasked with scraping a specific subset of book data from Books to Scrape to assist publishing companies in understanding the landscape of highly-rated books across different genres. Your insights will help shape future book marketing strategies and publishing decisions.\n", + "\n", + "**Background**\n", + "\n", + "In a world where data has become the new currency, businesses are leveraging big data to make informed decisions that drive success and profitability. The publishing industry, much like others, utilizes data analytics to understand market trends, reader preferences, and the performance of books based on factors such as genre, author, and ratings. Books to Scrape serves as a rich source of such data, offering detailed information about a diverse range of books, making it an ideal platform for extracting insights to aid in informed decision-making within the literary world.\n", + "\n", + "**Task**\n", + "\n", + "Your task is to create a Python script using BeautifulSoup and pandas to scrape Books to Scrape book data, focusing on book ratings and genres. The script should be able to filter books with ratings above a certain threshold and in specific genres. Additionally, the script should structure the scraped data in a tabular format using pandas for further analysis.\n", + "\n", + "**Expected Outcome**\n", + "\n", + "A function named `scrape_books` that takes two parameters: `min_rating` and `max_price`. The function should scrape book data from the \"Books to Scrape\" website and return a `pandas` DataFrame with the following columns:\n", + "\n", + "**Expected Outcome**\n", + "\n", + "- A function named `scrape_books` that takes two parameters: `min_rating` and `max_price`.\n", + "- The function should return a DataFrame with the following columns:\n", + " - **UPC**: The Universal Product Code (UPC) of the book.\n", + " - **Title**: The title of the book.\n", + " - **Price (£)**: The price of the book in pounds.\n", + " - **Rating**: The rating of the book (1-5 stars).\n", + " - **Genre**: The genre of the book.\n", + " - **Availability**: Whether the book is in stock or not.\n", + " - **Description**: A brief description or product description of the book (if available).\n", + " \n", + "You will execute this script to scrape data for books with a minimum rating of `4.0 and above` and a maximum price of `£20`. \n", + "\n", + "Remember to experiment with different ratings and prices to ensure your code is versatile and can handle various searches effectively!\n", + "\n", + "**Resources**\n", + "\n", + "- [Beautiful Soup Documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)\n", + "- [Pandas Documentation](https://pandas.pydata.org/pandas-docs/stable/index.html)\n", + "- [Books to Scrape](https://books.toscrape.com/)\n" + ] + }, + { + "cell_type": "markdown", + "id": "3519921d-5890-445b-9a33-934ed8ee378c", + "metadata": { + "id": "3519921d-5890-445b-9a33-934ed8ee378c" + }, + "source": [ + "**Hint**\n", + "\n", + "Your first mission is to familiarize yourself with the **Books to Scrape** website. Navigate to [Books to Scrape](http://books.toscrape.com/) and explore the available books to understand their layout and structure. \n", + "\n", + "Next, think about how you can set parameters for your data extraction:\n", + "\n", + "- **Minimum Rating**: Focus on books with a rating of 4.0 and above.\n", + "- **Maximum Price**: Filter for books priced up to £20.\n", + "\n", + "After reviewing the site, you can construct a plan for scraping relevant data. Pay attention to the details displayed for each book, including the title, price, rating, and availability. This will help you identify the correct HTML elements to target with your scraping script.\n", + "\n", + "Make sure to build your scraping URL and logic based on the patterns you observe in the HTML structure of the book listings!" + ] + }, + { + "cell_type": "markdown", + "id": "25a83a0d-a742-49f6-985e-e27887cbf922", + "metadata": { + "id": "25a83a0d-a742-49f6-985e-e27887cbf922" + }, + "source": [ + "\n", + "---\n", + "\n", + "**Best of luck! Immerse yourself in the world of books, and may the data be with you!**" + ] + }, + { + "cell_type": "markdown", + "id": "7b75cf0d-9afa-4eec-a9e2-befeac68b2a0", + "metadata": { + "id": "7b75cf0d-9afa-4eec-a9e2-befeac68b2a0" + }, + "source": [ + "**Important Note**:\n", + "\n", + "In the fast-changing online world, websites often update and change their structures. When you try this lab, the **Books to Scrape** website might differ from what you expect.\n", + "\n", + "If you encounter issues due to these changes, like new rules or obstacles preventing data extraction, don’t worry! Get creative.\n", + "\n", + "You can choose another website that interests you and is suitable for scraping data. Options like Wikipedia, The New York Times, or even library databases are great alternatives. The main goal remains the same: extract useful data and enhance your web scraping skills while exploring a source of information you enjoy. This is your opportunity to practice and adapt to different web environments!" + ] + }, + { + "cell_type": "markdown", + "id": "9030b080-c521-4c40-bb60-0018fe46e201", + "metadata": {}, + "source": [ + "Expected Outcome\n", + "\n", + "A function named scrape_books that takes two parameters: min_rating and max_price. The function should scrape book data from the \"Books to Scrape\" website and return a pandas DataFrame with the following columns:\n", + "\n", + "Expected Outcome\n", + "\n", + "A function named scrape_books that takes two parameters: min_rating and max_price.\n", + "The function should return a DataFrame with the following columns:\n", + "\n", + " UPC: The Universal Product Code (UPC) of the book.\n", + " Title: The title of the book.\n", + " Price (£): The price of the book in pounds.\n", + " Rating: The rating of the book (1-5 stars).\n", + " Genre: The genre of the book.\n", + " Availability: Whether the book is in stock or not.\n", + " Description: A brief description or product description of the book (if available).\n", + "You will execute this script to scrape data for books with a minimum rating of 4.0 and above and a maximum price of £20.\n", + "\n", + "Remember to experiment with different ratings and prices to ensure your code is versatile and can handle various searches effectively!" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "40359eee-9cd7-4884-bfa4-83344c222305", + "metadata": { + "id": "40359eee-9cd7-4884-bfa4-83344c222305" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " Book URL\n", + "0 https://books.toscrape.com/catalogue/catalogue...\n", + "1 https://books.toscrape.com/catalogue/catalogue...\n", + "2 https://books.toscrape.com/catalogue/catalogue...\n", + "3 https://books.toscrape.com/catalogue/catalogue...\n", + "4 https://books.toscrape.com/catalogue/catalogue...\n", + "5 https://books.toscrape.com/catalogue/catalogue...\n", + "6 https://books.toscrape.com/catalogue/catalogue...\n", + "7 https://books.toscrape.com/catalogue/catalogue...\n", + "8 https://books.toscrape.com/catalogue/catalogue...\n", + "9 https://books.toscrape.com/catalogue/catalogue...\n", + "10 https://books.toscrape.com/catalogue/catalogue...\n", + "11 https://books.toscrape.com/catalogue/catalogue...\n", + "12 https://books.toscrape.com/catalogue/catalogue...\n", + "13 https://books.toscrape.com/catalogue/catalogue...\n", + "14 https://books.toscrape.com/catalogue/catalogue...\n", + "15 https://books.toscrape.com/catalogue/catalogue...\n", + "16 https://books.toscrape.com/catalogue/catalogue...\n", + "17 https://books.toscrape.com/catalogue/catalogue...\n", + "18 https://books.toscrape.com/catalogue/catalogue...\n", + "19 https://books.toscrape.com/catalogue/catalogue...\n", + "20 https://books.toscrape.com/catalogue/in-her-wa...\n", + "21 https://books.toscrape.com/catalogue/how-music...\n", + "22 https://books.toscrape.com/catalogue/foolproof...\n", + "23 https://books.toscrape.com/catalogue/chase-me-...\n", + "24 https://books.toscrape.com/catalogue/black-dus...\n", + "25 https://books.toscrape.com/catalogue/birdsong-...\n", + "26 https://books.toscrape.com/catalogue/americas-...\n", + "27 https://books.toscrape.com/catalogue/aladdin-a...\n", + "28 https://books.toscrape.com/catalogue/worlds-el...\n", + "29 https://books.toscrape.com/catalogue/wall-and-...\n", + "30 https://books.toscrape.com/catalogue/the-four-...\n", + "31 https://books.toscrape.com/catalogue/the-five-...\n", + "32 https://books.toscrape.com/catalogue/the-eleph...\n", + "33 https://books.toscrape.com/catalogue/the-bear-...\n", + "34 https://books.toscrape.com/catalogue/sophies-w...\n", + "35 https://books.toscrape.com/catalogue/penny-may...\n", + "36 https://books.toscrape.com/catalogue/maude-188...\n", + "37 https://books.toscrape.com/catalogue/in-a-dark...\n", + "38 https://books.toscrape.com/catalogue/behind-cl...\n", + "39 https://books.toscrape.com/catalogue/you-cant-...\n" + ] + } + ], + "source": [ + "# making url list \n", + "import requests\n", + "#catalogue_base = \"https://books.toscrape.com/catalogue/\" \n", + "#a-light-in-the-attic_1000/index.html\n", + "#base_url = \"https://books.toscrape.com\"\n", + "\n", + "\n", + "#print(soup.prettify()) #<= MAKES THE HIEARCHY READABLE\n", + "\n", + "import requests\n", + "from bs4 import BeautifulSoup\n", + "import pandas as pd\n", + "\n", + "# Define base URL for the main site\n", + "base_url = 'https://books.toscrape.com/index.html'\n", + "# Define base URL for accessing book details\n", + "catalogue_base = 'https://books.toscrape.com/catalogue/'\n", + "# Initialize a list to store the complete URLs of the books\n", + "book_urls = []\n", + "\n", + "# Start with the main index page\n", + "page_url = base_url\n", + "\n", + "# Loop through pages until no more \"next\" button exists\n", + "while page_url:\n", + " # Send an HTTP request to the current page\n", + " response = requests.get(page_url)\n", + " soup = BeautifulSoup(response.content, 'html.parser')\n", + " \n", + " # Extract all book links from the current page\n", + " for h3 in soup.find_all('h3'):\n", + " relative_link = h3.find('a')['href']\n", + " # Construct the full URL\n", + " full_link = catalogue_base + relative_link.replace('../../../', '')\n", + " book_urls.append(full_link)\n", + " \n", + " # Check for the presence of the \"next\" button\n", + " next_button = soup.select_one('li.next a')\n", + " if next_button:\n", + " # Construct the next page URL correctly\n", + " next_page = next_button.get('href')\n", + " page_url = base_url.rsplit('/', 1)[0] + '/' + next_page # Ensure correct concatenation for base URL parts\n", + " else:\n", + " page_url = None\n", + "\n", + "# Convert the list of book URLs into a DataFrame for easy handling\n", + "book_urls_df = pd.DataFrame(book_urls, columns=['Book URL'])\n", + "\n", + "# Print the DataFrame of book URLs\n", + "print(book_urls_df)\n", + "#---------------------------------------------------------------------------------------\n", + "#two parameters: min_rating and max_price\n", + "#def scrape_books(soup):" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "43bb13eb-5a34-4fa9-b68a-4c68217ac054", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "books_data = []\n", + "\n", + "# Iterate over each URL in the 'Book URL' column of the DataFrame\n", + "for url in book_urls_df['Book URL']:\n", + " response = requests.get(url)\n", + " soup = BeautifulSoup(response.content, 'html.parser')\n", + " \n", + " # Extract the desired book information\n", + " title_element = soup.find('h1')\n", + " title = title_element.text.strip() if title_element else 'No title found'\n", + "\n", + " # Description\n", + " meta_description = soup.find('meta', attrs={'name': 'description'})\n", + " description = meta_description['content'].strip() if meta_description else 'No description available'\n", + "\n", + " # Price\n", + " price_element = soup.find('p', class_='price_color')\n", + " price = float(price_element.text[1:]) if price_element else None\n", + "\n", + " # Rating\n", + " rating_element = soup.find('p', class_='star-rating')\n", + " if rating_element:\n", + " rating_class = rating_element.get('class')\n", + " rating = rating_class[1] if len(rating_class) > 1 else \"Not Rated\"\n", + " else:\n", + " rating = \"Not Rated\"\n", + " rating_map = {'One': 1, 'Two': 2, 'Three': 3, 'Four': 4, 'Five': 5}\n", + " numerical_rating = rating_map.get(rating, 0)\n", + "\n", + " # Genre => error, so ignoring it for now \n", + " # breadcrumb_elements = soup.find('ul', class_='breadcrumb').find_all('a')\n", + " #genre = breadcrumb_elements[-1].text.strip() if breadcrumb_elements else 'No genre found'\n", + "\n", + " # Availability\n", + " availability_element = soup.find('p', class_='instock availability')\n", + " availability = availability_element.text.strip() if availability_element else 'Unknown'\n", + "\n", + " # UPC\n", + " upc_element = soup.find('th', string='UPC')\n", + " upc = upc_element.find_next_sibling('td').text.strip() if upc_element else 'No UPC found'\n", + "\n", + " # Append book data to list\n", + " books_data.append({\n", + " 'Title': title,\n", + " 'Description': description,\n", + " 'Price': price,\n", + " 'Rating': numerical_rating,\n", + "# 'Genre': genre,\n", + " 'Availability': availability,\n", + " 'UPC': upc,\n", + " 'URL': url\n", + " })\n", + "\n", + "# Create a DataFrame using the collected book data\n", + "book_df = pd.DataFrame(books_data)\n", + "\n", + "# Display resulting DataFrame\n", + "book_df.tail" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "3f46de88-33f7-47ca-beb3-a153342f53ac", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
UPCTitlePriceRatingAvailabilityDescription
306258a1f6a6dcfe50The Four Agreements: A Practical Guide to Pers...17.665In stock (18 available)In The Four Agreements, don Miguel Ruiz reveal...
346be3beb0793a53e7Sophie's World15.945In stock (18 available)A page-turning novel that is also an explorati...
\n", + "
" + ], + "text/plain": [ + " UPC Title \\\n", + "30 6258a1f6a6dcfe50 The Four Agreements: A Practical Guide to Pers... \n", + "34 6be3beb0793a53e7 Sophie's World \n", + "\n", + " Price Rating Availability \\\n", + "30 17.66 5 In stock (18 available) \n", + "34 15.94 5 In stock (18 available) \n", + "\n", + " Description \n", + "30 In The Four Agreements, don Miguel Ruiz reveal... \n", + "34 A page-turning novel that is also an explorati... " + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import pandas as pd\n", + "\n", + "def scrape_books(min_rating, max_price):\n", + " \n", + " filtered_books = book_df[\n", + " (book_df['Rating'] >= min_rating) & \n", + " (book_df['Price'] <= max_price)\n", + " ]\n", + " return filtered_books[['UPC', 'Title', 'Price', 'Rating', 'Availability', 'Description']]\n", + "\n", + "min_rating = 4.0\n", + "max_price = 20.0\n", + "\n", + "filtered_books_df = scrape_books(min_rating, max_price)\n", + "\n", + "display(filtered_books_df)" + ] + }, + { + "cell_type": "markdown", + "id": "7509d106-7b57-413d-9b44-b1026c721a35", + "metadata": {}, + "source": [ + "def scrape_product_details(product_url):\n", + " # Send a request to the book's specific URL\n", + " response = requests.get(product_url)\n", + " soup = BeautifulSoup(response.content, \"html.parser\")\n", + " \n", + " # --- Get Core Information (No checks needed) ---\n", + " title_text = soup.find('h1').text.strip()\n", + " price_text = soup.find('p', class_='price_color').text.strip() # Includes currency/symbols\n", + " \n", + " # Find the rating class (e.g., 'star-rating Three')\n", + " rating_classes = soup.find('p', class_='star-rating').get('class')\n", + " rating_text = rating_classes[1]\n", + " \n", + " # Find the Description (simplified)\n", + " # The description text is the

tag that follows the

with id='product_description'\n", + " description_tag = soup.find('div', id='product_description').find_next_sibling('p')\n", + " description_text = description_tag.text.strip()\n", + " \n", + " # --- Get the Table Data ---\n", + " table_data = {}\n", + " # Find the entire table\n", + " product_table = soup.find('table', class_='table-striped')\n", + " \n", + " # Loop through every row () in the table\n", + " for row in product_table.find_all('tr'):\n", + " key = row.find('th').text.strip()\n", + " value = row.find('td').text.strip()\n", + " table_data[key] = value\n", + "\n", + " # --- Append to Global Lists ---\n", + " Title.append(title_text)\n", + " Price.append(price_text)\n", + " Rating.append(rating_text)\n", + " \n", + " # Extract Category from the breadcrumb/link (a simple way)\n", + " category_from_link = product_url.split('/')[-3]\n", + " Category.append(category_from_link)\n", + " \n", + " # Append data from the table (assumes 'UPC' and 'Availability' keys exist)\n", + " UPC.append(table_data['UPC'])\n", + " Availability.append(table_data['Availability'])\n", + " Description.append(description_text)\n", + "\n", + "\n", + "# 4. Main Execution: Get Links from the Homepage\n", + "print(\"--- Step 1: Getting Links from Homepage ---\")\n", + "\n", + "# Request the main page\n", + "homepage_response = requests.get(url)\n", + "homepage_soup = BeautifulSoup(homepage_response.content, \"html.parser\")\n", + "\n", + "# Find all 20 book links on the homepage\n", + "for h3_tag in homepage_soup.find_all('h3'):\n", + " link_tag = h3_tag.find('a')\n", + " relative_link = link_tag.get('href')\n", + " \n", + " # Construct the full URL\n", + " full_link = url + 'catalogue/' + relative_link\n", + " \n", + " print(f\"-> Scraping book details for: {full_link.split('/')[-2]}\")\n", + " \n", + " # Call the function to scrape the details and fill the lists\n", + " scrape_product_details(full_link)\n", + "\n", + "print(f\"\\n✅ Scraping Complete! Total books saved: {len(Title)}\")\n", + "\n", + "# 5. Simple Display of the First Book's Data\n", + "print(\"\\n--- Results for the First Book ---\")\n", + "print(f\"Title: {Title[0]}\")\n", + "print(f\"Price: {Price[0]}\")\n", + "print(f\"UPC: {UPC[0]}\")\n", + "print(f\"Category: {Category[0]}\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "740cdc34-9b0d-459a-8cc4-e30232dfb971", + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "66ae605a-580b-483d-9ec9-e37ac086531f", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "data_env", + "language": "python", + "name": "data_env" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.9" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/lab-web-scraping.ipynb b/lab-web-scraping.ipynb index e552783..13f5d0b 100644 --- a/lab-web-scraping.ipynb +++ b/lab-web-scraping.ipynb @@ -1,148 +1,708 @@ { - "cells": [ - { - "cell_type": "markdown", - "id": "7e7a1ab8-2599-417d-9a65-25ef07f3a786", - "metadata": { - "id": "7e7a1ab8-2599-417d-9a65-25ef07f3a786" - }, - "source": [ - "# Lab | Web Scraping" - ] - }, - { - "cell_type": "markdown", - "id": "ce8882fc-4815-4567-92fa-b4816358ba7d", - "metadata": { - "id": "ce8882fc-4815-4567-92fa-b4816358ba7d" - }, - "source": [ - "Welcome to the \"Books to Scrape\" Web Scraping Adventure Lab!\n", - "\n", - "**Objective**\n", - "\n", - "In this lab, we will embark on a mission to unearth valuable insights from the data available on Books to Scrape, an online platform showcasing a wide variety of books. As data analyst, you have been tasked with scraping a specific subset of book data from Books to Scrape to assist publishing companies in understanding the landscape of highly-rated books across different genres. Your insights will help shape future book marketing strategies and publishing decisions.\n", - "\n", - "**Background**\n", - "\n", - "In a world where data has become the new currency, businesses are leveraging big data to make informed decisions that drive success and profitability. The publishing industry, much like others, utilizes data analytics to understand market trends, reader preferences, and the performance of books based on factors such as genre, author, and ratings. Books to Scrape serves as a rich source of such data, offering detailed information about a diverse range of books, making it an ideal platform for extracting insights to aid in informed decision-making within the literary world.\n", - "\n", - "**Task**\n", - "\n", - "Your task is to create a Python script using BeautifulSoup and pandas to scrape Books to Scrape book data, focusing on book ratings and genres. The script should be able to filter books with ratings above a certain threshold and in specific genres. Additionally, the script should structure the scraped data in a tabular format using pandas for further analysis.\n", - "\n", - "**Expected Outcome**\n", - "\n", - "A function named `scrape_books` that takes two parameters: `min_rating` and `max_price`. The function should scrape book data from the \"Books to Scrape\" website and return a `pandas` DataFrame with the following columns:\n", - "\n", - "**Expected Outcome**\n", - "\n", - "- A function named `scrape_books` that takes two parameters: `min_rating` and `max_price`.\n", - "- The function should return a DataFrame with the following columns:\n", - " - **UPC**: The Universal Product Code (UPC) of the book.\n", - " - **Title**: The title of the book.\n", - " - **Price (£)**: The price of the book in pounds.\n", - " - **Rating**: The rating of the book (1-5 stars).\n", - " - **Genre**: The genre of the book.\n", - " - **Availability**: Whether the book is in stock or not.\n", - " - **Description**: A brief description or product description of the book (if available).\n", - " \n", - "You will execute this script to scrape data for books with a minimum rating of `4.0 and above` and a maximum price of `£20`. \n", - "\n", - "Remember to experiment with different ratings and prices to ensure your code is versatile and can handle various searches effectively!\n", - "\n", - "**Resources**\n", - "\n", - "- [Beautiful Soup Documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)\n", - "- [Pandas Documentation](https://pandas.pydata.org/pandas-docs/stable/index.html)\n", - "- [Books to Scrape](https://books.toscrape.com/)\n" - ] - }, - { - "cell_type": "markdown", - "id": "3519921d-5890-445b-9a33-934ed8ee378c", - "metadata": { - "id": "3519921d-5890-445b-9a33-934ed8ee378c" - }, - "source": [ - "**Hint**\n", - "\n", - "Your first mission is to familiarize yourself with the **Books to Scrape** website. Navigate to [Books to Scrape](http://books.toscrape.com/) and explore the available books to understand their layout and structure. \n", - "\n", - "Next, think about how you can set parameters for your data extraction:\n", - "\n", - "- **Minimum Rating**: Focus on books with a rating of 4.0 and above.\n", - "- **Maximum Price**: Filter for books priced up to £20.\n", - "\n", - "After reviewing the site, you can construct a plan for scraping relevant data. Pay attention to the details displayed for each book, including the title, price, rating, and availability. This will help you identify the correct HTML elements to target with your scraping script.\n", - "\n", - "Make sure to build your scraping URL and logic based on the patterns you observe in the HTML structure of the book listings!" - ] - }, + "cells": [ + { + "cell_type": "markdown", + "id": "7e7a1ab8-2599-417d-9a65-25ef07f3a786", + "metadata": { + "id": "7e7a1ab8-2599-417d-9a65-25ef07f3a786" + }, + "source": [ + "# Lab | Web Scraping" + ] + }, + { + "cell_type": "markdown", + "id": "ce8882fc-4815-4567-92fa-b4816358ba7d", + "metadata": { + "id": "ce8882fc-4815-4567-92fa-b4816358ba7d" + }, + "source": [ + "Welcome to the \"Books to Scrape\" Web Scraping Adventure Lab!\n", + "\n", + "**Objective**\n", + "\n", + "In this lab, we will embark on a mission to unearth valuable insights from the data available on Books to Scrape, an online platform showcasing a wide variety of books. As data analyst, you have been tasked with scraping a specific subset of book data from Books to Scrape to assist publishing companies in understanding the landscape of highly-rated books across different genres. Your insights will help shape future book marketing strategies and publishing decisions.\n", + "\n", + "**Background**\n", + "\n", + "In a world where data has become the new currency, businesses are leveraging big data to make informed decisions that drive success and profitability. The publishing industry, much like others, utilizes data analytics to understand market trends, reader preferences, and the performance of books based on factors such as genre, author, and ratings. Books to Scrape serves as a rich source of such data, offering detailed information about a diverse range of books, making it an ideal platform for extracting insights to aid in informed decision-making within the literary world.\n", + "\n", + "**Task**\n", + "\n", + "Your task is to create a Python script using BeautifulSoup and pandas to scrape Books to Scrape book data, focusing on book ratings and genres. The script should be able to filter books with ratings above a certain threshold and in specific genres. Additionally, the script should structure the scraped data in a tabular format using pandas for further analysis.\n", + "\n", + "**Expected Outcome**\n", + "\n", + "A function named `scrape_books` that takes two parameters: `min_rating` and `max_price`. The function should scrape book data from the \"Books to Scrape\" website and return a `pandas` DataFrame with the following columns:\n", + "\n", + "**Expected Outcome**\n", + "\n", + "- A function named `scrape_books` that takes two parameters: `min_rating` and `max_price`.\n", + "- The function should return a DataFrame with the following columns:\n", + " - **UPC**: The Universal Product Code (UPC) of the book.\n", + " - **Title**: The title of the book.\n", + " - **Price (£)**: The price of the book in pounds.\n", + " - **Rating**: The rating of the book (1-5 stars).\n", + " - **Genre**: The genre of the book.\n", + " - **Availability**: Whether the book is in stock or not.\n", + " - **Description**: A brief description or product description of the book (if available).\n", + " \n", + "You will execute this script to scrape data for books with a minimum rating of `4.0 and above` and a maximum price of `£20`. \n", + "\n", + "Remember to experiment with different ratings and prices to ensure your code is versatile and can handle various searches effectively!\n", + "\n", + "**Resources**\n", + "\n", + "- [Beautiful Soup Documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)\n", + "- [Pandas Documentation](https://pandas.pydata.org/pandas-docs/stable/index.html)\n", + "- [Books to Scrape](https://books.toscrape.com/)\n" + ] + }, + { + "cell_type": "markdown", + "id": "3519921d-5890-445b-9a33-934ed8ee378c", + "metadata": { + "id": "3519921d-5890-445b-9a33-934ed8ee378c" + }, + "source": [ + "**Hint**\n", + "\n", + "Your first mission is to familiarize yourself with the **Books to Scrape** website. Navigate to [Books to Scrape](http://books.toscrape.com/) and explore the available books to understand their layout and structure. \n", + "\n", + "Next, think about how you can set parameters for your data extraction:\n", + "\n", + "- **Minimum Rating**: Focus on books with a rating of 4.0 and above.\n", + "- **Maximum Price**: Filter for books priced up to £20.\n", + "\n", + "After reviewing the site, you can construct a plan for scraping relevant data. Pay attention to the details displayed for each book, including the title, price, rating, and availability. This will help you identify the correct HTML elements to target with your scraping script.\n", + "\n", + "Make sure to build your scraping URL and logic based on the patterns you observe in the HTML structure of the book listings!" + ] + }, + { + "cell_type": "markdown", + "id": "25a83a0d-a742-49f6-985e-e27887cbf922", + "metadata": { + "id": "25a83a0d-a742-49f6-985e-e27887cbf922" + }, + "source": [ + "\n", + "---\n", + "\n", + "**Best of luck! Immerse yourself in the world of books, and may the data be with you!**" + ] + }, + { + "cell_type": "markdown", + "id": "7b75cf0d-9afa-4eec-a9e2-befeac68b2a0", + "metadata": { + "id": "7b75cf0d-9afa-4eec-a9e2-befeac68b2a0" + }, + "source": [ + "**Important Note**:\n", + "\n", + "In the fast-changing online world, websites often update and change their structures. When you try this lab, the **Books to Scrape** website might differ from what you expect.\n", + "\n", + "If you encounter issues due to these changes, like new rules or obstacles preventing data extraction, don’t worry! Get creative.\n", + "\n", + "You can choose another website that interests you and is suitable for scraping data. Options like Wikipedia, The New York Times, or even library databases are great alternatives. The main goal remains the same: extract useful data and enhance your web scraping skills while exploring a source of information you enjoy. This is your opportunity to practice and adapt to different web environments!" + ] + }, + { + "cell_type": "markdown", + "id": "9030b080-c521-4c40-bb60-0018fe46e201", + "metadata": {}, + "source": [ + "Expected Outcome\n", + "\n", + "A function named scrape_books that takes two parameters: min_rating and max_price. The function should scrape book data from the \"Books to Scrape\" website and return a pandas DataFrame with the following columns:\n", + "\n", + "Expected Outcome\n", + "\n", + "A function named scrape_books that takes two parameters: min_rating and max_price.\n", + "The function should return a DataFrame with the following columns:\n", + "\n", + " UPC: The Universal Product Code (UPC) of the book.\n", + " Title: The title of the book.\n", + " Price (£): The price of the book in pounds.\n", + " Rating: The rating of the book (1-5 stars).\n", + " Genre: The genre of the book.\n", + " Availability: Whether the book is in stock or not.\n", + " Description: A brief description or product description of the book (if available).\n", + "You will execute this script to scrape data for books with a minimum rating of 4.0 and above and a maximum price of £20.\n", + "\n", + "Remember to experiment with different ratings and prices to ensure your code is versatile and can handle various searches effectively!" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "40359eee-9cd7-4884-bfa4-83344c222305", + "metadata": { + "id": "40359eee-9cd7-4884-bfa4-83344c222305" + }, + "outputs": [ { - "cell_type": "markdown", - "id": "25a83a0d-a742-49f6-985e-e27887cbf922", - "metadata": { - "id": "25a83a0d-a742-49f6-985e-e27887cbf922" - }, - "source": [ - "\n", - "---\n", - "\n", - "**Best of luck! Immerse yourself in the world of books, and may the data be with you!**" - ] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + " Book URL\n", + "0 https://books.toscrape.com/catalogue/catalogue...\n", + "1 https://books.toscrape.com/catalogue/catalogue...\n", + "2 https://books.toscrape.com/catalogue/catalogue...\n", + "3 https://books.toscrape.com/catalogue/catalogue...\n", + "4 https://books.toscrape.com/catalogue/catalogue...\n", + "5 https://books.toscrape.com/catalogue/catalogue...\n", + "6 https://books.toscrape.com/catalogue/catalogue...\n", + "7 https://books.toscrape.com/catalogue/catalogue...\n", + "8 https://books.toscrape.com/catalogue/catalogue...\n", + "9 https://books.toscrape.com/catalogue/catalogue...\n", + "10 https://books.toscrape.com/catalogue/catalogue...\n", + "11 https://books.toscrape.com/catalogue/catalogue...\n", + "12 https://books.toscrape.com/catalogue/catalogue...\n", + "13 https://books.toscrape.com/catalogue/catalogue...\n", + "14 https://books.toscrape.com/catalogue/catalogue...\n", + "15 https://books.toscrape.com/catalogue/catalogue...\n", + "16 https://books.toscrape.com/catalogue/catalogue...\n", + "17 https://books.toscrape.com/catalogue/catalogue...\n", + "18 https://books.toscrape.com/catalogue/catalogue...\n", + "19 https://books.toscrape.com/catalogue/catalogue...\n", + "20 https://books.toscrape.com/catalogue/in-her-wa...\n", + "21 https://books.toscrape.com/catalogue/how-music...\n", + "22 https://books.toscrape.com/catalogue/foolproof...\n", + "23 https://books.toscrape.com/catalogue/chase-me-...\n", + "24 https://books.toscrape.com/catalogue/black-dus...\n", + "25 https://books.toscrape.com/catalogue/birdsong-...\n", + "26 https://books.toscrape.com/catalogue/americas-...\n", + "27 https://books.toscrape.com/catalogue/aladdin-a...\n", + "28 https://books.toscrape.com/catalogue/worlds-el...\n", + "29 https://books.toscrape.com/catalogue/wall-and-...\n", + "30 https://books.toscrape.com/catalogue/the-four-...\n", + "31 https://books.toscrape.com/catalogue/the-five-...\n", + "32 https://books.toscrape.com/catalogue/the-eleph...\n", + "33 https://books.toscrape.com/catalogue/the-bear-...\n", + "34 https://books.toscrape.com/catalogue/sophies-w...\n", + "35 https://books.toscrape.com/catalogue/penny-may...\n", + "36 https://books.toscrape.com/catalogue/maude-188...\n", + "37 https://books.toscrape.com/catalogue/in-a-dark...\n", + "38 https://books.toscrape.com/catalogue/behind-cl...\n", + "39 https://books.toscrape.com/catalogue/you-cant-...\n" + ] + } + ], + "source": [ + "# making url list \n", + "import requests\n", + "#catalogue_base = \"https://books.toscrape.com/catalogue/\" \n", + "#a-light-in-the-attic_1000/index.html\n", + "#base_url = \"https://books.toscrape.com\"\n", + "\n", + "\n", + "#print(soup.prettify()) #<= MAKES THE HIEARCHY READABLE\n", + "\n", + "import requests\n", + "from bs4 import BeautifulSoup\n", + "import pandas as pd\n", + "\n", + "# Define base URL for the main site\n", + "base_url = 'https://books.toscrape.com/index.html'\n", + "# Define base URL for accessing book details\n", + "catalogue_base = 'https://books.toscrape.com/catalogue/'\n", + "# Initialize a list to store the complete URLs of the books\n", + "book_urls = []\n", + "\n", + "# Start with the main index page\n", + "page_url = base_url\n", + "\n", + "# Loop through pages until no more \"next\" button exists\n", + "while page_url:\n", + " # Send an HTTP request to the current page\n", + " response = requests.get(page_url)\n", + " soup = BeautifulSoup(response.content, 'html.parser')\n", + " \n", + " # Extract all book links from the current page\n", + " for h3 in soup.find_all('h3'):\n", + " relative_link = h3.find('a')['href']\n", + " # Construct the full URL\n", + " full_link = catalogue_base + relative_link.replace('../../../', '')\n", + " book_urls.append(full_link)\n", + " \n", + " # Check for the presence of the \"next\" button\n", + " next_button = soup.select_one('li.next a')\n", + " if next_button:\n", + " # Construct the next page URL correctly\n", + " next_page = next_button.get('href')\n", + " page_url = base_url.rsplit('/', 1)[0] + '/' + next_page # Ensure correct concatenation for base URL parts\n", + " else:\n", + " page_url = None\n", + "\n", + "# Convert the list of book URLs into a DataFrame for easy handling\n", + "book_urls_df = pd.DataFrame(book_urls, columns=['Book URL'])\n", + "\n", + "# Print the DataFrame of book URLs\n", + "print(book_urls_df)\n", + "#---------------------------------------------------------------------------------------\n", + "#two parameters: min_rating and max_price\n", + "#def scrape_books(soup):" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "43bb13eb-5a34-4fa9-b68a-4c68217ac054", + "metadata": {}, + "outputs": [ { - "cell_type": "markdown", - "id": "7b75cf0d-9afa-4eec-a9e2-befeac68b2a0", - "metadata": { - "id": "7b75cf0d-9afa-4eec-a9e2-befeac68b2a0" - }, - "source": [ - "**Important Note**:\n", - "\n", - "In the fast-changing online world, websites often update and change their structures. When you try this lab, the **Books to Scrape** website might differ from what you expect.\n", - "\n", - "If you encounter issues due to these changes, like new rules or obstacles preventing data extraction, don’t worry! Get creative.\n", - "\n", - "You can choose another website that interests you and is suitable for scraping data. Options like Wikipedia, The New York Times, or even library databases are great alternatives. The main goal remains the same: extract useful data and enhance your web scraping skills while exploring a source of information you enjoy. This is your opportunity to practice and adapt to different web environments!" + "data": { + "text/plain": [ + "" ] - }, + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "books_data = []\n", + "\n", + "# Iterate over each URL in the 'Book URL' column of the DataFrame\n", + "for url in book_urls_df['Book URL']:\n", + " response = requests.get(url)\n", + " soup = BeautifulSoup(response.content, 'html.parser')\n", + " \n", + " # Extract the desired book information\n", + " title_element = soup.find('h1')\n", + " title = title_element.text.strip() if title_element else 'No title found'\n", + "\n", + " # Description\n", + " meta_description = soup.find('meta', attrs={'name': 'description'})\n", + " description = meta_description['content'].strip() if meta_description else 'No description available'\n", + "\n", + " # Price\n", + " price_element = soup.find('p', class_='price_color')\n", + " price = float(price_element.text[1:]) if price_element else None\n", + "\n", + " # Rating\n", + " rating_element = soup.find('p', class_='star-rating')\n", + " if rating_element:\n", + " rating_class = rating_element.get('class')\n", + " rating = rating_class[1] if len(rating_class) > 1 else \"Not Rated\"\n", + " else:\n", + " rating = \"Not Rated\"\n", + " rating_map = {'One': 1, 'Two': 2, 'Three': 3, 'Four': 4, 'Five': 5}\n", + " numerical_rating = rating_map.get(rating, 0)\n", + "\n", + " # Genre => error, so ignoring it for now \n", + " # breadcrumb_elements = soup.find('ul', class_='breadcrumb').find_all('a')\n", + " #genre = breadcrumb_elements[-1].text.strip() if breadcrumb_elements else 'No genre found'\n", + "\n", + " # Availability\n", + " availability_element = soup.find('p', class_='instock availability')\n", + " availability = availability_element.text.strip() if availability_element else 'Unknown'\n", + "\n", + " # UPC\n", + " upc_element = soup.find('th', string='UPC')\n", + " upc = upc_element.find_next_sibling('td').text.strip() if upc_element else 'No UPC found'\n", + "\n", + " # Append book data to list\n", + " books_data.append({\n", + " 'Title': title,\n", + " 'Description': description,\n", + " 'Price': price,\n", + " 'Rating': numerical_rating,\n", + "# 'Genre': genre,\n", + " 'Availability': availability,\n", + " 'UPC': upc,\n", + " 'URL': url\n", + " })\n", + "\n", + "# Create a DataFrame using the collected book data\n", + "book_df = pd.DataFrame(books_data)\n", + "\n", + "# Display resulting DataFrame\n", + "book_df.tail" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "3f46de88-33f7-47ca-beb3-a153342f53ac", + "metadata": {}, + "outputs": [ { - "cell_type": "code", - "execution_count": null, - "id": "40359eee-9cd7-4884-bfa4-83344c222305", - "metadata": { - "id": "40359eee-9cd7-4884-bfa4-83344c222305" - }, - "outputs": [], - "source": [ - "# Your solution goes here" + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
UPCTitlePriceRatingAvailabilityDescription
306258a1f6a6dcfe50The Four Agreements: A Practical Guide to Pers...17.665In stock (18 available)In The Four Agreements, don Miguel Ruiz reveal...
346be3beb0793a53e7Sophie's World15.945In stock (18 available)A page-turning novel that is also an explorati...
\n", + "
" + ], + "text/plain": [ + " UPC Title \\\n", + "30 6258a1f6a6dcfe50 The Four Agreements: A Practical Guide to Pers... \n", + "34 6be3beb0793a53e7 Sophie's World \n", + "\n", + " Price Rating Availability \\\n", + "30 17.66 5 In stock (18 available) \n", + "34 15.94 5 In stock (18 available) \n", + "\n", + " Description \n", + "30 In The Four Agreements, don Miguel Ruiz reveal... \n", + "34 A page-turning novel that is also an explorati... " ] + }, + "metadata": {}, + "output_type": "display_data" } - ], - "metadata": { - "colab": { - "provenance": [] - }, - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.13" - } + ], + "source": [ + "import pandas as pd\n", + "\n", + "def scrape_books(min_rating, max_price):\n", + " \n", + " filtered_books = book_df[\n", + " (book_df['Rating'] >= min_rating) & \n", + " (book_df['Price'] <= max_price)\n", + " ]\n", + " return filtered_books[['UPC', 'Title', 'Price', 'Rating', 'Availability', 'Description']]\n", + "\n", + "min_rating = 4.0\n", + "max_price = 20.0\n", + "\n", + "filtered_books_df = scrape_books(min_rating, max_price)\n", + "\n", + "display(filtered_books_df)" + ] + }, + { + "cell_type": "markdown", + "id": "7509d106-7b57-413d-9b44-b1026c721a35", + "metadata": {}, + "source": [ + "def scrape_product_details(product_url):\n", + " # Send a request to the book's specific URL\n", + " response = requests.get(product_url)\n", + " soup = BeautifulSoup(response.content, \"html.parser\")\n", + " \n", + " # --- Get Core Information (No checks needed) ---\n", + " title_text = soup.find('h1').text.strip()\n", + " price_text = soup.find('p', class_='price_color').text.strip() # Includes currency/symbols\n", + " \n", + " # Find the rating class (e.g., 'star-rating Three')\n", + " rating_classes = soup.find('p', class_='star-rating').get('class')\n", + " rating_text = rating_classes[1]\n", + " \n", + " # Find the Description (simplified)\n", + " # The description text is the

tag that follows the

with id='product_description'\n", + " description_tag = soup.find('div', id='product_description').find_next_sibling('p')\n", + " description_text = description_tag.text.strip()\n", + " \n", + " # --- Get the Table Data ---\n", + " table_data = {}\n", + " # Find the entire table\n", + " product_table = soup.find('table', class_='table-striped')\n", + " \n", + " # Loop through every row () in the table\n", + " for row in product_table.find_all('tr'):\n", + " key = row.find('th').text.strip()\n", + " value = row.find('td').text.strip()\n", + " table_data[key] = value\n", + "\n", + " # --- Append to Global Lists ---\n", + " Title.append(title_text)\n", + " Price.append(price_text)\n", + " Rating.append(rating_text)\n", + " \n", + " # Extract Category from the breadcrumb/link (a simple way)\n", + " category_from_link = product_url.split('/')[-3]\n", + " Category.append(category_from_link)\n", + " \n", + " # Append data from the table (assumes 'UPC' and 'Availability' keys exist)\n", + " UPC.append(table_data['UPC'])\n", + " Availability.append(table_data['Availability'])\n", + " Description.append(description_text)\n", + "\n", + "\n", + "# 4. Main Execution: Get Links from the Homepage\n", + "print(\"--- Step 1: Getting Links from Homepage ---\")\n", + "\n", + "# Request the main page\n", + "homepage_response = requests.get(url)\n", + "homepage_soup = BeautifulSoup(homepage_response.content, \"html.parser\")\n", + "\n", + "# Find all 20 book links on the homepage\n", + "for h3_tag in homepage_soup.find_all('h3'):\n", + " link_tag = h3_tag.find('a')\n", + " relative_link = link_tag.get('href')\n", + " \n", + " # Construct the full URL\n", + " full_link = url + 'catalogue/' + relative_link\n", + " \n", + " print(f\"-> Scraping book details for: {full_link.split('/')[-2]}\")\n", + " \n", + " # Call the function to scrape the details and fill the lists\n", + " scrape_product_details(full_link)\n", + "\n", + "print(f\"\\n✅ Scraping Complete! Total books saved: {len(Title)}\")\n", + "\n", + "# 5. Simple Display of the First Book's Data\n", + "print(\"\\n--- Results for the First Book ---\")\n", + "print(f\"Title: {Title[0]}\")\n", + "print(f\"Price: {Price[0]}\")\n", + "print(f\"UPC: {UPC[0]}\")\n", + "print(f\"Category: {Category[0]}\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "740cdc34-9b0d-459a-8cc4-e30232dfb971", + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "66ae605a-580b-483d-9ec9-e37ac086531f", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "data_env", + "language": "python", + "name": "data_env" }, - "nbformat": 4, - "nbformat_minor": 5 + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.9" + } + }, + "nbformat": 4, + "nbformat_minor": 5 }