diff --git a/.ipynb_checkpoints/lab-web-scraping-checkpoint.ipynb b/.ipynb_checkpoints/lab-web-scraping-checkpoint.ipynb new file mode 100644 index 0000000..e552783 --- /dev/null +++ b/.ipynb_checkpoints/lab-web-scraping-checkpoint.ipynb @@ -0,0 +1,148 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "7e7a1ab8-2599-417d-9a65-25ef07f3a786", + "metadata": { + "id": "7e7a1ab8-2599-417d-9a65-25ef07f3a786" + }, + "source": [ + "# Lab | Web Scraping" + ] + }, + { + "cell_type": "markdown", + "id": "ce8882fc-4815-4567-92fa-b4816358ba7d", + "metadata": { + "id": "ce8882fc-4815-4567-92fa-b4816358ba7d" + }, + "source": [ + "Welcome to the \"Books to Scrape\" Web Scraping Adventure Lab!\n", + "\n", + "**Objective**\n", + "\n", + "In this lab, we will embark on a mission to unearth valuable insights from the data available on Books to Scrape, an online platform showcasing a wide variety of books. As data analyst, you have been tasked with scraping a specific subset of book data from Books to Scrape to assist publishing companies in understanding the landscape of highly-rated books across different genres. Your insights will help shape future book marketing strategies and publishing decisions.\n", + "\n", + "**Background**\n", + "\n", + "In a world where data has become the new currency, businesses are leveraging big data to make informed decisions that drive success and profitability. The publishing industry, much like others, utilizes data analytics to understand market trends, reader preferences, and the performance of books based on factors such as genre, author, and ratings. Books to Scrape serves as a rich source of such data, offering detailed information about a diverse range of books, making it an ideal platform for extracting insights to aid in informed decision-making within the literary world.\n", + "\n", + "**Task**\n", + "\n", + "Your task is to create a Python script using BeautifulSoup and pandas to scrape Books to Scrape book data, focusing on book ratings and genres. The script should be able to filter books with ratings above a certain threshold and in specific genres. Additionally, the script should structure the scraped data in a tabular format using pandas for further analysis.\n", + "\n", + "**Expected Outcome**\n", + "\n", + "A function named `scrape_books` that takes two parameters: `min_rating` and `max_price`. The function should scrape book data from the \"Books to Scrape\" website and return a `pandas` DataFrame with the following columns:\n", + "\n", + "**Expected Outcome**\n", + "\n", + "- A function named `scrape_books` that takes two parameters: `min_rating` and `max_price`.\n", + "- The function should return a DataFrame with the following columns:\n", + " - **UPC**: The Universal Product Code (UPC) of the book.\n", + " - **Title**: The title of the book.\n", + " - **Price (£)**: The price of the book in pounds.\n", + " - **Rating**: The rating of the book (1-5 stars).\n", + " - **Genre**: The genre of the book.\n", + " - **Availability**: Whether the book is in stock or not.\n", + " - **Description**: A brief description or product description of the book (if available).\n", + " \n", + "You will execute this script to scrape data for books with a minimum rating of `4.0 and above` and a maximum price of `£20`. \n", + "\n", + "Remember to experiment with different ratings and prices to ensure your code is versatile and can handle various searches effectively!\n", + "\n", + "**Resources**\n", + "\n", + "- [Beautiful Soup Documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)\n", + "- [Pandas Documentation](https://pandas.pydata.org/pandas-docs/stable/index.html)\n", + "- [Books to Scrape](https://books.toscrape.com/)\n" + ] + }, + { + "cell_type": "markdown", + "id": "3519921d-5890-445b-9a33-934ed8ee378c", + "metadata": { + "id": "3519921d-5890-445b-9a33-934ed8ee378c" + }, + "source": [ + "**Hint**\n", + "\n", + "Your first mission is to familiarize yourself with the **Books to Scrape** website. Navigate to [Books to Scrape](http://books.toscrape.com/) and explore the available books to understand their layout and structure. \n", + "\n", + "Next, think about how you can set parameters for your data extraction:\n", + "\n", + "- **Minimum Rating**: Focus on books with a rating of 4.0 and above.\n", + "- **Maximum Price**: Filter for books priced up to £20.\n", + "\n", + "After reviewing the site, you can construct a plan for scraping relevant data. Pay attention to the details displayed for each book, including the title, price, rating, and availability. This will help you identify the correct HTML elements to target with your scraping script.\n", + "\n", + "Make sure to build your scraping URL and logic based on the patterns you observe in the HTML structure of the book listings!" + ] + }, + { + "cell_type": "markdown", + "id": "25a83a0d-a742-49f6-985e-e27887cbf922", + "metadata": { + "id": "25a83a0d-a742-49f6-985e-e27887cbf922" + }, + "source": [ + "\n", + "---\n", + "\n", + "**Best of luck! Immerse yourself in the world of books, and may the data be with you!**" + ] + }, + { + "cell_type": "markdown", + "id": "7b75cf0d-9afa-4eec-a9e2-befeac68b2a0", + "metadata": { + "id": "7b75cf0d-9afa-4eec-a9e2-befeac68b2a0" + }, + "source": [ + "**Important Note**:\n", + "\n", + "In the fast-changing online world, websites often update and change their structures. When you try this lab, the **Books to Scrape** website might differ from what you expect.\n", + "\n", + "If you encounter issues due to these changes, like new rules or obstacles preventing data extraction, don’t worry! Get creative.\n", + "\n", + "You can choose another website that interests you and is suitable for scraping data. Options like Wikipedia, The New York Times, or even library databases are great alternatives. The main goal remains the same: extract useful data and enhance your web scraping skills while exploring a source of information you enjoy. This is your opportunity to practice and adapt to different web environments!" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "40359eee-9cd7-4884-bfa4-83344c222305", + "metadata": { + "id": "40359eee-9cd7-4884-bfa4-83344c222305" + }, + "outputs": [], + "source": [ + "# Your solution goes here" + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/lab-web-scraping.ipynb b/lab-web-scraping.ipynb index e552783..35d567a 100644 --- a/lab-web-scraping.ipynb +++ b/lab-web-scraping.ipynb @@ -1,148 +1,404 @@ { - "cells": [ - { - "cell_type": "markdown", - "id": "7e7a1ab8-2599-417d-9a65-25ef07f3a786", - "metadata": { - "id": "7e7a1ab8-2599-417d-9a65-25ef07f3a786" - }, - "source": [ - "# Lab | Web Scraping" - ] - }, - { - "cell_type": "markdown", - "id": "ce8882fc-4815-4567-92fa-b4816358ba7d", - "metadata": { - "id": "ce8882fc-4815-4567-92fa-b4816358ba7d" - }, - "source": [ - "Welcome to the \"Books to Scrape\" Web Scraping Adventure Lab!\n", - "\n", - "**Objective**\n", - "\n", - "In this lab, we will embark on a mission to unearth valuable insights from the data available on Books to Scrape, an online platform showcasing a wide variety of books. As data analyst, you have been tasked with scraping a specific subset of book data from Books to Scrape to assist publishing companies in understanding the landscape of highly-rated books across different genres. Your insights will help shape future book marketing strategies and publishing decisions.\n", - "\n", - "**Background**\n", - "\n", - "In a world where data has become the new currency, businesses are leveraging big data to make informed decisions that drive success and profitability. The publishing industry, much like others, utilizes data analytics to understand market trends, reader preferences, and the performance of books based on factors such as genre, author, and ratings. Books to Scrape serves as a rich source of such data, offering detailed information about a diverse range of books, making it an ideal platform for extracting insights to aid in informed decision-making within the literary world.\n", - "\n", - "**Task**\n", - "\n", - "Your task is to create a Python script using BeautifulSoup and pandas to scrape Books to Scrape book data, focusing on book ratings and genres. The script should be able to filter books with ratings above a certain threshold and in specific genres. Additionally, the script should structure the scraped data in a tabular format using pandas for further analysis.\n", - "\n", - "**Expected Outcome**\n", - "\n", - "A function named `scrape_books` that takes two parameters: `min_rating` and `max_price`. The function should scrape book data from the \"Books to Scrape\" website and return a `pandas` DataFrame with the following columns:\n", - "\n", - "**Expected Outcome**\n", - "\n", - "- A function named `scrape_books` that takes two parameters: `min_rating` and `max_price`.\n", - "- The function should return a DataFrame with the following columns:\n", - " - **UPC**: The Universal Product Code (UPC) of the book.\n", - " - **Title**: The title of the book.\n", - " - **Price (£)**: The price of the book in pounds.\n", - " - **Rating**: The rating of the book (1-5 stars).\n", - " - **Genre**: The genre of the book.\n", - " - **Availability**: Whether the book is in stock or not.\n", - " - **Description**: A brief description or product description of the book (if available).\n", - " \n", - "You will execute this script to scrape data for books with a minimum rating of `4.0 and above` and a maximum price of `£20`. \n", - "\n", - "Remember to experiment with different ratings and prices to ensure your code is versatile and can handle various searches effectively!\n", - "\n", - "**Resources**\n", - "\n", - "- [Beautiful Soup Documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)\n", - "- [Pandas Documentation](https://pandas.pydata.org/pandas-docs/stable/index.html)\n", - "- [Books to Scrape](https://books.toscrape.com/)\n" - ] - }, - { - "cell_type": "markdown", - "id": "3519921d-5890-445b-9a33-934ed8ee378c", - "metadata": { - "id": "3519921d-5890-445b-9a33-934ed8ee378c" - }, - "source": [ - "**Hint**\n", - "\n", - "Your first mission is to familiarize yourself with the **Books to Scrape** website. Navigate to [Books to Scrape](http://books.toscrape.com/) and explore the available books to understand their layout and structure. \n", - "\n", - "Next, think about how you can set parameters for your data extraction:\n", - "\n", - "- **Minimum Rating**: Focus on books with a rating of 4.0 and above.\n", - "- **Maximum Price**: Filter for books priced up to £20.\n", - "\n", - "After reviewing the site, you can construct a plan for scraping relevant data. Pay attention to the details displayed for each book, including the title, price, rating, and availability. This will help you identify the correct HTML elements to target with your scraping script.\n", - "\n", - "Make sure to build your scraping URL and logic based on the patterns you observe in the HTML structure of the book listings!" - ] - }, - { - "cell_type": "markdown", - "id": "25a83a0d-a742-49f6-985e-e27887cbf922", - "metadata": { - "id": "25a83a0d-a742-49f6-985e-e27887cbf922" - }, - "source": [ - "\n", - "---\n", - "\n", - "**Best of luck! Immerse yourself in the world of books, and may the data be with you!**" - ] - }, - { - "cell_type": "markdown", - "id": "7b75cf0d-9afa-4eec-a9e2-befeac68b2a0", - "metadata": { - "id": "7b75cf0d-9afa-4eec-a9e2-befeac68b2a0" - }, - "source": [ - "**Important Note**:\n", - "\n", - "In the fast-changing online world, websites often update and change their structures. When you try this lab, the **Books to Scrape** website might differ from what you expect.\n", - "\n", - "If you encounter issues due to these changes, like new rules or obstacles preventing data extraction, don’t worry! Get creative.\n", - "\n", - "You can choose another website that interests you and is suitable for scraping data. Options like Wikipedia, The New York Times, or even library databases are great alternatives. The main goal remains the same: extract useful data and enhance your web scraping skills while exploring a source of information you enjoy. This is your opportunity to practice and adapt to different web environments!" - ] - }, + "cells": [ + { + "cell_type": "markdown", + "id": "7e7a1ab8-2599-417d-9a65-25ef07f3a786", + "metadata": { + "id": "7e7a1ab8-2599-417d-9a65-25ef07f3a786" + }, + "source": [ + "# Lab | Web Scraping" + ] + }, + { + "cell_type": "markdown", + "id": "ce8882fc-4815-4567-92fa-b4816358ba7d", + "metadata": { + "id": "ce8882fc-4815-4567-92fa-b4816358ba7d" + }, + "source": [ + "Welcome to the \"Books to Scrape\" Web Scraping Adventure Lab!\n", + "\n", + "**Objective**\n", + "\n", + "In this lab, we will embark on a mission to unearth valuable insights from the data available on Books to Scrape, an online platform showcasing a wide variety of books. As data analyst, you have been tasked with scraping a specific subset of book data from Books to Scrape to assist publishing companies in understanding the landscape of highly-rated books across different genres. Your insights will help shape future book marketing strategies and publishing decisions.\n", + "\n", + "**Background**\n", + "\n", + "In a world where data has become the new currency, businesses are leveraging big data to make informed decisions that drive success and profitability. The publishing industry, much like others, utilizes data analytics to understand market trends, reader preferences, and the performance of books based on factors such as genre, author, and ratings. Books to Scrape serves as a rich source of such data, offering detailed information about a diverse range of books, making it an ideal platform for extracting insights to aid in informed decision-making within the literary world.\n", + "\n", + "**Task**\n", + "\n", + "Your task is to create a Python script using BeautifulSoup and pandas to scrape Books to Scrape book data, focusing on book ratings and genres. The script should be able to filter books with ratings above a certain threshold and in specific genres. Additionally, the script should structure the scraped data in a tabular format using pandas for further analysis.\n", + "\n", + "**Expected Outcome**\n", + "\n", + "A function named `scrape_books` that takes two parameters: `min_rating` and `max_price`. The function should scrape book data from the \"Books to Scrape\" website and return a `pandas` DataFrame with the following columns:\n", + "\n", + "**Expected Outcome**\n", + "\n", + "- A function named `scrape_books` that takes two parameters: `min_rating` and `max_price`.\n", + "- The function should return a DataFrame with the following columns:\n", + " - **UPC**: The Universal Product Code (UPC) of the book.\n", + " - **Title**: The title of the book.\n", + " - **Price (£)**: The price of the book in pounds.\n", + " - **Rating**: The rating of the book (1-5 stars).\n", + " - **Genre**: The genre of the book.\n", + " - **Availability**: Whether the book is in stock or not.\n", + " - **Description**: A brief description or product description of the book (if available).\n", + " \n", + "You will execute this script to scrape data for books with a minimum rating of `4.0 and above` and a maximum price of `£20`. \n", + "\n", + "Remember to experiment with different ratings and prices to ensure your code is versatile and can handle various searches effectively!\n", + "\n", + "**Resources**\n", + "\n", + "- [Beautiful Soup Documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)\n", + "- [Pandas Documentation](https://pandas.pydata.org/pandas-docs/stable/index.html)\n", + "- [Books to Scrape](https://books.toscrape.com/)\n" + ] + }, + { + "cell_type": "markdown", + "id": "3519921d-5890-445b-9a33-934ed8ee378c", + "metadata": { + "id": "3519921d-5890-445b-9a33-934ed8ee378c" + }, + "source": [ + "**Hint**\n", + "\n", + "Your first mission is to familiarize yourself with the **Books to Scrape** website. Navigate to [Books to Scrape](http://books.toscrape.com/) and explore the available books to understand their layout and structure. \n", + "\n", + "Next, think about how you can set parameters for your data extraction:\n", + "\n", + "- **Minimum Rating**: Focus on books with a rating of 4.0 and above.\n", + "- **Maximum Price**: Filter for books priced up to £20.\n", + "\n", + "After reviewing the site, you can construct a plan for scraping relevant data. Pay attention to the details displayed for each book, including the title, price, rating, and availability. This will help you identify the correct HTML elements to target with your scraping script.\n", + "\n", + "Make sure to build your scraping URL and logic based on the patterns you observe in the HTML structure of the book listings!" + ] + }, + { + "cell_type": "markdown", + "id": "25a83a0d-a742-49f6-985e-e27887cbf922", + "metadata": { + "id": "25a83a0d-a742-49f6-985e-e27887cbf922" + }, + "source": [ + "\n", + "---\n", + "\n", + "**Best of luck! Immerse yourself in the world of books, and may the data be with you!**" + ] + }, + { + "cell_type": "markdown", + "id": "7b75cf0d-9afa-4eec-a9e2-befeac68b2a0", + "metadata": { + "id": "7b75cf0d-9afa-4eec-a9e2-befeac68b2a0" + }, + "source": [ + "**Important Note**:\n", + "\n", + "In the fast-changing online world, websites often update and change their structures. When you try this lab, the **Books to Scrape** website might differ from what you expect.\n", + "\n", + "If you encounter issues due to these changes, like new rules or obstacles preventing data extraction, don’t worry! Get creative.\n", + "\n", + "You can choose another website that interests you and is suitable for scraping data. Options like Wikipedia, The New York Times, or even library databases are great alternatives. The main goal remains the same: extract useful data and enhance your web scraping skills while exploring a source of information you enjoy. This is your opportunity to practice and adapt to different web environments!" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "40359eee-9cd7-4884-bfa4-83344c222305", + "metadata": { + "id": "40359eee-9cd7-4884-bfa4-83344c222305" + }, + "outputs": [ { - "cell_type": "code", - "execution_count": null, - "id": "40359eee-9cd7-4884-bfa4-83344c222305", - "metadata": { - "id": "40359eee-9cd7-4884-bfa4-83344c222305" - }, - "outputs": [], - "source": [ - "# Your solution goes here" + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
UPCTitlePrice (£)RatingGenreAvailabilityDescription
0ce6396b0f23f6eccSet Me Free17.465Young AdultIn stockAaron Ledbetter’s future had been planned ou...
16258a1f6a6dcfe50The Four Agreements: A Practical Guide to Pers...17.665SpiritualityIn stockIn The Four Agreements, don Miguel Ruiz reveal...
26be3beb0793a53e7Sophie's World15.945PhilosophyIn stockA page-turning novel that is also an explorati...
3657fe5ead67a7767Untitled Collection: Sabbath Poems 201414.274PoetryIn stockMore than thirty-five years ago, when the weat...
451653ef291ab7ddcThis One Summer19.494Sequential ArtIn stockEvery summer, Rose goes with her mom and dad t...
........................
709c96cd1329fbd82dThe Zombie Room19.695DefaultIn stockAn unlikely bond is forged between three men f...
71b78deb463531d078The Silent Wife12.345FictionIn stockA chilling psychological thriller about a marr...
724280ac3eab57aa5dThe Girl You Lost12.295MysteryIn stockEighteen years ago your baby daughter was snat...
7329fc016c459aeb14The Edge of Reason (Bridget Jones #2)19.184Womens FictionIn stockMonday 27 January“7:15 a.m. Hurrah! The wild...
7419fec36a1dfb4c16A Spy's Devotion (The Regency Spies of London #1)16.975Historical FictionIn stockIn England’s Regency era, manners and elegan...
\n", + "

75 rows × 7 columns

\n", + "
" + ], + "text/plain": [ + " UPC Title \\\n", + "0 ce6396b0f23f6ecc Set Me Free \n", + "1 6258a1f6a6dcfe50 The Four Agreements: A Practical Guide to Pers... \n", + "2 6be3beb0793a53e7 Sophie's World \n", + "3 657fe5ead67a7767 Untitled Collection: Sabbath Poems 2014 \n", + "4 51653ef291ab7ddc This One Summer \n", + ".. ... ... \n", + "70 9c96cd1329fbd82d The Zombie Room \n", + "71 b78deb463531d078 The Silent Wife \n", + "72 4280ac3eab57aa5d The Girl You Lost \n", + "73 29fc016c459aeb14 The Edge of Reason (Bridget Jones #2) \n", + "74 19fec36a1dfb4c16 A Spy's Devotion (The Regency Spies of London #1) \n", + "\n", + " Price (£) Rating Genre Availability \\\n", + "0 17.46 5 Young Adult In stock \n", + "1 17.66 5 Spirituality In stock \n", + "2 15.94 5 Philosophy In stock \n", + "3 14.27 4 Poetry In stock \n", + "4 19.49 4 Sequential Art In stock \n", + ".. ... ... ... ... \n", + "70 19.69 5 Default In stock \n", + "71 12.34 5 Fiction In stock \n", + "72 12.29 5 Mystery In stock \n", + "73 19.18 4 Womens Fiction In stock \n", + "74 16.97 5 Historical Fiction In stock \n", + "\n", + " Description \n", + "0 Aaron Ledbetter’s future had been planned ou... \n", + "1 In The Four Agreements, don Miguel Ruiz reveal... \n", + "2 A page-turning novel that is also an explorati... \n", + "3 More than thirty-five years ago, when the weat... \n", + "4 Every summer, Rose goes with her mom and dad t... \n", + ".. ... \n", + "70 An unlikely bond is forged between three men f... \n", + "71 A chilling psychological thriller about a marr... \n", + "72 Eighteen years ago your baby daughter was snat... \n", + "73 Monday 27 January“7:15 a.m. Hurrah! The wild... \n", + "74 In England’s Regency era, manners and elegan... \n", + "\n", + "[75 rows x 7 columns]" ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" } - ], - "metadata": { - "colab": { - "provenance": [] - }, - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.13" - } + ], + "source": [ + "# Your solution goes here\n", + "import requests\n", + "from bs4 import BeautifulSoup\n", + "import pandas as pd\n", + "\n", + "# Map ratings text to numbers\n", + "RATING_MAP = {\"One\": 1, \"Two\": 2, \"Three\": 3, \"Four\": 4, \"Five\": 5}\n", + "BASE_URL = \"https://books.toscrape.com/catalogue/page-{}.html\"\n", + "\n", + "min_rating = 4 # Minimum rating filter\n", + "max_price = 20 # Maximum price filter\n", + "books_data = [] # List to store result\n", + "\n", + "for page in range(1, 51): # Loop through up to 50 pages (site max)\n", + " res = requests.get(BASE_URL.format(page)) # Get page HTML\n", + " soup = BeautifulSoup(res.text, \"html.parser\") # Parse HTML\n", + " books = soup.find_all(\"article\", class_=\"product_pod\") # Find all book entries\n", + " if not books: # If no books found, stop (end of catalogue)\n", + " break\n", + " for book in books:\n", + " title = book.h3.a[\"title\"] # Get book title\n", + " # price = float(book.find(\"p\", class_=\"price_color\").text[1:])\n", + " price = book.find(\"p\", class_=\"price_color\").text[1:] # Get price, remove £\n", + " price = float(price.strip().replace(\"£\",\"\")) # Remove £\n", + " rating = RATING_MAP[book.find(\"p\", class_=\"star-rating\")[\"class\"][1]] # Get numeric rating\n", + " \n", + " # Filter by min_rating and max_price\n", + " if rating < min_rating or price > max_price:\n", + " continue\n", + " availability = book.find(\"p\", class_=\"instock availability\").text.strip() # Get availability text\n", + " href = book.h3.a[\"href\"].replace(\"../../../\", \"\") # Get relative link to detail page\n", + " detail_url = \"https://books.toscrape.com/catalogue/\" + href # Construct full detail page URL\n", + " detail = BeautifulSoup(requests.get(detail_url).text, \"html.parser\") # Scrape detail page for UPC, genre, and description\n", + " # upc = detail.find(\"th\", string=\"UPC\").find_next(\"td\").text\n", + " upc = detail.find(\"th\", string=\"UPC\").find_next(\"td\").text # Get UPC using updated 'string' argument\n", + " genre = detail.find(\"ul\", class_=\"breadcrumb\").find_all(\"li\")[2].text.strip() # Get genre from the breadcrumb navigation\n", + " desc_tag = detail.find(\"div\", id=\"product_description\") # Get book description (if available)\n", + " description = \"\"\n", + " \n", + " if desc_tag and desc_tag.find_next_sibling(\"p\"):\n", + " description = desc_tag.find_next_sibling(\"p\").text.strip()\n", + " \n", + " # Add book data to list\n", + " books_data.append({\n", + " \"UPC\": upc,\n", + " \"Title\": title,\n", + " \"Price (£)\": price,\n", + " \"Rating\": rating,\n", + " \"Genre\": genre,\n", + " \"Availability\": availability,\n", + " \"Description\": description\n", + " })\n", + "\n", + "df = pd.DataFrame(books_data) # Convert list of dicts to pandas DataFrame\n", + "df" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e4e81e93-6c1f-4c5c-a43a-3c642709b1cb", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" }, - "nbformat": 4, - "nbformat_minor": 5 + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.6" + } + }, + "nbformat": 4, + "nbformat_minor": 5 }