From 3fcfe8038b2c89ecc59a973837139b2ec4534bda Mon Sep 17 00:00:00 2001 From: Camilla Scandola <103769428+camilla-scandola@users.noreply.github.com> Date: Sun, 21 Sep 2025 23:01:52 +0200 Subject: [PATCH 1/2] Solved Lab --- ...structuring-and-combining-checkpoint.ipynb | 168 ++++++ lab-dw-data-structuring-and-combining.ipynb | 504 +++++++++++++++++- 2 files changed, 663 insertions(+), 9 deletions(-) create mode 100644 .ipynb_checkpoints/lab-dw-data-structuring-and-combining-checkpoint.ipynb diff --git a/.ipynb_checkpoints/lab-dw-data-structuring-and-combining-checkpoint.ipynb b/.ipynb_checkpoints/lab-dw-data-structuring-and-combining-checkpoint.ipynb new file mode 100644 index 0000000..ec4e3f9 --- /dev/null +++ b/.ipynb_checkpoints/lab-dw-data-structuring-and-combining-checkpoint.ipynb @@ -0,0 +1,168 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "25d7736c-ba17-4aff-b6bb-66eba20fbf4e", + "metadata": { + "id": "25d7736c-ba17-4aff-b6bb-66eba20fbf4e" + }, + "source": [ + "# Lab | Data Structuring and Combining Data" + ] + }, + { + "cell_type": "markdown", + "id": "a2cdfc70-44c8-478c-81e7-2bc43fdf4986", + "metadata": { + "id": "a2cdfc70-44c8-478c-81e7-2bc43fdf4986" + }, + "source": [ + "## Challenge 1: Combining & Cleaning Data\n", + "\n", + "In this challenge, we will be working with the customer data from an insurance company, as we did in the two previous labs. The data can be found here:\n", + "- https://raw.githubusercontent.com/data-bootcamp-v4/data/main/file1.csv\n", + "\n", + "But this time, we got new data, which can be found in the following 2 CSV files located at the links below.\n", + "\n", + "- https://raw.githubusercontent.com/data-bootcamp-v4/data/main/file2.csv\n", + "- https://raw.githubusercontent.com/data-bootcamp-v4/data/main/file3.csv\n", + "\n", + "Note that you'll need to clean and format the new data.\n", + "\n", + "Observation:\n", + "- One option is to first combine the three datasets and then apply the cleaning function to the new combined dataset\n", + "- Another option would be to read the clean file you saved in the previous lab, and just clean the two new files and concatenate the three clean datasets" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "492d06e3-92c7-4105-ac72-536db98d3244", + "metadata": { + "id": "492d06e3-92c7-4105-ac72-536db98d3244" + }, + "outputs": [], + "source": [ + "# Your code goes here" + ] + }, + { + "cell_type": "markdown", + "id": "31b8a9e7-7db9-4604-991b-ef6771603e57", + "metadata": { + "id": "31b8a9e7-7db9-4604-991b-ef6771603e57" + }, + "source": [ + "# Challenge 2: Structuring Data" + ] + }, + { + "cell_type": "markdown", + "id": "a877fd6d-7a0c-46d2-9657-f25036e4ca4b", + "metadata": { + "id": "a877fd6d-7a0c-46d2-9657-f25036e4ca4b" + }, + "source": [ + "In this challenge, we will continue to work with customer data from an insurance company, but we will use a dataset with more columns, called marketing_customer_analysis.csv, which can be found at the following link:\n", + "\n", + "https://raw.githubusercontent.com/data-bootcamp-v4/data/main/marketing_customer_analysis_clean.csv\n", + "\n", + "This dataset contains information such as customer demographics, policy details, vehicle information, and the customer's response to the last marketing campaign. Our goal is to explore and analyze this data by performing data cleaning, formatting, and structuring." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aa10d9b0-1c27-4d3f-a8e4-db6ab73bfd26", + "metadata": { + "id": "aa10d9b0-1c27-4d3f-a8e4-db6ab73bfd26" + }, + "outputs": [], + "source": [ + "# Your code goes here" + ] + }, + { + "cell_type": "markdown", + "id": "df35fd0d-513e-4e77-867e-429da10a9cc7", + "metadata": { + "id": "df35fd0d-513e-4e77-867e-429da10a9cc7" + }, + "source": [ + "1. You work at the marketing department and you want to know which sales channel brought the most sales in terms of total revenue. Using pivot, create a summary table showing the total revenue for each sales channel (branch, call center, web, and mail).\n", + "Round the total revenue to 2 decimal points. Analyze the resulting table to draw insights." + ] + }, + { + "cell_type": "markdown", + "id": "640993b2-a291-436c-a34d-a551144f8196", + "metadata": { + "id": "640993b2-a291-436c-a34d-a551144f8196" + }, + "source": [ + "2. Create a pivot table that shows the average customer lifetime value per gender and education level. Analyze the resulting table to draw insights." + ] + }, + { + "cell_type": "markdown", + "id": "32c7f2e5-3d90-43e5-be33-9781b6069198", + "metadata": { + "id": "32c7f2e5-3d90-43e5-be33-9781b6069198" + }, + "source": [ + "## Bonus\n", + "\n", + "You work at the customer service department and you want to know which months had the highest number of complaints by policy type category. Create a summary table showing the number of complaints by policy type and month.\n", + "Show it in a long format table." + ] + }, + { + "cell_type": "markdown", + "id": "e3d09a8f-953c-448a-a5f8-2e5a8cca7291", + "metadata": { + "id": "e3d09a8f-953c-448a-a5f8-2e5a8cca7291" + }, + "source": [ + "*In data analysis, a long format table is a way of structuring data in which each observation or measurement is stored in a separate row of the table. The key characteristic of a long format table is that each column represents a single variable, and each row represents a single observation of that variable.*\n", + "\n", + "*More information about long and wide format tables here: https://www.statology.org/long-vs-wide-data/*" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3a069e0b-b400-470e-904d-d17582191be4", + "metadata": { + "id": "3a069e0b-b400-470e-904d-d17582191be4" + }, + "outputs": [], + "source": [ + "# Your code goes here" + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/lab-dw-data-structuring-and-combining.ipynb b/lab-dw-data-structuring-and-combining.ipynb index ec4e3f9..3edbff5 100644 --- a/lab-dw-data-structuring-and-combining.ipynb +++ b/lab-dw-data-structuring-and-combining.ipynb @@ -36,14 +36,166 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "id": "492d06e3-92c7-4105-ac72-536db98d3244", "metadata": { "id": "492d06e3-92c7-4105-ac72-536db98d3244" }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "(9020, 11)\n", + " customer state gender education customer_lifetime_value \\\n", + "0 QZ44356 Arizona F Bachelor 697953 \n", + "1 AI49188 Nevada F Bachelor 1288743 \n", + "2 WW63253 California M Bachelor 764586 \n", + "3 GA49547 Washington M High School Or Below 536307 \n", + "4 OC83172 Oregon F Bachelor 825629 \n", + "\n", + " income monthly_premium_auto number_of_open_complaints policy_type \\\n", + "0 0 94.0 0 Personal Auto \n", + "1 48767 108.0 0 Personal Auto \n", + "2 0 106.0 0 Corporate Auto \n", + "3 36357 68.0 0 Personal Auto \n", + "4 62902 69.0 0 Personal Auto \n", + "\n", + " vehicle_class total_claim_amount \n", + "0 Four-Door Car 1131.0 \n", + "1 Two-Door Car 566.0 \n", + "2 Suv 529.0 \n", + "3 Four-Door Car 17.0 \n", + "4 Two-Door Car 159.0 \n" + ] + } + ], + "source": [ + "import pandas as pd\n", + "\n", + "urls = [\n", + " 'https://raw.githubusercontent.com/camilla-scandola/lab-dw-data-cleaning-and-formatting/refs/heads/main/cleaned_dataset.csv',\n", + " 'https://raw.githubusercontent.com/data-bootcamp-v4/data/main/file2.csv',\n", + " 'https://raw.githubusercontent.com/data-bootcamp-v4/data/main/file3.csv'\n", + "]\n", + "\n", + "dfs = []\n", + "for url in urls:\n", + " df = pd.read_csv(url)\n", + " # standardize column names\n", + " df = df.rename(columns={\n", + " 'ST': 'State',})\n", + " df.columns = df.columns.str.strip().str.lower().str.replace(\" \", \"_\")\n", + " dfs.append(df)\n", + "\n", + "df_combined = pd.concat(dfs, ignore_index=True)\n", + "\n", + "print(df_combined.shape)\n", + "print(df_combined.head())\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "6fb1556d-fa58-419e-86a2-4cfecda2f33e", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "customer 0\n", + "state 0\n", + "gender 5\n", + "education 0\n", + "customer_lifetime_value 4\n", + "income 0\n", + "monthly_premium_auto 0\n", + "number_of_open_complaints 0\n", + "policy_type 0\n", + "vehicle_class 0\n", + "total_claim_amount 0\n", + "dtype: int64" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df_combined.isnull().sum()" + ] + }, + { + "cell_type": "markdown", + "id": "373d2d80-9d5c-4141-a41c-91b57a3d6387", + "metadata": {}, + "source": [ + "In this case, I will apply the mode to the gender column, as I need to fill 5 cells and it won't throw off the stats of the dataset. Same for the customer value column, I will replace the null values with the mean" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "f5c2de4f-e228-4a60-ac05-de8ea607d3d8", + "metadata": {}, "outputs": [], "source": [ - "# Your code goes here" + "df_combined['gender'] = df_combined['gender'].fillna(df_combined['gender'].mode()[0])" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "a60c1275-677c-432a-b2e5-2c76706d2e82", + "metadata": {}, + "outputs": [], + "source": [ + "#adding def from my previous lab\n", + "def clean_customer_lifetime_value(df):\n", + " \"\"\"remove % sign and convert to float\"\"\"\n", + " if 'customer_lifetime_value' in df.columns:\n", + " df['customer_lifetime_value'] = df['customer_lifetime_value'].str.strip().str.rstrip('%').astype(float)\n", + " \n", + " #fill missing values with mean\n", + " df['customer_lifetime_value'] = df['customer_lifetime_value'].fillna(\n", + " df['customer_lifetime_value'].mean()\n", + " )\n", + " return df\n", + "\n", + "df_combined = clean_customer_lifetime_value(df_combined)" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "12c2bd7b-f2fc-4226-b35e-42e3b75abbec", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "customer 0\n", + "state 0\n", + "gender 0\n", + "education 0\n", + "customer_lifetime_value 0\n", + "income 0\n", + "monthly_premium_auto 0\n", + "number_of_open_complaints 0\n", + "policy_type 0\n", + "vehicle_class 0\n", + "total_claim_amount 0\n", + "dtype: int64" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df_combined.isnull().sum()" ] }, { @@ -72,14 +224,301 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 8, "id": "aa10d9b0-1c27-4d3f-a8e4-db6ab73bfd26", "metadata": { "id": "aa10d9b0-1c27-4d3f-a8e4-db6ab73bfd26" }, - "outputs": [], + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + " | unnamed:_0 | \n", + "customer | \n", + "state | \n", + "customer_lifetime_value | \n", + "response | \n", + "coverage | \n", + "education | \n", + "effective_to_date | \n", + "employmentstatus | \n", + "gender | \n", + "... | \n", + "number_of_policies | \n", + "policy_type | \n", + "policy | \n", + "renew_offer_type | \n", + "sales_channel | \n", + "total_claim_amount | \n", + "vehicle_class | \n", + "vehicle_size | \n", + "vehicle_type | \n", + "month | \n", + "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | \n", + "0 | \n", + "DK49336 | \n", + "Arizona | \n", + "4809.216960 | \n", + "No | \n", + "Basic | \n", + "College | \n", + "2011-02-18 | \n", + "Employed | \n", + "M | \n", + "... | \n", + "9 | \n", + "Corporate Auto | \n", + "Corporate L3 | \n", + "Offer3 | \n", + "Agent | \n", + "292.800000 | \n", + "Four-Door Car | \n", + "Medsize | \n", + "A | \n", + "2 | \n", + "
1 | \n", + "1 | \n", + "KX64629 | \n", + "California | \n", + "2228.525238 | \n", + "No | \n", + "Basic | \n", + "College | \n", + "2011-01-18 | \n", + "Unemployed | \n", + "F | \n", + "... | \n", + "1 | \n", + "Personal Auto | \n", + "Personal L3 | \n", + "Offer4 | \n", + "Call Center | \n", + "744.924331 | \n", + "Four-Door Car | \n", + "Medsize | \n", + "A | \n", + "1 | \n", + "
2 | \n", + "2 | \n", + "LZ68649 | \n", + "Washington | \n", + "14947.917300 | \n", + "No | \n", + "Basic | \n", + "Bachelor | \n", + "2011-02-10 | \n", + "Employed | \n", + "M | \n", + "... | \n", + "2 | \n", + "Personal Auto | \n", + "Personal L3 | \n", + "Offer3 | \n", + "Call Center | \n", + "480.000000 | \n", + "SUV | \n", + "Medsize | \n", + "A | \n", + "2 | \n", + "
3 | \n", + "3 | \n", + "XL78013 | \n", + "Oregon | \n", + "22332.439460 | \n", + "Yes | \n", + "Extended | \n", + "College | \n", + "2011-01-11 | \n", + "Employed | \n", + "M | \n", + "... | \n", + "2 | \n", + "Corporate Auto | \n", + "Corporate L3 | \n", + "Offer2 | \n", + "Branch | \n", + "484.013411 | \n", + "Four-Door Car | \n", + "Medsize | \n", + "A | \n", + "1 | \n", + "
4 | \n", + "4 | \n", + "QA50777 | \n", + "Oregon | \n", + "9025.067525 | \n", + "No | \n", + "Premium | \n", + "Bachelor | \n", + "2011-01-17 | \n", + "Medical Leave | \n", + "F | \n", + "... | \n", + "7 | \n", + "Personal Auto | \n", + "Personal L2 | \n", + "Offer1 | \n", + "Branch | \n", + "707.925645 | \n", + "Four-Door Car | \n", + "Medsize | \n", + "A | \n", + "1 | \n", + "
5 rows × 27 columns
\n", + "