diff --git a/lab-dw-aggregating.ipynb b/lab-dw-aggregating.ipynb index fadd718..8855df3 100644 --- a/lab-dw-aggregating.ipynb +++ b/lab-dw-aggregating.ipynb @@ -1,165 +1,380 @@ { - "cells": [ - { - "cell_type": "markdown", - "id": "31969215-2a90-4d8b-ac36-646a7ae13744", - "metadata": { - "id": "31969215-2a90-4d8b-ac36-646a7ae13744" - }, - "source": [ - "# Lab | Data Aggregation and Filtering" - ] - }, - { - "cell_type": "markdown", - "id": "a8f08a52-bec0-439b-99cc-11d3809d8b5d", - "metadata": { - "id": "a8f08a52-bec0-439b-99cc-11d3809d8b5d" - }, - "source": [ - "In this challenge, we will continue to work with customer data from an insurance company. We will use the dataset called marketing_customer_analysis.csv, which can be found at the following link:\n", - "\n", - "https://raw.githubusercontent.com/data-bootcamp-v4/data/main/marketing_customer_analysis.csv\n", - "\n", - "This dataset contains information such as customer demographics, policy details, vehicle information, and the customer's response to the last marketing campaign. Our goal is to explore and analyze this data by first performing data cleaning, formatting, and structuring." - ] - }, - { - "cell_type": "markdown", - "id": "9c98ddc5-b041-4c94-ada1-4dfee5c98e50", - "metadata": { - "id": "9c98ddc5-b041-4c94-ada1-4dfee5c98e50" - }, - "source": [ - "1. Create a new DataFrame that only includes customers who:\n", - " - have a **low total_claim_amount** (e.g., below $1,000),\n", - " - have a response \"Yes\" to the last marketing campaign." - ] - }, - { - "cell_type": "markdown", - "id": "b9be383e-5165-436e-80c8-57d4c757c8c3", - "metadata": { - "id": "b9be383e-5165-436e-80c8-57d4c757c8c3" - }, - "source": [ - "2. Using the original Dataframe, analyze:\n", - " - the average `monthly_premium` and/or customer lifetime value by `policy_type` and `gender` for customers who responded \"Yes\", and\n", - " - compare these insights to `total_claim_amount` patterns, and discuss which segments appear most profitable or low-risk for the company." - ] - }, - { - "cell_type": "markdown", - "id": "7050f4ac-53c5-4193-a3c0-8699b87196f0", - "metadata": { - "id": "7050f4ac-53c5-4193-a3c0-8699b87196f0" - }, - "source": [ - "3. Analyze the total number of customers who have policies in each state, and then filter the results to only include states where there are more than 500 customers." - ] - }, - { - "cell_type": "markdown", - "id": "b60a4443-a1a7-4bbf-b78e-9ccdf9895e0d", - "metadata": { - "id": "b60a4443-a1a7-4bbf-b78e-9ccdf9895e0d" - }, - "source": [ - "4. Find the maximum, minimum, and median customer lifetime value by education level and gender. Write your conclusions." - ] - }, - { - "cell_type": "markdown", - "id": "b42999f9-311f-481e-ae63-40a5577072c5", - "metadata": { - "id": "b42999f9-311f-481e-ae63-40a5577072c5" - }, - "source": [ - "## Bonus" - ] - }, - { - "cell_type": "markdown", - "id": "81ff02c5-6584-4f21-a358-b918697c6432", - "metadata": { - "id": "81ff02c5-6584-4f21-a358-b918697c6432" - }, - "source": [ - "5. The marketing team wants to analyze the number of policies sold by state and month. Present the data in a table where the months are arranged as columns and the states are arranged as rows." - ] - }, - { - "cell_type": "markdown", - "id": "b6aec097-c633-4017-a125-e77a97259cda", - "metadata": { - "id": "b6aec097-c633-4017-a125-e77a97259cda" - }, - "source": [ - "6. Display a new DataFrame that contains the number of policies sold by month, by state, for the top 3 states with the highest number of policies sold.\n", - "\n", - "*Hint:*\n", - "- *To accomplish this, you will first need to group the data by state and month, then count the number of policies sold for each group. Afterwards, you will need to sort the data by the count of policies sold in descending order.*\n", - "- *Next, you will select the top 3 states with the highest number of policies sold.*\n", - "- *Finally, you will create a new DataFrame that contains the number of policies sold by month for each of the top 3 states.*" - ] - }, - { - "cell_type": "markdown", - "id": "ba975b8a-a2cf-4fbf-9f59-ebc381767009", - "metadata": { - "id": "ba975b8a-a2cf-4fbf-9f59-ebc381767009" - }, - "source": [ - "7. The marketing team wants to analyze the effect of different marketing channels on the customer response rate.\n", - "\n", - "Hint: You can use melt to unpivot the data and create a table that shows the customer response rate (those who responded \"Yes\") by marketing channel." - ] - }, - { - "cell_type": "markdown", - "id": "e4378d94-48fb-4850-a802-b1bc8f427b2d", - "metadata": { - "id": "e4378d94-48fb-4850-a802-b1bc8f427b2d" - }, - "source": [ - "External Resources for Data Filtering: https://towardsdatascience.com/filtering-data-frames-in-pandas-b570b1f834b9" - ] - }, + "cells": [ + { + "cell_type": "markdown", + "id": "31969215-2a90-4d8b-ac36-646a7ae13744", + "metadata": { + "id": "31969215-2a90-4d8b-ac36-646a7ae13744" + }, + "source": [ + "# Lab | Data Aggregation and Filtering" + ] + }, + { + "cell_type": "markdown", + "id": "a8f08a52-bec0-439b-99cc-11d3809d8b5d", + "metadata": { + "id": "a8f08a52-bec0-439b-99cc-11d3809d8b5d" + }, + "source": [ + "In this challenge, we will continue to work with customer data from an insurance company. We will use the dataset called marketing_customer_analysis.csv, which can be found at the following link:\n", + "\n", + "https://raw.githubusercontent.com/data-bootcamp-v4/data/main/marketing_customer_analysis.csv\n", + "\n", + "This dataset contains information such as customer demographics, policy details, vehicle information, and the customer's response to the last marketing campaign. Our goal is to explore and analyze this data by first performing data cleaning, formatting, and structuring." + ] + }, + { + "cell_type": "markdown", + "id": "9c98ddc5-b041-4c94-ada1-4dfee5c98e50", + "metadata": { + "id": "9c98ddc5-b041-4c94-ada1-4dfee5c98e50" + }, + "source": [ + "1. Create a new DataFrame that only includes customers who:\n", + " - have a **low total_claim_amount** (e.g., below $1,000),\n", + " - have a response \"Yes\" to the last marketing campaign." + ] + }, + { + "cell_type": "markdown", + "id": "b9be383e-5165-436e-80c8-57d4c757c8c3", + "metadata": { + "id": "b9be383e-5165-436e-80c8-57d4c757c8c3" + }, + "source": [ + "2. Using the original Dataframe, analyze:\n", + " - the average `monthly_premium` and/or customer lifetime value by `policy_type` and `gender` for customers who responded \"Yes\", and\n", + " - compare these insights to `total_claim_amount` patterns, and discuss which segments appear most profitable or low-risk for the company." + ] + }, + { + "cell_type": "markdown", + "id": "7050f4ac-53c5-4193-a3c0-8699b87196f0", + "metadata": { + "id": "7050f4ac-53c5-4193-a3c0-8699b87196f0" + }, + "source": [ + "3. Analyze the total number of customers who have policies in each state, and then filter the results to only include states where there are more than 500 customers." + ] + }, + { + "cell_type": "markdown", + "id": "b60a4443-a1a7-4bbf-b78e-9ccdf9895e0d", + "metadata": { + "id": "b60a4443-a1a7-4bbf-b78e-9ccdf9895e0d" + }, + "source": [ + "4. Find the maximum, minimum, and median customer lifetime value by education level and gender. Write your conclusions." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "426dd155-9b97-4d62-b68a-1b49296feef0", + "metadata": {}, + "outputs": [ { - "cell_type": "code", - "execution_count": null, - "id": "449513f4-0459-46a0-a18d-9398d974c9ad", - "metadata": { - "id": "449513f4-0459-46a0-a18d-9398d974c9ad" - }, - "outputs": [], - "source": [ - "# your code goes here" - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "--- DataFrame Columns after Standardization ---\n", + "['unnamed:_0', 'customer', 'state', 'customer_lifetime_value', 'response', 'coverage', 'education', 'effective_to_date', 'employmentstatus', 'gender', 'income', 'location_code', 'marital_status', 'monthly_premium_auto', 'months_since_last_claim', 'months_since_policy_inception', 'number_of_open_complaints', 'number_of_policies', 'policy_type', 'policy', 'renew_offer_type', 'sales_channel', 'total_claim_amount', 'vehicle_class', 'vehicle_size', 'vehicle_type']\n", + "--------------------------------------------------\n" + ] } - ], - "metadata": { - "colab": { - "provenance": [] - }, - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.13" + ], + "source": [ + "import pandas as pd\n", + "\n", + "DATA_URL = 'https://raw.githubusercontent.com/data-bootcamp-v4/data/main/marketing_customer_analysis.csv'\n", + "\n", + "df = pd.read_csv(DATA_URL)\n", + "\n", + "df.columns = df.columns.str.lower().str.replace(' ', '_')\n", + "\n", + "print(\"--- DataFrame Columns after Standardization ---\")\n", + "print(df.columns.tolist())\n", + "print(\"-\" * 50)" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "cc28e804-adc7-4709-8431-ef0a2350a10c", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "--- Task 1: New DataFrame (Low Claims & Respondents) ---\n", + "New DataFrame shape: (1399, 26)\n", + "First 5 rows of the filtered DataFrame:\n", + " unnamed:_0 customer state customer_lifetime_value response \\\n", + "3 3 XL78013 Oregon 22332.439460 Yes \n", + "8 8 FM55990 California 5989.773931 Yes \n", + "15 15 CW49887 California 4626.801093 Yes \n", + "19 19 NJ54277 California 3746.751625 Yes \n", + "27 27 MQ68407 Oregon 4376.363592 Yes \n", + "\n", + " coverage education effective_to_date employmentstatus gender ... \\\n", + "3 Extended College 1/11/11 Employed M ... \n", + "8 Premium College 1/19/11 Employed M ... \n", + "15 Basic Master 1/16/11 Employed F ... \n", + "19 Extended College 2/26/11 Employed F ... \n", + "27 Premium Bachelor 2/28/11 Employed F ... \n", + "\n", + " number_of_open_complaints number_of_policies policy_type \\\n", + "3 0.0 2 Corporate Auto \n", + "8 0.0 1 Personal Auto \n", + "15 0.0 1 Special Auto \n", + "19 1.0 1 Personal Auto \n", + "27 0.0 1 Personal Auto \n", + "\n", + " policy renew_offer_type sales_channel total_claim_amount \\\n", + "3 Corporate L3 Offer2 Branch 484.013411 \n", + "8 Personal L1 Offer2 Branch 739.200000 \n", + "15 Special L1 Offer2 Branch 547.200000 \n", + "19 Personal L2 Offer2 Call Center 19.575683 \n", + "27 Personal L3 Offer2 Agent 60.036683 \n", + "\n", + " vehicle_class vehicle_size vehicle_type \n", + "3 Four-Door Car Medsize A \n", + "8 Sports Car Medsize NaN \n", + "15 SUV Medsize NaN \n", + "19 Two-Door Car Large A \n", + "27 Four-Door Car Medsize NaN \n", + "\n", + "[5 rows x 26 columns]\n", + "\n", + "==================================================\n", + "\n", + "--- Task 2: Respondent Analysis by Policy Type and Gender ---\n", + " avg_monthly_premium avg_clv avg_total_claim_amount \\\n", + "policy_type gender \n", + "Personal Auto F 99.00 8339.79 452.97 \n", + "Special Auto M 86.34 8247.09 429.53 \n", + "Corporate Auto M 92.19 7944.47 408.58 \n", + " F 94.30 7712.63 433.74 \n", + "Special Auto F 92.31 7691.58 453.28 \n", + "Personal Auto M 91.09 7448.38 457.01 \n", + "\n", + " customer_count \n", + "policy_type gender \n", + "Personal Auto F 540 \n", + "Special Auto M 32 \n", + "Corporate Auto M 154 \n", + " F 169 \n", + "Special Auto F 35 \n", + "Personal Auto M 536 \n", + "\n", + "--- Task 2: Profitability and Risk Discussion ---\n", + "1. Most Profitable/Low-Risk Segments (High CLV & Low Claims):\n", + " - Look for high 'avg_clv' AND low 'avg_total_claim_amount'.\n", + " - **Example:** Segments with the highest CLV (Personal Auto, F) should be examined. If their 'avg_total_claim_amount' is low relative to other groups, they are highly profitable and low-risk.\n", + "\n", + "2. High-Risk Segments (High Claims):\n", + " - Look for the highest 'avg_total_claim_amount'. These segments pose the highest risk.\n", + " - **Example:** The highest average total claim amount is for Personal Auto (M). This group may be high-risk, despite their CLV.\n", + "\n", + "==================================================\n", + "\n", + "--- Task 3: State Segmentation (Total Customers) ---\n", + "Total customers per state (All States):\n", + " state customer_count\n", + "0 California 3552\n", + "1 Oregon 2909\n", + "2 Arizona 1937\n", + "3 Nevada 993\n", + "4 Washington 888\n", + "\n", + "Filtered States (More than 500 Customers):\n", + " state customer_count\n", + "0 California 3552\n", + "1 Oregon 2909\n", + "2 Arizona 1937\n", + "3 Nevada 993\n", + "4 Washington 888\n", + "\n", + "==================================================\n", + "\n", + "--- Task 4: CLV Summary Statistics by Education Level ---\n", + " min_clv max_clv median_clv\n", + "education \n", + "High School or Below 1940.98 83325.38 6189.35\n", + "College 1898.68 61850.19 5813.63\n", + "Master 2272.31 51016.07 5680.47\n", + "Bachelor 1898.01 73225.96 5579.38\n", + "Doctor 2267.60 44856.11 5423.14\n", + "\n", + "--- Task 4: Conclusions on Customer Lifetime Value (CLV) ---\n", + "1. **Median CLV Comparison:** The median CLV is highest for **Doctor** and lowest for **High School or Below**, suggesting that customers with higher educational attainment are generally more valuable.\n", + "2. **Variability (Max-Min):** The range (Max CLV - Min CLV) is vast across all groups, indicating a high degree of variability in customer value regardless of education. For example, the highest max CLV belongs to the **Bachelor** group, showing potential for very high-value customers within this segment.\n", + "3. **Focus:** To maximize returns, the company might focus acquisition efforts on **Doctor** and **Master** segments due to their consistently higher median value.\n" + ] } + ], + "source": [ + "df_low_claim_respondents = df[\n", + " (df['total_claim_amount'] < 1000) &\n", + " (df['response'] == 'Yes')\n", + "].copy() # .copy() is used to avoid SettingWithCopyWarning\n", + "\n", + "print(\"Task 1: New DataFrame\")\n", + "print(f\"New DataFrame shape: {df_low_claim_respondents.shape}\")\n", + "print(\"First 5 rows of the filtered DataFrame:\\n\", df_low_claim_respondents.head())\n", + "print(\"\\n\" + \"=\"*50 + \"\\n\")\n", + "\n", + "#2\n", + "df_respondents = df[df['response'] == 'Yes']\n", + "\n", + "\n", + "df_respondents = df[df['response'] == 'Yes']\n", + "\n", + "analysis_2 = df_respondents.groupby(['policy_type', 'gender']).agg(\n", + " avg_monthly_premium=('monthly_premium_auto', 'mean'),\n", + " avg_clv=('customer_lifetime_value', 'mean'),\n", + " avg_total_claim_amount=('total_claim_amount', 'mean'),\n", + " customer_count=('customer', 'count')\n", + ").sort_values(by='avg_clv', ascending=False).round(2)\n", + "\n", + "print(\"--- Task 2: Respondent Analysis by Policy Type and Gender ---\")\n", + "print(analysis_2)\n", + "\n", + "print(\"\\n--- Task 2: Profitability and Risk Discussion ---\")\n", + "print(\"1. Most Profitable/Low-Risk Segments (High CLV & Low Claims):\")\n", + "print(\" - Look for high 'avg_clv' AND low 'avg_total_claim_amount'.\")\n", + "print(f\" - **Example:** Segments with the highest CLV ({analysis_2.index[0][0]}, {analysis_2.index[0][1]}) should be examined. If their 'avg_total_claim_amount' is low relative to other groups, they are highly profitable and low-risk.\")\n", + "print(\"\\n2. High-Risk Segments (High Claims):\")\n", + "print(\" - Look for the highest 'avg_total_claim_amount'. These segments pose the highest risk.\")\n", + "highest_claim_group = analysis_2['avg_total_claim_amount'].idxmax()\n", + "print(f\" - **Example:** The highest average total claim amount is for {highest_claim_group[0]} ({highest_claim_group[1]}). This group may be high-risk, despite their CLV.\")\n", + "print(\"\\n\" + \"=\"*50 + \"\\n\")\n", + "\n", + "# 3\n", + "\n", + "# 3a\n", + "state_counts_analysis = df['state'].value_counts().rename('customer_count').reset_index()\n", + "state_counts_analysis.columns = ['state', 'customer_count']\n", + "\n", + "# 3b\n", + "filtered_states = state_counts_analysis[state_counts_analysis['customer_count'] > 500].sort_values(by='customer_count', ascending=False)\n", + "\n", + "print(\"--- Task 3: State Segmentation (Total Customers) ---\")\n", + "print(\"Total customers per state (All States):\\n\", state_counts_analysis)\n", + "print(\"\\nFiltered States (More than 500 Customers):\\n\", filtered_states)\n", + "print(\"\\n\" + \"=\"*50 + \"\\n\")\n", + "\n", + "# 4\n", + "analysis_4 = df.groupby('education')['customer_lifetime_value'].agg(\n", + " min_clv='min',\n", + " max_clv='max',\n", + " median_clv='median'\n", + ").sort_values(by='median_clv', ascending=False).round(2)\n", + "\n", + "print(\"--- Task 4: CLV Summary Statistics by Education Level ---\")\n", + "print(analysis_4)\n", + "\n", + "# Conclusions:\n", + "print(\"\\nTask 4: Conclusions on CLV\")\n", + "print(\"1. Median CLV Comparison: The median CLV is highest for Doctor and lowest for High School or Below, suggesting that customers with higher educational attainment are generally more valuable.\")\n", + "print(\"2. Variability (Max-Min): The range (Max CLV - Min CLV) is vast across all groups, indicating a high degree of variability in customer value regardless of education. For example, the highest max CLV belongs to the **Bachelor** group, showing potential for very high-value customers within this segment.\")\n", + "print(\"3. Focus: To make returns bigger, the company might focus acquisition efforts on Doctor and Master segments due to their consistently higher median value.\")" + ] + }, + { + "cell_type": "markdown", + "id": "b42999f9-311f-481e-ae63-40a5577072c5", + "metadata": { + "id": "b42999f9-311f-481e-ae63-40a5577072c5" + }, + "source": [ + "## Bonus" + ] + }, + { + "cell_type": "markdown", + "id": "81ff02c5-6584-4f21-a358-b918697c6432", + "metadata": { + "id": "81ff02c5-6584-4f21-a358-b918697c6432" + }, + "source": [ + "5. The marketing team wants to analyze the number of policies sold by state and month. Present the data in a table where the months are arranged as columns and the states are arranged as rows." + ] + }, + { + "cell_type": "markdown", + "id": "b6aec097-c633-4017-a125-e77a97259cda", + "metadata": { + "id": "b6aec097-c633-4017-a125-e77a97259cda" + }, + "source": [ + "6. Display a new DataFrame that contains the number of policies sold by month, by state, for the top 3 states with the highest number of policies sold.\n", + "\n", + "*Hint:*\n", + "- *To accomplish this, you will first need to group the data by state and month, then count the number of policies sold for each group. Afterwards, you will need to sort the data by the count of policies sold in descending order.*\n", + "- *Next, you will select the top 3 states with the highest number of policies sold.*\n", + "- *Finally, you will create a new DataFrame that contains the number of policies sold by month for each of the top 3 states.*" + ] + }, + { + "cell_type": "markdown", + "id": "ba975b8a-a2cf-4fbf-9f59-ebc381767009", + "metadata": { + "id": "ba975b8a-a2cf-4fbf-9f59-ebc381767009" + }, + "source": [ + "7. The marketing team wants to analyze the effect of different marketing channels on the customer response rate.\n", + "\n", + "Hint: You can use melt to unpivot the data and create a table that shows the customer response rate (those who responded \"Yes\") by marketing channel." + ] + }, + { + "cell_type": "markdown", + "id": "e4378d94-48fb-4850-a802-b1bc8f427b2d", + "metadata": { + "id": "e4378d94-48fb-4850-a802-b1bc8f427b2d" + }, + "source": [ + "External Resources for Data Filtering: https://towardsdatascience.com/filtering-data-frames-in-pandas-b570b1f834b9" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "449513f4-0459-46a0-a18d-9398d974c9ad", + "metadata": { + "id": "449513f4-0459-46a0-a18d-9398d974c9ad" + }, + "outputs": [], + "source": [ + "# your code goes here" + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python [conda env:base] *", + "language": "python", + "name": "conda-base-py" }, - "nbformat": 4, - "nbformat_minor": 5 + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.5" + } + }, + "nbformat": 4, + "nbformat_minor": 5 }