diff --git a/lab-dw-aggregating.ipynb b/lab-dw-aggregating.ipynb index fadd718..08af4c9 100644 --- a/lab-dw-aggregating.ipynb +++ b/lab-dw-aggregating.ipynb @@ -1,165 +1,1279 @@ { - "cells": [ - { - "cell_type": "markdown", - "id": "31969215-2a90-4d8b-ac36-646a7ae13744", - "metadata": { - "id": "31969215-2a90-4d8b-ac36-646a7ae13744" - }, - "source": [ - "# Lab | Data Aggregation and Filtering" - ] - }, - { - "cell_type": "markdown", - "id": "a8f08a52-bec0-439b-99cc-11d3809d8b5d", - "metadata": { - "id": "a8f08a52-bec0-439b-99cc-11d3809d8b5d" - }, - "source": [ - "In this challenge, we will continue to work with customer data from an insurance company. We will use the dataset called marketing_customer_analysis.csv, which can be found at the following link:\n", - "\n", - "https://raw.githubusercontent.com/data-bootcamp-v4/data/main/marketing_customer_analysis.csv\n", - "\n", - "This dataset contains information such as customer demographics, policy details, vehicle information, and the customer's response to the last marketing campaign. Our goal is to explore and analyze this data by first performing data cleaning, formatting, and structuring." - ] - }, - { - "cell_type": "markdown", - "id": "9c98ddc5-b041-4c94-ada1-4dfee5c98e50", - "metadata": { - "id": "9c98ddc5-b041-4c94-ada1-4dfee5c98e50" - }, - "source": [ - "1. Create a new DataFrame that only includes customers who:\n", - " - have a **low total_claim_amount** (e.g., below $1,000),\n", - " - have a response \"Yes\" to the last marketing campaign." - ] - }, - { - "cell_type": "markdown", - "id": "b9be383e-5165-436e-80c8-57d4c757c8c3", - "metadata": { - "id": "b9be383e-5165-436e-80c8-57d4c757c8c3" - }, - "source": [ - "2. Using the original Dataframe, analyze:\n", - " - the average `monthly_premium` and/or customer lifetime value by `policy_type` and `gender` for customers who responded \"Yes\", and\n", - " - compare these insights to `total_claim_amount` patterns, and discuss which segments appear most profitable or low-risk for the company." - ] - }, - { - "cell_type": "markdown", - "id": "7050f4ac-53c5-4193-a3c0-8699b87196f0", - "metadata": { - "id": "7050f4ac-53c5-4193-a3c0-8699b87196f0" - }, - "source": [ - "3. Analyze the total number of customers who have policies in each state, and then filter the results to only include states where there are more than 500 customers." - ] - }, + "cells": [ + { + "cell_type": "markdown", + "id": "31969215-2a90-4d8b-ac36-646a7ae13744", + "metadata": { + "id": "31969215-2a90-4d8b-ac36-646a7ae13744" + }, + "source": [ + "# Lab | Data Aggregation and Filtering" + ] + }, + { + "cell_type": "markdown", + "id": "a8f08a52-bec0-439b-99cc-11d3809d8b5d", + "metadata": { + "id": "a8f08a52-bec0-439b-99cc-11d3809d8b5d" + }, + "source": [ + "In this challenge, we will continue to work with customer data from an insurance company. We will use the dataset called marketing_customer_analysis.csv, which can be found at the following link:\n", + "\n", + "https://raw.githubusercontent.com/data-bootcamp-v4/data/main/marketing_customer_analysis.csv\n", + "\n", + "This dataset contains information such as customer demographics, policy details, vehicle information, and the customer's response to the last marketing campaign. Our goal is to explore and analyze this data by first performing data cleaning, formatting, and structuring." + ] + }, + { + "cell_type": "markdown", + "id": "9c98ddc5-b041-4c94-ada1-4dfee5c98e50", + "metadata": { + "id": "9c98ddc5-b041-4c94-ada1-4dfee5c98e50" + }, + "source": [ + "1. Create a new DataFrame that only includes customers who:\n", + " - have a **low total_claim_amount** (e.g., below $1,000),\n", + " - have a response \"Yes\" to the last marketing campaign." + ] + }, + { + "cell_type": "markdown", + "id": "b9be383e-5165-436e-80c8-57d4c757c8c3", + "metadata": { + "id": "b9be383e-5165-436e-80c8-57d4c757c8c3" + }, + "source": [ + "2. Using the original Dataframe, analyze:\n", + " - the average `monthly_premium` and/or customer lifetime value by `policy_type` and `gender` for customers who responded \"Yes\", and\n", + " - compare these insights to `total_claim_amount` patterns, and discuss which segments appear most profitable or low-risk for the company." + ] + }, + { + "cell_type": "markdown", + "id": "7050f4ac-53c5-4193-a3c0-8699b87196f0", + "metadata": { + "id": "7050f4ac-53c5-4193-a3c0-8699b87196f0" + }, + "source": [ + "3. Analyze the total number of customers who have policies in each state, and then filter the results to only include states where there are more than 500 customers." + ] + }, + { + "cell_type": "markdown", + "id": "b60a4443-a1a7-4bbf-b78e-9ccdf9895e0d", + "metadata": { + "id": "b60a4443-a1a7-4bbf-b78e-9ccdf9895e0d" + }, + "source": [ + "4. Find the maximum, minimum, and median customer lifetime value by education level and gender. Write your conclusions." + ] + }, + { + "cell_type": "markdown", + "id": "b42999f9-311f-481e-ae63-40a5577072c5", + "metadata": { + "id": "b42999f9-311f-481e-ae63-40a5577072c5" + }, + "source": [ + "## Bonus" + ] + }, + { + "cell_type": "markdown", + "id": "81ff02c5-6584-4f21-a358-b918697c6432", + "metadata": { + "id": "81ff02c5-6584-4f21-a358-b918697c6432" + }, + "source": [ + "5. The marketing team wants to analyze the number of policies sold by state and month. Present the data in a table where the months are arranged as columns and the states are arranged as rows." + ] + }, + { + "cell_type": "markdown", + "id": "b6aec097-c633-4017-a125-e77a97259cda", + "metadata": { + "id": "b6aec097-c633-4017-a125-e77a97259cda" + }, + "source": [ + "6. Display a new DataFrame that contains the number of policies sold by month, by state, for the top 3 states with the highest number of policies sold.\n", + "\n", + "*Hint:*\n", + "- *To accomplish this, you will first need to group the data by state and month, then count the number of policies sold for each group. Afterwards, you will need to sort the data by the count of policies sold in descending order.*\n", + "- *Next, you will select the top 3 states with the highest number of policies sold.*\n", + "- *Finally, you will create a new DataFrame that contains the number of policies sold by month for each of the top 3 states.*" + ] + }, + { + "cell_type": "markdown", + "id": "ba975b8a-a2cf-4fbf-9f59-ebc381767009", + "metadata": { + "id": "ba975b8a-a2cf-4fbf-9f59-ebc381767009" + }, + "source": [ + "7. The marketing team wants to analyze the effect of different marketing channels on the customer response rate.\n", + "\n", + "Hint: You can use melt to unpivot the data and create a table that shows the customer response rate (those who responded \"Yes\") by marketing channel." + ] + }, + { + "cell_type": "markdown", + "id": "e4378d94-48fb-4850-a802-b1bc8f427b2d", + "metadata": { + "id": "e4378d94-48fb-4850-a802-b1bc8f427b2d" + }, + "source": [ + "External Resources for Data Filtering: https://towardsdatascience.com/filtering-data-frames-in-pandas-b570b1f834b9" + ] + }, + { + "cell_type": "markdown", + "id": "4437078b-a14b-400a-986f-f2c074901b49", + "metadata": { + "id": "449513f4-0459-46a0-a18d-9398d974c9ad" + }, + "source": [ + "# 1\n", + "Create a new DataFrame that only includes customers who:\n", + "have a low total_claim_amount (e.g., below $1,000),\n", + "have a response \"Yes\" to the last marketing campaign.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "1f6594e0-1f23-4da3-bb2f-bd5e9c48a320", + "metadata": {}, + "outputs": [], + "source": [ + "import pandas as pd" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "029c2835-622a-418f-9d9d-0307170e8f07", + "metadata": {}, + "outputs": [ { - "cell_type": "markdown", - "id": "b60a4443-a1a7-4bbf-b78e-9ccdf9895e0d", - "metadata": { - "id": "b60a4443-a1a7-4bbf-b78e-9ccdf9895e0d" - }, - "source": [ - "4. Find the maximum, minimum, and median customer lifetime value by education level and gender. Write your conclusions." - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "RangeIndex: 10910 entries, 0 to 10909\n", + "Data columns (total 26 columns):\n", + " # Column Non-Null Count Dtype \n", + "--- ------ -------------- ----- \n", + " 0 Unnamed: 0 10910 non-null int64 \n", + " 1 Customer 10910 non-null object \n", + " 2 State 10279 non-null object \n", + " 3 Customer Lifetime Value 10910 non-null float64\n", + " 4 Response 10279 non-null object \n", + " 5 Coverage 10910 non-null object \n", + " 6 Education 10910 non-null object \n", + " 7 Effective To Date 10910 non-null object \n", + " 8 EmploymentStatus 10910 non-null object \n", + " 9 Gender 10910 non-null object \n", + " 10 Income 10910 non-null int64 \n", + " 11 Location Code 10910 non-null object \n", + " 12 Marital Status 10910 non-null object \n", + " 13 Monthly Premium Auto 10910 non-null int64 \n", + " 14 Months Since Last Claim 10277 non-null float64\n", + " 15 Months Since Policy Inception 10910 non-null int64 \n", + " 16 Number of Open Complaints 10277 non-null float64\n", + " 17 Number of Policies 10910 non-null int64 \n", + " 18 Policy Type 10910 non-null object \n", + " 19 Policy 10910 non-null object \n", + " 20 Renew Offer Type 10910 non-null object \n", + " 21 Sales Channel 10910 non-null object \n", + " 22 Total Claim Amount 10910 non-null float64\n", + " 23 Vehicle Class 10288 non-null object \n", + " 24 Vehicle Size 10288 non-null object \n", + " 25 Vehicle Type 5428 non-null object \n", + "dtypes: float64(4), int64(5), object(17)\n", + "memory usage: 2.2+ MB\n", + "None\n" + ] }, { - "cell_type": "markdown", - "id": "b42999f9-311f-481e-ae63-40a5577072c5", - "metadata": { - "id": "b42999f9-311f-481e-ae63-40a5577072c5" - }, - "source": [ - "## Bonus" + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
Unnamed: 0CustomerStateCustomer Lifetime ValueResponseCoverageEducationEffective To DateEmploymentStatusGender...Number of Open ComplaintsNumber of PoliciesPolicy TypePolicyRenew Offer TypeSales ChannelTotal Claim AmountVehicle ClassVehicle SizeVehicle Type
00DK49336Arizona4809.216960NoBasicCollege2/18/11EmployedM...0.09Corporate AutoCorporate L3Offer3Agent292.800000Four-Door CarMedsizeNaN
11KX64629California2228.525238NoBasicCollege1/18/11UnemployedF...0.01Personal AutoPersonal L3Offer4Call Center744.924331Four-Door CarMedsizeNaN
22LZ68649Washington14947.917300NoBasicBachelor2/10/11EmployedM...0.02Personal AutoPersonal L3Offer3Call Center480.000000SUVMedsizeA
33XL78013Oregon22332.439460YesExtendedCollege1/11/11EmployedM...0.02Corporate AutoCorporate L3Offer2Branch484.013411Four-Door CarMedsizeA
44QA50777Oregon9025.067525NoPremiumBachelor1/17/11Medical LeaveF...NaN7Personal AutoPersonal L2Offer1Branch707.925645Four-Door CarMedsizeNaN
\n", + "

5 rows × 26 columns

\n", + "
" + ], + "text/plain": [ + " Unnamed: 0 Customer State Customer Lifetime Value Response \\\n", + "0 0 DK49336 Arizona 4809.216960 No \n", + "1 1 KX64629 California 2228.525238 No \n", + "2 2 LZ68649 Washington 14947.917300 No \n", + "3 3 XL78013 Oregon 22332.439460 Yes \n", + "4 4 QA50777 Oregon 9025.067525 No \n", + "\n", + " Coverage Education Effective To Date EmploymentStatus Gender ... \\\n", + "0 Basic College 2/18/11 Employed M ... \n", + "1 Basic College 1/18/11 Unemployed F ... \n", + "2 Basic Bachelor 2/10/11 Employed M ... \n", + "3 Extended College 1/11/11 Employed M ... \n", + "4 Premium Bachelor 1/17/11 Medical Leave F ... \n", + "\n", + " Number of Open Complaints Number of Policies Policy Type Policy \\\n", + "0 0.0 9 Corporate Auto Corporate L3 \n", + "1 0.0 1 Personal Auto Personal L3 \n", + "2 0.0 2 Personal Auto Personal L3 \n", + "3 0.0 2 Corporate Auto Corporate L3 \n", + "4 NaN 7 Personal Auto Personal L2 \n", + "\n", + " Renew Offer Type Sales Channel Total Claim Amount Vehicle Class \\\n", + "0 Offer3 Agent 292.800000 Four-Door Car \n", + "1 Offer4 Call Center 744.924331 Four-Door Car \n", + "2 Offer3 Call Center 480.000000 SUV \n", + "3 Offer2 Branch 484.013411 Four-Door Car \n", + "4 Offer1 Branch 707.925645 Four-Door Car \n", + "\n", + " Vehicle Size Vehicle Type \n", + "0 Medsize NaN \n", + "1 Medsize NaN \n", + "2 Medsize A \n", + "3 Medsize A \n", + "4 Medsize NaN \n", + "\n", + "[5 rows x 26 columns]" ] - }, + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df = pd.read_csv('https://raw.githubusercontent.com/data-bootcamp-v4/data/main/marketing_customer_analysis.csv')\n", + "print(df.info())\n", + "df.head()" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "28be9bad-5e0a-4364-87d8-6204b33c669f", + "metadata": {}, + "outputs": [ { - "cell_type": "markdown", - "id": "81ff02c5-6584-4f21-a358-b918697c6432", - "metadata": { - "id": "81ff02c5-6584-4f21-a358-b918697c6432" - }, - "source": [ - "5. The marketing team wants to analyze the number of policies sold by state and month. Present the data in a table where the months are arranged as columns and the states are arranged as rows." - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "Index: 1399 entries, 3 to 10897\n", + "Data columns (total 26 columns):\n", + " # Column Non-Null Count Dtype \n", + "--- ------ -------------- ----- \n", + " 0 Unnamed: 0 1399 non-null int64 \n", + " 1 Customer 1399 non-null object \n", + " 2 State 1399 non-null object \n", + " 3 Customer Lifetime Value 1399 non-null float64\n", + " 4 Response 1399 non-null object \n", + " 5 Coverage 1399 non-null object \n", + " 6 Education 1399 non-null object \n", + " 7 Effective To Date 1399 non-null object \n", + " 8 EmploymentStatus 1399 non-null object \n", + " 9 Gender 1399 non-null object \n", + " 10 Income 1399 non-null int64 \n", + " 11 Location Code 1399 non-null object \n", + " 12 Marital Status 1399 non-null object \n", + " 13 Monthly Premium Auto 1399 non-null int64 \n", + " 14 Months Since Last Claim 1324 non-null float64\n", + " 15 Months Since Policy Inception 1399 non-null int64 \n", + " 16 Number of Open Complaints 1324 non-null float64\n", + " 17 Number of Policies 1399 non-null int64 \n", + " 18 Policy Type 1399 non-null object \n", + " 19 Policy 1399 non-null object \n", + " 20 Renew Offer Type 1399 non-null object \n", + " 21 Sales Channel 1399 non-null object \n", + " 22 Total Claim Amount 1399 non-null float64\n", + " 23 Vehicle Class 1317 non-null object \n", + " 24 Vehicle Size 1317 non-null object \n", + " 25 Vehicle Type 684 non-null object \n", + "dtypes: float64(4), int64(5), object(17)\n", + "memory usage: 295.1+ KB\n" + ] }, { - "cell_type": "markdown", - "id": "b6aec097-c633-4017-a125-e77a97259cda", - "metadata": { - "id": "b6aec097-c633-4017-a125-e77a97259cda" - }, - "source": [ - "6. Display a new DataFrame that contains the number of policies sold by month, by state, for the top 3 states with the highest number of policies sold.\n", - "\n", - "*Hint:*\n", - "- *To accomplish this, you will first need to group the data by state and month, then count the number of policies sold for each group. Afterwards, you will need to sort the data by the count of policies sold in descending order.*\n", - "- *Next, you will select the top 3 states with the highest number of policies sold.*\n", - "- *Finally, you will create a new DataFrame that contains the number of policies sold by month for each of the top 3 states.*" + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
Unnamed: 0CustomerStateCustomer Lifetime ValueResponseCoverageEducationEffective To DateEmploymentStatusGender...Number of Open ComplaintsNumber of PoliciesPolicy TypePolicyRenew Offer TypeSales ChannelTotal Claim AmountVehicle ClassVehicle SizeVehicle Type
33XL78013Oregon22332.439460YesExtendedCollege1/11/11EmployedM...0.02Corporate AutoCorporate L3Offer2Branch484.013411Four-Door CarMedsizeA
88FM55990California5989.773931YesPremiumCollege1/19/11EmployedM...0.01Personal AutoPersonal L1Offer2Branch739.200000Sports CarMedsizeNaN
1515CW49887California4626.801093YesBasicMaster1/16/11EmployedF...0.01Special AutoSpecial L1Offer2Branch547.200000SUVMedsizeNaN
1919NJ54277California3746.751625YesExtendedCollege2/26/11EmployedF...1.01Personal AutoPersonal L2Offer2Call Center19.575683Two-Door CarLargeA
2727MQ68407Oregon4376.363592YesPremiumBachelor2/28/11EmployedF...0.01Personal AutoPersonal L3Offer2Agent60.036683Four-Door CarMedsizeNaN
\n", + "

5 rows × 26 columns

\n", + "
" + ], + "text/plain": [ + " Unnamed: 0 Customer State Customer Lifetime Value Response \\\n", + "3 3 XL78013 Oregon 22332.439460 Yes \n", + "8 8 FM55990 California 5989.773931 Yes \n", + "15 15 CW49887 California 4626.801093 Yes \n", + "19 19 NJ54277 California 3746.751625 Yes \n", + "27 27 MQ68407 Oregon 4376.363592 Yes \n", + "\n", + " Coverage Education Effective To Date EmploymentStatus Gender ... \\\n", + "3 Extended College 1/11/11 Employed M ... \n", + "8 Premium College 1/19/11 Employed M ... \n", + "15 Basic Master 1/16/11 Employed F ... \n", + "19 Extended College 2/26/11 Employed F ... \n", + "27 Premium Bachelor 2/28/11 Employed F ... \n", + "\n", + " Number of Open Complaints Number of Policies Policy Type \\\n", + "3 0.0 2 Corporate Auto \n", + "8 0.0 1 Personal Auto \n", + "15 0.0 1 Special Auto \n", + "19 1.0 1 Personal Auto \n", + "27 0.0 1 Personal Auto \n", + "\n", + " Policy Renew Offer Type Sales Channel Total Claim Amount \\\n", + "3 Corporate L3 Offer2 Branch 484.013411 \n", + "8 Personal L1 Offer2 Branch 739.200000 \n", + "15 Special L1 Offer2 Branch 547.200000 \n", + "19 Personal L2 Offer2 Call Center 19.575683 \n", + "27 Personal L3 Offer2 Agent 60.036683 \n", + "\n", + " Vehicle Class Vehicle Size Vehicle Type \n", + "3 Four-Door Car Medsize A \n", + "8 Sports Car Medsize NaN \n", + "15 SUV Medsize NaN \n", + "19 Two-Door Car Large A \n", + "27 Four-Door Car Medsize NaN \n", + "\n", + "[5 rows x 26 columns]" ] - }, + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Filter the dataframe to include only rows where:\n", + "# 1. Total Claim Amount is less than 1000\n", + "# 2. Response is 'Yes'\n", + "\n", + "df_claim_amount_and_response_yes = df[\n", + " (df['Total Claim Amount'] < 1000) &\n", + " (df['Response'] == 'Yes')\n", + "]\n", + "\n", + "df_claim_amount_and_response_yes.info()\n", + "df_claim_amount_and_response_yes.head()" + ] + }, + { + "cell_type": "markdown", + "id": "465f42be-e532-4d0a-b488-7c6053a8608e", + "metadata": {}, + "source": [ + "# 2\n", + "Using the original Dataframe, analyze:\n", + "the average monthly_premium and/or customer lifetime value by policy_type and gender for customers who responded \"Yes\", and\n", + "compare these insights to total_claim_amount patterns, and discuss which segments appear most profitable or low-risk for the company." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "dfe80241-2ec7-48b2-be59-4d5d0cf10cbf", + "metadata": {}, + "outputs": [], + "source": [ + "# Filter the dataframe to only include rows where the 'Response' column equals 'Yes'\n", + "# This creates a new dataframe containing only the positive responses\n", + "df_response_yes = df[\n", + " (df['Response'] == 'Yes')\n", + "]" + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "686b90d6-e19b-4489-a0bc-92228c5590f9", + "metadata": {}, + "outputs": [ { - "cell_type": "markdown", - "id": "ba975b8a-a2cf-4fbf-9f59-ebc381767009", - "metadata": { - "id": "ba975b8a-a2cf-4fbf-9f59-ebc381767009" - }, - "source": [ - "7. The marketing team wants to analyze the effect of different marketing channels on the customer response rate.\n", - "\n", - "Hint: You can use melt to unpivot the data and create a table that shows the customer response rate (those who responded \"Yes\") by marketing channel." - ] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + "Gender\n", + "F 8166.837762\n", + "M 7589.595500\n", + "Name: Customer Lifetime Value, dtype: float64\n" + ] + } + ], + "source": [ + "# Calculate the mean Customer Lifetime Value grouped by Gender\n", + "mean_by_gender_customer_lifetime_value = df_response_yes.groupby('Gender')['Customer Lifetime Value'].mean()\n", + "print(mean_by_gender_customer_lifetime_value)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f98998f9-926e-42d9-b67d-b9c14be8bf38", + "metadata": {}, + "outputs": [], + "source": [ + "# Calculate the mean monthly auto premium for each gender group\n", + "mean_by_gender_monthly_premium_auto = df_response_yes.groupby('Gender')['Monthly Premium Auto'].mean() \n", + "print(mean_by_gender_monthly_premium_auto)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "33557a0b-8710-4c1b-8676-f56166f5e93b", + "metadata": {}, + "outputs": [], + "source": [ + "# Calculate the mean Customer Lifetime Value for each Policy Type\n", + "mean_by_policy_type_customer_lifetime_value = df_response_yes.groupby('Policy Type')['Customer Lifetime Value'].mean()\n", + "print(mean_by_policy_type_customer_lifetime_value)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e59bd99e-6a92-4457-af7f-a972c595a64a", + "metadata": {}, + "outputs": [], + "source": [ + "# Calculate the mean Monthly Premium Auto for each Policy Type\n", + "mean_by_policy_type_monthly_premium_auto = df_response_yes.groupby('Policy Type')['Monthly Premium Auto'].mean()\n", + "print(mean_by_policy_type_monthly_premium_auto)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f30fe04e-d54a-4224-863e-aa45ed538dae", + "metadata": {}, + "outputs": [], + "source": [ + "# Group the data by 'Policy Type' and 'Gender', then calculate the mean of\n", + "# 'Monthly Premium Auto', 'Customer Lifetime Value', and 'Total Claim Amount'\n", + "\n", + "summary = df_response_yes.groupby(['Policy Type', 'Gender'])[\n", + " ['Monthly Premium Auto', 'Customer Lifetime Value', 'Total Claim Amount']\n", + "].mean().round(2)\n", + "print(summary)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c0faa467-bb81-4efc-a410-19dc0d510360", + "metadata": {}, + "outputs": [], + "source": [ + "from tabulate import tabulate\n", + "\n", + "print(tabulate(summary.reset_index(), headers='keys', tablefmt='pretty'))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dbc8c607-f6ed-4a20-a220-1e31d8abb023", + "metadata": {}, + "outputs": [], + "source": [ + "import seaborn as sns\n", + "import matplotlib.pyplot as plt\n", + "\n", + "\n", + "df_plot = summary.reset_index()\n", + "\n", + "\n", + "plt.figure(figsize=(10,6))\n", + "sns.barplot(data=df_plot, x='Policy Type', y='Customer Lifetime Value', hue='Gender')\n", + "plt.title('Customer Lifetime Value per Policy Type and Gender')\n", + "plt.ylabel('Average CLV')\n", + "plt.xticks(rotation=45)\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a59ec059-5ab3-4152-98dc-c039eeacf931", + "metadata": {}, + "outputs": [], + "source": [ + "plt.figure(figsize=(10,6))\n", + "sns.barplot(data=df_plot, x='Policy Type', y='Total Claim Amount', hue='Gender')\n", + "plt.title('TCA per Policy Type and Gender')\n", + "plt.ylabel('Average Total Claim Amount')\n", + "plt.xticks(rotation=45)\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f23b657f-7768-4272-b21c-5d194af3c047", + "metadata": {}, + "outputs": [], + "source": [ + "sns.scatterplot(data=df_plot, x='Total Claim Amount', y='Customer Lifetime Value', hue='Policy Type', style='Gender')\n", + "plt.title('CLV vs TCA')\n", + "plt.xlabel('Total Claim Amount')\n", + "plt.ylabel('Customer Lifetime Value')\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "159a6f39-6ce1-4844-9f74-e54cbc8e9157", + "metadata": {}, + "source": [ + "A segment that stands out for its high profitability is male individuals with 'corporate auto' and 'special auto' policies." + ] + }, + { + "cell_type": "markdown", + "id": "d9d1ec07-bc53-4b49-9213-0c96427d368a", + "metadata": {}, + "source": [ + "# 3\n", + "Analyze the total number of customers who have policies in each state, and then filter the results to only include states where there are more than 500 customers." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "45dfc4ce-52fd-4512-8631-cc8f667d7a1b", + "metadata": {}, + "outputs": [], + "source": [ + "df.State.value_counts()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "caa0c1f5-f617-46d6-9f36-de7c3bba5cbe", + "metadata": {}, + "outputs": [], + "source": [ + "clients_per_state = df['State'].value_counts()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "871d6c29-8920-4f9a-a4ac-343559732245", + "metadata": {}, + "outputs": [], + "source": [ + "more_than_500 = clients_per_state[clients_per_state > 500]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "620a218d-66af-467f-a40c-a198ae8a44da", + "metadata": {}, + "outputs": [], + "source": [ + "df_states = more_than_500.reset_index()\n", + "df_states.columns = ['State', 'Total Clients'] " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e0bdef2c-1fa6-470b-ab4a-6d949749a1e4", + "metadata": {}, + "outputs": [], + "source": [ + "df_states.head()" + ] + }, + { + "cell_type": "markdown", + "id": "9585a927-cf8f-40c7-bca5-fad628e6cafb", + "metadata": {}, + "source": [ + "# 4\n", + "Find the maximum, minimum, and median customer lifetime value by education level and gender. Write your conclusions." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "055b3cab-c83a-4e31-a4d9-a2bb93944b04", + "metadata": {}, + "outputs": [], + "source": [ + "# Group the dataframe by 'Education' and 'Gender' columns\n", + "# For each group, calculate min, max, and median of 'Customer Lifetime Value'\n", + "\n", + "summary2 = df.groupby(['Education', 'Gender'])['Customer Lifetime Value'].agg(['min', 'max', 'median']).round(2)\n", + "print(summary2)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cd56a735-ef69-4d18-bb8e-7cc88834549c", + "metadata": {}, + "outputs": [], + "source": [ + "import seaborn as sns\n", + "import matplotlib.pyplot as plt\n", + "\n", + "plt.figure(figsize=(10,6))\n", + "sns.boxplot(data=df, x='Education', y='Customer Lifetime Value', hue='Gender')\n", + "plt.title('Distribuição de CLV por Educação e Género')\n", + "plt.ylabel('Customer Lifetime Value')\n", + "plt.xticks(rotation=45)\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "9b1d6d86-7128-41fb-88f0-64cadc5aada6", + "metadata": {}, + "source": [ + "1. Customers with \"High School or Below\" have a higher median CLV\n", + "- Men with high school or less have the highest median CLV: 6286.73\n", + "- Women in this group also have a high median CLV: 6039.55\n", + "\n", + "\n", + "2. Doctorates have lower maximum values\n", + "Despite being highly qualified, the maximum CLV values for doctorates are the lowest of the group:\n", + "- F: 44856.11\n", + "- M: 32677.34\n", + "\n", + "\n", + "3. Gender differences are subtle, but consistent\n", + "At almost all levels, women have a slightly lower median CLV than men—except in the \"Master\" group, where women lead.\n", + "\n", + "\n", + "4. Customers with \"Bachelor\", \"College\" and 'High School or Below' degrees have greater dispersion\n", + "- Very high maximums (e.g., F with Bachelor's degree: 73225.96), but lower medians → indicates high variability\n", + "- There may be very valuable subgroups mixed with low-return customers\n" + ] + }, + { + "cell_type": "markdown", + "id": "435d9fd5-ab04-4dd7-ab48-51b2dbc21a3e", + "metadata": {}, + "source": [ + "# Bonus\n", + "# 5 \n", + "The marketing team wants to analyze the number of policies sold by state and month. Present the data in a table where the months are arranged as columns and the states are arranged as rows." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "4baf37d5-cd19-49c0-a0a7-cc0efa306c93", + "metadata": {}, + "outputs": [], + "source": [ + "df['Month'] = pd.to_datetime(df['Effective To Date'], format='%m/%d/%y').dt.month_name()" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "88f2335b-1df4-4660-90c6-ee0ab286b176", + "metadata": {}, + "outputs": [], + "source": [ + "policies_grouped = df.groupby(['State', 'Month']).size().reset_index(name='Total Policies')" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "b4de1957-127c-4770-81fe-daa9ecbe911e", + "metadata": {}, + "outputs": [], + "source": [ + "policies = policies_grouped.pivot(index='State', columns='Month', values='Total Policies').fillna(0).astype(int)" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "08a853b5-7b26-4a71-87a0-f1e8dec7d07e", + "metadata": {}, + "outputs": [], + "source": [ + "from calendar import month_name\n", + "\n", + "\n", + "ordered_months = list(month_name)[1:] \n", + "\n", + "\n", + "total_policies = policies.reindex(columns=ordered_months)" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "1d0ebeb7-4c38-488d-a9dd-3c6f5420ec55", + "metadata": {}, + "outputs": [ { - "cell_type": "markdown", - "id": "e4378d94-48fb-4850-a802-b1bc8f427b2d", - "metadata": { - "id": "e4378d94-48fb-4850-a802-b1bc8f427b2d" - }, - "source": [ - "External Resources for Data Filtering: https://towardsdatascience.com/filtering-data-frames-in-pandas-b570b1f834b9" - ] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + "Month January February March April May June July August \\\n", + "State \n", + "Arizona 1008 929 NaN NaN NaN NaN NaN NaN \n", + "California 1918 1634 NaN NaN NaN NaN NaN NaN \n", + "Nevada 551 442 NaN NaN NaN NaN NaN NaN \n", + "Oregon 1565 1344 NaN NaN NaN NaN NaN NaN \n", + "Washington 463 425 NaN NaN NaN NaN NaN NaN \n", + "\n", + "Month September October November December \n", + "State \n", + "Arizona NaN NaN NaN NaN \n", + "California NaN NaN NaN NaN \n", + "Nevada NaN NaN NaN NaN \n", + "Oregon NaN NaN NaN NaN \n", + "Washington NaN NaN NaN NaN \n" + ] + } + ], + "source": [ + "print(total_policies)" + ] + }, + { + "cell_type": "markdown", + "id": "be888c80-08aa-4536-9c32-fa3f77b65e4d", + "metadata": {}, + "source": [ + "# 6\n", + "Display a new DataFrame that contains the number of policies sold by month, by state, for the top 3 states with the highest number of policies sold.\n", + "Hint:\n", + "\n", + "To accomplish this, you will first need to group the data by state and month, then count the number of policies sold for each group. Afterwards, you will need to sort the data by the count of policies sold in descending order.\n", + "Next, you will select the top 3 states with the highest number of policies sold.\n", + "Finally, you will create a new DataFrame that contains the number of policies sold by month for each of the top 3 states." + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "724436fe-25d4-4a80-9259-919a5279ce50", + "metadata": {}, + "outputs": [], + "source": [ + "# Group the dataframe by 'State' and 'Month' columns\n", + "# Count the number of policies in each group\n", + "# Reset the index to convert the result back to a dataframe\n", + "# Name the count column as 'Total Policies'\n", + "\n", + "policies_by_state_month = df.groupby(['State', 'Month']).size().reset_index(name='Total Policies')" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "4d9e35b2-f2cf-420e-b2a6-edd32d98325d", + "metadata": {}, + "outputs": [], + "source": [ + "# Group policies by state, sum the 'Total Policies' column for each state, and sort in descending order\n", + "total_by_state = policies_by_state_month.groupby('State')['Total Policies'].sum().sort_values(ascending=False)\n", + "# Get the state names (index values) of the top 3 states with the highest total policies\n", + "top_3_states = total_by_state.head(3).index.tolist()" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "6c60bfea-af49-4954-bc5d-a1bec6ead235", + "metadata": {}, + "outputs": [], + "source": [ + "# Filter the dataframe to only include rows where the State column value is in the top 3 states list\n", + "top3_df = policies_by_state_month[policies_by_state_month['State'].isin(top_3_states)]" + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "137e2252-3ae3-400e-b343-c7beb39256bb", + "metadata": {}, + "outputs": [], + "source": [ + "# Create a pivot table from top3_df with states as rows and months as columns\n", + "# Values in the table represent total policies for each state-month combination\n", + "# Fill any missing values with 0 and convert all values to integers\n", + "final_table = top3_df.pivot(index='State', columns='Month', values='Total Policies').fillna(0).astype(int)" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "6fadf8b2-c31c-4f2b-a124-27f5ec932481", + "metadata": {}, + "outputs": [ { - "cell_type": "code", - "execution_count": null, - "id": "449513f4-0459-46a0-a18d-9398d974c9ad", - "metadata": { - "id": "449513f4-0459-46a0-a18d-9398d974c9ad" - }, - "outputs": [], - "source": [ - "# your code goes here" - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "Month February January\n", + "State \n", + "Arizona 929 1008\n", + "California 1634 1918\n", + "Oregon 1344 1565\n" + ] } - ], - "metadata": { - "colab": { - "provenance": [] - }, - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.13" + ], + "source": [ + "print(final_table)" + ] + }, + { + "cell_type": "markdown", + "id": "6c82ff27-417b-4953-849a-f59cbc6d8229", + "metadata": {}, + "source": [ + "# 7\n", + "The marketing team wants to analyze the effect of different marketing channels on the customer response rate.\n", + "Hint: You can use melt to unpivot the data and create a table that shows the customer response rate (those who responded \"Yes\") by marketing channel." + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "id": "5c76ac9e-d34e-4058-b031-3208ac5cb4e8", + "metadata": {}, + "outputs": [], + "source": [ + "# Filter the dataframe to keep only rows where 'Response' column is not null (NaN)\n", + "df_valid = df[df['Response'].notna()]" + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "id": "98052eea-145f-469d-94b4-91f0625a2a06", + "metadata": {}, + "outputs": [], + "source": [ + "# Group the validation dataframe by 'Sales Channel' column and count the number of records in each group\n", + "# This creates a Series with sales channels as index and counts as values\n", + "total_channel = df_valid.groupby('Sales Channel').size()" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "id": "a64a889f-10d6-4467-88e1-30c0a8a79d71", + "metadata": {}, + "outputs": [], + "source": [ + "# Count the number of customers who responded \"Yes\" for each sales channel\n", + "response_yes = df_valid[df_valid['Response'] == 'Yes'].groupby('Sales Channel').size()" + ] + }, + { + "cell_type": "code", + "execution_count": 39, + "id": "6d3858d8-02ff-4176-87d3-9e7fe698273e", + "metadata": {}, + "outputs": [], + "source": [ + "# Calculate the response rate by dividing \"Yes\" responses by total customers per channel\n", + "# Round the result to 3 decimal places and reset the index for a clean DataFrame\n", + "response_rate = (response_yes / total_channel).round(3).reset_index(name='Response Rate')" + ] + }, + { + "cell_type": "code", + "execution_count": 40, + "id": "77bb23d0-e284-437c-92f3-1ca9f6ca9ba8", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " Sales Channel Response Rate\n", + "0 Agent 0.191\n", + "1 Branch 0.114\n", + "2 Call Center 0.110\n", + "3 Web 0.117\n" + ] } + ], + "source": [ + "print(response_rate)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c6055025-99eb-4b4e-bbde-426ea4f09274", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python [conda env:base] *", + "language": "python", + "name": "conda-base-py" }, - "nbformat": 4, - "nbformat_minor": 5 + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.5" + } + }, + "nbformat": 4, + "nbformat_minor": 5 }