diff --git a/.ipynb_checkpoints/lab-dw-data-structuring-and-combining-checkpoint.ipynb b/.ipynb_checkpoints/lab-dw-data-structuring-and-combining-checkpoint.ipynb new file mode 100644 index 0000000..ec4e3f9 --- /dev/null +++ b/.ipynb_checkpoints/lab-dw-data-structuring-and-combining-checkpoint.ipynb @@ -0,0 +1,168 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "25d7736c-ba17-4aff-b6bb-66eba20fbf4e", + "metadata": { + "id": "25d7736c-ba17-4aff-b6bb-66eba20fbf4e" + }, + "source": [ + "# Lab | Data Structuring and Combining Data" + ] + }, + { + "cell_type": "markdown", + "id": "a2cdfc70-44c8-478c-81e7-2bc43fdf4986", + "metadata": { + "id": "a2cdfc70-44c8-478c-81e7-2bc43fdf4986" + }, + "source": [ + "## Challenge 1: Combining & Cleaning Data\n", + "\n", + "In this challenge, we will be working with the customer data from an insurance company, as we did in the two previous labs. The data can be found here:\n", + "- https://raw.githubusercontent.com/data-bootcamp-v4/data/main/file1.csv\n", + "\n", + "But this time, we got new data, which can be found in the following 2 CSV files located at the links below.\n", + "\n", + "- https://raw.githubusercontent.com/data-bootcamp-v4/data/main/file2.csv\n", + "- https://raw.githubusercontent.com/data-bootcamp-v4/data/main/file3.csv\n", + "\n", + "Note that you'll need to clean and format the new data.\n", + "\n", + "Observation:\n", + "- One option is to first combine the three datasets and then apply the cleaning function to the new combined dataset\n", + "- Another option would be to read the clean file you saved in the previous lab, and just clean the two new files and concatenate the three clean datasets" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "492d06e3-92c7-4105-ac72-536db98d3244", + "metadata": { + "id": "492d06e3-92c7-4105-ac72-536db98d3244" + }, + "outputs": [], + "source": [ + "# Your code goes here" + ] + }, + { + "cell_type": "markdown", + "id": "31b8a9e7-7db9-4604-991b-ef6771603e57", + "metadata": { + "id": "31b8a9e7-7db9-4604-991b-ef6771603e57" + }, + "source": [ + "# Challenge 2: Structuring Data" + ] + }, + { + "cell_type": "markdown", + "id": "a877fd6d-7a0c-46d2-9657-f25036e4ca4b", + "metadata": { + "id": "a877fd6d-7a0c-46d2-9657-f25036e4ca4b" + }, + "source": [ + "In this challenge, we will continue to work with customer data from an insurance company, but we will use a dataset with more columns, called marketing_customer_analysis.csv, which can be found at the following link:\n", + "\n", + "https://raw.githubusercontent.com/data-bootcamp-v4/data/main/marketing_customer_analysis_clean.csv\n", + "\n", + "This dataset contains information such as customer demographics, policy details, vehicle information, and the customer's response to the last marketing campaign. Our goal is to explore and analyze this data by performing data cleaning, formatting, and structuring." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aa10d9b0-1c27-4d3f-a8e4-db6ab73bfd26", + "metadata": { + "id": "aa10d9b0-1c27-4d3f-a8e4-db6ab73bfd26" + }, + "outputs": [], + "source": [ + "# Your code goes here" + ] + }, + { + "cell_type": "markdown", + "id": "df35fd0d-513e-4e77-867e-429da10a9cc7", + "metadata": { + "id": "df35fd0d-513e-4e77-867e-429da10a9cc7" + }, + "source": [ + "1. You work at the marketing department and you want to know which sales channel brought the most sales in terms of total revenue. Using pivot, create a summary table showing the total revenue for each sales channel (branch, call center, web, and mail).\n", + "Round the total revenue to 2 decimal points. Analyze the resulting table to draw insights." + ] + }, + { + "cell_type": "markdown", + "id": "640993b2-a291-436c-a34d-a551144f8196", + "metadata": { + "id": "640993b2-a291-436c-a34d-a551144f8196" + }, + "source": [ + "2. Create a pivot table that shows the average customer lifetime value per gender and education level. Analyze the resulting table to draw insights." + ] + }, + { + "cell_type": "markdown", + "id": "32c7f2e5-3d90-43e5-be33-9781b6069198", + "metadata": { + "id": "32c7f2e5-3d90-43e5-be33-9781b6069198" + }, + "source": [ + "## Bonus\n", + "\n", + "You work at the customer service department and you want to know which months had the highest number of complaints by policy type category. Create a summary table showing the number of complaints by policy type and month.\n", + "Show it in a long format table." + ] + }, + { + "cell_type": "markdown", + "id": "e3d09a8f-953c-448a-a5f8-2e5a8cca7291", + "metadata": { + "id": "e3d09a8f-953c-448a-a5f8-2e5a8cca7291" + }, + "source": [ + "*In data analysis, a long format table is a way of structuring data in which each observation or measurement is stored in a separate row of the table. The key characteristic of a long format table is that each column represents a single variable, and each row represents a single observation of that variable.*\n", + "\n", + "*More information about long and wide format tables here: https://www.statology.org/long-vs-wide-data/*" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3a069e0b-b400-470e-904d-d17582191be4", + "metadata": { + "id": "3a069e0b-b400-470e-904d-d17582191be4" + }, + "outputs": [], + "source": [ + "# Your code goes here" + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/lab-dw-data-structuring-and-combining.ipynb b/lab-dw-data-structuring-and-combining.ipynb index ec4e3f9..6d1878b 100644 --- a/lab-dw-data-structuring-and-combining.ipynb +++ b/lab-dw-data-structuring-and-combining.ipynb @@ -36,14 +36,157 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 48, "id": "492d06e3-92c7-4105-ac72-536db98d3244", "metadata": { "id": "492d06e3-92c7-4105-ac72-536db98d3244" }, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 48, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Your code goes here\n", + "import pandas as pd\n", + "url_1 = \"https://raw.githubusercontent.com/data-bootcamp-v4/data/main/file2.csv\"\n", + "url_2 = \"https://raw.githubusercontent.com/data-bootcamp-v4/data/main/file3.csv\"\n", + "df_1 = pd.read_csv(url_1)\n", + "df_2 = pd.read_csv(url_2)\n", + "df = pd.concat([df_1, df_2], axis =0)\n", + "\n", + "# If you want to concatenate DataFrames horizontally, use axis=1\n", + "# combined_df = pd.concat([df1, df2], axis=1)\n", + "df.info" + ] + }, + { + "cell_type": "code", + "execution_count": 50, + "id": "d7f18771-16ba-4996-8f22-eea0f6661336", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Empty DataFrame\n", + "Columns: [customer, st, gender, education, customer_lifetime_value, income, monthly_premium_auto, number_of_open_complaints, total_claim_amount, policy_type, vehicle_class, state, gender]\n", + "Index: []\n", + "\n", + "Index: 0 entries\n", + "Data columns (total 13 columns):\n", + " # Column Non-Null Count Dtype \n", + "--- ------ -------------- ----- \n", + " 0 customer 0 non-null object \n", + " 1 st 0 non-null object \n", + " 2 gender 0 non-null object \n", + " 3 education 0 non-null object \n", + " 4 customer_lifetime_value 0 non-null float64\n", + " 5 income 0 non-null int64 \n", + " 6 monthly_premium_auto 0 non-null int64 \n", + " 7 number_of_open_complaints 0 non-null int64 \n", + " 8 total_claim_amount 0 non-null float64\n", + " 9 policy_type 0 non-null object \n", + " 10 vehicle_class 0 non-null object \n", + " 11 state 0 non-null object \n", + " 12 gender 0 non-null object \n", + "dtypes: float64(2), int64(3), object(8)\n", + "memory usage: 0.0+ bytes\n", + "None\n" + ] + } + ], + "source": [ + "\n", + "# Step 1: Clean column names\n", + "df.columns = df.columns.str.strip().str.lower().str.replace(\" \", \"_\")\n", + "\n", + "# Step 2: Fix 'customer_lifetime_value' (remove %, convert to float)\n", + "if 'customer_lifetime_value' in df.columns:\n", + " df['customer_lifetime_value'] = (\n", + " df['customer_lifetime_value']\n", + " .astype(str)\n", + " .str.replace('%', '', regex=False)\n", + " .astype(float)\n", + " )\n", + "\n", + "# Step 3: Clean 'number_of_open_complaints'\n", + "df['number_of_open_complaints'] = pd.to_numeric(\n", + " df['number_of_open_complaints'], errors='coerce'\n", + ").fillna(0).astype(int)\n", + "\n", + "# Step 4: Clean 'total_claim_amount' to ensure it's numeric\n", + "df['total_claim_amount'] = pd.to_numeric(df['total_claim_amount'], errors='coerce')\n", + "\n", + "# Step 5: Handle missing values\n", + "# Drop rows with critical missing data\n", + "df.dropna(subset=['policy_type', 'state', 'gender', 'education'], inplace=True)\n", + "\n", + "# Optional: Fill other missing values\n", + "df['income'] = df['income'].fillna(0)\n", + "df['monthly_premium_auto'] = df['monthly_premium_auto'].fillna(df['monthly_premium_auto'].median())\n", + "\n", + "# Step 6: (Optional) Preview cleaned DataFrame\n", + "print(df.head())\n", + "print(df.info())\n" + ] + }, + { + "cell_type": "code", + "execution_count": 51, + "id": "99ecf157-1394-42d9-a71a-5338b9d66485", + "metadata": {}, "outputs": [], "source": [ - "# Your code goes here" + "# Save the DataFrame to a new CSV file\n", + "df.to_csv('lab_dw_data_structure.csv', index=False)" ] }, { @@ -72,14 +215,255 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 53, "id": "aa10d9b0-1c27-4d3f-a8e4-db6ab73bfd26", "metadata": { "id": "aa10d9b0-1c27-4d3f-a8e4-db6ab73bfd26" }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
unnamed:_0customerstatecustomer_lifetime_valueresponsecoverageeducationeffective_to_dateemploymentstatusgender...number_of_policiespolicy_typepolicyrenew_offer_typesales_channeltotal_claim_amountvehicle_classvehicle_sizevehicle_typemonth
00DK49336Arizona4809.216960NoBasicCollege2011-02-18EmployedM...9Corporate AutoCorporate L3Offer3Agent292.800000Four-Door CarMedsizeA2
11KX64629California2228.525238NoBasicCollege2011-01-18UnemployedF...1Personal AutoPersonal L3Offer4Call Center744.924331Four-Door CarMedsizeA1
22LZ68649Washington14947.917300NoBasicBachelor2011-02-10EmployedM...2Personal AutoPersonal L3Offer3Call Center480.000000SUVMedsizeA2
33XL78013Oregon22332.439460YesExtendedCollege2011-01-11EmployedM...2Corporate AutoCorporate L3Offer2Branch484.013411Four-Door CarMedsizeA1
44QA50777Oregon9025.067525NoPremiumBachelor2011-01-17Medical LeaveF...7Personal AutoPersonal L2Offer1Branch707.925645Four-Door CarMedsizeA1
\n", + "

5 rows × 27 columns

\n", + "
" + ], + "text/plain": [ + " unnamed:_0 customer state customer_lifetime_value response \\\n", + "0 0 DK49336 Arizona 4809.216960 No \n", + "1 1 KX64629 California 2228.525238 No \n", + "2 2 LZ68649 Washington 14947.917300 No \n", + "3 3 XL78013 Oregon 22332.439460 Yes \n", + "4 4 QA50777 Oregon 9025.067525 No \n", + "\n", + " coverage education effective_to_date employmentstatus gender ... \\\n", + "0 Basic College 2011-02-18 Employed M ... \n", + "1 Basic College 2011-01-18 Unemployed F ... \n", + "2 Basic Bachelor 2011-02-10 Employed M ... \n", + "3 Extended College 2011-01-11 Employed M ... \n", + "4 Premium Bachelor 2011-01-17 Medical Leave F ... \n", + "\n", + " number_of_policies policy_type policy renew_offer_type \\\n", + "0 9 Corporate Auto Corporate L3 Offer3 \n", + "1 1 Personal Auto Personal L3 Offer4 \n", + "2 2 Personal Auto Personal L3 Offer3 \n", + "3 2 Corporate Auto Corporate L3 Offer2 \n", + "4 7 Personal Auto Personal L2 Offer1 \n", + "\n", + " sales_channel total_claim_amount vehicle_class vehicle_size \\\n", + "0 Agent 292.800000 Four-Door Car Medsize \n", + "1 Call Center 744.924331 Four-Door Car Medsize \n", + "2 Call Center 480.000000 SUV Medsize \n", + "3 Branch 484.013411 Four-Door Car Medsize \n", + "4 Branch 707.925645 Four-Door Car Medsize \n", + "\n", + " vehicle_type month \n", + "0 A 2 \n", + "1 A 1 \n", + "2 A 2 \n", + "3 A 1 \n", + "4 A 1 \n", + "\n", + "[5 rows x 27 columns]" + ] + }, + "execution_count": 53, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Your code goes here\n", + "import pandas as pd\n", + "url = 'https://raw.githubusercontent.com/data-bootcamp-v4/data/main/marketing_customer_analysis_clean.csv'\n", + "df = pd.read_csv(url)\n", + "df.head()" + ] + }, + { + "cell_type": "code", + "execution_count": 54, + "id": "fb0b8bc9-b896-42e3-956e-6bdfd71904ff", + "metadata": {}, "outputs": [], "source": [ - "# Your code goes here" + "# Clean column names (optional but recommended)\n", + "df.columns = df.columns.str.strip().str.lower().str.replace(\" \", \"_\")\n", + "\n", + "# Clean 'customer_lifetime_value' - check type first and convert appropriately\n", + "if df['customer_lifetime_value'].dtype == 'object': # If it's string type\n", + " df['customer_lifetime_value'] = df['customer_lifetime_value'].str.replace('%', '').astype(float)\n", + "else: # If it's already numeric\n", + " df['customer_lifetime_value'] = df['customer_lifetime_value'].astype(float)\n", + "\n", + "# Clean 'number_of_open_complaints' (assuming it's malformed)\n", + "df['number_of_open_complaints'] = pd.to_numeric(df['number_of_open_complaints'], errors='coerce')\n", + "\n", + "# Drop rows with missing values in required columns\n", + "df.dropna(subset=['sales_channel', 'total_claim_amount'], inplace=True)\n" ] }, { @@ -93,6 +477,37 @@ "Round the total revenue to 2 decimal points. Analyze the resulting table to draw insights." ] }, + { + "cell_type": "code", + "execution_count": 55, + "id": "2dad06d9-671b-4245-9e82-24ea35153813", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " total_claim_amount\n", + "sales_channel \n", + "Agent 1810226.82\n", + "Branch 1301204.00\n", + "Call Center 926600.82\n", + "Web 706600.04\n" + ] + } + ], + "source": [ + "# Pivot table: Total revenue by Sales Channel\n", + "revenue_summary = df_clean.pivot_table(\n", + " index='sales_channel',\n", + " values='total_claim_amount',\n", + " aggfunc='sum'\n", + ").round(2)\n", + "\n", + "# Display table\n", + "print(revenue_summary)" + ] + }, { "cell_type": "markdown", "id": "640993b2-a291-436c-a34d-a551144f8196", @@ -103,6 +518,40 @@ "2. Create a pivot table that shows the average customer lifetime value per gender and education level. Analyze the resulting table to draw insights." ] }, + { + "cell_type": "code", + "execution_count": 36, + "id": "9e548d6c-72c6-4a01-90fd-f1f1f1557c43", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "education Bachelor College Doctor High School or Below Master\n", + "gender \n", + "F 7874.27 7748.82 7328.51 8675.22 8157.05\n", + "M 7703.60 8052.46 7415.33 8149.69 8168.83\n" + ] + } + ], + "source": [ + "\n", + "# Drop rows with missing Gender or Education (if any)\n", + "df.dropna(subset=['gender', 'education'], inplace=True)\n", + "\n", + "# Create pivot table\n", + "clv_pivot = df.pivot_table(\n", + " index='gender',\n", + " columns='education',\n", + " values='customer_lifetime_value',\n", + " aggfunc='mean'\n", + ").round(2)\n", + "\n", + "# Display the table\n", + "print(clv_pivot)" + ] + }, { "cell_type": "markdown", "id": "32c7f2e5-3d90-43e5-be33-9781b6069198", @@ -130,15 +579,51 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 56, "id": "3a069e0b-b400-470e-904d-d17582191be4", "metadata": { "id": "3a069e0b-b400-470e-904d-d17582191be4" }, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " policy_type month number_of_open_complaints\n", + "2 Personal Auto 1 1635\n", + "3 Personal Auto 2 1363\n", + "0 Corporate Auto 1 415\n", + "1 Corporate Auto 2 361\n", + "5 Special Auto 2 91\n", + "4 Special Auto 1 84\n" + ] + } + ], "source": [ - "# Your code goes here" + "# Your code goes here\n", + "# Ensure 'number_of_open_complaints' is numeric\n", + "df['number_of_open_complaints'] = pd.to_numeric(df['number_of_open_complaints'], errors='coerce').fillna(0).astype(int)\n", + "\n", + "# Drop rows with missing 'policy_type' or 'month'\n", + "df.dropna(subset=['policy_type', 'month'], inplace=True)\n", + "\n", + "# Group by policy type and month\n", + "complaints_summary = df.groupby(['policy_type', 'month'])['number_of_open_complaints'].sum().reset_index()\n", + "\n", + "# Sort by total complaints (optional)\n", + "complaints_summary = complaints_summary.sort_values(by='number_of_open_complaints', ascending=False)\n", + "\n", + "# Display result\n", + "print(complaints_summary)" ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d3fd9537-4bba-4573-9e00-5c1143b5b64a", + "metadata": {}, + "outputs": [], + "source": [] } ], "metadata": { @@ -146,9 +631,9 @@ "provenance": [] }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "Python [conda env:base] *", "language": "python", - "name": "python3" + "name": "conda-base-py" }, "language_info": { "codemirror_mode": { @@ -160,7 +645,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.13" + "version": "3.13.2" } }, "nbformat": 4, diff --git a/lab_dw_data_structure.csv b/lab_dw_data_structure.csv new file mode 100644 index 0000000..3b07990 --- /dev/null +++ b/lab_dw_data_structure.csv @@ -0,0 +1 @@ +customer,st,gender,education,customer_lifetime_value,income,monthly_premium_auto,number_of_open_complaints,total_claim_amount,policy_type,vehicle_class,state,gender