H2O/Machine Learning Regression problem with H2O (XGBoost & Deeplearning).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning Regression problem with H2O (XGBoost & Deeplearning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we will use the subset of the Freddie Mac Single-Family dataset to try to predict the interest rate for a loan using H2O's XGBoost and Deep Learning models. We will explore how to use these models for a regression problem, and we will also demonstrate how to use H2O's grid search to tune the hyper-parameters of both models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data set we're using comes from Freddie Mac and contains 20 years of mortgage history for each loan and contains information about \"loan-level credit performance data on a portion of fully amortizing fixed-rate mortgages that Freddie Mac bought between 1999 to 2017. Features include demographic factors, monthly loan performance, credit performance including property disposition, voluntary prepayments, MI Recoveries, non-MI recoveries, expenses, current deferred UPB and due date of last paid installment.\"[1]\n",
    "\n",
    "We're going to use machine learning with H2O-3 to predict the interest rate for each loan. To do this, we will build two regression models: an XGBoost model and a Deep Learning model that will help us find the interest rate that a loan should be assigned. Complete this tutorial to see how we achieved those results.\n",
    "\n",
    "We will start by importing H2O, the estimators for the algorithms that we will use, and also the function to perform grid search on those algorithms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Import H2O and other libraries that will be used in this tutorial \n",
    "import h2o\n",
    "import matplotlib as plt\n",
    "%matplotlib inline\n",
    "\n",
    "#Import the Estimators\n",
    "from h2o.estimators.deeplearning import H2ODeepLearningEstimator\n",
    "from h2o.estimators import H2OXGBoostEstimator\n",
    "\n",
    "#Import h2o grid search \n",
    "import h2o.grid \n",
    "from h2o.grid.grid_search import H2OGridSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import h2o\n",
    "\n",
    "startup  = '/home/h2o/bin/aquarium_startup'\n",
    "shutdown = '/home/h2o/bin/aquarium_stop'\n",
    "\n",
    "if os.path.exists(startup):\n",
    "    os.system(startup)\n",
    "    local_url = 'http://localhost:54321/h2o'\n",
    "    aquarium = True\n",
    "else:\n",
    "    local_url = 'http://localhost:54321'\n",
    "    aquarium = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking whether there is an H2O instance running at http://localhost:54321 . connected.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td>H2O_cluster_uptime:</td>\n",
       "<td>57 mins 22 secs</td></tr>\n",
       "<tr><td>H2O_cluster_timezone:</td>\n",
       "<td>America/Los_Angeles</td></tr>\n",
       "<tr><td>H2O_data_parsing_timezone:</td>\n",
       "<td>UTC</td></tr>\n",
       "<tr><td>H2O_cluster_version:</td>\n",
       "<td>3.30.1.1</td></tr>\n",
       "<tr><td>H2O_cluster_version_age:</td>\n",
       "<td>2 days </td></tr>\n",
       "<tr><td>H2O_cluster_name:</td>\n",
       "<td>H2O_from_python_bokhy_ughxpn</td></tr>\n",
       "<tr><td>H2O_cluster_total_nodes:</td>\n",
       "<td>1</td></tr>\n",
       "<tr><td>H2O_cluster_free_memory:</td>\n",
       "<td>2.943 Gb</td></tr>\n",
       "<tr><td>H2O_cluster_total_cores:</td>\n",
       "<td>8</td></tr>\n",
       "<tr><td>H2O_cluster_allowed_cores:</td>\n",
       "<td>8</td></tr>\n",
       "<tr><td>H2O_cluster_status:</td>\n",
       "<td>locked, healthy</td></tr>\n",
       "<tr><td>H2O_connection_url:</td>\n",
       "<td>http://localhost:54321</td></tr>\n",
       "<tr><td>H2O_connection_proxy:</td>\n",
       "<td>{\"http\": null, \"https\": null}</td></tr>\n",
       "<tr><td>H2O_internal_security:</td>\n",
       "<td>False</td></tr>\n",
       "<tr><td>H2O_API_Extensions:</td>\n",
       "<td>Amazon S3, Algos, AutoML, Core V3, TargetEncoder, Core V4</td></tr>\n",
       "<tr><td>Python_version:</td>\n",
       "<td>3.7.6 final</td></tr></table></div>"
      ],
      "text/plain": [
       "--------------------------  ---------------------------------------------------------\n",
       "H2O_cluster_uptime:         57 mins 22 secs\n",
       "H2O_cluster_timezone:       America/Los_Angeles\n",
       "H2O_data_parsing_timezone:  UTC\n",
       "H2O_cluster_version:        3.30.1.1\n",
       "H2O_cluster_version_age:    2 days\n",
       "H2O_cluster_name:           H2O_from_python_bokhy_ughxpn\n",
       "H2O_cluster_total_nodes:    1\n",
       "H2O_cluster_free_memory:    2.943 Gb\n",
       "H2O_cluster_total_cores:    8\n",
       "H2O_cluster_allowed_cores:  8\n",
       "H2O_cluster_status:         locked, healthy\n",
       "H2O_connection_url:         http://localhost:54321\n",
       "H2O_connection_proxy:       {\"http\": null, \"https\": null}\n",
       "H2O_internal_security:      False\n",
       "H2O_API_Extensions:         Amazon S3, Algos, AutoML, Core V3, TargetEncoder, Core V4\n",
       "Python_version:             3.7.6 final\n",
       "--------------------------  ---------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "h2o.init(url=local_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "#Import the dataset \n",
    "loan_level = h2o.import_file(\"https://s3.amazonaws.com/data.h2o.ai/DAI-Tutorials/loan_level_500k.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regression Analysis/Regressor Model\n",
    "Regression tries to predict a continuous number (as different from classification, which only categorizes). With a regressor model, you try to predict the exact number from your response column. In our case, we will try to predict the interest rate. You will see later on that some samples might have a 7.5% interest rate, and our regression model will try to predict that number.\n",
    "\n",
    "There are different types of regression analysis; for example, there is a linear regression, where you fit a straight line to your response based on your predictors, and you try to predict your output with it. Linear regressions are only advised for datasets with a linear distribution. There are also non-linear regressions, which are more useful when you do not have a linear distribution of your response variable.\n",
    "\n",
    "For regression use cases, H2O supports the following metrics:\n",
    "\n",
    "- R Squared (R2)\n",
    "- Mean Squared Error (MSE)\n",
    "- Root Mean Squared Error (RMSE)\n",
    "- Root Mean Squared Logarithmic Error (RMSLE)\n",
    "- Mean Absolute Error (MAE)\n",
    "- In this tutorial, we'll just focus on the RMSE and MAE.\n",
    "\n",
    "#### MSE\n",
    "The MSE metric measures the average of the squares of the errors or deviations. MSE takes the distances from the points to the regression line (these distances are the \"errors\") and squaring them to remove any negative signs. MSE incorporates both the variance and the bias of the predictor.\n",
    "MSE also gives more weight to larger differences. The bigger the error, the more it is penalized. For example, if your correct answers are 2,3,4 and the algorithm guesses 1,4,3, then the absolute error on each one is exactly 1, so squared error is also 1, and the MSE is 1. But if the algorithm guesses 2,3,6, then the errors are 0,0,2, the squared errors are 0,0,4, and the MSE is a higher 1.333.\n",
    "\n",
    "#### RMSE\n",
    "The RMSE metric evaluates how well a model can predict a continuous value. The RMSE units are the same as the predicted target, which is useful for understanding if the size of the error is of concern or not. The smaller the RMSE, the better the model's performance. RMSE is sensitive to outliers, meaning that when the error is larger, the more it is penalized.\n",
    "\n",
    "#### MAE\n",
    "The MAE is an average of the absolute errors. The MAE units are the same as the predicted target as well as the RMSW, which is also useful for understanding whether the size of the error is of concern or not. The smaller the MAE, the better the model's performance. MAE is robust to outliers, meaning all errors are penalized equally."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Look at the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  CREDIT_SCORE</th><th style=\"text-align: right;\">  FIRST_PAYMENT_DATE</th><th>FIRST_TIME_HOMEBUYER_FLAG  </th><th style=\"text-align: right;\">  MATURITY_DATE</th><th style=\"text-align: right;\">  METROPOLITAN_STATISTICAL_AREA</th><th style=\"text-align: right;\">  MORTGAGE_INSURANCE_PERCENTAGE</th><th style=\"text-align: right;\">  NUMBER_OF_UNITS</th><th>OCCUPANCY_STATUS  </th><th style=\"text-align: right;\">  ORIGINAL_COMBINED_LOAN_TO_VALUE</th><th style=\"text-align: right;\">  ORIGINAL_DEBT_TO_INCOME_RATIO</th><th style=\"text-align: right;\">  ORIGINAL_UPB</th><th style=\"text-align: right;\">  ORIGINAL_LOAN_TO_VALUE</th><th style=\"text-align: right;\">  ORIGINAL_INTEREST_RATE</th><th>CHANNEL  </th><th>PREPAYMENT_PENALTY_MORTGAGE_FLAG  </th><th>PRODUCT_TYPE  </th><th>PROPERTY_STATE  </th><th>PROPERTY_TYPE  </th><th style=\"text-align: right;\">  POSTAL_CODE</th><th>LOAN_SEQUENCE_NUMBER  </th><th>LOAN_PURPOSE  </th><th style=\"text-align: right;\">  ORIGINAL_LOAN_TERM</th><th style=\"text-align: right;\">  NUMBER_OF_BORROWERS</th><th>SELLER_NAME  </th><th>SERVICER_NAME  </th><th>PREPAID  </th><th>DELINQUENT  </th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">           669</td><td style=\"text-align: right;\">              200206</td><td>N                          </td><td style=\"text-align: right;\">         202901</td><td style=\"text-align: right;\">                            nan</td><td style=\"text-align: right;\">                              0</td><td style=\"text-align: right;\">                1</td><td>O                 </td><td style=\"text-align: right;\">                               80</td><td style=\"text-align: right;\">                             33</td><td style=\"text-align: right;\">        162000</td><td style=\"text-align: right;\">                      80</td><td style=\"text-align: right;\">                   7.12 </td><td>R        </td><td>N                                 </td><td>FRM           </td><td>WV              </td><td>SF             </td><td style=\"text-align: right;\">        26100</td><td>F199Q1000004          </td><td>P             </td><td style=\"text-align: right;\">                 320</td><td style=\"text-align: right;\">                    2</td><td>Other sellers</td><td>Other servicers</td><td>TRUE     </td><td>FALSE       </td></tr>\n",
       "<tr><td style=\"text-align: right;\">           732</td><td style=\"text-align: right;\">              199904</td><td>N                          </td><td style=\"text-align: right;\">         202903</td><td style=\"text-align: right;\">                          17140</td><td style=\"text-align: right;\">                              0</td><td style=\"text-align: right;\">                1</td><td>O                 </td><td style=\"text-align: right;\">                               25</td><td style=\"text-align: right;\">                             10</td><td style=\"text-align: right;\">         53000</td><td style=\"text-align: right;\">                      25</td><td style=\"text-align: right;\">                   6.5  </td><td>R        </td><td>N                                 </td><td>FRM           </td><td>OH              </td><td>SF             </td><td style=\"text-align: right;\">        45200</td><td>F199Q1000005          </td><td>N             </td><td style=\"text-align: right;\">                 360</td><td style=\"text-align: right;\">                    1</td><td>Other sellers</td><td>Other servicers</td><td>TRUE     </td><td>FALSE       </td></tr>\n",
       "<tr><td style=\"text-align: right;\">           679</td><td style=\"text-align: right;\">              200208</td><td>N                          </td><td style=\"text-align: right;\">         202902</td><td style=\"text-align: right;\">                          15940</td><td style=\"text-align: right;\">                             30</td><td style=\"text-align: right;\">                1</td><td>O                 </td><td style=\"text-align: right;\">                               91</td><td style=\"text-align: right;\">                             48</td><td style=\"text-align: right;\">        133000</td><td style=\"text-align: right;\">                      91</td><td style=\"text-align: right;\">                   6.75 </td><td>R        </td><td>N                                 </td><td>FRM           </td><td>OH              </td><td>SF             </td><td style=\"text-align: right;\">        44700</td><td>F199Q1000007          </td><td>P             </td><td style=\"text-align: right;\">                 319</td><td style=\"text-align: right;\">                    1</td><td>Other sellers</td><td>Other servicers</td><td>TRUE     </td><td>FALSE       </td></tr>\n",
       "<tr><td style=\"text-align: right;\">           721</td><td style=\"text-align: right;\">              200209</td><td>N                          </td><td style=\"text-align: right;\">         202902</td><td style=\"text-align: right;\">                          38060</td><td style=\"text-align: right;\">                              0</td><td style=\"text-align: right;\">                1</td><td>O                 </td><td style=\"text-align: right;\">                               39</td><td style=\"text-align: right;\">                             13</td><td style=\"text-align: right;\">        174000</td><td style=\"text-align: right;\">                      39</td><td style=\"text-align: right;\">                   6.625</td><td>T        </td><td>N                                 </td><td>FRM           </td><td>AZ              </td><td>SF             </td><td style=\"text-align: right;\">        85200</td><td>F199Q1000013          </td><td>N             </td><td style=\"text-align: right;\">                 318</td><td style=\"text-align: right;\">                    2</td><td>Other sellers</td><td>Other servicers</td><td>TRUE     </td><td>FALSE       </td></tr>\n",
       "<tr><td style=\"text-align: right;\">           618</td><td style=\"text-align: right;\">              200210</td><td>N                          </td><td style=\"text-align: right;\">         202902</td><td style=\"text-align: right;\">                          10420</td><td style=\"text-align: right;\">                             25</td><td style=\"text-align: right;\">                1</td><td>O                 </td><td style=\"text-align: right;\">                               85</td><td style=\"text-align: right;\">                             24</td><td style=\"text-align: right;\">        122000</td><td style=\"text-align: right;\">                      85</td><td style=\"text-align: right;\">                   6.375</td><td>R        </td><td>N                                 </td><td>FRM           </td><td>OH              </td><td>SF             </td><td style=\"text-align: right;\">        44200</td><td>F199Q1000015          </td><td>N             </td><td style=\"text-align: right;\">                 317</td><td style=\"text-align: right;\">                    2</td><td>Other sellers</td><td>Other servicers</td><td>TRUE     </td><td>FALSE       </td></tr>\n",
       "<tr><td style=\"text-align: right;\">           738</td><td style=\"text-align: right;\">              200211</td><td>N                          </td><td style=\"text-align: right;\">         202903</td><td style=\"text-align: right;\">                          10420</td><td style=\"text-align: right;\">                              0</td><td style=\"text-align: right;\">                1</td><td>O                 </td><td style=\"text-align: right;\">                               73</td><td style=\"text-align: right;\">                             44</td><td style=\"text-align: right;\">        218000</td><td style=\"text-align: right;\">                      73</td><td style=\"text-align: right;\">                   6    </td><td>R        </td><td>N                                 </td><td>FRM           </td><td>OH              </td><td>SF             </td><td style=\"text-align: right;\">        44300</td><td>F199Q1000016          </td><td>P             </td><td style=\"text-align: right;\">                 317</td><td style=\"text-align: right;\">                    2</td><td>Other sellers</td><td>Other servicers</td><td>TRUE     </td><td>FALSE       </td></tr>\n",
       "<tr><td style=\"text-align: right;\">           761</td><td style=\"text-align: right;\">              200211</td><td>N                          </td><td style=\"text-align: right;\">         202904</td><td style=\"text-align: right;\">                            nan</td><td style=\"text-align: right;\">                              0</td><td style=\"text-align: right;\">                1</td><td>O                 </td><td style=\"text-align: right;\">                               73</td><td style=\"text-align: right;\">                             31</td><td style=\"text-align: right;\">        138000</td><td style=\"text-align: right;\">                      73</td><td style=\"text-align: right;\">                   6.375</td><td>R        </td><td>N                                 </td><td>FRM           </td><td>SC              </td><td>PU             </td><td style=\"text-align: right;\">        29500</td><td>F199Q1000017          </td><td>P             </td><td style=\"text-align: right;\">                 318</td><td style=\"text-align: right;\">                    2</td><td>Other sellers</td><td>Other servicers</td><td>TRUE     </td><td>FALSE       </td></tr>\n",
       "<tr><td style=\"text-align: right;\">           707</td><td style=\"text-align: right;\">              200211</td><td>N                          </td><td style=\"text-align: right;\">         202903</td><td style=\"text-align: right;\">                          33340</td><td style=\"text-align: right;\">                              0</td><td style=\"text-align: right;\">                1</td><td>O                 </td><td style=\"text-align: right;\">                               60</td><td style=\"text-align: right;\">                             57</td><td style=\"text-align: right;\">        136000</td><td style=\"text-align: right;\">                      60</td><td style=\"text-align: right;\">                   6.25 </td><td>R        </td><td>N                                 </td><td>FRM           </td><td>WI              </td><td>SF             </td><td style=\"text-align: right;\">        53000</td><td>F199Q1000018          </td><td>C             </td><td style=\"text-align: right;\">                 317</td><td style=\"text-align: right;\">                    2</td><td>Other sellers</td><td>Other servicers</td><td>TRUE     </td><td>FALSE       </td></tr>\n",
       "<tr><td style=\"text-align: right;\">           760</td><td style=\"text-align: right;\">              200211</td><td>N                          </td><td style=\"text-align: right;\">         202903</td><td style=\"text-align: right;\">                          33340</td><td style=\"text-align: right;\">                              0</td><td style=\"text-align: right;\">                1</td><td>O                 </td><td style=\"text-align: right;\">                               63</td><td style=\"text-align: right;\">                             30</td><td style=\"text-align: right;\">         79000</td><td style=\"text-align: right;\">                      63</td><td style=\"text-align: right;\">                   6.125</td><td>R        </td><td>N                                 </td><td>FRM           </td><td>WI              </td><td>SF             </td><td style=\"text-align: right;\">        53000</td><td>F199Q1000019          </td><td>N             </td><td style=\"text-align: right;\">                 317</td><td style=\"text-align: right;\">                    2</td><td>Other sellers</td><td>Other servicers</td><td>TRUE     </td><td>FALSE       </td></tr>\n",
       "<tr><td style=\"text-align: right;\">           691</td><td style=\"text-align: right;\">              200302</td><td>N                          </td><td style=\"text-align: right;\">         202901</td><td style=\"text-align: right;\">                          15940</td><td style=\"text-align: right;\">                              0</td><td style=\"text-align: right;\">                1</td><td>O                 </td><td style=\"text-align: right;\">                               65</td><td style=\"text-align: right;\">                             25</td><td style=\"text-align: right;\">        130000</td><td style=\"text-align: right;\">                      65</td><td style=\"text-align: right;\">                   5.875</td><td>R        </td><td>N                                 </td><td>FRM           </td><td>OH              </td><td>SF             </td><td style=\"text-align: right;\">        44700</td><td>F199Q1000023          </td><td>P             </td><td style=\"text-align: right;\">                 312</td><td style=\"text-align: right;\">                    2</td><td>Other sellers</td><td>Other servicers</td><td>TRUE     </td><td>FALSE       </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loan_level.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rows:500137\n",
      "Cols:1\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>       </th><th>ORIGINAL_INTEREST_RATE  </th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>type   </td><td>real                    </td></tr>\n",
       "<tr><td>mins   </td><td>4.625                   </td></tr>\n",
       "<tr><td>mean   </td><td>7.18268686379932        </td></tr>\n",
       "<tr><td>maxs   </td><td>11.5                    </td></tr>\n",
       "<tr><td>sigma  </td><td>0.5799408623980559      </td></tr>\n",
       "<tr><td>zeros  </td><td>0                       </td></tr>\n",
       "<tr><td>missing</td><td>0                       </td></tr>\n",
       "<tr><td>0      </td><td>7.12                    </td></tr>\n",
       "<tr><td>1      </td><td>6.5                     </td></tr>\n",
       "<tr><td>2      </td><td>6.75                    </td></tr>\n",
       "<tr><td>3      </td><td>6.625                   </td></tr>\n",
       "<tr><td>4      </td><td>6.375                   </td></tr>\n",
       "<tr><td>5      </td><td>6.0                     </td></tr>\n",
       "<tr><td>6      </td><td>6.375                   </td></tr>\n",
       "<tr><td>7      </td><td>6.25                    </td></tr>\n",
       "<tr><td>8      </td><td>6.125                   </td></tr>\n",
       "<tr><td>9      </td><td>5.875                   </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "loan_level[\"ORIGINAL_INTEREST_RATE\"].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the description above, we can see that the maximum interest rate value is 11.5%, and the minimum is 4.625%, while the average or mean is 7.183%. Our models will have to predict a numerical value from 4.625 to 11.5 based on the predictors that we choose. Let's take a look at a more graphical representation of our data with a histogram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEXCAYAAACQ3VJYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de7xcZXn28d8lEQWRc6RpAoZDPABqhDRiqRRNhWhVoIKEWomWGkVsPVEF27dQbCyoSOFVsFgiBxGCoIBVihGq1tdwCIpCOJQgEUIiRBJOGsCE6/1jPSNrD7P3niR77cneub6fz3z2mnutZ829Zif7nud51qwl20RERAy15/Q6gYiIGJ1SYCIiohEpMBER0YgUmIiIaEQKTERENCIFJiIiGpECs5GRtFDS/r3Oo5ckHSLpPkmPS3p1r/OJGK1SYEYRSYsl/Vlb7N2SftR6bnsP298fZD8TJVnSmIZS7bXPAR+0vYXtn7avVOXvJd0laZWkeyWdLOl5tW3OlfRUKVIrJM2T9LLa+j7ve4nNkHS9pN9IerAsf0CSavv8l7Lc+h18u20fX5V0YltsZ0lPSzqzw7FY0m7dvjHteZd/Uw9IekEt9jeSvi9pp3L8rYfLsbWev67tfWo9ftZ2jK34YknHteWzuPwO6u2/UNZtKulUSUtK/B5Jp5V19e2fbtvHOwc4/hMl/a5s97CkH0t6bYftzpW0WtIfluefrO3/CUlras8X1n4Xv2nL7ePd/m5GohSYGHYbQOF6MbBwgPVnALOAI4EXAm8C3gBc0rbdZ2xvAYwH7gfO6W+Hkj4GnA58FvgDYAfg/cC+wKYD5LKPpH0HOpiS50pgRr0IDqExwIfag7bvLUV6i/I+ALyqFvufEvtMfTvbr2rb1dal/aHA/5H0xrb1b21r/8ESPx6YAkyl+j29Hvhpya2e171t+7hwkOOdW9ptD/w38PX6ylJs3w48AryzvN6na6/3fmB+7fX2qDV/VduxfGaQXEa0FJiNTL2XI2mqpAWSHi2fUj9fNvth+flw+ZT1WknPkfSPkn5ZPn2fL2mr2n6PLOsekvR/2l7nREmXlk/fjwLvLq89v3xKXCbpC5I2re3P5dP9XZIek/QpSbuWNo9KuqS+fdsxdsxV0vMkPQ5sAvxM0t0d2k4CPgC80/Z826ttL6T6gzJd0hva29heRVV8JveTz1bAScAHbF9q+zFXfmr7nbafHOBX9hngXwZYD1WB+Ufgd8BbB9l2XXwWOFbS1g3s+/dsL6Aq/B3fxw7+CPim7aXl/Vxs+/whzGc1cCEwXtLY2qq3Aw9T/U5nDtXrjUYpMBu304HTbW8J7Mozn9D3Kz+3Lp+y5gPvLo/XA7sAWwCtoYrdgTOpPs2NA7ai+lRfdxBwKbA11X/aNcBHqD4lvhaYRvWHvW46sDewD/Bx4OzyGjsCewJH9HNcHXO1/WTbJ+1dO7SdBiyxfUM9aPs+4Dqg/dN16xPtEcCifvJ5LfA84Ip+1g/ki8BL1Db0WXvt1wETgIupfn9HrsNrDGYB8H3g2Ab2/XuS9qH6vfb3Pra7Dvho+SDyCqkaahzCfDalej8fouohtswELqJ6z18maa+hfN3RJAVm9Lm89AoelvQw1R/+/vwO2E3S9rYft33dANu+E/i87V/YfpxqeGJGGe46FPiW7R/Zfgr4J6D9InfzbV9u+2nbq2zfZPu60kNYDPw78KdtbU6x/WjpQdwKfLe8/iPAVUB/E/QD5TqY7YFl/axbVta3HFve48eAPwHeNcA+f10+EQNQxvYfLnMD+/XTDuAJYDb992JmAlfZXgl8DXiTpBcNsL919U/A37Z9ku/WsfV/k5LOa1v/a0mrgPlU/14vb1t/eVv795b4vwKnUP2+FwD3SxqKHsU7yu91FfBe4NDW707STlQfXL5m+wHgGtauF/OTtmM5cAjy3WClwIw+B9veuvXg2b2CuqOAlwB3SLpR0lsG2PYPgV/Wnv+Samx+h7LuvtYK27+l+tRXd1/9iaSXSPpPSb8qw2afpu8fb4AHasurOjzfgs4GynUwv6bqhXUyrqxv+Vx5jyeWfF7aT7uHgO3rBc72H5e2DzH4/8MvAztI6jP8JWkz4DCqHiGlp3kv8JeD7G+t2b4V+E/guMG27eBz9X+Tttv/IG9P9bs8FtgfeG7b+oPb2n+55LTG9hdt70vVM54NzJH08nXIse6S8rvZgeqDzd61de8Cbrd9c3l+IfCXktpz7s9ebcdy9XrmukFLgdmI2b7L9hHAi6g+CV5ahns6XWJ7KdXkeMtOwGqqP/rLqIZpgN//4duu/eXanp8F3AFMKkN0nwSGaohjoFwHcy2wo6Sp9aCkHamG6q5pb2D7XqpJ8NPLsbebDzxJNUy41mz/Dvhn4FP0fY8OAbYEziyF+ldUQ5NNDJMBnED1ib59+HO9lWJxKlWPbaAPRf21X2X7i1RDWbsPUU6/Bt4HnCip9aHjSGCX2vv9eaoC+aaheM3RJgVmIybprySNtf001aQlVHMjy4GnqeYvWi4CPqLqlNgtqHocc8vQwaXAWyX9cRm3/mcGLxYvBB4FHld1eu/RQ3ZgA+c6INv/C3wJuFDSPpI2kbQHcBnwPdvf66fdPKrCNqvDuoep3pMzJR0qaYtyIsJk4AXt2/fjAqp5nOm12ExgDvAKqonxyVRnpU2W9IradptKen7tsUmXr9l+HIuAucDfrUv7Lp0MfFzS8wfbUNKHJe0vaTNJY8rw2AspZ5INBdt3AFeXnF5LNVc5lWfe7z2phiYz2d9BCszGbTqwsJxZdToww/YTZYhrNvD/yjjxPlR/yC6gOsPsHqpPmn8LUOZI/pZq0nMZ1ZzEg1Sf2vtzLNVQzmNUQ0Bzh/C4+s21Sx8E/gP4KvA48F9Uk9xvH6TdZ6n+ED3rVOFyOupHqU5WeJCqN/XvwCeAHw+WkO01VD2IbQEkjac6IeHfbP+q9rip5Fv/g7eQagiv9XjPYK83gJPovii2fFx9v/vx6wG2/TZVL+S9tdi32tp/s8RXAacCv6IaujwGeLvtX6xlfoP5LNUHh/cCV9i+pf6eU/3feYukbbvY18/ajuXfhjjXDYqcG47FECu9hoephr/u6XU+EdEb6cHEkJD0VkmblzmczwG3AIt7m1VE9FIKTAyVg6jmIJYCk6iG29I93gBJ+lLbME3r8aVe5zYcJF3Vz/F/ste5jTYZIouIiEakBxMREY3o9UUHNxjbb7+9J06c2Os0IiJGlJtuuunXtjte4SEFppg4cSILFizodRoRESOKpF/2ty5DZBER0YgUmIiIaEQKTERENCIFJiIiGpECExERjUiBiYiIRqTAREREI1JgIiKiESkwERHRiHyTP0aFicd9e73aLz75z4cok4hoSQ8mIiIakQITERGNSIGJiIhGpMBEREQjUmAiIqIRjRUYSTtK+m9Jt0taKOlDJb6tpHmS7io/t6m1OV7SIkl3SjqwFt9b0i1l3RmSVOLPkzS3xK+XNLHWZmZ5jbskzWzqOCMiorMmezCrgY/ZfjmwD3CMpN2B44BrbE8CrinPKetmAHsA04EzJW1S9nUWMAuYVB7TS/woYKXt3YDTgFPKvrYFTgBeA0wFTqgXsoiIaF5jBcb2Mts/KcuPAbcD44GDgPPKZucBB5flg4CLbT9p+x5gETBV0jhgS9vzbRs4v61Na1+XAtNK7+ZAYJ7tFbZXAvN4pihFRMQwGJY5mDJ09WrgemAH28ugKkLAi8pm44H7as2WlNj4stwe79PG9mrgEWC7AfbVntcsSQskLVi+fPm6H2BERDxL4wVG0hbAZcCHbT860KYdYh4gvq5tngnYZ9ueYnvK2LFjB0gtIiLWVqMFRtJzqYrLhba/UcIPlGEvys8HS3wJsGOt+QRgaYlP6BDv00bSGGArYMUA+4qIiGHS5FlkAs4Bbrf9+dqqK4HWWV0zgStq8RnlzLCdqSbzbyjDaI9J2qfs88i2Nq19HQpcW+ZprgYOkLRNmdw/oMQiImKYNHmxy32BdwG3SLq5xD4JnAxcIuko4F7gMADbCyVdAtxGdQbaMbbXlHZHA+cCmwFXlQdUBewCSYuoei4zyr5WSPoUcGPZ7iTbK5o60IiIeLbGCoztH9F5LgRgWj9tZgOzO8QXAHt2iD9BKVAd1s0B5nSbb0REDK18kz8iIhqRAhMREY1IgYmIiEakwERERCNSYCIiohEpMBER0YgUmIiIaEQKTERENCIFJiIiGpECExERjUiBiYiIRqTAREREI1JgIiKiESkwERHRiBSYiIhoRApMREQ0oslbJs+R9KCkW2uxuZJuLo/FrTtdSpooaVVt3ZdqbfaWdIukRZLOKLdNptxaeW6JXy9pYq3NTEl3lcdMIiJi2DV5y+RzgS8A57cCtg9vLUs6FXiktv3dtid32M9ZwCzgOuA7wHSqWyYfBay0vZukGcApwOGStgVOAKYABm6SdKXtlUN4bBERMYjGejC2fwis6LSu9ELeAVw00D4kjQO2tD3ftqmK1cFl9UHAeWX5UmBa2e+BwDzbK0pRmUdVlCIiYhj1ag7mdcADtu+qxXaW9FNJP5D0uhIbDyypbbOkxFrr7gOwvZqqN7RdPd6hTR+SZklaIGnB8uXL1/eYIiKiplcF5gj69l6WATvZfjXwUeBrkrYE1KGty8/+1g3Upm/QPtv2FNtTxo4d23XyERExuGEvMJLGAH8BzG3FbD9p+6GyfBNwN/ASqt7HhFrzCcDSsrwE2LG2z62ohuR+H+/QJiIihkkvejB/Btxh+/dDX5LGStqkLO8CTAJ+YXsZ8Jikfcr8ypHAFaXZlUDrDLFDgWvLPM3VwAGStpG0DXBAiUVExDBq7CwySRcB+wPbS1oCnGD7HGAGz57c3w84SdJqYA3wftutEwSOpjojbTOqs8euKvFzgAskLaLqucwAsL1C0qeAG8t2J9X2FRERw6SxAmP7iH7i7+4Quwy4rJ/tFwB7dog/ARzWT5s5wJy1SDciIoZYvskfERGNSIGJiIhGpMBEREQjUmAiIqIRKTAREdGIFJiIiGhECkxERDQiBSYiIhqRAhMREY1IgYmIiEakwERERCNSYCIiohEpMBER0YgUmIiIaEQKTERENCIFJiIiGtFYgZE0R9KDkm6txU6UdL+km8vjzbV1x0taJOlOSQfW4ntLuqWsO6PcOhlJz5M0t8SvlzSx1mampLvKo3Vb5YiIGEZN9mDOBaZ3iJ9me3J5fAdA0u5Utzzeo7Q5U9ImZfuzgFnApPJo7fMoYKXt3YDTgFPKvrYFTgBeA0wFTpC0zdAfXkREDKSxAmP7h8CKLjc/CLjY9pO27wEWAVMljQO2tD3ftoHzgYNrbc4ry5cC00rv5kBgnu0VtlcC8+hc6CIiokG9mIP5oKSflyG0Vs9iPHBfbZslJTa+LLfH+7SxvRp4BNhugH09i6RZkhZIWrB8+fL1O6qIiOhjuAvMWcCuwGRgGXBqiavDth4gvq5t+gbts21PsT1l7NixA+UdERFraVgLjO0HbK+x/TTwZao5Eqh6GTvWNp0ALC3xCR3ifdpIGgNsRTUk19++IiJiGA1rgSlzKi2HAK0zzK4EZpQzw3ammsy/wfYy4DFJ+5T5lSOBK2ptWmeIHQpcW+ZprgYOkLRNGYI7oMQiImIYjWlqx5IuAvYHtpe0hOrMrv0lTaYasloMvA/A9kJJlwC3AauBY2yvKbs6muqMtM2Aq8oD4BzgAkmLqHouM8q+Vkj6FHBj2e4k292ebBAREUOksQJj+4gO4XMG2H42MLtDfAGwZ4f4E8Bh/exrDjCn62QjImLI5Zv8ERHRiBSYiIhoRApMREQ0IgUmIiIakQITERGNSIGJiIhGpMBEREQjUmAiIqIRKTAREdGIFJiIiGhEVwVG0rMu1RIRETGQbnswX5J0g6QPSNq60YwiImJU6KrA2P4T4J1U91lZIOlrkt7YaGYRETGidT0HY/su4B+BTwB/Cpwh6Q5Jf9FUchERMXJ1OwfzSkmnAbcDbwDeavvlZfm0BvOLiIgRqtsezBeAnwCvsn2M7Z8A2F5K1at5FklzJD0o6dZa7LOl1/NzSd9szedImihplaSby+NLtTZ7S7pF0iJJZ5Q7W1Lufjm3xK+XNLHWZqaku8pjJhERMey6LTBvBr5mexWApOdI2hzA9gX9tDkXmN4WmwfsafuVwP8Cx9fW3W17cnm8vxY/C5hFdRvlSbV9HgWstL0bVS/qlJLbtlR3z3wNMBU4odw6OSIihlG3BeZ7VLcsbtm8xPpl+4dUtzKux75re3V5eh0wYaB9SBoHbGl7vm0D5wMHl9UHAeeV5UuBaaV3cyAwz/YK2yupilp7oYuIiIZ1W2Ceb/vx1pOyvPl6vvZfA1fVnu8s6aeSfiDpdSU2HlhS22ZJibXW3VfyWQ08AmxXj3do04ekWZIWSFqwfPny9TyciIio67bA/EbSXq0nkvYGVq3ri0r6B2A1cGEJLQN2sv1q4KPA1yRtCahDc7d208+6gdr0Ddpn255ie8rYsWPX5hAiImIQY7rc7sPA1yUtLc/HAYevywuWSfe3ANPKsBe2nwSeLMs3SbobeAlV76M+jDYBaOWwhOp7OUskjQG2ohqSWwLs39bm++uSa0RErLtuv2h5I/Ay4GjgA8DLbd+0ti8maTrV92jeZvu3tfhYSZuU5V2oJvN/YXsZ8Jikfcr8ypHAFaXZlUDrDLFDgWtLwboaOEDSNmVy/4ASi4iIYdRtDwbgj4CJpc2rJWH7/P42lnQRVU9ie0lLqM7sOh54HjCvnG18XTljbD/gJEmrgTXA+223ThA4muqMtM2o5mxa8zbnABdIWkTVc5kBYHuFpE8BN5btTqrtKyIihklXBUbSBcCuwM1UBQCqeY1+C4ztIzqEz+ln28uAy/pZtwB41sU2bT8BHNZPmznAnP5yi4iI5nXbg5kC7N6aM4mIiBhMt2eR3Qr8QZOJRETE6NJtD2Z74DZJN1DO9gKw/bZGsoqIiBGv2wJzYpNJRETE6NNVgbH9A0kvBibZ/l65DtkmzaYWEREjWbeX638v1fW+/r2ExgOXN5VURESMfN1O8h8D7As8Cr+/+diLmkoqIiJGvm4LzJO2n2o9KZdmySnLERHRr24LzA8kfRLYTNIbga8D32ourYiIGOm6LTDHAcuBW4D3Ad+hnztZRkREQPdnkT0NfLk8IiIiBtXttcjuocOci+1dhjyjiIgYFdbmWmQtz6e6yOS2Q59ORESMFt3eD+ah2uN+2/8GvKHh3CIiYgTrdohsr9rT51D1aF7YSEYRETEqdDtEdmpteTWwGHjHkGcTERGjRrdDZK+vPd5o+7227xyojaQ5kh6UdGsttq2keZLuKj+3qa07XtIiSXdKOrAW31vSLWXdGeXWyUh6nqS5JX69pIm1NjPLa9wlqXVb5YiIGEbdDpF9dKD1tj/fIXwu8AX63vXyOOAa2ydLOq48/4Sk3aluebwH8IfA9yS9xPYa4CxgFnAd1fdvplPdNvkoYKXt3STNAE4BDpe0LdXtmadQnfl2k6Qrba/s5lgjImJodPtFyynA0VQXuRwPvB/YnWoepuNcjO0fAivawgcB55Xl84CDa/GLbT9p+x5gETBV0jhgS9vzy900z29r09rXpcC00rs5EJhne0UpKvOoilJERAyjtbnh2F62HwOQdCLwddt/s5avt4PtZQC2l0lqXTBzPFUPpWVJif2uLLfHW23uK/taLekRYLt6vEObPiTNouodsdNOO63loURExEC67cHsBDxVe/4UMHEI81CHmAeIr2ubvkH7bNtTbE8ZO3ZsV4lGRER3uu3BXADcIOmbVH+sD6Hv3Eq3HpA0rvRexgEPlvgSYMfadhOApSU+oUO83mZJubrzVlRDckuA/dvafH8dco2IiPXQ7Vlks4H3ACuBh4H32P70OrzelUDrrK6ZwBW1+IxyZtjOwCTghjKc9pikfcr8ypFtbVr7OhS4tszTXA0cIGmbcpbaASUWERHDqNseDMDmwKO2vyJprKSdy4R8R5IuoupJbC9pCdWZXScDl0g6CriX6pIz2F4o6RLgNqrv2RxTziCD6uSCc4HNqM4eu6rEzwEukLSIqucyo+xrhaRPATeW7U6y3X6yQURENKzb05Rbp/2+FPgK8Fzgq1R3uezI9hH9rJrWz/azgdkd4guAPTvEn6AUqA7r5gBz+sstIiKa1+0k/yHA24DfANheSi4VExERA+i2wDxV5jcMIOkFzaUUERGjQbcF5hJJ/w5sLem9wPfIzcciImIAg87BlLO35gIvAx6lmof5J9vzGs4tIiJGsEELjG1Lutz23lSXXYmIiBhUt0Nk10n6o0YziYiIUaXb78G8Hni/pMVUZ5KJqnPzyqYSi4iIkW3AAiNpJ9v3Am8apnwiemLicd9er/aLT/7zIcokYvQYrAdzOdVVlH8p6TLbbx+OpCIiYuQbbA6mfmXiXZpMJCIiRpfBCoz7WY6IiBjQYENkr5L0KFVPZrOyDM9M8m/ZaHYRETFiDVhgbG8yXIlERMTo0u33YCIiItZKCkxERDQiBSYiIhox7AVG0ksl3Vx7PCrpw5JOlHR/Lf7mWpvjJS2SdKekA2vxvSXdUtadUS7MSbn18twSv17SxOE+zoiIjd2wFxjbd9qebHsysDfwW+CbZfVprXW2vwMgaXeq2yHvAUwHzpTUOvngLGAWMKk8ppf4UcBK27sBpwGnDMOhRURETa+HyKYBd9v+5QDbHARcbPtJ2/cAi4CpksYBW9qeX26Gdj5wcK3NeWX5UmBaq3cTERHDo9cFZgZwUe35ByX9XNIcSduU2Hjgvto2S0psfFluj/dpY3s18AiwXfuLS5olaYGkBcuXLx+K44mIiKJnBUbSpsDbgK+X0FnArsBkYBlwamvTDs09QHygNn0D9tm2p9ieMnbs2LXIPiIiBtPLHsybgJ/YfgDA9gO219h+mup2zFPLdkuAHWvtJgBLS3xCh3ifNpLGAFsBKxo6joiI6KCXBeYIasNjZU6l5RDg1rJ8JTCjnBm2M9Vk/g22lwGPSdqnzK8cCVxRazOzLB8KXFvmaSIiYph0e8OxISVpc+CNwPtq4c9Imkw1lLW4tc72QkmXALcBq4FjbK8pbY4GzgU2A64qD4BzgAskLaLqucxo8ngiIuLZelJgbP+Wtkl32+8aYPvZwOwO8QXAnh3iTwCHrX+mERGxrnp9FllERIxSKTAREdGIFJiIiGhECkxERDQiBSYiIhqRAhMREY1IgYmIiEakwERERCNSYCIiohEpMBER0YgUmIiIaEQKTERENKInF7uMaDfxuG/3OoWIGGIpMBFDYH0L5OKT/3yIMonYcGSILCIiGtGTAiNpsaRbJN0saUGJbStpnqS7ys9tatsfL2mRpDslHViL7132s0jSGeXOlpS7X84t8eslTRzuY4yI2Nj1sgfzetuTbU8pz48DrrE9CbimPEfS7lR3pNwDmA6cKWmT0uYsYBbVbZQnlfUARwErbe8GnAacMgzHExERNRvSENlBwHll+Tzg4Fr8YttP2r4HWARMlTQO2NL2fNsGzm9r09rXpcC0Vu8mIiKGR68KjIHvSrpJ0qwS28H2MoDy80UlPh64r9Z2SYmNL8vt8T5tbK8GHqHtFs0REdGsXp1Ftq/tpZJeBMyTdMcA23bqeXiA+EBt+u64Km6zAHbaaaeBM46IiLXSkx6M7aXl54PAN4GpwANl2Ivy88Gy+RJgx1rzCcDSEp/QId6njaQxwFbAig55nG17iu0pY8eOHZqDi4gIoAcFRtILJL2wtQwcANwKXAnMLJvNBK4oy1cCM8qZYTtTTebfUIbRHpO0T5lfObKtTWtfhwLXlnmaiIgYJr0YItsB+GaZcx8DfM32f0m6EbhE0lHAvcBhALYXSroEuA1YDRxje03Z19HAucBmwFXlAXAOcIGkRVQ9lxnDcWAREfGMYS8wtn8BvKpD/CFgWj9tZgOzO8QXAHt2iD9BKVAREdEbG9JpyhERMYqkwERERCNSYCIiohEpMBER0YgUmIiIaEQKTERENCIFJiIiGpECExERjUiBiYiIRqTAREREI1JgIiKiESkwERHRiBSYiIhoRK/uaBkRQ2jicd9er/aLT/7zIcok4hnpwURERCPSg4nYAKxvDyRiQ9SLWybvKOm/Jd0uaaGkD5X4iZLul3Rzeby51uZ4SYsk3SnpwFp8b0m3lHVnlFsnU26vPLfEr5c0cbiPMyJiY9eLIbLVwMdsvxzYBzhG0u5l3Wm2J5fHdwDKuhnAHsB04ExJm5TtzwJmAZPKY3qJHwWstL0bcBpwyjAcV0RE1Ax7gbG9zPZPyvJjwO3A+AGaHARcbPtJ2/cAi4CpksYBW9qeb9vA+cDBtTbnleVLgWmt3k1ERAyPnk7yl6GrVwPXl9AHJf1c0hxJ25TYeOC+WrMlJTa+LLfH+7SxvRp4BNiuw+vPkrRA0oLly5cPyTFFRESlZwVG0hbAZcCHbT9KNdy1KzAZWAac2tq0Q3MPEB+oTd+AfbbtKbanjB07di2PICIiBtKTAiPpuVTF5ULb3wCw/YDtNbafBr4MTC2bLwF2rDWfACwt8Qkd4n3aSBoDbAWsaOZoIiKik16cRSbgHOB225+vxcfVNjsEuLUsXwnMKGeG7Uw1mX+D7WXAY5L2Kfs8Erii1mZmWT4UuLbM00RExDDpxfdg9gXeBdwi6eYS+yRwhKTJVENZi4H3AdheKOkS4DaqM9COsb2mtDsaOBfYDLiqPKAqYBdIWkTVc5nR8DFFRESbYS8wtn9E5zmS7wzQZjYwu0N8AbBnh/gTwGHrkWZERKynXComIiIakQITERGNSIGJiIhG5GKXEZHL/Ucj0oOJiIhGpMBEREQjUmAiIqIRKTAREdGIFJiIiGhECkxERDQiBSYiIhqRAhMREY3IFy0jYr3li5rRSQpMDIn1/QMTEaNPhsgiIqIRKTAREdGIUV1gJE2XdKekRZKO63U+EREbk1FbYCRtAnwReBOwO9UtmXfvbVYRERuP0TzJPxVYZPsXAJIuBg4CbutpVhuoTNJHL/X631/OYmvGaC4w44H7as+XAK+pbyBpFjDYz+cAAAh5SURBVDCrPH1c0p2D7HN74NdDluHwGqm5j9S8Ibn3wjrlrVMayGTtjdT3/MX9rRjNBUYdYu7zxD4bOLvrHUoLbE9Z38R6YaTmPlLzhuTeCyM1bxjZufdn1M7BUPVYdqw9nwAs7VEuEREbndFcYG4EJknaWdKmwAzgyh7nFBGx0Ri1Q2S2V0v6IHA1sAkwx/bC9dxt18NpG6CRmvtIzRuSey+M1LxhZOfekWwPvlVERMRaGs1DZBER0UMpMBER0YgUmC5JWizpFkk3S1rQ63y6JWlrSZdKukPS7ZJe2+ucuiHppeW9bj0elfThXufVDUkfkbRQ0q2SLpL0/F7n1C1JHyp5L9zQ329JcyQ9KOnWWmxbSfMk3VV+btPLHDvpJ+/Dynv+tKRRc6pyCszaeb3tySPsXPXTgf+y/TLgVcDtPc6nK7bvLO/1ZGBv4LfAN3uc1qAkjQf+Dphie0+qE0xm9Dar7kjaE3gv1VUwXgW8RdKk3mY1oHOB6W2x44BrbE8CrinPNzTn8uy8bwX+AvjhsGfToBSYUUzSlsB+wDkAtp+y/XBvs1on04C7bf+y14l0aQywmaQxwOaMnO9fvRy4zvZvba8GfgAc0uOc+mX7h8CKtvBBwHll+Tzg4GFNqgud8rZ9u+3BriQy4qTAdM/AdyXdVC4xMxLsAiwHviLpp5L+Q9ILep3UOpgBXNTrJLph+37gc8C9wDLgEdvf7W1WXbsV2E/SdpI2B95M3y8rjwQ72F4GUH6+qMf5bNRSYLq3r+29qK7OfIyk/XqdUBfGAHsBZ9l+NfAbNswhg36VL8m+Dfh6r3PpRhnzPwjYGfhD4AWS/qq3WXXH9u3AKcA84L+AnwGre5pUjGgpMF2yvbT8fJBqLmBqbzPqyhJgie3ry/NLqQrOSPIm4Ce2H+h1Il36M+Ae28tt/w74BvDHPc6pa7bPsb2X7f2ohnHu6nVOa+kBSeMAys8He5zPRi0FpguSXiDpha1l4ACq4YQNmu1fAfdJemkJTWPk3a7gCEbI8FhxL7CPpM0lieo9HxEnVgBIelH5uRPVpPNIeu+huhzUzLI8E7iih7ls9PJN/i5I2oVnzmAaA3zN9uweptQ1SZOB/wA2BX4BvMf2yt5m1Z0yD3AfsIvtR3qdT7ck/TNwONXw0k+Bv7H9ZG+z6o6k/wG2A34HfNT2NT1OqV+SLgL2p7rM/QPACcDlwCXATlTF/jDb7ScC9FQ/ea8A/i8wFngYuNn2gb3KcaikwERERCMyRBYREY1IgYmIiEakwERERCNSYCIiohEpMBER0YgUmIiIaEQKTIwIkiZIuqJchv1uSadL2lTS/pIeKddau0PS52pt3i3pC7XnfyXp5+Wy6D8r12bbuqz7fusy6eXWDJfV2h0q6dy2fK6QNL8tdqKkY7s8nsfLz4mSLOlva+u+UHL/YrlVwW2SVtVuXXCopHMl3VOL/bh2zMtL7A5JH2nL7/622yBsXb4UeqGq21HcKulHkl5c2+ZXbe027eeY1pT1t0r6Vuu9ra3/WfkOCJLeU9vfU3rmVhgntx1D67F7N+9rbFhSYGKDV74R/w3g8nIZ9pcAWwCtL7v+T7nW2qupLjG/b4d9TAc+ArzJ9h5Ul8z5MbBDPy87RdIe/eSzdWm/taSd1/3Ifu9B4EPtf7htH1NuV/BmqqtJTy6PS8smf1+L1S9HM7e02xf4B0n1C1aeVmszuVxd+0PAA7ZfUW4xcBTwq9rtEr7U1u6pfo5jVVm/J9UXB49prZD0cqq/N/tJeoHtr9T2v5RnboXRulbe3LY8R9oVKIIUmBgZ3gA8YfsrALbXUBWLv6a6HD4lvgq4GRjfYR//ABxbrnaM7TW25wxwifTPAZ/sZ93bgW8BFzM093pZTnXvkpmDbbg2bD8ELALGDbLpOOD+Wrs7h+DKA/Pp+3v4S+AC4LtUFy+NjUAKTIwEewA31QO2H6W6FMhurVi5kvEkOt+0aQ/gJ2vxmpcAe0narcO61vXRLirLQ+Fk4GOSNlmLNp+tDSFd2L6yXE/s+cDPa+GP1Nr8d4nNAT4hab6kf9F63mSsHMM0quuCtRwOzKX79+zwtiGyzdYnp+iNFJgYCUR1P57+4q+T9HPgV8B/lot89r8z6RXlj9bdkg7vZ7M1wGeB49va7kBV1H5k+3+B1aruBLlebN8D3ED1Sb9b9SGyd9bih0taSHXtudNtP1FbVx/qen157Zup7h30WWBb4MYypLW2NpN0M/BQ2c88AEl/BCwvN4y7hqpwD3Yr4/YhslXrkE/0WApMjAQLgT63qVZ1t84dgbup5mBeCbwCOFrVBT477WMvANu3lLH/q4CBPhlfQHVH0J1qscOBbYB7JC0GJjJ0t0T+NPAJ1v//5dwyz/Q64FRJfzBYA9uP2/6G7Q8AX6Wa91lbq8r7+mKqi6u25mCOAF5W3q+7gS2phhljlEuBiZHgGmBzSUfC74dgTqW6t/lvWxuVHsW/Uv2RbvevwOckTajFBhx2KfdzOQ34cC18BDDd9kTbE4G9GaICY/sOqtspvGWI9jefqkh+aKDtJO3b6lGUEw12B9b59tTlytd/Bxwr6XnAYcAra+/ZQQzd0GJswFJgYoPn6pLfhwCHSboL+F/gCTpPwn+J6kylPmd32f4OcAZwVTnt98dUw2BXD/Ly51DdogFJE6l6M9fV9nsP8Kik15TQP0pa0nqs1YFWZgMTBt2qUp+D6e/04VOA96jcz4i+czA3l2PaFfiBpFuobi+wALisw766ZvunVHfEfAdwf+vkiuKHwO4qNwbrR/sczIi5aVs8I5frj4iIRqQHExERjRjT6wQiRitJ21HNH7WbVr6jMuKMxmOK5mSILCIiGpEhsoiIaEQKTERENCIFJiIiGpECExERjfj/7vlsF7KUZnEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "loan_level[\"ORIGINAL_INTEREST_RATE\"].hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The histogram verifies that the interest rate around 7 and 7.5 is the most frequent one. Now that we have an idea of what our data and our response variable looks like, we can split the dataset.\n",
    "\n",
    "**Please note that the main goal of this tutorial is to show the usage of some models for regression problems, as well as to tune some of the hyper-parameters of the models. For that reason, we will be skipping any data visualization and manipulation, as well as feature engineering. The aforementioned stages in machine learning are very important, and should always be done; however, they will be covered in later tutorials.**\n",
    "\n",
    "We will split the dataset into three sets, training, validation, and test set. The reason for having validation and test sets is because we will use the validation set to tune our models, and we will treat the test set as some unseen data in which we will see how our models perform. We will assign 70% of our data to the training set, and 15% to both the validation and test sets.\n",
    "Split the dataset and print the distribution of each set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train:350268 valid:74971 test:74898\n"
     ]
    }
   ],
   "source": [
    "train, valid, test = loan_level.split_frame([0.70, 0.15], seed=42)\n",
    "print(\"train:%d valid:%d test:%d\" % (train.nrows, valid.nrows, test.nrows))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set Variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = \"ORIGINAL_INTEREST_RATE\"\n",
    "\n",
    "ignore = [\"ORIGINAL_INTEREST_RATE\", \n",
    "          \"FIRST_PAYMENT_DATE\", \n",
    "          \"MATURITY_DATE\", \n",
    "          \"MORTGAGE_INSURANCE_PERCENTAGE\", \n",
    "          \"PREPAYMENT_PENALTY_MORTGAGE_FLAG\", \n",
    "          \"LOAN_SEQUENCE_NUMBER\", \n",
    "          \"PREPAID\", \n",
    "          \"DELINQUENT\", \n",
    "          \"PRODUCT_TYPE\"] \n",
    "\n",
    "x = list(set(train.names) - set(ignore))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['SELLER_NAME',\n",
       " 'ORIGINAL_COMBINED_LOAN_TO_VALUE',\n",
       " 'OCCUPANCY_STATUS',\n",
       " 'PROPERTY_STATE',\n",
       " 'FIRST_TIME_HOMEBUYER_FLAG',\n",
       " 'NUMBER_OF_BORROWERS',\n",
       " 'ORIGINAL_UPB',\n",
       " 'PROPERTY_TYPE',\n",
       " 'SERVICER_NAME',\n",
       " 'ORIGINAL_DEBT_TO_INCOME_RATIO',\n",
       " 'LOAN_PURPOSE',\n",
       " 'CREDIT_SCORE',\n",
       " 'ORIGINAL_LOAN_TO_VALUE',\n",
       " 'POSTAL_CODE',\n",
       " 'NUMBER_OF_UNITS',\n",
       " 'CHANNEL',\n",
       " 'ORIGINAL_LOAN_TERM',\n",
       " 'METROPOLITAN_STATISTICAL_AREA']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Build an XGBoost Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb = H2OXGBoostEstimator(seed=42, model_id='XGBoost', nfolds=0, keep_cross_validation_predictions = False)\n",
    "\n",
    "%time \n",
    "xgb.train(x=x, y=y, training_frame=train, validation_frame=valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: You do not need to specify the cross-validation parameters, but we want to show the parameters that you would have to change in order to enable H2O internal cross-validation. If you wanted to use the H2O cross-validation, you would have to set keep_cross_validation_predictions to True and change nfolds to 3, 5, or 10, depending on the number of folds that you want; by doing cross-validation, you no longer need a validation frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When printing the model summary, you will see a complete summary with some of the parameters in your model (in this case, we only see the number of trees). You will see some scoring metrics on your training data, as well as on your validation data, as shown in the image above. From the report, we can see that the MAE, in fact, is lower than the RMSE, this might be due to having some of the interest rates at the far end of our range (such as interest rates as high as 10 and 11, when our mean is 7.2). The training and validation RMSE are 0.4212 and 0.4244, respectively. This means that according to the RMSE, on average, the predictions are 0.42 percent off from the actual values. While the training and validation MAE are 0.3071 and 0.3108 respectively, which tells us that the predictions are about 0.31 percent off from the actual values, according to the MAE. Since we have a target interest rate range between 4 to 12 percent, we will try to reduce the RMSE when tuning our models as it will penalize larger errors or outliers.\n",
    "\n",
    "In the model summary, you will also see two tables; the first table is the scoring history, and the second one is the variable importance table (not shown here). Besides looking at the table, we can also take a look at both graphing representations of the scoring history and the variable importance.\n",
    "\n",
    "Let's first plot the scoring histor, and variable importance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb.varimp_plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will make some predictions on our validation set. To compare the predictions to the actual value, we are going to bind the two data frames, the predictions and the validation set, as follow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb_def_pred = xgb.predict(valid)\n",
    "xgb_def_pred.cbind(valid['ORIGINAL_INTEREST_RATE'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the first ten predictions, we can see that they are somewhat close to the actual value, except for the first sample, which is more than 1.5% off. We can see that samples such as this one, might make the RMSE higher than the MAE.\n",
    "\n",
    "We can also check the model performance with a test, or in this case, with a validation set, and print some of the scores for our model. For now, we will just save the model performance, as we will use later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "default_xgb_per = xgb.model_performance(valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Build a Deep Learning Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deep learning estimator is fairly easy to use. For our default Deep Learning model, we do not need to define any parameters, but we will define the seed, model id, and we will also make sure that cross-validation is disabled, as we are using a validation frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 0 ns\n",
      "deeplearning Model Build progress: |██████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "dl = H2ODeepLearningEstimator(seed=42, model_id='DL',\n",
    "                              nfolds= 0,\n",
    "                              keep_cross_validation_predictions = False\n",
    "                              )\n",
    "%time \n",
    "dl.train(x=x, y=y, training_frame=train, validation_frame=valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Details\n",
      "=============\n",
      "H2ODeepLearningEstimator :  Deep Learning\n",
      "Model Key:  DL\n",
      "\n",
      "\n",
      "Status of Neuron Layers: predicting ORIGINAL_INTEREST_RATE, regression, gaussian distribution, Quadratic loss, 73,601 weights/biases, 880.4 KB, 3,598,847 training samples, mini-batch size 1\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>layer</th>\n",
       "      <th>units</th>\n",
       "      <th>type</th>\n",
       "      <th>dropout</th>\n",
       "      <th>l1</th>\n",
       "      <th>l2</th>\n",
       "      <th>mean_rate</th>\n",
       "      <th>rate_rms</th>\n",
       "      <th>momentum</th>\n",
       "      <th>mean_weight</th>\n",
       "      <th>weight_rms</th>\n",
       "      <th>mean_bias</th>\n",
       "      <th>bias_rms</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td>1</td>\n",
       "      <td>165</td>\n",
       "      <td>Input</td>\n",
       "      <td>0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td>2</td>\n",
       "      <td>200</td>\n",
       "      <td>Rectifier</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0556208</td>\n",
       "      <td>0.213166</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.0133035</td>\n",
       "      <td>0.18641</td>\n",
       "      <td>-0.016015</td>\n",
       "      <td>0.240832</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "      <td>200</td>\n",
       "      <td>Rectifier</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0565888</td>\n",
       "      <td>0.0597781</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.055436</td>\n",
       "      <td>0.140429</td>\n",
       "      <td>0.0572095</td>\n",
       "      <td>0.523543</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td></td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Linear</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00114034</td>\n",
       "      <td>0.000922778</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.00921151</td>\n",
       "      <td>0.0805816</td>\n",
       "      <td>0.53918</td>\n",
       "      <td>1.09713e-154</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     layer  units       type dropout l1 l2   mean_rate     rate_rms momentum  \\\n",
       "0        1    165      Input       0                                           \n",
       "1        2    200  Rectifier       0  0  0   0.0556208     0.213166        0   \n",
       "2        3    200  Rectifier       0  0  0   0.0565888    0.0597781        0   \n",
       "3        4      1     Linear          0  0  0.00114034  0.000922778        0   \n",
       "\n",
       "  mean_weight weight_rms  mean_bias      bias_rms  \n",
       "0                                                  \n",
       "1  -0.0133035    0.18641  -0.016015      0.240832  \n",
       "2   -0.055436   0.140429  0.0572095      0.523543  \n",
       "3 -0.00921151  0.0805816    0.53918  1.09713e-154  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "ModelMetricsRegression: deeplearning\n",
      "** Reported on train data. **\n",
      "\n",
      "MSE: 0.19353740345708922\n",
      "RMSE: 0.43992886181414514\n",
      "MAE: 0.31480976947191924\n",
      "RMSLE: 0.05217880725894603\n",
      "Mean Residual Deviance: 0.19353740345708922\n",
      "\n",
      "ModelMetricsRegression: deeplearning\n",
      "** Reported on validation data. **\n",
      "\n",
      "MSE: 0.19590097755469604\n",
      "RMSE: 0.4426070238424782\n",
      "MAE: 0.31819382877238667\n",
      "RMSLE: 0.052454661814392047\n",
      "Mean Residual Deviance: 0.19590097755469604\n",
      "\n",
      "Scoring History: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>duration</th>\n",
       "      <th>training_speed</th>\n",
       "      <th>epochs</th>\n",
       "      <th>iterations</th>\n",
       "      <th>samples</th>\n",
       "      <th>training_rmse</th>\n",
       "      <th>training_deviance</th>\n",
       "      <th>training_mae</th>\n",
       "      <th>training_r2</th>\n",
       "      <th>validation_rmse</th>\n",
       "      <th>validation_deviance</th>\n",
       "      <th>validation_mae</th>\n",
       "      <th>validation_r2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 14:53:53</td>\n",
       "      <td>0.000 sec</td>\n",
       "      <td>None</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 14:53:57</td>\n",
       "      <td>5.830 sec</td>\n",
       "      <td>21051 obs/sec</td>\n",
       "      <td>0.284882</td>\n",
       "      <td>1</td>\n",
       "      <td>99785.0</td>\n",
       "      <td>0.485477</td>\n",
       "      <td>0.235688</td>\n",
       "      <td>0.359848</td>\n",
       "      <td>0.282620</td>\n",
       "      <td>0.487921</td>\n",
       "      <td>0.238067</td>\n",
       "      <td>0.361016</td>\n",
       "      <td>0.286977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 14:54:09</td>\n",
       "      <td>17.895 sec</td>\n",
       "      <td>31854 obs/sec</td>\n",
       "      <td>1.425885</td>\n",
       "      <td>5</td>\n",
       "      <td>499442.0</td>\n",
       "      <td>0.461583</td>\n",
       "      <td>0.213058</td>\n",
       "      <td>0.329545</td>\n",
       "      <td>0.351498</td>\n",
       "      <td>0.465912</td>\n",
       "      <td>0.217074</td>\n",
       "      <td>0.332636</td>\n",
       "      <td>0.349852</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 14:54:26</td>\n",
       "      <td>35.597 sec</td>\n",
       "      <td>28908 obs/sec</td>\n",
       "      <td>2.566495</td>\n",
       "      <td>9</td>\n",
       "      <td>898961.0</td>\n",
       "      <td>0.456045</td>\n",
       "      <td>0.207977</td>\n",
       "      <td>0.316504</td>\n",
       "      <td>0.366965</td>\n",
       "      <td>0.463335</td>\n",
       "      <td>0.214679</td>\n",
       "      <td>0.320577</td>\n",
       "      <td>0.357025</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 14:54:49</td>\n",
       "      <td>57.807 sec</td>\n",
       "      <td>28839 obs/sec</td>\n",
       "      <td>4.278681</td>\n",
       "      <td>15</td>\n",
       "      <td>1498685.0</td>\n",
       "      <td>0.619442</td>\n",
       "      <td>0.383708</td>\n",
       "      <td>0.489340</td>\n",
       "      <td>-0.167921</td>\n",
       "      <td>0.620816</td>\n",
       "      <td>0.385412</td>\n",
       "      <td>0.490522</td>\n",
       "      <td>-0.154329</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 14:55:03</td>\n",
       "      <td>1 min 11.748 sec</td>\n",
       "      <td>30894 obs/sec</td>\n",
       "      <td>5.705702</td>\n",
       "      <td>20</td>\n",
       "      <td>1998525.0</td>\n",
       "      <td>0.443726</td>\n",
       "      <td>0.196893</td>\n",
       "      <td>0.314879</td>\n",
       "      <td>0.400704</td>\n",
       "      <td>0.449752</td>\n",
       "      <td>0.202277</td>\n",
       "      <td>0.320370</td>\n",
       "      <td>0.394171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 14:55:16</td>\n",
       "      <td>1 min 24.364 sec</td>\n",
       "      <td>32807 obs/sec</td>\n",
       "      <td>7.132162</td>\n",
       "      <td>25</td>\n",
       "      <td>2498168.0</td>\n",
       "      <td>0.450594</td>\n",
       "      <td>0.203035</td>\n",
       "      <td>0.318042</td>\n",
       "      <td>0.382008</td>\n",
       "      <td>0.457518</td>\n",
       "      <td>0.209322</td>\n",
       "      <td>0.322146</td>\n",
       "      <td>0.373069</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 14:55:28</td>\n",
       "      <td>1 min 36.902 sec</td>\n",
       "      <td>34270 obs/sec</td>\n",
       "      <td>8.562215</td>\n",
       "      <td>30</td>\n",
       "      <td>2999070.0</td>\n",
       "      <td>0.439929</td>\n",
       "      <td>0.193537</td>\n",
       "      <td>0.314810</td>\n",
       "      <td>0.410916</td>\n",
       "      <td>0.442607</td>\n",
       "      <td>0.195901</td>\n",
       "      <td>0.318194</td>\n",
       "      <td>0.413267</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 14:55:41</td>\n",
       "      <td>1 min 49.604 sec</td>\n",
       "      <td>35340 obs/sec</td>\n",
       "      <td>9.989340</td>\n",
       "      <td>35</td>\n",
       "      <td>3498946.0</td>\n",
       "      <td>0.457844</td>\n",
       "      <td>0.209621</td>\n",
       "      <td>0.328640</td>\n",
       "      <td>0.361962</td>\n",
       "      <td>0.463597</td>\n",
       "      <td>0.214922</td>\n",
       "      <td>0.332610</td>\n",
       "      <td>0.356297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 14:55:45</td>\n",
       "      <td>1 min 53.112 sec</td>\n",
       "      <td>35525 obs/sec</td>\n",
       "      <td>10.274553</td>\n",
       "      <td>36</td>\n",
       "      <td>3598847.0</td>\n",
       "      <td>0.447597</td>\n",
       "      <td>0.200343</td>\n",
       "      <td>0.308114</td>\n",
       "      <td>0.390201</td>\n",
       "      <td>0.453395</td>\n",
       "      <td>0.205567</td>\n",
       "      <td>0.312505</td>\n",
       "      <td>0.384317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 14:55:46</td>\n",
       "      <td>1 min 54.328 sec</td>\n",
       "      <td>35523 obs/sec</td>\n",
       "      <td>10.274553</td>\n",
       "      <td>36</td>\n",
       "      <td>3598847.0</td>\n",
       "      <td>0.439929</td>\n",
       "      <td>0.193537</td>\n",
       "      <td>0.314810</td>\n",
       "      <td>0.410916</td>\n",
       "      <td>0.442607</td>\n",
       "      <td>0.195901</td>\n",
       "      <td>0.318194</td>\n",
       "      <td>0.413267</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                timestamp           duration training_speed     epochs  \\\n",
       "0     2020-08-12 14:53:53          0.000 sec           None   0.000000   \n",
       "1     2020-08-12 14:53:57          5.830 sec  21051 obs/sec   0.284882   \n",
       "2     2020-08-12 14:54:09         17.895 sec  31854 obs/sec   1.425885   \n",
       "3     2020-08-12 14:54:26         35.597 sec  28908 obs/sec   2.566495   \n",
       "4     2020-08-12 14:54:49         57.807 sec  28839 obs/sec   4.278681   \n",
       "5     2020-08-12 14:55:03   1 min 11.748 sec  30894 obs/sec   5.705702   \n",
       "6     2020-08-12 14:55:16   1 min 24.364 sec  32807 obs/sec   7.132162   \n",
       "7     2020-08-12 14:55:28   1 min 36.902 sec  34270 obs/sec   8.562215   \n",
       "8     2020-08-12 14:55:41   1 min 49.604 sec  35340 obs/sec   9.989340   \n",
       "9     2020-08-12 14:55:45   1 min 53.112 sec  35525 obs/sec  10.274553   \n",
       "10    2020-08-12 14:55:46   1 min 54.328 sec  35523 obs/sec  10.274553   \n",
       "\n",
       "    iterations    samples  training_rmse  training_deviance  training_mae  \\\n",
       "0            0        0.0            NaN                NaN           NaN   \n",
       "1            1    99785.0       0.485477           0.235688      0.359848   \n",
       "2            5   499442.0       0.461583           0.213058      0.329545   \n",
       "3            9   898961.0       0.456045           0.207977      0.316504   \n",
       "4           15  1498685.0       0.619442           0.383708      0.489340   \n",
       "5           20  1998525.0       0.443726           0.196893      0.314879   \n",
       "6           25  2498168.0       0.450594           0.203035      0.318042   \n",
       "7           30  2999070.0       0.439929           0.193537      0.314810   \n",
       "8           35  3498946.0       0.457844           0.209621      0.328640   \n",
       "9           36  3598847.0       0.447597           0.200343      0.308114   \n",
       "10          36  3598847.0       0.439929           0.193537      0.314810   \n",
       "\n",
       "    training_r2  validation_rmse  validation_deviance  validation_mae  \\\n",
       "0           NaN              NaN                  NaN             NaN   \n",
       "1      0.282620         0.487921             0.238067        0.361016   \n",
       "2      0.351498         0.465912             0.217074        0.332636   \n",
       "3      0.366965         0.463335             0.214679        0.320577   \n",
       "4     -0.167921         0.620816             0.385412        0.490522   \n",
       "5      0.400704         0.449752             0.202277        0.320370   \n",
       "6      0.382008         0.457518             0.209322        0.322146   \n",
       "7      0.410916         0.442607             0.195901        0.318194   \n",
       "8      0.361962         0.463597             0.214922        0.332610   \n",
       "9      0.390201         0.453395             0.205567        0.312505   \n",
       "10     0.410916         0.442607             0.195901        0.318194   \n",
       "\n",
       "    validation_r2  \n",
       "0             NaN  \n",
       "1        0.286977  \n",
       "2        0.349852  \n",
       "3        0.357025  \n",
       "4       -0.154329  \n",
       "5        0.394171  \n",
       "6        0.373069  \n",
       "7        0.413267  \n",
       "8        0.356297  \n",
       "9        0.384317  \n",
       "10       0.413267  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Variable Importances: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>variable</th>\n",
       "      <th>relative_importance</th>\n",
       "      <th>scaled_importance</th>\n",
       "      <th>percentage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>SELLER_NAME.Other sellers</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.011794</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>SELLER_NAME.NORWESTMORTGAGE,INC</td>\n",
       "      <td>0.990678</td>\n",
       "      <td>0.990678</td>\n",
       "      <td>0.011684</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>SELLER_NAME.FIRST UNION CAPITAL</td>\n",
       "      <td>0.950101</td>\n",
       "      <td>0.950101</td>\n",
       "      <td>0.011206</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>SELLER_NAME.WELLSFARGOHOMEMORTGA</td>\n",
       "      <td>0.882134</td>\n",
       "      <td>0.882134</td>\n",
       "      <td>0.010404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>SELLER_NAME.CHASEMANHATTANMTGECO</td>\n",
       "      <td>0.882006</td>\n",
       "      <td>0.882006</td>\n",
       "      <td>0.010402</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>SELLER_NAME.CROSSLANDMTGECORP</td>\n",
       "      <td>0.846646</td>\n",
       "      <td>0.846646</td>\n",
       "      <td>0.009985</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>SELLER_NAME.STANDARD FEDERAL BAN</td>\n",
       "      <td>0.831295</td>\n",
       "      <td>0.831295</td>\n",
       "      <td>0.009804</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>SELLER_NAME.BANKOFAMERICA,NA</td>\n",
       "      <td>0.824306</td>\n",
       "      <td>0.824306</td>\n",
       "      <td>0.009722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>SELLER_NAME.BISHOPSGATERESIDENTI</td>\n",
       "      <td>0.823274</td>\n",
       "      <td>0.823274</td>\n",
       "      <td>0.009710</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>SELLER_NAME.FIRSTARBANK,NA</td>\n",
       "      <td>0.813628</td>\n",
       "      <td>0.813628</td>\n",
       "      <td>0.009596</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>SERVICER_NAME.Other servicers</td>\n",
       "      <td>0.807938</td>\n",
       "      <td>0.807938</td>\n",
       "      <td>0.009529</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>SELLER_NAME.NATLCITYMTGECO</td>\n",
       "      <td>0.794619</td>\n",
       "      <td>0.794619</td>\n",
       "      <td>0.009372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>SELLER_NAME.PNCMTGECORPOFAMERICA</td>\n",
       "      <td>0.789251</td>\n",
       "      <td>0.789251</td>\n",
       "      <td>0.009308</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>SELLER_NAME.NORWEST MORTGAGE, IN</td>\n",
       "      <td>0.785579</td>\n",
       "      <td>0.785579</td>\n",
       "      <td>0.009265</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>SELLER_NAME.ABNAMROMTGEGROUP,INC</td>\n",
       "      <td>0.780304</td>\n",
       "      <td>0.780304</td>\n",
       "      <td>0.009203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>SERVICER_NAME.BANKOFAMERICA,NA</td>\n",
       "      <td>0.771477</td>\n",
       "      <td>0.771477</td>\n",
       "      <td>0.009099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>SELLER_NAME.PRINCIPALRESIDENTIAL</td>\n",
       "      <td>0.771051</td>\n",
       "      <td>0.771051</td>\n",
       "      <td>0.009094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>SERVICER_NAME.CHASEMTGECO</td>\n",
       "      <td>0.750546</td>\n",
       "      <td>0.750546</td>\n",
       "      <td>0.008852</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>SELLER_NAME.OLDKENTMTGECO</td>\n",
       "      <td>0.748508</td>\n",
       "      <td>0.748508</td>\n",
       "      <td>0.008828</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>SERVICER_NAME.WASHINGTONMUTUALBANK</td>\n",
       "      <td>0.747740</td>\n",
       "      <td>0.747740</td>\n",
       "      <td>0.008819</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              variable  relative_importance  \\\n",
       "0            SELLER_NAME.Other sellers             1.000000   \n",
       "1      SELLER_NAME.NORWESTMORTGAGE,INC             0.990678   \n",
       "2      SELLER_NAME.FIRST UNION CAPITAL             0.950101   \n",
       "3     SELLER_NAME.WELLSFARGOHOMEMORTGA             0.882134   \n",
       "4     SELLER_NAME.CHASEMANHATTANMTGECO             0.882006   \n",
       "5        SELLER_NAME.CROSSLANDMTGECORP             0.846646   \n",
       "6     SELLER_NAME.STANDARD FEDERAL BAN             0.831295   \n",
       "7         SELLER_NAME.BANKOFAMERICA,NA             0.824306   \n",
       "8     SELLER_NAME.BISHOPSGATERESIDENTI             0.823274   \n",
       "9           SELLER_NAME.FIRSTARBANK,NA             0.813628   \n",
       "10       SERVICER_NAME.Other servicers             0.807938   \n",
       "11          SELLER_NAME.NATLCITYMTGECO             0.794619   \n",
       "12    SELLER_NAME.PNCMTGECORPOFAMERICA             0.789251   \n",
       "13    SELLER_NAME.NORWEST MORTGAGE, IN             0.785579   \n",
       "14    SELLER_NAME.ABNAMROMTGEGROUP,INC             0.780304   \n",
       "15      SERVICER_NAME.BANKOFAMERICA,NA             0.771477   \n",
       "16    SELLER_NAME.PRINCIPALRESIDENTIAL             0.771051   \n",
       "17           SERVICER_NAME.CHASEMTGECO             0.750546   \n",
       "18           SELLER_NAME.OLDKENTMTGECO             0.748508   \n",
       "19  SERVICER_NAME.WASHINGTONMUTUALBANK             0.747740   \n",
       "\n",
       "    scaled_importance  percentage  \n",
       "0            1.000000    0.011794  \n",
       "1            0.990678    0.011684  \n",
       "2            0.950101    0.011206  \n",
       "3            0.882134    0.010404  \n",
       "4            0.882006    0.010402  \n",
       "5            0.846646    0.009985  \n",
       "6            0.831295    0.009804  \n",
       "7            0.824306    0.009722  \n",
       "8            0.823274    0.009710  \n",
       "9            0.813628    0.009596  \n",
       "10           0.807938    0.009529  \n",
       "11           0.794619    0.009372  \n",
       "12           0.789251    0.009308  \n",
       "13           0.785579    0.009265  \n",
       "14           0.780304    0.009203  \n",
       "15           0.771477    0.009099  \n",
       "16           0.771051    0.009094  \n",
       "17           0.750546    0.008852  \n",
       "18           0.748508    0.008828  \n",
       "19           0.747740    0.008819  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "See the whole table with table.as_data_frame()\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our deep learning model, we see that our validation scores are higher than the training scores, but not by much. The training and validation RMSE are 0.4395 and 0.4480, respectively. And the training and validation MAE are 0.3240 and 0.3298, respectively. You can also see the scoring history and variable importance table. You can plot both the scoring history and variable importance plot."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: The default number of epochs for a Deep Learning model is 10. We will try to find a number of epochs that will give us a better score when we tune our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeXxU9bn48c+TfZ+ELEAygYQshCULEEABFbRatRZr1ar1ti73ttqqbe1tq72/Vu2+aG+9rVuta1stWnctGhRFENl3wqIQQUImkAWykH3m+/vjTHAIISQwk5lJnvfrNa/MOfM95zwnhDz5rkeMMSillFL9FeLvAJRSSgUXTRxKKaUGRBOHUkqpAdHEoZRSakA0cSillBoQTRxKKaUGRBOHUqdARB4RkZ/66dpjRKRZREL9cX2lNHGoIUVE5ojIhyLSICL1IrJcRKZ7+zrGmJuNMb/w9nlFJEtEjIiE9dj/lIj80n3tT40xccYY50nOdb2IfODtGJUKO3kRpYKDiCQAbwDfAp4HIoCzgHYvXyf0ZL+0hwIRCTPGdPk7DhV4tMahhpJ8AGPMP40xTmNMqzFmkTFmc3cBEfmGiGwXkSYR2SYiU937J4jIEhE5LCLlIjLf45inRORhEVkoIkeAeZ41ABGZKyKVIvLfInJQRBwicoPH8cki8rqINIrIGhH55enUBHrWStw1iwr3PX0iIteKyATgEeBMd7PWYXdZm4j8TURqRGSviPxEREI8zrNcRP4oIvXAL9y1tkKPa6eJSKuIpJ5q/Cr4aeJQQ8lHgFNEnhaRi0QkyfNDEbkSuAf4OpAAzAfqRCQceB1YBKQBtwHPiMh4j8O/CvwKiAd6+6U/CrABGcB/Ag96XP9B4Ii7zHXul1eISCzwJ+AiY0w8MAvYaIzZDtwMrHA3ayW6D/mzO85xwDlY34sbPE45E6jA+j78HFgA/IfH59cA7xhjarx1Dyr4aOJQQ4YxphGYAxjgr0CNiLwmIiPdRf4L+L0xZo2x7DLG7AXOAOKA3xpjOowx72I1eV3jcfpXjTHLjTEuY0xbL5fvBH5ujOk0xiwEmoHx7g7sy4G7jTEtxphtwNP9uJ1ad+3nsLu28NU+yrqAySISbYxxGGPKeyvkjuUq4MfGmCZjzB7gD8DXPIpVGWP+bIzpMsa0umP9anetxF327/2IXw1hmjjUkGKM2W6Mud4YYwcmA+nA/e6PM4HdvRyWDuwzxrg89u3Fqj1023eSS9f16A9owUpGqVh9iZ7Hn+xcACnGmMTuF/Bsb4WMMUewksHNgENE/i0iBSc6J1a/z16PfX3epzFmFVZt6Rz3eXOB1/oRvxrCNHGoIcsYswN4CiuBgPVLMaeXolVApsdf1QBjgP2epzvFMGqALsDusS/zFM/VK2NMmTHmfGA0sAOrtgXHx1yLVTMa67GvP/f5NFZz1deAF05Q41LDiCYONWSISIG7g9ru3s7Eam5a6S7yGPADEZkmllwRGQt0/1X9IxEJF5G5wBex2vdPi3v01UvAPSIS4/6r/eune95uIjJSROa7+zrasZrIukd8HQDsIhLhEcvzwK9EJN59798H/nGSy/wduAwrefzNW7Gr4KWJQw0lTVidu6vco59WAluB/wYwxvwLq4P7WXfZV4ARxpgOrI7yi7D+Kn8I+Lq7xuINt2J1SFdj/RL+J94bIhyCdX9VQD1Wh/e33Z+9C5QD1SJS6953G1aSrMDq5H8WeKKvCxhjKoH1WLWRZV6KWwUx0Qc5KTW4ROR3wChjjNdGV/maiDyB1XH+E3/HovxPJwAq5WPu5qkIYAswHWu47n/5NagBEJEs4MvAFP9GogKFNlUp5XvxWP0cR7D6GP4AvOrXiPpJRH6B1dx3rzHmE3/HowKDNlUppZQaEK1xKKWUGpBh0ceRkpJisrKy/B2GCkgGDm2E8ESIy/Z3MEoFlHXr1tUaY45bl2xYJI6srCzWrl3r7zBUIDq4FN45h5bIBGIu158RpTyJyN7e9mtTlRrW2j5dCEBM+x5or/NvMEoFCU0calhr2/cWzc5oAEztaj9Ho1Rw0MShhq+2WhJaN/OPuotwGaFx/4f+jkipoDAs+jiU6lX1O4RgWOM8l3Pb12I7sAKbv2NSJ9XZ2UllZSVtbbrWordERUVht9sJDw/vV3lNHGrYOrL333R0xTNz2vlsKX+FC5vXgzEg4u/QVB8qKyuJj48nKysL0X+r02aMoa6ujsrKSrKz+zeyUJuq1PBkDFL9NsubS5g7YTQHI4qINfVwZI+/I1Mn0dbWRnJysiYNLxERkpOTB1SD08ShhqeGcmKcB1jfOYO8tDhIngGAq3aVnwNT/aFJw7sG+v3UxKGGJVfVW9bXkecjIqRkTqfdFU5D5XI/R6ZU4NPEoYallr1v8lHbGCbnTQKgKDOVra05OGu0xqH6VldXR0lJCSUlJYwaNYqMjIyj2x0dHX0eu3btWr7zne+c9BqzZs3yVrg+oZ3javjpaiXq0HKWNV3IxbnJAOSmxfFM+3iKWsvA1QUh+l9D9S45OZmNGzcCcM899xAXF8cPfvCDo593dXURFtb7z09paSmlpaUnvcaHHwb20HCtcajhp2YZYbTzcegsRtusyX+hIcKhqCmE0wYN5X4OUAWb66+/nu9///vMmzePO+64g9WrVzNr1iymTJnCrFmz2LlzJwBLlizhkksuAaykc+ONNzJ37lzGjRvHn/70p6Pni4uLO1p+7ty5XHHFFRQUFHDttdfSvaL5woULKSgoYM6cOXznO985et7BoH9WqWGna/+bOF3hxNnnHbM/NG0GNENXzUrCkor9FJ0aiJ+9Xs62qkavnnNiegJ3f3HSgI/76KOPeOeddwgNDaWxsZGlS5cSFhbGO++8w//8z//w4osvHnfMjh07eO+992hqamL8+PF861vfOm4uxYYNGygvLyc9PZ3Zs2ezfPlySktLuemmm1i6dCnZ2dlcc801p3y/p0JrHGrY6dhXxuojk5iRl3nM/szMQg53xdGkM8jVKbjyyisJDQ0FoKGhgSuvvJLJkydz++23U17eey32C1/4ApGRkaSkpJCWlsaBAweOKzNjxgzsdjshISGUlJSwZ88eduzYwbhx447OuxjsxKE1DjW8tOwnpnU7y5pv5Nac5GM+Ks5MYtPqfIrq1vgpODVQp1Iz8JXY2Nij73/6058yb948Xn75Zfbs2cPcuXN7PSYyMvLo+9DQULq6uvpVxt8P4NMahxpeqt8G4GDs2SREHdskMDY5hu0dBdg6dkJnsz+iU0NEQ0MDGRkZADz11FNeP39BQQEVFRXs2bMHgOeee87r1+iLJg41rHRWvsnBziQys2ce95mI0Bw7jRBccGi9H6JTQ8WPfvQjfvzjHzN79mycTqfXzx8dHc1DDz3EhRdeyJw5cxg5ciQ22+CttDYsnjleWlpq9EFOCpeTjn+l8nrNFDIufp4zxiUfV+SBhcu59fAcOot+R/jkH/khSHUy27dvZ8KECf4Ow++am5uJi4vDGMMtt9xCXl4et99++ymfr7fvq4isM8YcN35Yaxxq+Di0gQjnIVa0ljJlTGKvRXLHjGNfx0iaq7SDXAW2v/71r5SUlDBp0iQaGhq46aabBu3a2jmuhg9HGQBtyfOIDAvttUiRPZH1S/OZe0hrqCqw3X777adVwzgdWuNQw0Z75VtsacmhODf/hGVG26L42DmBOOd+aD1+aKRSyseJQ0QuFJGdIrJLRO48QZm5IrJRRMpF5P2THSsiI0TkbRH52P01yZf3oIaIzkbC61eyrHkKs3NTTlhMRGhLmGZt1OuwXKV647PEISKhwIPARcBE4BoRmdijTCLwEDDfGDMJuLIfx94JLDbG5AGL3dtK9e3AEkLoYmPnTApGxfdZND59Bk4TQseBlYMUnFLBxZc1jhnALmNMhTGmA1gAXNqjzFeBl4wxnwIYYw7249hLgafd758GvuTDe1BDhHG8RYsripiMswkJ6fvZA5My0/mobQwtDu0gV6o3vkwcGcA+j+1K9z5P+UCSiCwRkXUi8vV+HDvSGOMAcH9N6+3iIvJNEVkrImtrampO81ZUsOusLOPD5kLOzBt10rKFdhsbWsYT3bjOepSsUh7mzp1LWVnZMfvuv/9+vv3tb5+wfPd0gIsvvpjDhw8fV+aee+7hvvvu6/O6r7zyCtu2bTu6fdddd/HOO+8MNHyv8GXi6O3Pup7/C8OAacAXgM8DPxWR/H4e2ydjzKPGmFJjTGlqaupADlVDTXMFEa0VLG2a2mf/RreUuEg+ZRKRphGadg1CgCqYXHPNNSxYsOCYfQsWLOjXelELFy4kMbH3oeAn0zNx/PznP+dzn/vcKZ3rdPkycVQCnqvI2YGqXsq8ZYw5YoypBZYCxSc59oCIjAZwfz2IUn1xLAKgImwW9qSYfh3SlTTdelO32ldRqSB1xRVX8MYbb9De3g7Anj17qKqq4tlnn6W0tJRJkyZx991393psVlYWtbW1APzqV79i/PjxfO5znzu67DpY8zOmT59OcXExl19+OS0tLXz44Ye89tpr/PCHP6SkpITdu3dz/fXX88ILLwCwePFipkyZQmFhITfeeOPR2LKysrj77ruZOnUqhYWF7NixwyvfA1/O41gD5IlINrAfuBqrT8PTq8ADIhIGRAAzgT8CO/o49jXgOuC37q+v+vAe1BDgqnqLqo6RjB1b0u9jUjKm0LI/kpADK4jKvtaH0anTsu57cGijd8+ZVALT7j/hx8nJycyYMYO33nqLSy+9lAULFnDVVVfx4x//mBEjRuB0OjnvvPPYvHkzRUVFvYe9bh0LFixgw4YNdHV1MXXqVKZNs0bzffnLX+Yb3/gGAD/5yU94/PHHue2225g/fz6XXHIJV1xxxTHnamtr4/rrr2fx4sXk5+fz9a9/nYcffpjvfe97AKSkpLB+/Xoeeugh7rvvPh577LHT/hb5rMZhjOkCbgXKgO3A88aYchG5WURudpfZDrwFbAZWA48ZY7ae6Fj3qX8LnC8iHwPnu7eV6p2rE1O9mKVNU5iT1/8my8LMZLa05OrIKtUrz+aq7maq559/nqlTpzJlyhTKy8uPaVbqadmyZVx22WXExMSQkJDA/Pnzj362detWzjrrLAoLC3nmmWdOuCR7t507d5KdnU1+vjU/6brrrmPp0qVHP//yl78MwLRp044uini6fDpz3BizEFjYY98jPbbvBe7tz7Hu/XXAed6NVA1ZtasIdTaztHkqv805fm2qE5mcYeO51nymHfk3ODsgNMKHQapT1kfNwJe+9KUv8f3vf5/169fT2tpKUlIS9913H2vWrCEpKYnrr7+etra2Ps8h0vvovuuvv55XXnmF4uJinnrqKZYsWdLneU623mD3suwnWrb9VOjMcTW0OcpwmlAaEuaQGNP/X/626HAcoYWE0QENW3wYoApGcXFxzJ07lxtvvJFrrrmGxsZGYmNjsdlsHDhwgDfffLPP488++2xefvllWltbaWpq4vXXXz/6WVNTE6NHj6azs5Nnnnnm6P74+HiampqOO1dBQQF79uxh1y5rIMff//53zjnnHC/dae80caghzVlVxsaWfEpysgd+cPIM62vtKu8GpYaEa665hk2bNnH11VdTXFzMlClTmDRpEjfeeCOzZ8/u89ipU6dy1VVXUVJSwuWXX85ZZ5119LNf/OIXzJw5k/PPP5+CgoKj+6+++mruvfdepkyZwu7du4/uj4qK4sknn+TKK6+ksLCQkJAQbr75Zu/fsAddVl0NXe11mBdTub/6GmbM/79+DcX19PiyCuZXTCMu+xKiz/67j4JUA6XLqvuGLquuFED1YgTDitZSpo0d+JJmxZmJbGrNw1mjNQ6lPGniUEOXo4wmVxxRI2cQFd77Mup9mZRuY3NrPrHtu6CjwQcBKhWcNHGoockYnFVlLG0s5szcky8z0pvoiFDqIksQDNSv83KA6nQMhyb2wTTQ76cmDjU0NW4ntG0/S5unMmeAfRuewlKtDnJTp81VgSIqKoq6ujpNHl5ijKGuro6oqKh+H6NPAFRDk3uZkU1dM/h1esIpnyYvcyyfbB/NyOqVxEzyVnDqdNjtdiorK9HFS70nKioKu93e7/KaONSQZBxl7O0cw7ixEwg9yTLqfSmy29i0Lp/P1+lDnQJFeHg42dmnMLxaeY02Vamhx9mGOfA+7zUUD3gIbk8FoxLY2jae6C4HtOz3UoBKBTdNHGroqfmAEFcr7zedXv8GQERYCA0xU60NXSlXKUAThxqKHGV0mnAqw6czZkT/llHvS+yoaXSaUIzOIFcK0MShhiDjWMS61kmU5mSecCG5gZiYOZLtrdm0VutKuUqBJg411LQ6kMObWdJQctr9G92K7YlsasknvGEdGJdXzqlUMNPEoYYWx9sALG2ayqwBLKPel5zUWLZ1FBDuaobGnSc/QKkhThOHGlocZRx2jUCSikiOi/TKKcNCQ2hL0A5ypbpp4lBDh3Fhqt/m/cZiZuelefXUyaOLaHZG49QOcqU0cagh5NBGpL2G9xqneq1/o1th5gg2t+bRUb3Cq+dVKhhp4lBDh3uZkVUtU5meNfBl1PtSZE9kY0s+kc1bwdn3I0GVGuo0caihw1HG7q48xtqziYnw7mo6WckxfNQ1gRC64NBGr55bqWCjiUMNDZ3NmJrlvF1fdNqzxXsjIjiTplsb2kGuhjmfJg4RuVBEdorILhG5s5fP54pIg4hsdL/ucu8f77Fvo4g0isj33J/dIyL7PT672Jf3oILEwSWI6eT9Ju/3b3SzZ+RR3ZmsTwRUw57PVscVkVDgQeB8oBJYIyKvGWO29Si6zBhziecOY8xOoMTjPPuBlz2K/NEYc5+vYldByFFGB1HsdBVSmGHzySWK7TY2fZLHOQdXMfDnCSo1dPiyxjED2GWMqTDGdAALgEtP4TznAbuNMXu9Gp0aWhyLWNdaTGn2aMJCffNjXeSeQR7Vthva631yDaWCgS8TRwawz2O70r2vpzNFZJOIvCkivT0q52rgnz323Soim0XkCRHpdfiMiHxTRNaKyFp94MsQ17wHmj5iUX0Rc/J800wFMNoWxSfG/SNav9Zn11Eq0PkycfS2ulzPZz2uB8YaY4qBPwOvHHMCkQhgPvAvj90PAzlYTVkO4A+9XdwY86gxptQYU5qamnpqd6CCQ7U1DHdps+/6N8DqIA9JmY7LCOhEQDWM+TJxVAKZHtt2oMqzgDGm0RjT7H6/EAgXEc//+RcB640xBzyOOWCMcRpjXMBfsZrE1HDmKKPejKIlKpdxKbE+vVSePYPd7Xa6tINcDWO+TBxrgDwRyXbXHK4GXvMsICKjxL3utYjMcMdT51HkGno0U4nIaI/Ny4CtPohdBQtXF6Z6MUsaS5idm+qVZdT7UmS3saklH1O3GkzPCrRSw4PPEocxpgu4FSgDtgPPG2PKReRmEbnZXewKYKuIbAL+BFxtjPW/UURisEZkvdTj1L8XkS0ishmYB9zuq3tQQaBuNdLZwNuHS3wyf6OnInsiG1vzCe+sgZZPfX49pQKRz4bjwtHmp4U99j3i8f4B4IETHNsCHLcutjHma14OUwUzxyJchLC8uYSf5XpnGfW+pMRF4ggttDbqVkPsWJ9fU6lAozPHVXBzlLHbNZHRKaNJi48alEtGp02hw4TrDHI1bGniUMGr4xCmbjVldUU+HU3V00R7CuUt4+g6qB3kanjSxKGCV/ViBBfvNkxhTp7vm6m6Fbv7OeTQOnB1Ddp1lQoUmjhU8HKU0SbxlLeNZ0b24CWOyRnWyKpQVws09FxBR6mhTxOHCk7GgGMR69unUjQmmbhIn47zOIYtOpzayBJrQ/s51DCkiUMFp8ad0PIpb9RMHtT+jW7JoybS6IzTxKGGJU0cKjh1LzPSOHVQ5m/0VJSZxMaWPDp1BrkahjRxqODkKKNGxnJI0inOTBz0yxfZbWxsySe0sRy6jgz69ZXyJ00cKvg42+HAEpY1TeWMccmE+2gZ9b5MSk9gc2s+ITihfsOgX18pf9LEoYJPzXJwtvBGTaFf+jcAYiLCOBI3xdrQfg41zGjiUMGnehFOwll5pNCnz984mcz0cTg606wFD5UaRjRxqODjKKOCYmJjE8lLi/NbGIX2RNYfycNZs9JvMSjlD5o4VHBpPQCHNlJWV8ic3BSfL6Pel2K7jY0t4wlr3QttB/0Wh1KDTROHCi7VbwPwZn2x3/o3uo0fFU95+3hro26NX2NRajBp4lDBxVFGa8gItrWOY/YgLKPel8iwULpsJbhMiHaQq2FFE4cKHsYF1W+zsXM641LjGW2L9ndE5NtHs6tjjHaQq2FFE4cKHoc3Q9sBXj0wyS+zxXtTZE9k/ZF8XLWr9FGyatjQxKGCh8NaZmTx4RK/9290634GeWjnIWiu8Hc4Sg0KTRwqeDjKqAkdT51zBGfk+Ld/o1tuahw7OgqsDW2uUsOEJg4VHLqOQM0HLD8yleLMRBKiwv0dEQBhoSFEpBTSbiKhThc8VMODJg4VHA68D64OXnQETv9Gt0kZKWxpzcXUao1DDQ8+TRwicqGI7BSRXSJyZy+fzxWRBhHZ6H7d5fHZHhHZ4t6/1mP/CBF5W0Q+dn9N8uU9qABRvQinRLO6eWLA9G90K860seFIHqZ+Pbg6/R2OUj7ns8QhIqHAg8BFwETgGhGZ2EvRZcaYEvfr5z0+m+feX+qx705gsTEmD1js3lZDnaOMipBphIRFM2XM4C+j3pcieyKbWvIJMe1weIu/w1HK53xZ45gB7DLGVBhjOoAFwKVeOO+lwNPu908DX/LCOVUgO/IpNO7gnUPFzMgeQWRYqL8jOkZWcgy7nO6/ibSDXA0DvkwcGcA+j+1K976ezhSRTSLypohM8thvgEUisk5Evumxf6QxxgHg/prW28VF5JsislZE1tbU1JzenSj/cg/DfdExye+zxXsjIiSPyqfBZdPEoYYFXyaO3laf6zlDaj0w1hhTDPwZeMXjs9nGmKlYTV23iMjZA7m4MeZRY0ypMaY0NTV1IIeqQOMooyVsFLvaMwOuf6NbkT2J9UfycGkHuRoGfJk4KoFMj207UOVZwBjTaIxpdr9fCISLSIp7u8r99SDwMlbTF8ABERkN4P6qy5IOZS4nVL/DFudMRsRGMmFUgr8j6lWx3cbGI/lI4zbobPJ3OEr5lC8TxxogT0SyRSQCuBp4zbOAiIwS97rYIjLDHU+diMSKSLx7fyxwAbDVfdhrwHXu99cBr/rwHpS/1a+BzsO8emAys3KSCQnx3zLqfSm0J7KxZTyCgfq1Jz9AqSDms8RhjOkCbgXKgO3A88aYchG5WURudhe7AtgqIpuAPwFXG2MMMBL4wL1/NfBvY8xb7mN+C5wvIh8D57u31VDlWIRBWFgTePM3PKXboqgMcXfRaT+HGuLCfHlyd/PTwh77HvF4/wDwQC/HVQDFJzhnHXCedyNVActRRm1EEYedCQHbvwFWB/nY9DHs70onQxOHGuJ05rgKXB2HoW4VK1unMTY5hswRMf6OqE+FGTbWNmsHuRr6NHGowHXgXTBOnt8feLPFe1Ocaa2UG9JaCS1VJz9AqSCliUMFLscinKFxrDicG9D9G90KMxLZ2JJvbdTro2TV0KWJQwUmY8BRxiehZ+CUMM4cF3gT/3pKjY+kPmIiTkK1g1wNaZo4VGBq2gVH9vBeQwmT020kxUb4O6J+KcgYye6ObE0cakjTxKECk6MMgGc/LQiK/o1uRZk2VjflWUusG5e/w1HKJzRxqMDkKKMlMotP2kcFRf9Gt2L3SrnS1QiNH/k7HKV8QhOHCjzODjj4HuWuM4gIC6E0K3geuTI5w/ZZB7k2V6khqt+JQ0TmiMgN7vepIpLtu7DUsFb7IXQd4d81hUzPSiIqPLCWUe+LLTocV/x42kyMJg41ZPUrcYjI3cAdwI/du8KBf/gqKDXMORZhJIwXKvOCqn+j22T7CMrb8jRxqCGrvzWOy4D5wBE4unJtvK+CUsOco4y66Gk0u2KCqn+jW2GGjTVNuZhDG8HZ7u9wlPK6/iaODvfigwaOrlirlPe11cCh9axpK8UWHc6kdJu/Ixqw4kxrIqCYTji0yd/hKOV1/U0cz4vIX4BEEfkG8A7wV9+FpYat6rcB+Nf+iczKSSY0QJdR78uk9AS2tGkHuRq6+pU4jDH3AS8ALwLjgbuMMX/2ZWBqmHIswhk+giU19qDs3wCIiQgjLimbQyYF6lb5OxylvK6/neOxwLvGmB9i1TSiRSTcp5Gp4ccYqF7EpxGzcREalP0b3YrsiWxsycNojUMNQf1tqloKRIpIBlYz1Q3AU74KSg1TDVuh1cHSphIyEqMZmxzYy6j3pSgzkTWNeUjTR9BxyN/hKOVV/U0cYoxpAb4M/NkYcxkw0XdhqWHJvczI3/aMZ05uCu6nCgelYruNTa3d/Rz6KFk1tPQ7cYjImcC1wL/d+3z69EA1DDnKaI0pYHdzIrPzgreZCmD8qHi2t+dZG9pcpYaY/iaO72FN/nvZ/dzwccB7vgtLDTtdLXBwGTtlFgCzcgJ/GfW+RIaFYh+Zzn7XGE0casjpV63BGPM+8L7HdgXwHV8FpYahg0vB1c6btYVMGJ1ASlykvyM6bUV2G+sceaTXrUKMgSBuelPKU39HVZWKyEsisl5ENne/fB2cGkYcizAhUTz7SRZzcoO7ttGtKCORtc25SNsBaKn0dzhKeU1/m6qewRpFdTnwRY9Xn0TkQhHZKSK7ROTOXj6fKyINIrLR/brLvT9TRN4Tke0iUi4i3/U45h4R2e9xzMX9vAcVyKrLOBR3Bk1d4UE7f6OnokwbG1vGWxs6n0MNIf3t4K4xxrw2kBOLSCjwIHA+UAmsEZHXjDHbehRdZoy5pMe+LuC/jTHrRSQeWCcib3sc+0f3pEQ1FLRUQsM2NkR9gfBQYUb2CH9H5BW5qXHscebQRThhdathzBX+Dkkpr+hv4rhbRB4DFgNHV20zxrzUxzEzgF3u/hBEZAFwKdAzcRzHGOMAHO73TSKyHcjoz7EqCDkWAfCiYxJTxyQREzE0BuyFhYaQPzqFiq488rWDXA0h/W2qugEoAS7ks8hRj+kAACAASURBVGaqnrWEnjKAfR7ble59PZ0pIptE5E0RmdTzQxHJAqYAnnX9W939LE+ISPA85Uf1zrEIV1Q6C/clB/Vs8d4U2RNZ3ZiDqV8LLqe/w1HKK/qbOIqNMaXGmOuMMTe4Xzee5JjehpCYHtvrgbHGmGLgz8Arx5xAJA5rfazvGWMa3bsfBnKwEpkD+EOvFxf5poisFZG1NTU1JwlV+Y3LCdVvsz9qDiBBP3+jp+JMG+ua85CuI9C43d/hKOUV/U0cK0VkoDPFK4FMj207UOVZwBjTaIxpdr9fCISLSAqAey2sF4FnPJvEjDEHjDFOY4wLa92sGb1d3BjzqDvZlaampg4wdDVoDq2HjnqWH5lKfGQYRRnBt4x6XwozPGeQa3OVGhpOmjjEWvfhPGCje4TUZhHZ0o/huGuAPBHJFpEI4GrgmA52ERnlPj8iMsMdT5173+PAdmPM//Y4ZrTH5mXA1pPdgwpgjjJA+MfefM7ISSYstN9PMw4KWcmx1MoYWonXxKGGjJP2QhpjjIgkAnkDObExpktEbgXKgFDgCfes85vdnz8CXAF8S0S6gFbgavf15gBfA7aIyEb3Kf/HXSv5vYiUYDV77QFuGkhcKsA4ymhPKGHr5giunDW0mqkAQkKEQnsSOzrGM0UThxoi+jt85Z9AmjFmzUBO7v5Fv7DHvkc83j8APNDLcR/Qex8JxpivDSQGFcA6G6F2BR/bbgYYMvM3eiqyJ7JiZw4lkS8gXS0QFryr/ioF/e/jmAesEJHdA2iqUqpv1e+CcfL2oWJGJUSRkzo0n0hclGFj/ZF8xDjh0AZ/h6PUaetvjeMin0ahhqfqRZiwOJ7Zkc45BcG9jHpfijITuavFY6Xc1Nn+DUip09TfRQ73+joQNQw5ymiynUVtC8zJGxrrU/Um3RaFiRrFIUaRpP0caggYWkNYVPBo2g3NFWzqskZTz84Zmv0bACJCYYaNrW35OrJKDQmaOJR/uJ/298qByeSPjCMtIcrPAflWkT2R5YfGQXMFtNX6OxylTosmDuUf1YtwxWbzRkXMkB1N5ak408bGFvdEwPoBDU5UKuBo4lCDz9UJ1e9yMPYc2rvMkFufqjeFGYlsac3FINpcpYKeJg41+GpXQlcTK1qmEhoizBw3dDvGu6XGR2KLH4GDHKjVZ3Oo4KaJQw0+RxlIKM/vy2NKZiJxkUNjGfWTKbInsuFIHtSvBtNzvU+lgocmDjX4HGV0Jc1kZaVzWPRvdCu026wO8vY6OPKJv8NR6pRp4lCDq60W6tfxSfhsjIE5Q2wZ9b4U2xPZ1N1BXqv9HCp4aeJQg6v6HcCw+HAxsRGhlGQm+juiQVNot7GzbSxdRGoHuQpqmjjU4KpeBBEj+Ncno5g5LpnwIbaMel9s0eFkptjYY8Zb/RxKBanh879W+Z8x4CijZcQ8dte2Dav+jW6FGTbWNuVC/XprWLJSQUgThxo8DdugtYpy10yAYTF/o6ciu40PD+eAsxUayv0djlKnRBOHGjzuZUZePziZlLhI8kfG+TmgwVecmciGlvHWhs7nUEFKE4caPNWLMAkT+PfuMObkJg/ZZdT7Mik9gf2dI2mVRO0gV0FLE4caHF2tcPB96hPmUXekY1j2bwDERISRl5bAR10TNHGooKWJQw2Omg/A2caatmnA0H1MbH8U2W2sbMjBNJRDZ5O/w1FqwDRxqMHhKIOQCF7an8u41FjSE6P9HZHfFNltrGjIQTDW6CqlgowmDjU4HGW4Us5iWUXLsBxN5anInsjm7hnk2lylgpBPE4eIXCgiO0Vkl4jc2cvnc0WkQUQ2ul93nexYERkhIm+LyMfur0m+vAflBS1V0LCVyqg5tHYOr/WpelMwOp4mEjkUYtfEoYKSzxKHiIQCDwIXAROBa0RkYi9FlxljStyvn/fj2DuBxcaYPGCxe1sFsupFACxpnEKIwBnDYBn1vkSGhTJhdAI7Ogo0caig5MsaxwxglzGmwhjTASwALvXCsZcCT7vfPw18yYsxK19wLIKoUbyyJ5kieyK26HB/R+R3hRk2ltWPg5ZPobXa3+EoNSC+TBwZwD6P7Ur3vp7OFJFNIvKmiEzqx7EjjTEOAPfXNO+GrbzKuKD6bTrSPsemysZh37/RrdieyKrGXGtDax0qyPgycfQ2u6vn02vWA2ONMcXAn4FXBnBs3xcX+aaIrBWRtTU1NQM5VHnToQ3QXstOzsTpMsO+f6NbUaaN8tZxuAjVxKGCji8TRyWQ6bFtB6o8CxhjGo0xze73C4FwEUk5ybEHRGQ0gPvrwd4ubox51BhTaowpTU1N9cb9qFPhXmbkrdrJRIWHMHXs8FlGvS+5qXEQFsOBkDxNHCro+DJxrAHyRCRbRCKAq4HXPAuIyChxrzshIjPc8dSd5NjXgOvc768DXvXhPajT5VgESVMoqzDMyE4mMizU3xEFhLDQECan29jSOh7q1lhNekoFCZ8lDmNMF3ArUAZsB543xpSLyM0icrO72BXAVhHZBPwJuNpYej3WfcxvgfNF5GPgfPe2CkSdTVCznOYR57HrYDNzcof3aKqeiuyJvF+XBZ2HoWmXv8NRqt/CfHlyd/PTwh77HvF4/wDwQH+Pde+vA87zbqTKJw4sAdPF+o7pwPBeZqQ3RXYbD6/LszbqVkNCvn8DUqqfdOa48h1HGYTF8vr+LEbERjBhVIK/IwooRXYbH7dl0ikx2s+hgoomDuU7jjJM2lze393IrJxkQkKG3zLqfclKjiU2KpJ9MhHq9NkcKnho4lC+0VwBzbuoiTuHg03tOn+jFyEhQmGGjQ1H8uDQRnC2+zskpfpFE4fyDYe1zMiy5qmA9m+cSJE9kSW1WeDqgMOb/R2OUv2iiUP5hmMRxI7lzb3xjE2OIXNEjL8jCkjFdhvrmj06yJUKApo4lPe5OuHAYpwjL2BlxSGtbfShKDORqs5UWkNTNHGooKGJQ3lf3WrobGRP+Cya27u0f6MP6bYokmMjqXBN0sShgoYmDuV9jjKQEBbVFSICZw7zZdT7IiIU2W2sacqBxh3Q0eDvkJQ6KU0cfdn3Mqy5RUe7DJRjESTP5L2KDian20iKjfB3RAGtyJ7IuzXZ1kb9Wv8Go1Q/aOLoy6FN8PFD8PZsaP7E39EEh/Z6qF9DR+r5rP9U+zf6o8huY2NLdwe5zudQgU8TR1+K7sGc9ZK1jtCbU6HytZMfM9wdWAzGxRbnDLpcRvs3+qHInkijM47DYdnaz6GCgiaOPry8oZLvfpBF++fWQFw2LL0UNvzIGjWkeucog/BE3nRkEhEWQmmWPhL+ZFLjI0m3RfFR1wSoXQVmQI+eUWrQaeLoQ21TB69vruLKZw5QPfNdyL0Ztt8Li8+Flv3+Di/wGGMljlHn8cHuw0zPSiIqXJdR749Cu40Vh8dBWzW06s+WCmyaOPrwjbPH8ejXStl9sJn5D69l0+jfwKxnrKfavTkFqt/xd4iBpXEHtFTSmHQuO6qbtH9jAIrsibzX3UGuzVUqwGniOInzJ47kxW/PIiIshK/8ZQWvHj4HPr8GolLh3Qtgy8/A5fR3mIHB/bS/lS3TALR/YwCK7Ylsb8vGJeGaOFTA08TRDwWjEnj1ltkUZyby3QUbuXel4Dp/FWRdC1vugSUXQ5s+1xzHIkgYz9t7o7BFhzMp3ebviIJGYYaNdhNBTViBJg4V8DRx9FNyXCT/+M+ZXDMjkwff281NC3bQPPUJmPEoHHzfarqqWe7vMP3H2QYHl2BGXcDyXbXMykkmVJdR7zdbTDhZyTFsay+AurVai1UBTRPHAESEhfDrywq554sTeXfHQa54ZAX7RlwLF6yA0Ch45xzY/ofhOSqmZjk4W6mOOZuqhjbt3zgFRfZEltVnQ1eT1V+kVIDSxDFAIsL1s7N56obpVB1u5dIHl7Pq8Bi4cB1kzIcNP4Bll0HHYX+HOrgcZRASzpJDEwHt3zgVRXYb79dpB7kKfJo4TtFZeam8cstsEqPD+Y/HV7FgYwOc9SJM/V/Y/29rwmD9On+HOXgciyB1DksqWshIjGZssi6jPlBF9kQq2jPoConXxKECmiaO0zAuNY6Xb5nNmTkp3PnSFn72xja68r4Ln1sKphMWzYKPHxn6TVet1XB4E65RF/Dh7jrm5KYgov0bAzU5IwGREPaHTtbEoQKaTxOHiFwoIjtFZJeI3NlHueki4hSRK9zb40Vko8erUUS+5/7sHhHZ7/HZxb68h5OxRYfzxHWl3Dg7myeX7+GGp9bQEFcKF26AkfNgzbfgw/+AzmZ/hulb1W8D8HHILJraupidp81UpyImIoy8tHg2t+ZbTwPsavV3SEr1ymeJQ0RCgQeBi4CJwDUiMvEE5X4HlHXvM8bsNMaUGGNKgGlAC/Cyx2F/7P7cGLPQV/fQX2GhIdz1xYn8/vIiVlbUcdmDy6loioK5C6HoF/DpAiibAQ3b/B2qbzjKICqNdxyjAJiVo8uon6pCu40lNVlguqznkCsVgHxZ45gB7DLGVBhjOoAFwKW9lLsNeBE4eILznAfsNsbs9U2Y3vOV6Zk8+40zaGjt5EsPLmfpx3Uw+Scw723oqIO3psMn//B3mN5lXFb/xqjz+WBXPRNGJ5ASF+nvqIJWsd3GskPjrA1trlIBypeJIwPY57Fd6d53lIhkAJcBj/RxnquBf/bYd6uIbBaRJ0QkoFbRm541gldumU16YjTXP7maJz74BDNyntV0lVwKK74Gq2+y5j0MBYc2QXsNHanns27vIebkam3jdBTZEznYlUxr+GhNHCpg+TJx9NY72rOX+H7gDmNMr7OdRCQCmA/8y2P3w0AOUAI4gD+c4NhvishaEVlbUzO4s7ozR8Tw4rdmcd6Ekfz8jW38+KUtdESMgnMXw8Q7YdejsOhMaNo9qHH5hHuZkXUd0+hwunT+xmkqGB1PeKjwKZP12RwqYPkycVQCmR7bdqCqR5lSYIGI7AGuAB4SkS95fH4RsN4Yc6B7hzHmgDHGaYxxAX/FahI7jjHmUWNMqTGmNDU19fTvZoBiI8P4y39M49Z5uSxYs4//eGwVdS1OKPkNnPM6HNkLb02FfS8NemxeVb0IEotZsjeU8FBhRvYIf0cU1CLDQikYlcD6I3nQvBva6/wdklLH8WXiWAPkiUi2u+ZwNXDMk5CMMdnGmCxjTBbwAvBtY8wrHkWuoUczlYiM9ti8DNjqi+C9ISRE+MHnx/N/V5ewqfIw8x9Yzo7qRsi4BC5cD/HjYdnlsO52cHb4O9yB62yGmg9g9Of5YFctU8YkERMR5u+ogl6R3cbb1WOtjbo1/g1GqV74LHEYY7qAW7FGS20HnjfGlIvIzSJy88mOF5EY4Hyg55/kvxeRLSKyGZgH3O7l0L3u0pIMnr/pTLpcLi5/6EMWlVdDXBacvwzyb4Od98PiuXBk38lOFVgOvg+uThqT5lFe1aizxb2k2J7I6sZsDKL9HCog+XQehzFmoTEm3xiTY4z5lXvfI8aY4zrDjTHXG2Ne8NhuMcYkG2MaepT7mjGm0BhTZIyZb4xx+PIevKU4M5HXbp1DblocN/1jHQ++twsTEgGlf4LZz8HhLfDWFKh6y9+h9p+jDEKj+eDweADt3/CSQruNZlcMTRF5mjhUQNKZ44NoZEIUz910JvOL07m3bCffe24jbZ1OGPsVa62r6HRrifZNPw2O1VGrF0HaXJZVNBEXGUaxXZdR94a8tDiiwkPY7ZpoJY6hvvKACjqaOAZZVHgo919Vwg8/P55XN1Zx1V9WcKCxDRLy4YKVMO56KP8lvHcBtB446fn85sheaNx5tH/jjHHJhIXqj5M3hIWGMDndxpqmXGivsb7XSgUQ/Z/uByLCLfNyefRr0/j4YDPzH/iATfsOQ1gMnPEEzHwCaj+0mq4OLvV3uL1zLAKgKvos9tW36vwNLyu023ireoy1oc1VKsBo4vCjCyaN4qVvzyI81P1Y2o37rQ9yboALVkFYPCyeB+W/tWZoBxJHGcRkssRhJYw5uj6VVxXbE9ncNAZXSKTO51ABRxOHnx19LK3deiztfWU7cbkMJBXBhWsg8wrY9GN4/1Kraaizyf9t3q4uqH4HRl/A8t11jEyIJCc1zr8xDTFFdhtdhFEfMUlrHCrg6KD7AJAcF8k//msmd726lQfe28VHB5r441UlxEYmwOwFkHoWbPg+vPGGdUBoNESN7PFKO35f9EgITwRvL3FetwY6G3CNvIDlZbWcW5Cmy6h7WVZyLPGRYezsnEBK/UtWsg7R/64qMOhPYoCICAvhN18uZPyoeH7xxjYuf/hD/vr1UjJHxMD4W2HU56y/PNsOHPs6stfa314Dva3cEhIOkWnHJpOe293vI5IhJPTkwTrKQELYyQwOt5Tr/A0fCAkRCu02VhzOYXZ8KzSUQ1Kxv8NSCtDEEVBEhBtmZ5OTGsctz67n0geX88h/TLOW8bAVWK8TMS5reYqjSeXg8Umm7SA0bLXeu3qZqS4hEJlybK0lMu3Y5BI1EqoWwojpvL/XSlQ6f8M3iuyJvLk6kx/EY/1xoIlDdTMGDrwLCeMhxj7ol9fEEYDOzrceS/uNp9dy7WMr+eWXJnPV9DF9HyQhEJVqvZjcd1ljoLOh7yTTegCaK6z3XUeOP8fku1i+ppa8tDhGJkSd8r2qEyuy23jk/VF0hSUSVrcacr/h75BUIDi0GdZ/Dw68BwkTrDlgYdGDGoImjgCVkxrHy9+eza3/XM8dL25hR3UT/+/iCd6ZKyECEYnWK2H8yct3HfksmbQfhI5DtI38IqufX801M06S0NQpK7LbAOFAeBEZ2kGu2utg812w6xGr73L87bDzj7DxR1D650ENRRNHALPFhPPk9dP59cIdPLH8ExZucVCYYWNiuo1J6QlMSk8gIzHa9x3TYbEQN856ua3fVUt7l0v7N3woIzGa5NgIytsLyGh7zErgYbH+DksNNlcXfPwIbLkLOhsh7xYovAci3StR7/wjjL4IMgbvKdqaOAJc92Npp41NYtG2asqrGnl3x0Fc7hG5iTHhTByd4E4kVkIZlxpHaIhvk8kHu2oJDRFmjtNl1H1FRCiy21hWN44LklxQvw7SzvZ3WGowVb8D675nDY4YeR5Mux8SPZqiS34NB96BVTfAxVus0ZWDQBNHkPhC0Wi+UGStKN/a4WR7dSPlVY1sq2qgvKqRp1fspaPLmiQYFR5Cwahjk8n4UfFEhfdjxFQ/Ld9VS0lmIvFR4V47pzpeoT2Rf75v5xdJWB3kmjiGh+YKWP/fUPkKxGbDWS+D/dLjh9aHRsGsZ+GtUlh5o/Wsn0EYGq+JIwhFR4QydUwSU8d89tTcTqeL3TXNlO+3Ekp5VQOvbarimVWfAhAaIuSmxjEpPYGJ7oQyMT0BW/TAf/E3tHSyeX8Dt52b57V7Ur0rttv4U1ci7ZFjiNR+jqGvsxnKfw07/mANpS/+NRTcbiWIE0mcDFN+D+u+a/V/5H3L52Fq4hgiwkOtWkbBqAQun2btM8awr76VcnetpLyqgQ921fLShv1Hj8scEc3ko30m1te0k4ySWlFRizFo/8YgKHSvOLw/ZDLjgjFxdLVYv/REF6nok3HBnmdg4x3Q6oCsr1lPC43J6LV4e5fzaJ9nblq89VyfqoVWLSVtLtgm+DRcTRxDmIgwJjmGMckxXFT42YMTDza1uZu5Go8mlTe3Vh/9PCUu8mjne3cyGTMihhB3v8kHu2qJiQilJDNx0O9puEmLj2K0LYpNreMZx0JrZFv0SH+H1TtXp/VcmbpVULvK+tq4A+JyIe8mGHcDROpimMepWwNrvwN1K2HEdJjzIqSe2WvRtk4nz6/dx8NLduNoaMOeFM2b3z3LajI+40lYWAQfXmuttB0a4bOQNXEMQ2nxUaSNj2Le+M860hrbOtle1d3MZSWU5btq6XL3wsdFhjFxtNXMtXj7QWZmjyAiTP+KHAxFdhvv1ozlslSgfo316GF/MwZaPv0sQdStgvr14Gy1Po9MheSZ1lprB9+HDT+ETT+BsVdZTSnJMwelLT6gtVZb69BVPGVNrD3jScj+eq+1s7ZOJwtWf8oj71dQ3dhG6dgkbjp7HD9/Yxt3v1bO/36lBKJHw8zHYemlsPmnMOV3PgtdE4cCICEqnJnjkpk57rO/CNu7nHxU3XxMU9dza/bR2unk2/Ny/Rjt8FJkT+SBbXZMWihSt9o/iaOz0frL2LM20eZ+XkxIJIyYCrk3WQkhZSbEZh2bGA5vhY8fhk/+Dp/8DZKmWAkk66vDb4ixsx12/h9s/SW42mDCj2Dy/4PwhOOKtnU6eXbVpzzy/m4ONrUzI2sEf/hKMbNykhER6ls6+dPijzm3II1LitLBPh9yvwnb74X0C2HkPJ/cghh/r7Q6CEpLS83atWv9HcaQ4HQZqhvbGJ0QdbTpSvnWso9r+Nrjq9k68w7iEu0wz8ePF3Z1WUvTeNYmGrYD7t8V8fmfJYjkmZBY1P9mkc4mqy3/44fh8Gbrl2X2160kYpvos1sKCMbA/jdg/feheRekXwJT/xcSjh9k0trh5JlVe3nk/Qpqm9s5Y9wIvntePmfmHNvU1+l0ceUjK6ioaabs9rMZbYu25vu8OdX6evHmz+Z7nAIRWWeMKT1uvyYOpQJbQ0snxT9fxL9n/p1JzkVweZ33mnmMgZbKzxJE7Sprvoizxfo8MtlKDkdf00/rF9Ex161dAR8/BJ/+y1o7Le0cyPs22L/k0/Z5v2jYDutvtxYITSiAqX+0agQ9tHR08Y+Ve3l0aQW1zR3Myknmu+flHdMS0NOe2iNc/KdllGQm8o//nGn9QVe/DsrOgMzLYPZzp/zzcqLEoU1VSgU4W0w4WckxrDuSy6SQ56BpV69/pfZLZxPUrz22NtHqsD4LibCakHL+67PaRNw43/RFiEDqLOs19Y9Q8aQ1O3r5VVZ7f85/WU0usUG+pE3HYdjyM/joAatJbuofIf8Wa6ithyPtXfxtxV7+uqyC+iMdnJWXwnfOy2N61smTdFZKLHd/cSJ3vLiFxz6o4Jtn58CIaVD0M9j0/6Dg+5ByhldvSxOHUkGg0J7I2/vH8vV0rImA/UkcLqc149izNtG47bOnScblwshzP6tNJBVDaKRP76NXUakw8Ucw4QfWX+QfPwzbfmO90i+xmrFGXxBcQ3pdTqh43PrF3V5nLVBZ9Ev3IqSfaW7v4ukP9/DYsgoOtXRydn4q3z0vj2ljk05w4t59pTSTxdsPcm/ZTubkpjIxPcH6twUreXmZTxOHiFwI/B8QCjxmjPntCcpNB1YCVxljXnDv2wM0AU6gq7u6JCIjgOeALGAP8BVjzCFf3odS/lZst/GbTSMxmTFWB3n2tccXatl/bOd1/drPVjaOSHKPcrrcXZuYEXhDYyUE0i+yXkf2wq5HYfdjsP81q+aTexOMuxGiAnz+0MFlsO47cGij9RC2af8HI6YcU6SxrZOnl+/h8eWfcLilk7njrYQxZczAEkY3EeG3lxdx4f1L+e6CDbx+2xx8uWa1zxKHiIQCDwLnA5XAGhF5zRizrZdyvwPKejnNPGNMbY99dwKLjTG/FZE73dt3eP0GlAogRfZEnIRyOKqIpLrVVkKoW3tsbaLVPbEzJBwSS6x5E921ifjcQRn+6nQZKmqa2eawhnXvqG7CnhTNuePTmJ2bQnREP5e9iR0Lxb+CyXdD5ctWX8jGO6zVYcdcadVCUs4MrCG9Rz6FDT+CT5+DmEzr6Z1jvnJMjA2tnTy1fA+Pf1BBY1sX5xWk8Z3z8ij2wpyoEbER3HtlMdc9sZrfvrmDe2ad9ilPyJc1jhnALmNMBYCILAAuBbb1KHcb8CIwvZ/nvRSY637/NLAETRxqiJuUnkCIwG7XRErrnoR/JXg0OY2z1rDqHumUVNL3EhVe0trhZEd149EkUV7VyM7qRto6rbgiQkPISYtj3Z56nl31KZFhIZyZk8y5BWnMG59mPd3yZEIjrLkfY6+Cw+XWkhoVT8Oef0BisXtI77UQ7sdn3ne1WMNft/0OMFaym/gjCPvs/hpaOnl8+Sc8ufwTmtq6+NyEkXz3vLyjKwN4yzn5qVw/K4unPtzD/AxhqlfP/hlfJo4MYJ/HdiUw07OAiGQAlwHncnziMMAiETHAX4wxj7r3jzTGOACMMQ4R6XU5SBH5JvBNgDFjgryDTQ17sZFh5KbF8WrDuZTmHAbbJHdtYsZx7ea+cOhIh7XagKPh6KoDu2uaj67SHB8VxqT0BL46Y6y14kBGAjmpcYSHhtDR5WL1J/W8u+Mg7+08yF2vlgPl5I+MY15BGucVjGTqmMSTP2smcZL13Ini38DeZ62+kDU3W5MLu4f0Jk7y+ffiKGOsEWEbfmhNhhzzFWvNqNixR4scbung8Q8+4anle2hq7+Lzk0Zy27l5TM7wbsLwdOdFBXy4u5Y/v7eJJ9N9cw1fJo7e6pA9x/7eD9xhjHH28kyJ2caYKndieFtEdhhjlvb34u5E8yhYw3EHELdSAanInsjCHR38/LoXfPYMFmMMlYdaj9YitlU1sK2qkaqGtqNlRtuimJSewEWTRx19Now96cTPhYkIC2FOXgpz8lK464sTqahpPppEnvjgE/7yfgW26HDOzk/l3IJUzslPY0RsH8Nxw+OsEVc534DalVYC2f0YfPyg1aeQ923I/LJvh/Qe2mgtKnhwqVXzOfNvMPKcox/XH+ngsWUVPP3hHo50OLlo8ihuOzfP6rT2sajwUO6/agp3P7EesP5Nvf3T4svEUQlkemzbgaoeZUqBBe4fuBTgYhHpMsa8YoypAjDGHBSRl7GavpYCB0RktLu2MRo46MN7UCpgFNttvLCukqqGNjIST/9Rod0rKm/zWGZmW1UjjW1dAIQIjEuN4/+3d+fBVZZXHMe/vwAF2aEkkgCyBQnIqlJRRBG0apVCW60bDtOxndHiguO0tY6tjmNnGVKceQAACnFJREFU6NRudhRUXLAi1oIU6zguBUSZWkBR9lBWIRBNUEkATSDJ6R/PmzRgQG7IvW/IPZ+ZTG5e3tycZxLuuc/zvO85Z/fqXFO3bGBO+2O/qB+HPplt6ZPZlh+P7sO+skMs3bQnSiTF/HPVbjIEw0/rxNi8LMbmZZHXtV3dSUkKNZ0yzw030m19Oixl/fu60Jeiz00hwbTtdULxHqasOJTz2PJEuOBgxIxw6XBG2Lv5dH85T7yzjWff3c6Xhyr5zuBsbh/bj/5d2zVcDMdhYE57rjunBxTD5uL99Ku7VmK9JTNxrAD6SeoN7AKuBa6vfYKZ9a5+LOkZ4BUz+4ekNkCGme2LHn8beCA69WVgMjAt+rwgiWNwrtEY3D1soK7euTfhxHGgvCLsR0RJYn1h2Liu7uHSsnkGedntuWJITk2By7yu7Y9/M7ue2rVqweWDs7l8cDZVVcaaXSU1s5Hfvb6R372+kewOraIlrSzO63uUDfZWXWDgz2DAXVD4JmyeDht+C+unQc4V0SW9l9a8wCes6hD891FYcz9U7AvVaAffF5IHsGd/OY+/vZW/vvsRZRWVXDkkh9vG5nL6qalNGLUNzG4PxWEvqqElLXGYWYWkWwlXSzUDnjKzdZJujv59xjG+/VRgfvQuoznwvJlV11mYBrwo6SZgB3B1ssbgXGMyILsdLZqJVQUlh1U7PtKe/eWHzyIKS9m25wDVRSI6nNKCM3LaM/ncnjWziD5d2jRMP/sTkJEhhvboyNAeHbnzktMpKi3jrY3FLMovYsEHuw7bYB+Xl8VFeVl073TEBrsyIOfS8HFgB2x+IswOlrwS6mf1uzm6pDeBfaHCN0IXvtIN0PWS0IUvKo9StK+Mx5ds5blloZHa+KEhYeRmxZcwUsFLjjh3Ehn/l6W0a9Wc538yEjNjx2dfHDaLWLe7hE9Ky2vO79bxlKhx1/+XmnI6tEp+n/oGVl5RyYptn7Mov4hF+Z+w/dNQEuW4NtgrD4ZOepumQ9Fb4Q75HleFWUjmqKNf0rtvc+hvsetlaNs3LId1Gw8SRaVlzFiyldnLPuJQZRUTh3Vjythc+mbGeHXXEfLXvkHe6ktZnTuHId+6tl7P4bWqPHG4JuCe+WuYv3IXg7t3YMPuUvaVh/2I6g6P1UmiugR+x9ZNrOZTpHqDfVF+Ecu3fUZFldVssI/Ly+LC0zPpVNdeTMn6UNpk26xQ8bfj4OiS3knQIpolHNoH634D+X8MSWbQvdB/KjRrycclZcxYsoXnl++gssqYOKwbt47NpXeXxlfhN5mJw0uOOHcSuXhAFq+uKeRQZRUThueEWUR2w/eUb+zq2mBfmF/EWxuLjr3B3mEgnP1w6K63fU64sXDFT8ONe70mhTasax+Eso+h9+Rw3inZ7N77JTOWrOWF5TupNOMHZ3ZjykW59Pxm40sYqeAzDudck1G9wb4wv4jF+UWs2VUCQE60wT72yA12s1D7a9N0+OgFqCoP98ac9TB0OYdde7/k0cWb+ft7BVSZcdVZ3ZlyUe7x3bwYM1+qOkGeOJxLT0WlZSzeGJa0lm7aw4GDlUffYC//FEo3QpeR7Py8jEff2sLc98M9zFef3YNbLux7UiSMaqu2bOGxF2cyafwNnDdoUL2ew5eqnHNpJ6t9K64ZcRrXjDiN8orKmjvYF+UX8asF62DB4XewZ7Ybyox5a5m3soAMiWtG9OCWMbkNct9MqlW26MyrJefzwxZ1Ftc4IZ44nHNpoWXzZozul8nofpn8+sqBbN1zgMVREnnynXAHO4QaW9efcxq3jOkbOuq5r0iLpSpJxcBHdfxTF+DI6rtNnY85PfiY00Oyx9zTzL5y00taJI6jkfReXet3TZmPOT34mNNDXGM+iVpqOeecaww8cTjnnEtIuieOx7/+lCbHx5wefMzpIZYxp/Ueh3POucSl+4zDOedcgjxxOOecS0jaJg5Jl0naKGmzpLvjjifZJPWQtFjSBknrJN0Rd0ypIqmZpA8kvRJ3LKkgqaOkuZLyo9/3uXHHlGyS7oz+rtdKmiOpVdwxNTRJT0kqkrS21rHOkt6UtCn63CkVsaRl4pDUDHgEuBwYCFwnaWC8USVdBXCXmQ0ARgJT0mDM1e4ANsQdRAr9GXjNzPKAoTTxsUvqBtwOnG1mgwiN4+pX1a9xewa47IhjdwMLzawfsDD6OunSMnEQ+pdvNrOtZnYQeAGYEHNMSWVmhWa2Mnq8j/Bi0sCdiBsfSd2BK4CZcceSCpLaAxcATwKY2UEz2xtvVCnRHDhFUnOgNbA75nganJm9DXx2xOEJwKzo8SxgYipiSdfE0Q3YWevrAtLgRbSapF7AcGBZvJGkxJ+AnwNVcQeSIn2AYuDpaHlupqQm3TTCzHYBDxFaSRcCJWb2RrxRpcypZlYI4c0h0PAVDeuQromjrl6RaXFdsqS2wDxgqpmVxh1PMkm6Eigys/fjjiWFmgNnAtPNbDhwgBQtX8QlWtefAPQGcoA2kibFG1XTlq6JowDoUevr7jTBqe2RJLUgJI3ZZvZS3PGkwCjgu5K2E5Yjx0p6Lt6Qkq4AKDCz6tnkXEIiacouBraZWbGZHQJeAs6LOaZU+URSNkD0uSgVPzRdE8cKoJ+k3pK+QdhIeznmmJJKkgjr3hvM7A9xx5MKZvZLM+tuZr0Iv+NFZtak34ma2cfATkn9o0PjgPUxhpQKO4CRklpHf+fjaOIXBNTyMjA5ejwZWJCKH5qW/TjMrELSrcDrhCswnjKzdTGHlWyjgBuBNZI+jI7dY2avxhiTS47bgNnRm6KtwI9ijiepzGyZpLnASsLVgx/QBMuPSJoDjAG6SCoA7gOmAS9KuomQQK9OSSxecsQ551wi0nWpyjnnXD154nDOOZcQTxzOOecS4onDOedcQjxxOOecS4gnDucaIUlj0qWarzv5eOJwzjmXEE8czp0ASZMkLZf0oaTHot4f+yX9XtJKSQslZUbnDpP0H0mrJc2v7p0gKVfSvyStir6nb/T0bWv11Zgd3RWNpGmS1kfP81BMQ3dpzBOHc/UkaQBwDTDKzIYBlcANQBtgpZmdCSwh3OEL8CzwCzMbAqypdXw28IiZDSXUWCqMjg8HphJ6xvQBRknqDHwPOCN6ngeTO0rnvsoTh3P1Nw44C1gRlXEZR3iBrwL+Fp3zHHC+pA5ARzNbEh2fBVwgqR3QzczmA5hZmZl9EZ2z3MwKzKwK+BDoBZQCZcBMSd8Hqs91LmU8cThXfwJmmdmw6KO/md1fx3nHqutTV4n/auW1HlcCzc2sgtCIbB6hac9rCcbs3AnzxOFc/S0ErpKUBTX9n3sS/l9dFZ1zPbDUzEqAzyWNjo7fCCyJeqIUSJoYPUdLSa2P9gOjfiodouKUU4FhyRiYc8eSltVxnWsIZrZe0r3AG5IygEPAFELzpDMkvQ+UEPZBIJS9nhElhtpVa28EHpP0QPQcx6pw2g5YIKkVYbZyZwMPy7mv5dVxnWtgkvabWdu443AuWXypyjnnXEJ8xuGccy4hPuNwzjmXEE8czjnnEuKJwznnXEI8cTjnnEuIJw7nnHMJ+R8fXyvl0Cxh5gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "dl.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA/kAAAJTCAYAAABTtdYwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd9glRZn38e9PMIAoKiZ0kVEEVASG4KIgCophhVVUFEbeVVxzes1hBRUzK7iIOQO6CrgiBhDFwLxEwwgDCAKCoAvKLhhYERZluN8/ug709JwnTGKYw/dzXec651RVV1d3n5557u6q6lQVkiRJkiRp9Xe7Vd0ASZIkSZK0YhjkS5IkSZI0IQzyJUmSJEmaEAb5kiRJkiRNCIN8SZIkSZImhEG+JEmSJEkTwiBfknSbk2SnJJVk/+WsZ59Wzz5LscxhbZk5y7NuSVpdLMu/lZKWnUG+JOkWk+TL7Q+9l82i7Pda2d1vibZNit4FjPmrui0rm4HDeEnmt/2y0zRlDhvuu3SenOQjSRYm+WOS/01yQZIPJbnPDOvdJMnHkpyf5Jokf2nLfjzJpsuxHaPXDa1N5yf5SpLnJ1lnaeu9NUqy/4q48ChJAGuu6gZIkm5TPg3MA14EfGKqQu0u9+OB3wHHroR2/AR4KHDVSqhbWl3dETge+CtwEvB9YA3gccCrgb2S7FhVvxwumOT/Av9GdwPpJLrztoBtgJcCL07yuqr68DK063DgUiDAXYEHArsAzwLel+QFVfXtZahXt5xjgB/R/ZsuaSUzyJck3WKqan6SC4GtkmxdVWdMUfSFdH/QH1pVN6yEdlwLnL+i65VWc4uA/YCPV9UfR4lJbgd8HHgJXSD/j/2FkjwXOAT4A/D0qjppkL8j8HXgkCR/qqovLGW7Dquq+YM67wS8HngXcEySJwzXq1uPqroauHpVt0O6rbC7viTplvaZ9v6icZlJ1gD2obsL+NmWtnuSf09yYesCfE2SnyX5vy0AGdYx6or8oCSvSnJ2kutGXdinGpOfZJskhyQ5K8kfWlflXyb5YJK7T7dRSXZNclpr3x+TfDXJxkuzY5Js15a7Islfk/xnkk8lud/S1DNF3Td1bU/yhCQnt/14ZZJDk9ytldsqybFtG65J8s1x8wf0ulLfMcl7klyS5PokFyd5R5I7TNGOxyf5Tm//XpjkgCTrTrOOOyR5e+v6fX07vvOBQ1vRQwfduue05e/Xlju1t09/m27YyEPHrG9OW/6w9vnIJFe1di5Ists0+3fPJD/obdelSY5Isu2YsvOSnJibu8P/Isl+Se44Vf23hKr6W1W9tx/gt/Qb6YJpgJ36eUnuAnyofX3OuEC7qk4G9m5fD27LLG9b/7eq3gu8B7gD3UWGxSRZM8nLk/woyf8kuTbJmUleOe7fjbbMrM/BZT0HVoQkf5fko0l+1db5+3auPmJM2eU5DzZJclSS/05yY7p/O5f6PMkUQ2vaeXJpkrWTHJjkN217Lkry5iQZU1eSvDrJeW2dl7d9se6ovuXbu9LqzyBfknRLO5yuO/Bzkqw9Jv8pwP2B71fVJS3tAGBr4MfAR4AvAuvQ/WF/+DTrOgR4N3BO+3zqDG17EbAXcAFdAPlJuu6lrwNOnSY4eQbdncrL2npOB54J/CizHIuc5Pmtff8AnEgXOC2g69WwIMkDZlPPLDwVOA64km77fkl3UeXrSR4JnELX0+9zrT3/CBw3VVAEfAX4Z+BbwEfpLs7sDxw9/AM9yUuA7wE70O2vD9Hd/X0zcFrahYYxjgZeDpzWljkHOAz4Rsv/BvDO3utPLf0xwFva96OBg+m6DO8B/DTJllOsb0O6IR1z6H5rRwEPB76RZOfBNiXJYcCRwBbA19p6TgZ2BHYblP8c8GXgwa3sx9o+eDfwnSRrDsrfWsZq/7W9D3vW7AHcHfhJVX13qoWr6jvAT4F7tGVWlIOA64C5STYbJSa5Pd2QgY8Bd6Pb55+m+9v3I4z5d2M5zsFZnwMrQpKtgYV058QFdNvzLbrf+ylJnjJYZFnPg43o/s2dA3yJbv/9Ty9/1ufJDG4PnED3b+bxdBd316L7d//tY8p/jO7YrNvadATwRLp/W26/FOuVJldV+fLly5cvX7foi+6PwQL2GZP3jZa3Ry9tozHlbkf3h3oB2w3yDmvplwMPHLPsTi1//0H6hsAaY8q/oJV/8yB9n5ZewG6DvFe39B9M0bY5vbRN6IKoi4D7D8o/jq4b9TGz3LejbZs/RVtvAB472I/fa3l/APYeLPe5lve0Qfr8ln4hcPde+p3oLnIU8E+DfXs9XZDwkEFdH2/lPz3FOs4G7jlmW0fbtMTvqOXfG7jLmPQtgWuA4wfpc3rH8x2DvCe19G8P0l/c0n8CrDvIWwNYf0x7vwasNSi7f8t79RTp+4/bxim2e7TfDmvLj3stnG7fjanzza38EVP8Pt47izre28p+dim3Y6cZyp3cyj1/zH77CL1zuh2TJX7TLMM5yFKeAzNsw6yOM90FuIuA/6V3Hre8+9H9m/c74I4r6Dx435jl+vmzPU9Gv/19BumXjsrTOydam//UXrfvpe/Yyl8A3K2Xfge6uSAKuHS254ovX5P6WuUN8OXLly9ft70X3aR6BZwySF8f+BtwRf8Pu2nq2brV8/ZB+mGMCZh6+TvN5g/qXvnQjSf94SB99IfrD8Yss0b7Y7yADce0bU4v7eCWtusU6z+GLjhf4g/1abZt/hRt/eKYZZ7b8k4ak/fYKf6Yn88UQUyvDSf20vadJmi4O13wfx2LByejdTxtuMxgm/ZZht/gN+kCpX4AMWcUJDD+Ys+vgasGaee0ZbaaxTrPbL/vu43JW4NuIsifDNLvCTyEMRc5plnPaL/N5jXjvgMeAVzbjtFGg7xvt3peOot6XsqYAHAW27HTDOWObOXe1L7fru3L3wFrjil/N+BG4Cu9tKU+B5f2HJhhG/ZndkH+01q5A6fIH11cfMoKOA+u6J+Py3me7DPu98bNQf6Dx9Qzuoj78F7aZ1vac8eU32HUrtmeK758TerLifckSavCD4GLgR2SPLSqftHSn093p+qwqvrbqHCS9YA30nXlfxBw50F9959iPT9Zmka1Lr4voeuy/zC67qD9bupTref/DROqalGSU+i6vG5F94fvVB7V3h87bkwt3V2tNejuNv5sum2YhQVj0n7b3sfVfXl7/7sp6lti2+nurN5At90jW7f3Hw4LV9Ufk5xJ1634IcBZgyJLdRz7kuxKF1xuSxcwD//2uSdLzvi9sKoWjanuP7n5WJHkznTdk/+rqs6coR1r0905vQp4zRS9uK+ne+rDTarqKpb9KRA712DCul57DgOeN1MFSTah6wp+e2Cvqrp4WGTU1Fm0Z2nKLo1hvZsA69ENRdlvin19HYvv6+U5B2d7DqwIo3ZuOMUQjtE8IA+luwADLPN5cFZVXT9NW2Z1nszC1VV10RT1QHcRcGS0P08ZU/5HLDmcRLpNMsiXJN3iqqqSfBZ4P91419e3saujbvGfHZVt47R/SvfYrJ8AX6DrVn4D3R25V9M9+mucK5ayaUcBTwd+RTds4Aq6wAvgNdOs579mWP8Sk8oNrNfe3zhDuRXxTPBxM1zfMIu8qca6LrHt7QLH7+kCo5HRPpjqEVqj9HHj8pf2OAI3PdbtEOCPdEMSfkN3R7qA3emC7nHH9E9j0qDbF/2LPqO2Xj6m7NDd6YLRewHvmEX5VS7dxJEn0o2j36uqvjmm2Oi4zWbOiNGFohX9GLXRpHhXtvfR+bQx0+/r/vm0POfgbM+BFWHUzmfNUO6mdi7HeTDTeTfb82Qm09UD3cWVkdG/I9Ptc+k2zyBfkrSqHEo3Y/dzk/wL3VjLB9F1ie/f1XkhXYD/zqrav19BkkfRBflTmfUdw3SzoD+d7tngTxn0JLgd8KZpFr/PFOn3be8zPTpqlL9uVf3PtCVvfe5DFzTcJN0TEtZj8Um6Rtt4X+DcMfWsPyh3k6pa6ju/bQK7d9IFKltX1e8G+Utzp3Eqo+Bkqh4efaPtOrOqtp625K1Am3X9B3TH8VlV9Y0pip5C1wNnF7ohGdPZpb3PNAHmrLXJMLdpX3/c3kf7+piqesYsq1qec3C258CKMGrn06a46LKY5TwPVnSPixVhtD/vQ3cx9ia9fT6bi27SRHN2fUnSKlFV/0U3HvSedHeTRo/U+/Sg6IPb+9FjqnnsCmzSaD3f7Af4zd/TzfY8lSXa0f7gfHT7Om1XbrpuptBd6FjdjDsGO9LdSOhv9+jzTsPCrbfGXLqxwb8Y5k9j1FV4jTF596S7037amMBmHW4ePrDMquovwM+B+ySZtlt2VV1Dd3FjsyT3WN51r0xJNqcbb34P4JnTBPgAX6W72PH3SZ4wTZ1PoDuP/tiWWVHeSHduntkb9nN+a9Mj2xCc2Viec3C258CKsLTtXOnnwS1stD8fPSbvkXgDUwIM8iVJq9Zn2vvr6QL9q+gmuOq7tL3v1E9sQdW/rMC2TLWee9M9smk6jxvzbOhX0o3HP7GqphuPD91jt/5G9wzxTYaZ6Z4Tf2u9APC2JDeNmU1yJ7phGHDzc+wB/p1uG1+V5MEs7t3AXYF/n2EM8NCoa+64ruL/TdcleZsWzIzad3u6rsv3XIr1TOfD7f1TSRYblpHkdknW7yX9G90s4J8f97jAJHdvj0frp90zyUOSrKj2TivJXLou+nehu1t87HTl213v17evX06yw5g6t6d7hB3Aa6rqzyugnXdK8la63gN/Bf5vr0030M2qvz7w4SRLXKBLsn6Sh/WSluccnO05sCJ8g24+k1eMeVTeaP2P6j2e9JY6D24pX2jv+/bPtyR3AN63apok3fp4tUuStCqdAFxCd4cP4KNV9ddBmS/Q3a37UHv28i/pxtruRvcosj1XUFt+SteN+BlJTqPrhnwfumdmX8DNk9ON8y3gmCTH0M2ovyXdJIF/oHuW9bSq6vwk/wx8Hjg3yXfoHst1e7oAdke68cYPWbZNW6l+Qdfmr9IFSU+ju7hxHN2zswGoqkuTvIbugskZSb5Ct02PpZuk63y6x7QtjdPpApjXtLvjo3G6H6mqq5N8mO754Ock+QZdgL0z3R3qE9vn5fVZuruKzwV+2dZzJd048cfRHdP9Aarq80m2oftNXJzku3TdvO9BNyTlMXRB4Ut79b+Sblz5O0f1rCwtUP1Ba88PgEdN0Z37Q1V10zjqtl13Az4AnJxkPt3kdEXXlX5nutnsX1NVXxhT30z2SbJT+7wO3e/rMa2dvwP+uaqGE7G9m+48fCnwj0l+SNeN+950/37sQHeB4Ly2DctzDs7qHJil3ZPMmSLvhKr6cpJnAN8Fjmv/Vi2kOw82oHsSwoPoLnBcW1U33kLnwS2iqv5fkk/TPbry3CRH0+3zf6QbyvBbut+adJtmkC9JWmXaBHyfA97Tkj4zpsxv2x20A+iCqSfRBYQvpxs/v0KC/DZp01NbW55Cd2fwcrog7j20YGAKX6MbZrAvsCvdH51fA/6lqi6c5fr/PclZdHdFdwaeCPyF7o/Wr9JNCnhr9GzgbcDe3Pyc7v2BA4Zj6avq40kuAt4APBNYm24G7QPpHq031QRcY7VZ+Z9JFwQ/n5ufuvDvdH/wv40uMHsh3VMTrqabeGw/uqB5ubVtfF6SE+gCj2fTTWL2O7oZ1r85KP+KJMfTBZ+70HWl/gNdsH9ga/uqsi5d4AfdYy4fP0W5wxhMllZV/5bk23RzZDyOrus0wGXAp4BDqur8ZWzX6CkAi+ie634F3bl/PPAfbdjEYqrqb0l2B/4P3ePbdqO7QHAl3YXFtwFfGiyzrOfgrM+BWdiyvcb5E/Dlqjo7yZbA69p2PZ8usP0dXXf2d7D4ExlW+nlwC3sZ3f8BL6E7j35P1wPsrXS/t+ETIKTbnCzDXDaSJOk2rt2tfWxVjX0+mTTpPAduXdrTIC4Ejqyqeau6PdKq5Jh8SZIkSauFJPdtTzzpp60NfKh9Hc7rIt3m2F1fkiRJ0uriNcC81pPid3SP5Xw88He0IRyrrmnSrYNBviRJkqTVxffo5i14It0cEjfQddP/MN2kkI5F1m2eY/IlSZIkSZoQ3smXbmUOP/zwet7znjdzQUmSJEm3VVNO+unEe9KtzF/+ssSTgCRJkiRpVgzyJUmSJEmaEAb5kiRJkiRNCIN8SZIkSZImhEG+JEmSJEkTwiBfkiRJkqQJYZAvSZIkSdKEMMiXJEmSJGlCGORLkiRJkjQhDPIlSZIkSZoQBvmSJEmSJE0Ig3xJkiRJkiaEQb4kSZIkSRPCIF+SJEmSpAlhkC9JkiRJ0oQwyJckSZIkaUIY5EuSJEmSNCEM8iVJkiRJmhAG+ZIkSZIkTQiDfEmSJEmSJoRBviRJkiRJE8IgX5IkSZKkCWGQL0mSJEnShDDIlyRJkiRpQhjkS5IkSZI0IQzyJUmSJEmaEAb5kiRJkiRNCIN8SZIkSZImhEG+JEmSJEkTYs1V3QBJizvn8quZ85bjVnUzJEmSJAGXHrDrqm7CUvFOviRJkiRJE8IgX5IkSZKkCWGQL0mSJEnShDDIlyRJkiRpQhjkS5IkSZI0IQzyJUmSJEmaEAb5kiRJkiRNCIN8SZIkSZImhEG+JEmSJEkTwiBfkiRJkqQJYZAvSZIkSdKEMMiXJEmSJGlCGORLkiRJkjQhDPIlSZIkSZoQBvmSJEmSJE0Ig3xJkiRJkiaEQb4kSZIkSRPCIH8lSLJvknOTnJ1kYZLtWvr8JBe0tIVJvtrS90/yhjH1XDMmbf8kl/fqWJjkbkl2SnJ1kjOTnJ/koBnauE+SG5Ns0Uv7eZI5ve9bJakkTxosW0m+2Pu+ZpIrkxzbq/vKQRsfNqYN6yb5QpKL2+sLSdZteXOSPGfQ3o9Ot00rS2vLz9vnnUbbKUmSJEm3Ngb5K1iSRwG7AVtX1RbALsB/9orsXVVz22uPZVzNwb065lbVn1r6yVW1FbAVsFuSHWao5zJg32ny5wGntPe+vwAPT7JW+/4E4PJBmaMGbTxvTP2fA35VVRtV1UbAJcBnW94c4DljllkmSdZYUXUt5XrXXBXrlSRJknTbZJC/4q0PXFVV1wNU1VVV9dtbsgFVdR2wELj/DEWPBTZLsukwI0mAPYB9gCcmudOgyPHAru3zPOCIpWljkgcD2wDv7iW/C9g2yUbAAcCOrRfAa1v+/ZJ8J8kvk3ygV9cTk5ye5Iwk/5FknZZ+aZK3JzkFeNZg/c9qPRfOSnJSS1sjyYFJftp6Ybxkhm24c5LPt/JnJnlaS9+nteNbwAlJ1k9yUtuWnyfZcUxdL06yIMmCRddevTS7UpIkSZJuYpC/4p0AbJDkwiQfT/LYQf6Xel3YD1zGdby2V8eJw8wkdwc2Bk6aoZ4bgQ8Abx2TtwNwSVVdDMwHnjLIPxLYqwX/WwA/HuTvOeiuv9Yg/2HAwqpaNEponxcCmwFvoeuZMLeqDm5F5gJ7Apu3+jdIck9gP2CXqtoaWAC8rree/62qR1fVkYP1vx14UlVtCTy1pb0AuLqqHgE8AnhRkgeO2Tcj+wI/bOV3Bg5McueW9yjgeVX1OLoeCd+tqrnAlm0bF1NVn66qbatq2zXWXneaVUqSJEnS1OxKvIJV1TVJtgF2pAv8jkrylqo6rBXZu6oWLOdqDq6qcWPud0xyNrApcEBVXTGLur4M7DsmmJ1HF8jT3v8J+Noos6rObuP35wHfHlPvUVX1ymnWG6CWIh3gB1V1NUCS84ANgbvRXTA4tet8wB2A0/vtmKKuU4HDknyFm7fricAWSUbDKNalu1hy4RR1PBF4am8+hTsBD2ifv1dVf2iffwp8Psntga9X1RJBviRJkiStCAb5K0G7Iz0fmJ/kHOB5wGG3wKpPrqrdkmwCnJLkmJkCyqq6IckHgTeP0tr49WfSBbD70gXe6yW5S1X9ubf4N4GDgJ2A9ZayrecCWyW5XVXd2NZ7O7o73b8A/m7MMtf3Pi+i+/2GLqAezhsw8pdxiVX10nQTIu4KLEwyt9X1qqr6br9sfzLCgQDPrKoLBuW366+3qk5K8pi2ri8mObCqvjBFnZIkSZK0zOyuv4Il2TTJxr2kucCvb8k2VNWFwPvpBe4zOIxugsB7te+7AGdV1QZVNaeqNgSOBnYfLPd54F1Vdc4ytPEi4Ey6rvYj+wFntLw/A3eZRVU/AnZoY/xJsna7yDGtJBtV1Y+r6u3AVcAGwHeBl7U77iTZpNf9fpzvAq9q8xeQZKsp1rUh8N9V9Rm6yQa3nsV2SZIkSdJSM8hf8dYBDk9yXus6/zBg/15+f0z+93vp+yW5bPRqaWv305KMxpr3x+QvnOJO8yeBx8wwphyAqvor8GHg3i1pHnDMoNjRDGa7r6rLquqQKaodjsnfHiBJv2fBC4BNklyU5GJgk5YGcDZwQ5sY77VMoaqupJsc8Ii2v38EPGT6LQa68fPnpHs03knAWXQz+58HnNHSP8X0vV3eDdweOLuVf/cU5Xai6y1wJl0Pian2mSRJkiQtl1RNNfxZ0qrwsn3fX8cv2mJVN0OSJEkScOkBu85c6JaXqTK8ky9JkiRJ0oRw4r0Jl+T5wKsHyadW1StWRXskSZIkSSuPQf6Eq6pDgUNXdTskSZIkSSuf3fUlSZIkSZoQBvmSJEmSJE0Ig3xJkiRJkiaEQb4kSZIkSRPCIF+SJEmSpAlhkC9JkiRJ0oQwyJckSZIkaUIY5EuSJEmSNCEM8iVJkiRJmhBrruoGSFrc5vdfl0+8fNdV3QxJkiRJqyHv5EuSJEmSNCEM8iVJkiRJmhAG+ZIkSZIkTQiDfEmSJEmSJoRBviRJkiRJE8IgX5IkSZKkCWGQL0mSJEnShDDIlyRJkiRpQhjkS5IkSZI0IdZc1Q2QtLhzLr+aOW85blU3Q5IkSbrNuPSAXVd1E1YY7+RLkiRJkjQhDPIlSZIkSZoQBvmSJEmSJE0Ig3xJkiRJkiaEQb4kSZIkSRPCIF+SJEmSpAlhkC9JkiRJ0oQwyJckSZIkaUIY5EuSJEmSNCEM8iVJkiRJmhAG+ZIkSZIkTQiDfEmSJEmSJoRBviRJkiRJE8IgX5IkSZKkCWGQL0mSJEnShDDIlyRJkiRpQhjkS5IkSZI0IVZakJ9k3yTnJjk7ycIk2yWZn+SC9n1hkq+2svsnecOYOq4Zk7Z/kst7dSxMcrckOyW5OsmZSc5PctAM7dsnyY1Jtuil/TzJnN73rZJUkicNlq0kX+x9XzPJlUmO7dV95aCNDxvThkuTHN37vkeSw3rfd2/77/wk5yTZvZd3WJJLWt1nJXl8Sz9mUO6CJPv1vh+d5Bm9/dVv4y6tzLhjd0z7fNFgue3bcf1NkvTW8/X+8UuyWZIfJrkwyS+TvG1UfrC/zk/y2iRP6q3jmt7v5gttmb9v6/1lkjOSHJdk88H+PSvJEYO0NZO8ry03qn/fXv6iwT55y5jjdliSPdrn+UkW9PK2TTK/9/3vk5zU2n9+ks8mWXtYpyRJkiStCGuujEqTPArYDdi6qq5Pck/gDi1776paMPXSs3JwVS0WxLd48eSq2i3JWsCZSY6pqlOnqecyYF9gzyny5wGntPfv9tL/Ajw8yVpVdR3wBODywbJHVdUrZ7Et2ybZrKrOHWzPlsBBwBOq6pIkDwS+l+RXVXV2K/bGqvpqkp2BTwMbA6cB2wNfT7IecA3wqF7VjwJeATyEtr8G6x177Krq6S1/J+AN/eXavv8TsANwSpK7Aev38tcCvgm8rKpOaEHu0cDLgY/191dr8wXAVlU1ty0/v61zQft+H+ArwHOq6rSW9mhgI+Cc9v2hdBexHpPkzlX1l7ae9wD3BTavqv9Nchfg9b1dcN1ovUvh3kn+oaqOH+zL+wD/AexVVae3ixrPBO4CXLuU65AkSZKkGa2sO/nrA1dV1fUAVXVVVf12Ja1rCS3wXgjcf4aixwKbJdl0mNECsj2AfYAnJrnToMjxwK7t8zzgCJbNQcBbx6S/AXhfVV0C0N7fD7xxTNnTuXlbT6UL8mnvxwL3SueBdEHsFdO0Z1mP3ZHAXu3zM4Cv9fKeA5xaVSe0Oq8FXgkscZe8qn4PXETvIsEYrwQOHwX4bblTqurrg3V+ETgBeCpAu7jwIuBVVfW/bbk/V9X+s9i+6RwI7Dcm/RWtnae3dVVVfbWq/mtYMMmLkyxIsmDRtVcvZ3MkSZIk3VatrCD/BGCD1jX740ke28v7Uq8r9IHLWP9re3WcOMxMcne6u9onzVDPjcAHGB9k7wBcUlUXA/OBpwzyjwT2asH/FsCPB/l7Drp9rzVFG74CbJ3kwYP0zYCfDdIWtPShJwOjAPdndL0M7kAX5J9Od2f8oe17v2fDjoM2bsT0x246P6C7a74GXbB/1HTb0vbrOknu2k9P8gDgTsDZTG0z4IwZ2rNna8MRdBdhAB4M/Kaq/jzNcmsN9slUvTz6Tgeubz0q+h7OksdwrKr6dFVtW1XbrrH2urNZRJIkSZKWsFKC/Kq6BtgGeDFwJXBUkn1a9t5VNbe9xt2Vno2De3X0A6sdk5wNXAEcO8Md65EvA49sd7n75tEF8rT3ef3M1mV+Tkv/9ph6j+q1cW7rXTDOIro7wf8ySA9QM6QdmORXwL8D72vtuh44F9gaeCTdxYfT6QL87em684+cPGjjxTMcu+ksohvasCewVlVdOsO2jIzS90xyLvAr4JDRnfbZSPLjJL9Ickj7/gjgyqr6Nd3Fh63bhZ/hcs9vgfx/JtmgJV832CdHDZebwnsYfzdfkiRJkm4xK23ivapaVFXzq+oddN2rn7my1tVzclVtAWwOvCzJjGOrq+oG4IPAm0dp7W70M4G3J7kU+AjwD238dt836brbL2tX/ZEvAo8BHtBLOxfYdlBua+C83vc30t2d3g84vJd+WqvvLlX1R+BH3BzkTzdHAbBcx+5Iun31lUH6EtuS5EHANb276kdV1WbAjsAHk9x3mvWMLmKM2rsd8DZgdAt8HvCQduwuBu7atuEi4AGj41hVh7bx91cDa8xyG8eqqh/S9UB45KCd2yxPvZIkSZK0NFZKkJ9k0yQb95LmAr9eGesap6oupBu//uaZyjaHAbsA91LMSf0AACAASURBVGrfdwHOqqoNqmpOVW1IN1Hc7oPlPg+8q6rOWc72/g04GHhNL/kg4F/SZvtv72+luyDRX/ZG4BDgdrn5KQCnAi8Bzmrfz6YLPh9AF3hOaTmP3cl0+3140eNLwKNz8+z9awEfphsqsZg2fv2LwKunWc/HgH2SbN9LW7vVfTvgWcAW7djNAZ4GzGtzAXwO+OhojoV2QecOzCDJ+5M8fYZi7wXe1Pv+UeB5Sbbr1fN/ZriAIUmSJEnLbGXdyV8HODzJea37/MOA/Vtef0z+93vL7JfkstGrpa3dT0vyupbeH5O/ML3H3vV8km6M+LAb/hKq6q90Qee9W9I84JhBsaPpJnPrL3dZVR0yRbXDMfnbAyRZOEX5z9F72kFVLaS7SPGtJOcD3wLe1NKH7S+67uKjAPM04EF03fRHvRX+G1jQLgqMDMfk78H0x25abWK5g6rqqkH6dXSB9n5JLqCbAf+ndEHwOP8KPH9Mz4lRfVfQDQt4f7pH+p1GN0niR+l6MFxeVf2nHZwEPCzJ+nRPU/gd8PMkZ9JdmDgcGE0uOByTf0BL35xuGMh02/9tuiEOo+//RTc/wUHpHqH3C7qeCv8zXT2SJEmStKzSxYeSppPku1X1pJlLLr+X7fv+On7RFrfEqiRJkiQBlx6w68yFbl0yVcZKG5MvTZJbKsCXJEmSpOWx5sxFVm9Jns+S47tPrapXrIr2SJIkSZK0skx8kF9VhwKHrup2SJIkSZK0stldX5IkSZKkCWGQL0mSJEnShDDIlyRJkiRpQhjkS5IkSZI0IQzyJUmSJEmaEAb5kiRJkiRNCIN8SZIkSZImhEG+JEmSJEkTwiBfkiRJkqQJYZAvSZIkSdKEWHNVN0DS4ja//7p84uW7rupmSJIkSVoNeSdfkiRJkqQJYZAvSZIkSdKEMMiXJEmSJGlCGORLkiRJkjQhDPIlSZIkSZoQBvmSJEmSJE0Ig3xJkiRJkiaEQb4kSZIkSRPCIF+SJEmSpAmx5qpugKTFnXP51cx5y3GruhmSJEnSLe7SA3Zd1U1Y7XknX5IkSZKkCWGQL0mSJEnShDDIlyRJkiRpQhjkS5IkSZI0IQzyJUmSJEmaEAb5kiRJkiRNCIN8SZIkSZImhEG+JEmSJEkTwiBfkiRJkqQJYZAvSZIkSdKEMMiXJEmSJGlCGORLkiRJkjQhDPIlSZIkSZoQBvmSJEmSJE0Ig3xJkiRJkiaEQb4kSZIkSRNitQnyk+yb5NwkZydZmGS7lj4/yQUtbWGSr7b0/ZO8YUw914xJ2z/J5b06Fia5W5Kdklyd5Mwk5yc5aIY27pPkxiRb9NJ+nmRO7/tWSSrJkwbLVpIv9r6vmeTKJMf26r5y0MaHjWnDpUnO6ZXZPsmcJD9v+VNuU5L7JDk2yVlJzkvy7SSb9+r6Q5JL2ufvD9Z70zoG+/UN7fNhbR/fsX2/Z5JLxy2b5NFJftLad36SFw/qvDbJvXtpSxzTlr5Okk8lubj9dk4a/W5a/tPbfn/IYDuua9t4XpJPJrndqI1JntTbH9f0fntfmKHOnyNJkiRJK9maq7oBs5HkUcBuwNZVdX2SewJ36BXZu6oWLOdqDq6qxYL4JAAnV9VuSdYCzkxyTFWdOk09lwH7AntOkT8POKW9f7eX/hfg4UnWqqrrgCcAlw+WPaqqXjmLbdm5qq7qbcecQf5U2/Qu4HtVdUhbbouqOgeY274fBhxbVV+dRRvGWQT8M/CJqQokuS/wZWD3qjqjHevvJrm8qo5rxa4CXg+8eYb1fRa4BNi4qm5M8iDgob380bHYC9i/l35xVc1NsibwQ2B34AyAqvou7bglmQ+8YfDbm6pOSZIkSVrpVpc7+esDV1XV9QBVdVVV/faWbEALvBcC95+h6LHAZkk2HWaku2qwB7AP8MQkdxoUOR7YtX2eBxyxPG2eyZhtWp/uIsUo/+wVvMoPAa9twfNUXgEcVlWjoPoq4E3AW3plPg/smeQeU1WSZCNgO2C/qrqx1fWr0YWCJOsAOwAvoAvIl1BVNwCnAQ+ezcbNps5pln1xkgVJFiy69uqlWVSSJEmSbrK6BPknABskuTDJx5M8dpD/pV4X6gOXcR2v7dVx4jAzyd2BjYGTZqjnRuADwFvH5O0AXFJVFwPzgacM8o8E9mrB/xbAjwf5ew666681RRtObPnD5RczZps+BnwuyYnphkfcb7rll8Fv6O5y/9M0ZTYDfjZIW9DSR66hC/RfPUM9C6tq0RT5uwPfqaoLgT8k2XpYIMnawOOBc6ZZz1LVOZWq+nRVbVtV266x9rqzXUySJEmSFrNaBPlVdQ2wDfBi4ErgqCT79IrsXVVz2+uNy7iag3t17NxL3zHJ2cAVdF3Vr5hFXV8GHpnkgYP0eXSBPO19Xj+z3Tmf09K/Pabeo3ptnNvuxI+zc8vfbor8sdvUuqI/CPgM8BC6rvz3mm5D+82fZfr7gDcy9W8vU9Q1TPsw8Lwkd51l+4amOxYbJVkInAocV1XHr4A6JUmSJGmlWy3G5AO0O7LzgflJzgGeBxx2C6x6NH59E+CUNn594XQLVNUNST5Ib8x4kjWAZwJPTbIvXTC7XpK7VNWfe4t/EzgI2AlYbwVvy8iU21RVf6C7SPHldJP+PQY4ehZ1/h64+yDtHnRj4m9SVRe1APrZU9RzLrAt3X4Y2QY4b1DPn5J8GXj5NPVsmeR2o+76I0nWAx5HNwdCAWsAleRNrcjFVTV3inrHmkWdkiRJkrTSrRZ38pNsmmTjXtJc4Ne3ZBtaF+z3M/NkbyOHAbsAozvhuwBnVdUGVTWnqjakC553Hyz3eeBdbcK7lWq4TUke17qok+QuwEZ0XexnU9c1wO+SPL4tfw/gyXTd84feCyzx5IPmY8A+SUaT/a0H/CvdEIihfwNewpiLVW1IxALgnW0uBJJsnORpdPMifKGqNmzHYgO6ixGPns22TmFl1ClJkiRJS2W1CPKBdYDD2yPNzgYexuIzl/fH5Pcf7bZfkstGr5a2dj8tyetaen9M/sIxM9IDfBJ4zJhu+Euoqr/SdSkfPeptHnDMoNjRwHMGy102mt1+jOGY/O0B2p3xZdXfpm2ABW0fnw58tqp+uhR1PZduny+km5X+nS3YXkxVnUubrX5M3u+A/wN8Jsn5dBPffb6qvjWm7FV0+/SOU7TnhcB9gYta74/PAL9llsdiKc1U56aD392zlmNdkiRJkjRWqqYaSi1pVXjZvu+v4xdtsaqbIUmSJN3iLj1g15kLCbrh32OtLnfyJUmSJEnSDFabifduTZI8nyUf33ZqVb1iVbRHkiRJkiQwyF8mVXUocOiqbockSZIkSX1215ckSZIkaUIY5EuSJEmSNCEM8iVJkiRJmhAG+ZIkSZIkTQiDfEmSJEmSJoRBviRJkiRJE8IgX5IkSZKkCWGQL0mSJEnShDDIlyRJkiRpQhjkS5IkSZI0IdZc1Q2QtLjN778un3j5rqu6GZIkSZJWQ97JlyRJkiRpQhjkS5IkSZI0IQzyJUmSJEmaEAb5kiRJkiRNCIN8SZIkSZImhEG+JEmSJEkTwiBfkiRJkqQJYZAvSZIkSdKEMMiXJEmSJGlCrLmqGyBpcedcfjVz3nLcqm6GJEmSVhOXHrDrqm6CbkW8ky9JkiRJ0oQwyJckSZIkaUIY5EuSJEmSNCEM8iVJkiRJmhAG+ZIkSZIkTQiDfEmSJEmSJoRBviRJkiRJE8IgX5IkSZKkCWGQL0mSJEnShDDIlyRJkiRpQhjkS5IkSZI0IQzyJUmSJEmaEAb5kiRJkiRNCIN8SZIkSZImhEG+JEmSJEkTwiBfkiRJkqQJYZAvSZIkSdKEmFWQn2TfJOcmOTvJwiTbJZmf5IL2fWGSr7ay+yd5w5g6rhmTtn+Sy3t1LExytyQ7Jbk6yZlJzk9y0Azt2yfJjUm26KX9PMmc3vetklSSJw2WrSRf7H1fM8mVSY7t1X3loI0PG9Tx6iQf6n3/VJLv976/KsmH2+dFg7re0tLnJ9l2UO9Oo3YM0ndr++asJOcleckU+/OA3jZdleT9g3pGx/CsJD9NMreXt06STyS5uK3rZ0le1MvfLMkPk1yY5JdJ3pYkvX320THr2rZ9XjfJF1rdF7fP67a8Oe2YvLu37D2T/G1U5wy/m0rygjHH/Q3te5Ls19p8YZITk2zWK39pknPab/3/JdkwyTFtHRe13+Vondu3ffu+Vt8ofd/Btj+9teEhw2MpSZIkSSvSjEF+kkcBuwFbV9UWwC7Af7bsvatqbnvtsYxtOLhXx9yq+lNLP7mqtgK2AnZLssMM9VwG7DtN/jzglPbe9xfg4UnWat+fAFw+KHPUoI3nDfJPA7bvfZ8LrJtkjfZ9e+DU9vm6QV0HzLBdi0lye+DTwD9W1ZZ0+2d+r0h/f76lpT0RuAB49igQ79m71fNx4MBe+meBPwIbt+PwZOAerQ1rAd8EDqiqTYAt2za+fJab8TngV1W1UVVtBFzS1jfyK7rf3MizgHMHdUz1uzkH2LNXbi/grN73V7S2btna/n7gm0nu1Cuzc/utzwf2q6qnV9Vc4IV0v8vROk8D3gPcD9i8ldkRuP2graPf3l4z7BdJkiRJWi6zuZO/PnBVVV0PUFVXVdVvV26zblZV1wELgfvPUPRYYLMkmw4zWmC7B7AP8MRBQAdwPLBr+zwPOGIpm3kmsEmStdod6Wtbmzdv+dvTXQhYEe4CrAn8HqCqrq+qC2ZYZh5wCPAb4JFTlDmdto+TbAT8PV2Ae2Nbz5VV9a+t7HOAU6vqhJZ3LfBK4C1L1DqQ5MHANsC7e8nvArZt6wW4DvhFr2fDnsBXZqq7+Q1wpyT3acf9yXTHd+TNwKtam2nbcBqw95i6btonU2zL2sCLWn3/2+r7c1Xt3yuzDrAD8AKmCfKTvDjJgiQLFl179aw2VJIkSZKGZhPknwBs0Lo2fzzJY3t5X+p1UT5wqgpm8NpeHScOM5PcHdgYOGmGem4EPgC8dUzeDsAlVXUx3d3ZpwzyjwT2asH/FsCPB/l7DrqGr9XPrKob6IL6R9AF0T8GfgRsn+R+QKpq1PthrUFd/bvOM6qqP9DdRf91kiOS7J2kfxz7+/NJra2Pp7sIcgRL9mQYeTLw9fZ5M+CsUYA/xmbAzwbtuhhYJ8ldW9Ji+wwYBewPAxZW1aLesovo9t9mvSpHx+TvgEXA8MLSdL+br9Ld/d8eOAO4HqC17c6trX0LBuse6e+TcR4M/Kaq/jxNmd2B71TVhcAfkmw9rlBVfbqqtq2qbddYe91pqpMkSZKkqa05U4GquibJNnTdkHcGjkobR07X1XvBcrbh4KoaN+Z+xyRnA5vSdQu/YhZ1fRnYN8kDB+nz6IJG2vs/AV8bZVbV2enG788Dvj2m3qOq6pUzrPtUuqByLbo7wL+ku+BwJYvfxb+udeteZlX1wiSb0w2deAPdEIN9WvZi+zPJs4ATq+raJEcDb0vy2l6Q/aUkdwbWAMYGoG2M+bOAe1fV/YAANVXz2vti+yzJ/NHHKZYdpn+H7m7/fwFHjSk/1e8Gurv+RwEPobuwsf0U5aZa94lJ7gP8N7DfDMveXEnyfODVwHrA9u3CzjxgNF/Dke37GbOtU5IkSZKWxqwm3quqRVU1v6reQdct+5krt1lAN/Z5C7ou7y9Lb1K4qbQ76h+k65INQBsX/0zg7UkuBT4C/EOSuwwW/yZwEEvfVX9kNC7/UXRB/i/o7lr3x+OvMFV1TlUdTBfgT3c85gG7tG3/GV0AunMvf2/ggXQXSD7W0s4Dthz1EKiq97YLE6O79Ody8515AJI8CLhmhrvao2W36vc+aJ+3pNtno+37a2vv64GjZ6hzMe2C0N/o9s0Peun/A/yltbVva7ptHtkZ2LC19V3TrOoi4AGj31JVHdr209XAGknWAx4HfLbt/zfS9XAYzosgSZIkSSvEbCbe2zTJxr2kucCvV16TFte6Ob+fXuA+g8Po7nDfq33fha7r+QZVNaeqNqQLGncfLPd54F1Vdc4yNvU0uq7696qq/66qoruL/zRW3Hj80az3O/WSpjwerXv6o4EHtG2fQzfx3GJd9qvqb3R3rB+Z5KFVdRFdF/b3jCYPbEMZRsHpl4BHJ9ml5a0FfJhuuMS0Wt1nsvgd8v2AM1pe3weBN1fV72eqd4y3t2UXDdIPBD48GnLRtuHRdBc5+u28DngN8Nwk95hiW66lm0Two6N5Htr+ukMrsgfwharasO3/DegmGXz0MmyPJEmSJM1oNnfy1wEOT/eotrPp7k7v3/L6Y/K/31tmvySXjV4tbe1+WpLXtfT+2OqF6T32rueTwGPGdMNfQrsD/GHg3i1pHnDMoNjRdJPH9Ze7rKoOmaLa4Zj87QHaWPPR8n+kC+r7s8Cf3trRn919OCa/P7v+cb398x8t7fGDfbkV8Ka0xxcC7+TmrvpDzwB+OJo0sfkG8NQkdxxs/3V0QfXo8YcvpLvrf1GSnwHfp11oaWWfRnecL6Cb0f6nwGKPzZvGC+gmKrwoycXAJi1tMVV1blUdPkUd0/5uquq0qho3nv4jra3ntLa/DXha26bh+n9H17PjFdNsy77A74CfJzkTOBk4nG4OgVn99iRJkiRpRUl3w1nSrcXL9n1/Hb9oi1XdDEmSJK0mLj1g15kLadJMOQR4VmPyJUmSJEnSrd+Ms+vfmvRmL+87taqm604tSZIkSdJtwmoV5FfVocChq7odkiRJkiTdGtldX5IkSZKkCWGQL0mSJEnShDDIlyRJkiRpQhjkS5IkSZI0IQzyJUmSJEmaEAb5kiRJkiRNCIN8SZIkSZImhEG+JEmSJEkTwiBfkiRJkqQJYZAvSZIkSdKEWHNVN0DS4ja//7p84uW7rupmSJIkSVoNeSdfkiRJkqQJYZAvSZIkSdKEMMiXJEmSJGlCGORLkiRJkjQhDPIlSZIkSZoQBvmSJEmSJE0Ig3xJkiRJkiaEQb4kSZIkSRPCIF+SJEmSpAmx5qpugKTFnXP51cx5y3GruhmSJElaTVx6wK6rugm6FfFOviRJkiRJE8IgX5IkSZKkCWGQL0mSJEnShDDIlyRJkiRpQhjkS5IkSZI0IQzyJUmSJEmaEAb5kiRJkiRNCIN8SZIkSZImhEG+JEmSJEkTwiBfkiRJkqQJYZAvSZIkSdKEMMiXJEmSJGlCGORLkiRJkjQhDPIlSZIkSZoQBvmSJEmSJE0Ig3xJkiRJkibErIP8JPsmOTfJ2UkWJtkuyfwkF7TvC5N8tZXdP8kbxtRxzZi0/ZNc3qtjYZK7JdkpydVJzkxyfpKDZmjfPkluTLJFL+3nSeb0vm+VpJI8abBsJfli7/uaSa5Mcmyv7isHbXzYmDask+RTSS5u++qkJNuN2/ZW50cHaWclOWKQ9sgkP27r/EWS/adrU5I5bXve3avjnkn+Nsv1HdaOxx17y17aPs9J8vNB+cWOddt3VyV5f/u+b699i3qfq72fl+S6XvoebblvJDl9zLquTXLvXto1vc9THsckz++t469JzmmfD2hln5zkJ+23tjDJUUke0Nsnl/SWP623jn9IsqAdm8V+p0le3NLOb3U/GkmSJElaidacTaEkjwJ2A7auquuT3BO4Q8veu6oWLGc7Dq6qxYL4JAAnV9VuSdYCzkxyTFWdOk09lwH7AntOkT8POKW9f7eX/hfg4UnWqqrrgCcAlw+WPaqqXjnDdnwWuATYuKpuTPIg4KEzLANAkofSXXR5TJI7V9VfWtbhwLOr6qwkawCbTtemdlHjV3TH620t+VnAubNcH8Ai4J+BT8ym7QNPBC4Anp3krVX1XuC9bZ3XVNXcMe09tp+e5G7A1sA1SR5YVZf0FrkKeD3w5jHrnvI4VtWhwKGt/kuBnavqqvb94cBHgKdW1S9a2lOBOcBvWt1vrKqvDtr+cOCjwK5VdX6SNYEXt7zdgJcAj66qq5JsDXw9yd9X1RUz70ZJkiRJWnqzvZO/PnBVVV0PUFVXVdVvV16zFtcCtoXA/WcoeiywWZJNhxnprhrsAewDPDHJnQZFjgd2bZ/nAUewFJJsBGwH7FdVN7Z2/6qqjptlFc8BvgicADy1l35v4HetvkVVdd4s6roO+EWSbdv3PYGvzHJ9AB8CXtuC1qU1DziELjh+5DIsD/BM4FvAkcBeg7zPA3smuccUyy7LcXwz8L5RgA9QVd+sqpNmWO5NwHur6vy2zA1V9fFenW8cXUioqjPoLti8YlxF7a7/giQLFl179SyaLEmSJElLmm2QfwKwQZILk3w8yWN7eV/qdWM+cBnb8dpeHScOM5PcHdgYmCnouhH4APDWMXk7AJdU1cXAfOApg/wjgb1a8L8F8ONB/p6DrvFrDfI3AxZW1aIp2rZWf3ngXcP6gaPogtJ5vfSDgQuSHJPkJYOLE9O1abQ9f0d3Z354UWaq9UEXoJ8C/NOY7dhosB0vHWW09T+e7mLLuHpnaxScj6vjGrpA/9VTLDvTcRxnM+CMGcoc2NvuL7W0hwM/m6bOYd6Clr6Eqvp0VW1bVduusfa6s2iyJEmSJC1pVkF+VV0DbEPXFflK4Kgk+7Tsvatqbnu9cRnbcXCvjp176TsmORu4gq5L92y6OX8ZeGSSBw7S59EFgLT3xYLHqjqbrnv2PODbY+o9qtfGua13wdK4rr888PZRRpJHAFdW1a+BHwBbtwsbVNW7gG3pLrQ8B/jOLNv0Hbru6vPogvmbTLe+nvcBb2TJ38jFg+34ZC9vN+DEqroWOBp4ehtiMGtJ7gM8GDilqi4Ebmjd4vs+DDwvyV2Hy8/iOM60/vVaIH9hFp9X4o297d57aesdVQ/UMi4rSZIkSTOa9cR7rav4/Kp6B/BKui7VK9vJVbUFsDnwsiRzZ1qgqm4APkhvzHYLNJ8JvL2Nx/4I8A9J7jJY/JvAQSxlV/3mXGDLJMvyxIJ5wENa2y4G7kpv/1bVxVX1Cbq75FsmWW+mCqvqr3R3kl9PF3DPen1t+Yvohkg8eym3Y5dW78+A9YCdp11iSXsCdwcuafXMYdBlv6r+RHcx5+X/v717j7arLO89/v1J1EJRsKitIhq0iHIJ4XKOVlGhoqJB0UoLkdriqYceL1WxKFQ4lVoVKiAVET3oEdQhSgdIRRRvFSreRQ0JF6EiqEGtRCuKUCzhOX/Mdx9nVtbeeyUk2ezJ9zPGHnuv9zafeRHzzPm+c00zxrqexyvp3gFAVf203bw4A9hygn57TlN31Zi6PVq5JEmSJG0UEyWkSXZMskOvaDHwvY0T0traE93jGf+ytXHOAvYDHtQ+7wdcXlXbVdXCqnoEXeL73JF+7wXeUFUr1iPG6+imY/9dW/9Pkh2SHDhTv3ZT4I+BRS22hcCBtJkGSZZMjUe3ZGE18PMJwzoZOKqqfjrp9ka8CVjrWxKm2Y/7A3sDD++N+7Jpxp3JUmD/3hh7sva6fIC30r3Ybtx7A9b1PL4FOKa9jHDKFhP0OxF4XZJHQ3dsk7y6N+Y/TN2QaTeoDgNOHzeQJEmSJG0Ikz513hJ4X7qvO1sO7AQc1+r6a/I/2+tzbJKVUz+tbIt+WS8h6q/JX5be1971vIvuTfCj0/DX0p5in0r30jroEsfzR5qdRzf9vd9vZVW9bZphR9e/PwGgrUuf8mLg94DvJFkBvJu118KPejJwY1X13+b/eWCnJA+hWxd/TdvOB+iWR0yt+x8bU29/rqyq963j9tboz+xr1af8EfC5qZczNh8FnpP2dXyzaef94cBXejFcD/wi7asIe+Wr6M7pWmPPch7X0m4GvBJ4f7qvu/si3bcinN1rduLIsb5PWxrwKuBDSa4GrqB7SSVVdQHdzYYvJfk23bXwp1X1o0njkiRJkqR1lSqXCEt3Jy855vi6aPWiuQ5DkiRJ88QNJyyZvZGGJtNVrM/6cUmSJEmSdDe0Pt+DPqeSvIi1vz7ti1U19vvHJUmSJEm6p5h3SX5VnQmcOddxSJIkSZJ0d+N0fUmSJEmSBsIkX5IkSZKkgTDJlyRJkiRpIEzyJUmSJEkaCJN8SZIkSZIGwiRfkiRJkqSBMMmXJEmSJGkgTPIlSZIkSRoIk3xJkiRJkgbCJF+SJEmSpIFYMNcBSFrTrttuxTtfumSuw5AkSZI0D/kkX5IkSZKkgTDJlyRJkiRpIEzyJUmSJEkaCJN8SZIkSZIGwiRfkiRJkqSBMMmXJEmSJGkgTPIlSZIkSRoIk3xJkiRJkgbCJF+SJEmSpIFYMNcBSFrTihtvZuHRH5/rMCRJknQX3XDCkrkOQfdAPsmXJEmSJGkgTPIlSZIkSRoIk3xJkiRJkgbCJF+SJEmSpIEwyZckSZIkaSBM8iVJkiRJGgiTfEmSJEmSBsIkX5IkSZKkgTDJlyRJkiRpIEzyJUmSJEkaCJN8SZIkSZIGwiRfkiRJkqSBMMmXJEmSJGkgTPIlSZIkSRoIk3xJkiRJkgbCJF+SJEmSpIEwyZckSZIkaSDmPMlPckySK5MsT7IsyeOSXJLkmvZ5WZJzW9vjkhw5ZoxbxpQdl+TG3hjLkmydZJ8kNyf5VpJvJzlplvgOS3JnkkW9siuSLOx93j1JJXnGSN9K8oHe5wVJbkpyYW/sm0Zi3GlMDFsm+T9JrmvH6vNJHtfqVrd+VyT5WJKte/12TvK5JNcm+bck/ztJWt3vJrkwyeVJrkryiVZ+rySntvFWJPl6ku1b3Q1JHjjNcToiyX8m2apXtk87Bs/ulV2YZJ/299R5Xt7OxWkj8U97/JK8qHfMft1iXZbkhNZ2/yRfa+MuS3JOkoe3urOSXN/r/6XeNp6Z5LIkV49eH0kOb2XfbmPv3aub2pfL2zFb3Ku7ocV3eZJPJ/m9ccdQkiRJku6qOU3yk/wBcACwR1UtAvYDftCqD62qxe3noPXcxCm9MRZX1c9b+aVVtTuwO3BAkifOMs5K4JgZ6pcCX2i/+34FDsUY/wAAIABJREFU7JJk8/b5acCNI23OGYnxqjHjvwf4GbBDVe0MHAZMJdu3tX67tDYvA2jbvAA4oaoeDewGPAF4aev3BuAzVbVbVe0EHN3KDwYeCiyqql2B5wFTx20mS4Gvt/Z9sx27Q9u5XwTcDny0Vzft8auqM6eOGfBDYN/2+egkuwBvB/68qh7T2nwQWNgb+zW9Y/4EgNbvNOBPq+qxwC7Ad1vdAcBfAntX1WOA/wWcPZKwH1pVuwGnAyeO7Oe+re4y4HUzHA9JkiRJWm9z/ST/IcCqqrodoKpWVdUPN9XGq+o2YBmw7SxNLwR2TrLjaEV7Mn4QXeL99CS/NdLkImBJ+3sp8KF1iTHJo4DHAcdW1Z0t7u9W1cfHNP8yv9mXFwBfrKpPtz63Ai/nN8n8Q+gScFr98l75j3rbWllV/zFBjFsCx7L2jY7LgZuTPG2mMarq18BrgYcn2a1XtT7H7yjgzVV1dW/8C6rq87P0ey3wpqr6dutzR1Wd3hvzNVW1qtV9E3gf7abKiP55GPV54PdHC9ssgcuSXLb61ptnCVOSJEmSxpvrJP/TwHZtOvnpSZ7Sq/tgbzr16FPRSR3RG+Pi0cokDwB2oEu8ZnIn8BbGP4F9InB9VV0HXAI8a6T+w8AhLflfBHx1pP7gken6m4/U7wwsq6rVMwWYZDPgqXRP76f6faPfpsW4ZZL7A+8A/m+Si9MtmXhoa/ZPwLNbLCcn2X2m7TZTyfelwI5JHjxS/0a6GwAzavt4OfCYXvFsx2+cnYFvztLmxN4x/2Ar24WRYzYy5mjdZa181P7AP08zzgHAitHCqjqjqvaqqr0222KrMd0kSZIkaXZzmuRX1S3AnsDhwE3AOUkOa9X96fqvWc9N9Kfr79srf1KS5cCPgQur6scTjHU28Pip9ek9S+kSUdrvNZ5ktyfkC1v5J8aMOzpd/7YJYunbPMky4KfA7wCfaeUBapo+VVWfAh4JvJsuqf5WkgdV1UpgR+Bv6G5u/EuSp84SwyHAh9vT/48AfzyysUsBkjxpgv3JSN/Zjt/MgyXbtET+2qz5Pof+dP1D13XcXqz9Y/zBJCvpnvq/faTtxe083R84fj23J0mSJEkzmusn+VTV6qq6pKpeTzed/PmbYLOXtnXguwIv6b8kbTpVdQdwMl0CB/z/p+fPB/42yQ10id0zk9xvpPsFwEms41T95kpgtyTTnavb2przRwD34TfTx68E9uo3TPJI4Jaq+mXbp59V1dlV9UK69fRPbuW3V9VF7ebKm4HnThdcuhcS7gB8ph2DQ1h7yj7Am5h5bf7U8dwVuHqkal2P35XAHgBV9dN2fM6gW1IwW789p6m7akzdHq18yqHA9nQ3hN4x0nbqnQF/1ns3hCRJkiRtUHP94r0dk+zQK1oMfG9Tbb+qrqV7qnrUbG2bs+heDvig9nk/4PKq2q6qFlbVI4DzWDspfi/whqpaa5r2BDFeRzct/O/a+n+S7JDkwJF2NwOvAI5Mcm+6F83tnWS/1mdz4FS6ZQck+cMkW7S/7wc8Cvh+kj2mpu63GwuLmPmcLAWOa/u/sKoeCmyb5BEj8X0aeADdCwDX0mI+HvhB7/0AU9b1+L0FOCbJY3tlW0zQ70TgdUke3WK6V5JX98b8hyTbtLrFdO9hOL0/QFX9F93ShMePbF+SJEmSNrq5fpK/JfC+dF/hthzYCTiu1fXX5H+21+fYJCunflrZFv2yXmLWX5O/LL2vvet5F/DkMdPw19JeDncqMLXmfClw/kiz8+heetfvt7Kq3jbNsKNr8qfe9L6s1+bFwO8B30mygm6K/VovKKyqb9GtaT+kTfs/kO54XUO3DvzrdG+Ph+6p9GXtuH8ZeE9Vfb3t28eSXAEsB+7o9QFY3jvOb6V7cj96DM5v5aPeBDxspOyDLYYrgN9uMY/u10zHby3tZsArgfen+7q7LwKPpXvCPuXEkeN+n3Zz4VXAh5Jc3WJ6SBvzArqbDV9K8m26c/CnVfWjMdu/jW7Wx1pf9yhJkiRJG1Oqplu2LWkuvOSY4+ui1YvmOgxJkiTdRTecsGT2RtL6yXQVc/0kX5IkSZIkbSAL5jqAu4skL6Kb4t33xaoa9z3okiRJkiTd7ZjkN1V1JnDmXMchSZIkSdL6crq+JEmSJEkDYZIvSZIkSdJAmORLkiRJkjQQJvmSJEmSJA2ESb4kSZIkSQNhki9JkiRJ0kCY5EuSJEmSNBAm+ZIkSZIkDYRJviRJkiRJA7FgrgOQtKZdt92Kd750yVyHIUmSJGke8km+JEmSJEkDYZIvSZIkSdJAmORLkiRJkjQQJvmSJEmSJA2ESb4kSZIkSQNhki9JkiRJ0kCY5EuSJEmSNBAm+ZIkSZIkDYRJviRJkiRJA7FgrgOQtKYVN97MwqM/PtdhSJIkaYwbTlgy1yFIM/JJviRJkiRJA2GSL0mSJEnSQJjkS5IkSZI0ECb5kiRJkiQNhEm+JEmSJEkDYZIvSZIkSdJAmORLkiRJkjQQJvmSJEmSJA2ESb4kSZIkSQNhki9JkiRJ0kCY5EuSJEmSNBAm+ZIkSZIkDYRJviRJkiRJA2GSL0mSJEnSQJjkS5IkSZI0ECb5kiRJkiQNhEm+JEmSJEkDsdGS/CTHJLkyyfIky5I8rpVfkuSaVrYsybmt/LgkR44Z55YxZcclubE3xrIkWyfZJ8nNSb6V5NtJTpolxsOS3JlkUa/siiQLe593T1JJnjHSt5J8oPd5QZKbklzYG/umkRh3GhPD/0iyoh2nK5IcmOQdrf1VSW7r9T+o9floki+POSa3JnnwuGOXZHUb48oklyd5dZJ7jYwx3bg39uJZ2qs7K8n1bbxrk7w/ybbTHOvR8z61L6uz5jE6eqT98nYuT0uy9Zj9ma7f5Um+nmTxSBxva/tzr17ZYUlOGxf3NPFfneTwkfqZrpOTe5+PTHLcTNuSJEmSpPW1YGMMmuQPgAOAParq9iQPBO7Ta3JoVV12FzdzSlWtkcQnAbi0qg5IsjnwrSTnV9UXZxhnJXAMcPA09UuBL7Tfn+qV/wrYJcnmVXUb8DTgxpG+51TVy6fbcJKHtW3vUVU3J9kSeFBVfbTVLwQurKrFvT5bA3sAtyTZvqqu7w25Cvhr4Kgxm7ttapx2I+BsYCvg9ROMe0pVnZRkB+AbSc6tqv9qda+pqnPTHfxXARcn2aWqfj0mhnHn/bb+/o1rn+Q+wPHAR4GnrEO/FwEn0p0bWmL/POAHwJOBS6bpP52pcX8HuC7JWb39nO46uR34oyTHV9WqddyeJEmSJK2TjfUk/yHAqqq6HaCqVlXVDzfStsZqifcyYOyT5Z4LgZ2T7Dha0RLXg4DDgKcn+a2RJhcBS9rfS4EPrWOYDwZ+CdzSYr5lJLke5/nAx4APA4eM1L0XOLglodOqqp8AhwMvb/s427hT/f4NuBV4wJi6qqpTgB8Dz5xlH9ZJS6RfCzw8yW7r0PXLrHn+9wWuAN5Jd77W15Z0N3lWw6zXyR3AGcARMw2Y5PAklyW5bPWtN9+F0CRJkiTdk22sJP/TwHZtCvfpSZ4yUv/B3jTrE9dzG0f0xrh4tDLJA4AdgM/PMs6dwFuA142peyJwfVVdR/fU91kj9R8GDmlJ3SLgqyP1B49MKd98pP5y4N+B65OcmeTZs8QKv7mZ8CHWTlRvoUv0XznbIFX1XbrzPzW9f6ZxAUiyB/Bv7SbBdL4JPGaauv5536aVbT5yjMbOqKiq1XTH6zHr0G9/4J97n6f28XzggCT3nmE/pot/OXAN8PctJpj9OnkHcGiSraYbuKrOqKq9qmqvzbaYtpkkSZIkzWijTNevqluS7Ak8ie7p6TlJjq6qs1qTjTJdv3lSS8R2BE6oqh9PMNbZwDFJth8pX0qXyNN+vxD4yFRlVS1vU+qXAp8YM+6M0/WranWS/YH/BjwVOCXJnlV13Lj2SX4X+H3gC1VVSe5oU+Ov6DU7FVjWXwc+g0w47hFJ/ifwSLrEedYxp7Gu0/VnGnumfh9M8tvAZnRLEGhT/p8FHFFVv0zyVeDpwMcn3Db8Zrr+g4AvJflkVX2P2a+TXyR5P/AK4LZ12J4kSZIkrZON9uK9qlpdVZdU1euBl9NNB98ULq2qRcCuwEtGX7w2TlXdAZxMby17ks3oYv7bJDcAbweemeR+I90vAE5i3afqT227quprVXU83TT5mY7TwXRT5a9vMS1kZGp9Vf2c7qbFS2fabpJH0k03/8kE455SVTu2du8fs2yhb3fg6pm2vT7a+dh1wrEPBbanOw7vaGX7072DYEXbx71Zzyn7VXUT3YyFx63DdfKPwF8Av70+25QkSZKkSWyUJD/Jju0lbVMWA9/bGNuaTlVdS/eytnEvoRvnLGA/4EHt837A5VW1XVUtrKpHAOcBzx3p917gDVW1Yl1jTPLQNgV+ymzHaSmwf4tnIbAn49fPvxX4S6aZqdGeRL8LOK2qatJxq+ojwGXAn48ZM0leQfc+hk/OsA/rrE2rPx74QVUtn6RPezHgscDjkzyWbh9f3NvH7enWz2+xHvFsQXcz4zomvE6q6mfAP9El+pIkSZK0UWysJ/lbAu9L95Vry4GdgON69f212Z/tlR+bZOXUTyvbol+W5NWtvL8mf1l6X3vX8y7gyWOm4a+lvdztVNZco37+SLPzgBeM9FtZVW+bZtjRNflPAEiyrNXfGzgp3VfELaN7Uj52PX3bv4cDX+lt+3rgF2lfT9grX9Viv2+veGoN+5XAZ+nem/B36zJu8wag//V7Jya5HLiWbtnBvtO8WX86o2vrT+jVTa2Bv4LuCfiBE/ab2o/b6GZovBZ4Br2p+VX1K7q34U+9B+GwkevsYWNi/WA7T98AzqqqbzDhddKcDDxw2iMhSZIkSXdRuge5ku4uXnLM8XXR6kVzHYYkSZLGuOGEJbM3kja+ad+FttHW5EuSJEmSpE1ro7xd/+4myYtYexr8F6vqZXMRjyRJkiRJG8M9IsmvqjOBM+c6DkmSJEmSNian60uSJEmSNBAm+ZIkSZIkDYRJviRJkiRJA2GSL0mSJEnSQJjkS5IkSZI0ECb5kiRJkiQNhEm+JEmSJEkDYZIvSZIkSdJAmORLkiRJkjQQJvmSJEmSJA3EgrkOQNKadt12K9750iVzHYYkSZKkecgn+ZIkSZIkDYRJviRJkiRJA2GSL0mSJEnSQJjkS5IkSZI0ECb5kiRJkiQNhEm+JEmSJEkDYZIvSZIkSdJAmORLkiRJkjQQJvmSJEmSJA3EgrkOQNKaVtx4MwuP/vhchyFJknSPdsMJS+Y6BGm9+CRfkiRJkqSBMMmXJEmSJGkgTPIlSZIkSRoIk3xJkiRJkgbCJF+SJEmSpIEwyZckSZIkaSBM8iVJkiRJGgiTfEmSJEmSBsIkX5IkSZKkgTDJlyRJkiRpIEzyJUmSJEkaCJN8SZIkSZIGwiRfkiRJkqSBMMmXJEmSJGkgTPIlSZIkSRoIk3xJkiRJkgZiXif5SY5JcmWS5UmWJXlckkuSXNM+L0tybmt7XJIjx4xxy5iy45Lc2BtjWZKtk+yT5OYk30ry7SQnzRLfYUnuTLKoV3ZFkoW9z7snqSTPGOlbST7Q+7wgyU1JLuyNfdNIjDuNieGGJCta/YokB47UP69t6zG9soWt7K96ZaclOaz9fVaSg9rfv9OOx4vSOTbJvyW5NsnFSXaeJpZlSZ4wQQx/3yt7YJL/SnLa+p6ndtxO633+s3ZOrkxyVf8aacd8VZLj1zq5v2lzVovhvr0Ybxhpc0SS/0yy1XTjSJIkSdKGMG+T/CR/ABwA7FFVi4D9gB+06kOranH7OWg9N3FKb4zFVfXzVn5pVe0O7A4ckOSJs4yzEjhmhvqlwBfa775fAbsk2bx9fhpw40ibc0ZivGqabexbVYuBg4BTp9n+ISPlPwFemeQ+0wXektZPAWdU1ZnAy4AnALtV1aOB44ELkvzWaCzt50uzxPBdunM85Y+BK0farPd5SvJM4FXA06tqZ2AP4OZek6cD1wB/kiTTHQdgNfA/ZqhfCnwdeN4MbSRJkiTpLpu3ST7wEGBVVd0OUFWrquqHm2rjVXUbsAzYdpamFwI7J9lxtKIljgcBhwFPH0mGAS4ClrS/lwIfuisxA/cH/qO3/S2BJwJ/wdoJ9k3AvwB/Ps1YW7b4zq6qd7ayo4C/qqpbAarq08CXgEOnC2iWGG4Drk6yV/t8MPBPM+zfWmY5T38DHDl13VTVf1bVu3v1S4G3Ad8HHj/DZv4ROCLJgtGKJI+iO1bHsvaNHEmSJEnaoOZzkv9pYLs2Lfz0JE/p1X2wN337xPUc/4jeGBePViZ5ALAD8PlZxrkTeAvwujF1TwSur6rrgEuAZ43Ufxg4pCX/i4CvjtQfPDJVfXPGuzjJFcC/0iWbU54LfLKqrgV+lmSPkX4nAH+dZLMxY74V+EJVnQKQ5P7Ab7d96bsM2Ln3+eIW69S+zBbD1DF4GN0T89EbOXflPO0CfGNMOe1YPpXuJs2HmDlB/z7dTIQXjqmbujlzKbBjkgdPs73Dk1yW5LLVt948rokkSZIkzWreJvlVdQuwJ3A43VPnc6bWjLPmdP3XrOcm+tPA9+2VPynJcuDHwIVV9eMJxjobeHyS7UfKl9IlsbTfaySSVbUcWNjKPzFm3NHp+rdNs/19q2oXYFfgtPb0fJLtXw98DXjBmDE/Bxw4XdLaE6BGYllcVY+bJAbgk3RLFZYC54wZf0Oep74DgIvbrITzgOdNc7NjypuB17D2/6YOAT5cVXcCH6FbcrCWqjqjqvaqqr0228Kl+5IkSZLWz1rTi+eTqlpN9wT8kiQrmH5q+YZ0aVUdkOTRwBeSnF9Vy2aJ844kJ9NNZwegJYzPB56T5Bi6ZHibJPerql/2ul8AnATsA2xzVwKvquuS/DuwU5LrgD+kW/dfwGZAJXntSLc3A+ey9pPwD9M9vf5Ekn2r6hdJfpXkkVX13V67PehmEKwlyTazxVBVv07yDeCv6WYEPHvC3Z3kPF1Jd6Poc2P6LwWe2HuJ3jbAvsBnx22sqr6TZBnwJ739W0Q3i+AzbUn/fejeM/COCfdBkiRJktbJvH2Sn2THJDv0ihYD39tU22/Ty4+nl7jP4iy6lwM+qH3eD7i8qrarqoVV9Qi6J8bPHen3XuANVbXirsbcnrpvT3ecDgLeX1WPaNvfDrge2Lvfp6q+DVzFmi/Am6r7R7p1++e3F/SdCJw6tWwgyX5tvLOnCWmiGICTgaOq6qfrus+znKfjgbck+b0W732TvKItPdgbeHiLayHdSwWXtnbHJxn3Er03Af1vcFgKHDc1RlU9FNg2ySPWdT8kSZIkaRLzNsmne5nZ+9rXni0HdgKOa3X9Nfn9J6/HJlk59dPKtuiXJXl1K++v9V6W3tfe9bwLePKYafhrqapf073Zfmp6+1Lg/JFm5zEyNb6qVlbV26YZdnRN/tRX0o0+sb64lV0MHF1V/z7p9ps3AQ+bZr+OovtWgw/QPaH+OrAiyTXA/wYOnGEZwaTH4Mqqet80Y6z3eaqqT7SYP5vkSrr1+QuAPwI+N/VSx+ajdLMu7ku37GGt6f9VdSXwzV7RIWP273zWfsGgJEmSJG0QqarZW0n6/5J8qqqesbHGf8kxx9dFqxdtrOElSZI0gRtOWDJ7I2nuTPsV3/P5Sb40JzZmgi9JkiRJd8W8fvHe3UWSFwGvHCn+YlW9bC7ikSRJkiTdM5nkbwBVdSZw5lzHIUmSJEm6Z3O6viRJkiRJA2GSL0mSJEnSQJjkS5IkSZI0ECb5kiRJkiQNhEm+JEmSJEkDYZIvSZIkSdJAmORLkiRJkjQQJvmSJEmSJA2ESb4kSZIkSQNhki9JkiRJ0kAsmOsAJK1p12234p0vXTLXYUiSJEmah3ySL0mSJEnSQJjkS5IkSZI0ECb5kiRJkiQNhEm+JEmSJEkDYZIvSZIkSdJAmORLkiRJkjQQJvmSJEmSJA2ESb4kSZIkSQNhki9JkiRJ0kAsmOsAJK1pxY03s/Doj891GJIkSfdoN5ywZK5DkNaLT/IlSZIkSRoIk3xJkiRJkgbCJF+SJEmSpIEwyZckSZIkaSBM8iVJkiRJGgiTfEmSJEmSBsIkX5IkSZKkgTDJlyRJkiRpIEzyJUmSJEkaCJN8SZIkSZIGwiRfkiRJkqSBMMmXJEmSJGkgTPIlSZIkSRoIk3xJkiRJkgbCJF+SJEmSpIEwyZckSZIkaSBM8iVJkiRJGohNmuQnOSbJlUmWJ1mW5HFJLklyTfu8LMm5re1xSY4cM8YtY8qOS3Jjb4xlSbZOsk+Sm5N8K8m3k5w0S3yHJbkzyaJe2RVJFvY+756kkjxjpG8l+UDv84IkNyW5sDf2TSMx7jQmhhuSrGj1K5IcOLrvSe6V5NQW24okX0+yfa//A3t99pmKoX1+bjv+3259n9urOyvJ9W3b30zyB6388Um+2sqvTnJcr8/+Sb7WxluW5JwkDx85DquSHN8+H9Pb/9W9v1+xPudxuuM66THqxXBlksuTvDrJvXrH7uaRsffrne+Te3Ec2eKfbf/WuqYlSZIkaUNZsKk21BLGA4A9qur2lmTdp1UfWlWX3cVNnFJVayTxSQAuraoDkmwOfCvJ+VX1xRnGWQkcAxw8Tf1S4Avt96d65b8CdkmyeVXdBjwNuHGk7zlV9fIJ9mXfqlqVZEfg08BHR+oPBh4KLKqqO5M8rG1/Rkl2A04CnlZV17ek9zNJvltVy1uz11TVuUmeDvwfYBHwPuBPquryJJsBO7bxdgHeDjynqq5uZc8BFgLfb+M9HbgG+JMkr6uqNwFvam1vqarFvfiOY/3O41rHNcnSCY/RbVMxJHkwcDawFfD6Vn9pVR0wpt/twB8lOb6qVk0VTrB/kiRJkrTRbMon+Q8BVlXV7QBVtaqqfripNt4S72XAtrM0vRDYuSXYa0iXbR4EHAY8PclvjTS5CFjS/l4KfOiuxAzcH/iPMeUPAX5UVXcCVNXKqhrXbtSRwJur6vrW73rgeOA1Y9p+Hvj99veDgR+1Pqur6qpWflQb7+qpTlV1QVV9vjfOUuBtdEn/4yeIcUbrcB7X+RhV1U+Aw4GXt3M9kzuAM4AjJgp8FkkOT3JZkstW33rzhhhSkiRJ0j3QpkzyPw1sl+TaJKcneUqv7oO9ac0nruf4R/TGuHi0MskDgB3okteZ3Am8BXjdmLonAtdX1XXAJcCzRuo/DBzSkv9FwFdH6g8emfq9+TQxXJzkCuBfgWPH1P8T8Ow2xslJdh/Tf1mSZcB7euU7A98YaXtZKx/1bGBF+/sU4Jok5yf5y97NjZ2Bb06zD7T9eyrdjZMP0SX8s1mf8zjuuM52jMaqqu/S/e/iwa3oSSNjP6rX/B3AoUm2mmTsWbZ7RlXtVVV7bbbFXR5OkiRJ0j3UJkvyq+oWYE+6J6U3AeckOaxVH1pVi9vPuKfKkzilN8a+vfInJVkO/Bi4sKp+PMFYZwOPn1rD3bOULpGn/V4jaW1T3he28k+MGfecXoyL21Ppcfatql2AXYHTkmw5sp2VdFPm/4bupsS/JHnqSP/Fbar4i3vlAWpkW6NlJ7abA4cDf9G29wZgL7obNS8APjkacJJtWhJ8bW/d+QHAxVV1K3Ae8Lw23X8m63Me1zquExyjmfSf4l86MvZ1UxVV9Qvg/cArJhxXkiRJkjaqTbYmH7qp3nRPwC9JsgL4802w2am13I8GvtDWci+bJc472kvVjpoqa8np84HnJDmGLhHcJsn9quqXve4X0K173wfY5q4EXlXXJfl3YCfgayN1t9MtD7iotXku8C+zDHklXbK+vFe2B3BV7/NrqurccbEA70zybuCmJNu08fYALq+qnwKLW4I/dVNiKfDEJDe0z9sA+wKfnSXOcdbnPK7zMUrySGA18BPgsRPE9Y90sxnOnKCtJEmSJG1Um+xJfpIdk+zQK1oMfG9Tbb+qrqVbf37UbG2bs4D9gAe1z/vRJbPbVdXCqnoE3dPp5470ey/whqpawV3UXgS3PSPHKckeSR7a/r4X3dKASY7lScDfpH1bQPv9OuDkaXt07Zb01qjvQJcE/5xuWcMxSfrJ8Batz/2BvYGHt+O1EHgZk03Zn9ak53F9jlGSBwHvAk6rqtEZD9PF8zO6pQF/MUl7SZIkSdqYNuWT/C2BtyfZmu6lZd+hmxJ+Lt2a/Kmp66uqar/297FJXjU1QFU9DNgiycreuG9tv49I8qe98tHkG7oE7sgk20+9fG46VfXrJKfSvTQOuuT0/JFm5wEvAT7Q67ey12fUwUn27n1+aVV9Kcmy/lvY6dbUrwbuDRxdVf8+Ms6DgXcnuW/7/DXgtJn2p8W2LMlRwMeS3Bv4L+C1sz0RB14InJLkVrpzd2iblbEiySuB9ye5H/BTuhfsvR74I+BzUy9abD4KvCXJfUfK+9bpPLbPax1XupcWTnKMNm/LE+7d9u0D/OaagrYmv/f5jWNmOpwMTPKtCZIkSZK0UWXCB5aSNpGXHHN8XbR60VyHIUmSdI92wwlLZm8kzZ1pvw1sU75dX5IkSZIkbUSb9MV7dxdJXgS8cqT4i1X1srmIR5IkSZKkDeEemeRX1Zn4NnRJkiRJ0sA4XV+SJEmSpIEwyZckSZIkaSBM8iVJkiRJGgiTfEmSJEmSBsIkX5IkSZKkgTDJlyRJkiRpIEzyJUmSJEkaCJN8SZIkSZIGwiRfkiRJkqSBMMmXJEmSJGkgFsx1AJLWtOu2W/HOly6Z6zAkSZIkzUM+yZckSZIkaSBM8iVJkiRJGgiTfEmSJEmSBsIkX5IkSZKkgTDJlyRJkiRpIEzyJUmSJEkaCJN8SZIkSZIGwiRfkiRJkqSBMMmXJEmSJGkgFsx1AJLWtOLGm1l49MfnOgxJkqRBu+GEJXMdgrRR+CRfkiRJkqSBMMmXJEmSJGkgTPIlSZIkSRoIk3yrs+oQAAAMCklEQVRJkiRJkgbCJF+SJEmSpIEwyZckSZIkaSBM8iVJkiRJGgiTfEmSJEmSBsIkX5IkSZKkgTDJlyRJkiRpIEzyJUmSJEkaCJN8SZIkSZIGwiRfkiRJkqSBMMmXJEmSJGkgTPIlSZIkSRoIk3xJkiRJkgbiHp/kJzkmyZVJlidZluRxSS5Jck37vCzJua3tcUmOHDPGLWPKjktyY2+MZUm2TrJPkpuTfCvJt5OcNEt8hyW5M8miXtkVSRb2Pu+epJI8Y6RvJflA7/OCJDclubA39k0jMe40JoYbkqzotXlCkoVJrmj10+5Tkt9NcmGSy5NcleQTSXbtjfWzJNe3vz87y/6sbu2uSPKxJFu38oVJbmt1lyf5UpIdR/q+rZ2Pe/XKZjy2bb8f2P7es8W5+8i4+7RYn90ruzDJPr3PD0ryX0n+cvTYSpIkSdKGdI9O8pP8AXAAsEdVLQL2A37Qqg+tqsXt56D13MQpvTEWV9XPW/mlVbU7sDtwQJInzjLOSuCYGeqXAl9ov/t+BeySZPP2+WnAjSNtzhmJ8apptrFvr82XxtRPt09vAD5TVbtV1U7A0VW1Ymos4ALgNe3zfrPsz22t3S7Az4CX9equa3W7Ae8DXjdV0RL759Gd2yePjDnbsaXdBDgXOLiqvjWmyWxj/DHwlTH7I0mSJEkb1D06yQceAqyqqtsBqmpVVf1wU228qm4DlgHbztL0QmDn0afTAEkCHAQcBjw9yW+NNLkIWNL+Xgp86K7EPJsx+/QQuiR4qn75TP0n2J8pX2b643Z/4D96n/cFrgDeydqJ9rTHtnks8M/AC6vqa9O0uRy4OcnTpqlfCvw18LAkY2NOcniSy5JctvrWm6cZRpIkSZJmdk9P8j8NbJfk2iSnJ3lKr+6DvSnlJ67n+Ef0xrh4tDLJA4AdgM/PMs6dwFvoPZ3ueSJwfVVdB1wCPGuk/sPAIS1ZXgR8daT+4JHp+psz3sWtfrT/Gsbs0zuA/5vk4nRLIx46U/8J9ockmwFPpZsFMOVRLb7rgFcDb+3VTd3cOJ9ulsG9e3UzHVuAjwIvr6ovzBL3G4Fjx8S6HfB77QbBPwEHj+tcVWdU1V5VtddmW2w1y6YkSZIkabx7dJJfVbcAewKHAzcB5yQ5rFX3p+u/Zj030Z+uv2+v/ElJlgM/Bi6sqh9PMNbZwOOTbD9SvpQukaf9XuNJdXtyvrCVf2LMuKPT9W+bZvtT0/UfN0392H2qqk8BjwTeDTwG+FaSB82wnzPtz+ZJlgE/BX4H+Eyvbmq6/qOAVwFnACS5D92Ngn+uql/Q3eR4+sg2pzu2AJ8FXtxuLEyrqi5t23vSSNUhdMn9uP2RJEmSpA3qHp3kA1TV6qq6pKpeD7wceP4m2Oyl7R0AuwIvSbJ4tg5VdQdwMnDUVFlLPJ8P/G2SG4C3A89Mcr+R7hcAJ7Fxp+pPu09V9bOqOruqXgh8nbXXxQMT7c9tbR3/I4D7sOaa/L4LetvYH9gKWNHG3Ju1b4SsdWx7Xt5+nz7NtvrexNpr85cCh7VtXwDslmSHCcaSJEmSpHV2j07yk+w4knAtBr63qbZfVdcCxzM+uRznLLqXA049Cd8PuLyqtquqhVX1COA84Lkj/d4LvKGqVtz1qGc2uk9J/jDJFu3v+wGPAr4/TfeJ9qeqbgZeARw5MvV+yt7Ade3vpcCL23gLge3p1vpvMdLnLNY8tlPubGPsmOQNbT/+e5L3j9n3TwMPAHZr7XYEfruqtu1t/3i6p/uSJEmStMHdo5N8YEvgfe2r3ZYDOwHHtbr+mvzP9vocm2Tl1E8r26JfluTVrby/Jn9Zel971/Mu4MnTTBVfQ1X9GjgVeHArWkq3zrzvPOAFI/1WVtXbphl2dE3+EwDatPj11d+nPYHL2vH9MvCeqvr6NP0m2h+A9pb7y/lNwjy1Jv9y4M10U+y3AJ4BfLzX71d0b+5/9sh4o8e2X3c7cCDwnCQvAx4OTLes4U3Aw2bZH6fsS5IkSdooUlVzHYM0r7QXMX5gtm8KWF8vOeb4umj1oo0xtCRJkpobTlgyeyPp7ivTVSzYlFFIQ3AXXsQoSZIkSRuVSf7dRJIXAa8cKf5iVU33cjlJkiRJktZgkn83UVVnAmfOdRySJEmSpPnrnv7iPUmSJEmSBsMkX5IkSZKkgTDJlyRJkiRpIEzyJUmSJEkaCJN8SZIkSZIGwiRfkiRJkqSBMMmXJEmSJGkgTPIlSZIkSRoIk3xJkiRJkgbCJF+SJEmSpIFYMNcBSFrTrttuxTtfumSuw5AkSZI0D/kkX5IkSZKkgTDJlyRJkiRpIEzyJUmSJEkaCJN8SZIkSZIGwiRfkiRJkqSBMMmXJEmSJGkgTPIlSZIkSRoIk3xJkiRJkgbCJF+SJEmSpIEwyZckSZIkaSBM8iVJkiRJGgiTfEmSJEmSBsIkX5IkSZKkgTDJlyRJkiRpIEzyJUmSJEkaCJN8SZIkSZIGwiRfkiRJkqSBMMmXJEmSJGkgTPIlSZIkSRoIk3xJkiRJkgbCJF+SJEmSpIEwyZckSZIkaSBM8iVJkiRJGgiTfEmSJEmSBsIkX5IkSZKkgTDJlyRJkiRpIEzyJUmSJEkaCJN8SZIkSZIGwiRfkiRJkqSBSFXNdQySeo466qhf3vve975mruPQcNxyyy0P3HLLLVfNdRwaDq8pbUheT9rQvKa0od1Nr6lVb3zjG/cfV2GSL93NJLmsqvaa6zg0HF5T2tC8prQheT1pQ/Oa0oY2364pp+tLkiRJkjQQJvmSJEmSJA2ESb5093PGXAegwfGa0obmNaUNyetJG5rXlDa0eXVNuSZfkiRJkqSB8Em+JEmSJEkDYZIvSZIkSdJAmORLcyTJ/kmuSfKdJEePqU+SU1v98iR7zEWcmh8muJ4ObdfR8iRfSrLbXMSp+WO2a6rX7r8lWZ3koE0Zn+afSa6pJPskWZbkyiT/uqlj1Pwywf/3bZXkY0kub9fUi+YiTs0PSd6b5CdJrpimft7829wkX5oDSTYD3gE8E9gJWJpkp5FmzwR2aD+HA+/cpEFq3pjweroeeEpVLQL+nnn2AhltWhNeU1Pt/gH41KaNUPPNJNdUkq2B04HnVNXOwB9v8kA1b0z436mXAVdV1W7APsDJSe6zSQPVfHIWsP8M9fPm3+Ym+dLc+O/Ad6rqu1X1a+DDwIEjbQ4E3l+drwBbJ3nIpg5U88Ks11NVfamq/qN9/ArwsE0co+aXSf4bBfBXwHnATzZlcJqXJrmmXgB8pKq+D1BVXleaySTXVAH3SxJgS+BnwB2bNkzNF1X1ebprZDrz5t/mJvnS3NgW+EHv88pWtq5tJFj3a+UvgIs2akSa72a9ppJsCzwPeNcmjEvz1yT/nXo08IAklyT5RpI/22TRaT6a5Jo6DXgs8ENgBfDKqrpz04SnAZo3/zZfMNcBSPdQGVM2+n2Wk7SRYB2ulST70iX5e2/UiDTfTXJN/SNwVFWt7h6SSTOa5JpaAOwJPBXYHPhykq9U1bUbOzjNS5NcU88AlgF/CDwK+EySS6vqFxs7OA3SvPm3uUm+NDdWAtv1Pj+M7i7zuraRYMJrJcki4D3AM6vqp5soNs1Pk1xTewEfbgn+A4FnJbmjqv5504SoeWbS/99bVVW/An6V5PPAboBJvsaZ5Jp6EXBCVRXwnSTXA48BvrZpQtTAzJt/mztdX5obXwd2SLJ9ewHMIcAFI20uAP6svcnz8cDNVfWjTR2o5oVZr6ckDwc+ArzQp2KawKzXVFVtX1ULq2ohcC7wUhN8zWCS/9/7KPCkJAuSbAE8Drh6E8ep+WOSa+r7dDNDSPK7wI7AdzdplBqSefNvc5/kS3Ogqu5I8nK6N1JvBry3qq5M8r9a/buATwDPAr4D3Ep3N1pay4TX098C2wCntyevd1TVXnMVs+7eJrympIlNck1V1dVJPgksB+4E3lNVY7/KSprwv1N/D5yVZAXdVOujqmrVnAWtu7UkH6L7FoYHJlkJvB64N8y/f5unm70iSZIkSZLmO6frS5IkSZI0ECb5kiRJkiQNhEm+JEmSJEkDYZIvSZIkSdJAmORLkiRJkjQQJvmSJEmSJA2ESb4kSZIkSQPx/wAEjJuEbIaSLwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1008x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "dl.varimp_plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see in the image above, the variable importance plot changes a lot compared to the XGBoost. For our Deep Learning model, we see most of the SELLER_NAME variables, and the most important variable for our Deep Learning model \"Other sellers\" is not even included in the top ten for the XGBoost. Also, the top 10 most important variables for the Deep Learning model have almost the same significance, as opposed to the other model.\n",
    "\n",
    "Save the model performance on the validation set, as we will use it later on for comparison purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "default_dl_per = dl.model_performance(valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "ModelMetricsRegression: deeplearning\n",
      "** Reported on test data. **\n",
      "\n",
      "MSE: 0.19590097755469604\n",
      "RMSE: 0.4426070238424782\n",
      "MAE: 0.31819382877238667\n",
      "RMSLE: 0.052454661814392047\n",
      "Mean Residual Deviance: 0.19590097755469604\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "default_dl_per"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Tune the XGBoost Model with H2O GridSearch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will first try to find the max_depth for our XGBoost, as this is one of the most important parameters for an XGBoost model.\n",
    "\n",
    "max_depth defines the number of nodes along the longest path from the start of the tree to the farthest leaf node. By default, the value is 6. We could do a random search along with the other parameters, but when max_depth is large, the model takes longer to train; therefore, in order to do a more efficient random search with the other parameters, we will first find the best value max_depth, and we will use 100 trees with early stopping to tune our hyper-parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb = H2OXGBoostEstimator(model_id='xgb', ntrees=100,\n",
    "                          stopping_rounds=5, #default\n",
    "                          stopping_tolerance=1e-4, #default\n",
    "                          stopping_metric = \"rmse\", #default\n",
    "                          seed=42\n",
    "    )\n",
    "\n",
    "hyper_params = {'max_depth' : [5,7,9,10,12,13,15,20]\n",
    "               }\n",
    "\n",
    "grid_id = 'depth_grid'\n",
    "\n",
    "search_criteria = { \"strategy\":\"Cartesian\"}\n",
    "\n",
    "xgb_grid = H2OGridSearch(model=xgb, \n",
    "                         hyper_params=hyper_params,\n",
    "                         grid_id=grid_id,\n",
    "                         search_criteria=search_criteria\n",
    "                         )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time \n",
    "xgb_grid.train(x=x, y=y, training_frame=train, validation_frame = valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can get the models trained by the GridSearch with the .get_grid() function, and print it in a nice table format with **.sorted_metric_table()** function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_xgb = xgb_grid.get_grid(sort_by='rmse',decreasing=False)\n",
    "sorted_xgb.sorted_metric_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use max_depth equal to 9 to try to tune the next parameters. We will keep the same number of trees, at ntrees=100. We will start by searching over five parameters, and see if we get any improvement. Here is the list of the five parameters that we will tune.\n",
    "\n",
    "1. reg_alpha: Specify a value for L1 regularization. L1 regularization encourages sparsity, meaning it will make the weights at the leaves become 0. This value defaults to 0.\n",
    "\n",
    "2. reg_lambda: Specify a value for L2 regularization. L2 Regularization makes some of the weights at the leaves to be small, but not zero. This defaults to 1.\n",
    "\n",
    "3. learn_rate (alias: eta): Specify the learning rate by which to shrink the feature weights. Shrinking feature weights after each boosting step makes the boosting process more conservative and prevents overfitting. The range is 0.0 to 1.0. This value defaults to 0.3.\n",
    "\n",
    "4. distribution: Specify the distribution (i.e., the loss function). The options are AUTO, Bernoulli, multinomial, gaussian, poisson, gamma, or tweedie. Since our response is numeric, we will just include poisson, Tweedie, Gaussian, and gamma.\n",
    "\n",
    "5. booster: Specify the booster type. This can be one of the following: \"gbtree,\" \"gblinear,\" or \"dart.\" Note that \"gbtree\" and \"dart\" use a tree-based model while \"gblinear\" uses a linear function. This value defaults to \"gbtree.\"\n",
    "\n",
    "Since we have several parameters, we will be doing a random search; we will be using early stopping, and for our stopping criteria, we will set a limit of 100 models or 15 minutes. You can change these settings in the \"search_criteria_tune\" parameter for the grid search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb = H2OXGBoostEstimator(model_id='xgb_grid', max_depth=9, ntrees=100, \n",
    "                          stopping_rounds=5, #default\n",
    "                          stopping_tolerance=1e-4, #default\n",
    "                          stopping_metric = \"rmse\", #default\n",
    "                          seed=42\n",
    "                          )\n",
    "\n",
    "hyper_params = {'reg_alpha' : [x*0.01 for x in range(0, 101)],\n",
    "                'reg_lambda' : [x*0.01 for x in range(0, 101)],\n",
    "                'learn_rate' : [x*0.01 for x in range(1, 101)],\n",
    "                'distribution' : ['poisson', 'tweedie', 'gaussian','gamma'],\n",
    "                'booster' : ['gbtree','gblinear','dart']\n",
    "               }\n",
    "\n",
    "search_criteria_tune = {'strategy': \"RandomDiscrete\",\n",
    "                   'max_runtime_secs': 900, #15 min  \n",
    "                   'max_models': 100,  ## build no more than 100 models\n",
    "                   'seed' : 42 }\n",
    "\n",
    "xgb_grid = H2OGridSearch(xgb, hyper_params,\n",
    "                         grid_id = 'random_grid',\n",
    "                         search_criteria=search_criteria_tune,\n",
    "                         )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time \n",
    "xgb_grid.train(x=x, y=y, training_frame=train, validation_frame = valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The parameters for the best model in the grid search are shown in the image below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_xgb = xgb_grid.get_grid(sort_by='rmse',decreasing=False)\n",
    "sorted_xgb.sorted_metric_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the grid search that we just did allowed us to train a total of 41 models (not all the models are shown). As you can see, the top model has a gbtree booster with a gaussian distribution, and the second-best model also has a gbtree booster with a tweedie distribution. We can also see that the next two models with the dart booster yield good scores. Since the dart booster configuration yields good results, we will try to tune some extra parameters for the third model, which uses the dart booster. Dart booster is a \"method to add dropout techniques from the deep neural net community to boosted trees\" [1]. We could further tune the best model from the above grid search, and to do so, we could try tuning the sample_rate,col_sample_rate,col_sample_rate_per_tree,min_rows and gamma among others. However, we want to show you the Dart functionality of XGBoost.\n",
    "\n",
    "These are five extra parameters that we can tune when using Dart boosting, and we will try to see if we can get any improvement by changing some of those parameters\n",
    "\n",
    "1. normalize_type: specify whether the normalization method. This can be one of the following:\n",
    "\n",
    "- tree (default): New trees have the same weight as each of the dropped trees 1 / (k + learning_rate).\n",
    "- forest: New trees have the same weight as the sum of the dropped trees (1 / (1 + learning_rate).\n",
    "\n",
    "2. one_drop: specify whether to enable one drop, which causes at least one tree to always drop during the dropout. This value defaults to FALSE.\n",
    "\n",
    "3. rate_drop: specify a float value from 0 to 1 for the rate at which to drop previous trees during dropout. This value defaults to 0.0.\n",
    "\n",
    "4. sample_type: specify whether the sampling type should be one of the following:\n",
    "\n",
    "- uniform (default): Dropped trees are selected uniformly.\n",
    "- weighted: Dropped trees are selected in proportion to weight.\n",
    "\n",
    "5. skip_drop: specify a float value from 0 to 1 for the skip drop. This determines the probability of skipping the dropout procedure during a boosting iteration. If a dropout is skipped, new trees are added in the same manner as \"gbtree.\" Note that non-zero skip_drop has a higher priority than rate_drop or one_drop. This value defaults to 0.0.\n",
    "\n",
    "We will update the parameters that we found in our previous grid search, and to save some time, we will set normalize_type='forest',one_drop=False, and sample_type='uniform'. We will do a quick search for rate_drop and skip_drop. Based on results from previous experiments, combinations of lowrate_drop with highskip_drop yielded slightly better results, so we will do that for this grid search. Because this is the third grid search you can skip this step, and use the values that we find."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb = H2OXGBoostEstimator(model_id='xgb', \n",
    "                          max_depth=9, \n",
    "                          ntrees=100, \n",
    "                          distribution='gamma',\n",
    "                          reg_alpha= 0.4,\n",
    "                          reg_lambda= 0.1,\n",
    "                          learn_rate= 0.33,\n",
    "                          stopping_rounds=5, #default\n",
    "                          stopping_tolerance=1e-4, #default\n",
    "                          stopping_metric = \"rmse\", #default\n",
    "                          seed=42,\n",
    "                          booster = 'dart',\n",
    "                          normalize_type='forest',\n",
    "                          one_drop=False,\n",
    "                          sample_type='uniform'\n",
    "                          )\n",
    "\n",
    "hyper_params = { 'rate_drop' : [x*0.025 for x in range(0, 21)], #start from 0 to 0.5 in increments of 0.025\n",
    "                 'skip_drop' : [x*0.025 for x in range(20, 41)] #start from 0.5 to 1 in increments of 0.025\n",
    "               }\n",
    "\n",
    "search_criteria_tune = {'strategy': \"RandomDiscrete\",\n",
    "                   'max_runtime_secs': 600, #10 min  \n",
    "                   'max_models': 100,  # build no more than 100 models\n",
    "                   'seed' : 42\n",
    "                       }\n",
    "\n",
    "xgb_grid = H2OGridSearch(xgb, hyper_params,\n",
    "                         grid_id = 'dart_booster_grid',\n",
    "                         search_criteria=search_criteria_tune,\n",
    "                         )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time \n",
    "xgb_grid.train(x=x, y=y, training_frame=train, validation_frame = valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's get the models from the grid search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_xgb = xgb_grid.get_grid(sort_by='rmse',decreasing=False)\n",
    "sorted_xgb.sorted_metric_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The score slightly improved from what we obtained in the first random grid search; keep in mind that we are just exploring a small set of models; if we were trying to find the best model, we would have to add a wider range for each parameter and run the grid search for a longer period of time. We could also do a local search with the values we found and see if we could get better results. Let's do a quick random grid search for our XGBoost in Flow.\n",
    "\n",
    "Start by building an XGBoost model the same way we did in Task 4. Find the training set (the frame with 350k rows and 27 columns), assign a model id such flow-xgb-grid so that it is easy to find the models later on; for now, you can set\n",
    "nfolds to 3. Choose ORIGINAL_INTEREST_RATE for the response_column, select the same eight columns as before to be ignored.\n",
    "\n",
    "- FIRST_PAYMENT_DATE,\n",
    "- MATURITY_DATE,\n",
    "- MORTGAGE_INSURANCE_PERCENTAGE,\n",
    "- PREPAYMENT_PENALTY_MORTGAGE_FLAG,\n",
    "- PRODUCT_TYPE,\n",
    "- LOAN_SEQUENCE_NUMBER,\n",
    "- PREPAID,\n",
    "- DELINQUENT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's retrieve the best model from the grid search and then compare the results from the default model to the tuned model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_xgb_model = xgb_grid.models[0] \n",
    "best_xgb_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please note that you can also retrieve any of your models by changing the number inside the brackets. Now save the model performance on the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuned_xgb_per = best_xgb_model.model_performance(valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Check both RMSE and MAE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Default XGB RMSE: %.4f \\nTuned XGB RMSE:%.4f\" % (default_xgb_per.rmse(), tuned_xgb_per.rmse()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Default XGB MAE: %.4f \\nTuned XGB MAE:%.4f\" % (default_xgb_per.mae(), tuned_xgb_per.mae()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both RMSE and MAE slightly improved with the tuning that we did. As we mentioned before, we only let each random search run for 15 minutes. To see if we can obtain a much better model, we would have to let it run for much longer.\n",
    "\n",
    "One more thing that we could do is build an XGBoost model with all the parameters that we found, and increase the number of trees, similar to what we did in the first tutorial.\n",
    "\n",
    "Maybe for an interest rate decision, an MAE of 0.30 is good enough as it tells us that, on average, the model would predict a very close interest rate to what someone with \"x\" characteristics would get. However, for some companies that MAE might be too high, so it all depends on the application to decide whether or not our MAE results are satisfactory. We will see if the Deep Learning model can yield a lower RMSE and MAE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Tune the Deeplearning Model with H2O GridSearch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will try to tune our Deep Learning model and see if we can improve the scores from the default model.\n",
    "\n",
    "Two of the most important parameters of a Deep Learning model are the number of neurons in the hidden layers and the number of epochs. The parameter for both hidden neurons and layers is \"hidden\", and we can specify the hidden layer sizes. For example, to define a two-hidden-layer model, with 200 neurons in each layer, we would define it as [200,200]. If you wanted to have three hidden layers with 300 neurons in the first layer, 200 in the second one, and 100 in the third one, we would do it as follows [300,200,100]. The epochs allows us to specify the number of times to iterate (stream) the dataset, and this value can be a fraction. We will try to find a good size for our hidden layer, and we will take care of the number of epochs with early stopping.\n",
    "\n",
    "Since there are so many combinations for the size of hidden layers, the easiest thing to do is to just do a random search and use one of the models that you find. In this tutorial, we will try to find a good size for the hidden layer taking into consideration the time it takes to train, and then we will try to tune that model. We will explore models with up to three hidden layers for you to see, but for the purpose of this tutorial, we will not try to tune a complex model.\n",
    "\n",
    "With the code shown below, you can do a random search to explore several sizes of hidden layers. The way we chose the number of neurons is selecting multiples of 165, which is the size of our input layer (although we do not have 165 predictors, we have predictors that have multiple categories, and they all add up to 165, and thus the number of neurons) and we also added two random sizes, 100 and 200, just to see how they perform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl = H2ODeepLearningEstimator(seed=623, model_id='DL',\n",
    "                              nfolds= 0,\n",
    "                              keep_cross_validation_predictions = False,\n",
    "                              stopping_metric='RMSE',\n",
    "                              epochs=10\n",
    "                              )\n",
    "\n",
    "hyper_params = {'hidden' : [[165], [330], [495], [100], [200], \n",
    "                            [165, 33], [165, 165], [200, 165], [330, 330], \n",
    "                            [165, 330, 165]]             \n",
    "               }\n",
    "\n",
    "search_criteria_tune = {'strategy': \"Cartesian\",\n",
    "                       }\n",
    "\n",
    "dl_grid = H2OGridSearch(model=dl, \n",
    "                        hyper_params=hyper_params,\n",
    "                         grid_id = 'hidden_layer_grid',\n",
    "                         search_criteria=search_criteria_tune\n",
    "                         )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 0 ns\n",
      "deeplearning Grid Build progress: |███████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "%time \n",
    "dl_grid.train(x=x, y=y, training_frame=train, validation_frame = valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>hidden</th>\n",
       "      <th>model_ids</th>\n",
       "      <th>rmse</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td>[200, 165]</td>\n",
       "      <td>hidden_layer_grid_model_8</td>\n",
       "      <td>0.4341071484294671</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td>[165]</td>\n",
       "      <td>hidden_layer_grid_model_1</td>\n",
       "      <td>0.43439024000085696</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td>[165, 33]</td>\n",
       "      <td>hidden_layer_grid_model_6</td>\n",
       "      <td>0.43507784102688246</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td></td>\n",
       "      <td>[165, 330, 165]</td>\n",
       "      <td>hidden_layer_grid_model_10</td>\n",
       "      <td>0.43740278087118734</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td></td>\n",
       "      <td>[165, 165]</td>\n",
       "      <td>hidden_layer_grid_model_7</td>\n",
       "      <td>0.43750844915001474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td></td>\n",
       "      <td>[200]</td>\n",
       "      <td>hidden_layer_grid_model_5</td>\n",
       "      <td>0.4397165228189012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td></td>\n",
       "      <td>[495]</td>\n",
       "      <td>hidden_layer_grid_model_3</td>\n",
       "      <td>0.4458597302592869</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td></td>\n",
       "      <td>[330]</td>\n",
       "      <td>hidden_layer_grid_model_2</td>\n",
       "      <td>0.44999944542552284</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td></td>\n",
       "      <td>[330, 330]</td>\n",
       "      <td>hidden_layer_grid_model_9</td>\n",
       "      <td>0.45161253741728424</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td></td>\n",
       "      <td>[100]</td>\n",
       "      <td>hidden_layer_grid_model_4</td>\n",
       "      <td>0.47344688626506826</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              hidden                   model_ids                 rmse\n",
       "0         [200, 165]   hidden_layer_grid_model_8   0.4341071484294671\n",
       "1              [165]   hidden_layer_grid_model_1  0.43439024000085696\n",
       "2          [165, 33]   hidden_layer_grid_model_6  0.43507784102688246\n",
       "3    [165, 330, 165]  hidden_layer_grid_model_10  0.43740278087118734\n",
       "4         [165, 165]   hidden_layer_grid_model_7  0.43750844915001474\n",
       "5              [200]   hidden_layer_grid_model_5   0.4397165228189012\n",
       "6              [495]   hidden_layer_grid_model_3   0.4458597302592869\n",
       "7              [330]   hidden_layer_grid_model_2  0.44999944542552284\n",
       "8         [330, 330]   hidden_layer_grid_model_9  0.45161253741728424\n",
       "9              [100]   hidden_layer_grid_model_4  0.47344688626506826"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hidden_per = dl_grid.get_grid(sort_by='rmse', decreasing=False)\n",
    "hidden_per.sorted_metric_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model with 165 neurons was one of the most consistent ones, so we chose one hidden layer with 165 neurons to tune the other parameters.\n",
    "For deep learning models, the more complex the model, the longer it will take to train. For that reason, in this tutorial, we explore a fairly simple model, with only one hidden layer with 165 neurons.\n",
    "\n",
    "For the activation function, we could use the rectifier activation function, as it is one of the best amongst the three activation functions in H2O. We could try the maxout activation function, but the model usually takes much longer to build. H2O gives us the option to add dropout ratios to our activation functions, and for that reason, we will use the rectifier_with_dropout activation function. Therefore, we need to find a good dropout rate for our hidden layer.\n",
    "\n",
    "hidden_dropout_ratios improves generalization, which could help our model perform better. The range is >= 0 to <1, and the default is 0.5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl = H2ODeepLearningEstimator(\n",
    "    epochs=10,\n",
    "    hidden=[165],\n",
    "    seed=42,\n",
    "    model_id='DL',\n",
    "    activation='rectifier_with_dropout'\n",
    "    )\n",
    "\n",
    "hyper_params = {'hidden_dropout_ratios' : [[0], [0.01], [0.1], [0.15], [0.25], [0.3], [0.2],\n",
    "                                           [0.35],[0.5], [0.6], [0.8]]\n",
    "                                            \n",
    "    }\n",
    "\n",
    "search_criteria_tune = {'strategy': \"Cartesian\"\n",
    "                }\n",
    "\n",
    "dl_grid = H2OGridSearch(model=dl, \n",
    "                         hyper_params=hyper_params,\n",
    "                         grid_id = 'dropout_grid',\n",
    "                         search_criteria=search_criteria_tune,\n",
    "                         )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 0 ns\n",
      "deeplearning Grid Build progress: |███████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "%time \n",
    "dl_grid.train(x=x, y=y, training_frame=train, validation_frame = valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>hidden_dropout_ratios</th>\n",
       "      <th>model_ids</th>\n",
       "      <th>rmse</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td>[0.6]</td>\n",
       "      <td>dropout_grid_model_10</td>\n",
       "      <td>0.4356616951678924</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td>[0.5]</td>\n",
       "      <td>dropout_grid_model_9</td>\n",
       "      <td>0.43621413719586943</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td>[0.15]</td>\n",
       "      <td>dropout_grid_model_4</td>\n",
       "      <td>0.4368963938555745</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td></td>\n",
       "      <td>[0.3]</td>\n",
       "      <td>dropout_grid_model_6</td>\n",
       "      <td>0.43798861598244315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td></td>\n",
       "      <td>[0.2]</td>\n",
       "      <td>dropout_grid_model_7</td>\n",
       "      <td>0.4397494239154581</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td></td>\n",
       "      <td>[0.25]</td>\n",
       "      <td>dropout_grid_model_5</td>\n",
       "      <td>0.44038035525401653</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td></td>\n",
       "      <td>[0.8]</td>\n",
       "      <td>dropout_grid_model_11</td>\n",
       "      <td>0.4423341348749736</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td></td>\n",
       "      <td>[0.0]</td>\n",
       "      <td>dropout_grid_model_1</td>\n",
       "      <td>0.4424075548395576</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td></td>\n",
       "      <td>[0.35]</td>\n",
       "      <td>dropout_grid_model_8</td>\n",
       "      <td>0.4513440269002026</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td></td>\n",
       "      <td>[0.1]</td>\n",
       "      <td>dropout_grid_model_3</td>\n",
       "      <td>0.4537795786037289</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td></td>\n",
       "      <td>[0.01]</td>\n",
       "      <td>dropout_grid_model_2</td>\n",
       "      <td>0.46803037953434745</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     hidden_dropout_ratios              model_ids                 rmse\n",
       "0                    [0.6]  dropout_grid_model_10   0.4356616951678924\n",
       "1                    [0.5]   dropout_grid_model_9  0.43621413719586943\n",
       "2                   [0.15]   dropout_grid_model_4   0.4368963938555745\n",
       "3                    [0.3]   dropout_grid_model_6  0.43798861598244315\n",
       "4                    [0.2]   dropout_grid_model_7   0.4397494239154581\n",
       "5                   [0.25]   dropout_grid_model_5  0.44038035525401653\n",
       "6                    [0.8]  dropout_grid_model_11   0.4423341348749736\n",
       "7                    [0.0]   dropout_grid_model_1   0.4424075548395576\n",
       "8                   [0.35]   dropout_grid_model_8   0.4513440269002026\n",
       "9                    [0.1]   dropout_grid_model_3   0.4537795786037289\n",
       "10                  [0.01]   dropout_grid_model_2  0.46803037953434745"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dropout = dl_grid.get_grid(sort_by='rmse', decreasing=False)\n",
    "dropout.sorted_metric_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on our grid search, the best value for the hidden_dropout_ratios is 0.3, so for the next grid searches, we will use that value.\n",
    "\n",
    "Another parameter that is important in deep learning models is the learning rate. H2O offers a parameter called adaptive_rate(ADADELTA) and is enabled by default. Enabling the adaptive learning rate is roughly equivalent to turning on momentum training (as each model coefficient keeps track of its history), it is only slightly more computationally expensive (it uses an approximate square-root function that only costs a few clock cycles). Also, different hyper-parameters for adaptive learning rate can affect the training speed (at least for Rectifier activation functions, where back-propagation depends on the activation values).\n",
    "\n",
    "For cases where fastest model training is required (possibly at the expense of highest achievable accuracy), manual learning rates without momentum can be a good option, but in general, adaptive learning rate simplifies the usage of H2O Deep Learning and makes this tool highly usable by non-experts. However, let's try to disable the adaptive rate for now, and let's try to find a good learning rate for our model. For the learning rate, higher values result in a less stable model, while lower values lead to slower convergence. In order to specify the learning rate, we must disable adaptive_rate, as well as specify a value for the L2 regularization, which will add stability to our model. A suggested value for L2 is 1e-5. Please keep in mind that we could just use the adaptive learning rate feature, but we would like to show you the option to tune the parameters related to the learning rate.\n",
    "\n",
    "We will first start by finding a good learning rate for our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl = H2ODeepLearningEstimator(\n",
    "    epochs=10,\n",
    "    hidden=[165],\n",
    "    seed=42,\n",
    "    model_id='DL',\n",
    "    activation = 'rectifier_with_dropout',\n",
    "    hidden_dropout_ratios=[0.3],\n",
    "    distribution='auto',\n",
    "    adaptive_rate=False\n",
    "    )\n",
    "\n",
    "hyper_params = { 'rate' : [0.0001, 0.0005, 0.0008, 0.001, 0.0015, 0.0020, 0.003, 0.004, 0.007, 0.009]\n",
    "                \n",
    "    }\n",
    "\n",
    "search_criteria_tune = {'strategy': \"Cartesian\",\n",
    "                       }\n",
    "\n",
    "dl_grid = H2OGridSearch(model=dl, hyper_params=hyper_params,\n",
    "                         grid_id = 'rate_grid',\n",
    "                         search_criteria=search_criteria_tune,\n",
    "                         )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 0 ns\n",
      "deeplearning Grid Build progress: |███████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "%time \n",
    "dl_grid.train(x=x, y=y, training_frame=train, validation_frame = valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>rate</th>\n",
       "      <th>model_ids</th>\n",
       "      <th>rmse</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td>8.0E-4</td>\n",
       "      <td>rate_grid_model_3</td>\n",
       "      <td>0.433155557691245</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td>5.0E-4</td>\n",
       "      <td>rate_grid_model_2</td>\n",
       "      <td>0.43547853068737996</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td>0.001</td>\n",
       "      <td>rate_grid_model_4</td>\n",
       "      <td>0.43657606247967</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td></td>\n",
       "      <td>0.002</td>\n",
       "      <td>rate_grid_model_6</td>\n",
       "      <td>0.4370730518274487</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td></td>\n",
       "      <td>0.004</td>\n",
       "      <td>rate_grid_model_8</td>\n",
       "      <td>0.44132720783534013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td></td>\n",
       "      <td>0.003</td>\n",
       "      <td>rate_grid_model_7</td>\n",
       "      <td>0.4430098298549247</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td></td>\n",
       "      <td>0.009</td>\n",
       "      <td>rate_grid_model_10</td>\n",
       "      <td>0.44350017657233665</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td></td>\n",
       "      <td>0.0015</td>\n",
       "      <td>rate_grid_model_5</td>\n",
       "      <td>0.44359704501530195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td></td>\n",
       "      <td>1.0E-4</td>\n",
       "      <td>rate_grid_model_1</td>\n",
       "      <td>0.44385568648565144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td></td>\n",
       "      <td>0.007</td>\n",
       "      <td>rate_grid_model_9</td>\n",
       "      <td>0.44390107834121134</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       rate           model_ids                 rmse\n",
       "0    8.0E-4   rate_grid_model_3    0.433155557691245\n",
       "1    5.0E-4   rate_grid_model_2  0.43547853068737996\n",
       "2     0.001   rate_grid_model_4     0.43657606247967\n",
       "3     0.002   rate_grid_model_6   0.4370730518274487\n",
       "4     0.004   rate_grid_model_8  0.44132720783534013\n",
       "5     0.003   rate_grid_model_7   0.4430098298549247\n",
       "6     0.009  rate_grid_model_10  0.44350017657233665\n",
       "7    0.0015   rate_grid_model_5  0.44359704501530195\n",
       "8    1.0E-4   rate_grid_model_1  0.44385568648565144\n",
       "9     0.007   rate_grid_model_9  0.44390107834121134"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_per = dl_grid.get_grid(sort_by='rmse', decreasing=False)\n",
    "learn_per.sorted_metric_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best learning rate according to our results is 0.0015. Now that we found a good learning rate, we need to tune some parameters that are related to the learning rate. We will try to tune five more parameters, and to do so, we will do a random search with them and see if our model is improved. Below is a short description of the parameters we are going to tune.\n",
    "\n",
    "1. rate_annealing: Learning rate annealing reduces the learning rate to \"freeze\" into local minima in the optimization landscape. The annealing rate is the inverse of the number of training samples it takes to cut the learning rate in half.\n",
    "\n",
    "2. rate_decay: The learning rate decay parameter controls the change of learning rate across layers. Meaning that depending on your learning rate and the rate_decay, the learning rate for the weights connecting the input and first hidden layer will be different to the learning rate for the weights connecting the first and the second hidden layer, as for the second layer it will be a smaller value.\n",
    "\n",
    "3. momentum_ramp: The momentum_ramp parameter controls the amount of learning for which momentum increases (assuming momentum_stable is larger than momentum_start). The ramp is measured in the number of training samples.\n",
    "\n",
    "4. momentum_stable: The momentum_stable parameter controls the final momentum value reached after momentum_ramp training samples. The momentum used for training will remain the same for training beyond reaching that point.\n",
    "\n",
    "5. momentum_start: The momentum_start parameter controls the amount of momentum at the beginning of training\n",
    "\n",
    "We will run the grid search for 10 minutes and see the results of our last grid search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl = H2ODeepLearningEstimator(\n",
    "    epochs=10,\n",
    "    hidden=[165],\n",
    "    seed=42,\n",
    "    model_id='DL',\n",
    "    activation = 'rectifier_with_dropout',\n",
    "    hidden_dropout_ratios=[0.3],\n",
    "    distribution='auto',\n",
    "    adaptive_rate=False,\n",
    "    l2=1e-5,\n",
    "    rate=0.0015\n",
    "    )\n",
    "\n",
    "hyper_params = { 'rate_annealing' : [1e-6, 1e-7, 1e-8, 1e-5],\n",
    "                'rate_decay': [1, 0.8, 0.9, 1.1, 1.2],\n",
    "                'momentum_ramp' : [10000, 15000, 5000, 20000, 50000, 100000],\n",
    "                'momentum_stable' : [0.9, 0.95, 0.99, 0.999],\n",
    "                'momentum_start' : [0.9, 0.4, 0.5, 0.7, 0.8]\n",
    "    }\n",
    "\n",
    "search_criteria_tune = {'strategy': \"RandomDiscrete\",\n",
    "                   'max_runtime_secs': 600, #10 min  \n",
    "                   'max_models': 100,  ## build no more than 100 models\n",
    "                   'seed' : 42 \n",
    "                       }\n",
    "\n",
    "dl_grid = H2OGridSearch(model=dl, \n",
    "                        hyper_params=hyper_params,\n",
    "                        grid_id = 'rate_random_grid',\n",
    "                        search_criteria=search_criteria_tune,\n",
    "                         )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 0 ns\n",
      "deeplearning Grid Build progress: |███████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "%time \n",
    "dl_grid.train(x=x, y=y, training_frame=train, validation_frame = valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>momentum_ramp</th>\n",
       "      <th>momentum_stable</th>\n",
       "      <th>momentum_start</th>\n",
       "      <th>rate_annealing</th>\n",
       "      <th>rate_decay</th>\n",
       "      <th>model_ids</th>\n",
       "      <th>rmse</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td>10000.0</td>\n",
       "      <td>0.99</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.0E-6</td>\n",
       "      <td>1.0</td>\n",
       "      <td>rate_random_grid_model_6</td>\n",
       "      <td>0.4301771103515655</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td>5000.0</td>\n",
       "      <td>0.999</td>\n",
       "      <td>0.7</td>\n",
       "      <td>1.0E-5</td>\n",
       "      <td>0.9</td>\n",
       "      <td>rate_random_grid_model_2</td>\n",
       "      <td>0.4304419302072751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td>100000.0</td>\n",
       "      <td>0.99</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.0E-6</td>\n",
       "      <td>0.9</td>\n",
       "      <td>rate_random_grid_model_10</td>\n",
       "      <td>0.4320296415928409</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td></td>\n",
       "      <td>50000.0</td>\n",
       "      <td>0.999</td>\n",
       "      <td>0.8</td>\n",
       "      <td>1.0E-6</td>\n",
       "      <td>1.0</td>\n",
       "      <td>rate_random_grid_model_5</td>\n",
       "      <td>0.4333514969750732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td></td>\n",
       "      <td>15000.0</td>\n",
       "      <td>0.99</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.0E-5</td>\n",
       "      <td>1.2</td>\n",
       "      <td>rate_random_grid_model_12</td>\n",
       "      <td>0.43338024198151337</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td></td>\n",
       "      <td>10000.0</td>\n",
       "      <td>0.95</td>\n",
       "      <td>0.4</td>\n",
       "      <td>1.0E-7</td>\n",
       "      <td>0.9</td>\n",
       "      <td>rate_random_grid_model_9</td>\n",
       "      <td>0.43497180119109613</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td></td>\n",
       "      <td>5000.0</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.0E-6</td>\n",
       "      <td>0.9</td>\n",
       "      <td>rate_random_grid_model_8</td>\n",
       "      <td>0.4356958871262241</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td></td>\n",
       "      <td>10000.0</td>\n",
       "      <td>0.95</td>\n",
       "      <td>0.9</td>\n",
       "      <td>1.0E-5</td>\n",
       "      <td>0.8</td>\n",
       "      <td>rate_random_grid_model_1</td>\n",
       "      <td>0.43607634668201295</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td></td>\n",
       "      <td>10000.0</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.9</td>\n",
       "      <td>1.0E-5</td>\n",
       "      <td>1.2</td>\n",
       "      <td>rate_random_grid_model_11</td>\n",
       "      <td>0.4361884657454211</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td></td>\n",
       "      <td>100000.0</td>\n",
       "      <td>0.999</td>\n",
       "      <td>0.9</td>\n",
       "      <td>1.0E-7</td>\n",
       "      <td>0.8</td>\n",
       "      <td>rate_random_grid_model_4</td>\n",
       "      <td>0.43699592947680355</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td></td>\n",
       "      <td>10000.0</td>\n",
       "      <td>0.95</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.0E-6</td>\n",
       "      <td>0.8</td>\n",
       "      <td>rate_random_grid_model_3</td>\n",
       "      <td>0.43746520622364776</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td></td>\n",
       "      <td>10000.0</td>\n",
       "      <td>0.99</td>\n",
       "      <td>0.9</td>\n",
       "      <td>1.0E-8</td>\n",
       "      <td>1.2</td>\n",
       "      <td>rate_random_grid_model_13</td>\n",
       "      <td>0.44042900883919595</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td></td>\n",
       "      <td>5000.0</td>\n",
       "      <td>0.999</td>\n",
       "      <td>0.7</td>\n",
       "      <td>1.0E-8</td>\n",
       "      <td>1.1</td>\n",
       "      <td>rate_random_grid_model_7</td>\n",
       "      <td>0.44057160181040145</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td></td>\n",
       "      <td>5000.0</td>\n",
       "      <td>0.99</td>\n",
       "      <td>0.7</td>\n",
       "      <td>1.0E-6</td>\n",
       "      <td>1.1</td>\n",
       "      <td>rate_random_grid_model_14</td>\n",
       "      <td>0.4869247348597894</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     momentum_ramp momentum_stable momentum_start rate_annealing rate_decay  \\\n",
       "0          10000.0            0.99            0.5         1.0E-6        1.0   \n",
       "1           5000.0           0.999            0.7         1.0E-5        0.9   \n",
       "2         100000.0            0.99            0.5         1.0E-6        0.9   \n",
       "3          50000.0           0.999            0.8         1.0E-6        1.0   \n",
       "4          15000.0            0.99            0.5         1.0E-5        1.2   \n",
       "5          10000.0            0.95            0.4         1.0E-7        0.9   \n",
       "6           5000.0             0.9            0.5         1.0E-6        0.9   \n",
       "7          10000.0            0.95            0.9         1.0E-5        0.8   \n",
       "8          10000.0             0.9            0.9         1.0E-5        1.2   \n",
       "9         100000.0           0.999            0.9         1.0E-7        0.8   \n",
       "10         10000.0            0.95            0.5         1.0E-6        0.8   \n",
       "11         10000.0            0.99            0.9         1.0E-8        1.2   \n",
       "12          5000.0           0.999            0.7         1.0E-8        1.1   \n",
       "13          5000.0            0.99            0.7         1.0E-6        1.1   \n",
       "\n",
       "                    model_ids                 rmse  \n",
       "0    rate_random_grid_model_6   0.4301771103515655  \n",
       "1    rate_random_grid_model_2   0.4304419302072751  \n",
       "2   rate_random_grid_model_10   0.4320296415928409  \n",
       "3    rate_random_grid_model_5   0.4333514969750732  \n",
       "4   rate_random_grid_model_12  0.43338024198151337  \n",
       "5    rate_random_grid_model_9  0.43497180119109613  \n",
       "6    rate_random_grid_model_8   0.4356958871262241  \n",
       "7    rate_random_grid_model_1  0.43607634668201295  \n",
       "8   rate_random_grid_model_11   0.4361884657454211  \n",
       "9    rate_random_grid_model_4  0.43699592947680355  \n",
       "10   rate_random_grid_model_3  0.43746520622364776  \n",
       "11  rate_random_grid_model_13  0.44042900883919595  \n",
       "12   rate_random_grid_model_7  0.44057160181040145  \n",
       "13  rate_random_grid_model_14   0.4869247348597894  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_per = dl_grid.get_grid(sort_by='rmse', decreasing=False)\n",
    "learn_per.sorted_metric_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the list above, you can retrieve the parameters from the best model. In our case, the best model was the 17th model that was built, or model_id = rate_random_grid_model_17. We can see that the RMSE value slightly improved with the grid search that we just did. If you recall, at the beginning of the tutorial, we mentioned that H2O's DNN model allows you to do checkpointing. The checkpoint option allows you to specify a model key associated with a previously trained model. This will build a new model as a continuation of a previously generated model. Since we were training our model with only 10 epochs, let's now train the same model with 200 epochs while using early stopping to see if we get better results. Also, set reproducible=True, this model will take longer to train, but it will yield similar results when you run it.\n",
    "\n",
    "To be able to use checkpointing, we need to retrieve the model from the grid search that we did, and then use the model id to continue training the same model. There are several parameters that can not be changed, and for that reason, we need to specify them one more time; however, this guarantees us that we are training the same model from the grid search, and is not a new model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_dl_model = dl_grid.models[0]\n",
    "\n",
    "dl_checkpoint = H2ODeepLearningEstimator(checkpoint=best_dl_model.model_id,\n",
    "    epochs=200,\n",
    "    hidden=[165],\n",
    "    seed=42,\n",
    "    model_id='DL_checkpoint',\n",
    "    activation = 'rectifier_with_dropout',\n",
    "    hidden_dropout_ratios=[0.3],\n",
    "    distribution='auto',\n",
    "    adaptive_rate=False,\n",
    "    l2=1e-5,\n",
    "    rate=0.0015,\n",
    "    momentum_ramp=10000.0,\n",
    "    momentum_stable=0.99,\n",
    "    momentum_start=0.5,\n",
    "    rate_annealing=1.0e-6,\n",
    "    rate_decay=1.0,\n",
    "    reproducible=True,                                     \n",
    "                                         \n",
    "    stopping_metric='RMSE',\n",
    "    stopping_tolerance=0.0005,\n",
    "    stopping_rounds=5\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 0 ns\n",
      "deeplearning Model Build progress: |██████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "%time \n",
    "dl_checkpoint.train(x=x, y=y, training_frame=train, validation_frame=valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please note that we need to specify the parameters that we tuned, because if we don't, H2O assigns the default values, and most of the parameters that we tuned can not be changed when we do a checkpoint. Print the model summary by typing the name of your model in a new cell and running it, as shown in the image below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Details\n",
      "=============\n",
      "H2ODeepLearningEstimator :  Deep Learning\n",
      "Model Key:  DL_checkpoint\n",
      "\n",
      "\n",
      "Status of Neuron Layers: predicting ORIGINAL_INTEREST_RATE, regression, gaussian distribution, Quadratic loss, 27,556 weights/biases, 228.2 KB, 13,406,351 training samples, mini-batch size 1\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>layer</th>\n",
       "      <th>units</th>\n",
       "      <th>type</th>\n",
       "      <th>dropout</th>\n",
       "      <th>l1</th>\n",
       "      <th>l2</th>\n",
       "      <th>mean_rate</th>\n",
       "      <th>rate_rms</th>\n",
       "      <th>momentum</th>\n",
       "      <th>mean_weight</th>\n",
       "      <th>weight_rms</th>\n",
       "      <th>mean_bias</th>\n",
       "      <th>bias_rms</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td>1</td>\n",
       "      <td>165</td>\n",
       "      <td>Input</td>\n",
       "      <td>0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td>2</td>\n",
       "      <td>165</td>\n",
       "      <td>RectifierDropout</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>0.000104121</td>\n",
       "      <td>0</td>\n",
       "      <td>0.99</td>\n",
       "      <td>-0.0126665</td>\n",
       "      <td>0.106814</td>\n",
       "      <td>0.222765</td>\n",
       "      <td>0.100227</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Linear</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>0.000104121</td>\n",
       "      <td>0</td>\n",
       "      <td>0.99</td>\n",
       "      <td>0.0917103</td>\n",
       "      <td>0.347631</td>\n",
       "      <td>-0.0538511</td>\n",
       "      <td>1.09713e-154</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     layer  units              type dropout l1     l2    mean_rate rate_rms  \\\n",
       "0        1    165             Input       0                                   \n",
       "1        2    165  RectifierDropout      30  0  1e-05  0.000104121        0   \n",
       "2        3      1            Linear          0  1e-05  0.000104121        0   \n",
       "\n",
       "  momentum mean_weight weight_rms  mean_bias      bias_rms  \n",
       "0                                                           \n",
       "1     0.99  -0.0126665   0.106814   0.222765      0.100227  \n",
       "2     0.99   0.0917103   0.347631 -0.0538511  1.09713e-154  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "ModelMetricsRegression: deeplearning\n",
      "** Reported on train data. **\n",
      "\n",
      "MSE: 0.16911364387429423\n",
      "RMSE: 0.4112342931642426\n",
      "MAE: 0.2978915073004836\n",
      "RMSLE: 0.04879573205664998\n",
      "Mean Residual Deviance: 0.16911364387429423\n",
      "\n",
      "ModelMetricsRegression: deeplearning\n",
      "** Reported on validation data. **\n",
      "\n",
      "MSE: 0.17369076195434607\n",
      "RMSE: 0.4167622367181869\n",
      "MAE: 0.3025638227638908\n",
      "RMSLE: 0.04940479384357021\n",
      "Mean Residual Deviance: 0.17369076195434607\n",
      "\n",
      "Scoring History: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>duration</th>\n",
       "      <th>training_speed</th>\n",
       "      <th>epochs</th>\n",
       "      <th>iterations</th>\n",
       "      <th>samples</th>\n",
       "      <th>training_rmse</th>\n",
       "      <th>training_deviance</th>\n",
       "      <th>training_mae</th>\n",
       "      <th>training_r2</th>\n",
       "      <th>validation_rmse</th>\n",
       "      <th>validation_deviance</th>\n",
       "      <th>validation_mae</th>\n",
       "      <th>validation_r2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 15:34:49</td>\n",
       "      <td>0.000 sec</td>\n",
       "      <td>None</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 15:34:50</td>\n",
       "      <td>3 min 59.691 sec</td>\n",
       "      <td>66479 obs/sec</td>\n",
       "      <td>0.284882</td>\n",
       "      <td>1</td>\n",
       "      <td>99785.0</td>\n",
       "      <td>0.473595</td>\n",
       "      <td>0.224292</td>\n",
       "      <td>0.344861</td>\n",
       "      <td>0.317307</td>\n",
       "      <td>0.475430</td>\n",
       "      <td>0.226034</td>\n",
       "      <td>0.347454</td>\n",
       "      <td>0.323018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 15:34:55</td>\n",
       "      <td>4 min  4.864 sec</td>\n",
       "      <td>80620 obs/sec</td>\n",
       "      <td>1.425885</td>\n",
       "      <td>5</td>\n",
       "      <td>499442.0</td>\n",
       "      <td>0.479125</td>\n",
       "      <td>0.229561</td>\n",
       "      <td>0.349255</td>\n",
       "      <td>0.301269</td>\n",
       "      <td>0.481767</td>\n",
       "      <td>0.232099</td>\n",
       "      <td>0.351324</td>\n",
       "      <td>0.304851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 15:35:01</td>\n",
       "      <td>4 min 10.414 sec</td>\n",
       "      <td>88837 obs/sec</td>\n",
       "      <td>2.852801</td>\n",
       "      <td>10</td>\n",
       "      <td>999245.0</td>\n",
       "      <td>0.434289</td>\n",
       "      <td>0.188607</td>\n",
       "      <td>0.315483</td>\n",
       "      <td>0.425924</td>\n",
       "      <td>0.438293</td>\n",
       "      <td>0.192101</td>\n",
       "      <td>0.318645</td>\n",
       "      <td>0.424649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 15:35:06</td>\n",
       "      <td>4 min 15.709 sec</td>\n",
       "      <td>93504 obs/sec</td>\n",
       "      <td>4.278681</td>\n",
       "      <td>15</td>\n",
       "      <td>1498685.0</td>\n",
       "      <td>0.436966</td>\n",
       "      <td>0.190939</td>\n",
       "      <td>0.311575</td>\n",
       "      <td>0.418824</td>\n",
       "      <td>0.441851</td>\n",
       "      <td>0.195233</td>\n",
       "      <td>0.314874</td>\n",
       "      <td>0.415269</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 15:35:12</td>\n",
       "      <td>4 min 21.793 sec</td>\n",
       "      <td>97294 obs/sec</td>\n",
       "      <td>5.990696</td>\n",
       "      <td>21</td>\n",
       "      <td>2098349.0</td>\n",
       "      <td>0.460790</td>\n",
       "      <td>0.212328</td>\n",
       "      <td>0.317598</td>\n",
       "      <td>0.353723</td>\n",
       "      <td>0.465161</td>\n",
       "      <td>0.216374</td>\n",
       "      <td>0.320481</td>\n",
       "      <td>0.351948</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 15:35:18</td>\n",
       "      <td>4 min 27.735 sec</td>\n",
       "      <td>99815 obs/sec</td>\n",
       "      <td>7.704435</td>\n",
       "      <td>27</td>\n",
       "      <td>2698617.0</td>\n",
       "      <td>0.483816</td>\n",
       "      <td>0.234078</td>\n",
       "      <td>0.377289</td>\n",
       "      <td>0.287520</td>\n",
       "      <td>0.482648</td>\n",
       "      <td>0.232949</td>\n",
       "      <td>0.377641</td>\n",
       "      <td>0.302306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 15:35:23</td>\n",
       "      <td>4 min 32.789 sec</td>\n",
       "      <td>101184 obs/sec</td>\n",
       "      <td>9.133138</td>\n",
       "      <td>32</td>\n",
       "      <td>3199046.0</td>\n",
       "      <td>0.435031</td>\n",
       "      <td>0.189252</td>\n",
       "      <td>0.309273</td>\n",
       "      <td>0.423961</td>\n",
       "      <td>0.440107</td>\n",
       "      <td>0.193694</td>\n",
       "      <td>0.312866</td>\n",
       "      <td>0.419877</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 15:35:27</td>\n",
       "      <td>4 min 36.866 sec</td>\n",
       "      <td>102196 obs/sec</td>\n",
       "      <td>10.274553</td>\n",
       "      <td>36</td>\n",
       "      <td>3598847.0</td>\n",
       "      <td>0.427385</td>\n",
       "      <td>0.182658</td>\n",
       "      <td>0.307923</td>\n",
       "      <td>0.444032</td>\n",
       "      <td>0.430177</td>\n",
       "      <td>0.185052</td>\n",
       "      <td>0.310515</td>\n",
       "      <td>0.445759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 16:15:25</td>\n",
       "      <td>4 min 49.789 sec</td>\n",
       "      <td>83747 obs/sec</td>\n",
       "      <td>11.274553</td>\n",
       "      <td>37</td>\n",
       "      <td>3949115.0</td>\n",
       "      <td>0.416845</td>\n",
       "      <td>0.173760</td>\n",
       "      <td>0.303457</td>\n",
       "      <td>0.471114</td>\n",
       "      <td>0.420429</td>\n",
       "      <td>0.176761</td>\n",
       "      <td>0.306785</td>\n",
       "      <td>0.470593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 16:15:37</td>\n",
       "      <td>5 min  2.432 sec</td>\n",
       "      <td>72709 obs/sec</td>\n",
       "      <td>12.274553</td>\n",
       "      <td>38</td>\n",
       "      <td>4299383.0</td>\n",
       "      <td>0.416790</td>\n",
       "      <td>0.173714</td>\n",
       "      <td>0.301917</td>\n",
       "      <td>0.471255</td>\n",
       "      <td>0.420726</td>\n",
       "      <td>0.177011</td>\n",
       "      <td>0.305360</td>\n",
       "      <td>0.469844</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 16:15:52</td>\n",
       "      <td>5 min 17.211 sec</td>\n",
       "      <td>63424 obs/sec</td>\n",
       "      <td>13.274553</td>\n",
       "      <td>39</td>\n",
       "      <td>4649651.0</td>\n",
       "      <td>0.415833</td>\n",
       "      <td>0.172917</td>\n",
       "      <td>0.301946</td>\n",
       "      <td>0.473679</td>\n",
       "      <td>0.420251</td>\n",
       "      <td>0.176611</td>\n",
       "      <td>0.305808</td>\n",
       "      <td>0.471042</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 16:16:05</td>\n",
       "      <td>5 min 30.216 sec</td>\n",
       "      <td>58335 obs/sec</td>\n",
       "      <td>14.274553</td>\n",
       "      <td>40</td>\n",
       "      <td>4999919.0</td>\n",
       "      <td>0.415331</td>\n",
       "      <td>0.172500</td>\n",
       "      <td>0.299878</td>\n",
       "      <td>0.474949</td>\n",
       "      <td>0.419767</td>\n",
       "      <td>0.176204</td>\n",
       "      <td>0.303847</td>\n",
       "      <td>0.472260</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 16:16:18</td>\n",
       "      <td>5 min 43.084 sec</td>\n",
       "      <td>54629 obs/sec</td>\n",
       "      <td>15.274553</td>\n",
       "      <td>41</td>\n",
       "      <td>5350187.0</td>\n",
       "      <td>0.415730</td>\n",
       "      <td>0.172832</td>\n",
       "      <td>0.301811</td>\n",
       "      <td>0.473939</td>\n",
       "      <td>0.419371</td>\n",
       "      <td>0.175872</td>\n",
       "      <td>0.305220</td>\n",
       "      <td>0.473254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 16:16:32</td>\n",
       "      <td>5 min 57.568 sec</td>\n",
       "      <td>50975 obs/sec</td>\n",
       "      <td>16.274553</td>\n",
       "      <td>42</td>\n",
       "      <td>5700455.0</td>\n",
       "      <td>0.414813</td>\n",
       "      <td>0.172070</td>\n",
       "      <td>0.301114</td>\n",
       "      <td>0.476258</td>\n",
       "      <td>0.419024</td>\n",
       "      <td>0.175581</td>\n",
       "      <td>0.304988</td>\n",
       "      <td>0.474125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 16:16:46</td>\n",
       "      <td>6 min 10.720 sec</td>\n",
       "      <td>48637 obs/sec</td>\n",
       "      <td>17.274553</td>\n",
       "      <td>43</td>\n",
       "      <td>6050723.0</td>\n",
       "      <td>0.414156</td>\n",
       "      <td>0.171526</td>\n",
       "      <td>0.300813</td>\n",
       "      <td>0.477915</td>\n",
       "      <td>0.418609</td>\n",
       "      <td>0.175233</td>\n",
       "      <td>0.304769</td>\n",
       "      <td>0.475168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 16:16:58</td>\n",
       "      <td>6 min 23.417 sec</td>\n",
       "      <td>46920 obs/sec</td>\n",
       "      <td>18.274553</td>\n",
       "      <td>44</td>\n",
       "      <td>6400991.0</td>\n",
       "      <td>0.414864</td>\n",
       "      <td>0.172112</td>\n",
       "      <td>0.301545</td>\n",
       "      <td>0.476131</td>\n",
       "      <td>0.418661</td>\n",
       "      <td>0.175277</td>\n",
       "      <td>0.304999</td>\n",
       "      <td>0.475037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 16:17:11</td>\n",
       "      <td>6 min 36.322 sec</td>\n",
       "      <td>45391 obs/sec</td>\n",
       "      <td>19.274553</td>\n",
       "      <td>45</td>\n",
       "      <td>6751259.0</td>\n",
       "      <td>0.414041</td>\n",
       "      <td>0.171430</td>\n",
       "      <td>0.299571</td>\n",
       "      <td>0.478207</td>\n",
       "      <td>0.418404</td>\n",
       "      <td>0.175062</td>\n",
       "      <td>0.303188</td>\n",
       "      <td>0.475681</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 16:17:23</td>\n",
       "      <td>6 min 48.558 sec</td>\n",
       "      <td>44279 obs/sec</td>\n",
       "      <td>20.274553</td>\n",
       "      <td>46</td>\n",
       "      <td>7101527.0</td>\n",
       "      <td>0.413539</td>\n",
       "      <td>0.171015</td>\n",
       "      <td>0.300449</td>\n",
       "      <td>0.479470</td>\n",
       "      <td>0.418248</td>\n",
       "      <td>0.174932</td>\n",
       "      <td>0.304628</td>\n",
       "      <td>0.476071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td></td>\n",
       "      <td>2020-08-12 16:17:37</td>\n",
       "      <td>7 min  1.798 sec</td>\n",
       "      <td>43082 obs/sec</td>\n",
       "      <td>21.274553</td>\n",
       "      <td>47</td>\n",
       "      <td>7451795.0</td>\n",
       "      <td>0.413240</td>\n",
       "      <td>0.170767</td>\n",
       "      <td>0.300487</td>\n",
       "      <td>0.480224</td>\n",
       "      <td>0.418626</td>\n",
       "      <td>0.175248</td>\n",
       "      <td>0.304802</td>\n",
       "      <td>0.475124</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                timestamp           duration  training_speed     epochs  \\\n",
       "0     2020-08-12 15:34:49          0.000 sec            None   0.000000   \n",
       "1     2020-08-12 15:34:50   3 min 59.691 sec   66479 obs/sec   0.284882   \n",
       "2     2020-08-12 15:34:55   4 min  4.864 sec   80620 obs/sec   1.425885   \n",
       "3     2020-08-12 15:35:01   4 min 10.414 sec   88837 obs/sec   2.852801   \n",
       "4     2020-08-12 15:35:06   4 min 15.709 sec   93504 obs/sec   4.278681   \n",
       "5     2020-08-12 15:35:12   4 min 21.793 sec   97294 obs/sec   5.990696   \n",
       "6     2020-08-12 15:35:18   4 min 27.735 sec   99815 obs/sec   7.704435   \n",
       "7     2020-08-12 15:35:23   4 min 32.789 sec  101184 obs/sec   9.133138   \n",
       "8     2020-08-12 15:35:27   4 min 36.866 sec  102196 obs/sec  10.274553   \n",
       "9     2020-08-12 16:15:25   4 min 49.789 sec   83747 obs/sec  11.274553   \n",
       "10    2020-08-12 16:15:37   5 min  2.432 sec   72709 obs/sec  12.274553   \n",
       "11    2020-08-12 16:15:52   5 min 17.211 sec   63424 obs/sec  13.274553   \n",
       "12    2020-08-12 16:16:05   5 min 30.216 sec   58335 obs/sec  14.274553   \n",
       "13    2020-08-12 16:16:18   5 min 43.084 sec   54629 obs/sec  15.274553   \n",
       "14    2020-08-12 16:16:32   5 min 57.568 sec   50975 obs/sec  16.274553   \n",
       "15    2020-08-12 16:16:46   6 min 10.720 sec   48637 obs/sec  17.274553   \n",
       "16    2020-08-12 16:16:58   6 min 23.417 sec   46920 obs/sec  18.274553   \n",
       "17    2020-08-12 16:17:11   6 min 36.322 sec   45391 obs/sec  19.274553   \n",
       "18    2020-08-12 16:17:23   6 min 48.558 sec   44279 obs/sec  20.274553   \n",
       "19    2020-08-12 16:17:37   7 min  1.798 sec   43082 obs/sec  21.274553   \n",
       "\n",
       "    iterations    samples  training_rmse  training_deviance  training_mae  \\\n",
       "0            0        0.0            NaN                NaN           NaN   \n",
       "1            1    99785.0       0.473595           0.224292      0.344861   \n",
       "2            5   499442.0       0.479125           0.229561      0.349255   \n",
       "3           10   999245.0       0.434289           0.188607      0.315483   \n",
       "4           15  1498685.0       0.436966           0.190939      0.311575   \n",
       "5           21  2098349.0       0.460790           0.212328      0.317598   \n",
       "6           27  2698617.0       0.483816           0.234078      0.377289   \n",
       "7           32  3199046.0       0.435031           0.189252      0.309273   \n",
       "8           36  3598847.0       0.427385           0.182658      0.307923   \n",
       "9           37  3949115.0       0.416845           0.173760      0.303457   \n",
       "10          38  4299383.0       0.416790           0.173714      0.301917   \n",
       "11          39  4649651.0       0.415833           0.172917      0.301946   \n",
       "12          40  4999919.0       0.415331           0.172500      0.299878   \n",
       "13          41  5350187.0       0.415730           0.172832      0.301811   \n",
       "14          42  5700455.0       0.414813           0.172070      0.301114   \n",
       "15          43  6050723.0       0.414156           0.171526      0.300813   \n",
       "16          44  6400991.0       0.414864           0.172112      0.301545   \n",
       "17          45  6751259.0       0.414041           0.171430      0.299571   \n",
       "18          46  7101527.0       0.413539           0.171015      0.300449   \n",
       "19          47  7451795.0       0.413240           0.170767      0.300487   \n",
       "\n",
       "    training_r2  validation_rmse  validation_deviance  validation_mae  \\\n",
       "0           NaN              NaN                  NaN             NaN   \n",
       "1      0.317307         0.475430             0.226034        0.347454   \n",
       "2      0.301269         0.481767             0.232099        0.351324   \n",
       "3      0.425924         0.438293             0.192101        0.318645   \n",
       "4      0.418824         0.441851             0.195233        0.314874   \n",
       "5      0.353723         0.465161             0.216374        0.320481   \n",
       "6      0.287520         0.482648             0.232949        0.377641   \n",
       "7      0.423961         0.440107             0.193694        0.312866   \n",
       "8      0.444032         0.430177             0.185052        0.310515   \n",
       "9      0.471114         0.420429             0.176761        0.306785   \n",
       "10     0.471255         0.420726             0.177011        0.305360   \n",
       "11     0.473679         0.420251             0.176611        0.305808   \n",
       "12     0.474949         0.419767             0.176204        0.303847   \n",
       "13     0.473939         0.419371             0.175872        0.305220   \n",
       "14     0.476258         0.419024             0.175581        0.304988   \n",
       "15     0.477915         0.418609             0.175233        0.304769   \n",
       "16     0.476131         0.418661             0.175277        0.304999   \n",
       "17     0.478207         0.418404             0.175062        0.303188   \n",
       "18     0.479470         0.418248             0.174932        0.304628   \n",
       "19     0.480224         0.418626             0.175248        0.304802   \n",
       "\n",
       "    validation_r2  \n",
       "0             NaN  \n",
       "1        0.323018  \n",
       "2        0.304851  \n",
       "3        0.424649  \n",
       "4        0.415269  \n",
       "5        0.351948  \n",
       "6        0.302306  \n",
       "7        0.419877  \n",
       "8        0.445759  \n",
       "9        0.470593  \n",
       "10       0.469844  \n",
       "11       0.471042  \n",
       "12       0.472260  \n",
       "13       0.473254  \n",
       "14       0.474125  \n",
       "15       0.475168  \n",
       "16       0.475037  \n",
       "17       0.475681  \n",
       "18       0.476071  \n",
       "19       0.475124  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "See the whole table with table.as_data_frame()\n",
      "\n",
      "Variable Importances: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>variable</th>\n",
       "      <th>relative_importance</th>\n",
       "      <th>scaled_importance</th>\n",
       "      <th>percentage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>SELLER_NAME.NORWESTMORTGAGE,INC</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.014304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>SELLER_NAME.CROSSLANDMTGECORP</td>\n",
       "      <td>0.912073</td>\n",
       "      <td>0.912073</td>\n",
       "      <td>0.013046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CHANNEL.R</td>\n",
       "      <td>0.856126</td>\n",
       "      <td>0.856126</td>\n",
       "      <td>0.012246</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>SELLER_NAME.PNCMTGECORPOFAMERICA</td>\n",
       "      <td>0.833582</td>\n",
       "      <td>0.833582</td>\n",
       "      <td>0.011923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>SELLER_NAME.COUNTRYWIDE</td>\n",
       "      <td>0.824922</td>\n",
       "      <td>0.824922</td>\n",
       "      <td>0.011800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>SELLER_NAME.OLDKENTMTGECO</td>\n",
       "      <td>0.824168</td>\n",
       "      <td>0.824168</td>\n",
       "      <td>0.011789</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>SELLER_NAME.FIRST UNION CAPITAL</td>\n",
       "      <td>0.737920</td>\n",
       "      <td>0.737920</td>\n",
       "      <td>0.010555</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>SERVICER_NAME.ABNAMROMTGEGROUP,INC</td>\n",
       "      <td>0.719728</td>\n",
       "      <td>0.719728</td>\n",
       "      <td>0.010295</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>SELLER_NAME.FLAGSTARBANK,FSB</td>\n",
       "      <td>0.709444</td>\n",
       "      <td>0.709444</td>\n",
       "      <td>0.010148</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>SELLER_NAME.Other sellers</td>\n",
       "      <td>0.700602</td>\n",
       "      <td>0.700602</td>\n",
       "      <td>0.010021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>SELLER_NAME.STANDARD FEDERAL BAN</td>\n",
       "      <td>0.700266</td>\n",
       "      <td>0.700266</td>\n",
       "      <td>0.010016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>LOAN_PURPOSE.N</td>\n",
       "      <td>0.700000</td>\n",
       "      <td>0.700000</td>\n",
       "      <td>0.010013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>CHANNEL.T</td>\n",
       "      <td>0.688046</td>\n",
       "      <td>0.688046</td>\n",
       "      <td>0.009842</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>FIRST_TIME_HOMEBUYER_FLAG.</td>\n",
       "      <td>0.667278</td>\n",
       "      <td>0.667278</td>\n",
       "      <td>0.009545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>SELLER_NAME.FIFTHTHIRDBANK</td>\n",
       "      <td>0.664047</td>\n",
       "      <td>0.664047</td>\n",
       "      <td>0.009498</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>SELLER_NAME.FIRSTHORIZONHOMELOAN</td>\n",
       "      <td>0.663091</td>\n",
       "      <td>0.663091</td>\n",
       "      <td>0.009485</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>SELLER_NAME.NATLCITYMTGECO</td>\n",
       "      <td>0.653488</td>\n",
       "      <td>0.653488</td>\n",
       "      <td>0.009347</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>OCCUPANCY_STATUS.I</td>\n",
       "      <td>0.649291</td>\n",
       "      <td>0.649291</td>\n",
       "      <td>0.009287</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>OCCUPANCY_STATUS.O</td>\n",
       "      <td>0.648064</td>\n",
       "      <td>0.648064</td>\n",
       "      <td>0.009270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>SELLER_NAME.ABNAMROMTGEGROUP,INC</td>\n",
       "      <td>0.640946</td>\n",
       "      <td>0.640946</td>\n",
       "      <td>0.009168</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              variable  relative_importance  \\\n",
       "0      SELLER_NAME.NORWESTMORTGAGE,INC             1.000000   \n",
       "1        SELLER_NAME.CROSSLANDMTGECORP             0.912073   \n",
       "2                            CHANNEL.R             0.856126   \n",
       "3     SELLER_NAME.PNCMTGECORPOFAMERICA             0.833582   \n",
       "4              SELLER_NAME.COUNTRYWIDE             0.824922   \n",
       "5            SELLER_NAME.OLDKENTMTGECO             0.824168   \n",
       "6      SELLER_NAME.FIRST UNION CAPITAL             0.737920   \n",
       "7   SERVICER_NAME.ABNAMROMTGEGROUP,INC             0.719728   \n",
       "8         SELLER_NAME.FLAGSTARBANK,FSB             0.709444   \n",
       "9            SELLER_NAME.Other sellers             0.700602   \n",
       "10    SELLER_NAME.STANDARD FEDERAL BAN             0.700266   \n",
       "11                      LOAN_PURPOSE.N             0.700000   \n",
       "12                           CHANNEL.T             0.688046   \n",
       "13          FIRST_TIME_HOMEBUYER_FLAG.             0.667278   \n",
       "14          SELLER_NAME.FIFTHTHIRDBANK             0.664047   \n",
       "15    SELLER_NAME.FIRSTHORIZONHOMELOAN             0.663091   \n",
       "16          SELLER_NAME.NATLCITYMTGECO             0.653488   \n",
       "17                  OCCUPANCY_STATUS.I             0.649291   \n",
       "18                  OCCUPANCY_STATUS.O             0.648064   \n",
       "19    SELLER_NAME.ABNAMROMTGEGROUP,INC             0.640946   \n",
       "\n",
       "    scaled_importance  percentage  \n",
       "0            1.000000    0.014304  \n",
       "1            0.912073    0.013046  \n",
       "2            0.856126    0.012246  \n",
       "3            0.833582    0.011923  \n",
       "4            0.824922    0.011800  \n",
       "5            0.824168    0.011789  \n",
       "6            0.737920    0.010555  \n",
       "7            0.719728    0.010295  \n",
       "8            0.709444    0.010148  \n",
       "9            0.700602    0.010021  \n",
       "10           0.700266    0.010016  \n",
       "11           0.700000    0.010013  \n",
       "12           0.688046    0.009842  \n",
       "13           0.667278    0.009545  \n",
       "14           0.664047    0.009498  \n",
       "15           0.663091    0.009485  \n",
       "16           0.653488    0.009347  \n",
       "17           0.649291    0.009287  \n",
       "18           0.648064    0.009270  \n",
       "19           0.640946    0.009168  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "See the whole table with table.as_data_frame()\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dl_checkpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see in the image above, the RMSE and MAE slightly decreased, meaning that by increasing the number of epochs, we were able to improve the performance of our model.\n",
    "\n",
    "You can also tune a Deep Learning model in Flow, and you would do it in the same way we did with the XGBoost. Give it a try on your own, using the parameters that we found, do a local search with them. For example, since we found the rate to be 0.0015, you could do a grid search where you use the values 0.0010; 0.0012; 0.0017; 0.0019;\n",
    "Try to do the same with some of the other parameters that we found and see if you can get better results. Once you are done with Flow, go back to your Jupyter Notebook.\n",
    "\n",
    "Let's see how our default model performance compares to the tuned one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuned_dl_per = dl_checkpoint.model_performance(valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "ModelMetricsRegression: deeplearning\n",
      "** Reported on test data. **\n",
      "\n",
      "MSE: 0.17369076195434607\n",
      "RMSE: 0.4167622367181869\n",
      "MAE: 0.3025638227638908\n",
      "RMSLE: 0.04940479384357021\n",
      "Mean Residual Deviance: 0.17369076195434607\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tuned_dl_per"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Check both RMSE and MAE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Default DL Model RMSE: 0.4426 \n",
      "Tuned DL Model RMSE:0.4168\n"
     ]
    }
   ],
   "source": [
    "print(\"Default DL Model RMSE: %.4f \\nTuned DL Model RMSE:%.4f\" % (default_dl_per.rmse(), tuned_dl_per.rmse()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Default DL Model MAE: 0.3182 \n",
      "Tuned DL Model MAE:0.3026\n"
     ]
    }
   ],
   "source": [
    "print(\"Default DL Model MAE: %.4f \\nTuned DL Model MAE:%.4f\" % (default_dl_per.mae(), tuned_dl_per.mae()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, both scores improved with the grid searches that we did. In fact, our deep learning model had the greatest improvement out of the two models we tuned; however, the XGBoost still performed slightly better than the Deep Learning model.\n",
    "\n",
    "We will now see how our models perform on the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Evaludate Test Set Performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have done some tuning for our models, we will see how the models would perform on unseen data. Let's first print the final RMSE for both models. To do so, we need to evaluate the performance of our models on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl_test_per = dl_checkpoint.model_performance(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb_test_per = best_xgb_model.model_performance(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Check both RMSE and MAE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nDeep Learning model Test MAE: 0.3036 \n"
     ]
    }
   ],
   "source": [
    "print(\"nDeep Learning model Test MAE: %.4f \" % \n",
    "      (dl_test_per.mae()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deep Learning Model Test RMSE: 0.4195 \n"
     ]
    }
   ],
   "source": [
    "print(\"Deep Learning Model Test RMSE: %.4f \" % \n",
    "      (dl_test_per.rmse()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"XGBoost Test RMSE: %.4f  \\nDeep Learning Model Test RMSE: %.4f \" % \n",
    "      (xgb_test_per.rmse(), dl_test_per.rmse()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The test RMSE for both models is very close to the validation results, which means that our validation set cross-validation approach worked well. As we mentioned before, the RMSE score adds more penalty to larger errors. Based on the RMSE, on average, the tuned models' predictions on the test set are about 0.41 off.\n",
    "\n",
    "In terms of test MAE, we see that the predictions of the interest rate are 0.3 off from the actual values, which is a good indication that the model is making good predictions for the interest rate. Let's see some of the predictions for both models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### We will take a look at the first ten predictions of both models, compared to the actual interest value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb_tuned_pred = best_xgb_model.predict(test) #get predictions from xgboost\n",
    "test_rate_pred=test['ORIGINAL_INTEREST_RATE'].cbind(xgb_tuned_pred)#combined xgb predictions with actual interest rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deeplearning prediction progress: |███████████████████████████████████████| 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  ORIGINAL_INTEREST_RATE</th><th style=\"text-align: right;\">  predict</th><th style=\"text-align: right;\">  predict0</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">                   6    </td><td style=\"text-align: right;\">  6.53058</td><td style=\"text-align: right;\">   6.53058</td></tr>\n",
       "<tr><td style=\"text-align: right;\">                   6    </td><td style=\"text-align: right;\">  6.17022</td><td style=\"text-align: right;\">   6.17022</td></tr>\n",
       "<tr><td style=\"text-align: right;\">                   7.25 </td><td style=\"text-align: right;\">  6.97823</td><td style=\"text-align: right;\">   6.97823</td></tr>\n",
       "<tr><td style=\"text-align: right;\">                   7.125</td><td style=\"text-align: right;\">  7.03079</td><td style=\"text-align: right;\">   7.03079</td></tr>\n",
       "<tr><td style=\"text-align: right;\">                   7.125</td><td style=\"text-align: right;\">  7.17262</td><td style=\"text-align: right;\">   7.17262</td></tr>\n",
       "<tr><td style=\"text-align: right;\">                   7.75 </td><td style=\"text-align: right;\">  7.03528</td><td style=\"text-align: right;\">   7.03528</td></tr>\n",
       "<tr><td style=\"text-align: right;\">                   6.99 </td><td style=\"text-align: right;\">  6.79611</td><td style=\"text-align: right;\">   6.79611</td></tr>\n",
       "<tr><td style=\"text-align: right;\">                   7.125</td><td style=\"text-align: right;\">  7.18585</td><td style=\"text-align: right;\">   7.18585</td></tr>\n",
       "<tr><td style=\"text-align: right;\">                   7.25 </td><td style=\"text-align: right;\">  7.04871</td><td style=\"text-align: right;\">   7.04871</td></tr>\n",
       "<tr><td style=\"text-align: right;\">                   7.125</td><td style=\"text-align: right;\">  6.88045</td><td style=\"text-align: right;\">   6.88045</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dl_tuned_pred = dl_checkpoint.predict(test)#get predictions from Deep Learning Model\n",
    "test_rate_pred=test['ORIGINAL_INTEREST_RATE'].cbind(dl_tuned_pred)#combined deeplearning predictions with actual interest rate\n",
    "test_rate_pred.cbind(dl_tuned_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please note that the predict column has the predictions for the XGBoost model, and the predict0 column has the predictions for the Deep Learning model. As we can see in the image above, both predictions are close to the actual values. Although there might be some predictions that might be very off, the RMSE and MAE proved to us that both models make good predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Shutdown the cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "H2O session _sid_88f3 closed.\n"
     ]
    }
   ],
   "source": [
    "h2o.cluster().shutdown()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}