{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "76dc69f3-2413-4686-bfe9-cafc08f20c27",
   "metadata": {},
   "source": [
    "# AutoML for wage prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8342cb7",
   "metadata": {
    "papermill": {
     "duration": 0.013777,
     "end_time": "2021-03-24T11:24:18.450894",
     "exception": false,
     "start_time": "2021-03-24T11:24:18.437117",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Automatic Machine Learning with H2O AutoML using Wage Data from 2015"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fbd05b7",
   "metadata": {
    "papermill": {
     "duration": 0.014076,
     "end_time": "2021-03-24T11:24:18.478815",
     "exception": false,
     "start_time": "2021-03-24T11:24:18.464739",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We illustrate how to predict an outcome variable Y in a high-dimensional setting, using the AutoML package *H2O* that covers the complete pipeline from the raw dataset to the deployable machine learning model. In last few years, AutoML or automated machine learning has become widely popular among data science community. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5333433f",
   "metadata": {
    "papermill": {
     "duration": 0.013915,
     "end_time": "2021-03-24T11:24:18.508556",
     "exception": false,
     "start_time": "2021-03-24T11:24:18.494641",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can use AutoML as a benchmark and compare it to the methods that we used in the previous notebook where we applied one machine learning method after the other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "dd30c6a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import relevant packages\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import pyreadr\n",
    "import os\n",
    "from urllib.request import urlopen\n",
    "from sklearn import preprocessing\n",
    "import patsy\n",
    "from h2o.automl import H2OAutoML\n",
    "\n",
    "from numpy import loadtxt\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "fb1130e2-716b-49c7-b6f3-e7061fc1dd9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#pip install h2o"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "0e6d7d98",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking whether there is an H2O instance running at http://localhost:54321 . connected.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td>H2O_cluster_uptime:</td>\n",
       "<td>4 hours 11 mins</td></tr>\n",
       "<tr><td>H2O_cluster_timezone:</td>\n",
       "<td>America/Bogota</td></tr>\n",
       "<tr><td>H2O_data_parsing_timezone:</td>\n",
       "<td>UTC</td></tr>\n",
       "<tr><td>H2O_cluster_version:</td>\n",
       "<td>3.36.1.3</td></tr>\n",
       "<tr><td>H2O_cluster_version_age:</td>\n",
       "<td>26 days </td></tr>\n",
       "<tr><td>H2O_cluster_name:</td>\n",
       "<td>H2O_from_python_User_fi8ht0</td></tr>\n",
       "<tr><td>H2O_cluster_total_nodes:</td>\n",
       "<td>1</td></tr>\n",
       "<tr><td>H2O_cluster_free_memory:</td>\n",
       "<td>2.467 Gb</td></tr>\n",
       "<tr><td>H2O_cluster_total_cores:</td>\n",
       "<td>4</td></tr>\n",
       "<tr><td>H2O_cluster_allowed_cores:</td>\n",
       "<td>4</td></tr>\n",
       "<tr><td>H2O_cluster_status:</td>\n",
       "<td>locked, healthy</td></tr>\n",
       "<tr><td>H2O_connection_url:</td>\n",
       "<td>http://localhost:54321</td></tr>\n",
       "<tr><td>H2O_connection_proxy:</td>\n",
       "<td>{\"http\": null, \"https\": null}</td></tr>\n",
       "<tr><td>H2O_internal_security:</td>\n",
       "<td>False</td></tr>\n",
       "<tr><td>Python_version:</td>\n",
       "<td>3.9.12 final</td></tr></table></div>"
      ],
      "text/plain": [
       "--------------------------  -----------------------------\n",
       "H2O_cluster_uptime:         4 hours 11 mins\n",
       "H2O_cluster_timezone:       America/Bogota\n",
       "H2O_data_parsing_timezone:  UTC\n",
       "H2O_cluster_version:        3.36.1.3\n",
       "H2O_cluster_version_age:    26 days\n",
       "H2O_cluster_name:           H2O_from_python_User_fi8ht0\n",
       "H2O_cluster_total_nodes:    1\n",
       "H2O_cluster_free_memory:    2.467 Gb\n",
       "H2O_cluster_total_cores:    4\n",
       "H2O_cluster_allowed_cores:  4\n",
       "H2O_cluster_status:         locked, healthy\n",
       "H2O_connection_url:         http://localhost:54321\n",
       "H2O_connection_proxy:       {\"http\": null, \"https\": null}\n",
       "H2O_internal_security:      False\n",
       "Python_version:             3.9.12 final\n",
       "--------------------------  -----------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# load the H2O package\n",
    "import h2o\n",
    "\n",
    "# start h2o cluster\n",
    "h2o.init()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6de4a8bc-161e-4e0a-ae67-45bab3517933",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "link=\"https://raw.githubusercontent.com/d2cml-ai/14.388_py/main/data/wage2015_subsample_inference.Rdata\"\n",
    "response = urlopen(link)\n",
    "content = response.read()\n",
    "fhandle = open( 'wage2015_subsample_inference.Rdata', 'wb')\n",
    "fhandle.write(content)\n",
    "fhandle.close()\n",
    "result = pyreadr.read_r(\"wage2015_subsample_inference.Rdata\")\n",
    "os.remove(\"wage2015_subsample_inference.Rdata\")\n",
    "\n",
    "# Extracting the data frame from rdata_read\n",
    "data = result[ 'data' ]\n",
    "n = data.shape[0]\n",
    "type(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "305bb8e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import relevant packages for splitting data\n",
    "import random\n",
    "import math\n",
    "\n",
    "# Set Seed\n",
    "# to make the results replicable (generating random numbers)\n",
    "np.random.seed(0)\n",
    "random = np.random.randint(0, data.shape[0], size=math.floor(data.shape[0]))\n",
    "data[\"random\"] = random\n",
    "random    # the array does not change \n",
    "data_2 = data.sort_values(by=['random'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "52dd607c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3862, 21)\n",
      "(1288, 21)\n"
     ]
    }
   ],
   "source": [
    "# Create training and testing sample \n",
    "train = data_2[ : math.floor(n*3/4)]    # training sample\n",
    "test =  data_2[ math.floor(n*3/4) : ]   # testing sample\n",
    "print(train.shape)\n",
    "print(test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "bed9f791",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking whether there is an H2O instance running at http://localhost:54321 . connected.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td>H2O_cluster_uptime:</td>\n",
       "<td>4 hours 12 mins</td></tr>\n",
       "<tr><td>H2O_cluster_timezone:</td>\n",
       "<td>America/Bogota</td></tr>\n",
       "<tr><td>H2O_data_parsing_timezone:</td>\n",
       "<td>UTC</td></tr>\n",
       "<tr><td>H2O_cluster_version:</td>\n",
       "<td>3.36.1.3</td></tr>\n",
       "<tr><td>H2O_cluster_version_age:</td>\n",
       "<td>26 days </td></tr>\n",
       "<tr><td>H2O_cluster_name:</td>\n",
       "<td>H2O_from_python_User_fi8ht0</td></tr>\n",
       "<tr><td>H2O_cluster_total_nodes:</td>\n",
       "<td>1</td></tr>\n",
       "<tr><td>H2O_cluster_free_memory:</td>\n",
       "<td>2.467 Gb</td></tr>\n",
       "<tr><td>H2O_cluster_total_cores:</td>\n",
       "<td>4</td></tr>\n",
       "<tr><td>H2O_cluster_allowed_cores:</td>\n",
       "<td>4</td></tr>\n",
       "<tr><td>H2O_cluster_status:</td>\n",
       "<td>locked, healthy</td></tr>\n",
       "<tr><td>H2O_connection_url:</td>\n",
       "<td>http://localhost:54321</td></tr>\n",
       "<tr><td>H2O_connection_proxy:</td>\n",
       "<td>{\"http\": null, \"https\": null}</td></tr>\n",
       "<tr><td>H2O_internal_security:</td>\n",
       "<td>False</td></tr>\n",
       "<tr><td>Python_version:</td>\n",
       "<td>3.9.12 final</td></tr></table></div>"
      ],
      "text/plain": [
       "--------------------------  -----------------------------\n",
       "H2O_cluster_uptime:         4 hours 12 mins\n",
       "H2O_cluster_timezone:       America/Bogota\n",
       "H2O_data_parsing_timezone:  UTC\n",
       "H2O_cluster_version:        3.36.1.3\n",
       "H2O_cluster_version_age:    26 days\n",
       "H2O_cluster_name:           H2O_from_python_User_fi8ht0\n",
       "H2O_cluster_total_nodes:    1\n",
       "H2O_cluster_free_memory:    2.467 Gb\n",
       "H2O_cluster_total_cores:    4\n",
       "H2O_cluster_allowed_cores:  4\n",
       "H2O_cluster_status:         locked, healthy\n",
       "H2O_connection_url:         http://localhost:54321\n",
       "H2O_connection_proxy:       {\"http\": null, \"https\": null}\n",
       "H2O_internal_security:      False\n",
       "Python_version:             3.9.12 final\n",
       "--------------------------  -----------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# start h2o cluster\n",
    "h2o.init()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "bb269c76",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%\n",
      "Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%\n",
      "Rows:3862\n",
      "Cols:21\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>       </th><th>wage              </th><th>lwage             </th><th>sex               </th><th>shs                 </th><th>hsg                </th><th>scl                </th><th>clg                </th><th>ad                 </th><th>mw                </th><th>so                 </th><th>we                 </th><th>ne                 </th><th>exp1              </th><th>exp2              </th><th>exp3              </th><th>exp4              </th><th>occ              </th><th>occ2              </th><th>ind              </th><th>ind2              </th><th>random            </th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>type   </td><td>real              </td><td>real              </td><td>int               </td><td>int                 </td><td>int                </td><td>int                </td><td>int                </td><td>int                </td><td>int               </td><td>int                </td><td>int                </td><td>int                </td><td>real              </td><td>real              </td><td>real              </td><td>real              </td><td>int              </td><td>int               </td><td>int              </td><td>int               </td><td>int               </td></tr>\n",
       "<tr><td>mins   </td><td>3.021978021978022 </td><td>1.1059115911497213</td><td>0.0               </td><td>0.0                 </td><td>0.0                </td><td>0.0                </td><td>0.0                </td><td>0.0                </td><td>0.0               </td><td>0.0                </td><td>0.0                </td><td>0.0                </td><td>0.0               </td><td>0.0               </td><td>0.0               </td><td>0.0               </td><td>10.0             </td><td>1.0               </td><td>370.0            </td><td>2.0               </td><td>0.0               </td></tr>\n",
       "<tr><td>mean   </td><td>23.465417731467202</td><td>2.969427791239379 </td><td>0.446918694976696 </td><td>0.023562920766442258</td><td>0.24702226825479026</td><td>0.2780942516830658 </td><td>0.3125323666494045 </td><td>0.13878819264629724</td><td>0.2553081305023304</td><td>0.29829104091144487</td><td>0.21569135163127914</td><td>0.23070947695494562</td><td>13.672190574831728</td><td>2.9923032107716137</td><td>8.15417316157433  </td><td>24.849760334671146</td><td>5243.418436043504</td><td>11.69135163127914 </td><td>6667.996116002073</td><td>13.333764888658706</td><td>1914.2263076126374</td></tr>\n",
       "<tr><td>maxs   </td><td>528.845673076923  </td><td>6.270696655981913 </td><td>1.0               </td><td>1.0                 </td><td>1.0                </td><td>1.0                </td><td>1.0                </td><td>1.0                </td><td>1.0               </td><td>1.0                </td><td>1.0                </td><td>1.0                </td><td>47.0              </td><td>22.09             </td><td>103.823           </td><td>487.9681          </td><td>100000.0         </td><td>22.0              </td><td>100000.0         </td><td>22.0              </td><td>3825.0            </td></tr>\n",
       "<tr><td>sigma  </td><td>21.430085743506766</td><td>0.5750893708933925</td><td>0.4972387709704293</td><td>0.15170256601034887 </td><td>0.4313356487428773 </td><td>0.44811810407097313</td><td>0.46358551980855045</td><td>0.3457701367976389 </td><td>0.436090737870262 </td><td>0.45756716236312467</td><td>0.4113543571812672 </td><td>0.4213414081729523 </td><td>10.598613687032655</td><td>3.987480265404191 </td><td>14.424744487903597</td><td>53.27996156215637 </td><td>11579.91114662104</td><td>6.9783416585657125</td><td>5588.264282354911</td><td>5.691380293915183 </td><td>1104.7025392074318</td></tr>\n",
       "<tr><td>zeros  </td><td>0                 </td><td>0                 </td><td>2136              </td><td>3771                </td><td>2908               </td><td>2788               </td><td>2655               </td><td>3326               </td><td>2876              </td><td>2710               </td><td>3029               </td><td>2971               </td><td>48                </td><td>48                </td><td>48                </td><td>48                </td><td>0                </td><td>0                 </td><td>0                </td><td>0                 </td><td>3                 </td></tr>\n",
       "<tr><td>missing</td><td>0                 </td><td>0                 </td><td>0                 </td><td>0                   </td><td>0                  </td><td>0                  </td><td>0                  </td><td>0                  </td><td>0                 </td><td>0                  </td><td>0                  </td><td>0                  </td><td>0                 </td><td>0                 </td><td>0                 </td><td>0                 </td><td>0                </td><td>0                 </td><td>0                </td><td>0                 </td><td>0                 </td></tr>\n",
       "<tr><td>0      </td><td>26.442307692307693</td><td>3.274965291519244 </td><td>1.0               </td><td>0.0                 </td><td>1.0                </td><td>0.0                </td><td>0.0                </td><td>0.0                </td><td>0.0               </td><td>0.0                </td><td>0.0                </td><td>1.0                </td><td>29.0              </td><td>8.41              </td><td>24.389            </td><td>70.7281           </td><td>340.0            </td><td>1.0               </td><td>8660.0           </td><td>20.0              </td><td>0.0               </td></tr>\n",
       "<tr><td>1      </td><td>19.23076923076923 </td><td>2.9565115604007097</td><td>0.0               </td><td>0.0                 </td><td>0.0                </td><td>1.0                </td><td>0.0                </td><td>0.0                </td><td>0.0               </td><td>0.0                </td><td>0.0                </td><td>1.0                </td><td>33.5              </td><td>11.2225           </td><td>37.595375         </td><td>125.94450625      </td><td>9620.0           </td><td>22.0              </td><td>1870.0           </td><td>5.0               </td><td>0.0               </td></tr>\n",
       "<tr><td>2      </td><td>48.07692307692308 </td><td>3.872802292274865 </td><td>1.0               </td><td>0.0                 </td><td>0.0                </td><td>0.0                </td><td>0.0                </td><td>1.0                </td><td>1.0               </td><td>0.0                </td><td>0.0                </td><td>0.0                </td><td>2.0               </td><td>0.04              </td><td>0.008             </td><td>0.0016            </td><td>3060.0           </td><td>10.0              </td><td>8190.0           </td><td>18.0              </td><td>0.0               </td></tr>\n",
       "<tr><td>3      </td><td>12.01923076923077 </td><td>2.486507931154974 </td><td>0.0               </td><td>0.0                 </td><td>1.0                </td><td>0.0                </td><td>0.0                </td><td>0.0                </td><td>0.0               </td><td>1.0                </td><td>0.0                </td><td>0.0                </td><td>29.0              </td><td>8.41              </td><td>24.389            </td><td>70.7281           </td><td>6440.0           </td><td>19.0              </td><td>770.0            </td><td>4.0               </td><td>2.0               </td></tr>\n",
       "<tr><td>4      </td><td>39.90384615384615 </td><td>3.6864727140833713</td><td>1.0               </td><td>0.0                 </td><td>0.0                </td><td>0.0                </td><td>0.0                </td><td>1.0                </td><td>0.0               </td><td>1.0                </td><td>0.0                </td><td>0.0                </td><td>12.0              </td><td>1.44              </td><td>1.728             </td><td>2.0736            </td><td>1820.0           </td><td>5.0               </td><td>7860.0           </td><td>17.0              </td><td>2.0               </td></tr>\n",
       "<tr><td>5      </td><td>13.157894736842104</td><td>2.5770219386958058</td><td>0.0               </td><td>0.0                 </td><td>1.0                </td><td>0.0                </td><td>0.0                </td><td>0.0                </td><td>0.0               </td><td>1.0                </td><td>0.0                </td><td>0.0                </td><td>11.0              </td><td>1.21              </td><td>1.331             </td><td>1.4641            </td><td>8810.0           </td><td>21.0              </td><td>3895.0           </td><td>6.0               </td><td>3.0               </td></tr>\n",
       "<tr><td>6      </td><td>20.192307692307693</td><td>3.005301724570142 </td><td>0.0               </td><td>1.0                 </td><td>0.0                </td><td>0.0                </td><td>0.0                </td><td>0.0                </td><td>0.0               </td><td>0.0                </td><td>1.0                </td><td>0.0                </td><td>17.0              </td><td>2.89              </td><td>4.913             </td><td>8.3521            </td><td>7200.0           </td><td>20.0              </td><td>8770.0           </td><td>21.0              </td><td>4.0               </td></tr>\n",
       "<tr><td>7      </td><td>12.01923076923077 </td><td>2.486507931154974 </td><td>0.0               </td><td>0.0                 </td><td>1.0                </td><td>0.0                </td><td>0.0                </td><td>0.0                </td><td>0.0               </td><td>1.0                </td><td>0.0                </td><td>0.0                </td><td>7.0               </td><td>0.49              </td><td>0.343             </td><td>0.2401            </td><td>5610.0           </td><td>17.0              </td><td>4265.0           </td><td>7.0               </td><td>5.0               </td></tr>\n",
       "<tr><td>8      </td><td>28.846153846153847</td><td>3.361976668508874 </td><td>1.0               </td><td>1.0                 </td><td>0.0                </td><td>0.0                </td><td>0.0                </td><td>0.0                </td><td>0.0               </td><td>0.0                </td><td>0.0                </td><td>1.0                </td><td>8.0               </td><td>0.64              </td><td>0.512             </td><td>0.4096            </td><td>5240.0           </td><td>17.0              </td><td>6970.0           </td><td>12.0              </td><td>7.0               </td></tr>\n",
       "<tr><td>9      </td><td>34.13461538461539 </td><td>3.530311983328089 </td><td>0.0               </td><td>0.0                 </td><td>1.0                </td><td>0.0                </td><td>0.0                </td><td>0.0                </td><td>0.0               </td><td>1.0                </td><td>0.0                </td><td>0.0                </td><td>38.0              </td><td>14.44             </td><td>54.872            </td><td>208.5136          </td><td>5550.0           </td><td>17.0              </td><td>6370.0           </td><td>10.0              </td><td>7.0               </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# convert data as h2o type\n",
    "train_h = h2o.H2OFrame(train)\n",
    "test_h = h2o.H2OFrame(test)\n",
    "\n",
    "# have a look at the data\n",
    "train_h.describe()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "1f0bfa3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the variables\n",
    "y = 'lwage'\n",
    "\n",
    "data_columns = list(data)\n",
    "no_relev_col = ['wage','occ2', 'ind2', 'random', 'lwage']\n",
    "\n",
    "# This gives us: new_list = ['carrot' , 'lemon']\n",
    "x = [col for col in data_columns if col not in no_relev_col]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "57c48dce",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AutoML progress: |█\n",
      "05:14:21.267: AutoML: XGBoost is not available; skipping it.\n",
      "\n",
      "██████████████████████████████████████████████████████████████| (done) 100%\n",
      "Model Details\n",
      "=============\n",
      "H2OStackedEnsembleEstimator :  Stacked Ensemble\n",
      "Model Key:  StackedEnsemble_AllModels_1_AutoML_2_20220804_51421\n",
      "\n",
      "No model summary for this model\n",
      "\n",
      "ModelMetricsRegressionGLM: stackedensemble\n",
      "** Reported on train data. **\n",
      "\n",
      "MSE: 0.14963411645818037\n",
      "RMSE: 0.38682569260350375\n",
      "MAE: 0.29493414661545647\n",
      "RMSLE: 0.09962964024727533\n",
      "R^2: 0.5474439172253015\n",
      "Mean Residual Deviance: 0.14963411645818037\n",
      "Null degrees of freedom: 3861\n",
      "Residual degrees of freedom: 3855\n",
      "Null deviance: 1276.9399854672324\n",
      "Residual deviance: 577.8869577614926\n",
      "AIC: 3639.772059449712\n",
      "\n",
      "ModelMetricsRegressionGLM: stackedensemble\n",
      "** Reported on cross-validation data. **\n",
      "\n",
      "MSE: 0.21847845950742845\n",
      "RMSE: 0.4674167942077268\n",
      "MAE: 0.35554195972164104\n",
      "RMSLE: 0.11935352477830528\n",
      "R^2: 0.3392298618411601\n",
      "Mean Residual Deviance: 0.21847845950742845\n",
      "Null degrees of freedom: 3861\n",
      "Residual degrees of freedom: 3854\n",
      "Null deviance: 1277.1043650451534\n",
      "Residual deviance: 843.7638106176887\n",
      "AIC: 5103.517182973992\n",
      "\n",
      "Cross-Validation Metrics Summary: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>sd</th>\n",
       "      <th>cv_1_valid</th>\n",
       "      <th>cv_2_valid</th>\n",
       "      <th>cv_3_valid</th>\n",
       "      <th>cv_4_valid</th>\n",
       "      <th>cv_5_valid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mae</td>\n",
       "      <td>0.355537</td>\n",
       "      <td>0.011149</td>\n",
       "      <td>0.358856</td>\n",
       "      <td>0.365665</td>\n",
       "      <td>0.336852</td>\n",
       "      <td>0.354934</td>\n",
       "      <td>0.361377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mean_residual_deviance</td>\n",
       "      <td>0.218658</td>\n",
       "      <td>0.018164</td>\n",
       "      <td>0.232228</td>\n",
       "      <td>0.226281</td>\n",
       "      <td>0.191559</td>\n",
       "      <td>0.208852</td>\n",
       "      <td>0.234372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mse</td>\n",
       "      <td>0.218658</td>\n",
       "      <td>0.018164</td>\n",
       "      <td>0.232228</td>\n",
       "      <td>0.226281</td>\n",
       "      <td>0.191559</td>\n",
       "      <td>0.208852</td>\n",
       "      <td>0.234372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>null_deviance</td>\n",
       "      <td>255.420870</td>\n",
       "      <td>18.944805</td>\n",
       "      <td>270.143950</td>\n",
       "      <td>279.337650</td>\n",
       "      <td>235.192600</td>\n",
       "      <td>252.053120</td>\n",
       "      <td>240.377060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>r2</td>\n",
       "      <td>0.339152</td>\n",
       "      <td>0.024681</td>\n",
       "      <td>0.332046</td>\n",
       "      <td>0.342915</td>\n",
       "      <td>0.367003</td>\n",
       "      <td>0.352354</td>\n",
       "      <td>0.301443</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>residual_deviance</td>\n",
       "      <td>168.744060</td>\n",
       "      <td>13.997013</td>\n",
       "      <td>180.441440</td>\n",
       "      <td>183.513900</td>\n",
       "      <td>148.841520</td>\n",
       "      <td>163.113280</td>\n",
       "      <td>167.810170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>rmse</td>\n",
       "      <td>0.467278</td>\n",
       "      <td>0.019686</td>\n",
       "      <td>0.481901</td>\n",
       "      <td>0.475690</td>\n",
       "      <td>0.437675</td>\n",
       "      <td>0.457003</td>\n",
       "      <td>0.484120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>rmsle</td>\n",
       "      <td>0.119307</td>\n",
       "      <td>0.004619</td>\n",
       "      <td>0.122457</td>\n",
       "      <td>0.121297</td>\n",
       "      <td>0.111312</td>\n",
       "      <td>0.119422</td>\n",
       "      <td>0.122048</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 mean         sd  cv_1_valid  cv_2_valid  \\\n",
       "0                     mae    0.355537   0.011149    0.358856    0.365665   \n",
       "1  mean_residual_deviance    0.218658   0.018164    0.232228    0.226281   \n",
       "2                     mse    0.218658   0.018164    0.232228    0.226281   \n",
       "3           null_deviance  255.420870  18.944805  270.143950  279.337650   \n",
       "4                      r2    0.339152   0.024681    0.332046    0.342915   \n",
       "5       residual_deviance  168.744060  13.997013  180.441440  183.513900   \n",
       "6                    rmse    0.467278   0.019686    0.481901    0.475690   \n",
       "7                   rmsle    0.119307   0.004619    0.122457    0.121297   \n",
       "\n",
       "   cv_3_valid  cv_4_valid  cv_5_valid  \n",
       "0    0.336852    0.354934    0.361377  \n",
       "1    0.191559    0.208852    0.234372  \n",
       "2    0.191559    0.208852    0.234372  \n",
       "3  235.192600  252.053120  240.377060  \n",
       "4    0.367003    0.352354    0.301443  \n",
       "5  148.841520  163.113280  167.810170  \n",
       "6    0.437675    0.457003    0.484120  \n",
       "7    0.111312    0.119422    0.122048  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# run AutoML for 10 base models and a maximal runtime of 100 seconds\n",
    "# Run AutoML for 30 seconds\n",
    "aml = H2OAutoML(max_runtime_secs = 100, max_models = 10, seed = 1)\n",
    "aml.train(x = x, y = y, training_frame = train_h, leaderboard_frame = test_h)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "88df97d7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>model_id                                               </th><th style=\"text-align: right;\">    rmse</th><th style=\"text-align: right;\">     mse</th><th style=\"text-align: right;\">     mae</th><th style=\"text-align: right;\">   rmsle</th><th style=\"text-align: right;\">  mean_residual_deviance</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>StackedEnsemble_AllModels_1_AutoML_1_20220722_170939   </td><td style=\"text-align: right;\">0.47064 </td><td style=\"text-align: right;\">0.221502</td><td style=\"text-align: right;\">0.353943</td><td style=\"text-align: right;\">0.120368</td><td style=\"text-align: right;\">                0.221502</td></tr>\n",
       "<tr><td>GBM_2_AutoML_1_20220722_170939                         </td><td style=\"text-align: right;\">0.471849</td><td style=\"text-align: right;\">0.222642</td><td style=\"text-align: right;\">0.355941</td><td style=\"text-align: right;\">0.12074 </td><td style=\"text-align: right;\">                0.222642</td></tr>\n",
       "<tr><td>GBM_5_AutoML_1_20220722_170939                         </td><td style=\"text-align: right;\">0.47191 </td><td style=\"text-align: right;\">0.222699</td><td style=\"text-align: right;\">0.35582 </td><td style=\"text-align: right;\">0.120501</td><td style=\"text-align: right;\">                0.222699</td></tr>\n",
       "<tr><td>StackedEnsemble_BestOfFamily_1_AutoML_1_20220722_170939</td><td style=\"text-align: right;\">0.473798</td><td style=\"text-align: right;\">0.224484</td><td style=\"text-align: right;\">0.357598</td><td style=\"text-align: right;\">0.121491</td><td style=\"text-align: right;\">                0.224484</td></tr>\n",
       "<tr><td>GBM_3_AutoML_1_20220722_170939                         </td><td style=\"text-align: right;\">0.474585</td><td style=\"text-align: right;\">0.22523 </td><td style=\"text-align: right;\">0.359309</td><td style=\"text-align: right;\">0.121264</td><td style=\"text-align: right;\">                0.22523 </td></tr>\n",
       "<tr><td>GBM_1_AutoML_1_20220722_170939                         </td><td style=\"text-align: right;\">0.478149</td><td style=\"text-align: right;\">0.228626</td><td style=\"text-align: right;\">0.362211</td><td style=\"text-align: right;\">0.122278</td><td style=\"text-align: right;\">                0.228626</td></tr>\n",
       "<tr><td>GBM_4_AutoML_1_20220722_170939                         </td><td style=\"text-align: right;\">0.479072</td><td style=\"text-align: right;\">0.22951 </td><td style=\"text-align: right;\">0.362916</td><td style=\"text-align: right;\">0.122456</td><td style=\"text-align: right;\">                0.22951 </td></tr>\n",
       "<tr><td>GBM_grid_1_AutoML_1_20220722_170939_model_1            </td><td style=\"text-align: right;\">0.48001 </td><td style=\"text-align: right;\">0.23041 </td><td style=\"text-align: right;\">0.36381 </td><td style=\"text-align: right;\">0.122671</td><td style=\"text-align: right;\">                0.23041 </td></tr>\n",
       "<tr><td>XRT_1_AutoML_1_20220722_170939                         </td><td style=\"text-align: right;\">0.491218</td><td style=\"text-align: right;\">0.241295</td><td style=\"text-align: right;\">0.375089</td><td style=\"text-align: right;\">0.125318</td><td style=\"text-align: right;\">                0.241295</td></tr>\n",
       "<tr><td>DRF_1_AutoML_1_20220722_170939                         </td><td style=\"text-align: right;\">0.50224 </td><td style=\"text-align: right;\">0.252245</td><td style=\"text-align: right;\">0.382606</td><td style=\"text-align: right;\">0.12847 </td><td style=\"text-align: right;\">                0.252245</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# AutoML Leaderboard\n",
    "lb = aml.leaderboard\n",
    "print(lb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d295cdc1",
   "metadata": {},
   "source": [
    "We see that two Stacked Ensembles are at the top of the leaderboard. Stacked Ensembles often outperform a single model. The out-of-sample (test) MSE of the leading model is given by"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "800f6aba",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.22150159010610537"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aml.leaderboard['mse'][0,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c9c7ec9",
   "metadata": {},
   "source": [
    "The in-sample performance can be evaluated by"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "7e47ac17",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Details\n",
      "=============\n",
      "H2OStackedEnsembleEstimator :  Stacked Ensemble\n",
      "Model Key:  StackedEnsemble_AllModels_1_AutoML_1_20220722_170939\n",
      "\n",
      "No model summary for this model\n",
      "\n",
      "ModelMetricsRegressionGLM: stackedensemble\n",
      "** Reported on train data. **\n",
      "\n",
      "MSE: 0.15027004489538795\n",
      "RMSE: 0.38764680431468534\n",
      "MAE: 0.2955371675604367\n",
      "RMSLE: 0.09990016533775842\n",
      "R^2: 0.5455206039510314\n",
      "Mean Residual Deviance: 0.15027004489538795\n",
      "Null degrees of freedom: 3861\n",
      "Residual degrees of freedom: 3854\n",
      "Null deviance: 1276.9399854672324\n",
      "Residual deviance: 580.3429133859883\n",
      "AIC: 3658.1503537307485\n",
      "\n",
      "ModelMetricsRegressionGLM: stackedensemble\n",
      "** Reported on cross-validation data. **\n",
      "\n",
      "MSE: 0.2185070080606413\n",
      "RMSE: 0.46744733185744175\n",
      "MAE: 0.35554501482828577\n",
      "RMSLE: 0.11935774800872184\n",
      "R^2: 0.3391435190891413\n",
      "Mean Residual Deviance: 0.2185070080606413\n",
      "Null degrees of freedom: 3861\n",
      "Residual degrees of freedom: 3853\n",
      "Null deviance: 1277.1043650451534\n",
      "Residual deviance: 843.8740651301968\n",
      "AIC: 5106.021797066896\n",
      "\n",
      "Cross-Validation Metrics Summary: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>sd</th>\n",
       "      <th>cv_1_valid</th>\n",
       "      <th>cv_2_valid</th>\n",
       "      <th>cv_3_valid</th>\n",
       "      <th>cv_4_valid</th>\n",
       "      <th>cv_5_valid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mae</td>\n",
       "      <td>0.355575</td>\n",
       "      <td>0.011150</td>\n",
       "      <td>0.358694</td>\n",
       "      <td>0.365597</td>\n",
       "      <td>0.336775</td>\n",
       "      <td>0.355430</td>\n",
       "      <td>0.361377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mean_residual_deviance</td>\n",
       "      <td>0.218685</td>\n",
       "      <td>0.018149</td>\n",
       "      <td>0.232207</td>\n",
       "      <td>0.226213</td>\n",
       "      <td>0.191451</td>\n",
       "      <td>0.209182</td>\n",
       "      <td>0.234372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mse</td>\n",
       "      <td>0.218685</td>\n",
       "      <td>0.018149</td>\n",
       "      <td>0.232207</td>\n",
       "      <td>0.226213</td>\n",
       "      <td>0.191451</td>\n",
       "      <td>0.209182</td>\n",
       "      <td>0.234372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>null_deviance</td>\n",
       "      <td>255.420870</td>\n",
       "      <td>18.944805</td>\n",
       "      <td>270.143950</td>\n",
       "      <td>279.337650</td>\n",
       "      <td>235.192600</td>\n",
       "      <td>252.053120</td>\n",
       "      <td>240.377060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>r2</td>\n",
       "      <td>0.339071</td>\n",
       "      <td>0.024654</td>\n",
       "      <td>0.332108</td>\n",
       "      <td>0.343113</td>\n",
       "      <td>0.367360</td>\n",
       "      <td>0.351331</td>\n",
       "      <td>0.301443</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>residual_deviance</td>\n",
       "      <td>168.764340</td>\n",
       "      <td>13.983558</td>\n",
       "      <td>180.424550</td>\n",
       "      <td>183.458680</td>\n",
       "      <td>148.757460</td>\n",
       "      <td>163.370830</td>\n",
       "      <td>167.810170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>rmse</td>\n",
       "      <td>0.467306</td>\n",
       "      <td>0.019674</td>\n",
       "      <td>0.481878</td>\n",
       "      <td>0.475618</td>\n",
       "      <td>0.437551</td>\n",
       "      <td>0.457364</td>\n",
       "      <td>0.484120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>rmsle</td>\n",
       "      <td>0.119310</td>\n",
       "      <td>0.004632</td>\n",
       "      <td>0.122437</td>\n",
       "      <td>0.121279</td>\n",
       "      <td>0.111271</td>\n",
       "      <td>0.119514</td>\n",
       "      <td>0.122048</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 mean         sd  cv_1_valid  cv_2_valid  \\\n",
       "0                     mae    0.355575   0.011150    0.358694    0.365597   \n",
       "1  mean_residual_deviance    0.218685   0.018149    0.232207    0.226213   \n",
       "2                     mse    0.218685   0.018149    0.232207    0.226213   \n",
       "3           null_deviance  255.420870  18.944805  270.143950  279.337650   \n",
       "4                      r2    0.339071   0.024654    0.332108    0.343113   \n",
       "5       residual_deviance  168.764340  13.983558  180.424550  183.458680   \n",
       "6                    rmse    0.467306   0.019674    0.481878    0.475618   \n",
       "7                   rmsle    0.119310   0.004632    0.122437    0.121279   \n",
       "\n",
       "   cv_3_valid  cv_4_valid  cv_5_valid  \n",
       "0    0.336775    0.355430    0.361377  \n",
       "1    0.191451    0.209182    0.234372  \n",
       "2    0.191451    0.209182    0.234372  \n",
       "3  235.192600  252.053120  240.377060  \n",
       "4    0.367360    0.351331    0.301443  \n",
       "5  148.757460  163.370830  167.810170  \n",
       "6    0.437551    0.457364    0.484120  \n",
       "7    0.111271    0.119514    0.122048  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aml.leader"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca0b36af",
   "metadata": {
    "papermill": {
     "duration": 0.027663,
     "end_time": "2021-03-24T11:25:13.491063",
     "exception": false,
     "start_time": "2021-03-24T11:25:13.463400",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "This is in line with our previous results. To understand how the ensemble works, let's take a peek inside the Stacked Ensemble \"All Models\" model.  The \"All Models\" ensemble is an ensemble of all of the individual models in the AutoML run.  This is often the top performing model on the leaderboard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "95549783",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_ids = h2o.as_list(aml.leaderboard['model_id'][0], use_pandas=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "c2236931",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'StackedEnsemble_AllModels_1_AutoML_1_20220722_170939'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = model_ids[model_ids['model_id'].str.contains(\"StackedEnsemble_AllModels\")].values.tolist()\n",
    "model_id = model[0][0]\n",
    "model_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "615b33c4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Details\n",
      "=============\n",
      "H2OStackedEnsembleEstimator :  Stacked Ensemble\n",
      "Model Key:  StackedEnsemble_AllModels_1_AutoML_1_20220722_170939\n",
      "\n",
      "No model summary for this model\n",
      "\n",
      "ModelMetricsRegressionGLM: stackedensemble\n",
      "** Reported on train data. **\n",
      "\n",
      "MSE: 0.15027004489538795\n",
      "RMSE: 0.38764680431468534\n",
      "MAE: 0.2955371675604367\n",
      "RMSLE: 0.09990016533775842\n",
      "R^2: 0.5455206039510314\n",
      "Mean Residual Deviance: 0.15027004489538795\n",
      "Null degrees of freedom: 3861\n",
      "Residual degrees of freedom: 3854\n",
      "Null deviance: 1276.9399854672324\n",
      "Residual deviance: 580.3429133859883\n",
      "AIC: 3658.1503537307485\n",
      "\n",
      "ModelMetricsRegressionGLM: stackedensemble\n",
      "** Reported on cross-validation data. **\n",
      "\n",
      "MSE: 0.2185070080606413\n",
      "RMSE: 0.46744733185744175\n",
      "MAE: 0.35554501482828577\n",
      "RMSLE: 0.11935774800872184\n",
      "R^2: 0.3391435190891413\n",
      "Mean Residual Deviance: 0.2185070080606413\n",
      "Null degrees of freedom: 3861\n",
      "Residual degrees of freedom: 3853\n",
      "Null deviance: 1277.1043650451534\n",
      "Residual deviance: 843.8740651301968\n",
      "AIC: 5106.021797066896\n",
      "\n",
      "Cross-Validation Metrics Summary: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>sd</th>\n",
       "      <th>cv_1_valid</th>\n",
       "      <th>cv_2_valid</th>\n",
       "      <th>cv_3_valid</th>\n",
       "      <th>cv_4_valid</th>\n",
       "      <th>cv_5_valid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mae</td>\n",
       "      <td>0.355575</td>\n",
       "      <td>0.011150</td>\n",
       "      <td>0.358694</td>\n",
       "      <td>0.365597</td>\n",
       "      <td>0.336775</td>\n",
       "      <td>0.355430</td>\n",
       "      <td>0.361377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mean_residual_deviance</td>\n",
       "      <td>0.218685</td>\n",
       "      <td>0.018149</td>\n",
       "      <td>0.232207</td>\n",
       "      <td>0.226213</td>\n",
       "      <td>0.191451</td>\n",
       "      <td>0.209182</td>\n",
       "      <td>0.234372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mse</td>\n",
       "      <td>0.218685</td>\n",
       "      <td>0.018149</td>\n",
       "      <td>0.232207</td>\n",
       "      <td>0.226213</td>\n",
       "      <td>0.191451</td>\n",
       "      <td>0.209182</td>\n",
       "      <td>0.234372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>null_deviance</td>\n",
       "      <td>255.420870</td>\n",
       "      <td>18.944805</td>\n",
       "      <td>270.143950</td>\n",
       "      <td>279.337650</td>\n",
       "      <td>235.192600</td>\n",
       "      <td>252.053120</td>\n",
       "      <td>240.377060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>r2</td>\n",
       "      <td>0.339071</td>\n",
       "      <td>0.024654</td>\n",
       "      <td>0.332108</td>\n",
       "      <td>0.343113</td>\n",
       "      <td>0.367360</td>\n",
       "      <td>0.351331</td>\n",
       "      <td>0.301443</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>residual_deviance</td>\n",
       "      <td>168.764340</td>\n",
       "      <td>13.983558</td>\n",
       "      <td>180.424550</td>\n",
       "      <td>183.458680</td>\n",
       "      <td>148.757460</td>\n",
       "      <td>163.370830</td>\n",
       "      <td>167.810170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>rmse</td>\n",
       "      <td>0.467306</td>\n",
       "      <td>0.019674</td>\n",
       "      <td>0.481878</td>\n",
       "      <td>0.475618</td>\n",
       "      <td>0.437551</td>\n",
       "      <td>0.457364</td>\n",
       "      <td>0.484120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>rmsle</td>\n",
       "      <td>0.119310</td>\n",
       "      <td>0.004632</td>\n",
       "      <td>0.122437</td>\n",
       "      <td>0.121279</td>\n",
       "      <td>0.111271</td>\n",
       "      <td>0.119514</td>\n",
       "      <td>0.122048</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 mean         sd  cv_1_valid  cv_2_valid  \\\n",
       "0                     mae    0.355575   0.011150    0.358694    0.365597   \n",
       "1  mean_residual_deviance    0.218685   0.018149    0.232207    0.226213   \n",
       "2                     mse    0.218685   0.018149    0.232207    0.226213   \n",
       "3           null_deviance  255.420870  18.944805  270.143950  279.337650   \n",
       "4                      r2    0.339071   0.024654    0.332108    0.343113   \n",
       "5       residual_deviance  168.764340  13.983558  180.424550  183.458680   \n",
       "6                    rmse    0.467306   0.019674    0.481878    0.475618   \n",
       "7                   rmsle    0.119310   0.004632    0.122437    0.121279   \n",
       "\n",
       "   cv_3_valid  cv_4_valid  cv_5_valid  \n",
       "0    0.336775    0.355430    0.361377  \n",
       "1    0.191451    0.209182    0.234372  \n",
       "2    0.191451    0.209182    0.234372  \n",
       "3  235.192600  252.053120  240.377060  \n",
       "4    0.367360    0.351331    0.301443  \n",
       "5  148.757460  163.370830  167.810170  \n",
       "6    0.437551    0.457364    0.484120  \n",
       "7    0.111271    0.119514    0.122048  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "se = h2o.get_model(model_id)\n",
    "se"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "439e8999",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Details\n",
      "=============\n",
      "H2OGeneralizedLinearEstimator :  Generalized Linear Modeling\n",
      "Model Key:  metalearner_AUTO_StackedEnsemble_AllModels_1_AutoML_1_20220722_170939\n",
      "\n",
      "\n",
      "GLM Model: summary\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>family</th>\n",
       "      <th>link</th>\n",
       "      <th>regularization</th>\n",
       "      <th>lambda_search</th>\n",
       "      <th>number_of_predictors_total</th>\n",
       "      <th>number_of_active_predictors</th>\n",
       "      <th>number_of_iterations</th>\n",
       "      <th>training_frame</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td>gaussian</td>\n",
       "      <td>identity</td>\n",
       "      <td>Elastic Net (alpha = 0.5, lambda = 0.002551 )</td>\n",
       "      <td>nlambda = 100, lambda.max = 0.2219, lambda.min = 0.002551, lambda....</td>\n",
       "      <td>10</td>\n",
       "      <td>7</td>\n",
       "      <td>49</td>\n",
       "      <td>levelone_training_StackedEnsemble_AllModels_1_AutoML_1_20220722_17...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       family      link                                 regularization  \\\n",
       "0    gaussian  identity  Elastic Net (alpha = 0.5, lambda = 0.002551 )   \n",
       "\n",
       "                                                           lambda_search  \\\n",
       "0  nlambda = 100, lambda.max = 0.2219, lambda.min = 0.002551, lambda....   \n",
       "\n",
       "   number_of_predictors_total number_of_active_predictors  \\\n",
       "0                          10                           7   \n",
       "\n",
       "   number_of_iterations  \\\n",
       "0                    49   \n",
       "\n",
       "                                                          training_frame  \n",
       "0  levelone_training_StackedEnsemble_AllModels_1_AutoML_1_20220722_17...  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "ModelMetricsRegressionGLM: glm\n",
      "** Reported on train data. **\n",
      "\n",
      "MSE: 0.21727419974117612\n",
      "RMSE: 0.4661268065035266\n",
      "MAE: 0.35448759076481845\n",
      "RMSLE: 0.11902010738109643\n",
      "R^2: 0.3428720464937892\n",
      "Mean Residual Deviance: 0.21727419974117612\n",
      "Null degrees of freedom: 3861\n",
      "Residual degrees of freedom: 3854\n",
      "Null deviance: 1276.9399854672324\n",
      "Residual deviance: 839.1129594004221\n",
      "AIC: 5082.170839083037\n",
      "\n",
      "ModelMetricsRegressionGLM: glm\n",
      "** Reported on cross-validation data. **\n",
      "\n",
      "MSE: 0.2185070080606413\n",
      "RMSE: 0.46744733185744175\n",
      "MAE: 0.35554501482828577\n",
      "RMSLE: 0.11935774800872184\n",
      "R^2: 0.3391435190891413\n",
      "Mean Residual Deviance: 0.2185070080606413\n",
      "Null degrees of freedom: 3861\n",
      "Residual degrees of freedom: 3853\n",
      "Null deviance: 1277.1043650451534\n",
      "Residual deviance: 843.8740651301968\n",
      "AIC: 5106.021797066896\n",
      "\n",
      "Cross-Validation Metrics Summary: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>sd</th>\n",
       "      <th>cv_1_valid</th>\n",
       "      <th>cv_2_valid</th>\n",
       "      <th>cv_3_valid</th>\n",
       "      <th>cv_4_valid</th>\n",
       "      <th>cv_5_valid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mae</td>\n",
       "      <td>0.355575</td>\n",
       "      <td>0.011150</td>\n",
       "      <td>0.358694</td>\n",
       "      <td>0.365597</td>\n",
       "      <td>0.336775</td>\n",
       "      <td>0.355430</td>\n",
       "      <td>0.361377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mean_residual_deviance</td>\n",
       "      <td>0.218685</td>\n",
       "      <td>0.018149</td>\n",
       "      <td>0.232207</td>\n",
       "      <td>0.226213</td>\n",
       "      <td>0.191451</td>\n",
       "      <td>0.209182</td>\n",
       "      <td>0.234372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mse</td>\n",
       "      <td>0.218685</td>\n",
       "      <td>0.018149</td>\n",
       "      <td>0.232207</td>\n",
       "      <td>0.226213</td>\n",
       "      <td>0.191451</td>\n",
       "      <td>0.209182</td>\n",
       "      <td>0.234372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>null_deviance</td>\n",
       "      <td>255.420870</td>\n",
       "      <td>18.944805</td>\n",
       "      <td>270.143950</td>\n",
       "      <td>279.337650</td>\n",
       "      <td>235.192600</td>\n",
       "      <td>252.053120</td>\n",
       "      <td>240.377060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>r2</td>\n",
       "      <td>0.339071</td>\n",
       "      <td>0.024654</td>\n",
       "      <td>0.332108</td>\n",
       "      <td>0.343113</td>\n",
       "      <td>0.367360</td>\n",
       "      <td>0.351331</td>\n",
       "      <td>0.301443</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>residual_deviance</td>\n",
       "      <td>168.764340</td>\n",
       "      <td>13.983558</td>\n",
       "      <td>180.424550</td>\n",
       "      <td>183.458680</td>\n",
       "      <td>148.757460</td>\n",
       "      <td>163.370830</td>\n",
       "      <td>167.810170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>rmse</td>\n",
       "      <td>0.467306</td>\n",
       "      <td>0.019674</td>\n",
       "      <td>0.481878</td>\n",
       "      <td>0.475618</td>\n",
       "      <td>0.437551</td>\n",
       "      <td>0.457364</td>\n",
       "      <td>0.484120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>rmsle</td>\n",
       "      <td>0.119310</td>\n",
       "      <td>0.004632</td>\n",
       "      <td>0.122437</td>\n",
       "      <td>0.121279</td>\n",
       "      <td>0.111271</td>\n",
       "      <td>0.119514</td>\n",
       "      <td>0.122048</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 mean         sd  cv_1_valid  cv_2_valid  \\\n",
       "0                     mae    0.355575   0.011150    0.358694    0.365597   \n",
       "1  mean_residual_deviance    0.218685   0.018149    0.232207    0.226213   \n",
       "2                     mse    0.218685   0.018149    0.232207    0.226213   \n",
       "3           null_deviance  255.420870  18.944805  270.143950  279.337650   \n",
       "4                      r2    0.339071   0.024654    0.332108    0.343113   \n",
       "5       residual_deviance  168.764340  13.983558  180.424550  183.458680   \n",
       "6                    rmse    0.467306   0.019674    0.481878    0.475618   \n",
       "7                   rmsle    0.119310   0.004632    0.122437    0.121279   \n",
       "\n",
       "   cv_3_valid  cv_4_valid  cv_5_valid  \n",
       "0    0.336775    0.355430    0.361377  \n",
       "1    0.191451    0.209182    0.234372  \n",
       "2    0.191451    0.209182    0.234372  \n",
       "3  235.192600  252.053120  240.377060  \n",
       "4    0.367360    0.351331    0.301443  \n",
       "5  148.757460  163.370830  167.810170  \n",
       "6    0.437551    0.457364    0.484120  \n",
       "7    0.111271    0.119514    0.122048  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Scoring History: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>duration</th>\n",
       "      <th>iteration</th>\n",
       "      <th>lambda</th>\n",
       "      <th>predictors</th>\n",
       "      <th>deviance_train</th>\n",
       "      <th>deviance_xval</th>\n",
       "      <th>deviance_se</th>\n",
       "      <th>alpha</th>\n",
       "      <th>iterations</th>\n",
       "      <th>training_rmse</th>\n",
       "      <th>training_deviance</th>\n",
       "      <th>training_mae</th>\n",
       "      <th>training_r2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.000 sec</td>\n",
       "      <td>1</td>\n",
       "      <td>,22E0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.330642</td>\n",
       "      <td>0.330652</td>\n",
       "      <td>0.008216</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.000 sec</td>\n",
       "      <td>2</td>\n",
       "      <td>,2E0</td>\n",
       "      <td>5</td>\n",
       "      <td>0.316066</td>\n",
       "      <td>0.330609</td>\n",
       "      <td>0.008220</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.000 sec</td>\n",
       "      <td>3</td>\n",
       "      <td>,18E0</td>\n",
       "      <td>6</td>\n",
       "      <td>0.302049</td>\n",
       "      <td>0.330627</td>\n",
       "      <td>0.008216</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.000 sec</td>\n",
       "      <td>4</td>\n",
       "      <td>,17E0</td>\n",
       "      <td>6</td>\n",
       "      <td>0.289848</td>\n",
       "      <td>0.322157</td>\n",
       "      <td>0.008512</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.016 sec</td>\n",
       "      <td>5</td>\n",
       "      <td>,15E0</td>\n",
       "      <td>7</td>\n",
       "      <td>0.279287</td>\n",
       "      <td>0.307694</td>\n",
       "      <td>0.008407</td>\n",
       "      <td>0.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.528476</td>\n",
       "      <td>0.279287</td>\n",
       "      <td>0.407732</td>\n",
       "      <td>0.155318</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.016 sec</td>\n",
       "      <td>6</td>\n",
       "      <td>,14E0</td>\n",
       "      <td>7</td>\n",
       "      <td>0.270007</td>\n",
       "      <td>0.294801</td>\n",
       "      <td>0.008323</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.016 sec</td>\n",
       "      <td>7</td>\n",
       "      <td>,13E0</td>\n",
       "      <td>7</td>\n",
       "      <td>0.262086</td>\n",
       "      <td>0.283621</td>\n",
       "      <td>0.008253</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.016 sec</td>\n",
       "      <td>8</td>\n",
       "      <td>,12E0</td>\n",
       "      <td>7</td>\n",
       "      <td>0.255335</td>\n",
       "      <td>0.273875</td>\n",
       "      <td>0.008226</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.016 sec</td>\n",
       "      <td>9</td>\n",
       "      <td>,11E0</td>\n",
       "      <td>8</td>\n",
       "      <td>0.249586</td>\n",
       "      <td>0.265471</td>\n",
       "      <td>0.008206</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.016 sec</td>\n",
       "      <td>10</td>\n",
       "      <td>,96E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.244598</td>\n",
       "      <td>0.258304</td>\n",
       "      <td>0.008188</td>\n",
       "      <td>0.5</td>\n",
       "      <td>10.0</td>\n",
       "      <td>0.494568</td>\n",
       "      <td>0.244598</td>\n",
       "      <td>0.377588</td>\n",
       "      <td>0.260234</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.032 sec</td>\n",
       "      <td>11</td>\n",
       "      <td>,88E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.240460</td>\n",
       "      <td>0.252204</td>\n",
       "      <td>0.008175</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.032 sec</td>\n",
       "      <td>12</td>\n",
       "      <td>,8E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.236926</td>\n",
       "      <td>0.246950</td>\n",
       "      <td>0.008189</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.032 sec</td>\n",
       "      <td>13</td>\n",
       "      <td>,73E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.233943</td>\n",
       "      <td>0.242530</td>\n",
       "      <td>0.008194</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.032 sec</td>\n",
       "      <td>14</td>\n",
       "      <td>,66E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.231423</td>\n",
       "      <td>0.238774</td>\n",
       "      <td>0.008196</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.032 sec</td>\n",
       "      <td>15</td>\n",
       "      <td>,6E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.229293</td>\n",
       "      <td>0.235608</td>\n",
       "      <td>0.008204</td>\n",
       "      <td>0.5</td>\n",
       "      <td>15.0</td>\n",
       "      <td>0.478846</td>\n",
       "      <td>0.229293</td>\n",
       "      <td>0.364087</td>\n",
       "      <td>0.306521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.032 sec</td>\n",
       "      <td>16</td>\n",
       "      <td>,55E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.227495</td>\n",
       "      <td>0.232935</td>\n",
       "      <td>0.008210</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.032 sec</td>\n",
       "      <td>17</td>\n",
       "      <td>,5E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.226154</td>\n",
       "      <td>0.230682</td>\n",
       "      <td>0.008216</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.047 sec</td>\n",
       "      <td>18</td>\n",
       "      <td>,46E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.224642</td>\n",
       "      <td>0.228783</td>\n",
       "      <td>0.008220</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.047 sec</td>\n",
       "      <td>19</td>\n",
       "      <td>,42E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.223737</td>\n",
       "      <td>0.227337</td>\n",
       "      <td>0.008211</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.047 sec</td>\n",
       "      <td>20</td>\n",
       "      <td>,38E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.222642</td>\n",
       "      <td>0.225825</td>\n",
       "      <td>0.008238</td>\n",
       "      <td>0.5</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.471849</td>\n",
       "      <td>0.222642</td>\n",
       "      <td>0.358377</td>\n",
       "      <td>0.326639</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                timestamp    duration  iteration  lambda  predictors  \\\n",
       "0     2022-07-22 17:10:18   0.000 sec          1   ,22E0           1   \n",
       "1     2022-07-22 17:10:18   0.000 sec          2    ,2E0           5   \n",
       "2     2022-07-22 17:10:18   0.000 sec          3   ,18E0           6   \n",
       "3     2022-07-22 17:10:18   0.000 sec          4   ,17E0           6   \n",
       "4     2022-07-22 17:10:18   0.016 sec          5   ,15E0           7   \n",
       "5     2022-07-22 17:10:18   0.016 sec          6   ,14E0           7   \n",
       "6     2022-07-22 17:10:18   0.016 sec          7   ,13E0           7   \n",
       "7     2022-07-22 17:10:18   0.016 sec          8   ,12E0           7   \n",
       "8     2022-07-22 17:10:18   0.016 sec          9   ,11E0           8   \n",
       "9     2022-07-22 17:10:18   0.016 sec         10  ,96E-1           8   \n",
       "10    2022-07-22 17:10:18   0.032 sec         11  ,88E-1           8   \n",
       "11    2022-07-22 17:10:18   0.032 sec         12   ,8E-1           8   \n",
       "12    2022-07-22 17:10:18   0.032 sec         13  ,73E-1           8   \n",
       "13    2022-07-22 17:10:18   0.032 sec         14  ,66E-1           8   \n",
       "14    2022-07-22 17:10:18   0.032 sec         15   ,6E-1           8   \n",
       "15    2022-07-22 17:10:18   0.032 sec         16  ,55E-1           8   \n",
       "16    2022-07-22 17:10:18   0.032 sec         17   ,5E-1           8   \n",
       "17    2022-07-22 17:10:18   0.047 sec         18  ,46E-1           8   \n",
       "18    2022-07-22 17:10:18   0.047 sec         19  ,42E-1           8   \n",
       "19    2022-07-22 17:10:18   0.047 sec         20  ,38E-1           8   \n",
       "\n",
       "    deviance_train  deviance_xval  deviance_se  alpha  iterations  \\\n",
       "0         0.330642       0.330652     0.008216    0.5         NaN   \n",
       "1         0.316066       0.330609     0.008220    0.5         NaN   \n",
       "2         0.302049       0.330627     0.008216    0.5         NaN   \n",
       "3         0.289848       0.322157     0.008512    0.5         NaN   \n",
       "4         0.279287       0.307694     0.008407    0.5         5.0   \n",
       "5         0.270007       0.294801     0.008323    0.5         NaN   \n",
       "6         0.262086       0.283621     0.008253    0.5         NaN   \n",
       "7         0.255335       0.273875     0.008226    0.5         NaN   \n",
       "8         0.249586       0.265471     0.008206    0.5         NaN   \n",
       "9         0.244598       0.258304     0.008188    0.5        10.0   \n",
       "10        0.240460       0.252204     0.008175    0.5         NaN   \n",
       "11        0.236926       0.246950     0.008189    0.5         NaN   \n",
       "12        0.233943       0.242530     0.008194    0.5         NaN   \n",
       "13        0.231423       0.238774     0.008196    0.5         NaN   \n",
       "14        0.229293       0.235608     0.008204    0.5        15.0   \n",
       "15        0.227495       0.232935     0.008210    0.5         NaN   \n",
       "16        0.226154       0.230682     0.008216    0.5         NaN   \n",
       "17        0.224642       0.228783     0.008220    0.5         NaN   \n",
       "18        0.223737       0.227337     0.008211    0.5         NaN   \n",
       "19        0.222642       0.225825     0.008238    0.5        20.0   \n",
       "\n",
       "   training_rmse training_deviance training_mae training_r2  \n",
       "0                                                            \n",
       "1                                                            \n",
       "2                                                            \n",
       "3                                                            \n",
       "4       0.528476          0.279287     0.407732    0.155318  \n",
       "5                                                            \n",
       "6                                                            \n",
       "7                                                            \n",
       "8                                                            \n",
       "9       0.494568          0.244598     0.377588    0.260234  \n",
       "10                                                           \n",
       "11                                                           \n",
       "12                                                           \n",
       "13                                                           \n",
       "14      0.478846          0.229293     0.364087    0.306521  \n",
       "15                                                           \n",
       "16                                                           \n",
       "17                                                           \n",
       "18                                                           \n",
       "19      0.471849          0.222642     0.358377    0.326639  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "See the whole table with table.as_data_frame()\n",
      "\n",
      "Variable Importances: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>variable</th>\n",
       "      <th>relative_importance</th>\n",
       "      <th>scaled_importance</th>\n",
       "      <th>percentage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>GBM_5_AutoML_1_20220722_170939</td>\n",
       "      <td>0.156444</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.459637</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>GBM_3_AutoML_1_20220722_170939</td>\n",
       "      <td>0.064177</td>\n",
       "      <td>0.410220</td>\n",
       "      <td>0.188552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>GBM_1_AutoML_1_20220722_170939</td>\n",
       "      <td>0.045077</td>\n",
       "      <td>0.288132</td>\n",
       "      <td>0.132436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>GBM_grid_1_AutoML_1_20220722_170939_model_1</td>\n",
       "      <td>0.042987</td>\n",
       "      <td>0.274777</td>\n",
       "      <td>0.126298</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>GBM_4_AutoML_1_20220722_170939</td>\n",
       "      <td>0.022251</td>\n",
       "      <td>0.142227</td>\n",
       "      <td>0.065373</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>DeepLearning_1_AutoML_1_20220722_170939</td>\n",
       "      <td>0.009429</td>\n",
       "      <td>0.060268</td>\n",
       "      <td>0.027701</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>DRF_1_AutoML_1_20220722_170939</td>\n",
       "      <td>0.000001</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>0.000003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>GBM_2_AutoML_1_20220722_170939</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>XRT_1_AutoML_1_20220722_170939</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>GLM_1_AutoML_1_20220722_170939</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      variable  relative_importance  \\\n",
       "0               GBM_5_AutoML_1_20220722_170939             0.156444   \n",
       "1               GBM_3_AutoML_1_20220722_170939             0.064177   \n",
       "2               GBM_1_AutoML_1_20220722_170939             0.045077   \n",
       "3  GBM_grid_1_AutoML_1_20220722_170939_model_1             0.042987   \n",
       "4               GBM_4_AutoML_1_20220722_170939             0.022251   \n",
       "5      DeepLearning_1_AutoML_1_20220722_170939             0.009429   \n",
       "6               DRF_1_AutoML_1_20220722_170939             0.000001   \n",
       "7               GBM_2_AutoML_1_20220722_170939             0.000000   \n",
       "8               XRT_1_AutoML_1_20220722_170939             0.000000   \n",
       "9               GLM_1_AutoML_1_20220722_170939             0.000000   \n",
       "\n",
       "   scaled_importance  percentage  \n",
       "0           1.000000    0.459637  \n",
       "1           0.410220    0.188552  \n",
       "2           0.288132    0.132436  \n",
       "3           0.274777    0.126298  \n",
       "4           0.142227    0.065373  \n",
       "5           0.060268    0.027701  \n",
       "6           0.000007    0.000003  \n",
       "7           0.000000    0.000000  \n",
       "8           0.000000    0.000000  \n",
       "9           0.000000    0.000000  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the Stacked Ensemble metalearner model\n",
    "metalearner = se.metalearner()\n",
    "metalearner"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06402906",
   "metadata": {},
   "source": [
    "Examine the variable importance of the metalearner (combiner) algorithm in the ensemble. This shows us how much each base learner is contributing to the ensemble. The AutoML Stacked Ensembles use the default metalearner algorithm (GLM with non-negative weights), so the variable importance of the metalearner is actually the standardized coefficient magnitudes of the GLM."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7425b332",
   "metadata": {},
   "source": [
    "The table above gives us the variable importance of the metalearner in the ensemble. The AutoML Stacked Ensembles use the default metalearner algorithm (GLM with non-negative weights), so the variable importance of the metalearner is actually the standardized coefficient magnitudes of the GLM. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "4d86b390",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Intercept': 2.969427791239395,\n",
       " 'GBM_2_AutoML_1_20220722_170939': 0.0,\n",
       " 'GBM_5_AutoML_1_20220722_170939': 0.1564442116943833,\n",
       " 'GBM_3_AutoML_1_20220722_170939': 0.06417658529927933,\n",
       " 'GBM_1_AutoML_1_20220722_170939': 0.04507655475235933,\n",
       " 'GBM_4_AutoML_1_20220722_170939': 0.02225052417703666,\n",
       " 'GBM_grid_1_AutoML_1_20220722_170939_model_1': 0.0429872925804153,\n",
       " 'XRT_1_AutoML_1_20220722_170939': 0.0,\n",
       " 'DRF_1_AutoML_1_20220722_170939': 1.1143189727895769e-06,\n",
       " 'GLM_1_AutoML_1_20220722_170939': 0.0,\n",
       " 'DeepLearning_1_AutoML_1_20220722_170939': 0.009428554928469175}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metalearner.coef_norm()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "b4160a06",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<h2o.plot._plot_result._MObject at 0x265c10daf70>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABCYAAAJTCAYAAADUlBd3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAACAcklEQVR4nOzde/zu1Zz//8dTB2miEDEVO6NxTEVTzqKQcagcpvYYlDBUXwxjKhmMmflpNOMsOYdJOVROOSVROiCddulImyKTBolMKa/fH2td7XdX1+fzuXZ75+rwuN9u1+3zvtZ7vdd7vQ/XrvV6r7XeqSokSZIkSZJm4XazroAkSZIkSbrtMjAhSZIkSZJmxsCEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJtwFJFiWpJAfNuB6V5JtjaW/s6VvPpFJjkixNsnTW9VgZktwpyTv7MV3Tz/Nms67XzcnN4Xrf3H4DkvSnZmBCkqQJkqyS5MVJvpXkl0n+kOTSJGck+WCSZ4zl36U3LHaZUZU1Y0k2TLJfku8n+dXgnvl6klckWXsG1XoL8P+AJcCbgX8Bfr68hfR7u5L8MclfzJPvmEHeXW5spWfN3/Nko2u7QJ6lPd+iQdqfJXlukk8kOSfJ75JckeTkJK9OsvoCZW6b5JNJfpLk//rv63tJ3pDkzitwPH+Z5K1JThn8O//LJN9J8p9JHjZhm1EQ6Y1TlD+6jyrJt+bJt6j/thY8v9Kt1aqzroAkSTc3SVYBvghsB/waOBK4GLgL8BfA3wL3Bz4/oyre2rwbOBT4yawrcmMleRHtOG4PnA4cAvwKuCvwaODtwD8D6/6Jq/Y04LyqevpKKOsa2v877ga8dnxlko2Bxw3y3VJsM+sK3AY8Bvhv4JfAMcBnaf+ePh34T+CZSbapqv8bbpTk9sAHgb8Dfg98GTgPWAt4AvBGYM8kz6qqY6etTJIAr++f2wGnAJ/s9bsj8BBaQO/VSfasqvfcqKNe5hrgsUnuV1XnTlj/IiDc8n470krjjS9J0g0tpgUlTgceV1WXD1cmWRPYahYVuzWqqsuAy2Zdjxsryd8CH6AFIp5VVUdOyPMoYEUbNzfGnwNTN9gW8D/AJcCuSV5fVdeMrR81rr4I7LCS9nmTq6ofzroOtwE/pwUXPl1VV48Sk9wR+CbwSGAP4L/Gtntv3+4UYIequmiwbfo27wCOTLJlVZ09ZX1eTwtqXAQsrqrjxzMkuTvwSmBl9HQa/SZeBLxmbD+rALsC36P9XtdfCfuTbnEcyiFJ0g09sv89aDwoAVBVV1bVMaPvfc6Ej/SvHxl03b2uO3OSP0/y+iTHJ/l5kquT/Kx3bX7A+D4ymBOiLx+a5LLejfnkJE+bVPEkd+xdky/uec9J8irm+G9+78q8Xy/zF0muSvLjJO9PssGE/FuPujEn2TLJkb3r8/BYV0/yz0l+2Mu7MMm/9aefk+pwg/H1Sb45dh7HP98cK2PVJLsnOSnJb5JcmeTUJHsmucGxp9kzyVn9PP00ybuznMMtesPqXf3rzpOCEgC94XODYFaSbZJ8pZ/D/0tyXr8eE+uR5C5J3pzk7CS/T3J5kqOTPGks3zd7l/AAj5vrvN0IHwDuQeuJMdzfasALgBOAs+ao+8OSvCPJ6YPjPT/Jf2WO7vhJ1k7y9vH7Ocl9Rr+PsfwHje7FJH+fZEnf7n/6PX2D85qxOSYy3e/5oIwNVxhsf91vZI5z8JW0YQy/SRvm84hJxz7Y5v59fxf139P/pP27cb8JeddLG4JwbtpwiV/35YOS3Ge+/dyUquq0qjp4GJTo6VewLBix9XBdkkfTGuy/Ap42DEr0bauq3g3sT+tB8c5p6tLPw+uAq4GnTApK9PIvrarX0oZDraizgBOBF/TfytBTaQGJD6yE/Ui3WPaYkCTphv63//3LKfMfRBvysT3wOeC0wbpf97+PBfamdWM+DPgtsDHwbOAZSR5VVadPKPvewHeBHwEfp3V/3gn4XJJtxwIktweOBv6K1tvjYGAd2hCCx81R92cCL+31OoH2P+sPoj3Ze3qSLarqpxO2ewSwD/Bt4MO0IQpXJwnwqX4ufkgb3rA68EJgkznqMMlBtCep4x5D68J95Sih/4/+F4AnA+cCnwD+D3g8LWiwFfC8sXLeDryc1gPg/cAfep236vW9muk8m3ZNTqqqr82XsaquGn5P8ve0J8K/Az4NXEprnO1FO/ePqqpfD/Lfm3ZOFgHHAV8B/owWJPhKkr+vqlHj5qCe9w3Aj/t3gKVTHtdcDgHeSrs/PjtIfwawHu0ev+8c274Y2BH4FvB1YBXgocCrgKck2ao3VAFIsgbwjZ7nVNr9vDawL+0+mM9baPfDF4Cv0e6FF/e6PWGBbQ9i4d/zckvySNpxrw4cDlwAbEa7Tt+YY5vtet7RPX4BsAHtd/vUJI+vqlN63jWB42nDzY7q+UP7N2R74DO0f0dGZX+T9u/C46vqmzf2uFaCP/S/4z1wXtz/fqCqLpln+/+g9WzYNslGVXXhAvvbldYG+kRVTQyiDU3oGXRjfYD2b+XoWoy8mPbfg0Npv1fptqmq/Pjx48ePHz+DD7A5rWH6R1ow4JnAvRfYZheggF3mWH934I4T0jel/U/pl8fSF/XyCnjD2Lon9/QvjaW/tqcfBtxukL4Rbex00XqBDLdZH7j9hHo9CbgWeO9Y+taDev39hO3+tq87EVhjkH4XWqCigG+ObfPGnr71Auf4IcBvgF8A952w/buAVQbpqwAf6uu2H6Q/sqddANxlkL5Gr3cBS6e8V0bl/9ty3mP3Bq7qx3P/sXUH9DLfP5b+zX5P7jyWvg6t8fx7YL2xdTc43zfyN1HAxX35g7RG5AaD9V8BLgfWBP5t0m+hH/MqE8rereffayz9n3v6IUAG6Rv2e2DS/XxQT/8JcK9B+qq0IS0FbDm2zdLx683Cv+fRfhZNWLd1X/fGQVqAc8bvxb7uFSz7TW09SL8zrbfAZcADx7Z5EO3fjVMGaU/vZbxtQp1WZ+zfn34/Lfi7m3AfFO03N9fn13OdmznK/DIT/j1h2b8XT5yijON73r+bIu83et7dbuRv4Y3j13eevKP76N9ov43Lga8O1q9P+y19oH+/mNYZZIV+r3783BI/DuWQJGlMVZ1KG9f8P/3vYcDSJP+b5Igkyz2RYLVuwVdMSD+d9j/Kj5/QxRfa0+5/G9vmq7SG15ZjeXelNVz/qar+OMh/IXN0c66qn9bYk/ye/jVa9+Mnz3FIp1XV+yak79r/vrYGE9lV1S+Bf52jrAUl+XPaJKSr0Rp2F/T02wF70saw/0NVXTvY57XAq2kNg+dOqOO/93qN8v8frRfI8rhn/3vxcm73d7TG4rur6pyxdfsCVwDP671gSLIp7en2YVV16DBztV4Vb6AFVp61nPW4MT5AC/q8sNft3sATgYOr6sq5NqqqHw+vz8CHaQGa8XvtBbT7eZ+qqkE5F9F6vMznTVV13WSq1Z56f6R/Hf/d/Ck8ErgfcGxVfW5s3btpjfBxz6cFnd5QVT8Yrqj2pP8DwOZJHji23e/HC6qqqyf8+/N84AG0HlnL6w3zfNaetpAke9Lm8zmNdh8MjX5bF7GwUZ4/nyLvPfrfG/QE60OA3jj2eeUUZS6o/zY+ATxxMATohbTfksM4dJvnUA5Jkiaoqk8lOYLWBfzRtF4Uj6ZNYLZDko/RnqbW3KVcX5Kn0oZNbEEb+jD+3+F1aUMLhk6bozF3EW04xajsO9K6qV9Ukyfz+yYTugn3oRfPpT3Z25T2lHaVQZa5hjTM1Zh5KK0x+e056rDckqxFmzxufdpEdScMVv8l7c0X5wOva4dzA7+nNcCGdYQ2pGDccdywS/m81et/p74Pxupwgy78VfWrJKfShv/cnzYsZ3St1540dwFwt/73BvOVrGxV9Z0kS4AXJvk32rCO27FA46oH3v4e2Bl4IK0BO3xItv4g751oQxIuqqqlE4qbdH8NnTwhbdR4vdGvl1wBc95zVXVtkm/TjndodM03neOaj4aaPQD4QS/7p8DeSR4KfInWk2DivyHDwM3yqqqJPzRoc3bQesfMK8kzaQGmn9Mmjf3DHFmn+W0tz+9wvryLuOG/kz9m4UDYtD5A+2/AbkneQOstdEZV3ZjgkHSrYmBCkqQ59P9R/lr/jGZPfxbtyd7zgSO4/jj7OSV5OW32+F/Rxn//hDZPQtGCHZvSXjU57tdzFHkN12/UjZ5S/s8c+X8+R/pbaeOzLwG+SmvYjJ647sLcDYy5ylsb+OUcjYy5tplTP+eH0gJD+1TVJ8ey3LX/3Zj5x2evNVZHmHCueiPxf8fT5/Gz/vcGE4UuYFSHucbOj9LX6X9Hx/nE/pnLWvOsW5k+QOuFsx2tB8r3e0+j+XySNsfEj2hzN/ycNpwF2j04vP/v1P/OdT/PlT7y6wlpo4DTKhPW3dRuzO9zdM1fPGHd0FoAVfWbJA8H/oU258eoB8plSQ6gDTeaq/H/J5VkB9rv+lLaHBc/mpDt57RhaPeizR0zn9Hvb765KEYuoQX8bvD2i2pzbaTXcVWWzX+xUlTVKUlOof1mTqL9+/r/VuY+pFsqAxOSJE2pP3X8VJJNaLO6P4EpAhP9f3D/hfY/2g+tsYncFpqVf0qjt4esN8f6e4wnpL0O7+XAmcAjx7t6J1k8z/7mejJ5OXCXJKtNaATdoA5TeCdt1voPVNV+c+wP4IiqeuaUZQ7P1fUaRD0QclcmdPOew7dp3bG3oc2JMK1RHe7B5LdY3HMs3+jvK6pqqrcP3MQ+Tpt08H20Bt6b5sucZAtaUOLrwF8P740+HOefxjb5Tf871/08V/qfymio1KT/l15nQtpy/z4H22xaVWdMU6mqupj2ND60XilPoL1S8/W0QOby3KM3iSTPoQ1p+DnwhKo6f46s36YFJralBXPnKu/OwMP614lv2BhzPK0n3DbccPjIn8L7gQP75/fAf8+gDtLNjnNMSJK0/EYN+GF35lFX6UlPY9elNVZOmBCUWItl3bxvtB5UuABYP8l4l3AYexVfdx/a/wt8bUJQYoO+fnmd0st89JR1mFOSVwO703qs7D5HtnNoT8cfPsccHXPVESa/qeQxLN+Dm8/QJhZ9RJJt58uY678uddS7YOsJ+dahva3h/4Cze/JJg/rNXJ/X4jO0J9W/o01QOZ/Rmzo+PyFgtSVwh7Hyf0MLGq0/6ZWcTL6/Vqb5fs/Qej5Bm4hz3BYT0ua853owbNLx3OhrXs1ZVfUulvWw2WF5y1nZkvwt7V75GfC4eYIS0CZZBXhRkvkCUf9I623z9Vr4jRzQJi69Bnh2Jryq+U/gE7TfzAbAp2vw5h3ptszAhCRJY5IsTvLE/iR3fN09WNa1+tjBqlH3/3tNKPJS2rCNh/VAxKis1WjDO9ZdKRVvk/vdDviPYd2TbETrGTFuaf/76N44GuVfi9ZV/8b0rBxNMPjv/XWPozLvQutlMpU+/vwtwBLgOTXHK/t6+rtoPQzemeQO43mS3HNsgsCD+t99e71G+dYA3jxtHfv+r2DZuf1kkomThfYu9icOkv6b1k38/yUZf73mv9KGMvz3aGLSqjqZNv/FM5O8cI59bNJ7wcwrybpJ7p9kRe+719F6QTx50sSuY5b2v1uP1eXuwHvm2OZjtPv5zRlMHpJkQ9rQj5vSfL9nWDbHyvWGWfTeVK+YkP8E2nCExybZfmzdntxwfglov6VfA29IcoMJO5PcLsnWg+8PniOIM2rUX29i0iT36vfBmhO2WemSvIDW0+YnwGPnGL5xnao6lmWvSP5iD5aOl/lS2ut1f8vk8z6p3B/SJhReHfhyf43rJOtMU97y6r+V7Wi/nan/TZRu7RzKIUnSDW1F+5/cn/dJ6UZP4TaiDSu4A22M/PBd9CfS/sf/lb2xOxpL/q6qujzJO4G9gSVJPkf7n+LH0/6n+5i+vKL+i/ZU9FnAKUm+ShvbvhMtiPKMYeaq+nmSQ2mTEZ6W5Gs9/xNpT+tPoz25Xx6H9P09AzizH+tqwLOB7zG5ATbJf9Mapd8DXjVhUsulVXVQX/5X2hwdLwWenuQbtKEYd6fNPfEo2psufgBQVccneRdtbPeZST5DCxJsT3sSPs049etU1cE9IPJu4CtJTqM1RH9FGxbyiF6/ywbbLO2z/b+Hdq0+RXsF5uN6/nNoDa6hv6VNlvmhPmfJd2gN1w1or1J9cN/20gWqvCdtPo5/ob368EbpkydOO4Hi92hd6J+Z5ARaN/31gKfQGuw/m7DNW2j3887A/Qb359/Q7ucdWDakYmWb9/dM+/2fDyzuDebv0IIY2/d1fzMsrKoqyW60IQmHJTmc1sNpU9pQha/QGqvDbf43ybNpc9mclORo2rCfP/Z9PYJ2f40CgNsCb+3n9xzafbBBr9Mfgf3HjvFjtPvt8dzIiWmnleTxtGETt6P9e7frhN/0r6vq7WNpL6G1VxYD5yb5Mu28/xmt3g+mBZGeNf7mkgW8idbj7Z+B45N8nxZs+iUtILGIdj7h+gHooR3mCARB64X2ibl2XlULTd4q3fYsz7tF/fjx48ePn9vCh9Y9ew9ag+Bc2nj3q2kN1i/RXvV4uwnbbUdr0PyWNgdDAYv6ulWBV9Eax7+nja/+OG3ys4OGeXv+RT3toDnq+E0mvO+e9qT9rbSG+f/RGiivpg3LuEF5wJrAv9MaSf9He3PBe2gNnhvsg/bEu4A3znP+VqeNaf8RbXLDpX0ft+/bfnMs/xt7+taDtFrgM15GgOcBR9MaF1f3c/Bt4LXAhhPy70kbKnEVrWH8HlrDdykt8HFj7pv/oHXb/zUt2PELWkPslcCdJmzzJNpQlV/1elxAa5CvM8c+7tiP5/v9Pvs9LXB2JK0R92dj+ec733Newwn7LeDiKfP+W8+/y1j6XYAD+vn9P9orMv+/fg9OPOe0RuI7+/W5imX385Z9H28fy38QY7+lhe7defY95+95cL0/2e+339OCL8+caz99m4fRghBX9M/XaQGG0TXZesI2i2hBr/P7eftNPw8fB3YY5HsA7bd/cr/vRr+9z9DmkJn4b8ikfS5wH9zg350J53P8XO3Cwr/pOX9ztN/Jp2mv5b2KNv/G9/t5u8vy/lYH5d4PeBstCPtr2m/2l/1avo02J9D4Nm+c5xhGn7ePHfe/TVmfixc6v3783Fo/qZpr7ipJkiTp5ifJi2mTCL60qt436/pIklaMgQlJkiTdLCX586r62VjahrRhIfekPZWf9g0qkqSbKeeYkCRJ0s3VYX2S2O/TutovAp5GG/6xj0EJSbp1sMeEJEmSbpaS7E6bO2Rj2vwfv6W9avXdVXX4LOsmSVp5DExIkiRJkqSZcSiHpBX20Y9+tF7wghfMuhqSJEmSbr5u8J7gkdv9KWsh6dbpd7/73ayrIEmSJOkWysCEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmbGwIQkSZIkSZoZAxOSJEmSJGlmDExIkiRJkqSZMTAhSZIkSZJmxsCEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmbGwIQkSZIkSZoZAxOSJEmSJGlmDExIkiRJkqSZWXXWFZB0y7fkp5ezaO8jZ10NSZIkScDS/Z466yosF3tMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmbGwIQkSZIkSZoZAxOSJEmSJGlmDExIkiRJkqSZMTAhSZIkSZJmxsCEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMLESJFkvySeS/CjJ95OcmGTHJFsnuTzJaUnOSPL1JHfv2+ySpJJsMyhnx5727Hn2dVCSC3uZpyXZbIr6fS7JiVMey6IkfztFvq17XXcbpG3e0/5xUNc5j2WsvK8k+XWSL06R9+Ak5yY5M8mHk6zW05PknUku6Of7oT19wyTHJDk7yVlJXjEoa/8k5/T8RyRZZ7Bun17WuUme3NPuODj3pyW5LMnb+7pXJflBL+voJPfu6Zv1e+Ksvm6nBY5vz77fSrLuIP01g/2emeTaJHfp67br9bwgyd6DbTbt+16S5AtJ7tTTtxyUdXqSHQfb7NTreVaStyx0PSRJkiRpRRiYWEFJAnwWOLaq7lNVDwN2BjboWY6rqs2q6iHA94A9BpsvARYPvu8MnD7Fbl/Ty9ysqk5boH7rAA8F1kmy0RRlLwIWDEx0S4BhI3va+k+yP/C8KfMeDNwf2AS4A/Cinv4UYOP+eQnw3p5+DfDqqnoA8HBgjyQP7OuOAh7cr895wD4Aff3OwIOA7YADkqxSVVcMzv1mwI+Bw3tZpwJb9LI+A4wa9VcCz6+qUVlvHwZAJjge2LaXfZ2q2n+w332Ab1XVL5OsArynH/8DgcWD4/sgsHdVbQIcAbymp5/Z67pZr9P7kqya5K60a7FNr+96w+CZJEmSJK1sBiZW3BOAq6vqwFFCVf24qt41zNQDGHcEfjVIPg7YMslqSdYC7guctpLr9yzgC8ChtIb2qD7X682Q5Ld9cT/gMf1J+j8kWSPJR/oT91OTPH5Q9k+ANdJ6jITWwP3yjalkVR0NXDFl3i9VB3yXZUGg7YGP9VUn0YIx96yqS6rqlL7tFcDZwPr9+9eq6pq+/UljZR1aVVdV1YXABcCWw3ok2Ri4O+06UlXHVNWV42VV1XlVdX5f/hlwKXC3eY7v1KpausBpWAwc0pe3BC6oqh9V1dW0a719X3c/4Ni+fBTtfqCqrhwc9xpA9eX7AOdV1S/696+PthmX5CVJTk5y8rVXXr5AdSVJkiRpMgMTK+5BwCnzrH9MktNojfhtgQ8P1hWt4fdkWkPy81Pu8997V/u3Jbn9AnlHDdhDuH7vjLnszbJeHm+j9/DoT9wXAx9NssYg/2eA5wCPpJ2Hq6Y8hhXWh3A8D/hKT1ofuGiQ5eKeNtxmEbA58J0JRb6QZYGVBcuinY9P9gDJuN2YEKRJsiWwOvDDCdtMJcmatCDQYVPU9UzgGX35OcCGg3K2SnIWrefLS3ug4gLg/mlDelYFdhhuM1RV76+qLapqi1XWXPvGHo4kSZKk2zgDEytZkvf0Mfvf60mjRv6GwEdY1r1/ZNSTYWeWPQGfzz60YQx/BdwF2GueuqxH64Xx7ao6D7gmyYOX64Dg0cDHAarqHNrwgr8crP8UrcE7fIL/p3IAbQjNcf17JuS5LmjQe6UcBryyqn4zzJRkX9qQj4OnKaubeM2S/B2wBW1IxDD9nrRzuWtV/XGOY5rG04Hjq+qXU9T1hbShK9+n9di5+roMVd/pwzX+CtgnyRpV9SvgZcAnaT1BltLOiyRJkiTdJAxMrLizaHM4AFBVewDbMLmr/ueBxw4Tquq7wIOBdXvwYF59WEJV1VW0QMeW82TfCbgzcGGSpbT5I0bDOa6hX/8+DGP1OcqY1Ogd1ufnwB+AJwJHL1T/lSXJG2jn+FWD5Iu5/tP9DYCf9fyr0YISB1fV4YM8JHkB8DTguYPeD3OW1bfZFFi1qr4/Vta2wL7AM/o1GqXfCTgSeF0fZrIixgMic9a1qs6pqif1uU8OYUJPjao6G/gd7T6kqr5QVVtV1SOAc4HzV7C+kiRJkjQnAxMr7hu0eRZeNkhbc468j2ZyF/59gNdOs7P+1H0UTNiB1lV/LouB7apqUVUtAkYTc0J7Ev6wvrw9sFpfvoL2ZH3kWOC5fZ9/CdyL1lgdej2wV1VdO80xrKgkL6INf1k81vPg88Dz0zwcuLyqLunn6kPA2VX11rGytqP1OnnGYH6IUVk7J7l9nzR0Y9p8FiM36CGSZHPgfb2sSwfpq9MmnvxYVX16BY99beBxwOcGyd8DNk6yUd/Xzr3+ZNlbYG4HvA44sH/fqA/VIO3tIfej3RPDbe4M7E6bQFOSJEmSbhKrzroCt3RVVUl2AN6W5J+AX9CePo+GWIzmmAhwOcveIDEsY3kmjDw4yd16eacBL52Uqc+lcC/aJIyj/VyY5DdJtgI+AHwuyXdpPR1+17OdQRvycTpwEG24xIFJltB6WexSVVe1tv515Z4wT33fl/46TeCi/hR+Un2Pow1RWSvJxcBuVfXVOco8kDak5MRej8Or6k3Al4C/ps2TcCWwa8//KNpcFEv6tQB4bVV9CXg3cHvgqF7WSVX10qo6K8mngB/0495jLPDyN31fQ/sDawGf7mX9pKqe0fM+Frhrkl163l3meqNKkpcD/wTcAzgjyZeqanTf7Ah8rapG14uquibJnsBXgVWAD1fVWX314iSjN8EcTutlAy1ItneSPwB/BHavqsv6unf0HiEAb5qmJ48kSZIk3ViZPG+fJE3vZfu+ub587UNmXQ1JkiRJwNL9njrrKkwy5zQBDuWQJEmSJEkz41COm6kkRwAbjSXvNWl4Q5JdgVeMJR/fJ+K82UiyCf0NHwNXVdVWc+Sf+hzcEt3aj0+SJEmSpmFg4maqqnZcjrwfYdncATdbVbUE2Gw58k99Dm6Jbu3HJ0mSJEnTcCiHJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmbGwIQkSZIkSZoZAxOSJEmSJGlmVp11BSTd8m2y/tq8d/enzroakiRJkm6B7DEhSZIkSZJmxsCEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmZm1VlXQNIt35KfXs6ivY+cdTUk6UZZut9TZ10FSZJu0+wxIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmbGwIQkSZIkSZoZAxOSJEmSJGlmDExIkiRJkqSZMTAhSZIkSZJmxsCEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmbGwMRKkGS9JJ9I8qMk309yYpIdk2yd5PIkpyU5I8nXk9y9b7NLkkqyzaCcHXvas+fZ14eSnN7L+0yStaao3+eSnDjlsSxK8rdT5Nu613W3QdrmPe0f+/eD5juWsfK+kuTXSb44Rd6Dk5yb5MwkH06yWk9PkncmuaCfn4f29A2THJPk7CRnJXnFoKz9k5zT8x+RZJ3Bun16WecmeXJPu2O/nqPPZUne3te9KskPellHJ7l3T9+s3xNn9XU7LXB8e/b9VpJ1B+mvGez3zCTXJrlLX7ddr+cFSfYebLNp3/eSJF9IcqeevuWgrNOT7DjYZqdez7OSvGWh6yFJkiRJK8LAxApKEuCzwLFVdZ+qehiwM7BBz3JcVW1WVQ8BvgfsMdh8CbB48H1n4PQFdvkPVbVpL+8nwJ4L1G8d4KHAOkk2muKQFgELBia6JcCwkT1N/eeyP/C8KfMeDNwf2AS4A/Cinv4UYOP+eQnw3p5+DfDqqnoA8HBgjyQP7OuOAh7cz+d5wD4Aff3OwIOA7YADkqxSVVf067lZVW0G/Bg4vJd1KrBFL+szwKhRfyXw/KoalfX2YQBkguOBbXvZ16mq/Qf73Qf4VlX9MskqwHv68T8QWDw4vg8Ce1fVJsARwGt6+pm9rpv1Or0vyapJ7kq7Ftv0+q43DJ5JkiRJ0spmYGLFPQG4uqoOHCVU1Y+r6l3DTD2AcUfgV4Pk44Atk6zWez7cFzhtvp1V1W8G5d0BqAXq9yzgC8ChtIb2qD7X682Q5Ld9cT/gMf1J+j8kWSPJR/oT91OTPH5Q9k+ANdJ6jITWwP3yAvWZ67iOBq6YMu+XqgO+y7Ig0PbAx/qqk2jBmHtW1SVVdUrf9grgbGD9/v1rVXVN3/6ksbIOraqrqupC4AJgy2E9kmwM3J12HamqY6rqyvGyquq8qjq/L/8MuBS42zzHd2pVLV3gNCwGDunLWwIXVNWPqupq2rXevq+7H3BsXz6Kdj9QVVcOjnsNlt1H9wHOq6pf9O9fH20jSZIkSTcFAxMr7kHAKfOsf0yS02iN+G2BDw/WFa3h92RaQ/Lz0+wwyUeAn9N6DbxrgeyjBuwhXL93xlz2Zlkvj7fRe3j0J+6LgY8mWWOQ/zPAc4BH0s7DVdMcw8rQh3A8D/hKT1ofuGiQ5eKeNtxmEbA58J0JRb6QZYGVBcuinY9P9gDJuN2YEKRJsiWwOvDDCdtMJcmatCDQYVPU9UzgGX35OcCGg3K2SnIWrefLS3ug4gLg/mlDelYFdhhuM1aPlyQ5OcnJ1155+Y09HEmSJEm3cQYmVrIk7+lj9r/Xk0aN/A2Bj7Cse//IqCfDzix7Aj6vqtoV+HPak/855ytIsh6tF8a3q+o84JokD16uA4JHAx/v+z2HNrzgLwfrP0Vr8A6f4P+pHEAbQnNc/54Jea4LGvReKYcBrxz1PBms25c25OPgacrqJl6zJH8HbEEbEjFMvyftXO5aVX+c45im8XTg+Kr65RR1fSFt6Mr3aT12rr4uQ9V3+nCNvwL2SbJGVf0KeBnwSVpPkKW083LDHVS9v6q2qKotVllz7RU4HEmSJEm3ZQYmVtxZtDkcAKiqPYBtmNxV//PAY4cJVfVd4MHAuj14MJWqupbWeJyvm/1OwJ2BC5Mspc0fMRrOcQ39+vdhGKvPUcakRu+wHj8H/gA8ETh6utqvuCRvoJ3jVw2SL+b6T/c3AH7W869GC0ocXFWHD/KQ5AXA04DnDno/zFlW32ZTYNWq+v5YWdsC+wLPqKqrBul3Ao4EXteHmayI8YDInHWtqnOq6kl97pNDmNBTo6rOBn5Huw+pqi9U1VZV9QjgXOD8FayvJEmSJM3JwMSK+wZtnoWXDdLWnCPvo5nchX8f4LUL7SjNfUfLtCfn58yzyWJgu6paVFWLgNHEnNCehD+sL28PrNaXr6A9WR85Fnhu3+dfAveiNVaHXg/s1YMlN7kkL6INf1k81vPg88Dz+3l6OHB5VV3Sz9WHgLOr6q1jZW0H7EULJFw5VtbOSW7fJw3dmDafxcgNeogk2Rx4Xy/r0kH66rSJJz9WVZ9ewWNfG3gc8LlB8veAjZNs1Pe1c68/WfYWmNsBrwMO7N836kM1SHt7yP1o98RwmzsDu9Mm0JQkSZKkm8Sqs67ALV1VVZIdgLcl+SfgF7Snz3v1LKM5JgJczrI3SAzLmHbCyNDmeLhTXz6d1u3+hhnbXAr3ok3CONrPhUl+k2Qr4APA55J8l9bT4Xc92xm0IR+nAwfRhkscmGQJrZfFLlV1VWvrX1fuCfPU+X3pr9MELupP4SfV9zjanBlrJbkY2K2qvjpHmQfShpSc2OtxeFW9CfgS8Ne0eRKuBHbt+R9Fm4tiSb8WAK+tqi8B7wZuDxzVyzqpql5aVWcl+RTwg37ce4wFXv6m72tof2At4NO9rJ9U1TN63scCd02yS8+7S1WdxgRJXg78E3AP4IwkX6qq0X2zI/C1qhpdL6rqmiR7Al8FVgE+XFVn9dWLk4zeBHM4bTgRtCDZ3kn+APwR2L2qLuvr3tF7hAC8aXl68kiSJEnS8srkefskaXov2/fN9eVrHzLrakjSjbJ0v6fOugqSJN0WzDlNgEM5JEmSJEnSzDiU42YqyRHARmPJe00a3pBkV+AVY8nH94k4bzaSbEJ/w8fAVVW11Rz5pz4Ht0S39uOTJEmSpGkYmLiZqqodlyPvR1g2d8DNVlUtATZbjvxTn4Nbolv78UmSJEnSNBzKIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmbGwIQkSZIkSZoZAxOSJEmSJGlmDExIkiRJkqSZMTAhSZIkSZJmxsCEJEmSJEmaGQMTkiRJkiRpZladdQUk3fJtsv7avHf3p866GpIkSZJugewxIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmbGwIQkSZIkSZoZAxOSJEmSJGlmDExIkiRJkqSZMTAhSZIkSZJmZtVZV0DSLd+Sn17Oor2PnHU1pFuVpfs9ddZVkCRJ+pOwx4QkSZIkSZoZAxOSJEmSJGlmDExIkiRJkqSZMTAhSZIkSZJmxsCEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmbGwIQkSZIkSZoZAxOSJEmSJGlmDExIkiRJkqSZMTAhSZIkSZJmxsCEJEmSJEmaGQMTK0GS9ZJ8IsmPknw/yYlJdkyydZLLk5yW5IwkX09y977NLkkqyTaDcnbsac+eZ197Jrmg51t3yvp9LsmJU+ZdlORvp8i3da/DboO0zXvaP/bvB813LGPlfSXJr5N8cYq8Byc5N8mZST6cZLWeniTv7OfnjCQP7ekbJjkmydlJzkryikFZ+yc5p+c/Isk6g3X79LLOTfLknnbHfj1Hn8uSvL2ve1WSH/Syjk5y756+Wb8nzurrdlrg+CZe4ySvGez3zCTXJrlLX7ddr+cFSfYebLNp3/eSJF9IcqeevuWgrNOT7DjYZqdez7OSvGWh6yFJkiRJK8LAxApKEuCzwLFVdZ+qehiwM7BBz3JcVW1WVQ8BvgfsMdh8CbB48H1n4PQFdnk8sC3w4ynrtw7wUGCdJBtNsckiYMHARLcEGDayp6n/XPYHnjdl3oOB+wObAHcAXtTTnwJs3D8vAd7b068BXl1VDwAeDuyR5IF93VHAg/v1OQ/YB6Cv3xl4ELAdcECSVarqin49N6uqzWjX4fBe1qnAFr2szwCjRv2VwPOralTW24cBkAkmXuOq2n+w332Ab1XVL5OsArynH/8DgcWD4/sgsHdVbQIcAbymp5/Z67pZr9P7kqya5K60a7FNr+96w+CZJEmSJK1sBiZW3BOAq6vqwFFCVf24qt41zNQDGHcEfjVIPg7YMslqSdYC7gucNt/OqurUqlq6HPV7FvAF4FBaQ3tUn+v1Zkjy2764H/CY/iT9H5KskeQj/Yn7qUkePyj7J8AaaT1GQmvgfnk56jY8rqOBK6bM+6XqgO+yLAi0PfCxvuokWjDmnlV1SVWd0re9AjgbWL9//1pVXdO3P2msrEOr6qqquhC4ANhyWI8kGwN3p11HquqYqrpyvKyqOq+qzu/LPwMuBe42z/FNc40XA4f05S2BC6rqR1V1Ne1ab9/X3Q84ti8fRbsfqKorB8e9BlB9+T7AeVX1i/7966NtxiV5SZKTk5x87ZWXL1BdSZIkSZrMwMSKexBwyjzrH5PkNFojflvgw4N1RWv4PZnWkPz8TVC/UQP2EK7fO2Mue7Osl8fb6D08+hP3xcBHk6wxyP8Z4DnAI2nn4aqVWPd59SEczwO+0pPWBy4aZLm4pw23WQRsDnxnQpEvZFlgZcGyaOfjkz1AMm43JgRpkmwJrA78cMI2U0myJi0IdNgUdT0TeEZffg6w4aCcrZKcRev58tIeqLgAuH/akJ5VgR2G2wxV1furaouq2mKVNde+sYcjSZIk6TbOwMRKluQ9fcz+93rSqJG/IfARlnXvHxn1ZNiZZU/AV1Zd1qP1wvh2VZ0HXJPkwctZzKOBjwNU1Tm04QV/OVj/KVqDd/gE/0/lANoQmuP690zIc13QoPdKOQx4ZVX9Zpgpyb60IR8HT1NWN/GaJfk7YAvakIhh+j1p53LXqvrjHMc0jacDx1fVL6eo6wtpQ1e+T+uxc/V1Gaq+04dr/BWwT5I1qupXwMuAT9J6giylnRdJkiRJukkYmFhxZ9HmcACgqvYAtmFyV/3PA48dJlTVd4EHA+v24MHKtBNwZ+DCJEtp80eMhnNcQ7/+fRjG6nOUManRe52q+jnwB+CJwNErXOMpJXkD7Ry/apB8Mdd/ur8B8LOefzVaUOLgqjp8kIckLwCeBjx30PthzrL6NpsCq1bV98fK2hbYF3hGVV01SL8TcCTwuj7MZEWMB0TmrGtVnVNVT+pznxzChJ4aVXU28DvafUhVfaGqtqqqRwDnAuevYH0lSZIkaU4GJlbcN2jzLLxskLbmHHkfzeQu/PsAr13ZFaP1YtiuqhZV1SJgNDEntCfhD+vL2wOr9eUraE/WR44FnguQ5C+Be9Eaq0OvB/aqqmtXcv0nSvIi2vCXxWM9Dz4PPD/Nw4HLq+qSHnj5EHB2Vb11rKztgL1ogYQrx8raOcnt+6ShG9Pmsxi5QQ+RJJsD7+tlXTpIX5028eTHqurTK3jsawOPAz43SP4esHGSjfq+du71J8veAnM74HXAgf37Rn2oBv3tIfej3RPDbe4M7E6bQFOSJEmSbhIGJlZQf8K+A/C4JBcm+S7wUVpjF5ZNJHk6bT6EV08o48tVdcw0+0vy8iQX056Kn5FkYqOxz6VwL9okjKP9XAj8JslWwAd6nb8LbEV7Yg5wBm3Ix+lJ/oE2XGKVJEto3ft3GfYE6OWeUFWfnaPK70tycf/M+crSJMcBnwa26XmfPM9pOBBYDzixn9vX9/QvAT+izZPwAVqjGuBRtHP/hMErMv+6r3s3LRBzVE8/sB/TWbRhKj+gzWGxx1jg5W+44TCO/YG1gE/3sj4/yPtYYJfB/jeb51zMd413BL5WVaPrRZ8bYk/gq7SJPT/V6w/tDR3nAefQelF8pKc/Gji9z39yBLB7VV3W170jyQ9obwfZ7yboySNJkiRJ18nkefskaXov2/fN9eVrHzLraki3Kkv3e+qsqyBJkrQyzTlNgD0mJEmSJEnSzKw66wposiRHABuNJe9VVV+dkHdX4BVjycf3iThvNpJsQn/Dx8BVVbXVHPmnPge3RLf245MkSZKkaRiYuJmqqh2XI+9HWDZ3wM1WVS0BNluO/FOfg1uiW/vxSZIkSdI0HMohSZIkSZJmxsCEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmbGwIQkSZIkSZqZVWddAUm3fJusvzbv3f2ps66GJEmSpFsge0xIkiRJkqSZMTAhSZIkSZJmxsCEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJlZddYVkHTLt+Snl7No7yNnXQ3pFmHpfk+ddRUkSZJuVuwxIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmbGwIQkSZIkSZoZAxOSJEmSJGlmDExIkiRJkqSZMTAhSZIkSZJmxsCEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmZmqsBEkvWSfCLJj5J8P8mJSXZMsnWSy5OcluSMJF9Pcve+zS5JKsk2g3J27GnPXtGKJ9kiyTvnWLc0ybrzbPvhJJcmOXPKfa2a5LIkb54y/9ZJHjlFvjf283HfQdo/9LQtpjmWwXb379flqiT/uEDeDZMck+TsJGclecVg3V2SHJXk/P73zj39if3aL+l/n9DT10xyZJJzeln7Dcq6fZJPJrkgyXeSLOrpj+/3zOjzf0l26OsOTnJukjP7dVqtpz+332NnJDkhyaYLHOPEa9zrM9rv0iSnDdbt0+t6bpInD9J36vs9K8lbBukv7efjtCTfTvLAwbr/6MdwZpKd5qvrypTkoIV+XwvlSbJnPw81zb0nSZIkSStiwcBEkgCfBY6tqvtU1cOAnYENepbjqmqzqnoI8D1gj8HmS4DFg+87A6evaKWTrFpVJ1fVy29kEQcB2y1H/icB5wJ/08/HQrYGFgxMdEto52Xk2cAPlqNuI78EXg785xR5rwFeXVUPAB4O7DFoVO8NHF1VGwNH9+8AlwFPr6pNgBcAHx+U959VdX9gc+BRSZ7S03cDflVV9wXeBvwHQFUd0++ZzYAnAFcCX+vbHAzcH9gEuAPwop5+IfC4fp/9K/D+BY7xICZc46raabDvw4DDAfrx7ww8qG93QJJVktwV2B/YpqoeBKw3CLZ9oqo26WW9BXhrL+upwEOBzYCtgNckudMC9b05OR7YFvjxrCsiSZIk6dZvmh4TTwCurqoDRwlV9eOqetcwU2+w3xH41SD5OGDLJKslWQu4L3DafDtL8tf96fu3k7wzyRd7+huTvD/J14CP9V4Jo3V3TfK1JKcmeR8wb/Cgqo6lNeSntRh4B/ATWkN+VNfrejOk9eD4Zu8V8FLgH/qT9MckuXeSo/tT96OT3GtQ9meB7XsZ9wEuB36xHHUbHdOlVfU94A9T5L2kqk7py1cAZwPr99XbAx/tyx8Fduj5Tq2qn/X0s4A1kty+qq6sqmN6nquBU1gWtBqW9RlgmwmBnWcDX66qK3sZX6oO+O6orKo6oapG99ZJg33MdYzzXuNej78BDhnU9dCquqqqLgQuALYE7gOcV1Wja/J14Fl9H78ZFPlnQPXlBwLfqqprqup3tGDcnIGwfh/9f2k9Xk5O8tAkX03ywyQvHdU3yf69B8aSUS+Mnv7uJD9IciRw90G5D0vyrbQeLl9Ncs/5ztng3J1aVUsXypfkJb2+J1975eXTFC1JkiRJNzBNYOJBtMbmXB7Tu8P/hPaU9cODdUVryD2Z1vD7/Hw7SrIG8D7gKVX1aOBuY1keBmxfVX87lv4G4NtVtXnfx71YSZLcAdgG+CKtEbt4vvy9QXcg8Lb+ZP444N3Ax/rT/oOB4RCU3wAXJXlwL/uTK6vu0+iBlM2B7/Sk9arqEmgBDAYN3YFnAadW1VVjZa0DPJ3W0wJasOOiXtY1tKDLXcfK2pllwYFhWasBzwO+MmH/uwFfnv/IFvQY4H+q6vzxunYX97QLgPsnWZRkVVqgZsNBPfdI8kNaj4lRD57TgaekDXNZF3j8cJs5XFRVj6AF8w6iBWweDrypr38mrQfGprTf2f490LAjcD9aD5MX03vq9PP3LuDZvZfTh4F/n+bETKuq3l9VW1TVFqusufbKLFqSJEnSbchyT36Z5D1JTk/yvZ40GsqxIfARWgNt6FBa43NiA3TM/YEf9SfWTMj/+ar6/YTtHgv8N0BVHcn1e22sqKcBx/Qn+ocBOyZZZTnLeATwib78ceDRY+tH52gH4IgbX9Xl03uxHAa8cuzp/3zbPIg2JOPvx9JXpV2vd1bVj0bJE4qowTb3pDWovzoh3wG04UPHje3n8bTAxF7T1Hcei7n+/TWxrr2XxstoAaPjgKW0oTCjDO+pqr/o9XldT/sa8CXghL6PE4fbzGEUtFsCfKeqrui9NP6vB3weDRxSVddW1f8A3wL+inbvj9J/Bnyjl3M/4MHAUT1w+DoW6GUiSZIkSbMwTWDiLNp4eQCqag9aD4Lx3gzQGlePHSZU1XdpDaR1q+q8Bfa10PwNv5tnXc2zbkUsBrZNshT4Pu2J/+P7umtYdg7XWI4yx+v6BVrvgJ9MGyBYUf2J+mHAwVV1+GDV/4y6/Pe/lw622YAWOHl+Vf1wrMj3A+dX1dsHaRfTewr0wMXaXH94xd8AR1TV9YafJHkD7f561Vj6Q4AP0nrN/O9yHfD1y1mV1gNh2Dvlurp2GwA/A6iqL1TVVr1Hw7nA+dzQofRhL32bf+8BuyfS7utJ2wyNep/8cbA8+r4q8/82Jt37Ac4azafR58J40gJ1kCRJkqQ/uWkCE9+gzSfwskHamnPkfTQw3mAF2Ad47RT7Oge4Tx9eADDt2wyOBZ4L0CdevPOU282rT1j4aOBeVbWoqhbRJvccDedYShteAn3ege4K2nwbIyewbILL5wLfHu6n9wLZi5Xc1X4ufX6FDwFnV9Vbx1Z/nja5Jf3v5/o26wBHAvtU1fFj5f0bLejwynnKejbwjT53xMh4rwWSvIg29GdxVf1xkH4v2kSVz5siwLWQbYFzqurisbrunPYmkY2AjWlzXJBlb5q5M7A7LThCko0H2z+VHnwYTJo5CqY8hGWTe95YxwI79bLvRgsAfren79zT78myoNm5wN2SPKLXY7Xe20WSJEmSblZWXShDVVXaqxzfluSfaBMz/o5lXelHc0yENofAiyaUMdV8AFX1+yS7A19Jchm9YTiFfwEOSXIKrYv7T+bLnOQQ2psz1k1yMfCGqvrQhKzPpDWmh0+wPwe8Jcnt+34/lOS1LJujAVoPiM8k2R74f7S5Bz6c5DW087fr+I6q6tB5qnxGklEj/VNV9arxDEnuAZwM3An4Y5JXAg+cowfGo2g9NJZk2esyX1tVXwL2Az6VZDfaeXxOX78nbfLSf07yzz3tScDqwL60oNIpfW7Ld1fVB2nBj48nuYDWU+K6t4/04NOGtOs1dCDtbRAn9rIOr6o3Aa+n9VY5oKdfU1VbzHXCFrjGNxhWVFVnJfkU7Y0o1wB7VNW1ffU7suz1pG8aBEb2TLItbcLRX7EsCLMacFyv52+Av+tzbKyII2hDgk6n9ZD4p6r6eZIjaBPULgHOo5/Pqro67ZWg70yyNu23/nZaD6h5JXk58E/APWj33peq6ga/a0mSJElaGXL9B9izl2Stqvptf6r/HtrwgLfNul6S5vayfd9cX772IbOuhnSLsHS/p866CpIkSbMw5/D05Z788k/gxf0p/lm04QHvm211JEmSJEnSTWXBoRw3ld4FfaOx5L1674gV7iHRx/gfPWHVNpMmTkzyHtoQh6F3VNVHVrQuK1OSXYFXjCUf3yclHc+7XOfgluaWdnzz3POT3kpym6uPJEmSpNumm91QDkm3PA7lkKbnUA5JknQbdYsayiFJkiRJkm4jDExIkiRJkqSZMTAhSZIkSZJmxsCEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmZm1VlXQNIt3ybrr817d3/qrKshSZIk6RbIHhOSJEmSJGlmDExIkiRJkqSZMTAhSZIkSZJmxsCEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWZWnXUFJN3yLfnp5Sza+8hZV0MLWLrfU2ddBUmSJOkG7DEhSZIkSZJmxsCEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmbGwIQkSZIkSZoZAxOSJEmSJGlmDExIkiRJkqSZMTAhSZIkSZJmxsCEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAxEqQZL0kn0jyoyTfT3Jikh2TbJ3k8iSnJTkjydeT3L1vs0uSSrLNoJwde9qzp9jnu5L8dsr6fS7JiVPmXZTkb6fIt3Wv626DtM172j/27wdNcyw971eS/DrJF6fIe3CSc5OcmeTDSVbr6UnyziQX9PP90J6+YZJjkpyd5KwkrxiUtX+Sc3r+I5KsM1i3Ty/r3CRP7ml37Ndz9Lksydv7ulcl+UEv6+gk9+7pm/V74qy+bqcFjm/Pvt9Ksu4g/TWD/Z6Z5Nokd+nrtuv1vCDJ3oNtNu37XpLkC0nu1NO3HJR1epIdB9vs1Ot5VpK3LHQ9JEmSJGlFGJhYQUkCfBY4tqruU1UPA3YGNuhZjquqzarqIcD3gD0Gmy8BFg++7wycPsU+twDWmbJ+6wAPBdZJstEUmywCFgxMdEuAYSN7qvrPYX/geVPmPRi4P7AJcAfgRT39KcDG/fMS4L09/Rrg1VX1AODhwB5JHtjXHQU8uF+f84B9APr6nYEHAdsBByRZpaqu6Ndzs6raDPgxcHgv61Rgi17WZ4BRo/5K4PlVNSrr7cMAyATHA9v2sq9TVfsP9rsP8K2q+mWSVYD39ON/ILB4cHwfBPauqk2AI4DX9PQze10363V6X5JVk9yVdi226fVdbxg8kyRJkqSVzcDEinsCcHVVHThKqKofV9W7hpl6AOOOwK8GyccBWyZZLclawH2B0+bbWW+E7g/805T1exbwBeBQWkN7VM71ejMMel/sBzymP0n/hyRrJPlIf+J+apLHD8r+CbBGWo+R0Bq4X56yXtdTVUcDV0yZ90vVAd9lWRBoe+BjfdVJtGDMPavqkqo6pW97BXA2sH7//rWquqZvf9JYWYdW1VVVdSFwAbDlsB5JNgbuTruOVNUxVXXleFlVdV5Vnd+XfwZcCtxtnuM7taqWLnAaFgOH9OUtgQuq6kdVdTXtWm/f190POLYvH0W7H6iqKwfHvQZQffk+wHlV9Yv+/eujbcYleUmSk5OcfO2Vly9QXUmSJEmazMDEinsQcMo86x+T5DRaI35b4MODdUVr+D2Z1pD8/BT72xP4fFVdMmX9Rg3YQ7h+74y57M2yXh5vo/fw6E/cFwMfTbLGIP9ngOcAj6Sdh6umrNcK60M4ngd8pSetD1w0yHJxTxtuswjYHPjOhCJfyLLAyoJl0c7HJ3uAZNxuTAjSJNkSWB344YRtppJkTVoQ6LAp6nom8Iy+/Bxgw0E5WyU5i9bz5aU9UHEBcP+0IT2rAjsMtxmqqvdX1RZVtcUqa659Yw9HkiRJ0m2cgYmVLMl7+pj97/WkUSN/Q+AjLOvePzLqybAzy56Az1X2n9Mal++aL98g/3q0XhjfrqrzgGuSPHj6owHg0cDHAarqHNrwgr8crP9Ur9PwCf6fygG0ITTH9e+ZkOe6oEHvlXIY8Mqq+s0wU5J9aUM+Dp6mrG7iNUvyd8AWtJ4tw/R70s7lrlX1xzmOaRpPB46vql9OUdcX0oaufJ/WY+fq6zJUfacP1/grYJ8ka1TVr4CXAZ+k9QRZSjsvkiRJknSTMDCx4s6izeEAQFXtAWzD5K76nwceO0yoqu8CDwbW7cGD+WxOCzRckGQpsGaSC+bJvxNwZ+DCnn8Ry4ZzXEO//n0YxupzlDGp0Tus/8+BPwBPBI5eoP4rTZI30M7xqwbJF3P9p/sbAD/r+VejBSUOrqrDB3lI8gLgacBzB70f5iyrb7MpsGpVfX+srG2BfYFnVNVVg/Q7AUcCr+vDTFbEeEBkzrpW1TlV9aQ+98khTOipUVVnA7+j3YdU1ReqaquqegRwLnD+CtZXkiRJkuZkYGLFfYM2z8LLBmlrzpH30Uzuwr8P8NqFdlRVR1bVPapqUVUtAq6sqvvOs8liYLtB/tHEnNCehD+sL28PrNaXr6A9WR85FnguQJK/BO5Fa6wOvR7Yq6quXegYVoYkL6INf1k81vPg88Dz0zwcuLyqLumBlw8BZ1fVW8fK2g7YixZIuHKsrJ2T3L5PGroxbT6LkRv0EEmyOfC+Xtalg/TVaRNPfqyqPr2Cx7428Djgc4Pk7wEbJ9mo72vnXn+y7C0wtwNeBxzYv2/Uh2qQ9vaQ+9HuieE2dwZ2p02gKUmSJEk3iVVnXYFbuqqqJDsAb0vyT8AvaE+f9+pZRnNMBLicZW+QGJZxoyaMnE+fS+FetEkYR/u5MMlvkmwFfAD4XJLv0no6/K5nO4M25ON04CDacIkDkyyh9bLYpaquam3968o9YZ6qvC/9dZrARf0p/KT6Hkd708ZaSS4Gdquqr85R5oG0ISUn9nocXlVvAr4E/DVtnoQrgV17/kfR5qJY0q8FwGur6kvAu4HbA0f1sk6qqpdW1VlJPgX8oB/3HmOBl7/p+xraH1gL+HQv6ydV9Yye97HAXZPs0vPuUlWnMUGSl9MmN70HcEaSL1XV6L7ZEfhaVY2uF1V1TZI9ga8CqwAfrqqz+urFSUZvgjmcNpwIWpBs7yR/AP4I7F5Vl/V17+g9QgDeNEVPHkmSJEm60TJ53j5Jmt7L9n1zffnah8y6GlrA0v2eOusqSJIk6bZrzmkCHMohSZIkSZJmxqEcN1NJjgA2Gkvea9LwhiS7Aq8YSz6+T8R5s5FkE/obPgauqqqt5sg/9Tm4Jbq1H58kSZIkTcPAxM1UVe24HHk/wrK5A262qmoJsNly5J/6HNwS3dqPT5IkSZKm4VAOSZIkSZI0MwYmJEmSJEnSzBiYkCRJkiRJM2NgQpIkSZIkzYyBCUmSJEmSNDMGJiRJkiRJ0swYmJAkSZIkSTNjYEKSJEmSJM2MgQlJkiRJkjQzBiYkSZIkSdLMrDrrCki65dtk/bV57+5PnXU1JEmSJN0C2WNCkiRJkiTNjIEJSZIkSZI0MwYmJEmSJEnSzBiYkCRJkiRJM2NgQpIkSZIkzYyBCUmSJEmSNDMGJiRJkiRJ0swYmJAkSZIkSTNjYEKSJEmSJM3MqrOugKRbviU/vZxFex8562osl6X7PXXWVZAkSZKEPSYkSZIkSdIMGZiQJEmSJEkzY2BCkiRJkiTNjIEJSZIkSZI0MwYmJEmSJEnSzBiYkCRJkiRJM2NgQpIkSZIkzYyBCUmSJEmSNDMGJiRJkiRJ0swYmJAkSZIkSTNjYEKSJEmSJM2MgQlJkiRJkjQzBiYkSZIkSdLMGJiQJEmSJEkzY2BCkiRJkiTNjIEJSZIkSZI0MwYmJEmSJEnSzNxsAhNJrk1yWpKzkpye5FVJVnr9knwzyRYru9x59vfBJA9cyWV+Jcmvk3xxyvx3S/KHJH8/Zf4dpqlzkoOSXJnkjoO0dySpJOv277+dcp+PTXJKkmuSPHuBvJslObHfK2ck2WmwbqMk30lyfpJPJlm9pz+35z0jyQlJNu3pGyY5JsnZvbxXDMq6S5KjellHJbnzoKzTBp8/9jqtmeTIJOf0svYblPWqJD/o+z86yb0XOMaJ1zjJcYP9/izJZ3t6krwzyQV9Hw8dbPOKJGf2Or1ykP6vPe9pSb6W5M97+upJPpJkSf8tbj1fXSVJkiRpRdxsAhPA76tqs6p6EPBE4K+BN8y4TgtKssp866vqRVX1g5W82/2B5y1H/ucAJwGLp8y/AzBtMOUCYHuAHkh6PPDT5ajbyE+AXYBPTJH3SuD5/V7ZDnh7knX6uv8A3lZVGwO/Anbr6RcCj6uqhwD/Cry/p18DvLqqHgA8HNhjEJTZGzi6l3V0/05VHdzv1c1o12FpVZ3Wt/nPqro/sDnwqCRP6emnAlv0/X8GeMsCxzjxGlfVYwb7PhE4vK96CrBx/7wEeC9AkgcDLwa2BDYFnpZk49E+quohvawvAq/v6S/u+9qE9lv8r5siSChJkiRJcPMKTFynqi6lNa727E+CV0myf5Lv9Se81z35T/KaQfq/9LRF/an1R3v6Z5KsOdf+kvxZkg/3ck5Nsv2gnOP6k/xTkjyyp2/dn7J/AljSv3+z7+ecJAcnSc97XQ+NJL9N8u/9KfRJSdbr6X/Rv38vyZuyQC+DqjoauGI5Tuli4NXABknWHxz3bwfLz07rAfFI4BnA/v1J+l/03gAn9XN5xKjnQHcIMOqxsDVwPK2xv1yqamlVnQH8cYq851XV+X35Z8ClwN36OX8CreEP8FFakIWqOqGqftXTTwI26OmXVNUpffkK4GxgdI6272Vcr6wxi2nngKq6sqqO6ctXA6cM9nNMVV05vv95jnHea5zWS+UJwGcHdf1YNScB6yS5J/AA4KRet2uAbwE79n38ZlDknwHVlx9IC8SMfou/Bm7QyyjJS5KcnOTka6+8fL7DkSRJkqQ53SwDEwBV9SNa/e5Oe+p9eVX9FfBXwIvTuuw/ifaEeEtgM+BhSR7bi7gf8P7+hPo3wO7z7G5f4Bu9/MfTGuV/RmvwPrGqHkprfL9zsM2WwL5VNXq6vjnwSlqj7j7Aoybs589ojcRNgWPpT6aBdwDv6Pv/2ULnZnkk2RC4R1V9F/gUy4IIE1XVCcDngdf0J/M/BD4G7NXP5RKu35PlfFpQ4M60RvqhK7P+C0myJbA68EPgrsCvewMc4GKWBRmGdgO+PKGsRbTr+J2etF5VXQItgEG7F8ftRA9MjJW1DvB0egN/mv0vpx1pvTlGwYX1gYsG60fHfibw2CR37cG5vwY2HNTz35NcBDyXZT0mTge2T7Jqko2Ahw23Gamq91fVFlW1xSprrr2ChyNJkiTptupmG5jo0v8+CXh+ktNojca70gIST+qfU2lPp+/f0wEuqqrj+/J/A4+eZz9PAvbu5X8TWAO4F7Aa8IEkS4BPc/3hDd+tqgvHvl9cVX8ETgMWTdjP1bQu8wDfH+R5RC8fphvKsDx2pgUkoAUNph3OAUCStYF1qupbPemjwGPHsh3e97MVcNyNr+ry6T0CPg7s2s97JmSrsW0eTwsM7DWWvhZwGPDKsZ4E8+1/K+DKqjpzLH1VWrDinT3ANlz3d7TeB/tPs495XNdTY1T0hDxVVWfThrccBXyFFnS4ZpBh36raEDgY2LMnf5gW2DgZeDtwAjeiF4wkSZIkTWPVWVdgLknuA1xL67UQ4P9V1VfH8jwZeHNVvW8sfRFjDdIJ36+3CfCsqjp3rJw3Av9DG5t/O+D/Bqt/N1bGVYPla5l8bv9QVbVAnpVtMbBekuf273+eZOM+FGJ4TtZYgX0cSgsMfbSq/thHsdykktwJOBJ4XR+6AHAZbQjDqr3XxAYMeqAkeQjwQeApVfW/g/TVaEGJg6tqNGcDwP8kuWdVXdKDIJeOVWNnJvSWoM1fcX5VvX2sztvSeuc8rqqumrDdVJLcldZjZ8dB8sVcv1fDdcdeVR8CPtS3/f963nGfoJ3PN/Rz9w+D/Z1A6xkjSZIkSSvdzbLHRJK7AQcC7+4N+a8CL+sNSJL8ZR9q8VXghf1pN0nWTzLqbn+vJI/oy4uBb8+zy68C/28wL8TmPX1t4JL+NP55wLwTXa6Ak4Bn9eWdV1ahSe4H/FlVrV9Vi6pqEfDmwT7+J8kD0iY2HDZyrwDuCFBVlwO/SvKYvu55tHkKrlNVP6E1uA9YWXWfT9qbNo6gzakw6mlCv1eOAUZv9XgB8Lm+zb1oPTueV1XnDcoKrdF+dlW9dWxXn+9lXK+svt3taJOKXm/oSpJ/o903rxxL3xx4H/CMPm/DingO8MWqGgbKPk/rVZQkD6cNfbqk7/vu/e+9gGfSgylZNgkmtHlFzunpa/bfF0meCFxzE0zgKkmSJEnAzavHxB36UIrVaN3GPw6MGoofpA17OKU3JH8B7FBVX0vyAODEHlP4LfB3tN4IZwMvSPI+2tPe9w72dWSSP/TlE4Hn07qsn9HLXwo8jdbQPizJc2gN3vFeEivLK4H/TvJq2lPreWcSTHIcbdjKWkkuBnYb703SLaY14IcOozWm/5X2lokv0uYmOBNYq+c5lDaE5eW0Rv4LgAP7HAU/AnYd39F4r5WBNXsdR946IQBAkr/qdb0z8PQk/9LfujHJ39CGk9w1yS49bZf+Zoy9gEN7gOBUek8B2vwJdwUO6PfKNVW1BW0ukOfRJjE9red9bVV9CdgP+FSS3WhvDXnOoA6PBS4eDtVIsgEtQHMO7V6FFlz7IG3oxlrAp3v6T6rqGXMc30LXeOdet6Ev0eaPuID21pLhNTqs97L4A7DHYBLQ/Xrw6o/Aj4GX9vS7A19N8kfaG1aW5w0wkiRJkrRcsmxkwa1HH8rxxap68KzrMo3e4P99VVWSnYHFVbX9rOslTetl+765vnztQ2ZdjeWydL+nzroKkiRJ0m3JnGP+b049Jm7LHga8u/fW+DXwwtlWR5IkSZKkP41bZWCiqpYCt4jeEgBVdRxtgs3rJNmENpxl6Kqq2mpSGUmOADYaS95rjiEeM5NkX64/JALg01X17xPyLtc5uKW5tR+fJEmSJE3jVhmYuDWoqiXAZsuRf8eFc81eD0DcIAgxR97lOge3NLf245MkSZKkadws38ohSZIkSZJuGwxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmbGwIQkSZIkSZoZAxOSJEmSJGlmDExIkiRJkqSZWXXWFZB0y7fJ+mvz3t2fOutqSJIkSboFsseEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmbGwIQkSZIkSZoZAxOSJEmSJGlmDExIkiRJkqSZMTAhSZIkSZJmxsCEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmbGwIQkSZIkSZoZAxOSJEmSJGlmDExIkiRJkqSZMTAhSZIkSZJmxsCEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAxEqU5NokpyU5K8npSV6V5HZ93dZJLk9yapJzkvznYLtdkvyib3tako/Ns4/n9PL/mGSLKev1jiQ/HdVlgbzrJNl9inyLklSSfx2krZvkD0ne3b+/Mck/TlnHDye5NMmZU+Tdv5/DM5IckWSdwbp9klyQ5NwkT+5payY5sm9zVpL9BvlfleQHvayjk9x7sO4FSc7vnxcM0o8bXKufJflsT39uL+eMJCck2bSnb5jkmCRn9/2/YoHjm3iNe/mnDT5/TLJZX/ewJEv6sb8zSXr6vftxnZHkm0k2GKR/f3C/vnSwnyckOSXJmUk+mmTVha6JJEmSJN1YBiZWrt9X1WZV9SDgicBfA28YrD+uqjYHNgeeluRRg3Wf7NtuVlXPn2cfZwLPBI6dpkI9GLEjcBHw2Ck2WQdYMDDR/Qh42uD7c4Czptx23EHAdlPmPQp4cFU9BDgP2AcgyQOBnYEH9bIOSLJK3+Y/q+r+tHP/qCRP6emnAlv0sj4DvKWXdRfatdsK2BJ4Q5I7A1TVY0bXCjgROLyXdSHwuF7WvwLv7+nXAK+uqgcADwf26HWdy8RrXFUHD/b7PGBpVZ3WV78XeAmwcf+MzuV/Ah/rdXoT8OaefgnwyF7WVsDeSf683y8fBXauqgcDPwauC8pIkiRJ0spmYOImUlWX0hqKe46eXg/W/R44DVj/RpR7dlWduxybPJ7W0H0vsHiUON6boT8dXwTsB/xFf5K+f5r9+/olSXYalP174OzBU/2dgE8t7zH14zoW+OWUeb9WVdf0rycBG/Tl7YFDq+qqqroQuADYsqqurKpj+rZXA6eMtqmqY6rqygllPRk4qqp+WVW/ogVDrhc4SXJH4AnAZ3tZJ/S81yurqi6pqlP68hXA2cxz7ae8xouBQ3o97gncqapOrKoCPgbs0PM9EDi6Lx/TzxFVdXVVXdXTb8+yfwvuClxVVef170cBz5pUgSQvSXJykpMvu+yyBaorSZIkSZMZmLgJVdWPaOf47sP0/uR9Y67/RHynQRf9XVdiNUYN2CNovTRWWyD/3sAP+5P519Ce3G8GbApsC+zfG8IjhwI79yEC1wI/W4l1n8YLgS/35fVpPUNGLmYsANCHfTydZY31od2WpyxaT5Sjq+o3C5Q13P8iWq+N70zYZnnsRA9M9HpdPEddT2dZYGFH4I5J7trrsmGSM2jH+R9V9TPgMmC1QbDp2cCGkypQVe+vqi2qaot11113BQ9HkiRJ0m2VgYmb3rC3xGN6Q/DnwBer6ueDdcOhHB9ZKTtOVqcNJ/lsbzx/B3jSchbzaOCQqrq2qv4H+BbwV4P1X6ENW1kMfHLFaz29JPvShkkcPEqakK0G+VelNebf2YNGw7L+DtgC2H+asrrrei2MlfV4WmBir7H0tYDDgFfOEcyYSpKtgCurajQfx3x1/UfgcUlOBR4H/JR2zqiqi/oQj/sCL0iyXu9xsTPwtiTfBa4Y5ZckSZKkm4KBiZtQkvvQehFc2pOO6w3BTYCXjSYuvAltB6wNLEmylBZkGA3nuIbrX/815ihjUqP3On1oxPeBV9Ma3X8SfTLKpwHP7Y1paD0Fhk/3N+D6PTjeD5xfVW8fK2tbYF/gGYPhDfOW1XsdbAkcOVbWQ4APAttX1f8O0lejnZ+Dq+pwVszOXD8gcjHLhqBcr65V9bOqemaf22Tfnnb5sLDeU+Is4DH9+4l9Ho0tab16zl/B+kqSJEnSnAxM3ESS3A04EHj3oOEMQB+//2bGnqjfBBYDL6qqRVW1CNgIeFKSNYGlwEN7XR/a10F7Qn7HQRnH0oaZrNKP6bHAd8f281/AXsOG+E0pyXa0c/eMwfwQAJ+nDSu5fZKNaMNlvtu3+TdakOaVY2VtDryvl3XpYNVXaefqzn3ozZN62shzaL1e/m9Q1r1oE2E+bzBHA32OkQ8BZ1fVW1fw2G/X933oKK2qLgGuSPLwvq/nA5/r+dfNsrex7AN8uKdvkOQOffnOwKOAc/v3u/e/t6ed5wNXpM6SJEmSNB8DEyvXHUavXwS+DnwN+Jc58h4IPLY3oKeWZMckFwOPAI5M8tU58q1Jm8Dxuif6VfU74Nu0ORYOA+6S5DTgZbS3W9CDC8f3yS73p81NcQZtroJvAP80NgSFqjqrqj46R5Vfl+Ti0Wee4zqE9oaL+/W8u81zGt5NC54c1c/3gaN60Cbf/AFtiMkeVXVtn/9iX9pEkKf0bV7Uy9ofWAv4dE//fC/rl7Q3a3yvf97U00bGey0AvJ42eeQBvayTe/qjaG/ReMJgHpG/nudczHeNHwtcPD4UhXYNP0ib8POHLJvfYmvg3CTnAesB/97THwB8J8nptOE5/1lVS/q61yQ5m3bdv1BV35irrpIkSZK0ojL2MF+SltsBBxxQu+8+7VtmJUmSJN0GzTlNgD0mJEmSJEnSzKw66wposiTvoQ0BGHrHpDd2JHky8B9jyRdW1Y43Vf1ujD5h5KTXdG4zaX6K5TkHt0S39uOTJEmSpGkYmLiZqqo9liPvV7n+xIw3Sz34sNly5J/6HNwS3dqPT5IkSZKm4VAOSZIkSZI0MwYmJEmSJEnSzBiYkCRJkiRJM2NgQpIkSZIkzYyBCUmSJEmSNDMGJiRJkiRJ0swYmJAkSZIkSTNjYEKSJEmSJM2MgQlJkiRJkjQzBiYkSZIkSdLMGJiQJEmSJEkzY2BCkiRJkiTNjIEJSZIkSZI0MwYmJEmSJEnSzBiYkCRJkiRJM2NgQpIkSZIkzYyBCUmSJEmSNDMGJiRJkiRJ0swYmJAkSZIkSTNjYEKSJEmSJM2MgQlJkiRJkjQzBiYkSZIkSdLMGJiQJEmSJEkzY2BCkiRJkiTNjIEJSZIkSZI0MwYmJEmSJEnSzBiYkCRJkiRJM2NgQpIkSZIkzYyBCUmSJEmSNDMGJiRJkiRJ0swYmJAkSZIkSTNjYEKSJEmSJM2MgQlJkiRJkjQzBiYkSZIkSdLMGJiQJEmSJEkzY2BCkiRJkiTNjIEJSZIkSZI0MwYmJEmSJEnSzBiYWAmSrJfkE0l+lOT7SU5MsmOSrZNcnuS0JGck+XqSu/dtdklSSbYZlLNjT3v2PPs6OMm5Sc5M8uEkq01Rv88lOXHKY1mU5G+nyLd1r+tug7TNe9o/9u8HzXcsY+V9Jcmvk3xxirwTz0Gadya5oJ/vh/b0DZMck+TsJGclecWgrP2TnNPzH5FkncG6fXpZ5yZ5ck+7Y7+eo89lSd7e170qyQ96WUcnuXdP36zfE2f1dTstcHx79v1WknUH6a8Z7PfMJNcmuUtft12v5wVJ9h5ss2nf95IkX0hyp56+5aCs05PsONhmp17Ps5K8ZaHrIUmSJEkrwsDECkoS4LPAsVV1n6p6GLAzsEHPclxVbVZVDwG+B+wx2HwJsHjwfWfg9AV2eTBwf2AT4A7Aixao3zrAQ4F1kmw0xSEtAhYMTHRLgGEje5r6z2V/4HlT5p3rHDwF2Lh/XgK8t6dfA7y6qh4APBzYI8kD+7qjgAf363MesA9AX78z8CBgO+CAJKtU1RX9em5WVZsBPwYO72WdCmzRy/oMMGrUXwk8v6pGZb19GACZ4Hhg2172dapq/8F+9wG+VVW/TLIK8J5+/A8EFg+O74PA3lW1CXAE8Jqefmav62a9Tu9LsmqSu9KuxTa9vusNg2eSJEmStLIZmFhxTwCurqoDRwlV9eOqetcwUw9g3BH41SD5OGDLJKslWQu4L3DafDurqi9VB3yXZQGQuTwL+AJwKK2hParP9XozJPltX9wPeEx/kv4PSdZI8pH+xP3UJI8flP0TYI20HiOhNXC/vEB95jquo4Erpsw71znYHvhYX3USLRhzz6q6pKpO6dteAZwNrN+/f62qrunbnzRW1qFVdVVVXQhcAGw5rEeSjYG7064jVXVMVV05XlZVnVdV5/flnwGXAneb5/hOraqlC5yGxcAhfXlL4IKq+lFVXU271tv3dfcDju3LR9HuB6rqysFxrwFUX74PcF5V/aJ///pom3FJXpLk5CQnX3bZZQtUV5IkSZImMzCx4h4EnDLP+sckOY3WiN8W+PBgXdEafk+mNSQ/P+1O+/CF5wFfWSDrqAF7CNfvnTGXvVnWy+Nt9B4e/Yn7YuCjSdYY5P8M8BzgkbTzcNW0x7CiJpyD9YGLBlku7mnDbRYBmwPfmVDkC1kWWFmwLNr5+GQPkIzbjQlBmiRbAqsDP5ywzVSSrEkLAh02RV3PBJ7Rl58DbDgoZ6skZ9F6vry0ByouAO6fNqRnVWCH4TZDVfX+qtqiqrZYd911J2WRJEmSpAUZmFjJkrynj9n/Xk8aNfI3BD7Csu79I6OeDDuz7An4NA6gDR85bp66rEfrhfHtqjoPuCbJg5djHwCPBj4OUFXn0IYX/OVg/adoDd7hE/w/lfFzkAl5rgsa9F4phwGvrKrfDDMl2Zc25OPgacrqJl6zJH8HbEEbEjFMvyftXO5aVX+c45im8XTg+Kr65RR1fSFt6Mr3aT12rr4uQ9V3+nCNvwL2SbJGVf0KeBnwSVpPkKW08yJJkiRJNwkDEyvuLNocDgBU1R7ANkzuqv954LHDhKr6LvBgYN0ePFhQkjf08l+1QNadgDsDFyZZSps/YjSc4xr69e/DMFafa3fz7aCqfg78AXgicPTCtV855jgHF3P9p/sbAD/r+VejBSUOrqrDB3lI8gLgacBzB70f5iyrb7MpsGpVfX+srG2BfYFnVNVVg/Q7AUcCr+vDTFbEeEBkzrpW1TlV9aQ+98khTOipUVVnA7+j3YdU1ReqaquqegRwLnD+CtZXkiRJkuZkYGLFfYM2z8LLBmlrzpH30Uzuwr8P8NppdpbkRbShH4uneOq+GNiuqhZV1SJgNDEntCfhD+vL2wOjt3tcQXuyPnIs8Ny+778E7kVrrA69Htirqq6d5hhW1Dzn4PPA89M8HLi8qi7pgZcPAWdX1VvHytoO2IsWSLhyrKydk9y+Txq6MW0+i5Eb9BBJsjnwvl7WpYP01WkTT36sqj69gse+NvA44HOD5O8BGyfZqO9r515/suwtMLcDXgcc2L9v1IdqkPb2kPvR7onhNncGdqdNoClJkiRJN4lVZ12BW7qqqiQ7AG9L8k/AL2hPn/fqWUZzTAS4nAlv0aiq5Zkw8kDacIoTW3ubw6vqTeOZ+lwK96JNwjjaz4VJfpNkK+ADwOeSfJfW0+F3PdsZtCEfpwMH0YZLHJhkCa2XxS5VdVXf96jcE+ap7/vSX6cJXNSfwt9AkuNob9pYK8nFwG5V9dXlPAdfAv6aNk/ClcCuPf+jaHNRLOnXAuC1VfUl4N3A7YGjelknVdVLq+qsJJ8CftCPe4+xwMvf9H0N7Q+sBXy6l/WTqnpGz/tY4K5Jdul5d6mq05ggycuBfwLuAZyR5EtVNbpvdgS+VlWj60VVXZNkT+CrwCrAh6vqrL56cZLRm2AOpw0nghYk2zvJH4A/ArtX1WgGy3f0HiEAb5q2J48kSZIk3RiZPG+fJE3vgAMOqN13333W1ZAkSZJ08zXnNAEO5ZAkSZIkSTPjUI6bqSRHABuNJe81aXhDkl2BV4wlH98n4rzZSLIJ/Q0fA1dV1VZz5J/6HNwS3dqPT5IkSZKmYWDiZqqqdlyOvB9h2dwBN1tVtQTYbDnyT30Obolu7ccnSZIkSdNwKIckSZIkSZoZAxOSJEmSJGlmDExIkiRJkqSZMTAhSZIkSZJmxsCEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmbGwIQkSZIkSZoZAxOSJEmSJGlmDExIkiRJkqSZMTAhSZIkSZJmxsCEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmbGwIQkSZIkSZoZAxOSJEmSJGlmDExIkiRJkqSZMTAhSZIkSZJmxsCEJEmSJEmaGQMTkiRJkiRpZgxMSJIkSZKkmTEwIUmSJEmSZsbAhCRJkiRJmhkDE5IkSZIkaWYMTEiSJEmSpJkxMCFJkiRJkmbGwIQkSZIkSZoZAxOSJEmSJGlmDEysZEk2THJhkrv073fu3x+X5PdJTkvygyQfS7Je/35akp8n+eng++pzlP/hJJcmOXPK+qya5LIkb54y/9ZJHjlFvjcmqST3HaT9Q0/bon9fmmTdKcq6f5ITk1yV5B8XyLthkmOSnJ3krCSvGKy7S5Kjkpzf/965pz8xyfeTLOl/n9DT10xyZJJzeln7Dcq6fZJPJrkgyXeSLOrpjx9co9OS/F+SHfq6g5Ocm+TMfp1W6+nPTXJG/5yQZNMFjnHiNe71Ge13aZLTBuv26XU9N8mTB+k79f2eleQtg/SX9vNxWpJvJ3ngYN1/9GM4M8lO89VVkiRJklaUgYmVrKouAt4LjBq5+wHvB34M/LCqNgM2ATYAtq2qzXragcDbRt+r6uo5dnEQsN1yVOlJwLnA3yTJFPm3BhYMTHRLgJ0H358N/GA56jbyS+DlwH9Okfca4NVV9QDg4cAeg0b13sDRVbUxcHT/DnAZ8PSq2gR4AfDxQXn/WVX3BzYHHpXkKT19N+BXVXVf4G3AfwBU1TGDa/YE4Erga32bg4H7067vHYAX9fQLgcdV1UOAf6XdD/M5iAnXuKp2Guz7MOBwgH78OwMP6tsdkGSVJHcF9ge2qaoHAesl2aYX94mq2qSX9Rbgrb2spwIPBTYDtgJek+ROC9RXkiRJkm40AxM3jbcBD0/ySuDRwH8NV1bVtcB3gfWXt+CqOpbWkJ/WYuAdwE9oDXng+r0ZkmyR5Ju9V8BLgX/oT9Ifk+TeSY7uT92PTnKvQdmfBbbvZdwHuBz4xY04pkur6nvAH6bIe0lVndKXrwDOZtl53B74aF/+KLBDz3dqVf2sp58FrJHk9lV1ZVUd0/NcDZxCCxiNl/UZYJsJgZ1nA1+uqit7GV+qjnZ9N+jpJ1TVr/o2Jw32MdcxznuNez3+BjhkUNdDq+qqqroQuADYErgPcF5Vja7J14Fn9X38ZlDknwHVlx8IfKuqrqmq3wGnM0cgLMlLkpyc5OTLLrtsvkOSJEmSpDkZmLgJVNUfgNfQAhSvHO/9kGQN2tPor9yU9UhyB2Ab4Iu0Ruzi+fJX1VKu33PjOODdwMf60/6DgXcONvkNcFGSB/eyP7nSD2IePZCyOfCdnrReVV0CLYAB3H3CZs8CTq2qq8bKWgd4Oq2nBbRgx0W9rGtoQZe7jpW1M8uCA8OyVgOex+Truxvw5fmPbEGPAf6nqs4fr2t3cU+7ALh/kkVJVqUFajYc1HOPJD+k9Zh4eU8+HXhKH+ayLvD44TZDVfX+qtqiqrZYd90FR+xIkiRJ0kQGJm46TwEuAR48SPuLPi/A/wI/qaozbuI6PA04pj/RPwzYMckqy1nGI4BP9OWP03qADB1Ka6DvABxx46u6fJKsRTumV449/Z9vmwfRhmT8/Vj6qrQAwzur6kej5AlF1GCbe9KGbHx1Qr4DgGN7YGe4n8fTAhN7TVPfeSzm+gGRiXXtvTReRgsYHQcspQ2FGWV4T1X9Ra/P63ra14AvASf0fZw43EaSJEmSVjYDEzeBJJsBT6QNnfiH3oiFZXNM3Jc21OMZN3FVFgPbJlkKfJ/2xP/xfd01LLv+ayxHmTX2/Qu03gE/mTZAsKJ6j4TDgIOr6vDBqv8Znev+99LBNhvQAifPr6ofjhX5fuD8qnr7IO1iek+BHrhYm+sPr/gb4IjeO2ZYtzcAdwNeNZb+EOCDwPZV9b/LdcDXL2dV4Jlcv3fKdXXtNgB+BlBVX6iqrarqEbS5Rs7nhg6lD3vp2/x77zHzRFrQY9I2kiRJkrRSGJhYyfr4//fSnuT/hDb54PUmdezDDPYG9rkJ63EnWu+Ge1XVoqpaBOzBsuEcS4GH9eVnDTa9Arjj4PsJLJvg8rnAt4f7qarf0564//tKrP6c+vn9EHB2Vb11bPXnaZNb0v9+rm+zDnAksE9VHT9W3r/Rgg6vnKesZwPf6HNHjIz3WiDJi4AnA4ur6o+D9HvRJqp8XlWdN+2xzmFb4Jyqunisrjv3N4lsBGxMm+OCJHfvf+8M7E4LjpBk48H2T6UHHwaTZo6CKQ9h2eSekiRJkrTSGZhY+V5M6z1wVP9+AO1NDfcey/dZYM0kj1mewpOMutffL8nFSXabI+szaY3p4VwKnwOekeT2wL8A70hyHHDtIM8XaEM+Tut1ezmwa5IzaD0jXsGYqjp0NCHlBGf0el6cZDyQMDqmeyS5mNbL4HU971xvgnhUr8cTBq/O/Ou+bj/giUnOp/VYGb0ZZU9aL5V/Hmxz996LYl/ahI+n9PTRmzQ+BNw1yQW9XqM3fIzmttgQ+NZY3Q4E1gNO7GW9vqe/ntZb5YCefvIcxzYqf75rfIN5LarqLOBTtDeifAXYo0+wCu0a/wA4HthvEBjZM+0Voqf14xsFYVYDjuvbvB/4uz7HhiRJkiTdJHL9h8CStPwOOOCA2n333WddDUmSJEk3X5PmxgPsMSFJkiRJkmZo1VlXQDfUx/gfPWHVNpMmTkzyHtoQh6F3VNVHbor63VhJduWGQ0GOr6o9JuRdrnNwS3NrPz5JkiRJmpaBiZuh3jDdbDny36Bhf3PUAyVTBUuW9xzc0tzaj0+SJEmSpuVQDkmSJEmSNDMGJiRJkiRJ0swYmJAkSZIkSTNjYEKSJEmSJM2MgQlJkiRJkjQzBiYkSZIkSdLMGJiQJEmSJEkzY2BCkiRJkiTNjIEJSZIkSZI0MwYmJEmSJEnSzBiYkCRJkiRJM2NgQpIkSZIkzYyBCUmSJEmSNDMGJiRJkiRJ0swYmJAkSZIkSTNjYEKSJEmSJM2MgQlJkiRJkjQzBiYkSZIkSdLMGJiQJEmSJEkzY2BCkiRJkiTNjIEJSZIkSZI0MwYmJEmSJEnSzBiYkCRJkiRJM2NgQpIkSZIkzYyBCUmSJEmSNDMGJiRJkiRJ0swYmJAkSZIkSTNjYEKSJEmSJM2MgQlJkiRJkjQzBiYkSZIkSdLMGJiQJEmSJEkzY2BCkiRJkiTNjIEJSZIkSZI0MwYmJEmSJEnSzBiYkCRJkiRJM2NgQpIkSZIkzYyBCUmSJEmSNDMGJiRJkiRJ0swYmJAkSZIkSTNjYGIlSbJekk8k+VGS7yc5McmOSbZO8sUJ+b+Z5CdJMkj7bJLfLrCfryT59aQy58h/tyR/SPL3U+bfIckDp8h3UJIrk9xxkPaOJJVk3f593mMZbPfYJKckuSbJsxfIu1k/t2clOSPJToN1GyX5TpLzk3wyyeo9/bk97xlJTkiyaU/fMMkxSc7u5b1iUNZdkhzVyzoqyZ0HZZ02+Pyx12nNJEcmOaeXtd+grFcl+UHf/9FJ7r3AMU68xkmOG+z3Z0k+29OT5J1JLuj7eOhgm1ckObPX6ZWD9H/teU9L8rUkf97TV0/ykSRLkpyeZOv56ipJkiRJK8rAxErQgwufBY6tqvtU1cOAnYENFtj018CjehnrAPecYnf7A89bjuo9BzgJWDxl/h2ABQMT3QXA9gBJbgc8HvjpctRt5CfALsAnpsh7JfD8qnoQsB3w9n7uAP4DeFtVbQz8Ctitp18IPK6qHgL8K/D+nn4N8OqqegDwcGCPQVBmb+DoXtbR/TtVdXBVbVZVm9Guw9KqOq1v859VdX9gc+BRSZ7S008Ftuj7/wzwlgWOceI1rqrHDPZ9InB4X/UUYOP+eQnwXoAkDwZeDGwJbAo8LcnGo31U1UN6WV8EXt/TX9z3tQnwROC/+rWVJEmSpJuEDY6V4wnA1VV14Cihqn5cVe9aYLtDaQEMgGeyrKE5p6o6GrhiOeq2GHg1sEGS9UeJw94MSZ7de0A8EngGsH9/kv4XvTfASf3p+hGjngPdIcCox8LWwPG0xv5yqaqlVXUG8Mcp8p5XVef35Z8BlwJ368GhJ9Aa/gAfpQVZqKoTqupXPf0kesCoqi6pqlP68hXA2cDoHG3fy7heWWMW084BVXVlVR3Tl68GThns55iqunJ8//Mc47zXuPdSeQItGDaq68eqOQlYJ8k9gQcAJ/W6XQN8C9ix7+M3gyL/DKi+/EBaIIaqupQWPNtijnq8JMnJSU6+7LLL5jskSZIkSZqTgYmV40G0hujyOhp4bJJVaAGKT67MSiXZELhHVX0X+BTLgggTVdUJwOeB1/Qn8z8EPgbs1Z/2LwHeMNjkfFpQ4M60RvqhK7P+C0myJbA68EPgrsCvewMc4GKWBRmGdgO+PKGsRbSeDt/pSetV1SXQAhjA3SeUtRM9MDFW1jrA0+kN/Gn2v5x2pPXmGAUX1gcuGqwfHfuZtPvrrknWBP4a2HBQz39PchHwXJb1mDgd2D7Jqkk2Ah423Gaoqt5fVVtU1RbrrrvuCh6SJEmSpNsqAxM3gSTv6ePzv7dA1muBb9MauHeoqqUruSo70wIS0IIG0w7nACDJ2sA6VfWtnvRR4LFj2Q7v+9kKOO7GV3X59B4BHwd2rao/ApmQrca2eTwtMLDXWPpawGHAK8d6Esy3/62AK6vqzLH0VWnBindW1Y/G1v0drffB/tPsYx7X9dQYFT0hT1XV2bThLUcBX6EFHa4ZZNi3qjYEDgb27MkfpgU2TgbeDpzAjegFI0mSJEnTMjCxcpwFXDfhYFXtAWwD3G2KbQ8F3sWyAMLKtBjYJclSWk+ITQdzDAwb7WuswD4Opc3bcFQPENzkktyJ/7+9+4+xqkwPOP59dGbcyrgMQjZUfjhojC5ptaitqHQ1VUHboh2bVUBlGcRisd11U7MuK8Q0wbh/bOi2UdQtAhJ0jTq7iiv+SuQP4o91KZjiqitmqAxFBVFYxkYBefvHPcwOw1yYuWdmjjN8P8lE55z3Pec5z72cd+5zz3sOPAPMy6YuAHxMaQpDVfb7SGBruz5nAouBq1JKO9otr6ZUlHg4pdR+Ks1HWfHjQBFkW4cwptDJ1RKU7l+xMaX00w4xXwrcAVyZUvqiG4d7kIgYSumeEc+0W7yFg69qaDv2lNKDKaWzU0rfAj6hdJVLR48Af5+135dS+n52xcxVQF2ZPpIkSZLUIyxM9IyXgK9FxD+2W3Z8F/uuAe6m8w+5FYuI04FBKaURKaX6lFJ9tp8D97T4KCK+md3YsKFd193ACQAppV3ApxHxl9m6Gyjdp6BNSmkzpQ/ci3oy/nKi9KSNX1K6p8Lj7eJIwGrgwFM9vgM8lfUZTenKjhtSSu+221YADwJvp5QWdtjVymwbB20r63cMpZuKHjR1JSIWAIOBWzssHwc8QKko0bHA0V3fBn6VUvq8Q6zTs6dzjAd2HZiGEhHfyP47mtJ9TH6e/X5au/5XAu9ky4+PiEHZ/18G7EspvZUzZkmSJEkqq+rITXQkKaUUEX8H/FtE/ADYDnzGH6YMXBIRW9p1+Xb7vsBPurqviFgDnAHUZtu8MaX0fCdNp1L6AN9eE3+4wuGHlJ7G0ELpXgS1WZtHgf+MiO9S+pD/HeD+7B4FzUBjxx2llB4oE+7xHY57YScFACLiz7NYhwCTI+Jfs6dudOYaStNJhkbEjGzZjOzJGLcDj2YFgvWUig5Qun/CUGBRqRbBvpTSuZSeiHIDsCEi3sja/iiltAr4MfBYRNxI6akhba9Ztv8t7adqRMRISgWad4B12X7uSSktpjR1oxZ4PFu+OaV0ZZnjO9JrPCWLrb1VlO4f8R6lp5a0f42asqss9gK3tLsJ6I+z4tV+4H3g5mz5N4DnI2I/pSesdOcJMJIkSZLUbVH6XCxJlVu0aFGaM2dO0WFIkiRJ+urq7N54gFM5JEmSJElSgZzK8RUUEX9K6YkT7X2RUjqvTPtfAmM6LL69zBSPwkTEHRw8JQLg8ZTSXZ207VYO+puBfnySJEmS1FUWJr6CUkobgD/rRvuGI7cqXlaAOKQIUaZtt3LQ3wz045MkSZKkrnIqhyRJkiRJKoyFCUmSJEmSVBgLE5IkSZIkqTAWJiRJkiRJUmEsTEiSJEmSpMJYmJAkSZIkSYWxMCFJkiRJkgpjYUKSJEmSJBXGwoQkSZIkSSqMhQlJkiRJklQYCxOSJEmSJKkwVUUHIEkdnbvgRT5u3XPEdsNqa1g777I+iKgyN998M1VVVdxzzz252kiSJEkDmVdMSPrK6UpRojvtuuLiiy/muOOOo7a2lsGDBzNu3DiamppybfP+++8/qOBQX1/PihUrDttGkiRJOtpYmJCkzPz582ltbWXHjh1MnTqVa6+9lnfffbfosCRJkqQBzcKEJHVQVVXFnDlz+PLLL9mwYQP33Xcfp59+OoMHD2b8+PGsWbOmre369euZMGECgwcP5sQTT+SCCy7g008/BWDGjBnMmjULgMmTJ7N582ZmzZpFbW0tEydOPKTNbbfdRkNDw0GxrF69mhNOOIHPPvsMgDfffJNJkyYxbNgwRo8ezdy5c9m7d2+v50SSJEnqLRYmJKmDPXv2cO+991JdXc1bb73F/PnzWb58OTt27OCmm27i8ssv5/333wfglltuYeLEiXzyySd89NFHLFy4kJqamkO2+fTTTzN69GgWL15Ma2srL7zwwiFtZs6cyTPPPMP27dvbli1btoxrrrmGQYMGsW3bNi666CKuvvpqtm7dyquvvsqLL77I3Xff3XvJkCRJknqZhQlJytx1113U1dUxcuRInnrqKZqamlizZg2zZ8/mvPPOo6qqihtvvJEzzzyTRx55BICamho2b95MS0sL1dXVjB8/nkGDBlW0/7FjxzJu3Li2+1Ds3r2bpqYmZs6cCcDy5cs566yzmD17NjU1NYwYMYK5c+eyfPnynkmAJEmSVAALE5KUueOOO9i5cyfbtm3jlVdeYfLkybS0tHDKKacc1O7UU0+lpaUFgKVLl7J//34mTJjAmDFjmD9/Pvv27as4hsbGRpYuXQrAY489xogRI7jwwgsB2LRpEy+//DJ1dXVtPzNnzuTDDz+seH+SJElS0SxMSNJhjBo1ik2bNh20rLm5mVGjRgEwZswYlixZwpYtW1i5ciWLFy8uewXDMccc+ZQ7ZcoUNm7cyLp161i2bBmNjY1t604++WQuvfRSdu7c2faza9cuWltbcxyhJEmSVCwLE5J0GDNmzOCBBx7g9ddfZ9++fSxbtow33niDqVOnAvDQQw+xdetWAOrq6qiqqqKqqqrTbQ0fPpyNGzcedn91dXU0NDQwb948XnvtNaZPn962bvr06axdu5YlS5bw+eefs3//fpqbm3nuued66GglSZKkvmdhQpIOY9q0adx5551cf/31DB06lEWLFrFq1Srq6+sBeOmllzjnnHOora3l/PPPZ9q0aVx33XWdbmvevHmsWLGCIUOGcMUVV5TdZ2NjI88++yyTJk3ipJNOals+fPhwVq9ezZNPPkl9fT1DhgyhoaGB5ubmHj1mSZIkqS9FSqnoGCT1c4sWLUpz5szpse2du+BFPm7dc8R2w2prWDvvsh7bryRJkqReE+VWdH69sSQVyGKDJEmSdPRwKockSZIkSSqMhQlJkiRJklQYCxOSJEmSJKkwFiYkSZIkSVJhLExIkiRJkqTCWJiQJEmSJEmFsTAhSZIkSZIKY2FCkiRJkiQVxsKEJEmSJEkqjIUJSZIkSZJUGAsTkiRJkiSpMBYmJEmSJElSYSKlVHQMkvq522+/fXd1dfXvio7jaNTa2jqstrb246LjOBqZ++KY++KY++KY+2KZ/+KY++L0Qu4/XrBgweWdrbAwISm3iFibUjq36DiORua+OOa+OOa+OOa+OOa+WOa/OOa+OH2Ze6dySJIkSZKkwliYkCRJkiRJhbEwIakn/KzoAI5i5r445r445r445r445r5Y5r845r44fZZ77zEhSZIkSZIK4xUTkiRJkiSpMBYmJEmSJElSYSxMSCorIi6PiN9FxHsR8cNO1kdE/Ee2/r8j4uyu9tXhVZr7iBgVEasj4u2I+G1EfK/vo+/f8rzvs/XHRsT6iPhV30U9cOQ879RFxBMR8U72b+D8vo2+f8uZ++9n55w3I+LnEfG1vo2+f+tC7s+IiFcj4ouIuK07fXV4lebe8Ta/PO/7bL3jbYVynnN6Zay1MCGpUxFxLHAvcAUwFpgaEWM7NLsCOC37+Qfgvm70VRl5cg/sA/4lpfRNYDxwi7nvupy5P+B7wNu9HOqA1AP5/3fguZTSGcBZ+Dp0Wc5z/gjgu8C5KaU/AY4FpvRR6P1eF3P/CaUc/6SCviojT+5xvM0lZ+4PcLytQA/kvlfGWgsTksr5C+C9lFJzSmkP8ChwVYc2VwHLU8lrQF1E/HEX+6q8inOfUvogpbQOIKW0m9JgMaIvg+/n8rzviYiRwN8Ai/sy6AGk4vxHxNeBbwEPAqSU9qSUdvZh7P1drvc+UAX8UURUAccDW/sq8AHgiLlPKW1LKf0G2NvdvjqsinPveJtbnve9420+Fee+N8daCxOSyhkBtLT7fQuHDrjl2nSlr8rLk/s2EVEPjAN+3fMhDlh5c/9T4AfA/l6Kb6DLk/9TgO3A0uzS3sURMag3gx1gKs59Sul/KX2rthn4ANiVUnqhF2MdaPKMmY63+fRI/hxvK5I39z/F8bZSeXLfa2OthQlJ5UQnyzo+X7hcm670VXl5cl9aGVELNAG3ppR+34OxDXQV5z4i/hbYllL6r54P66iR571fBZwN3JdSGgd8BjjfvuvyvPeHUPq2bQxwEjAoIq7v4fgGsjxjpuNtPrnz53hbsYpz73ibW573fa+NtRYmJJWzBRjV7veRHHppbrk2Xemr8vLknoiopvRH0sMppV/0YpwDUZ7cXwhcGRH/Q+myyL+KiBW9F+qAlPe8syWldOAbyyco/fGkrsmT+0uBTSml7SmlvcAvgAt6MdaBJs+Y6XibT678Od7mkif3jrf55D3n9MpYa2FCUjm/AU6LiDERUUPpRmYrO7RZCUzP7tQ+ntLlux90sa/Kqzj3ERGU5v29nVJa2LdhDwgV5z6lNDelNDKlVJ/1eyml5LfG3ZMn/x8CLRFxetbuEuCtPou8/8tzzt8MjI+I47Nz0CV4Q7ruyDNmOt7mU3H+HG9zqzj3jre55cl9r421VT2xEUkDT0ppX0T8E/A8pTusL0kp/TYibs7W3w+sAv4aeA/4P6DxcH0LOIx+KU/uKX2LcAOwISLeyJb9KKW0qg8Pod/KmXvl1AP5/2fg4ewPrWZ8bbos5zn/1xHxBLCO0pMK1gM/6/uj6J+6kvuIGA6sBb4O7I+IW4GxKaXfO95WLk/ugTNxvK1Y3vd9UXEPBD2Q+14ZayMlp6FJkiRJkqRiOJVDkiRJkiQVxsKEJEmSJEkqjIUJSZIkSZJUGAsTkiRJkiSpMBYmJEmSJElSYSxMSJIkSZKkwliYkCRJkiRJhfl/p4qsusAVUEMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1008x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "metalearner.std_coef_plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "d08c1586",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Details\n",
      "=============\n",
      "H2OGeneralizedLinearEstimator :  Generalized Linear Modeling\n",
      "Model Key:  metalearner_AUTO_StackedEnsemble_AllModels_1_AutoML_1_20220722_170939\n",
      "\n",
      "\n",
      "GLM Model: summary\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>family</th>\n",
       "      <th>link</th>\n",
       "      <th>regularization</th>\n",
       "      <th>lambda_search</th>\n",
       "      <th>number_of_predictors_total</th>\n",
       "      <th>number_of_active_predictors</th>\n",
       "      <th>number_of_iterations</th>\n",
       "      <th>training_frame</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td>gaussian</td>\n",
       "      <td>identity</td>\n",
       "      <td>Elastic Net (alpha = 0.5, lambda = 0.002551 )</td>\n",
       "      <td>nlambda = 100, lambda.max = 0.2219, lambda.min = 0.002551, lambda....</td>\n",
       "      <td>10</td>\n",
       "      <td>7</td>\n",
       "      <td>49</td>\n",
       "      <td>levelone_training_StackedEnsemble_AllModels_1_AutoML_1_20220722_17...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       family      link                                 regularization  \\\n",
       "0    gaussian  identity  Elastic Net (alpha = 0.5, lambda = 0.002551 )   \n",
       "\n",
       "                                                           lambda_search  \\\n",
       "0  nlambda = 100, lambda.max = 0.2219, lambda.min = 0.002551, lambda....   \n",
       "\n",
       "   number_of_predictors_total number_of_active_predictors  \\\n",
       "0                          10                           7   \n",
       "\n",
       "   number_of_iterations  \\\n",
       "0                    49   \n",
       "\n",
       "                                                          training_frame  \n",
       "0  levelone_training_StackedEnsemble_AllModels_1_AutoML_1_20220722_17...  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "ModelMetricsRegressionGLM: glm\n",
      "** Reported on train data. **\n",
      "\n",
      "MSE: 0.21727419974117612\n",
      "RMSE: 0.4661268065035266\n",
      "MAE: 0.35448759076481845\n",
      "RMSLE: 0.11902010738109643\n",
      "R^2: 0.3428720464937892\n",
      "Mean Residual Deviance: 0.21727419974117612\n",
      "Null degrees of freedom: 3861\n",
      "Residual degrees of freedom: 3854\n",
      "Null deviance: 1276.9399854672324\n",
      "Residual deviance: 839.1129594004221\n",
      "AIC: 5082.170839083037\n",
      "\n",
      "ModelMetricsRegressionGLM: glm\n",
      "** Reported on cross-validation data. **\n",
      "\n",
      "MSE: 0.2185070080606413\n",
      "RMSE: 0.46744733185744175\n",
      "MAE: 0.35554501482828577\n",
      "RMSLE: 0.11935774800872184\n",
      "R^2: 0.3391435190891413\n",
      "Mean Residual Deviance: 0.2185070080606413\n",
      "Null degrees of freedom: 3861\n",
      "Residual degrees of freedom: 3853\n",
      "Null deviance: 1277.1043650451534\n",
      "Residual deviance: 843.8740651301968\n",
      "AIC: 5106.021797066896\n",
      "\n",
      "Cross-Validation Metrics Summary: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>sd</th>\n",
       "      <th>cv_1_valid</th>\n",
       "      <th>cv_2_valid</th>\n",
       "      <th>cv_3_valid</th>\n",
       "      <th>cv_4_valid</th>\n",
       "      <th>cv_5_valid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mae</td>\n",
       "      <td>0.355575</td>\n",
       "      <td>0.011150</td>\n",
       "      <td>0.358694</td>\n",
       "      <td>0.365597</td>\n",
       "      <td>0.336775</td>\n",
       "      <td>0.355430</td>\n",
       "      <td>0.361377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mean_residual_deviance</td>\n",
       "      <td>0.218685</td>\n",
       "      <td>0.018149</td>\n",
       "      <td>0.232207</td>\n",
       "      <td>0.226213</td>\n",
       "      <td>0.191451</td>\n",
       "      <td>0.209182</td>\n",
       "      <td>0.234372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mse</td>\n",
       "      <td>0.218685</td>\n",
       "      <td>0.018149</td>\n",
       "      <td>0.232207</td>\n",
       "      <td>0.226213</td>\n",
       "      <td>0.191451</td>\n",
       "      <td>0.209182</td>\n",
       "      <td>0.234372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>null_deviance</td>\n",
       "      <td>255.420870</td>\n",
       "      <td>18.944805</td>\n",
       "      <td>270.143950</td>\n",
       "      <td>279.337650</td>\n",
       "      <td>235.192600</td>\n",
       "      <td>252.053120</td>\n",
       "      <td>240.377060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>r2</td>\n",
       "      <td>0.339071</td>\n",
       "      <td>0.024654</td>\n",
       "      <td>0.332108</td>\n",
       "      <td>0.343113</td>\n",
       "      <td>0.367360</td>\n",
       "      <td>0.351331</td>\n",
       "      <td>0.301443</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>residual_deviance</td>\n",
       "      <td>168.764340</td>\n",
       "      <td>13.983558</td>\n",
       "      <td>180.424550</td>\n",
       "      <td>183.458680</td>\n",
       "      <td>148.757460</td>\n",
       "      <td>163.370830</td>\n",
       "      <td>167.810170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>rmse</td>\n",
       "      <td>0.467306</td>\n",
       "      <td>0.019674</td>\n",
       "      <td>0.481878</td>\n",
       "      <td>0.475618</td>\n",
       "      <td>0.437551</td>\n",
       "      <td>0.457364</td>\n",
       "      <td>0.484120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>rmsle</td>\n",
       "      <td>0.119310</td>\n",
       "      <td>0.004632</td>\n",
       "      <td>0.122437</td>\n",
       "      <td>0.121279</td>\n",
       "      <td>0.111271</td>\n",
       "      <td>0.119514</td>\n",
       "      <td>0.122048</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 mean         sd  cv_1_valid  cv_2_valid  \\\n",
       "0                     mae    0.355575   0.011150    0.358694    0.365597   \n",
       "1  mean_residual_deviance    0.218685   0.018149    0.232207    0.226213   \n",
       "2                     mse    0.218685   0.018149    0.232207    0.226213   \n",
       "3           null_deviance  255.420870  18.944805  270.143950  279.337650   \n",
       "4                      r2    0.339071   0.024654    0.332108    0.343113   \n",
       "5       residual_deviance  168.764340  13.983558  180.424550  183.458680   \n",
       "6                    rmse    0.467306   0.019674    0.481878    0.475618   \n",
       "7                   rmsle    0.119310   0.004632    0.122437    0.121279   \n",
       "\n",
       "   cv_3_valid  cv_4_valid  cv_5_valid  \n",
       "0    0.336775    0.355430    0.361377  \n",
       "1    0.191451    0.209182    0.234372  \n",
       "2    0.191451    0.209182    0.234372  \n",
       "3  235.192600  252.053120  240.377060  \n",
       "4    0.367360    0.351331    0.301443  \n",
       "5  148.757460  163.370830  167.810170  \n",
       "6    0.437551    0.457364    0.484120  \n",
       "7    0.111271    0.119514    0.122048  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Scoring History: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>duration</th>\n",
       "      <th>iteration</th>\n",
       "      <th>lambda</th>\n",
       "      <th>predictors</th>\n",
       "      <th>deviance_train</th>\n",
       "      <th>deviance_xval</th>\n",
       "      <th>deviance_se</th>\n",
       "      <th>alpha</th>\n",
       "      <th>iterations</th>\n",
       "      <th>training_rmse</th>\n",
       "      <th>training_deviance</th>\n",
       "      <th>training_mae</th>\n",
       "      <th>training_r2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.000 sec</td>\n",
       "      <td>1</td>\n",
       "      <td>,22E0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.330642</td>\n",
       "      <td>0.330652</td>\n",
       "      <td>0.008216</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.000 sec</td>\n",
       "      <td>2</td>\n",
       "      <td>,2E0</td>\n",
       "      <td>5</td>\n",
       "      <td>0.316066</td>\n",
       "      <td>0.330609</td>\n",
       "      <td>0.008220</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.000 sec</td>\n",
       "      <td>3</td>\n",
       "      <td>,18E0</td>\n",
       "      <td>6</td>\n",
       "      <td>0.302049</td>\n",
       "      <td>0.330627</td>\n",
       "      <td>0.008216</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.000 sec</td>\n",
       "      <td>4</td>\n",
       "      <td>,17E0</td>\n",
       "      <td>6</td>\n",
       "      <td>0.289848</td>\n",
       "      <td>0.322157</td>\n",
       "      <td>0.008512</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.016 sec</td>\n",
       "      <td>5</td>\n",
       "      <td>,15E0</td>\n",
       "      <td>7</td>\n",
       "      <td>0.279287</td>\n",
       "      <td>0.307694</td>\n",
       "      <td>0.008407</td>\n",
       "      <td>0.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.528476</td>\n",
       "      <td>0.279287</td>\n",
       "      <td>0.407732</td>\n",
       "      <td>0.155318</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.016 sec</td>\n",
       "      <td>6</td>\n",
       "      <td>,14E0</td>\n",
       "      <td>7</td>\n",
       "      <td>0.270007</td>\n",
       "      <td>0.294801</td>\n",
       "      <td>0.008323</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.016 sec</td>\n",
       "      <td>7</td>\n",
       "      <td>,13E0</td>\n",
       "      <td>7</td>\n",
       "      <td>0.262086</td>\n",
       "      <td>0.283621</td>\n",
       "      <td>0.008253</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.016 sec</td>\n",
       "      <td>8</td>\n",
       "      <td>,12E0</td>\n",
       "      <td>7</td>\n",
       "      <td>0.255335</td>\n",
       "      <td>0.273875</td>\n",
       "      <td>0.008226</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.016 sec</td>\n",
       "      <td>9</td>\n",
       "      <td>,11E0</td>\n",
       "      <td>8</td>\n",
       "      <td>0.249586</td>\n",
       "      <td>0.265471</td>\n",
       "      <td>0.008206</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.016 sec</td>\n",
       "      <td>10</td>\n",
       "      <td>,96E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.244598</td>\n",
       "      <td>0.258304</td>\n",
       "      <td>0.008188</td>\n",
       "      <td>0.5</td>\n",
       "      <td>10.0</td>\n",
       "      <td>0.494568</td>\n",
       "      <td>0.244598</td>\n",
       "      <td>0.377588</td>\n",
       "      <td>0.260234</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.032 sec</td>\n",
       "      <td>11</td>\n",
       "      <td>,88E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.240460</td>\n",
       "      <td>0.252204</td>\n",
       "      <td>0.008175</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.032 sec</td>\n",
       "      <td>12</td>\n",
       "      <td>,8E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.236926</td>\n",
       "      <td>0.246950</td>\n",
       "      <td>0.008189</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.032 sec</td>\n",
       "      <td>13</td>\n",
       "      <td>,73E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.233943</td>\n",
       "      <td>0.242530</td>\n",
       "      <td>0.008194</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.032 sec</td>\n",
       "      <td>14</td>\n",
       "      <td>,66E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.231423</td>\n",
       "      <td>0.238774</td>\n",
       "      <td>0.008196</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.032 sec</td>\n",
       "      <td>15</td>\n",
       "      <td>,6E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.229293</td>\n",
       "      <td>0.235608</td>\n",
       "      <td>0.008204</td>\n",
       "      <td>0.5</td>\n",
       "      <td>15.0</td>\n",
       "      <td>0.478846</td>\n",
       "      <td>0.229293</td>\n",
       "      <td>0.364087</td>\n",
       "      <td>0.306521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.032 sec</td>\n",
       "      <td>16</td>\n",
       "      <td>,55E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.227495</td>\n",
       "      <td>0.232935</td>\n",
       "      <td>0.008210</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.032 sec</td>\n",
       "      <td>17</td>\n",
       "      <td>,5E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.226154</td>\n",
       "      <td>0.230682</td>\n",
       "      <td>0.008216</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.047 sec</td>\n",
       "      <td>18</td>\n",
       "      <td>,46E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.224642</td>\n",
       "      <td>0.228783</td>\n",
       "      <td>0.008220</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.047 sec</td>\n",
       "      <td>19</td>\n",
       "      <td>,42E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.223737</td>\n",
       "      <td>0.227337</td>\n",
       "      <td>0.008211</td>\n",
       "      <td>0.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td></td>\n",
       "      <td>2022-07-22 17:10:18</td>\n",
       "      <td>0.047 sec</td>\n",
       "      <td>20</td>\n",
       "      <td>,38E-1</td>\n",
       "      <td>8</td>\n",
       "      <td>0.222642</td>\n",
       "      <td>0.225825</td>\n",
       "      <td>0.008238</td>\n",
       "      <td>0.5</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.471849</td>\n",
       "      <td>0.222642</td>\n",
       "      <td>0.358377</td>\n",
       "      <td>0.326639</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                timestamp    duration  iteration  lambda  predictors  \\\n",
       "0     2022-07-22 17:10:18   0.000 sec          1   ,22E0           1   \n",
       "1     2022-07-22 17:10:18   0.000 sec          2    ,2E0           5   \n",
       "2     2022-07-22 17:10:18   0.000 sec          3   ,18E0           6   \n",
       "3     2022-07-22 17:10:18   0.000 sec          4   ,17E0           6   \n",
       "4     2022-07-22 17:10:18   0.016 sec          5   ,15E0           7   \n",
       "5     2022-07-22 17:10:18   0.016 sec          6   ,14E0           7   \n",
       "6     2022-07-22 17:10:18   0.016 sec          7   ,13E0           7   \n",
       "7     2022-07-22 17:10:18   0.016 sec          8   ,12E0           7   \n",
       "8     2022-07-22 17:10:18   0.016 sec          9   ,11E0           8   \n",
       "9     2022-07-22 17:10:18   0.016 sec         10  ,96E-1           8   \n",
       "10    2022-07-22 17:10:18   0.032 sec         11  ,88E-1           8   \n",
       "11    2022-07-22 17:10:18   0.032 sec         12   ,8E-1           8   \n",
       "12    2022-07-22 17:10:18   0.032 sec         13  ,73E-1           8   \n",
       "13    2022-07-22 17:10:18   0.032 sec         14  ,66E-1           8   \n",
       "14    2022-07-22 17:10:18   0.032 sec         15   ,6E-1           8   \n",
       "15    2022-07-22 17:10:18   0.032 sec         16  ,55E-1           8   \n",
       "16    2022-07-22 17:10:18   0.032 sec         17   ,5E-1           8   \n",
       "17    2022-07-22 17:10:18   0.047 sec         18  ,46E-1           8   \n",
       "18    2022-07-22 17:10:18   0.047 sec         19  ,42E-1           8   \n",
       "19    2022-07-22 17:10:18   0.047 sec         20  ,38E-1           8   \n",
       "\n",
       "    deviance_train  deviance_xval  deviance_se  alpha  iterations  \\\n",
       "0         0.330642       0.330652     0.008216    0.5         NaN   \n",
       "1         0.316066       0.330609     0.008220    0.5         NaN   \n",
       "2         0.302049       0.330627     0.008216    0.5         NaN   \n",
       "3         0.289848       0.322157     0.008512    0.5         NaN   \n",
       "4         0.279287       0.307694     0.008407    0.5         5.0   \n",
       "5         0.270007       0.294801     0.008323    0.5         NaN   \n",
       "6         0.262086       0.283621     0.008253    0.5         NaN   \n",
       "7         0.255335       0.273875     0.008226    0.5         NaN   \n",
       "8         0.249586       0.265471     0.008206    0.5         NaN   \n",
       "9         0.244598       0.258304     0.008188    0.5        10.0   \n",
       "10        0.240460       0.252204     0.008175    0.5         NaN   \n",
       "11        0.236926       0.246950     0.008189    0.5         NaN   \n",
       "12        0.233943       0.242530     0.008194    0.5         NaN   \n",
       "13        0.231423       0.238774     0.008196    0.5         NaN   \n",
       "14        0.229293       0.235608     0.008204    0.5        15.0   \n",
       "15        0.227495       0.232935     0.008210    0.5         NaN   \n",
       "16        0.226154       0.230682     0.008216    0.5         NaN   \n",
       "17        0.224642       0.228783     0.008220    0.5         NaN   \n",
       "18        0.223737       0.227337     0.008211    0.5         NaN   \n",
       "19        0.222642       0.225825     0.008238    0.5        20.0   \n",
       "\n",
       "   training_rmse training_deviance training_mae training_r2  \n",
       "0                                                            \n",
       "1                                                            \n",
       "2                                                            \n",
       "3                                                            \n",
       "4       0.528476          0.279287     0.407732    0.155318  \n",
       "5                                                            \n",
       "6                                                            \n",
       "7                                                            \n",
       "8                                                            \n",
       "9       0.494568          0.244598     0.377588    0.260234  \n",
       "10                                                           \n",
       "11                                                           \n",
       "12                                                           \n",
       "13                                                           \n",
       "14      0.478846          0.229293     0.364087    0.306521  \n",
       "15                                                           \n",
       "16                                                           \n",
       "17                                                           \n",
       "18                                                           \n",
       "19      0.471849          0.222642     0.358377    0.326639  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "See the whole table with table.as_data_frame()\n",
      "\n",
      "Variable Importances: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>variable</th>\n",
       "      <th>relative_importance</th>\n",
       "      <th>scaled_importance</th>\n",
       "      <th>percentage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>GBM_5_AutoML_1_20220722_170939</td>\n",
       "      <td>0.156444</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.459637</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>GBM_3_AutoML_1_20220722_170939</td>\n",
       "      <td>0.064177</td>\n",
       "      <td>0.410220</td>\n",
       "      <td>0.188552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>GBM_1_AutoML_1_20220722_170939</td>\n",
       "      <td>0.045077</td>\n",
       "      <td>0.288132</td>\n",
       "      <td>0.132436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>GBM_grid_1_AutoML_1_20220722_170939_model_1</td>\n",
       "      <td>0.042987</td>\n",
       "      <td>0.274777</td>\n",
       "      <td>0.126298</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>GBM_4_AutoML_1_20220722_170939</td>\n",
       "      <td>0.022251</td>\n",
       "      <td>0.142227</td>\n",
       "      <td>0.065373</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>DeepLearning_1_AutoML_1_20220722_170939</td>\n",
       "      <td>0.009429</td>\n",
       "      <td>0.060268</td>\n",
       "      <td>0.027701</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>DRF_1_AutoML_1_20220722_170939</td>\n",
       "      <td>0.000001</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>0.000003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>GBM_2_AutoML_1_20220722_170939</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>XRT_1_AutoML_1_20220722_170939</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>GLM_1_AutoML_1_20220722_170939</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      variable  relative_importance  \\\n",
       "0               GBM_5_AutoML_1_20220722_170939             0.156444   \n",
       "1               GBM_3_AutoML_1_20220722_170939             0.064177   \n",
       "2               GBM_1_AutoML_1_20220722_170939             0.045077   \n",
       "3  GBM_grid_1_AutoML_1_20220722_170939_model_1             0.042987   \n",
       "4               GBM_4_AutoML_1_20220722_170939             0.022251   \n",
       "5      DeepLearning_1_AutoML_1_20220722_170939             0.009429   \n",
       "6               DRF_1_AutoML_1_20220722_170939             0.000001   \n",
       "7               GBM_2_AutoML_1_20220722_170939             0.000000   \n",
       "8               XRT_1_AutoML_1_20220722_170939             0.000000   \n",
       "9               GLM_1_AutoML_1_20220722_170939             0.000000   \n",
       "\n",
       "   scaled_importance  percentage  \n",
       "0           1.000000    0.459637  \n",
       "1           0.410220    0.188552  \n",
       "2           0.288132    0.132436  \n",
       "3           0.274777    0.126298  \n",
       "4           0.142227    0.065373  \n",
       "5           0.060268    0.027701  \n",
       "6           0.000007    0.000003  \n",
       "7           0.000000    0.000000  \n",
       "8           0.000000    0.000000  \n",
       "9           0.000000    0.000000  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h2o.get_model(model_id).metalearner()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8f423ad",
   "metadata": {
    "papermill": {
     "duration": 0.030956,
     "end_time": "2021-03-24T11:25:14.345344",
     "exception": false,
     "start_time": "2021-03-24T11:25:14.314388",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Generating Predictions Using Leader Model\n",
    "\n",
    "We can also generate predictions on a test sample using the leader model object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "90dd2625",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stackedensemble prediction progress: |███████████████████████████████████████████| (done) 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  predict</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">  2.88073</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  3.32158</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  3.08133</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  2.73111</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  2.51044</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  3.13355</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  3.18375</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  3.74316</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  2.58496</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  3.3082 </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred = aml.predict(test_h)\n",
    "pred.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbb85276",
   "metadata": {},
   "source": [
    "This allows us to estimate the out-of-sample (test) MSE and the standard error as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "d38d7686",
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_2 = pred.as_data_frame()\n",
    "pred_aml = pred_2.to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "3b5cb41a",
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_test = test_h['lwage'].as_data_frame().to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "3b070b47",
   "metadata": {},
   "outputs": [],
   "source": [
    "import statsmodels.api as sm\n",
    "import statsmodels.formula.api as smf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "7602a9e8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Coef.       0.221502\n",
       "Std.Err.    0.012942\n",
       "Name: const, dtype: float64"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resid_basic = (Y_test-pred_aml)**2\n",
    "\n",
    "MSE_aml_basic = sm.OLS( resid_basic , np.ones( resid_basic.shape[0] ) ).fit().summary2().tables[1].iloc[0, 0:2]\n",
    "MSE_aml_basic"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6725164f",
   "metadata": {},
   "source": [
    "We observe both a lower MSE and a lower standard error compared to our previous results (see [here](https://www.kaggle.com/janniskueck/pm3-notebook-newdata))."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e03b4c5b",
   "metadata": {
    "tags": []
   },
   "source": [
    "### By using model_performance()\n",
    "If needed, the standard model_performance() method can be applied to the AutoML leader model and a test set to generate an H2O model performance object.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "caaedabd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "ModelMetricsRegressionGLM: stackedensemble\n",
      "** Reported on test data. **\n",
      "\n",
      "MSE: 0.22150159010610537\n",
      "RMSE: 0.4706395543365489\n",
      "MAE: 0.353942656628514\n",
      "RMSLE: 0.12036767274774818\n",
      "R^2: 0.2835426043371053\n",
      "Mean Residual Deviance: 0.22150159010610537\n",
      "Null degrees of freedom: 1287\n",
      "Residual degrees of freedom: 1280\n",
      "Null deviance: 398.23902107893576\n",
      "Residual deviance: 285.2940480566637\n",
      "AIC: 1731.7504037354604\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "perf = aml.leader.model_performance(test_h)\n",
    "perf"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}