{ "cells": [ { "cell_type": "markdown", "metadata": { "_execution_state": "idle", "_uuid": "051d70d956493feee0c6d64651c6a088724dca2a", "id": "DfBfWzYzWitP", "papermill": { "duration": 0.037179, "end_time": "2021-07-22T21:33:20.414432", "exception": false, "start_time": "2021-07-22T21:33:20.377253", "status": "completed" }, "tags": [] }, "source": [ "# ML for wage prediction" ] }, { "cell_type": "markdown", "metadata": { "id": "Kk13PYstWitS", "papermill": { "duration": 0.035632, "end_time": "2021-07-22T21:33:20.486115", "exception": false, "start_time": "2021-07-22T21:33:20.450483", "status": "completed" }, "tags": [] }, "source": [ "We illustrate how to predict an outcome variable Y in a high-dimensional setting, where the number of covariates $p$ is large in relation to the sample size $n$. So far we have used linear prediction rules, e.g. Lasso regression, for estimation.\n", "Now, we also consider nonlinear prediction rules including tree-based methods." ] }, { "cell_type": "markdown", "metadata": { "id": "fQLyPTHhWitT", "papermill": { "duration": 0.035698, "end_time": "2021-07-22T21:33:20.558154", "exception": false, "start_time": "2021-07-22T21:33:20.522456", "status": "completed" }, "tags": [] }, "source": [ "## Data" ] }, { "cell_type": "markdown", "metadata": { "id": "QAJp34YkWitU", "papermill": { "duration": 0.035998, "end_time": "2021-07-22T21:33:20.630093", "exception": false, "start_time": "2021-07-22T21:33:20.594095", "status": "completed" }, "tags": [] }, "source": [ "Again, we consider data from the U.S. March Supplement of the Current Population Survey (CPS) in 2015.\n", "The preproccessed sample consists of $5150$ never-married individuals." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 69 }, "execution": { "iopub.execute_input": "2021-07-22T21:33:20.735227Z", "iopub.status.busy": "2021-07-22T21:33:20.732866Z", "iopub.status.idle": "2021-07-22T21:33:20.898796Z", "shell.execute_reply": "2021-07-22T21:33:20.897685Z" }, "executionInfo": { "elapsed": 6809, "status": "ok", "timestamp": 1658250110912, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "BGctl_l4WitW", "outputId": "9b78359d-a132-4e38-92c9-7c5f83859541", "papermill": { "duration": 0.232703, "end_time": "2021-07-22T21:33:20.899004", "exception": false, "start_time": "2021-07-22T21:33:20.666301", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Warning message in system(\"timedatectl\", intern = TRUE):\n", "“running command 'timedatectl' had status 1”\n" ] }, { "data": { "text/html": [ "\n", "
  1. 5150
  2. 21
\n" ], "text/latex": [ "\\begin{enumerate*}\n", "\\item 5150\n", "\\item 21\n", "\\end{enumerate*}\n" ], "text/markdown": [ "1. 5150\n", "2. 21\n", "\n", "\n" ], "text/plain": [ "[1] 5150 21" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "install.packages(\"librarian\", quiet = T)\n", "librarian::shelf(\n", " tidyverse\n", " , randomForest\n", " , rpart\n", " , glmnnet\n", " , gbm\n", " , rpart.plot\n", " , keras\n", " , hdm\n", " , quiet = T\n", ")\n", "data = read_csv(\"https://raw.githubusercontent.com/d2cml-ai/14.388_R/main/Data/wage2015_subsample_inference.csv\", show_col_types = F)\n", "dim(data)" ] }, { "cell_type": "markdown", "metadata": { "id": "SijRfuBnWitY", "papermill": { "duration": 0.036385, "end_time": "2021-07-22T21:33:20.972103", "exception": false, "start_time": "2021-07-22T21:33:20.935718", "status": "completed" }, "tags": [] }, "source": [ "The outcomes $Y_i$'s are hourly (log) wages of never-married workers living in the U.S. The raw regressors $Z_i$'s consist of a variety of characteristics, including experience, education and industry and occupation indicators." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "execution": { "iopub.execute_input": "2021-07-22T21:33:21.078896Z", "iopub.status.busy": "2021-07-22T21:33:21.048765Z", "iopub.status.idle": "2021-07-22T21:33:21.095997Z", "shell.execute_reply": "2021-07-22T21:33:21.094629Z" }, "executionInfo": { "elapsed": 38, "status": "ok", "timestamp": 1658250110915, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "iqAhALdZWitZ", "outputId": "a0c2a5b7-7f0d-49f3-9445-8213247eb3a7", "papermill": { "duration": 0.087405, "end_time": "2021-07-22T21:33:21.096144", "exception": false, "start_time": "2021-07-22T21:33:21.008739", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "data": { "text/html": [ "\n", "
  1. 'rownames'
  2. 'sex'
  3. 'shs'
  4. 'hsg'
  5. 'scl'
  6. 'clg'
  7. 'ad'
  8. 'mw'
  9. 'so'
  10. 'we'
  11. 'ne'
  12. 'exp1'
  13. 'exp2'
  14. 'exp3'
  15. 'exp4'
  16. 'occ'
  17. 'occ2'
  18. 'ind'
  19. 'ind2'
\n" ], "text/latex": [ "\\begin{enumerate*}\n", "\\item 'rownames'\n", "\\item 'sex'\n", "\\item 'shs'\n", "\\item 'hsg'\n", "\\item 'scl'\n", "\\item 'clg'\n", "\\item 'ad'\n", "\\item 'mw'\n", "\\item 'so'\n", "\\item 'we'\n", "\\item 'ne'\n", "\\item 'exp1'\n", "\\item 'exp2'\n", "\\item 'exp3'\n", "\\item 'exp4'\n", "\\item 'occ'\n", "\\item 'occ2'\n", "\\item 'ind'\n", "\\item 'ind2'\n", "\\end{enumerate*}\n" ], "text/markdown": [ "1. 'rownames'\n", "2. 'sex'\n", "3. 'shs'\n", "4. 'hsg'\n", "5. 'scl'\n", "6. 'clg'\n", "7. 'ad'\n", "8. 'mw'\n", "9. 'so'\n", "10. 'we'\n", "11. 'ne'\n", "12. 'exp1'\n", "13. 'exp2'\n", "14. 'exp3'\n", "15. 'exp4'\n", "16. 'occ'\n", "17. 'occ2'\n", "18. 'ind'\n", "19. 'ind2'\n", "\n", "\n" ], "text/plain": [ " [1] \"rownames\" \"sex\" \"shs\" \"hsg\" \"scl\" \"clg\" \n", " [7] \"ad\" \"mw\" \"so\" \"we\" \"ne\" \"exp1\" \n", "[13] \"exp2\" \"exp3\" \"exp4\" \"occ\" \"occ2\" \"ind\" \n", "[19] \"ind2\" " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "Z <- data |> select(-c(lwage, wage)) # regressors\n", "colnames(Z)" ] }, { "cell_type": "markdown", "metadata": { "id": "Oo2wY8oIWitc", "papermill": { "duration": 0.036941, "end_time": "2021-07-22T21:33:21.171426", "exception": false, "start_time": "2021-07-22T21:33:21.134485", "status": "completed" }, "tags": [] }, "source": [ "The following figure shows the weekly wage distribution from the US survey data." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 437 }, "execution": { "iopub.execute_input": "2021-07-22T21:33:21.250222Z", "iopub.status.busy": "2021-07-22T21:33:21.249037Z", "iopub.status.idle": "2021-07-22T21:33:21.551533Z", "shell.execute_reply": "2021-07-22T21:33:21.550113Z" }, "executionInfo": { "elapsed": 39, "status": "ok", "timestamp": 1658250110918, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "5Xsm3cUyWite", "outputId": "33fd519e-3ee5-4c10-d693-2f9805aeb98a", "papermill": { "duration": 0.343165, "end_time": "2021-07-22T21:33:21.551709", "exception": false, "start_time": "2021-07-22T21:33:21.208544", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAADAFBMVEUAAAABAQECAgIDAwMEBAQFBQUGBgYHBwcICAgJCQkKCgoLCwsMDAwNDQ0ODg4PDw8QEBARERESEhITExMUFBQVFRUWFhYXFxcYGBgZGRkaGhobGxscHBwdHR0eHh4fHx8gICAhISEiIiIjIyMkJCQlJSUmJiYnJycoKCgpKSkqKiorKyssLCwtLS0uLi4vLy8wMDAxMTEyMjIzMzM0NDQ1NTU2NjY3Nzc4ODg5OTk6Ojo7Ozs8PDw9PT0+Pj4/Pz9AQEBBQUFCQkJDQ0NERERFRUVGRkZHR0dISEhJSUlKSkpLS0tMTExNTU1OTk5PT09QUFBRUVFSUlJTU1NUVFRVVVVWVlZXV1dYWFhZWVlaWlpbW1tcXFxdXV1eXl5fX19gYGBhYWFiYmJjY2NkZGRlZWVmZmZnZ2doaGhpaWlqampra2tsbGxtbW1ubm5vb29wcHBxcXFycnJzc3N0dHR1dXV2dnZ3d3d4eHh5eXl6enp7e3t8fHx9fX1+fn5/f3+AgICBgYGCgoKDg4OEhISFhYWGhoaHh4eIiIiJiYmKioqLi4uMjIyNjY2Ojo6Pj4+QkJCRkZGSkpKTk5OUlJSVlZWWlpaXl5eYmJiZmZmampqbm5ucnJydnZ2enp6fn5+goKChoaGioqKjo6OkpKSlpaWmpqanp6eoqKipqamqqqqrq6usrKytra2urq6vr6+wsLCxsbGysrKzs7O0tLS1tbW2tra3t7e4uLi5ubm6urq7u7u8vLy9vb2+vr6/v7/AwMDBwcHCwsLDw8PExMTFxcXGxsbHx8fIyMjJycnKysrLy8vMzMzNzc3Ozs7Pz8/Q0NDR0dHS0tLT09PU1NTV1dXW1tbX19fY2NjZ2dna2trb29vc3Nzd3d3e3t7f39/g4ODh4eHi4uLj4+Pk5OTl5eXm5ubn5+fo6Ojp6enq6urr6+vs7Ozt7e3u7u7v7+/w8PDx8fHy8vLz8/P09PT19fX29vb39/f4+Pj5+fn6+vr7+/v8/Pz9/f3+/v7////isF19AAAACXBIWXMAABJ0AAASdAHeZh94AAAgAElEQVR4nO3dCXwU9dnA8Sckm5BACKAgcqNSrVpB0GorKArVeiB4VMTj5VJQQaIv+iKC4k2LVWsVD9Si4omAt1YpaKv1ptqqWI2goiICBpU7Jpl3Zvaa3cxOlsmzuyHz+34+ZifLf/Y/s7u/7DUmYgBoMMn1BgBNASEBCggJUEBIgAJCAhQQEqCAkAAFhAQoICRAASEBCggJUEBIgAJCAhQQEqCAkAAFhAQoICRAASEBCggJUEBIgAJCAhQQEqCAkAAFhAQoICRAASEBCggJUEBIgAJCAhQQEqCAkAAFhAQoICRAASEBCggJUEBIgAJCAhQQEqCAkAAFTSSkx0SKvL6vdwU1j4vkp3v5kUHpbcv2bfG8Ps2Ld9uc/viGzpcrkas79xphSHeJw7j01iGkRG/YV96GtMf7m69cZJfI4jiRbtZp5cyBuxaG2h548ecNmTt9hJSan5AqbrrpFq/v68h0SK4bsDpfPnJ+HxlUz7ZE1qp3l5zM+3XZTfdXpT3eZT5fIf19p+gN1/wJX5NvL9eQkq/nbGikIR0zJOL2DE2S6ZBc/Vlcb+B6tiXFWp6OTvtHUMr5/IRUaXbU/LQrpxzdTKR0jd8N2B6uV7efa6yhGmlI6zM9SU5COsRXSCnW8jRQZNL2rpM0n5+Q/iJS8G/rW/NakD/63YDt4Xp1+7nGGmoHCOkekf7GY72Lu02tMpYNbt3iNx+YZ94tsrfx6CGtSgf90xoTudXNoYf+NHHn9vF7wVcX/ryk+d6T7Z+OtQ8f2a6g9Jd/rjYS7ybVZSLLzdOJ5sRvmqc3iZybONq8i/QtaXvs2yvMEVusb/87bo+i0gNu/sm53XP6lrQ55u0nEl8j/XTnwHYF7Q64bq1hHBt+zlNeZyvNk2JjYf9WpQNeslaZKnKQdRq+j8TWim3xD9ce1CbU/si/VMeunH8Mat2i36L4loyLPL3aEJvJba3kqzTMOV9xwiXX3ec6IU0X6RH+/pobHv8sPtBxLSTtXuzKMNP/jT32fjPGNc7ZRonsY//LfeY1EL9jJFzdzlsrtgdJN2Fm7QAhPSKy77w867oZv2Jn66Td94bxgEjH6+0rrOAFI3a/fUhkvxusqzZ6r1tSFr5Wd/mP+c3pkfvXcbVJP2+PE3nIPNnP/LeZ5ukwkXmJo40L7cWiWeYXa4UFzcP/dsSW+GZOCg+ZlhBS1YDIpez2afwGTt5K86Tl7fa/NXvGqDek9zpFLvJX6yJXzguF1rf5f4ttSiyk2Ewua9W5SsOc87WJXPKLKfa5Tki3mbvwfN1b1HktJIcU3UQzqMIfrfNPNJ/YJ8z2T/PkY+tffifyuxRXt/PWioeUcBNm2A4QknmL7tr5+AlmEkVHdyw/yPzXmwzjUfO7ohEPX9fSvAGrnO9/de8S6r1n9Ptv2oocNv+B3iJ7/mQ8bd7Kt71/T4GdSUJIN9hX+7o8aS/Hmd92l7w1iaPfMiftfdfcX7cM32wrikUu/vjtw0SmxC7jTetWX/jUbwsSQjL72Ovh1/56krkZxodPmkMefGVFna20TkrPenimuYddtiXf05xrWYMrzSJ63P7EZHOeYyNXTvfeU440B/0yti2fvnKAyKmvvFITm8llrTpXaZhzvp0jl3xgin2uE9LH+eY2H3Pnh0l3XOe1kBxSdBPXF9rXtLGlhfVTLWG2vUVmmP9SZW7uMymubuetFduDhJsw0xppSNE3G8427Kva+km0wDxpvtzYtqfI4PCZow37Z6u84AxJen5pxL6fYt4ZNhnGGvNmmW/ceuyxZi7G8SL/kxTSv8wf1NYEeTOkdY2xWuQXRuLosSKtzZ/km7uFb7YJIgPMk7UtpTT24/ls88f6RvNJzF4JIY0UucE8qRo+4fc1xjcSe/WRsJXW9+eYp/PN0+fq3NMca1mDrxRp9bV5Otc8+53wuv232D98mznepIu8RorN5LJWnas0wrmVh26NXbLbPtd91+668ENAm8Gzf3TcognXQt2Qwps4RORM88SsoHRT4mw3hFNeLNI+9rwy6epOuLWie5BwZqY10pCiuhnhq/ot82YoEhlufnuR+eAQPvM987ufSkWuSLhLWk/Sot/vK3KW9e0/n3++Inrx54scmRRSTRtpXmX+w37/FXnXuinLjcTRe4dvZOOy8M22u8jULaZDRWLPp34eDtu4OiEk82VX1/tXR4Y476LOrbS+f988rW4lcnl9IfUSGWn9a3UbkavC6y4xv/2beer46MYRkj2Ty1p1rlKXrYxe8mfu++zyOdLz/fPCt1zrOfGtSbgWXEKyN9H8kbhTtR3diKTZ1poPVivtp9cXxi4y6epOuLWie5BwZqbtICFZn9F3Dj/Cmz+fdrfPDNVYo/cXGZVwl7RvsPD3tfnhVSIWDdmtyL7UgcnvSQ21fk7vKxONXeRm62HsiaTRJSLXWePm2zdbbbP4Bt4cvYjiyFQLE0J6t8QatPuYBdbrXedd1LGV8X3pY9/dPUOqLQj/dDeMX9kRWJdl/fCviMQY4QjJmsltrTpXaYRzK+OX7LrPLiEZxlf3j9vPjmlB7CITrgWXkOwrY7P5vPkfRrX5km1R8myn2F97Wj/lUlzdCbdWLCTnmZnWSENKeo1k3+fNn1LWp0q3RENqbf9rP5Fhzrtkfk18nY3mJd0RuxzzlbC0+HmvnV1Cullk1to8WWicLCcaR0iz9Ymja6P3nr/aN9tGR+nTI5eQNCR2+S/tEx7X/bWEu6hzK5P3xTMka+477cGDxHpBF53ny1Qh2TOlWivhKo1IegSMXLLbPhv/az6IRBbNB5Ke8em/vDTfelc1xnkt1A0pfGVYzyEvNl4S6ViTPNuL1ourT0T2i11g8tWdcNtG9yDhzEzbgUMKh7C//RiffLBNeKGmWfQHsWmD+WPxNPPH8HkuIf3HfD4xX/K+M/4s7apLpW/y6KLIz7/Hwo9I5r3kT8lb3TzyoDUv6RCh2teu+q311qH5jD75Lurc6tr4vkyNPNG6M9Uj0vX2ygeJnJ5GSPa/plor/ZBc9/ka8+XTxvBif5FDrNOqyPsM10Ye8yIc10LS7sVviGdE9rQe5C6qcw3Xdpf8NTckfDKVeHUn3lqRPUg8M9N24JDEetnzU8vok36XkIw9I6+RHrz66uf/IeGnBgNdQqrdWfYaL70M4z2RR60fjEbi6D0ir5Eib7b+zPkqKuJnkZe0U1yOtat+so31NoJHSPb51us9c19mmC+qrbvjONfXSL3tEsx7rDn4D+mGlGqt9ENy3WfrY9e59pL1dt2FxvIT9yqIfOj0F4l84hYXuRaSdi9+RVXtJPJxt/BL36TZrhK563DJ/ybV1Z14a0X2IPHMTNuRQ7rECL/xtCRlSObr052/N4xK8xa8a5HYL7A/NB+lDq3zuf3JktfNuulq2sgB9ltniaPPEClba/6M6xK+8c2Xrx03mfeM00Zd8lX0EkaYz88qzSGdnfePzdeNPN5+3nKkyOPW24HyistWWvtivYx+NLwv1jtrTxvGRyXhS0pay7xPtbDe5Jotkvdx2iGlWMstpOStjFyy2z5vaG1ev8+aCx/tbV7uv4wtbc1XSvZWfGs+uP48OizxWkjaPccNYaZ1uv2OaZ3ZvsqXQ0JytOMGS7y6E2+tyB4knplpjTSk2LF2Q35MGVJBaPpLt5k/xfasThnSF+YP4IMefbCvSNeNX5tX6OD3n+pkPkq1ev3bpJCsT1rNG9n+bFYKNhhG4ujF5rn73XvPgS3CN/5y86XuIc++cKLIPrEPzV82h/R95L4DzUfIZobz8eOk5975x5UhKfrWqA6J9J/3Qp2tNAPKL7zwyRtahfdluflSvcW4i9r+OjxZ0lrrzbvOHrMWXFwUfrBNM6QUa7mFlLyVkUt222fjVutq63LoHtZ7A+cb4Vcleb864/QB1ge5d8eGJVwLSbvnuCFetl8T/cFwuYaPtf7lEccNlnh1J95akT1IPDONO16DNNKQ4tamDKnNJfaA5tYxQilCMp4pCV9Kh/fsn3Kmjp91tF7BJoW0zLoHfGcuzBT7M6Wk0dYPQFPJHyLvET0WfjNIOjneZR1ln9PCum/VxDbg/c6R3Wh2j2EfSmp9IJq8lQ+YT3auje+LcY693PNVc5Nq664VO0bhxC1G2iGlWMstpOT5opfsts/GFfmRi82baD/mXFcQvd3yL4+PSrwWEnfPcUPUWtvY7EvD5Rq2nkSWJTxVTLy6E2+tyB4knplhO3BILYw7ezdvc7x9kGSqkIwV5/YsLt5ninWQV9Uf9i7udNbXxqI9Czo/khSS0UGsl0jh/5FnqpE82qi5fs+i9if/57noWstG9ygq2XdqpeMSam7Ys7D9yR98KPa7xtHLX331AbuESvYaa2/lV0NbN+9xbZ2tvFtkL2OOtS/v2WdXz+hZ2Gnst1+Zl7TJZa0frjmwLLTrCU8l7Gh9Ibmv5RpS0nyxS3bZZ8P4z4RerfLL9p/4XuT75VN/3S5U2O6QS/7rHJVwLSTunvOG+F/zrMMjy4mz/dQu8mo3xdWdeGtF9iDxzAxrhCGlJTmE7LjP/AGX/VlhrDafK76a643wREjpWDbjvFOto1OOFxmavVkRc7ZYn0k0ZoSUjgrzBfLQl1+9wHwe8WL2ZkXY3D+eYF7xLkeVNyaElJYroi/ZLsvipAg7zLrix+Z6K+pBSOlZfHLnUFG3YS9nc06EnVxUsOcNNbneinrsqCEBjQohAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQEEuQ1q1yN3ftuVwowA/chnSmFArV3mP53CjAD9yGdLIIe+7avNYDjcK8IOQAAWEBCggJEABIQEKCAlQQEiAAkICFBASoICQAAWEBCggJEABIQEKCAlQQEiAAkICFBASoICQAAWEBCggJEABIQEKCAlQQEiAAkICFBASoICQAAWEBCggJEABIQEKCAlQQEiAAkICFBASoICQAAWEBCggJEABIQEKCAlQQEiAgoaEVLt80cKFi1f6Xp+Q0GT4D6lyUnuxdb1qs79LICQ0Gb5DWtVDeo6cPnPmtOEdpVelr4sgJDQZvkMaE5oXWaqelVfu6yIICU2G75A6jI4vD+vi6yIICU2G75BC18aXryj0dRGEhCbDd0jdTokvD+nu6yIICU2G75DK867fGl7aeLlM9nURhIQmw3dI6/tI6cCRE8aPGFAi/Tf4ughCQpPh/3OkbTf2zrc+RgodPLva3yUQEpqMBh0itOWTpUsrtvlenZDQZHCIEKCAQ4QABRwiBCjgECFAAYcIAQo4RAhQwCFCgAIOEQIUcIgQoIBDhAAFmTlEaM1pv4s5ft+aFKsTEpqMhv46rm1vLVlR99wfysfGDJFUR+MREpoM3yFdvcT6ekcb88ld33e9Bv6TkND0+Q7JfqfuGSk6YdwhUvapx0BCQgA0LKSeZcvMrwvyRnkMJCQEQINCWiOX2stDO3kMJCQEQINCWilz7eVpIY+BhIQAaFBI1WUz7OXRbT0GEhICwH9Iw9+uWDtlj03m4kctBnsMJCQEgP+QwuYbxoMtmr3lMZCQEAC+Q5pz0/TyEUMHLDaMWZ2e9hpISAgAhT80tiHVIUBhhIQAUPmLfZWfefwjISEA/If072O69ZsVPu57stelEBICwHdIrxZJSUgOs39/ECEh6HyHdGzo8dqtN4YO3GgQEuA7pC5nWF8XFx5TTUiA/98idLl9cr9MJCTAd0idjw+fTpGZhITA8x3SxLxbqqzT2hFywfmEhIDzHdK6rjLIXqidKEJICDj/nyOtPe+CyNKC3QkJAadyZIMnQkIAEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABQ0JqXb5ooULF6+sZxQhIQD8h1Q5qb3Yul612WscISEAfIe0qof0HDl95sxpwztKr0qPgYSEAPAd0pjQvMhS9ay8co+BhIQA8B1Sh9Hx5WFdPAYSEgLAd0iha+PLVxR6DCQkBIDvkLqdEl8e0t1jICEhAHyHVJ53/dbw0sbLZbLHQEJCAPgOaX0fKR04csL4EQNKpP8Gj4GEhADw/znStht751sfI4UOnl3tNY6QEAANOkRoyydLl1akyiSKkBAAHCIEKOAQIUABhwgBCjhECFDAIUKAAg4RAhRwiBCggEOEAAUcIgQo4BAhQEFmDhGqfmJezNWEhKYvM4cIfb5rm5hS2ZpifUJCk8EhQoACDhECFHCIEKCAQ4QABRwiBCjgECFAAYcIAQo4RAhQwCFCgAJ+ixCgQOUv9lV+5vGPhIQA8B/Sv4/p1m9W+EndZK9LISQEgO+QXi2SkpAcZh8cREgIOt8hHRt6vHbrjaEDNxqEBPgOqcsZ1tfFhcdUExLg/xChy+2T+2UiIQG+Q+p8fPh0iswkJASe75Am5t1SZZ3WjpALzickBJzvkNZ1lUH2Qu1EEUJCwPn/HGnteRdElhbsTkgIOJUjGzwREgKAkAAFhAQoICRAASEBCggJUEBIgAJCAhQQEqCAkAAFhAQoaIwh5Yu7dp6/9gvIocYYUt45d7mZIlsyvrGAP40ypBtcz36QkNBoERKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKgwBnSwXd8n4EZCAkB4AypQIqHv1ijPQMhIQCcIa27c2C+dJlaoTsDISEAkl4jrbn98GbS7+4fFWcgJARA3TcbVt3US0rO+VhtBkJCANQJafNjJxVL11Doitp6161dvmjhwsUr6xlFSAiApJBePauVFJ/+krHyJJlez5qVk9qLretVm73GERICwBnSymt6iux/63pruXZQe+8VV/WQniOnz5w5bXhH6VXpMZCQEADOkJpJ2TnvRL+5Nc97xTGheZGl6ll55R4DCQkB4Ayp/72O52gVC71X7DA6vjysi8dAQkIAJL5G+mCt9eVf6awYuja+fEWhx0BCQgA4Q6oaLS+ZJ7fIyOr6V+x2Snx5SHePgYSEAHCGdIMcu8I8+e8w+VP9K5bnXb81vLTxcpnsMZCQEADOkH5xXGThmD3qX3F9HykdOHLC+BEDSqT/Bo+BhIQAcIZUfENkYWYojTW33dg73/oYKXTwbM9ngoSEAHCGtMv5kYXzdklv5S2fLF1akSqTKEJCADhDGl3yrHVSNbvgzLTW5RAhIMIZ0qpdpetvjuvXVnb9Io01OUQIiEn4HGn1OTuZYbQ7+6s0VuQQISAu6aDV2q8/3ZjeihwiBMT5/uUnHCIExDlDqp13XO99wupfkUOEgDhnSNeLlJSF1b8ihwgBcc6QOh+1PP0VOUQIiHOGFHpjO1bkECEgLuER6fXtWZNDhIAYZ0gXn7edK6c8RGh5kThsTbE6IaHJcIa04ajT/rqswpb2+tveWrKi7rm1/1gU8ycekdD0OUNyPIjUv+LVS6yvd7QxB/d912sgT+0QAM5kho8YE5XGitY7dc9I0QnjDpGyTz0GEhICwPeRDXZIPcuWmV8X5I3yGEhICICkkH78YH26K5ohrZFL7eWhnTwGEhICICGkl/uKPG8Yg/+WzopmSCtlrr08zev/qCUkBIAzpDcLS48yQ1rTofCdlOPjK5ohVZfNsJdHt/UYSEgIAGdIx3b98hvrEenbrkPSWHH42xVrp+yxyVz8qMVgj4GEhABwhrTTDMMOybiuTRorhs03jAdbNHvLYyAhIQAS/vTlA5GQ5qTxW4Tm3DS9fMTQAYsNY1anp70GEhICIOFYu6mRkEZ1256L2OD9Z2cJCQHgDGlsm6VWSJWXynYedLfO65AiQkIAOEP6pktBH+ndu0i6rt6+C5ns9bEuISEAEhL49lzrtwjtfO6323khhISgS/4tQqsrtvPRyEJICDrfx9r1dehASAg4ZwIDY/rXv2KzZkUx+YSEgHP9/5FKO9a/4uTS+Ft1PLVD0DkT+Mm26YOLDv2h/hWr9j+gKrpMSAg61wQuOSeNNZcVXxRdJCQEnWsCr6fx1M4wfvguuvTyDI9hhIQAcA3pxRLFGQgJAeAMaX3Ympd6p/G7v9NGSAgA998iNFdxBkJCACT8j31hQ89N5381TxshIQB8H9mQNkJCABASoMAZUq9fHuSkNAMhIQCcIe1SLCJ55n/F+RalGQgJAeAMqbLf+H9tMX74+4lHpnGIUNoICQHgDGnUyMjCb89SnIGQEADOkNrdE1n4Y3vFGQgJAeAMqSj6h8r/r0hxBkJCADhD2r9j+I/IvrpzL8UZCAkB4AzpyXzpMWjwoN0kb77iDISEAEj8axRHNReRwiMWac5ASAiApCMbar765EvPv1G+/QgJAeD7D42ljZAQAL7/0FjaCAkB4PsPjaWNkBAAvv/QWNoICQHg+w+NpY2QEAC+/9BY2ggJAaDwh8bqQUgIAJU/NOaJkBAAKn9ozBMhIQBU/tCYJ0JCAKj8oTFPhIQASDj6+4NMzEBICABnSM1/n4kZCAkB4Axp0NE1GZiBkBAAzpBWD//tQ+9U2BRnICQEgPsv0df8/auEhABwJjPszNFjIhRnICQEAL/7G1AQC+mWV+yTd7/SnoGQEACxkKQ8fDJeewZCQgAQEqCAkAAFhAQoICRAASEBCggJUBAP6aDpFjnQPlGcgZAQAPGQEijOQEgIgFgycxMozkBICACOtQMUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoaElLt8kULFy5eWc8oQkIA+A+pclL78P8E2PWqzV7jCAkB4DukVT2k58jpM2dOG95RelV6DCQkBIDvkMaE5kWWqmfllXsMJCQEgO+QOoyOLw/r4jGQkBAAvkMKXRtfvqLQYyAhIQB8h9TtlPjykO4eAwkJAeA7pPK867eGlzZeLpM9BhISAsB3SOv7SOnAkRPGjxhQIv03eAwkJASA/8+Rtt3YO9/6GCl08Oxqr3GEhABo0CFCWz5ZurQiVSZRhIQA4BAhQAGHCAEKOEQIUMAhQoACDhECFHCIEKCAQ4QABRwiBCjgECFAAYcIAQoyc4jQujN/F3MEIaHpy8whQusnjI0ZQkho+jhECFDAIUKAAg4RAhRwiBCggEOEAAUcIgQo4BAhQAGHCAEKOEQIUMBvEQIUqPzFvnUVHv9ISAgAlZAme10KISEACAlQQEiAAt8h9XXoQEgION8hNWtWFJNPSAg43yFNLo2/VcdTOwSd75Cq9j+gKrpMSAg6/282LCu+KLpISAi6Brxr98N30aWXZ3gMIyQEgMrb354ICQFASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVBASIACQgIUEBKggJAABYQEKCAkQAEhAQoICVDQkJBqly9auHDxynpGERICwH9IlZPai63rVZu9xhESAsB3SKt6SM+R02fOnDa8o/Sq9BhISAgA3yGNCc2LLFXPyiv3GEhICADfIXUYHV8e1sVjICEhAHyHFLo2vnxFocdAQkIA+A6p2ynx5SHdPQYSEgLAd0jleddvDS9tvFwmewwkJASA75DW95HSgSMnjB8xoET6b/AYSEgIAP+fI227sXe+9TFS6ODZ1V7jCAkB0KBDhLZ8snRpRapMoggJAcAhQoACDhECFHCIEKCAQ4QABZk5RKjmmXkxVxMSmr7MHCK0ol2bmFJCQtPHIUKAAg4RAhRwiBCggEOEAAUcIgQoaOiv49r21pIV3iMICQHgO6Srl1hf72hjPrnr+67XQEJCAPgOyX6n7hkpOmHcIVL2qcdAQkIANCyknmXLzK8L8kZ5DCQkBECDQlojl9rLQzt5DCQkBECDQlopc+3laSGPgYSEAGhQSNVlM+zl0W09BhISAsB/SMPfrlg7ZY9N5uJHLQZ7DCQkBID/kMLmG8aDLZq95TGQkBAAvkOac9P08hFDByw2jFmdnvYaSEgIAIU/NLahxvOfCQkBwF/sAxQQEqCAkAAFhAQoICRAASEBCggJUEBIgAJCAhQQEqCAkAAFhAQoICRAwQ4U0l0yZqyrRzK+D0A9dqCQrpRBR7rZbVDG9wGoxw4V0juu548jJOQcIQEKCAlQQEiAAkICFBASoICQAAWEBCggJEABIQEKCAlQQEiAAkICFBASoICQAAWEBCggJEABIQEKCAlQQEiAAkICFBASoICQAAWEBCggJEABIQEKCAlQQEiAAkICFBASoICQAAWEBCggJKW1hnkAAAv4SURBVEABIQEKCAlQQEiAAkICFBASoICQAAWEBCggJEABIQEKCAlQQEiAAkICFBASoICQAAWEBCggJEABIQEKCAlQ0ARCOqpVX1cHLsn4vgERTSCkPl0vdNXupozvGxDRFEI62P1i9iIkZA0hAQoICVBASIACQgIUEBKggJAABYQEKCAkQEETDqnTwMmurkq1PYBvTTik5t0OdtNHPsn4PiNwmnJIF7uevZiQoK8hIdUuX7Rw4eKV9YwiJASA/5AqJ7UXW9erNnuNIyQEgO+QVvWQniOnz5w5bXhH6VXpMZCQEAC+QxoTmhdZqp6VV+4xsLGF9JSkcJjfqwLwH1KH0fHlYV08Bja2kB6Va+5yc3iqwH411tVE94fhzffe6erWRctdrahxv3JWug//+D3385dXuV/MqhTD13ncYtvh8xQXv0Hn4lP4IcWsX7gP3zjH/Sa5s0Jxm3yHFLo2vnxFYdI/rmjXJqZUUtzG5oNaK1dS7Hp2sZS6np9fkOJiilzPbiEtXc8PpQopldI2blpu78W0dL2Ysu29mBLXi2mdani+6/A2zYvdzy9o5Xp2q1QXX+h+MSH3nW2d39r1/JapLibVtO5bmfImGev3zu/Cd0jdTokvD+me9I81Ly2KefGBVBexapG72c+7nv3i7e7DF851P3/Ok+7n3+Z+9jP3uJ//4Hz38+94wfXs5+90Hz7vIffz737W9exUO/vEfe7n3/eE+/m3v+h69rN3uw9/aJ77+Xe63yQv3OE+fP6D7uff84z7+SlukifnuJ8/d6H7+Sl29vnZ7sMXrfJ753fhO6TyvOu3hpc2Xi6TtTYH2DH5Dml9HykdOHLC+BEDSqR/Zp8SA42e/8+Rtt3YO996ohk6eHa14gYBO6IGHSK05ZOlSys4BBTIwrF2QAAQEqCAkAAFhAQoICRAASEBCggJUEBIgAJCAhQQEqCAkAAFhAQoICRAASEBCggJUEBIgAJCAhTkMqSD0/1VU0Ba/pa7O3MuQzpt8Ds58JwszMW0Jx+Zi1mXyMO5mPaMQ3Mx62vyeu7uzLkMaeTIXMz6VW5+9/e5w3Ix63fy71xMO2lwLmbdQkjZREiZR0hZRUgZR0jZQkhZQkgZR0hZRUiZR0hZRUgZR0jZQkhZQkgZR0hZRUiZR0hZRUgZR0jZQkhZQkgZF9iQxmr+6cG0rZHPczFt+Zm5mPXHvI9yMe2Uk3Ixa1X+0lxMG5bLkCrd/5xxpi3PyazfK/354+2Um539cU1Ops3Nzobxv1EACggJUEBIgAJCAhQQEqCAkAAFhAQoICRAASEBCggJUEBIgAJCAhQQEqCAkAAFhAQoICRAQe5CWl/eLbTrmFVZmavqkmZ968ya8Q2onNS1sPuQ17M87fKzdyvcecibWZ7VdqGMye60cyJ/heLqrM7qKmchbesjJ107OtQjG/+X7LI+pZGQHLNmfAO+6y7HXnZ6QfP/ZHXa/+5UeMb000Oh17K7s5a38+2QsjjtTTJ8smVJ9nc2Wc5CulH+YH59VCZlfqofig+oKOqbPGvGN2C83GJ+XSDHZHXa3+T93fy6UE7J7s6afurdyw4pi9NOl7eji1ne2TpyFlLv0q3WyR7tazM+1XeTqoxISI5ZM74BFwysMr/WFnfL6rTTplhfq0O9sruzpt/nPW+HlMVpy6Uiupjlna0jVyFtyR9on46U7PzGinBIjlmztQFbQ4fkYNqvZGi2Z/20+Nz1VkjZnHaErK3+cq21lINbNlGuQvpEwr/Ubrosysp84ZAcs2ZrA242n+Ble9pNL+1X+na2Zx246/d2SNmcdqhMbSPyswdzcssmylVIS2W8fXq9LMzKfOGQHLNmaQNeLuz3U7anLRM5Y3m2d3aOzDfskLI57QDZbcb9U1rJHTm4ZZPkLqQJ9ulMeTwr80VDis2anQ14qKjPd1mf9pKxv27Wb3l2Z/227XFGNKTsTbt4/kbz64dFbbdl/ZZNlquQKmSEfTotS3+JOhySY9ZsbEDt5fLbH7M/remlFvvVZHXWU1t+EQkp+ztrnCBv5WDWRLkKaVvBAPt0uHyRlfnCITlmzcIG1I6W86uzP63tNFmWzVmfk8u+/PLLD2X4lz/kYGfHyZIczJooZ29/H1Syyfxa07FLdqaLvP3tmDXzG1Au10WWsjftV/uFf8f4ifJ2Nnd2kkRNzuK0G257yD7tJ8uze8u6yFlIs+UK8+vtcmV2pouE5Jg14xuwQMqji1mctnPhG+bXj1u23JLNWZc9bXlEjnz6oyxOW9OppfU3Ap6Q/bN7y7rJWUjV/WXIlafm/WJT5qd6efLkyfkdzC/rnLNmfAN2l/Pt41cmV2Zz2sfzQ6dOHdlCbjWyurM2+zVSNqd9Mq/FmMtOyGu1NAc7myR3B61uuKhbqNP477Iw04zo046KhFkzvQGxZzufZXXaN4a2y2896KmkqbJybYdDyua0rx3duqDj/1RkeVY3/G8UgAJCAhQQEqCAkAAFhAQoICRAASEBCggJUEBIgAJCAhQQEqCAkAAFhAQoICRAASEBCggJUEBIgAJCAhQQEqCAkAAFhAQoICRAASEBCggJUEBIgAJCAhQQEqCAkAAFhAQoICRAASEBCggJUEBIgAJCaiTGSEVa44bJlxneEvhBSI0EIe3YCKmRIKQdGyE1EoS0YyOkRmKMLP99j8IuV9Way5+P7BjaafCb5tKxst78+pMMtAr6dlDzJ62Q+jVbaa2xruDg8Kodf2F93VueNb8+JHONN4fuFOp2xmfWmc8cWLzLxM2d9zcXV5/XNbTzkLeyv2fBQEiNxBgZtf+MmV3kIcNY2b7lxfde26nolYSQzpTTjr7ufSuke+Uaa4075Y7wqiPyKg3jW2n5f+by2LzV7zTveNXsS0rbrzOMv+d3uHLWgOPLDjKMNd3KJs+9rnPRy7nbxSaNkBqJMdKvyjCWyvFmGLLQPGNZ/sEJIY2WI2sM+6ndprKe1hoDm38fXvUBecowHikYZT1A/ay3cVufl8yFW+QWw/iNvG0Y1YeLGdK5BeaisbL0gJzsXdNHSI3EGHnc/Fqbf4BRW7aL9fzO6CfrnCGNkQetc63XSOfIq+ZDTP7wyKqr8y4yjHH731+w0Vglk+2zqrYslkmG0Xwv65u/miHV7tznG8tRsiH7+xYEhNRIjJEPrJOyfcwYjoic81piSO9Y51ohvSNnGcbt8mJ03f3MR5yflX8ui4wHZbFh3H9oazGVG+vlOOuffzRDWi1RH2Z/34KAkBqJyLt2ZkgVMtg+Z4IZRkJI9gD7Xbv9W202Du9SE113UsHGr82ng12mGWe32GZMkQPmvPz63WZIn8op9r/nH2ReaO/nw9ZnfdcCgZAaiXhI30QekUbJG5GQNiWHdKs89k2zqbF1/yqLHshbaww/1Oh5jLGluMsG+7xy4wvrFZe1uvWI1DvbexQshNRIxEMy2u5qv0Y6KG+9MVTWmEsfJIe0vvjkPzk+d9pcNH3MvoZxW9EKudn4TE6wzptihrStWS9rcYn1ZsPOze2HojXZ3avgIKRGwhHSWfb7Du/mmfWcK383F/8vOSTj9JLe/RwrH3HUbuMN4305Xz4yNudZnxq920nGGcYv8z4yjOqj7Hft5FLz7DUdjsv2jgUEITUSjpC+7tDy0vuubF/6b8N4XfoueWNK/9LkkF4Sudux8oyWMs8watu27Gp+c5yMe/iyNs8VdH5o42PS4/o7+48oMkP6tquMuve6rqEXXeZGwxFSI+EIyVg5ateC9qcus76/d+/iXcZ+37FfUkhG15IfHSu/I7LaPBlsvZtnrDmtXdkRrxhXtuzwjXHPnoXdplYV/to8+5tzuxS0Pv7NrO5UgBDSDmll6Jz0B/8Qfs8BmURIO6TfhT5OZ9hfDrM+e7pZZmZ4c0BIO6CKWUfK9LRGvlHU4cq7zivoymdHGUdIO54Fee2uq01v6KtHtw91Gv11ZrcHBiEBKggJUEBIgAJCAhQQEqCAkAAFhAQoICRAASEBCggJUEBIgAJCAhQQEqCAkAAFhAQoICRAASEBCggJUEBIgAJCAhQQEqCAkAAFhAQoICRAASEBCggJUEBIgIL/B0PcTl9QGzHZAAAAAElFTkSuQmCC", "text/plain": [ "Plot with title “Empirical wage distribution from the US survey data”" ] }, "metadata": { "image/png": { "height": 420, "width": 420 } }, "output_type": "display_data" } ], "source": [ "hist(data$wage, xlab= \"hourly wage\", main=\"Empirical wage distribution from the US survey data\", breaks= 35)" ] }, { "cell_type": "markdown", "metadata": { "id": "XCwz2ZNoWitg", "papermill": { "duration": 0.038762, "end_time": "2021-07-22T21:33:21.629360", "exception": false, "start_time": "2021-07-22T21:33:21.590598", "status": "completed" }, "tags": [] }, "source": [ "Wages show a high degree of skewness. Hence, wages are transformed in almost all studies by\n", "the logarithm." ] }, { "cell_type": "markdown", "metadata": { "id": "cp2Tf15pWiti", "papermill": { "duration": 0.038159, "end_time": "2021-07-22T21:33:21.706383", "exception": false, "start_time": "2021-07-22T21:33:21.668224", "status": "completed" }, "tags": [] }, "source": [ "## Analysis" ] }, { "cell_type": "markdown", "metadata": { "id": "jjN31ZYbWitj", "papermill": { "duration": 0.038644, "end_time": "2021-07-22T21:33:21.783461", "exception": false, "start_time": "2021-07-22T21:33:21.744817", "status": "completed" }, "tags": [] }, "source": [ "Due to the skewness of the data, we are considering log wages which leads to the following regression model\n", "\n", "$$log(wage) = g(Z) + \\epsilon.$$" ] }, { "cell_type": "markdown", "metadata": { "id": "jNsHcgDTWitl", "papermill": { "duration": 0.038767, "end_time": "2021-07-22T21:33:21.861283", "exception": false, "start_time": "2021-07-22T21:33:21.822516", "status": "completed" }, "tags": [] }, "source": [ "We will estimate the two sets of prediction rules: Linear and Nonlinear Models.\n", "In linear models, we estimate the prediction rule of the form\n", "\n", "$$\\hat g(Z) = \\hat \\beta'X.$$\n", "Again, we generate $X$ in two ways:\n", " \n", "1. Basic Model: $X$ consists of a set of raw regressors (e.g. gender, experience, education indicators, regional indicators).\n", "\n", "\n", "2. Flexible Model: $X$ consists of all raw regressors from the basic model plus occupation and industry indicators, transformations (e.g., ${exp}^2$ and ${exp}^3$) and additional two-way interactions.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "DpTuFFQ-Witn", "papermill": { "duration": 0.038121, "end_time": "2021-07-22T21:33:21.938272", "exception": false, "start_time": "2021-07-22T21:33:21.900151", "status": "completed" }, "tags": [] }, "source": [ "To evaluate the out-of-sample performance, we split the data first." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2021-07-22T21:33:22.019774Z", "iopub.status.busy": "2021-07-22T21:33:22.018129Z", "iopub.status.idle": "2021-07-22T21:33:22.037820Z", "shell.execute_reply": "2021-07-22T21:33:22.036538Z" }, "executionInfo": { "elapsed": 38, "status": "ok", "timestamp": 1658250110920, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "alXZbuZSWitp", "papermill": { "duration": 0.061626, "end_time": "2021-07-22T21:33:22.037969", "exception": false, "start_time": "2021-07-22T21:33:21.976343", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [], "source": [ "set.seed(1234)\n", "training <- sample(nrow(data), nrow(data)*(3/4), replace=FALSE)\n", "\n", "data_train <- data[training,]\n", "data_test <- data[-training,]" ] }, { "cell_type": "markdown", "metadata": { "id": "s0AlDE-nWitp", "papermill": { "duration": 0.039021, "end_time": "2021-07-22T21:33:22.115014", "exception": false, "start_time": "2021-07-22T21:33:22.075993", "status": "completed" }, "tags": [] }, "source": [ "We construct the two different model matrices $X_{basic}$ and $X_{flex}$ for both the training and the test sample:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2021-07-22T21:33:22.198405Z", "iopub.status.busy": "2021-07-22T21:33:22.197691Z", "iopub.status.idle": "2021-07-22T21:33:22.248902Z", "shell.execute_reply": "2021-07-22T21:33:22.246885Z" }, "executionInfo": { "elapsed": 39, "status": "ok", "timestamp": 1658250110921, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "TVNFi2fuWits", "papermill": { "duration": 0.09478, "end_time": "2021-07-22T21:33:22.249085", "exception": false, "start_time": "2021-07-22T21:33:22.154305", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [], "source": [ "X_basic <- \"sex + exp1 + exp2+ shs + hsg+ scl + clg + mw + so + we + occ2+ ind2\"\n", "X_flex <- \"sex + exp1 + exp2 + shs+hsg+scl+clg+occ2+ind2+mw+so+we + (exp1+exp2+exp3+exp4)*(shs+hsg+scl+clg+occ2+ind2+mw+so+we)\"\n", "formula_basic <- as.formula(paste(\"lwage\", \"~\", X_basic))\n", "formula_flex <- as.formula(paste(\"lwage\", \"~\", X_flex))\n", "\n", "model_X_basic_train <- model.matrix(formula_basic,data_train)\n", "model_X_basic_test <- model.matrix(formula_basic,data_test)\n", "p_basic <- dim(model_X_basic_train)[2]\n", "model_X_flex_train <- model.matrix(formula_flex,data_train)\n", "model_X_flex_test <- model.matrix(formula_flex,data_test)\n", "p_flex <- dim(model_X_flex_train)[2]" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2021-07-22T21:33:22.331274Z", "iopub.status.busy": "2021-07-22T21:33:22.329613Z", "iopub.status.idle": "2021-07-22T21:33:22.347093Z", "shell.execute_reply": "2021-07-22T21:33:22.345856Z" }, "executionInfo": { "elapsed": 39, "status": "ok", "timestamp": 1658250110922, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "7VvnCqX5Witt", "papermill": { "duration": 0.059764, "end_time": "2021-07-22T21:33:22.347237", "exception": false, "start_time": "2021-07-22T21:33:22.287473", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [], "source": [ "Y_train <- data_train$lwage\n", "Y_test <- data_test$lwage" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 52 }, "execution": { "iopub.execute_input": "2021-07-22T21:33:22.429312Z", "iopub.status.busy": "2021-07-22T21:33:22.428202Z", "iopub.status.idle": "2021-07-22T21:33:22.445258Z", "shell.execute_reply": "2021-07-22T21:33:22.443989Z" }, "executionInfo": { "elapsed": 39, "status": "ok", "timestamp": 1658250110923, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "HpKaJlTgWitt", "outputId": "df8c7f4f-e0bb-4c62-de65-b3ed914c9994", "papermill": { "duration": 0.059354, "end_time": "2021-07-22T21:33:22.445413", "exception": false, "start_time": "2021-07-22T21:33:22.386059", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "data": { "text/html": [ "13" ], "text/latex": [ "13" ], "text/markdown": [ "13" ], "text/plain": [ "[1] 13" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "51" ], "text/latex": [ "51" ], "text/markdown": [ "51" ], "text/plain": [ "[1] 51" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "p_basic\n", "p_flex" ] }, { "cell_type": "markdown", "metadata": { "id": "qUvQOieNWitu", "papermill": { "duration": 0.039589, "end_time": "2021-07-22T21:33:22.525196", "exception": false, "start_time": "2021-07-22T21:33:22.485607", "status": "completed" }, "tags": [] }, "source": [ "As known from our first lab, the basic model consists of $10$ regressors and the flexible model of $246$ regressors. Let us fit our models to the training sample using the two different model specifications. We are starting by running a simple ols regression. " ] }, { "cell_type": "markdown", "metadata": { "id": "QpogkfgpWitu", "papermill": { "duration": 0.039781, "end_time": "2021-07-22T21:33:22.604883", "exception": false, "start_time": "2021-07-22T21:33:22.565102", "status": "completed" }, "tags": [] }, "source": [ "### OLS" ] }, { "cell_type": "markdown", "metadata": { "id": "9VZOD13IWitu", "papermill": { "duration": 0.039692, "end_time": "2021-07-22T21:33:22.684509", "exception": false, "start_time": "2021-07-22T21:33:22.644817", "status": "completed" }, "tags": [] }, "source": [ "We fit the basic model to our training data by running an ols regression and compute the mean squared error on the test sample." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2021-07-22T21:33:22.770034Z", "iopub.status.busy": "2021-07-22T21:33:22.769510Z", "iopub.status.idle": "2021-07-22T21:33:22.791709Z", "shell.execute_reply": "2021-07-22T21:33:22.790718Z" }, "executionInfo": { "elapsed": 37, "status": "ok", "timestamp": 1658250110924, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "YRjB3xgOWitv", "papermill": { "duration": 0.067482, "end_time": "2021-07-22T21:33:22.791854", "exception": false, "start_time": "2021-07-22T21:33:22.724372", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [], "source": [ "# ols (basic model)\n", "fit_lm_basic <- lm(formula_basic, data_train) " ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "execution": { "iopub.execute_input": "2021-07-22T21:33:22.876209Z", "iopub.status.busy": "2021-07-22T21:33:22.875015Z", "iopub.status.idle": "2021-07-22T21:33:22.900006Z", "shell.execute_reply": "2021-07-22T21:33:22.897996Z" }, "executionInfo": { "elapsed": 1203, "status": "ok", "timestamp": 1658250112090, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "7VpQneW6Witv", "outputId": "c365c34c-f7bc-4161-f45a-949c9e659e5b", "papermill": { "duration": 0.068516, "end_time": "2021-07-22T21:33:22.900213", "exception": false, "start_time": "2021-07-22T21:33:22.831697", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The mean squared error (MSE) using the basic model is equal to 0.2496282" ] } ], "source": [ "# compute out-of-sample performance\n", "yhat_lm_basic <- predict(fit_lm_basic, newdata = data_test)\n", "cat(\"The mean squared error (MSE) using the basic model is equal to\" , mean((Y_test - yhat_lm_basic)^2)) # MSE OLS (basic model) " ] }, { "cell_type": "markdown", "metadata": { "id": "aUdqcpXAWitw", "papermill": { "duration": 0.056431, "end_time": "2021-07-22T21:33:23.026875", "exception": false, "start_time": "2021-07-22T21:33:22.970444", "status": "completed" }, "tags": [] }, "source": [ "To determine the out-of-sample $MSE$ and the standard error in one step, we can use the function *lm*:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "execution": { "iopub.execute_input": "2021-07-22T21:33:23.116493Z", "iopub.status.busy": "2021-07-22T21:33:23.115227Z", "iopub.status.idle": "2021-07-22T21:33:23.142144Z", "shell.execute_reply": "2021-07-22T21:33:23.140424Z" }, "executionInfo": { "elapsed": 23, "status": "ok", "timestamp": 1658250112093, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "-3VIzYKCWitw", "outputId": "faefb36a-56df-4a67-c824-3ffd2476013b", "papermill": { "duration": 0.075002, "end_time": "2021-07-22T21:33:23.142292", "exception": false, "start_time": "2021-07-22T21:33:23.067290", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "data": { "text/html": [ "\n", "
  1. 0.249628205480316
  2. 0.0155845190765748
\n" ], "text/latex": [ "\\begin{enumerate*}\n", "\\item 0.249628205480316\n", "\\item 0.0155845190765748\n", "\\end{enumerate*}\n" ], "text/markdown": [ "1. 0.249628205480316\n", "2. 0.0155845190765748\n", "\n", "\n" ], "text/plain": [ "[1] 0.24962821 0.01558452" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "MSE_lm_basic <- summary(lm((Y_test - yhat_lm_basic)^2~1))$coef[1:2]\n", "MSE_lm_basic" ] }, { "cell_type": "markdown", "metadata": { "id": "WSJe5AWiWitw", "papermill": { "duration": 0.040507, "end_time": "2021-07-22T21:33:23.223781", "exception": false, "start_time": "2021-07-22T21:33:23.183274", "status": "completed" }, "tags": [] }, "source": [ "We also compute the out-of-sample $R^2$:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "execution": { "iopub.execute_input": "2021-07-22T21:33:23.310177Z", "iopub.status.busy": "2021-07-22T21:33:23.308577Z", "iopub.status.idle": "2021-07-22T21:33:23.323143Z", "shell.execute_reply": "2021-07-22T21:33:23.321848Z" }, "executionInfo": { "elapsed": 24, "status": "ok", "timestamp": 1658250112095, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "6iCw0TS4Witw", "outputId": "b7a5abc6-be2d-4852-9c68-dffdd5ce07ce", "papermill": { "duration": 0.058991, "end_time": "2021-07-22T21:33:23.323276", "exception": false, "start_time": "2021-07-22T21:33:23.264285", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The R^2 using the basic model is equal to 0.2185201" ] } ], "source": [ "R2_lm_basic <- 1 - MSE_lm_basic[1] / var(Y_test)\n", "cat(\"The R^2 using the basic model is equal to\", R2_lm_basic) # MSE OLS (basic model) " ] }, { "cell_type": "markdown", "metadata": { "id": "OxxeDdauWity", "papermill": { "duration": 0.040697, "end_time": "2021-07-22T21:33:23.404884", "exception": false, "start_time": "2021-07-22T21:33:23.364187", "status": "completed" }, "tags": [] }, "source": [ "We repeat the same procedure for the flexible model." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "execution": { "iopub.execute_input": "2021-07-22T21:33:23.491381Z", "iopub.status.busy": "2021-07-22T21:33:23.490191Z", "iopub.status.idle": "2021-07-22T21:33:23.637474Z", "shell.execute_reply": "2021-07-22T21:33:23.635485Z" }, "executionInfo": { "elapsed": 21, "status": "ok", "timestamp": 1658250112096, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "4PUFKVx1Wity", "outputId": "9b400580-29da-42ad-ddbb-5a6b9442279d", "papermill": { "duration": 0.192034, "end_time": "2021-07-22T21:33:23.637702", "exception": false, "start_time": "2021-07-22T21:33:23.445668", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The R^2 using the flexible model is equal to 0.2165618" ] } ], "source": [ "# ols (flexible model)\n", "fit_lm_flex <- lm(formula_flex, data_train) \n", "# Compute the Out-Of-Sample Performance\n", "options(warn=-1)\n", "yhat_lm_flex <- predict(fit_lm_flex, newdata = data_test)\n", "MSE_lm_flex <- summary(lm((Y_test - yhat_lm_flex)^2~1))$coef[1:2]\n", "R2_lm_flex <- 1 - MSE_lm_flex[1] / var(Y_test)\n", "cat(\"The R^2 using the flexible model is equal to\", R2_lm_flex) # MSE OLS (flexible model) " ] }, { "cell_type": "markdown", "metadata": { "id": "smBG74geWity", "papermill": { "duration": 0.055232, "end_time": "2021-07-22T21:33:23.764568", "exception": false, "start_time": "2021-07-22T21:33:23.709336", "status": "completed" }, "tags": [] }, "source": [ "We observe that ols regression works better for the basic model with smaller $p/n$ ratio. We are proceeding by running lasso regressions and its versions." ] }, { "cell_type": "markdown", "metadata": { "id": "4gRVP4umWitz", "papermill": { "duration": 0.041825, "end_time": "2021-07-22T21:33:23.870407", "exception": false, "start_time": "2021-07-22T21:33:23.828582", "status": "completed" }, "tags": [] }, "source": [ "### Lasso, Ridge and Elastic Net\n" ] }, { "cell_type": "markdown", "metadata": { "id": "8_l0nfHyWitz", "papermill": { "duration": 0.04152, "end_time": "2021-07-22T21:33:23.953937", "exception": false, "start_time": "2021-07-22T21:33:23.912417", "status": "completed" }, "tags": [] }, "source": [ "Considering the basic model, we run a lasso/post-lasso regression first and then we compute the measures for the out-of-sample performance. Note that applying the package *hdm* and the function *rlasso* we rely on a theory-based choice of the penalty level $\\lambda$ in the lasso regression." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "execution": { "iopub.execute_input": "2021-07-22T21:33:24.047109Z", "iopub.status.busy": "2021-07-22T21:33:24.041516Z", "iopub.status.idle": "2021-07-22T21:33:24.535625Z", "shell.execute_reply": "2021-07-22T21:33:24.532403Z" }, "executionInfo": { "elapsed": 19, "status": "ok", "timestamp": 1658250112097, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "Is9ir6zuWit0", "outputId": "05db9dec-f46a-4f8b-ee43-fee29e3ea4af", "papermill": { "duration": 0.539956, "end_time": "2021-07-22T21:33:24.535881", "exception": false, "start_time": "2021-07-22T21:33:23.995925", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The R^2 using the basic model is equal to 0.2184129 for lasso and 0.2220093 for post-lasso" ] } ], "source": [ "# lasso and versions\n", "# library(hdm) \n", "fit.rlasso <- rlasso(formula_basic, data_train, post=FALSE)\n", "fit.rlasso.post <- rlasso(formula_basic, data_train, post=TRUE)\n", "yhat.rlasso <- predict(fit.rlasso, newdata=data_test)\n", "yhat.rlasso.post <- predict(fit.rlasso.post, newdata=data_test)\n", "\n", "MSE.lasso <- summary(lm((Y_test-yhat.rlasso)^2~1))$coef[1:2]\n", "MSE.lasso.post <- summary(lm((Y_test-yhat.rlasso.post)^2~1))$coef[1:2]\n", "\n", "R2.lasso <- 1-MSE.lasso[1]/var(Y_test)\n", "R2.lasso.post <- 1-MSE.lasso.post[1]/var(Y_test)\n", "cat(\"The R^2 using the basic model is equal to\",R2.lasso,\"for lasso and\",R2.lasso.post,\"for post-lasso\") # R^2 lasso/post-lasso (basic model) " ] }, { "cell_type": "markdown", "metadata": { "id": "4D3os7iXWit1", "papermill": { "duration": 0.051245, "end_time": "2021-07-22T21:33:24.660529", "exception": false, "start_time": "2021-07-22T21:33:24.609284", "status": "completed" }, "tags": [] }, "source": [ "Now, we repeat the same procedure for the flexible model." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "execution": { "iopub.execute_input": "2021-07-22T21:33:24.749884Z", "iopub.status.busy": "2021-07-22T21:33:24.748227Z", "iopub.status.idle": "2021-07-22T21:33:27.945305Z", "shell.execute_reply": "2021-07-22T21:33:27.942260Z" }, "executionInfo": { "elapsed": 315, "status": "ok", "timestamp": 1658250112397, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "GWGYhDYUWit1", "outputId": "4f3718c4-a1ab-4126-8092-e921add22961", "papermill": { "duration": 3.242671, "end_time": "2021-07-22T21:33:27.945527", "exception": false, "start_time": "2021-07-22T21:33:24.702856", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The R^2 using the flexible model is equal to 0.2167083 for lasso and 0.2220093 for post-lasso" ] } ], "source": [ "fit.rlasso.flex <- rlasso(formula_flex, data_train, post=FALSE)\n", "fit.rlasso.post.flex <- rlasso(formula_flex, data_train, post=TRUE)\n", "yhat.rlasso.flex <- predict(fit.rlasso.flex, newdata=data_test)\n", "yhat.rlasso.post.flex <- predict(fit.rlasso.post.flex, newdata=data_test)\n", "\n", "MSE.lasso.flex <- summary(lm((Y_test-yhat.rlasso.flex)^2~1))$coef[1:2]\n", "MSE.lasso.post.flex <- summary(lm((Y_test-yhat.rlasso.post.flex)^2~1))$coef[1:2]\n", "\n", "R2.lasso.flex <- 1-MSE.lasso.flex[1]/var(Y_test)\n", "R2.lasso.post.flex <- 1-MSE.lasso.post.flex[1]/var(Y_test)\n", "cat(\"The R^2 using the flexible model is equal to\",R2.lasso.flex,\"for lasso and\",R2.lasso.post.flex,\"for post-lasso\") # R^2 lasso/post-lasso (flexible model) " ] }, { "cell_type": "markdown", "metadata": { "id": "oVYQoNUSWit2", "papermill": { "duration": 0.051694, "end_time": "2021-07-22T21:33:28.071250", "exception": false, "start_time": "2021-07-22T21:33:28.019556", "status": "completed" }, "tags": [] }, "source": [ "The lasso regression works better for the more complex model." ] }, { "cell_type": "markdown", "metadata": { "id": "p19rDtdYWit2", "papermill": { "duration": 0.043014, "end_time": "2021-07-22T21:33:28.156922", "exception": false, "start_time": "2021-07-22T21:33:28.113908", "status": "completed" }, "tags": [] }, "source": [ "In contrast to a theory-based choice of the tuning parameter $\\lambda$ in the lasso regression, we can also use cross-validation to determine the penalty level by applying the package *glmnet* and the function cv.glmnet. In this context, we also run a ridge and a elastic net regression by adjusting the parameter *alpha*." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "execution": { "iopub.execute_input": "2021-07-22T21:33:28.245441Z", "iopub.status.busy": "2021-07-22T21:33:28.244334Z", "iopub.status.idle": "2021-07-22T21:33:30.309699Z", "shell.execute_reply": "2021-07-22T21:33:30.308186Z" }, "executionInfo": { "elapsed": 676, "status": "ok", "timestamp": 1658250113060, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "VO4vXSlsWit3", "outputId": "fb9c95e3-14a6-4153-996c-6ed658693581", "papermill": { "duration": 2.110744, "end_time": "2021-07-22T21:33:30.309881", "exception": false, "start_time": "2021-07-22T21:33:28.199137", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Loading required package: Matrix\n", "\n", "\n", "Attaching package: ‘Matrix’\n", "\n", "\n", "The following objects are masked from ‘package:tidyr’:\n", "\n", " expand, pack, unpack\n", "\n", "\n", "Loaded glmnet 4.1-4\n", "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "R^2 using cross-validation for lasso, ridge and elastic net in the basic model: 0.2203463 0.2055758 0.2192991" ] } ], "source": [ "library(glmnet)\n", "fit.lasso.cv <- cv.glmnet(model_X_basic_train, Y_train, family=\"gaussian\", alpha=1)\n", "fit.ridge <- cv.glmnet(model_X_basic_train, Y_train, family=\"gaussian\", alpha=0)\n", "fit.elnet <- cv.glmnet(model_X_basic_train, Y_train, family=\"gaussian\", alpha=.5)\n", "\n", "yhat.lasso.cv <- predict(fit.lasso.cv, newx = model_X_basic_test)\n", "yhat.ridge <- predict(fit.ridge, newx = model_X_basic_test)\n", "yhat.elnet <- predict(fit.elnet, newx = model_X_basic_test)\n", "\n", "MSE.lasso.cv <- summary(lm((Y_test-yhat.lasso.cv)^2~1))$coef[1:2]\n", "MSE.ridge <- summary(lm((Y_test-yhat.ridge)^2~1))$coef[1:2]\n", "MSE.elnet <- summary(lm((Y_test-yhat.elnet)^2~1))$coef[1:2]\n", "\n", "R2.lasso.cv <- 1-MSE.lasso.cv[1]/var(Y_test)\n", "R2.ridge <- 1-MSE.ridge[1]/var(Y_test)\n", "R2.elnet <- 1-MSE.elnet[1]/var(Y_test)\n", "\n", "# R^2 using cross-validation (basic model) \n", "cat(\"R^2 using cross-validation for lasso, ridge and elastic net in the basic model:\",R2.lasso.cv,R2.ridge,R2.elnet)" ] }, { "cell_type": "markdown", "metadata": { "id": "HXyV0kqUWit4", "papermill": { "duration": 0.04689, "end_time": "2021-07-22T21:33:30.403846", "exception": false, "start_time": "2021-07-22T21:33:30.356956", "status": "completed" }, "tags": [] }, "source": [ "Note that the following calculations for the flexible model require significant computation time." ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "execution": { "iopub.execute_input": "2021-07-22T21:33:30.499405Z", "iopub.status.busy": "2021-07-22T21:33:30.498099Z", "iopub.status.idle": "2021-07-22T21:33:44.141438Z", "shell.execute_reply": "2021-07-22T21:33:44.139990Z" }, "executionInfo": { "elapsed": 883, "status": "ok", "timestamp": 1658250113936, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "-CtIV7FDWit4", "outputId": "f1154812-104e-4a4b-946d-b98eca9e2097", "papermill": { "duration": 13.691447, "end_time": "2021-07-22T21:33:44.141768", "exception": false, "start_time": "2021-07-22T21:33:30.450321", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "R^2 using cross-validation for lasso, ridge and elastic net in the flexible model: 0.2201523 0.2062207 0.2211247" ] } ], "source": [ "fit.lasso.cv.flex <- cv.glmnet(model_X_flex_train, Y_train, family=\"gaussian\", alpha=1)\n", "fit.ridge.flex <- cv.glmnet(model_X_flex_train, Y_train, family=\"gaussian\", alpha=0)\n", "fit.elnet.flex <- cv.glmnet(model_X_flex_train, Y_train, family=\"gaussian\", alpha=.5)\n", "\n", "yhat.lasso.cv.flex <- predict(fit.lasso.cv.flex , newx = model_X_flex_test)\n", "yhat.ridge.flex <- predict(fit.ridge.flex , newx = model_X_flex_test)\n", "yhat.elnet.flex <- predict(fit.elnet.flex , newx = model_X_flex_test)\n", "\n", "MSE.lasso.cv.flex <- summary(lm((Y_test-yhat.lasso.cv.flex )^2~1))$coef[1:2]\n", "MSE.ridge.flex <- summary(lm((Y_test-yhat.ridge.flex )^2~1))$coef[1:2]\n", "MSE.elnet.flex <- summary(lm((Y_test-yhat.elnet.flex )^2~1))$coef[1:2]\n", "\n", "R2.lasso.cv.flex <- 1-MSE.lasso.cv.flex [1]/var(Y_test)\n", "R2.ridge.flex <- 1-MSE.ridge.flex [1]/var(Y_test)\n", "R2.elnet.flex <- 1-MSE.elnet.flex [1]/var(Y_test)\n", "\n", "# R^2 using cross-validation (flexible model) \n", "cat(\"R^2 using cross-validation for lasso, ridge and elastic net in the flexible model:\",R2.lasso.cv.flex,R2.ridge.flex,R2.elnet.flex)" ] }, { "cell_type": "markdown", "metadata": { "id": "TjeCfaakWit6", "papermill": { "duration": 0.043216, "end_time": "2021-07-22T21:33:44.229372", "exception": false, "start_time": "2021-07-22T21:33:44.186156", "status": "completed" }, "tags": [] }, "source": [ "The performance of the lasso regression with cross-validated penalty is quite similar to the performance of lasso using a theoretical based choice of the tuning parameter." ] }, { "cell_type": "markdown", "metadata": { "id": "ypNrjgnwWit6", "papermill": { "duration": 0.043279, "end_time": "2021-07-22T21:33:44.316231", "exception": false, "start_time": "2021-07-22T21:33:44.272952", "status": "completed" }, "tags": [] }, "source": [ "## Non-linear models" ] }, { "cell_type": "markdown", "metadata": { "id": "qkm_TFteWit6", "papermill": { "duration": 0.043761, "end_time": "2021-07-22T21:33:44.403514", "exception": false, "start_time": "2021-07-22T21:33:44.359753", "status": "completed" }, "tags": [] }, "source": [ "Besides linear regression models, we consider nonlinear regression models to build a predictive model. We are apply regression trees, random forests, boosted trees and neural nets to estimate the regression function $g(X)$. First, we load the relevant libraries." ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2021-07-22T21:33:44.494395Z", "iopub.status.busy": "2021-07-22T21:33:44.493268Z", "iopub.status.idle": "2021-07-22T21:33:44.752574Z", "shell.execute_reply": "2021-07-22T21:33:44.751346Z" }, "executionInfo": { "elapsed": 5, "status": "ok", "timestamp": 1658250113937, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "LHITdYUFWit6", "papermill": { "duration": 0.30601, "end_time": "2021-07-22T21:33:44.752736", "exception": false, "start_time": "2021-07-22T21:33:44.446726", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [], "source": [ "library(randomForest)\n", "library(rpart)\n", "library(nnet)\n", "library(gbm)\n", "library(rpart.plot)\n", "library(keras)" ] }, { "cell_type": "markdown", "metadata": { "id": "s8QdDVokWit7", "papermill": { "duration": 0.043594, "end_time": "2021-07-22T21:33:44.840683", "exception": false, "start_time": "2021-07-22T21:33:44.797089", "status": "completed" }, "tags": [] }, "source": [ "and we illustrate the application of regression trees." ] }, { "cell_type": "markdown", "metadata": { "id": "npHGkYKsWit7", "papermill": { "duration": 0.043219, "end_time": "2021-07-22T21:33:44.927600", "exception": false, "start_time": "2021-07-22T21:33:44.884381", "status": "completed" }, "tags": [] }, "source": [ "### Regression Trees" ] }, { "cell_type": "markdown", "metadata": { "id": "GBDE9z5yWit7", "papermill": { "duration": 0.043147, "end_time": "2021-07-22T21:33:45.014486", "exception": false, "start_time": "2021-07-22T21:33:44.971339", "status": "completed" }, "tags": [] }, "source": [ "We fit a regression tree to the training data using the basic model. The variable *cp* controls the complexity of the regression tree, i.e. how deep we build the tree." ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 454 }, "execution": { "iopub.execute_input": "2021-07-22T21:33:45.108375Z", "iopub.status.busy": "2021-07-22T21:33:45.106930Z", "iopub.status.idle": "2021-07-22T21:33:46.833179Z", "shell.execute_reply": "2021-07-22T21:33:46.833557Z" }, "executionInfo": { "elapsed": 1544, "status": "ok", "timestamp": 1658250115476, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "z4-0ZKerWit7", "outputId": "cdd56787-b884-4954-9572-c873e7829de7", "papermill": { "duration": 1.774435, "end_time": "2021-07-22T21:33:46.833743", "exception": false, "start_time": "2021-07-22T21:33:45.059308", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "cex 0.2 xlim c(0, 1) ylim c(0, 1)\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAACDVBMVEUeHh40NDQ4ODhAQEBERERGRkZHR0dKSkpMTExOTk5QUFBRUVFSUlJVVVVWVlZYWFhbW1tdXV1eXl5gYGBhYWFiYmJkZGRlZWVmZmZoaGhpaWlra2tsbGxtbW1ubm5vb29wcHBxcXFycnJzc3N0dHR1dXV2dnZ3d3d4eHh5eXl6enp8fHx9fX1+fn5/f3+AgICBgYGCgoKDg4OEhISFhYWGhoaHh4eIiIiJiYmKioqLi4uMjIyNjY2Ojo6Pj4+QkJCRkZGSkpKTk5OUlJSVlZWWlpaXl5eYmJiZmZmampqbm5ucnJydnZ2enp6fn5+goKChoaGioqKjo6OkpKSlpaWmpqanp6eoqKipqamqqqqrq6usrKytra2urq6vr6+wsLCxsbGysrKzs7O0tLS1tbW2tra3t7e4uLi5ubm6urq7u7u8vLy9vb2+vr6/v7/AwMDBwcHCwsLDw8PExMTFxcXGxsbHx8fIyMjJycnKysrLy8vMzMzNzc3Ozs7Pz8/Q0NDR0dHS0tLT09PU1NTV1dXW1tbX19fY2NjZ2dna2trb29vc3Nzd3d3e3t7f39/g4ODh4eHi4uLj4+Pk5OTl5eXm5ubn5+fo6Ojp6enq6urr6+vs7Ozt7e3u7u7v7+/w8PDx8fHy8vLz8/P09PT19fX29vb39/f4+Pj5+fn6+vr7+/v8/Pz9/f3+/v7///8uuT0YAAAACXBIWXMAABJ0AAASdAHeZh94AAAgAElEQVR4nOy9iX8Vx5Uvnpk3eRnPYzLjx8SZOJkxQckYZrEd5RfbY4dgnDiLh3E83l4c9sUGSdwjsWgBsSM2gVjMIkBIIEBcgaT+G3+3+/ZSy6m1q7er/n5At29XddW5VfXtqjp16tS3vBo1aqTGt4oWoEaNTkBNpBo1HKAmUo0aDlATqUYNB6iJVKOGA9REqlHDAWoi1ajhADWRatRwgJpINWo4QE2kGjUcoCZSjRoOUBOpRg0HqIlUo4YD1ESqUcMBaiLVqOEANZFq1HCAmkg1ajhATaQaNRygJlKNGg5QE6lGDQeoiVSjhgPURKpRwwFqItWo4QA1kWrUcICaSDVqOEBNpBo1HKAmUo0aDlATqUYNB6iJVKOGA9REqlHDAWoi1ajhADWRatRwgJpINWo4QE2kGjUcoCZSjRoOUBOpRg0HqIlUo4YD1ESqUcMBaiLVqOEANZFq1HCAmkg1ajhATaQaNRygJlKNGg5QE6lGDQeoiVSjhgPURKpRwwFqItWo4QA1kWrUcICaSDVqOEBNpBo1HKAmUo0aDlATqUYNB6iJVKOGA9REqlHDAWoilR3rVnrT/ufG7o3dRctSQ4iaSEVjby+PvjE/ZO3q7254dcP1l7y9az74cHqmeyYg0jY++vFif0ENryZS4XgG7c+dWz7e9utz0d0ds60/a9esWP/j9d9+e/7znZs+mtm84pMVTzzvYX87xh83fbkpjj09nafINRDURCoUSwO90L56++bvzuzoi+7//pLoifMhkfZ89cmeOPbo6NdDGUlYQw81kYrEmV0Pw6Harm5qsHZU/MwX/NBusHV7tv9AbmLX4FETqTjc7L3mMrnZ/ftdJlfDCDWRisKTfSdcJznVe8R1kjU0UROpGCwN9i9lkOy93tEMUq2hRk2kQnBqz1xGKd/bMZJRyjVkqIlUAL7pvZ1h6nd7T2WYeg0cNZFyx6OBsxnncKf3dMY51GBREylnLA4O5JDL9R1ncsilRoKaSPli9KtmPhld7x3LJ6MaAWoi5YlLvXfyy+zajnqAlx9qIuWH+70X8s3wau94vhkuY9REygvPBobzz/Rqr9Bor4ZT1ETKCYd2PS8k3/M7LheS73JDTaRccHrLg8LyvtA7UVjeywc1kXLArR1XCs3//I6aSlmjJlLmmOs5VrQI3vneG0WL0OGoiZQxlg72LBQtg49zvTeLFqGjURMpW5wJNo2XAsd23CpahA5GTaQscWPH1aJFIHEiU2PZ5Y2aSNnh8VfFT44YHN8xWbQIHYqaSFlhsdGTxda9tDi+937RInQkaiJlhJFtT4sWQYATX9fOu9yjJlImuFTqIdTw5pmiReg41ETKADN7z6kjFYqju4qztOhM1ERyjufQX7QIGji282HRInQUaiK5xujunLbupcTSYM+TomXoINREcotL2+8WLYI2loYy82W0/FATySWmdpR9ckRjsX9bTSU3qInkDs96DhUtgjEW+7eWVU1fLdREcobBnaWwTjXF4lBe/lg6GjWRHOHU5soqlJ/1V0Q/UmbURHKCm+WyTjXFs4G+YjbCdw5qIjnAk77KHz75bH9PTaU0qImUGkuH9i4WLYMDzPfsquQcrySoiZQWp7c/KloER5jf31tTyRY1kdLhxg6np+4VjObBnk7oXItATaQ0ePx16bbupcTjffvLuIuq/KiJZI9F2NuBje5xY6ADf1XmqIlkjZFOtQl49HVvTSVT1ESyxHipt+6lxGz/gaJFqBpqIllhuud80SJki9mB/UWLUC3URLLAs34oWoTsMdV7pGgRqoSaSOY4unO+aBFywb3e0aJFqA5qIplifNu9okXIDfd2jBQtQlVQE8kMUzs6fHLE4G7vqaJFqAZqIplgvnewaBFyx53e+ihaDdREMsBQNbfupcX1HWeKFqH8qImkjZOblq0Dq296ayopUBNJEzd7K711Ly2u76gHeFLURNLCk30nihahaFzrHS9ahDKjJpIGlgb7a+Mzz7vae6loEcqLmkhqnKr9KIY4v+Ny0SKUFTWRVPimPuaOwIXe+oB0FDWR5HjUf7ZoEUqGC3WvhKEmkgyLgwNFi1BCnO+9UbQI5UNNJAlGax+kOM713ixahLKhJpIQl3vvFC1CeXFsx62iRSgXaiIJcL93eVmnGuNErYQhURMJxbOB4aJFKD9OdPJue1PURMJwtLf236uD49vq0W+Imkg8Ln5dH/qtixNfTxctQjlQE4nFvd4rRYtQKQxvqV87Xk0kFvP7O813avYY3lTZo6HcoSYShdGeZbl1Ly2O7Vy2W7Ui1EQicGbnbNEiVBRLg9uXOZVqIsWo+Kl7BWNpaO+ytpGviRTi8e56cpQOS+cOL2OLqppIARYbPfXWPTfY2O15c083dq9bWbQkuaImko/D2zr0YIlcsXrVz1avXDPc7b25cW6m+8ZLRcuTK2oied6l2tTFCbree2P1iz/5xYonp35zf/OK//X28nDsHKIm0szec0WLUHU8vXw5sgXvYoKSkA7HcifS80Z/0SJUH1vFdLl8eWh5uMRb5kQa3b2MFU2u0JAbNgwui02Ay5pIl3bcLVqEDsAVlevIncvhZbWMiTTTV7tpc4DmV6oYi3/OQ46CsYyJVMMJtqqtE2cbOchRMJYfkUK90s//uVgxOgUj32hEutz5ViPLjUhrN/y460enN384/epry9o0zBXu6m3JH+14M8blRqSfrn/lRy+f3vbRTPeqokXpBCwpJ0ghvnqcqRzFo9OJ9M3u3gjRycJfrpqKg0fj0K9qV7wW+Eq3W1/6c4fbMnY4kS4e1I56Qj9qjRBn9I1C5nozlKME6GwiHTM51eju9meZCdKZeGyijfumsw9I72giDZgtFD3fXbs8NMKXRsO1kx19UFknE6lhbJtyeHnYhTnC4JQ6DomeTnY31LlEWthi4UXg0i73gnQqbhgvDm3qYM8yHUukx5usZjyPtnW6ntYVnutqvhM825OBHCVBpxJpcqftkwMXXcrRudhm8aKa7FzNaIcS6WoK665aD66DE9dsnjrbsecfdiaRzhxJ83StB1dj+oDdcwOdunGlI4k0lHLzeK0HV2K77YN2U9fyoxOJ1Hs9dRK1HlyOPdYuaRe+dilHedB5RFrae89BKpdrPbgEF07ZP3u/MzcndRyRmpvcbI+o9eBiPO1L8/R51db0SqLTiDS9ZdFVUrUeXISt6Sy5hzrRG0qHEemmyyW/Wg+O43BaVUwnekPpLCJdcNv0az04hptDaVPoRG8oHUWkoycdJ1jrwXksmpsGcehAbyidRKSBy+7TrPXgLHa7cOk90XHeUDqISH2ZHFVf68FpjLl5W3WcN5SOIdLC5oyOraz14CQeuHKV3mneUDqFSJa7JrRQ68ETbHGVUKd5Q+kQItnvmtBBrQeP0DvtLKkO84bSGUS6krEWqNaDt3ElhWkQh87yhtIRRDqdateEDmo9uI9nbjuRjvKG0glESrtrQgtHaj24jr98I3SSN5QOIFKvjhv39Kj14CPpt6cw6CBvKJUn0tJeQ6dQ1ljuevC7qU2DOHSQN5SqE6m56Wl+mS1rPbi2v3wTTFruWC8fKk6k6c3Odk3oYDnrwbX95RuhY7yhVJtI11PtMLPA8tWDj2Wk0dnfId5QKk2kC4dyz3K56sEf78sq5Q7xhlJlIjnfNaGF5akHz86gZyGLuVf+qDCRstg1oYPlqAc39ZdvgvuQXdr5obpE2pfJrgkdLD89uLm/fBNc7gSFQ1WJlNmuCS0sMz34893Zpt8J3lAqSqTHm58Xmv/y0oPb+Ms3Qgd4Q6kmkSZ3FC3BctKD2/nLN0EHeEOpJJGuDRYtwXLSg9v6yzdB9b2hVJFI2e+a0MJy0YNb+8s3QeW9oVSQSIN57JrQwfLQg9v7yzdC1b2hVI9IOe2a0MFy0IOn8ZdvhIp7Q6kakZa+LtVmsI7Xg6fzl2+CintDqRiRct01oYNO14On9Jdvgmp7Q6kWkaY35bprQgedrQdP7S/fBJX2hlIpIl3Le9eEDjpZD57eX74RquwNpUpEKmDXhBY6Vg++mLFpEIcKe0OpEJGK2TWhg07Vg3/lwl++EarrDaU6ROovaNeEDjpTD+7IX74JqusNpSpEWtpd2K4JLXSgHvwBFJDpZFW1oBUh0vNdT4oWQYHO04M785dvhKp6Q6kGkR4VvGtCB52mB+9z5y/fCBX1hlIJIt0ufNeEDjpLD+7UX74RNuWu4nCBKhBpfH/REmiig/Tgjv3lm6Ca3lAqQKTTo0VLoI3O0YO79pdvgkp6Qyk/kQbPFy2BATpFD+7eX74JqugNpfRE6ivPrgktdIQe/O5wsflX0BtKyYm09PWDokUwRQfowZfyNg3iUD1vKOUmUul2TejgXuX14F8XvmhXPW8opSbS/fLtmtDBwsj9okVIhaz85ZvgyfGiJTBEmYlUyl0TmtjY7XlzT9et9L7/adGimOJx8cXeFV/55VgJlJhIYwXPeC2xetXPVq9cM9ztvblx7sZL3u7uqhk0l2DL9z/+8G9e7/rR6c0fTs/UREqL0cKW1tOh6703Vr/4k1+seHLqN/e/8/af3vyHwiccZsjSX74uur74pxdefvn0to9mNq+oSPGVlkj9E0VLkAJd6ihlRbb+8g3w5aoSMFofJSXS0lfVNF2sPLL2l9+xKCeRyr9rQgXwvMWBooWwQOb+8nXRiP9UA6UkUhV2TSgA4f+K4VhptiEHHBqpjsFVGYk0VcVXOY15308LFC2FMR6WZ/UmINLd00WLoY0yEqkDcP2SV0UilQdL7ZcpFCuFAUpGpHUr/XXMjd2/eKNoSdLhsD8mgaKl0IR/uMeH31vpBVtiN3aXYgn5xoXgozqTpNIQae3q7254dcP1l/x1zJnuv//PShoHxdjn/4GChdBF1w9W/+3Le17y9q75wF8BLcUS8qG2kWVNJGOsXbNi/Y/Xf/vt+VO/ub95xWvfq+R+4xhBA4CChdBFV9cXf/nDb709//nOTf4K6L+XYQm5QX1UAGUg0mHB/YFHuYrhElVT3pYONZHM8UC4gadytvQxwP9zrCJ7qZit/LNl2JoYMmigBKNMPZSASJuFIbOV1YOD/+d2NXZMz7OlvLcQMWiERJq4UqgUBiieSEOT4rDzJdgZYwXw/1TEtKHBGjN8M1aIHBRCIi1UxYFU8UQSD+x89DzMSw63gOBvNUb4Pdyd4rukhegk9WoUoVcCIokHdgEq4RqSR4P4W3Jc4E9BPnetADkoXIls/ytRhD6KJtKwZGDn4xHkIoZjPG4rIivRCvgOqQRd0v5ouFmJIvRRMJEeHFDFuFCNGTuN8bZbOChWCi3MDSI3Dxe9FajBXZQdBRNJ48SDfRVRIpM42PYmBcVKoYUeVMFctNuGmD/7ipTCBMUS6ahiYBegBD4ETAHUR6mBj+L2z+UsBoOYSKeK7ht1USiRHmgpNx9X79h4aH9UYFxyGvdpulhwlxSX3BOR2UvZUCiRNI+yOn8iWzHcA9ofR8uvvMdUDcH9Yk0KGshVuVEkkY7oeng+UJX+PUJY+7dLv578YEQQ0FRqgbLEfHJ+fU0kJR7qL/xvqtg0Kaz9xdKvy/cIC7ZQDfh4cnJCTSQltupHfVoZ5U0bUe1DkULoQDz9fFDkqVT9yW60mkgqHLlhEHniZGZyZAFgPsuKwxIn5UV2SQ30stQojEgGAzsfB+9lJEcWWOoPL6BIKTQgUjX4uFHgOZ4Ee4YKVsTrojAibTOMv6VC06T7Uf8JRUqhxj2pOrTALokg0p0SmKLroCgiGQ3sfDyVvT1LhrE74QUUKYUaX0tDLxbnNJocz0FRQpihICI97FfHYXClOtOkeF8nFCmFEkuK+UdxXVJDcF1iFEQk04Gdj0OVmSZVxOTy4Kw8fLSoAp8jD/QpdxnGKIZIo6YDuwCbqrKBP677UUVTLRbKwXJRo+mzt4kvNZHEeGy3UNmUj+nLA4gubp0vUAoVbir9AQ8U5HobSMVSTSQxtls+N1ERC0aILkrtckA9BVoqaB2c4k5/NcYhRRDJbmDnY/i2Ok4JAMhV6fBc41XfV8ypIJRklwvf966FAohkObALsKUSrydArkqH/qfqOE1z5aoLUER6Xqj5rDYKIFIafybPCvcmoIOkIUBxQqigpUn4upBl8IbkW1mRP5FGv1HHEeObIm0pNUHMjKA4KRS4PK4T66Fom0WmqImkgccpvSYevuVGjgxxK9mGBMVJoYBm117ECGD2CPW1JhKKzWkHC9tLP006PhNfQnFSyNHUnKhOFmC6yvhpqImEId3Azsfzr1zIkSUguSxtI+jTPXS5gC6pIf1aUuRMpMcO1EA3jqjjFApILo+U9WQabaOF8fyPZ2aYc0Kyaao8yJlIqQd2PoauO0gkQ0ByefNCYVJIcV5/cSb/Lokh0qOyvzgD5Euko6kHdgG26I5LigHREBZKughiwI4TuS+Cs2O5SoztciWSi4Gdj+e73KSTEcq/m+bxIXWcGLl3STWRVHC2y/WGSUPIHeXfTdNrctT1wbznecB8L2cZMsiTSI4Gdj6OlXmaBMR1ORuB0f6IpZy9rk6zG+DLWYYMciTSE3CY2PYSn3oOxHUpG8FxHZfrCXrznZIeZU9NKGUZssiRSE7dlyyWdzWpOUR8KWUjMJz1PMv3CE+uyA5pmNcWjvyIdMrdwM7HnSF1nGJwlXQaUkYizZjqk/fmarrKFVn5PT97ORLpiesmdbys+1SGnxBfykgkY17M5vrS4ousmM0cZsiNSFudv9V2NV2n6AZUQzhcQtMGc+VBrhpwnkhlfBuxyItIJ/gjf9NisaSrSVS137hYlBhCDE8bPzJ5PAM5RKiJJMGTLHb/38GOPy0eZd/fadO95NklAXenJlKMbZlMV49eyiLVtADJtxLg7imLh67kZ7qK+CiuiRThxJVs0t1RRg/rIPlWAtitG+TXJY3wPsD6TQwxCkIuRHqS1dr4oq1jrywBkm/Fw/J02JO5ma4i3c+425WTTJALkbZntg5xv4SO40DyrXgctHT6mFuXhBDpWfkmmhzyINLJK9mlfbp80ySQfCsetoQ4yBruZAVsQlSBSVIORMpsYBdg9xN1nFzxkPZzBMVIIcL1s7ZP5uV1tSaSCNkN7HwslW016QLtRxaKkUIEe7/4+3IyE66JJECWAzsf0yWbJh2gbaXLZdrw3N7a5hm4E0OCJcxCtiaS5z2FrHM4o+XqMDcA/fWbUkkHKQyp9+SihL6BebmoieR5O7K3HN5TqmkS0F+fHyxECgHS6N4e52K6iu6ZOG5u1ZQ3sibSyRwOIl3alH0e+gDF9yJxOZWOMxcNONr5zJbfUXXGRGrm0inPlMnOnv3FUIQQAqQ7gu/OMUdiyIA3mPKP7TImUg4DOx+nrZW67lFiIjVTDjPz6JJqImHIY2AX4OvyHNZaYiL1pHS+cCUHd5c1kRA08zs68c+55aQCKL4XiNQ9Sg5dUk0kBFsqYLXrGug6SDkwZn3kaH4o9am7MmRJpFN5DewCbOz2vLmnG7vX/mMhp8xFuKc8K7wwpFM15IOrubYZh8iQSLkN7Fav+tnqlWuGu703N87NdP/L9wrtB0/fiy99Zz0ffm+lFyyC3PuLoiSK8MjBOlCX/8d/ZWWF/fg0LvN8UyNDIm3Nq0F3vffG6hd/8osVT0795v7mFS//wHKjgBsQm9C6frD6b1/e85K3d80HH053/12BQgXoSVkfr7y+dsOPvc0fTs9k2KCRyVAu+aZGdkTKd2DXlWdmMhAtoavri7/84bfenv9856aPZl77zpT4oVyQ1gj/4xdWr3/F2/bRzOYV2ZmSIETKJd/UyIxIzZw9RpcFpdUvHb1TtAQ6oIuvVAa/cmRGpNwGdgucWUMRBwjHAPprI/5TNFKVyqPDwUf2pjp0USEyj9zNWgQ7ZEWkU7m5ndnP9fc3eD80+QHor0HDyG85TYyZo2mePhLujwUHkkhBEekZpgvfk7UIdsiISM9SVZsREKVuMV3S3V4fG/w/EDuBDRpGGU5BPZxqUSB6FUB6QYRYGuzt3fPr3t6BWHG3D1PhDZezS8r7VHPnwPZOn76VvxyeR2zVXfg6ugjeqeHAqMLIg0gQalufxe/BXjReoSN3IdwSKVo58eaerlu5bqXTtAVAi7WQsobW/12/8v74vpc0uOvtfQsFTpLcVEn0A8CNUCj8tLe9Q2RyeAaNN3QPvV0w3BIpWjl5c+PcjZduvOQ0bRzP0TZ6oAh1D7T+z/3e20i2hUPtMV6BRHJTJdEPyNLDHLT+z77pJYUnssQoZZfkmEjhysmp39z/ztt/9XYO7jIAXVpYKmJ6D63/r+04u/HXi63rqTa2TU35VCqSSC6qZD46s3cpQ1M4aP3f8lar8La3y270aLvwOAyXYMrJoepzJMFba+9CvmL42J1cLn400cZnE0UTyQlOxaMpyC6TRuh/euHjdtn9z8QETqRSGg26JlJ4/gd7nm5WuH4ev/+0gGPPb/X2frX5k/d9/Gb/SBsflIBIx7kLYySL65BKFCmW9rfL74ODQdH1fjIiIpLFwTSZwzWRwjEVOE5WBOG7qZhxdHNqpEEhJNJoXm5KMcQsth/vJi8CSCWKCs2pibgA324IiSRQ5xWKjIh0Jh97FLGbtjt5Ho0VQ0Skh/ktq/GIWXDd2jNYQqTBTM//IIjUs15CpMHydUkZEWkxH7fn/WI3bYV0SRyRfhu2hQLHdtPJMNu6S0rEf5ipAxSCSO/0SIhUwi7JMZEeD4cX4DZdASRsGS9ihxhLpO2fFk+kgUTvYkukBWLXL6QSRgGCSO82ZEQqX5fkmEiHH4YX4DZdHBMCVUOAIroklkh/6CueSETW+ywNiUl35pBKGAUSIv1mp5RIqfeEOIdjIsXvPHCbLg4pV4pYbWCJtG6kVESastwG3yDs9CCVMAokRHqnISfSIdzqoThkRaSxSbcJY3gm9zNSwDi6hER6RvqysxzbNQTXzhET6ZPPFEQqnRF4VkTKQ9vQEBdzEJzTOSQE/IZA4oOoLSAHo+aEI6Tm3QGRLt5MI40Ccfm9H/yVEelgybokx0TKa8EhgGKB+1n+joxbDYHC59Hi/NTJ3GUJQfUg/XYeIsk0FrP0lxWV3+iu9qeESGWzuHNLpNvJngZwmjCGCZVH+D25++VqTtHwrcbabQHyFiUCRSS7Ha5LQH4DPJITNMMC/H9RAUqIdKBcXZJbIg08jy/BacIYlBZXD49kLgODJoO9rf/tkMImSXS3bDW2u0q9scBeFjXaxfawERWgLG65uiS3RCLqaSzr8+Tn1SO3oo0bm4knuaKINHGF+mpFJKAsgMFaFm3se66OU7YuKTMiZa5tALUu4VoOTt9luJisvxRFJCbfQzYWPnQa6ElgbqFJ96LfkxQyI1Lmby6dnr3gou5PXuUlIdLTwdRpzGRuxTikaeJ7sDxHkDgmEmlKkjWRtA6fO5X1+FIOogVma+ypJUIAm7EdkwZYiqIN7ddfmSzunBLpHLnGAC5T5qFX2sV2SUQLnCzGRRhndm5DJJB+dY7xK7oxD5SoS3JKJMoA6mymvcG83nLGQKF+wMlXORQiwX52InnkIRpPBvYdANbS6MHg5VeiLskpkajXXaYrd15Db2lxEbIUQoXcTGu0JAjw3Pz4pgMMGTP+JXcNdlfvL0+XlB2Rsn1z6b620h7BkAolJJLF2I5N43y2E0+T0fhSeYzAK0qki1c0Iz4pwHlDhEXy7V8IkRb4NYj0RMr2UD2zA6PL0yW5JNJj2qEoOEyahf6qdoHr3zfJZaxCiIT4Sj5lbJbPSQ52wuihz8j/U3m6JJdEGqFfD+eyGwI09Vd77+Tl0IjHEKnyZmcauQBh75KxKW++RDJ84QyUpUtySSRm1LCY3bHEvQZWzMV1SVSbuFGElQXWKk3HdjOclwawkkUPhwy1iktlsbjLkEgZFrjJhPT8tczEUIBqxuY9gWsJQpgSaYjzZXsow87VeOGvvySHkVWSSBeumsQu7J3VkHzLBTcvIjfHjQoPk3smu81V56+bPrFUEos7l0QCxXdnMCs7XdMt5yicSPvQDVlglggit2EKBrBgBZSjS3JIpJvnmBvnMjqmaM7QsLwozQ5Q3wogEp6l4dgOiQ7moujBRjO0VA7zBodE6md3kWSlbdDbrpLARDXhEkB9q4mkhtUwbaAUZ507JBJf4uAucRKmkx70KNLswTgl3W9I//SYG0Fv35J5A+TwBEkkq0Nzmlar5+XokjIlUjYvYXM1XDHqhpNT1NeruR1PHUG0Bc+IB0cRn6ZnM/K11mN3GE9/GbqkCvZI5rSYzvxUewzMayTDZTU9AWIYEQlLZMHIjEcbtmYKi2Uwb3BHJMSu+HwW2oY5i/6/kC6JbYK5T5JEGYKJIS+aCBjLooNBW/0bfmxjvnBHJGT/USYHJZoZY7VxpQxmBXkTaUnUBd432SyeI5GsV4TK0CW5IxI2YABnqSewKu4iuqSiiXRGaOtoMrbLj0jn7A96hoI28hOoHJHGbqjj8BjNwRU5i6KJJM7PgEjz6HToYBZGQilMFBaLOHybRrZEyuD3WfYtBRiSsA3ZaGqSQf4JBvR5gB+9eP+UuTgq3EljeNQovEvKlkjutQ1PbPxJtdCfe0nPs0qRSzlbz4qJ9OSwMIiF4FUIhrJoINWrbrHwc+OdEekRVjnutQ177ZYavIXcS5rjDbJdNUvcOyMO0x8pCIoNzGTRwFw6lfq+orskZ0QaRnWX4Cr5CNZag715O28Y4CwZ8uWybCRZQiLZviFDLBRt3uCMSHjdgKvkQ4xZn87zZFgdxyn4JpgvkWS5DWl7KcuLSEtpp9N9BXdJGRPJddtJMZDOW91QZiLhujgEC4K1KFwHkQIH026GKLpLyphIF9ye7/ZoSB1HhJuSOUMWKJhITalWRvf9Py7YaPfctZFQ+oW+grskZ0TCW4lwdYFj9kAAACAASURBVN0OvWkmOjl3SXyBQJ4Hn+Fz1gi6RMK3BnrOx3Zn7RdjIywUa97gikgi3x7gKP02Ur23Uqyc24An0vksj19VZ08Cs+nGICQcmMiihgvLE8WRwhnDFZFAoHQBR+kHOJOuJebaJSHOTriVpSwhJ9KCpiuWnIh004XPtIVCzRtcEUn0I5zOC1K+t0xdPaXC5Fn+Xo6TJPrcVx6ajU4Y7YDTXcduTCH3FdklZU0kl6fJP0qrwc5ztj+CaJhzzP+CYhyrRySxDzHZcq8xHPmVLrRLyppILrUNqddUTZ09pAFGmhyJpMpqTMt46/q4MAgMhFHB1UkH+4rwZhvCEZGeC22BwE0GPlKvFDRz3KRaciItackyIDY3AANhFNCTRQP524ElcESkMeE2BXCTQQsn0i8Cfu1ADE0Acq9ERNIb20lSAW1RlDjg7DS43uK6JEdEEuvw3TUeB0q3+5wf68wAyD3n9gBCTCvVYFrdez5EcrfrcgGcJWUKR0QSv98uWm3EQzCrb/svRn47ZbE2+DSFYYYZOB+DHK5OaCQjIdKpewbiSHHK4fpecV1S5kRypm3Y48IuYDw35w1oG8xtbKeRkc7YTpLMM2eLYi5fbs8KM2/InEjOxgBuiiivLunREexuxYh0R6bjBl1ZFHBrA9lTVJfkhkizaLNpw1HjOepmfjHqbEQix5m72N28iPRMYw+h0IwuwUHZCidoSyOH21fbs6LWktwQSba/xVHjcVXeOVnbN9BmmheRRjUO4LgzpowiFRd0hZHD1neACHsL6pLcEEn2Ghh3Mpd84ELV4KMh8OPrGE781zvOnYFaGGkyjnyZu965/KygtaTsieQGPXluQXAAtjrv7emNsTtzmzBHjSmHNrkI2eeRC6pCpMK9xBiC88VFXD88mnXujo7ZFBZ6V3y1sTtdFgPOFmMLhhsigTho3Uov2PuysXvDi/YZHEHn7imQtgGogBFp2zvel595Io2eQ1zRWSMK8PN/lgSKCf+PP/yb17t+dHrzh9MzKcvRparBL9YPvxe3t+9/6jBtJZwQ6RvsrFJv7ervbnh1w/WXvL1rPvBL/G4KIjkq71deX73qZ6ufO2gAKmBEmn3Te3dr6/PRoWaMTDJXrxS88vra1ZdaxfDqa5IN2uJNtl1f/NMLL798ettHM5tXpHJgf9ql48OuH6z+25f3xO1td3c6x0RmcEKkBirx2jUr1v94/bffnv985ya/xP/1RWuFyoyjc1k+fmHVe2+snk/fAFR4zuqfofV/y1tnt25oza1n903FyCR39Tj44xf+Zc2lVjF0r5JEkg/Yv1zlQHinuu+uri/+8offitvbv/9DnodUOCFS5lMkJ1YNeYIbXN38OlE2bD8zEaEoImkh+5nvDbUKvirInEi3zztYcnC2+BPtVANXCQqAbSBtTk2MMMiGSLNa/XdDab4lJZJPVtAWSQDHdia0wPn658qcSCem078hj9xPmUCMaKcauEpQAOwX+0RqUMiISAe0htDgqYohcyI9duu0c5FWVua5jdMNkRZOSwL9Jf6hlC7H3J1cGfloyVqbjh3llRuR9M4RA0/BhCbidYJ+PO2a7BG3I/YR2ifHXG629j5SE2ljd1vduG6lh+obofX/ypW0ubhC9JJ1Y25hhhyIZLK+0/BSvU/Ac7hFxg1YZWWuY7sURPI1ySvXDHe31Y03XvJQfSN41vtwqHWBtf/o4vUVtZxMDuX05EsZORDJZH0HPK31P9FqjP/okxFjCRk4Xc4D5ntqN8gmSEGkrvfeWP3iT36x4kmgbvzO2396E9M3NuI/5hmQ6wL/8j0XRlmxIOAgMQSypYw8eiSD9R3wBKVAr/+JVmMg/mODaB3L5XLeCXbRftGRgYcWUg7tupQxIP5jnji5LtD1AwfGJIm3NzuJlJAtZeQ0R9Jd3xFqC+j1P/TtGFk9oM/rIFrHcrmcx+tG8hzbuTuxTwCI/5QA1y5FV5B/5hyRslJ/6wG8NHOkC7eiNEoDnkgnczw5OD2RRlt9qMSnCPh/bCssGYMvOtHcJY6VR7PwuvqYO1p1kZhFtIjEwT2R/FmarD4iBGtIA+LhciudJYlxbVthB2ayEQh3xbjTx3oXEf1Rjrv80hMJPClRGlEcGxAF4aSbTuScy+Lcsa95fcihZAmsOYXAuQy6c9L7/tnHY2JbYJAnA8RfG+wjk3ECjDQ5ju3cEAmEoW1TAnG4FOTK+34XOx+JlgEOkmPwFFMFJk7Emhhcy9A+uEiDSME52VPi08nBk5ZRI45kh3yIdA21ps4E6YkkX+IOXny2QztSDyM/OEsTRGGDg+QY7MH0Wy6dTWngcnAEtEZ5H/RfTGLf3irLBSD+2iB88sisdQoMrl7C7ubXJWVNpLPBfM/yiC3qJePAixB5RAOIIllD4J0wR++uXuRkWINIDeKvKBzEj7eDsJMCtHDrXPvTnac/vHn05mbtnPXQrn229ym7uQBFJAdlcvVycu3I5QCBBj74vJTf8MKLh1zqNTcg/orCxaFh0DdiJ/tyxA4sndlq4USaznwzcoSsidQOuWvlu4w+iPy+ngmZDGT7uuvagn9J5FI51y6p3TAxDRYDIP6KwsWhYT7ctitdxK9IV0SaFFRnbmM7N0TqF/YWEPyVjMUlGKCn4unHdlQPB6mTo7FfZJByy8VxdLoIG7j6rGSVtsAP2S/eYtpIYtkgrgnbBFiIGke/0wPRJEhNpMf+igDuDtFHmgJn9DDpXy6ZEkks3leOc5KhQX1IAPJ4Td8jsfhM+vlBMhVzxDUx7MiyQbRi9NS2zzRFaiIFaiLx4QdAfZiBKZwrtgNyPEFImxqNYfE08EGO9vyGRAJR8HVfCSb20nJtgkzFGIkTrsduHMHMCMf9eY3tUhNpMHC4KBy6QfvDZih84zxzI+3Yjh5gOt6StEcS5toJohhR009NpCOPpcFRRyKMIMflq4wcaSH2vzyk4XTWBVITCYi/CFIM7bjJe1oikUq7cEXSGU7Kzsp9mpsZ8smwX9QlUr+I4oopVJS+MIIchENnyxQYSE5DyansUxNJUeLgycMl4HhzPKUDfFopvOh0S5KsQ2oVUl7nbUcNCnc9TgKCv8LJbbZEcr0w/kTi0DqnsV3WPZK9sxH+gPullDaIzONc+ilwEV1Xj7GYt9Pvs7cVEefa1rT3RF4CgPgrDHax0n7QiemXZFHwXD7beDMmkv6YnQPS/6Qc2zFCQLrUKMg7JM87MO0wMwmin6g0qAq1BUI/QkD8FQa7WGl/4GJxQNow8umSMibSpevtz0PmYxvkJX4wna40OyJdVy4453WaDHchwGho5AaCcJCGxgESq1cJqOObhHnooyldNqsIkeSD6cHwEJW2LaURECI9SzWtYaedI+629Ks6JM87mc8AQ5tIDeYTDwdlPlZTeepQamEe+hiQHtVzU+oNyRUcEUk00oXw03z7zxPsiVRjO9b7qWyGaoY7Ghvp8jEUitu1ikigiNe+Pyzyo5bO9wVVjQ6WIRTNIpcuydHQTtTj2Bf4fsy2A3cyrokGq+qFFIlRUHdInjeeh+3qZPzy1SUSSMOvC3QoiwN0PENQg4301sMLivO++/JYxktLpPC1LjKHB+5CF6ieS3ZWrRJc24IUiZF4qOWVKo8uaSieRKrUhMB8MghNgEQ2e3diwgqel4MSTqg41MawYjPHw6yP0fGRlkhRVyR4AwJ3oQu8JaQZ22VGpN1aOuCbOdiuJj/xlGLNDZhPBlFXJKjWE7ESUvC8HHTdWiUhTg5BHmO7tESKtAkqIpmOhG9cQG+LtirogCtviesPEzxVm1oH2OUkNymSUkanmASAe4LCkUfS4GS9F/AIUjCy2SRBQm27MJDDenhaIkH4KSpxNp4uBF3PzXOG6STgy/sWa8xnhx5NPs44U24I0UAvMUD4KbARUij1ktuAR5CCOXPdJgkSR5VrdOm0vXpwRSSQB7siUopeeuIKd8uJGZb+MdpfZz7pNSfSGXwICMynINjzDlioChjRhPZ+mtCwGslhbJcXkQyHdsLDru0nSZzSztEkqU/bxuWpQruUHuZEEkz1gfkUBNudRsC0/Jv4KF4boIzhjc6ky0IDaYkkt14Mtoe1w82Mso6JTkQ6Y22zjbQssE2LwJIBtfsyHquTSjY5kRYi2wLBBENhlJrcnrPwo892IekGBjrH0Ka10tSAKyLh9ZasQ5wwMzYT/3Dr9TskSbBNi0C/gX3EQsZDjEvEap68nO4oFpwURErnH5CtCYskCGi9yLIf27ka2o2iDspG41Y2aWanISaS9dgOSXLMgW9oI3kOZOEoOUE/sV59TPrmSrZPABoe3T2JjwxSEWma3c1qngQJrQoYv5IqDw24IhKuAUsKfNFofvBYPGAYtnQpiJ3xsZB+Q/+wWU+7N3WGMpC9yyOpW+2BmHKAhkd3FVMo4fMyHGLtjtSbpySYmFDH8XLoklwRCd8mZ/vmGhD7flmwHFEjSjsXYzsd6yACx1T7hFKhIfzCApArLByfQj08xsfUBydYqgMdNWc/2b7BvNRESuaagAUDeqmGrHQs3y3oaw/s0kpwwlT3kamhkA2R0Gjzh/iIBMhd+mgEKfjKTaNt0CTSHSvPigZISaSJ2IsFYMHETSMlgax0GnaeytAkwSopAoYdkudd0huJ2MFZj5R4UEWDSX93aAQp+JpIYQDOecgRIeuxXUoiDcU7QQALthzafSOzlH5i59wKrayhlF7VLlxWx2GQZZfkjEiJkggNJpNGI0jBC5aCSNp67b40Gwc0kJJIgFyhwUYFLlfE2L1b0MqSz8jVMO6QWq/Qk+mylIB2Q6dJJNSuQGECRN409qOPWHnlQaTHWRyIRSAlkfRLHA0XQEEkGyWP4GBesEgqwRUby7+dqbKU4STlQEF6YgrEV6cxGyEDIt0w3WfVzxsVXbTePjyl/1pycJqJDGXskRSnUd+x8RNAuCQkARZJJbDokDxvOrPdMfTbWex81CN/N3q+ASBXaLDFEgLS/QhdsCjRp/9SPSja7esGzoiEDREWCJ24gduloyL7oBA27xbBWgVYJBVjUmOHOYLdWdmuNqRfKUB8hTbiJBjdv0qlDEgEGbDBmGkaMQyawvNsLR2dEWnsDh9KmjMYWDeqxr02kyTRfqk0jdqqQ/K8uaz8uhsQiejzAQlO7qGjrtIQyWjHdLZ6O2dzJGwkQXo9e6y/G0dFJBt7D0Grsj4pq4UHtvuLevQ7ZyPoE4lcUcWiERsEsR8Jwi9qoON2wzRiGPnwSOuoVw5nRMIKg3rfI+E41JSzGNuJyJliLXCXrWXLQka2yPpEitwNiqLJt2M0qQUIUAtGZY1NVs9YGj2aFWSmJuDOhnbKbQrAh+OQ2AeFMCeSUH8BxklFeGI/Qtvv7AhiCsxvlBBpiJh4AxJO3ENSuUotKmPPS4CeG2Fp9Gh4Am2mert0RCJ3o6iIpL1YoH5xHDF2AHxJ5KASTFOKkeasljSuJ4S4zRjYS1abQXAd4tmh5BqptiFK/4U8LwPeDAwTCdFvNka+nKVVSToiTRD9tLMeSU2kReMVPKGBMZimFOFZGgOxYw72b3BgiXNX7OdKYZpAzhyV+yENt2y6JJLpWC1LdUM6IpHvJqSErLQ7OvN/405ayDx0PVIH6TQGWRgKGbgbI0OQoiE3lymJZOhHH2/8VrYNxj5NsjQBT0ckEFxjt5BwFDqvDbmzZwTCd5fGscUoUh7Uci6DUQbXGMWtE4hrZAFQ0WHRt+4b2Tw9w2dDxxVLhyiMz1QwE9UM+RFJ96Wj09vIjx9AIG72YJhSiIGU6+QZdEkGRCJDTvM9Cgiu8VtGY9zTyHKjR8/KtGGuPMhQ3eCOSMjJLfrWyAn0ttIalogkUTBLKUJadYH6IBhjWPZIiI0QGYwYpTLpAhdBAlHFGSXShtDTlBj7UvsZFyIdkcgineAXCKh31TG9U3FHtTp58eG7KIRKO1siDab27+S+S7IkEvKSIR9Epqwg/SqHSCajRNo4bO4A46nq/DV7uOuR+EPiZihv17f0TKX15h4zZmZuEt4Nmk63AlhaBxG459rvKj/7EJ/uptACkXee85MakH6VwyGRbGap2ent3BGJLwv63PAFPR2LZvGYje0kiT60sTw13mGO4Os0Lj8Q8L2u+NUFwi8awWwXxj8vgagmpJs+cBjlG2LQ3eFyDLIkEnPEEReO4bGmGaIZkfQMofWRvkPyvDmb+bUE/bzdmUgNME8NHoALB2kw61mNf14MoW//p8b77k6KDmSXQbFDJwXczZH4AgXpVxz9mh4ZrhrtJ3NMpLP45iZD7HFru6paxyNAb8ZTbP0GNvQ4Mz/kIkhwRDivMUklgN36Q2Zju1REog+0BDaYuaGlttPfg68b0VNoAsEgoRAuOqTW7MMiZwkMiERTgYv1jFpcADaYvXHAwBuNnv5DC3ZEOpWVP7RURKIdkgEbbKMmzYRI49clgYYmLi1MuDkOxut3artqQCT6JwMbTHdYXDB7w8SPvjsiXZTVqQRZmYCnItKw3HoRpF9RXBUcWsrjhIG5mtSV5zVjT0BuOiTPW3I6zjAgEn0f2OCjs9Jg9oaJH30xkYYMV7htCZHVmmwqIgH1TWXrpTO0M1jnNDDPkkc1nX/eljlDMMJRfJnfDsiPRA+09tiK4WyEGJ6xbyEDmz4O4ub/yNCVhS0hrmV0KHYqItFFepSdSIL0KwqTM1L0o8qJBPoJBXDVIXlOV2WxXdfXBfa/QH3jrE4bJsFccjJw/vNlyUpx9YJRdAIZqRsc9ki32WULpmg0JqUmjr0P6TtUkw8DQD9PH/cdrqSeu+IsqZOIEbbodFW6Yu6MMcFAfZtinTaBp7ohxCHJ6rcZkexHaHsdr9+FcEKkZzt6Y/Rc54L1Yb5jTwcLTv3HfOUyMXcwmTNA+Kmot+dk8FUuOGMh80rJEZwQKXA1s+tX3saNxE3vCT0J3djdmpc+3di94UVxggbvmXUrvel2sn/6a0VXd1GkVvJHQx9+b2VbLl8+DTx2tY5KZf7zf06bHPZC7/L/IL8Lws8LN1t/tr3jffmZR/Ijugq2lLeChz+lgulurqv9oVd64l6n9XxYnd//VCOhSbYX1YaRvAZwMkcKKmTu917ffywSJR5bsa5e9bPVK9cMd3tvbpyb6b4rIZLWe2bt6u9ueHXD9Ze8vWs++HC6leC7KjFFnXnXD1b/7ct7XmrLNaNXtM5OVKYyf/W1tO4L2Tb6yutrN/zY2+wXkDBuYMM1+6b37laPZEoUHKxutIK9T8jgB6RNVSuPrh+dxnPRENJH3DrC6tzdreEayLZDMpTXAO6I9NqOs9t/1/rsaYbYPxMaTXa998bqF3/yixVPTv3m/uYV//qicEVfz03Z2jUr1v94/bffnv9856aPZjav+PCoppgcurq++Msffuvt+UCuT1bo7BN75myYSGW+dlXa5Ngf+fELq9e/4m3zC4j7XRB+BkTa8tbZrRsWiXpr7g3rLSBSK/jYT8ngk9cIW9ifrn/lRy+fxnPRENJH3DrC6vz3f9BIyZZIhvIawN3QLsL2qRDbp4gS79JKz+7EFhVcjqf7stvRkg4Gc/XY0CN4AUaI621qe1hvlLegOHjnHdqo/MtV2nvNhWewaLUOAobG/xQM5DVBGiLFhobkrHTP4ESIz6dMD/HOaAbpMNmFjF2xW2MJ9OPGerjn24l6G4rqbXx3WG8LZPAh+2qNgdjV2sFwO1oeSEMkykJoamJkpEFgZOQzusSB9kuIwajFN7AtUBgUSruGZ/A6B4ee2MlMUy9uML646JJ8TPt/O0fam/H1NniYqLcmF8wQqUF9yCGrX/C8m9prQ/avxlBObs0zNdIQaZgcZrYqpEEBKXHF6vVVI2MdbQIoDg3x0wDdTB2u5p0n3yqpHd3RvrjY3Qq02AfJETRfbzuu0ERigj+zJ5IsUgPdQ4jjiYFREi5C0/lpSXZEerrP7+5/H3T64TZYskK+ePedd955///b/FVvb1/89gNP1WDNTG+0CXBcPg4wIdIZ01O1JKAGiTfPiqIp8bjPr4XfBXURuV9oMDO5kdDc+2lfUm+hRx2y3np+Gddbb1gZJJH+H1Gt8c5GEyKpDpoBnURaOGE/RIzk1M1KG3ZEIs/6OtR+F1IVQkaNLsBzKn4wrHORnlGP5BD0bMu+qyOfjOqFnckt7aPDfRxoj1LJenuXUO2PtF1sNAXVGl3fC+dbqeeh4OVSDSUjEnjBAuw6vy8+0bZGICskDKbX+cQN1mppMrAjw9MjEa3brlvZukKhM0QkJFy30s1a3lN6YdeeSOAldRGekswfQ9CbxN32jv8v3pRE1tu6dlIj/9Yi1Nm2dX2Tr1Zq2T1yeTGs7qujqsAX0PXfZ0mVakWnEJ/wp5eVAeyJNPd77/SW1ueJ28HywuT4MFXireD2Ol+0/tDTbAoarNXS5Kg/XZQRgF63vfHSjZfQaIEjdhWRCAlvvORmLe8gbXZ2znJ7DV0X+5+0F4L8DyrSNxfiuLNvBous3vFJrt7WtZOa39b6PHs9CJ7lq7W97B6vObVzki4B0lWBL6D7iauqga1SRXQEB6NiQQ9QSwN7Ir224+yWX7XEOXEpWF64OjZIlXgrOFjng3D54d7O1iQV0NSsliYb8R8B6HXb77z9V2+jC8HX/R1QA4pBNyHhd97+wslaHjv6starg5fUxf47QVlvaf1ndNS9cdwtb53d8larYo5f5uptXTupoSutz7PnguBJvlqDZfeoWrdFOcnGdnRVfIYuoIOn7ibYKjXfqx9Ledd+UorDjkgHxy7H2HIhWF4YO3YgLvJtSeiljeHyw7GvhESyQsAhB+kFfq7PZbUBWQyWONZjOziflPaXl/2iPr19YoIjUvAGPnA2ibv5Ildv759JgredbS8q4dV6eWdYrZ9FOVl5704QeC2AdGloIKG767ws1d/fXD53tH9LgMZIgMEDB46Nh6V7ob+/sXfnnzZtbgUPtYNHvhyWE+lIq6ZNdJJBxQ2rh4GtNIdl5nGBSA9VVkatyUD7w+xEHjG+YddM7A3fryV10R8U9a9bZT7BEGmuLb+83iYGk3rb1w4eJoIvItX6gTaR7lyWVXCwTxnUP/ZOuERiqb4uHZFas1BuAWJwbDK2MpmavDpGh/7hwpRojhTAVHkWxNXYJt6I582SdDRy3hcn5wRcB5TGKz9TF79sNDgixRmq623QILjnDxMaQ7sAvnIDhKFHVVPeEAPhS9GyHpLHxKLYwSGRhsdnY7vH5uzkOB0avLrEvj+N1Z9BXA3f68DYK6PpaNSLYyLxU6IUa710Xez4PUak0XtYXKzehuXBVOBHmyIiHVdZsPlb1kEc6v/RsDeIUrCrh+nE969YFDs4JBJVea1gOvS3folfEjucAU8952fj65SnH0+271Y3nTDGZbEXcRMgZkv7rXwnB6Dr4t19GJEi6qrrzaRa321ERGqq3mngKYl0U+1PIUphwErrdiiZCkiHKRbIl0gSGyHwWB/HckDwV02Afk/+9gHirwSPRtufS262UYzwJwo0TY/NIh6licTWRRu9WNy0RPplTCTl2A48JZE0DquKUrjPboDXAtFcLI+tFSI3IrVnpSBMzw/R9VccxdcgwOyoIpZmOvGCozKmFjBlt729HVUX2/+AE+nmGT6uSyKp3ml+uHj1pqGVBrHUbDW2Ix8CmwTEyLdHUjkIBP38+5OnZAi8F8pigV468ftWGVMLGJH6rC3IqLr45T6cSGGXVCiRxPbDoJWGNxI7wV9GRKKDf6vRIxn8uFBhrXzgkL9op7A6VsQI4JZI97AzGB9Y2zTTROLqIkR722RqIlHB60a0iQSezGwbiL+KRPSyQ0EOP5V5mSEVkWhMsCVOQTG0axexMJhD6CdXWZygitXOUmkxEleBk7VbfGOa9diOLOy9n3F1EWL+IBtXp95GOCIR2PN5QqTz8t0qippoJHGUiQRAThJUgnKrN2Sv3MGQhkgsppqS4M/lRGoTQxjMYVBzGRAUsR60ezZq0zWGOIlFe51AAtweKA2RYnxymauLCL1sXJ16kwb/+VwSrHJ6Bv4fYU0A8VeVSICnFkvjg+T6/ax6Ed4E9kRqTnGgSpzB9inpHKntPRK0sw9jKt9LQYYSv4Shqy6Vw9x7p9mc02AJL4abtj6myLrYwtdFhBN3PXW9GVXrVjJYrrZrnzOnIJLqvUi+xSzGdgqf56mQ7nwkfYSjYxCFa05VmPjYCac0ggzpUzMoREvEIE/mQGIfqYipg5MCn98O9t8iRysncO4Tg0pQXnltP7yiOOFxjqDIj1wfWaZECj3SgCjckkjKlYcgw3mxb4cG8ykA0WIc2DaILL0dEElqRL43ffo0GsIvHE4E259AEBq6TRYFY1ksUyKNto8KOYiMOAJA8Pe4tuWmJgGUY29zIp28p8hSDVFrl57jpAdpp3PntCzUAlSpcadaUGgfagGC0LF2H62yESIfP2NumsAQyakroryIFP4G4aFUEPy9p13TukRSaYOA+RSAaJ/aLjqE4DewRnDtTMh5+jTmqfHANekwW14Rodb0hsJGiHxcegwjCsYbHuW2LzXyIhK0P2ZHpeGi4xN4uOqRgPnEQXmNSz22E/vBTN3QFYq/fmuXdCjGqE5B7lqvHShSDoWPPlOM1KkcjOthiNmQ6fRg5pyJJBQe6GhKaDqxCE3kxLGA+cRxjTxIUJGlGuJ5zGja0zgU6gR3Dpex3KSZQ/BX1OUA84njLqXWNCYS+4A8M0PkTSSQh4uCOWgSKRxKimNFIfJaAXL4r8hSDXGDM/GXiuGW6nRbt2O7hvQrDQj+ikxFgfnEQS+bG89xOopIgsKObEcAD+YQLygoHgiVG+JYUYj8FDSq6R/lLbeNcPGqOCxlQ1fqt88a2NerYUKkMBDwUKBjCUA/rLHpggZbOgJZ7FASIkX6KsCDOdyJlksUBTxLlwAAIABJREFUTUfpvjDKUH40N5XLfMrt5jIN9YF0divqhaIsz38mFtuEcQEPBTqWACD9qgRbOsdmDBOQoSRDu6hDEARzOH03vDgyK40XpndM1IksRD2b3AsuXQUgzVIJGZGepVIJXleN7LIl0j3ZHiGgPgShiqJlQg0nSZxjDj3X8ZooCZFMHWDGe2kVjtfD9CZFWuHk/FRZzowZmSyqGvNSzVSqhq5huTB1Qh1HF9yxeZL8o8kR4MHgSYPbmGM8nhgSifdhKc3NECUhUnRbwy0QnYxio2MYT7jmcDpeXZXVynnapBXwWJo4KP2J+9KcfKJjAuSwS+JW1yX5Ryd1Ax4MnjS4jVFmufaAmTKfr2FpbobIiUhxexe0Vwg/lbZzTHxVYah6umQ1XkYkpoHYWPAnkHuC1Du0EAelpBdhwN3BNFyRSYgUOrZONUdinzW03+4IIt2JxlYH8QkphJ+6O+khvtKbnoIgXM89E9NAUpwq4ildqqboMbRsUp+7W4bkil5SF5GuGtdZx5QYlU15wSA/BHxsl36LcyLSmUg5MI5v/wLuQo4kWgFESmXbcOucPDwFkfS8Hrsb23HFkGwE5wDh5ynUa1fsm+mGbERiQlydx70pmam8IXIiUqwcENgIAXchRxKtakRS7d6zP/j+st45bfbu+llwxaBwEuXjPrbJPjmyTmYjxDtwMqqHh0jLA/6WLXIiEiBXaDgebJxeiLhmRbEayBUHzmVbGiIpuw3rHkPXDb+rLmme92QnHlzGRYaOLIGPx4Nf6jtqshA0jHSXwN+yRe5EQktqboSPqJme9IH40FpR/ST3hRs8ECeikh23KiwopyjW7Vx3214vOk0xxxl+e6IGkdCaAD6eJIkYGo52ZY9Xm0iABSenNKPBsvSkvUPsIll0nnbytMSFKpeFhs99EY4o7YtsNyVd1PUB+8CRswKk5MVEAslTmu9FJMhkaIDJ5mCTZoRyECkxLtH7acSatNTyLc5M5Psnkeap+HwDl3pTjfGXZZekf8CSo7EdUlVqU6xyEem8u+N8ykGk5KaeRvJ6otyRbmWL08Vmmow0gMfwsCoQx1WhDEQ65OKgNLQVnxB60gfJU/ZEMnFCgcVddOe3uBxzpCT4otR4NAKxxi3dKKlSyjUJ+1Px280hkaY1Tm6325R0QeFWjsCim6UkpMCEnvRjm0b7OdIUsntar7UEwDWKgN20Qtl6JNaeCgdZ3GiCetm2z71E0qSAGBvIN11IsE/q16ANu01JJi9nN2M7MJAiMXbEntLqkbAy199R7R1Gl3rF2ZmibETSG/a6IhK5kC5MBzmwe9rqLARPc/xl1c5NpgvjLrwVoGNqEZFOxr0s+g6C+EpjlkXC2O2UTqJWyIlIyZsDtREC9FIMXbdMgFwJ8hVn7FJvqvXcAQtzuDMqV7EUXHRJ6O4jEZESy6BLmFIS4iuxjRBgN9MSadjNdNHLi0gPE/8t6IEEgF6KoUsk1RwJBNfCzJSR5TittUH1mYVXZLND0XscLCWh5a5er8PG7sTSrthqGbCbomUNmQQkHomc8RgjHyIRryF0DQbQSzGy6JGE6WBvWTw5JTSbu4UTcDCK/VDoEEwfaHmJVB6AXkYgNpUh5hJt4LvwpI5lKSi2HaRGPkQaIgYryPyQfEuBTnqacyTi4DI8Fnl3UGSugBHJ5JBOAppEMt+UdNrQW6KDsR3aMkWe9AG9jHBiWhocQLAvXHds90jw6hBlZ4x8iASC6xCXrkmDeWgSiegIcZqQ6YgchKNOK4VbbqV4rHmo/axxh2Hq1XtIvkNfB3gTFggC6GWE/iVpsOy+LpEOC3y4irIzRv5EQn46qYDQcrJEpnFCvOxCHKKOu9Uk0xE5CMddelgZlwzoas2Nx3am4qT1+yV0eCAgEiEfkjN5S/RLkMdk0VmI3jT9DmaLAUpBJDIY37AieeCOeJhMZIXbN1KyABZDVFVWRNLWCJgS6bjgeAsxUo/tTuEu0PMm0pCmhlNEpOt6e0/UKMXQjrw1reGdg1JYSBblFNkytSaoQocLENpEmjTccWams/NxJW0DEo3h8OkdoJfYLSTYh+g0ed3dysKxryvfs6Ujko5HZnpnCpKgXrbMTQGR8CqwOcpgQr/xGnYYFjY/abskQWldR9XXC4RCH/hgEFwTmBB51dQcGgijCfIzRvmIhISzGKT2DokfUGTLbCjA0xHMJq7rOBphYNBvmDXzY/cNJfFzSOOuyDMcgpG6mcP8BjsQXJO3RS8uPSKpToFOj9LNkXR+GUi+kVAp9y5q9GxXBYyx6AQMiDR+JaOEY+hqEEUQNWC0Az9JMB3xRAjqdAG/rUukUeFmG2HChsifSIixFQi/qJNL0yMd0ujZQGBmKs5VBNWZJRRMuiQ7FVzKsZ0oT5RIZH+ywNttkGkdwR2oiLITKT0YiOkmUfoaIX8iXeStwshgnVcMFV/TzhEU4abqOSw9OQ6ZuMMzaeZHrNrCSCrP1zdF3pFxIgm/cHe+wUcA/EMhEF4iELcRybGoRsiFSJSwvI0QbX5VGJFwTaraekwXRgOwUZ1VgBB2Zywv2QwIY+wXLYmh7k9B+CWQhNSc4TZCM2JdrvF2AQacOHbIhUj08VLABtPuEoyJJJyHUkdMHkdewHQ6uKNSUStFrZilMGq4S/pEFWmGVUg1thOKh3rSly8z0C660JQPoavicklISN41oPO8GrkQiW7EwAbTDnwQrQ4LOgXhEi5FUGzdlk4HXbMVjhyMm+9tM4d1+s38sOV5Td/ws359iJuv0liei3CesokHNlh4s419GgsRTySaFUnSJsiFSHSfAVww9U04+k5AV8WUaJcdZV+HrduC9GuAc8J9PqZjuz6zlaf92jtlrIdoabokk435KiLRW9SADRbebOOqxuqczAGe9W5nGrkQiS46YIPpG4vq2SOdntC0ASTfsFtIDB1PbZowbO/aJyXZe2BopFhKSkEkYEMVzUN4M4TG75fNIoXvYTPk0yNR37g6AOlXHqytvp4+AEmWuYWlYzaEkWARyV8K3f5iSGDXrMacvcLqqfjUQqX/OM5OFMRxQ0g9QWpUhMYx0WlRRSKx0x3RA9kS6ayZT7RRU+uDPs3+wk5nF8B+bHdSrFQc5ndo0J6e4gMVIgD1DStxsZsvwQMmUUAWqI3yDe3UP+w0UxOiB0DyzQfbs2HpiKtAYwhKwngmI9qKxkDtA1mMUa3FTAySlon4vbpD6Vk4s2T10A67F0OoiRflYJK4NgrokUAerP5h7EZ90QOKbHV6Ntkxo6JscZirBPT6i8E0p55Z92Z6jjIinKJ7Y5b6oEwakHsxJGtMISRedFWJa6MAInFqEqbowFOALWrRAyD55kOjZ5OZg4qyRTFj7nB7r1asVMuq1mM7GZF4djKzImCC6e+YjRD7hL4wAY5JTT8uGDlgEiHXod2tr3tj7IxXPyQTV3lyrVddH5GgMN54Eql3G+ESOTn38koPEYNUmqeYgNBIdTasBM8tPA45gJmmJY69JynmvlEu+O5XSfAualaUctOQ1S5MQ+TaI/mf297x/5FmQ5exrSYbu8XJ0U62ogT5TcNktrt+5W3c6GdGGCRw6Qx/6jEDenEVrFvpTbcFvfcXYlFjmHccSQZ/+mvxLOBQqiNhW3l4c083dv/8n82ek3k49Wtu7mmrUL7/aXQLyItWObcqg+xlgA5dd4B6poVr6qUi6S/QIZJpEbDIg0ix0gZa/2ff9P+RNlWEDfYrr69e9bPVzzd/OD2jTaQoQY5ITw4TseZ+7/X9xyJt7cCl433iaRBp7ervbnh1w/WXvL1rPvAF7f47sajShERgM7j7rjiu7cguzuPNjXMz3a++ZsbHG6hRhF93K9cMd/tptgpld3fUC4NHXLTKuVUZIiK1Qk9voZ7xJEZgfotZu/pSq71If4FwXPGD7//r0Ov/+2fzqgQ0kAeR4iUvaP3f8tbZLW+1WvR8fzNET0Kkj19Y9d4bq+e3fTSzeYV4ZR+oqyjB/qfNGEFgzBk/1ms7zm7/nX9zPInVYNM59tNWOneOE+mgRFqzYv2P13/77fnPd27yBf3pd9QGpmMm43A2gw/F8ysbX5J0Hqd+c3/zirWrzJ7Gvbh0teruxZ/8YsWTVpqvfee/3vyHqArBIy5a5fzajlY590SFPB3ZuYahW37lD797FLUQ4uMX/mXNpVZ76Zb9AiGRut7cubXrD//XUyWggTyIFB9OBMTNp3umQmyfEh+WhyJJhtq5dy9KMEwvthAiY108lcTC07k9FMeYEpx4a45UKgEJDtifHZgG4pbdJY1OPrctKuTjRyMiER3P4o4prKKs8NTRTgkp8iDSwej99YCcTZ6fCPG5PZEuEUqCTyditNOLa+3W3iTWf1+OY00l6dwlZsH/PZ7EGEqjWyaRFZGcKUPMYDh7j6MPEGqf3qiUd1wN6//x7iRw91hSC2mJdNxiJ74x8iASWezNqYmRkQaBkRGGSKCsJyC/+Ak26ARDIoEi1pQynSlZOwXV+gSBOVPFZJjBSUUTEDr41UCf4FoHsgqCFl8WuVsxmlMjTDl/RtQ/0jwmpu4e82QAleGpvDn5Z22BNIYW8iASENdce20gRBqV7qRoUq0SSVCHSA0lkfwYigajXQEDhn1ugH61a+v9Num2QR155phIY/QCHWUD0iKSpP7RWhiQj19bGV6QHk4gJ1LDqwCRbgfrPB8GnXXbkQxVUh++08L7b21ujfj64ndvQ3YkgY+71MQdK/oL++Js911iY508HuDc7Ycn0HSOHT8ex5Cdr2dSAfJXKo6Z4+qDtGzSDdEge40RM8PX5jlZwoR2aQmCFb5Pg5poZ0gSacu7cf33ng5Dw1o4TNSCYqLayvCBtCDkpySCVwEi8cMHqt2jowuQj1g2dtPLKxiRpNkK0o1jUFkFqyy/eAN7oCVnpit9wYI7ZJY8NWpdcJgPEPwnPW432/tCSCKRan2mnhJmRLUgXE0DL1U5+Y9i26cNkS2RwAuWQv/4vhf9Vqrdh8FffuYlRRGMA4BNyEe8TEEtr2BECtMNVva4bKk0k2U4ikjkishM99//J+ZJCERyEvDtNz/8Xriy2vWK0da+YLubLAPb1dQ26JSNxnbRUnGy4somHCUeTJZaNTHyb4tJTSTVtc5j12cpItG1IFxNIzMUQVJGfoXfMdu9jCF7Is393tvoWx5AsCYwOzk+TBGpFfzuVi9ZN7jpK0MBSyxepqCWV0REaqUbrOxx2foPvf6/f/ajVc+pZTiKSOSKyOYVr31vHhFHZ2jX9YPVf/vynnBl9Xf/x+gICIj/cEi3mhrgPO1yArQOk2eWipMVVwKBK0YIvwREatXE/DYvquHZyaT+fSLF67N0PflEomtBtJoWDF8AD9NZr/UflZ7orYfsifTajrMbf+2/kII1gcmrY4MUkVrBWze0greHiwaD51pzTxAlyC9TiIjUSjdY2eOyDZL5w//tem+JWobjhnboigjz24RyRml0ffGXP/xWuLL6K7OeA+I/HNKtpgZg9JGa7iLppeJkxZVAcOwHhF8CIrVqYuiKF9Xw5NWk/n0iReuzTD1FQztVLYSedUAQqLFeC/GfVMiWSL2HR2NsChYFxseOHRgcHmnjiyR0+E/hqsGX41PSpWwWgcKUxsSuJN3RHWy2qnR0M/bnAVnOkcDLMgNWse9soSs4BDYS++repCKgL6yJA3ED+IBoHn+m6wm1QUIReHhMUU6N+E8qZKz+fjB9+9LxvgBDQdkNDx44NhYteY4fPz469NVXe3v7+hoRCz6YkPVInudPoUh3Bi0CcGhOJ9kOs9lebWO0/763H0vnaoQLu+57YscJgaJIImeEi+0TV+a1CRohqFtZBi3pn9paXnBeXw02VBxc9CTlQos9Nz0dVcTACN8ATrbqvz+o/77BKPRAEHolrobN95uHZKbzgT0EiMMPyKT1KtIjedjQa/DqZGL+Qfb0AX6rIBI7NWlO8Whi2Y6R2U5NNa40KXfEfDpHT0xJBAlOxBhWz04azKc2gqwbEv0EePYNgNsOeEv/CEJ5vuAxwXz9kzVBDfX55jE1NdGYaopOpsUzpOEftywO9cJthtIYWiiASMOTs4lBIjn3tCMSBizbcTLbZnNvK9a527J0Bh7IxphD/iqh6LhMApGsdkSSeaYGm1RD8CM5/bGdPF+I/4Tg65+sCUr5xDePZvPEtab8ECQuQxr+C09WSu2d8eLndVEAkUZISwZuoTsY2kl+uR8kPDdZlu0EZ0CRbLQQ5qRwIoSddc8AyPgmCB6QmDY8P2CTaog0RNIol36iq8dsFaSmDHQ9Bcdeg0Icid8G3/uozNKrvRtOloEeSkekoEcSHEngwy833LmwIluESPJ2GDQY4dAK4jhyQPgp9jQpe1DizzXQVlkSaeokd+uM1MqGhJxIECRG2AilJBLEf2QZnhdK70s6Kll3GH4SxUqHchJJcoyX/5Ofq50nOiCSH0N8oi0QfyWIuxRTF8ehgZo4g+Blo8wfB/Z+0O6SQJpvEEKeBJ8LkWaEZkB+8G2JSVM76fSmDaUjUjC0k5y1oamtdEQksRvORpKKDPvjlU5lVBohA8VPgUWiEbCxjiMiBeVCvjVSEklZ3+0woVGiL6lsvbX9+KS+rkWA/Ik0otEjiSuq3RlZEGnEikhiq1Eg/ioSacNw9HDmjiKDIMTmKFtBqtp+vfyHxW9wYDPImkhchkiwMDQKSm/akAuRmPVSmkhMcJtIwpL7JjghRo9Ikmx150iKClILkjwv02QjGFhQZBCkfNrKxeNZbEbxXLMtBe8WsXEaBH8JsZEFc3YDkrKeQCIPlyEN1XIcENHSIA8isWCIROHMdnmPNBKoIbSIJMtWt0cSC6I5tEsi3NJfrCceFGcQCMB5LdUCrsXSHNsF8x/hjG+hPX2F5M4UVxNTNJE06knS8YaSgCgcpKEa5ayJ7ImEr/OIgs9fDIJBlFyD+Jsi2/DcQPpAEQYQ/xGGKsuf3IRo9tID6sNaABQ4kTQPpzhzT5ZvONkgguU1oaonXgvIIJzFiuRREqlBREuDfM6Q1cfh9kgdROG6RFKhrS8NLCwFmBuRCgKeNDjEMWIuYSa0soL1BMAg2O71UM/ZeHuNSJRv2ORFwcZob+SdEW/cCxXfwgxBIQ5QH/YoG5Gg/SEf8noHZD2JFtraNJkefUK6Uhc6BcfPnU1APm7VI4lNG4DLQBcHBULrje3kY1rl1M4QSp1L6FlHlKHScAGoD3uUlEggCm8XGOqc1QgN6gNDmyMiQW60d8NPKAQhHzdjf/ikcFYfTkZsWqxooV+PSED8FYW665FCGwlxeooM20f99mM7MwNEallxBpooKZEUPZLskAg9qInUluSAYMdbqP+dUxh1A3EtdyzAIpRMOKu/eYGMZgQRka5rqUOA+CsKlbRcQwD1IY4gqqe2G9+xO6LHo5VccQaaKCmRhDZCagLoQZdIgZE3FrpExhKBpoGR0MB8sghLyKIg7gnt97S6JCD+ikJl2gEzAPUhjiCyEWoHi99h0VEUxyxPtI5RUiIJbYRcEQmU6bSDRF0O0PIIcPUKn6ImgPlkYa9tEp8KrUWkdsYDAhVfKA9pI5QKyvoOMxTZCIUPCtfVI+elqU0bSkqkpsCPUKRwyo1IopYKdCxRLlSbNRH6ebTzTvRQKMARIz8QAcS20Cd0+pG2QPIeQOlHTBu6RBJRRfW+iRJObdpQUiKJfvmVUF+dA5FAKgh40mA09LCB97ib5xUZhJLfOi8IF2JJ/ZulaMd5JNibC8xnWoTpqIkkyBCkoUSAMIYmKkakaCeSINg4n1Fx2w5jCKoQmE95rDZMVCSx2luUQXjf8CTbFk5LDpHWGNtF075U5aKPMB3xFEaRYXhbSETgLixRMSI15MHaiHwWsMfI8pI4JJJJPxqf9iTKQE8ABDKyHFTvmIxmP/kQaSlMR1xPigzD2wotsPh5bZSMSM8UU4Po9v6UJ0lGTpElY3n56Bw8aXAbDxlXbAZEAu5CEEEULoSMSJxPFB6RPk7wW6LydLQiG+1AVFnhqxrMsuuRbiumBlGBXJS5w9DA4Ui9LizhaBd5mjfvEPOCtyGSaMOClgAI5IcFqcd20QoR/lseRnpmMJFJjNi6XVR01LGMCED+eMcSKd6RCnh4VCCqhVAV4uSFJXzlqjRClADLFTyXEOj2BcWjAtOGhf1sRE0clB5gIVaNR4jyw8vl0nUmWkooR7ixtSQe4fGR9uch0a8G7sISJSNSvCIOeLirIW38uJBIkRXdYXS/W6wtldoIAfN9SV8zED8q0MvejKwQTMdQ8qPJJNahIYD5pBHbHuLBxgDugkG8co9HiIgtdPfU4C4sUTIiQXQhqO/8iBQFBD5GOExFC47SrhHYG/q1lcTkEgkQ+/NwSyT12E6u7onvghsbIWU7jzPcj9oIRT2RyHdBovRMa9pQViLhHYF7IgnTiQIWUAPxxCOeRBD+Uf1WnySLPxPflW6p4nFbsYKvJBKEn7jdexRquXWXgz6RzqFafeAuaCTuzlQFo0JZiXT9MhacvP8BC7bIR7j3ErgLEslZixJBzk5yiSqnIHyyCiKp7M8ZqA6dle3QCgDh5zW0gqJQiUGfEeL0lESaQfcKA3dB40xszZrWtKGsRMIXL8evcxFT5iO0CwbuAg2VCcIHyY8ixJ/Fm1AcLnG4hEF5erOqS4oyxjXlUaix9zFFbqH7OUkERT2hoZTJYMpJUlmJhP+upCNI+bPjfB4dEcSQDyKTm2YGN2hiCIjTm/FHQBEugNoGTkUkB+VigDg9fKqqzFA1FQD00gbVIlJyM92Gl2jBXJSPlxWRdJsXMXBC87clktos9ajE27hHMhHNN7nphEhJbgsChaciw5pI+O9KCuusxGJMjfvJgFo5+EYFAeRKEkmZGwtiP5ZiaGfWANQLrsQ7BkPiMRONB8hVCtxNdmMo1vMsh3aAXtqgqkSaFQ3JtEDMjLIj0h1ED3RA6OydBiEVbtqgIwAC5RRJxbWxuEtD8wXkKgVOxmfdpyWS8vHOIhIxhAcsvIFemoMYGArSSc6ZQDVtgFyxwOwBH8iPqsdSRbecESMdsQA8nmiccLlfqk8fiEsOzReQqxQgyl6QniLD+OZR3Mwf0EsblItIM4ohlysigTKdZCkcXRFJEhAfMQPaN6XRUL0ssQ1JM8UAA1L7oDbkmz3kC8XEs07mSIBeEiDcQGEZPo49jAl2wLpqUGUjEmGLClh4fkRK1K33+SNQyAQuX9PIRHFTHg2TkDioxETtojGyU4ztIL7CxCJ0a4AEG0NZ3wQ/sAyTyhEsExEPHU1n2lAuIhG2hYCFEzddEUngJIuIgWmMk2DxYWWA3dQUuyG4Ru7xy75i6BFJtmoM8RXmcY+YzznxI6QkEjFcwGyEiOECoM8Td2Vnv2igXESC5BKzEXpI7G92RaSJK6oYwIc+PiINDtBE5yOH9XwskIliv5QIn9U/kPnGBZ1Y4qNsqIyxwwT7Ew46sRFKchOYQhELqpiNEKCXCZ4Tq8opTRtKSyTMBOUcsQ3BFZEEe9nk70LSmFgkCL6LnXQGLgEIrrF7+iWh1SHJx3ZJZtjaLiSXkkM79UGkh5tCERGw8wRUPRrl8QKwGNooLZGwOS/pA8rwlBRhPsp1GiwC2V0CHyy7r9fsQXCN3dMnkuZ5E7JoRMZIvkRo+hOH6PRwUygigoLYaClRw1PAYmijtERSLWeeSjN0INO2IRJ5C/hg2X33RBIJwGFRM+a4xA6WSEJOJBfaBoqMaMEpMiRu4Y+Tb2PkeQNUlkjTmisyGKjezoZIILgWxVFmx4Ca92AJ6QjA4dh9dZwA4i6JtEXNnkiT5O7gLIhE3USeN0B5iaToCNL8bmqvgGKbgrLBCJghMn8+o6Nki3ds+8C2nIHgWgrNKZKMSKQiwq5cTEAZdSiJJG8w/Zi7HBB+MUZ1iZSioqhNaYBGIVJHzHrIZwRyiPYJac0dBkkBkbVEyoQTuGABtIk0Ijwhltx1guRL3nJAJPWLU5EhcQvd+AfCL8YoL5GAD3ZFJBB+ifCU0F1T3QP/DPq8J97DpyM3lSZCPUrbpFsQj/UOEvNzFG62INU9wAeD4NoSZkRCIhC3WNdo/CPpmF9dIiHhNtkknrlIXL2SXM/xC0JkAgIbIUDvehZEQp6gVOu63gYGNA1mPcnYDohrxJYQBNeWUL845RlSi+WYPNQ9gTmeJkpFJLIfUL3wnBEJ3axPbcjkcyLvCGyE+IdC6Ojt6Yf5JkTdEZ8wTkN7ZNdKX8Q5IK6jE1ESUH0nsKHmoJJAz78gu04+Q6pqMHmoe7dSmTaUikjUYS5886Gt3sA+HyppdNOoou8j7whGTPxDIa6JTqwRZa/IX/vkBwMiCa3EScH4H07ZRDjwIwTkl/Mcb5lzjwY4GyFqsABsKHvP3I06iVIRiepcRzgbIdrqBOzzoR9VmOAoGzIqyEOxdl6j3YPkG39Hb3Cv7y/CE4/tQPjFBzWhj3192oPKAHthUcfAIb5mhF9CqF5YBigVkagJ+rUJNrifGhSJfPnq5EN9MyfS84PSYB/i3RU67R4k3/g7ekTq0YoVYq/gPgi/+DhIjgjvao44JVC2c8rP7TRnpg/CL208pe21kBj6KBWRqIJ7ylmlAfUtxaFwdELmRKINhblg4U1xftKn+bToO3pE0rQPamNSwAIQfuEESW8jxKSgWg/hh7gg/NIGY8+JxNBHqYgE1Deu4Ohg+0PhlpQVpJgj0aMWpTqJwX619ox+ms8AJN8EWDArLgHt5OVC30i9kMRsbFAsACMRVFpe5qRiJIY+qksk+9/NGCabE4nea2NKJNyVIQmG6bx+GyTfBDgi9w7EAicS/fLi8qVvpCYSo49OSSRksM08AFwEA5SKSIoxMUi/6oOZBmP6aCpt7oh19Rz1OeroGH0cAbMpl9tyxuiXMAE4GOhFZbn6AAAgAElEQVTsfJxBz81R6E0dzt355JQLi4oGg/iQBelXM5SKSCD5xt/gwrWzoZlzGvH1RqUdn/yA54zJcUbkwNWHkkjn6ONfuNkGs+IhUWwkMCQSLuRZ6mcBGwySbxZQE0mRIXVjnj/fm3kgVRdaYiKphnbWv5tJZ5Y3HqEVHZxzQjoBpbKChVJw1rqPfYAZ8+BuuGkYuy9Dx3b0ougw60cfqG+ph3ZMAsjIQZEhSL4hD0gOFFajVERqSL7x9tSuiIQkxAwD5A9gh1ixT1AYwYySZE/L82e08Tj60VNPJBjE9sTTGV9nl5bl9WcMoL8iO9DkGTJH7jDJ8XfMT4gnUCoigeQbbwqTIZGYxWA2Ap0AYtQqJxKv2Fc8bfodgZHy2wfqI5jOiJsIguSbBZgEkB1o8gwZA3yV8lPoFlkLZSJS/ENO9SbYFGvIUqzA0oD4andfnE/fITTGs81h+E4keHp7IuceqtbAS4OkyvftizPYl8xyLFI3JhKKKOOFzckP7z3NBY8RoZv1TWVFualjnCUznI9C4+XaPrQUiXJe3IT+HhOUiUixWhpa/3f9yvvj+x5JH/qNsm6lN93+bF2ZAairbe/4edHVllz7ioZWDDo4kiSYJLRCm11MhNuyQUIk+cbuP/013so4ARkJwEOwsVuS5xWNaRSf3NzTDS963/+UFyxYDOfkekwejNwKGvm3Rcy0VRvokIP8mfFO5zDDoMEklqrg0cGBPJA8nlwHjlNatcg1BH2UiUjxrjFo/Z/7vbfRr6aESBB+rl393Q2vbrj+krd3zQcfTt946cZLhvkAdTX7pp8XzpQ2kVox8AoIiNQK3foZEwG1VGYln+m++66mgIwERFavvL561c9WP9/sJ4cnFsBEZ+cnuHLNcLf35sa5uy96u7v5M4QCInFyxb6VQqnnt3lpiESOtPCfGZsPhhkGDSYhUlyLpDyQPJ5YPQZEatUi1xD0USYixS7+oPX/tR1nN/669QY5PtkM0RNO6teuWbH+x+u//fb85zs3fTTznbf/6u15QYICAHW15a2zr+3wX1XNBEkMn0hb3qKDn0YVEBCp9fxnr7dE60Gfp0BLvnnFh9hmM0xARoKeRL3x8Qur3ntj9fw2PznBWVw+TIjU1UrwxZ/8YsWTU7+5//MX//TmPyTpRoIFRIrkin/4oakmJfXQldbnhStJsRgI4dGrDvjPjF+yYYZBg7k8zhUTKQ9Rj/dizxgBkT57/WTQEMiK1Je2TERqcBctHLk6FWK7YUWI8CRR5pAZNaZiEESIjc6I4Mlo1nuE1GxtQ5+3QvI4LqBxSRhPo7vkglHz/viHb4uIREo9di4pFjMZuHVwXpxIIU5meOF0XExTSDBRj2fjvov0PEpVpL60ZSISRBe3enq/2vzJJ+/7+PVIiInP6Z/l16XS2AbDpWS710ivn1GQz/t/jvKZoIjQ04rii/JfsRxT8ar/869ac9tQzvX9+PMITqgETx4/SQj4aSTBpZ1USTy5qCyIw8YrJMeJvzGSBYi+sFRa2PBVXEFRw7uzN6m/90eScjETAeivJ3h54hh3iQw3JA0myvAEUYqfJfW4P1kHI37P+7uJitSXtpREaqE5NTHSIDEywhCp4VkOZw9S6fAZtcqPnuY2J0ZGCDmmqDMd2Mf952foBQwWDYXqnu4/kJIYon6Br35XLAWYmjVE/meZI14oCyGqVAK5PicaHiK1KZHY9VhPatuCZPjZlCx4YoreWML/nsoSiSw57oc3WCKBZ0kk+qFWRhwReCKR4VN7yUCMSIfkHQAoGj69h4EviR0TVEn4PpFcEyk0UmK2qlBb55rcD/9MRqSGMZGA/8rcUjQYhkh0aGOCrkfk91SQSGcDRf9/B4r+9skgVLn86p0W3v/P1kCqtyeaagY21GCTGT0MYkp43Tvvf7J51x99SXqinp8s4o9/+f7mP/ih0fJWKOfw8Qjnbj9UuLXvV5zcd4Wy9iZLYuc7ATb7JRGbM4CnKohHgpOMxYiYlzTVa60q+ngvUQFkqfzPL4MK8uXad56T2phIF4jmEC8kACnORSLGWT7D99oNJljtehAF+wH9p+J6YqupI4i0j7smy+WzCSRq8HaE9FnTRPojoaxdiqqNLOIo+7loCSOUk3Ae0F6B6XpliVyBIeC7Ub0oVgqzj5Ml8Usi3vHIb6r1GFeCBvOZlLu0VJKIqYjEJ+cxP1Oe4daTWLAfkgyaW8VMr+Z1BJHAC1Zgv0zWY8hy+fhWO5ha/gs8ZwKXkgKRNipZ1qOJ9F/321lRa3dkEfv3fDninTkkkcgVmN/9n1lyBYaAv9CBelnDHydLYh1RTrGDBFAURLQAfO8vpEVDgSeSnwNVAUipEOGpiBQmN/wpkR39M8MYxEo4meGfz2PBEZHiYqZX8zqGSHO/997d6kWK/tnJ8WGKSK3gYPkv0vLv9ZX8huZ2r7z+jz/8m9c9alkPIVIrq2DtLspqNhElaDK+HEu9YWAoZ0AkcgXmV/+8lVyBITDyxMNbPv44S6SonM5di5dLmoKCoBeAu/9Ov6Si9GizJKoCkFJpr89y9WdFJD+tT+Lkwp8ZeyYKY7RXwrkMAyJFwT1RcESkuJjp1byOIdJrO85u3eD3A+21mqtjgxSRWsHB8t/2eJGg9TN1fSOG+PiFVV/80wsL1LIeQqRWVsHaXZTVZCJK0GR8OZZ2hIFXxxIi+RCswBBoRL8YA/84S6SonM6djUuiSR6FSYBeAH7tO/qz/Ui84cfkHaoC+FIJ1425+rMiUiu5Yz+Nk5uaurur9TNjP1JhjGAlnM8wIFIUvD0KjojkA6mljiDSLsJm8I8TPsbHjh0YHG4r9BtfEraheyba+Nz/mbq+ESXw22mCPsJ68eueMKvxY5EkIyO7w8Adg2HgWDvQwAIf4j/GAv7P3kS8Ly7EJdH0bpj42lLjbqStS7YxQVQBe5FSWU/U339z9Rcv3Ohm/zWX3MTEkb2t+o61iHuIGLu4DA98kYTu3UkGC0xJfDQnRhhUkEge26BHhgcPHBsbn4jRKocDVL381v+ZFg5QGh6nNmUwNkY3gJFhSpKJyUlS0kE60G/YU01uyxsJiP+g8I3TyFBGQL6F/rZFJMG5g1F+hj23dyjec0vO6ukqYkqFkYutPx8m6m+mOYyMfH7Ur+8BYQw2w/FWgxkUBwfVRB1P0GTbQUUXZDlF/+DY1cnEWIMbKXwQ/EwwzocnEotwsJZgmJJkyieSMHBq6k5rDCJ1xtiI/6DwNePkz2IE5MdMPpEk6fkvG9NzKBvIFatAYEuFk4stGCMTIU5Z8atLU/TP5NUZVIaTrDiDjDzbpprUHn6+IVTTRIhfMRufnE2sB7m5qy2RwGPaKQcuq2FKkta8dUIc2GxePdX0FmS+TyD+YyMgP4sPiCROz/cjLDqsSQQtIrGlwsrFFoyR0SpHk3VsffNEojKcVdRiW3lB5ohBX+ASE4nqWblS+60jIvHAFLdkidKSciPpYN4vywE8aQSFgFz76fkvOZECn0kyeRDwy0d81nypMMVmNDJiYUMkKkOl7gA8p8tvVSXSnv+2I1Jwpq9ca56SSFKlnEYEkD/OtZ8vN8mJFJhIy+QRiRjguNBerRAiSYd2NZF8GBLps+12RLriW0kUS6QgTGwj5D9PezmnwLWfD/vkcySlPDzmE9VFk9iBXwIigSSGIZEcG4RUlUjvH7AjUuADbr/UpY4LIsmoGsgs3jjqP3oGcbWX5M60sGENIpmtXI8RxqnE2K6jiBQoGojk0qJMRJJZsXPl2g4W7OkWA/w/1GkgHDSIJG5PYQ6ShvsseMdTJ/twz0+L9xfxLWxEMbRTyIMAdRBiSiSzZRgWXHLcnFiRoYpIwRIkkVxalIpIkvWwqQk6+LftYPSIXRnA//NUdI4WmhW7MsdIakqkcAe1cAUMpKF8Of12Qt4jqeRB0MCvpT9cUX+mQH6mnxzhVFaRYVNRi6d8GxbaiXsqlIlIsvWwKSb484lAyz+r2LHAAYK/0nbFZjUxMcU2GXFgmINk8HjyPiEIgn5pqL/cQQv3WVBOuI1QnNSQjlfjGGTx0EM77VIxXc9kwSX3RZAc4apOkSEfTMsbTEP5s8msUR4iydX4bEhPGAyG2TSIv3aSIOHU44FEksFj+BYUidC2CxcLyGXeEwggXgIO5OGcokpB5n4+8aZvWCpGyzAs2JQmR4LkiLGEIkOVPMFvRI3w7VAaIhkiqmswfA6snjJA+6QI5CR0WgKhCBeDhmswFGsnhLiIb6NtRaXj1ZhNso2F1OeFOcGF8FyB1H6QIwDx1wVqIjlG6EBanAN40gjt4yMN2ktDnl54QLJYHh70VnfjbeqZIFouAFcJthNyxsvKEgmYT7PnTJ8yQHikgTgHRcMHMpIOQJ5e6HpeLA8P2juMu7aWBrYvTiGASjY9Kkqk2JgNzJ4L/ddn2DgUPIlDRCJoTOKM0gt9v4nlEYmAfysKwHymhnE5K1BRIsUncIDZc5fbWh+J5UBa6BJJFKF9/+C8bn6RPaooPaU8wkdCOFQRpwCEn/UcyS3io7bA7LnwLKMx2YF66QDUB4KwKYhshNoPJg6sVQjnQApipiDSNbd7Bi0BzKejBJ0lV1UixXWtdexjAmh/OFR7CnJQLJCKbYTawcojlGKcv0Ulay4PDzrukijlXAHMZ1qEx/C6Sq6yRILoAjliV+c5kMRJhzBl7IRnKoLIRsh0KBYNAoVzpPaHgS0Vq0kvhdoOok9Hg/LQSYHhe1iCqhPpmWj9BAfvZso1oP2BnNPYRrw7HvBwUyJFERVEuoCeUo6CPUi6FNoGCD9P3pfF0kd4sr3he1iCqhPJsG9xrkXlEKYsNDuN/WcLRABpqDC+8OyGMPyJ3B05CfZwv1IQKRLCgbubAKEKRbiObYyKEol2XKgPYD6dYzFy+CTKIZ4b4RGiI9ZFj3OIIn6DGwGpOkAELHEOaGsQs0N8rvKiI0MLYD5To6JEAuTK5Dmzpwxw+5wih0Pz0ggXb8gf5xBFFLxbVR0gApZI905pP5oZrsTGqo76R2A+U2OZEimz4YpSLx/fx0WIdAeix03To1wq6oFTLpRA25D4NwM3CTof41eUSMl+HTB5LB4hZEakOGEQRAB5hOiutoCq9JakwdIUI5SASIBcOUnQUXJVJRIxlQeT58avhxfJEc+OAdyFWYTornh/kV16+uV0+yx7pwTaBoiv6qGdSxCWCWDyXGxOwJyg5Q7AXZhFiO7GNlApM1TKw+EAt4uoVEQCYRyrBB0lV1UiEfZfRn0LRBcWvo4NcwBVhP2ojVDUaBd1j0+O08Obu1IeDvxAbkR5KnLmsNXSKhN09pKoJpEguTRaWEiey+otG+dwWDA2iyNEtj2CYMBCeSTrQwoindT1vM0T6dERJFq+AOQqDWItuqlXdCEqTyQjb7zOX2wc4nRFY7M4wsxxaTBgoTySpXn8gfgudZCyDIhqoXhtA8RX8fJBKtyNXr+uFngrSiTy7QtWz5k8ZYI4XdHKYZIxOroE7kKOxKBB0SNp/2IknTIR6dJ1cSx9nIr6Z1Ov6EIsLyIBcuUUBH0EOShEiG9qjj2TRHAbIeN1guYgf69MRHp82El68STb1Ri/mkQCwbX+cxnNkWLDBpFcc4nNGyZC4l5MU8AkGuooiBjQ4fJwOI04eS1cbUf6bnEiDH7cRhosLyK5Lz8GR5O5K6ARrl2WRkjceBn3SFhXQtICl4cD1vucMDkiLAuETjUDgIsEk0RqImHXCjxKhgUjjyXx7EFUC6ARiIVWLEIylQa9Hd6AXsYg9tTj8nDAGlbTmY20JUjdGrhIMElkWROJ8hoH+s9dTDblkO84hwD0Uj9Cck9zN7x2emmIVPgkCYhrJy0/SXBZE4nyaGCw9/NA4khYe8HTDIBe4hGwKkzuib3sm6SnlEciAYEyEQkEcYzgXn1bSSIdIq1Y8IVNFEBcZzNJInLAHQEBeqkbnCY9zcHs8iBSkojxeSYCVJJIQH55pO9HX3DKgkNAckn4e8cjHEAc7QN6qZch9gBxT28we+scdrdotZ3tcocQSSIG72Epqk8kg5J1Xh8ciFRRLRoZAatCQC/1MsQeIO4taA1mcS9hBh4fMgEQ1y5ITSz3GbyHpagkkeiyBO3nQHDtDmSqaI0TETAbIaXWjwF5fLpijqTXAvFIRXvSB+LaxQ6Yu2fwtFNg2RIpk9EKZRekIhJmIwTopRiTxOYhbCJDZpGCSEVPkkipbqODTzOcJnw8QfrkfFSSSCD5pvtcJkSiNsUpiQR8cL80lMcZokUc5rUJ90lfRhUmEhDXLnpH0g0zCGMZYdkS6WCKU7CEoIzyLYhEHkDIhyIgdU7XL3PB1KHO1SUSbVkK6RMEwXUKLCciPSTnlVeuOBGFBtUMAYlAzmmQCORBf4d0mE4mgWg3KPMITB4WIlfJ4EhJbAd6PzOkTzADrVMVifSc1j9pD9IoNZmhi1Y9gPBLCGr7Cy842U3i6nNZhnx6SnlYiLb/XS/Ukz7VsdZEcgZmRUTbf/N+6rWaxSQJyC/YeShUkwAumLyDq88ZKLpA6o7o/AsSrJdVSdo5YoAqSQcVB4LrFKgikRgPO1f5uQEOoL5lTiTsyGyKy8AFg/CLToaK9HSOihEWSqGTJJB8S53g0JwolhGqSCSmtrUHabZac31Qac4iZ8eA8ItOcLr0dI6KKSeRnFccmcQ13fewHFUkEjDfdfuWnImE5UDd2s/ZCCmVFdIHFHMknQTLSSSgvjkYSpBJOJosV5FIbEnqlixIvrmBMgfq1rnb0mDkcXl6/ANKeeQPEOC93eUIoL7pzPXkcK5O95YzkTKYIzEOT4CPQd2a4Y5+AeEXARQPgOQbhhtCw9ZCPekD9e1iasO/e2fIbyCIZYYqEgmY73ZEcmU/T2CS9vaLyAWSb+wNnZ9FWRlxDzCHNKkTxJ1WakuTFei8Z1ObmZ6mToFz88uWL5FEh7imAONtUEkkLgJIQ3nQu/84GyF6/UUjQUmMIidJIP1qkZ5rdbq3rIjE7Dl94sSvk0wQYyLRLk3jE2LEoJ28cfonZiWrskRirHshbXoNyTdbVJBI3DHbmnXMruu4H6yAMgOQfGN5wR8LwYFeA+GWcJn0CU8oApSTSOwZ9JA2QTqBZUskzvmu5g4V1tIgcyIhfqXpdysTnzGv0zjlEaRf2e9qfyqSIhnWPWfGPVjtQuqKA7fJBaggkc6wLgzvnEHjsWALDFwII0uR9yvNvFtZCRgJ2WBlhuwDzHelPxXKOxODAj3ps+6+IW2CdDmfmE6bno8KEmlAz+Mbh+z1TqCKEI/dDu7rjdC3NQ6OJdwch/buk5ncxRlu7UseOMgHa6JwR5A4wHWCUTl/szsp5680DKhkqCCRgL3R5f/5+T8bPrexu/UKfrrhxV+84UYsNodW+sGrbt3KDS9G9+KjUP2Y297x/5FPgUcH7/oVm6gow/CBZheeno+uWCwhxK8ajYezAyPWupV+xW3sVte4CEB+topt40YvNV0rSCSqXF95fe2GH3ubP5x+9TWV8SFEF6tX/Wz1yjXD3d6bG+fuvvj3/2nZwwkli9Pfu+aDD6dvvHT3RSZCW5bZN/1/QiK1wuZ+7xkQqfXA1s+oB4iSapVS149Ot8ppxoJIWg9nB4jlWP3dDa9uuP6SX3Ez3eoaVyUYfLaKre8/Fpc9kT5+YfX6V7xtH810r1I9B9FF13tvrH7xJ79Y8eTUb+7//MXu781LHjJBpB2I0/9856aPZr7z9n+8GOUQi+BfbHnr7Ja3/BpsRuhp0sGv7WgF9zQTCH9S+MBnrzfJB3qSB366/pUfvXy6VU6bVzwR/gARkbQezg4QXaxds2L9j9d/++35VsVtXrFWWeMCxBZCQcKtYtv+O09ezhqoIJHA8jmqmXSll4MDadiApw+YLPumImxrIsFb4+ApMZH2YQ9c6qce+HKVagokmUWqH84Orie38YZb0mrv+S5JOWug+kQ6Tvw1eG7+rNZDJmDU8KcWveeMPjFuEtdas9vNn7wf4KOREBPboxocJoJ3xsESIh0mHtgePbD3PPXA2Ly3JLWY4xboSJwI/hUCoL+mliRW+z7f45dau9jeOxQVM1LOGqg8kcJdNvuV4zPG8Oza/9/e2T9HcaR3PMkvqVRdKpW6SqqS/JCUnUs55ToucVXsnC+QnH2+4xxVnMOSX8ESEIMk2AezsNoFLIPxCoxACLBBNgJsYcOCwfM3Zt665+nXnZnt2ZlePZ+Sdnr2mZ5+5uW702/Tfdn5T52m9hrUrxiDzbVeC9PrzQ2wuSeZbULSRnjr64G8seSPyI1LZlv0soE1coWAsgqarfKDGxSVq6A9zznwUEji/ZqOMiCOj6FDan6KXrOFoU39hQB1VfMVQ72Ex46IQhp6gR91AluEmc1iQvpIM4YyI5oS1xq5QkBZBc1W5fa3LYX0dCGq8F94M/qcZxM9slEGLF1Y7sTx9sSffDMI/8/dceTZlbgZ5w9xgwRvuIxS4KI/tci3OLIWf4Mv4fwvX45448PQfPhxarZd4Mfz0Z4OvBufioESYfpX4d5e+Y+90f5Oaf3RYcvpRqdq7B3Af4xP2sKe+JP1rHjYLe/JOrpOydSJ21JIn+EuLi3DUgM2XWIDXkGQ5zGWE+3o/IBX1C3wJfxP9ErH43Zqtl1gXEx+ekyJsHOoP0WJskNjF9IKfvORpR4N33emZH8l+1XYNkKK+3Ic3BXsjG41SL/MISRA0fhwdtH2P2onFi9BmkLvedweAYG0Em6Bmv7wJdyJrNxsu8DxNmGMN17RRtippCb7Y8DYyBlFtGX9KiGee1xut446xd8oOToYJPtDN9D2FdKD3cH5fUFW7T+f1vpbsnaAol39nLexDNzl+tMUHs0GqFkoSgGkLeKmv8T1rY3VLr7xmZWbu5YLHO83jDH1sjYC3p/eH4m//8U/P3PZ1qwdRbygGRapUmIhsXZrfsEfDkqPswDJ/uI7AZSrsL2EtGP/yr5d4U/jXFrnP5dW+3fN82gBinZ1mcdzLKQwhaWrUXgTpQDSFnHTX+L6xnq/g298ZuXmjuUCx/sNY0y9+lQXAe9P74/E//7ZPz132dasHUW8O+7671hIrN1auOBQboeQ7i+6E0C5CttGSN+gvpyHYC2mczQ97i1z7+T+fBbvvQtJvLXptU13QjqRJbA4t8ZSCB3jGc5ZtMX7sXm1v9zudJO2iwMHM+v+bmpuc7PajnRrfxbh/04qEf4bpTaL/ck5xbOK05xwXr5FF/xwK7twg7LFtSV0Xj5QroK5vS4HPgkpIszSCuw+yY7b+ubZQI7X+20kJIfFZ00KoWO4VBw19CC6nfZyf5Xd5coVlcwR0gWWEhyyv8SfT839E8D65h/wjzGju3Cln0iBcp3Cs9aWTvO2EZLwIN7Jf0CsolDLlFUISUjg4B9Cx75YxVtILnT66xu8W4qaxxDMmq4ryiHZ9zc/Fcb/WnlFigOaN70Ea21CUi7cqEKSTpt8mklIZmoQ0pv7Q8cen8BbSC50Vze2eD9JtdQrmDWdKZVDsu/vnb2hP5a3biEsBJlbklrpJmNHOcrXXAupaz/NOdmuQjr4RlxGcte1QUnh1eORYy28ha2itXg9rD2Csr/dRyV/JCILGK3ANhk3+idSeU8057nUE0hmcoT0ia17snL6puciITnr2qBJYWdPqhesWUg7u9Z6ynjmJuP9mdQzGCNXSPVZOxJSyKHf8LtnYGtb0DwvIiF9nW+0hzw0X0iyPyK3LgYWISU138bIFaLP2tETaVSk0/D6bHb32KrtNLdVJCSHFbq+Cyl+B8RoTUbTNJorRD6MI7vXpNrQYpCQEqTTsBPdPbZfKYOQHN4aSgq/7allJMuLEZULSfFHpBUVF8FkTQYKq2CU56HIh7Hvregoy/YR0r69sk2FJDQDvNbLLySpNaICIQnEWRBRSHKTiHTjS+ZhDRr2CMP9EbFXyyXRdFOnVY18lG8fjo7jUem5WOxXoTzeCUlkFlX724UkEXU4CdwKSU1BztqpDKzWIUIqtj/FHxHgH0arOjTnGJAP4+CG/TiGYb8K5fFNSFKN/zyq979ome5jIDOfRIOqHIv7iEpPJB12q/UKF9xf4o+x8wJEHydM7xnHVmleofGgO4yRhFT0NOfENyGJCKMMPClymdOIFbaMJLsuXyp2TeKPsY8QRB/GaWZbaKN6aY4nIn4LSRxloMg47yAsqiDZdflSsWsg/lSHUU6Jb1DjNLOANqoXSBYN8ETEbyGJr5oVObmQRnE7agMiHU+hfKnYMU+P44UKoE+TtQnPARAWzcFvIZWe6IY1ILkZQF3HF+nDEqpKoCDstXow2MFqbU6GqjmeiEySkOR5Wyywd9Qcdm2QYIPcQVUJFIRVXYPe/KRtszboOQDCojlMkpCu5S+PXLyZLKurh2L9YZuSmWfDpRj8uXUhXoAhNlgjjxNIFg3wRGSShGQdKVSEj8KTP0pBmGuVJVCQIf6keVyDlU2cZDCPk/Q4Tt+3bzZ2vBbS9z1xPX+1HSgB14C0rBuQlrL5R5uVzU57vIY+QhJp4fbG5XrdUPBaSMtSVQEJyQxIS73ZkGNi80LX0UdIZOtMsmxMbSjDayHJlz1/xhmKRylIjvH2xsoQf9KvDe3HrJWgjj5CIrzJuCnnlTFRQmqberiYY4IrX2TYjptywYEt9S1nqZs3L1gjj30cIYUum6CpKeeVMVFCum2dtQQDSsA1bMdNKRVDujS8FJyaDWM6gBKoC1ACDWGihJS/kARKwDVsx00pFUO6/GrFatb/0IMSqAtQAg3BZyGpGY3iQqqsawNLoSmlYkiXhj5CzExCKonPQrqm/NjnFhK/X4ydOEckyyM1JDPPf3T0/tibmUDerDaABWr3RMJnIR1TZkfIe3af8J/lqro2bPLiGlSTQEHuC9M2KfATorU26DkASqAh+GCURwcAABD2SURBVCwkVTY980j6Aug5BI6ckbjIp/apKIGCXF1nIdCZNy7YrA26fbkD9ERyh3ou75tH0hc4n43NC46ckTjBZwKrKIGCnOI/MaAz86Ki1tpEITWlNpQxWULKW0hCw+FU9MsGVSdQEOAhrT/8vSxtD/osJ1zHOEICwALXG1IbytieQgLrTlyQpeCHkFig/5XGmg29rzWPE2AB63igNeCxkL47pX7XQCGBcZtxApqQzvzNssaayWczd5N3NaDGBKjPCx0eC+nje+p3OWWBNgMnvihku23aE0nbRwg0oYzjvHq0jnGEMF9+zoNQnxc6PBaS7g49axtJPwO0QZdku21GqRh4SNtHCDQhnbXu2/dc1n4O9XmhY8KElDPnDFnwE81zzQFZCs0oFQMPbej6CLU0G+oi1/18RcPyNeNJz5kwIeUsJEEWrKhrQ5bCoFNJAgUBHnqi6yMEmpDOWvdzALTBJrDthWSZw24UQBusD9AGNd/ZrXUfDGiDTcBfIT3VvhtTWEgVXRDQBusDtEEGekpprA06GNAGm4C/Qlq/ovs2V875MS5JVZLXxlXzTcjM47lsNf6gchNoYqPvaj4YyIJFxtUdA/4K6Zi2kf1zy0j6nC8+RyuV3BrrV7MwVJFAQeL5+FJANaOavJOaEeVBG6wDlPypnP0qx4S/QtILINdI+kJNXSVCwnXeUEUCBcFj54NqRm1La+sacxb8KP8gnFUAWfC6NkdSG5MmpFxPfKFNEhz4ooCdqySBguAudKCa0Ve6SkZkrrmPEGTBZtSGcrankIwrrsA7bUIZCVBY40/Lat76OAvX3EcIULgJ5zVjWwqpZVxxBaBwE0rFgMKaGTjsQsKzJtU7jpDQQ4mE5ATTq0eQo6M/4JVKujbgFJpQKgYU1gxTgc3q/cmHwJI3HTvCuHpQlxdavBXSmW/131/PMZI+4BVtl5lRwSk0oVQMKLyhjl2HzaqQwLgybj6/iVagLi+0eCskY4YJhscV7pVKujZgJ5pQKgYU1vQRAkNY841qHiPCHLdQlxdaJk9IOYokYFlzg7DPBmTmAa+o/tgrGYdEHiOAVxpwWhEkpOqFVEUCBQHjSoQw1l2Ts3ZgXKkdElIlv2xgXKkHMK5ECKOvfqwUPu2RxwkYV2rHVyHpu6xGHNf0cRFh02alwOjeyPzQrjiBgoiV1iCbhXf91MoXMK6MGyHxJtSGZvgqpLU1k2X4SPrrYlwY3RsZsWKs/sz8vbN4TfFH6OmhVr4AXikwUa97AK9ca0BtaIavQrI0Fw29b6WfMhjZGYXzt/Fa/UK6JHTlVYVkWZO/+Oxr2TxGAK80oTY0w1chWUpCQwtJ0o0EI7qiQRz+rYIECiJ26QbZDJY1+Yvb5x34UxawrNXMdhQSiKvL34zoy7AU6n8igbgm9xECy5o8n0atfYTAslYzvgrJcnMWFVIFXRtE7+ovFYv+KH2ERDNIVjQElrrxeGlZ1mpmAoXU1c+DmgHiqmHGoFEQU6i/VAzC2i25j5BdSJLuarx9H4iz2JOQRodNbq3jO91goRh0Ae5G7I8+vjdvXxxggSfRrm/NR58PXSZQ1p+nkSe3P5T84eaH/HTczSrnsiGwvuXmYb9V1YB/ke4lntQ9FjnHUyGdNnRZzQXw0NtnOEttc4TCMKl+/16WwPFh+q4Q5s/g3cyfdvZbBOky/I7zzkCOHMyd5tZTh8bhtgyac52d2Pec/gCOgKdC0j/VozcrXv+rnwZxXuT2H5sigxCafTnovhm4LbqyfcU/oWECB3c5TqAgLOk4Uxf6E/7hxxSeZSw07Yx+U1Y25MjM/O50UNPBSCNND57VvxlfCxMlpGf/5pk//9vDfx3MP/fa63df+AtTZBBCWy8GwZ6gQiGFCTzY7TiBgrCkYyGF/kTHnPnDx8iExHp+X2AUUmj+1QdBTQcDYvCDaRLSqBiE9Oxbf/J3f/Tvj2YOvP8/93b8qWkccBBC+15aWf5ZWCJYGGS4ci8WUpjAjv2OEyjpTyyk0J99L2F/ztxM/YHUuuuHUEjXmXmeeZuaP/h1GBnqOBjpiTT9i0Gwdqm2kyoyUULKx7fZEARH0NdvrqVsbo58SVhby63P0JfTDhMoCKTLO7icxg94/+3UH3w6Pv6EmeeYt9h8ZI0fzfgOBlD66bLfy87q2PzQ4aeQrC/jnY3/zKze4MEzi4sf7t3zSszhXsrot0Y2ydDxMIE9SQK/PuYugYI8XGKhk8gfYP7MMH+25rPT8Tr3doZ5exGdrX11HAxkwXOpL7/nfqyRkIpzxZIzHiwNeWCdFCqiB5u9Xgvj4NZAUg0TWBP37yKBgggTYqj+TCN/BpuqGXmrmMd6MOJlFa9cj4RUArB0Qb5xeUhRWDSGl8P5fd7BUm2AkIRXjAoLaUYUkmht1fRECuQrR0JyTdR1DvJv7lZINxczFtIE6hXSytHUG/bOEfbn90e5t0eT7CgW0tFfoqP5iplrEdLxzJGjrFWWhDQqUy8EwYOHUy/oW4qiLpmQf2duhYQzH/2kArlmIbXkgOCPsiEW0tuXdeZahKQbeI+EVJ5n/uFfn/npc90XghenHtx7Qd9SBIFpFntte61bIUWJH9wV9J4P854Xk7GjahZS5JHQ/or9Sd2dmuJ2LKSZq4HcnFyXkNKjiJpgswMhIZXm2f/6l2f+8h//7Sfff/qbO3t/8jNtSxEEphGqte217oX0YHfwaDZcXlyPWze2Vrt1Cyluf2XNRtif1N0jP+ctQ1sbmTkWEmtOXmDm2oQUehI1wWYHsoFPLAmpKM8OsUNgGqFa217rXkg79q8sXQ2XF/ubERv9Tt1Cittf5zY3FX9Sd+d+F22YWNczcywk1pw8x8y1HExyFCtRE2x2IOsd0ZEx+GHGPyENA4JCs9hHlagio9waF+azUvH7q3FD4epyu91xlkBhllJvDn2UNlyuLne6zJPXUG3CTGLtL7eZ/cT7mfVwSzGP82BmUR3OAXYg/X5XcGQMfpjxUUigeclTsNpqG0Bqzg1z/TKj3RpRKQPTbbeX+y4TGNGjbr+/inyRpNHthN4ie/QzgG5X2TzGg1F+8aQDISEVpiWPLiIA6SbmyILMBpsqIwtJzP50+usuExjRo25/fQO5omTWWoJdyZh2xOjjOxglDy4dyNj7XYn4KCSQpiVQrEOeSKKQdIzinSqk7uqGw/2P6lHozRZyRa0+EOxKVYkUfXxHowpJ9mQ8fhjwVEiBeQgO4B9Gs9nqAE3fgJpzHaJHUqlmSIV2A/plcEeb4okWD4UUj2cGJmtSADKaSUgkpErwUEhxF0wwWZNx1/KXkZxDQqoGEpJr4hnDwWRN3uw0mumJREKqBA+FFD9twGRt/xB9GmexfxKNR1Dp+NXqmwZNF5LtxYhQSDW+N4Gp4n0Xh3goJOAfRqt5FvubFwM8HkEFRA1TUgtvA4QkeCMLydIgPdAcTF1CsjpaNx4KqcU/jFbzLPbx1MvoZXP3OG+Ycu/RwGYU7c05muZ4osVXIbUfG6wQfxpHqG79mG1UDc4bpirwyGoU7M05muZ4osVDIUH0cfm6zWqWSvJ9owa7JSYBX4UkzbqX0UIbmSKTkAjX+CokoxZAWBSMTBAl8VBIiQrAYAW8UdHIBFES/4SUznsFBnOurJ3JShAl8U9IN5IhOcBghiCH2WQliJL4J6S4h5BRC/dPBzZz+n2nzsmKiEnEPyHZ825sdgJDL6AnyQwm62uOnSK2O/4JCYSFTC+deMowi30666M4vzBBjIy3QkpzeDKstm5TP4v92XvCTgjCEd4KSZprm8GEZOgjxAZNAacuEYR/Qkql8kQ/FzlIm+kjk5AIx3grJIMWQN6sSGSCKIt/QgJpKdKSNysSmSDKMmlCAiVQIDJBlMU7IfGaa9CaQQlozdRrlXCLd0K6sZoGQGtWpgPSm6PZyAjCHd4J6cxWGgCtGZSA1vxV35U/BBHhnZD4k+aYbhz9xye4WddHKB5DKKLAdBUEkQPvhAQsoO0EFA8SZDanPYQCKiQRjvFOSFwBdz/RWD9lUw6nI65KnL2r7IYgXOCvkLSloCxDp+0j1OLZQRIS4RTvhAQ8pNMCaIM5IxNEaUhIBOEAj4UEVusQs85KEKXxTkj2MhL6TvfMyb776Ikrhwgi8E9IWUPRaE+kizfd+EMQMb4J6YtVHgSNGbRBzXf3z7jxhyBifBMS7yEUBCceqWbQBvOaCaIsvgkJlXyuXFPNgIJqHyH8Vi0oVoIoj29Cgiz4cEmx4lbY87cV88aKdkcEMTIeC0mjhbtns/DtzxTzubtZWI1MEOWZLCFd+jILazp4A+owrkYmiPL4JiTcOgSK9SSewk1tSLJHJojyTJaQhG9ISMT48E1IYAhrvrGblx448IYgUratkK5dceANQaR4JiTUQ0iXdwPjivLNYxpHn3CIZ0L64hJaQb0cUgCv2MtIVEgiXOKZkD7+Fq1kAzQwwLiSx0wQpfFMSIBXniotRYJZGUdIHHefXu0jHOKzkFQtCGaljxDuIURCIpzimZBalrVAEtLXch+hc3fwGgmJcIhnQgJhTdbCwy5eU3J+IAwpSUIiHDJRQroutg2BZLZHJogR8FpIIFmFOr1hQjp7NyAIV3gmJP4Y+epKyEz0cSXr68PHf3yyzs3XflQj348Mpw9Hn5tjcJrYBngqpKVzVzjv8y7fwAKzl7n10kEl8u1DWeQzuoGPCaIofgnpB9ZDCML/g7uCqakAV3ML817Ovhz96cayiyOoZoIoj19C+pL1EILw/8Hu4MjPn2IhAQ5svRj9GYWkmgmiPH4JidcmQPi/Y//K3O/C5fkvBykLA2Te99LKvpdCncFANsdCYuaFQcZ4D4aYJPwSEs+7HUFNQt3zaymzTAqLKM4is14+kJqvoFryH2aYeXOThESUxi8hAQs8WFj8cO+eV2Le6KWszTApXFnMzAeY+TR/Ef3k4uLePam5w2OTkIjyeCqkkMHmWq8l0JsZWMy9Q6vIrEYmIREjMFFCmhaFJFnfuk1CIqrCFyH1ji4yjp6Ov8FCOvIity4ufMnMXCTTmXVxPtkfCYlwii9CAiWMhTS9rjNzkexE1t738YKERDjFJyEd3BX0nn+qE9Kb14OsgZabBSGF1jdeCZenvovNJCTCKT4J6cHu4NFswFqGtjZWu4KQWAPtAjMLQgqtU1FPhlN3YjOKTEIiRscnIe3Yv7J0NQpvRmys9zs8a3ctyBpo55hZEFJonXo1lNmpG7EZRSYhEaPji5COo/qC9+IG1NX+crvTTdqATn6AahM+UszvLGTmuaT5NbS2uz0MCYkYAV+EFCSFIkS3017ur65xkHA05lXJ3Gm3l1HkuGsDCYkojV9CEnNjrf76xiZHyax1sHlDY0aRY0hIRGm8FtLqxlbW41SpPuhi85bGjCJTn1ViNLwWkpAbs1doa/pBrJFyCGeQkAjCASQkgnAACYkgHOCXkHpmpURCspjVyCQkwiF+CUlGFJLNrIlMQiLc4ZWQVAZ5zUMiE8Ro+CQkHXnNQyITxGh4JCSCaC4kJIJwAAmJIBxAQiIIB5CQCMIBJCSCcAAJiSAcQEIiCAeQkAjCASQkgnAACYkgHEBCIggHkJAIwgEkJIJwAAmJIBxAQiIIB5CQCMIBJCSCcAAJiSAcQEIiCAeQkAjCASQkgnAACYkgHEBCIggHkJAIwgEkJIJwAAmJIBxAQiIIB5CQCMIBJCSCcAAJiSAcQEIiCAeQkAjCASQkgnAACYkgHEBCIggHkJAIwgEkJIJwAAmJIBxAQiIIB5CQCMIBJCSCcMD/AyAdLGc+EYDNAAAAAElFTkSuQmCC", "text/plain": [ "plot without title" ] }, "metadata": { "image/png": { "height": 420, "width": 420 } }, "output_type": "display_data" } ], "source": [ "# fit the tree\n", "fit.trees <- rpart(formula_basic, data_train,cp = 0.001)\n", "prp(fit.trees,leaf.round=1, space=2, yspace=2,split.space=2,shadow.col = \"gray\",trace = 1) # plotting the tree" ] }, { "cell_type": "markdown", "metadata": { "id": "G3QZT03wWit8", "papermill": { "duration": 0.047977, "end_time": "2021-07-22T21:33:46.929812", "exception": false, "start_time": "2021-07-22T21:33:46.881835", "status": "completed" }, "tags": [] }, "source": [ "An important method to improve predictive performance is called \"Pruning the Tree\". This\n", "means the process of cutting down the branches of a tree. We apply pruning to the complex tree above to reduce the depth. Initially, we determine the optimal complexity of the regression tree." ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "execution": { "iopub.execute_input": "2021-07-22T21:33:47.029943Z", "iopub.status.busy": "2021-07-22T21:33:47.028067Z", "iopub.status.idle": "2021-07-22T21:33:47.046676Z", "shell.execute_reply": "2021-07-22T21:33:47.045274Z" }, "executionInfo": { "elapsed": 36, "status": "ok", "timestamp": 1658250115477, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "EumyiAhqWit8", "outputId": "44570ce8-fcd1-4478-d787-289d8c8c77f2", "papermill": { "duration": 0.069407, "end_time": "2021-07-22T21:33:47.046815", "exception": false, "start_time": "2021-07-22T21:33:46.977408", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "data": { "text/html": [ "0.00188444410871555" ], "text/latex": [ "0.00188444410871555" ], "text/markdown": [ "0.00188444410871555" ], "text/plain": [ "[1] 0.001884444" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "bestcp<- fit.trees$cptable[which.min(fit.trees$cptable[,\"xerror\"]),\"CP\"]\n", "bestcp" ] }, { "cell_type": "markdown", "metadata": { "id": "IG7NtCahWit9", "papermill": { "duration": 0.04778, "end_time": "2021-07-22T21:33:47.142808", "exception": false, "start_time": "2021-07-22T21:33:47.095028", "status": "completed" }, "tags": [] }, "source": [ "Now, we can prune the tree and visualize the prediction rule." ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 454 }, "execution": { "iopub.execute_input": "2021-07-22T21:33:47.245777Z", "iopub.status.busy": "2021-07-22T21:33:47.243764Z", "iopub.status.idle": "2021-07-22T21:33:47.717872Z", "shell.execute_reply": "2021-07-22T21:33:47.717444Z" }, "executionInfo": { "elapsed": 666, "status": "ok", "timestamp": 1658250116115, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "-mIXdvQmWit9", "outputId": "d66e227a-bdbb-472a-aa22-707070a59da8", "papermill": { "duration": 0.526641, "end_time": "2021-07-22T21:33:47.718019", "exception": false, "start_time": "2021-07-22T21:33:47.191378", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "cex 0.438 xlim c(0, 1) ylim c(0, 1)\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAC2VBMVEUAAAADAwMICAgJCQkKCgoLCwsMDAwNDQ0PDw8QEBARERESEhIVFRUWFhYYGBgZGRkaGhobGxsdHR0eHh4fHx8gICAhISEiIiIjIyMkJCQnJycoKCgpKSkqKiorKyssLCwtLS0uLi4vLy8wMDAxMTEyMjIzMzM0NDQ1NTU2NjY3Nzc4ODg5OTk6Ojo7Ozs8PDw9PT0+Pj4/Pz9AQEBBQUFCQkJDQ0NERERFRUVGRkZHR0dISEhJSUlKSkpLS0tMTExNTU1OTk5PT09QUFBRUVFSUlJTU1NUVFRVVVVWVlZXV1dYWFhZWVlaWlpbW1tcXFxdXV1eXl5fX19gYGBhYWFiYmJjY2NkZGRlZWVmZmZnZ2doaGhpaWlqampra2tsbGxtbW1ubm5vb29wcHBxcXFycnJzc3N0dHR1dXV2dnZ3d3d4eHh5eXl6enp7e3t8fHx9fX1+fn5/f3+AgICBgYGCgoKDg4OEhISFhYWGhoaHh4eIiIiJiYmKioqLi4uMjIyNjY2Ojo6Pj4+QkJCRkZGSkpKTk5OUlJSVlZWWlpaXl5eYmJiZmZmampqbm5ucnJydnZ2enp6fn5+goKChoaGioqKjo6OkpKSlpaWmpqanp6eoqKipqamqqqqrq6usrKytra2urq6vr6+wsLCxsbGysrKzs7O0tLS1tbW2tra3t7e4uLi5ubm6urq7u7u8vLy9vb2+vr6/v7/AwMDBwcHCwsLDw8PExMTFxcXGxsbHx8fIyMjJycnKysrLy8vMzMzNzc3Ozs7Pz8/Q0NDR0dHS0tLT09PU1NTV1dXW1tbX19fY2NjZ2dna2trb29vc3Nzd3d3e3t7f39/g4ODh4eHi4uLj4+Pk5OTl5eXm5ubn5+fo6Ojp6enq6urr6+vs7Ozt7e3u7u7v7+/w8PDx8fHy8vLz8/P09PT19fX29vb39/f4+Pj5+fn6+vr7+/v8/Pz9/f3+/v7////kI6qvAAAACXBIWXMAABJ0AAASdAHeZh94AAAgAElEQVR4nO2992MU17n4rRsHp7wxif3m3m9uguMEO8ZfJ/diOwE7fn0hLuA48XVsmuhVoosimkD0JkwRxVTTey8Cg8EIMKaYDhIdBBJFQiBptTt/wbu7WkmzO6fNzDlnztl9Pj8Idmb2nDM789mdOXPO8yQZAAC4JsnrBgBAPAAiAQAHQCQA4ACIBAAcAJEAgAMgEgBwAEQCAA6ASADAARAJADgAIgEAB0AkAOAAiAQAHACRAIADIBIAcABEAgAOgEgAwAEQCQA4ACIBAAdAJADgAIgEABwAkQCAAyASAHAARAIADoBIAMABEAkAOAAiAQAHQCQA4ACIBAAcAJEAgAMgEgBwAEQCAA6ASADAARAJADgAIgEAB0AkAOAAiAQAHACRAIADIBIAcABEAgAOgEgAwAEQCQA4ACIBAAdAJADgAIgEABwAkQCAAyASAHAARAIADoBIAMABEAkAOAAiAQAHQCQA4ACIBAAcAJEAgAMgEgBwAEQCAA6ASADAARAJADgAIgEAB0AkAOAAiAQAHACRAIADIBIAcABEAgAOgEjecfmVHza6bxhb8moX3W+TtLvqD6AXIJJEips/99t9D5vUbXgn/Cer4826CwxjwbNvr/F9kpSU9IZhrF1WZ3fVH0AvQCSJZL7oy3wh41V/2rrwH8PY//TZ4OKKZa818ft8vsrQNmGHQCTtAJEk0vIzY9tTLVuE/hf6Y6ysuzH0T8GI55JrtgGR9AREksiolyozGgxr4E9dFP6z+aeHS8oNY9wzXU4ZkUs7EElXQCSJlDR/7sUjRW/Vfb0g/KdpUJ4BhnG0tHaLyXWSftA+/Me7VgJOAJEAgAMgEgBwAERyxv4xmWjm+4TXvQFV76TrwusFCIBIjrjY7xKGPZmi6948A1XvuWT6OwFxgEiO2LkZu6qX6Lqn5ntTL0ACRHJERKR+N692vja835fb+7c+V70KREpIQCRHRERatK1TXuqg9LHL+9XeocgSaUfygDmXuvZcI61egASI5IiISMe6djDaGHcr/JebPaleJUukid9cHdjxQeBTafUCJEAkR0REKqtzyZiT2rEkdUSXQPUqWSK1r9i8rKVxeri0egESIJIjIiIVdbeukiVSS2P02fn9uxZJqxcgASI5okqkky0KrKugsyEhAZEccXICbs3VwVX/+tfge8gdcnPeDn/wn2XIgn1teVcH2AFEcsaS3n3RDC0Or7/bJldArd8kbwwqOqkPot4+xwTUBzADIgnhcKci+kYOCCxve1xIwYBLQCQRzBrkF1V0+ZSut0SVDTgHROJPSeetIosvHDaoWGT5gBNAJO6ca5EvuoYeMysFVwHYBETizdre5eIr+SZ5k/hKABuASHzxpc+VUg/0OigGiMSVgvYnZFUFvQ5KASLx5Kv29yXWBr0OCgEi8SOQNSJA34on0OugDCASN4o77pRf6QHodVADEIkXZ9rd8KJa6HVQAxCJE4tSKjyqGXodVABE4kLZgGUe1l44bFCJh9UDBojEh2stv/e2AdDr4DUgEgdyenj/gwC9Dt4CIrkmMG6s100IAb0OngIiueVhl/1eNyEC9Dp4CIjkkqOtFDp5odfBM0AkdyzoJz5ovh3OQq+DN4BIbnjSc6XXTbAAvQ6eACK54Erbi143AUFgeVtpQ9CBakAk52zsVkrfyAvKp3ZR6MYtMQCRnBKYOMXrJuApTBsAvQ5SAZEccjf5oNdNIAK9DnIBkZxxpKOYwHUc2fUp9DrIA0RyhMDAdfwILE+GXgdZgEgOeNxri9dNYOPxGOh1kASIZJ/zLfK8bgIzhWnDH3vdhoQARLLNuq5anZrQ6yAFEMkmvvQ5XjfBLruSc7xuQvwDItnjTgcN5ypAr4N4QCRb7G0nM3AdPx6P6w29DkIBkewwO0ODXm800OsgFhCJneJOO7xughug10EkIBIzZ1pc9roJLoFeB3GASKwsSZGQrkUw0OsgDBCJDd/wpV43gQuPx/W67XUb4hIQiYnrLU963QRe3IVeBxGASCzs7vDA6yZw5GwK9DpwB0Sio0jgOo5ArwN3QCQqD7t+7XUTuBNYnhw316pqACLR+E6lwHX8KIVeB66ASBQW9vMqXYtooNeBJyASkbKBK7xugkDOpMzUdsiTaoBIJK62OO11E8SyK3m3102IE0AkApu6PfK6CaLxQ68DH0AkLIGJU71uggyg14ELIBKOwuRvvG6CJKDXgQMgEoYjrQu8boI8oNfBNSASGi0C13FkJ/Q6uANEQvGk92avmyAb//I20OvgAhAJgU6B6/gBvQ5uAJGsrO+WoPfe0OvgHBAplspR2gWu4wf0OjgFRIrhTptDXjcB0BAQKZp97e553QRAR0CkKDQOXAd4CohkoqTzdq+bAGgKiFTLWe0D1wGeASLVsLq3/oHrAK8AkSL4hn/pdRMAjQGRqoijwHWAF4BIYfa0j6fAdYB8QKQggazxAa/bAOgNiGQYDzvu8roJunC/TdLuqv/dXh15dfmVHzbSM/saV0Ak41irm143QV0evP37v5f1e/PJrxZMf/7jXyxZu6zO7uDSwM4PnxljGOFXWR1v1l3gdTO9B0Ra1D9eA9fxYFhj483FZc+/8Tcj+yelk39jGGGROtefXTVMPPxq/9NnPWyhIiS6SGUDl3vdBKVp+bNXfz3ayEraaWT/zthcJ6LOsGf65odXh16trLvRwwaqQoKLdLXlKa+boDZDGxk7Lz94tk2Dyuwf3R9dLyKSkd//F91D/wZfbf7p4RJ4kp3gIu3qEfeB61xS1PiXr91u84n/lczs//zLz1ZMrpP0g/bhFRW5hhF+1TQpKWmAx61UgEQWKUEC13Eiu77XLVCaBBYpcQLXcQFEIpK4In3bGkJ9ANxIWJEWDoH0jwA/ElSkJ6mrvW4CEFckpkgXWlzyuglAfJGQIm3oVup1E4A4IwFFqhw12+smAHFH4ol0t02u100A4o+EE+lQJwhcB/An0URKtHQtgCQSS6SSztu8bgIQnySUSOda5HvdBI0o2mNv+8ptN8Q0RAsSSaQ1ELjOBo8+sZvipTIlgafsJ45IviFzvW6CVvSwn2wtMDjhEh3WkDgiAbaYsdbJuyYs490OXQCRABTHhjt734xEfdgNIgEI7rV2OjZ+8USuDdEGEAmwEujgPELZhnR+7dAIEAmwMvYrF2/e3ScRn3mDSICF3e4uzw53S8BIgSASEMvtdi4joZ9qb/cRlP6ASEAMvk+L3BaR1+Yhj5boBIgExDDoW/dl3Gp1130hWgEiAdFsmMOjlHutrvMoRh/iSCQIvMaDy135lFOSfIFPQZoQxyLd/0udjxKxI9YVT1ryCmdR1vEEp5K0ID5EetikbsM72fVvvlTv3UaG8UlSUtIbhjHuvwt/vsnrlulGyhluRflSEimSbXyIlPGqP21ddv2hL/ubNzYMv8/nqzSMz7oab430umWasWQFx8ICaTs4lqY48SFSyxZG6NKu9WfG8MY1Cz/rZrw1yrMmaclJzmklRiZOGM74EGlYA3/qouz6A14JhH6RIpd2ExsW/hxmltuhpCXvmY9jv+BcoLLEh0hFb9V9vSC7/pXfPP/emzULHzZ+upWHbdKQ9pe5FzlzCvci1SQ+RIpwIdvo1NrrRujLlK0CCv1yoIBCFSSuRLr16i//cM7rRmjLwUwhxW7s63Lknh7ElUiAC4raCMpz81XvRHiaByIBYfxthY2OO9I1AaZVgEhAmOFfiyv7TLv4n1YBIgEhdk4TWfrl+J9WASIBQa51Flv+7VZ3xFbgOSASYBgVrUT/YtxrfU1wDR4DIgGG0fc74VU8anteeB1eAiIBxrL5Eiop73RMQi2eASIB5/tIqaYydb+UerxBf5EqN5OfnD/aBJmXiTxuIalzOjAwjscQ6y/SkO8pG9xrfVVKQ3SlxyVpVU1YKa0q2Wgv0rGx1E0etTsroSG6MkvmnKFpcZtaR3eRfK0Zhp+UdzkqviWacmyY1OoWTpZanTx0F2nUEZatKlMTaNKzLe639smtcG263PpkoblIZxjT+AQGrxLbEE0JdJCe93VX/7gcDK63SP42T1g3zZonsB3aMtaDX+rc7pJ/BKWgt0gTbQxZXhCvV+cu2DPei1qPtmX++tMHrUXKG2Rn6zXpgpqhLa7TTjjkbIsST+oVic4iBdraOx67+sXl1bljKtsUelTzlX+5TnihGjqLNMPuBf6hRMyAhSedqcdTCLf/Jb2TQzAai3Sjr+23JGIGLCwbsz2s/P6n8sZTSEFjkTrct/+e/DYP+DdET3ilnXBIaZwNN9FXpPnrnLzrVqsC3g3RkzJuaSccUt5Z/CQoiWgr0p1UZ++73/oK34ZoSs/TXregsqfAeCvS0VakDk6DAJS2Psm1IXqyZJHXLQgNN9nidRP4oatI65wPyC/veJBjQ/Tk+/5etyDMhOVet4AbmopU1NbFm2EIK/+0Ew6ZziVhrQpoKlL3W67ePjHBh7B2y/e6BdUsnuR1Czihp0ibF7gsICtuJ5ixMFWhe5P16V63gA9aivSwnesiFsTLN6EDcpVKCJrTNy4GbmkpUm8OPdjxOsGMTlFrQWknHHKoWzxMq9BRpN0zeZQSJ9+EtvG3VS168Pcd4mBahYYilbbhM/j/cGIOYc1Q7zFoXptir5vgGg1FGnCBU0GnE3EI664sr1uA4GYrryZ0cEM/kQ7ym+n6YHMZt7I04Vqykokon2zS/TdJO5HKFLtV1gtfKxj9LgbtRBpGC6wKEOgPAf4EoZtIx0Z73QKdWfuF1y2IWzQTydcqIXvaOHGhl9ctiF80E2m0d2EG9OeJrLQTiYheIp2VG6g6zki56HUL4hitRPK3joNH4J4xO8GHvItFK5Em7/O6BRpzfIjXLYhrdBIpf6DXLdCY4pbQTSMSjUSyG1gVMBHoqENExuz6XrfAMRqJNHO76Brut0naXfW/26sN4/IrP2x03zDm/ludOrtF1yyc8cI/PB7gRKo9MMqij0g3BOTefvD27/9e1u/NJ79aMP35j3+xZO2ysDKBnR8+M8YwsjrerLvAMCb8D/96pXNgnNctYCL7d+/8ZNm5l/7jL/duvlTv3UaG8UlSUtIbhhE5MCqjj0gdHQRWpTGssfHm4rLn3/ibkf2T0sm/MYzw8epcf3bVA5f9T581jMH//qu/6h5UssCjtBN2yX76Zsobo1+/s+rK0Jf9zRsbht/n84XHVoJIvFi4VkChLX/26q9HG1lJO4PfhcbmOpHjNeyZvvmhtSvrbgz+3TPz9ItqBK9yjN+ztBM2yf69kf3ytb8+9cf81p8Zwxub1oBInLibIqLUoY2MnZcfPNumQWX2j+6Prld9vPL7/6K7YWz+6eGS8jOr5ky48yfNRRp62OsWMBK8R8p++UDuk0bjB7wSCP0iRS7tQCRudBQyP7qo8S9fu93mE/8rmdn/+ZefrZhcJ+kH7cMrKnINo2nwIA6Y/tujLzzVSO9Lu01cpubLICzS3nrP/zH/ym+ef+/NmuW1B0ZZNBFpveiQnBp3vNK40sXrFjjgQrbRqbXXjbCDHiI96CC6hvgVqaKljpNPb736yz+c87oRdtBDpG7XvW6BvvQ65XULEgItRNo53+sW6MuXC71uQWKgg0iP2urxFERFzqZ53YIEQQeReuV53QJtKWmZcHGSPEIDkfZM97oF+qJO2ol4R32RHqsZiU0Lpm32ugUJg/oiDTzvdQu05dAIr1uQOCgvUu5Er1ugLfcglqY8VBepHE4GpwTau0trCNhBdZGGQwpyp4yCCBcSUVyk46MkVpbxqPb/JxZJrFgIOVO8boFjzs0zv+rrUSvsobZIvtYys2+bZ2oEekisWAS32uvb2Tn9rPnVtLO47VRCbZGOyUwuVxE1l72rvudhCP8emV9BnOkW9dnna9HdpLZIUjm02PxqJoz19IyYSZzdvWmFPUCkGiblm19d1PcWQ3fOfR79up8OYdhApBpivgd1v0nSl9iLge0bvWmHLUCkGlKjX6bAAyyP6B5ze1qmQ78diFTN9QnRrxfnetMOIDV2QTcvWmETEKmapQejX98e6U07Ep4LU2OXfK5BBziIVE2f2P5iHb4H45HZJ2KX6NABDiJVYwmc1wfy23lCd791kQfNsAmIFOGxJWfMup1etANAdJdq0AEOIkXYbQmJXDzIi3YkPHmTrct2bJDfDpuASBFG3rYsUvgmaUHD6v+NakzaTkPmHrMuKxeQiYQzaokUSUlkVCUoCmfFefKPOq/JiHCIiC0++J6Eet0yqnHkQ9tiChET/uQ0TeyUYr1FUvo7LYLHIhU3f+63+x42qdvwTvhPJCVRJEFROCvOzP97Z9oV8Q0J9LQuyxGRAIMP2fVnvPz3Z1Zcf7Feo8aRD23Bs2+v8ZnzCWma2Ak5okT9DnCPRcp80Zf5Qsar/rR14T+RlEQ1CYqCp0Pyn/7f/5IQw/7MDOuyMnUvKLLrZ/+4eNAbg14N/LNx5EMzKpa91sScT0jPxE5XJ6CWXlG+A9xjkVp+Zmx7qmWL0P9CfyIpiaoTFIVOh8/eKfu/Emb3zUHNxFW31zUoUihxQ6twFqGqD80oGPFccs0GwU9Oz8RO879FLlb3UETwWKRRL1VmNBjWwJ+6KPwnnJLIqE5QFD4dxv33g+fHi29IKurKfJSyIccjIg1q4G/WOPKhjXumy6mofEJ6JnbCDHHsr3oHuMcilTR/7sUjRW/Vfb0g/Ceckii8IpSgKJwVp7hJnUYCkl7GgsxjdkjZ+eYRka7Ve/6jP0c+tKOltavDn5yeiZ0wPz3Kd4Cr1WvnGfeGo5ZWWoZPAoK5hskarXwHOIgUZuMO5OKukpsBLMRl6VS9AxxECjPoIXLxVIjyKpkUH2bF9DNS22EbECkMJtXzaYjfLxnsvGR0t7g6gEghfLgrcJhvLpebo7GrFO8AB5FCHMGltUMM6QcEsuQgdpXiHeAgUogplzArvjgqtR0JT88K7CrFO8BBpBDYK7jrY2U2AyBcvyneAQ4ihcD0NRjKX5nHGQWkwWBqd4CDSEFuYJ4CBukJOVglsvQAYaXaHeAgUpDl+OO3cq/EdiQ8PUkBy9XuAFdCpAvYX/S8zjJi2ffF/+wUDZVQv0NO4/uKDaOyi9Jf4GjITxuUvsxWQqTu6HEFIb7OkFA//hZJ6Svzjk9Ia8vb55FWq8gd8sFWugNcBZHOkn6zl+Ee8fDjcRphZX+85B6zdjV5/ePkG3Iawo0V5ByDO9dLaocTVBCpSylp7Ygc0fV/RTojt24SXb1DyjvQtihto1kS2d7knh2lO8AVEOlEFnF1oLvokaOZpPOtVNW5caMtAUktPGhdJKEh/KDdBCl8ma2CSB0pPczl7QVH8yHdIil79G5ZAloiuPuphDmR3ChCTgozoXIHuPciHcqmbVEoNpNsgDx9b7jM9JvsdC1k2aqgtco36DGs+oqygcod4N6LlEy35IzQmaqxCeJi2L9MZOVOOTCTbbtrbYlde0rRh9pUhTvAPRdp3wKGjfaIjMY0DxHa04Svl8C6neJvy5oG7Uo7bcZm0DVRuAPcc5Ha4qZERjHnS3Et6Ek5J1X8Gpy9h3nT852YPmHvYXj2rXAHuNcibVnOtt0A/EQVt5D7GgxjQr6wqp1SbGfC4QlNJlWtpT/nULgD3GuRkhkPsr+DqLjF92lfhMfnCKrZOQMu29n6u1QZ46xc05f4OLGKLuKb4RCPRVrD/Fv9uNUDMU3Yso2yQYD2kyWd85n2tj9AGrqhDCyX0DNOC2+GQ7wVyZ/M/l15k/n+2h7pVEG7qfaF3o7huzuKXQOEtIMrD9IZNroqIeiuM7wVaQk6nByao2LOBvrvzSxUXHAPWb/S9lu2qT/TdwNTgkQVe37CeCpSZVtbm28SERyrsjd1k0uTBNTrnPL2Dt60Rtmv8mr6P2LaStUOcE9FmsveiRtmooD4F0fn07dRa5TQ2ONO3rWS8QmuZ7D91ijbAe6lSBWd7L6jr6NziEjWBfo2PfCxbeRz2+Eo2vmz+baDM8WDmTYrp19AeIOXIk23/WzIl2zN9OoSli65Jd/wrtUFPe46fOPnAh9qu2cTrfM0gqrh2D0U6YmDhwIPW9ntsKLBIlKBjGm6jBxGpBZkZMoKju3gTRpjomBVO8A9FGkSeYwbmvxOfB/T3x7DspU6fUX+Vi4G/IxQdZKiwf4Rq9oB7p1Ij5ydnYf4/jqsJM9ujtD3MddKXTB3t5t3D3X1bpGUsEyvCqPOl1oU3ok02uFv9HLq/CU79GNSZIOd510iKXbZgThI1ehiW7ewbsl6DSgZz0R64HiO0UieQRzYWlE8iGOVbhiU7+79gd64RF4eM5A5xsyudSLb4RjPRBqBi1tPJTCO3/jVMsZRaPhQrFK5MNJtCf6xF3k0hDvsF2wVkwU2wzleiVSoxoD4ffZH23hJe6an/xpSqsFgQDJeiTT4qkcVRzP6utctsMN2JWe982D7Rq9b4BaPRLrD3EkjFuVmSJAob6PaMHRuDNYp2hESj0Tqz32EgjOERlXhzXgnD970QNE+bRt4I9KNIZ5Ua+ECOTalWtzu53ULhPFY1Sic7HgjUm+moGzima9TZssUp4Ps1GeXqmO62fFEpHzXvbic6KlJgJ0QR6Z53QJxpOsVWhmFJyJ1V+XWUqO+hkA7leZycEb/WyRPRDqnyONN46Eit2oszGecZqAjZXFw9+eFSJ1VmS68jXmAl+eUqDoPhwe713jdAvd4INKJKTxLu98maXfV/26vjrw6VO+ZBvks7x0qOM0FR9K1S79ng6Fx0I3igUgd3M1JePD27/9e1u/NJ79aMP35j3+xZO2yOruDSwM7P3xmjGGEXw140/9/5rIURb5FWtCw+n+jGrtqMQcujhBVcnZ9USWzY/sWSYVGxyBfpMMuo3AMa2y8ubjs+Tf+ZmT/pHTybwwjLFLn+rOr/Ay+ulr3B/VYrh79rOHxRzW+/MoPG903jC1RvwsfJ0nr9OsgbJCd6Zys/XGXS3lfu+9QoNGxyBeprctkRy1/9uqvRxtZSTuN7N8Zm+tERBr2TN/88Orgq07/9P2JZeLrsXnE1dn1Z7z892dWXH+xXqPGWR1v1l0Q/JF69u01vk+SkpLeCK7f+idpIu3gHW7h3Ev/8Zd7D5vUbXgndE5G9ijy4y6dvYwjh5VqdCzSRdrHEP6KyNBGxs7LD55t06Ay+0f3R9eLiGTk9/9F+Aoh+Op/PzMaskyP+PwccXV2/ewfFw96Y9CrgX8GL+32P302uKxi2WtN/D6fr9IwSn+3T5ZIvna8B9mNfv3OqisZr/rT1oXOycgeVX+UshlewLadUo2ORbpIbGlcCBQ1/uVrt9t84n8lM/s///KzFZPrJP2gKmJiRa5hhF+d+P3P/otlVHcP8ukZFKm+kf1yq8+M4Y2NlXXD45MLRjyXHFmdOilflkgTuc/Gu/bXp/6Y37KFEXO74c05yXqLpFSjY5Et0ral/Mpye8tJeRwbEWlQA3+zxpt/ergkeEk67pkup6qvKX794x8lNSSXwIkC/nO3DuQ+aTR+WAN/6iLTVZJH52QF6+6p1GgLskViTePCgkuRCig5HSIiXav3/Ed/bho8agMM42hUMDBZv0i9+Gex3Vvv+T/mF71V9/UC04dY++Mula8ZU2Qp1WgLkkVat1ZufSRW03L/KsK3U71ugVhGKjKlxh1yRQrYSOMinDTesSbFENeD7ELYyT6oLnJFWrZVanVkNBmxulClz0wAPlWjedtDqkiVyfRtpFGmx2SyUrVSYfDnG6VDkjMjVaT5u2XWRuEA6z2ut6Sf97oFgsm84XULuCBTpIoOEiujMu6a1y1g4dIwr1sgmjiYixRCpkizVMqOosktUidVppyIwtfT6xbwQaJI5WpNqdFCpJ2LvW6BaHLjZA8lijRVqUgjl7jOihIE/0F2yjFGjUihrpEn0mO1LoYXHvG6BQxMPuR1C4QTH0+RZIo07hT/Msu3m15ssDVgp5edrW+bTmiJyRBEB0i/b04O5U1ErEo7t0gPkGNRzn7PqS2ukCZSiYibyv3mZI47bc38t3WLNM8U43SwvHnRPW+JLX+J+QdvlCdh0I8ssLHx2HzkYiWGy0gTabGI3uYsc44SP/kqoezQDhMb/mV+tZcSvKGnaZTOIbcTqpi5sFpwBb3McywP2Tml3eH7tuaT77i4+n9f0b+gMEmHl21HL5eKl1nN3ZMS9V3Uj9RVvLrj0h1Y1qWNIH6rmRUNqHWv54aobx6/tH7onOSFiGOwYXAa5Wr74iT0cn9n/m20jeYiRb36ZhF+y4OUORN7MAcpTHnUvUovPQa70nk4OOplN0lXSOdxIRqODie/cSTu4nPO127awwetRfLF3IwTfit60PoWMJcNYY4sNL/armbuRfts2xz1crqA3iAUadgsl6nkcB7YQ1RBOniS0FqkEzFBtwbjY/NTr1uG3y6rxrJu+lnzq4o4eRZvDIm+M7woKack/uObdhZ/DAzjzOfYN07x/hGl1iLN/y769Ql8pC9q5K1R3+aFKSy0HsWU6Gm93TQKvU8ituNS0s0f/lDM3IU/BoYxFB8j5Yn3w1S0FqlP7KUAfhASXaRtuSHCRzF2XUx34HJNptZSqIydCJQiZwYhQaQVkWOQZz0GZM8zPR8jr7VIlu8h7O0og0hrckIED6PlIMZmCi7RYyYTjeg7vyDL5NyzE0RagD0GhnF0DqHMh57PDtRZpIAlcSWug5RdpBzEQfx6RcwCFbpb3TM5Npx4IaXXjBNUkVDHwDAGEnMopV9x3zBX6CxSvvXuGNt9wy5SruUgTrwcsyA7LpK5WqfeypmMSxcpaJJVJPIdXKHX2b11Fmn1XsuiCbhrZWaRcqwiWQ5hgZyvbsFYb9DxHdM8cSbSN7EXojH04x+zzBY6i5T+wLLo5ijMtm5Esg49UmtmlTMuWy+Dt2+QUbEzkVe5IWgAACAASURBVPpRJL8mLF8HG3xFkpsJBdXn2RGzrenoFTRt/u4jw7jwfnIz02U3XqQH1rx+47hlK/Iud8xiaxzkJ1Iy55kOxdlGzd4xPcwiiOSnXnamWL9WZSLqF6kmE4pRlQEs/Kp2ERdQIs08gd7WdPS2f2X032oYWXOMvqbRjniRdlq/p0U8u5TxiZnphejslnKTZDoUJ/KNzqapHASRvqKGqrk4nlP7nMFXJEQmlEgGsPCryCJOFKJ+zO+lozeOup4ob3I3+EPzYvM/V9Yuw4s06qa1OG7XdlI/sShQY+WlTKUwH4pzDT80jfAjiNSLniCqq7sEdi7hLZIlE0pNBrDwq6rkKHzYsRG1tBN66KX56N36KDQiYvgKY4wppBpeJNQZl85rUpLUT8zMw3TEwsMyplJE3yOlmbIq4EWqZJhIeww/hEgC3EWKzYRSnQEs/CqSHIUPY5EB0RahQxWZjl7RB+GfmNQcY+a02qX2RDr8hb22YpH6iZnZikpE7bc8mROA6VBk7DWmzqh9iReJaaBwey9jOwsRyZwJJZIBLPyqehEf0F9Sj9DDDkxHb/Af3ntv0+EWee+3aWa6/cCKVIDqCAzwuraT+omZSUfeesmYSmE6FPlvN2taXPsSL1LKE4aCv5E25RKBEJHMmVCMqgxg4Ve1i3iAGajYrRK11Hn394adqM17c0rqKvUTM4P+9GZImErhoPubccB9O445g+yi73Okx5gTbPUu1FLnIg1BfnXbCxChHpjQ9TKmUjgQaQPqOtTKllWuGuYKfUU6iAm+XoY8UM5FQl9B+rwfuO+KI5jZxBI6wB2I1IPx7sfDMCj6ijQD15uVgpoVxlsko7vek5Im5aOXp4q/Ybcv0hPWsd0rtrlpmCv0FSkVd0G8DRWhzbFIlyeit1+5m1ai0uCGgEqYSmFfpFU5jEX7O7lpmCv0FQl7bYV85uBYpBWYM6tU70lJuE+vUHz2C/siofuPUMzzLAyKtiJV4o9HH0RYLscipeG657qoEJbQKfnYiVvib5Jsi2TjO6vCs6li2op0ahZ21deIbghq/+mIdWiRsI/UZ3+HW6MBi7CRzwcIn0qBPxSfL0KKtGQ/e+FZ3zpvmCu0FQkxeLmaAOJbtTvtEUPXHKRI1lm41dzVOQUYPvL5duExwPtgx8T13ooUyc5vv2dhULQVqT/hWTdiVvJOfIChMMfboEU6Ox37FgWCqTkGf7qJn0pxbCRmxcX/zUGJ9NDW5Ncx51w0zQXaikT65vlutnXZrN57jmA5MP6j7WiRCNlfJlxy3HiveWidYlWD+Juk5d12Io7Bwanvb0GK9AX+2gPBQ+rdsBiEiXT4uOnF+c3Y7ZxCHF6JOhmKVs3JGpmOZtjCnBy0SH1QD6WqyMd0jDtlQVTf1E6R+Zu2bMWvyxSfW/fhWsShGDLPdAzMInW216szxJswKMJESokaa9nvIm47h1wlTuMajhoYXlaYl5tDJ1ok0vB9zhPOY+zvKjCny2DCbNIjEoZ+0g6FWaQiwq8nikIxYxNpiBIp5mFOKe/BG+uJz+guovJaOhHJEkTRzFCu8TYuT4h+fV9g2kvS14OMqRR2RMLNecbS35MwKKJEip0avGse3/KHEaOcITsCnIh0bB6hkm/nElbaJjv2hNk7lWfxZmKTD0TTVfwDMjsi2R6scM2T7lRRIvUvjlnQje8sZkovJyo6iRORsolpFbkGy7ZOBRoq6knVYWIm8RniU0naEOmW/ehAKYKiXBARJZLl/uE+3zsKikhXR1uXOREphTg4pS+nSUkhAtY98rUWlIlpUmzEyyjwox64YUOkKadtl35xLNfGsiFIpOPWcQfLeU4WuT+UsgHi2o5NpOhwueRQAbs45qb8DhHa+pKgZMyUX1LxHeCUQ2E+Bk6e1nkRBkWQSMMRXU6dKIla7bCbNof/c+v1SejoUU2KDuBOeTpZyfEx+rh8xMIFy/hVYILSnSB+KkXwUBBMMh+Dq2McFH8si2dj2RAkEup75BbHBF0TaA8L7lp/ssoKQ4ePRlRKEdzkwWqoeQDZQf9KdBHxUCSPMg12+T7yevdQDoXpGIxz9Nykg5g4FyTEiHR5HGrpbH7Trug/BdbHeGXhw0cjKsnVVMphRE9rd0I5up+9pA3zDAJ2CKM1whTSrpvdQz4UpmPgbBzWN/O4tZQVMSJNQg54CrQlpR23BV2k+Ycsi8pCKlExJ4vrQekJLuU2MG3PSvTyQ8ivJHf0pP2OSrhJIh+KmmNwyeHoEflhUMSIhOmhy+eVe6OMfgIXp3GohxqWkNszlyG452KZ6Dh9bqB+C4mfSsGK09CvW2NTWglHiEjYUR0TOU1gxIXuMNPF/UVRySDaFnN5TX/BdqT5k3mf1Q+pV247hE+lYMXxCHvpYVCEiIQdr+tvjR8Dagfyc9IqOARV2EONucXrdqIYr+w13jNsNlNvVZ/05VylU85Mo2+DZiVbAC9+CBEJPxPrNC5/kT3w89JqYQ49g2cs/cKC02PmdTvw61YRxyHYZxA9/4mc1H10ht12+s4ALr+PKESI9JjwHHHESR41MH1Jd3fdB8pwPmFTBNoDP2s0SE9u2ZjCMMSjlzCVggkXQn8hvA8/GhEirSV8vZa34tCf62d6IrV5k9t6GE65fD5ZeYhjDUo/5fmElDxitQoZUykY+C7b+XsrJEfmEiFSD9JPwUEOQXHP4ed/m/C5nQ9QxJIqlsuE81uZxNUncJOznXBoCX0bvxpxZAcVunjzNJEzI60IEIkycCbN/Ry/pWxdwj1djilFZj6JZRiP+XcLrA+9opjIGiGRgYksgyUkTKVgwNXo+idcx+ZTESDSbsyzxQgc5vgNYDOEni6RzHCWGWKo+BC26U653g20J0+/slUXy0YzxU+loJPrLu3ZWDEZ2jAIEKlv7FSkGHIQ45ztwXjhEWC4xyHB9nYePVzUk7uAX+BDpg+PdzwKR/R3l165mOPYTjoCRKKeWd3dzvFjvYJPczfDi00k2vcGA2foSRu38JqNewk1C9+KAh3gbr8HjaHEaVec4S8S/VrnQQd3NdxkHVp/ZJ6EaihXsixMZehD73fGdTVhFrCNxejpZRrJKva6nUNSyGOUGCv8RWK4+17vbiTUpu2sW7rqUlu7m2kzDpOSWJ7qlrXmMzWAOmK1iuV7udTmBvdJEdMKeLSDDf4isZwVnVzdO2cwh4kZ6vjJeJBBjIPcurv98q5k6qc/NdhlNVUwai9hKgWFSve9btdtRvJyA3eRLiOfUPpn9eiLpXdPexGXCOdC6eie+Hp6pl21Uw32Ev1OOqmSYbZvkQ9ixgDdGEiqZ6SjOSnUOfrKsEOv5KLcRZqIut4PdCEO+34ykPliLQRepNKWxMjP99rbuNFARCOp4npr4u/c1RZ2HyNmoC+GLyYTC7rYwkkvx2Zbn7SXpLIkMlcH7iIhb0uW00IWt7UxcOhhOnZVBuVhb5mNbo5LuLBy3Skn8B27Y6cxPWSdKCE8rji5cGEYsaoGFRLiVPKEt0joa+uetGewc74ri0CvYi8+HBH1w+9bwlwPdvgEtZLUMvadCfIEnUfLT40Gb7OeMG67lKWxiX+4eKHwFgk90416TqzehJrpjWYKfjA0tZ6MCzX10Dbth/tBoFbSi31fQmzbiFxcSu1csFlPiAr3c0sk4X7svlx4i4ROHUAXaUVt3AvatoRACnSRjjLXg/3upouUx74zQdLQvYMMItXUw2xSLsOIVSUo08b4CJxFwkQDoYs0CxEMCwOh044u0g7WevBTNegioSJ74cHcIjGIZK+eEBNs9Vt6CL/4TJLgLBJm/+kiTUHFOUVSQbiXp4u0DhEGEsn32LkwdJFQsSax4PKIM4hkq54wckdEu6Arv4iBcuAsUgp6/9lFikkMj+DYF/h1rCIxCPsFNoI9q0gs3woGPhwjs0iM9YRRY54RHX5xzmTBVyTcM3pmkXLoIs07jl/HLFJs7nIr+MFmnEXCVSRCpIvCMsVwZqnkieLu4StSDqZnmqdIvQndORxFwvcTM4uUw3SC48ZZixBp/lHaFtn12UoSTFfpAR7dwlckXJ4TniKRrk74iVSBj2zAV6TYRH01iBAplXrngRPpfpuk3Wx18OChm/SVCxpW/29UYw5tYYSvSLh7WdO5V9C0+btB3a6+0yzZtAG7SAHS81BqPcwiHVnIUsnZRs3euRdsU6f3/2ZKZWRLJEuivmrMIkXqCV7XNjRtYV8k+uPY7N+985Nl5176j7/cu/lSvXcbGcYnSUlJbxjG2mV1drPVwQNEvGknjGp8+ZUfNrpvGFtMjx7DXwlz/60O5/3hKhI2F6Tp3Nv+ldF/q2H032T02VO7lF0k7MgdpnqYRZqBH5VnquREvtE5eDW/Kd341jSe3ZZI1kR9EcwiReoxbrU0f8faFuk+PSlk9tM3U94Y/fqdVVeGvuxvHqzN7/P5wuO3ZIpkM5F5NNn1Z7z892dWXH+xXqPGWR1v1l0Q/JF69u01PvNXwoT/4dXUariKhM1OHHU1VN7kbvCMXmz0Mc0LZRdpJWn4K7UeZpFS8Bfp5krONfwweMyH9G7X2XTnZkck7NDY6Eu7qnqMltddibSJECctQvbvjeyXr/31qT/mt/7MGG6uTaJI99LdvDu7fvaPiwe9MejVwD+D7d//dCh0Q8Wy15qYvxIG//uv/sp3shJXkbBTkczn3q2PQh3LD5JbfWr6/WIXiThLiFoPs0iEi6Doe6S0pcGfvWnGGFMkODsioRL1VRFzjxSqZ9ksw5VIDDOsgvdI2S8fyH3SaPyAVwKhX6TI97hUkWYdc/PuoEihnWgV/iJYWTc8AKtgxHO1V/jBPdkz8/SL6CGOTuEpEj5ihuncK/rgZuifi2eMFqagq+wiEZ+EUOthFamUMEvZVEnGXmPqDMNYPMWYPsO0gQ2RkIn6qppgEilSz2ctWvzc9K1gWySGEavhc3Bvvef/mH/lN8+/92bN8sl1kn7QnqkSDriL9BIRaVADf7PGm396uCR4rTDumS6nor4S5ky48yd1RZpwCbfGdO4N/sN772063OLanz8wf+PyFwldD6tIXxMmw5sqyX+7WdPiwy3KP/3He6a5FXZEwg81MIsUqSf4Pze/SISeSAQXso1Ore28gRt37CcyNxMR6Vq95z/6c9OgPAMM46ipKyj8lXD0hacaqXtph59kzq/7+w4x6Ci37m9S3m+e3d+YRH0h+Hd/0xJ5RnPr1V/+gThNUhhZpzyp1h0cRcINGjN4irSNGP2Um0ikMWk8RcIl6jNEiDRekxGrXKJAy4ajSHPwj835iZRJjFHETSTS3QRPkdLxUWD4i6TJQLsbo71ugRM4ioTPisRRJPLwZV4iPSDN4uYpEiEOI3+RNJm8PeGC1y1wAj+RSgm9IPxEIn+r8hJp5wY3lbCLREjUx1+ki1kMGymAlld2HEUipZrkJlIpOZ0zL5FG3XRTCbtI63bi13EXCT8xRCnycIMP1YafSN0JAyK5iXSAnGCCl0jEBy4cRepNCBTEXaRUDineJJApM2I3P7iJ5CPF/qfmBZg7h02kz8lRsqn19NvuXiRqJT2ZRSKFqvdToxb0tCeSJgGE9Lyy4yfSTlJgzCWES5gw7XewiUQYAhdiGOXbrOJvOSwikR9WdaXEpC76iFUkcqK+jhQ7bnxqS6R79BGrKnBWkzu5WLiJ1Id0evnbE2eUVYzIyGEUidyI4hbEdHSPOs9mEmkjcXDn5eR7pNUFLVexirTgMGntmQ7ESJTXW6y3JdJG2peZGgy74XULnMFLpAA5dr5vUrcO7XB0/XB6DptIlbRBLsVDU7HVtOvx0aIcJpGGkGP83+ibgq8k9V9rclhFoiTqy+/dg1BPjUeMIg1yn8dJBgokZnIEL5GoqYjKCvNyc+iQRTpJzzPJWg/p3KPeTQQrYaiDLhK9HrbPjEUkW49jB5tTtMlMzXx8FucCd5rCb4oMJ85LpPS7lA24iLSQniVLI5Hoifo4ikSKYmYlWvEe8vr70t0kMkcxyjQURmQYZMcizeuKTznSt5t1lg36BBeUzuqsfZF2dO2N2JEemchQK3iROO/IY3cireraB7FT3afSQos8jn64/iVbFnklMX8pllGfXDjHqUjp24ird1ge2aNFEhR16ZJtkRZnoQc4nWqH+jbWRKTJmOHeB2nXarvWR728p01SJStRVxcCgxM5FOnsWMoGE2KHwqstUhn26cV2VCZT/PUj5x1xJdJd7KCt+ZQrgdixtNoEaLVQMMr8avEBYRU5FCmblrHrfFZMzhG1RIo9977FJM0LBVVAJE+RJxLLDR9OpI3Y1twbQs4IEyvOCInJWPmyIarT/wFhbKNLHIo0ntbbf2toTM6R0LlnPSlUEWnPelxRRmdE8hT0zngkEjb295fYx1S+7sSMME9iAwYfxgcnU5wh96NedhJWkTCR+sekNgmee3m5Fthy1dvmmrWmaGITOBBEaotInoLemRCcd+Qxrh7SzlRDEKkDMfHM7thRKjI7wPkS0wE77bSoioSJ1K36KEeOVVno5LNAHIjgnHuIqqIpZBbpE8TpWlaI2pkQnHfkMa4e0s5UQxCpRczRiWao5VmGxA5wvsSIdGMUejP3CBOpQ+x1R9AkK+QhBI4pRVQVQxmrSB+hLqBQ+xKG845U0nckvDPINxNE+ph4VWjtW9C1A/xybGAr8gAcF4gWyXQ3UoZAUH5DH6quaKLfQBcp5raKrVj30HcEXytdJHQ/Rbn12a2uHeArYsOJjromqCaJIqmMbZF0wKlIqGzXmnaAp8UOpT49TVBNIFIY8SJ5kC/FqUjDEZ3dI25zbZosrIO9RF3bcRApkvdhzQtNmtT+broWSW5yDrNINakfqhAsksB8KWaRoneKLBLq1+fIAgENRMH180CkLhlInAPjHA4iRfI+zI9KScnxFwmZnONQvWca5Lsr14xZpOrUDxF4iSQ/X4pZpOidIoqEDMgaEDK/Nvx5FDd/7rf7xHweZ6dbFh3APnp3B5dLu3DehynvfNy1tu/AtUiU5BwD3vT/H1wSGQdEXdpFUj9E4CWS/HwpUZd2UTtFFOlrZFwMIR3g4c8j80Vf5gtiPo+FRyyL/ILu9niIVJX34foNY1jtdCEOIhGTc1yt+4N6Jc7ajiLmHimU+qEaXiLJz5cSc49k2imiSCORt0NCOsDDn0fLz4xtT5kWcvw8+iCOWE8xk5I4iBTJ+7D9spFVm5KBh0ik5Byd/un70xhnbUdhFimS+qEafvdIsvOlmEWK3imiSOivbCEhH8Kfx6iXKjMaiPk8UNejmzdxK94MB5EieR+OvfVhs9rORl4i4ZJz/O9nRkNC6hW7mEUKp36ofclTJLn5UswiRe8USSQfJnqRiEui8OdR0vy5F2svwXh+HpWoXSmjRoFyhLLd35TkHCd+/7P/uu6kXDQynyNJy5firPv7G8wkpgz9OsCPzUMt7SpkuJOyIslFpkjS8qU4EykTc2ildYDzI/t71NIvSclTHQMihYGRDTXgLuGI6eTVJAX521NMDnvtEBApDIhUswob10C/EeCYZ19CYrmCSGFApGpysQ8sl4qbpy0Gy/zECNORV3wuAZHCgEjVjMGm9dNuBDgu1edtEZOSQKQwIFI1hKFAuo0An3oRs0LEwFUQKQyIFKGS8JRFtw7wHrgUkpkCkuk6FOmLY5QNTvbR6vw7iR23V/FPrXbEDD5x9a32uJ0idXLr1gGO/XE9LyDjhUORbtBGFQxcptX5V5mM+/ZaNFarHTHzCNs9NW4ObqfGEaJoaNYBXoKPvSXg2s5ppNVZUysIayuyUnL0Ov/29HmIWhxY84lmO2Jm1QhkRsDKLzpgd4p4H6RXB/gefMYuAZOSHMf+3p/SpcNnH6P5x0fVmSy5nH8HH5hefHvHZWkYzqelIvbnw6E8dyTEdvOLXYLVPNYHcZD+8fex2J3yEweiiekAL99jenGJX9f0WPwQsm8WcaulGufZKJjTS7g+/6KuUEppKZIcU0beHz4iXYpKNXxSVASBGigHKXanjn6BKyiEmA7wFWaRjO7cejQImZYETFPUQKSLU6JeDuAd8qoaKSLFRLHpIC6qexU2RZqYTyxNSAd4dBbvQm7jDkiy9OI+KUkDkUZEd7Xnc5yGFIUUkWJOk/Ub0Ztxw6ZIlLtw9Jw/d1TEdGF8uZpPuUXDCSu3buBTSS0aiBR7cDuSujlcIEOk0zOiXwdE5/C2JxKtX+7b+fxbuGNtzIKOfHoCtmI7/4NUcM+UpL5IlmSIu1CJVjggQyRLYsMpx12XScSeSMetCeKiENEB3rs0ZsFNzMxCmwwn9kpR0vfaR32RBlvuiQSlFCjDZZjgKJLlBriYz1mDxZ5IUy5RiuPfAR6w9gnM3I7Yzjbk/oSlvPOguBKJJecILlUCM9YPetYhVwXiIH8zYJOn2OGgNT1KGi35rjvIByl2p6idCfw7wA9ZrxYDybHxUZ1AFqlkAIcqzLgQKZwkiJ5zBBfinZGD1nEpj8UkGcHnauGyIyH6Wp/65o92WygRWztFv3K7N4R3Awcjvkjy0t2Xe5PSJ8X75tRFVnN8bhOGlCPM9HtgXZZ+y7rMPZT9wSTlsoMf9S3ZWVDfSRX0nTJtfDKbWh73DnBkP+F49z98a3eT18846bqKKNyIhM1tQsifYhc/6rHa9Qw3ReKg7Y/7VBO7UD27e5a4LZaErZ3KOk8tj/cI8AtTUEsrW7lOUzIIOeSrltsj3dYQjQuRGFOOuDv99iDDfnYWM7RG5I6ESEUOfRMUjKsaGztFGAtQDe8O8LHo1GzHXc+9o45d4Hxt50YkGfRC3nfu0zKnaQV6INs8VZJ4sQyc4d0BjnsCPMzlmLsA9T56NN9skYqL5MN8HoK/xcWwfhtycZmQ+PQOiH1ajCTFR9+GnYIRmBVlrd11tF+aStviPPKi0jGKi4QbyjFft0AcIbphzsHhAiZsOmH6WYaNlu3nWeUs7PPog1QTiCyl/8zznZSkuEgpmJvOct3iBwQptWaUrKKAe5+yM7rhJjea4dsBTni23pf2cJhIP+TtaBSDuI5+Vlukcuxz/2GicoGKAx/hswePB5DuYXs8x/MrrIQw0foRdtIyCwyXy7lcZ86rLdKaHbg1BbrFhjKMLtgT4+hMme3AcfZzps0ybvKrctlewspdxLlRZMgTFKsIcL2qUVukrvg72+703261uI8PIWC4+vLlxcxTTJvx7AAnx7Pv6jxLwvf0R8uG0St2uKwblBYJe1cR5AhloLJyzP4Ov24Vuj9PLt3ZbObYAY55HlBNUWfHJX9B+LBr2IaPwWYfpUX6kjREt60K3+I2IJ0VlaKnJbHAKgi/DnDambxsldOSe7KMu6J4bA+lRepCmoj95W5ZzeDCLdwTkzATTstqB5YLrP3N/DrAqddWnZzO8WN7NMczU5LKIj0k5t/wMQxoUYjJxJRIDzHx3iUym3UUJ7cOcPrd/k2HvxnIxOxWiH0dNlFZpHnWnNRmRqGHaSkK5eKtb5GcZuDpzhyHhVdvF0P/c7azOX5H2IaQlfR3VDoSlUWizIS9S+gGU47L48nrL4yT0w487AOVRnLqAB9M//IItHWUun7GGbbtOM61VlikElqkl+UaRT4ddZmywRLRgbko5E1m3vQqp/FZ6B/px2tnmhjxsfnVzM1sk7dSsB/m2S/MxbUaaX71BcsQKRwKixRXqNAtR2QuLS0Cd84jeze+Sc65hOXillZMD7twv66BtHHf40v/fnya855gEEkKp6d73QIa+C9xUYxGjdV92IF8LvtaMXS+l+JGHn1O6V3Y6zzyLYgkhfQCr1tAQ/4oYOTo6wXYnE4RNmyiT7P8egVmBbWjt5vjSZwgkhSU76q/OoG+DV/Qc72HI0J0RHF2In3m/0TcDSm1N72n47gCIJIM+A40FsH8b2XXiI4+MqSY8rbzI+hRdbC/rtT4qr0cR7oBkWSACoWkFinSUx+hu1/oIvWLBBIjhBnE9uTTRaop3G7wKxBJAn71pyFKbyEmQiNdpJTq0JZYkx5gh17QRXIcDRREkkCO48GXsrgm/XkwJmYwq0ikc30n9gEkq0gO4lODSBLoqfzcqYW0zjLudENfSzKLRDjXR2FHXoBIWhObAUhB7EyNyK7PoUJcXhUeIuEHOzGLlAMiKciGrcybcjlJHWAnIlhsG++3Sdptu8ItmAxrIJInXH7lh43uB4+KaRx4+LC2SEpKIiVsk0t39q9700nq6Px0yE3GSP4Pm9RteCe7/s2X6r3byDA+CX7KbxjG2mV1dtuuEZd70iTS2UbN3rlX8081dJHu4IMRm0SKFBvo9P7fTLOiEkak4ubP/XZf+HCG/2R1vFl3gWEsePbtNT7zYS29f+VZ6Vf9OOjpo8+99B9/uRc5Sd2dnw5ZcpBtu4xX/WnrsusPfdnfvLFh+H0+X/hOx35DsUFdTSKdyDc676v5pxq6SBuxEXPMIkWK3ZRufGsahJ4wImW+6Mt8IXw4w38MY//ToSG7FcteaxJ9WDupE4uVntNq9Ot3Vl2JnKSuzk+nMM3MDtKyhRH61Wz9mTG8sWmx/YZ+swizwnxpd67hh4HafyLQRRqCn51hvrSrKnZI73adTbETE0aklp8Z254KH87wH2Nl3fC1dsGI55Jrtgkd1gs/4Bg0yiXECfNhrv31qT/mR05S02J5IrE+RRrWwJ+6KLv+gFcCoV+kyE+nk4YOxM0hj75HSltq/icMXSTC/V70PVKo2P7TjDGmsEgJI9KolyozGoQPZ/jP5p8eLgl+n4x7psup6MOa/pa37TTxkD7/8EDuk0bjIyepq/PTIQWZjBsWvVX39YLs+ld+8/x7b9YsnFwn6Qd2f/+x4YJNImXsNabOqPmnGl4iRYpdPMWYbio9zkTCj6Qvaf7ci0fChzP8p2nwnBtgGEdNt4tVh/V9VQLTG8aco9RN/AFGegAAIABJREFU9tZ7/o/5kZO0ZqGT89MhdhNaXsg2OrV2U+H5LNwak0j5bzdrWny4Rfgf01tpIl0hDL81iRQpvfzTf7xnKj3OROKbcMNjnAdnk0ZPm2m9br36yz8QY7nQyMSmDXDf/b2ScEuaaN3fu71uAEewiUsUQvrPN36+sHuR0giB1BNMpIrdXreAI1O8j1hH446QTKIEbuDz8bkXifStkGAibdQqzhYF5YM1GMYKav88Zz7HB15wLRIxonKCidTD44A6PLnieZgtOr1lR2Mi3Da6FukcKadGYon0hC1Mph5k0sJwKYDsuUgPCBF0XYu0iDSiJbFEWrnb6xZwRIMruyLZoxKXEIKHuxaJ+POaWCLxDG3uNWfYknd5yqqvJFfYlXDpPpKWj/L7PmSRiD2Q9OAncSRSMceAzJ4zRPkwXIbRBzMQWxRlpN+FtbRg3/M/J4pUiU2WGoL6TO/9OBJpIT0ftT7wzZwtBtlPkTZvIqysaPWQ+OaCJjlEkY7PJb170wzSWsOYOSqOROqkWQIxEoc4pokURZHsbLypxF/AGx1GLl2O48uhzdaRRaIkp1nUeQ628OVzuvTPiR+R7g22s7Upx8NKV+nkxdDfThiuOaYR0UXzeDcFy9ocGxvPvoNautDOR08NqXR11/r5M9DMXFtzpmNESqVMovR9t2s9pvAZ83bkxJFI2SzZP6u5a3omfxv/uNwr7IXhirrEkheata+dlMToS9VH1N4wE/uXUDYoK8zLzaGDFol6nRosnaFwHUW6M6N/XxNvml+kzSKn0NlsvjVV4H4ksHOYufntPzS/yiAPIAhEdSn1ERd46Oa0fuZmvWV+MWAu8Tf0+Ez08uG0p2UnxtRU0bRH9f/S1yF/PuyJdHqsuf19/2p+MXildTRu3Iq0vftp/K+x70Rn4tk3zNxZih9SLIuy9isI52HRF8SMDyej0rSvx8+Xdsna3ufxzxcqjrYnpUkcjLyyC+4apad11ATUPMtHW1uhvidtiTRh9HV8tY93tbLUG68i3aM9sWxHuvKI+iHHT3KRxTDK3IKDUwgro7PMFYtKR3iD2D8cvBxtTZh2jv3RH0CckLwJdzVXkIJYaEeknNmkeg3jgSUnX7yKtIAWvH33ckKmjejj4PkoAuotUXfSvkT3Voq6SZp6gbLBuq3YRh7+AvemW8SuP/wvce9ia2V2ROpJC9A09GZMBWyl6xcgchTtSXbe6JpEG5ZduxqdmHWQ02TyvHCVNSTmNjn9LrdmRdGfFufk2OfYD3wA/sK1131CkfjPZeIJ6wcSOtXp53pVyGLqJz5vX0wF8SpSBi0hb/4QfBqPVdEJ2A6y5bIWh5usIQUxnY5freTYMBP9aN/hx8fhPvAA4Rf3EinsA/5zmbS/uq7ayoKnOt2kkEfBt1A/8S+2xOxNWFNK6bkaBtGni1SbacP6DRk9whEbLE0WrFlDUJkUVsdkZSy306VsA7pI6bg2fv0l4W1dCenHCSKtt1ZWVmUShap30EVaElNBWSFD6RqmdaGL1K36G8h68sWKgwvfKQs3WUP6xXaqCLpJoovUr/YDj17Tl+CKcYrQkUIQaSniAwme64V5NKp+w+gizY6poIyxdN0SjbGKhDr5LJMht2LzecjBTbIDy6/puHxezYqCVaTwBx61wk/+we+Ej6BCFSnmAyljg1hyhIhIob2pvkliL90O2oiEOPnOx+agxqU4kIULkcosT2KOzYldwgVmkXJiRdpFzvKUi28vXaScXLuXUrSSI1SLlFMjkiB0FmnxodiNu3s7lcnFvLF9lr6FAOoZi3uci0TL8tQW++GDSOJxI1Jfy0dDD7MtFBcijbY+0ewmZBi8Y5F8tCxPO5bi1oBI4jGJVNC0+buPDGNOkyYvmJ5XE0SyXrNjEpPKwnRYIzuz5oUmTa6ZNsCKhLj/yD4hoo0mkSJtvPpOs2TTBjiRcBmNaumAM9+aTaUajiLVlDyvoWmDRBRp+1dG/3BKrooPTBcReJEqEU/jvB3cYDqskZ2ZHz0WACsS6jruMiH2rnNMIkXa2H+T0WdP7QY4kXpQw7GuwalmzaZSDUeRqku+1dKcKCMRRTKM8ibhx/lTl5uW4UU6gbi7nSHkW5yVqAuN8M5MeefjrixZQ85NRxQnpAM86tIu3MaMxUYfU2gJjEgMz7UCyZgV1mwq1fC8tIuU3PJ6wot066PwXKTAn82XCHiR5iCkuS07aGgU5sNatTPXbxjDTNepWJHmHkcUx5q1yBZmkara+CC51aemCdoYkajBFIIs3INebs2mUg3fe6RQyctmGYkuUtEHVTfcB6PuF/AiIQcsejopyXRYIzuz/bKRxZI1JBXV47VcRNeJSaRIGy+eMVqYJmhjROrGkL+zEhNcxJpNpRqOIkVK/qxFi5+bvhYSUaTBf3jvvU2HWxjzoh4P4UVCPh8c42UmC9NhjezMsbc+bGYK6o4VCdnVXThMQBtNIkXaeO3PH5in96NFoufvDDEDHZ0xNptK7UuOIkXytAT/l+i/SGiwIpX3RW1+gTTlRzSOu7/voWM0irhJctj9vZwp+F0Z+oIAur/F41ykQ4uR23t5bedYpA07kZunkUNTOcKhSPT8nWHGf49aCiKJx7lI084jtx9Em+AkEMciDUQH6t1GfXRjH2ciFaexlf4IOUEIRBKPc5F6oB//5S7g1TT7OBYJMxy0tB+fdplxJtLCg4zFj8hHLASRxONCJPT2pNlnonEqUgXuTl7AvjgTiTlqZxHqzhVEEo9jkUpxuUF62YnTxhenIuXigoNk3OLTMBOORLrPHrUTFQYFRBKPY5H24gb1b1vPqWn2cSrSBFyf/TfoDhU3OBJpNj0zezW30q3LQCTxjL9B2eBMT7RI43FR7Cpoo5TF4TRrCPYKDjWc0CXppGmuIXIHW0Wyk5m9tzUEDX4vRq92JxL185mxIEFEyllB2WDudLRI+GcsXRmewYvBEkQthsoP0CLh1effmb9mG2WDiQssIt2x82Q4zxo6Gt+T2nmXO5GGE+PpBemxPUFECiQTImUGyXs3By0Sftob27NDEWwnJ4wNDJiBFClvMvYtU9B9/C7wtSKnbDrdPMci0ufIp0M4ull+8651xIwcX9onx51IBW3JYTo2dclJEJGM4j6pmXhSPtmCFukePiJhCeMzDwFsaD8Mvy9D2k7KQYpESHt6GjUq3B1FKb0IH3j3FtutItmbnHJ6kmXRqXYDEXWN6FTjkVORjPMdBuB3JqNzj9oK4l0kw3h0bM/qxWiW7swxfRLmz3o7IVuVnSt63tz8dg9mXxZvrN2XaJF6EUZ5i4gwVvId9gNftivqA4/sk808H50Rp+y9b611rjZ9IE5FMoz7iJIjrDRXEP8iOUrjkUG4QJmFmpMgCSehpUnBGUREoWD/wKu2n2Tz+vJQtv06nYvEvDcgUs1xNX0UpG/qghHC24zFgUgP0wnlzSdlu3feRlsi2Z523M7a3QMiiYe7SF4OXHUg0rbNhPJukEIBO2+jHZGukLtQEOyyhmQFkcTjRKRbo0kljs0T3mgcbCJF7csQ4jNpAVMpbIo0Jt92DdYwKCCSeJyItH4XqUQPJyU5SHZA7k/ozT91n02RHASUWWcJeUupMxdEco+dNB7VDCZlEfHy2s6+SJXkYS5riV8ZjtvI9oGHtr7o4Fsp0NZmnfazP9jdG+cVsKGESAwfRSSNRzWUXuHBgpIL0WHJSRKdEOA7bPauMMW2srwztpH1Aw9tnUEbxIVi8W5LnYTPxUkaFbaSeVTAhgIihT8KWqaNvOhkFJTxdPjUcqKxnzUki5JCj/9NUpmtD9xRqEBLGBTy5+IgjQpjyRwqYEMJkcoKWTAlCMi3PjyPIiAqcSQVxn0x7QwtxPdg/nkI7bTx5AxqcShmxcZlp9VpO/2Do70Rhwoi2UjjEWH5fkqJvT2blGQ7awht7MLu1Z41MrRpOnloHrYGy20q+9EVuDfiUEIk2/R/RNlgx1op7eDAjbGUDcqZAmEJw2kQ6Ikn6dvEE3qKRB2B5uGkJJssP0DbwrPL1BBH5tK3QfJImyPABy1FCtDnu3X3bFKSTfpQLznG4qYwymAAbQYzloyLPNuhPFqKdHomdZMVmDDUykEf3n10nvhW4HDRa/PAkoMwrtFSpAXfUjcp0eQwPqZndKLkbhXKARdBIwY5eQClLVqK1JuaqsfbSUk22M3QK9JVSOo+Jvq6iPV6O51bMzRAS5FYvqKzjwlvBg9GMvQuz7Q105snfleB9Xp7GPRWOjqKVNGbYaMCdFx61WD5TrhEefwsjt2WFNF2uGJzZq3W6CjS0fksW3mbBZORAFMnsWcd4C6Hnnenhf6KI3QUadYplq3GeTcpiZ1TiFnZVlJEpO5jwEcbvkTh9EQ+7dABHUVKYYpjcNGzCyIbsH0nLKWNiBKE66C1HcnBsuIJHUVi7A72NAsmIz2Y0g7d9eh+L9Xt+LSjs7i0Qwc0FOkJ4yOiIZ5NSmKH8TvBm5skhkTmNNrrMsDENRqKdGAZft09EzlZ5le0Ya6ecHcC23bDPGn9elp0Yzp7aRGp4wYNRZpyCbPibkrKSHyc086fe/dcE8cOrxtARJvxiiqgoUjdMUI8bkl+ALhDQGQrlzD+IHnDE2/nb2iGhiLh7itm0ZL49CNHTPGAkV43gMSK3V63QCf0EwkbDYQ6t2LVHkmzJVkpG+91C0h0FRAtOX7RT6ScdZgV1D6mzaslzd9nZf8Wr1tA4JEm4+cVQT+RRuNG59NFyqoOjyM4ogwrY/iniOXHYurUXcCEfiJhH73QRRonKcYZKx5ONKLTWb1OTpVJPJFCUTfz1Bjgr3JYg/u4rPEAEu1EuovN2cIskjnNsJdcmOZ1CwjMpc9CBkxoJ9Jm7ON27UT64juvW0CAllgaiEY7kYZhHdBOpJ4K9y/fxefotc39Nkm7q/53Oxzs8uMkX/AX79/q1NnNrw7P0U4k/A26SaSzjZq9c6/mn2pUE8nGbJ/Lr/ywUdXj5NC5GD4z7/+lzkdMY8cdMd1NfMcHb//+72X93nzyqwXTn//4F0vWLgsrE9j54TNjgv9u/VNIpAn/w6ehqqCdSPizzyTSiXyj876af6pRTKT7pC/94ubP/XbfwyZ1G94J/8nqeLPugppzMXxmjvvvwp8TMlK7pKObNw9rbLy5uOz5N/5mZP+kdPJvDCMsUuf6s0MTbkt/ty8k0uB//9VfnUVDVhPdRLqGz8NovrQ71/DDQO0/ERQTadN2wsrMF32ZL2S86k9bF/5jGPufPltzLobPzM+6Gm8JG2J0K8PNu1v+7NVfjzayknYa2b8zNteJiDTsmb75wX9SJ+WHRNoz8/SL8fTIVzeRVn+FXRV9j5S21PxPGMVEGvSAsLLlZ8a2p1q2CP0v9MdYWXejUXMuVonUzXhLWHSRyefcvHtoI2Pn5QfPtmlQmf2j+6PrRUQy8vv/orth/PrHP0pqeGbVnAl3/gQiecfAYuwqk0gZe42pM2r+qUYxkYiPY0e9VJnRYFgDf+qi8J/NPz1cEgrmV3Uuhs/MiQ0Lf+5+whAGd5Fjihr/8rXbbT7xv5KZ/Z9/+dmKyXWSftA+vKIiN/Q3+Is0/bdHX3iqEVzaeQfhBt0kUv7bzZoWH24R/qd2qVoi+YizFEqaP/fikaK36r5eEP7TNCkpqSoma+hcDJ+ZDxs/3UpU266O4VRQdn1OBSmPZiKRwldp1v19eJHXLcAzNp9TQSCSolzIwq/TTKRJ+V63AI8WMQHVQjORluTi12kmksIjVrWIZKYYmonUhxAoTTORXAZfFMnI6163QD80E4n0Na6XSJcVjkIKV3b20UukStJ0cr1EIl2kesyZz71ugYboJdLJ2YSVeonEkuPJI4bG0/MdWegl0tzjhJXU4CebxyskksK3SK6SIiUqeonUkxSyMK2U8u5Z89QRqQQXC8l7js7xugU6opdIxC7jI5RO24rWu9QRacdGr1uAZaDjROaJjFYilfclrs4aQzgFAifbzs1RR6Rh6p6tcGXnBK1Eoo2qOZne4uP3m6L5IGVDjkIi2XocezO/9v+VBzm3JJZvFnIu8HRtgNti0j2u3mgl0ue0wf1lhXm5OXS8F8lP7RkxM9mcfLCH4PR9/VwkMkdizgEVv0+oFBep9JKZVhfMrxATKhQWyX/N3PZNo82vblBCyKWY128QG57VzzFYQxXmH99hKofEdIXSIhX36jdjJpYB3S06qCvSso6T8Xsysd1m4pujusorVI6Gh6A0zfTiO9KDQK1RWqT214ir77aKDcOjrEibKCHsMkhZYiui+1hS9ErM+nVUrjHPErSLRmWRDs+jbLA+9ipHWZE6U9b7STcPx+ZFvdyxxm1rpBI9XaQvfoaz3qgs0hxayu+CjJg8LWwi5coXidq30LMMn3ImJpCkX+FBEQiie9N3r/SoGaJRWaTPL1I2eNCvsCZRS3hBUCS6SV4E0aeOA+xViE850ydmWF6fEn4NE090R3+lXt8C7OgtUrvc6kQtVadfWWHIJBp5KopU0zRr22LPva+/5NYu8TwYEv26u+Dee6/QW6RWNT8xNT9JQZOoyM+PRBepek8QkseKFNDpjn3nhujXa3Z60w7RxINIpou1srJCBuTnvmQUKdwPEtu2y5YxhGnqji+ykHkz+vXjOE3xHBcimU+/MhZEt9yCG5HW7Ind+Mhcbg0TjmUsVJwmMIs7kdTEjUhD7lm21mikjUWkLw570QzhgEhSYBYpx7oniH6uIdrMYb1jCU9eNAS1nfboIlJB0+bvPjKMC+8nN6u9QdBSJOSe2BSJe1SFSNqYLabRsXwSGG3cYVnU1XWhKqKLSNu/MvpvNYysOUbf2iQOWoqE3BOCSMjsL66v7dBpYxY8+/Ya3ydJSUlv8EpgNNTaLzLJVYB+VdFFJMMob3I3qM6Lzf9cO8BOS5GQe0IQafc6RHmj3EaeQ6eNMSqWvdbE7/P5KnklMELMu7qMz8yjMdqIdOuj0ECZ4SuMMbWPI/UUCbUnBJEmXkGUd8ltVDx02hijYMRzyZEt+CQwQg1Vj8trO11EKvog/DwiNceYWTuQWkuRkHtCEKk7srvY7bUdOm3MuGe6nDIil3ZcEhhdmYBYGJeTknQRafAf3ntv0+EWee+3aVY7c1lLkZB7QhAJPThtwgV3DUKnjTlqCsTEJYHRyn2Ihcey3RarILqIhERLkTAb4EQq74fc/kYmn4YJZsAj1NJ4vLYDkaTgXKSj89Fv0OOZLDrGS/84nJQEIknBuUi44LLTvufSMLEE0NelXy2X3A4JgEhScC5SH8y0gzvDuDRMLOfQD44rFc4N5RQQSQrORcJOhOvEoVmiWYQZV5eibgYBp4BIUnAsUgA7SX3utzwaJpY+mKOybjt6ucaASFJwLNKlKbh3PBzEo2FiwV3CxeGkJJVFmnGessG91rqIRA9+ghFp1V7sW7ooP7GnEvv9oX7b7aKySJu2UTY40k8XkaiTw7thREp/gH3Lkq85NEwox7ETEOcfktkOGagsUlkbP3mDTut1ESmTEj1+XwpGJELQnVL0o1qFmH0St+Z+usRmSEFlkYwjbfffvYej6HCnCTm6iOTrOvcadk/uXZn2z132RTK6kNKuqUAqvoF6PE+2gdIiGY8Wpvfp3BZD/6g8LWqLZBgHxw3ug9uTLtk1exIjUuEIQpGreQXkOWQOUHLQ5Sg+E4THRVPOcqtFDdQWyUYMYtVFCu0Kw57EiBQbzCq6RGpfICNRAZUD7R9zKraMcO15fSynSlQBRJKFI5HGEdMI9ODzXLMkerbEuXQupRpG7hLCyngbuAoiycKRSOSxNFv4JKJdERPvayynOD9ZpIvEETcJKzUERJKFI5HIkbJ9fFIldY/pE/C14hNWuAfpYdGJmVzqUAYQSRaMmTKi9qSMMkW1F4/bGWvmsuNjOBRL+zmNs5zPGohk+/RTEyciHaFkn969gryeiR1rLYvSaQl1WCgZSFzdn3euWm9RXyQGk7zIL2Ebpu+EmJQz2ZRJR34eX+u9Sy2LylpTHoWz8NVq4uq9y9xXoRCKixQxiSFPiw4i0fck5iuhJ+2RK4cU5MjcFgc4RKAk9zjqli+NhvoiqZmnxT5lTLsSnWiMeq59s9B1uw4h57L3vuS6YNqvZWpcTUpSXiSmRC1eJJiwDWvKmdp34Ccj1eD+2m7wXdTSRx1dF0ybBrthq+sqFEJ1kVTN0+IIu3tyIYta5CDX2XAxT0Y3u/2tK6LNhS+Lq0lJ6ouUwKzYT93EdYy4C7iZg91uuyt422baFt3iaVISiKQwgxiyLrsdajM2D7Oi0GXJI6jRJRcedFeDUoBICsPSrzX8hrs68LosWeOqYHqkoPuDXVWgFiCSwrCIdHaqqyoKCPM0OlpTBdqAIeRWPA1cBZHUpdCS7Q6FuylyswhTd2+46Q24xTDMaGocTUoCkdRl+yaWrUbnu6mjM2nlTGu6PWbW5dC3uT7aefmqASKpyxim258rbvJ2laSR1gaSkTHwmRh0n75NPE04B5HUhTGwr5uzcRk+2leIvHTHJTM1fiR5FJFOgEjqwjgYbYqLnKxdK8nrxzsNm4UJnx/D99MdFq8eIJKyPCZedtVyK8NxFRW0MUiVrRwOiMNHiI2CGu9PG0AkZSGGPDDjPJz+tvW0LY47TGi27ADTZmksd1JaACIpy8zTrBtSok/i6WWdihTLUGd5mPqxzd39+kv6NnoAIilLT8r9Sw330h3WEGAYO17WmrUZUTB2lPjjJlMSiKQshNv1C4N69zXxrvlF397MMcFzF6CWBpb2NBf3r+SowscyDXfwY2++rg8nNL2nvjMrQCRV8ePDP24h3lqUT5/IWMXgIlS9nYm5C/La5DOUfGoWZsUh4qDyysXaDr8DkVTlLLZr+El7ylvHYoPXR4N8AjWTMnejlGXK3/zv0MsDtGAQC10MpvAUEElVln6DW7OLFMc4xN2hTBMezyO7qKkPgNLu00vvhYmMdxr3S1WNL1WzyZrVgEiqMgA7GWkp9SlpJ9TEdQujr6CWUgOKTzxBLxzXh7DXGvorhs5MTVcPEElV8L8MdJHaMgWFQc9ioIt0gFp6BW7cOF2ktsgoMOoDIqmKG5E+qQ7uRTLpNnqWBl2kDajIYVF8i+wONFhEYmq6goBIilKAH1JAF+kjlsCZM9BdEnSRlqFiWUYXfQazgi7SR8hImeoDIinKVvwjFWaRyKGcMaPGWUUiFZ6C65tjFkmLKNRmQCRFGYV/3sJHpJIB6OXMIuXkYgvHjlcAkQDJEHqh+Yi0dB96OQeR8OPWQSRAMsJF6o4ZQ8dBpP3Lce8FkQC5lBJyophFOtuo2Tv3DKOgafN3TdPCGc7GCpwwpuWRwiP/VEMXaVI+rulmkSLFzmnS5IXZtpquJCCSmnyzFL/OLNKJfKNz8Bpt+1dGf1PvBMPZiE2baRIpUnjkn2roIuGHdJtFqim24gPTnAsQCeDJdML88ahLu3MNPwxH/i1vYgqGz3A29nqCW2EtvLqOKniJVFPsVPOVIIgE8CSVMLgz5h4pLfTjdesj8zBR+tkYwJ7s0Zd8aUvN/4ShivQgHdv0mHukcLGBP5tjgINIAE9IQ0fNImXsNabOMIyiD6KShNPPxm+wSTVNIkUKj/xTDVWkXfgJ7GaRqos9GOU0iARwpJLUd2YWKf/tZk2LD7cY/If33jOFk6SfjQOx8/NMNUcKD/9Tu5QqUuZN5OIQZpEipRvzppm3AJEAjpyeSVjJo/sbH3bbffc3Yfo4dH8DUlmSS1jJQaTz+BRmIJIjQCQlSSOF9+EgUuZV7Htdi3SHEGcPRAKkQpymykEkQphj1yJt2o5/L4gESCWVtNK9SDcICWNcizSUkNYWRAJkcoOYXMi9SJ+fwr/XtUikUHUgEiAT0tURD5FIWZFci0T6NQWRAJlkEBMZb6Tl8Cr9lHw2PiCMiDVocfWN4euJIl0dT3jv0cW00kEkgB/kkFgFgyhvX51JPhuXkELXDb1OKb1dDlGkVaSUS49pQZIP9gaRAG5QYstl4kZuV3Hqb7vIZ2NXUpTGopbEoMSB4WPJIqUVIxbWMGceaa1x5f/bDiIBvCihBe6d3y0qgHY03dtszyGejWXk26CCnj3xhfd+f3IOWSRKWPy1XQhNT/nXJnLT1QVEUpCvV1A3KczLzaGDPhs305I8lzGWjio8QOy5D5fuounqAiIpSNYF6iZuROqJm4pUjRuRzk+zLospHUQCVOMA69l4YyDqei11HNIpWyLdTidcCfYcaQ28DCIBynGI8Wy80A493OBUK1RmPTsiXWlNytNiXGzx0FI6iASoBqtInXBXc+dQAV3tiNQVG+u/iuuWvnoQCVAORpEqe2NL6I5Io8ImUm5YJGrfgiVPS1AkltJBJEAeZJFqAmg/HIotoRcijUpIJPq5Hi7c14/WREsFTJpC7G9AJiSRTCkdSCLlVedRqTUpeKoHTSKf7ZFsFAwixVYQ1pRWODn+v4KASFpzPY9M5OQliZRrTdJSVlhIKbcmPxKDSLEVsJYO+ZEAaTwoJBM5GUkioS6lyijF1pbOIFJsBWVspevlEYikN+VlFKo2o4sUc3NPK7amdGaRTBUwF64TIFIiYFskZpyIFJeASImACiLlgEiA7oBIwgGRVOLyKz9sdN8wtuSZF36c5HNbrlmkmCQtHEVCZ4EBkQDxFDd/7rf7Hjap2/BO+E9Wx5t1FxjGgmffXuP7JCkp6Y3gJlv/xFekmCQtHEVCZ4EBkQDxZL7oy3wh41V/2rrwH8PY//TZ4OKKZa818ft8vkrDKP3dPr4ixSRp4Xlph8wCAyIB4mn5mbHtqZYtQv8L/TFW1g3PIi8Y8VxyZIvUSfmcRYpO0sL3HgmRBQZEAsQz6qXKjAbDGvhTF4X/bP7p4ZJywxj3TJdTRuTS7tc//lFSQ7fVmEWKSdLCUSRcKmqCAAAHQUlEQVR0FhgQCRBPSfPnXjxS9Fbd1wvCf5oG5RlgGEejIn9z/kWKSdLCUSR0FhgQCYgboPtbOCBSIgAiCQdESgRAJOGASIkAiCQcECkRAJGEAyIlAiCScECkRABEEg6IlAj4+mJXuUyjEqBmgekJIgFxQ4cKzIrLrdyJZHSmxD++8zGIBMQNJ7ug06RfbbHRpUjn2hPTuNxuuQZEAuKHC72Qsb9rPHI+Ezy/Nyn297/W5oBIgFSOzDO/6kFLnGeXskJiLDkXIRXKCvNIJYNIgFSG3DW/etCBc/EgklBAJGXoGv1yJT3bmC1AJKGASKpwcVLMgk5FXMsHkYQCIqnCuLyYBbeoiR5s4bVIENcOkEIXy5K5W3mWT0kC4U6kOMzTYhMQSREKhluXtaUk8bIF+XR3k0YlPvO02AREUoTsY9Zl+QM4VhDJ1YLDRRoVSsnuK9ABEEkROqMWTvqaXwW0bCrO0z/YyAITv4BIalDSH7XU35oyks0OtGwqzk9z1iww/PZFPUAkNVixB7n4TAbHOsSlUYnHPC02AZHUoDsm5NaIo3LbATgERFKCCtwzo/LWlVIbAjgERFKCHWtxa3JjBzwASgIiKUFv9HShEAMuSGwH4BQQSQUC3fDrStsE8CsBVQCRVODQfMLKnNnS2gE4BkRSgcF3SWtTrslqB+AYEEkFuhLXPmgvqRmAc0AkBbgwhbx+/XI57QCcAyIpwNjYqUixdLgjpR2Ac0AkBSBf2QW5lSKjGYALQCTvKRhB3WTeFgntAFwAInnPrOP0bdoSwzACngMieU8n9OITa5bXMu0fphfLV30LD2kVA0TynJI01NJrbWbtwPNFq3OymwkQAZE8Z9lexEJ/K3K8hnKeU/4A94BIntMVNVHiK1p4yL3wbEkpQCSvqUBmGMo+Q3lb0YDI/O04n3mqCyCS12xbj1o6nTZ54mGP6pAi8R0LQRdAJK/phZyKRBepXW2UKzDJe0Akjwl0Ry6mi9SyOu5ifMeL0wUQyWNyFyAXs4oU9xFMdQFE8pjB6JwTzCLFe0xtXQCRPMYaOz8MiKQZIJK3nJ+KXg4iaQaI5C2jr6KXm0U626jZO/cMozKzrmkDEEktQCRvwU1FMot0It/ovM8wbu77b9MGIJJagEiecnskZkXUpd25hh+GR3uDSOoCInnKjJOYFTH3SGlLQ39BJHUBkTwF02cXLVLGXmPqjNB/QCR1AZG8pASbks8sUv7bzZoWH25x4L3/573acXkgklqASF6ydB9uDXR/awaI5CXdsDlbQCTNAJE8pKIXdhWIpBkgkods2YhdBSJpBojkIb3wcRdAJM0Akbwj0AO/DkTSDBDJO75ZhF8HImkGiOQdA+/h19GDn7QBkVQCRPIOUuz8PSspb943GERSCRDJM85nEVb6Wz0ivrmi9RYQSSVAJM/IxExFquJq6zkHjuA4uLDVFzkgkkqASJ6BHbAaIXd+1sh0NEOm78oBkZQCRPKKG7ipSNWUFebl5tABkZQARJLHwTkzTbRPN79aas1rDiLpBIgkiwdtF5y5hOVgT0t8OxBJJ0AkWaTcIq8f823MAhBJJ0AkSTwYRNngcc+yKqoXgEg6ASJJ4uQ82hadCqMTtYREopsEIYvVAESSxPH5tC3a5lUnaqkWicEkCKKvCCCSJOgifVKTp6XGpLy8XBqQ1kUNQCRJ0EX6qPqeJ6/2J6kwjwYkGlMDEEkSzCKZeg/KygrpQO5LJQCRJOFEJEAfQCRJgEjxDYgkCRApvgGRJGEWqaBp83cfGcbVd5olm7YAkXQGRJKEWaTtXxn9txpG/01Gnz21S0EknQGRJBF9aVfe5K5hZCw2+nxeuwxE0hkQSRJRIt366Lvg3wfJrT6dW7sQRNIZEEkSZpGKPrgZ+ufiGaOFKT8SiKQzIJIkzCIN/sN772063OLanz8YbNoCRNIZEEkS0P0d34BIkgCR4hsQSRIgUnwDIkkCRIpvQCRJgEjxDYgkCRApvgGRJAEixTcgkiROz6ZtASLpDIgkice9KRsUfgYiaQyIJIthR4ir/T2+BJE0BkSShW9I2uLlWKa3zsoBkTQGRJLHw9z182dgWAB5WvQGRJJIGcQgjltAJImASPELiCQRECl+AZEkAiLFLyCSRECk+AVEkghbxiMQSUdAJImwZDyCPC16AiJJJJSnhZqopTatC6ARIJJEQtklIE9LfAIiyaSMEa/bCdgGRAIADoBIAMABEAkAOAAiAQAHQCQA4ACIBAAcAJEAgAMgEgBwAEQCAA6ASADAARAJADgAIgEAB0AkAOAAiAQAHACRAIADIBIAcABEAgAOgEgAwAEQCQA4ACIBAAdAJADgAIgEABwAkQCAAyASAHAARAIADoBIAMABEAkAOAAiAQAHQCQA4ACIBAAcAJEAgAMgEgBwAEQCAA6ASADAARAJADgAIgEAB0AkAOAAiAQAHACRAIADIBIAcABEAgAOgEgAwIH/H58xaVIk+Vd6AAAAAElFTkSuQmCC", "text/plain": [ "plot without title" ] }, "metadata": { "image/png": { "height": 420, "width": 420 } }, "output_type": "display_data" } ], "source": [ "fit.prunedtree <- prune(fit.trees,cp=bestcp)\n", "prp(fit.prunedtree,leaf.round=1, space=3, yspace=3, split.space=7, shadow.col = \"gray\",trace = 1,yesno=1)" ] }, { "cell_type": "markdown", "metadata": { "id": "pYAym1xTWit9", "papermill": { "duration": 0.051023, "end_time": "2021-07-22T21:33:47.820426", "exception": false, "start_time": "2021-07-22T21:33:47.769403", "status": "completed" }, "tags": [] }, "source": [ "E.g., in the pruned tree the predicted hourly log wage for high-school graduates with more than $9.5$ years of experience is $2.8$, and otherwise is $2.6$." ] }, { "cell_type": "markdown", "metadata": { "id": "Leso28bjWit9", "papermill": { "duration": 0.050261, "end_time": "2021-07-22T21:33:47.921859", "exception": false, "start_time": "2021-07-22T21:33:47.871598", "status": "completed" }, "tags": [] }, "source": [ "Finally, we calculate the mean-squared error and the $R^2$ on the test sample to evaluate the out-of-sample performance of the pruned tree." ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "execution": { "iopub.execute_input": "2021-07-22T21:33:48.028361Z", "iopub.status.busy": "2021-07-22T21:33:48.027097Z", "iopub.status.idle": "2021-07-22T21:33:48.049467Z", "shell.execute_reply": "2021-07-22T21:33:48.048085Z" }, "executionInfo": { "elapsed": 15, "status": "ok", "timestamp": 1658250116117, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "-GdlHf6sWit9", "outputId": "395779d8-1795-4775-b811-b457792799d9", "papermill": { "duration": 0.077314, "end_time": "2021-07-22T21:33:48.049633", "exception": false, "start_time": "2021-07-22T21:33:47.972319", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "R^2 of the pruned tree: 0.2250066" ] } ], "source": [ "yhat.pt <- predict(fit.prunedtree,newdata=data_test)\n", "MSE.pt <- summary(lm((Y_test-yhat.pt)^2~1))$coef[1:2]\n", "R2.pt <- 1-MSE.pt[1]/var(Y_test)\n", "\n", "# R^2 of the pruned tree\n", "cat(\"R^2 of the pruned tree:\",R2.pt)" ] }, { "cell_type": "markdown", "metadata": { "id": "FRT62uP-Wit-", "papermill": { "duration": 0.050968, "end_time": "2021-07-22T21:33:48.152253", "exception": false, "start_time": "2021-07-22T21:33:48.101285", "status": "completed" }, "tags": [] }, "source": [ "### Random Forest and Boosted Trees" ] }, { "cell_type": "markdown", "metadata": { "id": "0u9KdQZrWit-", "papermill": { "duration": 0.050209, "end_time": "2021-07-22T21:33:48.253970", "exception": false, "start_time": "2021-07-22T21:33:48.203761", "status": "completed" }, "tags": [] }, "source": [ "In the next step, we apply the more advanced tree-based methods: random forest and boosted trees." ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "execution": { "iopub.execute_input": "2021-07-22T21:33:48.360982Z", "iopub.status.busy": "2021-07-22T21:33:48.360463Z", "iopub.status.idle": "2021-07-22T21:34:43.865598Z", "shell.execute_reply": "2021-07-22T21:34:43.864364Z" }, "executionInfo": { "elapsed": 37648, "status": "ok", "timestamp": 1658250153754, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "hAvYz1zbWit-", "outputId": "6d7ad332-f3f6-461c-8547-50d0d6947d38", "papermill": { "duration": 55.561432, "end_time": "2021-07-22T21:34:43.865807", "exception": false, "start_time": "2021-07-22T21:33:48.304375", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "OOB generally underestimates the optimal number of iterations although predictive performance is reasonably competitive. Using cv_folds>1 when calling gbm usually results in improved predictive performance.\n", "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "R^2 of the random forest and boosted trees: 0.26582 0.2675069" ] } ], "source": [ "## Applying the methods\n", "# random forest\n", "fit.rf <- randomForest(formula_basic, ntree=2000, nodesize=5, data=data_train)\n", "# for tuning: adjust input \"mtry\" to change the number of variables randomly sampled as candidates at each split\n", "\n", "# boosting\n", "fit.boost <- gbm(formula_basic, data=data_train, distribution= \"gaussian\", bag.fraction = .5, interaction.depth=2, n.trees=1000, shrinkage=.01)\n", "best.boost <- gbm.perf(fit.boost, plot.it = FALSE) # cross-validation to determine when to stop\n", "\n", "## Evaluating the methods\n", "yhat.rf <- predict(fit.rf, newdata=data_test) # prediction\n", "yhat.boost <- predict(fit.boost, newdata=data_test, n.trees=best.boost)\n", "\n", "MSE.rf = summary(lm((Y_test-yhat.rf)^2~1))$coef[1:2]\n", "MSE.boost = summary(lm((Y_test-yhat.boost)^2~1))$coef[1:2]\n", "R2.rf <- 1-MSE.rf[1]/var(Y_test)\n", "R2.boost <- 1-MSE.boost[1]/var(Y_test)\n", "\n", "# printing R^2\n", "cat(\"R^2 of the random forest and boosted trees:\",R2.rf,R2.boost)\n" ] }, { "cell_type": "markdown", "metadata": { "id": "Y3SItRnPWit_", "papermill": { "duration": 0.050943, "end_time": "2021-07-22T21:34:43.969202", "exception": false, "start_time": "2021-07-22T21:34:43.918259", "status": "completed" }, "tags": [] }, "source": [ "To conclude, let us have a look at our results." ] }, { "cell_type": "markdown", "metadata": { "id": "Sj1p6gTaWit_", "papermill": { "duration": 0.051543, "end_time": "2021-07-22T21:34:44.072428", "exception": false, "start_time": "2021-07-22T21:34:44.020885", "status": "completed" }, "tags": [] }, "source": [ "## Results" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 537 }, "execution": { "iopub.execute_input": "2021-07-22T21:34:44.181135Z", "iopub.status.busy": "2021-07-22T21:34:44.179921Z", "iopub.status.idle": "2021-07-22T21:34:44.286892Z", "shell.execute_reply": "2021-07-22T21:34:44.285724Z" }, "executionInfo": { "elapsed": 17, "status": "ok", "timestamp": 1658250153756, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "P_ijvjVPWit_", "outputId": "cf17f27c-ebbe-49de-d476-0be22b447326", "papermill": { "duration": 0.163242, "end_time": "2021-07-22T21:34:44.287033", "exception": false, "start_time": "2021-07-22T21:34:44.123791", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\t\n", "\n", "\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\n", "
A matrix: 15 × 3 of type dbl
MSES.E. for MSER-squared
Least Squares (basic)0.24962820.015584520.2185201
Least Squares (flexible)0.25025380.015578960.2165618
Lasso0.24966250.015111490.2184129
Post-Lasso0.24851370.015374090.2220093
Lasso (flexible)0.25020690.015030370.2167083
Post-Lasso (flexible)0.24851370.015374090.2220093
Cross-Validated lasso0.24904490.015192970.2203463
Cross-Validated ridge0.25376300.015360350.2055758
Cross-Validated elnet0.24937940.015206800.2192991
Cross-Validated lasso (flexible)0.24910680.015128940.2201523
Cross-Validated ridge (flexible)0.25355700.015356790.2062207
Cross-Validated elnet (flexible)0.24879620.015158420.2211247
Random Forest0.23451920.015500910.2658200
Boosted Trees0.23398040.015028810.2675069
Pruned Tree0.24755620.015485390.2250066
\n" ], "text/latex": [ "A matrix: 15 × 3 of type dbl\n", "\\begin{tabular}{r|lll}\n", " & MSE & S.E. for MSE & R-squared\\\\\n", "\\hline\n", "\tLeast Squares (basic) & 0.2496282 & 0.01558452 & 0.2185201\\\\\n", "\tLeast Squares (flexible) & 0.2502538 & 0.01557896 & 0.2165618\\\\\n", "\tLasso & 0.2496625 & 0.01511149 & 0.2184129\\\\\n", "\tPost-Lasso & 0.2485137 & 0.01537409 & 0.2220093\\\\\n", "\tLasso (flexible) & 0.2502069 & 0.01503037 & 0.2167083\\\\\n", "\tPost-Lasso (flexible) & 0.2485137 & 0.01537409 & 0.2220093\\\\\n", "\tCross-Validated lasso & 0.2490449 & 0.01519297 & 0.2203463\\\\\n", "\tCross-Validated ridge & 0.2537630 & 0.01536035 & 0.2055758\\\\\n", "\tCross-Validated elnet & 0.2493794 & 0.01520680 & 0.2192991\\\\\n", "\tCross-Validated lasso (flexible) & 0.2491068 & 0.01512894 & 0.2201523\\\\\n", "\tCross-Validated ridge (flexible) & 0.2535570 & 0.01535679 & 0.2062207\\\\\n", "\tCross-Validated elnet (flexible) & 0.2487962 & 0.01515842 & 0.2211247\\\\\n", "\tRandom Forest & 0.2345192 & 0.01550091 & 0.2658200\\\\\n", "\tBoosted Trees & 0.2339804 & 0.01502881 & 0.2675069\\\\\n", "\tPruned Tree & 0.2475562 & 0.01548539 & 0.2250066\\\\\n", "\\end{tabular}\n" ], "text/markdown": [ "\n", "A matrix: 15 × 3 of type dbl\n", "\n", "| | MSE | S.E. for MSE | R-squared |\n", "|---|---|---|---|\n", "| Least Squares (basic) | 0.2496282 | 0.01558452 | 0.2185201 |\n", "| Least Squares (flexible) | 0.2502538 | 0.01557896 | 0.2165618 |\n", "| Lasso | 0.2496625 | 0.01511149 | 0.2184129 |\n", "| Post-Lasso | 0.2485137 | 0.01537409 | 0.2220093 |\n", "| Lasso (flexible) | 0.2502069 | 0.01503037 | 0.2167083 |\n", "| Post-Lasso (flexible) | 0.2485137 | 0.01537409 | 0.2220093 |\n", "| Cross-Validated lasso | 0.2490449 | 0.01519297 | 0.2203463 |\n", "| Cross-Validated ridge | 0.2537630 | 0.01536035 | 0.2055758 |\n", "| Cross-Validated elnet | 0.2493794 | 0.01520680 | 0.2192991 |\n", "| Cross-Validated lasso (flexible) | 0.2491068 | 0.01512894 | 0.2201523 |\n", "| Cross-Validated ridge (flexible) | 0.2535570 | 0.01535679 | 0.2062207 |\n", "| Cross-Validated elnet (flexible) | 0.2487962 | 0.01515842 | 0.2211247 |\n", "| Random Forest | 0.2345192 | 0.01550091 | 0.2658200 |\n", "| Boosted Trees | 0.2339804 | 0.01502881 | 0.2675069 |\n", "| Pruned Tree | 0.2475562 | 0.01548539 | 0.2250066 |\n", "\n" ], "text/plain": [ " MSE S.E. for MSE R-squared\n", "Least Squares (basic) 0.2496282 0.01558452 0.2185201\n", "Least Squares (flexible) 0.2502538 0.01557896 0.2165618\n", "Lasso 0.2496625 0.01511149 0.2184129\n", "Post-Lasso 0.2485137 0.01537409 0.2220093\n", "Lasso (flexible) 0.2502069 0.01503037 0.2167083\n", "Post-Lasso (flexible) 0.2485137 0.01537409 0.2220093\n", "Cross-Validated lasso 0.2490449 0.01519297 0.2203463\n", "Cross-Validated ridge 0.2537630 0.01536035 0.2055758\n", "Cross-Validated elnet 0.2493794 0.01520680 0.2192991\n", "Cross-Validated lasso (flexible) 0.2491068 0.01512894 0.2201523\n", "Cross-Validated ridge (flexible) 0.2535570 0.01535679 0.2062207\n", "Cross-Validated elnet (flexible) 0.2487962 0.01515842 0.2211247\n", "Random Forest 0.2345192 0.01550091 0.2658200\n", "Boosted Trees 0.2339804 0.01502881 0.2675069\n", "Pruned Tree 0.2475562 0.01548539 0.2250066" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# library(xtable)\n", "table<- matrix(0, 15, 3)\n", "table[1,1:2] <- MSE_lm_basic\n", "table[2,1:2] <- MSE_lm_flex\n", "table[3,1:2] <- MSE.lasso\n", "table[4,1:2] <- MSE.lasso.post\n", "table[5,1:2] <- MSE.lasso.flex\n", "table[6,1:2] <- MSE.lasso.post.flex\n", "table[7,1:2] <- MSE.lasso.cv\n", "table[8,1:2] <- MSE.ridge\n", "table[9,1:2] <- MSE.elnet\n", "table[10,1:2] <- MSE.lasso.cv.flex\n", "table[11,1:2] <- MSE.ridge.flex\n", "table[12,1:2] <- MSE.elnet.flex\n", "table[13,1:2] <- MSE.rf\n", "table[14,1:2] <- MSE.boost\n", "table[15,1:2] <- MSE.pt\n", "\n", "table[1,3] <- R2_lm_basic\n", "table[2,3] <- R2_lm_flex\n", "table[3,3] <- R2.lasso\n", "table[4,3] <- R2.lasso.post\n", "table[5,3] <- R2.lasso.flex\n", "table[6,3] <- R2.lasso.post.flex\n", "table[7,3] <- R2.lasso.cv\n", "table[8,3] <- R2.ridge\n", "table[9,3] <- R2.elnet\n", "table[10,3] <- R2.lasso.cv.flex\n", "table[11,3] <- R2.ridge.flex\n", "table[12,3] <- R2.elnet.flex\n", "table[13,3] <- R2.rf\n", "table[14,3] <- R2.boost\n", "table[15,3] <- R2.pt\n", "\n", "colnames(table)<- c(\"MSE\", \"S.E. for MSE\", \"R-squared\")\n", "rownames(table)<- c(\"Least Squares (basic)\",\"Least Squares (flexible)\", \"Lasso\", \"Post-Lasso\",\"Lasso (flexible)\",\"Post-Lasso (flexible)\", \n", " \"Cross-Validated lasso\", \"Cross-Validated ridge\",\"Cross-Validated elnet\",\"Cross-Validated lasso (flexible)\",\"Cross-Validated ridge (flexible)\",\"Cross-Validated elnet (flexible)\", \n", " \"Random Forest\",\"Boosted Trees\", \"Pruned Tree\")\n", "table" ] }, { "cell_type": "markdown", "metadata": { "id": "TpqFypSbWit_", "papermill": { "duration": 0.052757, "end_time": "2021-07-22T21:34:44.392711", "exception": false, "start_time": "2021-07-22T21:34:44.339954", "status": "completed" }, "tags": [] }, "source": [ "Above, we have displayed the results for a single split of data into the training and testing part. The table shows the test MSE in column 1 as well as the standard error in column 2 and the test $R^2$ in column 3. \n", "\n", "We see that the prediction rule produced by the Elastic Net using the flexible model performs the best here, giving the lowest test MSE. Cross-Validated Lasso and Ridge, perform nearly as well. For any two of these methods, their testing MSEs are within one standard error of each other. Remarkably, OLS on a simple model performs extremely well, almost as well as best tree based method Random Forest. On the other hand, OLS on a flexible model with many regressors performs very poorly giving the highest test MSE. Notice that the nonlinear models, e.g. Random Forest, are not tuned. Thus, there is a lot of potential to improve the performance of the nonlinear methods we used in the analysis." ] }, { "cell_type": "markdown", "metadata": { "id": "YFol0C-XWiuA", "papermill": { "duration": 0.052773, "end_time": "2021-07-22T21:34:44.497668", "exception": false, "start_time": "2021-07-22T21:34:44.444895", "status": "completed" }, "tags": [] }, "source": [ "## Ensemble learning" ] }, { "cell_type": "markdown", "metadata": { "id": "P40FBHxRWiuA", "papermill": { "duration": 0.052345, "end_time": "2021-07-22T21:34:44.603135", "exception": false, "start_time": "2021-07-22T21:34:44.550790", "status": "completed" }, "tags": [] }, "source": [ "In the final step, we can build a prediction model by combining the strengths of the models we considered so far. This ensemble method is of the form\n", "\n", "$$ f(x) = \\sum_{k=1}^K \\alpha_k f_k(x) $$\n", " \n", "where the $f_k$'s denote our prediction rules from the table above and the $\\alpha_k$'s are the corresponding weights." ] }, { "cell_type": "markdown", "metadata": { "id": "nYVckjqeWiuA", "papermill": { "duration": 0.052659, "end_time": "2021-07-22T21:34:44.710467", "exception": false, "start_time": "2021-07-22T21:34:44.657808", "status": "completed" }, "tags": [] }, "source": [ "We focus on the prediction rules based on OLS, Post-Lasso, Elastic Net, Pruned Tree, Random Forest, Boosted Trees, and Neural Network and combine these methods into an ensemble method. The appropriate weights can be determined by a simple ols regression:" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 434 }, "execution": { "iopub.execute_input": "2021-07-22T21:34:44.820377Z", "iopub.status.busy": "2021-07-22T21:34:44.819096Z", "iopub.status.idle": "2021-07-22T21:34:44.841481Z", "shell.execute_reply": "2021-07-22T21:34:44.840197Z" }, "executionInfo": { "elapsed": 621, "status": "ok", "timestamp": 1658250154361, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "Xs9ndCggWiuA", "outputId": "3c5c35fc-7a9d-4882-daa1-27b50ffda431", "papermill": { "duration": 0.078475, "end_time": "2021-07-22T21:34:44.841619", "exception": false, "start_time": "2021-07-22T21:34:44.763144", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "data": { "text/plain": [ "\n", "Call:\n", "lm(formula = Y_test ~ yhat_lm_basic + yhat.rlasso.post.flex + \n", " yhat.elnet.flex + yhat.pt + yhat.rf + yhat.boost)\n", "\n", "Residuals:\n", " Min 1Q Median 3Q Max \n", "-1.7261 -0.2821 -0.0141 0.2717 3.6299 \n", "\n", "Coefficients:\n", " Estimate Std. Error t value Pr(>|t|) \n", "(Intercept) -0.13636 0.28142 -0.485 0.62809 \n", "yhat_lm_basic -0.05963 0.18749 -0.318 0.75050 \n", "yhat.rlasso.post.flex 0.24141 0.41866 0.577 0.56429 \n", "yhat.elnet.flex -0.18802 0.57863 -0.325 0.74529 \n", "yhat.pt 0.01050 0.10543 0.100 0.92070 \n", "yhat.rf 0.48543 0.09208 5.272 1.58e-07 ***\n", "yhat.boost 0.55427 0.17909 3.095 0.00201 ** \n", "---\n", "Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1\n", "\n", "Residual standard error: 0.4789 on 1281 degrees of freedom\n", "Multiple R-squared: 0.2855,\tAdjusted R-squared: 0.2821 \n", "F-statistic: 85.31 on 6 and 1281 DF, p-value: < 2.2e-16\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "ensemble.ols <- summary(lm(Y_test~ yhat_lm_basic + yhat.rlasso.post.flex + yhat.elnet.flex+ yhat.pt+ yhat.rf + yhat.boost))\n", "ensemble.ols" ] }, { "cell_type": "markdown", "metadata": { "id": "Y07LSBYrWiuB", "papermill": { "duration": 0.053593, "end_time": "2021-07-22T21:34:44.948758", "exception": false, "start_time": "2021-07-22T21:34:44.895165", "status": "completed" }, "tags": [] }, "source": [ "Alternatively, we can determine the weights via lasso regression. " ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 712 }, "execution": { "iopub.execute_input": "2021-07-22T21:34:45.060856Z", "iopub.status.busy": "2021-07-22T21:34:45.059440Z", "iopub.status.idle": "2021-07-22T21:34:45.174027Z", "shell.execute_reply": "2021-07-22T21:34:45.172952Z" }, "executionInfo": { "elapsed": 37, "status": "ok", "timestamp": 1658250154363, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "hlm-aAX4WiuB", "outputId": "a17029d8-fa3b-4b63-bdaf-b9f89037e36b", "papermill": { "duration": 0.171986, "end_time": "2021-07-22T21:34:45.174182", "exception": false, "start_time": "2021-07-22T21:34:45.002196", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Call:\n", "rlasso.formula(formula = Y_test ~ yhat_lm_basic + yhat.rlasso.post.flex + \n", " yhat.elnet.flex + yhat.pt + yhat.rf + yhat.boost)\n", "\n", "Post-Lasso Estimation: TRUE \n", "\n", "Total number of variables: 6\n", "Number of selected variables: 2 \n", "\n", "Residuals: \n", " Min 1Q Median 3Q Max \n", "-1.72393 -0.28094 -0.01085 0.27243 3.61820 \n", "\n", " Estimate\n", "(Intercept) -0.196\n", "yhat_lm_basic 0.000\n", "yhat.rlasso.post.flex 0.000\n", "yhat.elnet.flex 0.000\n", "yhat.pt 0.000\n", "yhat.rf 0.475\n", "yhat.boost 0.589\n", "\n", "Residual standard error: 0.4779\n", "Multiple R-squared: 0.2851\n", "Adjusted R-squared: 0.284\n", "Joint significance test:\n", " the sup score statistic for joint significance test is 3.399 with a p-value of 0.058\n" ] }, { "data": { "text/plain": [ "\n", "Call:\n", "rlasso.formula(formula = Y_test ~ yhat_lm_basic + yhat.rlasso.post.flex + \n", " yhat.elnet.flex + yhat.pt + yhat.rf + yhat.boost)\n", "\n", "Coefficients:\n", " (Intercept) yhat_lm_basic yhat.rlasso.post.flex \n", " -0.1963 0.0000 0.0000 \n", " yhat.elnet.flex yhat.pt yhat.rf \n", " 0.0000 0.0000 0.4750 \n", " yhat.boost \n", " 0.5892 \n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "ensemble.lasso <- summary(rlasso(Y_test~ yhat_lm_basic + yhat.rlasso.post.flex + yhat.elnet.flex+ yhat.pt+ yhat.rf + yhat.boost))\n", "ensemble.lasso" ] }, { "cell_type": "markdown", "metadata": { "id": "sceeSDLpWiuB", "papermill": { "duration": 0.05475, "end_time": "2021-07-22T21:34:45.284948", "exception": false, "start_time": "2021-07-22T21:34:45.230198", "status": "completed" }, "tags": [] }, "source": [ "The estimated weights are shown in the following table." ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 286 }, "execution": { "iopub.execute_input": "2021-07-22T21:34:45.398344Z", "iopub.status.busy": "2021-07-22T21:34:45.397127Z", "iopub.status.idle": "2021-07-22T21:34:45.432227Z", "shell.execute_reply": "2021-07-22T21:34:45.430974Z" }, "executionInfo": { "elapsed": 34, "status": "ok", "timestamp": 1658250154365, "user": { "displayName": "Jhon Kevin Flores Rojas", "userId": "10267608749788811245" }, "user_tz": 300 }, "id": "KPAWuQiMWiuC", "outputId": "30a8ea04-e39b-4ce4-e679-c76c56f92325", "papermill": { "duration": 0.09307, "end_time": "2021-07-22T21:34:45.432382", "exception": false, "start_time": "2021-07-22T21:34:45.339312", "status": "completed" }, "tags": [], "vscode": { "languageId": "r" } }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\t\n", "\n", "\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\n", "
A matrix: 7 × 2 of type dbl
Weight OLSWeight Lasso
Constant-0.13635940-0.1963149
Least Squares (basic)-0.05963050 0.0000000
Post-Lasso (flexible) 0.24140935 0.0000000
Cross-Validated elnet (flexible)-0.18801716 0.0000000
Pruned Tree 0.01049777 0.0000000
Random Forest 0.48543389 0.4749684
Boosted Trees 0.55426895 0.5891756
\n" ], "text/latex": [ "A matrix: 7 × 2 of type dbl\n", "\\begin{tabular}{r|ll}\n", " & Weight OLS & Weight Lasso\\\\\n", "\\hline\n", "\tConstant & -0.13635940 & -0.1963149\\\\\n", "\tLeast Squares (basic) & -0.05963050 & 0.0000000\\\\\n", "\tPost-Lasso (flexible) & 0.24140935 & 0.0000000\\\\\n", "\tCross-Validated elnet (flexible) & -0.18801716 & 0.0000000\\\\\n", "\tPruned Tree & 0.01049777 & 0.0000000\\\\\n", "\tRandom Forest & 0.48543389 & 0.4749684\\\\\n", "\tBoosted Trees & 0.55426895 & 0.5891756\\\\\n", "\\end{tabular}\n" ], "text/markdown": [ "\n", "A matrix: 7 × 2 of type dbl\n", "\n", "| | Weight OLS | Weight Lasso |\n", "|---|---|---|\n", "| Constant | -0.13635940 | -0.1963149 |\n", "| Least Squares (basic) | -0.05963050 | 0.0000000 |\n", "| Post-Lasso (flexible) | 0.24140935 | 0.0000000 |\n", "| Cross-Validated elnet (flexible) | -0.18801716 | 0.0000000 |\n", "| Pruned Tree | 0.01049777 | 0.0000000 |\n", "| Random Forest | 0.48543389 | 0.4749684 |\n", "| Boosted Trees | 0.55426895 | 0.5891756 |\n", "\n" ], "text/plain": [ " Weight OLS Weight Lasso\n", "Constant -0.13635940 -0.1963149 \n", "Least Squares (basic) -0.05963050 0.0000000 \n", "Post-Lasso (flexible) 0.24140935 0.0000000 \n", "Cross-Validated elnet (flexible) -0.18801716 0.0000000 \n", "Pruned Tree 0.01049777 0.0000000 \n", "Random Forest 0.48543389 0.4749684 \n", "Boosted Trees 0.55426895 0.5891756 " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "table<- matrix(0, 7, 2)\n", "table[1:7,1] <- ensemble.ols$coef[1:7]\n", "table[1:7,2] <- ensemble.lasso$coef[1:7]\n", "\n", "colnames(table)<- c(\"Weight OLS\", \"Weight Lasso\")\n", "rownames(table)<- c(\"Constant\",\"Least Squares (basic)\",\"Post-Lasso (flexible)\", \"Cross-Validated elnet (flexible)\", \"Pruned Tree\",\n", " \"Random Forest\",\"Boosted Trees\")\n", "table" ] }, { "cell_type": "markdown", "metadata": { "id": "ijgxlqKMWiuC", "papermill": { "duration": 0.055235, "end_time": "2021-07-22T21:34:45.543519", "exception": false, "start_time": "2021-07-22T21:34:45.488284", "status": "completed" }, "tags": [] }, "source": [ "Further, the $R^2$ for the test sample improves from $30\\%$ obtained by OLS to about $31\\%$ obtained by the ensemble method. We see that it is very powerful to aggregate prediction rules into an ensemble rule. Nevertheless, it is worth noticing that we should compare the ensemble method and the single rules on an additional validation set to ensure a fair comparison." ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "08_ml-for-wage-prediction.ipynb", "provenance": [] }, "jupytext": { "formats": "ipynb,auto:light" }, "kernelspec": { "display_name": "R", "language": "R", "name": "ir" }, "language_info": { "codemirror_mode": "r", "file_extension": ".r", "mimetype": "text/x-r-source", "name": "R", "pygments_lexer": "r", "version": "4.2.1" }, "papermill": { "default_parameters": {}, "duration": 88.350056, "end_time": "2021-07-22T21:34:45.707112", "environment_variables": {}, "exception": null, "input_path": "__notebook__.ipynb", "output_path": "__notebook__.ipynb", "parameters": {}, "start_time": "2021-07-22T21:33:17.357056", "version": "2.2.2" } }, "nbformat": 4, "nbformat_minor": 4 }