\n This is Set 10: Capstone Projects (Exercises 91-100) of DatatableTon: 💯 datatable exercises\n \n You can find all the exercises and solutions on GitHub\n
","metadata":{"papermill":{"duration":0.026862,"end_time":"2021-07-30T05:45:24.207627","exception":false,"start_time":"2021-07-30T05:45:24.180765","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"**Prerequisites**\n* The `datatable` package should be upgraded to the latest version (or v1.0.0+).\n* The datasets from the [Avazu](https://www.kaggle.com/c/avazu-ctr-prediction/data) competition on kaggle will be used for the exercises and needs to be downloaded manually.\n* It is encouraged to use your own datasets for the last exercise but you can use the datasets from the [Tradeshift](https://www.kaggle.com/c/tradeshift-text-classification) competition on kaggle for practise.","metadata":{"papermill":{"duration":0.028431,"end_time":"2021-07-30T05:45:24.263005","exception":false,"start_time":"2021-07-30T05:45:24.234574","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!python3 -m pip install -U pip\n!python3 -m pip install -U datatable","metadata":{"_kg_hide-output":true,"execution":{"iopub.execute_input":"2021-07-30T05:45:24.32717Z","iopub.status.busy":"2021-07-30T05:45:24.326493Z","iopub.status.idle":"2021-07-30T05:46:00.476959Z","shell.execute_reply":"2021-07-30T05:46:00.476276Z","shell.execute_reply.started":"2021-07-26T10:21:32.941725Z"},"papermill":{"duration":36.187058,"end_time":"2021-07-30T05:46:00.477133","exception":false,"start_time":"2021-07-30T05:45:24.290075","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## edit the paths to the downloaded datasets of Avazu\nPATH_TRAIN = '../input/avazu-ctr-prediction/train.gz'\nPATH_TEST = '../input/avazu-ctr-prediction/test.gz'\n\nimport datatable as dt","metadata":{"papermill":{"duration":0.257167,"end_time":"2021-07-30T05:46:00.769177","exception":false,"start_time":"2021-07-30T05:46:00.51201","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T09:48:21.049607Z","iopub.execute_input":"2021-07-31T09:48:21.050013Z","iopub.status.idle":"2021-07-31T09:48:21.239483Z","shell.execute_reply.started":"2021-07-31T09:48:21.04998Z","shell.execute_reply":"2021-07-31T09:48:21.238466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Exercise 91: Read `train.gz` and `test.gz` and assign it to `train` and `test` respectively**","metadata":{"papermill":{"duration":0.035509,"end_time":"2021-07-30T05:46:00.84039","exception":false,"start_time":"2021-07-30T05:46:00.804881","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train = dt.fread(PATH_TRAIN)\ntrain","metadata":{"papermill":{"duration":74.305796,"end_time":"2021-07-30T05:47:15.181684","exception":false,"start_time":"2021-07-30T05:46:00.875888","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T07:40:30.059612Z","iopub.execute_input":"2021-07-31T07:40:30.061652Z","iopub.status.idle":"2021-07-31T07:41:40.913148Z","shell.execute_reply.started":"2021-07-31T07:40:30.061615Z","shell.execute_reply":"2021-07-31T07:41:40.912187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test = dt.fread(PATH_TEST)\ntest","metadata":{"papermill":{"duration":9.186154,"end_time":"2021-07-30T05:47:24.409695","exception":false,"start_time":"2021-07-30T05:47:15.223541","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T07:41:40.914511Z","iopub.execute_input":"2021-07-31T07:41:40.914779Z","iopub.status.idle":"2021-07-31T07:41:49.761825Z","shell.execute_reply.started":"2021-07-31T07:41:40.914749Z","shell.execute_reply":"2021-07-31T07:41:49.761076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Exercise 92: Drop the `id` column and update the `hour` column with the hour as an integer in `train`**","metadata":{"papermill":{"duration":0.036149,"end_time":"2021-07-30T05:47:24.485936","exception":false,"start_time":"2021-07-30T05:47:24.449787","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train[:, dt.update(hour=dt.as_type(dt.str.slice(dt.as_type(dt.f.hour, str), 6, 8), int))]\ndel train['id']\ntrain.head()","metadata":{"papermill":{"duration":0.112396,"end_time":"2021-07-30T05:47:24.634665","exception":false,"start_time":"2021-07-30T05:47:24.522269","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T07:41:49.764837Z","iopub.execute_input":"2021-07-31T07:41:49.765347Z","iopub.status.idle":"2021-07-31T07:41:49.838424Z","shell.execute_reply.started":"2021-07-31T07:41:49.765315Z","shell.execute_reply":"2021-07-31T07:41:49.837295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Exercise 93: Assign the column `id` to `test_ids`, drop it from `test` and update the `hour` column with the hour as an integer in `test`**","metadata":{"papermill":{"duration":0.037156,"end_time":"2021-07-30T05:47:24.708983","exception":false,"start_time":"2021-07-30T05:47:24.671827","status":"completed"},"tags":[]}},{"cell_type":"code","source":"test_ids = test['id']\ntest_ids.head()","metadata":{"papermill":{"duration":0.046486,"end_time":"2021-07-30T05:47:24.792759","exception":false,"start_time":"2021-07-30T05:47:24.746273","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T07:41:49.83979Z","iopub.execute_input":"2021-07-31T07:41:49.840094Z","iopub.status.idle":"2021-07-31T07:41:49.850328Z","shell.execute_reply.started":"2021-07-31T07:41:49.840067Z","shell.execute_reply":"2021-07-31T07:41:49.849393Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test[:, dt.update(hour=dt.as_type(dt.str.slice(dt.as_type(dt.f.hour, str), 6, 8), int))]\ndel test['id']\ntest.head()","metadata":{"papermill":{"duration":0.051004,"end_time":"2021-07-30T05:47:24.883118","exception":false,"start_time":"2021-07-30T05:47:24.832114","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T07:41:49.851438Z","iopub.execute_input":"2021-07-31T07:41:49.851814Z","iopub.status.idle":"2021-07-31T07:41:49.861453Z","shell.execute_reply.started":"2021-07-31T07:41:49.851787Z","shell.execute_reply":"2021-07-31T07:41:49.86078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Exercise 94: Split `train` into `valid_data`, `valid_target` for rows where `hour=23` and rest as `train_data`, `train_target`**","metadata":{"papermill":{"duration":0.038081,"end_time":"2021-07-30T05:47:24.959297","exception":false,"start_time":"2021-07-30T05:47:24.921216","status":"completed"},"tags":[]}},{"cell_type":"code","source":"valid_data = train[dt.f.hour == 23, [col for col in train.names if col != 'click']]\nvalid_data.head()","metadata":{"papermill":{"duration":13.556117,"end_time":"2021-07-30T05:47:38.553274","exception":false,"start_time":"2021-07-30T05:47:24.997157","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T07:41:49.862366Z","iopub.execute_input":"2021-07-31T07:41:49.862775Z","iopub.status.idle":"2021-07-31T07:42:03.618856Z","shell.execute_reply.started":"2021-07-31T07:41:49.862748Z","shell.execute_reply":"2021-07-31T07:42:03.617757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_target = train[dt.f.hour == 23, 'click']\nvalid_target.head()","metadata":{"papermill":{"duration":13.141821,"end_time":"2021-07-30T05:47:51.733845","exception":false,"start_time":"2021-07-30T05:47:38.592024","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T07:42:03.62008Z","iopub.execute_input":"2021-07-31T07:42:03.620361Z","iopub.status.idle":"2021-07-31T07:42:16.4837Z","shell.execute_reply.started":"2021-07-31T07:42:03.620336Z","shell.execute_reply":"2021-07-31T07:42:16.482933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data = train[dt.f.hour != 23, [col for col in train.names if col != 'click']]\ntrain_data.head()","metadata":{"papermill":{"duration":13.859594,"end_time":"2021-07-30T05:48:05.634015","exception":false,"start_time":"2021-07-30T05:47:51.774421","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T07:42:16.485723Z","iopub.execute_input":"2021-07-31T07:42:16.486149Z","iopub.status.idle":"2021-07-31T07:42:29.424854Z","shell.execute_reply.started":"2021-07-31T07:42:16.48611Z","shell.execute_reply":"2021-07-31T07:42:29.424013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_target = train[dt.f.hour != 23, 'click']\ntrain_target.head()","metadata":{"papermill":{"duration":13.712134,"end_time":"2021-07-30T05:48:19.385323","exception":false,"start_time":"2021-07-30T05:48:05.673189","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T07:42:29.426506Z","iopub.execute_input":"2021-07-31T07:42:29.427145Z","iopub.status.idle":"2021-07-31T07:42:42.804657Z","shell.execute_reply.started":"2021-07-31T07:42:29.427109Z","shell.execute_reply":"2021-07-31T07:42:42.803668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Exercise 95: Train a FTRL model `model_ftrl_1` using `train_data` and `train_target` and assign the predictions of `valid_data` to `preds_valid_1` and of `test` to `preds_test_1`**","metadata":{"papermill":{"duration":0.040217,"end_time":"2021-07-30T05:48:19.465825","exception":false,"start_time":"2021-07-30T05:48:19.425608","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from datatable.models import Ftrl\n\nmodel_ftrl_1 = Ftrl()\nmodel_ftrl_1.fit(train_data, train_target)\nmodel_ftrl_1","metadata":{"papermill":{"duration":37.285443,"end_time":"2021-07-30T05:48:56.790808","exception":false,"start_time":"2021-07-30T05:48:19.505365","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T07:42:42.805929Z","iopub.execute_input":"2021-07-31T07:42:42.806472Z","iopub.status.idle":"2021-07-31T07:43:19.447473Z","shell.execute_reply.started":"2021-07-31T07:42:42.806432Z","shell.execute_reply":"2021-07-31T07:43:19.446482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds_valid_1 = model_ftrl_1.predict(valid_data)\npreds_valid_1.head()","metadata":{"papermill":{"duration":0.682371,"end_time":"2021-07-30T05:48:57.514261","exception":false,"start_time":"2021-07-30T05:48:56.83189","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T07:43:19.4488Z","iopub.execute_input":"2021-07-31T07:43:19.449103Z","iopub.status.idle":"2021-07-31T07:43:20.10483Z","shell.execute_reply.started":"2021-07-31T07:43:19.449075Z","shell.execute_reply":"2021-07-31T07:43:20.103336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds_test_1 = model_ftrl_1.predict(test)\npreds_test_1.head()","metadata":{"papermill":{"duration":2.940894,"end_time":"2021-07-30T05:49:00.505684","exception":false,"start_time":"2021-07-30T05:48:57.56479","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T07:43:20.107041Z","iopub.execute_input":"2021-07-31T07:43:20.108355Z","iopub.status.idle":"2021-07-31T07:43:23.028389Z","shell.execute_reply.started":"2021-07-31T07:43:20.108301Z","shell.execute_reply":"2021-07-31T07:43:23.027485Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Exercise 96: Display the feature importances of `model_ftrl_1` in descending order and calculate the logloss of `valid_target` and `preds_valid_1`**","metadata":{"papermill":{"duration":0.040494,"end_time":"2021-07-30T05:49:00.596807","exception":false,"start_time":"2021-07-30T05:49:00.556313","status":"completed"},"tags":[]}},{"cell_type":"code","source":"model_ftrl_1.feature_importances[:, :, dt.sort(-dt.f.feature_importance)]","metadata":{"papermill":{"duration":0.05395,"end_time":"2021-07-30T05:49:00.691645","exception":false,"start_time":"2021-07-30T05:49:00.637695","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T07:43:23.03234Z","iopub.execute_input":"2021-07-31T07:43:23.034399Z","iopub.status.idle":"2021-07-31T07:43:23.045047Z","shell.execute_reply.started":"2021-07-31T07:43:23.034352Z","shell.execute_reply":"2021-07-31T07:43:23.044252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds = dt.cbind(valid_target, preds_valid_1)\npreds[:, -dt.mean(dt.f.click * dt.math.log(dt.f['True']) + (1-dt.f.click) * dt.math.log(dt.f['False']))][0, 0]","metadata":{"papermill":{"duration":0.285179,"end_time":"2021-07-30T05:49:01.027141","exception":false,"start_time":"2021-07-30T05:49:00.741962","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T07:43:23.048485Z","iopub.execute_input":"2021-07-31T07:43:23.050448Z","iopub.status.idle":"2021-07-31T07:43:23.283926Z","shell.execute_reply.started":"2021-07-31T07:43:23.050406Z","shell.execute_reply":"2021-07-31T07:43:23.283171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Exercise 97: Create `submission_1` in the submission format of the competition, write it as `submission_1.csv` and submit it on Kaggle**","metadata":{"papermill":{"duration":0.041889,"end_time":"2021-07-30T05:49:01.110962","exception":false,"start_time":"2021-07-30T05:49:01.069073","status":"completed"},"tags":[]}},{"cell_type":"code","source":"submission_1 = dt.Frame(id=test_ids, click=preds_test_1['True'])\nsubmission_1.to_csv('submission_1.csv')\nsubmission_1.head()","metadata":{"papermill":{"duration":0.433389,"end_time":"2021-07-30T05:49:01.586659","exception":false,"start_time":"2021-07-30T05:49:01.15327","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T07:43:23.284856Z","iopub.execute_input":"2021-07-31T07:43:23.285248Z","iopub.status.idle":"2021-07-31T07:43:23.662628Z","shell.execute_reply.started":"2021-07-31T07:43:23.28522Z","shell.execute_reply":"2021-07-31T07:43:23.661869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Exercise 98: Train another FTRL model `model_ftrl_2` with `nepochs=3`, `nbins=10 ** 8`, display it's feature importances, score & evaluate it's logloss on `valid_data` and submit the predictions `preds_test_2` of `test` as `submission_2`**","metadata":{"papermill":{"duration":0.042453,"end_time":"2021-07-30T05:49:01.680226","exception":false,"start_time":"2021-07-30T05:49:01.637773","status":"completed"},"tags":[]}},{"cell_type":"code","source":"model_ftrl_2 = Ftrl(nepochs=3, nbins=10**8)\nmodel_ftrl_2.fit(train_data, train_target)\nmodel_ftrl_2","metadata":{"papermill":{"duration":125.278369,"end_time":"2021-07-30T05:51:07.000945","exception":false,"start_time":"2021-07-30T05:49:01.722576","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T07:43:23.663757Z","iopub.execute_input":"2021-07-31T07:43:23.664317Z","iopub.status.idle":"2021-07-31T07:45:24.733461Z","shell.execute_reply.started":"2021-07-31T07:43:23.664284Z","shell.execute_reply":"2021-07-31T07:45:24.732465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_ftrl_2.feature_importances[:, :, dt.sort(-dt.f.feature_importance)]","metadata":{"papermill":{"duration":0.056882,"end_time":"2021-07-30T05:51:07.100717","exception":false,"start_time":"2021-07-30T05:51:07.043835","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T08:02:28.881237Z","iopub.execute_input":"2021-07-31T08:02:28.88164Z","iopub.status.idle":"2021-07-31T08:02:28.889606Z","shell.execute_reply.started":"2021-07-31T08:02:28.881609Z","shell.execute_reply":"2021-07-31T08:02:28.888827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds_valid_2 = model_ftrl_2.predict(valid_data)\npreds_valid_2.head()","metadata":{"papermill":{"duration":0.753067,"end_time":"2021-07-30T05:51:07.906117","exception":false,"start_time":"2021-07-30T05:51:07.15305","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T08:02:30.553565Z","iopub.execute_input":"2021-07-31T08:02:30.553903Z","iopub.status.idle":"2021-07-31T08:02:31.312558Z","shell.execute_reply.started":"2021-07-31T08:02:30.553859Z","shell.execute_reply":"2021-07-31T08:02:31.311695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds_test_2 = model_ftrl_2.predict(test)\npreds_test_2.head()","metadata":{"papermill":{"duration":3.276748,"end_time":"2021-07-30T05:51:11.237583","exception":false,"start_time":"2021-07-30T05:51:07.960835","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T08:02:31.923125Z","iopub.execute_input":"2021-07-31T08:02:31.924151Z","iopub.status.idle":"2021-07-31T08:02:35.113852Z","shell.execute_reply.started":"2021-07-31T08:02:31.924103Z","shell.execute_reply":"2021-07-31T08:02:35.112715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds = dt.cbind(valid_target, preds_valid_2)\npreds[:, -dt.mean(dt.f.click * dt.math.log(dt.f['True']) + (1-dt.f.click) * dt.math.log(dt.f['False']))][0, 0]","metadata":{"papermill":{"duration":0.285736,"end_time":"2021-07-30T05:51:11.579052","exception":false,"start_time":"2021-07-30T05:51:11.293316","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T08:02:35.11843Z","iopub.execute_input":"2021-07-31T08:02:35.11883Z","iopub.status.idle":"2021-07-31T08:02:35.352818Z","shell.execute_reply.started":"2021-07-31T08:02:35.118797Z","shell.execute_reply":"2021-07-31T08:02:35.351777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_2 = dt.Frame(id=test_ids, click=preds_test_2['True'])\nsubmission_2.to_csv('submission_2.csv')\nsubmission_2.head()","metadata":{"papermill":{"duration":0.449484,"end_time":"2021-07-30T05:51:12.073424","exception":false,"start_time":"2021-07-30T05:51:11.62394","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T08:02:35.354406Z","iopub.execute_input":"2021-07-31T08:02:35.354698Z","iopub.status.idle":"2021-07-31T08:02:35.808824Z","shell.execute_reply.started":"2021-07-31T08:02:35.354671Z","shell.execute_reply":"2021-07-31T08:02:35.808011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Exercise 99: Submit a ensemble of `model_ftrl_1` and `model_ftrl_2` by averaging the predictions as `submission_ensemble`**","metadata":{"papermill":{"duration":0.044929,"end_time":"2021-07-30T05:51:12.173085","exception":false,"start_time":"2021-07-30T05:51:12.128156","status":"completed"},"tags":[]}},{"cell_type":"code","source":"submission_ensemble = dt.cbind(submission_1, submission_2)\nsubmission_ensemble[:, dt.update(click = 0.5 * dt.f.click + 0.5 * dt.f['click.0'])]\ndel submission_ensemble[:, ['id.0', 'click.0']]\nsubmission_ensemble.to_csv('submission_ensemble.csv')\nsubmission_ensemble.head()","metadata":{"execution":{"iopub.execute_input":"2021-07-30T05:51:13.13867Z","iopub.status.busy":"2021-07-30T05:51:13.137654Z","iopub.status.idle":"2021-07-30T05:51:13.144961Z","shell.execute_reply":"2021-07-30T05:51:13.145864Z","shell.execute_reply.started":"2021-07-30T05:36:25.948007Z"},"papermill":{"duration":0.925722,"end_time":"2021-07-30T05:51:13.146076","exception":false,"start_time":"2021-07-30T05:51:12.220354","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Exercise 100: Use datatable on your own datasets and projects**","metadata":{"papermill":{"duration":0.047787,"end_time":"2021-07-30T05:51:13.248878","exception":false,"start_time":"2021-07-30T05:51:13.201091","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import datatable as dt\nfrom datatable.models import Ftrl","metadata":{"execution":{"iopub.status.busy":"2021-07-31T09:48:50.091153Z","iopub.execute_input":"2021-07-31T09:48:50.09154Z","iopub.status.idle":"2021-07-31T09:48:50.097492Z","shell.execute_reply.started":"2021-07-31T09:48:50.091503Z","shell.execute_reply":"2021-07-31T09:48:50.096011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = dt.fread('../input/tradeshift-text-classification/train.csv.gz')\ntrain.head()","metadata":{"papermill":{"duration":0.044849,"end_time":"2021-07-30T05:51:13.338768","exception":false,"start_time":"2021-07-30T05:51:13.293919","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-31T09:52:06.453642Z","iopub.execute_input":"2021-07-31T09:52:06.454022Z","iopub.status.idle":"2021-07-31T09:52:33.46913Z","shell.execute_reply.started":"2021-07-31T09:52:06.453992Z","shell.execute_reply":"2021-07-31T09:52:33.468031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test = dt.fread('../input/tradeshift-text-classification/test.csv.gz')\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T09:49:25.309642Z","iopub.execute_input":"2021-07-31T09:49:25.310001Z","iopub.status.idle":"2021-07-31T09:49:39.224502Z","shell.execute_reply.started":"2021-07-31T09:49:25.309969Z","shell.execute_reply":"2021-07-31T09:49:39.221035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels = dt.fread('../input/tradeshift-text-classification/trainLabels.csv.gz')\ntrain_labels.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T09:49:39.227685Z","iopub.execute_input":"2021-07-31T09:49:39.22847Z","iopub.status.idle":"2021-07-31T09:49:40.144943Z","shell.execute_reply.started":"2021-07-31T09:49:39.228413Z","shell.execute_reply":"2021-07-31T09:49:40.14388Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_ids = test['id']\n\ndel train['id']\ndel test['id']\n\nsubmission = dt.Frame()\n\nfor target in train_labels.names[1:]:\n print(f'Model for target {target}')\n\n model_ftrl = Ftrl(nepochs=5, nbins=10**8, lambda1=0.1)\n model_ftrl.fit(train, train_labels[target])\n\n preds_test = model_ftrl.predict(test)\n submission_target = dt.Frame(id_label=test_ids[:, dt.as_type(dt.f.id, str) + f'_{target}'],\n pred=preds_test['True'])\n submission.rbind(submission_target)","metadata":{"execution":{"iopub.status.busy":"2021-07-31T09:57:36.704999Z","iopub.execute_input":"2021-07-31T09:57:36.705374Z","iopub.status.idle":"2021-07-31T10:03:08.590146Z","shell.execute_reply.started":"2021-07-31T09:57:36.705342Z","shell.execute_reply":"2021-07-31T10:03:08.589314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv')\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-31T10:08:15.895763Z","iopub.execute_input":"2021-07-31T10:08:15.896152Z","iopub.status.idle":"2021-07-31T10:08:17.483483Z","shell.execute_reply.started":"2021-07-31T10:08:15.89612Z","shell.execute_reply":"2021-07-31T10:08:17.481332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"✅ This completes Set 10: Capstone Projects (Exercises 91-100) and concludes **DatatableTon**: *💯 datatable exercises*\n\nHope these exercises enabled you to learn more about the Python datatable library. If this project made you go from an amateur to a datatable ninja or helped you get a deeper understanding of datatable or was useful in teaching datatable to others, please share your thoughts and any other feedback or suggestions on our [GitHub Repository](https://github.com/vopani/datatableton) where you can also find all the exercises and solutions along with future updates.","metadata":{"papermill":{"duration":0.044639,"end_time":"2021-07-30T05:51:13.429778","exception":false,"start_time":"2021-07-30T05:51:13.385139","status":"completed"},"tags":[]}}]}
\n \n \n 
\n \n \n 
\n \n