{"cells":[{"metadata":{},"cell_type":"markdown","source":"# 2020 NCAAW: Fast data loading with feather\n\nOriginal csv format takes time to load data, especially for event data. Here I converted them and uploaded with feather format.\nIt is about **5 times faster**.\n\nYou can see dataset here: https://www.kaggle.com/corochann/ncaa-march-madness-2020-womens\nPlease upvote both dataset and this kernel if you like it! :)\n\nThis kernel describes how to load this dataset.\n\n# How to add dataset\n\nWhen you write kernel, click \"+ Add Data\" botton on right top.\nThen inside window pop-up, you can see \"Search Datasets\" text box on right top.\nYou can type **\"ncaa-march-madness-2020-womens\"** to find this dataset and press \"Add\" botton to add the data."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import gc\nimport os\nfrom pathlib import Path\nimport random\nimport sys\n\nfrom tqdm.notebook import tqdm\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom IPython.core.display import display, HTML\n\n# --- plotly ---\nfrom plotly import tools, subplots\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.express as px\nimport plotly.figure_factory as ff\n\n# --- models ---\nfrom sklearn import preprocessing\nfrom sklearn.model_selection import KFold\nimport lightgbm as lgb\nimport xgboost as xgb\nimport catboost as cb\n\n# --- setup ---\npd.set_option('max_columns', 50)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls /kaggle/input","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load data in feather format\n\nYou can use `pd.read_feather` to read feather format."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ndatadir = Path('/kaggle/input/ncaa-march-madness-2020-womens')\nstage1dir = datadir / 'WDataFiles_Stage1'\n\n# --- read data ---\nevent_2015_df = pd.read_feather(datadir / 'WEvents2015.feather')\nevent_2016_df = pd.read_feather(datadir / 'WEvents2016.feather')\nevent_2017_df = pd.read_feather(datadir / 'WEvents2017.feather')\nevent_2018_df = pd.read_feather(datadir / 'WEvents2018.feather')\nevent_2019_df = pd.read_feather(datadir / 'WEvents2019.feather')\nplayers_df = pd.read_feather(datadir / 'WPlayers.feather')\nsample_submission = pd.read_feather(datadir / 'WSampleSubmissionStage1_2020.feather')\n\ncities_df = pd.read_feather(stage1dir / 'Cities.feather')\nconferences_df = pd.read_feather(stage1dir / 'Conferences.feather')\n# conference_tourney_games_df = pd.read_feather(stage1dir / 'WConferenceTourneyGames.feather')\ngame_cities_df = pd.read_feather(stage1dir / 'WGameCities.feather')\n# massey_ordinals_df = pd.read_feather(stage1dir / 'WMasseyOrdinals.feather')\ntourney_compact_results_df = pd.read_feather(stage1dir / 'WNCAATourneyCompactResults.feather')\ntourney_detailed_results_df = pd.read_feather(stage1dir / 'WNCAATourneyDetailedResults.feather')\n# tourney_seed_round_slots_df = pd.read_feather(stage1dir / 'WNCAATourneySeedRoundSlots.feather')\ntourney_seeds_df = pd.read_feather(stage1dir / 'WNCAATourneySeeds.feather')\ntourney_slots_df = pd.read_feather(stage1dir / 'WNCAATourneySlots.feather')\nregular_season_compact_results_df = pd.read_feather(stage1dir / 'WRegularSeasonCompactResults.feather')\nregular_season_detailed_results_df = pd.read_feather(stage1dir / 'WRegularSeasonDetailedResults.feather')\nseasons_df = pd.read_feather(stage1dir / 'WSeasons.feather')\n# secondary_tourney_compact_results_df = pd.read_feather(stage1dir / 'WSecondaryTourneyCompactResults.feather')\n# secondary_tourney_teams_df = pd.read_feather(stage1dir / 'WSecondaryTourneyTeams.feather')\n# team_coaches_df = pd.read_feather(stage1dir / 'WTeamCoaches.feather')\nteam_conferences_df = pd.read_feather(stage1dir / 'WTeamConferences.feather')\nteams_df = pd.read_feather(stage1dir / 'WTeams.feather')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load data in original csv format\n\nLet's compare how long it takes to load original csv format."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"%%time\n\ndatadir = Path('/kaggle/input/google-cloud-ncaa-march-madness-2020-division-1-womens-tournament')\nstage1dir = datadir/'WDataFiles_Stage1'\n\nevent_2015_df = pd.read_csv(datadir / 'WEvents2015.csv')\nevent_2016_df = pd.read_csv(datadir / 'WEvents2016.csv')\nevent_2017_df = pd.read_csv(datadir / 'WEvents2017.csv')\nevent_2018_df = pd.read_csv(datadir / 'WEvents2018.csv')\nevent_2019_df = pd.read_csv(datadir / 'WEvents2019.csv')\nplayers_df = pd.read_csv(datadir / 'WPlayers.csv')\nsample_submission = pd.read_csv(datadir / 'WSampleSubmissionStage1_2020.csv')\n\ncities_df = pd.read_csv(stage1dir / 'Cities.csv')\nconferences_df = pd.read_csv(stage1dir / 'Conferences.csv')\n# conference_tourney_games_df = pd.read_csv(stage1dir / 'WConferenceTourneyGames.csv')\ngame_cities_df = pd.read_csv(stage1dir / 'WGameCities.csv')\n# massey_ordinals_df = pd.read_csv(stage1dir / 'WMasseyOrdinals.csv')\ntourney_compact_results_df = pd.read_csv(stage1dir / 'WNCAATourneyCompactResults.csv')\ntourney_detailed_results_df = pd.read_csv(stage1dir / 'WNCAATourneyDetailedResults.csv')\n# tourney_seed_round_slots_df = pd.read_csv(stage1dir / 'WNCAATourneySeedRoundSlots.csv')\ntourney_seeds_df = pd.read_csv(stage1dir / 'WNCAATourneySeeds.csv')\ntourney_slots_df = pd.read_csv(stage1dir / 'WNCAATourneySlots.csv')\nregular_season_compact_results_df = pd.read_csv(stage1dir / 'WRegularSeasonCompactResults.csv')\nregular_season_detailed_results_df = pd.read_csv(stage1dir / 'WRegularSeasonDetailedResults.csv')\nseasons_df = pd.read_csv(stage1dir / 'WSeasons.csv')\n# secondary_tourney_compact_results_df = pd.read_csv(stage1dir / 'WSecondaryTourneyCompactResults.csv')\n# secondary_tourney_teams_df = pd.read_csv(stage1dir / 'WSecondaryTourneyTeams.csv')\n# team_coaches_df = pd.read_csv(stage1dir / 'WTeamCoaches.csv')\nteam_conferences_df = pd.read_csv(stage1dir / 'WTeamConferences.csv')\nteams_df = pd.read_csv(stage1dir / 'WTeams.csv')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"As you can see Wall time between loading feather format and original csv format, feather format loading is much faster!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}