{"cells":[{"metadata":{},"cell_type":"markdown","source":"# 2020 NCAAM: Fast data loading with feather\n\nOriginal csv format takes time to load data, especially for event data. Here I converted them and uploaded with feather format.\nIt is about **10 times faster**.\n\nYou can see dataset here: https://www.kaggle.com/corochann/ncaa-march-madness-2020-mens\nPlease upvote both dataset and this kernel if you like it! :)\n\nThis kernel describes how to load this dataset.\n\n# How to add dataset\n\nWhen you write kernel, click \"+ Add Data\" botton on right top.\nThen inside window pop-up, you can see \"Search Datasets\" text box on right top.\nYou can type **\"ncaa-march-madness-2020-mens\"** to find this dataset and press \"Add\" botton to add the data."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import gc\nimport os\nfrom pathlib import Path\nimport random\nimport sys\n\nfrom tqdm.notebook import tqdm\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom IPython.core.display import display, HTML\n\n# --- plotly ---\nfrom plotly import tools, subplots\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.express as px\nimport plotly.figure_factory as ff\n\n# --- models ---\nfrom sklearn import preprocessing\nfrom sklearn.model_selection import KFold\nimport lightgbm as lgb\nimport xgboost as xgb\nimport catboost as cb\n\n# --- setup ---\npd.set_option('max_columns', 50)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load data in feather format\n\nYou can use `pd.read_feather` to read feather format."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"%%time\n\ndatadir = Path('/kaggle/input/ncaa-march-madness-2020-mens')\nstage1dir = datadir/'MDataFiles_Stage1'\n\nevent_2015_df = pd.read_feather(datadir / 'MEvents2015.feather')\nevent_2016_df = pd.read_feather(datadir / 'MEvents2016.feather')\nevent_2017_df = pd.read_feather(datadir / 'MEvents2017.feather')\nevent_2018_df = pd.read_feather(datadir / 'MEvents2018.feather')\nevent_2019_df = pd.read_feather(datadir / 'MEvents2019.feather')\nplayers_df = pd.read_feather(datadir / 'MPlayers.feather')\nsample_submission = pd.read_feather(datadir / 'MSampleSubmissionStage1_2020.feather')\n\ncities_df = pd.read_feather(stage1dir / 'Cities.feather')\nconferences_df = pd.read_feather(stage1dir / 'Conferences.feather')\nconference_tourney_games_df = pd.read_feather(stage1dir / 'MConferenceTourneyGames.feather')\ngame_cities_df = pd.read_feather(stage1dir / 'MGameCities.feather')\nmassey_ordinals_df = pd.read_feather(stage1dir / 'MMasseyOrdinals.feather')\ntourney_compact_results_df = pd.read_feather(stage1dir / 'MNCAATourneyCompactResults.feather')\ntourney_detailed_results_df = pd.read_feather(stage1dir / 'MNCAATourneyDetailedResults.feather')\ntourney_seed_round_slots_df = pd.read_feather(stage1dir / 'MNCAATourneySeedRoundSlots.feather')\ntourney_seeds_df = pd.read_feather(stage1dir / 'MNCAATourneySeeds.feather')\ntourney_slots_df = pd.read_feather(stage1dir / 'MNCAATourneySlots.feather')\nregular_season_compact_results_df = pd.read_feather(stage1dir / 'MRegularSeasonCompactResults.feather')\nregular_season_detailed_results_df = pd.read_feather(stage1dir / 'MRegularSeasonDetailedResults.feather')\nseasons_df = pd.read_feather(stage1dir / 'MSeasons.feather')\nsecondary_tourney_compact_results_df = pd.read_feather(stage1dir / 'MSecondaryTourneyCompactResults.feather')\nsecondary_tourney_teams_df = pd.read_feather(stage1dir / 'MSecondaryTourneyTeams.feather')\nteam_coaches_df = pd.read_feather(stage1dir / 'MTeamCoaches.feather')\nteam_conferences_df = pd.read_feather(stage1dir / 'MTeamConferences.feather')\nteams_df = pd.read_feather(stage1dir / 'MTeams.feather')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load data in original csv format\n\nLet's compare how long it takes to load original csv format."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ndatadir = Path('/kaggle/input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament')\nstage1dir = datadir/'MDataFiles_Stage1'\n\nevent_2015_df = pd.read_csv(datadir/'MEvents2015.csv')\nevent_2016_df = pd.read_csv(datadir/'MEvents2016.csv')\nevent_2017_df = pd.read_csv(datadir/'MEvents2017.csv')\nevent_2018_df = pd.read_csv(datadir/'MEvents2018.csv')\nevent_2019_df = pd.read_csv(datadir/'MEvents2019.csv')\nplayers_df = pd.read_csv(datadir/'MPlayers.csv')\nsample_submission = pd.read_csv(datadir/'MSampleSubmissionStage1_2020.csv')\n\ncities_df = pd.read_csv(stage1dir/'Cities.csv')\nconferences_df = pd.read_csv(stage1dir/'Conferences.csv')\nconference_tourney_games_df = pd.read_csv(stage1dir/'MConferenceTourneyGames.csv')\ngame_cities_df = pd.read_csv(stage1dir/'MGameCities.csv')\nmassey_ordinals_df = pd.read_csv(stage1dir/'MMasseyOrdinals.csv')\ntourney_compact_results_df = pd.read_csv(stage1dir/'MNCAATourneyCompactResults.csv')\ntourney_detailed_results_df = pd.read_csv(stage1dir/'MNCAATourneyDetailedResults.csv')\ntourney_seed_round_slots_df = pd.read_csv(stage1dir/'MNCAATourneySeedRoundSlots.csv')\ntourney_seeds_df = pd.read_csv(stage1dir/'MNCAATourneySeeds.csv')\ntourney_slots_df = pd.read_csv(stage1dir/'MNCAATourneySlots.csv')\nregular_season_compact_results_df = pd.read_csv(stage1dir/'MRegularSeasonCompactResults.csv')\nregular_season_detailed_results_df = pd.read_csv(stage1dir/'MRegularSeasonDetailedResults.csv')\nseasons_df = pd.read_csv(stage1dir/'MSeasons.csv')\nsecondary_tourney_compact_results_df = pd.read_csv(stage1dir/'MSecondaryTourneyCompactResults.csv')\nsecondary_tourney_teams_df = pd.read_csv(stage1dir/'MSecondaryTourneyTeams.csv')\nteam_coaches_df = pd.read_csv(stage1dir/'MTeamCoaches.csv')\nteam_conferences_df = pd.read_csv(stage1dir/'MTeamConferences.csv')\nteams_df = pd.read_csv(stage1dir/'MTeams.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see Wall time between loading feather format and original csv format, feather format loading is much faster!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}