{ "id": 80732, "title": "Overfitting train - Reasons behind massive difference in LB vs CV", "url": "/competitions/favorita-grocery-sales-forecasting/discussion/80732", "author_name": "bharathsivaraman", "post_date": "2019-02-15T21:54:08.957000", "votes": 0, "comment_count": 0, "views": 0, "content": "

I seem to be massively overfitting my train dataset and I am at loss trying to figure the reason out. Any help is much appreciated

\n\n

These are the steps I have done:

\n\n
    \n

  1. Split training dataset into 2 subsets training

    \n\n
      2017-07-05 to 2017-07-27\n\n  2017-01-05 to 2017-02-28\n
    2. \n

  2. Log transforming the unit_sales and where negative making it 0

    3. \n

  3. I have weighted the dataset with perishable column before converting into a Dmatrix for XGboost

    4. \n

  4. I am using 5fold CV with XgBoost(Dmatrix) with early stopping to train the above two datasets. The results of the first fold is shown below and all the 10 folds ( for both data sets) are consistent

    \n\n
    Training dataset with start date: 2017-01-05T00:00:00.000000000 end date: 2017-02-27T00:00:00.000000000\n\nFold 1\n[0] train-rmse:1.02536  eval-rmse:1.0255\nMultiple eval metrics have been passed: 'eval-rmse' will be used for early stopping.\n\nWill train until eval-rmse hasn't improved in 50 rounds.\n[50]    train-rmse:0.744308 eval-rmse:0.744343\n[100]   train-rmse:0.725045 eval-rmse:0.724968\n[150]   train-rmse:0.703867 eval-rmse:0.703891\n[156]   train-rmse:0.702914 eval-rms\n
    \n\n

    e:0.702929\n> The RMSE Score for this round is: 0.49415837094493603

    5. \n
\n\n

My first question is: Why is the overall RMSE on the eval dataset lower than the Eval RMSE used by XgBoost's for early stopping? I am not able to figure the reason

\n\n

Also, I am splitting the train dataset into two more subsets for testing. SO the model hasn't seen the data

\n\n
1. '2017-08-02 to '2017-08-15'\n2. '2017-03-01' to '2017-03-16'\n
\n\n

The overall RMSE on these two datasets in 0.495 which is consistent with my overall CV score. When I do the same thing on the public test dataset my RMSE is 0.909 ( I am using a exponential transform of the output)

\n\n

I do not understand what I am doing wrong when the CV and the test data( that I have partitioned from the training set) show the same RMSE but the public test dataset is so different. I have never had such a massive overfitting problem which leads me to think there is something I am doing wrong rather than this being a CV issue. Below is the code I have used for CV

\n\n

Code for preprocess function:

\n\n
def preprocess_(df):\n    global data_clean,weightData\n    df['week']=df['date'].dt.week\n    df['day']=df['date'].dt.day_name()\n    data_prep=pd.merge(df,data_items,on=\"item_nbr\",how=\"left\")\n    data_prep=pd.merge(data_prep,data_stores,on=\"store_nbr\",how=\"left\")\n    weightData=1+ (data_prep['perishable']*.25)\n    data_clean=pd.get_dummies(data_prep,columns=['onpromotion','day'])\n    del data_prep \n    data_clean.drop(['id','perishable',],axis=1,inplace=True)\n
\n\n

Code for CV: ( The indentation of loops is messed up in Kaggle and I am not able to fix it)

\n\n
for idx,date in enumerate(train_start_date):\nselect_date(train_start_date[idx],train_end_date[idx])\nprint(\"Training dataset with\",\"start date:\",min(data_train_sel.date.values),\"end date:\",max(data_train_sel.date.values))\npreprocess_(data_train_sel)\ndel data_train_sel\n\nX=data_clean.drop([\"unit_sales\",'date'],axis=1)\nfeature_names=X.columns\ny=data_clean['unit_sales']\ndel data_clean\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\ndel X,y\nK=5\nkf=KFold(n_splits=5,random_state=1234,shuffle=True)\nnp.random.seed(100)\nscores=np.zeros(X_train[:].shape[0])\nimp_df=np.zeros((len(feature_names), K))\n\n\n\nfor i,(train_index,test_index) in enumerate(kf.split(X_train,y_train)):\n    x_train_cv,y_train_cv=X_train.iloc[train_index].copy(),y_train.iloc[train_index].copy()\n    w_train_cv = weightData[train_index]\n    x_train_cv=xgb.DMatrix(x_train_cv.valueslabel=y_train_cv,feature_names=feature_names,weight=w_train_cv)\n    x_eval_cv,y_eval_cv=X_train.iloc[test_index].copy(),y_train.iloc[test_index].copy()\n    w_eval_cv = weightData[test_index]\n    x_eval_cv=xgb.DMatrix(x_eval_cv.values,label=y_eval_cv,feature_names=feature_names,weight=w_eval_cv)\n    watchlist = [(x_train_cv, 'train'), (x_eval_cv, 'eval')]\n    print(\"\\nFold\",i+1)\n    clf=xgb.train(plst, x_train_cv, num_rounds,watchlist, early_stopping_rounds=50, verbose_eval=50)\n    y_pred_cv=clf.predict(x_eval_cv)\n    scores[test_index]=metrics.mean_squared_error(y_eval_cv, y_pred_cv)\n    print(\"The RMSE Score for this round is:\",metrics.mean_squared_error(y_eval_cv,y_pred_cv))\n\nprint ((\"Overall Mean score: {0:.3f} (+/-{1:.3f})\").format(np.mean(scores), sem(scores)))\nprep_predict(X_test,y_test)\ndel X_train, X_test, y_train, y_test\n\ndel data_train\n
", "messages": [ { "id": 472425, "postDate": "2019-02-15T21:54:08.957Z", "content": "

I seem to be massively overfitting my train dataset and I am at loss trying to figure the reason out. Any help is much appreciated

\n\n

These are the steps I have done:

\n\n
    \n

  1. Split training dataset into 2 subsets training

    \n\n
      2017-07-05 to 2017-07-27\n\n  2017-01-05 to 2017-02-28\n
    2. \n

  2. Log transforming the unit_sales and where negative making it 0

    3. \n

  3. I have weighted the dataset with perishable column before converting into a Dmatrix for XGboost

    4. \n

  4. I am using 5fold CV with XgBoost(Dmatrix) with early stopping to train the above two datasets. The results of the first fold is shown below and all the 10 folds ( for both data sets) are consistent

    \n\n
    Training dataset with start date: 2017-01-05T00:00:00.000000000 end date: 2017-02-27T00:00:00.000000000\n\nFold 1\n[0] train-rmse:1.02536  eval-rmse:1.0255\nMultiple eval metrics have been passed: 'eval-rmse' will be used for early stopping.\n\nWill train until eval-rmse hasn't improved in 50 rounds.\n[50]    train-rmse:0.744308 eval-rmse:0.744343\n[100]   train-rmse:0.725045 eval-rmse:0.724968\n[150]   train-rmse:0.703867 eval-rmse:0.703891\n[156]   train-rmse:0.702914 eval-rms\n
    \n\n

    e:0.702929\n> The RMSE Score for this round is: 0.49415837094493603

    5. \n
\n\n

My first question is: Why is the overall RMSE on the eval dataset lower than the Eval RMSE used by XgBoost's for early stopping? I am not able to figure the reason

\n\n

Also, I am splitting the train dataset into two more subsets for testing. SO the model hasn't seen the data

\n\n
1. '2017-08-02 to '2017-08-15'\n2. '2017-03-01' to '2017-03-16'\n
\n\n

The overall RMSE on these two datasets in 0.495 which is consistent with my overall CV score. When I do the same thing on the public test dataset my RMSE is 0.909 ( I am using a exponential transform of the output)

\n\n

I do not understand what I am doing wrong when the CV and the test data( that I have partitioned from the training set) show the same RMSE but the public test dataset is so different. I have never had such a massive overfitting problem which leads me to think there is something I am doing wrong rather than this being a CV issue. Below is the code I have used for CV

\n\n

Code for preprocess function:

\n\n
def preprocess_(df):\n    global data_clean,weightData\n    df['week']=df['date'].dt.week\n    df['day']=df['date'].dt.day_name()\n    data_prep=pd.merge(df,data_items,on=\"item_nbr\",how=\"left\")\n    data_prep=pd.merge(data_prep,data_stores,on=\"store_nbr\",how=\"left\")\n    weightData=1+ (data_prep['perishable']*.25)\n    data_clean=pd.get_dummies(data_prep,columns=['onpromotion','day'])\n    del data_prep \n    data_clean.drop(['id','perishable',],axis=1,inplace=True)\n
\n\n

Code for CV: ( The indentation of loops is messed up in Kaggle and I am not able to fix it)

\n\n
for idx,date in enumerate(train_start_date):\nselect_date(train_start_date[idx],train_end_date[idx])\nprint(\"Training dataset with\",\"start date:\",min(data_train_sel.date.values),\"end date:\",max(data_train_sel.date.values))\npreprocess_(data_train_sel)\ndel data_train_sel\n\nX=data_clean.drop([\"unit_sales\",'date'],axis=1)\nfeature_names=X.columns\ny=data_clean['unit_sales']\ndel data_clean\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\ndel X,y\nK=5\nkf=KFold(n_splits=5,random_state=1234,shuffle=True)\nnp.random.seed(100)\nscores=np.zeros(X_train[:].shape[0])\nimp_df=np.zeros((len(feature_names), K))\n\n\n\nfor i,(train_index,test_index) in enumerate(kf.split(X_train,y_train)):\n    x_train_cv,y_train_cv=X_train.iloc[train_index].copy(),y_train.iloc[train_index].copy()\n    w_train_cv = weightData[train_index]\n    x_train_cv=xgb.DMatrix(x_train_cv.valueslabel=y_train_cv,feature_names=feature_names,weight=w_train_cv)\n    x_eval_cv,y_eval_cv=X_train.iloc[test_index].copy(),y_train.iloc[test_index].copy()\n    w_eval_cv = weightData[test_index]\n    x_eval_cv=xgb.DMatrix(x_eval_cv.values,label=y_eval_cv,feature_names=feature_names,weight=w_eval_cv)\n    watchlist = [(x_train_cv, 'train'), (x_eval_cv, 'eval')]\n    print(\"\\nFold\",i+1)\n    clf=xgb.train(plst, x_train_cv, num_rounds,watchlist, early_stopping_rounds=50, verbose_eval=50)\n    y_pred_cv=clf.predict(x_eval_cv)\n    scores[test_index]=metrics.mean_squared_error(y_eval_cv, y_pred_cv)\n    print(\"The RMSE Score for this round is:\",metrics.mean_squared_error(y_eval_cv,y_pred_cv))\n\nprint ((\"Overall Mean score: {0:.3f} (+/-{1:.3f})\").format(np.mean(scores), sem(scores)))\nprep_predict(X_test,y_test)\ndel X_train, X_test, y_train, y_test\n\ndel data_train\n
", "rawMarkdown": "I seem to be massively overfitting my train dataset and I am at loss trying to figure the reason out. Any help is much appreciated\n\nThese are the steps I have done:\n\n 1. Split training dataset into 2 subsets training\n\n 2017-07-05 to 2017-07-27\n\n 2017-01-05 to 2017-02-28\n\n 3. Log transforming the unit_sales and where negative making it 0\n\n 4. I have weighted the dataset with perishable column before converting into a Dmatrix for XGboost\n 5. I am using 5fold CV with XgBoost(Dmatrix) with early stopping to train the above two datasets. The results of the first fold is shown below and all the 10 folds ( for both data sets) are consistent\n\n Training dataset with start date: 2017-01-05T00:00:00.000000000 end date: 2017-02-27T00:00:00.000000000\n \n Fold 1\n [0]\ttrain-rmse:1.02536\teval-rmse:1.0255\n Multiple eval metrics have been passed: 'eval-rmse' will be used for early stopping.\n \n Will train until eval-rmse hasn't improved in 50 rounds.\n [50]\ttrain-rmse:0.744308\teval-rmse:0.744343\n [100]\ttrain-rmse:0.725045\teval-rmse:0.724968\n [150]\ttrain-rmse:0.703867\teval-rmse:0.703891\n [156]\ttrain-rmse:0.702914\teval-rms\n e:0.702929\n > The RMSE Score for this round is: 0.49415837094493603\n\nMy first question is: Why is the overall RMSE on the eval dataset lower than the Eval RMSE used by XgBoost's for early stopping? I am not able to figure the reason\n\nAlso, I am splitting the train dataset into two more subsets for testing. SO the model hasn't seen the data\n\n 1. '2017-08-02 to '2017-08-15'\n 2. '2017-03-01' to '2017-03-16'\n\nThe overall RMSE on these two datasets in 0.495 which is consistent with my overall CV score. When I do the same thing on the public test dataset my RMSE is 0.909 ( I am using a exponential transform of the output)\n\nI do not understand what I am doing wrong when the CV and the test data( that I have partitioned from the training set) show the same RMSE but the public test dataset is so different. I have never had such a massive overfitting problem which leads me to think there is something I am doing wrong rather than this being a CV issue. Below is the code I have used for CV\n\nCode for preprocess function:\n\n def preprocess_(df):\n global data_clean,weightData\n df['week']=df['date'].dt.week\n df['day']=df['date'].dt.day_name()\n data_prep=pd.merge(df,data_items,on=\"item_nbr\",how=\"left\")\n data_prep=pd.merge(data_prep,data_stores,on=\"store_nbr\",how=\"left\")\n weightData=1+ (data_prep['perishable']*.25)\n data_clean=pd.get_dummies(data_prep,columns=['onpromotion','day'])\n del data_prep \n data_clean.drop(['id','perishable',],axis=1,inplace=True)\n\nCode for CV: ( The indentation of loops is messed up in Kaggle and I am not able to fix it)\n\n for idx,date in enumerate(train_start_date):\n select_date(train_start_date[idx],train_end_date[idx])\n print(\"Training dataset with\",\"start date:\",min(data_train_sel.date.values),\"end date:\",max(data_train_sel.date.values))\n preprocess_(data_train_sel)\n del data_train_sel\n\n X=data_clean.drop([\"unit_sales\",'date'],axis=1)\n feature_names=X.columns\n y=data_clean['unit_sales']\n del data_clean\n \n X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n del X,y\n K=5\n kf=KFold(n_splits=5,random_state=1234,shuffle=True)\n np.random.seed(100)\n scores=np.zeros(X_train[:].shape[0])\n imp_df=np.zeros((len(feature_names), K))\n \n \n\n for i,(train_index,test_index) in enumerate(kf.split(X_train,y_train)):\n x_train_cv,y_train_cv=X_train.iloc[train_index].copy(),y_train.iloc[train_index].copy()\n w_train_cv = weightData[train_index]\n x_train_cv=xgb.DMatrix(x_train_cv.valueslabel=y_train_cv,feature_names=feature_names,weight=w_train_cv)\n x_eval_cv,y_eval_cv=X_train.iloc[test_index].copy(),y_train.iloc[test_index].copy()\n w_eval_cv = weightData[test_index]\n x_eval_cv=xgb.DMatrix(x_eval_cv.values,label=y_eval_cv,feature_names=feature_names,weight=w_eval_cv)\n watchlist = [(x_train_cv, 'train'), (x_eval_cv, 'eval')]\n print(\"\\nFold\",i+1)\n clf=xgb.train(plst, x_train_cv, num_rounds,watchlist, early_stopping_rounds=50, verbose_eval=50)\n y_pred_cv=clf.predict(x_eval_cv)\n scores[test_index]=metrics.mean_squared_error(y_eval_cv, y_pred_cv)\n print(\"The RMSE Score for this round is:\",metrics.mean_squared_error(y_eval_cv,y_pred_cv))\n \n print ((\"Overall Mean score: {0:.3f} (+/-{1:.3f})\").format(np.mean(scores), sem(scores)))\n prep_predict(X_test,y_test)\n del X_train, X_test, y_train, y_test\n \n del data_train\n" } ], "comments": [], "raw_markdown_by_id": { "472425": "I seem to be massively overfitting my train dataset and I am at loss trying to figure the reason out. Any help is much appreciated\n\nThese are the steps I have done:\n\n 1. Split training dataset into 2 subsets training\n\n 2017-07-05 to 2017-07-27\n\n 2017-01-05 to 2017-02-28\n\n 3. Log transforming the unit_sales and where negative making it 0\n\n 4. I have weighted the dataset with perishable column before converting into a Dmatrix for XGboost\n 5. I am using 5fold CV with XgBoost(Dmatrix) with early stopping to train the above two datasets. The results of the first fold is shown below and all the 10 folds ( for both data sets) are consistent\n\n Training dataset with start date: 2017-01-05T00:00:00.000000000 end date: 2017-02-27T00:00:00.000000000\n \n Fold 1\n [0]\ttrain-rmse:1.02536\teval-rmse:1.0255\n Multiple eval metrics have been passed: 'eval-rmse' will be used for early stopping.\n \n Will train until eval-rmse hasn't improved in 50 rounds.\n [50]\ttrain-rmse:0.744308\teval-rmse:0.744343\n [100]\ttrain-rmse:0.725045\teval-rmse:0.724968\n [150]\ttrain-rmse:0.703867\teval-rmse:0.703891\n [156]\ttrain-rmse:0.702914\teval-rms\n e:0.702929\n > The RMSE Score for this round is: 0.49415837094493603\n\nMy first question is: Why is the overall RMSE on the eval dataset lower than the Eval RMSE used by XgBoost's for early stopping? I am not able to figure the reason\n\nAlso, I am splitting the train dataset into two more subsets for testing. SO the model hasn't seen the data\n\n 1. '2017-08-02 to '2017-08-15'\n 2. '2017-03-01' to '2017-03-16'\n\nThe overall RMSE on these two datasets in 0.495 which is consistent with my overall CV score. When I do the same thing on the public test dataset my RMSE is 0.909 ( I am using a exponential transform of the output)\n\nI do not understand what I am doing wrong when the CV and the test data( that I have partitioned from the training set) show the same RMSE but the public test dataset is so different. I have never had such a massive overfitting problem which leads me to think there is something I am doing wrong rather than this being a CV issue. Below is the code I have used for CV\n\nCode for preprocess function:\n\n def preprocess_(df):\n global data_clean,weightData\n df['week']=df['date'].dt.week\n df['day']=df['date'].dt.day_name()\n data_prep=pd.merge(df,data_items,on=\"item_nbr\",how=\"left\")\n data_prep=pd.merge(data_prep,data_stores,on=\"store_nbr\",how=\"left\")\n weightData=1+ (data_prep['perishable']*.25)\n data_clean=pd.get_dummies(data_prep,columns=['onpromotion','day'])\n del data_prep \n data_clean.drop(['id','perishable',],axis=1,inplace=True)\n\nCode for CV: ( The indentation of loops is messed up in Kaggle and I am not able to fix it)\n\n for idx,date in enumerate(train_start_date):\n select_date(train_start_date[idx],train_end_date[idx])\n print(\"Training dataset with\",\"start date:\",min(data_train_sel.date.values),\"end date:\",max(data_train_sel.date.values))\n preprocess_(data_train_sel)\n del data_train_sel\n\n X=data_clean.drop([\"unit_sales\",'date'],axis=1)\n feature_names=X.columns\n y=data_clean['unit_sales']\n del data_clean\n \n X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n del X,y\n K=5\n kf=KFold(n_splits=5,random_state=1234,shuffle=True)\n np.random.seed(100)\n scores=np.zeros(X_train[:].shape[0])\n imp_df=np.zeros((len(feature_names), K))\n \n \n\n for i,(train_index,test_index) in enumerate(kf.split(X_train,y_train)):\n x_train_cv,y_train_cv=X_train.iloc[train_index].copy(),y_train.iloc[train_index].copy()\n w_train_cv = weightData[train_index]\n x_train_cv=xgb.DMatrix(x_train_cv.valueslabel=y_train_cv,feature_names=feature_names,weight=w_train_cv)\n x_eval_cv,y_eval_cv=X_train.iloc[test_index].copy(),y_train.iloc[test_index].copy()\n w_eval_cv = weightData[test_index]\n x_eval_cv=xgb.DMatrix(x_eval_cv.values,label=y_eval_cv,feature_names=feature_names,weight=w_eval_cv)\n watchlist = [(x_train_cv, 'train'), (x_eval_cv, 'eval')]\n print(\"\\nFold\",i+1)\n clf=xgb.train(plst, x_train_cv, num_rounds,watchlist, early_stopping_rounds=50, verbose_eval=50)\n y_pred_cv=clf.predict(x_eval_cv)\n scores[test_index]=metrics.mean_squared_error(y_eval_cv, y_pred_cv)\n print(\"The RMSE Score for this round is:\",metrics.mean_squared_error(y_eval_cv,y_pred_cv))\n \n print ((\"Overall Mean score: {0:.3f} (+/-{1:.3f})\").format(np.mean(scores), sem(scores)))\n prep_predict(X_test,y_test)\n del X_train, X_test, y_train, y_test\n \n del data_train\n" } }