{"cells":[{"metadata":{"_uuid":"40096fdee8327b917ef0800425dbcddc154a353f"},"cell_type":"markdown","source":"# Summary of Non-stationarities and Anomalies in Feature Time Series"},{"metadata":{"trusted":false,"_uuid":"ea8bda6ff7f2d80a81916fe684c2ae3bed12e246"},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n%matplotlib inline\n\nfrom bokeh.io import output_notebook, show\nfrom bokeh.models import ColumnDataSource, HoverTool, Range1d, Span\nfrom bokeh.plotting import figure\nfrom bokeh.palettes import Set1_9\nfrom bokeh.transform import factor_cmap\n\nimport datetime\n\noutput_notebook()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bed4cf701ebc2df61818f7728a1042d045001baf"},"cell_type":"markdown","source":"After trying a few generic model-building techniques, I started to reflect on my data exploration and processing procedure to find out why I was not able to build better models that not only score higher but also produce predictions that make sense. I soon realised that I missed a very important step in the analysis: compare different parts of the data as well as compare the training set and the test set to see if the data samples are distributed identically and independently, if the training and test samples are distributed similarly, and if there are any inconsistencies preventing us from generalise from one part of the data to another. For any data that can be treated as a time series, if we are to perform any predictive analysis on it, it is important to look out for the stationarity and trend consistency of the series.\n\nNon-stationary time series is problematic for prediction. If the features next year will look completely different from this year, how can we be sure that we can generalise what we learned this year to the next year? Most machine learning algorithms are better at interpolation than extrapolation, so we always have to be careful when the range of the training features and features used to predict do not overlap well. Sudden changes in the time series may also indicate the occurrence of major events that should be taken into account in the analysis.\n\nFor our problem, we first look at the time ranges of the training and test dataset to see whether they are random samples from the same population or non-overlapping samples from different populations:"},{"metadata":{"trusted":false,"_uuid":"ee632391bac5a632fb1ac30cf864c1fc8e4d3ce0"},"cell_type":"code","source":"train = pd.read_pickle('../input/gstore-revenue-data-preprocessing/train.pkl')\ntest = pd.read_pickle('../input/gstore-revenue-data-preprocessing/test.pkl')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4d53387f4922fadfda7c7087b56d783fc31ef7af"},"cell_type":"markdown","source":"As we see below, the training and testing data are split according to time. There is no overlap between them, and apparently they cannot possibly have been drawn from the same sample, and it is likely that something has changed in the behaviour of the features over the transition from the training period to the testing period, and we have to watch out for it."},{"metadata":{"trusted":false,"_uuid":"2be73dfbe392f27d0431bae158b3b7ec890e279e"},"cell_type":"code","source":"train['visitStartTime'].describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"f147679d14467dd286a8d03a3fd878315b3df0d6"},"cell_type":"code","source":"test['visitStartTime'].describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dea03d5aa4162e9dd377e098bd6530b6a05bf41f"},"cell_type":"markdown","source":"Let us record the starting time of the testing samples:"},{"metadata":{"trusted":false,"_uuid":"3adcb1d9363b72118d2ef8eb0fee33a331ba2cf2"},"cell_type":"code","source":"test_start_time = np.min(test['visitStartTime'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"70827fcb1fa9fdc1040b9a1be03546ec3ba75de6"},"cell_type":"code","source":"test_start_time","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8e02ddf2c31ea39b55ccf1bc023c87fdb283df82"},"cell_type":"markdown","source":"### Sessions over time"},{"metadata":{"_uuid":"8769fae74a5c8cd083adcf074288c467c9696c20"},"cell_type":"markdown","source":"Now let us first look at the total sessions over time, including both training and testing data:"},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"0eebc09d37481b8e93f156baabafafd36c91d46f"},"cell_type":"code","source":"train_ts = train.set_index('visitStartTime')\ntest_ts = test.set_index('visitStartTime')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"6f7c4eab4673c5515c319541f98652912e9183be"},"cell_type":"code","source":"train_daily_sess_count = pd.DataFrame({'count': train_ts.groupby(pd.Grouper(freq='D')).size(), 'set': 'train'})\ntest_daily_sess_count = pd.DataFrame({'count': test_ts.groupby(pd.Grouper(freq='D')).size(), 'set': 'test'})\ndaily_sess_count = pd.concat([train_daily_sess_count, test_daily_sess_count], axis=0)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"0b40e22a9b907dc846675a1150cfa388013280d3"},"cell_type":"code","source":"p = figure(x_axis_type='datetime', width=700, height=400, title='daily session count over time')\np.line(\n    source=daily_sess_count[daily_sess_count['set'] == 'train'],\n    x='visitStartTime',\n    y='count',\n    color=Set1_9[0])\np.line(\n    source=daily_sess_count[daily_sess_count['set'] == 'test'],\n    x='visitStartTime',\n    y='count',\n    color=Set1_9[1])\np.legend.location = 'top_left'\ntest_start = Span(location=test_start_time.timestamp() * 1000, dimension='height', line_dash='dashed')\np.add_layout(test_start)\nshow(p)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ed4bb324993195a3f2ac31065de8cfa7b61dbf95"},"cell_type":"markdown","source":"Apart from one obvious outlier, we see that daily sessions fluctuates quite a bit over time. Trend-wise, there is a steep decline around the end of 2016 and a slow increasing trend afterwards. Overall, the a period from 2017 to the end of the training data appears to have more in common with the test portion of the data. I would like to find the reason for the decrease in daily sessions around the end of 2016."},{"metadata":{"_uuid":"198426e08e8885e61cd7314bd345d40e47f8786f"},"cell_type":"markdown","source":"### Pageviews and hits over time:"},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"efe7b30b5a83c46e1713bcb797f10da722c21e4b"},"cell_type":"code","source":"train_daily_pageviews = pd.DataFrame({\n    'count':\n    train_ts.groupby(pd.Grouper(freq='D'))['totals.pageviews'].sum(),\n    'set':\n    'train'\n})\ntest_daily_pageviews = pd.DataFrame({\n    'count':\n    test_ts.groupby(pd.Grouper(freq='D'))['totals.pageviews'].sum(),\n    'set':\n    'test'\n})\ndaily_pageviews = pd.concat(\n    [train_daily_pageviews, test_daily_pageviews], axis=0)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"dc5d67a5007cc355e6d1bae3889b5565b1fa1884"},"cell_type":"code","source":"train_daily_hit_count = pd.DataFrame({\n    'count':\n    train_ts.groupby(pd.Grouper(freq='D'))['totals.hits'].sum(),\n    'set':\n    'train'\n})\ntest_daily_hit_count = pd.DataFrame({\n    'count':\n    test_ts.groupby(pd.Grouper(freq='D'))['totals.hits'].sum(),\n    'set':\n    'test'\n})\ndaily_hit_count = pd.concat(\n    [train_daily_hit_count, test_daily_hit_count], axis=0)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"53be341ac1de0dcd8e563dd699a82c6995c53e31"},"cell_type":"code","source":"p = figure(x_axis_type='datetime', width=700, height=400, title='pageviews over time')\np.line(\n    source=daily_pageviews[daily_pageviews['set'] == 'train'],\n    x='visitStartTime',\n    y='count',\n    color=Set1_9[0])\np.line(\n    source=daily_pageviews[daily_pageviews['set'] == 'test'],\n    x='visitStartTime',\n    y='count',\n    color=Set1_9[1])\np.legend.location = 'top_left'\ntest_start = Span(location=test_start_time.timestamp() * 1000, dimension='height', line_dash='dashed')\np.add_layout(test_start)\nshow(p)\n\np = figure(x_axis_type='datetime', width=700, height=400, title='hits over time')\np.line(\n    source=daily_hit_count[daily_hit_count['set'] == 'train'],\n    x='visitStartTime',\n    y='count',\n    color=Set1_9[0])\np.line(\n    source=daily_hit_count[daily_hit_count['set'] == 'test'],\n    x='visitStartTime',\n    y='count',\n    color=Set1_9[1])\np.legend.location = 'top_left'\ntest_start = Span(\n    location=test_start_time.timestamp() * 1000,\n    dimension='height',\n    line_dash='dashed')\np.add_layout(test_start)\nshow(p)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"59c97314d5384dce1a6cb8f96a3f2a71e42a0c72"},"cell_type":"markdown","source":"We do not see a clear difference in the overall trend between pageviews and hits, but we can see that both have a yearly pattern that starts low at the beginning of the year, climbs throughout the year and drops at the end. This trend appears to be consistent in the slice of data we can see, however unfortunately we only have one cycle within the training set, so the cycle-to-cycle pattern and how that relates to overall revenue cannot be easily discovered."},{"metadata":{"_uuid":"95afdd365a71747ea18ded0e30336e79e676b82d"},"cell_type":"markdown","source":"### Hits/pageviews over time:"},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"1d89e78377b6360b7a7269c124424bf55e2a2321"},"cell_type":"code","source":"train_daily_hitrate = pd.DataFrame({\n    'count':\n    train_ts.groupby(pd.Grouper(freq='D')).apply(\n        lambda x: (x['totals.hits'].sum() + 1) / (x['totals.pageviews'].sum() + 1)\n    ),\n    'set':\n    'train'\n})\ntest_daily_hitrate = pd.DataFrame({\n    'count':\n    test_ts.groupby(pd.Grouper(freq='D')).apply(\n        lambda x: (x['totals.hits'].sum() + 1) / (x['totals.pageviews'].sum() + 1)\n    ),\n    'set':\n    'test'\n})\ndaily_hitrate = pd.concat([train_daily_hitrate, test_daily_hitrate], axis=0)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"5bd5d8558c143a39bef0a7dc832d59db499ccd75"},"cell_type":"code","source":"p = figure(x_axis_type='datetime', width=700, height=400, title='hits/pageviews ratio over time')\np.line(\n    source=daily_hitrate[daily_hitrate['set'] == 'train'],\n    x='visitStartTime',\n    y='count',\n    color=Set1_9[0])\np.line(\n    source=daily_hitrate[daily_hitrate['set'] == 'test'],\n    x='visitStartTime',\n    y='count',\n    color=Set1_9[1])\np.legend.location = 'top_left'\ntest_start = Span(location=test_start_time.timestamp() * 1000, dimension='height', line_dash='dashed')\np.add_layout(test_start)\nshow(p)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"28a820ff34cab7333a883a93037986999cdbfa5c"},"cell_type":"markdown","source":"### Daily mobile percentage over time:"},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"fcc6fb08ecf276131e5ca6928053a9a77e01a84e"},"cell_type":"code","source":"train_daily_mobile_rate = pd.DataFrame({\n    'count':\n    train_ts.groupby(pd.Grouper(freq='D'))['device.isMobile'].mean(),\n    'set':\n    'train'\n})\ntest_daily_mobile_rate = pd.DataFrame({\n    'count':\n    test_ts.groupby(pd.Grouper(freq='D'))['device.isMobile'].mean(),\n    'set':\n    'test'\n})\ndaily_mobile_rate = pd.concat([train_daily_mobile_rate, test_daily_mobile_rate], axis=0)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"f5dc2c05ee9be6b5cb4d89ab1f919f9a97dc44f3"},"cell_type":"code","source":"p = figure(x_axis_type='datetime', width=700, height=400, title='daily mobile percentage over time')\np.line(\n    source=daily_mobile_rate[daily_mobile_rate['set'] == 'train'],\n    x='visitStartTime',\n    y='count',\n    color=Set1_9[0])\np.line(\n    source=daily_mobile_rate[daily_mobile_rate['set'] == 'test'],\n    x='visitStartTime',\n    y='count',\n    color=Set1_9[1])\np.legend.location = 'top_left'\ntest_start = Span(location=test_start_time.timestamp() * 1000, dimension='height', line_dash='dashed')\np.add_layout(test_start)\nshow(p)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a9e7e8226513659a6bd70ba639d1372ae8b4e04d"},"cell_type":"markdown","source":"We might have expected the percentage of mobile users to gradually increase over time, but we also observed a rather peculiar jump around the end of 2016. Does it have more to do with the overall mobile market, or is it just because Google launched or revamped an online store for mobile? Is it related to the dip in daily sessions we see earlier? We have yet to answer these questions.\n\nAnyway, the observation here indicates that the condition for before the end of 2016 and after might be different enough that the usefulness of this feature at predicting revenue might not carry over to afterwards, including the test portion of the data, which is more similar to what happened after 2016. Now I will not be surprised if in some models, removing this feature from data points prior to 2017 actually increases its performance."},{"metadata":{"_uuid":"7839c70ff4d0623ec0e00749172c350995c13afa"},"cell_type":"markdown","source":"### ChannelGrouping categories weight over time:"},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"99ea21698e4d362c6f34d05877964d6320aac7af"},"cell_type":"code","source":"train_daily_channel_grouping = train_ts.groupby([pd.Grouper(freq='D'), 'channelGrouping']).size().reset_index()\ntrain_daily_channel_grouping['set'] = 'train'\ntest_daily_channel_grouping = test_ts.groupby([pd.Grouper(freq='D'), 'channelGrouping']).size().reset_index()\ntest_daily_channel_grouping['set'] = 'test'\ndaily_channel_grouping = pd.concat([train_daily_channel_grouping, test_daily_channel_grouping], axis=0)\ndaily_channel_grouping.rename(columns={0: 'count'}, inplace=True)\ndaily_channel_grouping['perc'] = daily_channel_grouping['count']","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"3bd9ce6ea1337fb5253f463edf3984e897bd4ce4"},"cell_type":"code","source":"daily_channel_grouping_merged = pd.merge(\n    daily_channel_grouping,\n    daily_sess_count.reset_index()[['visitStartTime', 'count'\n                                    ]].rename(columns={'count': 'sess_count'}),\n    on='visitStartTime')\ndaily_channel_grouping_merged[\n    'perc'] = daily_channel_grouping_merged['count'] / daily_channel_grouping_merged['sess_count']","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"b169141ce1b51d3be161db02947a557655514efe"},"cell_type":"code","source":"p = figure(\n    x_axis_type='datetime',\n    width=700,\n    height=600,\n    title='channelGrouping percentage shares over time')\nchannels = list(daily_channel_grouping['channelGrouping'].unique())\nfor idx, channel in enumerate(channels):\n    p.line(\n        source=daily_channel_grouping_merged[\n            (daily_channel_grouping_merged['channelGrouping'] == channel)\n            & (daily_channel_grouping_merged['perc'] < 1)],\n        x='visitStartTime',\n        y='perc',\n        color=Set1_9[idx],\n        legend=channel)\np.y_range = Range1d(0, 1)\np.legend.click_policy = 'hide'\np.add_tools(\n    HoverTool(tooltips=[('category',\n                         '@channelGrouping'), ('percentage',\n                                               '@perc'), ('count', '@count')]))\ntest_start = Span(\n    location=test_start_time.timestamp() * 1000,\n    dimension='height',\n    line_dash='dashed')\np.add_layout(test_start)\nshow(p)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bd366e299eb6326ca3ba2a326978b3e85552074d"},"cell_type":"markdown","source":"**(Click on the legend to hide one or more lines in the plot)**\n\nApparently a lot was going on at the end of 2016. Not only did 'Social' channel suffered a massive drop in percentage share, 'Organic Search' recovered from a dip and the 'Direct' channel enjoyed a moderate increase. This is also around the time that the overall session count decreased, so the drop in the absolute number of 'Social' channel is likely to driving force for the series of changes. 'Social' channel further suffered a drop around May of 2017. We also noticed that the 'Display' channel enjoyed a massive share spike in the test period, which had not happened at all throughout the training peroid. Does this mean we have to treat each of the channels differently and assign them different 'usefulness rating' for predicting user revenue? It appears to be the case here. Treating this attribute as a single column looks like a less appealing option after seeing this."},{"metadata":{"_uuid":"432fcc79644e3b7335ead58ae8aa797bb8f1d221"},"cell_type":"markdown","source":"### Top browser weight over time:"},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"b5b5e0b25cb747269b0e699384fc2cc78dc51716"},"cell_type":"code","source":"train_daily_browser = train_ts.groupby([pd.Grouper(freq='D'), 'device.browser']).size().reset_index()\ntrain_daily_browser['set'] = 'train'\ntest_daily_browser = test_ts.groupby([pd.Grouper(freq='D'), 'device.browser']).size().reset_index()\ntest_daily_browser['set'] = 'test'\ndaily_browser = pd.concat([train_daily_browser, test_daily_browser], axis=0)\ndaily_browser.rename(columns={0: 'count'}, inplace=True)\ndaily_browser['perc'] = daily_browser['count']\n\ndaily_browser_merged = pd.merge(\n    daily_browser,\n    daily_sess_count.reset_index()[['visitStartTime', 'count'\n                                    ]].rename(columns={'count': 'sess_count'}),\n    on='visitStartTime')\ndaily_browser_merged[\n    'perc'] = daily_browser_merged['count'] / daily_browser_merged['sess_count']","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"c1c62f9ccaaa288fa9893f0693926623c1cbd384"},"cell_type":"code","source":"top_browsers = set(train['device.browser'].value_counts().index[:5])\ntop_browsers","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"61001d32f57540e39b184d4175aaee6b9345c362"},"cell_type":"code","source":"p = figure(\n    x_axis_type='datetime',\n    width=700,\n    height=600,\n    title='browser percentage shares over time')\nfor idx, browser in enumerate(top_browsers):\n    p.line(\n        source=daily_browser_merged[\n            (daily_browser_merged['device.browser'] == browser)\n            & (daily_browser_merged['perc'] < 1)],\n        x='visitStartTime',\n        y='perc',\n        color=Set1_9[idx],\n        legend=browser)\np.y_range = Range1d(0, 1)\np.legend.click_policy = 'hide'\np.add_tools(\n    HoverTool(tooltips=[('category',\n                         '@{device.browser}'), ('percentage',\n                                                '@perc'), ('count',\n                                                           '@count')]))\ntest_start = Span(\n    location=test_start_time.timestamp() * 1000,\n    dimension='height',\n    line_dash='dashed')\np.add_layout(test_start)\nshow(p)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"059c36a61bde2c99d21ebeb49e4c497ef3f3de9b"},"cell_type":"markdown","source":"We can see from this plot that the browser percentage shares are not behaving normally around a short period of time, but otherwise appear to be someewhat stationary over time."},{"metadata":{"_uuid":"fa0947f27b9f7299470c7ec6f1ab8d4aa7963555"},"cell_type":"markdown","source":"### Top OS weight over time:"},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"3c19318b7af76472cb5423661cd728f71da171b6"},"cell_type":"code","source":"train_daily_os = train_ts.groupby([pd.Grouper(freq='D'), 'device.operatingSystem']).size().reset_index()\ntrain_daily_os['set'] = 'train'\ntest_daily_os = test_ts.groupby([pd.Grouper(freq='D'), 'device.operatingSystem']).size().reset_index()\ntest_daily_os['set'] = 'test'\ndaily_os = pd.concat([train_daily_os, test_daily_os], axis=0)\ndaily_os.rename(columns={0: 'count'}, inplace=True)\ndaily_os['perc'] = daily_os['count']\n\ndaily_os_merged = pd.merge(\n    daily_os,\n    daily_sess_count.reset_index()[['visitStartTime', 'count'\n                                    ]].rename(columns={'count': 'sess_count'}),\n    on='visitStartTime')\ndaily_os_merged[\n    'perc'] = daily_os_merged['count'] / daily_os_merged['sess_count']","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"f5fd685302d6362af6d2e47d1ba846e64c4e5249"},"cell_type":"code","source":"top_os = set(train['device.operatingSystem'].value_counts().index[:6])\ntop_os","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"06ee7d12bb1e2ff76abf0d074377623e192d2d89"},"cell_type":"code","source":"p = figure(\n    x_axis_type='datetime',\n    width=700,\n    height=600,\n    title='OS percentage shares over time')\nfor idx, os in enumerate(top_os):\n    p.line(\n        source=daily_os_merged[\n            (daily_os_merged['device.operatingSystem'] == os)\n            & (daily_os_merged['perc'] < 1)],\n        x='visitStartTime',\n        y='perc',\n        color=Set1_9[idx],\n        legend=os)\np.y_range = Range1d(0, 0.7)\np.legend.click_policy = 'hide'\np.add_tools(\n    HoverTool(tooltips=[('category',\n                         '@{device.operatingSystem}'), ('percentage',\n                                                        '@perc'), ('count',\n                                                                   '@count')]))\ntest_start = Span(\n    location=test_start_time.timestamp() * 1000,\n    dimension='height',\n    line_dash='dashed')\np.add_layout(test_start)\nshow(p)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"284a0fc3aff47981d4f2fcbc9f2e905e877ecd1d"},"cell_type":"markdown","source":"Here we see a large jump of mobile OS share around the same time we saw the overall mobile percentage numbers jump earlier. There are also a few spikes for Android in the test portion of the data, which is quite unlike what it looked like before. There is also a spike followed by continue decline for Mac users."},{"metadata":{"_uuid":"93b96757a5b723a240ef381bbaa79a3e11da8065"},"cell_type":"markdown","source":"### Revenue over time?"},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"d1aca95da2f37f1bbdc4620833273876555c5174"},"cell_type":"code","source":"train_daily_revenue = pd.DataFrame({\n    'sum':\n    train_ts.groupby(pd.Grouper(freq='D'))['totals.transactionRevenue'].sum(),\n    'set':\n    'train'\n})\ntest_daily_revenue = pd.DataFrame({'dummy': test_ts.groupby(pd.Grouper(freq='D')).size(), 'set': 'test'})\ntest_daily_revenue['sum'] = train_daily_revenue['sum'].mean()\ntest_daily_revenue.drop('dummy', axis=1, inplace=True)\ndaily_revenue = pd.concat([train_daily_revenue, test_daily_revenue], axis=0, sort=True)\n\np = figure(\n    x_axis_type='datetime',\n    width=700,\n    height=400,\n    title='revenue over time?')\np.line(\n    source=daily_revenue[daily_revenue['set'] == 'train'],\n    x='visitStartTime',\n    y='sum',\n    color=Set1_9[0])\np.line(\n    source=daily_revenue[daily_revenue['set'] == 'test'],\n    x='visitStartTime',\n    y='sum',\n    color=Set1_9[1])\np.legend.location = 'top_left'\ntest_start = Span(\n    location=test_start_time.timestamp() * 1000,\n    dimension='height',\n    line_dash='dashed')\np.add_layout(test_start)\nshow(p)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9dcc6756052a7aa2bc077aa0b2485460560a9fea"},"cell_type":"markdown","source":"As we see above, many of the attributes are either non-stationary, have peculiar trends during a certain period of time, or behaves quite differently in the training and test set. Surely, some of these non-stationarities and train/test distinctions will have to be taken into account for making predictions about future.\n\n(Many questions to answer still)"},{"metadata":{"trusted":false,"_uuid":"0abc01afcb15b917d47b3c8b13f36378989a5974"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"hide_input":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}