{"cells":[{"metadata":{"_uuid":"e67cab7753a00a88f6d338a89c6017eb4326e45b"},"cell_type":"markdown","source":"# **Now you see me...TMDB!!!**\n![](https://vuejsexamples.com/content/images/2017/06/656d6f2e676966.gif)"},{"metadata":{"_uuid":"2599b4d39db44b9635864218cfeb463dabb24697"},"cell_type":"markdown","source":"## **1.Load Libraries**"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":false},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n# import kaggle\nimport pandas as pd\nimport matplotlib.pyplot as plt\nplt.style.use('fivethirtyeight')\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\nimport warnings\nwarnings.filterwarnings('ignore')\nimport seaborn as sns\nfrom sklearn import *\nimport xgboost as xg\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true},"cell_type":"markdown","source":"## **2. Load Data**"},{"metadata":{"_uuid":"a636badfdd5840a289f40d6a9a323421e61144f9","trusted":false},"cell_type":"code","source":"train_data = pd.read_csv('../input/train.csv', sep=None, engine='python')\ntest_data = pd.read_csv('../input/test.csv', sep=None, engine='python')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4957830dee48e8369ff577380369a5fb30d4270d"},"cell_type":"markdown","source":"## **3.Data Statistics Analysis**"},{"metadata":{"_uuid":"a4ba5fd5ac36ed1e03e5abcebb795a6864736bae","trusted":false},"cell_type":"code","source":"m_train, n1 = train_data.shape\nm_test, n2 = test_data.shape\nprint(\"{} training samples\".format(m_train))\nprint(\"{} test samples\".format(m_test))\nprint(\"{} features to t\".format(n1))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"13fbdb2f1259c4f0dbd5e816059777fc04f00c61","trusted":false},"cell_type":"code","source":"test_data.dtypes.value_counts().plot(kind=\"barh\", figsize=(20,8))\nfor i, v in enumerate(test_data.dtypes.value_counts()):\n    plt.text(v, i, str(v), fontweight='bold', fontsize = 20)\nplt.xlabel(\"Value Count\")\nplt.ylabel(\"Data Types\")\nplt.title(\"Count Columns By Datatypes\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f059f00a57ac7bdf8e13e91da47f4211c51f0197","trusted":false},"cell_type":"code","source":"print(\"- The target is: \\n{} \".format([item for item in train_data.columns.tolist() if item not in test_data.columns.tolist()]))\nprint(train_data[\"\".join([item for item in train_data.columns.tolist() if item not in test_data.columns.tolist()])].dtypes)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dc9e46d219df0a933965c7b9e081d1588db20c59","trusted":false},"cell_type":"code","source":"train_data.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"37222909b68b20b1c8209e9cbaa3e69c1a787b86"},"cell_type":"markdown","source":"### **Some Insight from Statistics**\n*   We should filter out non-sense revenue values: look at min=1.0\n*   We should filter out non-sense runtime values: look at min=0.0\n*   We should inspect why there are that many budget=0: look at min=25%=0.0\n*   We should understand the meaning the 'popularity' to deal with it: look at max=294.3 vs 75%=10.9"},{"metadata":{"_uuid":"108788e9e470ac285155027e3dcdb8b6d0d86d6c","trusted":false},"cell_type":"code","source":"train_data.describe(include=\"all\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1729c39d3c017480ce21325a50a2b625adf6f01b"},"cell_type":"markdown","source":"### **All Features insight from Statistics**\n*   Be careful with > 'original title': 25 are repeated, 'title': 31 repeated\n*   86% of movies in English\n*   'status' only have two values. 99.8% are Released\n*   To be dropped for sure (4/23): 'homepage' (may be useful to get external data), 'tmdb\\_id' (may be useful to get external data), 'original\\_title' (as we have English title), 'poster\\_path' (useless)."},{"metadata":{"_uuid":"9ade0eae6afde6cdd371242ae07aac8548db9876"},"cell_type":"markdown","source":"## **4.Drop unwanted Columns**"},{"metadata":{"_uuid":"25d207d7f6343b9bc8aba13cf5787d5ba13755a3","trusted":false},"cell_type":"code","source":"# Delete unused columns\ntrain_data.drop(columns=['homepage', 'imdb_id', 'original_title', 'poster_path'], inplace=True)\ntest_data.drop(columns=['homepage', 'imdb_id', 'original_title', 'poster_path'], inplace=True)\n\n# Cast budget and revenue to float\ntrain_data.budget = train_data.budget.astype(float)\ntrain_data.revenue = train_data.revenue.astype(float)\ntest_data.budget = test_data.budget.astype(float)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6097aaa331f4b72803d7dd29b1f281e2913faeb6","trusted":false},"cell_type":"code","source":"train_data.isna().sum().plot(kind=\"barh\", figsize=(20,10))\nfor i, v in enumerate(train_data.isna().sum()):\n    plt.text(v, i, str(v), fontweight='bold', fontsize = 15)\nplt.xlabel(\"Missing Value Count\")\nplt.ylabel(\"Features\")\nplt.title(\"Missing Value count By Features\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a514ab4cb64f06f97d7f1ec735b0d87440581583"},"cell_type":"markdown","source":"## **5. Important Feature Analysis**\n\n**Three main features**   \n**1.Budget**  \n**2.Popularity**  \n**3.Runtime**\n\n### **1.Budget**"},{"metadata":{"_kg_hide-input":true,"_uuid":"3b29db718cfa2b23d3b86c54dd730197897e53e3","trusted":false},"cell_type":"code","source":"# We plot the scatter of the budget / revenue\nplt.figure(figsize=(20,10))\nplt.scatter(x = train_data['budget'], y = train_data['revenue'], marker = 'x', color = 'black')\n\n# We fit a linear model\nmodel = linear_model.HuberRegressor()\nmodel.fit(X = np.array(train_data['budget']).reshape(-1,1), y = train_data['revenue'])\nrev_pred = model.predict(np.array(train_data['budget']).reshape(-1,1))\n\n# We plot the fit \nplt.plot(train_data['budget'], rev_pred, color = 'orange', label = 'linear')\n\n\n# We fit an xgboost model \nparams = {'eval_metric' : 'rmse', 'silent' : 1}\ndtrain = xg.DMatrix(np.array(train_data['budget']).reshape(-1,1), np.array(train_data['revenue']))\nxg_model = xg.train(params, dtrain)\ndtest = xg.DMatrix(np.array(train_data['budget']).reshape(-1,1))\nrev_pred = xg_model.predict(dtest)\n\n# We plot the fit \nplt.scatter(train_data['budget'], rev_pred, color = 'blue', label = 'xgboost')\n\n\nplt.xlabel('budget')\nplt.ylabel('revenue')\nplt.title('Buged impact')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"37c229b606be0b2c87a8df19684450ceaeff18f0"},"cell_type":"markdown","source":"### **2.Popularity**"},{"metadata":{"_kg_hide-input":true,"_uuid":"4dfdd9dbb76c7e7448ecf0efaa057b46564c0ed6","trusted":false},"cell_type":"code","source":"# We plot the scatter of the popularity / revenue\nplt.figure(figsize=(20,10))\nplt.scatter(x = train_data['popularity'], y = train_data['revenue'], marker = 'x', color = 'black')\n\n# We fit a linear model\nmodel = linear_model.HuberRegressor()\nmodel.fit(X = np.array(train_data['popularity']).reshape(-1,1), y = train_data['revenue'])\nrev_pred = model.predict(np.array(train_data['popularity']).reshape(-1,1))\n\n# We plot the fit \nplt.plot(train_data['popularity'], rev_pred, color = 'orange', label = 'linear')\n\n\n# We fit an xgboost model \nparams = {'eval_metric' : 'rmse', 'silent' : 1}\ndtrain = xg.DMatrix(np.array(train_data['popularity']).reshape(-1,1), np.array(train_data['revenue']))\nxg_model = xg.train(params, dtrain)\ndtest = xg.DMatrix(np.array(train_data['popularity']).reshape(-1,1))\nrev_pred = xg_model.predict(dtest)\n\n# We plot the fit \nplt.scatter(train_data['popularity'], rev_pred, color = 'blue', label = 'xgboost')\n\n\nplt.xlabel('popularity')\nplt.ylabel('revenue')\nplt.title('Popularity impact')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7e5c2ae8f6f9371660885dacbdc8b61d7303b727"},"cell_type":"markdown","source":"### **3.Runtime**"},{"metadata":{"_kg_hide-input":true,"_uuid":"243326a2df5957c092a222a8d6af4259a27bdd08","trusted":false},"cell_type":"code","source":"idxs = np.where(train_data['runtime'].isna() == False)[0]\nruntime = np.array(train_data['runtime'][idxs]).reshape(-1,1)\nrevenue = np.array(train_data['revenue'][idxs])\n\n# We plot the scatter of the budget / revenue\nplt.figure(figsize=(20,10))\nplt.scatter(x = runtime, y = revenue, marker = 'x', color = 'black', label = 'runtime')\n\n# We fit a linear model\nmodel = linear_model.HuberRegressor()\nmodel.fit(X = runtime, y = revenue)\nrev_pred = model.predict(runtime)\n\n# We plot the fit \nplt.plot(runtime, rev_pred, color = 'orange', label = 'linear')\n\n\n# We fit an xgboost model \nparams = {'eval_metric' : 'rmse', 'silent' : 1}\ndtrain = xg.DMatrix(runtime, revenue)\nxg_model = xg.train(params, dtrain)\ndtest = xg.DMatrix(runtime)\nrev_pred = xg_model.predict(dtest)\n\n# We plot the fit \nplt.scatter(runtime, rev_pred, color = 'blue', label = 'xgboost')\n\n\nplt.xlabel('runtime')\nplt.ylabel('revenue')\nplt.title('Run time impact')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a704de05c8c582a87c739be2e754d5729d61f4f3"},"cell_type":"markdown","source":"## **6. Deal with NaN**\n\n** Belonging to Collection**"},{"metadata":{"_kg_hide-input":true,"_uuid":"eae2c066d58f4db00f520a64d9fede43be7cea68","trusted":false},"cell_type":"code","source":"# \"belongs_to_collection\": Clean, EDA and create bool > True if belongs to a saga, False if not\ncolor = {'boxes': 'DarkGreen', 'whiskers': 'DarkOrange','medians': 'DarkBlue', 'caps': 'Gray'}\nplt.figure(figsize=(20,10))\nbelongs_df = train_data[['belongs_to_collection', 'revenue']]\nbelongs_df['belongs_to_collection_bool'] = True\nbelongs_df.belongs_to_collection_bool[train_data.belongs_to_collection.isnull()] = False\nbelongs_df['in_saga'] = np.nan\nbelongs_df.in_saga[belongs_df.belongs_to_collection_bool == True] = belongs_df.revenue[belongs_df.belongs_to_collection_bool == True] \nbelongs_df['no_saga'] = np.nan\nbelongs_df.no_saga[belongs_df.belongs_to_collection_bool == False] = belongs_df.revenue[belongs_df.belongs_to_collection_bool == False] \nbelongs_df.belongs_to_collection[~train_data.belongs_to_collection.isnull()] = train_data.belongs_to_collection[~train_data.belongs_to_collection.isnull()].str.split(\"'name': '\").str[1].str.split(\"', 'poster\").str[0]\nbelongs_df.drop(columns=['revenue', 'belongs_to_collection', 'belongs_to_collection_bool'], inplace=True)\nprint(\"Movies in saga: {}/{}={}% \".format(len(~belongs_df.in_saga.isnull()), len(belongs_df), 100.0*len(~belongs_df.in_saga.isnull())/len(belongs_df)))\n# belongs_boxplot = belongs_df.boxplot(figsize=(15, 10), rot=90)\nsns.boxplot(data=belongs_df, orient='h')\nplt.xlabel('value')\nplt.ylabel('Data')\nplt.title('Box plot of Belongig to Collection')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c78e82c50d5d5b17b7edc8334d48c1965b3c332d"},"cell_type":"markdown","source":"### **Genre**"},{"metadata":{"_kg_hide-input":true,"_uuid":"aa7c817d5152df45f24afafabcdfe7d7622cf58f","trusted":false},"cell_type":"code","source":"# \"genres\": Clean, EDA and create bools > True if belongs to a specific genre, False if not\ngenres_df = train_data[['genres', 'revenue']]\ngenres_df['genres_test'] = genres_df['genres']\n# genres_df['number_genres'] = genres_df.genres.str.count(\"'id'\").fillna(0.0).astype(int)\ngenres_df.genres_test = genres_df.genres_test.str.strip('[]')\ngenres_df.genres_test[genres_df.genres_test.isnull()] = ''\ngenres_list = pd.Series(list(set(\", \".join(genres_df.genres_test.unique().tolist()).split('}, ')))).str.split(\"'name': '\").str[1].str.split(\"'\").str[0].tolist()\nfor i, genre in enumerate(genres_list):\n    genres_df[genre] = np.nan\n    genres_df[genre][genres_df.genres_test.str.contains(genre)] = genres_df.revenue[genres_df.genres_test.str.contains(genre)]\ngenres_df.drop(columns=['revenue', 'genres', 'genres_test'], inplace=True)\n# genres_boxplot = genres_df.boxplot(figsize=(15, 10), rot=90)\nplt.figure(figsize=(20,12))\nsns.boxplot(data=genres_df)\nplt.xlabel('Data')\nplt.ylabel('Value')\nplt.title('Box plot of genres')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e62c5f2b8c35d189c31311a5749a03811d8ad2a2"},"cell_type":"markdown","source":"### **Revenue**"},{"metadata":{"_uuid":"0c53a614ba9e8cb1e1538622a97521589785df72","trusted":false},"cell_type":"code","source":"revenue_df = train_data[['title', 'revenue']].sort_values('revenue')\nprint(len(revenue_df[revenue_df.revenue<2e4]))\nrevenue_boxplot = revenue_df[['revenue']][revenue_df.revenue < 1e6]\nplt.figure(figsize=(20,10))\nsns.distplot(revenue_boxplot)\nplt.xlabel('Data')\nplt.ylabel('Value')\nplt.title('Box plot of genres')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5aef2770a0d80ce7b6b5aef57bd3edb6c82446d9"},"cell_type":"markdown","source":"### **Budget**"},{"metadata":{"_uuid":"ef86ea29fcb821e5208e78ac715b6209d8e651b7","trusted":false},"cell_type":"code","source":"budget_df = train_data[['title', 'budget']].sort_values('budget')\nprint(len(budget_df[budget_df.budget<1e5]))\nbudget_boxplot = budget_df[['budget']][budget_df.budget < 1e9]\nplt.figure(figsize=(20,10))\nsns.distplot(budget_boxplot)\nplt.title('Distribution of Budget')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6a53076c1ed2465bb89b9f9c2dd7bd442e797d07"},"cell_type":"markdown","source":"### **Insight**\n\n*   Reason 1: I checked some of the movies with very low budget and it does not coincide with Google info (our data is probably wrong)\n*   Reason 2: my experience says there's no way a movie can cost hundreds of dollars only (or less). Only with camera rental and actors' payroll that should be a couple hundreds/thousands.\n*   Reason 3: check the amount of movies when a) budget<10K -> 835 b)budget<100K -> 849. The distribution points to an error for movies with budget<10K"},{"metadata":{"_uuid":"e83046b9eeabe6c28260bd3e2194f3053583165d"},"cell_type":"markdown","source":"### **Status**"},{"metadata":{"_uuid":"cddc2d532ac208ef0d96a7a7664969ea0a1bc1ac","trusted":false},"cell_type":"code","source":"# \"status\": Study the two existing values\nstatus_df = train_data[['status', 'revenue']]\n\nstatus_df = status_df.groupby('status', as_index=False).agg({'revenue' : ['min', 'max', 'mean', 'count']})\nprint(\"Possible status: {} \".format(status_df.status.unique().tolist()))\nstatus_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f3a66aa0f8d44dcd1b1d52acc75244f7ce3d9ce1"},"cell_type":"markdown","source":"### **Insight**\n\nStatus should not be included as a feature: not enough data points to derive a conclusion while the max and mean values for both categories are acceptable."},{"metadata":{"_uuid":"07a3b54de5ad3bbfb5ca465cf056b7569fd8bb02","trusted":false},"cell_type":"code","source":"popularity_df = train_data[['title', 'popularity']].sort_values('popularity')\nprint(len(popularity_df[popularity_df.popularity<7]))\npopularity_boxplot = popularity_df[['popularity']]\nplt.figure(figsize=(20,8))\nsns.distplot(popularity_boxplot)\nplt.title('Distribution of popularity')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"722381a93e36ca35c8004cb269dac9cfef2b8b24"},"cell_type":"markdown","source":"### **Insight**\n\n* Popularity seems to have coherent values for all movies, where movies with 0 popularity are not popular while with higher popularity are very popular.\n* Approximately half of the training set has a value below 7, while some movies have a value around 200. We should keep this is mind when building the model."},{"metadata":{"_uuid":"ef5541b6bbd4671165445de6190b63d38ef4fd93"},"cell_type":"markdown","source":"### **Title**"},{"metadata":{"_uuid":"2f010768eb1f1d13ee392acd436212785843ef2d","trusted":false},"cell_type":"code","source":"title_df = train_data[['title', 'belongs_to_collection']].sort_values('title')\nprint(\"Non-null titles: {}, Non-null unique titles: {} \".format(len(title_df.title), len(title_df.title.unique())))\nprint(\"Duplicated movies: {} \".format(title_df.title[title_df.duplicated(keep='first')].tolist()))\nprint(\"Titles that belong to a collection {} \".format(len(title_df[~title_df.belongs_to_collection.isnull()])))\nnumber_tuple = (' 0', ' 1', ' 2', ' 3', ' 4', ' 5', ' 6', ' 7', ' 8', ' 9')\ntitle_df = title_df[(~title_df.belongs_to_collection.isnull()) & (title_df.title.str.endswith(number_tuple))]\nprint(\"Titles that belong to a collection and indicate the number of movie {} \".format(len(title_df)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b6f27cfebccdb15889c9e0ea845701168a4cf7c9"},"cell_type":"markdown","source":"### **Insight**\n\n* I don't see the title being a feature worth considering as it gives no important information.\n* Only possible alternative is creating a new feature indicating \"the movie is not the first in the saga\". This may be important because if the first movie has high revenue, the  second may have regardless of other features. There are 57/604=9.4% movies in sagas we can say they are 2nd parts (or 3rd, 4th, etc.)."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}