{"cells":[{"metadata":{"id":"IVALDdaVnJBP","colab_type":"text","_uuid":"fe290ae2090acae98f459a2c5bfd54cecdbf9635"},"cell_type":"markdown","source":"<a name=\"series\"></a>\n## Data Series\nData series are one of the fundamental data structures in Pandas. You can think of them like a dictionary; they have a key (index) and value (data/values) like a dictionary, but also have some handy functionality attached to them.\n\nTo start out, let's create a series from scratch. We'll imagine these are temperature observations."},{"metadata":{"id":"qdI27uY0nJBh","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":111},"outputId":"43c6914b-b3ad-41de-e723-f997540e17af","trusted":false,"_uuid":"9ab58575c7f9a37a1740b88f86866a39a88284c9"},"cell_type":"code","source":"from pandas import Series\ntemperatures = Series([23, 20, 25, 18])\ntemperatures","execution_count":null,"outputs":[]},{"metadata":{"id":"IA8ti7c-nJCH","colab_type":"text","_uuid":"ab58eea4cd0a080fb72a8efad23278229d984bb3"},"cell_type":"markdown","source":"The values on the left are the index (zero based integers by default) and on the right are the values. Notice that the data type is an integer. Any NumPy datatype is acceptable in a series.\n\nThat's great, but it'd be more useful if the station were associated with those values. In fact you could say we want the values *indexed* by station name."},{"metadata":{"id":"MBuI0UIbnJCT","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":111},"outputId":"5dcd5acf-04a3-496d-92e3-44d8059e26d8","trusted":false,"_uuid":"39780b0eaeb96730b6d58d483db737f289da8823"},"cell_type":"code","source":"temperatures = Series([23, 20, 25, 18], index=['TOP', 'OUN', 'DAL', 'DEN'])\ntemperatures","execution_count":null,"outputs":[]},{"metadata":{"id":"9HM2Xi5inJC0","colab_type":"text","_uuid":"bbd430f2a139600e975d5758cdffaa9d1f922fa1"},"cell_type":"markdown","source":"Now, very similar to a dictionary, we can use the index to access and modify elements."},{"metadata":{"id":"TJaaC2oxnJC3","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":36},"outputId":"809ae796-06a3-49d8-d4a6-6893a6f6b4c9","trusted":false,"_uuid":"a3414d7ef4a1514f5b851c7a0d704e362bf70cb2"},"cell_type":"code","source":"temperatures['DAL']","execution_count":null,"outputs":[]},{"metadata":{"id":"0itqP1uenJDC","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":73},"outputId":"99055265-e8f1-4fe7-bfdd-a4308332fb7f","trusted":false,"_uuid":"aef184b86eacadfe54a7ab1b18c4dfd3078eaa59"},"cell_type":"code","source":"temperatures[['DAL', 'OUN']]","execution_count":null,"outputs":[]},{"metadata":{"id":"fE4a2c9xnJDW","colab_type":"text","_uuid":"10e42d2bdfc23d3df805bd05215d16fa7d7867d2"},"cell_type":"markdown","source":"We can also do basic filtering, math, etc."},{"metadata":{"id":"T6CB0ZCdnJDZ","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":73},"outputId":"eb8362ee-a082-4123-b99c-9e45ffed6f25","trusted":false,"_uuid":"d941ce0fd76fc89bb187688b8f979a14833da534"},"cell_type":"code","source":"temperatures[temperatures > 20]","execution_count":null,"outputs":[]},{"metadata":{"id":"hHJl1KyRnJDo","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":111},"outputId":"db8deb79-dbef-4e42-c461-f3f16451af5f","trusted":false,"_uuid":"ea2891dc85123e1e8f3ab5b6139b435950624ef5"},"cell_type":"code","source":"temperatures + 2","execution_count":null,"outputs":[]},{"metadata":{"id":"ZJSMLcYBnJD7","colab_type":"text","_uuid":"5102559dd71b41f212755a04f2380134b27d9209"},"cell_type":"markdown","source":"Remember how I said that series are like dictionaries? We can create a series striaght from a dictionary."},{"metadata":{"id":"rt0DellSnJEA","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":129},"outputId":"fff402a2-fac3-4aee-89b2-e2fbd827cbf4","trusted":false,"_uuid":"3ad4dd62b1f53d3df2a44d2fede535220a3c9968"},"cell_type":"code","source":"dps = {'TOP': 14,\n       'OUN': 18,\n       'DEN': 9,\n       'PHX': 11,\n       'DAL': 23}\n\ndewpoints = Series(dps)\ndewpoints","execution_count":null,"outputs":[]},{"metadata":{"id":"b7FfSu5UnJEQ","colab_type":"text","_uuid":"d9bdaf083edac3306953a16d1ca88283132ed59f"},"cell_type":"markdown","source":"It's also easy to check and see if an index exists in a given series:"},{"metadata":{"id":"b31GoHFBnJEV","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":36},"outputId":"3ccd5cac-8ed4-4a87-e12e-03193c240c29","trusted":false,"_uuid":"ed9bdb25200f264378c4b3f10adbea34403bc743"},"cell_type":"code","source":"'PHX' in dewpoints","execution_count":null,"outputs":[]},{"metadata":{"id":"a3dIlxXKnJEr","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":36},"outputId":"3019bb2a-692c-4f17-bcc8-d6097167217a","trusted":false,"_uuid":"5b0151ff450afd6490e4647f2790923583b0cba2"},"cell_type":"code","source":"'PHX' in temperatures","execution_count":null,"outputs":[]},{"metadata":{"id":"id1a9xV-nJE6","colab_type":"text","_uuid":"10d9e5c9cf6fd5ac7f86e7a0da0257d384fa8f11"},"cell_type":"markdown","source":"Series have a name attribute and their index has a name attribute."},{"metadata":{"id":"09JTsaW9nJE9","colab_type":"code","colab":{},"trusted":false,"_uuid":"7dae0ef485275a9f2b465e5d2ae53c40a28d4894"},"cell_type":"code","source":"temperatures.name = 'temperature'\ntemperatures.index.name = 'station'","execution_count":null,"outputs":[]},{"metadata":{"id":"KVfHIGBPnJFQ","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":129},"outputId":"571bf291-9224-4d35-ef4e-2fa4cb43eab1","trusted":false,"_uuid":"9507c77741f6113966360b6be2dc94debff811d0"},"cell_type":"code","source":"temperatures","execution_count":null,"outputs":[]},{"metadata":{"id":"RyViInn0nJF5","colab_type":"text","_uuid":"7ae6810975c89a622c3409c39d4809569244bfe3"},"cell_type":"markdown","source":"<a name=\"frames\"></a>\n## Data Frames\nSeries are great, but what about a bunch of related series? Something like a table or a spreadsheet? Enter the data frame. A data frame can be thought of as a dictionary of data series. They have indexes for their rows and their columns. Each data series can be of a different type , but they will all share a common index.\n\nThe easiest way to create a data frame by hand is to use a dictionary."},{"metadata":{"id":"9qmhnOjVnJF7","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":175},"outputId":"71a30033-6e34-4668-d375-4b3387f55e87","trusted":false,"_uuid":"e24b7c958faa7144461abe42a2765a7d29b9c607"},"cell_type":"code","source":"from pandas import DataFrame\n\ndata = {'station': ['TOP', 'OUN', 'DEN', 'DAL'],\n        'temperature': [23, 20, 25, 18],\n        'dewpoint': [14, 18, 9, 23]}\n\ndf = DataFrame(data)\ndf","execution_count":null,"outputs":[]},{"metadata":{"id":"FkCx24TBnJGM","colab_type":"text","_uuid":"7586436ff04093587cd367cfd97591a40bea7dae"},"cell_type":"markdown","source":"You can access columns (data series) using dictionary type notation or attribute type notation."},{"metadata":{"id":"TgiwHFyInJGP","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":111},"outputId":"f551471e-97d9-4698-ef27-ec44e7b7e43e","trusted":false,"_uuid":"07bc5b8f9902b76923920a8a503f7aa120a00d72"},"cell_type":"code","source":"df['temperature']","execution_count":null,"outputs":[]},{"metadata":{"id":"vk2uHakynJGj","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":111},"outputId":"daea55a0-dad4-4a53-b079-02dcdf61276b","trusted":false,"_uuid":"41c4eab9a72e6e1d5b0939506b4bbae62bd9137b"},"cell_type":"code","source":"df.dewpoint","execution_count":null,"outputs":[]},{"metadata":{"id":"TJgjFunNnJGu","colab_type":"text","_uuid":"aa500df1a51b91e615d08922b326cabfd21e6150"},"cell_type":"markdown","source":"Notice the index is shared and that the name of the column is attached as the series name.\n\nYou can also create a new column and assign values. If I only pass a scalar it is duplicated."},{"metadata":{"id":"9wRHgMTxnJGx","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":175},"outputId":"9c6c8968-82fe-4a8b-f0bb-177207dc2613","trusted":false,"_uuid":"e9162b69448c151052723f6cf53448462c782c9d"},"cell_type":"code","source":"df['wspeed'] = 0.\ndf","execution_count":null,"outputs":[]},{"metadata":{"id":"0pV4urT2nJHJ","colab_type":"text","_uuid":"0c756deab0bdb09a869e96198903cb5062dd9cdd"},"cell_type":"markdown","source":"Let's set the index to be the station."},{"metadata":{"id":"4UONKZyInJHR","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":206},"outputId":"a1c63328-cde7-47ae-d6b3-576a7388e1d4","trusted":false,"_uuid":"1d9371bcda406b3856f39fe0ce13891aaebe50ef"},"cell_type":"code","source":"df.index = df.station\ndf","execution_count":null,"outputs":[]},{"metadata":{"id":"nPEsNDz3nJHj","colab_type":"text","_uuid":"04a5b9e7e9bdd5e9538d224765f6e3ffdb816f44"},"cell_type":"markdown","source":"Well, that's close, but we now have a redundant column, so let's get rid of it."},{"metadata":{"id":"P0-sJWaynJHu","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":206},"outputId":"431bb2bc-449e-4490-ead4-ec715b26e9a0","trusted":false,"_uuid":"5060acc1a74a6acd2246f741d6eed81caa6e91ae"},"cell_type":"code","source":"df.drop('station', 1, inplace=True)\ndf","execution_count":null,"outputs":[]},{"metadata":{"id":"9QyefX6knJH3","colab_type":"text","_uuid":"d45c583ce439c1e3cd21976fdcd7c67fd22a8141"},"cell_type":"markdown","source":"Now let's get a row from the dataframe instead of a column."},{"metadata":{"id":"9RbvCmJOnJH8","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":92},"outputId":"d7efee5f-287b-45f7-f98a-0cf1b706d33b","trusted":false,"_uuid":"9dcaa7e80c514a324ace658e85b574a9f747181e"},"cell_type":"code","source":"df.loc['DEN']","execution_count":null,"outputs":[]},{"metadata":{"id":"aDPaa2V-nJIB","colab_type":"text","_uuid":"478379696e46f704c56f2b9284730d10a9224e61"},"cell_type":"markdown","source":"We can even transpose the data easily if we needed that do make things easier to merge/munge later."},{"metadata":{"id":"B1By2qTknJIO","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":143},"outputId":"471c9e97-6bc2-495a-a110-cd6920a8ddf7","trusted":false,"_uuid":"421a4632db496717b27155b60d1520330cafb41c"},"cell_type":"code","source":"df.T","execution_count":null,"outputs":[]},{"metadata":{"id":"KhDgMkwanJIk","colab_type":"text","_uuid":"34ded25d4c8526929ce00197de59e01eadf549ab"},"cell_type":"markdown","source":"Look at the `values` attribute to access the data as a 1D or 2D array for series and data frames recpectively."},{"metadata":{"id":"9pi-9kCCnJIn","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":92},"outputId":"3ce322fd-397d-4ef3-a076-703eedf2df92","trusted":false,"_uuid":"fb0bf3bb1a5cb7e3b88f304d651321bc54bfe498"},"cell_type":"code","source":"df.values","execution_count":null,"outputs":[]},{"metadata":{"id":"twayJLjOnJJE","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":36},"outputId":"653ddf81-3fd0-472d-8764-b276dc9c5cca","trusted":false,"_uuid":"d96ae3a9653c80c189ad760c2f3adb8a23cf417f"},"cell_type":"code","source":"df.temperature.values","execution_count":null,"outputs":[]},{"metadata":{"id":"0VdmYGRFnJJa","colab_type":"text","_uuid":"7c52dc5d67925169d174d0de272524671f5ab7b7"},"cell_type":"markdown","source":"<a name=\"loading\"></a>\n## Loading Data in Pandas\nThe real power of pandas is in manupulating and summarizing large sets of tabular data. To do that, we'll need a large set of tabular data. We've included a file in this directory called `JAN17_CO_ASOS.txt` that has all of the ASOS observations for several stations in Colorado for January of 2017. It's a few hundred thousand rows of data in a tab delimited format. Let's load it into Pandas."},{"metadata":{"id":"uhxAQGOwnJJf","colab_type":"code","colab":{},"trusted":true,"_uuid":"87ba5115e1e3dce63fe2682d9f0358cc693ec81d"},"cell_type":"code","source":"import pandas as pd","execution_count":null,"outputs":[]},{"metadata":{"id":"PMw0_CVonJJn","colab_type":"code","colab":{},"trusted":true,"_uuid":"e5cac3ef9b5ff04534aa5457f455aecbd74ef55e"},"cell_type":"code","source":"df = pd.read_csv('../input/train_V2.csv')","execution_count":null,"outputs":[]},{"metadata":{"id":"VOiG0mFYnJJq","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":206},"outputId":"454ac36e-80e8-450b-f8c3-f8d0bb859a99","trusted":true,"_uuid":"5bde340063a21819e7ba377cdf00fab70f5cd76a"},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"S2mkpsFjnJKM","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":92},"outputId":"9bd40595-e3aa-4a04-eb63-ac395bf96e72","trusted":true,"_uuid":"b8f6af617be58a458852496a32f8c201bd39b2da"},"cell_type":"code","source":"df.columns","execution_count":null,"outputs":[]},{"metadata":{"id":"wcKJkh6ZnJKu","colab_type":"text","_uuid":"ae324d2264eb32bf5b99d71ab7fefa21dcc19fcd"},"cell_type":"markdown","source":"<a name=\"missing\"></a>\n## Missing Data\nWe've already dealt with some missing data by turning the 'M' string into actual NaN's while reading the file in. We can do one better though and delete any rows that have all values missing. There are similar operations that could be performed for columns. You can even drop if any values are missing, all are missing, or just those you specify are missing."},{"metadata":{"id":"jAsqjHKpnJKw","colab_type":"code","colab":{},"trusted":true,"_uuid":"8aeb236f4b9b47a985d4f9482cf5c8e180e47883"},"cell_type":"code","source":"len(df)","execution_count":null,"outputs":[]},{"metadata":{"id":"0G_8xI57nJK8","colab_type":"code","colab":{},"trusted":true,"_uuid":"b1b27431a6a18ab33ac995d58c18d45b5adf47c0"},"cell_type":"code","source":"df.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"id":"gSDGYmMlnJLA","colab_type":"code","colab":{},"trusted":false,"_uuid":"25a0e5362457b17663dcef5e8eb8c59c6c2f4a1f"},"cell_type":"code","source":"len(df)","execution_count":null,"outputs":[]},{"metadata":{"id":"9bRJPrCxnJLG","colab_type":"code","colab":{},"trusted":false,"_uuid":"a39611a78308d506e2a1ba8595b9702fcb80ccf2"},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"3J4oI5hknJMQ","colab_type":"code","colab":{},"trusted":false,"_uuid":"504145116e237064c7001903a720e0ee647aedf0"},"cell_type":"code","source":"df.reset_index(drop=True, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"id":"ZALnTk_RnJMZ","colab_type":"code","colab":{},"trusted":false,"_uuid":"2eb2a6f240114bfdf4d6b1bc420ab52f14fcdff0"},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"sP79o15onJMf","colab_type":"text","_uuid":"baf5ca66cc40b66be1dc6d36c4765b26d704d2a5"},"cell_type":"markdown","source":"<a href=\"#top\">Top</a>\n<hr style=\"height:2px;\">"},{"metadata":{"id":"orJiDh7ynJMj","colab_type":"text","_uuid":"bb7e588da7afc8d9774bfe0c66fc475259e13c80"},"cell_type":"markdown","source":"<a name=\"manipulating\"></a>\n## Manipulating Data\nWe can now take our data and do some intersting things with it. Let's start with a simple min/max."},{"metadata":{"id":"QViVr61LnJM7","colab_type":"code","colab":{},"trusted":true,"_uuid":"4ed2e13bd64bb21a6c18d39738c3635d6c81f617"},"cell_type":"code","source":"print('Min: {}\\nMax: {}'.format(df.damageDealt.min(), df.damageDealt.max()))","execution_count":null,"outputs":[]},{"metadata":{"id":"ym1hH5eInJND","colab_type":"text","_uuid":"bc24e437087e6dc183ac69b35d9afc8cc108f611"},"cell_type":"markdown","source":"You can also do some useful statistics on data with attached methods like corr for correlation coefficient."},{"metadata":{"id":"2B-zHAI_nJNF","colab_type":"code","colab":{},"trusted":true,"_uuid":"5dcd29a2ea9988650ae53c558425fb62dd29542e"},"cell_type":"code","source":"df.teamKills.corr(df.damageDealt)","execution_count":null,"outputs":[]},{"metadata":{"id":"sv-vkdycnJNs","colab_type":"text","_uuid":"8f6f5f9ecc85373fe98a9cb7a0cf6dec2862e368"},"cell_type":"markdown","source":"We can also call a `groupby` on the data frame to start getting some summary information for each station."},{"metadata":{"id":"IeNrhB13nJNz","colab_type":"code","colab":{},"trusted":true,"_uuid":"d2e672b1bd385e6355dce314b7f6470469c3aa0f"},"cell_type":"code","source":"df.groupby('teamKills').mean()","execution_count":null,"outputs":[]},{"metadata":{"id":"1ayZOOb6nJPX","colab_type":"code","colab":{},"trusted":true,"_uuid":"c7b323d8d6ba12216f1b25187e5870f234b2a034"},"cell_type":"code","source":"df.groupby('teamKills').describe()","execution_count":null,"outputs":[]},{"metadata":{"id":"JKD9pWgQnJQa","colab_type":"text","_uuid":"1b74d1763fed8fd4866474222d97d02617766400"},"cell_type":"markdown","source":"<a href=\"#top\">Top</a>\n<hr style=\"height:2px;\">"}],"metadata":{"colab":{"name":"00-pandas-intro.ipynb","version":"0.3.2","provenance":[]},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}