{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# An Unformal Intro 💬\n## 328,770,000,000,000,000,000\n### This is approximately how much data is generated [*](https://explodingtopics.com/blog/data-generated-per-day) in Kilobytes per day ( 328.77 billion GB )","metadata":{}},{"cell_type":"markdown","source":"In a world where **data is king** 👑 and information is overflowing like a waterfall, it's important to know which data types rule the kingdom. Enter tabular and time series data - the dynamic duo of the data realm!\n\nTabular data, the life of every party, comes dressed in a stylish row-column outfit. It's the popular kid in school, with rows strutting their stuff as individual data points and columns showcasing their unique attributes. Think of it as a spreadsheet extravaganza where each row spills the beans on a different customer, while columns spill their secrets about their names, orders, and even the date they decided to splurge!\n\nBut hold on, folks! Time series data steps onto the scene with a swagger that can't be ignored. It's like tabular data's cool cousin, capturing observations over time and bringing the concept of \"chronological charisma\" to the data universe. Imagine a time series of stock prices, where each row unveils the mysteries of opening prices, closing prices, and dates that make the stock market feel like a thrilling roller coaster ride!\n\nBoth tabular and time series data are the superheroes of business and organizations, saving the day with their invaluable insights. Tabular data is the multitasker, helping businesses track inventory, analyze customer behavior, and make marketing magic happen. Meanwhile, time series data swoops in with its superpower of forecasting future trends, identifying problems before they explode, and optimizing operations like a data-driven ninja.\n","metadata":{}},{"cell_type":"markdown","source":"# So what it look like ?\nBefore going to explanations, just look at the below simple data representing  **tabular_data** and **time_series_data**. \nIn the example data - The tabular data consists of information about individuals including their names, ages, and countries, while the time series data captures the stock prices over consecutive dates.","metadata":{}},{"cell_type":"code","source":"# Something like this\n\ntabular_data = [\n    ['Name', 'Age', 'Country'],\n    ['John', 25, 'USA'],\n    ['Emily', 30, 'Canada'],\n    ['Michael', 28, 'UK']\n]\n\ntime_series_data = [\n    ['Date', 'Stock Price'],\n    ['2023-06-01', 50.2],\n    ['2023-06-02', 49.8],\n    ['2023-06-03', 51.5],\n    ['2023-06-04', 52.1]\n]\n","metadata":{"execution":{"iopub.status.busy":"2023-07-04T14:37:17.324842Z","iopub.execute_input":"2023-07-04T14:37:17.325250Z","iopub.status.idle":"2023-07-04T14:37:17.332091Z","shell.execute_reply.started":"2023-07-04T14:37:17.325218Z","shell.execute_reply":"2023-07-04T14:37:17.330915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# The Tab 📅\n### Officialy and unofficialy **The Tabular Data**","metadata":{}},{"cell_type":"markdown","source":"Tabular data, also known as structured data, is like a well-organized spreadsheet. It's arranged in rows and columns, with each row representing a single thing or event, and each column containing different details about those things or events. People use tabular data to store and show all kinds of information, like customer records, financial data, survey responses, and more. It's a convenient and straightforward way to keep track of and present different types of data.\n\n![Tabular Data](https://cloud.google.com/static/vertex-ai/docs/beginner/images/intro_tables-version.png)","metadata":{}},{"cell_type":"markdown","source":"### Pandas\nPandas, the reigning champion of tabular data manipulation in Python, takes the center stage. It presents itself as a powerful dataframe, capable of gracefully handling mixed column types just like the intriguing table we've been exploring. With the addition of an index that elegantly spans across columns and rows, pandas grants us seamless access to specific elements using their respective names. As a true ally in the realm of data science, pandas offers a robust collection of functions tailored for filtering, selecting, and grouping data, making it an incredibly versatile tool for a multitude of analytical tasks.\n\n*Example for Creating Pandas DataFrame from lists of lists*","metadata":{}},{"cell_type":"code","source":"# Import pandas library\nimport pandas as pd\n  \n# initialize list of lists\ndata = [['tom', 10], ['nick', 15], ['juli', 14]]\n  \n# Create the pandas DataFrame\ndf = pd.DataFrame(data, columns=['Name', 'Age'])\n  \n# print dataframe.\ndf","metadata":{"execution":{"iopub.status.busy":"2023-07-04T16:10:24.732597Z","iopub.execute_input":"2023-07-04T16:10:24.732986Z","iopub.status.idle":"2023-07-04T16:10:24.790495Z","shell.execute_reply.started":"2023-07-04T16:10:24.732957Z","shell.execute_reply":"2023-07-04T16:10:24.789426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"However, **every hero has its limitations**. While pandas shines brightly for most endeavors, it does encounter challenges when dealing with larger datasets. When faced with data weighing in at hundreds of megabytes or beyond, pandas may exhibit higher memory consumption and longer processing times for desired operations. It's a tradeoff that arises from pandas' emphasis on flexibility and user-friendly features.\n\n\n\n![Python Data Structures for Tabular Data](https://outerbounds.com/assets/images/python-tabular-data-structures-01-5adb4fc75f04a59f1784fb64a24b2c07.png)\n\n\nNevertheless, fear not! pandas remains a formidable choice for many data science applications, serving as an invaluable foundation for numerous analytical journeys. In scenarios where larger datasets demand efficient processing, exploring alternative tools and techniques could prove fruitful to ensure optimal performance.\n\n### NumPy \n#### Efficient, Interoperable Arrays for Numeric Data\nNumPy is a fast and efficient array library for numeric data. It excels at handling uniform data arrays, like columns in a table. While it offers limited data processing operations compared to pandas, it serves as a seamless bridge between various libraries. NumPy's integration with other data science tools enables efficient data transfer without explicit copying, making it a valuable asset for moving large datasets swiftly.\n### Arrow\n#### Efficient, Interoperable Tables\nApache Arrow is a fast, modern library for tabular data. It can handle mixed columns like pandas and offers faster processing when operations are expressed using its API. With efficient data representation, it allows for larger in-memory loads compared to pandas. Seamless data movement between pandas, Arrow, and NumPy is achieved with zero-copy operations.\n### Scipy.Sparse\n#### Efficient Sparse Arrays for Numeric Data\nThe three libraries pointed are versatile, offering a wide range of use cases with rich APIs and modules. In contrast, Scipy.Sparse is specialized for sparse matrices, where most values are empty. It's particularly useful for handling large datasets with empty column entries, enabling efficient processing and compatibility with libraries like XGBoost and Scikit Learn.\n### Tensors and Other Library-specific Arrays\nModern machine learning libraries like XGBoost, TensorFlow, and PyTorch efficiently handle large datasets. However, optimizing their performance requires configuring suitable data loaders and specialized structures. While library-specific data structures are optimized, they limit generality. Thankfully, efficient interchange between tensors and NumPy arrays is often possible.","metadata":{}},{"cell_type":"markdown","source":"# The Time 📈\n### Officialy and unofficialy **The Time Series Data**\n\nTime series data, like a mesmerizing melody, unfolds as a sequence of time-stamped data points. It captures the rhythm of change, with each point a measurement from a single source, all harmoniously arranged in chronological order.\n\n![](https://www.scylladb.com/wp-content/uploads/time-series-data-diagram.png)","metadata":{}},{"cell_type":"markdown","source":"Apart from being poetic : \n> Time series data, also referred to as time-stamped data, is a sequence of data points indexed in time order. These data points typically consist of successive measurements made from the same source over a fixed time interval and are used to track change over time.\n\n### Time series data that we are a part of ! \n* **Financial time series data** captures the movement of data points, like stock prices, over a specified period. It can be short-term, monitoring prices throughout the day, or long-term, tracking monthly closing prices over years. Collecting both short and long-term data optimizes returns, just as autonomous trading algorithms do.\n* **Healthcare time series data**  captures various patient health metrics, such as ECG readings, COVID-19 hospitalizations and deaths, and patient progress over time. It empowers providers to analyze trends and enhance care delivery.\n* **Self-driving cars** cars gather data on their surroundings, adapting to obstacles, weather, road conditions, and more.\n* **Forensic patterns**. Time-series databases uncover forensic patterns that go beyond simple snapshot numbers. For instance, regular monetary transactions between two individuals at consistent intervals reveal a relationship or agreement, even if they claim otherwise.\n* **Weather data**, such as mean daily temperature (MDT), has long been used to assess energy efficiency for buildings. However, considering detailed time series data, including hourly temperature fluctuations, cloud cover, precipitation, and wind speed, enables better optimization of energy efficiency by accounting for varying environmental factors.","metadata":{}},{"cell_type":"markdown","source":"### There are some important steps (Like always)\nBy following these steps and utilizing techniques like ARIMA, Python facilitates comprehensive time series analysis and forecasting.\n\n#### Stationarity and Autocorrelation\nBefore analyzing time series data, it is crucial to check for stationarity and autocorrelation. Stationarity assesses structural patterns like seasonal trends, while autocorrelation examines the linear dependency of future values on past values. These assumptions are fundamental to widely used methods like ARIMA and linear regression for time series analysis.\n#### Trend Decomposition and Visualization\nTrend decomposition helps visualize patterns in time series data, providing clear insights into their behavior. This step allows for a comprehensive understanding of the underlying trends within the data.\n#### Forecasting with ARIMA\nForecasting future values is an essential aspect of time series analysis. ARIMA (Autoregressive Integrated Moving Average) is a popular technique used for time series forecasting. It enables anticipating future events, aiding decision-making processes","metadata":{}},{"cell_type":"markdown","source":"### Let's spray 🖌️ the **Essay** some **Code** ","metadata":{}},{"cell_type":"markdown","source":"Time series forecasting predicts future values using current and past data. In this case, we will utilize the ARIMA method for forecasting the number of passengers. By leveraging the auto_arima package, we can skip the time-consuming task of hyperparameter tuning.","metadata":{}},{"cell_type":"code","source":"!pip install pmdarima","metadata":{"_kg_hide-input":true,"_kg_hide-output":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/air-passengers/AirPassengers.csv\")\ndf['Month'] = pd.to_datetime(df['Month'], format='%Y-%m')\ndf.index = df['Month']\ndel df['Month']\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-04T16:54:32.745650Z","iopub.execute_input":"2023-07-04T16:54:32.746062Z","iopub.status.idle":"2023-07-04T16:54:32.764046Z","shell.execute_reply.started":"2023-07-04T16:54:32.746020Z","shell.execute_reply":"2023-07-04T16:54:32.763133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport warnings\nwarnings.filterwarnings('ignore') \n\ndf['Date'] = df.index\ntrain = df[df['Date'] < pd.to_datetime(\"1960-08\", format='%Y-%m')]\ntrain['train'] = train['#Passengers']\ndel train['Date']\ndel train['#Passengers']\ntest = df[df['Date'] >= pd.to_datetime(\"1960-08\", format='%Y-%m')]\ndel test['Date']\ntest['test'] = test['#Passengers']\ndel test['#Passengers']\n\nfrom pmdarima.arima import auto_arima\nmodel = auto_arima(train, trace=True, error_action='ignore', suppress_warnings=True)\nmodel.fit(train)\nforecast = model.predict(n_periods=len(test))\nforecast = pd.DataFrame(forecast,index = test.index,columns=['Prediction'])","metadata":{"execution":{"iopub.status.busy":"2023-07-04T17:06:20.262689Z","iopub.execute_input":"2023-07-04T17:06:20.263193Z","iopub.status.idle":"2023-07-04T17:06:35.987198Z","shell.execute_reply.started":"2023-07-04T17:06:20.263153Z","shell.execute_reply":"2023-07-04T17:06:35.985965Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(train, color=\"black\", label=\"Training Data\")\nplt.plot(test, color=\"red\", label=\"Testing Data\")\nplt.plot(forecast, color=\"blue\", label=\"Forecasted Data\")\nplt.title(\"Passenger Prediction\")\nplt.ylabel(\"Actual Passenger\")\nplt.xlabel(\"Year-Month\")\nplt.legend()\nsns.set()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-04T17:04:54.390907Z","iopub.execute_input":"2023-07-04T17:04:54.391350Z","iopub.status.idle":"2023-07-04T17:04:55.047623Z","shell.execute_reply.started":"2023-07-04T17:04:54.391318Z","shell.execute_reply":"2023-07-04T17:04:55.046431Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Time Series Analysis Types\n\n#### Classification: \nInvolves identifying and assigning categories or labels to the data based on specific criteria or patterns observed in the time series. This can be useful for grouping similar data points together or identifying distinct patterns within the dataset.\n#### Curve fitting: \nInvolves fitting a mathematical curve or function to the time series data in order to study the relationships and patterns among the variables within the data. This technique helps in understanding the underlying structure of the data and can assist in making predictions or extrapolations.\n#### Descriptive analysis: \nFocuses on identifying and analyzing patterns and characteristics present in the time series data. This includes examining trends, cycles, seasonal variations, and other patterns that may be inherent in the data. Descriptive analysis provides insights into the behavior of the time series and aids in understanding its overall properties.\n#### Explanative analysis: \nAims to understand the relationships and causal links between variables within the time series data. It involves investigating the factors or events that influence the observed patterns or changes in the data. Explanatory analysis goes beyond describing the patterns and seeks to explain why and how certain phenomena occur within the time series.\n#### Exploratory analysis: \nInvolves exploring the main characteristics and properties of the time series data. This typically includes visualizing the data through charts, graphs, or other visual representations to gain insights into its patterns, distributions, outliers, and other relevant features. Exploratory analysis helps in gaining a preliminary understanding of the data before conducting further analysis.\n#### Forecasting: \nFocuses on predicting future data points in the time series based on historical trends and patterns. It uses statistical models or machine learning algorithms to estimate future values, leveraging the knowledge gained from the past behavior of the time series. Forecasting is useful for making informed decisions, planning, and anticipating future scenarios based on the available historical data.\n#### Intervention analysis: \nExamines the impact of specific events or interventions on the time series data. It seeks to understand how external factors or interventions can affect the patterns, trends, or behavior of the time series. Intervention analysis helps in assessing the effectiveness of interventions or understanding the consequences of certain events on the data.\n#### Segmentation: \nInvolves dividing the time series data into distinct segments or subgroups based on specific criteria or characteristics. This can help in revealing underlying properties, patterns, or behaviors within different segments of the data. Segmentation allows for a more detailed analysis of different aspects or subsets of the time series, providing insights into specific patterns or trends within each segment.\n","metadata":{}},{"cell_type":"markdown","source":"# When the Tab and Time join together 🪄\nTime series forecasting is a valuable tool used by businesses to make predictions about the future based on past data. By analyzing patterns and trends in historical data, businesses can gain insights into potential future outcomes and make informed decisions. However, to further improve the accuracy of these predictions, combining time series data with other relevant data sources is crucial. By integrating additional data, such as economic indicators or customer demographics, businesses can create more robust machine learning models that provide deeper insights and enable better decision-making. This integration allows businesses to consider a wider range of factors that may impact future outcomes and make more accurate and informed predictions.\n\n> For example, in the [Home Credit Default Risk Kaggle competition](https://www.kaggle.com/c/home-credit-default-risk), time-series data of bank transactions are provided to predict credit default risks. This kind of use case can be easily found in the healthcare industry as well. For example, after collecting patients’ time-series data (vital signs, lab results, exam results) and structured data(base information, demography information), we can model on the outcome of clinical procedures and patients’ complication risks.\n> \n> We have two options to deal with the mentioned problems. We can either merge structured data into time-series data or aggregate time-series data into the structured data.\n> \n> To be clarified, we do not need to make a one-off decision, actually, sometimes a combination of both solutions could get more accurate and stable results.","metadata":{}},{"cell_type":"markdown","source":"# Key Differences\n\n* **Data Organization**: \nTabular data, resembling a spreadsheet, arranges information in a structured row-column format. On the other hand, time series data takes a sequential approach, organizing data points based on time intervals, capturing the flow of events over time like a captivating storyline.\n\n* **Purpose**: \nTabular data serves as a versatile tool, accommodating various data needs, storage, and analysis requirements. In contrast, time series data is purpose-built for unraveling the secrets of time, unveiling hidden temporal patterns, and empowering us to make predictions about the future. It's like peering through a crystal ball to uncover the mysteries of time.\n\n* **Data Types**: \nTabular data embraces a diverse range of information, from numerical values to categorical labels and textual descriptions. In contrast, time series data primarily deals with numeric observations, capturing the essence of change and progression over time. It's like witnessing the evolving dance of numbers as time unfolds.\n    \n* **Analysis Techniques**: \nExploring tabular data involves a plethora of techniques, including descriptive statistics, filtering, sorting, and visualizing data from different angles. However, delving into the realm of time series data demands a specialized set of tools and techniques. Time domain analysis allows us to dissect the intricate patterns hidden within the temporal fabric, while stationarity testing helps us uncover the stability of our data. With forecasting models, we can unleash our predictive powers and venture into the realm of future possibilities.\n\n* **Patterns**: \nWithin the realm of tabular data, patterns may arise, but they are not inherently tied to time. In contrast, time series data offers a captivating landscape of temporal patterns, revealing the ebb and flow of seasonality, uncovering trends that emerge over time. It's like decoding the secret language of time, where each moment holds a clue to the larger picture.","metadata":{}},{"cell_type":"markdown","source":"# Conclusion\nTabular and time series data represent two fundamental data formats that offer distinct perspectives and valuable insights. Tabular data provides a structured and organized representation of information, while time series data reveals patterns and trends that unfold over time. Effectively analyzing these data formats requires a combination of statistical techniques, data visualization, and machine learning models. In the realms of business, research, and governance, the importance of harnessing the potential of tabular and time series data cannot be overstated. These data formats serve as invaluable tools for informed decision-making, process optimization, and future prediction. As we continue to witness the exponential growth of data, embracing the possibilities offered by tabular and time series data will be essential in unlocking valuable knowledge and driving progress across various domains.","metadata":{}},{"cell_type":"markdown","source":"# Moving Forward\nFuture research in tabular and time series data holds immense potential for advancing data analysis. Exploring innovative techniques for data organization, integration, and modeling can unlock hidden insights and enhance predictive capabilities. By combining these data formats synergistically, new opportunities for informed decision-making and scientific discoveries can emerge. Investing in research will drive progress across domains, empowering businesses, researchers, and policymakers to harness the full potential of tabular and time series data.","metadata":{}},{"cell_type":"markdown","source":"Reference Links :\n1. https://cloud.google.com/vertex-ai/docs/tabular-data/tabular101  \n2. https://outerbounds.com/docs/python-tabular-data-structures/  \n3. https://www.forbes.com/sites/tomcoughlin/2018/11/27/175-zettabytes-by-2025/?sh=53e887c15459        \n4. https://www.statista.com/statistics/871513/worldwide-data-created/    \n5. https://www.geeksforgeeks.org/different-ways-to-create-pandas-dataframe/         \n6. https://www.scylladb.com/glossary/time-series-data/\n7. https://builtin.com/data-science/time-series-python\n8. https://www.tableau.com/learn/articles/time-series-analysis               \n9. https://dxiaochuan.medium.com/combining-time-series-and-tabular-data-for-prediction-9815a5a17cd\n\nAI Generated ( ChatGPT / Bard ) - Prompt used :\n> Make it interesting to read","metadata":{}},{"cell_type":"code","source":"import pandas as pd\ndf = pd.read_csv('/kaggle/input/2023-kaggle-ai-report/sample_submission.csv')\ndf.iloc[0,1] = 'Tabular and/or time series data'\ndf.iloc[1,1] = 'https://www.kaggle.com/code/abdulkhadarn/the-tab-and-the-time'\ndf.iloc[2,1] = 'https://www.kaggle.com/code/poojach7611/tabular-and-time-series-data/comments#2330191'\ndf.iloc[3,1] = 'https://www.kaggle.com/code/varunkabra123/data-and-time-series/comments#2330207'\ndf.iloc[4,1] = 'https://www.kaggle.com/code/pawanthapaliya/tabular-and-time-series-data/comments#2330226'\ndf.to_csv('submission.csv', index=False)\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-04T20:29:59.378338Z","iopub.execute_input":"2023-07-04T20:29:59.378889Z","iopub.status.idle":"2023-07-04T20:29:59.445231Z","shell.execute_reply.started":"2023-07-04T20:29:59.378862Z","shell.execute_reply":"2023-07-04T20:29:59.444294Z"},"trusted":true},"execution_count":null,"outputs":[]}]}