{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"> Dan Heath quotes, \"**Data are just summaries of thousands of stories-tell a few of those stories to help make the data meaningful.**\"\n\nResearch in this context aims to focus on two important aspects:\n* To leverage AI/ML to extract meaningful features that are more relevant to grasp the hacks to proceed to solving a problem that has complex patterns often difficult to interpret.\n* To find the most secured and robust form of data intended to be private to an agency or an organization to forecast,classify and summarize the outcomes of a desired task using AI/ML to proceed to moving on to a subsequent action.\n\n\nText Data is thus a common mode of obtaining vital data used to derive AI models with appropriate encoding facilitated in addition by XML.\n\n(Further Reading:\nhttps://aws.amazon.com/what-is/xml/\n)\n\n> \"Quotes Gram\" displays a funny quote that goes by, \"**What's freaking us out here is that we've found a correlation between owning cats and being struck by lightning.**\"\n\nIrrelevant relationship fit is often a common problem encountered when Text Data is utilized in the process of building a model to be deployed for business or scientific scenarios.\n","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport json","metadata":{"execution":{"iopub.status.busy":"2023-07-05T04:31:31.066836Z","iopub.execute_input":"2023-07-05T04:31:31.067348Z","iopub.status.idle":"2023-07-05T04:31:31.074453Z","shell.execute_reply.started":"2023-07-05T04:31:31.067311Z","shell.execute_reply":"2023-07-05T04:31:31.073047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kaggle_summary=pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/kaggle_writeups_20230510.csv\")","metadata":{"execution":{"iopub.status.busy":"2023-07-05T04:31:31.076761Z","iopub.execute_input":"2023-07-05T04:31:31.077341Z","iopub.status.idle":"2023-07-05T04:31:31.325500Z","shell.execute_reply.started":"2023-07-05T04:31:31.077304Z","shell.execute_reply":"2023-07-05T04:31:31.324125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Kaggle made it interesting and easy for students exploring a Data Science and Machine Learning path, Scientists investigating Machine Learning's aid to complex computation required to assess orientation and probability of occurrence of events,Data Scientists and Machine Learning Engineers to arrive at improving measures to deal with bulky datasets.The key requirement is to make optimal use of processors on the cloud as Machine Learning Models are trained. The competitions organized so far in collaboration with agencies provided real time *Text Data* in different formats and proved understanding the hidden test case practices to be dealt with for successful submission to the competitions adds to discovery of new methods by the community while refining a study.","metadata":{}},{"cell_type":"code","source":"contest_urls=kaggle_summary['Competition URL'].unique()","metadata":{"execution":{"iopub.status.busy":"2023-07-05T04:31:31.327035Z","iopub.execute_input":"2023-07-05T04:31:31.327425Z","iopub.status.idle":"2023-07-05T04:31:31.332792Z","shell.execute_reply.started":"2023-07-05T04:31:31.327392Z","shell.execute_reply":"2023-07-05T04:31:31.331842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"count_unique=kaggle_summary['Title of Competition'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-07-05T04:31:31.335440Z","iopub.execute_input":"2023-07-05T04:31:31.335884Z","iopub.status.idle":"2023-07-05T04:31:31.349062Z","shell.execute_reply.started":"2023-07-05T04:31:31.335849Z","shell.execute_reply":"2023-07-05T04:31:31.347520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"count_unique","metadata":{"execution":{"iopub.status.busy":"2023-07-05T04:31:31.350865Z","iopub.execute_input":"2023-07-05T04:31:31.351560Z","iopub.status.idle":"2023-07-05T04:31:31.368673Z","shell.execute_reply.started":"2023-07-05T04:31:31.351523Z","shell.execute_reply":"2023-07-05T04:31:31.367686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It can be noted that most competitions organized so far involve **.csv**,**.hdf**, **.parquet**, **.json** formats of text data collected through the databases accessed as a part of User Interaction ensured by the organizers or the hierarchical storage of data recorded by detectors of the agencies encoded to be presented via a dashboard.\n\nOther formats in use are **.cdf**, **netcdf (.nc gz)** that are self-describing with the inclusion of metadata, with the records interpreted as xarrays or decoded as XML and parsed to gather relevant entries.ASCII format is utilized for a simplified representation and readability of *TextData*.\n\nThe **joblib** and **pickle** libraries in python help store the trained Machine Learning models for the parameters recorded to be used outside the training area for predictions. The **.pt**,**.pkl** formats are common text encoding of the weights and bias between different layers of neural networks(.pt files) while .pkl files store any regression and classification model. ","metadata":{}},{"cell_type":"markdown","source":"The different ways text data was made use of could be analysed in the context of the Kaggle Competition called \"IceCube-Neutrinos in Deep Ice\", one of the research competitions hosted in collaboration with **\"IceCube Neutrino Observatory\"** stationed at South Pole to understand the cosmos from deep within the ice. The goal of the competition was to improve the Machine Learning model in terms of computational efficiency and prediction accuracy to track the direction of neutrinos and their origin. \n\n The data collection process by the observatory can be described as follows:\n* 5160 Digital Optical Modules are deployed in the ice with 86 strings of DOM set 125 meters apart.The activity that produces Cherenkov Light as neutrinos interact with ice is recorded by the sensors.\n* Data is collected at the IceCube Laboratory that has its detectors placed in between the Ice Top and the Antarctic bedrock and sent via a satellite to the Data Warehouse at UW-Madison.\n* Data is split into batches with events ordered based on the occurrence in the recorded instant.\n* Data is made available for further use as a text-based .parquet file where training data  holds the ***batch_id, event_id, first_pulse_index, last_pulse_index,azimuth/zenith angle***; training-batch holds the ***event_id,time,sensor_id,charge,auxiliary***. The .parquet files are together used to train Machine Learning models to assess the direction of the neutrino at a given time.\n\nThe competition, **\"G2-Net,Detecting Continuous Gravitational Waves\"** , provided .hdf files, (hierarchical data format) where each level stored within a single file could be accessed as a key-value pair with attributes stored as keys. Though .hdf5 highlights abstract storage advantage of scientific data,it could lead to software crash during retrieval owing to its combination with metadata within the same file.\n\nParquet files could be widely used for scientific research purposes owing to their ability to store big data of any kind and saving cloud storage space by using highly efficient column-wise compression, and flexible encoding schemes, supporting columns with different data types.They ensure data throughput and performance using techniques like data skipping,as specific column values can be fetched through a **lazyframe**** supported by the **polars** library. \n\n\n(Further reading:\n 1. [https://www.databricks.com/glossary/what-is-parquet](http://)\n    \n 2.[https://pnavaro.github.io/big-data/14-FileFormats.html](http://))","metadata":{}},{"cell_type":"code","source":"count_unique['IceCube - Neutrinos in Deep Ice']","metadata":{"execution":{"iopub.status.busy":"2023-07-05T04:31:31.370561Z","iopub.execute_input":"2023-07-05T04:31:31.371358Z","iopub.status.idle":"2023-07-05T04:31:31.379836Z","shell.execute_reply.started":"2023-07-05T04:31:31.371297Z","shell.execute_reply":"2023-07-05T04:31:31.378776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The Writeups in the code and discussion sections of the competition page could be considered to gather insights into the different ML techniques adopted as a part of enhancing the precision and computational practices.Probing into the various techniques and algorithms considered would help justify the proofs derived by the ML research community. Since arxiv is an open-access archive of **Computer Science**, **Statistics** and **Mathematics** papers vital to investigating Data Science, Machine Learning and AI, the significance of these algorithms on *text data* would be noted.","metadata":{}},{"cell_type":"code","source":"kaggle_summary_i=kaggle_summary[kaggle_summary['Title of Competition']=='IceCube - Neutrinos in Deep Ice']","metadata":{"execution":{"iopub.status.busy":"2023-07-05T04:31:31.381459Z","iopub.execute_input":"2023-07-05T04:31:31.382909Z","iopub.status.idle":"2023-07-05T04:31:31.396110Z","shell.execute_reply.started":"2023-07-05T04:31:31.382858Z","shell.execute_reply":"2023-07-05T04:31:31.394990Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kaggle_summary_i[\"Writeup URL\"]","metadata":{"execution":{"iopub.status.busy":"2023-07-05T04:31:31.397917Z","iopub.execute_input":"2023-07-05T04:31:31.399109Z","iopub.status.idle":"2023-07-05T04:31:31.413157Z","shell.execute_reply.started":"2023-07-05T04:31:31.399060Z","shell.execute_reply":"2023-07-05T04:31:31.412154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The key algorithms observed in the 15 discussions above include:\n* Multiple Layer Perceptron Stack\n* LSTM\n* GRU\n* Transformer\n* Graph Neural Networks\n* Fourier Encoding\n* KNN\n* Regression with Attention and XG Boost\n* CNN\n\nSome of the solutions considered binning and ensemble learning to arrive at the best results.Normalization is easy on text data and normalizing the values of x,y,z was proved to necessary to catch the correlation between other parameters and deduce the azimuth and zenith angles. The neural network approaches in particular considered different optimizers and loss functions each with a justification of better performance. The loss functions observed include:\n* CrossEntropyLoss\n* VonMisesFisher3DLoss\n* MSE\n* Sparse_categorical_crossentropy loss\n* DirectionReconstructionWithKappa\n* VMFLoss\n\n\n(Futher Reading: \n \n 1.[https://www.kaggle.com/c/38257/discussion/403713](http://)\n \n 2.[https://www.kaggle.com/c/38257/discussion/402920](http://)\n \n 3.https://www.kaggle.com/c/38257/discussion/402888\n \n 4.https://www.kaggle.com/c/38257/discussion/402882\n )","metadata":{}},{"cell_type":"markdown","source":"Prior research included the use of :\n* Convolutional Neural Network for event reconstruction\n* Monte Carlo Simulations for probability of detection of photons hitting the detectors\n* Graph Neural Networks for Low-Energy Event Classification and Reconstruction\n* Bayesian Posterior via MCMC sampling for updation of event parameters.\n* Statistical approach for track construction\n\nThese approaches aim to transform the given text data according to uncertainty and loss in energy. The angular resoultion and energy resolution from the Monte Carlo Simulations are arrived at to make pseudo-images through hexagonal array construction to be passed into CNNs for non-linear regression.It is observed that CNNs efficiently compress similar data points to achieve a “latent space” representation.Commonly used inputs are the XYZ-TP areas which are compressed to the required hexagonal size and operational for larger x,y,z values.\n\nGraph Neural Networks are found to optimize the energy range to consider low-energy events unlike CNNs that result in lossy operations. Improvement is possible owing to the Multi Layer Perceptron(MLP) employed with the N-neighbours of a node being *pixels* of the kernel to which the node is mapped to rather than the position of the kernel.Thus, inputs can be sent directly and state graphs are derived in the training period. The use of text data provides every range of values to be examined and ease of **vectorization** relative to the computational power and time required for event proximity and direction analysis.\n\n(Further Reading:\n\n 1. https://arxiv.org/pdf/2004.08254.pdf\n \n 2. https://arxiv.org/pdf/2209.03042.pdf\n)","metadata":{}},{"cell_type":"markdown","source":"The following is the count of research papers on the algorithms employed above on text data as noted from Arxiv:\n* Graph Neural Network-130\n* LSTM- 346\n* GRU-63\n* Transformer-2880\n* Fourier Encoding-7\n* KNN-39\n* Regression-647\n* CNN-529\n\nMajor use cases include Natural Language Processing(NLP) and Sentiment Analysis, Scientific Analysis including space and quantum computing, Geospatial Analysis, Spam detection, Forensics and Healthcare.","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nalg=[\"Graph Neural Network\",\"LSTM\",\"GRU\",\"Transformer\",\"Fourier Encoding\",\"KNN\" ,\"Regression\",\"CNN\"]\ncount=[130,346,63,2880,7,39,647,529]\nplt.scatter(count,alg,color ='maroon')\nplt.xlabel(\"Count\")\nplt.ylabel(\"Algorithms\")\nplt.title(\"Arxiv Statistics\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-05T04:31:31.414384Z","iopub.execute_input":"2023-07-05T04:31:31.415697Z","iopub.status.idle":"2023-07-05T04:31:31.699981Z","shell.execute_reply.started":"2023-07-05T04:31:31.415655Z","shell.execute_reply":"2023-07-05T04:31:31.698848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Text data is thus observed as an easy means of retrieval of both unstructured and structured data in a tabular, hierarchical or parsed formats to perform efficient feature engineering and search for alternatives to reduce computational complexity and loss of data.Feature Selection based on variability is necessary to avoid meaningless correlation. It is relatively easy to derive and rectify observations according to requirements as statistical measures used to tune text data are verified. This is noteworthy to the Scientific Community.","metadata":{}},{"cell_type":"code","source":"sub=pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/sample_submission.csv\")","metadata":{"execution":{"iopub.status.busy":"2023-07-05T04:31:31.703247Z","iopub.execute_input":"2023-07-05T04:31:31.703672Z","iopub.status.idle":"2023-07-05T04:31:31.711421Z","shell.execute_reply.started":"2023-07-05T04:31:31.703642Z","shell.execute_reply":"2023-07-05T04:31:31.710070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub[\"value\"][0]=\"Text data\"\nsub[\"value\"][1]=\"https://www.kaggle.com/code/sumamallapragada/textdata-and-ai/edit/run/130294295\"\nsub[\"value\"][2]=\"https://www.kaggle.com/code/anthony35813/ai-report-a-first-look-at-the-data-for-text-data/notebook\"\nsub[\"value\"][3]=\"https://www.kaggle.com/code/mauromauro/artificially-intelligent-textual-data-processing/comments#2330527\"\nsub[\"value\"][4]=\"https://www.kaggle.com/code/vinitkp/text-data/comments#2330490\"","metadata":{"execution":{"iopub.status.busy":"2023-07-05T04:31:31.713567Z","iopub.execute_input":"2023-07-05T04:31:31.714705Z","iopub.status.idle":"2023-07-05T04:31:31.723677Z","shell.execute_reply.started":"2023-07-05T04:31:31.714656Z","shell.execute_reply":"2023-07-05T04:31:31.722399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub.to_csv(\"submission.csv\",index=False)","metadata":{"execution":{"iopub.status.busy":"2023-07-05T04:31:31.725536Z","iopub.execute_input":"2023-07-05T04:31:31.725980Z","iopub.status.idle":"2023-07-05T04:31:31.738222Z","shell.execute_reply.started":"2023-07-05T04:31:31.725950Z","shell.execute_reply":"2023-07-05T04:31:31.736884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I hope everything is covered. Thanks for reading!","metadata":{}}]}