{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport shutil\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input/'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n#os.mkdir('/kaggle/working/AnkitaDutta/')\n#os.mkdir('/kaggle/working/AnkitaDutta/submissioncsv/')\n#os.listdir('/kaggle/working/AnkitaDutta/submissioncsv')\n#os.listdir('/kaggle/AnkitaDutta/submissioncsv/')\n#os.remove('/kaggle/working/AnkitaDutta/submissioncsv/state.db')\n#shutil.copyfile('/kaggle/input/2023-kaggle-ai-report/sample_submission.csv', '/kaggle/working/AnkitaDutta/submissioncsv/submission.csv')\n#df = pd.read_csv(\"/kaggle/working/AnkitaDutta/submissioncsv/submission.csv\")\n#df.loc[0, 'type'] = 'essay_category'\n#df.loc[0, 'value'] = 'Other (anything that does not fall into any other category)'\n#df.loc[1, 'value'] = 'https://www.kaggle.com/code/ankitadutta1993/big-data-analytics-using-machine-learning/edit/run/135653947'\n#df.loc[2, 'value'] = 'Feedback 1 (till now no feedback)'\n#df.loc[3, 'value'] = 'Feedback 2 (till now no feedback)'\n#df.loc[4, 'value'] = 'Feedback 3 (till now no feedback)'\n#df.to_csv(\"/kaggle/working/AnkitaDutta/submissioncsv/submission.csv\", index=False)\n\n#print(df)\n#os.savedir('/kaggle/AnkitaDutta/submissioncsv/')\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"d0cb4cff-bd25-46a7-a521-ba6eba35256b","_cell_guid":"c4868e38-cd8f-4e56-a315-b7bd1e9071a2","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-07-04T07:13:32.595117Z","iopub.execute_input":"2023-07-04T07:13:32.595511Z","iopub.status.idle":"2023-07-04T07:13:32.604031Z","shell.execute_reply.started":"2023-07-04T07:13:32.595484Z","shell.execute_reply":"2023-07-04T07:13:32.602777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"# What is Big Data\nMassive amounts of data from numerous sources are combined to form \"big data\". Big data (BD) has received significant consideration in several fields like electronic commerce, e-health, the Internet of Things (IoT), bioinformatics, and Online Social Networks (OSN). Huge volumes of data are being generated every day in a variety of fields, from social networks to engineering and commerce to biomolecular research and psychology. Worldwide, the amount of digital data produced by different digital platforms and devices is expanding at incredible speeds. The gathering of massive amounts of data (BD) is made possible by recent advancements in platforms, tools, and technologies that support high-velocity data capture, storage, and analysis. By volume, digital information increased nine times in 2011 compared to 2006, and by volume, it was predicted to increase to 44 zettabytes by 2020. Globally generated data reached 59 zettabytes as of December 16th, 2020. As we move into a more data-driven future, it is predicted that it will reach 149 zettabytes in 2024. The growing volume in data is the principal attribute of “big data”. Volume, velocity, variety, value, and veracity are five key characteristics (5 Vs) of BD. There are numerous benefits to being able to handle these enormous amounts of data in real-time using Big Data Analytics (BDA) tools and Machine Learning (ML) techniques.\n# Difficulties in Big Data Analytics\nBD and its newly developed tools and methods, such as Big Data Analytics (BDA), have recently revolutionised how organisations and businesses work, creating important new opportunities for corporations, professionals, and academia. Governmental and non-governmental organisations now routinely provide huge unique scope and complexity data, in addition to corporations and research institutes. As a result, it has become crucial globally to extract useful information and valuable advantages from the accessible big data. Nevertheless, research demonstrates that it is difficult to rapidly and easily gain useful insights using BD with efficiency and skill. In order to realise BD's full potential to boost business performance and expand market share for the majority of enterprises, BDA has thus become unavoidably crucial. However, it can be difficult to choose the best BDA tool for a given work due to the abundance of free BDA applications, platforms, and data mining tools.\n# What is MapReduce\nMap Reduce is primarily intended for use by software developers. Programming model rather than language is what Map Reduce is. Because it is simple to use, effective, and can be completed quickly, big data management has gained a lot of attention. Standard SQL, the language used by relational databases, makes it challenging to develop applications like search, indexing, text analysis, graph analysis, data transformation, machine learning, and many more. Because Map Reduce is procedural, experienced programmers in these fields can easily understand it. Additionally, since the system handles parallel computing implementation transparently, it frees developers from having to deal with it. Although Map Reduce was intended for programmers, pre-built Map Reduce apps and function libraries can be useful to non-programmers as well.\n\nThe MapReduce paradigm is a programming model used for processing a huge amount of unstructured data (big data) in a distributed computing environment. It is the foundation of Apache Hadoop and other big data processing frameworks. The model consists of two fundamental tasks: the map task and the reduce task. The map task is responsible for processing input data and generating intermediate key-value pairs. The input data is read and processed in parallel by multiple map tasks, each one generating its own set of key-value pairs. The reduce task is responsible for aggregating and processing the intermediate key-value pairs generated by the map tasks. It takes the key-value pairs as input and generates the final output. The reduce tasks also run in parallel, each one processing a subset of the key-value pairs. The map-reduce paradigm allows for efficient and parallel processing of large data sets, as the data is divided into smaller chunks and processed in parallel by multiple map and reduce tasks.\n# BDA tools using Machine Learning\nIn this literature, we only discuss about the feature selection and classification techniques devised so far for big data analytics.\n## Feature selection in BD\nBig data can be seen as gigantic volumes of data with high dimensional features growing exponentially in different domains. Handling these high-dimensional data suffers from additional difficulties for effective and efficient data processing. In order to overcome these difficulties, feature selection (FS), one of the most popular dimensionality reduction techniques, reduces the high dimensionality of large-scale data by selecting a small subset of related and important features and removing unrelated and redundant features in order to build efficient prediction models. \n\nDimensionality reduction is the powerful data pre-processing technique which aims to map high-dimensional space into a lower-dimensional one while preserving a significant amount of information. There are two categories of dimensionality reduction techniques: Feature Extraction and FS. The feature extraction techniques often map the original features onto a new feature space with smaller dimensions; the newly added features are typically a composite of the original features. In contrast, FS typically extracts a significant subset of features from the initial dataset by eliminating unnecessary, redundant, or noisy features according on a predefined assessment score. \n\n### LASSO-based feature selection\nLeast Absolute Shrinkage and Selection Operator (LASSO) is a commonly used state-of-the-art embedded mechanism of feature selection in the context of big data. Embedded mechanisms are based on the interaction with the classifiers being learnt. LASSO works based on a regression analysis strategy that changes coeffcient estimation of a linear classifer and makes some of the feature coeffcients zero to perform FS. It uses the nonzero as the selected features. The basic LASSO was not capable of handling multi-label classifcation. An upgraded version of LASSO called Relaxed Lasso and Generalized Multi-Class SVM (RL-GenSVM) was developed to handle the problem. The LASSO does not perform good in the non-linear scenario since it assumes that a linear correlation between features and the target class exists. To select an optimal number of features from high dimensional datasets by performing non-linear regression tasks, a feature-wise kernelized LASSO was considered for picking up a nonlinear input-output feature values dependency. Then non-redundant features with a high statistical dependence on output values can be found in terms of kernel-based independence measures such as the Hilbert-Schmidt Independence Criterion (HSIC LASSO). Finally, the overall optimum solution can be calculated effciently. \n\n### SVM-based feature selection\nIn order to perform the FS of high dimension low sample size (HDLSS) datasets, a novel embedded FS technique named Sparse Proximal SVM (sPSVMs) was proposed by combining sparsity with proximal SVM. Another unsupervised FS technique was proposed to choose an optimal feature set based on Locality and Similarity Preserving Embedding (LSPE). The SVM for Recursive Feature Elimination (SVM-RFE) is another technique which is an embedded mechanism that employed SVM as a classifer, then chooses features and removes the least signifcant features based on the weights result from the SVM classifer. An upgraded method of RFE named Variable Step Size with RFE (VSSRFE) was introduced to reduce SVM-RFE time consumption by adding a large step size and makes the step size keep decreasing whereas the number of features to be removed is decreased.\n\n## Big Data classification techniques\n### KNN-based classification methods\nThe research papers corresponding to KNN-based classification methods are elaborated as follows. For prototype reduction methods in nearest neighbour classification, a distributed partitioning methodology was put forth. These methods centred on minimising the count of occurrences utilised to represent the first training datasets. This method's main contribution was to improve the categorization mechanism and reduce storage. However, processing such large datasets proved to be a challenge for the qualified prototype alleviation strategies. A MapReduce-based architecture was developed to address these issues, disseminating the processing of these algorithms through a collection of executing components, designing various algorithmic methodologies to incorporate various partial results into a single one, and employing prototype reduction methods to be taken into account for classification limitations without any loss inaccuracy. The main advantage of this MapReduce based Prototype Reduction (MRPR) was that it allowed PR methods to solve large-scale problems and reduced storage needs.\n\nAnother MapReduce-based scheme for big data categorisation was developed based on a K-Nearest Neighbor classifier. Here, many unseen instances were simultaneously classified against a large dataset. In different informational divisions, the Mapper phase identified K-Nearest Neighbours. The Reduction stage then calculated the definitive neighbours using the corpus obtained in the mapper stage. The developed method caused the K-Nearest Neighbour classifier to expand the dataset to any size by including the executing nodes as needed. The concurrent experimentation also produced a categorization rate that was appropriate for the genuine K-NN model.\n\nIn literature, there exists a MapReduce paradigm for addressing the problem of\nhigh-scale datasets using the K-Nearest Neighbor algorithm. For small or medium amounts of data, the developed model was very much practicable. As a result of this method's accurate parallelization of the KNN model, the precision that could be attained with it remained constant but the efficiency was much improved.\n\nAn incremental and distributed classifier was also developed depending on the common nearest neighbor technique. However, in order to carry out quick searches, this developed technique experimented using Apache Spark.Additionally, an efficient incremental selection method was created to update and remove the outdated examples from the basis. The created system made the original classifier more dependable by cutting down on its computational time. However, mining vast and unbalanced data was not practical.\n\nThere exists an effective solution to execute an appropriate K-Nearest neighbor categorisation depending on spark architecture. Comparing a large training dataset to a large number of hidden cases, the proposed approach successfully categorised them. The Mapper stage runs KNN using different training sets. Additionally, a number of reducers served as conclusive neighbours from the corpus, which was accomplished in the mapper step. The large-scale issues were solved using the developed technique. With regards to Hadoop 2.6.0-cdh5.4.2 and Spark 1.5.1, MapReduce was used for this implementation.\n\nLastly, a novel Data-Independent Reusable Projection (DIRP) method demonstrated how the projection matrix may be used for different types of databases with similar sizes of dimensions by developing a Data-Independent Reusable Projection (DIRP) method to reflect high-dimension data to low-dimension data. The method that was developed protected the separation between two locations in the dataset. The major goal of this strategy was to reduce the computational complexity of existing feature extraction methods. Additionally, the DIRP technique's execution time was shorter than the PCA's. The strategy, however, did not use in-memory processing or fine-tune the RandPro programme.\n### SVM-based classification methods\nThis section deliberates the review of SVM-based classification methods for big data classification collected from various existing research works. In literature, there exists a large-scale data categorisation technique using a one-class SVM model for individual data classes. The three budget-driven characteristics of the proposed model were as follows: The classification process can be managed if the data cannot be stored in memory, the labelling and training processes are user-controllable, and the classification algorithms employed in this technique are very flexible to any dynamic changes with minimal computing cost. Utilising the SVM classifier had the primary advantage of effectively conserving memory.\n\nThere exists an LDA scheme with an enhanced SVM model to e®ectively classify big data. FIR and IIR filters were initially used to distinguish a linear and nonlinear delay present in the ECG signal and remove the extraneous frequency components. In order to reduce the characteristics present in the input ECG data, linear discriminant analysis (LDA) was used, whilst SVM was frequently used for pattern recognition. Here, different elements from the input signal were categorised using the SVM technique with a weighted kernel parameter method. Additionally, the Q wave, R wave, and S wave were identified to classify the heartbeat level. The created approach performed better and successfully classified the large amounts of data.\n\nTo select the ideal set of characteristics, a Hybrid Multi-Objective Firefly and Simulated Annealing (HMOFSA) method was developed. This method's main contribution was the introduction of an Online Feature Selection (OFS) algorithm that enabled the MapReduce framework with the use of a met heuristic algorithm. In the mapper stage, the initial dataset was divided into groups of examples. As a result, the HMOFSA algorithm was used to choose the right characteristics. Finally, Kernel Support Vector Machine (KSVM) was used to evaluate the feature vector. In comparison to other classifiers, the execution time was extremely short.\n### Decision tree-based classification methods\nThis section elaborated on the detailed Survey of Decision tree-based classification techniques utilised for big data classification.\n\nA hybrid algorithm was devised by combining the decision tree logic of the machine learning approach. This method's main contribution was to include a map reduce-based architecture, which increased the efficiency of the traditional decision tree classifier. This approach successfully addressed the properties that feature a lot of information and the problem of handling large databases.\n\nBy developing a novel Divergence-based Grey Wolf Optimisation (DGWO) algorithm, a large data categorization system was created for the healthcare industry. Three steps made up the developed approach: the MapReduce framework, SVM, and Optimised Decision Tree (DT). Data was initially processed using the MapReduce framework, where various actions were handled to reduce the data's overall content. To reduce the dimensionality of the data, the MapReduce architecture used a principal component analysis (PCA). Additionally, classifiers received the reduced data after passing it via the SVM. The findings and information generated by the SVM functions, known as data conversion, allowed the DT classifier to produce the best rules possible. Finally, the converted data were sent to DT, where the established DGWO algorithm was used to do the categorization.The developed scheme achieved higher classification accuracy but failed to implement the technique over a wide range of medical applications.\n\nThe DeTE_SD method, a unique strategy based on Decision Tree Ensemble with smart data, was created to address the imbalanced classification problem in big data sectors. The basic idea behind this approach was to learn several decision trees and use distributed quality data for the ensemble process. By combining Random Discretization, PCA, and clustering-based Random Oversampling, several smart data versions of the original data were produced that were of high quality.\n### Random Forest-based classification methods\nThere exists a novel technique for categorising multi-class unbalanced data that mainly consists of two steps. In the first stage, the authentic database was divided into a collection of binary classes using binarization techniques like OVA and OVO. In order to obtain data, SMOTE was used in the second step in opposition to each subgroup of the unbalanced binary class. The Random Forest (RF) classifier was then used to categorise the data. The data was classified specifically using the oversampling method and the MapReduce technology, which was capable of processing the enormous dataset. The newly created SMOTE+OVA algorithm performed admirably when it came to problems with imbalanced data.\n\nOn the Apache Spark architecture, a Parallel Random Forest (PRF) method was developed. The PRF algorithm was an integrated approach that combined task- and data-parallel optimisation. A vertical data-splitting strategy was used in data-parallel optimisation to decrease information transmission costs, while a data multiplexing technique was used to reduce data volume. In terms of task-parallel optimisation, the dual-parallel training mechanism for RF was used, and a task Directed Acyclic Graph (DAG) was produced based on PRF's parallel training executions. Using the dimension-reduction method, high-dimensional and huge data were made more effective. The developed technique also offered greater accuracy and scalability. Here, a Spark was used for the experiments.\n\nA heuristic bootstrap sampling technique was modelled which was merged with an ensemble learning model on business data mining and introduced an ensemble RFC that utilised parallel computing. Here, data on the insurance industry was acquired from China Life Insurance Company to assess the method's potential for clients. By incorporating deep learning technology into the established scheme, the algorithm could handle several types of analytics while increasing the detection level's accuracy.\n\nAn effective implementation of multi-class resampling techniques was developed also. The two key additions to this established approach are unique segmenting for SMOTE in spark nodes and informed resampling. The created method was used to carry out a selective resampling that focused on particular kinds of occurrences, enhancing its presence in well-balanced databases. Given its significant investigative capacity, the devised clustering technique is very scalable for distributed areas.\n\n## References\n1. Nti, Isaac Kofi, et al. \"A mini-review of machine learning in big data analytics: Applications, challenges, and prospects.\" Big Data Mining and Analytics 5.2 (2022): 81-97.\n2. Gujjeti, Sridhar, and Suresh Pabboju. \"Optimized Neural Network for Big Data Classification Using MapReduce Approach.\" Applied Information Processing Systems: Proceedings of ICCET 2021. Springer Singapore, 2022.\n3. Abdulwahab, Hudhaifa Mohammed, S. Ajitha, and Mufeed Ahmed Naji Saif. \"Feature selection techniques in the context of big data: taxonomy and analysis.\" Applied Intelligence 52.12 (2022): 13568-13613.\n4. Jayasri, N. P., and R. Aruna. \"Big data analytics in health care by data mining and classification techniques.\" ICT Express 8.2 (2022): 250-257.\n5. Gandomi, Amir H., Fang Chen, and Laith Abualigah. \"Machine learning technologies for big data analytics.\" Electronics 11.3 (2022): 421.\n6. Yang, Shengying, et al. \"Optimized hadoop map reduce system for strong analytics of cloud big product data on amazon web service.\" Information Processing & Management 60.3 (2023): 103271.\n7. Abdalla, Hemn Barzan, and Belal Abuhaija. \"Comprehensive Analysis of Various Big Data Classification Techniques: A Challenging Overview.\" Journal of Information & Knowledge Management 22.01 (2023): 2250083.\n","metadata":{}}]}