{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Short-term Solar Energy Prediction Using Statistical and Machine Learning Models\n\n* Aman Solanke 520238\n* Satyam Kumar 520235\n* Rajkishore Kumar 520225\n* Suman Kumar Verma 520241\n\n* **Problem Statement**:\nGiven the Global Ensemble Forecast System(GFES) data for a period of 10 years and the elevation and location data of each mesonet site, we have to train a machine learning model that will predict the total daily incoming solar energy.  \n\n* **Our Approach**:\n1. Gathering required data: The necessary data is provided on kaggle.com, and will be used directly.\n\n2. Exploratory Data Analysis: To understand the provided data and the various technicalities associated with it. To find relation between various variables or the distribution of a single variable.\n\n3. Apply the Algorithm: To write the code and produce the first prediction result and calculate the accuracy of the model. We will be using ensemble learning techniques.\n\n4. Improvization: To improve the accuracy of the model using different optimization techniques.","metadata":{}},{"cell_type":"markdown","source":"# Understanding Ensemble Learning\n\nIn machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. The different types of ensemble techniques are Bagging, Boosting, etc. The output of every \"weak learner\" is averaged(normally or weighted) to make a better prediction.\n\n","metadata":{}},{"cell_type":"markdown","source":"# Coding Environment\n\nWe will be using Notebook feature available on kaggle.com","metadata":{}},{"cell_type":"markdown","source":"# Downloading data","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-03-05T18:02:49.204593Z","iopub.execute_input":"2023-03-05T18:02:49.205048Z","iopub.status.idle":"2023-03-05T18:02:49.250785Z","shell.execute_reply.started":"2023-03-05T18:02:49.205002Z","shell.execute_reply":"2023-03-05T18:02:49.249457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.set_option('display.max_columns', None)\npd.set_option('display.max_rows', None)","metadata":{"execution":{"iopub.status.busy":"2023-03-05T18:02:49.253528Z","iopub.execute_input":"2023-03-05T18:02:49.254456Z","iopub.status.idle":"2023-03-05T18:02:49.260539Z","shell.execute_reply.started":"2023-03-05T18:02:49.254391Z","shell.execute_reply":"2023-03-05T18:02:49.259167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Understanding the different data files and the variables associated with them\n\n1. solar_training_199401_200712.csv and train.csv: It contains the daily incoming solar energy of 98 Oklahoma Mesonet sites and the concerned Dates. \n\n2. station_info.csv: contains the latitude, longitudes, and elevation (meters) of each Mesonet station.\n\n3. gefs_elevations.nc: It is a netCDF4 file that contains the model elevations of the GEFS grid points.","metadata":{}},{"cell_type":"markdown","source":"**solar_training_199401_200712.csv**","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport os\ndataf=pd.read_csv('/kaggle/input/ams-2014-solar-energy-prediction-contest/solar_training_199401_200712.csv')\ndataf.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-05T18:02:49.261768Z","iopub.execute_input":"2023-03-05T18:02:49.262125Z","iopub.status.idle":"2023-03-05T18:02:49.474055Z","shell.execute_reply.started":"2023-03-05T18:02:49.262091Z","shell.execute_reply":"2023-03-05T18:02:49.473052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataf.info()","metadata":{"execution":{"iopub.status.busy":"2023-03-05T18:02:49.476448Z","iopub.execute_input":"2023-03-05T18:02:49.476854Z","iopub.status.idle":"2023-03-05T18:02:49.499467Z","shell.execute_reply.started":"2023-03-05T18:02:49.476806Z","shell.execute_reply":"2023-03-05T18:02:49.498340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\ndataf.isnull().sum()[df.isnull().sum() > 0].sort_values(ascending=False)","metadata":{"execution":{"iopub.status.busy":"2023-03-05T18:05:56.623566Z","iopub.execute_input":"2023-03-05T18:05:56.623988Z","iopub.status.idle":"2023-03-05T18:05:56.638916Z","shell.execute_reply.started":"2023-03-05T18:05:56.623927Z","shell.execute_reply":"2023-03-05T18:05:56.637617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**train.csv**","metadata":{}},{"cell_type":"code","source":"train1=pd.read_csv('/kaggle/input/ams-2014-solar-energy-prediction-contest/train.csv')\ntrain1.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-05T18:03:01.298554Z","iopub.execute_input":"2023-03-05T18:03:01.299165Z","iopub.status.idle":"2023-03-05T18:03:01.431923Z","shell.execute_reply.started":"2023-03-05T18:03:01.299105Z","shell.execute_reply":"2023-03-05T18:03:01.430899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train1.info()","metadata":{"execution":{"iopub.status.busy":"2023-03-05T18:03:01.433112Z","iopub.execute_input":"2023-03-05T18:03:01.434162Z","iopub.status.idle":"2023-03-05T18:03:01.454469Z","shell.execute_reply.started":"2023-03-05T18:03:01.434123Z","shell.execute_reply":"2023-03-05T18:03:01.453054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nfrom pandas import read_csv\nfrom matplotlib import pyplot\nseries = read_csv('/kaggle/input/ams-2014-solar-energy-prediction-contest/solar_training_199401_200712.csv', header=0, index_col=0, parse_dates=True, squeeze=True)\nseries[0:365]['TALI'].plot(legend=None,figsize=(10,6),style='.')\n\npyplot.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-05T19:53:30.806900Z","iopub.execute_input":"2023-03-05T19:53:30.807344Z","iopub.status.idle":"2023-03-05T19:53:31.237389Z","shell.execute_reply.started":"2023-03-05T19:53:30.807304Z","shell.execute_reply":"2023-03-05T19:53:31.236072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"series[0:365].plot(legend=None,figsize=(10,6),style='.')\n\npyplot.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-05T19:56:58.371551Z","iopub.execute_input":"2023-03-05T19:56:58.371960Z","iopub.status.idle":"2023-03-05T19:56:59.329077Z","shell.execute_reply.started":"2023-03-05T19:56:58.371922Z","shell.execute_reply":"2023-03-05T19:56:59.328162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"series['ACME'].plot(legend=None,figsize=(10,6))\n\npyplot.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-05T18:55:28.247741Z","iopub.execute_input":"2023-03-05T18:55:28.248201Z","iopub.status.idle":"2023-03-05T18:55:28.530254Z","shell.execute_reply.started":"2023-03-05T18:55:28.248160Z","shell.execute_reply":"2023-03-05T18:55:28.529086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"series['FAIR'].plot(legend=None,figsize=(10,6))\n\npyplot.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-05T18:55:45.291461Z","iopub.execute_input":"2023-03-05T18:55:45.291884Z","iopub.status.idle":"2023-03-05T18:55:45.577900Z","shell.execute_reply.started":"2023-03-05T18:55:45.291841Z","shell.execute_reply":"2023-03-05T18:55:45.576732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"series.plot(legend=None,figsize=(10,6))\n\npyplot.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-05T19:05:32.185382Z","iopub.execute_input":"2023-03-05T19:05:32.186268Z","iopub.status.idle":"2023-03-05T19:05:38.687450Z","shell.execute_reply.started":"2023-03-05T19:05:32.186218Z","shell.execute_reply":"2023-03-05T19:05:38.686326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"series.plot.area(figsize=(6,100), subplots=True)\npyplot.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-05T19:04:50.248069Z","iopub.execute_input":"2023-03-05T19:04:50.248518Z","iopub.status.idle":"2023-03-05T19:05:12.913695Z","shell.execute_reply.started":"2023-03-05T19:04:50.248477Z","shell.execute_reply":"2023-03-05T19:05:12.912245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\n%matplotlib inline\n","metadata":{"execution":{"iopub.status.busy":"2023-03-05T18:03:01.456209Z","iopub.execute_input":"2023-03-05T18:03:01.456635Z","iopub.status.idle":"2023-03-05T18:03:01.463894Z","shell.execute_reply.started":"2023-03-05T18:03:01.456595Z","shell.execute_reply":"2023-03-05T18:03:01.462446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"**stationInfo**","metadata":{}},{"cell_type":"code","source":"station_info=pd.read_csv('/kaggle/input/ams-2014-solar-energy-prediction-contest/station_info.csv')\nstation_info.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-05T18:03:01.465286Z","iopub.execute_input":"2023-03-05T18:03:01.465626Z","iopub.status.idle":"2023-03-05T18:03:01.485656Z","shell.execute_reply.started":"2023-03-05T18:03:01.465592Z","shell.execute_reply":"2023-03-05T18:03:01.484447Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"station_info.info()","metadata":{"execution":{"iopub.status.busy":"2023-03-05T18:03:01.489150Z","iopub.execute_input":"2023-03-05T18:03:01.489502Z","iopub.status.idle":"2023-03-05T18:03:01.503498Z","shell.execute_reply.started":"2023-03-05T18:03:01.489468Z","shell.execute_reply":"2023-03-05T18:03:01.502237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"# Understanding netCDF4 files\nNetCDF4 is a file format used for storing and sharing scientific data. It stands for Network Common Data Form and is designed to work with large, complex data sets commonly used in fields such as atmospheric science, oceanography, and climate research.\n\nNetCDF4 files can contain multiple variables and dimensions, allowing for easy access and manipulation of data. The format is designed to be platform-independent, so files can be shared and used across different operating systems and software packages.","metadata":{}},{"cell_type":"code","source":"from netCDF4 import Dataset\nelev_station=Dataset('/kaggle/input/ams-2014-solar-energy-prediction-contest/gefs_elevations.nc')\nprint(elev_station)","metadata":{"execution":{"iopub.status.busy":"2023-03-05T18:03:01.505423Z","iopub.execute_input":"2023-03-05T18:03:01.506019Z","iopub.status.idle":"2023-03-05T18:03:01.516266Z","shell.execute_reply.started":"2023-03-05T18:03:01.505945Z","shell.execute_reply":"2023-03-05T18:03:01.515184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for dim in elev_station.dimensions.values():\n    print(dim)","metadata":{"execution":{"iopub.status.busy":"2023-03-05T18:08:48.020291Z","iopub.execute_input":"2023-03-05T18:08:48.021408Z","iopub.status.idle":"2023-03-05T18:08:48.027279Z","shell.execute_reply.started":"2023-03-05T18:08:48.021338Z","shell.execute_reply":"2023-03-05T18:08:48.026130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for var in elev_station.variables.values():\n    print(var)\n    \n    \n    ","metadata":{"execution":{"iopub.status.busy":"2023-03-05T18:11:30.703738Z","iopub.execute_input":"2023-03-05T18:11:30.704181Z","iopub.status.idle":"2023-03-05T18:11:30.711995Z","shell.execute_reply.started":"2023-03-05T18:11:30.704135Z","shell.execute_reply":"2023-03-05T18:11:30.710715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plans for the month of March\n\n1. To study the different machine learning algorithms and apply them using ensemble learning. To create a learning model and test run.\n\n# Plans for April\n\n1. Improving the accuracy of the model.","metadata":{}}]}