![\"EMI](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/2969/media/emi.png\")
Imagine an energy feedback system that displays not only your total power consumption, but also continuously shows real-time usage, broken down by electrical appliance. Such a system could provide personalized and cost-effective energy saving recommendations. For example, it could report, \"Based on your usage patterns, you could save $215 per year by switching to a more efficient heating unit, which will pay for itself in 27 months.\" The challenge in this scenario is to sense end-uses of energy to provide feedback at the fine-grained, appliance level.
\r\nThere has been substantial prior research in this area [1,5,6,7,8], however most of this work has concentrated on the use of power consumption patterns and using changes in power draw as features to identify what appliance is being used and how much energy it is consuming. We recommend the reader refer to [2,3,9] for a detailed overview of machine learning features for energy disaggregation.
\r\nA more recent approach to estimate appliance usage is to examine the Electromagnetic Interference (EMI) that most consumer electronic appliances produce as identifying signatures [4]. This EMI is measured using a special sensor built at the Ubicomp Lab at the University of Washington as part of Sidhant Gupta's thesis work. The figure below shows an example of EMI captured from a home. The plot is in frequency domain and shows the signatures of various appliances.
\r\n
The presence or absence of such EMI signatures would ideally tell us when a particular appliance is in use. However, due to the large numbers of appliances in a home, the solution is not straightforward. Machine learning is required not only to make an inference about the appliance class given a particular signature, but probabilistic models are needed that take into account, for example, human appliance usage patterns (think using coffee machine and toaster in morning vs. lights in evening), weather patterns (very unlikely that AC came on during winters), and appliance electrical model. The signature of an appliance can also drift or vary over time due to operating conditions and the mode in which they are used (for instance, a washing machine has many modes). We encourage participants to review [4] to better understand the use of EMI for electrical appliance use detection and classification.
\r\nHere are a few lab quality videos that may helo you grasp the big picture:
\r\nVideo of the signal: http://youtu.be/o-SqO8y8XUA
Video of the technology applied to energy monitoring: http://www.youtube.com/watch?v=dcPI1Cp0VZI
Slides from conference talk for ElectriSense can be accessed here: http://homes.cs.washington.edu/~sidhant/slides/ElectriSense_PDF.pdf
\r\n1. Berges, M., Goldman, E., Matthews, H.S., and Soibelman, L. Training Load Monitoring Algorithms on Highly Sub-Metered Home Electricity Consumption Data. Tsinghua Science & Technology 13, Supple, 0 (2008), 406–411.
\r\n2. Carrie Armel, K., Gupta, A., Shrimali, G., and Albert, A. Is disaggregation the holy grail of energy efficiency? The case of electricity. Energy Policy 52, (2012), 213–234.
\r\n3. Froehlich, J., Larson, E., Gupta, S., Cohn, G., Reynolds, M., and Patel, S. Disaggregated End-Use Energy Sensing for the Smart Grid. IEEE Pervasive Computing 10, 1 (2011), 28–39.
\r\n4. Gupta, S., Reynolds, M., and Patel, S. ElectriSense: Single-Point Sensing Using EMI for Electrical Event Detection and Classification in the home. Ubicomp 2010, (2010).
\r\n5. Hart, G. Nonintrusive appliance load monitoring. Proceedings of the IEEE, (1992).
\r\n6. Laughman, C., Lee, K., and Cox, R. Power signature analysis. IEEE Power and Energy, april 2003 (2003).
\r\n7. Leeb, S.B., Shaw, S.R., and Kirtley, J.L. Transient Event Detection in Spectral Envelope Estimates. IEEE Transactions on Power Delivery 10, 3 (1995), 1200–1210.
\r\n8. Norford, L.K. and Leeb, S.B. Non-intrusive electrical load monitoring in commercial buildings based on steady-state and transient load-detection algorithms. Energy and Buildings 24, 1 (1996), 51–64.
\r\n9. Zeifman, M., Ph, D., and Roth, K. Non-Intrusive Appliance Load Monitoring ( NIALM ): Review and Outlook * Fraunhofer : A Leading Force in Applied R & D. Consumer Electronics, January (2011).
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "rules", "content": "\r\n\t\r\nYou cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.
\r\n\r\n Privately sharing code or data outside of teams is not permitted.\r\n It's okay to share code if made available to all participants on the forums.\r\n
\r\n\r\n Team mergers are allowed and can be performed by the team leader. In order to merge, the combined team must have a total submission count less than or equal to the maximum allowed as of the merge date. The maximum allowed is the number of submissions per day multiplied by the number of days the competition has been running.
\r\n
There is no maximum team size.
\r\nYou may submit a maximum of 2 entries per day.
\r\nYou may select up to 4 final submissions for judging.
\r\nStart Date: 7/2/2013 2:02 AM UTC
\r\nMerger Deadline: None
\r\nEntry Deadline: None
\r\nEnd Date: 10/30/2013 11:59 PM UTC
\r\n\t\r\nParticipants cannot use data other than that provided. Papers may be written summarizing the results from this challenge, but must cite the source of the data and technology.
\r\nThe Competition Sponsor reserves the right to make any changes to the structure of this competition. All decisions made by the Competition Sponsor are final.
\r\nI agree to the competition rules.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Evaluation", "content": "Entries are evaluated on the mean Hamming Loss. The Hamming Loss measures accuracy in a multi-label classification task. The formula is given by:
\r\nSubmission files should contain five columns (Id,House,Appliance,TimeStamp,Predicted) and include a header. See SampleSubmission.csv for an example of the format.
\r\nId,House,Appliance,TimeStamp,Predicted", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Prizes", "content": "
1,H1,30,1334300400,0
2,H1,29,1334300400,0
3,H1,15,1334300400,0
...\r\n
The prizes will be distributed to the top three places as follows:
\r\nThe visualization winner will be chosen by the hosts. You are not required to submit a prediction to the main competition to be eligible for the visualization prize.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "data-description", "content": "You do not need to download both versions of the files.
\r\nH1.zip - H4.zip = official competition data in .mat format
H1_CSV.zip - H4_CSV.zip = unofficial competition data in .csv format
You are provided with data from 4 homes (H1-H4) consisting of both training datasets and testing datasets. The goal is to use the training datasets to learn how each appliance in each home looks and behaves from a machine-learning perspective and build a model which can be applied to the test datasets to make predictions. Refer to SampleSubmission.csv to understand which appliances to predict at which times.
\r\nThe official data for this competition is provided as MATLAB .mat files. This is the language and storage structure being being used by the prototype and its researchers.
\r\nWe recognize that many people do not have access to MATLAB. As a courtesy, we have provided .csv files in addition to the .mat files. Note that the conversion results in some loss of decimal precision, contains complex numbers, and has scientific notation. The uncompressed .csv files can also be quite large. These files are provided without any guarantees regarding their correctness or usability!
\r\nParticipants without MATLAB are encouraged to use the forums to discuss issues around loading/converting the data. Note that in addition to the free language OCTAVE, there are packages in most other languages to read and write .mat files. Python/SciPy/NumPy works well for loading the Matlab files (see scipy.io.loadmat).
\r\nData for each home is placed in its respective directory. You will find 3 kinds of files for each home:
\r\nAllTaggingInfo.mat
This file contains Nx4 elements, where N is the number of labeled training examples provided for a particular home. Since training examples can span multiple days and hence multiple files (see item 2 below), we have included this file such that all of the training labels are in one place. Each row is of the following form:
ApplianceID, ApplianceName, ON_Time, OFF_Time\r\n
Tagged_Training_*.mat
These MATLAB files are the actual data from Belkin’s proprietary hardware. A detailed description of the contents of these files is provided later, however files beginning with “Tagged_Training_” are those that are provided as part of the training data. The reason these are training sets are because they include information about which appliance was turned ON or OFF and at what timestamps. This information is embedded in the .mat file and named “TaggingInfo”. The TaggingInfo array is also Nx4 with same format as the AllTaggingInfo.mat. Essentially, if all TaggingInfo arrays for each home were concatenated, it would yield the global AllTaggingInfo.mat file.
Testing_*.mat
These files are similar to the Tagged_Training_*.mat files, except they do not include a TaggingInfo structure. Instead, participants are expected to figure out when and which appliance is being operated.
Note: All timestamps in the entire dataset are in the UNIX time stamp format.
\r\nIf you load any of the training or testing mat files, it will result in a structure called Buffer that has several member elements:
\r\n| \r\n Buffer.HF \r\n | \r\n\r\n 4096xN spectrogram of high frequency noise captured in the home using Belkin hardware. Refer to [4] for background. N = number of FFT vectors, with each being computed every ~1.0667 seconds. \r\n | \r\n
| \r\n Buffer.TimeTicksHF \r\n | \r\n\r\n NX1 vector of UNIX timestamps corresponding to the N FFTs of Buffer.HF \r\n | \r\n
| \r\n Buffer.LF1V \r\n | \r\n\r\n NX6 array of fundamental and first 5 harmonics of 60Hz voltage measurement of Phase-1. \r\n | \r\n
| \r\n Buffer.LF1I \r\n | \r\n\r\n NX6 array of fundamental and first 5 harmonics of 60Hz current measurement of Phase-1. \r\n | \r\n
| \r\n Buffer.TimeTicks1 \r\n | \r\n\r\n NX1 vector of UNIX timestamps corresponding to Phase-1 current and voltage measurements. \r\n | \r\n
| \r\n Buffer.LF2V \r\n | \r\n\r\n NX6 array of fundamental and first 5 harmonics of 60Hz voltage measurement of Phase-2. \r\n | \r\n
| \r\n Buffer.LF2I \r\n | \r\n\r\n NX6 array of fundamental and first 5 harmonics of 60Hz current measurement of Phase-2. \r\n | \r\n
| \r\n Buffer.Timeticks2 \r\n | \r\n\r\n NX1 vector of UNIX timestamps corresponding to Phase-2 current and voltage measurements. \r\n | \r\n
| \r\n *Buffer.TaggingInfo \r\n | \r\n\r\n NX4 array of labels for training purposes that belong to the loaded file. Same format as AllTaggingInfo.mat file. \r\n | \r\n
*Only present in training data files.
\r\nFor bootstrapping purposes, we have provided sample matlab scripts that can be used to both load and analyze the data files. These scripts can be found in the “LoaderScripts” folder. The main file to load is “LoadData.m”. By default, we load data for H3 and select the first training file. The script is commented and is expected to be self-explanatory such that it can be adapted as needed. It is provided only as an example to understand how the Buffer can be processed to yield certain features like real, reactive and apparent power in addition to the power factors for each phase. The following figure shows the resulting figure when the script is run. A particular section of the plot is zoomed into for clarity.
\r\n![\"Buffer](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/2969/media/buffer.png\")
The default script assumes that the LoadData.m is run with the current directory set to \\LoaderScripts. The DATA_DIR_PATH variable in the script can be changed to point to the correct absolute path as necessary.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Timeline", "content": "First Place:
\r\nSecond Place:
\r\nThird Place:
\r\nVisualization First Place:
\r\n