This thread is a place to ask questions about competition setup, rules, and submissions.
\n\nYou can find information about the competition overview here.
\n\n\n\n", "messages": [ { "id": 742831, "postDate": "2020-02-11T15:11:03.843Z", "content": "This thread is a place to ask questions about competition setup, rules, and submissions.
\n\nYou can find information about the competition overview here.
\n\n\n\n", "rawMarkdown": "This thread is a place to ask questions about competition setup, rules, and submissions.\n\nYou can find information about the competition overview [here](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/overview/description).\n\n[How to participate](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/overview/evaluation)\n\n[Timeline](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/overview/timeline)", "votes": 11 }, { "id": 744169, "postDate": "2020-02-12T16:30:00.513Z", "content": "Since I do kaggle for fun and self improvement I always like to do a dive into the domain. I very strongly believe that domain knowledge is of equal importance to ML skills.
\n\nAs I dive into power generation in Puerto Rico it seems to me the island serves as a heck of an example of how NOT to manage and build a regions electrical systems. The list of factors that one could add to a model of how to not to do it is very wide ranging (IMO) and extends from the basics of \"not in my backyard thinking\" to a political system that treats the island as a poor cousin. The twin landing of hurricanes in 2017 shows that when we humans do it wrong, mother nature is only too happy to pile on and make it worse.
\n\nA stated goal (or is it a law?) of 40% renewal energy by 2025 might serve in a few years as a model for how to do it right. Sadly IMO taking the levels of renewal energy plants we see in the current data set to 40% in 5 years seems to be a goal beyond the resources that will be committed. Puerto Rico electrical power does hit the news cycle on my IPad on reasonable frequency but the main focus seems to be to point fingers rather than to provide resources or solutions.
\n\nHopefully this exposure will at least leave a bunch of Kaggler's with an appreciation for the island and its struggles.
", "rawMarkdown": "Since I do kaggle for fun and self improvement I always like to do a dive into the domain. I very strongly believe that domain knowledge is of equal importance to ML skills.\n\nAs I dive into power generation in Puerto Rico it seems to me the island serves as a heck of an example of how NOT to manage and build a regions electrical systems. The list of factors that one could add to a model of how to not to do it is very wide ranging (IMO) and extends from the basics of \"not in my backyard thinking\" to a political system that treats the island as a poor cousin. The twin landing of hurricanes in 2017 shows that when we humans do it wrong, mother nature is only too happy to pile on and make it worse.\n\nA stated goal (or is it a law?) of 40% renewal energy by 2025 might serve in a few years as a model for how to do it right. Sadly IMO taking the levels of renewal energy plants we see in the current data set to 40% in 5 years seems to be a goal beyond the resources that will be committed. Puerto Rico electrical power does hit the news cycle on my IPad on reasonable frequency but the main focus seems to be to point fingers rather than to provide resources or solutions.\n\nHopefully this exposure will at least leave a bunch of Kaggler's with an appreciation for the island and its struggles.", "votes": 4 }, { "id": 799849, "postDate": "2020-04-06T20:23:11.400Z", "content": "Hi @paultimothymooney! Is there any information about the announcement date for the competition results? :)
", "rawMarkdown": "Hi @paultimothymooney! Is there any information about the announcement date for the competition results? :)", "votes": 1, "replies": [ { "id": 799854, "postDate": "2020-04-06T20:26:12.717Z", "content": "There were a lot of very excellent and detailed submissions and it took us a long time to do the grading. We hope to announce the winners tomorrow morning (MDT).
", "rawMarkdown": "There were a lot of very excellent and detailed submissions and it took us a long time to do the grading. We hope to announce the winners tomorrow morning (MDT)." }, { "id": 799860, "postDate": "2020-04-06T20:30:57.507Z", "content": "Great, thank you!
", "rawMarkdown": "Great, thank you!", "votes": 1 } ] }, { "id": 750520, "postDate": "2020-02-19T13:21:05.360Z", "content": "Hi Paul,
\n\nWhy do we choose Puerto Rico as the sample of study? \nIs it possible to use additional countries for this case?
", "rawMarkdown": "Hi Paul,\n\nWhy do we choose Puerto Rico as the sample of study? \nIs it possible to use additional countries for this case?", "votes": 1, "replies": [ { "id": 750857, "postDate": "2020-02-19T18:47:56.780Z", "content": "We’re challenging the Kaggle community to see if it’s possible to use remote sensing techniques to calculate emissions factors for a sub-national region. We’ve provided an initial list of datasets covering the geographic boundary of Puerto Rico to serve as the foundation for this analysis. As an island, there are fewer confounding factors from nearby areas. Puerto Rico also offers a unique fuel mix and distinctive energy system layout that should make it easier to isolate pollution attributable to power generation in the remote sensing data. For example, many power plants are isolated from major population centers by the mountains in the middle of the island. Participants will be also be asked to provide a recommendation for how the methodology could be applied to calculate the emissions factor of electricity for another geospatial area using similar techniques. Ideally a model developed for Puerto Rico will also be able to be applied to additional sub-national regions -- and demonstrating so will make your submission very strong.
\n\nYou can find more details about the competition overview here: https://www.kaggle.com/c/ds4g-environmental-insights-explorer/overview.
", "rawMarkdown": "We’re challenging the Kaggle community to see if it’s possible to use remote sensing techniques to calculate emissions factors for a sub-national region. We’ve provided an initial list of datasets covering the geographic boundary of Puerto Rico to serve as the foundation for this analysis. As an island, there are fewer confounding factors from nearby areas. Puerto Rico also offers a unique fuel mix and distinctive energy system layout that should make it easier to isolate pollution attributable to power generation in the remote sensing data. For example, many power plants are isolated from major population centers by the mountains in the middle of the island. **Participants will be also be asked to provide a recommendation for how the methodology could be applied to calculate the emissions factor of electricity for another geospatial area using similar techniques.** Ideally a model developed for Puerto Rico will also be able to be applied to additional sub-national regions -- and demonstrating so will make your submission very strong. \n\nYou can find more details about the competition overview here: https://www.kaggle.com/c/ds4g-environmental-insights-explorer/overview.", "votes": 1 } ] }, { "id": 749807, "postDate": "2020-02-18T23:07:38.357Z", "content": "From the problem statement, \"Current emissions factors methodologies are based on time-consuming data collection and may include errors derived from a lack of access to granular datasets...\", does the \"time-consuming data collection\" refer to gppd_120_pr.csv? If this is the case, does it imply that we can somewhat get the ground truth, average historical emissions factor from gppd_120_pr.csv? Thank you.
The goal of this competition is to develop a methodology to calculate emissions factors using remote sensing data instead of manually collected data. The file gppd_120_pr.csv does not contain any direct measurements concerning annual historical emissions factors. You are welcome to attach additional datasets to your notebook (for example, datasets that contain more traditional measurements of emissions factors) if you think it will be beneficial to your analysis -- but we do not provide any of this data for you.
major cause of NO2 emmision is transportaion sector contribution of more than 50%\nand there is no data for estimation of distribution of NO2 emmision from different activities.
", "rawMarkdown": "major cause of NO2 emmision is transportaion sector contribution of more than 50%\nand there is no data for estimation of distribution of NO2 emmision from different activities.\n", "votes": 1, "replies": [ { "id": 746717, "postDate": "2020-02-15T13:01:48.933Z", "content": "Participants are free to use any datasets in addition to the official Kaggle dataset, but those datasets must also be publicly available on either Earth Engine or Kaggle for the submission to be valid.
\n\nAnother idea is to focus on the large power plants on the south of the island that are far away from the population centers.
", "rawMarkdown": "Participants are free to use any datasets in addition to the [official Kaggle dataset](https://www.kaggle.com/c/ds4g-environmental-insights-explorer), but those datasets must also be publicly available on either Earth Engine or Kaggle for the submission to be valid.\n\nAnother idea is to focus on the large power plants on the south of the island that are far away from the population centers.", "votes": 2 } ] }, { "id": 745599, "postDate": "2020-02-14T01:24:36.203Z", "content": "Paul
\n\nQuestion on the submission. As you know things often go bad on kaggle, limits exist for usage in a kernel, etc.
\n\nSo doing this work in a single kernel is not something I want to do nor do I think a 1st place submission can be completed within a single kernel. Can our submission link to a main kernel, within that kernel have links existing to others? All kernels to be public by the submission deadline. Can those kernels use kaggle data sets - also all public by the submission deadline.
\n\nFor example, training a model is likely to require use of GPU. I would rather not burn my hours while I am making simple plots and graphs, etc. So my model build would be in its own kernel.
", "rawMarkdown": "Paul\n\nQuestion on the submission. As you know things often go bad on kaggle, limits exist for usage in a kernel, etc.\n\nSo doing this work in a single kernel is not something I want to do nor do I think a 1st place submission can be completed within a single kernel. Can our submission link to a main kernel, within that kernel have links existing to others? All kernels to be public by the submission deadline. Can those kernels use kaggle data sets - also all public by the submission deadline.\n\nFor example, training a model is likely to require use of GPU. I would rather not burn my hours while I am making simple plots and graphs, etc. So my model build would be in its own kernel. \n\n", "votes": 1, "replies": [ { "id": 745607, "postDate": "2020-02-14T01:40:12.197Z", "content": "Your submission can contain more than one notebook. Here is an example of a multi-notebook submission that won first prize in a previous DS4G competition: https://www.kaggle.com/shivamb/1-solution-workflow-science-of-policing-equity
", "rawMarkdown": "Your submission can contain more than one notebook. Here is an example of a multi-notebook submission that won first prize in a previous DS4G competition: https://www.kaggle.com/shivamb/1-solution-workflow-science-of-policing-equity", "votes": 4 }, { "id": 745619, "postDate": "2020-02-14T02:15:59.457Z", "content": "Very nice - it raises a big question for me. Why such a short time span for the competition?
", "rawMarkdown": "Very nice - it raises a big question for me. Why such a short time span for the competition?" }, { "id": 746104, "postDate": "2020-02-14T16:22:38.807Z", "content": "The competition timeline is as follows:\n- February 11, 2020: Competition launch\n- March 24, 2020: Final submission deadline
\n\nWe do not plan to take down the dataset following the end of the competition. The Kaggle community can continue to answer the problem and benefit from the public notebooks long after the end of the competition.
", "rawMarkdown": "The competition timeline is as follows:\n- February 11, 2020: Competition launch\n- March 24, 2020: Final submission deadline\n\nWe do not plan to take down the dataset following the end of the competition. The Kaggle community can continue to answer the problem and benefit from the public notebooks long after the end of the competition." } ] }, { "id": 744175, "postDate": "2020-02-12T16:33:32.790Z", "content": "Paul
\n\nAdding additional google engine factors to a model would be helped if you can provide the coordinates that were used to generate the existing Tiff's.
\n\nAny chance you can pull those values from the Java or Python script that was used to generate the data set images?
", "rawMarkdown": "Paul\n\nAdding additional google engine factors to a model would be helped if you can provide the coordinates that were used to generate the existing Tiff's. \n\nAny chance you can pull those values from the Java or Python script that was used to generate the data set images?", "votes": 1, "replies": [ { "id": 744189, "postDate": "2020-02-12T16:39:51.343Z", "content": "\n pr = ee.Geometry.Polygon(\n [[[-67.32297404549217, 18.563112930177304],\n [-67.32297404549217, 17.903121359128956],\n [-65.19437297127342, 17.903121359128956],\n [-65.19437297127342, 18.563112930177304]]], None, False);\n
Thanks a lot
", "rawMarkdown": "Thanks a lot", "votes": 1 }, { "id": 750861, "postDate": "2020-02-19T18:51:00.197Z", "content": "You can also find the same information in the image metadata (example notebook here).
\n\nhttps://rasterio.readthedocs.io/en/latest/quickstart.html#dataset-georeferencing
\n\nHopefully that helps!
", "rawMarkdown": "You can also find the same information in the image metadata ([example notebook here](https://www.kaggle.com/paultimothymooney/explore-image-metadata-s5p-gfs-gldas)).\n\nhttps://rasterio.readthedocs.io/en/latest/quickstart.html#dataset-georeferencing\n\nHopefully that helps!" }, { "id": 767011, "postDate": "2020-03-09T03:09:03.877Z", "content": "Hi Paul! Thanks for listing out the coordinates above. Our team tried to iterate through most of the metadata of the images, it's a bit confusing that they are slightly different from each others, e.g.\npython3\n with rio.open(data) as img_filename:\n print(img_filename.bounds)\n\nproduces the following outputs:
\nBoundingBox(left=-67.32530371380956, bottom=18.565893662264774, right=-65.1918049140257, top=17.901140352016327)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\nBoundingBox(left=-67.32530371380956, bottom=18.565893662264774, right=-65.1918049140257, top=17.901140352016327)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\nBoundingBox(left=-67.32530371380956, bottom=18.565893662264774, right=-65.1918049140257, top=17.901140352016327)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\n...\n\nas we could see, the bouding box varies, although very slightly, between the tiff images. Given the coordinates you noted earlier are four fixed points on the earth, it's hard to reason why the metadata is changing between images over time, unless they all have their own bounding box coordinates. What's more interesting is that they all have the same resolution and shape (in terms of the underlying 2d-array size). Could you help clarify/explain here? Thanks!
Why doesn't this competition award points?
", "rawMarkdown": "Why doesn't this competition award points?", "votes": 2, "replies": [ { "id": 743253, "postDate": "2020-02-11T23:42:49.693Z", "content": "This competition is an \"Analytics Competition\". Analytics Competitions, Getting Started Competitions, and Playground Competitions do not award Kaggle ranking points. The vast majority of Featured Competitions on Kaggle are supervised machine learning competitions with a known answer and a quantitative evaluation metric. We reserve our Kaggle ranking points for these types of more traditional supervised machine learning competitions (where it is easy to quantify the difference in accuracy between a 999th place solution and a 1000th place solution). For this competition the incentive is mostly the bragging rights and the prize money.
", "rawMarkdown": "This competition is an \"[Analytics Competition](https://www.kaggle.com/competitions?sortBy=grouped&group=general&page=1&pageSize=20&category=analytics)\". Analytics Competitions, Getting Started Competitions, and Playground Competitions do not award Kaggle ranking points. The vast majority of Featured Competitions on Kaggle are supervised machine learning competitions with a known answer and a quantitative evaluation metric. We reserve our Kaggle ranking points for these types of more traditional supervised machine learning competitions (where it is easy to quantify the difference in accuracy between a 999th place solution and a 1000th place solution). For this competition the incentive is mostly the bragging rights and the prize money.\n\n", "votes": 5 } ] }, { "id": 779591, "postDate": "2020-03-19T14:03:31.443Z", "content": "Hi Paul,\nAre we suppose to submit only kaggle notebooks or submission through Google colab scripts are accepted. Thanks.
", "rawMarkdown": "Hi Paul,\nAre we suppose to submit only kaggle notebooks or submission through Google colab scripts are accepted. Thanks.", "replies": [ { "id": 779617, "postDate": "2020-03-19T14:28:49.407Z", "content": "You can read more about the competition rules here and here. Submissions in this challenge will take the form of Python and/or R notebooks, hosted on Kaggle.
", "rawMarkdown": "You can read more about the competition rules [here](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/overview/evaluation) and [here](https://www.kaggle.com/page/environmental-insights-explorer-submission-form). Submissions in this challenge will take the form of Python and/or R notebooks, hosted on Kaggle." }, { "id": 779621, "postDate": "2020-03-19T14:35:24.003Z", "content": "Thank you
", "rawMarkdown": "Thank you" } ] }, { "id": 772296, "postDate": "2020-03-15T09:51:36.660Z", "content": "Is Rmarkdown script a valid submission form ?\nThanks
", "rawMarkdown": "Is Rmarkdown script a valid submission form ?\nThanks", "replies": [ { "id": 775728, "postDate": "2020-03-17T00:31:21.287Z", "content": "Yes. You can read more about the competition rules here and here. Submissions in this challenge will take the form of Python and/or R notebooks, hosted on Kaggle.
", "rawMarkdown": "Yes. You can read more about the competition rules [here](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/overview/evaluation) and [here](https://www.kaggle.com/page/environmental-insights-explorer-submission-form). Submissions in this challenge will take the form of Python and/or R notebooks, hosted on Kaggle.", "votes": 1 } ] }, { "id": 767060, "postDate": "2020-03-09T05:24:25.563Z", "content": "Many questions. Some of them have been addressed by the examples thank you for those. \nQuestions:\n1) the time frame seems to be the average emissions across a year and we are given datasets with 2 years of data, so effectively two validation points. Yet the s5p data seems to be in 6 day increments (i.e., one entry for a 6 day interval) although the files in s5p dataset seem to overlap (i.e. one file for 7/1-7/7 another for 7/2-7/8, etc) and the GFS-GLDAS data are on 3 or 6 hour increments with 8 or 4 files per day across the two years. Is your expectation that a winning entry would take in 2 years of all this data and produce one EF for all of Puerto Rico based on the GLDAS-GFS data that could effectively reproduce one or more of the s5p bands?\n2) NO2 is not the only emission from power plants. Are we to try to find data for other emissions or just use the data we have to create an EF for NO2. Wouldn’t there be a separate EF for each gas?\n3) The GFS dataset information says it’s a 3 hour and 384 hour forecast (not measurements but a model forecast based on measurements). If the file is labeled lets say “GFS_2019070112” is it A) the actual (3hr) forecast for 1200hrs on 7/1/2019 B) The forecast generated on 7/1/2018 at 1200hrs for 3 hours in the future or C) the forecast generated on 7/1/2018 at 1200 hrs for 384 hours in the future? None of the dataset information I’ve read seems to clarify this but I am assuming it’s answer A above. But if it were B or C I could adjust accordingly
", "rawMarkdown": "Many questions. Some of them have been addressed by the examples thank you for those. \nQuestions:\n1) the time frame seems to be the average emissions across a year and we are given datasets with 2 years of data, so effectively two validation points. Yet the s5p data seems to be in 6 day increments (i.e., one entry for a 6 day interval) although the files in s5p dataset seem to overlap (i.e. one file for 7/1-7/7 another for 7/2-7/8, etc) and the GFS-GLDAS data are on 3 or 6 hour increments with 8 or 4 files per day across the two years. Is your expectation that a winning entry would take in 2 years of all this data and produce one EF for all of Puerto Rico based on the GLDAS-GFS data that could effectively reproduce one or more of the s5p bands?\n2) NO2 is not the only emission from power plants. Are we to try to find data for other emissions or just use the data we have to create an EF for NO2. Wouldn’t there be a separate EF for each gas?\n3) The GFS dataset information says it’s a 3 hour and 384 hour forecast (not measurements but a model forecast based on measurements). If the file is labeled lets say “GFS_2019070112” is it A) the actual (3hr) forecast for 1200hrs on 7/1/2019 B) The forecast generated on 7/1/2018 at 1200hrs for 3 hours in the future or C) the forecast generated on 7/1/2018 at 1200 hrs for 384 hours in the future? None of the dataset information I’ve read seems to clarify this but I am assuming it’s answer A above. But if it were B or C I could adjust accordingly", "replies": [ { "id": 767431, "postDate": "2020-03-09T16:10:09.270Z", "content": "You can read more about the competition overview here. The ultimate goal of this challenge is to test if calculations of emissions factors using remote sensing techniques are possible and on par with calculations of emissions factors from current methodologies. This is not a supervised machine learning project where you try to predict NO2 emissions from weather data but rather the goal is to develop a methodology that combines various forms of remote sensing data (including both NO2 emissions data, weather data, and additional data) in order to calculate an average historical emissions factor for a sub-national region.
\n\nYou are welcome to include measurements of additional greenhouse gases into your calculations -- although that would come with its own set of challenges. An advantage of NO2 is that we have access to high resolution remote sensing data, whereas CO2 is difficult to measure via remote sensing. NO2 is often used as a tracer gas for NO2 because it is difficult to generate NO2 in an urban environment without also generating CO2. You could also consider looking at NO emissions and SO2 emissions, but we are particularly optimistic about models that are based off of NO2 emissions because we have high resolution remote sensing data that covers the entire globe.
\n\nYou can find out more detail about the dataset by reviewing the dataset description or by reviewing the public notebooks. For example, you will find that there is one years worth of data for all three datasets and that the first date in the S5P_NO2 file names is the date that corresponds to when the image was acquired (while the second date in the file name is the date that the image was turned into a data product). Likewise, you will find that the 384-hour forecasts, with 3-hour forecast interval, are updated four times daily.
", "rawMarkdown": "You can read more about the competition overview [here](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/overview/description). The ultimate goal of this challenge is to test if calculations of emissions factors using remote sensing techniques are possible and on par with calculations of emissions factors from current methodologies. This is not a supervised machine learning project where you try to predict NO2 emissions from weather data but rather the goal is to develop a methodology that combines various forms of remote sensing data (including both NO2 emissions data, weather data, and additional data) in order to calculate an average historical emissions factor for a sub-national region. \n\nYou are welcome to include measurements of additional greenhouse gases into your calculations -- although that would come with its own set of challenges. An advantage of NO2 is that we have access to high resolution remote sensing data, whereas CO2 is difficult to measure via remote sensing. NO2 is often used as a tracer gas for NO2 because it is difficult to generate NO2 in an urban environment without also generating CO2. You could also consider looking at NO emissions and SO2 emissions, but we are particularly optimistic about models that are based off of NO2 emissions because we have high resolution remote sensing data that covers the entire globe.\n\nYou can find out more detail about the dataset by reviewing the [dataset description](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/data) or by reviewing the [public notebooks](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/notebooks). For example, you will find that there is one years worth of data for all three datasets and that the first date in the S5P_NO2 file names is the date that corresponds to when the image was acquired (while the second date in the file name is the date that the image was turned into a data product). Likewise, you will find that the 384-hour forecasts, with 3-hour forecast interval, are updated four times daily. \n" }, { "id": 768353, "postDate": "2020-03-10T17:22:06.997Z", "content": "Thanks for your help.
", "rawMarkdown": "Thanks for your help.", "votes": 1 } ] }, { "id": 765941, "postDate": "2020-03-07T12:16:03.913Z", "content": "It is not very clear for me what are the requirements of this competition from the overview description and the available data. Do we need to predict the emission factor per pollutant (NO2, CO2,..) from satellite images? If it is the case, I do not see any historical data about emission factor in the given dataset. Or, we need to come with new quantity (new definition of emission factor) that can give an insight on the pollution produced by a given activity.
", "rawMarkdown": "It is not very clear for me what are the requirements of this competition from the overview description and the available data. Do we need to predict the emission factor per pollutant (NO2, CO2,..) from satellite images? If it is the case, I do not see any historical data about emission factor in the given dataset. Or, we need to come with new quantity (new definition of emission factor) that can give an insight on the pollution produced by a given activity. ", "replies": [ { "id": 766317, "postDate": "2020-03-08T01:46:41.940Z", "content": "You can read more about the competition overview here. The goal is to develop and explain a methodology that uses remote sensing data to measure and model emissions factors. It is not a supervised machine learning competition where you try to predict a ground truth value.
", "rawMarkdown": "You can read more about the competition overview [here](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/overview/description). The goal is to develop and explain a methodology that uses remote sensing data to measure and model emissions factors. It is not a supervised machine learning competition where you try to predict a ground truth value. ", "votes": 2 }, { "id": 766587, "postDate": "2020-03-08T12:19:31.217Z", "content": "Thank you Paul, the emission factor should be computed per power plant ( separate the contribution of each one) or just the total emission factor from all power plant
", "rawMarkdown": "Thank you Paul, the emission factor should be computed per power plant ( separate the contribution of each one) or just the total emission factor from all power plant" }, { "id": 767395, "postDate": "2020-03-09T15:21:13.510Z", "content": "You can read more about the competition overview here. Participants will be tasked with developing a methodology to reliably calculate emissions factors for a sub-national region. For the purpose of this competition we are focusing on a geographic area the size of Puerto Rico.
", "rawMarkdown": "You can read more about the competition overview [here](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/overview/description). Participants will be tasked with developing a methodology to reliably calculate emissions factors for a sub-national region. For the purpose of this competition we are focusing on a geographic area the size of Puerto Rico." } ] }, { "id": 762706, "postDate": "2020-03-03T18:37:56.583Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 755122, "postDate": "2020-02-24T13:19:55.243Z", "rawMarkdown": "", "isDeleted": true }, { "id": 763354, "postDate": "2020-03-04T11:51:36.077Z", "content": "Thank you for your work.
", "rawMarkdown": "Thank you for your work." } ], "comments": [ { "id": 744169, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-02-12T16:30:00.513000", "content": "Since I do kaggle for fun and self improvement I always like to do a dive into the domain. I very strongly believe that domain knowledge is of equal importance to ML skills.
\n\nAs I dive into power generation in Puerto Rico it seems to me the island serves as a heck of an example of how NOT to manage and build a regions electrical systems. The list of factors that one could add to a model of how to not to do it is very wide ranging (IMO) and extends from the basics of \"not in my backyard thinking\" to a political system that treats the island as a poor cousin. The twin landing of hurricanes in 2017 shows that when we humans do it wrong, mother nature is only too happy to pile on and make it worse.
\n\nA stated goal (or is it a law?) of 40% renewal energy by 2025 might serve in a few years as a model for how to do it right. Sadly IMO taking the levels of renewal energy plants we see in the current data set to 40% in 5 years seems to be a goal beyond the resources that will be committed. Puerto Rico electrical power does hit the news cycle on my IPad on reasonable frequency but the main focus seems to be to point fingers rather than to provide resources or solutions.
\n\nHopefully this exposure will at least leave a bunch of Kaggler's with an appreciation for the island and its struggles.
", "votes": 4, "replies": [] }, { "id": 799849, "author_name": "Ekaterina Melianova", "author_url": "", "post_date": "2020-04-06T20:23:11.400000", "content": "Hi @paultimothymooney! Is there any information about the announcement date for the competition results? :)
", "votes": 1, "replies": [ { "id": 799854, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-04-06T20:26:12.717000", "content": "There were a lot of very excellent and detailed submissions and it took us a long time to do the grading. We hope to announce the winners tomorrow morning (MDT).
", "votes": 0, "replies": [] }, { "id": 799860, "author_name": "Ekaterina Melianova", "author_url": "", "post_date": "2020-04-06T20:30:57.507000", "content": "Great, thank you!
", "votes": 1, "replies": [] } ] }, { "id": 750520, "author_name": "Rasyid Ridha", "author_url": "", "post_date": "2020-02-19T13:21:05.360000", "content": "Hi Paul,
\n\nWhy do we choose Puerto Rico as the sample of study? \nIs it possible to use additional countries for this case?
", "votes": 1, "replies": [ { "id": 750857, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-19T18:47:56.780000", "content": "We’re challenging the Kaggle community to see if it’s possible to use remote sensing techniques to calculate emissions factors for a sub-national region. We’ve provided an initial list of datasets covering the geographic boundary of Puerto Rico to serve as the foundation for this analysis. As an island, there are fewer confounding factors from nearby areas. Puerto Rico also offers a unique fuel mix and distinctive energy system layout that should make it easier to isolate pollution attributable to power generation in the remote sensing data. For example, many power plants are isolated from major population centers by the mountains in the middle of the island. Participants will be also be asked to provide a recommendation for how the methodology could be applied to calculate the emissions factor of electricity for another geospatial area using similar techniques. Ideally a model developed for Puerto Rico will also be able to be applied to additional sub-national regions -- and demonstrating so will make your submission very strong.
\n\nYou can find more details about the competition overview here: https://www.kaggle.com/c/ds4g-environmental-insights-explorer/overview.
", "votes": 1, "replies": [] } ] }, { "id": 749807, "author_name": "wjsheng", "author_url": "", "post_date": "2020-02-18T23:07:38.357000", "content": "From the problem statement, \"Current emissions factors methodologies are based on time-consuming data collection and may include errors derived from a lack of access to granular datasets...\", does the \"time-consuming data collection\" refer to gppd_120_pr.csv? If this is the case, does it imply that we can somewhat get the ground truth, average historical emissions factor from gppd_120_pr.csv? Thank you.
The goal of this competition is to develop a methodology to calculate emissions factors using remote sensing data instead of manually collected data. The file gppd_120_pr.csv does not contain any direct measurements concerning annual historical emissions factors. You are welcome to attach additional datasets to your notebook (for example, datasets that contain more traditional measurements of emissions factors) if you think it will be beneficial to your analysis -- but we do not provide any of this data for you.
major cause of NO2 emmision is transportaion sector contribution of more than 50%\nand there is no data for estimation of distribution of NO2 emmision from different activities.
", "votes": 1, "replies": [ { "id": 746717, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-15T13:01:48.933000", "content": "Participants are free to use any datasets in addition to the official Kaggle dataset, but those datasets must also be publicly available on either Earth Engine or Kaggle for the submission to be valid.
\n\nAnother idea is to focus on the large power plants on the south of the island that are far away from the population centers.
", "votes": 2, "replies": [] } ] }, { "id": 745599, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-02-14T01:24:36.203000", "content": "Paul
\n\nQuestion on the submission. As you know things often go bad on kaggle, limits exist for usage in a kernel, etc.
\n\nSo doing this work in a single kernel is not something I want to do nor do I think a 1st place submission can be completed within a single kernel. Can our submission link to a main kernel, within that kernel have links existing to others? All kernels to be public by the submission deadline. Can those kernels use kaggle data sets - also all public by the submission deadline.
\n\nFor example, training a model is likely to require use of GPU. I would rather not burn my hours while I am making simple plots and graphs, etc. So my model build would be in its own kernel.
", "votes": 1, "replies": [ { "id": 745607, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-14T01:40:12.197000", "content": "Your submission can contain more than one notebook. Here is an example of a multi-notebook submission that won first prize in a previous DS4G competition: https://www.kaggle.com/shivamb/1-solution-workflow-science-of-policing-equity
", "votes": 4, "replies": [] }, { "id": 745619, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-02-14T02:15:59.457000", "content": "Very nice - it raises a big question for me. Why such a short time span for the competition?
", "votes": 0, "replies": [] }, { "id": 746104, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-14T16:22:38.807000", "content": "The competition timeline is as follows:\n- February 11, 2020: Competition launch\n- March 24, 2020: Final submission deadline
\n\nWe do not plan to take down the dataset following the end of the competition. The Kaggle community can continue to answer the problem and benefit from the public notebooks long after the end of the competition.
", "votes": 0, "replies": [] } ] }, { "id": 744175, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-02-12T16:33:32.790000", "content": "Paul
\n\nAdding additional google engine factors to a model would be helped if you can provide the coordinates that were used to generate the existing Tiff's.
\n\nAny chance you can pull those values from the Java or Python script that was used to generate the data set images?
", "votes": 1, "replies": [ { "id": 744189, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-12T16:39:51.343000", "content": "\n pr = ee.Geometry.Polygon(\n [[[-67.32297404549217, 18.563112930177304],\n [-67.32297404549217, 17.903121359128956],\n [-65.19437297127342, 17.903121359128956],\n [-65.19437297127342, 18.563112930177304]]], None, False);\n
Thanks a lot
", "votes": 1, "replies": [] }, { "id": 750861, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-19T18:51:00.197000", "content": "You can also find the same information in the image metadata (example notebook here).
\n\nhttps://rasterio.readthedocs.io/en/latest/quickstart.html#dataset-georeferencing
\n\nHopefully that helps!
", "votes": 0, "replies": [] }, { "id": 767011, "author_name": "Wang Rex", "author_url": "", "post_date": "2020-03-09T03:09:03.877000", "content": "Hi Paul! Thanks for listing out the coordinates above. Our team tried to iterate through most of the metadata of the images, it's a bit confusing that they are slightly different from each others, e.g.\npython3\n with rio.open(data) as img_filename:\n print(img_filename.bounds)\n\nproduces the following outputs:
\nBoundingBox(left=-67.32530371380956, bottom=18.565893662264774, right=-65.1918049140257, top=17.901140352016327)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\nBoundingBox(left=-67.32530371380956, bottom=18.565893662264774, right=-65.1918049140257, top=17.901140352016327)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\nBoundingBox(left=-67.32530371380956, bottom=18.565893662264774, right=-65.1918049140257, top=17.901140352016327)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\nBoundingBox(left=-67.32431391288841, bottom=18.564903861343627, right=-65.19081511310455, top=17.90015055109518)\n...\n\nas we could see, the bouding box varies, although very slightly, between the tiff images. Given the coordinates you noted earlier are four fixed points on the earth, it's hard to reason why the metadata is changing between images over time, unless they all have their own bounding box coordinates. What's more interesting is that they all have the same resolution and shape (in terms of the underlying 2d-array size). Could you help clarify/explain here? Thanks!
Why doesn't this competition award points?
", "votes": 2, "replies": [ { "id": 743253, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-11T23:42:49.693000", "content": "This competition is an \"Analytics Competition\". Analytics Competitions, Getting Started Competitions, and Playground Competitions do not award Kaggle ranking points. The vast majority of Featured Competitions on Kaggle are supervised machine learning competitions with a known answer and a quantitative evaluation metric. We reserve our Kaggle ranking points for these types of more traditional supervised machine learning competitions (where it is easy to quantify the difference in accuracy between a 999th place solution and a 1000th place solution). For this competition the incentive is mostly the bragging rights and the prize money.
", "votes": 5, "replies": [] } ] }, { "id": 779591, "author_name": "Meenakshi", "author_url": "", "post_date": "2020-03-19T14:03:31.443000", "content": "Hi Paul,\nAre we suppose to submit only kaggle notebooks or submission through Google colab scripts are accepted. Thanks.
", "votes": 0, "replies": [ { "id": 779617, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-03-19T14:28:49.407000", "content": "You can read more about the competition rules here and here. Submissions in this challenge will take the form of Python and/or R notebooks, hosted on Kaggle.
", "votes": 0, "replies": [] }, { "id": 779621, "author_name": "Meenakshi", "author_url": "", "post_date": "2020-03-19T14:35:24.003000", "content": "Thank you
", "votes": 0, "replies": [] } ] }, { "id": 772296, "author_name": "Vincent Larmet", "author_url": "", "post_date": "2020-03-15T09:51:36.660000", "content": "Is Rmarkdown script a valid submission form ?\nThanks
", "votes": 0, "replies": [ { "id": 775728, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-03-17T00:31:21.287000", "content": "Yes. You can read more about the competition rules here and here. Submissions in this challenge will take the form of Python and/or R notebooks, hosted on Kaggle.
", "votes": 1, "replies": [] } ] }, { "id": 767060, "author_name": "RayLSCI", "author_url": "", "post_date": "2020-03-09T05:24:25.563000", "content": "Many questions. Some of them have been addressed by the examples thank you for those. \nQuestions:\n1) the time frame seems to be the average emissions across a year and we are given datasets with 2 years of data, so effectively two validation points. Yet the s5p data seems to be in 6 day increments (i.e., one entry for a 6 day interval) although the files in s5p dataset seem to overlap (i.e. one file for 7/1-7/7 another for 7/2-7/8, etc) and the GFS-GLDAS data are on 3 or 6 hour increments with 8 or 4 files per day across the two years. Is your expectation that a winning entry would take in 2 years of all this data and produce one EF for all of Puerto Rico based on the GLDAS-GFS data that could effectively reproduce one or more of the s5p bands?\n2) NO2 is not the only emission from power plants. Are we to try to find data for other emissions or just use the data we have to create an EF for NO2. Wouldn’t there be a separate EF for each gas?\n3) The GFS dataset information says it’s a 3 hour and 384 hour forecast (not measurements but a model forecast based on measurements). If the file is labeled lets say “GFS_2019070112” is it A) the actual (3hr) forecast for 1200hrs on 7/1/2019 B) The forecast generated on 7/1/2018 at 1200hrs for 3 hours in the future or C) the forecast generated on 7/1/2018 at 1200 hrs for 384 hours in the future? None of the dataset information I’ve read seems to clarify this but I am assuming it’s answer A above. But if it were B or C I could adjust accordingly
", "votes": 0, "replies": [ { "id": 767431, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-03-09T16:10:09.270000", "content": "You can read more about the competition overview here. The ultimate goal of this challenge is to test if calculations of emissions factors using remote sensing techniques are possible and on par with calculations of emissions factors from current methodologies. This is not a supervised machine learning project where you try to predict NO2 emissions from weather data but rather the goal is to develop a methodology that combines various forms of remote sensing data (including both NO2 emissions data, weather data, and additional data) in order to calculate an average historical emissions factor for a sub-national region.
\n\nYou are welcome to include measurements of additional greenhouse gases into your calculations -- although that would come with its own set of challenges. An advantage of NO2 is that we have access to high resolution remote sensing data, whereas CO2 is difficult to measure via remote sensing. NO2 is often used as a tracer gas for NO2 because it is difficult to generate NO2 in an urban environment without also generating CO2. You could also consider looking at NO emissions and SO2 emissions, but we are particularly optimistic about models that are based off of NO2 emissions because we have high resolution remote sensing data that covers the entire globe.
\n\nYou can find out more detail about the dataset by reviewing the dataset description or by reviewing the public notebooks. For example, you will find that there is one years worth of data for all three datasets and that the first date in the S5P_NO2 file names is the date that corresponds to when the image was acquired (while the second date in the file name is the date that the image was turned into a data product). Likewise, you will find that the 384-hour forecasts, with 3-hour forecast interval, are updated four times daily.
", "votes": 0, "replies": [] }, { "id": 768353, "author_name": "RayLSCI", "author_url": "", "post_date": "2020-03-10T17:22:06.997000", "content": "Thanks for your help.
", "votes": 1, "replies": [] } ] }, { "id": 765941, "author_name": "essadkim", "author_url": "", "post_date": "2020-03-07T12:16:03.913000", "content": "It is not very clear for me what are the requirements of this competition from the overview description and the available data. Do we need to predict the emission factor per pollutant (NO2, CO2,..) from satellite images? If it is the case, I do not see any historical data about emission factor in the given dataset. Or, we need to come with new quantity (new definition of emission factor) that can give an insight on the pollution produced by a given activity.
", "votes": 0, "replies": [ { "id": 766317, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-03-08T01:46:41.940000", "content": "You can read more about the competition overview here. The goal is to develop and explain a methodology that uses remote sensing data to measure and model emissions factors. It is not a supervised machine learning competition where you try to predict a ground truth value.
", "votes": 2, "replies": [] }, { "id": 766587, "author_name": "essadkim", "author_url": "", "post_date": "2020-03-08T12:19:31.217000", "content": "Thank you Paul, the emission factor should be computed per power plant ( separate the contribution of each one) or just the total emission factor from all power plant
", "votes": 0, "replies": [] }, { "id": 767395, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-03-09T15:21:13.510000", "content": "You can read more about the competition overview here. Participants will be tasked with developing a methodology to reliably calculate emissions factors for a sub-national region. For the purpose of this competition we are focusing on a geographic area the size of Puerto Rico.
", "votes": 0, "replies": [] } ] }, { "id": 762706, "author_name": "", "author_url": "", "post_date": "2020-03-03T18:37:56.583000", "content": "", "votes": 1, "replies": [] }, { "id": 755122, "author_name": "", "author_url": "", "post_date": "2020-02-24T13:19:55.243000", "content": "", "votes": 0, "replies": [] }, { "id": 763354, "author_name": "Louis Bouvier", "author_url": "", "post_date": "2020-03-04T11:51:36.077000", "content": "Thank you for your work.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "742831": "This thread is a place to ask questions about competition setup, rules, and submissions.\n\nYou can find information about the competition overview [here](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/overview/description).\n\n[How to participate](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/overview/evaluation)\n\n[Timeline](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/overview/timeline)", "744169": "Since I do kaggle for fun and self improvement I always like to do a dive into the domain. I very strongly believe that domain knowledge is of equal importance to ML skills.\n\nAs I dive into power generation in Puerto Rico it seems to me the island serves as a heck of an example of how NOT to manage and build a regions electrical systems. The list of factors that one could add to a model of how to not to do it is very wide ranging (IMO) and extends from the basics of \"not in my backyard thinking\" to a political system that treats the island as a poor cousin. The twin landing of hurricanes in 2017 shows that when we humans do it wrong, mother nature is only too happy to pile on and make it worse.\n\nA stated goal (or is it a law?) of 40% renewal energy by 2025 might serve in a few years as a model for how to do it right. Sadly IMO taking the levels of renewal energy plants we see in the current data set to 40% in 5 years seems to be a goal beyond the resources that will be committed. Puerto Rico electrical power does hit the news cycle on my IPad on reasonable frequency but the main focus seems to be to point fingers rather than to provide resources or solutions.\n\nHopefully this exposure will at least leave a bunch of Kaggler's with an appreciation for the island and its struggles.", "799849": "Hi @paultimothymooney! Is there any information about the announcement date for the competition results? :)", "750520": "Hi Paul,\n\nWhy do we choose Puerto Rico as the sample of study? \nIs it possible to use additional countries for this case?", "749807": "From the problem statement, \"Current emissions factors methodologies are based on time-consuming data collection and may include errors derived from a lack of access to granular datasets...\", does the \"time-consuming data collection\" refer to `gppd_120_pr.csv`? If this is the case, does it imply that we can somewhat get the ground truth, `average historical emissions factor` from `gppd_120_pr.csv`? Thank you.", "746671": "major cause of NO2 emmision is transportaion sector contribution of more than 50%\nand there is no data for estimation of distribution of NO2 emmision from different activities.\n", "745599": "Paul\n\nQuestion on the submission. As you know things often go bad on kaggle, limits exist for usage in a kernel, etc.\n\nSo doing this work in a single kernel is not something I want to do nor do I think a 1st place submission can be completed within a single kernel. Can our submission link to a main kernel, within that kernel have links existing to others? All kernels to be public by the submission deadline. Can those kernels use kaggle data sets - also all public by the submission deadline.\n\nFor example, training a model is likely to require use of GPU. I would rather not burn my hours while I am making simple plots and graphs, etc. So my model build would be in its own kernel. \n\n", "744175": "Paul\n\nAdding additional google engine factors to a model would be helped if you can provide the coordinates that were used to generate the existing Tiff's. \n\nAny chance you can pull those values from the Java or Python script that was used to generate the data set images?", "743249": "Why doesn't this competition award points?", "779591": "Hi Paul,\nAre we suppose to submit only kaggle notebooks or submission through Google colab scripts are accepted. Thanks.", "772296": "Is Rmarkdown script a valid submission form ?\nThanks", "767060": "Many questions. Some of them have been addressed by the examples thank you for those. \nQuestions:\n1) the time frame seems to be the average emissions across a year and we are given datasets with 2 years of data, so effectively two validation points. Yet the s5p data seems to be in 6 day increments (i.e., one entry for a 6 day interval) although the files in s5p dataset seem to overlap (i.e. one file for 7/1-7/7 another for 7/2-7/8, etc) and the GFS-GLDAS data are on 3 or 6 hour increments with 8 or 4 files per day across the two years. Is your expectation that a winning entry would take in 2 years of all this data and produce one EF for all of Puerto Rico based on the GLDAS-GFS data that could effectively reproduce one or more of the s5p bands?\n2) NO2 is not the only emission from power plants. Are we to try to find data for other emissions or just use the data we have to create an EF for NO2. Wouldn’t there be a separate EF for each gas?\n3) The GFS dataset information says it’s a 3 hour and 384 hour forecast (not measurements but a model forecast based on measurements). If the file is labeled lets say “GFS_2019070112” is it A) the actual (3hr) forecast for 1200hrs on 7/1/2019 B) The forecast generated on 7/1/2018 at 1200hrs for 3 hours in the future or C) the forecast generated on 7/1/2018 at 1200 hrs for 384 hours in the future? None of the dataset information I’ve read seems to clarify this but I am assuming it’s answer A above. But if it were B or C I could adjust accordingly", "765941": "It is not very clear for me what are the requirements of this competition from the overview description and the available data. Do we need to predict the emission factor per pollutant (NO2, CO2,..) from satellite images? If it is the case, I do not see any historical data about emission factor in the given dataset. Or, we need to come with new quantity (new definition of emission factor) that can give an insight on the pollution produced by a given activity. ", "762706": "", "755122": "", "763354": "Thank you for your work." } }