Hi, guys,
\n\nWe have some question, it will be great if you can answer\n1. Does the data given in the challenge are subtracted from the original data with modifications?\n2. On this page https://developers.google.com/earth-engine/datasets/catalog/COPERNICUS_S5P_OFFL_L3_NO2, the resolution is 0.01 arc degrees. On this page https://sentinel.esa.int/web/sentinel/news/-/article/copernicus-sentinel-5-p-increased-spatial-resolution-change-now-operational it is 5.5 km. And on the page 98: https://sentinel.esa.int/documents/247904/2474726/Sentinel-5P-Level-2-Product-User-Manual-Nitrogen-Dioxide it is 7 km. Do you know what resolution of each pixel is finally?
\n\nWe are also looking for some recommendations:\n- For instance, we use rasterio, could you recommend some library in python to work with satellite data?\n- Could you recommend to learn more about the bands? For instance, we got that tropospheric_NO2_column_number_density is important. Do you use other bands?
", "messages": [ { "id": "764696", "postDate": "03/05/2020 18:50:34", "content": "Hi, guys,
\n\nWe have some question, it will be great if you can answer\n1. Does the data given in the challenge are subtracted from the original data with modifications?\n2. On this page https://developers.google.com/earth-engine/datasets/catalog/COPERNICUS_S5P_OFFL_L3_NO2, the resolution is 0.01 arc degrees. On this page https://sentinel.esa.int/web/sentinel/news/-/article/copernicus-sentinel-5-p-increased-spatial-resolution-change-now-operational it is 5.5 km. And on the page 98: https://sentinel.esa.int/documents/247904/2474726/Sentinel-5P-Level-2-Product-User-Manual-Nitrogen-Dioxide it is 7 km. Do you know what resolution of each pixel is finally?
\n\nWe are also looking for some recommendations:\n- For instance, we use rasterio, could you recommend some library in python to work with satellite data?\n- Could you recommend to learn more about the bands? For instance, we got that tropospheric_NO2_column_number_density is important. Do you use other bands?
", "rawMarkdown": "Hi, guys,\n\nWe have some question, it will be great if you can answer\n1. Does the data given in the challenge are subtracted from the original data with modifications?\n2. On this page https://developers.google.com/earth-engine/datasets/catalog/COPERNICUS_S5P_OFFL_L3_NO2, the resolution is 0.01 arc degrees. On this page https://sentinel.esa.int/web/sentinel/news/-/article/copernicus-sentinel-5-p-increased-spatial-resolution-change-now-operational it is 5.5 km. And on the page 98: https://sentinel.esa.int/documents/247904/2474726/Sentinel-5P-Level-2-Product-User-Manual-Nitrogen-Dioxide it is 7 km. Do you know what resolution of each pixel is finally?\n\nWe are also looking for some recommendations:\n- For instance, we use rasterio, could you recommend some library in python to work with satellite data?\n- Could you recommend to learn more about the bands? For instance, we got that tropospheric_NO2_column_number_density is important. Do you use other bands?", "votes": null }, { "id": "764718", "postDate": "03/05/2020 19:46:41", "content": "The remote sensing data in the Kaggle dataset was exported from the Earth Engine platform as .tif files. You can find links to each individual Earth Engine dataset here. Each data export was only a subset of the larger dataset -- covering only the island of Puerto Rico. All bands were exported for the S5P_NO2 and GFS datasets but only 12 of the 36 bands were exported for the GLDAS dataset. You can explore the image metadata to see additional details, including the resolution of the image itself. Another good resource to learn more about the differences between the different bands is this document. I really like Folium for working with geospatial data using Python.
", "rawMarkdown": "The remote sensing data in the [Kaggle dataset](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/data) was exported from the Earth Engine platform as .tif files. You can find links to each individual Earth Engine dataset [here](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/data). Each data export was only a subset of the larger dataset -- covering only the island of Puerto Rico. All bands were exported for the S5P_NO2 and GFS datasets but only 12 of the 36 bands were exported for the GLDAS dataset. You can [explore the image metadata](https://www.kaggle.com/paultimothymooney/explore-image-metadata-s5p-gfs-gldas) to see additional details, including the resolution of the image itself. Another good resource to learn more about the differences between the different bands is [this document](https://sentinel.esa.int/web/sentinel/user-guides/sentinel-5p-tropomi/document-library/-/asset_publisher/w9Mnd6VPjXlc/content/sentinel-5p-nitrogen-dioxide-level-2-product-readme-file?redirect=https%3A%2F%2Fsentinel.esa.int%2Fweb%2Fsentinel%2Fuser-guides%2Fsentinel-5p-tropomi%2Fdocument-library%3Fp_p_id%3D101_INSTANCE_w9Mnd6VPjXlc%26p_p_lifecycle%3D0%26p_p_state%3Dnormal%26p_p_mode%3Dview%26p_p_col_id%3Dcolumn-1%26p_p_col_count%3D1). I really like [Folium](https://python-visualization.github.io/folium/) for working with geospatial data using Python.", "votes": null }, { "id": "765358", "postDate": "03/06/2020 14:23:23", "content": "If you want to look into the images without the hastle of Python, you could also look into QGIS which is a free software that allows you to look at geo-referenced rasters by simply drag & dropping them in there. You can have access to the metadata there, and measurements tools so you can also measure how big objects are in the image, for instance pixel sizes.
", "rawMarkdown": "If you want to look into the images without the hastle of Python, you could also look into [QGIS](https://www.qgis.org/en/site/index.html) which is a free software that allows you to look at geo-referenced rasters by simply drag & dropping them in there. You can have access to the metadata there, and measurements tools so you can also measure how big objects are in the image, for instance pixel sizes.", "votes": null }, { "id": "765551", "postDate": "03/06/2020 19:35:43", "content": "Thanks!
", "rawMarkdown": "Thanks!", "votes": null }, { "id": "765552", "postDate": "03/06/2020 19:35:46", "content": "Thanks!
", "rawMarkdown": "Thanks!", "votes": null }, { "id": "769412", "postDate": "03/11/2020 21:56:52", "content": "@paultimothymooney I have more questions about data, could you answer, please?\n1. GFS: Global Forecast System 384-Hour Predicted Atmosphere Data. We have the data 4 times per day. As we work with past data, It is not a prediction? it is the real weather? To get the weather at another time, should we interpolate from that data?\nIn prediction, we have 688 images per day( = (5*24+5*8+6*2)*4 ) as \"GFS forecast interval for an individual forecast is hourly for the first 5 days then goes to 3-hour intervals up to day 10 then goes to 12-hour interval for the rest up to 16 days\".\n2. Gldas, we have 8 times per day the data. For example, let's take this file gldas_20180701_1800.tif.\n20180701 - date\n1800 - start hour\nWhat means for the last 3 hours? Does it mean from 1500 to 1800 or from 1800 to 2100 ?\nThanks
", "rawMarkdown": "paultimothymooney I have more questions about data, could you answer, please?\n1. GFS: Global Forecast System 384-Hour Predicted Atmosphere Data. We have the data 4 times per day. As we work with past data, It is not a prediction? it is the real weather? To get the weather at another time, should we interpolate from that data?\nIn prediction, we have 688 images per day( = (5*24+5*8+6*2)*4 ) as \"GFS forecast interval for an individual forecast is hourly for the first 5 days then goes to 3-hour intervals up to day 10 then goes to 12-hour interval for the rest up to 16 days\".\n2. Gldas, we have 8 times per day the data. For example, let's take this file gldas_20180701_1800.tif.\n20180701 - date\n1800 - start hour\nWhat means for the last 3 hours? Does it mean from 1500 to 1800 or from 1800 to 2100 ?\nThanks", "votes": null }, { "id": "769421", "postDate": "03/11/2020 22:13:08", "content": "The GFS dataset and the GLDAS dataset are two weather datasets that are complimentary to each other. GFS contains weather forecasts and covers bodies of water whereas GLDAS contains weather measurements and does not cover bodies of water.
\n\nThe timestamp in the file name for the GLDAS dataset refers to the beginning of the measurement interval.
", "rawMarkdown": "The GFS dataset and the GLDAS dataset are two weather datasets that are complimentary to each other. GFS contains weather forecasts and covers bodies of water whereas GLDAS contains weather measurements and does not cover bodies of water. \n\nThe timestamp in the file name for the GLDAS dataset refers to the beginning of the measurement interval.", "votes": null }, { "id": "769900", "postDate": "03/12/2020 11:27:32", "content": "I did not get the answer to the first question. We have only 4 images per day and not 688 images per day in GFS from Kaggle. What 4 images are referred to? Real weather?
", "rawMarkdown": "I did not get the answer to the first question. We have only 4 images per day and not 688 images per day in GFS from Kaggle. What 4 images are referred to? Real weather?", "votes": null }, { "id": "770139", "postDate": "03/12/2020 16:00:43", "content": "The GFS dataset on Earth Engine contains 384-hour forecasts made at 6-hour temporal resolution (i.e. updated four times daily). Each time point from within the 384-hour forecast can be exported as an individual file (e.g. f000 for the first time point or f384 for the very last time point) and because the data is weather forecast data the early time points will be more accurate and the late time points will be less accurate. Each GFS file in the Kaggle EIE dataset starter pack contains data from time point f000 only and then you have a new file generated every 6 hours again for time point f000. Because the GFS forecasts use only the f000 time point, perhaps it is fair to consider these data to be more of a measurement than a prediction.
\n\nThank you for surfacing that these details were poorly documented -- I will add in a few more details to the file description for the GFS folder.
", "rawMarkdown": "The [GFS dataset on Earth Engine](https://developers.google.com/earth-engine/datasets/catalog/NOAA_GFS0P25) contains 384-hour forecasts made at 6-hour temporal resolution (i.e. updated four times daily). Each time point from within the 384-hour forecast can be exported as an individual file (e.g. f000 for the first time point or f384 for the very last time point) and because the data is weather forecast data the early time points will be more accurate and the late time points will be less accurate. Each GFS file in the [Kaggle EIE dataset starter pack ](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/data) contains data from time point f000 only and then you have a new file generated every 6 hours again for time point f000. Because the GFS forecasts use only the f000 time point, perhaps it is fair to consider these data to be more of a measurement than a prediction. \n\nThank you for surfacing that these details were poorly documented -- I will add in a few more details to the file description for the GFS folder.", "votes": null } ], "comments": [ { "id": 764718, "author_name": "paultimothymooney", "author_url": "", "post_date": "03/05/2020 19:46:41", "content": "The remote sensing data in the Kaggle dataset was exported from the Earth Engine platform as .tif files. You can find links to each individual Earth Engine dataset here. Each data export was only a subset of the larger dataset -- covering only the island of Puerto Rico. All bands were exported for the S5P_NO2 and GFS datasets but only 12 of the 36 bands were exported for the GLDAS dataset. You can explore the image metadata to see additional details, including the resolution of the image itself. Another good resource to learn more about the differences between the different bands is this document. I really like Folium for working with geospatial data using Python.
", "votes": null, "replies": [ { "id": 765551, "author_name": "vict0ra", "author_url": "", "post_date": "03/06/2020 19:35:43", "content": "Thanks!
", "votes": null, "replies": [] } ] }, { "id": 765358, "author_name": "maxlenormand", "author_url": "", "post_date": "03/06/2020 14:23:23", "content": "If you want to look into the images without the hastle of Python, you could also look into QGIS which is a free software that allows you to look at geo-referenced rasters by simply drag & dropping them in there. You can have access to the metadata there, and measurements tools so you can also measure how big objects are in the image, for instance pixel sizes.
", "votes": null, "replies": [ { "id": 765552, "author_name": "vict0ra", "author_url": "", "post_date": "03/06/2020 19:35:46", "content": "Thanks!
", "votes": null, "replies": [] } ] }, { "id": 769412, "author_name": "vict0ra", "author_url": "", "post_date": "03/11/2020 21:56:52", "content": "@paultimothymooney I have more questions about data, could you answer, please?\n1. GFS: Global Forecast System 384-Hour Predicted Atmosphere Data. We have the data 4 times per day. As we work with past data, It is not a prediction? it is the real weather? To get the weather at another time, should we interpolate from that data?\nIn prediction, we have 688 images per day( = (5*24+5*8+6*2)*4 ) as \"GFS forecast interval for an individual forecast is hourly for the first 5 days then goes to 3-hour intervals up to day 10 then goes to 12-hour interval for the rest up to 16 days\".\n2. Gldas, we have 8 times per day the data. For example, let's take this file gldas_20180701_1800.tif.\n20180701 - date\n1800 - start hour\nWhat means for the last 3 hours? Does it mean from 1500 to 1800 or from 1800 to 2100 ?\nThanks
", "votes": null, "replies": [ { "id": 769421, "author_name": "paultimothymooney", "author_url": "", "post_date": "03/11/2020 22:13:08", "content": "The GFS dataset and the GLDAS dataset are two weather datasets that are complimentary to each other. GFS contains weather forecasts and covers bodies of water whereas GLDAS contains weather measurements and does not cover bodies of water.
\n\nThe timestamp in the file name for the GLDAS dataset refers to the beginning of the measurement interval.
", "votes": null, "replies": [] }, { "id": 769900, "author_name": "vict0ra", "author_url": "", "post_date": "03/12/2020 11:27:32", "content": "I did not get the answer to the first question. We have only 4 images per day and not 688 images per day in GFS from Kaggle. What 4 images are referred to? Real weather?
", "votes": null, "replies": [] }, { "id": 770139, "author_name": "paultimothymooney", "author_url": "", "post_date": "03/12/2020 16:00:43", "content": "The GFS dataset on Earth Engine contains 384-hour forecasts made at 6-hour temporal resolution (i.e. updated four times daily). Each time point from within the 384-hour forecast can be exported as an individual file (e.g. f000 for the first time point or f384 for the very last time point) and because the data is weather forecast data the early time points will be more accurate and the late time points will be less accurate. Each GFS file in the Kaggle EIE dataset starter pack contains data from time point f000 only and then you have a new file generated every 6 hours again for time point f000. Because the GFS forecasts use only the f000 time point, perhaps it is fair to consider these data to be more of a measurement than a prediction.
\n\nThank you for surfacing that these details were poorly documented -- I will add in a few more details to the file description for the GFS folder.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "764696": "Hi, guys,\n\nWe have some question, it will be great if you can answer\n1. Does the data given in the challenge are subtracted from the original data with modifications?\n2. On this page https://developers.google.com/earth-engine/datasets/catalog/COPERNICUS_S5P_OFFL_L3_NO2, the resolution is 0.01 arc degrees. On this page https://sentinel.esa.int/web/sentinel/news/-/article/copernicus-sentinel-5-p-increased-spatial-resolution-change-now-operational it is 5.5 km. And on the page 98: https://sentinel.esa.int/documents/247904/2474726/Sentinel-5P-Level-2-Product-User-Manual-Nitrogen-Dioxide it is 7 km. Do you know what resolution of each pixel is finally?\n\nWe are also looking for some recommendations:\n- For instance, we use rasterio, could you recommend some library in python to work with satellite data?\n- Could you recommend to learn more about the bands? For instance, we got that tropospheric_NO2_column_number_density is important. Do you use other bands?", "764718": "The remote sensing data in the [Kaggle dataset](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/data) was exported from the Earth Engine platform as .tif files. You can find links to each individual Earth Engine dataset [here](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/data). Each data export was only a subset of the larger dataset -- covering only the island of Puerto Rico. All bands were exported for the S5P_NO2 and GFS datasets but only 12 of the 36 bands were exported for the GLDAS dataset. You can [explore the image metadata](https://www.kaggle.com/paultimothymooney/explore-image-metadata-s5p-gfs-gldas) to see additional details, including the resolution of the image itself. Another good resource to learn more about the differences between the different bands is [this document](https://sentinel.esa.int/web/sentinel/user-guides/sentinel-5p-tropomi/document-library/-/asset_publisher/w9Mnd6VPjXlc/content/sentinel-5p-nitrogen-dioxide-level-2-product-readme-file?redirect=https%3A%2F%2Fsentinel.esa.int%2Fweb%2Fsentinel%2Fuser-guides%2Fsentinel-5p-tropomi%2Fdocument-library%3Fp_p_id%3D101_INSTANCE_w9Mnd6VPjXlc%26p_p_lifecycle%3D0%26p_p_state%3Dnormal%26p_p_mode%3Dview%26p_p_col_id%3Dcolumn-1%26p_p_col_count%3D1). I really like [Folium](https://python-visualization.github.io/folium/) for working with geospatial data using Python.", "765358": "If you want to look into the images without the hastle of Python, you could also look into [QGIS](https://www.qgis.org/en/site/index.html) which is a free software that allows you to look at geo-referenced rasters by simply drag & dropping them in there. You can have access to the metadata there, and measurements tools so you can also measure how big objects are in the image, for instance pixel sizes.", "765551": "Thanks!", "765552": "Thanks!", "769412": "paultimothymooney I have more questions about data, could you answer, please?\n1. GFS: Global Forecast System 384-Hour Predicted Atmosphere Data. We have the data 4 times per day. As we work with past data, It is not a prediction? it is the real weather? To get the weather at another time, should we interpolate from that data?\nIn prediction, we have 688 images per day( = (5*24+5*8+6*2)*4 ) as \"GFS forecast interval for an individual forecast is hourly for the first 5 days then goes to 3-hour intervals up to day 10 then goes to 12-hour interval for the rest up to 16 days\".\n2. Gldas, we have 8 times per day the data. For example, let's take this file gldas_20180701_1800.tif.\n20180701 - date\n1800 - start hour\nWhat means for the last 3 hours? Does it mean from 1500 to 1800 or from 1800 to 2100 ?\nThanks", "769421": "The GFS dataset and the GLDAS dataset are two weather datasets that are complimentary to each other. GFS contains weather forecasts and covers bodies of water whereas GLDAS contains weather measurements and does not cover bodies of water. \n\nThe timestamp in the file name for the GLDAS dataset refers to the beginning of the measurement interval.", "769900": "I did not get the answer to the first question. We have only 4 images per day and not 688 images per day in GFS from Kaggle. What 4 images are referred to? Real weather?", "770139": "The [GFS dataset on Earth Engine](https://developers.google.com/earth-engine/datasets/catalog/NOAA_GFS0P25) contains 384-hour forecasts made at 6-hour temporal resolution (i.e. updated four times daily). Each time point from within the 384-hour forecast can be exported as an individual file (e.g. f000 for the first time point or f384 for the very last time point) and because the data is weather forecast data the early time points will be more accurate and the late time points will be less accurate. Each GFS file in the [Kaggle EIE dataset starter pack ](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/data) contains data from time point f000 only and then you have a new file generated every 6 hours again for time point f000. Because the GFS forecasts use only the f000 time point, perhaps it is fair to consider these data to be more of a measurement than a prediction. \n\nThank you for surfacing that these details were poorly documented -- I will add in a few more details to the file description for the GFS folder." }, "source": "meta" }