![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2605845%2Fcb118f9e66a603e81d72735d6db7419b%2FQGIS_snapshot.png?generation=1581530802637946&alt=media\")
Hi!
\n\nEDIT: bare with me, Kaggle somehow chopped my post in half, I have to re-write everything. This is really frustrating, it's happen twice now. If anybody knows how to solve this, I'd be really grateful! This is only half of what I had initially wrote. I hope to be able to continue editing this post without it all going away when I hit save.
\n\nFor once, a competition where I can bring something to the table :P I've been working with satellite imagery for over a year now, and have 6 months of experience working in a remote sensing company. There are a few tools to help us visualize and use satellite / aerial imagery.
\n\nA little bit of technical jargon first:
\n\nThe difference is that a raster is a grid of pixels, with multiple bands of said pixels. A vector on the other hand is a set of points, that form polygons (or individual points).
\n\nHere's a quick recap:\n| | Raster | Vector |\n| --- | --- | --- |\n| Data content | Image (grid of pixels) | Points or set of points to form polygons| \n| Data format | .tif , . tiff | .shp (shapefile) , .geojson |
In this competition, it seems like the data we have is mostly rasters. But both types of data work together.
\n\nFirst of all, satellite imagery is different than regular .jpeg for example, in mostly 3 ways:
\n\nHigher pixel bit compression. Regular .jpegs are in 8bits, meaning the value of your pixel ranges from 0 to 255. Most remote sensing images are taken from sensors on satellites that have a much higher precision. It is pretty common to have 16 bits for example (2¹⁶ = 65,536, roughly ~200 times more than 8 bit).
Remote sensing images are geo-located. They are taken at a specific time at a specific location. It is important to know exactly when and where an image was taken, as it represents an area both spatial but temporally as well.
Finally, regular color images have 3 channels: Red, Green & Blue, or RGB for short. Satellite images can have many more than that. For example, the Sentinel 2 satellites, which are one of the most widely used for agricultural purposes have 10 bands. One might think that the image would be of shape (Height, Width, 10), however each of those band does not necessarily have the same resolution.
As we saw, satellite imagery is pretty different from regular images that we as data scientists might be more used to. The typical format for geo-referenced images if .tif which is the format of the images we have in this competition. You can't just open these in your regular image visualizer on your laptop, or do plt.imshow().
One great, free, open-source tool for this is QGIS. It's the best friend you can have when working on satellite imagery. In a nutshell, it allows you to drag and drop geo-referenced images into it, and visualize them very easily:
\n\n
This is what the first image in the gfs folder looks like. This took me literally 10 seconds to make. It is much more practical that having to create a script for every file that I want to look at. One can also stack different images, look at different images temporally, etc.
One great tool that I suggest making use of is Right Click on the image you're looking at and -> Properties. This opens up this window:
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2605845%2F5aff8b38169ebda32db693b6cba00bda%2FQGIS_propreties.png?generation=1581531020748074&alt=media\")
Here we can see a lot of different information:\n- Name of our file\n- Its path\n- CRS (Coordinate Reference System): This is maybe a lot to get into for now, but basically it's to know what projection system we're using. Here it is the very common WGS 84. Think of it as this: The Earth is round, but our screen in flat. There are many different ways to project a spherical surface down to a 2D one. How we do this will distort the image differently. So we want to have the control over this. For more info, check here.\n- Units: each pixel has a size, which allows us to know the resolution of the image on the ground (10m resolution means 1 pixel is 10m on the ground). Except meters aren't always the unit used, degrees are. Welcome to the world of remote sensing, where you have to keep all of these in mind.\n- Data: This is actually the type of data we have, we can see here that we actually have a 64 bit image! So the precision of each pixel is much, much higher than our 0 -> 255 in 8bit.\n- Then statistics and other useful information, but I'll leave that to you to find out :)
\n\nGDAL is IMHO a huge pain when you're only used to Python. But it is really, really useful! GDAL is basically what makes QGIS and rasterio, a library for Python that allows us to manipulate satellite images, possible. It can be tricky to use, but understanding how to use can make your life so much easier. One of the best things is its command line use. QGIS uses a lot of RAM (satellite images are big); so there's a better way to get access to the info we saw above, without needing to open the image in QGIS:
\n\ngdaldinfo <your tif>
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2605845%2F54400e17a0cacf64dcaf03252ddfa52d%2Fgdalinfo.png?generation=1581672866458659&alt=media\")
This allows us to have access to the metadata of a raster very quickly and without needing to open it.
\n\nGDAL is useful for many other things though. QGIS, rasterio, and many others rely on GDAL for their spatial operations (warping a raster to a different coordinate system, clipping it to a mask, etc.) GDAL can be directly used in the command line, but for this competition first of all I doubt that Kaggle has GDAL accessible in the command line, and these operations would probably be a bit too much. Each tile we have for this competition is quite small. Nevertheless, I'll put a shameless plug to on of my GitHub repos on useful GDAL command line operations.
", "messages": [ { "id": 744406, "postDate": "2020-02-12T20:07:50.463Z", "content": "Hi!
\n\nEDIT: bare with me, Kaggle somehow chopped my post in half, I have to re-write everything. This is really frustrating, it's happen twice now. If anybody knows how to solve this, I'd be really grateful! This is only half of what I had initially wrote. I hope to be able to continue editing this post without it all going away when I hit save.
\n\nFor once, a competition where I can bring something to the table :P I've been working with satellite imagery for over a year now, and have 6 months of experience working in a remote sensing company. There are a few tools to help us visualize and use satellite / aerial imagery.
\n\nA little bit of technical jargon first:
\n\nThe difference is that a raster is a grid of pixels, with multiple bands of said pixels. A vector on the other hand is a set of points, that form polygons (or individual points).
\n\nHere's a quick recap:\n| | Raster | Vector |\n| --- | --- | --- |\n| Data content | Image (grid of pixels) | Points or set of points to form polygons| \n| Data format | .tif , . tiff | .shp (shapefile) , .geojson |
In this competition, it seems like the data we have is mostly rasters. But both types of data work together.
\n\nFirst of all, satellite imagery is different than regular .jpeg for example, in mostly 3 ways:
\n\nHigher pixel bit compression. Regular .jpegs are in 8bits, meaning the value of your pixel ranges from 0 to 255. Most remote sensing images are taken from sensors on satellites that have a much higher precision. It is pretty common to have 16 bits for example (2¹⁶ = 65,536, roughly ~200 times more than 8 bit).
Remote sensing images are geo-located. They are taken at a specific time at a specific location. It is important to know exactly when and where an image was taken, as it represents an area both spatial but temporally as well.
Finally, regular color images have 3 channels: Red, Green & Blue, or RGB for short. Satellite images can have many more than that. For example, the Sentinel 2 satellites, which are one of the most widely used for agricultural purposes have 10 bands. One might think that the image would be of shape (Height, Width, 10), however each of those band does not necessarily have the same resolution.
As we saw, satellite imagery is pretty different from regular images that we as data scientists might be more used to. The typical format for geo-referenced images if .tif which is the format of the images we have in this competition. You can't just open these in your regular image visualizer on your laptop, or do plt.imshow().
One great, free, open-source tool for this is QGIS. It's the best friend you can have when working on satellite imagery. In a nutshell, it allows you to drag and drop geo-referenced images into it, and visualize them very easily:
\n\n
This is what the first image in the gfs folder looks like. This took me literally 10 seconds to make. It is much more practical that having to create a script for every file that I want to look at. One can also stack different images, look at different images temporally, etc.
One great tool that I suggest making use of is Right Click on the image you're looking at and -> Properties. This opens up this window:
\n\n
Here we can see a lot of different information:\n- Name of our file\n- Its path\n- CRS (Coordinate Reference System): This is maybe a lot to get into for now, but basically it's to know what projection system we're using. Here it is the very common WGS 84. Think of it as this: The Earth is round, but our screen in flat. There are many different ways to project a spherical surface down to a 2D one. How we do this will distort the image differently. So we want to have the control over this. For more info, check here.\n- Units: each pixel has a size, which allows us to know the resolution of the image on the ground (10m resolution means 1 pixel is 10m on the ground). Except meters aren't always the unit used, degrees are. Welcome to the world of remote sensing, where you have to keep all of these in mind.\n- Data: This is actually the type of data we have, we can see here that we actually have a 64 bit image! So the precision of each pixel is much, much higher than our 0 -> 255 in 8bit.\n- Then statistics and other useful information, but I'll leave that to you to find out :)
\n\nGDAL is IMHO a huge pain when you're only used to Python. But it is really, really useful! GDAL is basically what makes QGIS and rasterio, a library for Python that allows us to manipulate satellite images, possible. It can be tricky to use, but understanding how to use can make your life so much easier. One of the best things is its command line use. QGIS uses a lot of RAM (satellite images are big); so there's a better way to get access to the info we saw above, without needing to open the image in QGIS:
\n\ngdaldinfo <your tif>
This allows us to have access to the metadata of a raster very quickly and without needing to open it.
\n\nGDAL is useful for many other things though. QGIS, rasterio, and many others rely on GDAL for their spatial operations (warping a raster to a different coordinate system, clipping it to a mask, etc.) GDAL can be directly used in the command line, but for this competition first of all I doubt that Kaggle has GDAL accessible in the command line, and these operations would probably be a bit too much. Each tile we have for this competition is quite small. Nevertheless, I'll put a shameless plug to on of my GitHub repos on useful GDAL command line operations.
", "rawMarkdown": "Hi!\n\nEDIT: bare with me, Kaggle somehow chopped my post in half, I have to re-write everything. This is really frustrating, it's happen twice now. If anybody knows how to solve this, I'd be really grateful! This is only half of what I had initially wrote. I hope to be able to continue editing this post without it all going away when I hit save.\n\nFor once, a competition where I can bring something to the table :P I've been working with satellite imagery for over a year now, and have 6 months of experience working in a remote sensing company. There are a few tools to help us visualize and use satellite / aerial imagery.\n\n# Some technical terms\n\nA little bit of technical jargon first:\n\n- A *raster* is defines an image, which as we'll see later is geo-referenced. By contrast:\n- A *vector* is not an image, but rather a polygon that is also geo-referenced.\n\nThe difference is that a *raster* is a grid of pixels, with multiple bands of said pixels. A *vector* on the other hand is a set of points, that form polygons (or individual points).\n\nHere's a quick recap:\n| | Raster | Vector |\n| --- | --- | --- |\n| Data content | Image (grid of pixels) | Points or set of points to form polygons| \n| Data format | `.tif` , `. tiff` | `.shp` (shapefile) , `.geojson` | \n\nIn this competition, it seems like the data we have is mostly rasters. But both types of data work together.\n\n# Handling satellite imagery\n\nFirst of all, satellite imagery is different than regular .jpeg for example, in mostly 3 ways:\n\n- Higher pixel bit compression. Regular .jpegs are in 8bits, meaning the value of your pixel ranges from 0 to 255. Most remote sensing images are taken from sensors on satellites that have a much higher precision. It is pretty common to have 16 bits for example (2¹⁶ = 65,536, roughly ~200 times more than 8 bit).\n\n- Remote sensing images are geo-located. They are taken at a specific time at a specific location. It is important to know exactly when and where an image was taken, as it represents an area both spatial but temporally as well.\n\n- Finally, regular color images have 3 channels: Red, Green & Blue, or RGB for short. Satellite images can have many more than that. For example, the Sentinel 2 satellites, which are one of the most widely used for agricultural purposes have 10 bands. One might think that the image would be of shape (Height, Width, 10), however each of those band does not necessarily have the same resolution.\n\n# Some very useful tools for remote sensing visualization\n\n### QGIS: Your next best friend\n\nAs we saw, satellite imagery is pretty different from regular images that we as data scientists might be more used to. The typical format for geo-referenced images if `.tif` which is the format of the images we have in this competition. You can't just open these in your regular image visualizer on your laptop, or do `plt.imshow()`.\n\nOne great, free, open-source tool for this is [QGIS](https://www.qgis.org/en/site/). It's the best friend you can have when working on satellite imagery. In a nutshell, it allows you to drag and drop geo-referenced images into it, and visualize them very easily:\n\n\n\nThis is what the first image in the `gfs` folder looks like. This took me literally 10 seconds to make. It is much more practical that having to create a script for every file that I want to look at. One can also stack different images, look at different images temporally, etc.\n\nOne great tool that I suggest making use of is Right Click on the image you're looking at and -> Properties. This opens up this window:\n\n\n\nHere we can see a lot of different information:\n- Name of our file\n- Its path\n- CRS (Coordinate Reference System): This is maybe a lot to get into for now, but basically it's to know what projection system we're using. Here it is the very common WGS 84. Think of it as this: The Earth is round, but our screen in flat. There are many different ways to project a spherical surface down to a 2D one. How we do this will distort the image differently. So we want to have the control over this. [For more info, check here](https://georepository.com/crs_4326/WGS-84.html).\n- Units: each pixel has a size, which allows us to know the resolution of the image on the ground (10m resolution means 1 pixel is 10m on the ground). Except meters aren't always the unit used, degrees are. Welcome to the world of remote sensing, where you have to keep all of these in mind.\n- Data: This is actually the type of data we have, we can see here that we actually have a 64 bit image! So the precision of each pixel is much, much higher than our 0 -> 255 in 8bit.\n- Then statistics and other useful information, but I'll leave that to you to find out :)\n\n### GDAL: Your maybe not next best friend, but that one friend you're glad is around when you need him\n\nGDAL is IMHO a huge pain when you're only used to Python. But it is really, _really_ useful! GDAL is basically what makes QGIS and rasterio, a library for Python that allows us to manipulate satellite images, possible. It can be tricky to use, but understanding how to use can make your life so much easier. One of the best things is its command line use. QGIS uses a lot of RAM (satellite images are big); so there's a better way to get access to the info we saw above, without needing to open the image in QGIS:\n\n`gdaldinfo ", "votes": 63 }, { "id": 754124, "postDate": "2020-02-23T04:44:36.037Z", "content": "Just wanted to say thanks for sharing, QGIS has been really helpful for quickly exploring the data and output from various Python scripts.
\n\nI was able to make 'pretty' pictures like this with ease thanks to you!
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F473521%2F2027824f487f717f4bd08930dc06c82d%2FScreenshot%202020-02-23%20at%201.44.05%20PM.png?generation=1582433065476978&alt=media\")
Awesome! Glad I could help! I think these are what these competitions also are for :)
\n\nI’d be curious to know how you made that map?
", "rawMarkdown": "Awesome! Glad I could help! I think these are what these competitions also are for :)\n\nI’d be curious to know how you made that map? ", "votes": 1 }, { "id": 754251, "postDate": "2020-02-23T09:39:54.870Z", "content": "Three layers:\n- OpenStreetMap layer added like this.\n- Powerplants from the gppd csv, show the name + estimated usage and colour according to the plant's main fuel.\n- Heatmap added from the output of this kernel.
", "rawMarkdown": "Three layers:\n- OpenStreetMap layer added [like this](https://gis.stackexchange.com/questions/20191/adding-basemaps-from-google-or-bing-in-qgis/217670#217670).\n- Powerplants from the gppd csv, show the name + estimated usage and colour according to the plant's main fuel.\n- Heatmap added from the output of [this kernel](https://www.kaggle.com/jyesawtellrickson/averaging-satellite-data).", "votes": 3 }, { "id": 754272, "postDate": "2020-02-23T10:24:31.447Z", "content": "I saw your notebook yesterday, didn't put 2 and 2 together, my bad!
\n\nReally nice stuff there :)
", "rawMarkdown": "I saw your notebook yesterday, didn't put 2 and 2 together, my bad!\n\nReally nice stuff there :)" }, { "id": 754291, "postDate": "2020-02-23T11:02:40.183Z", "content": "Thanks! I'm really enjoying working with the dataset. Something exciting about calculating the impact of humanity on the climate through the data collected with these high-tech satellites spinning around the Earth. Feels much more 'real' compared to previous competitions I've participated in.
", "rawMarkdown": "Thanks! I'm really enjoying working with the dataset. Something exciting about calculating the impact of humanity on the climate through the data collected with these high-tech satellites spinning around the Earth. Feels much more 'real' compared to previous competitions I've participated in.", "votes": 1 }, { "id": 754331, "postDate": "2020-02-23T12:15:26.123Z", "content": "That's one of the reasons I got into remote sensing! The datasets themselves are really interesting and a lot of applications are linked to climate change or human activity monitoring!
\n\nIf you have any questions, I'm nowhere near mastering these tools, but don't hesitate to reach out, I'd love to try to help if I can!
", "rawMarkdown": "That's one of the reasons I got into remote sensing! The datasets themselves are really interesting and a lot of applications are linked to climate change or human activity monitoring!\n\nIf you have any questions, I'm nowhere near mastering these tools, but don't hesitate to reach out, I'd love to try to help if I can!", "votes": 1 }, { "id": 772009, "postDate": "2020-03-14T23:01:31.373Z", "content": "good work @jyesawtellrickson !
", "rawMarkdown": "good work @jyesawtellrickson !" } ] }, { "id": 745592, "postDate": "2020-02-14T01:15:45.533Z", "content": "Great ! Very useful @maxlenormand ! Thanks !
", "rawMarkdown": "Great ! Very useful @maxlenormand ! Thanks !\n", "votes": 3, "replies": [ { "id": 745593, "postDate": "2020-02-14T01:16:31.587Z", "content": "I search for some toops and QGIS its very useful, this is another remote sensing tools: https://gisgeography.com/open-source-remote-sensing-software-packages/
", "rawMarkdown": "I search for some toops and QGIS its very useful, this is another remote sensing tools: https://gisgeography.com/open-source-remote-sensing-software-packages/", "votes": 2 }, { "id": 745792, "postDate": "2020-02-14T07:51:01.393Z", "content": "Thanks! That link you gave points to some interesting tools! Those might be a bit overkill for this competition, but there still interesting to know! There is a lot of work that goes into pre-processing satellite imagery: atmosphere correction, coregristration (making sure the images are correctly geo-referenced), etc. But that probably aren’t necessary for this competition, as the data sources are already cleaned.
", "rawMarkdown": "Thanks! That link you gave points to some interesting tools! Those might be a bit overkill for this competition, but there still interesting to know! There is a lot of work that goes into pre-processing satellite imagery: atmosphere correction, coregristration (making sure the images are correctly geo-referenced), etc. But that probably aren’t necessary for this competition, as the data sources are already cleaned.", "votes": 2 } ] }, { "id": 744759, "postDate": "2020-02-13T05:43:56.207Z", "content": "Hi @maxlenormand , thanks for the great info. Very helpful.
\n\nHave you looked at xview2 challenge? You might find it interesting.
", "rawMarkdown": "Hi @maxlenormand , thanks for the great info. Very helpful.\n\nHave you looked at [xview2](https://www.challenge.gov/challenge/diu-xview2-assessing-building-damage/) challenge? You might find it interesting.", "votes": 1, "replies": [ { "id": 744860, "postDate": "2020-02-13T08:09:13.193Z", "content": "I didn’t know about that particular challenge, but I’ve worked with people that have been doing similar tasks! I’ll have a look if there are any public solutions! Thanks for pointing it out!
", "rawMarkdown": "I didn’t know about that particular challenge, but I’ve worked with people that have been doing similar tasks! I’ll have a look if there are any public solutions! Thanks for pointing it out!", "votes": 1 } ] }, { "id": 744505, "postDate": "2020-02-12T22:28:06.550Z", "content": "+1 for using QGIS, great tool :)
", "rawMarkdown": "+1 for using QGIS, great tool :)", "votes": 1, "replies": [ { "id": 744522, "postDate": "2020-02-12T22:44:00.433Z", "content": "It’s the go-to open source software for remote sensing! The more I learn how to use it, the more blown away by how powerful it is I get!
", "rawMarkdown": "It’s the go-to open source software for remote sensing! The more I learn how to use it, the more blown away by how powerful it is I get!", "votes": 3 } ] }, { "id": 744426, "postDate": "2020-02-12T20:32:05.163Z", "content": "Excellent topic Maxime.
", "rawMarkdown": "Excellent topic Maxime.", "votes": 1 }, { "id": 745629, "postDate": "2020-02-14T02:44:22.203Z", "content": "Thanks for wonderful content
", "rawMarkdown": "Thanks for wonderful content", "votes": 2, "replies": [ { "id": 745789, "postDate": "2020-02-14T07:47:20.390Z", "content": "Thanks! I’m glad to see people find it useful!
", "rawMarkdown": "Thanks! I’m glad to see people find it useful!", "votes": 1 } ] }, { "id": 744599, "postDate": "2020-02-13T01:08:06.197Z", "content": "Good job @maxlenormand .
", "rawMarkdown": "Good job @maxlenormand . ", "votes": 2, "replies": [ { "id": 744859, "postDate": "2020-02-13T08:07:16.053Z", "content": "Thanks! Really like your notebooks, so glad that it’s coming from you! :)
", "rawMarkdown": "Thanks! Really like your notebooks, so glad that it’s coming from you! :)", "votes": 1 } ] }, { "id": 744415, "postDate": "2020-02-12T20:19:09.637Z", "content": "Thanks for sharing this. The QGIS tool in particular seems like a great way to help Kagglers get familiar with the data as quickly as possible.
", "rawMarkdown": "Thanks for sharing this. The [QGIS](https://www.qgis.org/en/site/) tool in particular seems like a great way to help Kagglers get familiar with the data as quickly as possible.", "votes": 2, "replies": [ { "id": 744417, "postDate": "2020-02-12T20:21:58.923Z", "content": "Thanks!\nI just saw you pined my topic, that's really cool, thanks!
\n\nAnd yes, QGIS is an awesome tool! There would probably be enough to cover a whole new chapter of the Kaggle lessons out there, that could be useful for many people interested in getting into remote sensing here!
", "rawMarkdown": "Thanks!\nI just saw you pined my topic, that's really cool, thanks!\n\nAnd yes, QGIS is an awesome tool! There would probably be enough to cover a whole new chapter of the Kaggle lessons out there, that could be useful for many people interested in getting into remote sensing here!", "votes": 3 } ] }, { "id": 744654, "postDate": "2020-02-13T02:52:56.497Z", "content": "Wonderful content
", "rawMarkdown": "Wonderful content", "votes": 1 }, { "id": 771715, "postDate": "2020-03-14T14:26:22.163Z", "content": "Thank you so much for sharing, it's really helpful
", "rawMarkdown": "Thank you so much for sharing, it's really helpful", "replies": [ { "id": 771973, "postDate": "2020-03-14T21:43:47.357Z", "content": "Glad it helps! :)
", "rawMarkdown": "Glad it helps! :)" } ] }, { "id": 748560, "postDate": "2020-02-17T16:58:23.397Z", "content": "Great post! I come from an Earth Science / remote sensing background too! I'm excited to participate and it's awesome to see overlap between the communities. QGIS is an awesome program, it has a lot more functionality than a purely visual display such as Panoply, and is easier to plug and play than ArcGIS.
", "rawMarkdown": "Great post! I come from an Earth Science / remote sensing background too! I'm excited to participate and it's awesome to see overlap between the communities. QGIS is an awesome program, it has a lot more functionality than a purely visual display such as Panoply, and is easier to plug and play than ArcGIS.", "replies": [ { "id": 748563, "postDate": "2020-02-17T17:07:14.880Z", "content": "Thanks! Glad to see more people join with an Earth Observation background! I’m pretty excited for this competition!
\n\nOh absolutely, there are a lot of features in QGIS!
\n\nBut first of all, I wanted to keep this simple and just a gentle intro for people that might not know it, and as you have probably noticed, the post gets chopped. I still haven’t managed to solve that issue even after trying several times. I’ll see if I eventually find a way. Either way, I can only recommend people wanting to deep a bit more to do so :)
", "rawMarkdown": "Thanks! Glad to see more people join with an Earth Observation background! I’m pretty excited for this competition!\n\nOh absolutely, there are a lot of features in QGIS!\n\nBut first of all, I wanted to keep this simple and just a gentle intro for people that might not know it, and as you have probably noticed, the post gets chopped. I still haven’t managed to solve that issue even after trying several times. I’ll see if I eventually find a way. Either way, I can only recommend people wanting to deep a bit more to do so :)", "votes": 1 }, { "id": 748572, "postDate": "2020-02-17T17:14:44.900Z", "content": "Hopefully you are able to figure out why your notebook is not completely saving! 😃
", "rawMarkdown": "Hopefully you are able to figure out why your notebook is not completely saving! 😃 " }, { "id": 751203, "postDate": "2020-02-20T04:19:11.270Z", "content": "I keep a CURSE jar near my computer station - Kaggle manages to generate most of the contributions, with the discussion and notebook comments being a real pain to get right and use. Maybe if they had a better and larger organization supporting them they could fix these issues but like most code - new features and fix later seems to be the google way.
", "rawMarkdown": "I keep a CURSE jar near my computer station - Kaggle manages to generate most of the contributions, with the discussion and notebook comments being a real pain to get right and use. Maybe if they had a better and larger organization supporting them they could fix these issues but like most code - new features and fix later seems to be the google way." }, { "id": 753062, "postDate": "2020-02-21T17:09:37.853Z", "content": "The bug that resulted in lost work for discussion comments and notebook comments (that had been edited more than once) has been resolved. Please let us know if you continue to have any issues.
", "rawMarkdown": "The bug that resulted in lost work for discussion comments and notebook comments (that had been edited more than once) has been resolved. Please let us know if you continue to have any issues." }, { "id": 754274, "postDate": "2020-02-23T10:28:20.300Z", "content": "@paultimothymooney still didn't manage to get it working :/
", "rawMarkdown": "@paultimothymooney still didn't manage to get it working :/" } ] }, { "id": 748161, "postDate": "2020-02-17T07:53:57.557Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 748564, "postDate": "2020-02-17T17:07:27.243Z", "content": "Thanks! Glad it helps!
", "rawMarkdown": "Thanks! Glad it helps!" } ] }, { "id": 747774, "postDate": "2020-02-16T20:38:01.427Z", "content": "Thanks, very useful!
", "rawMarkdown": "Thanks, very useful! " } ], "comments": [ { "id": 754124, "author_name": "Jye", "author_url": "", "post_date": "2020-02-23T04:44:36.037000", "content": "Just wanted to say thanks for sharing, QGIS has been really helpful for quickly exploring the data and output from various Python scripts.
\n\nI was able to make 'pretty' pictures like this with ease thanks to you!
\n\n
Awesome! Glad I could help! I think these are what these competitions also are for :)
\n\nI’d be curious to know how you made that map?
", "votes": 1, "replies": [] }, { "id": 754251, "author_name": "Jye", "author_url": "", "post_date": "2020-02-23T09:39:54.870000", "content": "Three layers:\n- OpenStreetMap layer added like this.\n- Powerplants from the gppd csv, show the name + estimated usage and colour according to the plant's main fuel.\n- Heatmap added from the output of this kernel.
", "votes": 3, "replies": [] }, { "id": 754272, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2020-02-23T10:24:31.447000", "content": "I saw your notebook yesterday, didn't put 2 and 2 together, my bad!
\n\nReally nice stuff there :)
", "votes": 0, "replies": [] }, { "id": 754291, "author_name": "Jye", "author_url": "", "post_date": "2020-02-23T11:02:40.183000", "content": "Thanks! I'm really enjoying working with the dataset. Something exciting about calculating the impact of humanity on the climate through the data collected with these high-tech satellites spinning around the Earth. Feels much more 'real' compared to previous competitions I've participated in.
", "votes": 1, "replies": [] }, { "id": 754331, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2020-02-23T12:15:26.123000", "content": "That's one of the reasons I got into remote sensing! The datasets themselves are really interesting and a lot of applications are linked to climate change or human activity monitoring!
\n\nIf you have any questions, I'm nowhere near mastering these tools, but don't hesitate to reach out, I'd love to try to help if I can!
", "votes": 1, "replies": [] }, { "id": 772009, "author_name": "Caesar Lupum", "author_url": "", "post_date": "2020-03-14T23:01:31.373000", "content": "good work @jyesawtellrickson !
", "votes": 0, "replies": [] } ] }, { "id": 745592, "author_name": "Caesar Lupum", "author_url": "", "post_date": "2020-02-14T01:15:45.533000", "content": "Great ! Very useful @maxlenormand ! Thanks !
", "votes": 3, "replies": [ { "id": 745593, "author_name": "Caesar Lupum", "author_url": "", "post_date": "2020-02-14T01:16:31.587000", "content": "I search for some toops and QGIS its very useful, this is another remote sensing tools: https://gisgeography.com/open-source-remote-sensing-software-packages/
", "votes": 2, "replies": [] }, { "id": 745792, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2020-02-14T07:51:01.393000", "content": "Thanks! That link you gave points to some interesting tools! Those might be a bit overkill for this competition, but there still interesting to know! There is a lot of work that goes into pre-processing satellite imagery: atmosphere correction, coregristration (making sure the images are correctly geo-referenced), etc. But that probably aren’t necessary for this competition, as the data sources are already cleaned.
", "votes": 2, "replies": [] } ] }, { "id": 744759, "author_name": "veritasium", "author_url": "", "post_date": "2020-02-13T05:43:56.207000", "content": "Hi @maxlenormand , thanks for the great info. Very helpful.
\n\nHave you looked at xview2 challenge? You might find it interesting.
", "votes": 1, "replies": [ { "id": 744860, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2020-02-13T08:09:13.193000", "content": "I didn’t know about that particular challenge, but I’ve worked with people that have been doing similar tasks! I’ll have a look if there are any public solutions! Thanks for pointing it out!
", "votes": 1, "replies": [] } ] }, { "id": 744505, "author_name": "Tom Aindow", "author_url": "", "post_date": "2020-02-12T22:28:06.550000", "content": "+1 for using QGIS, great tool :)
", "votes": 1, "replies": [ { "id": 744522, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2020-02-12T22:44:00.433000", "content": "It’s the go-to open source software for remote sensing! The more I learn how to use it, the more blown away by how powerful it is I get!
", "votes": 3, "replies": [] } ] }, { "id": 744426, "author_name": "Marília Prata", "author_url": "", "post_date": "2020-02-12T20:32:05.163000", "content": "Excellent topic Maxime.
", "votes": 1, "replies": [] }, { "id": 745629, "author_name": "Aninda Goswamy", "author_url": "", "post_date": "2020-02-14T02:44:22.203000", "content": "Thanks for wonderful content
", "votes": 2, "replies": [ { "id": 745789, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2020-02-14T07:47:20.390000", "content": "Thanks! I’m glad to see people find it useful!
", "votes": 1, "replies": [] } ] }, { "id": 744599, "author_name": "Parul Pandey", "author_url": "", "post_date": "2020-02-13T01:08:06.197000", "content": "Good job @maxlenormand .
", "votes": 2, "replies": [ { "id": 744859, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2020-02-13T08:07:16.053000", "content": "Thanks! Really like your notebooks, so glad that it’s coming from you! :)
", "votes": 1, "replies": [] } ] }, { "id": 744415, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-12T20:19:09.637000", "content": "Thanks for sharing this. The QGIS tool in particular seems like a great way to help Kagglers get familiar with the data as quickly as possible.
", "votes": 2, "replies": [ { "id": 744417, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2020-02-12T20:21:58.923000", "content": "Thanks!\nI just saw you pined my topic, that's really cool, thanks!
\n\nAnd yes, QGIS is an awesome tool! There would probably be enough to cover a whole new chapter of the Kaggle lessons out there, that could be useful for many people interested in getting into remote sensing here!
", "votes": 3, "replies": [] } ] }, { "id": 744654, "author_name": "AtulVerma", "author_url": "", "post_date": "2020-02-13T02:52:56.497000", "content": "Wonderful content
", "votes": 1, "replies": [] }, { "id": 771715, "author_name": "Nawal B.", "author_url": "", "post_date": "2020-03-14T14:26:22.163000", "content": "Thank you so much for sharing, it's really helpful
", "votes": 0, "replies": [ { "id": 771973, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2020-03-14T21:43:47.357000", "content": "Glad it helps! :)
", "votes": 0, "replies": [] } ] }, { "id": 748560, "author_name": "Henry Sue", "author_url": "", "post_date": "2020-02-17T16:58:23.397000", "content": "Great post! I come from an Earth Science / remote sensing background too! I'm excited to participate and it's awesome to see overlap between the communities. QGIS is an awesome program, it has a lot more functionality than a purely visual display such as Panoply, and is easier to plug and play than ArcGIS.
", "votes": 0, "replies": [ { "id": 748563, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2020-02-17T17:07:14.880000", "content": "Thanks! Glad to see more people join with an Earth Observation background! I’m pretty excited for this competition!
\n\nOh absolutely, there are a lot of features in QGIS!
\n\nBut first of all, I wanted to keep this simple and just a gentle intro for people that might not know it, and as you have probably noticed, the post gets chopped. I still haven’t managed to solve that issue even after trying several times. I’ll see if I eventually find a way. Either way, I can only recommend people wanting to deep a bit more to do so :)
", "votes": 1, "replies": [] }, { "id": 748572, "author_name": "Henry Sue", "author_url": "", "post_date": "2020-02-17T17:14:44.900000", "content": "Hopefully you are able to figure out why your notebook is not completely saving! 😃
", "votes": 0, "replies": [] }, { "id": 751203, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-02-20T04:19:11.270000", "content": "I keep a CURSE jar near my computer station - Kaggle manages to generate most of the contributions, with the discussion and notebook comments being a real pain to get right and use. Maybe if they had a better and larger organization supporting them they could fix these issues but like most code - new features and fix later seems to be the google way.
", "votes": 0, "replies": [] }, { "id": 753062, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-21T17:09:37.853000", "content": "The bug that resulted in lost work for discussion comments and notebook comments (that had been edited more than once) has been resolved. Please let us know if you continue to have any issues.
", "votes": 0, "replies": [] }, { "id": 754274, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2020-02-23T10:28:20.300000", "content": "@paultimothymooney still didn't manage to get it working :/
", "votes": 0, "replies": [] } ] }, { "id": 748161, "author_name": "", "author_url": "", "post_date": "2020-02-17T07:53:57.557000", "content": "", "votes": 0, "replies": [ { "id": 748564, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2020-02-17T17:07:27.243000", "content": "Thanks! Glad it helps!
", "votes": 0, "replies": [] } ] }, { "id": 747774, "author_name": "Paulo Geusau", "author_url": "", "post_date": "2020-02-16T20:38:01.427000", "content": "Thanks, very useful!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "744406": "Hi!\n\nEDIT: bare with me, Kaggle somehow chopped my post in half, I have to re-write everything. This is really frustrating, it's happen twice now. If anybody knows how to solve this, I'd be really grateful! This is only half of what I had initially wrote. I hope to be able to continue editing this post without it all going away when I hit save.\n\nFor once, a competition where I can bring something to the table :P I've been working with satellite imagery for over a year now, and have 6 months of experience working in a remote sensing company. There are a few tools to help us visualize and use satellite / aerial imagery.\n\n# Some technical terms\n\nA little bit of technical jargon first:\n\n- A *raster* is defines an image, which as we'll see later is geo-referenced. By contrast:\n- A *vector* is not an image, but rather a polygon that is also geo-referenced.\n\nThe difference is that a *raster* is a grid of pixels, with multiple bands of said pixels. A *vector* on the other hand is a set of points, that form polygons (or individual points).\n\nHere's a quick recap:\n| | Raster | Vector |\n| --- | --- | --- |\n| Data content | Image (grid of pixels) | Points or set of points to form polygons| \n| Data format | `.tif` , `. tiff` | `.shp` (shapefile) , `.geojson` | \n\nIn this competition, it seems like the data we have is mostly rasters. But both types of data work together.\n\n# Handling satellite imagery\n\nFirst of all, satellite imagery is different than regular .jpeg for example, in mostly 3 ways:\n\n- Higher pixel bit compression. Regular .jpegs are in 8bits, meaning the value of your pixel ranges from 0 to 255. Most remote sensing images are taken from sensors on satellites that have a much higher precision. It is pretty common to have 16 bits for example (2¹⁶ = 65,536, roughly ~200 times more than 8 bit).\n\n- Remote sensing images are geo-located. They are taken at a specific time at a specific location. It is important to know exactly when and where an image was taken, as it represents an area both spatial but temporally as well.\n\n- Finally, regular color images have 3 channels: Red, Green & Blue, or RGB for short. Satellite images can have many more than that. For example, the Sentinel 2 satellites, which are one of the most widely used for agricultural purposes have 10 bands. One might think that the image would be of shape (Height, Width, 10), however each of those band does not necessarily have the same resolution.\n\n# Some very useful tools for remote sensing visualization\n\n### QGIS: Your next best friend\n\nAs we saw, satellite imagery is pretty different from regular images that we as data scientists might be more used to. The typical format for geo-referenced images if `.tif` which is the format of the images we have in this competition. You can't just open these in your regular image visualizer on your laptop, or do `plt.imshow()`.\n\nOne great, free, open-source tool for this is [QGIS](https://www.qgis.org/en/site/). It's the best friend you can have when working on satellite imagery. In a nutshell, it allows you to drag and drop geo-referenced images into it, and visualize them very easily:\n\n\n\nThis is what the first image in the `gfs` folder looks like. This took me literally 10 seconds to make. It is much more practical that having to create a script for every file that I want to look at. One can also stack different images, look at different images temporally, etc.\n\nOne great tool that I suggest making use of is Right Click on the image you're looking at and -> Properties. This opens up this window:\n\n\n\nHere we can see a lot of different information:\n- Name of our file\n- Its path\n- CRS (Coordinate Reference System): This is maybe a lot to get into for now, but basically it's to know what projection system we're using. Here it is the very common WGS 84. Think of it as this: The Earth is round, but our screen in flat. There are many different ways to project a spherical surface down to a 2D one. How we do this will distort the image differently. So we want to have the control over this. [For more info, check here](https://georepository.com/crs_4326/WGS-84.html).\n- Units: each pixel has a size, which allows us to know the resolution of the image on the ground (10m resolution means 1 pixel is 10m on the ground). Except meters aren't always the unit used, degrees are. Welcome to the world of remote sensing, where you have to keep all of these in mind.\n- Data: This is actually the type of data we have, we can see here that we actually have a 64 bit image! So the precision of each pixel is much, much higher than our 0 -> 255 in 8bit.\n- Then statistics and other useful information, but I'll leave that to you to find out :)\n\n### GDAL: Your maybe not next best friend, but that one friend you're glad is around when you need him\n\nGDAL is IMHO a huge pain when you're only used to Python. But it is really, _really_ useful! GDAL is basically what makes QGIS and rasterio, a library for Python that allows us to manipulate satellite images, possible. It can be tricky to use, but understanding how to use can make your life so much easier. One of the best things is its command line use. QGIS uses a lot of RAM (satellite images are big); so there's a better way to get access to the info we saw above, without needing to open the image in QGIS:\n\n`gdaldinfo ", "754124": "Just wanted to say thanks for sharing, QGIS has been really helpful for quickly exploring the data and output from various Python scripts.\n\nI was able to make 'pretty' pictures like this with ease thanks to you!\n\n\n", "745592": "Great ! Very useful @maxlenormand ! Thanks !\n", "744759": "Hi @maxlenormand , thanks for the great info. Very helpful.\n\nHave you looked at [xview2](https://www.challenge.gov/challenge/diu-xview2-assessing-building-damage/) challenge? You might find it interesting.", "744505": "+1 for using QGIS, great tool :)", "744426": "Excellent topic Maxime.", "745629": "Thanks for wonderful content", "744599": "Good job @maxlenormand . ", "744415": "Thanks for sharing this. The [QGIS](https://www.qgis.org/en/site/) tool in particular seems like a great way to help Kagglers get familiar with the data as quickly as possible.", "744654": "Wonderful content", "771715": "Thank you so much for sharing, it's really helpful", "748560": "Great post! I come from an Earth Science / remote sensing background too! I'm excited to participate and it's awesome to see overlap between the communities. QGIS is an awesome program, it has a lot more functionality than a purely visual display such as Panoply, and is easier to plug and play than ArcGIS.", "748161": "", "747774": "Thanks, very useful! " } }