{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Dataset description and relation to competition\nThis dataset is available here: http://citycarbonfootprints.info/ and derives from a recent publication on carbon footprints of world cities: https://iopscience.iop.org/article/10.1088/1748-9326/aac72a \"Carbon footprints of 13 000 cities\", published in 2018. The carbon footprint data refer to the year 2013. To create the footprints for the 13,000 cities around the world, the authors created a 250m grid spanning the globe:\n\n>Units are Gg (1 Gg=1Kt) of CO2 emissions from fossil fuel combustion. The data year is 2013. The map is full-world extent (-90° to +90° and -180° to 180°) in the equal-area World Mollewiede (EPSG:54009) projection, with 250m cells. The GeoTIFF files are 200mb uncompressed but require a minimum of 5gb RAM to view or analyse.\n\nThe study appears to be one of the most extensive efforts yet to characterize carbon footprints with as much spatial coverage as possible. This opens up numerous possibilities for analysis and incorporation into KPIs for the CDP: Unlocking Climate Solutions competition. The global extent should enable actual carbon footprint estimates to be spatially joined to the competition data, which include:\n- City-level information about commitments to improving carbon budgets, as well as\n- Fine-grained spatial analysis within cities at the zip code and census tract level\n\nThe data presented here could be used for both large, multi-city analysis, as well as within-city analysis due to the 250m spatial resolution.\n\nIn this notebook I load the data and visualize for Los Angeles, California, USA, then join with cities in the competition dataset (TBD)."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"collapsed":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import rasterio\nfrom rasterio.enums import Resampling\nfrom rasterio.plot import show\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\nimport geopandas as gpd\nimport re","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Carbon Footprint Data\nThese data come in the form of a GeoTIFF, which is essentially a 2D array of numbers that can be mapped to spatial locations on the earth's surface."},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset = rasterio.open('../input/global-gridded-model-of-carbon-footprints-ggmcf/GGMCF_v1.0.tif')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset.count","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Image data often have multiple \"bands\", e.g. red/green/blue, but there is just one band here, which is the array of numbers representing carbon footprints. Let's visualize the data. For visibility, downsample the data by a factor of 100 plot it on a log scale."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nupscale_factor = 0.01\n\n# resample data to target shape\ndata = dataset.read(\n    out_shape=(\n        dataset.count,\n        int(dataset.height * upscale_factor),\n        int(dataset.width * upscale_factor)\n    ),\n    resampling=Resampling.bilinear\n)\n\n# scale image transform\ntransform = dataset.transform * dataset.transform.scale(\n    (dataset.width / data.shape[-1]),\n    (dataset.height / data.shape[-2])\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mpl.rcParams['figure.dpi'] = 200\nmpl.rcParams['font.size'] = 8\nimage_hidden = plt.imshow(np.log10(data[0,:,:]))\nplt.colorbar(image_hidden)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It's apparent that more populated areas around the world have larger carbon footprints, which makes sense.\n\nWhat is the coordinate reference system of these data?"},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset.crs","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The coordinate reference system of these data need to match other systems, to be spatially joined."},{"metadata":{},"cell_type":"markdown","source":"# World urban areas shapefile\nIn order to join the global gridded data, we need to identify which regions of the gridded data correspond to cities in the competition data set. This public data set has polygons describing major cities around the world (https://geo.nyu.edu/catalog/stanford-yk247bg4748)."},{"metadata":{"trusted":true},"cell_type":"code","source":"gdf = gpd.read_file('../input/world-urban-areas-landscan-110-million-2012/ne_10m_urban_areas_landscan.shp')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gdf.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gdf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%time gdf.plot(figsize=(12,8))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The city polygons across the world form a faint impression of global land masses. The largest concentrations of cities appear to be in India and China, which are the most populated countries."},{"metadata":{},"cell_type":"markdown","source":"Let's try to find Los Angeles in here to plot one city."},{"metadata":{"trusted":true},"cell_type":"code","source":"LA_mask = gdf['name_conve'].str.contains('Angeles')\nsum(LA_mask)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gdf[LA_mask]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It looks like there are two cities in the data called \"Los Angeles\". What do they look like?"},{"metadata":{"trusted":true},"cell_type":"code","source":"gdf[gdf['name_conve'] == 'Los Angeles1'].plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gdf[gdf['name_conve'] == 'Los Angeles2'].plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The first one appears to be the Southern California metropolis. I'll assume that for other cities in these data, the first one is the largest city and the one I'll take for matching with other data."},{"metadata":{},"cell_type":"markdown","source":"What's the coordinate reference system here?"},{"metadata":{"trusted":true},"cell_type":"code","source":"gdf.crs","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is different than the carbon footprint, requiring reprojection."},{"metadata":{},"cell_type":"markdown","source":"# Reproject carbon footprints data and save"},{"metadata":{"trusted":true},"cell_type":"code","source":"from rasterio.warp import calculate_default_transform, reproject, Resampling","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This EPSG code comes from the metadata shown above."},{"metadata":{"trusted":true},"cell_type":"code","source":"dst_crs = 'EPSG:4326'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Reprojection code adapted from https://rasterio.readthedocs.io/en/latest/topics/reproject.html"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nwith rasterio.open('../input/global-gridded-model-of-carbon-footprints-ggmcf/GGMCF_v1.0.tif') as src:\n    transform, width, height = calculate_default_transform(\n        src.crs, dst_crs, src.width, src.height, *src.bounds)\n    kwargs = src.meta.copy()\n    kwargs.update({\n        'crs': dst_crs,\n        'transform': transform,\n        'width': width,\n        'height': height\n    })\n\n    with rasterio.open('/kaggle/working/reprojected.tif', 'w', **kwargs) as dst:\n        for i in range(1, src.count + 1):\n            reproject(\n                source=rasterio.band(src, i),\n                destination=rasterio.band(dst, i),\n                src_transform=src.transform,\n                src_crs=src.crs,\n                dst_transform=transform,\n                dst_crs=dst_crs,\n                resampling=Resampling.nearest)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Plot carbon footprint for one city"},{"metadata":{},"cell_type":"markdown","source":"Now I show how to mask the carbon footprint data, which span the globe, to an individual city, LA."},{"metadata":{"trusted":true},"cell_type":"code","source":"shapes = gdf[gdf['name_conve'] == 'Los Angeles1']['geometry']\nshapes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from rasterio.mask import mask","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with rasterio.open('reprojected.tif') as src:\n    out_image, out_transform = mask(src, shapes, crop=True)\n    out_meta = src.meta","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axs = plt.subplots(1,2)\ngdf[gdf['name_conve'] == 'Los Angeles1'].plot(ax=axs[0])\nshow(out_image, ax=axs[1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can also gather statistics on the carbon footprint data, such as adding up the carbon footprint for the LA metro area. I will just use the sum to add up the footprint over the whole city, but may come back later to add in other statistics."},{"metadata":{"trusted":true},"cell_type":"code","source":"out_image.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"out_image.sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Cities in the CDP data"},{"metadata":{},"cell_type":"markdown","source":"Now that we've done it for one city, the next step is to match up city names from the CDP competition data with the world urban areas data, in order to capture the carbon footprints. Here are the cities in the CDP data:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/cdp-unlocking-climate-solutions/Cities/Cities Disclosing/2019_Cities_Disclosing_to_CDP.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df['City'].isnull().mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looks like `City` is the field I want, but sometimes it's null. I'll fill the missing data with `Organization`."},{"metadata":{"trusted":true},"cell_type":"code","source":"df['City_filled'] = df['City'].fillna(df['Organization'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df[['City', 'Organization', 'City_filled']].head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now search for cities in the map data that match the city name in the CDP data, `break`ing the first time a match is found in order to take a city name ending in 1 in case of multiples."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nmatched = []\nmatch_count = []\nfor city_1 in df['City_filled'].tolist():\n    for city_2 in gdf['name_conve'].tolist():\n        if re.match(city_1, city_2):\n            matched.append(city_2)\n            match_count.append(1)\n            break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(match_count)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So we've found polygons for..."},{"metadata":{"trusted":true},"cell_type":"code","source":"345/861","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"about 40% of the cities. Which isn't perfect but seems like a reasonable amount to do some analysis. A more detailed look here may yield more matches."},{"metadata":{},"cell_type":"markdown","source":"# Carbon footprints for cities in CDP data\nTBD"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}