{"cells":[{"metadata":{"trusted":true},"cell_type":"markdown","source":"# KPIs to Keep an eye on\n\n## Quantifying Environmental Issues with a Social Context"},{"metadata":{},"cell_type":"markdown","source":"## Content:\n- [Introduction](#introduction)\n    * [Problem Statement](#problem-statement)\n- [The Data](#the-data)\n- [Analysis](#analysis)\n    * [Climate Hazards](#climatehazards)\n    * [Adaptation](#adaptation)\n    * [Clustering with various metrics](#clustering)\n        * [Clustering Categorical Features](#catclustering)\n        * [Clustering Numerical Features](#numclustering)\n- [Conclusion](#conclusion)"},{"metadata":{},"cell_type":"markdown","source":"# Introduction<a class=\"anchor\" id=\"introduction\"></a>\nThe Carbon Disclosure Project (CDP) is a global non-profit that drives companies and governments to reduce their greenhouse gas emissions, safeguard water resources, and protect forests. Each year, CDP takes the information supplied in its annual reporting process and scores companies and cities based on their journey through disclosure and towards environmental leadership."},{"metadata":{},"cell_type":"markdown","source":"### Problem Statement <a class=\"anchor\" id=\"problem-statement\"></a>\nDevelop a methodology for calculating key performance indicators (KPIs) that relate to the environmental and social issues that are discussed in the CDP survey data. Leverage external data sources and thoroughly discuss the intersection between environmental issues and social issues. Mine information to create automated insight generation demonstrating whether city and corporate ambitions take these factors into account."},{"metadata":{},"cell_type":"markdown","source":"## The Data<a class=\"anchor\" id=\"the-data\"></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\npd.set_option('display.max_columns', None)\npd.set_option('display.max_rows', None)\npd.set_option('display.max_colwidth', -1)\nimport re\nimport matplotlib.pyplot as plt\nimport plotly.graph_objects as go\n\nfrom kmodes.kmodes import KModes\nfrom sklearn import preprocessing\nfrom sklearn.cluster import KMeans\nfrom sklearn.preprocessing import StandardScaler\n\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Climate crisis is acknowledged throughout the world. Putting some numbers to the issue takes away the ambiguity and gives us an objective perspective to move forward."},{"metadata":{},"cell_type":"markdown","source":"The data consists of responses from 1032 different cities reported between 2018 and 2020. For simplicity, the 2019 dataset was used as it had the most reporting cities (861) and fairly up-to-date questionaire. Cities reporting in 2019 account for approximately 2.2 billion people. To add perspective, the current world population is 7.6 billion.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# loading city details and concatenating data\ncities18 = pd.read_csv('../input/cdp-unlocking-climate-solutions/Cities/Cities Disclosing/2018_Cities_Disclosing_to_CDP.csv')\ncities19 = pd.read_csv('../input/cdp-unlocking-climate-solutions/Cities/Cities Disclosing/2019_Cities_Disclosing_to_CDP.csv')\ncities20 = pd.read_csv('../input/cdp-unlocking-climate-solutions/Cities/Cities Disclosing/2020_Cities_Disclosing_to_CDP.csv')\ncitiesall = pd.concat([cities18,cities19, cities20])\n\n# loading responses and concatening data\nresp18 = pd.read_csv('../input/cdp-unlocking-climate-solutions/Cities/Cities Responses/2018_Full_Cities_Dataset.csv')\nresp19 = pd.read_csv('../input/cdp-unlocking-climate-solutions/Cities/Cities Responses/2019_Full_Cities_Dataset.csv')\nresp20 = pd.read_csv('../input/cdp-unlocking-climate-solutions/Cities/Cities Responses/2020_Full_Cities_Dataset.csv')\nrespall = pd.concat([resp18,resp19,resp20])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def filtq(qdf, qnum, cnum, rnum):\n    '''filters CDP questionaire dataset by question, column and row number'''\n    filcon = (qdf['Question Number'].apply(lambda x: x in qnum)) & (qdf['Column Number'].apply(lambda x: x in cnum)) & (qdf['Row Number'].apply(lambda x: x in rnum))\n    return qdf[filcon]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"city_count_yearly = [len(citdf['Account Number'].unique()) for citdf in [cities18, cities19, cities20]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"city_series = pd.concat([cities18['Account Number'],cities19['Account Number'],cities20['Account Number']])\ncity_count_total = len(city_series.unique())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"city_count_yearly","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"city_count_total","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import geopandas\nimport matplotlib.pyplot as plt\nfrom matplotlib.pyplot import figure\nfrom shapely import wkt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"world = geopandas.read_file(geopandas.datasets.get_path('naturalearth_lowres'))\nax = world.plot(color='white', edgecolor='black', figsize=(20,20))\ndf = cities19[['City','Country','City Location']].dropna(subset=['City Location'])\ndf['City Location'] = df['City Location'].apply(wkt.loads)\ngdf = geopandas.GeoDataFrame(df, geometry='City Location')\ngdf.plot(ax=ax, color='red')\nplt.title('World Map of Disclosing Cities in 2019')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Acknowledgement and Planning is the first step in tackling any problem. Most of the reporting cities have created climate vulnerability assessments or intend to do so in the near future."},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# filtering to vulnerability assesment question\nax = resp19[resp19['Question Number']=='2.0']\\\n.groupby('Response Answer').count().sort_values(by=['Account Number'], ascending=False)['Account Number']\\\n.plot.barh()\n\nplt.title('Climate Change Vulnerability Assessments')\nplt.xlabel('Number of Cities')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# filtering hazards and probability of hazards\ntempdf2 = resp19[(resp19['Question Number']=='2.1') & ((resp19['Column Number']==1) | (resp19['Column Number']==3))]\npivind = pd.MultiIndex.from_tuples(list(zip(tempdf2['Account Number'],tempdf2['Row Number'])), names=['Account Number', 'Row Number'])\ntempdf2['Pivot Index'] = pivind\ntempdf2 = tempdf2.pivot(values='Response Answer',columns='Column Name',index='Pivot Index')\ntempdf2 = tempdf2.reset_index().groupby(['Climate Hazards','Current probability of hazard']).count().reset_index()\ntempdf2 = tempdf2.pivot(values='Pivot Index',index='Climate Hazards',columns='Current probability of hazard')\ntempdf2 = tempdf2[['High','Medium High','Medium','Medium Low','Low','Does not currently impact the city','Do not know']]\ntempdf2['Total']=tempdf2.sum(axis=1)\ntempdf2 = tempdf2.sort_values(by='Total', ascending=False).head(16).drop(columns=['Total'])\n# top most listed hazards\ntophaz = tempdf2.index","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntempdf2.plot.barh(stacked=True, figsize=(10,10), color=['red','darkorange','orange','gold','khaki','lightgreen','lightblue'])\nplt.gca().invert_yaxis()\nplt.title('Top 15 most listed Climate Hazards')\nplt.xlabel('Number of Cities')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# hazards listed\ntempdf = resp19[(resp19['Question Number']=='2.1')\\\n                & (resp19['Column Number']==1)]\ntempdf = tempdf[tempdf['Response Answer'].apply(lambda x: x in tophaz)]\n# specific population affected by hazard\ntempdf2 = resp19[(resp19['Question Number']=='2.1')\\\n       & ((resp19['Column Number']==10))]\n\npivind = list(zip(tempdf['Account Number'],tempdf['Row Number']))\npivind2 = list(zip(tempdf2['Account Number'],tempdf2['Row Number']))\ntempdf['Pivot Index'] = pivind\ntempdf2['Pivot Index'] = pivind2\n\ntopcols = tempdf2.groupby('Response Answer').count().sort_values(by=['Account Number'],ascending=False).head(16).index\ntempdf2 = tempdf2.pivot(values='Account Number',index='Pivot Index',columns='Response Answer')\n\ntempdf3 = tempdf.merge(tempdf2[topcols].notna().reset_index(), left_on='Pivot Index', right_on='Pivot Index',\\\n          suffixes=('_left', '_right'))\ntempdf3 = tempdf3.groupby('Response Answer').sum()[topcols].loc[tophaz[0:10]]\ntempdf4 = tempdf3.reset_index().melt(id_vars=['Climate Hazards'])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_nodes = tempdf4['Climate Hazards'].values.tolist() + tempdf4['variable'].values.tolist()\nsource_indices = [all_nodes.index(haz) for haz in tempdf4['Climate Hazards']]\ntarget_indices = [all_nodes.index(affpop) for affpop in tempdf4['variable']]\n\nfig = go.Figure(data=[go.Sankey(\n    node = dict(\n      pad = 20,\n      thickness = 20,\n      line = dict(color = \"black\", width = 1.0),\n      label =  all_nodes,\n    ),\n\n    link = dict(\n      source =  source_indices,\n      target =  target_indices,\n      value =  tempdf4.value,\n))])\n\nfig.update_layout(title_text=\"Popluations Affected by Climate Hazards\",\n                  font_size=10)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The sankey diagram shows the number cities listing a specific popluation as most affected by a given hazard. Although Extreme Precipitation > rainstorm was the most listed hazard, certain populations such as elderly, childern and low-income households, are most impacted by extremely hot temperatures."},{"metadata":{},"cell_type":"markdown","source":"### Adaptation<a class=\"anchor\" id=\"adaptation\"></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"adap19 = filtq(resp19, '3.0', [1,2,8,10], list(range(1000)))\npivind = pd.MultiIndex.from_tuples(list(zip(adap19['Account Number'],adap19['Row Number'])), names=['Account Number', 'Row Number'])\nadap19['Pivot Index'] = pivind\nadap19 = adap19.pivot(values='Response Answer',columns='Column Name', index='Pivot Index')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"adact19 = adap19.reset_index().groupby('Action').count().sort_values(by=['Pivot Index'], ascending=False).head(10)['Pivot Index']\nax = adact19.plot.barh()\nplt.gca().invert_yaxis()\nplt.title('Action taken for Hazards')\nplt.xlabel('Number of times listed')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The top actions taken to combate climate hazards were planting trees and flood mapping. These actions directly address the top climate hazards which were extreme temperatures, precipitation/flooding and drought. These steps are the appropriate approach to handle the most common issues."},{"metadata":{"trusted":true},"cell_type":"code","source":"# factors affecting adaption\nadaff = filtq(resp19,'2.2',[1,2,3],[1])\npivind = pd.MultiIndex.from_tuples(list(zip(adaff['Account Number'],adaff['Row Number'])), names=['Account Number', 'Row Number'])\nadaff['Pivot Index'] = pivind\nadaff = adaff.pivot(values='Response Answer',columns='Column Name', index='Pivot Index')\n\nadaff = adaff.reset_index()\\\n.groupby(['Factors that affect ability to adapt','Support / Challenge'])\\\n.count()\\\n.reset_index()[['Factors that affect ability to adapt','Pivot Index','Support / Challenge']]\nadaff['Factors that affect ability to adapt'] = adaff['Factors that affect ability to adapt'].apply(lambda x: x[0:70])\n\n\nadaff = adaff.pivot(index='Factors that affect ability to adapt',values='Pivot Index', columns='Support / Challenge')\\\n.fillna(0)\nadaff=adaff[['Support','Challenge']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"adaff.sort_values(by='Challenge')\\\n.plot.barh(figsize=(10,10))\nplt.title('Factors Affecting ability to Adapt')\nax1 = plt.axes()\ny_axis = ax1.axes.get_yaxis()\ny_label = y_axis.get_label()\ny_label.set_visible(False)\nplt.xlabel('Number of Cities')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Cities listed the above factors as an issue or strength in adapting to climate hazards. 'Access to basic services' was listed about 100 times. About 30 cities have adequate access to basic services and have listed them as a support adapting to climate change. 70 cities still need bolstering of basic services and studying the strategy of the aformentioned 30 could reveal insights to address this challenge. <br>\n<br>\nBudgetary capacity was almost always listed as a challenge suggesting more funds are needed for adaption plans."},{"metadata":{},"cell_type":"markdown","source":"### Clustering with various metrics <a class=\"anchor\" id=\"clustering\"></a>"},{"metadata":{},"cell_type":"markdown","source":"The data, as it is, reveals valuable insights and simple calculations can be adequate KPIs. However, the purpose of this endeavor is to devise a KPI that can measure whether an individual city's response effectively addresses looming environmental issues while also considering social issues."},{"metadata":{},"cell_type":"markdown","source":"Most pertinent data from the emissions, emission reduction, energy, oppurtunity, buildings, transport and waste sections of the questionaire have been extracted to compile a city level data frame."},{"metadata":{},"cell_type":"markdown","source":"It is difficult to create a standard methodology for all cities as there are many different standards of reporting. The reported metrics are not always directly comparable or substitutable. Therefore, many fields have been imputed. Perhaps once a KPI methodology is established, it will be easier to create a standard and collect the specific data readily from participating cities."},{"metadata":{"trusted":true},"cell_type":"code","source":"# compile data at city level\n# emissions, emission reduction, energy, oppurtunity, buildings, transport, waste\n\n# getting city population and area\nfiltdf = filtq(resp19, ['0.5', '0.6'], [1,3],[1])\ncitycompile = filtdf.pivot(values='Response Answer', columns='Column Name', index='Account Number')\ncitycompile = citycompile.rename(columns={\"Land area of the city boundary as defined in question 0.1 (in square km)\":\"Land Area (sq km)\"})\ncitycompile = citycompile.apply(pd.to_numeric)\n\n# getting climate if vulnerability report made\nfiltdf = filtq(resp19, ['2.0'], [0],[0])\npivdf = filtdf.pivot(values='Response Answer', columns='Row Name', index='Account Number')\npivdf.columns = ['Risk Vulnerability Assessment']\ncitycompile = citycompile.join(pivdf, how='outer')\n\n# getting emissions, CO2 metric tonnes from different scopes\nfiltdf = filtq(resp19, ['4.6b'],[1], range(100))\npivdf = filtdf.pivot(values='Response Answer', columns='Row Name', index='Account Number')\npivdf = pivdf.apply(pd.to_numeric)\ncitycompile = citycompile.join(pivdf, how='outer')\n\n# getting emiision reduction targets, %target and % achievedd\nfiltdf = filtq(resp19, ['5.0a'], [1,7,10], range(100))\nfiltdf['pivind'] = index = pd.MultiIndex.from_frame(filtdf[['Account Number','Row Number']])\npivdf = filtdf.pivot(values='Response Answer', columns='Column Name', index='pivind')\npivdf = pivdf[pivdf.Sector=='All emissions sources included in city inventory']\npivdf = pivdf.reset_index()\npivdf['Account Number'] = pivdf.pivind.apply(lambda x: x[0])\npivdf = pivdf.set_index('Account Number')[['Percentage of target achieved so far', 'Percentage reduction target']]\npivdf.columns = ['Emission Reduction Achieved (%)', 'Emission Reduction Target (%)']\npivdf = pivdf.apply(pd.to_numeric)\ncitycompile = citycompile.join(pivdf, how='outer')\n\n# getting energy data, renewable target exists and percentage of different sources\nfiltdf = filtq(resp19, ['8.0'], [0], [0])\npivdf = filtdf.pivot(values='Response Answer', columns='Column Name', index='Account Number')\npivdf.columns = ['Renewable Target Exists']\ncitycompile = citycompile.join(pivdf, how='outer')\n\nfiltdf = filtq(resp19, ['8.0a'], [5], range(20))\nfiltdf['Response Answer'] = pd.to_numeric(filtdf['Response Answer'])\nfiltdf = filtdf.groupby(['Account Number', 'Column Name']).max()['Response Answer']\nfiltdf = filtdf.reset_index(level='Column Name', drop=True).fillna(0)\ncitycompile = citycompile.join(filtdf, how='outer')\ncitycompile = citycompile.rename(columns={'Response Answer':'Renewable Energy (%)'})\n\n# getting oppurtinity data, ESG in investing, business collab, established funds,\n# green growth strategy, development bank funding, green jobs\nfiltdf = filtq(resp19, ['6.1','6.3','6.4','6.6','6.9','6.11'],[0],[0])\npivdf = filtdf.pivot(values='Response Answer', columns='Question Name', index='Account Number')\npivdf = pivdf.rename(columns={'How many people within your City are employed in green jobs/ industries?':'Green Jobs'})\npivdf['Green Jobs'] = pd.to_numeric(pivdf['Green Jobs'])\ncitycompile = citycompile.join(pivdf, how='outer')\n\n# getting buildings data, CO2 from different buildings\nfiltdf = filtq(resp19, ['9.0'], [1],range(10))\npivdf = filtdf.pivot(values='Response Answer', columns='Row Name', index='Account Number')\npivdf = pivdf.rename(columns={'Municipal' : 'Municipal Buildings', \n                              'Commercial':'Commerical Buildings',\n                             'Residential': 'Residential Buildings'})\npivdf.columns = [colname+' (CO2 Tonnes per capita)' for colname in pivdf.columns]\npivdf = pivdf.apply(pd.to_numeric)\ncitycompile = citycompile.join(pivdf, how='outer')\n\n# getting transportation data, % of different transport modes\nfiltdf = filtq(resp19, ['10.1'], range(10),range(10))\npivdf = filtdf.pivot(values='Response Answer', columns='Column Name', index='Account Number').fillna(0)\npivdf = pivdf.apply(pd.to_numeric)\ncitycompile = citycompile.join(pivdf, how='outer')\n\n# getting waste data, waste per capita\nfiltdf = filtq(resp19, ['13.0'], [1],[2])\npivdf = filtdf.pivot(values='Response Answer', columns='Row Name', index='Account Number')\npivdf = pivdf.apply(pd.to_numeric)\npivdf2 = pivdf\ncitycompile = citycompile.join(pivdf, how='outer')\n\n# copy before imputing\nunimpcitycomp = citycompile.copy()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### Imputing Missing Values"},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# imputing categorical column NA values with no\ncitycompile.loc[:,citycompile.dtypes=='object'] = citycompile.loc[:,citycompile.dtypes=='object'].fillna('No').replace('Do not know', 'No')\n\n# imputing base population with mean\ncitycompile['Current population'] = citycompile['Current population'].fillna(citycompile['Current population'].mean())\n\n# dividing emission data and green jobscolumns with base year population\nstdcolumns=['Agriculture, Forestry and Land Use – Scope 1 (V)',\n           'Industrial Processes and Product Use – Scope 1 (IV)',\n           'Stationary Energy: energy generation supplied to the grid – Scope 1 (I.4.4)',\n           'Stationary Energy: energy use – Scope 1 (I.X.1)',\n           'Stationary Energy: energy use – Scope 2 (I.X.2)',\n           'Stationary Energy: energy use – Scope 3 (I.X.3)',\n           'TOTAL BASIC emissions', 'TOTAL BASIC+ emissions',\n           'TOTAL Scope 1 (Territorial) emissions', 'TOTAL Scope 2 emissions',\n           'TOTAL Scope 3 emissions', 'Transportation – Scope 1 (II.X.1)',\n           'Transportation – Scope 2 (II.X.2)',\n           'Transportation – Scope 3 (II.X.3)',\n           'Waste: waste generated outside the city boundary – Scope 1 (III.X.3)',\n           'Waste: waste generated within the city boundary – Scope 1 (III.X.1)',\n           'Waste: waste generated within the city boundary – Scope 3 (III.X.2)',\n           'Green Jobs']\n\ncitycompile[stdcolumns] = citycompile[stdcolumns].div(citycompile['Current population'],axis=0)\n\ncitycompile.drop(columns = ['Current population','Land Area (sq km)', 'Projected population'], inplace=True)\n\n# imputing emission columns with mean\nimpmncols = ['Agriculture, Forestry and Land Use – Scope 1 (V)',\n               'Industrial Processes and Product Use – Scope 1 (IV)',\n               'Stationary Energy: energy generation supplied to the grid – Scope 1 (I.4.4)',\n               'Stationary Energy: energy use – Scope 1 (I.X.1)',\n               'Stationary Energy: energy use – Scope 2 (I.X.2)',\n               'Stationary Energy: energy use – Scope 3 (I.X.3)',\n               'TOTAL BASIC emissions', 'TOTAL BASIC+ emissions',\n               'TOTAL Scope 1 (Territorial) emissions', 'TOTAL Scope 2 emissions',\n               'TOTAL Scope 3 emissions', 'Transportation – Scope 1 (II.X.1)',\n               'Transportation – Scope 2 (II.X.2)',\n               'Transportation – Scope 3 (II.X.3)',\n               'Waste: waste generated outside the city boundary – Scope 1 (III.X.3)',\n               'Waste: waste generated within the city boundary – Scope 1 (III.X.1)',\n               'Waste: waste generated within the city boundary – Scope 3 (III.X.2)',\n               'Emission Reduction Target (%)',\n               'All building types (CO2 Tonnes per capita)',\n               'Commerical Buildings (CO2 Tonnes per capita)',\n               'Municipal Buildings (CO2 Tonnes per capita)',\n               'Residential Buildings (CO2 Tonnes per capita)',\n               'Buses (including BRT)', \n               'Cycling', \n               'Ferries/ River boats', \n               'Other',\n               'Private motorized transport', \n               'Rail/Metro/Tram',\n               'Taxis or For Hire Vehicles', \n               'Walking',\n               'Waste generation per capita (kg/person/year)'\n             ]\n\ncitycompile[impmncols] = citycompile[impmncols].fillna(citycompile[impmncols].mean())\n\n# imputing 0 for na values\nimp0cols = ['Green Jobs',\n            'New buildings (CO2 Tonnes per capita)',\n            'Emission Reduction Achieved (%)',\n            'Renewable Energy (%)'            \n           ]\ncitycompile[imp0cols] = citycompile[imp0cols].fillna(0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Clustering Categorigal Features<a class=\"anchor\" id=\"catclustering\"></a>"},{"metadata":{},"cell_type":"markdown","source":"Finding appropriate number of clusters."},{"metadata":{"trusted":true},"cell_type":"code","source":"# get categorical variables and try various number of clusters\ncitycatvars = citycompile.select_dtypes(include='object')\ncatcols = citycatvars.columns\ncost = []\nfor num_clusters in list(range(1,6)):\n    kmode = KModes(n_clusters=num_clusters, init = \"Cao\", n_init = 1)\n    kmode.fit_predict(citycatvars)\n    cost.append(kmode.cost_)\n\n# plot error vs clusters\ny = np.array([i for i in range(1,6,1)])\nplt.plot(y,cost)\nplt.title('Cost vs Number of Clusters')\nplt.xlabel('Clusters')\nplt.ylabel('Cost')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Observing the cost vs number of clusters plot, 3 seems to be a good number of clusters."},{"metadata":{"trusted":true},"cell_type":"code","source":"citycatvars = citycompile.select_dtypes(include='object')\ncatcols = citycatvars.columns\n\n# define the k-modes model\nkm = KModes(n_clusters=3, \n            init='Cao', n_init=11)\n\n# fit the clusters to the skills dataframe\nclusters = km.fit_predict(citycatvars)\n\n# get an array of cluster modes\nkmodes = km.cluster_centroids_\nshape = kmodes.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Categorigcal Features Cluster Centroids**"},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.DataFrame(data=kmodes, columns=catcols)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Kmodes clustering showed that the most distinguishing features were **whether a Renewable Energy Target exists** and **whether a Risk and Vulnerability assessment was created**. **Business partnerships for sustainability projects** was the next most distinguishing feature. The other categorical features did not serve well to create distinct clusters from the rest of the data. "},{"metadata":{},"cell_type":"markdown","source":"The Kmodes clustering revealed there are 3 most common tiers of cities: <br>\n\n-Cities who have Risk Vulnerability assessments but do not Renewable Energy Targets or Collaborate with Businesses.\n\n-Cities who Intend to create Risk Vulnerability Assessments, Renewable Energy Targets and Business Partnerships.\n\n-Cities who have Risk Vulnerability Assesments, have Renewable Energy Targets and Collaborate with Businesses on Sustainability."},{"metadata":{},"cell_type":"markdown","source":"These clusters can be thought to be a graded system in which cities are graded as being **limited, intermediate or advanced** in their progress of addressing environmental issues. Extending this idea, cities can even be scored by simply adding the number of steps taken. For example if a city has a renewable target and risk assessment, it would have covered 2 of the 7 steps (represented as columns in the above dataframe). Of course this is a rudimentary system, and needs more complexity to truly represent a cities readiness for the issues at hand. Another idea is adding more weight to certain steps. For example, establishing funds to invest in sustainability projects should hold more weight than creating a risk assessment. Such a system would be subject and it would be best to get expert opinion to assess how much weight experts would give to each step."},{"metadata":{"trusted":true},"cell_type":"code","source":"citycompile['Are Environmental, Social and Governance (ESG) issues incorporated into investment decisions of any of the city retirement funds?'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Clustering Numerical Features<a class=\"anchor\" id=\"numclustering\"></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"# scaling numerical features\ncitynumvars = citycompile.select_dtypes(exclude='object')\nscaler = StandardScaler()\nscaled_features = scaler.fit_transform(citynumvars)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"kmeans_kwargs = {\"init\": \"random\",\n                 \"n_init\": 10,\n                 \"max_iter\": 300,\n                 \"random_state\": 42,\n                }\n# finding appropriate number of clusters\nsse = []\nfor k in range(1, 11):\n    kmeans = KMeans(n_clusters=k, **kmeans_kwargs)\n    kmeans.fit(scaled_features)\n    sse.append(kmeans.inertia_)\n\n# plot error vs number of clusters\nplt.plot(range(1, 11), sse)\nplt.title('SSE vs Number of Clusters')\nplt.xticks(range(1, 11))\nplt.xlabel(\"Number of Clusters\")\nplt.ylabel(\"SSE\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Observing the above plot, about 2 to 3 clusters seem to be a good number of clusters."},{"metadata":{"trusted":true},"cell_type":"code","source":"kmeans = KMeans(n_clusters=2, **kmeans_kwargs)\nkmeans.fit(scaled_features)\nsse.append(kmeans.inertia_)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"###### Numerical Features Cluster Centers"},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.DataFrame(kmeans.cluster_centers_, columns=citynumvars.columns)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Kmeans clustering revealed there to be two common classes of cities. Cities with low emissions and cities with high emissions. The emmission data was divided by the city's base year population, therfore it is analagous to per capita emmissions. Emmissions data was the key distinguishing factor. Data on waste, renewable energy, and transportation did not play highly decisive roles in clustering. <br>\n<br>\nThere was quite a bit of imputed data. 84 of 910 cities reported waste per capita, therefore it likely did not hold great weight in distinguishing cities (the rest of the 826 cities were imputed to have 36222kg/person/year waste). The number of green jobs had a similar amount of missing data with about 53 cities reporting the metric.<br>\n<br>\nSimilar to clustering with categorical variables, these clusters can also be thought to be graded. The high emmissions cluster being limited in progress tier and low emissions cluster being the advanced tier.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"unimpcitycomp['Waste generation per capita (kg/person/year)'].count()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"unimpcitycomp['Waste generation per capita (kg/person/year)'].mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"unimpcitycomp['Green Jobs'].count()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"unimpcitycomp['Green Jobs'].mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Conclusion<a class=\"anchor\" id=\"conclusion\"></a>"},{"metadata":{},"cell_type":"markdown","source":"The data as it is provides valuable insight to the state of cities and their progress in the on going battle against environmental issues and social issues.\n\nThe Climate Hazards and Vulnerability and Adaptation sections ask key questions that provide a comprehensive view on pressing issues for cities and their strategies. From the data we found extreme temperatures and precipitation to be most common climate hazards on city radars. Children and elderly are most vulnerable to these specific hazards. Planting trees and flood mapping are the ubiquitous strategies used to address these issues.\n\nFurthermore, access to basic services and budget restrictions are the most common factors that prevent cities from adapting and implementing their strategies.\n\nThese insights were derived from basic exploratory data analysis on the given data (simple counting and summation for the most part). These insights show us how cities in general are adapting and the most risk social demographic. However it does not measure an individual city's preparedness and appropriateness of its response. It also does not reveal if the city takes into account social issues.\n\nClustering analysis provided some insight to the question at hand. Clustering on numerical data showed whether a city relatively produced low emmissions or high emmissions. These clusters were analagous to grading cities as being limited or advanced in their readiness to combat climate change and reduce emmissions. Some metrics, such as the number of Green Jobs, could have added insight into a city taking advantage of oppurtunities posed by environmental issues and tackling social issues. However, as this metric is not a standard, most cities did not report the number of Green Jobs being created.\n\nClustering on categorical data showed there to be 3 most common classes of cities based on whether they created risk assessments, created renewable energy targets and partnered with businesses for sustainability. The business partnership question in particular could shed light on cities tackling social issues as these partnerships could increase employment. The clustering could act as a grading system to classify a city as being limited, intermediate or advanced in their ability to handle environmental and social issues alike. The scoring system can be simplified to simply summing 'Yes' answers to precisely designed questions. The summed result could be a KPI indicating the cities effectiveness against environmental and social issues.\n\nIt is important to keep in mind the clustering results are relative. Even if a city is classified into an advanced tier does not mean it can't improve. It means they have set the precedent for other cities to follow but should still continue pioneering until climate change is no longer a threat. This scoring system would also change as time passes and new technologies develop. Cities in the current advance tier could fall behind into lower tiers if they do not keep up with new technology.\n\nClimate change is an impending catastrophe. This study showed that climate change is on the minds' of city administrations. Most cities have risk assessments to plan for the future and are already taking action. The plans being set into action are a promising foundation for a sustainable future for all. Continuing to improve the design of this KPI will help us stay on track by quantifying our progress towards that future.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}