{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.9","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":57988,"databundleVersionId":6263341,"sourceType":"competition"},{"sourceId":285984,"sourceType":"datasetVersion","datasetId":5779}],"dockerImageVersionId":30055,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Respiratory Disease Incidence Prediction\n\n## Project Overview\nThis project aims to predict the incidence of chronic respiratory diseases in the United States using demographic information and air pollution indicators.\n\nThe primary dataset contains health incidence data across multiple U.S. states, age groups, and years. In later stages, this dataset will be integrated with EPA hitorical air quality data.\n\nThe goal is to explore relationships between air pollution and respiratory health outcomes and build machine learning models to predict disease incidence.\n","metadata":{}},{"cell_type":"markdown","source":"## Data Loading\n\nIn this step, we load the primary dataset containing chronic respiratory disease incidence data and perform an initial inspection to understand its structure.\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\ntrain_path = \"/kaggle/input/air-toxicity-and-chronic-respiratory-diseases-us/train.csv\"\ndf = pd.read_csv(train_path)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:56.395493Z","iopub.execute_input":"2025-12-18T08:04:56.395830Z","iopub.status.idle":"2025-12-18T08:04:56.429422Z","shell.execute_reply.started":"2025-12-18T08:04:56.395736Z","shell.execute_reply":"2025-12-18T08:04:56.428088Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Dataset Structure\n\nWe examine the size of the dataset, column names, data types, and check for missing values to ensure data quality before proceeding with further analysis.\n","metadata":{}},{"cell_type":"code","source":"print(\"Shape (rows, cols):\", df.shape)\ndf.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:56.431174Z","iopub.execute_input":"2025-12-18T08:04:56.431620Z","iopub.status.idle":"2025-12-18T08:04:56.470297Z","shell.execute_reply.started":"2025-12-18T08:04:56.431586Z","shell.execute_reply":"2025-12-18T08:04:56.468886Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:56.473315Z","iopub.execute_input":"2025-12-18T08:04:56.473620Z","iopub.status.idle":"2025-12-18T08:04:56.495193Z","shell.execute_reply.started":"2025-12-18T08:04:56.473590Z","shell.execute_reply":"2025-12-18T08:04:56.494025Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Target Variable Exploration (Incidence)\n\nThe target variable represents the incidence rate of chronic respiratory diseases. We analyze its distribution and summary statistics to understand its range and variability.This means how many new cases there are in a year divided by how many people there are in the middle of the year.","metadata":{}},{"cell_type":"code","source":"df[\"Incidence\"].describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:56.497882Z","iopub.execute_input":"2025-12-18T08:04:56.498253Z","iopub.status.idle":"2025-12-18T08:04:56.510983Z","shell.execute_reply.started":"2025-12-18T08:04:56.498223Z","shell.execute_reply":"2025-12-18T08:04:56.510066Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df[\"Incidence\"].hist(bins=30)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:56.512448Z","iopub.execute_input":"2025-12-18T08:04:56.512756Z","iopub.status.idle":"2025-12-18T08:04:56.766616Z","shell.execute_reply.started":"2025-12-18T08:04:56.512725Z","shell.execute_reply":"2025-12-18T08:04:56.765290Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Interpretation of Target Variable Distribution (Incidence)\n\nThe target variable *Incidence* represents the annual incidence rate of chronic respiratory diseases.\n\n### Summary Statistics Interpretation\n- The dataset contains **4,500 observations**, indicating sufficient data for statistical analysis and machine learning modeling.\n- The **mean incidence rate** is approximately **4,705**, while the **median (50%)** is around **4,000**, suggesting a slightly right-skewed distribution.\n- The incidence values range from approximately **2,725 (minimum)** to **9,200 (maximum)**, indicating substantial variability across age groups, states, and years.\n- The relatively high standard deviation (~1,526) reflects notable dispersion in disease incidence across the dataset.\n\n### Distribution Shape Analysis\nThe histogram shows that:\n- Most observations are concentrated between **3,000 and 6,000** incidence units.\n- A noticeable right tail exists, with fewer observations at higher incidence values (above 7,000).\n- This right-skewness suggests that while most populations experience moderate incidence rates, certain state-age-year combinations suffer significantly higher disease incidence.\n\n### Implications for Modeling\n- The wide range and skewed distribution indicate that **non-linear models** (e.g., Random Forest, Gradient Boosting) may capture patterns better than purely linear approaches.\n- Scaling and potential transformation of the target variable may be considered during the modeling phase.\n","metadata":{}},{"cell_type":"markdown","source":"## Age Group Analysis\n\nThe dataset categorizes the elderly population into predefined age ranges (e.g., 65–69, 70–74, etc.).\nIn this step, we analyze how the incidence of chronic respiratory diseases varies across different age groups.\n\nUnderstanding age-related patterns is important because susceptibility to chronic respiratory diseases is expected to increase with age due to physiological decline and prolonged exposure to environmental risk factors.\n","metadata":{}},{"cell_type":"code","source":"#Are all categories represented?\ndf[\"Age\"].value_counts().sort_index()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:56.768447Z","iopub.execute_input":"2025-12-18T08:04:56.768936Z","iopub.status.idle":"2025-12-18T08:04:56.780990Z","shell.execute_reply.started":"2025-12-18T08:04:56.768871Z","shell.execute_reply":"2025-12-18T08:04:56.778238Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#The relation between age and Incidence\ndf.groupby(\"Age\")[\"Incidence\"].mean()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:56.783477Z","iopub.execute_input":"2025-12-18T08:04:56.784015Z","iopub.status.idle":"2025-12-18T08:04:56.801541Z","shell.execute_reply.started":"2025-12-18T08:04:56.783960Z","shell.execute_reply":"2025-12-18T08:04:56.799919Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df.groupby(\"Age\")[\"Incidence\"].mean().plot(kind=\"bar\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:56.803829Z","iopub.execute_input":"2025-12-18T08:04:56.804331Z","iopub.status.idle":"2025-12-18T08:04:56.962033Z","shell.execute_reply.started":"2025-12-18T08:04:56.804285Z","shell.execute_reply":"2025-12-18T08:04:56.960973Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Interpretation of Age–Incidence Relationship\n\nThe analysis shows that the incidence of chronic respiratory diseases generally increases with age up to the 75–79 age group, where the highest average incidence is observed.\n\nAfter this peak, a gradual decline in incidence is noticeable in older age groups (80+). This pattern suggests that the relationship between age and disease incidence is non-linear rather than strictly increasing.","metadata":{}},{"cell_type":"markdown","source":"## Geographic Analysis (State-Level Patterns)\n\nThis section explores how the incidence of chronic respiratory diseases varies across different U.S. states.\n\nGeographic analysis is important because environmental exposure, healthcare access, urbanization, and industrial activity can differ significantly between states, potentially influencing respiratory health outcomes.\n","metadata":{}},{"cell_type":"code","source":"df[\"State.Name\"].nunique()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:56.963558Z","iopub.execute_input":"2025-12-18T08:04:56.963987Z","iopub.status.idle":"2025-12-18T08:04:56.973045Z","shell.execute_reply.started":"2025-12-18T08:04:56.963954Z","shell.execute_reply":"2025-12-18T08:04:56.971812Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"state_incidence = df.groupby(\"State.Name\")[\"Incidence\"].mean().sort_values(ascending=False)\nstate_incidence","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:56.975017Z","iopub.execute_input":"2025-12-18T08:04:56.976095Z","iopub.status.idle":"2025-12-18T08:04:56.992996Z","shell.execute_reply.started":"2025-12-18T08:04:56.976046Z","shell.execute_reply":"2025-12-18T08:04:56.991981Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"state_incidence.head(25).plot(kind=\"bar\", figsize=(8,4))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:56.994634Z","iopub.execute_input":"2025-12-18T08:04:56.995071Z","iopub.status.idle":"2025-12-18T08:04:57.280367Z","shell.execute_reply.started":"2025-12-18T08:04:56.995035Z","shell.execute_reply":"2025-12-18T08:04:57.279072Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Interpretation of Geographic Analysis\n\nThe geographic analysis reveals substantial variation in the incidence of chronic respiratory diseases across U.S. states.\n\nStates such as Montana, Colorado, and Indiana exhibit notably higher average incidence rates, while regions like the District of Columbia, Arkansas, and North Dakota show comparatively lower values.\n\nSince age groups are equally represented in the dataset, these differences cannot be attributed to demographic imbalance. Instead, they suggest that regional factors—such as air quality, industrial activity, climate conditions, and environmental exposure—may play a significant role in influencing respiratory health outcomes.\n\nThese findings strongly motivate the integration of external air pollution data to further investigate environmental determinants of chronic respiratory disease incidence.\n","metadata":{}},{"cell_type":"markdown","source":"## Temporal Analysis (Year)\n\nThis section examines how the incidence of chronic respiratory diseases changes over time.\n\nAnalyzing temporal trends is important to identify whether disease incidence is increasing, decreasing, or remaining stable across years. Such trends may reflect changes in environmental exposure, healthcare quality, regulations, or population-level risk factors.","metadata":{}},{"cell_type":"code","source":"df.groupby(\"Year\")[\"Incidence\"].mean()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:57.282297Z","iopub.execute_input":"2025-12-18T08:04:57.282612Z","iopub.status.idle":"2025-12-18T08:04:57.294402Z","shell.execute_reply.started":"2025-12-18T08:04:57.282583Z","shell.execute_reply":"2025-12-18T08:04:57.293235Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df.groupby(\"Year\")[\"Incidence\"].mean().plot(figsize=(8,4))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:57.296041Z","iopub.execute_input":"2025-12-18T08:04:57.296396Z","iopub.status.idle":"2025-12-18T08:04:57.475431Z","shell.execute_reply.started":"2025-12-18T08:04:57.296342Z","shell.execute_reply":"2025-12-18T08:04:57.474178Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Observations\n\n- From the early 1990s to the mid-1990s, the average incidence shows a noticeable decline.\n- Between approximately 2005 and 2015, there is a sharp and sustained increase in incidence rates.\n- After reaching a peak around 2016–2017, the incidence slightly decreases toward the most recent years.\n\nThis upward trend in later years may indicate increasing exposure to environmental pollutants, aging population effects, or improved disease detection and reporting.\n","metadata":{}},{"cell_type":"markdown","source":"## Exploratory Data Analysis Summary\n\nKey findings from the exploratory analysis include:\n\n- The dataset is well-balanced across age groups, with equal representation.\n- Incidence rates vary significantly across different age categories.\n- Clear geographical differences exist between states, indicating spatial variability.\n- A strong temporal trend is observed, especially after 2005.\n- No missing values were detected in the main dataset.\n","metadata":{}},{"cell_type":"code","source":"df.duplicated().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:57.477104Z","iopub.execute_input":"2025-12-18T08:04:57.477534Z","iopub.status.idle":"2025-12-18T08:04:57.488996Z","shell.execute_reply.started":"2025-12-18T08:04:57.477497Z","shell.execute_reply":"2025-12-18T08:04:57.488009Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df.isnull().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:57.490529Z","iopub.execute_input":"2025-12-18T08:04:57.490880Z","iopub.status.idle":"2025-12-18T08:04:57.501416Z","shell.execute_reply.started":"2025-12-18T08:04:57.490846Z","shell.execute_reply":"2025-12-18T08:04:57.500381Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Preliminary checks confirmed that:\n- No duplicate records are present.\n- No missing values were detected.\n- The dataset is already well-structured and clean.","metadata":{}},{"cell_type":"markdown","source":"##  Data Integration\n\n","metadata":{}},{"cell_type":"markdown","source":"## 🌫️ Air Pollution Variables Description\n\nThis study incorporates key air pollutant indicators commonly used in environmental and public health research. Each pollutant represents a distinct source and health impact mechanism.\n\n### 🔹 PM2.5 (Fine Particulate Matter ≤ 2.5 µm)\nPM2.5 refers to fine inhalable particles with diameters of 2.5 micrometers or smaller. Due to their small size, these particles can penetrate deep into the lungs and enter the bloodstream. PM2.5 exposure is strongly associated with respiratory and cardiovascular diseases, including asthma, chronic bronchitis, and premature mortality.\n\n### 🔹 PM10 (Particulate Matter ≤ 10 µm)\nPM10 consists of inhalable particles with diameters up to 10 micrometers. These particles primarily affect the upper respiratory tract and are commonly generated by dust, construction activities, and road traffic. Long-term exposure can exacerbate respiratory conditions such as asthma and reduce lung function.\n\n### 🔹 SO₂ (Sulfur Dioxide)\nSulfur dioxide is a gaseous pollutant mainly produced by the combustion of fossil fuels, especially coal and oil. Short-term exposure to SO₂ can cause airway inflammation, bronchoconstriction, and respiratory symptoms, particularly in individuals with asthma or pre-existing lung conditions.\n\n### 🔹 NO₂ (Nitrogen Dioxide)\nNitrogen dioxide is a reactive gas formed during high-temperature combustion processes, primarily from vehicle emissions and power plants. NO₂ contributes to airway irritation and reduced lung function and plays a significant role in the formation of secondary pollutants such as ozone and particulate matter.\n\n### 🔹 CO (Carbon Monoxide)\nCarbon monoxide is a colorless, odorless gas produced by incomplete combustion of carbon-containing fuels. CO reduces the oxygen-carrying capacity of blood by binding to hemoglobin, leading to hypoxia. Prolonged exposure is associated with cardiovascular stress and increased risk of respiratory-related hospital admissions.\n\n### 🔹 O₃ (Ozone)\nGround-level ozone is a secondary pollutant formed through photochemical reactions between nitrogen oxides and volatile organic compounds under sunlight. Ozone is a powerful respiratory irritant that can cause inflammation, decreased lung function, and worsening of asthma and other chronic respiratory diseases.\n\n### 🔹 Summary\nCollectively, these pollutants represent major environmental risk factors influencing respiratory health. Their inclusion in predictive modeling allows for a comprehensive assessment of air quality impacts on chronic respiratory disease incidence.\n","metadata":{}},{"cell_type":"code","source":"states_25 = sorted(df[\"State.Name\"].unique())\nyears = sorted(df[\"Year\"].unique())\n\nprint(\"Number of states:\", len(states_25))\nprint(\"Year range:\", min(years), \"to\", max(years))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:57.502850Z","iopub.execute_input":"2025-12-18T08:04:57.503210Z","iopub.status.idle":"2025-12-18T08:04:57.516258Z","shell.execute_reply.started":"2025-12-18T08:04:57.503179Z","shell.execute_reply":"2025-12-18T08:04:57.515242Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import bq_helper\nfrom bq_helper import BigQueryHelper\n\n# create a helper object for our bigquery dataset\nhistorical_air_quality = bq_helper.BigQueryHelper(active_project=\"bigquery-public-data\", \\\n                                                  dataset_name='epa_historical_air_quality')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:57.517598Z","iopub.execute_input":"2025-12-18T08:04:57.518086Z","iopub.status.idle":"2025-12-18T08:04:57.534098Z","shell.execute_reply.started":"2025-12-18T08:04:57.518054Z","shell.execute_reply":"2025-12-18T08:04:57.532959Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"bq_assistant = BigQueryHelper('bigquery-public-data', 'epa_historical_air_quality')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:57.535984Z","iopub.execute_input":"2025-12-18T08:04:57.536316Z","iopub.status.idle":"2025-12-18T08:04:57.555515Z","shell.execute_reply.started":"2025-12-18T08:04:57.536282Z","shell.execute_reply":"2025-12-18T08:04:57.554286Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"historical_air_quality.list_tables()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:57.556850Z","iopub.execute_input":"2025-12-18T08:04:57.557304Z","iopub.status.idle":"2025-12-18T08:04:58.046068Z","shell.execute_reply.started":"2025-12-18T08:04:57.557257Z","shell.execute_reply":"2025-12-18T08:04:58.044846Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"historical_air_quality.head('co_daily_summary')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:58.047881Z","iopub.execute_input":"2025-12-18T08:04:58.048312Z","iopub.status.idle":"2025-12-18T08:04:58.667465Z","shell.execute_reply.started":"2025-12-18T08:04:58.048279Z","shell.execute_reply":"2025-12-18T08:04:58.666332Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 1. Ortak Değişkenler (Eyalet Listesi ve Tarih)\ntarget_states = [\n    'Alabama', 'Alaska', 'Arizona', 'Arkansas', 'California', 'Colorado', \n    'Connecticut', 'District of Columbia', 'Florida', 'Georgia', 'Hawaii', \n    'Idaho', 'Indiana', 'Iowa', 'Michigan', 'Mississippi', 'Montana', \n    'North Carolina', 'North Dakota', 'Ohio', 'Oklahoma', 'Oregon', \n    'Tennessee', 'Texas', 'Washington'\n]\n\n# SQL için liste formatı\nstates_tuple = \"'\" + \"', '\".join(target_states) + \"'\"\n\n# 2. Genel Sorgu Fonksiyonu\ndef get_epa_pollutant_data(table_name, value_col_name):\n    \"\"\"\n    Verilen tablo ismine göre BigQuery'den optimize edilmiş veriyi çeker.\n    \"\"\"\n    QUERY = f\"\"\"\n        SELECT \n            state_name as STATE, \n            date_local as DATE,\n            -- Birimi bilgi olarak alalım (Tekrarı önlemek için ANY_VALUE)\n            ANY_VALUE(units_of_measure) as UNIT, \n            -- Ortalamayı alıp istediğimiz kolon ismini verelim\n            AVG(arithmetic_mean) as {value_col_name}\n        FROM \n            `bigquery-public-data.epa_historical_air_quality.{table_name}`\n        WHERE \n            state_name IN ({states_tuple})\n            AND date_local BETWEEN '1990-01-01' AND '2019-12-31'\n        GROUP BY\n            state_name, date_local\n        ORDER BY\n            date_local\n    \"\"\"\n    \n    print(f\"Veri çekiliyor: {table_name} ...\")\n    return bq_assistant.query_to_pandas(QUERY)\n\n# 3. Her Kirletici İçin Fonksiyonu Çağırma\n\n# NO2 (Azot Dioksit)\ndf_no2 = get_epa_pollutant_data('no2_daily_summary', 'NO2')\n\n# PM10 (Partikül Madde 10)\ndf_pm10 = get_epa_pollutant_data('pm10_daily_summary', 'PM10')\n\n# PM2.5 (Partikül Madde 2.5 - FRM yani Resmi Metod tablosunu kullanıyoruz)\ndf_pm25 = get_epa_pollutant_data('pm25_frm_daily_summary', 'PM25')\n\n# O3 (Ozon)\ndf_o3 = get_epa_pollutant_data('o3_daily_summary', 'O3')\n\n# SO2 (Kükürt Dioksit)\ndf_so2 = get_epa_pollutant_data('so2_daily_summary', 'SO2')\n\n# co (Kükürt Dioksit)\ndf_co = get_epa_pollutant_data('co_daily_summary', 'CO')\n\nprint(\"\\n--- İŞLEM TAMAMLANDI ---\")\nprint(f\"df_no2 satır sayısı : {len(df_no2)}\")\nprint(f\"df_pm10 satır sayısı: {len(df_pm10)}\")\nprint(f\"df_pm25 satır sayısı: {len(df_pm25)}\")\nprint(f\"df_o3 satır sayısı  : {len(df_o3)}\")\nprint(f\"df_so2 satır sayısı : {len(df_so2)}\")\nprint(f\"df_co satır sayısı : {len(df_co)}\")\n\n\n# Sonucu kontrol et\nprint(df_no2.head())\nprint(df_o3.head())\nprint(df_pm25.head())\nprint(df_pm10.head())\nprint(df_so2.head())\nprint(df_co.head())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:04:58.669131Z","iopub.execute_input":"2025-12-18T08:04:58.669426Z","iopub.status.idle":"2025-12-18T08:06:18.407706Z","shell.execute_reply.started":"2025-12-18T08:04:58.669397Z","shell.execute_reply":"2025-12-18T08:06:18.406599Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def get_annual_average(df, pollutant_col):\n    \"\"\"\n    Günlük veriyi alır, yıllık ortalamaya çevirir.\n    \"\"\"\n    # 1. Veri boş mu kontrol et (Hata almamak için)\n    if df is None or df.empty:\n        return None\n\n    # 2. Tarih formatını garantiye al ve YIL sütunu oluştur\n    df['DATE'] = pd.to_datetime(df['DATE'])\n    df['YEAR'] = df['DATE'].dt.year\n    \n    # 3. Eyalet ve Yıla göre grupla, ilgili kirleticinin ortalamasını al\n    df_annual = df.groupby(['STATE', 'YEAR'])[pollutant_col].mean().reset_index()\n    \n    # 4. Okunabilirlik için yuvarla (2 basamak)\n    df_annual[pollutant_col] = df_annual[pollutant_col].round(2)\n    \n    return df_annual\n\n# --- FONKSİYONU TÜM DATAFRAME'LERE UYGULAMA ---\n\n# df_CO, df_no2 vb. değişkenlerin daha önce tanımlı olduğunu varsayıyoruz\ndf_co_annual   = get_annual_average(df_co, 'CO')\ndf_no2_annual  = get_annual_average(df_no2, 'NO2')\ndf_pm10_annual = get_annual_average(df_pm10, 'PM10')\ndf_pm25_annual = get_annual_average(df_pm25, 'PM25')\ndf_o3_annual   = get_annual_average(df_o3, 'O3')\ndf_so2_annual  = get_annual_average(df_so2, 'SO2')\n\n# Örnek Kontrol\nprint(\"--- PM2.5 Yıllık Ortalama (İlk 5 Satır) ---\")\nif df_pm25_annual is not None:\n    print(df_pm25_annual.head())\n\nprint(\"\\n--- İŞLEM TAMAMLANDI ---\")\nprint(f\"df_co satır sayısı : {len(df_co_annual)}\")\nprint(f\"df_no2 satır sayısı: {len(df_no2_annual)}\")\nprint(f\"df_pm10 satır sayısı: {len(df_pm10_annual)}\")\nprint(f\"df_pm25 satır sayısı  : {len(df_pm25_annual)}\")\nprint(f\"df_o3 satır sayısı : {len(df_o3_annual)}\")\nprint(f\"df_so2 satır sayısı : {len(df_so2_annual)}\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:18.409550Z","iopub.execute_input":"2025-12-18T08:06:18.409866Z","iopub.status.idle":"2025-12-18T08:06:19.382873Z","shell.execute_reply.started":"2025-12-18T08:06:18.409833Z","shell.execute_reply":"2025-12-18T08:06:19.381847Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Kontrol edilecek DataFrame listesi ve isimleri\npollutant_list = [\n    ('CO (Karbonmonoksit)', df_co_annual),\n    ('NO2 (Azot Dioksit)', df_no2_annual),\n    ('PM10 (Partikül Madde 10)', df_pm10_annual),\n    ('PM2.5 (Partikül Madde 2.5)', df_pm25_annual),\n    ('O3 (Ozon)', df_o3_annual),\n    ('SO2 (Kükürt Dioksit)', df_so2_annual)\n]\n\nsummary_data = []\n\nfor name, df in pollutant_list:\n    if df is not None and not df.empty:\n        min_year = df['YEAR'].min()\n        max_year = df['YEAR'].max()\n        unique_years = df['YEAR'].nunique() # Kaç farklı yıl var?\n        \n        summary_data.append({\n            'Kirletici Türü': name,\n            'Başlangıç Yılı': int(min_year),\n            'Bitiş Yılı': int(max_year),\n            'Toplam Yıl Sayısı': unique_years,\n            'Eksik Yıl Var mı?': 'Evet' if (max_year - min_year + 1) != unique_years else 'Hayır'\n        })\n    else:\n        summary_data.append({\n            'Kirletici Türü': name,\n            'Başlangıç Yılı': '-',\n            'Bitiş Yılı': '-',\n            'Toplam Yıl Sayısı': 0,\n            'Eksik Yıl Var mı?': '-'\n        })\n\n# Tabloyu oluştur ve göster\ndf_summary_ranges = pd.DataFrame(summary_data)\n\nprint(\"--- VERİ SETİ YIL ARALIKLARI ÖZETİ ---\")\nprint(df_summary_ranges.to_string(index=False))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:19.384412Z","iopub.execute_input":"2025-12-18T08:06:19.384735Z","iopub.status.idle":"2025-12-18T08:06:19.402381Z","shell.execute_reply.started":"2025-12-18T08:06:19.384703Z","shell.execute_reply":"2025-12-18T08:06:19.401175Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"target_set = set(target_states)\n\n# 2. Analiz Edilecek Yıl Aralığı (Referans Zaman)\nall_years = list(range(1990, 2020)) # 1990'dan 2019'a kadar\n\n# 3. Dataframe Sözlüğü (Döngüye sokmak için)\npollutants = {\n    'CO (Karbonmonoksit)': df_co_annual,\n    'NO2 (Azot Dioksit)': df_no2_annual,\n    'PM10 (Partikül Madde 10)': df_pm10_annual,\n    'PM2.5 (Partikül Madde 2.5)': df_pm25_annual,\n    'O3 (Ozon)': df_o3_annual,\n    'SO2 (Kükürt Dioksit)': df_so2_annual\n}\n\n# --- ANA ANALİZ DÖNGÜSÜ ---\nfor name, df in pollutants.items():\n    print(f\"\\n{'='*20} {name} ANALİZİ {'='*20}\")\n    \n    if df is None or df.empty:\n        print(\"❌ Veri Yok veya Çekilemedi.\\n\")\n        continue\n\n    # --- A. HİÇ OLMAYAN EYALETLERİ BULMA ---\n    present_states = set(df['STATE'].unique())\n    missing_states = target_set - present_states\n    \n    if missing_states:\n        print(f\"⚠️ HİÇ VERİSİ OLMAYAN EYALETLER ({len(missing_states)} Adet):\")\n        print(f\"   {', '.join(sorted(missing_states))}\")\n    else:\n        print(\"✅ Hedeflenen 25 eyaletin hepsinde istasyon var.\")\n\n    # --- B. EKSİK YILLARI BULMA (GAP ANALYSIS) ---\n    # Her eyalet için mevcut yılları 1 ve 0 olarak tabloya döküyoruz\n    pivot = pd.crosstab(df['STATE'], df['YEAR'])\n    \n    # Tabloyu 1990-2019 aralığına zorluyoruz (Ölçülmeyen yılları 0 yapar)\n    pivot = pivot.reindex(columns=all_years, fill_value=0)\n    \n    # 0 olan yılları say (Eksik Yıl Sayısı)\n    missing_counts = (pivot == 0).sum(axis=1)\n    \n    # Sadece eksiği olanları filtrele\n    gaps = missing_counts[missing_counts > 0]\n    \n    if not gaps.empty:\n        print(f\"\\n📉 YIL EKSİĞİ OLAN EYALETLER (Toplam 30 yıl üzerinden):\")\n        # En çok eksiği olan ilk 5 eyaleti göster\n        print(gaps.sort_values(ascending=False).head(5).to_string())\n        \n        # Kritik Uyarı: Eğer ortalama eksik yıl sayısı çok yüksekse\n        avg_missing = gaps.mean()\n        if avg_missing > 10:\n            print(f\"\\n   ℹ️ NOT: Bu kirletici için ortalama {int(avg_missing)} yıl eksik.\")\n            print(\"   (Muhtemelen ölçümler 1990'dan çok sonra başladı, örn: PM2.5)\")\n    else:\n        print(\"\\n✨ Harika! Mevcut eyaletlerin hepsinde 30 yılın verisi tam.\")\n\nprint(\"\\n\" + \"=\"*60)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:19.411094Z","iopub.execute_input":"2025-12-18T08:06:19.411525Z","iopub.status.idle":"2025-12-18T08:06:19.661601Z","shell.execute_reply.started":"2025-12-18T08:06:19.411484Z","shell.execute_reply":"2025-12-18T08:06:19.660435Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from functools import reduce\n\n# 1. Eldeki DataFrame'leri ve İsimlerini Listeye Alalım\npollutant_list = [\n    ('CO', df_co_annual),\n    ('NO2', df_no2_annual),\n    ('PM10', df_pm10_annual),\n    ('PM2.5', df_pm25_annual),\n    ('O3', df_o3_annual),\n    ('SO2', df_so2_annual)\n]\n\n# Boş olmayanları filtrele\nvalid_dfs = [(name, df) for name, df in pollutant_list if df is not None and not df.empty]\n\nif not valid_dfs:\n    print(\"Hiç geçerli veri çerçevesi yok!\")\nelse:\n    # 2. ORTAK ANAHTARLARI BULMA (Intersection)\n    # İlk DataFrame'in (State, Year) ikililerini alarak başlıyoruz\n    common_keys = set(zip(valid_dfs[0][1]['STATE'], valid_dfs[0][1]['YEAR']))\n\n    # Diğerleriyle kesişim kümesini alıyoruz\n    for name, df in valid_dfs[1:]:\n        current_keys = set(zip(df['STATE'], df['YEAR']))\n        common_keys = common_keys.intersection(current_keys)\n\n    print(f\"✅ TÜM TABLOLARDA ORTAK OLAN SATIR SAYISI: {len(common_keys)}\")\n    print(\"-\" * 60)\n\n    # 3. SİLİNEN VERİ RAPORU (Hangi Tablodan Ne Gitti?)\n    print(\"--- SİLİNEN VERİ ÖZETİ (Ortak Olmadığı İçin Çıkarılanlar) ---\")\n    \n    for name, df in valid_dfs:\n        # Bu tablodaki mevcut anahtarlar\n        current_keys = set(zip(df['STATE'], df['YEAR']))\n        # Ortak kümede olmayanlar (Silinecekler)\n        dropped_keys = current_keys - common_keys\n        \n        if dropped_keys:\n            print(f\"\\n📌 {name} Tablosundan Silinenler ({len(dropped_keys)} satır):\")\n            \n            # Silinenleri DataFrame yapıp Eyalet bazında özetleyelim\n            dropped_df = pd.DataFrame(list(dropped_keys), columns=['STATE', 'YEAR'])\n            \n            # Her eyalet için hangi yılların silindiğini grupla\n            summary = dropped_df.groupby('STATE')['YEAR'].apply(list)\n            \n            # İlk 5 eyaleti örnek olarak göster (ekranı doldurmamak için)\n            for state, years in summary.head(5).items():\n                years.sort()\n                # Yılları \"1990-1998\" gibi aralık olarak göstermek yerine min-max yazalım\n                min_y, max_y = years[0], years[-1]\n                if min_y == max_y:\n                    print(f\"   - {state}: {min_y}\")\n                else:\n                    print(f\"   - {state}: {min_y}-{max_y} ({len(years)} yıl)\")\n            \n            if len(summary) > 5:\n                print(f\"   ... ve {len(summary)-5} eyalet daha.\")\n        else:\n            print(f\"\\n✨ {name}: Hiç veri kaybı yok (Tam uyumlu).\")\n\n    # 4. BİRLEŞTİRME İŞLEMİ (INNER JOIN)\n    # Sadece verileri (df'leri) bir listede topla\n    dfs_to_merge = [item[1] for item in valid_dfs]\n    \n    # Inner Join ile birleştir\n    df_final_common = reduce(lambda left, right: pd.merge(left, right, on=['STATE', 'YEAR'], how='inner'), dfs_to_merge)\n    \n    # Düzenleme: Eyalet ve Yıla göre sırala\n    df_final_common = df_final_common.sort_values(by=['STATE', 'YEAR']).reset_index(drop=True)\n\n    print(\"\\n\" + \"=\"*60)\n    print(\"SONUÇ: BİRLEŞTİRİLMİŞ ORTAK VERİ SETİ\")\n    print(\"=\"*60)\n    print(df_final_common.head(10))\n    print(f\"\\nToplam Satır Sayısı: {len(df_final_common)}\")\n    print(f\"Toplam Sütunlar: {df_final_common.columns.tolist()}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:19.663862Z","iopub.execute_input":"2025-12-18T08:06:19.664437Z","iopub.status.idle":"2025-12-18T08:06:19.764786Z","shell.execute_reply.started":"2025-12-18T08:06:19.664386Z","shell.execute_reply":"2025-12-18T08:06:19.763660Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if 'df_final_common' in locals() and not df_final_common.empty:\n    # 1. Temel İstatistikler\n    min_year = int(df_final_common['YEAR'].min())\n    max_year = int(df_final_common['YEAR'].max())\n    total_years = df_final_common['YEAR'].nunique()\n    \n    # 2. Teorik olarak olması gereken yıl sayısı (Aralık farkı)\n    expected_duration = max_year - min_year + 1\n    \n    print(\"-\" * 45)\n    print(\"📊 SONUÇ TABLOSU (df_final_common) TARİH ARALIĞI\")\n    print(\"-\" * 45)\n    print(f\"📅 Başlangıç Yılı : {min_year}\")\n    print(f\"📅 Bitiş Yılı     : {max_year}\")\n    print(f\"⏱️ Kapsanan Süre  : {expected_duration} Yıl\")\n    print(f\"✅ Dolu Yıl Sayısı: {total_years}\")\n    \n    # 3. Kesinti Kontrolü (Gap Check)\n    if total_years == expected_duration:\n        print(\"   Seçilen aralıktaki tüm yıllar mevcut.\")\n    else:\n        print(f\"\\nArada {expected_duration - total_years} yıl eksik!\")\n        # Eksik yılları bulup gösterelim\n        all_years = set(range(min_year, max_year + 1))\n        existing_years = set(df_final_common['YEAR'].unique())\n        missing = sorted(list(all_years - existing_years))\n        print(f\"   Eksik Yıllar: {missing}\")\n\nelse:\n    print(\"❌ Tablo boş veya oluşturulmamış.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:19.766379Z","iopub.execute_input":"2025-12-18T08:06:19.766705Z","iopub.status.idle":"2025-12-18T08:06:19.778497Z","shell.execute_reply.started":"2025-12-18T08:06:19.766673Z","shell.execute_reply":"2025-12-18T08:06:19.777278Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_final_common.head(10)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:19.780423Z","iopub.execute_input":"2025-12-18T08:06:19.780880Z","iopub.status.idle":"2025-12-18T08:06:19.815763Z","shell.execute_reply.started":"2025-12-18T08:06:19.780832Z","shell.execute_reply":"2025-12-18T08:06:19.814673Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_final_common.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:19.817173Z","iopub.execute_input":"2025-12-18T08:06:19.817573Z","iopub.status.idle":"2025-12-18T08:06:19.833396Z","shell.execute_reply.started":"2025-12-18T08:06:19.817523Z","shell.execute_reply":"2025-12-18T08:06:19.832242Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_path = \"/kaggle/input/air-toxicity-and-chronic-respiratory-diseases-us/train.csv\"\ndf = pd.read_csv(train_path)\ndf.head(8)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:19.834609Z","iopub.execute_input":"2025-12-18T08:06:19.834978Z","iopub.status.idle":"2025-12-18T08:06:19.871498Z","shell.execute_reply.started":"2025-12-18T08:06:19.834940Z","shell.execute_reply":"2025-12-18T08:06:19.870148Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df = df[~df['Year'].between(1990, 1997)]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:19.873036Z","iopub.execute_input":"2025-12-18T08:06:19.873437Z","iopub.status.idle":"2025-12-18T08:06:19.881043Z","shell.execute_reply.started":"2025-12-18T08:06:19.873403Z","shell.execute_reply":"2025-12-18T08:06:19.879724Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df['Year'].min(), df['Year'].max()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:19.882741Z","iopub.execute_input":"2025-12-18T08:06:19.883232Z","iopub.status.idle":"2025-12-18T08:06:19.899099Z","shell.execute_reply.started":"2025-12-18T08:06:19.883197Z","shell.execute_reply":"2025-12-18T08:06:19.897720Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df = df.rename(columns={\"State.Name\": \"STATE\", \"Year\": \"YEAR\"})\n\ndf[\"YEAR\"] = df[\"YEAR\"].astype(int)\ndf_final_common[\"YEAR\"] = df_final_common[\"YEAR\"].astype(int)\n\ndf[\"STATE\"] = df[\"STATE\"].astype(str).str.strip()\ndf_final_common[\"STATE\"] = df_final_common[\"STATE\"].astype(str).str.strip()\n\ndf_merged = df.merge(\n    df_final_common,\n    on=[\"STATE\", \"YEAR\"],\n    how=\"left\",\n    indicator=True   \n)\n\ndf_merged.head(10)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:19.900527Z","iopub.execute_input":"2025-12-18T08:06:19.901117Z","iopub.status.idle":"2025-12-18T08:06:19.956172Z","shell.execute_reply.started":"2025-12-18T08:06:19.901069Z","shell.execute_reply":"2025-12-18T08:06:19.955176Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"missing_cases = df_merged[df_merged[\"_merge\"] == \"left_only\"]\nmissing_cases.count()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:19.957369Z","iopub.execute_input":"2025-12-18T08:06:19.957646Z","iopub.status.idle":"2025-12-18T08:06:19.973426Z","shell.execute_reply.started":"2025-12-18T08:06:19.957620Z","shell.execute_reply":"2025-12-18T08:06:19.972365Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_merged.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:19.975212Z","iopub.execute_input":"2025-12-18T08:06:19.975540Z","iopub.status.idle":"2025-12-18T08:06:19.994525Z","shell.execute_reply.started":"2025-12-18T08:06:19.975509Z","shell.execute_reply":"2025-12-18T08:06:19.993100Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_merged = df_merged[df_merged[\"_merge\"] != \"left_only\"]\ndf_merged.head(10)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:19.996443Z","iopub.execute_input":"2025-12-18T08:06:19.997013Z","iopub.status.idle":"2025-12-18T08:06:20.027738Z","shell.execute_reply.started":"2025-12-18T08:06:19.996949Z","shell.execute_reply":"2025-12-18T08:06:20.026406Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_merged.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:20.029322Z","iopub.execute_input":"2025-12-18T08:06:20.029728Z","iopub.status.idle":"2025-12-18T08:06:20.043644Z","shell.execute_reply.started":"2025-12-18T08:06:20.029692Z","shell.execute_reply":"2025-12-18T08:06:20.042213Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"dup_all = df_merged[df_merged.duplicated(keep=False)]\nprint(\"Duplıcated rows:\", dup_all.shape[0])\ndf_merged.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:20.045141Z","iopub.execute_input":"2025-12-18T08:06:20.045451Z","iopub.status.idle":"2025-12-18T08:06:20.083314Z","shell.execute_reply.started":"2025-12-18T08:06:20.045423Z","shell.execute_reply":"2025-12-18T08:06:20.082198Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_merged.head(8)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:20.084673Z","iopub.execute_input":"2025-12-18T08:06:20.084970Z","iopub.status.idle":"2025-12-18T08:06:20.107099Z","shell.execute_reply.started":"2025-12-18T08:06:20.084942Z","shell.execute_reply":"2025-12-18T08:06:20.106124Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"cols_to_drop = [c for c in [\"ID\", \"id\", \"STATE\",\"_merge\"] if c in df_merged.columns]\n\ndf_merged = df_merged.drop(columns=cols_to_drop)\n\ndf_merged.head()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:20.108564Z","iopub.execute_input":"2025-12-18T08:06:20.108842Z","iopub.status.idle":"2025-12-18T08:06:20.138422Z","shell.execute_reply.started":"2025-12-18T08:06:20.108815Z","shell.execute_reply":"2025-12-18T08:06:20.137158Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import re\nimport numpy as np\n\ndef age_to_midpoint(x):\n    if pd.isna(x):\n        return np.nan\n    x = str(x).strip()\n    m = re.match(r\"(\\d+)\\s*-\\s*(\\d+)\", x)\n    if m:\n        a, b = int(m.group(1)), int(m.group(2))\n        return (a + b) / 2\n    return np.nan\n\nif \"Age\" in df_merged.columns:\n    df_merged[\"Age_mid\"] = df_merged[\"Age\"].apply(age_to_midpoint)\n    df_merged = df_merged.drop(columns=[\"Age\"])\n\ndf_merged[[\"Age_mid\"]].head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:20.140300Z","iopub.execute_input":"2025-12-18T08:06:20.140730Z","iopub.status.idle":"2025-12-18T08:06:20.173394Z","shell.execute_reply.started":"2025-12-18T08:06:20.140685Z","shell.execute_reply":"2025-12-18T08:06:20.172299Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_merged.head(5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:20.175090Z","iopub.execute_input":"2025-12-18T08:06:20.175450Z","iopub.status.idle":"2025-12-18T08:06:20.196342Z","shell.execute_reply.started":"2025-12-18T08:06:20.175418Z","shell.execute_reply":"2025-12-18T08:06:20.195425Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"Missing values per column:\")\nprint(df_merged.isna().sum().sort_values(ascending=False).head(20))\n\nprint(\"\\nTotal duplicated rows:\", df_merged.duplicated().sum())\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:20.197584Z","iopub.execute_input":"2025-12-18T08:06:20.197918Z","iopub.status.idle":"2025-12-18T08:06:20.219263Z","shell.execute_reply.started":"2025-12-18T08:06:20.197869Z","shell.execute_reply":"2025-12-18T08:06:20.218082Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"target = \"Incidence\"  \n\nx = df_merged.drop(columns=[target])\ny = df_merged[target]\n\nprint(\"X shape:\", x.shape)\nprint(\"y shape:\", y.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:06:20.221262Z","iopub.execute_input":"2025-12-18T08:06:20.221588Z","iopub.status.idle":"2025-12-18T08:06:20.233478Z","shell.execute_reply.started":"2025-12-18T08:06:20.221555Z","shell.execute_reply":"2025-12-18T08:06:20.232341Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\ntmp_num = df_merged.select_dtypes(include=[np.number]).copy()\n\ncorr_matrix = tmp_num.corr()\n\nplt.figure(figsize=(12, 10))\nplt.imshow(corr_matrix, aspect=\"auto\")\n\nplt.title(\"Correlation Matrix (Heatmap)\")\nplt.colorbar()\n\nticks = np.arange(len(corr_matrix.columns))\nplt.xticks(ticks, corr_matrix.columns, rotation=90)\nplt.yticks(ticks, corr_matrix.columns)\n\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:08:42.582831Z","iopub.execute_input":"2025-12-18T08:08:42.583265Z","iopub.status.idle":"2025-12-18T08:08:42.974686Z","shell.execute_reply.started":"2025-12-18T08:08:42.583232Z","shell.execute_reply":"2025-12-18T08:08:42.973712Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nX_all = x\n\nX_train, X_test, y_train, y_test = train_test_split(\n    X_all, y, test_size=0.2, random_state=42\n)\n\nX_train.shape, X_test.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:09:03.742576Z","iopub.execute_input":"2025-12-18T08:09:03.743016Z","iopub.status.idle":"2025-12-18T08:09:05.004560Z","shell.execute_reply.started":"2025-12-18T08:09:03.742964Z","shell.execute_reply":"2025-12-18T08:09:05.003148Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestRegressor\nfrom sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score\nimport numpy as np\n\nrf = RandomForestRegressor(n_estimators=300, random_state=42, n_jobs=-1)\nrf.fit(X_train, y_train)\n\npred = rf.predict(X_test)\n\nmae = mean_absolute_error(y_test, pred)\nrmse = np.sqrt(mean_squared_error(y_test, pred))\nr2 = r2_score(y_test, pred)\n\nprint(f\"MAE  : {mae:.4f}\")\nprint(f\"RMSE : {rmse:.4f}\")\nprint(f\"R2   : {r2:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:09:09.728803Z","iopub.execute_input":"2025-12-18T08:09:09.729158Z","iopub.status.idle":"2025-12-18T08:09:11.831373Z","shell.execute_reply.started":"2025-12-18T08:09:09.729127Z","shell.execute_reply":"2025-12-18T08:09:11.830074Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from sklearn.linear_model import LinearRegression\nfrom sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score\nimport numpy as np\n\nlr = LinearRegression()\nlr.fit(X_train, y_train)\n\ny_pred_lr = lr.predict(X_test)\n\nmae_lr = mean_absolute_error(y_test, y_pred_lr)\nrmse_lr = np.sqrt(mean_squared_error(y_test, y_pred_lr))\nr2_lr = r2_score(y_test, y_pred_lr)\n\nmae_lr, rmse_lr, r2_lr","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:09:26.054483Z","iopub.execute_input":"2025-12-18T08:09:26.054860Z","iopub.status.idle":"2025-12-18T08:09:26.100324Z","shell.execute_reply.started":"2025-12-18T08:09:26.054829Z","shell.execute_reply":"2025-12-18T08:09:26.099190Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from xgboost import XGBRegressor\nfrom sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score\n\nxgb = XGBRegressor(\n    n_estimators=500,\n    learning_rate=0.05,\n    max_depth=6,\n    subsample=0.8,\n    colsample_bytree=0.8,\n    random_state=42,\n    n_jobs=-1\n)\n\nxgb.fit(X_train, y_train)\n\ny_pred_xgb = xgb.predict(X_test)\n\nmae_xgb  = mean_absolute_error(y_test, y_pred_xgb)\nrmse_xgb = np.sqrt(mean_squared_error(y_test, y_pred_xgb))\nr2_xgb   = r2_score(y_test, y_pred_xgb)\n\nmae_xgb, rmse_xgb, r2_xgb\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:09:30.845607Z","iopub.execute_input":"2025-12-18T08:09:30.846061Z","iopub.status.idle":"2025-12-18T08:09:32.182963Z","shell.execute_reply.started":"2025-12-18T08:09:30.846019Z","shell.execute_reply":"2025-12-18T08:09:32.181977Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"feature_importance = pd.DataFrame({\n    \"feature\": X_train.columns,\n    \"importance\": xgb.feature_importances_\n}).sort_values(by=\"importance\", ascending=False)\n\nfeature_importance\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:09:37.592570Z","iopub.execute_input":"2025-12-18T08:09:37.592976Z","iopub.status.idle":"2025-12-18T08:09:38.028403Z","shell.execute_reply.started":"2025-12-18T08:09:37.592927Z","shell.execute_reply":"2025-12-18T08:09:38.026799Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Model Results & Performance Analysis\n\nAfter completing the data preprocessing steps (including removing ID columns and excluding geographic categorical features such as State and its encoded derivatives), we trained and evaluated multiple regression models to predict respiratory disease incidence based on air pollution indicators and demographic variables.\n\n🔹 Models Evaluated\n\nLinear Regression (Baseline)\n\nRandom Forest Regressor\n\nXGBoost Regressor\n\nEach model was evaluated using:\n\nMAE (Mean Absolute Error)\n\nRMSE (Root Mean Squared Error)\n\nR² Score (Coefficient of Determination)\nInterpretation:\n\nLinear Regression underperformed, indicating that the relationship between air pollutants and disease incidence is highly non-linear.\n\nRandom Forest significantly improved performance by capturing non-linear interactions.\n\nXGBoost achieved the best results, with very low prediction error and an R² close to 1, showing excellent explanatory power.\n\n🔹 Feature Importance (XGBoost)\n\nFeature importance analysis from the XGBoost model revealed the following ranking:\n\nYear\n\nCarbon Monoxide (CO)\n\nAge Group (Mid)\n\nOzone (O₃)\n\nPM2.5\n\nSulfur Dioxide (SO₂)\n\nNitrogen Dioxide (NO₂)\n\nPM10\n\nKey Insights:\n\nThe Year feature dominates, suggesting strong temporal trends in respiratory disease incidence (e.g., policy changes, healthcare improvements, or long-term exposure effects).\n\nCO, PM2.5, and O₃ emerge as the most influential pollutants, aligning with established epidemiological findings.\n\nRemoving geographic identifiers (State) did not degrade model performance, indicating that pollution and time variables sufficiently capture spatial effects.\n\n🔹 Conclusion\n\nTree-based ensemble models (especially XGBoost) are highly effective for modeling environmental health data.\n\nFeature importance analysis provides interpretable insights into which pollutants most strongly influence respiratory disease incidence.\n\nThe final model demonstrates strong predictive accuracy and robustness, making it suitable for downstream analysis or policy-related simulations.","metadata":{}},{"cell_type":"code","source":"import shap\n\n#  SHAP explainer\nexplainer = shap.Explainer(xgb, X_train)\n\nshap_values = explainer(X_test)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:09:45.101574Z","iopub.execute_input":"2025-12-18T08:09:45.102009Z","iopub.status.idle":"2025-12-18T08:10:15.766698Z","shell.execute_reply.started":"2025-12-18T08:09:45.101966Z","shell.execute_reply":"2025-12-18T08:10:15.765384Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"shap.summary_plot(shap_values, X_test, plot_type=\"bar\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:10:19.090308Z","iopub.execute_input":"2025-12-18T08:10:19.090656Z","iopub.status.idle":"2025-12-18T08:10:19.260487Z","shell.execute_reply.started":"2025-12-18T08:10:19.090628Z","shell.execute_reply":"2025-12-18T08:10:19.259338Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"shap.dependence_plot(\"PM25\", shap_values.values, X_test)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:10:27.777054Z","iopub.execute_input":"2025-12-18T08:10:27.777445Z","iopub.status.idle":"2025-12-18T08:10:28.099690Z","shell.execute_reply.started":"2025-12-18T08:10:27.777411Z","shell.execute_reply":"2025-12-18T08:10:28.098171Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"shap.dependence_plot(\"CO\", shap_values.values, X_test)\nshap.dependence_plot(\"O3\", shap_values.values, X_test)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T08:10:35.219570Z","iopub.execute_input":"2025-12-18T08:10:35.219949Z","iopub.status.idle":"2025-12-18T08:10:35.733175Z","shell.execute_reply.started":"2025-12-18T08:10:35.219880Z","shell.execute_reply":"2025-12-18T08:10:35.732118Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🔍 Model Explainability using SHAP\n\nTo enhance model interpretability, SHAP (SHapley Additive exPlanations) was applied to the XGBoost regression model. SHAP provides both global and local explanations by quantifying each feature’s contribution to the model’s predictions.\n\n### 🔹 SHAP Summary Plot\nThe SHAP summary plot reveals that **Year**, **CO**, and **PM2.5** are the most influential features affecting respiratory disease incidence. High pollutant concentrations generally push predictions toward higher incidence values, indicating a positive association between air pollution and respiratory health risks.\n\n### 🔹 Feature Importance (SHAP Bar Plot)\nThe bar plot confirms that temporal trends (Year) and key pollutants such as CO and PM2.5 dominate the prediction process, aligning with epidemiological findings reported in environmental health literature.\n\n### 🔹 Feature Interaction & Non-Linearity\nSHAP dependence plots illustrate non-linear relationships between pollutants and disease incidence, particularly for PM2.5 and O₃, demonstrating that tree-based models are well-suited for capturing complex exposure–response patterns.\n\n### 🔹 Conclusion\nSHAP analysis validates the robustness of the XGBoost model and provides transparent, interpretable insights into how air pollution variables contribute to respiratory disease incidence.\n","metadata":{}}]}