{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":10445,"databundleVersionId":200747,"sourceType":"competition"}],"dockerImageVersionId":30732,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-06-20T06:44:07.972137Z","iopub.execute_input":"2024-06-20T06:44:07.972544Z","iopub.status.idle":"2024-06-20T06:44:08.595406Z","shell.execute_reply.started":"2024-06-20T06:44:07.972511Z","shell.execute_reply":"2024-06-20T06:44:08.594210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"特徴量選択（変数選択）は、機械学習系の予測モデルなどを構築するとき、最初に行う重要なステップの1つです。\n\n予測に寄与しない特徴量（説明変数）を取り除くことで、よりシンプルな予測モデルを構築を目指します。","metadata":{}},{"cell_type":"markdown","source":"## 特徴量選択の３つの手法\n\n特徴量選択には幾つかやり方がありますが、基本となるのは次の3つ考え方です。\n\n- フィルター法（Filter Method）\n- ラッパー法（Wrapper Method）\n- 埋め込み法（Embedded Method）","metadata":{}},{"cell_type":"markdown","source":"## フィルター法（Filter Method）\nデータの特性に基づいて特徴量を選択する手法で、最も一般的な方法は、目的変数との相関係数を用いるものとなります。  \n高い相関を持つ特徴量を選択し、低い相関のものは除外します。さらに、選択された特徴量間の相関関係も考慮し、強い相関を持つ特徴量がある場合は、そのうちの一つだけを選択します。\n\n\n今回は相関係数で説明しましたが、相関係数である必要はありません。","metadata":{}},{"cell_type":"markdown","source":"## ラッパー法（Wrapper Method）\nモデルの構築を行いながら特徴量を選択する手法です。  \nモデルの性能を指標として、最適な特徴量の組み合わせを見つけ出します。  \n代表的な手法には変数減少法（Backward Elimination）や変数増加法（Forward Selection）があります。これらはモデルの性能を基にして特徴量を追加または除去していきます。","metadata":{}},{"cell_type":"markdown","source":"統計的検定でよく利用されるp値をもとに、変数減少法（Backward Elimination）を簡単に説明します。\n\n1. 先ず、全ての特徴量（説明変数）を使いモデル構築します\n2. 最もp値の大きな係数を持つ特徴量（説明変数）を探します\n3. p値が0.05よりも大きい場合、その特徴量（説明変数）を除去します\n4. 残りの特徴量（説明変数）を使いモデル構築します\n5. 最もp値の大きな係数を持つ特徴量（説明変数）を探します\n6. p値が0.05よりも大きい場合、その特徴量（説明変数）を除去します\n7. 以後、最もp値の大きな係数のp値が、0.05よりも小さくなるまで続けます\n\n今回はp値で説明しましたが、p値である必要はありません。\n\n最近では、RFE（Recursive Feature Elimination）法がよく利用されます。RFE法は、予測モデルのパフォーマンス（予測精度など）を見ながら特徴量選択（変数選択）します。","metadata":{}},{"cell_type":"markdown","source":"### RFE法の簡単な説明\n\n特徴に重みを割り当てる外部推定機（ランダムフォレストや勾配ブースティング、線形モデルなど）が与えられた時、特徴集合を走査、どの特徴が重要であるかを発見し、指定した特徴数になるまで特徴の消去を行う手法です。\n\n例えば２０個の特徴をもつデータがあったとする。そのままの形でモデルにデータを渡して学習を行わせると、２０個に含まれる重要度の低い特徴までも考慮して学習することになってしまいます。\n（天気を予測させるためのデータの中に「ここ一週間の株価」という特徴があったとして、これは明らかに結果に対して重要でない）\n\nその時RFEを用いると、指定した数、例えば１０とすると、データ中の特徴量を重要度でランキング化し、その上位１０の特徴量を抽出して学習を行わせることができるようになります。\n","metadata":{}},{"cell_type":"markdown","source":"## 埋め込み法（Embedded Method）\nモデル構築プロセス自体に特徴量選択のメカニズムが組み込まれている手法です。  \nLasso回帰のように、特徴量の係数を0にすることで、自動的に特徴量選択を行います。この方法は、モデルの複雑さを調整するパラメーターによって、選択される特徴量が変わります。","metadata":{}},{"cell_type":"markdown","source":"ただ、Lasso回帰のハイパーパラメータであるλ（Complexity Parameter）によって、選択される特徴量（説明変数）が変わります。scikit-learnではαという記号で表現されています。\n\n- λ（Complexity Parameter）が0の場合、通常の重回帰と同じになります\n- λ（Complexity Parameter）の値が大きいほど、選択される特徴量（説明変数）の数は減少します","metadata":{}},{"cell_type":"code","source":"# 必要なモジュールのimport\n\nimport warnings\nimport time\nimport sys\nimport datetime\nimport pickle\n\nimport numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport lightgbm as lgb\nfrom sklearn.feature_selection import RFE, RFECV\nfrom sklearn.linear_model import Lasso, LassoCV\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import mean_squared_error\n\nfrom sklearn.linear_model import LinearRegression, Lasso\nfrom sklearn.feature_selection import RFE\nfrom sklearn.preprocessing import StandardScaler\nimport statsmodels.api as sm\n\nwarnings.simplefilter(action='ignore', category=FutureWarning)\npd.set_option('display.max_columns', 500)","metadata":{"execution":{"iopub.status.busy":"2024-06-20T06:44:11.845450Z","iopub.execute_input":"2024-06-20T06:44:11.846042Z","iopub.status.idle":"2024-06-20T06:44:16.376158Z","shell.execute_reply.started":"2024-06-20T06:44:11.846007Z","shell.execute_reply":"2024-06-20T06:44:16.375052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# データの読み込み\n\n# ファイルパスを変数に保存\nfile_paths = {\n    \"sample_submission\": \"/kaggle/input/elo-merchant-category-recommendation/sample_submission.csv\",\n    \"new_merchant_transactions\": \"/kaggle/input/elo-merchant-category-recommendation/new_merchant_transactions.csv\",\n    \"merchants\": \"/kaggle/input/elo-merchant-category-recommendation/merchants.csv\",\n    \"historical_transactions\": \"/kaggle/input/elo-merchant-category-recommendation/historical_transactions.csv\",\n    \"train\": \"/kaggle/input/elo-merchant-category-recommendation/train.csv\",\n    \"test\": \"/kaggle/input/elo-merchant-category-recommendation/test.csv\"\n}\n\nnew_transactions = pd.read_csv(file_paths['new_merchant_transactions'],\n                               parse_dates=['purchase_date'])\n\nhistorical_transactions = pd.read_csv(file_paths['historical_transactions'],\n                                      parse_dates=['purchase_date'])\n\ndef binarize(df):\n    for col in ['authorized_flag', 'category_1']:\n        df[col] = df[col].map({'Y':1, 'N':0})\n    return df\n\nhistorical_transactions = binarize(historical_transactions)\nnew_transactions = binarize(new_transactions)","metadata":{"execution":{"iopub.status.busy":"2024-06-20T06:44:16.378067Z","iopub.execute_input":"2024-06-20T06:44:16.378637Z","iopub.status.idle":"2024-06-20T06:46:43.826084Z","shell.execute_reply.started":"2024-06-20T06:44:16.378603Z","shell.execute_reply":"2024-06-20T06:46:43.824262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# メモリの消費を抑えるコード\n\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df","metadata":{"execution":{"iopub.status.busy":"2024-06-20T06:46:43.828176Z","iopub.execute_input":"2024-06-20T06:46:43.828755Z","iopub.status.idle":"2024-06-20T06:46:43.850551Z","shell.execute_reply.started":"2024-06-20T06:46:43.828716Z","shell.execute_reply":"2024-06-20T06:46:43.848482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_data(input_file):\n    df = pd.read_csv(input_file)\n    df['first_active_month'] = pd.to_datetime(df['first_active_month'])\n    df['elapsed_time'] = (pd.Timestamp('2018-02-01') - df['first_active_month']).dt.days\n    return df\n\ntrain = read_data(file_paths['train'])\ntest = read_data(file_paths['test'])\n\ntarget = train['target']\ndel train['target']","metadata":{"execution":{"iopub.status.busy":"2024-06-20T06:46:43.854166Z","iopub.execute_input":"2024-06-20T06:46:43.854907Z","iopub.status.idle":"2024-06-20T06:46:44.473251Z","shell.execute_reply.started":"2024-06-20T06:46:43.854848Z","shell.execute_reply":"2024-06-20T06:46:44.471871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 特徴量作成","metadata":{}},{"cell_type":"code","source":"historical_transactions['month_diff'] = ((datetime.datetime.today() - historical_transactions['purchase_date']).dt.days)//30\nhistorical_transactions['month_diff'] += historical_transactions['month_lag']\n\nnew_transactions['month_diff'] = ((datetime.datetime.today() - new_transactions['purchase_date']).dt.days)//30\nnew_transactions['month_diff'] += new_transactions['month_lag']","metadata":{"execution":{"iopub.status.busy":"2024-06-20T06:46:44.474449Z","iopub.execute_input":"2024-06-20T06:46:44.474900Z","iopub.status.idle":"2024-06-20T06:46:46.275397Z","shell.execute_reply.started":"2024-06-20T06:46:44.474868Z","shell.execute_reply":"2024-06-20T06:46:46.274195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# データの確認\nhistorical_transactions[:5]","metadata":{"execution":{"iopub.status.busy":"2024-06-20T06:46:46.276922Z","iopub.execute_input":"2024-06-20T06:46:46.277330Z","iopub.status.idle":"2024-06-20T06:46:46.308187Z","shell.execute_reply.started":"2024-06-20T06:46:46.277299Z","shell.execute_reply":"2024-06-20T06:46:46.306912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"historical_transactions = pd.get_dummies(historical_transactions, columns=['category_2', 'category_3'])\nnew_transactions = pd.get_dummies(new_transactions, columns=['category_2', 'category_3'])\n\nhistorical_transactions = reduce_mem_usage(historical_transactions)\nnew_transactions = reduce_mem_usage(new_transactions)\n\nagg_fun = {'authorized_flag': ['mean']}\nauth_mean = historical_transactions.groupby(['card_id']).agg(agg_fun)\nauth_mean.columns = ['_'.join(col).strip() for col in auth_mean.columns.values]\nauth_mean.reset_index(inplace=True)\n\nauthorized_transactions = historical_transactions[historical_transactions['authorized_flag'] == 1]\nhistorical_transactions = historical_transactions[historical_transactions['authorized_flag'] == 0]","metadata":{"execution":{"iopub.status.busy":"2024-06-20T06:46:46.309673Z","iopub.execute_input":"2024-06-20T06:46:46.310083Z","iopub.status.idle":"2024-06-20T06:47:12.443789Z","shell.execute_reply.started":"2024-06-20T06:46:46.310052Z","shell.execute_reply":"2024-06-20T06:47:12.442059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"historical_transactions['purchase_month'] = historical_transactions['purchase_date'].dt.month\nauthorized_transactions['purchase_month'] = authorized_transactions['purchase_date'].dt.month\nnew_transactions['purchase_month'] = new_transactions['purchase_date'].dt.month","metadata":{"execution":{"iopub.status.busy":"2024-06-20T06:47:12.446177Z","iopub.execute_input":"2024-06-20T06:47:12.447589Z","iopub.status.idle":"2024-06-20T06:47:13.953235Z","shell.execute_reply.started":"2024-06-20T06:47:12.447538Z","shell.execute_reply":"2024-06-20T06:47:13.952151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def aggregate_transactions(history):\n    \n    history.loc[:, 'purchase_date'] = pd.DatetimeIndex(history['purchase_date']).\\\n                                      astype(np.int64) * 1e-9\n    \n    agg_func = {\n                'category_1': ['sum', 'mean'],\n                'category_2_1.0': ['mean'],\n                'category_2_2.0': ['mean'],\n                'category_2_3.0': ['mean'],\n                'category_2_4.0': ['mean'],\n                'category_2_5.0': ['mean'],\n                'category_3_A': ['mean'],\n                'category_3_B': ['mean'],\n                'category_3_C': ['mean'],\n                'merchant_id': ['nunique'],\n                'merchant_category_id': ['nunique'],\n                'state_id': ['nunique'],\n                'city_id': ['nunique'],\n                'subsector_id': ['nunique'],\n                'purchase_amount': ['sum', 'mean', 'max', 'min', 'std'],\n                'installments': ['sum', 'mean', 'max', 'min', 'std'],\n                'purchase_month': ['mean', 'max', 'min', 'std'],\n                'purchase_date': [np.ptp, 'min', 'max'],\n                'month_lag': ['mean', 'max', 'min', 'std'],\n                'month_diff': ['mean']\n    }\n    \n    agg_history = history.groupby(['card_id']).agg(agg_func)\n    agg_history.columns = ['_'.join(col).strip() for col in agg_history.columns.values]\n    agg_history.reset_index(inplace=True)\n    \n    df = (history.groupby('card_id')\n          .size()\n          .reset_index(name='transactions_count'))\n    \n    agg_history = pd.merge(df, agg_history, on='card_id', how='left')\n    \n    return agg_history","metadata":{"execution":{"iopub.status.busy":"2024-06-20T06:47:13.954676Z","iopub.execute_input":"2024-06-20T06:47:13.955151Z","iopub.status.idle":"2024-06-20T06:47:13.967868Z","shell.execute_reply.started":"2024-06-20T06:47:13.955113Z","shell.execute_reply":"2024-06-20T06:47:13.966666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = aggregate_transactions(historical_transactions)\nhistory.columns = ['hist_' + c if c != 'card_id' else c for c in history.columns]\nhistory[:5]","metadata":{"execution":{"iopub.status.busy":"2024-06-20T06:47:13.972885Z","iopub.execute_input":"2024-06-20T06:47:13.974079Z","iopub.status.idle":"2024-06-20T06:48:08.111817Z","shell.execute_reply.started":"2024-06-20T06:47:13.974041Z","shell.execute_reply":"2024-06-20T06:48:08.110271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"authorized = aggregate_transactions(authorized_transactions)\nauthorized.columns = ['auth_' + c if c != 'card_id' else c for c in authorized.columns]\nauthorized[:5]","metadata":{"execution":{"iopub.status.busy":"2024-06-20T06:48:08.113601Z","iopub.execute_input":"2024-06-20T06:48:08.114233Z","iopub.status.idle":"2024-06-20T06:49:55.062653Z","shell.execute_reply.started":"2024-06-20T06:48:08.114187Z","shell.execute_reply":"2024-06-20T06:49:55.061535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new = aggregate_transactions(new_transactions)\nnew.columns = ['new_' + c if c != 'card_id' else c for c in new.columns]\nnew[:5]","metadata":{"execution":{"iopub.status.busy":"2024-06-20T06:49:55.064374Z","iopub.execute_input":"2024-06-20T06:49:55.064856Z","iopub.status.idle":"2024-06-20T06:50:54.070025Z","shell.execute_reply.started":"2024-06-20T06:49:55.064824Z","shell.execute_reply":"2024-06-20T06:50:54.068700Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def aggregate_per_month(history):\n    grouped = history.groupby(['card_id', 'month_lag'])\n\n    agg_func = {\n            'purchase_amount': ['count', 'sum', 'mean', 'min', 'max', 'std'],\n            'installments': ['count', 'sum', 'mean', 'min', 'max', 'std'],\n            }\n\n    intermediate_group = grouped.agg(agg_func)\n    intermediate_group.columns = ['_'.join(col).strip() for col in intermediate_group.columns.values]\n    intermediate_group.reset_index(inplace=True)\n\n    final_group = intermediate_group.groupby('card_id').agg(['mean', 'std'])\n    final_group.columns = ['_'.join(col).strip() for col in final_group.columns.values]\n    final_group.reset_index(inplace=True)\n    \n    return final_group\n#___________________________________________________________\nfinal_group =  aggregate_per_month(authorized_transactions) \nfinal_group[:10]","metadata":{"execution":{"iopub.status.busy":"2024-06-20T06:50:54.071767Z","iopub.execute_input":"2024-06-20T06:50:54.072183Z","iopub.status.idle":"2024-06-20T06:51:08.015276Z","shell.execute_reply.started":"2024-06-20T06:50:54.072145Z","shell.execute_reply":"2024-06-20T06:51:08.013938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def successive_aggregates(df, field1, field2):\n    t = df.groupby(['card_id', field1])[field2].mean()\n    u = pd.DataFrame(t).reset_index().groupby('card_id')[field2].agg(['mean', 'min', 'max', 'std'])\n    u.columns = [field1 + '_' + field2 + '_' + col for col in u.columns.values]\n    u.reset_index(inplace=True)\n    return u","metadata":{"execution":{"iopub.status.busy":"2024-06-20T06:51:08.016898Z","iopub.execute_input":"2024-06-20T06:51:08.017303Z","iopub.status.idle":"2024-06-20T06:51:08.025172Z","shell.execute_reply.started":"2024-06-20T06:51:08.017270Z","shell.execute_reply":"2024-06-20T06:51:08.023786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"additional_fields = successive_aggregates(new_transactions, 'category_1', 'purchase_amount')\nadditional_fields = additional_fields.merge(successive_aggregates(new_transactions, 'installments', 'purchase_amount'),\n                                            on = 'card_id', how='left')\nadditional_fields = additional_fields.merge(successive_aggregates(new_transactions, 'city_id', 'purchase_amount'),\n                                            on = 'card_id', how='left')\nadditional_fields = additional_fields.merge(successive_aggregates(new_transactions, 'category_1', 'installments'),\n                                            on = 'card_id', how='left')","metadata":{"execution":{"iopub.status.busy":"2024-06-20T06:51:08.027587Z","iopub.execute_input":"2024-06-20T06:51:08.028013Z","iopub.status.idle":"2024-06-20T06:51:16.183792Z","shell.execute_reply.started":"2024-06-20T06:51:08.027970Z","shell.execute_reply":"2024-06-20T06:51:16.182632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# データの結合\ntrain = pd.merge(train, history, on='card_id', how='left')\ntest = pd.merge(test, history, on='card_id', how='left')\n\ntrain = pd.merge(train, authorized, on='card_id', how='left')\ntest = pd.merge(test, authorized, on='card_id', how='left')\n\ntrain = pd.merge(train, new, on='card_id', how='left')\ntest = pd.merge(test, new, on='card_id', how='left')\n\ntrain = pd.merge(train, final_group, on='card_id', how='left')\ntest = pd.merge(test, final_group, on='card_id', how='left')\n\ntrain = pd.merge(train, auth_mean, on='card_id', how='left')\ntest = pd.merge(test, auth_mean, on='card_id', how='left')\n\ntrain = pd.merge(train, additional_fields, on='card_id', how='left')\ntest = pd.merge(test, additional_fields, on='card_id', how='left')\n\nfeatures = [c for c in train.columns if c not in ['card_id', 'first_active_month']]\n\ncategorical_feats = ['feature_2', 'feature_3']","metadata":{"execution":{"iopub.status.busy":"2024-06-20T06:51:16.185585Z","iopub.execute_input":"2024-06-20T06:51:16.186046Z","iopub.status.idle":"2024-06-20T06:51:21.499732Z","shell.execute_reply.started":"2024-06-20T06:51:16.186002Z","shell.execute_reply":"2024-06-20T06:51:21.498042Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# LightGBMのパラメータ\nparam = {'num_leaves': 111,\n         'min_data_in_leaf': 149, \n         'objective':'regression',\n         'max_depth': 9,\n         'learning_rate': 0.005,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.7522,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.7083 ,\n         \"bagging_seed\": 11,\n         \"metric\": 'rmse',\n         \"lambda_l1\": 0.2634,\n         \"random_state\": 133,\n         \"verbosity\": -1}","metadata":{"execution":{"iopub.status.busy":"2024-06-20T06:51:21.501665Z","iopub.execute_input":"2024-06-20T06:51:21.502103Z","iopub.status.idle":"2024-06-20T06:51:21.509262Z","shell.execute_reply.started":"2024-06-20T06:51:21.502071Z","shell.execute_reply":"2024-06-20T06:51:21.507832Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ベースライン(特徴量選択なしの精度)の確認\nまずは特徴量なしの場合での精度を確認してみましょう。  \n精度指標として今回はRMSEを利用します。\n\n### 評価指標の説明(RMSE)\nhttps://pythondatascience.plavox.info/scikit-learn/%E5%9B%9E%E5%B8%B0%E3%83%A2%E3%83%87%E3%83%AB%E3%81%AE%E8%A9%95%E4%BE%A1","metadata":{}},{"cell_type":"code","source":"# データをKFoldで5分割して学習\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof = np.zeros(len(train))\npredictions = np.zeros(len(test))\nstart = time.time()\nfeature_importance_df = pd.DataFrame()\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train.values, target.values)):\n    print(\"fold n°{}\".format(fold_))\n    trn_data = lgb.Dataset(train.iloc[trn_idx][features],\n                           label=target.iloc[trn_idx],\n                           categorical_feature=categorical_feats\n                          )\n    val_data = lgb.Dataset(train.iloc[val_idx][features],\n                           label=target.iloc[val_idx],\n                           categorical_feature=categorical_feats\n                          )\n\n    num_round = 10000\n    clf = lgb.train(params=param,\n                    train_set=trn_data,\n                    num_boost_round=num_round,\n                    valid_sets=[val_data],\n                    callbacks=[lgb.early_stopping(stopping_rounds=200),\n                               lgb.log_evaluation(100)])\n\n    oof[val_idx] = clf.predict(train.iloc[val_idx][features], num_iteration=clf.best_iteration)\n    \n    fold_importance_df = pd.DataFrame()\n    fold_importance_df[\"feature\"] = features\n    fold_importance_df[\"importance\"] = clf.feature_importance()\n    fold_importance_df[\"fold\"] = fold_ + 1\n    feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\n    \n    predictions += clf.predict(test[features], num_iteration=clf.best_iteration) / folds.n_splits\n\n    # モデルを保存\n    with open(f'model_fold_baseline_{fold_}.pkl', 'wb') as f:\n        pickle.dump(clf, f)\n\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(oof, target)**0.5))","metadata":{"execution":{"iopub.status.busy":"2024-06-20T06:51:21.510968Z","iopub.execute_input":"2024-06-20T06:51:21.511463Z","iopub.status.idle":"2024-06-20T07:09:32.089326Z","shell.execute_reply.started":"2024-06-20T06:51:21.511403Z","shell.execute_reply":"2024-06-20T07:09:32.087850Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## フィルター法（Filter Method）で特徴量選択（変数選択）\n相関係数を利用する方法で、特徴量選択（変数選択）を実施します。  \n目的変数targetと、説明変数の相関係数を求めます。","metadata":{}},{"cell_type":"code","source":"# 相関係数に基づいた特徴量選択\ncor = pd.concat([train[features], target], axis=1).corr()\ntarget_cor = abs(cor['target'])","metadata":{"execution":{"iopub.status.busy":"2024-06-20T07:09:32.091193Z","iopub.execute_input":"2024-06-20T07:09:32.091639Z","iopub.status.idle":"2024-06-20T07:09:53.627135Z","shell.execute_reply.started":"2024-06-20T07:09:32.091597Z","shell.execute_reply":"2024-06-20T07:09:53.625260Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"ある基準を設定し、特徴量選択（変数選択）を実施します。   \n今回は、目的変数との相関係数の絶対値が0.05より大きい、という基準を設けます。","metadata":{}},{"cell_type":"code","source":"# 特徴量抽出\nselected_features_filter = target_cor[target_cor > 0.05].index.drop('target')","metadata":{"execution":{"iopub.status.busy":"2024-06-20T07:09:53.629158Z","iopub.execute_input":"2024-06-20T07:09:53.629548Z","iopub.status.idle":"2024-06-20T07:09:53.638015Z","shell.execute_reply.started":"2024-06-20T07:09:53.629517Z","shell.execute_reply":"2024-06-20T07:09:53.636627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ラッパー法（Wrapper Method）で特徴量選択（変数選択）\n\n- p値を利用した変数減少法で、特徴量選択（変数選択）を実施します。\n\n\n学習データで重回帰モデルを構築し、各説明変数の係数のp値を求めます。","metadata":{}},{"cell_type":"code","source":"# p値を使ったラッパー法\n\ndef backward_elimination(X, y, p_value_threshold=0.05):\n    \n    # 欠損値があると線形回帰ができないため、一旦０埋め\n    X = X.fillna(0)\n    cols = list(X.columns)\n    while len(cols) > 0:\n        X_ = sm.add_constant(X[cols])\n        model = sm.OLS(y, X_).fit()\n        p_values = model.pvalues.iloc[1:]\n        max_p_value = p_values.max()\n        if max_p_value > p_value_threshold:\n            excluded_feature = p_values.idxmax()\n            cols.remove(excluded_feature)\n        else:\n            break\n    return cols\n\n# 特徴量の抽出\nselected_features_pval = backward_elimination(train[features], target)","metadata":{"execution":{"iopub.status.busy":"2024-06-20T07:09:53.661884Z","iopub.execute_input":"2024-06-20T07:09:53.662261Z","iopub.status.idle":"2024-06-20T07:15:46.522726Z","shell.execute_reply.started":"2024-06-20T07:09:53.662232Z","shell.execute_reply":"2024-06-20T07:15:46.520380Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## RFE法（重回帰）による変数選択\n\nRFE法で、特徴量選択（変数選択）を実施します。  \nRFE法の中で変数選択で利用するモデルは重回帰モデル（線形回帰モデル）です。","metadata":{}},{"cell_type":"code","source":"# RFE法\n# n_features_to_selectで選択する特徴量の数を決めることもできる(設定しないと特徴量の数は半分になる)\n\n# 欠損値があると線形回帰ができないため、一旦０埋めして学習する\nrfe = RFE(LinearRegression())\nrfe.fit(train[features].fillna(0), target)\n\n# 特徴量の抽出\nselected_features_rfe = train[features].columns[rfe.support_]","metadata":{"execution":{"iopub.status.busy":"2024-06-20T07:15:46.526083Z","iopub.execute_input":"2024-06-20T07:15:46.527484Z","iopub.status.idle":"2024-06-20T07:19:42.623159Z","shell.execute_reply.started":"2024-06-20T07:15:46.527415Z","shell.execute_reply":"2024-06-20T07:19:42.621561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 埋め込み法（Embedded Method）で特徴量選択（変数選択）\n- Lasso回帰を用いて変数選択をしてみます。","metadata":{}},{"cell_type":"code","source":"# 埋め込み法（Lasso回帰）\n\n# データの標準化\nscaler = StandardScaler()\nX_train_std = scaler.fit_transform(train[features].fillna(0))\nX_test_std = scaler.transform(test[features].fillna(0))\n\n# ラッソ回帰の学習\nlasso = Lasso(alpha=0.01)\nlasso.fit(X_train_std, target)\n\n# 特徴量の抽出\nselected_features_lasso = train[features].columns[lasso.coef_ != 0]","metadata":{"execution":{"iopub.status.busy":"2024-06-20T07:19:42.625339Z","iopub.execute_input":"2024-06-20T07:19:42.625834Z","iopub.status.idle":"2024-06-20T07:19:49.052408Z","shell.execute_reply.started":"2024-06-20T07:19:42.625791Z","shell.execute_reply":"2024-06-20T07:19:49.051337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 特徴量選択を適用したモデルの学習と評価\ndef train_and_evaluate(selected_features, feature_name):\n    folds = KFold(n_splits=5, shuffle=True, random_state=15)\n    oof = np.zeros(len(train))\n    predictions = np.zeros(len(test))\n    feature_importance_df = pd.DataFrame()\n\n    for fold_, (trn_idx, val_idx) in enumerate(folds.split(train.values, target.values)):\n        print(f\"fold n°{fold_} - {feature_name}\")\n        \n        # filter法を使った特徴選択の場合categorical_featsの変数が消えてしまうので別処理にする\n        if feature_name == \"corr\":\n            trn_data = lgb.Dataset(train.iloc[trn_idx][selected_features],\n                                   label=target.iloc[trn_idx])\n            val_data = lgb.Dataset(train.iloc[val_idx][selected_features],\n                                   label=target.iloc[val_idx])\n        else:    \n            trn_data = lgb.Dataset(train.iloc[trn_idx][selected_features],\n                                   label=target.iloc[trn_idx],\n                                   categorical_feature=categorical_feats)\n            val_data = lgb.Dataset(train.iloc[val_idx][selected_features],\n                                   label=target.iloc[val_idx],\n                                   categorical_feature=categorical_feats)\n\n        num_round = 10000\n        clf = lgb.train(params=param,\n                        train_set=trn_data,\n                        num_boost_round=num_round,\n                        valid_sets=[val_data],\n                        callbacks=[lgb.early_stopping(stopping_rounds=200),\n                                   lgb.log_evaluation(100)])\n\n        oof[val_idx] = clf.predict(train.iloc[val_idx][selected_features], num_iteration=clf.best_iteration)\n\n        fold_importance_df = pd.DataFrame()\n        fold_importance_df[\"feature\"] = selected_features\n        fold_importance_df[\"importance\"] = clf.feature_importance()\n        fold_importance_df[\"fold\"] = fold_ + 1\n        feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\n\n        predictions += clf.predict(test[selected_features], num_iteration=clf.best_iteration) / folds.n_splits\n\n        # モデルを保存\n        with open(f'{feature_name}_model_fold_{fold_}.pkl', 'wb') as f:\n            pickle.dump(clf, f)\n\n    cv_score = mean_squared_error(oof, target) ** 0.5\n    print(f\"CV score with {feature_name}: {cv_score:<8.5f}\")\n\n# 各特徴量選択方法での学習と評価\ntrain_and_evaluate(selected_features_filter, \"corr\")\nprint('-'* 100)\ntrain_and_evaluate(selected_features_pval, \"p-value\")\nprint('-'* 100)\ntrain_and_evaluate(selected_features_rfe, \"RFE\")\nprint('-'* 100)\ntrain_and_evaluate(selected_features_lasso, \"Lasso\")","metadata":{"execution":{"iopub.status.busy":"2024-06-20T07:29:21.132789Z","iopub.execute_input":"2024-06-20T07:29:21.134518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}