![\"Concept](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4310004%2F9b1674b9b9a6a326f741acdb1b61967e%2FDSC082821_res.png?generation=1741227470938082&alt=media\")
First of all, I want to thank to my teammates @introverstein, @yuogawa and my love @daphne4sg who put a lot of effort and support me. This is the first time I paricipant a competititon for totally three month. It was hard because I was in the Taiwan military service for 4 months. They only allow me to use my cellphone so I write all my codes on my cellphone, typing code using my finger is furstrated. During this competition, I usually have quarrel with my girlfriend @daphne4sg because I spend all day long on coding and didn't talk to her and relax. So I made a promise that if we are in top 5 then I'll take you to Japan (maybe visit my teammate @yuogawa). It turns out to be 108th place. That's fine, \"Failure is the mother of success\".
\nMy solution
\nOne day I attempted GNN on raw data and it didin't show good results. I create edges purely based on the categorical features. Here is an example using race_group:
():\n race_groups = .data.groupby()[].apply()\n edges = []\n group tqdm(race_groups):\n edges.extend(combinations(group, ))\n edge_index = torch.tensor(edges, dtype=torch.long).T\n edge_index\nThe moment I realize that GNN might works is seeing @andreasbis public codebase. He hardly tried a lot of targets and model combinations. So I'm thinkg about how to use GNN to aggregate them. What if using GNN as stack meta model instead of base model? The edge connection is now built based on the similarity between the model predictions. We don't need to worry about using a lot of categorical features to build edges. It turns out a very good model.
\n(The full version is using the predicitons from @andreasbis codebase, PRL NN, Yunbase and Autogluon to train GNN)
| model | \npublic | \nprivate | \ncv | \n
|---|---|---|---|
| GBDTs | \n0.690 | \n0.689 | \n0.6805 | \n
| GNN | \n0.693 | \n0.693 | \n0.6830 | \n
| GNN (Full) | \n0.694 | \n0.693 | \n0.6911 | \n
I think it works because predictions implicitly indicate the property of each data point and hidden relationship between them. Obviously better than categorical-based edge connection. Here are the details of our solution:
\nRandomNodeLoader to allow the Graphsage training on large graph without OOM.knn_graph is better than similarity thresholding.Fit CoxPH using predictions base models.\nTrain GNN using Cox Target. \npropagation = LabelPropagation(num_layers = , alpha = )\ny = np.array(.y.copy())\ny = np.concatenate([y, * np.ones(X_test.shape[],)], axis=)\ny[valid_index] = \ny[test_index] = \ny_tensor = torch.tensor(y, dtype=torch.long)\nmask = y_tensor.clone() != \nefs_preds = propagation(y_tensor, edge_index, mask=mask)\ntest_efs = torch.argmax(efs_preds, dim=).numpy()\nraces = df_sub.race_group.unique()\nshift = [, -, , , ,]\n i, race (races):\n prl_preds[(test_efs==)&(test.race_group == race)] += shift[i]\n![\"Label](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4310004%2F75eab84303d5339b13ad1f4cb5b45f0d%2F123.png?generation=1741227645957802&alt=media\")
aux_pred from PRL NN and prediction from LabelPropogation to determine whether to eliminate the decision of PRL NN as I observe the loss function giving bias on the race. So my hypothesis is that the graph structure is more convicing than the PRL NN.![\"elminiate\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4310004%2F77162eb766237e1076edd5cfd7a62f1e%2Felmi.png?generation=1741227971695543&alt=media\")
pip install Mambular notebookImplementations
\nModel:
(nn.Module):\n ():\n ().__init__()\n .graphsage = GraphSAGE(in_channels, hidden_channels,\n , out_channels, jk=, norm = LayerNorm(hidden_channels),\n dropout = )\n\n ():\n\n x, edge_index = data.x, data.edge_index\n\n .training:\n edge_index = graph_augmentation(data)\n\n edge_index.shape[]==:\n edge_index = torch.tensor([[, ]]).T.cuda()\n\n \n x = x.cuda()\n edge_index = edge_index.cuda()\n logits = .graphsage(x, edge_index)\n logits\n\n\n ():\n edge_index, _ = dropout_edge(data.edge_index, p=p, training=)\n num_edges = edge_index.size()\n perm = torch.randperm(num_edges)[:num_edges // ]\n edge_index[:, perm] = torch.randint(, data.num_nodes, edge_index[:, perm].shape)\n edge_index\nTraining:
\n mambular.preprocessing.preprocessor Preprocessor\n torch_geometric.nn.models GraphSAGE, LabelPropagation\n torch_geometric.loader RandomNodeLoader\n torch.optim.lr_scheduler ExponentialLR\n torch.utils.data DataLoader, TensorDataset\n\n ():\n\n X = pd.DataFrame(np.vstack(oof_model_preds).T)\n y = .data[target]\n\n \n models, pps, fold_scores = [], [], []\n\n cv, oof_preds = .targets._prepare_cv()\n\n fold, (train_index, valid_index) (cv.split(X, y)):\n \n X_train = X.iloc[train_index]\n X_valid = X.iloc[valid_index]\n y_train = y.iloc[train_index]\n y_valid = y.iloc[valid_index]\n\n \n ()\n pp = Preprocessor()\n X_train_dict = pp.fit_transform(X_train, y_train)\n X_valid_dict = pp.transform(X_valid)\n\n \n X_train = pd.DataFrame()\n k,v X_train_dict.items():\n i (v.shape[]):\n X_train[] = v[:, i]\n\n X_valid = pd.DataFrame()\n k,v X_valid_dict.items():\n i (v.shape[]):\n X_valid[] = v[:, i]\n\n \n set_seed()\n best_model, best_score, best_oof_preds = .train_fold(\n X_train, y_train, X_valid, y_valid, oof_preds,\n train_index, valid_index)\n oof_preds = best_oof_preds\n models.append(best_model)\n pps.append(pp)\n fold_scores.append(best_score)\n .eda._plot_cv(fold_scores, title)\n .targets.validate_model(oof_preds, title)\n models, pps, oof_preds\n\n\n ():\n\n\n \n X_train = torch.tensor(X_train.values, dtype=torch.float32)\n X_valid = torch.tensor(X_valid.values, dtype=torch.float32)\n\n\n \n y_train = torch.tensor(y_train.values, dtype=torch.float32).unsqueeze()\n y_valid = torch.tensor(y_valid.values, dtype=torch.float32).unsqueeze()\n\n\n \n X = torch.cat([torch.zeros_like(X_train), torch.zeros_like(X_valid)], dim=)\n y = torch.cat([torch.zeros_like(y_train), torch.zeros_like(y_valid)], dim=)\n X[train_index] = X_train\n X[valid_index] = X_valid\n y[train_index] = y_train\n y[valid_index] = y_valid\n\n \n train_mask = torch.zeros((X), dtype=torch.)\n valid_mask = torch.zeros((X), dtype=torch.)\n train_mask[train_index] = \n valid_mask[valid_index] = \n\n \n \n ()\n edge_index = .build_edges(X, CFG.k)\n\n\n \n data = Data(x=X, edge_index=edge_index, y=y)\n data.train_mask = train_mask\n data.test_mask = valid_mask\n data.x = data.x.contiguous()\n data.edge_index = data.edge_index.contiguous()\n data.y = data.y.contiguous()\n data.train_mask = data.train_mask.contiguous()\n data.test_mask = data.test_mask.contiguous()\n\n ()\n\n \n train_loader = RandomNodeLoader(data, num_parts = CFG.num_parts, shuffle=)\n valid_loader = RandomNodeLoader(data, num_parts = CFG.num_parts, shuffle=)\n\n \n params = {\n : X_train.shape[]\n }\n set_seed()\n model = GNN_Model(**params).cuda()\n\n \n criterion = nn.MSELoss()\n optimizer = optim.Adam(\n model.parameters(),\n lr=CFG.lr,\n weight_decay=CFG.weight_decay\n )\n scheduler = ExponentialLR(optimizer, gamma=)\n best_score = \n best_oof_preds = oof_preds\n best_model = \n patience = \n epoch (CFG.epochs):\n model.train()\n loop = tqdm(train_loader)\n o=[]\n batch loop:\n y = batch.y.cuda()\n out = model(batch)\n loss = criterion(out[[batch.train_mask]], y[[batch.train_mask]])\n optimizer.zero_grad()\n loss.backward()\n optimizer.step()\n loop.set_postfix(loss=)\n scheduler.step()\n model.()\n oof_preds_ = []\n torch.no_grad():\n batch valid_loader:\n out = model(batch).cpu()\n oof_preds_.append(out[batch.test_mask][:, ])\n oof_preds_ = torch.cat(oof_preds_, dim=)\n oof_preds[valid_index] = oof_preds_.numpy()\n y_true_fold = .data.iloc[valid_index][[, , , ]].copy()\n y_pred_fold = .data.iloc[valid_index][[]].copy()\n y_pred_fold[] = oof_preds[valid_index]\n fold_score = score(y_true_fold, y_pred_fold, )\n fold_score > best_score:\n best_score = fold_score\n best_oof_preds\n best_oof_preds = copy(oof_preds)\n best_model\n best_model = deepcopy(model)\n patience = \n :\n patience += \n ()\n patience >= CFG.patience:\n \n best_model, best_score, best_oof_preds\n\n ():\n num_samples = features.shape[]\n edges = []\n start tqdm((, num_samples, batch_size)):\n end = (start + batch_size, num_samples)\n batch_features = features[start:end]\n \n batch_edges = knn_graph(batch_features, k=k, batch=, loop=)\n \n batch_edges += start\n edges.append(batch_edges)\n torch.cat(edges, dim=)\nNote
\nBest regards, Tom Huang.
", "comments": [ { "id": 3145550, "author_name": "Chris Deotte", "author_url": "", "post_date": "2025-03-10T01:03:32.057000", "content": "Great job Tom! I love how your team made a strong GNN model!
", "votes": 1, "replies": [ { "id": 3145840, "author_name": "Tom", "author_url": "", "post_date": "2025-03-10T09:53:25.997000", "content": "@cdeotte Thank you chris! You also provide a lot of good material in this comp. Maybe next time we can solve the problem together!
", "votes": 1, "replies": [] } ] }, { "id": 3141931, "author_name": "Zhu Siqi", "author_url": "", "post_date": "2025-03-06T01:40:23.410000", "content": "Thanks for your amazing solution! Could I ask what are the inputs and outputs when using Coxph as stack meta model?
", "votes": 1, "replies": [ { "id": 3141935, "author_name": "Tom", "author_url": "", "post_date": "2025-03-06T01:45:48.830000", "content": "Assume you have a predicitons from other base models named data.
(): \n\n \n\n cv, oof_preds = ._prepare_cv()\n\n train_index, valid_index cv.split(data):\n\n train_data = data.iloc[train_index]\n valid_data = data.iloc[valid_index]\n\n \n train_data = train_data.loc[:, train_data.nunique() > ]\n valid_data = valid_data[train_data.columns]\n\n cph = CoxPHFitter(penalizer=._penalizer)\n cph.fit(train_data, duration_col=, event_col=)\n\n oof_preds[valid_index] = cph.predict_partial_hazard(valid_data) \n .data[] = oof_preds\n .validate_model(oof_preds, ) \n\n .data\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4310004%2Fc32e9d80616a34446e88a972c33c6a50%2FDSC082821_res.png?generation=1741225622034558&alt=media\")
Thanks for the mention @tom99763 . These past three months have been an incredible learning experience with the team, and I got to learn a lot from you all - both in terms of knowledge and work ethic.
", "votes": 1, "replies": [ { "id": 3141940, "author_name": "Tom", "author_url": "", "post_date": "2025-03-06T01:48:02.733000", "content": "@introverstein Thank you for being my teammate. I aslo learn a lot from you.
", "votes": 1, "replies": [] } ] }, { "id": 3141912, "author_name": "Sheikh Muhammad Abdullah", "author_url": "", "post_date": "2025-03-06T01:21:54.727000", "content": "Congrats, @tom99763.
", "votes": 1, "replies": [ { "id": 3141949, "author_name": "Tom", "author_url": "", "post_date": "2025-03-06T01:53:54.070000", "content": "@abdmental01 Thank you! I use your codebase also. Great implementation.
", "votes": 0, "replies": [] } ] }, { "id": 3141907, "author_name": "eikyou", "author_url": "", "post_date": "2025-03-06T01:20:06.163000", "content": "Man, u are just nuts, in a positive way of course. Congratulations!
", "votes": 1, "replies": [ { "id": 3141943, "author_name": "Tom", "author_url": "", "post_date": "2025-03-06T01:53:01.100000", "content": "@eikyou Thank you!
", "votes": 0, "replies": [] } ] }, { "id": 3141900, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2025-03-06T01:16:15.293000", "content": "Great effort and great result @tom99763
\nThanks for the mambular reference also
@ravi20076 Thank you! See you in another competition.
", "votes": 0, "replies": [] } ] }, { "id": 3142454, "author_name": "YuOgawa", "author_url": "", "post_date": "2025-03-06T12:02:41.547000", "content": "@tom99763 ,Thanks for the mention.
\n It was an honor to be part of such an amazing team.
\nOur team leader, @tom99763 , led us with incredible dedication from start to finish. Honestly, I have no idea when he even sleeps.
\nMy teammates, @introverstein and @daphne4sg ,did an outstanding job—precise, reliable, and creative.
\nI have so much respect for my teammates, and I will continue striving to be like them.
\nThank you!
@yuogawa You hardly try a lot of ideas. Although we miss a lot of improvement directions, we cannot get this place without your effort. Maybe next time we can team up together again.
", "votes": 0, "replies": [] } ] }, { "id": 3143476, "author_name": "SCRIPTCHEF", "author_url": "", "post_date": "2025-03-07T09:10:47.447000", "content": "Hi Tom, I was competing whilst on holiday and thought that was bad, but competing from military service from a phone is insane (and you got a great position!). I am very interested in learning more about GNNs and will definitely use your implementation to do so. If you had any other recommendations on reading material or videos you used to learn about them, that would be great! Congrats again!
", "votes": 0, "replies": [ { "id": 3143534, "author_name": "Tom", "author_url": "", "post_date": "2025-03-07T10:07:24.823000", "content": "@noodl35 You can learn from Stanford Machine Learning with Graph . See the final projects made by their students. Graph is usually used for recommender system. But if your data is structural, feel free to try it.
", "votes": 0, "replies": [] } ] }, { "id": 3142689, "author_name": "Andreas Bisiadis", "author_url": "", "post_date": "2025-03-06T15:51:58.960000", "content": "@tom99763 Congrats for your team's work and it's impressive you managed to serve in military, have a girlfriend and participate in this competition from a smartphone at the same time period! That's something to look up to!
\nI have one question with your solution:
\n\n\nUsing Coxph as stack meta model fitted the predictions and treat its prediction as targets (Knowledge Distillation).
\n
From my understanding, did you use CoxPH as (regression) target for the GNN?
", "votes": 0, "replies": [ { "id": 3143065, "author_name": "Tom", "author_url": "", "post_date": "2025-03-06T22:46:18.283000", "content": "@andreasbis In your public notebook, you feed raw data to coxph, but I feed predictions from base models . I found this method outperforms other targets for meta model.
", "votes": 1, "replies": [ { "id": 3143296, "author_name": "Andreas Bisiadis", "author_url": "", "post_date": "2025-03-07T04:49:26.183000", "content": "@tom99763 how did you get thus target engineering approach to work? When I tried this approach I was getting values up to 600B and the GBDT models that use this target were random guessing.
", "votes": 0, "replies": [] }, { "id": 3143301, "author_name": "Tom", "author_url": "", "post_date": "2025-03-07T04:55:05.587000", "content": "@andreasbis Here is an example. I have made it public. https://www.kaggle.com/code/tom99763/graphsage-stack-ensemble-cox-target-augment
", "votes": 1, "replies": [] } ] } ] }, { "id": 3141939, "author_name": "Vladimir Demidov", "author_url": "", "post_date": "2025-03-06T01:47:57.367000", "content": "In your own words, how is Mambular doing?
", "votes": 0, "replies": [ { "id": 3141942, "author_name": "Tom", "author_url": "", "post_date": "2025-03-06T01:49:30.067000", "content": "@yekenot It provides ResNet, TabM, FTTransformer and Saint models for training tabular data with their preprocessing method.
", "votes": 1, "replies": [] } ] }, { "id": 3142478, "author_name": "", "author_url": "", "post_date": "2025-03-06T12:29:46.570000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 24, "messages": [ { "id": 3141896, "postDate": "2025-03-06T01:13:44.847Z", "content": "First of all, I want to thank to my teammates @introverstein, @yuogawa and my love @daphne4sg who put a lot of effort and support me. This is the first time I paricipant a competititon for totally three month. It was hard because I was in the Taiwan military service for 4 months. They only allow me to use my cellphone so I write all my codes on my cellphone, typing code using my finger is furstrated. During this competition, I usually have quarrel with my girlfriend @daphne4sg because I spend all day long on coding and didn't talk to her and relax. So I made a promise that if we are in top 5 then I'll take you to Japan (maybe visit my teammate @yuogawa). It turns out to be 108th place. That's fine, \"Failure is the mother of success\".
\nMy solution
\nOne day I attempted GNN on raw data and it didin't show good results. I create edges purely based on the categorical features. Here is an example using race_group:
():\n race_groups = .data.groupby()[].apply()\n edges = []\n group tqdm(race_groups):\n edges.extend(combinations(group, ))\n edge_index = torch.tensor(edges, dtype=torch.long).T\n edge_index\nThe moment I realize that GNN might works is seeing @andreasbis public codebase. He hardly tried a lot of targets and model combinations. So I'm thinkg about how to use GNN to aggregate them. What if using GNN as stack meta model instead of base model? The edge connection is now built based on the similarity between the model predictions. We don't need to worry about using a lot of categorical features to build edges. It turns out a very good model.
\n(The full version is using the predicitons from @andreasbis codebase, PRL NN, Yunbase and Autogluon to train GNN)
| model | \npublic | \nprivate | \ncv | \n
|---|---|---|---|
| GBDTs | \n0.690 | \n0.689 | \n0.6805 | \n
| GNN | \n0.693 | \n0.693 | \n0.6830 | \n
| GNN (Full) | \n0.694 | \n0.693 | \n0.6911 | \n
I think it works because predictions implicitly indicate the property of each data point and hidden relationship between them. Obviously better than categorical-based edge connection. Here are the details of our solution:
\nRandomNodeLoader to allow the Graphsage training on large graph without OOM.knn_graph is better than similarity thresholding.Fit CoxPH using predictions base models.\nTrain GNN using Cox Target. \npropagation = LabelPropagation(num_layers = , alpha = )\ny = np.array(.y.copy())\ny = np.concatenate([y, * np.ones(X_test.shape[],)], axis=)\ny[valid_index] = \ny[test_index] = \ny_tensor = torch.tensor(y, dtype=torch.long)\nmask = y_tensor.clone() != \nefs_preds = propagation(y_tensor, edge_index, mask=mask)\ntest_efs = torch.argmax(efs_preds, dim=).numpy()\nraces = df_sub.race_group.unique()\nshift = [, -, , , ,]\n i, race (races):\n prl_preds[(test_efs==)&(test.race_group == race)] += shift[i]\n
aux_pred from PRL NN and prediction from LabelPropogation to determine whether to eliminate the decision of PRL NN as I observe the loss function giving bias on the race. So my hypothesis is that the graph structure is more convicing than the PRL NN.
pip install Mambular notebookImplementations
\nModel:
(nn.Module):\n ():\n ().__init__()\n .graphsage = GraphSAGE(in_channels, hidden_channels,\n , out_channels, jk=, norm = LayerNorm(hidden_channels),\n dropout = )\n\n ():\n\n x, edge_index = data.x, data.edge_index\n\n .training:\n edge_index = graph_augmentation(data)\n\n edge_index.shape[]==:\n edge_index = torch.tensor([[, ]]).T.cuda()\n\n \n x = x.cuda()\n edge_index = edge_index.cuda()\n logits = .graphsage(x, edge_index)\n logits\n\n\n ():\n edge_index, _ = dropout_edge(data.edge_index, p=p, training=)\n num_edges = edge_index.size()\n perm = torch.randperm(num_edges)[:num_edges // ]\n edge_index[:, perm] = torch.randint(, data.num_nodes, edge_index[:, perm].shape)\n edge_index\nTraining:
\n mambular.preprocessing.preprocessor Preprocessor\n torch_geometric.nn.models GraphSAGE, LabelPropagation\n torch_geometric.loader RandomNodeLoader\n torch.optim.lr_scheduler ExponentialLR\n torch.utils.data DataLoader, TensorDataset\n\n ():\n\n X = pd.DataFrame(np.vstack(oof_model_preds).T)\n y = .data[target]\n\n \n models, pps, fold_scores = [], [], []\n\n cv, oof_preds = .targets._prepare_cv()\n\n fold, (train_index, valid_index) (cv.split(X, y)):\n \n X_train = X.iloc[train_index]\n X_valid = X.iloc[valid_index]\n y_train = y.iloc[train_index]\n y_valid = y.iloc[valid_index]\n\n \n ()\n pp = Preprocessor()\n X_train_dict = pp.fit_transform(X_train, y_train)\n X_valid_dict = pp.transform(X_valid)\n\n \n X_train = pd.DataFrame()\n k,v X_train_dict.items():\n i (v.shape[]):\n X_train[] = v[:, i]\n\n X_valid = pd.DataFrame()\n k,v X_valid_dict.items():\n i (v.shape[]):\n X_valid[] = v[:, i]\n\n \n set_seed()\n best_model, best_score, best_oof_preds = .train_fold(\n X_train, y_train, X_valid, y_valid, oof_preds,\n train_index, valid_index)\n oof_preds = best_oof_preds\n models.append(best_model)\n pps.append(pp)\n fold_scores.append(best_score)\n .eda._plot_cv(fold_scores, title)\n .targets.validate_model(oof_preds, title)\n models, pps, oof_preds\n\n\n ():\n\n\n \n X_train = torch.tensor(X_train.values, dtype=torch.float32)\n X_valid = torch.tensor(X_valid.values, dtype=torch.float32)\n\n\n \n y_train = torch.tensor(y_train.values, dtype=torch.float32).unsqueeze()\n y_valid = torch.tensor(y_valid.values, dtype=torch.float32).unsqueeze()\n\n\n \n X = torch.cat([torch.zeros_like(X_train), torch.zeros_like(X_valid)], dim=)\n y = torch.cat([torch.zeros_like(y_train), torch.zeros_like(y_valid)], dim=)\n X[train_index] = X_train\n X[valid_index] = X_valid\n y[train_index] = y_train\n y[valid_index] = y_valid\n\n \n train_mask = torch.zeros((X), dtype=torch.)\n valid_mask = torch.zeros((X), dtype=torch.)\n train_mask[train_index] = \n valid_mask[valid_index] = \n\n \n \n ()\n edge_index = .build_edges(X, CFG.k)\n\n\n \n data = Data(x=X, edge_index=edge_index, y=y)\n data.train_mask = train_mask\n data.test_mask = valid_mask\n data.x = data.x.contiguous()\n data.edge_index = data.edge_index.contiguous()\n data.y = data.y.contiguous()\n data.train_mask = data.train_mask.contiguous()\n data.test_mask = data.test_mask.contiguous()\n\n ()\n\n \n train_loader = RandomNodeLoader(data, num_parts = CFG.num_parts, shuffle=)\n valid_loader = RandomNodeLoader(data, num_parts = CFG.num_parts, shuffle=)\n\n \n params = {\n : X_train.shape[]\n }\n set_seed()\n model = GNN_Model(**params).cuda()\n\n \n criterion = nn.MSELoss()\n optimizer = optim.Adam(\n model.parameters(),\n lr=CFG.lr,\n weight_decay=CFG.weight_decay\n )\n scheduler = ExponentialLR(optimizer, gamma=)\n best_score = \n best_oof_preds = oof_preds\n best_model = \n patience = \n epoch (CFG.epochs):\n model.train()\n loop = tqdm(train_loader)\n o=[]\n batch loop:\n y = batch.y.cuda()\n out = model(batch)\n loss = criterion(out[[batch.train_mask]], y[[batch.train_mask]])\n optimizer.zero_grad()\n loss.backward()\n optimizer.step()\n loop.set_postfix(loss=)\n scheduler.step()\n model.()\n oof_preds_ = []\n torch.no_grad():\n batch valid_loader:\n out = model(batch).cpu()\n oof_preds_.append(out[batch.test_mask][:, ])\n oof_preds_ = torch.cat(oof_preds_, dim=)\n oof_preds[valid_index] = oof_preds_.numpy()\n y_true_fold = .data.iloc[valid_index][[, , , ]].copy()\n y_pred_fold = .data.iloc[valid_index][[]].copy()\n y_pred_fold[] = oof_preds[valid_index]\n fold_score = score(y_true_fold, y_pred_fold, )\n fold_score > best_score:\n best_score = fold_score\n best_oof_preds\n best_oof_preds = copy(oof_preds)\n best_model\n best_model = deepcopy(model)\n patience = \n :\n patience += \n ()\n patience >= CFG.patience:\n \n best_model, best_score, best_oof_preds\n\n ():\n num_samples = features.shape[]\n edges = []\n start tqdm((, num_samples, batch_size)):\n end = (start + batch_size, num_samples)\n batch_features = features[start:end]\n \n batch_edges = knn_graph(batch_features, k=k, batch=, loop=)\n \n batch_edges += start\n edges.append(batch_edges)\n torch.cat(edges, dim=)\nNote
\nBest regards, Tom Huang.
", "rawMarkdown": "First of all, I want to thank to my teammates @introverstein, @yuogawa and my love @daphne4sg who put a lot of effort and support me. This is the first time I paricipant a competititon for totally three month. It was hard because I was in the Taiwan military service for 4 months. They only allow me to use my cellphone so I write all my codes on my cellphone, typing code using my finger is furstrated. During this competition, I usually have quarrel with my girlfriend @daphne4sg because I spend all day long on coding and didn't talk to her and relax. So I made a promise that if we are in top 5 then I'll take you to Japan (maybe visit my teammate @yuogawa). It turns out to be 108th place. That's fine, \"Failure is the mother of success\". \n\n***My solution***\nOne day I attempted GNN on raw data and it didin't show good results. I create edges purely based on the categorical features. Here is an example using race_group:\n\n```python\ndef build_race_edges(self):\n race_groups = self.data.groupby('race_group')['ID'].apply(list)\n edges = []\n for group in tqdm(race_groups):\n edges.extend(combinations(group, 2))\n edge_index = torch.tensor(edges, dtype=torch.long).T\n return edge_index\n```\nThe moment I realize that GNN might works is seeing @andreasbis public codebase. He hardly tried a lot of targets and model combinations. So I'm thinkg about how to use GNN to aggregate them. What if using GNN as stack meta model instead of base model? The edge connection is now built based on the similarity between the model predictions. We don't need to worry about using a lot of categorical features to build edges. It turns out a very good model. \n(The full version is using the predicitons from @andreasbis codebase, PRL NN, Yunbase and Autogluon to train GNN)\n\n| model | public | private | cv |\n| --- | --- | --- | --- |\n| GBDTs | 0.690 | 0.689 | 0.6805 |\n| GNN | 0.693 | 0.693 | 0.6830 |\n| GNN (Full) | 0.694 | 0.693 | 0.6911 |\n\n\n\n**I think it works because predictions implicitly indicate the property of each data point and hidden relationship between them. Obviously better than categorical-based edge connection.** Here are the details of our solution:\n* To reduce the computation complexity of building edges (a graph with 28800 nodes), we use `RandomNodeLoader` to allow the Graphsage training on large graph without OOM.\n* Building edges using `knn_graph` is better than similarity thresholding.\n* Knowledge Distillation from CoxPH\n\n```python\nFit CoxPH using predictions from base models.\nTrain GNN using Cox Target. \n```\n* The final solution of our team using quantile, coxph, NAF, Race-wise KMF. Among them, quantile and coxph give very good performance for training the stack meta model.\n* Using the prediction from label propogation to mask PRL NN race-wisely as it also indicates the determination of your graph structure. (It did better than XGBoost)\n\n```python\npropagation = LabelPropagation(num_layers = 10, alpha = 0.9)\ny = np.array(self.y.copy())\ny = np.concatenate([y, 2 * np.ones(X_test.shape[0],)], axis=0)\ny[valid_index] = 2\ny[test_index] = 2\ny_tensor = torch.tensor(y, dtype=torch.long)\nmask = y_tensor.clone() != 2\nefs_preds = propagation(y_tensor, edge_index, mask=mask)\ntest_efs = torch.argmax(efs_preds, dim=1).numpy()\nraces = df_sub.race_group.unique()\nshift = [0.1, -0.5, 0.0, 0. ,0.4 ,0.]\nfor i, race in enumerate(races):\n prl_preds[(test_efs==1)&(test.race_group == race)] += shift[i]\n``` \n\n\n\n* Continue to LabelPropogation., the first method I used is the eliminiation I mention [before](https://www.kaggle.com/competitions/equity-post-HCT-survival-predictions/discussion/561146#3115494). I use the disagreement between `aux_pred` from PRL NN and prediction from LabelPropogation to determine whether to eliminate the decision of PRL NN as I observe the loss function giving bias on the race. So my hypothesis is that the graph structure is more convicing than the PRL NN.\n\n\n\n* I also add ResNet, TabM, FTTransformer, Saint in @@andreasbis codebase to increase the target diversity, which are trained with Coxph targets. And let me introduce the library I use [Mambular](https://mambular.readthedocs.io/en/latest/index.html). Welcome to use my `pip install Mambular` [notebook](https://www.kaggle.com/code/tom99763/pip-install-mambular)\n\n\n***Implementations***\nModel:\n```python\nclass GNN_Model(nn.Module):\n def __init__(self, in_channels, hidden_channels = 256, out_channels = 1):\n super().__init__()\n self.graphsage = GraphSAGE(in_channels, hidden_channels,\n 8, out_channels, jk='lstm', norm = LayerNorm(hidden_channels),\n dropout = 0.3)\n\n def forward(self, data):\n \n x, edge_index = data.x, data.edge_index\n\n if self.training:\n edge_index = graph_augmentation(data)\n \n if edge_index.shape[0]==0:\n edge_index = torch.tensor([[0, 0]]).T.cuda()\n\n #process input\n x = x.cuda()\n edge_index = edge_index.cuda()\n logits = self.graphsage(x, edge_index)\n return logits\n\n#That's a trick.\ndef graph_augmentation(data, p=0.5):\n edge_index, _ = dropout_edge(data.edge_index, p=p, training=True)\n num_edges = edge_index.size(1)\n perm = torch.randperm(num_edges)[:num_edges // 5]\n edge_index[:, perm] = torch.randint(0, data.num_nodes, edge_index[:, perm].shape)\n return edge_index\n```\n\nTraining:\n```python\nfrom mambular.preprocessing.preprocessor import Preprocessor\nfrom torch_geometric.nn.models import GraphSAGE, LabelPropagation\nfrom torch_geometric.loader import RandomNodeLoader\nfrom torch.optim.lr_scheduler import ExponentialLR\nfrom torch.utils.data import DataLoader, TensorDataset\n\ndef train_model(self, target, title, oof_model_preds):\n \n X = pd.DataFrame(np.vstack(oof_model_preds).T)\n y = self.data[target]\n\n #train\n models, pps, fold_scores = [], [], []\n \n cv, oof_preds = self.targets._prepare_cv()\n \n for fold, (train_index, valid_index) in enumerate(cv.split(X, y)):\n #split\n X_train = X.iloc[train_index]\n X_valid = X.iloc[valid_index]\n y_train = y.iloc[train_index]\n y_valid = y.iloc[valid_index]\n\n #preprocess\n print('preprocess')\n pp = Preprocessor()\n X_train_dict = pp.fit_transform(X_train, y_train)\n X_valid_dict = pp.transform(X_valid)\n\n #convert dict to dataframe\n X_train = pd.DataFrame()\n for k,v in X_train_dict.items():\n for i in range(v.shape[1]):\n X_train[f'{k}_{i}'] = v[:, i]\n\n X_valid = pd.DataFrame()\n for k,v in X_valid_dict.items():\n for i in range(v.shape[1]):\n X_valid[f'{k}_{i}'] = v[:, i]\n \n #modeling\n set_seed(42)\n best_model, best_score, best_oof_preds = self.train_fold(\n X_train, y_train, X_valid, y_valid, oof_preds,\n train_index, valid_index)\n oof_preds = best_oof_preds\n models.append(best_model)\n pps.append(pp)\n fold_scores.append(best_score)\n self.eda._plot_cv(fold_scores, title)\n self.targets.validate_model(oof_preds, title)\n return models, pps, oof_preds\n\n \n def train_fold(self, X_train, y_train,\n X_valid, y_valid, oof_preds,\n train_index, valid_index):\n \n \n #data\n X_train = torch.tensor(X_train.values, dtype=torch.float32)\n X_valid = torch.tensor(X_valid.values, dtype=torch.float32)\n\n\n #label\n y_train = torch.tensor(y_train.values, dtype=torch.float32).unsqueeze(1)\n y_valid = torch.tensor(y_valid.values, dtype=torch.float32).unsqueeze(1)\n\n\n #assign\n X = torch.cat([torch.zeros_like(X_train), torch.zeros_like(X_valid)], dim=0)\n y = torch.cat([torch.zeros_like(y_train), torch.zeros_like(y_valid)], dim=0)\n X[train_index] = X_train\n X[valid_index] = X_valid\n y[train_index] = y_train\n y[valid_index] = y_valid\n\n #mask\n train_mask = torch.zeros(len(X), dtype=torch.bool)\n valid_mask = torch.zeros(len(X), dtype=torch.bool)\n train_mask[train_index] = True\n valid_mask[valid_index] = True\n\n # Create Graph Edges\n # Connect if they have high similarity between numerical features\n print('build_edges')\n edge_index = self.build_edges(X, CFG.k)\n \n\n # Create PyG Data Object\n data = Data(x=X, edge_index=edge_index, y=y)\n data.train_mask = train_mask\n data.test_mask = valid_mask\n data.x = data.x.contiguous()\n data.edge_index = data.edge_index.contiguous()\n data.y = data.y.contiguous()\n data.train_mask = data.train_mask.contiguous()\n data.test_mask = data.test_mask.contiguous()\n\n print('train')\n\n # Partition the graph\n train_loader = RandomNodeLoader(data, num_parts = CFG.num_parts, shuffle=True)\n valid_loader = RandomNodeLoader(data, num_parts = CFG.num_parts, shuffle=False)\n\n #model\n params = {\n 'in_channels': X_train.shape[1]\n }\n set_seed(42)\n model = GNN_Model(**params).cuda()\n\n #optimizer\n criterion = nn.MSELoss()\n optimizer = optim.Adam(\n model.parameters(),\n lr=CFG.lr,\n weight_decay=CFG.weight_decay\n )\n scheduler = ExponentialLR(optimizer, gamma=0.99)\n best_score = 0\n best_oof_preds = oof_preds\n best_model = None\n patience = 0\n for epoch in range(CFG.epochs):\n model.train()\n loop = tqdm(train_loader)\n o=[]\n for batch in loop:\n y = batch.y.cuda()\n out = model(batch)\n loss = criterion(out[[batch.train_mask]], y[[batch.train_mask]])\n optimizer.zero_grad()\n loss.backward()\n optimizer.step()\n loop.set_postfix(loss=f'epoch:{epoch}--loss:{loss}')\n scheduler.step()\n model.eval()\n oof_preds_ = []\n with torch.no_grad():\n for batch in valid_loader:\n out = model(batch).cpu()\n oof_preds_.append(out[batch.test_mask][:, 0])\n oof_preds_ = torch.cat(oof_preds_, dim=0)\n oof_preds[valid_index] = oof_preds_.numpy()\n y_true_fold = self.data.iloc[valid_index][['ID', 'efs', 'efs_time', 'race_group']].copy()\n y_pred_fold = self.data.iloc[valid_index][['ID']].copy()\n y_pred_fold['prediction'] = oof_preds[valid_index]\n fold_score = score(y_true_fold, y_pred_fold, 'ID')\n if fold_score > best_score:\n best_score = fold_score\n del best_oof_preds\n best_oof_preds = copy(oof_preds)\n del best_model\n best_model = deepcopy(model)\n patience = 0\n else:\n patience += 1\n print(f'epoch:{epoch}--c-index:{fold_score}--best score :{best_score}')\n if patience >= CFG.patience:\n break\n return best_model, best_score, best_oof_preds\n\n def build_edges(self, features, k, batch_size=5000):\n num_samples = features.shape[0]\n edges = []\n for start in tqdm(range(0, num_samples, batch_size)):\n end = min(start + batch_size, num_samples)\n batch_features = features[start:end]\n # Compute k-NN for the current batch\n batch_edges = knn_graph(batch_features, k=k, batch=None, loop=False)\n # Adjust node indices for the batch offset\n batch_edges += start\n edges.append(batch_edges)\n return torch.cat(edges, dim=1)\n```\n\n***Note***\n* I introduced our method during my job interview, and they were very satisfied with my introduction. Hope can find a data scientist job in Taiwan next week!\n* I also explain this method to my fellow in the military who don't know about data science. He was shocked to the perspective of how I see the data.\n* Want to say sorry to my love @daphne4sg I'll take you to a trip and put my concentration on you instead of the competition.\n\nBest regards, Tom Huang. ", "votes": 38 }, { "id": 3145550, "postDate": "2025-03-10T01:03:32.057Z", "content": "Great job Tom! I love how your team made a strong GNN model!
", "rawMarkdown": "Great job Tom! I love how your team made a strong GNN model!", "votes": 1, "replies": [ { "id": 3145840, "postDate": "2025-03-10T09:53:25.997Z", "content": "@cdeotte Thank you chris! You also provide a lot of good material in this comp. Maybe next time we can solve the problem together!
", "rawMarkdown": "@cdeotte Thank you chris! You also provide a lot of good material in this comp. Maybe next time we can solve the problem together!", "votes": 1 } ] }, { "id": 3141931, "postDate": "2025-03-06T01:40:23.410Z", "content": "Thanks for your amazing solution! Could I ask what are the inputs and outputs when using Coxph as stack meta model?
", "rawMarkdown": "Thanks for your amazing solution! Could I ask what are the inputs and outputs when using Coxph as stack meta model?", "votes": 1, "replies": [ { "id": 3141935, "postDate": "2025-03-06T01:45:48.830Z", "content": "Assume you have a predicitons from other base models named data.
(): \n\n \n\n cv, oof_preds = ._prepare_cv()\n\n train_index, valid_index cv.split(data):\n\n train_data = data.iloc[train_index]\n valid_data = data.iloc[valid_index]\n\n \n train_data = train_data.loc[:, train_data.nunique() > ]\n valid_data = valid_data[train_data.columns]\n\n cph = CoxPHFitter(penalizer=._penalizer)\n cph.fit(train_data, duration_col=, event_col=)\n\n oof_preds[valid_index] = cph.predict_partial_hazard(valid_data) \n .data[] = oof_preds\n .validate_model(oof_preds, ) \n\n .data\n
Thanks for the mention @tom99763 . These past three months have been an incredible learning experience with the team, and I got to learn a lot from you all - both in terms of knowledge and work ethic.
", "rawMarkdown": "Thanks for the mention @tom99763 . These past three months have been an incredible learning experience with the team, and I got to learn a lot from you all - both in terms of knowledge and work ethic. ", "votes": 1, "replies": [ { "id": 3141940, "postDate": "2025-03-06T01:48:02.733Z", "content": "@introverstein Thank you for being my teammate. I aslo learn a lot from you.
", "rawMarkdown": "@introverstein Thank you for being my teammate. I aslo learn a lot from you. ", "votes": 1 } ] }, { "id": 3141912, "postDate": "2025-03-06T01:21:54.727Z", "content": "Congrats, @tom99763.
", "rawMarkdown": "Congrats, @tom99763.", "votes": 1, "replies": [ { "id": 3141949, "postDate": "2025-03-06T01:53:54.070Z", "content": "@abdmental01 Thank you! I use your codebase also. Great implementation.
", "rawMarkdown": "@abdmental01 Thank you! I use your codebase also. Great implementation. " } ] }, { "id": 3141907, "postDate": "2025-03-06T01:20:06.163Z", "content": "Man, u are just nuts, in a positive way of course. Congratulations!
", "rawMarkdown": "Man, u are just nuts, in a positive way of course. Congratulations!", "votes": 1, "replies": [ { "id": 3141943, "postDate": "2025-03-06T01:53:01.100Z", "content": "@eikyou Thank you!
", "rawMarkdown": "@eikyou Thank you!" } ] }, { "id": 3141900, "postDate": "2025-03-06T01:16:15.293Z", "content": "Great effort and great result @tom99763
\nThanks for the mambular reference also
@ravi20076 Thank you! See you in another competition.
", "rawMarkdown": "@ravi20076 Thank you! See you in another competition." } ] }, { "id": 3142454, "postDate": "2025-03-06T12:02:41.547Z", "content": "@tom99763 ,Thanks for the mention.
\n It was an honor to be part of such an amazing team.
\nOur team leader, @tom99763 , led us with incredible dedication from start to finish. Honestly, I have no idea when he even sleeps.
\nMy teammates, @introverstein and @daphne4sg ,did an outstanding job—precise, reliable, and creative.
\nI have so much respect for my teammates, and I will continue striving to be like them.
\nThank you!
@yuogawa You hardly try a lot of ideas. Although we miss a lot of improvement directions, we cannot get this place without your effort. Maybe next time we can team up together again.
", "rawMarkdown": "@yuogawa You hardly try a lot of ideas. Although we miss a lot of improvement directions, we cannot get this place without your effort. Maybe next time we can team up together again. " } ] }, { "id": 3143476, "postDate": "2025-03-07T09:10:47.447Z", "content": "Hi Tom, I was competing whilst on holiday and thought that was bad, but competing from military service from a phone is insane (and you got a great position!). I am very interested in learning more about GNNs and will definitely use your implementation to do so. If you had any other recommendations on reading material or videos you used to learn about them, that would be great! Congrats again!
", "rawMarkdown": "Hi Tom, I was competing whilst on holiday and thought that was bad, but competing from military service from a phone is insane (and you got a great position!). I am very interested in learning more about GNNs and will definitely use your implementation to do so. If you had any other recommendations on reading material or videos you used to learn about them, that would be great! Congrats again!", "replies": [ { "id": 3143534, "postDate": "2025-03-07T10:07:24.823Z", "content": "@noodl35 You can learn from Stanford Machine Learning with Graph . See the final projects made by their students. Graph is usually used for recommender system. But if your data is structural, feel free to try it.
", "rawMarkdown": "@noodl35 You can learn from [Stanford Machine Learning with Graph ](https://www.youtube.com/watch?v=JAB_plj2rbA&list=PLoROMvodv4rPLKxIpqhjhPgdQy7imNkDn&ab_channel=StanfordOnline). See the final projects made by their students. Graph is usually used for recommender system. But if your data is structural, feel free to try it. " } ] }, { "id": 3142689, "postDate": "2025-03-06T15:51:58.960Z", "content": "@tom99763 Congrats for your team's work and it's impressive you managed to serve in military, have a girlfriend and participate in this competition from a smartphone at the same time period! That's something to look up to!
\nI have one question with your solution:
\n\n\nUsing Coxph as stack meta model fitted the predictions and treat its prediction as targets (Knowledge Distillation).
\n
From my understanding, did you use CoxPH as (regression) target for the GNN?
", "rawMarkdown": "@tom99763 Congrats for your team's work and it's impressive you managed to serve in military, have a girlfriend and participate in this competition from a smartphone at the same time period! That's something to look up to!\n\nI have one question with your solution:\n\n> Using Coxph as stack meta model fitted the predictions and treat its prediction as targets (Knowledge Distillation).\n\nFrom my understanding, did you use CoxPH as (regression) target for the GNN?", "replies": [ { "id": 3143065, "postDate": "2025-03-06T22:46:18.283Z", "content": "@andreasbis In your public notebook, you feed raw data to coxph, but I feed predictions from base models . I found this method outperforms other targets for meta model.
", "rawMarkdown": "@andreasbis In your public notebook, you feed raw data to coxph, but I feed predictions from base models . I found this method outperforms other targets for meta model.", "votes": 1, "replies": [ { "id": 3143296, "postDate": "2025-03-07T04:49:26.183Z", "content": "@tom99763 how did you get thus target engineering approach to work? When I tried this approach I was getting values up to 600B and the GBDT models that use this target were random guessing.
", "rawMarkdown": "@tom99763 how did you get thus target engineering approach to work? When I tried this approach I was getting values up to 600B and the GBDT models that use this target were random guessing." }, { "id": 3143301, "postDate": "2025-03-07T04:55:05.587Z", "content": "@andreasbis Here is an example. I have made it public. https://www.kaggle.com/code/tom99763/graphsage-stack-ensemble-cox-target-augment
", "rawMarkdown": "@andreasbis Here is an example. I have made it public. https://www.kaggle.com/code/tom99763/graphsage-stack-ensemble-cox-target-augment", "votes": 1 } ] } ] }, { "id": 3141939, "postDate": "2025-03-06T01:47:57.367Z", "content": "In your own words, how is Mambular doing?
", "rawMarkdown": "In your own words, how is Mambular doing?", "replies": [ { "id": 3141942, "postDate": "2025-03-06T01:49:30.067Z", "content": "@yekenot It provides ResNet, TabM, FTTransformer and Saint models for training tabular data with their preprocessing method.
", "rawMarkdown": "@yekenot It provides ResNet, TabM, FTTransformer and Saint models for training tabular data with their preprocessing method. ", "votes": 1 } ] }, { "id": 3142478, "postDate": "2025-03-06T12:29:46.570Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:46:02Z", "competition_id": 70942, "competition_slug": "equity-post-HCT-survival-predictions", "team_id": 13064779, "solution_url": "/competitions/equity-post-HCT-survival-predictions/writeups/i-back-to-save-our-team-104th-solution-gnn" }