# 104th solution (GNN) First of all, I want to thank to my teammates [@introverstein](https://www.kaggle.com/introverstein) , [@yuogawa](https://www.kaggle.com/yuogawa) and my love [@daphne4sg](https://www.kaggle.com/daphne4sg) who put a lot of effort and support me. This is the first time I paricipant a competititon for totally three month. It was hard because I was in the Taiwan military service for 4 months. They only allow me to use my cellphone so I write all my codes on my cellphone, typing code using my finger is furstrated. During this competition, I usually have quarrel with my girlfriend [@daphne4sg](https://www.kaggle.com/daphne4sg) because I spend all day long on coding and didn't talk to her and relax. So I made a promise that if we are in top 5 then I'll take you to Japan (maybe visit my teammate [@yuogawa](https://www.kaggle.com/yuogawa) ). It turns out to be 108th place. That's fine, "Failure is the mother of success". My solution One day I attempted GNN on raw data and it didin't show good results. I create edges purely based on the categorical features. Here is an example using race_group: (): race_groups = .data.groupby()[].apply() edges = [] group tqdm(race_groups): edges.extend(combinations(group, )) edge_index = torch.tensor(edges, dtype=torch.long).T edge_index The moment I realize that GNN might works is seeing [@andreasbis](https://www.kaggle.com/andreasbis) public codebase. He hardly tried a lot of targets and model combinations. So I'm thinkg about how to use GNN to aggregate them. What if using GNN as stack meta model instead of base model? The edge connection is now built based on the similarity between the model predictions. We don't need to worry about using a lot of categorical features to build edges. It turns out a very good model. (The full version is using the predicitons from [@andreasbis](https://www.kaggle.com/andreasbis) codebase, PRL NN, Yunbase and Autogluon to train GNN) model public private cv GBDTs 0.690 0.689 0.6805 GNN 0.693 0.693 0.6830 GNN (Full) 0.694 0.693 0.6911 ![Concept of GNN](assets/4017cc8735866dedcfc5e24b.png) I think it works because predictions implicitly indicate the property of each data point and hidden relationship between them. Obviously better than categorical-based edge connection. Here are the details of our solution: To reduce the computation complexity of building edges (a graph with 28800 nodes), we use RandomNodeLoader to allow the Graphsage training on large graph without OOM. Building edges using knn_graph is better than similarity thresholding. Knowledge Distillation from CoxPH Fit CoxPH using predictions base models. Train GNN using Cox Target. The final solution of our team using quantile, coxph, NAF, Race-wise KMF. Among them, quantile and coxph give very good performance for training the stack meta model. Using the prediction from label propogation to mask PRL NN race-wisely as it also indicates the determination of your graph structure. (It did better than XGBoost) propagation = LabelPropagation(num_layers = , alpha = ) y = np.array(.y.copy()) y = np.concatenate([y, * np.ones(X_test.shape[],)], axis=) y[valid_index] = y[test_index] = y_tensor = torch.tensor(y, dtype=torch.long) mask = y_tensor.clone() != efs_preds = propagation(y_tensor, edge_index, mask=mask) test_efs = torch.argmax(efs_preds, dim=).numpy() races = df_sub.race_group.unique() shift = [, -, , , ,] i, race (races): prl_preds[(test_efs==)&(test.race_group == race)] += shift[i] ![Label propogation](assets/7506ccac33197b79ed2454aa.png) Continue to LabelPropogation., the first method I used is the eliminiation I mention [before](https://www.kaggle.com/competitions/equity-post-HCT-survival-predictions/discussion/561146#3115494) . I use the disagreement between aux_pred from PRL NN and prediction from LabelPropogation to determine whether to eliminate the decision of PRL NN as I observe the loss function giving bias on the race. So my hypothesis is that the graph structure is more convicing than the PRL NN. ![elminiate](assets/a0cee89d2893c159c389c5e7.png) I also add ResNet, TabM, FTTransformer, Saint in @@andreasbis codebase to increase the target diversity, which are trained with Coxph targets. And let me introduce the library I use [Mambular](https://mambular.readthedocs.io/en/latest/index.html) . Welcome to use my pip install Mambular [notebook](https://www.kaggle.com/code/tom99763/pip-install-mambular) Implementations Model: (nn.Module): (): ().__init__() .graphsage = GraphSAGE(in_channels, hidden_channels, , out_channels, jk=, norm = LayerNorm(hidden_channels), dropout = ) (): x, edge_index = data.x, data.edge_index .training: edge_index = graph_augmentation(data) edge_index.shape[]==: edge_index = torch.tensor([[, ]]).T.cuda() x = x.cuda() edge_index = edge_index.cuda() logits = .graphsage(x, edge_index) logits (): edge_index, _ = dropout_edge(data.edge_index, p=p, training=) num_edges = edge_index.size() perm = torch.randperm(num_edges)[:num_edges // ] edge_index[:, perm] = torch.randint(, data.num_nodes, edge_index[:, perm].shape) edge_index Training: mambular.preprocessing.preprocessor Preprocessor torch_geometric.nn.models GraphSAGE, LabelPropagation torch_geometric.loader RandomNodeLoader torch.optim.lr_scheduler ExponentialLR torch.utils.data DataLoader, TensorDataset (): X = pd.DataFrame(np.vstack(oof_model_preds).T) y = .data[target] models, pps, fold_scores = [], [], [] cv, oof_preds = .targets._prepare_cv() fold, (train_index, valid_index) (cv.split(X, y)): X_train = X.iloc[train_index] X_valid = X.iloc[valid_index] y_train = y.iloc[train_index] y_valid = y.iloc[valid_index] () pp = Preprocessor() X_train_dict = pp.fit_transform(X_train, y_train) X_valid_dict = pp.transform(X_valid) X_train = pd.DataFrame() k,v X_train_dict.items(): i (v.shape[]): X_train[] = v[:, i] X_valid = pd.DataFrame() k,v X_valid_dict.items(): i (v.shape[]): X_valid[] = v[:, i] set_seed() best_model, best_score, best_oof_preds = .train_fold( X_train, y_train, X_valid, y_valid, oof_preds, train_index, valid_index) oof_preds = best_oof_preds models.append(best_model) pps.append(pp) fold_scores.append(best_score) .eda._plot_cv(fold_scores, title) .targets.validate_model(oof_preds, title) models, pps, oof_preds (): X_train = torch.tensor(X_train.values, dtype=torch.float32) X_valid = torch.tensor(X_valid.values, dtype=torch.float32) y_train = torch.tensor(y_train.values, dtype=torch.float32).unsqueeze() y_valid = torch.tensor(y_valid.values, dtype=torch.float32).unsqueeze() X = torch.cat([torch.zeros_like(X_train), torch.zeros_like(X_valid)], dim=) y = torch.cat([torch.zeros_like(y_train), torch.zeros_like(y_valid)], dim=) X[train_index] = X_train X[valid_index] = X_valid y[train_index] = y_train y[valid_index] = y_valid train_mask = torch.zeros((X), dtype=torch.) valid_mask = torch.zeros((X), dtype=torch.) train_mask[train_index] = valid_mask[valid_index] = () edge_index = .build_edges(X, CFG.k) data = Data(x=X, edge_index=edge_index, y=y) data.train_mask = train_mask data.test_mask = valid_mask data.x = data.x.contiguous() data.edge_index = data.edge_index.contiguous() data.y = data.y.contiguous() data.train_mask = data.train_mask.contiguous() data.test_mask = data.test_mask.contiguous() () train_loader = RandomNodeLoader(data, num_parts = CFG.num_parts, shuffle=) valid_loader = RandomNodeLoader(data, num_parts = CFG.num_parts, shuffle=) params = { : X_train.shape[] } set_seed() model = GNN_Model(**params).cuda() criterion = nn.MSELoss() optimizer = optim.Adam( model.parameters(), lr=CFG.lr, weight_decay=CFG.weight_decay ) scheduler = ExponentialLR(optimizer, gamma=) best_score = best_oof_preds = oof_preds best_model = patience = epoch (CFG.epochs): model.train() loop = tqdm(train_loader) o=[] batch loop: y = batch.y.cuda() out = model(batch) loss = criterion(out[[batch.train_mask]], y[[batch.train_mask]]) optimizer.zero_grad() loss.backward() optimizer.step() loop.set_postfix(loss=) scheduler.step() model.() oof_preds_ = [] torch.no_grad(): batch valid_loader: out = model(batch).cpu() oof_preds_.append(out[batch.test_mask][:, ]) oof_preds_ = torch.cat(oof_preds_, dim=) oof_preds[valid_index] = oof_preds_.numpy() y_true_fold = .data.iloc[valid_index][[, , , ]].copy() y_pred_fold = .data.iloc[valid_index][[]].copy() y_pred_fold[] = oof_preds[valid_index] fold_score = score(y_true_fold, y_pred_fold, ) fold_score > best_score: best_score = fold_score best_oof_preds best_oof_preds = copy(oof_preds) best_model best_model = deepcopy(model) patience = : patience += () patience >= CFG.patience: best_model, best_score, best_oof_preds (): num_samples = features.shape[] edges = [] start tqdm((, num_samples, batch_size)): end = (start + batch_size, num_samples) batch_features = features[start:end] batch_edges = knn_graph(batch_features, k=k, batch=, loop=) batch_edges += start edges.append(batch_edges) torch.cat(edges, dim=) Note I introduced our method during my job interview, and they were very satisfied with my introduction. Hope can find a data scientist job in Taiwan next week! I also explain this method to my fellow in the military who don't know about data science. He was shocked to the perspective of how I see the data. Want to say sorry to my love [@daphne4sg](https://www.kaggle.com/daphne4sg) I'll take you to a trip and put my concentration on you instead of the competition. Best regards, Tom Huang.