{
  "id": 90350,
  "title": "BERT -6 layer solution [LB 0.23101]",
  "content": "<p>My solution was based on <a href=\"https://www.kaggle.com/mateiionita/taming-the-bert-a-baseline\">Matei's kernel</a> and <a href=\"https://www.aclweb.org/anthology/D17-1018\">End-to-end paper</a>. \nMy model is a simple BERT(no fine-tuning) + FFNN model, and the FFNN structure is from this End to End paper, out of the box.</p>\n\n<p>Feature:\nWord vectors of A, B and Pronoun (from BERT large layer -6);\nDistance between P and A/B;</p>\n\n<p>Argumentation:\nSwapping A and B, I got 0.006 improvement.</p>\n\n<p>Hyper parameter:\nAdam optimizer learning rate=0.0004;\nFFNN hidden dimensions = 40;\ndrop_rate=0.65;\nbatchsize = 16;\n10 folds cross-validation;</p>\n\n<p>Thanks for your reading.</p>",
  "comments": [],
  "comment_count": 0,
  "messages": [
    {
      "id": 521546,
      "postDate": "2019-04-23T03:02:00.460Z",
      "content": "<p>My solution was based on <a href=\"https://www.kaggle.com/mateiionita/taming-the-bert-a-baseline\">Matei's kernel</a> and <a href=\"https://www.aclweb.org/anthology/D17-1018\">End-to-end paper</a>. \nMy model is a simple BERT(no fine-tuning) + FFNN model, and the FFNN structure is from this End to End paper, out of the box.</p>\n\n<p>Feature:\nWord vectors of A, B and Pronoun (from BERT large layer -6);\nDistance between P and A/B;</p>\n\n<p>Argumentation:\nSwapping A and B, I got 0.006 improvement.</p>\n\n<p>Hyper parameter:\nAdam optimizer learning rate=0.0004;\nFFNN hidden dimensions = 40;\ndrop_rate=0.65;\nbatchsize = 16;\n10 folds cross-validation;</p>\n\n<p>Thanks for your reading.</p>",
      "rawMarkdown": "My solution was based on [Matei's kernel](https://www.kaggle.com/mateiionita/taming-the-bert-a-baseline) and [End-to-end paper](https://www.aclweb.org/anthology/D17-1018). \nMy model is a simple BERT(no fine-tuning) + FFNN model, and the FFNN structure is from this End to End paper, out of the box.\n\nFeature:\nWord vectors of A, B and Pronoun (from BERT large layer -6);\nDistance between P and A/B;\n\nArgumentation:\nSwapping A and B, I got 0.006 improvement.\n\nHyper parameter:\nAdam optimizer learning rate=0.0004;\nFFNN hidden dimensions = 40;\ndrop_rate=0.65;\nbatchsize = 16;\n10 folds cross-validation;\n\nThanks for your reading.",
      "votes": 2
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T22:44:44Z",
  "competition_id": 12797,
  "competition_slug": "gendered-pronoun-resolution",
  "team_id": 2840509,
  "solution_url": "/competitions/gendered-pronoun-resolution/writeups/all-0-33-baseline-bert-6-layer-solution-lb-0-23101"
}