{
  "id": 239336,
  "title": "Replacing Bert Self Attention with Fourier Transform",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/239336",
  "author_name": "",
  "post_date": "2021-05-15T20:25:18.276506Z",
  "votes": 3,
  "comment_count": 2,
  "views": 0,
  "content": "<p><a href=\"https://arxiv.org/pdf/2105.03824.pdf\" target=\"_blank\">Paper -- FNet: Mixing Tokens with Fourier Transforms</a></p>\n<p>FNet is a layer normalized ResNet architecture with multiple layers, each of which consists of a Fourier mixing sublayer followed by a feedforward sublayer. The team replaces the self-attention sublayer of each transformer encoder layer with a Fourier Transform sublayer. They apply 1D Fourier Transforms along both the sequence dimension and the hidden dimension. The result is a complex number that can be written as a real number multiplied by the imaginary unit (the number “i” in mathematics, which enables solving equations that do not have real number solutions). Only the result’s real number is kept, eliminating the need to modify the (nonlinear) feedforward sublayers or output layers to handle complex numbers.</p>\n<p><img src=\"https://i0.wp.com/syncedreview.com/wp-content/uploads/2021/05/image-49.png?w=591&amp;ssl=1\" alt=\"FNet Architecture\"></p>",
  "messages": [
    {
      "id": "1309326",
      "postDate": "05/15/2021 20:25:18",
      "content": "<p><a href=\"https://arxiv.org/pdf/2105.03824.pdf\" target=\"_blank\">Paper -- FNet: Mixing Tokens with Fourier Transforms</a></p>\n<p>FNet is a layer normalized ResNet architecture with multiple layers, each of which consists of a Fourier mixing sublayer followed by a feedforward sublayer. The team replaces the self-attention sublayer of each transformer encoder layer with a Fourier Transform sublayer. They apply 1D Fourier Transforms along both the sequence dimension and the hidden dimension. The result is a complex number that can be written as a real number multiplied by the imaginary unit (the number “i” in mathematics, which enables solving equations that do not have real number solutions). Only the result’s real number is kept, eliminating the need to modify the (nonlinear) feedforward sublayers or output layers to handle complex numbers.</p>\n<p><img src=\"https://i0.wp.com/syncedreview.com/wp-content/uploads/2021/05/image-49.png?w=591&amp;ssl=1\" alt=\"FNet Architecture\"></p>",
      "rawMarkdown": "[Paper -- FNet: Mixing Tokens with Fourier Transforms](https://arxiv.org/pdf/2105.03824.pdf)\n\nFNet is a layer normalized ResNet architecture with multiple layers, each of which consists of a Fourier mixing sublayer followed by a feedforward sublayer. The team replaces the self-attention sublayer of each transformer encoder layer with a Fourier Transform sublayer. They apply 1D Fourier Transforms along both the sequence dimension and the hidden dimension. The result is a complex number that can be written as a real number multiplied by the imaginary unit (the number “i” in mathematics, which enables solving equations that do not have real number solutions). Only the result’s real number is kept, eliminating the need to modify the (nonlinear) feedforward sublayers or output layers to handle complex numbers.\n\n![FNet Architecture](https://i0.wp.com/syncedreview.com/wp-content/uploads/2021/05/image-49.png?w=591&ssl=1)",
      "votes": null
    },
    {
      "id": "1309382",
      "postDate": "05/15/2021 22:18:23",
      "content": "<p>Interesting! What does keeping only the real part intuitively means? Isn't it similar to applying kernels in CNNs?</p>",
      "rawMarkdown": "Interesting! What does keeping only the real part intuitively means? Isn't it similar to applying kernels in CNNs?",
      "votes": null
    },
    {
      "id": "1314972",
      "postDate": "05/19/2021 13:26:25",
      "content": "<p>Interesting!!</p>",
      "rawMarkdown": "Interesting!!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1309382,
      "author_name": "shivammiglani",
      "author_url": "",
      "post_date": "05/15/2021 22:18:23",
      "content": "<p>Interesting! What does keeping only the real part intuitively means? Isn't it similar to applying kernels in CNNs?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1314972,
      "author_name": "udaygurugubelli",
      "author_url": "",
      "post_date": "05/19/2021 13:26:25",
      "content": "<p>Interesting!!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1309326": "[Paper -- FNet: Mixing Tokens with Fourier Transforms](https://arxiv.org/pdf/2105.03824.pdf)\n\nFNet is a layer normalized ResNet architecture with multiple layers, each of which consists of a Fourier mixing sublayer followed by a feedforward sublayer. The team replaces the self-attention sublayer of each transformer encoder layer with a Fourier Transform sublayer. They apply 1D Fourier Transforms along both the sequence dimension and the hidden dimension. The result is a complex number that can be written as a real number multiplied by the imaginary unit (the number “i” in mathematics, which enables solving equations that do not have real number solutions). Only the result’s real number is kept, eliminating the need to modify the (nonlinear) feedforward sublayers or output layers to handle complex numbers.\n\n![FNet Architecture](https://i0.wp.com/syncedreview.com/wp-content/uploads/2021/05/image-49.png?w=591&ssl=1)",
    "1309382": "Interesting! What does keeping only the real part intuitively means? Isn't it similar to applying kernels in CNNs?",
    "1314972": "Interesting!!"
  },
  "source": "meta"
}