Game of Life forwards can be solved with only 10 neurons and 13 weights!
\nThis is the actual minimum network architecture and weights. It passed a 10,000 board unit test.
https://www.kaggle.com/jamesmcguigan/its-easy-for-neural-networks-to-learn-game-of-life
\nepoch: 6900 | board_count: 3450000 | loss: 0.0246548884 | accuracy = 1.0000000000 | time: 0.974ms/board | 11m 23s \nFinished Training: GameOfLifeForward_11N - 6945 epochs in 687.9s\nGameOfLifeForward_11N.savefile(): models/GameOfLifeForward_11N.pth = 2.2 kB\n--------------------\nGameOfLifeForward_11N\nGameOfLifeForward_11N(\n (criterion): MSELoss()\n (layers): ModuleList(\n (0): Conv2d(1, 1, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), padding_mode=circular)\n (1): Conv2d(1, 1, kernel_size=(1, 1), stride=(1, 1))\n )\n (dropout): Dropout(p=0.0, inplace=False)\n (activation): PReLU(num_parameters=1)\n)\nactivation.weight [-0.6601397]\n\nlayers.0.weight\n[[[[-0.8748493 -0.87457496 -0.8745291 ]\n [-0.8742993 -0.6293647 -0.8741275 ]\n [-0.8746812 -0.87455815 -0.874589 ]]]]\nlayers.0.bias [2.6234446]\n\nlayers.1.weight [[[[-3.8328729]]]]\nlayers.1.bias [1.6233934]\nThis also acts as my response to the recent paper:
\nIt's Hard for Neural Networks To Learn the Game of Life
\nby Jacob M. Springer, Garrett T. Kenyon
\n\nThis notebook is part of series exploring the Neural Network implementations of the Game of Life Foward Problem
\nGame of Life forwards can be solved with only 10 neurons and 13 weights!
\nThis is the actual minimum network architecture and weights. It passed a 10,000 board unit test.
https://www.kaggle.com/jamesmcguigan/its-easy-for-neural-networks-to-learn-game-of-life
\nepoch: 6900 | board_count: 3450000 | loss: 0.0246548884 | accuracy = 1.0000000000 | time: 0.974ms/board | 11m 23s \nFinished Training: GameOfLifeForward_11N - 6945 epochs in 687.9s\nGameOfLifeForward_11N.savefile(): models/GameOfLifeForward_11N.pth = 2.2 kB\n--------------------\nGameOfLifeForward_11N\nGameOfLifeForward_11N(\n (criterion): MSELoss()\n (layers): ModuleList(\n (0): Conv2d(1, 1, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), padding_mode=circular)\n (1): Conv2d(1, 1, kernel_size=(1, 1), stride=(1, 1))\n )\n (dropout): Dropout(p=0.0, inplace=False)\n (activation): PReLU(num_parameters=1)\n)\nactivation.weight [-0.6601397]\n\nlayers.0.weight\n[[[[-0.8748493 -0.87457496 -0.8745291 ]\n [-0.8742993 -0.6293647 -0.8741275 ]\n [-0.8746812 -0.87455815 -0.874589 ]]]]\nlayers.0.bias [2.6234446]\n\nlayers.1.weight [[[[-3.8328729]]]]\nlayers.1.bias [1.6233934]\nThis also acts as my response to the recent paper:
\nIt's Hard for Neural Networks To Learn the Game of Life
\nby Jacob M. Springer, Garrett T. Kenyon
\n\nThis notebook is part of series exploring the Neural Network implementations of the Game of Life Foward Problem
\nJames, the paper was claiming that it's easy to hand-craft a small neural network that can perfectly implement the Game of Life (in the forwards direction), but that standard training algorithms can't reliably find that solution. Which doesn't seem inconsistent with what you've found?
", "rawMarkdown": "James, the paper was claiming that it's easy to hand-craft a small neural network that can perfectly implement the Game of Life (in the forwards direction), but that standard training algorithms can't reliably find that solution. Which doesn't seem inconsistent with what you've found?", "votes": null }, { "id": "1043355", "postDate": "10/08/2020 22:36:23", "content": "The paper inspired me to experiment with miniature architectures and figure out how to handcode neural network weights.
\n\nThe neural networks in my response notebook were all trained from random weights.
\n\nWhat I did different was to use the PReLU activation function rather than ReLU. This has a trainable weight that allows it to form different shapes, such as with a -0.5 weight creating a V shape (mimicking a support vector machine), rather than the L shape of ReLU or the S shape of Sigmoid/Tanh. This greatly improved the ability of the network to converge to 100% accuracy.
\nI did encounter some aspect of lottery ticket initialization but was getting training success rates between 40-80-100% depending on the size of the network. \"4(3x3) + 4 + 1\" was large enough to always train 100% successfully but \"1(3x3) + 2 + 1\" only had a 40% success rate (which is better than lottery odds).
\nI discovered another new thing, that if you reduce the network size even smaller still, down to 10 neurons \"3x3 + 1\", then there is only one mathematically valid solution, and the training success rate jumps back upto 90%. This is compared to \"1*(3x3) + 2 + 1\" with only a 40% success rate, but reports 3 different solution designs to the network weights (different signs or relative weights).
", "rawMarkdown": "The paper inspired me to experiment with miniature architectures and figure out how to handcode neural network weights.\n- https://www.kaggle.com/jamesmcguigan/pytorch-game-of-life-hardcoding-network-weights/\n\nThe neural networks in my response notebook were all trained from random weights. \n- https://www.kaggle.com/jamesmcguigan/its-easy-for-neural-networks-to-learn-game-of-life\n\nWhat I did different was to use the PReLU activation function rather than ReLU. This has a trainable weight that allows it to form different shapes, such as with a -0.5 weight creating a V shape (mimicking a support vector machine), rather than the L shape of ReLU or the S shape of Sigmoid/Tanh. This greatly improved the ability of the network to converge to 100% accuracy.\n\nI did encounter some aspect of lottery ticket initialization but was getting training success rates between 40-80-100% depending on the size of the network. \"4*(3x3) + 4 + 1\" was large enough to always train 100% successfully but \"1*(3x3) + 2 + 1\" only had a 40% success rate (which is better than lottery odds).\n\nI discovered another new thing, that if you reduce the network size even smaller still, down to 10 neurons \"3x3 + 1\", then there is only one mathematically valid solution, and the training success rate jumps back upto 90%. This is compared to \"1*(3x3) + 2 + 1\" with only a 40% success rate, but reports 3 different solution designs to the network weights (different signs or relative weights).", "votes": null }, { "id": "1043367", "postDate": "10/08/2020 22:53:23", "content": "I see - thanks for the detailed reply.
", "rawMarkdown": "I see - thanks for the detailed reply.", "votes": null } ], "comments": [ { "id": 1043249, "author_name": "johnlucas0", "author_url": "", "post_date": "10/08/2020 19:54:07", "content": "James, the paper was claiming that it's easy to hand-craft a small neural network that can perfectly implement the Game of Life (in the forwards direction), but that standard training algorithms can't reliably find that solution. Which doesn't seem inconsistent with what you've found?
", "votes": null, "replies": [ { "id": 1043355, "author_name": "jamesmcguigan", "author_url": "", "post_date": "10/08/2020 22:36:23", "content": "The paper inspired me to experiment with miniature architectures and figure out how to handcode neural network weights.
\n\nThe neural networks in my response notebook were all trained from random weights.
\n\nWhat I did different was to use the PReLU activation function rather than ReLU. This has a trainable weight that allows it to form different shapes, such as with a -0.5 weight creating a V shape (mimicking a support vector machine), rather than the L shape of ReLU or the S shape of Sigmoid/Tanh. This greatly improved the ability of the network to converge to 100% accuracy.
\nI did encounter some aspect of lottery ticket initialization but was getting training success rates between 40-80-100% depending on the size of the network. \"4(3x3) + 4 + 1\" was large enough to always train 100% successfully but \"1(3x3) + 2 + 1\" only had a 40% success rate (which is better than lottery odds).
\nI discovered another new thing, that if you reduce the network size even smaller still, down to 10 neurons \"3x3 + 1\", then there is only one mathematically valid solution, and the training success rate jumps back upto 90%. This is compared to \"1*(3x3) + 2 + 1\" with only a 40% success rate, but reports 3 different solution designs to the network weights (different signs or relative weights).
", "votes": null, "replies": [] }, { "id": 1043367, "author_name": "johnlucas0", "author_url": "", "post_date": "10/08/2020 22:53:23", "content": "I see - thanks for the detailed reply.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1043005": "Game of Life forwards can be solved with only 10 neurons and 13 weights!\nThis is the actual minimum network architecture and weights. It passed a 10,000 board unit test.\n\nhttps://www.kaggle.com/jamesmcguigan/its-easy-for-neural-networks-to-learn-game-of-life\n\n```\nepoch: 6900 | board_count: 3450000 | loss: 0.0246548884 | accuracy = 1.0000000000 | time: 0.974ms/board | 11m 23s \nFinished Training: GameOfLifeForward_11N - 6945 epochs in 687.9s\nGameOfLifeForward_11N.savefile(): models/GameOfLifeForward_11N.pth = 2.2 kB\n--------------------\nGameOfLifeForward_11N\nGameOfLifeForward_11N(\n (criterion): MSELoss()\n (layers): ModuleList(\n (0): Conv2d(1, 1, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), padding_mode=circular)\n (1): Conv2d(1, 1, kernel_size=(1, 1), stride=(1, 1))\n )\n (dropout): Dropout(p=0.0, inplace=False)\n (activation): PReLU(num_parameters=1)\n)\nactivation.weight [-0.6601397]\n\nlayers.0.weight\n[[[[-0.8748493 -0.87457496 -0.8745291 ]\n [-0.8742993 -0.6293647 -0.8741275 ]\n [-0.8746812 -0.87455815 -0.874589 ]]]]\nlayers.0.bias [2.6234446]\n\nlayers.1.weight [[[[-3.8328729]]]]\nlayers.1.bias [1.6233934]\n```\n\nThis also acts as my response to the recent paper:\n\n**It's Hard for Neural Networks To Learn the Game of Life** \n\nby Jacob M. Springer, Garrett T. Kenyon\n- https://arxiv.org/abs/2009.01398\n\nThis notebook is part of series exploring the Neural Network implementations of the Game of Life Foward Problem\n- [Pytorch Game of Life - First Attempt](https://www.kaggle.com/jamesmcguigan/pytorch-game-of-life-first-attempt)\n- [Pytorch Game of Life - Hardcoding Network Weights](https://www.kaggle.com/jamesmcguigan/pytorch-game-of-life-hardcoding-network-weights)\n- [Its Easy for Neural Networks To Learn Game of Life](https://www.kaggle.com/jamesmcguigan/its-easy-for-neural-networks-to-learn-game-of-life)", "1043249": "James, the paper was claiming that it's easy to hand-craft a small neural network that can perfectly implement the Game of Life (in the forwards direction), but that standard training algorithms can't reliably find that solution. Which doesn't seem inconsistent with what you've found?", "1043355": "The paper inspired me to experiment with miniature architectures and figure out how to handcode neural network weights.\n- https://www.kaggle.com/jamesmcguigan/pytorch-game-of-life-hardcoding-network-weights/\n\nThe neural networks in my response notebook were all trained from random weights. \n- https://www.kaggle.com/jamesmcguigan/its-easy-for-neural-networks-to-learn-game-of-life\n\nWhat I did different was to use the PReLU activation function rather than ReLU. This has a trainable weight that allows it to form different shapes, such as with a -0.5 weight creating a V shape (mimicking a support vector machine), rather than the L shape of ReLU or the S shape of Sigmoid/Tanh. This greatly improved the ability of the network to converge to 100% accuracy.\n\nI did encounter some aspect of lottery ticket initialization but was getting training success rates between 40-80-100% depending on the size of the network. \"4*(3x3) + 4 + 1\" was large enough to always train 100% successfully but \"1*(3x3) + 2 + 1\" only had a 40% success rate (which is better than lottery odds).\n\nI discovered another new thing, that if you reduce the network size even smaller still, down to 10 neurons \"3x3 + 1\", then there is only one mathematically valid solution, and the training success rate jumps back upto 90%. This is compared to \"1*(3x3) + 2 + 1\" with only a 40% success rate, but reports 3 different solution designs to the network weights (different signs or relative weights).", "1043367": "I see - thanks for the detailed reply." }, "source": "meta" }