I have implemented a conv net that predicts only one time step forward and then does this iteratively for the other delta predictions. My conv net is nothing super complicated but I am getting stuck on my accuracy. It looks like the top scorers are using hashing or some brute force method (perhaps…) I'm curious on how well anyone using a conv net is doing. I have a LB of 0.13086 which is middle of the pack. I have looked at notebooks ahead of me but it seems most of the better scores use a different method or are private. Just trying to learn and would love some open discussion without giving away any of your secrets to help me improve my understanding. Thanks! https://www.kaggle.com/robintwhite/iterative-cnn-with-forward-prediction-loss?scriptVersionId=46265988
", "messages": [ { "id": "1074730", "postDate": "11/11/2020 01:43:44", "content": "I have implemented a conv net that predicts only one time step forward and then does this iteratively for the other delta predictions. My conv net is nothing super complicated but I am getting stuck on my accuracy. It looks like the top scorers are using hashing or some brute force method (perhaps…) I'm curious on how well anyone using a conv net is doing. I have a LB of 0.13086 which is middle of the pack. I have looked at notebooks ahead of me but it seems most of the better scores use a different method or are private. Just trying to learn and would love some open discussion without giving away any of your secrets to help me improve my understanding. Thanks! https://www.kaggle.com/robintwhite/iterative-cnn-with-forward-prediction-loss?scriptVersionId=46265988
", "rawMarkdown": "I have implemented a conv net that predicts only one time step forward and then does this iteratively for the other delta predictions. My conv net is nothing super complicated but I am getting stuck on my accuracy. It looks like the top scorers are using hashing or some brute force method (perhaps...) I'm curious on how well anyone using a conv net is doing. I have a LB of 0.13086 which is middle of the pack. I have looked at notebooks ahead of me but it seems most of the better scores use a different method or are private. Just trying to learn and would love some open discussion without giving away any of your secrets to help me improve my understanding. Thanks! https://www.kaggle.com/robintwhite/iterative-cnn-with-forward-prediction-loss?scriptVersionId=46265988", "votes": null }, { "id": "1074767", "postDate": "11/11/2020 03:17:16", "content": "Hi @robintwhite,
\nI tried with CNN's, but I could not exceed 0.13, then I tried an autoencoder with CNN's and went down to 0.12. I think the problem is that CNN's bring non-linearity to each individual cell, but not to the neighborhood. Convolution is a linear operation on the neighborhood and the problem requires that the non-linearity be both in the cell and in the neighborhood. The convolutional layer could be changed to some layer that combines the convolution with some non-linearity in the neighborhood, although I can't think of how (it's just an idea that came to me while writing this).
\nI am working with genetic algorithms with a modification of adjusting the search areas in the neighborhoods where there are living cells. I have tried other heuristic optimization algorithms like PSO (among others), but genetic algorithms have given me better results. The problem with genetic algorithms is that the solutions are not exact.
\nThe other common approach is a SAT solver, which has the advantage that the solutions are exact, but may not find a solution. So it is possible not to find solutions to some boards.
\nWell there is also a brute force optimization, there is not much to say about this, but apparently it is what has delivered the best results.
\nIn any case, the problem with any optimization technique described above is the processing time (several days). Although it improves in each pass of the algorithm it is very little, but I will keep trying.
", "rawMarkdown": "Hi @robintwhite,\n\nI tried with CNN's, but I could not exceed 0.13, then I tried an autoencoder with CNN's and went down to 0.12. I think the problem is that CNN's bring non-linearity to each individual cell, but not to the neighborhood. Convolution is a linear operation on the neighborhood and the problem requires that the non-linearity be both in the cell and in the neighborhood. The convolutional layer could be changed to some layer that combines the convolution with some non-linearity in the neighborhood, although I can't think of how (it's just an idea that came to me while writing this).\n\nI am working with genetic algorithms with a modification of adjusting the search areas in the neighborhoods where there are living cells. I have tried other heuristic optimization algorithms like PSO (among others), but genetic algorithms have given me better results. The problem with genetic algorithms is that the solutions are not exact.\n\nThe other common approach is a SAT solver, which has the advantage that the solutions are exact, but may not find a solution. So it is possible not to find solutions to some boards.\n\nWell there is also a brute force optimization, there is not much to say about this, but apparently it is what has delivered the best results.\n\nIn any case, the problem with any optimization technique described above is the processing time (several days). Although it improves in each pass of the algorithm it is very little, but I will keep trying.", "votes": null }, { "id": "1076930", "postDate": "11/13/2020 03:38:15", "content": "Thanks very much for that info! Would a pooling layer give some level of non-linearity, is that part of the improvement in the autoencoder? Do you think a custom pooling layer would work, something other than max, or average as you were mentioning? How did you handle the image size, since it is a awkward odd number? I was considering trying this as well, just as you had for a next step but also trying skip connections in something like a U-net if that might help with any attention…I'm not sure, would like your thoughts on this.
\nYour genetic algorithm notebook is quite impressive.. and the score too! It is a lot to try and follow for me though, I also don't know R…
\nThis has been very informative for me, thank you again for your help
Yes, a pooling layer could work, I haven't tried it. A custom pooling is also a good idea, but how to do it? You have to have an idea of how the GoL behaves backwards, it could be with a probabilistic, should be tried.
\nActually, I did very basic tests with CNN's and I gave up on this method. I think now I'll get to do some research on this.
\nThere is a pretty good kernel with a 1 layer CNN, I used these weights (without retraining) for the output of the autoencoder.
\nThe idea of this autoencoder is to find the start state in intermediate layers using the stop states at the input and at the output, so in the last layer I use the weights that I mentioned.
\nTo adjust the size I used an layer_zero_padding_2d between a couple of layers, the problem is that it generated little variability in the edge weights, so I added it in the first layers, which improved the result a bit.
", "rawMarkdown": "Yes, a pooling layer could work, I haven't tried it. A custom pooling is also a good idea, but how to do it? You have to have an idea of how the GoL behaves backwards, it could be with a [probabilistic](https://www.kaggle.com/yakuben/crgl-probability-extension-true-target-problem), should be tried.\n\nActually, I did very basic tests with CNN's and I gave up on this method. I think now I'll get to do some research on this.\n\nThere is a pretty good kernel with a [1 layer CNN](https://www.kaggle.com/c/conways-reverse-game-of-life-2020/discussion/189764), I used these weights (without retraining) for the output of the autoencoder.\n\nThe idea of this autoencoder is to find the start state in intermediate layers using the stop states at the input and at the output, so in the last layer I use the weights that I mentioned.\n\nTo adjust the size I used an layer_zero_padding_2d between a couple of layers, the problem is that it generated little variability in the edge weights, so I added it in the first layers, which improved the result a bit.", "votes": null }, { "id": "1077741", "postDate": "11/13/2020 21:58:29", "content": "I have tried several different approaches to using neural networks. I once got a network that claimed to have achieved 99% accuracy after training overnight, but I forgot to checkin the model savefile and I was never able to reproduce it again (it may have been training on a 5x5 board).
\nThese are my various attempts, but so far I am not guessing better than sample_submission.csv
\nOn the plus side, I've completely solved the Game of Life Foward Problem using a neural network (using only 10 neurons), and have written a tutorial on how to hardcode neural network weights for minimalist architectures
\nHi @robintwhite,
\nI tried with CNN's, but I could not exceed 0.13, then I tried an autoencoder with CNN's and went down to 0.12. I think the problem is that CNN's bring non-linearity to each individual cell, but not to the neighborhood. Convolution is a linear operation on the neighborhood and the problem requires that the non-linearity be both in the cell and in the neighborhood. The convolutional layer could be changed to some layer that combines the convolution with some non-linearity in the neighborhood, although I can't think of how (it's just an idea that came to me while writing this).
\nI am working with genetic algorithms with a modification of adjusting the search areas in the neighborhoods where there are living cells. I have tried other heuristic optimization algorithms like PSO (among others), but genetic algorithms have given me better results. The problem with genetic algorithms is that the solutions are not exact.
\nThe other common approach is a SAT solver, which has the advantage that the solutions are exact, but may not find a solution. So it is possible not to find solutions to some boards.
\nWell there is also a brute force optimization, there is not much to say about this, but apparently it is what has delivered the best results.
\nIn any case, the problem with any optimization technique described above is the processing time (several days). Although it improves in each pass of the algorithm it is very little, but I will keep trying.
", "votes": null, "replies": [ { "id": 1076930, "author_name": "robintwhite", "author_url": "", "post_date": "11/13/2020 03:38:15", "content": "Thanks very much for that info! Would a pooling layer give some level of non-linearity, is that part of the improvement in the autoencoder? Do you think a custom pooling layer would work, something other than max, or average as you were mentioning? How did you handle the image size, since it is a awkward odd number? I was considering trying this as well, just as you had for a next step but also trying skip connections in something like a U-net if that might help with any attention…I'm not sure, would like your thoughts on this.
\nYour genetic algorithm notebook is quite impressive.. and the score too! It is a lot to try and follow for me though, I also don't know R…
\nThis has been very informative for me, thank you again for your help
Yes, a pooling layer could work, I haven't tried it. A custom pooling is also a good idea, but how to do it? You have to have an idea of how the GoL behaves backwards, it could be with a probabilistic, should be tried.
\nActually, I did very basic tests with CNN's and I gave up on this method. I think now I'll get to do some research on this.
\nThere is a pretty good kernel with a 1 layer CNN, I used these weights (without retraining) for the output of the autoencoder.
\nThe idea of this autoencoder is to find the start state in intermediate layers using the stop states at the input and at the output, so in the last layer I use the weights that I mentioned.
\nTo adjust the size I used an layer_zero_padding_2d between a couple of layers, the problem is that it generated little variability in the edge weights, so I added it in the first layers, which improved the result a bit.
", "votes": null, "replies": [] } ] }, { "id": 1077741, "author_name": "jamesmcguigan", "author_url": "", "post_date": "11/13/2020 21:58:29", "content": "I have tried several different approaches to using neural networks. I once got a network that claimed to have achieved 99% accuracy after training overnight, but I forgot to checkin the model savefile and I was never able to reproduce it again (it may have been training on a 5x5 board).
\nThese are my various attempts, but so far I am not guessing better than sample_submission.csv
\nOn the plus side, I've completely solved the Game of Life Foward Problem using a neural network (using only 10 neurons), and have written a tutorial on how to hardcode neural network weights for minimalist architectures
\n", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1074730": "I have implemented a conv net that predicts only one time step forward and then does this iteratively for the other delta predictions. My conv net is nothing super complicated but I am getting stuck on my accuracy. It looks like the top scorers are using hashing or some brute force method (perhaps...) I'm curious on how well anyone using a conv net is doing. I have a LB of 0.13086 which is middle of the pack. I have looked at notebooks ahead of me but it seems most of the better scores use a different method or are private. Just trying to learn and would love some open discussion without giving away any of your secrets to help me improve my understanding. Thanks! https://www.kaggle.com/robintwhite/iterative-cnn-with-forward-prediction-loss?scriptVersionId=46265988", "1074767": "Hi @robintwhite,\n\nI tried with CNN's, but I could not exceed 0.13, then I tried an autoencoder with CNN's and went down to 0.12. I think the problem is that CNN's bring non-linearity to each individual cell, but not to the neighborhood. Convolution is a linear operation on the neighborhood and the problem requires that the non-linearity be both in the cell and in the neighborhood. The convolutional layer could be changed to some layer that combines the convolution with some non-linearity in the neighborhood, although I can't think of how (it's just an idea that came to me while writing this).\n\nI am working with genetic algorithms with a modification of adjusting the search areas in the neighborhoods where there are living cells. I have tried other heuristic optimization algorithms like PSO (among others), but genetic algorithms have given me better results. The problem with genetic algorithms is that the solutions are not exact.\n\nThe other common approach is a SAT solver, which has the advantage that the solutions are exact, but may not find a solution. So it is possible not to find solutions to some boards.\n\nWell there is also a brute force optimization, there is not much to say about this, but apparently it is what has delivered the best results.\n\nIn any case, the problem with any optimization technique described above is the processing time (several days). Although it improves in each pass of the algorithm it is very little, but I will keep trying.", "1076930": "Thanks very much for that info! Would a pooling layer give some level of non-linearity, is that part of the improvement in the autoencoder? Do you think a custom pooling layer would work, something other than max, or average as you were mentioning? How did you handle the image size, since it is a awkward odd number? I was considering trying this as well, just as you had for a next step but also trying skip connections in something like a U-net if that might help with any attention...I'm not sure, would like your thoughts on this.\nYour genetic algorithm notebook is quite impressive.. and the score too! It is a lot to try and follow for me though, I also don't know R...\nThis has been very informative for me, thank you again for your help", "1077560": "Yes, a pooling layer could work, I haven't tried it. A custom pooling is also a good idea, but how to do it? You have to have an idea of how the GoL behaves backwards, it could be with a [probabilistic](https://www.kaggle.com/yakuben/crgl-probability-extension-true-target-problem), should be tried.\n\nActually, I did very basic tests with CNN's and I gave up on this method. I think now I'll get to do some research on this.\n\nThere is a pretty good kernel with a [1 layer CNN](https://www.kaggle.com/c/conways-reverse-game-of-life-2020/discussion/189764), I used these weights (without retraining) for the output of the autoencoder.\n\nThe idea of this autoencoder is to find the start state in intermediate layers using the stop states at the input and at the output, so in the last layer I use the weights that I mentioned.\n\nTo adjust the size I used an layer_zero_padding_2d between a couple of layers, the problem is that it generated little variability in the edge weights, so I added it in the first layers, which improved the result a bit.", "1077741": "I have tried several different approaches to using neural networks. I once got a network that claimed to have achieved 99% accuracy after training overnight, but I forgot to checkin the model savefile and I was never able to reproduce it again (it may have been training on a 5x5 board). \n\nThese are my various attempts, but so far I am not guessing better than sample_submission.csv\n- https://www.kaggle.com/jamesmcguigan/ouroboroslife-function-reversal-gan\n- https://www.kaggle.com/jamesmcguigan/reverse-game-of-life-inception-cnn\n- https://www.kaggle.com/jamesmcguigan/reverse-game-of-life-rnn\n\nOn the plus side, I've completely solved the Game of Life Foward Problem using a neural network (using only 10 neurons), and have written a tutorial on how to hardcode neural network weights for minimalist architectures \n- [Pytorch Game of Life - First Attempt](https://www.kaggle.com/jamesmcguigan/pytorch-game-of-life-first-attempt)\n- [Pytorch Game of Life - Hardcoding Network Weights](https://www.kaggle.com/jamesmcguigan/pytorch-game-of-life-hardcoding-network-weights)\n- [Its Easy for Neural Networks To Learn Game of Life](https://www.kaggle.com/jamesmcguigan/its-easy-for-neural-networks-to-learn-game-of-life)" }, "source": "meta" }