Hi there,
\na novice here.
\nI was just watching the competition and toying around with the dataset and it got me curious regarding the improvement of data-preparation.
\nA naive approach yielded an error-rate of around 0.17 so I thought it best to compute some more data for e.g. delta=1.
A very direct approach for the computation would run ~5-6h and not use the GPU.
\nI came up with the following approach which runs ~45-60min(if projections pan out) and utilizes ~13-15% of the GPU.
\nI understand that the bottle-neck here is computation of a cell's state in the next iteration.
\nIs there a way to treat everything as a matrix-/vector-operation and thus utilizing the GPU more gaining more speed?
import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom fastai.tabular.all import *\nfrom torch.nn import *\n\n\ndef calculate_all_masks(matrixWidth):\n columnCount = int(matrixWidth ** 2)\n result = torch.as_tensor(np.zeros((columnCount,columnCount), int), device=device)\n\n for position in range(0,columnCount):\n for y in [-1,0,1]:\n for z in [-1,0,1]:\n calculated_row = (position + columnCount +y*matrixWidth) % columnCount\n index_with_column_chosen = int(calculated_row/matrixWidth) * matrixWidth + (calculated_row + 1*z)%matrixWidth\n result[position][index_with_column_chosen] = 1\n\n return result\n\ndef calculate_gol_state_for_tensor_with_masks(start, masks, matrixWidth):\n columnCount = int(matrixWidth ** 2)\n result = torch.as_tensor(np.zeros((columnCount,), int), device=device)\n zero_vector = torch.as_tensor(np.zeros((columnCount,),int), device=device).float().unsqueeze(0)\n\n for current_cells in range(0, columnCount):\n result[current_cells] = int(\n math.ceil(\n torch.cdist((masks[current_cells] & start).float().unsqueeze(0), zero_vector)[0].item() ** 2\n ) in [3,4]\n )\n\n return result\n\ndef calculate_gol_state_with_masks_per_delta(start, mask, matrixWidth, delta):\n result = start\n\n for i in range(0,delta):\n result = calculate_gol_state_for_tensor_with_masks(result, masks, matrixWidth)\n\n return result\n\ndataset = pd.read_csv('/kaggle/input/conways-reverse-game-of-life-2020/train.csv') \ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nto_generate_one_delta_from = dataset.loc[dataset['delta'] > 1].reset_index(drop=True)\nmasks = calculate_all_masks(25)\ncalculated_result = torch.as_tensor(np.zeros((len(list(to_generate_one_delta_from.index)), 625), int), device=device)\nstart_positions = torch.as_tensor(to_generate_one_delta_from.filter(regex='^start_').values, device=device)\n\nfor i in range(0, start_positions.shape[0]):\n calculated_result[i] |= calculate_gol_state_with_masks_per_delta(start_positions[i], masks, 25, 1)\n if ((i % 100) == 0):\n print(i, end=',')\nI am aware that I could deduce the width of the vector/dimension of the metrix from the shap, but chose to be a bit more explicit here for no good reason.
\nTL;DR;
\nIs there a way to improve the performance of the code above further?
Cheers
\ndud
", "messages": [ { "id": 1086152, "postDate": "2020-11-21T12:43:07.973Z", "content": "Hi there,
\na novice here.
\nI was just watching the competition and toying around with the dataset and it got me curious regarding the improvement of data-preparation.
\nA naive approach yielded an error-rate of around 0.17 so I thought it best to compute some more data for e.g. delta=1.
A very direct approach for the computation would run ~5-6h and not use the GPU.
\nI came up with the following approach which runs ~45-60min(if projections pan out) and utilizes ~13-15% of the GPU.
\nI understand that the bottle-neck here is computation of a cell's state in the next iteration.
\nIs there a way to treat everything as a matrix-/vector-operation and thus utilizing the GPU more gaining more speed?
import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom fastai.tabular.all import *\nfrom torch.nn import *\n\n\ndef calculate_all_masks(matrixWidth):\n columnCount = int(matrixWidth ** 2)\n result = torch.as_tensor(np.zeros((columnCount,columnCount), int), device=device)\n\n for position in range(0,columnCount):\n for y in [-1,0,1]:\n for z in [-1,0,1]:\n calculated_row = (position + columnCount +y*matrixWidth) % columnCount\n index_with_column_chosen = int(calculated_row/matrixWidth) * matrixWidth + (calculated_row + 1*z)%matrixWidth\n result[position][index_with_column_chosen] = 1\n\n return result\n\ndef calculate_gol_state_for_tensor_with_masks(start, masks, matrixWidth):\n columnCount = int(matrixWidth ** 2)\n result = torch.as_tensor(np.zeros((columnCount,), int), device=device)\n zero_vector = torch.as_tensor(np.zeros((columnCount,),int), device=device).float().unsqueeze(0)\n\n for current_cells in range(0, columnCount):\n result[current_cells] = int(\n math.ceil(\n torch.cdist((masks[current_cells] & start).float().unsqueeze(0), zero_vector)[0].item() ** 2\n ) in [3,4]\n )\n\n return result\n\ndef calculate_gol_state_with_masks_per_delta(start, mask, matrixWidth, delta):\n result = start\n\n for i in range(0,delta):\n result = calculate_gol_state_for_tensor_with_masks(result, masks, matrixWidth)\n\n return result\n\ndataset = pd.read_csv('/kaggle/input/conways-reverse-game-of-life-2020/train.csv') \ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nto_generate_one_delta_from = dataset.loc[dataset['delta'] > 1].reset_index(drop=True)\nmasks = calculate_all_masks(25)\ncalculated_result = torch.as_tensor(np.zeros((len(list(to_generate_one_delta_from.index)), 625), int), device=device)\nstart_positions = torch.as_tensor(to_generate_one_delta_from.filter(regex='^start_').values, device=device)\n\nfor i in range(0, start_positions.shape[0]):\n calculated_result[i] |= calculate_gol_state_with_masks_per_delta(start_positions[i], masks, 25, 1)\n if ((i % 100) == 0):\n print(i, end=',')\nI am aware that I could deduce the width of the vector/dimension of the metrix from the shap, but chose to be a bit more explicit here for no good reason.
\nTL;DR;
\nIs there a way to improve the performance of the code above further?
Cheers
\ndud
", "rawMarkdown": "Hi there,\n\na novice here.\nI was just watching the competition and toying around with the dataset and it got me curious regarding the improvement of data-preparation.\nA naive approach yielded an error-rate of around 0.17 so I thought it best to compute some more data for e.g. delta=1.\n\nA very direct approach for the computation would run ~5-6h and not use the GPU.\n\nI came up with the following approach which runs ~45-60min(if projections pan out) and utilizes ~13-15% of the GPU.\n\nI understand that the bottle-neck here is computation of a cell's state in the next iteration.\nIs there a way to treat everything as a matrix-/vector-operation and thus utilizing the GPU more gaining more speed?\n\n```\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom fastai.tabular.all import *\nfrom torch.nn import *\n\n\ndef calculate_all_masks(matrixWidth):\n columnCount = int(matrixWidth ** 2)\n result = torch.as_tensor(np.zeros((columnCount,columnCount), int), device=device)\n \n for position in range(0,columnCount):\n for y in [-1,0,1]:\n for z in [-1,0,1]:\n calculated_row = (position + columnCount +y*matrixWidth) % columnCount\n index_with_column_chosen = int(calculated_row/matrixWidth) * matrixWidth + (calculated_row + 1*z)%matrixWidth\n result[position][index_with_column_chosen] = 1\n \n return result\n\ndef calculate_gol_state_for_tensor_with_masks(start, masks, matrixWidth):\n columnCount = int(matrixWidth ** 2)\n result = torch.as_tensor(np.zeros((columnCount,), int), device=device)\n zero_vector = torch.as_tensor(np.zeros((columnCount,),int), device=device).float().unsqueeze(0)\n \n for current_cells in range(0, columnCount):\n result[current_cells] = int(\n math.ceil(\n torch.cdist((masks[current_cells] & start).float().unsqueeze(0), zero_vector)[0].item() ** 2\n ) in [3,4]\n )\n \n return result\n\ndef calculate_gol_state_with_masks_per_delta(start, mask, matrixWidth, delta):\n result = start\n \n for i in range(0,delta):\n result = calculate_gol_state_for_tensor_with_masks(result, masks, matrixWidth)\n \n return result\n \ndataset = pd.read_csv('/kaggle/input/conways-reverse-game-of-life-2020/train.csv') \ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nto_generate_one_delta_from = dataset.loc[dataset['delta'] > 1].reset_index(drop=True)\nmasks = calculate_all_masks(25)\ncalculated_result = torch.as_tensor(np.zeros((len(list(to_generate_one_delta_from.index)), 625), int), device=device)\nstart_positions = torch.as_tensor(to_generate_one_delta_from.filter(regex='^start_').values, device=device)\n\nfor i in range(0, start_positions.shape[0]):\n calculated_result[i] |= calculate_gol_state_with_masks_per_delta(start_positions[i], masks, 25, 1)\n if ((i % 100) == 0):\n print(i, end=',')\n```\n\nI am aware that I could deduce the width of the vector/dimension of the metrix from the shap, but chose to be a bit more explicit here for no good reason.\n\nTL;DR;\nIs there a way to improve the performance of the code above further?\n\nCheers\n\ndud\n", "votes": 3 }, { "id": 1097002, "postDate": "2020-11-30T23:36:37.123Z", "content": "The \"secret\" of the top leaderboard entries is that they are written in C. Same with the ConnectX competition. Native python is 100x slower than C.
\nIf you want to stick with Python, then get familiar with numba @njit which compiles to the LLVM (ie C)
\n\nimport numpy as np\nfrom numba import njit\n\n@njit()\ndef count_neighbours( input: np.ndarray ) -> np.ndarray:\n output = np.zeros(array.shape, dtype=np.int) \n directions = [ (1,1), (1,0), (1,-1), (0,1), (1,-1), (-1,1), (-1,0), (-1,-1), ]\n shape = input.shape\n for x in range(shape[0]):\n for y in range(shape[1]):\n for (dx,dy) in directions:\n output[x,y] += input[ (x+dx)%shape[0], (y+dy)%shape[1] ]\n return output \nIn practice using numba can be a bit fiddly to use:
\nI was actually thinking of restructuring things to harness the GPU better, but did not expect s.th. like numba to be in the mix, as well. I havn't even heard of it before. And by the looks of it, is also able to work with Cuda.
\nThank you for this tip.👍
Out of curiosity, what is the idea behind this code? What is it supposed to do?
\nIt looks like some kind of magic.
Hi!
\nThank you for answering.
\nThat escalated quickly.
\nWasn't expecting a discussion about the motivation behind it, but I do appreciate it, as I might have made mistakes I am not aware of.
So let me structure this in 2 parts.
\nOne for what the code actually does and another with the motivation behind this.
Let's start with the motivation and how I viewed the problem.
\nThe rules for the Game of Life allow for an easy forward computation, but as soon as you try to reverse it you are faced with a combinatory problem which grows greated with the size of the grid and the number of iterations.
\nML is hungry for data.
\nI was motivated by the thought that more data allows for my Neural Net to train more and become better at calculating a solution. Every iteration with delta > 1 is a composition of x iterations of delta==1 predictions where x is the actual delta for the dataset to find a solution for.
\nIn order get more data quickly I chose the start-positions of the training set as they provide a state that lives up to or past delta == 1.
\nThe code above is meant to calculate the first iteration for every row in the dataset with delta > 1.
\nI still have an error rate of 0.14 so I must be doing s.th. else wrong, but I am working on it.
As far as the code is concerned.
\nThe state of every cell is only concerned with it's immediat surrounding.
\nSo I can mask out all other cells for the purpose of computing a cell's state after one iteration.
\nTo speed things up, I precompute a combination of all the masks.
\nThen the state of the cell in question is calculated by using bit-wise & on the vector and it will become a matter distance to a 0-vector.
\nLength 3 or 4 means the cell lives after the current iteration regardless of it's current state.
\nAny other length means that it is not alive after the current iteration.
\n2 or 3 neighbors => cell get's to live on so the length can either be 3 or 4(3 neighbors(1) and the alive cell(1))
\nIf the cell is dead and 3 neighbors are present => cell gets to live next round and the length has to be 3
Cheers
\ndud
", "rawMarkdown": "Hi!\nThank you for answering.\nThat escalated quickly.\nWasn't expecting a discussion about the motivation behind it, but I do appreciate it, as I might have made mistakes I am not aware of.\n\nSo let me structure this in 2 parts.\nOne for what the code actually does and another with the motivation behind this.\n\nLet's start with the motivation and how I viewed the problem.\nThe rules for the Game of Life allow for an easy forward computation, but as soon as you try to reverse it you are faced with a combinatory problem which grows greated with the size of the grid and the number of iterations.\nML is hungry for data.\nI was motivated by the thought that more data allows for my Neural Net to train more and become better at calculating a solution. Every iteration with delta > 1 is a composition of x iterations of delta==1 predictions where x is the actual delta for the dataset to find a solution for.\nIn order get more data quickly I chose the start-positions of the training set as they provide a state that lives up to or past delta == 1.\nThe code above is meant to calculate the first iteration for every row in the dataset with delta > 1.\nI still have an error rate of 0.14 so I must be doing s.th. else wrong, but I am working on it.\n\nAs far as the code is concerned.\nThe state of every cell is only concerned with it's immediat surrounding.\nSo I can mask out all other cells for the purpose of computing a cell's state after one iteration.\nTo speed things up, I precompute a combination of all the masks.\nThen the state of the cell in question is calculated by using bit-wise & on the vector and it will become a matter distance to a 0-vector.\nLength 3 or 4 means the cell lives after the current iteration regardless of it's current state.\nAny other length means that it is not alive after the current iteration.\n2 or 3 neighbors => cell get's to live on so the length can either be 3 or 4(3 neighbors(1) and the alive cell(1))\nIf the cell is dead and 3 neighbors are present => cell gets to live next round and the length has to be 3\n\nCheers\n\ndud\n" }, { "id": 1088601, "postDate": "2020-11-23T19:29:35.593Z", "content": "Interesting idea.
\nAlthough it is still unclear to me how you code solves reverse game of life problem and not direct one, a have few comments:
\n\n\nThe state of every cell is only concerned with its immediate surrounding.
\n
Unfortunately, while working with SAT solvers, I found that adding additional constraint on dead cell surrounded by dead cells to be dead previously (0-length in your terms) often makes problem of finding previous step unsolvable.
\nSometimes even 5x5 surrounding is not enough to determine cell state.
\n\nEvery iteration with delta > 1 is a composition of x iterations of delta==1 predictions where x is the actual delta for the dataset to find a solution for.
\n
While this is true, only small subset of possible 1st iteration solutions will be solvable on next iteration.
\nFinding solutions with delta at least (x-1) is crucial for the competition.
I think we are talking about s.th. different.
\nThis is not ment to solve the reverse Game of Life, but rather provide data for it.
\nIt is a forward solver to give the Neural Net more data, and for that I only need the surrounding cells.
\nSince it is a forward solver for data-generation and and zeroed out all the non-neighborly cells, I think I actually removed unnecessary input. Looking at my conditions for a living/reanimated cell, I actually end up with one less condition compared to the step-instructions for the classical non-vector approach.
Do you mind elaborating on
\n\n\nWhile this is true, only small subset of possible 1st iteration solutions will be solvable on next iteration.
\n
\n Finding solutions with delta at least (x-1) is crucial for the competition.
Why is that? Could you furnish me with a link?
\nAlso, why does it matter? Any valid starting-position resulting in the given game is valid according to the competition's description
\n", "rawMarkdown": "I think we are talking about s.th. different.\nThis is not ment to solve the reverse Game of Life, but rather provide data for it.\nIt is a forward solver to give the Neural Net more data, and for that I only need the surrounding cells.\nSince it is a forward solver for data-generation and and zeroed out all the non-neighborly cells, I think I actually removed unnecessary input. Looking at my conditions for a living/reanimated cell, I actually end up with one less condition compared to the step-instructions for the classical non-vector approach.\n\nDo you mind elaborating on \n> While this is true, only small subset of possible 1st iteration solutions will be solvable on next iteration.\nFinding solutions with delta at least (x-1) is crucial for the competition.\n\nWhy is that? Could you furnish me with a link?\nAlso, why does it matter? Any valid starting-position resulting in the given game is valid according to the competition's description\n\n> Submissions are solved forward by the appropriate number of steps, so that any correct starting solution will achieve a maximum score. This article contains the stepping function that is used for this competition.\n\n" } ] }, { "id": 1088018, "postDate": "2020-11-23T08:43:40.060Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1097002, "author_name": "James McGuigan", "author_url": "", "post_date": "2020-11-30T23:36:37.123000", "content": "Submissions are solved forward by the appropriate number of steps, so that any correct starting solution will achieve a maximum score. This article contains the stepping function that is used for this competition.
\n
The \"secret\" of the top leaderboard entries is that they are written in C. Same with the ConnectX competition. Native python is 100x slower than C.
\nIf you want to stick with Python, then get familiar with numba @njit which compiles to the LLVM (ie C)
\n\nimport numpy as np\nfrom numba import njit\n\n@njit()\ndef count_neighbours( input: np.ndarray ) -> np.ndarray:\n output = np.zeros(array.shape, dtype=np.int) \n directions = [ (1,1), (1,0), (1,-1), (0,1), (1,-1), (-1,1), (-1,0), (-1,-1), ]\n shape = input.shape\n for x in range(shape[0]):\n for y in range(shape[1]):\n for (dx,dy) in directions:\n output[x,y] += input[ (x+dx)%shape[0], (y+dy)%shape[1] ]\n return output \nIn practice using numba can be a bit fiddly to use:
\nI was actually thinking of restructuring things to harness the GPU better, but did not expect s.th. like numba to be in the mix, as well. I havn't even heard of it before. And by the looks of it, is also able to work with Cuda.
\nThank you for this tip.👍
Out of curiosity, what is the idea behind this code? What is it supposed to do?
\nIt looks like some kind of magic.
Hi!
\nThank you for answering.
\nThat escalated quickly.
\nWasn't expecting a discussion about the motivation behind it, but I do appreciate it, as I might have made mistakes I am not aware of.
So let me structure this in 2 parts.
\nOne for what the code actually does and another with the motivation behind this.
Let's start with the motivation and how I viewed the problem.
\nThe rules for the Game of Life allow for an easy forward computation, but as soon as you try to reverse it you are faced with a combinatory problem which grows greated with the size of the grid and the number of iterations.
\nML is hungry for data.
\nI was motivated by the thought that more data allows for my Neural Net to train more and become better at calculating a solution. Every iteration with delta > 1 is a composition of x iterations of delta==1 predictions where x is the actual delta for the dataset to find a solution for.
\nIn order get more data quickly I chose the start-positions of the training set as they provide a state that lives up to or past delta == 1.
\nThe code above is meant to calculate the first iteration for every row in the dataset with delta > 1.
\nI still have an error rate of 0.14 so I must be doing s.th. else wrong, but I am working on it.
As far as the code is concerned.
\nThe state of every cell is only concerned with it's immediat surrounding.
\nSo I can mask out all other cells for the purpose of computing a cell's state after one iteration.
\nTo speed things up, I precompute a combination of all the masks.
\nThen the state of the cell in question is calculated by using bit-wise & on the vector and it will become a matter distance to a 0-vector.
\nLength 3 or 4 means the cell lives after the current iteration regardless of it's current state.
\nAny other length means that it is not alive after the current iteration.
\n2 or 3 neighbors => cell get's to live on so the length can either be 3 or 4(3 neighbors(1) and the alive cell(1))
\nIf the cell is dead and 3 neighbors are present => cell gets to live next round and the length has to be 3
Cheers
\ndud
", "votes": 0, "replies": [] }, { "id": 1088601, "author_name": "Serhii Hrynko", "author_url": "", "post_date": "2020-11-23T19:29:35.593000", "content": "Interesting idea.
\nAlthough it is still unclear to me how you code solves reverse game of life problem and not direct one, a have few comments:
\n\n\nThe state of every cell is only concerned with its immediate surrounding.
\n
Unfortunately, while working with SAT solvers, I found that adding additional constraint on dead cell surrounded by dead cells to be dead previously (0-length in your terms) often makes problem of finding previous step unsolvable.
\nSometimes even 5x5 surrounding is not enough to determine cell state.
\n\nEvery iteration with delta > 1 is a composition of x iterations of delta==1 predictions where x is the actual delta for the dataset to find a solution for.
\n
While this is true, only small subset of possible 1st iteration solutions will be solvable on next iteration.
\nFinding solutions with delta at least (x-1) is crucial for the competition.
I think we are talking about s.th. different.
\nThis is not ment to solve the reverse Game of Life, but rather provide data for it.
\nIt is a forward solver to give the Neural Net more data, and for that I only need the surrounding cells.
\nSince it is a forward solver for data-generation and and zeroed out all the non-neighborly cells, I think I actually removed unnecessary input. Looking at my conditions for a living/reanimated cell, I actually end up with one less condition compared to the step-instructions for the classical non-vector approach.
Do you mind elaborating on
\n\n\nWhile this is true, only small subset of possible 1st iteration solutions will be solvable on next iteration.
\n
\n Finding solutions with delta at least (x-1) is crucial for the competition.
Why is that? Could you furnish me with a link?
\nAlso, why does it matter? Any valid starting-position resulting in the given game is valid according to the competition's description
\n", "votes": 0, "replies": [] } ] }, { "id": 1088018, "author_name": "", "author_url": "", "post_date": "2020-11-23T08:43:40.060000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1086152": "Hi there,\n\na novice here.\nI was just watching the competition and toying around with the dataset and it got me curious regarding the improvement of data-preparation.\nA naive approach yielded an error-rate of around 0.17 so I thought it best to compute some more data for e.g. delta=1.\n\nA very direct approach for the computation would run ~5-6h and not use the GPU.\n\nI came up with the following approach which runs ~45-60min(if projections pan out) and utilizes ~13-15% of the GPU.\n\nI understand that the bottle-neck here is computation of a cell's state in the next iteration.\nIs there a way to treat everything as a matrix-/vector-operation and thus utilizing the GPU more gaining more speed?\n\n```\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom fastai.tabular.all import *\nfrom torch.nn import *\n\n\ndef calculate_all_masks(matrixWidth):\n columnCount = int(matrixWidth ** 2)\n result = torch.as_tensor(np.zeros((columnCount,columnCount), int), device=device)\n \n for position in range(0,columnCount):\n for y in [-1,0,1]:\n for z in [-1,0,1]:\n calculated_row = (position + columnCount +y*matrixWidth) % columnCount\n index_with_column_chosen = int(calculated_row/matrixWidth) * matrixWidth + (calculated_row + 1*z)%matrixWidth\n result[position][index_with_column_chosen] = 1\n \n return result\n\ndef calculate_gol_state_for_tensor_with_masks(start, masks, matrixWidth):\n columnCount = int(matrixWidth ** 2)\n result = torch.as_tensor(np.zeros((columnCount,), int), device=device)\n zero_vector = torch.as_tensor(np.zeros((columnCount,),int), device=device).float().unsqueeze(0)\n \n for current_cells in range(0, columnCount):\n result[current_cells] = int(\n math.ceil(\n torch.cdist((masks[current_cells] & start).float().unsqueeze(0), zero_vector)[0].item() ** 2\n ) in [3,4]\n )\n \n return result\n\ndef calculate_gol_state_with_masks_per_delta(start, mask, matrixWidth, delta):\n result = start\n \n for i in range(0,delta):\n result = calculate_gol_state_for_tensor_with_masks(result, masks, matrixWidth)\n \n return result\n \ndataset = pd.read_csv('/kaggle/input/conways-reverse-game-of-life-2020/train.csv') \ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nto_generate_one_delta_from = dataset.loc[dataset['delta'] > 1].reset_index(drop=True)\nmasks = calculate_all_masks(25)\ncalculated_result = torch.as_tensor(np.zeros((len(list(to_generate_one_delta_from.index)), 625), int), device=device)\nstart_positions = torch.as_tensor(to_generate_one_delta_from.filter(regex='^start_').values, device=device)\n\nfor i in range(0, start_positions.shape[0]):\n calculated_result[i] |= calculate_gol_state_with_masks_per_delta(start_positions[i], masks, 25, 1)\n if ((i % 100) == 0):\n print(i, end=',')\n```\n\nI am aware that I could deduce the width of the vector/dimension of the metrix from the shap, but chose to be a bit more explicit here for no good reason.\n\nTL;DR;\nIs there a way to improve the performance of the code above further?\n\nCheers\n\ndud\n", "1097002": "The \"secret\" of the top leaderboard entries is that they are written in C. Same with the ConnectX competition. Native python is 100x slower than C.\n\nIf you want to stick with Python, then get familiar with numba @njit which compiles to the LLVM (ie C)\n- https://numba.pydata.org/numba-doc/latest/index.html\n```\nimport numpy as np\nfrom numba import njit\n\n@njit()\ndef count_neighbours( input: np.ndarray ) -> np.ndarray:\n output = np.zeros(array.shape, dtype=np.int) \n directions = [ (1,1), (1,0), (1,-1), (0,1), (1,-1), (-1,1), (-1,0), (-1,-1), ]\n shape = input.shape\n for x in range(shape[0]):\n for y in range(shape[1]):\n for (dx,dy) in directions:\n output[x,y] += input[ (x+dx)%shape[0], (y+dy)%shape[1] ]\n return output \n```\n\nIn practice using numba can be a bit fiddly to use:\n- It has to statically type everything using C types and will throw compile-time exceptions if it can't figure out deterministic typing for all variables\n- It doesn't like object oriented programming - using @jit(nopython=False) eliminates most of the performance gains\n- It doesn't like some common python idioms \n- It doesn't like dictionaries unless you use numba.typed.Dict() and be explicit about typing\n- It has problems with reflected lists if it can't statically determine type\n- It likes numpy arrays (which are already C typed)\n- Not all numpy functions are implemented, and you often can't use itertools or other libraries\n- There is very high one-time compile cost the first time the function is run\n- There is significant overhead in converting python objects to C types and back, so ading @njit to a single small function may actually make things slower - please profile your code and avoid passing in unnecessary data\n- It can be tricky to get working sometimes, and some code may be impossible to port\n- Used correctly it can provide a 100-1000x performance increase vs native python\n- use NUMBA_DISABLE_JIT=1 to disable for debugging", "1087990": "Out of curiosity, what is the idea behind this code? What is it supposed to do?\nIt looks like some kind of magic.", "1088018": "" } }Submissions are solved forward by the appropriate number of steps, so that any correct starting solution will achieve a maximum score. This article contains the stepping function that is used for this competition.
\n