{ "id": 200235, "title": "Unknown Problem with static MAE", "url": "/competitions/conways-reverse-game-of-life-2020/discussion/200235", "author_name": "", "post_date": "2020-11-29T15:24:43.353935900Z", "votes": 1, "comment_count": 2, "views": 0, "content": "

Hi again,

\n

I seem to be doing s.th. fundamentally wrong trying to adapt the learning rate.
\nI have commented out some approaches I have tried before.
\nPlease see my code below.
\nA hint as to what I am doing wrong is much appreciated.
\nI am stuck at an error-rate of 0.15 with MAE.

\n

Cheers

\n
def getModel():\n    model = Sequential(\n      Conv2d(in_channels=1, out_channels=1, kernel_size=(3,3), padding=(1,1), padding_mode='circular'),\n      ReLU(),\n      Conv2d(in_channels=1, out_channels=1, kernel_size=(3,3), padding=(1,1), padding_mode='circular'),\n      ReLU(),\n    )\n\n    model.to(device)\n    model =  parallel.DataParallel(model)\n\n    # the current field is of no interest to us\n    #model.module[0].weight[0][0][1][1] = 0\n    return model\n\ndef getOptimizer(model):\n    return optim.SGD(model.parameters(), lr=2.5)\n    # return optim.Adadelta(model.parameters())\n\ndef getScheduler(optimizer):\n    return torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.5)\n\ndef trainModel(model, optimizer, scheduler, xs_matrix, y_matrix):\n    charToUse = [\".\", \"\\b\"]\n    charIndex = 0\n    mae = L1Loss()\n\n    # TODO precompute the squeeze or unsqueeze operations\n    for i in train_index_nn:\n        # in your training loop:\n        input = xs_matrix[i].unsqueeze(0).unsqueeze(0)\n        target = y_matrix[i]\n        output = model(input)\n\n        loss = mae(output[0].squeeze(),target)\n        loss.backward()\n        #scheduler.optimizer.step()   \n        optimizer.step()    \n        #scheduler.optimizer.zero_grad()  \n        optimizer.zero_grad()  \n        #scheduler.step(loss) \n        if (i%100 == 0):\n            print(charToUse[charIndex],end='')\n            charIndex = (charIndex + 1) % 2\n\n    return model\n\ndef calculateMAE(model, xs_valid, y_valid):\n    rowsize = np.zeros((25,25),float)\n    columnsize = [rowsize for  i in range(0,len(valid_index_nn))]\n    result_estimates = torch.as_tensor(torch.as_tensor(columnsize,device=device), device=device)\n\n    #TODO try to get rid of the unqueeze operation or at least precompute it\n    #by predefining the tensor we get rid of costly copy operations and estimate faster\n    for  i in range(0,len(xs_valid)):\n        result_estimates[i] = model(xs_valid[i].unsqueeze(0).unsqueeze(0)).detach()[0].squeeze()\n\n\n    return mean_absolute_error(y_valid.cpu().numpy().reshape(y_valid.shape[0],625), result_estimates.cpu().numpy().reshape(result_estimates.shape[0],625))\n\n\ndef calculateMAERighted(model, xs_valid, y_valid):\n    rowsize = np.zeros((25,25),float)\n    columnsize = [rowsize for  i in range(0,len(valid_index_nn))]\n    result_estimates = torch.as_tensor(torch.as_tensor(columnsize,device=device), device=device)\n\n    #TODO try to get rid of the unqueeze operation or at least precompute it\n    #by predefining the tensor we get rid of costly copy operations and estimate faster\n    for  i in range(0,len(xs_valid)):\n        result_estimates[i] = (model(xs_valid[i].unsqueeze(0).unsqueeze(0)).detach()[0].squeeze() > 0.).float()\n\n    return mean_absolute_error(y_valid.cpu().numpy().reshape(y_valid.shape[0],625), result_estimates.cpu().numpy().reshape(result_estimates.shape[0],625))\n
", "messages": [ { "id": "1095414", "postDate": "11/29/2020 15:24:43", "content": "

Hi again,

\n

I seem to be doing s.th. fundamentally wrong trying to adapt the learning rate.
\nI have commented out some approaches I have tried before.
\nPlease see my code below.
\nA hint as to what I am doing wrong is much appreciated.
\nI am stuck at an error-rate of 0.15 with MAE.

\n

Cheers

\n
def getModel():\n    model = Sequential(\n      Conv2d(in_channels=1, out_channels=1, kernel_size=(3,3), padding=(1,1), padding_mode='circular'),\n      ReLU(),\n      Conv2d(in_channels=1, out_channels=1, kernel_size=(3,3), padding=(1,1), padding_mode='circular'),\n      ReLU(),\n    )\n\n    model.to(device)\n    model =  parallel.DataParallel(model)\n\n    # the current field is of no interest to us\n    #model.module[0].weight[0][0][1][1] = 0\n    return model\n\ndef getOptimizer(model):\n    return optim.SGD(model.parameters(), lr=2.5)\n    # return optim.Adadelta(model.parameters())\n\ndef getScheduler(optimizer):\n    return torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.5)\n\ndef trainModel(model, optimizer, scheduler, xs_matrix, y_matrix):\n    charToUse = [\".\", \"\\b\"]\n    charIndex = 0\n    mae = L1Loss()\n\n    # TODO precompute the squeeze or unsqueeze operations\n    for i in train_index_nn:\n        # in your training loop:\n        input = xs_matrix[i].unsqueeze(0).unsqueeze(0)\n        target = y_matrix[i]\n        output = model(input)\n\n        loss = mae(output[0].squeeze(),target)\n        loss.backward()\n        #scheduler.optimizer.step()   \n        optimizer.step()    \n        #scheduler.optimizer.zero_grad()  \n        optimizer.zero_grad()  \n        #scheduler.step(loss) \n        if (i%100 == 0):\n            print(charToUse[charIndex],end='')\n            charIndex = (charIndex + 1) % 2\n\n    return model\n\ndef calculateMAE(model, xs_valid, y_valid):\n    rowsize = np.zeros((25,25),float)\n    columnsize = [rowsize for  i in range(0,len(valid_index_nn))]\n    result_estimates = torch.as_tensor(torch.as_tensor(columnsize,device=device), device=device)\n\n    #TODO try to get rid of the unqueeze operation or at least precompute it\n    #by predefining the tensor we get rid of costly copy operations and estimate faster\n    for  i in range(0,len(xs_valid)):\n        result_estimates[i] = model(xs_valid[i].unsqueeze(0).unsqueeze(0)).detach()[0].squeeze()\n\n\n    return mean_absolute_error(y_valid.cpu().numpy().reshape(y_valid.shape[0],625), result_estimates.cpu().numpy().reshape(result_estimates.shape[0],625))\n\n\ndef calculateMAERighted(model, xs_valid, y_valid):\n    rowsize = np.zeros((25,25),float)\n    columnsize = [rowsize for  i in range(0,len(valid_index_nn))]\n    result_estimates = torch.as_tensor(torch.as_tensor(columnsize,device=device), device=device)\n\n    #TODO try to get rid of the unqueeze operation or at least precompute it\n    #by predefining the tensor we get rid of costly copy operations and estimate faster\n    for  i in range(0,len(xs_valid)):\n        result_estimates[i] = (model(xs_valid[i].unsqueeze(0).unsqueeze(0)).detach()[0].squeeze() > 0.).float()\n\n    return mean_absolute_error(y_valid.cpu().numpy().reshape(y_valid.shape[0],625), result_estimates.cpu().numpy().reshape(result_estimates.shape[0],625))\n
", "rawMarkdown": "Hi again,\n\nI seem to be doing s.th. fundamentally wrong trying to adapt the learning rate.\nI have commented out some approaches I have tried before.\nPlease see my code below.\nA hint as to what I am doing wrong is much appreciated.\nI am stuck at an error-rate of 0.15 with MAE.\n\nCheers\n\n```\ndef getModel():\n model = Sequential(\n Conv2d(in_channels=1, out_channels=1, kernel_size=(3,3), padding=(1,1), padding_mode='circular'),\n ReLU(),\n Conv2d(in_channels=1, out_channels=1, kernel_size=(3,3), padding=(1,1), padding_mode='circular'),\n ReLU(),\n )\n \n model.to(device)\n model = parallel.DataParallel(model)\n \n # the current field is of no interest to us\n #model.module[0].weight[0][0][1][1] = 0\n return model\n \ndef getOptimizer(model):\n return optim.SGD(model.parameters(), lr=2.5)\n # return optim.Adadelta(model.parameters())\n\ndef getScheduler(optimizer):\n return torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.5)\n \ndef trainModel(model, optimizer, scheduler, xs_matrix, y_matrix):\n charToUse = [\".\", \"\\b\"]\n charIndex = 0\n mae = L1Loss()\n \n # TODO precompute the squeeze or unsqueeze operations\n for i in train_index_nn:\n # in your training loop:\n input = xs_matrix[i].unsqueeze(0).unsqueeze(0)\n target = y_matrix[i]\n output = model(input)\n\n loss = mae(output[0].squeeze(),target)\n loss.backward()\n #scheduler.optimizer.step() \n optimizer.step() \n #scheduler.optimizer.zero_grad() \n optimizer.zero_grad() \n #scheduler.step(loss) \n if (i%100 == 0):\n print(charToUse[charIndex],end='')\n charIndex = (charIndex + 1) % 2\n\n return model\n \ndef calculateMAE(model, xs_valid, y_valid):\n rowsize = np.zeros((25,25),float)\n columnsize = [rowsize for i in range(0,len(valid_index_nn))]\n result_estimates = torch.as_tensor(torch.as_tensor(columnsize,device=device), device=device)\n\n #TODO try to get rid of the unqueeze operation or at least precompute it\n #by predefining the tensor we get rid of costly copy operations and estimate faster\n for i in range(0,len(xs_valid)):\n result_estimates[i] = model(xs_valid[i].unsqueeze(0).unsqueeze(0)).detach()[0].squeeze()\n\n \n return mean_absolute_error(y_valid.cpu().numpy().reshape(y_valid.shape[0],625), result_estimates.cpu().numpy().reshape(result_estimates.shape[0],625))\n\n\ndef calculateMAERighted(model, xs_valid, y_valid):\n rowsize = np.zeros((25,25),float)\n columnsize = [rowsize for i in range(0,len(valid_index_nn))]\n result_estimates = torch.as_tensor(torch.as_tensor(columnsize,device=device), device=device)\n\n #TODO try to get rid of the unqueeze operation or at least precompute it\n #by predefining the tensor we get rid of costly copy operations and estimate faster\n for i in range(0,len(xs_valid)):\n result_estimates[i] = (model(xs_valid[i].unsqueeze(0).unsqueeze(0)).detach()[0].squeeze() > 0.).float()\n\n return mean_absolute_error(y_valid.cpu().numpy().reshape(y_valid.shape[0],625), result_estimates.cpu().numpy().reshape(result_estimates.shape[0],625))\n```", "votes": null }, { "id": "1095764", "postDate": "11/29/2020 23:33:58", "content": "

I am not sure a direct out-of-box use of DNN or CNN would useful. I first tried Keras out of box. It gave me about 15% error rates. Since the test data has about 15% live cells, so the start state of 0 for all entries would get 15% error rates. A highly customized CNN would be needed.
\nI don't think you are doing anything wrong. Out of box NN just does not work well.

", "rawMarkdown": "I am not sure a direct out-of-box use of DNN or CNN would useful. I first tried Keras out of box. It gave me about 15% error rates. Since the test data has about 15% live cells, so the start state of 0 for all entries would get 15% error rates. A highly customized CNN would be needed.\nI don't think you are doing anything wrong. Out of box NN just does not work well.", "votes": null }, { "id": "1096019", "postDate": "11/30/2020 06:43:52", "content": "

That is a relief.
\nThanks.
\nI will look through the results of the competition then and try to get a grasp on what a more ideal solution for this problem constitutes.

", "rawMarkdown": "That is a relief.\nThanks.\nI will look through the results of the competition then and try to get a grasp on what a more ideal solution for this problem constitutes.", "votes": null } ], "comments": [ { "id": 1095764, "author_name": "mathtester", "author_url": "", "post_date": "11/29/2020 23:33:58", "content": "

I am not sure a direct out-of-box use of DNN or CNN would useful. I first tried Keras out of box. It gave me about 15% error rates. Since the test data has about 15% live cells, so the start state of 0 for all entries would get 15% error rates. A highly customized CNN would be needed.
\nI don't think you are doing anything wrong. Out of box NN just does not work well.

", "votes": null, "replies": [ { "id": 1096019, "author_name": "badhairday", "author_url": "", "post_date": "11/30/2020 06:43:52", "content": "

That is a relief.
\nThanks.
\nI will look through the results of the competition then and try to get a grasp on what a more ideal solution for this problem constitutes.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1095414": "Hi again,\n\nI seem to be doing s.th. fundamentally wrong trying to adapt the learning rate.\nI have commented out some approaches I have tried before.\nPlease see my code below.\nA hint as to what I am doing wrong is much appreciated.\nI am stuck at an error-rate of 0.15 with MAE.\n\nCheers\n\n```\ndef getModel():\n model = Sequential(\n Conv2d(in_channels=1, out_channels=1, kernel_size=(3,3), padding=(1,1), padding_mode='circular'),\n ReLU(),\n Conv2d(in_channels=1, out_channels=1, kernel_size=(3,3), padding=(1,1), padding_mode='circular'),\n ReLU(),\n )\n \n model.to(device)\n model = parallel.DataParallel(model)\n \n # the current field is of no interest to us\n #model.module[0].weight[0][0][1][1] = 0\n return model\n \ndef getOptimizer(model):\n return optim.SGD(model.parameters(), lr=2.5)\n # return optim.Adadelta(model.parameters())\n\ndef getScheduler(optimizer):\n return torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.5)\n \ndef trainModel(model, optimizer, scheduler, xs_matrix, y_matrix):\n charToUse = [\".\", \"\\b\"]\n charIndex = 0\n mae = L1Loss()\n \n # TODO precompute the squeeze or unsqueeze operations\n for i in train_index_nn:\n # in your training loop:\n input = xs_matrix[i].unsqueeze(0).unsqueeze(0)\n target = y_matrix[i]\n output = model(input)\n\n loss = mae(output[0].squeeze(),target)\n loss.backward()\n #scheduler.optimizer.step() \n optimizer.step() \n #scheduler.optimizer.zero_grad() \n optimizer.zero_grad() \n #scheduler.step(loss) \n if (i%100 == 0):\n print(charToUse[charIndex],end='')\n charIndex = (charIndex + 1) % 2\n\n return model\n \ndef calculateMAE(model, xs_valid, y_valid):\n rowsize = np.zeros((25,25),float)\n columnsize = [rowsize for i in range(0,len(valid_index_nn))]\n result_estimates = torch.as_tensor(torch.as_tensor(columnsize,device=device), device=device)\n\n #TODO try to get rid of the unqueeze operation or at least precompute it\n #by predefining the tensor we get rid of costly copy operations and estimate faster\n for i in range(0,len(xs_valid)):\n result_estimates[i] = model(xs_valid[i].unsqueeze(0).unsqueeze(0)).detach()[0].squeeze()\n\n \n return mean_absolute_error(y_valid.cpu().numpy().reshape(y_valid.shape[0],625), result_estimates.cpu().numpy().reshape(result_estimates.shape[0],625))\n\n\ndef calculateMAERighted(model, xs_valid, y_valid):\n rowsize = np.zeros((25,25),float)\n columnsize = [rowsize for i in range(0,len(valid_index_nn))]\n result_estimates = torch.as_tensor(torch.as_tensor(columnsize,device=device), device=device)\n\n #TODO try to get rid of the unqueeze operation or at least precompute it\n #by predefining the tensor we get rid of costly copy operations and estimate faster\n for i in range(0,len(xs_valid)):\n result_estimates[i] = (model(xs_valid[i].unsqueeze(0).unsqueeze(0)).detach()[0].squeeze() > 0.).float()\n\n return mean_absolute_error(y_valid.cpu().numpy().reshape(y_valid.shape[0],625), result_estimates.cpu().numpy().reshape(result_estimates.shape[0],625))\n```", "1095764": "I am not sure a direct out-of-box use of DNN or CNN would useful. I first tried Keras out of box. It gave me about 15% error rates. Since the test data has about 15% live cells, so the start state of 0 for all entries would get 15% error rates. A highly customized CNN would be needed.\nI don't think you are doing anything wrong. Out of box NN just does not work well.", "1096019": "That is a relief.\nThanks.\nI will look through the results of the competition then and try to get a grasp on what a more ideal solution for this problem constitutes." }, "source": "meta" }