I am new to kaggle and have 2 questions about entering prize competitions:
\n\nAm I allowed to add to the data set myself, like by manually categorizing the 25,000 dog/cat images prior to training the nn with them, to improve the accuracy of my nn on the unlabeled images in the test set?
Without modifying the kaggle data set, can I pretrain my nn on other similar data sets that I have access to? Like could I first train my kaggle nn on 100,000 of my own cat/dog photos and then have it go to work on the 25,000 images from kaggle?
I am new to kaggle and have 2 questions about entering prize competitions:
\n\nAm I allowed to add to the data set myself, like by manually categorizing the 25,000 dog/cat images prior to training the nn with them, to improve the accuracy of my nn on the unlabeled images in the test set?
Without modifying the kaggle data set, can I pretrain my nn on other similar data sets that I have access to? Like could I first train my kaggle nn on 100,000 of my own cat/dog photos and then have it go to work on the 25,000 images from kaggle?
I'm relatively new to Kaggle, so I'll try to answer your questions. I'm sure if I'm incorrect, someone will chime in.
\n\nThe rules for each prize competition are pretty well spelled out. This particular competition--Dogs vs Cats--is not a "prized" competition. It's in the "Playground" category. So, the rules are pretty lax. Yes, for this particular competition, you could go and categorize the images manually (I'm not sure how adding categories will improve classification, but you never know I suppose). In some cases, you could even cull images manually (like in one of the training sets, there's a picture of a dog's paw, but no dog . . . this is probably not helping classification much . . . it might even be providing confusion for the overall class of "dog"). But for "prized" competitions, often there are strict rules about how you handle the data, how many people can be on a team, how many team can merge or if they can merge at all, etc. So, the answer to both of your questions is "it depends". It depends mainly on the rules of that particular competition.
\n\nFor this competition, I think it's interesting to pre-train on 100,000 separate images and in then fine tune with Kaggle's training images. But where did you get 100K images of doggies and kitties? Some people already do this by using well-known models, such as VGG19 or Inception, for the front-end weights and then fine-tune the backend neural net with the Kaggle data.
", "rawMarkdown": "I'm relatively new to Kaggle, so I'll try to answer your questions. I'm sure if I'm incorrect, someone will chime in.\r\n\r\nThe rules for each prize competition are pretty well spelled out. This particular competition--Dogs vs Cats--is not a \"prized\" competition. It's in the \"Playground\" category. So, the rules are pretty lax. Yes, for this particular competition, you could go and categorize the images manually (I'm not sure how adding categories will improve classification, but you never know I suppose). In some cases, you could even cull images manually (like in one of the training sets, there's a picture of a dog's paw, but no dog . . . this is probably not helping classification much . . . it might even be providing confusion for the overall class of \"dog\"). But for \"prized\" competitions, often there are strict rules about how you handle the data, how many people can be on a team, how many team can merge or if they can merge at all, etc. So, the answer to both of your questions is \"it depends\". It depends mainly on the rules of that particular competition.\r\n\r\nFor this competition, I think it's interesting to pre-train on 100,000 separate images and in then fine tune with Kaggle's training images. But where did you get 100K images of doggies and kitties? Some people already do this by using well-known models, such as VGG19 or Inception, for the front-end weights and then fine-tune the backend neural net with the Kaggle data.", "votes": null }, { "id": "139105", "postDate": "10/12/2016 14:44:52", "content": "Dogs vs Cats was an actual competition back in 2013, see https://www.kaggle.com/c/dogs-vs-cats . This new version is the playground version of the original competition. The original version did not have any "external data" rules, but I was not sure if it was more of an unwritten rule for all the competitions. I do see now that some prize competitions do have rules against using external data.
\n\nI don't have any dog/cat images, I was just giving that as a hypothetical example.
\nYes, fine-tuning an existing model like VGG19 or Inception is like what I was talking about.
Thanks for answering, now I know how to approach these issues in the competitions.
", "rawMarkdown": "Dogs vs Cats was an actual competition back in 2013, see https://www.kaggle.com/c/dogs-vs-cats . This new version is the playground version of the original competition. The original version did not have any \"external data\" rules, but I was not sure if it was more of an unwritten rule for all the competitions. I do see now that some prize competitions do have rules against using external data.\r\n\r\nI don't have any dog/cat images, I was just giving that as a hypothetical example. \r\nYes, fine-tuning an existing model like VGG19 or Inception is like what I was talking about.\r\n\r\nThanks for answering, now I know how to approach these issues in the competitions.", "votes": null } ], "comments": [ { "id": 139097, "author_name": "edjansing", "author_url": "", "post_date": "10/12/2016 14:31:03", "content": "I'm relatively new to Kaggle, so I'll try to answer your questions. I'm sure if I'm incorrect, someone will chime in.
\n\nThe rules for each prize competition are pretty well spelled out. This particular competition--Dogs vs Cats--is not a "prized" competition. It's in the "Playground" category. So, the rules are pretty lax. Yes, for this particular competition, you could go and categorize the images manually (I'm not sure how adding categories will improve classification, but you never know I suppose). In some cases, you could even cull images manually (like in one of the training sets, there's a picture of a dog's paw, but no dog . . . this is probably not helping classification much . . . it might even be providing confusion for the overall class of "dog"). But for "prized" competitions, often there are strict rules about how you handle the data, how many people can be on a team, how many team can merge or if they can merge at all, etc. So, the answer to both of your questions is "it depends". It depends mainly on the rules of that particular competition.
\n\nFor this competition, I think it's interesting to pre-train on 100,000 separate images and in then fine tune with Kaggle's training images. But where did you get 100K images of doggies and kitties? Some people already do this by using well-known models, such as VGG19 or Inception, for the front-end weights and then fine-tune the backend neural net with the Kaggle data.
", "votes": null, "replies": [] }, { "id": 139105, "author_name": "impulsecorp", "author_url": "", "post_date": "10/12/2016 14:44:52", "content": "Dogs vs Cats was an actual competition back in 2013, see https://www.kaggle.com/c/dogs-vs-cats . This new version is the playground version of the original competition. The original version did not have any "external data" rules, but I was not sure if it was more of an unwritten rule for all the competitions. I do see now that some prize competitions do have rules against using external data.
\n\nI don't have any dog/cat images, I was just giving that as a hypothetical example.
\nYes, fine-tuning an existing model like VGG19 or Inception is like what I was talking about.
Thanks for answering, now I know how to approach these issues in the competitions.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "138995": "I am new to kaggle and have 2 questions about entering prize competitions:\r\n\r\n1. Am I allowed to add to the data set myself, like by manually categorizing the 25,000 dog/cat images prior to training the nn with them, to improve the accuracy of my nn on the unlabeled images in the test set?\r\n\r\n2. Without modifying the kaggle data set, can I pretrain my nn on other similar data sets that I have access to? Like could I first train my kaggle nn on 100,000 of my own cat/dog photos and then have it go to work on the 25,000 images from kaggle?", "139097": "I'm relatively new to Kaggle, so I'll try to answer your questions. I'm sure if I'm incorrect, someone will chime in.\r\n\r\nThe rules for each prize competition are pretty well spelled out. This particular competition--Dogs vs Cats--is not a \"prized\" competition. It's in the \"Playground\" category. So, the rules are pretty lax. Yes, for this particular competition, you could go and categorize the images manually (I'm not sure how adding categories will improve classification, but you never know I suppose). In some cases, you could even cull images manually (like in one of the training sets, there's a picture of a dog's paw, but no dog . . . this is probably not helping classification much . . . it might even be providing confusion for the overall class of \"dog\"). But for \"prized\" competitions, often there are strict rules about how you handle the data, how many people can be on a team, how many team can merge or if they can merge at all, etc. So, the answer to both of your questions is \"it depends\". It depends mainly on the rules of that particular competition.\r\n\r\nFor this competition, I think it's interesting to pre-train on 100,000 separate images and in then fine tune with Kaggle's training images. But where did you get 100K images of doggies and kitties? Some people already do this by using well-known models, such as VGG19 or Inception, for the front-end weights and then fine-tune the backend neural net with the Kaggle data.", "139105": "Dogs vs Cats was an actual competition back in 2013, see https://www.kaggle.com/c/dogs-vs-cats . This new version is the playground version of the original competition. The original version did not have any \"external data\" rules, but I was not sure if it was more of an unwritten rule for all the competitions. I do see now that some prize competitions do have rules against using external data.\r\n\r\nI don't have any dog/cat images, I was just giving that as a hypothetical example. \r\nYes, fine-tuning an existing model like VGG19 or Inception is like what I was talking about.\r\n\r\nThanks for answering, now I know how to approach these issues in the competitions." }, "source": "meta" }