I was wondering if adding image tags as feature data would increase overall model performance, so ran a small experiment where I used the 1000 ResNet features to predict if the image should be tagged with each tag value. Using a random forest classifier without tuning, it was possible to reach 100% training accuracy and 95-100% testing accuracy for almost all tags. This leads me to believe that the tag data wouldn't add significant value to the overall model performance, since information in the tags is already represented in the ResNet features (with a few caveats below)
\nA few tags that stood out with testing accuracy less than 95% (these specific tags could be beneficial to add as features if you validate tags are accurate) :
\nHere's some example cases where the random forests classifier made a prediction that didn't match the tags:
\nI was wondering if adding image tags as feature data would increase overall model performance, so ran a small experiment where I used the 1000 ResNet features to predict if the image should be tagged with each tag value. Using a random forest classifier without tuning, it was possible to reach 100% training accuracy and 95-100% testing accuracy for almost all tags. This leads me to believe that the tag data wouldn't add significant value to the overall model performance, since information in the tags is already represented in the ResNet features (with a few caveats below)
\nA few tags that stood out with testing accuracy less than 95% (these specific tags could be beneficial to add as features if you validate tags are accurate) :
\nHere's some example cases where the random forests classifier made a prediction that didn't match the tags:
\nEDIT: I didn't realize that tags are matched with ResNet features in the test set and not with descriptions. So your conclusion is most likely right.
\nHey! Interesting approach! I have a question though: the conclusion of your post is that adding tags to our explanatory variables won't significantly improve our predictions because the information is already in the ResNet features of the images.
\nHowever, the ResNet features are our Y (the variables we are trying to predict). I wonder if we could not conclude from your experiment that, because tags seem to be well explained by ResNet features, they could significantly improve our predictions of the latter.
What do you think?
", "rawMarkdown": "EDIT: I didn't realize that tags are matched with ResNet features in the test set and not with descriptions. So your conclusion is most likely right.\n\nHey! Interesting approach! I have a question though: the conclusion of your post is that adding tags to our explanatory variables won't significantly improve our predictions because the information is already in the ResNet features of the images.\nHowever, the ResNet features are our Y (the variables we are trying to predict). I wonder if we could not conclude from your experiment that, because tags seem to be well explained by ResNet features, they could significantly improve our predictions of the latter.\n\nWhat do you think?", "votes": 1 }, { "id": 1612426, "postDate": "2021-12-08T23:40:34.430Z", "content": "I have the same question, so I investigated the tags the opposite way, and get similar results as you, but I'm a little confused.
\nSo I used the embedded descriptions to train the embedded tags, and I also got nearly perfect results, and then I use the predicted tags to find the top 20 nearest neighbors indices without even incorporating the ResNet features in to the model. the result is almost the same as training a model between descriptions and ResNet features, and also same as training a model between descriptions and ResNet + tags.
", "rawMarkdown": "I have the same question, so I investigated the tags the opposite way, and get similar results as you, but I'm a little confused. \n\nSo I used the embedded descriptions to train the embedded tags, and I also got nearly perfect results, and then I use the predicted tags to find the top 20 nearest neighbors indices without even incorporating the ResNet features in to the model. the result is almost the same as training a model between descriptions and ResNet features, and also same as training a model between descriptions and ResNet + tags. ", "votes": 2, "replies": [ { "id": 1612444, "postDate": "2021-12-09T00:26:37.440Z", "rawMarkdown": "", "votes": 2, "isDeleted": true } ] }, { "id": 1617199, "postDate": "2021-12-14T00:01:22.827Z", "content": "My team analyzed the number of tag-description matches and came to a similar conclusion. Since the number of tags that exist are limited, and the number of image features are much higher, we saw higher performance gains when focusing on the image features over the tags.
", "rawMarkdown": "My team analyzed the number of tag-description matches and came to a similar conclusion. Since the number of tags that exist are limited, and the number of image features are much higher, we saw higher performance gains when focusing on the image features over the tags." }, { "id": 1615852, "postDate": "2021-12-12T19:20:04.457Z", "content": "Thank you for your sharing. It's very useful!
", "rawMarkdown": "Thank you for your sharing. It's very useful!" }, { "id": 1615086, "postDate": "2021-12-11T19:37:16.183Z", "content": "Thx for the tip and the documents
", "rawMarkdown": "Thx for the tip and the documents" }, { "id": 1614871, "postDate": "2021-12-11T14:25:28.647Z", "content": "thanks for your sharing! We thought tags and images are in one-to-one correspondence when we checked the data so we didn't do experiments to verify it!
", "rawMarkdown": "thanks for your sharing! We thought tags and images are in one-to-one correspondence when we checked the data so we didn't do experiments to verify it!" }, { "id": 1614240, "postDate": "2021-12-10T19:39:49.090Z", "content": "This is very helpful especially when adding upon other models, thanks so much for sharing!
", "rawMarkdown": "This is very helpful especially when adding upon other models, thanks so much for sharing!" }, { "id": 1612336, "postDate": "2021-12-08T19:11:05.293Z", "content": "Thank you for sharing this! Super helpful going forward!
", "rawMarkdown": "Thank you for sharing this! Super helpful going forward!" }, { "id": 1610537, "postDate": "2021-12-07T09:56:25.317Z", "content": "👍👍👍👍👍👍👍👍👍👍
", "rawMarkdown": "👍👍👍👍👍👍👍👍👍👍" }, { "id": 1614437, "postDate": "2021-12-11T05:17:24.360Z", "content": "Thank you so much for sharing it!
", "rawMarkdown": "Thank you so much for sharing it!" }, { "id": 1614401, "postDate": "2021-12-11T03:37:45.303Z", "content": "Thank you for the useful tips!
", "rawMarkdown": "Thank you for the useful tips! " }, { "id": 1614392, "postDate": "2021-12-11T03:18:09.683Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" } ], "comments": [ { "id": 1612377, "author_name": "rfaures", "author_url": "", "post_date": "2021-12-08T20:36:00.970000", "content": "EDIT: I didn't realize that tags are matched with ResNet features in the test set and not with descriptions. So your conclusion is most likely right.
\nHey! Interesting approach! I have a question though: the conclusion of your post is that adding tags to our explanatory variables won't significantly improve our predictions because the information is already in the ResNet features of the images.
\nHowever, the ResNet features are our Y (the variables we are trying to predict). I wonder if we could not conclude from your experiment that, because tags seem to be well explained by ResNet features, they could significantly improve our predictions of the latter.
What do you think?
", "votes": 1, "replies": [] }, { "id": 1612426, "author_name": "Caroline Ming", "author_url": "", "post_date": "2021-12-08T23:40:34.430000", "content": "I have the same question, so I investigated the tags the opposite way, and get similar results as you, but I'm a little confused.
\nSo I used the embedded descriptions to train the embedded tags, and I also got nearly perfect results, and then I use the predicted tags to find the top 20 nearest neighbors indices without even incorporating the ResNet features in to the model. the result is almost the same as training a model between descriptions and ResNet features, and also same as training a model between descriptions and ResNet + tags.
", "votes": 2, "replies": [ { "id": 1612444, "author_name": "", "author_url": "", "post_date": "2021-12-09T00:26:37.440000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 1617199, "author_name": "Jenny Juarez", "author_url": "", "post_date": "2021-12-14T00:01:22.827000", "content": "My team analyzed the number of tag-description matches and came to a similar conclusion. Since the number of tags that exist are limited, and the number of image features are much higher, we saw higher performance gains when focusing on the image features over the tags.
", "votes": 0, "replies": [] }, { "id": 1615852, "author_name": "IreneNaa", "author_url": "", "post_date": "2021-12-12T19:20:04.457000", "content": "Thank you for your sharing. It's very useful!
", "votes": 0, "replies": [] }, { "id": 1615086, "author_name": "Khaled Abughoush", "author_url": "", "post_date": "2021-12-11T19:37:16.183000", "content": "Thx for the tip and the documents
", "votes": 0, "replies": [] }, { "id": 1614871, "author_name": "Siyi Chen12138", "author_url": "", "post_date": "2021-12-11T14:25:28.647000", "content": "thanks for your sharing! We thought tags and images are in one-to-one correspondence when we checked the data so we didn't do experiments to verify it!
", "votes": 0, "replies": [] }, { "id": 1614240, "author_name": "TGCF", "author_url": "", "post_date": "2021-12-10T19:39:49.090000", "content": "This is very helpful especially when adding upon other models, thanks so much for sharing!
", "votes": 0, "replies": [] }, { "id": 1612336, "author_name": "Clare Dobbin", "author_url": "", "post_date": "2021-12-08T19:11:05.293000", "content": "Thank you for sharing this! Super helpful going forward!
", "votes": 0, "replies": [] }, { "id": 1610537, "author_name": "omar Programmer", "author_url": "", "post_date": "2021-12-07T09:56:25.317000", "content": "👍👍👍👍👍👍👍👍👍👍
", "votes": 0, "replies": [] }, { "id": 1614437, "author_name": "Kexin Chen", "author_url": "", "post_date": "2021-12-11T05:17:24.360000", "content": "Thank you so much for sharing it!
", "votes": 0, "replies": [] }, { "id": 1614401, "author_name": "Ziwei Ge", "author_url": "", "post_date": "2021-12-11T03:37:45.303000", "content": "Thank you for the useful tips!
", "votes": 0, "replies": [] }, { "id": 1614392, "author_name": "Sophia Song", "author_url": "", "post_date": "2021-12-11T03:18:09.683000", "content": "Thanks for sharing!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1608885": "I was wondering if adding image tags as feature data would increase overall model performance, so ran a small experiment where I used the 1000 ResNet features to predict if the image should be tagged with each tag value. Using a random forest classifier without tuning, it was possible to reach 100% training accuracy and 95-100% testing accuracy for almost all tags. This leads me to believe that the tag data wouldn't add significant value to the overall model performance, since information in the tags is already represented in the ResNet features (with a few caveats below) \n\nA few tags that stood out with testing accuracy less than 95% (these specific tags could be beneficial to add as features if you validate tags are accurate) :\n- person:person (86%)\n- furniture:chair (91%)\n- vehicle:car (92%)\n- furniture:dining table (93%)\n- kitchen:bottle (93%)\n- kitchen:cup (94%)\n\nHere's some example cases where the random forests classifier made a prediction that didn't match the tags:\n- Testing image 1826. Actual tags={'person:person', 'animal:giraffe'}. Predicted {'person:person'} https://drive.google.com/file/d/1CmPWCbPoMbhsocyLF324RCLOVD6OTFQG/view?usp=sharing\n- Testing image 23. Actual tags={'person:person', 'electronic:cell_phone', 'kitchen:cup', 'furniture:dining_table', 'food:carrot'}. Predicted {'furniture:dining_table', 'food:carrot'} https://drive.google.com/file/d/1ioRU8s3ULLw-bLRZsirX2uYHdeSOHRLW/view?usp=sharing\n- Testing image 953. Actual tags={'outdoor:traffic_light', 'vehicle:car'}. Predicted {'outdoor:traffic_light', 'person:person'} https://drive.google.com/file/d/1dLHyeVxoDq8jIWekP1a9xUmUC9vP1HIj/view?usp=sharing", "1612377": "EDIT: I didn't realize that tags are matched with ResNet features in the test set and not with descriptions. So your conclusion is most likely right.\n\nHey! Interesting approach! I have a question though: the conclusion of your post is that adding tags to our explanatory variables won't significantly improve our predictions because the information is already in the ResNet features of the images.\nHowever, the ResNet features are our Y (the variables we are trying to predict). I wonder if we could not conclude from your experiment that, because tags seem to be well explained by ResNet features, they could significantly improve our predictions of the latter.\n\nWhat do you think?", "1612426": "I have the same question, so I investigated the tags the opposite way, and get similar results as you, but I'm a little confused. \n\nSo I used the embedded descriptions to train the embedded tags, and I also got nearly perfect results, and then I use the predicted tags to find the top 20 nearest neighbors indices without even incorporating the ResNet features in to the model. the result is almost the same as training a model between descriptions and ResNet features, and also same as training a model between descriptions and ResNet + tags. ", "1617199": "My team analyzed the number of tag-description matches and came to a similar conclusion. Since the number of tags that exist are limited, and the number of image features are much higher, we saw higher performance gains when focusing on the image features over the tags.", "1615852": "Thank you for your sharing. It's very useful!", "1615086": "Thx for the tip and the documents", "1614871": "thanks for your sharing! We thought tags and images are in one-to-one correspondence when we checked the data so we didn't do experiments to verify it!", "1614240": "This is very helpful especially when adding upon other models, thanks so much for sharing!", "1612336": "Thank you for sharing this! Super helpful going forward!", "1610537": "👍👍👍👍👍👍👍👍👍👍", "1614437": "Thank you so much for sharing it!", "1614401": "Thank you for the useful tips! ", "1614392": "Thanks for sharing!" } }