Team: TILAPIA
\nThe baseline is basically three steps:
\nWe used BERT embeddings, instead of the provided word2vec, to get the feature vectors and boost the baseline performance by 30% (0.12->0.16). For anyone wanna try BERT:
\nOne interesting point using BERT: text processing (remove punctuation, stopwords, stemmer, feature reduction via PCA) doesn't lead to significantly better results. In fact, most of the processing methods lead to noticeably worse performance. Our hypothesis is that BERT embeddings are sophisticated pre-trained models on paragraphs and generate embedding vectors when applied to new paragraphs. So processing complete sentences to a bunch of random words actually ruins the semantic meaning of sentences. We haven't tested the hypothesis yet so feel free to comment your ideas!
\nUpdate:
\nFor anyone interested in BERT - @jamesan2020 and @epickens provide great explanations and resources in the comment below!
Team: TILAPIA
\nThe baseline is basically three steps:
\nWe used BERT embeddings, instead of the provided word2vec, to get the feature vectors and boost the baseline performance by 30% (0.12->0.16). For anyone wanna try BERT:
\nOne interesting point using BERT: text processing (remove punctuation, stopwords, stemmer, feature reduction via PCA) doesn't lead to significantly better results. In fact, most of the processing methods lead to noticeably worse performance. Our hypothesis is that BERT embeddings are sophisticated pre-trained models on paragraphs and generate embedding vectors when applied to new paragraphs. So processing complete sentences to a bunch of random words actually ruins the semantic meaning of sentences. We haven't tested the hypothesis yet so feel free to comment your ideas!
\nUpdate:
\nFor anyone interested in BERT - @jamesan2020 and @epickens provide great explanations and resources in the comment below!
hey any reason you used the ‘bert-base-nli-mean-tokens’ model? on huggingface site has warning:
\n This model is deprecated. Please don't use it as it produces sentence embeddings of low quality. You can find recommended sentence embedding models here: SBERT.net - Pretrained Models
We tried 5 different models suggested in 'SBERT.net - Pretrained Models': some take much longer to run, some are much larger to download but produce similar overall results as ‘bert-base-nli-mean-tokens’. Since the descriptions are short and relatively simple (compared to long texts with complicated semantics), I would say we used 'bert-base-nli-mean-tokens' because of the tradeoff between training time and training result. We didn't have time to try more models but it's definitely interesting to see how the results look like!
", "rawMarkdown": "We tried 5 different models suggested in 'SBERT.net - Pretrained Models': some take much longer to run, some are much larger to download but produce similar overall results as ‘bert-base-nli-mean-tokens’. Since the descriptions are short and relatively simple (compared to long texts with complicated semantics), I would say we used 'bert-base-nli-mean-tokens' because of the tradeoff between training time and training result. We didn't have time to try more models but it's definitely interesting to see how the results look like!" }, { "id": 1616925, "postDate": "2021-12-13T17:58:36.210Z", "content": "If regular BERT is taking too long to compute embeddings for you I would suggest using DistilBERT. It is a lot lighter weight, but with almost the same performance as regular BERT.
\nCheck it out: https://huggingface.co/docs/transformers/model_doc/distilbert
", "rawMarkdown": "If regular BERT is taking too long to compute embeddings for you I would suggest using DistilBERT. It is a lot lighter weight, but with almost the same performance as regular BERT. \n\nCheck it out: https://huggingface.co/docs/transformers/model_doc/distilbert", "votes": 1 } ] }, { "id": 1615828, "postDate": "2021-12-12T18:33:24.830Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 1615837, "postDate": "2021-12-12T18:42:18.373Z", "content": "That's a pro-level explanation! Thanks, James!
", "rawMarkdown": "That's a pro-level explanation! Thanks, James!" }, { "id": 1616071, "postDate": "2021-12-13T05:52:04.020Z", "content": "For a little more background you might want to check out the paper \"Attention Is All You Need:\" https://arxiv.org/pdf/1706.03762.pdf
\nIt introduced transformers, which underpin many modern embedding methods. It's pretty cool and you can even get some of the stuff up in running in pytorch without too much trouble.
\nHere's some documentation for how you can go about getting started on some language modeling with transformers: https://pytorch.org/tutorials/beginner/transformer_tutorial.html
", "rawMarkdown": "For a little more background you might want to check out the paper \"Attention Is All You Need:\" https://arxiv.org/pdf/1706.03762.pdf\n\nIt introduced transformers, which underpin many modern embedding methods. It's pretty cool and you can even get some of the stuff up in running in pytorch without too much trouble. \n\nHere's some documentation for how you can go about getting started on some language modeling with transformers: https://pytorch.org/tutorials/beginner/transformer_tutorial.html\n\n", "votes": 2 }, { "id": 1616906, "postDate": "2021-12-13T17:44:28.203Z", "content": "Thanks for sharing all the great resources!
", "rawMarkdown": "Thanks for sharing all the great resources! " } ] } ], "comments": [ { "id": 1616808, "author_name": "Achal Amin", "author_url": "", "post_date": "2021-12-13T16:28:31.990000", "content": "hey any reason you used the ‘bert-base-nli-mean-tokens’ model? on huggingface site has warning:
\n This model is deprecated. Please don't use it as it produces sentence embeddings of low quality. You can find recommended sentence embedding models here: SBERT.net - Pretrained Models
We tried 5 different models suggested in 'SBERT.net - Pretrained Models': some take much longer to run, some are much larger to download but produce similar overall results as ‘bert-base-nli-mean-tokens’. Since the descriptions are short and relatively simple (compared to long texts with complicated semantics), I would say we used 'bert-base-nli-mean-tokens' because of the tradeoff between training time and training result. We didn't have time to try more models but it's definitely interesting to see how the results look like!
", "votes": 0, "replies": [] }, { "id": 1616925, "author_name": "Elliot", "author_url": "", "post_date": "2021-12-13T17:58:36.210000", "content": "If regular BERT is taking too long to compute embeddings for you I would suggest using DistilBERT. It is a lot lighter weight, but with almost the same performance as regular BERT.
\nCheck it out: https://huggingface.co/docs/transformers/model_doc/distilbert
", "votes": 1, "replies": [] } ] }, { "id": 1615828, "author_name": "", "author_url": "", "post_date": "2021-12-12T18:33:24.830000", "content": "", "votes": 0, "replies": [ { "id": 1615837, "author_name": "JiaZZZZZ", "author_url": "", "post_date": "2021-12-12T18:42:18.373000", "content": "That's a pro-level explanation! Thanks, James!
", "votes": 0, "replies": [] }, { "id": 1616071, "author_name": "Elliot", "author_url": "", "post_date": "2021-12-13T05:52:04.020000", "content": "For a little more background you might want to check out the paper \"Attention Is All You Need:\" https://arxiv.org/pdf/1706.03762.pdf
\nIt introduced transformers, which underpin many modern embedding methods. It's pretty cool and you can even get some of the stuff up in running in pytorch without too much trouble.
\nHere's some documentation for how you can go about getting started on some language modeling with transformers: https://pytorch.org/tutorials/beginner/transformer_tutorial.html
", "votes": 2, "replies": [] }, { "id": 1616906, "author_name": "JiaZZZZZ", "author_url": "", "post_date": "2021-12-13T17:44:28.203000", "content": "Thanks for sharing all the great resources!
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1615821": "Team: TILAPIA\n\nThe baseline is basically three steps:\n1. vectorize descriptions;\n2. reduce the dimension of ResNet features;\n3. Use Ridge regression model to find the relationship between descriptions and ResNet features.\n\nWe used BERT embeddings, instead of the provided word2vec, to get the feature vectors and boost the baseline performance by 30% (0.12->0.16). For anyone wanna try BERT:\n1. you need GPU and Google Colab is a good choice.\n2. download sentence_transformers and import SentenceTransformer.\n3. choose a model from https://huggingface.co/sentence-transformers?sort=downloads. We chose ‘bert-base-nli-mean-tokens’.\n\nOne interesting point using BERT: text processing (remove punctuation, stopwords, stemmer, feature reduction via PCA) doesn't lead to significantly better results. In fact, most of the processing methods lead to noticeably worse performance. Our hypothesis is that BERT embeddings are sophisticated pre-trained models on paragraphs and generate embedding vectors when applied to new paragraphs. So processing complete sentences to a bunch of random words actually ruins the semantic meaning of sentences. We haven't tested the hypothesis yet so feel free to comment your ideas!\n\nUpdate:\nFor anyone interested in BERT - @jamesan2020 and @epickens provide great explanations and resources in the comment below!", "1616808": "hey any reason you used the ‘bert-base-nli-mean-tokens’ model? on huggingface site has warning:\n This model is deprecated. Please don't use it as it produces sentence embeddings of low quality. You can find recommended sentence embedding models here: SBERT.net - Pretrained Models", "1615828": "" } }