Hey, Folks!
\nRealizing that the competition is nearly at a close, I have two questions/comments regarding the competition metrics:
\n1) I've run a few models that have scored very highly (mid/high 40s) across hold out test tests. I'm fairly certain I'm not over-fitting (at least in one sense). Clearly, I am learning a translation distribution that does not correspond as closely to the test data as I would hope. But that makes me wonder: What is different about the test data? Is it messier? Is it not really 'sentence-level'? Does it disproportionately contain new vocabulary? I'm just curious how the train/test split was created. How many folks are chasing the test set's oddities rather than developing a robust solution?
\n2) The competition metric is rather wooden. I've noticed while trying some ensembling that \"better-scoring\" models are outputting translations that are very messy (lots of stray marks, odd capitalization and spacing, etc. - which, granted, are perhaps easy post-processing fixes); at the same time, some of my models are outputting translations that are good, just not what the reference has word-for-word - e.g., \"tablet\" or \"message\" rather than \"letter\" or sometimes just outputting the cleanly transliterated source word when it hasn't seen how to translate a word. I realize there is a huge expense here, but I do wonder how shaken up the leaderboard would be if the metric were something more like COMET. Of course, this is a known limitation of metrics like BLEU and chrf. I also wonder about compute: Some of the models folks are using and methods they are using are very computationally expensive. Could that be incorporated somehow into the competition metric? Most of what I have been exploring has been relatively cheap in regard to compute. That may be very relevant depending on how DP intends to use this solution.
\nOf course, I don't expect anything to change at this point, but for future efforts, wrestling with these questions may improve the quality of the community's solutions.
\nI absolutely love the mission of this competition and want to see DP get a quality solution! I'm only disappointed that I've only been able to try my hand for the last few days.
\nThanks for the opportunity!\nEli
", "messages": [ { "id": "3427137", "postDate": "03/23/2026 18:37:25", "content": "Hey, Folks!
\nRealizing that the competition is nearly at a close, I have two questions/comments regarding the competition metrics:
\n1) I've run a few models that have scored very highly (mid/high 40s) across hold out test tests. I'm fairly certain I'm not over-fitting (at least in one sense). Clearly, I am learning a translation distribution that does not correspond as closely to the test data as I would hope. But that makes me wonder: What is different about the test data? Is it messier? Is it not really 'sentence-level'? Does it disproportionately contain new vocabulary? I'm just curious how the train/test split was created. How many folks are chasing the test set's oddities rather than developing a robust solution?
\n2) The competition metric is rather wooden. I've noticed while trying some ensembling that \"better-scoring\" models are outputting translations that are very messy (lots of stray marks, odd capitalization and spacing, etc. - which, granted, are perhaps easy post-processing fixes); at the same time, some of my models are outputting translations that are good, just not what the reference has word-for-word - e.g., \"tablet\" or \"message\" rather than \"letter\" or sometimes just outputting the cleanly transliterated source word when it hasn't seen how to translate a word. I realize there is a huge expense here, but I do wonder how shaken up the leaderboard would be if the metric were something more like COMET. Of course, this is a known limitation of metrics like BLEU and chrf. I also wonder about compute: Some of the models folks are using and methods they are using are very computationally expensive. Could that be incorporated somehow into the competition metric? Most of what I have been exploring has been relatively cheap in regard to compute. That may be very relevant depending on how DP intends to use this solution.
\nOf course, I don't expect anything to change at this point, but for future efforts, wrestling with these questions may improve the quality of the community's solutions.
\nI absolutely love the mission of this competition and want to see DP get a quality solution! I'm only disappointed that I've only been able to try my hand for the last few days.
\nThanks for the opportunity!\nEli
", "rawMarkdown": "Hey, Folks!\n\nRealizing that the competition is nearly at a close, I have two questions/comments regarding the competition metrics:\n\n1) I've run a few models that have scored very highly (mid/high 40s) across hold out test tests. I'm fairly certain I'm not over-fitting (at least in one sense). Clearly, I am learning a translation distribution that does not correspond as closely to the test data as I would hope. But that makes me wonder: What is different about the test data? Is it messier? Is it not really 'sentence-level'? Does it disproportionately contain new vocabulary? I'm just curious how the train/test split was created. How many folks are chasing the test set's oddities rather than developing a robust solution?\n\n2) The competition metric is rather wooden. I've noticed while trying some ensembling that \"better-scoring\" models are outputting translations that are very messy (lots of stray marks, odd capitalization and spacing, etc. - which, granted, are perhaps easy post-processing fixes); at the same time, some of my models are outputting translations that are good, just not what the reference has word-for-word - e.g., \"tablet\" or \"message\" rather than \"letter\" or sometimes just outputting the cleanly transliterated source word when it hasn't seen how to translate a word. I realize there is a huge expense here, but I do wonder how shaken up the leaderboard would be if the metric were something more like COMET. Of course, this is a known limitation of metrics like BLEU and chrf. I also wonder about compute: Some of the models folks are using and methods they are using are very computationally expensive. Could that be incorporated somehow into the competition metric? Most of what I have been exploring has been relatively cheap in regard to compute. That may be very relevant depending on how DP intends to use this solution.\n\nOf course, I don't expect anything to change at this point, but for future efforts, wrestling with these questions may improve the quality of the community's solutions. \n\nI absolutely love the mission of this competition and want to see DP get a quality solution! I'm only disappointed that I've only been able to try my hand for the last few days.\n\nThanks for the opportunity!\nEli", "votes": null }, { "id": "3427143", "postDate": "03/23/2026 18:46:28", "content": "I do believe test data has a lot of new vocabulary that is not present in the train data set.
", "rawMarkdown": "I do believe test data has a lot of new vocabulary that is not present in the train data set.", "votes": null }, { "id": "3427150", "postDate": "03/23/2026 18:56:15", "content": "Which is totally fine! I'm just wondering if it has a disproportionate amount of new vocab; you know what I'm saying? If they selected the test/train split inadvertently along the lines of different translators/time periods/regions/genres, it's possible that what we're training on is especially not what we're being tested on.
", "rawMarkdown": "Which is totally fine! I'm just wondering if it has a disproportionate amount of new vocab; you know what I'm saying? If they selected the test/train split inadvertently along the lines of different translators/time periods/regions/genres, it's possible that what we're training on is *especially* not what we're being tested on.", "votes": null }, { "id": "3427162", "postDate": "03/23/2026 19:13:02", "content": "I think the test data has to be based on some unpublished translations, so it's not a random split with the train set. It might come from different sources which implies some level of domain shift.
", "rawMarkdown": "I think the test data has to be based on some unpublished translations, so it's not a random split with the train set. It might come from different sources which implies some level of domain shift.", "votes": null }, { "id": "3427194", "postDate": "03/23/2026 20:09:12", "content": "True. But who translated it? And did they choose texts that were relatively representative? It just affects the competition a lot. It could mean that it is less about model-choice and hyperparameter tuning and more about data scraping which may not really be what DP is looking for.
", "rawMarkdown": "True. But who translated it? And did they choose texts that were relatively representative? It just affects the competition a lot. It could mean that it is less about model-choice and hyperparameter tuning and more about data scraping which may not really be what DP is looking for.", "votes": null }, { "id": "3427247", "postDate": "03/23/2026 21:31:31", "content": "My observations are the same. Using a stock version of Comet likely wouldn't have worked well, and they might have thought that training one on the limited number of publicly available Old Assyrian to English translations might have made it too easy to game, but even something like cosine distance between the English candidate and English reference might have worked as a third metric to balance the other two.
\nI'm very interested in the final scores, I trained a lot of models in very different ways, yet several of them that obtained the same score on the public leaderboard varied greatly in quality and style and had scores all over the place when tested on external datasets.
\nI think train.csv might not be that representative of the test data… the model that scored the best for me is one that never even saw train.csv at any point. I'm really unsure about the ones I'm choosing as final submissions as I feel we might be in for a surprise. In any case, good luck to all.
", "rawMarkdown": "My observations are the same. Using a stock version of Comet likely wouldn't have worked well, and they might have thought that training one on the limited number of publicly available Old Assyrian to English translations might have made it too easy to game, but even something like cosine distance between the English candidate and English reference might have worked as a third metric to balance the other two.\n\nI'm very interested in the final scores, I trained a lot of models in very different ways, yet several of them that obtained the same score on the public leaderboard varied greatly in quality and style and had scores all over the place when tested on external datasets. \n\nI think train.csv might not be that representative of the test data... the model that scored the best for me is one that never even saw train.csv at any point. I'm really unsure about the ones I'm choosing as final submissions as I feel we might be in for a surprise. In any case, good luck to all.", "votes": null }, { "id": "3427261", "postDate": "03/23/2026 22:02:16", "content": "Style too. Some translators prefer to make ancient texts sound ancient and/or foreign to make the source felt through the translation (Thus speaks… say to…) while others prefer to reformulate as needed to make the text feel more idiomatic and modern (From… to…).
\nSome prefer to keep foreign words when they feel that it represents a concept that doesn't exist in the target language (kārum, wabartum), while others prefer to translate to the closest approximation that most people would understand (colony, trading station.)
\nThen there's the whole normalization issue (Kaniš, Kaneš, Kanesh, etc.)
\nThere are lots of ways to translate a text. Since we're dealing with BLEU and CHRF, what's evaluated are the words and characters used rather than the meaning in a fuzzier sense. So an accurate translation in a different style can score worse than an inaccurate one with more ngram overlap with the reference.
", "rawMarkdown": "Style too. Some translators prefer to make ancient texts sound ancient and/or foreign to make the source felt through the translation (Thus speaks... say to...) while others prefer to reformulate as needed to make the text feel more idiomatic and modern (From... to...).\n\nSome prefer to keep foreign words when they feel that it represents a concept that doesn't exist in the target language (kārum, wabartum), while others prefer to translate to the closest approximation that most people would understand (colony, trading station.)\n\nThen there's the whole normalization issue (Kaniš, Kaneš, Kanesh, etc.)\n\nThere are lots of ways to translate a text. Since we're dealing with BLEU and CHRF, what's evaluated are the words and characters used rather than the meaning in a fuzzier sense. So an accurate translation in a different style can score worse than an inaccurate one with more ngram overlap with the reference.", "votes": null }, { "id": "3427263", "postDate": "03/23/2026 22:03:51", "content": "yeah goodluck
", "rawMarkdown": "yeah goodluck", "votes": null }, { "id": "3427274", "postDate": "03/23/2026 22:18:05", "content": "Yep, same bizarre experience here: Models that differed by 10+ BLEU points in local testing were getting the exact same leaderboard score. Also, a lot of normalization techniques on the source that really should have seen big gains for my method did nothing or even negatively impacted the score - which makes me think the source data might also be funky in the test set. Yeah, same regarding picking. Same to you!
", "rawMarkdown": "Yep, same bizarre experience here: Models that differed by 10+ BLEU points in local testing were getting the exact same leaderboard score. Also, a lot of normalization techniques on the source that really should have seen big gains for my method did nothing or even negatively impacted the score - which makes me think the source data might also be funky in the test set. Yeah, same regarding picking. Same to you!", "votes": null }, { "id": "3427276", "postDate": "03/23/2026 22:19:28", "content": "Yes I think emulating the test set's translator's formulaic styles will be rewarded. When inspecting the model outputs, sometimes they get penalized just for word order or stylistic choices like those you mentioned. In my experience the higher scoring models did seem to be better in general too, but I didn't get top scoring ones.
\nFrom a competition perspective I guess there are ways to probe for that and maybe some people were able to do it.
", "rawMarkdown": "Yes I think emulating the test set's translator's formulaic styles will be rewarded. When inspecting the model outputs, sometimes they get penalized just for word order or stylistic choices like those you mentioned. In my experience the higher scoring models did seem to be better in general too, but I didn't get top scoring ones.\n\nFrom a competition perspective I guess there are ways to probe for that and maybe some people were able to do it.", "votes": null }, { "id": "3427279", "postDate": "03/23/2026 22:23:29", "content": "Exactly. I work in low-resource NLP, but typically we are augmenting the work of the same individuals who provided us the training data (as applicable). When we can't do so, we see large decreases in performance.
\nI think the big question is how will these machine translations be used by DP. As references to consult? As-is translations? As drafts to edit? To be able to leverage high-resource language NLP tools by first translating to English? Maybe these metrics line up with their intent, maybe they don't 🤷.
", "rawMarkdown": "Exactly. I work in low-resource NLP, but typically we are augmenting the work of the same individuals who provided us the training data (as applicable). When we can't do so, we see large decreases in performance. \n\nI think the big question is how will these machine translations be used by DP. As references to consult? As-is translations? As drafts to edit? To be able to leverage high-resource language NLP tools by first translating to English? Maybe these metrics line up with their intent, maybe they don't 🤷.", "votes": null }, { "id": "3427282", "postDate": "03/23/2026 22:33:34", "content": "I expect there will be a lot of work on extraction and normalization from the PDF data which is pretty valuable in itself, as a big hurdle is having a homogeneous dataset to work with.
", "rawMarkdown": "I expect there will be a lot of work on extraction and normalization from the PDF data which is pretty valuable in itself, as a big hurdle is having a homogeneous dataset to work with.", "votes": null }, { "id": "3427301", "postDate": "03/23/2026 23:35:38", "content": "Yep. I think the Pdf documents have been very helpful. Once the documents were released, I completely discarded the train.csv, and built a new dataset from the PDFs. This helped a lot because it's easy to look at the line-by-line text in the docs and predict where to split the text.
\nExtracting and processing the texts, however, was a nightmare.
", "rawMarkdown": "Yep. I think the Pdf documents have been very helpful. Once the documents were released, I completely discarded the train.csv, and built a new dataset from the PDFs. This helped a lot because it's easy to look at the line-by-line text in the docs and predict where to split the text.\n\nExtracting and processing the texts, however, was a nightmare.", "votes": null }, { "id": "3427317", "postDate": "03/24/2026 00:23:16", "content": "I believe it!
", "rawMarkdown": "I believe it!", "votes": null }, { "id": "3427389", "postDate": "03/24/2026 02:55:46", "content": "Thank you for the Competition Metrics @eliclowry
", "rawMarkdown": "Thank you for the Competition Metrics @eliclowry", "votes": null } ], "comments": [ { "id": 3427143, "author_name": "paritoshtripathi5", "author_url": "", "post_date": "03/23/2026 18:46:28", "content": "I do believe test data has a lot of new vocabulary that is not present in the train data set.
", "votes": null, "replies": [ { "id": 3427150, "author_name": "eliclowry", "author_url": "", "post_date": "03/23/2026 18:56:15", "content": "Which is totally fine! I'm just wondering if it has a disproportionate amount of new vocab; you know what I'm saying? If they selected the test/train split inadvertently along the lines of different translators/time periods/regions/genres, it's possible that what we're training on is especially not what we're being tested on.
", "votes": null, "replies": [ { "id": 3427162, "author_name": "tennogh", "author_url": "", "post_date": "03/23/2026 19:13:02", "content": "I think the test data has to be based on some unpublished translations, so it's not a random split with the train set. It might come from different sources which implies some level of domain shift.
", "votes": null, "replies": [ { "id": 3427194, "author_name": "eliclowry", "author_url": "", "post_date": "03/23/2026 20:09:12", "content": "True. But who translated it? And did they choose texts that were relatively representative? It just affects the competition a lot. It could mean that it is less about model-choice and hyperparameter tuning and more about data scraping which may not really be what DP is looking for.
", "votes": null, "replies": [] }, { "id": 3427261, "author_name": "epiktistes", "author_url": "", "post_date": "03/23/2026 22:02:16", "content": "Style too. Some translators prefer to make ancient texts sound ancient and/or foreign to make the source felt through the translation (Thus speaks… say to…) while others prefer to reformulate as needed to make the text feel more idiomatic and modern (From… to…).
\nSome prefer to keep foreign words when they feel that it represents a concept that doesn't exist in the target language (kārum, wabartum), while others prefer to translate to the closest approximation that most people would understand (colony, trading station.)
\nThen there's the whole normalization issue (Kaniš, Kaneš, Kanesh, etc.)
\nThere are lots of ways to translate a text. Since we're dealing with BLEU and CHRF, what's evaluated are the words and characters used rather than the meaning in a fuzzier sense. So an accurate translation in a different style can score worse than an inaccurate one with more ngram overlap with the reference.
", "votes": null, "replies": [ { "id": 3427276, "author_name": "tennogh", "author_url": "", "post_date": "03/23/2026 22:19:28", "content": "Yes I think emulating the test set's translator's formulaic styles will be rewarded. When inspecting the model outputs, sometimes they get penalized just for word order or stylistic choices like those you mentioned. In my experience the higher scoring models did seem to be better in general too, but I didn't get top scoring ones.
\nFrom a competition perspective I guess there are ways to probe for that and maybe some people were able to do it.
", "votes": null, "replies": [] }, { "id": 3427279, "author_name": "eliclowry", "author_url": "", "post_date": "03/23/2026 22:23:29", "content": "Exactly. I work in low-resource NLP, but typically we are augmenting the work of the same individuals who provided us the training data (as applicable). When we can't do so, we see large decreases in performance.
\nI think the big question is how will these machine translations be used by DP. As references to consult? As-is translations? As drafts to edit? To be able to leverage high-resource language NLP tools by first translating to English? Maybe these metrics line up with their intent, maybe they don't 🤷.
", "votes": null, "replies": [ { "id": 3427282, "author_name": "tennogh", "author_url": "", "post_date": "03/23/2026 22:33:34", "content": "I expect there will be a lot of work on extraction and normalization from the PDF data which is pretty valuable in itself, as a big hurdle is having a homogeneous dataset to work with.
", "votes": null, "replies": [ { "id": 3427301, "author_name": "anaphase21", "author_url": "", "post_date": "03/23/2026 23:35:38", "content": "Yep. I think the Pdf documents have been very helpful. Once the documents were released, I completely discarded the train.csv, and built a new dataset from the PDFs. This helped a lot because it's easy to look at the line-by-line text in the docs and predict where to split the text.
\nExtracting and processing the texts, however, was a nightmare.
", "votes": null, "replies": [ { "id": 3427317, "author_name": "eliclowry", "author_url": "", "post_date": "03/24/2026 00:23:16", "content": "I believe it!
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] } ] }, { "id": 3427247, "author_name": "epiktistes", "author_url": "", "post_date": "03/23/2026 21:31:31", "content": "My observations are the same. Using a stock version of Comet likely wouldn't have worked well, and they might have thought that training one on the limited number of publicly available Old Assyrian to English translations might have made it too easy to game, but even something like cosine distance between the English candidate and English reference might have worked as a third metric to balance the other two.
\nI'm very interested in the final scores, I trained a lot of models in very different ways, yet several of them that obtained the same score on the public leaderboard varied greatly in quality and style and had scores all over the place when tested on external datasets.
\nI think train.csv might not be that representative of the test data… the model that scored the best for me is one that never even saw train.csv at any point. I'm really unsure about the ones I'm choosing as final submissions as I feel we might be in for a surprise. In any case, good luck to all.
", "votes": null, "replies": [ { "id": 3427263, "author_name": "", "author_url": "", "post_date": "03/23/2026 22:03:51", "content": "yeah goodluck
", "votes": null, "replies": [] }, { "id": 3427274, "author_name": "eliclowry", "author_url": "", "post_date": "03/23/2026 22:18:05", "content": "Yep, same bizarre experience here: Models that differed by 10+ BLEU points in local testing were getting the exact same leaderboard score. Also, a lot of normalization techniques on the source that really should have seen big gains for my method did nothing or even negatively impacted the score - which makes me think the source data might also be funky in the test set. Yeah, same regarding picking. Same to you!
", "votes": null, "replies": [] } ] }, { "id": 3427389, "author_name": "navneetbende", "author_url": "", "post_date": "03/24/2026 02:55:46", "content": "Thank you for the Competition Metrics @eliclowry
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3427137": "Hey, Folks!\n\nRealizing that the competition is nearly at a close, I have two questions/comments regarding the competition metrics:\n\n1) I've run a few models that have scored very highly (mid/high 40s) across hold out test tests. I'm fairly certain I'm not over-fitting (at least in one sense). Clearly, I am learning a translation distribution that does not correspond as closely to the test data as I would hope. But that makes me wonder: What is different about the test data? Is it messier? Is it not really 'sentence-level'? Does it disproportionately contain new vocabulary? I'm just curious how the train/test split was created. How many folks are chasing the test set's oddities rather than developing a robust solution?\n\n2) The competition metric is rather wooden. I've noticed while trying some ensembling that \"better-scoring\" models are outputting translations that are very messy (lots of stray marks, odd capitalization and spacing, etc. - which, granted, are perhaps easy post-processing fixes); at the same time, some of my models are outputting translations that are good, just not what the reference has word-for-word - e.g., \"tablet\" or \"message\" rather than \"letter\" or sometimes just outputting the cleanly transliterated source word when it hasn't seen how to translate a word. I realize there is a huge expense here, but I do wonder how shaken up the leaderboard would be if the metric were something more like COMET. Of course, this is a known limitation of metrics like BLEU and chrf. I also wonder about compute: Some of the models folks are using and methods they are using are very computationally expensive. Could that be incorporated somehow into the competition metric? Most of what I have been exploring has been relatively cheap in regard to compute. That may be very relevant depending on how DP intends to use this solution.\n\nOf course, I don't expect anything to change at this point, but for future efforts, wrestling with these questions may improve the quality of the community's solutions. \n\nI absolutely love the mission of this competition and want to see DP get a quality solution! I'm only disappointed that I've only been able to try my hand for the last few days.\n\nThanks for the opportunity!\nEli", "3427143": "I do believe test data has a lot of new vocabulary that is not present in the train data set.", "3427150": "Which is totally fine! I'm just wondering if it has a disproportionate amount of new vocab; you know what I'm saying? If they selected the test/train split inadvertently along the lines of different translators/time periods/regions/genres, it's possible that what we're training on is *especially* not what we're being tested on.", "3427162": "I think the test data has to be based on some unpublished translations, so it's not a random split with the train set. It might come from different sources which implies some level of domain shift.", "3427194": "True. But who translated it? And did they choose texts that were relatively representative? It just affects the competition a lot. It could mean that it is less about model-choice and hyperparameter tuning and more about data scraping which may not really be what DP is looking for.", "3427247": "My observations are the same. Using a stock version of Comet likely wouldn't have worked well, and they might have thought that training one on the limited number of publicly available Old Assyrian to English translations might have made it too easy to game, but even something like cosine distance between the English candidate and English reference might have worked as a third metric to balance the other two.\n\nI'm very interested in the final scores, I trained a lot of models in very different ways, yet several of them that obtained the same score on the public leaderboard varied greatly in quality and style and had scores all over the place when tested on external datasets. \n\nI think train.csv might not be that representative of the test data... the model that scored the best for me is one that never even saw train.csv at any point. I'm really unsure about the ones I'm choosing as final submissions as I feel we might be in for a surprise. In any case, good luck to all.", "3427261": "Style too. Some translators prefer to make ancient texts sound ancient and/or foreign to make the source felt through the translation (Thus speaks... say to...) while others prefer to reformulate as needed to make the text feel more idiomatic and modern (From... to...).\n\nSome prefer to keep foreign words when they feel that it represents a concept that doesn't exist in the target language (kārum, wabartum), while others prefer to translate to the closest approximation that most people would understand (colony, trading station.)\n\nThen there's the whole normalization issue (Kaniš, Kaneš, Kanesh, etc.)\n\nThere are lots of ways to translate a text. Since we're dealing with BLEU and CHRF, what's evaluated are the words and characters used rather than the meaning in a fuzzier sense. So an accurate translation in a different style can score worse than an inaccurate one with more ngram overlap with the reference.", "3427263": "yeah goodluck", "3427274": "Yep, same bizarre experience here: Models that differed by 10+ BLEU points in local testing were getting the exact same leaderboard score. Also, a lot of normalization techniques on the source that really should have seen big gains for my method did nothing or even negatively impacted the score - which makes me think the source data might also be funky in the test set. Yeah, same regarding picking. Same to you!", "3427276": "Yes I think emulating the test set's translator's formulaic styles will be rewarded. When inspecting the model outputs, sometimes they get penalized just for word order or stylistic choices like those you mentioned. In my experience the higher scoring models did seem to be better in general too, but I didn't get top scoring ones.\n\nFrom a competition perspective I guess there are ways to probe for that and maybe some people were able to do it.", "3427279": "Exactly. I work in low-resource NLP, but typically we are augmenting the work of the same individuals who provided us the training data (as applicable). When we can't do so, we see large decreases in performance. \n\nI think the big question is how will these machine translations be used by DP. As references to consult? As-is translations? As drafts to edit? To be able to leverage high-resource language NLP tools by first translating to English? Maybe these metrics line up with their intent, maybe they don't 🤷.", "3427282": "I expect there will be a lot of work on extraction and normalization from the PDF data which is pretty valuable in itself, as a big hurdle is having a homogeneous dataset to work with.", "3427301": "Yep. I think the Pdf documents have been very helpful. Once the documents were released, I completely discarded the train.csv, and built a new dataset from the PDFs. This helped a lot because it's easy to look at the line-by-line text in the docs and predict where to split the text.\n\nExtracting and processing the texts, however, was a nightmare.", "3427317": "I believe it!", "3427389": "Thank you for the Competition Metrics @eliclowry" }, "source": "meta" }