\n
![\"https://i.imgur.com/hFsnexq.png\"](\"https://i.imgur.com/hFsnexq.png\")
The race is finally over, from the very beginning when the outer circle of the medal position upwards when the high spirits, to the code debug after occupying the front row when the complacency, to the second half of the stage fluctuates back and forth the sense of roller coaster, until the end of the race after the small ripples. This emotional journey is much more than the actual feeling brought by the race itself. In the end, the results were poor, at least a silver medal 🥈, and some results were considered to be a gold medal, but for various reasons (which I will discuss later) I could not submit my best results. Of course, the most important thing about the competition is not the results, now talk about the feelings and gains after more than 2 months of competition.
\n1. Difficulties
\nAt the beginning, when I faced this competition, I thought it was just Ner or QA, which is a common type of competition, and I had previous experience in related competitions. The most abundant time in school is not enough, basically only a few hours after work every day, so in the beginning of the period, a lot of time on weekends will be used to do the competition, because the joy of the competition is really attractive. The problems with this piece gradually settled in as the first few weeks were struggled with, and could only be alleviated by the relatively large time span of over 2 months.
\nBut as the model added and modified features gradually, the number of debug also rose, the first mainly used to unblock the entire pre-processing, training process is also more and more time-consuming, this problem is more prominent in the tone of the official white whoring kaggle machine! The problem of cpu makes the official training time, excluding debug, doubled several times to 30 minutes each time (you know I have so many hours of effective time off work every day 😞), while the official gpu The time limit for resources and the 9-hour limit for each training session made me experience gpu anxiety. After a few days of review and practice, using offline multi-threaded processing can significantly reduce the pre-processing time. Also, for training, renting a gpu cloud server properly is an option, but it is still not cheap, requiring tens of soft dollars for a training session.
What's worse is that I encountered the problem of \"memory explosion\" that few people would encounter, which is the problem of insufficient memory when making predictions on a private test set. Of course, this problem is caused by the roughness of data pre-processing and training process construction when training models by individuals. Since I was mostly doing domestic competitions before, I used enough resources from the lab, so I naturally did not pay attention to the problem of insufficient memory, but the kaggle competition has a 13g memory limit, which made me spend 1-2 weeks on dealing with this problem. Finally, I solved the problem by migrating the pre-trained inputs such as input id, type id to the dataset class for batch generation.
\nTo summarize, the problem of having little time to face the competition can be alleviated by participating in a long time span and making good use of the weekend. The use of multiple threads can effectively reduce the pre-processing time. Through process optimization, it can reduce the memory occupation. Properly renting a gpu cloud server can solve the urgent problem (of course I am also considering purchasing my own server host, although the current graphics card premium is too high).
2. Data
\nWhether it's a competition or a research lab, the model's understanding of the data determines whether your model is over-fitted or can have good generalization performance. Therefore the amount of quality data set can greatly influence the final result of your model, which is why many competitions do not allow the act of manually labeling additional data.
\nIn this competition, the primary thing that was performed was the data analysis, which determined the goal of our task, what kind of model we needed, and how we performed the pre-processing and post-processing. I personally use a model for reading comprehension, but of course it can also be seen as a ner task. It is especially important to be able to have a whole overview of the data after data analysis, such as the number of texts, the length distribution of texts, the number and length distribution of tags, the share of different tags, how serious the quantitative imbalance is, etc. This is a particularly important step.
In this contest, I also performed the above-mentioned operations, and the results were really surprising.
\nProblem 1:
\nOfficially, more than 1w samples were given, and the length of the samples was much longer than 512 so a sliding window, a common preprocessing method in reading comprehension, was needed. The sample labels are less than 200, which means that each label may be contained in 50 samples on average. It turns out that due to the long tail phenomenon, the label adni appears in more than 60% of the samples, which will prevent the model from learning other types of labels, and also make our validation meaningless due to the presence of leak, which seriously affects our training.
\nProblem 2:
\nIn addition to this, the presence of labeled text with inclusion relationships as TP at the same time in the labels makes it impossible to perform effective post-processing to filter out unsuitable FPs when generating prediction results (of course there is no good solution to this problem in the end).
\nProblem 3:
\nFinally, there is another problem with huge impact, that is, the training set is not completely labeled well. That is, there are some valid labels in the training set that are not labeled. Therefore, we cannot judge the performance of the model by the effect of the validation set, which can only be reflected by the score on the public test set.
Since problem one is a common problem and it is not the pain point of this competition, it is not optimized. For problems two and three, I tried a series of approaches to identify some strategies that can alleviate it.
\n- Division of training set and validation set
\nDue to the previous problem, to perform validation as effectively as possible, it should make the validation set without the labels seen during training, so I used the way of dividing the set by using and checking the set, so that the training set and the validation set do not have the same labels, and then the validation is closest to the score of the private dataset, but due to the poor results on the public test set, I did not optimize the strategy based on this However, due to the poor results on the public test set, I did not optimize the strategy on this basis, but focused on the fusion with the text matching method, which is the reason why my final model did not work well. Also it might be better if a similar operation could be performed on span, as this would allow a validation set containing a sufficient number of labels to verify the effect, although the score is lower.
\n- Input of text
\nSince there are some unlabeled labels in the training set, it is necessary to limit the length of the input qa model text while training without using negative samples to minimize the negative effect of hidden samples. I also added some possible valid labels (these labels are from the bigger-govt-dataset-list, without verifying its validity) to expand the training sample.
\n- Post-processing
\nFor post-processing, I am directly using the size judgment of logit values. Also when predicting the results, the answers do not have overlapping parts, and finally use whether to include Remove the duplicate prediction labels.
\n- Semi-supervised
\nUse the labels with higher logit values in all the results predicted by the model, while the length of the words is between 2 and 10, and include some special words such as survey, as the labels used for string matching, to improve the recall of the private test set.
Finally, the public validation set shows that a text length of 128 or 256 can have a score of about 0.5. In the private test set it can reach 0.42, but I did not choose this result because the score in the public test set is too low, which is also a reflection of problem 2.
\n
Finally, enjoy the game!
", "messages": [ { "id": 1364196, "postDate": "2021-06-24T17:38:05.243Z", "content": "The race is finally over, from the very beginning when the outer circle of the medal position upwards when the high spirits, to the code debug after occupying the front row when the complacency, to the second half of the stage fluctuates back and forth the sense of roller coaster, until the end of the race after the small ripples. This emotional journey is much more than the actual feeling brought by the race itself. In the end, the results were poor, at least a silver medal 🥈, and some results were considered to be a gold medal, but for various reasons (which I will discuss later) I could not submit my best results. Of course, the most important thing about the competition is not the results, now talk about the feelings and gains after more than 2 months of competition.
\n1. Difficulties
\nAt the beginning, when I faced this competition, I thought it was just Ner or QA, which is a common type of competition, and I had previous experience in related competitions. The most abundant time in school is not enough, basically only a few hours after work every day, so in the beginning of the period, a lot of time on weekends will be used to do the competition, because the joy of the competition is really attractive. The problems with this piece gradually settled in as the first few weeks were struggled with, and could only be alleviated by the relatively large time span of over 2 months.
\nBut as the model added and modified features gradually, the number of debug also rose, the first mainly used to unblock the entire pre-processing, training process is also more and more time-consuming, this problem is more prominent in the tone of the official white whoring kaggle machine! The problem of cpu makes the official training time, excluding debug, doubled several times to 30 minutes each time (you know I have so many hours of effective time off work every day 😞), while the official gpu The time limit for resources and the 9-hour limit for each training session made me experience gpu anxiety. After a few days of review and practice, using offline multi-threaded processing can significantly reduce the pre-processing time. Also, for training, renting a gpu cloud server properly is an option, but it is still not cheap, requiring tens of soft dollars for a training session.
What's worse is that I encountered the problem of \"memory explosion\" that few people would encounter, which is the problem of insufficient memory when making predictions on a private test set. Of course, this problem is caused by the roughness of data pre-processing and training process construction when training models by individuals. Since I was mostly doing domestic competitions before, I used enough resources from the lab, so I naturally did not pay attention to the problem of insufficient memory, but the kaggle competition has a 13g memory limit, which made me spend 1-2 weeks on dealing with this problem. Finally, I solved the problem by migrating the pre-trained inputs such as input id, type id to the dataset class for batch generation.
\nTo summarize, the problem of having little time to face the competition can be alleviated by participating in a long time span and making good use of the weekend. The use of multiple threads can effectively reduce the pre-processing time. Through process optimization, it can reduce the memory occupation. Properly renting a gpu cloud server can solve the urgent problem (of course I am also considering purchasing my own server host, although the current graphics card premium is too high).
2. Data
\nWhether it's a competition or a research lab, the model's understanding of the data determines whether your model is over-fitted or can have good generalization performance. Therefore the amount of quality data set can greatly influence the final result of your model, which is why many competitions do not allow the act of manually labeling additional data.
\nIn this competition, the primary thing that was performed was the data analysis, which determined the goal of our task, what kind of model we needed, and how we performed the pre-processing and post-processing. I personally use a model for reading comprehension, but of course it can also be seen as a ner task. It is especially important to be able to have a whole overview of the data after data analysis, such as the number of texts, the length distribution of texts, the number and length distribution of tags, the share of different tags, how serious the quantitative imbalance is, etc. This is a particularly important step.
In this contest, I also performed the above-mentioned operations, and the results were really surprising.
\nProblem 1:
\nOfficially, more than 1w samples were given, and the length of the samples was much longer than 512 so a sliding window, a common preprocessing method in reading comprehension, was needed. The sample labels are less than 200, which means that each label may be contained in 50 samples on average. It turns out that due to the long tail phenomenon, the label adni appears in more than 60% of the samples, which will prevent the model from learning other types of labels, and also make our validation meaningless due to the presence of leak, which seriously affects our training.
\nProblem 2:
\nIn addition to this, the presence of labeled text with inclusion relationships as TP at the same time in the labels makes it impossible to perform effective post-processing to filter out unsuitable FPs when generating prediction results (of course there is no good solution to this problem in the end).
\nProblem 3:
\nFinally, there is another problem with huge impact, that is, the training set is not completely labeled well. That is, there are some valid labels in the training set that are not labeled. Therefore, we cannot judge the performance of the model by the effect of the validation set, which can only be reflected by the score on the public test set.
Since problem one is a common problem and it is not the pain point of this competition, it is not optimized. For problems two and three, I tried a series of approaches to identify some strategies that can alleviate it.
\n- Division of training set and validation set
\nDue to the previous problem, to perform validation as effectively as possible, it should make the validation set without the labels seen during training, so I used the way of dividing the set by using and checking the set, so that the training set and the validation set do not have the same labels, and then the validation is closest to the score of the private dataset, but due to the poor results on the public test set, I did not optimize the strategy based on this However, due to the poor results on the public test set, I did not optimize the strategy on this basis, but focused on the fusion with the text matching method, which is the reason why my final model did not work well. Also it might be better if a similar operation could be performed on span, as this would allow a validation set containing a sufficient number of labels to verify the effect, although the score is lower.
\n- Input of text
\nSince there are some unlabeled labels in the training set, it is necessary to limit the length of the input qa model text while training without using negative samples to minimize the negative effect of hidden samples. I also added some possible valid labels (these labels are from the bigger-govt-dataset-list, without verifying its validity) to expand the training sample.
\n- Post-processing
\nFor post-processing, I am directly using the size judgment of logit values. Also when predicting the results, the answers do not have overlapping parts, and finally use whether to include Remove the duplicate prediction labels.
\n- Semi-supervised
\nUse the labels with higher logit values in all the results predicted by the model, while the length of the words is between 2 and 10, and include some special words such as survey, as the labels used for string matching, to improve the recall of the private test set.
Finally, the public validation set shows that a text length of 128 or 256 can have a score of about 0.5. In the private test set it can reach 0.42, but I did not choose this result because the score in the public test set is too low, which is also a reflection of problem 2.
\n
Finally, enjoy the game!
", "rawMarkdown": "The race is finally over, from the very beginning when the outer circle of the medal position upwards when the high spirits, to the code debug after occupying the front row when the complacency, to the second half of the stage fluctuates back and forth the sense of roller coaster, until the end of the race after the small ripples. This emotional journey is much more than the actual feeling brought by the race itself. In the end, the results were poor, at least a silver medal 🥈, and some results were considered to be a gold medal, but for various reasons (which I will discuss later) I could not submit my best results. Of course, the most important thing about the competition is not the results, now talk about the feelings and gains after more than 2 months of competition.\n\n**1. Difficulties**\nAt the beginning, when I faced this competition, I thought it was just Ner or QA, which is a common type of competition, and I had previous experience in related competitions. The most abundant time in school is not enough, basically only a few hours after work every day, so in the beginning of the period, a lot of time on weekends will be used to do the competition, because the joy of the competition is really attractive. The problems with this piece gradually settled in as the first few weeks were struggled with, and could only be alleviated by the relatively large time span of over 2 months.\nBut as the model added and modified features gradually, the number of debug also rose, the first mainly used to unblock the entire pre-processing, training process is also more and more time-consuming, this problem is more prominent in the tone of the official white whoring kaggle machine! The problem of cpu makes the official training time, excluding debug, doubled several times to 30 minutes each time (you know I have so many hours of effective time off work every day 😞), while the official gpu The time limit for resources and the 9-hour limit for each training session made me experience gpu anxiety. After a few days of review and practice, using offline multi-threaded processing can significantly reduce the pre-processing time. Also, for training, renting a gpu cloud server properly is an option, but it is still not cheap, requiring tens of soft dollars for a training session.\n\nWhat's worse is that I encountered the problem of \"memory explosion\" that few people would encounter, which is the problem of insufficient memory when making predictions on a private test set. Of course, this problem is caused by the roughness of data pre-processing and training process construction when training models by individuals. Since I was mostly doing domestic competitions before, I used enough resources from the lab, so I naturally did not pay attention to the problem of insufficient memory, but the kaggle competition has a 13g memory limit, which made me spend 1-2 weeks on dealing with this problem. Finally, I solved the problem by migrating the pre-trained inputs such as input id, type id to the dataset class for batch generation.\nTo summarize, the problem of having little time to face the competition can be alleviated by participating in a long time span and making good use of the weekend. The use of multiple threads can effectively reduce the pre-processing time. Through process optimization, it can reduce the memory occupation. Properly renting a gpu cloud server can solve the urgent problem (of course I am also considering purchasing my own server host, although the current graphics card premium is too high).\n\n**2. Data**\nWhether it's a competition or a research lab, the model's understanding of the data determines whether your model is over-fitted or can have good generalization performance. Therefore the amount of quality data set can greatly influence the final result of your model, which is why many competitions do not allow the act of manually labeling additional data.\nIn this competition, the primary thing that was performed was the data analysis, which determined the goal of our task, what kind of model we needed, and how we performed the pre-processing and post-processing. I personally use a model for reading comprehension, but of course it can also be seen as a ner task. It is especially important to be able to have a whole overview of the data after data analysis, such as the number of texts, the length distribution of texts, the number and length distribution of tags, the share of different tags, how serious the quantitative imbalance is, etc. This is a particularly important step.\n\nIn this contest, I also performed the above-mentioned operations, and the results were really surprising.\n**Problem 1:**\nOfficially, more than 1w samples were given, and the length of the samples was much longer than 512 so a sliding window, a common preprocessing method in reading comprehension, was needed. The sample labels are less than 200, which means that each label may be contained in 50 samples on average. It turns out that due to the long tail phenomenon, the label adni appears in more than 60% of the samples, which will prevent the model from learning other types of labels, and also make our validation meaningless due to the presence of leak, which seriously affects our training.\n**Problem 2:**\nIn addition to this, the presence of labeled text with inclusion relationships as TP at the same time in the labels makes it impossible to perform effective post-processing to filter out unsuitable FPs when generating prediction results (of course there is no good solution to this problem in the end).\n**Problem 3:**\nFinally, there is another problem with huge impact, that is, the training set is not completely labeled well. That is, there are some valid labels in the training set that are not labeled. Therefore, we cannot judge the performance of the model by the effect of the validation set, which can only be reflected by the score on the public test set.\n\nSince problem one is a common problem and it is not the pain point of this competition, it is not optimized. For problems two and three, I tried a series of approaches to identify some strategies that can alleviate it.\n**- Division of training set and validation set**\nDue to the previous problem, to perform validation as effectively as possible, it should make the validation set without the labels seen during training, so I used the way of dividing the set by using and checking the set, so that the training set and the validation set do not have the same labels, and then the validation is closest to the score of the private dataset, but due to the poor results on the public test set, I did not optimize the strategy based on this However, due to the poor results on the public test set, I did not optimize the strategy on this basis, but focused on the fusion with the text matching method, which is the reason why my final model did not work well. Also it might be better if a similar operation could be performed on span, as this would allow a validation set containing a sufficient number of labels to verify the effect, although the score is lower.\n**- Input of text**\nSince there are some unlabeled labels in the training set, it is necessary to limit the length of the input qa model text while training without using negative samples to minimize the negative effect of hidden samples. I also added some possible valid labels (these labels are from the bigger-govt-dataset-list, without verifying its validity) to expand the training sample.\n**- Post-processing**\nFor post-processing, I am directly using the size judgment of logit values. Also when predicting the results, the answers do not have overlapping parts, and finally use whether to include Remove the duplicate prediction labels.\n**- Semi-supervised**\nUse the labels with higher logit values in all the results predicted by the model, while the length of the words is between 2 and 10, and include some special words such as survey, as the labels used for string matching, to improve the recall of the private test set.\n\n\nFinally, the public validation set shows that a text length of 128 or 256 can have a score of about 0.5. In the private test set it can reach 0.42, but I did not choose this result because the score in the public test set is too low, which is also a reflection of problem 2.\n\n\n**Finally, enjoy the game!**\n", "votes": 8 } ], "comments": [], "raw_markdown_by_id": { "1364196": "The race is finally over, from the very beginning when the outer circle of the medal position upwards when the high spirits, to the code debug after occupying the front row when the complacency, to the second half of the stage fluctuates back and forth the sense of roller coaster, until the end of the race after the small ripples. This emotional journey is much more than the actual feeling brought by the race itself. In the end, the results were poor, at least a silver medal 🥈, and some results were considered to be a gold medal, but for various reasons (which I will discuss later) I could not submit my best results. Of course, the most important thing about the competition is not the results, now talk about the feelings and gains after more than 2 months of competition.\n\n**1. Difficulties**\nAt the beginning, when I faced this competition, I thought it was just Ner or QA, which is a common type of competition, and I had previous experience in related competitions. The most abundant time in school is not enough, basically only a few hours after work every day, so in the beginning of the period, a lot of time on weekends will be used to do the competition, because the joy of the competition is really attractive. The problems with this piece gradually settled in as the first few weeks were struggled with, and could only be alleviated by the relatively large time span of over 2 months.\nBut as the model added and modified features gradually, the number of debug also rose, the first mainly used to unblock the entire pre-processing, training process is also more and more time-consuming, this problem is more prominent in the tone of the official white whoring kaggle machine! The problem of cpu makes the official training time, excluding debug, doubled several times to 30 minutes each time (you know I have so many hours of effective time off work every day 😞), while the official gpu The time limit for resources and the 9-hour limit for each training session made me experience gpu anxiety. After a few days of review and practice, using offline multi-threaded processing can significantly reduce the pre-processing time. Also, for training, renting a gpu cloud server properly is an option, but it is still not cheap, requiring tens of soft dollars for a training session.\n\nWhat's worse is that I encountered the problem of \"memory explosion\" that few people would encounter, which is the problem of insufficient memory when making predictions on a private test set. Of course, this problem is caused by the roughness of data pre-processing and training process construction when training models by individuals. Since I was mostly doing domestic competitions before, I used enough resources from the lab, so I naturally did not pay attention to the problem of insufficient memory, but the kaggle competition has a 13g memory limit, which made me spend 1-2 weeks on dealing with this problem. Finally, I solved the problem by migrating the pre-trained inputs such as input id, type id to the dataset class for batch generation.\nTo summarize, the problem of having little time to face the competition can be alleviated by participating in a long time span and making good use of the weekend. The use of multiple threads can effectively reduce the pre-processing time. Through process optimization, it can reduce the memory occupation. Properly renting a gpu cloud server can solve the urgent problem (of course I am also considering purchasing my own server host, although the current graphics card premium is too high).\n\n**2. Data**\nWhether it's a competition or a research lab, the model's understanding of the data determines whether your model is over-fitted or can have good generalization performance. Therefore the amount of quality data set can greatly influence the final result of your model, which is why many competitions do not allow the act of manually labeling additional data.\nIn this competition, the primary thing that was performed was the data analysis, which determined the goal of our task, what kind of model we needed, and how we performed the pre-processing and post-processing. I personally use a model for reading comprehension, but of course it can also be seen as a ner task. It is especially important to be able to have a whole overview of the data after data analysis, such as the number of texts, the length distribution of texts, the number and length distribution of tags, the share of different tags, how serious the quantitative imbalance is, etc. This is a particularly important step.\n\nIn this contest, I also performed the above-mentioned operations, and the results were really surprising.\n**Problem 1:**\nOfficially, more than 1w samples were given, and the length of the samples was much longer than 512 so a sliding window, a common preprocessing method in reading comprehension, was needed. The sample labels are less than 200, which means that each label may be contained in 50 samples on average. It turns out that due to the long tail phenomenon, the label adni appears in more than 60% of the samples, which will prevent the model from learning other types of labels, and also make our validation meaningless due to the presence of leak, which seriously affects our training.\n**Problem 2:**\nIn addition to this, the presence of labeled text with inclusion relationships as TP at the same time in the labels makes it impossible to perform effective post-processing to filter out unsuitable FPs when generating prediction results (of course there is no good solution to this problem in the end).\n**Problem 3:**\nFinally, there is another problem with huge impact, that is, the training set is not completely labeled well. That is, there are some valid labels in the training set that are not labeled. Therefore, we cannot judge the performance of the model by the effect of the validation set, which can only be reflected by the score on the public test set.\n\nSince problem one is a common problem and it is not the pain point of this competition, it is not optimized. For problems two and three, I tried a series of approaches to identify some strategies that can alleviate it.\n**- Division of training set and validation set**\nDue to the previous problem, to perform validation as effectively as possible, it should make the validation set without the labels seen during training, so I used the way of dividing the set by using and checking the set, so that the training set and the validation set do not have the same labels, and then the validation is closest to the score of the private dataset, but due to the poor results on the public test set, I did not optimize the strategy based on this However, due to the poor results on the public test set, I did not optimize the strategy on this basis, but focused on the fusion with the text matching method, which is the reason why my final model did not work well. Also it might be better if a similar operation could be performed on span, as this would allow a validation set containing a sufficient number of labels to verify the effect, although the score is lower.\n**- Input of text**\nSince there are some unlabeled labels in the training set, it is necessary to limit the length of the input qa model text while training without using negative samples to minimize the negative effect of hidden samples. I also added some possible valid labels (these labels are from the bigger-govt-dataset-list, without verifying its validity) to expand the training sample.\n**- Post-processing**\nFor post-processing, I am directly using the size judgment of logit values. Also when predicting the results, the answers do not have overlapping parts, and finally use whether to include Remove the duplicate prediction labels.\n**- Semi-supervised**\nUse the labels with higher logit values in all the results predicted by the model, while the length of the words is between 2 and 10, and include some special words such as survey, as the labels used for string matching, to improve the recall of the private test set.\n\n\nFinally, the public validation set shows that a text length of 128 or 256 can have a score of about 0.5. In the private test set it can reach 0.42, but I did not choose this result because the score in the public test set is too low, which is also a reflection of problem 2.\n\n\n**Finally, enjoy the game!**\n" } }