{ "id": 5897, "title": "Duplicate Entries", "url": "/competitions/facebook-recruiting-iii-keyword-extraction/discussion/5897", "author_name": "", "post_date": "2013-09-27T12:16:04.960Z", "votes": 6, "comment_count": 3, "views": 1906, "content": "

There are duplicate entries in both Train.csv and Test.csv. These entries have identical titles and bodies with different Id's. Here is the first example from the training set:

\n

Id: 2383

\n

Tags: 'function anchor'

\n

Title: 'Compatibility issues with <a> and calling a function(); across different web browsers'

\n

Body: "&lt;p&gt;I am new to javascript. ...

\n

 

\n

Id: 3551

\n

Tags: 'function anchor'

\n

Title: 'Compatibility issues with <a> and calling a function(); across different web browsers'

\n

Body: "&lt;p&gt;I am new to javascript. ...

\n

There are 1,582,285 duplicates in the training set and 219,796 duplicates in the test set. There also may be triplicates and higher. I haven't checked. Additionally, 47,888 duplicates in the training set have different tags. For example:

\n

    13032 & 35742 have identical titles and body   

\n

    13032 has tags 'java jsf jsf-2.0'
    35742 has tags 'java jsf jsf-2'

\n

I found 1 post on stack overflow with this title and body:

\n

    http://stackoverflow.com/questions/3523998/jsf-2-0-validation-controller

\n

It has the tags of the 35742 entry in the training set.

\n

What do the duplicate entries mean? Especially with different tags? Is this a previous version of the question for which the tags have been modified?

\n

Since there are so many duplicates in the training and test set, it would be good to account for the duplicates.

\n

Thanks!

\n

Edit: Added quantities that were still being computed when post was accidentally submitted

", "messages": [ { "id": "31654", "postDate": "09/27/2013 12:16:04", "content": "

There are duplicate entries in both Train.csv and Test.csv. These entries have identical titles and bodies with different Id's. Here is the first example from the training set:

\n

Id: 2383

\n

Tags: 'function anchor'

\n

Title: 'Compatibility issues with <a> and calling a function(); across different web browsers'

\n

Body: "&lt;p&gt;I am new to javascript. ...

\n

 

\n

Id: 3551

\n

Tags: 'function anchor'

\n

Title: 'Compatibility issues with <a> and calling a function(); across different web browsers'

\n

Body: "&lt;p&gt;I am new to javascript. ...

\n

There are 1,582,285 duplicates in the training set and 219,796 duplicates in the test set. There also may be triplicates and higher. I haven't checked. Additionally, 47,888 duplicates in the training set have different tags. For example:

\n

    13032 & 35742 have identical titles and body   

\n

    13032 has tags 'java jsf jsf-2.0'
    35742 has tags 'java jsf jsf-2'

\n

I found 1 post on stack overflow with this title and body:

\n

    http://stackoverflow.com/questions/3523998/jsf-2-0-validation-controller

\n

It has the tags of the 35742 entry in the training set.

\n

What do the duplicate entries mean? Especially with different tags? Is this a previous version of the question for which the tags have been modified?

\n

Since there are so many duplicates in the training and test set, it would be good to account for the duplicates.

\n

Thanks!

\n

Edit: Added quantities that were still being computed when post was accidentally submitted

", "rawMarkdown": "", "votes": null }, { "id": "31675", "postDate": "09/27/2013 21:51:42", "content": "

Good Catch!

\n

I checked to see what is the size of largest set of duplicated ids in the training set. Turns out there are 5 instances where 6 ids have the same exact title, body & tags.

\n

1. {1069368,2121889,4250846,3681253,3881129,287001}
2. {406875,5218087,1542583,1004129,3026293,2939632}
3. {2474623,4238422,3402352,5145828,901958,5771519}
4. {894828,979395,5736232,5245229,2962126,5346578}
5. {1895154,322241,4279923,1149482,5920726,1488873}

\n

For 1.:

\n

Title: Removing VirTool:Win32/VBInject.gen!DG

\n

Body:<p>i get this little bugger poping up every start up, my microsoft security essentials stops initial attempt and there no threat till next start up. I was wondering, if there any tools to check and remove items that actually initiate execution.</p><p>I assume this is may be a registry entry that can be removed, how would i effectively search for it.</p><p>Really do not want to reinstall everything.</p><p>Any help very much appreciated. Thank you so much.</p>

\n

Tags: windows-7 virus

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "31676", "postDate": "09/27/2013 22:09:11", "content": "

If the tags are different and title and body match (in one of the example as Matt pointed out), what should be the strategy? Merge the tags?

", "rawMarkdown": "", "votes": null }, { "id": "31677", "postDate": "09/27/2013 22:17:19", "content": "

I think you can either try to learn the strategy itself or follow a predifined strategy. The compettion is scored with the f-measure, so if you should merge the tags or not, depends on the number of tags and the size of the overlapping tags. For instance, if you choose to focus on precision, you would probably pick the overlapping tags and discard the others.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 31675, "author_name": "sashikanthdareddy", "author_url": "", "post_date": "09/27/2013 21:51:42", "content": "

Good Catch!

\n

I checked to see what is the size of largest set of duplicated ids in the training set. Turns out there are 5 instances where 6 ids have the same exact title, body & tags.

\n

1. {1069368,2121889,4250846,3681253,3881129,287001}
2. {406875,5218087,1542583,1004129,3026293,2939632}
3. {2474623,4238422,3402352,5145828,901958,5771519}
4. {894828,979395,5736232,5245229,2962126,5346578}
5. {1895154,322241,4279923,1149482,5920726,1488873}

\n

For 1.:

\n

Title: Removing VirTool:Win32/VBInject.gen!DG

\n

Body:<p>i get this little bugger poping up every start up, my microsoft security essentials stops initial attempt and there no threat till next start up. I was wondering, if there any tools to check and remove items that actually initiate execution.</p><p>I assume this is may be a registry entry that can be removed, how would i effectively search for it.</p><p>Really do not want to reinstall everything.</p><p>Any help very much appreciated. Thank you so much.</p>

\n

Tags: windows-7 virus

\n

 

", "votes": null, "replies": [] }, { "id": 31676, "author_name": "krishnadubba", "author_url": "", "post_date": "09/27/2013 22:09:11", "content": "

If the tags are different and title and body match (in one of the example as Matt pointed out), what should be the strategy? Merge the tags?

", "votes": null, "replies": [] }, { "id": 31677, "author_name": "", "author_url": "", "post_date": "09/27/2013 22:17:19", "content": "

I think you can either try to learn the strategy itself or follow a predifined strategy. The compettion is scored with the f-measure, so if you should merge the tags or not, depends on the number of tags and the size of the overlapping tags. For instance, if you choose to focus on precision, you would probably pick the overlapping tags and discard the others.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "31654": "", "31675": "", "31676": "", "31677": "" }, "source": "meta" }