{ "id": 10775, "title": "Competition Restarted", "url": "/competitions/avazu-ctr-prediction/discussion/10775", "author_name": "", "post_date": "2014-10-28T19:28:03.647Z", "votes": 2, "comment_count": 32, "views": 7207, "content": "

Thanks for your patience as we corrected the data issue. Revised files are now available from the Data page. I have also invalidated all previous submissions to the competition. Good luck with the competition, everyone!

", "messages": [ { "id": "56923", "postDate": "10/28/2014 19:28:03", "content": "

Thanks for your patience as we corrected the data issue. Revised files are now available from the Data page. I have also invalidated all previous submissions to the competition. Good luck with the competition, everyone!

", "rawMarkdown": "", "votes": null }, { "id": "56926", "postDate": "10/28/2014 19:42:07", "content": "

Are you sure there aren't a few bugs left in your data export scripts and elsewhere? Have you run appropriate tests and sanity checks? 

\n

Besides the temporal aberration that now seems to be corrected, the data looks nothing like what I'm used to in the industry. Mobile CTRs between 10 to 25%, really? In practice you'll never see anything above 1%, and that's the best case (generally involving fraud and dark patterns in ad placement), industry average is more like 0.1%.

\n

I'm having a hard time believing this data is legit.

", "rawMarkdown": "", "votes": null }, { "id": "56928", "postDate": "10/28/2014 19:57:40", "content": "

It could be due to subsampling. https://www.kaggle.com/c/criteo-display-ad-challenge/forums/t/10047/leader-board-0-44xx-is-the-best/52066#post52066

", "rawMarkdown": "", "votes": null }, { "id": "56929", "postDate": "10/28/2014 20:10:32", "content": "

Also, please update the Data page to include the additional details shared in the forums about the meanings of some columns.

", "rawMarkdown": "", "votes": null }, { "id": "56930", "postDate": "10/28/2014 20:15:49", "content": "

Seems there is other mess to be cleaned up. When I enter our team's name, It says "team name is already in use." Apparently the system confuses with our team's old record before all this. And my submission has been evaluated for half an hour and it is still pending.

", "rawMarkdown": "", "votes": null }, { "id": "56931", "postDate": "10/28/2014 20:33:20", "content": "

[quote=Synthient;56929]

\n

Also, please update the Data page to include the additional details shared in the forums about the meanings of some columns.

\n

[/quote]

\n

The "additional details" are simply column header names, which are now included in the data file itself.

", "rawMarkdown": "", "votes": null }, { "id": "56933", "postDate": "10/28/2014 20:40:17", "content": "

Joyce,

\n

are submissions working? mine seems to be pending for >20mins now (and the uncompressed file is not that big)...

\n

thanks,

\n

K

", "rawMarkdown": "", "votes": null }, { "id": "56934", "postDate": "10/28/2014 20:42:08", "content": "

[quote=joycenv;56931]

\n

[quote=Synthient;56929]

\n

Also, please update the Data page to include the additional details shared in the forums about the meanings of some columns.

\n

[/quote]

\n

The "additional details" are simply column header names, which are now included in the data file itself.

\n

[/quote]

\n\n

...which is different from the Data page which still says "C1-C24: 24 anonymized variables". It's now C1-4, C20-24: 9 anonymized variables.

", "rawMarkdown": "", "votes": null }, { "id": "56935", "postDate": "10/28/2014 20:44:45", "content": "

Submissions issue seems to be there in all competitions, scoring engine is slow for some reason.
Submitted a 15kb file in the Social networks competition and its still pending after 15 minutes.

", "rawMarkdown": "", "votes": null }, { "id": "56936", "postDate": "10/28/2014 20:46:24", "content": "

@Rush:  thanks for the confirmation. Mind you, the disappearing leaderboard probably has sth to do with that as well :-)

", "rawMarkdown": "", "votes": null }, { "id": "56938", "postDate": "10/28/2014 20:53:48", "content": "

Hi Konrad,

\n

Looks like we're having some other, unrelated problems. Please hang tight and don't resubmit for a while. For reference: https://www.kaggle.com/forums/t/10776/submission-outage

", "rawMarkdown": "", "votes": null }, { "id": "56939", "postDate": "10/28/2014 20:57:19", "content": "

Joyce,

\n

Thanks for the update, commencing hang-on-tight mode :-)

\n\n

K

", "rawMarkdown": "", "votes": null }, { "id": "56940", "postDate": "10/28/2014 21:08:53", "content": "

[quote=Synthient;56934]

\n

[quote=joycenv;56931]

\n

[quote=Synthient;56929]

\n

Also, please update the Data page to include the additional details shared in the forums about the meanings of some columns.

\n

[/quote]

\n

The "additional details" are simply column header names, which are now included in the data file itself.

\n

[/quote]

\n

...which is different from the Data page which still says "C1-C24: 24 anonymized variables". It's now C1-4, C20-24: 9 anonymized variables.

\n

[/quote]

\n

Hi Synthient, I've made this change on the Data page.

", "rawMarkdown": "", "votes": null }, { "id": "56941", "postDate": "10/28/2014 21:26:29", "content": "

Thanks.

", "rawMarkdown": "", "votes": null }, { "id": "56946", "postDate": "10/28/2014 22:14:09", "content": "

[quote=rcarson;56928]

\n

It could be due to subsampling. https://www.kaggle.com/c/criteo-display-ad-challenge/forums/t/10047/leader-board-0-44xx-is-the-best/52066#post52066

\n

[/quote]

\n

Subsampling of non-click events by a factor 10-100 could explain the unrealistic CTR, thanks for pointing this out. It would be nice it that was clarified by the organizers. It changes the nature of the prediction problem...

\n

Still, the data doesn't check. I am 90% sure the fields are mislabeled. The field "device_ip" has 8320 unique entries, with the most common having over 3 million impressions and 460k clicks. Meanwhile the field "app_domain" has 6 million unique entries... makes total sense, right? I'm imagining that one guy on his smartphone, visiting 3MM pages across 6MM apps and clicking on ads 460k times. Precious.

\n

So here are the correct labels:

\n

"app_domain" is actually "device_ip"

\n

"app_id" is actually "device_id"

\n

"device_ip" is probably "app_id"

\n

"device_ip" is probably "app_domain"

\n

Other fields are also probably mis-labeled, but it's harder to tell from the data histogram. And there's no telling what other bugs (hash collisions, non random sub-sampling, etc) are lurking in there.

", "rawMarkdown": "", "votes": null }, { "id": "56951", "postDate": "10/29/2014 03:29:52", "content": "

[quote=fchollet;56946]

\n

[quote=rcarson;56928]

\n

It could be due to subsampling. https://www.kaggle.com/c/criteo-display-ad-challenge/forums/t/10047/leader-board-0-44xx-is-the-best/52066#post52066

\n

[/quote]

\n

Subsampling of non-click events by a factor 10-100 could explain the unrealistic CTR, thanks for pointing this out. It would be nice it that was clarified by the organizers. It changes the nature of the prediction problem...

\n

Still, the data doesn't check. I am 90% sure the fields are mislabeled. The field "device_ip" has 8320 unique entries, with the most common having over 3 million impressions and 460k clicks. Meanwhile the field "app_domain" has 6 million unique entries... makes total sense, right? I'm imagining that one guy on his smartphone, visiting 3MM pages across 6MM apps and clicking on ads 460k times. Precious.

\n

So here are the correct labels:

\n

"app_domain" is actually "device_ip"

\n

"app_id" is actually "device_id"

\n

"device_ip" is probably "app_id"

\n

"device_ip" is probably "app_domain"

\n

Other fields are also probably mis-labeled, but it's harder to tell from the data histogram. And there's no telling what other bugs (hash collisions, non random sub-sampling, etc) are lurking in there.

\n

[/quote]

\n

Hi fchollet

\n

  I have loaded files into mysql, checked the train data and it seems OK

\n

count(distinct site_id ): 4715
count(distinct site_domain ): 7733
count(distinct site_category ): 26
count(distinct app_id ): 7941
count(distinct app_domain ): 433
count(distinct app_category ): 38
count(distinct device_id ): 2787564
count(distinct device_ip ): 6081079
count(distinct device_os ): 15
count(distinct device_make ): 283
count(distinct device_model ): 8320
count(distinct device_type ): 4
count(distinct device_conn_type ): 4
count(distinct device_geo_country ): 232

", "rawMarkdown": "", "votes": null }, { "id": "56952", "postDate": "10/29/2014 04:05:39", "content": "

Hi Steve,

\n

It seems we do not have the same data... I have no idea how that could be possible, since I have no visibility into the code that generated the files, but apparently anything is possible. 

\n

I have the following entry counts:

\n

For train:

\n

hour: 240 - ex: 14100822
C1: 7 - ex: 1001
C2: 7 - ex: 0
C3: 4715 - ex: 8dd05c70
C4: 7733 - ex: 1928ea2e
banner_pos: 26 - ex: e878bf21
site_id: 7941 - ex: 0b54af4e
site_domain: 433 - ex: dc165517
site_category: 38 - ex: e878bf21
app_id: 2787564 - ex: 56ac8621
app_domain: 6081079 - ex: 1a62fb8f
app_category: 15 - ex: 9740a313
device_id: 283 - ex: 8f5da45e
device_ip: 8320 - ex: 3f8375f9
device_os: 4 - ex: 0
device_make: 4 - ex: 0
device_model: 232 - ex: 7efdfc94
device_type: 2592 - ex: 19115
device_conn_type: 8 - ex: 320
device_geo_country: 9 - ex: 480
C20: 457 - ex: 2048
C21: 4 - ex: 0
C22: 61 - ex: 1449
C23: 171 - ex: -1
C24: 53 - ex: 91
--
total lines: 47686351
total clicks: 8228679

\n

For test:

\n

hour: 24 - ex: 14101120
C1: 7 - ex: 1001
C2: 7 - ex: 0
C3: 2919 - ex: 8dd05c70
C4: 3681 - ex: 80568324
banner_pos: 21 - ex: bc1596e9
site_id: 2904 - ex: 3ad8fcfa
site_domain: 139 - ex: f90bd603
site_category: 22 - ex: 050fd349
app_id: 435864 - ex: fa5e1ef1
app_domain: 978837 - ex: ac4acb06
app_category: 10 - ex: 5ed26317
device_id: 246 - ex: af7e730c
device_ip: 5530 - ex: 257e8866
device_os: 5 - ex: 0
device_make: 4 - ex: 0
device_model: 227 - ex: 2c38b9e4
device_type: 1480 - ex: 10923
device_conn_type: 8 - ex: 320
device_geo_country: 9 - ex: 480
C20: 294 - ex: 2049
C21: 4 - ex: 0
C22: 39 - ex: 427
C23: 163 - ex: -1
C24: 31 - ex: 91
--
total lines: 4769401

\n

The headers in the files I downloaded have the following order (same for train and test):

\n

'hour', 'C1', 'C2', 'C3', 'C4', 'banner_pos', 'site_id', 'site_domain', 'site_category', 'app_id', 'app_domain', 'app_category', 'device_id', 'device_ip', 'device_os', 'device_make', 'device_model', 'device_type', 'device_conn_type', 'device_geo_country', 'C20', 'C21', 'C22', 'C23', 'C24'

", "rawMarkdown": "", "votes": null }, { "id": "56953", "postDate": "10/29/2014 04:15:43", "content": "

Hi fchollet,

\n

I've just learned that I've had a miscommunication with the sponsor so the headers on the data files are incorrect. I am in the process of correcting the error, but these are large files. Also, I want to confirm there are no other changes that need to be addressed since I do not want to have to again re-release the data set. Thanks for your careful eye, and for your patience as we get this sorted out.

", "rawMarkdown": "", "votes": null }, { "id": "56955", "postDate": "10/29/2014 04:38:16", "content": "

I have uploaded files for the correct training set and test set header rows. Please use this to identify the columns appropriately.

", "rawMarkdown": "", "votes": null }, { "id": "56956", "postDate": "10/29/2014 04:50:47", "content": "

[quote=fchollet;56926]

\n

Are you sure there aren't a few bugs left in your data export scripts and elsewhere? Have you run appropriate tests and sanity checks? 

\n

Besides the temporal aberration that now seems to be corrected, the data looks nothing like what I'm used to in the industry. Mobile CTRs between 10 to 25%, really? In practice you'll never see anything above 1%, and that's the best case (generally involving fraud and dark patterns in ad placement), industry average is more like 0.1%.

\n

I'm having a hard time believing this data is legit.

\n

[/quote]

\n

yes we did subsampling and the ctr for train data is about 0.1778

", "rawMarkdown": "", "votes": null }, { "id": "56957", "postDate": "10/29/2014 05:10:38", "content": "

Hi all,

\n

still downloading the data.. but can no longer wait for:

\n

Are all id of ad in the training set included in the test set ?

", "rawMarkdown": "", "votes": null }, { "id": "56958", "postDate": "10/29/2014 05:13:55", "content": "

[quote=joycenv;56955]

\n

I have uploaded files for the correct training set and test set header rows. Please use this to identify the columns appropriately.

\n

[/quote]

\n

Ok, mislabeling issue solved. We can officially start the contest. Thank you!

", "rawMarkdown": "", "votes": null }, { "id": "56975", "postDate": "10/29/2014 11:52:34", "content": "

Joyce,

\n

Two more questions:

\n

1. What is the difference between test_rev1 and test_rev2?

\n

2. the proper train set = {train_rev1 + train_header}, test = {test_rev1 + test_header}?

\n\n

thanks,

\n

K

", "rawMarkdown": "", "votes": null }, { "id": "56978", "postDate": "10/29/2014 13:14:42", "content": "

1. rev_2 includes the correct header. The header row is the only difference between _rev1 and _rev2.

\n

2. Yes. _rev1 files need the updated header row.

", "rawMarkdown": "", "votes": null }, { "id": "56979", "postDate": "10/29/2014 13:15:52", "content": "

FWIW: if anybody needs to replace the headers with the new ones, here's a fairly quick (if not particularly elegant :-) way to do it:

\n

wc -l ./data/test_rev1.csv

\n

tail -4769401 ./data/test_rev1.csv > ./data/temp.csv

\n

cat ./data/test_header.csv ./data/temp.csv > ./data/test_v2.csv

\n\n

EDIT: this approach takes forever for the train set, so try 

\n

cat ./data/train_rev1.csv | tail -n+2 > ./data/temp.csv

\n

instead

", "rawMarkdown": "", "votes": null }, { "id": "56980", "postDate": "10/29/2014 13:16:48", "content": "

Thank you, Joyce.

\n\n

K

", "rawMarkdown": "", "votes": null }, { "id": "58013", "postDate": "11/13/2014 17:33:51", "content": "

hello, can i know why i can't see "make submission" in my sidebar?

\n

really weird!!

", "rawMarkdown": "", "votes": null }, { "id": "58015", "postDate": "11/13/2014 17:53:16", "content": "

[quote=optimizer;58014]

\n

[quote=noooooo;58013]

\n

hello, can i know why i can't see "make submission" in my sidebar?

\n

really weird!!

\n

[/quote]

\n

it's on hold.

\n

http://www.kaggle.com/c/avazu-ctr-prediction/forums/t/10887/competition-put-on-hold/57606#post57606

\n

[/quote]

\n

thanks...i search for the button for 15mins already....

", "rawMarkdown": "", "votes": null }, { "id": "58031", "postDate": "11/14/2014 00:12:31", "content": "

Oh, someone just bumped this thread to the top, the competition is still on hold. I was so confused.

", "rawMarkdown": "", "votes": null }, { "id": "58033", "postDate": "11/14/2014 00:23:46", "content": "

Hey mods, can we delete or lock this thread? It's confusing new people, and I'm tired of seeing it pop up in my inbox and getting all excited about nothing.

", "rawMarkdown": "", "votes": null }, { "id": "58034", "postDate": "11/14/2014 00:24:08", "content": "

[quote=James King;58031]

\n

Uh, am I supposed to see data on the data page? I still see "Data downloads are temporarily disabled. "

\n

[/quote]

\n\n

Competition is still on hold (read related post on the forum).  

\n

If this was not Kaggle, I would think this post has been trolled. This thread is an old restart of the competition. 

", "rawMarkdown": "", "votes": null }, { "id": "58035", "postDate": "11/14/2014 00:26:26", "content": "

This post should NOT be pinned on top of the forum.

\n

It's really misleading.

", "rawMarkdown": "", "votes": null }, { "id": "58037", "postDate": "11/14/2014 00:29:06", "content": "

[quote=pi_informatics;58033]

\n

Hey mods, can we delete or lock this thread? It's confusing new people, and I'm tired of seeing it pop up in my inbox and getting all excited about nothing.

\n

[/quote]

\n

LOL . . . I was just about to cancel date night.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 56926, "author_name": "fchollet", "author_url": "", "post_date": "10/28/2014 19:42:07", "content": "

Are you sure there aren't a few bugs left in your data export scripts and elsewhere? Have you run appropriate tests and sanity checks? 

\n

Besides the temporal aberration that now seems to be corrected, the data looks nothing like what I'm used to in the industry. Mobile CTRs between 10 to 25%, really? In practice you'll never see anything above 1%, and that's the best case (generally involving fraud and dark patterns in ad placement), industry average is more like 0.1%.

\n

I'm having a hard time believing this data is legit.

", "votes": null, "replies": [] }, { "id": 56928, "author_name": "jiweiliu", "author_url": "", "post_date": "10/28/2014 19:57:40", "content": "

It could be due to subsampling. https://www.kaggle.com/c/criteo-display-ad-challenge/forums/t/10047/leader-board-0-44xx-is-the-best/52066#post52066

", "votes": null, "replies": [] }, { "id": 56929, "author_name": "brundage", "author_url": "", "post_date": "10/28/2014 20:10:32", "content": "

Also, please update the Data page to include the additional details shared in the forums about the meanings of some columns.

", "votes": null, "replies": [] }, { "id": 56930, "author_name": "jiweiliu", "author_url": "", "post_date": "10/28/2014 20:15:49", "content": "

Seems there is other mess to be cleaned up. When I enter our team's name, It says "team name is already in use." Apparently the system confuses with our team's old record before all this. And my submission has been evaluated for half an hour and it is still pending.

", "votes": null, "replies": [] }, { "id": 56931, "author_name": "joycenv", "author_url": "", "post_date": "10/28/2014 20:33:20", "content": "

[quote=Synthient;56929]

\n

Also, please update the Data page to include the additional details shared in the forums about the meanings of some columns.

\n

[/quote]

\n

The "additional details" are simply column header names, which are now included in the data file itself.

", "votes": null, "replies": [] }, { "id": 56933, "author_name": "konradb", "author_url": "", "post_date": "10/28/2014 20:40:17", "content": "

Joyce,

\n

are submissions working? mine seems to be pending for >20mins now (and the uncompressed file is not that big)...

\n

thanks,

\n

K

", "votes": null, "replies": [] }, { "id": 56934, "author_name": "brundage", "author_url": "", "post_date": "10/28/2014 20:42:08", "content": "

[quote=joycenv;56931]

\n

[quote=Synthient;56929]

\n

Also, please update the Data page to include the additional details shared in the forums about the meanings of some columns.

\n

[/quote]

\n

The "additional details" are simply column header names, which are now included in the data file itself.

\n

[/quote]

\n\n

...which is different from the Data page which still says "C1-C24: 24 anonymized variables". It's now C1-4, C20-24: 9 anonymized variables.

", "votes": null, "replies": [] }, { "id": 56935, "author_name": "rishis", "author_url": "", "post_date": "10/28/2014 20:44:45", "content": "

Submissions issue seems to be there in all competitions, scoring engine is slow for some reason.
Submitted a 15kb file in the Social networks competition and its still pending after 15 minutes.

", "votes": null, "replies": [] }, { "id": 56936, "author_name": "konradb", "author_url": "", "post_date": "10/28/2014 20:46:24", "content": "

@Rush:  thanks for the confirmation. Mind you, the disappearing leaderboard probably has sth to do with that as well :-)

", "votes": null, "replies": [] }, { "id": 56938, "author_name": "joycenv", "author_url": "", "post_date": "10/28/2014 20:53:48", "content": "

Hi Konrad,

\n

Looks like we're having some other, unrelated problems. Please hang tight and don't resubmit for a while. For reference: https://www.kaggle.com/forums/t/10776/submission-outage

", "votes": null, "replies": [] }, { "id": 56939, "author_name": "konradb", "author_url": "", "post_date": "10/28/2014 20:57:19", "content": "

Joyce,

\n

Thanks for the update, commencing hang-on-tight mode :-)

\n\n

K

", "votes": null, "replies": [] }, { "id": 56940, "author_name": "joycenv", "author_url": "", "post_date": "10/28/2014 21:08:53", "content": "

[quote=Synthient;56934]

\n

[quote=joycenv;56931]

\n

[quote=Synthient;56929]

\n

Also, please update the Data page to include the additional details shared in the forums about the meanings of some columns.

\n

[/quote]

\n

The "additional details" are simply column header names, which are now included in the data file itself.

\n

[/quote]

\n

...which is different from the Data page which still says "C1-C24: 24 anonymized variables". It's now C1-4, C20-24: 9 anonymized variables.

\n

[/quote]

\n

Hi Synthient, I've made this change on the Data page.

", "votes": null, "replies": [] }, { "id": 56941, "author_name": "brundage", "author_url": "", "post_date": "10/28/2014 21:26:29", "content": "

Thanks.

", "votes": null, "replies": [] }, { "id": 56946, "author_name": "fchollet", "author_url": "", "post_date": "10/28/2014 22:14:09", "content": "

[quote=rcarson;56928]

\n

It could be due to subsampling. https://www.kaggle.com/c/criteo-display-ad-challenge/forums/t/10047/leader-board-0-44xx-is-the-best/52066#post52066

\n

[/quote]

\n

Subsampling of non-click events by a factor 10-100 could explain the unrealistic CTR, thanks for pointing this out. It would be nice it that was clarified by the organizers. It changes the nature of the prediction problem...

\n

Still, the data doesn't check. I am 90% sure the fields are mislabeled. The field "device_ip" has 8320 unique entries, with the most common having over 3 million impressions and 460k clicks. Meanwhile the field "app_domain" has 6 million unique entries... makes total sense, right? I'm imagining that one guy on his smartphone, visiting 3MM pages across 6MM apps and clicking on ads 460k times. Precious.

\n

So here are the correct labels:

\n

"app_domain" is actually "device_ip"

\n

"app_id" is actually "device_id"

\n

"device_ip" is probably "app_id"

\n

"device_ip" is probably "app_domain"

\n

Other fields are also probably mis-labeled, but it's harder to tell from the data histogram. And there's no telling what other bugs (hash collisions, non random sub-sampling, etc) are lurking in there.

", "votes": null, "replies": [] }, { "id": 56951, "author_name": "stevewang0", "author_url": "", "post_date": "10/29/2014 03:29:52", "content": "

[quote=fchollet;56946]

\n

[quote=rcarson;56928]

\n

It could be due to subsampling. https://www.kaggle.com/c/criteo-display-ad-challenge/forums/t/10047/leader-board-0-44xx-is-the-best/52066#post52066

\n

[/quote]

\n

Subsampling of non-click events by a factor 10-100 could explain the unrealistic CTR, thanks for pointing this out. It would be nice it that was clarified by the organizers. It changes the nature of the prediction problem...

\n

Still, the data doesn't check. I am 90% sure the fields are mislabeled. The field "device_ip" has 8320 unique entries, with the most common having over 3 million impressions and 460k clicks. Meanwhile the field "app_domain" has 6 million unique entries... makes total sense, right? I'm imagining that one guy on his smartphone, visiting 3MM pages across 6MM apps and clicking on ads 460k times. Precious.

\n

So here are the correct labels:

\n

"app_domain" is actually "device_ip"

\n

"app_id" is actually "device_id"

\n

"device_ip" is probably "app_id"

\n

"device_ip" is probably "app_domain"

\n

Other fields are also probably mis-labeled, but it's harder to tell from the data histogram. And there's no telling what other bugs (hash collisions, non random sub-sampling, etc) are lurking in there.

\n

[/quote]

\n

Hi fchollet

\n

  I have loaded files into mysql, checked the train data and it seems OK

\n

count(distinct site_id ): 4715
count(distinct site_domain ): 7733
count(distinct site_category ): 26
count(distinct app_id ): 7941
count(distinct app_domain ): 433
count(distinct app_category ): 38
count(distinct device_id ): 2787564
count(distinct device_ip ): 6081079
count(distinct device_os ): 15
count(distinct device_make ): 283
count(distinct device_model ): 8320
count(distinct device_type ): 4
count(distinct device_conn_type ): 4
count(distinct device_geo_country ): 232

", "votes": null, "replies": [] }, { "id": 56952, "author_name": "fchollet", "author_url": "", "post_date": "10/29/2014 04:05:39", "content": "

Hi Steve,

\n

It seems we do not have the same data... I have no idea how that could be possible, since I have no visibility into the code that generated the files, but apparently anything is possible. 

\n

I have the following entry counts:

\n

For train:

\n

hour: 240 - ex: 14100822
C1: 7 - ex: 1001
C2: 7 - ex: 0
C3: 4715 - ex: 8dd05c70
C4: 7733 - ex: 1928ea2e
banner_pos: 26 - ex: e878bf21
site_id: 7941 - ex: 0b54af4e
site_domain: 433 - ex: dc165517
site_category: 38 - ex: e878bf21
app_id: 2787564 - ex: 56ac8621
app_domain: 6081079 - ex: 1a62fb8f
app_category: 15 - ex: 9740a313
device_id: 283 - ex: 8f5da45e
device_ip: 8320 - ex: 3f8375f9
device_os: 4 - ex: 0
device_make: 4 - ex: 0
device_model: 232 - ex: 7efdfc94
device_type: 2592 - ex: 19115
device_conn_type: 8 - ex: 320
device_geo_country: 9 - ex: 480
C20: 457 - ex: 2048
C21: 4 - ex: 0
C22: 61 - ex: 1449
C23: 171 - ex: -1
C24: 53 - ex: 91
--
total lines: 47686351
total clicks: 8228679

\n

For test:

\n

hour: 24 - ex: 14101120
C1: 7 - ex: 1001
C2: 7 - ex: 0
C3: 2919 - ex: 8dd05c70
C4: 3681 - ex: 80568324
banner_pos: 21 - ex: bc1596e9
site_id: 2904 - ex: 3ad8fcfa
site_domain: 139 - ex: f90bd603
site_category: 22 - ex: 050fd349
app_id: 435864 - ex: fa5e1ef1
app_domain: 978837 - ex: ac4acb06
app_category: 10 - ex: 5ed26317
device_id: 246 - ex: af7e730c
device_ip: 5530 - ex: 257e8866
device_os: 5 - ex: 0
device_make: 4 - ex: 0
device_model: 227 - ex: 2c38b9e4
device_type: 1480 - ex: 10923
device_conn_type: 8 - ex: 320
device_geo_country: 9 - ex: 480
C20: 294 - ex: 2049
C21: 4 - ex: 0
C22: 39 - ex: 427
C23: 163 - ex: -1
C24: 31 - ex: 91
--
total lines: 4769401

\n

The headers in the files I downloaded have the following order (same for train and test):

\n

'hour', 'C1', 'C2', 'C3', 'C4', 'banner_pos', 'site_id', 'site_domain', 'site_category', 'app_id', 'app_domain', 'app_category', 'device_id', 'device_ip', 'device_os', 'device_make', 'device_model', 'device_type', 'device_conn_type', 'device_geo_country', 'C20', 'C21', 'C22', 'C23', 'C24'

", "votes": null, "replies": [] }, { "id": 56953, "author_name": "joycenv", "author_url": "", "post_date": "10/29/2014 04:15:43", "content": "

Hi fchollet,

\n

I've just learned that I've had a miscommunication with the sponsor so the headers on the data files are incorrect. I am in the process of correcting the error, but these are large files. Also, I want to confirm there are no other changes that need to be addressed since I do not want to have to again re-release the data set. Thanks for your careful eye, and for your patience as we get this sorted out.

", "votes": null, "replies": [] }, { "id": 56955, "author_name": "joycenv", "author_url": "", "post_date": "10/29/2014 04:38:16", "content": "

I have uploaded files for the correct training set and test set header rows. Please use this to identify the columns appropriately.

", "votes": null, "replies": [] }, { "id": 56956, "author_name": "stevewang0", "author_url": "", "post_date": "10/29/2014 04:50:47", "content": "

[quote=fchollet;56926]

\n

Are you sure there aren't a few bugs left in your data export scripts and elsewhere? Have you run appropriate tests and sanity checks? 

\n

Besides the temporal aberration that now seems to be corrected, the data looks nothing like what I'm used to in the industry. Mobile CTRs between 10 to 25%, really? In practice you'll never see anything above 1%, and that's the best case (generally involving fraud and dark patterns in ad placement), industry average is more like 0.1%.

\n

I'm having a hard time believing this data is legit.

\n

[/quote]

\n

yes we did subsampling and the ctr for train data is about 0.1778

", "votes": null, "replies": [] }, { "id": 56957, "author_name": "herimanitra", "author_url": "", "post_date": "10/29/2014 05:10:38", "content": "

Hi all,

\n

still downloading the data.. but can no longer wait for:

\n

Are all id of ad in the training set included in the test set ?

", "votes": null, "replies": [] }, { "id": 56958, "author_name": "jiweiliu", "author_url": "", "post_date": "10/29/2014 05:13:55", "content": "

[quote=joycenv;56955]

\n

I have uploaded files for the correct training set and test set header rows. Please use this to identify the columns appropriately.

\n

[/quote]

\n

Ok, mislabeling issue solved. We can officially start the contest. Thank you!

", "votes": null, "replies": [] }, { "id": 56975, "author_name": "konradb", "author_url": "", "post_date": "10/29/2014 11:52:34", "content": "

Joyce,

\n

Two more questions:

\n

1. What is the difference between test_rev1 and test_rev2?

\n

2. the proper train set = {train_rev1 + train_header}, test = {test_rev1 + test_header}?

\n\n

thanks,

\n

K

", "votes": null, "replies": [] }, { "id": 56978, "author_name": "joycenv", "author_url": "", "post_date": "10/29/2014 13:14:42", "content": "

1. rev_2 includes the correct header. The header row is the only difference between _rev1 and _rev2.

\n

2. Yes. _rev1 files need the updated header row.

", "votes": null, "replies": [] }, { "id": 56979, "author_name": "konradb", "author_url": "", "post_date": "10/29/2014 13:15:52", "content": "

FWIW: if anybody needs to replace the headers with the new ones, here's a fairly quick (if not particularly elegant :-) way to do it:

\n

wc -l ./data/test_rev1.csv

\n

tail -4769401 ./data/test_rev1.csv > ./data/temp.csv

\n

cat ./data/test_header.csv ./data/temp.csv > ./data/test_v2.csv

\n\n

EDIT: this approach takes forever for the train set, so try 

\n

cat ./data/train_rev1.csv | tail -n+2 > ./data/temp.csv

\n

instead

", "votes": null, "replies": [] }, { "id": 56980, "author_name": "konradb", "author_url": "", "post_date": "10/29/2014 13:16:48", "content": "

Thank you, Joyce.

\n\n

K

", "votes": null, "replies": [] }, { "id": 58013, "author_name": "betterplace", "author_url": "", "post_date": "11/13/2014 17:33:51", "content": "

hello, can i know why i can't see "make submission" in my sidebar?

\n

really weird!!

", "votes": null, "replies": [] }, { "id": 58015, "author_name": "betterplace", "author_url": "", "post_date": "11/13/2014 17:53:16", "content": "

[quote=optimizer;58014]

\n

[quote=noooooo;58013]

\n

hello, can i know why i can't see "make submission" in my sidebar?

\n

really weird!!

\n

[/quote]

\n

it's on hold.

\n

http://www.kaggle.com/c/avazu-ctr-prediction/forums/t/10887/competition-put-on-hold/57606#post57606

\n

[/quote]

\n

thanks...i search for the button for 15mins already....

", "votes": null, "replies": [] }, { "id": 58031, "author_name": "jfkingiii", "author_url": "", "post_date": "11/14/2014 00:12:31", "content": "

Oh, someone just bumped this thread to the top, the competition is still on hold. I was so confused.

", "votes": null, "replies": [] }, { "id": 58033, "author_name": "piinformatics", "author_url": "", "post_date": "11/14/2014 00:23:46", "content": "

Hey mods, can we delete or lock this thread? It's confusing new people, and I'm tired of seeing it pop up in my inbox and getting all excited about nothing.

", "votes": null, "replies": [] }, { "id": 58034, "author_name": "fernandoconstantino", "author_url": "", "post_date": "11/14/2014 00:24:08", "content": "

[quote=James King;58031]

\n

Uh, am I supposed to see data on the data page? I still see "Data downloads are temporarily disabled. "

\n

[/quote]

\n\n

Competition is still on hold (read related post on the forum).  

\n

If this was not Kaggle, I would think this post has been trolled. This thread is an old restart of the competition. 

", "votes": null, "replies": [] }, { "id": 58035, "author_name": "fernandoconstantino", "author_url": "", "post_date": "11/14/2014 00:26:26", "content": "

This post should NOT be pinned on top of the forum.

\n

It's really misleading.

", "votes": null, "replies": [] }, { "id": 58037, "author_name": "inversion", "author_url": "", "post_date": "11/14/2014 00:29:06", "content": "

[quote=pi_informatics;58033]

\n

Hey mods, can we delete or lock this thread? It's confusing new people, and I'm tired of seeing it pop up in my inbox and getting all excited about nothing.

\n

[/quote]

\n

LOL . . . I was just about to cancel date night.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "56923": "", "56926": "", "56928": "", "56929": "", "56930": "", "56931": "", "56933": "", "56934": "", "56935": "", "56936": "", "56938": "", "56939": "", "56940": "", "56941": "", "56946": "", "56951": "", "56952": "", "56953": "", "56955": "", "56956": "", "56957": "", "56958": "", "56975": "", "56978": "", "56979": "", "56980": "", "58013": "", "58015": "", "58031": "", "58033": "", "58034": "", "58035": "", "58037": "" }, "source": "meta" }