{ "id": 10813, "title": "How to interpret training data?", "url": "/competitions/avazu-ctr-prediction/discussion/10813", "author_name": "", "post_date": "2014-11-01T06:21:40.490Z", "votes": 2, "comment_count": 4, "views": 1856, "content": "

I would request the competition organizers to provide more details about the nature of the data. Lets consider the first row of training data:

\n

id,click,hour,C1,banner_pos,site_id,site_domain,site_category,app_id,app_domain,app_category,device_id,device_ip,device_os,device_make,device_model,device_type,device_conn_type,device_geo_country,C17,C18,C19,C20,C21,C22,C23,C24

\n

10000222510487979663,0,14100100,1005,0,d41d8cd9,d41d8cd9,d41d8cd9,ee72efa5,85262c2b,7e5068fc,d41d8cd9,22f9c6ba,c31b3236,3d517f89,3e238c9b,1,0,fc9fdf08,11999,320,50,1248,2,39,-1,13

\n

A general comment is that you should provide a handful of rows that are NOT anonymized. They needn't be part of training and test data. They don't even have to be from real world dataset- just some possible scenarios. The purpose of these examples would be to allow us to understand the data better rather than trying to guess what they are.

\n

Back to the training data example. In this example, site_id = site_domain = site_category. Also device_model = device_make.

\n
    \n
  1. What does it mean for variables in different dimensions to have same values? Can they be compared? I presume equal values in the same dimensions are just a coincidence. 
    2. \n
  2. Is there any way to infer these anonymized values? E.g. aaaaa is closer to aaaaz than zzzzz?
    3. \n
  3. Does banner_pos = 1 for device_type X mean the same thing as banner_pos = 0 for device_type Y? As you can see, I can ask the same question for different variables (app_category, etc.) and there would be an endless combination of variables.
    4. \n
  4. I mentioned this before, but it will be incredibly useful if you can provide some sample values for each of these and other variables. 
    5. \n
", "messages": [ { "id": "57151", "postDate": "11/01/2014 06:21:40", "content": "

I would request the competition organizers to provide more details about the nature of the data. Lets consider the first row of training data:

\n

id,click,hour,C1,banner_pos,site_id,site_domain,site_category,app_id,app_domain,app_category,device_id,device_ip,device_os,device_make,device_model,device_type,device_conn_type,device_geo_country,C17,C18,C19,C20,C21,C22,C23,C24

\n

10000222510487979663,0,14100100,1005,0,d41d8cd9,d41d8cd9,d41d8cd9,ee72efa5,85262c2b,7e5068fc,d41d8cd9,22f9c6ba,c31b3236,3d517f89,3e238c9b,1,0,fc9fdf08,11999,320,50,1248,2,39,-1,13

\n

A general comment is that you should provide a handful of rows that are NOT anonymized. They needn't be part of training and test data. They don't even have to be from real world dataset- just some possible scenarios. The purpose of these examples would be to allow us to understand the data better rather than trying to guess what they are.

\n

Back to the training data example. In this example, site_id = site_domain = site_category. Also device_model = device_make.

\n
    \n
  1. What does it mean for variables in different dimensions to have same values? Can they be compared? I presume equal values in the same dimensions are just a coincidence. 
    2. \n
  2. Is there any way to infer these anonymized values? E.g. aaaaa is closer to aaaaz than zzzzz?
    3. \n
  3. Does banner_pos = 1 for device_type X mean the same thing as banner_pos = 0 for device_type Y? As you can see, I can ask the same question for different variables (app_category, etc.) and there would be an endless combination of variables.
    4. \n
  4. I mentioned this before, but it will be incredibly useful if you can provide some sample values for each of these and other variables. 
    5. \n
", "rawMarkdown": "", "votes": null }, { "id": "57153", "postDate": "11/01/2014 06:44:59", "content": "

They are strings and anonymized using md5

", "rawMarkdown": "", "votes": null }, { "id": "57154", "postDate": "11/01/2014 06:59:48", "content": "

Thanks for your quick response. 

\n

I'm not sure I understand what you mean by "no need to derive or infer physically meaningful features from them". If there is no need, then why is it part of dataset?

", "rawMarkdown": "", "votes": null }, { "id": "57155", "postDate": "11/01/2014 07:26:56", "content": "

sorry it's just a suggestion

\n

they are just md5ed to hide business information

", "rawMarkdown": "", "votes": null }, { "id": "57156", "postDate": "11/01/2014 07:52:52", "content": "

I understand its business information. Which is why I asked for some artificial examples that are "human readable". That way, we can focus on getting the best model rather than guessing what the variables are (which doesn't happen in a practical setting anyway)

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 57153, "author_name": "stevewang0", "author_url": "", "post_date": "11/01/2014 06:44:59", "content": "

They are strings and anonymized using md5

", "votes": null, "replies": [] }, { "id": 57154, "author_name": "vishalsurana", "author_url": "", "post_date": "11/01/2014 06:59:48", "content": "

Thanks for your quick response. 

\n

I'm not sure I understand what you mean by "no need to derive or infer physically meaningful features from them". If there is no need, then why is it part of dataset?

", "votes": null, "replies": [] }, { "id": 57155, "author_name": "stevewang0", "author_url": "", "post_date": "11/01/2014 07:26:56", "content": "

sorry it's just a suggestion

\n

they are just md5ed to hide business information

", "votes": null, "replies": [] }, { "id": 57156, "author_name": "vishalsurana", "author_url": "", "post_date": "11/01/2014 07:52:52", "content": "

I understand its business information. Which is why I asked for some artificial examples that are "human readable". That way, we can focus on getting the best model rather than guessing what the variables are (which doesn't happen in a practical setting anyway)

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "57151": "", "57153": "", "57154": "", "57155": "", "57156": "" }, "source": "meta" }