{ "id": 10853, "title": "DEVICE_GEO_COUNTRY", "url": "/competitions/avazu-ctr-prediction/discussion/10853", "author_name": "", "post_date": "2014-11-06T15:13:00.807Z", "votes": null, "comment_count": 26, "views": 7230, "content": "

Hi everybody!

\n

is there any way to know that is representing the values of device_geo_country variable? I mean, there are 232 different values for all length 8... then, every different value is a country? or maybe the first digits refers to the country and the following to the región?, if so, how many and which digits represent the country??

\n

thanks!

", "messages": [ { "id": "57405", "postDate": "11/06/2014 15:13:00", "content": "

Hi everybody!

\n

is there any way to know that is representing the values of device_geo_country variable? I mean, there are 232 different values for all length 8... then, every different value is a country? or maybe the first digits refers to the country and the following to the región?, if so, how many and which digits represent the country??

\n

thanks!

", "rawMarkdown": "", "votes": null }, { "id": "57411", "postDate": "11/06/2014 16:02:30", "content": "

It could be interesting to correlate each of the levels of country with peak activity time of day, and at least infer longitude for each country...

", "rawMarkdown": "", "votes": null }, { "id": "57424", "postDate": "11/06/2014 17:23:44", "content": "

Here is a plot of normalized hourly click ratios (ie click ratio for each hour / average ratio) for each country:

\n

\"click

\n

Not exactly what I would have expected to see. There seems to be some kind of general trend and not huge variations between the countries etc.

\n

Edit: Hmm, maybe this could be just a consequence of the reported subsampling of the data

", "rawMarkdown": "", "votes": null }, { "id": "57428", "postDate": "11/06/2014 18:45:41", "content": "

device_geo_country  = 'dc634e20'

\n

\n

I arbitrarily select several device-geo-country and find most of them have statistics like the picture above, but the location of peak is different.

", "rawMarkdown": "", "votes": null }, { "id": "57453", "postDate": "11/07/2014 02:45:32", "content": "

It's at least plausible that they are countries - ISO 3166 tabulates each recognised nation as a two character alpha, a three character alpha and a three character numeric - total 8 characters - and has a listing for around 250 nation-like entities.

\n

At the same time, it is also possible that a few countries - e.g. the USA - could be encoded at the state level. Not uncommon for online forms to request the user's country, and then request the user's state if and only they identify as from the US and/or a handful of other large countries.

", "rawMarkdown": "", "votes": null }, { "id": "57460", "postDate": "11/07/2014 10:09:39", "content": "

Hi OzRob,

\n

It really is a good idea but can not find the correspondence between the two codes!

\n\n

For example, dc634e20?

", "rawMarkdown": "", "votes": null }, { "id": "57486", "postDate": "11/07/2014 20:16:26", "content": "

Here's the top 10 countries in training data:

\n

1 13b5bfe9 India //md5('in') = "13b5bfe96f3e2fe411c9f66f4a582adf"
2 b80bb774 Indonesia
3 0b3b97fa United States
4 959848ca South Africa
5 1cd3c693 Pakistan
6 dc634e20 Brazil
7 c12e01f2 Saudi Arabia
8 795237fd Iraq
9 dcf0d7d2 Korea, Republic Of
10 dd07de85 Peru

\n\n

Now, let's go crack the other columns :)

", "rawMarkdown": "", "votes": null }, { "id": "57492", "postDate": "11/07/2014 22:34:02", "content": "

B Yang: it is amazing that you are decoding the top country - very curious about how you managed to do so if you don't mind sharing the technique :)

", "rawMarkdown": "", "votes": null }, { "id": "57493", "postDate": "11/07/2014 22:51:23", "content": "

[quote=DerekZH;57492]

\n

B Yang: it is amazing that you are decoding the top country - very curious about how you managed to do so if you don't mind sharing the technique :)

\n

[/quote]

\n\n

He's using the first 8 chars from the md5 hash (as stated in the comment from line 1).

\n

MD5 ("in") = 13b5bfe96f3e2fe411c9f66f4a582adf
MD5 ("id") = b80bb7740288fda1f201890375a60c8f
MD5 ("us") = 0b3b97fa66886c5688ee4ae80ec0c3c2
MD5 ("za") = 959848ca10cc8a60da818ac11523dc63
MD5 ("pk") = 1cd3c693132f4c31b5b5e5f4c5eed6bd
MD5 ("br") = dc634e2072827fe0b5be9a2063390544
MD5 ("sa") = c12e01f2a13ff5587e1e9e4aedb8242d
MD5 ("iq") = 795237fd9d107e63cd19b0db0f2fba2f
MD5 ("kr") = dcf0d7d2cd120bf42580d43f29785dd3
MD5 ("pe") = dd07de8561395a7c37f8fcc1752d45e1

", "rawMarkdown": "", "votes": null }, { "id": "57494", "postDate": "11/07/2014 23:38:41", "content": "

Full list

", "rawMarkdown": "", "votes": null }, { "id": "57499", "postDate": "11/08/2014 03:50:46", "content": "

[quote=Nissan Pow;57493]

\n

He's using the first 8 chars from the md5 hash (as stated in the comment from line 1).

\n

[/quote]

\n

Yes, you hash the likely suspects and see if the leading bytes are in the data. For example, the top 5 device_os:

\n\n

And the top 10 device_make:

\n\n

I wonder what the #2 device_make is, I suspect it's something like '-unknown-'. You can go to this web site:

\n

    http://www.md5hashgenerator.com/index.php

\n

enter your guess and click on "Generate MD5 Hash".

", "rawMarkdown": "", "votes": null }, { "id": "57552", "postDate": "11/09/2014 12:37:24", "content": "

How can one IP have two DEVICE_GEO_COUNTRY values associated with it?

\n

For eg. c1b8122b DEVICE_IP has b80bb774 and 0b3b97fa.

\n

Cheers!

", "rawMarkdown": "", "votes": null }, { "id": "57591", "postDate": "11/10/2014 11:46:18", "content": "

I would suspect that the owner of that phone has been in more than one country during the time period covered by the dataset. Whenever I travel I spend a fair amount of time messing around on my phone while waiting in airports, both while waiting for departure and waiting to get through arrivals - i.e. in 2 countries! I usually have data turned off when abroad due to roaming costs, but guess not everyone does so.

", "rawMarkdown": "", "votes": null }, { "id": "57607", "postDate": "11/10/2014 14:54:27", "content": "

I see, but the IP of your phone is differnet in Paris and London, the geolocation is different. The question is how can the same IP have two different geolocations.

", "rawMarkdown": "", "votes": null }, { "id": "57613", "postDate": "11/10/2014 16:00:21", "content": "

ivo@

\n

ivo wrote:

\n

The question is how can the same IP have two different geolocations.

\n

_________________________________________________________ 

\n

I think - this is not possible

", "rawMarkdown": "", "votes": null }, { "id": "57622", "postDate": "11/10/2014 17:22:08", "content": "

probably means they are going state-wise in some countries (india). and some small states in india share ip geolocations.

", "rawMarkdown": "", "votes": null }, { "id": "57623", "postDate": "11/10/2014 17:34:26", "content": "

on second thoughts there are 233 country codes and 232 values in the file. i'm guessing they are excluding only china.

\n

https://en.wikipedia.org/wiki/List_of_countries_by_IPv4_address_allocation

\n

there are 19 countries with 0 IPs. these probably use IP from nearby countries.

", "rawMarkdown": "", "votes": null }, { "id": "57650", "postDate": "11/10/2014 19:26:44", "content": "

Given the initial data debacle, I was fully expecting something like this. Seriously guys, you did not salt your hashes? Did you expect nobody would try using common hashes on simple fields to deanonymize the data? Like it has happened every single time unsalted hashes have been used in a competition? Guys.

\n

In case you're wondering, the reason this is problematic from a privacy perspective, is that by brute-forcing the device_ip column (by hashing a few billion IPs that have been active recently and matching the hashes with the Avazu data) and the site_domain column (by hashing the Alexa top 1M website domains and matching them against the Avazu data), you get access to the web histories of a few hundred thousand people (assuming a reasonable brute-forcing success rate), all personally identifiable. And that's bad.  

", "rawMarkdown": "", "votes": null }, { "id": "57686", "postDate": "11/10/2014 21:55:19", "content": "

Why bother with hashing at all ? In my view, the right way is to map each unique value to an incrementing integer ID:

\n

int new_id=0;
dictionary d;
for (each datum in data column) {
  if (!d.contains_key(datum)) {
     d[datum]=new_id; new_id++;
  }
  mapped_value=d[datum];
}

\n

This approach has the following advantages:

\n", "rawMarkdown": "", "votes": null }, { "id": "57687", "postDate": "11/10/2014 22:03:32", "content": "

[quote=B Yang;57686]

\n

Why bother with hashing at all ? In my view, the right way is to map each unique value to an incrementing integer ID:

\n

int new_id=0;
dictionary d;
for (each datum in data column) {
  if (!d.contains_key(datum)) {
     d[datum]=new_id; new_id++;
  }
  mapped_value=d[datum];
}

\n

This approach has the following advantages:

\n\n

[/quote]

\n\n

Agreed. I usually do this for long hashes anyway as it frees up more memory

", "rawMarkdown": "", "votes": null }, { "id": "57688", "postDate": "11/10/2014 22:03:39", "content": "

@B Yang: Actually this how I first processed this dataset. I sorted ids in decreasing order (so 0 value is the most frequent). It reduced the size of the data by 50%. I don't know if it helps someone but I'm attaching the script to do this. The list of columns is wrong (they changed the column labels) and I don't have the correct list.

", "rawMarkdown": "", "votes": null }, { "id": "57690", "postDate": "11/10/2014 22:43:34", "content": "

[quote=Paweł;57688]

\n

@B Yang: Actually this how I first processed this dataset. I sorted ids in decreasing order (so 0 value is the most frequent). It reduced the size of the data by 50%. ...

\n

[/quote]

\n

I do something similar see:

\n

pandas_extensions.py

\n\n

These are pandas extension methods, it includes a:

\n

DataFrame.compress(aggresiveness=0,to_sparse=False)

\n

method that reduces size of data by 60-85% it does the following:

\n\n\n

All these extensions use my naming conventions explained here:

\n

Guido's Naming Conventions

\n\n

So at least this fiasco has led to some ideas/code being shared.  

", "rawMarkdown": "", "votes": null }, { "id": "57692", "postDate": "11/10/2014 22:47:19", "content": "

@guido

\n

I'm going to have to create some sock-puppet accounts just so I can up-vote your pandas_extensions link!

\n

(j/k about the sock-puppet accounts)

", "rawMarkdown": "", "votes": null }, { "id": "57694", "postDate": "11/10/2014 22:58:40", "content": "

[quote=B Yang;57686]

\n

Why bother with hashing at all ? In my view, the right way is to map each unique value to an incrementing integer ID:

\n

int new_id=0;
dictionary d;
for (each datum in data column) {
  if (!d.contains_key(datum)) {
     d[datum]=new_id; new_id++;
  }
  mapped_value=d[datum];
}

\n

This approach has the following advantages:

\n\n

[/quote]

\n\n

Hashing with good hash functions is usually faster than dict lookup since dict does hashing as well and has additional random RAM access.

", "rawMarkdown": "", "votes": null }, { "id": "57695", "postDate": "11/10/2014 23:01:05", "content": "

@Guido: This extension is very nice!!! I like the _df_engineer method which lets you define your own column types. I did something like this in R for 1 competition.

\n

I have 2 additional lossless compression methods that I sometimes use:

\n

1. Removing columns with only 1 unique value

\n

2. Removing columns with duplicate data (I work with bag of words models very often so it happens very frequently).

\n

Both are pretty aggressive.

\n

https://gist.github.com/pjankiewicz/0773ab67e3dd5a4cc623

", "rawMarkdown": "", "votes": null }, { "id": "57717", "postDate": "11/11/2014 00:30:18", "content": "

[quote=Paweł;57695]

\n

1. Removing columns with only 1 unique value

\n

[/quote]

\n

Cool, but I think your get_useless_columns has a bug:

\n

line:

\n

for col in useless_columns

\n

should be:

\n

for col in data.columns:

\n

I will probably incorporate these into my compress function (if you dont object), my only concern is how fast is your hash_array function on large datasets, like the Avazu one?

", "rawMarkdown": "", "votes": null }, { "id": "57724", "postDate": "11/11/2014 00:59:50", "content": "

[quote=Guocong Song;57694]

\n

Hashing with good hash functions is usually faster than dict lookup since dict does hashing as well and has additional random RAM access.

\n

[/quote]

\n

I was thinking of binary tree-based dictionaries. There're still random RAM access, but I think they'd be faster than algorithms like MD5, especially for small input strings like those in this dataset.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 57411, "author_name": "jaymoore", "author_url": "", "post_date": "11/06/2014 16:02:30", "content": "

It could be interesting to correlate each of the levels of country with peak activity time of day, and at least infer longitude for each country...

", "votes": null, "replies": [] }, { "id": 57424, "author_name": "herrahuu", "author_url": "", "post_date": "11/06/2014 17:23:44", "content": "

Here is a plot of normalized hourly click ratios (ie click ratio for each hour / average ratio) for each country:

\n

\"click

\n

Not exactly what I would have expected to see. There seems to be some kind of general trend and not huge variations between the countries etc.

\n

Edit: Hmm, maybe this could be just a consequence of the reported subsampling of the data

", "votes": null, "replies": [] }, { "id": 57428, "author_name": "luyitian", "author_url": "", "post_date": "11/06/2014 18:45:41", "content": "

device_geo_country  = 'dc634e20'

\n

\n

I arbitrarily select several device-geo-country and find most of them have statistics like the picture above, but the location of peak is different.

", "votes": null, "replies": [] }, { "id": 57453, "author_name": "ozrobstar", "author_url": "", "post_date": "11/07/2014 02:45:32", "content": "

It's at least plausible that they are countries - ISO 3166 tabulates each recognised nation as a two character alpha, a three character alpha and a three character numeric - total 8 characters - and has a listing for around 250 nation-like entities.

\n

At the same time, it is also possible that a few countries - e.g. the USA - could be encoded at the state level. Not uncommon for online forms to request the user's country, and then request the user's state if and only they identify as from the US and/or a handful of other large countries.

", "votes": null, "replies": [] }, { "id": 57460, "author_name": "mike64", "author_url": "", "post_date": "11/07/2014 10:09:39", "content": "

Hi OzRob,

\n

It really is a good idea but can not find the correspondence between the two codes!

\n\n

For example, dc634e20?

", "votes": null, "replies": [] }, { "id": 57486, "author_name": "byang1", "author_url": "", "post_date": "11/07/2014 20:16:26", "content": "

Here's the top 10 countries in training data:

\n

1 13b5bfe9 India //md5('in') = "13b5bfe96f3e2fe411c9f66f4a582adf"
2 b80bb774 Indonesia
3 0b3b97fa United States
4 959848ca South Africa
5 1cd3c693 Pakistan
6 dc634e20 Brazil
7 c12e01f2 Saudi Arabia
8 795237fd Iraq
9 dcf0d7d2 Korea, Republic Of
10 dd07de85 Peru

\n\n

Now, let's go crack the other columns :)

", "votes": null, "replies": [] }, { "id": 57492, "author_name": "derekzh", "author_url": "", "post_date": "11/07/2014 22:34:02", "content": "

B Yang: it is amazing that you are decoding the top country - very curious about how you managed to do so if you don't mind sharing the technique :)

", "votes": null, "replies": [] }, { "id": 57493, "author_name": "nissanpow", "author_url": "", "post_date": "11/07/2014 22:51:23", "content": "

[quote=DerekZH;57492]

\n

B Yang: it is amazing that you are decoding the top country - very curious about how you managed to do so if you don't mind sharing the technique :)

\n

[/quote]

\n\n

He's using the first 8 chars from the md5 hash (as stated in the comment from line 1).

\n

MD5 ("in") = 13b5bfe96f3e2fe411c9f66f4a582adf
MD5 ("id") = b80bb7740288fda1f201890375a60c8f
MD5 ("us") = 0b3b97fa66886c5688ee4ae80ec0c3c2
MD5 ("za") = 959848ca10cc8a60da818ac11523dc63
MD5 ("pk") = 1cd3c693132f4c31b5b5e5f4c5eed6bd
MD5 ("br") = dc634e2072827fe0b5be9a2063390544
MD5 ("sa") = c12e01f2a13ff5587e1e9e4aedb8242d
MD5 ("iq") = 795237fd9d107e63cd19b0db0f2fba2f
MD5 ("kr") = dcf0d7d2cd120bf42580d43f29785dd3
MD5 ("pe") = dd07de8561395a7c37f8fcc1752d45e1

", "votes": null, "replies": [] }, { "id": 57494, "author_name": "matlab", "author_url": "", "post_date": "11/07/2014 23:38:41", "content": "

Full list

", "votes": null, "replies": [] }, { "id": 57499, "author_name": "byang1", "author_url": "", "post_date": "11/08/2014 03:50:46", "content": "

[quote=Nissan Pow;57493]

\n

He's using the first 8 chars from the md5 hash (as stated in the comment from line 1).

\n

[/quote]

\n

Yes, you hash the likely suspects and see if the leading bytes are in the data. For example, the top 5 device_os:

\n\n

And the top 10 device_make:

\n\n

I wonder what the #2 device_make is, I suspect it's something like '-unknown-'. You can go to this web site:

\n

    http://www.md5hashgenerator.com/index.php

\n

enter your guess and click on "Generate MD5 Hash".

", "votes": null, "replies": [] }, { "id": 57552, "author_name": "iamivo", "author_url": "", "post_date": "11/09/2014 12:37:24", "content": "

How can one IP have two DEVICE_GEO_COUNTRY values associated with it?

\n

For eg. c1b8122b DEVICE_IP has b80bb774 and 0b3b97fa.

\n

Cheers!

", "votes": null, "replies": [] }, { "id": 57591, "author_name": "danielhutty", "author_url": "", "post_date": "11/10/2014 11:46:18", "content": "

I would suspect that the owner of that phone has been in more than one country during the time period covered by the dataset. Whenever I travel I spend a fair amount of time messing around on my phone while waiting in airports, both while waiting for departure and waiting to get through arrivals - i.e. in 2 countries! I usually have data turned off when abroad due to roaming costs, but guess not everyone does so.

", "votes": null, "replies": [] }, { "id": 57607, "author_name": "iamivo", "author_url": "", "post_date": "11/10/2014 14:54:27", "content": "

I see, but the IP of your phone is differnet in Paris and London, the geolocation is different. The question is how can the same IP have two different geolocations.

", "votes": null, "replies": [] }, { "id": 57613, "author_name": "alexxanderlarko", "author_url": "", "post_date": "11/10/2014 16:00:21", "content": "

ivo@

\n

ivo wrote:

\n

The question is how can the same IP have two different geolocations.

\n

_________________________________________________________ 

\n

I think - this is not possible

", "votes": null, "replies": [] }, { "id": 57622, "author_name": "vvarkey", "author_url": "", "post_date": "11/10/2014 17:22:08", "content": "

probably means they are going state-wise in some countries (india). and some small states in india share ip geolocations.

", "votes": null, "replies": [] }, { "id": 57623, "author_name": "vvarkey", "author_url": "", "post_date": "11/10/2014 17:34:26", "content": "

on second thoughts there are 233 country codes and 232 values in the file. i'm guessing they are excluding only china.

\n

https://en.wikipedia.org/wiki/List_of_countries_by_IPv4_address_allocation

\n

there are 19 countries with 0 IPs. these probably use IP from nearby countries.

", "votes": null, "replies": [] }, { "id": 57650, "author_name": "fchollet", "author_url": "", "post_date": "11/10/2014 19:26:44", "content": "

Given the initial data debacle, I was fully expecting something like this. Seriously guys, you did not salt your hashes? Did you expect nobody would try using common hashes on simple fields to deanonymize the data? Like it has happened every single time unsalted hashes have been used in a competition? Guys.

\n

In case you're wondering, the reason this is problematic from a privacy perspective, is that by brute-forcing the device_ip column (by hashing a few billion IPs that have been active recently and matching the hashes with the Avazu data) and the site_domain column (by hashing the Alexa top 1M website domains and matching them against the Avazu data), you get access to the web histories of a few hundred thousand people (assuming a reasonable brute-forcing success rate), all personally identifiable. And that's bad.  

", "votes": null, "replies": [] }, { "id": 57686, "author_name": "byang1", "author_url": "", "post_date": "11/10/2014 21:55:19", "content": "

Why bother with hashing at all ? In my view, the right way is to map each unique value to an incrementing integer ID:

\n

int new_id=0;
dictionary d;
for (each datum in data column) {
  if (!d.contains_key(datum)) {
     d[datum]=new_id; new_id++;
  }
  mapped_value=d[datum];
}

\n

This approach has the following advantages:

\n", "votes": null, "replies": [] }, { "id": 57687, "author_name": "domcastro", "author_url": "", "post_date": "11/10/2014 22:03:32", "content": "

[quote=B Yang;57686]

\n

Why bother with hashing at all ? In my view, the right way is to map each unique value to an incrementing integer ID:

\n

int new_id=0;
dictionary d;
for (each datum in data column) {
  if (!d.contains_key(datum)) {
     d[datum]=new_id; new_id++;
  }
  mapped_value=d[datum];
}

\n

This approach has the following advantages:

\n\n

[/quote]

\n\n

Agreed. I usually do this for long hashes anyway as it frees up more memory

", "votes": null, "replies": [] }, { "id": 57688, "author_name": "paweljankiewicz", "author_url": "", "post_date": "11/10/2014 22:03:39", "content": "

@B Yang: Actually this how I first processed this dataset. I sorted ids in decreasing order (so 0 value is the most frequent). It reduced the size of the data by 50%. I don't know if it helps someone but I'm attaching the script to do this. The list of columns is wrong (they changed the column labels) and I don't have the correct list.

", "votes": null, "replies": [] }, { "id": 57690, "author_name": "guidotapia", "author_url": "", "post_date": "11/10/2014 22:43:34", "content": "

[quote=Paweł;57688]

\n

@B Yang: Actually this how I first processed this dataset. I sorted ids in decreasing order (so 0 value is the most frequent). It reduced the size of the data by 50%. ...

\n

[/quote]

\n

I do something similar see:

\n

pandas_extensions.py

\n\n

These are pandas extension methods, it includes a:

\n

DataFrame.compress(aggresiveness=0,to_sparse=False)

\n

method that reduces size of data by 60-85% it does the following:

\n\n\n

All these extensions use my naming conventions explained here:

\n

Guido's Naming Conventions

\n\n

So at least this fiasco has led to some ideas/code being shared.  

", "votes": null, "replies": [] }, { "id": 57692, "author_name": "inversion", "author_url": "", "post_date": "11/10/2014 22:47:19", "content": "

@guido

\n

I'm going to have to create some sock-puppet accounts just so I can up-vote your pandas_extensions link!

\n

(j/k about the sock-puppet accounts)

", "votes": null, "replies": [] }, { "id": 57694, "author_name": "songgc", "author_url": "", "post_date": "11/10/2014 22:58:40", "content": "

[quote=B Yang;57686]

\n

Why bother with hashing at all ? In my view, the right way is to map each unique value to an incrementing integer ID:

\n

int new_id=0;
dictionary d;
for (each datum in data column) {
  if (!d.contains_key(datum)) {
     d[datum]=new_id; new_id++;
  }
  mapped_value=d[datum];
}

\n

This approach has the following advantages:

\n\n

[/quote]

\n\n

Hashing with good hash functions is usually faster than dict lookup since dict does hashing as well and has additional random RAM access.

", "votes": null, "replies": [] }, { "id": 57695, "author_name": "paweljankiewicz", "author_url": "", "post_date": "11/10/2014 23:01:05", "content": "

@Guido: This extension is very nice!!! I like the _df_engineer method which lets you define your own column types. I did something like this in R for 1 competition.

\n

I have 2 additional lossless compression methods that I sometimes use:

\n

1. Removing columns with only 1 unique value

\n

2. Removing columns with duplicate data (I work with bag of words models very often so it happens very frequently).

\n

Both are pretty aggressive.

\n

https://gist.github.com/pjankiewicz/0773ab67e3dd5a4cc623

", "votes": null, "replies": [] }, { "id": 57717, "author_name": "guidotapia", "author_url": "", "post_date": "11/11/2014 00:30:18", "content": "

[quote=Paweł;57695]

\n

1. Removing columns with only 1 unique value

\n

[/quote]

\n

Cool, but I think your get_useless_columns has a bug:

\n

line:

\n

for col in useless_columns

\n

should be:

\n

for col in data.columns:

\n

I will probably incorporate these into my compress function (if you dont object), my only concern is how fast is your hash_array function on large datasets, like the Avazu one?

", "votes": null, "replies": [] }, { "id": 57724, "author_name": "byang1", "author_url": "", "post_date": "11/11/2014 00:59:50", "content": "

[quote=Guocong Song;57694]

\n

Hashing with good hash functions is usually faster than dict lookup since dict does hashing as well and has additional random RAM access.

\n

[/quote]

\n

I was thinking of binary tree-based dictionaries. There're still random RAM access, but I think they'd be faster than algorithms like MD5, especially for small input strings like those in this dataset.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "57405": "", "57411": "", "57424": "", "57428": "", "57453": "", "57460": "", "57486": "", "57492": "", "57493": "", "57494": "", "57499": "", "57552": "", "57591": "", "57607": "", "57613": "", "57622": "", "57623": "", "57650": "", "57686": "", "57687": "", "57688": "", "57690": "", "57692": "", "57694": "", "57695": "", "57717": "", "57724": "" }, "source": "meta" }