{
  "id": 10745,
  "title": "Beating the Benchmark ;)",
  "url": "/competitions/avazu-ctr-prediction/discussion/10745",
  "author_name": "",
  "post_date": "2014-10-25T11:30:27.297Z",
  "votes": 33,
  "comment_count": 44,
  "views": 14572,
  "content": "<p>Inspired by @tinrtgu (http://www.kaggle.com/users/185835/tinrtgu), I present to you a modified version of his script used in tradeshift challenge.</p>\n<p>LB Score: ~0.0885</p>\n<p>pypy takes around 5mins on my system.</p>\n<p>Vote up if this helped :D</p>",
  "messages": [
    {
      "id": "56731",
      "postDate": "10/25/2014 11:30:27",
      "content": "<p>Inspired by @tinrtgu (http://www.kaggle.com/users/185835/tinrtgu), I present to you a modified version of his script used in tradeshift challenge.</p>\n<p>LB Score: ~0.0885</p>\n<p>pypy takes around 5mins on my system.</p>\n<p>Vote up if this helped :D</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56732",
      "postDate": "10/25/2014 11:32:40",
      "content": "<p>Thanks Abhishek</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56733",
      "postDate": "10/25/2014 11:42:59",
      "content": "<p>This approach is so elegant. No dependencies. This stuff is 99.1% pure.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56734",
      "postDate": "10/25/2014 11:55:34",
      "content": "<p>Thanks Abhishek</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56735",
      "postDate": "10/25/2014 12:02:58",
      "content": "<p>Do I need to vote up too?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56736",
      "postDate": "10/25/2014 12:06:17",
      "content": "<p>[quote=tinrtgu;56735]</p>\n<p>Do I need to vote up too?</p>\n<p>[/quote]</p>\n<p>No, we will do that for you :D</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56737",
      "postDate": "10/25/2014 12:38:53",
      "content": "<p>Do you actually have a cron task which monitors new competitions, tests some methods and submits &quot;Beating ...&quot; posts? :)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56738",
      "postDate": "10/25/2014 12:41:19",
      "content": "<p>[quote=Eugene Nizhibitsky;56737]</p>\n<p>Do you actually have a cron task which monitors new competitions, tests some methods and submits &quot;Beating ...&quot; posts? :)</p>\n<p>[/quote]</p>\n<p>Nopes... But thats a nice idea :P</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56744",
      "postDate": "10/25/2014 13:24:57",
      "content": "<p>This code could be really popular and useful!&nbsp; I think&nbsp;tinrtgu can name it or make it a library so that we can easily reference it and give credits to him.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56746",
      "postDate": "10/25/2014 13:26:52",
      "content": "<p>@rcarson - not to take away from tinrtgu (I'm more than impressed by this piece of code) but it seems we already have vowpal wabbit which does exactly the same in a bigger package.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56747",
      "postDate": "10/25/2014 13:38:32",
      "content": "<p>[quote=Pawe&#322;;56746]</p>\n<p>@rcarson - not to take away from tinrtgu (I'm more than impressed by this piece of code) but it seems we already have vowpal wabbit which does exactly the same in a bigger package.</p>\n<p>[/quote]</p>\n<p>I agree! I am just trying to say&nbsp;this one shows exactly what's going on under the hood. One&nbsp;can simply apply other online algorithms with it and end up understanding the algorithm better. It can be a great tool for learning and educating.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56749",
      "postDate": "10/25/2014 14:23:36",
      "content": "<p>Isn't there a mistake in the code in the way how the function hashes the train/test variables?</p>\n<p>The hashing is inconsistent because the m index of the column is different for train/test. When I try to hash the same data with and without the &quot;click&quot; column it produces different indices. My solution is to replace this</p>\n<p><code>x[m] = abs(hash(str(m) + '_' + feat)) % D</code></p>\n<p>with this</p>\n<p><code>if traindata:<br>&nbsp; &nbsp; &nbsp;x[m] = abs(hash(str(m) + '_' + feat)) % D<br> else:<br>&nbsp; &nbsp; &nbsp;x[m+1] = abs(hash(str(m+1) + '_' + feat)) % D</code></p>\n<p>Strangely enough I got a worse result than the original code.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56750",
      "postDate": "10/25/2014 14:34:03",
      "content": "",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56754",
      "postDate": "10/25/2014 15:39:23",
      "content": "<p>There must be other bug in the code. This model should beat constant prediction easily - which it doesn't currently.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56755",
      "postDate": "10/25/2014 15:45:16",
      "content": "<p>that's because the seasonality is too strong. if you hash 'hh' rather than 'yymmddhh', you can beat the constant prediction, which is what we just did : )</p>\n<p>Edit: actually I believe that has something to do with no clicks after 2 am.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56774",
      "postDate": "10/25/2014 23:19:44",
      "content": "",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56948",
      "postDate": "10/28/2014 22:37:53",
      "content": "<p>This is a corrected version of the&nbsp;code.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57019",
      "postDate": "10/29/2014 21:36:04",
      "content": "<p>[quote=Pawe&#322;;56948]</p>\n<p>This is a corrected version of the&nbsp;code.</p>\n<p>[/quote]</p>\n\n<p>How was the code corrected?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57024",
      "postDate": "10/29/2014 22:01:40",
      "content": "<p>@Inspector: Look at my post above. Abhishek version has a bug.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57217",
      "postDate": "11/02/2014 21:06:12",
      "content": "<p>What do you recommend doing with features that are only present in the testing data?</p>\n\n<p>Do you use some code block like:</p>\n\n<p>try</p>\n<p style=\"padding-left: 30px\">wTx += w[i] * x[i]</p>\n<p>except KeyError:</p>\n<p style=\"padding-left: 30px\">wTx += 0</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57219",
      "postDate": "11/02/2014 21:57:18",
      "content": "<p>[quote=nkconnor;57217]</p>\n<p>What do you recommend doing with features that are only present in the testing data?</p>\n<p>Do you use some code block like:</p>\n<p>try</p>\n<p style=\"padding-left: 30px\">wTx += w[i] * x[i]</p>\n<p>except KeyError:</p>\n<p style=\"padding-left: 30px\">wTx += 0</p>\n<p>[/quote]</p>\n<p>Nope, hashing and mod by D already takes&nbsp;care of that. It will be a hashing collision.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57220",
      "postDate": "11/02/2014 23:10:57",
      "content": "<p>Ah, thanks. I see that now.&nbsp;I am doing things a bit differently by generating all the possible keys before the regression, so I will implement something a bit special for this case.</p>\n\n<p>I think&nbsp;I will form it so that features&nbsp;in the test set that were not&nbsp;encountered before will inherit w[i] from infrequently encountered features.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57230",
      "postDate": "11/03/2014 07:16:49",
      "content": "<p>Does the shared&nbsp;code handle intercept term in logistic regression correctly? Is zero-indexed weight intended&nbsp;to be intercept? If yes, then the code has a bug.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57235",
      "postDate": "11/03/2014 11:14:00",
      "content": "<p>[quote=rcarson;57219]</p>\n<p>[quote=nkconnor;57217]</p>\n<p>What do you recommend doing with features that are only present in the testing data?</p>\n<p>Do you use some code block like:</p>\n<p>try</p>\n<p style=\"padding-left: 30px\">wTx += w[i] * x[i]</p>\n<p>except KeyError:</p>\n<p style=\"padding-left: 30px\">wTx += 0</p>\n<p>[/quote]</p>\n<p>Nope, hashing and mod by D already takes&nbsp;care of that. It will be a hashing collision.</p>\n<p>[/quote]</p>\n<p>Hi,rcarson:</p>\n<p>I have two questions.</p>\n<p>1.you said that can process the features which are only present in the testing data. because of hashing collision,but if &nbsp;there maybe a case the value of hashed feature didn't have a hash collision.for example. x is never present in C10 in training data but test data.now hash(&quot;C10_&quot;+x) %D=1234,it still not present in history hashed value.at this time ,I think we still need to use&nbsp;</p>\n<p>try</p>\n<p>wTx += w[i] * x[i]</p>\n<p>except KeyError:</p>\n<p>wTx += 0</p>\n<p>to solve this problem.</p>\n<p>2.How can I decide the value of D about hash tracking.I know Someone make it 2^20,but maybe the feature size more than 2^20,that maybe cause large amount of hash collision.many different hash value may map into a same value.How do you think about this.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57236",
      "postDate": "11/03/2014 11:19:45",
      "content": "<p>I think nkconnor is right. Features that are not in the training data should be omitted. Sure, hashing trick will put them somewhere but they are just unwanted noise.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57238",
      "postDate": "11/03/2014 12:41:53",
      "content": "<p>[quote=Pawe&#322;;57236]</p>\n<p>I think nkconnor is right. Features that are not in the training data should be omitted. Sure, hashing trick will put them somewhere but they are just unwanted noise.</p>\n<p>[/quote]</p>\n<p>Yeah,according to training data set ,we would get feature list,hashing trick will put them somewhere,but we must remove it in actually computation.</p>\n<p>Is there some introduction about hash tracking.because this is my first time to touch this concept.</p>\n<p>do you know how to confirm the value of D.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57239",
      "postDate": "11/03/2014 12:43:48",
      "content": "<p>[quote=JerryZhong;57238]</p>\n<p>Is there some introduction about hash tracking.because this is my first time to touch this concept.</p>\n<p>[/quote]</p>\n<p>Some good discussions here:</p>\n<p>https://www.kaggle.com/c/tradeshift-text-classification/forums/t/10561/help-hash-values</p>\n<p>https://www.kaggle.com/c/tradeshift-text-classification/forums/t/10512/purpose-of-hashing-function</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57241",
      "postDate": "11/03/2014 13:44:53",
      "content": "<p>[quote=JerryZhong;57235]</p>\n\n<p>Hi,rcarson:</p>\n<p>I have two questions.</p>\n<p>1.you said that can process the features which are only present in the testing data. because of hashing collision,but if &nbsp;there maybe a case the value of hashed feature didn't have a hash collision.for example. x is never present in C10 in training data but test data.now hash(&quot;C10_&quot;+x) %D=1234,it still not present in history hashed value.at this time ,I think we still need to use&nbsp;</p>\n<p>try</p>\n<p>wTx += w[i] * x[i]</p>\n<p>except KeyError:</p>\n<p>wTx += 0</p>\n<p>to solve this problem.</p>\n<p>2.How can I decide the value of D about hash tracking.I know Someone make it 2^20,but maybe the feature size more than 2^20,that maybe cause large amount of hash collision.many different hash value may map into a same value.How do you think about this.</p>\n<p>[/quote]</p>\n<p>Hi,</p>\n<p>Sorry, I'm no expert of this code and by no means I mean hashing collision a good thing. In my opinion, it is all about speed and memory footprint.&nbsp;Hashing trick is fast and simple and adaptable to&nbsp;different memory capacity, that's why everybody loves it. If x is defined as [0.]*D, I can see no way there will be a key error. If you want to have a list of all possible features, I'm not sure it will fit into memory, given the data size. If you want to check for every sample, I'm not sure what the slowdown either.</p>\n<p>After all, with such a simple classifier, I don't expect it can differentiate such minor noise like unseen features. But don't let this stop you from correcting the error. Maybe it is significant and please share it whatever the result. It helps everybody understands better. Thank you.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57292",
      "postDate": "11/04/2014 12:13:42",
      "content": "<p>[quote=rcarson;57241]</p>\n<p>Hi,</p>\n<p>Sorry, I'm no expert of this code and by no means I mean hashing collision a good thing. In my opinion, it is all about speed and memory footprint.&nbsp;Hashing trick is fast and simple and adaptable to&nbsp;different memory capacity, that's why everybody loves it. If x is defined as [0.]*D, I can see no way there will be a key error. If you want to have a list of all possible features, I'm not sure it will fit into memory, given the data size. If you want to check for every sample, I'm not sure what the slowdown either.</p>\n<p>After all, with such a simple classifier, I don't expect it can differentiate such minor noise like unseen features. But don't let this stop you from correcting the error. Maybe it is significant and please share it whatever the result. It helps everybody understands better. Thank you.</p>\n<p>[/quote]</p>\n<p>Sorry for my late reply.Thank you for your answer,When I learn information from other paper,I think I have get a satisfying answer.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57293",
      "postDate": "11/04/2014 12:13:52",
      "content": "",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57465",
      "postDate": "11/07/2014 14:42:55",
      "content": "<p>Hi, can somebody enlighten me a bit on the following line in the code?</p>\n<p>x[m] = abs(hash(str(m) + '_' + feat)) % D</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57467",
      "postDate": "11/07/2014 15:17:16",
      "content": "<p>[quote=cipolla;57465]</p>\n<p>Hi, can somebody enlighten me a bit on the following line in the code?</p>\n<p>x[m] = abs(hash(str(m) + '_' + feat)) % D</p>\n<p>[/quote]</p>\n<p>Which part, exactly? What I mean is, do you understand what the line is doing conceptually, but don't understand the implementation (or a part of the implementation)? Or, are you asking why the values are being hashed in the first place. etc.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57468",
      "postDate": "11/07/2014 15:18:09",
      "content": "<p>[quote=cipolla;57465]</p>\n<p>Hi, can somebody enlighten me a bit on the following line in the code?</p>\n<p>x[m] = abs(hash(str(m) + '_' + feat)) % D</p>\n<p>[/quote]</p>\n<p>I am not an expert, but this is the idea from my research - hopefully others will correct is there are any errors.</p>\n<p>This is the 'hashing trick'. Essentially, feature values are 'hashed' (or 'sent to', if you will)&nbsp; to a given bucket in an array.</p>\n<p>The hash() function - which as noted is not terribly robust and there are much better ones available, such as murmurhash3 used by vowpal wabbit - will take a string that is the concatenation of the feature name and its value (to make it unique in case there are multiple features that share common values) and transform that string to an integer.</p>\n<p>The hash function will return an integer uniformly at random but should always return the same value for a given input (required of course for scoring a test set). The absolute value of that result is then divided by D and the remainder (modulo) is returned. This is the bin or bucket that the feature value is hashed to and will become essentially a categorical variable that is one-hot-encoded and used in the model.</p>\n<p>The larger is D, the larger the possible number of buckets in the array and thus the larger number of one-hot-encoded variables with weights from the training process. If D=100 then there are 100 possible values that can result from from x%D, where x is the integer being divided by D.</p>\n<p>The larger D, the less collisions (meaning different values of features are hashed into the same bucket and thus not distinguishable from each other). Sometimes a larger D is better, but not always, as a smaller D (note, smaller still means really large!) acts as dimensionality reduction. Hope this helps.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57470",
      "postDate": "11/07/2014 15:38:55",
      "content": "<p>Thank you inversion for the note. &nbsp;I am quite confused about the modulo operation. &nbsp;As I understand, by doing the hash trick, one can convert the string features into some integer numbers. &nbsp;What is the point of having the modulo operation on top of that?</p>\n\n<p>[quote=inversion;57467]</p>\n<p>[quote=cipolla;57465]</p>\n<p>Hi, can somebody enlighten me a bit on the following line in the code?</p>\n<p>x[m] = abs(hash(str(m) + '_' + feat)) % D</p>\n<p>[/quote]</p>\n<p>Which part, exactly? What I mean is, do you understand what the line is doing conceptually, but don't understand the implementation (or a part of the implementation)? Or, are you asking why the values are being hashed in the first place. etc.</p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57471",
      "postDate": "11/07/2014 15:49:04",
      "content": "<p>The reply covers almost everything I can imagine :) &nbsp;Thank you!</p>\n<p>So the only purpose of calculating the remainder is to reduce the number of categories of a feature so is the dimensionality?</p>\n\n<p>[quote=Inspector;57468]</p>\n<p>[quote=cipolla;57465]</p>\n<p>Hi, can somebody enlighten me a bit on the following line in the code?</p>\n<p>x[m] = abs(hash(str(m) + '_' + feat)) % D</p>\n<p>[/quote]</p>\n<p>I am not an expert, but this is the idea from my research - hopefully others will correct is there are any errors.</p>\n<p>This is the 'hashing trick'. Essentially, feature values are 'hashed' (or 'sent to', if you will)&nbsp; to a given bucket in an array.</p>\n<p>The hash() function - which as noted is not terribly robust and there are much better ones available, such as murmurhash3 used by vowpal wabbit - will take a string that is the concatenation of the feature name and its value (to make it unique in case there are multiple features that share common values) and transform that string to an integer.</p>\n<p>The hash function will return an integer uniformly at random but should always return the same value for a given input (required of course for scoring a test set). The absolute value of that result is then divided by D and the remainder (modulo) is returned. This is the bin or bucket that the feature value is hashed to and will become essentially a categorical variable that is one-hot-encoded and used in the model.</p>\n<p>The larger is D, the larger the possible number of buckets in the array and thus the larger number of one-hot-encoded variables with weights from the training process. If D=100 then there are 100 possible values that can result from from x%D, where x is the integer being divided by D.</p>\n<p>The larger D, the less collisions (meaning different values of features are hashed into the same bucket and thus not distinguishable from each other). Sometimes a larger D is better, but not always, as a smaller D (note, smaller still means really large!) acts as dimensionality reduction. Hope this helps.</p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57472",
      "postDate": "11/07/2014 15:54:25",
      "content": "<p>[quote=cipolla;57471]</p>\n<p>The reply covers almost everything I can imagine :) &nbsp;Thank you!</p>\n<p>So the only purpose of calculating the remainder is to reduce the number of categories of a feature so is the dimensionality?</p>\n<p>[/quote]</p>\n\n<p>Your welcome, glad it helped. I was confused when I first encountered it.</p>\n<p>Yes, I believe that is the reason for D, else I think hash() would end up hashing every value to a distinct integer (I am sure within some limit). I cant claim to know this for an absolute fact, but I *think* that is correct.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57473",
      "postDate": "11/07/2014 15:54:54",
      "content": "<p>[quote=cipolla;57471]</p>\n<p>The reply covers almost everything I can imagine :) &nbsp;Thank you!</p>\n<p>So the only purpose of calculating the remainder is to reduce the number of categories of a feature so is the dimensionality?</p>\n<p>[/quote]</p>\n<p>On my computer, when I type&nbsp;hash('somestring'),&nbsp;I get back 43209540836401356. Since you need to store&nbsp;a weight for each of&nbsp;your potential attributes, you'll need a weight array at least that long. Taking the modulo trims the number of possible weights&nbsp;down to a manageable amount.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57931",
      "postDate": "11/12/2014 17:26:09",
      "content": "<p>Hello all,</p>\n<p>I seperate the trans_rev2 to 4 small files.</p>\n<p>WHY i get my Logloss higer at each print,19.999807,19.999903....</p>\n<p>do you have the same problem like mine?</p>\n<p>I didn't change other default settings in corrected fast_solution.py</p>\n<p>and i tried smaller learning_rate, don't work.....</p>\n<p>Thanks!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57947",
      "postDate": "11/12/2014 21:40:44",
      "content": "<p>It's because the file is in sequence order.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57949",
      "postDate": "11/12/2014 22:34:39",
      "content": "<p>[quote=ACS69;57947]</p>\n<p>It's because the file is in sequence order.</p>\n<p>[/quote]</p>\n<p>sorry, i still don't get the point why the log loss get higher. can you explicate a little more?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57950",
      "postDate": "11/12/2014 22:35:31",
      "content": "<p>[quote=ACS69;57947]</p>\n<p>It's because the file is in sequence order.</p>\n<p>[/quote]</p>\n<p>sorry, i still don't get the point why the log loss get higher. can you explicate a little more?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57951",
      "postDate": "11/12/2014 22:43:04",
      "content": "<p>I'm no expert, but I would think it's because they're in similar groupings (hour) - the way the online code works will pick up the similar features - so for the same hour the log loss gets better, then the hour changes and there's new features so the log loss may get worse, then level out again or improve and so on . If you randomly shuffle the file before running the code, you will get the pattern that you are expecting</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57952",
      "postDate": "11/12/2014 22:53:55",
      "content": "<p>Thanks for you explication.</p>\n<p>But if it is the case, which means log loss will have a sudden drop in performance while reaching the next hour data, &nbsp;we'll see a drop each 200k data (as the Q&amp;A said each hour has 200k data). As the published print code print the performance each 100k data, so the log loss will not be go up each print.</p>\n<p>i don't get this because in the Criteo CTR competition, the data is also order by time but the log loss go down without data permutation.</p>\n<p>Anyone else has the same&nbsp;result like mine?&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57953",
      "postDate": "11/12/2014 23:06:33",
      "content": "<p>Yes - it will happen to everyone who runs the online code on the sequenced data.&nbsp; Shuffle the data to get consistent log loss improvement. I used &quot;hour&quot; as an example - I haven't looked at the details to see what is causing the drops</p>\n<p>edit: I didn't enter the other competition so I don't know what the data was like</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "313324",
      "postDate": "04/13/2018 05:42:01",
      "content": "<p>simple and effective method </p>",
      "rawMarkdown": "simple and effective method",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 56732,
      "author_name": "drpatrickchan",
      "author_url": "",
      "post_date": "10/25/2014 11:32:40",
      "content": "<p>Thanks Abhishek</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56733,
      "author_name": "paweljankiewicz",
      "author_url": "",
      "post_date": "10/25/2014 11:42:59",
      "content": "<p>This approach is so elegant. No dependencies. This stuff is 99.1% pure.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56734,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "10/25/2014 11:55:34",
      "content": "<p>Thanks Abhishek</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56735,
      "author_name": "ggglhf",
      "author_url": "",
      "post_date": "10/25/2014 12:02:58",
      "content": "<p>Do I need to vote up too?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56736,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "10/25/2014 12:06:17",
      "content": "<p>[quote=tinrtgu;56735]</p>\n<p>Do I need to vote up too?</p>\n<p>[/quote]</p>\n<p>No, we will do that for you :D</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56737,
      "author_name": "nizhibitsky",
      "author_url": "",
      "post_date": "10/25/2014 12:38:53",
      "content": "<p>Do you actually have a cron task which monitors new competitions, tests some methods and submits &quot;Beating ...&quot; posts? :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56738,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "10/25/2014 12:41:19",
      "content": "<p>[quote=Eugene Nizhibitsky;56737]</p>\n<p>Do you actually have a cron task which monitors new competitions, tests some methods and submits &quot;Beating ...&quot; posts? :)</p>\n<p>[/quote]</p>\n<p>Nopes... But thats a nice idea :P</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56744,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "10/25/2014 13:24:57",
      "content": "<p>This code could be really popular and useful!&nbsp; I think&nbsp;tinrtgu can name it or make it a library so that we can easily reference it and give credits to him.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56746,
      "author_name": "paweljankiewicz",
      "author_url": "",
      "post_date": "10/25/2014 13:26:52",
      "content": "<p>@rcarson - not to take away from tinrtgu (I'm more than impressed by this piece of code) but it seems we already have vowpal wabbit which does exactly the same in a bigger package.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56747,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "10/25/2014 13:38:32",
      "content": "<p>[quote=Pawe&#322;;56746]</p>\n<p>@rcarson - not to take away from tinrtgu (I'm more than impressed by this piece of code) but it seems we already have vowpal wabbit which does exactly the same in a bigger package.</p>\n<p>[/quote]</p>\n<p>I agree! I am just trying to say&nbsp;this one shows exactly what's going on under the hood. One&nbsp;can simply apply other online algorithms with it and end up understanding the algorithm better. It can be a great tool for learning and educating.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56749,
      "author_name": "paweljankiewicz",
      "author_url": "",
      "post_date": "10/25/2014 14:23:36",
      "content": "<p>Isn't there a mistake in the code in the way how the function hashes the train/test variables?</p>\n<p>The hashing is inconsistent because the m index of the column is different for train/test. When I try to hash the same data with and without the &quot;click&quot; column it produces different indices. My solution is to replace this</p>\n<p><code>x[m] = abs(hash(str(m) + '_' + feat)) % D</code></p>\n<p>with this</p>\n<p><code>if traindata:<br>&nbsp; &nbsp; &nbsp;x[m] = abs(hash(str(m) + '_' + feat)) % D<br> else:<br>&nbsp; &nbsp; &nbsp;x[m+1] = abs(hash(str(m+1) + '_' + feat)) % D</code></p>\n<p>Strangely enough I got a worse result than the original code.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56750,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "10/25/2014 14:34:03",
      "content": "",
      "votes": null,
      "replies": []
    },
    {
      "id": 56754,
      "author_name": "paweljankiewicz",
      "author_url": "",
      "post_date": "10/25/2014 15:39:23",
      "content": "<p>There must be other bug in the code. This model should beat constant prediction easily - which it doesn't currently.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56755,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "10/25/2014 15:45:16",
      "content": "<p>that's because the seasonality is too strong. if you hash 'hh' rather than 'yymmddhh', you can beat the constant prediction, which is what we just did : )</p>\n<p>Edit: actually I believe that has something to do with no clicks after 2 am.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56774,
      "author_name": "nicholasconnor",
      "author_url": "",
      "post_date": "10/25/2014 23:19:44",
      "content": "",
      "votes": null,
      "replies": []
    },
    {
      "id": 56948,
      "author_name": "paweljankiewicz",
      "author_url": "",
      "post_date": "10/28/2014 22:37:53",
      "content": "<p>This is a corrected version of the&nbsp;code.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57019,
      "author_name": "inspector",
      "author_url": "",
      "post_date": "10/29/2014 21:36:04",
      "content": "<p>[quote=Pawe&#322;;56948]</p>\n<p>This is a corrected version of the&nbsp;code.</p>\n<p>[/quote]</p>\n\n<p>How was the code corrected?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57024,
      "author_name": "paweljankiewicz",
      "author_url": "",
      "post_date": "10/29/2014 22:01:40",
      "content": "<p>@Inspector: Look at my post above. Abhishek version has a bug.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57217,
      "author_name": "nicholasconnor",
      "author_url": "",
      "post_date": "11/02/2014 21:06:12",
      "content": "<p>What do you recommend doing with features that are only present in the testing data?</p>\n\n<p>Do you use some code block like:</p>\n\n<p>try</p>\n<p style=\"padding-left: 30px\">wTx += w[i] * x[i]</p>\n<p>except KeyError:</p>\n<p style=\"padding-left: 30px\">wTx += 0</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57219,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "11/02/2014 21:57:18",
      "content": "<p>[quote=nkconnor;57217]</p>\n<p>What do you recommend doing with features that are only present in the testing data?</p>\n<p>Do you use some code block like:</p>\n<p>try</p>\n<p style=\"padding-left: 30px\">wTx += w[i] * x[i]</p>\n<p>except KeyError:</p>\n<p style=\"padding-left: 30px\">wTx += 0</p>\n<p>[/quote]</p>\n<p>Nope, hashing and mod by D already takes&nbsp;care of that. It will be a hashing collision.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57220,
      "author_name": "nicholasconnor",
      "author_url": "",
      "post_date": "11/02/2014 23:10:57",
      "content": "<p>Ah, thanks. I see that now.&nbsp;I am doing things a bit differently by generating all the possible keys before the regression, so I will implement something a bit special for this case.</p>\n\n<p>I think&nbsp;I will form it so that features&nbsp;in the test set that were not&nbsp;encountered before will inherit w[i] from infrequently encountered features.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57230,
      "author_name": "jiayuan",
      "author_url": "",
      "post_date": "11/03/2014 07:16:49",
      "content": "<p>Does the shared&nbsp;code handle intercept term in logistic regression correctly? Is zero-indexed weight intended&nbsp;to be intercept? If yes, then the code has a bug.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57235,
      "author_name": "jerryzhong",
      "author_url": "",
      "post_date": "11/03/2014 11:14:00",
      "content": "<p>[quote=rcarson;57219]</p>\n<p>[quote=nkconnor;57217]</p>\n<p>What do you recommend doing with features that are only present in the testing data?</p>\n<p>Do you use some code block like:</p>\n<p>try</p>\n<p style=\"padding-left: 30px\">wTx += w[i] * x[i]</p>\n<p>except KeyError:</p>\n<p style=\"padding-left: 30px\">wTx += 0</p>\n<p>[/quote]</p>\n<p>Nope, hashing and mod by D already takes&nbsp;care of that. It will be a hashing collision.</p>\n<p>[/quote]</p>\n<p>Hi,rcarson:</p>\n<p>I have two questions.</p>\n<p>1.you said that can process the features which are only present in the testing data. because of hashing collision,but if &nbsp;there maybe a case the value of hashed feature didn't have a hash collision.for example. x is never present in C10 in training data but test data.now hash(&quot;C10_&quot;+x) %D=1234,it still not present in history hashed value.at this time ,I think we still need to use&nbsp;</p>\n<p>try</p>\n<p>wTx += w[i] * x[i]</p>\n<p>except KeyError:</p>\n<p>wTx += 0</p>\n<p>to solve this problem.</p>\n<p>2.How can I decide the value of D about hash tracking.I know Someone make it 2^20,but maybe the feature size more than 2^20,that maybe cause large amount of hash collision.many different hash value may map into a same value.How do you think about this.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57236,
      "author_name": "paweljankiewicz",
      "author_url": "",
      "post_date": "11/03/2014 11:19:45",
      "content": "<p>I think nkconnor is right. Features that are not in the training data should be omitted. Sure, hashing trick will put them somewhere but they are just unwanted noise.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57238,
      "author_name": "jerryzhong",
      "author_url": "",
      "post_date": "11/03/2014 12:41:53",
      "content": "<p>[quote=Pawe&#322;;57236]</p>\n<p>I think nkconnor is right. Features that are not in the training data should be omitted. Sure, hashing trick will put them somewhere but they are just unwanted noise.</p>\n<p>[/quote]</p>\n<p>Yeah,according to training data set ,we would get feature list,hashing trick will put them somewhere,but we must remove it in actually computation.</p>\n<p>Is there some introduction about hash tracking.because this is my first time to touch this concept.</p>\n<p>do you know how to confirm the value of D.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57239,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "11/03/2014 12:43:48",
      "content": "<p>[quote=JerryZhong;57238]</p>\n<p>Is there some introduction about hash tracking.because this is my first time to touch this concept.</p>\n<p>[/quote]</p>\n<p>Some good discussions here:</p>\n<p>https://www.kaggle.com/c/tradeshift-text-classification/forums/t/10561/help-hash-values</p>\n<p>https://www.kaggle.com/c/tradeshift-text-classification/forums/t/10512/purpose-of-hashing-function</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57241,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "11/03/2014 13:44:53",
      "content": "<p>[quote=JerryZhong;57235]</p>\n\n<p>Hi,rcarson:</p>\n<p>I have two questions.</p>\n<p>1.you said that can process the features which are only present in the testing data. because of hashing collision,but if &nbsp;there maybe a case the value of hashed feature didn't have a hash collision.for example. x is never present in C10 in training data but test data.now hash(&quot;C10_&quot;+x) %D=1234,it still not present in history hashed value.at this time ,I think we still need to use&nbsp;</p>\n<p>try</p>\n<p>wTx += w[i] * x[i]</p>\n<p>except KeyError:</p>\n<p>wTx += 0</p>\n<p>to solve this problem.</p>\n<p>2.How can I decide the value of D about hash tracking.I know Someone make it 2^20,but maybe the feature size more than 2^20,that maybe cause large amount of hash collision.many different hash value may map into a same value.How do you think about this.</p>\n<p>[/quote]</p>\n<p>Hi,</p>\n<p>Sorry, I'm no expert of this code and by no means I mean hashing collision a good thing. In my opinion, it is all about speed and memory footprint.&nbsp;Hashing trick is fast and simple and adaptable to&nbsp;different memory capacity, that's why everybody loves it. If x is defined as [0.]*D, I can see no way there will be a key error. If you want to have a list of all possible features, I'm not sure it will fit into memory, given the data size. If you want to check for every sample, I'm not sure what the slowdown either.</p>\n<p>After all, with such a simple classifier, I don't expect it can differentiate such minor noise like unseen features. But don't let this stop you from correcting the error. Maybe it is significant and please share it whatever the result. It helps everybody understands better. Thank you.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57292,
      "author_name": "jerryzhong",
      "author_url": "",
      "post_date": "11/04/2014 12:13:42",
      "content": "<p>[quote=rcarson;57241]</p>\n<p>Hi,</p>\n<p>Sorry, I'm no expert of this code and by no means I mean hashing collision a good thing. In my opinion, it is all about speed and memory footprint.&nbsp;Hashing trick is fast and simple and adaptable to&nbsp;different memory capacity, that's why everybody loves it. If x is defined as [0.]*D, I can see no way there will be a key error. If you want to have a list of all possible features, I'm not sure it will fit into memory, given the data size. If you want to check for every sample, I'm not sure what the slowdown either.</p>\n<p>After all, with such a simple classifier, I don't expect it can differentiate such minor noise like unseen features. But don't let this stop you from correcting the error. Maybe it is significant and please share it whatever the result. It helps everybody understands better. Thank you.</p>\n<p>[/quote]</p>\n<p>Sorry for my late reply.Thank you for your answer,When I learn information from other paper,I think I have get a satisfying answer.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57293,
      "author_name": "jerryzhong",
      "author_url": "",
      "post_date": "11/04/2014 12:13:52",
      "content": "",
      "votes": null,
      "replies": []
    },
    {
      "id": 57465,
      "author_name": "cipolla",
      "author_url": "",
      "post_date": "11/07/2014 14:42:55",
      "content": "<p>Hi, can somebody enlighten me a bit on the following line in the code?</p>\n<p>x[m] = abs(hash(str(m) + '_' + feat)) % D</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57467,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "11/07/2014 15:17:16",
      "content": "<p>[quote=cipolla;57465]</p>\n<p>Hi, can somebody enlighten me a bit on the following line in the code?</p>\n<p>x[m] = abs(hash(str(m) + '_' + feat)) % D</p>\n<p>[/quote]</p>\n<p>Which part, exactly? What I mean is, do you understand what the line is doing conceptually, but don't understand the implementation (or a part of the implementation)? Or, are you asking why the values are being hashed in the first place. etc.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57468,
      "author_name": "inspector",
      "author_url": "",
      "post_date": "11/07/2014 15:18:09",
      "content": "<p>[quote=cipolla;57465]</p>\n<p>Hi, can somebody enlighten me a bit on the following line in the code?</p>\n<p>x[m] = abs(hash(str(m) + '_' + feat)) % D</p>\n<p>[/quote]</p>\n<p>I am not an expert, but this is the idea from my research - hopefully others will correct is there are any errors.</p>\n<p>This is the 'hashing trick'. Essentially, feature values are 'hashed' (or 'sent to', if you will)&nbsp; to a given bucket in an array.</p>\n<p>The hash() function - which as noted is not terribly robust and there are much better ones available, such as murmurhash3 used by vowpal wabbit - will take a string that is the concatenation of the feature name and its value (to make it unique in case there are multiple features that share common values) and transform that string to an integer.</p>\n<p>The hash function will return an integer uniformly at random but should always return the same value for a given input (required of course for scoring a test set). The absolute value of that result is then divided by D and the remainder (modulo) is returned. This is the bin or bucket that the feature value is hashed to and will become essentially a categorical variable that is one-hot-encoded and used in the model.</p>\n<p>The larger is D, the larger the possible number of buckets in the array and thus the larger number of one-hot-encoded variables with weights from the training process. If D=100 then there are 100 possible values that can result from from x%D, where x is the integer being divided by D.</p>\n<p>The larger D, the less collisions (meaning different values of features are hashed into the same bucket and thus not distinguishable from each other). Sometimes a larger D is better, but not always, as a smaller D (note, smaller still means really large!) acts as dimensionality reduction. Hope this helps.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57470,
      "author_name": "cipolla",
      "author_url": "",
      "post_date": "11/07/2014 15:38:55",
      "content": "<p>Thank you inversion for the note. &nbsp;I am quite confused about the modulo operation. &nbsp;As I understand, by doing the hash trick, one can convert the string features into some integer numbers. &nbsp;What is the point of having the modulo operation on top of that?</p>\n\n<p>[quote=inversion;57467]</p>\n<p>[quote=cipolla;57465]</p>\n<p>Hi, can somebody enlighten me a bit on the following line in the code?</p>\n<p>x[m] = abs(hash(str(m) + '_' + feat)) % D</p>\n<p>[/quote]</p>\n<p>Which part, exactly? What I mean is, do you understand what the line is doing conceptually, but don't understand the implementation (or a part of the implementation)? Or, are you asking why the values are being hashed in the first place. etc.</p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57471,
      "author_name": "cipolla",
      "author_url": "",
      "post_date": "11/07/2014 15:49:04",
      "content": "<p>The reply covers almost everything I can imagine :) &nbsp;Thank you!</p>\n<p>So the only purpose of calculating the remainder is to reduce the number of categories of a feature so is the dimensionality?</p>\n\n<p>[quote=Inspector;57468]</p>\n<p>[quote=cipolla;57465]</p>\n<p>Hi, can somebody enlighten me a bit on the following line in the code?</p>\n<p>x[m] = abs(hash(str(m) + '_' + feat)) % D</p>\n<p>[/quote]</p>\n<p>I am not an expert, but this is the idea from my research - hopefully others will correct is there are any errors.</p>\n<p>This is the 'hashing trick'. Essentially, feature values are 'hashed' (or 'sent to', if you will)&nbsp; to a given bucket in an array.</p>\n<p>The hash() function - which as noted is not terribly robust and there are much better ones available, such as murmurhash3 used by vowpal wabbit - will take a string that is the concatenation of the feature name and its value (to make it unique in case there are multiple features that share common values) and transform that string to an integer.</p>\n<p>The hash function will return an integer uniformly at random but should always return the same value for a given input (required of course for scoring a test set). The absolute value of that result is then divided by D and the remainder (modulo) is returned. This is the bin or bucket that the feature value is hashed to and will become essentially a categorical variable that is one-hot-encoded and used in the model.</p>\n<p>The larger is D, the larger the possible number of buckets in the array and thus the larger number of one-hot-encoded variables with weights from the training process. If D=100 then there are 100 possible values that can result from from x%D, where x is the integer being divided by D.</p>\n<p>The larger D, the less collisions (meaning different values of features are hashed into the same bucket and thus not distinguishable from each other). Sometimes a larger D is better, but not always, as a smaller D (note, smaller still means really large!) acts as dimensionality reduction. Hope this helps.</p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57472,
      "author_name": "inspector",
      "author_url": "",
      "post_date": "11/07/2014 15:54:25",
      "content": "<p>[quote=cipolla;57471]</p>\n<p>The reply covers almost everything I can imagine :) &nbsp;Thank you!</p>\n<p>So the only purpose of calculating the remainder is to reduce the number of categories of a feature so is the dimensionality?</p>\n<p>[/quote]</p>\n\n<p>Your welcome, glad it helped. I was confused when I first encountered it.</p>\n<p>Yes, I believe that is the reason for D, else I think hash() would end up hashing every value to a distinct integer (I am sure within some limit). I cant claim to know this for an absolute fact, but I *think* that is correct.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57473,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "11/07/2014 15:54:54",
      "content": "<p>[quote=cipolla;57471]</p>\n<p>The reply covers almost everything I can imagine :) &nbsp;Thank you!</p>\n<p>So the only purpose of calculating the remainder is to reduce the number of categories of a feature so is the dimensionality?</p>\n<p>[/quote]</p>\n<p>On my computer, when I type&nbsp;hash('somestring'),&nbsp;I get back 43209540836401356. Since you need to store&nbsp;a weight for each of&nbsp;your potential attributes, you'll need a weight array at least that long. Taking the modulo trims the number of possible weights&nbsp;down to a manageable amount.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57931,
      "author_name": "betterplace",
      "author_url": "",
      "post_date": "11/12/2014 17:26:09",
      "content": "<p>Hello all,</p>\n<p>I seperate the trans_rev2 to 4 small files.</p>\n<p>WHY i get my Logloss higer at each print,19.999807,19.999903....</p>\n<p>do you have the same problem like mine?</p>\n<p>I didn't change other default settings in corrected fast_solution.py</p>\n<p>and i tried smaller learning_rate, don't work.....</p>\n<p>Thanks!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57947,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "11/12/2014 21:40:44",
      "content": "<p>It's because the file is in sequence order.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57949,
      "author_name": "betterplace",
      "author_url": "",
      "post_date": "11/12/2014 22:34:39",
      "content": "<p>[quote=ACS69;57947]</p>\n<p>It's because the file is in sequence order.</p>\n<p>[/quote]</p>\n<p>sorry, i still don't get the point why the log loss get higher. can you explicate a little more?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57950,
      "author_name": "betterplace",
      "author_url": "",
      "post_date": "11/12/2014 22:35:31",
      "content": "<p>[quote=ACS69;57947]</p>\n<p>It's because the file is in sequence order.</p>\n<p>[/quote]</p>\n<p>sorry, i still don't get the point why the log loss get higher. can you explicate a little more?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57951,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "11/12/2014 22:43:04",
      "content": "<p>I'm no expert, but I would think it's because they're in similar groupings (hour) - the way the online code works will pick up the similar features - so for the same hour the log loss gets better, then the hour changes and there's new features so the log loss may get worse, then level out again or improve and so on . If you randomly shuffle the file before running the code, you will get the pattern that you are expecting</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57952,
      "author_name": "betterplace",
      "author_url": "",
      "post_date": "11/12/2014 22:53:55",
      "content": "<p>Thanks for you explication.</p>\n<p>But if it is the case, which means log loss will have a sudden drop in performance while reaching the next hour data, &nbsp;we'll see a drop each 200k data (as the Q&amp;A said each hour has 200k data). As the published print code print the performance each 100k data, so the log loss will not be go up each print.</p>\n<p>i don't get this because in the Criteo CTR competition, the data is also order by time but the log loss go down without data permutation.</p>\n<p>Anyone else has the same&nbsp;result like mine?&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57953,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "11/12/2014 23:06:33",
      "content": "<p>Yes - it will happen to everyone who runs the online code on the sequenced data.&nbsp; Shuffle the data to get consistent log loss improvement. I used &quot;hour&quot; as an example - I haven't looked at the details to see what is causing the drops</p>\n<p>edit: I didn't enter the other competition so I don't know what the data was like</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 313324,
      "author_name": "zhegeliang2",
      "author_url": "",
      "post_date": "04/13/2018 05:42:01",
      "content": "<p>simple and effective method </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "56731": "",
    "56732": "",
    "56733": "",
    "56734": "",
    "56735": "",
    "56736": "",
    "56737": "",
    "56738": "",
    "56744": "",
    "56746": "",
    "56747": "",
    "56749": "",
    "56750": "",
    "56754": "",
    "56755": "",
    "56774": "",
    "56948": "",
    "57019": "",
    "57024": "",
    "57217": "",
    "57219": "",
    "57220": "",
    "57230": "",
    "57235": "",
    "57236": "",
    "57238": "",
    "57239": "",
    "57241": "",
    "57292": "",
    "57293": "",
    "57465": "",
    "57467": "",
    "57468": "",
    "57470": "",
    "57471": "",
    "57472": "",
    "57473": "",
    "57931": "",
    "57947": "",
    "57949": "",
    "57950": "",
    "57951": "",
    "57952": "",
    "57953": "",
    "313324": "simple and effective method"
  },
  "source": "meta"
}