{
  "id": 77894,
  "title": "1st place solution # c1ph3rh4ck team",
  "content": "<p>Hi, </p>\n\n<p>Thank you Phil for organizing this competition, it has been an addictive and intense experience! </p>\n\n<p>We learnt a lot in the process and wanted to share our experience in this post. This post does not cover all technical details and everything we tried, we mostly wanted to tell the story of this competition from our eyes.</p>\n\n<p>And before we start, a massive thank you to our competitors, and special note for EtienneW, flal, kaggleuser58, RS Turley. You really pushed us to do better. THANK YOU.</p>\n\n<h2>Episode 1: The c1ph3r h4ck1ng. From 0 to 0.9847.</h2>\n\n<p>I called my friend and I was like \"hey there is this competition, are you up for it?\". Minutes later, we jumped on a mission to break the ciphers one by one. We kept going until success. For a detailed technical write-up about the ciphers breaking, we would like to recommend the great kernels by our competitors listed above. It took us a few days, we used a mix of automation, stats and manual fixes to get it right and reach the top of the leaderboard. We also painfully reverse-engineered the pre-processing (\"\\n \" vs \"\\n\" pain anyone?).</p>\n\n<p>However, we found some ambiguous rows. We submitted a random solution in the space of possible solutions. 0.9847. <em>Have we missed something?</em></p>\n\n<h2>Episode 2: The space fine-tuning. From 0.9847 to 0.9991.</h2>\n\n<p>We improved our solution:</p>\n\n<p>1) We counted how many times each piece of text appears in each class.  For example instead of mapping the '--' decoded text to the potential classes [1, 5], we mapped it to {1: 1, 5: 4} (keys = classes, values = number of occurrences). Then instead of uniformly sampling on [1, 5], we gave <strong>non-uniform probabilities</strong> (or equivalently uniform ones on [1, 5, 5, 5, 5]). This technique gives a higher probability to the correct class, and tends to increase the score.</p>\n\n<p>2) We grouped rows with identical decoded text. For example, if a decoded text is found twice in the test set, and that text maps to {1: 1, 5: 1} (classes 1 and 5 are both found once). Then we know the solution is</p>\n\n<p><code>row_1, 5</code><br>\n<code>row_2, 1</code><br>\nor <br>\n<code>row_1, 1</code><br>\n<code>row_2, 5</code><br></p>\n\n<p>In other words, using <strong>sampling without replacement</strong> for each group reduces the probability space.</p>\n\n<p>3) We <strong>removed the observations from the training space</strong>. For example, if a string, '---' appears in classes 7 and 8, but is in class 7 in the training set, then we know class 8 is the one we want.</p>\n\n<p>Jointly, this brought us up to 0.9991. Not bad. </p>\n\n<h2>Episode 3: The RNG gods incantations. From 0.9991 to 0.9992.</h2>\n\n<p>It was time to invoke the RNG gods (shoutout to Etienne for this expression ^^) with our tricks from Episode 2. Our incantations gave us solutions scoring in [0.9990, 0.9992]. We were not so happy to rely on luck at this point, but we were up +0.0001, so we'll take that.</p>\n\n<h2>Episode 3: The RNG gods on steroids. From 0.9992 to 0.9994.</h2>\n\n<p>At this point, we had a bunch of random submissions, some scoring better than others. Despited limited precision, we decided to aggregate that information. </p>\n\n<p>1) We used a <strong>simple counting technique</strong>. For example, for a row with 2 possible classes {5, 6}, we gave points to each class. We decided to give points as: 0.9990 -&gt; -1, 0.9991 -&gt; 0, 0.9992 -&gt; +1. Iterating over our random submissions, we got, for example, 3 points for class 5, -1 point for class 6. Then we created a submission with all winning combinations.</p>\n\n<p>2) We used <strong>dimensionality reduction techniques</strong> to somehow visualize the space we were sampling from. We sampled lots of random submissions, both for exploitation (submitting a solution close to our top scores in the 2D space) and exploration (submitting a solution which is far away from the ones already submitted -- hoping that it would provide useful information we would use at some point).</p>\n\n<p>Both techniques were enough to stay at the top of the leaderboard for a while. However EtienneW was catching up, and he finally took the #1 spot. We kept using our techniques for one or two days, then decided it was time for a change.</p>\n\n<h2>Episode 4: The mathematical formulation. From 0.9994 to 0.9994.</h2>\n\n<p>We were not excited by the idea of manually testing ambiguous rows one by one. Instead we preferred to look for an elegant way of finding the solution. So we started to wonder which information we had not exploited yet, and could be leveraged. At this point, we were a bit frustrated by the limited precision of the scores of our past submissions. We checked the HTML code and the Kaggle API, no luck, same limited precision. </p>\n\n<p>Looking for extra information, we found out that, on the Kaggle \"My submissions\" page, it is possible to sort past submissions according to their public scores. We hypothesized that, despite an apparent identical score, the submissions were sorted according to the full precision score. <strong>This gives much more information than the scores only.</strong> </p>\n\n<p><code>some submission, 0.9994  # rank = 1</code> <br>\n<code>another submission, 0.9994  # rank = 2</code><br>\n<code>...</code><br>\n<code>yet another submission, 0.9993  # rank = n</code><br>\n<code>...</code> </p>\n\n<p>The question was: how to leverage that information? Our first idea was to refine our point system from Episode 3. </p>\n\n<p>And then it hit us. <strong>We realized it was possible to evaluate candidate solutions without submitting them.</strong> Let me explain. If you have the correct solution, then you should be able to use it to score and rank your past submissions.  Let us say you have a candidate solution. Then if you compute the score of a past submission, assuming the candidate solution to be perfect, you should get the score visible on \"My submissions\", for any submission. In addition, given a candidate solution, if you compute all past submissions and sort them according to these scores, you should get the same ranking as the one visible on the \"My submissions\" page, when sorting by \"public score\". </p>\n\n<p>Of course, this is a <strong>necessary but not sufficient condition</strong>. Nevertheless, we could evaluate tons of solutions locally, and hopefully have an strong enough edge to win. Unfortunately, the space to bruteforce was just gigantic. 10^{39}. Even with a good implementation, we don't have that kind of computing power. So we decided to reduce the space size with manual submissions.</p>\n\n<h2>Episode 5: The manual improvements. From 0.9994 to 0.9995.</h2>\n\n<p>As we could not solve the nice optimization problems yet, we had to reduce the number of possible combinations. With the \"sort-by-public-score\" technique, we could <strong>evaluate the impact of a single change</strong>. For example, we take one row where there are two possible solutions, say classes 7 and 8. We take our best submission, change the value on a single row, and submit this new solution. If this new submission gets a better rank, then the new value is the correct one, otherwise the old value is the correct one. In any case, </p>\n\n<ol>\n<li>we get the information of the correct class for that row and,  </li>\n<li>we reduce the space size by a factor 2.</li>\n</ol>\n\n<p>Not super exciting phase, we got +0.0001 and we were still #2. The space was still too large to fully bruteforce.</p>\n\n<h2>Episode 6: The leaderboard shake-up. From 0.9995 to 0.9998.</h2>\n\n<p>At this moment, the leaderboard was taken by storm. Flal and kaggleuser58 scored 0.9997 (not sure who got there first). We were #4. Flal had only 4 submissions, so we knew he did something smart we missed. </p>\n\n<p>Again, the question was: which information could be exploited? We realized the <code>difficulty</code> field carried extra hidden information. <strong>All chunks for the same original document were encoded with the same <code>difficulty</code>.</strong> This reduces the number of ambiguous rows. For example, say some encrypted text is decoded as '--', which maps to classes 7 and 8. Now let's say we also know that </p>\n\n<ol>\n<li>'--' was encrypted in difficulty 3, </li>\n<li>one document with the chunk '--' was encrypted in difficulty 3 and is in class 7, </li>\n<li>another document with the chunk '--' was encrypted in difficulty 2 and is in class 8, </li>\n</ol>\n\n<p>then, we know that the correct class is 7.</p>\n\n<p>This is not enough to solve everything, but still it was a massive solution space reduction! We went from 128 ambiguous rows to 37 ambiguous rows. Unfortunately, the space was still too large (~ 10^{15}), but we were getting there. With our 5 daily submissions at hand, we solved 4 ambiguous rows manually (further reducing the space by 2^4). Finally we shot 6M random solutions in this space, and took the solution with the lowest scoring error, as explained in Episode 4. We were back to #1 with 0.9998.</p>\n\n<h2>Episode 7: The final strike. From 0.9998 to 1.</h2>\n\n<p>From here on, we had a plan to finish this: <strong>manual space reduction followed by bruteforcing</strong> (our secret weapon). </p>\n\n<p>The space was still huge, so we started to look into code efficiency and speed-up tricks. Several orders of magnitude later, we were a few days away to fully bruteforce the space with our python implementation. Nevertheless, the competition was tough, so we re-implemented our strategy in C++. We gained further orders of magnitude. It took around 20-30 min to evaluate 5x10^{14} (50k billion) solutions on a 24-CPU VM.</p>\n\n<p>We got 1280 solutions that were able to perfectly score all our past submissions. Not bad. Then we evaluated these guys in terms of ranking and found... no solution. We strongly believed in our approach, so with high probability there was a bug somewhere in our pipeline. The bug hunt was unsuccessful for a while, so in the meantime we submitted solutions with single row changes. This brought us to 0.9999.</p>\n\n<p>After about two days of checking assumptions, reviewing code, checking everything, and lots of sweat, we finally fixed our pipeline, and found the perfect solution. Submitted that. Kaggle called Watson. Got that final +0.0001 ^^ </p>\n\n<p><strong>JUNG N PBZCRGVGVBA! GUNAX LBH NTNVA!</strong> You know what we mean, right? ;)</p>\n\n<p>Team c1ph3rh4ck.</p>",
  "comments": [
    {
      "id": 457448,
      "author_name": "EtienneW",
      "author_url": "",
      "post_date": "2019-01-17T12:55:47.400000",
      "content": "<p>Great work guys, it was really fun competing with you!</p>\n\n<p>Just to give my own intuition on the final step (from 0.9998 to 1) : \nI got stuck like you after finding out we could use difficulties to reduce the possibilities to something like 20 unpredictable ciphers, which made me try to look for a pattern in the mapping possibilities -&gt; ground truth in the train set (since we had the actual targets for these problematic combinations in train). What I discovered was that for a particular cipher, if you had several possibilities, the actual target was always in the original 20 newsgroup dataset test part if there was such possibility, else it was in the original 20 newsgroups train set. That made the number of possibilities go down approxiamtely by half. To illustrate this :</p>\n\n<p>Possibilities : before -&gt; after\ncipher 45a4zr56az : [train_set1, train_set10, test_set2, test_set50] -&gt; [test_set2, test_set50]\ncipher 564z6gtzeg : [train_set45, train_set877, test_set23] -&gt; [test_set23]\ncipher zezfegz545 : [train_set48, train_set 456] -&gt; [train_set48, train_set 456]</p>\n\n<p>This made me realize that Phil probably loaded the train set of the original dataset, then the test set, then sliced it up, ciphered it, and kept for every duplicate only the last (ie pd.drop_duplicated(df, keep='last'), or first loaded the test set then the train set and kept the first of each duplicate.</p>\n\n<p>If my intuition was true, that meant that since in the original dataset the messages are organized by targets, I only had to find the order in which there were loaded (this seems to be random when you import the .zip) and keep either the last or the first of them.</p>\n\n<p>So if the categories were loaded like this : [2, 15, 14,...,10, 8, 19, 18], I could look up for each multi-possibilities ciphers the target that came up first or last in this list and keep it</p>\n\n<p>Possibilities (keep last) :\n[4, 19, 15, 3] -&gt; 19\n[14, 18] -&gt; 18\netc...</p>\n\n<p>The way the possibilities matched to targets in the competition train set seemed to confirm my intuition. For instance, each time you had a set of possibilities with a target 8 in it, you could be sure that the actual target was 8 (after trimming it down to only test_set or only train_set), as if the 8th category was one of the last one loaded in a 'keep last' scenario.</p>\n\n<p>Unfortunately I didn't have enough submissions remaining to probe the leaderboard to get that the actual order of the categories, so I stalled at 0.9999. I made as much guesses as I could before I ran out of time.\nThat being, maybe I just imagined a pattern out of a few samples and this was completely wrong lol.</p>\n\n<p>Anyway great competition, thanks again Phil for that, and great job c1ph3rh4ck, we better watch out for you in the next ciphertext competition ;)</p>",
      "votes": 4,
      "replies": [
        {
          "id": 457601,
          "author_name": "",
          "author_url": "",
          "post_date": "2019-01-17T19:28:54.237000",
          "content": "",
          "votes": 0,
          "replies": []
        },
        {
          "id": 457905,
          "author_name": "JBF",
          "author_url": "",
          "post_date": "2019-01-18T10:10:41.750000",
          "content": "<p>Thanks for the comment and for sharing your end-game strategy :) I like the reasoning, you found a cool pattern, great work!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 459744,
          "author_name": "Flal",
          "author_url": "",
          "post_date": "2019-01-22T09:29:02.860000",
          "content": "<p>\"if you had several possibilities, the actual target was always in the original 20 newsgroup dataset test part if there was such possibility, else it was in the original 20 newsgroups train set\" - excellent, thanks!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 597457,
          "author_name": "Tejashwi Kalp Taru",
          "author_url": "",
          "post_date": "2019-08-12T11:19:20.810000",
          "content": "<p>Thanks for the explanation, really informative </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 457549,
      "author_name": "RS Turley",
      "author_url": "",
      "post_date": "2019-01-17T17:05:53.723000",
      "content": "<p>Great work, and thanks so much for writing up your experience. Some really creative ideas in nailing the ending!</p>\n\n<p>It truly was fun to see everyone's progress on the leaderboard and feel the competitive pressure as someone from the early group of high scorers would make a breakthrough. Congrats to everyone and I hope we keep crossing paths in the future.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 457911,
          "author_name": "JBF",
          "author_url": "",
          "post_date": "2019-01-18T10:22:33.717000",
          "content": "<p>Thanks! I also liked the techniques you shared, especially the fully-automated aspect. </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 457447,
      "author_name": "Robin Hood",
      "author_url": "",
      "post_date": "2019-01-17T12:55:45.517000",
      "content": "<p>I had not noticed it was possible to sort the public scoreboard on the scores so my way to find the last 37 was like this.\nWith a few submissions I found a sweetspot where one more correct would raise the score from 0.9996 to 0.9997 and thereby I could fix 1 row pr submission whenever there was only two possibilities. If the score goes up the new target had to be right otherwise I assumed the other previous target was right. Going on like this I knew I would get to about 0.99995 and thought it could be enough to win.\nAt the end I thought I had all 37 tough one fixed - but somewhere before the 37 I think I have made 4 mistakes and still wonder where those mistakes are.</p>\n\n<p>But you found an other way to beat the entire field.\nCONGRATULATION\nKaggleuser58</p>",
      "votes": 2,
      "replies": [
        {
          "id": 457908,
          "author_name": "JBF",
          "author_url": "",
          "post_date": "2019-01-18T10:15:20.163000",
          "content": "<p>Thanks :) Nice job finding this sweet spot to go around the limited score precision!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 459738,
      "author_name": "Flal",
      "author_url": "",
      "post_date": "2019-01-22T09:14:40.147000",
      "content": "<p>Hi JBF, thank you very much for sharing your path to the perfect 1.00. Very instructive. Congratulations to your teammate and you. Really looking forward to the next one.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 460714,
          "author_name": "JBF",
          "author_url": "",
          "post_date": "2019-01-24T09:47:23.003000",
          "content": "<p>Thanks Flal :) Reaching 0.9997 in only 4 submissions was super elegant and impressive!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 458457,
      "author_name": "Sarfaraz Ahmed Abbasi",
      "author_url": "",
      "post_date": "2019-01-19T17:31:27.610000",
      "content": "<p>Congratulations!\nThank you for sharing your experience. It's very impressive.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 460713,
          "author_name": "JBF",
          "author_url": "",
          "post_date": "2019-01-24T09:39:15.787000",
          "content": "<p>Thanks!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 458022,
      "author_name": "Eric Vos",
      "author_url": "",
      "post_date": "2019-01-18T15:33:33.257000",
      "content": "<p>Congratulations,\nThanks for sharing.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 458319,
          "author_name": "JBF",
          "author_url": "",
          "post_date": "2019-01-19T11:12:59.633000",
          "content": "<p>Thanks :)</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 457667,
      "author_name": "Aian Shay",
      "author_url": "",
      "post_date": "2019-01-17T22:38:08.997000",
      "content": "<p>dope solution, keep going man!</p>",
      "votes": 0,
      "replies": [
        {
          "id": 457913,
          "author_name": "JBF",
          "author_url": "",
          "post_date": "2019-01-18T10:23:13.193000",
          "content": "<p>Thanks!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 17,
  "messages": [
    {
      "id": 457414,
      "postDate": "2019-01-17T11:15:22.043Z",
      "content": "<p>Hi, </p>\n\n<p>Thank you Phil for organizing this competition, it has been an addictive and intense experience! </p>\n\n<p>We learnt a lot in the process and wanted to share our experience in this post. This post does not cover all technical details and everything we tried, we mostly wanted to tell the story of this competition from our eyes.</p>\n\n<p>And before we start, a massive thank you to our competitors, and special note for EtienneW, flal, kaggleuser58, RS Turley. You really pushed us to do better. THANK YOU.</p>\n\n<h2>Episode 1: The c1ph3r h4ck1ng. From 0 to 0.9847.</h2>\n\n<p>I called my friend and I was like \"hey there is this competition, are you up for it?\". Minutes later, we jumped on a mission to break the ciphers one by one. We kept going until success. For a detailed technical write-up about the ciphers breaking, we would like to recommend the great kernels by our competitors listed above. It took us a few days, we used a mix of automation, stats and manual fixes to get it right and reach the top of the leaderboard. We also painfully reverse-engineered the pre-processing (\"\\n \" vs \"\\n\" pain anyone?).</p>\n\n<p>However, we found some ambiguous rows. We submitted a random solution in the space of possible solutions. 0.9847. <em>Have we missed something?</em></p>\n\n<h2>Episode 2: The space fine-tuning. From 0.9847 to 0.9991.</h2>\n\n<p>We improved our solution:</p>\n\n<p>1) We counted how many times each piece of text appears in each class.  For example instead of mapping the '--' decoded text to the potential classes [1, 5], we mapped it to {1: 1, 5: 4} (keys = classes, values = number of occurrences). Then instead of uniformly sampling on [1, 5], we gave <strong>non-uniform probabilities</strong> (or equivalently uniform ones on [1, 5, 5, 5, 5]). This technique gives a higher probability to the correct class, and tends to increase the score.</p>\n\n<p>2) We grouped rows with identical decoded text. For example, if a decoded text is found twice in the test set, and that text maps to {1: 1, 5: 1} (classes 1 and 5 are both found once). Then we know the solution is</p>\n\n<p><code>row_1, 5</code><br>\n<code>row_2, 1</code><br>\nor <br>\n<code>row_1, 1</code><br>\n<code>row_2, 5</code><br></p>\n\n<p>In other words, using <strong>sampling without replacement</strong> for each group reduces the probability space.</p>\n\n<p>3) We <strong>removed the observations from the training space</strong>. For example, if a string, '---' appears in classes 7 and 8, but is in class 7 in the training set, then we know class 8 is the one we want.</p>\n\n<p>Jointly, this brought us up to 0.9991. Not bad. </p>\n\n<h2>Episode 3: The RNG gods incantations. From 0.9991 to 0.9992.</h2>\n\n<p>It was time to invoke the RNG gods (shoutout to Etienne for this expression ^^) with our tricks from Episode 2. Our incantations gave us solutions scoring in [0.9990, 0.9992]. We were not so happy to rely on luck at this point, but we were up +0.0001, so we'll take that.</p>\n\n<h2>Episode 3: The RNG gods on steroids. From 0.9992 to 0.9994.</h2>\n\n<p>At this point, we had a bunch of random submissions, some scoring better than others. Despited limited precision, we decided to aggregate that information. </p>\n\n<p>1) We used a <strong>simple counting technique</strong>. For example, for a row with 2 possible classes {5, 6}, we gave points to each class. We decided to give points as: 0.9990 -&gt; -1, 0.9991 -&gt; 0, 0.9992 -&gt; +1. Iterating over our random submissions, we got, for example, 3 points for class 5, -1 point for class 6. Then we created a submission with all winning combinations.</p>\n\n<p>2) We used <strong>dimensionality reduction techniques</strong> to somehow visualize the space we were sampling from. We sampled lots of random submissions, both for exploitation (submitting a solution close to our top scores in the 2D space) and exploration (submitting a solution which is far away from the ones already submitted -- hoping that it would provide useful information we would use at some point).</p>\n\n<p>Both techniques were enough to stay at the top of the leaderboard for a while. However EtienneW was catching up, and he finally took the #1 spot. We kept using our techniques for one or two days, then decided it was time for a change.</p>\n\n<h2>Episode 4: The mathematical formulation. From 0.9994 to 0.9994.</h2>\n\n<p>We were not excited by the idea of manually testing ambiguous rows one by one. Instead we preferred to look for an elegant way of finding the solution. So we started to wonder which information we had not exploited yet, and could be leveraged. At this point, we were a bit frustrated by the limited precision of the scores of our past submissions. We checked the HTML code and the Kaggle API, no luck, same limited precision. </p>\n\n<p>Looking for extra information, we found out that, on the Kaggle \"My submissions\" page, it is possible to sort past submissions according to their public scores. We hypothesized that, despite an apparent identical score, the submissions were sorted according to the full precision score. <strong>This gives much more information than the scores only.</strong> </p>\n\n<p><code>some submission, 0.9994  # rank = 1</code> <br>\n<code>another submission, 0.9994  # rank = 2</code><br>\n<code>...</code><br>\n<code>yet another submission, 0.9993  # rank = n</code><br>\n<code>...</code> </p>\n\n<p>The question was: how to leverage that information? Our first idea was to refine our point system from Episode 3. </p>\n\n<p>And then it hit us. <strong>We realized it was possible to evaluate candidate solutions without submitting them.</strong> Let me explain. If you have the correct solution, then you should be able to use it to score and rank your past submissions.  Let us say you have a candidate solution. Then if you compute the score of a past submission, assuming the candidate solution to be perfect, you should get the score visible on \"My submissions\", for any submission. In addition, given a candidate solution, if you compute all past submissions and sort them according to these scores, you should get the same ranking as the one visible on the \"My submissions\" page, when sorting by \"public score\". </p>\n\n<p>Of course, this is a <strong>necessary but not sufficient condition</strong>. Nevertheless, we could evaluate tons of solutions locally, and hopefully have an strong enough edge to win. Unfortunately, the space to bruteforce was just gigantic. 10^{39}. Even with a good implementation, we don't have that kind of computing power. So we decided to reduce the space size with manual submissions.</p>\n\n<h2>Episode 5: The manual improvements. From 0.9994 to 0.9995.</h2>\n\n<p>As we could not solve the nice optimization problems yet, we had to reduce the number of possible combinations. With the \"sort-by-public-score\" technique, we could <strong>evaluate the impact of a single change</strong>. For example, we take one row where there are two possible solutions, say classes 7 and 8. We take our best submission, change the value on a single row, and submit this new solution. If this new submission gets a better rank, then the new value is the correct one, otherwise the old value is the correct one. In any case, </p>\n\n<ol>\n<li>we get the information of the correct class for that row and,  </li>\n<li>we reduce the space size by a factor 2.</li>\n</ol>\n\n<p>Not super exciting phase, we got +0.0001 and we were still #2. The space was still too large to fully bruteforce.</p>\n\n<h2>Episode 6: The leaderboard shake-up. From 0.9995 to 0.9998.</h2>\n\n<p>At this moment, the leaderboard was taken by storm. Flal and kaggleuser58 scored 0.9997 (not sure who got there first). We were #4. Flal had only 4 submissions, so we knew he did something smart we missed. </p>\n\n<p>Again, the question was: which information could be exploited? We realized the <code>difficulty</code> field carried extra hidden information. <strong>All chunks for the same original document were encoded with the same <code>difficulty</code>.</strong> This reduces the number of ambiguous rows. For example, say some encrypted text is decoded as '--', which maps to classes 7 and 8. Now let's say we also know that </p>\n\n<ol>\n<li>'--' was encrypted in difficulty 3, </li>\n<li>one document with the chunk '--' was encrypted in difficulty 3 and is in class 7, </li>\n<li>another document with the chunk '--' was encrypted in difficulty 2 and is in class 8, </li>\n</ol>\n\n<p>then, we know that the correct class is 7.</p>\n\n<p>This is not enough to solve everything, but still it was a massive solution space reduction! We went from 128 ambiguous rows to 37 ambiguous rows. Unfortunately, the space was still too large (~ 10^{15}), but we were getting there. With our 5 daily submissions at hand, we solved 4 ambiguous rows manually (further reducing the space by 2^4). Finally we shot 6M random solutions in this space, and took the solution with the lowest scoring error, as explained in Episode 4. We were back to #1 with 0.9998.</p>\n\n<h2>Episode 7: The final strike. From 0.9998 to 1.</h2>\n\n<p>From here on, we had a plan to finish this: <strong>manual space reduction followed by bruteforcing</strong> (our secret weapon). </p>\n\n<p>The space was still huge, so we started to look into code efficiency and speed-up tricks. Several orders of magnitude later, we were a few days away to fully bruteforce the space with our python implementation. Nevertheless, the competition was tough, so we re-implemented our strategy in C++. We gained further orders of magnitude. It took around 20-30 min to evaluate 5x10^{14} (50k billion) solutions on a 24-CPU VM.</p>\n\n<p>We got 1280 solutions that were able to perfectly score all our past submissions. Not bad. Then we evaluated these guys in terms of ranking and found... no solution. We strongly believed in our approach, so with high probability there was a bug somewhere in our pipeline. The bug hunt was unsuccessful for a while, so in the meantime we submitted solutions with single row changes. This brought us to 0.9999.</p>\n\n<p>After about two days of checking assumptions, reviewing code, checking everything, and lots of sweat, we finally fixed our pipeline, and found the perfect solution. Submitted that. Kaggle called Watson. Got that final +0.0001 ^^ </p>\n\n<p><strong>JUNG N PBZCRGVGVBA! GUNAX LBH NTNVA!</strong> You know what we mean, right? ;)</p>\n\n<p>Team c1ph3rh4ck.</p>",
      "rawMarkdown": "Hi, \n\nThank you Phil for organizing this competition, it has been an addictive and intense experience! \n\nWe learnt a lot in the process and wanted to share our experience in this post. This post does not cover all technical details and everything we tried, we mostly wanted to tell the story of this competition from our eyes.\n\nAnd before we start, a massive thank you to our competitors, and special note for EtienneW, flal, kaggleuser58, RS Turley. You really pushed us to do better. THANK YOU.\n\n## Episode 1: The c1ph3r h4ck1ng. From 0 to 0.9847.\n\nI called my friend and I was like \"hey there is this competition, are you up for it?\". Minutes later, we jumped on a mission to break the ciphers one by one. We kept going until success. For a detailed technical write-up about the ciphers breaking, we would like to recommend the great kernels by our competitors listed above. It took us a few days, we used a mix of automation, stats and manual fixes to get it right and reach the top of the leaderboard. We also painfully reverse-engineered the pre-processing (\"\\n \" vs \"\\n\" pain anyone?).\n\nHowever, we found some ambiguous rows. We submitted a random solution in the space of possible solutions. 0.9847. *Have we missed something?*\n\n## Episode 2: The space fine-tuning. From 0.9847 to 0.9991.\n\nWe improved our solution:\n\n1) We counted how many times each piece of text appears in each class.  For example instead of mapping the '--' decoded text to the potential classes [1, 5], we mapped it to {1: 1, 5: 4} (keys = classes, values = number of occurrences). Then instead of uniformly sampling on [1, 5], we gave **non-uniform probabilities** (or equivalently uniform ones on [1, 5, 5, 5, 5]). This technique gives a higher probability to the correct class, and tends to increase the score.\n\n2) We grouped rows with identical decoded text. For example, if a decoded text is found twice in the test set, and that text maps to {1: 1, 5: 1} (classes 1 and 5 are both found once). Then we know the solution is\n\n`row_1, 5`<br>\n`row_2, 1`<br>\nor <br>\n`row_1, 1`<br>\n`row_2, 5`<br>\n\nIn other words, using **sampling without replacement** for each group reduces the probability space.\n\n3) We **removed the observations from the training space**. For example, if a string, '---' appears in classes 7 and 8, but is in class 7 in the training set, then we know class 8 is the one we want.\n\nJointly, this brought us up to 0.9991. Not bad. \n\n## Episode 3: The RNG gods incantations. From 0.9991 to 0.9992.\n\nIt was time to invoke the RNG gods (shoutout to Etienne for this expression ^^) with our tricks from Episode 2. Our incantations gave us solutions scoring in [0.9990, 0.9992]. We were not so happy to rely on luck at this point, but we were up +0.0001, so we'll take that.\n\n## Episode 3: The RNG gods on steroids. From 0.9992 to 0.9994.\n\nAt this point, we had a bunch of random submissions, some scoring better than others. Despited limited precision, we decided to aggregate that information. \n\n1) We used a **simple counting technique**. For example, for a row with 2 possible classes {5, 6}, we gave points to each class. We decided to give points as: 0.9990 -&gt; -1, 0.9991 -&gt; 0, 0.9992 -&gt; +1. Iterating over our random submissions, we got, for example, 3 points for class 5, -1 point for class 6. Then we created a submission with all winning combinations.\n\n2) We used **dimensionality reduction techniques** to somehow visualize the space we were sampling from. We sampled lots of random submissions, both for exploitation (submitting a solution close to our top scores in the 2D space) and exploration (submitting a solution which is far away from the ones already submitted -- hoping that it would provide useful information we would use at some point).\n\nBoth techniques were enough to stay at the top of the leaderboard for a while. However EtienneW was catching up, and he finally took the #1 spot. We kept using our techniques for one or two days, then decided it was time for a change.\n\n## Episode 4: The mathematical formulation. From 0.9994 to 0.9994.\n\nWe were not excited by the idea of manually testing ambiguous rows one by one. Instead we preferred to look for an elegant way of finding the solution. So we started to wonder which information we had not exploited yet, and could be leveraged. At this point, we were a bit frustrated by the limited precision of the scores of our past submissions. We checked the HTML code and the Kaggle API, no luck, same limited precision. \n\nLooking for extra information, we found out that, on the Kaggle \"My submissions\" page, it is possible to sort past submissions according to their public scores. We hypothesized that, despite an apparent identical score, the submissions were sorted according to the full precision score. **This gives much more information than the scores only.** \n\n`some submission, 0.9994  # rank = 1` <br>\n`another submission, 0.9994  # rank = 2`<br>\n`...`<br>\n`yet another submission, 0.9993  # rank = n`<br>\n`...` \n\nThe question was: how to leverage that information? Our first idea was to refine our point system from Episode 3. \n\nAnd then it hit us. **We realized it was possible to evaluate candidate solutions without submitting them.** Let me explain. If you have the correct solution, then you should be able to use it to score and rank your past submissions.  Let us say you have a candidate solution. Then if you compute the score of a past submission, assuming the candidate solution to be perfect, you should get the score visible on \"My submissions\", for any submission. In addition, given a candidate solution, if you compute all past submissions and sort them according to these scores, you should get the same ranking as the one visible on the \"My submissions\" page, when sorting by \"public score\". \n\nOf course, this is a **necessary but not sufficient condition**. Nevertheless, we could evaluate tons of solutions locally, and hopefully have an strong enough edge to win. Unfortunately, the space to bruteforce was just gigantic. 10^{39}. Even with a good implementation, we don't have that kind of computing power. So we decided to reduce the space size with manual submissions.\n\n## Episode 5: The manual improvements. From 0.9994 to 0.9995.\n\nAs we could not solve the nice optimization problems yet, we had to reduce the number of possible combinations. With the \"sort-by-public-score\" technique, we could **evaluate the impact of a single change**. For example, we take one row where there are two possible solutions, say classes 7 and 8. We take our best submission, change the value on a single row, and submit this new solution. If this new submission gets a better rank, then the new value is the correct one, otherwise the old value is the correct one. In any case, \n\n 1. we get the information of the correct class for that row and,  \n 2. we reduce the space size by a factor 2.\n\nNot super exciting phase, we got +0.0001 and we were still #2. The space was still too large to fully bruteforce.\n\n## Episode 6: The leaderboard shake-up. From 0.9995 to 0.9998.\n\nAt this moment, the leaderboard was taken by storm. Flal and kaggleuser58 scored 0.9997 (not sure who got there first). We were #4. Flal had only 4 submissions, so we knew he did something smart we missed. \n\nAgain, the question was: which information could be exploited? We realized the `difficulty` field carried extra hidden information. **All chunks for the same original document were encoded with the same `difficulty`.** This reduces the number of ambiguous rows. For example, say some encrypted text is decoded as '--', which maps to classes 7 and 8. Now let's say we also know that \n\n 1. '--' was encrypted in difficulty 3, \n 2. one document with the chunk '--' was encrypted in difficulty 3 and is in class 7, \n 3. another document with the chunk '--' was encrypted in difficulty 2 and is in class 8, \n\nthen, we know that the correct class is 7.\n\nThis is not enough to solve everything, but still it was a massive solution space reduction! We went from 128 ambiguous rows to 37 ambiguous rows. Unfortunately, the space was still too large (~ 10^{15}), but we were getting there. With our 5 daily submissions at hand, we solved 4 ambiguous rows manually (further reducing the space by 2^4). Finally we shot 6M random solutions in this space, and took the solution with the lowest scoring error, as explained in Episode 4. We were back to #1 with 0.9998.\n\n## Episode 7: The final strike. From 0.9998 to 1.\n\nFrom here on, we had a plan to finish this: **manual space reduction followed by bruteforcing** (our secret weapon). \n\nThe space was still huge, so we started to look into code efficiency and speed-up tricks. Several orders of magnitude later, we were a few days away to fully bruteforce the space with our python implementation. Nevertheless, the competition was tough, so we re-implemented our strategy in C++. We gained further orders of magnitude. It took around 20-30 min to evaluate 5x10^{14} (50k billion) solutions on a 24-CPU VM.\n\nWe got 1280 solutions that were able to perfectly score all our past submissions. Not bad. Then we evaluated these guys in terms of ranking and found... no solution. We strongly believed in our approach, so with high probability there was a bug somewhere in our pipeline. The bug hunt was unsuccessful for a while, so in the meantime we submitted solutions with single row changes. This brought us to 0.9999.\n\nAfter about two days of checking assumptions, reviewing code, checking everything, and lots of sweat, we finally fixed our pipeline, and found the perfect solution. Submitted that. Kaggle called Watson. Got that final +0.0001 ^^ \n\n**JUNG N PBZCRGVGVBA! GUNAX LBH NTNVA!** You know what we mean, right? ;)\n\n\nTeam c1ph3rh4ck.\n",
      "votes": 22
    },
    {
      "id": 457448,
      "postDate": "2019-01-17T12:55:47.400Z",
      "content": "<p>Great work guys, it was really fun competing with you!</p>\n\n<p>Just to give my own intuition on the final step (from 0.9998 to 1) : \nI got stuck like you after finding out we could use difficulties to reduce the possibilities to something like 20 unpredictable ciphers, which made me try to look for a pattern in the mapping possibilities -&gt; ground truth in the train set (since we had the actual targets for these problematic combinations in train). What I discovered was that for a particular cipher, if you had several possibilities, the actual target was always in the original 20 newsgroup dataset test part if there was such possibility, else it was in the original 20 newsgroups train set. That made the number of possibilities go down approxiamtely by half. To illustrate this :</p>\n\n<p>Possibilities : before -&gt; after\ncipher 45a4zr56az : [train_set1, train_set10, test_set2, test_set50] -&gt; [test_set2, test_set50]\ncipher 564z6gtzeg : [train_set45, train_set877, test_set23] -&gt; [test_set23]\ncipher zezfegz545 : [train_set48, train_set 456] -&gt; [train_set48, train_set 456]</p>\n\n<p>This made me realize that Phil probably loaded the train set of the original dataset, then the test set, then sliced it up, ciphered it, and kept for every duplicate only the last (ie pd.drop_duplicated(df, keep='last'), or first loaded the test set then the train set and kept the first of each duplicate.</p>\n\n<p>If my intuition was true, that meant that since in the original dataset the messages are organized by targets, I only had to find the order in which there were loaded (this seems to be random when you import the .zip) and keep either the last or the first of them.</p>\n\n<p>So if the categories were loaded like this : [2, 15, 14,...,10, 8, 19, 18], I could look up for each multi-possibilities ciphers the target that came up first or last in this list and keep it</p>\n\n<p>Possibilities (keep last) :\n[4, 19, 15, 3] -&gt; 19\n[14, 18] -&gt; 18\netc...</p>\n\n<p>The way the possibilities matched to targets in the competition train set seemed to confirm my intuition. For instance, each time you had a set of possibilities with a target 8 in it, you could be sure that the actual target was 8 (after trimming it down to only test_set or only train_set), as if the 8th category was one of the last one loaded in a 'keep last' scenario.</p>\n\n<p>Unfortunately I didn't have enough submissions remaining to probe the leaderboard to get that the actual order of the categories, so I stalled at 0.9999. I made as much guesses as I could before I ran out of time.\nThat being, maybe I just imagined a pattern out of a few samples and this was completely wrong lol.</p>\n\n<p>Anyway great competition, thanks again Phil for that, and great job c1ph3rh4ck, we better watch out for you in the next ciphertext competition ;)</p>",
      "rawMarkdown": "Great work guys, it was really fun competing with you!\n\nJust to give my own intuition on the final step (from 0.9998 to 1) : \nI got stuck like you after finding out we could use difficulties to reduce the possibilities to something like 20 unpredictable ciphers, which made me try to look for a pattern in the mapping possibilities -&gt; ground truth in the train set (since we had the actual targets for these problematic combinations in train). What I discovered was that for a particular cipher, if you had several possibilities, the actual target was always in the original 20 newsgroup dataset test part if there was such possibility, else it was in the original 20 newsgroups train set. That made the number of possibilities go down approxiamtely by half. To illustrate this :\n\nPossibilities : before -&gt; after\ncipher 45a4zr56az : [train_set1, train_set10, test_set2, test_set50] -&gt; [test_set2, test_set50]\ncipher 564z6gtzeg : [train_set45, train_set877, test_set23] -&gt; [test_set23]\ncipher zezfegz545 : [train_set48, train_set 456] -&gt; [train_set48, train_set 456]\n\nThis made me realize that Phil probably loaded the train set of the original dataset, then the test set, then sliced it up, ciphered it, and kept for every duplicate only the last (ie pd.drop_duplicated(df, keep='last'), or first loaded the test set then the train set and kept the first of each duplicate.\n\nIf my intuition was true, that meant that since in the original dataset the messages are organized by targets, I only had to find the order in which there were loaded (this seems to be random when you import the .zip) and keep either the last or the first of them.\n\nSo if the categories were loaded like this : [2, 15, 14,...,10, 8, 19, 18], I could look up for each multi-possibilities ciphers the target that came up first or last in this list and keep it\n\nPossibilities (keep last) :\n[4, 19, 15, 3] -&gt; 19\n[14, 18] -&gt; 18\netc...\n\nThe way the possibilities matched to targets in the competition train set seemed to confirm my intuition. For instance, each time you had a set of possibilities with a target 8 in it, you could be sure that the actual target was 8 (after trimming it down to only test_set or only train_set), as if the 8th category was one of the last one loaded in a 'keep last' scenario.\n\nUnfortunately I didn't have enough submissions remaining to probe the leaderboard to get that the actual order of the categories, so I stalled at 0.9999. I made as much guesses as I could before I ran out of time.\nThat being, maybe I just imagined a pattern out of a few samples and this was completely wrong lol.\n\nAnyway great competition, thanks again Phil for that, and great job c1ph3rh4ck, we better watch out for you in the next ciphertext competition ;)",
      "votes": 4,
      "replies": [
        {
          "id": 457601,
          "postDate": "2019-01-17T19:28:54.237Z",
          "rawMarkdown": "",
          "isDeleted": true
        },
        {
          "id": 457905,
          "postDate": "2019-01-18T10:10:41.750Z",
          "content": "<p>Thanks for the comment and for sharing your end-game strategy :) I like the reasoning, you found a cool pattern, great work!</p>",
          "rawMarkdown": "Thanks for the comment and for sharing your end-game strategy :) I like the reasoning, you found a cool pattern, great work!"
        },
        {
          "id": 459744,
          "postDate": "2019-01-22T09:29:02.860Z",
          "content": "<p>\"if you had several possibilities, the actual target was always in the original 20 newsgroup dataset test part if there was such possibility, else it was in the original 20 newsgroups train set\" - excellent, thanks!</p>",
          "rawMarkdown": "\"if you had several possibilities, the actual target was always in the original 20 newsgroup dataset test part if there was such possibility, else it was in the original 20 newsgroups train set\" - excellent, thanks!"
        },
        {
          "id": 597457,
          "postDate": "2019-08-12T11:19:20.810Z",
          "content": "<p>Thanks for the explanation, really informative </p>",
          "rawMarkdown": "Thanks for the explanation, really informative "
        }
      ]
    },
    {
      "id": 457549,
      "postDate": "2019-01-17T17:05:53.723Z",
      "content": "<p>Great work, and thanks so much for writing up your experience. Some really creative ideas in nailing the ending!</p>\n\n<p>It truly was fun to see everyone's progress on the leaderboard and feel the competitive pressure as someone from the early group of high scorers would make a breakthrough. Congrats to everyone and I hope we keep crossing paths in the future.</p>",
      "rawMarkdown": "Great work, and thanks so much for writing up your experience. Some really creative ideas in nailing the ending!\n\nIt truly was fun to see everyone's progress on the leaderboard and feel the competitive pressure as someone from the early group of high scorers would make a breakthrough. Congrats to everyone and I hope we keep crossing paths in the future.",
      "votes": 2,
      "replies": [
        {
          "id": 457911,
          "postDate": "2019-01-18T10:22:33.717Z",
          "content": "<p>Thanks! I also liked the techniques you shared, especially the fully-automated aspect. </p>",
          "rawMarkdown": "Thanks! I also liked the techniques you shared, especially the fully-automated aspect. "
        }
      ]
    },
    {
      "id": 457447,
      "postDate": "2019-01-17T12:55:45.517Z",
      "content": "<p>I had not noticed it was possible to sort the public scoreboard on the scores so my way to find the last 37 was like this.\nWith a few submissions I found a sweetspot where one more correct would raise the score from 0.9996 to 0.9997 and thereby I could fix 1 row pr submission whenever there was only two possibilities. If the score goes up the new target had to be right otherwise I assumed the other previous target was right. Going on like this I knew I would get to about 0.99995 and thought it could be enough to win.\nAt the end I thought I had all 37 tough one fixed - but somewhere before the 37 I think I have made 4 mistakes and still wonder where those mistakes are.</p>\n\n<p>But you found an other way to beat the entire field.\nCONGRATULATION\nKaggleuser58</p>",
      "rawMarkdown": "I had not noticed it was possible to sort the public scoreboard on the scores so my way to find the last 37 was like this.\nWith a few submissions I found a sweetspot where one more correct would raise the score from 0.9996 to 0.9997 and thereby I could fix 1 row pr submission whenever there was only two possibilities. If the score goes up the new target had to be right otherwise I assumed the other previous target was right. Going on like this I knew I would get to about 0.99995 and thought it could be enough to win.\nAt the end I thought I had all 37 tough one fixed - but somewhere before the 37 I think I have made 4 mistakes and still wonder where those mistakes are.\n\nBut you found an other way to beat the entire field.\nCONGRATULATION\nKaggleuser58",
      "votes": 2,
      "replies": [
        {
          "id": 457908,
          "postDate": "2019-01-18T10:15:20.163Z",
          "content": "<p>Thanks :) Nice job finding this sweet spot to go around the limited score precision!</p>",
          "rawMarkdown": "Thanks :) Nice job finding this sweet spot to go around the limited score precision!"
        }
      ]
    },
    {
      "id": 459738,
      "postDate": "2019-01-22T09:14:40.147Z",
      "content": "<p>Hi JBF, thank you very much for sharing your path to the perfect 1.00. Very instructive. Congratulations to your teammate and you. Really looking forward to the next one.</p>",
      "rawMarkdown": "Hi JBF, thank you very much for sharing your path to the perfect 1.00. Very instructive. Congratulations to your teammate and you. Really looking forward to the next one.",
      "replies": [
        {
          "id": 460714,
          "postDate": "2019-01-24T09:47:23.003Z",
          "content": "<p>Thanks Flal :) Reaching 0.9997 in only 4 submissions was super elegant and impressive!</p>",
          "rawMarkdown": "Thanks Flal :) Reaching 0.9997 in only 4 submissions was super elegant and impressive!"
        }
      ]
    },
    {
      "id": 458457,
      "postDate": "2019-01-19T17:31:27.610Z",
      "content": "<p>Congratulations!\nThank you for sharing your experience. It's very impressive.</p>",
      "rawMarkdown": "Congratulations!\nThank you for sharing your experience. It's very impressive.",
      "replies": [
        {
          "id": 460713,
          "postDate": "2019-01-24T09:39:15.787Z",
          "content": "<p>Thanks!</p>",
          "rawMarkdown": "Thanks!"
        }
      ]
    },
    {
      "id": 458022,
      "postDate": "2019-01-18T15:33:33.257Z",
      "content": "<p>Congratulations,\nThanks for sharing.</p>",
      "rawMarkdown": "Congratulations,\nThanks for sharing.",
      "replies": [
        {
          "id": 458319,
          "postDate": "2019-01-19T11:12:59.633Z",
          "content": "<p>Thanks :)</p>",
          "rawMarkdown": "Thanks :)"
        }
      ]
    },
    {
      "id": 457667,
      "postDate": "2019-01-17T22:38:08.997Z",
      "content": "<p>dope solution, keep going man!</p>",
      "rawMarkdown": "dope solution, keep going man!",
      "replies": [
        {
          "id": 457913,
          "postDate": "2019-01-18T10:23:13.193Z",
          "content": "<p>Thanks!</p>",
          "rawMarkdown": "Thanks!"
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T17:10:38Z",
  "competition_id": 12133,
  "competition_slug": "20-newsgroups-ciphertext-challenge",
  "team_id": 2547845,
  "solution_url": "/competitions/20-newsgroups-ciphertext-challenge/writeups/c1ph3rh4ck-1st-place-solution-c1ph3rh4ck-team"
}