{
  "id": 11522,
  "title": "Anybody still using Vowpal Wabbit for this one?",
  "url": "/competitions/avazu-ctr-prediction/discussion/11522",
  "author_name": "",
  "post_date": "2015-01-15T05:55:27.597Z",
  "votes": null,
  "comment_count": 27,
  "views": 8601,
  "content": "<p>If so, want to share your best leaderboard scores so far?</p>\n<p>I was having a hard time breaking 0.40 before switching over to the code provided by tinrtgu. I'm sure I'll go back to it before too long, but just wanted to see if anyone out there was having success with it this time around.</p>",
  "messages": [
    {
      "id": "62738",
      "postDate": "01/15/2015 05:55:27",
      "content": "<p>If so, want to share your best leaderboard scores so far?</p>\n<p>I was having a hard time breaking 0.40 before switching over to the code provided by tinrtgu. I'm sure I'll go back to it before too long, but just wanted to see if anyone out there was having success with it this time around.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62739",
      "postDate": "01/15/2015 06:42:48",
      "content": "<p>liblinear&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62741",
      "postDate": "01/15/2015 07:08:07",
      "content": "<p>Yup, vowpal.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62757",
      "postDate": "01/15/2015 17:41:00",
      "content": "<p>Yep, vowpal wabbit and 0.3935X</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62928",
      "postDate": "01/20/2015 13:43:50",
      "content": "<p>Vowpal wabbit gives me 0.40. Am i going wrong?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62929",
      "postDate": "01/20/2015 14:45:03",
      "content": "<p>I use vowpal wabbit as part of a basic ensemble which lands at .3944 with very little preprocessing or tuning.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62931",
      "postDate": "01/20/2015 16:08:29",
      "content": "<p>@ Maineiac : big jump!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62937",
      "postDate": "01/20/2015 19:27:14",
      "content": "<p>I am also using vw:</p>\n<p>0.398660 validation</p>\n<p>0.3951709 lb</p>\n<p>Not sure how, but probably overfitting the lb...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62940",
      "postDate": "01/20/2015 21:59:37",
      "content": "<p>[quote=tom;62931]</p>\n<p>@ Maineiac : big jump!</p>\n<p>[/quote]</p>\n<p>Well, I just started a week ago, so I guess that was the jump from my first submission.</p>\n<p>Unfortunately, none of this so far is with VW :(</p>\n<p>I still can't crack 0.4 with it, and ensembling with my VW predictions so far has resulted in a lower leaderboard scores. But... I'm not giving up!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62955",
      "postDate": "01/21/2015 10:56:32",
      "content": "<p>Can anyone please tell me how to ensemble in vowpal?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62965",
      "postDate": "01/21/2015 14:05:59",
      "content": "<p>There are two ways of doing this:</p>\n\n<p>1. If you want to use bootstrapped VW, you can pass the --bootstrap flag to VW and it will create a set of bagged classifiers. I tried this in this competition but could not make it work</p>\n<p>2. You can actually build you model, output the predictions to a file, and use that output file as the ensemble input. How do you use it is up to you, but the most classical approaches are to average output files or to use output as the input of a stacked classifier.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62971",
      "postDate": "01/21/2015 17:29:59",
      "content": "<p>@Guillem This is what I was looking for, thank you!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62974",
      "postDate": "01/21/2015 18:25:27",
      "content": "<p>I still use vw.</p>\n<p>It easily cracks 0.393 with some tuning and very few preprocessing.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63044",
      "postDate": "01/23/2015 07:44:30",
      "content": "<p>Does anyhow use something different by VW or tinrtgu code? I'm using scikit-learn and I'm stuck at 0.397...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63045",
      "postDate": "01/23/2015 07:54:14",
      "content": "<p>@Mikhail Did you use ensembles? I cant get above 0.394 with VW with one model..</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63048",
      "postDate": "01/23/2015 09:06:53",
      "content": "<p>Vowpal Wabbit is not a toy, but really an excellent industry-class tool. It works very fast, has&nbsp;rich parameters and options to tune.</p>\n<p>You can easily break benchmark 0.40 by some simply steps.</p>\n<p>1) data munging</p>\n<p>&nbsp; &nbsp; You can get valuable ideas form <a href=\"http://mlwave.com/predicting-click-through-rates-with-online-machine-learning/\">this article</a>.</p>\n<p>2) increase -b param(default 18)&nbsp;</p>\n<p>&nbsp; &nbsp; It determines the size of hash table(2^b). Larger b is better for it can decrease hash collision, but too large one is not useful. Make sure size of table &nbsp;do not exceed your physical memory.</p>\n<p>&nbsp; &nbsp;Please refer to <a href=\"http://fastml.com/how-to-run-external-programs-from-python-and-capture-their-output/\">this article</a>.</p>\n<p>3) tune -l the learning rate(default 1.0)</p>\n<p>&nbsp; &nbsp;You can search learning rate between (0, 1.0]. &nbsp;Low learning rate make the model learn more&nbsp;from train examples and alleviate underfit, &nbsp;but maybe increase the possibility of the overfit and take&nbsp;&nbsp;longer train time.</p>\n\n<p>However, if you want to get a better score beyond the benchmark. You should build elaborate model, which is just what I am trying now. It include feature engineering, model ensemble and exploring other tools.</p>\n\n<p>So, I think, Vowpal Wabbit is a&nbsp;nice&nbsp;start, but not the end.</p>\n\n<p>I wish you good luck!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63052",
      "postDate": "01/23/2015 10:52:40",
      "content": "<p>I used VW and I could get to 0.3945 by tuning some parameters too. In addition of the parameters posted by LifeMatrix, you can try:</p>\n<p>- Regularization (l1, l2) to prevent overfitting</p>\n<p>- Feature interaction (-q for quadratic mostly)&nbsp;</p>\n<p>- Add a hidden layer with --nn so there is some non-linearity.</p>\n\n<p>Be careful on the values you choose, otherwise you will overfit really badly.</p>\n\n<p>After that, I used Xgboost to transform the input features and reach 0.3935. See the thread&nbsp;https://www.kaggle.com/c/criteo-display-ad-challenge/forums/t/10429/congratulations-to-the-winners for some ideas.</p>\n<p>Next step is probably tune the parameters and do some ensemble, but this is very time consuming at this point.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63181",
      "postDate": "01/26/2015 08:47:04",
      "content": "<p>I'm still with vw. Single model: 0.3893. Mostly by feature engeneering.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63196",
      "postDate": "01/26/2015 17:26:11",
      "content": "<p>Very nice. I was not able to beat 0.4 with VW for this one, but maybe I haven't tried hard enough :).</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63202",
      "postDate": "01/26/2015 19:48:56",
      "content": "<p>@Pavitrakumar no ensemble, single model.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63290",
      "postDate": "01/29/2015 03:39:13",
      "content": "<p>I am using all the features as it is and I am stuck at .397... with VW. If there is one change/parameter&nbsp;which can improve the score what could it be?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63293",
      "postDate": "01/29/2015 07:09:42",
      "content": "<p>-q</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63331",
      "postDate": "01/29/2015 22:02:46",
      "content": "<p>So 2 namespaces, one for int and one for text features and tried -q with i:c ,&nbsp;:1 ,&nbsp;i: ,&nbsp;:c or c: or even :: and no luck! It performs worse and gives me score of 2.xyz. I am sure there is something else needed or&nbsp;I might be doing something&nbsp;wrong. Anyways it couldn't&nbsp;be that simple :)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63365",
      "postDate": "01/30/2015 14:38:32",
      "content": "<p>-q doesn't work for me too! try -nn</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63378",
      "postDate": "01/31/2015 04:23:51",
      "content": "<p>This is my first competition and am having my butt handed to me using just using Windows world basic tools like&nbsp;SQL Server&nbsp;and R.&nbsp; Any basic advice on what I need to learn in order to even get VW installed and use it?&nbsp; Think VW for Windows users&nbsp;dummies.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63379",
      "postDate": "01/31/2015 04:40:00",
      "content": "<p>There is an installation guide here (using cygwin):</p>\n\n<p>http://mlwave.com/install-vowpal-wabbit-on-windows-and-cygwin/</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63383",
      "postDate": "01/31/2015 08:04:10",
      "content": "<p>[quote=jcordes;63378]</p>\n<p>This is my first competition and am having my butt handed to me using just using Windows world basic tools like&nbsp;SQL Server&nbsp;and R.&nbsp; Any basic advice on what I need to learn in order to even get VW installed and use it?&nbsp; Think VW for Windows users&nbsp;dummies.</p>\n<p>[/quote]</p>\n<p>Following&nbsp;the link which James King posted, you can pretty much install the 7.4 version. However, I had problems installing the 7.8 version.</p>\n<p>So, I resorted to the binary built by Guido.This is the 7.7 version. Thanks to him!</p>\n<p>Here's the link: http://www.picnet.com.au/blogs/guido/post/2014/11/13/vowpal-wabbit-for-windows-and-binary-for-win-x64/</p>\n<p>For a basic tutorial, here it is:&nbsp;http://zinkov.com/posts/2013-08-13-vowpal-tutorial/ It's pretty old one but it is written well.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63399",
      "postDate": "02/01/2015 02:28:54",
      "content": "<p>[quote=Moon;63331]</p>\n<p>So 2 namespaces, one for int and one for text features and tried -q with i:c ,&nbsp;:1 ,&nbsp;i: ,&nbsp;:c or c: or even :: and no luck! It performs worse and gives me score of 2.xyz. I am sure there is something else needed or&nbsp;I might be doing something&nbsp;wrong. Anyways it couldn't&nbsp;be that simple :)</p>\n<p>[/quote]</p>\n<p>According to <a href=\"http://fastml.com/vowpal-wabbit-eats-big-data-from-the-criteo-competition-for-breakfast/\">this</a>&nbsp;post, each categorical feature should have its own namespace to help avoid hash collisions. This gives you more options with q, too.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 62739,
      "author_name": "xuqiang1990",
      "author_url": "",
      "post_date": "01/15/2015 06:42:48",
      "content": "<p>liblinear&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62741,
      "author_name": "konradb",
      "author_url": "",
      "post_date": "01/15/2015 07:08:07",
      "content": "<p>Yup, vowpal.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62757,
      "author_name": "guillempalou",
      "author_url": "",
      "post_date": "01/15/2015 17:41:00",
      "content": "<p>Yep, vowpal wabbit and 0.3935X</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62928,
      "author_name": "ektagolchha",
      "author_url": "",
      "post_date": "01/20/2015 13:43:50",
      "content": "<p>Vowpal wabbit gives me 0.40. Am i going wrong?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62929,
      "author_name": "jwjohnson314",
      "author_url": "",
      "post_date": "01/20/2015 14:45:03",
      "content": "<p>I use vowpal wabbit as part of a basic ensemble which lands at .3944 with very little preprocessing or tuning.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62931,
      "author_name": "ektagolchha",
      "author_url": "",
      "post_date": "01/20/2015 16:08:29",
      "content": "<p>@ Maineiac : big jump!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62937,
      "author_name": "cronopio",
      "author_url": "",
      "post_date": "01/20/2015 19:27:14",
      "content": "<p>I am also using vw:</p>\n<p>0.398660 validation</p>\n<p>0.3951709 lb</p>\n<p>Not sure how, but probably overfitting the lb...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62940,
      "author_name": "maineiac",
      "author_url": "",
      "post_date": "01/20/2015 21:59:37",
      "content": "<p>[quote=tom;62931]</p>\n<p>@ Maineiac : big jump!</p>\n<p>[/quote]</p>\n<p>Well, I just started a week ago, so I guess that was the jump from my first submission.</p>\n<p>Unfortunately, none of this so far is with VW :(</p>\n<p>I still can't crack 0.4 with it, and ensembling with my VW predictions so far has resulted in a lower leaderboard scores. But... I'm not giving up!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62955,
      "author_name": "pavitrakumar78",
      "author_url": "",
      "post_date": "01/21/2015 10:56:32",
      "content": "<p>Can anyone please tell me how to ensemble in vowpal?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62965,
      "author_name": "guillempalou",
      "author_url": "",
      "post_date": "01/21/2015 14:05:59",
      "content": "<p>There are two ways of doing this:</p>\n\n<p>1. If you want to use bootstrapped VW, you can pass the --bootstrap flag to VW and it will create a set of bagged classifiers. I tried this in this competition but could not make it work</p>\n<p>2. You can actually build you model, output the predictions to a file, and use that output file as the ensemble input. How do you use it is up to you, but the most classical approaches are to average output files or to use output as the input of a stacked classifier.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62971,
      "author_name": "pavitrakumar78",
      "author_url": "",
      "post_date": "01/21/2015 17:29:59",
      "content": "<p>@Guillem This is what I was looking for, thank you!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62974,
      "author_name": "mikhailtrofimov",
      "author_url": "",
      "post_date": "01/21/2015 18:25:27",
      "content": "<p>I still use vw.</p>\n<p>It easily cracks 0.393 with some tuning and very few preprocessing.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63044,
      "author_name": "elenacuoco",
      "author_url": "",
      "post_date": "01/23/2015 07:44:30",
      "content": "<p>Does anyhow use something different by VW or tinrtgu code? I'm using scikit-learn and I'm stuck at 0.397...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63045,
      "author_name": "pavitrakumar78",
      "author_url": "",
      "post_date": "01/23/2015 07:54:14",
      "content": "<p>@Mikhail Did you use ensembles? I cant get above 0.394 with VW with one model..</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63048,
      "author_name": "lifematrix",
      "author_url": "",
      "post_date": "01/23/2015 09:06:53",
      "content": "<p>Vowpal Wabbit is not a toy, but really an excellent industry-class tool. It works very fast, has&nbsp;rich parameters and options to tune.</p>\n<p>You can easily break benchmark 0.40 by some simply steps.</p>\n<p>1) data munging</p>\n<p>&nbsp; &nbsp; You can get valuable ideas form <a href=\"http://mlwave.com/predicting-click-through-rates-with-online-machine-learning/\">this article</a>.</p>\n<p>2) increase -b param(default 18)&nbsp;</p>\n<p>&nbsp; &nbsp; It determines the size of hash table(2^b). Larger b is better for it can decrease hash collision, but too large one is not useful. Make sure size of table &nbsp;do not exceed your physical memory.</p>\n<p>&nbsp; &nbsp;Please refer to <a href=\"http://fastml.com/how-to-run-external-programs-from-python-and-capture-their-output/\">this article</a>.</p>\n<p>3) tune -l the learning rate(default 1.0)</p>\n<p>&nbsp; &nbsp;You can search learning rate between (0, 1.0]. &nbsp;Low learning rate make the model learn more&nbsp;from train examples and alleviate underfit, &nbsp;but maybe increase the possibility of the overfit and take&nbsp;&nbsp;longer train time.</p>\n\n<p>However, if you want to get a better score beyond the benchmark. You should build elaborate model, which is just what I am trying now. It include feature engineering, model ensemble and exploring other tools.</p>\n\n<p>So, I think, Vowpal Wabbit is a&nbsp;nice&nbsp;start, but not the end.</p>\n\n<p>I wish you good luck!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63052,
      "author_name": "guillempalou",
      "author_url": "",
      "post_date": "01/23/2015 10:52:40",
      "content": "<p>I used VW and I could get to 0.3945 by tuning some parameters too. In addition of the parameters posted by LifeMatrix, you can try:</p>\n<p>- Regularization (l1, l2) to prevent overfitting</p>\n<p>- Feature interaction (-q for quadratic mostly)&nbsp;</p>\n<p>- Add a hidden layer with --nn so there is some non-linearity.</p>\n\n<p>Be careful on the values you choose, otherwise you will overfit really badly.</p>\n\n<p>After that, I used Xgboost to transform the input features and reach 0.3935. See the thread&nbsp;https://www.kaggle.com/c/criteo-display-ad-challenge/forums/t/10429/congratulations-to-the-winners for some ideas.</p>\n<p>Next step is probably tune the parameters and do some ensemble, but this is very time consuming at this point.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63181,
      "author_name": "trufanov",
      "author_url": "",
      "post_date": "01/26/2015 08:47:04",
      "content": "<p>I'm still with vw. Single model: 0.3893. Mostly by feature engeneering.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63196,
      "author_name": "triskelion",
      "author_url": "",
      "post_date": "01/26/2015 17:26:11",
      "content": "<p>Very nice. I was not able to beat 0.4 with VW for this one, but maybe I haven't tried hard enough :).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63202,
      "author_name": "mikhailtrofimov",
      "author_url": "",
      "post_date": "01/26/2015 19:48:56",
      "content": "<p>@Pavitrakumar no ensemble, single model.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63290,
      "author_name": "monami",
      "author_url": "",
      "post_date": "01/29/2015 03:39:13",
      "content": "<p>I am using all the features as it is and I am stuck at .397... with VW. If there is one change/parameter&nbsp;which can improve the score what could it be?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63293,
      "author_name": "mikhailtrofimov",
      "author_url": "",
      "post_date": "01/29/2015 07:09:42",
      "content": "<p>-q</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63331,
      "author_name": "monami",
      "author_url": "",
      "post_date": "01/29/2015 22:02:46",
      "content": "<p>So 2 namespaces, one for int and one for text features and tried -q with i:c ,&nbsp;:1 ,&nbsp;i: ,&nbsp;:c or c: or even :: and no luck! It performs worse and gives me score of 2.xyz. I am sure there is something else needed or&nbsp;I might be doing something&nbsp;wrong. Anyways it couldn't&nbsp;be that simple :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63365,
      "author_name": "elenacuoco",
      "author_url": "",
      "post_date": "01/30/2015 14:38:32",
      "content": "<p>-q doesn't work for me too! try -nn</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63378,
      "author_name": "jaycordes",
      "author_url": "",
      "post_date": "01/31/2015 04:23:51",
      "content": "<p>This is my first competition and am having my butt handed to me using just using Windows world basic tools like&nbsp;SQL Server&nbsp;and R.&nbsp; Any basic advice on what I need to learn in order to even get VW installed and use it?&nbsp; Think VW for Windows users&nbsp;dummies.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63379,
      "author_name": "jfkingiii",
      "author_url": "",
      "post_date": "01/31/2015 04:40:00",
      "content": "<p>There is an installation guide here (using cygwin):</p>\n\n<p>http://mlwave.com/install-vowpal-wabbit-on-windows-and-cygwin/</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63383,
      "author_name": "phanisrikanth",
      "author_url": "",
      "post_date": "01/31/2015 08:04:10",
      "content": "<p>[quote=jcordes;63378]</p>\n<p>This is my first competition and am having my butt handed to me using just using Windows world basic tools like&nbsp;SQL Server&nbsp;and R.&nbsp; Any basic advice on what I need to learn in order to even get VW installed and use it?&nbsp; Think VW for Windows users&nbsp;dummies.</p>\n<p>[/quote]</p>\n<p>Following&nbsp;the link which James King posted, you can pretty much install the 7.4 version. However, I had problems installing the 7.8 version.</p>\n<p>So, I resorted to the binary built by Guido.This is the 7.7 version. Thanks to him!</p>\n<p>Here's the link: http://www.picnet.com.au/blogs/guido/post/2014/11/13/vowpal-wabbit-for-windows-and-binary-for-win-x64/</p>\n<p>For a basic tutorial, here it is:&nbsp;http://zinkov.com/posts/2013-08-13-vowpal-tutorial/ It's pretty old one but it is written well.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63399,
      "author_name": "jwjohnson314",
      "author_url": "",
      "post_date": "02/01/2015 02:28:54",
      "content": "<p>[quote=Moon;63331]</p>\n<p>So 2 namespaces, one for int and one for text features and tried -q with i:c ,&nbsp;:1 ,&nbsp;i: ,&nbsp;:c or c: or even :: and no luck! It performs worse and gives me score of 2.xyz. I am sure there is something else needed or&nbsp;I might be doing something&nbsp;wrong. Anyways it couldn't&nbsp;be that simple :)</p>\n<p>[/quote]</p>\n<p>According to <a href=\"http://fastml.com/vowpal-wabbit-eats-big-data-from-the-criteo-competition-for-breakfast/\">this</a>&nbsp;post, each categorical feature should have its own namespace to help avoid hash collisions. This gives you more options with q, too.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "62738": "",
    "62739": "",
    "62741": "",
    "62757": "",
    "62928": "",
    "62929": "",
    "62931": "",
    "62937": "",
    "62940": "",
    "62955": "",
    "62965": "",
    "62971": "",
    "62974": "",
    "63044": "",
    "63045": "",
    "63048": "",
    "63052": "",
    "63181": "",
    "63196": "",
    "63202": "",
    "63290": "",
    "63293": "",
    "63331": "",
    "63365": "",
    "63378": "",
    "63379": "",
    "63383": "",
    "63399": ""
  },
  "source": "meta"
}