{
  "id": 5609,
  "title": "Use of external modules/data",
  "url": "/competitions/facebook-recruiting-iii-keyword-extraction/discussion/5609",
  "author_name": "",
  "post_date": "2013-09-01T05:02:45.250Z",
  "votes": null,
  "comment_count": 11,
  "views": 3693,
  "content": "<p>Similar questions have been asked before, but I am still not clear about it.</p>\n<p>Specifically, what I want to know is:</p>\n<p>1) How is 'external data' defined? Can I have a file containing stopwords in my submission? Can I use the unix words list? Can I download a list of words from somewhere else and use it?</p>\n<p>2) Can I use existing NLP packages or modules in my code? What if some of these modules use data that might be considered external? e.g, can I use the nltk library in python? Can I use the stopwords corpus that comes with it?</p>\n<p>&nbsp;</p>\n<p>I understand the intent behind the rule, but I am new to Kaggle competitions and to NLP/Information retrieval competitions, so I am not sure to what extent it is applicable. Please help me out here. Thanks!</p>",
  "messages": [
    {
      "id": "29891",
      "postDate": "09/01/2013 05:02:45",
      "content": "<p>Similar questions have been asked before, but I am still not clear about it.</p>\n<p>Specifically, what I want to know is:</p>\n<p>1) How is 'external data' defined? Can I have a file containing stopwords in my submission? Can I use the unix words list? Can I download a list of words from somewhere else and use it?</p>\n<p>2) Can I use existing NLP packages or modules in my code? What if some of these modules use data that might be considered external? e.g, can I use the nltk library in python? Can I use the stopwords corpus that comes with it?</p>\n<p>&nbsp;</p>\n<p>I understand the intent behind the rule, but I am new to Kaggle competitions and to NLP/Information retrieval competitions, so I am not sure to what extent it is applicable. Please help me out here. Thanks!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29933",
      "postDate": "09/01/2013 14:31:15",
      "content": "<p>Bump! Really confused about this. I am going to go ahead and use nltk (http://nltk.org/) and its corpus for now.</p>\n<p>Please do let me know if you think it is not permitted.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29944",
      "postDate": "09/01/2013 17:37:50",
      "content": "<p>I am also wondering the same things - it would seem ridiculous to not be able to use the stop words corpus in NLTK, but the rules do make it sound that way.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "30056",
      "postDate": "09/03/2013 14:47:57",
      "content": "<p>Yes, you may use NLTK and things like stop words lists. The external data rule is meant to cover &quot;non-legitimate uses&quot;, like scraping the Stack website or using a similar corpus of tagged questions as training data.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "30059",
      "postDate": "09/03/2013 14:54:18",
      "content": "<p>Thanks William!&nbsp;Much appreciated.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "30067",
      "postDate": "09/03/2013 16:29:09",
      "content": "<p>Thanks for the clarification William!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "30394",
      "postDate": "09/07/2013 01:35:45",
      "content": "<p>Github has open-sourced the library that they use to detect what programming language some code is written in (<a href=\"https://github.com/github/linguist\" target=\"_blank\">https://github.com/github/linguist</a>). Is it okay to use the language training data they use for their&nbsp;Bayesian classifier?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "30511",
      "postDate": "09/09/2013 00:47:15",
      "content": "<p>Marc, I would tend to think no. I think a trained model or the data set github uses counts as &quot;using a similar corpus of tagged questions as training data.&quot;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "32637",
      "postDate": "10/24/2013 17:30:02",
      "content": "<p>Can I use wikipedia and similar semantic web implementations to discover relational mappings?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "32638",
      "postDate": "10/24/2013 17:31:42",
      "content": "<p>[quote=Orr Inbar;32637]</p>\n<p>Can I use wikipedia and similar semantic web implementations to discover relational mappings?</p>\n<p>[/quote]</p>\n<p>Nope. Wikipedia is external data.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33045",
      "postDate": "11/01/2013 18:05:19",
      "content": "<p>I am new to Kaggle and wasn't sure about the convention. Is it expected to write all the implementations from ground up, or can I use existing libraries for classification/etc.. in my code (eg: scikit for python -&nbsp;http://scikit-learn.org/stable/)?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33069",
      "postDate": "11/01/2013 22:56:08",
      "content": "<p>Yes, you can certainly use existing libraries.&nbsp;</p>\n<p>[quote=params;33045]</p>\n<p>I am new to Kaggle and wasn't sure about the convention. Is it expected to write all the implementations from ground up, or can I use existing libraries for classification/etc.. in my code (eg: scikit for python -&nbsp;http://scikit-learn.org/stable/)?</p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 29933,
      "author_name": "raze2dust",
      "author_url": "",
      "post_date": "09/01/2013 14:31:15",
      "content": "<p>Bump! Really confused about this. I am going to go ahead and use nltk (http://nltk.org/) and its corpus for now.</p>\n<p>Please do let me know if you think it is not permitted.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29944,
      "author_name": "dougcox",
      "author_url": "",
      "post_date": "09/01/2013 17:37:50",
      "content": "<p>I am also wondering the same things - it would seem ridiculous to not be able to use the stop words corpus in NLTK, but the rules do make it sound that way.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 30056,
      "author_name": "wcukierski",
      "author_url": "",
      "post_date": "09/03/2013 14:47:57",
      "content": "<p>Yes, you may use NLTK and things like stop words lists. The external data rule is meant to cover &quot;non-legitimate uses&quot;, like scraping the Stack website or using a similar corpus of tagged questions as training data.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 30059,
      "author_name": "raze2dust",
      "author_url": "",
      "post_date": "09/03/2013 14:54:18",
      "content": "<p>Thanks William!&nbsp;Much appreciated.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 30067,
      "author_name": "dougcox",
      "author_url": "",
      "post_date": "09/03/2013 16:29:09",
      "content": "<p>Thanks for the clarification William!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 30394,
      "author_name": "marcrosen",
      "author_url": "",
      "post_date": "09/07/2013 01:35:45",
      "content": "<p>Github has open-sourced the library that they use to detect what programming language some code is written in (<a href=\"https://github.com/github/linguist\" target=\"_blank\">https://github.com/github/linguist</a>). Is it okay to use the language training data they use for their&nbsp;Bayesian classifier?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 30511,
      "author_name": "dougcox",
      "author_url": "",
      "post_date": "09/09/2013 00:47:15",
      "content": "<p>Marc, I would tend to think no. I think a trained model or the data set github uses counts as &quot;using a similar corpus of tagged questions as training data.&quot;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 32637,
      "author_name": "orrinbar",
      "author_url": "",
      "post_date": "10/24/2013 17:30:02",
      "content": "<p>Can I use wikipedia and similar semantic web implementations to discover relational mappings?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 32638,
      "author_name": "wcukierski",
      "author_url": "",
      "post_date": "10/24/2013 17:31:42",
      "content": "<p>[quote=Orr Inbar;32637]</p>\n<p>Can I use wikipedia and similar semantic web implementations to discover relational mappings?</p>\n<p>[/quote]</p>\n<p>Nope. Wikipedia is external data.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33045,
      "author_name": "params",
      "author_url": "",
      "post_date": "11/01/2013 18:05:19",
      "content": "<p>I am new to Kaggle and wasn't sure about the convention. Is it expected to write all the implementations from ground up, or can I use existing libraries for classification/etc.. in my code (eg: scikit for python -&nbsp;http://scikit-learn.org/stable/)?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33069,
      "author_name": "rohits",
      "author_url": "",
      "post_date": "11/01/2013 22:56:08",
      "content": "<p>Yes, you can certainly use existing libraries.&nbsp;</p>\n<p>[quote=params;33045]</p>\n<p>I am new to Kaggle and wasn't sure about the convention. Is it expected to write all the implementations from ground up, or can I use existing libraries for classification/etc.. in my code (eg: scikit for python -&nbsp;http://scikit-learn.org/stable/)?</p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "29891": "",
    "29933": "",
    "29944": "",
    "30056": "",
    "30059": "",
    "30067": "",
    "30394": "",
    "30511": "",
    "32637": "",
    "32638": "",
    "33045": "",
    "33069": ""
  },
  "source": "meta"
}