{ "id": 9722, "title": "Sample code different from benchmark", "url": "/competitions/avito-prohibited-content/discussion/9722", "author_name": "", "post_date": "2014-07-12T03:41:29.353Z", "votes": null, "comment_count": 18, "views": 3734, "content": "

I'm unable to reproduce the 0.89061 result with the provided sample code.

\n

After uncommenting the lines in main(), I got 0.88598.  Many other entrants on the leaderboard also have this score, so it appears it's not just me.

\n

In the other thread, it's said that a fix to line 57 is required:

\n

item = {featureName:featureValue.decode('utf-8') for featureName,featureValue in item.iteritems() if featureValue is not None}

\n

However, I get the exact same result file (no diffs) with or without this fix.

\n\n

What other changes are required to the sample code to achieve the stated benchmark score?  Could this be a platform difference? I'm on Mac OS 10.9 / Python 2.7.7 / nltk 2.0.4.

", "messages": [ { "id": "50377", "postDate": "07/12/2014 03:41:29", "content": "

I'm unable to reproduce the 0.89061 result with the provided sample code.

\n

After uncommenting the lines in main(), I got 0.88598.  Many other entrants on the leaderboard also have this score, so it appears it's not just me.

\n

In the other thread, it's said that a fix to line 57 is required:

\n

item = {featureName:featureValue.decode('utf-8') for featureName,featureValue in item.iteritems() if featureValue is not None}

\n

However, I get the exact same result file (no diffs) with or without this fix.

\n\n

What other changes are required to the sample code to achieve the stated benchmark score?  Could this be a platform difference? I'm on Mac OS 10.9 / Python 2.7.7 / nltk 2.0.4.

", "rawMarkdown": "", "votes": null }, { "id": "50400", "postDate": "07/12/2014 13:19:26", "content": "

@Synthient

\n

Try to change a little bit on the alpha in the SGD classifier, and set correct words and stem to True, you will got 0.9.

\n

clf = SGDClassifier( loss="log", penalty="l2", alpha=1e-5, class_weight="auto")

", "rawMarkdown": "", "votes": null }, { "id": "50409", "postDate": "07/12/2014 16:41:08", "content": "

Thanks. I mostly just wanted to reproduce the benchmark score. When the published code doesn't work or doesn't produce the advertised result, sometimes there's something else missing...

", "rawMarkdown": "", "votes": null }, { "id": "50916", "postDate": "07/20/2014 08:20:52", "content": "

Hi Kevin,

\n

Do you know the sample size and other parameters used for Avito Best Prediction, e.g. learning rate and alpha? I am assuming it used the Corrected Words and Stemming.

", "rawMarkdown": "", "votes": null }, { "id": "50917", "postDate": "07/20/2014 08:22:36", "content": "

Hi Synth,

\n

I also think you need to set nIter = 100 or something, the default nIter for SGDClassiffier is 5, but it will hardly converge within 5 iteration

", "rawMarkdown": "", "votes": null }, { "id": "51147", "postDate": "07/24/2014 17:27:04", "content": "

Can anybody help me with this piece of code. I have no idea what is does? I would be glad if can somebody explain the code line by line. Thanks in advance.

\n

stopwords= frozenset(word.decode('utf-8') for word in nltk.corpus.stopwords.words("russian") if word!="не")
stemmer = SnowballStemmer('russian')
engChars = [ord(char) for char in u"cCyoOBaAKpPeE"]
rusChars = [ord(char) for char in u"сСуоОВаАКрРеЕ"]
eng_rusTranslateTable = dict(zip(engChars, rusChars))
rus_engTranslateTable = dict(zip(rusChars, engChars))

\n

logging.basicConfig(format = u'[LINE:%(lineno)d]# %(levelname)-8s [%(asctime)s] %(message)s', level = logging.NOTSET)

def correctWord (w):
""" Corrects word by replacing characters with written similarly depending on which language the word.
Fraudsters use this technique to avoid detection by anti-fraud algorithms."""

\n

if len(re.findall(ur"[а-я]",w))>len(re.findall(ur"[a-z]",w)):
return w.translate(eng_rusTranslateTable)
else:
return w.translate(rus_engTranslateTable)

", "rawMarkdown": "", "votes": null }, { "id": "51155", "postDate": "07/24/2014 19:48:05", "content": "

[quote]

\n

engChars = [ord(char) for char in u"cCyoOBaAKpPeE"]
rusChars = [ord(char) for char in u"сСуоОВаАКрРеЕ"]

\n

[/quote]

\n

- this is homoglyphs in english and russian layouts. 

\n

[quote]

\n

if len(re.findall(ur"[а-я]",w))>len(re.findall(ur"[a-z]",w)):

\n

...

\n

[/quote]

\n

- this is detection which language the word.

\n

The remaining lines are evident, IMHO...

", "rawMarkdown": "", "votes": null }, { "id": "51168", "postDate": "07/25/2014 06:54:24", "content": "

[quote=Kevin Hu;50400]

\n

@Synthient

\n

Try to change a little bit on the alpha in the SGD classifier, and set correct words and stem to True, you will got 0.9.

\n

clf = SGDClassifier( loss="log", penalty="l2", alpha=1e-5, class_weight="auto")

\n

[/quote]

\n

Even after this change i cant able to get 0.89

", "rawMarkdown": "", "votes": null }, { "id": "51169", "postDate": "07/25/2014 06:54:56", "content": "

still in 0.286

", "rawMarkdown": "", "votes": null }, { "id": "51179", "postDate": "07/25/2014 16:15:18", "content": "

[quote=Parthiban Gowthaman;51169]

\n

still in 0.286

\n

[/quote]

\n

Increase the number of iterations, using n_iter: parameter, additionally, changing the number of features also helpful.

", "rawMarkdown": "", "votes": null }, { "id": "51180", "postDate": "07/25/2014 17:06:43", "content": "

Double check if you are submitting the updated file created by the python script

", "rawMarkdown": "", "votes": null }, { "id": "51255", "postDate": "07/28/2014 06:39:23", "content": "

I am unable to reproduce 0.89 benchmark. The score was 0.37.

\n

I have made following changes:

\n

item = {featureName:featureValue.decode('utf-8') for featureName,featureValue in item.iteritems() if featureValue is not None}

\n

Changed the alpha value in SGD classifier to 1e-5, n_ter = 100

\n

Uncommented the 4 lines in main 

", "rawMarkdown": "", "votes": null }, { "id": "51285", "postDate": "07/29/2014 04:15:59", "content": "

I was getting the correct benchmark at first and then when I switched computers I started getting low values like 0.28. These only happened when I was running the code within Spyder, so it must have been using an old version of python or more likely loading outdated versions of the libraries. Make sure you have the most recent version of everything (by installing a distribution like Anaconda, for example) and then run the sample code directly from the command prompt rather than from an IDE.

", "rawMarkdown": "", "votes": null }, { "id": "51287", "postDate": "07/29/2014 04:31:25", "content": "

[quote=James King;51285]

\n

I was getting the correct benchmark at first and then when I switched computers I started getting low values like 0.28. These only happened when I was running the code within Spyder, so it must have been using an old version of python or more likely loading outdated versions of the libraries. Make sure you have the most recent version of everything (by installing a distribution like Anaconda, for example) and then run the sample code directly from the command prompt rather than from an IDE.

\n

[/quote]

\n

When I executed the same code in Ubuntu platform, I  am able to get 0.9. It might be that the libraries are outdated and older version of python.

", "rawMarkdown": "", "votes": null }, { "id": "51432", "postDate": "07/31/2014 18:13:08", "content": "

My question is around the existing python code given to us and the benchmark of 0.89. How did people get it to run with the existing code: "clf.fit(trainFeatures,trainTargets)"

\n

I get this error:

\n\n

[LINE:159]# INFO [2014-07-31 17:08:40,598] Feature preparation done, fitting model...
Traceback (most recent call last):
File "avito_ProhibitedContent_SampleCode.py", line 184, in <module>
main()
File "avito_ProhibitedContent_SampleCode.py", line 165, in main
clf.fit(trainFeatures,trainTargets)
File "/usr/lib/pymodules/python2.7/sklearn/linear_model/base.py", line 430, in fit
self._fit_binary(X, y, sample_weight)
File "/usr/lib/pymodules/python2.7/sklearn/linear_model/stochastic_gradient.py", line 131, in _fit_binary
X = np.asarray(X, dtype=np.float64, order='C')
File "/usr/lib/python2.7/dist-packages/numpy/core/numeric.py", line 235, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: setting an array element with a sequence.

", "rawMarkdown": "", "votes": null }, { "id": "51437", "postDate": "07/31/2014 20:37:11", "content": "

For those that had the ~0.30 result problem, how did you get around that?

\n

So far I have tried running the provided script in both Windows and Linux, with different versions of Anaconda environment (1.9 and 2.0) and all the latest updates. Results vary between 0.27 and 0.30 but not more.

\n

Has anyone been able to pinpoint if it is an issue with the feature calculation if you don't have specific library versions or if it is something in the model itself?

\n

Any specific versions of operating system and python libraries, etc. that work? Also have you installed the entire nltk or only the few parts required to run the script?

\n

Thanks,

\n

Andreas

", "rawMarkdown": "", "votes": null }, { "id": "51451", "postDate": "08/01/2014 05:38:04", "content": "

What version of scikit are you using ? I get error in the code below. Please help - 

\n\n\n

self._fit_binary(X, y, sample_weight)
File "/usr/lib/pymodules/python2.7/sklearn/linear_model/stochastic_gradient.py", line 131, in _fit_binary
X = np.asarray(X, dtype=np.float64, order='C')
File "/usr/lib/python2.7/dist-packages/numpy/core/numeric.py", line 235, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: setting an array element with a sequence.

\n\n

trainFeatures,trainTargets, trainItemIds=processData(os.path.join(dataFolder,"avito_train.tsv"), featureIndexes, itemsLimit=3000)
logging.info("Feature preparation done, fitting model...")
clf = SGDClassifier( loss="log",
penalty="l2",
alpha=1e-4,
class_weight="auto")
clf.fit(trainFeatures,trainTargets)
#clf.fit(trainFeatures.toarray(),trainTargets)

", "rawMarkdown": "", "votes": null }, { "id": "51452", "postDate": "08/01/2014 05:38:42", "content": "", "rawMarkdown": "", "votes": null }, { "id": "51506", "postDate": "08/01/2014 22:03:15", "content": "

Resolved : I had an older version of scikit. Got 0.15 now and it all worked.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 50400, "author_name": "kevinandhu", "author_url": "", "post_date": "07/12/2014 13:19:26", "content": "

@Synthient

\n

Try to change a little bit on the alpha in the SGD classifier, and set correct words and stem to True, you will got 0.9.

\n

clf = SGDClassifier( loss="log", penalty="l2", alpha=1e-5, class_weight="auto")

", "votes": null, "replies": [] }, { "id": 50409, "author_name": "brundage", "author_url": "", "post_date": "07/12/2014 16:41:08", "content": "

Thanks. I mostly just wanted to reproduce the benchmark score. When the published code doesn't work or doesn't produce the advertised result, sometimes there's something else missing...

", "votes": null, "replies": [] }, { "id": 50916, "author_name": "sayan21", "author_url": "", "post_date": "07/20/2014 08:20:52", "content": "

Hi Kevin,

\n

Do you know the sample size and other parameters used for Avito Best Prediction, e.g. learning rate and alpha? I am assuming it used the Corrected Words and Stemming.

", "votes": null, "replies": [] }, { "id": 50917, "author_name": "sayan21", "author_url": "", "post_date": "07/20/2014 08:22:36", "content": "

Hi Synth,

\n

I also think you need to set nIter = 100 or something, the default nIter for SGDClassiffier is 5, but it will hardly converge within 5 iteration

", "votes": null, "replies": [] }, { "id": 51147, "author_name": "novice7", "author_url": "", "post_date": "07/24/2014 17:27:04", "content": "

Can anybody help me with this piece of code. I have no idea what is does? I would be glad if can somebody explain the code line by line. Thanks in advance.

\n

stopwords= frozenset(word.decode('utf-8') for word in nltk.corpus.stopwords.words("russian") if word!="не")
stemmer = SnowballStemmer('russian')
engChars = [ord(char) for char in u"cCyoOBaAKpPeE"]
rusChars = [ord(char) for char in u"сСуоОВаАКрРеЕ"]
eng_rusTranslateTable = dict(zip(engChars, rusChars))
rus_engTranslateTable = dict(zip(rusChars, engChars))

\n

logging.basicConfig(format = u'[LINE:%(lineno)d]# %(levelname)-8s [%(asctime)s] %(message)s', level = logging.NOTSET)

def correctWord (w):
""" Corrects word by replacing characters with written similarly depending on which language the word.
Fraudsters use this technique to avoid detection by anti-fraud algorithms."""

\n

if len(re.findall(ur"[а-я]",w))>len(re.findall(ur"[a-z]",w)):
return w.translate(eng_rusTranslateTable)
else:
return w.translate(rus_engTranslateTable)

", "votes": null, "replies": [] }, { "id": 51155, "author_name": "vasilyleksin", "author_url": "", "post_date": "07/24/2014 19:48:05", "content": "

[quote]

\n

engChars = [ord(char) for char in u"cCyoOBaAKpPeE"]
rusChars = [ord(char) for char in u"сСуоОВаАКрРеЕ"]

\n

[/quote]

\n

- this is homoglyphs in english and russian layouts. 

\n

[quote]

\n

if len(re.findall(ur"[а-я]",w))>len(re.findall(ur"[a-z]",w)):

\n

...

\n

[/quote]

\n

- this is detection which language the word.

\n

The remaining lines are evident, IMHO...

", "votes": null, "replies": [] }, { "id": 51168, "author_name": "parthibangowthaman", "author_url": "", "post_date": "07/25/2014 06:54:24", "content": "

[quote=Kevin Hu;50400]

\n

@Synthient

\n

Try to change a little bit on the alpha in the SGD classifier, and set correct words and stem to True, you will got 0.9.

\n

clf = SGDClassifier( loss="log", penalty="l2", alpha=1e-5, class_weight="auto")

\n

[/quote]

\n

Even after this change i cant able to get 0.89

", "votes": null, "replies": [] }, { "id": 51169, "author_name": "parthibangowthaman", "author_url": "", "post_date": "07/25/2014 06:54:56", "content": "

still in 0.286

", "votes": null, "replies": [] }, { "id": 51179, "author_name": "upulbandara", "author_url": "", "post_date": "07/25/2014 16:15:18", "content": "

[quote=Parthiban Gowthaman;51169]

\n

still in 0.286

\n

[/quote]

\n

Increase the number of iterations, using n_iter: parameter, additionally, changing the number of features also helpful.

", "votes": null, "replies": [] }, { "id": 51180, "author_name": "snisioi", "author_url": "", "post_date": "07/25/2014 17:06:43", "content": "

Double check if you are submitting the updated file created by the python script

", "votes": null, "replies": [] }, { "id": 51255, "author_name": "novice7", "author_url": "", "post_date": "07/28/2014 06:39:23", "content": "

I am unable to reproduce 0.89 benchmark. The score was 0.37.

\n

I have made following changes:

\n

item = {featureName:featureValue.decode('utf-8') for featureName,featureValue in item.iteritems() if featureValue is not None}

\n

Changed the alpha value in SGD classifier to 1e-5, n_ter = 100

\n

Uncommented the 4 lines in main 

", "votes": null, "replies": [] }, { "id": 51285, "author_name": "jfkingiii", "author_url": "", "post_date": "07/29/2014 04:15:59", "content": "

I was getting the correct benchmark at first and then when I switched computers I started getting low values like 0.28. These only happened when I was running the code within Spyder, so it must have been using an old version of python or more likely loading outdated versions of the libraries. Make sure you have the most recent version of everything (by installing a distribution like Anaconda, for example) and then run the sample code directly from the command prompt rather than from an IDE.

", "votes": null, "replies": [] }, { "id": 51287, "author_name": "novice7", "author_url": "", "post_date": "07/29/2014 04:31:25", "content": "

[quote=James King;51285]

\n

I was getting the correct benchmark at first and then when I switched computers I started getting low values like 0.28. These only happened when I was running the code within Spyder, so it must have been using an old version of python or more likely loading outdated versions of the libraries. Make sure you have the most recent version of everything (by installing a distribution like Anaconda, for example) and then run the sample code directly from the command prompt rather than from an IDE.

\n

[/quote]

\n

When I executed the same code in Ubuntu platform, I  am able to get 0.9. It might be that the libraries are outdated and older version of python.

", "votes": null, "replies": [] }, { "id": 51432, "author_name": "avanigoe", "author_url": "", "post_date": "07/31/2014 18:13:08", "content": "

My question is around the existing python code given to us and the benchmark of 0.89. How did people get it to run with the existing code: "clf.fit(trainFeatures,trainTargets)"

\n

I get this error:

\n\n

[LINE:159]# INFO [2014-07-31 17:08:40,598] Feature preparation done, fitting model...
Traceback (most recent call last):
File "avito_ProhibitedContent_SampleCode.py", line 184, in <module>
main()
File "avito_ProhibitedContent_SampleCode.py", line 165, in main
clf.fit(trainFeatures,trainTargets)
File "/usr/lib/pymodules/python2.7/sklearn/linear_model/base.py", line 430, in fit
self._fit_binary(X, y, sample_weight)
File "/usr/lib/pymodules/python2.7/sklearn/linear_model/stochastic_gradient.py", line 131, in _fit_binary
X = np.asarray(X, dtype=np.float64, order='C')
File "/usr/lib/python2.7/dist-packages/numpy/core/numeric.py", line 235, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: setting an array element with a sequence.

", "votes": null, "replies": [] }, { "id": 51437, "author_name": "andreasmerentitis", "author_url": "", "post_date": "07/31/2014 20:37:11", "content": "

For those that had the ~0.30 result problem, how did you get around that?

\n

So far I have tried running the provided script in both Windows and Linux, with different versions of Anaconda environment (1.9 and 2.0) and all the latest updates. Results vary between 0.27 and 0.30 but not more.

\n

Has anyone been able to pinpoint if it is an issue with the feature calculation if you don't have specific library versions or if it is something in the model itself?

\n

Any specific versions of operating system and python libraries, etc. that work? Also have you installed the entire nltk or only the few parts required to run the script?

\n

Thanks,

\n

Andreas

", "votes": null, "replies": [] }, { "id": 51451, "author_name": "avanigoe", "author_url": "", "post_date": "08/01/2014 05:38:04", "content": "

What version of scikit are you using ? I get error in the code below. Please help - 

\n\n\n

self._fit_binary(X, y, sample_weight)
File "/usr/lib/pymodules/python2.7/sklearn/linear_model/stochastic_gradient.py", line 131, in _fit_binary
X = np.asarray(X, dtype=np.float64, order='C')
File "/usr/lib/python2.7/dist-packages/numpy/core/numeric.py", line 235, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: setting an array element with a sequence.

\n\n

trainFeatures,trainTargets, trainItemIds=processData(os.path.join(dataFolder,"avito_train.tsv"), featureIndexes, itemsLimit=3000)
logging.info("Feature preparation done, fitting model...")
clf = SGDClassifier( loss="log",
penalty="l2",
alpha=1e-4,
class_weight="auto")
clf.fit(trainFeatures,trainTargets)
#clf.fit(trainFeatures.toarray(),trainTargets)

", "votes": null, "replies": [] }, { "id": 51452, "author_name": "avanigoe", "author_url": "", "post_date": "08/01/2014 05:38:42", "content": "", "votes": null, "replies": [] }, { "id": 51506, "author_name": "avanigoe", "author_url": "", "post_date": "08/01/2014 22:03:15", "content": "

Resolved : I had an older version of scikit. Got 0.15 now and it all worked.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "50377": "", "50400": "", "50409": "", "50916": "", "50917": "", "51147": "", "51155": "", "51168": "", "51169": "", "51179": "", "51180": "", "51255": "", "51285": "", "51287": "", "51432": "", "51437": "", "51451": "", "51452": "", "51506": "" }, "source": "meta" }