{ "id": 9601, "title": "what python program is doing", "url": "/competitions/avito-prohibited-content/discussion/9601", "author_name": "", "post_date": "2014-06-28T06:42:13.860Z", "votes": null, "comment_count": 5, "views": 1694, "content": "

Hi ,

\n

Can someone explain what python program is doing , As per my knowledge it is taking the item and description words and creating a sparse matrix and running the SGDClassifier . what else is going on in this program ?,

\n

are russian characters are getting converted to english character ?

", "messages": [ { "id": "49757", "postDate": "06/28/2014 06:42:13", "content": "

Hi ,

\n

Can someone explain what python program is doing , As per my knowledge it is taking the item and description words and creating a sparse matrix and running the SGDClassifier . what else is going on in this program ?,

\n

are russian characters are getting converted to english character ?

", "rawMarkdown": "", "votes": null }, { "id": "49873", "postDate": "06/30/2014 17:35:33", "content": "

In the sample program, the words in the title and description are being taken into consideration. Each word acts as a binary feature. 

\n\n

Let us know if you need help with anything specific in the program. 

", "rawMarkdown": "", "votes": null }, { "id": "50736", "postDate": "07/16/2014 21:50:56", "content": "

[quote=Phani Vadrevu;49873]

\n

In the sample program, the words in the title and description are being taken into consideration. Each word acts as a binary feature. 

\n

Let us know if you need help with anything specific in the program. 

\n

[/quote]

\n

Hi,

\n

I don't think that each word acts as a binary feature.

\n

Looking at the given code each word can enter more than once for each item and in that case the entry in the sparse matrix will be greater than one.

\n

Take a look at that code:

\n

from scipy.sparse import *
from scipy import *
import numpy as np
row = array([0,0,1,2,2])
col = array([0,2,2,1,1])
m = csr_matrix( (np.ones(len(row)),(row,col)), shape=(3,3) ).todense()

\n

row=2, col=1 has two entries so the resulted matrix will have the value 2 at cell (2,1).

\n

m = matrix([[ 1., 0., 1.],
            [ 0., 0., 1.],
            [ 0., 2., 0.]])

\n

Hope that helps.

\n

C

", "rawMarkdown": "", "votes": null }, { "id": "50773", "postDate": "07/17/2014 13:16:56", "content": "

are russian characters are getting converted to english character ?

\n

The issue with Russian/English - or Cyrillic/Latin characters is a technique fraudsters use to bypass automated word detection. These characters look the same in both alphabets, but their codes are different.

\n

So if a word in Cyrillic is written such as: Начните these would be all character codes from Cyrillic code page, but fraudsters would replace latter н for example with Latin H because it looks the same. The ASCII code for Latin H is different to Cyrillic н so the auto-detection may miss the word, unless this replacement is made beforehand.

", "rawMarkdown": "", "votes": null }, { "id": "50793", "postDate": "07/17/2014 18:14:32", "content": "

[quote=Jano Petras;50773]

\n

are russian characters are getting converted to english character ?

\n

The issue with Russian/English - or Cyrillic/Latin characters is a technique fraudsters use to bypass automated word detection. These characters look the same in both alphabets, but their codes are different.

\n

So if a word in Cyrillic is written such as: Начните these would be all character codes from Cyrillic code page, but fraudsters would replace latter н for example with Latin H because it looks the same. The ASCII code for Latin H is different to Cyrillic н so the auto-detection may miss the word, unless this replacement is made beforehand.

\n

[/quote]

\n

In the `correctWord` function, I notice that eng2rus replacement is done only when there's more Cyrillic characters than Latin characters, and rus2eng replacement is done only when there's more Latin characters. Do you know why?

", "rawMarkdown": "", "votes": null }, { "id": "50794", "postDate": "07/17/2014 18:37:58", "content": "

It was mentioned that the fraudsters use this technique to evade autoamatic detection systems. 

\n

Think about how it works. If I am a fraudster and I would like to sell some porn material which is illegal, I will try to use the word 'pоrn' instead of 'porn' in the description. Notice I used the Cyrillic 'о' instead of Latin 'o'. From the fraudster's perspective, its only enough to change as few characters with look-alike characters. Even one change should be good enough to escape detection. 

\n

Vice-versa follows for original Russian text that needs to be hidden. I am guessing that that's why we should convert all characters to the alphabet that has more characters originally.

\n

EDIT: I just realized that I explained the opposite thing. Sorry, I don't know why it is being done this way!

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 49873, "author_name": "phanivadrevu", "author_url": "", "post_date": "06/30/2014 17:35:33", "content": "

In the sample program, the words in the title and description are being taken into consideration. Each word acts as a binary feature. 

\n\n

Let us know if you need help with anything specific in the program. 

", "votes": null, "replies": [] }, { "id": 50736, "author_name": "clustifier", "author_url": "", "post_date": "07/16/2014 21:50:56", "content": "

[quote=Phani Vadrevu;49873]

\n

In the sample program, the words in the title and description are being taken into consideration. Each word acts as a binary feature. 

\n

Let us know if you need help with anything specific in the program. 

\n

[/quote]

\n

Hi,

\n

I don't think that each word acts as a binary feature.

\n

Looking at the given code each word can enter more than once for each item and in that case the entry in the sparse matrix will be greater than one.

\n

Take a look at that code:

\n

from scipy.sparse import *
from scipy import *
import numpy as np
row = array([0,0,1,2,2])
col = array([0,2,2,1,1])
m = csr_matrix( (np.ones(len(row)),(row,col)), shape=(3,3) ).todense()

\n

row=2, col=1 has two entries so the resulted matrix will have the value 2 at cell (2,1).

\n

m = matrix([[ 1., 0., 1.],
            [ 0., 0., 1.],
            [ 0., 2., 0.]])

\n

Hope that helps.

\n

C

", "votes": null, "replies": [] }, { "id": 50773, "author_name": "janopetras", "author_url": "", "post_date": "07/17/2014 13:16:56", "content": "

are russian characters are getting converted to english character ?

\n

The issue with Russian/English - or Cyrillic/Latin characters is a technique fraudsters use to bypass automated word detection. These characters look the same in both alphabets, but their codes are different.

\n

So if a word in Cyrillic is written such as: Начните these would be all character codes from Cyrillic code page, but fraudsters would replace latter н for example with Latin H because it looks the same. The ASCII code for Latin H is different to Cyrillic н so the auto-detection may miss the word, unless this replacement is made beforehand.

", "votes": null, "replies": [] }, { "id": 50793, "author_name": "apophenia", "author_url": "", "post_date": "07/17/2014 18:14:32", "content": "

[quote=Jano Petras;50773]

\n

are russian characters are getting converted to english character ?

\n

The issue with Russian/English - or Cyrillic/Latin characters is a technique fraudsters use to bypass automated word detection. These characters look the same in both alphabets, but their codes are different.

\n

So if a word in Cyrillic is written such as: Начните these would be all character codes from Cyrillic code page, but fraudsters would replace latter н for example with Latin H because it looks the same. The ASCII code for Latin H is different to Cyrillic н so the auto-detection may miss the word, unless this replacement is made beforehand.

\n

[/quote]

\n

In the `correctWord` function, I notice that eng2rus replacement is done only when there's more Cyrillic characters than Latin characters, and rus2eng replacement is done only when there's more Latin characters. Do you know why?

", "votes": null, "replies": [] }, { "id": 50794, "author_name": "phanivadrevu", "author_url": "", "post_date": "07/17/2014 18:37:58", "content": "

It was mentioned that the fraudsters use this technique to evade autoamatic detection systems. 

\n

Think about how it works. If I am a fraudster and I would like to sell some porn material which is illegal, I will try to use the word 'pоrn' instead of 'porn' in the description. Notice I used the Cyrillic 'о' instead of Latin 'o'. From the fraudster's perspective, its only enough to change as few characters with look-alike characters. Even one change should be good enough to escape detection. 

\n

Vice-versa follows for original Russian text that needs to be hidden. I am guessing that that's why we should convert all characters to the alphabet that has more characters originally.

\n

EDIT: I just realized that I explained the opposite thing. Sorry, I don't know why it is being done this way!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "49757": "", "49873": "", "50736": "", "50773": "", "50793": "", "50794": "" }, "source": "meta" }