{ "id": 4858, "title": "1% prediction rate: what am I doing wrong?", "url": "/competitions/digit-recognizer/discussion/4858", "author_name": "gdowding", "post_date": "2013-06-16T22:47:22.160000", "votes": 1, "comment_count": 3, "views": null, "content": "

Hi,
\r\n
\r\nI followed this tutorial for using Pandas and scikit learn to do data recognition. I was expecting to get a reasonable result, to start with, but I am getting about 1% accuracy. It makes me think I am doing something wrong.

\r\n

http://scikit-learn.org/dev/auto_examples/plot_digits_classification.html#example-plot-digits-classification-py

\r\n


\r\nI created a classifier with 1/3 of the available training data.

\r\n
train_df = pd.read_csv('train.csv')\r\nclassifier = svm.SVC(gamma=0.001)\r\ntrain_size = train_df.shape[0] / 3\r\ntrain_start = 0\r\ntrain_set = train_df.ix[train_start:train_size]\r\nX = train_set.ix[0:, 1:]\r\ny = train_set.ix[0:, 0]\r\n%time classifier.fit(X, y)
\r\n

When I run the classifier with some of the reserved data, I get around 1% accuracy. It looks like the classifier is classifying most of the samples as 1.

\r\n
start = train_size + 500\r\nlength = 25\r\n#print test_data\r\ntest_slice = test_data.ix[start:start + length]\r\nexp_slice = expected.ix[start:start + length]\r\n#print test_slice\r\n%time predicted = classifier.predict(test_slice)\r\n\r\nprint zip(exp_slice, predicted)
\r\n

Here is the output

\r\n
CPU times: user 0.32 s, sys: 0.01 s, total: 0.33 s\r\nWall time: 0.78 s\r\n[(9, 1), (8, 1), (5, 1), (2, 1), (5, 1), (2, 1), (1, 1), (3, 1), (2, 1), (1, 1), (2, 1), (2, 1), (0, 1), (9, 1), (1, 1), (3, 1), (1, 1), (0, 1), (0, 1), (4, 1), (6, 1), (3, 1), (3, 1), (4, 1), (3, 1), (3, 1)]
\r\n


\r\nI feel like since the predictions are coming out as mostly 1's I am missing something pretty basic. Any help would be appreciated.

\r\n

\r\n

Thanks,

\r\n

George

", "messages": [ { "id": 32359, "postDate": "2013-10-16T10:08:22.250Z", "content": "

Hi theharshest,

\n

I think this is best understood pictorially in the link you provided. But if you prefer it in textual terms, this would be what I try to think.

\n

A low gamma (= far reached) makes it so that more similar/ambiguous cases will be considered same/similar. e.g. a very poorly written 7 which is almost like 1 in a digit recognition task will be considered as a 1.

\n

A high gamma (= close reached) makes it so that only very similar/unambiguous case gets considered same/similar. e.g. two similar 1 digits are considered same/similar, but the same very poorly written 7 which is almost like 1 above would not be considered to be 1 in this case, because its shape is much less like a 1, after all.

\n

 

\n

 

", "votes": 2 }, { "id": 26293, "postDate": "2013-06-27T12:49:50.907Z", "content": "

Hi George,

\r\n

I see that you're using a gamma of 0.001 for your RBF kernel SVM instance. My experience with this competition is that this gamma is too big (and I guess it's not converging and spiraling out).

\r\n

In fact, I verified with my SVM code (rank 206) that with a gamma of 0.001 accuracy drops to 11% immediately. You may like to search for different gamma values. Using a smaller dataset (2000 instances) should be good for the search initially.

\r\n

Concretely:

\r\n
    \r\n
  1. Create a smaller training set from the original training file
  2. Try out different combinations of C and gamma (for RBF kernels) on a 75% training and 25% testing\r\n
  3. Print out the results and observe how the gamma is good.
\r\n

I hope this helps.

", "votes": 2 }, { "id": 25772, "postDate": "2013-06-16T22:47:22.160Z", "content": "

Hi,
\r\n
\r\nI followed this tutorial for using Pandas and scikit learn to do data recognition. I was expecting to get a reasonable result, to start with, but I am getting about 1% accuracy. It makes me think I am doing something wrong.

\r\n

http://scikit-learn.org/dev/auto_examples/plot_digits_classification.html#example-plot-digits-classification-py

\r\n


\r\nI created a classifier with 1/3 of the available training data.

\r\n
train_df = pd.read_csv('train.csv')\r\nclassifier = svm.SVC(gamma=0.001)\r\ntrain_size = train_df.shape[0] / 3\r\ntrain_start = 0\r\ntrain_set = train_df.ix[train_start:train_size]\r\nX = train_set.ix[0:, 1:]\r\ny = train_set.ix[0:, 0]\r\n%time classifier.fit(X, y)
\r\n

When I run the classifier with some of the reserved data, I get around 1% accuracy. It looks like the classifier is classifying most of the samples as 1.

\r\n
start = train_size + 500\r\nlength = 25\r\n#print test_data\r\ntest_slice = test_data.ix[start:start + length]\r\nexp_slice = expected.ix[start:start + length]\r\n#print test_slice\r\n%time predicted = classifier.predict(test_slice)\r\n\r\nprint zip(exp_slice, predicted)
\r\n

Here is the output

\r\n
CPU times: user 0.32 s, sys: 0.01 s, total: 0.33 s\r\nWall time: 0.78 s\r\n[(9, 1), (8, 1), (5, 1), (2, 1), (5, 1), (2, 1), (1, 1), (3, 1), (2, 1), (1, 1), (2, 1), (2, 1), (0, 1), (9, 1), (1, 1), (3, 1), (1, 1), (0, 1), (0, 1), (4, 1), (6, 1), (3, 1), (3, 1), (4, 1), (3, 1), (3, 1)]
\r\n


\r\nI feel like since the predictions are coming out as mostly 1's I am missing something pretty basic. Any help would be appreciated.

\r\n

\r\n

Thanks,

\r\n

George

", "votes": 1 }, { "id": 32355, "postDate": "2013-10-16T09:35:24.967Z", "content": "

Hi,

\n

I'm also trying different values of gamma here, but don't understand its significance as such. From http://scikit-learn.org/stable/auto_examples/svm/plot_rbf_parameters.html, I can see that -

\n

the gamma parameter defines how far the influence of a single training example reaches, with low values meaning ‘far’ and high values meaning ‘close’

\n

But I don't understand what does that mean. Can you please explain?

\n

Thank you.

" } ], "comments": [ { "id": 32359, "author_name": "Log0", "author_url": "", "post_date": "2013-10-16T10:08:22.250000", "content": "

Hi theharshest,

\n

I think this is best understood pictorially in the link you provided. But if you prefer it in textual terms, this would be what I try to think.

\n

A low gamma (= far reached) makes it so that more similar/ambiguous cases will be considered same/similar. e.g. a very poorly written 7 which is almost like 1 in a digit recognition task will be considered as a 1.

\n

A high gamma (= close reached) makes it so that only very similar/unambiguous case gets considered same/similar. e.g. two similar 1 digits are considered same/similar, but the same very poorly written 7 which is almost like 1 above would not be considered to be 1 in this case, because its shape is much less like a 1, after all.

\n

 

\n

 

", "votes": 2, "replies": [] }, { "id": 26293, "author_name": "Log0", "author_url": "", "post_date": "2013-06-27T12:49:50.907000", "content": "

Hi George,

\r\n

I see that you're using a gamma of 0.001 for your RBF kernel SVM instance. My experience with this competition is that this gamma is too big (and I guess it's not converging and spiraling out).

\r\n

In fact, I verified with my SVM code (rank 206) that with a gamma of 0.001 accuracy drops to 11% immediately. You may like to search for different gamma values. Using a smaller dataset (2000 instances) should be good for the search initially.

\r\n

Concretely:

\r\n
    \r\n
  1. Create a smaller training set from the original training file
  2. Try out different combinations of C and gamma (for RBF kernels) on a 75% training and 25% testing\r\n
  3. Print out the results and observe how the gamma is good.
\r\n

I hope this helps.

", "votes": 2, "replies": [] }, { "id": 32355, "author_name": "theharshest", "author_url": "", "post_date": "2013-10-16T09:35:24.967000", "content": "

Hi,

\n

I'm also trying different values of gamma here, but don't understand its significance as such. From http://scikit-learn.org/stable/auto_examples/svm/plot_rbf_parameters.html, I can see that -

\n

the gamma parameter defines how far the influence of a single training example reaches, with low values meaning ‘far’ and high values meaning ‘close’

\n

But I don't understand what does that mean. Can you please explain?

\n

Thank you.

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "32359": "", "26293": "", "25772": "", "32355": "" } }