There have been a few questions about the competition. I try to answer them all in this thread so that everyone is at the same level.
\n\nQ: How can I use the training data in practice?
\n\nA: You need to generate pairs of matching and non-matching images. The absolute differences of the feature vectors can be the input for the classifier.
\n\nQ: If I sample training data at random, I get hugely unbalanced data (less than 1% matches and over 99% non-matches). Is this a problem?
\n\nA: This will most likely result in a dummy classifier that always predicts a non-match (and be 99% accurate). Solutions to this is to develop some procedure for sampling about 50% of both. Alternatively, your classifier may support class-specific weights. Then you can set the match-class weights 100 times larger, for example.
\n\nQ: I train an SVM but the result is worse than the sample solution.
\n\nA: Your classifier probably overfits to the training set. Either try to generate more data (see previous question, though), or regularize your solution. For example, Matlab's fitcsvm function has the parameter BoxConstraint, which should improve generalization and thus the accuracy as well. For Python's sklearn package, the corresponding regularizer is called C. Also LibSVM has the parameter -c for regularization.
", "messages": [ { "id": "98979", "postDate": "11/17/2015 16:05:39", "content": "There have been a few questions about the competition. I try to answer them all in this thread so that everyone is at the same level.
\n\nQ: How can I use the training data in practice?
\n\nA: You need to generate pairs of matching and non-matching images. The absolute differences of the feature vectors can be the input for the classifier.
\n\nQ: If I sample training data at random, I get hugely unbalanced data (less than 1% matches and over 99% non-matches). Is this a problem?
\n\nA: This will most likely result in a dummy classifier that always predicts a non-match (and be 99% accurate). Solutions to this is to develop some procedure for sampling about 50% of both. Alternatively, your classifier may support class-specific weights. Then you can set the match-class weights 100 times larger, for example.
\n\nQ: I train an SVM but the result is worse than the sample solution.
\n\nA: Your classifier probably overfits to the training set. Either try to generate more data (see previous question, though), or regularize your solution. For example, Matlab's fitcsvm function has the parameter BoxConstraint, which should improve generalization and thus the accuracy as well. For Python's sklearn package, the corresponding regularizer is called C. Also LibSVM has the parameter -c for regularization.
", "rawMarkdown": "There have been a few questions about the competition. I try to answer them all in this thread so that everyone is at the same level.\r\n\r\nQ: How can I use the training data in practice?\r\n\r\nA: You need to generate pairs of matching and non-matching images. The absolute differences of the feature vectors can be the input for the classifier.\r\n\r\nQ: If I sample training data at random, I get hugely unbalanced data (less than 1% matches and over 99% non-matches). Is this a problem?\r\n\r\nA: This will most likely result in a dummy classifier that always predicts a non-match (and be 99% accurate). Solutions to this is to develop some procedure for sampling about 50% of both. Alternatively, your classifier may support class-specific weights. Then you can set the match-class weights 100 times larger, for example.\r\n\r\nQ: I train an SVM but the result is worse than the sample solution.\r\n\r\nA: Your classifier probably overfits to the training set. Either try to generate more data (see previous question, though), or regularize your solution. For example, [Matlab's][1] fitcsvm function has the parameter BoxConstraint, which should improve generalization and thus the accuracy as well. For Python's [sklearn][2] package, the corresponding regularizer is called C. Also [LibSVM][3] has the parameter -c for regularization.\r\n\r\n\r\n [1]: http://se.mathworks.com/help/stats/fitcsvm.html#namevaluepairarguments\r\n [2]: http://scikit-learn.org/stable/modules/svm.html\r\n [3]: https://www.csie.ntu.edu.tw/~cjlin/libsvm/", "votes": null }, { "id": "99023", "postDate": "11/18/2015 11:56:45", "content": "More Q&A:
\n\nQ: SVM gives me only -1 and 1, not probabilities?
\n\nA: In Matlab, you can get probability estimates with fitPosterior. With LibSVM, use the -b switch (documented at LibSVM website). In sklearn, initialize with probability switch: clf = SVC(probability = True). Alternatively, you can look into Logistic Regression classifier, which returns probabilities by default. Implementations include lassoglm, liblinear or sklearn.
Q: Which values to use for the regularization parameter (BoxConstraint or C)?
\n\nA: Use a geometric range, such as 10 .^ (-5 : 5) instead of e.g., 0.1 : 0.001 : 1.
Q: How do I compare different C values?
\n\nA: Split your training data to training and testing, and check how well different versions work with the test part when trained with the train part. More advanced approaches involve cross validation. See e.g., Matlab example and Python example.
\n\nQ: Python seems so much nicer than Matlab :) How to get hold of it?
\n\nA: Anaconda distribution is probably the easiest starting point and includes sklearn package by default. At TUT, everything is readily installed at classroom TC407. Just go there and run python deepfeatures_distance_prediction.py of the other thread after setting up all data. My slides from last summer may help you to get started.
More Q&A:
\n\nQ: SVM gives me only -1 and 1, not probabilities?
\n\nA: In Matlab, you can get probability estimates with fitPosterior. With LibSVM, use the -b switch (documented at LibSVM website). In sklearn, initialize with probability switch: clf = SVC(probability = True). Alternatively, you can look into Logistic Regression classifier, which returns probabilities by default. Implementations include lassoglm, liblinear or sklearn.
Q: Which values to use for the regularization parameter (BoxConstraint or C)?
\n\nA: Use a geometric range, such as 10 .^ (-5 : 5) instead of e.g., 0.1 : 0.001 : 1.
Q: How do I compare different C values?
\n\nA: Split your training data to training and testing, and check how well different versions work with the test part when trained with the train part. More advanced approaches involve cross validation. See e.g., Matlab example and Python example.
\n\nQ: Python seems so much nicer than Matlab :) How to get hold of it?
\n\nA: Anaconda distribution is probably the easiest starting point and includes sklearn package by default. At TUT, everything is readily installed at classroom TC407. Just go there and run python deepfeatures_distance_prediction.py of the other thread after setting up all data. My slides from last summer may help you to get started.