Here is a new, simple tutorial on how to evaluate the quality of a classifier. The attached doc shows you how to construct a confusion matrix, compute the precision, recall, and f1 scores for a classifier, and to construct a precision/recall chart in R to compare the relative strengths and weaknesses of different classifiers.
Granted, these measures are not perfect. Powers (2011), in the Journal of Machine Learning Technologies, advises that they should not be used without a clear understanding of the biases, especially considering the power of intelligent prediction vs. the power of the guess. However, they should provide a decent basis for practitioners to compare different classification strategies. (Notice that you don’t even need algorithms to do this… you can generate a confusion matrix from any plant operation or business activity where classification is performed!)