# 机器学习评判指标

## 0.背景

• 准确度PR (Precision Recall)
• F测量
• [ ] MCC
• [ ] BM
• [ ] MK
• [ ] Gini系数
• ROC
• [ ] Z score
• AUC
• [ ] Cost Curve
• [ ] BLEU
• [ ] Matthews correlation coefficient
• [ ] METEOR
• [ ] Brier score
• [ ] NIST (metric)
• [ ] ROUGE (metric)
• [ ] Sørensen–Dice coefficient
• [ ] Uncertainty coefficient, aka Proficiency
• [ ] Word error rate (WER)

• true condition：列表示真实类别；predicted condition：行表示预测的类别；
• 真实正类=true positive+false negative；真实负类=false positive+true negative；
• 预测的正类=true positive+false positive； 预测的负类=false negative+true negative。

## 1. 不同指标的含义

### 1.1 accuracy&Precision Recall

• accuracy：（图0.1中ACC）即最常用的准确度，表示$\frac{所有预测对了的样本个数}{总的样本个数}$
• Precision：（图0.1中PPV），精确率，表示预测的正类中预测对的样本个数比例$\frac{true\, positive}{预测的正类}$
• Recall：（图0.1中TPR），召回率，表示真实正类中预测对的样本个数比例$\frac{true\, positive}{真实正类}$.

### 1.2 F measure&&G measure

#### 1.2.1 F measure

• 当F score为0的时候最差：即precision和recall中某个值或者都接近0，则该模型越差；
• 当F score为1的时候最好：即precision和recall同时越接近1则该模型越好。

ps：F1 score同样也被称为Sørensen–Dice coefficient或者说叫Dice similarity coefficient (DSC).

• $\beta=2$，则表示recall的影响要大于precision；
• $\beta=0.5$，则表示precision的影响要大于recall.

### 1.5 ROC

AUC：Aera under curve，即表示曲线下面积的意思

## 2. 不同指标之间的关系

### 2.3 AUC的探讨

1. [ROC绘制] .introduction-to-auc-and-roc
2. [F1] wiki.F1_score
3. [ROC] J. A. Hanley and B. J. McNeil. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 1982.
4. [ROC] Hanley J A, McNeil B J. A method of comparing the areas under receiver operating characteristic curves derived from the same cases[J]. Radiology, 1983, 148(3): 839-843.
5. [ROC] McNeil B J, Hanley J A. Statistical approaches to the analysis of receiver operating characteristic (ROC) curves[J]. Medical decision making, 1984, 4(2): 137-150.
6. [ROC] DeLong E R, DeLong D M, Clarke-Pearson D L. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach[J]. Biometrics, 1988: 837-845.
7. [ROC] Bradley, A. (1997). The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30, 1145{1159.
8. [ACC&ROC] Provost, F., Fawcett, T., & Kohavi, R. (1998). The case against accuracy estimation for comparing induction algorithms. Proceeding of the 15th International Conference on Machine Learning (pp. 445{453). Morgan Kaufmann, San Francisco, CA.
9. [ROC&CC] Chris Drummond and Robert C. Holte, ‘Explicitly representing expected cost: An alternative to roc representation’, in Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 198–207, (2000).
11. [AUC] Cortes, C., & Mohri, M. (2003). AUC optimization vs. error rate minimization. Neural Information Processing Systems 15 (NIPS). MIT Press
12. [ROC&CC] Drummond, C., & Holte, R. C. (2004). What ROC curves can't do (and cost curves can). ROCAI (pp. 19{26).
13. [ROC] Zhang, Jun; Mueller, Shane T. (2005). "A note on ROC analysis and non-parametric estimate of sensitivity". Psychometrika. 70: 203–212.
14. [ROC] Fan J, Upadhye S, Worster A. Understanding receiver operating characteristic (ROC) curves[J]. Canadian Journal of Emergency Medicine, 2006, 8(1): 19-20.
15. [ROC] Fawcett, Tom (2006). An Introduction to ROC Analysis. Pattern Recognition Letters. 27 (8): 861–874.
16. [PR&ROC] .The Relationship Between Precision-Recall and ROC Curves, Jesse Davis and Mark Goadrich, ICML 2006
17. [ROC] Brown C D, Davis H T. Receiver operating characteristics curves and related decision measures: A tutorial[J]. Chemometrics and Intelligent Laboratory Systems, 2006, 80(1): 24-38.
18. [ROC] Weng C G, Poon J. A new evaluation measure for imbalanced datasets[C]//Proceedings of the 7th Australasian Data Mining Conference-Volume 87. Australian Computer Society, Inc., 2008: 27-32.
19. [ROC] Powers, David M W (2011). Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness & Correlation . Journal of Machine Learning Technologies. 2 (1): 37–63.
20. [ROC] Flach P A. ROC analysis[M]//Encyclopedia of machine learning. Springer US, 2011: 869-875.
21. [ROC] Hernandez-Orallo, J. (2013). "ROC curves for regression". Pattern Recognition. 46 (12): 3395–3411 .
22. [ROC] .Using the Receiver Operating Characteristic (ROC) curve to analyze a classification model: A final note of historical interest. Department of Mathematics, University of Utah. Department of Mathematics, University of Utah. Retrieved May 25, 2017.
23. [CC] Drummond C, Holte R C. Cost curves: An improved method for visualizing classifier performance[J]. Machine learning, 2006, 65(1): 95-130.
posted @ 2017-11-22 18:09  仙守  阅读(5035)  评论(0编辑  收藏  举报