Receiver Operator Characteristic (ROC)

A receiver operating characteristic curve, or ROC curve, is a graphical plot that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold is varied. The method was originally developed for operators of military radar receivers

Videos

Topic	Access
Train, Test, & Validation Data Sets (7 min)	https://www.youtube.com/watch?v=Zi-0rlM4RDs
Confusion Matrix (7 min)	https://www.youtube.com/watch?v=Kdsp6soqA7o

ROC and AUC, Clearly Explained (16 min) This is good 🙂	https://www.youtube.com/watch?v=4jRBRDbJemM
Evaluating Classifiers: Gains and Lift Charts ( 14 min)	https://www.youtube.com/watch?v=1dYOcDaDJLY

Articles

Medium	Topic	Access
Excercise	Understanding Confusion Matrix in R	https://www.datacamp.com/community/tutorials/confusion-matrix-calculation-r
Exercise with Answer	Model Evaluation	https://www.r-exercises.com/2016/12/02/model-evaluation-exercise-1/
Exercise with Answer	Model Evaluation	https://www.r-exercises.com/2016/12/22/model-evaluation-2/
Tutorial	Lift Charts Updated by IBM	https://www.ibm.com/docs/en/spss-statistics/28.0.0?topic=customers-cumulative-gains-lift-charts
Tutorial	Generate ROC Curve Charts for Print and Interactive Use	https://cran.r-project.org/web/packages/plotROC/vignettes/examples.html

From a confusion matrix

condition positive (P): the number of real positive cases
condition negative (N): the number of real negative cases

true positive (TP): True positive Prediction=A test result that correctly indicates the presence of a condition or characteristic
true negative (TN): True negative Prediction=A test result that correctly indicates the absence of a condition or characteristic
false positive (FP): False positive Prediction=A test result which wrongly indicates that a particular condition or attribute is present
false negative (FN): False negative Prediction=A test result which wrongly indicates that a particular condition or attribute is absent

sensitivity
sensitivity, recall, hit rate, or true positive rate (TPR): $\mathrm {TPR} ={\frac {\mathrm {Predicted TP} }{\mathrm {known P} }}={\frac {\mathrm {TP} }{\mathrm {P} }}={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FN} }}=1-\mathrm {FNR}$
specificity
specificity, selectivity or true negative rate (TNR): $\mathrm {TNR} ={\frac {\mathrm {TN} }{\mathrm {N} }}={\frac {\mathrm {TN} }{\mathrm {TN} +\mathrm {FP} }}=1-\mathrm {FPR}$; precision
precision or positive predictive value (PPV): $\mathrm {PPV} ={\frac {\mathrm {PredictedTP} }{\mathrm {PredictedP} }}={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FP} }}=1-\mathrm {FDR}$
negative predictive value (NPV): $\mathrm {NPV} ={\frac {\mathrm {TN} }{\mathrm {TN} +\mathrm {FN} }}=1-\mathrm {FOR}$
miss rate or false negative rate (FNR): $\mathrm {FNR} ={\frac {\mathrm {FN} }{\mathrm {P} }}={\frac {\mathrm {FN} }{\mathrm {FN} +\mathrm {TP} }}=1-\mathrm {TPR}$
fall-out or false positive rate (FPR): $\mathrm {FPR} ={\frac {\mathrm {FP} }{\mathrm {N} }}={\frac {\mathrm {FP} }{\mathrm {FP} +\mathrm {TN} }}=1-\mathrm {TNR}$
false discovery rate (FDR): $\mathrm {FDR} ={\frac {\mathrm {FP} }{\mathrm {FP} +\mathrm {TP} }}=1-\mathrm {PPV}$
false omission rate (FOR): $\mathrm {FOR} ={\frac {\mathrm {FN} }{\mathrm {FN} +\mathrm {TN} }}=1-\mathrm {NPV}$
Positive likelihood ratio (LR+): $\mathrm {LR+} ={\frac {\mathrm {TPR} }{\mathrm {FPR} }}$
Negative likelihood ratio (LR-): $\mathrm {LR-} ={\frac {\mathrm {FNR} }{\mathrm {TNR} }}$
prevalence threshold (PT): $\mathrm {PT} ={\frac {{\sqrt {\mathrm {TPR} (-\mathrm {TNR} +1)}}+\mathrm {TNR} -1}{(\mathrm {TPR} +\mathrm {TNR} -1)}}={\frac {\sqrt {\mathrm {FPR} }}{{\sqrt {\mathrm {TPR} }}+{\sqrt {\mathrm {FPR} }}}}$
threat score (TS) or critical success index (CSI): $\mathrm {TS} ={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FN} +\mathrm {FP} }}$

Prevalence

{\frac {\mathrm {P} }{\mathrm {P} +\mathrm {N} }}

rpp (Rate of Positive Predictions)

rpp = (tp+fp)/(tp+fn+fp+tn)

$rpp=\frac{TP+FP}{TP+TN+FP+FN}$

$rpp=\frac{TP+FP}{Total Number of predictions }$

When rpp is 0.1, the number of times we have had some positive prediction are 10% of the total(regardless of our goodness)

Accuracy

accuracy (ACC)

\mathrm {ACC} ={\frac {\mathrm {TP} +\mathrm {TN} }{\mathrm {P} +\mathrm {N} }}={\frac {\mathrm {TP} +\mathrm {TN} }{\mathrm {TP} +\mathrm {TN} +\mathrm {FP} +\mathrm {FN} }}

balanced accuracy (BA)

\mathrm {BA} ={\frac {TPR+TNR}{2}}

F1 score

is the harmonic mean of precision and sensitivity:

$\mathrm {F} _{1}=2\times {\frac {\mathrm {PPV} \times \mathrm {TPR} }{\mathrm {PPV} +\mathrm {TPR} }}={\frac {2\mathrm {TP} }{2\mathrm {TP} +\mathrm {FP} +\mathrm {FN} }}$

phi coefficient (φ or r_φ) or Matthews correlation coefficient (MCC)

$\mathrm {MCC} ={\frac {\mathrm {TP} \times \mathrm {TN} -\mathrm {FP} \times \mathrm {FN} }{\sqrt {(\mathrm {TP} +\mathrm {FP} )(\mathrm {TP} +\mathrm {FN} )(\mathrm {TN} +\mathrm {FP} )(\mathrm {TN} +\mathrm {FN} )}}}$

Fowlkes–Mallows index (FM)

$\mathrm {FM} ={\sqrt {{\frac {TP}{TP+FP}}\times {\frac {TP}{TP+FN}}}}={\sqrt {PPV\times TPR}}$

informedness or bookmaker informedness (BM)

$\mathrm {BM} =\mathrm {TPR} +\mathrm {TNR} -1$

markedness (MK) or deltaP (Δp)

$\mathrm {MK} =\mathrm {PPV} +\mathrm {NPV} -1$

Diagnostic odds ratio (DOR)

$\mathrm {DOR} ={\frac {\mathrm {LR+} }{\mathrm {LR-} }}$

Sources: Fawcett (2006),^[2] Piryonesi and El-Diraby (2020),^[3] Powers (2011),^[4] Ting (2011),^[5] CAWCR,^[6] D. Chicco & G. Jurman (2020, 2021),^[7]^[8] Tharwat (2018).^[9]

^{==================================================================================}

#To create a confusion matrix with complete analysis of measures

library(caret)

^{KNNconfusionMatrix<-caret::confusionMatrix(Predictionsvector,actualvector)

KNNconfusionMatrix}

^{#Easiest and best visualization gain and lift in R:}

#install.packages(‘CustomerScoringMetrics’)
library(CustomerScoringMetrics)
CustomerScoringMetrics::cumGainsChart(Predictionsvector,actualvector)
CustomerScoringMetrics::liftChart(Predictionsvector,actualvector)

# ROC gain lift With more elaboration in R

##################################################
# Create a prediction object from predictions and actuals of any test data
# Then by Various performance analysis you will have ROC , cumulative gain and lift charts
# rpp = (tp+fp)/(tp+fn+fp+tn) (Rate of Positive Predictions)
#################################################
ROCRpredictionObjfromAnyModel<-ROCR::prediction(as.numeric(PredictionsfromAnyTestset),as.numeric(ActualsfromAnyTestset))
plotableROC<- ROCR::performance(ROCRpredictionObjfromAnyModel,measure=”tpr”,x.measure=”fpr”)
plot(plotableROC, col=”orange”, lwd=2, main=”ROC curve for blah blah”)

plotableGain<- ROCR::performance(ROCRpredictionObjfromAnyModel,measure=”tpr”,x.measure=”rpp”)
plot(plotableGain, col=”orange”, lwd=2, main=”Gaincurve for blah blah”)
#################################################
# For example we will create a prediction object from KNN model prediction vector
#####################################################

ROCRpredictionObjfromKNN<-ROCR::prediction(as.numeric(amirKNNextraxtedPredictionsvector),as.numeric(actualvectoradmitancefromtestdata))

# Now we will use the prediction object we created above as a parameter that we pass to performance function
# this performance object will contain tpr and fpr
# ROC Chart:
ROCKNN <- ROCR::performance(ROCRpredictionObjfromKNN,measure=”tpr”,x.measure=”fpr”)

# if we plot tpr vs fpr it is ROC by definition 🙂
plot(ROCKNN, col=”orange”, lwd=2, main=”ROC curve ROCRpredictionObjKNN”)

# Now we will use the prediction object we created above as a parameter that we pass to performance function
# this performance object will contain tpr and rpp (Rate of Positive Predictions)

# if we plot tpr vs rpp it is Gains chart by definition 🙂
# Gains Chart:
gainKNN <- ROCR::performance(ROCRpredictionObjfromKNN,measure=”tpr”,x.measure=”rpp”)
plot(gainKNN, col=”orange”, lwd=2, main=”gain curve KNN”)

# Lift Chart:
liftchartKNN<-ROCR::performance(ROCRpredictionObjfromKNN,”lift”,”rpp”)
plot(liftchartKNN, main=”Lift curve KNN”, colorize=T)

#AUC
library(ROCit)
roc_empirical <- ROCit::rocit(score = as.numeric(amirKNNextraxtedPredictionsvector), class = as.numeric(actualvectoradmitancefromtestdata), negref = 1)
# to get the area under the curve = AUC
summary(roc_empirical)
plot(roc_empirical)

=============================================

http://mlwiki.org/index.php/Cumulative_Gain_Chart

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4608333/

What is a gain chart?

https://idswater.com/2020/09/19/what-is-a-gain-chart/

http://www2.cs.uregina.ca/~dbd/cs831/notes/lift_chart/lift_chart.html

AUC: https://www.r-bloggers.com/2016/11/calculating-auc-the-area-under-a-roc-curve/

https://www.rdocumentation.org/packages/caret/versions/5.07-001/topics/predict.train

https://www.datanovia.com/en/lessons/determining-the-optimal-number-of-clusters-3-must-know-methods/

https://cran.r-project.org/web/packages/ROCit/index.html

=================================

https://en.wikipedia.org/wiki/Confusion_matrix

Other metrics can be included in a confusion matrix, each of them having their significance and use.

		Predicted condition		^Sources:^[22]^[23]^[24]^[25]^[26]^[27]^[28]^[29]^[30] view talk edit
	Total population = P + N	Positive (PP)	Negative (PN)	Informedness, bookmaker informedness (BM) = TPR + TNR − 1	Prevalence threshold (PT) = ${\mathsf {\tfrac {{\sqrt {{\text{TPR}}\times {\text{FPR}}}}-{\text{FPR}}}{{\text{TPR}}-{\text{FPR}}}}}$
Actual condition	Positive (P)	True positive (TP), hit	False negative (FN), type II error, miss, underestimation	True positive rate (TPR), recall, sensitivity (SEN), probability of detection, hit rate, power = TP/P= 1 − FNR	False negative rate (FNR), miss rate = FN/P= 1 − TPR
Actual condition	Negative (N)	False positive (FP), type I error, false alarm, overestimation	True negative (TN), correct rejection	False positive rate (FPR), probability of false alarm, fall-out = FP/N= 1 − TNR	True negative rate (TNR), specificity (SPC), selectivity = TN/N= 1 − FPR
	Prevalence = P/P + N	Positive predictive value (PPV), precision = TP/PP= 1 − FDR	False omission rate (FOR) = FN/PN= 1 − NPV	Positive likelihood ratio (LR+) = TPR/FPR	Negative likelihood ratio (LR−) = FNR/TNR
	Accuracy (ACC) = TP + TN/P + N	False discovery rate (FDR) = FP/PP= 1 − PPV	Negative predictive value (NPV) = TN/PN= 1 − FOR	Markedness (MK), deltaP (Δp) = PPV + NPV − 1	Diagnostic odds ratio (DOR) = LR+/LR−
	Balanced accuracy (BA) = TPR + TNR/2	F₁ score = 2 PPV × TPR/PPV + TPR = 2 TP/2 TP + FP + FN	Fowlkes–Mallows index (FM) = $\scriptstyle {\mathsf {\sqrt {{\text{PPV}}\times {\text{TPR}}}}}$	Matthews correlation coefficient (MCC) = $\scriptstyle {\mathsf {\sqrt {{\text{TPR}}\times {\text{TNR}}\times {\text{PPV}}\times {\text{NPV}}}}}$ $\scriptstyle -{\mathsf {\sqrt {{\text{FNR}}\times {\text{FPR}}\times {\text{FOR}}\times {\text{FDR}}}}}$	Threat score (TS), critical success index (CSI), Jaccard index = TP/TP + FN + FP

Confusion matrices with more than two categories

https://stackoverflow.com/questions/31324218/scikit-learn-how-to-obtain-true-positive-true-negative-false-positive-and-fal

Confusion matrix is not limited to binary classification and can be used in multi-class classifiers as well.^[31] The confusion matrices discussed above have only two conditions: positive and negative. For example, the table below summarizes communication of a whistled language between two speakers, zero values omitted for clarity.^[32]

Perceived vowel Vowel produced	i	a	o	u
i	15	1
e	1	1
a		79	5
o		4	15	3
u			2	2

Demosophy.org

Evaluation of Classification Models: Confusion Matrix, ROC , AUC ,Gains ,and Lift Charts