Accuracy measures

Selecting the best model

Once we develop the models, we need to check whether the model is perfect for the data. That is why we use a lot of statistical ways to select the best model. Some of the most commonly and widely used measures are listed below.

Coefficient of determination

R-squared estimates the relationship between the movements of a dependent variable and the movements of an independent variable. If the R-squared value is 0.89, it means 89% of the variance of the dependent variable is exemplified by the variance of the independent variable. The regression model can be evaluated using the R-squared value. You can use error measures instead of  R-squared values.

Error Measures

• MAE - Mean Absolute Error

• MAPE - Mean Absolute Percentage Error

• MPE - Mean Percentage Error

• RMSE - Root Mean Squared Error

• MSE - Mean Scaled Error

• MASE - Mean Absolute Scaled Error

• Scale-dependent method, such as mean absolute error (MAE). Although these are simple to compute, they cannot be used to compare different series due to scale dependence. Scale-dependent means that the error statistics represent the underlying data units (e.g., Money, inches, etc.).The fundamental advantage of scale-dependent metrics is that they are usually simple to calculate and comprehend.
• Mean Absolute Percent Error (MAPE), for example, can be used to evaluate two series because it is scale-independent. However, we can't use them if the data contains zeros. It is called as percentage error method.
• MAE and RMSE are the most widely used error measures to select the better model for the data.

All these measures are used to observe the error (which means the difference between the actual value and the predicted value) from the model, these error measures will help us to compare one model with another model and then select the best model with the lowest error measures. The famous accuracy measures which are used to evaluate the machine learning models are listed below:

Confusion matrix:

A confusion matrix is a table-like summary that is used to define how well a classification method performs. It is mostly used to check the classification algorithm's performance. The format of the confusion matrix is shown in the below image. 

There are many measurements in the confusion matrix, and some of them are mentioned below with the formula.

Here,   P = TP + FN;   N = FP + TN;   PP = TP + FP;  PN = FN + TN

All the above-mentioned are measurements that can be calculated by the confusion matrix. By these measurements, we can understand model performance.

Gain and lift chart

          The gain chart and lift chart are two components used in commercial situations such as target marketing to quantify the benefits of adopting the model. It's not only limited to marketing analysis. It may also be applied in other areas such as risk modelling, supply chain analytics, and so on. It helps to evaluate, how the model is good compared to the random model. If the model line is higher when compared to the random line then the model is a better one.

ROC Curve

The ROC (Receiver Operating Characteristic) curve is a chart that shows the performance of a classification model at all classification thresholds. In this curve, it has two parameters True positive rate and false positive rate. The ROC curve that falls on the diagonal line depicts the results of a diagnostic test. i.e. a test that produces positive or negative findings that are unrelated to the Underlying true medical condition.

In Receiver Operating Characteristic (ROC) analysis, the Area Under the Curve (AUC) is a statistic used to assess how well a binary classification model can distinguish between positive and negative classes. AUC values typically represent how well the model distinguishes between the two classes. 

• AUC = 0.5 denotes how well the model approximates a random chance, while AUC values higher than 0.5 but below 1 suggest superior discriminatory power.
• A classifier that perfectly distinguishes between classes without producing any false positives or negatives has an AUC of 1, which denotes perfection. 
• AUC = 1 denotes a flawless classifier, demonstrating the model's error-free capacity to differentiate between positive and negative classifications.

Models can also be chosen by using the Criterion, which makes it simple to choose the model with the lowest value. The criterions are the Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC).