Скачать или смотреть Creating a Custom accuracy_score for Ordinal Classification Problems in Python

Discover how to implement a performance metric that incorporates ordinality in your classification tasks using a custom accuracy score in Python.
---
This video is based on the question https://stackoverflow.com/q/65872754/ asked by the user 't.pellegrom' ( https://stackoverflow.com/u/14208556/ ) and on the answer https://stackoverflow.com/a/65873113/ provided by the user 'XavierBrt' ( https://stackoverflow.com/u/8276531/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Make variation on accuracy score to incorporate ordinality

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Understanding and Solving Ordinal Classification Problems

In classification problems, we often rely on metrics like accuracy to gauge how well our models are performing. However, when dealing with ordinal data classes—where the classes have a meaningful order (like 1, 2, 3, 4, 5)—the regular accuracy score may not provide a complete picture.

The Problem at Hand

Let’s say you have a classification problem where your target variable has classes numbered from 1 to 5. The challenge here is to determine how well your model predicts these classes. In strictly accurate terms, a model predicting 2 for an actual 3 would be considered incorrect. However, since 2 is only one notch away from 3, this may not be a fair representation of the model's performance.

For example, consider the following data:

Actual Values (y_test): [1, 3, 3, 3, 5]

Predicted Values (y_pred): [5, 4, 3, 2, 1]

Using the default accuracy score will yield 0.2, which seems harsh for a model that predicted three values reasonably close to the actual classes. Instead, we want to create a modified version of the accuracy score that counts predictions that are within one class of the actual value as correct.

This gives us a more forgiving, yet still informative metric to assess model predictions.

Creating a Custom Accuracy Score

Here's how you can implement your own version of the accuracy score in Python, which we’ll refer to as accuracy_score_within1. This score will consider predictions that are within one unit of the actual values as correct classifications.

Code Implementation

Let’s walk through the code needed to create this custom accuracy score function.

[[See Video to Reveal this Text or Code Snippet]]

Explanation of the Code

Importing Libraries: We start by importing NumPy, which provides support for arrays and mathematical operations.

Function Definition: We define our function accuracy_score_within1 that takes two parameters: y_test (the actual values) and y_pred (the predicted values).

Calculating Differences: We compute the absolute difference between the actual and predicted values using NumPy arrays.

Correct Classification Logic: We iterate over the differences, marking them as 1 (correct) if the difference is 1 or less, and 0 (incorrect) otherwise.

Calculating the Score: Lastly, we compute the mean of the result list, which gives us our custom accuracy score.

Testing the Function

Now, let's see our function in action with the initial example:

[[See Video to Reveal this Text or Code Snippet]]

Expected Output

When you run the test case, you should get an output of 0.6, which reflects a more forgiving view of the model's performance, aligning better with our needs for understanding how “close” the predictions are to the actual values.

Conclusion

In conclusion, adapting the accuracy score to factor in ordinality is a great way to enhance the evaluation of models predicting ordinal classes. With our custom implementation in Python, you can readily assess your model's ability to predict values that may not be classically classified as correct but are still reasonably close.

This approach provides a more nuanced view of model performance, which can be incredibly useful in various domains, particularly where the cost of errors is significant.

Feel free to modify and integrate this function into your model evaluation process to yield more insightful assessments of your predictions!