Скачать или смотреть Solving the Mystery of Shannon's Entropy Algorithm Returning Negative Values

Discover why Shannon's Entropy algorithm in C might return negative values and how to fix it with a clear, step-by-step solution.
---
This video is based on the question https://stackoverflow.com/q/62616994/ asked by the user 'orORorOR' ( https://stackoverflow.com/u/13765694/ ) and on the answer https://stackoverflow.com/a/62617032/ provided by the user 'Eric Postpischil' ( https://stackoverflow.com/u/298225/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Shannon's Entropy Algorithm Returning negative values

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Understanding the Issue with Shannon's Entropy Algorithm

If you’re working on an algorithm that involves data analysis or information theory, you might have come across Shannon's Entropy. This mathematical concept helps in measuring the uncertainty or randomness of a set of values. However, you may encounter a frustrating issue where your algorithm returns negative values, which is mathematically incorrect.

This guide aims to clarify this problem and guide you through the solution with a comprehensive explanation.

The Problem at Hand

You are using C to implement Shannon's Entropy from scratch, and you've encountered unexpected outcomes:

When your map contains three occurrences of the value 15, the entropy value returned is -4.214...

When the map has three occurrences of 15 and one occurrence of 25, the entropy value is 0.000.

Clearly, something is amiss in the calculation. Your implementation should not produce negative entropy or zeros under normal circumstances.

Why Are You Getting Negative Values?

The issue lies within the probability calculation of the logarithmic term in your algorithm. Here's a closer look at the problematic part of your code:

[[See Video to Reveal this Text or Code Snippet]]

Understanding the Calculation

The variable x is intended to represent the probability of occurrence for a specific item, calculated as:

The number of times the item occurs (map->entry[i].occ)

Divided by the total occurrences of items (this should be the total of all occurrences, not the count of distinct values).

However, in your case, map->entry_count refers to the number of distinct values, leading to incorrect probability values greater than 1. This is a fundamental mistake because probabilities should always be between 0 and 1.

Example to Illustrate the Problem

Let’s consider an example with three different events (A, B, C) occurring 10, 13, and 17 times respectively. The correct probabilities should be calculated as follows:

Total occurrences = 10 + 13 + 17 = 40

Probability of A = 10 / 40 = 0.25

Probability of B = 13 / 40 = 0.325

Probability of C = 17 / 40 = 0.425

However, with your current method, if you replace the total occurrences with the count of distinct values (3), you’ll end up with probabilities:

Probability of A = 10 / 3 ≈ 3.33 (infeasible as it exceeds 1)

This calculation yields invalid results and can lead to negative values in your entropy computation.

The Solution

To fix this issue, follow these steps to correct your entropy calculation:

Calculate Total Occurrences: First, ensure you compute the total occurrences of all values in your map correctly.

Correct the Probability Calculation: Instead of dividing by the number of distinct values, divide by the total occurrences:

[[See Video to Reveal this Text or Code Snippet]]

Implement the Logarithm: Now, you can safely compute the entropy:

[[See Video to Reveal this Text or Code Snippet]]

Return the Final Entropy: Finally, you can return the computed entropy value.

Conclusion

With these changes, your algorithm should produce valid, non-negative entropy values, allowing you to fully leverage the power of Shannon's Entropy in your project.

Feel free to revisit the core ideas in probability and logarithmic functions if you need a deeper understanding. Remember, accurate calculations are key in programming, especially when dealing with complex mathematical concepts!

By addressing the underlying mistake in the probability calculation, you can enhance the reliability of your Shannon’s Entropy implementation.