Compute and expose χ² probability in EntropyReport #995
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
qkaiser
force-pushed
the
feat-chisquare-entropy
branch
from
65c20e2 to
aeaa3d2
Compare
qkaiser
changed the title
qkaiser
changed the title
qkaiser
force-pushed
the
feat-chisquare-entropy
branch
from
aeaa3d2 to
0459ac8
Compare
|
Entropy levels for a plaintext lorem ipsum:
Entropy levels of that file, now XOR'ed:
Entropy levels of that plaintext file, now gzip'ed:
Entropy levels of that plaintext file, now AES encrypted:
The χ² probability convey more precise information about the nature of analyzed data. We can spot weak encryption like XOR even when Shannon entropy reports constant high levels of entropy. We can differentiate between compressed and encrypted given the χ² probability distribution, while they share similar Shannon entropy levels. |
qkaiser
force-pushed
the
feat-chisquare-entropy
branch
from
0459ac8 to
b0228db
Compare
qkaiser
force-pushed
the
feat-chisquare-entropy
branch
from
b0228db to
448c685
Compare
e3krisztian
requested changes
Oct 29, 2024
There was a problem hiding this comment.
I would prefer a wording change both in the commit message and in sources to use randomness or randomness measure instead of entropy.
In the commit message of 448c685 chi^2 probability is categorized as a kind of entropy, probably because as we already have entropy as a randomness measure.
qkaiser
force-pushed
the
feat-chisquare-entropy
branch
2 times, most recently
from
93aec8d to
d123265
Compare
qkaiser
force-pushed
the
feat-chisquare-entropy
branch
2 times, most recently
from
cceeb8f to
42ad2a6
Compare
e3krisztian
reviewed
Nov 6, 2024
qkaiser
force-pushed
the
feat-chisquare-entropy
branch
3 times, most recently
from
3ed5c17 to
aaa7361
Compare
|
@e3krisztian now using unblob-native version 0.1.5 that was released today. Ready to be merged I think. |
We should update the required version as well to 0.1.5, as that is what matters when people install unblob from pypi |
qkaiser
force-pushed
the
feat-chisquare-entropy
branch
3 times, most recently
from
0b9cb0d to
7f034e2
Compare
|
@vlaci done. also I hate this caret notation. |
…sReport Introduce another randomness measure based on Chi Square probability by using unblob-native's chi_square_probability function. This function returns the Chi Square distribution probability. Chi-square tests are effective for distinguishing compressed from encrypted data because they evaluate the uniformity of byte distributions more rigorously than Shannon entropy. In compressed files, bytes often cluster around certain values due to patterns that still exist (albeit less detectable), resulting in a non-uniform distribution. Encrypted data, by contrast, exhibits nearly perfect uniformity, as each byte value from 0–255 is expected to appear with almost equal frequency, making it harder to detect any discernible patterns. The chi-square distribution is calculated for the stream of bytes in the chunk and expressed as an absolute number and a percentage which indicates how frequently a truly random sequence would exceed the value calculated. The percentage is the only value that is of interest from unblob's perspective, so that's why we only return it. According to ent doc⁰: > We [can] interpret the percentage as the degree to which the > sequence tested is suspected of being non-random. If the percentage is > greater than 99% or less than 1%, the sequence is almost certainly not > random. If the percentage is between 99% and 95% or between 1% and 5%, > the sequence is suspect. Percentages between 90% and 95% and 5% and 10% > indicate the sequence is “almost suspect”. [0] - https://www.fourmilab.ch/random/ This randomness measure is introduced by modifying the EntropyReport class so that it contains two RandomnessMeasurements: - shannon: for Shannon entropy, which was already there - chi_square: for Chi Square probability, which we introduce EntropyReport is renamed to RandomnessReport to reflect that all measurements are not entropy related. The format_entropy_plot has been adjusted to display two lines within the entropy graph. One for Shannon, the other for Chi Square. This commit breaks the previous API by converting entropy_depth and entropy_plot to randomness_depth and randomness_plot in ExtractionConfig. The '--entropy-depth' CLI option is replaced by '--randomness-depth'.
qkaiser
force-pushed
the
feat-chisquare-entropy
branch
from
7f034e2 to
8e2e11b
Compare
e3krisztian
approved these changes
Nov 8, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduce another entropy measure based on Chi Square probability by using unblob-native's chi_square_probability function. This function returns the Chi Square distribution probability.
Chi-square tests are effective for distinguishing compressed from encrypted data because they evaluate the uniformity of byte distributions more rigorously than Shannon entropy.
In compressed files, bytes often cluster around certain values due to patterns that still exist (albeit less detectable), resulting in a non-uniform distribution. Encrypted data, by contrast, exhibits nearly perfect uniformity, as each byte value from 0–255 is expected to appear with almost equal frequency, making it harder to detect any discernible patterns.
The chi-square distribution is calculated for the stream of bytes in the chunk and expressed as an absolute number and a percentage which indicates how frequently a truly random sequence would exceed the value calculated. The percentage is the only value that is of interest from unblob's perspective, so that's why we only return it.
According to ent doc⁰:
[0] - https://www.fourmilab.ch/random/
This entropy measure is introduced by modifying the
EntropyReportclass so that it contains twoEntropyMeasures:The
format_entropy_plothas been adjusted to display two lines within the entropy graph. One for Shannon, the other for Chi Square.