Add chi square calculation function to math module. #69
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
qkaiser
added
enhancement
qkaiser
force-pushed
the
chisquare
branch
4 times, most recently
from
bec8800 to
96d7d42
Compare
|
We should update the benchmarks as well. I am curious how the throughput compares between the algorithms. |
qkaiser
force-pushed
the
chisquare
branch
2 times, most recently
from
b721665 to
816251a
Compare
Done ! Throughput is quite similar: |
|
|
When using version 1.65, the following error message is received when running cargo: > error: package `clap_lex v0.7.2` cannot be built because it requires > rustc 1.74 or newer, while the currently active rustc version is > 1.65.0 Either upgrade to rustc 1.74 or newer, or use cargo update -p > [email protected] --precise ver where `ver` is the latest version of > `clap_lex` supporting rustc 1.65.0 When using version 1.74 and statrs we get: > error: unsupported output in build script of libm v0.2.9: > cargo::rustc-check-cfg=cfg(assert_no_panic) Found a cargo::key=value > build directive which is reserved for future use. Either change the > directive to cargo:key=value syntax (note the single :) or upgrade your > version of Rust.
Add chi_square_probability function to math_tools module. This function returns the Chi Square distribution probability. Chi-square tests are effective for distinguishing compressed from encrypted data because they evaluate the uniformity of byte distributions more rigorously than Shannon entropy. In compressed files, bytes often cluster around certain values due to patterns that still exist (albeit less detectable), resulting in a non-uniform distribution. Encrypted data, by contrast, exhibits nearly perfect uniformity, as each byte value from 0–255 is expected to appear with almost equal frequency, making it harder to detect any discernible patterns. The chi-square distribution is calculated for the stream of bytes in the chunk and expressed as an absolute number and a percentage which indicates how frequently a truly random sequence would exceed the value calculated. The percentage is the only value that is of interest from unblob's perspective, so that's why we only return it. According to ent doc⁰: > We [can] interpret the percentage as the degree to which the > sequence tested is suspected of being non-random. If the percentage is > greater than 99% or less than 1%, the sequence is almost certainly not > random. If the percentage is between 99% and 95% or between 1% and 5%, > the sequence is suspect. Percentages between 90% and 95% and 5% and 10% > indicate the sequence is “almost suspect”. [0] - https://www.fourmilab.ch/random/
vlaci
approved these changes
Nov 6, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, unblob-native expose the
shannon_entropyto calculate entropy levels of chunks. While valid, this approach is limited in that we cannot rely on it to differentiate between compressed and encrypted data streams.An improved approach involves Chi-square tests. Chi-square tests are effective for distinguishing compressed from encrypted data because they evaluate the uniformity of byte distributions more rigorously than Shannon entropy.
In compressed files, bytes often cluster around certain values due to patterns that still exist (albeit less detectable), resulting in a non-uniform distribution. Encrypted data, by contrast, exhibits nearly perfect uniformity, as each byte value from 0–255 is expected to appear with almost equal frequency, making it harder to detect any discernible patterns.
According to ent: