-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #52 from intel/update_dic_12_2024
update_dic_12_2024
- Loading branch information
Showing
48 changed files
with
2,844 additions
and
150 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
170 changes: 170 additions & 0 deletions
170
docs/AppNotes/CompressAndVerify/checksum_algorithms.html
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,170 @@ | ||
| <!DOCTYPE html> | ||
| <html class="writer-html5" lang="en" > | ||
| <head> | ||
| <meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" /> | ||
|
|
||
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> | ||
| <title>Summary of Checksum Algorithms — Intel® QuickAssist Technology documentation</title> | ||
| <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" /> | ||
| <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" /> | ||
| <link rel="stylesheet" href="../../_static/copybutton.css" type="text/css" /> | ||
| <link rel="stylesheet" href="../../_static/css/qat.css" type="text/css" /> | ||
| <!--[if lt IE 9]> | ||
| <script src="../../_static/js/html5shiv.min.js"></script> | ||
| <![endif]--> | ||
|
|
||
| <script data-url_root="../../" id="documentation_options" src="../../_static/documentation_options.js"></script> | ||
| <script src="../../_static/jquery.js"></script> | ||
| <script src="../../_static/underscore.js"></script> | ||
| <script src="../../_static/_sphinx_javascript_frameworks_compat.js"></script> | ||
| <script src="../../_static/doctools.js"></script> | ||
| <script src="../../_static/clipboard.min.js"></script> | ||
| <script src="../../_static/copybutton.js"></script> | ||
| <script src="../../_static/js/theme.js"></script> | ||
| <link rel="index" title="Index" href="../../genindex.html" /> | ||
| <link rel="search" title="Search" href="../../search.html" /> | ||
| <link rel="next" title="Intel® QAT Compress and Verify Tests" href="cnv_tests.html" /> | ||
| <link rel="prev" title="Checksum Discussion" href="checksums.html" /> | ||
| </head> | ||
|
|
||
| <body class="wy-body-for-nav"> | ||
| <div class="wy-grid-for-nav"> | ||
| <nav data-toggle="wy-nav-shift" class="wy-nav-side"> | ||
| <div class="wy-side-scroll"> | ||
| <div class="wy-side-nav-search" > | ||
|
|
||
|
|
||
|
|
||
| <a href="../../index.html" class="icon icon-home"> | ||
| Intel® QuickAssist Technology | ||
| </a> | ||
| <div class="version"> | ||
| Hardware Version 2.0 | ||
| </div> | ||
| <div role="search"> | ||
| <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get"> | ||
| <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" /> | ||
| <input type="hidden" name="check_keywords" value="yes" /> | ||
| <input type="hidden" name="area" value="default" /> | ||
| </form> | ||
| </div> | ||
| </div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu"> | ||
| <ul class="current"> | ||
| <li class="toctree-l1"><a class="reference internal" href="../../qat_general/legal.html">Legal Notices & Disclaimers</a></li> | ||
| <li class="toctree-l1"><a class="reference internal" href="../../Intro/introduction.html">Introduction</a></li> | ||
| <li class="toctree-l1"><a class="reference internal" href="../../Intro/terminology.html">Terminology</a></li> | ||
| <li class="toctree-l1"><a class="reference internal" href="../../RN/In-Tree/index.html">Release Notes - In-Tree</a></li> | ||
| <li class="toctree-l1"><a class="reference internal" href="../../RN/Linux/2.X/index.html">Release Notes - Linux*</a></li> | ||
| <li class="toctree-l1"><a class="reference internal" href="../../RN/VMware/2.X/index.html">Release Notes - VMware*</a></li> | ||
| <li class="toctree-l1"><a class="reference internal" href="../../GSG/2.X/index.html">Getting Started Guide</a></li> | ||
| <li class="toctree-l1"><a class="reference internal" href="../../PG/index.html">Programmer’s Guide</a></li> | ||
| <li class="toctree-l1"><a class="reference internal" href="../../VIRT/index.html">Virtualization Deployment Guide</a></li> | ||
| <li class="toctree-l1"><a class="reference internal" href="../../PERF/index.html">Performance Optimization Guide</a></li> | ||
| <li class="toctree-l1"><a class="reference internal" href="../../qatlib/index.html">QATlib User’s Guide</a></li> | ||
| <li class="toctree-l1 current"><a class="reference internal" href="../index.html">Application Notes</a><ul class="current"> | ||
| <li class="toctree-l2"><a class="reference internal" href="../Containers/index.html">Intel<sup>®</sup> QAT in Linux Containers</a></li> | ||
| <li class="toctree-l2"><a class="reference internal" href="../WhenToQAT/index.html">When to Use Intel<sup>®</sup> QAT</a></li> | ||
| <li class="toctree-l2 current"><a class="reference internal" href="index.html">Intel® QAT Compress and Verify White Paper</a><ul class="current"> | ||
| <li class="toctree-l3"><a class="reference internal" href="introduction.html">Introduction</a></li> | ||
| <li class="toctree-l3"><a class="reference internal" href="checksums.html">Checksum Discussion</a></li> | ||
| <li class="toctree-l3 current"><a class="current reference internal" href="#">Summary of Checksum Algorithms</a></li> | ||
| <li class="toctree-l3"><a class="reference internal" href="cnv_tests.html">Intel® QAT Compress and Verify Tests</a></li> | ||
| <li class="toctree-l3"><a class="reference internal" href="conclusion.html">Conclusion</a></li> | ||
| <li class="toctree-l3"><a class="reference internal" href="notes.html">The CRC64 Values Used in End-To-End mode</a></li> | ||
| <li class="toctree-l3"><a class="reference internal" href="notes.html#the-empirical-study">The Empirical Study</a></li> | ||
| <li class="toctree-l3"><a class="reference internal" href="references.html">Footnotes</a></li> | ||
| <li class="toctree-l3"><a class="reference internal" href="revision_history.html">Revision History</a></li> | ||
| </ul> | ||
| </li> | ||
| </ul> | ||
| </li> | ||
| <li class="toctree-l1"><a class="reference internal" href="../../API_PG/index.html">API Programmer’s Guide</a></li> | ||
| <li class="toctree-l1"><a class="reference internal" href="../../qat_general/contact.html">Contact & Support</a></li> | ||
| <li class="toctree-l1"><a class="reference internal" href="../../qat_general/collateral_list.html">Documentation & Resources</a></li> | ||
| </ul> | ||
|
|
||
| </div> | ||
| </div> | ||
| </nav> | ||
|
|
||
| <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" > | ||
| <i data-toggle="wy-nav-top" class="fa fa-bars"></i> | ||
| <a href="../../index.html">Intel® QuickAssist Technology</a> | ||
| </nav> | ||
|
|
||
| <div class="wy-nav-content"> | ||
| <div class="rst-content"> | ||
| <div role="navigation" aria-label="Page navigation"> | ||
| <ul class="wy-breadcrumbs"> | ||
| <li><a href="../../index.html" class="icon icon-home" aria-label="Home"></a></li> | ||
| <li class="breadcrumb-item"><a href="../index.html">Application Notes</a></li> | ||
| <li class="breadcrumb-item"><a href="index.html">Intel® QAT Compress and Verify White Paper</a></li> | ||
| <li class="breadcrumb-item active">Summary of Checksum Algorithms</li> | ||
| <li class="wy-breadcrumbs-aside"> | ||
| </li> | ||
| </ul> | ||
| <hr/> | ||
| </div> | ||
| <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> | ||
| <div itemprop="articleBody"> | ||
|
|
||
| <section id="summary-of-checksum-algorithms"> | ||
| <span id="qat-2-0-cnv-checksum-algorithms"></span><h1>Summary of Checksum Algorithms<a class="headerlink" href="#summary-of-checksum-algorithms" title="Permalink to this heading"></a></h1> | ||
| <p>Each checksum algorithm utilizes distinct methods for error detection. | ||
| The Adler32 uses an additive process. The CRC32 uses a polynomial-based | ||
| division operation. The XXHash32 checksum employs addition, | ||
| multiplication, rotate, shift and xor operations.</p> | ||
| <p>Despite their different techniques, all checksum algorithms are, to varying extents, vulnerable to collisions—instances where distinct inputs produce the same hash output. This susceptibility impacts their performance and reliability, although the degree of vulnerability varies across the algorithms</p> | ||
| <p>Empirical testing, which involved over 500 billion error-injected dataset tests (detailed in the section The Empirical Study), supports the notion that each algorithm’s collision rate differs based on its design.</p> | ||
| <p>It was noted that when the dataset was sourced from the printable ASCII range (32 to 128), Adler32 exhibited a higher collision rate compared to when the dataset used the full ASCII character set. CRC32 and XXHash32, however, showed no significant difference in collision rates between these two datasets.</p> | ||
| <p>The increased collision rate for Adler32 when using printable ASCII characters can be attributed to the simplicity of its additive process. The nature of the data perturbations, combined with the smaller dataset size, makes Adler32 more prone to hash collisions—particularly when the modulo operation used in the addition step does not adequately account for the smaller dataset.</p> | ||
| <p>This paper suggests that the differing mechanisms behind Adler32, CRC32, and XXHash32 provide complementary checks for dataset integrity. This idea is further corroborated by the empirical data, supporting the rationale for using multiple checksums in the QAT CnV process.</p> | ||
| <p>From the testing, the following p-values for collisions were obtained:</p> | ||
| <blockquote> | ||
| <div><ul class="simple"> | ||
| <li><p><strong>CRC32: Pc</strong> = 2.22196E-10</p></li> | ||
| <li><p><strong>Adler32: Pa</strong> = 1.75051E-07</p></li> | ||
| <li><p><strong>XXHash32: Px</strong> = 2.87547E-10</p></li> | ||
| </ul> | ||
| </div></blockquote> | ||
| <p>Because the algorithms are independent of each other, calculating the probability of | ||
| an error inducing collisions with two checks is straightforward.</p> | ||
| <p>For the Deflate algorithm, using both CRC32 and Adler32 as integrity checks produces a P value of:</p> | ||
| <blockquote> | ||
| <div><p>Pdef = Pc x Pa = 2.22196E-10 * 1.75051E-07 = 5.03354E-17</p> | ||
| </div></blockquote> | ||
| <p>For LZ4 and LZ4s, using CRC32 and XXHash32:</p> | ||
| <blockquote> | ||
| <div><p>Plz4 = Pc x Px = 2.22196E-10 * 2.87547E-10 = 6.38918E-20</p> | ||
| </div></blockquote> | ||
| </section> | ||
|
|
||
|
|
||
| </div> | ||
| </div> | ||
| <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer"> | ||
| <a href="checksums.html" class="btn btn-neutral float-left" title="Checksum Discussion" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a> | ||
| <a href="cnv_tests.html" class="btn btn-neutral float-right" title="Intel® QAT Compress and Verify Tests" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a> | ||
| </div> | ||
|
|
||
| <hr/> | ||
|
|
||
| <div role="contentinfo"> | ||
| <p>© Copyright Intel Corporation.</p> | ||
| </div> | ||
|
|
||
|
|
||
|
|
||
| </footer> | ||
| </div> | ||
| </div> | ||
| </section> | ||
| </div> | ||
| <script> | ||
| jQuery(function () { | ||
| SphinxRtdTheme.Navigation.enable(true); | ||
| }); | ||
| </script> | ||
|
|
||
| </body> | ||
| </html> |
Oops, something went wrong.