-
Notifications
You must be signed in to change notification settings - Fork 1
/
cache-tool.en.html
255 lines (247 loc) · 13.7 KB
/
cache-tool.en.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<title>Cache Tool — AMC Traffic Server Documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="_static/sphinxdoc.css" />
<link rel="stylesheet" type="text/css" href="_static/graphviz.css" />
<script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
<script src="_static/jquery.js"></script>
<script src="_static/underscore.js"></script>
<script src="_static/doctools.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Command Registration" href="command-registration.en.html" />
<link rel="prev" title="Traffic Server Projects" href="ats-projects.en.html" />
</head><body>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="command-registration.en.html" title="Command Registration"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="ats-projects.en.html" title="Traffic Server Projects"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">SWOC Docs</a> »</li>
<li class="nav-item nav-item-1"><a href="ats-projects.en.html" accesskey="U">Traffic Server Projects</a> »</li>
<li class="nav-item nav-item-this"><a href="">Cache Tool</a></li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="toctree-wrapper compound" id="cache-tool">
</div>
<section id="id1">
<h1>Cache Tool<a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h1>
<p>CacheTool is a command line tool for interacting with the Traffic Server cache. CacheTool started as a side
project to not only explore the Traffic Server cache but as a test bed for various other technologies of
interest to me, some of which are not really cache related. The goal is for the eventual release
of these other technologies to open source as they mature. My view was the ancillary software would
be of better quality if validated in actual use first.</p>
<ul class="simple">
<li><p><a class="reference external" href="https://docs.trafficserver.apache.org/en/latest/developer-guide/internal-libraries/scalar.en.html">TS.Scalar</a>: a template for creating quantized integral types. This has been contributed to open source.</p></li>
<li><p><a class="reference external" href="https://docs.trafficserver.apache.org/en/latest/developer-guide/internal-libraries/memview.en.html">TextView</a>: read only view of string data. This has been contributed to open source.</p></li>
<li><p>Better <a class="reference internal" href="command-registration.en.html#command-registration"><span class="std std-ref">command line parsing</span></a> for action oriented tools like CacheTool and <strong class="program">traffic_ctl</strong>.</p></li>
<li><p>File system path and file handling class.</p></li>
<li><p>New data structures for the cache, as part of a larger scale redesign.</p></li>
<li><p>Improved stripe allocation algorithms.</p></li>
</ul>
<p>In addition to being a test bed, Cache Tool was intended to do useful work. An early feature, stripe
re-allocation, was built to get around a bug in Traffic Server where a disk that was replaced would not
actually be used by the cache. This was originally a request from the Oath operations team but it
was sufficiently useful to another member of the community that I was persuaded to make CacheTool
public before it was even alpha quality.</p>
<p>Some of the intended features of CacheTool are</p>
<ul class="simple">
<li><p>Fix stripe allocation for unused disks.</p></li>
<li><p>List stripe data for a cache.</p></li>
<li><p>Clear / reset cache disks independently.</p></li>
<li><p>Provide cache directory analysis, as is done by passing <code class="code docutils literal notranslate"><span class="pre">--command</span> <span class="pre">check</span></code> to <strong class="program">traffic_server</strong> now, but with sufficiently low resource requirements it can be run on live system.</p></li>
<li><p>Cache directory diagnostics.</p></li>
<li><p>Cache scanning</p>
<ul>
<li><p>Scan to purge.</p></li>
<li><p>Statistical analysis.</p></li>
</ul>
</li>
<li><p>Cache repartitioning.</p>
<ul>
<li><p>Store additional information about cache configuration.</p>
<ul>
<li><p>More efficient and reliable examination.</p></li>
<li><p>Required to repartition, if values are always computed there is a chicken and egg problem.</p></li>
</ul>
</li>
<li><p>Update stripes to shift storage between directory and content (change effect average object size).</p></li>
<li><p>Maintain backwards compatibility and some forward compatibility. In particular it must be
possible to run 8.0 on a box and revert to 7.X without clearing the cache.</p></li>
</ul>
</li>
</ul>
<section id="usage">
<h2>Usage<a class="headerlink" href="#usage" title="Permalink to this headline">¶</a></h2>
<p>CacheTool has an action oriented interface with commands and subcommands (and possibly
subsubcommands, etc.), modeled on <strong class="program">traffic_ctl</strong> but with a much better internal implementation.</p>
<dl>
<dt><code class="docutils literal notranslate"><span class="pre">list</span></code></dt><dd><p>List information about the cache.</p>
<dl class="simple">
<dt><code class="docutils literal notranslate"><span class="pre">stripes</span></code></dt><dd><p>List per stripe information.</p>
</dd>
</dl>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">alloc</span></code></dt><dd><p>Allocate cache space to stripes.</p>
<dl class="simple">
<dt><code class="docutils literal notranslate"><span class="pre">free</span></code></dt><dd><p>Allocate all free space to volumes.</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">sim</span></code></dt><dd><p>Simulate a full allocation of space to volumes.</p>
</dd>
</dl>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">dir</span></code></dt><dd><p>Stripe directory operations.</p>
<dl class="simple">
<dt><code class="docutils literal notranslate"><span class="pre">freelist</span></code></dt><dd><p>Scan and fix segment free lists in the stripe.</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">stats</span></code></dt><dd><p>Print analytical data about the stripe directory.</p>
</dd>
</dl>
</dd>
</dl>
<p><code class="docutils literal notranslate"><span class="pre">span</span></code></p>
<blockquote>
<div><dl>
<dt><code class="docutils literal notranslate"><span class="pre">clear</span></code></dt><dd><p>Clear a span. All space in the stripe is consolidated in a single free block.
This command takes a list of span identifiers, each of which is cleared.</p>
<dl class="simple">
<dt><code class="docutils literal notranslate"><span class="pre">all</span></code></dt><dd><p>Clear all spans.</p>
</dd>
</dl>
</dd>
</dl>
</div></blockquote>
</section>
<section id="implementation">
<h2>Implementation<a class="headerlink" href="#implementation" title="Permalink to this headline">¶</a></h2>
<p>This section has internal implementation notes.</p>
<section id="directory-free-lists">
<h3>Directory Free Lists<a class="headerlink" href="#directory-free-lists" title="Permalink to this headline">¶</a></h3>
<p>Each segment in a directory has a free list. It is possible for this to become corrupted and have a
loop in the list. There is some logic which can be conditionally compiled in the Traffic Server core that will
place a maximum length on the free list. Upon detection an error is generated.</p>
<p>The CacheTool will detect free list loops more directly by using a <code class="code docutils literal notranslate"><span class="pre">std::bitset</span></code> to mark
directory entries that have been encountered while walking the free list. If a duplicate is detected
(indicating a loop) then rather than attempt any sort of complex repair, the CacheTool will clear
and then reconstruct the free list in a way similar to how the free list was constructed during
stripe initialization. This is primarily constructing the list so that directories at higher layers
in the buckets are earlier in the free list.</p>
</section>
<section id="rate-limiting">
<h3>Rate Limiting<a class="headerlink" href="#rate-limiting" title="Permalink to this headline">¶</a></h3>
<p>While in general the CacheTool will be run while Traffic Server is offline it will frequently be desirable to
be able to run CacheTool while Traffic Server is also running. To make this possible CacheTool will need to be
able to rate limit its disk reading. This will be specified by placing a limit on the number of
bytes per second that will be read. It is not expected this limiting will be exact but it should be
reasonably close. An possible implementation is to start with the per second limit as a read limit
and then after the read completes, subtract the size of the read and add the limit value for each
second that has passed. It may also be good to limit the addition to a maximum number of seconds to
further restrict I/O when the disks are under excessive load.</p>
</section>
</section>
<section id="future-work">
<h2>Future Work<a class="headerlink" href="#future-work" title="Permalink to this headline">¶</a></h2>
<p>This section contains some of the larger work planned for the CacheTool.</p>
<section id="content-scanning">
<h3>Content Scanning<a class="headerlink" href="#content-scanning" title="Permalink to this headline">¶</a></h3>
<p>A key feature is the ability to scan content stored in the cache. This is extremely expensive in
terms of disk I/O. This is of less concern if the system is offline, in which case CacheTool can
consume all availale disk I/O. Even in that case, however, the amount of data to be read can still
large enough to make the overall runtime too large if the data is read in a simplistic fashion.</p>
<p>The primary purpose of cache scanning is to extract the request and response headers. All of this
data is contained in the first fragment of the cached object. This is noted in the directory entry
which means disk reads can be restricted to only these fragments rather than a full scan of the
content area.</p>
</section>
</section>
</section>
<div class="clearer"></div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<p class="logo"><a href="index.html">
<img class="logo" src="_static/balcora-gate-400x400.jpg" alt="Logo"/>
</a></p>
<h3><a href="index.html">Table of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Cache Tool</a><ul>
<li><a class="reference internal" href="#usage">Usage</a></li>
<li><a class="reference internal" href="#implementation">Implementation</a><ul>
<li><a class="reference internal" href="#directory-free-lists">Directory Free Lists</a></li>
<li><a class="reference internal" href="#rate-limiting">Rate Limiting</a></li>
</ul>
</li>
<li><a class="reference internal" href="#future-work">Future Work</a><ul>
<li><a class="reference internal" href="#content-scanning">Content Scanning</a></li>
</ul>
</li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="ats-projects.en.html"
title="previous chapter">Traffic Server Projects</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="command-registration.en.html"
title="next chapter">Command Registration</a></p>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/cache-tool.en.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
<div id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
<input type="submit" value="Go" />
</form>
</div>
</div>
<script>$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="command-registration.en.html" title="Command Registration"
>next</a> |</li>
<li class="right" >
<a href="ats-projects.en.html" title="Traffic Server Projects"
>previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">SWOC Docs</a> »</li>
<li class="nav-item nav-item-1"><a href="ats-projects.en.html" >Traffic Server Projects</a> »</li>
<li class="nav-item nav-item-this"><a href="">Cache Tool</a></li>
</ul>
</div>
<div class="footer" role="contentinfo">
© Copyright 2017, [email protected].
Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 4.1.2.
</div>
</body>
</html>