-
Notifications
You must be signed in to change notification settings - Fork 0
/
RunningTheUnitTests.html
256 lines (213 loc) · 13.4 KB
/
RunningTheUnitTests.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
<!DOCTYPE html>
<html lang="en" data-content_root="./">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Running the Unit Tests</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=fa44fd50" />
<link rel="stylesheet" type="text/css" href="_static/bootstrap-sphinx.css?v=fadd4351" />
<link rel="stylesheet" type="text/css" href="_static/custom.css?v=77160d70" />
<script src="_static/documentation_options.js?v=a8da1a53"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Debugging Unit Tests" href="DebuggingUnitTests.html" />
<link rel="prev" title="Conda Package Manager" href="CondaPackageManager.html" />
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-59110517-1', 'auto');
ga('send', 'pageview');
</script>
</head><body>
<div id="navbar" class="navbar navbar-default ">
<div class="container">
<div class="navbar-header">
<!-- .btn-navbar is used as the toggle for collapsed navbar content -->
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".nav-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="http://www.mantidproject.org">
</a>
<span class="navbar-text navbar-version pull-left"><b>main</b></span>
</div>
<div class="collapse navbar-collapse nav-collapse">
<ul class="nav navbar-nav">
<li class="divider-vertical"></li>
<li><a href="index.html">Home</a></li>
<li><a href="https://download.mantidproject.org">Download</a></li>
<li><a href="https://docs.mantidproject.org">User Documentation</a></li>
<li><a href="http://www.mantidproject.org/contact">Contact Us</a></li>
</ul>
<form class="navbar-form navbar-right" action="search.html" method="get">
<div class="form-group">
<input type="text" name="q" class="form-control" placeholder="Search" />
</div>
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<p>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="nav-item nav-item-0"><a href="index.html">Documentation</a> »</li>
<li class="nav-item nav-item-this"><a href="">Running the Unit Tests</a></li>
</ul>
</div> </p>
</div>
<div class="container">
<div class="row">
<div class="body col-md-12 content" role="main">
<section id="running-the-unit-tests">
<span id="runningtheunittests"></span><h1>Running the Unit Tests<a class="headerlink" href="#running-the-unit-tests" title="Link to this heading">¶</a></h1>
<nav class="contents local" id="contents">
<ul class="simple">
<li><p><a class="reference internal" href="#overview" id="id1">Overview</a></p></li>
<li><p><a class="reference internal" href="#cmake-ctest-command-line" id="id2">CMake/CTest: Command Line</a></p></li>
<li><p><a class="reference internal" href="#useful-ctest-options" id="id3">Useful CTest Options</a></p></li>
<li><p><a class="reference internal" href="#running-unit-tests-directly" id="id4">Running Unit Tests Directly</a></p></li>
<li><p><a class="reference internal" href="#running-unit-tests-with-visual-studio-and-ctest" id="id5">Running Unit Tests With Visual Studio and ctest</a></p></li>
<li><p><a class="reference internal" href="#running-unit-tests-with-visual-studio" id="id6">Running Unit Tests With Visual Studio</a></p></li>
<li><p><a class="reference internal" href="#debugging-unit-tests" id="id7">Debugging unit tests</a></p></li>
</ul>
</nav>
<section id="overview">
<h2><a class="toc-backref" href="#id1" role="doc-backlink">Overview</a><a class="headerlink" href="#overview" title="Link to this heading">¶</a></h2>
<p>We use <a class="reference external" href="https://cmake.org/cmake/help/latest/manual/ctest.1.html">CTest</a>
for building and running our unit tests. This wraps the underlying
<a class="reference external" href="cxxtest">cxxtest</a> or other (e.g. pyunit) test code.</p>
</section>
<section id="cmake-ctest-command-line">
<h2><a class="toc-backref" href="#id2" role="doc-backlink">CMake/CTest: Command Line</a><a class="headerlink" href="#cmake-ctest-command-line" title="Link to this heading">¶</a></h2>
<p>The unit tests are currently excluded from the ‘all’ target. To build
all of the tests, including the dependent framework code run:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>cmake<span class="w"> </span>--build<span class="w"> </span>.<span class="w"> </span>--target<span class="w"> </span>AllTests
</pre></div>
</div>
<p>To build only one package of tests (and its dependencies):</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>cmake<span class="w"> </span>--build<span class="w"> </span>.<span class="w"> </span>--target<span class="w"> </span>KernelTest
</pre></div>
</div>
<p>To run all the tests:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>ctest<span class="w"> </span>-j8
</pre></div>
</div>
<p>To build and run all the tests in one shot:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>cmake<span class="w"> </span>--build<span class="w"> </span>.<span class="w"> </span>--target<span class="w"> </span>check
</pre></div>
</div>
<p>To run a specific test or set of tests (will run all those that match
the search string):</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>ctest<span class="w"> </span>-R<span class="w"> </span>KernelTest_TimerTest
</pre></div>
</div>
<p>So to run all tests in a suite (using a search string):</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>ctest<span class="w"> </span>-j8<span class="w"> </span>-R<span class="w"> </span>KernelTest
</pre></div>
</div>
<p>To exclude things from your tests (matches the string as with the -R
option) - useful for those building the performance tests:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>ctest<span class="w"> </span>-j8<span class="w"> </span>-E<span class="w"> </span>Performance
</pre></div>
</div>
</section>
<section id="useful-ctest-options">
<h2><a class="toc-backref" href="#id3" role="doc-backlink">Useful CTest Options</a><a class="headerlink" href="#useful-ctest-options" title="Link to this heading">¶</a></h2>
<p><code class="docutils literal notranslate"><span class="pre">ctest</span> <span class="pre">-h</span></code> gives the full list of command line arguments. Some useful
ones to note are:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">--output-on-failure</span></code>: displays the log and any stderr output that
is otherwise hidden by CTest.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">--schedule-random</span></code>: run the tests in a random order. Useful to
weed out accidental dependencies</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">--repeat-until-fail</span></code>: require each test to run times without
failing in order to pass. Useful to try and find random failures</p></li>
</ul>
</section>
<section id="running-unit-tests-directly">
<h2><a class="toc-backref" href="#id4" role="doc-backlink">Running Unit Tests Directly</a><a class="headerlink" href="#running-unit-tests-directly" title="Link to this heading">¶</a></h2>
<p>Starting in your build folder:</p>
<ul>
<li><p>Running an entire test suite (e.g. <cite>KernelTest</cite>):</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>./bin/KernelTest
</pre></div>
</div>
</li>
<li><p>Running a specific test class.</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>./bin/KernelTest<span class="w"> </span>MyTestClassName
</pre></div>
</div>
</li>
<li><p>Running a specific test from a CxxTest test class (not possible via CTest).</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>./bin/KernelTest<span class="w"> </span>MyTestClassName<span class="w"> </span>MySingleTestName
</pre></div>
</div>
</li>
<li><p>Running a specific test from a Python <code class="docutils literal notranslate"><span class="pre">unittest</span></code> test class (not possible
via CTest).</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="nv">PYTHONPATH</span><span class="o">=</span>/path/to/build/bin<span class="w"> </span>python3<span class="w"> </span>/path/to/src/Framework/PythonInterface/test/python/plugins/algorithms/MeanTest.py<span class="w"> </span>MeanTest.test_mean
</pre></div>
</div>
</li>
</ul>
</section>
<section id="running-unit-tests-with-visual-studio-and-ctest">
<h2><a class="toc-backref" href="#id5" role="doc-backlink">Running Unit Tests With Visual Studio and ctest</a><a class="headerlink" href="#running-unit-tests-with-visual-studio-and-ctest" title="Link to this heading">¶</a></h2>
<p>Open the Mantid solution in Visual Studio. To run a subset of tests (for example <code class="docutils literal notranslate"><span class="pre">UnitTests/AlgorithmsTest</span></code>);</p>
<ul>
<li><p>In the Solution Explorer, right click the project for the tests (in this case <code class="docutils literal notranslate"><span class="pre">UnitTests/AlgorithmsTest</span></code>) and select Properties.</p></li>
<li><p>In the Debugging tab of Properties change the Command Arguments box to the name of the test, for example “AddPeakTest”.</p></li>
<li><p>Right click the directory again and select Debug->Start new instance.</p></li>
<li><p>Once the build has finished, open a file browser and navigate to the mantid build directory, run the command-prompt.bat file to open a command prompt and run</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>ctest<span class="w"> </span>-C<span class="w"> </span>Debug<span class="w"> </span>-V<span class="w"> </span>-R<span class="w"> </span>AddPeakTest
</pre></div>
</div>
<p>For this example, there should be several lines of output ending with the time taken and the line</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="m">100</span>%<span class="w"> </span>tests<span class="w"> </span>passed,<span class="w"> </span><span class="m">0</span><span class="w"> </span>tests<span class="w"> </span>failed<span class="w"> </span>out<span class="w"> </span>of<span class="w"> </span><span class="m">1</span>
</pre></div>
</div>
<p>Omitting the <code class="docutils literal notranslate"><span class="pre">-R</span> <span class="pre">AddPeakTest</span></code> option runs all the tests, but note that any tests which were not built according to the above instructions will fail. Adding the <code class="docutils literal notranslate"><span class="pre">-V</span></code> increases the output verbosity.</p>
</li>
</ul>
</section>
<section id="running-unit-tests-with-visual-studio">
<h2><a class="toc-backref" href="#id6" role="doc-backlink">Running Unit Tests With Visual Studio</a><a class="headerlink" href="#running-unit-tests-with-visual-studio" title="Link to this heading">¶</a></h2>
<p>The unit tests can be run from within Visual Studio, following steps 1-3 above, with the addition in step 2 of;</p>
<ul class="simple">
<li><p>Add the name of the test to the Target Name field in the General tab of Properties. Then add a breakpoint somewhere in the test header file.</p></li>
</ul>
</section>
<section id="debugging-unit-tests">
<h2><a class="toc-backref" href="#id7" role="doc-backlink">Debugging unit tests</a><a class="headerlink" href="#debugging-unit-tests" title="Link to this heading">¶</a></h2>
<p>See the instructions <a class="reference external" href="DebuggingUnitTests">here</a></p>
</section>
</section>
</div>
</div>
</div>
<footer class="footer">
<div class="container">
<ul class="nav navbar-nav" style=" float: right;">
<li>
<a href="CondaPackageManager.html" title="Previous Chapter: Conda Package Manager"><span class="glyphicon glyphicon-chevron-left visible-sm"></span><span class="hidden-sm hidden-tablet">« Conda Package Manager</span>
</a>
</li>
<li>
<a href="DebuggingUnitTests.html" title="Next Chapter: Debugging Unit Tests"><span class="glyphicon glyphicon-chevron-right visible-sm"></span><span class="hidden-sm hidden-tablet">Debugging Unit Tests »</span>
</a>
</li>
<li><a href="#">Back to top</a></li>
</ul>
<p>
</p>
</div>
</footer>
</body>
</html>