tests_dependencies.html 23 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224
  1. <html>
  2. <head>
  3. <meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
  4. <title>Managing test dependencies</title>
  5. <link rel="stylesheet" href="../../boostbook.css" type="text/css">
  6. <meta name="generator" content="DocBook XSL Stylesheets V1.79.1">
  7. <link rel="home" href="../../index.html" title="Boost.Test">
  8. <link rel="up" href="../tests_organization.html" title="Declaring and organizing tests">
  9. <link rel="prev" href="fixtures/global.html" title="Global fixture">
  10. <link rel="next" href="tests_grouping.html" title="Grouping tests into logical units by labels">
  11. </head>
  12. <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
  13. <table cellpadding="2" width="100%"><tr>
  14. <td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../boost.png"></td>
  15. <td align="center"><a href="../../../../../../index.html">Home</a></td>
  16. <td align="center"><a href="../../../../../../libs/libraries.htm">Libraries</a></td>
  17. <td align="center"><a href="http://www.boost.org/users/people.html">People</a></td>
  18. <td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td>
  19. <td align="center"><a href="../../../../../../more/index.htm">More</a></td>
  20. </tr></table>
  21. <hr>
  22. <div class="spirit-nav">
  23. <a accesskey="p" href="fixtures/global.html"><img src="../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../tests_organization.html"><img src="../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="tests_grouping.html"><img src="../../../../../../doc/src/images/next.png" alt="Next"></a>
  24. </div>
  25. <div class="section">
  26. <div class="titlepage"><div><div><h3 class="title">
  27. <a name="boost_test.tests_organization.tests_dependencies"></a><a class="link" href="tests_dependencies.html" title="Managing test dependencies">Managing
  28. test dependencies</a>
  29. </h3></div></div></div>
  30. <p>
  31. In general, it is a good practice to write any test as independent as possible
  32. from any other, there are however cases where a dependency cannot be avoided
  33. and an order for executing the tests is needed.
  34. </p>
  35. <p>
  36. In the general setup and for any two test cases <code class="computeroutput"><span class="identifier">TA</span></code>
  37. and <code class="computeroutput"><span class="identifier">TB</span></code>, <code class="computeroutput"><span class="identifier">TB</span></code>
  38. should not take for granted that <code class="computeroutput"><span class="identifier">TA</span></code>
  39. has already executed, even if <code class="computeroutput"><span class="identifier">TA</span></code>
  40. is declared before <code class="computeroutput"><span class="identifier">TB</span></code> in
  41. the same translation unit. The only ordering-related guarantee that <span class="emphasis"><em>Unit
  42. Test Framework</em></span> makes by default is that if test cases <code class="computeroutput"><span class="identifier">TA</span></code> and <code class="computeroutput"><span class="identifier">TB</span></code>
  43. are declared in the same test suite, no test case (call it <code class="computeroutput"><span class="identifier">TX</span></code>) from any other test suite is executed
  44. between <code class="computeroutput"><span class="identifier">TA</span></code> and <code class="computeroutput"><span class="identifier">TB</span></code>, even if the declaration of <code class="computeroutput"><span class="identifier">TX</span></code> appears between the declarations of
  45. <code class="computeroutput"><span class="identifier">TA</span></code> and <code class="computeroutput"><span class="identifier">TB</span></code>.
  46. In other words, all tests from a suite are executed in one go, even if the
  47. test suite namespace is opened multiple times.
  48. </p>
  49. <p>
  50. Even though the order is not guaranteed, it may accidentally be preserved
  51. across the different runs. In order to make sure the test cases do not depend
  52. on one another, the test module may be called with an additional command-line
  53. argument, <a class="link" href="../utf_reference/rt_param_reference/random.html" title="random"><code class="computeroutput"><span class="identifier">random</span></code></a>, to shuffle the tests unit
  54. ordering and to be more robust against an erroneous implicit ordering.
  55. </p>
  56. <h4>
  57. <a name="boost_test.tests_organization.tests_dependencies.h0"></a>
  58. <span class="phrase"><a name="boost_test.tests_organization.tests_dependencies.declaring_a_test_case_dependency"></a></span><a class="link" href="tests_dependencies.html#boost_test.tests_organization.tests_dependencies.declaring_a_test_case_dependency">Declaring
  59. a test case dependency</a>
  60. </h4>
  61. <p>
  62. If there exist a dependency between test units, and an ordering is required
  63. between the execution of those tests, it has to be declared explicitly. Dependencies
  64. in the <span class="emphasis"><em>Unit Test Framework</em></span> affect two dimensions of
  65. test units, which are:
  66. </p>
  67. <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
  68. <li class="listitem">
  69. the order of execution of these units
  70. </li>
  71. <li class="listitem">
  72. the execution of a test unit, which is conditioned by the state of its
  73. parents
  74. </li>
  75. </ul></div>
  76. <p>
  77. <a class="link" href="decorators.html" title="Decorators">Decorator</a>
  78. <a class="link" href="../utf_reference/test_org_reference/decorator_depends_on.html" title="depends_on (decorator)"><code class="computeroutput"><span class="identifier">depends_on</span></code></a> associates the decorated
  79. test case (call it <code class="computeroutput"><span class="identifier">TB</span></code>) with
  80. another test case (call it <code class="computeroutput"><span class="identifier">TA</span></code>)
  81. specified by name. This affects the processing the test tree in two ways:
  82. </p>
  83. <div class="orderedlist"><ol class="orderedlist" type="1">
  84. <li class="listitem">
  85. first, test case <code class="computeroutput"><span class="identifier">TA</span></code> is
  86. ordered to be run before <code class="computeroutput"><span class="identifier">TB</span></code>,
  87. irrespective of the order in which they were declared or added to the
  88. test tree,
  89. </li>
  90. <li class="listitem">
  91. second, the execution of <code class="computeroutput"><span class="identifier">TB</span></code>
  92. is skipped if <code class="computeroutput"><span class="identifier">TA</span></code> is either
  93. disabled or skipped or is executed and marked as failed.
  94. </li>
  95. </ol></div>
  96. <h6>
  97. <a name="boost_test.tests_organization.tests_dependencies.h1"></a>
  98. <span class="phrase"><a name="boost_test.tests_organization.tests_dependencies.example_descr"></a></span><a class="link" href="tests_dependencies.html#boost_test.tests_organization.tests_dependencies.example_descr">Example:
  99. decorator depends_on</a>
  100. </h6>
  101. <div class="informaltable"><table class="table">
  102. <colgroup><col></colgroup>
  103. <thead><tr><th>
  104. <p>
  105. Code
  106. </p>
  107. </th></tr></thead>
  108. <tbody><tr><td>
  109. <pre xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" class="table-programlisting"><span class="preprocessor">#define</span> <span class="identifier">BOOST_TEST_MODULE</span> <span class="identifier">decorator_07</span>
  110. <span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">test</span><span class="special">/</span><span class="identifier">included</span><span class="special">/</span><span class="identifier">unit_test</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
  111. <span class="keyword">namespace</span> <span class="identifier">utf</span> <span class="special">=</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">unit_test</span><span class="special">;</span>
  112. <span class="comment">// test1 and test2 defined at the bottom</span>
  113. <span class="identifier">BOOST_AUTO_TEST_CASE</span><span class="special">(</span><span class="identifier">test3</span><span class="special">,</span> <span class="special">*</span> <span class="identifier">utf</span><span class="special">::</span><span class="identifier">depends_on</span><span class="special">(</span><span class="string">"s1/test1"</span><span class="special">))</span>
  114. <span class="special">{</span>
  115. <span class="identifier">BOOST_TEST</span><span class="special">(</span><span class="keyword">false</span><span class="special">);</span>
  116. <span class="special">}</span>
  117. <span class="identifier">BOOST_AUTO_TEST_CASE</span><span class="special">(</span><span class="identifier">test4</span><span class="special">,</span> <span class="special">*</span> <span class="identifier">utf</span><span class="special">::</span><span class="identifier">depends_on</span><span class="special">(</span><span class="string">"test3"</span><span class="special">))</span>
  118. <span class="special">{</span>
  119. <span class="identifier">BOOST_TEST</span><span class="special">(</span><span class="keyword">false</span><span class="special">);</span>
  120. <span class="special">}</span>
  121. <span class="identifier">BOOST_AUTO_TEST_CASE</span><span class="special">(</span><span class="identifier">test5</span><span class="special">,</span> <span class="special">*</span> <span class="identifier">utf</span><span class="special">::</span><span class="identifier">depends_on</span><span class="special">(</span><span class="string">"s1/test2"</span><span class="special">))</span>
  122. <span class="special">{</span>
  123. <span class="identifier">BOOST_TEST</span><span class="special">(</span><span class="keyword">false</span><span class="special">);</span>
  124. <span class="special">}</span>
  125. <span class="identifier">BOOST_AUTO_TEST_SUITE</span><span class="special">(</span><span class="identifier">s1</span><span class="special">)</span>
  126. <span class="identifier">BOOST_AUTO_TEST_CASE</span><span class="special">(</span><span class="identifier">test1</span><span class="special">)</span>
  127. <span class="special">{</span>
  128. <span class="identifier">BOOST_TEST</span><span class="special">(</span><span class="keyword">true</span><span class="special">);</span>
  129. <span class="special">}</span>
  130. <span class="identifier">BOOST_AUTO_TEST_CASE</span><span class="special">(</span><span class="identifier">test2</span><span class="special">,</span> <span class="special">*</span> <span class="identifier">utf</span><span class="special">::</span><span class="identifier">disabled</span><span class="special">())</span>
  131. <span class="special">{</span>
  132. <span class="identifier">BOOST_TEST</span><span class="special">(</span><span class="keyword">false</span><span class="special">);</span>
  133. <span class="special">}</span>
  134. <span class="identifier">BOOST_AUTO_TEST_SUITE_END</span><span class="special">()</span>
  135. </pre>
  136. </td></tr></tbody>
  137. </table></div>
  138. <div class="informaltable"><table class="table">
  139. <colgroup><col></colgroup>
  140. <thead><tr><th>
  141. <p>
  142. Output
  143. </p>
  144. </th></tr></thead>
  145. <tbody><tr><td>
  146. <pre xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" class="table-programlisting"><span class="special">&gt;</span> <span class="identifier">decorator_07</span> <span class="special">--</span><span class="identifier">report_level</span><span class="special">=</span><span class="identifier">detailed</span> <span class="special">--</span><span class="identifier">log_level</span><span class="special">=</span><span class="identifier">all</span>
  147. <span class="identifier">Running</span> <span class="number">4</span> <span class="identifier">test</span> <span class="identifier">cases</span><span class="special">...</span>
  148. <span class="identifier">Entering</span> <span class="identifier">test</span> <span class="identifier">module</span> <span class="string">"decorator_07"</span>
  149. <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">:</span><span class="number">31</span><span class="special">:</span> <span class="identifier">Entering</span> <span class="identifier">test</span> <span class="identifier">suite</span> <span class="string">"s1"</span>
  150. <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">:</span><span class="number">33</span><span class="special">:</span> <span class="identifier">Entering</span> <span class="identifier">test</span> <span class="keyword">case</span> <span class="string">"test1"</span>
  151. <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">:</span><span class="number">35</span><span class="special">:</span> <span class="identifier">info</span><span class="special">:</span> <span class="identifier">check</span> <span class="keyword">true</span> <span class="identifier">has</span> <span class="identifier">passed</span>
  152. <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">:</span><span class="number">33</span><span class="special">:</span> <span class="identifier">Leaving</span> <span class="identifier">test</span> <span class="keyword">case</span> <span class="string">"test1"</span><span class="special">;</span> <span class="identifier">testing</span> <span class="identifier">time</span><span class="special">:</span> <span class="number">100u</span><span class="identifier">s</span>
  153. <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">:</span><span class="number">31</span><span class="special">:</span> <span class="identifier">Leaving</span> <span class="identifier">test</span> <span class="identifier">suite</span> <span class="string">"s1"</span><span class="special">;</span> <span class="identifier">testing</span> <span class="identifier">time</span><span class="special">:</span> <span class="number">129u</span><span class="identifier">s</span>
  154. <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">:</span><span class="number">16</span><span class="special">:</span> <span class="identifier">Entering</span> <span class="identifier">test</span> <span class="keyword">case</span> <span class="string">"test3"</span>
  155. <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">:</span><span class="number">18</span><span class="special">:</span> <span class="identifier">error</span><span class="special">:</span> <span class="identifier">in</span> <span class="string">"test3"</span><span class="special">:</span> <span class="identifier">check</span> <span class="keyword">false</span> <span class="identifier">has</span> <span class="identifier">failed</span>
  156. <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">:</span><span class="number">16</span><span class="special">:</span> <span class="identifier">Leaving</span> <span class="identifier">test</span> <span class="keyword">case</span> <span class="string">"test3"</span><span class="special">;</span> <span class="identifier">testing</span> <span class="identifier">time</span><span class="special">:</span> <span class="number">48u</span><span class="identifier">s</span>
  157. <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">:</span><span class="number">26</span><span class="special">:</span> <span class="identifier">Test</span> <span class="keyword">case</span> <span class="string">"test5"</span> <span class="identifier">is</span> <span class="identifier">skipped</span> <span class="identifier">because</span> <span class="identifier">dependency</span> <span class="identifier">test</span> <span class="keyword">case</span> <span class="string">"s1/test2"</span> <span class="identifier">is</span> <span class="identifier">disabled</span>
  158. <span class="identifier">test</span><span class="special">.</span><span class="identifier">cpp</span><span class="special">:</span><span class="number">21</span><span class="special">:</span> <span class="identifier">Test</span> <span class="keyword">case</span> <span class="string">"test4"</span> <span class="identifier">is</span> <span class="identifier">skipped</span> <span class="identifier">because</span> <span class="identifier">dependency</span> <span class="identifier">test</span> <span class="keyword">case</span> <span class="string">"test3"</span> <span class="identifier">has</span> <span class="identifier">failed</span>
  159. <span class="identifier">Leaving</span> <span class="identifier">test</span> <span class="identifier">module</span> <span class="string">"decorator_07"</span><span class="special">;</span> <span class="identifier">testing</span> <span class="identifier">time</span><span class="special">:</span> <span class="number">263u</span><span class="identifier">s</span>
  160. <span class="identifier">Test</span> <span class="identifier">module</span> <span class="string">"decorator_07"</span> <span class="identifier">has</span> <span class="identifier">failed</span> <span class="identifier">with</span><span class="special">:</span>
  161. <span class="number">1</span> <span class="identifier">test</span> <span class="keyword">case</span> <span class="identifier">out</span> <span class="identifier">of</span> <span class="number">4</span> <span class="identifier">passed</span>
  162. <span class="number">1</span> <span class="identifier">test</span> <span class="keyword">case</span> <span class="identifier">out</span> <span class="identifier">of</span> <span class="number">4</span> <span class="identifier">failed</span>
  163. <span class="number">2</span> <span class="identifier">test</span> <span class="identifier">cases</span> <span class="identifier">out</span> <span class="identifier">of</span> <span class="number">4</span> <span class="identifier">skipped</span>
  164. <span class="number">1</span> <span class="identifier">assertion</span> <span class="identifier">out</span> <span class="identifier">of</span> <span class="number">2</span> <span class="identifier">passed</span>
  165. <span class="number">1</span> <span class="identifier">assertion</span> <span class="identifier">out</span> <span class="identifier">of</span> <span class="number">2</span> <span class="identifier">failed</span>
  166. <span class="identifier">Test</span> <span class="keyword">case</span> <span class="string">"test3"</span> <span class="identifier">has</span> <span class="identifier">failed</span> <span class="identifier">with</span><span class="special">:</span>
  167. <span class="number">1</span> <span class="identifier">assertion</span> <span class="identifier">out</span> <span class="identifier">of</span> <span class="number">1</span> <span class="identifier">failed</span>
  168. <span class="identifier">Test</span> <span class="keyword">case</span> <span class="string">"test4"</span> <span class="identifier">was</span> <span class="identifier">skipped</span>
  169. <span class="identifier">Test</span> <span class="keyword">case</span> <span class="string">"test5"</span> <span class="identifier">was</span> <span class="identifier">skipped</span>
  170. <span class="identifier">Test</span> <span class="identifier">suite</span> <span class="string">"s1"</span> <span class="identifier">has</span> <span class="identifier">passed</span> <span class="identifier">with</span><span class="special">:</span>
  171. <span class="number">1</span> <span class="identifier">test</span> <span class="keyword">case</span> <span class="identifier">out</span> <span class="identifier">of</span> <span class="number">1</span> <span class="identifier">passed</span>
  172. <span class="number">1</span> <span class="identifier">assertion</span> <span class="identifier">out</span> <span class="identifier">of</span> <span class="number">1</span> <span class="identifier">passed</span>
  173. <span class="identifier">Test</span> <span class="keyword">case</span> <span class="string">"s1/test1"</span> <span class="identifier">has</span> <span class="identifier">passed</span> <span class="identifier">with</span><span class="special">:</span>
  174. <span class="number">1</span> <span class="identifier">assertion</span> <span class="identifier">out</span> <span class="identifier">of</span> <span class="number">1</span> <span class="identifier">passed</span>
  175. </pre>
  176. </td></tr></tbody>
  177. </table></div>
  178. <p>
  179. In the above scenario:
  180. </p>
  181. <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
  182. <li class="listitem">
  183. test case <code class="computeroutput"><span class="identifier">test3</span></code> is run
  184. (and fails) because <code class="computeroutput"><span class="identifier">s1</span><span class="special">/</span><span class="identifier">test1</span></code>
  185. has been run and succeeded,
  186. </li>
  187. <li class="listitem">
  188. <code class="computeroutput"><span class="identifier">test4</span></code> is skipped because
  189. <code class="computeroutput"><span class="identifier">test3</span></code> has failed,
  190. </li>
  191. <li class="listitem">
  192. <code class="computeroutput"><span class="identifier">test5</span></code> is skipped because
  193. <code class="computeroutput"><span class="identifier">s1</span><span class="special">/</span><span class="identifier">test2</span></code> is disabled.
  194. </li>
  195. </ul></div>
  196. </div>
  197. <table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
  198. <td align="left"></td>
  199. <td align="right"><div class="copyright-footer">Copyright &#169; 2001-2019 Boost.Test
  200. contributors<p>
  201. Distributed under the Boost Software License, Version 1.0. (See accompanying
  202. file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>)
  203. </p>
  204. </div></td>
  205. </tr></table>
  206. <hr>
  207. <div class="spirit-nav">
  208. <a accesskey="p" href="fixtures/global.html"><img src="../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../tests_organization.html"><img src="../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="tests_grouping.html"><img src="../../../../../../doc/src/images/next.png" alt="Next"></a>
  209. </div>
  210. </body>
  211. </html>