-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathpython-testing-style-guidelines.html
231 lines (197 loc) · 16.2 KB
/
python-testing-style-guidelines.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<title> Python testing style guidelines
</title>
<meta name="description" content="">
<meta name="viewport" content="width=device-width">
<link rel="stylesheet" href="https://jrsmith3.github.io/theme/css/normalize.css">
<link href='//fonts.googleapis.com/css?family=Lato' rel='stylesheet' type='text/css'>
<link href='//fonts.googleapis.com/css?family=Oswald' rel='stylesheet' type='text/css'>
<link rel="stylesheet" href="https://jrsmith3.github.io/theme/css/font-awesome.min.css">
<link rel="stylesheet" href="https://jrsmith3.github.io/theme/css/main.css">
<link rel="stylesheet" href="https://jrsmith3.github.io/theme/css/blog.css">
<link rel="stylesheet" href="https://jrsmith3.github.io/theme/css/github.css">
<link href="https://jrsmith3.github.io/feeds/all.atom.xml" type="application/atom+xml" rel="alternate" title="Generic Surname Atom Feed" />
<script src="https://jrsmith3.github.io/theme/js/vendor/modernizr-2.6.2.min.js"></script>
</head>
<body>
<!--[if lt IE 7]>
<p class="chromeframe">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> or <a href="http://www.google.com/chromeframe/?redirect=true">activate Google Chrome Frame</a> to improve your experience.</p>
<![endif]-->
<div id="wrapper">
<header id="sidebar" class="side-shadow">
<hgroup id="site-header">
<a id="site-title" href="https://jrsmith3.github.io"><h1><i class="icon-coffee"></i> Generic Surname</h1></a>
<p id="site-desc"></p>
</hgroup>
<nav>
<ul id="nav-links">
<li><a href="https://jrsmith3.github.io/pages/about.html">About</a></li>
<li><a href="https://jrsmith3.github.io/pages/contact.html">Contact</a></li>
<li><a href="https://jrsmith3.github.io/pages/curriculum-vitae.html">Curriculum Vitae</a></li>
</ul>
</nav>
<footer id="site-info">
<p>
Proudly powered by <a href="http://getpelican.com/" target="pelican">Pelican</a> and <a href="http://python.org/" target="python">Python</a>. Theme by <a href="https://github.com/hdra/pelican-cait" target="github">hndr</a>.
</p>
<p>
Textures by <a href="http://subtlepatterns.com/" target="subtlepatterns">Subtle Pattern</a>. Font Awesome by <a href="http://fortawesome.github.io/Font-Awesome/" target="github">Dave Grandy</a>.
</p>
</footer></header>
<div id="post-container">
<ol id="post-list">
<li>
<article class="post-entry">
<header class="entry-header">
<time class="post-time" datetime="2015-01-20T00:00:00-05:00" pubdate>
Tue 20 January 2015
</time>
<a href="https://jrsmith3.github.io/python-testing-style-guidelines.html" rel="bookmark"><h1>Python testing style guidelines</h1></a>
</header>
<section class="post-content">
<p>This post was updated on 2016-03-26 and is revision 0.2.0.</p>
<p>Testing a module you've written in python can easily spiral out of control, particularly one which simulates a physical phenomenon. You want to be sure that your tests are comprehensive, but its quite easy to fall down a rabbit hole of writing completely inane tests for edge cases that will never occur in practice. After writing and refactoring my <a href="https://github.com/jrsmith3/tec"><code>tec</code></a> module several times, I've gotten a sense of the various types of tests that need to be written and how to organize them. I intend this strategy to be along the lines of a "style guidelines" type document like <a href="https://www.python.org/dev/peps/pep-0008/">pep8</a> or <a href="https://en.wikipedia.org/wiki/The_Elements_of_Style">Strunk and White</a>.</p>
<p>These guidelines assume the reader has an understanding of testing python code and the <code>unittest</code> framework. Resources for testing code with Python are given at the end of this essay.</p>
<h1>Overview</h1>
<p>Tests tend to fall into two categories: API tests and numerical tests. API tests test the interface of the module: ensuring proper return types of methods, ensuring exceptions are raised under appropriate conditions, methods that return units actually return the correct unit, etc. Numerical tests verify the numerical accuracy of the methods that return the results of the physical simulation.</p>
<h1>Test style guidelines</h1>
<h2>Organization of test files in the filesystem</h2>
<p>Files containing tests for a python module should be located in a <code>test</code> directory in the root of the repo <a href="http://pytest.org/latest/goodpractices.html?highlight=inline#choosing-a-test-layout-import-rules">for the sake of separation of concerns</a>. Each file in the <code>test</code> directory should contain tests for one and only one class/function defined in the module. Files containing tests should be named according to the rubric </p>
<div class="highlight"><pre>test_ClassName.py
</pre></div>
<p>At the top of the file containing the tests, I have the typical python import statements. Below that, I usually define some common initialization parameters needed for the tests. These initialization parameters aren't used directly in the tests as I will explain momentarily.</p>
<p>Next, I usually define a base class (called <code>Base</code>) which is a subclass of <code>unittest.TestCase</code>. In this class I define a <code>setUp</code> method (and possibly <code>tearDown</code>) because I've found that most of the tests I write require a common environment in which to be run. This base class allows me to split up tests according to functionality by writing subclasses of <code>Base</code> while avoiding copying/pasting the same <code>setUp</code> method between these classes. Therefore I avoid introducing the category of bugs associated with copying and pasting code for reuse. In this <code>Base</code> class I <code>copy</code> the common initialization parameters I've defined at the top of the test file into attributes of the class so that a uniform environment is created for each test via <code>Base.setUp</code>. I've found there's a category of bugs that appear if I directly use the initialization parameters defined at the top of the test file: some tests require the initialization parameters to be changed slightly. Its possible to define a parameter and have it change in memory as a result of a test. Subsequent tests will therefore throw errors.</p>
<p>I organize tests according to functionality by subclassing <code>Base</code> (and thus <code>unittest.TestCase</code>). For classes that implement a physical phenomenon, tests tend to fall into the following categories:</p>
<ul>
<li><code>Instantiation</code> - test all aspects of instantiating an object. Includes input of wrong type, input outside of a bound, etc.</li>
<li><code>Set</code> - test all aspects of instantiating an object. Includes similar tests to the instantiation tests.</li>
<li><code>MethodsInput</code> - test methods that take input. Includes passing data of the wrong type, data that's outside of a constraint, etc.</li>
<li><code>MethodsReturnType</code> - test that methods return the proper type of data.</li>
<li><code>MethodsReturnUnits</code> - test that methods return the proper units, assuming the method returns data of type <code>astropy.units.Unit</code>.</li>
<li><code>MethodsReturnValues</code> - tests that the methods return appropriate values. These tests are particularly important for methods which actually implement the physical model; I call them "calculator methods."</li>
</ul>
<p>Each class contains methods that implement a test. These methods are named according to the rubric</p>
<div class="highlight"><pre>test_name_condition
</pre></div>
<p>Where "name" refers to the name of the attribute or method being tested and "condition" refers to the condition being tested. The name of the class which contains the attribute or method under test should not appear in the test method name, nor should the functionality of the test. For example, the following test names should not be used:</p>
<div class="highlight"><pre>test_ClassName_temp_less_than_zero
test_input_temp_less_than_zero
</pre></div>
<p>The first example contains the name of the class (<code>ClassName</code>) in which the attribute <code>temp</code> is found. The class name is superfluous in this case because this test is presumably located in a file named <code>test_ClassName.py</code>.</p>
<p>The second example contains the name of the functionality being tested, i.e. "input." The functionality in this case is superfluous because the test is presumably a method of the class <code>Instantiation</code>
.</p>
<p>Docstrings should include the name of the class under test, the functionality being tested, the name of the attribute/method being tested, and the condition being tested. In this way, <code>nosetests</code> will display all the information required to locate the failed/erronious test within this classification scheme. The order is not strictly required; sometimes it is easier to write something like "Setting ClassName.strictly_positive_attribute < 0 is invalid." The docstrings should describe logically what the test method does. In other words, it should be clear from the docstring how the test is passed or failed.</p>
<h2>Additional style guidelines for numerical tests</h2>
<p>Each calculator method requires the following items in order to be considered to be fully tested.</p>
<ul>
<li>A unit listed in its docstring. No mention of unit indicates the quantity is unitless.</li>
<li>A value of uncertainty listed in its docstring.</li>
<li>An analysis of the method's uncertainty documented in that method's test's docstring.</li>
<li>A unit test that tests the method against standard data.</li>
<li>Additional unit tests that test any other special, edge, or corner cases.</li>
</ul>
<h2>Remarks</h2>
<p>If a class has several methods that do similar things, many of the tests in a class will be very similar. I wrote a really dumb command line tool I call <a href="https://github.com/jrsmith3/testwriter"><code>testwriter</code></a> to automatically generate code for these tests.</p>
<h1>Example/skeleton test file</h1>
<div class="highlight"><pre><span class="c1"># -*- coding: utf-8 -*-</span>
<span class="c1"># Import statements</span>
<span class="c1"># =================</span>
<span class="c1"># Common input parameters</span>
<span class="c1"># =======================</span>
<span class="c1"># Base classes</span>
<span class="c1"># ============</span>
<span class="k">class</span> <span class="nc">Base</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Base class for tests</span>
<span class="sd"> This class defines a common `setUp` method that defines attributes which are used in the various tests.</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">pass</span>
<span class="c1"># Test classes</span>
<span class="c1"># ============</span>
<span class="k">class</span> <span class="nc">Instantiation</span><span class="p">(</span><span class="n">Base</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Tests all aspects of instantiation</span>
<span class="sd"> Tests include: instantiation with args of wrong type, instantiation with input values outside constraints, etc.</span>
<span class="sd"> """</span>
<span class="k">pass</span>
<span class="c1"># Input arguments wrong type</span>
<span class="c1"># ==========================</span>
<span class="c1"># Input arguments outside constraints</span>
<span class="c1"># ===================================</span>
<span class="k">class</span> <span class="nc">Set</span><span class="p">(</span><span class="n">Base</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Tests all aspects of setting attributes</span>
<span class="sd"> Tests include: setting attributes of wrong type, setting attributes outside their constraints, etc.</span>
<span class="sd"> """</span>
<span class="k">pass</span>
<span class="c1"># Set attribute wrong type</span>
<span class="c1"># ========================</span>
<span class="c1"># Set attribute outside constraint</span>
<span class="c1"># ================================</span>
<span class="k">class</span> <span class="nc">MethodsInput</span><span class="p">(</span><span class="n">Base</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Tests methods which take input parameters</span>
<span class="sd"> Tests include: passing invalid input, etc.</span>
<span class="sd"> """</span>
<span class="k">pass</span>
<span class="k">class</span> <span class="nc">MethodsReturnType</span><span class="p">(</span><span class="n">Base</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Tests methods' output types</span>
<span class="sd"> """</span>
<span class="k">pass</span>
<span class="k">class</span> <span class="nc">MethodsReturnUnits</span><span class="p">(</span><span class="n">Base</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Tests methods' output units where applicable</span>
<span class="sd"> """</span>
<span class="k">pass</span>
<span class="k">class</span> <span class="nc">MethodsReturnValues</span><span class="p">(</span><span class="n">Base</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Tests values of methods against known values</span>
<span class="sd"> """</span>
<span class="k">pass</span>
</pre></div>
<h1>Resources</h1>
<p><a href="http://www.amazon.com/dp/1847198848">Arbuckle (isbn 9781847198846)</a> gives an excellent, but slightly dated, introduction to python testing.
The <a href="http://pythontesting.net">Python testing blog</a>.
The <a href="http://pytest.org/latest/">py.test module</a>.</p>
</section>
<hr/>
<aside class="post-meta">
<p>Category: <a href="https://jrsmith3.github.io/category/blog.html">Blog</a></p>
</aside>
<hr/>
<div class="comments">
<div id="disqus_thread"></div>
<script type="text/javascript">
var disqus_shortname = 'genericsurname';
(function() {
var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
})();
</script>
<noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>
<a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a>
</div>
</article>
</li>
</ol>
</div>
</div>
<script>
var _gaq=[['_setAccount','UA-1567200-4'],['_trackPageview']];
(function(d,t){var g=d.createElement(t),s=d.getElementsByTagName(t)[0];
g.src=('https:'==location.protocol?'//ssl':'//www')+'.google-analytics.com/ga.js';
s.parentNode.insertBefore(g,s)}(document,'script'));
</script>
<script src="https://jrsmith3.github.io/theme/js/main.js"></script>
</body>
</html>