115:
record the output values (or state changes) and generate a set of characterization tests. When the generated tests are executed against a new version of the code, they will produce one or more failures/warnings if that version of the code has been modified in a way that changes a previously established behavior.
114:
One of the interesting aspects of characterization tests is that, since they are based on existing code, it's possible to generate some characterization tests automatically. An automated characterization test tool will exercise existing code with a wide range of relevant and/or random input values,
94:
When creating a characterization test, one must observe what outputs occur for a given set of inputs. Given an observation that the legacy code gives a certain output based on given inputs, then a test can be written that asserts that the output of the legacy code matches the observed result for the
110:
behavior of the code, which can be impossible to determine. Instead they verify the behavior that was observed when they were written. Often no specification or test suite is available, leaving only characterization tests as an option, since the conservative path is to assume that the old behavior
98:
Unfortunately, as with any testing, it is generally not possible to create a characterization test for every possible input and output. As such, many people opt for either statement or branch coverage. However, even this can be difficult. Test writers must use their judgment to decide how much
95:
given inputs. For example, if one observes that f(3.14) == 42, then this could be created as a characterization test. Then, after modifications to the system, the test can determine if the modifications caused changes in the results when given the same inputs.
51:
The goal of characterization tests is to help developers verify that the modifications made to a reference version of a software system did not modify its behavior in unwanted or undesirable ways. They enable, and provide a safety net for, extending and
79:, the outcome of the test is not determined by individual values or properties (that are checked with assertions), but by comparing a complex result of the tested software-process as a whole with the result of the same process in a previous
111:
is the required behavior. Characterization tests are, essentially, change detectors. It is up to the person analyzing the results to determine if the detected change was expected and/or desirable, or unexpected and/or undesirable.
181:/ removed, both from the Golden Master as well as from the result of the process. If too many elements need to be removed or removing them is too complex, it can render Golden Master testing impractical.
99:
testing is appropriate. It is often sufficient to write characterization tests that only cover the specific inputs and outputs that are known to occur, paying special attention to edge cases.
156:, images, etc. where checking all relevant attributes with assertions would be both insensible due to the amount of attributes and result in unreadable/
232:
204:
83:
of the software. In a sense, characterization testing inverts traditional testing: Traditional tests check individual properties (
187:
Golden Master testing does not infer correctness of the results. It merely helps detect unwanted effects of software changes.
184:
It depends not only on the software to be repeatable but also on the stability of the environment and input values.
303:
119:
308:
134:
Golden Master testing has the following advantages over the traditional assertions-based software testing:
169:
Golden Master testing has the following disadvantages over traditional assertions-based software testing:
246:
20:
80:
103:
228:
64:
40:
67:
and
Michael Bolton's classification of test oracles, this kind of testing corresponds to the
178:
76:
281:
157:
123:
72:
35:
behavior of an existing piece of software, and therefore protect existing behavior of
297:
174:
139:
88:
87:
them), where characterization testing checks all properties that are not removed (
287:
53:
36:
126:
to create complex test cases that capture use cases and special cases thereof.
276:
84:
57:
205:"J. B. Rainsberger - Surviving Legacy Code with Golden Master and Sampling"
106:, to which they are very similar, characterization tests do not verify the
284:
first in a blog-based series of tutorials on characterization tests.
148:
It is generally a sensible approach for complex results such as
153:
149:
122:
level, characterization testing can be combined with
177:. Volatile and non-deterministic values need to be
282:Working Effectively With Characterization Tests
138:It is relatively easy to implement for complex
43:. This term was coined by Michael Feathers.
8:
31:) is a means to describe (characterize) the
196:
71:. In contrast to the usual approach of
290:DDJ article on characterization tests.
7:
225:Working Effectively with Legacy Code
14:
56:code that does not have adequate
245:Bolton, Michael (January 2005).
39:against unintended changes via
1:
145:As such allows refactoring.
325:
124:intelligent monkey testing
256:. Sticky Minds / TechWell
288:Change Code Without Fear
247:"Testing Without a Map"
277:Characterization Tests
223:Feathers, Michael C.
29:Golden Master Testing
25:characterization test
16:Type of software test
118:When testing on the
21:computer programming
69:historical oracle
41:automated testing
316:
304:Software testing
265:
264:
262:
261:
251:
242:
236:
221:
215:
214:
212:
211:
201:
104:regression tests
77:software testing
324:
323:
319:
318:
317:
315:
314:
313:
294:
293:
273:
268:
259:
257:
254:Better Software
249:
244:
243:
239:
222:
218:
209:
207:
203:
202:
198:
194:
167:
132:
49:
27:(also known as
17:
12:
11:
5:
322:
320:
312:
311:
309:Legacy systems
306:
296:
295:
292:
291:
285:
279:
272:
271:External links
269:
267:
266:
237:
216:
195:
193:
190:
189:
188:
185:
182:
173:It depends on
166:
163:
162:
161:
158:unmaintainable
146:
143:
140:legacy systems
131:
128:
48:
45:
15:
13:
10:
9:
6:
4:
3:
2:
321:
310:
307:
305:
302:
301:
299:
289:
286:
283:
280:
278:
275:
274:
270:
255:
248:
241:
238:
234:
233:0-13-117705-2
230:
226:
220:
217:
206:
200:
197:
191:
186:
183:
180:
176:
175:repeatability
172:
171:
170:
165:Disadvantages
164:
159:
155:
151:
147:
144:
141:
137:
136:
135:
129:
127:
125:
121:
116:
112:
109:
105:
100:
96:
92:
90:
86:
82:
78:
74:
70:
66:
61:
59:
55:
46:
44:
42:
38:
34:
30:
26:
22:
258:. Retrieved
253:
240:
224:
219:
208:. Retrieved
199:
168:
133:
117:
113:
107:
101:
97:
93:
68:
65:James Bach's
62:
50:
32:
28:
24:
18:
89:blacklisted
54:refactoring
37:legacy code
298:Categories
260:2017-05-30
210:2017-05-30
192:References
160:test code.
130:Advantages
85:whitelists
73:assertions
58:unit tests
47:Overview
108:correct
102:Unlike
81:version
75:-based
231:
179:masked
33:actual
250:(PDF)
229:ISBN
150:PDFs
23:, a
154:XML
120:GUI
91:).
63:In
19:In
300::
252:.
235:).
152:,
60:.
263:.
227:(
213:.
142:.
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.