1 """
2 This is a top level package, hosting the entire CSB test framework. It is divided
3 into several major parts:
4
5 - test cases, located under csb.test.cases
6 - test data, in C{/csb/test/data} (not a package)
7 - test console, in C{/csb/test/app.py}
8
9 This module, csb.test, contains all the glue-code functions, classes and
10 decorators you would need in order to write tests for CSB.
11
12 1. Configuration and Tree
13
14 L{Config<csb.test.Config>} is a common config object shared between CSB
15 tests. Each config instance contains properties like:
16
17 - data: the data folder, automatically discovered and loaded in
18 csb.test.Config.DATA at module import time
19 - temp: a default temp folder, which test cases can use
20
21 Each L{Config<csb.test.Config>} provides a convenient way to retrieve
22 files from C{/csb/test/data}. Be sure to check out L{Config.getTestFile}
23 and L{Config.getPickle}. In case you need a temp file, use
24 L{Config.getTempStream} or have a look at L{csb.io.TempFile} and
25 L{csb.io.TempFolder}.
26
27 All test data files should be placed in the C{data} folder. All test
28 modules must be placed in the root package: csb.test.cases. There is
29 a strict naming convention for test modules: the name of a test module
30 should be the same as the name of the CSB API package it tests. For
31 example, if you are writing tests for C{csb/bio/io/__init__.py}, the
32 test module must be C{csb/test/cases/bio/io/__init__.py}. C{csb.test.cases}
33 is the root package of all test modules in CSB.
34
35 2. Writing Tests
36
37 Writing a test is easy. All you need is to import csb.test and then
38 create your own test cases, derived from L{csb.test.Case}:
39
40 >>> import csb.test
41 >>> @csb.test.unit
42 class TestSomeClass(csb.test.Case):
43 def setUp(self):
44 super(TestSomeClass, self).setUp()
45 # do something with self.config here...
46
47 In this way your test case instance is automatically equipped with a
48 reference to the test config, so your test method can be:
49
50 >>> @csb.test.unit
51 class TestSomeClass(csb.test.Case):
52 def testSomeMethod(self):
53 myDataFile = self.config.getTestFile('some.file')
54 self.assert...
55
56 The "unit" decorator marks a test case as a collection of unit tests.
57 All possibilities are: L{csb.test.unit}, L{csb.test.functional}, L{csb.test.custom},
58 and L{csb.test.regression}.
59
60 Writing custom (a.k.a. "data", "slow", "dynamic") tests is a little bit
61 more work. Custom tests must be functions, not classes. Basically a
62 custom test is a function, which builds a unittest.TestSuite instance
63 and then returns it when called without arguments.
64
65 Regression tests are usually created in response to reported bugs. Therefore,
66 the best practice is to mark each test method with its relevant bug ID:
67
68 >>> @csb.test.regression
69 class SomeClassRegressions(csb.test.Case)
70 def testSomeFeature(self)
71 \"""
72 @see: [CSB 000XXXX]
73 \"""
74 # regression test body...
75
76 3. Style Guide:
77
78 - name test case packages as already described
79 - group tests in csb.test.Case-s and name them properly
80 - prefix test methods with "test", like "testParser" - very important
81 - use camelCase for methods and variables. This applies to all the
82 code under csb.test (including test) and does not apply to the rest
83 of the library!
84 - for functional tests it's okay to define just one test method: runTest
85 - for unit tests you should create more specific test names, for example:
86 "testParseFile" - a unit test for some method called "parse_file"
87 - use csb.test decorators to mark tests as unit, functional, regression, etc.
88 - make every test module executable::
89
90 if __name__ == '__main__':
91 csb.test.Console() # Discovers and runs all test cases in the module
92
93 4. Test Execution
94
95 Test discovery is handled by C{test builders} and a test runner
96 C{app}. Test builders are subclasses of L{AbstractTestBuilder}.
97 For every test type (unit, functional, regression, custom) there is a
98 corresponding test builder. L{AnyTestBuilder} is a special builder which
99 scans for unit, regression and functional tests at the same time.
100
101 Test builder classes inherit the following test discovery methods:
102
103 - C{loadTests} - load tests from a test namespace. Wildcard
104 namespaces are handled by C{loadAllTests}
105 - C{loadAllTests} - load tests from the given namespace, and
106 from all sub-packages (recursive)
107 - C{loadFromFile} - load tests from an absolute file name
108 - C{loadMultipleTests} - calls C{loadTests} for a list of
109 namespaces and combines all loaded tests in a single suite
110
111 Each of those return test suite objects, which can be directly executed
112 with python's unittest runner.
113
114 Much simpler way to execute a test suite is to use our test app
115 (C{csb/test/app.py}), which is simply an instance of L{csb.test.Console}::
116
117 $ python csb/test/app.py --help
118
119 The app has two main arguments:
120
121 - test type - tells the app which TestBuilder to use for test dicsovery
122 ("any" triggers L{AnyTestBuilder}, "unit" - L{UnitTestBuilder}, etc.)
123 - test namespaces - a list of "dotted" test modules, for example::
124
125 csb.test.cases.bio.io.* # io and sub-packages
126 csb.test.cases.bio.utils # only utils
127 . # current module
128
129 In addition to running the app from the command line, you can run it
130 also programmatically by instantiating L{csb.test.Console}. You can
131 construct a test console object by passing a list of test namespace(s)
132 and a test builder class to the Console's constructor.
133
134
135 5. Commit Policies
136
137 Follow these guidelines when making changes to the repository:
138
139 - B{no bugs in "trunk"}: after fixing a bug or implementing a new
140 feature, make sure at least the default test set passes by running
141 the test console without any arguments. This is equivalent to:
142 app.py -t any "csb.test.cases.*". (If no test case from this set covers
143 the affected code, create a test case first, as described in the other
144 policies)
145
146 - B{no recurrent issues}: when a bug is found, first write a regression
147 test with a proper "@see: BugID" tag in the docstring. Run the test
148 to make sure it fails. After fixing the bug, run the test again before
149 you commit, as required by the previous policy
150
151 - B{test all new features}: there should be a test case for every new feature
152 we implement. One possible approach is to write a test case first and
153 make sure it fails; when the new feature is ready, run the test again
154 to make sure it passes
155
156 @warning: for compatibility reasons do NOT import and use the unittest module
157 directly. Always import unittest from csb.test, which is guaranteed
158 to be python 2.7+ compatible. The standard unittest under python 2.6
159 is missing some features, that's why csb.test will take care of
160 replacing it with unittest2 instead.
161 """
162 import os
163 import sys
164 import imp
165 import types
166 import time
167 import tempfile
168 import traceback
169 import argparse
170
171 import csb.io
172 import csb.core
173
174 try:
175 from unittest import skip, skipIf
176 import unittest
177 except ImportError:
178 import unittest2 as unittest
179
180 from abc import ABCMeta, abstractproperty
189
191 """
192 General CSB Test Config. Config instances contain the following properties:
193
194 - data - path to the CSB Test Data directory. Default is L{Config.DATA}
195 - temp - path to the system's temp directory. Default is L{Config.TEMP}
196 - config - the L{Config} class
197 """
198
199 DATA = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'data')
200 """
201 @cvar: path to the default test data directory: <install dir>/csb/test/data
202 """
203 GENERATED_DATA = DATA
204 """
205 @cvar: path to the default data directory for generated test files
206 """
207 TEMP = os.path.abspath(tempfile.gettempdir())
208 """
209 @cvar: path to the default system's temp directory
210 """
211
212 @staticmethod
214 """
215 Override the default L{Config.DATA} with a new data root directory.
216
217 @param path: full directory path
218 @type path: str
219 """
220 if not os.path.isdir(path):
221 raise IOError('Path not found: {0}'.format(path))
222
223 Config.DATA = os.path.abspath(path)
224
225 @staticmethod
227 """
228 Override the default L{Config.GENERATED_DATA} with a new data root directory.
229
230 @param path: full directory path
231 @type path: str
232 """
233 if not os.path.isdir(path):
234 raise IOError('Path not found: {0}'.format(path))
235
236 Config.GENERATED_DATA = os.path.abspath(path)
237
238 @property
240 """
241 Test data directory
242 @rtype: str
243 """
244 return Config.DATA
245
246 @property
248 """
249 Test data directory for generated files
250 @rtype: str
251 """
252 return Config.GENERATED_DATA
253
254 @property
256 """
257 Test temp directory
258 @rtype: str
259 """
260 return Config.TEMP
261
263 """
264 Search for C{fileName} in the L{Config.DATA} directory. If not found,
265 try also L{Config.GENERATED_DATA} (if different).
266
267 @param fileName: the name of a test file to retrieve
268 @type fileName: str
269 @param subDir: scan a sub-directory of L{Config.DATA}
270 @type subDir: str
271
272 @return: full path to C{fileName}
273 @rtype: str
274
275 @raise IOError: if no such file is found
276 """
277 for data in [self.data, self.generated_data]:
278 file = os.path.join(data, subDir, fileName)
279
280 if os.path.isfile(file):
281 return file
282
283 raise IOError('Test file not found: {0}'.format(fileName))
284
286 """
287 Same as C{self.getTestFile}, but try to unpickle the the file
288 and return the unpickled object. Pickles are usually stored in
289 L{Config.GENERATED_DATA}.
290
291 @param fileName: the name of a test file to retrieve
292 @type fileName: str
293 @param subDir: scan a sub-directory of L{Config.DATA}
294 @type subDir: str
295
296 @rtype: object
297 """
298 file = self.getTestFile(fileName, subDir)
299 return csb.io.Pickle.load(open(file, 'rb'))
300
301 - def getContent(self, fileName, subDir=''):
302 """
303 Same as C{self.getTestFile}, but also read and return the contents of
304 the file.
305
306 @param fileName: the name of a test file to retrieve
307 @type fileName: str
308 @param subDir: scan a sub-directory of L{Config.DATA}
309 @type subDir: str
310
311 @rtype: str
312 """
313 with open(self.getTestFile(fileName, subDir)) as f:
314 return f.read()
315
317 """
318 Return a temporary file stream::
319
320 with self.getTempStream() as tmp:
321 tmp.write(something)
322 tmp.flush()
323 file_name = tmp.name
324
325 @param mode: file open mode (text, binary), default=t
326 @type mode: str
327 @rtype: file stream
328 """
329 return csb.io.TempFile(mode=mode)
330
332 """
333 Try to deserialize some pickled data files. Call L{Config.updateDataFiles}
334 if the pickles appeared incompatible with the current interpreter.
335 """
336 try:
337 self.getPickle('1nz9.model1.pickle')
338 except:
339 self.updateDataFiles()
340
364
365 -class Case(unittest.TestCase):
366 """
367 Base class, defining a CSB Test Case. Provides a default implementation
368 of C{unittest.TestCase.setUp} which grabs a reference to a L{Config}.
369 """
370
371 @property
373 """
374 Test config instance
375 @rtype: L{Config}
376 """
377 return self.__config
378
380 """
381 Provide a reference to the CSB Test Config in the C{self.config} property.
382 """
383 self.__config = Config()
384 assert hasattr(self.config, 'data'), 'The CSB Test Config must contain the data directory'
385 assert self.config.data, 'The CSB Test Config must contain the data directory'
386
388 """
389 Re-raise the last exception with its full traceback, but modify the
390 argument list with C{addArgs} and the original stack trace.
391
392 @param addArgs: additional arguments to append to the exception
393 @type addArgs: tuple
394 """
395 klass, ex, _tb = sys.exc_info()
396 ex.args = list(ex.args) + list(addArgs) + [''.join(traceback.format_exc())]
397
398 raise klass(ex.args)
399
401
402 if first == second:
403 return
404 if delta is not None and places is not None:
405 raise TypeError("specify delta or places not both")
406
407 if delta is not None:
408
409 if abs(first - second) <= delta:
410 return
411
412 m = '{0} != {1} within {2} delta'.format(first, second, delta)
413 msg = self._formatMessage(msg, m)
414
415 raise self.failureException(msg)
416
417 else:
418 if places is None:
419 places = 7
420
421 return super(Case, self).assertAlmostEqual(first, second, places=places, msg=msg)
422
424 """
425 Fail if it took more than C{duration} seconds to invoke C{callable}.
426
427 @param duration: maximum amount of seconds allowed
428 @type duration: float
429 """
430
431 start = time.time()
432 callable(*args, **kargs)
433 execution = time.time() - start
434
435 if execution > duration:
436 self.fail('{0}s is slower than {1}s)'.format(execution, duration))
437
438 @classmethod
440 """
441 Run this test case.
442 """
443 suite = unittest.TestLoader().loadTestsFromTestCase(cls)
444 runner = unittest.TextTestRunner()
445
446 return runner.run(suite)
447
450
452 """
453 This is a base class, defining a test loader which exposes the C{loadTests}
454 method.
455
456 Subclasses must override the C{labels} abstract property, which controls
457 what kind of test cases are loaded by the test builder.
458 """
459
460 __metaclass__ = ABCMeta
461
462 @abstractproperty
465
467 """
468 Load L{csb.test.Case}s from a module file.
469
470 @param file: test module file name
471 @type file: str
472
473 @return: a C{unittest.TestSuite} ready for the test runner
474 @rtype: C{unittest.TestSuite}
475 """
476 mod = self._loadSource(file)
477 suite = unittest.TestLoader().loadTestsFromModule(mod)
478 return unittest.TestSuite(self._filter(suite))
479
481 """
482 Load L{csb.test.Case}s from the given CSB C{namespace}. If the namespace
483 ends with a wildcard, tests from sub-packages will be loaded as well.
484 If the namespace is '__main__' or '.', tests are loaded from __main__.
485
486 @param namespace: test module namespace, e.g. 'csb.test.cases.bio' will
487 load tests from '/csb/test/cases/bio/__init__.py'
488 @type namespace: str
489
490 @return: a C{unittest.TestSuite} ready for the test runner
491 @rtype: C{unittest.TestSuite}
492 """
493 if namespace.strip() == '.*':
494 namespace = '__main__.*'
495 elif namespace.strip() == '.':
496 namespace = '__main__'
497
498 if namespace.endswith('.*'):
499 return self.loadAllTests(namespace[:-2])
500 else:
501 loader = unittest.TestLoader()
502 tests = loader.loadTestsFromName(namespace)
503 return unittest.TestSuite(self._filter(tests))
504
506 """
507 Load L{csb.test.Case}s from a list of given CSB C{namespaces}.
508
509 @param namespaces: a list of test module namespaces, e.g.
510 ('csb.test.cases.bio', 'csb.test.cases.bio.io') will
511 load tests from '/csb/test/cases/bio.py' and
512 '/csb/test/cases/bio/io.py'
513 @type namespaces: tuple of str
514
515 @return: a C{unittest.TestSuite} ready for the test runner
516 @rtype: C{unittest.TestSuite}
517 """
518 if not csb.core.iterable(namespaces):
519 raise TypeError(namespaces)
520
521 return unittest.TestSuite(self.loadTests(n) for n in namespaces)
522
524 """
525 Load L{csb.test.Case}s recursively from the given CSB C{namespace} and
526 all of its sub-packages. Same as::
527
528 builder.loadTests('namespace.*')
529
530 @param namespace: test module namespace, e.g. 'csb.test.cases.bio' will
531 load tests from /csb/test/cases/bio/*'
532 @type namespace: str
533
534 @return: a C{unittest.TestSuite} ready for the test runner
535 @rtype: C{unittest.TestSuite}
536 """
537 suites = []
538
539 try:
540 base = __import__(namespace, level=0, fromlist=['']).__file__
541 except ImportError:
542 raise InvalidNamespaceError('Namespapce {0} is not importable'.format(namespace))
543
544 if os.path.splitext(os.path.basename(base))[0] != '__init__':
545 suites.append(self.loadTests(namespace))
546
547 else:
548
549 for entry in os.walk(os.path.dirname(base)):
550
551 for item in entry[2]:
552 file = os.path.join(entry[0], item)
553 if extension and item.endswith(extension):
554 suites.append(self.loadFromFile(file))
555
556 return unittest.TestSuite(suites)
557
559 """
560 Import and return the Python module identified by C{path}.
561
562 @note: Module objects behave as singletons. If you import two different
563 modules and give them the same name in imp.load_source(mn), this
564 counts for a redefinition of the module originally named mn, which
565 is basically the same as reload(mn). Therefore, you need to ensure
566 that for every call to imp.load_source(mn, src.py) the mn parameter
567 is a string that uniquely identifies the source file src.py.
568 """
569 name = os.path.splitext(os.path.abspath(path))[0]
570 name = name.replace('.', '-').rstrip('__init__').strip(os.path.sep)
571
572 return imp.load_source(name, path)
573
575 """
576 Extract test cases recursively from a test C{obj} container.
577 """
578 cases = []
579 if isinstance(obj, unittest.TestSuite) or csb.core.iterable(obj):
580 for item in obj:
581 cases.extend(self._recurse(item))
582 else:
583 cases.append(obj)
584 return cases
585
587 """
588 Filter a list of objects using C{self.labels}.
589 """
590 filtered = []
591
592 for test in self._recurse(tests):
593 for label in self.labels:
594 if hasattr(test, label) and getattr(test, label) is True:
595 filtered.append(test)
596
597 return filtered
598
600 """
601 Build a test suite of cases, marked as either unit, functional or regression
602 tests. For detailed documentation see L{AbstractTestBuilder}.
603 """
604 @property
607
609 """
610 Build a test suite of cases, marked as unit tests.
611 For detailed documentation see L{AbstractTestBuilder}.
612 """
613 @property
616
618 """
619 Build a test suite of cases, marked as functional tests.
620 For detailed documentation see L{AbstractTestBuilder}.
621 """
622 @property
625
627 """
628 Build a test suite of cases, marked as regression tests.
629 For detailed documentation see L{AbstractTestBuilder}.
630 """
631 @property
634
636 """
637 Build a test suite of cases, marked as custom tests. CustomTestBuilder will
638 search for functions, marked with the 'custom' test decorator, which return
639 a dynamically built C{unittest.TestSuite} object when called without
640 parameters. This is convenient when doing data-related tests, e.g.
641 instantiating a single type of a test case many times iteratively, for
642 each entry in a database.
643
644 For detailed documentation see L{AbstractTestBuilder}.
645 """
646 @property
649
651
652 mod = self._loadSource(file)
653 suites = self._inspect(mod)
654
655 return unittest.TestSuite(suites)
656
673
675
676 objects = map(lambda n: getattr(module, n), dir(module))
677 return self._filter(objects)
678
680 """
681 Filter a list of objects using C{self.labels}.
682 """
683 filtered = []
684
685 for obj in factories:
686 for label in self.labels:
687 if hasattr(obj, label) and getattr(obj, label) is True:
688 suite = obj()
689 if not isinstance(suite, unittest.TestSuite):
690 raise ValueError('Custom test function {0} must return a '
691 'unittest.TestSuite, not {1}'.format(obj.__name__, type(suite)))
692 filtered.append(suite)
693
694 return filtered
695
697 """
698 A class decorator, used to label unit test cases.
699
700 @param klass: a C{unittest.TestCase} class type
701 @type klass: type
702 """
703 if not isinstance(klass, type):
704 raise TypeError("Can't apply class decorator on {0}".format(type(klass)))
705
706 setattr(klass, Attributes.UNIT, True)
707 return klass
708
710 """
711 A class decorator, used to label functional test cases.
712
713 @param klass: a C{unittest.TestCase} class type
714 @type klass: type
715 """
716 if not isinstance(klass, type):
717 raise TypeError("Can't apply class decorator on {0}".format(type(klass)))
718
719 setattr(klass, Attributes.FUNCTIONAL, True)
720 return klass
721
723 """
724 A class decorator, used to label regression test cases.
725
726 @param klass: a C{unittest.TestCase} class type
727 @type klass: type
728 """
729 if not isinstance(klass, type):
730 raise TypeError("Can't apply class decorator on {0}".format(type(klass)))
731
732 setattr(klass, Attributes.REGRESSION, True)
733 return klass
734
736 """
737 A function decorator, used to mark functions which build custom (dynamic)
738 test suites when called.
739
740 @param function: a callable object, which returns a dynamically compiled
741 C{unittest.TestSuite}
742 @type function: callable
743 """
744 if isinstance(function, type):
745 raise TypeError("Can't apply function decorator on a class")
746 elif not hasattr(function, '__call__'):
747 raise TypeError("Can't apply function decorator on non-callable {0}".format(type(function)))
748
749 setattr(function, Attributes.CUSTOM, True)
750 return function
751
752 -def skip(reason, condition=None):
753 """
754 Mark a test case or method for skipping.
755
756 @param reason: message
757 @type reason: str
758 @param condition: skip only if the specified condition is True
759 @type condition: bool/expression
760 """
761 if isinstance(reason, types.FunctionType):
762 raise TypeError('skip: no reason specified')
763
764 if condition is None:
765 return unittest.skip(reason)
766 else:
767 return unittest.skipIf(condition, reason)
768
770 """
771 Build and run all tests of the specified namespace and kind.
772
773 @param namespace: a dotted name, which specifies the test module
774 (see L{csb.test.AbstractTestBuilder.loadTests})
775 @type namespace: str
776 @param builder: test builder to use
777 @type builder: any L{csb.test.AbstractTestBuilder} subclass
778 @param verbosity: verbosity level for C{unittest.TestRunner}
779 @type verbosity: int
780 @param update: if True, refresh all pickles in csb/test/data
781 @type update: bool
782 @param generated_data: where to cache generated test files (directory)
783 @type generated_data: str
784 """
785
786 BUILDERS = {'unit': UnitTestBuilder, 'functional': FunctionalTestBuilder,
787 'custom': CustomTestBuilder, 'any': AnyTestBuilder,
788 'regression': RegressionTestBuilder}
789
790
812
813 @property
815 return self._namespace
816 @namespace.setter
818 if csb.core.iterable(value):
819 self._namespace = list(value)
820 else:
821 self._namespace = [value]
822
823 @property
826 @builder.setter
828 self._builder = value
829
830 @property
832 return self._verbosity
833 @verbosity.setter
835 self._verbosity = value
836
837 @property
840
841 @property
844
845 @property
848 @update.setter
850 self._update = bool(value)
851
852 @property
855 @generated_data.setter
858
873
875
876 parser = argparse.ArgumentParser(prog=self.program, description="CSB Test Runner Console.")
877
878 parser.add_argument("-t", "--type", type=str, default="any", choices=list(Console.BUILDERS),
879 help="Type of tests to load from each namespace (default=any)")
880 parser.add_argument("-v", "--verbosity", type=int, default=1,
881 help="Verbosity level passed to unittest.TextTestRunner (default=1).")
882 parser.add_argument("-u", "--update-files", default=False, action="store_true",
883 help="Force update of the test pickles in " + Config.GENERATED_DATA)
884 parser.add_argument("-g", "--generated-resources", type=str, default=Config.GENERATED_DATA,
885 help="Generate, store and load additional test resources in this directory"
886 " (default=" + Config.GENERATED_DATA + ")")
887
888 parser.add_argument("namespaces", nargs='*',
889 help="""An optional list of CSB test dotted namespaces, from which to
890 load tests. '__main__' and '.' are interpreted as the
891 current module. If a namespace ends with an asterisk
892 '.*', all sub-packages will be scanned as well.
893
894 Examples:
895 "csb.test.cases.bio.*"
896 "csb.test.cases.bio.io" "csb.test.cases.bio.utils"
897 ".")""")
898
899 args = parser.parse_args(argv)
900
901 self.builder = Console.BUILDERS[args.type]
902 self.verbosity = args.verbosity
903 self.update = args.update_files
904 self.generated_data = args.generated_resources
905
906 if args.namespaces:
907 self.namespace = args.namespaces
908
909
910 if __name__ == '__main__':
911
912 Console()
913