Pytest passing test I think it should fail on sys.exit() call - python

I am trying to check the exit code I have for a script I'm writing in python3 on a Mac (10.14.4). When I run the test it doesn't fail which I think is wrong. But I can't see what it is that I've got wrong.
The test file looks like this:
import pytest
import os
import sys
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
import my_script
class TestMyScript():
def test_exit(self):
with pytest.raises(SystemExit) as pytest_wrapped_e:
my_script.main()
assert pytest_wrapped_e.type == SystemExit
def test_exit_code(self):
with pytest.raises(SystemExit) as pytest_wrapped_e:
my_script.main()
self.assertEqual(pytest_wrapped_e.exception.code, 42)
My script looks like:
#!/usr/bin/env python3
import sys
def main():
print('Hello World!')
sys.exit(0)
if __name__ == '__main__':
main()
The output I get is:
$ py.test -v
============================= test session starts ==============================
platform darwin -- Python 3.7.3, pytest-3.10.1, py-1.8.0, pluggy-0.9.0 -- /usr/local/opt/python/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/robertpostill/software/gateway, inifile:
plugins: shutil-1.6.0
collected 2 items
test/test_git_refresh.py::TestGitRefresh::test_exit PASSED [ 50%]
test/test_git_refresh.py::TestGitRefresh::test_exit_code PASSED [100%]
=========================== 2 passed in 0.02 seconds ===========================
$
I would expect the second test(test_exit_code) to fail as the exit call is getting a code of 0, not 42. But for some reason, the assert is happy whatever value I put in the sys.exit call.

Good question, that's because your Asserts are never called (either of them). When exit() is called the program is done (at least within the with clause), it turns off the lights, packs up its bags, and goes home. No further functions will be called. To see this add an assert before and after you call main:
def test_exit_code(self):
with pytest.raises(SystemExit) as pytest_wrapped_e:
self.assertEqual(0, 1) # This will make it fail
my_script.main()
self.assertEqual(0, 1) # This will never be called because main `exits`
self.assertEqual(pytest_wrapped_e.exception.code, 42)
A test passes if no asserts fail and nothing breaks, so in your case both tests passed because the assert was never hit.
To fix this pull your asserts out of the with statement:
def test_exit_code(self):
with pytest.raises(SystemExit) as pytest_wrapped_e:
my_script.main()
self.assertEqual(pytest_wrapped_e.exception.code, 42)
Though now you will need to fix the pytest syntax because you are missing some other stuff.
See: Testing sys.exit() with pytest

Related

Python unittest AssertionError not being raised

I am trying to write unit tests for my python package and I am finding that when I run the tests AssertionErrors are not being raised, when they should be. Here is a MWE:
In exampleModule.py I have:
#! /usr/bin/env python
import unittest
class UnitTest(unittest.TestCase):
def runTest(self):
print("Starting test...")
a = 4
b = 5
self.assertEqual(a,b)
print("TEST COMPLETE")
return
and in testError.py I have:
#! /usr/bin/env python
import unittest
class AllTests(unittest.TestCase):
def testExample(self):
from exampleModule import UnitTest
UT = UnitTest()
UT.run()
return
if __name__ == '__main__':
unittest.main()
When I run testError.py I expect to see the AssertionError reported from the UnitTest in exampleModule.py, however, I simply see the following:
> ./testError.py
Starting test...
.
----------------------------------------------------------------------
Ran 1 test in 0.001s
OK
Why is the AssertionError not being raised? If I place the UnitTest() class in testError.py (i.e. have everything in the same file) then the AssertionError is raised. So why when UnitTest is stored in a different file does the error not get raised?
Thanks!
TestCase.run()
creates or updates a
TestResult
object, on the assumption you intend to do something interesting with those results.
But you immediately throw them away:
UT.run()
Any failures or errors --- including exceptions raised --- would have been in that results object.
For example:
def testExample(self):
from exampleModule import UnitTest
UT = UnitTest()
result = UT.run()
print('Errors: {!r}'.format(result.errors))
print('Failures: {!r}'.format(result.failures))
return
This prints:
Failures: [(<exampleModule.UnitTest testMethod=runTest>, 'Traceback (most recent call last):\n File "/home/kjc/tmp/exampleModule.py", line 11, in runTest\n self.assertEqual(a, b)\nAssertionError: 4 != 5\n')]
It catches all kinds of exceptions, like this example where I added sys.exit() before the assertion:
Errors: [(<exampleModule.UnitTest testMethod=runTest>, 'Traceback (most recent call last):\n File "/home/kjc/tmp/exampleModule.py", line 11, in runTest\n import sys; sys.exit()\nSystemExit\n')]
For the record, the one passing test your example produced is testExample itself.
You can verify this by calling self.fail() early in testExample.
(As one commentator said, calling unit tests from a unit test is a very strange thing to do.)
One Solution
You seem to want to run tests normally, but from multiple files.
If so, you can make a bunch of typical
unittest.main()-style
test modules, and then load them manually.
Your test-everything script would look something like this:
#!/usr/bin/env python
import unittest
# Edit this to include new modules or TestCases to run.
_NAMES = ['testmoda', 'testmodb'] # testothermodule.SomeTestCase, etc...
def _main():
suite = unittest.defaultTestLoader.loadTestsFromNames(_NAMES)
unittest.TextTestRunner().run(suite)
return # WARNING: This always returns a successful exit value.
if __name__ == '__main__':
_main()
Your test modules would be the usual one-or-more TestCases, with conditional calls to unittest.main(), so each could be run independently:
#!/usr/bin/env python
import unittest
class TestModA(unittest.TestCase):
def testThatPasses(self):
self.assertEqual(0, -0)
return
def testThatFails(self):
self.assertEqual(0, float('inf'))
return
def testThatBlowsUp(self):
raise RuntimeError('OMG!')
return
if '__main__' == __name__:
unittest.main()
If that's not fiddly enough for you, you can also read up on the
discover() method
or the module-level
load_tests protocol.
The convention to structure unittests is like this:
Hold all tests in tests directory.
test module will be named as test_modulea.py with class TestClassA
nosetests -vw tests to run all tests.

Why Does “re.match” Behavior Change when Under Test?

I've run into a bizarre situation where I wrote a test to check for a "re.match" that would definitely fail, but passes when tested in py.test. The code fails as expected when ran outside a test environment.
OS: Ubuntu 16.04
Python: 2.7.12
pytest: 3.5.0
regex_code.py
import re
import sys
def main():
regex = re.compile(r'tests\.(test_\w+)\.?(Test\w+)?$')
input_text = 'test.tests.test_default'
if regex.match(input_text):
print('Pass')
return 0
else:
print('Fail')
return 1
if __name__ == "__main__":
sys.exit(main())
test_regex_code.py
import pytest
from regex_code import main
def test_main():
assert 1 == main()
When I run regex_code.py from the command-line I get following output:
$ python regex_code.py
Fail
Process finished with exit code 1
When I run test_regex_code.py test suite I get the following output:
$ pytest test_regex_code.py
============================= test session starts ==============================
platform linux2 -- Python 2.7.12, pytest-3.5.0, py-1.5.3, pluggy-0.6.0
rootdir: /mnt/c/Users/rackspace/code/dummy, inifile:
plugins: helpers-namespace-2017.11.11, flake8dir-1.2.0
collected 1 item
test_regex_code.py .Fail
[100%]
=========================== 1 passed in 0.03 seconds ===========================
Process finished with exit code 0
Any thoughts on what the root cause is for this difference in behavior?

Debugger output not shown

I am running into a problem while running ipdb within a test containing the capfd fixture. A cut down version of the test code is as thus:
import pytest
import sys
def test_foo(capfd):
def foo():
print("Hello World!")
foo()
out, err = capfd.readouterr()
import ipdb
ipdb.set_trace()
assert out == "Hello World!\n"
When I run py.test -s test/test_capfb.py, all the output of the debugger is captured and I see nothing. I can still issue commands (for example the c in the output below) to the debugger but get no output until the whole process finishes its run. A sample of the output is here:
; py.test -s tests/test_capfb.py
============================= test session starts ==============================
platform linux2 -- Python 2.7.5, pytest-2.8.7, py-1.4.31, pluggy-0.3.1
rootdir: /home/usr/repos/junk, inifile: setup.cfg
plugins: bdd-2.16.0, colordots-0.1, cov-2.2.1, html-1.7, pep8-1.0.6, xdist-1.14, catchlog-1.2.2
collected 1 items
tests/test_capfb.py
c <---- I typed this!
--Return--
None
> /home/usr/repos/junk/tests/test_capfb.py(12)test_foo()
11 import ipdb
---> 12 ipdb.set_trace()
13
ipdb> .
========================== 1 passed in 228.90 seconds ==========================
Is there any way to tell pytest to stop capturing stdout/stderr when ipdb is involved?
As a work around, I can use iocapture to capture the output like thus:
import pytest
import sys
import iocapture
def test_foo():
def foo():
print("Hello World!")
out = None
with iocapture.capture() as captured:
foo()
out = captured.stdout
import ipdb
ipdb.set_trace()
assert out == "Hello World!\n"

Python unittest __del__ behaviour wrt modules

I'm writing a Lua wrapper and the highest level of abstraction calls lua_close in it's __del__ method. As far as I can tell every test of this passes except the setuptools test (i.e. regular unit testing works, unit testing w/ setuptools does not). Am I doing something wrong, or is there a bug in setuptools/unittest?
My setup.py:
from setuptools import setup
setup(name="PyLua",
version="0.1",
description="A cffi based lua package.",
packages=['lua'],
author="Alex Orange",
author_email="crazycasta#gmail.com",
license="AGPLv3",
test_suite='test',
)
My lua/lua.py:
from math import sin
class Test(object):
def __del__(self):
print sin(1)
My test/lua.py:
from __future__ import absolute_import
import unittest
def load_tests(loader, tests, pattern):
suite = unittest.TestSuite()
tests = loader.loadTestsFromTestCase(RealCodeTestCase)
suite.addTests(tests)
return suite
class RealCodeTestCase(unittest.TestCase):
def setUp(self):
from lua.lua import Test
self.L = Test()
def testCallStuff(self):
self.assertEqual(1,1)
My test2.py:
import unittest
import test.lua
suite = unittest.TestLoader().loadTestsFromTestCase(test.lua.RealCodeTestCase)
unittest.TextTestRunner(verbosity=2).run(suite)
Results of python setup.py test:
running test
running egg_info
writing PyLua.egg-info/PKG-INFO
writing top-level names to PyLua.egg-info/top_level.txt
writing dependency_links to PyLua.egg-info/dependency_links.txt
reading manifest file 'PyLua.egg-info/SOURCES.txt'
writing manifest file 'PyLua.egg-info/SOURCES.txt'
running build_ext
testCallStuff (test.lua.RealCodeTestCase) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.001s
OK
Exception TypeError: "'NoneType' object is not callable" in <bound method Test.__del__ of <lua.lua.Test object at 0x7fa546ccc350>> ignored
Results of python test2.py
testCallStuff (test.lua.RealCodeTestCase) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.002s
OK
0.841470984808
P.S. Python is CPython-2.7
It appears you can prevent this from happening by providing an explicit tearDown in your test:
class RealCodeTestCase(unittest.TestCase):
def setUp(self):
from lua.lua import Test
self.L = Test()
def tearDown(self):
del self.L
def testCallStuff(self):
self.assertEqual(1,1)
I don't really know why the error occurs, but since the tearDown method prevents the error, my guess is that behind the scenes, setuptools implements its own variant of tearDown, which clears a bit too much, including imports, such as the from math import sin.
The error message indicates sin can't be called, since while the name exists, it has been turned into a None. I can only guess this happens somewhere in that tearDown method implemented by setuptools.

How to exit the script in unittest test case

Here is a sample script that checks for a precondition in the very first test case and my intention is to abort the script if the precondition is not met.
#!/usr/bin/python
import unittest
import sys
class TestMyScript(unittest.TestCase):
def test_000_prerequisite(self):
a = 0
if not a:
sys.exit()
return
def test_001_test1(self):
print "Inside test 1"
return
def test_002_test2(self):
print "Inside test 2"
return
if __name__ == "__main__":
unittest.main()
However, the sys.exit() only exits from the individual test case of the suite. It doesn't exit the whole script.
I understand that unittest treats each test case individually which is why any exceptions caused by any testcase are handled by the test runner and it proceeds to the next test case.
But I want the script to kill itself, how do I do that?
Here is the output of my script:
./temp.py
EInside test 1
.Inside test 2
.
======================================================================
ERROR: test_000_prerequisite (__main__.TestMyScript)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./temp.py", line 9, in test_000_prerequisite
sys.exit()
SystemExit
----------------------------------------------------------------------
Ran 3 tests in 0.000s
FAILED (errors=1)
My guess is that I have to mess around with TestRunner and kill the script if a test case returns some signal. But not sure how to really achieve it.
Here is the answer:
Stop testsuite if a testcase find an error
Here is the change I need to make when calling unittest.main(). The failfast keyword argument stops the script after the first failure.
if __name__ == "__main__":
unittest.main(failfast=True)
p.s. failfast keyword argument is only available for python 2.7+
p.p.s. you can also use failfast on unittest.TextTestRunner()

Resources