Unit Testing¶
Unit testing ensures that the general execution of the module code is correct. It is the fastest way to test, requires no F5 products, and is what the developers recommended be used when doing the initial development of a module.
While both forms of testing are important, the unit tests will not tell you if you have a fully functioning module. Functional tests are the only things that can provide anything close to this assurance. Nevertheless, unit tests are required by the F5 module developers. When contributing code to upstream Ansible, only unit tests may be submitted to the core product. This is because the Ansible developers do not have the ability to test F5 products.
Filesystem location¶
All unit tests are located in the following directory:
tests/units/modules/network/f5/
Changing to this directory will show a number of files that are named after different modules. For example:
-rw-r--r-- 1 trupp OLYMPUS\Domain Users 3507 Jan 24 17:20 test_bigip_config.py
-rw-r--r-- 1 trupp OLYMPUS\Domain Users 4138 Feb 15 11:21 test_bigip_configsync_action.py
-rw-r--r-- 1 trupp OLYMPUS\Domain Users 16007 Feb 22 09:27 test_bigip_data_group.py
-rw-r--r-- 1 trupp OLYMPUS\Domain Users 12692 Jan 24 17:20 test_bigip_device_connectivity.py
-rw-r--r-- 1 trupp OLYMPUS\Domain Users 4323 Jan 24 17:20 test_bigip_device_dns.py
-rw-r--r-- 1 trupp OLYMPUS\Domain Users 5547 Jan 24 17:20 test_bigip_device_group.py
These files are the unit test files themselves. The test/units/modules/network/f5/
directory also includes another
directory of interest:
fixtures/
This directory contains a number of static data files that are used by the different unit tests.
As will be seen later during test development, the files in the fixtures/
directory can be
easily loaded by using functions in the unit test file. Examples of fixture files are:
-rw-r--r-- 1 trupp OLYMPUS\Domain Users 912 Nov 14 19:22 load_tm_sys_syslog.json
-rw-r--r-- 1 trupp OLYMPUS\Domain Users 893 Jan 24 17:20 load_tm_sys_ucs.json
-rw-r--r-- 1 trupp OLYMPUS\Domain Users 969 Nov 14 19:22 load_vcmp_guest.json
-rw-r--r-- 1 trupp OLYMPUS\Domain Users 808 Nov 14 19:22 load_vlan.json
-rw-r--r-- 1 trupp OLYMPUS\Domain Users 510 Dec 18 18:37 load_vlan_interfaces.json
Note
Fixture files are often in JSON format, because the REST API returns information in this format. Unit tests use these REST response payloads to verify the tests’ correctness.
Tutorial module implementation¶
The implementation of the tutorial module’s unit tests can be found here. Additionally, you
will need to have the following fixture files downloaded and placed in the fixtures
directory.
General things to know about unit tests¶
Unit tests for the F5 Modules for Ansible are written using pytest.
For pytest
to be able to run your unit tests, your tests must follow these rules.
- Classes, if used, must start with the string
Test
. Spelling must be exact. - Methods or functions containing tests must start with the string
test_
. Spelling must be exact. - Unit tests do not need to do any form of cleanup. Pytest handles cleanup for you automatically.
Writing a unit test¶
Let’s take the time now to write the unit tests for the module that was developed in this
tutorial. During the initial stubber run, the inv
command produced a unit test file that
included a sampling of what will need to be done.
Let’s touch on those boilerplate blocks before investigating the actual testing code.
Import block¶
At the top of the unit test file (like at the top of many Python source code) there are a series
of import
statements. These tell Python to include different bodies of code that come either
pre-installed with Python, or as separate packages that you should have installed.
Note
All of the dependencies for typical F5 modules for Ansible are pre-installed for you in the development Docker containers that were mentioned at the beginning of the tutorial.
Some of the imports of interest are:
- The SkipTest import
- The dev versus prod import
First, the SkipTest
import. This import is defined as such:
from nose.plugins.skip import SkipTest
if sys.version_info < (2, 7):
raise SkipTest("F5 Ansible modules require Python >= 2.7")
The purpose of this import is to declare that the F5 modules require Python versions greater than, or equal to, 2.7. Over time, it is expected that this check will change to require Python 3 and beyond. Therefore, be sure to keep aware of this and do not find yourself in a situation where you are unable to upgrade either your operating system, or Python, to later versions.
Next, the dev/prod import. This import is defined as such:
try:
from library.bigip_policy_rule import Parameters
from library.bigip_policy_rule import ModuleParameters
from library.bigip_policy_rule import ApiParameters
from library.bigip_policy_rule import ModuleManager
from library.bigip_policy_rule import ArgumentSpec
from library.module_utils.network.f5.common import F5ModuleError
from library.module_utils.network.f5.common import iControlUnexpectedHTTPError
from test.unit.modules.utils import set_module_args
except ImportError:
from ansible.modules.network.f5.bigip_policy_rule import Parameters
from ansible.modules.network.f5.bigip_policy_rule import ModuleParameters
from ansible.modules.network.f5.bigip_policy_rule import ApiParameters
from ansible.modules.network.f5.bigip_policy_rule import ModuleManager
from ansible.modules.network.f5.bigip_policy_rule import ArgumentSpec
from ansible.module_utils.network.f5.common import F5ModuleError
from ansible.module_utils.network.f5.common import iControlUnexpectedHTTPError
from units.modules.utils import set_module_args
The purpose of this import block is the same as the purpose of a similar import block that
existed in the actual module code. The content in the try
section attempts to import
development code (code in the f5-ansible Github repository) and if that fails, it will attempt
to load product code (code in the upstream Ansible Github repository).
This differentiation is used by the F5 module developers to allow for development out-of-band of the upstream Ansible product.
Therefore, this import block serves a similar purpose to the module’s block. The major difference is that the things that are imported are different. The unit test is interested in importing the classes that are defined in the module. It will test these classes later.
Note
An ongoing disagreement exists among developers as to what constitutes a “unit” for test. F5 considers the “unit” under test the class, not the methods of the class.
Fixture setup¶
After the import block, the fixture setup block can be found. It is implemented like so.
fixture_path = os.path.join(os.path.dirname(__file__), 'fixtures')
fixture_data = {}
def load_fixture(name):
path = os.path.join(fixture_path, name)
if path in fixture_data:
return fixture_data[path]
with open(path) as f:
data = f.read()
try:
data = json.loads(data)
except Exception:
pass
fixture_data[path] = data
return data
The first assignment in this block is used to declare two things:
- Where the fixtures can be found
- A cache for the fixtures to prevent re-reads from disk
After the assignment statements comes the definition of the load_fixture
function. This
function is what is responsible for using the two assignments above.
Parameter unit tests¶
The first set of unit tests that are stubbed (and the tests which are likely to be written
first) are the Parameters
class unit tests.
The parameters tests are typically defined by a class named TestParameters
. The purpose of
this class is to test the different combinations of arguments that one can send to the different
parameter classes (ApiParameters
and ModuleParameters
).
Usually, you will provide the class an argument, and then assert that some property of the
Parameters
class is equal to an expected value.
Using the module being developed as an example, refer to the code below.
def test_module_parameters_policy(self):
args = dict(
policy='Policy - Foo'
)
p = ModuleParameters(params=args)
assert p.policy == 'Policy - Foo'
As stated previously, the test sets some property to some known value. It then creates an
instance of the Parameters
class under test–in this case ModuleParameters
. It provides
the defined arguments to this class in the same way that the Ansible module does.
Finally, it performs an assertion to check that some expected @property
is equal to some
expected value.
All of the Parameter
tests resemble this format.
F5 imposes no limit on the number of tests you are allowed to write. The general rule of thumb is to follow code-coverage reports to determine what tests are missing.
ModuleManager unit tests¶
The second set of unit tests that will be stubbed out are the ModuleManager
tests. There
may be either a single class, or multiple classes, for testing the module manager(s). For
instance, if the Ansible module under test is a factory module (such as several GTM modules)
there may be two classes for module manager tests.
The basic definition of a ModuleManager
test class is shown below.
class TestManager(unittest.TestCase):
def setUp(self):
self.spec = ArgumentSpec()
In the above stub, a method names setUp
is defined. This is typical of all manager test
classes. The job of this method is to, (according to the unittest documentation)
…define instructions that will be executed before and after each test method
In this case, the unit tests will require an ArgumentSpec
definition before they can run.
By putting this definition here, it can be used in all of the remaining unit tests in the class.
Actual tests¶
The actual unit tests of the ModuleManager
should include (at a minimum) the following
tests:
- A creation test
- An update test
- A deletion test
- An idempotent creation test
- An idempotent update test
- An idempotent deletion test
You are unlikely to find all of these tests for every module that exists, but it is still a goal of module development to produce this minimum set of tests.
Below is the implementation of a creation test.
def test_create_policy_rule_no_existence(self, *args):
set_module_args(dict(
name="rule1",
state='present',
policy='policy1',
actions=[
dict(
type='forward',
pool='baz'
)
],
conditions=[
dict(
type='http_uri',
path_begins_with_any=['/ABC']
)
],
password='password',
server='localhost',
user='admin'
))
module = AnsibleModule(
argument_spec=self.spec.argument_spec,
supports_check_mode=self.spec.supports_check_mode
)
# Override methods to force specific logic in the module to happen
mm = ModuleManager(module=module)
mm.exists = Mock(return_value=False)
mm.publish_on_device = Mock(return_value=True)
mm.draft_exists = Mock(return_value=False)
mm._create_existing_policy_draft_on_device = Mock(return_value=True)
mm.create_on_device = Mock(return_value=True)
results = mm.exec_module()
assert results['changed'] is True
The basic design of a test follows these steps:
- Define some parameters using
set_module_args
- Create an instance of
AnsibleModule
- Create an instance of
ModuleManager
- Stub out all of the methods that communicate with the API using simple
Mock
classes - Call
exec_module
to drive the test - Assert changes on the result
Most of the above is self-explanatory, but the fourth item on the list warrants some explanation.
The purpose of the F5 Ansible module unit tests is to confirm that:
- a series of arguments
- invokes a known series of methods
- to produce a known result
That’s it. You do not need to mock the actual API calls. The best way to test actual API calls is via functional tests.
Therefore, to put it simply, the F5 module unit tests are there to test drive code execution paths.
Using the above as an example, given the parameters that are set, if the ``Mock``ed calls are called during execution of the module, then the module will logically return the asserted result.
If, however, a problem exists in the logic of the module such that a different code path
is taken than expected, then pytest
will fail because it will attempt to call an API
method. This failure should pique your interest because it means there is a bug in the module.
Unit tests are meant to confirm code path execution. Nothing more.
Conclusion¶
This section introduced you to tests, showed how and where they are laid out, and introduced
you to writing two forms of test: a Parameters
test and a ModuleManager
test. With these
tools, the remainder of the work falls on the shoulders of the developer. Ansible will
run these tests as part of their basic test suite. Therefore, it is important that they are:
- Correct
- Fast
Hundreds of tests exist in the F5 Ansible code-base. If the F5 unit tests are slowing down the total execution time of the test suite (beyond reason of course) then this should be considered a bug and fixed.
In the next section, the concept of integration tests will be explored in greater depth. Integration tests are the most important tests that can be run because they confirm or reject the correctness of a module.