MS RFC 129: Update msautotest to use Pytest¶
- Last update:
This RFC proposes to use the Pytest framework, to replace the current ad-hoc Python testing infrastructure for its msautotest regression test suite.
MapServer uses a ad-hoc Python testing infrastructure, similar to what GDAL used up to GDAL 2.3 (starting with GDAL 2.4, pytest has been adopted)
While it is fullyfunctional, it is non-standard. Using a standard testing framework brings more familiarity to developers, more capabilities, and less boilerplate for tests under msautotest/mspython, due to the ability of using assertions instead of the adhoc post_reason()/return 'fail' current combo.
Pytest offers, among many other features:
ability to run a subset of tests based on expressions. For example, "pytest -k wfs" to run all tests containing wfs in their name.
standard solution for skipping tests, expected failures, etc.
adding custom options with documentation
Tests under msautotest/mspython are modified to be compatible of pytest: methods are prefixed with test_, adhoc post_reason()/return 'fail' replaced by simple assert statements
msautotest/pymod/mstestlib.py, the workhorse to run tests based on mapfiles, is modified to use pytest infrastructure
Python mapscript tests, under mapscript/python/tests, are not modified. They use Python unittest module, which can be used transparently with pytest.
It is now possible to just run pytest from msautotest to run all tests. Or pytest misc renderers to just run tests in those directories.
Backwards Compatibility Issues¶
None for users.
What is not changed:
the syntax of test MapFiles is not changed
the current run_test.py launcher scripts in wxs/, gdal/, etc. subdirectories are preserved and are wrapper scripts over pytest.
A few changes:
pytest must be installed: "pip install -r requirements.txt" from msautotest/
a few changes in options: the strict, valgrind and run_under_asan options are now prefixed with double hyphen
https://mapserver.org/development/tests/autotest.html will be updated
Ticket ID and references¶
Pull request available at: https://github.com/MapServer/MapServer/pull/5984
Adopted with +1 from PSC members SethG, EvenR, JeffM, SteveL, JukkaR, JeromeB, StephanS, TomK