During my career as a software engineer so far, I’ve tended to err on the side of caution when writing tests, taking the view that “more is more”. I think it’s usually better to have one more test than not to have it, even though there is of course a cost to each extra test. Sometimes this attitude results in tests that I admit might be a bit excessive in terms of what they cover.
I worked in permanent roles at various start ups in London for about eight years, and in 2021 I made the switch to contracting and freelancing. I knew that as a contractor I would be thrown into all sorts of projects with different levels of design, development and maintenance. One thing that has struck me from taking on this kind of work is that I actually feel a vindication of the “excessive” testing described above.
For example, in a permanent role in the past, I developed a system involving an XML message processor which needed to verify XML signatures on the incoming messages. At the time, I wrote tests that generated a fresh private key and certificate, signed an XML message with the key, configured the system under test to use the certificate, and confirmed that it could verify the signed XML and also perform whatever business logic was being covered in each particular test case. A lot of this might seem a bit excessive or unnecessary for testing a fairly bog standard system.
For example, it might have been easier to not bother with the XML signature and verification in tests, as that part was mostly just calls to a library anyway. Alternatively, it might have been easier to just generate a test private key and certificate manually and perhaps keep it in a password manager somewhere or even commit it into the repo as it’s only for testing purposes.
However, having the test code generate fresh keys and certificates to use on the fly does have some benefits. It exercises more of the system from start to end. The tests confirm the configuration of the system and its use of the library, as well as the business logic. If we need to upgrade the library or switch to another one, the tests will be able to validate that change with quite a lot of confidence. Having the key and certificate generation code committed in the repo is also a form of documentation of how that works, and makes it easier to modify the tests in future if we switch to a different signing algorithm or key strength. It also means the tests “just work”, with no set up required other than running the tests.
With more detailed and extensive tests, it’s less mysterious what is happening in the system and in the tests, and that can be useful for people working on both of those in the future. It can save money for the company, because future development can happen quickly, confidently and safely by leaning on these kinds of tests.
Coming back to the contracting work, the above was somewhat vindicated recently when I worked on a similar system involving XML messages with signatures. In this case, whoever built the system in the past hadn’t included tests that covered the signature verification part of the system. Lo and behold, by the time I came in to work on this project, the changes that were needed involved modifying the signature verification as well as how the XML structure was parsed. It took a lot of digging around in sparse documentation, trawling through old commits and tracking down people who might have some more knowledge. This was the only way to gather all of the necessary information to make the changes safely and then deploy them with confidence. A lot of the time and effort could have been saved had more extensive tests been put in at the time of development.
In summary, I’d say this confirms that it’s always worth considering other people or your future self when deciding how detailed and extensive to make your test cases. More often than not, it will pay off to cover more of the system with explicit tests.