Gian Sampson-Wild explains her problem with testability in the W3C‘s Web Content Accessibility Guidelines 2.0 (WCAG). Testability is the ability of, in our case, software or content, to be testable either by machines or humans.
One of the properties of a good establishment of tests is that it is not dependent on the machine’s or the human’s preference. The test, in itself, is complete, reproducible and the outcome is dependent only on the subject of test, so that it does not vary by who is testing or using it. This removes the burden on the tester and makes it consistent across a variety of them.
Gian illustrates that WCAG has dropped certain success criteria because they are not testable.
This is a generic problem with testability. In spite of many developments in the testing discipline, we have not been able to represent meaning, purpose and behavior within the test itself. Sometimes the success criteria are not easily measurable. And so they are left to human interpretation. While the criteria is important, it is not testable, and gets omitted from the tests a lot of times. Which means that we do not perform certain checks, in spite of being important, because we cannot test them.
I do not think the solution to this is to do away with testability. Roger Johansson gives an instance where testability can be really useful. We should acknowledge that some of the checks have be performed cognitively, with human participation. Which means that the human should be sufficiently knowledgeable and skilled to understand the problem and solution both. Also, the outcomes of such checks can be subjective and a feedback loop from the users can help consolidate it.
Should testability determine what checks we should do, or should their importance and validity? I think it should be the latter, but with explicit acknowledgement that they require a different approach.