Better Software - September 2008 - (Page 27) I first heard of “specification by example,” also known as “inductive inference of programs from input-output examples,” from Brian Marick, but it has been around for some time. Much of the original writings, however, were focused on automatically generating code from the examples. We’re not quite there yet with ATDD/TDD, but that isn’t to say we won’t make it a reality in the future. Landmark 3: It’s Not About Testing That’s it, then; it’s all about specification by example—right? The whole point is to elicit and comprehend the customer’s requirements, deriving the logic and rules from specific examples of how the system is expected to function. Is it only about discovery of detail and creation of a specification that emphasizes the concrete over the abstract? So, the fact that we express these examples as automated tests and that these tests help find regressions in the system is merely a useful side effect of the process. Or is it? Well, in short—no. When you see this landmark, I want you to turn around and double back until you see landmark 4. volves testing. From the moment we start discussing the acceptance criteria for a user story and even while we are automating them, I find myself visualizing the application. Mentally I am attempting different scenarios, thinking, based on the examples provided so far, “How would I expect the system to respond if … ?” or asking the customer “If I do this, what would you then expect?” I’m thinking creatively; I’m applying testing techniques to generate more potential examples or reduce overwhelming numbers of examples into a more comprehensible set, and none of this is scripted or automated. I’m learning as I discover more. I’m creating new test ideas or applying test-design techniques to identify another concrete example to ask about. Sound familiar? Indeed, this is very much like exploratory testing. In fact, discussions with exploratory testing expert James Bach led us both to conclude that this part of ATDD is a form of exploratory testing—just not as most people know it. It explores the idea rather than the implementation, avoiding the introduction of many of the bugs that would otherwise not be discovered until the code is deployed to testers. Landmark 4: It Is Also About Testing Martin Fowler provides a nice analogy on this subject in a blog post about specification by example [2]: These days it’s terribly unfashionable to talk about Test Driven Development (TDD) having anything to do with testing … I do think that having a comprehensive automated test suite is more valuable than the term “side-effect” implies. Rather like if someone offered me a million dollars to hike up a mountain. I may say that the main purpose of the hike is the enjoyment of nature, but the side-effect to my wallet is hardly trivial. So, ATDD is about requirements elicitation, and it is about testing. Just because the first benefit you encounter is requirements elicitation doesn’t necessarily mean that it is more about requirements than testing. There is another rarely discussed aspect of ATDD in practice that also in- Landmark 5: Executable Specifications & Automated Validation Barry Boehm’s book Software Engineering Economics [3] contains a clear distinction between validation and verification: Validation: Am I building the right product? Verification: Am I building the product right? The right product is a product that meets the real requirements. The real requirements are the things that people will actually need to do with the system once they can use it. Traditional specifications (e.g., UML models, use cases, naturallanguage functional specifications, and the like) attempt to model our anticipation of these real-world requirements in a conceptualizing and generalizing way. I say generalizing because the rules within the specification contain variables rather than specific values—i.e., these rules apply generically for any relevant value used in place of the variables. www.StickyMinds.com Of course, we assume that such generalizing specifications weren’t plucked out of thin air and were derived from concrete examples of anticipated usage in the real world. So on projects that derive tests (or examples of anticipated usage in the real world) from such a generalizing specification that was itself derived from examples of anticipated real-world usage is at best verification against the abstraction and at worst the software development equivalent of the “Telephone Game.” The very nature of such generalizing specifications makes it difficult, if not impossible, to use them in any effort to validate the software against its requirements. So, testing our system against a conceptualization of anticipated requirements can’t tell us if we are building the right product. It can only tell us if the product matches the conceptualization. It can only tell us if we are building the product right. Let’s also not forget that a significant reason for conceptualizing software formally on paper before writing the code is that software processes and technologies of old made software hard and expensive to change. If the technology is designed to make code easy to change and the process is designed to allow for the iterative discovery of requirements by making code safe and inexpensive to change, then formally modeling or conceptualizing on paper becomes of limited value. In fact, due to the additional maintenance overhead, it becomes a hindrance. ATDD starts with concrete examples of anticipated real-world usage and automatically evaluates the software against those examples. Short of mind reading and time travel, this is about as close as we can get to validation—as close as we can get to knowing if we are building the right product. Landmark 6: It’s an Automated Test, Jim, But Not as We Know It Because in ATDD our suite of automated acceptance tests is an executable requirements specification of sorts, this affects how we express the tests. Typical automated test tools just don’t cut it. I learned this the hard way on my first XP project in 2000. Kevin Lawrence gave me (and a SEPTEMBER 2008 BETTER SOFTWARE 27 http://www.StickyMinds.com
For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.