Thursday, 16 February 2012

Non-Testable Acceptance Criteria

Non-Testable Acceptance Criteria

I recently discovered the INVEST criteria for user stories. A quick recap - user stories should be:


Setting aside the interesting tension between Independent and Small, I have a difficulty with Testable.

Non-testable stories

A canonical example:

The software should be easy to use.
thanks to Vabihav

To make it testable, this might get broken down:

A novice user can is able to complete common workflows without training.

For testability that might become: "80% of novice users can use the core functionality without instruction in 5 minutes." OK that's not automatable, but at least it's objectively testable. But does it capture the business value?

A real-world example

The promo area on our homepage contains beautifully designed links to 6 articles in our CMS. We have to select those 6 articles in the CMS. Consider three ways to do this:

  1. We open each of the 6 most important articles and click the "make homepage promo" link.
  2. The CMS contains a checkbox list of the articles, and we select 6 of them.
  3. The CMS contains 6 empty slots, each of which we populate with an article title.

1. is plainly impractical, and almost certainly fails our test.

2. is a clear UX and can certainly be used by novice users. It passes the test. And it looks nice and easy to use in development. But we have 2000 articles and more every week. Selecting 6 of them in an alphabetical list is anything but easy to use.

3. might be nice and easy, if coupled with a decent search facility (eg auto-complete). The development team implemented 2.

Tightening up the acceptance criterion

80% of novice users can set the right number of homepage promos without instruction in 5 minutes, for any set of 6 top articles, for a CMS load of up to 10000 articles.

Advantages of this approach:

  • It's testable
  • It's specific in a dispute (the legal department will love it)

Disadvantages of this approach:

  • I've no idea whether I've missed any ways the statement needs tightening up
  • It's plainly absurd

Frankly I think we're better off with "The software must be easy to use" and an adult conversation about what that means for the user group.

Some business needs just aren't necessarily black-and-white testable.