Thursday, 16 February 2012

Non-Testable Acceptance Criteria

Non-Testable Acceptance Criteria

I recently discovered the INVEST criteria for user stories. A quick recap - user stories should be:


Setting aside the interesting tension between Independent and Small, I have a difficulty with Testable.

Non-testable stories

A canonical example:

The software should be easy to use.
thanks to Vabihav

To make it testable, this might get broken down:

A novice user can is able to complete common workflows without training.

For testability that might become: "80% of novice users can use the core functionality without instruction in 5 minutes." OK that's not automatable, but at least it's objectively testable. But does it capture the business value?

A real-world example

The promo area on our homepage contains beautifully designed links to 6 articles in our CMS. We have to select those 6 articles in the CMS. Consider three ways to do this:

  1. We open each of the 6 most important articles and click the "make homepage promo" link.
  2. The CMS contains a checkbox list of the articles, and we select 6 of them.
  3. The CMS contains 6 empty slots, each of which we populate with an article title.

1. is plainly impractical, and almost certainly fails our test.

2. is a clear UX and can certainly be used by novice users. It passes the test. And it looks nice and easy to use in development. But we have 2000 articles and more every week. Selecting 6 of them in an alphabetical list is anything but easy to use.

3. might be nice and easy, if coupled with a decent search facility (eg auto-complete). The development team implemented 2.

Tightening up the acceptance criterion

80% of novice users can set the right number of homepage promos without instruction in 5 minutes, for any set of 6 top articles, for a CMS load of up to 10000 articles.

Advantages of this approach:

  • It's testable
  • It's specific in a dispute (the legal department will love it)

Disadvantages of this approach:

  • I've no idea whether I've missed any ways the statement needs tightening up
  • It's plainly absurd

Frankly I think we're better off with "The software must be easy to use" and an adult conversation about what that means for the user group.

Some business needs just aren't necessarily black-and-white testable.


  1. You might be interested to have a read about Planguage:

    It’s used more in a manufacturing context in this paper, but mirrors some of the points you make in your post.

  2. Hello Guy,

    It's been two years, but perhaps the discussion can still be carried off (all of us are wiser in the meantime - me from my post back in 2007). You are right on point - and not all stories are indeed testable - or provide a directly measurable business value either. So, let's say, the INVEST model or something similar can work very well for what can be categorized as functional requirements - as compared to what falls under non-functional requirements.

    Non-functional requirements (NFRs) are harder to test, describe, objectify, and measure. Your example above is for instance an NFR.

    1. You're absolutely right Vabihav that's it's an NFR. A hard one to describe too. If many NFRs can be summed up quantitatively 'must respond in x milliseconds', 'must have y up-time' etc, usability is a harder case. Of course an Agile project addresses this by people having conversations!