Systematic study of test automation pyramid

by Will   Last Updated August 07, 2017 17:05 PM

The Test Automation Pyramid concept was conceived by Mike Cohn, popularized by Martin Fowler, and blogged about by tech companies such as Google.

From Cohn's "The Forgotten Layer of the Test Automation Pyramid":

Test Automation Pyramid image by Mike Cohn of Mountain Goat Software, originally obtained August 7th, 2017

At the base of the test automation pyramid is unit testing. Unit testing should be the foundation of a solid test automation strategy and as such represents the largest part of the pyramid...

Automated user interface testing is placed at the top of the test automation pyramid because we want to do as little of it as possible...

Without service-level testing to fill the gap between unit and user interface testing, all other testing ends up being performed through the user interface, resulting in tests that are expensive to run, expensive to write, and brittle.

Is there empirical research or a systematic study validating (or debunking) the Test Automation Pyramid concept? Beyond anecdotal evidence. I realize the Test Automation Pyramid concept, at least defined by Cohn, is not written as a formal scientific theory, but it does contain or imply hypotheses that could be tested. Specifically, is there empirical research into any of the following metrics, which arise from the pyramid concept?

  • relative marginal utility of any 2+ sets of tests (i.e. layers of the pyramid: unit tests, service tests, user interface tests), or
  • absolute or relative return on investment of any 2+ sets of tests, or
  • measure (or absence) of diminishing marginal return of user interface tests

Related Questions

Using parameters in automatic integration tests

Updated May 27, 2015 22:02 PM

Manipulate database from within integration test?

Updated December 12, 2017 09:05 AM

Integration tests, but how much?

Updated February 28, 2017 08:05 AM