Monday, June 17, 2013

Using the Agile Process makes testing activities more difficult, not easier

One the innovative, great processes for reducing risk of achieving any goal, improving the efficiency of work teams, and reducing resource investment when business decisions change, and one being adopted en mass currently amongst companies that develop software is the Agile Process.

I love Agile process and believe it is absolutely the best way to develop software and protect business from over-investment (resources and time) and risk in a dramatically changing business climate.

However, the ideal (and most practiced) Agile sprints last 2 or 3 weeks. 10 - 15 working days. Some companies, especially SAAS companies, deliver the software to the production environment after each sprint. This is really great for the end customer; they get a new update of the product every 2-3 weeks and since the changes are "incremental", it helps reduce the massive changes in software functionality and user experience that used to accompany major software updates (does anyone recall their user experience going from Windows 2000 to Windows XP to Windows Vista or Windows 7 to Windows 8?).

The compressed timeframes, and dependency of the development resources to complete the new functionality or software changes greatly reduce the amount of time for testing activities. Testing resources have no control as to when the software changes get finished and ready to test. In most cases, testers have multiple stories to test in each sprint. This scenario creates testing schedule pressures. New changes must be tested, plus the regression testing.

In most cases, I have experienced that developers will not compete new functionality and be ready for testing into the testing environment until 50-70% of the entire sprint time duration.

This leaves 50-30% of sprint time for testing activities per story. 3 to 5 working days for a 10 day sprint duration possibly for testing each story implementation.

During this time, bugs may or may not be found. If they are, bug reports are created and assigned for fixing. Re-testing must be done for bugs fixed. Regression is run many times over again to verify the product is still working as expected. For large software development tasks, bug finding might be greater in occurrence.

Testing difficulty, and time to test, occurs between white and black box testing. The company needs to be aware of what impact this has on testing it's software in short time durations. Risk varies amongst functional goal to be implemented.

Using the Agile process for software development puts a lot of schedule pressure on the testing time to validate the Acceptance Criteria of the software for each story. This could be between 3-5 days. This is why so many companies (almost every company) are putting testing emphasis on automated testing to assist the team in testing-related activities. It is a wise choice, but the organization needs to realize the stress that the Agile process puts on testing activities and be sure to adequately provide testing resources to ensure acceptance criteria for new stories and successful regression testing is accomplished and all required tests to be run pass.


  1. If you think development spans most of your sprint, then maybe you should revisit the way the user stories are created. If you break down your epic in small enough user stories, it might be FULLY possible to complete each user story within 2 days.

    It generally depends on the way they are broken down.

    Moreover, team might allow dev completed items to slip into the next iteration for QA activities. We have been doing this for our current project and its working well for us.

    1. Rumadak,

      I agree that you can let some feature complete items to slip into the next iteration. However, this may set the next iteration up for failure as you start the new iteration with negative equity. Really, the only way to make this work is to obviously evaluate the risk if this feature were to get released into the wild, and also to make considerations for the next iteration that will be impacted by the initial testing that wasn't initially planned.

      Also, be clear that this is an exception and not the rule.

  2. Micheal,

    It sounds like what you're describing is more waterfall than agile. In an effective agile environment, you want to start testing activities as soon as the developer starts working on the feature. You should be involved in all the design and implementation meetings, and work with the developer on testing strategies for unit and functional tests.

    I believe that both developer and QE are responsible for delivering a feature and If the testing isn't done, then the developer needs chips in and writes tests or execute manual tests. When the testing is completed then the feature is moved to done. This will helps to keep developers from getting too far ahead.

  3. Hi Rudnak and Jason, thanks for the great comments.

    In response to Rudnak: I would say that it is difficult to get the development team to complete all story tasks in 2 days, at least I have not yet been successful in convincing dev teams of the importance of this. Usually we do move stories into the next sprint if dev is not compkete by day 7 of a 10-day sprint because not enough testing will be done to verify acceptance criteria for new story implementation and regression.

    In response to Jason: I understand that the perception is waterfall method adjusted into an Agile process, but it really is not. There are certain tasks that need to be done as a team to allow for shared responsibility of test cases and acceptance criteria and then waiting for the developer to complete the development task and get the software ready for testing. There are many cases where testing cannot be done until the software is ready to be tested. In some case, you are right, some mock testing can be done in parallel until software is complete, but that is not always the case.