Testing as a Creative Endeavor

Recently, I did a Google search on “agile test automation”. I’m struggling to figure out how, if at all, test automation fits in to a very agile and loosely structured software development model. I come from a world of three- to twelve-month software release cycles. Where I work now, we release new code to production at least once a week. It’s exciting and invigorating.  What I most enjoyed about writing test automation was the challenge of building in extensibility. Without extensibility, automation is one-dimensional: it does one test or set of tests and reports what it finds. In agile development, software changes in small and not so small ways almost constantly. As the software changes, simplistic automation can’t change with it (at least not without more human intervention), and as such has limited value. By the time we make the test code robust enough to add sustained value, the product it’s designed to test has probably morphed two or three times over. Is robust automation being fast-paced into obsolescence, and if so, is there a new, nimble automation approach to replace it? Or is my search string an oxymoron?

In pursuing these questions, I stumbled upon articles slightly off topic but decidedly more interesting, about the mysterious process we call “exploratory testing”. This is the farthest thing from automation. It involves creative thinking about how you’ll approach the software, some knowledge of the target market, a notebook to record findings, and a chunk of uninterrupted time. In addition to these basics, skilled testers bring to the table some knowledge of the risk areas, and my favorite, intuition. The thrill of the hunt is on: you don’t know exactly what you’re looking for, but you’ll know it when you find it.

You may have heard software testing referred to as an art. I was delighted to discover Jonathan Kohl’s article likening exploratory testing to music. He offers that key both to successful software testing and great music is the proper balance of tension and resolution. Without this balance, he says, neither testing nor music is at its best, and both are boring. James Bach defines exploratory testing as “simultaneous learning, test design, and test execution.” Remove the word “test” from this definition, and it sounds a whole lot like musical improvisation. Testing — like music — as art. Nice. I agree with my test junkie colleagues that exploratory testing, rather than “don’t color outside the lines” scripted testing, keeps us engaged and nets better results: more bug finds, in shorter test sessions.

So the search didn’t lead to what I thought I was looking for. I still haven’t satisfied my question about the place of automation in agile development. But, like exploratory testing, following the unintended path led to the richer, more interesting discovery of like-minded thinkers in the field.

Advertisements

9 thoughts on “Testing as a Creative Endeavor

  1. DaveM, the closest I’ve seen to ‘matching’ development’s cycle would be to automate the tests in parallel with their development, which of course will mean that there will be some tweaking needed to your scripts once integration complete occurs, as you’ll undoubtedly need to correct a few libraries/scripts once you have a product to test against.

    @ Soapstone, we had 45-60 day cycles, and we broke them down by:

    1-2 week test plan authoring
    3-4 weeks test implementation (java/python/etc)
    1 week fixing the script up at the end once dev is close to releasing code and we can see where there were disconnects between SQA implementation and dev implementation

    then, dev’s integration complete (sqa handoff) occurs, and we spend

    2 weeks running the tests via automation as well as manual GUI testing.

    Nightly runs allow us to catch regressions introduced as dev attempts bug fixes, and at the end of the 2 weeks or so, the iteration is considered complete.

    I can’t help but think that the way you describe must snowball. It seems like if the automation isn’t fully debugged and running, you cannot call the iteration ‘complete’ or release it, or else the ‘leftover work’ will accumulate more and more after each iteration, until you’re an entire iteration behind. It must ‘wrap up’ before the cross functional teams move to the next iteration.

    Best regards,

    -mike landman

    Like

  2. In our new Agile-like environment our automation lags behind the development cycle but a small amount. We found that what works best is utilizing the automation as purely regression testing of all previous functionality. We are striving to build some of the automation during the Agile-like cycle (our Agile cycle is 2-3 months and isn’t really Agile by anything other than name; but I digress) and then complete that automation after the release to production. Parallel to the development of automation is manual testing of all new functionality.

    This method seems to be working. We’re ten or so months into this new development style so the jury is still out deliberating.

    – .dave.

    Like

    • Hi Dave –

      The way your group is doing things makes a whole lot of sense. The manual testers can also serve as suppliers of great test plans on which to base the new automation.

      Thanks for reading.

      Karen

      Like

  3. Hi Karen, WOW you know your stuff. Never realized what intricate details went into programing software — I’m one of those who definitely utilizes the IT department. Great article!

    Like

  4. Beebles Babbles is the first name that came to my mind. But, only a select few could truly understand.
    Still Smiling …
    The Fatman

    Like

  5. I have no earthly idea what most of all that you wrote about means except for one thing, technology sure the heck is moving along rapidly, more so than I.
    Smile ….
    The Fatman

    Like

  6. I’m impressed. I do get it, though. A human will use the software and discover the bugs. The human may or may not have a structure for doing this. Automation would be structured; the human element will discover more. There may be a place for both methods in testing.

    Like

  7. Good stuffs Karen
    It says a lot that you can write about this in a way that someone who understands nothing of why anyone would want to code software (“isn’t there enough out there already?”) can start to relate–
    Jake A

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s