Automating repetitive
procedures can provide
real value to software
development projects. In
this article, we will
explore the value of and
barriers to automation
and provide some
guidance for automating
aspects of the
development process.
Although few experienced
developers and project
managers would argue the
merits of automating
development and testing
procedures, when push
comes to shove many
teams place a low
priority on implementing
automated processes. The
result is usually that,
if automation is
considered at all, it is
given lip service early
in the project life
cycle, but falls quickly
by the wayside.
Experience teaches us
over and over again that
trying to run a "simple"
project by implementing
a series of "simple"
manual protocols, backed
by "simple" written
(sometimes, even just
verbal) instructions,
just doesn’t work well.
Even so, many of us
still tend to allow
ourselves to start the
next project with the
thought that the manual,
protocol-based method
will "do just fine."
After all, aren’t we all
professionals? Can’t we
read a set of simple
instructions and just be
disciplined enough to
follow those
instructions when the
time is right? Isn’t it
a waste of time and
money to invest in
automating procedures
for such a small
project? The development
team is only a
half-dozen people after
all—and the argument
against automation goes
on and on.
If you’re a developer or
tester who enjoys
spending your time
actually adding value to
your project, rather
than repeating the same
routine tasks over and
over, you’ll want to
consider advocating the
concept of automation to
your team (especially,
to your project
manager). If you’re a
project manager who’s
committed to maximizing
the talents and time of
the members of your
technical team, as well
as minimizing the risk
of your project failing
to deliver on time and
on quality, you will
want to encourage your
team to invest the
necessary time and
effort required to
automate the types of
tasks that will be
identified in this
article.
Why Should I Automate?
You may already be
familiar with many of
the benefits of
automating development
processes. Some of the
more commonly cited ones
are:
Repeatability.
Scripts can be repeated,
and, unless your
computer is having a
particularly bad day,
you can be reasonably
certain that the same
instructions will be
executed in the same
order each time the same
script is run.
Reliability.
Scripts reduce chances
for human error.
Efficiency.
Automated tasks will
often be faster than the
same task performed
manually. (Some people
might question whether
gains in efficiency are
typical, noting that
they have worked on
projects where, in their
view, trying to automate
tasks actually cost the
project more time than
it saved. Depending on
the situation, this may
be a real concern. In
addition, automation
might have been
implemented poorly or
carried too far on some
projects—but keep
reading for more on what
to automate, and when.)
Testing. Scripted
processes undergo
testing throughout the
development cycle, in
much the same way the
system code does. This
greatly improves chances
for successful process
execution as the project
progresses. Automated
scripts eventually
represent a mature,
proven set of repeatable
processes.
Versioning.
Scripts are artifacts
that can be placed under
version control. With
manual processes, the
only artifacts that can
be versioned and tracked
are procedure documents.
Versioning of human
beings—the other factor
in the manual process
equation—is
unfortunately not
supported by your
typical source control
system.
Leverage. Another
big benefit to
automating is that
developers and testers
can focus on the areas
where they add real
value to a
project—developing and
testing new code and
features—instead of
worrying about the
underlying development
infrastructure issues.
For example, instead of
requiring everyone to
become intimately
familiar with all the
little nuances of the
build procedure, you can
have one person focus on
automating the build and
have that person provide
the team with a greatly
simplified method of
building, hopefully as
simple as running a
command or two. Less
time spent on builds
leaves more time for the
tasks that add the most
value to the project.
What Should I Automate?
If you’re convinced that
automating your
development processes is
a good idea, the next
logical question is
which processes should
be automated. While the
answer, to some extent,
is different for every
project, there are some
obvious ones, as well as
some general guidelines
that I’d like to offer.
Some of the typical
targets for automation
are:
-
Build and deployment
of the system under
design.
-
Unit test execution
and report
generation.
-
Code coverage report
generation.
-
Functional test
execution and report
generation.
-
Load test execution
and report
generation.
-
Code quality metrics
report generation.
-
Coding conventions
report generation.
The above list is
obviously not
exhaustive, and every
project has its own
unique characteristics.
Here’s a general, and
perhaps obvious, rule of
thumb to help identify
any process that should
be considered for
automation: Consider
automating processes
that you expect to be
repeated frequently
throughout a system’s
life cycle. The more
often the procedures
will be repeated, the
higher the value of
automating them.
Once a process has been
identified, spend a
little time
investigating how you
might be able to
automate the process,
including researching
tools that could assist
with automation, and
estimating the level of
effort required to
implement the automation
versus the total cost
and risk of requiring
team members to manually
perform the procedures.
As with any other
business decision, it
really should come down
to a cost versus benefit
analysis.
You probably noticed
that the term "report
generation" appears in
the above list of
automation candidates.
The repetition points
out another important
aspect of automating
development processes:
the end result of every
automated process
execution should be a
report that is easily
interpreted by the team.
Ideally, such reports
will focus on making
anomalous conditions
(for example, test
failures) obvious at a
glance. Also, these
reports should be
readily accessible to
the appropriate team
members.
In many situations, it’s
even a good idea to
"push" reports to the
team (perhaps via email
or RSS), instead of
requiring people to
remember to go out and
search for them. The
basic idea is that the
development team should
be notified as soon as
possible when problems
are introduced to the
system or environment. A
side benefit is that
management is provided a
more tangible real-time
view into the progress
and health of the
project than is possible
with team status reports
alone.
When Should I Automate?
Consider automation as
soon as you realize that
team members are
starting to execute a
process that meets the
criteria discussed in
the previous section.
For example, automating
the build process when
the project is nearly
over provides very
little benefit. It can,
however, save dozens, if
not hundreds of hours
when automated as soon
as development begins.
However, before you go
off and start automating
everything, one caution:
be reasonably certain
that the procedure will
be required for your
project, that it will be
repeated more than two
or three times during
the project’s life
cycle, and that you
understand the steps
that need to be executed
for the procedure. There
are some things you just
don’t do very often, and
in these cases the costs
may outweigh the
benefits.
Even for processes where
automation is warranted,
if you’re truly guessing
at the steps involved,
there’s a high
likelihood that you’ll
end up re-writing the
whole thing. Re-writing
is much different than
fine-tuning or
refactoring the
automated scripts.
Refactoring is expected,
but scrapping and
starting over again
means you tried to
automate before you
understood the process
you were trying to
automate.
Another danger is
allowing your project to
become the "proving
ground" for automation
for the entire
organization. If your
organization doesn’t
already have the
infrastructure in place
to support development
automation, you should
provide what makes sense
for your project, but
try not to allow your
team to lose sight of
your project’s goals. A
project team whose goal
is to deliver working
software is in no
position to develop
enterprise-wide
strategies and tools.
Trying to do so is
asking for trouble.
If outside entities
begin to get involved in
these efforts, I’d
suggest that you
recommend that a
separate project be
undertaken to work out
the automation
infrastructure for the
enterprise. Members of
this "automation
project" team are free
to review your project’s
implementation for ideas
or for use as a
launching point. Once
this infrastructure is
in place, the question
of When to automate?
is easier to answer for
future projects, and
less costly.
Obstacles to Automation
If there are so many
benefits to automation,
why don’t we see more of
it on software projects?
All software development
teams struggle to
balance the need to show
immediate results with
the long-term goals of
the project. There are
many obstacles to
implementation of
development process
automation. Here are a
few of the more common
ones:
-
Stakeholder
pressure. Faced
with the desire to
show early and rapid
progress, teams
often overlook or
choose to ignore
practices that do
not seem, at first
glance, to directly
contribute to
getting working code
up and running as
quickly as possible.
-
Focus on core
requirements.
The team has an
overwhelming desire
to begin producing
working code and
tests as soon as
possible. Writing
code that can be
demonstrated is much
more satisfying to
developers than
writing automation
scripts.
-
Lack of
management support.
Management may not
have a good
understanding of
automation—how it
works and/or the
costs and benefits.
-
Lack of
organizational
support. There
is no
enterprise-wide
policy or
infrastructure for
development
automation
(standards, tools,
expertise, etc.)
-
Lack of follow
through. Even
with the best of
intentions, teams
can quickly lose
their commitment to
plans for
implementing
automated processes.
As with any fundamental
change, there must be
someone who’s both
committed to the concept
and who has the
authority to make sure
that teams follow
through with the plan.
Not following through is
much worse than never
having taken the time to
investigate and discuss
automation. First of
all, there are probably
other problems that you
could have successfully
solved using the time
and resources and,
second, if you fail to
implement any
automation, when the
idea surfaces again
people in the
organization will point
out that it didn’t work
the "last time we tried
it".
Selling Automation
As implied in the
preceding section, the
primary obstacle to
automation is short-term
thinking. Therefore,
your primary task is to
get your team and
management to pause to
think about overall
project costs and
schedule, instead of
focusing solely on
meeting next week’s
promised deliverables.
The message should be
clear. Meeting
milestones is critical,
but it’s very possible
that you could meet that
next milestone to the
detriment of the
project. But before you
try to convince people
who are under a great
deal of pressure that it
sometimes makes sense to
slow down momentarily in
order to improve the
chances for success,
you’d better have more
than a set of maxims at
hand.
This is where Return On
Investment (ROI) can
help. Most people in the
business world,
regardless of field,
have at least a basic
understanding of ROI and
agree that, when based
on valid assumptions, it
can be one of the best
mechanisms for
evaluating whether or
not to take action or
adopt a particular
solution. The process of
calculating ROI is well
beyond the scope of this
series of articles.
However, the Suggested
Reading section at the
end of the article
provides a link to an
outstanding article from
Edward Adams that
describes how to do
this, as well as some
additional advice on
selling the case for
test automation, in a
very straightforward and
easy to understand way.
Although many managers
really like to see
quantifiable benefits,
such as ROI, before
making major decisions
about adding tasks to
the project schedule,
others have sufficient
experience with software
development and are just
as comfortable with a
common-sense explanation
of automation’s
benefits. With that in
mind, rather than going
into a full-blown ROI
calculation, let’s just
take a look at one
scenario that most
projects will face at
some point to see, from
a common sense
perspective, the likely
outcome of a purely
manual process versus a
process that had been
automated early on in
the project.
Manual testing
approach. Your team
is in the final month of
a twelve-month project,
and the system test team
is starting to perform
regression testing by
manually re-running the
entire set of test
cases. Some test cases
haven’t been executed in
several months, because,
frankly, the testers
have just been spread
too thin, and, due to
perceived pressure to
deliver, management
chose to move testers
onto other tasks as soon
as they were finished
wrapping up a set of
test cases.
Regular regression
testing was put into the
schedule early on, but
repeating the same tests
over and over throughout
the project was (whether
or not anyone wants to
admit it) eventually
considered a "nice to
have" and definitely
much less critical than
other, more pressing
tasks. Now some of the
regression tests are
failing and the test
team is logging defect
reports.
The development team
decides to manually
re-run the entire unit
test suite (which also
hasn’t been executed in
a few months, because it
wasn’t automated) to
hopefully zero in on the
source of the problem.
The developers find two
problems as a result of
the unit test run:
first, apparently there
were never any unit
tests written for the
parts of the system in
question; and second,
there are other,
seemingly non-related
unit tests that are
failing. To make things
worse, the components
that are failing were
developed by someone
who’s no longer with the
team. Even if he were
available, he’d probably
require significant
spin-up time to
re-acquaint himself with
the failing code, since
he hasn’t even looked at
it for months.
Automated testing
approach: Early on
in the project, the
decision was made to
automate at least the
following: unit testing,
code coverage, and
system testing. The same
bug that caused the
situation above was
introduced in month
three of the project.
The code coverage report
that’s automatically
generated as part of the
daily build clearly
indicated to the
development team that
there were no unit tests
for the related
components as soon as
those components were
placed under source
control. Therefore, the
developer went back and
immediately implemented
the unit tests.
After implementing the
unit tests and working
out the bugs that were
initially revealed as a
result, the developer
checks the unit tests
into source control. The
daily build
automatically picks up
the new unit tests, the
coverage report reflects
that the components are
now being tested (as
well as to what extent
they’re being tested),
and the unit test report
indicates test success
or failure each day. As
soon as any change is
made to the system and
placed under source
control, the team will
become aware of the
impact that change has
on the overall system,
while the changes are
still fresh in the minds
of the implementer.
The system tester who’s
responsible for testing
that same set of
features now designs his
test case, implements
the tests, reports any
defects he discovers,
etc. until he feels that
the code and test case
are operating as
expected. He then uses
the test automation
tools to record the
execution steps for the
test case and checks the
resulting scripts and
files into source
control. The next day,
the automated test
runner picks up his new
test case for execution
and the results become
available to the test
team as part of the
daily test run. The
development and test
teams are leveraging the
automated tools and
processes for proactive
detection as problems
are introduced.
Without performing a
detailed ROI analysis,
which of the two above
scenarios is intuitively
more desirable? Assuming
that the exact same bug
is introduced into the
system at the same
time—month three of the
project—in each case,
which approach do you
think will result in the
least expense and least
risk? Which will result
in the least stress to
the team and to
management?
Start Small
Unless you’re lucky
enough to have a team
with multiple members
who’ve had a good deal
of experience automating
development processes,
you won’t want to try to
take the idea of
automation to the
extreme. Trying to do so
with an inexperienced
team or a team that’s
not completely sold on
the idea of automation
will likely result in
failure, as both
expectations and
barriers are set high
from the start. Pick one
or two areas where you
believe there is both
high potential value for
automation and a low
risk that the team will
fail implementing
automation. You might
ask yourself the
following questions:
It’s better to succeed
in a small way than to
fail in a big way. Your
successes just might
build on themselves, and
when you suggest
additional areas for
automation, either later
in the project or on
your next project, you
may face little or no
resistance at all. To
reach the ideal takes a
long-term plan,
management commitment,
adequate tools and
training, and of course
time. If it were easy,
everyone would already
be doing it.
Conclusion
I’m well aware that
there may be other
factors that affect the
decision of whether and
what to automate than
those we’ve addressed in
this article. For
instance, you may find
that the only automation
tools your organization
will approve for your
team’s use are beyond
your budget or that, due
to your project’s
special needs, there are
more important factors.
Maybe you’re a vendor
and your marketing
department has
determined that
time-to-market for your
first release is more
critical than the
quality or the long-term
success of the current
product.
I’m a big believer in
the "it’s all about
trade offs" principle.
To gain one advantage,
you typically accept
other disadvantages, and
in the end you should
try to make decisions
that tip the scales most
in favor of the
"advantages." I would
say, however, that if
your project is truly
one of those projects
that is better off not
automating, you’re in
the minority
category—the exception,
not the rule. If the
primary obstacle to
adoption of automation
for your organization is
lack or expense of
tools, there are several
free open source tools
that can go a long way
towards solving that
problem. The time you
spend investigating
automation strategies
and tools could very
well make the difference
in whether your projects
successfully meet
schedule and quality
goals.