How effective is it to purchase automation scripts rather than developing in-house?
morde last edited by
I recently moved into the Software Testing department of my organization.The business has little history of automating testing, and most software development and testing is conducted by third parties on our behalf. However, we do employ testers directly, and are now looking to increase automation within software testing. To do this, we have begun to purchase test scripts for regression packages from these third parties. I have a bit of programming experience, and I worry that purchasing scripts will not allow us to have a) the knowledge to evaluate these scripts effectively and b) to fully access the benefits of automation. By the full benefits of automation, I mean the capacity to understand on the spot whether or not a process is a potential candidate for automation, how long it would take to automate versus manual execution, how to effectively re-use existing function libraries (test partners and the individuals working for them regularly change) etc. This knowledge could lead us beyond simply automating regression packs for long term maintenance purposes, to more sophisticated automation approaches. However, as no such capacity exists within the organization at the moment, there would be costs associated with creating and maintaining the skill base required. As someone new to testing, I wonder - are my concerns justified, or are the cost savings associated with buying test scripts sufficient justification for a lack of in-house expertise?
You're going to get a lot of "it depends" answers for this. Whether it's better to stay with primarily third-party automation will change depending on the quality of your third party automation providers, the nature of your business applications, your internal infrastructure, the method your third party people use and a whole lot more. Some of the factors to consider are: Does your business build small, self-contained applications or maintain a large, complex application suite that gets new features added to it? In the former case, third parties could be the better option, where a continually changing complex application requiring a lot of domain knowledge is a better candidate for internally developed and maintained automation. What is the automation/coding skill base of your internal test team? If you have a number of testers with good formal logic skills and a decent programming knowledge, you have enough to build good, maintainable internal automation. If your test team is primarily manual without much in the way of formal logic, it's going to be a lot more difficult to do this and you'll probably hit resistance from your test team. What are your organization's budget priorities? Typically a third party automation solution is going to automate existing manual test cases, something that can be done fairly quickly using the third party's pre-existing structures and frameworks. Building your own from scratch (because chances are there's a lot of proprietary code in the third party automation) is very expensive in terms of resources, time, and money. The third party automation could use a tool that is outside your budget, giving you no ability to reuse their material. What are your organization's long term plans? Your test strategy - including what gets automated and how - needs to support the organization's long range goals. Using a third party to build and maintain automation may be more aligned to the long range goals than building your own. Some of the advantages of building your own automation are: the automation is built by people with domain knowledge. if it's done right, adding new tests to existing automation is a lot easier. your organization retains full control over the automation. your team is more familiar with the automation, making it easier to ensure that any test failures are real problems with the software. Some of the disadvantages are: it can take a long time to build up a stable, reliable automation suite. depending on the application technologies you need, automation tools can be extremely expensive. you need testers who can code to at least intermediate developer level. Ideally you want people who can fill the Software Developer in Test role, but those aren't all that easy to come by (the combination of good test skills and good coding skills isn't terribly common). you will need to convince management of the need and then keep them convinced. from the management perspective, it's easier to contract someone to build automation, particularly when that's the way things already work and the third party is experienced with building automation. It may also be cheaper. it's extremely difficult to quantify potential cost savings from moving automation in-house, especially when the only thing management is likely to see is increased expense.