testing

Introduction to Test Automation:

Introduction to Test Automation Trainer: Anup Patnaik, Director Engineering | 10.SEP.2010 |

What are we going to talk about …:

What are we going to talk about … The good & the bad What & when Myths & facts Tools & the market Record & playback - why not Key pieces of any automated test

Understanding Test Automation:

Understanding Test Automation

Lets get automated:

Lets get automated 4 Test Engineers Test Automation Engineers “The Product”

Wow bugs – So many of them, really!:

Wow bugs – So many of them, really! 5 Test Case Result TC_1 Play video TC_2 Auto play TC_3 Check start screen TC_4 Check channels TC_5 Check upload TC_6 Check syndication TC_7 Check login

Oops! What went wrong:

Oops! What went wrong 6 Test Engineers Test Automation Engineers “The Product” !!??##??!!

The divide between goals and reality:

The divide between goals and reality 7

Learning:

Learning Software Test Automation is No Silver Bullet Let’s not paint a rosy picture 8

Learning:

Learning There is No Silver Bullet to Software Test Automation Doing Test Automation Right is Hard! 9

Software Test Automation:

Software Test Automation What is test automation? Test Automation is the use of software to execute tests without Human intervention Why to automate? Manual validation of application functionality is increasingly inefficient in an environment where developer productivity is increasing as programmers make greater use of higher level APIs and more modern, effective programming interfaces. Ever expanding application feature sets are making comprehensive manual testing more tedious and difficult. Ever expanding software/hardware configurations. During the past decade, companies, irrespective of which domain/field they operate in, have strived to achieve operational efficiencies by adopting various processing systems. The focus has been on automation and moving as close as possible to straight-through processing (without manual intervention) More recently, in the wake of tougher competition and market requirements, companies increasingly have begun adopting so-called next-generation information-technology applications. It has been right recognized that it is critical that these upgrades and new operating systems be thoroughly tested Achieving good test coverage after every change, no matter how minor, using manual techniques is virtually impossible. It also saves time, particularly, for the business-acceptance tester who has to perform regression testing. 10

The Good – Textual view:

The Good – Textual view Reusability : Assuming automation has been designed and implemented properly, acceptance tests could be run as many times as necessary to test a software release or any future software releases. Over time, this may amount to very significant productivity gains Accuracy : Repeating the same tests over and over inevitably lead to boredom-induced complacency that allows bugs that would otherwise be caught, to be overlooked. Oftentimes, this also leads to testing shortcuts which could also have detrimental effects. With automation, test cases are executed with 100% accuracy and repeatability every time. Relentlessness : Can be run, day and night, 24 hours a day potentially delivering the equivalent of several full-time SQA manual testers. Efficiency : Automating boring repetitive tasks not only improves employee morale, but also frees up time for staff to pursue other tasks they otherwise could not or would not pursue. Therefore, greater breadth and depth of testing is possible this way. Speed : Test cases generally execute faster when automated. More importantly, different batches of test cases can be executed on multiple computers simultaneously. 11

The Good – Pictorial view:

The Good – Pictorial view 12 Traditional testing approaches result in defect discovery late in the process Late defect discovery results in significant rework Time Rate of Discovery Requirements Design & Build Defects Release to Test Release to Field Exponential Increase in Cost of Removing Defects

The Bad:

The Bad Significant Investment: The investment required to implement test automation is expensive but the incremental cost is very cheap. Test scripts created during the initial automation exercise need to be maintained to keep pace with changes to the relevant application. Maintenance: Requires constant maintenance. Error Detection Difficult: Errors introduced during the automation process are more difficult to detect because once a test has been fully automated and become part of the test regime, Human interaction is minimized and errors will only come to light if the automated test itself includes robust error detection routines or the manual tester is actively monitoring every automated test as they execute. Cannot Think: Cannot detect and intercede when unexpected situations arise. Dependency on automation expert: Test automation is largely a technical exercise performed by a skilled automation expert. Moreover, once the test automation platform changes, we require a completely fresh team skilled in the new test automation scripting language. 13

Slide 14:

What to automate? Regression Tests: Stabilized tests that verify stabilized functionality Tests rerun often: Tests that are executed regularly vs. rarely Tests that will not expire shortly: Most tests have a finite lifetime during which its automated script must recoup the additional cost required for its automation Tedious/Boring tests: tests with many calculations and number verifications repetitive tests performing the same operations over and over tests requiring many performance measurements Just plain boring tests Reliably repeatable What NOT to automate? Unstable functionality: Not reliably repeatable Rarely executed tests: poor Return-On-Investment Tests that will soon expire: poor Return-On-Investment Requiring in-depth business analysis: some tests require so much business specific knowledge that it becomes prohibitive time wise to include every verification required to make its automated script robust enough to be effective exceedingly complex tests are sometimes not possible to automate because computers cannot think white box testing without ready access to business analysts When to automate? 14 What & When

Myths & Facts (0):

Myths & Facts (0) Automation: Facts Manual scripts needed Ideal for regression testing Benefits come from discipline in analysis and planning Causes significant schedule impact at introduction Require more programming and design skills from testers Require continuous maintenance Automation: Myths High immediate ROI from automation One tool that fits perfectly Need to substitute skilled manual testers Preclude manual testing 100% Automation of all existing manual tests – All or none 15

Myths & Facts (1):

Myths & Facts (1) Myth: Recorders make test automation easier Truth: They make recording easy … But they also make it easy to create automation that is difficult and expensive to maintain A recorder can serve as a training device to help you learn a scripting language Myth: You can’t develop test automation for an application until it is stable Truth: You should develop test plans as soon as the Requirements are documented You can also develop automated tests from wire-frames, mock-ups and prototypes In the case of wire-frames, you can even test some of your methods If tests are structured properly, only a small portion of your work will have to be maintained when GUI changes are made to an application that has not yet stabilized. 16

Myths & Facts (2):

Myths & Facts (2) Myth: Test automation is most effective for regression testing. Truth: Test automation is very effective for regression testing. It is also effective for first pass testing when a data driven approach is used Automated tests should be designed to find bugs not just verify that regression tests don’t break Myth: It is difficult to automate a dynamic application where data changes constantly Truth: Data can be separated from test code so that only the data needs to be changed Better yet, in some cases input data can be captured at runtime to meet the criteria required for a test Or a database query can be executed to get data at runtime Verification data can also be captured at runtime so that tests remain flexible in a dynamic environment 17

Myths & Facts (3):

Myths & Facts (3) Myth: Test automation is not a software development task. Truth: Automated tests should be designed, developed and tested Automated test components are assets that should be treated like application source code Test automation standards should be developed Test automation should be subject to peer reviews Myth: Productivity gains will be realized from test tools right away Truth: Test automation is an investment Quality gains will be realized right away Productivity gains will be realized over time as skills develop and manual testing tasks are offset by completed automation 18

Myths & Facts (4):

Myths & Facts (4) Myth: Automated tests are difficult to maintain Truth: Separate business rules, test methods and data to promote maintainability If properly structured, automated tests can be maintained across many releases of a target application It’s important to run automated tests on every build and make incremental changes as needed 19

Myths & Facts (5):

Myths & Facts (5) Misconceptions Testing tools don’t work Testing tools are too difficult to use My application is too complex Our project schedules are too tight Management will never support it Automation will eliminate the need for all manual testing Manual testing must have taken place before automating a feature Things to be taken care: Setting realistic goals Evaluating staff skills Designing good test cases Choosing best automation candidates Preparing test data Building a strong framework Establishing source code control Developing Standards Fact: 63% of Test Automation Projects Fail as per some study 20

Tools & more tools:

Tools & more tools 21

A special note …:

A special note … Record & Playback Add a test case to the test management tool Capture the user actions in a macro like fashion Externalize data Add synchronization points/think times Does NOT work Hard coded values (e.g. Test data, Object Identifiers) Script does not replay – test creation tied to test execution environment or not all scenarios taken care of No re-usability/modularity Too much hidden from the engineer Disclaimer: Tools like QTP have developed a lot of hybrid strategies and worked at length at solving the above mentioned problems 22

Key pieces of any Automated Test:

Key pieces of any Automated Test Actions / Scripts Object Identification/Map Test Data 23

Questions & Answers:

Questions & Answers anuppatnaik@qainfotech.net | www.qainfotech.com/blog | www.qainfotech.com Thank you .