Software Testing (2IW30): Software Testing (2IW30) Lecture 1 Introduction Data-based black box test generation Judi Romijn jromijn@win.tue.nl HG 6.82


Outline: Outline Introduction to the subject what is... error/bug/fault/failure/testing? overview of the testing process classifying aspects of testing Course overview Data-based black box test generation Homework for next week


What is...: What is... error/fault/bug: something wrong in software failure: manifestation of an error (observable in software behaviour) something wrong in software behaviour (deviates from requirements)


What is...: What is... testing: by experiment, find errors in software (Myers, 1979) establish quality of software (Hetzel, 1988) a succesful test: finds at least one error gives quality judgement with maximum confidence with minimum effort test-to-fail test-to-pass


What’s been said?: What’s been said? Dijkstra: Testing can show the presence of bugs, but not the absence Beizer: 1st law: (Pesticide paradox) Every method you use to prevent or find bugs leaves a residue of subtler bugs, for which other methods are needed 2nd law: Software complexity grows to the limits of our ability to manage it Beizer: Testers are not better at test design than programmers are at code design Humphreys: Coders introduce bugs at the rate of 4.2 defects per hour of programming. If you crack the whip and force people to move more quickly, things get even worse. ... Developing software & testing are truly difficult jobs! Let’s determine what goes on in the testing process


Concept map of the testing process: Concept map of the testing process  


Dimensions of software testing: Dimensions of software testing What is the surrounding software development process? (v-model/agile, unit/system/user level, planning, documentation, ...) What is tested? Software characteristics (design/code/binary, embedded?, language, ...) Requirements (functional/performance/reliability/..., behaviour/data oriented, precision) Which tests? Purpose (kind of coding errors, missing/additional requirements, development/regression) Technique (adequacy criterion: how to generate how many tests) Assumptions (limitations, simplifications, heuristics) How to test? (manual/automated, platform, reproducable) How are the results evaluated? (quality model, priorities, risks) Who performs which task? (programmer, tester, user, third party) Test generation, implementation, execution, evaluation


Dimensions + concept map: Dimensions + concept map 2 1 6 2 6 6 4 3 5


1: Test process in software development: 1: Test process in software development V-model:


1: Test process in software development: 1: Test process in software development Agile/spiral model:


2: Software: 2: Software (phase) Unit vs. integrated system (language) imperative/object-oriented/hardware design/binary/… (interface) data-oriented/interactive/ embedded/distributed/…


2: Requirements: 2: Requirements functional: the behaviour of the system should be correct requirements can be precise, but often are not non-functional: performance, reliability, compatibility, robustness (stress/volume/recovery), usability, ... requirements are possibly quantifiable, and always vague


3: Test generation: purpose: 3: Test generation: purpose What errors to find? Related to software development phase: unit phase typical typos, functional mistakes integration interface errors system/acceptance: errors w.r.t. requirements unimplemented required features ‘software does not do all it should do’ implemented non-required features ‘software does things it should not do’


3: Test generation: technique: 3: Test generation: technique black box: we don’t have acces to the software to be tested white box: we have access to the software to be tested Dimensions:


3: Test generation: 3: Test generation Assumptions, limitations single/multiple fault: clustering/dependency of errors perfect repair heuristics: knowledge about usual programming mistakes history of the software pesticide paradox ...


4: Test implementation & execution: 4: Test implementation & execution Implementation platform batch? inputs, outputs, coordination, ... Execution actual duration manual/interactive or automated in parallel on several systems reproducible?


5: Who performs which task: 5: Who performs which task Software producer programmer testing department Software consumer end user management Third party testers hired externally certification organization


6: Result evaluation: 6: Result evaluation Per test: pass/fail result diagnostical output which requirement was (not) met Statistical information: coverage (program code, requirements, input domain, output domain) progress of testing (#errors found per test-time unit: decreasing?) Decide to: stop (satisfied) create/run more tests (not yet enough confidence) adjust software and/or requirements, create/run more tests (errors to be repaired)


Dimensions + concept map: Dimensions + concept map 2 1 6 2 6 6 4 3 5


Dimensions + concept map: Dimensions + concept map Classify terminology according to aspects of testing process: smoke testing: typically an initial testing effort to determine if a new software version is performing well enough to accept it for a major testing effort software development phase: usually system testing requirements: only major functions purpose: regression testing technique: error guessing assumption: a bit of testing suffices capture/replay tool: a tool that records test input behaviour and saves it for automatic reuse, very useful for graphical user interfaces test implementation & execution: here the tool is used purpose: regression testing traceability matrix: a document that maps each requirement to other work products in the development process such as design components, software modules, test cases, and test results requirements, test set, test results: the matrix shows relationship between these result evaluation: the matrix helps deciding coverage


Outline: Outline Introduction to the subject Course overview goals subjects organization required knowledge & skills self-study Data-based black box testing Homework for next week


Course goals: Course goals knowledge of (general) testing process, terminology, classification test generation methods software reliability techniques ability to classify test terminology skill in several test generation techniques understanding of software reliability techniques understanding of possibilities, limitations, required effort, mutual relations for techniques of C. and D. ability to judge, given a situation: which technique from C. and D. is suitable when to stop testing


Course overview: Course overview 1 Introduction, data-based black box testing 2 Structure-based white box testing 3-5 Structure-based black box testing 6 Integration testing 7-8 Statistical testing 9 Industrial testing of embedded systems (guest lecture: Bart Broekman, Sogeti NL) 10 Summary, exam preparation


Organization of the course: Organization of the course 4 ects = 112 hours = 1/5 of trimester load 10 lectures: (mind the schedule!) literature: handouts exercises global feedback on homework written examination: Monday July 22, 9:00-12:00 self-study: 112 – (15 + 3) = 94 hours contact by e-mail: jromijn@win.tue.nl Information on the web: http://studyweb.tue.nl http://www.win.tue.nl/~jromijn/2IW30/ register!!


Required knowledge, skills: Required knowledge, skills English (speak/understand/write) Computer science: skill in programming (making+correcting errors) software development process behavioural specification languages: automata, state machines practical homework: windows+linux, a bit of C Mathematics: logic, sets, reasoning statistics: D.C. Montgomery,and G.C. Runger, Applied Statistics and Probability for Engineers, 3rd. ed., Wiley 1999 Chapter 3: Poisson distribution (2nd. ed.: Ch 4) Chapter 4: exponential, Weibull distribution (2nd. ed.: Ch 5)


Self-study: Self-study As mentioned, 95.5 hrs DIY: global feedback during lectures possibly on web page material: handouts (lecture, studyweb) slides, also with narration see web page for links to other relevant literature contact feedback on homework only at request e-mail It’s your own responsibility to keep up! Written exam is much work, so practice!


Outline: Outline Introduction to the subject Course overview Data-based black box testing techniques example Homework for next week