black box testing

Views:
 
Category: Education
     
 

Presentation Description

No description available

Comments

Presentation Transcript

Black Box Testing (revisited) : 

Black Box Testing (revisited) Csci 565 Spring 2007

Objectives : 

Objectives Cause-Effect Graphs in Functional testing Input validation and Syntax-driven Testing Decision Table-Based Testing State transition testing (revisited)

Syntax-Driven testing (SDT) : 

Syntax-Driven testing (SDT) Applicable to Specification validation written by a certain grammar (BNF) Input data validation Test generation Generate test cases such that each production rule is tested at least once

Example: BNF of simple Arithmetic expressions : 

Example: BNF of simple Arithmetic expressions Example <exp> ::= <exp> + <term>| <exp> - <term>| <term> <term> ::= <term> * <factor> | <term> / <factor>|<factor> <factor> ::= <id> | <exp> <id>::= a|b|…|z

More on SDT : 

More on SDT The set of test cases for SDT should include expressions that exercise the production rules Examples: a+b * c (w+v) + z

Input validation and syntax testing : 

Input validation and syntax testing Data validation is the first line of defense against a hostile world Casual Malicious users Good designs won’t accept garbage Good testers will test system against possible garbage

Formal speciation of input : 

Formal speciation of input Use Backus-Naur Form (BNF) to specify data input E.g., line::= (alpha:char)7..80 end:line alpha:char::= A/B/C/…/Z end:line ::= CR CR LF/ CR LF LF

Test Case generation : 

Test Case generation A data validation routine is designed to recognize strings defined by BNF String recognizer routine Accepts the string (or recognizes) Rejects the string String generator When test designer attempts to generate strings

Types of incorrect actions : 

Types of incorrect actions There are three possible kinds of incorrect actions The recognizer does not recognize a good string It accepts a bad string It may accept or reject a good string or a bad string, but in so doing, it fails

Strings’ errors : 

Strings’ errors Even small specification may result in many strings Strings’ errors can be classified as Syntax error Delimiter errors Field-value errors

Syntax error : 

Syntax error E.g., Item ::= atype/btype/ctype/dtype etype Possible test cases Wrong combination E.g, /dtype atype Don’t do enough Dtype/ etype Don’t do nothing Do too much Atype btype ctype dtype etype/ atype atype atype/ dtype etype atype/dtype etype

Delimiter errors : 

Delimiter errors Delimiters are chars placed between two field Excellent source of test cases Test for Missing delimiter Wrong delimiter Not a delimiter Too many delimiters Paired delimiters

Field-value errors : 

Field-value errors Fields have meaning and must be tested Boundary values Excluded values Troublesome values

Problems with SDT : 

Problems with SDT It is easy to forget the normal cases Don’t get carry a way with combinations Don't ignore structure

Applications of SDT : 

Applications of SDT Here are some examples All user interfaces Operating command processing All communications interfaces and protocols Internal interface and protocol (call sequences)

Decision table-based Testing (DTT) : 

Decision table-based Testing (DTT) Applicable to the software requirements written using “if-then” statements Can be automatically translated into code Assume the independence of inputs Example If c1 AND c2 OR c3 then A1

Sample of Decision table : 

Sample of Decision table A decision table is consists of a number of columns (rules) that comprise all test situations Action ai will take place if c1 and c2 are true

Example: Simple editor : 

Example: Simple editor A simple text editor should provide the following features Copy Paste Boldface Underline select

Slide 19: 

In general, for n conditions, we need 2n rules

Decision tables as a basis for test case design : 

Decision tables as a basis for test case design The use of decision-table model to design test cases is applicable The spec is given by table or is easily converted to one The order in which the conditions are evaluated does not affect the interpretation of rules or the resulting action The order in which the rules are evaluated does not affect the resulting action Once a rule has been satisfied and an action is selected, no other rule need be examined If multiple actions result from the satisfaction of a rule, the order in which the actions take place is not important

The implications of rules : 

The implications of rules The above conditions have the following implications Rules are complete (i.e., every combination of decision table values including default combinations are inherent in the decision table) The rules are consistent (i.e, there is not two actions for the same combinations of conditions)

Cause-effect graphs in black box testing : 

Cause-effect graphs in black box testing Captures the relationships between specific combinations of inputs (causes) and outputs (effects) Deals with specific cases, Avoids combinatorial explosion Explore combinations of possible inputs Causes/effects are represented as nodes of a cause effect graph The graph also includes a number of intermediate nodes linking causes and effects

Example 2: : 

Example 2: Consider the following spec: The character in column 1 must be an “A” or a “B”. The character in column 2 must be a digit In that case, the file update is made If the first character is incorrect, message X12 is issued If the second character is not a digit, message X13 is issued

The causes : 

The causes Character in column 1 is “A” Character in column 1 is “B” Character in column 2 is a digit

The effects : 

The effects The effects are 70: update made 71: message X12 is issued 72: message X13 is issued

Drawing Cause-Effect Graphs : 

Drawing Cause-Effect Graphs A B If A then B (identity) A C If (A and B)then C B

Drawing Cause-Effect Graphs : 

Drawing Cause-Effect Graphs A C If (A or B) then C B A C If (not(A and B)) then C B ??

Drawing Cause-Effect Graphs : 

Drawing Cause-Effect Graphs A C If (not (A or B))then C B A B If (not A) then B ? ?

Constraint Symbols : 

Constraint Symbols a b E: at most, one of a and b can be 1 a b O: one and only one, of a and b must be 1 a b I: at least one of a,or b must be 1

More Constraint symbols : 

More Constraint symbols a b R Require (i.e. a is 1, b must be 1) a b M Mask (i.e. a is 1, b must be 0)

Example: ATM : 

Example: ATM For a simple ATM banking transaction system Causes (inputs) C1: Command is credit C2: command is debit C3: account number is valid C4: transaction amount is valid Effects E1: Print “invalid command” E2: Print “ invalid account number” E3: Print “debit amount not valid” E4: debit account E5: Credit account

Slide 32: 

C1 C4 C3 C2 and and and and E3 E2 E1 E5 E4 or ? ? ?

Description of processing nodes used in a Cause-effect graph : 

Description of processing nodes used in a Cause-effect graph ?

ATM Cause-effect decision table : 

ATM Cause-effect decision table Don’t Care condition

Example 3: : 

Example 3: Consider the following NL specifications: The boiler should be shut down if any of the following conditions occurs: If the water level in a boiler is below the 20000lb mark The water level is above the 120000 lb, or The boiler is operating in degraded mode and the steam meter fails Being in degraded mode means If either water pump or a pump monitors fails

Example 3: Translate NL into workable pieces (atomic specifications) : 

Example 3: Translate NL into workable pieces (atomic specifications) Atomic sentences are C1. the water level in boiler is below the 20000 lb (F/T) C2. the water level is above the 120000 lb mark (F/T) C3. A water pump has failed (F/T) C4. A pump monitor has failed (F/T) C5. Steam meter has failed/T E. Shut the boiler down

Steps to create cause-effect graph : 

Steps to create cause-effect graph Study the functional requirements and divide the requirements into workable pieces E.g. of workable piece, in eCom, can be verifying a single item placed in shopping cart Identify causes and effects in the specifications Causes: distinct input conditions Effects: an output condition or a system transformations. Assign unique number to each cause and effect Use the semantic content of the spec and trasnform it into a Boolean graph Annotate the graph with constrains describing combinations of causes and/or effects Convert the graph into a limited-entry decision table Use each column as a test case

Possible research topics based on CEG : 

Possible research topics based on CEG Comparison of CEG ,FSM-based test sets, and randomly generated test cases (functional) For effectiveness in terms of cost and fault detection capabilities For fault detection capabilities For number of test cases generated (cost) For automatic generation of actual test cases

Black-box testing : 

Black-box testing

Equivalence Partitioning (EP) : 

Equivalence Partitioning (EP) Input data and output results often fall into different classes where all members of a class are related Each of these classes is an equivalence partition where the program behaves in an equivalent way for each class member The entire set (union of all classes serves Completeness and non-redundancy Test cases should be chosen from each partition (or class)

Equivalence partitioning : 

Equivalence partitioning

Guidelines for equivalence classes : 

Guidelines for equivalence classes If an input condition specifies range, one valid and two invalid equivalence classes are needed If a condition requires a specific value, then one valid and two invalid equivalence classes are needed If an input condition specifies a member of a set, one valid and one invalid equivalence class are needed If an input condition is Boolean, one valid and one invalid class are needed

Example: ATM : 

Example: ATM Consider data maintained for ATM User should be able to access the bank using PC and modem User should provide six-digit password Need to follow a set of typed commands

Data format : 

Data format Software accepts Area code: Might be blank or three-digit Prefix: three-digit number not beginning with 0 or 1 Suffix: four digits number Password: six digit alphanumeric value Command: {“check”, “deposit,” “ bill pay”, “transfer” etc.}

Input conditions for ATM : 

Input conditions for ATM Input conditions Area code: Boolean: the area code may or may not be present Range: values defined between 200-999 Specific value: no value > 905 Prefix: range –specific value >200 Suffix: value (four-digit length) Password: Boolean: password may or may not be present Or value – six char string Command: set containing commands noted previously

Boundary Value Analysis (BVA) : 

Boundary Value Analysis (BVA) complements equivalence partitioning Works with single fault assumption in reliability theory Focuses is on the boundaries of the input If input condition specifies a range bounded by a certain values, say, a and b, then test cases should include The values for a and b The values just above and just below a and b If an input condition specifies any number of values, test cases should be exercise the minimum and maximum numbers, the values just above and just below the minimum and maximum values

More on BVA : 

More on BVA Generalizing BVA are done Using the number of variables Example for Function of F with n variables, need at least 4n+1 test cases The type of ranges Logical quantity vs physical bounded quantities E.g. of discrete and bounded: months of year, commissions E.g., discrete and unbounded: length of sides in triangle problems Need to create artificial min/max bounds

Example 2: Equivalence Partitioning : 

Example 2: Equivalence Partitioning

Valid partitions : 

Valid partitions The valid partitions can be 0<=exam mark <=75 0<=coursework <=25

Invalid partitions : 

Invalid partitions The most obvious partitions are Exam mark > 75 Exam mark < 0 Coursework mark > 25 Coursework nark <0

Exam mark and c/w mark : 

Exam mark and c/w mark

Less obvious invalid input EP : 

Less obvious invalid input EP invalid INPUT EP should include

Partitions for the OUTPUTS : 

Partitions for the OUTPUTS EP for valid OUTPUTS should include

The EP and boundaries : 

The EP and boundaries The EP and boundaries for total mark

Unspecified Outputs : 

Unspecified Outputs Three unspecfied Outputs can be identified (very subjective) Output = “E” Output = “A+” Output = “null”

Total PE : 

Total PE

Test Cases corresponding to PE exam mark (INPUT) : 

Test Cases corresponding to PE exam mark (INPUT)

Test Case 4-6 (coursework) : 

Test Case 4-6 (coursework)

test case for Invalid inputs : 

test case for Invalid inputs

Test cases for invalid outputs:1 : 

Test cases for invalid outputs:1

Test cases for invalid outputs:2 : 

Test cases for invalid outputs:2

Test cases for invalid outputs:3 : 

Test cases for invalid outputs:3

Minimal Test cases:1 : 

Minimal Test cases:1

Minimal Test cases:2 : 

Minimal Test cases:2

For Boundary Value analysis (BVA) : 

For Boundary Value analysis (BVA)

BVA : 

BVA For each boundary three values are used One on the boundary One either side of it

Test cases corresponding to BVA (Exam mark) : 

Test cases corresponding to BVA (Exam mark)

Test cases corresponding to BVA (Course work) : 

Test cases corresponding to BVA (Course work)

PE + BVA : 

PE + BVA

Test case derived from valid outputs:1 : 

Test case derived from valid outputs:1

Test case derived from valid outputs:2 : 

Test case derived from valid outputs:2

State Transition Testing (revisited) : 

State Transition Testing (revisited) States The word “state” is used in much the same way as in “state of the Union” or “state of economy” Refers to a combination of circumstances (or snapshots of program attributes)

Number of states : 

Number of states Suppose the following factors are identified for a car GEAR (R,N, 1,2,3, 4) = 6 values Direction (Forward, reveres, stopped) =3 values Engines Operation (Running, ideal) = 2 values Engine condition (Okay, broken) = 2 values Total number of states 6*3*2*2*=72 Impossible states Can broken engine run? A car with a broken transmission won’t move

Example of state graph : 

Example of state graph A program that detects the char sequence “ZCZC” cab be in the following states Neither ZCZC nor any part of it has been detected Z has been detected ZC has been visited ZCZ has been visited or detected ZCZC has been detected

ZCZC sequence detection state graphs : 

ZCZC sequence detection state graphs none ZCZC C Z C ZC Z ZCZ A, C Z,C,Z Z A C, A Z A Z

GOOD STATE GRAPHS : 

GOOD STATE GRAPHS Some general rules The total number of states is equal to the product of the possibilities of factors that make up the states Deterministic behavior For every transition there is one outcome specified For every state there is a sequence of inputs that will drive the system back to the same state

More on states : 

More on states impossible states Equivalent states Two states are equivalent if every sequence of inputs starting from one state produces exactly the same sequence of outputs when stared from the other state Unreachable states A state that no input sequence will reach Dead states A state which once entered cannot be left

Dead State : 

A B 2 1 1, 2 State B can never be left (Dead State) The initial state never be entered gain Dead State

Unreachable State : 

A B 2 1 1, 2 State B can never be left (Dead State) The initial state never be entered gain Unreachable State

Ambitious state : 

A B 1 1 2 Two transitions are specified for an input of 1 in state A Ambitious state 1, 2

State testing : 

State testing The starting point of state testing is Define a set of covering input sequence that get back to the initial state when starting from the initial state For each step in each input sequence, define the expected next state, the expected transition, and the expected outcome

More on state testing : 

More on state testing A set of tests consists of Input sequence Corresponding transitions or next-state names Outcome sequences

References : 

References 1. G. Myers. The Art of Software Testing, second edition, Wiley 2004 2. Standard for Software Component Testing Produced by the British Computer Society, April 2001, www.testingstandards.co.uk