Research... : Research... The systematic application of a family of methods employed to provide trustworthy information about problems
an ongoing process based on many accumulated understandings and explanations that, when taken together lead to generalizations about problems and the development of theories The basic steps of research... : The basic steps of research... Scientific and disciplined inquiry is an orderly process, involving:
recognition and identification of a topic to be studied (“problem”)
description and execution of procedures to collection information (“method”)
objective data analysis
statement of findings (“results”) Research methods... : Research methods... Quantitative…
…collects and analyzes numerical data obtained from formal instruments Quantitative methods... : Quantitative methods... descriptive research (“survey research”)
experimental research experimental research : experimental research …the researcher selects participants and divides them into two or more groups having similar characteristics and, then, applies the treatment(s) to the groups and measures the effects upon the groups Conducting an experimental study… : Conducting an experimental study… select the problem
select participants and instrument
selection and execution of a research plan
data analysis and formulation of conclusions Types of experimental comparison… : Types of experimental comparison… 1. comparison of two different approaches (A versus B)
2. comparison of an existing approach to a new approach (A and ~ A)
3. comparison of differing amounts of a single approach (A and a or a and A) where: : where: A – experimental (“treatment”) group
B – control (“no treatment,” “nonmanipulated”) group Variable... : Variable... …a concept (e.g., intelligence, height, aptitude) that can assume any one of a range of values Research variables... : Research variables... Independent…
…an activity of characteristic believed to make a difference with respect to some behavior
…(syn.) experimental variable, active variable, cause, treatment Dependent… : Dependent… …the change or difference occurring as a result of the independent variable
…(syn.) criterion variable, assigned variable, effect, outcome, posttest Confounding… : Confounding… …the fact that the effects of the independent variable may intertwine with extraneous variables, such that it is difficult to determine the unique effects of each variable
…(syn.) criterion variable, assigned variable, effect, outcome, posttest Validity of Experimental Design : Validity of Experimental Design Internal validity—The accuracy in concluding that the outcome of an experiment is due to the independent variable
External validity—The extent to which the results of an experiment can be generalized INTERNAL OR EXTERNAL VALIDITY: A TRADE-OFF? : INTERNAL OR EXTERNAL VALIDITY: A TRADE-OFF? Too much control reduces ability to generalize
Too little control reduces ability to make causal statements
Attempt to find a good balance Factors Jeopardizing Internal Validity : Factors Jeopardizing Internal Validity History - The events occurring between the first and second measurements in addition to the experimental variable which might affect the measurement.
Example: Researcher collects gross sales data before and after a 5 day 50% off sale. During the sale a hurricane occurs and results of the study may be affected because of the hurricane, not the sale. Slide 16: Maturation - The process of maturing which takes place in the individual during the duration of the experiment which is not a result of specific events but of simply growing older, growing more tired, or similar changes.
Example: Subjects become tired after completing a training session, and their responses on the Posttest are affected. Slide 17: Pre-testing - The effect created on the second measurement by having a measurement before the experiment.
Example: Subjects take a Pretest and think about some of the items. On the Posttest they change to answers they feel are more acceptable. Experimental group learns from the pretest. Slide 18: Measuring Instruments - Changes in instruments, calibration of instruments, observers, or scorers may cause changes in the measurements.
Example: Interviewers are very careful with their first two or three interviews but on the 4th, 5th, 6th become fatigued and are less careful and make errors. Slide 19: Statistical Regression - Groups are chosen because of extreme scores of measurements; those scores or measurements tend to move toward the mean with repeated measurements even without an experimental variable.
Example: Managers who are performing poorly are selected for training. Their average Posttest scores will be higher than their Pretest scores because of statistical regression, even if no training were given. Slide 20: Differential Selection - Different individuals or groups would have different previous knowledge or ability which would affect the final measurement if not taken into account.
Example: A group of subjects who have viewed a TV program is compared with a group which has not. There is no way of knowing that the groups would have been equivalent since they were not randomly assigned to view the TV program. Slide 21: Experimental Mortality - The loss of subjects from comparison groups could greatly affect the comparisons because of unique characteristics of those subjects. Groups to be compared need to be the same after as before the experiment.
Example: Over a 6 month experiment aimed to change accounting practices, 12 accountants drop out of the experimental group and none drop out of the control group. Not only is there differential loss in the two groups, but the 12 dropouts may be very different from those who remained in the experimental group. Slide 22: Interaction of Factors, such as Selection Maturation, etc. - Combinations of these factors may interact especially in multiple group comparisons to produce erroneous measurements. Factors Jeopardizing External Validity or Generalizability : Factors Jeopardizing External Validity or Generalizability Pre-Testing -Individuals who were pretested might be less or more sensitive to the experimental variable or might have "learned" from the pre-test making them unrepresentative of the population who had not been pre-tested.
Example: Prior to viewing a film on Environmental Effects of Chemical, a group of subjects is given a 60 item antichemical test. Taking the Pretest may increase the effect of the film. The film may not be effective for a nonpretested group. Slide 24: Differential Selection - The selection of the subjects determines how the findings can be generalized. Subjects selected from a small group or one with particular characteristics would limit generalizability. Randomly chosen subjects from the entire population could be generalized to the entire population.
Example: Researcher, requesting permission to conduct experiment, is turned down by 11 corporations, but the 12th corporation grant permission. The 12th corporation is obviously different then the others because they accepted. Thus subjects in the 12th corporation may be more accepting or sensitive to the treatment. Slide 25: Experimental Procedures - The experimental procedures and arrangements have a certain amount of effect on the subjects in the experimental settings. Generalization to persons not in the experimental setting may be precluded.
Example: Department heads realize they are being studied, try to guess what the experimenter wants and respond accordingly rather than respond to the treatment. Slide 26: Multiple Treatment Interference - If the subjects are exposed to more than one treatment then the findings could only be generalized to individuals exposed to the same treatments in the same order of presentation.
Example: A group of CPA’s is given training in working with managers followed by training in working with comptrollers. Since training effects cannot be deleted, the first training will affect the second. Tools of Experimental Design Used to Control Factors Jeopardizing Validity : Tools of Experimental Design Used to Control Factors Jeopardizing Validity Pre-Test - The pre-test, or measurement before the experiment begins, can aid control for differential selection by determining the presence or knowledge of the experimental variable before the experiment begins. It can aid control of experimental mortality because the subjects can be removed from the entire comparison by removing their pre-tests.
However, pre-tests cause problems by their effect on the second measurement and by causing generalizability problems to a population not pre-tested and those with no experimental arrangements. Slide 28: Control Group -The use of a matched or similar group which is not exposed to the experimental variable can help reduce the effect of History, Maturation, Instrumentation, and Interaction of Factors. The control group is exposed to all conditions of the experiment except the experimental variable. Slide 29: Randomization - Use of random selection procedures for subjects can aid in control of Statistical Regression, Differential Selection, and the Interaction of Factors. It greatly increases generalizability by helping make the groups representative of the populations Slide 30: Additional Groups - The effects of Pre-tests and Experimental Procedures can be partially controlled through the use of groups which were not pre-tested or exposed to experimental arrangements. They would have to be used in conjunction with other pre-tested groups or other factors jeopardizing validity would be present. Group experimental designs… : Group experimental designs… 1. single-variable
2. factorial single-variable designs : single-variable designs …involve one manipulated independent variable
true experimental Experimental Designs : Experimental Designs Pre-Experimental Design - loose in structure, could be biased
True Experimental Design - greater control and refinement, greater control of validity
Quasi-Experimental Design - not randomly selected Pre-Experimental Design : Pre-Experimental Design True Experimental Design : True Experimental Design Experimental design (cont) : Experimental design (cont) Experimental design (cont) : Experimental design (cont) Quasi-Experimental Design : Quasi-Experimental Design Quasi-Experimental Design (cont) : Quasi-Experimental Design (cont) Quasi-Experimental Design (cont) : Quasi-Experimental Design (cont) Quasi-Experimental Design (cont) : Quasi-Experimental Design (cont)