Slide1:
Reviewer’s
Secrets
Revealed!
James R. Van Scotter, Ph.D., ISDS Department, Louisiana State University
About This Presentation: About This Presentation Is it about learning how to review papers?
About This Presentation: About This Presentation ..or, is it about how reviewers evaluate your papers?
About This Presentation: About This Presentation …or, both!
Basic Questions for Reviewers: Should this paper be published?
(Why or why not?)
Does it contribute to scientific knowledge? (How?)
Can the paper be strengthened?
Basic Questions for Reviewers
Criteria: Schwab’s (1985) Criteria
1. Technical merit
Internal validity, construct validity, causal issues.
2. Craftsmanship
Organization, clarity
3. Significance or importance
Tests a reasonable theory in non-trivial way.
Improves an existing body of empirical research.
Has implications for an important policy issue. Criteria Source: Schwab, D. (1985). Reviewing empirically-based manuscripts: Perspectives on
Process. In L. Cummings andamp; R. Frost (Eds.). Publishing in the organizational sciences. Homewood, IL: Richard D. Irwin, 171-181.
Criteria: Why I Recommended Rejecting Your Manuscript…
No theory
Theory should explain what the variables mean,
how they are related and why they are expected to be
related to one another.
Concepts and operationalization not aligned
Variables do not reflect the concepts or the logical links
have not been developed.
Criteria Source: Daft, R. (1985). Why I recommended rejecting your manuscript and what you can do about it. In L. Cummings andamp; R. Frost (Eds.). Publishing in the organizational sciences. Homewood, IL: Richard D. Irwin, 193-209
Criteria: Why I Recommended Rejecting Your Manuscript…
3. Insufficient definition - Theory
Concepts are not defined or justified. It is not clear why
variables were included. Some were included simply because
they had appeared in other studies.
4. Insufficient rationale – Design
Lack of explanation for study procedures, description of
sample, etc. Nothing is obvious or assumed. Criteria Source: Daft, R. (1985). Why I recommended rejecting your manuscript and what you can do about it. In L. Cummings andamp; R. Frost (Eds.). Publishing in the organizational sciences. Homewood, IL: Richard D. Irwin, 193-209
Criteria: Why I Recommended Rejecting Your Manuscript…
5. Insufficient research design
Usually fatal. Using survey questions to study subtle processes.
Using undergrad sample for strategy research (CEO, CIO level).
6. Lack of organization and flow
No single story line. Mismatch between tests and hypotheses.
7. Amateurish style and tone
Overly negative approach to previous literature.
Exaggerated sense of importance. Logical gaps in writing. Criteria Source: Daft, R. (1985). Why I recommended rejecting your manuscript and what you can do about it. In L. Cummings andamp; R. Frost (Eds.). Publishing in the organizational sciences. Homewood, IL: Richard D. Irwin, 193-209
Criteria: Leading Reasons for Rejection by JIS
Lack of theory
Weakness in Design
Poor Motivation for the Study
Other Reasons for Rejection
Lack of Fit with Journal
Insufficient Development
Insufficient Contribution
Methodological Problems Criteria Source: Murthy, U. andamp; Wiggins, C., (2002). Why manuscripts are rejected:
An analysis of JIS rejections. Journal of Information Systems, 16, 1, 41-48.
Review Process: Peer review is a complex social act.
All referees are not experts in the areas they review.
Recommendations are based on the editor’s or
reviewer’s personal preferences.
3. Reviewers’ recommendations may conflict.
4. When recommendations become too detailed,
reviewers become ghost writers.
5. Nobel prize winning research has been rejected. Review Process Source: Bedeian, A. (2004). Peer Review and the social construction of knowledge
In the management discipline. Academy of Management Learning and Education,
3, 2, 198-216.
Review Process: Survey of Reviewer Practices
Typical reviewer spends 5.4 hours per paper.
They reviewed 1.5 papers per month.
Two dominant roles: Critic and Coach. Review Process Source: Jauch, L. andamp; Wall, J. (1989). What do they do when they get your
manuscript: A survey of Academy of Management Reviewer Practices. Academy
of Management Journal, 32, 1, 157-173.
Process: Developmental Reviewing
More than just avoiding a negative tone.
Two main ingredients:
Helping the author(s) understand what is wrong.
Helping the author(s) figure out how to fix it.
Process Source: Feldman, D. (2004). Editorial: Being a developmental reviewer:
Easier said than done. Journal of Management, 30, 2, 161-164.
Process: Developmental Reviewing
3. Procedure:
Start with overall evaluation.
Describe aspects of the paper that are strong/weak.
Identify major and minor issues.
Suggest solutions for methods problems.
Suggest areas that should be cut or expanded.
Process Source: Feldman, D. (2004). Editorial: Being a developmental reviewer:
Easier said than done. Journal of Management, 30, 2, 161-164.
Process: Developmental Reviewing
4. Specific recommendations.
No sample is perfect. Help the author(s) make
the best use of the data.
b. Give examples of improvements (e.g., clearer
hypotheses) you would like to see.
If appropriate, provide references for papers
with an alternative perspective. Process Source: Feldman, D. (2004). Editorial: Being a developmental reviewer:
Easier said than done. Journal of Management, 30, 2, 161-164.
What to Do: What to Do
What to Do: What to Do 'Science depends on qualitative, common-sense knowing … (Campbell, 1979, p.50)'
Campbell, D. T. (1979). Degrees of freedom and the case study, in T. D. Cook, andamp; C. S. Reichardt (Eds.), Qualitative and quantitative methods in evaluation research, Beverly Hills, CA: Sage Publications, pp.49-67.
Influential Articles: Analysis of Influential Articles
1. Obvious contribution to knowledge
2. Results of major theoretical significance
3. Useful new theory/theoretical framework
4. Generates new research
5. Provides new and exciting ideas
6. Integrates several areas
7. Integrates existing theories into a simpler framework Influential Articles Source: Sternberg, R. andamp; Gordeeva, T. (1996). The anatomy of impact: What
Makes an article influential. Psychological Science, 7, 2, 69-75.
Slide19: A research paper should be like a present.
The general concept should be appealing,
the contents should be valuable, and everything necessary should be wrapped up in the package.
Six General Goals of Science: Six General Goals of Science Organize andamp; categorize things (typologies)
Explain past events
Predict future events
Control future events
Provide a sense of understanding
Generalize results
Portions Adapted from:
Reynolds, P. D. (1971). A primer in theory construction. Indianapolis, The Bobbs-Merrill Company.
How does science accomplish its goals?: How does science accomplish its goals? Formulating hypotheses andamp; descriptive statements that:
Are derived from theories
Are falsifiable
Can be tested empirically
Portions Adapted from:
Reynolds, P. D. (1971). A primer in theory construction. Indianapolis, The Bobbs-Merrill Company.
How does science accomplish its goals?: 2. Developing models that summarize important attributes/relationships
Portions Adapted from:
Reynolds, P. D. (1971). A primer in theory construction. Indianapolis, The Bobbs-Merrill Company. How does science accomplish its goals?
How does science accomplish its goals?: 3. Collecting data
Manipulating relevant variables
Measuring relevant variables
Controlling and/or measuring nuisance variables
Portions Adapted from:
Reynolds, P. D. (1971). A primer in theory construction. Indianapolis, The Bobbs-Merrill Company. How does science accomplish its goals?
How does science accomplish its goals?: 4. Using empirical data to test the hypotheses
Portions Adapted from:
Reynolds, P. D. (1971). A primer in theory construction. Indianapolis, The Bobbs-Merrill Company. How does science accomplish its goals?
How does science accomplish its goals?: 5. Generalizing research results to a larger context
Portions Adapted from:
Reynolds, P. D. (1971). A primer in theory construction. Indianapolis, The Bobbs-Merrill Company. How does science accomplish its goals?
General Principles: General Principles
General Principles: General Principles Research as story telling
and good writing.
2. Favor simple explanations (parsimony).
3. Use models to summarize important relationships.
Portions adapted from:
Ableson, R. P. (1995). Statistics as principled argument. Hillsdale, NJ: Lawrence Erlbaum Publishers.
General Principles: General Principles Collecting data to test models/hypotheses.
5. Using comparison as the evaluation framework.
6. Ruling out alternative explanations.
Portions adapted from:
Ableson, R. P. (1995). Statistics as principled argument. Hillsdale, NJ: Lawrence Erlbaum Publishers.
Story Telling: Story Telling From general to specific
From specific to general Hour glass shape
Story Telling: Story Telling Introduction
Literature Review
Methods
Results
Discussion
Conclusion andamp; Recommendations
Story Telling: 1. Why is the topic important and interesting?
Theoretical andamp; practical value
Appropriateness for journal
Timeliness
Novelty
Story Telling
Portions adapted from:
Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles
in applied psychology. Personnel Psychology, 46, 1-14.
Story Telling: 2. Literature review
Link to important literature
Logic for hypotheses andamp; propositions
Context for study (samples/methods)
Use literature to develop arguments
Thorough exposition
Accurate interpretation (including nuances)
c. Lead to a conclusion about what is needed
d. Establish incremental value of the paper
Story Telling Portions adapted from:
Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles
in applied psychology. Personnel Psychology, 46, 1-14.
Story Telling:
3. Conceptual development
Adequacy andamp; scope of research hypotheses
Clarity and logical coherence (one main story line)
Explains how things work without leaps of faith or logical gaps
4. Have a point (not just a point of view)
Story Telling
Portions adapted from:
Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles
in applied psychology. Personnel Psychology, 46, 1-14.
Parsimony: Parsimony
Simple explanations are better.
Simple explanations are more likely to be testable.
Parsimony: Parsimony Parsimony is related to goals of scientific research
Simplifying complex issues
Separating important from unimportant factors
Generalizing across time, place, samples
Providing a benchmark
Slide36: Theoretical statement goes here Theoretical statements should be concise enough to fit on a bumper sticker. …and simple enough that your grandmother could understand them.
Models: Models
Simplify communication of complex relationships
Consistent with idea that measures of the same variable are interchangeable
Encourage thinking about alternative explanations
Models: Models
4. Testing process must be guided by theory,
not the modeling method.
5. All 6 models above have the same fit
(Bollen, 1989, p.70)
Methods: Methods Sample design
1. Representativeness andamp; generalizability
Appropriateness for research question
Demographics for subjects
Research setting andamp; context
2. Suitability for tests andamp; inference (statistical assumptions)
Sampling method (random, purposive, convenience)
Comparison with population, or samples in related studies
Sufficiency (sample size, response rate, attrition) Portions adapted from:
Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14.
Methods: Methods Measurement
Justify constructs
Purpose
Theory andamp; Prior research
Operationalization of constructs
Use well known instruments when possible
Use archival data when appropriate
Use multiple sources of measures when possible
Avoid single item measures
Portions adapted from:
Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14.
Methods: Methods Measurement
3. Distinguish between measures of perceptions, intentions versus behaviors and outcomes.
4. Avoid criterion contamination or assess andamp; report it.
5. Use appropriate level of analysis; avoid ecological fallacy.
Portions adapted from:
Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14.
Methods: Methods Measurement
6. Provide appropriate reliability evidence.
Internal consistency
Inter-rater
Test-retest, Alternate forms
Average Variance Extracted
7. Provide appropriate validity evidence.
Convergent andamp; discriminant
Construct
Content Portions adapted from:
Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14.
Methods: Methods Design
1. Use pilot studies
Test manipulations/treatments and instruments
Test sampling methods and procedures
2. Use standard administration methods to avoid biasing results
3. Use ethical procedures; comply with rules for treatment of human subjects Portions adapted from:
Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14.
Methods: Methods Design (cross-sectional studies)
4. Use designs appropriate to the hypotheses
Tests of alternative models/explanations
Differential predictions
NOT the general null hypothesis
5. Control for statistical and sampling artifacts
Restriction in range
Common method bias
Uses appropriate control variables Portions adapted from:
Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14.
Methods: Methods Design (experimental/quasi-experimental)
6. Use random assignment procedures; provide information on comparability of groups.
7. Use adequate manipulations/perform checks.
8. Avoid threats to validity.
Internal andamp; external threats
Experimenter bias
Portions adapted from:
Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14.
Framework for Evaluation: Framework for Evaluation Empirical research tests relationships between
observed variables
Framework for Evaluation: Framework for Evaluation Theories focus on relationships between
Latent (unobservable) variables
Framework for Evaluation: Framework for Evaluation Latent Independent Variable Latent Dependent Variable Observed Independent Variable Observed Dependent Variable What we
want to know What we
can test Construct
validity Construct
validity Statistical
Conclusion
Validity Statistical
Conclusion
Validity Construct
validity
Methods: Methods Comparative Evaluation
1. Tests of different groups
Natural categories
Treatment/control groups; multiple treatment groups
2. Tests of alternative hypotheses
Competing explanations
Nested models
Goal is to find evidence that contradicts one explanation and supports another Portions adapted from:
Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14.
Methods: Methods Data Analysis
Use appropriate statistics
Demonstrate awareness of major assumptions
Conduct supplemental analyses suggested by the study
2. Avoid capitalization on chance
Examine outliers as needed
Control experiment-wise error rate
3. Report effect sizes, confidence intervals, significance levels Portions adapted from:
Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14.
Ruling Out Alternatives: Ruling Out Alternatives
Some sources of errors
Design
Sample
Instruments
Experimenter andamp; situational errors
Demand artifacts
Results: Results Summarize results briefly
Link findings to hypotheses and interpret them
Relate findings to other studies
Derive specific theoretical and practical implications
Provide suggestions for future research Portions adapted from:
Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14.
Story Telling: Story Telling Writing quality, simple sentences, proper grammar, active voice
Logical organization andamp;
sense of direction
3. Succinct and shows attention to detail
4. Tell a consistent, integrated story Portions adapted from:
Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14.
Story Telling: Why should this paper NOT be published?
Does the theoretical development make sense?
Do the authors explain why AB?
Are there any logical gaps or leaps of faith? Story Telling
Story Telling: Why should this paper NOT be published?
Do the authors test what they said they would?
Are the data and methods appropriate?
What are the theoretical implications?
What does this paper add to the literature?
Story Telling
Additional: Journals with 75% or more articles by IS Authors
Journal
Journal Pct Ranking
MIS Quarterly .96 A
Information Systems Research .95 A
Journal of Management Information Systems .92 A
Communications of the AIS .91 B
The Data Base for Advances in Info Systems .84 A
International Journal of Electronic Commerce .80 A
Journal of the AIS .80 B
Journal of Global Information Management .76 ? Additional Source: Chua, C., Cao, L., Cousins, K., andamp; Straub, D. (2002). Measuring researcher
production in information systems. Journal of the AIS, 3, 145-215.
Publish or Perish: The 7 P’s
Proper
Prior
Planning
Prevents
Poor
Publication
Performance
Publish or Perish Source: Unknown
Additional References: Additional References Bhattacharjee, S., Tung, Y., andamp; Pathak, B. (2004). Author experiences with
the IS journal review process. Communications of the AIS, 13, 629-653.
Mingers, J. (2002). The long and winding road: Getting papers published
In top journals. Communications of the AIS, 8, 330-339.
Slide59: The End