Editorial summary. This is our text summary of an article published by plos. Charts, figures, and the author’s full voice are at the original — read it there .
Licence. The original is licensed under CC BY 4.0. This page adapts it: the summary and analysis are ours, not the authors’.
Editorial verdict
Methodologically rigorous reanalysis. The publication bias finding — that conscientiousness validity is overestimated by about 30% overall, driven by journal-published studies (about 63% for journals alone) — is well-supported across multiple convergent methods, though the study is limited to a single existing dataset and cannot fully resolve heterogeneity concerns.
Executive summary
This study addresses the robustness of meta-analytic validity estimates for conscientiousness as a predictor of job performance in personnel selection contexts. The authors argue that existing meta-analytic estimates overstate the true predictive validity of conscientiousness due to publication bias, a phenomenon rarely examined through sensitivity analyses in organizational science. Using data from Shaffer and Postlethwaite's comprehensive meta-analysis (k = 113 correlation coefficients), the authors applied multiple sensitivity analysis methods including contour-enhanced funnel plots, trim-and-fill analysis, selection models, PET-PEESE, and excess significance tests. Results across methods converged on a finding that the random-effects mean validity of .16 is overestimated by approximately 30%, with bias-adjusted estimates ranging from .12 to .13. Publication bias was found to originate primarily from journal-published studies (mean r = .19) rather than non-journal sources (mean r = .12), consistent with suppression of statistically non-significant small-sample findings. The authors conclude that inflated validity estimates lead to overestimates of personnel selection dollar utility by millions of dollars, and call for mandatory sensitivity analyses in all meta-analytic reviews in organizational science.
Key insights
- 1The observed meta-analytic validity of conscientiousness for predicting job performance (.16) is overestimated by approximately 30%, with bias-corrected estimates clustering around .12–.13 across multiple methods.
- 2Journal-published studies are the primary driver of publication bias, reporting a mean validity of .19 compared to .12 for non-journal sources, with journal distributions showing misestimation of up to 63%.
- 3Sensitivity analyses — including outlier detection, trim-and-fill, selection models, and PET-PEESE — are rarely conducted in organizational science meta-analyses despite being recommended by the American Psychological Association and the Cochrane Collaboration, leaving cumulative knowledge potentially distorted.
Practical takeaways
- Organizations relying on conscientiousness validity estimates from journal-based meta-analyses to calculate dollar utility of personnel selection programs may be overestimating the financial value of such programs by millions of dollars when using inflated validity coefficients.
- Meta-analytic reviews in applied psychology and management gain credibility when reporting a range of parameter estimates derived from multiple publication bias detection methods rather than a single point estimate, as convergence across methods strengthens confidence in conclusions.
Frameworks mentioned
Big Five Personality Traits
A model describing five broad dimensions of human personality — including conscientiousness — used as a framework for personality assessment in personnel selection research.
PET-PEESE
Precision-Effect Test and Precision Effect Estimate with Standard Error — a two-stage meta-regression method used to detect and correct for publication bias by modeling the relationship between effect sizes and their standard errors.
References
- Shaffer and Postlethwaite (2012).Meta-analytic assessment of the validity of conscientiousness.
- Biostat (2005).Comprehensive Meta-Analysis (CMA, version 2.0).
- Greenhouse and Iyengar (2009).The Handbook of Research Synthesis and Meta-Analysis (chapter).
- Kepes et al. (2012).Publication bias in the organizational sciences.
- Hedges and Olkin (1985).Statistical Methods for Meta-Analysis.
- Hunter and Schmidt (2004).Methods of Meta-Analysis: Correcting Error and Bias in Research Findings.
- Duval and Tweedie (2000).Trim and fill: A simple funnel-plot-based method for testing and adjusting for publication bias in meta-analysis.
- Vevea and Woods (2005).Step function selection models for meta-analysis.
- Francis (2014).The frequency of excess success for articles in Psychological Science.
- Stanley and Doucouliagos (2014).PET-PEESE meta-regression procedures.
- Egger et al. (1997).Egger's test of the intercept.
- van Assen et al. (2015).p-uniform method for publication bias correction.
- Viechtbauer and Cheung (2010).Outlier and influence diagnostics for meta-analysis.
- McDaniel, Rothstein, and Whetzel (2006).Publication bias and the validity of employment tests.
- American Psychological Association (2008).Meta-analysis Reporting Standards.
- Cochrane Collaboration (2011).Cochrane Handbook for Systematic Reviews of Interventions.
- Costa and McCrae (1992).NEO Personality Inventory.
Source & Provenance
plos
Sven Kepes, Michael A. McDaniel
October 30, 2015
Research Study
Global
Original source metadata is preserved. AI analysis is generated separately.
Like this? Get the Monday Decision Brief — free, every week.
No spam, unsubscribe anytime.
Rate this article
Want the full article? Read it at the original source — free, no paywall.
Read original article