Testing the value of expert insight: Comparing local versus general expert judgment models

Expert judges often claim to utilize expert insight to tailor judgments to maximize predictive validity for a specific context. We evaluated multi-organizational assessment data regarding the prediction of supervisory ratings of job performance from ratings on individual assessment dimensions, finding no evidence that the average expert assessor effectively tailored judgments to specific organizations to maximize prediction. Expert judgment was outperformed in all organizational contexts by linear models of expert judgment, optimal weighted regression models, as well as simple sum composites. Critically, the dimension weighting policies of the expert assessors were not consistent with optimal weights for predicting job performance at any organization. We discuss why expertise tends not to contribute to predictive validity and describe methods for improving overall judgmental accuracy.