5

In the article Oh & Berry (2009), p. 1506, in the note for Table 2, a certain statistic is used:

"Operational (true) validity is the LISREL estimated correlation corrected for measurement error in the criterion measure"

Can anyone explain a) what this means, b) why/when it's used (why not just use path coefficients as in a standard structural equations model?), and c) how to interpret this statistic?

Reference

Oh, I.-S., & Berry, C. M. (2009). The five-factor model of personality and managerial performance: validity gains through the use of 360 degree performance ratings. The Journal of Applied Psychology, 94(6), 1498–513. doi:10.1037/a0017221

Glen_b
  • 282,281
histelheim
  • 2,993

2 Answers2

2

Operational validities is a term used in psychology research for correlation. The reason they are called validities is that they can be use to test the validity of a construct through confirming that it correlates with other constructs according to a theoretically expected pattern.

Adjusting for error in the criterion measure simply means that the dependent variable has been adjusted for measurement error. This reduces the total amount of variance, and therefore the ratio between shared variance and total variance goes up. This means that the operational (true) validity, which is adjusted for error, is slightly higher than just the operational validity, which is the raw correlation.

histelheim
  • 2,993
0

In personnel selection settings (e.g., industrial/organisational psychology), a validity coefficient typically refers to the Pearson correlation between a predictor (e.g., general mental ability) and an outcome of interest (e.g., job performance or training performance).

When you do research to estimate the correlation between predictor and criterion, there are a range of corrections that can be performed. You can correct for reliability in the predictor and you can correct for reliability in the criterion. In the context of employee selection, you can also correct for range restriction. For example, if you correlate general mental ability measured in a job applicant setting and job performance and you hire only those with higher general mental ability, then your correlation will be reduced due to range restriction on the predictor.

Operational validity corrects for measurement error in the criterion and range restriction. The aim is to estimate validity of the predictor measure in operational settings. Thus, you don't correct for measurement error on the predictor (e.g., GMA) because to some extent that is intrinsic to using the measurement approach. You do correct for measurement error on the criterion because the operational value of the predictor is determined by how well you can predict the true outcome (i.e., true performance) and not the observed criteria (which is measured with error).

Here is a quote and a reference. The article includes relevant formulas for applying these corrections:

In personnel selection settings, operational validity refers to an estimate of the relationship between a predictor used in the practical context of selection and the theoretical construct that a criterion intends to measure (Binning & Barrett, 1989). However, top- down selection on the predictor (as might be typical in selection) obviously restricts predictor scores, and consequently, it also incidentally restricts scores on the criterion. This not only leads to a range-restricted validity coefficient; it also restricts the estimate of reliability for criterion scores (Sackett, Laczo, & Arvey, 2002). In addition to considering and correcting for range restriction, operational validities are corrected for measurement error in the criterion measure but not in the predictor measure. (Brown et al 2017)

  • Brown, R. D., Oswald, F. L., & Converse, P. D. (2017). Estimating Operational Validity Under Incidental Range Restriction: Some Important but Neglected Issues. Practical Assessment, Research & Evaluation, 22(6). https://pareonline.net/getvn.asp?v=22&n=6
Jeromy Anglim
  • 44,984