larc stated:
Alan,
Since you are mathematicly gifted, I would like run an idea by you.
AlanF has not yet responded, so I'll put my oar in if I may. What I don't comment on, I don't have an issue with.
It has to do with the degree of accuracy in human judgements. The standard method for determining this is to correlate one person's quantified judgements, e.g., on a rating scale, with another person's judgements. The correlation coefficient is assumed to be the proper index of accuracy. This has been the standard metric for the past 90 years. However, there is a problem here. One person is being compared to imperfect standard, i.e., another person. Therefore, the correlation is an underestimate of one person's accuracy.
The last statement is incorrect. The two people could be making the same, though wrong, judgements (could happen for a variety of reasons - to name but one, common socio-economic-educational sources of "error") - in which case the correlation coefficient would be an overestimate (not an underestimate) of the target person's accuracy.
If a person could be compared to a "true score", this would be a better index. Empirical tests have been done by correlating one person's score with the sum of a larger number of others.
Please be very explicit here - there are at least two very different ways to interpret what you have just stated.
Did you mean using a population "average" instead of a second individual for the purpose of comparison?
Measurement theory assumes that as a larger sum is used, the error term decreases, since error between people is random,
Please justify the last clause. Most "random" things aren't.
therefore, the sum approaches the theoretical true score.
I do not understand this statement - please be specific and use precise terminology. Is regression towards the mean meant?
There is a body of research using this line of reasoning. Another approach is to correct the correlation between two people by taking the square root of the correlation.
As a typically-defined correlation coefficient belongs to [-1,1], half the range will produce an imaginary result when square-rooted.
No one, to date, has tyed these to methods together. I conducted an experiment and found the correlation between (1) the summed scores approach and (2) the square root approach is .982.
You do not mention sample size. One cuckoo doth not a spring make, or however the saying goes.
Not bad, for a result in the "soft science" of psychology. I met great resistance from the journal reviewers and the journal editor. Finally, the editor wrote that if I obtained the empirical results from a second sample and applied it back to math corrections in the first sample, he would give it another look. I am in the process of doing this within the next two weeks. Anyway, I ran this by you to see if you had any thoughts on my logic or the math involved.
If you are more specific, I am sure meaningful opinions can be expressed.
--
Focus
(The Generation of Random Numbers is FAR too important to be left to Pure Chance! Class)