← Prev in month ← Prev in thread

RSQ - Unstable Algorithm

From
Leonard Mada <>
Date
2007-06-07T19:59:18+00:00
ID
Thread
RSQ - Unstable Algorithm
Hi,

section 6.16.63 of the OpenDocument Formula document describes  the 
RSQ() function:

''Summary: Returns the square of the Pearson product moment correlation 
coefficient through data points in known_y's and known_x's.''

Unfortunately, the formulas/algorithms given for 'a' and 'b' seem to be 
computationally unstable
 - they both use terms of the form n*sum(x^2) - (sum(x))^2 which are 
notoriously unstable

Although I did NOT do a formal calculation of the error term, having 
*two* occurrences of  this unstable calculation, and both times as 
*divisors* makes the formula very unreliable.

SUGGESTIONS:
 - remove formula from specification
 - consult statistician knowledgeable of stable coding algorithms

Sincerely,

Leonard Mada
← Prev in month ← Prev in thread