Lists Home |
Date Index |
- To: firstname.lastname@example.org
- Subject: Avoiding roundoff errors in calculations?
- From: "Roger L. Costello" <email@example.com>
- Date: Fri, 29 Aug 2003 14:57:10 -0400
- Organization: The MITRE Corporation
Suppose that I would like to convert this Length to meters (there are 6
picas per inch):
Doing the arithmetic:
13 picas * in/6 picas * 2.54 cm/in * m/100 cm
= 33.02/600 m
= 0.055033333333333333333333333333 m (the "3" repeats infinitely)
The problem is that everyone has different precision requirements. Some
people would be happy with this as the result:
For other people such a roundoff could be catastrophic.
Even if I were to return a result that is as precise as my machine can
there is still a roundoff that may be unacceptable to some.
I am considering returning the result in numerator/denominator form, and
leaving it up to the recipient to do the division (and decide what
precision is suitable), i.e.,
What are your thoughts on this? What techniques do people use to avoid
introducing roundoff errors? /Roger