[
Lists Home |
Date Index |
Thread Index
]
- To: xml-dev@lists.xml.org
- Subject: Avoiding roundoff errors in calculations?
- From: "Roger L. Costello" <costello@mitre.org>
- Date: Fri, 29 Aug 2003 14:57:10 -0400
- Organization: The MITRE Corporation
Hi Folks,
Suppose that I would like to convert this Length to meters (there are 6
picas per inch):
<Length>
<value>13</value>
<unit rdf:resource="#Pica"/>
</Length>
Doing the arithmetic:
13 picas * in/6 picas * 2.54 cm/in * m/100 cm
= 33.02/600 m
= 0.055033333333333333333333333333 m (the "3" repeats infinitely)
The problem is that everyone has different precision requirements. Some
people would be happy with this as the result:
<Length>
<value>0.055</value>
<unit rdf:resource="#Meter"/>
</Length>
For other people such a roundoff could be catastrophic.
Even if I were to return a result that is as precise as my machine can
produce,
<Length>
<value>0.055033333333333333333333333333</value>
<unit rdf:resource="#Meter"/>
</Length>
there is still a roundoff that may be unacceptable to some.
I am considering returning the result in numerator/denominator form, and
leaving it up to the recipient to do the division (and decide what
precision is suitable), i.e.,
<Length>
<value>
<numerator>33.02</numerator>
<denominator>600</denominator>
</value>
<unit rdf:resource="#Meter"/>
</Length>
What are your thoughts on this? What techniques do people use to avoid
introducing roundoff errors? /Roger
|