← Prev in month ← Prev in thread

Avoiding roundoff errors in calculations?

From
Roger L. Costello <>
To
Date
2003-08-29T18:57:10Z
ID
<>
Thread
Avoiding roundoff errors in calculations?
Hi Folks,

Suppose that I would like to convert this Length to meters (there are 6
picas per inch):

<Length>
    <value>13</value>
    <unit rdf:resource="#Pica"/>
</Length>

Doing the arithmetic:

13 picas * in/6 picas * 2.54 cm/in * m/100 cm 

    = 33.02/600 m
 
    = 0.055033333333333333333333333333 m (the "3" repeats infinitely)

The problem is that everyone has different precision requirements.  Some
people would be happy with this as the result:

<Length>
    <value>0.055</value>
    <unit rdf:resource="#Meter"/>
</Length>

For other people such a roundoff could be catastrophic.  

Even if I were to return a result that is as precise as my machine can
produce, 

<Length>
    <value>0.055033333333333333333333333333</value>
    <unit rdf:resource="#Meter"/>
</Length>

there is still a roundoff that may be unacceptable to some.

I am considering returning the result in numerator/denominator form, and
leaving it up to the recipient to do the division (and decide what
precision is suitable), i.e.,

<Length>
    <value>
        <numerator>33.02</numerator>
        <denominator>600</denominator>
    </value>
    <unit rdf:resource="#Meter"/>
</Length>

What are your thoughts on this?  What techniques do people use to avoid
introducing roundoff errors?  /Roger
← Prev in month ← Prev in thread