OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.

 


Help: OASIS Mailing Lists Help | MarkMail Help

 


 

   Avoiding roundoff errors in calculations?

[ Lists Home | Date Index | Thread Index ]
  • To: xml-dev@lists.xml.org
  • Subject: Avoiding roundoff errors in calculations?
  • From: "Roger L. Costello" <costello@mitre.org>
  • Date: Fri, 29 Aug 2003 14:57:10 -0400
  • Organization: The MITRE Corporation

Hi Folks,

Suppose that I would like to convert this Length to meters (there are 6
picas per inch):

<Length>
    <value>13</value>
    <unit rdf:resource="#Pica"/>
</Length>

Doing the arithmetic:

13 picas * in/6 picas * 2.54 cm/in * m/100 cm 

    = 33.02/600 m
 
    = 0.055033333333333333333333333333 m (the "3" repeats infinitely)

The problem is that everyone has different precision requirements.  Some
people would be happy with this as the result:

<Length>
    <value>0.055</value>
    <unit rdf:resource="#Meter"/>
</Length>

For other people such a roundoff could be catastrophic.  

Even if I were to return a result that is as precise as my machine can
produce, 

<Length>
    <value>0.055033333333333333333333333333</value>
    <unit rdf:resource="#Meter"/>
</Length>

there is still a roundoff that may be unacceptable to some.

I am considering returning the result in numerator/denominator form, and
leaving it up to the recipient to do the division (and decide what
precision is suitable), i.e.,

<Length>
    <value>
        <numerator>33.02</numerator>
        <denominator>600</denominator>
    </value>
    <unit rdf:resource="#Meter"/>
</Length>

What are your thoughts on this?  What techniques do people use to avoid
introducing roundoff errors?  /Roger





 

News | XML in Industry | Calendar | XML Registry
Marketplace | Resources | MyXML.org | Sponsors | Privacy Statement

Copyright 2001 XML.org. This site is hosted by OASIS