OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help



   Avoiding roundoff errors in calculations?

[ Lists Home | Date Index | Thread Index ]
  • To: xml-dev@lists.xml.org
  • Subject: Avoiding roundoff errors in calculations?
  • From: "Roger L. Costello" <costello@mitre.org>
  • Date: Fri, 29 Aug 2003 14:57:10 -0400
  • Organization: The MITRE Corporation

Hi Folks,

Suppose that I would like to convert this Length to meters (there are 6
picas per inch):

    <unit rdf:resource="#Pica"/>

Doing the arithmetic:

13 picas * in/6 picas * 2.54 cm/in * m/100 cm 

    = 33.02/600 m
    = 0.055033333333333333333333333333 m (the "3" repeats infinitely)

The problem is that everyone has different precision requirements.  Some
people would be happy with this as the result:

    <unit rdf:resource="#Meter"/>

For other people such a roundoff could be catastrophic.  

Even if I were to return a result that is as precise as my machine can

    <unit rdf:resource="#Meter"/>

there is still a roundoff that may be unacceptable to some.

I am considering returning the result in numerator/denominator form, and
leaving it up to the recipient to do the division (and decide what
precision is suitable), i.e.,

    <unit rdf:resource="#Meter"/>

What are your thoughts on this?  What techniques do people use to avoid
introducing roundoff errors?  /Roger


News | XML in Industry | Calendar | XML Registry
Marketplace | Resources | MyXML.org | Sponsors | Privacy Statement

Copyright 2001 XML.org. This site is hosted by OASIS