Overcoming rounding errors

September 25, 2008 at 8:39 pm 1 comment

Floating point variables are prone to rounding errors.  (That is, the Number type in ActionScript)  Often these don’t matter.  But in some applications, especially those that process financial information, it is important to avoid errors.  Especially errors that propagate.  I am interested in eliminating rounding errors because of my spreadsheet AIR application.  e2spreadsheet.  The current beta version doesn’t do anything about these errors, but I want to remedy that in the next version.

For example, if you open up e2spreadsheet, and type in the following formula: =74.725*100 what answer do you expect?  7472.5 right?  Well, the answer that you actually get is 74.499999999999

Because the binary representation has a finite length, it cannot accurately represent some decimal values.

So, how do we get around this?  Well, my first thought was a number representation called Binary Coded Decimal (BCD).  I think Cobol used to use this representation.  But that would have involved many program operations to simulate each BCD calculation.  Then this blog by Josh Tynjala gave me another idea.  (Ok, probably not a new idea – but it was a revelation to me).

What if I represent my numbers using a radix-10 (decimal) mantissa and exponent.

Actually floating point numbers are represented internally in binary mantissa-exponent form. So that…


In other words, the mantissa is shifted left or right depending on the value of the exponent.

Rounding errors occur because finite fractional binary numbers don’t translate well into decimal numbers.  So the solution is to simulate this mantissa-exponent form in decimal (radix-10).  We would write a class so that value=mantissa*Math.pow(10,exponent);  And this class would contain all the methods necessary to do arithmetic operations on this representation.  Add, Subtract, Multiply, etc..

So, with that idea in my head, I wrote the following code. (previous blog entry which appears below).  Don’t rely on this yet.  I wrote and tested it quickly, and I wanted to see what other developers think about the idea.  Is this a good way to eliminate rounding errors?  Or is there a better way I haven’t considered?


Entry filed under: Adobe AIR.

Overcoming rounding errors (code) Flash CS4 tweening. Nothing new.

1 Comment Add your own

  • 1. Clint  |  June 19, 2009 at 5:32 pm

    Store it as a string 😉 Of course, then you lose all speed benefits, tee-hee.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed

  RSS feed          View Daniel Freeman's LinkedIn profileView my profile

Add to Technorati Favorites

September 2008
    Oct »

%d bloggers like this: