Here is an example in which a value rounds up in one type and down in another. Dividing 1 by 3 yields the binary result
, where the bar indicates a repeating bit pattern. To round to the 24 significant bits of the float format, express the intermediate result in the normalized form
, where the braces indicate bits beyond float precision.
rounds up to
when rounding to nearest. In double, the intermediate value is
, which rounds down to
. The situation is different because double has an odd number (53) of significant bits, versus an even 24 for float.
[Contents]
[Previous]
[Next]
Click the icon to mail questions or corrections about this material to Taligent personnel.
Copyright©1995 Taligent,Inc. All rights reserved.
Generated with WebMaker