Data Access Worldwide Knowledge Base
Article Text
QUESTION:
I have a function that converts an integer to a number using the specified number of decimal places. The number returned is sometimes wrong. Is this a bug?
The function is
Function ConvertNumber Number nValue Integer iDecimals Returns Number nNewValue
Move (nValue * (10^(-iDecimals ))) to nNewValue
Function_Return nNewValue
End_Function
If nValue is 12345678901 and iDecimals is 2, nNewValue will be 123456789.01000001.
ANSWER:
This is not a bug. What happens is that the ^ (power of) operator operates on the Real data type and the result of that calculation will be a Real.
From your example,
Move (nValue * (10^(-iDecimals ))) to nNewValue
the whole expression will be converted to Real -- since the result for the "power of" expression was a Real -- and then the calculations are performed using Real data type and then the result is converted to Number.
If you separate that expression into two (like below), the first expression will generate a Real as a result of the "power of" expression, and then the result will be converted to Number (stored in nCalc).
When calculating the second expression, only the Number data type will be used -- resulting in the expected value stored in nNewValue.
Number nCalc
Move (10^(-iDecimals)) to nCalc
Move (nValue * nCalc) to nNewValue
So, in order to have the behavior you were expecting, you can change the function to the above or you can use the Number function as in
Move (nValue * (Number(10^(-iDecimals))) ) to nNewValue
Remember: whenever using "power of", the return value will always be Real.
Contributed By:
Sonny Falk
Company: Data Access Worldwide
Web Site: http://www.dataaccess.com
Email this Article
Email this Article to a Colleague
Send Feedback on this Article to Data Access Worldwide
Copyright ©2010 Data Access Corporation. All rights reserved.
| The information provided in the Data Access Technical Knowledge Base is provided "as is" without warranty of any kind. Data Access Corporation disclaims all warranties, either express or implied, including the warranties of merchantability and fitness for a particular purpose. In no event shall Data Access Corporation or its suppliers be liable for any damages whatsoever including direct, indirect, incidental, consequential, loss of business profits or special damages, even if Data Access Corporation or its suppliers have been advised of the possibility of such damages. Some states do not allow the exclusion or limitation of liability for consequential or incidental damages so the foregoing limitation may not apply. |