In the "Lua Programming Manual" it states
| Quote |
|---|
2.3 - Numbers
The number type represents real (double-precision floating-point) numbers. Lua has no integer type, as it does not need it. There is a widespread misconception about floating-point arithmetic errors and some people fear that even a simple increment can go weird with floating-point numbers. The fact is that, when you use a double to represent an integer, there is no rounding error at all (unless the number is greater than 100,000,000,000,000). Specifically, a Lua number can represent any long integer without rounding problems. Moreover, most modern CPUs do floating-point arithmetic as fast as (or even faster than) integer arithmetic. |
I had some troubles with one of my program loops where I wanted to subdivide the picture into discrete cells. Occasionally I would have overlapping cells. As the map input range is scaled in the range [0..1] for the Y axis I needed to change this to an integer value. It took me some time to find that the trouble in my algorithm lay with the maths.floor() function giving incorrect results. Contrary to what the manual says, it may be necessary at times to make allowances for small rounding errors. For example the following code shows how maths.floor() does not give the expected answer.
| Code |
|---|
for i=0,1,0.1 do
j = i * 10
k = math.floor(j)
print(i,j,k)
end
|
The output from this is:
| Code |
|---|
0 0 0
0.1 1 1
0.2 2 2
0.3 3 3
0.4 4 4
0.5 5 5
0.6 6 6
0.7 7 7
0.8 8 7 << incorrect
0.9 9 9
1 10 9 << incorrect |
To ensure you get the correct results you should add a small offset
| Code |
|---|
| k = math.floor(j+0.000001) |
Such a small offset should not affect your application but will ensure that the results are as expected. Using this offset the results are:
| Code |
|---|
0 0 0
0.1 1 1
0.2 2 2
0.3 3 3
0.4 4 4
0.5 5 5
0.6 6 6
0.7 7 7
0.8 8 8 << correct
0.9 9 9
1 10 10 << correct
|
which is correct.
After I modified my math.floor() functions in this way my algorithm worked correctly.
I hope this helps someone who has troubles with such logic.
