tests sc_array_functions_test fail due to floating point precision

classic Classic list List threaded Threaded
7 messages Options
Milton Vandersloot Milton Vandersloot
Reply | Threaded
Open this post in threaded view
|

tests sc_array_functions_test fail due to floating point precision

Dear LibreOffice developers

I was told to report this bug here[2]. So here we go:

The test ArrayFunctionsTest::testArrayFormulasFODS (files sc/qa/unit/functions_test.cxx, sc/qa/unit/functions_array.cxx) fails the test document sc/qa/unit/data/functions/array/fods/linest.fods. Error message in the compile run is (Note that 1 = TRUE, 0 = FALSE in the document)


functions_test.cxx:43:Assertion
Test name: ArrayFunctionsTest::testArrayFormulasFODS
double equality assertion failed
- Expected: 1
- Actual  : 0
- Delta   : 1e-14


The issue seems to be (again) floating point precision. Indeed, I've compiled LibreOffice with FMA (fused multiply-add) instructions, if I disable them, i.e. passing -ffp-contract=off to GCC, then the tests succeed. Note that FMA has slightly different floating point precision than ordinary plus and mult [0, 1].

I've also opened the other test documents in sc/qa/unit/data/functions/array/fods/ using LibreOffice. The documents which fail are linest.fods and also logest.fods. The others succeed.


Some mentionable commits:

* Add logest.fods test: https://cgit.freedesktop.org/libreoffice/core/commit/?id=588ebef9e8d1e2c703b79f51f2a1c8004cd7d96d
* Add linest.fods test: https://cgit.freedesktop.org/libreoffice/core/commit/?id=0523e11a2530ed101c580f93f94a4180e9dc0c9e
* Disable tests on 32-bit x86-linux due to floating point weirdness: https://cgit.freedesktop.org/libreoffice/core/commit/?id=b9a27d5856f60688456762bfcc29c38670009254

Best
Milton

[0] https://en.wikipedia.org/wiki/Fused_multiply-add
[1] https://github.com/uclouvain/openjpeg/issues/1017
[2] https://bugs.documentfoundation.org/114051
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Eike Rathke-2 Eike Rathke-2
Reply | Threaded
Open this post in threaded view
|

Re: tests sc_array_functions_test fail due to floating point precision

Hi Milton,

On Saturday, 2017-12-02 02:47:03 -0500, Milton Vandersloot wrote:

> The issue seems to be (again) floating point precision. Indeed, I've compiled LibreOffice with FMA (fused multiply-add) instructions, if I disable them, i.e. passing -ffp-contract=off to GCC, then the tests succeed. Note that FMA has slightly different floating point precision than ordinary plus and mult [0, 1].

From what I understood by browsing shortly gcc enables this by default
for -std=gnu* and it can be unset by

#pragma STDC FP_CONTRACT OFF

https://stackoverflow.com/questions/15933100/how-to-use-fused-multiply-add-fma-instructions-with-sse-avx#15933677

Could you try if adding the pragma to include/sal/config.h helps?

Thanks
  Eike

--
LibreOffice Calc developer. Number formatter stricken i18n transpositionizer.
GPG key 0x6A6CD5B765632D3A - 2265 D7F3 A7B0 95CC 3918  630B 6A6C D5B7 6563 2D3A
Care about Free Software, support the FSFE https://fsfe.org/support/?erack

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (849 bytes) Download Attachment
Milton Vandersloot Milton Vandersloot
Reply | Threaded
Open this post in threaded view
|

Re: tests sc_array_functions_test fail due to floating point precision

Hi Eike

On December 4, 2017 10:22 PM, Eike Rathke wrote:
> From what I understood by browsing shortly gcc enables
> this by default for -std=gnu* and it can be unset by
> #pragma STDC FP_CONTRACT OFF
> [...]
> Could you try if adding the pragma to include/sal/config.h
> helps?

I gave it a try but GCC didn't appreciate it and threw warnings back at me:

    include/sal/config.h:100:0: warning: ignoring #pragma STDC FP_CONTRACT [-Wunknown-pragmas]

I didn't let the compilation finish since in light of the
warning I expect the outcome to be the same as without pragma.
(If you think otherwise, let me know then I'll invest a couple
of hours into the compilation.)

The GCC version used was 6.4.0 but GCC 5 shows the same
behaviour. Sadly, I currently have no GCC 7 at my disposal to
test if that version supports the STDC FP_CONTRACT pragma (but
I don't think it does).

I took the liberty to google around a little. Here is what
I've found:

The GCC manual [0] (for version 6.4 but also for version 7.2)
states:

| Whether and how floating expressions are contracted when
| not disallowed by the FP_CONTRACT pragma (C99 and C11 6.5).
|
| Expressions are currently only contracted if
| -ffp-contract=fast, -funsafe-math-optimizations or
| -ffast-math are used. This is subject to change.

The C99 status page of GCC[1] reads:

| standard pragmas: Not implemented. Associated command-line
| options to control the state of the pragmas for the whole
| translation unit are available.

which hint to me that instead of the pragma we should use the
command line option -ffp-contract.

There is also GCC bug #37845 "gcc ignores FP_CONTRACT pragma
set to OFF". The last entry is:

| [...]
| Note: This bug was about FP contraction being done despite
| the use of:
| #pragma STDC FP_CONTRACT OFF
| not about the pragma being ignored (the implementation of
| this pragma is not required by the ISO C standard if the
| default is OFF, which is now the case with GCC). So, this
| bug is really fixed. A new enhancement bug could be opened
| (if not already done) for the implementation of the STDC
| FP_CONTRACT pragma.

So GCC does not recognize the pragma and its "default"
behaviour is "FP_CONTRACT=OFF". However, GCC enables
FP_CONTRACT as soon as -O2 (or higher) is enabled and the
target hardware supports it.

LLVM/clang seems to understand the pragma FP_CONTRACT, though
[3,4]

Best Milton


[0] https://gcc.gnu.org/onlinedocs/gcc-6.4.0/gcc/Floating-point-implementation.html
[1] https://gcc.gnu.org/c99status.html
[2] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=37845
[3] https://reviews.llvm.org/D24481
[4] https://github.com/llvm-mirror/clang/blob/master/test/Preprocessor/pragma_unknown.c
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
sberg sberg
Reply | Threaded
Open this post in threaded view
|

Re: tests sc_array_functions_test fail due to floating point precision

In reply to this post by Milton Vandersloot
On 12/02/2017 08:47 AM, Milton Vandersloot wrote:
> functions_test.cxx:43:Assertion
> Test name: ArrayFunctionsTest::testArrayFormulasFODS
> double equality assertion failed
> - Expected: 1
> - Actual  : 0
> - Delta   : 1e-14
>
>
> The issue seems to be (again) floating point precision. Indeed, I've compiled LibreOffice with FMA (fused multiply-add) instructions, if I disable them, i.e. passing -ffp-contract=off to GCC, then the tests succeed. Note that FMA has slightly different floating point precision than ordinary plus and mult [0, 1].

What is ultimately missing, I think, is a spec what the acceptable
results of calculations in Calc are, and tests written so that they
check whether or not the calculated results are in the acceptable ranges.
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Eike Rathke-2 Eike Rathke-2
Reply | Threaded
Open this post in threaded view
|

Re: tests sc_array_functions_test fail due to floating point precision

In reply to this post by Milton Vandersloot
Hi Milton,

On Saturday, 2017-12-02 02:47:03 -0500, Milton Vandersloot wrote:

> I've also opened the other test documents in sc/qa/unit/data/functions/array/fods/ using LibreOffice. The documents which fail are linest.fods and also logest.fods. The others succeed.

If your build completed ignoring these tests (i.e. produced a usable
office under ./instdir/), could you please open the failing documents
and tell us the cell addresses on the second sheet where tests fail and
the actual values they produced?

Thanks
  Eike

--
LibreOffice Calc developer. Number formatter stricken i18n transpositionizer.
GPG key 0x6A6CD5B765632D3A - 2265 D7F3 A7B0 95CC 3918  630B 6A6C D5B7 6563 2D3A
Care about Free Software, support the FSFE https://fsfe.org/support/?erack

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (849 bytes) Download Attachment
Milton Vandersloot Milton Vandersloot
Reply | Threaded
Open this post in threaded view
|

Re: tests sc_array_functions_test fail due to floating point precision

Hi Eike

On December 7, 2017 at 12:40 PM Eike Rathke wrote:
> On Saturday, 2017-12-02 02:47:03 -0500, Milton Vandersloot wrote:
> > I've also opened the other test documents in
> > sc/qa/unit/data/functions/array/fods/ using LibreOffice. The documents which
> > fail are linest.fods and also logest.fods. The others succeed.
>
>If your build completed ignoring these tests (i.e. produced a
>usable office under ./instdir/), could you please open the
>failing documents and tell us the cell addresses on the second
>sheet where tests fail and the actual values they produced?

Below are the numbers. Some remarks: Many errors are in the
range of 1E-11 to 1E-12 but some are way larger >1E30. I assume
this is due some division of a very small error.

Has someone done a/some theoretical error analysis of the
algorithms involved? (e.g. numeric (forward/backward) stability
and error estimates similar to what is well-known for LU
factorization and other algorithms.)

Who/What provided the reference values -- or reformulated --
what error do these numbers have (wrt to the mathematical
solution).

Best
Milton


Below are the first three columns (A-C). They are prefixed by the
line number (e.g. 'l21'). At the end I added the formula of the
TRUE/FALSE column. I included this as some tests round to
something different than 1E-12, e.g. line 5 in linest.fods.


logest.fods Sheet 2:
--------------------

l21: 538.89039831558 538.890398315583 FALSE (=ROUND(A21,12)=ROUND(B21,12))
off by 3E-12



linest.fods Sheet 2:
--------------------

l39: 12237.3616028624 12237.3616028623 FALSE (=ROUND(A39,12)=ROUND(B39,12))
off by 1E-10

l46: 1.15420554120705E+030 5.5523702096735E+030 FALSE (=ROUND(A46,12)=ROUND(B46,12))
off by 4.4E30

l66: 1.36570269395312E+030 3.73790456956461E+029 FALSE (=ROUND(A66,12)=ROUND(B66,12))
off by 1.E30

l99: 2029957.60481663 2029957.60481605 FALSE (=ROUND(A99,12)=ROUND(B99,12))
off by 6.E-7

l113: 1.11356775201247E+027 7.3824164159995E+026 FALSE (=ROUND(A113,12)=ROUND(B113,12))
off by 4E26

l156: 9942283.62964539 9942283.62965002 FALSE (=ROUND(A156,12)=ROUND(B156,12))
off by 5E-6
l157: 11577901.4177444 11577901.4177444 FALSE (=ROUND(A157,12)=ROUND(B157,12))
off by 0. This keeps me puzzling ... (increasing the number of significant digits yields zeros on both entries)

l163: -108.083609022555 -108.083609022579 FALSE (=ROUND(A163,12)=ROUND(B163,12))
off by 2.4E-11
l164: 34.8435504725302 34.8435504725221 FALSE (=ROUND(A164,12)=ROUND(B164,12))
off by 1.2E-11

l171: 2484916.49382756 2484916.49382742 FALSE (=ROUND(A171,12)=ROUND(B171,12))
off by 1.4E-6

l184: 2.7060018272122E-14 9.25081260404044E-13 FALSE (=ROUND(A184,12)=ROUND(B184,12))
off by 9E-13

l186: 4.04703891013103E+033 3.46285083263994E+030 FALSE (=ROUND(A186,12)=ROUND(B186,12))
off by 4E33

l189: 5.41123064912622E-14 1.84989825908131E-12 FALSE (=ROUND(A189,12)=ROUND(B189,12))
off by 1.8E-12

l201: 786024.499999976 786024.50000001 FALSE (=ROUND(A201,12)=ROUND(B201,12))
off by 4E-8

l216: 786024.499999976 786024.50000001 FALSE (=ROUND(A216,12)=ROUND(B216,12))
off by 4E-8

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Winfried Donkers-3 Winfried Donkers-3
Reply | Threaded
Open this post in threaded view
|

Re: tests sc_array_functions_test fail due to floating point precision

Hi Milton,


>>> I've also opened the other test documents in
>>> sc/qa/unit/data/functions/array/fods/ using LibreOffice. The documents which
>>> fail are linest.fods and also logest.fods. The others succeed.
>> If your build completed ignoring these tests (i.e. produced a
>> usable office under ./instdir/), could you please open the
>> failing documents and tell us the cell addresses on the second
>> sheet where tests fail and the actual values they produced?
> Below are the numbers. Some remarks: Many errors are in the
> range of 1E-11 to 1E-12 but some are way larger >1E30. I assume
> this is due some division of a very small error.
>
> Has someone done a/some theoretical error analysis of the
> algorithms involved? (e.g. numeric (forward/backward) stability
> and error estimates similar to what is well-known for LU
> factorization and other algorithms.)
>
> Who/What provided the reference values -- or reformulated --
> what error do these numbers have (wrt to the mathematical
> solution).
>
Looking at the numbers it would be better to use ROUNDSIG instead of ROUND.
ROUND(1.2345E30,4) has no effect, ROUNDSIG(1.2345E30,4) rounds to 4
significant digits, i.e. 1.234E30.
ROUNDSIG was only added to Calc last February, so most unit test
documents don't use it (yet).

Winfried

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice