Document Foundation Mail Archive › LibreOffice › Dev

test infrastructure ideas appreciated ...

Classic

List

Threaded

52 messages Options

Next » 123

Michael Meeks-5

test infrastructure ideas appreciated ...

Hi guys,

The ESC are interested in improving unit (and other automated) testing
and are interested in concrete ideas for implementing new automated
tests to prevent regressions. We hope to turn this into a proposal for
the Board, which if approved -could- turn into a tender to fund more
work in this area. Having said that writing unit tests is every
developers' responsibility where feasible - so this ask focuses on new
infrastructure.

It is noticeable that where there are existing tests & infrastructure,
new tests are easier to create and often there are more of them - so are
there places where we should work to create infrastructure ?

Is there something we can improve by throwing hardware at it ? as we
have done for the crash-testing; if so what hardware is needed ?

Is there a large area of code that is completely un-tested or
under-tested that you'd love us to invest in making test-able ?

Is there a test that currently only runs in an obscure setup /
corner-case that we can invest in standardizing and making more easy to
use ?

Constructive thoughts appreciated in reply here.

ATB,

Michael.

--
[hidden email] <><, Pseudo Engineer, itinerant idiot

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Robert Antoni

Re: test infrastructure ideas appreciated ...

We could add an automatic validation test for checking the discovery of dynamic library dependencies on OS X & Linux.

OS X: otool -L file
Linux: ldd file

2015-06-03 15:33 GMT+02:00 Michael Meeks <[hidden email]>:

Hi guys,

The ESC are interested in improving unit (and other automated) testing
and are interested in concrete ideas for implementing new automated
tests to prevent regressions. We hope to turn this into a proposal for
the Board, which if approved -could- turn into a tender to fund more
work in this area. Having said that writing unit tests is every
developers' responsibility where feasible - so this ask focuses on new
infrastructure.

It is noticeable that where there are existing tests & infrastructure,
new tests are easier to create and often there are more of them - so are
there places where we should work to create infrastructure ?

Is there something we can improve by throwing hardware at it ? as we
have done for the crash-testing; if so what hardware is needed ?

Is there a large area of code that is completely un-tested or
under-tested that you'd love us to invest in making test-able ?

Is there a test that currently only runs in an obscure setup /
corner-case that we can invest in standardizing and making more easy to
use ?

Constructive thoughts appreciated in reply here.

ATB,

Michael.

--
[hidden email] <><, Pseudo Engineer, itinerant idiot

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

http://about.me/rbuj

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Terrence Enger

Re: test infrastructure ideas appreciated ...

In reply to this post by Michael Meeks-5

Thanks to Michael Meeks for asking for suggestions for test
infrastructure.

I suggest a bunch of virtual partitions provisioned with all the
world's freely available database engines. This would enpower
interested testers to work on Base bugs without having to have the
database engine referenced in the bug report.

This idea has been floating around for years. The wiki page
"User:Drew/baseQA VM"
<https://wiki.documentfoundation.org/User:Drew/baseQA_VM> has not been
updated since 2011.

Terry.

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Noel Grandin-2

Re: test infrastructure ideas appreciated ...

In reply to this post by Michael Meeks-5

On 2015-06-03 03:33 PM, Michael Meeks wrote:
>
> The ESC are interested in improving unit (and other automated) testing
> and are interested in concrete ideas for implementing new automated
> tests to prevent regressions.

Hi

mjayfrancis(IRC) is doing some interesting automated-UI testing work using the UNO accessibility API - perhaps get him
on contract to finish it up and make it nice?

Perhaps a contract to convert the Java unit tests to C++?

Perhaps hand out some SSD's to our top QA people (and any other developers that need them, but I think most of them
already have one), should speed up their workflow a little.

Regards, Noel

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Michael Meeks-5

Re: test infrastructure ideas appreciated ...

In reply to this post by Robert Antoni

Hi Robert,

On Wed, 2015-06-03 at 15:44 +0200, Robert Antoni Buj i Gelonch wrote:
> We could add an automatic validation test for checking the discovery
> of dynamic library dependencies on OS X & Linux.
> * OS X: otool -L file
> * Linux: ldd file

Sounds interesting - the concern would be having added new
dependencies ? or what could we check with that ?

Thanks,

Michael.

--
[hidden email] <><, Pseudo Engineer, itinerant idiot

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Michael Meeks-5

Re: test infrastructure ideas appreciated ... - a11y based testing

In reply to this post by Noel Grandin-2

On Thu, 2015-06-04 at 16:14 +0200, Noel Grandin wrote:
> mjayfrancis(IRC) is doing some interesting automated-UI testing work using
> the UNO accessibility API - perhaps get him on contract to finish it up and make it nice?

Ah - it'd be great to use our UNO API directly, rather than via some
wrapper - then it can be effortlessly cross-platform ... will split your
ideas up - perhaps into threads ?

Thanks =)

Michael.

--
[hidden email] <><, Pseudo Engineer, itinerant idiot

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Michael Meeks-5

Re: test infrastructure ideas appreciated ... - junit ...

In reply to this post by Noel Grandin-2

On Thu, 2015-06-04 at 16:14 +0200, Noel Grandin wrote:
> Perhaps a contract to convert the Java unit tests to C++?

Might be interesting - I wonder if that can be partially automated in
some way ;->

At a minimum for JUnit it'd be good to further accelerate the tests by
processing the idle messages instead of sleeping I guess.

=)

Michael.

--
[hidden email] <><, Pseudo Engineer, itinerant idiot

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Michael Meeks-5

Re: test infrastructure ideas appreciated ... - Help / Doc screenshots ...

In reply to this post by Michael Meeks-5

This was mostly a Norbert idea from the ESC call :-)

* Find some way to automate the creation of (translated)
screenshots for help and documentation
+ this can be used to keep the help up-to-date
+ and also to test significant dialog open/close with it
+ and to add more screenshots (cheaply) to
the documentation

* Ideally would include some way of highlighting / annotating
certain bits of the UI, and intelligently cropping too.

I imagine it would need some form of annotation / markup / RDF magic on
images in documentation to allow those to be re-built easily.

=)

Michael.

--
[hidden email] <><, Pseudo Engineer, itinerant idiot

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Michael Meeks-5

Re: test infrastructure ideas appreciated ... - layout tests ...

In reply to this post by Michael Meeks-5

I'm no expert on the current state here, but my feeling is that there
are a lot of document / chart layout issues that are a real problem to
regression test.

I imagine that funding an investigation of the best of breed of
whatever we have, plus some combination of custom fonts,
built-in-everything (including freetype, harfbuz), no system hyphenation
data etc. etc. and basebmp rendering to ensure consistency across
platforms (& so the widest default reach) -might- get us to a place that
people can routinely run (at least on Linux) some layout tests (?) -
assuming we can find some way to reasonably describe workable and
maintainable constraints for that.

Thoughts much appreciated ;-)

Michael.

--
[hidden email] <><, Pseudo Engineer, itinerant idiot

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Matthew Francis

Re: test infrastructure ideas appreciated ... - a11y based testing

In reply to this post by Michael Meeks-5

On 05/06/2015 00:39, Michael Meeks wrote:
>
> On Thu, 2015-06-04 at 16:14 +0200, Noel Grandin wrote:
>> mjayfrancis(IRC) is doing some interesting automated-UI testing work using
>> the UNO accessibility API - perhaps get him on contract to finish it up and make it nice?
>
> Ah - it'd be great to use our UNO API directly, rather than via some
> wrapper - then it can be effortlessly cross-platform ... will split your
> ideas up - perhaps into threads ?

The framework I've got so far is based on a "UNO sandwich" - setting up
a test environment, and the final check of the results of actions are
done through (Py)UNO, while the manipulation of the UI itself (the
"filling") is done through Dogtail, a Red Hat project which is itself a
wrapper around the pyatspi accessibility library (and hence currently
Linux only)

Replacing the "filling" with more UNO could be done, but it would need
at least a replacement for the Dogtail convenience layer (which makes a
lot of the work of finding and interacting with the correct UI elements
very easy), and possibly some other work on the raw UNO accessibility
API. Ultimately this is an achievable volume of work, but it will take
some time to investigate, and I want to get as far as possible through
some other aspects of making UI testing easier before tackling it - much
of the rest of the approach/framework will still be applicable.

At present I'm on a side project to improve the usability of PyUNO by
making UNO collections (the XIndexAccess, XNameAccess,
XEnumerationAccess family) function as native Python lists, dicts and
iterators. Hopefully this too should help make it easier to write tests
for LO. As of today the basic functionality of this all works, and I
hope to be able to push it for review some time next week.

Regards
Matthew Francis

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Michael Stahl-2

Re: test infrastructure ideas appreciated ...

In reply to this post by Michael Meeks-5

On 04.06.2015 18:32, Michael Meeks wrote:

> Hi Robert,
>
> On Wed, 2015-06-03 at 15:44 +0200, Robert Antoni Buj i Gelonch wrote:
>> We could add an automatic validation test for checking the discovery
>> of dynamic library dependencies on OS X & Linux.
>> * OS X: otool -L file
>> * Linux: ldd file
>
> Sounds interesting - the concern would be having added new
> dependencies ? or what could we check with that ?

that's actually a good idea, as the recent bibisect Linux 5.0 accident
has demonstrated, see also commit
f4844a9abebcb0451161625c42a1e2b48796102d

we could have a test that runs something like readelf -d | grep
"(NEEDED)", filter out our own libraries, and then filters against a
whilelist of known-good system libraries; anything outside the whitelist
shouldn't be required.

it might even be useful on Mac OS X: while you do have a proper SDK
there and not a random collection of -devel packages, it might still be
possible that some bundled library's crazy configure script finds some
random stuff installed via Fink/MacPorts/etc., which would cause the
same issues.

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Robert Antoni

Re: test infrastructure ideas appreciated ...

$ curl -OL https://llvm.org/svn/llvm-project/openmp/trunk/runtime/tools/check-depends.pl

$ curl -OL https://llvm.org/svn/llvm-project/openmp/trunk/runtime/tools/lib/tools.pm

$ curl -OL https://llvm.org/svn/llvm-project/openmp/trunk/runtime/tools/lib/Platform.pm

$ curl -OL https://llvm.org/svn/llvm-project/openmp/trunk/runtime/tools/lib/Uname.pm

$ chmod +x check-depends.pl

$ ./check-depends.pl /Applications/LibreOfficeDev.app/Contents/Frameworks/libwpd-0.10.10.dylib

check-depends.pl: (x) Fail to parse otool output:

check-depends.pl: (x) /Applications/LibreOfficeDev.app/Contents/Frameworks/libwpd-0.10.10.dylib:

check-depends.pl: (x) >>> /usr/local/lib/libwpd-0.10.10.dylib (compatibility version 11.0.0, current version 11.0.0)

check-depends.pl: (x) /@.__________________________________________________OOO/lib/librevenge-0.0.0.dylib (compatibility version 1.0.0, current version 1.2.0)

check-depends.pl: (x) /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)

check-depends.pl: (x) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

check-depends.pl: (x) (eof)

2015-06-04 21:04 GMT+02:00 Michael Stahl <[hidden email]>:

On 04.06.2015 18:32, Michael Meeks wrote:
> Hi Robert,
>
> On Wed, 2015-06-03 at 15:44 +0200, Robert Antoni Buj i Gelonch wrote:
>> We could add an automatic validation test for checking the discovery
>> of dynamic library dependencies on OS X & Linux.
>> * OS X: otool -L file
>> * Linux: ldd file
>
> Sounds interesting - the concern would be having added new
> dependencies ? or what could we check with that ?

that's actually a good idea, as the recent bibisect Linux 5.0 accident
has demonstrated, see also commit
f4844a9abebcb0451161625c42a1e2b48796102d

we could have a test that runs something like readelf -d | grep
"(NEEDED)", filter out our own libraries, and then filters against a
whilelist of known-good system libraries; anything outside the whitelist
shouldn't be required.

it might even be useful on Mac OS X: while you do have a proper SDK
there and not a random collection of -devel packages, it might still be
possible that some bundled library's crazy configure script finds some
random stuff installed via Fink/MacPorts/etc., which would cause the
same issues.

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

http://about.me/rbuj

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Thorsten Behrens

Re: test infrastructure ideas appreciated ... - layout tests ...

In reply to this post by Michael Meeks-5

Michael Meeks wrote:
> I'm no expert on the current state here, but my feeling is that there
> are a lot of document / chart layout issues that are a real problem to
> regression test.
>
Curious to hear Moggi's input on that one - what I recall, was that
Chart layout is just too randomly unstable to make that work?

Or is that something where there is a plan how to fix that, just no
cycles/motivation to do so?

And tangentially - for anything doing layout checks, I'd of course
much prefer something that compares xml layout dumps, rather than
bitmaps. ;)

Cheers,

-- Thorsten

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (968 bytes) Download Attachment

Thorsten Behrens

Re: test infrastructure ideas appreciated ... - Help / Doc screenshots ...

In reply to this post by Michael Meeks-5

Michael Meeks wrote:
> * Find some way to automate the creation of (translated)
> screenshots for help and documentation
> + this can be used to keep the help up-to-date
> + and also to test significant dialog open/close with it
> + and to add more screenshots (cheaply) to
> the documentation
>
Nice - and we'd get a 'smoke-test all our dialogs' for free with
it. :)

Ideally that then has a way to register dialogs with some central
service inside LibreOffice, which appears Easy-Hackable to me
(instead of maintaining a set of out-of-tree scripts that click
through UI to make a particular dialog appear...).

Cheers,

-- Thorsten

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (968 bytes) Download Attachment

Markus Mohrhard

Re: test infrastructure ideas appreciated ... - layout tests ...

In reply to this post by Thorsten Behrens

On Fri, Jun 5, 2015 at 2:21 PM, Thorsten Behrens <[hidden email]> wrote:

Michael Meeks wrote:
> I'm no expert on the current state here, but my feeling is that there
> are a lot of document / chart layout issues that are a real problem to
> regression test.
>
Curious to hear Moggi's input on that one - what I recall, was that
Chart layout is just too randomly unstable to make that work?

At least in my opinion it is. Fonts are surely one problem but not the only one. The whole layouting in chart is based on dynamic layouting instead of manual layouting. So I have never been able to get it stable across more than my machines and gandalf. I suppose you can increase the number of systems that support it if you really know what you are doing. However that requires a lot of changes to how chart calculates its layout.

Or is that something where there is a plan how to fix that, just no
cycles/motivation to do so?

The layout tests work, just on a limited subset of machines. I looked into the failures when I implemented the layout tests. They are a combination of font issues, rounding errors, ...

And tangentially - for anything doing layout checks, I'd of course
much prefer something that compares xml layout dumps, rather than
bitmaps. ;)

Of course the current chart layout tests use the XShape xml dump. Actually I wrote the first version of the layout dump for chart and just reused it later for impress and draw ;)

Regards,

Markus

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Thorsten Behrens

Re: test infrastructure ideas appreciated ... - layout tests ...

Markus Mohrhard wrote:
> So I have never been able to get it stable across more than my
> machines and gandalf. I suppose you can increase the number of
> systems that support it if you really know what you are
> doing.
>
With the above - pretty much rules it out for me at this stage. I have
not so fond memories of having exactly one machine to login to & try
to make tests pass again...

Or are there plans to independently revamp the Chart layout?

Cheers,

-- Thorsten

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (968 bytes) Download Attachment

Thorsten Behrens

Re: test infrastructure ideas appreciated ... - format validity checks

In reply to this post by Michael Meeks-5

Hi,

another angle to tests - the various file format compliance checkers.

For example ODF:

- have an up-to-date rng file in our tree, that needs to have
extensions added the moment we start writing that out
- make --with-export-validation pickup that file instead
- have a host (e.g. the crashtest box) run that permanently & shout
when it breaks

Same applies to OOXML, having one box running the Open XML SDK there &
its validator would be awesome
(http://openxmldeveloper.org/blog/b/openxmldeveloper/archive/2010/03/08/8242.aspx)

Cheers,

-- Thorsten

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (968 bytes) Download Attachment

sberg

Re: test infrastructure ideas appreciated ...

In reply to this post by Robert Antoni

On 06/04/2015 09:12 PM, Robert Antoni Buj i Gelonch wrote:

> $ curl -OL
> https://llvm.org/svn/llvm-project/openmp/trunk/runtime/tools/check-depends.pl
> $ curl -OL
> https://llvm.org/svn/llvm-project/openmp/trunk/runtime/tools/lib/tools.pm
> $ curl -OL
> https://llvm.org/svn/llvm-project/openmp/trunk/runtime/tools/lib/Platform.pm
> $ curl -OL
> https://llvm.org/svn/llvm-project/openmp/trunk/runtime/tools/lib/Uname.pm
> $ chmod +x check-depends.pl <http://check-depends.pl>
> $ ./check-depends.pl <http://check-depends.pl>
> /Applications/LibreOfficeDev.app/Contents/Frameworks/libwpd-0.10.10.dylib
> check-depends.pl <http://check-depends.pl>: (x) Fail to parse otool output:
> check-depends.pl <http://check-depends.pl>: (x)
> /Applications/LibreOfficeDev.app/Contents/Frameworks/libwpd-0.10.10.dylib:
> check-depends.pl <http://check-depends.pl>: (x) >>>
> /usr/local/lib/libwpd-0.10.10.dylib (compatibility version 11.0.0,
> current version 11.0.0)
> check-depends.pl <http://check-depends.pl>: (x)
> /@.__________________________________________________OOO/lib/librevenge-0.0.0.dylib
> (compatibility version 1.0.0, current version 1.2.0)
> check-depends.pl <http://check-depends.pl>: (x) /usr/lib/libc++.1.dylib
> (compatibility version 1.0.0, current version 120.0.0)
> check-depends.pl <http://check-depends.pl>: (x)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
> 1213.0.0)
> check-depends.pl <http://check-depends.pl>: (x) (eof)

should be fixed now with
<http://cgit.freedesktop.org/libreoffice/core/commit/?h=libreoffice-5-0&id=670f78ee3c89434d965970f6220eff811bc8b7ab>
"Fix Mac OS X install names of external libwpd/libwpg"

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Michael Meeks-5

Re: test infrastructure ideas appreciated ... - format validity checks

In reply to this post by Thorsten Behrens

On Fri, 2015-06-05 at 18:59 +0200, Thorsten Behrens wrote:
> another angle to tests - the various file format compliance checkers.
> For example ODF:
...
> Same applies to OOXML, having one box running the Open XML SDK there &
> its validator would be awesome
>
(http://openxmldeveloper.org/blog/b/openxmldeveloper/archive/2010/03/08/8242.aspx)

What of this don't we cover with the crashtester validation ?

> - have an up-to-date rng file in our tree, that needs to have
> extensions added the moment we start writing that out

Is it that ? =)

Michael.

--
[hidden email] <><, Pseudo Engineer, itinerant idiot

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Michael Meeks-5

Re: test infrastructure ideas appreciated ... - layout tests ...

In reply to this post by Thorsten Behrens

On Fri, 2015-06-05 at 14:46 +0200, Thorsten Behrens wrote:
> > So I have never been able to get it stable across more than my
> > machines and gandalf. I suppose you can increase the number of
> > systems that support it if you really know what you are
> > doing.
>
> With the above - pretty much rules it out for me at this stage.

So - given that we can apply some funding to it - I'm fairly certain
that we can't test layout without fixing this problem, and if we fix it
- then lots of things become possible.

My suggestion here would be to stub or otherwise clobber the low-level
font API in VCL for this purpose - we used to ship various standard AFMs
- I don't see why we can't have a couple of those - and use their
metrics for every font requested from VCL and just stub the font
rendering for such tests.

That should / could also be made to work cross-platform too so we'd
capture and find any cross-platform layout issues as well.

It's not glorious & fun work, but I don't see a good way of testing
precise layout without something like that =) and if TDF can fund it -
all the better.

ATB,

Michael.

--
[hidden email] <><, Pseudo Engineer, itinerant idiot

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Next » 123