Quantcast

subsequenttests now run headless

classic Classic list List threaded Threaded
15 messages Options
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

subsequenttests now run headless

Hi all,

subsequenttests runs headless now by default and seem to be very stable
that way. These tests are very good to parallelize. On my machine I see:

> subsequenttests
[ __.oO SCK ] loaded modules: chart2 comphelper configmgr dbaccess
editeng forms framework linguistic ooo padmin qadevOOo sal sfx2 sot svl
svtools svx sw toolkit tools ucb unotools unoxml vbahelper xmloff

real 17m51.257s
user 3m6.720s
sys 0m18.650s

> subsequenttests -j
[ __.oO SCK ] loaded modules: chart2 comphelper configmgr dbaccess
editeng forms framework linguistic ooo padmin qadevOOo sal sfx2 sot svl
svtools svx sw toolkit tools ucb unotools unoxml vbahelper xmloff

real 5m33.555s
user 3m4.350s
sys 0m18.020s

In these five and a half minutes, you get a _lot_ of code coverage. If
you want to run the tests without headless, you do a:

> subsequenttests gb_JunitTest_HEADLESS=

and get all the blinkenlights. Be aware that by doing that you are not
only testing LibreOffice, but also stress testing your X11
implementation and WM (esp. when you enable parallelization). ;)

Best Regards,

Bjoern

--
https://launchpad.net/~bjoern-michaelsen


_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Miklos Vajna Miklos Vajna
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: subsequenttests now run headless

On Tue, Mar 29, 2011 at 10:22:43PM +0200, Bjoern Michaelsen <[hidden email]> wrote:
> subsequenttests runs headless now by default and seem to be very stable
> that way. These tests are very good to parallelize. On my machine I see:

Hi,

I'm sure this is a newbie question, but what is the relation between
unit tests, smoketest and subsequenttests?

If I'm not mistaken, the later two is integration test, while cppunit is
about lowlevel unit testing - does that mean that we want to get rid of
smoketest in the long run?

Thanks.

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

attachment0 (205 bytes) Download Attachment
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: subsequenttests now run headless

Hi Miklos,

lengthy answer to your innocent question ahead -- brace for impact.

On Tue, 29 Mar 2011 23:18:20 +0200
Miklos Vajna <[hidden email]> wrote:
 
> I'm sure this is a newbie question, but what is the relation between
> unit tests, smoketest and subsequenttests?

There is a whole lot of terms flying around wrt tests, which is rather
unfortunate.
There are:
  - unit tests (the real ones, somewhere in $MODULE/qa)
  - smoketest (in module smoketestoo_native)
  - subsequent tests including:
    - complex tests (mostly in $MODULE/qa/complex)
    - unoapi tests (in $MODULE/qa/unoapi and qadevOOo)
  - testtool tests

Unittests are run during the build. The smoketest tests the full
product so that it doesnt crash right upon start or while loading a
document. It requires a full installation for that of course.
Subsequent tests do various additional tests that require a full
installation. Complex tests are manually written "realworld scenarios"
that use a feature in the way intended on development. Unoapi tests
just do some very basic tests (setting a value, reading the value back,
checking if it is the same). They are no useful "realworld scenarios"
and highly synthetic. And finally there is the VclTesttool which
simulates mouseclicks and GUI events and hooks into the UI toolkit.

> If I'm not mistaken, the later two is integration test, while cppunit
> is about lowlevel unit testing - does that mean that we want to get
> rid of smoketest in the long run?

IMHO not at all. The list above is exactly in order of the
test usefulness (with real unittests being the most useful). However,
the real unittests are also the hardest to get: Some parts of code
require so much infrastructure (UNO, Config, GUI etc.) that you need to
do a lot of trickery to test it without a complete installation. That
is unfortunate because:

 - it is a lot of extra work.
 - the trickery might be fragile and hard to maintain.

The best solution would be to refactor your code to real small and
nifty units. But that is a lot of code change and for refactoring you
might want to have tests before you change (and break) everything. This
is were the complex tests come in IMHO:

You first write a "complex test" running real world scenarios on your
code area. It requires a full installation, but still it is better than
manually testing over and over again. Then you refactor mercilessly,
kill of useless dependencies and end up with a nifty small unit worth
the name "unit". Then you write a small unit test for the code that can
be run during the build, which should be easy (or at least easier) now.
You leave the complex test around because it doesnt hurt to have it (if
it fails, it might be worth a look).

The unoapi tests are still somewhat useful because of their code
coverage. But I dont think it is worth it to invest too much time into
writing new ones, when we can better write "real unittest" (and complex
tests -- to get real unittests).

As for the testtool tests -- I dont find them worth the effort. They
are very fragile (because they rely on timing and things like that) and
rarely are helpful in triaging bugs. There is one scenario I remember
them to be useful though: When migrating modules to the new build
system, they found missing resources sometimes. But that is a rather
special case and not really worth the long runtime and fragile results.

Running smoketest before pushing a nontrivial changeset is a good idea.

tl;dr: I dont think we should get rid of subsequenttests or smoketest
as of now as they are cheap to keep. But whenever it is possible to
write "real unit tests" that should be preferred.

Best Regards,

Bjoern

P.S.:
BTW not all subsequent tests are in Java, some are in C++ too. Thus also
not every cppunit test is run during the build currently.

P.P.S.:
Here is how I personally usually run the tests:
unit tests:     on every build (doh)
smoketest:      before nontrivial pushes along with manual testing
                after a major rebuild because of changes on master
                  (some branch integration for example)
complex tests:  on the module I code along with manual testing
                twice a week (before going to lunch)
                on releases (starting with 3.4.X)
unoapi tests:   as complex tests
testtool tests: not at all unless forced by some policy  

--
https://launchpad.net/~bjoern-michaelsen

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (501 bytes) Download Attachment
Michael Meeks Michael Meeks
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: subsequenttests now run headless


On Wed, 2011-03-30 at 01:22 +0200, Bjoern Michaelsen wrote:
> BTW not all subsequent tests are in Java, some are in C++ too. Thus also
> not every cppunit test is run during the build currently.

        Ooh - good point; do the java tests get skipped if we have Java
disabled ? [ a number of people build that way for speed / ease ].

        Otherwise - great write-up; though personally I'd encourage writing
many more unit tests run during the build, rather than subsequent-tests.
And of course converting as many of our existing C++ tests / workben
fragments to similar unit-tests :-)

        ATB,

                Michael.

--
 [hidden email]  <><, Pseudo Engineer, itinerant idiot


_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Caolán McNamara Caolán McNamara
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: subsequenttests now run headless

On Thu, 2011-03-31 at 10:50 +0100, Michael Meeks wrote:
> personally I'd encourage writing many more unit tests run during the
> build,

Currently unfortunately I think that all of our "large app" unit tests,
sw, sc, starmath and sd are now all disabled/not ported to new build
system yet, so we should try and get those back up and running.

C.

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: subsequenttests now run headless

In reply to this post by Michael Meeks
Hi Michael,

On Thu, 31 Mar 2011 10:50:05 +0100
Michael Meeks <[hidden email]>
wrote:

> Ooh - good point; do the java tests get skipped if we have
> Java disabled ? [ a number of people build that way for speed / ease
> ].

Not yet, but it would not be hard to implement. Although the subsequent
c++ tests are currently only in sal (which has no Java tests). So you
can pragmatically already run those with:

 make -sr subsequentcheck

there.

> Otherwise - great write-up; though personally I'd encourage
> writing many more unit tests run during the build, rather than
> subsequent-tests. And of course converting as many of our existing
> C++ tests / workben fragments to similar unit-tests :-)

That is not at all different from my position on unit tests at all
really. I just think we should not dogmatically refuse to use subsequent
tests where they are the pragmatic solution (because a real unit test
is hard or fragile). And the five minutes to run all subsequent tests
in headless mode now are not really any barrier at all.

Best Regards,

Bjoern

--
https://launchpad.net/~bjoern-michaelsen


_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Thorsten Behrens Thorsten Behrens
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: subsequenttests now run headless

In reply to this post by Bjoern Michaelsen
Bjoern Michaelsen wrote:
> subsequenttests runs headless now by default and seem to be very stable
> that way. These tests are very good to parallelize. On my machine I see:
>
Neat, let me try that on one of the tinderboxen - would then be in
favour of running it unconditionally.

Cheers,

-- Thorsten

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

attachment0 (205 bytes) Download Attachment
Caolán McNamara Caolán McNamara
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: subsequenttests now run headless

On Fri, 2011-04-01 at 12:18 +0200, Thorsten Behrens wrote:
> Bjoern Michaelsen wrote:
> > subsequenttests runs headless now by default and seem to be very stable
> > that way. These tests are very good to parallelize. On my machine I see:
> >
> Neat, let me try that on one of the tinderboxen - would then be in
> favour of running it unconditionally.

add running the subsequenttests to the default make check target ?

C.

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: subsequenttests now run headless

Hi Caolán, Hi Thorsten,

On Fri, 01 Apr 2011 11:52:13 +0100
Caolán McNamara <[hidden email]> wrote:

> On Fri, 2011-04-01 at 12:18 +0200, Thorsten Behrens wrote:
> > Bjoern Michaelsen wrote:
> > > subsequenttests runs headless now by default and seem to be very
> > > stable that way. These tests are very good to parallelize. On my
> > > machine I see:
> > >
> > Neat, let me try that on one of the tinderboxen - would then be in
> > favour of running it unconditionally.
>
> add running the subsequenttests to the default make check target ?
Since about a minute that is a good idea. The Junit tests are skipped
now if there is no Junit jar available (or if Java is disabled).

Best Regards,


Bjoern

--
https://launchpad.net/~bjoern-michaelsen

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (501 bytes) Download Attachment
Michael Meeks Michael Meeks
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: subsequenttests now run headless

In reply to this post by Caolán McNamara

On Thu, 2011-03-31 at 10:56 +0100, Caolán McNamara wrote:
> On Thu, 2011-03-31 at 10:50 +0100, Michael Meeks wrote:
> > personally I'd encourage writing many more unit tests run during the
> > build,
>
> Currently unfortunately I think that all of our "large app" unit tests,
> sw, sc, starmath and sd are now all disabled/not ported to new build
> system yet, so we should try and get those back up and running.

        Since I started on this, and inadvertently duplicated Norbert's work -
it is worth saying that Norbert is working on this for sc/ - which will
hopefully result in a nice cookie-cutter to re-use elsewhere.

        HTH,

                Michael.

--
 [hidden email]  <><, Pseudo Engineer, itinerant idiot


_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Caolán McNamara Caolán McNamara
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: subsequenttests now run headless

On Fri, 2011-04-01 at 12:59 +0100, Michael Meeks wrote:

> On Thu, 2011-03-31 at 10:56 +0100, Caolán McNamara wrote:
> > On Thu, 2011-03-31 at 10:50 +0100, Michael Meeks wrote:
> > > personally I'd encourage writing many more unit tests run during the
> > > build,
> >
> > Currently unfortunately I think that all of our "large app" unit tests,
> > sw, sc, starmath and sd are now all disabled/not ported to new build
> > system yet, so we should try and get those back up and running.
>
> Since I started on this, and inadvertently duplicated Norbert's work -
> it is worth saying that Norbert is working on this for sc/ - which will
> hopefully result in a nice cookie-cutter to re-use elsewhere.

What's the current status of this. I see that Norbert has got the sc
example to the stage of being able to build and launch which is great.
       
Looking at why the sc one fails afterwards we just really need to...
       
a) add a gbuild rule to make a services.rdb from the test_components
list in e.g. sc/qa/unit roughly the same way as the makefile.mk does it
b) launch cppunittester with the additional arguments listed in the
old-style makefile.mk as well
c) set the STAR_RESOURCEPATH variable appropriately
       
I have to admit the gmake files have defeated me on "a)", but it should
be fairly trivial if someone can give me a sample of correctly adding a
dependency to the cppunittester target to echo hello world to a file :-)

C.

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: subsequenttests now run headless

Hi Caolán,

On Thu, 07 Apr 2011 12:18:02 +0100
Caolán McNamara <[hidden email]> wrote:

> a) add a gbuild rule to make a services.rdb from the test_components
> list in e.g. sc/qa/unit roughly the same way as the makefile.mk does
> it
> I have to admit the gmake files have defeated me on "a)", but it
> should be fairly trivial if someone can give me a sample of correctly
> adding a dependency to the cppunittester target to echo hello world
> to a file :-)

Doing something "custom" unfortunately has still way too much
boilerplate in gbuild esp. for small tasks. You can find an example in
tools where:

in Library_tl:

 $(eval $(call gb_Library_add_package_headers,tl,tools_reversemap))

this creates a dependency on the tools_reversemap package.
in Package_reversemap:

 $(eval $(call
 gb_Package_add_customtarget,tools_reversemap,tools/source/reversemap))

says it does a recursive GNU make call in tools/source/reversemap.
The stuff it does itself is in tools/source/reversemap/Makefile which
will be started with the present work dir in

 $(WORKDIR)/CustomTarget/tools/source/reversemap/Makefile

and the dir of the makefile identifiable by:

MYDIR := $(realpath $(dir $(firstword $(MAKEFILE_LIST))))

in the recursive Makefile (the one at tools/source/reversemap/Makefile).

The stuff is part of a package (like header) because you might want to
"deliver" some of the products to the solver (not the case in the
tools example).

Now flame me.

Best Regards,

Bjoern


--
https://launchpad.net/~bjoern-michaelsen

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (501 bytes) Download Attachment
Caolán McNamara Caolán McNamara
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: subsequenttests now run headless

On Thu, 2011-04-07 at 14:06 +0200, Bjoern Michaelsen wrote:
> Doing something "custom" unfortunately has still way too much
> boilerplate in gbuild esp. for small tasks. You can find an example in
> tools where:

I see that you ported the sc one over, thanks for that :-)

C.

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: subsequenttests now run headless

Hi Caolán,

On Tue, 12 Apr 2011 15:34:24 +0100
Caolán McNamara <[hidden email]> wrote:

> I see that you ported the sc one over, thanks for that :-)

Yes, I also wanted to answer here, but could not find the thread
anymore ;)

Anyway: I also just cherrypicked that to 3-4 upon koheis request. I hope
I didnt break anything with that.

Best,

Bjoern

--
https://launchpad.net/~bjoern-michaelsen

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (501 bytes) Download Attachment
Caolán McNamara Caolán McNamara
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: subsequenttests now run headless

On Tue, 2011-04-12 at 18:37 +0200, Bjoern Michaelsen wrote:
> Anyway: I also just cherrypicked that to 3-4 upon koheis request.

After a bit of struggling today I see that under windows we need to

a) convert the /cygdrive/path/to/each/.o in the extraobjects back to
C:\path\to\each\.o so I hazarded a change as
1cd742e270cc60510232f86fbc7b52ad7efa2b8a in bootstrap to do that

b) sed out the program off BRAND_BASE under windows, i.e.
06920dced71fb5182c58349737fc9da5ca91a4ca in calc

these are in master, assuming a) is done the right way, then they
presumably also need cherry-picking back for the win builds.

C.

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Loading...