15 messages
Open this post in threaded view
|

Open this post in threaded view
|

## Re: subsequenttests now run headless

 On Tue, Mar 29, 2011 at 10:22:43PM +0200, Bjoern Michaelsen <[hidden email]> wrote: > subsequenttests runs headless now by default and seem to be very stable > that way. These tests are very good to parallelize. On my machine I see: Hi, I'm sure this is a newbie question, but what is the relation between unit tests, smoketest and subsequenttests? If I'm not mistaken, the later two is integration test, while cppunit is about lowlevel unit testing - does that mean that we want to get rid of smoketest in the long run? Thanks. _______________________________________________ LibreOffice mailing list [hidden email] http://lists.freedesktop.org/mailman/listinfo/libreoffice attachment0 (205 bytes) Download Attachment
Open this post in threaded view
|

 Hi Miklos, lengthy answer to your innocent question ahead -- brace for impact. On Tue, 29 Mar 2011 23:18:20 +0200 Miklos Vajna <[hidden email]> wrote:   > I'm sure this is a newbie question, but what is the relation between > unit tests, smoketest and subsequenttests? There is a whole lot of terms flying around wrt tests, which is rather unfortunate. There are:   - unit tests (the real ones, somewhere in $MODULE/qa) - smoketest (in module smoketestoo_native) - subsequent tests including: - complex tests (mostly in$MODULE/qa/complex)     - unoapi tests (in $MODULE/qa/unoapi and qadevOOo) - testtool tests Unittests are run during the build. The smoketest tests the full product so that it doesnt crash right upon start or while loading a document. It requires a full installation for that of course. Subsequent tests do various additional tests that require a full installation. Complex tests are manually written "realworld scenarios" that use a feature in the way intended on development. Unoapi tests just do some very basic tests (setting a value, reading the value back, checking if it is the same). They are no useful "realworld scenarios" and highly synthetic. And finally there is the VclTesttool which simulates mouseclicks and GUI events and hooks into the UI toolkit. > If I'm not mistaken, the later two is integration test, while cppunit > is about lowlevel unit testing - does that mean that we want to get > rid of smoketest in the long run? IMHO not at all. The list above is exactly in order of the test usefulness (with real unittests being the most useful). However, the real unittests are also the hardest to get: Some parts of code require so much infrastructure (UNO, Config, GUI etc.) that you need to do a lot of trickery to test it without a complete installation. That is unfortunate because: - it is a lot of extra work. - the trickery might be fragile and hard to maintain. The best solution would be to refactor your code to real small and nifty units. But that is a lot of code change and for refactoring you might want to have tests before you change (and break) everything. This is were the complex tests come in IMHO: You first write a "complex test" running real world scenarios on your code area. It requires a full installation, but still it is better than manually testing over and over again. Then you refactor mercilessly, kill of useless dependencies and end up with a nifty small unit worth the name "unit". Then you write a small unit test for the code that can be run during the build, which should be easy (or at least easier) now. You leave the complex test around because it doesnt hurt to have it (if it fails, it might be worth a look). The unoapi tests are still somewhat useful because of their code coverage. But I dont think it is worth it to invest too much time into writing new ones, when we can better write "real unittest" (and complex tests -- to get real unittests). As for the testtool tests -- I dont find them worth the effort. They are very fragile (because they rely on timing and things like that) and rarely are helpful in triaging bugs. There is one scenario I remember them to be useful though: When migrating modules to the new build system, they found missing resources sometimes. But that is a rather special case and not really worth the long runtime and fragile results. Running smoketest before pushing a nontrivial changeset is a good idea. tl;dr: I dont think we should get rid of subsequenttests or smoketest as of now as they are cheap to keep. But whenever it is possible to write "real unit tests" that should be preferred. Best Regards, Bjoern P.S.: BTW not all subsequent tests are in Java, some are in C++ too. Thus also not every cppunit test is run during the build currently. P.P.S.: Here is how I personally usually run the tests: unit tests: on every build (doh) smoketest: before nontrivial pushes along with manual testing after a major rebuild because of changes on master (some branch integration for example) complex tests: on the module I code along with manual testing twice a week (before going to lunch) on releases (starting with 3.4.X) unoapi tests: as complex tests testtool tests: not at all unless forced by some policy -- https://launchpad.net/~bjoern-michaelsen_______________________________________________ LibreOffice mailing list [hidden email] http://lists.freedesktop.org/mailman/listinfo/libreoffice signature.asc (501 bytes) Download Attachment Reply | Threaded Open this post in threaded view | ## Re: subsequenttests now run headless  On Wed, 2011-03-30 at 01:22 +0200, Bjoern Michaelsen wrote: > BTW not all subsequent tests are in Java, some are in C++ too. Thus also > not every cppunit test is run during the build currently. Ooh - good point; do the java tests get skipped if we have Java disabled ? [ a number of people build that way for speed / ease ]. Otherwise - great write-up; though personally I'd encourage writing many more unit tests run during the build, rather than subsequent-tests. And of course converting as many of our existing C++ tests / workben fragments to similar unit-tests :-) ATB, Michael. -- [hidden email] <><, Pseudo Engineer, itinerant idiot _______________________________________________ LibreOffice mailing list [hidden email] http://lists.freedesktop.org/mailman/listinfo/libreoffice Reply | Threaded Open this post in threaded view | ## Re: subsequenttests now run headless  On Thu, 2011-03-31 at 10:50 +0100, Michael Meeks wrote: > personally I'd encourage writing many more unit tests run during the > build, Currently unfortunately I think that all of our "large app" unit tests, sw, sc, starmath and sd are now all disabled/not ported to new build system yet, so we should try and get those back up and running. C. _______________________________________________ LibreOffice mailing list [hidden email] http://lists.freedesktop.org/mailman/listinfo/libreoffice Reply | Threaded Open this post in threaded view | ## Re: subsequenttests now run headless  In reply to this post by Michael Meeks Hi Michael, On Thu, 31 Mar 2011 10:50:05 +0100 Michael Meeks <[hidden email]> wrote: > Ooh - good point; do the java tests get skipped if we have > Java disabled ? [ a number of people build that way for speed / ease > ]. Not yet, but it would not be hard to implement. Although the subsequent c++ tests are currently only in sal (which has no Java tests). So you can pragmatically already run those with: make -sr subsequentcheck there. > Otherwise - great write-up; though personally I'd encourage > writing many more unit tests run during the build, rather than > subsequent-tests. And of course converting as many of our existing > C++ tests / workben fragments to similar unit-tests :-) That is not at all different from my position on unit tests at all really. I just think we should not dogmatically refuse to use subsequent tests where they are the pragmatic solution (because a real unit test is hard or fragile). And the five minutes to run all subsequent tests in headless mode now are not really any barrier at all. Best Regards, Bjoern -- https://launchpad.net/~bjoern-michaelsen_______________________________________________ LibreOffice mailing list [hidden email] http://lists.freedesktop.org/mailman/listinfo/libreoffice Reply | Threaded Open this post in threaded view | ## Re: subsequenttests now run headless  In reply to this post by Bjoern Michaelsen Bjoern Michaelsen wrote: > subsequenttests runs headless now by default and seem to be very stable > that way. These tests are very good to parallelize. On my machine I see: > Neat, let me try that on one of the tinderboxen - would then be in favour of running it unconditionally. Cheers, -- Thorsten _______________________________________________ LibreOffice mailing list [hidden email] http://lists.freedesktop.org/mailman/listinfo/libreoffice attachment0 (205 bytes) Download Attachment Reply | Threaded Open this post in threaded view | ## Re: subsequenttests now run headless  On Fri, 2011-04-01 at 12:18 +0200, Thorsten Behrens wrote: > Bjoern Michaelsen wrote: > > subsequenttests runs headless now by default and seem to be very stable > > that way. These tests are very good to parallelize. On my machine I see: > > > Neat, let me try that on one of the tinderboxen - would then be in > favour of running it unconditionally. add running the subsequenttests to the default make check target ? C. _______________________________________________ LibreOffice mailing list [hidden email] http://lists.freedesktop.org/mailman/listinfo/libreoffice Reply | Threaded Open this post in threaded view | ## Re: subsequenttests now run headless  Hi Caolán, Hi Thorsten, On Fri, 01 Apr 2011 11:52:13 +0100 Caolán McNamara <[hidden email]> wrote: > On Fri, 2011-04-01 at 12:18 +0200, Thorsten Behrens wrote: > > Bjoern Michaelsen wrote: > > > subsequenttests runs headless now by default and seem to be very > > > stable that way. These tests are very good to parallelize. On my > > > machine I see: > > > > > Neat, let me try that on one of the tinderboxen - would then be in > > favour of running it unconditionally. > > add running the subsequenttests to the default make check target ? Since about a minute that is a good idea. The Junit tests are skipped now if there is no Junit jar available (or if Java is disabled). Best Regards, Bjoern -- https://launchpad.net/~bjoern-michaelsen_______________________________________________ LibreOffice mailing list [hidden email] http://lists.freedesktop.org/mailman/listinfo/libreoffice signature.asc (501 bytes) Download Attachment Reply | Threaded Open this post in threaded view | ## Re: subsequenttests now run headless  In reply to this post by Caolán McNamara On Thu, 2011-03-31 at 10:56 +0100, Caolán McNamara wrote: > On Thu, 2011-03-31 at 10:50 +0100, Michael Meeks wrote: > > personally I'd encourage writing many more unit tests run during the > > build, > > Currently unfortunately I think that all of our "large app" unit tests, > sw, sc, starmath and sd are now all disabled/not ported to new build > system yet, so we should try and get those back up and running. Since I started on this, and inadvertently duplicated Norbert's work - it is worth saying that Norbert is working on this for sc/ - which will hopefully result in a nice cookie-cutter to re-use elsewhere. HTH, Michael. -- [hidden email] <><, Pseudo Engineer, itinerant idiot _______________________________________________ LibreOffice mailing list [hidden email] http://lists.freedesktop.org/mailman/listinfo/libreoffice Reply | Threaded Open this post in threaded view | ## Re: subsequenttests now run headless  On Fri, 2011-04-01 at 12:59 +0100, Michael Meeks wrote: > On Thu, 2011-03-31 at 10:56 +0100, Caolán McNamara wrote: > > On Thu, 2011-03-31 at 10:50 +0100, Michael Meeks wrote: > > > personally I'd encourage writing many more unit tests run during the > > > build, > > > > Currently unfortunately I think that all of our "large app" unit tests, > > sw, sc, starmath and sd are now all disabled/not ported to new build > > system yet, so we should try and get those back up and running. > > Since I started on this, and inadvertently duplicated Norbert's work - > it is worth saying that Norbert is working on this for sc/ - which will > hopefully result in a nice cookie-cutter to re-use elsewhere. What's the current status of this. I see that Norbert has got the sc example to the stage of being able to build and launch which is great. Looking at why the sc one fails afterwards we just really need to... a) add a gbuild rule to make a services.rdb from the test_components list in e.g. sc/qa/unit roughly the same way as the makefile.mk does it b) launch cppunittester with the additional arguments listed in the old-style makefile.mk as well c) set the STAR_RESOURCEPATH variable appropriately I have to admit the gmake files have defeated me on "a)", but it should be fairly trivial if someone can give me a sample of correctly adding a dependency to the cppunittester target to echo hello world to a file :-) C. _______________________________________________ LibreOffice mailing list [hidden email] http://lists.freedesktop.org/mailman/listinfo/libreoffice Reply | Threaded Open this post in threaded view | ## Re: subsequenttests now run headless  Hi Caolán, On Thu, 07 Apr 2011 12:18:02 +0100 Caolán McNamara <[hidden email]> wrote: > a) add a gbuild rule to make a services.rdb from the test_components > list in e.g. sc/qa/unit roughly the same way as the makefile.mk does > it > I have to admit the gmake files have defeated me on "a)", but it > should be fairly trivial if someone can give me a sample of correctly > adding a dependency to the cppunittester target to echo hello world > to a file :-) Doing something "custom" unfortunately has still way too much boilerplate in gbuild esp. for small tasks. You can find an example in tools where: in Library_tl:$(eval $(call gb_Library_add_package_headers,tl,tools_reversemap)) this creates a dependency on the tools_reversemap package. in Package_reversemap:$(eval $(call gb_Package_add_customtarget,tools_reversemap,tools/source/reversemap)) says it does a recursive GNU make call in tools/source/reversemap. The stuff it does itself is in tools/source/reversemap/Makefile which will be started with the present work dir in$(WORKDIR)/CustomTarget/tools/source/reversemap/Makefile and the dir of the makefile identifiable by: MYDIR := $(realpath$(dir $(firstword$(MAKEFILE_LIST)))) in the recursive Makefile (the one at tools/source/reversemap/Makefile). The stuff is part of a package (like header) because you might want to "deliver" some of the products to the solver (not the case in the tools example). Now flame me. Best Regards, Bjoern -- https://launchpad.net/~bjoern-michaelsen_______________________________________________ LibreOffice mailing list [hidden email] http://lists.freedesktop.org/mailman/listinfo/libreoffice signature.asc (501 bytes) Download Attachment
Open this post in threaded view
|

## Re: subsequenttests now run headless

 On Thu, 2011-04-07 at 14:06 +0200, Bjoern Michaelsen wrote: > Doing something "custom" unfortunately has still way too much > boilerplate in gbuild esp. for small tasks. You can find an example in > tools where: I see that you ported the sc one over, thanks for that :-) C. _______________________________________________ LibreOffice mailing list [hidden email] http://lists.freedesktop.org/mailman/listinfo/libreoffice