Brace for impact - gnumake4 to hit master

classic Classic list List threaded Threaded
13 messages Options
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|

Brace for impact - gnumake4 to hit master

Hi all,

feature/gnumake4 is about to be merged into master. As this changes a
lot of low-level build system stuff, there will be quite few temporary
instabilities and problems (esp. on non-Linux platforms) after the
merge.

Some highlights of the changes to the build system:
- gb_{Library,LinkTarget,Executable}_set_{defs,cflags,cxxflags} have
  been obsoleted. Use
  gb_{Library,LinkTarget,Executable}_add_{defs,cflags,cxxflags}_add_{defs,cflags,cxxflags}
  instead.
- Zip and Jar Targets
- Faster and simpler dep generation
- a mini gbuild-system "gbuild-simple.mk" for use in recursive
  CustomTargets
- The linking against system libs is abstracted in one file
  RepositoryExternal.mk making the module makefiles a lot more readable
  by having simple gb_Library_use_externals calls instead of errorprone
  LDFLAGS twiddling etc. There might be modules which need to be
  adapted still to this simpler scheme.

One nasty sideeffect of the new dep-generation is that if you declare a
cxx to compile for a library that is nonexistant it might result in
the build looping.(*)

Attached you find a list with the ~220 directories touched by this, if
there are changes at an area you feel at home, please have a curious
look at the changes.

I have reverted the gbuildization of gnumake4 in writerfilter and kept
"our" gbuildization. Still I think, we(**) should take a very close look
at that, as it seems to be doing some things quite right. I got the
gnumake4 gbuildization working, but I did not resync that state with the
latest changes in writerfilter. I created two temporary tags in filters:
-
http://cgit.freedesktop.org/libreoffice/filters/tag/?id=feature/gnumake4_writerfilter_base
-
http://cgit.freedesktop.org/libreoffice/filters/tag/?id=feature/gnumake4_writerfilter_head

which are the working gnumake4 gbuildization state (head) and "our"
master state it was based on (base). That diff (in writerfilter only) is
pushed as a featurebranch:
 http://cgit.freedesktop.org/libreoffice/filters/commit/?h=feature/gnumake4_writerfilter

which we can merge to master, if we choose to. Otherwise, we should
delete those.

If nobody protests, I will merge the branch to master on Sunday
afternoon, so that I wont break the master for volunteers on the
weekend, and so that we have good tinderbox result on Monday.

Best,

Bjoern

(*) EasyHack: Make gbuild bail out early if nonexistent cxx files are
declared.

(**) we = Miklos and David, maybe?

P.S.: Test status of the branch:
Smoketest passes.
In-build tests pass.
JunitTest/toolkit_unoapi fails, but does so on master too.
JunitTest/sw_complex fails, but does so on master too.
Junittest/dbaccess_complex seems to hang, but does seem to _NOT_ fail
on master. Worth a look.
Other subsequenttests pass.

--
https://launchpad.net/~bjoern-michaelsen

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

gnumake4_changed_dirs.txt (4K) Download Attachment
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|

Re: Brace for impact - gnumake4 to hit master

Hi all,

On Sat, 23 Jul 2011 04:10:37 +0200
Bjoern Michaelsen
<[hidden email]> wrote:

> Junittest/dbaccess_complex seems to hang, but does seem to _NOT_ fail
> on master. Worth a look.

I reverted back dbaccess to the gbuildization from master. However, the
dbacces_complex failure was not a regression introduced by the gnumake4
gbuildization: dbaccess_complex was missing on master (which is of
course a very sneaky, cheap way of not-failing). I added
dbacess_complex with:
 http://cgit.freedesktop.org/libreoffice/base/commit/?h=feature/gnumake4&id=e851f4185bcda511f33fb38e0aba9162252b7fce

and it fails against the master gbuildization for the test
complex.dbaccess.DatabaseDocument. I disabled that test with:
 http://cgit.freedesktop.org/libreoffice/base/diff/dbaccess/JunitTest_dbaccess_complex.mk?h=feature/gnumake4&id=e851f4185bcda511f33fb38e0aba9162252b7fce
for now as it is not a regression. If you know what dbaccess does or
should do, please revert that change and run:
 cd $SOLARSRC && \
     make -f dbaccess/Makefile subsequentcheck gb_JunitTest_HEADLESS=
and have a look.

Best,

Bjoern

--
https://launchpad.net/~bjoern-michaelsen


_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Miklos Vajna Miklos Vajna
Reply | Threaded
Open this post in threaded view
|

Re: Brace for impact - gnumake4 to hit master

In reply to this post by Bjoern Michaelsen
[ Sorry, replied off-list unintentionally. ]

On Sat, Jul 23, 2011 at 04:10:37AM +0200, Bjoern Michaelsen <[hidden email]> wrote:

> I have reverted the gbuildization of gnumake4 in writerfilter and kept
> "our" gbuildization. Still I think, we(**) should take a very close look
> at that, as it seems to be doing some things quite right. I got the
> gnumake4 gbuildization working, but I did not resync that state with the
> latest changes in writerfilter. I created two temporary tags in filters:
> -
> http://cgit.freedesktop.org/libreoffice/filters/tag/?id=feature/gnumake4_writerfilter_base
> -
> http://cgit.freedesktop.org/libreoffice/filters/tag/?id=feature/gnumake4_writerfilter_head
>
> which are the working gnumake4 gbuildization state (head) and "our"
> master state it was based on (base). That diff (in writerfilter only) is
> pushed as a featurebranch:
>  http://cgit.freedesktop.org/libreoffice/filters/commit/?h=feature/gnumake4_writerfilter
>
> which we can merge to master, if we choose to. Otherwise, we should
> delete those.
Hi Bjoern,

As we discussed on IRC in the long term it's a good idea to do what is
in gnumake4 (merge the various small libs to a single writerfilter lib);
but probably it makes more sense to do it once rtftok is not under heavy
development. Of course if you / David want to do it right now I can live
with that as well.

Thanks.

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

attachment0 (205 bytes) Download Attachment
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|

Re: Brace for impact - gnumake4 to hit master

On Sun, 24 Jul 2011 03:13:57 +0200
Miklos Vajna <[hidden email]> wrote:

> As we discussed on IRC in the long term it's a good idea to do what is
> in gnumake4 (merge the various small libs to a single writerfilter
> lib); but probably it makes more sense to do it once rtftok is not
> under heavy development. Of course if you / David want to do it right
> now I can live with that as well.

Right. I now merged the Cpp- and Junittests manually in the branch
too and also the source file generation per CustomTarget, but I kept the
separation of libs out of it for now, to make things not more confusing
as they need to be for merging. Joining the libs is the only change left
out from gnumake4 in writerfilter for now and should not be too much of
an issue, but can be done independently of the gnumake4 merge as this
also needs some annoying changes in scp2 etc. which are a pain to keep
on a separate branch.

Yes, doing that change with rtftok makes a lot of sense.

Best,

Bjoern

--
https://launchpad.net/~bjoern-michaelsen

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (501 bytes) Download Attachment
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|

[MERGED and PUSHED] Brace for impact - gnumake4 to hit master

In reply to this post by Bjoern Michaelsen
On Sat, 23 Jul 2011 04:10:37 +0200
Bjoern Michaelsen
<[hidden email]> wrote:


> If nobody protests, I will merge the branch to master on Sunday
> afternoon, so that I wont break the master for volunteers on the
> weekend, and so that we have good tinderbox result on Monday.

Done.

Best,

Bjoern


--
https://launchpad.net/~bjoern-michaelsen


_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Marc-André Laverdière Marc-André Laverdière
Reply | Threaded
Open this post in threaded view
|

Re: [MERGED and PUSHED] Brace for impact - gnumake4 to hit master

Monday 11AM IST:
./g pull -r; autogen.sh; make dev-install

Environment: Fedora 15 with Gnu Make 3.82

resulted in a successful build and I was able to open a document.
So no apparent breakage for the older build systems in place.

Marc-André Laverdière
Software Security Scientist
Innovation Labs, Tata Consultancy Services
Hyderabad, India

On 07/25/2011 12:23 AM, Bjoern Michaelsen wrote:

> On Sat, 23 Jul 2011 04:10:37 +0200
> Bjoern Michaelsen
> <[hidden email]> wrote:
>
>
>> If nobody protests, I will merge the branch to master on Sunday
>> afternoon, so that I wont break the master for volunteers on the
>> weekend, and so that we have good tinderbox result on Monday.
>
> Done.
>
> Best,
>
> Bjoern
>
>
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Francois Tigeot Francois Tigeot
Reply | Threaded
Open this post in threaded view
|

Re: [MERGED and PUSHED] Brace for impact - gnumake4 to hit master

In reply to this post by Bjoern Michaelsen
Hi,

On Sun, Jul 24, 2011 at 08:53:36PM +0200, Bjoern Michaelsen wrote:
> > If nobody protests, I will merge the branch to master on Sunday
> > afternoon, so that I wont break the master for volunteers on the
> > weekend, and so that we have good tinderbox result on Monday.
>
> Done.

A little breakage here (DragonFly/x86_64):

bootstrap/solenv/gbuild/Library.mk:52: *** gb_Deliver_deliver: file does not
exist in solver, and cannot be delivered:
/home/ftigeot/LibreOffice/bootstrap/solver/350/unxdfly.pro/lib/libcppunit.so.
Stop.

Libcppunit.so is only present in:
solver/350/unxdfly.pro/workdir/Headers/Library/libcppunit.so

--
Francois Tigeot
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|

Re: [MERGED and PUSHED] Brace for impact - gnumake4 to hit master

On Mon, 25 Jul 2011 08:25:24 +0200
Francois Tigeot <[hidden email]> wrote:

> bootstrap/solenv/gbuild/Library.mk:52: *** gb_Deliver_deliver: file
> does not exist in solver, and cannot be delivered:
> /home/ftigeot/LibreOffice/bootstrap/solver/350/unxdfly.pro/lib/libcppunit.so.
> Stop.

Well, there should be a symlink at:
solver/350/unxlngx6.pro/lib/libcppunit.so
created by:
 http://opengrok.libreoffice.org/xref/libs-extern/cppunit/prj/d.lst#26

> Libcppunit.so is only present in:
> solver/350/unxdfly.pro/workdir/Headers/Library/libcppunit.so

That is only a touch target in the new build system telling us that
it expects the headers for libcppunit.so in the solver and up-to-date.
Given that cppunit is still build by the old build system you also got
a message:
LinkTarget cppunit not defined: Assuming headers to be there!

Best,

Bjoern

--
https://launchpad.net/~bjoern-michaelsen


_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Caolán McNamara Caolán McNamara
Reply | Threaded
Open this post in threaded view
|

Re: sw, complex test, bookmarks

In reply to this post by Bjoern Michaelsen
On Sat, 2011-07-23 at 04:10 +0200, Bjoern Michaelsen wrote:
> JunitTest/sw_complex fails, but does so on master too.

I've not been able to carve out time to look at this failure (for way
too long), but last time I checked it was a bookmarks test which failed
for me. So if the words "bookmarks" in writer, make anyone go, "oh I did
something there", it might be good to have a look at the bookmarks
complex test to see if the test itself needs updating.

C.

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Michael Meeks Michael Meeks
Reply | Threaded
Open this post in threaded view
|

tree open for gnumake hacking again ...

In reply to this post by Bjoern Michaelsen
Hi Bjoern,

        First - this is great work :-) thanks for pulling this all together.

On Sat, 2011-07-23 at 04:10 +0200, Bjoern Michaelsen wrote:
> I have reverted the gbuildization of gnumake4 in writerfilter and kept
> "our" gbuildization.

        Great :-) always good to prefer our changes I guess.

        So - thanks again, for doing this, and - hopefully, this is something
we can continue on master.

        Matus - have you backed your work up as git format-patch output before
trying to get git to re-sync them to master ? :-) And of course, I'm
looking forward to your seeing work there: what module are you working
on currently ?

        ATB,

                Michael.

--
 [hidden email]  <><, Pseudo Engineer, itinerant idiot


_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|

Re: sw, complex test, bookmarks

In reply to this post by Caolán McNamara
Hi all,

On Mon, 25 Jul 2011 12:14:14 +0100
Caolán McNamara <[hidden email]> wrote:

> I've not been able to carve out time to look at this failure (for way
> too long), but last time I checked it was a bookmarks test which
> failed for me. So if the words "bookmarks" in writer, make anyone go,
> "oh I did something there", it might be good to have a look at the
> bookmarks complex test to see if the test itself needs updating.

Running this against a debug build I get a lot of:
sw/source/core/unocore/unobkm.cxx, Line 180:
<SwXBookmark::GetObject(..)>SwXBookmark requested for non-bookmark mark.
when executing getBookmarksHash in the test, which suggests something
at odd in general with the the mark container, as a generic mark is
cast to a bookmark, while it is not one.

Best,

Bjoern

P.S.:
As the original author of that bookmark test, here is a mea culpa:
The test is not really suited well for debugging. It creates a huge set
of bookmarks and then does a lot of pseudorandom delete/insert
operations on the document and compares a hash of the bookmark contents
with the one from a previous run. If the test fails, you just changed
the behavior of bookmarks changed. If one fixed a broken behavior, that
is of cause a Good Thing(tm), and the test needs to be updated. If
nobody intended to change bookmark behavior it is likely a Bad
Thing(tm). At least with onegit, we will soon be able to bisect to the
suspicious commit and make the decision, if that was intended or not
easier. ;)
Of course, in this case it doesnt really matter because getting the
hash should always work.

--
https://launchpad.net/~bjoern-michaelsen


_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Matúš Kukan Matúš Kukan
Reply | Threaded
Open this post in threaded view
|

Re: tree open for gnumake hacking again ...

In reply to this post by Michael Meeks
Hi Michael, Bjoern, all

On 25 July 2011 13:34, Michael Meeks <[hidden email]> wrote:
> Hi Bjoern,
>
>        First - this is great work :-) thanks for pulling this all together.
>
I agree, thanks.

>        Matus - have you backed your work up as git format-patch output before
> trying to get git to re-sync them to master ? :-) And of course, I'm
> looking forward to your seeing work there: what module are you working
> on currently ?
>
Well, there are more. Is it bad idea to push more than one at the same time?
But nothing is ready now. Almost in every module is something I can't deal with.
Some questions: (maybe not all necessary, I will then try to learn and
decide more myself)

First: I can use templates from solenv/gbuild/templates? Asking mainly
because of license part of file.

Do we need for each module or even library FOO_DLLPUBLIC? Or I can
just use SAL_DLLPUBLIC_EXPORT.

What to do with libraries?
Can I break things on other platforms and somebody else will fix that?
or should I really try to not.
In basctl there was: IF WNT SHL1STDLIBS+= $(SHELLLIB)
(SHELLLIB=-lgdi32 -lshell32 -ladvapi32 -lcomdlg32)
So I should add all of them (but I don't think all are really used) or
just nothing and somebody will add the right one ?

Can I also change names of libraries?
-        Name = SPECIAL_NAME(animcore);
+       Name = LIBNAME(animcore);
And because of things like
-SHL1IMPLIB= i$(TARGET) I should add something to RepositoryFixes?
And RepositoryFixes has section about WNT which has two parts but
there are common things mentioned twice. Maybe we could reorganize
that.

Is postprocess/rebase/coffbase.txt used? Do I need to change also that?

What to do with unused files or whole directories? remove or igore or
ask each time?

I've seen somewhere .IF "$(L10N_framework)"=="". Is the same variable
present in gbuild?

Last important: In scaddins there were idl files processed:
http://opengrok.libreoffice.org/xref/calc/scaddins/source/analysis/makefile.mk#125
Is it possible to add them into offapi or somewhere?
But I was able to use generated hpp files just with adding:
    -I$(realpath $(WORKDIR)/UnoApiHeaders/offapi) \
gb_Library_add_api,analysis,offapi was not enough, not sure why and
how it works.

Finally: Is there any preferred log message we want to use?

Thanks,
Matus

PS: What is the right place for this kind of questions? (to know in
the future, hopefully not /dev/null, I can try to ask less)
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|

Re: tree open for gnumake hacking again ...

Hi Matúš,
On Tue, 26 Jul 2011 11:18:05 +0200
Matúš Kukan <[hidden email]> wrote:

> First: I can use templates from solenv/gbuild/templates? Asking mainly
> because of license part of file.

IANAL, so I will refrain from answering that one (I assume you intend
to change the license).

> Do we need for each module or even library FOO_DLLPUBLIC? Or I can
> just use SAL_DLLPUBLIC_EXPORT.

Almost. Every library where the header is used to link against the
symbol from another Library. In those cases SAL_DLLPUBLIC_EXPORT would
work on *nix, but not on Windows. SAL_DLLPUBLIC_EXPORT is enough for
plain UNO-libs which just export the UNO factory functions.

see
http://webcache.googleusercontent.com/search?q=cache:1iVDdEXF5bcJ:blogs.oracle.com/GullFOSS/entry/why_some_compilers_suck_more
for details.


> What to do with libraries?
> Can I break things on other platforms and somebody else will fix that?
> or should I really try to not.

At least trying is a good idea ;)

> In basctl there was: IF WNT SHL1STDLIBS+= $(SHELLLIB)
> (SHELLLIB=-lgdi32 -lshell32 -ladvapi32 -lcomdlg32)
> So I should add all of them (but I don't think all are really used) or
> just nothing and somebody will add the right one ?

While it is good to remove superfluous linking, I dont think breaking
master for that is a good idea. In case of doubt, file an easyhack
listing the suspicious linking and work through those later (or let
somebody else pick that task up).

> Can I also change names of libraries?
> -        Name = SPECIAL_NAME(animcore);
> +       Name = LIBNAME(animcore);

Yes, as long as they are not part of the URE (for UNO-API stability).

> And because of things like
> -SHL1IMPLIB= i$(TARGET) I should add something to RepositoryFixes?

RepositoryFixes is for all the adjustments where libs are not following
the usual conventions. The idea was, that when we once go binary
incompatible (LibreOffice 4.0 API), we can make RepositoryFixes.mk
empty and have alot more consistent naming (modulo external libs, where
we have no say in the naming). Using RepositoryFixes right now is also
a good idea, as renaming a lib in two build systems is just needless
pain (migrating to gbuild and renaming only there is much easier).

> And RepositoryFixes has section about WNT which has two parts but
> there are common things mentioned twice. Maybe we could reorganize
> that.
Sure, go ahead! RepositoryFixes will always be messy, I am afraid
though. Its part of its nature. (It was once called the "bad bank" of
the new build system).
 
> What to do with unused files or whole directories? remove or igore or
> ask each time?

If it is plain dead code: remove. If it is test code with the hope to
be turned in a unittest: keep, maybe open a EasyHack/Bug about it.

> I've seen somewhere .IF "$(L10N_framework)"=="". Is the same variable
> present in gbuild?

No.

> Last important: In scaddins there were idl files processed:
> http://opengrok.libreoffice.org/xref/calc/scaddins/source/analysis/makefile.mk#125
> Is it possible to add them into offapi or somewhere?

Possible: sure, if there are no good reasons to keep the separation.

> But I was able to use generated hpp files just with adding:
>     -I$(realpath $(WORKDIR)/UnoApiHeaders/offapi) \
> gb_Library_add_api,analysis,offapi was not enough, not sure why and
> how it works.

gb_Library_add_api adds an include path to $(OUTDIR)/inc/$(api) and
depends on the Package $(api). So to keep the scaddins stuff separate
you would need to create a Package that generates the headers to
$(OUTDIR)/inc/scaddinsapi for example. See the offapi/UnoApi_offapi.mk
and offapi/Package_offapi_inc.mk files for reference.

Best,

Bjoern

--
https://launchpad.net/~bjoern-michaelsen
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice