What is bibisect? And what is it doing in my office?

Hi all,

bibisect stands for "binary bisect" and is intended to help QA for LibreOffice
3.5. Regressions are a most annoying artifact that unfortunately come with
software development and QA. However, regressions are a misfeature we want to
deal with quick and early as they might get harder and harder to triage and fix
as time passes.

Because the way git stores its stuff, this download:

 http://people.canonical.com/~bjoern/bibisect-3.5.lzma

contains:

 - 53 complete office installs between the creation of the core repo and the
   -3-5 branchoff (thats >5000 commits)
 - at 450MB each, that would be ~22GB total
 - however, it is only 749MB total download size, thats <15MB per installation

And one does not need to install them in parallel as one can switch through all
of them with a quick "git checkout source-hash-XXXXXX" -- one switch costs <1
second).

As developers are helped extraordinary by knowing as exact as possible when
some regression first showed up in the product there is some value in making
this task as easy and fast as possible. In source code there is the possibility
to bisect:

 http://book.git-scm.com/5_finding_issues_-_git_bisect.html

and with the core repository, we have -- in theory -- the ability to exactly
identify where the regression was introduced. In theory, because you need a
compile and install to triage the bug and different from most other projects,
this still takes quite some time for LibreOffice and thus we cannot fix
regressions as quick as we should.

This is were bibisect comes in: It contains fully completed LibreOffice
installations between two major releases(1) and you can bisect your regression
(to identify when the offending change happened). Once the range where the bug
was introduced is identified, developers will be much more eager to fix the
issue (as the scope can not only be guessed better, it is also known to be
quite limited now).

Bibisect also has the binary installs tagged with the commit-id from the source
repository -- which is the only thing identifying a build that really helps
developers. And by the order they are on the branch one can quickly see which
build is older and which is newer.

The script this was generated with(2) is here:

  http://cgit.freedesktop.org/libreoffice/contrib/dev-tools/tree/bibisect

and I sincerly wish it is worth the carbon footprint needed to generate the
output in the last days. ;)

So I hope with this, we make regression finding some more fun and allow us --
QA and developers together -- to stabilize this release more quickly that ever
before.

If there are questions about how bisecting works, I am pretty sure developers
will be happy to help out people interested to get started as this allows us to
distribute the work on more shoulders.

Opinions/Comments?

Best,

Bjoern

(1) Not quite for 3.4-3.5 as the stuff as is only works from the point in time,
where we merged repositories, but from the merge point (early August 2011) to
the point of 3.5 branchoff (early December).

(2) Almost: I forgot to disable-oolink, so I had to filter the branch to tweak
the links again. :-/
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

On Fri, 2011-12-09 at 14:59 +0100, Bjoern Michaelsen wrote:
That is fairly amazing :-)

C.

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

On 12/09/2011 02:59 PM, Bjoern Michaelsen wrote:
> (2) Almost: I forgot to disable-oolink, so I had to filter the branch to tweak
> the links again. :-/

Note that configure now has --disable-oolink.  In case you'll do this
again for 3.5--3.6.

Stephan
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Hi,

On Fri, Dec 09, 2011 at 03:12:53PM +0100, Stephan Bergmann wrote:
> On 12/09/2011 02:59 PM, Bjoern Michaelsen wrote:
> >(2) Almost: I forgot to disable-oolink, so I had to filter the branch to tweak
> >the links again. :-/
>
> Note that configure now has --disable-oolink.  In case you'll do
> this again for 3.5--3.6.

Already commited like that:
http://cgit.freedesktop.org/libreoffice/contrib/dev-tools/tree/bibisect/build.sh#n37

Best,

Bjoern
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Hi Bjoern,

On Friday, 2011-12-09 14:59:53 +0100, Bjoern Michaelsen wrote:

> bibisect stands for "binary bisect"

Way cool!

  Eike

--
LibreOffice Calc developer. Number formatter stricken i18n transpositionizer.
GnuPG key 0x293C05FD : 997A 4C60 CE41 0149 0DB3  9E96 2F1A D073 293C 05FD

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

On Fri, 2011-12-09 at 14:59 +0100, Bjoern Michaelsen wrote:
> bibisect stands for "binary bisect" and is intended to help QA for LibreOffice
> 3.5. Regressions are a most annoying artifact that unfortunately come with
> software development and QA.

        Wow - I assume this was the cryptic & magic item you wanted to discuss
at the TSC next week :-)

> If there are questions ...

        How did you get so awesome ? :-)

        A couple of points:

        + a quick re-send to the [hidden email]
          would be cool :-) so more QA guys get it.

        + are these binaries 'generic' ? ie. did you build them
          on an old enough distribution, with a clean enough config
          that they are usable on things other than
          Debian/Ubuntu-latest ? :-)

        Either way - it seems pretty cool to me. I wonder if we could find a
way to completely automate the production of this using our generic
linux tinderboxen ?

        ATB,

                Michael.

--
[hidden email]  <><, Pseudo Engineer, itinerant idiot

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Hi,

On Fri, Dec 09, 2011 at 02:47:47PM +0000, Michael Meeks wrote:
> Wow - I assume this was the cryptic & magic item you wanted to discuss
> at the TSC next week :-)

Yes. ;)

> + are these binaries 'generic' ? ie. did you build them
>  on an old enough distribution, with a clean enough config
>  that they are usable on things other than
>  Debian/Ubuntu-latest ? :-)

They are generic only in the sense that I didnt do a --with-system-foo, but
other than that, they have been compile on Ubuntu 11.10, so might not work on
older stuff. I was driven by the "Now is better than never." line from the Zen
of Python for this decision. ;)

> Either way - it seems pretty cool to me. I wonder if we could find a
> way to completely automate the production of this using our generic
> linux tinderboxen ?

Should be doable. There were quite a few of tweaks and turns in the beginning
(you want to get this stuff halfway right, when you are churning away 24 cores
nonstop for 3 days on your own electricity bill), but in the end it should
integrate well with the tinderboxes. However, I assume this will not work quite
as well incrementally as the binary repo was ~4GB before I purged and repacked
it with:

git purge && git repack -ad --window 2000 --depth 55 --funroll-loops --abuse-my-ram --yes-all-of-it

Which:
- takes quite a bit of time
- packs everything in one pack, which sucks for incremental downloads

So, it is a proof-of-concept prototype, but unless somebody finds out I packed
the same version 53 times or something like that, it should be a good starting
point.

Best,

Bjoern
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

On Fri, Dec 09, 2011 at 02:59:53PM +0100, Bjoern Michaelsen wrote:
> bibisect stands for "binary bisect" and is intended to help QA for LibreOffice
> 3.5.

I just elaborated a bit more about this on http://sweetshark.livejournal.com/7683.html

Best,

Bjoern
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

The bibisect is only possible for Bugs which are reproducible under Linux, if Linux has already been excluded (maybe bug history or comments will contain required information) bibisecting will not be possible. But of course, you can use bibisect to find out whether the bug is reproducible under Linux.

Hi.
What is needed to prepare a Windows version? There are Windows tinderboxes with daily builds of LO already (with new builds made twice a day even). Repacking them as ready to go Windows installation by using PortableApps technology would allow to do bibisecting on Windows without a hassle (if I understand correctly it is a manual run exe-check exe-mark source process). Am I missing something?
Best regards.

Hi,

On 2012-06-16 at 02:22 -0700, bfo wrote:

Not at all - it is even possible to generate a zipped installation, that
would be just unpacked, and checked into git; ie. nothing really hard to
do.  But nobody has done that yet - are you interested?

If yes: The best would be to modify the tinbuild2 to be able also to
commit the installation to a git repository, and push it to some
repository.  We wouldn't get the same granularity as a dedicated
bibisect build (with strict 64 commits interval), but still good enough,
considering that we usually have 40-100 commits a day.

The repository would have to be setup (on the server) to repack after
every push; but that is just a trivial hook.

The tinbuild2 is here:

http://cgit.freedesktop.org/libreoffice/contrib/buildbot/

You probably want to hack somewhere here:

http://cgit.freedesktop.org/libreoffice/contrib/buildbot/tree/bin/tinbuild_phases.sh#n85

(making it part of the do_push() function).

All the best,
Kendy

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Jan Holesovsky wrote

Hi,
On 2012-06-16 at 02:22 -0700, bfo wrote:
Not at all - it is even possible to generate a zipped installation, that
would be just unpacked, and checked into git; ie. nothing really hard to
do.  But nobody has done that yet - are you interested?
If yes: The best would be to modify the tinbuild2 to be able also to
commit the installation to a git repository, and push it to some
repository. [...]
The repository would have to be setup (on the server) to repack after
every push; but that is just a trivial hook. [..]
You probably want to hack somewhere here:
http://cgit.freedesktop.org/libreoffice/contrib/buildbot/tree/bin/tinbuild_phases.sh#n85
(making it part of the do_push() function).

Hi.
Thanks for the info, its good for search engines out there. Starred this thread already... If anyone can add other useful informations - please do comment.
Best regards.

Hi,
just to say thanks for bibisect
Impressive example with fdo49610 to find the problematic commit.

Regards
Pierre-André

On 12/09/2011 02:59 PM, Bjoern Michaelsen wrote:
> Hi all,

> So I hope with this, we make regression finding some more fun and allow us --
> QA and developers together -- to stabilize this release more quickly that ever
> before.
>
> Bjoern
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Hi,

On Thu, Jun 21, 2012 at 07:11:48PM +0200, Pierre-André Jacquod wrote:
> just to say thanks for bibisect
> Impressive example with fdo49610 to find the problematic commit.

Thanks for the feedback, and also kudos to Florian for actually doing the
bibisect ;)

(An this reminds me that I need to update bibisect again ;) )

Best,

Bjoern
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

On Mon, Jun 18, 2012 at 3:11 AM, Jan Holesovsky <[hidden email]> wrote:
>
> Not at all - it is even possible to generate a zipped installation, that
> would be just unpacked, and checked into git; ie. nothing really hard to
> do.  But nobody has done that yet - are you interested?

Has anyone else started to implement this feature?

> If yes: The best would be to modify the tinbuild2 to be able also to
> commit the installation to a git repository, and push it to some
> repository.  We wouldn't get the same granularity as a dedicated
> bibisect build (with strict 64 commits interval), but still good enough,
> considering that we usually have 40-100 commits a day.

It's certainly good enough for Windows, where we don't have any
bibisect repos, but it would still be great on GNU/Linux so that we'd
have up-to-date bibisect that can be applied to any regressions that
pop up during beta testing.

> The tinbuild2 is here:
>
> http://cgit.freedesktop.org/libreoffice/contrib/buildbot/
>
> You probably want to hack somewhere here:
>
> http://cgit.freedesktop.org/libreoffice/contrib/buildbot/tree/bin/tinbuild_phases.sh#n85
>
> (making it part of the do_push() function).

Hmm. The buildbots are spread out in different locations, right? It
seems like the simplest thing would be to push the zipped install up
to the remote (source) server with something fault-tolerant like
rsync, compare a sha1 to make sure it got transfered properly, then
unzip and commit the files to the bibisect repo for that buildbot.
That would eliminate the need for each buildbot to have a local copy
of the bibisect repo.

--R
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Hi,

On Fri, Jul 06, 2012 at 03:13:52PM -0400, Robinson Tryon wrote:
>
> Hmm. The buildbots are spread out in different locations, right? It
> seems like the simplest thing would be to push the zipped install up
> to the remote (source) server with something fault-tolerant like
> rsync, compare a sha1 to make sure it got transfered properly, then
> unzip and commit the files to the bibisect repo for that buildbot.
> That would eliminate the need for each buildbot to have a local copy
> of the bibisect repo.

If you commit to a local bibisect repo, and then push to a central one:
- you dont need sha1/rsync whatever because git itself makes sure your data
  gets transmitted correctly
- you will save bandwidth as the delta between too installs is much smaller
  than a full install. And git stores the zipped delta.

No need to reinvent the wheel: Using git for transmission and compreesion is
already likely a lot better than any custom tooling anyway.


Best,

Bjoern
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

On Sat, Jul 7, 2012 at 8:43 AM, Bjoern Michaelsen
<[hidden email]> wrote:
>
> If you commit to a local bibisect repo, and then push to a central one:
> - you dont need sha1/rsync whatever because git itself makes sure your data
>   gets transmitted correctly
> - you will save bandwidth as the delta between too installs is much smaller
>   than a full install. And git stores the zipped delta.
>
> No need to reinvent the wheel: Using git for transmission and compreesion is
> already likely a lot better than any custom tooling anyway.

Oh, okay, thanks. I've gotten burned in the past when trying to push a
lot of binary data around in git repos, because AFAIK git still can't
resume like rsync. I guess the deltas will be small enough that this
shouldn't be a problem. I'll see about making the code spam someone if
the git operations have a hiccup.

I'll set up a machine as a buildbot so that I can test my changes.
Anything I should know that's not in the README or the Tinderbox pages
on the wiki?

--R
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

On Sat, Jul 7, 2012 at 10:19 AM, Robinson Tryon
<[hidden email]> wrote:
>
> I'll set up a machine as a buildbot so that I can test my changes.
>

Here's my first shot at bibisect integration into tinbuild2. I threw
together a couple of test files to make it easier to test the bibisect
part without running a whole new build, I plundered Bjoern's bibisect
code to make the output more in line with the existing bibisect repos
(probably still needs improvements), and I added a number of tests
throughout the do_bibisect_push() function to ensure that all of the
steps of the tinbuild phase are executed properly.

I've tested these changes on Ubuntu 12.04, but I'd love to get
feedback from people running buildbots on other OSes.

Couple of notes:
The code currently adds every build to the bibisect repository, but
nightly builds might be often enough (or should we shoot for a "strict
64 commits interval" ?)

Git push to a local-filesystem remote (e.g. "fake-remote" made by
redo-repos.sh) is fast. It'll be interesting to see how slow push is
to a physically distant remote. Will there be a big repack hit as the
repository grows to include dozens of builds?

--R

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Hi Norbert,

On Sun, Jul 15, 2012 at 10:39:36AM -0400, Robinson Tryon wrote:
> Here's my first shot at bibisect integration into tinbuild2.

Could you look into this? I have never used tinbuild and have no box for
testing it -- still it would be awesome to have this. We could use this on this
one box we still have idling? IIRC Caolan suggested something like that.

Best,

Bjoern
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

On Sun, Jul 15, 2012 at 4:39 PM, Robinson Tryon
<[hidden email]> wrote:
> Here's my first shot at bibisect integration into tinbuild2.

Sorry, I've been afk for some time.. I'll be back soon.
I took a quick glance, and up front:

+ I'd like not to create a new step for that but to fold it in the 'push' step
+ I'd prefer if the global variable (and default) be step early
(before sourcing the .tinbuild/config/<profile>.cfg so that they can
easely be overridden there (notably the location of the 'artefact' git
repo
+ I suspect that a good part of the new bisect function should really
be on the per-platform source, since it is likely that that will be
quite different on Mac, Linux or Windows no ?

I'll take a deeper look in few days, and probably take care of the
first 2 items above, unless you beat me to it :-)

Norbert
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

On Tue, Jul 17, 2012 at 11:30 AM, Norbert Thiebaud <[hidden email]> wrote:
> On Sun, Jul 15, 2012 at 4:39 PM, Robinson Tryon
> <[hidden email]> wrote:
>> Here's my first shot at bibisect integration into tinbuild2.
>
> Sorry, I've been afk for some time.. I'll be back soon.
> I took a quick glance, and up front:
>
> + I'd like not to create a new step for that but to fold it in the 'push' step

For clarity I'll keep the bulk of the code factored-out into a
separate function, but I'll have that function called from within
do_push(). If it's called deep inside do_push() (perhaps just before
prepare_upload_manifest), then we'll only get nightlies in the
bibisect repository, but we can always tweak it to use a strict
interval in the future.

> + I'd prefer if the global variable (and default) be step early
> (before sourcing the .tinbuild/config/<profile>.cfg so that they can
> easely be overridden there (notably the location of the 'artefact' git
> repo

I'll put all of the global variables right at the top of tinbuild2 --
the PUSH_TO_BIBISECT_REPO is already there. (I copied bjoern's naming
conventions for the variables -- I'm not sure why it's called ARTIFACT
DIR, but I'm sure there's a good reason!)

> + I suspect that a good part of the new bisect function should really
> be on the per-platform source, since it is likely that that will be
> quite different on Mac, Linux or Windows no ?

I think that all of the git and file operations should run on most
*nix systems, including OSX. There might be a couple of tweaks to make
sure that we use the right cross-platform switches for 'find', etc..
but nothing major.

Windows is a more...challenging creature. Whenever I've had to support
git on windows I've done it through cygwin. I believe that the cygwin
shell should have all of the command-line utils necessary for these
scripts to run, and I assume that windows buildbots have to run
tinbuild2 through cygwin (or something else that provides a bash
shell), so I hope (knock on wood) that these scripts will run cleanly
without any special porting code.

> I'll take a deeper look in few days, and probably take care of the
> first 2 items above, unless you beat me to it :-)

It is very beautiful outside today, but thankfully I can take my
laptop out under the tree and poke at the code from there, so I should
have some new diffs for you tonight :-)

--R
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice