some hints needed for fdo 42286 in calc

hello,

I am trying to debug fdo 42286 in calc. It has to do with autofilter,
where some empty lines (rows) are shown despite the filter criteria that
is set. (and a strange filter behaviour).

As entry point, I found sc/.. tools/queryparam.cxx & dbdata.cxx but it
seems that there all is Ok.

As I understand it, there the query is just "prepared" but I am not able
to find the code where the actual calculation for displaying it is done.
could someone hint me / give me a search criteria ?

Just an additional remark: "disassembling" the bug's attached file, I
see that the content.xml ends with some strange empty row definitions...
But this still does not justify to appear witin the filter in my
opinion. Or does this indicates the file is ill formatted ? (sorry, I do
not know the oasis spec)

<table:table-row table:style-name="ro1"
table:number-rows-repeated="1048532">
<table:table-cell table:number-columns-repeated="1024"/>
</table:table-row>

<table:table-row table:style-name="ro1"
table:visibility="filter"><table:table-cell
table:number-columns-repeated="1024"/>
</table:table-row>

<table:table-row table:style-name="ro1">
<table:table-cell table:number-columns-repeated="1024"/>
</table:table-row>

<table:table-row table:style-name="ro1"
table:visibility="filter"><table:table-cell
table:number-columns-repeated="1024"/>
</table:table-row>

<table:table-row table:style-name="ro1">
<table:table-cell table:number-columns-repeated="1024"/>
</table:table-row>

<table:table-row table:style-name="ro1" table:visibility="filter"
table:number-rows-repeated="5">
<table:table-cell table:number-columns-repeated="1024"/>
</table:table-row>

etc.... all of this kind

thanks for a hint
regards

Pierre-André
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Hello,

2011/11/1 Pierre-André Jacquod <[hidden email]>:
> hello,
>
> I am trying to debug fdo 42286 in calc. It has to do with autofilter, where
> some empty lines (rows) are shown despite the filter criteria that is set.
> (and a strange filter behaviour).
>
> As entry point, I found sc/.. tools/queryparam.cxx & dbdata.cxx but it seems
> that there all is Ok.

These are only the data structures holding the information for db data
and query information. The query is executed in ScDBDocFunc::Query and
related functions.

I hope that helps you a bit otherwise just ask again or ask in libreoffice-dev.

Regards,
Markus

P.S. Do you know if this is a regression against 3.3? Then it might be
much easier to search for the bug.
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

hello,

> and query information. The query is executed in ScDBDocFunc::Query and
> related functions.

thanks I will look there & come back if needed

> P.S. Do you know if this is a regression against 3.3? Then it might be

No 3.3 available to test (and a bit short of place on HD).

But happens on

LibreOffice 3.4.2
OOO340m1 (Build:1206)
(OpenSuse 11.4 distribution)

and on master, build by myself (also OpenSuse 3.4.2)

regards
Pierre-André
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Hi Pierre-André,

On Tuesday, 2011-11-01 19:18:08 +0100, Pierre-André Jacquod wrote:

> Just an additional remark: "disassembling" the bug's attached file,
> I see that the content.xml ends with some strange empty row
> definitions... But this still does not justify to appear witin the
> filter in my opinion. Or does this indicates the file is ill
> formatted ? (sorry, I do not know the oasis spec)

It is ill-formatted by the user ;-)  Cells in columns A:G down to the
very last row (with the exception of rows 1048571 and 1048572, maybe
more?) are formatted to have borders, for those a cell style has to be
applied hence the repeated rows.

However, strange are the spurious entries with table:visibility="filter"
in between that apparently cause the misbehavior, didn't investigate
further.

  Eike

--
LibreOffice Calc developer. Number formatter stricken i18n transpositionizer.
GnuPG key 0x293C05FD : 997A 4C60 CE41 0149 0DB3  9E96 2F1A D073 293C 05FD

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Hello,

Thanks for your hints. It took some times, but know I have clearly
identified the problem , but I do not if my solution is right or could
create more problems.

In short, the problem is that GetAreadData is called each time the
filter determines its values, instead of using the current area. This
does not have influence, as long as your selection area does not end
with empties rows or empties columns.

But this behaviour produce a wrong result as soon as your filtered area
ends with empties rows, and as special case, if you select all rows,
since it decreases the area of 1 row if the last row is empty.

Basically this happens with the call of function ExtendDataArea. This
originates from documen3.cxx, function GetFilterEntries line 1390 (the
call itself is at line 1398). (more details below, if needed)

My solution: suppress this call...
I have tested it and it seems to work. I did not managed to find a case
where something is broken with the filter..

But it seems me too simple.
Could you check this before I push it ?

If Ok, would you like then to approve it & cherry-pick it to 3.4.5 ?

Thanks & best regards
Pierre-André

ps: if what I did is right, I will then update the bug ticket

-------------------

What happens?
As I understand after debugging and following the logic, each time the
filter is activated (it is an auto-filter) it tries do determine the
data range when it search for its value to show in the filter drop-down.

For that it searches the biggest contiguous area of data, as when
applying the filter without selection.

Except the fact that here it goes wrong, since an area has already
selected (either manually or stored as <table:database-range ... tag.
which contains and finish (important in this analysis) with several
empty rows (columns would produce the same problem). And it tries to fit
the Area with the GetAreaData algorithm, starting with a predefined
area. The result: it does not extend it (next row is emtpy), but
thinking he as extended the area of one row to far (this the last row
int the area is empty), it reduce the selected area of one row.

And this produce the strange comportment described. (more detailed
analysis below).

Each time the filter value is changed , the function
ScTAble::GetAreaData (in  sc/source/core/data/table1.cxx, line 740 in my
check-out) is called.
At the end of the called function, there is a check: if the last row is
empty, then the Area is reduced by one, deacreasing its last row value
by 1 (line 835/836)
if (!bFound)
     --rEndRow;

This is obviously wrong, especially if you select e.g the 3 first
columns (completely) and then apply auto-filter. Since empty lines *are*
part of the selection... and not only the last one is empty. So each
time I change the filter value, I reduce by 1 the last row number...

>> I see that the content.xml ends with some strange empty row
> It is ill-formatted by the user ;-)  Cells in columns A:G down to the
....
> However, strange are the spurious entries with
> table:visibility="filter"

no, this is the result of this filter changes coupled with the order of
"ALL" value filter or partial filters.

This also creates a mess with the storage of
<table:database-range table:name="__Anonymous_Sheet_DB__0" ....
reducing the area within the range, depending when you save your file...




_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

On Tue, 2011-11-15 at 21:12 +0100, Pierre-André Jacquod wrote:

> My solution: suppress this call...
> I have tested it and it seems to work. I did not managed to find a case
> where something is broken with the filter..
>
> But it seems me too simple.

Yes it is too simple, isn't it?

The good news is that, you are in the right neighborhood.  The bad news
is that, you've just removed a valid functionality.  I'm afraid I can't
push your change, since the line you wish to remove is there for a
reason.

Also, your use case is a weird one.  Normally Calc users don't want to
filter out the bottom empty rows.  In fact, if you have an empty row it
actually acts as a data range separator, which affects what range is
used for so-called "database" operations; filtering, subtotal, sorting
etc.  That's an intended feature, not a bug, and supporting use cases
like yours will not be easy & require non-trivial code change and
perhaps several configuration options too (since the default behavior
can't possibly please all users).

Having said that, this auto range selection code for database ranges has
grown too complex & probably needs some cleanup.  But I'm afraid
removing that line will not be the way to go.  Sorry.

Kohei

--
Kohei Yoshida, LibreOffice hacker, Calc

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Hello,
thanks for the answer

>> My solution: suppress this call...
> is that, you've just removed a valid functionality.  I'm afraid I can't
> push your change, since the line you wish to remove is there for a
> reason.
which is ?? (not to argue, just to understand since I really tried to
find a sense at this call before writing the mail. And did not found any
and I have tested in different cases and did not noticed any side effect).

Ok for all what you say about auto-range I agree with it and I
understood from the code that this is the intention. BUT where I do not
see the rational :

this removed call i.e. the determination of the auto-range is called
only (and each time) the drop-down button of the filter is clicked, just
when building the drop-down list value. This path is NOT used on filter
activation. This path is used each time you change the value by which
the column is filtered, (in the drop down list) the auto-range is again
calculated and changes.

In other words, it means, that each time you change the filtered value,
the area which is under "filter control" changes: is this really the
intended design ?

As user, I do not think that this is intuitive and awaited. I would
await that the filter is working on the selected area and that changing
the value active for filtering does not alter its working area.

(If I repeat, that's not because I think you are dumb, that's because I
really do not know if I am able to express myself in an understandable
way in English :-(

OK, that said, where did I missed the steps and fall one floor down ?

> Also, your use case is a weird one.  Normally Calc users don't want to
> filter out the bottom empty rows.
ergh, just a point of view.... :- ) And no, I see a lot of people doing
this... and giving up. (btw I do not know the one who opened the ticket)

just a basic action (which is not weird):
cell A1: a, cell B1 b
cell A2 aa, cell B2 bb
select the whole sheet, activate standard filter
filter for value like aa in column a
.....

and to have funnier results just play changing the filtered value, not
forgiving to go regularly through the filter value (ALL)
cell A1: a, cell B1 b
cell A2 aa, cell B2 bb
select column A and B, active auto-filter (with header, that's nice)
filter for value aa in column A
filter for non empty in column B
filter for ALL in B
filter for ALL in A
filter for value aa in column A


I hope not to waste (too much of) your time. Just I am very rational and
like to understand, Especially when I have the impression that this is
against the (or my ?) logic.

Thanks
Best regards
Pierre-André

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice