Package: aptitude
Version: 0.6.8.2-1
Severity: normal
Hi,
this may be the cause for the commenter's issues at
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=317841#10 but is likely
not the cause for the issue of the original reporter of #317841 as there
were no .gz in the file names he reported.
If I set Acquire::GzipIndexes to "true" in apt.conf, remove all files
from /var/lib/apt/lists/ and run "apt-get update" again, the downloaded
files don't get uncompressed but stay in compressed form on disk.
When I now start aptitude's TUI and move the selection up and down with
the cursor keys (or click on a package with the mouse), it takes several
hundred milliseconds until I see a reaction from aptitude if the
selected line contains a package. It does not happen if I select a
package group or if I don't use compressed package files. I suspect this
is because aptitude needs to get the short description from the package
list, because it's displayed in the status bar upon package selection.
So it seems that aptitude uncompresses the package list which contains
the selected package each time I select a package. An "strace -efile" on
selecting the package initscripts with and without Acquire::GzipIndexes
shows the following amounts of stat64() calls on the package lists:
$ sort /tmp/aptitude-one-down-stat.strace | awk '{print $1}' | uniq -c | sort -n
[...]
4 stat64("/var/lib/apt/lists/debian.ethz.ch_debian_dists_wheezy_non-free_binary-i386_Packages",
4 stat64("/var/lib/apt/lists/debian.ethz.ch_debian_dists_wheezy_non-free_i18n_Translation-en",
56 stat64("/var/lib/apt/lists/debian.ethz.ch_debian_dists_sid_main_binary-i386_Packages",
$ sort /tmp/aptitude-gzip-indexes-one-down-stat.strace | awk '{print $1}' | uniq -c | sort -n
[...]
4 stat64("/var/lib/apt/lists/debian.ethz.ch_debian_dists_wheezy_non-free_binary-i386_Packages.gz",
4 stat64("/var/lib/apt/lists/debian.ethz.ch_debian_dists_wheezy_non-free_i18n_Translation-en",
27 stat64("/var/lib/apt/lists/debian.ethz.ch_debian_dists_sid_main_binary-i386_Packages",
54 stat64("/var/lib/apt/lists/debian.ethz.ch_debian_dists_sid_main_binary-i386_Packages.gz",
So aptitude at least stat64()s several times on every package selection
if the uncompressed package lists.
I though don't expect that this causes the obvious speed difference
between with and without Acquire::GzipIndexes, so I suspect it's some
decompression which slows down selecting packages in the TUI.
So maybe it can cache the information gathered at the first time read or
read all the information into memory on startup (which would likely
raise memory consumption which would be not ideal either).
-- Package-specific info:
-- System Information:
Debian Release: 7.0
APT prefers unstable
APT policy: (990, 'unstable'), (600, 'stable')
Architecture: i386 (i686)
Kernel: Linux 3.7-trunk-686-pae (SMP w/1 CPU core)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages aptitude depends on:
ii aptitude-common 0.6.8.2-1
ii libapt-pkg4.12 0.9.7.7
ii libboost-iostreams1.49.0 1.49.0-3.1
ii libc6 2.13-38
ii libcwidget3 0.5.16-3.4
ii libept1.4.12 1.0.9
ii libgcc1 1:4.7.2-5
ii libncursesw5 5.9-10
ii libsigc++-2.0-0c2a 2.2.10-0.2
ii libsqlite3-0 3.7.15.1-1
ii libstdc++6 4.7.2-5
ii libtinfo5 5.9-10
ii libxapian22 1.2.12-2
ii zlib1g 1:1.2.7.dfsg-13
Versions of packages aptitude recommends:
pn apt-xapian-index <none>
pn aptitude-doc-en | aptitude-doc <none>
ii libparse-debianchangelog-perl 1.2.0-1
ii sensible-utils 0.0.7
Versions of packages aptitude suggests:
pn debtags <none>
pn tasksel <none>
-- no debconf information
Acknowledgement sent
to Daniel Hartwig <[email protected]>:
Extra info received and forwarded to list. Copy sent to Aptitude Development Team <[email protected]>.
(Wed, 09 Jan 2013 03:33:03 GMT) (full text, mbox, link).
Subject: Re: [Aptitude-devel] Bug#697724: aptitude: selecting packages in the
curses interface slow iff Acquire::GzipIndexes="true"
Date: Wed, 9 Jan 2013 11:31:13 +0800
On 9 January 2013 05:55, Axel Beckert <[email protected]> wrote:
> If I set Acquire::GzipIndexes to "true" in apt.conf, remove all files
> from /var/lib/apt/lists/ and run "apt-get update" again, the downloaded
> files don't get uncompressed but stay in compressed form on disk.
These indices (Packages files mainly) are part of APTs database and
contain much information that is not present in the binary cache. While
apt-get performs fine with this setting, aptitude frequently accesses
package fields that are not in the binary cache and wants for
uncompressed access to the indices.
> I though don't expect that this causes the obvious speed difference
> between with and without Acquire::GzipIndexes, so I suspect it's some
> decompression which slows down selecting packages in the TUI.
>
> So maybe it can cache the information gathered at the first time read or
> read all the information into memory on startup (which would likely
> raise memory consumption which would be not ideal either).
I believe that synaptic does something like this as part of converting
the data to a form suitable for GTK+ to use. However, implementing
such caching above the APT level is a major burden from a maintenance
perspective. Better to do this in APT, by setting the aforementioned
option to “false”.
Given the ease of configuring this setting appropriately I doubt
anyone will be interested in implementing “on-demand” caching of
the uncompressed data. This setting is really for situations where
programs such as aptitude are not used, and the extra MBs of disk
space are important.
Regards
Acknowledgement sent
to Axel Beckert <[email protected]>:
Extra info received and forwarded to list. Copy sent to Aptitude Development Team <[email protected]>.
(Wed, 09 Jan 2013 03:57:03 GMT) (full text, mbox, link).
Subject: Re: [Aptitude-devel] Bug#697724: aptitude: selecting packages in
the curses interface slow iff Acquire::GzipIndexes="true"
Date: Wed, 9 Jan 2013 04:55:14 +0100
Hi Daniel,
Daniel Hartwig wrote:
> On 9 January 2013 05:55, Axel Beckert <[email protected]> wrote:
> > If I set Acquire::GzipIndexes to "true" in apt.conf, remove all files
> > from /var/lib/apt/lists/ and run "apt-get update" again, the downloaded
> > files don't get uncompressed but stay in compressed form on disk.
>
> These indices (Packages files mainly) are part of APTs database and
> contain much information that is not present in the binary cache. While
> apt-get performs fine with this setting, aptitude frequently accesses
> package fields that are not in the binary cache and wants for
> uncompressed access to the indices.
I see.
> > I though don't expect that this causes the obvious speed difference
> > between with and without Acquire::GzipIndexes, so I suspect it's some
> > decompression which slows down selecting packages in the TUI.
> >
> > So maybe it can cache the information gathered at the first time read or
> > read all the information into memory on startup (which would likely
> > raise memory consumption which would be not ideal either).
>
> I believe that synaptic does something like this as part of converting
> the data to a form suitable for GTK+ to use. However, implementing
> such caching above the APT level is a major burden from a maintenance
> perspective.
Indeed, also because other tools have way worse issues with that
setting, e.g. http://bugs.debian.org/617856
> Better to do this in APT, [...]
*nod* Reassigning, affects aptitude and downgrading to wishlist?
> Given the ease of configuring this setting appropriately I doubt
> anyone will be interested in implementing “on-demand” caching of
> the uncompressed data.
*nod*
> This setting is really for situations where programs such as
> aptitude are not used, and the extra MBs of disk space are
> important.
Here I disagree. These things are not mutually exclusive. If there
wasn't #617856, I may have reported this much earlier (*), because on
my EeePC with just 4 GB of hard-wired disk space, I'd be very happy
about that 87 MB more disk space I'd have if I could use that feature.
There are also a few boxes which run on CF cards with 512 MB to 2 GB
where I'm happy about every MB I can spare without compensating in
features.
(*) I mostly reported it because
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=317841#10 looks a lot
like this and I didn't want to close #317841 (which was definitely a
different issue) without giving that commenter's issue a proper bug
report.
Regards, Axel
--
,''`. | Axel Beckert <[email protected]>, http://people.debian.org/~abe/
: :' : | Debian Developer, ftp.ch.debian.org Admin
`. `' | 1024D: F067 EA27 26B9 C3FC 1486 202E C09E 1D89 9593 0EDE
`- | 4096R: 2517 B724 C5F6 CA99 5329 6E61 2FF9 CD59 6126 16B5
Acknowledgement sent
to Daniel Hartwig <[email protected]>:
Extra info received and forwarded to list. Copy sent to Aptitude Development Team <[email protected]>.
(Wed, 09 Jan 2013 07:57:07 GMT) (full text, mbox, link).
Subject: Re: Bug#697724: Bug#697724: aptitude: selecting packages in the
curses interface slow iff Acquire::GzipIndexes="true"
Date: Wed, 9 Jan 2013 15:54:51 +0800
On 9 January 2013 11:55, Axel Beckert <[email protected]> wrote:
>> > So maybe it can cache the information gathered at the first time read or
>> > read all the information into memory on startup (which would likely
>> > raise memory consumption which would be not ideal either).
>>
>> I believe that synaptic does something like this as part of converting
>> the data to a form suitable for GTK+ to use. However, implementing
>> such caching above the APT level is a major burden from a maintenance
>> perspective.
>
> Indeed, also because other tools have way worse issues with that
> setting, e.g. http://bugs.debian.org/617856
That program is parsing Packages directly, bypassing the libapt
interface. Not much that APT could do to facilitate that.
>
>> Better to do this in APT, [...]
>
> *nod* Reassigning, affects aptitude and downgrading to wishlist?
Best place for it.
Regards
Acknowledgement sent
to "Manuel A. Fernandez Montecelo" <[email protected]>:
Extra info received and forwarded to list. Copy sent to Aptitude Development Team <[email protected]>.
(Fri, 15 Jan 2016 00:45:03 GMT) (full text, mbox, link).
Subject: Re: aptitude: selecting packages in the curses interface slow iff
Acquire::GzipIndexes="true"
Date: Fri, 15 Jan 2016 00:41:41 +0000
Control: tags -1 + moreinfo
Hi Axel,
2013-01-08 21:55 Axel Beckert:
>Package: aptitude
>Version: 0.6.8.2-1
>Severity: normal
>
>Hi,
>
>this may be the cause for the commenter's issues at
>http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=317841#10 but is likely
>not the cause for the issue of the original reporter of #317841 as there
>were no .gz in the file names he reported.
>
>If I set Acquire::GzipIndexes to "true" in apt.conf, remove all files
>from /var/lib/apt/lists/ and run "apt-get update" again, the downloaded
>files don't get uncompressed but stay in compressed form on disk.
>
>When I now start aptitude's TUI and move the selection up and down with
>the cursor keys (or click on a package with the mouse), it takes several
>hundred milliseconds until I see a reaction from aptitude if the
>selected line contains a package. It does not happen if I select a
>package group or if I don't use compressed package files. I suspect this
>is because aptitude needs to get the short description from the package
>list, because it's displayed in the status bar upon package selection.
>
>So it seems that aptitude uncompresses the package list which contains
>the selected package each time I select a package. An "strace -efile" on
>selecting the package initscripts with and without Acquire::GzipIndexes
>shows the following amounts of stat64() calls on the package lists:
Did you try this with apt-1.1 or 1.2?
Since you experience the original issue you are in a better position to
see if things improved since then.
Apart from other speed improvements, I think that the number of fields
cached are now more, and we made some changes in aptitude to use better
ways to access those.
Cheers.
--
Manuel A. Fernandez Montecelo <[email protected]>