Debian Bug report logs - #688303
Please support concatenated bz2 files in zlib-functions

Reply or subscribe to this bug.

Display info messages

Message #5 received at [email protected] (full text, mbox, reply):

From: Thijs Kinkhorst <[email protected]>

To: Debian Bug Tracking System <[email protected]>

Subject: fails on concatenated bzip2 compressed Packages files (regression)

Date: Fri, 21 Sep 2012 11:28:30 +0200

Package: apt
Version: 0.9.7.4
Severity: important

Hi,

Using apt from wheezy with our APT repository fails:
201 URI Done: bzip2:/var/lib/apt/lists/partial/non-free.uvt.nl_debian_dists_squeeze_uvt_binary-amd64_Packages
RecivedHash: SHA256:99efd14ebb9fe582cc986f50502540030299595badefad71f5819e78c86a0c8d
ExpectedHash: SHA256:56ea2d1eb412a7121ad50f96e389c587e870ed7e80e893de0a66b3dbd9793015
[...]
W: Failed to fetch bzip2:/var/lib/apt/lists/partial/non-free.uvt.nl_debian_dists_squeeze_uvt_binary-amd64_Packages  Hash Sum mismatch

The problem is that our APT archive contains a Packages.bz2 which is a
concatenation of other bz2 files (of sub-archives). Two concatenated
bzip2 files again form a valid bzip2 file. The b(un)zip2 command line
utils support this and it's a documented feature. APT only decompresses
the first bzip2 file in Packages.bz2 leading indeed to a different
hash sum than expected.

This is a regression from squeeze, because apt in squeeze used the
command line utils which do this correctly. In 0.9, apt switched to
using libbz2, however, it only calls the function that decompresses
one bzip2 stream. The bzip2 command line utils handle the looping over
input themselves with the appropriate library calls (search for
bzReadGetUnused).


Cheers,
Thijs

-- Package-specific info:

-- apt-config dump --

APT "";
APT::Architecture "amd64";
APT::Build-Essential "";
APT::Build-Essential:: "build-essential";
APT::Install-Recommends "false";
APT::Install-Suggests "0";
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image.*";
APT::NeverAutoRemove:: "^kfreebsd-image.*";
APT::NeverAutoRemove:: "^linux-restricted-modules.*";
APT::NeverAutoRemove:: "^linux-ubuntu-modules-.*";
APT::NeverAutoRemove:: "^gnumach$";
APT::NeverAutoRemove:: "^gnumach-image.*";
APT::Never-MarkAuto-Sections "";
APT::Never-MarkAuto-Sections:: "metapackages";
APT::Never-MarkAuto-Sections:: "restricted/metapackages";
APT::Never-MarkAuto-Sections:: "universe/metapackages";
APT::Never-MarkAuto-Sections:: "multiverse/metapackages";
APT::Never-MarkAuto-Sections:: "oldlibs";
APT::Never-MarkAuto-Sections:: "restricted/oldlibs";
APT::Never-MarkAuto-Sections:: "universe/oldlibs";
APT::Never-MarkAuto-Sections:: "multiverse/oldlibs";
APT::Periodic "";
APT::Periodic::AutocleanInterval "14";
APT::Cache-Limit "134217728";
APT::Architectures "";
APT::Architectures:: "amd64";
APT::Compressor "";
APT::Compressor::. "";
APT::Compressor::.::Name ".";
APT::Compressor::.::Extension "";
APT::Compressor::.::Binary "";
APT::Compressor::.::Cost "1";
APT::Compressor::gzip "";
APT::Compressor::gzip::Name "gzip";
APT::Compressor::gzip::Extension ".gz";
APT::Compressor::gzip::Binary "gzip";
APT::Compressor::gzip::Cost "2";
APT::Compressor::gzip::CompressArg "";
APT::Compressor::gzip::CompressArg:: "-9n";
APT::Compressor::gzip::UncompressArg "";
APT::Compressor::gzip::UncompressArg:: "-d";
APT::Compressor::bzip2 "";
APT::Compressor::bzip2::Name "bzip2";
APT::Compressor::bzip2::Extension ".bz2";
APT::Compressor::bzip2::Binary "bzip2";
APT::Compressor::bzip2::Cost "3";
APT::Compressor::bzip2::CompressArg "";
APT::Compressor::bzip2::CompressArg:: "-9";
APT::Compressor::bzip2::UncompressArg "";
APT::Compressor::bzip2::UncompressArg:: "-d";
APT::Compressor::xz "";
APT::Compressor::xz::Name "xz";
APT::Compressor::xz::Extension ".xz";
APT::Compressor::xz::Binary "xz";
APT::Compressor::xz::Cost "4";
APT::Compressor::xz::CompressArg "";
APT::Compressor::xz::CompressArg:: "-6";
APT::Compressor::xz::UncompressArg "";
APT::Compressor::xz::UncompressArg:: "-d";
APT::Compressor::lzma "";
APT::Compressor::lzma::Name "lzma";
APT::Compressor::lzma::Extension ".lzma";
APT::Compressor::lzma::Binary "xz";
APT::Compressor::lzma::Cost "5";
APT::Compressor::lzma::CompressArg "";
APT::Compressor::lzma::CompressArg:: "--format=lzma";
APT::Compressor::lzma::CompressArg:: "-9";
APT::Compressor::lzma::UncompressArg "";
APT::Compressor::lzma::UncompressArg:: "--format=lzma";
APT::Compressor::lzma::UncompressArg:: "-d";
APT::Compressor::::Name "";
APT::Compressor::::Extension ".";
APT::Compressor::::Binary "";
APT::Compressor::::Cost "100";
APT::Compressor::::CompressArg "";
APT::Compressor::::CompressArg:: "-9";
APT::Compressor::::UncompressArg "";
APT::Compressor::::UncompressArg:: "-d";
Dir "/";
Dir::State "var/lib/apt/";
Dir::State::lists "lists/";
Dir::State::cdroms "cdroms.list";
Dir::State::mirrors "mirrors/";
Dir::State::extended_states "extended_states";
Dir::State::status "/var/lib/dpkg/status";
Dir::Cache "var/cache/apt/";
Dir::Cache::archives "archives/";
Dir::Cache::srcpkgcache "srcpkgcache.bin";
Dir::Cache::pkgcache "pkgcache.bin";
Dir::Etc "etc/apt/";
Dir::Etc::sourcelist "sources.list";
Dir::Etc::sourceparts "sources.list.d";
Dir::Etc::vendorlist "vendors.list";
Dir::Etc::vendorparts "vendors.list.d";
Dir::Etc::main "apt.conf";
Dir::Etc::netrc "auth.conf";
Dir::Etc::parts "apt.conf.d";
Dir::Etc::preferences "preferences";
Dir::Etc::preferencesparts "preferences.d";
Dir::Etc::trusted "trusted.gpg";
Dir::Etc::trustedparts "trusted.gpg.d";
Dir::Bin "";
Dir::Bin::methods "/usr/lib/apt/methods";
Dir::Bin::solvers "";
Dir::Bin::solvers:: "/usr/lib/apt/solvers";
Dir::Bin::dpkg "/usr/bin/dpkg";
Dir::Bin::bzip2 "/bin/bzip2";
Dir::Bin::xz "/usr/bin/xz";
Dir::Media "";
Dir::Media::MountPath "/media/apt";
Dir::Log "var/log/apt";
Dir::Log::Terminal "term.log";
Dir::Log::History "history.log";
Dir::Ignore-Files-Silently "";
Dir::Ignore-Files-Silently:: "~$";
Dir::Ignore-Files-Silently:: "\.disabled$";
Dir::Ignore-Files-Silently:: "\.bak$";
Dir::Ignore-Files-Silently:: "\.dpkg-[a-z]+$";
Dir::Ignore-Files-Silently:: "\.save$";
Dir::Ignore-Files-Silently:: "\.orig$";
Acquire "";
Acquire::cdrom "";
Acquire::cdrom::mount "/media/cdrom/";
Acquire::Pdiffs "false";
Acquire::Languages "";
Acquire::Languages:: "en";
Acquire::Languages:: "none";
DPkg "";
DPkg::Post-Invoke "";
DPkg::Post-Invoke:: "exec sync";
DPkg::Pre-Install-Pkgs "";
DPkg::Pre-Install-Pkgs:: "/usr/bin/apt-listchanges --apt || test $? -ne 10";
DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
DPkg::Tools "";
DPkg::Tools::Options "";
DPkg::Tools::Options::/usr/bin/apt-listchanges "";
DPkg::Tools::Options::/usr/bin/apt-listchanges::Version "2";
Aptitude "";
Aptitude::Recommends-Important "false";
CommandLine "";
CommandLine::AsString "apt-config dump";

-- System Information:
Debian Release: wheezy/sid
  APT prefers testing
  APT policy: (400, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 3.2.0-3-amd64 (SMP w/8 CPU cores)
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
Shell: /bin/sh linked to /bin/dash

Versions of packages apt depends on:
ii  debian-archive-keyring  2012.4
ii  gnupg                   1.4.12-4+b1
ii  libapt-pkg4.12          0.9.7.4
ii  libc6                   2.13-35
ii  libgcc1                 1:4.7.1-7
ii  libstdc++6              4.7.1-7

apt recommends no packages.

Versions of packages apt suggests:
pn  apt-doc     <none>
ii  aptitude    0.6.8.1-1
ii  dpkg-dev    1.16.8
ii  python-apt  0.8.4
ii  xz-utils    5.1.1alpha+20120614-1

-- no debconf information

Message #10 received at [email protected] (full text, mbox, reply):

From: David Kalnischkies <[email protected]>

To: Thijs Kinkhorst <[email protected]>, [email protected]

Cc: [email protected]

Subject: Re: Bug#688303: fails on concatenated bzip2 compressed Packages files (regression)

Date: Wed, 26 Sep 2012 16:09:16 +0200

reassign 688303 bzip2 1.0.6-4
severity 688303 wishlist
retitle 688303 Please support concatenated bz2 files in zlib-functions
thanks

On Fri, Sep 21, 2012 at 11:28 AM, Thijs Kinkhorst <[email protected]> wrote:
> The problem is that our APT archive contains a Packages.bz2 which is a
> concatenation of other bz2 files (of sub-archives). Two concatenated
> bzip2 files again form a valid bzip2 file. The b(un)zip2 command line
> utils support this and it's a documented feature.

It may be a documented feature of bzip2, but not of APT as you have noticed.
It is also not part of any documentation for a repository format, which all
say that the Packages.bz2 file is a compressed Packages file.
Nowhere is said that this could be a concatenation of different streams a
compression algorithm knows by accident how to combine into one file.
(It speaks volumes that the documentation of the higher level API mentions
 indeed the method you point to, but in the example code showing writing and
 reading a file on the very same page it is not used, nor even mentioned -
 maybe because they also see a difference between file and stream)


Anyway, we don't even use the higher level functions, we use the zlib-
compatibility functions which do not expose this "detail", so I am
reassigning this to libbz2 for them to consider implementing support for
this one way or another.


Either way, it's a bug in your archive to assume that such bz2-files would
be supported by APT or anything else working with these indexes without it
ever be documented as supported.

The reference implementation for an archive are the ones operated by
ftp-masters - it is reasonable to assume everything they don't use is
not safe to use in yours.


Best regards

David Kalnischkies

Send a report that this bug log contains spam.