Debian Bug report logs - #784310
apt update always fails on experimental/non-free

Reply or subscribe to this bug.

Display info messages

Message #5 received at [email protected] (full text, mbox, reply):

From: Cesare Leonardi <[email protected]>

To: Debian Bug Tracking System <[email protected]>

Subject: apt update always fails on experimental/non-free

Date: Tue, 05 May 2015 09:43:53 +0200

Package: apt
Version: 1.0.9.9
Severity: normal

Since months i have an error during the apt update phase, always on
experimental/non-free. Initially i thought it was a temporary mirror
problem, because changing the repository the error went away.
But when in the next days i was doing another update, the error came back,
even with the changed mirror.

It happens on at least two machine of mine, an i386 and an amd64 arch, with
substantially the same sources.list and i think it started to show this
behaviour at the same time.
And it happens with aptitude too.

Apart from sources.list, i think i've never changed nothing from default
apt setup.

Try these steps, to me are always reproducible:
- Take my sources.list.
- If you are currently using the same mirror like mine, change it.
- apt update
- Look that no error are displayed: everything is correctly updated.
- Rerun: apt update
- Here i see an "Err" line on experimental/non-free Packages.

Here is my output:
-------------------------
root@etna:~# apt update
Get:1 http://httpredir.debian.org unstable InRelease [204 kB]
Get:2 http://httpredir.debian.org experimental InRelease [162 kB]
Get:3 http://httpredir.debian.org unstable/contrib Translation-en [42,1 kB]
Get:4 http://httpredir.debian.org unstable/main Translation-en [4.878 kB]    
Get:5 http://httpredir.debian.org unstable/non-free Translation-en [76,4 kB]         
Get:6 http://httpredir.debian.org experimental/contrib Translation-en [4.527 B]                                            
Get:7 http://httpredir.debian.org experimental/main Translation-en [429 kB]                           
Get:8 http://httpredir.debian.org unstable/main amd64 Packages [7.189 kB]                            
Get:9 http://httpredir.debian.org experimental/non-free Translation-en [10,2 kB]                                                              
Get:10 http://httpredir.debian.org experimental/contrib amd64 Packages [6.144 B]                                                                               
Get:11 http://httpredir.debian.org experimental/non-free amd64 Packages [13,9 kB]                                                   
Get:12 http://httpredir.debian.org unstable/non-free amd64 Packages [89,2 kB]                                                                                          
Get:13 http://httpredir.debian.org unstable/contrib amd64 Packages [54,1 kB]                                                                                           
Get:14 http://httpredir.debian.org experimental/main amd64 Packages [737 kB]                                                                                           
Fetched 13,9 MB in 24s (568 kB/s)                                                                                                                                      
Reading package lists... Done
Building dependency tree       
Reading state information... Done
34 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@etna:~#
root@etna:~# apt update
Get:1 http://httpredir.debian.org unstable InRelease [204 kB]
Hit http://httpredir.debian.org experimental InRelease          
Get:2 http://httpredir.debian.org experimental/contrib amd64 Packages/DiffIndex [7.819 B]
Get:3 http://httpredir.debian.org experimental/main amd64 Packages/DiffIndex [7.819 B]
Get:4 http://httpredir.debian.org experimental/non-free amd64 Packages/DiffIndex [7.819 B]
Get:5 http://httpredir.debian.org experimental/contrib Translation-en/DiffIndex [7.819 B]
Get:6 http://httpredir.debian.org experimental/main Translation-en/DiffIndex [7.819 B]
Get:7 http://httpredir.debian.org experimental/non-free Translation-en/DiffIndex [7.819 B]
Get:8 http://httpredir.debian.org unstable/main amd64 Packages/DiffIndex [7.876 B]
Get:9 http://httpredir.debian.org unstable/non-free amd64 Packages/DiffIndex [7.819 B]
Get:10 http://httpredir.debian.org unstable/contrib amd64 Packages/DiffIndex [7.819 B]                        
Get:11 http://httpredir.debian.org unstable/contrib Translation-en/DiffIndex [7.819 B]
Get:12 http://httpredir.debian.org unstable/main Translation-en/DiffIndex [7.876 B]
Get:13 http://httpredir.debian.org unstable/non-free Translation-en/DiffIndex [7.819 B]
Err http://httpredir.debian.org experimental/non-free amd64 Packages
  
Hit http://httpredir.debian.org experimental/non-free amd64 Packages
Fetched 298 kB in 4s (66,2 kB/s)
Reading package lists... Done
Building dependency tree       
Reading state information... Done
34 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@etna:~# 
-------------------------

So when you change mirror and update for the first time you get no error.
And that's why in the past i thought was a mirror problem: because changing
mirror the error apparently went away.
But on subsequently updates the error is always present.

Cesare.


-- Package-specific info:

-- apt-config dump --

APT "";
APT::Architecture "amd64";
APT::Build-Essential "";
APT::Build-Essential:: "build-essential";
APT::Install-Recommends "1";
APT::Install-Suggests "0";
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^linux-headers-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^linux-image-extra-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^linux-signed-image-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^kfreebsd-image-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^kfreebsd-headers-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^gnumach-image-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^.*-modules-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^.*-kernel-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^linux-backports-modules-.*-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^linux-tools-3\.16\.0-4-amd64$";
APT::VersionedKernelPackages "";
APT::VersionedKernelPackages:: "linux-image";
APT::VersionedKernelPackages:: "linux-headers";
APT::VersionedKernelPackages:: "linux-image-extra";
APT::VersionedKernelPackages:: "linux-signed-image";
APT::VersionedKernelPackages:: "kfreebsd-image";
APT::VersionedKernelPackages:: "kfreebsd-headers";
APT::VersionedKernelPackages:: "gnumach-image";
APT::VersionedKernelPackages:: ".*-modules";
APT::VersionedKernelPackages:: ".*-kernel";
APT::VersionedKernelPackages:: "linux-backports-modules-.*";
APT::VersionedKernelPackages:: "linux-tools";
APT::Never-MarkAuto-Sections "";
APT::Never-MarkAuto-Sections:: "metapackages";
APT::Never-MarkAuto-Sections:: "restricted/metapackages";
APT::Never-MarkAuto-Sections:: "universe/metapackages";
APT::Never-MarkAuto-Sections:: "multiverse/metapackages";
APT::Never-MarkAuto-Sections:: "oldlibs";
APT::Never-MarkAuto-Sections:: "restricted/oldlibs";
APT::Never-MarkAuto-Sections:: "universe/oldlibs";
APT::Never-MarkAuto-Sections:: "multiverse/oldlibs";
APT::Architectures "";
APT::Architectures:: "amd64";
APT::Compressor "";
APT::Compressor::. "";
APT::Compressor::.::Name ".";
APT::Compressor::.::Extension "";
APT::Compressor::.::Binary "";
APT::Compressor::.::Cost "1";
APT::Compressor::gzip "";
APT::Compressor::gzip::Name "gzip";
APT::Compressor::gzip::Extension ".gz";
APT::Compressor::gzip::Binary "gzip";
APT::Compressor::gzip::Cost "2";
APT::Compressor::gzip::CompressArg "";
APT::Compressor::gzip::CompressArg:: "-9n";
APT::Compressor::gzip::UncompressArg "";
APT::Compressor::gzip::UncompressArg:: "-d";
APT::Compressor::bzip2 "";
APT::Compressor::bzip2::Name "bzip2";
APT::Compressor::bzip2::Extension ".bz2";
APT::Compressor::bzip2::Binary "bzip2";
APT::Compressor::bzip2::Cost "3";
APT::Compressor::bzip2::CompressArg "";
APT::Compressor::bzip2::CompressArg:: "-9";
APT::Compressor::bzip2::UncompressArg "";
APT::Compressor::bzip2::UncompressArg:: "-d";
APT::Compressor::xz "";
APT::Compressor::xz::Name "xz";
APT::Compressor::xz::Extension ".xz";
APT::Compressor::xz::Binary "xz";
APT::Compressor::xz::Cost "4";
APT::Compressor::xz::CompressArg "";
APT::Compressor::xz::CompressArg:: "-6";
APT::Compressor::xz::UncompressArg "";
APT::Compressor::xz::UncompressArg:: "-d";
APT::Compressor::lzma "";
APT::Compressor::lzma::Name "lzma";
APT::Compressor::lzma::Extension ".lzma";
APT::Compressor::lzma::Binary "xz";
APT::Compressor::lzma::Cost "5";
APT::Compressor::lzma::CompressArg "";
APT::Compressor::lzma::CompressArg:: "--format=lzma";
APT::Compressor::lzma::CompressArg:: "-9";
APT::Compressor::lzma::UncompressArg "";
APT::Compressor::lzma::UncompressArg:: "--format=lzma";
APT::Compressor::lzma::UncompressArg:: "-d";
Dir "/";
Dir::State "var/lib/apt/";
Dir::State::lists "lists/";
Dir::State::cdroms "cdroms.list";
Dir::State::mirrors "mirrors/";
Dir::State::extended_states "extended_states";
Dir::State::status "/var/lib/dpkg/status";
Dir::Cache "var/cache/apt/";
Dir::Cache::archives "archives/";
Dir::Cache::srcpkgcache "srcpkgcache.bin";
Dir::Cache::pkgcache "pkgcache.bin";
Dir::Etc "etc/apt/";
Dir::Etc::sourcelist "sources.list";
Dir::Etc::sourceparts "sources.list.d";
Dir::Etc::vendorlist "vendors.list";
Dir::Etc::vendorparts "vendors.list.d";
Dir::Etc::main "apt.conf";
Dir::Etc::netrc "auth.conf";
Dir::Etc::parts "apt.conf.d";
Dir::Etc::preferences "preferences";
Dir::Etc::preferencesparts "preferences.d";
Dir::Etc::trusted "trusted.gpg";
Dir::Etc::trustedparts "trusted.gpg.d";
Dir::Bin "";
Dir::Bin::methods "/usr/lib/apt/methods";
Dir::Bin::solvers "";
Dir::Bin::solvers:: "/usr/lib/apt/solvers";
Dir::Bin::dpkg "/usr/bin/dpkg";
Dir::Bin::bzip2 "/bin/bzip2";
Dir::Bin::xz "/usr/bin/xz";
Dir::Bin::lzma "/usr/bin/xz";
Dir::Media "";
Dir::Media::MountPath "/media/apt";
Dir::Log "var/log/apt";
Dir::Log::Terminal "term.log";
Dir::Log::History "history.log";
Dir::Ignore-Files-Silently "";
Dir::Ignore-Files-Silently:: "~$";
Dir::Ignore-Files-Silently:: "\.disabled$";
Dir::Ignore-Files-Silently:: "\.bak$";
Dir::Ignore-Files-Silently:: "\.dpkg-[a-z]+$";
Dir::Ignore-Files-Silently:: "\.save$";
Dir::Ignore-Files-Silently:: "\.orig$";
Dir::Ignore-Files-Silently:: "\.distUpgrade$";
Acquire "";
Acquire::cdrom "";
Acquire::cdrom::mount "/media/cdrom/";
Acquire::Languages "";
Acquire::Languages:: "en";
Acquire::Languages:: "none";
DPkg "";
DPkg::Pre-Install-Pkgs "";
DPkg::Pre-Install-Pkgs:: "/usr/bin/apt-listchanges --apt || test $? -ne 10";
DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
DPkg::Tools "";
DPkg::Tools::Options "";
DPkg::Tools::Options::/usr/bin/apt-listchanges "";
DPkg::Tools::Options::/usr/bin/apt-listchanges::Version "2";
DPkg::Post-Invoke "";
DPkg::Post-Invoke:: "test -x /usr/lib/needrestart/apt-pinvoke && /usr/lib/needrestart/apt-pinvoke || true";
CommandLine "";
CommandLine::AsString "apt-config dump";

-- (no /etc/apt/preferences present) --


-- /etc/apt/sources.list --

deb http://httpredir.debian.org/debian unstable main
deb http://httpredir.debian.org/debian unstable contrib
deb http://httpredir.debian.org/debian unstable non-free
deb http://httpredir.debian.org/debian experimental main
deb http://httpredir.debian.org/debian experimental contrib
deb http://httpredir.debian.org/debian experimental non-free

#deb http://ftp.it.debian.org/debian unstable main
#deb http://ftp.it.debian.org/debian unstable contrib
#deb http://ftp.it.debian.org/debian unstable non-free
#deb http://ftp.it.debian.org/debian experimental main
#deb http://ftp.it.debian.org/debian experimental contrib
#deb http://ftp.it.debian.org/debian experimental non-free

#deb http://mirrors.kernel.org/debian unstable main
#deb http://mirrors.kernel.org/debian unstable contrib
#deb http://mirrors.kernel.org/debian unstable non-free
#deb http://mirrors.kernel.org/debian experimental main
#deb http://mirrors.kernel.org/debian experimental contrib
#deb http://mirrors.kernel.org/debian experimental non-free


-- System Information:
Debian Release: stretch/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 3.16.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages apt depends on:
ii  debian-archive-keyring  2014.3
ii  gnupg                   1.4.18-7
ii  libapt-pkg4.12          1.0.9.9
ii  libc6                   2.19-18
ii  libgcc1                 1:5.1.1-3
ii  libstdc++6              5.1.1-3

apt recommends no packages.

Versions of packages apt suggests:
pn  apt-doc     <none>
ii  aptitude    0.6.11-1+b1
pn  dpkg-dev    <none>
ii  python-apt  0.9.3.11

-- no debconf information

Message #10 received at [email protected] (full text, mbox, reply):

From: Axel Beckert <[email protected]>

To: [email protected]

Cc: Cesare Leonardi <[email protected]>

Subject: Re: Bug#784310: apt update always fails on experimental/non-free

Date: Tue, 5 May 2015 14:58:45 +0200

Control: found -1 1.0.9.8
Control: affects -1 aptitude

Hi,

Cesare Leonardi wrote:
> Since months i have an error during the apt update phase, always on
> experimental/non-free.

Same here.

> Initially i thought it was a temporary mirror problem,

Dito.

> Err http://httpredir.debian.org experimental/non-free amd64 Packages

Also the http redirection is not the cause as I get this with using
plain http on ftp.ch.debian.org:

Err http://ftp.ch.debian.org experimental/non-free amd64 Packages            

I though currently can't reproduce it on i386.

		Regards, Axel
-- 
 ,''`.  |  Axel Beckert <[email protected]>, http://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5
  `-    |  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE

Message #19 received at [email protected] (full text, mbox, reply):

From: David Kalnischkies <[email protected]>

To: [email protected]

Cc: Cesare Leonardi <[email protected]>

Subject: Re: Bug#784310: apt update always fails on experimental/non-free

Date: Thu, 7 May 2015 13:29:46 +0200

[Message part 1 (text/plain, inline)]

Hi,

On Tue, May 05, 2015 at 09:43:53AM +0200, Cesare Leonardi wrote:
> root@etna:~# apt update
> Get:1 http://httpredir.debian.org unstable InRelease [204 kB]
> Hit http://httpredir.debian.org experimental InRelease
[…]
> Err http://httpredir.debian.org experimental/non-free amd64 Packages
>
> Hit http://httpredir.debian.org experimental/non-free amd64 Packages
> Fetched 298 kB in 4s (66,2 kB/s)

Would be nice if you could run
apt update -o Debug::Acquire::http=1

(This enables a lot of output resembling the messages send between
server(s) and the apt client(s). Probably best to redirect the output to
a file with e.g.:  2>&1 | tee apt-update.log
as the different interactions happen in parallel, so the output can be
a confusing mix of stuff on first look).

We are working in /experimental on revamping the acquire system at large
which is what is behind downloading stuff and shared between all
package managers based on libapt, so apt and aptitude do exactly the
same here.

I am in particular working currently in my branch on the behavior of
getting a 'Hit' for a (In)Release(.gpg) file currently, which is what
you got in your example and likely most of the time as non-free
experimental isn't changing a lot.

Note that this Err isn't critical, apt happily recovers from many
'errors' as getting them is "normal". apt implements an HTTP1.1 client
which tries to use many features of HTTP1.1, but many servers running on
mirrors implement only a subset of the features. And many different
subsets. Not to mention that some downright refuse to implement them
correctly (one of the sillier examples is e.g. #778375 or pipelining, my
personal favorite). Unfortunately the internet isn't implemented based
on specs, but based on what popular browsers use and support (which
isn't much – its pretty ironic that our selfwritten test webserver
supports more of HTTP1.1 than most 'real' webservers. And its only
getting worse)…


Usually you are not told about such errors, but it looks like one
slipped through here. I haven't looked at the 'old' source, but
I have a suspicion, which the debug output hopefully confirms:

The server is supporting Range-requests, but not If-Range and hence
answers with a 416, but not with Content-Range set. I at least have
a faint memory of encountering such a server recently with these
symptoms.


So, summary: If the debug output isn't proofing me horribly wrong
(possible), I am going to close this bugreport (if unreproducible with
our forthcoming new acquire system) as this is a 'display issue' and the
fix is basically our 'webservers are idiots' rewrite^Wrework… ;)

[signature.asc (application/pgp-signature, inline)]

Message #24 received at [email protected] (full text, mbox, reply):

From: Cesare Leonardi <[email protected]>

To: Axel Beckert <[email protected]>, [email protected]

Subject: Re: Bug#784310: apt update always fails on experimental/non-free

Date: Fri, 08 May 2015 01:56:49 +0200

On 05/05/2015 14:58, Axel Beckert wrote:
> I though currently can't reproduce it on i386.

I was almost sure that i got the error on a i386 PC of mine but since i 
haven't really tried for some time, the doubt started to arise and since 
your reply i've run some tests.
I confirm what Axel said: the error doesn't show under i386. And i've 
verified also on another i386 PCs (up to date Sid).

Then i've verified with two KVM virtual machine, both with updated Sid, 
same sources.list, but one i386 and one amd64: i can see the error only 
on amd64.

To be honest, now, while writing this mail, on the first i386 PC tested, 
i get error on experimental/main, but only using httpredir in 
sources.list and only in the first run after changing mirror. Don't know 
what to think...

Cesare.

Message #29 received at [email protected] (full text, mbox, reply):

From: Cesare Leonardi <[email protected]>

To: David Kalnischkies <[email protected]>, [email protected]

Cc: [email protected]

Subject: Re: Bug#784310: apt update always fails on experimental/non-free

Date: Fri, 08 May 2015 01:58:22 +0200

[Message part 1 (text/plain, inline)]

On 07/05/2015 13:29, David Kalnischkies wrote:
> Would be nice if you could run
> apt update -o Debug::Acquire::http=1

Ok, i've attached a ZIP with 4 log:
- 2 from i386 and 2 from amd64
- 2 for the first run, after changing mirror (run1) and 2 for the next 
run (run2).

Under i386 i see that in the second run are searched only diff files, 
while under amd64, only for experimental/main, are searched Packages.bz2 
and Packages.xz: the first one generates the error, because doesn't exist.

> We are working in /experimental on revamping the acquire system at large

Thank you very much for sharing information about the development of apt.

> Note that this Err isn't critical, apt happily recovers from many
> 'errors' as getting them is "normal".
[...]
> Usually you are not told about such errors, but it looks like one
> slipped through here.

In fact in the first run i see a lot of error about apt searching for 
bz2 files first (that are missing), then for xz (ok).

> The server is supporting Range-requests, but not If-Range and hence
> answers with a 416, but not with Content-Range set.

To me it doesn't look the case.

Cesare.

[apt-update.zip (application/zip, attachment)]

Send a report that this bug log contains spam.