Package: apt
Version: 0.5.27
Severity: normal
Bug number #152711 reported that apt did not handle chunked encoding from the
server properly. A fix for that bug was applied for v0.5.25 of apt. I have
tested apt again recently and a different problem is now present with chunked
encoding.
When I retrieve a file from a server that uses chunked encoding apt will
sometimes corrupt the file by failing to remove the chunked encoding information
from the data stream.
For example the following is the file from /var/lib/apt/lists/partial/ for one
of the Release files (I have replaced the character 13 by the two characters ^M
so that it shows clearly in plain text).
--------------------
58^M
Archive: testing
Component: non-US/main
Origin: Debian
Label: Debian
Architecture: i386
^M
0^M
^M
--------------------
This is clearly the body that was sent from the server without removing the
chunked encoding. The data is valid since it says the length is 0x58 bytes
which it clearly is.
The Packages.gz files all contain the ^M, 0^M, ^M lines at the end and are
corrupted in the middle as well.
-- System Information:
Debian Release: 3.1
APT prefers testing
APT policy: (990, 'testing'), (50, 'unstable')
Architecture: i386 (i686)
Kernel: Linux 2.6.7-67
Locale: LANG=C, LC_CTYPE=C
Versions of packages apt depends on:
ii libc6 2.3.2.ds1-13 GNU C Library: Shared libraries an
ii libgcc1 1:3.4.1-4sarge1 GCC support library
ii libstdc++5 1:3.3.4-6sarge1.2 The GNU Standard C++ Library v3
-- no debconf information
Here we see the effect reading from local wwwoffle on an offline system:
# aptitude install ftm
Need to get 27.2kB of archives. After unpacking 73.7kB will be used.
Get:1 http://debian.linux.org.tw sid/main ftm 0.0.7 [27.2kB]
Err http://debian.linux.org.tw sid/main ftm 0.0.7
Error reading from server Remote end closed connection
# aptitude install ftm
Need to get 27.2kB of archives. After unpacking 73.7kB will be used.
Get:1 http://debian.linux.org.tw sid/main ftm 0.0.7 [27.2kB]
Fetched 27.2kB in 0s (2411kB/s)
One cannot reproduce it usually without rebooting, as internal buffers
filling affect timing issues, etc.
On Sun, Oct 10, 2004 at 04:42:11AM +0800, Dan Jacobson wrote:
> Here we see the effect reading from local wwwoffle on an offline system:
>
> # aptitude install ftm
> Need to get 27.2kB of archives. After unpacking 73.7kB will be used.
> Get:1 http://debian.linux.org.tw sid/main ftm 0.0.7 [27.2kB]
> Err http://debian.linux.org.tw sid/main ftm 0.0.7
> Error reading from server Remote end closed connection
> # aptitude install ftm
> Need to get 27.2kB of archives. After unpacking 73.7kB will be used.
> Get:1 http://debian.linux.org.tw sid/main ftm 0.0.7 [27.2kB]
> Fetched 27.2kB in 0s (2411kB/s)
>
> One cannot reproduce it usually without rebooting, as internal buffers
> filling affect timing issues, etc.
I can reproduce the problem, and I think I know why it happens.
I wrote a proxy that converts every data stream going over it to
Transfer-Encoding: Chunked. It's available at:
<URL:http://[email protected]/~greek0/div/chunked-proxy.py>
It works quite well AFAICT, but it relies on python's httplib, which
turned out to have some problems as well, at least if the servers
send strange data (\r in the headers without \r\n), as
ftp.at.debian.org does. So if you try it out and get garbled data
please make sure that it really is apt's fault.
The problem seems to be timing sensitive, since singlestepping in
gdb, as well as adding adding sleep(10) calls to the proxy prevent
it from happening.
The error message and the corrupted data comes from
HttpMethod::ServerDie, which is called by HttpMethod::Go when it
fails to read data from the server (ie. the server closed the
connection).
ServerDie then flushes all the Data that we have already read
(usually either all the Data the server sent us excluding headers,
or just the last part of the file that we haven't written out yet).
It does this even if we are using a Transfer-Encoding, so the
resulting file is corrupted, since the Encoding is not stripped.
Additionally we usually get a "Remote host closed connection" error.
I don't really know what's the right point to fix the problem. IMHO
ServerDie might be too late. HttpMethod::Go might be right, but I
don't know how we should detect a failing connection for
chunked encoding then.
Cheers,
Greek0
Acknowledgement sent
to "Eugene V. Lyubimkin" <[email protected]>:
Extra info received and forwarded to list. Copy sent to APT Development Team <[email protected]>.
(Sun, 16 Nov 2008 11:18:13 GMT) (full text, mbox, link).
Hello Andrew, hello Dan.
Is the problem present in apt >= 0.7.16 ?
--
Eugene V. Lyubimkin aka JackYF, JID: jackyf.devel(maildog)gmail.com
Ukrainian C++ developer, Debian APT contributor
EVL> Is the problem present in apt >= 0.7.16 ?
I'm not sure... I'm no power user. All I know is I sometimes have to
do apt-get update twice on my tiny Openmoko phone...
Acknowledgement sent
to "Eugene V. Lyubimkin" <[email protected]>:
Extra info received and forwarded to list. Copy sent to APT Development Team <[email protected]>.
(Sun, 16 Nov 2008 22:18:05 GMT) (full text, mbox, link).
[email protected] wrote:
> EVL> Is the problem present in apt >= 0.7.16 ?
>
> I'm not sure... I'm no power user. All I know is I sometimes have to
> do apt-get update twice on my tiny Openmoko phone...
>
I meant "Is the problem reproducable?".
Regards,
--
Eugene V. Lyubimkin aka JackYF, JID: jackyf.devel(maildog)gmail.com
Ukrainian C++ developer, Debian APT contributor
Acknowledgement sent
to [email protected] (Andrew M. Bishop):
Extra info received and forwarded to list. Copy sent to APT Development Team <[email protected]>.
(Thu, 20 Nov 2008 10:20:21 GMT) (full text, mbox, link).
Acknowledgement sent
to "Eugene V. Lyubimkin" <[email protected]>:
Extra info received and forwarded to list. Copy sent to APT Development Team <[email protected]>.
(Thu, 20 Nov 2008 11:48:17 GMT) (full text, mbox, link).