Subject: autopkgtest: fail early when disk space is insufficient to download and decompress source
Date: Sat, 04 Jun 2022 12:24:28 +0200
Package: autopkgtest
Version: 5.22
Severity: normal
Hi,
This came up in bug #1011646 and I just discussed this with pabs on
IRC. autopkgtest could fail early once it's known that disk space is
insufficient for the source to be downloaded and decompressed, instead
of what we see now [1].
Ideally $(apt-get source) and $(dpkg-source) do this natively (I need
to figure out what they do already and if new bugs need to be filed),
but until they do, autopkgtest could do something at least after the
download by $(apt-get source). An example of what could be done (not
usable straight away, but it can serve as inspiration) was shared by
pabs and I paste it below [2,3].
Paul
[1] https://ci.debian.net/data/autopkgtest/testing/ppc64el/n/nvidia-cuda-toolkit/22025556/log.gz
autopkgtest [12:35:33]: @@@@@@@@@@@@@@@@@@@@ apt-source nvidia-cuda-toolkit
Get:1 http://deb.debian.org/debian unstable/non-free nvidia-cuda-toolkit 11.4.3-3 (dsc) [9,029 B]
Get:2 http://deb.debian.org/debian unstable/non-free nvidia-cuda-toolkit 11.4.3-3 (tar) [1,793 MB]
Get:3 http://deb.debian.org/debian unstable/non-free nvidia-cuda-toolkit 11.4.3-3 (tar) [1,248 MB]
Get:4 http://deb.debian.org/debian unstable/non-free nvidia-cuda-toolkit 11.4.3-3 (tar) [28.3 MB]
Get:5 http://deb.debian.org/debian unstable/non-free nvidia-cuda-toolkit 11.4.3-3 (tar) [28.1 MB]
Get:6 http://deb.debian.org/debian unstable/non-free nvidia-cuda-toolkit 11.4.3-3 (tar) [66.6 MB]
Get:7 http://deb.debian.org/debian unstable/non-free nvidia-cuda-toolkit 11.4.3-3 (tar) [1,469 MB]
Get:8 http://deb.debian.org/debian unstable/non-free nvidia-cuda-toolkit 11.4.3-3 (tar) [196 B]
Get:9 http://deb.debian.org/debian unstable/non-free nvidia-cuda-toolkit 11.4.3-3 (diff) [60.1 kB]
gpgv: unknown type of key resource 'trustedkeys.kbx'
gpgv: keyblock resource '/tmp/dpkg-verify-sig.wAbyHvXr/trustedkeys.kbx': General error
gpgv: Signature made Mon 23 May 2022 12:14:31 AM UTC
gpgv: using RSA key EBF30A30A8D9C63BDA44C6945FB33F9359E9ED08
gpgv: issuer "[email protected]"
gpgv: Can't check signature: No public key
dpkg-source: warning: cannot verify signature ./nvidia-cuda-toolkit_11.4.3-3.dsc
tar: nvidia-cuda-toolkit-11.4.3.orig-arm64/libnpp/targets/sbsa-linux/lib/libnpps_static.a: Cannot write: No space left on device
tar: nvidia-cuda-toolkit-11.4.3.orig-arm64/libnpp/targets/sbsa-linux/lib/stubs: Cannot mkdir: No space left on device
tar:
[2]
def uncompressed_size(filename):
uc_size = 0
file_size = os.path.getsize(filename)
with open(filename, 'rb') as f:
magic = f.read(6)
# *.gz
if magic[:2] == "\x1f\x8b":
f.seek(-4, 2)
data = f.read()
uc_size = struct.unpack('<I', data)[0]
# *.bz2
elif magic[:3] == 'BZh':
# Crude estimate based on average compression ratio of 25%
uc_size = file_size*4
# *.xz
elif magic == "\xfd7zXZ\x00":
cmdline = ['xz', '--verbose', '--list', filename]
process = subprocess.Popen(cmdline, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = process.communicate()[0].decode('utf-8')
if process.returncode:
logging.warning('xz reported failure to check size of %s:', filename)
logging.warning(output)
else:
for line in output.splitlines():
line = line.strip()
if line.startswith('Uncompressed size:'):
match = re.match(r'Uncompressed size: .*?([0-9,]+) B', line)
if match: uc_size = int(''.join(match.group(1).split(',')))
else: logging.warning('xz reported weird output for %s: %s', filename, line)
# *.lz
elif magic[:4] == 'LZIP':
f.seek(-16, 2)
data = f.read(8)
uc_size = struct.unpack('<Q', data)[0]
return max(file_size, uc_size)
[3]
[08:22:01] <elbrus> pabs: can I quote your paste.d.n in a bug report?
[08:23:00] <pabs> sure. the code is currently MIT/Expat, but feel free to apply any DFSG-free license
[08:23:58] <pabs> its from https://salsa.debian.org/deriv-team/census/blob/master/bin/compare-source-package-list