Subject: gdb: extremely slow response to basic commands
Date: Thu, 09 Nov 2023 17:17:32 +0100
Package: gdb
Version: 13.1-3
Severity: important
X-Debbugs-Cc: [email protected]
The gdb compiled for Debian12 is extremely slow - it takes tens of seconds to
minutes
to execute even the simplest commands, like "info args", or "backtrace" - for a
medium size program.
I've modified Codeblocks IDE to measure gdb response times, here's the link:
https://forums.codeblocks.org/index.php/topic,25561.msg174050.html#msg174050
Apparently the problem is with the build flags used to compile gdb.
I've made a test build of the source package:
"gdb-source_13.1-3_all.deb"
build flags:
> export target_configargs="--enable-threading --with-gnu-ld --enable-
libbacktrace --with-zstd --with-system-readline --with-system-zlib --with-
xxhash=yes --disable-unit-tests --disable-sim"
> ./configure --enable-lto --enable-vtable-verify --disable-libstdcxx
The newly built gdb is working as expected - measured response times are close
to zero (see the link posted above)
Regards
-- System Information:
Debian Release: 12.2
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable-debug'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 6.1.0-13-amd64 (SMP w/2 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages gdb depends on:
ii libbabeltrace1 1.5.11-1+b2
ii libc6 2.36-9+deb12u3
ii libdebuginfod1 0.188-2.1
ii libexpat1 2.5.0-1
ii libgcc-s1 12.2.0-14
ii libgmp10 2:6.2.1+dfsg1-1.1
ii libipt2 2.0.5-1
ii liblzma5 5.4.1-0.2
ii libmpfr6 4.2.0-1
ii libncursesw6 6.4-4
ii libpython3.11 3.11.2-6
ii libreadline8 8.2-1.3
ii libsource-highlight4v5 3.1.9-4.2+b3
ii libstdc++6 12.2.0-14
ii libtinfo6 6.4-4
ii libxxhash0 0.8.1-1
ii libzstd1 1.5.4+dfsg2-5
ii zlib1g 1:1.2.13.dfsg-1
Versions of packages gdb recommends:
ii libc6-dbg [libc-dbg] 2.36-9+deb12u3
Versions of packages gdb suggests:
ii gdb-doc 13.1-1
pn gdbserver <none>
-- no debconf information
Acknowledgement sent
to tomazzi <[email protected]>:
Extra info received and forwarded to list. Copy sent to Héctor Orón Martínez <[email protected]>.
(Sat, 11 Nov 2023 12:15:05 GMT) (full text, mbox, link).
Subject: gdb: extremely slow response to basic commands
Date: Sat, 11 Nov 2023 13:05:02 +0100
Today I've tried to do the same trick on some other machine (using build
flags from previous mail), but the build failed with error message:
"ar: `u' modifier ignored since `D' is the default (see `U')"
while compiling "libgdbsupport.a"
So I've just spent few hours trying to figure out why it worked on the
other PC ...
Long story short:
During testing, I've first run ./configure from the gdb/gdb directory
with the following flags:
./configure --enable-threading --with-gnu-ld --enable-libbacktrace
--with-xxhash=yes --with-debuginfod=yes
Then I've run ./configure from top /gdb dir (with the aforementioned
flags) and the compilation went fine.
So now the process is reproducible.
Regards
Acknowledgement sent
to tomazzi <[email protected]>:
Extra info received and forwarded to list. Copy sent to Héctor Orón Martínez <[email protected]>.
(Mon, 13 Nov 2023 20:51:02 GMT) (full text, mbox, link).
Subject: gdb: extremely slow response to basic commands
Date: Mon, 13 Nov 2023 21:46:37 +0100
Hello,
I've just built gdb "the Debian way", to check what are the differences
in the build process:
$> apt source gdb
$> apt build-dep gdb
$> debuild -b -uc -us
From debuild output:
gdb-default: configured with:
...
--without-babeltrace
--with-babeltrace
...
gdb-minimal: configured with:
...
--without-babeltrace
--with-babeltrace
--without-babeltrace
...
This looks bad, but the results are even more "interresting":
The gdb binary installed in the system (**debsums OK**) reports the
following configuration (the "show configuration" command, differences
only):
...
--with-babeltrace
--with-mpfr
--with-xxhash
--with-python=/usr (relocatable)
--with-python-libdir=/usr/lib (relocatable)
--enable-source-highlight
...
The newly "debuild" gdb reports something *different*:
...
--without-babeltrace << this: conflicting build flags
--without-mpfr
--with-xxhash
--without-python
--without-python-libdir
--disable-source-highlight
...
The gdb which I've compiled using ./configure script (so *without*
Debian patches) reports:
...
--with-babeltrace
--with-mpfr
--without-xxhash << this: libxxhash-dev is installed, but
flag is disabled ?
--with-python=/usr
--with-python-libdir=/usr/lib
--disable-source-highlight << this
...
Regards
Acknowledgement sent
to Héctor Orón Martínez <[email protected]>:
Extra info received and forwarded to list. Copy sent to Héctor Orón Martínez <[email protected]>.
(Mon, 13 Nov 2023 21:24:03 GMT) (full text, mbox, link).
Subject: Re: Bug#1055646: gdb: extremely slow response to basic commands
Date: Mon, 13 Nov 2023 22:20:28 +0100
Hello Tomazzi,
On Mon, 13 Nov 2023 at 21:51, tomazzi <[email protected]> wrote:
>
> Hello,
>
> I've just built gdb "the Debian way", to check what are the differences
> in the build process:
> $> apt source gdb
> $> apt build-dep gdb
> $> debuild -b -uc -us
>
> From debuild output:
>
> gdb-default: configured with:
> ...
> --without-babeltrace
> --with-babeltrace
> ...
> gdb-minimal: configured with:
> ...
> --without-babeltrace
> --with-babeltrace
> --without-babeltrace
> ...
>
> This looks bad, but the results are even more "interresting":
Latest switch is the one that should be in-use
> The gdb binary installed in the system (**debsums OK**) reports the
> following configuration (the "show configuration" command, differences
> only):
>
> ...
> --with-babeltrace
> --with-mpfr
> --with-xxhash
> --with-python=/usr (relocatable)
> --with-python-libdir=/usr/lib (relocatable)
> --enable-source-highlight
> ...
>
> The newly "debuild" gdb reports something *different*:
> ...
> --without-babeltrace << this: conflicting build flags
> --without-mpfr
> --with-xxhash
> --without-python
> --without-python-libdir
> --disable-source-highlight
> ...
>
>
> The gdb which I've compiled using ./configure script (so *without*
> Debian patches) reports:
> ...
> --with-babeltrace
> --with-mpfr
> --without-xxhash << this: libxxhash-dev is installed, but
> flag is disabled ?
> --with-python=/usr
> --with-python-libdir=/usr/lib
> --disable-source-highlight << this
> ...
>
Could you share your conclusion or a patch? I quite do not understand
what you are trying to say with all those examples.
Regards
--
Héctor Orón -.. . -... .. .- -. -.. . ...- . .-.. --- .--. . .-.
Acknowledgement sent
to tomazzi <[email protected]>:
Extra info received and forwarded to list. Copy sent to Héctor Orón Martínez <[email protected]>.
(Wed, 15 Nov 2023 22:09:03 GMT) (full text, mbox, link).
Subject: Re: Bug#1055646: gdb: extremely slow response to basic commands
Date: Wed, 15 Nov 2023 23:05:57 +0100
Hello Héctor,
Well, I want to fix this bug, because gdb is a very important tool for
me, and compiling gdb from ./configure script is just a hack - not a
real solution (which would eliminate the problem for good).
To do this, I need to have a complete build log with all the
compiler/linker flags, compiler warnings, etc.
Unfortunately, the methods described on the official Debian Wiki page
does not allow to build gdb from source package -> "debuild" fails to
produce the binary shipped by Debian. (I would say that this deserves a
separate BUG report).
Now, the "hacked" gdb is also different - so the first step is to find
differences in the build process, like the exact list of features
enabled/disabled.
So, can You please explain how did You build the gdb?
Regards.
On Mon, 13 Nov 2023 22:20:28 +0100
=?UTF-8?B?SMOpY3RvciBPcsOzbiBNYXJ0w61uZXo=?= <[email protected]> wrote:
> Hello Tomazzi,
>
> On Mon, 13 Nov 2023 at 21:51, tomazzi <[email protected]> wrote:
> >
> > Hello,
> >
> > I've just built gdb "the Debian way", to check what are the differences
> > in the build process:
> > $> apt source gdb
> > $> apt build-dep gdb
> > $> debuild -b -uc -us
> >
> > From debuild output:
> >
> > gdb-default: configured with:
> > ...
> > --without-babeltrace
> > --with-babeltrace
> > ...
> > gdb-minimal: configured with:
> > ...
> > --without-babeltrace
> > --with-babeltrace
> > --without-babeltrace
> > ...
> >
> > This looks bad, but the results are even more "interresting":
>
> Latest switch is the one that should be in-use
>
> > The gdb binary installed in the system (**debsums OK**) reports the
> > following configuration (the "show configuration" command, differences
> > only):
> >
> > ...
> > --with-babeltrace
> > --with-mpfr
> > --with-xxhash
> > --with-python=/usr (relocatable)
> > --with-python-libdir=/usr/lib (relocatable)
> > --enable-source-highlight
> > ...
> >
> > The newly "debuild" gdb reports something *different*:
> > ...
> > --without-babeltrace << this: conflicting build flags
> > --without-mpfr
> > --with-xxhash
> > --without-python
> > --without-python-libdir
> > --disable-source-highlight
> > ...
> >
> >
> > The gdb which I've compiled using ./configure script (so *without*
> > Debian patches) reports:
> > ...
> > --with-babeltrace
> > --with-mpfr
> > --without-xxhash << this: libxxhash-dev is installed, but
Acknowledgement sent
to Héctor Orón Martínez <[email protected]>:
Extra info received and forwarded to list. Copy sent to Héctor Orón Martínez <[email protected]>.
(Tue, 21 Nov 2023 12:57:05 GMT) (full text, mbox, link).
Subject: Re: Bug#1055646: gdb: extremely slow response to basic commands
Date: Tue, 21 Nov 2023 13:53:00 +0100
Hello,
On Wed, 15 Nov 2023 at 23:09, tomazzi <[email protected]> wrote:
> To do this, I need to have a complete build log with all the
> compiler/linker flags, compiler warnings, etc.
Find logs at https://buildd.debian.org/status/logs.php?pkg=gdb&arch=amd64
> Unfortunately, the methods described on the official Debian Wiki page
> does not allow to build gdb from source package -> "debuild" fails to
> produce the binary shipped by Debian. (I would say that this deserves a
> separate BUG report).
I am unsure about that, usually you can build using dpkg-buildpackage.
> Now, the "hacked" gdb is also different - so the first step is to find
> differences in the build process, like the exact list of features
> enabled/disabled.
>
> So, can You please explain how did You build the gdb?
Build daemons build it, see logs posted previously. Build daemons use sbuild.
--
Héctor Orón -.. . -... .. .- -. -.. . ...- . .-.. --- .--. . .-.
Acknowledgement sent
to tomazzi <[email protected]>:
Extra info received and forwarded to list. Copy sent to Héctor Orón Martínez <[email protected]>.
(Thu, 23 Nov 2023 11:03:05 GMT) (full text, mbox, link).
Subject: Re: Bug#1055646: gdb: extremely slow response to basic commands
Date: Thu, 23 Nov 2023 11:59:32 +0100
Hello,
On Tue, 21 Nov 2023 13:53:00 +0100
=?UTF-8?B?SMOpY3RvciBPcsOzbiBNYXJ0w61uZXo=?= <[email protected]> wrote:
> Hello,
>
> On Wed, 15 Nov 2023 at 23:09, tomazzi <[email protected]> wrote:
>
> > To do this, I need to have a complete build log with all the
> > compiler/linker flags, compiler warnings, etc.
>
> Find logs at https://buildd.debian.org/status/logs.php?pkg=gdb&arch=amd64
Many thanks for the link - I have the log, but I need some time to
analyze it.
> > Unfortunately, the methods described on the official Debian Wiki page
> > does not allow to build gdb from source package -> "debuild" fails to
> > produce the binary shipped by Debian. (I would say that this deserves a
> > separate BUG report).
>
> I am unsure about that, usually you can build using dpkg-buildpackage.
The gdb compiled with:
$> dpkg-buildpackage --build=binary --host-type=x86_64-linux-gnu -us -uc
reports configuration identical with the installed gdb, but the files
are of different size: installed 9.9MiB, compiled 9.3MiB. The newly
build gdb is also very slow.
I've also compiled the "hacked" gdb with Debian patches applied, and
it's fast - so no regression here.
Didn't tried sbuild yet.
Regards
Acknowledgement sent
to tomazzi <[email protected]>:
Extra info received and forwarded to list. Copy sent to Héctor Orón Martínez <[email protected]>.
(Sun, 26 Nov 2023 18:45:05 GMT) (full text, mbox, link).
Subject: Re: Bug#1055646: gdb: extremely slow response to basic commands
Date: Sun, 26 Nov 2023 19:39:53 +0100
Hello,
I have found the root cause of this problem:
It doesn't matter if the gdb is compiled with sbuild, debuild or
dpkg-buildpackage. It's slow, because it reads BFDs from a *hidden*
directory "/usr/lib/debug/.build-id/"
The gdb compiled using "configure" script had different prefix, so its
"relocatable" paths were pointing to an empty "debug" directory - that's
why it was "fast".
But, this is not the end of problems:
When the gdb is started, it reads symbols from target program and from
libraries which are loaded by the target program. In my test case
(2.5MiB executable with debug symbols included + wxWidgets library with
dbgsym package installed) the symbol table size is ~8MiB, and the gdb
process is using ~50MiB.
This is sufficient to debug the program, but after issuing the "run"
command, the BFDs from "/usr/lib/debug/.build-id/" are read - and gdb
creates a Monster Symbol Table with size over 200MiB (~280Mib used in
total).
Then things are getting only worse - any gdb command for displaying type
or value of target program' variables causes the the memory consumption
to double, in my case from ~280MiB to ~570MiB and the response times
become deadly slow.
But, this is not the end of problems: (2)
In all test cases the gdb was compiled with --enable-threading - and
indeed, gdb creates (num_CPU +1) threads on startup (in my case it's 17
for 16-core CPU). But the threading support in gdb is an illusion -
those additional threads are *never* used, with a single exception -
they are used only after the "run" command is issued, probably to read
the BFDs. For any other gdb command only the main thread is consuming
CPU time.
I thought that this BUG should be easy to eliminate - but it turns out
that there are several serious BUGs in the gdb, and the only way to
eliminate them is to debug the gdb first ;)
Regards
Acknowledgement sent
to Héctor Orón Martínez <[email protected]>:
Extra info received and forwarded to list. Copy sent to Héctor Orón Martínez <[email protected]>.
(Tue, 28 Nov 2023 12:39:02 GMT) (full text, mbox, link).
Subject: Re: Bug#1055646: gdb: extremely slow response to basic commands
Date: Tue, 28 Nov 2023 13:37:40 +0100
Hello tomazzi,
On Sun, 26 Nov 2023 at 19:45, tomazzi <[email protected]> wrote:
> I have found the root cause of this problem:
Thanks very much! This is a very valuable investigation.
I wanted to drop a link to addr2line bug[1], could that be related?
[1] https://sourceware.org/bugzilla/show_bug.cgi?id=29785
> I thought that this BUG should be easy to eliminate - but it turns out
> that there are several serious BUGs in the gdb, and the only way to
> eliminate them is to debug the gdb first ;)
Which several serious bugs? I understand you mean in the upstream GNU
GDB, instead of the Debian GNU GDB package. Are those reported
upstream?
Regards
--
Héctor Orón -.. . -... .. .- -. -.. . ...- . .-.. --- .--. . .-.
Acknowledgement sent
to tomazzi <[email protected]>:
Extra info received and forwarded to list. Copy sent to Héctor Orón Martínez <[email protected]>.
(Tue, 28 Nov 2023 16:30:02 GMT) (full text, mbox, link).
Subject: Re: Bug#1055646: gdb: extremely slow response to basic commands
Date: Tue, 28 Nov 2023 17:27:25 +0100
Hello Héctor,
On Tue, 28 Nov 2023 13:37:40 +0100
=?UTF-8?B?SMOpY3RvciBPcsOzbiBNYXJ0w61uZXo=?= <[email protected]> wrote:
> I wanted to drop a link to addr2line bug[1], could that be related?
>
> [1] https://sourceware.org/bugzilla/show_bug.cgi?id=29785
Thanks for this link.
It looks like it can be related, but I can't confirm this at the moment.
> > I thought that this BUG should be easy to eliminate - but it turns out
> > that there are several serious BUGs in the gdb, and the only way to
> > eliminate them is to debug the gdb first ;)
>
> Which several serious bugs? I understand you mean in the upstream GNU
> GDB, instead of the Debian GNU GDB package. Are those reported
> upstream?
Well, I think that at least 2 problems are obvious:
- huge memory consumption (can be related to the upstream bug or just a
memory leak)
- creation of useless threads: I would expect that "enable-threading"
really enables SMP for searching symbol tables, memory ranges, etc.
I haven't reported this yet, because first I want to fully understand
whats going on.
And yes, I mean upstream GNU GDB bugs.
Regards.