Bug 1210168 - [Build 20230404] firefox no longer existing on i586 because LLVM out of memory
Summary: [Build 20230404] firefox no longer existing on i586 because LLVM out of memory
Status: RESOLVED FIXED
Alias: None
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Development (show other bugs)
Version: Current
Hardware: i586 Other
: P5 - None : Normal (vote)
Target Milestone: ---
Assignee: Aaron Puchert
QA Contact: E-mail List
URL: https://openqa.opensuse.org/tests/321...
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-04-05 15:06 UTC by Dominique Leuenberger
Modified: 2023-10-13 07:34 UTC (History)
3 users (show)

See Also:
Found By: openQA
Services Priority:
Business Priority:
Blocker: Yes
Marketing QA Status: ---
IT Deployment: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Dominique Leuenberger 2023-04-05 15:06:52 UTC 
## Observation


The latest submission for MozillaFirefox has set ExcludeArch %ix86, which means FF is no longer built.

openQA test in scenario opensuse-Tumbleweed-LegacyX86-DVD-i586-gnome@32bit fails in
[firefox](https://openqa.opensuse.org/tests/3210979/modules/firefox/steps/16)

## Test suite description
Revert schedule to working settings - dleuenberger, 20211117


## Reproducible

Fails since (at least) Build [20230404](https://openqa.opensuse.org/tests/3210979) (current job)


## Expected result

Last good: [20230403](https://openqa.opensuse.org/tests/3208208) (or more recent)


## Further details

Always latest result in this scenario: [latest](https://openqa.opensuse.org/tests/latest?arch=i586&distri=opensuse&flavor=LegacyX86-DVD&machine=32bit&test=gnome&version=Tumbleweed)
Comment 1 Wolfgang Rosenauer 2023-04-05 17:44:39 UTC 
It has an excludearch because it fails to compile due to processing huge libxul.so.
I wasn't able to resolve the issue in FF yet.

I will add the compile error here in a bit. Probably others have ideas how to solve the issue.
Comment 2 Wolfgang Rosenauer 2023-04-05 21:01:26 UTC 
[ 1677s] 26:24.66 /usr/bin/g++ -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fno-sized-deallocation -fno-aligned-new -fomit-frame-pointer -O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -g -march=i686 -mtune=generic -msse2 -fimplicit-constexpr -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -freorder-blocks -O2 -fomit-frame-pointer -funwind-tables  -shared -Wl,-z,defs -Wl,-h,libxul.so -o libxul.so /home/abuild/rpmbuild/BUILD/obj/toolkit/library/build/libxul_so.list   -lpthread -Wl,--no-keep-memory -Wl,--reduce-memory-overheads -fPIC -Wl,-z,relro,-z,now -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,nocopyreloc -Wl,-Bsymbolic-functions -Wl,--build-id=sha1 -fstack-protector-strong -Wl,-rpath-link,/home/abuild/rpmbuild/BUILD/obj/dist/bin -Wl,-rpath-link,/usr/lib  ../../../js/src/build/libjs_static.a /home/abuild/rpmbuild/BUILD/obj/i686-unknown-linux-gnu/release/libgkrust.a ../../../security/sandbox/linux/libmozsandbox.so ../../../config/external/lgpllibs/liblgpllibs.so ../../../config/external/sqlite/libmozsqlite3.so ../../../widget/gtk/mozgtk/libmozgtk.so ../../../widget/gtk/mozwayland/libmozwayland.so -Wl,--version-script,symverscript  -lasound -lrt -lm -ldl -lX11 -lXcomposite -lXdamage -lXext -lXfixes -lXrandr -lXrender -lXtst -lpthread -lc -lplds4 -lplc4 -lnspr4 -lz -lssl3 -lsmime3 -lnss3 -lnssutil3 -lfreetype -lfontconfig -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -ldbus-glib-1 -ldbus-1 -lxcb-shm -lX11-xcb -lxcb -lXcursor -lXi -lproxy
[ 1798s] 28:25.57 /usr/lib/gcc/i586-suse-linux/13/../../../../i586-suse-linux/bin/ld: error: libxul.so(.debug_info) is too large (0x5a166fec bytes)
[ 1802s] 28:29.57 /home/abuild/rpmbuild/BUILD/obj/_virtualenvs/build/bin/python -m mozbuild.action.check_binary --target libxul.so
[ 1802s] 28:29.87 LLVM ERROR: out of memory
[ 1802s] 28:29.87 Allocation failed
[ 1802s] 28:29.88 PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
[ 1802s] 28:29.88 Stack dump:
[ 1802s] 28:29.88 0.	Program arguments: /usr/bin/llvm-readelf -d libxul.so
[ 1802s] 28:29.97  #0 0xf3abbee5 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/lib/libLLVM.so.16+0x3cbbee5)
[ 1802s] 28:29.97  #1 0xf3abc562 (/lib/libLLVM.so.16+0x3cbc562)
[ 1802s] 28:29.97  #2 0xf3ab95cc llvm::sys::RunSignalHandlers() (/lib/libLLVM.so.16+0x3cb95cc)
[ 1802s] 28:29.97  #3 0xf3abcd9d (/lib/libLLVM.so.16+0x3cbcd9d)
[ 1802s] 28:29.97  #4 0xf7fb9570 (linux-gate.so.1+0x570)
[ 1802s] 28:29.97  #5 0xf7fb9559 (linux-gate.so.1+0x559)
[ 1802s] 28:29.97  #6 0xef68ac57 __pthread_kill_implementation (/lib/libc.so.6+0x8ac57)
[ 1802s] 28:29.97  #7 0xef63a2e1 gsignal (/lib/libc.so.6+0x3a2e1)
[ 1802s] 28:29.97  #8 0xef622277 abort (/lib/libc.so.6+0x22277)
[ 1802s] 28:29.97  #9 0xf39d4acc llvm::report_bad_alloc_error(char const*, bool) (/lib/libLLVM.so.16+0x3bd4acc)
[ 1802s] 28:29.97 #10 0xf39d4b41 (/lib/libLLVM.so.16+0x3bd4b41)
[ 1802s] 28:29.97 #11 0xefa9b022 operator new(unsigned int) (/lib/libstdc++.so.6+0x9b022)
[ 1802s] 28:29.97 #12 0xefa9b04d operator new(unsigned int, std::nothrow_t const&) (/lib/libstdc++.so.6+0x9b04d)
[ 1802s] 28:29.97 #13 0xf3a22d85 llvm::WritableMemoryBuffer::getNewUninitMemBuffer(unsigned int, llvm::Twine const&, std::optional<llvm::Align>) (/lib/libLLVM.so.16+0x3c22d85)
[ 1802s] 28:29.97 #14 0xf3a234f5 (/lib/libLLVM.so.16+0x3c234f5)
[ 1802s] 28:29.97 #15 0xf3a22376 (/lib/libLLVM.so.16+0x3c22376)
[ 1802s] 28:29.97 #16 0xf3a2202d llvm::MemoryBuffer::getFileOrSTDIN(llvm::Twine const&, bool, bool, std::optional<llvm::Align>) (/lib/libLLVM.so.16+0x3c2202d)
[ 1802s] 28:29.97 #17 0x56835766 (/usr/bin/llvm-readelf+0x22c766)
[ 1802s] 28:29.97 #18 0x5685a7d0 (/usr/bin/llvm-readelf+0x2517d0)
[ 1802s] 28:29.97 #19 0xef623855 __libc_start_call_main (/lib/libc.so.6+0x23855)
[ 1802s] 28:29.97 #20 0xef623918 __libc_start_main@GLIBC_2.0 (/lib/libc.so.6+0x23918)
[ 1802s] 28:29.97 #21 0x566a9ba7 (/usr/bin/llvm-readelf+0xa0ba7)
Comment 3 Wolfgang Rosenauer 2023-04-11 21:28:25 UTC 
Any LLVM specialist any hints?
Comment 4 Aaron Puchert 2023-04-15 16:19:32 UTC 
(In reply to Wolfgang Rosenauer from comment #2)
> [ 1798s] 28:25.57
> /usr/lib/gcc/i586-suse-linux/13/../../../../i586-suse-linux/bin/ld: error:
> libxul.so(.debug_info) is too large (0x5a166fec bytes)

Seems that the linker is already having trouble, but presumably still produces a shared library?

> [ 1802s] 28:29.57
> /home/abuild/rpmbuild/BUILD/obj/_virtualenvs/build/bin/python -m
> mozbuild.action.check_binary --target libxul.so
> [ 1802s] 28:29.87 LLVM ERROR: out of memory
> [ 1802s] 28:29.87 Allocation failed
> [ 1802s] 28:29.88 PLEASE submit a bug report to
> https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
> [ 1802s] 28:29.88 Stack dump:
> [ 1802s] 28:29.88 0.	Program arguments: /usr/bin/llvm-readelf -d libxul.so

At first it seems strange that such a simple request would run out of memory. However, this isn't the OOM killer, it's an allocation that fails. That is usually because we've exhausted virtual memory.

> [ 1802s] 28:29.97  #0 0xf3abbee5
> llvm::sys::PrintStackTrace(llvm::raw_ostream&, int)
> (/lib/libLLVM.so.16+0x3cbbee5)
> [ 1802s] 28:29.97  #1 0xf3abc562 (/lib/libLLVM.so.16+0x3cbc562)
> [ 1802s] 28:29.97  #2 0xf3ab95cc llvm::sys::RunSignalHandlers()
> (/lib/libLLVM.so.16+0x3cb95cc)
> [ 1802s] 28:29.97  #3 0xf3abcd9d (/lib/libLLVM.so.16+0x3cbcd9d)
> [ 1802s] 28:29.97  #4 0xf7fb9570 (linux-gate.so.1+0x570)
> [ 1802s] 28:29.97  #5 0xf7fb9559 (linux-gate.so.1+0x559)
> [ 1802s] 28:29.97  #6 0xef68ac57 __pthread_kill_implementation
> (/lib/libc.so.6+0x8ac57)
> [ 1802s] 28:29.97  #7 0xef63a2e1 gsignal (/lib/libc.so.6+0x3a2e1)
> [ 1802s] 28:29.97  #8 0xef622277 abort (/lib/libc.so.6+0x22277)
> [ 1802s] 28:29.97  #9 0xf39d4acc llvm::report_bad_alloc_error(char const*,
> bool) (/lib/libLLVM.so.16+0x3bd4acc)
> [ 1802s] 28:29.97 #10 0xf39d4b41 (/lib/libLLVM.so.16+0x3bd4b41)
> [ 1802s] 28:29.97 #11 0xefa9b022 operator new(unsigned int)
> (/lib/libstdc++.so.6+0x9b022)
> [ 1802s] 28:29.97 #12 0xefa9b04d operator new(unsigned int, std::nothrow_t
> const&) (/lib/libstdc++.so.6+0x9b04d)
> [ 1802s] 28:29.97 #13 0xf3a22d85
> llvm::WritableMemoryBuffer::getNewUninitMemBuffer(unsigned int, llvm::Twine
> const&, std::optional<llvm::Align>) (/lib/libLLVM.so.16+0x3c22d85)
> [ 1802s] 28:29.97 #14 0xf3a234f5 (/lib/libLLVM.so.16+0x3c234f5)
> [ 1802s] 28:29.97 #15 0xf3a22376 (/lib/libLLVM.so.16+0x3c22376)
> [ 1802s] 28:29.97 #16 0xf3a2202d
> llvm::MemoryBuffer::getFileOrSTDIN(llvm::Twine const&, bool, bool,
> std::optional<llvm::Align>) (/lib/libLLVM.so.16+0x3c2202d)
> [ 1802s] 28:29.97 #17 0x56835766 (/usr/bin/llvm-readelf+0x22c766)
> [ 1802s] 28:29.97 #18 0x5685a7d0 (/usr/bin/llvm-readelf+0x2517d0)
> [ 1802s] 28:29.97 #19 0xef623855 __libc_start_call_main
> (/lib/libc.so.6+0x23855)
> [ 1802s] 28:29.97 #20 0xef623918 __libc_start_main@GLIBC_2.0
> (/lib/libc.so.6+0x23918)
> [ 1802s] 28:29.97 #21 0x566a9ba7 (/usr/bin/llvm-readelf+0xa0ba7)

The call to getNewUninitMemBuffer should be in getOpenFileImpl [1]:

  if (shouldUseMmap(FD, FileSize, MapSize, Offset, RequiresNullTerminator,
                    PageSize, IsVolatile)) {
    std::error_code EC;
    std::unique_ptr<MB> Result(
        new (NamedBufferAlloc(Filename)) MemoryBufferMMapFile<MB>(
            RequiresNullTerminator, FD, MapSize, Offset, EC));
    if (!EC)
      return std::move(Result);
  }

  auto Buf =
      WritableMemoryBuffer::getNewUninitMemBuffer(MapSize, Filename, Alignment);

My guess is that the mmap fails because of the virtual memory limit (2G in user space on i586?), and then we try to allocate the same amount of memory directly, which also fails. Based on what the linker says, the debug info alone is well over 1G, so we could reasonably run out of virtual memory space, especially considering that we need that amount contiguously.

I see two solutions:
* Try to use binutils' readelf instead of llvm-readelf. Maybe that's a bit more careful with allocating virtual memory.
* With LLVM itself we had similar issues in the past. We build all 32-bit architectures (%{arm} ppc %{ix86}) with -g1 instead of -g. That's often good enough to get symbols for a crash, and better than not having the package at all.

[1] https://github.com/llvm/llvm-project/blob/llvmorg-16.0.1/llvm/lib/Support/MemoryBuffer.cpp#L446-L516
Comment 5 Aaron Puchert 2023-04-15 16:25:51 UTC 
(In reply to Aaron Puchert from comment #4)
> * With LLVM itself we had similar issues in the past. We build all 32-bit
> architectures (%{arm} ppc %{ix86}) with -g1 instead of -g. That's often good
> enough to get symbols for a crash, and better than not having the package at
> all.

Since LLVM 4 actually, when the libraries where still rather small compared to today:

-------------------------------------------------------------------
Wed Aug 30 12:08:17 UTC 2017 - msrb@suse.com

[...]
- Reduce debuginfo on x86 architecture. LLVM libraries are so big that they
  exhaust all memory on 32 bit machine if linked with full debuginfo.
Comment 6 Dominique Leuenberger 2023-05-09 11:35:55 UTC 
https://build.opensuse.org/request/show/1085402 re-enabled Firefox for i586 => FIXED
Comment 9 Maintenance Automation 2023-09-15 08:30:13 UTC 
SUSE-SU-2023:3610-1: An update that solves one vulnerability and has two security fixes can now be installed.

Category: security (critical)
Bug References: 1210168, 1215231, 1215245
CVE References: CVE-2023-4863
Sources used:
openSUSE Leap 15.4 (src): MozillaFirefox-115.2.1-150200.152.105.1
openSUSE Leap 15.5 (src): MozillaFirefox-115.2.1-150200.152.105.1
Desktop Applications Module 15-SP4 (src): MozillaFirefox-115.2.1-150200.152.105.1
Desktop Applications Module 15-SP5 (src): MozillaFirefox-115.2.1-150200.152.105.1
SUSE Linux Enterprise High Performance Computing 15 SP2 LTSS 15-SP2 (src): MozillaFirefox-115.2.1-150200.152.105.1
SUSE Linux Enterprise High Performance Computing ESPOS 15 SP3 (src): MozillaFirefox-115.2.1-150200.152.105.1
SUSE Linux Enterprise High Performance Computing LTSS 15 SP3 (src): MozillaFirefox-115.2.1-150200.152.105.1
SUSE Linux Enterprise Server 15 SP2 LTSS 15-SP2 (src): MozillaFirefox-115.2.1-150200.152.105.1
SUSE Linux Enterprise Server 15 SP3 LTSS 15-SP3 (src): MozillaFirefox-115.2.1-150200.152.105.1
SUSE Linux Enterprise Server for SAP Applications 15 SP2 (src): MozillaFirefox-115.2.1-150200.152.105.1
SUSE Linux Enterprise Server for SAP Applications 15 SP3 (src): MozillaFirefox-115.2.1-150200.152.105.1
SUSE Enterprise Storage 7.1 (src): MozillaFirefox-115.2.1-150200.152.105.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 10 Maintenance Automation 2023-09-15 08:30:18 UTC 
SUSE-SU-2023:3609-1: An update that solves one vulnerability and has two security fixes can now be installed.

Category: security (critical)
Bug References: 1210168, 1215231, 1215245
CVE References: CVE-2023-4863
Sources used:
SUSE Linux Enterprise High Performance Computing 15 SP1 LTSS 15-SP1 (src): MozillaFirefox-115.2.1-150000.150.103.1
SUSE Linux Enterprise Server 15 SP1 LTSS 15-SP1 (src): MozillaFirefox-115.2.1-150000.150.103.1
SUSE Linux Enterprise Server for SAP Applications 15 SP1 (src): MozillaFirefox-115.2.1-150000.150.103.1
SUSE CaaS Platform 4.0 (src): MozillaFirefox-115.2.1-150000.150.103.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 11 Maintenance Automation 2023-09-15 16:30:01 UTC 
SUSE-SU-2023:3626-1: An update that solves one vulnerability and has two security fixes can now be installed.

Category: security (critical)
Bug References: 1210168, 1215231, 1215245
CVE References: CVE-2023-4863
Sources used:
SUSE Linux Enterprise Software Development Kit 12 SP5 (src): MozillaFirefox-115.2.1-112.179.1
SUSE Linux Enterprise High Performance Computing 12 SP5 (src): MozillaFirefox-115.2.1-112.179.1
SUSE Linux Enterprise Server 12 SP5 (src): MozillaFirefox-115.2.1-112.179.1
SUSE Linux Enterprise Server for SAP Applications 12 SP5 (src): MozillaFirefox-115.2.1-112.179.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 13 Maintenance Automation 2023-10-09 20:28:54 UTC 
SUSE-SU-2023:4016-1: An update that solves six vulnerabilities can now be installed.

Category: security (critical)
Bug References: 1210168, 1215309, 1215575, 1215814
CVE References: CVE-2023-5168, CVE-2023-5169, CVE-2023-5171, CVE-2023-5174, CVE-2023-5176, CVE-2023-5217
Sources used:
openSUSE Leap 15.4 (src): MozillaThunderbird-115.3.1-150200.8.133.1
openSUSE Leap 15.5 (src): MozillaThunderbird-115.3.1-150200.8.133.1
SUSE Package Hub 15 15-SP4 (src): MozillaThunderbird-115.3.1-150200.8.133.1
SUSE Package Hub 15 15-SP5 (src): MozillaThunderbird-115.3.1-150200.8.133.1
SUSE Linux Enterprise Workstation Extension 15 SP4 (src): MozillaThunderbird-115.3.1-150200.8.133.1
SUSE Linux Enterprise Workstation Extension 15 SP5 (src): MozillaThunderbird-115.3.1-150200.8.133.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.