|
|
Subscribe / Log in / New account

5.1 Merge window part 1

By Jonathan Corbet
March 8, 2019
As of this writing, 6,135 non-merge changesets have been pulled into the mainline repository for the 5.1 release. That is approximately halfway through the expected merge-window volume, which is a good time for a summary. A number of important new features have been merged for this release; read on for the details.

Core kernel

  • Support for the ancient a.out executable format has been deprecated, with an eye toward removing it entirely later this year if nobody objects.
  • The BPF verifier has gained the ability to detect and remove dead code from BPF programs.
  • The BPF spinlock patches have been merged, giving BPF programs increased control over concurrency.
  • There is a new prctl() option for controlling speculative execution; PR_SPEC_DISABLE_NOEXEC disables enough speculation to block the speculative store bypass vulnerability, but only until the process calls exec().
  • A whole new set of system calls using 64-bit time values has been added; they are part of the kernel's year-2038 preparation. See this commit for the full list.
  • A new sysctl knob (kernel/sched_energy_aware) can be used to disable energy-aware scheduling. This feature is already disabled on most platforms, but by default it will be turned on for asymmetric systems where it makes sense. There are also two new documents on energy-aware scheduling and the energy model framework.
  • There is a new F_SEAL_FUTURE_WRITE operation for memfd regions that allows the caller to continue to write to the region but prevents any others from doing so. This feature supports an Android use case; see this commit for some more information.
  • The timer-events oriented CPU-frequency governor has been merged; it should yield better power-saving results on systems with tickless scheduling enabled.

Filesystems

  • The Btrfs filesystem now supports the use of multiple ZSTD compression levels. See this commit for some information about the feature and the performance characteristics of the various levels.

Hardware support

  • Audio: Cirrus Logic CS4341 and CS35L36 codecs, Rockchip RK3328 audio codecs, NXP pulse density modulation microphone interfaces, Mediatek MT8183 audio platforms, MediaTek MT6358 codecs, Qualcomm WCD9335 codecs, and Ingenic JZ4725B codecs.
  • Industrial I/O: Sensirion SPS30 particulate matter sensors, Plantower PMS7003 particulate matter sensors, Texas Instruments ADS124S08 analog-to-digital converters, Analog Devices ad7768-1 analog-to-digital converters, Analog Devices AD5674R/AD5679R digital-to-analog converters, STMicroelectronics LPS22HH pressure sensors, Nuvoton NPCM analog-to-digital converters, Invensense ICM-20602 motion trackers, Maxim MAX44009 ambient light sensors, and Texas Instruments DAC7612 digital-to-analog converters.
  • Miscellaneous: STMicroelectronics FMC2 NAND controllers, Amlogic Meson NAND flash controllers, ROHM BD70528 power regulators, Maxim MAX77650/77651 power regulators, Loongson-1 interrupt controllers, Broadcom STB reset controllers, OP-TEE-based random number generators, Habana AI accelerators, Mediatek GNSS receivers, and NVIDIA Tegra combined UARTs.
  • Networking: NXP ENETC gigabit Ethernet controller physical and virtual devices, and MediaTek MT7603E (PCIe) and MT76x8 wireless interfaces.
  • SPI: Freescale quad SPI controllers, NXP Flex SPI controllers, and SiFive SPI controllers.
  • USB: Marvell A3700 comphy and UTMI PHYs and Cadence D-PHYs.

Networking

  • There is a new socket option called "SO_BINDTOIFINDEX": "It behaves similar to SO_BINDTODEVICE, but takes a network interface index as argument, rather than the network interface name.". See this changelog for more information.
  • As part of the "make WiFi fast" effort, the mac80211 layer has gained an awareness of airtime fairness and the ability to share the available airtime across stations.
  • The year-2038-safe socket timestamp options described in this article have been merged.
  • The new "devlink health" mechanism provides notifications when an interface device has problems. See this merge commit and this documentation patch for details.
  • The mac80211 layer now has support for multiple BSSIDs (MAC addresses, essentially) for wireless devices.

Internal kernel changes

  • The ancient get_ds() function, which originally retrieved the value of the %ds segment on x86 systems, has been removed; all in-tree callers (of which there are few) have been changed to simply use the KERNEL_DS constant instead.
  • Much of the API for dealing with atomic variables is now automatically generated from a set of descriptions; the intent is to provide better consistency across architectures and to make it easier to add instrumentation. This commit gives an overview of how it all works.
  • The definitions of the GFP_* memory-allocation flags have been changed to allow the most frequently used bits to be represented more compactly. According to this commit, that reduces the size of Arm kernels by about 0.1%.
  • Memory compaction has been reworked, resulting in significant improvements in success rates and CPU time required.
  • The new "on-chip interconnect API" provides a generic mechanism for the control of connections within complex peripheral devices; see Documentation/interconnect/interconnect.rst for details.

The 5.1 merge window will probably stay open through March 17, with the final 5.1 release due at the beginning of May. As always, we'll catch up with the rest of the merge-window activity once it has closed; stay tuned.

Index entries for this article
KernelReleases/5.1


to post comments

5.1 Merge window part 1

Posted Mar 8, 2019 19:24 UTC (Fri) by johill (subscriber, #25196) [Link] (1 responses)

For the record, you wrote:

  • The mac80211 layer now has support for multiple BSSIDs (MAC addresses, essentially) for wireless devices.

This is rather misleading. Depending on how you look at it and what exactly you're thinking of, this either makes no sense or has been supported all along (depending on driver support).

Rather, what we did add was support for the "multi-BSSID" part of the spec, and there only scanning for and connecting to them. It's been defined for a while but is now seeing uptake in the upcoming HE specification (802.11ax). What "multi-BSSID" does isn't add support for multiple BSSIDs, but rather it adds support for an AP to have multiple BSSIDs without sending multiple beacons. Until now, if an AP wanted to advertise multiple networks, it would use multiple BSSIDs but then would have to send a beacon for each one. The "multi-BSSID" extension allows it to send a single beacon which inside the data advertises that in fact the AP has a few more BSSIDs, along with their SSID (network name) and other capabilities. This optimises air time usage because sending a lot of very similar beacons is a waste thereof.

But in a sense even that is only half the story because so far we only added support for the client-side, i.e. a mac80211-based driver connecting to such an AP. The driver also still has to support it separately due to changes in the TIM Element used for powersave; changes to iwlwifi will be coming to do so, though those are mostly in firmware with a little driver pass-through support.

What's still missing, unfortunately, is the AP-side support for this, which requires having multiple AP interfaces (netdevs) but linking them together in a suitable way that not each one will beacon. How we do that is TBD, perhaps we'll hash out a solution at netdev 0x13 in ~2 weeks.

5.1 Merge window part 1

Posted Mar 18, 2019 19:40 UTC (Mon) by jg (guest, #17537) [Link]

The MAC 802.11 airtime fairness work is important for people to pick up on. Drivers need to be updated to use it, so initially, it depends what wifi chip you have as to whether you will see benefit.

The feature not only ensures that you get a "fair share" of available WiFi bandwidth, but this also has the effect of using the available bandwidth much more efficiently (multiple stations end up able to get more total bandwidth), so it helps busy networks.

See:
https://www.usenix.org/system/files/conference/atc17/atc1...

5.1 Merge window part 1

Posted Mar 9, 2019 10:28 UTC (Sat) by hrw (subscriber, #44826) [Link]

https://fedora.juszkiewicz.com.pl/syscalls.html got updated to use latest syscall data.

Firoz Khan did great job with moving system calls lists from header files into plain text. Looks like all architectures got recent syscall updates at same time.

5.1 Merge window part 1

Posted Mar 12, 2019 14:34 UTC (Tue) by anton (subscriber, #25547) [Link] (23 responses)

Support for the ancient a.out executable format has been deprecated, with an eye toward removing it entirely later this year if nobody objects.
Where do I register my objections?

Just today I found that my system can run my good old QMAGIC binaries with a little help from me (I have yet to find out how to do it as non-root, and how to run ZMAGIC binaries). I use this to compare the old stuff to the new one, both in features and in performance, and want to continue to do so in the next decades.

Couldn't the same be achieved by always recompiling from the source? No, for several reasons: 1) One of the things I compare is what the compiler produced. 2) Unlike the Linux kernel maintainers, compiler maintainers usually do not give a guarantee not to break existing code, and consequently, the chances of 20+-year old source code compiling on a current compiler are not that great.

Interestingly, my recent investigation of a.out binaries was in reaction to a posting containing the claim:

So, [Windows has] a window of ~ 25 years where binaries still typically work, vs Linux being "maybe a few years, and only on a single distro".
Do the Linux kernel maintainers want to prove that poster right? And that despite giving the guarantee not to break existing code? Of course, binary compatibility problems are not just a kernel issue, but kernel maintainers should do their part in maintaining binary compatibility (as should others).

5.1 Merge window part 1

Posted Mar 12, 2019 16:35 UTC (Tue) by willy (subscriber, #9762) [Link] (2 responses)

I was the one who proposed it be removed entirely.

Continuing to support the a.out format has risks. The initial proposal to remove the a.out coredump support was due to bugs being found in it. Given the amount of testing the a.out code receives as a whole, I have little doubt that a determined attacker could find a security hole to exploit in it.

Since you appear to be a programmer yourself, how would you like to take up Alan Cox's suggestion of writing an a.out loader entirely in userspace?

5.1 Merge window part 1

Posted Mar 13, 2019 18:16 UTC (Wed) by anton (subscriber, #25547) [Link] (1 responses)

Unfortunately, I do not have the time to pursue such a project. I can provide binaries and libraries for testing, if someone wants to do it, though.

As for the security excuse: 1) You don't even have any proof of such a vulnerability; is your belief enough justification to break existing code? 2) Even if there was such a vulnerability, the attacker would have to break into my system first before exploiting it. 3) Those who are nervous about such a hypothetical vulnerability can disable the module.

5.1 Merge window part 1

Posted Mar 22, 2019 8:15 UTC (Fri) by flussence (guest, #85566) [Link]

If you can hold on to 20 year old binaries then surely you can afford to keep a compatible disk image and qemu around?

5.1 Merge window part 1

Posted Mar 14, 2019 13:51 UTC (Thu) by nilsmeyer (guest, #122604) [Link] (2 responses)

> Do the Linux kernel maintainers want to prove that poster right? And that despite giving the guarantee not to break existing code? Of course, binary compatibility problems are not just a kernel issue, but kernel maintainers should do their part in maintaining binary compatibility (as should others).

If that guarantee is broken you can of course demand your money back - you see where I'm going with this?

5.1 Merge window part 1

Posted Mar 19, 2019 8:54 UTC (Tue) by anton (subscriber, #25547) [Link] (1 responses)

Looks to me like you want to insult the Linux kernel maintainers by insinuating that their guarantee has no value.

5.1 Merge window part 1

Posted Mar 19, 2019 13:39 UTC (Tue) by nilsmeyer (guest, #122604) [Link]

You're the one who expects others to maintain your pet feature for free, indefinitely. I find that unreasonable.

5.1 Merge window part 1

Posted Mar 18, 2019 4:37 UTC (Mon) by dvdeug (subscriber, #10998) [Link] (13 responses)

ELF support came to Linux in 1995 (e.g. Slackware 3.0), the same year as Windows 95 (the first 32-bit consumer Windows) was released. And modern Windows doesn't support 16-bit programs, so Linux without a.out support offers support as far back as Windows does, about 24 years. It's a lot easier to run an old Linux in a VM, but there's more support for emulating old Windows programs.

In any case, proprietary code has been rare on Linux, and definitely pre-1995. Besides bragging rights, what do you gain from having a.out support in Linux?

5.1 Merge window part 1

Posted Mar 18, 2019 16:55 UTC (Mon) by deater (subscriber, #11746) [Link] (2 responses)

> In any case, proprietary code has been rare on Linux, and
> definitely pre-1995.

Not necessarily true. A lot of software back then depended on the Motif gui library, so even nominally free software ended up being statically linked against that.

I know I had an a.out "xmcd" binary that I used for many many years after the ELF transition, though I do admit I no longer run a.out files on a regular basis (though I do regularly run binary-only software written in the 1980s, just not on Linux).

5.1 Merge window part 1

Posted Mar 19, 2019 8:33 UTC (Tue) by dvdeug (subscriber, #10998) [Link] (1 responses)

Wow, https://www.amb.org/xmcd/ is still up and that page is a blast from the past. Last change was in 2004, so you might be able to compile a new version, but I don't know how well it would work with PulseAudio.

Hopefully current virtual machines would handle the need for running Motif-linked binaries. It seems unlikely a GUI program would need the blistering speed of native hardware, and the more complex library-wise you get, the more headaches you get trying to install that on a modern system.

5.1 Merge window part 1

Posted Mar 26, 2019 13:12 UTC (Tue) by nix (subscriber, #2304) [Link]

The painful part might be actually accessing a CD drive from inside the VM. Even with various forms of USB and ATA passthrough, in my experience it's still a toss-up whether that sort of thing works or causes an obscure and ridiculous error or even crashes the host (not with USB, but ATA/PCI passthrough is evil).

5.1 Merge window part 1

Posted Mar 19, 2019 9:32 UTC (Tue) by anton (subscriber, #25547) [Link] (9 responses)

32-bit Windows programs were supported since Windows NT (release 1993), and Win32s for Windows 3.1 (beta 1992). And Windows 10 (the most modern Windows) actually does support 16-bit applications in its 32-bit variant. Anyway, the issue is about running 32-bit binaries, just a different format.

I have 5 OMAGIC, 9 ZMAGIC, and 14 QMAGIC binaries in my /usr/local/bin. One of the QMAGIC binaries is from a program that I have in source but that does not compile with current gcc (unfortunately, gcc maintainers like to break existing programs in newer gcc versions; of course they justify it by claiming that the programs were broken before, but that does not change the fact that they don't compile programs that they used to compile in earlier versions). I have not checked the others whether they can be compiled.

So what do I gain? I can run old binaries, without needing to recompile them, which may not be possible, and in any case, requires some effort.

There is value in being able to run old software: We can better study and compare it if we can run it. And the basis of all that is a willingness to support it.

If the Linux kernel maintainers retract their support for old software, why should those who you point to as alternative not retract support, too. E.g., you claim we can run old Linux kernels in a VM, and old binaries in that. But why should VM maintainers support old Linux kernels? Do you guarantee that they won't change their VM a few years down the road in a way that supports only modern kernels that don't have a.out support if you get your way?

5.1 Merge window part 1

Posted Mar 19, 2019 10:00 UTC (Tue) by edeloget (subscriber, #88392) [Link] (4 responses)

> But why should VM maintainers support old Linux kernels?
> Do you guarantee that they won't change their VM a few years
> down the road in a way that supports only modern kernels
> that don't have a.out support if you get your way?

Wouldn't that be antithetical to the purpose of a virtual machine, which is to provide a kind of emulation layer for the hardware? A VM that can only run a selected list of OS is of no use, and they cannot assume what you'll feed to them.

5.1 Merge window part 1

Posted Mar 19, 2019 12:44 UTC (Tue) by anton (subscriber, #25547) [Link] (3 responses)

I am not an expert on the business of VMs, but I imagine that, just like some here don't consider it the purpose of the kernel to run old binaries, VM maintainers might also consider it not their purpose to run old OSs: They could (and probably do) decide to emulate hardware that is too recent for OSs above a certain age.

5.1 Merge window part 1

Posted Mar 19, 2019 19:08 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

Nobody stops you from using an old version of QEMU. It should be fine for the next 25-30 years or so - before Linux developers remove ELF in favor of something newer like TROLL.

5.1 Merge window part 1

Posted Mar 21, 2019 21:47 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

I feel like options such as ENT (executable notation table) or HOBBIT (handy object bundles built in tandem) would be better.

5.1 Merge window part 1

Posted Mar 19, 2019 23:41 UTC (Tue) by mpr22 (subscriber, #60784) [Link]

Given that SIMH exists, and includes emulations for computers that are older than the first operating systems, I am not overly worried on this score :)

5.1 Merge window part 1

Posted Mar 19, 2019 21:24 UTC (Tue) by dvdeug (subscriber, #10998) [Link] (3 responses)

I thought about Windows NT, but I forgot about Win32s. But by that standard, Linux wins hands-down; it can run applications written for Unix in 1977, provided they were written in Bourne Shell. The question is running the applications that people want to run on the modern systems they're running, and the last 16-bit Windows programs came out quite a bit after the last Linux a.out programs.

Most versions of Linux can't run ix86 binaries; only those compiled for ix86 or AMD64 can. Had Intel had their way, we'd be running Itanium without direct support for ix86 binaries. Perhaps in five or ten years, we'll be running ARM64 without direct support for ix86 or AMD64 binaries.

We can better study something in its home environment, not by running it on a modern system where various things might not work (audio, modern resolutions, USB).

Out of the box, Retropie supports the ZX-Spectrum, the TI-99, the MSX, the Atari ST, the Coco, the Colecovision, the Amstrad CPC, Amiga, SAM Coupé, the Macintosh, MS-DOS and the Commodore 64, along with dozens of other, usually more gaming focused, systems. It seems unlikely emulating x86, one of the most common computer hardware platforms of all time, will be a big problem for the foreseeable future.

You're demanding that Linux accept your solution for your obscure problem; there's not that many people who ran Linux before 1996, and many of those have given up all their old binaries from the time. If you were volunteering as a maintainer, you'd have more say in if this feature stays; if you were looking for solutions to a problem, people will be more willing to help. But to demand that a feature that doesn't fully work (i.e. core dumps were broken) be the only possible solution to your problem isn't helpful.

5.1 Merge window part 1

Posted Mar 21, 2019 17:57 UTC (Thu) by anton (subscriber, #25547) [Link] (2 responses)

Most versions of Linux can't run ix86 binaries; only those compiled for ix86 or AMD64 can.
That's beside the point. This is not about running binaries for one architecture on other architectures, this is about removing support for a binary format, i.e., a pure software issue.

[But anyway, the statement is wrong. IA-32 binaries could be run on Linux-Alpha. Someone reported running an ARM binary of Gforth on an Intel-based Android tablet; and I expect that Android also has emulation the other way round.]

We can better study something in its home environment, not by running it on a modern system where various things might not work (audio, modern resolutions, USB).
I study and compare things in many different environments. One thing that I study is how the various versions of the software perform on hardware, both on old hardware and new hardware. And to do that, I need to run the old software on new hardware, with (necessarily) new kernels. Retropie won't help me with that. And fortunately, audio, resolutions, and USB are no problems for the stuff I do.
You're demanding that Linux accept your solution for your obscure problem
I just object to removing a feature from the kernel that I use.
If you were volunteering as a maintainer, you'd have more say in if this feature stays
Did anybody ask for a maintainer? In any case, are you suggesting that, when Linus Torvalds says "Breaking user programs simply isn't acceptable", he means "Breaking maintainer programs simply is not acceptable, but breaking non-maintainer programs is"?
But to demand that a feature that doesn't fully work (i.e. core dumps were broken) be the only possible solution to your problem isn't helpful.
The feature works better now than with the proposed change; that's why I object to the change.

Someone suggested a userspace loader for a.out binaries. When that works with little overhead, it will be acceptable for me, too, but somebody has to write it, while the kernel support exists, and works better than the current state (non-existence) of the userspace loader.

5.1 Merge window part 1

Posted Mar 21, 2019 19:53 UTC (Thu) by excors (subscriber, #95769) [Link]

> Someone reported running an ARM binary of Gforth on an Intel-based Android tablet; and I expect that Android also has emulation the other way round.

Intel developed an ARM-to-x86 binary translator (Houdini) for their Android BSP, for compatibility with popular apps that contain ARM-only native code; I suspect that's why it worked. I doubt anyone has done it the other way round, because there are approximately zero Android apps with Intel-only native code.

5.1 Merge window part 1

Posted Mar 22, 2019 0:42 UTC (Fri) by dvdeug (subscriber, #10998) [Link]

Then Windows is moot? Cross-Linux compatibility versus Windows was certainly was part of the original discussion.

At this point, I question the relevance of any numbers you get comparing these a.out binaries to modern systems. You're comparing a program compiled for the original Pentium with GCC 2.7 (at the latest) and libc4 to a different program natively running on a recent version of AMD64 using modern versions of GCC and GNU libc. Where the faults lay for speedups and slowdowns is going to be very hard to tell. In any case, for the near future, you can run a Linux 5.0 kernel in an virtual machine if you want to compare things.

I'm not Linus, but I'm certainly more worried about some of the lost filesystem support making file recovery quite complex than I am about forcing a.out binaries to be run via qemu or a userspace loader, if anyone cares to write one. Supporting ancient binaries is much better and easier handled by emulators than trying to maintain a 25-year old environment mixed in with a modern one.

5.1 Merge window part 1

Posted Mar 21, 2019 12:56 UTC (Thu) by wojtekka (guest, #131045) [Link] (2 responses)

If Wine can load and run binaries built for a completely different operating system, how hard can it be to create an application that loads and runs a.out binaries in user-space? This and binfmt_misc should make things completely transparent.

5.1 Merge window part 1

Posted Mar 23, 2019 17:22 UTC (Sat) by Wol (subscriber, #4433) [Link] (1 responses)

If you're not a programmer, then getting Wine to run Windows binaries can actually be quite tricky ...

If you're a company with programmers on tap, then most things are *possible*.

Thing is, I'm a private individual with no real kernel experience (although my C was pretty decent), so getting old binaries to work is not *practical*.

That's what I feel is often missed here - what is *possible* is not always *practical*.

I'd love a guarantee like the IBM mainframes - I believe you can run "dawn of computing" mainframe software on the latest megabeasts, because the current system can emulate the previous system, which can emulate the one before that, which can emulate the one before that ... turtles all the way down. But that COSTS MONEY, which us little guys don't have, and most people don't seem to consider important enough to do as a project, be it commercial or private.

Cheers,
Wol

5.1 Merge window part 1

Posted Mar 26, 2019 10:27 UTC (Tue) by wojtekka (guest, #131045) [Link]

If there's a need for running a.out binaries without kernel support someone *will* come up with a solution. Huge part of the software I use everyday was merely *possible* at some point and most probably was not *practical* because there were already alternatives available. Just sayin'.


Copyright © 2019, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds