Tracking Issue for feature(unix_socket_ancillary_data)

[LinkTed]

Tracking issue for feature(unix_socket_ancillary_data) to extend UnixStream and UnixDatagram to send and receive ancillary data. For example file descriptors.

Unresolved Questions

• Review the SocketAncillary struct before stabilization to ensure it is compatible with all OSes.
• fuchsia, haiku, illumos, ios, macos and solaris does not have MSG_CMSG_CLOEXEC constant in libc
• fuchsia and uclibc(x86_64) does not have cmsghdr struct in libc
• The current version of rust libc does not have ucred struct for the target OS emscripten, but emscripten has this struct in the standard C library

Implementation history

PR #69864, #82589, #83374

[tadeokondrak]

send_vectored_with_ancillary should switch to non-mutable references for its parameters, since sendmsg doesn't write through them.

#79753 changes just the bufs parameter, but it was closed for inactivity.

(Edit: Hmm, actually changing the ancillary data too would require both a SocketAncillary and SocketAncillaryMut, like IoSlice. Probably not worth it there)

[LinkTed]

@tadeokondrak I open a new PR #82589.

[LinkTed]

I create a new PR to send and receive TTL for UdpSocket from the ancillary interface. #82858

[reyk]

Hi @LinkTed, I posted a fix in #83374 for macOS, OpenBSD, and possibly other BSDs.

[LinkTed]

What has to be done to stabilize this API?

[the8472]

I think the API still needs polish. All those niche socket features are tricky and right now it's probably not extensible enough for 3rd party libraries to fill platform-specific gaps such as flags or cmsgs only supported on one particular OS. These basically are wrappers for the recvmsg and sndmsg syscalls, we should expose them in a way that's forwards-compatible so we don't have to add recv_vectored_with_ancillary_from_with_flags or something like that in the future.

Before stabilizing we should consider whether recv_vectored_with_ancillary_from should gain another argument. I would recommend adding a ReceiveOptions argument that allows one to specify flags, similar to OpenOptions. The options could then be used to set linux' MSG_DONTWAIT, MSG_PEEK or MSG_ERRQUEUE for example, maybe via custom_flags() similar to OpenOptions. That way we don't have to support all OS-specific peculiarities.

Or we could put the flags into SocketAncillary or at least reserve the right to do that in the future, but in that case the struct would need to be renamed because it would serve several purposes at once.

Similar could be done for sending.

Additionally the AncillaryData enum should be marked as #[non_exhaustive] and maybe gain a Unknown variant for things that the standard library doesn't support. And a way to get the raw ancillary message. That way one could for example decode struct sock_extended_err in a 3rd party crate without waiting for std to implement it.

[LinkTed]

@the8472 Thank you for your feedback. If this PR request is #82858 merged I will start implementing your suggestions.

[tv42]

What about CMSG_SPACE? Without it, we're forced to either guess or probe the appropriate buffer size.

https://man7.org/linux/man-pages/man3/cmsg.3.html SCM_RIGHTS example shows how it's used to compute the correct buffer size (with some tricky business wrt alignment).

[LinkTed]

Hi @tv42,
what are the difference between CMSG_SPACE and CMSG_LEN?

Is there a function, which calculate the alignment for a struct to a specific address in Rust?

[comex]

what are the difference between CMSG_SPACE and CMSG_LEN?

CMSG_SPACE includes alignment bytes, CMSG_LEN doesn't.

Is there a function, which calculate the alignment for a struct to a specific address in Rust?

I don't know what this means.

[LinkTed]

what are the difference between CMSG_SPACE and CMSG_LEN?

CMSG_SPACE includes alignment bytes, CMSG_LEN doesn't.

Is there a function, which calculate the alignment for a struct to a specific address in Rust?

I don't know what this means.

I thought that the alignment depends on the address, where you want to store the struct.

[zopsicle]

What it means for CMGS_SPACE to include “alignment bytes” is that it rounds the size up to include a number of padding bytes to make sure the next header is properly aligned (to the alignment of cmsghdr).

[LinkTed]

So, the address have to be a multiple of the return value of CMSG_ALIGN, right?

[zopsicle]

The address of the cmsghdr object has to be a multiple of std::mem::align_of::<cmsghdr>().

[LinkTed]

Sorry, that I ask again. Does the macro CMSG_ALIGN the same as std::mem::align_of::<cmsghdr>()?

[zopsicle]

CMSG_ALIGN(len) rounds len up to a multiple of std::mem::align_of::<cmsghdr>(). For instance, if the required alignment is 8, and len is 11, CMSG_ALIGN(len) will return 16.

As an example, consider the following two headers:

let A = cmsghdr{ cmsg_len = 19, cmsg_level = X, cmsg_type = Y };
let B = cmsghdr{ cmsg_len = 22, cmsg_level = W, cmsg_type = Z };

If cmsg_len were not rounded up, this sequence of headers would look like this in memory:

          ~~~~~~~~~A~~~~~~~~~ ~~~~~~~~~B~~~~~~~~~
         +----+----+----+----+----+----+----+----+
Address: |  0 |  8 | 12 | 16 | 19 | 27 | 31 | 35 |
         +----+----+----+----+----+----+----+----+
Data:    | 19 |  X |  Y |  _ | 22 |  W |  Z |  _ |
         +----+----+----+----+----+----+----+----+

Notice how header B is at improperly aligned address 19, which is not a multiple of the alignment of cmsghdr.

By rounding up the lengths (assuming an alignment of 8 for this example), we get this:

          ~~~~~~~~~A~~~~~~~~~      ~~~~~~~~~B~~~~~~~~~
         +----+----+----+----+----+----+----+----+----+----+
Address: |  0 |  8 | 12 | 16 | 19 | 24 | 32 | 36 | 40 | 46 |
         +----+----+----+----+----+----+----+----+----+----+
Data:    | 19 |  X |  Y |  _ |    | 22 |  W |  Z |  _ |    |
         +----+----+----+----+----+----+----+----+----+----+
                              ~~~~ padding bytes       ~~~~ padding bytes

Now, every header and each field in every header is at the aligned address it should be.

[LinkTed]

Thanks, @chloekek. Currently, the code is using CMSG_LEN, which includes the alignment bytes. So, technically we have to ensure that the first cmsg struct is properly aligned, right?

But, if cmsg_len is rounded up, this means that the receiver interpreted the alignment bytes as data. For example, if the sender set 3 bytes, which then is aligned to 8, the receiver will think that he got 8 bytes. How do we handle this?

[zopsicle]

Actually, the man page says that you should include the padding in cmsg_len:

CMSG_LEN() returns the value to store in the cmsg_len member of the cmsghdr structure

This means that the receiver will indeed interpret those bytes as part of the data, I don’t think there’s any way around that?

This would mean that, if alignof(cmsghdr) = 8, and sizeof(int) = 4, you could only send an even number of file descriptors using SCM_RIGHTS. 😅 I guess if you want to send an odd number of file descriptors you could program the sender to set the odd one to -1 and program the receiver to ignore any -1s.

[LinkTed]

Interesting, currently there is a test, which send one FD and I tested on a 64-bit system, where the alignment of cmsghdr should be 8. 😀

[zopsicle]

What is suspicious is that CMSG_NXTHDR adds the alignment if necessary.

This could mean that either:

• CMSG_NXTHDR is just being cautious to avoid undefined behavior (dereferencing pointer to unaligned data is UB).
• You don’t actually have to include the padding in cmsg_len, despite what the man page suggests.

Note that while you could test this, other Unix implementations (e.g. BSD) may behave differently from Linux in this regard. It would be best to check the Official Spec™ (POSIX?).

As for file descriptors, the kernel will indeed divide the data length by the size of a file descriptor in order to compute how many file descriptors to send. And it seems that you indeed do have to set all of them to valid file descriptors, not to -1.

[LinkTed]

I think the man page is wrong. CMSG_LEN does not include the alignment. Only CMSG_SPACE as @comex suggested.

[zopsicle]

So to summarize:

• Set cmsg_len to the return value of CMSG_LEN. cmsg_len can be any value greater than sizeof(cmsghdr).
• Make sure that the next header is at a properly aligned address. CMSG_NXTHDR will round the address up for you, or you can use CMSG_ALIGN if you do the pointer arithmetic yourself.

[joshtriplett]

I do care about the state of this API, and I'm in particular quite interested in the fd-sending mechanism.

It's certainly a good candidate for using whatever new where Os: Unix mechanism or similar that we introduce in the future.

[jmillikin]

I'm interested in this feature and have time to work on it. However, the current API is very different from what I would expect:

• The current API for AncillaryData is closely tied with UNIX sockets and doesn't seem to have any support for ancillary data from (for example) TCP/IPv6 sockets.
• Putting an enum for possible SCM_* values into std::os::unix prevents use of OS-specific values such as Linux's SCM_SECURITY.

How much flexibility is available for making changes to the API at this point? If I were to write an RFC with an API approximating the following, would that be a plausible next step toward stabilization?

// ===== API generic to any UNIX platform
trait unix::net::SocketSend {
  // equivalent to sendmsg()
  fn send_message(..., Option<&mut AncillaryData>) -> io::Result<usize>;
}
trait unix::net::SocketRecv {
  // equivalent to recvmsg()
  fn receive_message(..., Option<&mut AncillaryData>) -> io::Result<usize>;
}

struct unix::net::AncillaryData;
  fn control_messages<'a>(&'a self) -> impl Iter<ControlMessage<'a>>;

struct unix::net::ControlMessage<'a>;
  fn level() -> c_int;
  fn type() -> c_int;
  fn data() -> &'a [u8]

// ===== Platform-specific API (Linux)
#[non_exhaustive]
enum linux::net::ControlMessage<'a> {
  ScmRights(ScmRights<'a>),
  ScmCredentials(ScmCredentials<'a>),
}
trait linux::net::ControlMessageExt: Sealed {
  fn try_get(&self) -> Option<linux::net::ControlMessage<'a>>,
}

struct linux::net::ScmCredentials<'a>;
  fn new(&unix::net::ControlMessage) -> Option<Self>;

struct linux::net::ScmRights<'a>;
  fn new(&unix::net::ControlMessage) -> Option<Self>;

// ===== Platform-specific API (etc)
trait freebsd::net::ControlMessageExt: Sealed { ... }
trait macos::net::ControlMessageExt: Sealed { ... }
trait netbsd::net::ControlMessageExt: Sealed { ... }

Advantages:

• API layout not affected by #[cfg(target_os)]
• New platform-specific functionality can be added freely via non_exhaustive enums.
• Exposing the ControlMessage fields directly allows users to self-service control message types not yet supported by the Rust standard library.

[kotauskas]

I found a potential critical concern with how ScmRights deals with its file descriptors upon being dropped – or, more specifically, how it does not.

rlimit DoS via SCM_RIGHTS is a pretty insidious issue that can easily go unnoticed and needs to be manually mitigated by the user of the standard library to make sure that file descriptor table pollution is avoided and no resources are leaked.

My own implementation of Unix domain sockets and ancillary data in the interprocess crate (actual current ancillary data code not pushed to main yet) does this differently and opts for a wrapper struct for SCM_RIGHTS (not too unlike ScmRights) that holds conditional ownership over its file descriptors, depending on whether it was constructed with &[BorrowedFd] or received from the kernel. If it's logically &[OwnedFd] with logical quasi-ownership through an immutable reference (as it would be if the FDs came from the kernel), it closes the descriptors upon being dropped, averting descriptor table pollution. It's not the cleanest implementation and I'm still working on it, but it's better than relegating prevention of rlimit DoS onto the downstream user and it takes care not to double-free.

[jmillikin]

I wrote a draft RFC for a v2 of Unix socket ancillary data support: rust-lang/rfcs#3430

In my testing, that proposal solves all the known issues with the current implementation (platform-specific control messages, third-party library extensibility, and file descriptor ownership). I would be happy if folks interested in stabilized ancillary data support could take a look at that RFC.

[the8472]

Are you planning to implement it too? Asking since I intended to work on this eventually.

[jmillikin]

Yes, if that RFC is accepted then I plan to implement it. I already have it working in a local branch, so it'd just be a matter of adding docs and tests.

[jmillikin]

The RFC looks like it's moving now (tagged libs-api-nominated) so I've started work on the implementation of non-contentious parts.

• Convert Unix{Datagram,Stream}::{set_}passcred() to per-OS traits #117156 is a change that needs to happen anyway, moving OS-specific socket methods to extension traits.
• Add types for inspecting control messages in Unix socket ancillary data #117196 adds ControlMessage{s,Iter,Buf} for constructing and inspecting the control messages buffer used by ancillary data.

[purplesyringa]

As far as I can see, the current implementation is unsound: it takes a user-provided buffer and performs unaligned reads and writes. Is this a known issue?

[GrahamBarnett]

1. Would if be possible to make the constructor public?

  pub fn new(buffer: &'a mut [u8]), current_length: usize, truncated: bool -> Self {
    SocketAncillary { buffer, length: current_length, truncated }
  }

this allows us to work on an existing ancillary buffer or re-use a SocketAncillary on different read buffers if we want, and as a by-product makes testing a lot easier

2. Please add #[derive(Debug)] to

  SocketCred, 
  ScmRights, 
  AncillaryDataIter<'a, T>
  AncillaryData<'a>

and:

  impl<'a> std::fmt::Debug for ScmRights<'a> {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        f.debug_list()
            .entries(Self(unsafe { AncillaryDataIter::<'a, RawFd>::new(self.0.data) }).collect::<Vec<RawFd>>())
            .finish()
    }
  }
  
  impl<'a> std::fmt::Debug for ScmCredentials<'a> {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        f.debug_list()
            .entries(Self(unsafe { AncillaryDataIter::<'a, libc::ucred>::new(self.0.data) }).collect::<Vec<SocketCred>>())
            .finish()
    }
  }
  
  impl<'a> std::fmt::Debug for Messages<'a> {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        let messages = Messages{buffer: self.buffer, current: self.current};
        f.debug_list().entries(messages.collect::<Vec<Result<AncillaryData<'a>, AncillaryError>>>()).finish()
    }
  }`

this allows us to println!("{:?}", ancillary.messages()) which makes logging/ debugging a lot easier.
It also means that you can debug the SocketAncillary struct itself (and get buffer status), or can show messages themselves as needed.

3. I had these changes suggested by the linter on 1.75:

    let data_len = (*cmsg).cmsg_len as usize - cmsg_len_zero; -->linter says should be--> let data_len = cmsg.cmsg_len - cmsg_len_zero;
    match (*cmsg).cmsg_level { -->linter says should be--> match cmsg.cmsg_level {
    cmsg_type: (*cmsg).cmsg_type, -->linter says should be--> cmsg_type: cmsg.cmsg_type,

Many thanks

[andrewbaxter]

1. Would if be possible to make the constructor public?

My use case is I don't know exactly where in a stream ancillary data will appear - I'd like to use several reads to parse a message and aggregate ancillary data across them (in a single buffer) to be used at the end. It doesn't appear trivially possible with the current design.

[GrahamBarnett]

I am doing exactly this. By making the constructor public and adding the length and truncated to the constructor, I can customise the mio behaviour as I need by managing the buffer/length/truncated collection myself using the mio io call and only later creating this class to do the unpacking

…

On Sat, 4 May 2024, 12:50 Andrew Baxter, ***@***.***> wrote: 1. Would if be possible to make the constructor public? My use case is I don't know exactly where in a stream ancillary data will appear. I'd like to use several reads to parse a message and aggregate ancillary data across them (in a single buffer) to be used at the end. It doesn't appear trivially possible with the current design. — Reply to this email directly, view it on GitHub <#76915 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASLCFDRQSLPJPQPNENYEB2TZATDRHAVCNFSM4RS6TJ6KU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBZGQYTGNJQGY2A> . You are receiving this because you commented.Message ID: ***@***.***>

[MaxVerevkin]

Wouldn't it be better for SocketAncillary's add_creds and add_fds to return a Result<(), NotEnoughSpaceErrror> instead of a bool?

[jonleivent]

Would it be better for SocketAncillary::new to take a MaybeUninit buffer? I assume that the original buffer should not be read from after calling SocketAncillary::new. Also, this saves the initialization expense.

[the8472]

When initializing a buffer that will contain a series of
              cmsghdr structures (e.g., to be sent with [sendmsg(2)](https://man7.org/linux/man-pages/man2/sendmsg.2.html)),
              that buffer should first be zero-initialized to ensure the
              correct operation of CMSG_NXTHDR().

at least sending it requires zero-initialization anyway.

[jonleivent]

that buffer should first be zero-initialized to ensure the
correct operation of CMSG_NXTHDR().

Is that the responsibility of the caller of, or of SocketAncillary::new itself? Because if this is a safety concern, how do you enforce it on the caller? Wouldn't a common mistake be to re-use the buffer for subsequent SocketAncillary::new calls without resetting its contents?