DDSSEC12 — Underspecified RSASSA-PSS-SHA256 Salt Length

The DDS Security version 1.1 specification mentions:

If the Participant Private Key is a RSA key, then:

The value of the c.dsign_algo property shall be “RSASSA-PSS-SHA256”.
The digital signature shall be computed using the RSASSA-PSS algorithm specified in PKCS #1 (IETF 3447) RSA Cryptography Specifications Version 2.1 [44], using SHA256 as hash function, and MGF1 with SHA256 (mgf1sha256) as mask generation function.
This RFC is the one aforementioned. It states:

This encoding method is parameterized by the choice of hash function, mask generation function, and salt length. These options should be fixed for a given RSA key, except that the salt length can be variable (see [31] for discussion). Suggested hash and mask generation functions are given in Appendix B.

In the appendix, we can see an example where the salt length is the same as the hash length:

RSASSA-PSS-params ::= SEQUENCE
Unknown macro:
Unknown macro:

However, the salt length doesn't have to be the hash length. The WolfSSL and OpenSSL crypto libraries support several options (Openssl doc here and WolfSSL doc here). Our current approach is to use OpenSSL's default option: "maximum permissible value" (pkcsBlockLen - hLen - 2) when signing and auto (detect salt length from the signature) when verifying.

proposed solution
The specification should mention the salt length used when generating the signature. Otherwise, it should say that auto must be used when verifying the signature. This would allow interoperability (WolfSSL for example doesn't use auto by default).

Specify how to configure the salt length for RSA digital signatures

Merging with DDSSEC12-90 because the section referenced in the issue description ( 9.3.2.5.2) no longer describes algorithm-specific details, rather the crypto algorithms description are in a new dedicated section under section 8 "Common Cryptographic Algorithms" . So it is simpler to handle adding this detail to Issue 90.