RSA-PSS vs PKCS#1 v1.5: which RSA signature padding

RSA-PSS vs PKCS#1 v1.5 is a choice of how a hash is encoded before the RSA private-key operation. Both come from PKCS #1 (RFC 8017), both are approved in FIPS 186-5, and both work with the same RSA key pair. The difference is in their security arguments, their determinism and the parameters that must match between signer and verifier.

At a glance

Security differences

PKCS#1 v1.5 signature padding dates from 1991. It is deterministic and has never had a security proof as clean as PSS's, but after three decades no practical forgery exists against verifiers that check the full encoded block. The real-world breaks were implementation bugs: Bleichenbacher's 2006 attack forged signatures for public exponent 3 against parsers that ignored trailing bytes after the DigestInfo, and variants of that mistake resurfaced in libraries such as NSS (BERserk, 2014) and others. RFC 8017 §8.2.2 avoids the whole class by re-encoding the expected block and comparing bytes, and FIPS 186-5 §5.4(h) codifies the checks.

RSASSA-PSS (Bellare and Rogaway, standardized in PKCS #1 v2.1, 2002) hashes a random salt with the message hash and masks the result with MGF1. Its security reduces tightly to the RSA problem in the random oracle model, and its randomization means a fault or side channel in one signature reveals less. With a zero-length salt PSS becomes deterministic but keeps its proof under slightly different assumptions.

Do not confuse either with PKCS #1 v1.5 encryption padding, which is genuinely broken by padding-oracle attacks (Bleichenbacher 1998, ROBOT 2017) and was removed from TLS 1.3. Signature padding and encryption padding are separate schemes that share a name.

Interoperability: the salt length trap

PSS needs signer and verifier to agree on three parameters: the hash, the MGF1 hash and the salt length. JOSE fixes all three (PS256 means SHA-256, MGF1-SHA-256, 32-byte salt), and TLS 1.3 requires the salt length to equal the hash length. Elsewhere you must check: OpenSSL 1.1.1 and 3.0 signed with the maximum salt by default (222 bytes for RSA-2048 with SHA-256), while OpenSSL 3.1 and later default to auto-digestmax, which yields a salt the size of the digest. Verifiers configured for a fixed salt length then reject signatures from the other generation.

The script below, run with OpenSSL 3.6.5, shows both behaviors.

#!/bin/sh
dir=$(mktemp -d)
cd "$dir" || exit 1
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out key.pem 2>/dev/null
openssl pkey -in key.pem -pubout -out pub.pem
printf 'hello' > msg

openssl dgst -sha256 -sign key.pem -out v15a.sig msg
openssl dgst -sha256 -sign key.pem -out v15b.sig msg
cmp -s v15a.sig v15b.sig && echo "PKCS#1 v1.5: identical signatures" || echo "PKCS#1 v1.5: different signatures"

openssl dgst -sha256 -sign key.pem -sigopt rsa_padding_mode:pss -out pssa.sig msg
openssl dgst -sha256 -sign key.pem -sigopt rsa_padding_mode:pss -out pssb.sig msg
cmp -s pssa.sig pssb.sig && echo "PSS: identical signatures" || echo "PSS: different signatures"

for saltlen in digest max; do
  printf 'verify PSS with rsa_pss_saltlen:%s -> ' "$saltlen"
  openssl dgst -sha256 -verify pub.pem -sigopt rsa_padding_mode:pss \
    -sigopt "rsa_pss_saltlen:$saltlen" -signature pssa.sig msg 2>&1 | grep -o -E "Verified OK|salt length check failed"
done
cd / && rm -rf "$dir"

PKCS#1 v1.5: identical signatures
PSS: different signatures
verify PSS with rsa_pss_saltlen:digest -> Verified OK
verify PSS with rsa_pss_saltlen:max -> salt length check failed

Verifiers that accept auto detect the salt length from the signature, which is the most compatible setting when you do not control the signer.

Keys and certificates

The same RSA key can produce both kinds of signature, and an rsaEncryption key in a certificate is usable for both (TLS calls that rsa_pss_rsae). X.509 also defines id-RSASSA-PSS keys whose parameters restrict them to PSS (rsa_pss_pss in TLS); they prevent cross-scheme use but many tools and some browsers handle them poorly, so rsaEncryption keys signing with PSS are the practical choice. Certificate signatures in the WebPKI are almost all PKCS#1 v1.5 or ECDSA, and verifiers must support v1.5 for that reason alone.

When to choose which

• TLS 1.3 handshakes: PSS, mandatory; your library does this automatically with any RSA certificate.
• JWTs: PS256 when all consumers support it, RS256 otherwise; better still, consider ES256 (see ES256 vs RS256).
• X.509 certificates and CRLs you issue: PKCS#1 v1.5 (sha256WithRSAEncryption) for maximum compatibility, unless a closed ecosystem standardizes on PSS.
• New document, firmware or API signatures with RSA keys: PSS with salt length equal to the hash length.
• Interop with HSMs and cloud KMS: both are supported (for example RSASSA_PSS_SHA_256 and RSASSA_PKCS1_V1_5_SHA_256 in AWS KMS); match the verifier's expectations.
• Verifying either: use a library's high-level verify function, never manual decoding of the encoded message.

Migration notes

Moving from v1.5 to PSS needs no new keys, which makes it easy to stage.

1. Teach verifiers PSS first: in JOSE, allow PS256 for the key (one algorithm per key is still the rule, so publish the same key material under a new kid with alg PS256, or rotate).
2. Fix the parameters explicitly: hash SHA-256, MGF1 SHA-256, salt length 32 bytes; avoid library defaults that differ between versions.
3. Switch signers and watch for verification failures that mention salt length or padding.
4. Remove v1.5 acceptance for those keys once old signatures have expired, keeping v1.5 only where certificates or third parties require it.
5. If you are rotating keys anyway, weigh ECDSA P-256 or Ed25519 instead, which are smaller and faster to sign than any RSA padding.

References

• RFC 8017, PKCS #1 v2.2 (§8.1, §8.2, §9.1, §9.2)
• FIPS 186-5 §5.4
• RFC 9846 §4.3.3, TLS 1.3 signature algorithms
• RFC 7518 §3.3 and §3.5, RS256 and PS256
• RFC 4055, RSASSA-PSS in X.509
• OpenSSL EVP_SIGNATURE-RSA, rsa_pss_saltlen values