Specified in OpenSSH's PROTOCOL.key: a magic string openssh-key-v1 plus a zero byte, the cipher name (none or aes256-ctr), the KDF name (none or bcrypt) with salt and rounds (16 by default, ssh-keygen -a raises it), the public keys and an encrypted section holding a check integer twice (to detect a wrong passphrase), the private keys and comments. Ed25519 keys have always used this format; since 7.8 RSA and ECDSA keys do too unless -m PEM is given.
Older libraries and tools read only PEM (PKCS #1, SEC 1 or PKCS #8): early JSch, some CI services and Java code using standard key specs fail with errors such as invalid privatekey. Convert with ssh-keygen -p -m PEM -f key (rewrites the file in place) or export a public key with ssh-keygen -e -m PKCS8. Public keys stay in the one-line type base64 comment form.