It seems you haven't followed the JWT bearer flow documentation correctly.
You're not sending the private key directly (and if you have, you should consider it compromised and generate a new keypair and certificate), but rather using it to "sign" a copy of your payload (which, as I understand is hashing with sha256 followed by encrypting with the private key).
Having the certificate in Salesforce is then what allows you (well, Salesforce) to verify the signature (decrypt the signature with the public key in the cert, hash the original data, compare the two hashes, and make a determination whether or not the request was modified in-flight).
The structure of the JWT assertion looks like this
Base64URL({header}).Base64URL({claims}).Base64URL(RSA(SHA256(Base64URL({header}).Base64URL({claims}))))
or grouped a bit differently for readability
encodedHeader = Base64URL({header})
encodedClaims = Base64URL({claims})
token = encodedHeader.encodedClaims
assertion = token.Base64URL(RSA(SHA256(token)))
The private key is used in generating the signature (i.e. the second part of the "assertion"). If your assertion doesn't start with eyJhbGciOiJSUzI1NiJ9 (the Base64 encoded version of {"alg":"RS256"}) then it is wrong.
You also need to ensure the following:
- The scopes granted by your connected app include "access and manager your data (api)", "perform requests on your behalf at any time (refresh_token, offline_access)"
- The connected app is installed in your target org
- The connected app is configured to "Use Digital Signatures", and you've uploaded the certificate (.crt file)
- If you're installing the connected app in another org (i.e. the connected app is in your production org and you want to use it to authenticate with a new sandbox), you'll need to go through another OAuth flow (like the Web-Server or User-Agent flows) to avoid the user hasn't approved this consumer error. This question goes into more detail on that.
Signature.getInstance("SHA256withRSA");and the like in Java would be handled by something likeopenssl_sign()with OPENSSL_ALGO_SHA256 passed as the algorithm in PHP) – Derek F Jun 03 '20 at 21:51