Unable to verify signature returned by JSON-RPC `eth_sign` method

Question

I am attempting to verify a signature returned by the eth_sign JSON-RPC method.

https://github.com/ethereum/wiki/wiki/JSON-RPC#eth_sign

The verification is done using the following python code and test vectors. geth is running behind the scenes, being accessed over RPC with all API's enabled.

• Private Key being used for testing: 0x5e95384d8050109aab08c1922d3c230739bc16976553c317e5d0b87b59371f2a
• Message being signed: 1234567890abcdefghijklmnopqrstuvwxyz
• Hash of message: 0x089c33d56ed10bd8b571a0f493cedb28db1ae0f40c6cd266243001528c06eab3
• Signature Returned: 0xcb8a42e69ff562a4391ef5bb6346bdb3dd144e96c6462ea950a71fb707d2816b1b636277fcef51ead6139376cef5d5d244d677da06d655c42727de3e340a5bb701
• Public Key extracted from returned signature: 0x15f8c20c46c734b12527925392937890e7eba985

from sha3 import sha3_256
from secp256k1 import PrivateKey, PublicKey, ALL_FLAGS

from bitcoin import encode_pubkey, ecdsa_raw_sign

from ethereum.utils import privtoaddr

from eth_tester_client.utils import (
    mk_random_privkey,  # generates a random byte-string of length 32
    encode_address,  # encodes an ethereum address into it's hex representation
    encode_data,  # encodes a string into it's hex representation
    encode_number,  # encodes an integer into it's big_endian hex representation
    coerce_return_to_bytes,  # decorates a function, forcing it's return value to bytes.
)

from web3.web3.rpcprovider import TestRPCProvider
from web3.utils.encoding import (
    force_bytes,  # forces a string to bytes
    encode_hex,  # converts a string into `hex`
    decode_hex,  # converts a `hex` encoded value to `bytes`
    add_0x_prefix,  # adds a `0x` prefix if not present
    remove_0x_prefix,  # removes a `0x` prefix if present
)


@coerce_return_to_bytes
def sha3(s):
    return add_0x_prefix(sha3_256(s).hexdigest())

# sanity check that the correct sha3 (keccak) is being used.
assert sha3(b'') == b'0xc5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470'


def test_eth_sign(web3):
    private_key_hex = b'0x5e95384d8050109aab08c1922d3c230739bc16976553c317e5d0b87b59371f2a'
    private_key = decode_hex(private_key_hex)

    # This imports the private key into the running geth instance and unlocks
    # the account so that it can sign things.
    # `0xa5df35f30ba0ce878b1061ae086289adff3ba1e0`
    address = web3.personal.importRawKey(private_key, "password")
    web3.personal.unlockAccount(address, "password")

    assert add_0x_prefix(encode_hex(privtoaddr(private_key))) == add_0x_prefix(address)

    # the data to be signed
    data = b'1234567890abcdefghijklmnopqrstuvwxyz'
    # the hash of the data `0x089c33d56ed10bd8b571a0f493cedb28db1ae0f40c6cd266243001528c06eab3`
    data_hash = web3.sha3(data)

    assert force_bytes(data_hash) == sha3(data)

    priv_key = PrivateKey(flags=ALL_FLAGS)
    priv_key.set_raw_privkey(private_key)
    pub_key = priv_key.pubkey
    pub_key_serialized = pub_key.serialize(compressed=False)
    pub_key_address = sha3(encode_pubkey(pub_key_serialized, 'bin')[1:])[-40:]

    # This verifies that the address recovery code is working as expected since
    # the roundtrip operation matches the address being used to sign.
    assert add_0x_prefix(pub_key_address) == add_0x_prefix(address)

    # This is a sanity check using the pattern found internally in `pyethereum`
    v, r, s = ecdsa_raw_sign(decode_hex(data_hash), private_key)
    verify_pub_key = PublicKey(flags=ALL_FLAGS)
    verify_pub_key.public_key = verify_pub_key.ecdsa_recover(
        decode_hex(data_hash),
        verify_pub_key.ecdsa_recoverable_deserialize(
            decode_hex(encode_number(r, 32)) + decode_hex(encode_number(s, 32)), v - 27,
        ),
        raw=True,
    )
    verify_pub_key_serialized = verify_pub_key.serialize(compressed=False)
    verify_pub_key_address = sha3(encode_pubkey(verify_pub_key_serialized, 'bin')[1:])[-40:]

    assert add_0x_prefix(verify_pub_key_address) == add_0x_prefix(address)

    # Now have geth sign some data.
    # `0xcb8a42e69ff562a4391ef5bb6346bdb3dd144e96c6462ea950a71fb707d2816b1b636277fcef51ead6139376cef5d5d244d677da06d655c42727de3e340a5bb701`
    signature_hex = web3.eth.sign(remove_0x_prefix(address), data)
    signature_bytes = decode_hex(signature_hex)


    # Now lets try to recover the public key from the signature returned from
    # the `geth` JSON-RPC `eth_sign` endpoint.
    rec_pub_key = PublicKey(flags=ALL_FLAGS)
    rec_pub_key.public_key = rec_pub_key.ecdsa_recover(
        decode_hex(data_hash),
        rec_pub_key.ecdsa_recoverable_deserialize(
            signature_bytes[:64], signature_bytes[64] - 1,
        ),
        raw=True,
    )
    rec_pub_key_serialized = rec_pub_key.serialize(compressed=False)

    # This address is expected to be the same one used to sign.  Instead I get
    # `0x15f8c20c46c734b12527925392937890e7eba985` which is wrong
    rec_pub_key_address = sha3(encode_pubkey(rec_pub_key_serialized, 'bin')[1:])[-40:]

    # this fails because the addresses don't match
    assert add_0x_prefix(rec_pub_key_address) == add_0x_prefix(address)

    # try to verify the signature against the public key derived from the
    # original private key.  It fails.
    #
    # It of course passes if I use the recovered public key
    recoverable_signature = pub_key.ecdsa_recoverable_deserialize(signature_bytes[:64], signature_bytes[64])
    signature = pub_key.ecdsa_recoverable_convert(recoverable_signature)
    is_valid = pub_key.ecdsa_verify(
        msg=data,
        raw_sig=signature,
        digest=sha3_256,
    )
    assert is_valid

I have verified the following things to try and debug this.

• I am using the correct sha3 implementation (keccak)
• I can produce signatures using the given private key that recover the correct address as well as verify against the recovered and original public key.
• The address returned by web3 is the expected address that would be created by the private_key bytes.
• The returned signature also does not match the coinbase account or any of the other accounts present in the running geth instance.

I've mucked around with the following.

• Changing the v value passed into ecdsa_recoverable_deserialize
• Calling ecdsa_verify with raw=True and msg=sha3_256(data).digest().

In addition to all of this, I'm confused why the eth_sign calls return different signatures for the same inputs. I've verified that I see this same behavior from the javascript console.

Answer 1

I have had a similar issue and discovered that the eth_sign RPC call does not sign the passed message, but a transformation of it.

Quote from https://github.com/ethereum/wiki/wiki/JSON-RPC#eth_sign

The sign method calculates an Ethereum specific signature with: sign(keccak256("\x19Ethereum Signed Message:\n" + len(message) + message))).

(Where len(message) is the ASCII decimal representation of the length)

One option is to pass keccak256("\x19Ethereum Signed Message:\n" + len(message) + message) instead of message as first argument to ecrecover(). Another is to use a more low-level version of eth_sign.