soliditySha parameter encoding issue

[Amxx]

• Version: 4.8.2
• Python: 3.7.1
• OS: linux
• pip freeze output

ppdirs==1.4.3
APScheduler==3.5.3
attrdict==2.0.0
backcall==0.1.0
borgbackup==1.1.7
Brlapi==0.6.7
btrfsutil==1.0.0
CacheControl==0.12.5
chardet==3.0.4
chrome-gnome-shell==0.0.0
colorama==0.4.1
cycler==0.10.0
cytoolz==0.9.0.1
decorator==4.3.0
distlib==0.2.8
distro==1.3.0
eth-abi==1.2.2
eth-account==0.3.0
eth-hash==0.2.0
eth-keyfile==0.5.1
eth-keys==0.2.0b3
eth-rlp==0.1.2
eth-typing==1.3.0
eth-utils==1.3.0
Glances==3.0.2
hexbytes==0.1.0
html5lib==1.0.1
idna==2.7
ipython==7.1.1
ipython-genutils==0.1.0
jedi==0.13.1
kiwisolver==1.0.1
lensfun==0.3.2
lockfile==0.12.2
louis==3.7.0
lru-dict==1.1.6
matplotlib==3.0.2
meld==3.19.1
msgpack==0.5.6
mutagen==1.41.1
numpy==1.15.4
packaging==18.0
parsimonious==0.8.1
parso==0.3.1
pexpect==4.6.0
pickleshare==0.7.5
progress==1.4
prompt-toolkit==2.0.7
psutil==5.4.8
ptyprocess==0.6.0
pycairo==1.18.0
pycryptodome==3.7.2
Pygments==2.3.0
PyGObject==3.30.4
pyparsing==2.3.0
PyQt5==5.11.3
PyQt5-sip==4.19.13
python-dateutil==2.7.5
pytoml==0.1.20
pytz==2018.7
pyxdg==0.26
requests==2.20.1
retrying==1.3.3
rlp==1.0.3
simplegeneric==0.8.1
six==1.11.0
team==1.0
toolz==0.9.0
traitlets==4.3.2
tzlocal==1.5.1
urllib3==1.24.1
wcwidth==0.1.7
web3==4.8.2
webencodings==0.5.1
websockets==6.0

What was wrong?

I am trying to implement EIP712 signatures in python. In order to do that, I need to follow a very strict hashing mechanism. I've got it working in web3.js and in solidity, but I fail to achieve the same hashes in python.

In order to compute the domain separator I use the following code:

return web3.soliditySha3([
	"bytes32",
	"bytes32",
	"bytes32",
	"uint256",
	"address",
],[
	EIP712DOMAIN_TYPEHASH,
	web3.sha3(text=domain.name   ),
	web3.sha3(text=domain.version),
	domain.chainId,
	domain.verifyingContract,
])

when checked, all 5 elements are the same as what web3.js produces. However, the result of soliditySha3 is not the same. I noticed that when I change the description of verifyingContract from 'address' to 'bytes32', the resulting hash doesn't change. Is it possible python's soliditySha3 does not abi.encode correctly ?

Just for the record, here is the corresponding javascript code:

return web3.utils.keccak256(web3.eth.abi.encodeParameters([
	"bytes32",
	"bytes32",
	"bytes32",
	"uint256",
	"address",
],[
	EIP712DOMAIN_TYPEHASH,
	web3.utils.keccak256(domain.name   ),
	web3.utils.keccak256(domain.version),
	domain.chainId,
	domain.verifyingContract,
]));

[carver]

Can you provide the specific inputs? When I smoke test the same concept, I get a matching result:

pragma solidity ^0.4.25;

contract KeccakTest {
    function keccak() public pure returns (bytes32) {
        bytes32 a = 0x1212121212121212121212121212121212121212121212121212121212121212;
        uint256 b = 3;
        address c = 0x3232323232323232323232323232323232323232;
        bytes memory d = abi.encodePacked(a, b, c);
        return keccak256(d);
    }
}

This returns 0x016376890565182a6ad9dc8211606eafa4aeec7a8be286fce25d1fd6ae6f9745 in remix.

Same result in web3.py:

w3.soliditySha3(
    ['bytes32', 'uint256', 'address'],
    [b'\x12'*32, 3, "0x3232323232323232323232323232323232323232"],
)

which returns HexBytes('0x016376890565182a6ad9dc8211606eafa4aeec7a8be286fce25d1fd6ae6f9745')

[molten-firescar96]

I have a similar problem on web3=4.8.2, solidity=0.5.0:

JS:

// 0xd5652966cc71c9ac8b3bb876a1ebe6985fc71f10a1ad2896a0c4393537c4948f
web3.utils.soliditySha3({t: 'bytes32', value: "0x8ddc1a359f25f52858404a15f1098181"})

Python:

# 0xe87822a6c1ab85ca5fd5f675c14c3d94f15727a2e7e700223bb5dcfb668ac0b2
Web3.soliditySha3(['bytes32'], ["0x8ddc1a359f25f52858404a15f1098181"]).hex()

Maybe something to do with padding differences between the two libraries. The 'string' type gets handled the same by both

[carver]

JS:

// 0xd5652966cc71c9ac8b3bb876a1ebe6985fc71f10a1ad2896a0c4393537c4948f
web3.utils.soliditySha3({t: 'bytes32', value: "0x8ddc1a359f25f52858404a15f1098181"})

Python:

# 0xe87822a6c1ab85ca5fd5f675c14c3d94f15727a2e7e700223bb5dcfb668ac0b2
Web3.soliditySha3(['bytes32'], ["0x8ddc1a359f25f52858404a15f1098181"]).hex()

Hm, actually both of those look wrong when the bytes32 value is short. My Remix run suggests that the result should be:
0xc5c3c026c2d7e0acf858a8142a4664f2fef9d227a4a8e904282ef05c63e58012

Source:

pragma solidity ^0.5.3;

contract KeccakTest {
    function shaShortBytes() public pure returns (bytes32) {
        bytes32 a = 0x000000000000000000000000000000008ddc1a359f25f52858404a15f1098181;
        return keccak256(abi.encodePacked(a));
    }
}

(Note that remix won't let you compile with a short bytes32 value)

If you pre-pad the value in web3.py, it looks right:

>>>: Web3.soliditySha3(['bytes32'], ['0x000000000000000000000000000000008ddc1a359f25f52858404a15f1098181'])
HexBytes('0xc5c3c026c2d7e0acf858a8142a4664f2fef9d227a4a8e904282ef05c63e58012')

---

Some obvious options are to:

1. disallow incomplete bytes32 values
2. left-pad with 0s

I'm leaning toward # 1

[tommyz7]

Above answer was helpful and a bit misleading for me at the same time. I had to right-pad bytes32 to achieve the same hash. I'm using web3==5.0.0a9 and solidityKeccak but I'm guessing that it's the same as soliditySha3 in previous version.

pragma solidity ^0.5.3;

contract HashTest {
    function encodePacked(bytes32 b) public view returns (bytes memory) {
        return abi.encodePacked(b);
    }
    
    function getHash(bytes32 mcode) public view returns (bytes32) {
        return keccak256(abi.encodePacked(mcode));
    }
}

getHash(0x757361) returns 0x1866d505cb5df2e484fa00f0290697b8f7888b0fc27822a24acd662fad3c087c but web3.solidityKeccak(["bytes32"], ["0x0000000000000000000000000000000000000000000000000000000000757361"]) returns this HexBytes('0x8fd6f4d044e5640b2e02d74593afe6b42f5e057740bea1c0122e5adfa05ff5ae')

encodePacked confirms that 0x757361 becomes this 0x7573610000000000000000000000000000000000000000000000000000000000
so to generate correct hash in python you have to right-pad:
web3.solidityKeccak(["bytes32"], ["0x7573610000000000000000000000000000000000000000000000000000000000"]) returns this HexBytes('0x1866d505cb5df2e484fa00f0290697b8f7888b0fc27822a24acd662fad3c087c')

As @carver mentioned this is not an issue if you fill 0s yourself. However, I often find myself storing short strings in bytes32 and filling it with 0s is kind of cumbersome. Perhaps coping solidity default behaviour is not the worst idea.

Hopefully this will save someone debugging time.

[carver]

So it looks like web3.js is left-padding and solidity is right-padding. That's even more reason for us to not take a stance in web3.py until it is standardized (and require explicit padding by the caller). In order to generate the hash, the caller must know which tool created the "short bytes" value, and pad it appropriately.

I agree that manual padding is a bit annoying, and would like to see a generally accepted standard so we can guess the correct padding direction. In the meantime, explicitly padding is a short-ish python built-in:

# right pad:
filled_val = short_val.ljust(32, b'\0')
# left pad:
filled_val = short_val.rjust(32, b'\0')

[carver]

cc @njgheorghita @kclowes Can we get the strictness of bytes input into v5? Much easier to relax it later than constrain it. That means that any constant-length bytes value would have to be explicitly padded by the caller.

[kclowes]

Sure! To be clear, this is a request for option 1 in this comment, right?

[carver]

Yup!

[MatthiasLohr]

Anything new here..?

[carver]

Anything new here..?

I don't know if there have been any standardization discussions between Solidity and web3.js.

---

As for web3.py, it looks like our answer will have to be that too-short values are rejected, and the caller must pad themselves using:

# right pad:
filled_val = short_val.ljust(32, b'\0')
# left pad:
filled_val = short_val.rjust(32, b'\0')

According to the summary on #1419 the too-short values are only rejected if you first call w3.enable_strict_bytes_type_checking() first. But it looks like that will be enabled by default in v6.