deploy contract example broken with Quorum / Geth

[drandreaskrueger]

summary:

Your web3.py contract-deployment-example (or latest) does work fine with TestRPCProvider() and with (the Parity fork) Energy Web; but the example code does not work with (the Geth fork) Quorum 2.0.2.

Error:

web3/middleware/pythonic.py
ValueError: The value HexBytes('0xd78301070...') is 589 bytes, but should be 32
-->
web3/utils/formatters.py", line 72, in apply_formatters_to_dict
ValueError: Could not format value '0xd78301070...' as field 'extraData'

Good

This is similar to #808 but different. With the bugfix of issue 808, I could get it running on Web3.TestRPCProvider() AND on the Parity fork "Energy Web". All good there. No errors:

versions: web3 4.2.0, py-solc: 2.1.0, solc 0.4.23+commit.124ca40d.Linux.gpp, testrpc 1.3.4, python 3.5.3 
node ID string: Energy Web//v1.12.0-unstable-a7d13fb69-20180509/x86_64-linux-gnu/rustc1.25.0
sender address 0x0056D95f4c3F1f0B32B538E3BdD393D8e4850857 

submitted:  0x49289bdfd96e7dec86831aa0dc9c7afb65d64e16b3112eddc7d1c9c4ea0b454d
Default contract greeting: Hello
Setting the greeting to Nihao...
Updated contract greeting: Nihao

---

Bad

However now I am trying the Geth-fork Quorum, and the same Python code results in an error message:

versions: web3 4.2.0, py-solc: 2.1.0, solc 0.4.23+commit.124ca40d.Linux.gpp, testrpc 1.3.4, python 3.5.3   
node ID string: Geth/v1.7.2-stable-ee498061/linux-amd64/go1.9.3
sender address 0xed9d02e382b34818e88B88a309c7fe71E65f419d 

Traceback (most recent call last):
  File "...site-packages/web3/utils/formatters.py", line 70, in apply_formatters_to_dict
    yield key, formatters[key](item)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "...site-packages/web3/middleware/pythonic.py", line 83, in to_hexbytes
    result, len(result), num_bytes
ValueError: The value HexBytes('0xd783010702846765746887676f312e392e33856c696e75780000000000000000f9022af893946571d97f340c8495b661a823f2c2145ca47d63c2948157d4437104e3b8df4451a85f7b2438ef6699ff94b131288f355bc27090e542ae0be213c20350b76794b912de287f9b047b4228436e94b5b78e3ee1617194d8dba507e85f116b1f7e231ca8525fc9008a696694e36cbeb565b061217930767886474e3cde903ac594f512a992f3fb749857d758ffda1330e590fa915eb8415e0e622c15cebb79797e7740b8c12ddff9f3a9787fcb12fd704a07aded9bf89878189067d065ec9d7542e625af562d95d62b23cf2144727f63408e73c67f32f100f9014fb841a1c7248aa00eaceb5ee0b2b2baeb5775ed6d0249bbc760c12815c0dfd5a7707d5b47005c06a80e0d2b36a6bf9c7e03589499199f0a6891a13f5e0f083822723700b841ad216c6e61cb292b2a9c617fc126a53b593c8cef457e9fea18ba65a9a86491096f54a167d579e09f4ab52425b190d44cd5026e91b7cc9a8a32d577ff1362359001b841b84709f873924c3e43590b957d6c6883eb85c4169419f45117bb2bfd865205420841fe548e9da94a084df9cdbfe0b6989e8b05028f246dce123b8106e745616e01b841d5f2a050b2326a61ae42e32fc65e3a0a751906db0a354d388f3117fa23b6e3c34ad1788166a0358ccd3b0d352183e9b0df507f1c1a5def45fd7125b061a8889801b8413d19295613de41d180996c89c7b3ddef1c6d8f8fbd760a301514ff64c991d7af2322c91fb9b4152865c6ed01e221146cf82c75260223a6b3d89ef5985c5a489400') is 589 bytes, but should be 32

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "...gitlab/electronDLT/chainhammer/DeployContractExample_stable20180612.py", line 74, in <module>
    tx_hash = Greeter.constructor().transact()
  File "...site-packages/web3/utils/decorators.py", line 14, in _wrapper
    return self.method(obj, *args, **kwargs)
  File "...site-packages/web3/contract.py", line 746, in transact
    return self.web3.eth.sendTransaction(transact_transaction)
  File "...site-packages/web3/eth.py", line 239, in sendTransaction
    get_buffered_gas_estimate(self.web3, transaction),
  File "...site-packages/web3/utils/transactions.py", line 74, in get_buffered_gas_estimate
    gas_limit = get_block_gas_limit(web3)
  File "...site-packages/web3/utils/transactions.py", line 65, in get_block_gas_limit
    block = web3.eth.getBlock(block_identifier)
  File "...site-packages/web3/eth.py", line 141, in getBlock
    [block_identifier, full_transactions],
  File "...site-packages/web3/manager.py", line 103, in request_blocking
    response = self._make_request(method, params)
  File "...site-packages/web3/manager.py", line 86, in _make_request
    return request_func(method, params)
  File "...site-packages/web3/middleware/gas_price_strategy.py", line 18, in middleware
    return make_request(method, params)
  File "...site-packages/web3/middleware/formatting.py", line 23, in middleware
    response = make_request(method, params)
  File "...site-packages/web3/middleware/attrdict.py", line 18, in middleware
    response = make_request(method, params)
  File "...site-packages/web3/middleware/formatting.py", line 30, in middleware
    formatter(response['result']),
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "...site-packages/web3/utils/formatters.py", line 59, in apply_formatter_if
    return formatter(value)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "...site-packages/eth_utils/functional.py", line 22, in inner
    return callback(fn(*args, **kwargs))
  File "...site-packages/web3/utils/formatters.py", line 72, in apply_formatters_to_dict
    raise type(exc)("Could not format value %r as field %r" % (item, key)) from exc
ValueError: Could not format value '0xd783010702846765746887676f312e392e33856c696e75780000000000000000f9022af893946571d97f340c8495b661a823f2c2145ca47d63c2948157d4437104e3b8df4451a85f7b2438ef6699ff94b131288f355bc27090e542ae0be213c20350b76794b912de287f9b047b4228436e94b5b78e3ee1617194d8dba507e85f116b1f7e231ca8525fc9008a696694e36cbeb565b061217930767886474e3cde903ac594f512a992f3fb749857d758ffda1330e590fa915eb8415e0e622c15cebb79797e7740b8c12ddff9f3a9787fcb12fd704a07aded9bf89878189067d065ec9d7542e625af562d95d62b23cf2144727f63408e73c67f32f100f9014fb841a1c7248aa00eaceb5ee0b2b2baeb5775ed6d0249bbc760c12815c0dfd5a7707d5b47005c06a80e0d2b36a6bf9c7e03589499199f0a6891a13f5e0f083822723700b841ad216c6e61cb292b2a9c617fc126a53b593c8cef457e9fea18ba65a9a86491096f54a167d579e09f4ab52425b190d44cd5026e91b7cc9a8a32d577ff1362359001b841b84709f873924c3e43590b957d6c6883eb85c4169419f45117bb2bfd865205420841fe548e9da94a084df9cdbfe0b6989e8b05028f246dce123b8106e745616e01b841d5f2a050b2326a61ae42e32fc65e3a0a751906db0a354d388f3117fa23b6e3c34ad1788166a0358ccd3b0d352183e9b0df507f1c1a5def45fd7125b061a8889801b8413d19295613de41d180996c89c7b3ddef1c6d8f8fbd760a301514ff64c991d7af2322c91fb9b4152865c6ed01e221146cf82c75260223a6b3d89ef5985c5a489400' as field 'extraData'

---

Versions:

see above

The code which produced the error:

deployContract_example_web3.py

The full output of the error:

See above

What type of node you were connecting to.

Quorum. A fork of Geth.

Easiest way to have Quorum running is in vagrant virtualbox:
step 1 https://github.com/jpmorganchase/quorum-examples#vagrant-usage
step 2 https://github.com/jpmorganchase/quorum-examples/blob/master/examples/7nodes/README.md#7-nodes

How can it be fixed?

Is this caused by a web3.py ?
Is this caused by an again wrong example script?
Is this caused by Quorum?

I don't know.

[dylanjw]

Quorom sets a higher maximumExtraDataSize than the standard 32 bytes. The pythonic formatting middleware rejects extraData because it expects the 32 bytes maximum length. This also occurs when connecting web3 to some other chains with other configurations, namely "geth --dev".

You can try adding the geth_poa_middleware to the middleware stack, which moves extraData to new field proofOfAuthorityData which does validate that the value is under 32-bytes.

w3 = Web3(HTTPProvider('http://localhost:22000'))
from web3.middleware import geth_poa_middleware
# inject the poa compatibility middleware to the innermost layer
w3.middleware_stack.inject(geth_poa_middleware, layer=0)

Have a look at http://web3py.readthedocs.io/en/latest/middleware.html#geth-style-proof-of-authority

[dylanjw]

Feel free to ask any questions in this issue, but I am closing it as there is no web3 bug.

[voith]

@drandreaskrueger which web3 version are you using. I'm worried that the change I added in #756 is not working. It was suppose to give you a helpful error message whenever extraData is greater than 32 bytes. Or am I missing something?

[carver]

From the issue, they're using web3 4.2.0. That new error message was added in 4.2.1.

[voith]

Oops, I did not read the entire issue. Usually the web3 version is added at the start of the issue!

they're using

nice use of the neutral gender! 😆

[drandreaskrueger]

Fantastic, thank you all very much.

Yes, @dylanjw - this solves it - the error message disappears.

[drandreaskrueger]

as there is no web3 bug

true, @carver. However, anyone who wants to deploy contracts to quorum would encounter the same issue, right?

[drandreaskrueger]

they're using web3 4.2.0.

yes, because of previous version issues I had decided to fix all dependencies to definitive versions.

with this now, I have increased to 4.3.0 ...

That new error message was added in 4.2.1.

yes I see that now, when running the previous code:

Traceback (most recent call last):

  File ".../chainhammer/deployContract_example_web3.py", line 87, in <module>
    tx_hash = Greeter.constructor().transact()
  ...
  File ".../python3.5/site-packages/web3/middleware/validation.py", line 55, in check_extradata_length
    len(result), MAX_EXTRADATA_LENGTH, result

web3.exceptions.ValidationError: The field extraData is 589 bytes, but should be 32.   
It is quite likely that you are connected to a POA chain. Refer  
http://web3py.readthedocs.io/en/stable/middleware.html#geth-style-proof-of-authority  
for more details.   The full extraData is: HexBytes('0xd78301070...')

nice one, @voith . Thanks.

[drandreaskrueger]

... likely that you are connected to a POA chain ...

yes, the "quorum" (geth fork) I am running is PoA, and it did show that problem.

however, the "energy web" (parity fork) is also PoA - however it did not show the same problem.

See chapter "Good" in OP.

[voith]

however, the "energy web" (parity fork) is also PoA - however it did not show the same problem.

@drandreaskrueger You'll see the error message If and only if length of extraData is greater than 32 bytes.

[kigawas]

Quorom sets a higher maximumExtraDataSize than the standard 32 bytes. The pythonic formatting middleware rejects extraData because it expects the 32 bytes maximum length. This also occurs when connecting web3 to some other chains with other configurations, namely "geth --dev".

You can try adding the geth_poa_middleware to the middleware stack, which moves extraData to new field proofOfAuthorityData which does validate that the value is under 32-bytes.

w3 = Web3(HTTPProvider('http://localhost:22000'))
from web3.middleware import geth_poa_middleware
# inject the poa compatibility middleware to the innermost layer
w3.middleware_stack.inject(geth_poa_middleware, layer=0)

Have a look at http://web3py.readthedocs.io/en/latest/middleware.html#geth-style-proof-of-authority

It's w3.middleware_onion.inject(geth_poa_middleware, layer=0) now...