Support utxo query with kupo for Vasil (#121)

* incorporate split change logic

* incorporate split change logic

* incorporate split change logic

* minor modification to change combine logic

* minor modification to change combine logic

* minor modification to change combine logic

* integrate diff changes

* integrate diff changes and remove unused functions

* add change split test case. Modify txbuilder logic in handeling splits.

* Bump pytest from 7.0.1 to 7.1.1

Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.0.1 to 7.1.1.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](pytest-dev/pytest@7.0.1...7.1.1)

---
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* replace max fee with more accurately estimated fees

* replace max fee with more precise fee estimation

* remove max fee import

* modify format to be consistent with code base

* supports utxo queries using kupo

* support utxo query with Kupo

* reformat

* query utxo with Kupo when kupo url is provided

* remove the hardcoded genesis utxo in test_all file

* Create usage guides for Plutus and fix hyperlinks

* create separate section for datum/redeemer serialiation

* add -f flag to try to clean up the build if it exists

* Create usage guides for Plutus and fix hyperlinks

* create separate section for datum/redeemer serialiation

* add -f flag to try to clean up the build if it exists

* hdwallet implementation and unit test

* modify poetry

* modify poetry file

* modify poetry lock

* made changes to hdwallet and incorporated into bip32

* remove commented code block

* support utxo query with kupo

* move test_bip32

* update utxo query with kupo

* update plutus guide with vasil

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Yuanheng Wang <[email protected]>
Co-authored-by: Yuanheng Wang <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Author: 4 people

docs/source/guides/plutus.rst

@@ -161,3 +161,71 @@ Taker/Unlocker provides collateral. Collateral has been introduced in Alonzo tra
 
 The funds locked in script address is successfully retrieved to the taker address.
 
+-------------
+Vasil Upgrade
+-------------
+As part of the Basho phase of Cardano roadmap, the Vasil upgrade brings new capabilities on Plutus, namely reference inputs, inline datums, reference scripts, collateral output and Plutus V2 primitives.
+
+- **Reference inputs** (`CIP-31 <https://cips.cardano.org/cips/cip31/>`_): This upgrade enables data sharing on-chain. Previously, datums were carried in transaction outputs; they stored and provided access to information on the blockchain. However, to access information in this datum, one had to spend the output that the datum was attached to. This required the re-creation of a spent output. The addition of reference inputs now allows developers to look at the datum without extra steps. This facilitates access to information stored on the blockchain without the need for spending and re-creating UTXOs. This can be useful for oracles and other use cases where state need to be inspected.
+
+- **Inline datums** (`CIP-32 <https://cips.cardano.org/cips/cip32/>`_): Transaction datums were previously attached to outputs as hashes. With the implementation of inline datums, developers can now create scripts and attach datums directly to outputs instead of using their hashes. This simplifies how datums are used – a user can see the actual datum rather than supply it to match the given hash.
+
+- **Reference scripts** (`CIP-33 <https://cips.cardano.org/cips/cip33/>`_): In Alonzo, when spending an output locked within a Plutus script, one had to include the script in the spending transaction. This increased the size of the script and caused certain delays in its processing. The reference scripts upgrade allows developers to reference a script without including it in each transaction. This significantly reduces transaction size, improves throughput, and reduces script execution costs (since the script only needs to be paid for once).
+
+- **Explicit collateral output** (`CIP-40 <https://cips.cardano.org/cips/cip40/>`_): Transactions that call Plutus smart contracts are required to put up collateral to cover the potential cost of smart contract execution failure. If contract execution fails during phase 2 validation, all the funds stored in the chose UTXO for the collateral will be lost. After Vasil, user can specify a change address for the script collateral. If the script fails phase-2 validation, only the collateral amount will be taken, and the remaining funds will be sent to the change address.
+
+- **Plutus V2 scripts**: The Vasil upgrade includes a new cost model that's lower than before, and developers will be able to see redeemers for all inputs rather than just the one being passed to the currently executing script.
+
+Using the same FortyTwo example, now in Vasil, we show how reference scripts can be used. Reference script exists at a particular transaction output, and it can be used to witness UTxO at the corresponding script address::
+
+    >>> builder = TransactionBuilder(context)
+    >>> builder.add_input_address(giver_address)
+    >>> datum = 42
+    >>> # Include scripts in the script address
+    >>> builder.add_output(
+    >>>     TransactionOutput(script_address, 50000000, script=forty_two_script)
+    >>> )
+
+With reference script, actual script doesn't need to be included in the transaction anymore in order to spend UTxO sitting at script address::
+
+    >>> utxo_to_spend = None
+    >>> # Spend the utxo that has datum/datum hash but no script
+    >>> for utxo in chain_context.utxos(str(script_address)):
+    >>>     if not utxo.output.script and (
+    >>>        utxo.output.datum_hash == datum_hash(datum)
+    >>>         or utxo.output.datum == datum
+    >>>     ):
+    >>>         utxo_to_spend = utxo
+    >>>         break
+
+    >>> builder = TransactionBuilder(context)
+    >>> builder.add_script_input(utxo_to_spend, datum=datum, redeemer=redeemer)
+    >>> take_output = TransactionOutput(taker_address, 25123456)
+    >>> builder.add_output(take_output)
+    >>> signed_tx = builder.build_and_sign([extended_payment_skey], taker_address)
+
+Again, with the same example, we show that you can send funds to script address with inline datums directly::
+
+    >>> builder = TransactionBuilder(context)
+    >>> builder.add_input_address(giver_address)
+    >>> datum = 42
+    >>> builder.add_output(
+    >>>     TransactionOutput(script_address, 50000000, datum=datum, script=forty_two_script)
+    >>> )
+
+With inline datum, we no longer have to include a datum within our transaction for our plutus spending scripts. Instead we can specify the transaction output where our datum exists to be used in conjunction with our Plutus spending script. This reduces the overall size of our transaction::
+
+    >>> utxo_to_spend = None
+    >>> # Speed the utxo that has both inline script and inline datum
+    >>> for utxo in chain_context.utxos(str(script_address)):
+    >>>     if utxo.output.datum and utxo.output.script:
+    >>>         utxo_to_spend = utxo
+    >>>         break
+
+    >>> builder = TransactionBuilder(context)
+    >>> builder.add_script_input(utxo_to_spend, redeemer=redeemer)
+    >>> take_output = TransactionOutput(taker_address, 25123456)
+    >>> builder.add_output(take_output)
+    >>> signed_tx = builder.build_and_sign([extended_payment_skey], taker_address)
+
+

integration-test/docker-compose.yml

@@ -73,7 +73,7 @@ services:
         max-file: "10"
 
   kupo:
-    image: cardanosolutions/kupo
+    image: cardanosolutions/kupo:v2.1.0
     environment:
       NETWORK: "${NETWORK:-local}"
 

integration-test/test/base.py

@@ -21,9 +21,9 @@ class TestBase:
 
     OGMIOS_WS = "ws://localhost:1337"
 
-    KUPO_URL = "http://localhost:1442/v1/matches"
+    KUPO_URL = "http://localhost:1442"
 
-    chain_context = OgmiosChainContext(OGMIOS_WS, Network.TESTNET)
+    chain_context = OgmiosChainContext(OGMIOS_WS, Network.TESTNET, kupo_url=KUPO_URL)
 
     check_chain_context(chain_context)
 

integration-test/test/test_plutus.py

@@ -293,7 +293,7 @@ def test_plutus_v2_ref_script(self):
         for utxo in self.chain_context.utxos(str(script_address)):
             if not utxo.output.script and (
                 utxo.output.datum_hash == datum_hash(datum)
-                or utxo.output.datum == datum
+                or datum_hash(utxo.output.datum) == datum_hash(datum)
             ):
                 utxo_to_spend = utxo
                 break

pycardano/backend/ogmios.py

@@ -254,8 +254,8 @@ def _utxos_kupo(self, address: str) -> List[UTxO]:
                 "kupo_url object attribute has not been assigned properly."
             )
 
-        address_url = self._kupo_url + "/" + address
-        results = requests.get(address_url).json()
+        kupo_utxo_url = self._kupo_url + "/matches/" + address
+        results = requests.get(kupo_utxo_url).json()
 
         utxos = []
 
@@ -272,17 +272,42 @@ def _utxos_kupo(self, address: str) -> List[UTxO]:
 
                 lovelace_amount = result["value"]["coins"]
 
+                script = None
+                script_hash = result.get("script_hash", None)
+                if script_hash:
+                    kupo_script_url = self._kupo_url + "/scripts/" + script_hash
+                    script = requests.get(kupo_script_url).json()
+                    if script["language"] == "plutus:v2":
+                        script = PlutusV2Script(
+                            cbor2.loads(bytes.fromhex(script["script"]))
+                        )
+                    elif script["language"] == "plutus:v1":
+                        script = PlutusV1Script(
+                            cbor2.loads(bytes.fromhex(script["script"]))
+                        )
+                    else:
+                        raise ValueError("Unknown plutus script type")
+
+                datum = None
                 datum_hash = (
                     DatumHash.from_primitive(result["datum_hash"])
                     if result["datum_hash"]
                     else None
                 )
+                if datum_hash:
+                    kupo_datum_url = self._kupo_url + "/datums/" + result["datum_hash"]
+                    datum_result = requests.get(kupo_datum_url).json()
+                    if datum_result and datum_result["datum"] != datum_hash:
+                        datum = RawCBOR(bytes.fromhex(datum_result["datum"]))
+                        datum_hash = None
 
                 if not result["value"]["assets"]:
                     tx_out = TransactionOutput(
                         Address.from_primitive(address),
                         amount=lovelace_amount,
                         datum_hash=datum_hash,
+                        datum=datum,
+                        script=script,
                     )
                 else:
                     multi_assets = MultiAsset()
@@ -299,6 +324,8 @@ def _utxos_kupo(self, address: str) -> List[UTxO]:
                         Address.from_primitive(address),
                         amount=Value(lovelace_amount, multi_assets),
                         datum_hash=datum_hash,
+                        datum=datum,
+                        script=script,
                     )
                 utxos.append(UTxO(tx_in, tx_out))
             else: