Fix ExtendedSigning key generation from HDWallet

Author: Yuanheng Wang

.coveragerc

@@ -1,7 +1,7 @@
 [run]
 branch = True
 omit =
-    pycardano/crypto/*
+    pycardano/crypto/bech32.py
 
 [report]
 # Regexes for lines to exclude from consideration

.github/donate_addr.png

@@ -2,7 +2,7 @@ name: Build
 
 on:
   push:
-    branches: ["main"]
+    branches: "**"
   pull_request:
     branches: ["main"]
     types: [opened, reopened, edited, synchronize]
@@ -14,7 +14,11 @@ jobs:
     strategy:
       matrix:
         os: [ubuntu-latest, macos-latest]
-        python-version: [3.7, 3.8, 3.9, '3.10']
+        python-version: [3.7, 3.8, 3.9, '3.10', '3.11']
+        exclude:
+          # excludes python 3.11 on macOS
+          - os: macos-latest
+            python-version: '3.11'
 
     steps:
       - uses: actions/checkout@v3
@@ -28,16 +32,16 @@ jobs:
       - name: Install dependencies
         run: |
           poetry install
-      - name: Lint with flake8
-        run: |
-          poetry run flake8 pycardano
       - name: Run unit tests
         run: |
           poetry run pytest --doctest-modules --ignore=examples --cov=pycardano --cov-config=.coveragerc --cov-report=xml
       - name: "Upload coverage to Codecov"
-        uses: codecov/codecov-action@v2
+        uses: codecov/codecov-action@v3
         with:
           fail_ci_if_error: true
+      - name: Run static analyses
+        run: |
+          make qa
 
   continuous-integration:
     runs-on: ${{ matrix.os }}
@@ -59,3 +63,15 @@ jobs:
       - name: Run integration tests
         run: |
           cd integration-test && ./run_tests.sh
+
+      - name: Dump docker logs
+        if: failure()
+        run: |
+          cd integration-test && docker-compose logs --no-color > integration-test.log
+
+      - name: Upload integration-test.log
+        if: failure()
+        uses: actions/upload-artifact@v2
+        with:
+          name: integration-test.log
+          path: integration-test/integration-test.log

logo.png renamed to .github/logo.png

@@ -2,4 +2,8 @@
 .coverage
 cov_html
 docs/build
-dist
+dist
+
+# IDE
+.idea
+.code

.github/workflows/main.yml

@@ -4,6 +4,18 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/)
 and this project adheres to [Semantic Versioning](http://semver.org/).
 
+## [0.7.0] - 2022-10-16
+
+### Added
+
+- Support HDWallets and mnemonic phrases. (#85)
+
+### Fixed
+
+- Fix key error when there are duplicates in reference scripts.
+- If merging change into existing outputs is enabled, do not enforce min_utxo on changes.
+- Make script estimation more accurate.
+
 ## [0.6.3] - 2022-10-02
 
 ### Added

.gitignore

@@ -1,4 +1,4 @@
-.PHONY: clean clean-test clean-pyc clean-build format test help docs
+.PHONY: cov cov-html clean clean-test clean-pyc clean-build qa format test test-single help docs
 .DEFAULT_GOAL := help
 
 define BROWSER_PYSCRIPT
@@ -57,8 +57,13 @@ clean-test: ## remove test and coverage artifacts
 test: ## runs tests
 	poetry run pytest -s -vv -n 4
 
-qa: ## runs static analysis with flake8
+test-single: ## runs tests with "single" markers
+	poetry run pytest -s -vv -m single
+
+qa: ## runs static analyses
 	poetry run flake8 pycardano
+	poetry run mypy --install-types --non-interactive pycardano
+	poetry run black --check .
 
 format: ## runs code style and formatter
 	poetry run isort .

CHANGELOG.md

@@ -1,5 +1,5 @@
 <p align="center">
-  <img src="./logo.png" height=200 width=200 />
+  <img src="./.github/logo.png" height=200 width=200 />
 </p>
 
 ---
@@ -9,12 +9,14 @@
 [![PyPi version](https://badgen.net/pypi/v/pycardano)](https://pypi.python.org/pypi/pycardano/)
 [![PyPI pyversions](https://img.shields.io/pypi/pyversions/pycardano)](https://pypi.python.org/pypi/pycardano/)
 [![PyPI download month](https://img.shields.io/pypi/dm/pycardano)](https://pypi.python.org/pypi/pycardano/)
-[![Discord](https://img.shields.io/discord/949404918903631923.svg?label=chat&logo=discord&logoColor=ffffff&color=7389D8&labelColor=6A7EC2)](https://discord.gg/qT9Mn9xjgz)
 
-[![PyCardano](https://github.com/cffls/pycardano/actions/workflows/main.yml/badge.svg)](https://github.com/cffls/pycardano/actions/workflows/main.yml)
-[![codecov](https://codecov.io/gh/cffls/pycardano/branch/main/graph/badge.svg?token=62N0IL9IMQ)](https://codecov.io/gh/cffls/pycardano)
+[![PyCardano](https://github.com/Python-Cardano/pycardano/actions/workflows/main.yml/badge.svg)](https://github.com/Python-Cardano/pycardano/actions/workflows/main.yml)
+[![codecov](https://codecov.io/gh/Python-Cardano/pycardano/branch/main/graph/badge.svg?token=62N0IL9IMQ)](https://codecov.io/gh/Python-Cardano/pycardano)
 [![Documentation Status](https://readthedocs.org/projects/pycardano/badge/?version=latest)](https://pycardano.readthedocs.io/en/latest/?badge=latest)
 
+[![Discord](https://img.shields.io/discord/949404918903631923.svg?label=chat&logo=discord&logoColor=ffffff&color=7389D8&labelColor=6A7EC2)](https://discord.gg/qT9Mn9xjgz)
+[![Twitter](https://img.shields.io/twitter/follow/PyCardano?style=social&label=Follow%20%40PyCardano)](https://twitter.com/PyCardano)
+
 
 PyCardano is a Cardano library written in Python. It allows users to create and sign transactions without 
 depending on third-party Cardano serialization tools, such as
@@ -41,9 +43,9 @@ could be beneficial for faster R&D iterations.
 - [X] Plutus script
 - [X] Staking certificates
 - [X] Reward withdraw
-- [ ] Mnemonic 
+- [X] Mnemonic 
+- [X] HD Wallet
 - [ ] Byron Address
-- [ ] HD Wallet
 - [ ] Pool certificate
 - [ ] Protocol proposal update
 
@@ -158,7 +160,7 @@ context.submit_tx(signed_tx.to_cbor())
 ```
 </details>
 
-See more usages under [examples](https://github.com/cffls/pycardano/tree/main/examples).
+See more usages under [examples](https://github.com/Python-Cardano/pycardano/tree/main/examples).
 
 
 ### Development
@@ -170,7 +172,7 @@ See more usages under [examples](https://github.com/cffls/pycardano/tree/main/ex
 
 Clone the repository:
 
-`git clone https://github.com/cffls/pycardano.git`
+`git clone https://github.com/Python-Cardano/pycardano.git`
 
 PyCardano uses [poetry](https://python-poetry.org/) to manage its dependencies. 
 Install poetry for osx / linux / bashonwindows:
@@ -235,10 +237,25 @@ Build docs and open the docs in browser:
 
 </details>
 
+## Donation and Sponsor
+If you find this project helpful, please consider donate or sponsor us. Your donation and sponsor will allow us to
+ spend more time on improving PyCardano and adding more features in the future.
+
+You can support us by 1) sponsoring through Github, or 2) donating ADA to our ADA Handle `pycardano` or to the address below:
+
+[`addr1vxa4qadv7hk2dd3jtz9rep7sp92lkgwzefwkmsy3qkglq5qzv8c0d`](https://cardanoscan.io/address/61bb5075acf5eca6b632588a3c87d00955fb21c2ca5d6dc0910591f050)
+
+<p>
+  <img src="./.github/donate_addr.png" height=150 width=150/>
+</p>
+
+
 ## Sponsors :heart:
 
 <p align="left">
   <a href="https://www.blockery.io/"><img src="https://avatars.githubusercontent.com/u/97766045?s=50&v=4"/></a>
   <a href="https://github.com/KtorZ"><img src="https://avatars.githubusercontent.com/u/5680256?s=50&v=4"/></a>
   <a href="https://github.com/lacepool"><img src="https://avatars.githubusercontent.com/u/79309785?s=50&v=4"/></a>
+  <a href="https://github.com/muppetADA"><img width="50" src="https://avatars.githubusercontent.com/u/115075316?s=50&v=4"/></a>
+  <a href="https://github.com/CardanoDur"><img width="50" src="https://avatars.githubusercontent.com/u/1000466?s=50&v=4"/></a>
 </p>

Makefile

@@ -5,7 +5,7 @@ babel==2.10.3 ; python_version >= "3.7" and python_version < "4.0"
 black==22.8.0 ; python_version >= "3.7" and python_version < "4.0"
 blockfrost-python==0.5.1 ; python_version >= "3.7" and python_version < "4"
 cbor2==5.4.3 ; python_version >= "3.7" and python_version < "4.0"
-certifi==2022.6.15.2 ; python_version >= "3.7" and python_version < "4"
+certifi==2022.9.14 ; python_version >= "3.7" and python_version < "4"
 certvalidator==0.11.1 ; python_version >= "3.7" and python_version < "4.0"
 cffi==1.15.1 ; python_version >= "3.7" and python_version < "4.0"
 charset-normalizer==2.1.1 ; python_version >= "3.7" and python_version < "4"
@@ -17,6 +17,7 @@ cryptography==38.0.1 ; python_version >= "3.7" and python_version < "4.0"
 decorator==5.1.1 ; python_version >= "3.7" and python_version < "4.0"
 docutils==0.17.1 ; python_version >= "3.7" and python_version < "4.0"
 ecdsa==0.18.0 ; python_version >= "3.7" and python_version < "4.0"
+ecpy==1.2.5 ; python_version >= "3.7" and python_version < "4.0"
 execnet==1.9.0 ; python_version >= "3.7" and python_version < "4.0"
 flake8==4.0.1 ; python_version >= "3.7" and python_version < "4.0"
 flask==2.2.2 ; python_version >= "3.7" and python_version < "4.0"
@@ -29,6 +30,7 @@ itsdangerous==2.1.2 ; python_version >= "3.7" and python_version < "4.0"
 jinja2==3.1.2 ; python_version >= "3.7" and python_version < "4.0"
 markupsafe==2.1.1 ; python_version >= "3.7" and python_version < "4.0"
 mccabe==0.6.1 ; python_version >= "3.7" and python_version < "4.0"
+mnemonic==0.20 ; python_version >= "3.7" and python_version < "4.0"
 mypy-extensions==0.4.3 ; python_version >= "3.7" and python_version < "4.0"
 oscrypto==1.3.0 ; python_version >= "3.7" and python_version < "4.0"
 packaging==21.3 ; python_version >= "3.7" and python_version < "4.0"
@@ -50,7 +52,6 @@ pytest==7.1.3 ; python_version >= "3.7" and python_version < "4.0"
 pytz==2022.2.1 ; python_version >= "3.7" and python_version < "4.0"
 requests==2.28.1 ; python_version >= "3.7" and python_version < "4"
 retry==0.9.2 ; python_version >= "3.7" and python_version < "4.0"
-setuptools==65.3.0 ; python_version >= "3.7" and python_version < "4.0"
 six==1.16.0 ; python_version >= "3.7" and python_version < "4.0"
 snowballstemmer==2.2.0 ; python_version >= "3.7" and python_version < "4.0"
 sphinx-copybutton==0.5.0 ; python_version >= "3.7" and python_version < "4.0"

README.md

@@ -1,6 +1,8 @@
 CIP
 =====================
 
+Implementation of Cardano Improvement Proposals (CIPs)
+
 .. automodule:: pycardano.cip.cip8
    :members:
    :undoc-members:

docs/requirements.txt

@@ -0,0 +1,8 @@
+Crypto
+=============================
+A module that contains cryptographic functions.
+
+.. automodule:: pycardano.crypto.bip32
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/api/pycardano.cip.rst

@@ -161,3 +161,71 @@ Taker/Unlocker provides collateral. Collateral has been introduced in Alonzo tra
 
 The funds locked in script address is successfully retrieved to the taker address.
 
+-------------
+Vasil Upgrade
+-------------
+As part of the Basho phase of Cardano roadmap, the Vasil upgrade brings new capabilities on Plutus, namely reference inputs, inline datums, reference scripts, collateral output and Plutus V2 primitives.
+
+- **Reference inputs** (`CIP-31 <https://cips.cardano.org/cips/cip31/>`_): This upgrade enables data sharing on-chain. Previously, datums were carried in transaction outputs; they stored and provided access to information on the blockchain. However, to access information in this datum, one had to spend the output that the datum was attached to. This required the re-creation of a spent output. The addition of reference inputs now allows developers to look at the datum without extra steps. This facilitates access to information stored on the blockchain without the need for spending and re-creating UTXOs. This can be useful for oracles and other use cases where state need to be inspected.
+
+- **Inline datums** (`CIP-32 <https://cips.cardano.org/cips/cip32/>`_): Transaction datums were previously attached to outputs as hashes. With the implementation of inline datums, developers can now create scripts and attach datums directly to outputs instead of using their hashes. This simplifies how datums are used – a user can see the actual datum rather than supply it to match the given hash.
+
+- **Reference scripts** (`CIP-33 <https://cips.cardano.org/cips/cip33/>`_): In Alonzo, when spending an output locked within a Plutus script, one had to include the script in the spending transaction. This increased the size of the script and caused certain delays in its processing. The reference scripts upgrade allows developers to reference a script without including it in each transaction. This significantly reduces transaction size, improves throughput, and reduces script execution costs (since the script only needs to be paid for once).
+
+- **Explicit collateral output** (`CIP-40 <https://cips.cardano.org/cips/cip40/>`_): Transactions that call Plutus smart contracts are required to put up collateral to cover the potential cost of smart contract execution failure. If contract execution fails during phase 2 validation, all the funds stored in the chose UTXO for the collateral will be lost. After Vasil, user can specify a change address for the script collateral. If the script fails phase-2 validation, only the collateral amount will be taken, and the remaining funds will be sent to the change address.
+
+- **Plutus V2 scripts**: The Vasil upgrade includes a new cost model that's lower than before, and developers will be able to see redeemers for all inputs rather than just the one being passed to the currently executing script.
+
+Using the same FortyTwo example, now in Vasil, we show how reference scripts can be used. Reference script exists at a particular transaction output, and it can be used to witness UTxO at the corresponding script address::
+
+    >>> builder = TransactionBuilder(context)
+    >>> builder.add_input_address(giver_address)
+    >>> datum = 42
+    >>> # Include scripts in the script address
+    >>> builder.add_output(
+    >>>     TransactionOutput(script_address, 50000000, script=forty_two_script)
+    >>> )
+
+With reference script, actual script doesn't need to be included in the transaction anymore in order to spend UTxO sitting at script address::
+
+    >>> utxo_to_spend = None
+    >>> # Spend the utxo that has datum/datum hash but no script
+    >>> for utxo in chain_context.utxos(str(script_address)):
+    >>>     if not utxo.output.script and (
+    >>>        utxo.output.datum_hash == datum_hash(datum)
+    >>>         or utxo.output.datum == datum
+    >>>     ):
+    >>>         utxo_to_spend = utxo
+    >>>         break
+
+    >>> builder = TransactionBuilder(context)
+    >>> builder.add_script_input(utxo_to_spend, datum=datum, redeemer=redeemer)
+    >>> take_output = TransactionOutput(taker_address, 25123456)
+    >>> builder.add_output(take_output)
+    >>> signed_tx = builder.build_and_sign([extended_payment_skey], taker_address)
+
+Again, with the same example, we show that you can send funds to script address with inline datums directly::
+
+    >>> builder = TransactionBuilder(context)
+    >>> builder.add_input_address(giver_address)
+    >>> datum = 42
+    >>> builder.add_output(
+    >>>     TransactionOutput(script_address, 50000000, datum=datum, script=forty_two_script)
+    >>> )
+
+With inline datum, we no longer have to include a datum within our transaction for our plutus spending scripts. Instead we can specify the transaction output where our datum exists to be used in conjunction with our Plutus spending script. This reduces the overall size of our transaction::
+
+    >>> utxo_to_spend = None
+    >>> # Speed the utxo that has both inline script and inline datum
+    >>> for utxo in chain_context.utxos(str(script_address)):
+    >>>     if utxo.output.datum and utxo.output.script:
+    >>>         utxo_to_spend = utxo
+    >>>         break
+
+    >>> builder = TransactionBuilder(context)
+    >>> builder.add_script_input(utxo_to_spend, redeemer=redeemer)
+    >>> take_output = TransactionOutput(taker_address, 25123456)
+    >>> builder.add_output(take_output)
+    >>> signed_tx = builder.build_and_sign([extended_payment_skey], taker_address)
+
+

docs/source/api/pycardano.crypto.rst

@@ -142,15 +142,22 @@ Specify output amount::
 
 Step 6
 
+Add additional transaction information as needed:
+
+    >>> builder.ttl = 3600
+    >>> builder.reference_inputs.add(tx_in)
+
+Step 7
+
 Create a signed transaction using transaction builder. Unlike building a raw transaction, where we need to manually
 sign a transaction and build a transaction witness set, transaction builder can build and sign a transaction directly
 with its `build_and_sign` method. The code below tells the builder to build a transaction and sign the transaction
 with a list of signing keys (in this case, we only need the signature from one signing key, `sk`) and send the change
 back to sender's address::
 
-    >>> tx = builder.build_and_sign([sk], change_address=address)
+    >>> signed_tx = builder.build_and_sign([sk], change_address=address)
 
-Transaction ID could be obtained from the transaction obejct::
+Transaction ID could be obtained from the transaction object::
 
     >>> tx.id
     TransactionId(hex='1d40b950ded3a144fb4c100d1cf8b85719da91b06845530e34a0304427692ce4')

docs/source/guides/plutus.rst

@@ -35,6 +35,7 @@ making it a light-weight library that is easy and fast to set up in all kinds of
    api/pycardano.backend.base
    api/pycardano.certificate
    api/pycardano.cip
+   api/pycardano.crypto
    api/pycardano.coinselection
    api/pycardano.exception
    api/pycardano.hash

docs/source/guides/transaction.rst

@@ -46,5 +46,5 @@
         "pools": {},
         "stake": {}
     },
-    "epochLength": 2
+    "epochLength": 4
 }

docs/source/index.rst

@@ -73,7 +73,7 @@ services:
         max-file: "10"
 
   kupo:
-    image: cardanosolutions/kupo
+    image: cardanosolutions/kupo:v2.1.0
     environment:
       NETWORK: "${NETWORK:-local}"