Skip to content

docs: overhaul all #585

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 12, 2024
Merged

docs: overhaul all #585

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 0 additions & 20 deletions CONTRIBUTING.md
View file
Open in desktop

This file was deleted.

20 changes: 9 additions & 11 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,24 +5,22 @@ Monitor and analyze the emergent behaviors of Bitcoin networks.

## Major Features

* Launch a bitcoin network with a specified number of nodes connected to each other according to a network topology from a graphml file.
* Scenarios can be run across the network which can be programmed using the Bitcoin Core functional [test_framework language](https://github.com/bitcoin/bitcoin/tree/master/test/functional).
* Nodes can have traffic shaping parameters assigned to them via the graph using [tc-netem](https://manpages.ubuntu.com/manpages/trusty/man8/tc-netem.8.html) tool.
* Data from nodes can be collected and searched including log files and p2p messages.
* Performance data from containers can be monitored and visualized.
* Lightning Network nodes can be deployed and operated.
* Networks can be deployed using Kubernetes, e.g. via MiniKube (small network graphs) or a managed cluster for larger network graphs.
* Launch a bitcoin network with a specified number of nodes connected to each other according to a network topology.
* Run scenarios of network behavior across the network which can be programmed using the Bitcoin Core functional [test_framework language](https://github.com/bitcoin/bitcoin/tree/master/test/functional).
* Collect and search data from nodes including log files and p2p messages.
* Monitor and visualize performance data from Bitcoin nodes.
* Connect to a large network running in a remote cluster, or a smaller network running locally.

## Documentation

- [Installation](/docs/install.md)
- [CLI Commands](/docs/warnet.md)
- [Network configuration with yaml files](/docs/config.md)
- [Scenarios](/docs/scenarios.md)
- [Monitoring](/docs/logging_monitoring.md)
- [Lightning Network](/docs/lightning.md)
- [Snapshots](/docs/snapshots.md)
- [Connecting to local nodes outside the cluster](/docs/connecting-local-nodes.md)
- [Scaling](/docs/scaling.md)
- [Connecting to local nodes](/docs/connecting-local-nodes.md)
- [Understanding network configuration](/docs/config.md)
- [Contributing](CONTRIBUTING.md)
- [Contributing](/docs/developer-notes.md)

![warnet-art](https://raw.githubusercontent.com/bitcoin-dev-project/warnet/main/docs/machines.webp)
3 changes: 1 addition & 2 deletions docs/connecting-local-nodes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,7 @@
### Run Warnet network

```shell
warnet cluster deploy
warnet network start
warnet deploy path/to/network/directory
```

### Install Telepresence
Expand Down
54 changes: 54 additions & 0 deletions docs/developer-notes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,62 @@
# Developer notes

## Contributing / Local Warnet Development

### Download the code repository

```bash
git clone https://github.com/bitcoin-dev-project/warnet
cd warnet
```

### Recommended: use a virtual Python environment such as `venv`

```bash
python3 -m venv .venv # Use alternative venv manager if desired
source .venv/bin/activate
```

```bash
pip install --upgrade pip
pip install -e .
```

## Lint

This project primarily uses the `uv` python packaging tool: https://docs.astral.sh/uv/ along with the sister formatter/linter `ruff` https://docs.astral.sh/ruff/

With `uv` installed you can add/remove dependencies using `uv add <dep>` or `uv remove <dep>.
This will update the [`uv.lock`](https://docs.astral.sh/uv/guides/projects/#uvlock) file automatically.


`uv` can also run tools (like `ruff`) without external installation, simply run `uvx ruff check .` or `uvx ruff format .` to use a uv-managed format/lint on the project.

## Release process
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think this is better in a dedicated document? It's not really developer notes or even relevant for general contributors.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only if you insist. i prefer having all the meta stuff in one place whether its for new contributors or us old salts.


Once a tag is pushed to GH this will start an image build using the tag

### Prerequisites

- [ ] Update version in pyproject.toml
- [ ] Tag git with new version
- [ ] Push tag to GitHub

### Manual Builds

```bash
# Install build dependencies
pip install -e .[build]

# Remove previous release metadata
rm -i -Rf build/ dist/

# Build wheel
python3 -m build
```

#### Upload

```bash
# Upload to Pypi
python3 -m twine upload dist/*
```
97 changes: 0 additions & 97 deletions docs/lightning.md
View file
Open in desktop

This file was deleted.

93 changes: 36 additions & 57 deletions docs/logging_monitoring.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,31 +1,23 @@
# Logging and Monitoring

Warnet allows different granularity of logging.

## Logging

### Warnet network level logging

Fetch logs from the warnet RPC server `rpc-0`, which is in charge of orchestrating the network.

Examples of information provided:
### Pod logs

- how many tanks are running
- what scenarios are running
- warnet RPC requests
The command `warnet logs` will bring up a menu of pods to print log output from,
such as Bitcoin tanks, or scenario commanders. Follow the output with the `-f` option.

Commands: `warnet network logs` or `warnet network logs --follow`.

See more details in [warnet](/docs/warnet.md#warnet-network-logs)
See command [`warnet logs`](/docs/warnet.md#warnet-logs)

### Bitcoin Core logs

These are tank level or pod level log output from a Bitcoin Core node, useful for things like net logging and transaction propagation, retrieved by RPC `debug-log` using its network name and graph node index.
Entire debug log files from a Bitcoin tank can be dumped by using the tank's
pod name.

Example:

```sh
$ warnet bitcoin debug-log 0
$ warnet bitcoin debug-log tank-0000


2023-10-11T17:54:39.616974Z Bitcoin Core version v25.0.0 (release build)
Expand All @@ -34,50 +26,46 @@ $ warnet bitcoin debug-log 0
... (etc)
```

For logs of lightning nodes, kubectl is required.
See command [`warnet bitcoin debug-log`](/docs/warnet.md#warnet-bitcoin-debug-log)

### Aggregated logs from all nodes
### Aggregated logs from all Bitcoin nodes

Aggregated logs can be searched using `warnet bitcoin grep-logs` with regex patterns.

See more details in [`warnet bitcoin grep-logs`](/docs/warnet.md#warnet-bitcoin-grep-logs)

Example:

```sh
$ warnet bitcoin grep-logs 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d

warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.716582Z [miner] AddToWallet 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d newupdate
warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.717787Z [miner] Submitting wtx 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d to mempool for relay
warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.717929Z [validation] Enqueuing TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf
warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.718040Z [validation] TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf
warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.723017Z [miner] AddToWallet 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d
warnet_test_uhynisdj_tank_000007: 2023-10-11T17:44:52.173199Z [validation] Enqueuing TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf
tank-0001: 2023-10-11T17:44:48.716582Z [miner] AddToWallet 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d newupdate
tank-0001: 2023-10-11T17:44:48.717787Z [miner] Submitting wtx 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d to mempool for relay
tank-0001: 2023-10-11T17:44:48.717929Z [validation] Enqueuing TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf
tank-0001: 2023-10-11T17:44:48.718040Z [validation] TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf
tank-0001: 2023-10-11T17:44:48.723017Z [miner] AddToWallet 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d
tank-0002: 2023-10-11T17:44:52.173199Z [validation] Enqueuing TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf
... (etc)
```

See more details in [warnet](/docs/warnet.md#warnet-bitcoin-grep-logs)

## Monitoring and Metrics

## Install logging infrastructure

Ensure that [`helm`](https://helm.sh/docs/intro/install/) is installed, then simply run the following scripts:
If any tank in a network is configured with `collectLogs: true` or `metricsExport: true`
then the logging stack will be installed automatically when `warnet deploy` is executed.

```bash
./resources/scripts/install_logging.sh
```
The logging stack includes Loki, Prometheus, and Grafana. Together these programs
aggregate logs and data from Bitcoin RPC queries into a web-based dashboard.

To forward port `3000` and view the [Grafana](#grafana) dashboard run the `connect_logging` script:
## Connect to logging dashboard

```bash
./resources/scripts/connect_logging.sh
```

It might take a couple minutes to get the pod running. If you see `error: unable to forward port because pod is not running. Current status=Pending`, hang tight.

The Grafana dashboard (and API) will be accessible without requiring authentication
at `http://localhost:3000`.
The logging stack including the user interface web server runs inside the kubernetes cluster.
To access that from a local web browser, you must use kubernetes port-forwarding.

The `install_logging` script will need to be installed before starting the network in order to collect the information for monitoring and metrics. Restart the network with `warnet network down && warnet network up` if necessary.
Run the script `./resources/scripts/connect_logging.sh` to forward port 3000.
The Grafana dashboard will then be available locally at `localhost:3000`.

### Prometheus

Expand All @@ -86,40 +74,31 @@ to any Bitcoin Tank and configured to scrape any available RPC results.

The `bitcoin-exporter` image is defined in `resources/images/exporter` and
maintained in the BitcoinDevProject dockerhub organization. To add the exporter
in the Tank pod with Bitcoin Core add the `"exporter"` key to the node in the graphml file:

```xml
<node id="0">
<data key="version">27.0</data>
<data key="exporter">true</data>
</node>
```
in the Tank pod with Bitcoin Core add the `metricsExport: true` value to the node in the yaml file.

The default metrics are defined in the `bitcoin-exporter` image:
- Block count
- Number of inbound peers
- Number of outbound peers
- Mempool size (# of TXs)

Metrics can be configured by setting a `"metrics"` key to the node in the graphml file.
The metrics value is a space-separated list of labels, RPC commands with arguments, and
Metrics can be configured by setting an additional `metrics` value to the node in the yaml file. The metrics value is a space-separated list of labels, RPC commands with arguments, and
JSON keys to resolve the desired data:

```
label=method(arguments)[JSON result key][...]
```

For example, the default metrics listed above are defined as:
For example, the default metrics listed above would be explicitly configured as follows:

```xml
<node id="0">
<data key="version">27.0</data>
<data key="exporter">true</data>
<data key="metrics">blocks=getblockcount() inbounds=getnetworkinfo()["connections_in"] outbounds=getnetworkinfo()["connections_in"] mempool_size=getmempoolinfo()["size"]</data>
</node>
```yaml
nodes:
- name: tank-0000
metricsExport: true
metrics: blocks=getblockcount() inbounds=getnetworkinfo()["connections_in"] outbounds=getnetworkinfo()["connections_in"] mempool_size=getmempoolinfo()["size"]
```

The data can be retrieved from the Prometheus exporter on port `9332`, example:
The data can be retrieved directly from the Prometheus exporter container in the tank pod via port `9332`, example:

```
# HELP blocks getblockcount()
Expand All @@ -138,7 +117,7 @@ mempool_size 0.0

### Grafana

Data from Prometheus exporters can be collected and fed into Grafana for a
Data from Prometheus exporters is collected and fed into Grafana for a
web-based interface.

#### Dashboards
Expand Down
Binary file removed docs/random_internet_as_graph_n100.png
View file
Open in desktop
Binary file not shown.
Loading