Skip to content

Commit 918a30d

Browse files
ABastionOfSanityAbdul Rabbani
ABastionOfSanity
and
Abdul Rabbani
authored
V1.10.23 statediff v4 wip (#275)
* Statediff Geth Handle conflicts (#244) * Handle conflicts * Update go mod file versions * Make lint changes Disassociate block number from the indexer object Update ipld-eth-db ref Refactor builder code to make it reusable Use prefix comparison for account selective statediffing Update builder unit tests Add mode to write to CSV files in statediff file writer (#249) * Change file writing mode to csv files * Implement writer interface for file indexer * Implement option for csv or sql in file mode * Close files in CSV writer * Add tests for CSV file mode * Implement CSV file for watched addresses * Separate test configs for CSV and SQL * Refactor common code for file indexer tests Update indexer to include block hash in receipts and logs (#256) * Update indexer to include block hash in receipts and logs * Upgrade ipld-eth-db image in docker-compose to run tests Use watched addresses from direct indexing params by default while serving statediff APIs (#262) * Use watched addresses from direct indexing params in statediff APIs by default * Avoid using indexer object when direct indexing is off * Add nil check before accessing watched addresses from direct indexing params Rebase missed these changes needed at 1.10.20 Flags cleanup for CLI changes and linter complaints Linter appeasements to achieve perfection enforce go 1.18 for check (#267) * enforce go 1.18 for check * tests on 1.18 as well * adding db yml for possible change in docker-compose behavior in yml parsing Add indexer tests for handling non canonical blocks (#254) * Add indexer tests for header and transactions in a non canonical block * Add indexer tests for receipts in a non-canonical block and refactor * Add indexer tests for logs in a non-canonical block * Add indexer tests for state and storage nodes in a non-canonical block * Add indexer tests for non-canonical block at another height * Avoid passing address of a pointer * Update refs in GitHub workflow * Add genesis file path to stack-orchestrator config in GitHub workflow * Add descriptive comments fix non-deterministic ordering in unit tests Refactor indexer tests to avoid duplicate code (#270) * Refactor indexer tests to avoid duplicate code * Refactor file mode indexer tests * Fix expected db stats for sqlx after tx closure * Refactor indexer tests for legacy block * Refactor mainnet indexer tests * Refactor tests for watched addressess methods * Fix query in legacy indexer test rebase and resolve onto 1.10.23... still error out of index related to GetLeafKeys changed trie.Commit behavior was subtle about not not flushing to disk without an Update * no merge nodeset throws nil * linter appeasement Co-authored-by: Abdul Rabbani <[email protected]>
1 parent d901d85 commit 918a30d

File tree

181 files changed

+25377
-145
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

181 files changed

+25377
-145
lines changed

‎.github/workflows/checks.yml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
name: checks
2+
3+
on: [pull_request]
4+
5+
jobs:
6+
linter-check:
7+
runs-on: ubuntu-latest
8+
steps:
9+
- uses: actions/setup-go@v3
10+
with:
11+
go-version: "1.18"
12+
check-latest: true
13+
- uses: actions/checkout@v2
14+
- name: Run linter
15+
run: go run build/ci.go lint

‎.github/workflows/on-pr.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
name: Build and test
2+
3+
on: [pull_request]
4+
5+
jobs:
6+
run-tests:
7+
uses: ./.github/workflows/tests.yml

‎.github/workflows/publish.yaml‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
name: Publish geth to release
2+
on:
3+
release:
4+
types: [published]
5+
jobs:
6+
run-tests:
7+
uses: ./.github/workflows/tests.yml
8+
build:
9+
name: Run docker build and publish
10+
needs: run-tests
11+
runs-on: ubuntu-latest
12+
steps:
13+
- uses: actions/checkout@v2
14+
- name: Run docker build
15+
run: docker build -t vulcanize/go-ethereum -f Dockerfile .
16+
- name: Get the version
17+
id: vars
18+
run: echo ::set-output name=sha::$(echo ${GITHUB_SHA:0:7})
19+
- name: Tag docker image
20+
run: docker tag vulcanize/go-ethereum docker.pkg.github.com/vulcanize/go-ethereum/go-ethereum:${{steps.vars.outputs.sha}}
21+
- name: Docker Login
22+
run: echo ${{ secrets.GITHUB_TOKEN }} | docker login https://docker.pkg.github.com -u vulcanize --password-stdin
23+
- name: Docker Push
24+
run: docker push docker.pkg.github.com/vulcanize/go-ethereum/go-ethereum:${{steps.vars.outputs.sha}}
25+
push_to_registries:
26+
name: Publish assets to Release
27+
runs-on: ubuntu-latest
28+
needs: build
29+
steps:
30+
- name: Get the version
31+
id: vars
32+
run: |
33+
echo ::set-output name=sha::$(echo ${GITHUB_SHA:0:7})
34+
echo ::set-output name=tag::$(echo ${GITHUB_REF#refs/tags/})
35+
- name: Docker Login to Github Registry
36+
run: echo ${{ secrets.GITHUB_TOKEN }} | docker login https://docker.pkg.github.com -u vulcanize --password-stdin
37+
- name: Docker Pull
38+
run: docker pull docker.pkg.github.com/vulcanize/go-ethereum/go-ethereum:${{steps.vars.outputs.sha}}
39+
- name: Copy ethereum binary file
40+
run: docker run --rm --entrypoint cat docker.pkg.github.com/vulcanize/go-ethereum/go-ethereum:${{steps.vars.outputs.sha}} /usr/local/bin/geth > geth-linux-amd64
41+
- name: Docker Login to Docker Registry
42+
run: echo ${{ secrets.VULCANIZEJENKINS_PAT }} | docker login -u vulcanizejenkins --password-stdin
43+
- name: Tag docker image
44+
run: docker tag docker.pkg.github.com/vulcanize/go-ethereum/go-ethereum:${{steps.vars.outputs.sha}} vulcanize/vdb-geth:${{steps.vars.outputs.tag}}
45+
- name: Docker Push to Docker Hub
46+
run: docker push vulcanize/vdb-geth:${{steps.vars.outputs.tag}}
47+
- name: Get release
48+
id: get_release
49+
uses: bruceadams/[email protected]
50+
env:
51+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
52+
- name: Upload Release Asset
53+
id: upload-release-asset
54+
uses: actions/upload-release-asset@v1
55+
env:
56+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
57+
with:
58+
upload_url: ${{ steps.get_release.outputs.upload_url }}
59+
asset_path: geth-linux-amd64
60+
asset_name: geth-linux-amd64
61+
asset_content_type: application/octet-stream

‎.github/workflows/tests.yml‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
name: Tests for Geth that are used in multiple jobs.
2+
3+
on:
4+
workflow_call:
5+
6+
env:
7+
stack-orchestrator-ref: ${{ github.event.inputs.stack-orchestrator-ref || 'f2fd766f5400fcb9eb47b50675d2e3b1f2753702'}}
8+
ipld-eth-db-ref: ${{ github.event.inputs.ipld-ethcl-db-ref || 'be345e0733d2c025e4082c5154e441317ae94cf7' }}
9+
GOPATH: /tmp/go
10+
11+
jobs:
12+
build:
13+
name: Run docker build
14+
runs-on: ubuntu-latest
15+
steps:
16+
- uses: actions/checkout@v2
17+
- name: Run docker build
18+
run: docker build -t vulcanize/go-ethereum .
19+
20+
geth-unit-test:
21+
name: Run geth unit test
22+
runs-on: ubuntu-latest
23+
env:
24+
GO111MODULE: on
25+
steps:
26+
- name: Create GOPATH
27+
run: mkdir -p /tmp/go
28+
29+
- uses: actions/setup-go@v3
30+
with:
31+
go-version: "1.18"
32+
check-latest: true
33+
34+
- name: Checkout code
35+
uses: actions/checkout@v2
36+
37+
- name: Run unit tests
38+
run: |
39+
make test
40+
41+
statediff-unit-test:
42+
name: Run state diff unit test
43+
runs-on: ubuntu-latest
44+
steps:
45+
- name: Create GOPATH
46+
run: mkdir -p /tmp/go
47+
48+
- uses: actions/setup-go@v3
49+
with:
50+
go-version: "1.18"
51+
check-latest: true
52+
53+
- name: Checkout code
54+
uses: actions/checkout@v2
55+
56+
- name: Run docker compose
57+
run: |
58+
docker-compose up -d
59+
60+
- name: Give the migration a few seconds
61+
run: sleep 30;
62+
63+
- name: Run unit tests
64+
run: make statedifftest
65+
66+
private-network-test:
67+
name: Start Geth in a private network.
68+
runs-on: ubuntu-latest
69+
steps:
70+
- name: Create GOPATH
71+
run: mkdir -p /tmp/go
72+
73+
- uses: actions/setup-go@v3
74+
with:
75+
go-version: "1.18"
76+
check-latest: true
77+
78+
- name: Checkout code
79+
uses: actions/checkout@v3
80+
with:
81+
path: "./go-ethereum"
82+
83+
- uses: actions/checkout@v3
84+
with:
85+
ref: ${{ env.stack-orchestrator-ref }}
86+
path: "./stack-orchestrator/"
87+
repository: vulcanize/stack-orchestrator
88+
fetch-depth: 0
89+
90+
- uses: actions/checkout@v3
91+
with:
92+
ref: ${{ env.ipld-eth-db-ref }}
93+
repository: vulcanize/ipld-eth-db
94+
path: "./ipld-eth-db/"
95+
fetch-depth: 0
96+
97+
- name: Create config file
98+
run: |
99+
echo vulcanize_ipld_eth_db=$GITHUB_WORKSPACE/ipld-eth-db/ > $GITHUB_WORKSPACE/config.sh
100+
echo vulcanize_go_ethereum=$GITHUB_WORKSPACE/go-ethereum/ >> $GITHUB_WORKSPACE/config.sh
101+
echo db_write=true >> $GITHUB_WORKSPACE/config.sh
102+
echo genesis_file_path=start-up-files/go-ethereum/genesis.json >> $GITHUB_WORKSPACE/config.sh
103+
cat $GITHUB_WORKSPACE/config.sh
104+
105+
- name: Compile Geth
106+
run: |
107+
cd $GITHUB_WORKSPACE/stack-orchestrator/helper-scripts
108+
./compile-geth.sh -e docker -p $GITHUB_WORKSPACE/config.sh
109+
cd -
110+
111+
- name: Run docker compose
112+
run: |
113+
docker-compose \
114+
-f "$GITHUB_WORKSPACE/stack-orchestrator/docker/local/docker-compose-go-ethereum.yml" \
115+
-f "$GITHUB_WORKSPACE/stack-orchestrator/docker/local/docker-compose-db-sharding.yml" \
116+
--env-file $GITHUB_WORKSPACE/config.sh \
117+
up -d --build
118+
119+
- name: Make sure the /root/transaction_info/STATEFUL_TEST_DEPLOYED_ADDRESS exists within a certain time frame.
120+
shell: bash
121+
run: |
122+
COUNT=0
123+
ATTEMPTS=15
124+
docker ps
125+
until $(docker compose -f "$GITHUB_WORKSPACE/stack-orchestrator/docker/local/docker-compose-db-sharding.yml" -f "$GITHUB_WORKSPACE/stack-orchestrator/docker/local/docker-compose-go-ethereum.yml" cp go-ethereum:/root/transaction_info/STATEFUL_TEST_DEPLOYED_ADDRESS ./STATEFUL_TEST_DEPLOYED_ADDRESS) || [[ $COUNT -eq $ATTEMPTS ]]; do echo -e "$(( COUNT++ ))... \c"; sleep 10; done
126+
[[ $COUNT -eq $ATTEMPTS ]] && echo "Could not find the successful contract deployment" && (exit 1)
127+
cat ./STATEFUL_TEST_DEPLOYED_ADDRESS
128+
sleep 15;
129+
130+
- name: Create a new transaction.
131+
shell: bash
132+
run: |
133+
docker compose -f "$GITHUB_WORKSPACE/stack-orchestrator/docker/local/docker-compose-db-sharding.yml" -f "$GITHUB_WORKSPACE/stack-orchestrator/docker/local/docker-compose-go-ethereum.yml" exec go-ethereum /bin/bash /root/transaction_info/NEW_TRANSACTION
134+
echo $?
135+
136+
- name: Make sure we see entries in the header table
137+
shell: bash
138+
run: |
139+
rows=$(docker compose -f "$GITHUB_WORKSPACE/stack-orchestrator/docker/local/docker-compose-db-sharding.yml" -f "$GITHUB_WORKSPACE/stack-orchestrator/docker/local/docker-compose-go-ethereum.yml" exec ipld-eth-db psql -U vdbm -d vulcanize_testing -AXqtc "SELECT COUNT(*) FROM eth.header_cids")
140+
[[ "$rows" -lt "1" ]] && echo "We could not find any rows in postgres table." && (exit 1)
141+
echo $rows
142+
docker compose -f "$GITHUB_WORKSPACE/stack-orchestrator/docker/local/docker-compose-db-sharding.yml" -f "$GITHUB_WORKSPACE/stack-orchestrator/docker/local/docker-compose-go-ethereum.yml" exec ipld-eth-db psql -U vdbm -d vulcanize_testing -AXqtc "SELECT * FROM eth.header_cids"

‎.gitignore‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,3 +47,15 @@ profile.cov
4747
/dashboard/assets/package-lock.json
4848

4949
**/yarn-error.log
50+
foundry/deployments/local-private-network/geth-linux-amd64
51+
foundry/projects/local-private-network/geth-linux-amd64
52+
53+
# Helpful repos
54+
related-repositories/foundry-test/**
55+
related-repositories/hive/**
56+
related-repositories/ipld-eth-db/**
57+
statediff/indexer/database/sql/statediffing_test_file.sql
58+
statediff/statediffing_test_file.sql
59+
statediff/known_gaps.sql
60+
related-repositories/foundry-test/
61+
related-repositories/ipld-eth-db/

‎.gitmodules‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,4 +5,4 @@
55
[submodule "evm-benchmarks"]
66
path = tests/evm-benchmarks
77
url = https://github.com/ipsilon/evm-benchmarks
8-
shallow = true
8+
shallow = true

‎Dockerfile.amd64‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Build Geth in a stock Go builder container
2+
FROM golang:1.15.5 as builder
3+
4+
#RUN apk add --no-cache make gcc musl-dev linux-headers git
5+
6+
ADD . /go-ethereum
7+
RUN cd /go-ethereum && make geth

‎Makefile‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,31 @@
44

55
.PHONY: geth android ios evm all test clean
66

7+
BIN = $(GOPATH)/bin
8+
9+
## Migration tool
10+
GOOSE = $(BIN)/goose
11+
$(BIN)/goose:
12+
go get -u github.com/pressly/goose/cmd/goose
13+
714
GOBIN = ./build/bin
815
GO ?= latest
916
GORUN = env GO111MODULE=on go run
1017

18+
#Database
19+
HOST_NAME = localhost
20+
PORT = 5432
21+
USER = vdbm
22+
PASSWORD = password
23+
24+
# Set env variable
25+
# `PGPASSWORD` is used by `createdb` and `dropdb`
26+
export PGPASSWORD=$(PASSWORD)
27+
28+
#Test
29+
TEST_DB = vulcanize_public
30+
TEST_CONNECT_STRING = postgresql://$(USER):$(PASSWORD)@$(HOST_NAME):$(PORT)/$(TEST_DB)?sslmode=disable
31+
1132
geth:
1233
$(GORUN) build/ci.go install ./cmd/geth
1334
@echo "Done building."
@@ -48,3 +69,13 @@ devtools:
4869
env GOBIN= go install ./cmd/abigen
4970
@type "solc" 2> /dev/null || echo 'Please install solc'
5071
@type "protoc" 2> /dev/null || echo 'Please install protoc'
72+
73+
.PHONY: statedifftest
74+
statedifftest: | $(GOOSE)
75+
GO111MODULE=on go get github.com/stretchr/testify/[email protected]
76+
GO111MODULE=on MODE=statediff go test -p 1 ./statediff/... -v
77+
78+
.PHONY: statediff_filewriting_test
79+
statediff_filetest: | $(GOOSE)
80+
GO111MODULE=on go get github.com/stretchr/testify/[email protected]
81+
GO111MODULE=on MODE=statediff STATEDIFF_DB=file go test -p 1 ./statediff/... -v

0 commit comments

Comments
 (0)