Skip to content

Commit b091e4f

Browse files
colinlyguoThegaramlispc
authored
feat(rollup-sync-service): support codecv2 (#726)
* feat(codecv1): support zstd compression * revert .gitignore changes * change dependency * add zstd compression * bump golang version to 1.21.x * use codecv2 * remove an invalid command * tweak * tweak * remove libzstd, use .so files in da-codec release instead * fix CI * fix CI * try fix CI * fix CI * tweak * fix * fix CI * tweak * tweak * address comments * tweak * remove cp to /usr/local/lib * fix CI * remove cp to /usr/local/lib in CI * try fix * fix * fix * fix * remove env GO111MODULE=on * add .so files to Dockerfiles * revert make geth changes * fix Dockerfile changes * tweak * revert changes * tweak * trigger ci * update Dockerfiles * fix * add codecv2 in decode block ranges from encoded chunks * address comment * nit * update dependencies * use /scroll/lib/ in testing * trigger ci * add unit tests * chore: auto version bump [bot] * remove zkp-builder in Dockerfile.mockccc * fix Dockerfile * Update Makefile Co-authored-by: Péter Garamvölgyi <[email protected]> * use -Wl,-rpath,/scroll/lib/ * change some static values to env var * update dependency * change pwd to scroll_lib_path * fix * add sudo in mkdir * add sudo * update da-codec dependency * chore: auto version bump [bot] * chore: auto version bump [bot] * chore: auto version bump [bot] * update zstd dependency * update libscroll_zstd.so and remove libzktrie.so related to zstd * chore: auto version bump [bot] * uncomment ./rollup tests * fix CI * fix CI * upgrade libzkp to v0.11.0rc2 (#773) * upgrade libzkp * remove zktrie.so * done * fix build --------- Co-authored-by: colin <[email protected]> --------- Co-authored-by: colinlyguo <[email protected]> Co-authored-by: Péter Garamvölgyi <[email protected]> Co-authored-by: Zhang Zhuo <[email protected]>
1 parent e6cc960 commit b091e4f

File tree

20 files changed

+690
-1711
lines changed

20 files changed

+690
-1711
lines changed

‎.github/workflows/l2geth_ci.yml‎

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ jobs:
2020
- name: Install Go
2121
uses: actions/setup-go@v2
2222
with:
23-
go-version: 1.20.x
23+
go-version: 1.21.x
2424
- name: Checkout code
2525
uses: actions/checkout@v2
2626
- name: Build
@@ -33,7 +33,7 @@ jobs:
3333
- name: Install Go
3434
uses: actions/setup-go@v2
3535
with:
36-
go-version: 1.20.x
36+
go-version: 1.21.x
3737
- name: Install rust
3838
uses: actions-rs/toolchain@v1
3939
with:
@@ -46,7 +46,6 @@ jobs:
4646
run: |
4747
make libzkp
4848
sudo cp ./rollup/circuitcapacitychecker/libzkp/libzkp.so /usr/local/lib/
49-
sudo cp ./rollup/circuitcapacitychecker/libzkp/libzktrie.so /usr/local/lib/
5049
make geth
5150
check:
5251
if: github.event.pull_request.draft == false
@@ -55,7 +54,7 @@ jobs:
5554
- name: Install Go
5655
uses: actions/setup-go@v2
5756
with:
58-
go-version: 1.20.x
57+
go-version: 1.21.x
5958
- name: Checkout code
6059
uses: actions/checkout@v2
6160
- name: Lint
@@ -88,7 +87,7 @@ jobs:
8887
- name: Install Go
8988
uses: actions/setup-go@v2
9089
with:
91-
go-version: 1.20.x
90+
go-version: 1.21.x
9291
- name: Checkout code
9392
uses: actions/checkout@v2
9493
- run: go mod tidy
@@ -105,12 +104,11 @@ jobs:
105104
- name: Install Go
106105
uses: actions/setup-go@v2
107106
with:
108-
go-version: 1.20.x
107+
go-version: 1.21.x
109108
- name: Checkout code
110109
uses: actions/checkout@v2
111110
- name: Test
112111
run: |
113-
go get ./...
114112
make test
115113
- name: Upload coverage report
116114
run: bash <(curl -s https://codecov.io/bash)

‎Dockerfile‎

Lines changed: 26 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22
ARG COMMIT=""
33
ARG VERSION=""
44
ARG BUILDNUM=""
5+
ARG LIBSCROLL_ZSTD_VERSION=v0.1.0-rc0-ubuntu20.04
6+
ARG SCROLL_LIB_PATH=/scroll/lib
57

68
# Build libzkp dependency
79
FROM scrolltech/go-rust-builder:go-1.20-rust-nightly-2022-12-10 as chef
@@ -19,27 +21,43 @@ RUN cargo chef cook --release --recipe-path recipe.json
1921
COPY ./rollup/circuitcapacitychecker/libzkp .
2022
RUN cargo clean
2123
RUN cargo build --release
22-
RUN find ./ | grep libzktrie.so | xargs -I{} cp {} /app/target/release/
2324

2425
# Build Geth in a stock Go builder container
2526
FROM scrolltech/go-rust-builder:go-1.20-rust-nightly-2022-12-10 as builder
2627

2728
ADD . /go-ethereum
28-
COPY --from=zkp-builder /app/target/release/libzkp.so /usr/local/lib/
29-
COPY --from=zkp-builder /app/target/release/libzktrie.so /usr/local/lib/
30-
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/
29+
30+
ARG LIBSCROLL_ZSTD_VERSION
31+
ARG SCROLL_LIB_PATH
32+
33+
RUN mkdir -p $SCROLL_LIB_PATH
34+
35+
COPY --from=zkp-builder /app/target/release/libzkp.so $SCROLL_LIB_PATH
36+
RUN wget -O $SCROLL_LIB_PATH/libscroll_zstd.so https://github.com/scroll-tech/da-codec/releases/download/$LIBSCROLL_ZSTD_VERSION/libscroll_zstd.so
37+
38+
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$SCROLL_LIB_PATH
39+
ENV CGO_LDFLAGS="-L$SCROLL_LIB_PATH -Wl,-rpath,$SCROLL_LIB_PATH"
40+
3141
RUN cd /go-ethereum && env GO111MODULE=on go run build/ci.go install -buildtags circuit_capacity_checker ./cmd/geth
3242

3343
# Pull Geth into a second stage deploy alpine container
3444
FROM ubuntu:20.04
3545

3646
RUN apt-get -qq update \
37-
&& apt-get -qq install -y --no-install-recommends ca-certificates
47+
&& apt-get -qq install -y --no-install-recommends ca-certificates wget
3848

3949
COPY --from=builder /go-ethereum/build/bin/geth /usr/local/bin/
40-
COPY --from=zkp-builder /app/target/release/libzkp.so /usr/local/lib/
41-
COPY --from=zkp-builder /app/target/release/libzktrie.so /usr/local/lib/
42-
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/
50+
51+
ARG LIBSCROLL_ZSTD_VERSION
52+
ARG SCROLL_LIB_PATH
53+
54+
RUN mkdir -p $SCROLL_LIB_PATH
55+
56+
COPY --from=zkp-builder /app/target/release/libzkp.so $SCROLL_LIB_PATH
57+
RUN wget -O $SCROLL_LIB_PATH/libscroll_zstd.so https://github.com/scroll-tech/da-codec/releases/download/$LIBSCROLL_ZSTD_VERSION/libscroll_zstd.so
58+
59+
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$SCROLL_LIB_PATH
60+
ENV CGO_LDFLAGS="-L$SCROLL_LIB_PATH -Wl,-rpath,$SCROLL_LIB_PATH"
4361

4462
EXPOSE 8545 8546 30303 30303/udp
4563
ENTRYPOINT ["geth"]

‎Dockerfile.mockccc‎

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,18 +2,41 @@
22
ARG COMMIT=""
33
ARG VERSION=""
44
ARG BUILDNUM=""
5+
ARG LIBSCROLL_ZSTD_VERSION=v0.1.0-rc0-ubuntu20.04
6+
ARG SCROLL_LIB_PATH=/scroll/lib
57

68
# Build Geth in a stock Go builder container
79
FROM scrolltech/go-rust-builder:go-1.20-rust-nightly-2022-12-10 as builder
810

911
ADD . /go-ethereum
10-
RUN cd /go-ethereum && env GO111MODULE=on go run build/ci.go install ./cmd/geth
12+
13+
ARG LIBSCROLL_ZSTD_VERSION
14+
ARG SCROLL_LIB_PATH
15+
16+
RUN mkdir -p $SCROLL_LIB_PATH
17+
18+
RUN apt-get -qq update && apt-get -qq install -y wget
19+
RUN wget -O $SCROLL_LIB_PATH/libscroll_zstd.so https://github.com/scroll-tech/da-codec/releases/download/$LIBSCROLL_ZSTD_VERSION/libscroll_zstd.so
20+
21+
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$SCROLL_LIB_PATH
22+
ENV CGO_LDFLAGS="-L$SCROLL_LIB_PATH -Wl,-rpath,$SCROLL_LIB_PATH"
1123

1224
# Pull Geth into a second stage deploy alpine container
1325
FROM ubuntu:20.04
1426

1527
COPY --from=builder /go-ethereum/build/bin/geth /usr/local/bin/
1628

29+
ARG LIBSCROLL_ZSTD_VERSION
30+
ARG SCROLL_LIB_PATH
31+
32+
RUN mkdir -p $SCROLL_LIB_PATH
33+
34+
RUN apt-get -qq update && apt-get -qq install -y wget
35+
RUN wget -O $SCROLL_LIB_PATH/libscroll_zstd.so https://github.com/scroll-tech/da-codec/releases/download/$LIBSCROLL_ZSTD_VERSION/libscroll_zstd.so
36+
37+
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$SCROLL_LIB_PATH
38+
ENV CGO_LDFLAGS="-L$SCROLL_LIB_PATH -Wl,-rpath,$SCROLL_LIB_PATH"
39+
1740
EXPOSE 8545 8546 30303 30303/udp
1841
ENTRYPOINT ["geth"]
1942

‎Makefile‎

Lines changed: 14 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -2,21 +2,27 @@
22
# with Go source code. If you know what GOPATH is then you probably
33
# don't need to bother with make.
44

5-
.PHONY: geth android ios evm all test clean libzkp
5+
.PHONY: geth android ios evm all test clean libzkp libzstd
66

77
GOBIN = ./build/bin
88
GO ?= latest
9-
GORUN = env GO111MODULE=on go run
9+
GORUN = go run
10+
LIBSCROLL_ZSTD_VERSION = v0.1.0-rc0-ubuntu20.04
11+
SCROLL_LIB_PATH = /scroll/lib
1012

1113
libzkp:
1214
cd $(PWD)/rollup/circuitcapacitychecker/libzkp && make libzkp
1315

14-
nccc_geth: ## geth without circuit capacity checker
15-
$(GORUN) build/ci.go install ./cmd/geth
16+
libzstd:
17+
@sudo mkdir -p $(SCROLL_LIB_PATH)
18+
@sudo wget -O $(SCROLL_LIB_PATH)/libscroll_zstd.so https://github.com/scroll-tech/da-codec/releases/download/$(LIBSCROLL_ZSTD_VERSION)/libscroll_zstd.so
19+
20+
nccc_geth: libzstd ## geth without circuit capacity checker
21+
@LD_LIBRARY_PATH="$(LD_LIBRARY_PATH):$(SCROLL_LIB_PATH)" CGO_LDFLAGS="-L$(SCROLL_LIB_PATH) -Wl,-rpath,$(SCROLL_LIB_PATH)" $(GORUN) build/ci.go install ./cmd/geth
1622
@echo "Done building."
1723
@echo "Run \"$(GOBIN)/geth\" to launch geth."
1824

19-
geth: libzkp
25+
geth: libzkp libzstd
2026
$(GORUN) build/ci.go install -buildtags circuit_capacity_checker ./cmd/geth
2127
@echo "Done building."
2228
@echo "Run \"$(GOBIN)/geth\" to launch geth."
@@ -36,17 +42,14 @@ ios:
3642
@echo "Done building."
3743
@echo "Import \"$(GOBIN)/Geth.framework\" to use the library."
3844

39-
test: all
40-
# genesis test
41-
cd ${PWD}/cmd/geth; go test -test.run TestCustomGenesis
42-
# module test
43-
$(GORUN) build/ci.go test ./consensus ./core ./eth ./miner ./node ./trie ./rollup/fees
45+
test:
46+
./run_test.sh
4447

4548
lint: ## Run linters.
4649
$(GORUN) build/ci.go lint
4750

4851
clean:
49-
env GO111MODULE=on go clean -cache
52+
go clean -cache
5053
rm -fr build/_workspace/pkg/ $(GOBIN)/*
5154

5255
# The devtools target installs tools required for 'go generate'.

‎go.mod‎

Lines changed: 22 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,18 @@
11
module github.com/scroll-tech/go-ethereum
22

3-
go 1.20
3+
go 1.21
44

55
require (
66
github.com/Azure/azure-storage-blob-go v0.7.0
7-
github.com/VictoriaMetrics/fastcache v1.6.0
7+
github.com/VictoriaMetrics/fastcache v1.12.1
88
github.com/aws/aws-sdk-go-v2 v1.2.0
99
github.com/aws/aws-sdk-go-v2/config v1.1.1
1010
github.com/aws/aws-sdk-go-v2/credentials v1.1.1
1111
github.com/aws/aws-sdk-go-v2/service/route53 v1.1.1
1212
github.com/btcsuite/btcd v0.20.1-beta
1313
github.com/cespare/cp v0.1.0
1414
github.com/cloudflare/cloudflare-go v0.14.0
15-
github.com/consensys/gnark-crypto v0.10.0
15+
github.com/consensys/gnark-crypto v0.12.1
1616
github.com/crate-crypto/go-kzg-4844 v1.0.0
1717
github.com/davecgh/go-spew v1.1.1
1818
github.com/deckarep/golang-set v0.0.0-20180603214616-504e848d77ea
@@ -23,9 +23,9 @@ require (
2323
github.com/fatih/color v1.7.0
2424
github.com/fjl/memsize v0.0.0-20190710130421-bcb5799ab5e5
2525
github.com/gballet/go-libpcsclite v0.0.0-20190607065134-2772fd86a8ff
26-
github.com/go-stack/stack v1.8.0
26+
github.com/go-stack/stack v1.8.1
2727
github.com/golang/protobuf v1.4.3
28-
github.com/golang/snappy v0.0.4
28+
github.com/golang/snappy v0.0.5-0.20220116011046-fa5810519dcb
2929
github.com/google/gofuzz v1.1.1-0.20200604201612-c04b05f3adfa
3030
github.com/google/uuid v1.1.5
3131
github.com/gorilla/websocket v1.4.2
@@ -35,7 +35,7 @@ require (
3535
github.com/holiman/bloomfilter/v2 v2.0.3
3636
github.com/holiman/uint256 v1.2.4
3737
github.com/huin/goupnp v1.0.2
38-
github.com/iden3/go-iden3-crypto v0.0.12
38+
github.com/iden3/go-iden3-crypto v0.0.15
3939
github.com/influxdata/influxdb v1.8.3
4040
github.com/influxdata/influxdb-client-go/v2 v2.4.0
4141
github.com/jackpal/go-nat-pmp v1.0.2-0.20160603034137-1fa385a6f458
@@ -50,16 +50,17 @@ require (
5050
github.com/prometheus/tsdb v0.7.1
5151
github.com/rjeczalik/notify v0.9.1
5252
github.com/rs/cors v1.7.0
53+
github.com/scroll-tech/da-codec v0.0.0-20240516115958-db04f5e6772c
5354
github.com/scroll-tech/zktrie v0.8.4
54-
github.com/shirou/gopsutil v3.21.4-0.20210419000835-c7a38de76ee5+incompatible
55+
github.com/shirou/gopsutil v3.21.11+incompatible
5556
github.com/status-im/keycard-go v0.0.0-20190316090335-8537d3370df4
56-
github.com/stretchr/testify v1.8.2
57+
github.com/stretchr/testify v1.9.0
5758
github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7
5859
github.com/tyler-smith/go-bip39 v1.0.1-0.20181017060643-dbb3b84ba2ef
59-
golang.org/x/crypto v0.14.0
60-
golang.org/x/sync v0.4.0
61-
golang.org/x/sys v0.13.0
62-
golang.org/x/text v0.13.0
60+
golang.org/x/crypto v0.17.0
61+
golang.org/x/sync v0.6.0
62+
golang.org/x/sys v0.17.0
63+
golang.org/x/text v0.14.0
6364
golang.org/x/time v0.0.0-20210220033141-f8bda1e9f3ba
6465
gopkg.in/natefinch/npipe.v2 v2.0.0-20160621034901-c1b8fa8bdcce
6566
gopkg.in/olebedev/go-duktape.v3 v3.0.0-20200619000410-60c24ae608a6
@@ -69,36 +70,37 @@ require (
6970
require (
7071
github.com/Azure/azure-pipeline-go v0.2.2 // indirect
7172
github.com/Azure/go-autorest/autorest/adal v0.9.23 // indirect
72-
github.com/StackExchange/wmi v0.0.0-20180116203802-5d049714c4a6 // indirect
7373
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.0.2 // indirect
7474
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.0.2 // indirect
7575
github.com/aws/aws-sdk-go-v2/service/sso v1.1.1 // indirect
7676
github.com/aws/aws-sdk-go-v2/service/sts v1.1.1 // indirect
7777
github.com/aws/smithy-go v1.1.0 // indirect
78-
github.com/bits-and-blooms/bitset v1.5.0 // indirect
79-
github.com/cespare/xxhash/v2 v2.1.1 // indirect
78+
github.com/bits-and-blooms/bitset v1.12.0 // indirect
79+
github.com/cespare/xxhash/v2 v2.2.0 // indirect
8080
github.com/consensys/bavard v0.1.13 // indirect
8181
github.com/deepmap/oapi-codegen v1.8.2 // indirect
8282
github.com/dlclark/regexp2 v1.4.1-0.20201116162257-a2a8dda75c91 // indirect
83-
github.com/go-ole/go-ole v1.2.1 // indirect
83+
github.com/go-ole/go-ole v1.3.0 // indirect
8484
github.com/go-sourcemap/sourcemap v2.1.3+incompatible // indirect
8585
github.com/gotestyourself/gotestyourself v1.4.0 // indirect
8686
github.com/influxdata/line-protocol v0.0.0-20210311194329-9aa0e372d097 // indirect
8787
github.com/kylelemons/godebug v1.1.0 // indirect
8888
github.com/mattn/go-ieproxy v0.0.0-20190702010315-6dee0af9227d // indirect
89-
github.com/mattn/go-runewidth v0.0.9 // indirect
89+
github.com/mattn/go-runewidth v0.0.15 // indirect
9090
github.com/mitchellh/mapstructure v1.4.1 // indirect
9191
github.com/mitchellh/pointerstructure v1.2.0 // indirect
9292
github.com/mmcloughlin/addchain v0.4.0 // indirect
9393
github.com/naoina/go-stringutil v0.1.0 // indirect
9494
github.com/opentracing/opentracing-go v1.1.0 // indirect
9595
github.com/pkg/errors v0.9.1 // indirect
9696
github.com/pmezard/go-difflib v1.0.0 // indirect
97+
github.com/rivo/uniseg v0.4.4 // indirect
9798
github.com/supranational/blst v0.3.11-0.20230124161941-ca03e11a3ff2 // indirect
98-
github.com/tklauser/go-sysconf v0.3.10 // indirect
99-
github.com/tklauser/numcpus v0.4.0 // indirect
99+
github.com/tklauser/go-sysconf v0.3.12 // indirect
100+
github.com/tklauser/numcpus v0.6.1 // indirect
101+
github.com/yusufpapurcu/wmi v1.2.3 // indirect
100102
golang.org/x/net v0.16.0 // indirect
101-
golang.org/x/term v0.13.0 // indirect
103+
golang.org/x/term v0.15.0 // indirect
102104
google.golang.org/protobuf v1.23.0 // indirect
103105
gopkg.in/yaml.v2 v2.4.0 // indirect
104106
gopkg.in/yaml.v3 v3.0.1 // indirect

0 commit comments

Comments
 (0)