diff --git a/.github/workflows/fork-sync.yml b/.github/workflows/fork-sync.yml new file mode 100644 index 000000000000..a2196fb742f4 --- /dev/null +++ b/.github/workflows/fork-sync.yml @@ -0,0 +1,18 @@ +name: Sync Fork + +on: + schedule: + - cron: '*/30 * * * *' # every 30 minutes + workflow_dispatch: # on button click + +jobs: + sync: + + runs-on: ubuntu-latest + + steps: + - uses: tgymnich/fork-sync@v1.8 + with: + base: celestia-develop + head: develop + token: ${{ secrets.REPO_PAT }} diff --git a/README.md b/README.md index 9c189f9662c2..943a798a65f8 100644 --- a/README.md +++ b/README.md @@ -2,8 +2,11 @@

Optimism +

Optimism is a low-cost and lightning-fast Ethereum L2 blockchain, built with the OP Stack.


-

Optimism is Ethereum, scaled.

+

+

+ Celestia +

Celestia is a modular data availability network that securely scales with the number of users, making it easy for anyone to launch their own blockchain.


@@ -28,9 +31,25 @@ ## What is Optimism? +If you're looking to run the OP Stack + Celestia setup for this repository, please visit the [Optimism & Celestia guides and tutorials](https://docs.celestia.org/developers/intro-to-op-stack/) to get started. + [Optimism](https://www.optimism.io/) is a project dedicated to scaling Ethereum's technology and expanding its ability to coordinate people from across the world to build effective decentralized economies and governance systems. The [Optimism Collective](https://app.optimism.io/announcement) builds open-source software for running L2 blockchains and aims to address key governance and economic challenges in the wider cryptocurrency ecosystem. Optimism operates on the principle of **impact=profit**, the idea that individuals who positively impact the Collective should be proportionally rewarded with profit. **Change the incentives and you change the world.** -In this repository, you'll find numerous core components of the OP Stack, the decentralized software stack maintained by the Optimism Collective that powers Optimism and forms the backbone of blockchains like [OP Mainnet](https://explorer.optimism.io/) and [Base](https://base.org). Designed to be "aggressively open source," the OP Stack encourages you to explore, modify, extend, and test the code as needed. Although not all elements of the OP Stack are contained here, many of its essential components can be found within this repository. By collaborating on free, open software and shared standards, the Optimism Collective aims to prevent siloed software development and rapidly accelerate the development of the Ethereum ecosystem. Come contribute, build the future, and redefine power, together. +The OP Stack powers Optimism, an Ethereum L2 blockchain, and forms the technical foundation for the [the Optimism Collective](https://app.optimism.io/announcement)—a group committed to the **impact=profit** principle. This principle rewards individuals for their positive contributions to the collective. + +Optimism addresses critical coordination failures in the crypto ecosystem, such as funding public goods and infrastructure. The OP Stack focuses on creating a shared, open-source system for developing new L2 blockchains within the proposed Superchain ecosystem, promoting collaboration and preventing redundant efforts. + +As Optimism evolves, the OP Stack will adapt, encompassing components ranging from blockchain infrastructure to governance systems. This software suite aims to simplify L2 blockchain creation while supporting the growth and development of the Optimism ecosystem. + +## What is Celestia? + +Celestia is a modular consensus and data network, built to enable anyone to easily deploy their own blockchain with minimal overhead. + +Celestia is a minimal blockchain that only orders and publishes transactions and does not execute them. By decoupling the consensus and application execution layers, Celestia modularizes the blockchain technology stack and unlocks new possibilities for decentralized application builders. Lean more at [Celestia.org](https://celestia.org). + +## Maintenance + +The maintenance guide for this repository can be found in the Wiki tab of the repository or [here](https://github.com/celestiaorg/optimism/wiki). ## Documentation @@ -40,24 +59,67 @@ In this repository, you'll find numerous core components of the OP Stack, the de ## Specification If you're interested in the technical details of how Optimism works, refer to the [Optimism Protocol Specification](https://github.com/ethereum-optimism/specs). +If you want to build on top of Celestia, take a look at the documentation at [docs.celestia.org](https://docs.celestia.org). + +If you want to learn more about the OP Stack, check out the documentation at [stack.optimism.io](https://stack.optimism.io/). ## Community +### Optimism + General discussion happens most frequently on the [Optimism discord](https://discord.gg/optimism). Governance discussion can also be found on the [Optimism Governance Forum](https://gov.optimism.io/). +### Celestia + +General discussion happens most frequently on the [Celestia discord](https://discord.com/invite/YsnTPcSfWQ). +Other discussions can be found on the [Celestia forum](https://forum.celestia.org). + ## Contributing -Read through [CONTRIBUTING.md](./CONTRIBUTING.md) for a general overview of the contributing process for this repository. +Read through [CONTRIBUTING.md](./CONTRIBUTING.md) for a general overview of our contribution process. Use the [Developer Quick Start](./CONTRIBUTING.md#development-quick-start) to get your development environment set up to start working on the Optimism Monorepo. Then check out the list of [Good First Issues](https://github.com/ethereum-optimism/optimism/issues?q=is:open+is:issue+label:D-good-first-issue) to find something fun to work on! Typo fixes are welcome; however, please create a single commit with all of the typo fixes & batch as many fixes together in a PR as possible. Spammy PRs will be closed. -## Security Policy and Vulnerability Reporting +## e2e testing + +This repository has updated end-to-end tests in the `op-e2e` package to work with +Celestia as the data availability (DA) layer. + +Currently, the tests assume a working [Celestia devnet](https://github.com/rollkit/local-celestia-devnet) running locally: + +```bash +docker run -p 26650:26650 ghcr.io/rollkit/local-celestia-devnet:v0.12.7 +``` + +The e2e tests can be triggered with: + +```bash +cd $HOME/optimism +cd op-e2e +OP_E2E_DISABLE_PARALLEL=true OP_E2E_CANNON_ENABLED=false OP_NODE_DA_RPC=localhost:26650 OP_BATCHER_DA_RPC=localhost:26650 make test +``` + +## Bridging + +If you have the OP Stack + Celestia setup running, you can test out bridging from the L1 +to the L2. + +To do this, first navigate to the `packages/contracts-bedrock` directory and create a +`.env` file with the following contents: + +```bash +L1_PROVIDER_URL=http://localhost:8545 +L2_PROVIDER_URL=http://localhost:9545 +PRIVATE_KEY=bf7604d9d3a1c7748642b1b7b05c2bd219c9faa91458b370f85e5a40f3b03af7 +``` + +Then, run the following from the same directory: -Please refer to the canonical [Security Policy](https://github.com/ethereum-optimism/.github/blob/master/SECURITY.md) document for detailed information about how to report vulnerabilities in this codebase. -Bounty hunters are encouraged to check out [the Optimism Immunefi bug bounty program](https://immunefi.com/bounty/optimism/). -The Optimism Immunefi program offers up to $2,000,042 for in-scope critical vulnerabilities. +```bash +npx hardhat deposit --network devnetL1 --l1-provider-url http://localhost:8545 --l2-provider-url http://localhost:9545 --amount-eth --to
+``` ## Directory Structure diff --git a/bedrock-devnet/devnet/__init__.py b/bedrock-devnet/devnet/__init__.py index 3565916d25a3..c0389349e7a3 100644 --- a/bedrock-devnet/devnet/__init__.py +++ b/bedrock-devnet/devnet/__init__.py @@ -271,12 +271,18 @@ def devnet_deploy(paths): else: docker_env['L2OO_ADDRESS'] = l2_output_oracle - if DEVNET_PLASMA: - docker_env['PLASMA_ENABLED'] = 'true' - docker_env['PLASMA_DA_SERVICE'] = 'false' - else: - docker_env['PLASMA_ENABLED'] = 'false' - docker_env['PLASMA_DA_SERVICE'] = 'false' + log.info('Bringing up DA') + run_command(['docker', 'compose', 'up', '-d', 'da'], cwd=paths.ops_bedrock_dir, env=docker_env) + wait_up(26658) + time.sleep(15) + celestia_node_auth_token = os.getenv("CELESTIA_NODE_AUTH_TOKEN") + if not celestia_node_auth_token: + celestia_node_auth_token = run_command([ + 'docker', 'compose', 'exec', '--no-TTY', 'da', 'celestia', 'bridge', 'auth', 'admin', '--node.store', '/home/celestia/bridge' + ], cwd=paths.ops_bedrock_dir, env=docker_env, capture_output=True).stdout.decode().strip() + docker_env['CELESTIA_NODE_AUTH_TOKEN'] = celestia_node_auth_token + print('CELESTIA_NODE_AUTH_TOKEN: ', celestia_node_auth_token) + # Bring up the rest of the services. log.info('Bringing up `op-node`, `op-proposer` and `op-batcher`.') @@ -373,10 +379,11 @@ def run_command_preset(command: CommandPreset): return proc.returncode -def run_command(args, check=True, shell=False, cwd=None, env=None, timeout=None): +def run_command(args, check=True, shell=False, cwd=None, env=None, timeout=None, capture_output=False): env = env if env else {} return subprocess.run( args, + capture_output=capture_output, check=check, shell=shell, env={ diff --git a/docs/op-stack/src/assets/docs/understand/Celestia-logo-color-color.svg b/docs/op-stack/src/assets/docs/understand/Celestia-logo-color-color.svg new file mode 100644 index 000000000000..6d22cb408925 --- /dev/null +++ b/docs/op-stack/src/assets/docs/understand/Celestia-logo-color-color.svg @@ -0,0 +1 @@ +Celestia-logo-color-black \ No newline at end of file diff --git a/go.mod b/go.mod index 826e3bddc27f..3029c3e42646 100644 --- a/go.mod +++ b/go.mod @@ -1,6 +1,8 @@ module github.com/ethereum-optimism/optimism -go 1.21 +go 1.21.1 + +toolchain go1.21.10 require ( github.com/BurntSushi/toml v1.3.2 @@ -45,6 +47,7 @@ require ( github.com/pkg/errors v0.9.1 github.com/pkg/profile v1.7.0 github.com/prometheus/client_golang v1.19.0 + github.com/rollkit/go-da v0.5.0 github.com/stretchr/testify v1.9.0 github.com/urfave/cli/v2 v2.27.1 golang.org/x/crypto v0.23.0 @@ -93,6 +96,7 @@ require ( github.com/fatih/color v1.15.0 // indirect github.com/felixge/fgprof v0.9.3 // indirect github.com/ferranbt/fastssz v0.1.2 // indirect + github.com/filecoin-project/go-jsonrpc v0.3.1 // indirect github.com/fjl/memsize v0.0.2 // indirect github.com/flynn/noise v1.0.0 // indirect github.com/francoispqt/gojay v1.2.13 // indirect @@ -107,6 +111,7 @@ require ( github.com/gofrs/flock v0.8.1 // indirect github.com/gogo/protobuf v1.3.2 // indirect github.com/golang-jwt/jwt/v4 v4.5.0 // indirect + github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 // indirect github.com/golang/protobuf v1.5.3 // indirect github.com/google/gopacket v1.1.19 // indirect github.com/google/pprof v0.0.0-20231023181126-ff6d637d2a7b // indirect @@ -207,6 +212,7 @@ require ( github.com/tyler-smith/go-bip39 v1.1.0 // indirect github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 // indirect github.com/yusufpapurcu/wmi v1.2.3 // indirect + go.opencensus.io v0.22.3 // indirect go.uber.org/automaxprocs v1.5.2 // indirect go.uber.org/dig v1.17.1 // indirect go.uber.org/fx v1.20.1 // indirect @@ -218,6 +224,9 @@ require ( golang.org/x/sys v0.20.0 // indirect golang.org/x/text v0.15.0 // indirect golang.org/x/tools v0.17.0 // indirect + golang.org/x/xerrors v0.0.0-20220517211312-f3a8303e98df // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20231002182017-d307bd883b97 // indirect + google.golang.org/grpc v1.60.1 // indirect google.golang.org/protobuf v1.32.0 // indirect gopkg.in/ini.v1 v1.67.0 // indirect gopkg.in/natefinch/lumberjack.v2 v2.0.0 // indirect diff --git a/go.sum b/go.sum index 190e9eaaaa93..9b8231be6477 100644 --- a/go.sum +++ b/go.sum @@ -188,6 +188,8 @@ github.com/felixge/fgprof v0.9.3 h1:VvyZxILNuCiUCSXtPtYmmtGvb65nqXh2QFWc0Wpf2/g= github.com/felixge/fgprof v0.9.3/go.mod h1:RdbpDgzqYVh/T9fPELJyV7EYJuHB55UTEULNun8eiPw= github.com/ferranbt/fastssz v0.1.2 h1:Dky6dXlngF6Qjc+EfDipAkE83N5I5DE68bY6O0VLNPk= github.com/ferranbt/fastssz v0.1.2/go.mod h1:X5UPrE2u1UJjxHA8X54u04SBwdAQjG2sFtWs39YxyWs= +github.com/filecoin-project/go-jsonrpc v0.3.1 h1:qwvAUc5VwAkooquKJmfz9R2+F8znhiqcNHYjEp/NM10= +github.com/filecoin-project/go-jsonrpc v0.3.1/go.mod h1:jBSvPTl8V1N7gSTuCR4bis8wnQnIjHbRPpROol6iQKM= github.com/fjl/memsize v0.0.2 h1:27txuSD9or+NZlnOWdKUxeBzTAUkWCVh+4Gf2dWFOzA= github.com/fjl/memsize v0.0.2/go.mod h1:VvhXpOYNQvB+uIk2RvXzuaQtkQJzzIx6lSBe1xv7hi0= github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc= @@ -256,6 +258,8 @@ github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69 github.com/golang-jwt/jwt/v4 v4.5.0 h1:7cYmW1XlMY7h7ii7UhUyChSgS5wUJEnm9uZVTGqOWzg= github.com/golang-jwt/jwt/v4 v4.5.0/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= +github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 h1:ZgQEtGgCBiWRM39fZuwSd1LwSqqSW0hOdXCYYDX0R3I= +github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/lint v0.0.0-20180702182130-06c8688daad7/go.mod h1:tluoj9z5200jBnyusfRPU2LqT6J+DAorxEvtC7LHB+E= github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= @@ -712,6 +716,8 @@ github.com/rogpeppe/go-internal v1.6.1/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTE github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs= github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog= +github.com/rollkit/go-da v0.5.0 h1:sQpZricNS+2TLx3HMjNWhtRfqtvVC/U4pWHpfUz3eN4= +github.com/rollkit/go-da v0.5.0/go.mod h1:VsUeAoPvKl4Y8wWguu/VibscYiFFePkkrvZWyTjZHww= github.com/rs/cors v1.9.0 h1:l9HGsTsHJcvW14Nk7J9KFz8bzeAWXn3CG6bgt7LsrAE= github.com/rs/cors v1.9.0/go.mod h1:XyqrcTp5zjWr1wsJ8PIRZssZ8b/WMcMf71DJnit4EMU= github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ= @@ -814,6 +820,8 @@ github.com/yusufpapurcu/wmi v1.2.3 h1:E1ctvB7uKFMOJw3fdOW32DwGE9I7t++CRUEMKvFoFi github.com/yusufpapurcu/wmi v1.2.3/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0= github.com/zenazn/goji v0.9.0/go.mod h1:7S9M489iMyHBNxwZnk9/EHS098H4/F6TATF2mIxtB1Q= go.opencensus.io v0.18.0/go.mod h1:vKdFvxhtzZ9onBp9VKHK8z/sRpBMnKAsufL7wlDrCOA= +go.opencensus.io v0.22.3 h1:8sGtKOrtQqkN1bp2AtX+misvLIlOmsEsNd+9NIcPEm8= +go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= go.uber.org/atomic v1.5.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ= @@ -876,6 +884,7 @@ golang.org/x/exp v0.0.0-20240112132812-db7319d0e0e3/go.mod h1:idGWGoKP1toJGkd5/i golang.org/x/lint v0.0.0-20180702182130-06c8688daad7/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= +golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc= @@ -949,6 +958,7 @@ golang.org/x/sys v0.0.0-20190316082340-a2f829d7f35f/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20190403152447-81d4e9dc473e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -1051,6 +1061,7 @@ golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8T golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20220517211312-f3a8303e98df h1:5Pf6pFKu98ODmgnpvkJ3kFUOQGGLIzLIkbzUHp47618= golang.org/x/xerrors v0.0.0-20220517211312-f3a8303e98df/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8= google.golang.org/api v0.0.0-20180910000450-7ca32eb868bf/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0= google.golang.org/api v0.0.0-20181030000543-1d582fd0359e/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0= @@ -1064,10 +1075,16 @@ google.golang.org/genproto v0.0.0-20180831171423-11092d34479b/go.mod h1:JiN7NxoA google.golang.org/genproto v0.0.0-20181029155118-b69ba1387ce2/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= google.golang.org/genproto v0.0.0-20181202183823-bd91e49a0898/go.mod h1:7Ep/1NZk928CDR8SjdVbjWNpdIf6nzjE3BTgJDr2Atg= google.golang.org/genproto v0.0.0-20190306203927-b5d61aea6440/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/genproto/googleapis/rpc v0.0.0-20231002182017-d307bd883b97 h1:6GQBEOdGkX6MMTLT9V+TjtIRZCw9VPD5Z+yHY9wMgS0= +google.golang.org/genproto/googleapis/rpc v0.0.0-20231002182017-d307bd883b97/go.mod h1:v7nGkzlmW8P3n/bKmWBn2WpBjpOEx8Q6gMueudAmKfY= google.golang.org/grpc v1.14.0/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw= google.golang.org/grpc v1.16.0/go.mod h1:0JHn/cJsOMiMfNA9+DeHDlAU7KAAB5GDlYFpa9MZMio= google.golang.org/grpc v1.17.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs= google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= +google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38= +google.golang.org/grpc v1.60.1 h1:26+wFr+cNqSGFcOXcabYC0lUVJVRa2Sb2ortSK7VrEU= +google.golang.org/grpc v1.60.1/go.mod h1:OlCHIeLYqSSsLi6i49B5QGdzaMZK9+M7LXN2FKz4eGM= google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= diff --git a/op-batcher/batcher/config.go b/op-batcher/batcher/config.go index cc37961a056a..0e594199ec1c 100644 --- a/op-batcher/batcher/config.go +++ b/op-batcher/batcher/config.go @@ -11,6 +11,7 @@ import ( "github.com/ethereum-optimism/optimism/op-batcher/compressor" "github.com/ethereum-optimism/optimism/op-batcher/flags" "github.com/ethereum-optimism/optimism/op-node/rollup/derive" + celestia "github.com/ethereum-optimism/optimism/op-celestia" plasma "github.com/ethereum-optimism/optimism/op-plasma" oplog "github.com/ethereum-optimism/optimism/op-service/log" opmetrics "github.com/ethereum-optimism/optimism/op-service/metrics" @@ -100,7 +101,8 @@ type CLIConfig struct { MetricsConfig opmetrics.CLIConfig PprofConfig oppprof.CLIConfig RPC oprpc.CLIConfig - PlasmaDA plasma.CLIConfig + PlasmaDA plasma.CLIConfig + DaConfig celestia.CLIConfig } func (c *CLIConfig) Check() error { @@ -155,6 +157,9 @@ func (c *CLIConfig) Check() error { if err := c.RPC.Check(); err != nil { return err } + if err := c.DaConfig.Check(); err != nil { + return err + } return nil } @@ -188,5 +193,6 @@ func NewConfig(ctx *cli.Context) *CLIConfig { PprofConfig: oppprof.ReadCLIConfig(ctx), RPC: oprpc.ReadCLIConfig(ctx), PlasmaDA: plasma.ReadCLIConfig(ctx), + DaConfig: celestia.ReadCLIConfig(ctx), } } diff --git a/op-batcher/batcher/driver.go b/op-batcher/batcher/driver.go index ef0a2fbe15ac..3af4d806404e 100644 --- a/op-batcher/batcher/driver.go +++ b/op-batcher/batcher/driver.go @@ -2,6 +2,7 @@ package batcher import ( "context" + "encoding/hex" "errors" "fmt" "io" @@ -11,6 +12,7 @@ import ( "time" "github.com/ethereum-optimism/optimism/op-batcher/metrics" + celestia "github.com/ethereum-optimism/optimism/op-celestia" "github.com/ethereum-optimism/optimism/op-node/rollup" "github.com/ethereum-optimism/optimism/op-node/rollup/derive" plasma "github.com/ethereum-optimism/optimism/op-plasma" @@ -49,6 +51,7 @@ type DriverSetup struct { EndpointProvider dial.L2EndpointProvider ChannelConfig ChannelConfig PlasmaDA *plasma.DAClient + DAClient *celestia.DAClient } // BatchSubmitter encapsulates a service responsible for submitting L2 tx @@ -507,7 +510,11 @@ func (l *BatchSubmitter) sendTransaction(ctx context.Context, txdata txData, que // signal plasma commitment tx with TxDataVersion1 data = comm.TxData() } - candidate = l.calldataTxCandidate(data) + candidate, err = l.calldataTxCandidate(data) + if err != nil { + l.recordFailedTx(txdata.ID(), err) + return nil + } } intrinsicGas, err := core.IntrinsicGas(candidate.TxData, nil, false, true, true, false) @@ -538,12 +545,25 @@ func (l *BatchSubmitter) blobTxCandidate(data txData) (*txmgr.TxCandidate, error }, nil } -func (l *BatchSubmitter) calldataTxCandidate(data []byte) *txmgr.TxCandidate { +func (l *BatchSubmitter) calldataTxCandidate(data []byte) (*txmgr.TxCandidate, error) { l.Log.Info("building Calldata transaction candidate", "size", len(data)) + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Duration(l.RollupConfig.BlockTime)*time.Second) + ids, err := l.DAClient.Client.Submit(ctx, [][]byte{data}, -1, l.DAClient.Namespace) + cancel() + if err == nil && len(ids) == 1 { + l.Log.Info("celestia: blob successfully submitted", "id", hex.EncodeToString(ids[0])) + data = append([]byte{celestia.DerivationVersionCelestia}, ids[0]...) + } else { + if l.DAClient.EthFallbackDisabled { + return nil, fmt.Errorf("celestia: blob submission failed; eth fallback disabled: %w", err) + } + + l.Log.Info("celestia: blob submission failed; falling back to eth", "err", err) + } return &txmgr.TxCandidate{ To: &l.RollupConfig.BatchInboxAddress, TxData: data, - } + }, nil } func (l *BatchSubmitter) handleReceipt(r txmgr.TxReceipt[txID]) { diff --git a/op-batcher/batcher/service.go b/op-batcher/batcher/service.go index 4203f442c78e..c4c00f328c0d 100644 --- a/op-batcher/batcher/service.go +++ b/op-batcher/batcher/service.go @@ -16,6 +16,7 @@ import ( "github.com/ethereum-optimism/optimism/op-batcher/flags" "github.com/ethereum-optimism/optimism/op-batcher/metrics" "github.com/ethereum-optimism/optimism/op-batcher/rpc" + celestia "github.com/ethereum-optimism/optimism/op-celestia" "github.com/ethereum-optimism/optimism/op-node/chaincfg" "github.com/ethereum-optimism/optimism/op-node/rollup" plasma "github.com/ethereum-optimism/optimism/op-plasma" @@ -76,6 +77,7 @@ type BatcherService struct { stopped atomic.Bool NotSubmittingOnStart bool + DAClient *celestia.DAClient } // BatcherServiceFromCLIConfig creates a new BatcherService from a CLIConfig. @@ -124,6 +126,9 @@ func (bs *BatcherService) initFromCLIConfig(ctx context.Context, version string, if err := bs.initPlasmaDA(cfg); err != nil { return fmt.Errorf("failed to init plasma DA: %w", err) } + if err := bs.initDA(cfg); err != nil { + return fmt.Errorf("failed to start da server: %w", err) + } bs.initDriver() if err := bs.initRPCServer(cfg); err != nil { return fmt.Errorf("failed to start RPC server: %w", err) @@ -306,6 +311,7 @@ func (bs *BatcherService) initDriver() { EndpointProvider: bs.EndpointProvider, ChannelConfig: bs.ChannelConfig, PlasmaDA: bs.PlasmaDA, + DAClient: bs.DAClient, }) } @@ -339,6 +345,15 @@ func (bs *BatcherService) initPlasmaDA(cfg *CLIConfig) error { return nil } +func (bs *BatcherService) initDA(cfg *CLIConfig) error { + client, err := celestia.NewDAClient(cfg.DaConfig.Rpc, cfg.DaConfig.AuthToken, cfg.DaConfig.Namespace, cfg.DaConfig.EthFallbackDisabled) + if err != nil { + return err + } + bs.DAClient = client + return nil +} + // Start runs once upon start of the batcher lifecycle, // and starts batch-submission work if the batcher is configured to start submit data on startup. func (bs *BatcherService) Start(_ context.Context) error { diff --git a/op-batcher/flags/flags.go b/op-batcher/flags/flags.go index c91234b96a29..44b0b30af246 100644 --- a/op-batcher/flags/flags.go +++ b/op-batcher/flags/flags.go @@ -10,6 +10,7 @@ import ( "github.com/ethereum-optimism/optimism/op-batcher/compressor" "github.com/ethereum-optimism/optimism/op-node/rollup/derive" + celestia "github.com/ethereum-optimism/optimism/op-celestia" plasma "github.com/ethereum-optimism/optimism/op-plasma" opservice "github.com/ethereum-optimism/optimism/op-service" openum "github.com/ethereum-optimism/optimism/op-service/enum" @@ -43,7 +44,6 @@ var ( Usage: "HTTP provider URL for Rollup node. A comma-separated list enables the active L2 endpoint provider. Such a list needs to match the number of l2-eth-rpcs provided.", EnvVars: prefixEnvVars("ROLLUP_RPC"), } - // Optional flags SubSafetyMarginFlag = &cli.Uint64Flag{ Name: "sub-safety-margin", Usage: "The batcher tx submission safety margin (in #L1-blocks) to subtract " + @@ -187,6 +187,7 @@ func init() { optionalFlags = append(optionalFlags, oppprof.CLIFlags(EnvVarPrefix)...) optionalFlags = append(optionalFlags, txmgr.CLIFlags(EnvVarPrefix)...) optionalFlags = append(optionalFlags, plasma.CLIFlags(EnvVarPrefix, "")...) + optionalFlags = append(optionalFlags, celestia.CLIFlags(EnvVarPrefix)...) Flags = append(requiredFlags, optionalFlags...) } diff --git a/op-celestia/cli.go b/op-celestia/cli.go new file mode 100644 index 000000000000..919e3d95f9aa --- /dev/null +++ b/op-celestia/cli.go @@ -0,0 +1,73 @@ +package celestia + +import ( + "github.com/urfave/cli/v2" + + opservice "github.com/ethereum-optimism/optimism/op-service" +) + +const ( + // RPCFlagName defines the flag for the rpc url + RPCFlagName = "da.rpc" + // AuthTokenFlagName defines the flag for the auth token + AuthTokenFlagName = "da.auth_token" + // NamespaceFlagName defines the flag for the namespace + NamespaceFlagName = "da.namespace" + // EthFallbackDisabledFlagName defines the flag for disabling eth fallback + EthFallbackDisabledFlagName = "da.eth_fallback_disabled" + + // defaultRPC is the default rpc dial address + defaultRPC = "grpc://localhost:26650" +) + +func CLIFlags(envPrefix string) []cli.Flag { + return []cli.Flag{ + &cli.StringFlag{ + Name: RPCFlagName, + Usage: "dial address of the data availability rpc client; supports grpc, http, https", + Value: defaultRPC, + EnvVars: opservice.PrefixEnvVar(envPrefix, "DA_RPC"), + }, + &cli.StringFlag{ + Name: AuthTokenFlagName, + Usage: "authentication token of the data availability client", + EnvVars: opservice.PrefixEnvVar(envPrefix, "DA_AUTH_TOKEN"), + }, + &cli.StringFlag{ + Name: NamespaceFlagName, + Usage: "namespace of the data availability client", + EnvVars: opservice.PrefixEnvVar(envPrefix, "DA_NAMESPACE"), + }, + &cli.BoolFlag{ + Name: EthFallbackDisabledFlagName, + Usage: "disable eth fallback", + EnvVars: opservice.PrefixEnvVar(envPrefix, "DA_ETH_FALLBACK_DISABLED"), + }, + } +} + +type CLIConfig struct { + Rpc string + AuthToken string + Namespace string + EthFallbackDisabled bool +} + +func (c CLIConfig) Check() error { + return nil +} + +func NewCLIConfig() CLIConfig { + return CLIConfig{ + Rpc: defaultRPC, + } +} + +func ReadCLIConfig(ctx *cli.Context) CLIConfig { + return CLIConfig{ + Rpc: ctx.String(RPCFlagName), + AuthToken: ctx.String(AuthTokenFlagName), + Namespace: ctx.String(NamespaceFlagName), + EthFallbackDisabled: ctx.Bool(EthFallbackDisabledFlagName), + } +} diff --git a/op-celestia/da.go b/op-celestia/da.go new file mode 100644 index 000000000000..feecb3a82fad --- /dev/null +++ b/op-celestia/da.go @@ -0,0 +1,10 @@ +package celestia + +// DerivationVersionCelestia is a byte marker for celestia references submitted +// to the batch inbox address as calldata. +// Mnemonic 0xce = celestia +// version 0xce references are encoded as: +// [8]byte block height ++ [32]byte commitment +// in little-endian encoding. +// see: https://github.com/rollkit/celestia-da/blob/1f2df375fd2fcc59e425a50f7eb950daa5382ef0/celestia.go#L141-L160 +const DerivationVersionCelestia = 0xce diff --git a/op-celestia/da_client.go b/op-celestia/da_client.go new file mode 100644 index 000000000000..d3d407a10b94 --- /dev/null +++ b/op-celestia/da_client.go @@ -0,0 +1,33 @@ +package celestia + +import ( + "encoding/hex" + "time" + + "github.com/rollkit/go-da" + "github.com/rollkit/go-da/proxy" +) + +type DAClient struct { + Client da.DA + GetTimeout time.Duration + Namespace da.Namespace + EthFallbackDisabled bool +} + +func NewDAClient(rpc, token, namespace string, ethFallbackDisabled bool) (*DAClient, error) { + client, err := proxy.NewClient(rpc, token) + if err != nil { + return nil, err + } + ns, err := hex.DecodeString(namespace) + if err != nil { + return nil, err + } + return &DAClient{ + Client: client, + GetTimeout: time.Minute, + Namespace: ns, + EthFallbackDisabled: ethFallbackDisabled, + }, nil +} diff --git a/op-e2e/setup.go b/op-e2e/setup.go index 984b27a9edfe..153cbb23905e 100644 --- a/op-e2e/setup.go +++ b/op-e2e/setup.go @@ -41,6 +41,7 @@ import ( bss "github.com/ethereum-optimism/optimism/op-batcher/batcher" batcherFlags "github.com/ethereum-optimism/optimism/op-batcher/flags" + celestia "github.com/ethereum-optimism/optimism/op-celestia" "github.com/ethereum-optimism/optimism/op-chain-ops/genesis" "github.com/ethereum-optimism/optimism/op-e2e/config" "github.com/ethereum-optimism/optimism/op-e2e/e2eutils" @@ -132,6 +133,7 @@ func DefaultSystemConfig(t testing.TB) SystemConfig { RuntimeConfigReloadInterval: time.Minute * 10, ConfigPersistence: &rollupNode.DisabledConfigPersistence{}, Sync: sync.Config{SyncMode: sync.CLSync}, + DaConfig: celestia.CLIConfig{Rpc: "grpc://localhost:26650"}, }, "verifier": { Driver: driver.Config{ @@ -143,6 +145,7 @@ func DefaultSystemConfig(t testing.TB) SystemConfig { RuntimeConfigReloadInterval: time.Minute * 10, ConfigPersistence: &rollupNode.DisabledConfigPersistence{}, Sync: sync.Config{SyncMode: sync.CLSync}, + DaConfig: celestia.CLIConfig{Rpc: "grpc://localhost:26650"}, }, }, Loggers: map[string]log.Logger{ @@ -732,7 +735,8 @@ func (cfg SystemConfig) Start(t *testing.T, _opts ...SystemConfigOption) (*Syste l.Warn("closed op-node!") }() } - node, err := rollupNode.New(context.Background(), &c, l, snapLog, "", metrics.NewMetrics("")) + + node, err := rollupNode.New(context.Background(), &c, cfg.Loggers[name], snapLog, "", metrics.NewMetrics("")) if err != nil { return nil, err } @@ -855,6 +859,7 @@ func (cfg SystemConfig) Start(t *testing.T, _opts ...SystemConfigOption) (*Syste BatchType: batchType, DataAvailabilityType: sys.Cfg.DataAvailabilityType, CompressionAlgo: compressionAlgo, + DaConfig: celestia.CLIConfig{Rpc: "localhost:26650"}, } // Batch Submitter batcher, err := bss.BatcherServiceFromCLIConfig(context.Background(), "0.0.1", batcherCLIConfig, sys.Cfg.Loggers["batcher"]) diff --git a/op-node/flags/flags.go b/op-node/flags/flags.go index 579e3443fb08..31577e9b3526 100644 --- a/op-node/flags/flags.go +++ b/op-node/flags/flags.go @@ -6,6 +6,7 @@ import ( "github.com/urfave/cli/v2" + celestia "github.com/ethereum-optimism/optimism/op-celestia" "github.com/ethereum-optimism/optimism/op-node/rollup/sync" plasma "github.com/ethereum-optimism/optimism/op-plasma" openum "github.com/ethereum-optimism/optimism/op-service/enum" @@ -421,6 +422,7 @@ func init() { optionalFlags = append(optionalFlags, DeprecatedFlags...) optionalFlags = append(optionalFlags, opflags.CLIFlags(EnvVarPrefix, RollupCategory)...) optionalFlags = append(optionalFlags, plasma.CLIFlags(EnvVarPrefix, PlasmaCategory)...) + optionalFlags = append(optionalFlags, celestia.CLIFlags(EnvVarPrefix)...) Flags = append(requiredFlags, optionalFlags...) } diff --git a/op-node/node/config.go b/op-node/node/config.go index 88bba5ff85f7..238f117363cc 100644 --- a/op-node/node/config.go +++ b/op-node/node/config.go @@ -7,6 +7,7 @@ import ( "math" "time" + celestia "github.com/ethereum-optimism/optimism/op-celestia" "github.com/ethereum-optimism/optimism/op-node/flags" "github.com/ethereum-optimism/optimism/op-node/p2p" "github.com/ethereum-optimism/optimism/op-node/rollup" @@ -76,6 +77,8 @@ type Config struct { // Plasma DA config Plasma plasma.CLIConfig + + DaConfig celestia.CLIConfig } type RPCConfig struct { @@ -174,5 +177,8 @@ func (cfg *Config) Check() error { if err := cfg.Plasma.Check(); err != nil { return fmt.Errorf("plasma config error: %w", err) } + if err := cfg.DaConfig.Check(); err != nil { + return fmt.Errorf("da config error: %w", err) + } return nil } diff --git a/op-node/node/node.go b/op-node/node/node.go index 0fb1edcf5700..b11e8c11375c 100644 --- a/op-node/node/node.go +++ b/op-node/node/node.go @@ -128,6 +128,9 @@ func (n *OpNode) init(ctx context.Context, cfg *Config, snapshotLog log.Logger) if err := n.initL1BeaconAPI(ctx, cfg); err != nil { return err } + if err := n.initDA(ctx, cfg); err != nil { + return fmt.Errorf("failed to init da: %w", err) + } if err := n.initL2(ctx, cfg, snapshotLog); err != nil { return fmt.Errorf("failed to init L2: %w", err) } @@ -364,6 +367,10 @@ func (n *OpNode) initL1BeaconAPI(ctx context.Context, cfg *Config) error { } } +func (n *OpNode) initDA(ctx context.Context, cfg *Config) error { + return driver.SetDAClient(cfg.DaConfig) +} + func (n *OpNode) initL2(ctx context.Context, cfg *Config, snapshotLog log.Logger) error { rpcClient, rpcCfg, err := cfg.L2.Setup(ctx, n.log, &cfg.Rollup) if err != nil { diff --git a/op-node/rollup/derive/blob_data_source.go b/op-node/rollup/derive/blob_data_source.go index 7780b6065063..d21b3d3283d6 100644 --- a/op-node/rollup/derive/blob_data_source.go +++ b/op-node/rollup/derive/blob_data_source.go @@ -86,7 +86,10 @@ func (ds *BlobDataSource) open(ctx context.Context) ([]blobOrCalldata, error) { return nil, NewTemporaryError(fmt.Errorf("failed to open blob data source: %w", err)) } - data, hashes := dataAndHashesFromTxs(txs, &ds.dsCfg, ds.batcherAddr) + data, hashes, err := dataAndHashesFromTxs(txs, &ds.dsCfg, ds.batcherAddr) + if err != nil { + return nil, err + } if len(hashes) == 0 { // there are no blobs to fetch so we can return immediately @@ -115,11 +118,12 @@ func (ds *BlobDataSource) open(ctx context.Context) ([]blobOrCalldata, error) { // dataAndHashesFromTxs extracts calldata and datahashes from the input transactions and returns them. It // creates a placeholder blobOrCalldata element for each returned blob hash that must be populated // by fillBlobPointers after blob bodies are retrieved. -func dataAndHashesFromTxs(txs types.Transactions, config *DataSourceConfig, batcherAddr common.Address) ([]blobOrCalldata, []eth.IndexedBlobHash) { +func dataAndHashesFromTxs(txs types.Transactions, config *DataSourceConfig, batcherAddr common.Address) ([]blobOrCalldata, []eth.IndexedBlobHash, error) { data := []blobOrCalldata{} var hashes []eth.IndexedBlobHash blobIndex := 0 // index of each blob in the block's blob sidecar for _, tx := range txs { + logger := log.New("tx", tx.Hash()) // skip any non-batcher transactions if !isValidBatchTx(tx, config.l1Signer, config.batchInboxAddress, batcherAddr) { blobIndex += len(tx.BlobHashes()) @@ -127,8 +131,15 @@ func dataAndHashesFromTxs(txs types.Transactions, config *DataSourceConfig, batc } // handle non-blob batcher transactions by extracting their calldata if tx.Type() != types.BlobTxType { - calldata := eth.Data(tx.Data()) - data = append(data, blobOrCalldata{nil, &calldata}) + calldata, err := DataFromEVMTransactions(*config, batcherAddr, types.Transactions{tx}, logger) + if err != nil { + return nil, nil, err + } + if len(calldata) == 0 { + log.Warn("celestia: skipping empty calldata") + continue + } + data = append(data, blobOrCalldata{nil, &calldata[0]}) continue } // handle blob batcher transactions by extracting their blob hashes, ignoring any calldata. @@ -145,7 +156,7 @@ func dataAndHashesFromTxs(txs types.Transactions, config *DataSourceConfig, batc blobIndex += 1 } } - return data, hashes + return data, hashes, nil } // fillBlobPointers goes back through the data array and fills in the pointers to the fetched blob diff --git a/op-node/rollup/derive/blob_data_source_test.go b/op-node/rollup/derive/blob_data_source_test.go index aa9ef82cb991..0fea0cd56255 100644 --- a/op-node/rollup/derive/blob_data_source_test.go +++ b/op-node/rollup/derive/blob_data_source_test.go @@ -42,7 +42,8 @@ func TestDataAndHashesFromTxs(t *testing.T) { } calldataTx, _ := types.SignNewTx(privateKey, signer, txData) txs := types.Transactions{calldataTx} - data, blobHashes := dataAndHashesFromTxs(txs, &config, batcherAddr) + data, blobHashes, err := dataAndHashesFromTxs(txs, &config, batcherAddr) + require.NoError(t, err) require.Equal(t, 1, len(data)) require.Equal(t, 0, len(blobHashes)) @@ -57,14 +58,16 @@ func TestDataAndHashesFromTxs(t *testing.T) { } blobTx, _ := types.SignNewTx(privateKey, signer, blobTxData) txs = types.Transactions{blobTx} - data, blobHashes = dataAndHashesFromTxs(txs, &config, batcherAddr) + data, blobHashes, err = dataAndHashesFromTxs(txs, &config, batcherAddr) + require.NoError(t, err) require.Equal(t, 1, len(data)) require.Equal(t, 1, len(blobHashes)) require.Nil(t, data[0].calldata) // try again with both the blob & calldata transactions and make sure both are picked up txs = types.Transactions{blobTx, calldataTx} - data, blobHashes = dataAndHashesFromTxs(txs, &config, batcherAddr) + data, blobHashes, err = dataAndHashesFromTxs(txs, &config, batcherAddr) + require.NoError(t, err) require.Equal(t, 2, len(data)) require.Equal(t, 1, len(blobHashes)) require.NotNil(t, data[1].calldata) @@ -72,7 +75,8 @@ func TestDataAndHashesFromTxs(t *testing.T) { // make sure blob tx to the batch inbox is ignored if not signed by the batcher blobTx, _ = types.SignNewTx(testutils.RandomKey(), signer, blobTxData) txs = types.Transactions{blobTx} - data, blobHashes = dataAndHashesFromTxs(txs, &config, batcherAddr) + data, blobHashes, err = dataAndHashesFromTxs(txs, &config, batcherAddr) + require.NoError(t, err) require.Equal(t, 0, len(data)) require.Equal(t, 0, len(blobHashes)) @@ -81,7 +85,8 @@ func TestDataAndHashesFromTxs(t *testing.T) { blobTxData.To = testutils.RandomAddress(rng) blobTx, _ = types.SignNewTx(privateKey, signer, blobTxData) txs = types.Transactions{blobTx} - data, blobHashes = dataAndHashesFromTxs(txs, &config, batcherAddr) + data, blobHashes, err = dataAndHashesFromTxs(txs, &config, batcherAddr) + require.NoError(t, err) require.Equal(t, 0, len(data)) require.Equal(t, 0, len(blobHashes)) } diff --git a/op-node/rollup/derive/calldata_source.go b/op-node/rollup/derive/calldata_source.go index 0a5d7915770d..3ad40d9a467a 100644 --- a/op-node/rollup/derive/calldata_source.go +++ b/op-node/rollup/derive/calldata_source.go @@ -2,6 +2,7 @@ package derive import ( "context" + "encoding/hex" "errors" "fmt" "io" @@ -11,9 +12,20 @@ import ( "github.com/ethereum/go-ethereum/core/types" "github.com/ethereum/go-ethereum/log" + celestia "github.com/ethereum-optimism/optimism/op-celestia" "github.com/ethereum-optimism/optimism/op-service/eth" ) +var daClient *celestia.DAClient + +func SetDAClient(c *celestia.DAClient) error { + if daClient != nil { + return errors.New("da client already configured") + } + daClient = c + return nil +} + // CalldataSource is a fault tolerant approach to fetching data. // The constructor will never fail & it will instead re-attempt the fetcher // at a later point. @@ -32,7 +44,7 @@ type CalldataSource struct { // NewCalldataSource creates a new calldata source. It suppresses errors in fetching the L1 block if they occur. // If there is an error, it will attempt to fetch the result on the next call to `Next`. -func NewCalldataSource(ctx context.Context, log log.Logger, dsCfg DataSourceConfig, fetcher L1TransactionFetcher, ref eth.L1BlockRef, batcherAddr common.Address) DataIter { +func NewCalldataSource(ctx context.Context, log log.Logger, dsCfg DataSourceConfig, fetcher L1TransactionFetcher, ref eth.L1BlockRef, batcherAddr common.Address) (DataIter, error) { _, txs, err := fetcher.InfoAndTxsByHash(ctx, ref.Hash) if err != nil { return &CalldataSource{ @@ -42,12 +54,23 @@ func NewCalldataSource(ctx context.Context, log log.Logger, dsCfg DataSourceConf fetcher: fetcher, log: log, batcherAddr: batcherAddr, - } + }, nil + } + data, err := DataFromEVMTransactions(dsCfg, batcherAddr, txs, log.New("origin", ref)) + if err != nil { + return &CalldataSource{ + open: false, + ref: ref, + dsCfg: dsCfg, + fetcher: fetcher, + log: log, + batcherAddr: batcherAddr, + }, err } return &CalldataSource{ open: true, - data: DataFromEVMTransactions(dsCfg, batcherAddr, txs, log.New("origin", ref)), - } + data: data, + }, nil } // Next returns the next piece of data if it has it. If the constructor failed, this @@ -57,7 +80,11 @@ func (ds *CalldataSource) Next(ctx context.Context) (eth.Data, error) { if !ds.open { if _, txs, err := ds.fetcher.InfoAndTxsByHash(ctx, ds.ref.Hash); err == nil { ds.open = true - ds.data = DataFromEVMTransactions(ds.dsCfg, ds.batcherAddr, txs, ds.log) + ds.data, err = DataFromEVMTransactions(ds.dsCfg, ds.batcherAddr, txs, ds.log) + if err != nil { + // already wrapped + return nil, err + } } else if errors.Is(err, ethereum.NotFound) { return nil, NewResetError(fmt.Errorf("failed to open calldata source: %w", err)) } else { @@ -76,12 +103,38 @@ func (ds *CalldataSource) Next(ctx context.Context) (eth.Data, error) { // DataFromEVMTransactions filters all of the transactions and returns the calldata from transactions // that are sent to the batch inbox address from the batch sender address. // This will return an empty array if no valid transactions are found. -func DataFromEVMTransactions(dsCfg DataSourceConfig, batcherAddr common.Address, txs types.Transactions, log log.Logger) []eth.Data { +func DataFromEVMTransactions(dsCfg DataSourceConfig, batcherAddr common.Address, txs types.Transactions, log log.Logger) ([]eth.Data, error) { out := []eth.Data{} for _, tx := range txs { if isValidBatchTx(tx, dsCfg.l1Signer, dsCfg.batchInboxAddress, batcherAddr) { - out = append(out, tx.Data()) + data := tx.Data() + switch len(data) { + case 0: + out = append(out, data) + default: + switch data[0] { + case celestia.DerivationVersionCelestia: + log.Info("celestia: blob request", "id", hex.EncodeToString(tx.Data())) + ctx, cancel := context.WithTimeout(context.Background(), daClient.GetTimeout) + blobs, err := daClient.Client.Get(ctx, [][]byte{data[1:]}, daClient.Namespace) + cancel() + if err != nil { + return nil, NewResetError(fmt.Errorf("celestia: failed to resolve frame: %w", err)) + } + if len(blobs) != 1 { + log.Warn("celestia: unexpected length for blobs", "expected", 1, "got", len(blobs)) + if len(blobs) == 0 { + log.Warn("celestia: skipping empty blobs") + continue + } + } + out = append(out, blobs[0]) + default: + out = append(out, data) + log.Info("celestia: using eth fallback") + } + } } } - return out + return out, nil } diff --git a/op-node/rollup/derive/calldata_source_test.go b/op-node/rollup/derive/calldata_source_test.go index 01b2616cca3f..7dc17c6aa19f 100644 --- a/op-node/rollup/derive/calldata_source_test.go +++ b/op-node/rollup/derive/calldata_source_test.go @@ -121,8 +121,10 @@ func TestDataFromEVMTransactions(t *testing.T) { } } - out := DataFromEVMTransactions(DataSourceConfig{cfg.L1Signer(), cfg.BatchInboxAddress, false}, batcherAddr, txs, testlog.Logger(t, log.LevelCrit)) + out, err := DataFromEVMTransactions(DataSourceConfig{cfg.L1Signer(), cfg.BatchInboxAddress, false}, batcherAddr, txs, testlog.Logger(t, log.LevelCrit)) + require.NoError(t, err) require.ElementsMatch(t, expectedData, out) + require.NoError(t, err) } } diff --git a/op-node/rollup/derive/data_source.go b/op-node/rollup/derive/data_source.go index 6cb43a94165f..be5c2295d3e4 100644 --- a/op-node/rollup/derive/data_source.go +++ b/op-node/rollup/derive/data_source.go @@ -72,13 +72,17 @@ func (ds *DataSourceFactory) OpenData(ctx context.Context, ref eth.L1BlockRef, b // Creates a data iterator from blob or calldata source so we can forward it to the plasma source // if enabled as it still requires an L1 data source for fetching input commmitments. var src DataIter + var err error if ds.ecotoneTime != nil && ref.Time >= *ds.ecotoneTime { if ds.blobsFetcher == nil { return nil, fmt.Errorf("ecotone upgrade active but beacon endpoint not configured") } src = NewBlobDataSource(ctx, ds.log, ds.dsCfg, ds.fetcher, ds.blobsFetcher, ref, batcherAddr) } else { - src = NewCalldataSource(ctx, ds.log, ds.dsCfg, ds.fetcher, ref, batcherAddr) + src, err = NewCalldataSource(ctx, ds.log, ds.dsCfg, ds.fetcher, ref, batcherAddr) + if err != nil { + return src, err + } } if ds.dsCfg.plasmaEnabled { // plasma([calldata | blobdata](l1Ref)) -> data diff --git a/op-node/rollup/driver/da.go b/op-node/rollup/driver/da.go new file mode 100644 index 000000000000..cacef1afd0c0 --- /dev/null +++ b/op-node/rollup/driver/da.go @@ -0,0 +1,14 @@ +package driver + +import ( + celestia "github.com/ethereum-optimism/optimism/op-celestia" + "github.com/ethereum-optimism/optimism/op-node/rollup/derive" +) + +func SetDAClient(cfg celestia.CLIConfig) error { + client, err := celestia.NewDAClient(cfg.Rpc, cfg.AuthToken, cfg.Namespace, false) + if err != nil { + return err + } + return derive.SetDAClient(client) +} diff --git a/op-node/service.go b/op-node/service.go index bd7627c97255..1a951c683207 100644 --- a/op-node/service.go +++ b/op-node/service.go @@ -18,6 +18,7 @@ import ( "github.com/ethereum/go-ethereum/log" "github.com/urfave/cli/v2" + celestia "github.com/ethereum-optimism/optimism/op-celestia" "github.com/ethereum-optimism/optimism/op-node/flags" "github.com/ethereum-optimism/optimism/op-node/node" p2pcli "github.com/ethereum-optimism/optimism/op-node/p2p/cli" @@ -110,7 +111,8 @@ func NewConfig(ctx *cli.Context, log log.Logger) (*node.Config, error) { ConductorRpc: ctx.String(flags.ConductorRpcFlag.Name), ConductorRpcTimeout: ctx.Duration(flags.ConductorRpcTimeoutFlag.Name), - Plasma: plasma.ReadCLIConfig(ctx), + Plasma: plasma.ReadCLIConfig(ctx), + DaConfig: celestia.ReadCLIConfig(ctx), } if err := cfg.LoadPersisted(log); err != nil { diff --git a/ops-bedrock/docker-compose.yml b/ops-bedrock/docker-compose.yml index 37a5b29ace33..47bf55fb0b0e 100644 --- a/ops-bedrock/docker-compose.yml +++ b/ops-bedrock/docker-compose.yml @@ -14,6 +14,16 @@ volumes: services: + da: + image: ghcr.io/rollkit/local-celestia-devnet:v0.13.1 + ports: + - "26658:26658" + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:26659/header/1"] + interval: 10s + timeout: 5s + retries: 5 + start_period: 30s l1: build: @@ -52,6 +62,7 @@ services: depends_on: - l1 - l2 + - da build: context: ../ dockerfile: ops/docker/op-stack-go/Dockerfile @@ -60,6 +71,7 @@ services: command: > op-node --l1=ws://l1:8546 + --l1.beacon.ignore --l2=http://l2:8551 --l2.jwt-secret=/config/test-jwt-secret.txt --sequencer.enabled @@ -81,10 +93,10 @@ services: --metrics.port=7300 --pprof.enabled --rpc.enable-admin - --safedb.path=/db - --plasma.enabled=${PLASMA_ENABLED} - --plasma.da-service=${PLASMA_DA_SERVICE} - --plasma.da-server=http://da-server:3100 + environment: + OP_NODE_DA_RPC: "http://da:26658" + OP_NODE_DA_AUTH_TOKEN: "${CELESTIA_NODE_AUTH_TOKEN}" + OP_NODE_DA_NAMESPACE: "00000000000000000000000000000000000000000008e5f679bf7116cb" ports: - "7545:8545" - "9003:9003" @@ -133,6 +145,7 @@ services: - l1 - l2 - op-node + - da build: context: ../ dockerfile: ops/docker/op-stack-go/Dockerfile @@ -146,6 +159,7 @@ services: OP_BATCHER_L1_ETH_RPC: http://l1:8545 OP_BATCHER_L2_ETH_RPC: http://l2:8545 OP_BATCHER_ROLLUP_RPC: http://op-node:8545 + OFFLINE_GAS_ESTIMATION: null OP_BATCHER_MAX_CHANNEL_DURATION: 1 OP_BATCHER_SUB_SAFETY_MARGIN: 4 # SWS is 15, ChannelTimeout is 40 OP_BATCHER_POLL_INTERVAL: 1s @@ -156,9 +170,9 @@ services: OP_BATCHER_METRICS_ENABLED: "true" OP_BATCHER_RPC_ENABLE_ADMIN: "true" OP_BATCHER_BATCH_TYPE: 0 - OP_BATCHER_PLASMA_ENABLED: "${PLASMA_ENABLED}" - OP_BATCHER_PLASMA_DA_SERVICE: "${PLASMA_DA_SERVICE}" - OP_BATCHER_PLASMA_DA_SERVER: "http://da-server:3100" + OP_BATCHER_DA_RPC: "http://da:26658" + OP_BATCHER_DA_AUTH_TOKEN: "${CELESTIA_NODE_AUTH_TOKEN}" + OP_BATCHER_DA_NAMESPACE: "00000000000000000000000000000000000000000008e5f679bf7116cb" op-challenger: depends_on: diff --git a/ops/docker/op-stack-go/Dockerfile.dockerignore b/ops/docker/op-stack-go/Dockerfile.dockerignore index 9512fe9fbc7c..7eb8a255ef48 100644 --- a/ops/docker/op-stack-go/Dockerfile.dockerignore +++ b/ops/docker/op-stack-go/Dockerfile.dockerignore @@ -6,6 +6,7 @@ !/op-batcher !/op-bindings !/op-bootnode +!/op-celestia !/op-chain-ops !/op-challenger !/packages/contracts-bedrock/snapshots diff --git a/packages/contracts-bedrock/deploy-config/devnetL1-template.json b/packages/contracts-bedrock/deploy-config/devnetL1-template.json index 88fe5509dc1c..e3c196cf50a6 100644 --- a/packages/contracts-bedrock/deploy-config/devnetL1-template.json +++ b/packages/contracts-bedrock/deploy-config/devnetL1-template.json @@ -46,8 +46,9 @@ "eip1559Elasticity": 6, "l1GenesisBlockTimestamp": "0x64c811bf", "l2GenesisRegolithTimeOffset": "0x0", - "l2GenesisDeltaTimeOffset": null, "l2GenesisCanyonTimeOffset": "0x0", + "l2GenesisDeltaTimeOffset": "0x0", + "l2GenesisEcotoneTimeOffset": "0x0", "systemConfigStartBlock": 0, "requiredProtocolVersion": "0x0000000000000000000000000000000000000000000000000000000000000000", "recommendedProtocolVersion": "0x0000000000000000000000000000000000000000000000000000000000000000",