v2.9.0

Added

• chainlink health CLI command and HTML /health endpoint, to provide human-readable views of the underlying JSON health data.
• New job type stream to represent streamspecs. This job type is not yet used anywhere but will be required for Data Streams V1.
• Environment variables CL_MEDIAN_ENV, CL_SOLANA_ENV, and CL_STARKNET_ENV for setting environment variables in LOOP Plugins with an .env file.

  echo "Foo=Bar" >> median.env
  echo "Baz=Val" >> median.env  
  CL_MEDIAN_ENV="median.env"
  

Fixed

• Fixed the encoding used for transactions when resending in batches

Removed

• P2P.V1 is no longer supported and must not be set in TOML configuration in order to boot. Use P2P.V2 instead. If you are using both, V1 can simply be removed.
• Removed TelemetryIngress.URL and TelemetryIngress.ServerPubKey from TOML configuration, these fields are replaced by [[TelemetryIngress.Endpoints]]:

  [[TelemetryIngress.Endpoints]]
  Network = '...' # e.g. EVM. Solana, Starknet, Cosmos
  ChainID = '...' # e.g. 1, 5, devnet, mainnet-beta
  URL = '...'
  ServerPubKey = '...'

v2.8.0

Added

• Added distributed tracing in the OpenTelemetry trace format to the node, currently focused at the LOOPP Plugin development effort. This includes a new set of Tracing TOML configurations. The default for collecting traces is off - you must explicitly enable traces and setup a valid OpenTelemetry collector. Refer to .github/tracing/README.md for more details.
• Added a new, optional WebServer authentication option that supports LDAP as a user identity provider. This enables user login access and user roles to be managed and provisioned via a centralized remote server that supports the LDAP protocol, which can be helpful when running multiple nodes. See the documentation for more information and config setup instructions. There is a new [WebServer].AuthenticationMethod config option, when set to ldap requires the new [WebServer.LDAP] config section to be defined, see the reference docs/core.toml.
• New prom metrics for mercury transmit queue:
  mercury_transmit_queue_delete_error_count
mercury_transmit_queue_insert_error_count
mercury_transmit_queue_push_error_count
  Nops should consider alerting on these.
• Mercury now implements a local cache for fetching prices for fees, which ought to reduce latency and load on the mercury server, as well as increasing performance. It is enabled by default and can be configured with the following new config variables:

  [Mercury]
  
  # Mercury.Cache controls settings for the price retrieval cache querying a mercury server
  [Mercury.Cache]
  # LatestReportTTL controls how "stale" we will allow a price to be e.g. if
  # set to 1s, a new price will always be fetched if the last result was
  # from 1 second ago or older.
  # 
  # Another way of looking at it is such: the cache will _never_ return a
  # price that was queried from now-LatestReportTTL or before.
  # 
  # Setting to zero disables caching entirely.
  LatestReportTTL = "1s" # Default
  # MaxStaleAge is that maximum amount of time that a value can be stale
  # before it is deleted from the cache (a form of garbage collection).
  # 
  # This should generally be set to something much larger than
  # LatestReportTTL. Setting to zero disables garbage collection.
  MaxStaleAge = "1h" # Default
  # LatestReportDeadline controls how long to wait for a response from the
  # mercury server before retrying. Setting this to zero will wait indefinitely.
  LatestReportDeadline = "5s" # Default
  

• New prom metrics for the mercury cache:
  mercury_cache_fetch_failure_count
mercury_cache_hit_count
mercury_cache_wait_count
mercury_cache_miss_count
• Added new EVM.OCR TOML config fields DeltaCOverride and DeltaCJitterOverride for overriding the config DeltaC.
• Mercury v0.2 has improved consensus around current block that uses the most recent 5 blocks instead of only the latest one
• Two new prom metrics for mercury, nops should consider adding alerting on these:
  • mercury_insufficient_blocks_count
  • mercury_zero_blocks_count

Changed

• PromReporter no longer directly reads txm related status from the db, and instead uses the txStore API.
• L2Suggested mode is now called SuggestedPrice
• Console logs will now escape (non-whitespace) control characters
• Following EVM Pool metrics were renamed:
  • evm_pool_rpc_node_states → multi_node_states
  • evm_pool_rpc_node_num_transitions_to_alive → pool_rpc_node_num_transitions_to_alive
  • evm_pool_rpc_node_num_transitions_to_in_sync → pool_rpc_node_num_transitions_to_in_sync
  • evm_pool_rpc_node_num_transitions_to_out_of_sync → pool_rpc_node_num_transitions_to_out_of_sync
  • evm_pool_rpc_node_num_transitions_to_unreachable → pool_rpc_node_num_transitions_to_unreachable
  • evm_pool_rpc_node_num_transitions_to_invalid_chain_id → pool_rpc_node_num_transitions_to_invalid_chain_id
  • evm_pool_rpc_node_num_transitions_to_unusable → pool_rpc_node_num_transitions_to_unusable
  • evm_pool_rpc_node_highest_seen_block → pool_rpc_node_highest_seen_block
  • evm_pool_rpc_node_num_seen_blocks → pool_rpc_node_num_seen_blocks
  • evm_pool_rpc_node_polls_total → pool_rpc_node_polls_total
  • evm_pool_rpc_node_polls_failed → pool_rpc_node_polls_failed
  • evm_pool_rpc_node_polls_success → pool_rpc_node_polls_success

Removed

• Removed Optimism2 as a supported gas estimator mode

Fixed

• Corrected Ethereum Sepolia LinkContractAddress to 0x779877A7B0D9E8603169DdbD7836e478b4624789
• Fixed a bug that caused the Telemetry Manager to report incorrect health

Upcoming Required Configuration Changes

Starting in v2.9.0:

• TelemetryIngress.URL and TelemetryIngress.ServerPubKey will no longer be allowed. Any TOML configuration that sets this fields will prevent the node from booting. These fields will be replaced by [[TelemetryIngress.Endpoints]]
• P2P.V1 will no longer be supported and must not be set in TOML configuration in order to boot. Use P2P.V2 instead. If you are using both, V1 can simply be removed.

v2.7.2

Fixed

• Fixed a bug that caused nodes without OCR or OCR2 enabled to fail config validation if P2P.V2 was not explicitly disabled. With this fix, NOPs will not have to make changes to their config.

v2.7.1

Fixed

• Fixed a bug that causes the node to shutdown if all configured RPC's are unreachable during startup.

v2.7.0

Added

• Added new configuration field named LeaseDuration for EVM.NodePool that will periodically check if internal subscriptions are connected to the "best" (as defined by the SelectionMode) node and switch to it if necessary. Setting this value to 0s will disable this feature.
• Added multichain telemetry support. Each network/chainID pair must be configured using the new fields:

[[TelemetryIngress.Endpoints]]
Network = '...' # e.g. EVM. Solana, Starknet, Cosmos
ChainID = '...' # e.g. 1, 5, devnet, mainnet-beta
URL = '...'
ServerPubKey = '...'

These will eventually replace TelemetryIngress.URL and TelemetryIngress.ServerPubKey. Setting TelemetryIngress.URL and TelemetryIngress.ServerPubKey alongside [[TelemetryIngress.Endpoints]] will prevent the node from booting. Only one way of configuring telemetry endpoints is supported.

• Added bridge_name label to pipeline_tasks_total_finished prometheus metric. This should make it easier to see directly what bridge was failing out from the CL NODE perspective.

• LogPoller will now use finality tags to dynamically determine finality on evm chains if UseFinalityTags=true, rather than the fixed FinalityDepth specified in toml config

Changed

• P2P.V1 is now disabled (Enabled = false) by default. It must be explicitly enabled with true to be used. However, it is deprecated and will be removed in the future.
• P2P.V2 is now enabled (Enabled = true) by default.

Upcoming Required Configuration Changes

Starting in v2.9.0:

• TelemetryIngress.URL and TelemetryIngress.ServerPubKey will no longer be allowed. Any TOML configuration that sets this fields will prevent the node from booting. These fields will be replaced by [[TelemetryIngress.Endpoints]]
• P2P.V1 will no longer be supported and must not be set in TOML configuration in order to boot. Use P2P.V2 instead. If you are using both, V1 can simply be removed.

Removed

• Removed the ability to set a next nonce value for an address through CLI

v2.6.0

Added

• Simple password use in production builds is now disallowed - nodes with this configuration will not boot and will not pass config validation.
• Helper migrations function for injecting env vars into goose migrations. This was done to inject chainID into evm chain id not null in specs migrations.
• OCR2 jobs now support querying the state contract for configurations if it has been deployed. This can help on chains such as BSC which "manage" state bloat by arbitrarily deleting logs older than a certain date. In this case, if logs are missing we will query the contract directly and retrieve the latest config from chain state. Chainlink will perform no extra RPC calls unless the job spec has this feature explicitly enabled. On chains that require this, nops may see an increase in RPC calls. This can be enabled for OCR2 jobs by specifying ConfigContractAddress in the relay config TOML.

Removed

• Removed support for sending telemetry to the deprecated Explorer service. All nodes will have to remove Explorer related keys from TOML configuration and env vars.
• Removed default evmChainID logic where evmChainID was implicitly injected into the jobspecs based on node EVM chainID toml configuration. All newly created jobs(that have evmChainID field) will have to explicitly define evmChainID in the jobspec.
• Removed keyset migration that migrated v1 keys to v2 keys. All keys should've been migrated by now, and we don't permit creation of new v1 keys anymore

All nodes will have to remove the following secret configurations:

• Explorer.AccessKey
• Explorer.Secret

All nodes will have to remove the following configuration field: ExplorerURL

Fixed

• Unauthenticated users executing CLI commands previously generated a confusing error log, which is now removed:
  [ERROR] Error in transaction, rolling back: session missing or expired, please login again pg/transaction.go:118
• Fixed a bug that was preventing job runs to be displayed when the job chainID was disabled.
• chainlink txs evm create returns a transaction hash for the attempted transaction in the CLI. Previously only the sender, recipient and unstarted state were returned.
• Fixed a bug where evmChainId is requested instead of id or evm-chain-id in CLI error verbatim
• Fixed a bug that would cause the node to shut down while performing backup
• Fixed health checker to include more services in the prometheus health metric and HTTP /health endpoint

v2.5.0

Upcoming Required Configuration Change

• Starting in 2.6.0, chainlink nodes will no longer allow insecure configuration for production builds. Any TOML configuration that sets the following line will fail validation checks in node start or node validate:

AllowSimplePasswords=true

• To migrate on production builds, update the database password set in Database.URL to be 16 - 50 characters without leading or trailing whitespace. URI parsing rules apply to the chosen password - refer to RFC 3986 for special character escape rules.

Added

• Various Functions improvements

v2.4.0

Fixed

• Updated v2/keys/evm and v2/keys/eth routes to return 400 and 404 status codes where appropriate. Previously 500s were returned when requested resources were not found or client requests could not be parsed.
• Fixed withdrawing ETH from CL node for EIP1559 enabled chains. Previously would error out unless validation was overridden with allowHigherAmounts.

Added

• Added the ability to specify and merge fields from multiple secrets files. Overrides of fields and keys are not allowed.

Upcoming Required Configuration Change

• Starting in 2.6.0, chainlink nodes will no longer allow insecure configuration for production builds. Any TOML configuration that sets the following line will fail validation checks in node start or node validate:

AllowSimplePasswords=true

• To migrate on production builds, update the database password set in Database.URL to be 16 - 50 characters without leading or trailing whitespace. URI parsing rules apply to the chosen password - refer to RFC 3986 for special character escape rules.

v2.3.0

Added

• Add a new field called Order (range from 1 to 100) to EVM.Nodes that is used for the PriorityLevel node selector and also as a tie-breaker for HighestHead and TotalDifficulty. Order levels are considered in ascending order. If not defined it will default to Order = 100 (last level).
• Added new node selection mode called PriorityLevel for EVM, it is a tiered round-robin in ascending order of theOrder field. Example:

[EVM.NodePool]
SelectionMode = 'PriorityLevel'

[[EVM.Nodes]]
Name = '...'
WSURL = '...'
HTTPURL = '...'
Order = 5 

• The config keys WebServer.StartTimeout and WebServer.HTTPMaxSize. These keys respectively set a timeout for the node server to
  start and set the max request size for HTTP requests. Previously these attributes were set by
  JobPipeline.DefaultHTTPLimit/JobPipeline.DefaultHTTPTimeout. To migrate to these new fields, set their values to be identical to
  JobPipeline.DefaultHTTPLimit/JobPipeline.DefaultHTTPTimeout.

• Low latency oracle jobs now support in-protocol block range guarantees. This
  is necessary in order to produce reports with block number ranges that do not
  overlap. It can now be guaranteed at the protocol level, so we can use local
  state instead of relying on an unreliable round-trip to the Mercury server.

• New settings Evm.GasEstimator.LimitJobType.OCR2, OCR2.DefaultTransactionQueueDepth, OCR2.SimulateTransactions for OCR2
  jobs. These replace the settings Evm.GasEstimator.LimitJobType.OCR, OCR.DefaultTransactionQueueDepth, and OCR.SimulateTransactions
  for OCR2.

• Add new config parameter to OCR and OCR2 named TraceLogging that enables trace logging of OCR and OCR2 jobs, previously this behavior was controlled from the P2P.TraceLogging parameter. To maintain the same behavior set OCR.TraceLogging and OCR2.TraceLogging to the same value P2P.TraceLogging was set.

• Add two new config parameters WebServer.ListenIP and WebServer.TLS.ListenIP which allows binding Chainlink HTTP/HTTPS servers to a particular IP. The default is '0.0.0.0' which listens to all IP addresses (same behavior as before). Set to '127.0.0.1' to only allow connections from the local machine (this can be handy for local development).

• Add several new metrics for mercury feeds, related to WSRPC connections:

  • mercury_transmit_timeout_count
  • mercury_dial_count
  • mercury_dial_success_count
  • mercury_dial_error_count
  • mercury_connection_reset_count

Node operators may wish to add alerting based around these metrics.

Fixed

• Fixed a bug in the nodes xxx list command that caused results to not be displayed correctly

Changed

• Assumption violations for MaxFeePerGas >= BaseFeePerGas and MaxFeePerGas >= MaxPriorityFeePerGas in EIP-1559 effective gas price calculation will now use a gas price if specified
• Config validation now enforces protection against duplicate chain ids and node fields per provided TOML file. Duplicates accross multiple configuration files are still valid. If you have specified duplicate chain ids or nodes in a given configuration file, this change will error out of all node subcommands.
• Restricted scope of the Evm.GasEstimator.LimitJobType.OCR, OCR.DefaultTransactionQueueDepth, and OCR.SimulateTransactions settings so they
  apply only to OCR. Previously these settings would apply to OCR2 as well as OCR. You must use the OCR2 equivalents added above if you
  want your settings to apply to OCR2.

Removed

• Legacy chain types Optimism and Optimism2. OptimismBedrock is now used to handle Optimism's special cases.
• Optimism Kovan configurations along with legacy error messages.

v2.2.0

Added

• Experimental support of runtime process isolation for Solana data feeds. Requires plugin binaries to be installed and
  configured via the env vars CL_SOLANA_CMD and CL_MEDIAN_CMD. See plugins/README.md.

Fixed

• Fixed a bug which made it impossible to re-send the same transaction after abandoning it while manually changing the nonce.

Changed

• Set default for EVM.GasEstimator.BumpTxDepth to EVM.Transactions.MaxInFlight.
• Bumped batch size defaults for EVM specific configuration. If you are overriding any of these fields in your local config, please consider if it is necessary:
  • LogBackfillBatchSize = 1000
  • RPCDefaultBatchSize = 250
  • GasEstimator.BatchSize = 25
• Dropped support for Development Mode configuration. CL_DEV is now ignored on production builds.
• Updated Docker image's PostgreSQL client (used for backups) to v15 in order to support PostgreSQL v15 servers.