Lease key errors in postgres logs

[wesbillman]

2024-07-15 16:00:07 UTC:10.2.155.50(41150):adminuser@tbd:[61375]:ERROR:  duplicate key value violates unique constraint "leases_key_key"
2024-07-15 16:00:07 UTC:10.2.155.50(41150):adminuser@tbd:[61375]:DETAIL:  Key (key)=(/system/scheduledtask/reapStaleControllers) already exists.
2024-07-15 16:00:07 UTC:10.2.155.50(41150):adminuser@tbd:[61375]:STATEMENT:  -- name: NewLease :one
	INSERT INTO leases (
	  idempotency_key,
	  key,
	  expires_at,
	  metadata
	)
	VALUES (
	  gen_random_uuid(),
	  $1::lease_key,
	  (NOW() AT TIME ZONE 'utc') + $2::interval,
	  $3::JSONB
	)
	RETURNING idempotency_key
	
2024-07-15 16:00:08 UTC:10.2.111.21(46692):adminuser@tbd:[60308]:ERROR:  null value in column "key" of relation "leases" violates not-null constraint
2024-07-15 16:00:08 UTC:10.2.111.21(46692):adminuser@tbd:[60308]:DETAIL:  Failing row contains (1632753, 37a8d114-73ca-46a1-b03b-a17331d17181, null, 2024-07-15 16:00:08.260772+00, 2024-07-15 16:00:13.260772+00, null).
2024-07-15 16:00:08 UTC:10.2.111.21(46692):adminuser@tbd:[60308]:STATEMENT:  -- name: AcquireAsyncCall :one
	WITH async_call AS (
	  SELECT id
	  FROM async_calls
	  WHERE state = 'pending' AND scheduled_at <= (NOW() AT TIME ZONE 'utc')
	  LIMIT 1
	  FOR UPDATE SKIP LOCKED
	), lease AS (
	  INSERT INTO leases (idempotency_key, key, expires_at)
	  VALUES (gen_random_uuid(), '/system/async_call/' || (SELECT id FROM async_call), (NOW() AT TIME ZONE 'utc') + $1::interval)
	  RETURNING id, idempotency_key, key, created_at, expires_at, metadata
	)
	UPDATE async_calls
	SET state = 'executing', lease_id = (SELECT id FROM lease)
	WHERE id = (SELECT id FROM async_call)
	RETURNING
	  id AS async_call_id,
	  (SELECT idempotency_key FROM lease) AS lease_idempotency_key,
	  (SELECT key FROM lease) AS lease_key,
	  origin,
	  verb,
	  request,
	  scheduled_at,
	  remaining_attempts,
	  backoff,
	  max_backoff

[matt2e]

2024-07-15 16:00:07 UTC:10.2.155.50(41150):adminuser@tbd:[61375]:ERROR:  duplicate key value violates unique constraint "leases_key_key"
2024-07-15 16:00:07 UTC:10.2.155.50(41150):adminuser@tbd:[61375]:DETAIL:  Key (key)=(/system/scheduledtask/reapStaleControllers) already exists.
2024-07-15 16:00:07 UTC:10.2.155.50(41150):adminuser@tbd:[61375]:STATEMENT:  -- name: NewLease :one
	INSERT INTO leases (
	  idempotency_key,
	  key,
	  expires_at,
	  metadata
	)
	VALUES (
	  gen_random_uuid(),
	  $1::lease_key,
	  (NOW() AT TIME ZONE 'utc') + $2::interval,
	  $3::JSONB
	)
	RETURNING idempotency_key

This one seems fine to me as failures due to unique key constraints are how leases are supposed to work, for example:

• 2 controllers both try and be the ASM leader
• first controller inserts lease
• second controller gets this error saying lease already exists, and falls back to being a follower
  I'd expect this error for each scheduledtask that is set to run on only one controller as well, and the log above is one of those (/system/scheduledtask/reapStaleControllers)

[matt2e]

Second case is resolved here: #2081