Automatically run a server in a subprocess if no API URL is specified

[desertaxle]

Overview

This PR changes the mechanics of running an ephemeral server with Prefect. Instead of running the API in memory when no API URL is configured, Prefect will run a server in a subprocess only if ephemeral mode is explicitly allowed via settings. This will allow feature sets that rely on WebSockets (like events) to run even if the client is not configured to communicate with a persistent server.

New settings

This PR introduces the following settings:

• PREFECT_SERVER_ALLOW_EPHEMERAL_MODE
  • Controls whether or not a subprocess server will be started we an API URL is not set. Defaults to False.
• PREFECT_SERVER_EPHEMERAL_STARTUP_WAIT_SECONDS
  • Controls the maximum time to wait for a subprocess server to be available before raising an exception. Defaults to 10.

I'm open to alternative names for both of these settings

SubprocessASGIServer

The SubprocessASGIServer class can start and stop subprocess servers as needed. The class will return the same instance if called multiple times with the same port argument and .start() is idempotent to prevent running multiple subprocess servers. Started servers will shutdown at process exit via an atexit hook, or can be stopped manually via stop. SubprocessASGIServer is consumed by get_client and similar utilities used to retrieve a client. These utilities will start a subprocess server if the settings enable them to do so.

Test suite changes

Many tests implicitly relied on the in-memory implementation of the ephemeral server. To account for the new behavior, this PR updates the use_hosted_api_server fixture to be used by default in all tests. Each worker process will run a server in a subprocess which will be used for the duration of the test run. Most test changes are to address side effects from changing how the ephemeral server works.

Related to #14716

Checklist

• This pull request references any related issue by including "closes <link to issue>"
  • If no issue exists and your change is not a small fix, please create an issue first.
• If this pull request adds new functionality, it includes unit tests that cover the changes
• If this pull request removes docs files, it includes redirect settings in mint.json.
• If this pull request adds functions or classes, it includes helpful docstrings.

[codspeed-hq]

CodSpeed Performance Report

Merging #14722 will improve performances by 26.89%

_{Comparing subprocess-server (cec7998) with main (dc34a12)}

Summary

⚡ 1 improvements
✅ 4 untouched benchmarks

Benchmarks breakdown

	Benchmark	main	subprocess-server	Change
⚡	bench_flow_call[options1]	305.9 ms	241.1 ms	+26.89%

[zzstoatzz]

this is looking pretty good to me! just a couple comments / thoughts