Prefect Server Requires All Flow Dependancies To Be Installed At Deployment Time

[MrChadMWood]

First check

• I added a descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the Prefect documentation for this issue.
• I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

The Prefect server, being hosted separately from the Prefect agent, for some reason requires all 3rd party packages (e.g., Pandas) to be installed when building the deployment.

This is unexpected because why should the server require a copy of these packages when the agent will be running the code?

Reproduction

My setup:  
Running with Docker, image: `prefecthq/prefect:2.10.16-python3.11`. I have the following docker-compose:

version: "3.9"
services:

  ### Prefect Database
  operational_db:
    image: postgres:15.2-alpine
    restart: always
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=prefect
    expose:
      - 5432
    volumes: 
      - db:/var/lib/postgresql/data
    profiles: ["server"]

  ### Prefect Server API and UI
  server:
    image: prefecthq/prefect:2.10.16-python3.11
    restart: always
    volumes:
      - prefect:/root/.prefect
    entrypoint: ["/opt/prefect/entrypoint.sh", "prefect", "server", "start"]
    environment:
      - PREFECT_UI_URL=http://127.0.0.0:4200/api
      - PREFECT_API_URL=http://127.0.0.1:4200/api
      - PREFECT_SERVER_API_HOST=0.0.0.0
      - PREFECT_API_DATABASE_CONNECTION_URL=postgresql+asyncpg://postgres:postgres@operational_db:5432/prefect
    ports:
      - 4200:4200
    depends_on:
      - operational_db
    profiles: ["server"]

  ### Prefect Agent
  agent:
    image: prefecthq/prefect:2.10.16-python3.11
    restart: always
    volumes:
      - temp:/root/temp
    entrypoint: ["prefect", "agent", "start", "-q", "default"]
    environment:
      - PREFECT_API_URL=http://server:4200/api
      - KEEPASS_DB=/root/keepass/pefect-keys.kdbx
      - DATABASE_CONNECTION_URL=postgresql+asyncpg://postgres:postgres@database:5433/reporting_automations
    profiles: ["agent"]

  ### Prefect  CLI
  cli:
    image: prefecthq/prefect:2.10.16-python3.11
    entrypoint: "bash"
    working_dir: "/root"
    volumes:
      - ./flows:/root/flows
      - ./utils:/root/utils
    environment:
      - PREFECT_API_URL=http://server:4200/api
      - DATABASE_CONNECTION_URL=postgresql+asyncpg://postgres:postgres@database:5433/reporting_automations
    profiles: ["cli"]

volumes:
  prefect:
  flows:
  temp: 

networks:
  default:
    name: prefect-network

I make sure that the agent has all dependencies by actually using a modified image for the agent.
Then I go into the CLI and move to the /root/flows directory, and I git clone a flow into there. Finally I try to deploy it, knowing that the agent contains all dependencies.

root@bd3b3cbd2eb8:~/flows/prefect-flows# prefect deployment build -sb gitlab-repository/flows -n test-cases-depl -q default cases.py:testcases

### Error

```python3
Script at 'cases.py' encountered an exception: ModuleNotFoundError("No module named 'pandas'")

### Versions

```Text
Version:             2.10.16
API version:         0.8.4
Python version:      3.11.4
Git commit:          6cd7c3ee
Built:               Tue, Jun 20, 2023 2:59 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         server

Additional context

No response

[pierreloicq]

Seems similar to #9512