Skip to content

Commit

Permalink
Instruct model server to bailout if cannot becoming active after 1hr.
Browse files Browse the repository at this point in the history
PiperOrigin-RevId: 651222555
Change-Id: I35571d2f7e9287cf96d505d9c2c320dd42749beb
  • Loading branch information
Sax Authors authored and copybara-github committed Jul 11, 2024
1 parent 1a77046 commit dd75f08
Showing 1 changed file with 8 additions and 5 deletions.
13 changes: 8 additions & 5 deletions saxml/server/model_service_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -841,12 +841,15 @@ def _wait_for_backend_to_wake_up(self) -> None:
self._pending_wake_up = False
still_dormant = self._is_backend_dormant()
if still_dormant:
# Backend cannot be woken up after waiting for 1 hour, give-up and leave
# some messages.
logging.error(
'Timeout waiting for backend to wake up, backend is still dormant.'
# Backend cannot be woken up after waiting for 1 hour, bailout and leave
# message for user to start a model server with new backend.
logging.fatal(
'Abort as backend is still dormant after 1 hour of attempting to'
' wake. This is abnormal and probably suggest a bug or resource'
' exhausted. Please spin-up a model server with new backend instead'
' of waiting for current one back to active, and you may file a bug'
' via go/rightsizer-bug.'
)
return
logging.info(
'Backend woken up successfully after %s seconds.',
time.time() - begin_at,
Expand Down

0 comments on commit dd75f08

Please sign in to comment.