Feature/mc 514 consume logs from client side #226

mc-dorzo · 2025-01-07T15:38:30Z

CLI Logging Commands:
* Added a log command group to the CLI, allowing users to view logs in real-time with optional filters.
* Support for filtering logs by specific components:
* --guideline-proposer (-g)
* --tool-caller (-t)
* --message-event-generator (-m)
* Support for logical operators (AND/OR) between multiple filters.
* Added support for additional pattern matching on log messages.
* Example CLI commands:

# Streaming all logs
parlant log

# Filter logs by component
parlant log --guideline-proposer

# Combine filters using AND
parlant log -g Evaluation --operator AND

# Add a pattern filter
parlant log --tool-caller [ToolResult]

kichanyurd · 2025-01-13T21:14:26Z

src/parlant/bin/client.py

+    def stream_logs(ctx: click.Context, filters: list[str]) -> Iterator[dict[str, Any]]:
+        try:
+            context = zmq.Context.instance()
+            sub_socket = context.socket(zmq.SUB)


I think sub_socket needs to be initialized before the try for the finally section to make sense

kichanyurd · 2025-01-13T21:16:18Z

src/parlant/bin/client.py

+
+            while True:
+                try:
+                    message = cast(dict[str, Any], sub_socket.recv_json(flags=zmq.NOBLOCK))


Why are we using NOBLOCK here?
And then sleeping for 0.01s?
That's a lot of idle spinning

removed the block. I thought in the begining it will be async and maybe pulling multiple topics. nvm. ^^

kichanyurd · 2025-01-13T21:21:20Z

src/parlant/bin/client.py

@@ -1984,10 +2033,21 @@ class Config:
        metavar="ADDRESS[:PORT]",
        default="http://localhost:8800",
    )
+    @click.option(
+        "--log-server-address",


How it's written now we would have to write the following to access a remote server:

parlant --server http://1.2.3.4:8800 --log-server-address tcp://1.2.3.4:8799 [...]

Instead let's assume that the log server is on the same host as the server itself, and just allow specifying a different port. So the option would be --log-port and we'd use the host name from the --server option.

kichanyurd · 2025-01-13T21:24:00Z

src/parlant/core/engines/alpha/guideline_proposer.py

@@ -233,7 +233,7 @@ async def _process_guideline_batch(
            guideline = guidelines_dict[int(proposition.guideline_number)]


Let's add 2 logs here:

[GuidelineProposer][Prompt] (debug)
[GuidelineProposer][Completion] (debug) --> output all of the response's content as indented JSON

kichanyurd · 2025-01-13T21:25:46Z

src/parlant/core/engines/alpha/message_event_generator.py

                        return [MessageEventGenerationResult(generation_info, [])]
                except Exception as exc:
                    self._logger.warning(
-                        f"Generation attempt {generation_attempt} failed: {traceback.format_exception(exc)}"
+                        f"[MessageEventGenerator] Generation attempt {generation_attempt} failed: {traceback.format_exception(exc)}"


In MEG let's also add the log
[MessageEventGenerator][Completion] --> logging all of the generated content as indented JSON.

And BTW let's do the same for ToolCaller too... so we have [Prompt] and [Completion] for all components.

kichanyurd · 2025-01-13T21:27:07Z

src/parlant/core/engines/alpha/message_event_generator.py

@@ -171,7 +171,7 @@ async def generate_events(
                    )

                    if response_message is not None:
-                        self._logger.debug(f'Message production result: "{response_message}"')
+                        self._logger.debug(f'[MessageEventGenerator][Result]: "{response_message}"')


Rename, this isn't just "result", this is [MessageEventGenerator][GeneratedMessage]

kichanyurd · 2025-01-13T21:28:01Z

src/parlant/core/engines/alpha/message_event_generator.py

@@ -518,7 +522,7 @@ async def _generate_response_message(
            and not final_revision.is_repeat_message
        ):
            self._logger.warning(
-                f"PROBLEMATIC MESSAGE EVENT PRODUCER RESPONSE: {final_revision.content}"
+                f"[MessageEventGenerator] PROBLEMATIC MESSAGE EVENT PRODUCER RESPONSE: {final_revision.content}"


I think you'll have to rebase here as this part was changed already

kichanyurd · 2025-01-13T21:28:42Z

src/parlant/core/engines/alpha/tool_caller.py

@@ -555,7 +557,7 @@ async def _run_inference(

        log_message = json.dumps(inference.content.model_dump(mode="json"), indent=2)

-        self._logger.debug(f"Tool call request results: {log_message}")
+        self._logger.debug(f"[ToolCaller][RequestResults]: {log_message}")


Rename to [ToolCaller][Completion]

kichanyurd · 2025-01-13T21:29:47Z

src/parlant/core/engines/alpha/message_event_generator.py

@@ -142,7 +142,7 @@ async def generate_events(
                shots=await self.shots(),
            )

-            self._logger.debug(f"Message production prompt:\n{prompt}")
+            self._logger.debug(f"[MessageEventGenerator][Prompt]:\n{prompt}")


I see you sometimes have a colon (:) after the [..] part and sometimes don't.
Let's be consistent: go over all instances and let's go with no colon. Just a space after the last bracket (in this case a newline is fine instead of the space since it's a multiline prompt)

kichanyurd · 2025-01-13T21:32:35Z

src/parlant/core/logging.py

+            yield
+        finally:
+            for context in reversed(contexts):
+                context.__exit__(None, None, None)


There's a bug here. What if we got an exception? We'll be calling __exit__ here without one.

Why not use ExitStack to write this cleanly?

kichanyurd · 2025-01-13T21:33:23Z

tests/core/stable/test_loggers.py

+    sub_socket.connect(f"tcp://localhost:{PARLANT_LOG_PORT}")
+    sub_socket.setsockopt_string(zmq.SUBSCRIBE, "")
+
+    await asyncio.sleep(1)


Why do we need to sleep here?

It need to to connect. otherwise we the messages are sent without the subscriber receiving in

…parlant log "Revisions" "Evaluation"

…now exclusively listen to the log topic. - Replaced the `--log-server-address` parameter with `--log-port` for specifying the log server port.

…ntext management

mc-dorzo requested a review from kichanyurd January 7, 2025 15:38

mc-dorzo self-assigned this Jan 7, 2025

kichanyurd reviewed Jan 13, 2025

View reviewed changes

mc-dorzo added 17 commits January 13, 2025 21:47

Add ZMQLogger

46ae49a

Add ZMQLogger and test for subscribing

27afe2d

Add CLI for streaming logs

eb22312

Add [GuidelineProposer] to logs in guideline proposer

3b162f8

Add [MessageEventGenerator] to logs in messge_event_generator

58b8a64

Add extra bracketed prefix to tool caller logs

cf960c2

Add extra bracketed prefix to guideline proposer logs

ffc6faa

Fix default port for log serer

f9de41d

Add CompositeLogger and implement in server

4904c0f

Add pattern as filters

79ee39f

Add patterns as list of strings. now we can use streaming logs i.e. `…

62c7eea

…parlant log "Revisions" "Evaluation"

Remove douple brackets in ZMQLogger

2c668b4

Move exmaples.modules to tests.modules

8c57dc8

- Changed logging functionality from non-blocking to blocking, as we …

73ddc75

…now exclusively listen to the log topic. - Replaced the `--log-server-address` parameter with `--log-port` for specifying the log server port.

Use ExitStack in CompositeLogger operation functionality for safer co…

d7f5e01

…ntext management

Fix e2e test to support moving examples.modules to tests.modules

d6d243c

Polishing loggings conventions

2e7217c

mc-dorzo force-pushed the feature/mc-514-consume-logs-from-client-side branch from 9355906 to 2e7217c Compare January 14, 2025 15:56

mc-dorzo added 5 commits January 14, 2025 19:39

update CHANGELOG

e6eaed1

update changlog

773845b

Add union and intersection pattern flags

2ce78ed

Polishing logs

6d22677

update changelog

444274e

mc-dorzo merged commit 851c690 into develop Jan 15, 2025
1 of 2 checks passed

mc-dorzo deleted the feature/mc-514-consume-logs-from-client-side branch January 15, 2025 13:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/mc 514 consume logs from client side #226

Feature/mc 514 consume logs from client side #226

mc-dorzo commented Jan 7, 2025 •

edited

Loading

kichanyurd Jan 13, 2025

kichanyurd Jan 13, 2025

mc-dorzo Jan 14, 2025

kichanyurd Jan 13, 2025

kichanyurd Jan 13, 2025 •

edited

Loading

kichanyurd Jan 13, 2025

kichanyurd Jan 13, 2025

kichanyurd Jan 13, 2025

kichanyurd Jan 13, 2025

kichanyurd Jan 13, 2025

kichanyurd Jan 13, 2025 •

edited

Loading

kichanyurd Jan 13, 2025

mc-dorzo Jan 13, 2025

		@@ -233,7 +233,7 @@ async def _process_guideline_batch(
		guideline = guidelines_dict[int(proposition.guideline_number)]

Feature/mc 514 consume logs from client side #226

Feature/mc 514 consume logs from client side #226

Conversation

mc-dorzo commented Jan 7, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kichanyurd Jan 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kichanyurd Jan 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mc-dorzo commented Jan 7, 2025 •

edited

Loading

kichanyurd Jan 13, 2025 •

edited

Loading

kichanyurd Jan 13, 2025 •

edited

Loading