Instrumenting Custom Event Types

[enrico-stauss]

I created a custom node post-processor that bases the BaseNodePostprocessor. There is no suitable Start/EndEvent(BaseEvent) type for my post-processor, so I decided to add a custom event type, too, in order for my event to be displayed properly in Arize Phoenix. However this does not work, as I discovered that those events are registered explicitly in an external package (openinference.python.instrumentation.openinference-instrumentation-llama-index).

Do you know if there is a way to register my custom event without making modifications to openinference?

Kind Regards

[dosubot[bot]]

LlamaIndex does not have any dependencies on the openinference.python.instrumentation.openinference-instrumentation-llama-index package ^[1]. You can register your custom event type directly within LlamaIndex without needing to modify any external packages.

Here's how you can create and register a custom event type:

1. Define the Custom Event:

   from llama_index.core.instrumentation.event.base import BaseEvent
   from llama_index.core.bridge.pydantic import Field
   
   class MyCustomEvent(BaseEvent):
       """My custom Event."""
       new_field_1: str = Field(...)
       new_field_2: int = Field(...)

2. Create a Custom Event Handler:

   from typing import List
   from llama_index.core.instrumentation.events import BaseEvent
   from llama_index.core.instrumentation.event_handlers import BaseEventHandler
   
   class MyCustomEventHandler(BaseEventHandler):
       """My custom event handler."""
       events: List[BaseEvent] = []
   
       @classmethod
       def class_name(cls) -> str:
           """Class name."""
           return "MyCustomEventHandler"
   
       def handle(self, event: BaseEvent) -> None:
           """Logic for handling event."""
           print(f"Handling event: {event.class_name()}")
           self.events.append(event)

3. Register the Custom Event Handler:

   from llama_index.core.instrumentation import get_dispatcher
   
   dispatcher = get_dispatcher()
   my_event_handler = MyCustomEventHandler()
   dispatcher.add_event_handler(my_event_handler)

4. Fire the Custom Event:

   dispatcher.event(MyCustomEvent(new_field_1="example", new_field_2=123))

This will allow your custom event to be registered and handled within LlamaIndex without any dependency on external packages ^[2][3].

To continue talking to Dosu, mention @dosu.

[enrico-stauss]

I'm not saying that the event is ignored @dosu, it shows up in Arize Phoenix, even without using custom events. It however uses a generic display style that does not allow looking into the body of input/output nodes. For the reranking events for example, this is possible. I could just use the ReRankStart/EndEvent in my custom NodePostProcessor but this is the wrong type, so I'd like to use a dedicated type. But in order for custom event types to be displayed as desired in Arize Phoenix, I believe a customization to the openinference package would be required. Please correct me if I'm wrong!

[dosubot[bot]]

You are correct that customizing the openinference package is not necessary to display custom event types properly in Arize Phoenix. The OpenInferenceCallbackHandler already provides mechanisms to handle various event types, including custom ones, and ensures they are displayed correctly.

The OpenInferenceCallbackHandler captures and stores AI model inferences in a format that integrates seamlessly with observability solutions like Arize and Phoenix. It supports various event types such as QUERY, LLM, RETRIEVE, and EMBEDDING, and converts them into a pandas DataFrame for easy access and manipulation. This should ensure that custom event types are properly handled and displayed without additional customization ^[1][2][3].

If your specific use-case is not supported, you can open an issue on GitHub to request support for your particular query engine or use-case ^[1].

To continue talking to Dosu, mention @dosu.