StreamingResponse

[franzwilding]

In order to have a good LLM chat UX, we need to streame the response to the client. Langserve is doing this with an dedicated endpoint, hayhooks could do the same (pseudocode):

async def pipeline_stream(pipeline_run_req: PipelineRunRequest) -> StreamingResponse:
        buffer = ...
        result = pipe.run(data=pipeline_run_req.dict())

        return StreamingResponse(buffer_generator)

app.add_api_route(
        path=f"/{pipeline_def.name}/stream",
        endpoint=pipeline_stream,
        methods=["POST"],
        name=pipeline_def.name,
        response_model=PipelineRunResponse,
    )

Additionally haystack should provide a special streaming_callback that will write the chunk content to a buffer, that will be available to hayhooks. Maybe the Pipeline could add this logic and provides an pipe.stream method that will return a generator or simething like this.

[vblagoje]

Yes @franzwilding we have this item on our roadmap, thanks for raising this issue and voicing your preferred solution.

[Phlasse]

@vblagoje any idea yet, when this feature will become available? We are using haystack in quite some projects now and want to know if it is worth putting more energy in our work around solution or if we can expect proper streaming out of a pipeline soon :) ?

[vblagoje]

Yes, I understand totally! The support is currently being worked on 😎

[aymbot]

@vblagoje Any updates regarding an ETA for the feature? Thanks in advance for the heads-up

[vblagoje]

@aymbot on our immediate roadmap for Q3, starting soon 🙏

[ilkersigirci]

With this feature implemented, hayhooks would be a strong alternative to langserve. Thanks again for working on it

[ParseDark]

really need this feature. Is there any recent update? The streaming feature is very important because most of the other third-party UIs or pkgs are called in streaming mode.

[DavidSche]

any update ?

[aryaminus]

https://dev.to/arya_minus/async-haystack-streaming-over-fastapi-endpoint-2kj0

if anyone is following this thread