webhook verified source

[alberto-lanfranco-storebrand]

It would be very useful for us to use dlt to ingest data that is delivered from a webhook.
Is this something that dlt could officially support?

[burnash]

Hey @alberto-lanfranco-storebrand, thanks for the suggestion. Could you describe your use case in more detail?

[alberto-lanfranco-storebrand]

Hi @burnash ! Thank you for your prompt response, and sorry for the delay in my reply.

Here is a detailed description of our use case:

Project Overview

We are a data engineering team working on a project to integrate data from a SaaS tool into our data platform. We aim to switch to a direct integration using the tool's webhook interface to receive real-time updates.

Integration Requirements

The SaaS tool sends JSON payloads via HTTP POST requests to a specified endpoint whenever there is a change in the system.

Key requirements include:

• Real-time Data Ingestion: Webhook messages need to be processed as they arrive.
• Authentication: The tool supports token-based authentication.

Proposed Solution

We envision a solution where dlt can:

• Receive Webhook Data: Implement an endpoint to receive HTTP POST requests from the webhook.

Conclusion

In summary, integrating webhook support into dlt would significantly streamline our data ingestion process, allowing us to leverage real-time updates while maintaining cost efficiency and data freshness. We believe this feature would not only benefit our use case but also enhance dlt’s capabilities for other users with similar needs.

Thank you for considering our request, and we look forward to your response.

[burnash]

Hi @alberto-lanfranco-storebrand thanks for the details. Just to recap: you would like to have some endpoint which when triggered by SaaS tool receives the payload and calls dlt to do the ingestion. Is it his correct?
If my understanding is correct, you'd need to host this code somewhere. Is it something like AWS lambda / Cloud Functions?

[alberto-lanfranco-storebrand]

In our case the code would run either in an AKS cluster (together with dlt and dagter) or in Azure Container Apps