Question about lambda workers doing local operations

[adrianmiu]

Hello,

I'm wondering what would be the best approach to the following scenario:

1. There are "retrieve data" jobs are dispatched to make requests to external APIs. The worker retrieves the data, stores it in a file (json/xml), dispatches "process data" jobs and marks the request as being retrieved. There are 10s of thounthands of external requests to be made and I have a table to store the status for each one (pending/queued/retrieved/processed) so that I don't push the same request to the queue multiple times.
2. The "process data" jobs parses the file, extracts the required data and performs some MySql queries (delete/update/create) to insert the processed data into the database

I have a VPS for the site and Mysql and I want to offload the retrieval and processing to AWS Lambda but I don't think the lambda can perform the database queries. What are my options here? The best I could come up so far is this using webhooks:

1. Have a webhook to notify that the data was retrieved for a particular request
2. Store the results of the processing in a S3 bucket and notify the site about the location of the file
3. Have the main site perform the required queries to import the data

Are there better options?

Thank you.

[deleugpn]

Why can't the lambda perform the database queries?

[adrianmiu]

@deleugpn

The lambda function operates in a different environment so it cannot access the website (right?), unless I open the mysql instance to the world which I don't want to do. The VPS is a regular whm/cpanel instance

[bkuhl]

If you want to use Lambda, you'll need to either open mysql to the world, or a specific ip range in which lambda operates. Otherwise move your database to AWS. This is a networking item though, not an issue with this package so this can be closed.

[deleugpn]

@adrianmiu You can opt to deploy your AWS Lambda inside a VPC with a NAT Instance or a NAT Gateway. In both cases, the public IP will be static (and predefined by the NAT). You can then add the predefined IP to your Database allow-list so that Lambda can successfully connect to your database. NAT Gateway does add $25/month cost (per AZ). NAT Instance is cheaper but you have to manage it yourself.