No response from invoke container for Lambda inside docker-compose #2837
Comments
|
Is your Docker daemon running locally? Could you try with |
|
Hi @hoffa I've tested with --container-host-interface 0.0.0.0, at this point I can't even get a request to go through to the docker-compose container at that point. Trying http://0.0.0.0:3000/v1/openapi.json , from the host system is inaccessible
|
|
Hi @Andrewx82 could you try the command without |
|
Hi @xazhao The call is now accessible and I see it being processed in the logs , however I see no response from the containers |
|
@Andrewx82 |
|
Hi @xazhao I have changed the sam_entrypoint.sh to the following along with the docker-compose to Still appears to be no response |
|
@Andrewx82 I was testing using configurations from this comment: #2436 (comment). After adding |
|
@xazhao I can confirm that that does not work . See my bash entry script I also tried with and without the -v option Can we just get an official example instead of saying that it is supposed to work ? |
|
@Andrewx82 This is what I'm currently using.
Then open another terminal and use the following command: Please note: My development environment is macOS and I think mounting volume is different in Windows (Seems you already did properly). For your |
|
@xazhao I've tried to align my files to be similar to yours It appears that something is happening here as I'm getting information about interacting with the lambda, however it doesn't seem like its reading the values correctly. For clarity I don't want to use volumes for this as I want to bake a docker image with the lambda |
|
I'm having a very similar issue. I think that the root cause is related to the code path it is attempting to mount. In the output that @Andrewx82 posted earlier, you see this line: In order to properly evaluate paths for mounts, the docker engine needs the path on the host of the docker engine. In @xazhao 's examples, this is accomplished because of the (Note that if using the exact code shared by @xazhao , the WORKDIR is "aws" instead of "app", but the issue is the same regardless.) Here's what you see logged when In the use case of running from within VSCode devcontainers, which is one of the use cases mentioned in #2492, the option with I also tried this (working_dir commented out but passing $PWD to the In that case, here is the the output: I looked at the codebase related to that area of code:
Requiring the configuration to use The desired behavior being described in this issue, #2492, #963 and #1834 seem like they would be resolved if you used |
|
@xazhao any updates on the above? |
|
👍 |
|
I solved this problem just using |
@alexsanderp Can you please share your docker-compose file ? |
|
sam.Dockerfile docker-compose.yml sam-api-entrypoint.sh So I can access the API from host with http://localhost:3000. But I can't access from other containers :( |
|
I was able to connect to the API from other containers using extra_hosts: Connect to API from inside the other container: That is, I can now connect from within other containers and from the host :) If you want the lambda container to connect to other containers (example: dynamodb), configure them on the network specified in |
|
@alexsanderp We don't want to use volumes like your example is doing, we want a baked image that can be distributed |
|
@xazhao any movement around this, it has been more than a month |
|
I get one step further by using the official SAM CLI images. Perhaps it's worth a try @Andrewx82? version: '3.9'
services:
api:
image: public.ecr.aws/sam/build-nodejs14.x:1.26.0
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./dist:/var/task:ro
ports:
- 3000:3000
command: sam local start-api --template stack.yaml --host 0.0.0.0 --docker-network application --container-host host.docker.internal --warm-containers EAGER
networks:
default:
name: applicationMind you, I'm now stuck because the spawned container's |
|
I tried @NickDarvey's Tested on Debian Linux. I would really like to get a solution to this too. I have used |
|
Hi, I ran into the same problem of running sam in a docker container for a project I work on. I manged in the end to fix it by running sam as: notice that container-host is 172.17.0.2 because I override that in my docker daemon config, by default that would be 172.17.0.1 (on linux, on mac this is a hostname with a random ip I belive) |
|
@sbhvt @Andrewx82 Sorry for the delay. @sbhvt Thanks for the detailed response and analysis. It did help me identify the bug a lot. When we introduced nested stack feature, we started resolving CodeUri before it is resolved using the value of @Andrewx82 Could you please try the method that @sbhvt proposed using We will fix it once we confirm that is the root cause. |
|
I've tested this using the latest Sam CLI and an API Gateway project created from
FROM node:14-buster-slim
ENV NODE_ENV test
RUN apt-get update && \
apt-get install -y \
unzip \
curl && \
(cd /tmp/ && \
curl -OL https://github.com/aws/aws-sam-cli/releases/download/v1.28.0/aws-sam-cli-linux-x86_64.zip && \
unzip aws-sam-cli-linux-x86_64.zip -d sam-installation && \
./sam-installation/install)
version: '3.9'
services:
sam_app:
working_dir: $PWD
build:
context: .
dockerfile: Dockerfile
ports:
- "3000:3000"
command: sam local start-api --host 0.0.0.0 --container-host 172.17.0.1 --debug --docker-volume-basedir $PWD
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- $PWD:$PWDThis is the output I see from Creating network "sam-app_default" with the default driver
Creating sam-app_sam_app_1 ... done
Attaching to sam-app_sam_app_1
sam_app_1 |
sam_app_1 | SAM CLI now collects telemetry to better understand customer needs.
sam_app_1 |
sam_app_1 | You can OPT OUT and disable telemetry collection by setting the
sam_app_1 | environment variable SAM_CLI_TELEMETRY=0 in your shell.
sam_app_1 | Thanks for your help!
sam_app_1 |
sam_app_1 | Learn More: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-telemetry.html
sam_app_1 |
sam_app_1 | 2021-08-18 10:22:04,409 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
sam_app_1 | 2021-08-18 10:22:04,410 | Using config file: samconfig.toml, config environment: default
sam_app_1 | 2021-08-18 10:22:04,410 | Expand command line arguments to:
sam_app_1 | 2021-08-18 10:22:04,410 | --template_file=/home/graham/workspace/sam-app/template.yaml --host=0.0.0.0 --container_host=172.17.0.1 --docker_volume_basedir=/home/graham/workspace/sam-app --port=3000 --static_dir=public --layer_cache_basedir=/root/.aws-sam/layers-pkg --container_host_interface=127.0.0.1
sam_app_1 | 2021-08-18 10:22:04,493 | local start-api command is called
sam_app_1 | 2021-08-18 10:22:04,502 | No Parameters detected in the template
sam_app_1 | 2021-08-18 10:22:04,534 | 2 stacks found in the template
sam_app_1 | 2021-08-18 10:22:04,534 | No Parameters detected in the template
sam_app_1 | 2021-08-18 10:22:04,564 | 2 resources found in the stack
sam_app_1 | 2021-08-18 10:22:04,564 | No Parameters detected in the template
sam_app_1 | 2021-08-18 10:22:04,595 | Found Serverless function with name='HelloWorldFunction' and CodeUri='hello-world/'
sam_app_1 | 2021-08-18 10:22:04,595 | --base-dir is not presented, adjusting uri hello-world/ relative to /home/graham/workspace/sam-app/template.yaml
sam_app_1 | 2021-08-18 10:22:04,600 | No Parameters detected in the template
sam_app_1 | 2021-08-18 10:22:04,630 | No Parameters detected in the template
sam_app_1 | 2021-08-18 10:22:04,659 | Found '1' API Events in Serverless function with name 'HelloWorldFunction'
sam_app_1 | 2021-08-18 10:22:04,659 | Detected Inline Swagger definition
sam_app_1 | 2021-08-18 10:22:04,659 | Lambda function integration not found in Swagger document at path='/hello' method='get'
sam_app_1 | 2021-08-18 10:22:04,659 | Found '0' APIs in resource 'ServerlessRestApi'
sam_app_1 | 2021-08-18 10:22:04,660 | Removed duplicates from '0' Explicit APIs and '1' Implicit APIs to produce '1' APIs
sam_app_1 | 2021-08-18 10:22:04,660 | 1 APIs found in the template
sam_app_1 | 2021-08-18 10:22:04,667 | Mounting HelloWorldFunction at http://0.0.0.0:3000/hello [GET]
sam_app_1 | 2021-08-18 10:22:04,667 | You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
sam_app_1 | 2021-08-18 10:22:04,667 | Localhost server is starting up. Multi-threading = True
sam_app_1 | 2021-08-18 10:22:04 * Running on http://0.0.0.0:3000/ (Press CTRL+C to quit)And when I hit sam_app_1 | 2021-08-18 10:23:24,190 | Constructed String representation of Event to invoke Lambda. Event: {"body": null, "headers": {"Accept": "*/*", "Host": "localhost:3000", "User-Agent": "curl/7.64.0", "X-Forwarded-Port": "3000", "X-Forwarded-Proto": "http"}, "httpMethod": "GET", "isBase64Encoded": false, "multiValueHeaders": {"Accept": ["*/*"], "Host": ["localhost:3000"], "User-Agent": ["curl/7.64.0"], "X-Forwarded-Port": ["3000"], "X-Forwarded-Proto": ["http"]}, "multiValueQueryStringParameters": null, "path": "/hello", "pathParameters": null, "queryStringParameters": null, "requestContext": {"accountId": "123456789012", "apiId": "1234567890", "domainName": "localhost:3000", "extendedRequestId": null, "httpMethod": "GET", "identity": {"accountId": null, "apiKey": null, "caller": null, "cognitoAuthenticationProvider": null, "cognitoAuthenticationType": null, "cognitoIdentityPoolId": null, "sourceIp": "192.168.112.1", "user": null, "userAgent": "Custom User Agent String", "userArn": null}, "path": "/hello", "protocol": "HTTP/1.1", "requestId": "313b059f-6e74-4d2d-9033-d47f935ca834", "requestTime": "18/Aug/2021:10:22:04 +0000", "requestTimeEpoch": 1629282124, "resourceId": "123456", "resourcePath": "/hello", "stage": "Prod"}, "resource": "/hello", "stageVariables": null, "version": "1.0"}
sam_app_1 | 2021-08-18 10:23:24,191 | Found one Lambda function with name 'HelloWorldFunction'
sam_app_1 | 2021-08-18 10:23:24,191 | Invoking app.lambdaHandler (nodejs14.x)
sam_app_1 | 2021-08-18 10:23:24,191 | No environment variables found for function 'HelloWorldFunction'
sam_app_1 | 2021-08-18 10:23:24,191 | Environment variables overrides data is standard format
sam_app_1 | 2021-08-18 10:23:24,191 | Loading AWS credentials from session with profile 'None'
sam_app_1 | 2021-08-18 10:23:26,219 | Resolving code path. Cwd=/home/graham/workspace/sam-app, CodeUri=/home/graham/workspace/sam-app/hello-world
sam_app_1 | 2021-08-18 10:23:26,219 | Resolved absolute path to code is /home/graham/workspace/sam-app/hello-world
sam_app_1 | 2021-08-18 10:23:26,220 | Code /home/graham/workspace/sam-app/hello-world is not a zip/jar file
sam_app_1 | 2021-08-18 10:23:26,275 | Skip pulling image and use local one: public.ecr.aws/sam/emulation-nodejs14.x:rapid-1.28.0.
sam_app_1 |
sam_app_1 | 2021-08-18 10:23:26,275 | Mounting /home/graham/workspace/sam-app/hello-world as /var/task:ro,delegated inside runtime container
sam_app_1 | 2021-08-18 10:23:27,132 | Starting a timer for 1 seconds for function 'HelloWorldFunction'
sam_app_1 | 2021-08-18 10:23:28,866 | Cleaning all decompressed code dirs
sam_app_1 | 2021-08-18 10:23:28,866 | No response from invoke container for HelloWorldFunction
sam_app_1 | 2021-08-18 10:23:28,866 | Invalid lambda response received: Lambda response must be valid json
sam_app_1 | 2021-08-18 10:23:28 192.168.112.1 - - [18/Aug/2021 10:23:28] "GET /hello HTTP/1.1" 502 -From @sbhvt's version I had to add in the When I add the @xazhao, is this enough information for you to investigate further? If not then please let me know - I'm really keen to get this resolved as it's blocking my adoption of Sam over Serverless. |
|
Came here because of the same issue when I try to invoke locally my lambda function with sam cli. |
aws-sam-cli-bot
added
maintainer/need-followup
and removed
maintainer/need-followup
labels
|
Also interested about this. I have success running the commands locally, but when I run them in CircleCI, I am faced with the dreaded I am using the following two orbs: |
|
@austinben, what OS are you running on locally? In the example I posted above I got it working on macOS but then had to add something else to make it work on Linux. I don't remember exactly what but take a look and see if you've anything missing. |
|
Running it on macOS locally, but have confirmed it also runs locally on linux. I am not using docker compose. Simply invoking the sam cli commands |
aws-sam-cli-bot
added
maintainer/need-followup
and removed
maintainer/need-followup
labels
|
Hello my friends in aws suffering :) Just my two cents on this issue. Yes most of these configs work, but for some like me they simply don't. I was trying everything for a few days and then my boss just started the first configuration on (clean) mac and it worked. Before I tried to put aws sam cli in docker I installed everything locally (then it was 1.38 version) to test everything (mac os, dockers for lambdas, and experimenting with file sharing to make it work). After the docker-compose configuration worked for my boss and my very troubled mind I deleted everything locally, reseted docker and then it simply started to work in docker… So maybe an option to think about. Also meanwhile aws-sam-cli was upgraded to 1.40, i refuse to believe that this was cause of my suffering tho x) So this simple config should work, try it maybe on some other machine, linux or something, and take into consideration that problem isn't up to config. I hope this will help somebody.. the docker file in docker-compose |
aws-sam-cli-bot
added
maintainer/need-followup
and removed
maintainer/need-followup
labels
|
@grahamlyons Sorry for my slow response. Was on leave for a while at the end of last year/beginning of this year and then this just fell off my radar. There was some back and forth whether Discussions was useful. For now, a lightweight way (I can think of) is just move discussion about "docker in docker" into a [Discussion] titled issue. That will at least allow that to focus. We do have a slack channel, which could be another option. |
|
@jfuss, no worries - thanks for coming back on this. I would welcome any discussion on running Sam with Docker (although I'd avoid the phrase "docker in docker" as that implies running the docker daemon inside a container which is whole other thing) as it's something I've been doing for a while so would be good to air my use cases. |
aws-sam-cli-bot
added
maintainer/need-followup
and removed
maintainer/need-followup
labels
|
btw, |
aws-sam-cli-bot
added
maintainer/need-followup
and removed
maintainer/need-followup
labels
|
Closing as the fix has been released in SAM v1.57.0 |
|
Hey, @hawflau ; I'm not convinced this issue is the one fixed by the PR merged above. That PR seems to resolve Issue 3520 (not going to hotlink it to avoid associating it with this ticket). This ticket is to do with accessing APIs over a Docker user network. I'm not sure why that PR mentions this issue by ID. |
|
Hi @awilkins, I'll check with the PR contributor. If that's the case, we will reopen the issue. Thanks for flagging it! |
|
Hi, thanks for bringing up your observations! I can confirm that the linked PR fixed one of the original issues (not being able to target the correct directory on the host machine). It looks like there is another issue related to the usage of the We'll investigate further and look into next steps to resolve this. |
|
Hi, the thread was helpful, I managed to get the app working, however, I started facing issues when I was trying to use layers. My docker-compose file is as below. The error I am getting is that the path to the utilities-layer is not being found. The path is on the host machine and it exists. I am running a Mac OS M1 using a docker desktop. See the error I am facing below. I tried to add the flag |
|
I was able to get sam working inside docker. It turns out when running the application using docker-compose the utilities base path has to be the working directory inside docker. My working directory was I tried to find a way to make this dynamic. My attempt was to have the base path for the utilities as a parameter like My work around was to have the default of the parameter set to |
and others
Description:
Related to #2492
When running a docker-compose with an image that is using sam to run a local api, the api can never get a response from the lambda container. Everything works correctly when run locally not in a container.
I have read through all the answers in 2492 and tried the various permutations suggested, nothing has worked. There appeared to be no subsequent documentation follow up for getting this setup to work with docker.
Steps to reproduce:
DockerFile
sam_entrypoint.sh
Template.yml
Docker-compose
Observed result:
Expected result:
That the lambda can be executed correctly and a response received.
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
sam --version: 1.23.0The text was updated successfully, but these errors were encountered: