sam build very slow #3371

GuilhermeCunha · 2021-10-17T18:35:41Z

Description:

As can be seen in the logs below, this build took more than 10 minutes, as I only have 2 functions. I noticed that the "CustomMakeBuilder:CopySource" step takes too long, and I can't find an explanation to justify it.

I'm using a custom build for my project which is in Typescript, and the link to the repository can be found here.

Steps to reproduce:

Clone

git clone https://github.com/GuilhermeCunha/aws-typescript-sam-template.git

Do configuration steps
Do build steps

Observed result:

> [email protected] sam:build /Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template
> sam build --debug

2021-10-17 14:32:58,778 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-10-17 14:32:58,779 | Using config file: samconfig.toml, config environment: default
2021-10-17 14:32:58,779 | Expand command line arguments to:
2021-10-17 14:32:58,779 | --template_file=/Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template/template.yml --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache 
2021-10-17 14:32:58,968 | 'build' command is called
2021-10-17 14:32:58,982 | Collected default values for parameters: {}
2021-10-17 14:32:59,032 | 4 stacks found in the template
2021-10-17 14:32:59,032 | Collected default values for parameters: {}
2021-10-17 14:32:59,060 | 4 resources found in the stack 
2021-10-17 14:32:59,060 | Collected default values for parameters: {}
2021-10-17 14:32:59,094 | Collected default values for parameters: {}
2021-10-17 14:32:59,123 | --base-dir is not presented, adjusting uri ./ relative to /Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template/template.yml
2021-10-17 14:32:59,123 | Found Serverless function with name='hello' and CodeUri='.'
2021-10-17 14:32:59,123 | --base-dir is not presented, adjusting uri . relative to /Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template/template.yml
2021-10-17 14:32:59,123 | Collected default values for parameters: {}
2021-10-17 14:32:59,150 | --base-dir is not presented, adjusting uri ./ relative to /Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template/template.yml
2021-10-17 14:32:59,150 | Found Serverless function with name='world' and CodeUri='.'
2021-10-17 14:32:59,150 | --base-dir is not presented, adjusting uri . relative to /Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template/template.yml
2021-10-17 14:32:59,151 | Collected default values for parameters: {}
2021-10-17 14:32:59,181 | --base-dir is not presented, adjusting uri ./ relative to /Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template/template.yml
2021-10-17 14:32:59,183 | Instantiating build definitions
2021-10-17 14:32:59,194 | Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(nodejs14.x, /Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template, Zip, x86_64, , 38890194-01fc-4005-9aed-c9ac929bd550, {'BuildMethod': 'makefile'}, {}, []), Function: Function(name='hello', functionname='hello', runtime='nodejs14.x', memory=128, timeout=100, handler='dist/src/handlers/hello.hello', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template', environment={'Variables': {'ExampleVariable': 'ExampleVariable'}}, rolearn=None, layers=[<samcli.lib.providers.provider.LayerVersion object at 0x103c5abb0>], events={'Api': {'Type': 'Api', 'Properties': {'Path': '/hello', 'Method': 'POST', 'RestApiId': 'ServerlessRestApi'}}}, metadata={'BuildMethod': 'makefile'}, inlinecode=None, codesign_config_arn=None, architectures=None, stack_path=''))
2021-10-17 14:32:59,194 | Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(nodejs14.x, /Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template, Zip, x86_64, , 3ab0ee80-bdda-4684-be23-f9cf24f8103f, {'BuildMethod': 'makefile'}, {}, []), Function: Function(name='world', functionname='world', runtime='nodejs14.x', memory=128, timeout=100, handler='dist/src/handlers/world.world', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template', environment={'Variables': {'ExampleVariable2': 'ExampleVariable2'}}, rolearn=None, layers=[<samcli.lib.providers.provider.LayerVersion object at 0x103c66610>], events={'Api': {'Type': 'Api', 'Properties': {'Path': '/world', 'Method': 'POST', 'RestApiId': 'ServerlessRestApi'}}}, metadata={'BuildMethod': 'makefile'}, inlinecode=None, codesign_config_arn=None, architectures=None, stack_path=''))
2021-10-17 14:32:59,194 | Same Layer build definition found, adding layer (Previous: LayerBuildDefinition(RuntimeDependenciesLayer, /Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template, , c680ffc6-645a-410c-b327-e173bbd450ff, makefile, ['nodejs14.x'], x86_64, {}), Current: LayerBuildDefinition(RuntimeDependenciesLayer, /Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template, , 2790a233-cb47-409b-9489-055592dfbd82, makefile, ['nodejs14.x'], x86_64, {}), Layer: <samcli.lib.providers.provider.LayerVersion object at 0x103b82a60>)
2021-10-17 14:32:59,201 | Building codeuri: /Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template runtime: nodejs14.x metadata: {'BuildMethod': 'makefile'} architecture: x86_64 functions: ['hello']
2021-10-17 14:32:59,201 | Building to following folder /Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template/.aws-sam/build/hello
2021-10-17 14:32:59,202 | Loading workflow module 'aws_lambda_builders.workflows'
2021-10-17 14:32:59,212 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2021-10-17 14:32:59,217 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2021-10-17 14:32:59,223 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2021-10-17 14:32:59,228 | Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
2021-10-17 14:32:59,234 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2021-10-17 14:32:59,241 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2021-10-17 14:32:59,248 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2021-10-17 14:32:59,254 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2021-10-17 14:32:59,260 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2021-10-17 14:32:59,260 | Found workflow 'CustomMakeBuilder' to support capabilities 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2021-10-17 14:32:59,261 | Running workflow 'CustomMakeBuilder'
2021-10-17 14:32:59,261 | Running CustomMakeBuilder:CopySource
2021-10-17 14:36:04,061 | CustomMakeBuilder:CopySource succeeded
2021-10-17 14:36:04,061 | Running CustomMakeBuilder:MakeBuild
2021-10-17 14:36:04,062 | Current Artifacts Directory : /Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template/.aws-sam/build/hello
2021-10-17 14:36:04,063 | executing Make: ['make', '--makefile', '/Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template/Makefile', 'build-hello']
2021-10-17 14:36:16,294 | CustomMakeBuilder:MakeBuild succeeded
2021-10-17 14:36:21,699 | Building codeuri: /Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template runtime: nodejs14.x metadata: {'BuildMethod': 'makefile'} architecture: x86_64 functions: ['world']
2021-10-17 14:36:21,699 | Building to following folder /Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template/.aws-sam/build/world
2021-10-17 14:36:21,702 | Loading workflow module 'aws_lambda_builders.workflows'
2021-10-17 14:36:21,702 | Found workflow 'CustomMakeBuilder' to support capabilities 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2021-10-17 14:36:21,703 | Running workflow 'CustomMakeBuilder'
2021-10-17 14:36:21,703 | Running CustomMakeBuilder:CopySource
2021-10-17 14:39:32,799 | CustomMakeBuilder:CopySource succeeded
2021-10-17 14:39:32,799 | Running CustomMakeBuilder:MakeBuild
2021-10-17 14:39:32,800 | Current Artifacts Directory : /Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template/.aws-sam/build/world
2021-10-17 14:39:32,801 | executing Make: ['make', '--makefile', '/Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template/Makefile', 'build-world']
2021-10-17 14:39:44,718 | CustomMakeBuilder:MakeBuild succeeded
2021-10-17 14:39:49,768 | Building layer 'RuntimeDependenciesLayer'
2021-10-17 14:39:49,770 | First compatible runtime has been chosen as build runtime
2021-10-17 14:39:49,770 | Loading workflow module 'aws_lambda_builders.workflows'
2021-10-17 14:39:49,770 | Found workflow 'CustomMakeBuilder' to support capabilities 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2021-10-17 14:39:49,771 | Running workflow 'CustomMakeBuilder'
2021-10-17 14:39:49,771 | Running CustomMakeBuilder:CopySource


2021-10-17 14:42:59,056 | CustomMakeBuilder:CopySource succeeded
2021-10-17 14:42:59,057 | Running CustomMakeBuilder:MakeBuild
2021-10-17 14:42:59,058 | Current Artifacts Directory : /Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template/.aws-sam/build/RuntimeDependenciesLayer
2021-10-17 14:42:59,058 | executing Make: ['make', '--makefile', '/Users/guilherme.s.cunha/Documents/respositorios/aws-typescript-sam-template/Makefile', 'build-RuntimeDependenciesLayer']
2021-10-17 14:43:07,138 | CustomMakeBuilder:MakeBuild succeeded

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided
    
2021-10-17 14:43:12,303 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'e9bec35a-8b4c-4243-bbc1-e52a4a3e87e2', 'installationId': '4f66b770-763a-4404-b516-32a6e5d501f9', 'sessionId': '1975356e-64ba-4906-a554-e886dd6ba7f0', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.12', 'samcliVersion': '1.33.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 613519, 'exitReason': 'success', 'exitCode': 0}}]}
2021-10-17 14:43:13,232 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

Expected result:

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: macOs Catalina 10.15.7
sam --version: SAM CLI, version 1.33.0
AWS region:

The text was updated successfully, but these errors were encountered:

hawflau · 2021-10-18T15:28:22Z

@GuilhermeCunha Thanks for raising the issue. We will try to reproduce the issue and investigate.

joyacv2 · 2021-11-09T02:00:10Z

Hi,

sam build or sam build --cache extremely slow.

Thank you very much!

moelasmar · 2021-11-15T13:16:04Z

I was able to build the project in 52 seconds. From the logs you shared, it seems most of the time is always taken in the copy source step.

2021-10-17 14:32:59,261 | Running CustomMakeBuilder:CopySource
2021-10-17 14:36:04,061 | CustomMakeBuilder:CopySource succeeded

2021-10-17 14:36:21,703 | Running CustomMakeBuilder:CopySource
2021-10-17 14:39:32,799 | CustomMakeBuilder:CopySource succeeded

2021-10-17 14:39:49,771 | Running CustomMakeBuilder:CopySource
2021-10-17 14:42:59,056 | CustomMakeBuilder:CopySource succeeded

Could you please check that there is no not required files in your lambda functions source directories.

nthornton2010 · 2021-11-17T20:09:51Z

I'm running into this as well. This only happens if my makefile is in the root of my project, if it's in a subdirectory, the makefile finishes in seconds. Maybe it's copying node_modules when it doesn't need to, which makes it take so much longer than it needs to?

EDIT: I believe this can be solved by implementing something along the lines of what is in here:
#3350

yskkin · 2021-11-18T02:55:45Z

@moelasmar For JavaScript project, copy is slow due to large npm dependency even if there is no not required files in source directories
#3093

f-lombardo · 2021-11-24T08:00:03Z

Hi all,

I think problems arises because sam tries to copy the whole project directory into a working dir for each lambda in the project. In my use case it seems way too much, since I have a directory structure like this:

/project
  /lambdas
    /l1
    /l2
.....
 /other_complex_and_big_stuffs
template.yaml
Makefile

Is there a way to tell sam not to consider other parts of the project?

LuizPelegrini · 2022-03-15T17:22:35Z

sam build seems to be unsustainable for large projects. Every time I build (even using sam build --cached), it erases all .aws-sam/ folder content and start all over again (taking about 5-10 min for 5 functions), even if I change one or two functions. Please, someone can advise? I'm using makefile inside two subdirectories

/project
   /lambdas
       /domain1
          lambda1.ts
          lambda2.ts
          Makefile
      /domain2
          lambda1.ts
          lambda2.ts
          Makefile
template.yaml

mndeveci · 2022-04-13T05:00:18Z

Thanks for raising this issue.

SAM CLI now supports esbuild (which is still in beta). Can you try to use that workflow to build your functions (instead of Makefile)? You can read documentation here; https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-using-build-typescript.html

Please let us know if you have any improvement point for esbuild, we would love to hear them before beta stage ends.

Thanks!

intrealm · 2022-05-10T17:12:59Z

@mndeveci we also originally planned to use esbuild, but with esbuild we couldn't ship other extensions to the build folder and that where makefile worked, we are also facing a big pause in builds, and one more thing we are on m1 chips.

mndeveci · 2022-05-12T00:31:46Z

Thanks for the feedback @intrealm, I will take this to our team and PM to find a resolution for build's IO operations.

tunesmith · 2022-09-23T22:46:40Z

For what it's worth, our build of eight functions sped up from 55 minutes to 2 minutes when we upgraded to sam 1.56.0. For us I think it was related to the bloat of devDependencies, which sam/esbuild now knows to omit.

GuilhermeCunha changed the title ~~sam build very Slow~~ sam build very slow Oct 17, 2021

hawflau added stage/bug-repro The issue/bug needs to be reproduced area/build sam build command stage/needs-investigation Requires a deeper investigation labels Oct 18, 2021

aws-sam-cli-bot added the maintainer/need-followup label Nov 11, 2021

aws-sam-cli-bot removed the maintainer/need-followup label Nov 15, 2021

aws-sam-cli-bot added the maintainer/need-followup label Nov 20, 2021

wchengru removed the blocked/close-if-inactive Blocked for >14 days with no response, will be closed if still inactive after 7 days label Nov 21, 2021

f-lombardo mentioned this issue Nov 24, 2021

Ignore files/folders for CustomMakeWorkflow aws/aws-lambda-builders#227

Closed

aws-sam-cli-bot added maintainer/need-followup and removed maintainer/need-followup labels Nov 25, 2021

zhuhaow mentioned this issue Feb 11, 2022

Custom builder should not copy source files to temp folder aws/aws-lambda-builders#328

Closed

aws-sam-cli-bot added maintainer/need-followup and removed maintainer/need-followup labels Mar 16, 2022

aws-sam-cli-bot removed the maintainer/need-followup label Apr 14, 2022

aws-sam-cli-bot added the maintainer/need-followup label Sep 26, 2022

sriram-mv added blocked/close-if-inactive Blocked for >14 days with no response, will be closed if still inactive after 7 days and removed maintainer/need-followup labels Oct 6, 2022

aws-sam-cli-bot added the maintainer/need-followup label Oct 7, 2022

sriram-mv closed this as completed Oct 11, 2022

torresxb1 mentioned this issue Jan 12, 2023

Feature request: Build in source #4571

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sam build very slow #3371

sam build very slow #3371

GuilhermeCunha commented Oct 17, 2021 •

edited

Loading

hawflau commented Oct 18, 2021

joyacv2 commented Nov 9, 2021

moelasmar commented Nov 15, 2021

nthornton2010 commented Nov 17, 2021 •

edited

Loading

yskkin commented Nov 18, 2021

f-lombardo commented Nov 24, 2021 •

edited

Loading

LuizPelegrini commented Mar 15, 2022 •

edited

Loading

mndeveci commented Apr 13, 2022

intrealm commented May 10, 2022 •

edited

Loading

mndeveci commented May 12, 2022

tunesmith commented Sep 23, 2022

sam build very slow #3371

sam build very slow #3371

Comments

GuilhermeCunha commented Oct 17, 2021 • edited Loading

Description:

Steps to reproduce:

Observed result:

Expected result:

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

hawflau commented Oct 18, 2021

joyacv2 commented Nov 9, 2021

moelasmar commented Nov 15, 2021

nthornton2010 commented Nov 17, 2021 • edited Loading

yskkin commented Nov 18, 2021

f-lombardo commented Nov 24, 2021 • edited Loading

LuizPelegrini commented Mar 15, 2022 • edited Loading

mndeveci commented Apr 13, 2022

intrealm commented May 10, 2022 • edited Loading

mndeveci commented May 12, 2022

tunesmith commented Sep 23, 2022

GuilhermeCunha commented Oct 17, 2021 •

edited

Loading

nthornton2010 commented Nov 17, 2021 •

edited

Loading

f-lombardo commented Nov 24, 2021 •

edited

Loading

LuizPelegrini commented Mar 15, 2022 •

edited

Loading

intrealm commented May 10, 2022 •

edited

Loading