sam build fails with Python layer with makefile BuildMethod

[AbeGellis]

Description:

I have a Python 3.8 application in which I am attempting to include the psycopg2-binary library as a layer. I have already built the library against AWS Linux using EC2 and have been able to create and manage this layer within the Lambda console. However, integrating it into the sam workflow is proving troublesome.

When I specify a "python3.8" BuildMethod in my template.yaml, I encounter a similar issue to #1360: there are a number of .so files that I want to include which are automatically omitted from the deployed layer.

Since all I really need to do is copy the contents of my layer directory, I set BuildMethod to "makefile" and added a simple Makefile file to the ContentUri directory. This, however, never seems to be executed, and sam produces a somewhat incoherent error instead, as if I had requested a nonexistent build runtime.

Steps to reproduce:

On AWS Linux, install psycopg2-binary via pip to a target directory.
Copy that installation to a subdirectory of a sam project. In my case this was "layers/psycopg2-binary"
Add the following to template.yaml:

psycopg2binary:
Type: AWS::Serverless::LayerVersion
Properties:
LayerName: psycopg2
ContentUri: layers/psycopg2-binary
CompatibleRuntimes:
- python3.8
Metadata:
BuildMethod: makefile

Add a file named "Makefile" to the same subdirectory, with the contents:

build-psycopg2binary:
echo "executing makefile"
mkdir -p "$(ARTIFACTS_DIR)/python"
cp * "$(ARTIFACTS_DIR)/python"

Execute the command sam build psycopg2binary --debug from the root directory of the sam project.

Observed result:

2021-04-30 16:42:30,825 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-04-30 16:42:30,825 | Using config file: samconfig.toml, config environment: default
2021-04-30 16:42:30,826 | Expand command line arguments to:
2021-04-30 16:42:30,826 | --template_file=D:\Work\Projects\CTB-Serverless-Backend\ctb-serverless-backend\template.yaml --resource_logical_id=psycopg2binary
--build_dir=.aws-sam\build --cache_dir=.aws-sam\cache
2021-04-30 16:42:31,081 | 'build' command is called
2021-04-30 16:42:31,089 | No Parameters detected in the template
2021-04-30 16:42:31,146 | 4 stacks found in the template
2021-04-30 16:42:31,147 | No Parameters detected in the template
2021-04-30 16:42:31,178 | 4 resources found in the stack
2021-04-30 16:42:31,179 | No Parameters detected in the template
2021-04-30 16:42:31,206 | No Parameters detected in the template
2021-04-30 16:42:31,246 | --base-dir is not presented, adjusting uri layers/psycopg2-binary/ relative to D:\Work\Projects\CTB-Serverless-Backend\ctb-serverless-backend\template.yaml
2021-04-30 16:42:31,248 | Found Serverless function with name='getFLAparentactivities' and CodeUri='get_FLA_parent_activities/'
2021-04-30 16:42:31,248 | --base-dir is not presented, adjusting uri get_FLA_parent_activities/ relative to D:\Work\Projects\CTB-Serverless-Backend\ctb-serverless-backend\template.yaml
2021-04-30 16:42:31,250 | Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
2021-04-30 16:42:31,251 | --base-dir is not presented, adjusting uri hello_world/ relative to D:\Work\Projects\CTB-Serverless-Backend\ctb-serverless-backend\template.yaml
2021-04-30 16:42:31,252 | No Parameters detected in the template
2021-04-30 16:42:31,285 | --base-dir is not presented, adjusting uri layers/psycopg2-binary/ relative to D:\Work\Projects\CTB-Serverless-Backend\ctb-serverless-backend\template.yaml
2021-04-30 16:42:31,289 | Instantiating build definitions
--- Logging error ---
Traceback (most recent call last):
File "logging_init_.py", line 1081, in emit
File "logging_init_.py", line 925, in format
File "logging_init_.py", line 664, in format
File "logging_init_.py", line 369, in getMessage
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\build\build_graph.py", line 353, in str
f"LayerBuildDefinition({self.name}, {self.codeuri}, {self.source_md5}, {self.uuid}, "
AttributeError: 'NoneType' object has no attribute 'name'
Call stack:
File "runpy.py", line 194, in _run_module_as_main
File "runpy.py", line 87, in run_code
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli_main.py", line 12, in
cli(prog_name="sam")
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 829, in call
return self.main(*args, **kwargs)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 1259, in invoke
return process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\decorators.py", line 73, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\telemetry\metric.py", line 122, in wrapped
return_value = func(*args, **kwargs)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\utils\version_checker.py", line 42, in wrapped
actual_result = func(*args, **kwargs)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\cli\main.py", line 90, in wrapper
return func(*args, **kwargs)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\build\command.py", line 210, in cli
do_cli(
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\build\command.py", line 315, in do_cli
artifacts = builder.build()
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\build\app_builder.py", line 139, in build
build_graph = self.get_build_graph(self.container_env_var, self.container_env_var_file)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\build\app_builder.py", line 204, in get_build_graph
build_graph.put_layer_build_definition(layer_build_details, layer)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\build\build_graph.py", line 228, in put_layer_build_definition
LOG.debug(
File "logging_init.py", line 1430, in debug
File "logging_init.py", line 1585, in log
File "logging_init.py", line 1595, in handle
File "logging_init.py", line 1657, in callHandlers
File "logging_init.py", line 950, in handle
File "logging_init.py", line 1089, in emit
Message: 'Same Layer build definition found, adding layer (Previous: %s, Current: %s, Layer: %s)'
Arguments: (<samcli.lib.build.build_graph.LayerBuildDefinition object at 0x000001AA129C2760>, <samcli.lib.build.build_graph.LayerBuildDefinition object at 0x000001AA129CB130>, <samcli.lib.providers.provider.LayerVersion object at 0x000001AA12A763D0>)
2021-04-30 16:42:31,319 | Building layer 'psycopg2binary'
2021-04-30 16:42:31,322 | Loading workflow module 'aws_lambda_builders.workflows'
2021-04-30 16:42:31,352 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2021-04-30 16:42:31,362 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2021-04-30 16:42:31,373 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2021-04-30 16:42:31,383 | Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
2021-04-30 16:42:31,397 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2021-04-30 16:42:31,413 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2021-04-30 16:42:31,427 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2021-04-30 16:42:31,441 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2021-04-30 16:42:31,454 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2021-04-30 16:42:31,456 | Found workflow 'CustomMakeBuilder' to support capabilities 'Capability(language='provided', dependency_manager=None, application_framework=None)'

Build Failed
2021-04-30 16:42:31,507 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '1a758fdd-27a7-43b7-bfcf-025b2b4c6283', 'installationId': 'f0be4545-6dd3-4ec0-998b-dd72367a0a64', 'sessionId': '051a96cb-8c50-4690-8685-870febec6b18', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.8', 'samcliVersion': '1.23.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 681, 'exitReason': 'WorkflowFailedError', 'exitCode': 1}}]}
2021-04-30 16:42:34,215 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: CustomMakeBuilder:Resolver - Path resolution for runtime: provided of binary: make was not successful

From the debug logs it seems to recognize that I'm trying to use a custom makefile, but the last line makes it seem as though it's still attempting to build psycopg2binary with an unspecified workflow.

Expected result:

"executing makefile" to be output in the command line, and the layer copied into the destination directory.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Windows 10 Home version 2004
2. sam --version: 1.23.0
3. AWS region: us-east-1

[mndeveci]

Hi @AbeGellis,

I've tried your example, and I was able to build my layer with following;

template.yaml

 Resources:
   psycopg2binary:
     Type: AWS::Serverless::LayerVersion
     Properties:
       LayerName: psycopg2
       ContentUri: layers/psycopg2-binary
       CompatibleRuntimes:
         - python3.8
     Metadata:
       BuildMethod: makefile

layers/psycopg2-binary/Makefile

  build-psycopg2binary:
          mkdir -p "$(ARTIFACTS_DIR)/python"
          cp * "$(ARTIFACTS_DIR)/python"

One note, Makefile indentation should be tabs, not spaces. I tried with 2 spaces first and it failed.

Please let us know if you still experience the issue.

[AbeGellis]

That seems to have done the trick! I'm on sam --version 1.33.0 now, for what it's worth, and I've been working around this with a prebuilt layer referenced throughout the project.

If anyone else stumbles into this, I put the .py files and the formerly problematic _psycopg.cpython-38-x86_64-linux-gnu.so at the directory specified by ContentUri, then changed the Makefile provided by @mndeveci to

build-psycopg2binary:
		mkdir -p "$(ARTIFACTS_DIR)/python/psycopg2"
		cp * "$(ARTIFACTS_DIR)/python/psycopg2"

since $(ARTIFACTS_DIR)/python/ is the root from which Lambda searches for Python layers.

Thanks for your help!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[rushiikesh]

i have install make but also give error,
Error: CustomMakeBuilder:MakeBuild - Make Failed: Invalid number of parameters
make: * [E:\New folder\backend-main\src\layers\common-layer\Makefile:2: build-CommonLayer] Error 4