No module named '_lumigo' error when function handler not directly beneath serverless config

[lmmx]

I've followed the instructions on the plugin page and am getting an error that there's no such module as _lumigo.

I tried installing a few previous versions and it doesn't look like the lumigo python-tracer package (distributed as lumigo-tracer) has ever shipped this name, so I suspect the problem isn't that the import name has changed, but that the import command is meant to be doing something else.

{                                                                                                                       
    "errorMessage": "Unable to import module '_lumigo/pypi_release_check': No module named '_lumigo'",                  
    "errorType": "Runtime.ImportModuleError",                                                                           
    "requestId": "4d7f20d8-970a-433f-9317-6f504438b5a4",                                                                
    "stackTrace": []                                                                                                    
} 

I suspect the problem may be due to my project structure: I have a serverless.yml and below it I have directories containing Python packages (subservices) deployed in a single service as one stack.

My guess is that this has resulted in the tracer trying to import them from the wrong location.

I haven't read the source closely enough to be sure but if it's looking in the service directory and then assuming to find the service there, i.e. assuming that the handler must be at the top level. In my case the handler is down a level (there's a dotname that goes subpackage.module.funcdef rather than module.funcdef).

I copied the serverless config down a level and adjusted, and packaged there, and can see that when I enable the sls lumigo plugin I get the following code:

import importlib
from lumigo_tracer import lumigo_tracer
userHandler = getattr(importlib.import_module("release_check"), "check_handler")

@lumigo_tracer(token='t_XXXXXXXXXXX')
def check_handler(event, context):
  return userHandler(event, context)

I suspect this would work if the _lumigo directory created in the packaged zip were dropped down a level so that it ends up being within the bounds of where the individual subpackages can "see" (when the Python runtime is invoked on a subpackage it necessarily limits the parent folder which is "visible" to import from, and I think that's what you're relying on here to make the wrapper work).

Annoying because I can see the handler wrapper is right there, hopefully it will be a simple fix ! :-)

For clarity my "subservice" tree below my src directory containing serverless.yml is:

pypi_watcher/
├── functions.yml
├── __init__.py
├── pypi_watcher
│   ├── __init__.py
│   ├── lookout
│   │   ├── data_model.py
│   │   ├── __init__.py
│   │   ├── main.py
│   │   └── utils
│   │       ├── __init__.py
│   │       └── rss_reader.py
│   └── release_check.py
├── README.md
├── requirements.txt
└── stepfunctions.yml

and I'm using the functions.yml to define functions for the service above it:

service: ss-python-services

plugins:
  - serverless-python-requirements
  - serverless-lumigo
package:
  individually: true
  # Negated ** include optimises packaging time: further `include`s at function level
  include:
    - "!./**"
  exclude:
    - "**"

custom:
  lumigo:
    token: ${ssm:SS-LUMIGO-TRACING-TOKEN}
    skipReqCheck: true
  pythonRequirements:
    dockerizePip: non-linux
    noDeploy:
      - boto3 # boto3==1.26.90
      - botocore # botocore==1.29.90
      - pytest # pytest==7.1.2
      - pydantic # in layers only
      # https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

provider:
  name: aws
  runtime: python3.10
  versionFunctions: false
  deploymentBucket: ss-serverless-deployments
  stage: ${opt:stage, 'dev'}
  region: eu-west-1
  timeout: 300
  environment:
    STAGE: ${self:provider.stage}
  iamRoleStatements:
    - Effect: "Allow"
      Action: "secretsmanager:GetSecretValue"
      Resource: "arn:aws:secretsmanager:*"

functions:
  - ${file(../src/pypi_watcher/functions.yml)}

where the functions.yml file contains:

pypi_release_check:
  handler: pypi_watcher.release_check.check_handler
  module: pypi_watcher
  package:
    include:
      - pypi_watcher/pypi_watcher/**
  timeout: 300
  layers:
    - arn:aws:lambda:eu-west-1:017000801446:layer:AWSLambdaPowertoolsPythonV2:36

• Note that I un/commented out the package: include/exclude statements here to see if they were involved, this is used to achieve individual package separation in this project layout. I commented it out just to be able to see what was going on. It doesn't affect the result.

When I put the following serverless.yml file in the pypi_watcher directory I could see a _lumigo directory in the packaged zip (specifically the function zip pypi_release_check.zip):

service: pypi-watcher-monoservice

plugins:
  - serverless-python-requirements
  - serverless-lumigo
package:
  individually: true

custom:
  lumigo:
    token: ${ssm:SS-LUMIGO-TRACING-TOKEN}
    skipReqCheck: true
  pythonRequirements:
    dockerizePip: non-linux
    noDeploy:
      - boto3 # boto3==1.26.90
      - botocore # botocore==1.29.90
      - pytest # pytest==7.1.2
      - pydantic # in layers only
      # https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

provider:
  name: aws
  runtime: python3.10
  versionFunctions: false
  deploymentBucket: ss-serverless-deployments
  stage: ${opt:stage, 'dev'}
  region: eu-west-1
  timeout: 300
  environment:
    STAGE: ${self:provider.stage}
  iamRoleStatements:
    # pypi_watcher
    - Effect: "Allow"
      Action: "secretsmanager:GetSecretValue"
      Resource: "arn:aws:secretsmanager:*"

functions:
  pypi_release_check:
    handler: release_check.check_handler
    module: pypi_watcher
    timeout: 300
    layers:
      - arn:aws:lambda:eu-west-1:017000801446:layer:AWSLambdaPowertoolsPythonV2:36