Lambda ARM (Apple M1): AWS SAM Golang lambda debugging does not work

[ashprao]

Description:

After creating a plain vanilla AWS SAM application with sam init for golang, and then using the AWS Toolkit provided code lens in VS Code, to create an AWS SAM template file based debug configuration, I get the following error when trying to debug the lambda code.

Running the code without debugging works.

Note:
I had raised this issue in aws/aws-toolkit-vscode as aws/aws-toolkit-vscode#2020. There was a suggestion in that thread that I should raise an issue here.

Steps to reproduce:

1. Create a new AWS SAM application using Go with sam init.
2. Select the Hello world template.
3. In the main.go file, create a debug configuration from the code lens provided by the AWS Toolkit and associated with the handler.
4. Set a breakpoint and start debugging.

Observed result:

2021-09-11 10:55:02 [INFO]: The Delve repo was not found in your GOPATH. Downloading in a temporary directory...
2021-09-11 10:55:03 [INFO]: Preparing to debug locally: Lambda "hello-world"
2021-09-11 10:55:03 [INFO]: Building SAM application...
2021-09-11 10:55:03 [INFO]: Running command: (not started) [/opt/homebrew/bin/sam build --build-dir /tmp/aws-toolkit-vscode/vsctkeYZ6BB/output --template /Users/user1/dev/godev/aws/lambdas/hello-world/hello-world/app___vsctk___template.yaml --base-dir /Users/user1/dev/godev/aws/lambdas/hello-world/hello-world]
2021-09-11 10:55:03 [INFO]: Building codeuri: /Users/user1/dev/godev/aws/lambdas/hello-world/hello-world runtime: go1.x metadata: {} functions: ['helloworld']

2021-09-11 10:55:03 [INFO]: Running GoModulesBuilder:Build

2021-09-11 10:55:03 [INFO]: 
Build Succeeded

2021-09-11 10:55:03 [INFO]: 
Built Artifacts  : ../../../../../../../private/tmp/aws-toolkit-vscode/vsctkeYZ6BB/output
Built Template   : ../../../../../../../private/tmp/aws-toolkit-vscode/vsctkeYZ6BB/output/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke -t ../../../../../../../private/tmp/aws-toolkit-vscode/vsctkeYZ6BB/output/template.yaml
[*] Deploy: sam deploy --guided --template-file ../../../../../../../private/tmp/aws-toolkit-vscode/vsctkeYZ6BB/output/template.yaml
    

2021-09-11 10:55:04 [INFO]: Build complete.
2021-09-11 10:55:04 [INFO]: Starting SAM application locally
2021-09-11 10:55:05 [INFO]: AWS.running.command
Invoking hello-world (go1.x)
Requested to skip pulling images ...

Mounting /tmp/aws-toolkit-vscode/vsctkeYZ6BB/output/helloworld as /var/task:ro,delegated inside runtime container
START RequestId: 7c546e9e-8558-4fde-8dd6-fc6946034850 Version: $LATEST
API server listening at: [::]:5858
2021-09-11T00:55:07Z info layer=debugger launching process with args: [/var/task/hello-world]
2021-09-11 10:55:07 [INFO]: Waiting for SAM application to start...
could not launch process: fork/exec /var/task/hello-world: function not implemented
2021/09/11 00:55:07 exit status 1
time="2021-09-11T00:55:07.765" level=error msg="Init failed" InvokeID= error="Runtime exited with error: exit status 1"
time="2021-09-11T00:55:07.773" level=error msg="INIT DONE failed: Runtime.ExitError"
2021-09-11 10:55:08 [INFO]: Attaching debugger to SAM application...

Command stopped: "sam local invoke"

From the output listed above, it seems to me that when using the debugger, it thinks that the function is not implemented.
2021-09-11 10:55:07 [INFO]: Waiting for SAM application to start...
could not launch process: fork/exec /var/task/hello-world: function not implemented

Other considerations

• Go debugging works for vanilla go apps (non lambdas) on M1 based Apple Macs.
• Go lambdas run successfully with sam local invoke on M1 based Apple Macs.
• Go lambdas can be successfully debugged on x86 based Apple Macs.

Expected result:

• Lambda debugging working with breakpoints being hit.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: macOS Big Sur 11.6 (M1) / Darwin arm64 20.6.0
2. sam --version: 1.31.0
3. AWS region: any (currently working in ap-south-1)
4. VS Code version: 1.16
5. AWS Toolkit extension version: 1.29.0
6. Go version: 1.17.1
7. dlv and dlv-dap version: 1.7.1 Build: $Id: 3bde2354aafb5a4043fd59838842c4cd4a8b6f0b $
8. Docker version: 20.10.8, build 3967b7d

[ashprao]

Hello, just checking in to see if there is any update on this. My self interest here is to be able to work with Go based Lambdas on my M1 Mac and not rely on an intel based system, which is what I am doing now for just those Lambdas we use Go for. At the least it would be good to get some understanding regarding what is the cause for this issue from the aws-sam-cli team point of view?

[hawflau]

Hey @ashprao, sorry for the late response. I confirmed that Go lambdas can be successfully debugged on x86 based Apple Macs. But I personally don't have a M1 Mac, I'll ask a team member to help with this.

[ashprao]

Hello @hawflau, Thanks for getting to this. Would love to be able fully switch to my M1 based Mac for all of my development. :-)

To add more context. I did try debugging after updating, AWS CLI, AWS SAM, VS Code, VS Code AWS Toolkit to the latest versions and I am still facing the issue.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: macOS Big Sur 11.6 (M1) / Darwin arm64 20.6.0
aws --version: aws-cli/2.2.45 Python/3.8.8 Darwin/20.6.0 exe/x86_64 prompt/off
sam --version: SAM CLI, version 1.33.0
AWS region: any (currently working in ap-south-1)
VS Code version: 1.61.0 (Universal)
AWS Toolkit extension version: 1.31.0
Go version: 1.17.1
dlv and dlv-dap version: 1.7.1 Build: $Id: 3bde2354aafb5a4043fd59838842c4cd4a8b6f0b $
Docker version: 20.10.8, build 3967b7d

[ashprao]

Hello @hawflau, just checking in to see if your colleagues have had any success in reproducing this, and if so, whether there is a resolution or workaround that can be made available in the near term.

[ashprao]

Hi @hawflau, apologies for the second ping on this. Any luck with having this reproduced? If this is an issue with my setup only, then I would be happy to get some pointers to figure out how to resolve it? But I am hoping that this is a bug that would have an ETA for a fix. Thanks in advance.

[casimcdaniels]

Any update to this? I seem to be able to run this without debugging just fine as @ashprao

[jackmcguire1]

bump @jfuss

[rush51]

Any update on this @hawflau? I am also facing the issue with my M1 Pro.

[jackmcguire1]

@aahung can you help with this

[jeremyforan]

I am having the same issue.

Model Name:	MacBook Pro
Model Identifier: MacBookPro17,1
Chip: Apple M1
Total Number of Cores:	8 (4 performance and 4 efficiency)
Memory:	16 GB
System Firmware Version: 7429.61.2
OS Loader Version:	7429.61.2

[flythebluesky]

Same issue here: debug run fails on an M1 MacBook.

[jeremyforan]

Just checked again with version 1.38.0. Problem persists.

Invoking handler (go1.x)
Image was not found.
Removing rapid images for repo public.ecr.aws/sam/emulation-go1.x
Building image................................
Skip pulling image and use local one: public.ecr.aws/sam/emulation-go1.x:rapid-1.38.0-x86_64.

Mounting /Users/######/GolandProjects/#######/Development/#########/.aws-sam/build/Function as /var/task:ro,delegated inside runtime container
START RequestId: 0c52d50b-####-####-acdc-d864ae3b4862 Version: $LATEST
API server listening at: [::]:65486
2022-02-16T20:11:03Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
2022-02-16T20:11:03Z info layer=debugger launching process with args: [/var/task/handler]
could not launch process: fork/exec /var/task/handler: function not implemented
2022/02/16 20:11:03 exit status 1
16 Feb 2022 20:11:03,501 [ERROR] (rapid) Init failed error=Runtime exited with error: exit status 1 InvokeID=
API server listening at: [::]:65486
2022-02-16T20:11:04Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
2022-02-16T20:11:04Z info layer=debugger launching process with args: [/var/task/handler]
could not launch process: fork/exec /var/task/handler: function not implemented
2022/02/16 20:11:04 exit status 1
END RequestId: 5c7b6ef8-####-####-aa70-20440b6d1ff8
REPORT RequestId: 5c7b6ef8-####-####-aa70-20440b6d1ff8	Init Duration: 1.59 ms	Duration: 1732.04 ms	Billed Duration: 1733 ms	Memory Size: 128 MB	Max Memory Used: 128 MB	

[rush51]

@jackmcguire1 @aahung
Is there anything we can do to assist with this fix?

[dartthebart]

any updates?

[V4G4X]

Any updates here?

[V4G4X]

i think it's related to this, the problem being docker emulation for apple soc.

@georgedimac Unfortunately, that's related to this.
Only after SAM CLI supports M1, can others like VSCode-Toolkit use that feature.

[ravitejaavv]

Any update on this issue?

[rei-gun]

Having a similar issue on M2

Dockerfile:

FROM --platform=linux/arm64 public.ecr.aws/lambda/provided:al2 as build-image
# FROM public.ecr.aws/bitnami/golang:1.19 as build-image
RUN yum install -y golang
RUN go env -w GOPROXY=direct
WORKDIR /go/src
COPY . .
RUN go build -o /main

FROM public.ecr.aws/lambda/provided:al2
WORKDIR /var/task
COPY --from=build-image /main /main
#TODO: shouldn't add my local AWS creds here
# COPY .aws /root/.aws
# Command can be overwritten by providing a different command in the template directly.
ENTRYPOINT ["/main"]

Logs when running debugger via vscode:

2022-12-28 20:40:01 [INFO]: The Delve repo was not found in your GOPATH. Downloading in a temporary directory...
2022-12-28 20:40:01 [INFO]: Preparing to debug locally: Lambda "ParseCSVFunction"
2022-12-28 20:40:01 [INFO]: Building SAM application...
2022-12-28 20:40:01 [ERROR]: SamLaunchRequestError: Failed to build SAM application [BuildFailure]
2022-12-28 20:40:01 [INFO]: SAM CLI location: [object Object]
2022-12-28 20:40:01 [INFO]: Command: (not started) [/opt/homebrew/bin/sam build --build-dir /tmp/aws-toolkit-vscode/vsctkHg6ywY/output --template /Users/reigun/work/repos/parsley/template.yaml --use-container]
2022-12-28 20:40:01 [INFO]: SAM CLI not configured, using SAM found at: '/opt/homebrew/bin/sam'
2022-12-28 20:43:38 [INFO]: telemetry: sent batch (size=3)

[moelasmar]

Thank you for raising this issue. It seems that the issue is related to the fact that Go Lambda functions can only run on x86_64 architecture (https://docs.aws.amazon.com/lambda/latest/dg/lambda-golang.html), and that Docker Desktop does not support running containers using this architecture (see these issues docker/for-mac#5191 (comment), aws/aws-toolkit-vscode#2020).

As a workaround, you can use Colima instead of Docker Desktop allows you to start the container as x86_64 architecture. I tried this workaround and worked fine. These are the steps:

• You need first to uninstall the docker desktop
  /Applications/Docker.app/Contents/MacOS/uninstall
• Install Colima, and docker client

brew install colima
brew install docker 

• start colima using the x86_64 architecture

colima start --arch x86_64

• build a dlv binary on an Intel mac and copy it to your M1 Mac
  GOARCH=amd64 GOOS=linux go build -o x86_64/dlv github.com/go-delve/delve/cmd/dlv

• in vscode, update the launch.json file, and add the following configuration:

{
   "configurations": [
       {
           "name": "Connect to server",
           "type": "go",
           "request": "attach",
           "mode": "remote",
           "remotePath": "",
           "port": 9999,
           "host": "127.0.0.1"
       }
   ]
 }

• build your project using sam
  sam build
• run sam local invoke in debug mode
  sam local invoke -d 9999 --debugger-path x86_64/ --debug-args "-delveAPI=2"
• wait till you see in the logs that the server is listening to :9999, start the debugging

I hope this information helps, and please let me know if there is anything else I can assist with.

[sbilello]

It does not work!

[VaismanLior]

Hey @moelasmar, I've tried this workaround on my M1 and I keep getting an error when running sam local invoke -d 9999 --debugger-path x86_64/ --debug-args "-delveAPI=2".
The error message: [ERROR] (rapid) Init failed error=exec: "-delveAPI=2": executable file not found in $PATH.
Any idea why this would happen?
Everything went smoothly until this point.

[khlbrg]

I'm using AWS CDK and local invoke with SAM. I've the same issue, but what I've figured out so far is that is trying to mount a local directory which does not exist.
Mounting /private/var/folders/3g/1f23g14534q_71ljshdh3k7m0000gn/T/tmpr4_g071r as /var/task:ro,delegated, inside runtime container

Where /private/var/folders/3g/1f23g14534q_71ljshdh3k7m0000gn/T/tmpr4_g071r does not exist on my local filesystem.

[ashprao]

Thanks @moelasmar! This is great info. I will try it out when I get a chance. Much appreciate the update.

In the meantime I am wondering if you or anybody else in the forum has insights into if and when Go lambdas will be supported on arm64 architectures.

Also, I did briefly try to use the provided.al2 runtime as described here - Migrating AWS Lambda functions to Amazon Linux 2. I was not successful in getting this working, but I did not really persevere, I must admit, and this was a while back.

We have continued development with local debugging rather than running the Go code as lambdas in debugging mode as a stop gap, which is largely okay unless you are trying to debug issues relating to the code running in the Lambda context.

But it would be great to get Go Lambdas debuggable in Macs with M1 architectures out of the box as we do in Intel systems. My intel based Mac is nearing 10 years and I am not planning on getting a new Intel based Mac moving forward. :-)

[dkosasih]

Hi,
I also observe the same problem with dotnet. Don't know if this information helps narrow it down if it's a cli problem or go.
I create a simple solution using AWS Toolkit solution, both dotnet and Go have the same problem.
I am on M1 and I tried to ask my colleague who run the MacOS but with intel processor, it works for him.

[moelasmar]

Hi @dkosasih,
Both Lambda, and AWS SAM CLI supports dotnet 6 for ARM architecture. Could you please provide more details on the specific issue you are encountering?

[dkosasih]

Thanks for the quick reply @moelasmar.
Rather than replicating the behavior, this thread definitely sums it up better than I could, and it also point to this discussion from Aws-Toolkit-VsCode repo.

[moelasmar]

Thank you @dkosasih .. I will check it later

[benanderton]

Thank you for raising this issue. It seems that the issue is related to the fact that Go Lambda functions can only run on x86_64 architecture (https://docs.aws.amazon.com/lambda/latest/dg/lambda-golang.html), and that Docker Desktop does not support running containers using this architecture (see these issues docker/for-mac#5191 (comment), aws/aws-toolkit-vscode#2020).

As a workaround, you can use Colima instead of Docker Desktop allows you to start the container as x86_64 architecture. I tried this workaround and worked fine. These are the steps:

* You need first to uninstall the docker desktop
  `/Applications/Docker.app/Contents/MacOS/uninstall`

* Install Colima, and docker client

brew install colima
brew install docker 

* start colima using the x86_64 architecture

colima start --arch x86_64

* build a dlv binary on an Intel mac and copy it to your M1 Mac
  `GOARCH=amd64 GOOS=linux go build -o x86_64/dlv github.com/go-delve/delve/cmd/dlv`

* in vscode, update the launch.json file, and add the following configuration:

{
   "configurations": [
       {
           "name": "Connect to server",
           "type": "go",
           "request": "attach",
           "mode": "remote",
           "remotePath": "",
           "port": 9999,
           "host": "127.0.0.1"
       }
   ]
 }

* build your project using sam
  ` sam build`

* run sam local invoke in debug mode
  `sam local invoke -d 9999 --debugger-path x86_64/ --debug-args "-delveAPI=2"`

* wait till you see in the logs that the server is listening to :9999, start the debugging

I hope this information helps, and please let me know if there is anything else I can assist with.

Hi @moelasmar, thanks for the reply. I've tried following your instructions but receive the following still:

START RequestId: d632e759-34fa-41b8-b21c-57fcb6814c13 Version: $LATEST
API server listening at: [::]:9999
2023-02-13T17:21:56Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
2023-02-13T17:21:56Z info layer=debugger launching process with args: [/var/task/main]
could not launch process: fork/exec /var/task/main: function not implemented
2023/02/13 17:21:56 exit status 1

A cursory glance at Google indicates that this is a problem with delve running on arm architecture. I followed all previous steps including compiling for amd64, I also tried compiling natively on a Ryzen powered system I have next to me and had no luck with this either.

At this point we're largely engineered our way around being dependant on local sam debugging by having extensive unit tests but it would be nice to be able to debug from the context of a Lambda.

[moelasmar]

Hello @benanderton, I appreciate your feedback. I encountered a similar issue when trying to utilize Delve. When I used the M1 Mac compiled version on Colima or the Intel Mac compiled version on Docker Desktop, I received the same error. However, I did not have any problems when I used the Intel Mac compiled version on Colima. Unfortunately, I do not know why it is not working for you.

[sbilello]

Isn't there any workaround for Apple M1 chip? I tried to use GoLand and I encountered the same problem even by changing the debugging configuration.
The GoLand tutorial probably does not run on Mac M1

[moelasmar]

We are moving this issue to a discussion thread because it is related to the development setup and aligning it with the architecture of the Lambda function, rather than being directly related to SAM CLI.

[sbilello]

I created an hello world lambda project with GoLand and SAM and I was able to replicate the issue in a Mac M1

[rahul-aut-ind]

Issue still exists. Do we have an update here ?

[jfuss]

What is the issue exactly? Did you follow #4706 (comment)

[jackmcguire1]

@jfuss isit possible to use AWS finch as a work around instead, planning on getting an M2 soon, so this will help!

[aravind-craft]

Hi,

I got it working finally on M1 mac. But I'm currently unable to pass a JSON file as an input to my lambda. I am still trying to figure that out.

Here are the steps that worked for me.

I am extending the instructions provided by @moelasmar above.

1. On your M1 Mac, You need first to uninstall the docker desktop

/Applications/Docker.app/Contents/MacOS/uninstall

2. On your M1 Mac, Install Colima, and docker client

brew install colima
brew install docker 

3. On your M1 Mac, start colima using the x86_64 architecture
   colima start --arch x86_64

4. On your M1 Mac, Make a symlink to docker.sock so that VSCode can find it. Otherwise, SAM invoke will fail with an error that it cant find docker.

sudo ln -sf ~/.colima/docker.sock /var/run/docker.sock

5. On your M1 Mac, Update docker config file, otherwise sam will fail saying it couldnt find some docker credential package.
   vi ~/.docker/config.json

Change credSStore to credStore as described in https://stackoverflow.com/a/68202428

The file should look like this:

{
        "auths": {
                "https://index.docker.io/v1/": {}
        },
        "credStore": "desktop",
        "experimental": "disabled",
        "currentContext": "colima"
}

6. I tried building the dlv binary on my M1 mac, but for intel architecture, and that also worked. So I guess I don't need a 2nd laptop after all.

cd 
mkdir trash
cd trash
go mod init trash
go get github.com/go-delve/delve/cmd/dlv
GOARCH=amd64 GOOS=linux go build -o x86_64/dlv github.com/go-delve/delve/cmd/dlv

7. On your M1 Mac, in vscode, update the launch.json file

{
   "configurations": [
       {
           "name": "Connect to server",
           "type": "go",
           "request": "attach",
           "mode": "remote",
           "remotePath": "",
           "port": 9999,
           "host": "127.0.0.1"
       }
   ]
 }

8. On your M1 Mac, Build your project using sam
   sam build
9. On your M1 Mac, Run sam local invoke in debug mode.

sam local invoke -d 9999 --debugger-path ~/trash/x86_64 --debug-args "-delveAPI=2"

10. On your M1 Mac, Wait till you see in the logs that the server is listening to :9999, start the debugging within vscode using vscode run and debug panel.

EXTRA BONUS TIP:
I wanted to pass an SQS Event JSON file into my lambda as an input and for that I had to do the following.

I just had to change the sam command to pass the JSON event file.

sam local invoke -d 9999 --debugger-path ~/trash/x86_64 --debug-args "-delveAPI=2" --event events/sqsevent.json

[aravind-craft]

Nope sorry when the dlv is compiled even for x86_64 on an M1 mac, i am seeing weird behaviour (debugger crashes). Ended up using the dlv that I compiled on an Intel Windows Laptop (I did not have a Intel Mac lying around) instead. Good luck!