[Package Request] - Python 3.12

[matejsp]

What package is missing from Amazon Linux 2023? Please describe and include package name.
Python 3.12 that was released today.
https://pythoninsider.blogspot.com/2023/10/python-3120-final-now-available.html

Is this an update to existing package or new package request?
Update to existing package (would be nice to have python 3.11 and python 3.12 available as a transition)

Is this package available in Amazon Linux 2? If it is available via external sources such as EPEL, please specify.
Old versions of python

Any additional information you'd like to include. (use-cases, etc)
Using latest python 3.12 features.

[stewartsmith]

There's chatter over on aws/elastic-beanstalk-roadmap#174 about Beanstalk and up to date Python support. By having Python 3.12 packaged in AL2023, adding this as a Beanstalk platform is less work for the Beanstalk team.

[ccassidy-maxar]

Adding a note from my accidental duplicate ticket: Lambda now supports Python 3.12 using the AL2023 base image.
https://aws.amazon.com/about-aws/whats-new/2023/12/aws-lambda-support-python-3-12/

[bryantbiggs]

would this be switching the default python from 3.9 to 3.12, or is it simply adding support to install 3.12 (similar to 3.11)? I'd like to use just one python installation in the AL2023 minimal container image, ideally 3.12 due to the performance improvements

[stewartsmith]

would this be switching the default python from 3.9 to 3.12, or is it simply adding support to install 3.12 (similar to 3.11)? I'd like to use just one python installation in the AL2023 minimal container image, ideally 3.12 due to the performance improvements

It's for having it as an option, not replacing the system Python.

The good news is that the minimal container image doesn't have any version of Python in it due to it having microdnf rather than the full dnf, so if you added a version of Python to it, it's entirely your choice what version that is.

[aperuru]

Unable to install python 3.12.1 from source on AmazonLinux-2023 (2023.3.20231218.0)

here are the steps from my dockerfile

RUN dnf install -y gzip make tar wget which openssl \
         bzip2-devel gcc-c++ libffi-devel openssl-devel \
         openssh-clients sqlite-devel zlib-devel readline-devel \
         && dnf clean all
RUN wget -nv https://www.python.org/ftp/python/3.12.1/Python-3.12.1.tgz \
        && tar xzf Python-3.12.1.tgz \
        && cd Python-3.12.1 \ 
        && ./configure --enable-optimizations \
        && make -j $(nproc) \    #fails at this step
        && make altinstall \

Error on step make -j $(nproc)

gcc: internal compiler error: Segmentation fault signal terminated program cc1
Please submit a full bug report,
with preprocessed source if appropriate.
See <https://github.com/amazonlinux/amazon-linux-2022> for instructions.
make[2]: *** [Makefile:2706: Parser/string_parser.o] Error 4
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/tmp/Python-3.12.1'
make[1]: *** [Makefile:777: profile-gen-stamp] Error 2
make[1]: Leaving directory '/tmp/Python-3.12.1'
make: *** [Makefile:789: profile-run-stamp] Error 2

I also tried using just make

make[2]: *** [Makefile:1366: Python/frozen_modules/frozen_only.h] Segmentation fault (core dumped)
make[2]: Leaving directory '/tmp/Python-3.12.1'
make[1]: *** [Makefile:777: profile-gen-stamp] Error 2
make[1]: Leaving directory '/tmp/Python-3.12.1'
make: *** [Makefile:789: profile-run-stamp] Error 2

Has anyone run into similar issues? Any help would be greatly appreciated

[matejsp]

If you get core dumped or killed process try assigning more RAM to docker builder and try again.

I didn't try building it on al2023, but we used Python 3.12 from official lambda repository. Its not perfect but does the job for now. I would really like to have it in al2023 instead.

FROM amazonlinux/amazonlinux:2023.3.20240108.0-minimal as base-arm64
# Taken from al2023 lambda package https://github.com/aws/aws-lambda-base-images/blob/python3.12/x86_64/Dockerfile.python3.12
ADD a6a32126a5d6ad6ee275fd1b290d6a3381de37020554fa2b4c1818a90f8c15a4.tar.xz /

FROM amazonlinux/amazonlinux:2023.3.20240108.0-minimal as base-amd64
# Taken from al2023 lambda package https://github.com/aws/aws-lambda-base-images/blob/python3.12/x86_64/Dockerfile.python3.12
ADD 08e61bf1511d156676b2e9ac9eea5a8727c2c5a67fa4d7a957cee02af4a51252.tar.xz /

FROM base-${TARGETARCH}

RUN dnf install -y gzip make tar wget which openssl \
         bzip2-devel gcc-c++ libffi-devel openssl-devel \
         openssh-clients sqlite-devel zlib-devel readline-devel \
         && dnf clean all

...

[matejsp]

I tried your command running on docker minimal and got ir built without any problems on 8gb docker machine.

 docker run -it amazonlinux/amazonlinux:2023.3.20240108.0-minimal bash

echo  $(nproc)
8

make -j $(nproc)
...
Checked 111 modules (31 built-in, 73 shared, 1 n/a on linux-x86_64, 0 disabled, 6 missing, 0 failed on import)
make[1]: Leaving directory '/Python-3.12.1'
bash-5.2#

[maunzCache]

Adding a note from my accidental duplicate ticket: Lambda now supports Python 3.12 using the AL2023 base image. https://aws.amazon.com/about-aws/whats-new/2023/12/aws-lambda-support-python-3-12/

Sidenote: I doesn't seem to be available as a package in the base image itself judging by https://docs.aws.amazon.com/linux/al2023/release-notes/all-packages-AL2023.3.html

Edit: If you do not rely on AL2023, you might directly reference the runtime via https://hub.docker.com/r/amazon/aws-lambda-python

[dkavanagh]

Having this package would help me bundling python libs like cryptography which include native code. We'd like to build for 3.12 on AL2023.

[matejsp]

@maunzCache No it is not availabe as a package yet. In docker you can just add docker layer but for EC2 you are out of luck currently (unless you build it yourself). Really hope to have 3.12 available soon in AL2023.

[antm-pp]

Given the lambda python3.12 runtime is the only one based on AL2023 it'd be useful to have a common EC2 base for testing outside of lambda. Presumably it must already be in codified in repos in some form to be available via dnf and maintained in the lambda docker images?

[matejsp]

Any update on the python 3.12 when avaialable?
I was really looking forward to new 2024.4 release but then I saw there is still no new python available.
Lambda supports python 3.12 since december. And we would like to have the same python as in lambda on our ec2 or docker image. Lambda 3.11 is using AL2 and only 3.12 is using AL2023.

[matejsp]

I stumbled upon release notes for RHEL 9.4 (https://developers.redhat.com/articles/2024/05/01/whats-new-red-hat-enterprise-linux-94).
And they already support python 3.12.

I tried installed python 3.12 via AlmaLinux packages :

dnf install https://repo.almalinux.org/almalinux/9/AppStream/x86_64/os/Packages/python3.12-3.12.1-4.el9.x86_64.rpm https://repo.almalinux.org/almalinux/9/AppStream/x86_64/os/Packages/python3.12-libs-3.12.1-4.el9.x86_64.rpm        https://repo.almalinux.org/almalinux/9/AppStream/x86_64/os/Packages/python3.12-pip-wheel-23.2.1-4.el9.noarch.rpm https://repo.almalinux.org/almalinux/9/AppStream/x86_64/os/Packages/libnsl2-2.0.0-1.el9.x86_64.rpm

And got it working. Too bad still no official release in AL3.

[zztopping]

Chiming in here as well, anyone building packages (with OS lib deps) on Amazon Linux for python on Lambda are kinda stuck on 3.8 right now unless they build Python from source:

• Lambda's Python 3.12 is only available on AL2023, but AL2023 images do not have a managed package for py3.12
• Lambda's Python 3.11 is only available on AL2, but AL2 does not support 3.11 as a AWS-managed package (I assume it never will, given our place in the lifetime of AL2)
• Without building from source, the only AL<->Python match between an AL image and Lambda is python3.8-AL2. Which is EOS in early 2025

[abhishek-parative]

Any updates?

[maunzCache]

Any updates?

2023.4 did not bring a python update. Actually i would not expect a new python until AL2025. If you only need python 3.12 for e.g. local lambda tests or similar you may use https://hub.docker.com/r/amazon/aws-lambda-python instead.

[matejsp]

Any updates?

Just checked latest 2023.5 and unfortunately still lacking python 3.12 ... Is there any chance to get a least some timeline for python 3.12 since it was released 9 months ago.

[direvus]

* Lambda's Python 3.11 is only available on AL2, but AL2 does not support 3.11 as a AWS-managed package (I assume it never will, given our place in the lifetime of AL2)

Yes, and to make matters worse, AL2 is stuck on GLIBC 2.26 (coming to you live from August 2017), which means AL2 is totally non-viable for some libraries.

[matejsp]

@stewartsmith Any chance to get this finally in AL2023.6? Before python 3.13 is released in a couple of days.

[eric-spitler]

I'm going to guess they'll skip 3.12 altogether and jump to 3.13.

[matejsp]

Well I hope they don't, because we are still after looking for python version parity with ec2 and lambdas. Currently we use 3.11 on ec2 and 3.11 on lambda (but it still uses AL2 https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html). And then we have problems with libs like openssl and others for binary wheels (like cryptography).

Unless they start to support 3.13 in both environments. Together with opt in Free-threaded builds.

Would be nice to know what their plan is, so we can also plan accordingly.

[louis-jaris]

My two cents: given the Release Cadence (and maintenance) of Amazon Linux, and compared with the python release and maintenance cadence, it would make more sense for AL2025 to skip python3.12 and to directly support python3.13 (given that AL2025 and python3.13 have more overlapping in support than python3.12)

So, since python3.13 is out, I'd basically prefer to see #811 than this issue being done (as it would reduce the workload of AL maintainers)

[eithermaxmax]

@louis-jaris the progress has to be incremental to incermentally unblock audiences. There are tons of tested libraries that support 3.12 by now, I can't say the same about 3.13. If 3.12 is skipped, people will have to wait to upgrade untill all their transitive dependency chains are reliably supporting 3.13.

[louis-jaris]

the progress has to be incremental to incermentally unblock audiences

@eithermaxmax not necessarily -- c.f. AL2023 that does not support python3.10 at all -- it only expose 3.9 and python3.11.

I will see what Amazon Linux Maintainers are planning to do -- if I'm not mistaken, we might see AL2025 around march 2025, so we will have the final answer at that point.