Please upstream modules into mainline kernel

[bk2204]

I recently discovered that my ThinkPad X1 Carbon needs an out-of-tree driver (this one) for my camera. That seems atypical for Intel hardware, which is typically well supported upstream.

I've tried with both Debian sid's 5.18.0-2-amd64 and Ubuntu jammy's 5.15, but the driver is included in neither distro, probably because it isn't upstream. Since this hardware is considered important for most people who use an Alder Lake laptop, could you consider upstreaming this driver (and any dependent pieces) into the mainline kernel relatively soon, even if only in the staging area?

I did try to build this myself with DKMS, but due to #13 (both the missing header and then the modpost error), it's not possible to do so, so there is presently no way to make this hardware work on Debian.

[dennis-pries]

Fedora (and I think a lot of other distos) also will not include them, if not upstreamed.

Please do it.

[wbolster]

the new Dell XPS 13 Plus (9320) also requires this driver for the webcam, but building it fails on linux kernel 5.18.7 (arch linux) due to the issues in #13 (missing headers, and unresolved symbols after manually copying headers)

[naps62]

I also have a Dell XPS 13 plus, and just stumbled upon this issue. Also failed to build.
I'm not very profficient in kernel module compilation. Is there a way to get this working at the moment? I'm using Manjaro

[K-MeLeOn]

Dell XPS 13 Plus too, building error on modpost with message 'undefined!' on ov02c10.ko and ov01a10.ko

[paulmenzel]

I notified the Linux upstream folks about the issue. The current recommendation is to not buy such devices. From Greg KH:

On Thu, Aug 11, 2022 at 04:54:53PM +0300, Laurent Pinchart wrote:

For the time being, I agree with your recommendation to not buy these
devices if you care about camera support.

I second this, don't buy these devices if the vendor is not willing to
get their drivers upstreamed properly.

It also sounds like, that it won’t be easily be solved.

Those of you who can, I can only suggest to return the device to not further support vendors screwing up Linux support in this case, and save time fiddling with Linux driver issues.

No idea, who dropped the ball here, but the ChromiumOS/ChromeOS and Ubuntu/Canonical/Dell/Lenovo should have started the discussion with the upstream folks much earlier. It’s a very sad situation for the GNU/Linux ecosystem.

[kbingham]

We can certainly help over at https://libcamera.org if anyone from Intel wants to contact us.

[pinchartl]

@paulmenzel I share your disappointment, but I'd like to focus on how we could improve the situation instead of generating bad PR, as I'm sure the latter doesn't need my help :-)

I won't comment on what happened at Google, Canonical, Dell or Lenovo, as only them have a full picture. All I can say is that it's fairly common for discussions to start in private to decide how to support new product, especially before they are released. I have no reason to believe those discussions didn't start, but they obviously didn't bear fruits so far. The problem isn't easy to solve, and I do understand how development teams can be pressured with product deadlines. Still, this may explain the situation, but not excuse it.

Another point I'd like to make is that we have received technical support from multiple Intel developers when developing IPU3 support in libcamera. There are people with goodwill in their teams, I don't think it would be fair blaming all the developers for the lack of a high-level upstreaming strategy.

We won't be able to fix the mistakes that were made without a clear commitment from Intel, they need to engage directly with the Linux kernel and libcamera upstream developers. Given the amount of work that lies ahead, this has to happen now. My door is fully open.

[Thesola10]

hey, in the mean time I managed to create a patch for pure DKMS building in my AUR package intel-ipu6-dkms-git

[Vascro]

Doesn't appear there are any new updates regarding this ipu6 mipi camera issue. I just got my new XPS 9320 in the mail and noticed this issue immediately. Hopefully Intel fixes this ASAP.

[paulmenzel]

I can only recommend to return the system to not support such products. Otherwise vendors won’t change as long as they can sell such devices.

Additionally, Jonathan Corbet summarized the topic in the LWN article The growing image-processor unpleasantness.

[naps62]

I'm also stuck with a Dell XPS 13 with this issue. don't have a chance to return it unfortunately, although I do agree returning sends the more forceful message to vendors.

FYI @Thesola10 I tried your kernel module (although I'm not 100% sure what it would achieve) on my Manjaro installation. The boot hanged for a few minutes but ended up booting in the end. Stilll no working though, so I removed it again to fix the boot process

[Thesola10]

The lengthened boot process is due to including the intel_vsc drivers into the initramfs. Those can be excluded and they'll load after boot. Otherwise the driver works with icamerasrc on GStreamer so you can try installing that as well

[cjechlitschek]

This might be already known, but I still want to mention it. Canonical provides prebuilt packages for some Dell systems that enable the camera: https://wiki.ubuntu.com/Dell

[netllama]

@cjechlitschek those packages are completely useless if someone isn't running a Debian based distro. Also, see the previous comments which explain why those packages are flawed.

[corebots]

I also miss the webcam under Fedora, pleas fix who can!

[githubtefo]

This is crazy, I just bought the XPS 13 Plus because it was certified by Ubuntu 😩 and even under a Debian based distro, the drivers are not working on 5.17.0-1018-oem kernel.

[naps62]

@githubtefo For what it's worth, I've been running Ubuntu smoothly for a while now, with snapd and all the bloatware removed
I had already bought the laptop, after a 3-month wait, and shipped from US to Europe. so it's not like I had much of a choice

will happily switch back to Arch once possible. But all things considered, I'm quite happy with the performance of the laptop, including the webcam

[karolszk]

Hello, I would like also point out the need for camera drivers for the newest Lenovo Carbon X10 laptop. For 6 version of Linux kernel where are compilable and still lacking in the upstream.

[simisimis]

for my new job I was picking from Thinkpad X1 gen10 and Dell XPS Plus. Was afraid that I made a wrong choice. But from what you wrote I see I would have ended up in this thread regardless 😅

[corebots]

Hi, I've picked up the XPS 13 Plus 5 months ago, been using it with Fedora since then. Mostly good, except the webcam still not working after all that time :(

Please add the webcam drivers to the mainline kernel since it's a bummer needing to use an external usb webcam with such a great and expensive machine. Thanks!

[dmipx]

We are from RealSense working on a "webcam" support for IPU6.
This will support MIPI connected camera on IPU6 to act as "USB UVC", provide access to camera features through video node.
First of all, we want to support RealSense D457 MIPI camera on IPU6 platform.

The work is in progress - we estimate work done by endo of the year, you can observe PR's update.
#73
My repo with updates:
https://github.com/dmipx/ipu6-drivers/tree/iotg_ipu6_isys_video_ctrl

[netllama]

@dmipx that sounds like great news. Hopefully you can improve the README as well, so that there are clear build instructions, instead of the current outdated confusing mess.

[dmipx]

@netllama AFAIK the iotg_ipu6 branch is heavily developed so you can sync with it and see if there is any issue with build steps.
I'm not sure about all supported camera drivers but i know that IPU6 driver can be built without issues.

[pinchartl]

We are from RealSense working on a "webcam" support for IPU6. This will support MIPI connected camera on IPU6 to act as "USB UVC", provide access to camera features through video node. First of all, we want to support RealSense D457 MIPI camera on IPU6 platform.

The work is in progress - we estimate work done by endo of the year, you can observe PR's update. #73 My repo with updates: https://github.com/dmipx/ipu6-drivers/tree/iotg_ipu6_isys_video_ctrl

Are you planning to upstream this work, or just merge it in the out-of-tree IPU6 driver ?

[dmipx]

For now we have dispute about it with ipu6 team so we started to implement it as out-of-tree.
RealSense, as usual, will provide debian package for ubuntu distros with DKMS supporting d457 cameras.
If it will be demanded i will try to support laptops mipi cameras but for now - I have no hardware to work on it. Let's see how it goes.
In long term we want to upstream but this should be community demand.

[paulmenzel]

In long term we want to upstream but this should be community demand.

Please contact the Linux (subsystem) kernel mailing list with the involved developers in Cc notifying them about your efforts. This GitHub issue is not the right forum for this.

[MorganMLGman]

Hi, any updates regarding this topic? Is it possible to have this included in mainline kernel or people with alder lake laptops will be left with not working webcam on linux?

[chromer030]

Please upstream the module , people are suffering from lack of support.

[SmokinCaterpillar]

Hey is there any update? I bought a new X1Gen10 and, working in a remote setting without a camera, this machine is a bit of a very expensive paperweight.

[SmokinCaterpillar]

Small question / reminder, first post of this issue was on Jun 21, 2022 that is now over a year and a half ago. What has happened in the meantime? Any update from intel?

[arpad-m]

There is ongoing upstreaming efforts for various parts of the webcam driver stack:

• https://lore.kernel.org/linux-media/[email protected]/
• https://lore.kernel.org/linux-media/[email protected]/
• https://lists.libcamera.org/pipermail/libcamera-devel/2024-January/040113.html

jwrdegoede will also have a talk at fosdem about it. See also their recent blog post.

[paulmenzel]

Thank you Red Hat and Linaro, and, therefore, to all organizations buying RHEL licenses and not being parasites by using CentOS or other clones. Unfortunately that’s often banks and insurance companies. I am curious, if Dell or Google ChromeOS participated in this effort.

Anyway, big kudos again to Red Hat (@jwrdegoede) and Linaro.

[pinchartl]

I have good hopes that the ISYS driver will land upstream in a reasonably near future.

The PSYS is a different story. It seems much more likely to me that IPU6-based platforms will be supported by libcamera using a software-based ISP (possibly GPU-accelerated at some point). That will incur some performance and power consumption costs, but should provide decent-enough images for basic video conferencing needs.

[DemiMarie]

I have good hopes that the ISYS driver will land upstream in a reasonably near future.

The PSYS is a different story. It seems much more likely to me that IPU6-based platforms will be supported by libcamera using a software-based ISP (possibly GPU-accelerated at some point). That will incur some performance and power consumption costs, but should provide decent-enough images for basic video conferencing needs.

What is the reason for this? Is it because Linux has no framework for PSYS drivers, or because Intel is not willing to provide a driver or documentation, and therefore extensive reverse engineering would be required?

[pinchartl]

Only Intel can officially answer that question. You may be able to form your personal opinion on the matter by looking at how much information the existing PSYS driver exposes (or doesn't expose) about the parameters the ISP receives, the statistics it produces, and how it operates in general.

[DemiMarie]

Ah, so this is a situation where the actual driver runs in userspace, with the kernel driver just being a pipe between userspace and the on-device firmware.

[paulmenzel]

Slide 7 of the FOSDEM 2024 presentation slides says:

Reluctance/refusal of some vendors to disclose “secret sauce”

Doesn’t that contradict Intel’s pledge to openness in 2021?

[DemiMarie]

I have good hopes that the ISYS driver will land upstream in a reasonably near future.

The PSYS is a different story. It seems much more likely to me that IPU6-based platforms will be supported by libcamera using a software-based ISP (possibly GPU-accelerated at some point). That will incur some performance and power consumption costs, but should provide decent-enough images for basic video conferencing needs.

Does that mean that for those who need high-quality images or good battery life, an external high-end USB webcam will be needed for the forseeable future?

[pinchartl]

For those who need high-quality images and good battery life with upstream drivers, yes. That is my understanding, but I would be very happy if Intel proved me wrong :-)

[acegallagher]

See here for discussion and status on kernel mailing list:

https://lwn.net/ml/linux-media/[email protected]/#t

[oblique]

The v6 of the patch was posted for review two days ago: https://lwn.net/ml/linux-media/[email protected]/

[arpad-m]

Also, v8 of the softisp patch has been posted recently https://lists.libcamera.org/pipermail/libcamera-devel/2024-April/041448.html

the patches I've linked above have made it into Linux 6.8.

[chromer030]

the driver upstreamed in linux 6.10.

https://lore.kernel.org/lkml/[email protected]/

EDIT: relative commit which contains codes & documentation about how driver works:

torvalds/linux@6fd600d

[prochac]

the driver upstreamed in linux 6.10.

https://lore.kernel.org/lkml/[email protected]/

My for-sure-last Intel-based NTB will have a working webcam in 2024, yay.

[pinchartl]

the driver upstreamed in linux 6.10.
https://lore.kernel.org/lkml/[email protected]/

While it's an important part of the puzzle, the ISYS driver alone isn't enough. The PSYS driver is still missing, and that will take more time. In the meantime, libcamera has a software-based ISP implementation (contributed by Redhat and Linaro, kudos to them) that will be usable as a stop-gap measure.

My for-sure-last Intel-based NTB will have a working webcam in 2024, yay.

You will also need a driver for the camera sensor in your device. With some luck, there will already be one in the kernel. It may also need integration in libcamera, if not done yet.

TL;DR: Avoiding the extra CPU and power consumption of the software ISP will have to wait for next year, but the good news is that there's a chance you'll get images out of your camera in 2024 :-)

[DemiMarie]

the driver upstreamed in linux 6.10.
https://lore.kernel.org/lkml/[email protected]/

While it's an important part of the puzzle, the ISYS driver alone isn't enough. The PSYS driver is still missing, and that will take more time.

Does this mean that a PSYS driver is at least planned? I thought it was completely blocked by Intel not being willing to release any userspace for the PSYS and upstream (rightly) refusing to take a driver with no working userspace.

(...)

My for-sure-last Intel-based NTB will have a working webcam in 2024, yay.

You will also need a driver for the camera sensor in your device. With some luck, there will already be one in the kernel. It may also need integration in libcamera, if not done yet.

Does Linux have drivers for the sensors most commonly found in laptops?

[pinchartl]

Does this mean that a PSYS driver is at least planned? I thought it was completely blocked by Intel not being willing to release any userspace for the PSYS and upstream (rightly) refusing to take a driver with no working userspace.

I'm not sure I would say "planned". The effort isn't dead, but the relevant upstream kernel communities have not blessed any architecture and technical solution yet. Whether or not a driver could land upstream, and at what time, are not questions that can be answered with a complete guarantee at this point.

(...)

My for-sure-last Intel-based NTB will have a working webcam in 2024, yay.

You will also need a driver for the camera sensor in your device. With some luck, there will already be one in the kernel. It may also need integration in libcamera, if not done yet.

Does Linux have drivers for the sensors most commonly found in laptops?

Some of them are supported. Intel has upstreamed sensor drivers in the past few of years, so I would say they have done work in that area, but I don't know what shares of the Intel-based laptops that would cover, or what models.

[DemiMarie]

Does this mean that a PSYS driver is at least planned? I thought it was completely blocked by Intel not being willing to release any userspace for the PSYS and upstream (rightly) refusing to take a driver with no working userspace.

I'm not sure I would say "planned". The effort isn't dead, but the relevant upstream kernel communities have not blessed any architecture and technical solution yet. Whether or not a driver could land upstream, and at what time, are not questions that can be answered with a complete guarantee at this point.

I know that drivers for other ISPs have been accepted upstream. Is the lack of an open userspace for the PSYS still the main blocker, or are there other problems that also need to be resolved?

[Vascro]

the driver upstreamed in linux 6.10.

https://lore.kernel.org/lkml/[email protected]/

EDIT: relative commit which contains codes & documentation about how driver works:

torvalds/linux@6fd600d

Is there any talk of backporting the driver to 6.9? I'm aware that the PSYS driver would still be needed, but with 6.9 being so new, it makes sense to me to try supporting it.

[norru]

I've updated my Debian Sid to latest, which brought in kernel 6.10.6.

There is no binary kernel module for IPU6 (but I can see the ipu6-fw in the firmware package).

Any idea why the driver could be missing from the distro?

[rk9qn3j]

I'm running Fedora 40 with 6.10.6 - same here..

[otetard]

Any idea why the driver could be missing from the distro?

IPU6 was enabled in the 6.11~rc4-1~exp1 release of the Linux kernel (available in experimental). Unfortunately, at the moment, the only available module is ov2740.

More information on the next steps required to have support in Debian available in this bug: #1074441.

[norru]

I'll keep waiting, thanks for the update!

[norru]

For reference, vanilla Debian kernel 6.11.6 on sid/unstable now detects the camera out of the box, which is a great improvement:

nico@thinkpad-x1-debian:~$ sudo dmesg | grep ipu
[   24.321680] intel-ipu6 0000:00:05.0: enabling device (0000 -> 0002)
[   24.321944] intel-ipu6 0000:00:05.0: IPU6 in secure mode touch 0x0 mask 0xff
[   24.356330] intel-ipu6 0000:00:05.0: FW version: 20230925
[   24.357343] intel-ipu6 0000:00:05.0: Found supported sensor INT3474:01
[   24.357534] intel-ipu6 0000:00:05.0: Connected 1 cameras
[   24.358629] intel-ipu6 0000:00:05.0: Sending BOOT_LOAD to CSE
[   24.367915] intel-ipu6 0000:00:05.0: Sending AUTHENTICATE_RUN to CSE
[   24.438096] intel-ipu6 0000:00:05.0: CSE authenticate_run done
[   24.438126] intel-ipu6 0000:00:05.0: IPU6-v3[a75d] hardware version 5
[ 1739.162014] intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Single packet header error corrected
[ 1739.162032] intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Multiple packet header errors detected
[ 1739.162037] intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Payload checksum (CRC) error
[ 1739.162041] intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Incomplete long packet detected
[ 1739.162045] intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame short packet discarded
[ 1739.162049] intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded
[ 1739.397941] intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Single packet header error corrected
[ 1739.397961] intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Multiple packet header errors detected
[ 1739.397967] intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Payload checksum (CRC) error
[ 1739.397971] intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Incomplete long packet detected
[24034.354786] intel-ipu6 0000:00:05.0: IPU6 in secure mode
[32087.700242] intel-ipu6 0000:00:05.0: IPU6 in secure mode
[34059.922569] intel-ipu6 0000:00:05.0: IPU6 in secure mode
[36265.036903] intel-ipu6 0000:00:05.0: IPU6 in secure mode

Available cameras:
1: 'ov2740' (\_SB_.PC00.LNK1)

nico@thinkpad-x1-debian:~$ uname -a
Linux thinkpad-x1-debian 6.11.6-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.11.6-1 (2024-11-04) x86_64 GNU/Linux

product: 21HMCTO1WW (LENOVO_MT_21HM_BU_Think_FM_ThinkPad X1 Carbon Gen 11)
vendor: LENOVO
version: ThinkPad X1 Carbon Gen 11

Using camera \_SB_.PC00.LNK1 as cam0
0: 1924x1092-ABGR8888
 * Pixelformat: ABGR8888 (2x2)-(1924x1092)/(+2,+2)
  - 160x120
  - 240x160
  - 320x240
  - 400x240
  - 480x320
  - 640x360
  - 640x480
  - 720x480
  - 768x480
  - 854x480
  - 720x576
  - 800x600
  - 960x540
  - 1024x576
  - 960x640
  - 1024x600
  - 1024x768
  - 1280x720
  - 1152x864
  - 1280x800
  - 1360x768
  - 1366x768
  - 1440x900
  - 1280x1024
  - 1536x864
  - 1280x1080
  - 1600x900
  - 1400x1050
  - 1680x1050
  - 1920x1080
 * Pixelformat: XBGR8888 (2x2)-(1924x1092)/(+2,+2)
  - 160x120
  - 240x160
  - 320x240
  - 400x240
  - 480x320
  - 640x360
  - 640x480
  - 720x480
  - 768x480
  - 854x480
  - 720x576
  - 800x600
  - 960x540
  - 1024x576
  - 960x640
  - 1024x600
  - 1024x768
  - 1280x720
  - 1152x864
  - 1280x800
  - 1360x768
  - 1366x768
  - 1440x900
  - 1280x1024
  - 1536x864
  - 1280x1080
  - 1600x900
  - 1400x1050
  - 1680x1050
  - 1920x1080
 * Pixelformat: BGR888 (2x2)-(1924x1092)/(+2,+2)
  - 160x120
  - 240x160
  - 320x240
  - 400x240
  - 480x320
  - 640x360
  - 640x480
  - 720x480
  - 768x480
  - 854x480
  - 720x576
  - 800x600
  - 960x540
  - 1024x576
  - 960x640
  - 1024x600
  - 1024x768
  - 1280x720
  - 1152x864
  - 1280x800
  - 1360x768
  - 1366x768
  - 1440x900
  - 1280x1024
  - 1536x864
  - 1280x1080
  - 1600x900
  - 1400x1050
  - 1680x1050
  - 1920x1080
 * Pixelformat: RGB888 (2x2)-(1924x1092)/(+2,+2)
  - 160x120
  - 240x160
  - 320x240
  - 400x240
  - 480x320
  - 640x360
  - 640x480
  - 720x480
  - 768x480
  - 854x480
  - 720x576
  - 800x600
  - 960x540
  - 1024x576
  - 960x640
  - 1024x600
  - 1024x768
  - 1280x720
  - 1152x864
  - 1280x800
  - 1360x768
  - 1366x768
  - 1440x900
  - 1280x1024
  - 1536x864
  - 1280x1080
  - 1600x900
  - 1400x1050
  - 1680x1050
  - 1920x1080
 * Pixelformat: ARGB8888 (2x2)-(1924x1092)/(+2,+2)
  - 160x120
  - 240x160
  - 320x240
  - 400x240
  - 480x320
  - 640x360
  - 640x480
  - 720x480
  - 768x480
  - 854x480
  - 720x576
  - 800x600
  - 960x540
  - 1024x576
  - 960x640
  - 1024x600
  - 1024x768
  - 1280x720
  - 1152x864
  - 1280x800
  - 1360x768
  - 1366x768
  - 1440x900
  - 1280x1024
  - 1536x864
  - 1280x1080
  - 1600x900
  - 1400x1050
  - 1680x1050
  - 1920x1080
 * Pixelformat: XRGB8888 (2x2)-(1924x1092)/(+2,+2)
  - 160x120
  - 240x160
  - 320x240
  - 400x240
  - 480x320
  - 640x360
  - 640x480
  - 720x480
  - 768x480
  - 854x480
  - 720x576
  - 800x600
  - 960x540
  - 1024x576
  - 960x640
  - 1024x600
  - 1024x768
  - 1280x720
  - 1152x864
  - 1280x800
  - 1360x768
  - 1366x768
  - 1440x900
  - 1280x1024
  - 1536x864
  - 1280x1080
  - 1600x900
  - 1400x1050
  - 1680x1050
  - 1920x1080

I can capture frames using libcamera's tool but not got around to get a video stream yet. Doesn't look like there is an upstream Debian package for v4l2-relayd

UPDATE: I got the camera working as a source in OBS, qcam, cheese via v4l2-relayd + v4l2loopback. I had to build v4l2-relayd from source to manually fix a bug. The image quality is poor due to lack of calibration but the framerate is reasonable and CPU usage stays low.

[DemiMarie]

For those who need high-quality images and good battery life with upstream drivers, yes. That is my understanding, but I would be very happy if Intel proved me wrong :-)

Would PSYS support be sufficient for high image quality, or would that also require a lot of platform-specific tuning? Is this tuning something that could be automated so that it could easily be done by the community, or will Linux always lag behind the platform (Windows) driver?

[pinchartl]

Would PSYS support be sufficient for high image quality, or would that also require a lot of platform-specific tuning?

"PSYS support" means both a kernel driver and userspace ISP control algorithms (in libcamera) to compute the ISP parameters. The algorithms require a set of per-camera module data commonly called tuning data or calibration data. The nature and representation of the tuning data depends on the implementation of the ISP hardware and of the userspace algorithms. Two different algorithms implementations (for instance an open-source version in libcamera and a closed-source version from Intel) would both express similar data in different ways, and also use different data.

Is this tuning something that could be automated so that it could easily be done by the community, or will Linux always lag behind the platform (Windows) driver?

Tuning is a manual process performed with a tuning tool. libcamera has a tuning tool infrastructure that would need to be extended to support the open-source PSYS ISP control algorithms.

IPU6-based machines supported by Intel (both running Windows and Chrome OS) ship a tuning data file produced by a closed-source tuning tool (that is not available publicly even in binary form to the best of my knowledge). This tuning data file uses an undocumented binary format. Part of the parameters it contains could be reused by open-source ISP control algorithms, if the file format were to be documented.