Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upstream INT3472 GPIO type 0x12 / Lattice MIPI aggregator support #281

Open
jwrdegoede opened this issue Oct 13, 2024 · 11 comments
Open

Upstream INT3472 GPIO type 0x12 / Lattice MIPI aggregator support #281

jwrdegoede opened this issue Oct 13, 2024 · 11 comments

Comments

@jwrdegoede
Copy link
Contributor

Some Meteor Lake laptops return a GPIO/pin type of 0x12 from the INT3472 79234640-9e10-4fea-a5c1-b5aa8b19756f _DSM.

A while ago 2 patches have been posted upstream for this indicating that this is related to a new "Lattice MIPI aggregator" chip:

https://lore.kernel.org/platform-driver-x86/[email protected]/
https://lore.kernel.org/platform-driver-x86/[email protected]/

With mainline kernel + libcamera softISP support landing in Fedora 41:
https://hansdegoede.dreamwidth.org/28841.html

Users are now reporting issues with things not working on e.g. a HP Spectre x360 14-eu0xxx with an OVTI08F4/ov08x40 sensor.

dmesg shows the following line indicating that this laptop has the "Lattice MIPI aggregator" chip:

[    4.658940] int3472-discrete INT3472:01: GPIO type 0x12 unknown; the sensor may not work

I would like to start working on supporting this setup in the mainline kernel, but first I need to better understand what the "Lattice MIPI aggregator" chip exactly is.

The ACPI tables show that the sensor is directly connected to one of the designware I2C controllers of the main SoC, so there seems to be no USB attached IO-expander (LJCA) ?

I guess that the "Lattice MIPI aggregator" takes the place of the IO-expander and that it takes care of providing extclk/xvclk, voltages and control of reset/powerdown GPIOs? And that all of that then gets controlled by this one single handshake GPIO and the chip takes care of doing the necessary power up/down sequencing with all necessary delays itself ?

Can you please answer the following questions:

  1. What IOs are there between the "Lattice MIPI aggregator" and the main SoC/CPU
  2. Is there some bus like I2C/SPI/USB to the main SoC for e.g. do firmware updates?
  3. To get the sensor to work, do we need a driver for the "Lattice MIPI aggregator", or is is just driving the handshake GPIO high enough to power-up the sensor ?
  4. What IOs are there between the "Lattice MIPI aggregator" and the sensor?
  5. Does the Lattice MIPI aggregator also touch the MIPI CSI signals, like the IVSC chip does, or does it only control GPIOs / clks / regulators?
  6. Does the Lattice MIPI aggregator also touch the I2C signals going to the sensor ?
  7. Can you provide an example / simplified schematic with the signals going to/from the main SoC to the Lattice MIPI aggregator and the signals running between the aggregator and the sensor ?

My goal with these questions is to better understand what the exact function / purpose of the "Lattice MIPI aggregator" is.

Once I have a clearer picture of what the exact function / purpose of the "Lattice MIPI aggregator" is I'll send a proposal for how to get this supported in the upstream kernel to the linux-media mailinglist.
@jwrdegoede
Copy link
Contributor Author

Hmm, doing a duckduckgo search for "Lattice MIPI aggregator" ends up at:

https://www.latticesemi.com/Products/FPGAandCPLD/CrossLink

If the "Lattice MIPI aggregator" is a MIPI CSI-2 mux/demux then it really needs to be modelled as part of the media-controller graph similar to how upstream handles the IVSC chip.

@alexstore06 alexstore06 mentioned this issue Oct 13, 2024
Closed
@hao-yao
Copy link
Contributor

hao-yao commented Oct 14, 2024

Hi @jwrdegoede ,

  1. What IOs are there between the "Lattice MIPI aggregator" and the main SoC/CPU

SoC controls Lattice by USB command. The virtual I2C & GPIO are actually on USB connection between SoC and Lattice. There are also MIPI CSI-2 connections between SoC & Lattice.

  1. Is there some bus like I2C/SPI/USB to the main SoC for e.g. do firmware updates?

Yes, SoC can transfer firmware updates to Lattice chip. But I don't know the detail steps.

  1. To get the sensor to work, do we need a driver for the "Lattice MIPI aggregator", or is is just driving the handshake GPIO high enough to power-up the sensor ?

We need at least a USB virtual GPIO & I2C driver for Lattice device. Once the drivers are ready, we can drive the virtual handshake GPIO to high to powe-up the sensor.

  1. What IOs are there between the "Lattice MIPI aggregator" and the sensor?

I2C, MIPI CSI-2 and some power & clock pins.

  1. Does the Lattice MIPI aggregator also touch the MIPI CSI signals, like the IVSC chip does, or does it only control GPIOs / clks / regulators?

Yes.

  1. Does the Lattice MIPI aggregator also touch the I2C signals going to the sensor ?

For my point of view, the I2C is virtual so actually driver send I2C request to USBIO driver, then USBIO driver construct the USB command from SoC to Lattice, then Lattice hardware/firmware send the actual I2C command to camera sensor.

  1. Can you provide an example / simplified schematic with the signals going to/from the main SoC to the Lattice MIPI aggregator and the signals running between the aggregator and the sensor ?

I can draw what I know here:

+--------------+        +-------------+          +------------+
| SoC          |  USB   |             |   GPIOs  |            |
|              |========|             |----------|            |
|              |        |             |          |            |
|              |        |             |   I2C    |   Camera   |
|              |        |             |==========|   Sensor   |
|              |        |   Lattice   |          |            |
|   +------+   |  MIPI  |             |   MIPI   |            |
|   | IPU6 |===|========|             |==========|            |
|   +------+===|========|             |==========|            |
+--------------+        +-------------+          +------------+

@lwang47-amr Please help correct me if anything wrong, thanks!

@jwrdegoede
Copy link
Contributor Author

@hao-yao thank you for your answers and for the ASCII art schematic that is very helpful.

So it looks like the MIPI-CSI2 parts needs to become a media-controller entity on the media-controller graph just like how this is done for the iVSC chip.

I also see that the drivers for this from https://github.com/intel/usbio-drivers/ are still using the old MFD framework similar to how the iVSC drivers where changed from MFD to use the new auxbus interface, this will also need to use the auxbus with separate auxbus children devices for the I2C-controller, GPIO-controller and MIPI/vision parts.

@lwang47-amr, what are the plans / timeline for upstreaming the drivers for this ?

@n3thshan
Copy link

hi its been a month. any new updates on the issue?

@moozhub
Copy link

moozhub commented Nov 23, 2024

This is what I'm currently getting on Fedora 41, qcam shows no available sensors to use. I forced the usage of the ov08x40 kernel module and blacklisted ov2740 since it seemed to get automatically loaded for some reason, but didn't make any difference.

This is on a Lenovo X1C G12

v4l2-ctl --list-devices

ipu6 ():
	/dev/video0
	/dev/video1
	/dev/video2
	/dev/video3
	/dev/video4
	/dev/video5
	/dev/video6
	/dev/video7
	/dev/video8
	/dev/video9
	/dev/video10
	/dev/video11
	/dev/video12
	/dev/video13
	/dev/video14
	/dev/video15
	/dev/video16
	/dev/video17
	/dev/video18
	/dev/video19
	/dev/video20
	/dev/video21
	/dev/video22
	/dev/video23
	/dev/video24
	/dev/video25
	/dev/video26
	/dev/video27
	/dev/video28
	/dev/video29
	/dev/video30
	/dev/video31
	/dev/video32
	/dev/video33
	/dev/video34
	/dev/video35
	/dev/video36
	/dev/video37
	/dev/video38
	/dev/video39
	/dev/video40
	/dev/video41
	/dev/video42
	/dev/video43
	/dev/video44
	/dev/video45
	/dev/video46
	/dev/video47

ipu6 (PCI:0000:00:05.0):
	/dev/media0

dmesg

[    6.015418] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.018610] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.018639] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.019412] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.028520] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.028544] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.034398] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.060135] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.060173] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.069734] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.074921] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.074943] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.083135] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.085945] intel-ipu6 0000:00:05.0: enabling device (0000 -> 0002)
[    6.086035] intel-ipu6 0000:00:05.0: IPU6 in secure mode touch 0x80000000 mask 0x0
[    6.092570] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.092605] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.101185] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.113770] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.113804] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.118689] intel-ipu6 0000:00:05.0: FW version: 20230925
[    6.127676] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.127720] intel-ipu6 0000:00:05.0: Found supported sensor OVTI08F4:00
[    6.127778] intel-ipu6 0000:00:05.0: Connected 1 cameras
[    6.128957] intel-ipu6 0000:00:05.0: Sending BOOT_LOAD to CSE
[    6.130439] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.130598] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.141928] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.159891] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.159917] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.164446] intel-ipu6 0000:00:05.0: Sending AUTHENTICATE_RUN to CSE
[    6.169272] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.173361] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.173442] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.176585] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.182496] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.183398] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.192496] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.198030] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.198054] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.211120] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.211280] intel-ipu6 0000:00:05.0: CSE authenticate_run done
[    6.211287] intel-ipu6 0000:00:05.0: IPU6-v4[7d19] hardware version 6
[    6.229019] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.229578] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.230157] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.242916] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.243488] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.259828] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.267806] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.268601] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.269773] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.276817] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.277537] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.300887] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.304394] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.305310] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.312820] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.316217] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.317169] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.324922] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.332266] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.332343] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.335899] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.343871] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.343922] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.353561] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.373220] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.374162] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.377687] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.515352] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.515638] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.517426] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.518770] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.519095] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.521853] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.523991] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.524378] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.542964] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.544309] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.544599] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.548337] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.549570] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.549856] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    6.865025] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    6.866654] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    6.866923] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[    7.055045] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[    7.055768] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[    7.055797] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[   14.427982] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[   14.428809] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[   14.428841] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[   17.533445] int3472-discrete INT3472:0c: cannot find GPIO chip INTC1007:00, deferring
[   17.534376] int3472-discrete INT3472:0b: GPIO type 0x12 unknown; the sensor may not work
[   17.534448] int3472-discrete INT3472:0b: cannot find GPIO chip INTC1007:00, deferring
[   17.536525] platform INT3472:0c: deferred probe pending: int3472-discrete: Failed to get GPIO
[   17.536528] platform INT3472:0b: deferred probe pending: int3472-discrete: Failed to get GPIO

@mluessi
Copy link

mluessi commented Dec 6, 2024

Is this the Lattice MIPI aggregator USB device? This is from a thinkpad X1 2-in-1 Gen 9

Bus 003 Device 003: ID 2ac1:20c9 SLS Lattice AI USB 2.0

@mluessi
Copy link

mluessi commented Dec 7, 2024

I looked into this a bit more. To answer my own question, the USB device with ID 2ac1:20c9 is the MIPI aggregator. The ID is listed in the usbio-driver here as a Lattice NX33, which I assume is a Lattice Crosslink-NX. I assume the "SLS Lattice AI USB 2.0" name comes from the USB 2.0 IP core they use for the crosslink FPGA.

I compiled a 6.12.1 kernel with the handshake pin patch that @jwrdegoede linked above applied. In addition, I compiled the usbio-drivers kernel module and loaded it before the int3472-discrete module.

With those changes, both the INT3472:0b: GPIO type 0x12 unknown; the sensor may not work and INT3472:0c: cannot find GPIO chip INTC1007:00, deferring error messages are no longer present. However, the camera pin still does not work.

I assume it may work if the handshake pin is asserted. In the discussion of the patch, @hao-yao commented "It starts outputing MIPI packages when we pull handshake pins high". Unfortunately is is not clear which pin the handshake pin is. The gpioinfo tool shows 128 pins for the INTC1007:00 chip, two of them are "privacy-led" outputs, all others are unnamed inputs.

@hao-yao can you help with this?

@mluessi
Copy link

mluessi commented Dec 7, 2024

Also, when also loading the i2c-usbio module, I get an error saying that it can't detect the sensor:

[root@gms martin]# modprobe i2c-usbio
[root@gms martin]# dmesg
..
 4449.783916] usbio-i2c usbio-i2c.3.auto: hid INTC1008 uid 0 new uid1
[ 4449.788497] ov08x40 i2c-OVTI08F4:00: chip id mismatch: 560858!=0
[ 4449.788504] ov08x40 i2c-OVTI08F4:00: failed to find sensor: -6

@jwrdegoede
Copy link
Contributor Author

@mluessi

Hi, nice to meet you. I saw your reply about this to my blogpost https://hansdegoede.dreamwidth.org/28841.html but lets continue discussing this here.

I should receive a X1 Carbon with the Lattice MIPI aggregator chip soon. Once I have it I'll try to get things to work with the out of tree drivers for this and then once I have things working we can see from there. Note I'm taking 2 weeks of with Christmas so I likely won't get around to this until after January 1st.

Thank you for your offer to help with this, once we have things working I believe that the code from https://github.com/intel/usbio-drivers will need a bunch of work before it will be submitted upstream. Like moving it from being a MFD driver to the new auxiliary bus kernel framework similar to what was done for the LJCA driver before it was mainlined.

@hao-yao do you know what Intel's plans are for upstreaming the usb-io drivers ? I want to avoid us doing double work where e.g. Intel is already working on moving the code to the auxiliary bus infrastructure and we end up doing the same work a second time.

@hao-yao
Copy link
Contributor

hao-yao commented Dec 12, 2024

Hans @jwrdegoede ,

@hao-yao do you know what Intel's plans are for upstreaming the usb-io drivers ? I want to avoid us doing double work where e.g. Intel is already working on moving the code to the auxiliary bus infrastructure and we end up doing the same work a second time.

As far as I know, the owner team of usbio-drivers is working on the upstreaming in progress, however I don't know the detail plan or schedule. I think you can check with the usbio-drivers maintainer for the detail.

@mluessi
Copy link

mluessi commented Dec 12, 2024

@jwrdegoede Nice to meet you too. Sounds good, I will be happy to help early next year.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jwrdegoede @mluessi @moozhub @hao-yao @n3thshan