Error redirection a usb device to a qemu virtual machine

[kovalev0]

The error is reproduced for both the master and dev branches, and for the module from the dev branch it is not possible to run the printer and keyboard example application, but a proxy device is successfully created using the usb_proxy application.

master:
$ sudo ./keyboard
...
ioctl(USB_RAW_IOCTL_EP_WRITE): Cannot send after transport endpoint shutdown

$ sudo ./printer
...
ioctl(USB_RAW_IOCTL_EP_READ): Cannot send after transport endpoint shutdown
ioctl(USB_RAW_IOCTL_EP_WRITE): Cannot send after transport endpoint shutdown

usbredir (example printer):
$ sudo usbredirect --device 0525:a4a8 --as 127.0.0.1:4000 --verbose 5
(usbredirect:12580): usbredirect-ERROR **: 20:22:46.937: usbredirhost: error resetting device: LIBUSB_ERROR_NOT_FOUND
Ловушка трассировки/останова

=====================================================================

dev:
$ sudo ./keyboard

event: connect, length: 0

ep #0:
name: ep1in-bulk
addr: 1
type: ___ blk ___
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #1:
name: ep2out-bulk
addr: 2
type: ___ blk ___
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #2:
name: ep5in-int
addr: 5
type: ___ ___ int
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #3:
name: ep6in-bulk
addr: 6
type: ___ blk ___
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #4:
name: ep7out-bulk
addr: 7
type: ___ blk ___
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #5:
name: ep10in-int
addr: 10
type: ___ ___ int
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #6:
name: ep11in-bulk
addr: 11
type: ___ blk ___
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #7:
name: ep12out-bulk
addr: 12
type: ___ blk ___
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #8:
name: ep15in-int
addr: 15
type: ___ ___ int
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #9:
name: ep1out-bulk
addr: 1
type: ___ blk ___
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #10:
name: ep2in-bulk
addr: 2
type: ___ blk ___
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #11:
name: ep-aout
addr: 255
type: ___ blk int
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #12:
name: ep-bin
addr: 255
type: ___ blk int
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #13:
name: ep-cout
addr: 255
type: ___ blk int
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #14:
name: ep-dout
addr: 255
type: ___ blk int
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #15:
name: ep-ein
addr: 255
type: ___ blk int
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #16:
name: ep-fout
addr: 255
type: ___ blk int
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #17:
name: ep-gin
addr: 255
type: ___ blk int
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #18:
name: ep-hout
addr: 255
type: ___ blk int
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #19:
name: ep-iout
addr: 255
type: ___ blk int
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #20:
name: ep-jin
addr: 255
type: ___ blk int
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #21:
name: ep-kout
addr: 255
type: ___ blk int
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #22:
name: ep-lin
addr: 255
type: ___ blk int
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #23:
name: ep-mout
addr: 255
type: ___ blk int
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
int_in: addr = 5

event: 6 (unknown), length: 0

fail: unknown event

$ sudo ./printer

event: connect, length: 0

ep #0:
name: ep1in-bulk
addr: 1
type: ___ blk ___
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #1:
name: ep2out-bulk
addr: 2
type: ___ blk ___
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #2:
name: ep5in-int
addr: 5
type: ___ ___ int
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #3:
name: ep6in-bulk
addr: 6
type: ___ blk ___
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #4:
name: ep7out-bulk
addr: 7
type: ___ blk ___
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #5:
name: ep10in-int
addr: 10
type: ___ ___ int
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #6:
name: ep11in-bulk
addr: 11
type: ___ blk ___
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #7:
name: ep12out-bulk
addr: 12
type: ___ blk ___
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #8:
name: ep15in-int
addr: 15
type: ___ ___ int
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #9:
name: ep1out-bulk
addr: 1
type: ___ blk ___
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #10:
name: ep2in-bulk
addr: 2
type: ___ blk ___
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #11:
name: ep-aout
addr: 255
type: ___ blk int
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #12:
name: ep-bin
addr: 255
type: ___ blk int
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #13:
name: ep-cout
addr: 255
type: ___ blk int
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #14:
name: ep-dout
addr: 255
type: ___ blk int
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #15:
name: ep-ein
addr: 255
type: ___ blk int
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #16:
name: ep-fout
addr: 255
type: ___ blk int
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #17:
name: ep-gin
addr: 255
type: ___ blk int
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #18:
name: ep-hout
addr: 255
type: ___ blk int
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #19:
name: ep-iout
addr: 255
type: ___ blk int
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #20:
name: ep-jin
addr: 255
type: ___ blk int
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #21:
name: ep-kout
addr: 255
type: ___ blk int
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
ep #22:
name: ep-lin
addr: 255
type: ___ blk int
dir : in ___
maxpacket_limit: 65535
max_streams: 16
ep #23:
name: ep-mout
addr: 255
type: ___ blk int
dir : ___ out
maxpacket_limit: 65535
max_streams: 16
bulk_out: addr = 2
bulk_in: addr = 1

event: 6 (unknown), length: 0

fail: unknown event

usb-proxy:

$ lsusb
...
Bus 002 Device 004: ID 0bda:0129 Realtek Semiconductor Corp. RTS5129 Card Reader Controller
...
$ sudo ./usb-proxy --vendor_id=0bda --product_id=0129
...
ep0: transferred 0 bytes (out)
EP1(bulk_out): read 16 bytes from host
EP82(bulk_in): wrote 4 bytes to host
EP1(bulk_out): read 16 bytes from host
EP82(bulk_in): wrote 4 bytes to host
EP1(bulk_out): read 16 bytes from host
EP82(bulk_in): wrote 4 bytes to host
EP1(bulk_out): read 24 bytes from host
EP1(bulk_out): read 32 bytes from host
EP1(bulk_out): read 12 bytes from host
EP1(bulk_out): read 24 bytes from host

new proxy-usb device:
$ lsusb
Bus 001 Device 003: ID 0bda:0129 Realtek Semiconductor Corp. RTS5129 Card Reader Controller
...

redir with virt-manager:

(continue) $ sudo ./usb-proxy --vendor_id=0bda --product_id=0129
...
ioctl(USB_RAW_IOCTL_EP_READ): Cannot send after transport endpoint shutdown
ioctl(USB_RAW_IOCTL_EP_WRITE): Cannot send after transport endpoint shutdown

[xairy]

Thanks for the report!

Re fail: unknown event: you need to refetch the dev branch, I fixed this issue recently.

Looks like the issue with running the examples is that usbredirect resets the device before connecting it to QEMU's guest, and the reset event (6) is not handled by the examples. See AristoChen/usb-proxy#9 (comment) for a related discussion.

I'm not sure whether it makes to add reset handling to existing examples or add another one to demonstrate proper reset handling.

usb-proxy also doesn't know how to handle the reset event yet. I was planning to try addressing this once we get AristoChen/usb-proxy#9 resolved.

For now, could you share the command you use to run QEMU to enable USB redirection?

[kovalev0]

For now, could you share the command you use to run QEMU to enable USB redirection?

#!/bin/bash

qemu-system-x86_64 \
-hda altlinux.qcow2 \
-m 4G \
-enable-kvm \
-cpu host,nx \
-monitor stdio \
-nic user,hostfwd=tcp::8888-:22 \
-device ich9-usb-ehci1,id=ehci,addr=1d.7,multifunction=on \
-device ich9-usb-uhci1,id=uhci-1,addr=1d.0,multifunction=on,masterbus=ehci.0,firstport=0 \
-device ich9-usb-uhci2,id=uhci-2,addr=1d.1,multifunction=on,masterbus=ehci.0,firstport=2 \
-device ich9-usb-uhci3,id=uhci-3,addr=1d.2,multifunction=on,masterbus=ehci.0,firstport=4 \
-chardev socket,id=usbredirchardev1,port=4000,host=127.0.0.1 \
-device usb-redir,chardev=usbredirchardev1,id=usbredirdev1,bus=ehci.0,debug=4

The process of working in steps is described in this article:

https://www.altlinux.org/%D0%98%D1%81%D1%81%D0%BB%D0%B5%D0%B4%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5_%D0%B2%D0%BE%D0%B7%D0%BC%D0%BE%D0%B6%D0%BD%D0%BE%D1%81%D1%82%D0%B8_%D0%BF%D1%80%D0%B8%D0%BC%D0%B5%D0%BD%D0%B5%D0%BD%D0%B8%D1%8F_raw-gadget_%D0%BA%D0%B0%D0%BA_%D0%B8%D1%81%D1%82%D0%BE%D1%87%D0%BD%D0%B8%D0%BA%D0%B0_USB-%D1%82%D1%80%D0%B0%D1%84%D0%B8%D0%BA%D0%B0_%D0%B4%D0%BB%D1%8F_%D1%82%D0%B5%D1%81%D1%82%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D1%8F_usbredir

[xairy]

I pushed a few changes to the dev branch that make keyboard.c handle the reset event. I haven't tested them with QEMU usbredir though, only with the usbreset tool. Please check if it works for you now.

Re the link you shared: interesting! Could you explain what is the purpose of redirecting a virtual USB device emulated with Raw Gadget on host into QEMU guest? You could emulate a device within the QEMU guest directly. Or do you want to use a Raw Gadget-based proxy to MitM a real device while redirecting its traffic to the QEMU guest?

[kovalev0]

I pushed a few changes to the dev branch that make keyboard.c handle the reset event. I haven't tested them with QEMU usbredir though, only with the usbreset tool. Please check if it works for you now.

The module and the keyboard example (sleep(1) -> sleep(10) ) are compiled from the dev branch.
The redirected device falls off when the virtual machine desktop is loaded or before the user is invited to the terminal in multi-user mode, however, if you go to the grub menu, the 'x' symbol is printed.
Moreover, such a run does not always work the same way and falls with different errors. I attach one of the falls to the picture and log files:

device:
keyboard_0.txt

$ sudo ./keyboard
...
ep_int_in: key up: 8
ep_int_in: key down: 8
ep_int_in: key up: 8
event: reset
ep0: stopping ep_int_in thread
ep_int_in: thread interrupted, exiting
ep0: stopped ep_int_in thread
event: 5 (unknown), length: 0
ep0: ignoring unknown event
event: control, length: 8
  bRequestType: 0x80 (IN), bRequest: 0x6, wValue: 0x100, wIndex: 0x0, wLength: 64
  type = USB_TYPE_STANDARD
  req = USB_REQ_GET_DESCRIPTOR
  desc = USB_DT_DEVICE
ioctl(USB_RAW_IOCTL_EP0_WRITE): Value too large for defined data type

or

...
ep_int_in: key down: 8
ep_int_in: key up: 8
event: reset
ep0: stopping ep_int_in thread
ep_int_in: thread interrupted, exiting
ep0: stopped ep_int_in thread
ioctl(USB_RAW_IOCTL_EVENT_FETCH): Invalid argument

or

...
event: reset
ep0: stopping ep_int_in thread
usb_raw_ep_write_may_fail(): Cannot send after transport endpoint shutdown

usbredir:

$ sudo /usr/bin/usbredirect --device 046d:c312 --as 127.0.0.1:4000

(usbredirect:4336): usbredirect-CRITICAL **: 17:06:55.685: connection_handle_io_cb: Failed to read guest

qemu:
qemu_0.txt

...
qemu: usb-redir: reset device

qemu: usb-redir: ctrl-out type 0x80 req 0x6 val 0x100 index 0 len 64 id 17330560

qemu: usb-redir: adding packet id 17330560 to cancelled queue

qemu: usb-redir: reset device

qemu: usb-redir: ctrl-out type 0x80 req 0x6 val 0x100 index 0 len 64 id 17330272

qemu: usb-redir: detaching device

qemu: usb-redir: adding packet id 17330272 to cancelled queue

qemu: usb-redir: removing 2 packet-ids from cancelled queue

qemu: usb-redir: removing 0 packet-ids from already-in-flight queue

qemu: usb-redir: chardev close

qemu: usb-redir: removing 0 packet-ids from cancelled queue

qemu: usb-redir: removing 0 packet-ids from already-in-flight queue

qemu: usb-redir: destroying usbredirparser
...

usbdump (usage usbmon):
usbdump_0.txt

$ sudo ./usbdump -d 046d:c312
  0.000000 <--5 intr
  1.828025 <--0 ctrl [0100 0000] 18: 1201 0002 0000 0040 6d04 12c3 0000 0001  0201
  1.832031 <--0 ctrl [0200 0000] 34: 0902 2200 0101 03c0 3209 0400 0001 0301  0104 0921 1001 0001 2241 0007 0585 0308  0005
  1.836036 <--0 ctrl [0302 0409] 4: 0403 7800
  1.836061 -->0 ctrl [0001 0000]
 18.054838 <--0 ctrl [0100 0000] 8: 1201 0002 0000 0040
 18.059836 <--0 ctrl [0200 0000] 9: 0902 2200 0101 03c0 32
 18.064836 <--0 ctrl [0200 0000] 34: 0902 2200 0101 03c0 3209 0400 0001 0301  0104 0921 1001 0001 2241 0007 0585 0308  0005
 18.067802 -->0 ctrl [0000 0000]
 18.071983 -->0 ctrl [0800 0000]
 18.079837 <--5 intr 8: 0000 1b00 0000 0000
 18.079839 <--5 intr 8: 0000 0000 0000 0000
 28.081838 <--5 intr 8: 0000 1b00 0000 0000
 28.081842 <--5 intr 8: 0000 0000 0000 0000

Re the link you shared: interesting! Could you explain what is the purpose of redirecting a virtual USB device emulated with Raw Gadget on host into QEMU guest?

The initial goal is fuzzing the usbredir library.

You could emulate a device within the QEMU guest directly. Or do you want to use a Raw Gadget-based proxy to MitM a real device while redirecting its traffic to the QEMU guest?

In the process of exploring the capabilities of raw_gadget and usb_proxy, there was also such an idea that at the beginning of the session, use redirection of a regular device and use a standard handler, and then send a packet with mutated data and monitor the reaction of qemu entities.

[kovalev0]

Compared the logs with redirecting a real keyboard Bus 002 Device 008: ID 046d:c34b Logitech, Inc. USB Keyboard,
attach:
usbdump_real_keyboard_0.txt
и
qemu_real_keyboard_0.txt

At the time of the user's invitation, usbredir tries to execute qemu: usb-redir: set address 2, to which the keyboard emulator does not respond.

real:

...
qemu: usb-redir: reset device

qemu: usb-redir: reset device

qemu: usb-redir: ctrl-out type 0x80 req 0x6 val 0x100 index 0 len 64 id 17317952

qemu: usb-redir: ctrl-in status 0 len 18 id 17317952

qemu: usb-redir: reset device

qemu: usb-redir: set address 2

qemu: usb-redir: ctrl-out type 0x80 req 0x6 val 0x100 index 0 len 18 id 17318144

qemu: usb-redir: ctrl-in status 0 len 18 id 17318144

qemu: usb-redir: ctrl-out type 0x80 req 0x6 val 0x200 index 0 len 9 id 17318336

qemu: usb-redir: ctrl-in status 0 len 9 id 17318336

qemu: usb-redir: Removed remote wake 046D:C34B
...

emulator:

...
qemu: usb-redir: reset device

qemu: usb-redir: reset device

qemu: usb-redir: ctrl-out type 0x80 req 0x6 val 0x100 index 0 len 64 id 17330560

qemu: usb-redir: adding packet id 17330560 to cancelled queue

qemu: usb-redir: reset device

qemu: usb-redir: ctrl-out type 0x80 req 0x6 val 0x100 index 0 len 64 id 17330272

qemu: usb-redir: detaching device

qemu: usb-redir: adding packet id 17330272 to cancelled queue

qemu: usb-redir: removing 2 packet-ids from cancelled queue

qemu: usb-redir: removing 0 packet-ids from already-in-flight queue

qemu: usb-redir: chardev close
...

[xairy]

Pushed another change to dev, could you test?

[kovalev0]

Pushed another change to dev, could you test?

Thanks! Now keyboard redirection works fine both with qemu and via virt-manager.

[xairy]

Awesome! So Raw Gadget works with usbredir, the only thing is that the emulation code needs to handle the reset event properly.

Let's keep this issue open as a reminder for me to:

1. Upstream the Raw Gadget patches from the dev branch;
2. Add reset handling to usb-proxy.

[AristoChen]

Just noticed the discussion here :)

Probably not related to the issue, but I noticed that you are discussing about the reset. From my previous experience(I can't recall everything now, it was like 3-4 years ago), some USB devices behave very weird if they receive the reset event for the second time.

I was using https://github.com/usb-tools/USBProxy-legacy to do proxy, and I called libusb_reset_device() before expose the USB device to Host, but then Host send another reset event(which is a normal behaviour if the USB device is plugged to Host directly) to the device, which cause the device hang and not functioning

In the end, I added a config/variable to decide whether I shoud call libusb_reset_device() before expose the device to Host or not

[xairy]

@AristoChen Yeah, I wound't be surprised if some USB devices break on multiple resets :)

Most of them should handle them though. So I would say resetting the device if the host asks for it should be the default behavior of a USB proxy.

But yeah, this is not related to handling the reset event in Raw Gadget.

[xairy]

Sent the patches upstream and implemented usb-proxy support for resent handling (didn't send a PR yet).

[xairy]

Raw Gadget patches are now in usb-next, so I includes the changes into the master branch and updated the keyboard example to handle reset.

Closing this issue as resolved.

Handing of reset in USB proxy is tracked in AristoChen/usb-proxy#9.