Feature Proposal: Sinden Lightgun Support

[NolanNicholson]

Hello!

The Sinden Lightgun is a USB lightgun controller that works using a video camera in the barrel, instead of CRT beam tracking or infrared sensors. This means it works with modern flat-panel displays, requires no additional sensors or other hardware to be fixed to the display, and tracks position accurately with no drift over time.

The gun requires two unusual things: a special driver, and a screen border.

1. Driver: The Sinden hardware presents raw video data from the camera. To convert from video feed to aim data, special driver software is required. The driver is computationally intensive, and is proprietary.

2. Border: In order for the edge detection to work reliably (especially on dark screens), a thin white border must be displayed around the image at all times that the lightgun is in use.

Previously, a couple of users have been able to use the Sinden Lightgun on the MiSTer by using external computer hardware (e.g., a Raspberry Pi) to run the driver software, plus external video processing hardware (e.g., a RetroTINK 5X) to add a border to the image.

Recently, I worked with with Andy Sinden and wiggy808 to accomplish both of these onboard the MiSTer itself - meaning the lightgun can work with no other hardware than the DE-10, the lightgun, and an external display. I'd be interested in submitting the necessary changes as pull requests, but I'd like to describe them as proposals first rather than just dumping them on your doorstep.

For the border, I propose adding a vscale_frame parameter to MiSTer.ini, defining the size in pixels of a white border to be drawn around the image. This parameter would be passed through to the FPGA framework, particularly sys_top and ascal.

Since the driver is proprietary, the Sinden folks would distribute binaries and support scripts themselves. They have some kinks to work out: the driver works well on cores like NES and SNES, but its heavy calculations currently interfere somewhat with cores like PSX (where the main binary needs CPU cycles to perform disc reads.) To support the driver, the uvcvideo module would have to be enabled in the Linux kernel, and the lightgun PID/VID would need the usual quirks added in input.cpp.

I recognize that this is a bit much to support just one product. It might be better to just go halfway - like maybe add the border, but still require a Raspberry Pi for the position calculations. I'd be happy to hear your opinion.

[sorgelig]

I don't like this idea because of:

1. it requires special frame embedded in video.
2. it requires special and proprietary driver.
3. it loads CPU.

There is a much easier and already working solution - Wiimote. Its accuracy is enough and on-par with retro guns.
And it's very cheap.

[MrLightgun]

Hi Sorgelig

This is Andy. Thank you for your work with the MiSTer, I appreciate it's not easy being the custodian of a project like this.

The Sinden driver for the MiSTer has been specially coded to not have any CPU impact and was quite a lengthly implementation to remove any dependancies so it was just a single compiled binary. It only processes video frames when you pull the trigger, so the CPU usage is tiny and even when you pull the trigger it only needs to process a single frame which it does in much less than one frame's time. There is also minimal memory requirements.

So when using the custom driver we didn't see any performance impact on any of the cores, we did with the original driver which is why it needed a substantial re-write.

What we were hoping you would consider rather than proper mainline support and implementation of borders etc, would you consider the OS change to allow the driver to work so we can at least experiment on the proper MiSTer build. We don't want to encourage users to have any non-mainline custom OS components. We only need Linux UVC video to be enabled which Nolan says is very lightweight, along with a bug fix in the UVC driver so that 2 video devices can work through the same USB controller (via a hub) and the input.cpp changes so that everything works properly with the inputs.

In return we will make sure that any usage is clearly considered outside of mainline support with clear warnings for anyone downloading the software that it is not supported by the MiSTer community and is experimental. I would also consider it reasonable if you felt we reneged on this that the OS support was removed. We can support users on our Discord and if there was a thread on the main MiSTer forum then I would happily answer questions there whilst also being patient with users and re-inforcing that it is experimental and not part of mainline MiSTer.

Whilst the lightgun project is commercial I care about our community of existing customers and continue to develop software to improve their experience without any commercial payback and I probably get at least one email a day asking for MiSTer support. The lightgun is expensive compared to a wiiremote however this isn't really about new users it's about the 10,000+ community that already have a lightgun and there is a large crossover with the MiSTer community. The experimental driver works really well and we would really like to give the community a chance to experiment with it.

Thank you for your time and consideration.

Andy

[sorgelig]

There is a reasonable amount of change to support a specific device. It's also related to amount of users using such device.

For example, take Wiimote. What it needs? Some small proper driver emulating mouse or analog joystick and that's all. It doesn't need to change the framework or embed quite invasive code into FPGA part of framework where some cores already close to resource limits. And i believe amount of Wiimote users on MiSTer several times larger than Sinden's.
For Sinden you request enormous change in both HPS and FPGA to please just several users. It's not a reasonable change.
Why don't you want to update your device with ability to track for example 4 LEDs placed on corners of TV? Today it's very easy to make such source of lights with very specific parameters for easy tracking. Why you use such invasive system? Nintendo could track a stripe of leds back in 2007. Today embedded MCUs are much more powerful and can do it on a completely new level without requesting processing power from emulators.

I'm not even talking about aesthetic aspect where having white frame is quite distracting. And it even burns OLED displays.

[MrLightgun]

Thank you for the reply, all we would like to do is avoid users needing a non-standard Linux. If the UVC changes and input.cpp changes were added then people can experiment with Sinden Lightgun and still update their MiSTer system with no issues and not need to wait for an updated version of the same OS with the changes. People with outdated branched environments are a huge pain and clog up forums and other supportive channels.

Anything else can be managed separately and away from mainline. If you could kindly allow those small changes that won't create any other negativity for other users then I promise not to ask or even encourage the community to ask for any more level of mainline integration, no borders, no driver being included, no scripts etc etc.

Wii remotes are great and we support people using whatever they can as a lightgun, so all emulator development I have done personally or been involved in, have always encouraged making sure the support is there for all lightguns so people have a choice.

If we did a merge request for just the UVC support, UVC bug fix and input.cpp would you kindly consider it please.

[sorgelig]

MiSTer CPU is weak, USB host device is very buggy and barely can handle several USB devices. I can't even think about video capturing driver in such scenario. Even if you don't process the video whole time, the webcam still sends the video over USB whole time. You know how many people consider about input lag? And you offer to add a constant heavy traffic over USB.
May be try some offloading on RPi-like device, then convert it into mouse touchscreen coordinates and send it to MiSTer as an input device. MiSTer already supports touchscreen (RasPad3) as a gun, so can be easily adapted to another touchscreen device.

[MrLightgun]

There is no scenario where it would affect anyone who isn't using a Sinden Lightgun at the time. So it's not going to affect someone's response time while trying to speed run Super Mario Bros. You connect the lightgun to play lightgun games and disconnect when you have finished. We have a working driver that works really well and doesn't affect performance while being used on any of the cores. The video feed is compressed and comfortably handled by USB and the CPU is barely touched during use so it's not like we are going to damage people's MiSTer. I have spent a long time on the driver to make sure it is fit for purpose.

We are going to be releasing a proof of concept for people to experiment with but couldn't we do that without forcing them to use a modified OS also? If the UVC support was there already on the Linux and we released a driver, I wouldn't expect that you would go back and disable UVC support in the OS deliberately to cause us a blocker?

Not asking for official support but couldn't you kindly consider just removing the blocker for us? it's not going to do any harm or affect non Sinden Lightgun users. I think we have about 1000 people who would be interested in this. Like I said previously the RetroGaming community has a lot of crossover between cool devices.

Thank you for your time and consideration.

[sorgelig]

I don't block anything. Kernel source is available. You may compile a loadable kernel driver for your experiments.

[MrLightgun]

OK, thank you for your replies, we will proceed down this route.

[birdybro]

Hey @MrLightgun if you provide a way for me (and your customers of course) to test like with a script to either load the module or download an alternate linux image to the MiSTer, I'm sure real-world testing could be done. I've heard from multiple users on our discord who have your gun and they like it and they have been hoping for MiSTer support for awhile. I don't really do much software emulation these days but if you published some way to get it working for your current users that would make it worth it for me to buy one too and test (as well as others).

[MrLightgun]

Hey @MrLightgun if you provide a way for me (and your customers of course) to test like with a script to either load the module or download an alternate linux image to the MiSTer, I'm sure real-world testing could be done. I've heard from multiple users on our discord who have your gun and they like it and they have been hoping for MiSTer support for awhile. I don't really do much software emulation these days but if you published some way to get it working for your current users that would make it worth it for me to buy one too and test (as well as others).

Thank you for the encouragement, published how to guide coming very soon :-)

[alucryd]

Music to my ears, I've had 2 sindens attached to my arcade cabinet collecting dust for 2 years waiting for this moment <3

[alucryd]

@MrLightgun Any chance you could share the required changes so we can build the necessary parts ourselves ? I found that the current prebuilts don't work on the latest mr fusion.