Hardware Messages Monitoring

The driver communicates with the device using Ethernet packets called Ethernet Management Datagrams (EMADs). As of Linux 4.8, it has become possible to monitor all messages sent between the driver and the device using a kernel tracepoint called devlink:devlink_hwmsg.

Using a simple Python script called devlink-hwmsg.py it is possible to hook on the tracepoint and convert the generated events into a PCAP file. The script may be found in the following repository:

switch$ git clone https://github.com/jpirko/hwmsg_tracing.git

For an easier view, Wireshark may be used:

laptop$ ssh switch sudo ~/hwmsg_tracing/devlink-hwmsg.py | wireshark -k -i -

If devlink-hwmsg.py complains about a missing tracepoint object, then make sure to update the python-perf package to at least version 4.8. An adequate version is currently available via Koji:

switch$ dnf install https://kojipkgs.fedoraproject.org//packages/kernel/4.8.0/0.rc7.git4.1.fc25/x86_64/python-perf-4.8.0-0.rc7.git4.1.fc25.x86_64.rpm -y

Also, make sure that devlink module is loaded before running the script.

Processing the Message Stream

Hardware message packets contain data in a custom TLV-based format. The tool bwz.py can be used to filter this data based on values inside individual TLV fields, and to slice the data (i.e. omit some TLV records or remove the TLV headers altogether).

bwz includes a simple language for expressing filtering (-f) and slicing (-s). See --help for details of operation. E.g. to include just the payload of messages that the kernel sends out, you could pipe devlink-hwmsg.py output as follows:

sw:~$ devlink-hwmsg.py | bwz.py -f outgoing -s buf | ...

It's possible to query substrings of individual values using Python array subscript syntax. E.g. if you know the message buffer is in Mellanox EMAD format, you can do the following to select all messages related to the RAUHT register (whose ID is 0x8014):

sw:~$ ... | bwz.py -f 'buf[0x14:0x16] == "\x80\x14"' | ...

One can combine the conditions using & and | for "and" and "or". Make sure you parenthesize the combined expressions, & and | don't have the right precedence in Python. When in doubt, use the command line argument --show to have bwz dump how it understands the task.

sw:~$ bwz.py --show -f '(driver == "mlxsw_spectrum") & outgoing & \
                        (buf[0x14:0x16] == "\x80\x14")'
filter=(((driver == 'mlxsw_spectrum') & (~incoming)) & ((buf[slice(20, 22, None)]) == '\x80\x14'))
slice=tlv(bus, dev, driver, incoming, type, buf)

bwz depends on pcapy, a Python package for processing pcap files.