UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 146-147: invalid continuation byte

[Delido]

I found a bug in megaclisas-status. After some debugging i found the issue.

OS: Debian Bullseye
APPS:

• megaclisas-status 0.18+Debian.11.bullseye
• megacli 8.07.14-3+Debian.11.bullseye

Bug:

/usr/sbin/megaclisas-status
-- Controller information --
-- ID | H/W Model          | RAM    | Temp | BBU    | Firmware     
c0    | PERC H730P Adapter | 2048MB | 60C  | Good   | FW: 25.5.8.0001 

-- Array information --
Traceback (most recent call last):
  File "/usr/sbin/megaclisas-status", line 697, in <module>
    output = getOutput(cmd)
  File "/usr/sbin/megaclisas-status", line 148, in getOutput
    for line in output:
  File "/usr/lib/python3.9/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 146-147: invalid continuation byte

after some debugging the problem is the Name ob the Virtual Drive. megaclisas-status execute the following:

usr/sbin/megacli -LDInfo -l0 -a0 -NoLog       
                                     

Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :Virtual Disk ??
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 111.25 GB
Sector Size         : 512
Is VD emulated      : Yes
Mirror Data         : 111.25 GB
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 2
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
Default Power Savings Policy: Controller Defined
Current Power Savings Policy: None
Can spin up in 1 minute: No
LD has drives that support T10 power conditions: No
LD's IO profile supports MAX power savings with cached writes: No
Bad Blocks Exist: No
Is VD Cached: No

if i change all megacli commands with "xxxx | grep -v Name", megaclisas-status is working.

I dont know why my VD Name is "Virtual Disk ??" also i dont know to change this.

Regards
Sebastian

[eLvErDe]

It seems your megacli outputs some characters in a non-utf8 charset, that's very weird.
Can you send the output to a file and run "file" on it to see what charset it detects ?

[Delido]

/usr/sbin/megacli -LDInfo -l0 -a0 -NoLog > /tmp/mega
❯ file -0 /tmp/mega
/tmp/mega: ISO-8859 text, with CR, LF line terminators

on another working Server:

file -0 /tmp/mega
/tmp/mega: ASCII text, with CR, LF line terminators

[eLvErDe]

Good 'ol iso8859 :D

I'll send you a patch to try tomorrow

[Delido]

why is the output iso8859? all locale Settings between the Servers are the same and UTF-8 are default. Can / must i change something?

[krietvel]

We are running a bunch of old PERC 5/i adapters, for which MegaCLI 8.07.14 gives non-UTF8 and non-ASCII output (classified as "data" by the file command). I ended up using the patch below to get megaclisas-status to work with Python 3.9 on bullseye. It replaces popen with subprocess.Popen to be able to ignore the conversion errors ...

diff a/wrapper-scripts/megaclisas-status b/wrapper-scripts/megaclisas-status
--- a/wrapper-scripts/megaclisas-status
+++ b/wrapper-scripts/megaclisas-status
@@ -11,6 +11,7 @@ import sys
 import pdb
 import inspect
 import argparse
+from subprocess import Popen, PIPE, STDOUT
 
 if sys.platform == "win32":
     import ctypes
@@ -144,8 +145,10 @@ def getOutput(cmd):
         lines = Outputs[cmd]
     else:
         dbgprint("Not a Cached value: " + str(cmd))
-        output = os.popen(cmd)
-        for line in output:
+        process = Popen(cmd, shell=True, stdout=PIPE, stderr=STDOUT, text=True, errors='ignore')
+        process.wait()
+        output = process.stdout.read()
+        for line in output.split("\n"):
             if not re.match(r"^$", line.strip()):
                 lines.append(line.strip())
         Outputs[cmd] = lines