Use unicodedata2 if available

[jayvdb]

https://pypi.org/project/unicodedata2/ is usually more up to date than even the latest cpython release.

iirc, using it is simply a matter of checking if unicodedata2 data version is higher than unicodedata, and if so sys.modules['unicodedata'] = unicodedata2 . Need to check that though

[Ousret]

Hi,

Is this modification safe ? Can we test it first ? Do we gain anything ? What do they update compare to Python embed ?

[Ousret]

What do you think of this lib ? Are there any reason to use it compared to others (chardet and cchardet)?

[jayvdb]

Yes you can test it. To compare with Python 3.7, a feature in unicode 12 needs to be found, because Python 3.7 iirc still has unicode 11. The most well known differences are the emoji list https://emojipedia.org/unicode-12.0/ , but there are also 4 new scripts. As unicodedata only provides the characters, and doesnt provide more "encodings", using a more recent unicodedata is mostly useful for avoiding decode errors when newer characters are encountered in a stream being processed.

The case I am most interested in at the moment is streams that have 'magic' or 'BOM' which distinctly indicate their encoding. Surprisingly most of the libraries tested so far do not handle the known boms listed on Wikipedia. timrburnham/bom_open#13 (comment) I havent run this library against my bom tests yet to see how it compares with others.

I did look through the existing test suite here, and the code, and it looks quite good, and probably better for Asian languages than chardet, but cchardet is also quite good at Asian languages.

[Ousret]

        self.assertEqual(
            CnM.from_bytes(
                '我没有埋怨，磋砣的只是一些时间。'.encode('gb18030')
            ).best().first().encoding,
            'gb18030'
        )

        self.assertEqual(
            CnM.from_bytes(
                (u'\uFEFF' + '我没有埋怨，磋砣的只是一些时间。').encode('gb18030')
            ).best().first().encoding,
            'gb18030'
        )

        self.assertEqual(
            CnM.from_bytes(
                '我没有埋怨，磋砣的只是一些时间。'.encode('utf_8')
            ).best().first().encoding,
            'utf_8'
        )

        self.assertEqual(
            CnM.from_bytes(
               b'\xef\xbb\xbf' + '我没有埋怨，磋砣的只是一些时间。'.encode('utf_8')
            ).best().first().encoding,
            'utf_8'
        )

        self.assertEqual(
            CnM.from_bytes(
                'Bсеки човек има право на образование. Oбразованието трябва да бъде безплатно, '
                'поне що се отнася до началното и основното образование.'.encode('utf_8')
            ).best().first().encoding,
            'utf_8'
        )

        self.assertEqual(
            CnM.from_bytes(
                'Bсеки човек има право на образование.'.encode(
                    'utf_8')
            ).best().first().encoding,
            'utf_8'
        )

charset_normalizer 0.2.2 comply with those tests 🎉

[jayvdb]

And utf-7 ? :P

[Ousret]

I'm going to fix utf_7 support in next minor. (was always considered as ascii)

[Ousret]

@jayvdb did you have time to try the newest version ? 🎉

[Ousret]

Set as optional feature. Op as of release 1.3.0

pip install charset-normalizer[UnicodeDataBackport]