Add test case for `math.mode` #1009

Job	Run time
standard-build (--enable-debug --enable-cuckoo --enable-magic --enable-pb-tests --enable-address-...	2m 35s
bazel-build	1m 21s
mingw-build (--host=x86_64-w64-mingw32)	1m 33s
standard-build (--enable-debug --enable-profiling --enable-cuckoo --enable-magic --enable-pb-test...	2m 45s
mingw-build (--host=i686-w64-mingw32)	1m 15s
bigendian	11m 33s
macosx-build	1m 51s
	22m 53s

Provide feedback