Segmentation fault (looks like stack exhaustion)

[raincoats]

Hi, I'm getting segfaults when running any program with this library LD_PRELOADed.

I first compiled it normally, then when it didn't work, I unset all environment variables to be sure it wasn't some strange setting on my machine, but that didn't help at all.

Additionally, make runtests seems to work but make rundtests doesn't.

Running LD_PRELOAD=$(pwd)/libdheap.so LIBDHEAP_DEBUG=1 ./test, and then using gdb to analyse the coredump shows me that there was 63589 stack frames, basically just looping these:

#63579 0x00007fcf386f14e8 in vdprintf () from /usr/lib/libc.so.6
#63580 0x00007fcf386d14b4 in dprintf () from /usr/lib/libc.so.6
#63581 0x00007fcf387a6bcb in _dl_signal_error () from /usr/lib/libc.so.6
#63582 0x00007fcf38c42495 in _dl_signal_cerror () from /lib64/ld-linux-x86-64.so.2
#63583 0x00007fcf38c34218 in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#63584 0x00007fcf38c38b53 in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#63585 0x00007fcf38c400c7 in _dl_runtime_resolve_sse_vex () from /lib64/ld-linux-x86-64.so.2
#63586 0x00007fcf38a26afb in guard_chunk (ch=0x1286040) at ./src/canary.c:67
#63587 0x00007fcf38a278bd in malloc (size=0x1a) at ./src/libdheap.c:72

Shell output

Compiling and running the tests
Output of running the command (with a reduced stack size (256), otherwise it's huge)
Output of bt full in gdb (again with reduced stack size)

My machine

• OS: Arch Linux
• Architecture: x86_64
• glibc version:

GNU C Library (GNU libc) stable release version 2.25, by Roland McGrath et al.
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 7.1.1 20170516.
Available extensions:
    crypt add-on version 2.1 by Michael Glad and others
    GNU Libidn by Simon Josefsson
    Native POSIX Threads Library by Ulrich Drepper et al
    BIND-8.2.3-T5B
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<https://bugs.archlinux.org/>.

[DhavalKapil]

Thank you for a detailed description! The issue is that printf on your machine internally depends on malloc. This creates a recursion resulting in a segmentation fault.

[DhavalKapil]

A possible fix that comes to my mind is to set a particular variable before calling external functions. Also, within malloc, redirecting calls of to __libc_malloc whenever that variable is set. Though, I guess it'll create problems in mutexes. l I'll think over it and let you know when I fix it.

[ghost]

this is the output of make rundtests on arch, it works

test
buffer_overflow
[LIBDHEAP LOG] : Buffer overflow detected in heap chunk
[LIBDHEAP LOG] : Inconsistent heap canary!
[LIBDHEAP LOG] : Printing Stack Trace ====>
[LIBDHEAP LOG] : 0x400565
[LIBDHEAP LOG] : 0x7f89f0b8a43a
[LIBDHEAP LOG] : <==== End of Stack Trace

buffer_underflow
[LIBDHEAP LOG] : Buffer underflow detected in heap chunk
[LIBDHEAP LOG] : Inconsistent heap canary!
[LIBDHEAP LOG] : Printing Stack Trace ====>
[LIBDHEAP LOG] : 0x400574
[LIBDHEAP LOG] : 0x7f5c0969143a
[LIBDHEAP LOG] : <==== End of Stack Trace

double_free
[LIBDHEAP LOG] : Freeing non allocated chunk!
[LIBDHEAP LOG] : Printing Stack Trace ====>
[LIBDHEAP LOG] : 0x400596
[LIBDHEAP LOG] : 0x7fedd560843a
[LIBDHEAP LOG] : <==== End of Stack Trace

yes the problem seems to be the malloc used in the printf, but I don't have any fix in mind at the moment..