Debugging programs is not fun or easy but knowing how to create and analyze a core file dump from a dying program makes your life a bit easier.
First off check the system core dump size limit.
$ ulimit -c
0
Set the maximum size of core files created to unlimited. This must be done each time you start a new login session as it will only change this limit for the current shell.
$ ulimit -c unlimited
$ ulimit -c
unlimited
Here is the source code for a simple C program that will produce a segfault and a coredump.
vim coredump.c
#include <sys/types.h>
#include <signal.h>
main()
{
kill(getpid(), SIGSEGV);
}
Build the program.
$ make coredump
cc coredump.c -o coredump
Now you can run it.
$ ./coredump
Segmentation fault (core dumped)
The core file should now exist in the execution directory.
$ ls core.*
core.12345
Use gdb to look at the core file. This show you the stack trace for each thread in the program. This must be done on the same system since a different system may have different libraries and the results might not be correct.
$ gdb coredump core.12345
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `./coredump'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000031ba830717 in kill () from /lib64/libc.so.6
(gdb) thread apply all bt
Thread 1 (Thread 27701):
#0 0x00000031ba830717 in kill () from /lib64/libc.so.6
#1 0x00000000004004f2 in main ()
(gdb) q