•What is the kernel-debug package?

The kernel-debug package provides the same stable kernel provided by the common kernel package with some debug options enabled.

•What does the kernel-debug package contain?

The kernel-debug is not the same as kernel-debuginfo which has the goal to provide a executable image of the kernel with all debug symbols, while kernel-debug has the goal to enable some debugging code but does not have
the debug symbols on it.

Environment

•Red Hat Enterprise Linux 5
•Red Hat Enterprise Linux 6

Root Cause

•The kernel-debug enables the following options on the kernel that are disabled on the default kernel:

•CONFIG_DEBUG_SLAB
◦Makes kernel do limited verification on memory allocation as well as poisoning memory on free to catch use of freed memory. (performance impact mainly on kmalloc / mfree calls).

•CONFIG_DEBUG_MUTEXES
◦Allows mutexes semantics violations to be detected and reported.

•CONFIG_DEBUG_RT_MUTEXES
◦Allows rt mutex semantics violations and rt mutex related deadlocks (lockups) to be detected and reported automatically.

•CONFIG_DEBUG_RWSEMS
◦Allows read-write semaphore semantics violations to be detected and reported.

•CONFIG_DEBUG_LOCK_ALLOC
◦This feature will check whether any held lock (spinlock, rwlock, mutex or rwsem) is incorrectly freed by the kernel, via any of the memory-freeing routines (kfree(), kmem_cache_free(), free_pages(), vfree(), etc), whether a live lock is incorrectly reinitialized via spin_lock_init(),mutex_init(),etc, or whether there is any lock held during task exit.

•CONFIG_PROVE_LOCKING
◦This feature enables the kernel to prove that all locking that occurs in the kernel runtime is mathematically correct: that under no circumstance could an arbitrary (and not yet triggered) combination of observed locking sequences (on an arbitrary number of CPUs, running an arbitrary number of tasks and interrupt contexts) cause a deadlock.

•CONFIG_DEBUG_VM
◦Turn on extended checks in the virtual memory system (performance impact).

•CONFIG_DEBUG_SPINLOCK
◦Built SMP to catch missing spinlock initialization and certain other kinds of spinlock errors commonly made. This is best used in conjunction with the NMI watchdog so that spinlock deadlocks are also debuggable.

•CONFIG_DEBUG_SPINLOCK_SLEEP
◦Various routines which may sleep will become very noisy if they are called with a spinlock held.

•CONFIG_LOCK_STAT
◦Enables tracking lock contention points ( see /usr/share/doc/kernel-doc/Documentation/lockstat.txt ).

•CONFIG_XFS_DEBUG
◦Enable XFS debugging features, including ASSERT checks, function wrappers around macros, and extra sanity-checking functions in various code paths (make a huge and slow code).

Diagnostic Steps

•Install the kernel-debug package:

[root@host ~]# yum install kernel-debug

•Boot the system on the new installed kernel named kernel-.elXdebug (where

•The system will generate more information into the logs and also add a new file into the pseudo /proc filesystem named: /proc/lock_stat