Bugchecks Depicted 151
kd> kL
Child-SP RetAddr Call Site
fffffadf`dfcf19b8 fffffadf`dfee38c4 nt!KeBugCheck
fffffadf`dfcf19c0 fffff800`012ce9cf userdump!UdIoctl+0x104
fffffadf`dfcf1a70 fffff800`012df026 nt!IopXxxControlFile+0xa5a
fffffadf`dfcf1b90 fffff800`010410fd nt!NtDeviceIoControlFile+0x56
fffffadf`dfcf1c00 00000000`77ef0a5a nt!KiSystemServiceCopyEnd+0x3
00000000`01eadd58 00000001`0000a755 ntdll!NtDeviceIoControlFile+0xa
00000000`01eadd60 00000000`77ef30a5
userdump_100000000!UdServiceWorkerAPC+0x1005
00000000`01eaf970 00000000`77ef0a2a ntdll!KiUserApcDispatcher+0x15
00000000`01eafe68 00000001`00007fe2 ntdll!NtWaitForSingleObject+0xa
00000000`01eafe70 00000001`00008a39
userdump_100000000!UdServiceWorker+0xb2
00000000`01eaff20 000007ff`7fee4db6
userdump_100000000!UdServiceStart+0x139
00000000`01eaff50 00000000`77d6b6da ADVAPI32!ScSvcctrlThreadW+0x25
00000000`01eaff80 00000000`00000000 kernel32!BaseThreadStart+0x3a
This might be useful if we want to see kernel data that happened to be at the
exception time. In this case we can avoid requesting complete memory dump of physi-
cal memory and ask for kernel memory dump only together with a user dump.
Note: do not set this option if you are unsure. It can have your production servers
bluescreen in the case of false positive dumps.

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
152 PART 2: Professional Crash Dump Analysis
CF
Bugcheck CF name is the second longest one:
TERMINAL_SERVER_DRIVER_MADE_INCORRECT_MEMORY_REFERENCE (cf)
Arguments:

corresponding to display driver code and data might be unknown. This can also hap-
pen in an arbitrary process context if we access the code that belongs to a display driver
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
Bugchecks Depicted 153
that doesn’t correspond to the current session protocol. This can be illustrated with the
following example where TSDD can be either RDP or ICA display driver.
In the list of loaded modules we can see that ATI and TSDD drivers are loaded:
0: kd> lm
start end module name
...
...
...
77d30000 77d9f000 RPCRT4 (deferred)
77e10000 77e6f000 USER32 (deferred)
77f40000 77f7c000 GDI32 (deferred)
77f80000 77ffc000 ntdll (pdb symbols)
78000000 78045000 MSVCRT (deferred)
7c2d0000 7c335000 ADVAPI32 (deferred)
7c340000 7c34f000 Secur32 (deferred)
7c570000 7c624000 KERNEL32 (deferred)
7cc30000 7cc70000 winsrv (deferred)
80062000 80076f80 hal (deferred)
80400000 805a2940 nt (pdb symbols)
a0000000 a0190ce0 win32k (pdb symbols)
a0191000 a01e8000 ati2drad (deferred)
a01f0000 a0296000 tsdd (no symbols)
b4a60000 b4a72320 naveng (deferred)
b4a73000 b4b44c40 navex15 (deferred)
…
…

0: kd> .process /r /p 87540a60
Implicit process is now 87540a60
Loading User Symbols
................
Now TSDD display driver address is valid:
0: kd> !pte a020b1d4
A020B1D4 - PDE at C0300A00 PTE at C028082C
contains 5D2C2863 contains 20985863
pfn 5d2c2 –DA–KWV pfn 20985 –DA–KWV
but ATI driver address is not. It is unknown and this is expected because no real
display hardware is used:
0: kd> !pte a0191000
A0191000 - PDE at C0300A00 PTE at C0280644
contains 5D2C2863 contains 00000000
pfn 5d2c2 –DA–KWV not valid

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
Bugchecks Depicted 155
Let’s switch to session 0 where the display is “physical”:
PROCESS 8898a5e0 SessionId: 0 Cid: 0180 Peb: 7ffdf000 ParentCid:
0164
DirBase: 14c58000 ObjectTable: 8898b948 TableSize: 1322.
Image: csrss.exe
0: kd> .process /r /p 8898a5e0
Implicit process is now 8898a5e0
Loading User Symbols
..............
TSDD driver address is unknown and this is expected too because we no longer
use terminal services protocol:
0: kd> !pte a020b1d4

For it I created a small working multithreaded program:
#include "stdafx.h"
#include <stdio.h>
#include <process.h>
typedef void (*REQ_JUMP)();
typedef void (*REQ_RETURN)();
const char str[] = "\0\0\0\0\0\0\0";
bool loop = true;
void return_func()
{
puts("Return Func");
loop = false;
_endthread();
}
void jump_func()
{
puts("Jump Func");
}
void internal_func_2(void *param_jump,void *param_return)
{
REQ_JUMP f_jmp = (REQ_JUMP)param_jump;
REQ_RETURN f_ret = (REQ_RETURN)param_return;
puts("Internal Func 2");
// Uncomment memcpy to crash the program
// Overwrite f_jmp and f_ret with NULL
// memcpy(&f_ret, str, sizeof(str));
__asm
{
push f_ret;
mov eax, f_jmp

internal_func_2 gets two parameters: the function address to jump and the func-
tion address to call upon the return. The latter sets loop variable to false in order to
break infinite main thread loop and calls _endthread. Why we need this complexity in
the small sample? Because I wanted to simulate FPO optimization in an inner function
call and also gain control over a return address. This is why I set EBP to zero before
jumping and pushed the custom return address which I can change any time. If I used
the call instruction then the processor would have determined the return address as the
next instruction address.
The code also copies two internal_func_2 parameters into local variables f_jmp
and f_ret because the commented memcpy call is crafted to overwrite them with zeroes
and do not touch the saved EBP, return address and function arguments. This is all to
make the stack trace incorrect but at the same time to make manual stack reconstruc-
tion as easy as possible in this example.
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
Manual Stack Trace Reconstruction 159
Let’s suppose that memcpy call is a bug that overwrites local variables. Then we
obviously have a crash because EAX is zero and jump to zero address will cause access
violation. EBP is also 0 because we assigned 0 to it explicitly. Let’s pretend that we
wanted to pass some constant via EBP and it is zero.
What we have now:
EBP is 0
EIP is 0
Return address is 0
When we load a crash dump WinDbg is utterly confused because it has no clue
on how to reconstruct the stack trace:
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(bd0.ec8): Access violation – code c0000005 (first/second chance not
available)
eax=00000000 ebx=00595620 ecx=00000002 edx=00000000 esi=00000000

0069ff9c 0069ff88
0069ffa0 0069fb34
0069ffa4 0069ffdc
0069ffa8 78138cd9 msvcr80!_except_handler4
0069ffac d207e277
0069ffb0 00000000
0069ffb4 0069ffec
0069ffb8 781328c8 msvcr80!_endthread+0xcb
0069ffbc 7d4dfe21 kernel32!BaseThreadStart+0x34
0069ffc0 00595620
0069ffc4 00000000
0069ffc8 00000000
0069ffcc 00595620
0069ffd0 c0000005
Here we can start searching for the following pairs:
EBP: PreviousEBP
Function return address
…
…
…
PreviousEBP: PrePreviousEBP
Function return address
…
…
…
for example:
0:001> dds esp
0069ff54 00000000
0069ff58 00000000
0069ff5c 00000000

0069ff8c 00000000
0069ff90 00000000
0069ff94 00595620
0069ff98 c0000005
0069ff9c 0069ff88
0069ffa0 0069fb34
0069ffa4 0069ffdc
0069ffa8 78138cd9 msvcr80!_except_handler4
0069ffac d207e277
0069ffb0 00000000
0069ffb4 0069ffec
0069ffb8 781328c8 msvcr80!_endthread+0xcb
0069ffbc 7d4dfe21 kernel32!BaseThreadStart+0×34
0069ffc0 00595620
0069ffc4 00000000
0069ffc8 00000000
0069ffcc 00595620
0069ffd0 c0000005
We also double check return addresses to see if they are valid code indeed. The
best way is to disassemble them backwards. This should show call instructions resulted
in saved return addresses:
0:001> ub WrongIP!internal_func_1+0x1f
WrongIP!internal_func_1+0x1:
00401871 mov ebp,esp
00401873 push offset WrongIP!GS_ExceptionPointers+0x38 (00402124)
00401878 call dword ptr [WrongIP!_imp__puts (004020ac)]
0040187e add esp,4
00401881 push offset WrongIP!return_func (00401850)
00401886 mov eax,dword ptr [ebp+8]
00401889 push eax

781328b6 mov dword ptr [eax+4],edx
781328b9 call msvcr80!_freefls (78132e41)
781328be call msvcr80!_initp_misc_winxfltr (781493c1)
781328c3 call msvcr80!_endthread+0×30 (7813282d)
0:001> ub BaseThreadStart+0x34
kernel32!BaseThreadStart+0x10:
7d4dfdfd mov eax,dword ptr fs:[00000018h]
7d4dfe03 cmp dword ptr [eax+10h],1E00h
7d4dfe0a jne kernel32!BaseThreadStart+0x2e (7d4dfe1b)
7d4dfe0c cmp byte ptr [kernel32!BaseRunningInServerProcess
(7d560008)],0
7d4dfe13 jne kernel32!BaseThreadStart+0x2e (7d4dfe1b)
7d4dfe15 call dword ptr [kernel32!_imp__CsrNewThread (7d4d0310)]
7d4dfe1b push dword ptr [ebp+0Ch]
7d4dfe1e call dword ptr [ebp+8]
Now we can use extended version of k command and supply custom EBP, ESP
and EIP values. We set EBP to the first found address of EBP:PreviousEBP pair and set
EIP to 0:
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.