Tracking down stack overflows is tedious work. Especially when the entire stack is blown away leaving you with crash dumps like the excerpt following this paragraph. This crash dump is from an actual process in case you are curious. Specifically, it is from one of the bugs detailed in TPTI-07-02: Trend Micro ServerProtect eng50.dll Stack Overflow Vulnerabilities. It's pretty obvious that it's game over for our target here. The important question to answer at this juncture is, where is the bug? There are a number of approaches you can take to manually handle this situation. Please add any creative ones you may have as a comment.

    [INVALID]:41414141 Unable to disassemble at 41414141 from thread 568
    caused access violation when attempting to read from 0x41414141
    
    CONTEXT DUMP
      EIP: 41414141 Unable to disassemble at 41414141
      EAX: 00000001 (         1) - N/A
      EBX: 0259eedc (  39448284) - AAAAAAAAAAAAA.....AAAAAAAAAAAAAAAAAAAAA (stack)
      ECX: 00000000 (         0) - N/A
      EDX: ffffffff (4294967295) - N/A
      EDI: 00000000 (         0) - N/A
      ESI: 0259f102 (  39448834) - AAAAAAAAAAAAA.....AAAAAAAAAAAAAAAAAAAAA (stack)
      EBP: 00000001 (         1) - N/A
      ESP: 0259e2d4 (  39445204) - AAAAAAAAAAAAA.....AAAAAAAAAAAAAAAAAAAAA (stack)
      +00: 41414141 (1094795585) - N/A
      +04: 41414141 (1094795585) - N/A
      +08: 41414141 (1094795585) - N/A
      +0c: 41414141 (1094795585) - N/A
      +10: 41414141 (1094795585) - N/A
      +14: 41414141 (1094795585) - N/A
    
    disasm around:
            0x41414141 Unable to disassemble

In this blog entry, I present stack_integrity_monitor.py. A command line utility implemented in under 150 lines of Python code which provides an automated solution to the task of tracking down the source of a stack overflow. This Python utility leverages PyDbg, a pure-Python win32 debugger interface. PyDbg is part of the larger PaiMei Reverse Engineering Framework. If you've never heard of PaiMei before, follow the link to learn more.

The main reason stack overflows are exploitable is because control information is stored in the same medium as volatile user-controllable data. If we can move or mirror the call-chain "out of band", then we can verify the integrity of the stack at run-time. Skipping over the intricate details, here is the high level overview of how the utility works:

1. Instantiate a debugger object and attach to the target program.
2. Set a breakpoint where we want the trace to start, this can be as simple as setting a break on recv().
3. Once the breakpoint is hit, set the active thread to single step.
4. When a CALL instruction is reached, copy the stack and return addresses to an internal "mirror" list.
5. When a RET instruction is reached, walk through the "mirror" list and verify that the values match the actual stack.
6. When the last saved return address is reached, pop it off the internal "mirror" list.

If during the stack integrity check a mismatch is found, then not only do we know that a stack overflow has occurred, but we know which functions frame the overflow originated in and we can pinpoint the cause of the overflow. Using Trend Micro again as a real-world example, the previously shown (and mostly useless) crash dump turns into the following (very useful) output when launching the target under the peering eyes of stack_integrity_monitor.py:

    0259fc24: TmRpcSrv.dll.65741721
    0259e7b4: StRpcSrv.dll.65671190
    0259e7a8: Eng50.dll.61181d8c
    0259e790: Eng50.dll.611819a0
    0259e564: Eng50.dll.61181a50
    0259e2d0: Eng50.dll.61190fa4 -- 41414141
    0259e03c: Eng50.dll.61190fd2

Examining the vicinity of the last return address in the list, we find:

    61190FC7 lea edx, [esp+288h+szShortPath]
    61190FCB push esi
    61190FCC push edx
    61190FCD call _wcscpy  
    61190FD2 add esp, 8

The wcscpy() is the source of the stack overflow. The origin of the overflowed buffer is obvious in this case, it resides in the current function frame with a size of 600 bytes. Had the overflow occurred in a buffer originating further up the call chain the stack_integrity_monitor would have told us. In this case we see the stack mismatch occurred at stack address 0x0259e2d0 which should contain the return address 0x61190fa4 but instead contains 0x41414141. Had even a single byte of the return address been overwritten, stack_integrity_monitor would have detected it.

This handy command line utility has been checked into the PaiMei SVN repository and will be distributed with future releases of the reverse engineering framework. Future improvements may include speed boosts and perhaps additionally mirroring saved frame pointers. This quick hack was written in less than 2 hours and motivated from necessity, on a day where I happened to need to track down a dozen or so stack overflows. Give it a try, let me know what you think.