TippingPoint Digital Vaccine Laboratories
DID YOU KNOW... TippingPoint customers were protected against 0-day exploitation of MS07-017 two years prior to the exploit being discovered in the wild.

MindshaRE: Checking Return Values

When auditing applications it is necessary to track down every possible error that may exist. Though every programming error may not lead to code execution some may, and a good researcher will check every last one of them. One of these programming errors is the lack of return value checking. So today we take a quick look at how to check return values in an efficient manner.

MindshaRE is our sometimes bi-weekly look at some simple reverse engineering tips and tricks. The goal is to keep things small and discuss every day aspects of reversing. You can view previous entries here by going through our blog history.

My favorite book on vulnerability research The Art of Software Security Assessment[1] makes this statement regarding return values "If a return value is misinterpreted or simply ignored, the program might take incorrect code paths as a result, which can have severe security implications."[2] This means we must treat all return values as important, and ensure they are checked properly, and the appropriate code path is followed if the check fails. While return values of all types are important we will only be looking at those dealing with the allocation of memory. The reason for this is many vulnerabilities have been discovered when subsequent usage of unchecked values has led to an exploitable condition.

Several functions are provided by Windows libraries that allow the allocation of memory. Some of the most used are malloc, LocalAlloc, VirtualAlloc, and ExAllocatePoolWithTag. There also exists a handful of wrappers for these functions like MIDL_user_allocate, IoAllocateMdl, and so on. When auditing an application we must look at all of the functions called and verify that the usage of these allocation functions is safe and invulnerable. Part of this is verifying the return code.

Because almost every allocation function returns a pointer to the allocated block the developer of the software must ensure any operation on that block is valid. In cases where memory cannot be allocated via the memory manager a NULL will be returned. Here is an example of the proper checking of an allocation.
66656D18 push    [ebp+uBytes]    ; uBytes
66656D1B push    ebx             ; uFlags
66656D1C call    ds:LocalAlloc(x,x)
66656D22 cmp     eax, ebx
66656D24 mov     [ebp+hMem], eax
66656D27 jz      loc_76E56ED5
We can see in the example that after calling LocalAlloc the developer properly checks that the value is not zero (ebx is a zero register) and branches to failure. One down, hundreds more to go! Lets take a look at an example with a  lack of sanity checks.
7774982D push    eax             ; uBytes
7774982E push    ebx             ; uFlags
7774982F mov     [ebp+arg_4], eax
77749832 call    ds:LocalAlloc(x,x)
77749838 lea     ecx, [ebp+uBytes]
7774983B push    ecx
7774983C push    eax
7774983D push    [ebp+arg_4]
77749840 mov     [ebp+hMem], eax
77749843 push    [ebp+var_4]
77749846 push    ebx
77749847 push    [ebp+var_8]
7774984A call    edi ; NtAdjustPrivilegesToken(x,x,x,x,x,x)
You can immediately see that the return value of LocalAlloc is ignored and instead used directly as an argument to NtAdjustPrivilegesToken. This could potentially lead to a vulnerability if the right conditions are met, so we would want to investigate this function further.

Return value checking is not a hard process. You simply check that eax is tested for zero at each call of an allocation routine. The problem is that these calls happen hundreds of times. So we are going to write a script that automates this for us.

Our script is simple. We first locate all allocation routines. Next we cross reference those routines and build a list of all calling functions. From there we use some simple mnemonic comparison logic to test if a comparison of eax has occurred, and the appropriate branch taken. By doing this we target areas of interest requiring manual inspection quickly.

The meat of this checking can be seen below and the full script can be viewed here: find_unchecked_alloc.py.
while cur_ea != BADADDR:
    mnem = GetMnem(cur_ea)
    if mnem == "mov":
        op1 = GetOpnd(cur_ea, 0)
        op2 = GetOpnd(cur_ea, 1)
        if op2 in retreg:
            pretreg = retreg[-1]
            retreg.append(GetOpnd(cur_ea, 0))
        elif op1 in retreg:
        elif not len(retreg):
    elif mnem in tests:
        op1 = GetOpnd(cur_ea, 0)
        op2 = GetOpnd(cur_ea, 1)
        for r in retreg:
            if r in [op1, op2]:
                tested = True
    elif mnem in jumps:
        if tested:
            jumped = True
            dest = idaapi.get_instruction_operand(idaapi.get_current_instruction(), 0).addr
            cur_ea = dest
    elif mnem == "jmp":
        dest = idaapi.get_instruction_operand(idaapi.get_current_instruction(), 0).addr
        cur_ea = dest
    cur_ea = Rfirst(cur_ea)
if not jumped:
    print "%x: Did not check its return value" % (ea)
Executing this script will result in a list of all addresses calling an allocation that is not properly checked. We always want to reduce the time it takes when auditing applications. As we have seen a little bit of scripting can go a long way, and save time for the interesting aspects of vulnerability research.


[1] Dowd, McDonald, Schuh. The Art of Software Security Assessment: Identifying and Preventing Software Vulnerabilities. ISBN 0321444426
[2] Dowd, McDonald, Schuh. The Art Of Software Security Assessment. pg 341.
Published On: 2009-04-16 17:12:34

Comments post a comment

  1. gera commented on 2009-04-17 @ 07:13

    Simple still usefull, like I like it.

    When walking code and pretending to follow some sort of execution flow, I found Rfirst() to be much better than NextNotTail(). Rfirst() is guarantied to return the regular flow first, and is better for chunked functions (follow's straight JMPs)

  2. Cody Pierce commented on 2009-04-17 @ 14:01

    @gera: As always thanks for the tip, Rfirst works great!

  3. Thomas commented on 2009-06-17 @ 14:32

    After analysing some flashplayer bug, I made a little test program to dynamically test if a program checks the return values of memory allocations.


    Did you find any interesting issues with this? I found lots of dangerous looking situations (eip fscked). But the bugs are extremely hard to reproduce and exploit.

  4. Cody Pierce commented on 2009-06-30 @ 14:41

    @Thomas: Great post on your website. I hope anyone reading this article checks your link out as well.

    I have found lots of interesting situations where the perfect conditions could lead to exploitation, but as you stated it was extremely unlikely. In most cases I have seen what appears to be "memory corruption" presented as exploitable which turn out to simply be a failed memory allocation that wasn't checked, and is not exploitable.