TippingPoint Digital Vaccine Laboratories
DID YOU KNOW... DVLabs team members gave 20 presentations throughout 2010. Abstracts and slides are available here.

Sulley vs. HP OpenView

These bugs are the result of one of our weekly all-night audit sessions. This posting will quickly walk through the discovery aspects of TPTI-07-14: HP OpenView Multiple Product Shared Trace Service Stack Overflow Vulnerabilities, outlining a simple case study of applying the Sulley Fuzzing Framework released at BlackHat US 2007.

The Shared Trace Service is by default running on many of HP's OpenView products. The description found in the Services MMC tags it as: "HP OpenView Shared Service for diagnostic tracing facility". We discovered that it listens on the same TCP port across all the OpenView products we looked at, usually as OVTrace.exe or ovtrcsvc.exe.

The first phase of the audit was discovering where user input is first processed by the service. After attaching with a debugger, setting up breakpoints on receive functions, and shooting off some data to port 5053, we discover that the OVTrace.exe process calls recv via a pointer here:

    0x0040C090    call    dword ptr [edx] ; recv
    0x0040C092    test    al, al
    0x0040C094    jnz     short zero_check
    0x0040C096 recv_failed:
    0x0040C096    mov     ecx, [esi]
    0x0040C098    mov     eax, [ecx]
    0x0040C09A    call    dword ptr [eax+8]
    0x0040C09D    push    offset RecvFailed
                          ; "ReadBuf failed: native buffer read"...
    0x0040C0A2    mov     [esi+10h], eax
    0x0040C0A5    call    log_func
    0x0040C0B3 zero_check:
    0x0040C0B3    mov     eax, [esp+0Ch+var_4]
    0x0040C0B7    test    eax, eax
    0x0040C0B9    jnz     short recv_success

As the above disassembly shows, the process calls recv and does simple checks to ensure that there was data received. As you may have noticed, a nice little bit of information is disclosed here at 0x0040C09D. They apparently have some form of logging function they use internally in which they push a string with the function name--likely to help a developer debug problems. For the reverser, however, this introduces a reliable method to enumerate function names without symbols.

In fact, if we follow into that function, we can find all the cross-references to it, and then hopefully get a larger list of function names enumerated. Doing so in IDA (Ctrl+X from log_func) gives us a multitude of cross references. The following strings were pushed to this function throughout the program:

    "Debug tracing disabled"
    "ReadInt failed: ReadBuf returned fail"
    "ReadLong failed: ReadBuf returned fail"
    "ReadShort failed: ReadBuf returned fail"
    "ReadByte failed: ReadBuf returned fail"
    "ReadUTF failed: ReadShort returned fail"
    "ReadUTF failed: ReadByte returned fail"
    "ReadUTF failed: ReadShort returned fail"
    "[Sockets][SIStream_t::ReadBuf] Invalid socket_t pointer"
    "[Sockets][SISTream_t::ReadBuf] no timeout select"
    "[Sockets][SISTream_t::ReadBuf]  timeout select %d.%d sec"
    "[Sockets][SIStream_t::ReadBuf] select returns SOCKET_ERROR (0x%x)"
    "[Sockets][SIStream_t::ReadBuf] recv returns SOCKET ERROR (%d)"
    "[Sockets][SiSStream_t::ReadBuf] recv Extended error: Socket 0x%x"
    "[Sockets]Ready has invalid socket pointer"
    "[Sockets]SIStream_t Close has invalid socket pointer"
    "[Sockets]socket_t Close failed"
    "[Sockets]SOStream_t WriteBuf has invalid socket_t pointer"
    "[Sockets]SOStream_t WriteBuf returned SOCKET_ERROR"
    "[Sockets]SOStream_t WriteBuf failed to write any bytes"

As you can see, when lacking symbols or source code this type of information is very useful to a reverser.

Back to the user input... if stepping out of ReadBuf to see what called it, you'll stumble across the "ReadByte failed: ReadBuf returned fail" string. This told us that the process calls ReadByte which in turn called ReadBuf. So, we could safely assume that the process is reading one byte off our network buffer.

Stepping out of ReadByte to see what called it, you'll immediately notice a large switch statement following the call to ReadByte. This is generally an interesting progression; the process reads one byte and then performs a switch jump on it. The following screenshot shows the call to ReadByte and the subsequent switch:

click for full size image

A perceptive reader will notice that the process does a cmp against 0x1E before the switch. This told us that the 1 byte they read is likely an opcode and it's value can not exceed 0x1E.

Taking a look at that screenshot, you'll see a reference to off_403d88. This is the location of the jump table that the switch uses. Taking a look at that address will quickly show us the amount of cases to be dealt with:

    .text:00403D88  off_403D88      dd offset loc_403BE6
    .text:00403D8C     dd offset loc_403C42
    .text:00403D90     dd offset loc_403C2A
    .text:00403D94     dd offset loc_403C3A
    .text:00403D98     dd offset loc_403C32
    .text:00403D9C     dd offset loc_403C75
    .text:00403DA0     dd offset loc_403C5A
    .text:00403DA4     dd offset loc_403C62
    .text:00403DA8     dd offset loc_403C4A
    .text:00403DAC     dd offset loc_403C52
    .text:00403DB0     dd offset loc_403C6A
    .text:00403DB4     dd offset loc_403C80
    .text:00403DB8     dd offset loc_403C96
    .text:00403DBC     dd offset loc_403CA1
    .text:00403DC0     dd offset loc_403CAC
    .text:00403DC4     dd offset loc_403C8B
    .text:00403DC8     dd offset loc_403CB7
    .text:00403DCC     dd offset loc_403BF7

So, this switch has 17 cases. Note that there is not a 1:1 relationship between cases and the number of opcodes available. That is, different opcodes may result in the same case being hit.

At this point, we had 17 different functions to audit. Taking a peek into some of those reveals that reversing them all may take some time as they have a decent amount of basic blocks and downgraphs.

So, we decided to write a very simple fuzzer using Sulley, the framework Pedram and I released at our Blackhat talk in August 2007. Because we didn't know too much about this protocol yet, we made the fuzz script extremely simple. Here's what it looked like:

        HP Data Protector OVTrace service
    s_group("opcodes", values=map(chr, range(0, 0x1E + 1)))
    # packet body
    if s_block_start("body", group="opcodes"):
        # 1 byte opcode included here because we tied it into the block
        if s_block_start("data"):

All this simple script does is run through all 1 byte opcodes from 0 to 0x1E and appends a large string buffer after it. Sulley's fuzz library will replace the string values during each iteration using values historically known for causing problems.

After letting our fuzzer take its toll on the process for about an hour, our Sulley web interface showed the results:

click for full size image

As you can see, it found two access violations. Clicking the details will give us crash dump information:

click for full size image

The SEH chain at the bottom clearly shows that we are dealing with a stack overflow as it is overwriten with "AAAA". At this point, we can pull up the PCAPs that Sulley saved and see which opcode actually caused the crash. Once we pull out that opcode, we can look through the function it called in the switch. We discovered that it calls the handling function here:

    0x00403C32 vuln_1:
    0x00403C32    push    esi
    0x00403C33    call    vuln_1_handler
    0x00403C38    jmp     short loc_403BEC

Jumping into that vuln_1_handler, we were presented with this basic block graph:

Charting all the cross references down from this function using IDA's "Display graph of xrefs from current identifier" we get this colorful image.

Zooming in on one of the first nodes in the downgraph, we saw this:

click for full size image

sprintf is a function often abused as it performs no bounds checking on the destination buffer. While this may not be the offending function, it's worth a check. So, we set a breakpoint where this function calls sprintf, which is here:

    0x004046F8 lea     eax, [esp+0E8h+var_54]
    0x004046FF push    offset aUS      ; "%u(%s)"
    0x00404704 push    eax             ; char *
    0x00404705 call    ds:sprintf

This bodes well as the sprintf takes a %s which does not limit our buffer size. Next step, we simply send the vulnerable opcode followed by a large buffer of any non NULL value. Our breakpoint was then hit and it's clear that our data was being copied and the bug exists here.

Now for curious minds, how large is the stack buffer? A simple static check in IDA shows us that var_54 is a 48 byte stack buffer as shown below:

click for full size image

So, there you have it. These example bugs are very simple and could probably be found using some luck and random data. However, this post's goal was to give the reader a foundation for a comprehensive and repeatable procedure for black box auditing of software. Hope it helps.
Tags: reverse engineering,fuzzing,sulley
Published On: 2007-08-24 14:15:46

Comments post a comment

  1. Anonymous commented on 2007-08-25 @ 11:48

    Nice post! Do you guys plan to release more sully fuzzing scripts in the future?

  2. Aaron Portnoy commented on 2007-08-27 @ 13:51


    As of right now, the sulley distribution contains a bunch of examples in the requests/ directory. A web-based request library is in the works and will likely be hosted at http://sulley.fuzzing.org once it is finished.

    Thanks for the comment!