TippingPoint Digital Vaccine Laboratories
DID YOU KNOW... In December of 2007, Microsoft released seven security bulletins which fixed 11 new security vulnerabilities. TippingPoint and ZDI were credited with discovering a total of four of those vulnerabilities.

CA BrightStor Discovery Service Mailslot Buffer Overflow Vulnerability

TPTI-06-12: October 5th, 2006


Affected Vendors

Affected Products

TippingPoint™ IPS Customer Protection

TippingPoint IPS customers are protected against this vulnerability by Digital Vaccine protection filter ID 4267. For further product information on the TippingPoint IPS:

Vulnerability Details

This vulnerability allows remote attackers to execute arbitrary code on vulnerable installations of Computer Associates ARCserver Backup. Authentication is not required exploit this vulnerability and both the client and server are affected.

The problem specifically exists within the handling of long messages received over the Mailslot named 'CheyenneDS'. As no explicit MaxMessageSize is supplied in the call to CreateMailslot, an attacker can cause an exploitable stack-based buffer overflow.

The vulnerable Mailslot creation occurs:

casdscsvc.exe -> Asbrdcst.dll
20C14E8C push 0 ; lpSecurityAttributes
20C14E8E push 0 ; lReadTimeout
20C14E90 push 0 ; nMaxMessageSize
20C14E92 push offset Name ; "\\\\.\\mailslot\\CheyenneDS"
20C14E97 stosb
20C14E98 call ds:CreateMailslotA
20C14EA1 mov mailslot_handle, eax

Note there is no explicit MaxMessageSize specified. Later the mailslot handle is read from into a 4k buffer. The read data is also passed to a routine which calls vsprintf into a 1k buffer.

casdscsvc.exe -> Asbrdcst.dll
20C15024 mov eax, mailslot_handle
20C15029 lea edx, [esp+1044h+Buffer_4k]
20C1502D push ecx ; nNumberOfBytesToRead
20C1502E push edx ; lpBuffer
20C1502F push eax ; hFile
20C15030 call edi ; ReadFile
20C15032 test eax, eax
20C15034 jz short read_failed
20C15036 lea ecx, [esp+3Dh]
20C1503A push ecx ; char
20C1503B push offset str_ReadmailslotS ; "ReadMailSlot: %s\n"
20C15040 call not_interesting_call_to_vsnprtinf
20C15045 add esp, 8
20C15048 lea edx, [esp+3Dh]
20C1504C push edx ; va_list
20C1504D push offset str_ReadmailslotS_0 ; "ReadMailSlot: %s"
20C15052 push 0 ; for_debug_log
20C15054 call vsprintf_into_1024_stack_buf_and_debug_log

As mentioned in TSRT-06-02, exploitation of this vulnerability is possible due to the ability to exceeding the second-class Mailslot message size limitation.

Vendor Response

Computer Associates has issued an update to correct this vulnerability. More details can be found at:

Disclosure Timeline

    2006-04-27 - Vulnerability reported to vendor
    2006-10-05 - Coordinated public release of advisory


This vulnerability was discovered by: