...so I reversed your reversing tool to help you reverse better.

MindshaRE is our periodic look at some simple reverse engineering tips and tricks. The goal is to keep things small and discuss every day aspects of reversing. You can view previous entries by going through our blog history or querying a search engine for dvlabs mindshare.

Update: Igor Skochinsky pointed out that hooking specific actions can be accomplished via the idautils.ProcessUiActions function (which isn't documented in the official IDAPython documentation (but looks like it was introduced in revision 344). Regardless, we've found that the ability to get a reference to any IDA widget has other uses we will show in future blog posts.

So, apparently, sometimes there are bugs in code. One in particular was thwarting some of my attempts at getting access to some previously unavailable IDA functionality: http://bugs.pyside.org/show_bug.cgi?id=816.

Specifically, I was running into this problem when trying to get access to IDA's QCoreApplication instance.

Python>from PySide import QtCore
Python>qapp = QtCore.QCoreApplication.instance()
Python>qapp.children()
Traceback (most recent call last):
  File "", line 1, in 
RuntimeError: Internal C++ object (PySide.QtCore.QCoreApplication) already deleted.

The reason I wanted to access the application object was fairly trivial: I wanted to be able to hook the GraphColor1 action that you can specify in your idagui.cfg:

"GraphColor1"           =       0               // Set node color 1

As far as I could tell, there was no way to color a basic block in IDA programmatically without traversing over every instruction (and that assumes you've already determined basic block boundaries). However, if you set that shortcut in your idagui.cfg file, IDA colors a basic block for you. If you read the IDAPython documentation for SetColor you'll see it takes a CIC constant:

SetColor(ea, what, color)

Set item color

Parameters:
ea - address of the item
what - type of the item (one of CIC_* constants)
color - new color code in RGB (hex 0xBBGGRR)
Returns:
success (True or False)

The CIC_* constants are defined in idc.idc within the idc subfolder of your IDA program files directory:

// color item codes:
#define CIC_ITEM 1          // one instruction or data
#define CIC_FUNC 2          // function
#define CIC_SEGM 3          // segment

Notice the lack of a constant definition for a basic block. As I mentioned previously, the only way to color a basic block programmatically is to determine the boundaries yourself and iterate over the instructions, calling SetColor with CIC_ITEM yourself. So, we implemented that ourselves, but it still annoyed me, which is why I set out to try to hook the function IDA uses when you set a shortcut for GraphColor1.

In order to do this, I figured I needed access to that QCoreApplication object so that I could traverse through it's children and find the registered QKeySequence for GraphColor1. Unfortunately, I was using the PySide binaries distributed by Hex-Rays available at http://www.hexblog.com/?p=333. These binaries were compiled from PySide 1.0.1 and the issue that was getting in my way was fixed in 1.0.2.

Before having realized it was a PySide bug, I thought perhaps Hex-Rays had purposely deleted some reference in order to thwart attempts at introspecting their GUI...so, I decided to try debugging IDA and breaking on functions within QtCore4.dll that appeared to be freeing QCoreApplication objects. I patched out the calls and continued testing... no luck.

After a couple e-mails with Igor Skochinsky from Hex-Rays, we came to the conclusion that I'd have to compile PySide myself, making the appropriate source code changes that their team had made in 1.0.1, but ported to 1.0.8 (latest).

In order to compile PySide, you'll need to build Qt first. My first attempt was with Qt 4.8 (don't try this, its painful). It seems Qt hardcodes some paths within its qmake.exe process that caused all sorts of issues. Apparently, there is a program called qpatch.exe (reference: http://www.riverbankcomputing.com/pipermail/pyqt/2010-January/025574.html) that lets you relocate a Qt source tree to another location, updating the paths hardcoded into the binary. But, I found no such program in Qt 4.8, nor in the 1.7GB Qt SDK I downloaded and installed. So, I tried patching the qmake.exe process myself which led to all sorts of other problems... Eventually, I found the qpatch.exe process within qt-creator version 1.3.1. Long story short, I gave up on Qt version 4.8 and instead decided to use Qt 4.7.3 which didn't appear to have the hardcoded path problem.

Once I got Qt 4.7.3 downloaded, I compiled it using the options Elias Bachaalany from Hex-Rays talks about here. The most important tidbit from that page is that you must pass -qtnamespace QT to configure.exe when compiling Qt.

Once I had Qt properly compiled I moved on to PySide. It appears that the Hex-Rays developers had to make a bunch of changes to PySide in order to get it to compile properly against their given QT namespace. I ran across several posts by IDA developers from 2010 that helped me understand what problems they were running into:

http://groups.google.com/group/pyside/browse_thread/thread/a155fa446a47714f?fwc=2&pli=1
http://www.riverbankcomputing.com/pipermail/pyqt/2010-October/028198.html

The first post contains a zip with .diff files for the changes Elias made. I'll say that while those were useful, they were not the only changes I ended up having to make in PySide 1.0.8.

The guide I used to getting PySide compiled was from the Qt website. First, I had to get the build scripts repository (I used their gitorious repo as it had more up to date code than their github):

C:\repositories\pyside>git clone git://gitorious.org/pyside/packaging.git
Cloning into 'packaging'...
remote: Counting objects: 921, done.
remote: Compressing objects: 100% (787/787), done.
remote: Total 921 (delta 462), reused 117 (delta 55)
Receiving objects: 100% (921/921), 156.41 KiB | 147 KiB/s, done.
Resolving deltas: 100% (462/462), done.

C:\repositories\pyside>

So, this packaging repo has a build.py file within packaging\setuptools\ that can fetch the git repos for the rest of PySide and build them automatically for you. This can be accomplished by running (from a Visual Studio command prompt):

python build.py -d

Now, this script will then begin trying to compile PySide and will most definitely fail to link due to the fact that our Qt library was compiled with -qtnamespace.

So, if you ctrl+c the build process you can see it attempted to git clone repos into the modules subdirectory. So, we can just download those ourself. If you look at the build.py file you'll see they have the git URLs defined like so:

# Modules
modules = {
    'dev': [
        ["Apiextractor", "master", "https://github.com/PySide/Apiextractor.git"],
        ["Generatorrunner", "master", "https://github.com/PySide/Generatorrunner.git"],
        ["Shiboken", "master", "https://github.com/PySide/Shiboken.git"],
        ["PySide", "master", "https://github.com/PySide/PySide.git"],
        ["Tools", "master", "https://github.com/PySide/Tools.git"],
    ],
    'stable': [
        ["Apiextractor", "0.10.8", "https://github.com/PySide/Apiextractor.git"],
        ["Generatorrunner", "0.6.14", "https://github.com/PySide/Generatorrunner.git"],
        ["Shiboken", "1.0.9", "https://github.com/PySide/Shiboken.git"],
        ["PySide", "1.0.8", "https://github.com/PySide/PySide.git"],
        #["Tools", "0.2.13", "https://github.com/PySide/Tools.git"],
    ],
}

So, I'll start off by saying that I didn't need the Tools repo and at this point I commented it out in the 'stable' dictionary within build.py (as it leads to compile problems later on and its really not even neccessary). Next, I pulled down the repos manually:

C:\repositories\pyside\packaging\setuptools\modules>git clone https://github.com/Py
Side/Apiextractor.git && git clone https://github.com/PySide/Generatorrunner.git
 && git clone https://github.com/PySide/Shiboken.git && git clone https://github
.com/PySide/PySide.git
Cloning into 'Apiextractor'...
remote: Counting objects: 2103, done.
remote: Compressing objects: 100% (631/631), done.
remote: Total 2103 (delta 1464), reused 2094 (delta 1456)
Receiving objects: 100% (2103/2103), 757.97 KiB, done.
Resolving deltas: 100% (1464/1464), done.
Cloning into 'Generatorrunner'...
remote: Counting objects: 1255, done.
remote: Compressing objects: 100% (506/506), done.
remote: Total 1255 (delta 745), reused 1236 (delta 727)
Receiving objects: 100% (1255/1255), 761.00 KiB, done.
Resolving deltas: 100% (745/745), done.
Cloning into 'Shiboken'...
remote: Counting objects: 16996, done.
remote: Compressing objects: 100% (4625/4625), done.
remote: Total 16996 (delta 12244), reused 16897 (delta 12152)
Receiving objects: 100% (16996/16996), 4.41 MiB | 1.51 MiB/s, done.
Resolving deltas: 100% (12244/12244), done.
Cloning into 'PySide'...
remote: Counting objects: 12923, done.
remote: Compressing objects: 100% (3782/3782), done.
remote: Total 12923 (delta 8828), reused 12876 (delta 8790)
Receiving objects: 100% (12923/12923), 5.65 MiB | 1.40 MiB/s, done.
Resolving deltas: 100% (8828/8828), done.

C:\repositories\pyside\packaging\setuptools\modules>cd Apiextractor
C:\repositories\pyside\packaging\setuptools\modules\Apiextractor>git checkout 0.10.8
HEAD is now at 0208aae... Version bump to 0.10.8.
C:\repositories\pyside\packaging\setuptools\modules\Apiextractor>cd ..\Generatorrunner
C:\repositories\pyside\packaging\setuptools\modules\Generatorrunner>git checkout 0.6.14
HEAD is now at 96259b8... Version bump to 0.6.14.
C:\repositories\pyside\packaging\setuptools\modules\Generatorrunner>cd ..\Shiboken
C:\repositories\pyside\packaging\setuptools\modules\Shiboken>git checkout 1.0.9
HEAD is now at 980b4a9... Bump version to 1.0.9.
C:\repositories\pyside\packaging\setuptools\modules\Shiboken>cd ..\PySide
C:\repositories\pyside\packaging\setuptools\modules\PySide>git checkout 1.0.8
HEAD is now at 69e2171... Fix unit test of bug 829.
C:\repositories\pyside\packaging\setuptools\modules\PySide>

This is the point at which I began merging the changes Hex-Rays made to 1.0.1 into PySide 1.0.8. There were only 19 changes from the files distributed on the Hex-Rays blog. I made those changes and PySide compilation still failed pretty hard.

I'll spare you the method by which I figured out what changes had to be made and instead I'll just give you the diff file. Also, before you compile you need to set the environmental variable QT_NAMESPACE to QT (and restart your visual studio cmd.exe process).

Once you've applied the changes you can run build.py without the -d parameter and it should give you a PySide-1.0.8qt473.win32-py2.6.exe installer that will install PySide to your Python installation's lib\site-packages\PySide.

The next step was to figure to test if the new PySide fixed the original bug I was running into:

Python>from PySide import QtCore
Python>qapp = QtCore.QCoreApplication.instance()
Python>qapp.children()
[<PySide.QtCore.QAbstractEventDispatcher object at 0x06405F30>,
 <PySide.QtGui.QWindowsStyle object at 0x05E548A0>,
<PySide.QtGui.QSessionManager object at 0x05E547B0>,
 <PySide.QtCore.QObject object at 0x0677ADF0>]

Looks like it worked. At this point I decided to try to figure out how to hook the GraphColor1 action. The first thing I did was modify the idagui.cfg file and defined GraphColor1's hotkey to the Z key:

"GraphColor1"           =       'Z'               // Set node color 1

So, I defined a QKeySequence of my own with the Z key for comparison purposes and then gathered all the widgets within the application:

from PySide import QtCore

ours = QtGui.QKeySequence(QtCore.Qt.Key_Z)

a = QtCore.QCoreApplication.instance()
b = a.allWidgets()

Next, I gathered all the QAction instances within all the widgets:

all_actions = []
for x in b:
   actions = x.findChildren(QtGui.QAction)
   if actions != []:
      all_actions.extend(actions)

... and gathered all the QKeySequences for all the QActions so I could match the Z key one I defined prior:

all_ks = {}
for x in all_actions:
   pyside = x.shortcuts()
   if pyside != []:
      if all_ks.has_key(x):
         all_ks[x].extend(pyside)
      else:
         all_ks[x] = pyside

And then searched:

res = None
for k,vals in all_ks.iteritems():
   for v in vals:
      if v == ours:
         print "Found it"
         res = k

Luckily, it worked:

Python>print res
<PySide.QtGui.QAction object at 0x055D2EE0>

I can trigger the action via the activate() method of the QAction:

Python>res.activate(QtGui.QAction.Trigger)
Command "GraphColor1" failed

The reason it is failing is that the command assumes that your current mouse cursor is within the IDA disassembly view (and when I execute this via the command line my cursor is focused in the command line window).

To test that this works, we can define a new hotkey (let's do the Q key) and assign it to a function that will call res.activate:

Python>import idaapi
Python>idaapi.CompileLine('static _keypress() { RunPythonStatement("lolz()");}')
Python>idc.AddHotkey('Q', "_keypress")
0
Python>def lolz():
Python>   res.activate(QtGui.QAction.Trigger)
Python>

After we do this, whenever we hit the 'Q' key, the GraphColor1 action is invoked and our current basic block is colored. Great success.

If you'd like the installer I've generated, you can get it here.

Keep in mind, while the example shown is just to hook GraphColor1, the fact is we can actually retrieve any object that is part of IDA's GUI and manipulate, query, hook, replace, it or any data it contains.... which can be exceptionally useful if you think about it a bit...

--
Aaron
@aaronportnoy