We (the two Stephens) have spent a bunch of head-down time recently doing embedded research (as we mentioned in the last post). In fact, in our “Hardware  Hacking For Software People” talk, we ended with a demo of a neat crash/bug found in a VERY popular cable modem. “But what about exploitation of that bug?”. This is the question we kept getting. Well, we honestly hadn’t done much of  that at the time last year. But that has all changed, and we’ve collected all of our notes on embedded exploitation (specifically on the ARM architecture), added some additional research and developed a course! We are proud to announce the public release the course  we’ve developed entitled “Practical ARM Exploitation”.

A New Course:

The purpose of the course is to introduce students with prior basic exploitation experience (on other architectures) to “real world” exploitation scenarios on the ARM processor architecture. The reality is that exploitation these days is harder and a bit more nuanced than it was in the past with the advent of protection mechanisms like XN, ASLR, stack cookies, etc. As such, this course is called “practical” because it aims to teach exploitation on ARM under the real-world circumstances (with all these protection mechanisms) that the exploit developer will encounter and have to circumvent. The course materials focus on advanced exploitation topics using Linux as the target platform running on the ARM architecture. The goal here is to use Linux as a platform for circumventing “advanced protections” while also teaching about the ARM architecture itself,  although there is obvious application for these techniques against platforms running on mobile phones, tablets, net-books, embedded devices, etc.

Our hope is that students with some previous exploitation experience go from knowing nothing about ARM on the first day to exploiting custom heap implementation (bypassing ASLR, NX) using their hand-built ROP connect-back-shell payload on the the last day.
The course contains the following:

• 650+ slides across 12 decks
• 17 lab exercises (ranging from code auditing and simple stack overflows to advanced heap exploitation and application specific exploitation)
• some “CTF style” exploitation challenges
• 80+ page printed/bound/laminated lab manual with comprehensive notes including: architecture quick reference, ARM GDB and IDA ‘gotchas’, et al
• …and maybe some exploitation exercises running on real ARM hardware (instead of in QEmu which is where we do everything for the course)

We are proud to also announce that we will be publicly debuting the course at CanSecWest in Vancouver in March of 2012. This thang was a lotta work so we are really excited to be giving this course and we look forward to meeting and talking with folks who are also interested in this stuff.

As a teaser for the course, we’ve cut out the “Cliff’s Notes” reference section from our much larger Lab Manual and are giving it away as a free piece of reference material (for that directory you keep full of notes, slides, and presentations . Enjoy.

If you want to learn a bit more about the layout of the course, you can read our Syllabus which has a brief description of every Lab exercise and Slide deck. If you are curious about if you are right for the course the Syllabus also includes a short blurb about prerequisite skills and what you should bring if you decide to take it.

Anyway, thats it, we’re really excited and look forward to seeing you there! Feel free to post any questions about the course as comments below.

Syllabus, “Who Should Take This Course” and “What to bring if you do”

Free Give Away!: Lab Manual Sample: ARM Architecture Reference

Why we’ve been away:

This post is long overdue. Our apologies. We’ve been head-down focusing on research (mostly around embedded devices and mobiles) and in the upcoming year plan to debut tools, talks, and comprehensive training on ARM Exploitation.

The materials focus on and teach about advanced exploitation topics (circumventing ASLR, XN, stack cookies, etc.) using Linux as a basis to learn the ARM architecture but with obvious applications for embedded devices and mobiles. Students (with some previous exploitation experience) go from knowing nothing about ARM to exploiting custom heap implementations using their own hand-built ROP connect-back-shell payloads. The course is:

• 600+ slides
• 17 lab exercises (from simple stack up through advanced heap exploitation and ‘application-specific’ exploitation)
• 3 “CTF” style exploitation challenges
• 80+ page lab manual (comprehensive notes including: architecture quick reference, ARM GDB and IDA ‘gotchas’)
• neat ARM-specific exploitation techniques
• Many useful tools including a library of 30+ ARM ROP gadgets for use with the exercises (with several lab units focused on techniques for finding and building ROP libraries from scratch for specific targets).

All of this from a “use only what is on the box” approach. We want people to understand the core concepts so no relying on other people’s IDA plugins, debugger helpers (like DEPLIB), or anyone else’s shellcode. You have to do it all and we show you how. We haven’t seen anything like our materials circulated publicly so we are really really excited to be releasing all of the details in the coming weeks.

Anyway, our ReCon 2o11 talk is embedded below and is also available (along with more detail) in the “Hardware Hacking for Software People”  blogpost. All of the photos from our visit to ReCon 2011 are here. Stay tuned for more on the ARM stuff…

Cheers.

Then I grew up.

None of that ever happened …the closest I’ve come to any of this fantasy is attending/speaking at  ”hacker” conferences, finding a bunch of bugs, and writing mostly lame software and exploits. I did all of that amidst the hyper-focused software quality assurance that ostentatiously calls itself “computer security research”.

But still, the dream lingered like a dusty raisin swept underneath the refrigerator in some derelict apartment. I would occasionally read about stuff like elaborate custom atm skimmer hardware, Nintendo DSi hacking, iPhonehardware jailbreaks, automobile computer hacks, smart-meter fiascos or see presentations by Barnaby Jack or Travis Goodspeed and my chin would quiver with envy as I fought back tears of regret…”Man, I’d always wanted to do *that* kind of stuff…”…ok not really.

Over the years as I focused on software security professionally, I flirted with “hardware stuff” by periodically buying PICs, solderless breadboards, jumpers, and BASIC Stamp development kits. All of this stuff ultimately sat unused on my desks and bookshelves…

That is until the last few years…

On a pentest project a few years ago, I was tasked with attacking a network infrastructure that supported a series of wireless sensors. The project was overwhelming because I’d never done any professional “results-oriented” hardware reversing or penetration testing. However, in a few short weeks, armed with Python (pyserial), a dilettante’s google-knowledge of serial taps, two days of soldering parts from Fry’s, and some basic protocol reversing/replication we had some great findings with code injection through a blackbox telemetry sensor into to the infrastructure’s backend…..All from a serial cable.

This success taught me a lot: mostly that a little bit of knowledge of hardware can go a long way for a software reverse engineer and vulnerability researcher especially now that many of the most interesting targets are implemented on embedded systems.

Over the last few years I’d collected a number of Arduino development boards from SparkFun. These (like so much other crap I’ve accumulated over the years) were destined to become re-gifted stocking stuffers…that is until I started reading one day about the simple serial protocol common in embedded controllers and integrated circuits. Most of these protocols were so simple that they only required two wires (some only requiring one!).

Enter i2c/SPI/2-Wire:

I was surprised to discover not only that there were simple serial data protocols down in those little ICs on circuit boards, but also that they were so ubiquitous. As I began reading more about them, I began finding out that ICs using SPI and i2C were virtually everywhere like:

• EEPROM
• Accelerometers
• Nintendo Wii Controllers
• Medical Equipment
• Microcontrollers for Home Appliances
• Pinball Machines
• Notebook Batteries
• Routers and Cablemodems
• HDMI and VGA cables

Of all the above, the ones that piqued my interest most were how EEPROM, HDMI/VGA, and Microcontrollers used this serial protocol. To start, very first thing I wanted to learn was how to tap these busses to begin observing them in “the wild”. I needed to perfect my methods for blindly approaching a piece of hardware. I needed to know how to go from knowing nothing about a hardware target to determining of any tappable busses were exposed. The resulting techniques and notes (along with hardware used) turned into a presentation given at SummerCon (in New York) in the summer of 2011 and ReCon Reverse Engineering Conference 2011 (in Montreal, Canada).

During this talk I walk you from knowing nothing about exposed pins to investigating them, tapping them and injecting data onto them.  The presentation concludes with a demonstration of how these techniques were applied to finding and later exploiting a bug in a popular cable modem. Along the way I discuss the types of hardware (Beagle i2c, BusPirate, Arduinos, cheap oscilloscopes, etc) used (to do all the above) and I demonstrate how to use them (with diagrams/photos and video demontrations). There are also some extras thrown in like:

• building passive ethernet taps from Home Depot parts,
• assembling/soldering hardware 9/25 pin serial taps with Radio Shack parts
• building VGA “taps”

Additionally, while developing and practicing these techniques I briefly assisted Charlie Miller on a project he was privately researching that resulted in his talk at BlackHat 2011 on reverse engineering MacBook batteries. My assistance with his project is noted in his talk and whitepaper.

If you want a more detailed walkthrough please check out the video of the recon talk available here and embedded below. Slides are also available here.

The conference was held on Saturday and Sunday.  Almost all of Immunity was there, only a few Immuniacs were missing.  Probably the most important part of the conference was that Immunity had an open bar on Saturday night.  As a consequence, Beans actually visited the sandy part of the beach for a few minutes, and met some interesting folks, which would otherwise not have happened.

Another incredibly important part of the conference and the training was that the food was pretty good. I really hate eating bad food at conferences. Outside of the conference, I would highly recommend David’s Café on 11^th Street for awesome home-style Cuban food and La Sandwhicherie for affordable late-night sandwhiches. But otherwise be careful because South Beach can be damned expensive.

Ok now for the less interesting parts…

Halvar mentioned trying to overwrite compiled Javascript bytecode as an alternative to “getting EIP” which was a cool idea.  He also talked about the practical limits of static analysis such as regarding analysis of loops, which is what I’ve learned myself from trying to do automated static analysis.

Immunity gave a talk on how pathetically easy it is to own Android devices. Basically, your phone is always a year or two behind on the current set of WebKit and Linux kernel vulnerabilities. You are aided by the fact that TCMalloc has absolutely no security features in it whatsoever (the subject of another talk), and Android does not support ASLR or NX.  So…you can be 1337 by downloading shit off full disclosure again!

Valasek and Smith detailed how they owned the IIS FTPSVC vulnerability, which pretty much boiled down to overwriting the NextEntryOffset in a free LFH chunk with 0xffff after performing extensive heap grooming to ensure that the next allocated OVERLAPPED structure containing a function pointer would point to within the range of memory pointed to by the NextEntryOffset.  (Thus, when the “next entry” is popped off, it would overwrite the function pointer). This can happen basically because the NextEntryOffset is not validated (via a checksum) when entries are popped off the list of free LFH chunks. Of course there is a lot of voodoo that was glossed over (how to “feng sheu”, “massage”, “groom”, “whatever” the heap).  Also I don’t know how reliable it would be “IRL” or on multi-core systems, but a decent talk overall.

I was unfortunately really hung over on Sunday and missed Cesar’s and Tarjeh’s talks on Windows stuff. I assume they were good talks.  I also missed Nagy’s talk on Saturday for roughly the same reason, even though the bar hadn’t even opened yet. After being informed that the conference had not actually yet finished, I made my way back in time to see the arcade-hacking presentation by Ron.  Basically he has a USB stick he can use to upload privileged Lua code to certain DDR-like arcade machines to give free credits or “tweak” the dance program to give him an advantage in competition. The entire presentation was hilarious and I mean that in a good way.

The PAX bypass was interesting.  Basically, assume you have an arbitrary memory write privilege in the kernel, and assume you can get an uninitialized stack variable disclosure vulnerability as well. Normally with PAX installed you are S-O-L. So what you do is you fork a child, he does the stack disclosure vuln to leak his kstack address back to you. Then put him into a blocking syscall that will return data back to you via a user buffer (think “wait”) and, as a consequence, the syscall will have to write the destination address somewhere on the stack while it’s waiting. Use your arbitrary write to overwrite that saved destination address with “anywhere you’d like to read” (its just an offset from the known kstack address).  The memcpy back to user is in copy_to_user and is hence protected (mostly) from SEGV type issues. ‘NUFF SAID.

This guy from McAfee talked about a fast shellcode detection engine. Basically he played tricks with the LDT to make a heuristic shellcode scanning engine faster. My opinion on shellcode detection is that there’s always the vector you didn’t think about (such as perhaps “not shellcode”). What’s more interesting really is just the dumb/clever tricks you can do with LDT entries to safely segment memory accesses even if the face of untrusted code.

The MASTER CLASS was pretty decent, but I think Immunity was over-optimistic in what they could cover in a week.  The first two days were all in ho-hum heap stuff, but if you were unfamiliar with pwning heap vulnerabilities it would’ve been useful I guess. We went over client-side use-after-free vulnerabilities which was interesting, although that could have been a week long class in itself. Also I don’t think the (not-)Chinese IE_PEERS exploit was as much “pray-after-free” as Immunity let on. Basically in the in-the-wild IE_PEERS exploit they go in a loop going “blah.setAttribute(‘s’, window);” 10 times. But the refcount on the window object was exactly 9, so that loop does actually perfectly decrement and free the window object, and on the 10th iteration the call to VariantChangeTypeEx actually allocates a new size 0×28 object which perfectly replaces the tearoff that was freed when the refcount hit 0.  So it pretty much works “perfectly”.  The only weird part is they rely on the fact that at offset 0x1c of the new object is the value 0x0fxxxxxx instead of a legit function pointer, but that still pretty much can be relied upon to point into your standard 0x0c0c0c0c heapspray. The saddest part of the (not-)Chinese exploit really was that they were doing lazy 0x0c0c0c0c-style heapspray 7 years after it was already lazy and largely unnecessary in decent exploits.  Except what’s sadder is that kind of crap actually works

The real highlight of the MASTER CLASS was Sean and Pablo’s talks on using SAT/SMT solvers. Immunity has clearly put a lot of work into making SAT/SMT usable. They provide python bindings so you can manually code constraints in a nice, easy-to-read language like Python (as opposed to this awful pseudo-LISP SMT2 language), and they can obtain and parse the results of the solver and present them back to you in an easy-to-read way.   Their sequence analyzer is also really easy to use. Basically within a day they had us (at least me) writing python scripts to automatically parse x86 machine code to ask questions like “what do I need to put into EAX to make EDX equal to 0×100” or “in this block of horrific checksumming/crypto/hashing code, what are the possible values that EDX could be on output assuming I control ECX”?  They had some dumb tricks (e.g., binary search, avoid inequalities) to make using this stuff more efficient. Instead of spending months worrying about how to do x86->IR->SMT translation, or madly scribbling notes on paper, I was just sitting around answering questions about machine code using a SMT solver after a day of instruction.

Kostya lead the kernel part of the class, but again for me it was all old hat. A lot of the mystique of “kernel stuff” is mostly because most people don’t know how to write kernel drivers. If you didn’t know about METHOD_NEITHER or that usermode code is usually mapped into kernelmode space upon vulnerability trigger, it would probably have been more interesting than I found it. However they also had us work on that SMB function pointer vulnerability from a year or two ago which was amusing to try and exploit. I lucked into it by sorting the pointed-to functions by function size and stumbling into the fact that esi gets loaded with the SMB2 packet if you redirect the ValidationRoutine to a function that executes a “retn 0×10″.  Subsequently I doodled around in IDA Free looking for interesting code paths until I found a “write zero”, “call arbitrary address”, and lastly “increment”. It seemed like the ideal place to be able to code up some constraints in a SMT solver to figure out how to explore the different potential code paths based on SMB2 packet contents but I didn’t have time to do that. I may try and revisit this vulnerability and “do it with SMT” to see how that compares to trying to figure everything out “in your head”.

Also deplib is actually pretty sweet, I’ve never used it and as a publically available tool it’s great. It really does beat manually grepping through machine code using regular expressions.  Apparently 64-bit support is forthcoming and will mostly “just work” with a few tweaks to the Immunity Debugger.

So that’s pretty much it basically. I hope I haven’t left anything out. If you go to INFILTRATE next year (assuming it is next year) make sure to visit David’s Café and get the fried pork chunks.  And talk a walk around Collins Ave to watch 20-year-old kids drive Rolls-Royces and Lamborghinis up and down the street all night long.

In general (I personally) found the SMT Summerschool to be very academic and high-level. Much of it was over my head. But this was good and bad. It was a good experience because I (perhaps masochistically) like to be exposed to new things especially things that I know nothing about.  The “bad” part of all of this was that of what I could comprehend (with the exception of a few presentations)  I found the ideas to be spectacular but completely divorced from any kind of practical implementation.

For example: In the presentation: Liquid Types: SMT Solver-based Types the researcher (Ranjit Jhala) demonstrated how a simple high-level type system can be used to describe many “insecure patterns” in an abstract way so that a user needs not fumble with building these constraints by hand. During the first half of his presentation I began to get really excited and when it came closer to time for demonstration of the tools and such, it was partially implemented for use only with OCAML (with C support maybe later)! This to me was such a big let-down, and from the comments and questions of others in the audience, I was not the only one. These kinds of post-presentations discussions were a common theme that demonstrated how divorced academia is from industry. In many of the Q&A sessions small discussions would break out between presenters and audience members on the practicality of implementation. The age old tension between the purity of academia and the “get it done so it’s usable” mentality of industry.

Of all of the presentations most relevant to vulnerability research, these were my favorites because the concepts were the most accessible for me:

• BitBlaze & WebBlaze: Tools for computer security using SMT Solvers
• Constraint Solving Challenges in Dynamic Symbolic Execution
• Liquid Types: SMT Solver-based Types
• SMT Solver-based Compiler Optimization Verification
• SAGE: Automated Whitebox Fuzzing using SMT solvers
• Symbolic Execution and Automated Exploit Generation

Of all the talks, the Constraint Solving Challenges in Dynamic Symbolic Execution was probably the best introduction to the use of SMT Solvers for basic vulnerability research. The researchers leveraged symbolic execution and the STP constraint solver to find bugs laying dormant in GNU CoreUtils. Some of these bugs were simple enough to trigger with command-line arguments (see slide 36 “Ten Command Lines of Death”).

In the end, it was a great experience to have participated in the SMT Summer School. I learned a lot and got interested in new and different applications for SMT Solvers…but I will probably have to work with them a bit more before I can get more value from another conference like this.

Also a very short slideshow of some photos taken during the SMT Summer School are here.

SummerC0n was a great time. We hope to attend the next one…whenever or wherever it will be.

An annotated slideshow of our visit to Summercon 2011 is available here.

SourceBoston was held at the (dog friendly  Boston Seaport Hotel, a surprisingly fancy venue that was on the waterfront and adjacent to The World Trade Center of Boston. Unlike many Infosec conferences we’ve attended, SourceBoston did a good job of intermingling “suits” with “grunts”. In other words, the attendees and speakers hailed from many different levels in their organizations (with a healthy sprinkling of academic types mixed in). There was also quality representation from the different niches in the Information Security community (from “hardware hacking” to “management and policy”). In short, its a cool little conference.

My talk at Source Boston 2011 was entitled “GreyHat Ruby”. The talk was on the many ways that a devout Python coder has come to find Ruby very useful for Information Security work. Here is a bullet list from a section of the presentation entitled “12 Good Reasons for C/C++/Python coders” (see examples/comparisons/screenshots in the slides!):

1. Ruby has an equally useful Interactive Interpreter.
2. Ruby has “real” case/switch statements
3. Ruby has C style ternary statements
4. Ruby has “public” and “private” namespaces
5. Ruby (like C++ and Java) let’s you define classes in “piecemeal” (split that class def across files!).
6. Ruby has a “container” class called “module” that act as namespace “directories” letting you arrange things as you see fit.
7. Ruby doesn’t automatically make a namespace entry for an “included” file.
8. Ruby has better “sprintf” functionality.
9. Ruby has strong OOP paradigm and convenient “getter/setter” syntax.
10. You can modify Ruby Class definitions “on the fly” without cumbersome “get_attr”/”set_attr”.
11. Ruby is a “functional” programming language. It has “blocks” and “anonymous functions” (not kludgy lambdas).
12. Lots of other neat things like send() and __END__.

After that list, I dive into some of the specific things useful for Information Security professionals:

1. Accessing “foreign” functions: getting your Ruby code to call into DLLs and shared objects.
2. Existing pure ruby process debuggers and hit-tracers
3. Using JRuby to talk to Java RMI services
4. Ruby and IDA
5. Build quick user-friendly CLIs (like Python’s Cmd module).
6. Plugging Ruby into Burp
7. Using Ruby’s Win32OLE/Win32API/RubyDL  to “script” mouse clicks and keyboard actions on Windows via User32. (demo video here)

8. Complete browser automation (Firefox, IE, Safari) allowing you to “script” user interaction with the browser. (Web pen-testing, QA, or fuzzing!)
9. Writing distributed code with Drb and Rinda. (Don’t bother with socket code. Or, build a distributed fuzz farm!)
10. Using Ruby to create “Domain Specific Languages” for your tasks (like fuzzing .

That’s it in a nutshell, if you want to see more detail: 

Check out the slides!

In 2003 I read the book “Linked: How Everything is Connected to Everything Else and Why It Matters“. This was one of those books that comes out every few years and immediately becomes a bestseller for it’s ability to make a niche science comprehendible for us laymen. I am a sucker for these kinds of books, favorites being: “Hyperspace“, “Guns, Germs, and Steel“, “The Black Swan“, “Blink“, and “Freakonomics“. But “Linked” remains one of my all time favorites because it taught me a bit about the science of networks just as terms like “social network” and “internet meme” were entering the vernacular.  Before this, I’d never really though about these concepts beyond road-trip games of of “Six Degrees of Kevin Bacon“. This book also led me to other books that would become favorites like: “Turtles, Termites, and Traffic Jams” and strangely “The Hero With A Thousand Faces“. While the subjects differ, the themes remain the same: finding common unifying threads in seemingly disparate pieces of information.

Shortly after reading this book, I stumbled upon an article about a “Matrix-style” brain interface device created by a team from a university in Italy that was being demo’d at Neuroscience conferences. The device was supposedly revolutionary because it was non-invasive and adaptive. I was called “non-invasive” because it used only an Electro-Encepalagraph (aka EEG which is worn on the head) and “adaptive” because the device supposedly worked on virtually everyone. At the conferences, the demonstration of the device was essentially a booth, open to anyone willing to try it. Because every brain is different and produces different signals, a short 15 minute “training session” was needed for the device to adapt itself to the new user. During those 15 minutes the ABI device would hone in on the specific electro-encephalograph signals produced by parts of the brain responsible for motor control. After those fifteen minutes, when the user merely thought about moving their left hand, a mouse cursor would move left. When they thought about moving their right hand, the cursor would move right.

This blew my mind, not only because of the crazy Lawnmower Man powers it might give me or the potential of the device (in all it’s Neuromancer-esque cyberpunkish glory) to be a boon for paraplegics but because it implied that there was some fascinating signal processing going on inside the device. Something inside that thing was able to do the data mining necessary to find those needles in the proverbial haystack of brain noise.
I wanted to know what it was and how it worked, so I tried to consume the few research papers produced by the ESPRIT ABI team.  As a full scholarship-third year Physics dropout this was very ambitious for me (thats right ladies, I’m a quitter, take a number!). After a few hours googling every odd word in the research paper and instant messaging all the smart people in my buddy list, I had a general idea of how the Adaptive Brain Interface worked.

The device essentially worked like this:

After running the EEG signals through a lot of signal filters and transformations (things with fancy names like Fast-Fourier Transforms and Butterworth Filters) the system shat out massive matrices of information. All this data was ultimately passed into a ‘neural classifier‘ that was itself largely based upon a single simple classification algorithm that I learned was called Mahalanobis Distance Statistic.

If you are thinking: “That equation looks like Greek!” (which is kinda is and “What the hell is a ‘neural classifier‘?” Then you and I aren’t too dissimilar, ‘cuz that exactly what I first thought.

Firstly, it is important to note that just because the ESPRIT ABI is a neurological interface the ‘neural’ in the phrase ‘neural classifier’ is just a coincidence. The neural classifier has nothing to do with the fact that this is a neurological device. Don’t let it confuse you, the way it did me. The neural classifier is merely an algorithm that makes use of basic neural network concepts to optimize the task of separating data into classes. The neural network is the “decision engine” that decides if a piece of data is an apple or an orange. In the case of the ESPRIT Adaptive Brain Interface, it’s neural classifier relied heavily upon Mahalanobis Distance Statistic. Other than this, we don’t need to know anything more about neural networks to continue understanding how this thing works. ( I, myself, know nothing more about neural networks other than that oft-quoted Ant-Colony Optimization technique wherein computer programs mimic ants and their pheromone trails to do “AI stuff”.)

At the time that I was diving into this, there was very little information on the web about Mahalobis Distance (certainly not the Wikipedia page that there is now) so in early 2003 or 2004 I ordered a book from Amazon called the Mahalanobis-Taguchi System. In this book, I learned that a nerdy Japanese Quality Assurance guy was able to get famous and make a buncha money by applying the Mahalanobis Distance Statistic to manufacturing processes. How? Well, take the task of photo development as an example.

With photo processing you have to apply costly chemicals to film gradually over many stages before you can produce a final photo that can be determined to be “bad” (overexposed, out of focus, etc). Well if you are able to determine that the photo will turn out bad at earlier stages in the process, you don’t have to invest the resources in developing the full final product before you decide to throw it out. This is also true of other kinds of manufacturing: from automobiles, to Spacely Sprockets. (Especially for things like CPUs where almost half of your production yield gets thrown away.)

This japanese Quality Assurance guy was able to collect metrics about the “bad yields” of different products as they flowed through their manufacturing processes.

He applied Mahalanobis Distance Statistic to these metrics at “checkpoints” along the manufacturing process and was able to make early predictions about if a yield would be good or bad. This saves companies a whole grip of cash. He presumably made loads of money and got girls from it. I too wanted money and girls, and this math crap seemed like a perfect angle for my new “tortured genius” affectation. Although rare like black pearls, Black nerds like myself need additional gimmicks to snag chicks…”tortured genius” was my new attempt after “DJ”, “tortured writer”, and “international man of mystery” had all met with limited success. (Oh, and if you think Black Nerds aren’t that rare, I challenge you to name more than five: Steve Urkel, Barack Obama, Lamar from “Revenge of the Nerds”, that dude from 30 Rock, and ….yea, see? Not easy.)

Anyway, in addition to the Mahalanobis-Taguchi book, I found that Mahalanobis is used for lots of other sexy applications like “computer vision“. But perhaps my favorite other application of Mahalanobis Distance I found in some obscure articles on the web. These articles were about how Astronomers and Chemists use Mahalanobis with spectrographs (things that measure wavelengths from light sources) to determine the chemical composition of distant stars based solely on the type of light the star emits.

“Wait a minute. How the hell does this crap have anything to do with stars?” Well, the universe is imperfect. When you burn a chemical in a lab it produces light that you can

measure with a spectrograph. A pure sample of a chemical will basically always produce the same spectrograph reading. However, when the same chemical gets mixed in with a buncha other stuff and burns as part of a chemical compound (like in a star) it will produce an “imperfect” spectrograph reading. A human can overlay the perfect andimperfect graphs to see that they look kinda similar, but your average computer program would likely nitpick at all the little abberations and imperfections in the “real-world” sample and call the two samples vastly different. Mahalanobis Distance Statistic is one tool that Astronomers use (coded into computer programs) to measure the “sameness” between the real-world readings (of stuff burning in distant stars) to the samples of purer chemicals burning in their labs. This is how we (as humans) determine the chemical composition of real stuff in the universe. Effectively, Mahalanobis Distance Statistic helps humans discover what stars are made of without actually having to visit them. It’s real sh*t, that really works!

After reading all this crap, my mind was flush with crazy ideas:

• “If this equation could filter through seemingly chaotic radio noise from our brains, then surely it could be used to do something simple like classify mp3s on my filesystem based on the ‘way they sounded’” (A few short years later, I would read about Pandora which uses a similar classification system.)

Or:

• “Surely if this thing can data-mine brain noise then it can be used to help decide if an email is spam!” (It turns out that Mahalanobis Distance offers no real improvement over Bayesian filtering because it requires some computationally expensive matrix math and matrix transformations.)

Or even better:

• “I could use this to classify streams of traffic from libpcap or Snort to help discover previously overlooked intrusions and data exfiltrations!”

Or:

• “I could use this to find out the encoding or compression schemes of data inside of seemingly arbitrary data streams.”

Or:

• “I could use this to decide if an executable is malicious based on traces of it’s execution!”

My young over-zealous mind was awash in great applications for a flexible “decision engine” based on statistical classifiers like Mahalanobis Distance.  All anyone would need to do is extract “features” from any data set and feed it to my magical solver to have all the data classified! All these ideas teemed in my head for a few years and like most of my ideas and life aspirations only manifested as half-assed and unfinished code in a repository on some forgotten backup disk. (That is until a few  months ago

But lets rewind a bit. First I had to understand a bit of more about how Mahalanobis Distance worked. I will not bore you with the details in this blogpost (if you want to know there is a Wikipedia page and plenty of step-by-step tutorials) but what is important is why Mahalanobis Distance is uniquely useful over other types of classification.

Lets imagine that you and I were tasked with creating a system that was able to categorize the two dimensional data points like the ones in the chart below:

After a few minutes of looking at the data, we would probably come up with a scheme that probably worked something like this:

1. Graph all of the observations as points on a graph.

2. Finding the center-point (or average) of the cluster of points and plot this average as a point on the graph.

3. Calculate some kind of radius from this center-point. This radius will serve as our “class threshold”. (In this case we will use a radius of ~2.o)

4. Draw a circle using the radius or “class threshold” and the center-point of the cluster.

Using this scheme, any new points added to the graph would either fall within the circle our outside of the circle. With this system, those outside can be said to be “a part of the class” or “not part of the class”.

The fancy terms for this common-sense technique are “Euclidean Mean” or “Minimum Distance Classifier“.

Well if you think like a hacker, you are probably already imagining scenarios where this classification system breaks down.  Such as:

• “What happens if a point happens to fall on the line, is it part of the class or not? How close to the line is too close?”

• “If two points are plotted with one point falling just inside the circle and the other falling just outside of the circle, they themselves are very similar to one another but determined to be drastically different simply because they don’t both fall within the “class threshold”. Is that ok?”

And lastly, and most importantly:

• “How would this system scale?” That is to say, how would it hold up if we tried to classify richer sets of data.

For example: what about data sets that have more than just X and Y coordinates? If they had X, Y, and Z coordinates we could maybe extend this idea into three dimensions by drawing a sphere as our threshold. But what if we wanted to have the same system work for data with X, Y, Z, D, E, F, H coordinates? What if we want an equation that would classify data sets with N dimensions just as easily as 2 dimensions? As coders we don’t want to hand code a different classification system for every possible dimension, we want a single algorithm we can reuse for N dimensions!

That is precisely what Mahalanobis Distance gives us. It is a scheme that does not fall apart for any of these above issues (as well as a bunch of other ones that us noobs can’t yet dream up).

Well let’s bring this back the the ESPRIT ABI thingy I mentioned earlier. You’ll recall that the ABI uses an EEG to read electrical signals in the brain from electrodes attached the surface of the head.

At any given time in a recording there is a reading for each electrode of the EEG. This data, at any given time, can be represented as a “multi dimensionalarray”. Instead of the simplistic 2-dimensional data-points like the ones from earlier, these readings have many “features”. Instead of having just X and Y coordinates the EEG readings have F3, C3, F7, T7, P7, O1, P3, (and so on), with each coordinate representing a reading from a corresponding electrode. At any given time, the EEG is constantly producing a stream of data in the form of a matrix containing the frequencies of data for each electrode.

If a researcher wanted to add more electrodes, this would effectively increase the number of dimensions (or features) that the datastream from the EEG produced. Thus, the analysis engine that consumes this data would have to be flexible and able to process data with matrices of N-dimensions. This is precisely why the researchers chose to use Mahalanobis Distance in the Adaptive Brain Interface it is one of few statistical classifiers that can be used on matrices of N-dimensions.

Well this is all very interesting, but how the hell does this have anything to do with SEO, spam, and tricking search engines? When you perform a message digest (like md5) on two blocks of text that differ by only one byte, comparing the md5 sums only tells you that the two blocks differ.

Thus, message digests are not suited to give you a measure of similarity because they can only indicate uniqueness. Well Mahalanobis is suited for exactly this. (In fact, the S-Matrix can even be represented by something that looks like a SHA256 hash). With this in mind, I set out to try to use Mahalanobis Distance Statistic to tell me “how similar” two blocks of text were.

After implementing this, I found that I could also use a bit of Natural Language Processing and word databases to not only tell me how unique a block of text was (based on the characters it contained) but also to give me a measure of uniqueness based on the lexical sentence structure (i.e. placement of nouns, pronouns, determiners, etc within the text).

From this, I was able to write proof-of-concept applications that “borrowed” the lexical structure of one body of text to programmatically generate hundreds and thousands of articles that made absolutely no sense to a human reader but were grammatically and syntactically correct enough that Google considered it indexable (including the content and links it contained). This in turn increases page rank for the sites referenced in the generated text. When performed in an automated fashion you could effectively implement this on a scale large enough to influence search engine results for specific keywords….Sound familiar? Blog spam!

In the next blogpost I will dive into the technical details, code, and applications of all this. Some of the more interesting points are:

• SEO vernacular: templates, articles, keywords oh my!
• Combinatorics and Set Math: How your fuzzer is not so different from what spammers use to generate article content.
• WTF is a Markov Chain?: How Natural Language Processing is used to “game” the search engines by determining parts of speech and “spinning” blocks of text.
• Automation: How spammers and SEOs defeat Captchas. How spammers and SEOs mass mail and automate control of many web identities (blogs, webmail, etc).
• Code and technical details for all of the above.
• Lastly, my ideas on how to defeat/improve this stuff by using Statistical classifiers like Mahalanobis distance to see if blocks of text are lexically similar.

[spinner screenshot]

[nlp post screenshot]

[mahal nlp screenshot]

The conference location (the Emirates Palace) was lavish to say the least. Shortly after Blackhat Abu Dhabi it was the first to host the most expensive Christmas Tree in the world. And earlier that year, the hotel made headlines for hosting the world’s first gold vending machine. Funny story about Barnaby Jack and that ATM (who was speaking at Blackhat Abu Dhabi about breaking into ATMs .

What made the trip even better was that my dad also came along with me. With my dad about to meet some grungy hacker types, I worried a bit about worlds colliding but my dad is a chill dude so my worries soon abated…only to immediately return upon introducing him to a very inebriated Grugq and Barnaby Jack. (I heard: “Nice friends you have here son.” a few times on this trip  But in the end, everyone decided my dad was cooler than me anyway.

As is usual for these conferences, I got to hang with old friends, make some new ones, and to meet some interesting folks like Moxie and Vanity Fair’s Michael Joseph Gross. I even got a chance to bore the FireEye CEO with some of my wacky data correlation ideas for malware analysis.

The conference itself and my talk went very well despite a relatively small turnout (by Blackhat standards). The sponsors (the Crown Prince himself, AE Cert, and officials from the “Telecommunications Regulatory Authority“) were very gracious and forgiving .

Blackhat Abu Dhabi wrapped up the same weekend as the first Abu Dhabi Grand Prix, so as grungy hackers were leaving the Emirates Palace celebrities, aristocrats, and foreign dignitaries were all showing up.

All-in-all, I’m sure the next one will be even bigger and better. If you have the chance, you should go to the next one. The UAE is pretty blingin’.

Some sample images are below, but a complete photo-journal of the trip you can view as  a big labeled slideshow here.

Ok. So (as you may already know), compiled TLB data is stored in the resource section of the PE file it is associated with. Here is what I used to do:

1. Observe the section in the resource section of the target PE using PEView.

2. Find out the length of that specific section using the PE Header data.

3. Do some basic math to find where the TLB data begins inside the PE file (using the TLB magic bytes to identify the beginning).

4. Open the target PE file in a Hex Editor and manually extract the TLB file data into a new file (or otherwise extract it using file offsets calculated from PE Header values).

5. Open the newly created TLB file (that we extracted from the PE) in OLEView so that we can convert it back to IDL form.

Well, this process obviously got tedious for every single binary I encountered, so I wrote a tool called WhoHasTLB? (using Ero Carrera’s extremely useful PeFile) to automate steps 1-4. I also compiled it into a standalone executable so it was a bit more portable and easier to use (especially moving between VMs).

I then also wrapped the TLB extraction routines in something that will search a directory recursively for PE files containing TypeLib data. This is useful on those large build directories that you get from customers with lots of binaries in them.

After the TLB files have been extracted automatically by WhoHasTLB? you can then just open them in OleView to have them converted to human readable IDL’s.

I hope this is useful!

GitHub Repository and download is here.