I’ve been on a bit of a code obfuscation/deobfuscation kick these last few months (which is one of the forces driving my slop counter up :D). Maybe I should publish some of my thoughts on this whole thing. Maybe I have no idea about some things or am just plain wrong, but yeah.

Bin2bin solutions seem to dominate the commercial protection market for obvious reasons - who wants to alter their toolchain/ship their source code to run some compiler pass-level solution? Compiler pass level solutions seem to mainly be in-house stuff. For instance, the following obfuscation turning every call indirect (and adding a bunch of MBA) (pulled from a product protected by what is presumably an in-house solution mixing compiler pass and bin2bin), was probably done by a compiler pass. The surviving direct calls seem to be to functions like memcpy/memset and such - which I think a bin2bin solution would have obfuscated as well (also, the function is still in .text and the bin2bin protector the product uses seems to place most of its doings in its own section, sometimes also cannibalizing space presumably taken by rewritten functions/padding for a few VM handlers).

What’s the point of virtualization?

I’m not quite sure what the point of code virtualization is these days. What code virtualization seems to actually do:

• make it more difficult to understand function control flow (cause things are separated into handlers and control flow can be handled within the VM), tying into…
• making it more difficult to understand what function does by pressing F5
• and killing performance

Depending on the amount of protection applied to how the VM functions (selecting the next handler ETC), the VM can collapse with just a bit of compiler optimizations. And while making things not fold can make the next handler not trivially concretizable, what’s the point of doing VM handlers at all? Why not handle it closer to control flow flattening - split function into tailcallable individual basic blocks and handle control flow transfer using your fancy next handler selector MBA?

It’ll kill less performance because you’re now obfuscating way fewer things, but should still keep the main benefits of code virtualization - the control flow is harder to figure out and thus a F5 presser may have no idea what’s going on. If you’re worried about arithmetic/other program logic being easier to figure out, apply more MBA (or something else that fits) there - you’ll still probably get less of a performance drop than code virtualization. Maybe apply some instruction mutation too - while I’m not sure about the point of it in the modern binary lifting landscape and exploiting lifter gaps may be silly given frontier LLMs, I’m sure there are operations that are painful to represent in a lifted form and will confuse the F5 presser.