Polymorphic Encryption Methods


In this document I want to explain methods of polymorphic self-encryption methods and how they are used today. We see them in modern viruses and in packers. Mostly these methods are used in packers with the target to hide malicous software from anti virus scanners - but it's also worth mentioning that software companies use it to hide their software and algorithms (for example Google does it on Google Desktop). However, polymorphic encryption routines are very interesting because they do encryption technologies at runtime non static (you won't believe how far it could go and in fact does).

- Peter Kleissner, Software Engineer (July 2008)

Overview over polymorphic self-encryption

Self-decryption means the software decrypts a part of itself at runtime. Ideally the software just contains a decryptor stub and the encrypted code. There are many variants of encryption methods, some more static (with a static decryption key) and some with changing and the best with polymorph changing decryption keys. I have also seen malware that does not know its own decryption key, but did just brute force its own key, until the decrypted code 'makes sense'. This is done in order to let the logging debugger/emulation environment of anti-virus vendor stopping (and yes, it works fine).

A big difference between polymorphic decryption code is how the code is decrypted, thus I know 4 versions:

A simple polymorphic code

A very simple polymorphic self-decryption code looks like:

add/sub/xor BYTE PTR [BX],X

As you can see the bytes at a given pointer BX which points to the decrypted code will be decrypted using an easy arithmetic instruction (addition, subtraction and exlcusive-or), where the opposite instruction for encryption would be subtraction, addition and exclusive-or. The code shown there is copied from a tutorial site I've found, so don't blame me for this code.

Another code sample which uses at this time an instantly changing key (very easy to reconstruct for investigating software):

LEA SI,...
MOV SP,0682h

At the first look it may seems as the decryption key would change with multiple executions (because of the stack pointer), but as you can see SP is clearly initialized before the loop. It's a simple code, but a little bit more tricky than the last one.

It's possible to realize a decryption routine with multiple of the above described methods (so there are more). Methods they would come into my sense would be:

Problems with finding the perfect self-encryption algorithm

The biggest difference to conventional encryption methods is you can not apply them, until the principle of them is "security lies in the knowledge of the key, not the algorithm". Considering this fact we have to eliminate the key or making the key unreconstructable for out-standing code. This could be done if the encrypted code is one key for itselfs decryption process. Other keys could be then depending on the encrypted code or on the key that is calculated from the encrypted code. This could be done in layers.

Anti virus vendors implement decryption routines as calculation, not as the algorithm (they won't copy your polymorphic code). Giving an example, if you have (theoretically) for every 4 byte another random one, the AV vendor would not include a 4 GB translation file.

Polymorphic self-decrypting code in the real world

Last day I debugged an email worm which has a decryption stub (which means the virus has been packed):

WSNPOEM Decryption Code analyzed:

00409D17 WSNPOEM-.<ModuleEntryPoint>    $  BE 419D4000       MOV ESI,<encrypted code>
00409D1C                                .  BB 3997101A       MOV EBX,1A109739                               ;  decryption key
00409D21                                .  33C9              XOR ECX,ECX
00409D23 <WSNPOEM-.Decryption_Loop>     >  281E              SUB BYTE PTR DS:[ESI],BL
00409D25                                .  46                INC ESI
00409D26                                .  C1EB 08           SHR EBX,8
00409D29                                .  41                INC ECX
00409D2A                                .  83F9 04           CMP ECX,4                                      ;  4 bytes decrypted?
00409D2D                                .  75 0A             JNZ SHORT 00409D39                             ;  if not => next bye, else reload decryption key
00409D2F                                .  BB 3997101A       MOV EBX,1A109739                               ;  decryption key, for every 4 byte, 1 byte to decrypt
00409D34                                .  B9 00000000       MOV ECX,0
00409D39                                >  81FE 03A04000     CMP ESI,<End of Code>                          ;  end of encrypted code reached?
00409D3F                                .^ 72 E2             JB SHORT <Decryption_Loop>
00409D41 <WSNPOEM-.encrypted code>      .  C9                DB C9

Notice this is just Layer 1 decryption code. As a good developer sees already, this code is inefficient. Instead of using ecx as 4 byte counter the developer could test ebx for zero, and use or reg,reg jnz instead of cmp reg,0 jnz. Also not to forget to use SIMD with the 4 byte subtraction (and THEN you don't even need to test ebx for zero, and don't need to reset the key).


To come to a conclusion, there are variety ways of making your decryption routine more complex. Common used methods are using localized or system-specific values as key. This makes the program working only on the you-wanted systems, and unable to be debugged or emulated on others. For this, current malicious software uses addresses of system functions or structures, mostly stack values or kernel addresses. It is also worth to mention that current malware decryption code works via multiple layers, which means layer 1 decrypts the basic code, layer 2 some specific if necessary (if no debugger detected..). You should think of making more intelligent decryption routines. Remember nothing is impossible at software level, just at least at hardware level everything is defined (as someone sayed..). Not to know, thanks for your attention, let's go back to work ;).