Malware authors often encode their malicious payload, to avoid detection and make analysis more difficult.
I regurlarly see payloads encoded with the XOR function. Often, they will use a sequence of bytes as encoding key. For example, lets take Password as encoding key. Then the first byte of the payload is XORed with the first byte of the key (P), the second byte of the payload is XORed with the second byte of the key (a), and so on until all bytes of the key have been used. And then we start again with the first byte of the key: the ninth byte of the payload is XORed with the first byte of the key (P), …
Let width:889px” />
So just by opening a XOR encoded PE file with a binary editor, we can see the repeating key, provided that the key is smaller than the sequences of 0x00 bytes.
Second interesting property of the XOR function: if you XOR the original file (cleartext) with the encoded file (ciphertext), you get the key (or to be more precise, the keystream).
Lets take another example. We know that in many PE files, you can find the string This program can not be run in DOS mode. width:882px” />
So if we have the encoded file, and the partially unencoded file, we can also recover the key, provided again that the key is smaller than the unencoded text, and that we know where to line-up the encoded and unencoded text.
In a next diary entry, I will show a tool to automate this analysis process.
(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.