Is there a way to record CPU operations for a given process as they occur in production OS environment?
Doing malware analysis I'm interested in being able to log every instruction the CPU executes for a given process. I was hoping this was perhaps possible in a standard VM environment, even if not possible in a bare-metal execution scenario. I'm interested specifically in Windows 10 for now.
To be clear, I don't mean inside a debugger or special emulator sandbox designed specifically for malware detection. That's going to be detectable by the attacker and can be bypassed. I'm thinking of an approach where we monitor the execution on our real, production machines (perhaps having all systems operate as VMs/hypervisors if necessary) and as the forbidden tools execute we analyze and detect the behavior fingerprint. This would include any dynamically allocated executable data, for example data made executable after a
Solutions I'm considering
Processors already handle context switches as a regular part of multi-threaded architectures, for example Hyper Threading, and I guess one way to describe what I'm interested in doing is to track the state of one of those contexts as the VM simulates it.
Considering a general arithmetic addition operation A = B+1. The instruction is stored in the instruction register and the program counter is incremented. A and B are read from memory and are stored in registers R1, R2 respectively. In this case, B+1 is calculated and written in R1 as the final answer. This operation as there are sequential reads and writes and there's no waits for function calls used, hence no context switch/wait takes place in this case. - source
Perhaps I would need to hijack the VM's implementation of this process to track which process is currently in filling the register state and insert some code in the VM's hyperthreading emulation code to store that state somewhere else each time it updates?
Perhaps that's overkill. Perhaps no VMs actually virtualize systems on such a low level to make this possible. I'm still learning about the lower level working of VMs. I would expect such low-level emulation to cause a significant performance hit, especially if we're recording register states between each instruction - a massive performance hit probably.
Can this be achieved with existing virtual machine tools, or why not? What features should I look to in VM software or Windows OS documentation? Or rather, will I almost certainly have to make low-level modifications to a VM product to create a custom solution for doing this? Or perhaps would a new kind of VM have to be built to essentially do this kind of instruction processing virtualization?
Yes, some VMs can do that, but only by using emulation, never virtualization.
It's absolutely not possible to do on bare metal without JTAG (which, due to its limited bandwidth, reduces performance massively), nor is it possible to do in a VM while using hardware-accelerated virtualization like VT-x or AMD-v because only certain instructions can be configured to trap and return control to the hypervisor. You can't trap, for example, the
testinstruction when running in a virtual machine, so it will execute without you being able to record or influence it.
The only solution is to use a VM which emulates instructions, like QEMU in softmmu mode or Bochs. Note that this will also come with a massive performance hit, and the malware you're analyzing will likely realize that it's running in a VM, because there are going to be differences between the VM and real hardware. However, you will be able to record and even adjust individual instructions. You'll be able to do everything you could do with GDB, but on the level of the entire OS.
Obviously malware would have an easy time knowing that it's running under some sort of debugger or emulator, but since you can manipulate individual instructions, registers, and memory, you can force the malware to think it is running on bare metal. Of course, this requires an understanding of the malware and how it operates, which itself requires a working knowledge of reverse engineering. Being able to step through individual instructions and modify the execution context at will makes that easier, because you're not limited to changing the malware's environment and running it blindly.