Are there any existing JTAG (hardware debugging) based malware detection systems, and if not, why?
System software debug support is for many software developers the main reason to be interested in JTAG. Many silicon architectures such as PowerPC, MIPS, ARM, x86 built an entire software debug, instruction tracing, and data tracing infrastructure around the basic JTAG protocol. - source
I'd like to know if there are any malware detection solutions that use the dedicated debug port on x86 motherboards leverage the JTAG protocol to observe processes and detect malicious behavior signatures as they occur on the victim machine.
This port seems like a powerful solution to modern malware detection problems based on the fact that external hardware gets to monitor the system's every state change.
I have a lot of research left to do on how JTAG works, but some possibilities I considered for why it (using the dedicated physical debug port) might not work are:
Perhaps JTAG can only debug one core at a time, or not all cores at once, making it impossible to use for a system-wide monitoring solution. Relevant question
Perhaps the performance cost is too high. Relevant question
Perhaps I completely misunderstood the workings of this capability and various details make what I'm suggesting impossible.
Based on this related question I asked recently about using an OS's debugging API to track a process state, you should be able to understand this question about JTAG a little better.
To recap, that question is about my research on the application of machine learning against register and memory state change patterns to defeat evasive and polymorphic techniques used by modern malware to avoid behavior based signature recognition traditionally performed within emulator sandboxes.
By watching processes actually executing on the real machine where they must demonstrate their behavior in order to accomplish the desired goal, we can avoid the weaknesses experienced by emulator based approaches (which would be an already defeated layer in our defense strategy by the time the solution I'm asking about now would be relevant).
Are there any existing JTAG (hardware) based malware detection systems, and if not, why?
The issue was in fact bandwidth
JTAG is a bit-serial interface that runs at a maximum of 100 Mbits/sec, including all of the overhead bits for the protocol. That's something less than 12.5 Mbytes/sec of actual data. If you want to record 64 bits (8 bytes) @ 3 GHz, that would produce 24 Gbytes/second of data, more than 2000× what the JTAG interface can handle. - Dave Tweed