DETECTION AND PREVENTION OF A BUFFER OVERFLOW ATTACKS BASED ON HYBRID APPROACH

Buffer overflows vulnerabilities to compromise critical data structures. We present a black-box testing approach to detecting buffer overflow vulnerabilities. Our approach is motivated by a reflection on how buffer overflow vulnerabilities are exploited in practice. In most cases the attacker can influence the behavior of a target system only by controlling its external parameters. Therefore, launching a successful attack often amounts to a clever way of tweaking the values of external parameters. We simulate the process performed by the attacker, but in a more systematic manner. In particular, our approach exploits the fact that combinatorial testing often achieves a high level of code coverage. We have implemented our approach in a prototype tool called Trance. The results of applying Trance to five open-source programs show that our approach can be very effective in detecting buffer overflow vulnerabilities.