Automatically Identifying Security Checks for Detecting Kernel Semantic Bugs

OS kernels enforce a large number of security checks to validate system states. We observe that security checks are in fact very informative in inferring critical semantics in OS kernels. Specifically, security checks can reveal (1) whether an operation or a variable is critical but can be erroneous, (2) what particular errors may occur, and (3) constraints that should be enforced for the uses of a variable or a function. Such information is particularly valuable for detecting kernel semantic bugs because the detection typically requires understanding critical semantics. However, identifying security checks is challenging due to not only the lack of clear criteria but also the diversity of security checks. In this paper, we first systematically study security checks and propose a mostly-automated approach to identify security checks in OS kernels. Based on the information offered by the identified security checks, we then develop multiple analyzers that detect three classes of common yet critical semantic bugs in OS kernels, including NULL-pointer dereferencing, missing error handling, and double fetching. We implemented both the identification and the analyzers as LLVM passes and evaluated them using the Linux kernel and the FreeBSD kernel. Evaluation results show that our security-check identification has very low false-negative and false-positive rates. We also have found 164 new semantic bugs in both kernels, 88 of which have been fixed with our patches. The evaluation results confirm that our system can accurately identify security checks, which helps effectively identify numerous critical semantic bugs in complex OS kernels.