LLVM自定义调用约定
通过新增自定义调用约定，能够使IDA参数识别出现问题。自定义调用约定不像编写中端Pass那样简单，涉及了LLVM的后端修改和中端修改两部分。LLVM的后端部分代码主要分布在**/llvm/lib/Target**下。这里主要在X86下新增调用约定，其他架构下为一样的步骤。
0x1 修改调用约定的TableGen文件
LLVM的后端主要通过TableGen语言描述架构信息，包含寄存器，调用约定等。X86下的调用约定被定义在**/llvm/lib/Target/X86/X86CallingConv.td**文件中。该文件中定义了X86架构下支持的所有调用约定，并以如下格式进行定义。
1 2 3 4 5 6 | def CC_X86_64_C : CallingConv<[ CCIfType<[i32], CCAssignToReg<[EDI, ESI, EDX, ECX, R8D, R9D]>>, CCIfType<[i64], CCAssignToReg<[RDI, RSI, RDX, RCX, R8 , R9 ]>>, CCIfType<[i32, i64, f16, f32, f64], CCAssignToStack<8, 8>>, ..... // many rules]> |
可以发现TableGen语言实际上是一种规则描述语言，通过大量的If语句来定义调用约定的规则，例如上述约定的意义为如果参数类型为i32，那么存入EDI, ESI, EDX, ECX, R8D, R9D，如果参数类型为i64，则存入RDI, RSI, RDX, RCX, R8 , R9，如果上述规则无法满足，则使用第三条规则，即将多余参数压入栈中。
为了简单起见，我们想要新建一个调用约定，使得传参通过RAX，RBX实现，直接复制粘贴现成的调用约定。代码如下：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 | def CC_X86_64_Obfu1 : CallingConv<[ // Handles byval parameters. CCIfByVal<CCPassByVal<8, 8>>, // Promote i1/i8/i16/v1i1 arguments to i32. CCIfType<[i1, i8, i16, v1i1], CCPromoteToType<i32>>, // The 'nest' parameter, if any, is passed in R10. CCIfNest<CCIfSubtarget<"isTarget64BitILP32()", CCAssignToReg<[R10D]>>>, CCIfNest<CCAssignToReg<[R10]>>, // Pass SwiftSelf in a callee saved register. CCIfSwiftSelf<CCIfType<[i64], CCAssignToReg<[R13]>>>, // A SwiftError is passed in R12. CCIfSwiftError<CCIfType<[i64], CCAssignToReg<[R12]>>>, // Pass SwiftAsync in an otherwise callee saved register so that calls to // normal functions don't need to save it somewhere. CCIfSwiftAsync<CCIfType<[i64], CCAssignToReg<[R14]>>>, // For Swift Calling Conventions, pass sret in %rax. CCIfCC<"CallingConv::Swift", CCIfSRet<CCIfType<[i64], CCAssignToReg<[RAX]>>>>, CCIfCC<"CallingConv::SwiftTail", CCIfSRet<CCIfType<[i64], CCAssignToReg<[RAX]>>>>, // Pointers are always passed in full 64-bit registers. CCIfPtr<CCCustom<"CC_X86_64_Pointer">>, // The first 6 integer arguments are passed in integer registers. CCIfType<[i32], CCAssignToReg<[EAX, EBX ]>>, // 注意这个地方，我们进行了修改！ CCIfType<[i64], CCAssignToReg<[RAX, RBX ]>>, // 注意这个地方，我们进行了修改！ // The first 8 MMX vector arguments are passed in XMM registers on Darwin. CCIfType<[x86mmx], CCIfSubtarget<"isTargetDarwin()", CCIfSubtarget<"hasSSE2()", CCPromoteToType<v2i64>>>>, // Boolean vectors of AVX-512 are passed in SIMD registers. // The call from AVX to AVX-512 function should work, // since the boolean types in AVX/AVX2 are promoted by default. CCIfType<[v2i1], CCPromoteToType<v2i64>>, CCIfType<[v4i1], CCPromoteToType<v4i32>>, CCIfType<[v8i1], CCPromoteToType<v8i16>>, CCIfType<[v16i1], CCPromoteToType<v16i8>>, CCIfType<[v32i1], CCPromoteToType<v32i8>>, CCIfType<[v64i1], CCPromoteToType<v64i8>>, // The first 8 FP/Vector arguments are passed in XMM registers. CCIfType<[f16, f32, f64, f128, v16i8, v8i16, v4i32, v2i64, v8f16, v4f32, v2f64], CCIfSubtarget<"hasSSE1()", CCAssignToReg<[XMM0, XMM1, XMM2, XMM3, XMM4, XMM5, XMM6, XMM7]>>>, // The first 8 256-bit vector arguments are passed in YMM registers, unless // this is a vararg function. // FIXME: This isn't precisely correct; the x86-64 ABI document says that // fixed arguments to vararg functions are supposed to be passed in // registers. Actually modeling that would be a lot of work, though. CCIfNotVarArg<CCIfType<[v32i8, v16i16, v8i32, v4i64, v16f16, v8f32, v4f64], CCIfSubtarget<"hasAVX()", CCAssignToReg<[YMM0, YMM1, YMM2, YMM3, YMM4, YMM5, YMM6, YMM7]>>>>, // The first 8 512-bit vector arguments are passed in ZMM registers. CCIfNotVarArg<CCIfType<[v64i8, v32i16, v16i32, v8i64, v32f16, v16f32, v8f64], CCIfSubtarget<"hasAVX512()", CCAssignToReg<[ZMM0, ZMM1, ZMM2, ZMM3, ZMM4, ZMM5, ZMM6, ZMM7]>>>>, // Integer/FP values get stored in stack slots that are 8 bytes in size and // 8-byte aligned if there are no more registers to hold them. CCIfType<[i32, i64, f16, f32, f64], CCAssignToStack<8, 8>>, // Long doubles get stack slots whose size and alignment depends on the // subtarget. CCIfType<[f80, f128], CCAssignToStack<0, 0>>, // Vectors get 16-byte stack slots that are 16-byte aligned. CCIfType<[v16i8, v8i16, v4i32, v2i64, v8f16, v4f32, v2f64], CCAssignToStack<16, 16>>, // 256-bit vectors get 32-byte stack slots that are 32-byte aligned. CCIfType<[v32i8, v16i16, v8i32, v4i64, v16f16, v8f32, v4f64], CCAssignToStack<32, 32>>, // 512-bit vectors get 64-byte stack slots that are 64-byte aligned. CCIfType<[v64i8, v32i16, v16i32, v8i64, v32f16, v16f32, v8f64], CCAssignToStack<64, 64>>]>; |
我们将原有的CC_X86_64_C调用约定进行了复制修改，并将原有传递参数的寄存器改成了RAX和RBX。我们就得到了一个新的调用约定规则:CC_X86_64_Obfu1。
0x2 注册新增的调用约定
然后虽然拥有了定义，但是实际上并不能被调用。因此我们需要将这个规则注册到Root Argument Calling Conventions中，这表示X86架构下调用约定的选择逻辑。我们找到CC_X86_64，它是X86架构下调用约定的主约定。
可以发现这个调用约定根据不同的CallingConv的值将规则分发(通过CCDelegateTo语句)给了不同的子调用约定。我们新建一个调用约定取名为X86_64_Obfu1，并将调用约定逻辑分发给我们之前定义的CC_X86_64_Obfu1规则。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | // This is the root argument convention for the X86-64 backend.def CC_X86_64 : CallingConv<[ CCIfCC<"CallingConv::X86_64_Obfu1", CCDelegateTo<CC_X86_64_Obfu1>>, // 我们新增的调用约定 CCIfCC<"CallingConv::GHC", CCDelegateTo<CC_X86_64_GHC>>, CCIfCC<"CallingConv::HiPE", CCDelegateTo<CC_X86_64_HiPE>>, CCIfCC<"CallingConv::WebKit_JS", CCDelegateTo<CC_X86_64_WebKit_JS>>, CCIfCC<"CallingConv::AnyReg", CCDelegateTo<CC_X86_64_AnyReg>>, CCIfCC<"CallingConv::Win64", CCDelegateTo<CC_X86_Win64_C>>, CCIfCC<"CallingConv::X86_64_SysV", CCDelegateTo<CC_X86_64_C>>, CCIfCC<"CallingConv::X86_VectorCall", CCDelegateTo<CC_X86_Win64_VectorCall>>, CCIfCC<"CallingConv::HHVM", CCDelegateTo<CC_X86_64_HHVM>>, CCIfCC<"CallingConv::HHVM_C", CCDelegateTo<CC_X86_64_HHVM_C>>, CCIfCC<"CallingConv::X86_RegCall", CCIfSubtarget<"isTargetWin64()", CCDelegateTo<CC_X86_Win64_RegCall>>>, CCIfCC<"CallingConv::X86_RegCall", CCDelegateTo<CC_X86_SysV64_RegCall>>, CCIfCC<"CallingConv::X86_INTR", CCCustom<"CC_X86_Intr">>, // Mingw64 and native Win64 use Win64 CC CCIfSubtarget<"isTargetWin64()", CCDelegateTo<CC_X86_Win64_C>>, // Otherwise, drop to normal X86-64 CC CCDelegateTo<CC_X86_64_C>]>; |
然后我们转到中端，向头文件**/llvm/include/llvm/IR/CallingConv.h**中注册这个调用约定的存在，这样我们能够在Pass中使用这个调用约定。添加结果如下:
1 2 3 4 5 6 7 8 9 10 11 | namespace CallingConv {enum { C = 0, Fast = 8, Cold = 9, ....... AArch64_SME_ABI_Support_Routines_PreserveMost_From_X2 = 103, X86_64_Obfu1 = 104, // 新增的调用约定。 MaxID = 1023};} |
重新编译代码你就可以使用这个新增的调用约定。
0x3 编写Pass修改函数的调用约定
通过编写中端Pass，我们修改所有函数的调用约定为新增的调用约定。代码如下所示
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | #include "llvm/Transforms/Obfuscation/CustomCC.h"#include "llvm/IR/BasicBlock.h"#include "llvm/IR/CFG.h"#include "llvm/IR/Function.h"#include "llvm/IR/IRBuilder.h"#include "llvm/IR/Instruction.h"#include "llvm/Pass.h"#include "llvm/Passes/PassBuilder.h"#include "llvm/Transforms/Obfuscation/Utils.h"using namespace llvm;namespace polaris {PreservedAnalyses CustomCC::run(Module &M, ModuleAnalysisManager &AM) { for (Function &F : M) { if (readAnnotate(F).find("customcc") != std::string::npos) { F.setCallingConv(CallingConv::X86_64_Obfu1); // 修改函数调用约定 FixInstrCallingConv(M, F, CallingConv::X86_64_Obfu1); // 修改函数调用指令的调用约定 } } return PreservedAnalyses::none();}void CustomCC::FixInstrCallingConv(Module &M, Function &Target, CallingConv::ID CC) { std::vector<CallInst *> CIs; for (Function &F : M) { for (BasicBlock &BB : F) { for (Instruction &I : BB) { if (isa<CallInst>(I)) { // 收集所有的CallInst进行处理 CallInst *CI = (CallInst *)&I; Function *Func = CI->getCalledFunction(); if (Func && Func == &Target) { // 检查被调用的函数是否是被修改了调用约定的函数 CIs.push_back(CI); } } } } } for (CallInst *CI : CIs) { CI->setCallingConv(CC); // 修改这条指令的调用约定以匹配目标函数 }}} // namespace polaris |
下面是效果，调用函数的代码片段：
下面是被调用函数的效果，参数通过rax和rbx传递：
0x4 结语
通过添加大量的自定义调用约定，能够使IDA的反编译结果出现问题(虽然效果不大，也能被恢复)。但是如果每个函数的调用都不一样的话，恢复起来就比较麻烦了。
具体的代码可以查询，其中还包含了一些其他的混淆：https://github.com/za233/Polaris-Obfuscator
可以直接编译使用，通过开启ccc选项和添加注释即可。
mb_kqntebgh