Module ida_merge

Merge functionality.

NOTE: this functionality is available in IDA Teams (not IDA Pro)

There are 3 databases involved in merging: base_idb, local_db, and remote_idb. * base_idb: the common base ancestor of 'local_db' and 'remote_db'. in the UI this database is located in the middle. * local_idb: local database that will contain the result of the merging. in the UI this database is located on the left. * remote_idb: remote database that will merge into local_idb. It may reside locally on the current computer, despite its name. in the UI this database is located on the right. base_idb and remote_idb are opened for reading only. base_idb may be absent, in this case a 2-way merging is performed.

Conflicts can be resolved automatically or interactively. The automatic resolving scores the conflicting blocks and takes the better one. The interactive resolving displays the full rendered contents side by side, and expects the user to select the better side for each conflict.

Since IDB files contain various kinds of information, there are many merging phases. The entire list can be found in merge.cpp. Below are just some selected examples: * merge global database settings (inf and other global vars) * merge segmentation and changes to the database bytes * merge various lists: exports, imports, loaded tils, etc * merge names, functions, function frames * merge debugger settings, breakpoints * merge struct/enum views * merge local type libraries * merge the disassembly items (i.e. the segment contents) this includes operand types, code/data separation, etc * merge plugin specific info like decompiler types, dwarf mappings, etc

To unify UI elements of each merge phase, we use merger views: * A view that consists of 2 or 3 panes: left (local_idb) and right (remote_idb). The common base is in the middle, if present. * Rendering of the panes depends on the phase, different phases show different contents. * The conflicts are highlighted by a colored background. Also, the detail pane can be consulted for additional info. * The user can select a conflict (or a bunch of conflicts) and say "use this block". * The user can browse the panes as he wishes. He will not be forced to handle conflicts in any particular order. However, once he finishes working with a merge handler and proceeds to the next one, he cannot go back. * Scrolling the left pane will synchronously scroll the right pane and vice versa. * There are the navigation commands like "go to the prev/next conflict" * The number of remaining conflicts to resolve is printed in the "Progress" chooser. * The user may manually modify local database inside the merger view. For that he may use the regular hotkeys. However, editing the database may lead to new conflicts, so we better restrict the available actions to some reasonable minimum. Currently, this is not implemented.

IDA works in a new "merge" mode during merging. In this mode most events are not generated. We forbid them to reduce the risk that a rogue third-party plugin that is not aware of the "merge" mode would spoil something.

For example, normally renaming a function causes a cascade of events and may lead to other database modifications. Some of them may be desired, some - not. Since there are some undesired events, it is better to stop generating them. However, some events are required to render the disassembly listing. For example, ev_ana_insn, av_out_insn. This is why some events are still generated in the "merge" mode.

To let processor modules and plugins merge their data, we introduce a new event: ev_create_merge_handlers. It is generated immediately after opening all three idbs. The interested modules should react to this event by creating new merge handlers, if they need them.

While the kernel can create arbitrary merge handlers, modules can create only the standard ones returned by:

create_nodeval_merge_handler() create_nodeval_merge_handlers() create_std_modmerge_handlers()

We do not document merge_handler_t because once a merge handler is created, it is used exclusively by the kernel.

See mergemod.hpp for more information about the merge mode for modules.

Global variables

var MERGE_KIND_AFLAGS_EA

merge aflags for mapped EA

var MERGE_KIND_AUTOQ

auto queues

var MERGE_KIND_BOOKMARKS

merge bookmarks

var MERGE_KIND_BPTS

merge breakpoints

var MERGE_KIND_BYTEVAL

merge byte values

var MERGE_KIND_CREFS

merge crefs

var MERGE_KIND_CUSTDATA

merge custom data type and formats

var MERGE_KIND_DBG_MEMREGS

manual memory regions (debugger)

var MERGE_KIND_DEBUGGER

debugger data

var MERGE_KIND_DEKSTOPS

dekstops

var MERGE_KIND_DIRTREE

merge std dirtrees

var MERGE_KIND_DREFS

merge drefs

var MERGE_KIND_ENCODINGS

merge encodings

var MERGE_KIND_ENCODINGS2

merge default encodings

var MERGE_KIND_END

insert to the end of handler list, valid for merge_handler_params_t::insert_after

var MERGE_KIND_ENUMS

merge enums

var MERGE_KIND_EXPORTS

merge exports

var MERGE_KIND_EXTRACMT

merge extra next or prev lines

var MERGE_KIND_FILEREGIONS

merge fileregions

var MERGE_KIND_FIXUPS

merge fixups

var MERGE_KIND_FLAGS

merge flags64_t

var MERGE_KIND_FLOWS

merge flows

var MERGE_KIND_FRAME

merge function frame info (frame members)

var MERGE_KIND_FRAMEMGR

merge frames (globally: add/delete frames entirely)

var MERGE_KIND_FUNC

merge func info

var MERGE_KIND_GHSTRCMT

merge ghost structure comment

var MERGE_KIND_HIDDENRANGES

merge hidden ranges

var MERGE_KIND_IGNOREMICRO

IM ("$ ignore micro") flags.

var MERGE_KIND_IMPORTS

merge imports

var MERGE_KIND_INF

merge the inf variable (global settings)

var MERGE_KIND_LAST

last predefined merge handler type. please note that there can be more merge handler types, registered by plugins and processor modules.

var MERGE_KIND_LOADER

loader data

var MERGE_KIND_LUMINA

lumina function metadata

var MERGE_KIND_MAPPING

merge manual memory mapping

var MERGE_KIND_NETNODE

netnode (no merging, to be used in idbunits)

var MERGE_KIND_NOTEPAD

notepad

var MERGE_KIND_ORPHANS

merge orphan bytes

var MERGE_KIND_PATCHES

merge patched bytes

var MERGE_KIND_PROBLEMS

problems

var MERGE_KIND_SCRIPTS

merge scripts

var MERGE_KIND_SCRIPTS2

merge scripts common info

var MERGE_KIND_SEGGRPS

merge segment groups

var MERGE_KIND_SEGMENTS

merge segments

var MERGE_KIND_SEGREGS

merge segment registers

var MERGE_KIND_SELECTORS

merge selectors

var MERGE_KIND_SIGNATURES

signatures

var MERGE_KIND_SOURCEFILES

merge source files ranges

var MERGE_KIND_STKPNTS

merge SP change points

var MERGE_KIND_STRMEM

merge struct members

var MERGE_KIND_STRMEMCMT

merge member comments for ghost struc

var MERGE_KIND_STRUCTS

merge structs (globally: add/delete structs entirely)

var MERGE_KIND_STT

merge flag storage types

var MERGE_KIND_TILS

merge type libraries

var MERGE_KIND_TINFO

merge tinfo

var MERGE_KIND_TRYBLKS

merge try blocks

var MERGE_KIND_UDTMEM

merge UDT members (local types)

var MERGE_KIND_UI

UI.

var MERGE_KIND_VFTABLES

merge vftables

var MERGE_KIND_WATCHPOINTS

merge watchpoints

var MH_LISTEN

merge handler will receive merge events

var MH_TERSE

do not display equal lines in the merge results table

var MH_UI_CHAR_MASK

7-bit ASCII split character

var MH_UI_COLONNAME

ida will split the diffpos name by ':' to create chooser columns

var MH_UI_COMMANAME

ida will split the diffpos name by ',' to create chooser columns

var MH_UI_COMPLEX

diffpos details won't be displayed in the diffpos chooser

var MH_UI_DP_NOLINEDIFF

Detail pane: do not show differences inside the line.

var MH_UI_DP_SHORTNAME

Detail pane: use the first part of a complex diffpos name as the tree node name.

var MH_UI_INDENT

preserve indent for diffpos name in diffpos chooser

var MH_UI_NODETAILS

ida will not show the diffpos details

var MH_UI_SPLITNAME

ida will split the diffpos name by 7-bit ASCII char to create chooser columns

var NDS_BLOB

stored as netnode blobs

var NDS_EV_FUNC

enable default handling of mev_added_func/mev_deleting_func

var NDS_EV_RANGE

enable default handling of mev_modified_ranges, mev_deleting_segm

var NDS_INC

stored value is incremented (scalars only)

var NDS_IS_BOOL

boolean value

var NDS_IS_EA

EA value.

var NDS_IS_RELATIVE

value is relative to index (stored as delta)

var NDS_IS_STR

string value

var NDS_MAP_IDX

apply ea2node() to index (==NETMAP_IDX)

var NDS_MAP_VAL

apply ea2node() to value. Along with NDS_INC it gives effect of NETMAP_VAL, examples: altval_ea : NDS_MAP_IDX charval : NDS_VAL8 charval_ea: NDS_MAP_IDX|NDS_VAL8 eaget : NDS_MAP_IDX|NDS_MAP_VAL|NDS_INC

var NDS_SUPVAL

stored as netnode supvals (not scalar)

var NDS_UI_ND

UI: no need to show diffpos detail pane, MH_UI_NODETAILS, make sense if merge_node_helper_t is used

var NDS_VAL8

use 8-bit values (==NETMAP_V8)

Functions

def create_nodeval_merge_handler(mhp: merge_handler_params_t, label: char const *, nodename: char const *, tag: uchar, nds_flags: uint32, node_helper: merge_node_helper_t = None, skip_empty_nodes: bool = True)

create_nodeval_merge_handler(mhp, label, nodename, tag, nds_flags, node_helper=None, skip_empty_nodes=True) -> merge_handler_t Create a merge handler for netnode scalar/string values

@param mhp: (C++: const merge_handler_params_t &) merging parameters @param label: (C++: const char ) handler short name (to be be appended to mhp.label) @param nodename: (C++: const char ) netnode name @param tag: (C++: uchar) a tag used to access values in the netnode @param nds_flags: (C++: uint32) netnode value attributes (a combination of nds_flags_t) @param node_helper: merge_node_helper_t * @param skip_empty_nodes: (C++: bool) do not create handler in case of empty netnode @return: diff source object (normally should be attahced to a merge handler)

def create_nodeval_merge_handlers(out: merge_handlers_t *, mhp: merge_handler_params_t, nodename: char const *, valdesc: merge_node_info_t, skip_empty_nodes: bool = True)

create_nodeval_merge_handlers(out, mhp, nodename, valdesc, skip_empty_nodes=True) Create a serie of merge handlers for netnode scalar/string values (call create_nodeval_merge_handler() for each member of VALDESC)

@param out: (C++: merge_handlers_t ) [out] created handlers will be placed here @param mhp: (C++: const merge_handler_params_t &) merging parameters @param nodename: (C++: const char ) netnode name @param valdesc: (C++: const merge_node_info_t *) array of handler descriptions @param skip_empty_nodes: (C++: bool) do not create handlers for empty netnodes @return: diff source object (normally should be attahced to a merge handler)

def destroy_moddata_merge_handlers(data_id: int)

destroy_moddata_merge_handlers(data_id)

@param data_id: int

def get_ea_diffpos_name(ea: ea_t)

get_ea_diffpos_name(ea) -> str Get nice name for EA diffpos

@param ea: (C++: ea_t) diffpos @note @see: get_nice_colored_name

def is_diff_merge_mode() ‑> bool

is_diff_merge_mode() -> bool Return TRUE if IDA is running in diff mode (MERGE_POLICY_MDIFF/MERGE_POLICY_VDIFF)

Classes

class item_block_locator_t

Proxy of C++ merge_data_t::item_block_locator_t class.

init(self) -> item_block_locator_t

@param self: PyObject *

Instance variables

var thisown

The membership flag

Methods

def get_block_head(self, md: merge_data_t, idx: diff_source_idx_t, item_head: ea_t)

get_block_head(self, md, idx, item_head) -> ea_t

@param md: merge_data_t & @param idx: diff_source_idx_t @param item_head: ea_t

def setup_blocks(self, md: merge_data_t, _from: diff_source_idx_t, to: diff_source_idx_t, region: diff_range_t const &)

setup_blocks(self, md, _from, to, region) -> bool

@param md: merge_data_t & @param from: diff_source_idx_t @param to: diff_source_idx_t @param region: diff_range_t const &

class merge_data_t (*args, **kwargs)

Proxy of C++ merge_data_t class.

Instance variables

var dbctx_ids : int [3]

local, remote, base ids

var ev_handlers : merge_handlers_t

event handlers

var item_block_locator : merge_data_t::item_block_locator_t *

item_block_locator

var last_udt_related_merger : merge_handler_t *

last_udt_related_merger

var nbases : int

number of database participating in merge process, maybe 2 or 3

var thisown

The membership flag

Methods

def add_event_handler(self, handler: merge_handler_t *)

add_event_handler(self, handler)

@param handler: merge_handler_t *

def base_id(self) ‑> int

base_id(self) -> int

def compare_merging_tifs(self, tif1: tinfo_t, diffidx1: diff_source_idx_t, tif2: tinfo_t, diffidx2: diff_source_idx_t)

compare_merging_tifs(self, tif1, diffidx1, tif2, diffidx2) -> int compare types from two databases

@param tif1: (C++: const tinfo_t &) type @param diffidx1: (C++: diff_source_idx_t) database index, diff_source_idx_t @param tif2: (C++: const tinfo_t &) type @param diffidx2: (C++: diff_source_idx_t) database index, diff_source_idx_t @return: -1, 0, 1

def get_block_head(self, idx: diff_source_idx_t, item_head: ea_t)

get_block_head(self, idx, item_head) -> ea_t

@param idx: diff_source_idx_t @param item_head: ea_t

def has_existing_node(self, nodename: char const *)

has_existing_node(self, nodename) -> bool check that node exists in any of databases

@param nodename: (C++: const char *) char const *

def local_id(self) ‑> int

local_id(self) -> int

def map_privrange_id(self, tid: tid_t *, ea: ea_t, _from: diff_source_idx_t, to: diff_source_idx_t, strict: bool = True)

map_privrange_id(self, tid, ea, _from, to, strict=True) -> bool map IDs of structures, enumerations and their members

@param tid: (C++: tid_t *) item ID in TO database @param ea: (C++: ea_t) item ID to find counterpart @param from: (C++: diff_source_idx_t) source database index, diff_source_idx_t @param to: (C++: diff_source_idx_t) destination database index, diff_source_idx_t @param strict: (C++: bool) raise interr if could not map @return: success

def map_tinfo(self, tif: tinfo_t, _from: diff_source_idx_t, to: diff_source_idx_t, strict: bool = True)

map_tinfo(self, tif, _from, to, strict=True) -> bool migrate type, replaces type references into FROM database to references into TO database

@param tif: (C++: tinfo_t *) type to migrate, will be cleared in case of fail @param from: (C++: diff_source_idx_t) source database index, diff_source_idx_t @param to: (C++: diff_source_idx_t) destination database index, diff_source_idx_t @param strict: (C++: bool) raise interr if could not map @return: success

def remote_id(self) ‑> int

remote_id(self) -> int

def remove_event_handler(self, handler: merge_handler_t *)

remove_event_handler(self, handler)

@param handler: merge_handler_t *

def set_dbctx_ids(self, local: int, remote: int, base: int)

set_dbctx_ids(self, local, remote, base)

@param local: int @param remote: int @param base: int

def setup_blocks(self, dst_idx: diff_source_idx_t, src_idx: diff_source_idx_t, region: diff_range_t const &)

setup_blocks(self, dst_idx, src_idx, region) -> bool

@param dst_idx: diff_source_idx_t @param src_idx: diff_source_idx_t @param region: diff_range_t const &

class merge_handler_params_t (_md: merge_data_t, _label: qstring const &, _kind: merge_kind_t, _insert_after: merge_kind_t, _mh_flags: uint32)

Proxy of C++ merge_handler_params_t class.

init(self, _md, _label, _kind, _insert_after, _mh_flags) -> merge_handler_params_t

@param _md: merge_data_t & @param _label: qstring const & @param _kind: enum merge_kind_t @param _insert_after: enum merge_kind_t @param _mh_flags: uint32

Instance variables

var insert_after : merge_kind_t

desired position inside 'handlers' merge_kind_t

var kind : merge_kind_t

merge handler kind merge_kind_t

var label : qstring

label

var md : merge_data_t &

md

var mh_flags : uint32

mh_flags

var thisown

The membership flag

Methods

def ui_complex_details(self, *args) ‑> bool

ui_complex_details(self, _mh_flags) -> bool Do not display the diffpos details in the chooser. For example, the MERGE_KIND_SCRIPTS handler puts the script body as the diffpos detail. It would not be great to show them as part of the chooser.

@param _mh_flags: (C++: uint32)

ui_complex_details(self) -> bool

def ui_complex_name(self, *args) ‑> bool

ui_complex_name(self, _mh_flags) -> bool It customary to create long diffpos names having many components that are separated by any 7-bit ASCII character (besides of '\0'). In this case it is possible to instruct IDA to use this separator to create a multi-column chooser. For example the MERGE_KIND_ENUMS handler has the following diffpos name: enum_1,enum_2 If MH_UI_COMMANAME is specified, IDA will create 2 columns for these names.

@param _mh_flags: (C++: uint32)

ui_complex_name(self) -> bool

def ui_dp_shortname(self, *args) ‑> bool

ui_dp_shortname(self, _mh_flags) -> bool The detail pane shows the diffpos details for the current diffpos range as a tree-like view. In this pane the diffpos names are used as tree node names and the diffpos details as their children. Sometimes, for complex diffpos names, the first part of the name looks better than the entire name. For example, the MERGE_KIND_SEGMENTS handler has the following diffpos name: ,,, if MH_UI_DP_SHORTNAME is specified, IDA will use as a tree node name

@param _mh_flags: (C++: uint32)

ui_dp_shortname(self) -> bool

def ui_has_details(self, *args) ‑> bool

ui_has_details(self, _mh_flags) -> bool Should IDA display the diffpos detail pane?

@param _mh_flags: (C++: uint32)

ui_has_details(self) -> bool

def ui_indent(self, *args) ‑> bool

ui_indent(self, _mh_flags) -> bool In the ordinary situation the spaces from the both sides of diffpos name are trimmed. Use this UI hint to preserve the leading spaces.

@param _mh_flags: (C++: uint32)

ui_indent(self) -> bool

def ui_linediff(self, *args) ‑> bool

ui_linediff(self, _mh_flags) -> bool In detail pane IDA shows difference between diffpos details. IDA marks added or deleted detail by color. In the modified detail the changes are marked. Use this UI hint if you do not want to show the differences inside detail.

@param _mh_flags: (C++: uint32)

ui_linediff(self) -> bool

def ui_split_char(self, *args)

ui_split_char(self, _mh_flags) -> char

@param _mh_flags: uint32

ui_split_char(self) -> char

def ui_split_str(self, *args)

ui_split_str(self, _mh_flags) -> qstring

@param _mh_flags: uint32

ui_split_str(self) -> qstring

class merge_node_helper_t

Proxy of C++ merge_node_helper_t class.

init(self) -> merge_node_helper_t

@param self: PyObject *

Static methods

def append_eavec(s: qstring *, prefix: char const *, eas: eavec_t const &)

append_eavec(s, prefix, eas) can be used by derived classes

@param s: (C++: qstring ) @param prefix: (C++: const char ) char const * @param eas: (C++: const eavec_t &) eavec_t const &

Instance variables

var thisown

The membership flag

Methods

def get_column_headers(self, arg0: qstrvec_t *, arg1: uchar, arg2: void *)

get_column_headers(self, arg0, arg1, arg2) get column headers for chooser (to be used in linear_diff_source_t::get_column_headers)

@param arg0: qstrvec_t * @param arg1: uchar @param arg2: void *

def get_netnode(self)

get_netnode(self) -> netnode return netnode to be used as source. If this function returns BADNODE netnode will be created using netnode name passed to create_nodeval_diff_source

def is_mergeable(self, arg0: uchar, arg1: nodeidx_t)

is_mergeable(self, arg0, arg1) -> bool

@param filter: check if we should perform merging for given record @param arg1: nodeidx_t

def map_scalar(self, arg0: nodeidx_t *, arg1: void *, arg2: diff_source_idx_t, arg3: diff_source_idx_t)

map_scalar(self, arg0, arg1, arg2, arg3) map scalar/string/buffered value

@param arg0: nodeidx_t * @param arg1: void * @param arg2: diff_source_idx_t @param arg3: diff_source_idx_t

def map_string(self, arg0: qstring *, arg1: void *, arg2: diff_source_idx_t, arg3: diff_source_idx_t)

map_string(self, arg0, arg1, arg2, arg3)

@param arg0: qstring * @param arg1: void * @param arg2: diff_source_idx_t @param arg3: diff_source_idx_t

def print_entry_details(self, arg0: qstrvec_t *, arg1: uchar, arg2: nodeidx_t, arg3: void *)

print_entry_details(self, arg0, arg1, arg2, arg3) print the details of the specified entry usually contains multiple lines, one for each attribute or detail. (to be used in print_diffpos_details)

@param arg0: qstrvec_t * @param arg1: uchar @param arg2: nodeidx_t @param arg3: void *

def print_entry_name(self, arg0: uchar, arg1: nodeidx_t, arg2: void *)

print_entry_name(self, arg0, arg1, arg2) -> qstring print the name of the specified entry (to be used in print_diffpos_name)

@param arg0: uchar @param arg1: nodeidx_t @param arg2: void *

def refresh(self, arg0: uchar, arg1: void *)

refresh(self, arg0, arg1) notify helper that some data was changed in the database and internal structures (e.g. caches) should be refreshed

@param arg0: uchar @param arg1: void *

class merge_node_info_t (name: char const *, tag: uchar, nds_flags: uint32, node_helper: merge_node_helper_t = None)

Proxy of C++ merge_node_info2_t class.

init(self, name, tag, nds_flags, node_helper=None) -> merge_node_info_t

@param name: char const * @param tag: uchar @param nds_flags: uint32 @param node_helper: merge_node_helper_t *

Instance variables

var name : char const *

name of the array (label)

var nds_flags : uint32

node value attributes (a combination of nds_flags_t)

var node_helper : merge_node_helper_t *

node_helper

var tag : uchar

a tag used to access values in the netnode

var thisown

The membership flag

class moddata_diff_helper_t (_module_name: char const *, _netnode_name: char const *, _fields: idbattr_info_t)

Proxy of C++ moddata_diff_helper_t class.

init(self, _module_name, _netnode_name, _fields) -> moddata_diff_helper_t

@param _module_name: char const * @param _netnode_name: char const * @param _fields: idbattr_info_t const *

Instance variables

var additional_mh_flags : uint32

additional merge handler flags

var fields : idbattr_info_t const *

module data attribute descriptions

var module_name : char const *

will be used as a prefix for field desc

var netnode_name : char const *

name of netnode with module data attributes

var nfields : size_t

number of descriptions

var thisown

The membership flag

Methods

def get_struc_ptr(self, arg0: merge_data_t, arg1: diff_source_idx_t, arg2: idbattr_info_t)

get_struc_ptr(self, arg0, arg1, arg2) -> void *

@param arg0: merge_data_t & @param arg1: diff_source_idx_t @param arg2: idbattr_info_t const &

def merge_ending(self, arg0: diff_source_idx_t, arg1: void *)

merge_ending(self, arg0, arg1)

@param arg0: diff_source_idx_t @param arg1: void *

def merge_starting(self, arg0: diff_source_idx_t, arg1: void *)

merge_starting(self, arg0, arg1)

@param arg0: diff_source_idx_t @param arg1: void *

def print_diffpos_details(self, arg0: qstrvec_t *, arg1: idbattr_info_t)

print_diffpos_details(self, arg0, arg1)

@param arg0: qstrvec_t * @param arg1: idbattr_info_t const &

def str2val(self, arg0: uint64 *, arg1: idbattr_info_t, arg2: char const *)

str2val(self, arg0, arg1, arg2) -> bool

@param arg0: uint64 * @param arg1: idbattr_info_t const & @param arg2: char const *

def val2str(self, arg0: qstring *, arg1: idbattr_info_t, arg2: uint64)

val2str(self, arg0, arg1, arg2) -> bool

@param arg0: qstring * @param arg1: idbattr_info_t const & @param arg2: uint64