Skip to content

Stability and Compatibility Issues with QNX 6.5.0SP1 #3818

Closed

Description

@KazWolfe

I've recently been working on getting frida-core and the toolchain to run on QNX 6.5 on an ARM processor; specifically an automotive head unit (QNX localhost 6.5.0 2019/12/12-16:00:38CST TI-OMAP5432-Pasa-Ford-Sync3 armle). There's so far been a considerable amount of difficulty in getting things to work reliably, so I wanted to raise an Issue to discuss options and whether patches would be desired. I would also (ideally) like to get confirmation that these problems aren't just weird quirks of my own platform before upstreaming things. This is, unfortunately, more of a full list of (currently-discovered) issues; more specific ones can be opened if desired.

General QNX Issues

To assist in testing, I also spun up an x86 virtual machine via VMWare (QNX localhost 6.5.0 2012/06/20-13:50:50EDT x86pc x86) to validate certain reports.

  1. gumelfmodule.c mixes self->shdrs and self->sections. In cases where the executable itself cannot be read, Gum will fall back to using the in-memory copy. The loops in gum_elf_module_enumerate_sections and gum_elf_module_enumerate_relocations loop over the still-populated shdrs, but attempt to read from the now-empty sections.
  2. x86 is not a supported target. This one isn't really a flaw as much as it's just a general pain for debugging. Support in FRIDA only targets ARM (which makes sense given most QNX apps will be running there), but most QNX devkits will run on x86 hosts. In no particular order:
    1. libunwind doesn't understand QNX x86 at all.
    2. glib fails assertions for gint64 and int64_t.
    3. qinjector-glue.c doesn't have x86 support at all as well.
    4. qnx-tools only ships targets for armle and armle-v7, which is probably going to be the most painful to solve.

I've managed to get an x86 build running and passing (limited) self tests, but I'm unsure this is a desired platform in general. The code I have is probably none too pretty (and I'll admit to LLM assistance), but it is at least a proof of semi-viable concept. I also don't know what the licensing implications would be to get an x86 buildroot distributed; perhaps it'll need to just be "bring-your-own" if it's even desired?

Likewise, I have a patch for gumelfmodule.c; it's only two lines since frida/frida-gum#1210 was fixed.

ARM-Specific Issues

And things I've noticed only in ARM. Again, I don't have hardware or a VM to test whether this is ARM-general, or specific to my hardware.

  1. qinjector-glue.c doesn't handle low addresses well. put_call_address_with_arguments() will emit a pc-relative BL/BLX call if the target is in range (32MB). However, pc is not properly set to the remote_address, meaning payloads end up in weird places. This is especially notable with libc, which is loaded around 0x01000000.
  2. Problems with XIP page replacement. This one is probably the most convoluted here, and I don't really understand it fully yet either. When we attempt to write-enable an XIP'd page, mprotect() will fail with EACCES. The fallback in gum_try_mprotect() will then replace the entire page with a private copy. However, this copy is written through the data cache and doesn't get cleaned prior to thread release. On ARM, this fails since instruction fetch doesn't read from the data cache. When execution continues, the data cache is still dirty and the instruction cache is left in an inconsistent state causing unpredictable crashes.
  3. The Interceptor mangles small functions like exit. On this build, libc defines _exit as a four-byte BL followed immediately by exit proper. ExitMonitor hooks both, which causes _exit to overwrite the first instruction of exit proper.

Similar to the generic issues, I have potential patches:

  1. The low address issue can be fixed by setting cw.pc = remote_address.
  2. Block all signals while the replacement is happening and freeze threads. Clean the cache, unblock execution, and continue onwards.
  3. There are a few patches possible here, including trying to make the redirect size-aware, skipping a hook on _exit, or instead hooking __exit. None of these options are really great, so I'm not really sure which would be the "best" here.

Activity

  1. KazWolfe commented on Nov 6, 2026

    @KazWolfe
    Author

    Actually, if I can get some guidance on the exit/_exit hook issues, that'd be really helpful I think.

    Background

    frida-core's ExitMonitor currently attaches to exit, _exit, and abort. In the libc for this specific target:

    ; void __exit(int status) - 20 bytes
    __exit:
      01029088  e1a02000  mov   r2, r0
      0102908c  e3e00000  mvn   r0, #0
      01029090  e1a01000  mov   r1, r0
      01029094  ebffaef3  bl    j_ThreadDestroy
      01029098  eafffffe  b     .
    
    ; void _Exit(int status) - 4 bytes
    _Exit:
      0102909c  ebffa9c0  bl    __exit    ; via thunk
    
    ; void _exit(int status) - 4 bytes
    _exit:
      010290a0  ebffa9bf  bl    __exit    ; via thunk
    
    ; void exit(int status) - 16 bytes
    exit:
      010290a4  e1a04000  mov   r4, r0
      010290a8  ebffaa59  bl    __cleanup    ; via thunk
      010290ac  e1a00004  mov   r0, r4
      010290b0  ebffb1da  bl    _exit        ; via thunk
    

    Since the compiler decided to emit BL for _exit, Gum's ARM relocator gets a bit confused and continues onwards into exit proper, overwriting the initial mov r4, r0 with the _exit trampoline address.

    This theoretically isn't a QNX-specific flaw. Any one-instruction noreturn function followed immediately by another hooked one would risk this type of issue, especially with this strange BL instruction.

    Size-Aware Redirecting

    Gum already has a method redirect_space_hint, which (currently) only applies to write_redirect callbacks. This can be extended to expose the redirect's size, using a shorter B-only jump whenever the value is less than 8. To leverage this, frida-core can grab the function size (if present) from the symbol size table and passes that along.

    If we'd rather not reuse redirect_space_hint to avoid confusion (though I don't think it's used at all if write_redirect isn't present?), we can create a new var like redirect_max_size, but there may be semantic considerations here. This also still depends on us knowing the size, which works for libc but may not work elsewhere.

    Code Deflector Breaks

    This fix has been proven to work on hardware, but ran into another annoying problem: the code deflector. Gum seems to place this deflector in the padding/unused space at ELF Header + 8. Unfortunately, the only (discoverable) header I could find within the 32MB space around _exit is libc's own header. This means that _exit would be the only short hook that can apply as-is. This seems to affect the entire ARM platform, and is probably worth mentioning.

    However, specific to ARM QNX, we run into the same mprotect issue discussed above when trying to write the deflector. Gum falls back and maps its anonymous page over the memory holding the header, causing procfs to lose track of that page as-is, causing the entire libc ELF header to break (from procfs' perspective). The detach operation restores the 8 blank bytes, but the new page mapping remains.

    On a reinject, frida_find_library_base from qinjector-glue.c takes the first mapping that procfs reports as belonging to libc as its base address, meaning every relative call now becomes off by 0x1000. As an end result, calls to (for example) mmap will land in sigtimedwait, causing all sorts of fun problems.

    Code Deflector Fixes

    Probably the simplest way to "fix" this bug is to use gum_code_allocator_try_alloc_slice_near() like is done in standard x86 and ARM64, since that avoids touching the header and thereby forcing the binding. This doesn't, however, actually fix the underlying issue since we're just no longer using the deflector.

    If making sure the deflector works (e.g. as a fallback in case alloc_slice_near fails, or to allow otherwise tampering with low memory like this) is a priority, some other solution is probably necessary. On QNX, this would (probably) involve DCMD_PROC_MAPDEBUG (which returns the original linking address) and calculating the delta for the base, or by reading _r_debug/link_map from the process similar to how a debugger would work. There are almost certainly other solutions too, but these are at least the most obvious.

    MAPDEBUG works in theory, but feels pretty fragile as well. It assumes that the linking address remains constant, which may not hold on all operating systems or platform configurations. The _r_debug mechanism feels more robust, but we'll have to implement a lot more of QNX's internals to get that working and likely would need support of QNX-specific structs. Maybe that's fine?

    Just Hook __exit on QNX

    This is probably the simplest fix. Since __exit is 20 bytes long, it has plenty of space for a proper hook and trampoline. It's also hit from all codepaths, at least on this specific platform.

    The main downside here is that this fix only (really) would affect this specific platform. The overruns are still possible generally, and frida-core would need to be okay running after __cleanup. We're also relying on an internal libc symbol that might change out from under us, or can behave unpredictably on other systems.


    Any thoughts/opinions on how to work around this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions