When the Apple announced KEXTs deprecation and replacement for kernel-mode kAuth & MACF APIs with user-mode EndpointSecurity framework, many developers said “Wow”.
When they build apps with EndpointSecurity.framework and try to debug them, they said “Doh!”

Note: The article does NOT cover all details about EndpointSecurity framework. What it does – helps to debug applications that use it.

I’ve created the short Demo (~4 min) that shown whole process on both machines. It is in the end of the article.

Kernel Extensions (KEXT): a bit of history

kAuth (Kernel Authorization) and MACF(Mandatory Access Control framework) are mechanisms that reside in Kernel. They are designed to granulary authorize particular events that occur in the system (file events, process events, other).
For example, user of kAuth is able to subscribe to file open events and decide to allow or deny the action on exact file for a particular process.

Such mechanisms can be used by developers by writing KEXT (Kernel eXtension) – a special executable, that can be loaded into the Kernel of macOS.

Most popular products were built on top of kAuth and MACF:

• antivirus
• portable device read-only mode
• data leak protection & prevention
• user activity monitoring
• on-the-fly backup

And now (even since macOS 10.15) these mechanisms become deprecated.
Why? Because writing Kernel code:

• requires zero-bug policy. If you have a crash in your usual App – the App only is crashed.
  If you have a crash in your KEXT – whole macOS is crashed
• hard to debug. To use the debugger, you should make some magic and use 2 machines (debugger and debuggee)
• requires extra security. Having some issues in the code may allow the attacker to gain TOTAL control on the machine

EndpointSecurity framework to the rescue

EndpointSecurity.framework (ES.framework) is a convenient, user-mode based mechanism to be used instead of Kernel-level kAuth and MACF.
ES.framework is implemented in C, providing wide range of system events to deal with.

The huge benefit here is developers should not deal with Kernel anymore.

Debug: expectations and reality

When we hear of ‘user-mode’ approach, we may think: yeah, now development of security products would be much easier.
And we can finally debug it right from the project we built… but not so fast =\

ES.framework has two kind of events: NOTIFY and AUTH.

• NOTIFY event is just notification that some event already has occurred
• AUTH event is event which assumes to be responded (allow or deny)

For example, file open events occur about 20-1000 times per second in usual-working macOS.

And here comes the beasts: while the AUTH event is not responded, the OS does NOT allow corresponding action.
What we have on practice:

• our App use ES.framework. It listens file open AUTH events. And we want to debug it
• we set the breakpoint, ES.framework sends lots of file open AUTH events (also from system daemons, our IDE itself) and they hit breakpoint we set
• all the processes (including the IDE with debugger) are blocked by the OS (including our debugger/IDE) waiting the response from our app
• our app needs to open some files for debug purposes. There requests are also blocked by ourselves because of breakpoint
• whole scenario leads to kind of ‘interprocess deadlock’ resulting in whole system hang!

To deal with such situation, there are some approaches. I’ve used three:

1. Perform hard-muting of processes to limit ES events to very tiny set of processes
2. Move ES.framework client code to another process and perform events filtering there. Pass events of interest into the main app over IPC (interesting approach, but it is out of scope of this article)
3. Perform two-machine debug

Debug: Two-machine approach

Note: opposed to Kernel two-machine debug, user-mode two-machine debug is quite easy to setup and it has relaxed requirements.

How it works

In two-machine debug scenario, one machine (Host) acts like a debugger machine (where the debugger runs) and another one (Target) acts like a debuggee (where the application runs).
These machines must be visible to each other over the (local) network.
In few words, here is the scenario

• application in launched on Target machine
• debugserver is launched on Target machine too. It attaches to the application and listens on specific port for debugger commands
• on Host machine, the debugger (lldb / Xcode) is launched. It connects to the debugserver over the network
• debugger on Host machine interacts with debugserver on Target, performing real-time debug

Generally speaking, it works very similar to local debug. But in usual life we don’t face with such details of how exactly we debug.

Why it works with EndpointSecurity?

The main issue in debugging of EndpointSecurity events is the fact the machine freezes because of pending file AUTH events.
Actually, the machine and it’s processes are still running, but user interaction is blocked.

Because we keep user interaction on Host machine, even if Target machine has lots on pending AUTH events, the debugserver still running as usual, performing debug commands.

Two-machine debug

For two-machine debug we need two virtual or physical machines (unexpected, isn’t it?), visible over the network (you can ping between them).

Prepare Host machine

1. Install Xcode
2. Reveal hidden Xcode feature that allows remote debug.
   In Terminal.app, run: defaults write com.apple.dt.Xcode IDEDebuggerFeatureSetting 12
3. Copy debugserver process from /Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Versions/A/Resources/debugserver to shared location with Target machine

Prepare Target machine

1. Copy debugserver process to this machine

Debug it, finally

On Target machine, prepare for debug session

1. Place the executable (built on Host machine) you want to debug on the Target machine
2. On Target machine, attach debugserver to that application
   Important here that IP address is address of the Host machine
3. to attach to already running application, run
   ./debugserver ip:port --attach=pid
4. to attach on start, launch the application via
   ./debugserver ip:port /path/to/executable

On Host machine, perform attach-over-network.
In Xcode, adjust the Scheme you are going to debug

1. Switch Launch to ‘Custom LLDB commands’
2. Input the command process connect connect://ip:port
   Note that ip is the IP address of the Target machine
3. Press ‘Start’ button in Xcode to perform connect and debug

A piece of advice

While the technique is working, I’d like to put here some advice from my personal experience.
It could help you to perform two-machine debug more easy.

1. Consider using Virtual machine as Target
2. VM has snapshots
3. less latency Host <-> Target
4. makes whole process faster
5. Be fast
   AUTH events from EndpointSecruity framework have deadlines (in practice, its around 30-60 seconds).
   If deadline of any event is reached (the app does not respond in this time), the system will KILL your app. Which obviously terminates the debug session.

A little promotion: Swift libraries you may want to use in your macOS applications

While my career as macOS developer, I’ve faced the same problems in different projects. So I decided to create some set of libraries to gather most common things in one place. In topic of the article, this is sEndpointSecurity library which provides nice Swift wrapper around EndpointSecurity C API:

Few libraries also helps on everyday basis:

• SwiftConvenience: Swift common extensions and utilities used in everyday development
• sXPC: Swift type-safe wrapper around NSXPCConnection and proxy object
• sLaunchctl: Register and manage daemons and user-agents. Swift interface to launchctl tool
• sMock: Swift unit-test mocking framework similar to gtest/gmock