The Evolution of macOS Gatekeeper

The history of Mac endpoint security is like the history of Apple itself: a sprawling tale full of elegant innovations and memorable missteps. That story is too big for a single blog article (or even a single book), but within the larger security narrative, there’s a fascinating tale about one of its key components: Gatekeeper.

Gatekeeper is the central component of a Modern Mac’s famous “scan at launch” style of security which includes technologies like File Quarantine and XProtect. It’s a technology that feels uniquely Apple: simple, (mostly) effective, and at times feels nearly invisible to users.

To understand Gatekeeper is to understand Apple’s attitudes about security and IT. So let’s take a look at where Gatekeeper came from, where it’s going, and the role it plays in guarding the door to tech’s most fiercely-defended walled garden.

Part 1: “Macs Don’t Get PC Viruses”

In 2005, Apple was riding high with the release of OS X Tiger. With Steve Jobs’ return, they had avoided financial ruin and successfully transitioned to a new, modern OS. They also had their first blockbuster product since the Mac–the iPod. The irresistible (and expertly marketed) music player had a halo effect on Macs, prompting more and more users to switch over from their PCs.

At that time, Macs were coasting on the strong security story they’d inherited from their Unix-based OS, which was primarily based on Jobs’ previous venture at NeXT.

That year, Steve Jobs gave a revealing interview that touched on Mac’s relationship to security. Jobs acknowledged an internal battle over forcing users to authenticate in order to install applications.

“Avie Tevanian, the person that was running software at the time, showed us OS X and every time you wanted to load an application into OS X, whether it was off the internet or even off a disc, you had to type your name and password–you had to authenticate. And we gave him incredible shit for that. We said ‘Avie, are you nuts? This is the Mac!’ And he said, ‘trust me.’ And so we deferred to Avie on that after trying to twist his arm for a year. And boy, was he ahead of his time.”

It was a battle that security ultimately won, despite the added user friction.

Still, though Jobs claimed to be proud of Mac security, he wasn’t ready to make it a pillar of Apple’s messaging. “We don’t market this because that’s like the red cape and the bull,” he said, with the “bull” being hackers. Hubris aside, 2005 Jobs seemed to think trumpeting security would be bad form. “I don’t think it should be used as a competitive weapon,” he claimed.

Nevertheless, by 2006 Apple did exactly that. One of its first “Get a Mac” commercials was a shot across the bow at PC security, and implied that Macs were immune to PC viruses.

What those commercials didn’t acknowledge (but Jobs clearly knew) was that Macs weren’t impervious to viruses; they’d simply escaped the attention of hackers by virtue of their comparatively small user base. But with this ad, they’d waved the red cape at the bull, and the bull was about to charge.

Part 2: Introduction of File Quarantine

2007 marked the end of Mac’s security honeymoon. Suddenly, malware emerged that was written specifically for OS X, starting with the OSX/Leap-A (aka Oompa-A), which spread via iChat. Soon, Mac users started to experience the same problems with performance issues, freezes, and crashes they’d had with their PCs.

Apple’s first major action in its war with viruses was File Quarantine, which debuted in OS X Leopard (10.5). This feature attaches an extended attribute (com.apple.quarantine) to files downloaded from the internet (and later, copied using Airdrop). This tag prevents a user from executing a file until they have verified in a popup warning that they are aware of its contents.

File Quarantine attempted to take a light touch with users by letting them choose whether to proceed. Unfortunately, this approach didn’t do much to stem the tide of infected Macs for two reasons:

1. Since Macs stuck the quarantine tag on every downloaded file (not just applications), users quickly developed quarantine fatigue and grew accustomed to just breezing past the popup.

2. The warning failed to explain the potential consequences of opening applications or provide any instruction on how users could differentiate between trustworthy and untrustworthy sources.

The first issue, at least, is understandable since Apple was trying to protect users from malware disguising itself as harmless documents. As Eclectic Light explains, “[Malware] might install an innocent-looking document but set its OpenWith xattr to ensure that it’s processed (installed or run in some way) using a third-party tool instead.”

But the second problem is emblematic of a larger tendency in Mac security to withhold important information from users, presumably out of fear of bogging them down with tedious, technical information.

From a security practitioner’s perspective, however, the File Quarantine system is extremely valuable. When the warning dialog is shown to a user and the file is opened, Apple records this event in an on-disk database. Security tools like osquery can read these databases across all the Macs in your fleet giving you insight into the external files users are downloading and executing from the Internet.

Introducing XProtect

Mac’s next OS, Snow Leopard, significantly improved File Quarantine. Instead of simply issuing a blanket warning to all downloaded files, it introduced XProtect, which would compare downloaded files against a database of known malware.

While modest at first–the original list only contained a handful of Trojan horses–XProtect has since become the unsung hero of Mac’s malware fight. As we’ve written before, “XProtect updates itself automatically, silently, and separately from manually installed security updates.”

Like many of Apple’s security enhancements, Apple never mentioned the inclusion of XProtect in the Snow Leopard Release Notes or marketing materials. After Snow Leopard was released, this functionality was discovered by security vendor Intego.

XProtect was a valuable update to File Quarantine, but even if a file was deemed dangerous, all it could do was issue a more ominous-looking warning to users and advise them to move the offending file to the trash. But Apple was about to exert much stricter control over its ecosystem, wresting it from users and developers alike.

There’s an App for That

For Apple, the big news in 2007 wasn’t about Macs at all; that year, executives were preoccupied with the birth of the company’s beloved youngest child: the iPhone.

The advent of the iPhone is important to our story for two reasons:

1. When people suddenly started carrying around their private data in their back pockets, that data was exposed to vastly more risk than when it lived on a desktop or laptop. This change altered the balance of power between security and user experience in Apple’s design. It began a slow progression toward security and privacy taking center stage as Apple values instead of operating quietly in the background.

2. Hot on the heels of the iPhone came the App Store, a marketplace that invited third-party developers…if they played by Apple’s rules. (And even this level of access was apparently a tough sell for Steve Jobs.)

The App Store was an immediate success on iOS (it helped that it was the only game in town), and Apple hoped its popularity would cross over to Macs. If that theory had panned out, it would have been a coup for security since every app that users downloaded would have Apple’s seal of approval.

But, as every Mac user already knows, Apple’s dreams for the Mac App Store never came to fruition. The available apps were granted such limited permissions that users felt like they’d been stuck in the kiddy pool. And since users were already accustomed to downloading much more robust apps from the web, they had no motivation to change.

Despite the Mac App Store’s failure, it was clear the company’s bifurcated approach to security wasn’t working. Macs still needed a tool that ensured every app users downloaded met its standards, regardless of where it came from.

Part 3: The Walled Garden Gets a Gatekeeper

Apple included a sneak preview of Gatekeeper in Lion, to enable developer testing, and then rolled out the full version with Mountain Lion, in 2012.

The simplest description of Gatekeeper’s function comes from Objective See, and we won’t try to improve on it. “Gatekeeper checks the code signing information of downloaded items and blocks those that do not adhere to system policies. (For example, it checks that items are signed with a valid developer ID).”

Gatekeeper was designed to give the existing malware protections, File Quarantine and XProtect, some teeth¹. Whereas those earlier features would only recommend that users refrain from code that falls outside local system policies, Gatekeeper doesn’t give them the option to run it.

Like many of the other protections mentioned here, determined users can work around Gatekeeper’s strongly worded prompts. Luckily, like File Quarantine, tools like osquery and Kolide can list the questionable and unsigned software users ran anyway despite the warning.

Part 4: Notarization and The Future of Mac Security

The relationship between hackers and developers is an endless game of cat and mouse, in which there will never be a definitive winner, only a series of moves and countermoves.

Since introducing Gatekeeper, Apple has continued to improve its malware posture by exercising more and more control. In 2019, they introduced Application Notarization requirements, which ensure that no piece of software can run unless it has been scanned and approved.

In Ventura, Gatekeeper’s power was enhanced to check that apps have not been wrongfully modified after installation. However, SentinelOne reports that “the Gatekeeper check here is overridable by users. When an unauthorized modification is attempted, Gatekeeper throws a warning and asks the user if they want to allow it in System Settings.” This would seem to mark a (slight) return to user control, which hackers will inevitably seek to exploit.

Closing Thoughts

It’s easy to snipe at Apple, and there’s a fine line between knee-jerk snark and good faith attempts to hold the tech giant accountable. But the reality is there are no simple solutions that resolve the tension between preserving a user’s autonomy and safeguarding their security.

So, since there will never be a silver bullet that satisfies developers, security professionals, end users, and Apple itself, the best question we can ask is: does Apple’s approach to malware actually work?

In terms of public perception of Mac security, the answer is clearly: yes. In 2020, Jamf ran a survey of IT and security professionals, in which 77% of all respondents reported that they considered Mac to have the best out-of-the-box security. And 79% of professionals at Mac-first companies “said the perceived security of macOS influenced their purchasing decision.”

But does that perception mean that Macs actually get less Malware than PCs? That’s a much harder question to answer, given their wildly disproportionate market shares, and the fact that it’s tough to find an expert who doesn’t either love Apple or love to hate it.

One thing we can all agree on is that neither Macs nor PCs are immune from malware, especially the kind that stems from good old fashioned user error. And we can probably also agree that Macs have done a better job of facing those threats while still preserving the user experience of their platform.

Still, it’s fair to say that Apple’s habit of not wanting to “burden” users with technical information has become a liability. As Eclectic Light writes in their critique of the document quarantine issue, “determining document behaviours like this through opaque metadata prevents the user from making judgements of their own on which documents to trust.” And SentinelOne blames Mac OS’ “lack of transparency” for inadvertent data breaches committed by users (and admins) without a full understanding of what they’re doing.

As Apple continues to navigate these issues, they would do well to remember the lesson hiding in their own company’s name: people don’t want to be protected from knowledge or from choice, no matter how lovely the garden is.

For more original Mac and cybersecurity content, subscribe to our bi-weekly newsletter!