KN006: Apple’s Private Cloud Compute vs. Claude Opus 5.5, etc. for Categorizing Notifications
How well Apple’s Private Cloud Compute categorizes my notifications compared with Claude Opus 5.5 and other models, measured against my own manual categorization.
Results
Important notifications interrupt me, so whether something was marked as Important or not is the most important metric.
Private Cloud Compute, Opus, and GPT-6.1 Sol got Karui’s Private Cloud Sorting prompt. So did GPT-6 Luna Decisions, OpenAI’s decision model, which returns a probability for each category. Apple’s on-device model and Qwen 3.5 4B ran through Karui’s own on-device sort.
Winner
Important notifications caught
Notifications incorrectly marked Important
Overall, Important or not
GPT-6 Luna Decisions is the most accurate overall, and it catches the most Important notifications. It clearly beats Private Cloud Compute (exact McNemar p ≈ 0.001) and GPT-6.1 Sol (p ≈ 0.02). Its lead over Opus isn’t significant (p ≈ 0.29): it catches more Important notifications Opus misses, but raises more false alarms.
I tried different cutoffs for GPT-6 Luna Decisions. Anywhere from 30% to 55% scored about the same overall, so I kept the default, 50%.
Opus almost never interrupts for something that isn’t important. Overall, Opus beats Private Cloud Compute (59 against 37, p ≈ 0.03). Private Cloud Compute catches more Important notifications, but that gap isn’t significant (35 against 23, p ≈ 0.15). Every model called the same verification code Spam, which wouldn’t happen in the app: Karui’s one-time code rule sorts codes before any model sees them.
GPT-6.1 Sol answers much like Opus. In their first runs the two disagreed on Important or not 14 times, and Opus was right on 12 (p ≈ 0.01). Taking each one’s majority answer over three runs, they disagreed 15 times and Opus was right on 10 (p ≈ 0.30), so Opus’s lead is smaller than one run suggests. Sol’s lead over Private Cloud Compute isn’t significant (56 against 44, p ≈ 0.27).
Apple’s on-device model catches as many Important notifications as Private Cloud Compute, but raises more false alarms, and overall the two are close (60 against 50 for Private Cloud Compute, p ≈ 0.39). Opus beats it clearly (68 against 36, p ≈ 0.002). Apple’s on-device model is also quick to call things Spam: 23 of them were notifications I graded Important.
Why try a model of my own: over a month of my History, Apple’s on-device model refused 48% of Karui’s requests as rate limited (KN001), and Apple engineers told me at an Apple Intelligence lab that an on-device model you bring yourself isn’t rate limited. Foundation Models lets an app plug one in, and Karui’s classifier runs on it unchanged.
Qwen 3.5 4B is about as accurate overall as Apple’s model (61 against 55, p ≈ 0.64), with the opposite trade-off: 22 false alarms against 55, but only 62 Important notifications caught against 89. For a filter, missing what matters is the worse mistake. Qwen is also slower, at a median of 6.0 seconds an answer on an M5 Mac, and 5 answers took longer than Karui’s 20-second limit. Karui’s rules were written for Apple’s model, so a prompt tuned for Qwen, or another model such as Gemma 4 E2B, may do better.
How much answers change from run to run: I ran Opus, GPT-6.1 Sol, Apple’s on-device model, and GPT-6 Luna Decisions three times each, and Qwen twice. Opus changed between Important and not on 7 of the 463 notifications, Sol on 9, Qwen on 64 in just two runs, and Apple’s on-device model on 76. GPT-6 Luna Decisions gave exactly the same answers every time, down to the probabilities. The charts show each model’s first run; Private Cloud Compute has one run so far.
Counting Neutral and Spam apart only matters with the Daily Summary on. There, Opus and GPT-6 Luna Decisions matched my grade 68% of the time, GPT-6.1 Sol and Private Cloud Compute 62%, Qwen 54%, and Apple’s on-device model 52%.
Where Private Cloud Compute and Opus both missed
Private Cloud Compute and Opus both missed the same 47 notifications I graded Important, and raised the same 8 false alarms. Of the 47, 19 are account, payment, and service notices, such as an App Store review or an invoice, and 23 are messages from people. A prompt change could probably fix many of these.
How it works
- I graded 501 notifications from my own History by hand, without seeing Karui’s answers, and left out 31 I graded Unclear and 6 decided by Karui’s hard-coded rules. Private Cloud Compute still has one more to answer, waiting on Apple’s limit to reset, so the results cover 463 for now.
- Private Cloud Compute answered through Karui’s Private Cloud Sorting with my settings, Opus through the Claude API at medium effort, and GPT-6.1 Sol through the OpenAI API at medium reasoning effort. GPT-6 Luna Decisions answered through OpenAI’s Decisions API, by way of Vercel’s AI Gateway, with one question: choose Important, Neutral, or Spam.
- For Karui’s own sort, karui-eval runs Karui’s classifier itself, with no calendar events and nobody I’ve written to. Qwen 3.5 4B is Ollama’s qwen3.5:4b on the Mac, with thinking off, plugged into Foundation Models as a model of its own, so nothing left the Mac.
- Apple limits Private Cloud Compute to about 75 to 110 answers a sitting, so this test took days.
For this test I sent my own notifications to Anthropic and OpenAI, some of them through Vercel’s AI Gateway with zero data retention on. Karui itself does only with Karui Cloud Sorting, an invite-only option that sends notifications through Karui’s server to GPT-6 Luna Decisions. Otherwise it uses only Apple’s on-device model and, with Private Cloud Sorting on, Private Cloud Compute (KN004). The tool, karui-eval, is in Karui for Mac’s source.
Show the prompt
Private Cloud Compute, Opus, and GPT-6.1 Sol got this same text, in English, as one message, and GPT-6 Luna Decisions got it as its input; Apple’s on-device model and Qwen got the same rules as the session’s instructions. {{policy}} stands for my settings, such as my important contacts and apps and my urgent and silence terms, which I’ve left out here. {{notification}} stands for the notification. It’s also in the source.
Classify one notification as Important, Neutral, or Spam.
First write the reason, then choose the category the reason supports. When the reason says nothing is needed from the user, choose Neutral, unless the Sender is an important contact, it is a security alert, or it matches the user's urgent terms.
Only connect a notification to one of the user's calendar events when it names that event or matches its place or time. When it doesn't say which event, don't guess, and don't mention the user's events in the reason.
Important:
- The Sender matches an important contact by name, email, or phone. Merely mentioning an important person's name in the title or body does not count.
- A question or request written by a person directly to the user that needs their reply or decision.
- A person writes to the user and addresses them by one of the user's names. Marketing, newsletters, and automated messages that insert the user's name do not count.
- Security alerts about the user's own accounts, such as a new sign-in, a password change, or fraud.
- A reminder about an appointment or reservation the user has already booked, happening on the day the notification was received or the next day.
- A change to the user's own travel, such as a delay, cancellation, or gate change.
- A change to the user's plans happening soon, such as someone cancelling, running late, or moving the time.
- Something arriving now or waiting for the user, such as a ride pulling up, food at the door, or a courier outside.
- The app marked the notification time sensitive, unless it is marketing, a sale, or a promotion.
- Anything matching the user's urgent terms.
- A direct message to the user from a person in an important app, unless it is only a reaction or it is automated.
- When the source is an important app and you are unsure between Important and Neutral, choose Important.
Neutral:
- A message addressed to someone else, such as "Addressed to: @Bob, not the user", even when it asks a question or makes a request. It is for that person, not the user.
- Reactions such as "laughed at", "loved", "liked", or "emphasized", and casual group chat that asks nothing of the user.
- A question put to a whole channel or group, such as "Has anyone been here?" or "Does anybody know…?". It is not written directly to the user, unless it names or @mentions them.
- Someone confirming plans or saying they'll attend, such as "See you there", when it asks nothing and changes nothing.
- Messages generated by a system, bot, or service, such as receipts, digests, shipping and delivery updates that don't need the user right now, and status updates. Quoted or forwarded content inside a person's message does not make the message automated.
- Anything matching the user's summary terms.
- Everything that is not clearly Important or Spam.
Spam:
- The app or carrier labels the sender or message as spam, such as "Suspected spam", "Spam Risk", "Scam Likely", or "Potential fraud", unless the Sender matches an important contact.
- Messages matching the user's silence terms, and promotions when the user's policy says so, unless their Sender matches an important contact, even from important apps.
- Offers, sales, and deadlines to buy or sign up are promotions, not reminders.
Examples:
- A person asks "Can you pick up milk on your way home?": Important, a request that needs the user's reply.
- The user's names include "Aya", and a person writes "Aya, are you still coming tonight?": Important, written to the user by name.
- The user's names include "Aya", and a store writes "Aya, your exclusive offer ends tonight": a promotion that uses the user's name, not a personal message.
- Subtitle "Family", body "Ken laughed at “see you soon”": Neutral, a reaction in a group chat.
- Source Slack, body "@Bob can you send me that report?", addressed to @Bob, not the user: Neutral, a request to Bob.
- Received Tuesday, body "Reminder: your dentist appointment is Wednesday at 10:00": Important, a booked appointment the next day.
- The user has "Team all-hands 19:00" today, and a friend writes "Yes, I plan on attending as well": Neutral, "Confirms they'll attend an event; the message doesn't say which, so it isn't linked to the user's calendar."
- The user has "Book club 19:00" today, and a friend writes "Running 20 minutes late, start without me": Important, a change to today's plans.
- "Last chance! 30% off ends tomorrow": a promotion, not a reminder. Spam when the user silences promotions, otherwise Neutral.
- "Your package will arrive tomorrow": Neutral, a routine delivery update.
- Marked time sensitive, "Your driver is arriving. Jordan is 2 minutes away": Important, a ride arriving now.
- Marked time sensitive, "Flash sale ends in 1 hour!": a promotion, so the time sensitive mark doesn't make it Important.
- Sender "Suspected spam", title "Missed call from (555) 010-0199": Spam, the app labeled it spam.
Follow the user's policy conservatively. Never invent context, and never call something urgent unless the text says so.
Also rate interest from 1 through 5, independent of the category, using the whole range:
5: a real person wrote to the user personally, including a reply in a thread the user is part of (RE:), or it needs the user's decision or reply.
4: specific news about the user's own plans, orders, bills, accounts, or work.
3: a group conversation, community event, or update the user follows but need not act on.
2: routine automated status, such as delivered, synced, or completed, and newsletters the user subscribed to.
1: marketing, promotions, and generic broadcasts, even when worded as a personal invitation or sent under a person's name.
Write the reason in English.
Also look for a deadline: something the notification asks the user to do by a certain day.
task: what the user must do, at most 8 words, such as "Pay invoice #2041". Leave it empty when nothing is asked of the user by a certain day.
dueDate: that day as YYYY-MM-DD, taken from the Dates line, only when the notification itself names the day: a date, a weekday, "tomorrow", "end of the month". Leave it empty when there is no task or no day is named. Never guess a day from words like "soon", "later", or "before I leave".
Only these count, each something the user must do themselves by the day named: pay a bill, invoice, or fee; reply or RSVP when someone asks the user directly; submit, sign, or send a form, document, or application; renew something of the user's that would otherwise lapse, such as a membership, license, passport, or registration, or cancel a free trial before it starts charging; return something borrowed or rented, or an order within its return window.
None of these is a task: a sale, offer, or promotion that ends on a day; an event, appointment, flight, or delivery the user attends or waits for, which the calendar or the app already reminds them of; a confirmation or receipt saying something is done, scheduled, or paid; an automatic payment or renewal that needs nothing from the user; a message to a group or channel that doesn't ask the user by name; someone describing their own plans or what they are waiting for. Neither does a promise the sender makes about their own action, such as "I'll return it tomorrow" or "I'll send the slides Friday": that is the sender's task, even when the message mentions the user. Japanese often leaves out who acts; a sentence like "明日返します" or "金曜に送ります" from the sender is the sender's own promise. Leave both empty for all of these, and when in doubt.
Write the task in the same language as the reason.
{{policy}}
{{notification}}
Answer with exactly {{answer.count}} lines and nothing else:
{{answer.Category}}
{{answer.Confidence}}
{{answer.Reason}}
{{answer.Tags}}
{{answer.Task}}
{{answer.Due}}Caveats and next
- One grader, me, and only 158 Important notifications, so those percentages could be about 8 points off either way.
- Private Cloud Compute has answered each notification only once, so how often its answers change between runs isn’t measured yet.
- Qwen ran on a Mac with its GPU. On an iPhone, and in the background where Karui sorts, it would likely be slower.
- Apple can change Private Cloud Compute’s model and limits without notice.
- Next: try a prompt aimed at the notifications Private Cloud Compute and Opus both missed, first on Opus, then on Private Cloud Compute.
Revision History
- 2026-11-07 Added OpenAI’s decision model, GPT-6 Luna Decisions, which is now the most accurate overall. Ran Opus, GPT-6.1 Sol, Apple’s on-device model, and Qwen again to measure how much their answers change from run to run; Opus’s lead over Sol is smaller than one run suggested. Noted that invite-only Karui Cloud Sorting uses GPT-6 Luna Decisions.
- 2026-11-05 Finished all 463 notifications and added OpenAI’s GPT-6.1 Sol. Replaced Apple’s on-device model on the Private Cloud Sorting prompt, which isn’t how Karui uses it, with Apple’s model and Qwen 3.5 4B through Karui’s own on-device sort. Qwen’s first results, posted earlier the same day, listed its answer’s fields in the wrong order by mistake, which made it look worse. Tightened the commentary.
- 2026-11-03 Reworded the results and how the test works.
- 2026-11-02 Updated to 382 of 464 notifications, where Opus’s overall lead became significant and the on-device model caught the most Important notifications. Added the prompt, named the task in the title, and renamed Second Opinion to Private Cloud Sorting.
- 2026-11-01 Updated to 280 of 464 notifications, and noted that the one-time code rule catches the verification code first.
- 2026-10-31 Updated to 171 of 464 notifications, and added Apple’s on-device model.
- 2026-10-29 First published, with 87 of 464 notifications.