Karuiアプリアイコン

KN006:AppleのPrivate Cloud Compute対Claude Opus 5.5など(通知の分類)

Karui

AppleのPrivate Cloud Computeが私の通知をどれだけ正しく分類できるかを、Claude Opus 5.5などのモデルと比べ、私自身の手作業の分類と照らして測ります。

463件の通知 · macOS 27.2で実行 · 2026年11月7日更新

結果

重要な通知は私の手を止めるので、重要と判定されたかどうかが最も重要な指標です。

Private Cloud Compute、Opus、GPT-6.1 Solには、KaruiのPrivate Cloud分類のプロンプトを渡しました。OpenAIの判定用モデルGPT-6 Luna Decisionsにも同じプロンプトを渡しました。このモデルはカテゴリごとの確率を返します。AppleのオンデバイスモデルとQwen 3.5 4Bは、Karui自身のオンデバイス分類で動かしました。

勝者

重要な通知を拾えた割合

Private Cloud ComputePCC
56%(158件中88件)
Claude Opus 5.5Claude
48%(158件中76件)
GPT-6.1 SolGPT
46%(158件中72件)
Appleオンデバイスオンデバイス
56%(158件中89件)
Qwen 3.5 4BQwen
39%(158件中62件)
GPT-6 Luna DecisionsLuna
69%(158件中109件)

誤って重要と判定された通知

Private Cloud ComputePCC
14%(305件中44件)
Claude Opus 5.5Claude
3%(305件中10件)
GPT-6.1 SolGPT
5%(305件中16件)
Appleオンデバイスオンデバイス
18%(305件中55件)
Qwen 3.5 4BQwen
7%(305件中22件)
GPT-6 Luna DecisionsLuna
11%(305件中33件)

全体(重要かどうか)

Private Cloud ComputePCC
75%(463件中349件)
Claude Opus 5.5Claude
80%(463件中371件)
GPT-6.1 SolGPT
78%(463件中361件)
Appleオンデバイスオンデバイス
73%(463件中339件)
Qwen 3.5 4BQwen
75%(463件中345件)
GPT-6 Luna DecisionsLuna
82%(463件中381件)

全体で最も正確で、重要な通知を最も多く拾ったのはGPT-6 Luna Decisionsでした。Private Cloud Compute(正確なマクネマー検定でp ≈ 0.001)とGPT-6.1 Sol(p ≈ 0.02)にははっきり勝っています。Opusに対する優位は有意ではありません(p ≈ 0.29)。Opusが見逃す重要な通知を多く拾う一方、誤報も多くなります。

GPT-6 Luna Decisionsでは、重要とみなす確率の境目をいくつか試しました。30%から55%の間なら全体の正確さはほぼ同じだったので、標準の50%のままにしました。

Opusは重要でない通知で手を止めることがほとんどありません。全体ではOpusがPrivate Cloud Computeに勝っています(59件対37件、p ≈ 0.03)。重要な通知を拾う数はPrivate Cloud Computeのほうが多いものの、その差は有意ではありません(35件対23件、p ≈ 0.15)。どのモデルも同じ確認コードをスパムと判断しましたが、アプリではKaruiのワンタイムコードのルールがモデルより先に分類するため、こうはなりません。

GPT-6.1 SolはOpusとよく似た答え方をします。1回目の実行では、重要かどうかで両者の意見が分かれたのは14件で、そのうち12件はOpusが正解でした(p ≈ 0.01)。3回実行してそれぞれの多数決の答えで比べると、意見が分かれたのは15件で、Opusが正解だったのは10件です(p ≈ 0.30)。Opusの優位は1回の実行から見えるより小さいようです。Private Cloud Computeに対するSolの優位は有意ではありません(56件対44件、p ≈ 0.27)。

Appleのオンデバイスモデルは、重要な通知をPrivate Cloud Computeと同じくらい拾いますが、誤報が多く、全体では両者は近い結果です(Private Cloud Computeだけが正解したのが60件、その逆が50件、p ≈ 0.39)。Opusとの差ははっきりしています(68件対36件、p ≈ 0.002)。また、Appleのオンデバイスモデルはスパムと判断しがちで、そのうち23件は私が重要と採点した通知でした。

自分で用意したモデルを試す理由:私の1か月分の履歴では、AppleのオンデバイスモデルがKaruiのリクエストの48%をレート制限で断りました(KN001)。一方、Apple Intelligenceのラボで、自分で用意したオンデバイスモデルにはレート制限がかからないとAppleのエンジニアから聞きました。Foundation Modelsではアプリが独自のモデルを組み込めて、Karuiの分類器はそのまま動きます。

Qwen 3.5 4Bは全体ではAppleのモデルとほぼ同じ正確さです(61件対55件、p ≈ 0.64)が、傾向は逆です。誤報は55件に対して22件と少ない一方、拾えた重要な通知も89件に対して62件だけでした。通知のフィルターにとっては、大事なものを見逃すほうが悪い誤りです。Qwenは遅くもあり、M5のMacで1件あたりの中央値は6.0秒、5件はKaruiの20秒の制限を超えました。KaruiのルールはAppleのモデル向けに書かれているので、Qwen向けに調整したプロンプトや、Gemma 4 E2Bなど別のモデルなら良くなるかもしれません。

実行ごとの答えの変わりやすさ:Opus、GPT-6.1 Sol、Appleのオンデバイスモデル、GPT-6 Luna Decisionsはそれぞれ3回、Qwenは2回実行しました。463件のうち、重要かどうかの答えが実行によって変わったのは、Opusが7件、Solが9件、Qwenが2回だけで64件、Appleのオンデバイスモデルが76件でした。GPT-6 Luna Decisionsは毎回、確率まで含めてまったく同じ答えを返しました。グラフは各モデルの1回目の結果です。Private Cloud Computeはまだ1回だけです。

普通とスパムの区別は、毎日のまとめ機能がオンのときにだけ意味を持ちます。その場合、私の採点との一致はOpusとGPT-6 Luna Decisionsが68%、GPT-6.1 SolとPrivate Cloud Computeが62%、Qwenが54%、Appleのオンデバイスモデルが52%でした。

Private Cloud ComputeとOpusの両方が誤ったもの

Private Cloud ComputeとOpusは、私が重要と採点した同じ47件の通知をどちらも見逃し、同じ8件の誤報を出しました。47件のうち19件は、App Storeの審査や請求書のようなアカウント・支払い・サービスの通知で、23件は人からのメッセージです。多くはプロンプトの調整で直せそうです。

測定方法

このテストのために、私は自分の通知をAnthropicとOpenAIに送りました。一部は、データを保持しない設定(ZDR)をオンにしたVercelのAI Gateway経由です。Karui自体がそうするのは、招待制のKarui Cloud分類だけです。この機能は通知をKaruiのサーバー経由でGPT-6 Luna Decisionsに送ります。それ以外では、KaruiはAppleのオンデバイスモデルと、Private Cloud分類がオンのときはPrivate Cloud Computeだけを使います(KN004)。ツールのkarui-evalはKarui for Macのソースコードにあります。

プロンプトを表示

Private Cloud Compute、Opus、GPT-6.1 Solには、この同じ英語の文章を1つのメッセージとして送り、GPT-6 Luna Decisionsには入力として渡しました。AppleのオンデバイスモデルとQwenには、同じルールをセッションの指示として渡しました。{{policy}}の部分には、重要な連絡先やアプリ、緊急・消音の語句といった私の設定が入ります(ここでは省略しています)。{{notification}}の部分には通知が入ります。ソースコードでも読めます。

Classify one notification as Important, Neutral, or Spam.
First write the reason, then choose the category the reason supports. When the reason says nothing is needed from the user, choose Neutral, unless the Sender is an important contact, it is a security alert, or it matches the user's urgent terms.
Only connect a notification to one of the user's calendar events when it names that event or matches its place or time. When it doesn't say which event, don't guess, and don't mention the user's events in the reason.

Important:
- The Sender matches an important contact by name, email, or phone. Merely mentioning an important person's name in the title or body does not count.
- A question or request written by a person directly to the user that needs their reply or decision.
- A person writes to the user and addresses them by one of the user's names. Marketing, newsletters, and automated messages that insert the user's name do not count.
- Security alerts about the user's own accounts, such as a new sign-in, a password change, or fraud.
- A reminder about an appointment or reservation the user has already booked, happening on the day the notification was received or the next day.
- A change to the user's own travel, such as a delay, cancellation, or gate change.
- A change to the user's plans happening soon, such as someone cancelling, running late, or moving the time.
- Something arriving now or waiting for the user, such as a ride pulling up, food at the door, or a courier outside.
- The app marked the notification time sensitive, unless it is marketing, a sale, or a promotion.
- Anything matching the user's urgent terms.
- A direct message to the user from a person in an important app, unless it is only a reaction or it is automated.
- When the source is an important app and you are unsure between Important and Neutral, choose Important.

Neutral:
- A message addressed to someone else, such as "Addressed to: @Bob, not the user", even when it asks a question or makes a request. It is for that person, not the user.
- Reactions such as "laughed at", "loved", "liked", or "emphasized", and casual group chat that asks nothing of the user.
- A question put to a whole channel or group, such as "Has anyone been here?" or "Does anybody know…?". It is not written directly to the user, unless it names or @mentions them.
- Someone confirming plans or saying they'll attend, such as "See you there", when it asks nothing and changes nothing.
- Messages generated by a system, bot, or service, such as receipts, digests, shipping and delivery updates that don't need the user right now, and status updates. Quoted or forwarded content inside a person's message does not make the message automated.
- Anything matching the user's summary terms.
- Everything that is not clearly Important or Spam.

Spam:
- The app or carrier labels the sender or message as spam, such as "Suspected spam", "Spam Risk", "Scam Likely", or "Potential fraud", unless the Sender matches an important contact.
- Messages matching the user's silence terms, and promotions when the user's policy says so, unless their Sender matches an important contact, even from important apps.
- Offers, sales, and deadlines to buy or sign up are promotions, not reminders.

Examples:
- A person asks "Can you pick up milk on your way home?": Important, a request that needs the user's reply.
- The user's names include "Aya", and a person writes "Aya, are you still coming tonight?": Important, written to the user by name.
- The user's names include "Aya", and a store writes "Aya, your exclusive offer ends tonight": a promotion that uses the user's name, not a personal message.
- Subtitle "Family", body "Ken laughed at “see you soon”": Neutral, a reaction in a group chat.
- Source Slack, body "@Bob can you send me that report?", addressed to @Bob, not the user: Neutral, a request to Bob.
- Received Tuesday, body "Reminder: your dentist appointment is Wednesday at 10:00": Important, a booked appointment the next day.
- The user has "Team all-hands 19:00" today, and a friend writes "Yes, I plan on attending as well": Neutral, "Confirms they'll attend an event; the message doesn't say which, so it isn't linked to the user's calendar."
- The user has "Book club 19:00" today, and a friend writes "Running 20 minutes late, start without me": Important, a change to today's plans.
- "Last chance! 30% off ends tomorrow": a promotion, not a reminder. Spam when the user silences promotions, otherwise Neutral.
- "Your package will arrive tomorrow": Neutral, a routine delivery update.
- Marked time sensitive, "Your driver is arriving. Jordan is 2 minutes away": Important, a ride arriving now.
- Marked time sensitive, "Flash sale ends in 1 hour!": a promotion, so the time sensitive mark doesn't make it Important.
- Sender "Suspected spam", title "Missed call from (555) 010-0199": Spam, the app labeled it spam.

Follow the user's policy conservatively. Never invent context, and never call something urgent unless the text says so.
Also rate interest from 1 through 5, independent of the category, using the whole range:
5: a real person wrote to the user personally, including a reply in a thread the user is part of (RE:), or it needs the user's decision or reply.
4: specific news about the user's own plans, orders, bills, accounts, or work.
3: a group conversation, community event, or update the user follows but need not act on.
2: routine automated status, such as delivered, synced, or completed, and newsletters the user subscribed to.
1: marketing, promotions, and generic broadcasts, even when worded as a personal invitation or sent under a person's name.
Write the reason in English.
Also look for a deadline: something the notification asks the user to do by a certain day.
task: what the user must do, at most 8 words, such as "Pay invoice #2041". Leave it empty when nothing is asked of the user by a certain day.
dueDate: that day as YYYY-MM-DD, taken from the Dates line, only when the notification itself names the day: a date, a weekday, "tomorrow", "end of the month". Leave it empty when there is no task or no day is named. Never guess a day from words like "soon", "later", or "before I leave".
Only these count, each something the user must do themselves by the day named: pay a bill, invoice, or fee; reply or RSVP when someone asks the user directly; submit, sign, or send a form, document, or application; renew something of the user's that would otherwise lapse, such as a membership, license, passport, or registration, or cancel a free trial before it starts charging; return something borrowed or rented, or an order within its return window.
None of these is a task: a sale, offer, or promotion that ends on a day; an event, appointment, flight, or delivery the user attends or waits for, which the calendar or the app already reminds them of; a confirmation or receipt saying something is done, scheduled, or paid; an automatic payment or renewal that needs nothing from the user; a message to a group or channel that doesn't ask the user by name; someone describing their own plans or what they are waiting for. Neither does a promise the sender makes about their own action, such as "I'll return it tomorrow" or "I'll send the slides Friday": that is the sender's task, even when the message mentions the user. Japanese often leaves out who acts; a sentence like "明日返します" or "金曜に送ります" from the sender is the sender's own promise. Leave both empty for all of these, and when in doubt.
Write the task in the same language as the reason.

{{policy}}

{{notification}}

Answer with exactly {{answer.count}} lines and nothing else:
{{answer.Category}}
{{answer.Confidence}}
{{answer.Reason}}
{{answer.Tags}}
{{answer.Task}}
{{answer.Due}}

注意点と今後

改訂履歴

← 技術ノート一覧