Agentic Executive Assistant & Portfolio Command System

An always-on agentic system that watches my inbox, chats, and calendar, drafts replies, farms context from external systems into a structured knowledge vault, and renders an executive dashboard of my entire project portfolio - built and used daily to run a live consulting workload. It runs only on my own machine: personal infrastructure, not a deployed product.

1. Why this exists

I run a portfolio of concurrent client engagements across government and private-sector clients. The failure mode is not capability - it is state loss: a commitment made in a Teams thread, a deadline moved in an email, a project that quietly stops moving while attention is elsewhere.

Rather than adopt a task manager, I built the system I actually needed: an agentic layer that treats my own working context as a data-engineering problem. It is the most honest demonstration in this portfolio, because it is the one system whose failures I feel immediately and whose value I can measure directly - I use it every working day.

It also functions as deliberate R&D. Every agentic pattern I’ve built for a client - tool grounding, multi-agent decomposition, evaluation gating, human-in-the-loop approval - I tested here first, on a system where I am the one who bears the cost of getting it wrong.

2. Architecture - three layers

Layer 1 - Structured knowledge vault

A local markdown vault with strict YAML frontmatter as the state store: tasks, projects, people, ideas, daily briefs, weekly summaries, and outputs. Markdown rather than a database, deliberately - it is diffable, greppable, editable by hand, readable by any LLM without a schema translation step, and durable in a way a proprietary app’s storage is not.

Each entity type has a defined contract. Projects carry status, priority score, tier, stakeholder, strategic alignment, blocking factor, effort, working-directory link, and a display block driving how the project renders on the dashboard.

Layer 2 - Agentic layer (agents, skills, and “farmers”)

Built on an agentic CLI harness with three composable mechanisms:

• Skills - invocable workflows (/today, /sync, /prioritize, /review, /new, /history) that encode a repeatable process as instructions rather than code, so the agent executes them consistently.
• Farmers - specialised subagents that read an external system through MCP (Model Context Protocol) servers and write relevant context into the vault. One per source: Microsoft Planner, Outlook, Teams/Slack, and meeting transcripts. Each maintains its own state file with a last-run timestamp, so a run is incremental rather than reprocessing history. Each classifies what it finds (task / project / person / idea), scores it, deduplicates against existing vault content, and writes with source provenance in frontmatter.
• Rules - externalised decision logic (prioritisation model, sync behaviour, persona, brief format) kept as separate documents that both I and the agent read, so behaviour is inspectable and changeable without touching prompts scattered across files.

Access goes through Microsoft Graph with explicitly granted permission scopes: mail read, Teams read and write, calendar. The flagship routine: when a new engagement lands, the system consolidates the inputs from mail and Teams, downloads the referenced documents itself, and writes the project brief straight into the development directory.

The design principle: the subagent definition is simultaneously the configuration and the executor. There is no separate config format to keep in sync with behaviour.

Layer 3 - Always-on daemon and native app

A Python asyncio daemon with incremental change pollers over Microsoft 365, a hand-rolled JSON-RPC MCP client over stdio, a two-tier LLM router (a fast cheap model for triage, a stronger model for drafting, with prompt caching on the persistent style and profile context), a SQLite state store holding watermarks and seen-message dedup, and a Unix-socket server. A Swift/SwiftUI menu-bar app provides native notifications with action buttons, a draft-editing window, and a global voice hotkey routing through local on-device speech-to-text into an agent with full tool access.

Human-in-the-loop by design, with graduated autonomy:

The graduated-autonomy model is the interesting part: full automation for genuinely trivial acts (with an undo), approval for everything consequential, and explicit confirmation for anything destructive. Autonomy is calibrated to the cost of being wrong.

The allowlist is managed from the app’s own UI, per person and per channel: whitelist a contact for email, for Teams messages, or both, and the system replies to them automatically within the trivial-acts boundary. Everyone else gets draft-and-approve.

3. Portfolio intelligence - the dashboard and its instrumentation

Rendering

An executive dashboard is generated from the project markdown files by a Python + Jinja2 renderer - priority-tier grouping, per-project cards with status and progress, click-through detail modals, checklists, activity timelines, and stakeholders. It renders in ~100ms, and is embedded in the native app via WKWebView with auto-reload on file change.

The rule that keeps it honest: the markdown is the source of truth and the HTML is never hand-edited. Any change happens in the data and re-renders.

Prioritisation model

Every project scores 0–100 on a documented weighted model - deadline pressure (0.25), blocking factor (0.30), stakeholder visibility (0.20), strategic alignment (0.15), effort efficiency (0.10) - mapping onto P0–P4 tiers. The weights are explicit and the arithmetic is shown, so I can disagree with a score and override it. A model you can’t argue with is a model you stop trusting.

Freshness instrumentation - and the bug that taught me the most

I built an activity scanner that tags every project by how recently its working directory changed (this week / 2 weeks / this month / 1–3 months / dormant), surfacing projects that claim to be active but have stopped moving.

The first implementation was wrong, in an instructive way. Using newest-file modification time, six unrelated projects all reported activity on the same date - including ones whose last real commit was months earlier. A bulk file-sync operation had rewritten modification times across whole directory trees. Only 11 of 77 files in one project carried that date, and 24 of 11,120 in another: the signal was a sync artifact, not work.

The fix was to change the signal, not patch the symptom: git commit history for repositories (which a file copy cannot forge), file modification time only for non-git directories, and each project records which signal produced its answer so a wrong-looking date is diagnosable rather than mysterious.

Then a second, deeper correction. A project showed as dormant for 145 days - and was in fact days from launch. Its build sat with an external vendor, so all real activity lived in an email thread while the local repository sat still. Code activity is not the only kind of activity. The scanner now takes the newest of repository activity and a recorded last-correspondence date, reporting which won.

Both bugs share a lesson I now apply generally: an instrument that reports confidently from a single proxy signal will be confidently wrong. Cross-check the proxy, and make the system state which evidence it used.

Staleness as a review trigger, not a score

A deliberate modelling decision: freshness deliberately does not feed the priority score. A project waiting on a client is correctly idle, and penalising its score would bury genuinely blocked work beneath merely neglected work. Instead, staleness forces a resolution into one of three states - waiting (must name who, and since when), neglected (surfaces as a focus candidate), or finished/dead (archived with a reason). Distinguishing a measurement from an incentive is the point.

4. What is real, and what is a substitution

I hold this project to the same standard as the client work, so: the daemon is installed and running under a LaunchAgent, the app bundle is built and installed, and the pipeline is in daily use. Every component described above exists as working code rather than scaffolding - a hand-rolled JSON-RPC MCP client over stdio, the socket server, the LLM router with prompt caching on the style and profile context, the SQLite state store with watermarks and dedup, the whisper.cpp binding, and a genuinely substantial native app (settings, activity view, markdown rendering, chat UI, audio capture, global hotkey via Carbon).

Three places where the implementation substitutes for the ideal, all of which the code documents honestly rather than papering over:

• “Delta polling” is watermark polling, not Graph delta tokens. The MCP layer doesn’t expose real delta endpoints, so change detection is an ISO-timestamp watermark plus seen-message dedup in SQLite. Equivalent in effect at this cadence, but it is not a continuation-token delta feed and shouldn’t be described as one.
• Meeting RSVP is a polite email reply to the organiser, not a real calendar accept/decline - the tool layer has no RSVP endpoint. A pragmatic v1, labelled as such in the code.
• The email “undo” cannot unsend. It logs an undo record within the window; a delivered message is delivered. The auto-send path is therefore restricted to trivial acknowledgements to allowlisted contacts, which is what makes that acceptable - but “undo” is a generous word for it.

Also honest: 3 of 35 tests currently fail, and they fail because a stub in the test fixture didn’t gain a method the real service acquired in a refactor. Test drift from my own refactor, not broken behaviour - and precisely the kind of thing I’d want caught rather than hidden. Test coverage is decent on parsers, state and classification, and absent on the Swift app and the voice pipeline end-to-end.

On the Second Brain side, two capabilities in the design are unused rather than built-and-abandoned: the weekly/ directory is empty (no weekly review has ever run) and ideas/ is empty. The people directory holds a single record against a design that assumed rich relationship context. That gap is itself the finding that drove the review described below.

5. What I’d highlight

• Dogfooding as engineering discipline. This is where I test agentic patterns before proposing them to a client, on a system whose failures cost me directly.
• Graduated autonomy calibrated to the cost of error - auto-send with undo, draft-and-approve, explicit confirmation for destructive acts.
• MCP-based tool integration across multiple enterprise systems, with incremental state so runs are cheap.
• Markdown-as-database, chosen for diffability, LLM-readability, and longevity over convenience.
• Generated-not-authored dashboard, with a single source of truth.
• Debugging my own instrumentation twice - and both times fixing the signal, not the symptom, then making the system report its own evidence source.
• Separating measurement from incentive in the staleness design.
• Honest self-assessment. The system’s own audit surfaced that I had let it drift for three months. It reported that, rather than flattering me - which is the only reason the correction happened.

6. Skills demonstrated

agentic system architecture · multi-agent orchestration · Model Context Protocol (MCP) · Microsoft Graph integration · Python asyncio daemon engineering · Swift / SwiftUI native macOS development · local speech-to-text integration · LLM routing and cost tiering · human-in-the-loop design · graduated autonomy models · template-driven rendering (Jinja2) · weighted decision models · instrumentation and signal validation · LaunchAgent / background service deployment