# AI Secretary: Solving Calendar Chaos with an Intelligent Agent ## The Problem In 2018, I was working night shifts at Micron, running a machine shop out of my garage, and struggling to make any meaningful progress during my limited time off. That's when I discovered *Getting Things Done* by David Allen. It's no exaggeration to say it changed my life—I've been practicing and refining GTD ever since. ![[81RFSiwpvyL._UF1000,1000_QL80_.jpg|250]] My system has evolved significantly over the years as I've identified weaknesses in both the methodology and the tools I used to implement it. I migrated from OneNote to Obsidian, built custom workflows, and developed a personal productivity system that actually works for me. But one pain point remained stubbornly unsolved. When my wife messages me that she's scheduled something for us, my "solution" has been to leave the message unread until I can manually add it to my calendar. Inevitably, she sends another message that I *do* read, and the scheduling message gets buried. The event never makes it to my calendar. Compounding this, she uses a physical planner while I use a digital calendar. The result? We constantly double-book each other or miss commitments we've made together. Earlier this year, I rebuilt my homelab Linux server specifically to be a capable sandbox for AI development. That infrastructure became the foundation for solving this scheduling nightmare. ![[Adobe Express - InShot_20251223_085201524(1).gif|250]] # AI Secretary: Solving Calendar Chaos with an Intelligent Agent ## The Solution I built a personal assistant agent accessible via Telegram that acts as an intelligent scheduling coordinator for my wife and I. It handles both text and voice messages—she can literally speak into her phone, and the system transcribes her request using GPU-accelerated transcription on my homelab server. ![[Pasted image 20251223122455.png|500]] **The Intelligence Layer:** This isn't just a dumb calendar API wrapper. The agent actively coordinates and problem-solves: **Conflict Detection & Resolution** - When my wife requests: *"Schedule dentist appointment Thursday at 2pm"* - The agent checks my calendar, detects I already have a meeting at 1:30pm - Responds: *"Conflict: You have a Product Review meeting 1:30-3:00pm. I found open slots at 11am or 4pm Thursday. Which works?"* - She confirms, the agent books it, updates both our systems - I get a single notification: *"Ashlyn scheduled dentist Thu 4pm"* **Smart Availability Queries** - *"Is he free for dinner Friday night?"* - Agent doesn't just check for empty time blocks—it understands context - Looks for evening availability, accounts for typical work hours, checks both our calendars - *"Yes, free after 6pm. Latest commitment ends at 4:30pm."* **Proactive Confirmation Flow** - Request: *"Add dinner with the Johnsons Saturday at 7"* - Agent checks if either of us has conflicting events - If clear: Books it immediately, confirms with details - If potential conflict: *"You have 'Movie night?' on your planner for Saturday. Should I cancel that or pick a different time?"* - Only notifies me once everything is resolved **Multi-Modal Input** The Telegram interface accepts both text and voice messages. When my wife sends a voice message, the system: 1. Receives audio via Telegram Bot API 2. Transcribes using Whisper on my homelab's GPU (fast, local, private) 3. Routes transcription to the orchestrator agent 4. Processes the scheduling request identically to text input This is critical—she can literally talk to the calendar while driving, cooking, or wrangling our baby. ## Architecture: Orchestrator-Specialist Pattern **Orchestrator Agent** – Routes requests based on user identity and intent **Scoped Agent (for Ashlyn):** - **Permissions:** Read/write shared calendar, query my availability, basic home automation - **Home Automation:** Turn lights on/off, control Christmas tree, open garage door - **Restrictions:** No access to my work calendar details, personal tasks, or other system tools **Specialist Agents (for me):** - Full calendar access - Integration with my GTD system in Obsidian - Integration with my knowledge management in Obsidian - Advanced queries and automation ![[Pasted image 20251223122831.png|500]] **Tool Layer:** - Google Calendar API - Home Assistant integration (lights, garage, smart devices) - Telegram notifications - Whisper (speech-to-text) ## Technical Stack **Phase 1 (Current - n8n):** - n8n for orchestration and workflow logic - Telegram Bot API for messaging interface - Whisper (GPU-accelerated) for voice transcription - Google Calendar API for scheduling - Home Assistant API for smart home control **Phase 2 (Planned - pydantic-ai):** - pydantic-ai for type-safe agent framework - FastAPI for API layer - PostgreSQL for conversation history and state management - Same external integrations (Telegram, Calendar, Home Assistant, Whisper) ## Why This Matters This project demonstrates: 1. **Practical AI application** – Not a chatbot for fun; solves a real coordination failure that was costing us actual commitments 2. **Permission boundaries** – Scoped agents with different access levels based on user identity 3. **Multi-modal interfaces** – Voice and text, seamlessly handled 4. **Proactive problem-solving** – The agent doesn't just execute commands; it detects conflicts, suggests alternatives, and only escalates when necessary 5. **Infrastructure literacy** – Self-hosted GPU acceleration, homelab deployment, API orchestration Most importantly: **it works**. My wife manages our shared schedule without learning my productivity system, and I'm no longer the bottleneck in our family coordination. --- *Built with: n8n, Telegram Bot API, Google Calendar API, Whisper, Home Assistant, NVIDIA GPU (local inference)* *Planned migration: pydantic-ai, FastAPI, PostgreSQL* Tags [[AI]] [[n8n]] [[linux]] [[server]] [[networking]]