COMET for Healthcare Operations: Clinical Decision Support (Hybrid) vs Patient Scheduling (AI)
At ARKONA, we're building an autonomous multi-agent AI ecosystem applicable across domains – initially cyber-physical reverse engineering, now expanding to business operations, and increasingly, healthcare. A core component of this expansion is COMET, our 7-step human↔AI delegation framework, grounded in IEEE and NIST standards. I want to share some specifics on how we’re applying COMET, focusing on two distinctly different healthcare use cases: clinical decision support and patient scheduling. These represent contrasting levels of AI autonomy and highlight the challenges and successes we've had implementing COMET in a regulated environment.
Clinical Decision Support: A Hybrid Approach
Clinical decision support (CDS) is a highly sensitive area. Incorrect recommendations can have severe consequences, so complete AI autonomy isn’t viable – yet. Our approach leverages a hybrid system where the AI acts as an analytical assistant, augmenting (not replacing) the clinician’s expertise. This aligns with the MITRE ATT&CK for Healthcare framework’s emphasis on minimizing risks associated with automation in critical processes.
Within ARKONA, this manifests as integration between the hardware reverse-engineering pipeline (typically used for hardware RE, but adaptable for complex data analysis) and a dedicated CDS agent. Data -- anonymized patient records, lab results, imaging reports -- flows via encrypted overlay network to the RE service. The pipeline’s Ghidra integration isn’t used for disassembly here, but for structured data extraction and analysis. It prepares a concise summary, including statistically significant findings and potential differential diagnoses, along with references to supporting literature. This summary is then presented to the CDS agent, which formats it for presentation to the physician. Crucially, the agent *does not* make a diagnosis; it presents information for the physician’s consideration. This fits COMET Step 1: 'Human Oversight Defined' – the human remains the ultimate decision-maker.
The COMET framework's Step 3, 'Risk & Reward Assessment,' is paramount here. We utilize our NIST 800-30 grounded risk evaluation engine to continuously assess the confidence level of the AI's suggestions. Factors considered include data quality, the prevalence of the condition, and the physician's prior interactions with the system. Lower confidence scores trigger alerts within the interface, emphasizing the need for careful scrutiny. This is implemented using a simple JSON configuration file driving the risk engine:
{
"condition": "pneumonia",
"data_quality_threshold": 0.8,
"prevalence_threshold": 0.05,
"confidence_weight_dq": 0.4,
"confidence_weight_prev": 0.3,
"confidence_weight_hist": 0.3,
"alert_threshold": 0.7
}
This configuration dictates that for pneumonia, the data quality must be at least 80%, the prevalence in the patient population must be greater than 5%, and historical data relating to the physician's experience with similar cases must be considered. The final confidence score is a weighted average, and if it falls below 70%, an alert is triggered. The system communicates these scores via our inter-agent communication broker, using the ‘cds_risk’ topic.
Patient Scheduling: Full AI Autonomy
Patient scheduling, while critical, presents a lower risk profile than CDS. An incorrect appointment time is inconvenient, not life-threatening. This allows for a much higher degree of AI autonomy. We’ve implemented a dedicated scheduling agent, leveraging our MuXD hybrid LLM router. The agent manages appointment slots, considers physician availability, patient preferences (communicated via a secure web interface authenticated with WebAuthn/Face ID), and optimizes for efficient resource utilization.
The key here is COMET Step 5: 'Autonomous Execution with Monitoring'. The scheduling agent operates independently, booking appointments and sending confirmations. However, it’s constantly monitored by a separate ‘sync’ agent – one of our 26 autonomous agents on battle rhythm – which tracks key performance indicators (KPIs) like appointment no-show rates, wait times, and patient satisfaction. This agent publishes metrics to a dedicated monitoring service.
MuXD plays a critical role in cost-effectiveness. The initial prompt engineering and data pre-processing are handled locally by one of our 5 Ollama models (specifically, a fine-tuned Llama 2 variant). Complex reasoning – for example, handling conflicting appointment requests or accommodating last-minute cancellations – is routed to Claude via API. This reduces reliance on expensive cloud compute and minimizes latency. Token savings optimization, a key feature of MuXD, ensures we’re maximizing value from our Claude subscription.
This system is also integrated with our provenance signing process. Every scheduled appointment is signed with a SHA-256 hash, creating an immutable audit trail. This provides strong evidence of system behavior and allows for easy identification of any anomalies – crucial for regulatory compliance.
Comparing COMET Implementation & Lessons Learned
The contrasting approaches to CDS and patient scheduling demonstrate the flexibility of COMET. In CDS, we prioritize human oversight and risk mitigation, aligning with the ethical considerations of healthcare. In patient scheduling, we embrace AI autonomy, leveraging its ability to optimize complex processes. The level of autonomy directly correlates with the inherent risk of the application.
One of the biggest challenges has been defining the boundaries of AI responsibility. It’s easy to get caught up in the technical possibilities, but it’s vital to remember that AI is a tool, not a replacement for human judgment. This is where COMET Step 7, 'Continuous Refinement & Feedback,' becomes crucial. We’re constantly gathering feedback from clinicians and patients, analyzing system performance, and refining our algorithms. The 180 commits in the last 7 days reflect this ongoing iterative process.
Another key takeaway is the importance of data quality. Garbage in, garbage out. The pipeline’s data extraction capabilities are essential, but they can only work with high-quality, structured data. Investing in data standardization and quality control is paramount. Finally, a robust monitoring infrastructure (like our sync agent and monitoring service) is essential for identifying and addressing potential issues before they escalate.
Currently, 21 out of 22 services are online, demonstrating the stability of our infrastructure. However, maintaining this level of uptime requires constant vigilance and proactive maintenance.
Key Takeaway: Successful AI implementation in healthcare isn't about building the most advanced algorithms. It's about thoughtfully applying AI to solve specific problems, carefully considering the risks and benefits, and prioritizing human oversight where it matters most. The COMET framework provides a structured approach to this process, ensuring that AI is used responsibly and ethically.
```