The Aircraft Maintenance Case Study: 55 Tasks Classified Across 5 Delegation Levels with NAS 410 Citations
Over the past quarter, my team and I at ARKONA have been rigorously testing the COMET framework – our 7-step human↔AI delegation system – in a complex, real-world scenario: aircraft maintenance. This isn’t just about automating tasks; it's about establishing verifiable trust in an AI-augmented workflow adhering to stringent safety standards. We’ve focused on classifying 55 distinct maintenance tasks across five delegation levels, and grounding our approach in NAS 410, the standard for aviation maintenance.
COMET and the Five Levels of Delegation
COMET, built on top of our MuXD hybrid LLM router (currently serving requests across internal service ports), allows us to dynamically route tasks between human operators and our 26 autonomous agents. The core concept is a progressive increase in AI autonomy. We’ve mapped this to five delegation levels:
- Level 1: Human-in-the-Loop (HITL) - Monitoring. The AI *proposes* actions, but a human operator reviews and approves *every* step. This is used for critical, safety-related tasks like interpreting structural inspection reports.
- Level 2: AI-Assisted Execution. The AI executes tasks with constant human monitoring, providing real-time feedback and intervention capability. An example is an agent controlling a robotic arm for applying non-destructive testing (NDT) fluids.
- Level 3: AI-Supervised Autonomy. The AI performs the task autonomously, but the human operator monitors key performance indicators (KPIs) and can intervene if anomalies are detected. This is applied to tasks like bolt torque verification.
- Level 4: Autonomous Execution with Reporting. The AI completes the task, logs all actions with SHA-256 provenance signing, and delivers a comprehensive report to the human operator. This might include compiling a list of replaced components.
- Level 5: Full Autonomy. The AI handles the task end-to-end, including initiating and concluding the process, without direct human intervention. This is reserved for routine, well-defined tasks like environmental condition logging.
Task Classification and NAS 410 Alignment
We identified 55 tasks covering a representative range of aircraft maintenance procedures. Each task was evaluated against NAS 410’s requirements for accuracy, repeatability, and documentation. We then assigned a delegation level based on the criticality of the task, the potential for error, and the maturity of the AI agent capable of performing it.
Here's a breakdown of task distribution across the levels:
- Level 1 (HITL): 8 tasks – Primarily visual inspections requiring nuanced judgment (e.g., crack propagation assessment). Agents leverage our Ghidra-integrated hardware RE pipeline to pre-process image data for anomaly detection, but final assessment remains human-driven.
- Level 2 (AI-Assisted): 12 tasks – Automated NDT procedures (dye penetrant, ultrasonic testing) with human supervision. Agents control robotic manipulators via a dedicated internal service and interpret sensor data with AI-powered anomaly detection.
- Level 3 (AI-Supervised): 15 tasks – Routine maintenance checks (tire pressure, fluid levels, filter replacements) with constant KPI monitoring by a dedicated monitoring agent. Alerts are routed through our inter-agent communication broker (MCP server) if thresholds are exceeded.
- Level 4 (Autonomous Reporting): 10 tasks – Component replacement logging, parts inventory updates, and documentation generation. Agents utilize the REOps service to cross-reference parts with maintenance manuals and generate accurate reports.
- Level 5 (Full Autonomy): 10 tasks – Environmental data logging (temperature, humidity), operational hour tracking, and basic system health checks. These tasks are handled entirely by autonomous agents without human oversight.
Technical Implementation Details
The core of this delegation system lies in the interaction between several ARKONA services. When a maintenance task is initiated, it’s submitted to the BizOps domain. BizOps then routes the task to the appropriate agent via the inter-agent communication broker (MCP server). The agent's capabilities and the defined delegation level determine how the task is executed.
Consider a Level 2 task: performing an eddy current inspection. The workflow looks like this:
- BizOps submits the task to the MCP server.
- The MCP server dispatches the task to the ‘EddyCurrentAgent’.
- The EddyCurrentAgent initiates the robotic arm (controlled through a dedicated service) and scans the specified area.
- The agent analyzes the eddy current data in real-time, identifying potential flaws.
- The agent displays the results to a human operator *with* the data stream from the robotic arm and sensor readings, allowing immediate intervention if needed.
- All actions are logged with SHA-256 provenance signing by the CoreOps service.
Here’s an example of the task configuration file (YAML) used to define the delegation level and agent parameters:
task_id: "EC-INS-001"
task_name: "Eddy Current Inspection - Wing Leading Edge"
delegation_level: 2
agent: "EddyCurrentAgent"
agent_port: 7003
robot_control_port: 7002
kpi_thresholds:
signal_amplitude: 0.5
scan_speed: 20 # mm/s
anomaly_count: 5
human_intervention_required: true
nas410_compliance_check: "Visual confirmation of scan coverage and flaw identification"
provenance_logging: true
Our MuXD router is key to this. It optimizes token usage by intelligently routing requests. Tasks requiring complex reasoning or creative problem-solving (e.g., interpreting unusual inspection findings) are routed to Claude (cloud), while tasks involving data processing or code generation are handled by our local Ollama models (Mixtral 8x7B, running on internal service ports). This hybrid approach significantly reduces API costs and latency.
Risk Evaluation and COMET Integration
We’ve integrated a NIST 800-30 grounded risk evaluation engine into the COMET framework. Before delegating a task, the system automatically assesses the potential risks associated with AI autonomy at each level. This assessment considers factors like the severity of a potential failure, the likelihood of occurrence, and the availability of mitigation strategies.
For example, delegating a Level 4 task (autonomous component replacement) requires a higher level of risk mitigation than a Level 1 task (HITL monitoring). This might involve requiring multiple redundant checks, detailed logging, and automated rollback mechanisms.
Newsroom Verification and Battle Rhythm
The 5-agent newsroom editorial pipeline provides a vital check on the AI's output. Reports generated by agents (Level 4 tasks) are automatically routed to the newsroom for fact-checking and verification. This ensures the accuracy and reliability of the maintenance documentation. This pipeline runs on a battle rhythm, processing reports within a guaranteed timeframe. The agents are synchronized daily, drawing information from the central data lake maintained by the CoreOps service.
Key Takeaway
This case study demonstrates that achieving verifiable trust in AI-augmented workflows isn't just about building sophisticated algorithms. It’s about meticulously classifying tasks, carefully defining delegation levels, and grounding the entire process in established standards like NAS 410. The COMET framework, combined with our infrastructure and the intelligent routing provided by MuXD, provides a practical pathway towards realizing the full potential of autonomous agents in safety-critical domains. The biggest lesson learned? Don't focus solely on what the AI *can* do, but on *how* you can demonstrably verify that it's doing it correctly, and with acceptable risk.