AI agents fail on desktops. They hallucinate UI states. They click buttons that aren't there. They read text that doesn't exist. This breaks automation trust. Users stop relying on the tool. The solution is a verification layer. You intercept agent actions. You check the actual OS accessibility tree. You only allow the click if the target exists. This grounds the LLM in reality. Without this, your agent is guessing. Guessing fails in production.
1. Install the Observer
You need access to the OS accessibility API. On macOS, use AXUIElement. On Windows, use UI Automation. Standard screen scraping tools miss semantic data. Install the necessary bindings. For Python, pyautogui is not enough. It only sees pixels. You need pyobjc for macOS or comtypes for Windows to see elements. Run pip install pyobjc-framework-ApplicationServices. This gives you the tree structure of the current window. You can now query every button and field. Write a helper function to dump the current tree. Log the output to understand the structure. You will see nested dictionaries representing UI components.
2. Define a Strict Schema
Do not let the agent send raw clicks. Force it to propose an action first. The proposal must include the element ID or label. Use Pydantic to enforce this. The model outputs JSON. You parse the JSON. If the schema fails, reject the action immediately. This stops malformed commands before they reach the OS. Define fields for action_type, target_label, and expected_value. This structure makes validation possible. Require the model to state what it expects to happen. This creates a baseline for verification. If the button says "Submit", the model must know that.
3. Intercept and Verify
Before executing the click, query the accessibility tree. Search for the element label provided by the agent. If the element exists, proceed. If not, return an error to the LLM. Tell the model exactly what went wrong. This feedback loop corrects the hallucination. The model learns the current state. Add a timeout of 5 seconds for the search. If the UI is slow, wait before failing. Implement a retry logic. Try the search three times. If it fails三次，stop the action. Log the failure for debugging. This data helps you improve the system later.
Common pitfalls
Accessibility trees are noisy. Many elements have empty labels. You must filter out invisible items. Focus on interactive elements like buttons and inputs. Ignore static text unless specifically requested. Also, permissions are strict. macOS requires Accessibility permissions in System Settings. Your script will crash without them. Warn users during setup. Do not hide this requirement. Users need to know why the app needs control. Another issue is dynamic IDs. Some apps randomize element IDs. Rely on stable labels instead. If labels change, your agent breaks. Test on multiple app versions.
Next step
Add visual grounding. Combine accessibility trees with screenshot embeddings. This handles cases where accessibility data is missing. Use a multimodal model to verify the element visually. This adds a second layer of truth. Check out multimodal RAG patterns for implementing this.