click_ui
Click a control by accessible NAME or by a marks number in one call: the coordinate comes from the tree, the click goes through the real pointer (visible, same safety guarantees); an ambiguous name returns the candidates instead of guessing
How to use it
click_ui is exposed by the Hypruse MCP server. Add the server to your MCP client (Claude Desktop, Cursor, Windsurf and others), and the click_ui tool becomes available to the model automatically. See the full listing for setup details and every tool this server provides.
Install Hypruse
claude mcp add -s user hypruse -- uvx hypruseOther tools in Hypruse (15)
The user's own keybinds, decoded (SUPER+Q, action, description); the agent runs one with usebind
Read or write the text clipboard via wl-clipboard; opt-in, exists only with HYPRUSECLIPBOARD=1 in the server env
One-call semantic snapshot: monitors, workspaces, windows (address/class/title/geometry), active window, cursor, and layer surfaces (launchers, bars, notification popups) with a best-effort kind and geometry
Switch workspace, focus/move/close windows, fullscreen, floating (pure IPC, milliseconds; closewindow also waits up to 1s for the destroy event, so its result and then= observation reflect the close instead of racing it)
Type literal text (unicode-safe) or press app-level combos (ctrl+shift+t, esc, F5); optional window address focuses the target first so keystrokes land in the right app. Compositor binds (super+...) go through usebind, not here
Start an app (optionally silent on another workspace), block on its actual openwindow event, return its address; detects single-instance apps (browsers) whose window ignores exec rules and moves it to the requested workspace
Set-of-Marks capture: the window screenshot with every accessible control drawn as a numbered mark, plus a JSON legend (role, name, current value, exact click point per number); needs ImageMagick for the drawing, degrades to the legend alone without it
move / click / drag / scroll (discrete wheel notches) in global coordinates
Focused monitor, exact window crop by address, or x,y,WxH region; returns image + coordinate-mapping metadata; fast JPEG by default (lossless=true for PNG); stable=true waits for the frame to settle
Run an ordered list of actions (pointer/keyboard/clickui/hypr/waitfor) in one call; stops the moment the desktop changes in a way the current step did not expect, so a click/type/enter micro-sequence costs one round-trip instead of several
what it does
Read a window's accessibility tree (AT-SPI, GTK/Qt apps that expose one) and return clickable elements by name with exact global coordinates, no screenshot; reports current values too (typed text, slider position, checkbox state); falls back to vision when an app exposes nothing
Execute a keybind by combo (SUPER+F), running its bound action, so the agent drives the owner's own launchers and shortcuts. Not available for binds in a Lua hyprland.lua: those are anonymous closures Hyprland exposes no way to call, so binds shows them as action lua and usebind says so instead of f
Block on real compositor events (window open/close, workspace change, title change, layer surfaces appearing/closing, urgency, screen sharing on/off) with a match filter and timeout; a filtered wait whose condition already holds (window already gone, workspace already active, layer already mapped) a
Native-resolution re-capture around an estimated point (optionally clamped to a window): the precision step before clicking small controls, same metadata contract