OpenClaw

linux-gui-control

Control the Linux desktop GUI using xdotool, wmctrl, and dogtail. Use when you need to interact with non-browser applications, simulate mouse/keyboard input, manage windows, or inspect the UI hierarchy of applications on X11/GNOME. Supports: (1) Clicking/typing in apps, (2) Resizing/moving windows, (3) Extracting text-based UI trees from apps (A11y), (4) Taking screenshots for visual analysis.

dreamtraveler130 来源安装数☆ 0 仓库 Stars

查看来源 ↗安装与使用 SKILL.md ↗

安装与使用

请根据来源仓库和你的 AI 工具说明安装此技能。先阅读技能文件和依赖要求，再添加到工作流。

SkillHub CLI

skillhub install dreamtraveler13/guicountrol

打开来源仓库，查看说明和兼容环境后，按来源提供的命令或步骤安装。

将来源链接提供给你的 Agent，让它先读取说明、核对依赖，并向你说明将执行的安装步骤。

查看来源 ↗使用指南

技能说明

Linux GUI Control

This skill provides tools and procedures for automating interactions with the Linux desktop environment.

Quick Start

1. Identify Target Window

Use wmctrl to find the exact name of the window you want to control.

wmctrl -l

2. Inspect UI Hierarchy

For apps supporting accessibility (GNOME apps, Electron apps with --force-renderer-accessibility), use the inspection script to find button names without taking screenshots.

python3 scripts/inspect_ui.py "<app_name>"

3. Perform Actions

Use xdotool via the helper script for common actions.

# Activate window
./scripts/gui_action.sh activate "<window_name>"

# Click coordinates
./scripts/gui_action.sh click 500 500

# Type text
./scripts/gui_action.sh type "Hello World"

# Press a key
./scripts/gui_action.sh key "Return"

Workflows

Operating an App via Text UI

List windows with wmctrl -l.
Activate the target window.
Run scripts/inspect_ui.py to get the list of buttons and inputs.
Use xdotool key Tab and Return to navigate, or click if coordinates are known.
If text-based inspection fails, fallback to taking a screenshot and using vision.

Forcing Accessibility in Electron Apps

Many modern apps (VS Code, Discord, Cider, Chrome) need a flag to expose their UI tree:

pkill <app>
nohup <app> --force-renderer-accessibility > /dev/null 2>&1 &

Tool Reference

wmctrl: Window management (list, activate, move, resize).
xdotool: Input simulation (click, type, key, mousemove).
dogtail: UI tree extraction via AT-SPI (Accessibility bus).
scrot: Lightweight screenshot tool.

来源信息

Source: github
Repository: openclaw/skills
Path: skills/dreamtraveler13/guicountrol
Version: —
Commit: 8c0a8e667b0fa9726a1406fb8fdff706dbacc3d2
分类来源: rules
状态: active
可见性: public

这些信息来自收录时的来源分析，可能滞后；使用前请核对最新来源文件。

来源内容

SKILL.md

# Linux GUI Control

This skill provides tools and procedures for automating interactions with the Linux desktop environment.

## Quick Start

### 1. Identify Target Window
Use `wmctrl` to find the exact name of the window you want to control.
```bash
wmctrl -l
```

### 2. Inspect UI Hierarchy
For apps supporting accessibility (GNOME apps, Electron apps with `--force-renderer-accessibility`), use the inspection script to find button names without taking screenshots.
```bash
python3 scripts/inspect_ui.py "<app_name>"
```

### 3. Perform Actions
Use `xdotool` via the helper script for common actions.
```bash
# Activate window
./scripts/gui_action.sh activate "<window_name>"

# Click coordinates
./scripts/gui_action.sh click 500 500

# Type text
./scripts/gui_action.sh type "Hello World"

# Press a key
./scripts/gui_action.sh key "Return"
```

## Workflows

### Operating an App via Text UI
1. List windows with `wmctrl -l`.
2. Activate the target window.
3. Run `scripts/inspect_ui.py` to get the list of buttons and inputs.
4. Use `xdotool key Tab` and `Return` to navigate, or `click` if coordinates are known.
5. If text-based inspection fails, fallback to taking a screenshot and using vision.

### Forcing Accessibility in Electron Apps
Many modern apps (VS Code, Discord, Cider, Chrome) need a flag to expose their UI tree:
```bash
pkill <app>
nohup <app> --force-renderer-accessibility > /dev/null 2>&1 &
```

## Tool Reference

- **wmctrl**: Window management (list, activate, move, resize).
- **xdotool**: Input simulation (click, type, key, mousemove).
- **dogtail**: UI tree extraction via AT-SPI (Accessibility bus).
- **scrot**: Lightweight screenshot tool.

搜索