building-chat-widgets

Building Chat Widgets

Create interactive widgets for AI chat with actions and entity tagging.

Quick Start

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  widgets: {
    onAction: async (action, widgetItem) => {
      if (action.type === "view_details") {
        navigate(`/details/${action.payload.id}`);
      }
    },
  },
});

Action Handler Types

Widget Lifecycle

1. Agent tool generates widget → yield WidgetItem
2. Widget renders in chat with action buttons
3. User clicks action → action dispatched
4. Handler processes action:
   - client: onAction callback in frontend
   - server: action() method in ChatKitServer
5. Optional: Widget replaced with updated state

Core Patterns

1. Widget Templates

Define reusable widget layouts with dynamic data:

{
  "type": "ListView",
  "children": [
    {
      "type": "ListViewItem",
      "key": "item-1",
      "onClickAction": {
        "type": "item.select",
        "handler": "client",
        "payload": { "itemId": "item-1" }
      },
      "children": [
        {
          "type": "Row",
          "gap": 3,
          "children": [
            { "type": "Icon", "name": "check", "color": "success" },
            { "type": "Text", "value": "Item title", "weight": "semibold" }
          ]
        }
      ]
    }
  ]
}

2. Client-Handled Actions

Actions that update local state, navigate, or send follow-up messages:

Widget Definition:

{
  "type": "Button",
  "label": "View Article",
  "onClickAction": {
    "type": "open_article",
    "handler": "client",
    "payload": { "id": "article-123" }
  }
}

Frontend Handler:

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  widgets: {
    onAction: async (action, widgetItem) => {
      switch (action.type) {
        case "open_article":
          navigate(`/article/${action.payload?.id}`);
          break;

        case "more_suggestions":
          await chatkit.sendUserMessage({ text: "More suggestions, please" });
          break;

        case "select_option":
          setSelectedOption(action.payload?.optionId);
          break;
      }
    },
  },
});

3. Server-Handled Actions

Actions that mutate data, update widgets, or require backend processing:

Widget Definition:

{
  "type": "ListViewItem",
  "onClickAction": {
    "type": "line.select",
    "handler": "server",
    "payload": { "id": "blue-line" }
  }
}

Backend Handler:

from chatkit.types import (
    Action, WidgetItem, ThreadItemReplacedEvent,
    ThreadItemDoneEvent, AssistantMessageItem, ClientEffectEvent,
)

class MyServer(ChatKitServer[dict]):

    async def action(
        self,
        thread: ThreadMetadata,
        action: Action[str, Any],
        sender: WidgetItem | None,
        context: RequestContext,  # Note: Already RequestContext, not dict
    ) -> AsyncIterator[ThreadStreamEvent]:

        if action.type == "line.select":
            line_id = action.payload["id"]  # Use .payload, not .arguments

            # 1. Update widget with selection
            updated_widget = build_selector_widget(selected=line_id)
            yield ThreadItemReplacedEvent(
                item=sender.model_copy(update={"widget": updated_widget})
            )

            # 2. Stream assistant message
            yield ThreadItemDoneEvent(
                item=AssistantMessageItem(
                    id=self.store.generate_item_id("msg", thread, context),
                    thread_id=thread.id,
                    created_at=datetime.now(),
                    content=[{"text": f"Selected {line_id}"}],
                )
            )

            # 3. Trigger client effect
            yield ClientEffectEvent(
                name="selection_changed",
                data={"lineId": line_id},
            )

4. Entity Tagging (@mentions)

Allow users to @mention entities in messages:

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  entities: {
    onTagSearch: async (query: string): Promise<Entity[]> => {
      const results = await fetch(`/api/search?q=${query}`).then(r => r.json());

      return results.map((item) => ({
        id: item.id,
        title: item.name,
        icon: item.type === "person" ? "profile" : "document",
        group: item.type === "People" ? "People" : "Articles",
        interactive: true,
        data: { type: item.type, article_id: item.id },
      }));
    },

    onClick: (entity: Entity) => {
      if (entity.data?.article_id) {
        navigate(`/article/${entity.data.article_id}`);
      }
    },
  },
});

5. Composer Tools (Mode Selection)

Let users select different AI modes from the composer:

const TOOL_CHOICES = [
  {
    id: "general",
    label: "Chat",
    icon: "sparkle",
    placeholderOverride: "Ask anything...",
    pinned: true,
  },
  {
    id: "event_finder",
    label: "Find Events",
    icon: "calendar",
    placeholderOverride: "What events are you looking for?",
    pinned: true,
  },
];

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },
  composer: {
    placeholder: "What would you like to do?",
    tools: TOOL_CHOICES,
  },
});

Backend Routing:

async def respond(self, thread, item, context):
    tool_choice = context.metadata.get("tool_choice")

    if tool_choice == "event_finder":
        agent = self.event_finder_agent
    else:
        agent = self.general_agent

    result = Runner.run_streamed(agent, input_items)
    async for event in stream_agent_response(context, result):
        yield event

Widget Component Reference

Layout Components

Content Components

Interactive Components

Critical Implementation Details

Action Object Structure

IMPORTANT: Use action.payload, NOT action.arguments:

# WRONG - Will cause AttributeError
action.arguments

# CORRECT
action.payload

Context Parameter

The context parameter is RequestContext, not dict:

# WRONG - Tries to wrap RequestContext
request_context = RequestContext(metadata=context)

# CORRECT - Use directly
user_id = context.user_id

UserMessageItem Required Fields

When creating synthetic user messages:

from chatkit.types import UserMessageItem, UserMessageTextContent

# Include ALL required fields
synthetic_message = UserMessageItem(
    id=self.store.generate_item_id("message", thread, context),
    thread_id=thread.id,
    created_at=datetime.now(),
    content=[UserMessageTextContent(type="input_text", text=message_text)],
    inference_options={},
)

Anti-Patterns

1. Mixing handlers - Don't handle same action in both client and server
2. Missing payload - Always include data in action payload
3. Using action.arguments - Use action.payload
4. Wrapping RequestContext - Context is already RequestContext
5. Missing UserMessageItem fields - Include id, thread_id, created_at
6. Wrong content type - Use type="input_text" for user messages

Verification

Run: python3 scripts/verify.py

Expected: ✓ building-chat-widgets skill ready

If Verification Fails

1. Check: references/ folder has widget-patterns.md
2. Stop and report if still failing

References

• references/widget-patterns.md - Complete widget patterns
• references/server-action-handler.md - Backend action handling