yoto-smart-stream-testing

Yoto Smart Stream Testing

Comprehensive testing guide covering authentication testing, functional testing patterns, Playwright UI automation, and iterative test-and-fix development loops.

Overview

Testing strategy for Yoto Smart Stream focuses on:

1. Authentication Testing - Verifying login flows, session management, and OAuth integration
2. Functional Testing - API endpoints, business logic, and integration testing
3. Playwright Testing - UI automation for login workflows and end-to-end scenarios
4. Test-and-Fix Loops - Iterative development with automated validation

Test Philosophy

• Test First: Write tests before implementing features (TDD)
• Fast Feedback: Tests should run quickly and provide clear error messages
• Real Scenarios: Tests should mirror actual user workflows
• Automated: All tests should run in CI/CD without manual intervention
• Maintainable: Tests should be easy to understand and update

v0.3.0 Release Validation ✅

Playwright UI Testing Results:

• Dashboard Page: PWA registered, 2 players loaded, MQTT enabled, zero console errors
• Admin Page: Dark Mode widget visible and functional, all controls responsive, zero errors
• Login Page: Authentication form renders correctly, Dark Mode widget operational, zero errors
• Dark Mode Widget: "🌓 Activate dark mode" control verified on all pages
• Browser Console: No errors, warnings, or issues on any tested page
• Deployment: v0.3.0 stable on Railway, health checks passing, service running

Quick Start

Prerequisites

# Install development dependencies
pip install -e ".[dev]"

# Verify pytest is installed
pytest --version

# Verify Playwright is installed (for UI tests)
playwright --version

Run All Tests

# Run all tests with coverage
pytest --cov=yoto_smart_stream --cov-report=html

# Run specific test categories
pytest tests/test_auth.py          # Authentication tests
pytest tests/test_api.py           # API endpoint tests
pytest tests/test_login_flows.py   # Playwright UI tests

# Run with verbose output
pytest -v

# Run with output capture disabled (see print statements)
pytest -s

Basic Test Structure

import pytest
from fastapi.testclient import TestClient
from yoto_smart_stream.api.app import app

@pytest.fixture
def client():
    """Test client fixture."""
    return TestClient(app)

def test_health_endpoint(client):
    """Test health check endpoint."""
    response = client.get("/api/health")
    assert response.status_code == 200
    assert response.json()["status"] == "healthy"

def test_login_success(client):
    """Test successful login."""
    response = client.post("/api/user/login", json={
        "username": "admin",
        "password": "yoto"
    })
    assert response.status_code == 200
    assert "access_token" in response.json()

Reference Documentation

For detailed testing information, refer to:

• 📋 Testing Guide - Complete testing guide with unit tests, integration tests, and manual testing procedures
• 🔐 Login Workflows - Detailed documentation of authentication flows and Playwright UI testing patterns

Common Testing Patterns

1. Resolving Service URL

Pattern: Determine the correct service URL for testing

import os
import subprocess

def get_service_url():
    """Get service URL from environment or Railway CLI."""
    # Try environment variable first
    url = os.getenv("SERVICE_URL")
    if url:
        return url
    
    # Try Railway CLI
    try:
        result = subprocess.run(
            ["railway", "domain"],
            capture_output=True,
            text=True,
            check=True
        )
        domain = result.stdout.strip()
        return f"https://{domain}"
    except Exception:
        pass
    
    # Fall back to pattern
    env = os.getenv("RAILWAY_ENVIRONMENT", "develop")
    return f"https://yoto-smart-stream-{env}.up.railway.app"

# Use in tests
@pytest.fixture
def service_url():
    return get_service_url()

def test_service_health(service_url):
    response = requests.get(f"{service_url}/api/health")
    assert response.status_code == 200

2. UI Login (Playwright)

Pattern: Automate browser login for UI testing

import pytest
from playwright.sync_api import Page, expect

@pytest.fixture(scope="session")
def service_url():
    """Get service URL for testing."""
    return os.getenv("SERVICE_URL", "https://yoto-smart-stream-develop.up.railway.app")

def test_ui_login(page: Page, service_url: str):
    """Test login via UI."""
    # Navigate to login page
    page.goto(service_url)
    
    # Should redirect to login
    expect(page).to_have_url(f"{service_url}/login")
    
    # Fill login form
    page.fill('input[name="username"]', "admin")
    page.fill('input[name="password"]', "yoto")
    page.click('button[type="submit"]')
    
    # Should redirect to dashboard
    expect(page).to_have_url(f"{service_url}/")
    expect(page.locator("h1")).to_contain_text("Dashboard")

def test_oauth_button_visible(page: Page, service_url: str):
    """Test OAuth connect button appears after login."""
    # Login first
    page.goto(f"{service_url}/login")
    page.fill('input[name="username"]', "admin")
    page.fill('input[name="password"]', "yoto")
    page.click('button[type="submit"]')
    
    # Check OAuth button
    oauth_button = page.locator('button:has-text("Connect Yoto Account")')
    expect(oauth_button).to_be_visible()

3. API Login (Programmatic)

Pattern: Get JWT token for API testing

import requests

@pytest.fixture
def auth_token(service_url):
    """Get authentication token for API tests."""
    response = requests.post(
        f"{service_url}/api/user/login",
        json={"username": "admin", "password": "yoto"}
    )
    assert response.status_code == 200
    return response.json()["access_token"]

def test_authenticated_endpoint(service_url, auth_token):
    """Test endpoint requiring authentication."""
    headers = {"Authorization": f"Bearer {auth_token}"}
    response = requests.get(
        f"{service_url}/api/players",
        headers=headers
    )
    assert response.status_code == 200
    assert isinstance(response.json(), list)

4. Hybrid Testing (UI + API)

Pattern: Combine UI and API testing for complex scenarios

def test_complete_user_journey(page: Page, service_url: str):
    """Test complete user journey from login to playing audio."""
    # 1. Login via UI
    page.goto(f"{service_url}/login")
    page.fill('input[name="username"]', "admin")
    page.fill('input[name="password"]', "yoto")
    page.click('button[type="submit"]')
    
    # 2. Extract auth cookie/token from browser
    cookies = page.context.cookies()
    auth_cookie = next(c for c in cookies if c["name"] == "session")
    
    # 3. Use API to create content
    headers = {"Cookie": f"session={auth_cookie['value']}"}
    response = requests.post(
        f"{service_url}/api/cards",
        headers=headers,
        json={"title": "Test Card", "content": {...}}
    )
    assert response.status_code == 201
    card_id = response.json()["id"]
    
    # 4. Verify card appears in UI
    page.goto(f"{service_url}/library")
    expect(page.locator(f'[data-card-id="{card_id}"]')).to_be_visible()

5. Fixture-Based Testing

Pattern: Reusable test fixtures for common setups

import pytest
from playwright.sync_api import Page

@pytest.fixture
def logged_in_page(page: Page, service_url: str):
    """Fixture providing a logged-in browser page."""
    page.goto(f"{service_url}/login")
    page.fill('input[name="username"]', "admin")
    page.fill('input[name="password"]', "yoto")
    page.click('button[type="submit"]')
    expect(page).to_have_url(f"{service_url}/")
    return page

@pytest.fixture
def api_client(service_url: str):
    """Fixture providing authenticated API client."""
    response = requests.post(
        f"{service_url}/api/user/login",
        json={"username": "admin", "password": "yoto"}
    )
    token = response.json()["access_token"]
    
    class AuthClient:
        def __init__(self, base_url, token):
            self.base_url = base_url
            self.headers = {"Authorization": f"Bearer {token}"}
        
        def get(self, path):
            return requests.get(f"{self.base_url}{path}", headers=self.headers)
        
        def post(self, path, json):
            return requests.post(f"{self.base_url}{path}", headers=self.headers, json=json)
    
    return AuthClient(service_url, token)

# Use fixtures in tests
def test_dashboard_loads(logged_in_page: Page):
    """Test dashboard loads after login."""
    expect(logged_in_page.locator("h1")).to_contain_text("Dashboard")

def test_api_players_list(api_client):
    """Test listing players via API."""
    response = api_client.get("/api/players")
    assert response.status_code == 200

Test Organization

Directory Structure

tests/
├── __init__.py
├── conftest.py              # Shared fixtures
├── test_auth.py             # Authentication tests
├── test_api_endpoints.py    # API endpoint tests
├── test_login_flows.py      # Playwright UI tests
├── test_audio.py            # Audio management tests
├── test_cards.py            # MYO card tests
├── test_users.py            # User management tests
├── test_oauth.py            # OAuth flow tests
├── integration/             # Integration tests
│   ├── test_complete_flows.py
│   └── test_player_control.py
└── fixtures/                # Test data
    ├── sample_audio.mp3
    └── sample_icon.png

Test Categories

Unit Tests (test_*.py in root):

• Fast execution
• Isolated components
• Mocked dependencies
• High coverage target (>80%)

Integration Tests (integration/):

• Test component interactions
• Use real dependencies where possible
• Slower but more realistic
• Coverage target (>60%)

UI Tests (test_login_flows.py, test_ui_*.py):

• Playwright browser automation
• Real user workflows
• Slower execution
• Focus on critical paths

Test Pyramid

     /\
    /UI\         ← Few (critical workflows)
   /────\
  /INTEG\        ← Some (component interactions)
 /──────\
/  UNIT  \       ← Many (isolated components)
──────────

Best Practices

1. Test Naming

# Good: Descriptive test names
def test_login_with_valid_credentials_returns_token():
    ...

def test_login_with_invalid_password_returns_401():
    ...

# Bad: Vague test names
def test_login():
    ...

def test_case_1():
    ...

2. Arrange-Act-Assert

def test_create_user():
    # Arrange: Setup test data
    user_data = {
        "username": "testuser",
        "password": "testpass",
        "role": "user"
    }
    
    # Act: Execute the action
    response = client.post("/api/admin/users", json=user_data)
    
    # Assert: Verify the results
    assert response.status_code == 201
    assert response.json()["username"] == "testuser"
    assert response.json()["role"] == "user"

3. Test Isolation

# Good: Each test is independent
@pytest.fixture
def clean_database():
    db.clear()
    yield
    db.clear()

def test_create_user(clean_database):
    # Test creates user in clean database
    ...

def test_list_users(clean_database):
    # Test lists users in clean database
    ...

# Bad: Tests depend on each other
def test_create_user():
    global user_id
    user_id = create_user()

def test_delete_user():
    delete_user(user_id)  # Depends on previous test

4. Clear Error Messages

# Good: Descriptive assertions
def test_player_status():
    response = client.get("/api/players/123")
    assert response.status_code == 200, \
        f"Expected 200 but got {response.status_code}. Response: {response.text}"
    
    data = response.json()
    assert "online" in data, \
        f"'online' field missing from response. Got: {list(data.keys())}"

# Bad: Bare assertions
def test_player_status():
    response = client.get("/api/players/123")
    assert response.status_code == 200
    assert "online" in response.json()

5. Fixtures Over Setup/Teardown

# Good: Use fixtures
@pytest.fixture
def test_user(client):
    response = client.post("/api/admin/users", json={
        "username": "testuser",
        "password": "testpass"
    })
    user_id = response.json()["id"]
    yield user_id
    client.delete(f"/api/admin/users/{user_id}")

def test_with_user(client, test_user):
    # User automatically created and cleaned up
    ...

# Bad: Manual setup/teardown
def test_with_user(client):
    # Setup
    response = client.post("/api/admin/users", ...)
    user_id = response.json()["id"]
    
    try:
        # Test
        ...
    finally:
        # Teardown
        client.delete(f"/api/admin/users/{user_id}")

Running Tests

Local Development

# Run all tests
pytest

# Run specific test file
pytest tests/test_auth.py

# Run specific test
pytest tests/test_auth.py::test_login_success

# Run tests matching pattern
pytest -k "login"

# Run with coverage
pytest --cov=yoto_smart_stream --cov-report=html

# Run and stop on first failure
pytest -x

# Run with verbose output
pytest -v

# Run with print statements visible
pytest -s

# Run in parallel (requires pytest-xdist)
pytest -n auto

Playwright Tests

# Run Playwright tests
pytest tests/test_login_flows.py

# Run in headed mode (see browser)
pytest tests/test_login_flows.py --headed

# Run specific browser
pytest tests/test_login_flows.py --browser chromium
pytest tests/test_login_flows.py --browser firefox
pytest tests/test_login_flows.py --browser webkit

# Debug mode (opens Playwright Inspector)
PWDEBUG=1 pytest tests/test_login_flows.py

# Generate trace for debugging
pytest tests/test_login_flows.py --tracing on

CI/CD Testing

# Run in CI mode (non-interactive)
pytest --tb=short --maxfail=3

# Generate JUnit XML for CI reporting
pytest --junitxml=test-results.xml

# Generate coverage reports for CI
pytest --cov=yoto_smart_stream --cov-report=xml --cov-report=term

# Run with environment-specific config
SERVICE_URL=https://yoto-smart-stream-pr-61.up.railway.app pytest

Troubleshooting

Tests Failing Locally

Symptom: Tests pass in CI but fail locally (or vice versa)

Common Causes:

1. Environment Variables:

   # Check required environment variables
   echo $SERVICE_URL
   echo $YOTO_CLIENT_ID
   
   # Set for testing
   export SERVICE_URL=https://yoto-smart-stream-develop.up.railway.app
   

2. Database State:

   # Clear test database
   rm -f test_database.db
   
   # Or use fixture to ensure clean state
   @pytest.fixture(autouse=True)
   def clean_db():
       db.clear()
       yield
       db.clear()
   

3. Network Issues:

   # Test connectivity
   curl https://yoto-smart-stream-develop.up.railway.app/api/health
   
   # Check if service is running
   railway status
   

Playwright Tests Timing Out

Symptom: Playwright tests hang or timeout

Solutions:

1. Increase Timeout:

   @pytest.fixture(scope="session")
   def browser_context_args(browser_context_args):
       return {
           **browser_context_args,
           "timeout": 30000  # 30 seconds
       }
   

2. Wait for Elements:

   # Bad: No wait
   page.click("button")
   
   # Good: Wait for element
   page.wait_for_selector("button", state="visible")
   page.click("button")
   

3. Debug with Traces:

   # Generate trace
   pytest tests/test_login_flows.py --tracing on
   
   # View trace
   playwright show-trace test-results/trace.zip
   

OAuth Tests Failing

Symptom: Tests requiring OAuth fail with "Not authenticated"

Solutions:

1. Mock OAuth in Tests:

   @pytest.fixture
   def mock_oauth(monkeypatch):
       """Mock OAuth for testing."""
       def mock_get_players():
           return [{"id": "player-123", "name": "Test Player"}]
   
       monkeypatch.setattr("yoto_smart_stream.api.routes.get_players", mock_get_players)
   
   def test_with_oauth(client, mock_oauth):
       # OAuth is mocked
       response = client.get("/api/players")
       assert response.status_code == 200
   

2. Skip OAuth Tests:

   @pytest.mark.skipif(
       not os.getenv("OAUTH_CONFIGURED"),
       reason="OAuth not configured"
   )
   def test_real_oauth():
       # Only runs when OAuth is configured
       ...
   

Slow Test Suite

Symptom: Tests take too long to run

Solutions:

1. Run Tests in Parallel:

   pip install pytest-xdist
   pytest -n auto
   

2. Use Markers to Run Subsets:

   @pytest.mark.fast
   def test_fast():
       ...
   
   @pytest.mark.slow
   def test_slow():
       ...
   
   # Run only fast tests
   pytest -m fast
   

3. Mock External Services:

   # Instead of real Yoto API calls
   @pytest.fixture
   def mock_yoto_client(monkeypatch):
       monkeypatch.setattr("yoto_api.YotoClient", MockYotoClient)
   

Coverage Not Accurate

Symptom: Coverage report shows missing lines that are actually tested

Solutions:

1. Include Source in Coverage:

   # setup.cfg or pyproject.toml
   [tool.pytest.ini_options]
   testpaths = ["tests"]
   
   [tool.coverage.run]
   source = ["yoto_smart_stream"]
   omit = ["*/tests/*", "*/venv/*"]
   

2. Run Tests with Coverage:

   pytest --cov=yoto_smart_stream --cov-report=html
   open htmlcov/index.html
   

Test-and-Fix Development Loop

Iterative Development Pattern

1. Write Failing Test:

   def test_new_feature():
       response = client.post("/api/new-endpoint", json={...})
       assert response.status_code == 200
   

2. Run Test (Verify Failure):

   pytest tests/test_new_feature.py -v
   # Should fail with clear error
   

3. Implement Minimum Code:

   @app.post("/api/new-endpoint")
   def new_endpoint(data: dict):
       return {"status": "ok"}
   

4. Run Test Again:

   pytest tests/test_new_feature.py -v
   # Should pass
   

5. Refactor:

   • Improve implementation
   • Add error handling
   • Optimize performance

6. Run Full Test Suite:

   pytest --cov=yoto_smart_stream
   # Verify no regressions
   

Example: Adding New Endpoint

# Step 1: Write test
def test_get_player_volume(client, auth_token):
    headers = {"Authorization": f"Bearer {auth_token}"}
    response = client.get("/api/players/123/volume", headers=headers)
    assert response.status_code == 200
    assert "volume" in response.json()
    assert 0 <= response.json()["volume"] <= 100

# Step 2: Run test (fails)
# $ pytest tests/test_players.py::test_get_player_volume
# E   404 Not Found

# Step 3: Implement endpoint
@app.get("/api/players/{player_id}/volume")
async def get_player_volume(
    player_id: str,
    current_user: User = Depends(get_current_user)
):
    player = await get_player(player_id)
    return {"volume": player.config.volume}

# Step 4: Run test (passes)
# $ pytest tests/test_players.py::test_get_player_volume
# ✓ PASSED

# Step 5: Add more test cases
def test_get_player_volume_invalid_id(client, auth_token):
    headers = {"Authorization": f"Bearer {auth_token}"}
    response = client.get("/api/players/invalid/volume", headers=headers)
    assert response.status_code == 404

# Step 6: Run full suite
# $ pytest --cov=yoto_smart_stream
# ✓ All tests passed
# Coverage: 87%

Additional Resources

• Project Docs: See /docs folder for additional guides
• Service Operations: See yoto-smart-stream-service skill
• API Development: See yoto-smart-stream skill
• Example Tests: See /tests folder for test examples
• Playwright Docs: https://playwright.dev/python/docs/intro
• pytest Docs: https://docs.pytest.org/