Testing Guide

Comprehensive guide to Perpendicularity's test suite, CI/CD pipeline, and testing best practices.

📊 Test Coverage

Current Coverage: 82% (427 tests across 4 test suites)

Coverage Report:
-----------------
agent/          87%  (agent logic, models, tools)
cli/            93%  (command-line interface)
api/            88%  (FastAPI endpoints)
Overall:        82%

🎯 Test Organization

tests/
├── unit/                    # Fast unit tests (no I/O)
│   ├── test_config.py      # Configuration loading
│   ├── test_models.py      # Model initialization
│   ├── test_tools.py       # MCP tool management+
│   ├── test_react_agent.py # ReAct agent logic
│   ├── test_langgraph_agent.py  # LangGraph agent logic
│   ├── test_agent_factory.py   # Agent creation
│   └── test_output.py      # CLI output formatting
│
├── integration/             # Integration tests (requires MCP servers)
│   └── test_mcp_connection.py  # Live MCP server tests
│
├── cli/                     # CLI command tests
│   └── test_cli.py         # Click CLI testing
│
├── api/                     # API endpoint tests
│   └── test_api.py         # FastAPI endpoint testing
│
├── conftest.py             # Pytest fixtures and configuration
└── mcp_test_server.py      # Mock MCP server for testing

🚀 Quick Start

Run All Tests

# Using uv (recommended)
uv run pytest

# Using pytest directly
pytest

# With coverage
pytest --cov=agent --cov=cli --cov=api --cov-report=term-missing

Run Specific Test Suites

# Unit tests only (fast, no external dependencies)
pytest tests/unit/ -v

# Integration tests (requires MCP servers)
pytest tests/integration/ -v -m integration

# CLI tests
pytest tests/cli/ -v

# API tests
pytest tests/api/ -v

Run Individual Test File

# Single file
pytest tests/unit/test_config.py -v

# Single test
pytest tests/unit/test_config.py::TestAgentConfig::test_load_default_config -v

🏷️ Test Markers

Tests are organized using pytest markers:

Marker	Description	When to Use
`unit`	Fast unit tests	Default (no external dependencies)
`integration`	Tests requiring live MCP servers	Test real MCP integration
`cli`	CLI command tests	Test Click interface
`slow`	Tests with timeouts/delays	Tests that take >5 seconds

Usage:

# Run only unit tests
pytest -m unit

# Run only integration tests
pytest -m integration

# Skip slow tests
pytest -m "not slow"

# Run CLI tests
pytest -m cli

📝 Test Types

1. Unit Tests

Location: tests/unit/

Characteristics:

✅ Fast (< 1 second each)
✅ No network calls
✅ No file I/O
✅ Fully mocked external dependencies

Example:

# tests/unit/test_config.py

import pytest
from agent.config import AgentConfig

class TestAgentConfig:
    """Test configuration loading and validation."""
    
    def test_load_default_config(self):
        """Should load default configuration file."""
        config = AgentConfig("config/agent_config.yaml")
        assert config.config is not None
        assert "models" in config.config
        assert "agent" in config.config
    
    def test_get_default_model(self):
        """Should return configured default model."""
        config = AgentConfig("config/agent_config.yaml")
        default = config.get_default_model()
        assert default is not None
        assert isinstance(default, str)
    
    def test_invalid_config_path_raises_error(self):
        """Should raise error for non-existent config."""
        with pytest.raises(FileNotFoundError):
            AgentConfig("nonexistent.yaml")

Run:

pytest tests/unit/ -v

2. Integration Tests

Location: tests/integration/

Characteristics:

⚠️ Requires live MCP servers
⚠️ Slower (network latency)
⚠️ May fail if servers unavailable

Setup:

# Option A: Use test MCP server (included)
python tests/mcp_test_server.py &

# Option B: Configure real MCP servers
export TEST_MCP_GENOMIC_URL="http://localhost:8000/mcp"
export TEST_MCP_TXGEMMA_URL="http://localhost:8001/mcp"

Example:

# tests/integration/test_mcp_connection.py

import pytest
from agent.tools import MCPToolManager

@pytest.mark.integration
class TestMCPConnection:
    """Test live MCP server connections."""
    
    @pytest.mark.asyncio
    async def test_connect_to_genomic_ops(self):
        """Should connect to GenomicOps-MCP server."""
        config = {
            "servers": {
                "genomic_ops": {
                    "url": "http://test-server:8000/mcp",
                    "transport": "streamable-http"
                }
            }
        }
        
        manager = MCPToolManager(config)
        await manager.connect()
        
        assert len(manager.tools) > 0
        assert any("genomic" in tool.name.lower() for tool in manager.tools)
        
        await manager.disconnect()

Run:

pytest tests/integration/ -v -m integration

3. CLI Tests

Location: tests/cli/

Characteristics:

✅ Uses Click's test runner
✅ No real agent execution
✅ Mocked dependencies

Example:

# tests/cli/test_cli.py

from click.testing import CliRunner
from cli.main import cli

class TestAskCommand:
    """Test 'ask' CLI command."""
    
    def test_ask_command_basic(self):
        """Should accept question argument."""
        runner = CliRunner()
        result = runner.invoke(cli, ['ask', 'What is aspirin?', '--dry-run'])
        
        assert result.exit_code == 0
        assert 'aspirin' in result.output.lower()
    
    def test_ask_with_model_option(self):
        """Should accept --model flag."""
        runner = CliRunner()
        result = runner.invoke(cli, [
            'ask', 'test', 
            '--model', 'gemini',
            '--dry-run'
        ])
        
        assert result.exit_code == 0
        assert 'gemini' in result.output.lower()

Run:

pytest tests/cli/ -v

4. API Tests

Location: tests/api/

Characteristics:

✅ Uses FastAPI TestClient
✅ Mocked agent execution
✅ Tests HTTP endpoints

Example:

# tests/api/test_api.py

from fastapi.testclient import TestClient
from unittest.mock import MagicMock, AsyncMock, patch
from api.main import app

@pytest.fixture
def client():
    """FastAPI test client."""
    return TestClient(app)

class TestChatEndpoint:
    """Test /api/chat endpoint."""
    
    @patch('api.main.create_and_connect_agent')
    def test_chat_endpoint_accepts_post(self, mock_create_agent, client):
        """Should accept POST requests with question."""
        # Mock agent
        mock_agent = MagicMock()
        mock_agent.ask = AsyncMock(return_value="Aspirin is a pain reliever")
        mock_agent.disconnect = AsyncMock()
        mock_create_agent.return_value = mock_agent
        
        # Test request
        response = client.post("/api/chat", json={
            "question": "What is aspirin?",
            "agent_type": "langgraph",
            "model": "gemini",
            "stream": False
        })
        
        assert response.status_code == 200
        data = response.json()
        assert "answer" in data
        assert "aspirin" in data["answer"].lower()

Run:

pytest tests/api/ -v

🔧 Test Configuration

pytest.ini (embedded in pyproject.toml)

[tool.pytest.ini_options]
# Test discovery
testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]

# Output and behavior
addopts = [
    "--verbose",
    "--strict-markers",
    "--tb=short",
    "-ra",  # Show summary of all test outcomes
]

# Async support
asyncio_mode = "auto"

# Test markers
markers = [
    "unit: Fast unit tests with no I/O (default)",
    "integration: Tests using local test MCP server",
    "cli: CLI command tests using Click test runner",
    "slow: Tests involving timeouts or delays",
]

🤖 CI/CD Pipeline

GitHub Actions Workflow

Location: .github/workflows/tests.yml

Triggers:

Push to main or develop branches
Pull requests to main or develop

Matrix Testing:

Python versions: 3.11, 3.12
OS: Ubuntu latest

Steps:

name: Tests

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main, develop]

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest]
        python-version: ['3.11', '3.12']

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install uv
        run: |
          curl -LsSf https://astral.sh/uv/install.sh | sh
          echo "$HOME/.cargo/bin" >> $GITHUB_PATH

      - name: Cache dependencies
        uses: actions/cache@v4
        with:
          path: ~/.cache/uv
          key: ${{ runner.os }}-uv-${{ hashFiles('pyproject.toml') }}

      - name: Install dependencies
        run: uv sync --extra dev

      - name: Lint with ruff
        run: |
          uv run ruff check agent cli tests
          uv run ruff format --check agent cli tests

      - name: Run unit tests
        run: |
          uv run pytest tests/unit/ -v \
            --cov=agent --cov=cli \
            --cov-report=xml \
            --cov-report=term-missing

      - name: Run integration tests
        run: uv run pytest tests/integration/ -v -m integration

      - name: Run CLI tests
        run: uv run pytest tests/cli/ -v

      - name: Upload coverage
        uses: codecov/codecov-action@v4
        if: matrix.python-version == '3.12'
        with:
          file: ./coverage.xml

View Results:

Go to GitHub Actions tab
Check latest workflow run
View test results and coverage

📊 Coverage Reports

Generate Coverage Report

# Terminal report
pytest --cov=agent --cov=cli --cov=api --cov-report=term-missing

# HTML report (detailed)
pytest --cov=agent --cov=cli --cov=api --cov-report=html

# Open report
open htmlcov/index.html  # macOS
xdg-open htmlcov/index.html  # Linux

Coverage Configuration

[tool.coverage.run]
source = ["agent", "cli", "api"]

omit = [
    "*/tests/*",
    "*/__pycache__/*",
    "*/site-packages/*",
]

[tool.coverage.report]
exclude_lines = [
    "pragma: no cover",
    "def __repr__",
    "raise AssertionError",
    "raise NotImplementedError",
    "if __name__ == .__main__.:",
    "if TYPE_CHECKING:",
    "@abstractmethod",
]

🎓 Writing Tests

Best Practices

1. Use Descriptive Names

# ✅ Good
def test_agent_calls_mcp_tool_when_needed():
    """Agent should call appropriate MCP tool based on query."""
    pass

# ❌ Bad
def test_agent():
    pass

2. One Assertion Per Concept

# ✅ Good - focused tests
def test_config_loads_successfully():
    """Configuration should load without errors."""
    config = AgentConfig("config/agent_config.yaml")
    assert config.config is not None

def test_config_contains_models():
    """Configuration should have models section."""
    config = AgentConfig("config/agent_config.yaml")
    assert "models" in config.config

# ❌ Bad - multiple concepts
def test_config():
    config = AgentConfig("config/agent_config.yaml")
    assert config.config is not None
    assert "models" in config.config
    assert "agent" in config.config
    # Too much in one test

3. Mock External Dependencies

# ✅ Good - mocked
from unittest.mock import AsyncMock, patch

@patch('agent.tools.MCPToolManager')
async def test_agent_with_mocked_tools(mock_manager):
    """Agent should work with mocked tools."""
    mock_manager.return_value.tools = []
    # Test agent logic without real MCP servers
    pass

# ❌ Bad - real dependencies
async def test_agent_with_real_tools():
    """This requires actual MCP servers running!"""
    manager = MCPToolManager(real_config)
    await manager.connect()  # Fails if servers down

4. Use Fixtures for Common Setup

# conftest.py
import pytest

@pytest.fixture
def sample_config():
    """Provide sample configuration for tests."""
    return {
        "models": {"gemini": {"type": "gemini", "name": "gemini-2.5-flash"}},
        "agent": {"type": "langgraph", "max_steps": 5}
    }

# test_agent.py
def test_agent_initialization(sample_config):
    """Should initialize with sample config."""
    agent = Agent(sample_config)
    assert agent is not None

5. Test Error Cases

# Test happy path
def test_config_loads_valid_file():
    """Should load valid configuration."""
    config = AgentConfig("config/agent_config.yaml")
    assert config.config is not None

# Test error cases
def test_config_raises_on_missing_file():
    """Should raise FileNotFoundError for missing file."""
    with pytest.raises(FileNotFoundError):
        AgentConfig("nonexistent.yaml")

def test_config_raises_on_invalid_yaml():
    """Should raise ValueError for invalid YAML."""
    with pytest.raises(ValueError):
        AgentConfig("tests/fixtures/invalid.yaml")

🧪 Mocking Patterns

Mock Async Functions

from unittest.mock import AsyncMock

@pytest.mark.asyncio
async def test_async_function():
    """Test async function with AsyncMock."""
    mock_func = AsyncMock(return_value="result")
    result = await mock_func()
    assert result == "result"
    mock_func.assert_called_once()

Mock MCP Tools

from unittest.mock import MagicMock, AsyncMock

@pytest.fixture
def mock_mcp_manager():
    """Mock MCP tool manager."""
    manager = MagicMock()
    manager.connect = AsyncMock()
    manager.disconnect = AsyncMock()
    manager.tools = []
    return manager

Mock Agents

from unittest.mock import MagicMock, AsyncMock

@pytest.fixture
def mock_agent():
    """Mock LangGraph agent."""
    agent = MagicMock()
    agent.ask = AsyncMock(return_value="Mocked answer")
    agent.disconnect = AsyncMock()
    agent.set_step_callback = MagicMock()  # Sync method
    return agent

🐛 Debugging Tests

Run with Verbose Output

# Show full test names
pytest -v

# Show captured output
pytest -s

# Show local variables on failure
pytest -l

# Drop into debugger on failure
pytest --pdb

Run Specific Test with Debug

# Single test with all debug output
pytest tests/unit/test_config.py::TestAgentConfig::test_load_default_config -vsl

Use Print Statements

def test_with_debug():
    """Test with debug output."""
    config = AgentConfig("config/agent_config.yaml")
    print(f"Config: {config.config}")  # Shows in -s mode
    assert config.config is not None

📈 Test Metrics

Current Stats (v0.1.0)

Suite	Tests	Coverage	Status
Unit	206	76%	✅ Passing
Integration	11	28%	✅ Passing
CLI	114	83%	✅ Passing
API	97	75%	✅ Passing
Total	428	86%	✅ Passing

Coverage by Module

agent/__init__.py          100%
agent/agent_factory.py     100%
agent/config.py             88%
agent/langgraph_agent.py    91%
agent/models.py             75%
agent/react_agent.py        64%
agent/tools.py              88%

cli/__init__.py            100%
cli/main.py                 93%
cli/output.py               85%

api/__init__.py            100%
api/main.py                 75%

🎯 Testing Checklist

Before submitting a PR:

Run full check:

# Lint
uv run ruff check agent cli tests api
uv run ruff format --check agent cli tests api

# Test
uv run pytest --cov=agent --cov=cli --cov=api --cov-report=term-missing

# All passing? Ready to commit!

📚 Additional Resources

pytest Documentation - Testing framework
pytest-asyncio - Async testing
pytest-cov - Coverage plugin
unittest.mock - Mocking
Click Testing - CLI testing

Test early, test often! 🧪

For questions, see Contributing Guide or open an issue.

FilesExpand file tree

TESTING.md

Latest commit

History

TESTING.md

File metadata and controls

Testing Guide

📊 Test Coverage

🎯 Test Organization

🚀 Quick Start

Run All Tests

Run Specific Test Suites

Run Individual Test File

🏷️ Test Markers

📝 Test Types

1. Unit Tests

2. Integration Tests

3. CLI Tests

4. API Tests

🔧 Test Configuration

pytest.ini (embedded in pyproject.toml)

🤖 CI/CD Pipeline

GitHub Actions Workflow

📊 Coverage Reports

Generate Coverage Report

Coverage Configuration

🎓 Writing Tests

Best Practices

1. Use Descriptive Names

2. One Assertion Per Concept

3. Mock External Dependencies

4. Use Fixtures for Common Setup

5. Test Error Cases

🧪 Mocking Patterns

Mock Async Functions

Mock MCP Tools

Mock Agents

🐛 Debugging Tests

Run with Verbose Output

Run Specific Test with Debug

Use Print Statements

📈 Test Metrics

Current Stats (v0.1.0)

Coverage by Module

🎯 Testing Checklist

📚 Additional Resources