feat(embeddings): add support for multiple embedding providers

This change introduces support for multiple embedding model providers beyond the default OpenAI, allowing users to configure providers like Google Vertex AI, Google Gemini, AWS Bedrock, and Azure OpenAI via the `DOCS_MCP_EMBEDDING_MODEL` environment variable (format: `provider:model_name`).

**Key Changes:**

*   **Embedding Factory (`EmbeddingFactory.ts`):**
    *   Refactored to dynamically instantiate LangChain embedding classes based on the specified provider.
    *   Added support for `vertex`, `gemini`, `aws`, and `microsoft` providers.
    *   Includes checks for required environment variables per provider.
*   **Dimension Handling (`FixedDimensionEmbeddings.ts`):**
    *   Introduced a new `FixedDimensionEmbeddings` wrapper class.
    *   This wrapper ensures all vectors match the database's fixed dimension (1536).
    *   Pads vectors smaller than 1536 with zeros.
    *   Truncates vectors larger than 1536 *only* if `allowTruncate` is true (currently enabled for the `gemini` provider, which supports MRL).
    *   Throws a `DimensionError` if a non-truncatable model produces vectors > 1536.
*   **Factory Integration:**
    *   Updated `EmbeddingFactory.ts` to wrap the `gemini` provider's embeddings with `FixedDimensionEmbeddings(..., allowTruncate: true)`.
*   **Configuration (`.env.example`, `Dockerfile`):**
    *   Added necessary environment variables for all supported providers.
    *   Updated examples and comments.
*   **Testing:**
    *   Added comprehensive tests for `FixedDimensionEmbeddings.ts`.
    *   Updated tests for `EmbeddingFactory.ts` to cover new providers and the wrapper integration.
*   **Documentation (`README.md`, `ARCHITECTURE.md`):**
    *   Updated `README.md` to list supported providers, required environment variables, and simplified the vector dimension explanation.
    *   Updated `ARCHITECTURE.md` with details on the embedding factory, the `FixedDimensionEmbeddings` wrapper, and the dimension handling logic (padding, MRL truncation, errors).
    *   Removed examples using unsupported large-dimension models (e.g., `text-embedding-3-large`).

This enhancement provides greater flexibility in choosing embedding models while maintaining compatibility with the existing database schema.

Implements #28

Author: arabold

.clinerules

@@ -6,9 +6,8 @@
 - ALWAYS use the latest version of the programming language and libraries.
 - ALWAYS prefer the simplest solution.
 - When importing a relative path, avoid using file extensions like ".js" and ".ts".
-- ALWAYS add and update TSDoc for all classes, methods and functions. Focus on functionality and reasoning. Avoid documenting individual parameters or return values if their use can easily be derived from their name.
-- ALWAYS format Git commit messages as markdown.
-- ALWAYS adhere to the Conventional Commits specification for all Git commit messages
+- ALWAYS add and update TSDoc for all classes, methods and functions. Focus on functionality and reasoning.
+- NEVER document individual parameters or return values if their use can easily be derived from their name.
 
 ## Architecture Documentation Guidelines
 
@@ -19,3 +18,12 @@ Keep `ARCHITECTURE.md` high-level:
 - Use simple MermaidJS diagrams for visualization
 - Put implementation details in source code
 - Update when architecture changes
+
+## Git
+
+- The repository owner and name is `arabold/docs-mcp-server` on GitHub.
+- AWLAYS create new branches locally first before pushing them to the GitHub repository.
+- ALWAYS format Git commit messages as markdown.
+- ALWAYS adhere to the Conventional Commits specification for all Git commit messages
+- ALWAYS prefix branch names with the type of work being done, such as `feature/`, `bugfix/`, `chore/`, etc.
+- ALWAYS include the issue number in the branch name, such as `feature/1234-issue-name`.

.env.example

@@ -1,18 +1,43 @@
-# OpenAI Configuration
-# Required: Your OpenAI API Key
-OPENAI_API_KEY=your-key-here
+# Embedding Model Configuration
+# Optional: Format is "provider:model_name" or just "model_name" for OpenAI (default)
+# Examples:
+#   - openai:text-embedding-3-small (default if no provider specified)
+#   - vertex:text-embedding-004 (Google Cloud Vertex AI)
+#   - gemini:gemini-embedding-exp-03-07 (Google Generative AI)
+#   - aws:amazon.titan-embed-text-v1
+#   - microsoft:text-embedding-ada-002
+DOCS_MCP_EMBEDDING_MODEL=
 
-# Optional: Your OpenAI Organization ID (handled automatically by LangChain if set)
+# OpenAI Provider Configuration (Default)
+# Required for OpenAI provider or as fallback
+OPENAI_API_KEY=your-key-here
+# Optional: Your OpenAI Organization ID
 OPENAI_ORG_ID=
-
-# Optional: Custom base URL for OpenAI API (e.g., for Azure OpenAI or compatible APIs)
+# Optional: Custom base URL for OpenAI-compatible APIs (e.g., Ollama, Azure OpenAI)
 OPENAI_API_BASE=
 
-# Optional: Embedding model name (defaults to "text-embedding-3-small")
-# Must produce vectors with ≤1536 dimensions (smaller dimensions are padded with zeros)
-# Examples: text-embedding-3-small (1536), text-embedding-ada-002 (1536)
-# Note: text-embedding-3-large (3072) is not supported due to dimension limit
-DOCS_MCP_EMBEDDING_MODEL=
+# Google Cloud Vertex AI Configuration
+# Required for vertex provider: Path to service account JSON key file
+GOOGLE_APPLICATION_CREDENTIALS=/path/to/gcp-key.json
+
+# Google Generative AI (Gemini) Configuration
+# Required for gemini provider: Google API key
+GOOGLE_API_KEY=your-google-api-key
+
+# AWS Bedrock Configuration
+# Required for aws provider
+AWS_ACCESS_KEY_ID=your-aws-key
+AWS_SECRET_ACCESS_KEY=your-aws-secret
+AWS_REGION=us-east-1
+# Optional: Use BEDROCK_AWS_REGION instead of AWS_REGION if needed
+# BEDROCK_AWS_REGION=us-east-1
+
+# Azure OpenAI Configuration
+# Required for microsoft provider
+AZURE_OPENAI_API_KEY=your-azure-key
+AZURE_OPENAI_API_INSTANCE_NAME=your-instance
+AZURE_OPENAI_API_DEPLOYMENT_NAME=your-deployment
+AZURE_OPENAI_API_VERSION=2024-02-01
 
 # Optional: Specify a custom directory to store the SQLite database file (documents.db).
 # If set, this path takes precedence over the default locations.

ARCHITECTURE.md

@@ -153,6 +153,63 @@ graph TD
 
 The project uses SQLite for document storage, providing a lightweight and efficient database solution that requires no separate server setup.
 
+#### Embedding Generation
+
+Document embeddings are generated using a flexible provider system implemented in `src/store/embeddings/EmbeddingFactory.ts`. This factory supports multiple embedding providers through LangChain.js integrations:
+
+```mermaid
+graph TD
+    subgraph Input
+        EM[DOCS_MCP_EMBEDDING_MODEL]
+        DC[Document Content]
+    end
+
+    subgraph EmbeddingFactory
+        P[Parse provider:model]
+        PV[Provider Selection]
+        Config[Provider Configuration]
+        LangChain[LangChain Integration]
+    end
+
+    subgraph Providers
+        OpenAI[OpenAI Embeddings]
+        VertexAI[Google Vertex AI]
+        Bedrock[AWS Bedrock]
+        Azure[Azure OpenAI]
+    end
+
+    subgraph Output
+        Vec[1536d Vector]
+        Pad[Zero Padding if needed]
+    end
+
+    EM --> P
+    P --> PV
+    PV --> Config
+    Config --> LangChain
+    DC --> LangChain
+
+    LangChain --> |provider selection| OpenAI
+    LangChain --> |provider selection| VertexAI
+    LangChain --> |provider selection| Bedrock
+    LangChain --> |provider selection| Azure
+
+    OpenAI & VertexAI & Bedrock & Azure --> Vec
+    Vec --> |if dimension < 1536| Pad
+```
+
+The factory:
+
+- Parses the `DOCS_MCP_EMBEDDING_MODEL` environment variable to determine the provider and model
+- Configures the appropriate LangChain embeddings class based on provider-specific environment variables
+- Ensures consistent vector dimensions through the `FixedDimensionEmbeddings` wrapper:
+  - Models producing vectors < 1536 dimensions: Padded with zeros
+  - Models with MRL support (e.g., Gemini): Safely truncated to 1536 dimensions
+  - Other models producing vectors > 1536: Not supported, throws error
+- Maintains a fixed database dimension of 1536 for all embeddings for compatibility with `sqlite-vec`
+
+This design allows easy addition of new embedding providers while maintaining consistent vector dimensions in the database.
+
 **Database Location:** The application determines the database file (`documents.db`) location dynamically:
 
 1. It first checks for a `.store` directory in the current working directory (`process.cwd()`). If `.store/documents.db` exists, it uses this path. This prioritizes local development databases.
@@ -251,6 +308,44 @@ This hierarchy ensures:
    - Easy to add new tools
    - Simple to add new interfaces (e.g., REST API) using same tools
 
+## Testing Conventions
+
+This section outlines conventions and best practices for writing tests within this project.
+
+### Mocking with Vitest
+
+When mocking modules or functions using `vitest`, it's crucial to follow a specific order due to how `vi.mock` hoisting works. `vi.mock` calls are moved to the top of the file before any imports. This means you cannot define helper functions _before_ `vi.mock` and then use them _within_ the mock setup directly.
+
+To correctly mock dependencies, follow these steps:
+
+1.  **Declare the Mock:** Call `vi.mock('./path/to/module-to-mock')` at the top of your test file, before any imports or other code.
+2.  **Define Mock Implementations:** _After_ the `vi.mock` call, define any helper functions, variables, or mock implementations you'll need.
+3.  **Import the Actual Module:** Import the specific functions or classes you intend to mock from the original module.
+4.  **Apply the Mock:** Use the defined mock implementations to replace the behavior of the imported functions/classes. You might need to cast the imported item as a `Mock` type (`import { type Mock } from 'vitest'`).
+
+**Example Structure:**
+
+```typescript
+import { vi, type Mock } from "vitest";
+
+// 1. Declare the mock (hoisted to top)
+vi.mock("./dependency");
+
+// 2. Define mock function/variable *after* vi.mock
+const mockImplementation = vi.fn(() => "mocked result");
+
+// 3. Import the actual function/class *after* defining mocks
+import { functionToMock } from "./dependency";
+
+// 4. Apply the mock implementation
+(functionToMock as Mock).mockImplementation(mockImplementation);
+
+// ... rest of your test code using the mocked functionToMock ...
+// expect(functionToMock()).toBe('mocked result');
+```
+
+This structure ensures that mocks are set up correctly before the modules that depend on them are imported and used in your tests.
+
 ## Future Considerations
 
 When adding new functionality:

Dockerfile

@@ -30,13 +30,34 @@ RUN npm ci --omit=dev
 COPY --from=builder /app/dist ./dist
 RUN ln -s /app/dist/cli.js /app/docs-cli
 
-# Define the data directory environment variable and volume
-# Environment variables
+# Define environment variables with defaults
+# OpenAI (default provider)
+ENV OPENAI_API_BASE=""
+ENV OPENAI_ORG_ID=""
+
+# Google Cloud - Vertex AI
+ENV GOOGLE_APPLICATION_CREDENTIALS=""
+
+# Google Generative AI (Gemini)
+ENV GOOGLE_API_KEY=""
+
+# AWS Bedrock
+ENV AWS_ACCESS_KEY_ID=""
+ENV AWS_SECRET_ACCESS_KEY=""
+ENV AWS_REGION=""
+ENV BEDROCK_AWS_REGION=""
+
+# Azure OpenAI
+ENV AZURE_OPENAI_API_KEY=""
+ENV AZURE_OPENAI_API_INSTANCE_NAME=""
+ENV AZURE_OPENAI_API_DEPLOYMENT_NAME=""
+ENV AZURE_OPENAI_API_VERSION=""
+
+# Core configuration
 ENV DOCS_MCP_STORE_PATH=/data
-ENV OPENAI_API_BASE=
-ENV OPENAI_ORG_ID=
-ENV DOCS_MCP_EMBEDDING_MODEL=
+ENV DOCS_MCP_EMBEDDING_MODEL=""
 
+# Define volumes
 VOLUME /data
 
 # Set the command to run the application

README.md

@@ -28,14 +28,43 @@ The server exposes MCP tools for:
 
 ## Configuration
 
-The following environment variables are supported to configure the OpenAI API and embedding behavior:
+The following environment variables are supported to configure the embedding model behavior:
 
-- `OPENAI_API_KEY`: **Required.** Your OpenAI API key for generating embeddings.
-- `OPENAI_ORG_ID`: **Optional.** Your OpenAI Organization ID (handled automatically by LangChain if set).
-- `OPENAI_API_BASE`: **Optional.** Custom base URL for OpenAI API (e.g., for Azure OpenAI or compatible APIs).
-- `DOCS_MCP_EMBEDDING_MODEL`: **Optional.** Embedding model name (defaults to "text-embedding-3-small"). Must produce vectors with ≤1536 dimensions. Smaller dimensions are automatically padded with zeros.
+### Embedding Model Configuration
 
-The database schema uses a fixed dimension of 1536 for embedding vectors. Models that produce larger vectors are not supported and will cause an error. Models with smaller vectors (e.g., older embedding models) are automatically padded with zeros to match the required dimension.
+- `DOCS_MCP_EMBEDDING_MODEL`: **Optional.** Format: `provider:model_name` or just `model_name` (defaults to `text-embedding-3-small`). Supported providers and their required environment variables:
+
+  - `openai` (default): Uses OpenAI's embedding models
+
+    - `OPENAI_API_KEY`: **Required.** Your OpenAI API key
+    - `OPENAI_ORG_ID`: **Optional.** Your OpenAI Organization ID
+    - `OPENAI_API_BASE`: **Optional.** Custom base URL for OpenAI-compatible APIs (e.g., Ollama, Azure OpenAI)
+
+  - `vertex`: Uses Google Cloud Vertex AI embeddings
+
+    - `GOOGLE_APPLICATION_CREDENTIALS`: **Required.** Path to service account JSON key file
+
+  - `gemini`: Uses Google Generative AI (Gemini) embeddings
+
+    - `GOOGLE_API_KEY`: **Required.** Your Google API key
+
+  - `aws`: Uses AWS Bedrock embeddings
+
+    - `AWS_ACCESS_KEY_ID`: **Required.** AWS access key
+    - `AWS_SECRET_ACCESS_KEY`: **Required.** AWS secret key
+    - `AWS_REGION` or `BEDROCK_AWS_REGION`: **Required.** AWS region for Bedrock
+
+  - `microsoft`: Uses Azure OpenAI embeddings
+    - `AZURE_OPENAI_API_KEY`: **Required.** Azure OpenAI API key
+    - `AZURE_OPENAI_API_INSTANCE_NAME`: **Required.** Azure instance name
+    - `AZURE_OPENAI_API_DEPLOYMENT_NAME`: **Required.** Azure deployment name
+    - `AZURE_OPENAI_API_VERSION`: **Required.** Azure API version
+
+### Vector Dimensions
+
+The database schema uses a fixed dimension of 1536 for embedding vectors. Only models that produce vectors with dimension ≤ 1536 are supported, except for certain providers (like Gemini) that support dimension reduction.
+
+For OpenAI-compatible APIs (like Ollama), use the `openai` provider with `OPENAI_API_BASE` pointing to your endpoint.
 
 These variables can be set regardless of how you run the server (Docker, npx, or from source).
 
@@ -92,10 +121,54 @@ This is the recommended approach for most users. It's easy, straightforward, and
 Any of the configuration environment variables (see [Configuration](#configuration) above) can be passed to the container using the `-e` flag. For example:
 
 ```bash
+# Example 1: Using OpenAI embeddings (default)
+docker run -i --rm \
+  -e OPENAI_API_KEY="your-key-here" \
+  -e DOCS_MCP_EMBEDDING_MODEL="text-embedding-3-small" \
+  -v docs-mcp-data:/data \
+  ghcr.io/arabold/docs-mcp-server:latest
+
+# Example 2: Using OpenAI-compatible API (like Ollama)
 docker run -i --rm \
   -e OPENAI_API_KEY="your-key-here" \
-  -e DOCS_MCP_EMBEDDING_MODEL="text-embedding-3-large" \
-  -e OPENAI_API_BASE="http://your-api-endpoint" \
+  -e OPENAI_API_BASE="http://localhost:11434/v1" \
+  -e DOCS_MCP_EMBEDDING_MODEL="embeddings" \
+  -v docs-mcp-data:/data \
+  ghcr.io/arabold/docs-mcp-server:latest
+
+# Example 3a: Using Google Cloud Vertex AI embeddings
+docker run -i --rm \
+  -e OPENAI_API_KEY="your-openai-key" \  # Keep for fallback to OpenAI
+  -e DOCS_MCP_EMBEDDING_MODEL="vertex:text-embedding-004" \
+  -e GOOGLE_APPLICATION_CREDENTIALS="/app/gcp-key.json" \
+  -v docs-mcp-data:/data \
+  -v /path/to/gcp-key.json:/app/gcp-key.json:ro \
+  ghcr.io/arabold/docs-mcp-server:latest
+
+# Example 3b: Using Google Generative AI (Gemini) embeddings
+docker run -i --rm \
+  -e OPENAI_API_KEY="your-openai-key" \  # Keep for fallback to OpenAI
+  -e DOCS_MCP_EMBEDDING_MODEL="gemini:embedding-001" \
+  -e GOOGLE_API_KEY="your-google-api-key" \
+  -v docs-mcp-data:/data \
+  ghcr.io/arabold/docs-mcp-server:latest
+
+# Example 4: Using AWS Bedrock embeddings
+docker run -i --rm \
+  -e AWS_ACCESS_KEY_ID="your-aws-key" \
+  -e AWS_SECRET_ACCESS_KEY="your-aws-secret" \
+  -e AWS_REGION="us-east-1" \
+  -e DOCS_MCP_EMBEDDING_MODEL="aws:amazon.titan-embed-text-v1" \
+  -v docs-mcp-data:/data \
+  ghcr.io/arabold/docs-mcp-server:latest
+
+# Example 5: Using Azure OpenAI embeddings
+docker run -i --rm \
+  -e AZURE_OPENAI_API_KEY="your-azure-key" \
+  -e AZURE_OPENAI_API_INSTANCE_NAME="your-instance" \
+  -e AZURE_OPENAI_API_DEPLOYMENT_NAME="your-deployment" \
+  -e AZURE_OPENAI_API_VERSION="2024-02-01" \
+  -e DOCS_MCP_EMBEDDING_MODEL="microsoft:text-embedding-ada-002" \
   -v docs-mcp-data:/data \
   ghcr.io/arabold/docs-mcp-server:latest
 ```