Integration with Neuron AI framework

[ilvalerione]

This PR introduces a dedicated class to enable the use of the PHPVector database in Neuron AI applications.

The integration class accepts an instance of VectorDatabase and performs the translation between Neuron RAG Documents to PHPVector documents and vice-versa.

Developers are free to configure the VectorDatabase instance with direct access to PHPVector APIs, and just pass the instance to the adapter class:

use PHPVector\Integrations\NeuronPHPVector;

class MyRAG extends RAG 
{
    ...

    protected function vectorStore(): VectorStoreInterface {
        return new NeuronPHPVector(
            database: new VectorDatabase(path: '/var/data/mydb'),
            topK: 5
        );
    }
}

Limitations

It seems PHPVector doesn't support document deletion yet, so the delete methods raise an exception. If PR #3 will be finally merged, this integration can be updated accordingly.

[ezimuel]

HI @ilvalerione thanks for the PR! I've a question, why this integration cannot be done in https://github.com/neuron-core/neuron-ai direcly? I would like to avoid adding the neuron-core/neuron-ai dependency in the PHPVector code base.

Moreover, we just merged #3 with the document deletion. Thanks again for your contribution.

[ilvalerione]

Totally understand, I took the opportunity to build the adapter in the first place. In the plan for the next major version of Neuron I would deprecate the internal FileVectorStore implementation in favor of other local DBs like PHPVector. Since so many different solutions are emerging, I would prefer these integrations to be optional and leave developers free to choose.

I can create a dedicated package under neuron-core organization to distribute this adapter. Feel free to close the PR for now.

[ezimuel]

Thanks @ilvalerione. Me or @danielebarbaro, the other maintainer, can help to contribute to the PHPVector package under the neuron-core. Ping us when you have the repository in place. We would love to contribute to your amazing project.

[ilvalerione]

Here it is: https://github.com/neuron-core/php-vector

Since you already merged the document deletion feature, it could be better to integrate it in this adapter before tagging the first release.

Let me know if you document this feature so I can eventually complete the implementation.

One question

I'm not sure if the initial database creation is different from subsequent incremental updates. If so, we should check if the database already exists and call load instead of simply construct the instance, and also call "save()" after document addition?

This could require do design the adapter in a different way.

[ezimuel]

@ilvalerione the load of a database from the disk is performed by VectorDatabase::open('/var/data/mydb'). The constructor of VectorDatabase doesn't load from disk. The documents are stored in the filesystem when added (using addDocument()). The $path passed in the constructor is just to indicate the folder path where to write the documents.
The save is needed to persist the BM25 index and the HNSW graph. The save() does not store the documents, only the data structure to perform BM25 and HNSW searches.

[danielebarbaro]

Hey @ilvalerione, @ezimuel

Regarding the deleteBy() implementation: VectorStoreInterface expects deletion by sourceType / sourceName, which are metadata attributes.

At the moment, PHPVector supports deleteDocument($id) (PR #3, already merged), but there’s no way to query documents by metadata to determine which ones should be deleted.

I have an open PR (#4) that introduces metadata filtering (metadataSearch(), MetadataFilter, etc.). Once that’s merged, deleteBy() could be implemented like this:

public function deleteBy(string $sourceType, ?string $sourceName = null): VectorStoreInterface
{
$filters = [MetadataFilter::eq('sourceType', $sourceType)];

if ($sourceName !== null) {
    $filters[] = MetadataFilter::eq('sourceName', $sourceName);
}

$results = $this->database->metadataSearch(filters: $filters);

foreach ($results as $result) {
    $this->database->deleteDocument($result->document->id);
}

return $this;

}

In the meantime, a temp workaround could be to maintain an internal map of sourceType:sourceName -> document IDs during addDocument(), and use that for deletion. This is less elegant, but would work without waiting for #4.

That said, I’d suggest waiting for #4 to land and then implementing deleteBy() properly.

If you have time to take a look at PR #4 and help move it forward, that would be super helpful ... I’d love to get it merged soon as I need it for a project I’m working on 😇

Happy to open a PR on neuron-core/php-vector once that’s done.