Would this work at scale?

[seabass011]

I'm wondering if I could use autofaiss + pyspark to store ~100 billion vectors. I read in a thread on the faiss package that milvus is just basically faiss, but already distributed. I was wondering if I could use autofaiss and then distribute the data across a bunch of nodes.

Do you think this is a reasonable solution if I need to store a ton of vectors?

[rom1504]

https://github.com/facebookresearch/faiss/wiki/Indexing-1T-vectors yes