Hi! I'm one of the authors of this feature. It's something quite novel, because it's not just a HNSW plug-in for MySQL (like e.g. pgvector is for Postgres). It's a fully transactional index, integrated into InnoDB.
We based the implementation on two very new papers from Microsoft Research, SPANN and SPFresh. SPANN is a hybrid graph/tree algorithm that does a fantastic job of scaling larger-than-RAM indexes (https://arxiv.org/abs/2111.08566) and SPFresh expands upon it by defining a set of background operations that can be performed to maintain the index's performance and recall while it's continuously updated in-place (https://arxiv.org/html/2410.14452v1). The novel thing here is designing all the SPANN _and_ SPFresh operations to be transactional, and integrating them in MySQL's default storage engine.
This tight integration fundamentally means that inserting, updating and deleting vector data from MySQL is always reflected immediately in the index as part of committing your transaction. But it also means that the indexes are fully covered by the MySQL binlog; they recover from hard crashes just fine. They're also managed by MySQL's buffer pool, so they scale to terabytes of data, just like any other table. And also crucially, they're fully integrated with the query planner, so they can be used in any query, including JOINs and WHERE clauses (something that many other vector indexes really struggle with).
We plan to release a whitepaper on our transactionality extensions to SPFresh, which I think will be super interesting, but meanwhile please feel free to test the beta and ask me any questions (here, or by emailing PlanetScale support). Thanks!
Given that PlanetScale only operates in a handful of AWS and Google regions, isn't the latency brutal for any customer not hosting in 1 of the ~12 datacenter PlanetScale is in.
I work for another db vendor and we also have added vectors support. What’s crazy is that customers were not really asking for- just surfing the hype wave like everyone else. Why would someone store vectors in an SQL db rather than a dedicated vector db with support for native graph / vector semantics?
Will these converge on a common syntax for vector fields, indexes, and comparison functions in the near future? Or will vector implementations just add momentum to the increasing incompatibility in the MySQL-ish ecosystem?
tanoku ·6 hours ago
We based the implementation on two very new papers from Microsoft Research, SPANN and SPFresh. SPANN is a hybrid graph/tree algorithm that does a fantastic job of scaling larger-than-RAM indexes (https://arxiv.org/abs/2111.08566) and SPFresh expands upon it by defining a set of background operations that can be performed to maintain the index's performance and recall while it's continuously updated in-place (https://arxiv.org/html/2410.14452v1). The novel thing here is designing all the SPANN _and_ SPFresh operations to be transactional, and integrating them in MySQL's default storage engine.
This tight integration fundamentally means that inserting, updating and deleting vector data from MySQL is always reflected immediately in the index as part of committing your transaction. But it also means that the indexes are fully covered by the MySQL binlog; they recover from hard crashes just fine. They're also managed by MySQL's buffer pool, so they scale to terabytes of data, just like any other table. And also crucially, they're fully integrated with the query planner, so they can be used in any query, including JOINs and WHERE clauses (something that many other vector indexes really struggle with).
We plan to release a whitepaper on our transactionality extensions to SPFresh, which I think will be super interesting, but meanwhile please feel free to test the beta and ask me any questions (here, or by emailing PlanetScale support). Thanks!
Show replies
emmanueloga_ ·1 hours ago
Andy Pavlo, Jan 4, 2024
P.S.: R.I.P. Ottertune.
---
1: https://archive.is/rzMnq#selection-467.69-467.180
alberth ·7 hours ago
Show replies
Temporary_31337 ·1 hours ago
Show replies
cholmon ·4 hours ago
Will these converge on a common syntax for vector fields, indexes, and comparison functions in the near future? Or will vector implementations just add momentum to the increasing incompatibility in the MySQL-ish ecosystem?
Show replies