Technology is evolving rapidly as the world becomes a digital society. Artificial intelligence (AI) now permeates various aspects of our lives through the development of virtual entities that mimic human behavior, cognition, and emotions, such as Character AI.
While most people are aware of these significant technology developments, many are unaware that the technology enabling these new innovations is also advancing. One sector that is having to progress quickly to keep up with these innovations is the data management industry. The amount of data created by modern applications and required to train these applications is massive. For example, every character inputted into GPT-3 accounts for approximately 4 bytes, with a typical request averaging at about 2000 characters. With millions of requests being made per day, databases that are capable of handling such demands are becoming increasingly sought after.
While the most common type of database used for handling these large demands is the NoSQL database, another database is becoming more widely used across industries: the vector database. This database stores, organizes, and searches for data in a new way that is much more in line with what is required by modern applications, such as chatbots and Character AI. In this post, we will cover the evolution of the vector database, how it works, and how it is being used.
The Evolution of the Vector Database
Despite being one of the most modern types of databases used today, the evolution of the vector database began in the 1970s through the development of DNA sequencing. The need to be able to store vast amounts of DNA data led to the idea of storing data on high-dimensional vectors. Throughout the 80s and 90s, the vector database was developed and used to store medical data.
In 2005, Stanford was using a vector database to search for annotations and sequence information commonly used in molecular biology. As the 2000s progressed, Locality Sensitive Hashing (LSH) was used to develop the vector database even further. LSH is a technique that approximates a similarity search by reducing the dimensionality of data while preserving local distances between points. The similarity search would become the core foundation of the vector database and what it is most associated with today.
How Modern Vector Databases Work
Vector databases differ from both relational databases, which store structured data in tables and columns, and NoSQL databases, which can store both structured and unstructured data in various formats, in that they store data on vectors. A vector is an ordered list or sequence of numbers that can represent any type of data, including text, image, audio, and video. Each number in the vector represents a specific feature or attribute of that data, and because a vector can be represented in any number of dimensions, it can have an infinite number of components.
This makes it ideal for complex data requirements, such as training generative-AI applications. The vectors are then stored in a multi-dimensional space in the database, where data points with similar attributes or characteristics naturally gravitate toward each other, forming clusters. It is this clustering that allows vector databases to perform a similarity search. Instead of looking for exact matches between identical vectors, a vector database uses a similarity search to identify vectors that reside close to the given query vector. This approach not only more closely aligns with the inherent nature of the data but also offers a speed and efficiency that a traditional search can’t match. This is why vector databases are ideal for applications that require the rapid and accurate matching of data based on similarity rather than exact values.
Use Cases of Vector Databases
Vector databases are increasing in the number of real-world use cases. Below are the two most common:
Training and Supporting AI Applications
Vector databases have become closely associated with AI due to their ability to both train AI applications and continually update them. The most well-known example of this is through chatbot interfaces that can provide immediate responses to common questions. These chatbots are being used to personalize communication in businesses, from recruitment to customer service. Vector databases that can convert customer queries and support documents into embeddings can quickly find relevant responses to inquiries made on AI applications. A vector database can also be easily updated by generating new vectors, which are used to keep the application up-to-date with the latest relevant information.
E-commerce Recommendations
Modern e-commerce platforms are using vector databases to enhance their recommendation engines. By converting product descriptions, user reviews, and user profiles into embeddings, the platform can perform similarity searches to recommend products that closely match user preferences. This not only creates personalized recommendations for individual customers, which can significantly increase user engagement and sales, but it also ensures that recommendations are always relevant. Other industries that use recommendation systems powered by vector databases are streaming services, to recommend films, music, and games, and social media, to suggest friends, groups, or content.
Vector databases have evolved from DNA sequencing in the 70s to become a mainstream database that is redefining how we store data and adapt to an AI-integrated world.
