AI Search and Retrieval

TF-IDF (Term Frequency-Inverse Document Frequency) is a statistical method for measuring how significant a term is within a specific document relative to a larger collection of documents. It combines two signals: how often a term appears in a document (term frequency), and how rare that term is across the full collection (inverse document frequency). Terms…

BM25 (Best match 25) is a keyword-based ranking algorithm used in search systems to score how relevant a document is to a query. It works by analyzing term frequency, such as how often a search term appears in a document. It balances that that against how common the term is across the entire document collection.…

Reranking is a step in AI search and retrieval pipelines that reorders an initial set of retrieved results to improve their relevance before they are passed to a language model or returned to a user. An initial retrieval step (typically semantic search) casts a wide net and returns a set of candidate results. Reranking then…

Semantic search is a search approach that retrieves results based on the meaning of a query rather than exact keyword matches. Instead of looking for documents that contain the specific words a user typed, semantic search looks for content that is conceptually relevant, capturing intent and context rather than surface-level terms.

A vector database is a type of database designed to store, index, and search embeddings. Where a traditional database searches for exact or pattern-matched values, a vector database searches by similarity, finding content that is conceptually close to a query even when the words don’t match.

Embeddings are numerical representations of text or other content like images or audio that capture meaning in a form a machine can process and compare. When text is converted into an embedding, it becomes a series of numbers that position that text in a high-dimensional space. Content that is semantically similar ends up positioned close…

Retrieval-augmented generation (RAG) is a technique that combines information retrieval with text generation. Instead of relying solely on what a language model learned during training, a RAG system first retrieves relevant content from an external source and then uses that content to inform the generated response.