Semantic search in the enterprise: full text, vectors, hybrid

01

What is semantic search, and how does it differ from full-text search?

Full-text search finds documents that contain the words of the query, and weights rare words more heavily. Semantic search instead compares meaning: the query and the document are represented as vectors, and vectors that sit close together count as similar in content. It therefore also finds documents that describe the same thing in different words.

Say a new employee searches the intranet for how do I request holiday. The actual policy is called absence request via the HR portal and mentions the word holiday only once, in a footnote. Full-text search ranks that page far down, or does not find it at all. Semantic search recognises that requesting holiday and an absence request mean the same thing, and moves the page to the top.

The other way round, a service technician searches for the fault code E-4711 or the standard DIN EN 60204-1. Here, literal matching is everything, and a vector that only knows the rough meaning of a string is more likely to return documents about similar faults than the one correct document. Both methods, then, have complementary strengths. Search itself is covered here; Retrieval-augmented generation explains how it gets embedded into a system that formulates answers, and Knowledge management with AI places it within the wider knowledge management picture.

02

How does classic full-text search work with BM25?

BM25 scores a document by how often the search terms appear in it, how rare those terms are across the whole store, and how long the document is. Repetitions count with diminishing weight, and long documents get penalised. The method is fast, explainable, needs no training, and remains a strong baseline.

The foundation is an inverted index: for every word it stores which documents it appears in. BM25, described in the survey by Robertson and Zaragoza (2009), computes a score for each document from three components.

1. Term rarity (inverse document frequency): a word that appears in almost every document contributes little to distinguishing between them, while a rare technical term contributes a lot.
2. Frequency within the document, with saturation: more occurrences raise the score, but with diminishing returns. The parameter k1 controls how quickly this saturation sets in.
3. Length normalisation: a long document simply contains more words by chance. The parameter b determines how strongly length gets compensated for.

In common search engines, BM25 is the default, in Elasticsearch for instance with k1 = 1.2 and b = 0.75 according to its documentation. The strengths are speed, traceability (it is possible to show which word triggered a match) and precision on exact terms. The weakness is the vocabulary problem: synonyms, rephrasings and other word forms are only found if text analysis normalises them beforehand. For German text, this analysis is exactly the decisive part, addressed in detail later.

03

How does vector search work with embeddings?

An embedding model translates every chunk of text into a vector, a long sequence of numbers. The query gets translated with the same model, and then an index searches for the closest vectors, usually using cosine similarity. Because document vectors are computed in advance, search stays fast even for large stores.

The decisive trick is separating query and document. A model that reads both together scores the match very precisely, but has to re-read every document for every query. Reimers and Gurevych (EMNLP, 2019) calculated that finding the most similar pair among 10,000 sentences this way takes around 65 hours, versus about 5 seconds with precomputed sentence vectors (Sentence-BERT). These models are called bi-encoders, the jointly reading ones cross-encoders; the latter get used in reranking.

To avoid comparing every vector with every other one, indexes use approximation methods. HNSW graphs (Malkov and Yashunin, 2016) are common, where search jumps across several levels from coarse to fine neighbourhoods. An adjustable trade-off applies here: more search effort means a higher probability of finding the actual nearest neighbours, but more latency and memory. More on embeddings is in the glossary.

04

Vector search or full-text search: which method finds what better?

Full-text search wins on exact terms, identifiers and rare technical words, vector search wins on rephrasings, synonyms and everyday-language questions. Independent benchmarks show that dense models are not reliably better than BM25 outside their training data. Hybrid search captures both strengths and is therefore the usual starting point for enterprise document stores.

The two benchmarks explain why blanket judgements fail in either direction. BEIR (Thakur et al., 2021) tested search models on datasets they were not trained for: reranking and late-interaction models achieved the best scores on average, while dense models often generalised worse than BM25. MTEB (Muennighoff et al., 2022) found no embedding method that led across many tasks and languages. For an organisation's own store, only a measurement with its own questions counts.

05

How do you combine both methods (hybrid search with Reciprocal Rank Fusion)?

Hybrid search runs keyword search and vector search in parallel and merges the two rankings. Reciprocal Rank Fusion does this by adding, for each document and each list, the value 1 divided by 60 plus rank. Documents that rank high in both lists win out. The two methods' scores do not need to be comparable for this to work.

The problem with combining them is that BM25 scores and similarity scores sit on completely different scales. They can be normalised and added with weights, but the weights then have to be tuned for every store. Reciprocal Rank Fusion sidesteps this by using only the ranks. Cormack, Clarke and Büttcher (SIGIR, 2009) set the constant at 60 in a preliminary study and found that the fusion outperformed the best individual systems almost consistently. The constant dampens the advantage of the very top spots, so that agreement between the lists counts for more than a single top ranking.

function rrf(rankings, k = 60) {
  const scores = new Map();
  for (const list of rankings) {
    list.forEach((id, index) => {
      const rank = index + 1;
      scores.set(id, (scores.get(id) ?? 0) + 1 / (k + rank));
    });
  }
  return [...scores.entries()].sort((a, b) => b[1] - a[1]);
}

// rrf([["A", "D", "B"], ["B", "C", "X", "A"]])

Document B wins, even though keyword search only ranked it third, because both methods consider it relevant. RAG-Fusion uses the same principle, merging several rephrasings of a question, see F-RAG (RAG-Fusion). In good systems, fusion is followed by reranking of the best candidates; more on hybrid search is in the glossary.

06

What is distinctive about German-language documents (compounds, inflection, umlauts)?

German text forms long compound words, inflects heavily, and spells umlauts inconsistently. Keyword search therefore needs decompounding, stemming, and normalisation of umlauts and the letter ß. Embedding models absorb much of this, but not all of it, which is why both sides should be tested against German questions.

Without text analysis, keyword search will not find Tasse (cup) inside Kaffeetasse (coffee cup), or Prüfung (exam) inside Prüfungen (exams). Search engines ship filters for this. Elasticsearch's Hyphenation Decompounder, for example, searches for possible sub-words using hyphenation patterns and keeps only those that appear in a word list; from Kaffeetasse it thereby also produces the entries Kaffee and tasse. Apache Lucene's GermanNormalizationFilter replaces ß with ss and ä, ö, ü with a, o, u, so that Müller and Mueller match the same entry.

07

What must a vector database do for enterprise search?

A vector database stores embeddings and finds approximate nearest neighbours quickly. For enterprise stores, what also counts is filtering by metadata and access rights during search, updating and deleting individual entries, hybrid queries, multiple vectors per document where needed, and a plannable rebuild when the embedding model changes.

A vector database can be a standalone system or an extension of an existing database or search engine. The choice of product matters less than whether the following properties are met for a given store. Filtering is especially tricky: if rights are filtered only after the neighbour search, ten matches might shrink to two, and the rest of the correct documents were never even checked. Access control in AI knowledge systems describes how rights get carried cleanly through search.

08

What is late interaction (ColBERT), and when does it pay off?

Late interaction, known from ColBERT (Khattab and Zaharia, SIGIR 2020), stores one vector per token instead of one vector per chunk, and only compares each query term against its best-matching passage at search time. This matches more finely than a single vector, but costs considerably more storage, which is why it is often used only for reranking.

A single vector averages an entire chunk down to one mean. Late interaction (late interaction) keeps the individual parts and sums the best match for each query term. Carried over to page images, the same principle sits behind visual retrieval models, see Understanding documents with AI. Late interaction explained covers the setup with a fast first pass and precise rescoring.

09

How do you measure whether search is any good (Recall@k, nDCG)?

Search quality gets measured with a test set of real queries and known relevant documents. Recall@k states what share of the relevant documents falls within the first k results. nDCG additionally scores order and graded relevance: a highly relevant document at rank 1 counts for more than one at rank 8.

1. Gathering queries

   Real search queries from logs or conversations, including vague and misspelled ones.

2. Setting relevance

   Experts mark the matching documents for each query, ideally graded as highly, partially and not relevant.

3. Calculating metrics

   Recall@k and nDCG@k for every search variant on the same test set.

4. Comparing changes

   Every change to analysis, model or fusion runs against the same test set before it goes live.

A worked example: for one query there are three relevant documents, two of which fall within the first five results. Recall@5 then comes to 2 out of 3, or about 0.67. Recall only counts whether something was found, not where. The definitions appear in the textbook by Manning, Raghavan and Schütze (2008); the graded scoring with rank discount goes back to Järvelin and Kekäläinen (ACM TOIS, 2002). For RAG, Recall@k is the most important value, because the model only ever sees what got retrieved. Checking AI answers shows how whole answers get scored on top of that.