Teams rebuilding a search layer say a version of this often: we moved to vector search, some queries work much better, and some broke completely.
That outcome is expected. Vector search is good at capturing semantic similarity and weak at exact term matching. Ask "how long is the return window" and it works beautifully. Search for "TX-4410-B" and it confuses that code with codes that look like it.
A meaningful share of enterprise search traffic is of the second kind: product codes, invoice numbers, order references, regulation clause numbers, error codes. A search layer that loses those queries is a regression for users, however well it captures meaning.
The two methods are not alternatives
Classic text search ranks with functions such as BM25, which account for term frequency and document length. This approach measures directly whether the searched term occurs in the document and is very strong on rare terms (Robertson and Zaragoza, 2009).
Vector search places text in a semantic space and compares the proximity of query and documents in that space. It can match without lexical overlap, connecting "no invoice was issued" with "my invoice has not arrived".
The right architecture is not to pick one but to run both. What we see across enterprise data sets is consistent: a hybrid setup beats both standalone text search and standalone vector search on almost every corpus.
The correct way to merge two result lists
The common mistake here is averaging the two methods' scores. It does not work, because the scales are incompatible. BM25 scores are unbounded positive numbers; cosine similarity moves in a narrow range. Adding them means silently letting one method dominate the other.
The widely adopted solution is to merge ranks rather than scores. Reciprocal Rank Fusion scores each document by its position in each list and sums the contributions. The method was defined in Cormack, Clarke and Buettcher's 2009 SIGIR paper, and the constant of 60 from the original work is still a reasonable starting point.
The advantage of rank-based fusion is that it needs no normalisation, and when a new retrieval source is added (title search or synonym expansion, for example) the same formula keeps working.
What Turkish content additionally requires
The assumption that a search layer can be built language-agnostically hits a wall quickly on Turkish content. A few concrete issues.
Agglutinative morphology. In a lexical index, the Turkish words for invoice, my invoice, my invoices' and invoicing are five separate terms. A user searching "I cannot see my invoice" may fail to match a document containing "invoice viewing". The text side needs a stemming or subword analysis chain configured for Turkish. Without it, the lexical leg of a hybrid setup performs far below expectation.
Turkish characters and case conversion. Users often type without Turkish characters. Storing both the original and a folded form at index time closes that gap.
Related to this is a classic software bug: performing lowercase conversion without a locale. In Turkish, the lowercase of capital I is dotless ı, and the lowercase of capital İ is i. Conversions that ignore the locale can break the match between "İSTANBUL" and "istanbul". When you see unexplained inconsistencies in search behaviour, this is one of the first places to look.
Organisation-specific abbreviations. Every organisation has jargon with no equivalent in general models. A synonym dictionary is the cheapest way to improve search quality. Fifty lines of synonyms often make more difference than a model change.
The permission filter has to run before the query
In enterprise search not every user should see every document. Where that filter is applied is a security question, not a performance one.
Architectures that gather results and then strip them are problematic for two reasons. First, a document the user cannot access has already been processed and summarised. Second, post-filtering leaves the list shorter than expected and disturbs relevance ordering.
The correct approach is to narrow the searchable document set to what the user can access before the query runs. Filtered search in vector indexes requires deliberate design: access groups written to the index as attributes, with filtering delegated to the search engine.
You cannot improve what you do not measure
Most search improvement work proceeds on intuition: an engineer tries a few queries, results look better, the change ships. That method hides the queries that got worse while others improved.
Building an evaluation set is less work than people expect:
- Pull the 150 to 200 most frequent queries from your search logs
- Mark by hand what the correct result should be for each
- Run the set on every change and report two metrics: how often the correct result appears in the top k, and ranking quality such as nDCG
Click data can help but is not sufficient on its own, because users click the top result more regardless of quality. Optimising ranking on clicks can amount to confirming the ranking you already have.
Public benchmarks are a sensible starting point when choosing an embedding model; multi-task evaluation suites such as MTEB give a general sense of relative performance (Muennighoff et al., 2022). The ordering on your own data may differ, so make the final call with your own set.
Reranking is often the single highest-return step
Taking the top fifty results from hybrid retrieval and rescoring them with a model that evaluates query and document text together usually produces a clear quality gain. The cost is latency: rescoring fifty documents adds to search time.
The practical balance we settle on: in a search the user is waiting on directly, rerank the top twenty; in searches an assistant runs in the background, evaluate a wider set, because the user is already waiting for an answer.
Generating an answer does not replace results
Showing a short answer above the results is useful. Replacing the result list entirely with an answer removes the user's ability to verify.
The arrangement that works: a short answer, the sources it rests on, and the normal result list beneath. Users read the quick answer and go to the source if unconvinced. That structure also makes wrong answers easier to notice and report.
Metrics to track
- Share of queries returning zero results, and the list of those queries
- Click-through rate on the first result
- Rate at which a support ticket is opened after a search
- Average number of searches per intent, which shows how often users rewrite the same query
- Quality difference with reranking on and off
The zero-result list is the most concrete work item for a content team. Topics users search for and cannot find usually point at a missing document.
Checklist
- Do lexical and vector search run together
- Is merging score-based or rank-based
- Is stemming or subword analysis configured for Turkish
- Are character folding and locale-aware case conversion in place
- Do synonym and jargon dictionaries exist
- Does the permission filter run before the query
- Does an evaluation set exist and run on every change
- Are zero-result queries reviewed regularly
To see how the semantic search service connects over API, look at Smartsearch Services, and for grounded answer generation see the enterprise chatbot article.
References
- Robertson, S., Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval. staff.city.ac.uk
- Cormack, G. V., Clarke, C. L. A., Buettcher, S. (2009). Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. SIGIR '09. dl.acm.org
- Muennighoff, N. et al. (2022). MTEB: Massive Text Embedding Benchmark. arxiv.org