The interviewer is testing trade-off knowledge. You can’t just “add more compute.” They want caching, approximate methods, precomputation, and architectural shortcuts.
Askgenai In Answered question
- Query Cache: Cache embedding vectors for frequent queries (Redis/Memcached). Exact match or semantic cache (FAISS for similar queries).
- Prefetching: Pre-compute embeddings for hot documents and store in memory.
-
Smaller Models: Use distilled embedding models (
bge-small-en,all-MiniLM-L6-v2) instead of large ones. -
Approximate Search: HNSW with tuned
ef_constructionandef_searchparameters. Accept slight recall loss for speed. - Async Generation: Stream LLM tokens while retrieval happens in parallel.
- Edge Deployment: Run embedding model on edge (ONNX/TensorRT) to reduce network round-trip.
Askgenai In Answered question