This blog is jointly written by Md Rahman, Arkaprabho Ghosh, Navin Bilwar, and Desh Shukla.<\/em><\/p>\nExecutive summary<\/h2>\n Cisco IT recently evaluated fine-tuning embedding models using NVIDIA Nemotron RAG fine-tuning recipe as part of an effort to improve retrieval accuracy for domain-specific enterprise data. The objective was not to redesign existing retrieval-augmented generation (RAG) systems, but to understand whether targeted embedding fine-tuning could materially improve semantic search quality with reasonable effort and fast turnaround. Through this experiment, Cisco was able to validate firsthand that embedding fine-tuning, combined with synthetic data generation, can deliver measurable accuracy gains within a short time frame. The experiment also demonstrated strong time-to-value, enabling rapid iteration and clear performance signals without long training cycles or extensive manual labeling. The reduced turnaround of only a few days to understand the immediate benefits was a key outcome of this collaboration.
Problem statement<\/h2>\n Prior to conducting this experiment, Cisco had conducted similar embedding fine-tuning experiments using earlier generation models and smaller scale infrastructure. These prior efforts required significant manual tuning of hyperparameters such as batch size and number of epochs, and results were often difficult to stabilize. Iteration cycles were long, making it challenging to explore different configurations or scale experiments. Despite some localized improvements, keyword search remained necessary for many domain-specific retrieval scenarios. There was also no standardized, end-to-end workflow that engineering teams could execute quickly and evaluate consistently across runs. Often, these efforts would take weeks to months of manual effort for uncertain gains.<\/p>\n
How the fine\u2011tuning went and time to value<\/h2>\n In this experiment, Cisco used the NVIDIA NeMo Retriever embedding finetuning recipe, leveraging synthetic data generation to produce training signals from existing corpora. The recipe runs through five distinct stages: synthetic data generation (SDG), data preparation with hard-negative mining, contrastive fine-tuning, BEIR evaluation, and ONNX model export. The workflow was able to run end-to-end successfully. All experiments ran on a single NVIDIA H200 143 GB GPU hosted within Cisco AI Pods built on Cisco UCS 885A systems. Finetuning runs completed within hours of training time, enabling rapid experimentation across multiple dataset sizes and configurations. The use of synthetic data generation eliminated the need for manual labeling, significantly reducing overhead. This approach allowed Cisco to iterate quickly, observe performance trends early, and validate whether embedding fine-tuning was worth further investment. The overall time-to-value was substantially shorter than previous efforts, with meaningful insights gained after only a small number of runs.<\/p>\n
The five-stage pipeline architecture:<\/p>\n
Timings based on ~925 documents \/ ~9,200 QA pairs \/ ~7,800 training examples on a single NVIDIA H200 GPU running on Cisco AI Pods with Cisco UCS 885A infrastructure. Actual duration scales with data volume.<\/em><\/p>\nAccuracy gains observed<\/h2>\n Across multiple experiments, the results showed consistent, measurable improvements. Fine-tuning the NVIDIA 1-billion-parameter NV-EmbedQA model on synthetic domain-specific data yielded gains across all retrieval metrics, with NDCG@1 gains of +7.1 to +7.3 absolute points (+9.9% to +11.1% relative). Recall@10 improved by up to +6.8 points (+8.5%), and MAP@10 by up to +6.5 points (+9.7%). Using an on-premise 120B-parameter LLM for synthetic data generation, the entire pipeline ran with zero external API costs and with the data staying completely on prem ensured data privacy. These gains held even as dataset size increased and retrieval tasks became more challenging. Importantly, improvements were observed on domain-specific queries that previously performed poorly with base embedding models. While these results represent an initial baseline rather than a fully optimized outcome, they provided strong confirmation that embedding fine-tuning can materially improve retrieval quality for enterprise-specific data.<\/p>\n
\u00a0<\/strong><\/h3>\n\u00a0 \u00a0Summary of experiments<\/strong><\/h3>\nTable 1.<\/strong> Retrieval performance comparison between the base embedding model and the contrastively fine-tuned model across two dataset sizes (334 and 925 documents). Fine-tuning consistently improves ranking quality across all BEIR evaluation metrics.<\/em><\/p>\n\u00a0 \u00a0<\/strong><\/h3>\n\u00a0 \u00a0Key Observations: <\/strong><\/h3>\n\nFine-tuning consistently improved retrieval quality across all metrics.<\/li>\n NDCG@1 showed the largest improvement in top-level relevance.<\/li>\n Gains were stable across the two dataset sizes (334 and 925 documents).<\/li>\n Improved Recall@10 and Map@10 gains indicative of better coverage and ranking than the base embedding model.<\/li>\n<\/ul>\nWhat surprised us<\/h2>\n The most unexpected finding was how quickly the recipe delivered actionable results. Within a few days of starting the experiment, we had measurable accuracy improvements \u2014 a stark contrast to previous efforts that took weeks to months. The synthetic data generation approach produced training signals of sufficient quality to drive meaningful gains without a single manually labeled example. We were also surprised by how well the improvements generalized across query types, including the rare-token identifier queries that had historically been the weakest point for semantic search.<\/p>\n
Next steps with engagement<\/h2>\n Building on these results, Cisco will continue<\/strong> working with NVIDIA to systematically push accuracy further. The next phase of work will focus<\/strong> on:<\/p>\n\nUsing<\/strong> a fixed evaluation set across runs so that metrics will be<\/strong> directly comparable<\/li>\nTuning<\/strong> the learning rate (trying default, half, and double) and increasing<\/strong> epochs from 3 to 5<\/li>\nScaling<\/strong> training data to ~100K QA pairs to find the saturation point for the domain<\/li>\nUsing<\/strong> a larger or higher-quality LLM for synthetic data generation to improve QA pair fidelity<\/li>\nApplying<\/strong> 10% warmup with cosine decay for more stable convergence<\/li>\nIncreasing<\/strong> hard-negative mining from 5 to 10 negatives per query for a stronger contrastive signal<\/li>\nRefining<\/strong> synthetic data generation prompts to better emphasize rare and domain-specific terms \u2014 bug IDs, product identifiers, firmware versions \u2014 where base models struggle most<\/li>\nExploring<\/strong> chunk-aware training: using real document chunks from a production vector database as the retrieval corpus, generating questions against those chunks via the LLM, and mapping each question to its positive chunk and hard-negative chunks \u2014 training the model on the same data distribution it will encounter<\/strong> in production, where answers may be<\/strong> buried in longer text and chunking strategies will vary<\/strong><\/li>\n<\/ul>\nLonger term, the engagement will expand<\/strong> to include re-ranker fine-tuning and broader retrieval optimization as part of a full end-to-end RAG improvement effort.<\/p>\n
![\"Retail](\"https:\/\/dmsretail.com\/RetailNews\/wp-content\/uploads\/2022\/05\/RETAIL-ONLINE-TRAINING-728-X-90.png\")
<\/a><\/p>![\"\"](\"https:\/\/blogs.cisco.com\/gcs\/ciscoblogs\/1\/2026\/03\/image11-768x164.png\")
![\"\"](\"https:\/\/blogs.cisco.com\/gcs\/ciscoblogs\/1\/2026\/03\/qkGx1a86-image21-768x224.png\")