Transitioning to Dual RTX Pro 6000s: Empowering Local AI Model Deployment
As advancements in language models continue at a rapid pace, the ability to efficiently run large AI models locally is transforming the landscape for researchers, developers, and enthusiasts alike. Recent developments involving Qwen 3.5 and 3.6 models have demonstrated astonishing capabilities, allowing near-frontier model deployment on prosumer-grade hardware. Motivated by these breakthroughs, I made the decision to upgrade my workstation from a single NVIDIA RTX 5090 to a dual NVIDIA RTX Pro 6000 setup. This article shares my experience with this upgrade, the challenges faced, and the performance insights gained.
From Consumer GPU to High-Performance Workstation
Initially, I attempted to utilize vllm, a popular library for running large language models, but encountered difficulties due to the architectural differences between consumer-grade GPUs and data center accelerators. My previous RTX 5090, while powerful, was limited in handling larger, more demanding models like DeepSeek V4 Flash. Transitioning to a pair of RTX Pro 6000 GPUs provided the necessary compute capacity to explore these larger models effectively.
Achieving High Throughput in Local Inference
After overcoming initial hurdles—such as misconceptions about hardware compatibility—I successfully configured my setup to run DeepSeek V4 Flash. Benchmarking revealed inference speeds of approximately 80 to 100 tokens per second, with full context handling and additional room for key-value (KV) cache (around 4 million tokens with L2 cache). This performance marks a significant improvement over previous dense Qwen models, especially for tasks centered around research, coding, and exploratory AI applications.
Practical Applications and Limitations
The upgraded hardware has enabled me to replace about 80-90% of my previous remote session workflows, largely eliminating dependence on cloud-based inference. While the cost of inference on DeepSeek models remains relatively low—making it an attractive option for local deployment—the experience offers more than just performance metrics. There’s a certain satisfaction in hearing the subtle whine of the GPUs during intensive operations or observing the power fluctuations, as the system draws around 1.2 kW under load—sometimes causing alarm sounds from smoke detectors when power limits are overlooked.
Reflections on Cost, Convenience, and Satisfaction
While investing in dual RTX Pro 6000 cards might not always be justifiable purely from a cost perspective, the tangible benefits in usability and immediate responsiveness outweigh the expense for dedicated enthusiasts and researchers. Running complex models locally provides not just speed, but a sense of control and satisfaction that enhances the research and development process.
Conclusion
The upgrade to dual RTX Pro 6000 GPUs represents a meaningful step forward for those seeking to run large language models in a local environment. Despite some initial technical challenges, the improved inference speeds, larger context handling, and the overall experience of local AI experimentation are well worth the effort. As model sizes and capabilities continue to grow, powerful workstation setups like this will become increasingly essential for cutting-edge AI research and development.
Note: Always ensure your hardware setup is properly configured, and be mindful of power management to avoid unexpected issues.
