Harnessing a 1.5 Billion Parameter Language Model to Simplify Shell Command Generation
For over a decade, I’ve found myself repeatedly searching for the right tar flags to extract compressed archives—typing in queries like “tar extract gz” countless times. Frustrated by the repetitive nature of this task, I finally decided to explore an alternative solution: training an AI model to generate shell commands based on natural language prompts.
Developing a Compact, On-Device AI Solution
I fine-tuned a language model, Qwen2.5-Coder-1.5B, using a dataset of 125,000 natural language-to-command pairs. The model was then merged and quantized to a Q4_K_M format, resulting in a 941MB file compatible with llama.cpp, enabling inference directly on my laptop without any need for a GPU or external API.
Running on my laptop with an Intel i5-11320H processor (4 threads), the model achieves approximately 31.9 tokens per second with a median response time of 0.59 seconds per query, while maintaining a modest 1.6GB of RAM usage. This setup means the AI operates entirely locally, preserving privacy and eliminating dependency on external services.
Performance and Capabilities
While not matching the performance of larger models like GPT-4, this 1.5B parameter model delivers surprisingly effective results. Its accuracy is reflected in a score of 0.620 on the InterCode-ALFA benchmark, slightly above the untuned 7B variant, which scores around 0.613, yet well below GPT-4o at 0.73. Despite being a lightweight model, it provides answers comparable to larger models at a fraction of the size, all on a standard CPU.
Safety Measures and Usage
Given the potential risks—such as generating destructive commands that could wipe a system—I integrated a static safety checker. It includes 304 regression cases to prevent the model from issuing harmful commands, especially when prompted explicitly to do so.
You can access the code and models through the following links:
– Project Repository
– Pre-trained Weights
Both are licensed under Apache 2.0, encouraging community collaboration. If you’re interested in improving the approach or have ideas for enhancing safety and performance, I welcome feedback through comments or pull requests. And if you find this project useful, a star on GitHub would be greatly appreciated.
In summary, this project demonstrates that with careful fine-tuning and quantization, lightweight language models can be effectively run on consumer hardware to assist with technical tasks—bringing powerful AI capabilities directly to your own machine, without reliance on cloud services.
