Blog

September 16, 2024

Training 1M Context Length Models on NVIDIA L40S GPUs

In the past month, the enthusiasm surrounding our long context models has sparked engaging discussions within the open-source community. While we had the chance to share some of our initial thoughts into scaling long context models, we’ve yet to explore the hardware that made it possible. Join us as we deep dive into our decision to leverage NVIDIA L40S GPUs and how our partners at Crusoe made this process seamless. This work was also previously featured in VentureBeat – “How Gradient created an open LLM with a million-token context window.”