Scaling Up with Crusoe Cloud and VAST Data
Jua is a fast-growing AI startup based in Switzerland focused on transforming energy markets and weather forecasting through AI-driven weather models. Their cutting-edge models provide highly accurate weather predictions and enhanced risk management for energy traders.
To overcome the challenges of scaling their AI model training, Jua turned to Crusoe Cloud's high-performance GPU infrastructure. After trying hyperscaler GPU offerings, Crusoe Cloud proved to be a game-changer for Jua, providing them with the massive scalability they needed to manage petabyte-scale datasets. “Never again would I go back to a traditional hyperscaler for model training,” said Marvin Gabler, Co-Founder & CTO Jua.
Crusoe Cloud’s intuitive onboarding process and responsive support team made adoption of the platform seamless. Jua also leveraged Crusoe Cloud's comprehensive managed services, which allowed them to effortlessly spawn CPU machines for temporary MinIO clusters and utilize effective Terraform templates, further streamlining their workflows.
“We rely on Crusoe Cloud to train our models on massive amounts of data. Crusoe’s usability is a key differentiator,” said Gabler. “Their managed services free up precious resources to enable us to focus on our core expertise: building models.”
With Crusoe Cloud as its platform, Jua now needed an AI storage solution. The team initially relied on an on-premise cluster and then a self-managed MinIO object storage solution. However, the immense petabyte-scale data volumes required for training their sophisticated weather models proved difficult to manage. The latency associated with their object storage solution hindered training speeds, and the significant resources required to manage their MinIO cluster diverted their research team from core development tasks.
"The team worked four full weeks to optimize the whole data loading chain," explained Gabler. "And then around a quarter or half of a colleague’s time was spent on making sure that the MinIO cluster ran."
The company turned to VAST Data for its high-throughput AI storage solution, which proved transformative. VAST Data's ease of integration and high-speed data access eliminated the latency issues they faced with their previous setup, freeing up valuable engineering time.
"The story with VAST Data has been: we tested it, it's up, it just works – like no effort at all," Gabler noted. "The same extremely high performance. It frees up so much resources from my research team who can now focus on doing their real job."
"It really was a transition from being my main task towards something I have a look at every other weekend," said Henry Martin, a member of Jua's technical staff, highlighting the dramatic reduction in maintenance overhead.
The Results:
Significant Time Savings: Jua drastically reduced the time spent managing their AI storage infrastructure, freeing up their research team to focus on model development and innovation.
Enhanced Performance: VAST Data provided the high-throughput AI storage necessary to optimize training speeds, enabling faster model iterations.
Scalability and Reliability: Crusoe Cloud's enterprise-grade infrastructure and responsive support ensured reliable and scalable operations.
Focus on Innovation: By offloading infrastructure management, Jua could concentrate on developing their platform and expanding their market reach.
Strategic Recommendations:
Jua recommends that companies considering similar solutions prioritize managed services like Crusoe Cloud and VAST Data, especially in the early stages of growth. "Our recommendation would be to use managed solutions like Crusoe and VAST Data so that you can free up the resources to work on the actual task," he advised. They emphasized that managing complex infrastructure in-house requires significant expertise and resources, which can be better allocated to core business activities.
Looking Ahead:
With the technological hurdles overcome, Jua is now focused on building its platform and expanding its market reach within the energy sector and beyond. They are continuing to develop and deploy advanced weather models, leveraging the power of Crusoe Cloud and VAST Data to drive innovation and deliver value to their customers.
