[EN] Jean Zay Supercomputer, Large Language Models - Nathan Cassereau, Hatim Bourfoune

I met with Nathan Cassereau and Hatim Bourfoune from IDRIS, a national computing centre for the CNRS (the national research centre in France). Nathan and Hatim work on the Bloom project, an open source large language model, which was created using the Jean-Zay supercomputer. 
Thanks to Nathan and Hatim I had the chance to take a look at the machine after our interview. 
LLMs and AI/ML in general have created a lot of excitement. Hatim said he got into AI/ML himself, and he highlighted a Coursera course run by Andrew Ng. 

Here are a few links:

• https://arxiv.org/abs/2211.05100 a paper on BLOOM on ArXiv
• https://github.com/ncassereau-idris/lm-evaluation-harness Evaluation of LM 
• https://github.com/dptrsa-300/start_with_bloom Getting started with BLOOM on GitHub
• https://huggingface.co/bigscience/bloom Summary on BLOOM from Huggingface
•  https://www.technologyreview.com/2022/07/12/1055817/inside-a-radical-new-project-to-democratize-ai/ a technology review on BLOOM by MIT
• https://towardsdatascience.com/run-bloom-the-largest-open-access-ai-model-on-your-desktop-computer-f48e1e2a9a32 another BLOOM article
• https://www.youtube.com/@CNRS-FIDLE YouTube channel by CNRS 
• https://github.com/NVIDIA/Megatron-LM Megatron LM library used in the project
• https://github.com/microsoft/DeepSpeed DeepSpeed library used in the project
• https://pytorch.org PyTorch library 
• https://www.genci.fr/en a national infrastructure to provide access to HPC (Grand Equipement National de Calcul Intensif) in France
• https://en.wikipedia.org/wiki/Jean_Zay brief summary of Jean Zay's life
• http://www.idris.fr/eng/jean-zay/jean-zay-presentation-eng.html The Jean Zay supercomputer at IDRIS/Paris-Saclay