Jensen Huang, CUDA and the Rise of NVIDIA

Introduction

When Jensen Huang co-founded NVIDIA in 1993, its goal was to bring 3D graphics to computer games and multimedia. Today, the same company's processors are used to train and run artificial intelligence models. To understand that shift, it helps to start with Huang's engineering background and the software platform NVIDIA built, CUDA. ^[1]

1. Huang's engineering career

Born in Taiwan, Huang earned a bachelor's degree in electrical engineering from Oregon State University in 1984. He worked as a microprocessor designer at AMD from 1984 to 1985, then held several positions at LSI Logic from 1985 to 1993. He became director of Coreware, the division responsible for the company's strategy to integrate multiple components of a computer system onto a single chip. During that period, he also earned a master's degree in electrical engineering from Stanford University in 1992. ^[2]

On 5 April 1993, Huang founded NVIDIA with Chris Malachowsky and Curtis Priem. He has served as president, chief executive officer and a board member since the company began. In 1999, NVIDIA held its initial public offering. That same year, it introduced the GeForce 256, a milestone in the company's history of graphics processing units (GPUs). ^[1]

2. What is CUDA?

NVIDIA introduced CUDA in 2006 as a software platform and programming model for performing many calculations in parallel on its GPUs. The GPU is the hardware that performs the calculations. CUDA provides the programming tools developers use to run their calculations on that hardware. This opened up GPUs built for graphics to applications in scientific research, high-performance computing and artificial intelligence. ^[1]

Imagine brightening an image with a million pixels. The calculation for one pixel can proceed independently of the others. A CUDA program can divide that work into smaller calculations and run them on the GPU. A function that runs on the GPU is called a kernel. Many threads execute it in parallel, each working on its assigned portion of the data. The same principle of dividing work into parts that can run in parallel applies to AI calculations such as large matrix multiplications. ^[3]

3. How CUDA helped the AI industry

When training a neural network, the model first makes a prediction. The training process measures the error and updates the model's parameters. It repeats this process many times. GPUs speed up operations such as matrix multiplication, giving researchers time to run more experiments and making it practical to train larger models. GPUs are also used after training, during inference, when a model such as a language model generates a response for a user. ^[4]

AlexNet offered a practical demonstration in 2012. Trained on NVIDIA GPUs, it won the ImageNet image classification competition. Its performance showed how graphics processors used for computer games could also train models to recognise images. ^[1]

4. How CUDA gave NVIDIA a lasting advantage

NVIDIA built optimised libraries and tools for calculations developers need frequently. cuBLAS accelerates matrix operations, cuDNN supports common neural network operations, NCCL handles communication across GPUs, and TensorRT helps trained models run faster. Frameworks such as PyTorch support CUDA, so developers can use these capabilities without writing every GPU operation from scratch. ^[5]

Conclusion

That software ecosystem strengthened NVIDIA's hardware business. A team can move from a small experiment to a system running across many GPUs while continuing to use familiar software and tools. Switching hardware can require the team to review its CUDA dependencies, adapt parts of its software and test performance again. By investing in software alongside hardware for years, NVIDIA had tools ready for developers who already knew how to use them when demand for AI computing grew. ^[1]

References

1. https://www.nvidia.com/en-us/about-nvidia/corporate-timeline/
2. https://s201.q4cdn.com/141608511/files/doc_financials/annual/2013/NVDA_2013AR_Proxy_10K-FINAL.pdf
3. https://docs.nvidia.com/cuda/cuda-programming-guide/02-basics/writing-cuda-kernels.html
4. https://docs.nvidia.com/cuda/cuda-programming-guide/index.html
5. https://developer.nvidia.com/machine-learning

By Keshavan Thiagaraja