{"id":2223,"date":"2022-07-26T16:41:58","date_gmt":"2022-07-26T16:41:58","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2022\/07\/26\/what-is-an-exaflop\/"},"modified":"2022-07-26T16:41:58","modified_gmt":"2022-07-26T16:41:58","slug":"what-is-an-exaflop","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2022\/07\/26\/what-is-an-exaflop\/","title":{"rendered":"What Is an Exaflop?"},"content":{"rendered":"
\n

Computers are crunching more numbers than ever to crack the most complex problems of our time \u2014 how to cure diseases like COVID and cancer, mitigate climate change and more.<\/p>\n

These and other grand challenges ushered computing into today\u2019s exascale era when top performance is often measured in exaflops.<\/p>\n

So, What\u2019s an Exaflop?<\/b><\/h2>\n

An exaflop is a measure of performance for a supercomputer that can calculate at least 1018<\/sup> or one quintillion floating point operations per second.<\/p>\n

In exaflop, the exa- prefix means a quintillion, that\u2019s a billion billion, or one followed by 18 zeros. Similarly, an exabyte is a memory subsystem packing a quintillion bytes of data.<\/p>\n

The \u201cflop\u201d in exaflop is an abbreviation for floating point operations. The rate at which a system executes a flop in seconds is measured in exaflop\/s.<\/p>\n

Floating point refers to calculations made where all the numbers are expressed with decimal points.<\/p>\n

1,000 Petaflops = an Exaflop<\/b><\/h2>\n

The prefix peta- means 1015<\/sup>, or one with 15 zeros behind it. So, an exaflop is a thousand petaflops.<\/p>\n

<\/p>\n

\"The<\/p>\n

<\/a><\/p>\n

To get a sense of what a heady calculation an exaflop is, imagine a billion people, each holding a billion calculators. (Clearly, they\u2019ve got big hands!)<\/p>\n

If they all hit the equal sign at the same time, they\u2019d execute one exaflop.<\/p>\n

Indiana University, home to the Big Red 200<\/a> and several other supercomputers, puts it this way: To match what an exaflop computer can do in just one second, you\u2019d have to perform one calculation every second for 31,688,765,000 years.<\/p>\n

A Brief History of the Exaflop<\/b><\/h2>\n

For most of supercomputing\u2019s history, a flop was a flop, a reality that\u2019s morphing as workloads embrace AI.<\/p>\n

People used numbers expressed in the highest of several precision formats<\/a>, called double precision, as defined by the IEEE Standard for Floating Point Arithmetic. It\u2019s dubbed double precision, or FP64, because each number in a calculation requires 64 bits, data nuggets expressed as a zero or one. By contrast, single precision uses 32 bits.<\/p>\n

Double precision uses those 64 bits to ensure each number is accurate to a tiny fraction. It\u2019s like saying 1.0001 + 1.0001 = 2.0002, instead of 1 + 1 = 2.<\/p>\n

The format is a great fit for what made up the bulk of the workloads at the time \u2014 simulations of everything, from atoms to airplanes, that need to ensure their results come close to what they represent in the real world.<\/p>\n

So, it was natural that the LINPACK benchmark, aka HPL, that measures performance on FP64 math became the default measurement in 1993, when the TOP500 list of world\u2019s most powerful supercomputers debuted.<\/p>\n

The Big Bang of AI<\/b><\/h2>\n

A decade ago, the computing industry heard what NVIDIA CEO Jensen Huang describes as the big bang of AI<\/a>.<\/p>\n

This powerful new form of computing started showing significant results on scientific and business applications. And it takes advantage of some\u00a0 very different mathematical methods.<\/p>\n

Deep learning is not about simulating real-world objects; it\u2019s about sifting through mountains of data to find patterns that enable fresh insights.<\/p>\n

Its math demands high throughput, so doing many, many calculations with simplified numbers (like 1.01 instead of 1.0001) is much better than doing fewer calculations with more complex ones.<\/p>\n

That\u2019s why AI uses lower precision formats like FP32, FP16 and FP8. Their 32-, 16- and 8-bit numbers let users do more calculations faster.<\/p>\n

Mixed Precision Evolves<\/b><\/h2>\n

For AI, using 64-bit numbers would be like taking your whole closet when going away for the weekend.<\/p>\n

Finding the ideal lower-precision technique for AI is an active area of research.<\/p>\n

For example, the first NVIDIA Tensor Core GPU<\/a>, Volta, used mixed precision. It executed matrix multiplication in FP16, then accumulated the results in FP32 for higher accuracy.<\/p>\n

Hopper Accelerates With FP8<\/b><\/h2>\n

More recently, the NVIDIA Hopper architecture<\/a> debuted with a lower-precision method for training AI that\u2019s even faster. The Hopper Transformer Engine<\/a> automatically analyzes a workload, adopts FP8 whenever possible and accumulates results in FP32.<\/p>\n

When it comes to the less compute-intensive job of inference \u2014 running AI models in production \u2014 major frameworks such as TensorFlow and PyTorch support 8-bit integer numbers for fast performance. That\u2019s because they don\u2019t need decimal points to do their work.<\/p>\n

The good news is NVIDIA GPUs support all precision formats (above), so users can accelerate every workload optimally.<\/p>\n

Last year, the IEEE P3109 committee<\/a> started work on an industry standard for precision formats used in machine learning. This work could take another year or two.<\/p>\n

Some Sims Shine at Lower Precision<\/b><\/h2>\n

While FP64 remains popular for simulations, many use lower-precision math when it delivers useful results faster.<\/p>\n

<\/p>\n

\"Factors<\/p>\n

<\/a>

HPC apps vary in the factors that impact their performance.<\/figcaption><\/figure>\n

For example, researchers run in FP32 a popular simulator for car crashes, LS-Dyna from Ansys. Genomics is another field that tends to prefer lower-precision math.<\/p>\n

In addition, many traditional simulations are starting to adopt AI for at least part of their workflows. As workloads shift towards AI,\u00a0 supercomputers need to support lower precision to run these emerging applications well.<\/p>\n

Benchmarks Evolve With Workloads<\/b><\/h2>\n

Recognizing these changes, researchers including Jack Dongarra \u2014 the 2021 Turing award winner and a contributor to HPL \u2014 debuted HPL-AI<\/a> in 2019. It\u2019s a new benchmark that\u2019s better for measuring these new workloads.<\/p>\n

\u201cMixed-precision techniques have become increasingly important to improve the computing efficiency of supercomputers, both for traditional simulations with iterative refinement techniques as well as for AI applications,\u201d Dongarra said in a 2019 blog<\/a>. \u201cJust as HPL allows benchmarking of double-precision capabilities, this new approach based on HPL allows benchmarking of mixed-precision capabilities of supercomputers at scale.\u201d<\/p>\n

Thomas Lippert, director of the J\u00fclich Supercomputing Center, agreed.<\/p>\n

\u201cWe\u2019re using the HPL-AI benchmark because it\u2019s a good measure of the mixed-precision work in a growing number of our AI and scientific workloads \u2014 and it reflects accurate 64-bit floating point results, too,\u201d he said in a blog<\/a> posted last year.<\/p>\n

Today\u2019s Exaflop Systems<\/b><\/h2>\n

In a June report, 20 supercomputer centers around the world reported their HPL-AI results<\/a>, three of them delivering more than an exaflop.<\/p>\n

One of those systems, a supercomputer at Oak Ridge National Laboratory, also exceeded an exaflop in FP64 performance on HPL.<\/p>\n

<\/p>\n

\"Exaflop<\/p>\n

<\/a>

A sampler of the June 2022 HPL-AI results.<\/figcaption><\/figure>\n

Two years ago, a very unconventional system was the first to hit an exaflop<\/a>. The crowd-sourced supercomputer assembled by the Folding@home consortium passed the milestone after it put out a call for help fighting the COVID-19 pandemic and was deluged with donated time on more than a million computers.<\/p>\n

Exaflop in Theory and Practice<\/b><\/h2>\n

Since then, many organizations have installed supercomputers that deliver more than an exaflop in theoretical peak performance. It\u2019s worth noting that the TOP500 list reports both Rmax (actual) and Rpeak (theoretical) scores.<\/p>\n

Rmax is simply the best performance a computer actually demonstrated.<\/p>\n

Rpeak is a system\u2019s top theoretical performance if everything could run at its highest possible level, something that almost never really happens. It\u2019s typically calculated by multiplying the number of processors in a system by their clock speed, then multiplying the result by the number of floating point operations the processors can perform in one second.<\/p>\n

So, if someone says their system can do an exaflop, consider asking if that\u2019s using Rmax (actual) or Rpeak (theoretical).<\/p>\n

Many Metrics in the Exaflop Age<\/b><\/h2>\n

It\u2019s another one of the many nuances in this new exascale era.<\/p>\n

And it\u2019s worth noting that HPL and HPL-AI are synthetic benchmarks, meaning they measure performance on math routines, not real-world applications. Other benchmarks, like MLPerf<\/a>, are based on real-world workloads.<\/p>\n

In the end, the best measure of a system\u2019s performance, of course, is how well it runs a user\u2019s applications. That\u2019s a measure not based on exaflops, but on ROI.<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

https:\/\/blogs.nvidia.com\/blog\/2022\/07\/26\/what-is-an-exaflop\/<\/p>\n","protected":false},"author":0,"featured_media":2224,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/2223"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=2223"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/2223\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/2224"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=2223"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=2223"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=2223"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}