GPU使用率が上がらない理由、Adobeでは学習時間の2/3がデータ待ち

"At NVIDIA’s GTC conference , Adobe’s CTO Ely Greenfield walked the audience through the company’s three-year journey building frontier generative AI models from scratch."／"It revealed that roughly two-thirds of GPU time was spent simply waiting for data."／"“If we were putting a million dollars into training,” he told the room, “that was $600,000 we were burning away on GPUs sitting and doing nothing.” In other words, roughly sixty cents of every dollar spent on GPU compute was being wasted."（冒頭）— Backblaze より

"Greenfield’s team traced the waste to two culprits, neither of which is unique to Adobe: whether data can reach your GPUs fast enough, and whether the work it represents is distributed evenly once it arrives."／"Adobe’s training data lived in petabytes of distributed cloud storage and had to be shipped out to thousands of GPUs constantly, over standard Ethernet."／"Most of those checkpoints were never needed again. But writing and reading them still consumed real GPU time, which cost a lot of money whether they were used or not."／"they replaced standard Ethernet with a high-performance networking fabric designed for the petabyte-scale traffic distributed AI training generates."（Culprit #1 節）— Backblaze より

"Their pipeline used a straightforward data-parallelism approach: slice the training data into equal-sized chunks and assign one chunk to each GPU in the cluster."／"But equal-sized chunks weren’t equal work. Some GPUs got simple, low-resolution assets that processed in seconds; others got large, complex files that took minutes."／"They custom-built a balanced data loader that understood the processing cost of each asset and distributed work so every GPU finished at roughly the same time."／"The loader optimizes what happens once data is there, but storage determines whether it’s there at all."（Culprit #2 節）— Backblaze より

"After addressing both culprits, Adobe’s GPUs ran at roughly 80% utilization. After accounting for the coordination overhead inherent to running thousands of machines together, 80% is close to the practical ceiling."／"And building something like Adobe’s balanced data loader takes real engineering investment, including time spent profiling workloads, testing distribution strategies, and tuning until every GPU finishes at roughly the same pace."／"Teams still fighting storage bottlenecks rarely get to that work."（From 40 to 80 cents 節）— Backblaze より