Compute-Bound vs Memory-Bound

[𝗔𝗜] In AI, 𝗰𝗼𝗺𝗽𝘂𝘁𝗲-𝗯𝗼𝘂𝗻𝗱 means the chip is limited by how fast it can do math. 𝗠𝗲𝗺𝗼𝗿𝘆-𝗯𝗼𝘂𝗻𝗱 means it’s limited by how fast it can move data.

𝗖𝗼𝗺𝗽𝘂𝘁𝗲-𝗯𝗼𝘂𝗻𝗱
This is the usual bottleneck in 𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴. The GPU/TPU is busy crunching huge batches of matrix math, so performance depends on raw 𝗙𝗟𝗢𝗣𝗦.

𝗠𝗲𝗺𝗼𝗿𝘆-𝗯𝗼𝘂𝗻𝗱
This is the big issue in 𝗟𝗟𝗠 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲. Models generate 𝗼𝗻𝗲 𝘁𝗼𝗸𝗲𝗻 𝗮𝘁 𝗮 𝘁𝗶𝗺𝗲, and each token requires pulling the model’s weights through memory. The problem isn’t the math — it’s 𝗴𝗲𝘁𝘁𝗶𝗻𝗴 𝘁𝗵𝗲 𝘄𝗲𝗶𝗴𝗵𝘁𝘀 𝘁𝗼 𝘁𝗵𝗲 𝗰𝗼𝗺𝗽𝘂𝘁𝗲 𝗰𝗼𝗿𝗲𝘀 𝗳𝗮𝘀𝘁 𝗲𝗻𝗼𝘂𝗴𝗵.

Because on-chip memory is too small, the weights live in 𝗼𝗳𝗳-𝗰𝗵𝗶𝗽 𝗛𝗕𝗠, and the cores often sit idle waiting for data. So inference speed is capped by 𝗺𝗲𝗺𝗼𝗿𝘆 𝗯𝗮𝗻𝗱𝘄𝗶𝗱𝘁𝗵, not compute power.