What Are the Three Key Metrics Used to Measure Inference Speed?

[𝗔𝗜] The three key inference-speed metrics are:

• 𝗧𝗶𝗺𝗲 𝘁𝗼 𝗙𝗶𝗿𝘀𝘁 𝗧𝗼𝗸𝗲𝗻 (𝗧𝗧𝗙𝗧): how long it takes before the model starts responding
• 𝗧𝗼𝗸𝗲𝗻𝘀 𝗽𝗲𝗿 𝗦𝗲𝗰𝗼𝗻𝗱 (𝗧𝗣𝗦): how fast it generates output after it starts
• 𝗜𝗻𝘁𝗲𝗿-𝗧𝗼𝗸𝗲𝗻 𝗟𝗮𝘁𝗲𝗻𝗰𝘆 (𝗜𝗧𝗟): the delay between each generated token

In short: 𝗧𝗧𝗙𝗧 = 𝘀𝘁𝗮𝗿𝘁𝘂𝗽 𝘀𝗽𝗲𝗲𝗱, 𝗧𝗣𝗦 = 𝗼𝘂𝘁𝗽𝘂𝘁 𝘀𝗽𝗲𝗲𝗱, 𝗜𝗧𝗟 = 𝘀𝗺𝗼𝗼𝘁𝗵𝗻𝗲𝘀𝘀.