The time it takes to generate an answer from an AI chatbot. The inference speed is the time between a user asking a question and getting an answer. It is the execution speed that people actually witness. Although considerably more time consuming (days, weeks and months), people never experience the training phases that developed the models that the inference engine uses. See
AI inference vs. training.