Ykhjuy

In machine learning, people usually refer to the "performance of a model" or "performance of an optimizer". What is the exact definition of "performance"? What would be the "performance of an optimizer"?

I know that there are ways of measuring how far away current predictions of machine learning models are from the expected ones: for example, you can measure the accuracy of a model, perplexity, etc. Is this what the performance of a ML model refers to? Is it a name to refer to any way of measuring the correctness of the predictions (or, in general, outputs) of the model? Or is performance actually the time it takes to perform the prediction? Or something else?

In the absence of any specific clarifying context, "performance" is just a synonym for "quality."

The sentence "I want a model that performs better" is essentially the same as the sentence "I want a higher-quality model." Readers understand that the speaker is not satisfied with how well the model solves some particular problem, but the reader does not know, precisely, what about the model is dissatisfactory. Does the model predict too many false positives? Or false negatives? Does it predict incorrect classes for images that have a tilted horizon, or are taken on cloudy days? Understanding what about the model needs improvement would require further, specific elaboration.

Likewise, if someone says that Adam has better performance than another optimizer, they're making a claim that Adam does better at some task, which they would have to specify for it to be possible to assess the truthfulness of the claim. One way to assesses performance of an optimizer is how many iterations it takes to reach some neighborhood around a minimum; another, which is particular to machine learning classifiers, is how well the solutions obtained by an optimizer generalize to out-of-sample data.

As the other answer correctly points out, there is no universal definition or measurement of performance of a machine learning model. Rather, performance metrics are highly dependent on the domain and ultimate purpose of the model being built. Performance of an ML model is just "how good" it does at a particular task, but the definition of "good" can take many forms. A "good" model could be one that predicts well, one that trains quickly, one that finds a robust solution, or any combination of the above.

For example, an algorithm used for a medical screening test should be highly sensitive - we want to catch all possible cases of a disease, at the cost of misdiagnosing some people who aren't actually sick. These individuals can go on for further tests that may optimize other metrics like positive predictive value, indicating that a positive test result is likely a result of actually having the disease. Depending on the purpose of the test, we may want to put more weight on true positives/negatives at the cost of errors on the other side.

Performance can also be a function of the error measure used. Suppose your classifier outputs values on a continuous scale which are then thresholded for a binary classification. Do you care only if points are on the correct side of the boundary (accuracy measure)? Or do you care how badly you missed on the misclassified points (RMSE)? There is no universal best way to optimize performance.