Performance

Suppose you find yourself working with a dataset where performance is slower than you like? What does "slow" mean? In some cases, days of delay might be ok, e.g., for a final training run of a ML model In other cases, a few seconds may be too long, e.g., for an online visualization being rendered for a web page. So how do we systematically approach performance problems? 1) Find the bottleneck 2) Improve performance of bottleneck, e.g., by adding hardware, or redesigning software How do we quantify performance? [slide] Throughput: requests/time, for many waiting requests. Latency: time/request, for a single request. Are these always the inverse of each other? A: Sometimes E.g., 1 request takes 0.1 seconds of CPU, 1 CPU => tput 10 req/s. Often not inverses: Concurrency: with 2 CPUs, 0.1 second latency => tput 20 req/s. Which of these matters more? A: Depends on setting In most data science applications, focus is on latency, but if your work is being used in an online or multiuser setting, throughput may matter. Often a few hundred milliseconds of latency is immaterial, but may want to scale to very high throughputs. OTOH, adding seconds of latency, e.g., in a web server, is bad. And even 10s of milliseconds, if you are doing high frequency trading, may be super important. What do these look like as systems scale, typically? [slide: graph]: # users on x-axis, tput on y-axis, linear then flat (queueing). [slide: graph]: # users on x-axis, latency on y-axis, ~zero then linear (queueing).