Welcome back nerds! In Part 1 we discussed how we regularly use numbers to assign meaning and why it’s important to ask the most basic questions about those numbers lest we be bamboozled. The example provided (wine consumption and breast cancer rates) wasn’t presented to exemplify an instance of trickery – the example was intended to demonstrate how to look at a number and make a better, more informed judgement about what it means. Saying something increases your risk of cancer by X is hardly informative if you don’t know what your chances are in the first place.
Now we are going to talk about some examples that highlight why you should always ask this question when presented with a metric – “how do you define that?”
Think of metrics you might hear on a daily basis. Customer satisfaction. Quality of hire. Vehicle safety. Website Impact. All of these metrics mean something – or at least they are intended to mean something. But if we don’t ask the most basic question – how do you define that – what we think it means and what it is actually measuring could be completely different. What if I told you Website Impact is a metric I developed to understand how important my website is to visitors. That might make sense. But what if I told you I assess that website impact by number of views. Do views equal importance? This is a simple example of why understanding how a metric is defined is key to not only understanding the metric, but understanding what factors might influence the metric and how the metric could be improved, manipulated, and just plain wrong. Let’s talk about some other examples below.
As I was driving past a hospital one day, I noticed a roadside sign flashing the current Emergency Room wait time. With red lights against a black backdrop, the sign alerted me that should I suddenly lose power-steering and dislocate my shoulder in an evasive maneuver to avoid a motley crew of dockless scooter ruffians (stick with me) I’d only have to wait 9 minutes at the ER! Now, I have been to the ER more times than I would like to admit. Somewhere in my genetic code is a marker that indicates a propensity for calamitous accidents. If I recall anything from those visits, it is that a trip to the ER is an all day affair; nine minutes would have been an unheard of record. But before you decide to track this hospital down to alleviate your ER woes – let’s talk about metrics. Specifically, and remember this term because it has multiple syllables and will make you sound oh so smart, let’s talk about operationalization (8 syllables yeah!).
Hospitals, like any other business, want to know how well they are performing various tasks. Invariably, when some measure of performance is desired numbers are typically involved, and rightly so. A number is intuitive, comparable, and informative. I know what minutes are so nine minutes immediately makes sense to me (intuitive). I know nine minutes is less than 15 minutes but greater than five minutes (comparable). Lastly, I know how long I’ve waited in the ER before – knowing the wait at this particular ER is nine minutes tells me something about the speed of service/care (informative). Metrics like this are also valuable if they are standardized and adopted by other hospitals – it allows me to see which ER has the shortest wait time. But with all this information that is immediately provided to me from the flashing nine minutes, one important piece of information is missing. One detail that can change everything – how do we define (operationalize) ER wait time? As I looked at the sign in my rearview mirror, it hit me – ER wait time has to have a pretty specific definition and, because it is advertised, has to be something easily controlled and likely quickly completed. Fortunately for us, ER wait time is a standardized metric – so what is it?
According to ProPublica, ER wait time is defined as the “Average time patients spent in the emergency room before being seen by a doctor. Average time refers to the median waiting time (the midpoint of all patients’ waiting times). References to “doctor” indicate a doctor, nurse practitioner or physician assistant.“1 These metrics are set by the U.S. Centers for Medicare & Medicaid Services.2 Alas, nine minutes is the expected wait time from my initial check-in to contact with a healthcare professional who takes health/emergency information. This website (https://projects.propublica.org/emergency) allows me to see approximately how long an entire ER visit could take at a specific hospital and, just as I remembered, it takes up a significant portion of the day (although I was surprised that my childhood hospital listed the average time for discharge at 2 hours and 39 minutes as of this writing). Now this isn’t an indictment of ERs – I know people who work there and they are awesome at what they do and ERs are just crazy – organized chaos if you will. However, this is a great example of why we should understand how a metric is defined. We may feel silly asking, “Nine minute ER wait time, what does that mean?” But sometimes simple questions lead to big surprises – like realizing my hypothetical dislocated shoulder won’t be resolved with pit-crew like speed.
Now that we feel comfortable asking about (and saying the word) operationalization, let’s look at another metric – one that is likely to really drive home the point of why you should ask, “how do you define that?”
On-time departure – To those who travel by air, leaving on time is an expectation. You have booked your flight in advance, paid a hefty sum for your ticket, and likely made subsequent plans contingent upon getting to where you need to be on time – which usually necessitates leaving on time. Some people might even select their airline and flight based on the on-time departure historical performance (you can find this data here if you are interested: https://www.transtats.bts.gov/ONTIME/Departures.aspx). Now if you have flown a lot, you might be surprised to find such high performance. I can recall numerous times sitting in a queue of planes on the runway as if we were waiting for the lone porta potty at a marathon starting line – what gives? Well…even though the name sounds intuitive we should probably ask the question: how do you define on-time departure? And I hope you are near an ER with a short wait time, because you are about to have your mind blown. On-time departure is defined as whether or not the aircraft pulled away from the gate at the time it was scheduled to do so within a 15 minute margin of error.3 If your flight was scheduled to leave at 10:00 AM and the aircraft pulled away from the gate at that time but then waited on the runway for an hour, you still (based on this metric) had an on-time departure.
In Part 1 of this series we talked about the importance of asking basic questions about numbers we hear. In Part 2, we discussed how asking the most basic question (how do you define that) is usually the most important. We will bring all of these together in the last installment (Part 3) where we discuss examples of 1) tweaking the environment to improve a metric and 2) how an unchecked metric can lead to incorrect assumptions. The cartoon below is a nice little preview.
1 https://projects.propublica.org/emergency
2 https://www.medicare.gov/hospitalcompare/Data/Data-Updated.html#
3 https://www.bts.gov/explore-topics-and-geography/topics/airline-time-performance-and-causes-flight-delays