A Statistician’s Viewpoint on Metrics for Your Research Program

Dr. Wendy Tate
Director, Analytics, Forte
December 7th, 2017

Designing for transparency

There is a famous statistical quote that is often attributed to Mark Twain, but the original author is unknown: “There are three kinds of lies: lies, damn lies, and statistics.” When you think about it, this can be very true. You can use data and statistics to come up with just about any answer you want, including metrics for your research program. That is why you have to approach your question from an unbiased mindset. Now, unbiased doesn’t mean disinterested. It is quite the opposite. By removing bias from your analysis, you minimize the opportunity for people to dismiss your metrics as an inaccurate representation of your program. 

Removing bias from your analysis

How do you remove bias? By taking the time to think ahead and define the question to be answered up front. By utilizing the scientific method (define your question, come up with a hypothesis, design your methods, collect data, do the analysis, draw conclusions), you and your audience can be confident that the metric you present is accurate to the actual state of affairs. However, diligence doesn’t stop with defining the question. You have to give analytics the chance to answer the question honestly. 

Know the limitations of the data and the method of analysis (the “test”) used. By choosing a test that does not properly fit the type of data you are using, you can come up with relationships that do not exist (type I error) or miss associations that do exist (type II error). The result is you present something that is just plain wrong and can result in downstream action that is not in the best interest of the research program. Without proper consideration of your question and the data going into answering that question, you can come up with just about any answer. 

Learn more about using operational performance metrics to find answers to your critical questions. Download this free eBook.

Displaying the results accurately

In designing the methods to conduct your analysis, you also need to design the method to present the results. This extends to the use of charts. In choosing the wrong kind of chart to answer a question, you lead your audience down the wrong path of interpretation. For example, if you are looking at accrual between departments, you do not want to use a line graph. While the data points (number of people for each category) may be correct, the connection of the lines between categories indicates a relationship between items. 

The decreasing line or increasing line can indicate a trend that truly doesn’t exist because the ability of Cardiology to accrue subjects may have nothing to do with how Respiratory accrues subjects. Also, departments are a categorical variable. There is no intrinsic order. You can reorder the categories, and thus change the directionality of the lines, leading to different conclusions. An alternative would be to use a bar chart. This type of chart shows each category as an independent entity but still visualizes the relation of the metric between groups.

Creating a reliable, repeatable process

It can be a lot of work, but by considering the above advice ahead of time (and not designing on the fly), your reward is multi-faceted. You generate reproducible and explainable metrics, reduce bias and increase transparency in your analysis (which increases your audience’s confidence), and provide the best opportunity for your audience to quickly and easily understand the story you are telling. But, rewards only occur when you have utilized the right toolset and framework to examine your data.

We utilize the scientific method at Forte to develop our data visualization platform, Forte Insights. That, in combination with visualization best practices, make it possible to provide a consistent technology that allows for interpretation of a metric in an out-of-the-box, easy manner. Transparency with our customers allows for transparency between the customer and their audience.

Learn more about Forte Insights

Compliance and Regulatory Metrics

No Comments

Leave a Reply

Your email address will not be published. Required fields are marked *