Global Wireless Solutions (GWS) and Hiya recently released the findings from a new study that assessed how well spam detection solution providers Hiya, First Orion and TNS detect unwanted robocalls. The study, which focused specifically on spam identification rate and spam error rate, and assigned an overall score for the combination of these two metrics, found that Hiya is the best overall at accurately detecting unwanted calls. Of course, Hiya is pleased with the outcome, but that is less important.
Most importantly, this study introduces a first-of-its-kind methodology to set a new industry standard for how to transparently and accurately evaluate how well solution providers detect spam calls. As covered exclusively by FierceWireless in a story titled “Hiya calls for industry standard to compare robocall solution providers”, we believe the time is now to do just that!
A New Industry Standard to Cut Through the Confusion
Over the past year, a few different studies attempted to declare a spam detection “winner” among the big four carriers (recently, the Washington Post attempted their own study). At best, these studies can serve as cautionary tales as to the complexity of the spam detection space. At worst, they were erroneous and misleading. And given the lack of transparency surrounding who really commissioned them and what methodology was used to conduct them, in the end they did nothing but to neutralize each other.
In order to provide the most scientific way of testing each company’s ability to accurately detect spam calls, Hiya worked with GWS to implement a unique approach to the testing methodology. Instead of testing static “lists” of known spam numbers, this test examined actual phone traffic over the course of 24 hours and leveraged the resulting set of numbers to test how each company classified each number. This method allowed GWS to look at where providers agree and disagree on the reputations they assign to numbers, and through further analysis, determine how accurate they are at doing so.
Specifically, the results of this study looked at and revealed the following:
- Identification Rate: The Identification Rate is the portion of all confirmed spam phone numbers in the dataset tested that the provider correctly classified as spam.
- Error Rate: The Error Rate is the portion of all calls that a provider classified as spam, that were not actually spam (i.e. false positives).
- Detection Score: The Detection Score takes the Identification Rate and the Error Rate and applies a penalty for incorrectly classifying a number as spam.
Further details surrounding the industry standard set forth by Hiya and GWS can be found here.
Our hope is that the industry can rally together to create a fair and honest way to evaluate all of the spam detection solutions in the market. Healthy competition among the vendors will benefit the carrier customers, and perhaps more importantly, the consumer end users will enjoy a better calling experience with their carriers.
