It¡¯s widely accepted that, statistically speaking, a result shouldn¡¯t be considered real unless it¡¯s seen at least 19 out of 20 times¡ªi.e., it reaches the p=0.05 statistical significance level. This appears to have first been suggested by English statistician Sir Ronald Aylmer Fisher in 1925.1 Notably, he said this should not be taken as an absolute cutoff; in fact, he recommended looking at all probability levels. Unfortunately, that important qualification has been largely forgotten, resulting in a tendency to ignore
effects that don¡¯t reach 5-percent significance.
Today, many measuring instruments only show data that reaches a 5-, 2-, 1- or 0.5-percent significance level. The problem is that this eliminates potentially important information. Is an effect not worth noting because it only reaches the 7-percent significance level?
Researchers at the University of Iowa and City University in London, England, have now developed a way to reveal information that would otherwise be hidden by this protocol: a continuous-probability scale that presents all of the information found during a test or scan¡ªnot just the data passing the p¡Ü0.05 test¡ªalong with its level of statistical significance. Chris A. Johnson, PhD, professor of ophthalmology and visual sciences, and director of the Visual Field Reading Center at the
How the Scale Works
Dr. Johnson says the continuous probability scale uses color to show the statistical likelihood that each area detected to be non-normal is the result of chance. ¡°Green represents data that has a 50-percent or greater likelihood of representing a normal result for a person of that age,¡± he says. ¡°Representation of data closer to the suspicious zone, such as the 20- to 10-percent probability level, shifts to yellow. Data reaching the 5-percent probability level is shown in red; 1- or 0.5-percent probability is shown as black.¡± (Dr. Johnson notes that the colors were chosen to be readable even by clinicians who have hereditary colorblindness.) He adds that using a similar scale to display change over time could also improve a clinician¡¯s ability to detect progression.
Looking at perimetry results, the group has found that presenting data using this scale does indeed reveal useful information that was hidden in the traditional format. ¡°For example, in one case described in our recent paper on this topic,2 a patient with a pituitary adenoma was treated,¡± says Dr. Johnson. ¡°After treatment, one eye¡¯s visual field now looked normal. However, the continuous probability scale showed that the eye still had a residual defect. The perimeter didn¡¯t show the defect because it only ranked as having a 10-percent probability of being real. Clearly it¡¯s clinically relevant to know that a treatment was not a cure.¡±
One advantage of this approach to data analysis is that it can be used to create a level playing field for different tests whose results are normally difficult or impossible to compare. ¡°Different visual field tests use different measurements, different operating ranges and different dynamic ranges,¡± notes Dr. Johnson. ¡°And, they¡¯re susceptible to variations that occur from one test to another and one individual to another. Those differences make it very hard to compare the tests.
¡°The continuous probability scale solves that problem,¡± he continues. ¡°This scale simply shows the likelihood that someone of a specific age would have this result by chance. Furthermore, up until now, every instrument has established its own normative database using different criteria and different individuals. Our scale¡¯s normative database for all instruments is based on the same population and criteria for being normal. This will make it possible to truly compare data from different instruments on a level playing field.¡±
Testing with Real World Data
To assemble a normative database, Dr. Johnson says individuals with healthy eyes and some with glaucoma were scanned at multiple centers. ¡°We had sufficient numbers to be able to differentiate between pathology and normal aging,¡± he notes.
Dr. Johnson explains that his team is retroactively applying this probability scale to data collected in several major clinical trials. ¡°I¡¯ve been involved in visual field reading centers used by multicenter trials for more than 20 years,¡± he points out. ¡°We¡¯ve made sure we had raw data in a form that could be re-evaluated later using newer technology, to ensure that data comparisons would still be legitimate. As a result, we have all of the raw data from the patients in the Optic Neuritis Treatment Trial and the Ocular Hypertension Treatment Study¡ªmore than 2,000 in total. We¡¯re checking to see whether the continuous probability scale will reveal information that was overlooked in these studies because they used the 5-percent cutoff.¡±
Hurdles to Surmount
Dr. Johnson notes that there are some potential problems that come with switching to a different perimetry scale. ¡°Visual fields are usually measured using a logarithmic decibel scale that runs from 0 to 35 or 36,¡± he explains. ¡°The continuous probability scale is linear and goes from zero to 100. Because it¡¯s linear, there are times when a change from 98-percent probability to 97-percent probability will be a very small amount in decibels¡ªsmaller than can actually be measured. That could affect the meaningfulness of the numbers in some situations. Similarly, how big of a change over time is meaningful? If you go from the 77th percentile to the 63rd percentile, does that affect the person¡¯s quality of vision?¡±
Dr. Johnson adds that his group plans to increase the size of the normative database, and they still don¡¯t know what ways of analyzing this type of data will be most useful.
¡°These are all issues that will have to be worked out over time,¡± he says.
Potential for Widespread Use
Because this scale can reveal potentially useful data that clinicians have been missing, possible uses extend far beyond perimetry. ¡°Monitoring structural changes should benefit from removing the 5-percent probability cutoff, whether you¡¯re looking at retinal thickness or the amount of cupping,¡± he says. ¡°We¡¯re testing this type of scale with OCT technology, and
tests such as visual acuity, contrast sensitivity and color vision as well.¡±
Dr. Johnson says the team is currently focusing on a number of practical concerns, including making sure the scale is easy to use and understand. Furthermore, he hopes to make this technology widely available by remaining independent of any one instrument manufacturer. ¡°Whenever you¡¯re dealing with the private sector, you have issues like competition, intellectual property and proprietary interests,¡± he says. ¡°We¡¯d be most comfortable with a secure website that anyone can access to upload their data and have it transformed into our format and sent back to them. I believe we can provide this service on a noncommercial basis.¡± Dr. Johnson says he also plans to avoid any personal financial ties to the technology, which he believes could make him less objective and more resistant to correcting problems or introducing modifications.
R
O
Dr. Johnson is optimistic about the future. ¡°I¡¯ve been contacted by many individuals and groups who are interested in pursuing this,¡± he says. ¡°I¡¯m sure that with combined input from many sources we¡¯ll develop a much better assessment of what this technology can do and where it¡¯s most efficacious. This is just the beginning.¡±
1. Fisher RA. Statistical Methods for Research Workers.
2. Wall M, Johnson CA, Kardon RH, Crabb DP. Use of a Continuous Probability Scale to Display Visual Field Damage. Arch Ophthalmol 2009:127;6;749-756.
