However, not all studies are created equal. A study's conclusions may be more or less valid, depending on numerous factors, including the design of the study. In fact, when evaluating a scientific paper to determine the validity of its results, you should begin by considering its design.
It's useful to think of this in terms of a hierarchy, with the most reliable study design at the top and least reliable at the bottom. (See table, facing page.) Here, I'd like to review some of the ways a study's design affects the validity of its results.
Noncomparative Studies
Let's consider each type of study, beginning with the simplest (at the bottom of the chart).
• Case reports. Individual case reports may provide important observations that can serve as valuable starting points for further research. They're also a practical source of information when you're dealing with rare diseases or uncommon associations to a treatment, such as when a glaucoma patient taking a new medication gets a side effect. But without multiple cases, you don't know whether or not your observation is a chance association that only applies to the case under consideration.
• Case series. When you analyze a series of cases, such as one describing the results of a new glaucoma surgical procedure, you can get a sense of whether the effect you're studying is a fluke or a trend. However, without a control group for comparison you still can't draw any conclusions about whether the new procedure is superior to the standard one.
• Cross-sectional study. This is an epidemiologic study—basically a survey in which you find the prevalence of a disease in the population. It only considers the subjects' conditions at one point in time. These studies accumulate data rather than testing a hypothesis; however, they're important because their results often stimulate clinical trials. The Baltimore Eye Survey, for example, is the best known cross-sectional study in ophthalmology; it produced the significant finding that African Americans have a prevalence of glaucoma more than four times that of Caucasians.
Comparative Studies
Comparative studies allow us to draw more reliable conclusions because we can contrast the impact of different factors on the outcome. These studies can be either retrospective, using preexisting data, or prospective. Retrospective studies produce less valid conclusions because the quality of the data may not be as good, and key data may be missing. Also, they usually suffer from selection bias.
Prospective studies can be designed to avoid these problems. The downside is that a prospective study may require waiting a long time for outcomes to emerge, and it can be expensive. In either case, having a control group is better than not having one.
• Case-control study. This kind of study takes a group of individuals who have the same outcome and retrospectively examines the factors that are different between them. An example would be a study that compares a group of lung cancer patients to a group of same-age patients without lung cancer (controls) to determine whether the patients with lung cancer had a history of heavy smoking. Requires large numbers of patients, expensive Single site prospective, randomized, controlled, double-masked clinical trial Requires large numbers of patients, expensive; may not be generalizable Prospective
Study Design
Advantages
Disadvantages
Validity Scale*
Multicenter prospective, randomized, controlled,
double-masked
clinical trial Eliminates selection bias; minimizes
confounding
factors
4
Limits selection bias and confounding factors
4
cohort studyGood for generating hypotheses
and establishing risk ratioTime-consuming; expensive
3
Retrospective cohort study
Good for generating hypotheses and establishing risk ratio
Limited by pre-existing data and confounding variables
3
Case-control study
Good for generating hypotheses
Only one outcome is studied
3
Cross-sectional study
Establishes prevalence; good for generating hypotheses
Large numbers needed; expensive
2
Comparative, retrospective case series
Better than having no comparison group
Historical control data is less reliable; selection bias
2
Case series
Helps to reveal trends
No control group for comparison
1
Case reports
Good for rare diseases and rare associations, such as side effects
Observation could be chance association
1
*4 indicates high validity; 1 indicates low validity
• Cohort study. In this type of study a population is selected and followed to establish the sequence of events leading to different outcomes. For example, the St. Lucia Eye Survey looked at subjects over time to determine how many subjects progressed from glaucoma to blindness.
Unfortunately, these studies suffer from a lack of randomization; the patients themselves are selecting treatment or no treatment. This allows confounding variables to influence the results. For example, if a cohort study finds that smokers have a higher incidence of liver disease, you can't attribute that to smoking because smokers also consume more alcohol. Nevertheless, cohort studies provide plenty of impetus for further controlled, randomized studies.
A classic example is the study of heart disease in Framingham, Mass., residents that found previously unknown risk factors, including high blood pressure, high cholesterol, increasing age and smoking. That information led to clinical trials in which patients were randomized to receive (or not receive) drugs that reduced the different risk factors, to see whether this decreased the prevalence of heart disease.
Similarly, an ophthalmology paper published in 2004 reported an epidemiologic study in which the authors looked at multiple factors affecting a large database of patients to find possible correlations to glaucoma. The data suggested that taking oral statins, such as Lipitor or Crestor, was protective for glaucoma. This could lead to a clinical trial in which patients are randomized to receive or not receive an oral statin; the trial would then follow the patients to find out whether they develop glaucoma.
As you can see, although cohort studies don't produce conclusive evidence of causality, they are vitally important to the data-gathering process. They provide better evidence than cross-sectional studies and may lead to prospective randomized trials.
Eliminating Confounding Factors
Once a prospective study randomizes subjects into two or more groups, several sources of bias and confounding error are largely eliminated.
• Prospective, randomized, controlled, double-masked clinical trials. This type of study is close to ideal, for several reasons. First, randomization of group members eliminates selection bias, a potentially crucial factor. Second, when comparing two groups, you'd like everything except the treatment you're giving to be the same in both groups. Randomizing patients to both groups tends to equalize baseline factors such as number of men and women in each group, age distribution, and other confounding variables—as long as the groups are large enough. (If each group only contains five people, for example, you could easily end up with five women or men in one group.) Neutralizing potential confounding variables strengthens the study's conclusions.
The control group may be untreated or treated using an alternate treatment (sometimes the existing, standard treatment). If this kind of study involves multiple clinical sites, the data is considered even more reliable because the number and diversity of subjects is greater, and multiple participating researchers decrease the likelihood of experimental bias.
Unfortunately, not all treatments or disease states lend themselves to this type of study, because it requires a large number of subjects. If you only see one new case of a disease per year, you're not going to be able to randomize patients to one treatment or another and complete the study in your lifetime.
Also, in some circumstances this type of study may simply be inappropriate because it would be unethical to assign people to one of the groups. This could be true if the treatment is already known to have tremendous benefit, as when penicillin was discovered for infections, or if the object of study is known to be harmful, as in the case of cigarette smoking.
Pooling Your Resources
Another useful study design is one that doesn't involve conducting a new study; instead, it makes the most of existing data.
• Meta-analysis combining multiple studies. This type of study maximizes the value of existing related studies by combining information from them. This can be done in two ways: by statistically combining the results of multiple studies, or by combining the individual data from multiple studies.
The results of this kind of analysis are based upon a larger data set. Sometimes results conflict and the studies effectively cancel each other out; other times, all results point in the same direction, but the sample sizes were too small in the individual trials to achieve statistical significance. Combining them may increase the sample size enough to produce statistical significance.
Clearly, combining the individual data from each trial for analysis would be preferable to just combining study results. However, this is often not feasible. The data in the different studies, for example, are often collected using different time intervals or different measuring devices. This doesn't invalidate combining the results, but can make combining the original data difficult or impossible.
Proceed with Caution
It's tempting to take reported study results at face value, but even the most well-intentioned research can produce results that are more suggestive than conclusive. The nature of the study design, along with potential sources or error, bias and confounding, must be taken into consideration before basing clinical decisions on study results.
Dr. Budenz is associate professor in the departments of ophthalmology, epidemiology, and public health at the University of Miami Miller School of Medicine, where he teaches a course in clinical trials. He is also associate medical director of the Bascom Palmer Eye Institute, Miami.
