Scientific merit

Scientific merit is an important component of an IRB’s evaluation of a study. This Course will discuss concepts related to Scientific Merit. We will begin by examining the origins of scientific merit within individual disciplines and within the Nuremberg Code and the Helsinki Declaration. We will define scientific merit and examine the components of scientific merit in research design, grant funding and application evaluation, and in the dissemination of findings. Finally, we will look at how the NIH reviews studies for scientific merit. The module will conclude with a brief quiz.

Objectives:

  • Review the origins and definition of scientific merit
  • Examine the importance of scientific merit to research
  • Understand what IRBs review when evaluating the scientific merit of a study.

Scientific Merit

Introduction

Scientific merit is an important component of an IRB’s evaluation of a study. Under the criteria for review in 45 CFR 46.111(a), IRB’s are called to ensure that “(1) Risks to subjects are minimized: (i) by using procedures which are consistent with sound research design, and which do not, unnecessarily, expose subjects to risk”. (ref 13) Thus, the IRB has a responsibility to evaluate scientific merit, or quality, as part of its review, especially when a study design poses greater than minimal risk to participants. Amdur and Bankert (ref 78, p. 88) suggest that “if revising the study design will meaningfully decrease the risk to subjects without a major compromise in the persuasiveness of the study results, then the IRB should not approve the protocol”. Where risk to subjects is minimal, however, they go on to say “there is really no ethical justification for the IRB to make such revisions a condition for IRB approval. Each case must be evaluated independently” (ref 78, p. 89).

This Course will discuss concepts related to Scientific Merit. We will begin by examining the origins of scientific merit within individual disciplines and within the Nuremberg Code and the Helsinki Declaration. We will define scientific merit and examine the components of scientific merit in research design, grant funding and application evaluation, and in the dissemination of findings. Finally, we will look at how the NIH reviews studies for scientific merit.

The Origins of Scientific Merit

The origins of scientific merit lay within each scientific discipline’s desire to build the strongest knowledge base possible. However, within the last century, serious research abuses have given rise to documents like the Nuremberg Code and the Helsinki Declaration, which also specifically speak to the issue of scientific merit.

The Nuremberg Code arose out of the Nuremberg Trials, where Nazi doctors were tried for experiments conducted on both prisoners and concentration camp victims during the Second World War. And while, arguably, some useful information did come from some of the Nazi studies, such as the work they did on hypothermia, none of their studies involved any form of consent and much of the research they conducted would be considered useless by the scientific community in general, such as the studies involving attempting to change eye color with the injection of dye into the eye. Because of the significant abuses of the Nazi doctors, the Nuremberg Code states that experiments should yield scientifically useful results. Hellstroem and Jacob (ref 68) point out that scientific merit “should be judged by researchers from disciplines or research orientations outside of that of the research being considered. . . Scientific merit is also an issue of whether a scientific field is ‘ripe for exploitation’ in terms of technology. Thus, scientific merit is founded on the value of the proposed research to the rest of science and technology. “(ref 68, p. 388) research should not be engaged in on a whim or to simply repeat what is already known; the anticipated results of the study must contribute to science or benefit the field or discipline; and experiments must be conducted only by scientifically qualified persons. (ref 1) (ref 2)

The Helsinki Declaration also addresses scientific merit. It states that research must conform to generally accepted scientific principles. The design and performance of each experimental procedure should be outlined in an experimental protocol which should be reviewed by an independent committee. Research should be conducted by scientifically qualified persons. (ref 3)

Expectations of research with good scientific merit

Download

Scientific merit, in the most basic sense simply looks at whether or not a study represents good science. For any study to have scientific merit, it must contain the following components: address an area of importance to the discipline, utilize established scientific principles, exhibit alignment within the study, demonstrate how scientific knowledge will be garined (should be gained) from the study, and involve appropriately trained researchers.

Addresses an area of importance to the discipline

There is a key difference between what may be of interest to laypersons, practitioners and scientists. So while an individual may believe that a particular problem is worth studying because of casual observations made at work or in daily life, scientific merit requires that the problems being investigated are considered valuable to the larger scientific community. This means that the problem the researcher seeks to investigate comes from the appropriate academic literature. In turn, what the researcher is investigating should extend the literature in some way. This is how academic disciplines advance.

The study must address a problem that is supported by the scientific literature and is important to the scientific community in general. Hellstroem and Jacob (ref 68) point out that scientific merit is not judged by scientists within the field that the research is a part of but “should be judged by researchers from disciplines or research orientations outside of that of the research being considered . . . . Scientific merit is also an issue of whether a scientific field is ‘ripe for exploitation’ in terms of technology. Thus, scientific merit is founded on the value of the proposed research to the rest of science and technology. “(ref 68, p. 388)

Utilize established scientific principles

How research is conducted within the scientific community has evolved over a considerable period of time and what might have been considered acceptable even 25 years ago, may not be acceptable today. How a researcher conducts a study should always be based upon the ways in which studies are conducted in science in general and within the researcher’s particular field specifically. Research should conform to established scientific principles and procedures.

This means that the researcher, whether designing a quantitative or qualitative study, must turn to the literature extant for guidance on the appropriate methods and procedures to be used within that study. While there are advancements and changes in how research may be conducted over time, and, in fact, research into methods is an area of study in and of itself, researchers should not simply make-up how a study is conducted out of thin air. Researchers do not need to invent new methods to carry out each new study, because appropriate methods already exist within the literature—and when an appropriate method does not, that, in and of itself, will be the object of a study.

Exhibit Alignment within the study

Any research study that has scientific merit is a coherent whole. From beginning to end, the study flows together. It makes logical sense. When we read the research questions, we should know what methodology the study will use, and we should be able to determine what the variables in the study are. The selection of subjects should flow from the methodology. The methods of data collection will make sense as will the data analysis. Everything within the study is linked together. The language of the study will also make sense. Qualitative studies talk about research in one way and quantitative studies in another.

First and foremost, the research questions must be answerable. They should not be broad social or philosophical problems like “why is there war?” The research questions should be narrow in scope and they must be nested within the researcher’s discipline. Beyond having answerable research questions, they must also be answerable in the way that the study proposes. The entire study must represent a blueprint for answering the research questions.

Demonstrate how scientific knowledge will be gained from the study.

The outcome of any study should be an increase in scientific knowledge. This does not have to be a discipline-changing increase, or a paradigm shift, but it must be an increase none the less. A study cannot simply repeat what is already known—either from previous studies, or from general knowledge. Some of the most unethical studies that have been conducted violate this principle of scientific merit. For example, the San Antonio Contraception Study examined whether women who were given placebos rather than birth control pills were more likely to become pregnant. (ref 8)

The directive of adding to the pool of scientific knowledge does not imply that for each and every study something of statistical significance must be found. To the contrary, the results of many studies show us that something does not work, that there is no significant difference, or that the expected outcome does not occur. But, this lack of a finding is, in and of itself, a finding.

Involve appropriately trained researchers

Good science cannot come from individuals who are untrained. The researcher and everyone involved in the study must have appropriate training to conduct the study. This training will include training in dealing with the subjects in the study, the instruments in the study, and even the statistics used within the study. If a study involves children, the researchers must be able to show that they have appropriate training in the psychological/medical/social/physical development of children or whatever aspect of child development the study involves. If a study involves veterans with post traumatic stress disorder, then the researchers must show that they have special training in working with individuals with this disorder.

If a study involves administering psychological measures, like the Minnesota Multiphasic Personality Inventory, or the Wechsler Intelligence Scales then the researchers must possess the specialized knowledge to administer the test. If a researcher is interested in conducting research into Post Traumatic Stress Disorder (PTSD) and wants to measure PTSD, the researcher might decide to administer the Clinician-Administered PTSD Scale (CAPS) (ref 69). This is a good scale to make either a current or lifetime diagnosis of PTSD, but it must be administered by a clinician or clinical researcher who has a working knowledge of PTSD or an appropriately trained paraprofessional. If a researcher is investigating preschool settings, the Environment Rating Scales can be used to assess group programs for young children. And while certification is not required to administer the scales, the authors recommend extensive training. (ref 70) (ref 71)

Proposed recruitment and data collection plans should be based on the population characteristics, topic, and on expected participation rates. The proposed study should have a sample size appropriate for the data analysis. However, researchers do not have a crystal ball. This may mean the recruitment efforts do not yield the sample size needed. It may also mean when the data is entered and ready for analysis the expected range of responses is not present. In these instances, revisions to the original plan may be needed. This should be noted in any publications, reports or dissemination efforts. Common sections for this information may be in the description of the sample, data analysis section and also in the limitations area.

Scientific merit in grants and publication requirements

(VIDEO ref 97)

Scientific merit is a requirement for grants and publications. Funding agencies will often mention that the scientific merit of a particular proposal will be considered when making funding decisions. “The scientific merit of the proposal” is among the criteria that The Foundation for AIDS Research (amfAR), uses to evaluate the proposals and letters of intent that they receive. If a proposal does not have scientific merit, it will not be funded by them (ref 72). The Canadian Institutes of Health Research use both Scientific Merit and the potential impact of the research when reviewing research projects for funding (ref 73).

A Humerous Look at Scientific Merit

Download PDF (ref 74)

The parachute study was written to poke fun at researchers who insist that double-blind, placebo studies must be done to insure that an intervention works. Clearly, no one needs to assign individuals to “parachute” and “no-parachute” groups. [Permission to use the parachute article provided by Dr. Gordon Smith.] (ref 74)  We already know that parachutes do work, so there really is no need to create a study testing whether using a parachute as opposed to not using a parachute increases the survivability of a long “gravitational challenge.” There is no need to conduct a study to investigate something that we already know. And there is no need to put people in harm’s way to conduct this research. It’s not the case that research cannot be conducted on parachutes, but this would not be the way to design such a study—even though placebo studies may be appropriate in other studies, it certainly would not be appropriate in this study.

Conclusion

This Module discussed concepts related to Scientific Merit. We began by examining the origins of scientific merit within individual disciplines and within the Nuremberg Code and the Helsinki Declaration. We defined scientific merit and examined the components of scientific merit in research design, grant funding and application evaluation, and in the dissemination of findings. Finally, we learned how the NIH reviews a study for scientific merit.