|
By Clark Rundell
10/09/07
Quality Control in Molecular Diagnostics
The American Society for Quality defines quality as, “The totality of
features and characteristics of a product or service that bear on its ability
to satisfy stated or implied needs.” For Clark Rundell, PhD, DABCC, FACB,
Vice President of Research at Maine Molecular Quality Controls, Inc. (Scarborough,
ME) that translates to “usefulness to the recipient.” In other
words, it’s about assessing whether quality is improving and it is just
a simple measure of the usefulness of what is provided to the user (i.e., patient).
This article will provide an inside look at current quality control issues
in molecular diagnostics.
Defining Usefulness
According to Rundell, usefulness to the patient can be
measured by four factors. “The
first, obviously, is whether the test can be delivered to the right patient
at the right time. Otherwise, it will have no measure of quality because it
is of no value to the individual. Second, the test result has to get to the
place where it can be used and it must be available when it is needed. Third,
cost is a quality issue, but only as far as it limits the availability of a
test. Finally, accuracy (i.e., the right result) is a combination of how accurate
and precise the result is, as well as how often the right result is delivered
to the right patient.”
Defining Quality Control
Quality control is also about security for the laboratory
because it ensures that test results have value to the recipient. “Value
to the recipient is what we get paid for,” Rundell says. “There
is a lot of talk about how to market and sell tests. However, the bottom line
is that you have to demonstrate value to the user (i.e., the recipient), whether
it is the hospital, the patient, or even the insurer.” This is the key
to economic survival.
A properly executed quality control program also needs
to satisfy regulatory requirements and meet with peer acceptance. This is the
key to demonstrating proficiency. “While it’s
a given that whatever we do will satisfy the regulations, it might be shortsighted
to stop there,” Rundell
says. “What you really want to ask is, does it (i.e., the test) fulfill
the purpose for which it is intended?”
“Once we have defined the
purpose of the test, that delineates the level of precision and accuracy that
the test needs to have,” he says. “That
in turn allows us to build the scenario of how to monitor a test so that we
can ensure that it delivers that level of accuracy.”
Quality as an “Error Budget”
It is useful to think of quality in
terms of an “error budget.” Possible
sources of error in a test occur at collection of sample, through the analytical
processes, and in reporting of the result. “In some complex molecular
tests, the analytical component may be a significant contributor to error.
In others, such as testing of RNA from preserved tissue, the quality of the
sample can be a major source of error. To achieve the desired accuracy it is
necessary to control all error sources so that the total error is within allowable
limits. In some cases this means spending a lot of effort in controlling the
quality of the specimen,” Rundell explains. “However, there’s
one additional piece to that—even if the laboratory (the analytical or
technical portion of error) is only 1 percent of the total, you have to think
that you don’t want that to climb to 10 percent. No matter which testing
scenario, there is an important reason to think about monitoring and controlling
the technical errors in a test.”
When discussing performance, Rundell
notes that, for molecular tests, the actual error rate, or medically allowable
error rate, has not really been defined in many cases. “The public perception
is that a genetic test is 100 percent accurate—it should be done once
and done accurately. It’s something
that we have to be concerned about because we are going to be faced with loss
of confidence in laboratory testing, increased regulation, and loss of financial
support if our tests are perceived as not accurate.”
Regarding what is
known about molecular test error rates, “For simple
single nucleotide polymorphism (SNP) testing data, CAP proficiency statistics
and European Union experience suggests an error rate of between 0.1 and 1 percent
on proficiency samples,” Rundell explains. While this measured proficiency
sample error rate is for simple SNP testing, there is anecdotal evidence that
for rare mutations, not usually in proficiency samples, that laboratories have
seldom seen or tested, the error rate is, in fact, somewhat higher than 0.1
to 1 percent. While we haven’t defined what error rate is allowable,
public perception is certainly that we are going to be right most of the time.”
Quality
Issues
According to Rundell, one of the reasons there are quality issues
with new tests is that there are variables that are not yet known and, therefore,
are not being monitored. “For these tests, the big quality issues are
1) that we have not defined a quality goal, 2) we have not defined all important
variables, and 3) we have not previously had the technical capability nor the
necessary reference materials to monitor and control the tests closely. However,
instrumentation is now available to allow monitoring of test variables and
new reference materials are becoming available to facilitate long term performance
monitoring and proactive quality control.”
Regarding quality goal, he
notes, “Real data has given us something to
work with. Before we had these newer instruments, we were basically looking
at bands on a gel. Except for a few infectious disease tests, we were really
putting a checkmark in a positive or negative box. We really couldn’t
monitor the assay closely.”
The bottom line, according to Rundell, is
that while capabilities have not been defined, labs and clinicians are at the
point of identifying what they are capable of. “For gene expression assay
by real-time polymerase chain reaction (PCR), for example, we can achieve a
threshold cycle (Ct) precision of 10 to 15 percent, reliably from assay to
assay (both inter and intra). This is the precision required to reliably tell
the difference where there has been a 50 percent change in expression of a
gene. Is that good enough? It depends on what you’re trying to assay.”
 This
performance level however, identifies the capability of the test system and
is a minimal performance target for laboratories doing this type of testing.
De-facto performance targets are also being established for other assays as
data becomes available.
According to Rundell, although regulations
are not the entire answer, they are pointing in the right direction. “We
should have positive, negative, and sensitivity controls for each assay—but
it is very difficult with a multiplex test and it is going to pose a problem
with an array test. We really don’t want to run more controls. What we
really do want is a control system that tells us when our systems are not performing
optimally. The new test systems put out lots of data and we have emerging new
reference materials to attack the problem. In addition, we need some comprehensive
studies to determine what will constitute a sufficient quality strategy.”
To
accomplish the necessary improved monitoring of molecular tests, especially
of multiplex tests, synthetic quality controls have been developed. Current
synthetic control materials available include:
• Amplicons
• Ribonucleic
acid (RNA) based controls from the External RNA Control Consortium (ERCC) to
help measure from array system to array system
• Synthetic
oligonucleotides
• Plasma
• Recombinants
designed to be extracted
A list of materials is available at www.phppo.cdc.gov.
“These materials were designed to address the special problems of quality
control in molecular testing and particularly multiplex molecular testing.
Since the materials are synthetic, they can, and do, contain rare mutations
that are not available from other sources,” Rundell says. “They
also address special quality control needs of multiplex tests. For example,
it could be cost prohibitive to regularly run 25 quality controls to monitor
the performance of a 25 mutation multiplex test. The synthetic controls, however,
may contain all 25 mutations in one or two samples, thereby greatly simplifying
quality monitoring. Synthetic controls also do not contain potentially infectious
material nor are they from patient samples and therefore do not have confidentiality
concerns. Another necessary feature of a quality control material is
sample-to-sample and lot-to-lot consistency. The synthetic materials can be
manufactured to be highly consistent and therefore facilitate long term test
monitoring.”
MDx Quality in Summary
“To implement a quality molecular test you first need to establish a
goal, which is your target error rate. Finally, you need to establish criteria
for quality testing, which would include choosing the right system, validation,
developing a quality plan, and having a system to monitor quality through preventive
action,” Rundell says. In short, the growth and survival of molecular
diagnostics depends on efficient and accurate testing.
|
