As we posted here on the Blog yesterday, the Subcommittee on Oversight and Investigations of the House of Representatives Committee on Energy and Commerce held a hearing on “Direct-To-Consumer Genetic Testing and the Consequences to the Public Health.”
Central to this hearing was an investigation by the Government Accountability Office (GAO) into 15 DTC genetic testing companies.
The GAO refused to discuss its concerns or its report with 23andMe, and now that the report is public and we have had a chance to review it, we are troubled and find the report is deeply flawed. We note that while such an exercise as conducted by GAO has the potential to raise questions, it does not provide the answers that a more rigorous scientific study would provide. This report raises questions, but leads to few conclusions because of its unscientific nature. The GAO itself recognizes this, writing, “It is important to emphasize that we did not conduct a rigorous scientific study.…”
[The report is accessible from the Committee’s website (Click on PDF for “Testimony of Gregory Kutz.” Gregory Kutz is Managing Director, Forensic Audits and Special Investigations, Government Accountability Office).]
We are confident in our service’s accuracy and reliability. It is widely accepted that the technology we are using is sound. We understand that GAO did not find any problem with the underlying data that we provide — the As, Cs, Ts and Gs. What is at question is whether or not one part of the information about that data that we provide is of value, and we believe strongly that it is.
The GAO report focused only on disease risk probabilities. It did not focus on ancestry or the trait reports we offer. It also failed to address that we also provide information about carrier status for single gene diseases such as cystic fibrosis and Tay-Sachs disease, as well as the fact that we provide information about a customer’s likely response to certain prescription medications that have been shown in clinical trials to have differing effects and side effects depending on a person’s genetic make-up. This suggests that GAO found no problems with these parts of our service.
Carrier status and drug response information are clearly useful. In fact, Dr. James Evans, the Director of Adult Genetics Services at the University of North Carolina and the Editor-in-Chief of Genetics in Medicine, admitted during the Congressional hearing that drug response information would be of great interest to him as a physician. (He was specifically referring to results pertaining to a patient’s sensitivity to the anti-viral medication abacavir.) It should be noted that during the hearing it was not clear that Dr. Evans had been the primary consultant to GAO regarding the scientific and medical relevance of the results provided by DTC genetics testing companies.
The remarks made by 23andMe Co-Founder Anne Wojcicki and General Counsel Ashley Gould at the FDA public meeting on July 20, 2010 about laboratory developed tests demonstrate the importance of the work we are doing and our commitment to ensuring that members of the public are provided unfettered access to their DNA information in a responsible manner. We embrace the ideas that the FDA offered today about stepping in to provide a regulatory framework and help set scientific and transparency standards across the industry. We look forward to helping with this process.
Read on for discussion of some of the problems with the GAO report.
One of the most unfortunate parts of the GAO report is that it unfairly lumps together reputable and well known companies such as 23andMe with un-named companies making verifiably untrue endorsement claims, spurious scientific claims, and also selling potentially fraudulent supplements in addition to genetic testing services. Some of the most troubling of these interactions between the GAO and genetic testing companies can be found in a table on pages 15-16 of the GAO report, and in a Youtube video the Office has posted.
It must be noted however, that although the companies are not identified in the video or the report, at the hearing it was revealed that 23andMe is Company 1. Other than saying that we believe customers should consult with their physician or other healthcare professional when they have questions about their data, 23andMe/Company 1 is not implicated in any wrongdoing.
GAO seems to believe that directing consumers with questions about their genetic information to their health care professionals (a stance we continue to stand behind) is “misleading” because of a pronouncement by the Department of Health and Human Services’ Secretary’s Advisory Committee on Genetics, Health and Society that physicians “cannot keep up with the pace of genetic tests and are not adequately prepared to use test information to treat patients appropriately.”
It is important to note that most instances of wrongdoing highlighted by GAO come from the 11 still unidentified companies they investigated. (Companies 2-4 were identified as DecodeMe, Pathway Genomics and Navigenics, respectively.)
The rest of the GAO report revolved around five main points (beginning on page 4 of the document):
(1) “each donor’s factual profile received disease risk predictions that varied across all four companies, indicating that identical DNA can yield contradictory results depending solely on the company it was sent to for analysis”
It is well known that different DTC genetic testing companies provide different estimates of risk. A report published in the scientific journal Nature in 2009 (Ng, et al.) described this situation, and 23andme, along with Navigenics, responded in an open letter that was published on our blog, The Blog.
There are valid scientific reasons for different estimates from different companies, such as: companies employ slightly different statistical models for making risk estimates; companies establish different criteria for the inclusion of associations in their reports; new associations are being discovered at a faster rate than companies’ development cycles; companies may test for an imperfectly overlapping set of genetic variants for reasons including the ability of different genotyping technologies to assay certain variants.
We agree with the idea that consumers should be able to compare the risk predictions they might receive from different test providers. This is an issue that deserves serious attention and one that we believe can be solved by the implementation of broad standards throughout the industry. We have approached both the NIH and FDA for assistance in this matter (see this letter sent to the heads of both agencies and posted on our blog, The Blog). Instead of constructively adding to these efforts, GAO has instead implied that because results differ between companies, they are simply wrong. Their report fails to provide all relevant information, and perpetuates the misunderstandings of genetics in particular and science in general that 23andMe has since the very beginning been dedicated to changing.
It should be noted that the problem of different risk predictions from different sources is not unique to the direct-to-consumer genetic testing industry. Take for example, these two tools (here and here) available to the public to calculate risk for cardiovascular disease. Both are clinically valid, yet they give different answers. It should be noted that development of both of these two tools was at least partially funded by the federal government.
2) “these risk predictions often conflicted with the donors’ factual illnesses and family medical histories”
As an example of this the GAO points to one of their “donors” who has a family history of heart disease but is told by some DTC genetic testing companies that he has average risk for heart disease based on genetics alone.
As the average risk of heart disease is about 21% for males, it is unsurprising that an individual would have a family history of heart disease but also be at average risk based on genetics alone.
The GAO admits that risks are probabilistic by definition and that therefore there is no expectation that there would always be complete alignment between predicted risk and reality. But they then go on to quote an expert that “noted that the discrepancies between actual health and the predications made by these companies also serve to illustrate the lack of robustness of such predictive tests.” This is a criticism that could be said of any risk prediction, including those currently made based on measurements such as cholesterol. It is well accepted that not everyone with high cholesterol has a heart attack, and not all heart attacks happen in people with high cholesterol. Yet, cholesterol measurements are taken and used in clinical practice every day.
3) “none of the companies could provide the donors who submitted fictitious African American and Asian profiles with complete test results for their ethnicity but did not explicitly disclose this limitation prior to purchase”
Unfortunately, because of where and how biomedical research is funded, there are comparatively few studies looking for genetic associations in populations that form a minority in the countries where much of the latest research takes place. In other words, most of the associations have only been confirmed in populations of European ancestry. An FAQ on our website explains this situation and lays out how it can result in fewer results for people with African American or Asian ethnicity. In addition, our Terms of Service explain the limitations of genetic research, especially with regard to ethnic minorities, and note that a free demo version of our service is available for people to examine exactly what they will be getting if they choose to purchase.
23andMe actively looks for associations that have been repeated in populations other than the original study population, but we can only report data published by the scientific community at large. Unfortunately, most studies, both publicly and privately funded, do not include sufficient numbers of people from minority populations to meet the strict standards we have put in place to ensure that the information we provide through our Personal Genome Service is of high quality.
One of the strengths of 23andMe is our research platform, which we believe will be instrumental in bringing more diversity to the populations that can be completely served by DTC genetic testing. We are actively seeking to initiate a large study of African American individuals that will seek to replicate genetic associations already known in European populations (or show that these associations do not hold). As our database grows, we will be able to conduct studies in even more populations.
It should also be noted that the inability to accurately provide information for racial minorities is not a problem unique to the DTC genetic testing industry. The Gail breast cancer risk estimator provided by the National Cancer Institute has the following disclaimer:
“Assessments for Hispanic women are subject to greater uncertainty than those for white and African American women. Researchers are conducting additional studies, including studies with minority populations, to gather more data and to increase the accuracy of the tool for women in these populations.”
4) “one company provided donors with reports that showed conflicting predictions for the same DNA and profile, but did not explain how to interpret these different results”
This criticism specifically targets 23andMe and has to do with the fact that we provide both Established Research Reports and Preliminary Research Reports. The distinction between the two is that Established reports are supported by multiple, large, peer-reviewed studies, whereas markers included in Preliminary reports have not yet gained enough scientific consensus to be included in the Established reports.
Very often there will be some markers for a particular disease that have reached “Established” status, while some have not. In these cases there can be two reports (one Established, and one Preliminary) for the same condition. This allows us to provide customers with a fuller picture of the scientific research, while at the same time indicating differing levels of confidence in different markers.
The apparently conflicting predictions of an Established report and a Preliminary report are not inconsistent. Consider smoking, weight and your risk of heart disease. Your weight might suggest a low risk for heart disease, while your smoking habits may suggest otherwise. This example is analogous to cases of conflicting predictions in the GAO report, which rely on different genetic markers: one genetic variation may suggest an increased risk while another may suggest otherwise.
The GAO report does not reflect the great pains we have made on our website to make clear the differences between Established and Preliminary Research Reports. We are, however, always looking for ways to improve our service, and we will take the feedback from GAO on this matter into consideration and look for even more ways to increase customer understanding.
5) “follow-up consultations offered by three of the companies provided only general information and not the expert advice the companies promised to provide”
This criticism does not apply to 23andMe because we do not have in-house genetic counselors. We therefore can’t comment on the quality of the information provided by the “experts” the GAO had consultations with.
23andMe has always felt strongly that customers with questions about their data should consult with independent health care professionals so as to receive unbiased advice. We provide a list of resources in every report directing customers to the National Association of Genetic Counselors, which maintains a website that can help people find local health professionals trained in genetics.
23andMe is also collaborating with Informed Medical Decisions, Inc., to give customers direct access to board-certified genetic counselors that have been specifically trained to guide them through their 23andMe results. By design these counselors are independent of 23andMe, giving customers the opportunity to get unbiased information from a trained medical professional.
In conclusion, 23andMe is extremely disappointed that we did not have the opportunity to address all of these concerns at the Congressional hearing, or sooner, due to our lack of access to the report. These are serious issues that deserve serious and thoughtful discussion. Standards are needed in the genetic testing industry. We have been working towards these since the inception of our company, and we were pleased to hear FDA say that they are interested in developing a new type of regulatory framework that can deal with the many special aspects of direct-to-consumer genetic testing while still providing consumers with the protections they need and deserve. 23andMe is meeting with the FDA today and looks forward to fruitful discussion.