Rapid Responses to:

EDITORIALS: Roger Lemon and Stephen B Dunnett Surveying the literature from animal experiments BMJ 2005; 330: 977-978 [Full text]

Rapid Responses: Submit a response to this article

Rapid Responses published:

Nerves of steel

Alfred N Jackson   (1 May 2005)

Drug safety tests and subsequent clinical experience

Peter Fletcher   (4 May 2005)

Systematic review and meta-analysis make an important contribution to surveying the literature from animal experiments

Malcolm R Macleod, Paul Kelly, Peter Sandercock, Pandora Pound, Shah Ebrahim and Ian Roberts   (10 May 2005)

Avoidance of bias is the objective of systematic review, not meta-analysis

Khalid S. Khan, Luciano Mignini, Research fellow, Centro Rosarino de Estudios Perinatales, Rosario, Argentina   (1 June 2005)

Nerves of steel 1 May 2005
Alfred N Jackson, Primary care physician Bulawayo Zimbabwe Send response to journal: Re: Nerves of steel	The report on Fialuridin is scary.The extrapolation of animal experiments to humankind is a difficult one.There do not appear to be many alternatives. But the vast chasm between men and animals is real. Chromosome numbers are fixed by the protocols of meiosis.Chromosome numbers suggest no common genealogies. So applicability of animal findings to humans is a matter of serendipity. I wish to thank the editor for opening this contentious issue to debate. Competing interests: None declared
Drug safety tests and subsequent clinical experience 4 May 2005
Peter Fletcher, retired Little Maplestead, Halstead CO92RU Send response to journal: Re: Drug safety tests and subsequent clinical experience	It may be of passing interest to readers who are either for or aginst the use of animals in medical research that I published a rather scruffy little paper 27 years ago (J Roy Soc Med. Drug safety tests and subsequent clinical experience. 1978, 71, 693-696) on the matter of animal safety testing. The paper is remarkable for several reasons – (1) it has been cited extensively over many years, (2) it has never been repeated, (3) it has never been refuted and (4) it showed that animal safety studies predict human adverse drug reactions (ADRs) to a limited but useful extent. I was the Medical Assessor to the Committee on Safety of Medicines (CSM) at the time of writing so had access to the entire data submitted in support of Product Licence Applications. I scrutinised 45 major new drugs considered by the CSM over a period of about 9 months listing every toxic effect observed in any animal test for each drug and comparing those findings with reported ADRs for the same drug when administered to patients. Amongst several interesting observations was a technician’s opinion that dogs had hallucinations on a particular compound which was confirmed in humans and that hypotension as an ADR of three drugs which were not antihypertensive agents was not detected in animals probably because of a failure to measure blood pressure! Competing interests: None declared
Systematic review and meta-analysis make an important contribution to surveying the literature from animal experiments 10 May 2005
Malcolm R Macleod, SpR, Neurology Clinical Neurosciences, University of Edinburgh, EH4 2XU, Paul Kelly, Peter Sandercock, Pandora Pound, Shah Ebrahim and Ian Roberts Send response to journal: Re: Systematic review and meta-analysis make an important contribution to surveying the literature from animal experiments	We agree with Lemon and Dunnett (1) that “we need better methods of surveying the literature on animal experiments”, but we do not share their confidence in the utility of non-systematic reviews. We accept that the techniques of systematic review and meta-analysis have only recently been applied to the animal literature, but they still have an important contribution to make. Lemon and Dunnett state that “no mechanism exists for so called negative results to be published”. It is widely accepted that it is unethical not to publish negative results from clinical trials, because this may lead to the study being unwittingly repeated by other investigators, exposing trial participants to risk of drug side effects with no prospect of benefit. We consider that non- publication of data from animal studies is equally unethical, because this may lead to unnecessary repetition by investigators unaware of previous negative work, with attendant unnecessary use of experimental animals. Editorial policies of scientific Journals should acknowledge the scientific and ethical importance of publication of negative results. Sample size in animal experiments is small, for reasons of ethics and of cost. Lemon and Dunnett argued that tight control of experimental conditions minimises variance and obviates the need for larger samples. A systematic review of the 29 studies of FK506 in animal models of stroke (2) suggested otherwise; none of the studies reported a sample size calculation, yet the observed variance in these studies suggested that, to detect an improvement in outcome of 20% with 80% power, would require at least 65 animals per group. Of the studies in the review, the largest study reported 16 animals per group and the average was 8 animals per group (2). Performing underpowered studies is as unethical in animal studies as it is in human studies. The systematic approach to searching for studies for inclusion in a systematic review is intended to broaden the scope of included data (through identifying hitherto hidden information in obscure Journals or that published in abstract only) rather than to narrow it, and to reduce the risk of inclusion bias. Indeed, in 3 systematic reviews of candidate neuroprotective drugs the systematic search criteria did not state any exclusion criteria (2;3;4). In our view, the danger of “critical” but non- systematic reviews is that they are prone to inclusion bias and hence, by selective citation, may only serve to reinforce the prejudice of the authors. Basic science often proceeds with the testing of sequential hypotheses, in contrast to the testing of the same hypothesis, often in different populations, which is the basis of clinical trials. We would argue, however, that there are some circumstances – for instance the pre- clinical testing of candidate stroke drugs in different animals and in different models of ischaemia (5) – which have more in common with clinical trials than with fundamental science research, and for which the use of systematic review and meta-analysis is valid. Systematic review and meta-analysis have made an important contribution to our understanding of sources of bias in clinical trial methodology which in turn contributed to improving the quality of the design, conduct and reporting of clinical trials. We believe this approach might also lead to increased understanding of sources of bias in animal experiments, with the prospect of gains for experimental methodology, animal welfare and clinical science. Animal experiments have improved our understanding of disease pathophysiology and contributed to the development of effective treatments for many diseases. However, there are cases where the benefits seen in animal models have been lost in translation. Those animal studies may have been falsely positive, or the clinical trials may have been falsely negative. In either case, identifying the determinants of this discrepancy is a matter of some urgency, and a problem to which all available tools – including systematic review – should be brought to bear. To that end, developing the methodology for systematic reviews of animal experiments would be aided if funding agencies such as the MRC were to take a lead in bringing together methodologists, scientists working with animal models of disease and clinical scientists to agree priority areas for research and a coherent strategy to improve the transition from bench to bedside. References (1) Lemon R, Dunnett SB. Surveying the literature from animal experiments. BMJ 2005; 330(7498):977-978. (2) Macleod MR, O'Collins T, Horky LL, Howells DW, Donnan GA. Systematic review and metaanalysis of the efficacy of FK506 in experimental stroke. J Cereb Blood Flow Metab 2005; Epub 9th Feb 2005 DOI 10.1038/sj.jcbfm.9600064. (3) Macleod MR, O'Collins T, Howells DW, Donnan GA. Pooling of animal experimental data reveals influence of study design and publication bias. Stroke 2004; 35(5):1203-1208. (4) Macleod MR, O'Collins T, Horky LL, Howells DW, Donnan GA. Systematic review and meta-analysis of the efficacy of melatonin in experimental stroke. J Pineal Res 2005; 38(1):35-41. (5) Recommendations for Standards Regarding Preclinical Neuroprotective and Restorative Drug Development. Stroke 1999; 30(12):2752 -2758. Malcolm Macleod, Paul Kelly, Peter Sandercock; Clinical Neurosciences, University of Edinburgh Pandora Pound; Primary Care, Peninsula Medical School Shah Ebrahim; Social Medicine, University of Bristol Ian Roberts; Epidemiology and Population Health, London School of Hygeine and Tropical Medicine Competing interests: MRM and IR have conducted systematic reviews of animal experiments
Avoidance of bias is the objective of systematic review, not meta-analysis 1 June 2005
Khalid S. Khan, Professor University of Birmingham, UK, Luciano Mignini, Research fellow, Centro Rosarino de Estudios Perinatales, Rosario, Argentina Send response to journal: Re: Avoidance of bias is the objective of systematic review, not meta-analysis	The extent to which surveys of animal research can serve as useful tools in translation from laboratory to human research and healthcare depends on how reviewers employ strategies to minimise bias. We recently undertook a review of methodological quality of existing reviews of animal studies. We searched Medline and Embase (1996-2004) using a carefully designed search term combination with input from expert librarians, examined bibliographies of known reviews and contacted experts. From 4749 citations initially identified, 1517 were considered potentially relevant and their full manuscripts were evaluated. Finally, 30 reviews of animal studies which, as a minimum, performed search of a publicity available resource, were selected for detailed analysis (reference list available on request). We defined animal studies as those in which laboratory variables were measured or where treatments were administrated to live animals to examine their effects. Our analysis (Figure 1) showed that these reviews often lacked methodological features such as prior specification of a testable hypothesis (9/30, 30%); literature search without language restriction (8/30, 27%); and assessment for the risk of missing studies (5/30, 17%). Among these reviews 12/30 (40%) used a statistical combination of individual results, however, the proportion of reviews of animal research that would benefit from meta-analytic techniques is unknown. The features that we found most disturbing were that data syntheses among reivews usually ignored methods to assess the validity of included studies (15/30, 50%) and the heterogeneity between them (20/30, 66%). These are key features in assessing suitability of combining results statistically and gauging the strength of any inferences. We concur with Lemon and Dunnett's identification of the need for 'better methods of surveying the literature on animal experiments' as our study showed existing reviews of animal research were poor in methodological features increasing the risk of bias. However, we fail to understand their objection to the use of systematic reviews (1, 2). Their argument seems grounded in that it is 'difficult to combine data from different studies' in animal research. Perhaps it needs to be emphasised that meta-analysis is not the objective of a systematic review. Reviews earn the adjective systematic by use of explicit methods to minimise bias at every step of the reviewing process including literature search, study selection, critical appraisal and data synthesis. The need for rigour when reviewing animal research is undeniable and systematic review methodology provides a sound strategy for conducting such reviews. Figure 1: Methodological features of reviews of animal research (1996-2004). Data presented as 100% stacked bars; figures in the stacks represent number of studies. 1. Khan KS, Ter Riet G, Glanville J, Sowden AJ, Kleijnen J (eds) for the NHS Centre for Reviews and Dissemination (CRD). Undertaking Systematic Reviews of Research on Effectiveness. CRD's Guidance for Carrying Out or Commissioning Reviews. 2nd Edition. CRD Report No. 4. York: NHS Centre for Reviews and Dissemination, University of York, 2001. Available free at www.york.ac.uk/inst/crd/report4.htm. 2. Egger M, Davey-Smith G, Altman DG (eds). Systematic Reviews in Health Care. Meta-analysis in Context. London: BMJ Publishing Group, 2001. Available at www.systematicreviews.com. Competing interests: None declared