18-Aug-2016 Dear Dr. Nagendran Manuscript ID BMJ.2016.034309 entitled "Do randomised trials with very large treatment effects make further trials unnecessary? An empirical assessment." We sent it for external peer review and discussed it at our manuscript committee meeting. We recognise its potential importance and relevance to general medical readers, but I am afraid that we have not yet been able to reach a final decision on it because several important aspects of the work still need clarifying. We hope very much that you will be willing and able to revise your paper as explained below in the report from the manuscript meeting, so that we will be in a better position to understand your study and decide whether the BMJ is the right journal for it. We are looking forward to reading the revised version and, we hope, reaching a decision. Yours sincerely, Georg Roeggla [email protected] **Report from The BMJ’s manuscript committee meeting** These comments are an attempt to summarise the discussions at the manuscript meeting. They are not an exact transcript. Manuscript meeting 18.08.2016 Jose Merino (chair), Angela Wade (stats), Elizabeth Loder, Georg Roggla, Wim Weber, Tiago Villanueva, Jessamy Bagenal, Amy Price Decision: Ask for revision The committee was interested in the topic of your research. The following concerns were mentioned: • Were patients involved in discussion of this paper at outset? • There's a bit about patient recruitment in the PI statement, as if recruitment and involvement were synonymous. • What does this add to the recent JAMA paper (http://jama.jamanetwork.com/article.aspx?articleid=1386610) • The committee agreed with the issue about absolute risks vs relative risks raised by the reviewer Siegerink. • This is interesting and thought-provoking but the committee had concerns about the methods you've used to identify the initial VLE study and the subsequent larger study. It's not until Table 2 that we get a sense of the actual clinical questions and topics that are involved. One of them, we note is rizatriptan (where the initial VLE was upheld by a subsequent trial although it wasn't quite as large). We could not, however, locate which small study they referred to, or which subsequent large rizatriptan trial. However, having this concrete example made us realize that the authors don't take into account the context of a VLE. Riza was a me-too drug and there was every reason to think its effects would be in line with previous triptan results. It doesn't seem quite right to consider it in isolation from the known effects of similar drugs. We question whether a subsequent large trial was needed (or even ethical, if it involved placebo), yet the authors would say it was. • Unless we are interpreting table 2 incorrectly, it seems more VLEs were upheld than refuted. • Could you define what you mean by empirical? It appears in the title and then in the text. Also need to define "agreed with." • We thought this is an interesting read, although a bit abstract. First, please revise your paper to respond to all of the comments by the reviewers. Their reports are available at the end of this letter, below. Please also respond to the additional comments by the committee. In your response please provide, point by point, your replies to the comments made by the reviewers and the editors, explaining how you have dealt with them in the paper. ** Comments from the external peer reviewers** Reviewer: 1 Recommendation: Comments: Peer review report BMJ.2016.034309 Bob Siegerink, PhD [email protected] / bobsiegerink.com To the editor and authors I have read with great interest the paper by Nagendran et al with the title “Do randomised trials with very large treatment effects make further trials unnecessary? An empirical assessment”, aka BMJ.2016.034309. The authors ask whether a very large treatment effect (RR >5 or its inverse) in a trial can be seen as an indication of no need for follow up trials. This question, linked to the day-to-day practice that small trials provide an “obvious benefit” and therefore clinical equipoise is not applicable anymore, is a relevant question as it addresses concepts like reproducibility of research, potential source of publication bias as well as introducing new treatments that potentially could harm patients. The authors use data from the Cochrane reviews to which they apply several methods to identify RCT with very large effects and subsequent large trials that either refute or uphold that effect. The results indicate very large effects are not a reliable marker for true clinical benefit. The authors even draw the conclusion that therefore these smaller trials might better be skipped and that a new treatment should be investigated with a large trial at the end. The paper is well written, clear in its approach, results well reported and the topic is both relevant as well as attractive and relevant for readers of the BMJ. Nonetheless, I have several comments and question, which I will provide below in a point-by-point fashion. Please note that I have no conflict of interest with regard to this paper and that I am very happy to relinquish my anonymity in this peer review process. Bob siegerink // Major comments. 1. relative risks vs absolute risks. All calculations, selections and definitions are based on relative risks. Even though these measures are the most used measured to estimate the effect size of an intervention, they do not convey the true impact of an intervention. For this, a form of absolute measure is needed, which can either be an absolute risk reduction, number needed to treat or otherwise. This is not only relevant to understand what “very large effects” indeed mean in clinical practice, but is also important as the initial VLE trial and subsequent large trial can have similar RR while the NNT is different, and vice versa. The figure below shows a numeric example, in which only the absolute risk and total number of persons in the subsequent trials is changed. IN each of the subsequent trials, either the RR or the ARR is similar to the initial VLE trial even though the other measure is then not the same. This shows that there is a problem in interpretation of the RR between trials when there is no information on the absolute risk. It is not unrealistic to think that the subsequent analyses trial could have a different absolute risk compared to the VLE trial, either due to chance, different inclusion criteria, change in co-treatment over time periods, change in geographic location etc. I am not sure to what extend whether such scaling of the RR indeed hampers the interpretation of the results provided by the authors. Details on the absolute risk in both the VLE trials and subsequent trial (e.g. extra row in table 2) would help the reader to understand the impact of this scaling problem. << please refer to pdf for table>> In line with this question on the absolute risk, where the authors able to ensure that there was sufficient overlap in the patient populations of the initial and subsequent trials (overlapping external validity?) 2. upheld/refute. The definitions of “upheld” and “refute” are unclear to me. The authors describe their two methods on page 10 line 4 to 14. The first approach is clear, but the second approach is unclear, as it speaks of “agreed with the index trial” in which agreed which is not defined. After repeatedly reading the second approach, I think the authors mean that when multiple large trials are present, a meta analyses is performed and the result of this meta analyses is checked based on the same criteria as the first approach (effect in opposite direction or non-statistically significant effect). But if this is the case, does approach 1 not rule out that approach 2 is even possible? The figures provide some indication on the definitions, i.e. that everything that stays significant is upheld. Although understandably when it comes to clear definition, I doubt whether a statistical significant of RR 1.1-2 in a subsequent large trial should indeed be seen as “upheld”. It is quite dependent on the research question (PICO) whether the finding of this second trials should next to statistically significant also be seen as ‘just as clinically relevant” as the finding of the index trial. 3. Selection bias. I wonder whether this analyses suffers from selection bias due to the definition of index trial. As is stated at page 7 line 35 an index trial is a VLE trial followed by at least one large trial. If this is the case selection bias might be introduced through this selection: consider all VLE trials that are possible. It is likely that the characteristics of those who are followed by a large trial are indeed different from those VLE trials who are not followed by large trials – and as such never reach the status of index trial-. Perhaps VLE trials who are followed by a large trial (i.e. index trial) are more “disbelieved” by the medical community, perhaps due subject matter knowledge of problems with the initial VLE trial. VLE trials who are not followed by a large trial are could indeed be more solid in terms of effect and these are, as far as I can judge, not part of this analyses. If this is the case, it is no wonder that VLE in the index trials are indeed a poor predictor of success in a subsequent large trial. There is some information in the paper which makes the it not easier for the reader to understand what is going on, being the reference in the flowchart which reads “index RCT not followed by RCT=2”.