This page provides a list of questions you may like to ask when you are trying to work out how scientifically rigorous a specific research study is.
The more times you can answer 'Yes', the more likely the study will be valid and reliable - although this is only a very rough rule of thumb.
Please note: Scientists never draw firm conclusions from just one study or set of results. They consider the contribution it makes in the context of other work and their own experience.
(The peer review process helps to ensure that the information presented in a research study is scientifically rigorous because it has been checked by other scientists.)
(A well-defined and specific research question is more likely to provide useful information than a question that is vague and poorly-defined.)
(Some researchers may be setting out to prove that an intervention works, irrespective of what the data in their research shows. Other researchers may stand to gain financially if, for example, they are providers of the intervention.)
(A study run by different research teams on different sites is less likely to be biased than a study run by a single team on the same site.)
(If the study does not include enough participants the results may not be statistically significant or applicable to other people - Please see Note 1 at the end of this page.)
(It is important to record important information about the participants as this could affect the results of the intervention. For example, some interventions may not be effective for particular groups of people.)
(A control group is important because it can help to show that any changes in the experimental group are likely to be as a result of the intervention.)
(If the control group is not the same as the experimental group any differences in the results may not be the result of the treatment.)
(Randomisation ensures that no one is able to influence who is put in which group and this reduces the risk of bias.)
(This makes it easier for other researchers to replicate the study and check whether the findings are accurate.)
In some studies, the control group may be accessing other types of interventions, which may be having their own effects. Not reporting these makes it difficult to compare the effects of the intervention given to the treatment group compared to what the control group is receiving.
(Blinding ensures that the participants (and any parent/carers) do not behave in a particular way because they know which treatment they are receiving, which could alter the results of the study.)
(Blinding ensures that the assessors do not make judgements about the success of the intervention based on their beliefs about which participants are receiving which treatments.)
(Using widely recognised and relevant outcome measures ensures that the results are more likely to be valid and that they can be interpreted by other researchers.)
(If the study uses different measures before and after it can be difficult to compare the effect of the intervention.)
(Using appropriate and widely-accepted statistical techniques means that any results reported in the study are more likely to be valid and reliable.)
The following questions do not necessarily tell you how scientifically rigorous a specific research study is. But they are important questions you may want to ask anyway.
(Ethical approval safeguards the rights of participants to be treated as openly and fairly as possible within the research, and to consent fully to taking part.)
(Including autistic people on the research team, rather than just using them as study participants, means that the study is more likely to be of relevance to other autistic people.)
(Many research studies ask purely theoretical questions that have no practical application to people on the autism spectrum, their parents/ carers or service providers.)
There is no general agreement on how many participants is 'enough' for a research study because it depends on so many other factors. So some researchers may believe that three participants is enough - provided they are part of a study that uses a reasonably robust design, such as a multiple baseline design. Other researchers may believe that a minimum of 60 participants is required - provided they are part of a study that uses a very robust design, such as a randomised controlled trial.
These questions are taken from 'Choosing Autism Interventions: A Research-Based Guide' which was published in April 2015.