How video publication of laboratory experiments will solve the reproducibility problem: The Journal of Visualized Experiments
One day, my supervising professor came to me in the lab, pointing to an article in a prestigious research journal, and indicating that we must learn the methodology described in the paper in order to implement it in our laboratory. Back then, in 2005, I was a graduate student in a biology lab at Princeton University, New Jersey. I did my best, but replication of the method and results did not work out. The professor put someone else on the task. Replication still did not work out. The professor came to see me again, and suggested that I visit the lab initially conducting the research. So I got on a plane bound for Edinburgh, Scotland. I spent two weeks in the lab, immersed myself in the procedures, returned to Princeton, and successfully replicated the experiment.
Reproducibility in science is a big issue. In the 21st century, and given all the Internet technologies available, to me it seemed somewhat cumbersome and out-dated that we were seeking experimental replication based on a description in a paper, only to find that we had to travel far away, to the other lab.
In what follows, I would like to show how the video recording of experiments effectively and reliably addresses the problem of reproducibility in research. From this insight emerged the Journal of Visualized Experiments (JoVE), now serving ten broad research fields across the sciences, and subscribed to by more than 800 institutions.
1.Failures of reproducibility in science
Reproducibility in the biosciences is a known issue. But it had not been quantified until two pharmaceutical companies started systematic trials. In 2012, Bayer (Germany) and Amgen (USA) each took about 100 articles from highly cited journals and asked staff to reproduce the results from the descriptions contained in the articles. Full replication was possible for less than 20% of the research results. Conversely, in more than 70% of the cases, reproduction was not possible at all.
These data do not (necessarily) indicate that research results are not reproducible, only that this is not possible based on the information and data contained in the publication. To be sure, research publications have seen many enhancements, many of which have been presented and reviewed at the Academic Publishing in Europe conference in January 2015. In essence, however, the research publication is still a ‘paper’, much like the Philosophical Transactions of the Royal Society in 1665.
2.Facilitating reproducibility by recording experiments
The evidence says that text as a medium does not facilitate reproducibility in science. It does not work for the transfer of complex technical knowledge. Consequently, we need to explore if other mediums will enable the presentation of data and information in a manner that makes replication effective and easy.
When surveying which digital technologies are available for enhancing reproducibility, we found that the video recording of experiments seemed the most promising avenue.
3.Structure of scientific video article
When exploring potential formats of the video article we decided to re-use the structure of the traditional text article. This makes visualized experiments highly compatible with the habits and attitudes of researchers, facilitating adoption. The structure should be: (1) Abstract; (2) Introduction; (3) Experimental procedure; (4) Discussion of results (see Fig. 1).
Typically, we ask researchers to submit their manuscript and, if it passes peer review, we will produce the video.
Fig. 1.
Structure of the video article. (Colors are visible in the online version of the article; http://dx.doi.org/10.3233/ISU-150778.)
4.A videographer network
Currently, we publish about 80 video articles per month. We organize production, filming the experiment in the researchers’ lab. We rely on a network of videographers covering nearly 20 countries that host major research institutions. These include USA, Canada, UK, Germany, Holland, France, Israel, Australia, Japan, Korea, and others. Experiments typically last several hours, and the material we shoot will also be of substantial duration. From this we have to extract about 10 minutes of useful information, the information that makes the experiment reproducible. Video articles pass editorial review and go into production.
5.JoVE: Journal of Visualized Experiments
“By having the video protocol … anybody quickly can learn the technique whenever it is convenient for her or him, saving them lots of time, money and frustration.”
Dr Maria Rodriguez-Toledeo, University of Nottingham.
By 2015, JoVE has published about 4,000 video articles, currently adding about 1,000 articles per year. Fields that we cover are:
JoVE Biology;
JoVE Neuroscience;
JoVE Immunology and Infection;
JoVE Bioengineering;
JoVE Translational and Clinical Medicine;
JoVE Developmental Biology;
JoVE Behavior;
JoVE Applied Physics;
JoVE Chemistry;
JoVE Environment.
6.Calculating the return on investment for researchers
To better understand the benefits JoVE generates for researchers and their institutions, we are conducting case studies. Consider the following example: Dr Theresa Casey, Department of Animal Science, Purdue University, researches mammary development and lactation. She had discovered that activity of circadian clock genes changed during the transition from pregnancy to lactation in other tissues. The circadian clock is governed by the suprachiasmatic nucleus (SN) in the hypothalamus area of the brain. She needed to learn to dissect the SN brain area using surgery.
On JoVE she found the video article:
Slice Preparation, Organotypic Tissue Culturing and Luciferase Recording of Clock Gene Activity in the Suprachiasmatic Nucleus. By Savelyev et al., 2011, JoVE (Karolinska Institute).
Dr Casey says that after watching this JoVE video article, she spent only 2 weeks instead of 2 months to learn the experiment. She saved 2 month of work time and $15,000 of research funds ($7,700 reagents; $1,100 travel; $6,200 salary).
Our calculation for Purdue University is as follows. There are 100+ labs at Purdue. If between 10% and 50% of the laboratories can benefit from JoVE once during a year, then the annual saving is $150,000 to $750,000.
7.Users and subscribers
JoVE currently sees about 500,000 unique visitors per month, of which about 80% are from academic institutions. Our business model is based on subscriptions. The number of subscribing institutions has increased from 90 in 2009 to more than 800 by 2015 (Fig. 2).
Fig. 2.
Number of institutional subscriptions. (Colors are visible in the online version of the article; http://dx.doi.org/10.3233/ISU-150778.)
8.Conclusions
The success of JoVE clearly demonstrates the demand for the video-based publication in the scientific community. We expect that the growing debate on the research reproducibility and productivity will highlight this demand and increase the acceptance of this novel method of publishing among scientists, students, academic librarians, and scholarly publishers.
