Research transparency - Knowledge (XXG)

313: 733:(PREP). This initiative also aims to solve the structural scarcity of data and empirical information on editorial policies and peer review practices. As of 2022, this database contains partially crowdsourced information on the editorial procedures of 490 journals, from an initial base of 353 journals. The procedures evaluated include especially "the level of anonymity afforded to authors and reviewers; the use of digital tools such as plagiarism scanners; and the timing of peer review in the research and publication process". Despite this developments, research on editorial research still highlight the need for the "a comprehensive database that would allow authors or other stakeholders to compare journals based on their (…) requirements or recommendations" 696:

meta-analysis of 25 replications published between 1986 and 2019. It finds that the majority of the replication concern the medical and social sciences (especially, psychology and behavioral economics) and that there is for now no standardized evaluation criteria: "methods of assessing replicability are inconsistent and the replicability percentages depend strongly on the methods used." Consequently, at least as for 2019, replication studies cannot be aggregated to extrapolate a replicability rate: they "are not necessarily indicative of the actual rate of non-replicability across science for a number"

417:. Access is no longer the main dimension of open science, as it has been extended by more recent commitments toward transparency, collaborative work and social impact. Through this process, open science has been increasingly structured over a consisting set of ethical principles: "novel open science practices have developed in tandem with novel organising forms of conducting and sharing research through open repositories, open physical labs, and transdisciplinary research platforms. Together, these novel practices and organising forms are expanding the ethos of science at universities." 396:

which makes it harder to assess what could be the necessary steps to overcome the issue at plays. The Nature survey has also been criticized for its paradoxical lack of research transparency, since it was not based on a representative sample but an online survey: it has "relied on convenience samples and other methodological choices that limit the conclusions that can be made about attitudes among the larger scientific community" Despite mixed results, the Nature survey has been largely disseminated and ahs become a common entry data for any study of research transparency.

298:, two scientific journalists described scientific fraud as a structural problem: "As more cases of frauds broke into public view (…) we wondered if fraud wasn't a quite regular minor feature of the scientific landscape (…) Logic, replication, peer review — all had been successfully defied by scientific forgers, often for extended periods of time". The codification of research integrity has been the main institutional answer to this increased public scrutiny with "numerous codes of conduct field specific, national, and international alike." 578:

been motivated by the diversification and the complexification of the open science publishing landscape: "Triggered by a wide variety of expectations for journals’ editorial processes, journals have started to experiment with new ways of organizing their editorial assessment and peer review systems (...) The arrival of these innovations in an already diverse set of practices of peer review and editorial selection means we can no longer assume that authors, readers, and reviewers simply know how editorial assessment operates."

523:

are the original procedures matched and criterium may vary depending on the disciplines or, even on the field of research. Consequently, meta-analysis of reproducibility have faced significant challenges. A 2015 study of 100 psychology papers conducted by Open Science Collaboration has been confronted with the "lack of a single accepted definition" which "opened the door to controversy about their methodological approach and conclusions" and made it necessary to fall back on "subjective assessments" of result reproducibility.

255:, by describing in detail a research design that has only been run once For Friedrich Steinle, the gap between the postulated virtue of transparency and the material conditions of science has never been solved: "The rare cases in which replication actually is attempted are those that either are central for theory development (e.g., by being incompatible with existing theory) or promise broad attention due to major economical perspectives. Despite the formal ideal of replicability, we do not live in a culture of replication." 139:

Questionable research practices are more widespread as more than one third of the respondents admit to have done it once. A large 2021 survey of 6,813 respondents in the Netherlands found significantly higher estimate, with 4% of the respondents engaging in data fabrication and more than half of the respondents engaging in questionable research practices. Higher rates can be either attributed to a deterioration of ethic norms or to "the increased awareness of research integrity in recent years".

433:

this sense the most advanced of the four CoCs." First adopted in July 2020, the Hong Kong principles for assessing researchers acknowledge open science as one of the five pillars of scientific integrity: "It seems clear that the various modalities of open science need to be rewarded in the assessment of researchers because these behaviors strongly increase transparency, which is a core principle of research integrity."

172:

and research design were not significantly better conceived and the rate of false or partially false has likely remained approximately constant for the last decades. Consequently, proponents of research transparency have come to embrace more explicitly the discourse of open science: the culture of scientific transparency becomes a new ideal to achieve rather than a fundamental principle to re-establish.

385: 712:. The declaration underlined that journals "often do not contain information about reviewer selection, review criteria, blinding, the use of digital tools such as text similarity scanners, as well as policies on corrections and retractions" and this lack of transparency. The declaration identifies four main publication and peer review phases that should be better documented: 557:

scientific advising as pre-defined values may largely predate choices about the concepts, the protocols and the data used. Douglas argued instead in favor of a disclosure of the values held by researchers: "the values should be made as explicit as possible in this indirect role, whether in policy documents or in the research papers of scientists."

683:

has been argued that preregistration may also in some cases harm the quality of the research output by creating artificial constraints that do not fit with the reality of the research field: "Preregistration may interfere with valid inference because nothing prevents a researcher from preregistering a poor analytical plan."

488:, as the open code can rely on external programs which may not always be available or compatible. Two studies in 2018 and 2019 have shown that a large share of research notebook hosted on GitHub are no longer usable, either due to the of required extensions no longer being available or issues in the code. 682:

Preregistration do not solve all the range of questionable research practices. Selective reporting of the results would especially still be compatible with a predefined research plan: "preregistration does not fully counter publication bias as it does not guarantee that findings will be reported." It

535:

as the "expectation being that any skilled researcher placed in the same time and place would pick out, if not the same data, at least similar patterns". This expectation recovers a large range scientific and scholarly practices in non-experimental disciplines: "A tremendous amount of research in the

428:

were elaborated in 2014 by a committee for Transparency and Openness Promotion that included "disciplinary leaders, journal editors, funding agency representatives, and disciplinary experts largely from the social and behavioral sciences". The guidelines rely on eight standards, with different levels

420:

The global scale of the open science movement and its integration in a large variety of technical tools, standards and regulations makes it possible to overcome the "classic collective action problem" embodied by research transparency: there is a structural discrepancy between the stated objective of

246:

Beyond this lack of formalization, there was a significant drift between the institutional and disciplinary discourse on research transparency and the reality of research work, that has persisted till the 21st century. Due to the high cost of the apparatus and the lack of incentives, most experiences

171:

This open science framework of transparency has been in turn coopted by leading contributors and institutions on the topic of research transparency. After 2015, contributions from science historians underlined that there have been no significant deterioration of research quality, as past experiments

4369:

Squazzoni, Flaminio; Ahrweiler, Petra; Barros, Tiago; Bianchi, Federico; Birukou, Aliaksandr; Blom, Harry J. J.; Bravo, Giangiacomo; Cowley, Stephen; Dignum, Virginia; Dondio, Pierpaolo; Grimaldo, Francisco; Haire, Lynsey; Hoyt, Jason; Hurst, Phil; Lammey, Rachael; MacCallum, Catriona; Marušić, Ana;

577:

Editorial transparency has been recently acknowledged as a natural expansion of the debate over research reproducibility. Several principles laid in the 2015 TOP guidelines already implied the existence of explicit editorial standards. Unprecedented attention given to editorial transparency has also

522:

Result reproducibility is harder to achieve than other forms of research transparency. It involve a variety of issues that may include computational reproducibility, accuracy of scientific measurement and diversity of methodological approaches. There are no universal standard to determine how close

395:

highlighted that "more than 70% of researchers have tried and failed to reproduce another scientist's experiments, and more than half have failed to reproduce their own experiments" The survey also found "no consensus on what reproducibility is or should be", in part due to disciplinary differences,

62:

Differences between disciplines and epistemic cultures has largely contributed to different acceptions. The reproduction of past research has been a leading source of dissent. In an experimental setting, reproduction relies on the same set-up and apparatus, while replication only requires the use of

1876:

Nosek, B. A.; Alter, G.; Banks, G. C.; Borsboom, D.; Bowman, S. D.; Breckler, S. J.; Buck, S.; Chambers, C. D.; Chin, G.; Christensen, G.; Contestabile, M.; Dafoe, A.; Eich, E.; Freese, J.; Glennerster, R.; Goroff, D.; Green, D. P.; Hesse, B.; Humphreys, M.; Ishiyama, J.; Karlanflup, D.; Kraut, A.;

543:

Increased transparency of citations to primary sources or research materials has been framed by Andrew Moravcsik as a "revolution in qualitative research". Access to theses resources has been thoroughly transformed by digitization and the attribution of unique identifiers. Permanent digital object

432:

After 2015, theses initiatives have partly influenced new regulations and code of ethics. The European Code of Conduct for Research Integrity from 2017 is strongly structured around open science and open data: it "pays data management almost an equal amount of attention as publishing and is also in

289:

published one of the first generic evaluation of statistical methods in 67 leading medical journals. While few outright problematic papers were found, "in almost 73% of the reports read (those needing revision and those which should have been rejected), conclusions were drawn when the justification

229:

described extensively of the forms and benefits of procedural experimentation, that made it possible to check for random effects, the soundness of the experiment design, or causal relationships through repeated trials Replication and the open documentation of scientific experiments has become a key

590:

have set up an influential transdisciplinary standard to establish result reproducibility in an open science context. While experimental and computational disciplines remains a primary focus, the standards have strived to integrate concerns and formats more specific to other disciplinary practices

399:

Reproducibility crisis and other issues of research transparency have become a public topic addressed in the general press: "Reproducibility conversations are also unique compared to other methodological conversations because they have received sustained attention in both the scientific literature

539:

The development of open scientific infrastructure has radically transformed the status and the availability of scientific data and other primary sources. Access to theses resources has been thoroughly transformed by digitization and the attribution of unique identifiers. Permanent digital object

467:

in an experimental context. In the report of the National Academies of Science, that opted for an experimental terminology, the counterpart of method reproducibility was described as "obtaining consistent results using the same input data; computational steps, methods, and code; and conditions of

408:

Since 2000, the open science movement has expanded beyond access to scientific outputs (publication, data or software) to encompass the entire process of scientific production. In 2018, Vicente-Saez and Martinez-Fuentes have attempted to map the common values shared by the standard definitions of

320:

In the 2000s, long-standing issues on the standardization of research methodology have been increasingly presented as a structural crisis which "if not addressed the general public will inevitably lose its trust in science." The early 2010s is commonly considered to be a turning point: "it wasn’t

293:

In the 1970s and the 1980s, scientific misconducts gradually ceased to be presented as individual misconducts and became collective problems that need to be addressed by scientific institutions and communities. Between 1979 and 1981, several major cases of scientific frauds and plagiarism draw a

267:

has created unprecedented challenges for research transparency. The generalization of statistical methods across a large number of fields, as well as the increasing breadth and complexity of research projects, entailed a series of concerns about the lack of proper documentation of the scientific

243:

largely unclear and unharmonized across disciplines: "What one group means by one word, the other group means by the other word. These terms — and others, such as repeatability — have long been used in relation to the general concept of one experiment or study confirming the results of another."

242:

appeared in an article on the "Methods of illuminations" first published in 1902: one of the methods examined was deemed limited regarding "reproducibility and constancy" In 2019, the National Academies underlined that the distinction between reproduction, repetition and replication has remained

679:. It takes usually the form of "a timestamped uneditable research plan to a public archive states the hypotheses to be tested, target sample sizes". Preregistration acts as an ethical contract as it theoretically constrains "the researcher degrees of freedom that make QRPs and p-hacking work". 556:

underlined that the public discourse on science has been largely dominated by normative ideals of objective research: if the procedures have been correctly applied, science results should be "value-free". For Douglas, this ideal remains largely at loss with the effective process of research and

518:

met in experimental disciplines like psychology or medicine is mostly a crisis of "result reproducibility", since it concerns research that cannot been simply re-executed, but involve the independent recreation of the experimental design. As such it is arguably the most debated form of research

446:

introduced a taxonomy of eight dimensions of research transparency: purpose, audience, content, timeframe, actors, mechanism, venues and dangers. For Elliott not all forms of transparency are achievable and desirable, so that a proper terminology can help to make the more appropriate decisions:

138:

In 2009, a meta-analysis of 18 surveys estimated that less than 2% of scientists "admitted to have fabricated, falsified or modified data or results at least once". Real prevalence may be under-estimated due to self-reporting: regarding "the behaviour of colleagues admission rates were 14.12%".

126:

of problematic practices, which are frequently associated to deficiencies in research transparency. In 2016, a study identified as much as 34 questionable research practices or "degree of freedom", that can occur at all the steps of the project (the initial hypothesis, the design of the study,

167:

have significantly contributed to bring transparency on the agenda of the open science movements. They aim to promote an "open research culture" and implement "strong incentives to be more transparent". They rely on eight standards, with different levels of compliance. While the standards are

686:

While advocated as a relatively cost-free solution, preregistration may be in reality harder to implement as it relies on a significant commitment on the part of the researchers. An empiric study of the adoption of open science experiments in a psychology journals has shown that "Adoption of

475:

are common recommendations to enhance method reproducibility. In principle, the wider availability of research output makes it possible to assess and audit the process of analysis. In practice, Roger Peng already underlined in 2011, that many projects require "computing power that may not be

656:, "appropriate citation for data and materials" should be provided each publication. Consequently, scientific outputs like code or dataset are fully acknowledged as citable contributions: "Regular and rigorous citation of these materials credit them as original intellectual contributions." 695:

Replication studies or assessments of replicability aims to re-do one or several original studies. Although the concept has only appeared in the 2010s, replication studies have been existing for decades but were not acknowledged as such. The 2019 report of the National academies include a

441:

Research transparency has a large variety of forms depending on the disciplinary culture, the material condition of research and the interaction between scientists and other social circles (policy-makers, non-academic professionals, general audience). For Lyon, Jeng and Mattern, "the term

594:

Informal incentives like badges or indexes have been initially advocated as a way to support the adoption of harmonized policies in regard to research transparency. Due to the development of open science, regulation and standardized infrastructures or processes are increasingly favored.

29:

The definitions and norms of research transparency significantly differ depending on the disciplines and fields of research. Due to the lack of consistent terminology, research transparency has frequently been defined negatively by addressing non-transparent usages (which are part of

122:(or QRP) was first incepted in a 1992 report of the Committee on Science, Engineering, and Public Policy as a way to address potentially non-intentional research failures (such as inadequacies in the research data management process). Questionable research practices uncover a large 155:

identified transparency as a "third dimension" of open science, due to the fact that "the concept of transparency and the associated term ‘reproducibility’, have become increasingly important in the current interdisciplinary research environment." According to Kevin Elliott, the

328:. The main argument was based on the excessively lax experimental standards in place, with numerous weak result being presented as solid research: "the majority of modern biomedical research is operating in areas with very low pre- and post-study probability for true findings" 105:

argues that all disciplines, including the social sciences, currently face similar issues to medicine and physical sciences: "The problem, which has come to be known as the reproducibility crisis, affects almost all of science, not one or two individual disciplines."

114:

Due to lack of consistent terminology over research transparency, scientists, policy-makers and other major stake-holders have increasingly rely on negative definitions: what are the practices and forms that harm or disrupt any common ideal of research transparency.

365:

value above the common threshold of 0.05). In 2021, another Reproducibility Project, Cancer Biology, analyzed 53 top papers about cancer published between 2010 and 2012 and established that the effect sizes were 85% smaller on average than the original findings .

237:

Although transparency has been early on acknowledged as a key component of science, it was not defined consistently. Most concept associated today with research transparency have arisen as terms of the art with no clear and widespread definitions. The concept of

447:"While these are important objections, the taxonomy of transparency considered here suggests that the best response to them is typically not to abandon the goal of transparency entirely to consider what forms of transparency are best able to minimize them.". 1877:

Lupia, A.; Mabry, P.; Madon, T.; Malhotra, N.; Mayo-Wilson, E.; McNutt, M.; Miguel, E.; Paluck, E. Levy; Simonsohn, U.; Soderberg, C.; Spellman, B. A.; Turitto, J.; VandenBos, G.; Vazire, S.; Wagenmakers, E. J.; Wilson, R.; Yarkoni, T. (2015-06-26).

699:

The TOPs guidelines have called for an enhanced recognition and valorization of replication studies. The eighth standards state that compliant journals should use "registered Reports as a submission option for replication studies with peer review".

687:

pre-registration lags relative to other open science practices (…) from 2015 to 2020". Consequently "even within researchers who see field-wide benefits of pre-registration, there is uncertainty surrounding the costs and benefits to individuals."

573:

Until the 2010s, the editorial practices of scholarly publishing have remained largely unformal and little studied: "Despite 350 years of scholarly publishing (…) research on ItAs , and on their evolution and change, is scarce."

491:

In experimental sciences, there is no commonly agreed criterium of method reproducibility: "in practice, the level of procedural detail needed to describe a study as "methodologically reproducible" does not have consensus."

130:

Surveys of disciplinary practices have shown large differences in the admissibility and spread of questionable research practices. While data fabrication and, to a lesser extent, rounding of statistical indicators like the

484:, as the development of very large deep learning models makes it nearly impossible to recreate them (or at a prohibitive cost), even when the original code and data are open. Method reproducibility can also be affected by 2721: 37:

After 2010, recurrent issues of research methodology have been increasingly acknowledged as structural crisis, that involve deep changes at all stages of the research process. Transparency has become a key value of the

500:

Goodman, Fanelli and Ioannidis define result reproducibility as "obtaining the same results from the conduct of an independent study whose procedures are as closely matched". Result reproducibility is comparable to

564:

emphasized three conditions for value transparency in research, the first one involved "being as transparent as possible about (…) data, methods, models and assumptions so that value influence can be scrutinized".

429:

of compliance. While the standards are modular, they also aim to articulate a consistent ethos of science as "they also complement each other, in that commitment to one standard may facilitate adoption of others."

230:

component of the diffusion of scientific knowledge in society: once they attained a satisfying rate of success, experiments could be performed in a variety of social spaces such as courts, marketplaces or learned

513:

retained in the National Academies of Science, largely applies to it: "obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data.". The

351: 607: hypothesized that "some kind of registration or networking of data collections or investigators within fields may be more feasible than registration of each and every hypothesis-generating experiment." 204:

Transparency has been a fundamental criterion of experimental research for centuries. Successful replications have become an integral part of the institutional discourse of natural sciences (then called

4370:

Mehmani, Bahar; Murray, Hollydawn; Nicholas, Duncan; Pedrazzi, Giorgio; Puebla, Iratxe; Rodgers, Peter; Ross-Hellauer, Tony; Seeber, Marco; Shankar, Kalpana; Van Rossum, Joris; Willis, Michael (2020).

630:

While citation standards are commonly applied to academic reference, there is much less formalization for all the other research output, such as data, code, primary sources or qualitative assessments.

3361:

Munafò, Marcus; Nosek, Brian; Bishop, Dorothy; Button, Katherine; Chambers, Christopher; Percie du Sert, Nathalie; Simonsohn, Uri; Wagenmakers, Eric-Jan; Ware, Jennifer; Ioannidis, John (2017-01-10).

71:. Alternative taxonomies have proposed to make do entirely with the ambiguity of reproducibility/replicability/repeatability. Goodman, Fanelli and Ioannidis recommended instead a distinction between 536:

medical, historical and social sciences does not rest on experimentation, but rather on observational techniques such as surveys, descriptions and case reports documenting unique circumstances"

281:. These rely on the assumption that quantitative results and the details of the experimental and observational framework are sound (such as the size or the composition of the sample). In 1966, 471:

Method reproducibility is more attainable in computational sciences: as long as it behaves as expected, the same code should produce the same output. Open code, open data and more recently,

459:

as "the provision of enough detail about study procedures and data so the same procedures could, in theory or in actuality, be exactly repeated." This acception is largely synonymous with

4323: 637:

adopted new policies for open qualitative research. They covered three dimensions of transparency: data transparency (in the sense of precise bibliographic data to the original sources),

151:. Until the 2010s, definitions of open science have been mostly focused on technical access and enhanced participation and collaboration between academics and non-academics. In 2016, 369:

During the 2010s, the concept of reproducibility crisis has been expanded to a wider array of disciplines. The share of citations per year of the seminal paper of John Ioannidis,

622:(4). All the relevant data, code and research materials are to be stored on a "trusted repository" and all analysis being already reproduced independently prior to publication. 168:

modular, they also aim to articulate a consistent ethos of science as "they also complement each other, in that commitment to one standard may facilitate adoption of others.".

442:‘transparency’ has been applied in a range of contexts by diverse research stakeholders, who have articulated and framed the concept in a number of different ways." In 2020, 2773: 2712: 424:

The formalization of open science as a potential framework to ensure research transparency has been initially undertaken by institutional and communities initiatives. The

603:

Data sharing has been early on identified as major potential solution to the reproducibility crisis and the lack of solid guidelines for statistical indicators. In 2005,

552:

Transparency of research values has been a major focus of disciplines with strong involvements in policy-making such as environment studies or social sciences. In 2009,

42:

movement, which evolved from an initial focus on publishing to encompass a large diversity of research outputs. New common standards for research transparency, like the

86:

retained the experimental definition of replication and reproduction, which remains "at odds with the more flexible way they are used by major organizations". The

4494:

Brock, Theo C. M.; Elliott, Kevin C.; Gladbach, Anja; Moermond, Caroline; Romeis, Jörg; Seiler, Thomas-Benjamin; Solomon, Keith; Peter Dohmen, G. (November 2021).

4656:

Prevalence of questionable research practices, research misconduct and their potential explanatory factors: a survey among academic researchers in The Netherlands

370: 325: 391:

Several global surveys have reported a growing uneasiness of scientific communities over reproducibility and other issues of research transparency. In 2016,

345: 4261:

Moher, David; Bouter, Lex; Kleinert, Sabine; Glasziou, Paul; Sham, Mai Har; Barbour, Virginia; Coriat, Anne-Marie; Foeger, Nicole; Dirnagl, Ulrich (2020).

544:

identifiers (or DOI) have been first allocated to dataset since the early 2000s which solved a long-standing debate on the citability of scientific data.

540:

identifiers (or DOI) have been first allocated to dataset since the early 2000s which solved a long-standing debate on the citability of scientific data.

4155:"The dawn of an open exploration era: Emergent principles and practices of open science and innovation of university research teams in a digital world" 312: 3130:

Wicherts, Jelte M.; Veldkamp, Coosje L. S.; Augusteijn, Hilde E. M.; Bakker, Marjan; van Aert, Robbie C. M.; van Assen, Marcel A. L. M. (2016).

3034:

Wicherts, Jelte M.; Veldkamp, Coosje L. S.; Augusteijn, Hilde E. M.; Bakker, Marjan; van Aert, Robbie C. M.; van Assen, Marcel A. L. M. (2016).

634: 2874: 271:

Due to the expansion of the published research output, new quantitative methods for literature surveys have been developed under the label of

5139: 5108: 5036: 4991: 2304: 2127: 2108: 2089: 2070: 2051: 2032: 2013: 1994: 1968: 5203: 135:

are largely rejected, the non-publication of negative results or the adjonctions of supplementary data are not identified as major issues.

82:

Core institutional actors continue to disagree on the meaning and usage of key concepts. In 2019, the National Academies of Science of the

377:(6,349 citations as of June 2022) shows how this framing has especially expanded to computing sciences. In Economics, a replication of 18 645:(in reference to the editorial choices made in the selection of the sources). In 2014, Andrew Moravcsik advocated the implementation of 87: 5171: 4145: 672:(7). In both cases, for the highest level of compliance journal should provide "link and badge in article to meeting requirements". 649:, containing detailed quotes of original sources as well as annotations "explaining how the source supports the claim being made". 560:

In the 2010s, several philosopher of sciences attempted to systematize value transparency in the context of open science. In 2017,

2287:

Brase, Jan (2004). "Using Digital Library Techniques – Registration of Scientific Primary Data". In Rachel Heery, Liz Lyon (ed.).

1936: 1960: 357: 324:

An early significant contribution to the debate has been the controversial and influential claim of John Ioannidis from 2005: "

97:

Debate over research transparency has also created new convergences between different disciplines and academic circles. In the

321:

until sometime around 2011–2012 that the scientific community’s consciousness was bombarded with irreproducibility warnings".

5264: 4848:

Elliott, Kevin (2022). "Open Science for Non-Specialists: Making Open Science Meaningful Outside the Scientific Community".

361:): while nearly all paper had reproducible effects, it was found that only 36% of the replications were significant enough ( 553: 5008: 4084:"The Platform for Responsible Editorial Policies: An initiative to foster editorial transparency in scholarly publishing" 3132:"Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking" 3036:"Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking" 3230:

Goodman, Steven N.; Fanelli, Daniele; Ioannidis, John P. A. (2016-06-01). "What does research reproducibility mean?".

719:

During review: criteria for selection, timing of the review and model of peer review (double bind, single bind, open).

294:

larger focus to the issue from researchers and policy-makers in the United States In a well-publicized investigation,

5274: 3324:"Research Transparency: A Preliminary Study of Disciplinary Conceptualisation, Drivers, Tools and Support Services" 5061: 725:

Post-publication: "criteria and procedures for corrections, expressions of concern, retraction" and other changes.

4037:"Transparency in Literature Syntheses and Editorial Review: Introducing the Methodological Guidance Paper Series" 716:

At submission: details on the governance of the journal, its scope, the editorial board or the rejection rates.

4653:

Gopalakrishna, Gowri; Riet, Gerben ter; Vink, Gerko; Stoop, Ineke; Wicherts, Jelte; Bouter, Lex (2021-07-06).

335:

the study of Ioannidis had a considerable echo in psychology, medicine and biology. In the following decades,

5238: 1235: 477: 336: 160:

movement "encompasses a number of different initiatives aimed at somewhat different forms of transparency."

5093:

Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

2408:"How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data" 5269: 4139: 3916: 2767: 2706: 3801: 472: 4732:"Should We Strive to Make Science Bias-Free? A Philosophical Assessment of the Reproducibility Crisis" 4435:

Malički, Mario; Jerončić, Ana; Aalbersberg, IJsbrand Jan; Bouter, Lex; ter Riet, Gerben (2021-10-05).

3460: 22:

is a major aspect of scientific research. It covers a variety of scientific principles and practices:

4800: 4684: 4605: 4507: 4448: 4383: 4209: 3845: 3678: 3519: 3417: 3100: 2535: 2419: 1890: 378: 4971: 4437:"Systematic review and meta-analyses of studies analysing instructions to authors from 1987 to 2017" 247:

were not reproduced by contemporary researchers: even a committed proponent of experimentalism like

286: 282: 118:

The taxonomy of scientific misconducts has been gradually expanded since the 1980s. The concept of

5240:

Reproducible analysis and Research Transparency — Reproducible Research workshop 1.0 documentation

2782: 175:

The concept of transparency has contributed to create convergences between open science and other

5219: 5042: 4945: 4898: 4554: 4417: 4351: 4310: 4243: 4184: 4121: 4064: 4023: 3966: 3814: 3757: 3488: 3441: 3304: 3263: 3212: 2969: 2749: 2688: 2610: 2504: 2496: 2388: 2269: 2173: 307: 63:

the same methodology. Conversely, computational disciplines use reversed definitions of the term

4671:

Devezer, Berna; Navarro, Danielle J.; Vandekerckhove, Joachim; Ozge Buzbas, Erkan (2021-03-31).

2820:

Head, Megan L.; Holman, Luke; Lanfear, Rob; Kahn, Andrew T.; Jennions, Michael D. (2015-03-13).

373:

in the main fields of research according to the metadata recorded by the academic search engine

46:, aims to build and strengthen open research culture across disciplines and epistemic cultures. 2981:

Ioannidis, John P. A.; Fanelli, Daniele; Dunne, Debbie Drake; Goodman, Steven N. (2015-10-02).

2660: 2583:"Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling" 2469:"The Significance of Re-Doing Experiments: A Contribution to Historically Informed Methodology" 5135: 5104: 5032: 4987: 4937: 4890: 4836: 4818: 4769: 4751: 4718: 4700: 4641: 4623: 4574: 4541: 4523: 4482: 4464: 4409: 4343: 4302: 4284: 4235: 4176: 4113: 4056: 4015: 4007: 3958: 3902: 3898: 3881: 3863: 3806: 3749: 3714: 3696: 3653: 3635: 3600: 3582: 3547: 3480: 3433: 3392: 3343: 3296: 3255: 3247: 3204: 3171: 3153: 3118: 3075: 3057: 3022: 3004: 2961: 2912: 2894: 2861: 2843: 2802: 2741: 2680: 2641: 2602: 2569: 2551: 2488: 2455: 2437: 2355: 2337: 2300: 2261: 2253: 2214: 2206: 2165: 2123: 2104: 2085: 2066: 2047: 2028: 2009: 1990: 1964: 1924: 1906: 152: 102: 3938: 421:

scientific institutions and the lack of incentives to implement them at an individual level.

5211: 5158: 5127: 5096: 5024: 5016: 4979: 4929: 4882: 4857: 4826: 4808: 4759: 4743: 4708: 4692: 4631: 4613: 4566: 4531: 4515: 4472: 4456: 4399: 4391: 4335: 4292: 4274: 4225: 4217: 4166: 4103: 4095: 4048: 3997: 3950: 3871: 3853: 3796: 3788: 3739: 3704: 3686: 3643: 3625: 3590: 3572: 3537: 3527: 3472: 3425: 3382: 3374: 3335: 3288: 3239: 3196: 3161: 3143: 3108: 3065: 3047: 3012: 2994: 2951: 2943: 2902: 2886: 2851: 2833: 2794: 2733: 2672: 2594: 2582: 2559: 2543: 2480: 2445: 2427: 2380: 2368: 2345: 2327: 2292: 2245: 2198: 2157: 1914: 1898: 374: 264: 94:, where a different team of research use exactly the same measurement system and procedure. 5149:

Pimentel, João Felipe; Murta, Leonardo; Braganholo, Vanessa; Freire, Juliana (2019-05-26).

4789:"Mapping the discursive dimensions of the reproducibility crisis: A mixed methods analysis" 4594:"Mapping the discursive dimensions of the reproducibility crisis: A mixed methods analysis" 2186: 3929: 3277:"Science's reproducibility and replicability crisis: International business is not immune" 3087:

Camerer CF, Dreber A, Forsell E, Ho TH, Huber J, Johannesson M, et al. (March 2016).

1779: 1777: 1775: 1773: 1771: 1769: 485: 481: 221:

as a call for "repeated (public) performances of experimental trials" A key member of the

184: 23: 610:

The sharing of research outputs is covered by three standards of the TOPs guidelines: on

90:

opted in 2016, for the computational definition and added also an intermediary notion of

4804: 4688: 4609: 4511: 4452: 4387: 4213: 4083: 3849: 3832:

Fraser, Hannah; Parker, Tim; Nakagawa, Shinichi; Barnett, Ashley; Fidler, Fiona (2018).

3682: 3665:

Fraser, Hannah; Parker, Tim; Nakagawa, Shinichi; Barnett, Ashley; Fidler, Fiona (2018).

3523: 3421: 3104: 2539: 2423: 1894: 708:

In July 2018, several publishers, librarians, journal editors and researchers drafted a

381:

in two major journals, found a failure rate comparable to psychology or medicine (39%).

4831: 4788: 4764: 4713: 4672: 4636: 4593: 4536: 4495: 4477: 4436: 4297: 4262: 3876: 3833: 3709: 3666: 3648: 3613: 3595: 3560: 3542: 3507: 3387: 3362: 3166: 3131: 3070: 3035: 3017: 2982: 2907: 2856: 2821: 2564: 2523: 2450: 2407: 2369:"Why P Values Are Not a Useful Measure of Evidence in Statistical Significance Testing" 2350: 2315: 1919: 1878: 604: 414: 226: 5186:

Pellizzari, Edo D.; Lohr, Kathleen N.; Blatecky, Alan; Creel, Darryl V. (2017-08-29).

5095:. ESEM '10. Bolzano-Bozen, Italy: Association for Computing Machinery. pp. 1–10. 4035:

Murphy, P. Karen; Dowd, Alicia C.; Lloyd, Gwendolyn M.; List, Alexandra (2020-02-01).

3506:

Redish, A. David; Kummerfeld, Erich; Morris, Rebecca Lea; Love, Alan C. (2018-05-15).

5258: 5223: 5020: 4949: 4902: 4421: 4371: 4355: 4314: 4247: 4197: 4188: 4125: 4068: 4027: 3970: 3761: 3492: 3216: 2973: 2931: 2753: 2508: 2392: 2291:. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer. pp. 488–494. 2202: 561: 443: 332: 273: 83: 5126:. CHI '18. Montreal QC, Canada: Association for Computing Machinery. pp. 1–12. 5046: 3818: 3776: 3445: 3308: 2722:"Openness in Political Science: Data Access and Research Transparency: Introduction" 2692: 2273: 2177: 4171: 4154: 3561:"Opinion: Is science really facing a reproducibility crisis, and do we need it to?" 2614: 384: 248: 148: 39: 4263:"The Hong Kong Principles for assessing researchers: Fostering research integrity" 3476: 3429: 3267: 4813: 4618: 4279: 3858: 3691: 3243: 2999: 2838: 2432: 2332: 2296: 5155:

Proceedings of the 16th International Conference on Mining Software Repositories

4570: 3405: 2161: 4933: 4747: 4460: 4395: 4221: 4002: 3985: 3461:"Open Science now: A systematic literature review for an integrated definition" 3184: 5215: 4983: 4886: 3792: 3614:"Reproducibility vs. Replicability: A Brief History of a Confused Terminology" 3292: 3189:

LIBER Quarterly: The Journal of the Association of European Research Libraries

2737: 2676: 2484: 2249: 191:

describe transparency as a common "rationale for open science and open data".

4972:"Stability and Replication of Experimental Results: A Historical Perspective" 4941: 4917: 4894: 4822: 4755: 4731: 4704: 4627: 4578: 4527: 4468: 4347: 4288: 4180: 4117: 4060: 4052: 4011: 3962: 3867: 3810: 3753: 3700: 3639: 3630: 3586: 3484: 3437: 3347: 3300: 3251: 3208: 3157: 3148: 3061: 3052: 3008: 2983:"Meta-research: Evaluation and Improvement of Research Methods and Practices" 2898: 2847: 2806: 2745: 2684: 2645: 2598: 2555: 2492: 2441: 2384: 2341: 2257: 2210: 2169: 1910: 476:

available to all researchers". This issue has worsened in some areas such as

147:

Transparency has been increasingly acknowledged as an important component of

59:

There is no widespread consensus on the definition of research transparency.

5151:"A large-scale study about quality and reproducibility of jupyter notebooks" 5131: 5124:

Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

5100: 4870: 4654: 4153:

Vicente-Saez, Ruben; Gustafsson, Robin; Van den Brande, Lieve (2020-07-01).

3727: 3577: 3532: 3378: 3276: 3113: 3088: 2947: 2890: 2547: 2233: 2101:

The Problem with Science: The Reproducibility Crisis and What to do About It

1902: 180: 5162: 4840: 4787:

Nelson, Nicole C.; Ichikawa, Kelsey; Chung, Julie; Malik, Momin M. (2021).

4773: 4722: 4645: 4592:

Nelson, Nicole C.; Ichikawa, Kelsey; Chung, Julie; Malik, Momin M. (2021).

4545: 4486: 4413: 4306: 4239: 4036: 4019: 3986:"A Necessary Complement to Transparent Peer Review: Editorial Transparency" 3894: 3885: 3718: 3657: 3604: 3551: 3396: 3339: 3323: 3259: 3175: 3122: 3079: 3026: 2965: 2916: 2865: 2606: 2573: 2459: 2359: 2265: 1928: 1783: 5119: 5088: 3185:"Transparency: the emerging third dimension of Open Science and Open Data" 2629: 2218: 5150: 210: 4861: 4696: 4339: 2956: 2500: 729:

In 2020, the Leiden Declaration has been expanded and supplemented by a

343:

attempted to reproduced 100 studies from three top psychology journals (

5204:"Dutch agency launches first grants programme dedicated to replication" 5028: 4404: 4230: 4108: 3954: 3744: 2468: 1819: 1592: 1590: 1588: 132: 4519: 4099: 3200: 763: 761: 759: 757: 409:

open science in the English-speaking scientific literature indexed on

5068:(Winter 2018 ed.). Metaphysics Research Lab, Stanford University 2932:"PSYCHOLOGY. Estimating the reproducibility of psychological science" 410: 5188:

Reproducibility: A Primer on Semantics and Implications for Research

4082:

Horbach, Serge P.J.M.; Hepkema, Wytske M.; Halffman, Willem (2020).

2145: 2025:

Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life

1342: 5011:. In Luca Fiorito; Scott Scheall; Carlos Eduardo Suprinyak (eds.). 3777:"The possibility and desirability of replication in the humanities" 2798: 2783:"The Evolution of Data Citation: From Principles to Implementation" 2581:

John, Leslie K.; Loewenstein, George; Prelec, Drazen (2012-04-16).

2120:

A Journey into Open Science and Research Transparency in Psychology

1756: 1754: 1752: 1078: 1076: 1074: 1072: 1070: 352:

Journal of Experimental Psychology: Learning, Memory, and Cognition

383: 3410:

Journal of Experimental & Theoretical Artificial Intelligence

3275:

Aguinis, Herman; Cascio, Wayne F.; Ramani, Ravi S. (2017-08-01).

3089:"Evaluating replicability of laboratory experiments in economics" 5009:"Rethinking Reproducibility as a Criterion for Research Quality" 4198:"Hundreds of journals' editorial practices captured in database" 4196:

Horbach, Serge; Hepkema, Wytske; Halffman, Willem (2020-06-02).

985: 983: 188: 4555:"Pre-registration: Weighing costs and benefits for researchers" 1649: 1647: 1645: 1643: 1641: 934: 932: 930: 928: 926: 641:(in regards to claims extrapolated from the cited sources) and 339:

attempted to assess experimental reproducibility. In 2015, the

5015:. Vol. 36. Emerald Publishing Limited. pp. 129–146. 3508:"Reproducibility failures are essential to scientific inquiry" 2150:

Transactions of the American Institute of Electrical Engineers

200:

Discourse and practices of research transparency (before 1945)

26:, data and code sharing, citation standards or verifiability. 5118:

Rule, Adam; Tabard, Aurélien; Hollan, James D. (2018-04-19).

4918:"Addressing the Reproducibility Crisis: A Response to Hudson" 1879:"Promoting an open research culture [TOP Guidelines]" 722:

Publication: disclosure of the "roles in the review process".

5087:

Gómez, Omar S.; Juristo, Natalia; Vegas, Sira (2010-09-16).

4871:"Understanding Replication in a Way That Is True to Science" 2875:"Scientific Reproducibility, Human Error, and Public Policy" 1739: 1737: 1330: 1318: 675:

Pre-registrations aims to preventively address a variety of

5013:

Research in the History of Economic Thought and Methodology

3459:

Vicente-Saez, Ruben; Martinez-Fuentes, Clara (2018-07-01).

4496:"Open Science in regulatory environmental risk assessment" 3834:"Questionable research practices in ecology and evolution" 3667:"Questionable research practices in ecology and evolution" 2063:

A Tapestry of Values: An Introduction to Values in Science

1461: 1459: 1227: 1214: 3728:"Open science and codes of conduct on research integrity" 127:

collection of the data, the analysis and the reporting).

5120:"Exploration and Explanation in Computational Notebooks" 316:

Results from The Reproducibility Project: Cancer Biology

5157:. MSR '19. Montreal, Quebec, Canada. pp. 507–517. 2187:"Statistical Evaluation of Medical Journal Manuscripts" 1842: 1830: 1795: 1596: 1503: 1501: 1450: 913: 911: 767: 4673:"The case for formal methodology in scientific reform" 2661:"Transparency: The Revolution in Qualitative Research" 2289:

Research and Advanced Technology for Digital Libraries

1700: 1698: 862: 860: 209:) in the 17th century. An early scientific society of 3322:

Lyon, Liz; Jeng, Wei; Mattern, Eleanor (2017-09-16).

2822:"The Extent and Consequences of P-Hacking in Science" 2630:"Tutkimusetiikka Suomessa 1980-luvulta tähän päivään" 839: 710:

Leiden Declaration for Transparent Editorial Policies

664:

Pre-registrations are covered by two TOP guidelines:

4559:

Organizational Behavior and Human Decision Processes

3939:"Philosophy of science and the replicability crisis" 1488: 1486: 1266: 1264: 890: 878: 259:

Preconditions of the transparency crisis (1945–2000)

2367:Hubbard, Raymond; Lindsay, R. Murray (2008-02-01). 302:

The reproducibility/transparency debate (2000–2015)

4916:Douglas, Heather; Elliott, Kevin C. (2022-05-11). 4500:Integrated Environmental Assessment and Management 3897:. Rochester, NY: Social Science Research Network. 2873:Elliott, Kevin C.; Resnik, David B. (2015-01-01). 1760: 1438: 1343:Vicente-Saez, Gustafsson & Van den Brande 2020 1282: 1082: 5089:"Replications types in experimental disciplines" 2524:"Reproducible research in computational science" 2316:"Why Most Published Research Findings Are False" 1963:(Report). National Academies Press. 2019-09-20. 1807: 1784:Radboud and Leiden transparency declaration 2019 1414: 4553:Logg, Jennifer M.; Dorison, Charles A. (2021). 3565:Proceedings of the National Academy of Sciences 3512:Proceedings of the National Academy of Sciences 2042:Elliott, Kevin C.; Steel, Daniel (2017-03-27). 1426: 989: 827: 582:Transparent by design: developing open workflow 79:(different setup but same overall principles). 4978:. John Wiley & Sons, Ltd. pp. 39–63. 2772:: CS1 maint: DOI inactive as of August 2024 ( 2711:: CS1 maint: DOI inactive as of August 2024 ( 2023:Shapin, Steven; Schaffer, Simon (2011-08-15). 1854: 1728: 1579: 1477: 1255: 1130: 1094: 509:in a computational context. The definition of 404:Research transparency and open science (2015–) 371:Why Most Published Research Findings Are False 2185:Schor, Stanley; Karten, Irving (1966-03-28). 1306: 851: 779: 527:Observation reproducibility and verifiability 8: 2628:Löppönen, Paavo; Vuorio, Eero (2013-02-21). 1961:Reproducibility and Replicability in Science 1653: 1620: 1366: 950: 938: 346:Journal of Personality and Social Psychology 75:(same experimental/computational setup) and 4159:Technological Forecasting and Social Change 3802:1871.1/1286b1ed-f663-4f24-886e-f91520856464 3406:"Reproducible research: a minority opinion" 2044:Current Controversies in Values and Science 1938:Radboud and Leiden transparency declaration 1743: 1118: 731:Platform for Responsible Editorial Policies 4372:"Unlock ways to share data on peer review" 2930:Open Science Collaboration (August 2015). 1985:Broad, William J.; Wade, Nicholas (1983). 1665: 326:most published research findings are false 4922:Journal for General Philosophy of Science 4830: 4812: 4763: 4736:Journal for General Philosophy of Science 4712: 4635: 4617: 4535: 4476: 4403: 4296: 4278: 4229: 4170: 4107: 4001: 3875: 3857: 3800: 3743: 3708: 3690: 3647: 3629: 3594: 3576: 3541: 3531: 3386: 3328:International Journal of Digital Curation 3281:Journal of International Business Studies 3165: 3147: 3112: 3069: 3051: 3016: 2998: 2955: 2906: 2855: 2837: 2563: 2449: 2431: 2349: 2331: 2006:Science, Policy, and the Value-Free Ideal 1918: 1689: 1677: 1632: 1519: 1228:"Reproducibility Project: Cancer Biology" 1202: 1190: 1142: 1037: 1025: 1013: 263:The development of big science after the 1465: 1331:Vicente-Saez & Martinez-Fuentes 2018 1319:Vicente-Saez & Martinez-Fuentes 2018 1166: 311: 5066:The Stanford Encyclopedia of Philosophy 5062:"Reproducibility of Scientific Results" 4324:"A Taxonomy of Transparency in Science" 1608: 1567: 1555: 1543: 1531: 1390: 1378: 1294: 1178: 1106: 1049: 1001: 974: 962: 917: 866: 803: 791: 748: 741: 4137: 3925: 3914: 3775:Peels, Rik; Bouter, Lex (2018-08-07). 3363:"A manifesto for reproducible science" 2765: 2704: 1716: 1704: 1507: 815: 635:American Political Science Association 455:Goodman, Fanelli and Ioannidis define 1843:Horbach, Hepkema & Halffman 2020b 1831:Horbach, Hepkema & Halffman 2020a 1796:Horbach, Hepkema & Halffman 2020a 1597:Horbach, Hepkema & Halffman 2020a 1492: 1451:Goodman, Fanelli & Ioannidis 2016 1354: 1270: 1154: 768:Goodman, Fanelli & Ioannidis 2016 179:movements in different areas such as 7: 5060:Fidler, Fiona; Wilcox, John (2018). 2726:PS: Political Science & Politics 2720:Lupia, Arthur; Elman, Colin (2014). 2665:PS: Political Science & Politics 1402: 1061: 902: 2061:Elliott, Kevin Christopher (2017). 840:John, Loewenstein & Prelec 2012 341:Reproducibility Project: Psychology 290:for these conclusions was invalid" 88:Association for Computing Machinery 616:Analytic/code methods transparency 519:transparency in the recent years. 14: 3895:"p-Hacking: A Strategic Analysis" 2232:Pimple, Kenneth D. (2002-06-01). 2099:Bausell, R. Barker (2021-01-26). 2080:Pimple, Kenneth D. (2017-05-15). 670:Preregistration of analysis plans 569:Review and editorial transparency 5021:10.1108/S0743-41542018000036B009 4322:Elliott, Kevin C. (2020-06-16). 2234:"Six domains of research ethics" 2203:10.1001/jama.1966.03100130097026 2008:. University of Pittsburgh Pre. 531:In 2018 Sabina Leonelli defines 143:A new dimension of open science? 5007:Leonelli, Sabina (2018-10-24). 3612:Plesser, Hans E. (2018-01-18). 3559:Fanelli, Daniele (2018-03-13). 2004:Douglas, Heather (2009-07-15). 1215:Open Science Collaboration 2015 677:questionable research practices 620:Research materials transparency 505:in an experimental context and 120:questionable research practices 32:questionable research practices 4328:Canadian Journal of Philosophy 4172:10.1016/j.techfore.2020.120037 4041:Review of Educational Research 3893:MacCoun, Robert (2019-08-06). 3404:Drummond, Chris (2018-01-02). 3232:Science Translational Medicine 2314:Ioannidis, John P. A. (2005). 2238:Science and Engineering Ethics 2027:. Princeton University Press. 1415:Rule, Tabard & Hollan 2018 591:(such as research materials). 463:in a computational context or 437:Forms of research transparency 1: 5172:Making Open Science a Reality 5064:. In Zalta, Edward N. (ed.). 4869:Haig, Brian D. (2022-06-01). 4730:Hudson, Robert (2021-09-01). 4144:: CS1 maint: date and year ( 3618:Frontiers in Neuroinformatics 3477:10.1016/j.jbusres.2017.12.043 3430:10.1080/0952813X.2017.1413140 2522:Peng, Roger D. (2011-12-02). 990:Lyon, Jeng & Mattern 2017 4875:Review of General Psychology 4814:10.1371/journal.pone.0254090 4619:10.1371/journal.pone.0254090 4280:10.1371/journal.pbio.3000737 3859:10.1371/journal.pone.0200303 3692:10.1371/journal.pone.0200303 3465:Journal of Business Research 3244:10.1126/scitranslmed.aaf5027 3000:10.1371/journal.pbio.1002264 2839:10.1371/journal.pbio.1002106 2781:Crosas, Mercè (2014-05-26). 2433:10.1371/journal.pone.0005738 2333:10.1371/journal.pmed.0020124 2297:10.1007/978-3-540-30230-8_44 2144:Bell, Louis (January 1902). 251:had to devolve to a form of 4970:Steinle, Friedrich (2016). 4571:10.1016/j.obhdp.2021.05.006 3726:Laine, Heidi (2018-12-31). 2162:10.1109/T-AIEE.1902.4763952 2103:. Oxford University Press. 2065:. Oxford University Press. 626:Extended citation standards 599:Sharing of research outputs 533:observation reproducibility 163:First drafted in 2014, the 5291: 4934:10.1007/s10838-022-09606-5 4748:10.1007/s10838-020-09548-w 4677:Royal Society Open Science 4461:10.1038/s41467-021-26027-y 4396:10.1038/d41586-020-00500-y 4222:10.1038/d41586-020-01628-7 4003:10.1016/j.cels.2019.07.002 2659:Moravcsik, Andrew (2014). 1870:Standards and declarations 1131:Löppönen & Vuorio 2013 1095:Shapin & Schaffer 2011 666:Preregistration of studies 331:Due to being published in 305: 217:adopted in 1657 the motto 5216:10.1038/nature.2016.20287 4984:10.1002/9781118865064.ch3 4887:10.1177/10892680211046514 3984:Justman, Quincey (2019). 3793:10.1057/s41599-018-0149-x 3293:10.1057/s41267-017-0081-0 2738:10.1017/S1049096513001716 2677:10.1017/S1049096513001789 2485:10.1007/s10670-011-9332-9 2467:Schickore, Jutta (2011). 2406:Fanelli, Daniele (2009). 2250:10.1007/s11948-002-0018-1 2146:"Methods of Illumination" 2118:Grahe, Jon (2021-08-30). 1941:. Leiden University. 2019 891:Gopalakrishna et al. 2021 879:Gopalakrishna et al. 2021 4053:10.3102/0034654319901128 3631:10.3389/fninf.2017.00076 3183:Lyon, Liz (2016-06-23). 3149:10.3389/fpsyg.2016.01832 3053:10.3389/fpsyg.2016.01832 2599:10.1177/0956797611430953 2385:10.1177/0959354307086923 2046:. Taylor & Francis. 400:and the popular press". 5132:10.1145/3173574.3173606 5101:10.1145/1852786.1852790 3937:Romero, Felipe (2019). 3781:Palgrave Communications 3578:10.1073/pnas.1708272114 3533:10.1073/pnas.1806370115 3379:10.1038/s41562-016-0021 3136:Frontiers in Psychology 3114:10.1126/science.aaf0918 3040:Frontiers in Psychology 2948:10.1126/science.aac4716 2740:(inactive 2024-08-05). 2679:(inactive 2024-08-05). 2548:10.1126/science.1213847 2373:Theory & Psychology 1903:10.1126/science.aab2374 1820:ResponsibleJournals.org 1761:National Academies 2019 1744:Logg & Dorison 2021 1439:National Academies 2019 1283:National Academies 2019 1236:Center for Open Science 1119:Schor & Karten 1966 1083:National Academies 2019 704:Open editorial policies 643:production transparency 478:Artificial Intelligence 253:virtual experimentalism 5237:Psomopoulos, Fotis E. 5163:10.1109/MSR.2019.00077 3924:Cite journal requires 3367:Nature Human Behaviour 3340:10.2218/ijdc.v12i1.530 1989:. Simon and Schuster. 1987:Betrayers of the Truth 1666:Lupia & Elman 2014 516:reproducibility crisis 496:Result reproducibility 457:method reproducibility 451:Method reproducibility 388: 317: 77:result reproducibility 73:method reproducibility 55:Confused terminologies 5265:Ethics and statistics 5202:Baker, Monya (2016). 4850:Philosophy of Science 4441:Nature Communications 2891:10.1093/biosci/biu197 2587:Psychological Science 1808:Squazzoni et al. 2020 1143:Broad & Wade 1983 647:transparency appendix 639:analytic transparency 387: 358:Psychological Science 315: 306:Further information: 219:provando e riprovando 215:Accademia del Cimento 20:Research transparency 1427:Pimentel et al. 2019 953:, p. 1422-1423. 828:Wicherts et al. 2016 379:experimental studies 337:large range projects 296:Betrayers of Science 110:Negative definitions 103:Rufus Barker Bausell 5170:OECD (2015-10-15). 4862:10.1017/psa.2022.36 4805:2021PLoSO..1654090N 4697:10.1098/rsos.200805 4689:2021RSOS....800805D 4610:2021PLoSO..1654090N 4512:2021IEAM...17.1229B 4453:2021NatCo..12.5840M 4388:2020Natur.578..512S 4340:10.1017/can.2020.21 4214:2020Natur.582...32H 3850:2018PLoSO..1300303F 3732:Informaatiotutkimus 3683:2018PLoSO..1300303F 3524:2018PNAS..115.5042R 3422:2018JETAI..30....1D 3105:2016Sci...351.1433C 3099:(6280): 1433–1436. 2540:2011Sci...334.1226P 2534:(6060): 1226–1227. 2424:2009PLoSO...4.5738F 1895:2015Sci...348.1422N 1889:(6242): 1422–1425. 1855:Malički et al. 2021 1729:Devezer et al. 2021 1580:Malički et al. 2021 1478:Devezer et al. 2021 1256:Camerer et al. 2016 691:Replication studies 16:Scientific practice 4088:Learned Publishing 3955:10.1111/phc3.12633 3943:Philosophy Compass 3745:10.23978/inf.77414 2634:Tieteessä tapahtuu 2473:Erkenntnis (1975-) 1307:Nelson et al. 2021 852:Fraser et al. 2018 780:Nelson et al. 2021 548:Value transparency 486:library dependency 389: 318: 308:Replication crisis 207:natural philosophy 99:Problem of science 5275:Scientific method 5141:978-1-4503-5620-6 5110:978-1-4503-0039-1 5038:978-1-78756-424-4 4993:978-1-118-86506-4 4520:10.1002/ieam.4433 4382:(7796): 512–514. 4100:10.1002/leap.1312 3571:(11): 2628–2631. 3518:(20): 5042–5046. 3201:10.18352/lq.10113 2942:(6251): aac4716. 2787:IASSIST Quarterly 2306:978-3-540-30230-8 2197:(13): 1123–1128. 2138:Academic articles 2129:978-1-00-043049-3 2110:978-0-19-753654-4 2091:978-1-351-90400-1 2072:978-0-19-026081-1 2053:978-1-317-27399-8 2034:978-1-4008-3849-3 2015:978-0-8229-7357-7 1996:978-0-671-44769-4 1970:978-0-309-48619-4 1654:Nosek et al. 2015 1621:Nosek et al. 2015 1546:, p. 87 sq.. 1367:Moher et al. 2020 1097:, p. 60 sq.. 951:Nosek et al. 2015 939:Nosek et al. 2015 660:Pre-registrations 652:According to the 612:Data transparency 473:research notebook 225:, the naturalist 5282: 5250: 5248: 5247: 5233: 5231: 5230: 5191: 5182: 5180: 5179: 5166: 5145: 5114: 5076: 5074: 5073: 5056: 5054: 5053: 5003: 5001: 5000: 4959: 4957: 4956: 4912: 4910: 4909: 4865: 4844: 4834: 4816: 4783: 4781: 4780: 4767: 4726: 4716: 4667: 4665: 4664: 4649: 4639: 4621: 4588: 4586: 4585: 4549: 4539: 4506:(6): 1229–1242. 4490: 4480: 4431: 4429: 4428: 4407: 4365: 4363: 4362: 4318: 4300: 4282: 4257: 4255: 4254: 4233: 4192: 4174: 4149: 4143: 4135: 4133: 4132: 4111: 4078: 4076: 4075: 4031: 4005: 3980: 3978: 3977: 3933: 3927: 3922: 3920: 3912: 3910: 3909: 3889: 3879: 3861: 3828: 3826: 3825: 3804: 3771: 3769: 3768: 3747: 3722: 3712: 3694: 3661: 3651: 3633: 3608: 3598: 3580: 3555: 3545: 3535: 3502: 3500: 3499: 3455: 3453: 3452: 3400: 3390: 3357: 3355: 3354: 3318: 3316: 3315: 3271: 3226: 3224: 3223: 3179: 3169: 3151: 3126: 3116: 3083: 3073: 3055: 3030: 3020: 3002: 2993:(10): e1002264. 2977: 2959: 2926: 2924: 2923: 2910: 2869: 2859: 2841: 2816: 2814: 2813: 2777: 2771: 2763: 2761: 2760: 2716: 2710: 2702: 2700: 2699: 2655: 2653: 2652: 2624: 2622: 2621: 2577: 2567: 2518: 2516: 2515: 2463: 2453: 2435: 2402: 2400: 2399: 2363: 2353: 2335: 2310: 2283: 2281: 2280: 2228: 2226: 2225: 2181: 2133: 2114: 2095: 2076: 2057: 2038: 2019: 2000: 1979:Books and theses 1974: 1949: 1947: 1946: 1932: 1922: 1858: 1852: 1846: 1840: 1834: 1828: 1822: 1817: 1811: 1805: 1799: 1793: 1787: 1781: 1764: 1758: 1747: 1741: 1732: 1726: 1720: 1714: 1708: 1702: 1693: 1687: 1681: 1680:, p. 48-49. 1675: 1669: 1663: 1657: 1651: 1636: 1630: 1624: 1618: 1612: 1606: 1600: 1594: 1583: 1577: 1571: 1565: 1559: 1553: 1547: 1541: 1535: 1529: 1523: 1517: 1511: 1505: 1496: 1490: 1481: 1475: 1469: 1463: 1454: 1448: 1442: 1436: 1430: 1424: 1418: 1412: 1406: 1400: 1394: 1388: 1382: 1376: 1370: 1364: 1358: 1352: 1346: 1340: 1334: 1328: 1322: 1316: 1310: 1304: 1298: 1292: 1286: 1280: 1274: 1268: 1259: 1253: 1247: 1246: 1244: 1242: 1224: 1218: 1212: 1206: 1200: 1194: 1188: 1182: 1176: 1170: 1164: 1158: 1152: 1146: 1140: 1134: 1128: 1122: 1116: 1110: 1104: 1098: 1092: 1086: 1080: 1065: 1059: 1053: 1047: 1041: 1035: 1029: 1023: 1017: 1011: 1005: 999: 993: 987: 978: 972: 966: 960: 954: 948: 942: 936: 921: 915: 906: 900: 894: 888: 882: 876: 870: 864: 855: 849: 843: 837: 831: 825: 819: 813: 807: 801: 795: 789: 783: 777: 771: 765: 752: 746: 375:Semantic Scholar 265:Second World War 5290: 5289: 5285: 5284: 5283: 5281: 5280: 5279: 5255: 5254: 5253: 5245: 5243: 5236: 5228: 5226: 5201: 5198: 5185: 5177: 5175: 5169: 5148: 5142: 5117: 5111: 5086: 5083: 5071: 5069: 5059: 5051: 5049: 5039: 5006: 4998: 4996: 4994: 4976:Reproducibility 4969: 4966: 4954: 4952: 4915: 4907: 4905: 4868: 4847: 4799:(7): –0254090. 4786: 4778: 4776: 4729: 4670: 4662: 4660: 4652: 4604:(7): –0254090. 4591: 4583: 4581: 4552: 4493: 4434: 4426: 4424: 4368: 4360: 4358: 4321: 4273:(7): –3000737. 4260: 4252: 4250: 4195: 4152: 4136: 4130: 4128: 4081: 4073: 4071: 4034: 3983: 3975: 3973: 3936: 3923: 3913: 3907: 3905: 3892: 3844:(7): –0200303. 3831: 3823: 3821: 3774: 3766: 3764: 3725: 3677:(7): –0200303. 3664: 3611: 3558: 3505: 3497: 3495: 3458: 3450: 3448: 3403: 3360: 3352: 3350: 3321: 3313: 3311: 3274: 3238:(341): 341–12. 3229: 3221: 3219: 3182: 3129: 3086: 3033: 2980: 2929: 2921: 2919: 2872: 2832:(3): –1002106. 2819: 2811: 2809: 2780: 2764: 2758: 2756: 2719: 2703: 2697: 2695: 2658: 2650: 2648: 2627: 2619: 2617: 2580: 2521: 2513: 2511: 2466: 2405: 2397: 2395: 2366: 2313: 2307: 2286: 2278: 2276: 2231: 2223: 2221: 2184: 2143: 2140: 2130: 2117: 2111: 2098: 2092: 2082:Research Ethics 2079: 2073: 2060: 2054: 2041: 2035: 2022: 2016: 2003: 1997: 1984: 1981: 1971: 1959: 1956: 1944: 1942: 1935: 1875: 1872: 1866: 1861: 1853: 1849: 1841: 1837: 1829: 1825: 1818: 1814: 1806: 1802: 1794: 1790: 1782: 1767: 1759: 1750: 1742: 1735: 1727: 1723: 1715: 1711: 1703: 1696: 1688: 1684: 1676: 1672: 1664: 1660: 1656:, p. 1424. 1652: 1639: 1631: 1627: 1619: 1615: 1607: 1603: 1595: 1586: 1578: 1574: 1566: 1562: 1554: 1550: 1542: 1538: 1530: 1526: 1518: 1514: 1506: 1499: 1491: 1484: 1476: 1472: 1464: 1457: 1449: 1445: 1437: 1433: 1425: 1421: 1413: 1409: 1401: 1397: 1389: 1385: 1377: 1373: 1365: 1361: 1353: 1349: 1341: 1337: 1329: 1325: 1317: 1313: 1305: 1301: 1293: 1289: 1281: 1277: 1269: 1262: 1254: 1250: 1240: 1238: 1226: 1225: 1221: 1213: 1209: 1201: 1197: 1189: 1185: 1177: 1173: 1165: 1161: 1153: 1149: 1141: 1137: 1129: 1125: 1117: 1113: 1105: 1101: 1093: 1089: 1081: 1068: 1060: 1056: 1048: 1044: 1036: 1032: 1024: 1020: 1012: 1008: 1000: 996: 988: 981: 973: 969: 961: 957: 949: 945: 941:, p. 1423. 937: 924: 916: 909: 901: 897: 889: 885: 877: 873: 865: 858: 850: 846: 838: 834: 826: 822: 814: 810: 802: 798: 790: 786: 778: 774: 766: 755: 747: 743: 739: 706: 693: 662: 628: 601: 588:TOPs Guidelines 584: 571: 554:Heather Douglas 550: 529: 507:reproducibility 498: 482:Computer vision 465:reproducibility 453: 439: 406: 310: 304: 261: 240:reproducibility 202: 197: 187:. In 2015, the 185:open government 145: 112: 69:reproducibility 57: 52: 24:reproducibility 17: 12: 11: 5: 5288: 5286: 5278: 5277: 5272: 5267: 5257: 5256: 5252: 5251: 5234: 5197: 5194: 5193: 5192: 5183: 5167: 5146: 5140: 5115: 5109: 5082: 5079: 5078: 5077: 5057: 5037: 5004: 4992: 4965: 4962: 4961: 4960: 4928:(2): 201–209. 4913: 4881:(2): 224–240. 4866: 4845: 4784: 4742:(3): 389–405. 4727: 4668: 4650: 4589: 4550: 4491: 4432: 4366: 4334:(3): 342–355. 4319: 4258: 4193: 4150: 4094:(3): 340–344. 4079: 4032: 3981: 3949:(11): –12633. 3934: 3926:|journal= 3890: 3829: 3772: 3723: 3662: 3609: 3556: 3503: 3456: 3401: 3358: 3319: 3287:(6): 653–663. 3272: 3227: 3195:(4): 153–171. 3180: 3127: 3084: 3031: 2978: 2927: 2870: 2817: 2799:10.29173/iq504 2778: 2717: 2656: 2625: 2593:(5): 524–532. 2578: 2519: 2479:(3): 325–347. 2464: 2403: 2364: 2311: 2305: 2284: 2244:(2): 191–205. 2229: 2182: 2139: 2136: 2135: 2134: 2128: 2115: 2109: 2096: 2090: 2077: 2071: 2058: 2052: 2039: 2033: 2020: 2014: 2001: 1995: 1980: 1977: 1976: 1975: 1969: 1955: 1952: 1951: 1950: 1933: 1871: 1868: 1867: 1865: 1862: 1860: 1859: 1847: 1835: 1823: 1812: 1800: 1788: 1765: 1748: 1733: 1721: 1709: 1694: 1690:Moravcsik 2014 1682: 1678:Moravcsik 2014 1670: 1658: 1637: 1635:, p. 701. 1633:Ioannidis 2005 1625: 1613: 1601: 1584: 1572: 1560: 1558:, p. 176. 1548: 1536: 1534:, p. 175. 1524: 1520:Moravcsik 2014 1512: 1497: 1482: 1470: 1468:, p. 137. 1455: 1443: 1431: 1419: 1407: 1395: 1393:, p. 8-9. 1383: 1371: 1359: 1347: 1335: 1323: 1311: 1309:, p. 1-2. 1299: 1297:, p. 128. 1287: 1275: 1260: 1248: 1219: 1207: 1205:, p. 700. 1203:Ioannidis 2005 1195: 1191:Ioannidis 2005 1183: 1171: 1159: 1147: 1135: 1123: 1121:, p. 148. 1111: 1099: 1087: 1066: 1054: 1042: 1040:, p. 332. 1038:Schickore 2011 1030: 1028:, p. 330. 1026:Schickore 2011 1018: 1014:Schickore 2011 1006: 994: 979: 967: 955: 943: 922: 907: 905:, p. 160. 895: 883: 871: 856: 844: 832: 820: 818:, p. 202. 808: 796: 784: 772: 753: 751:, p. 1-2. 740: 738: 735: 727: 726: 723: 720: 717: 705: 702: 692: 689: 661: 658: 654:TOP Guidelines 627: 624: 605:John Ioannidis 600: 597: 583: 580: 570: 567: 549: 546: 528: 525: 497: 494: 452: 449: 438: 435: 426:TOP guidelines 415:Web of Science 405: 402: 303: 300: 260: 257: 227:Francesco Redi 201: 198: 196: 193: 165:TOP guidelines 144: 141: 111: 108: 56: 53: 51: 48: 44:TOP Guidelines 15: 13: 10: 9: 6: 4: 3: 2: 5287: 5276: 5273: 5271: 5268: 5266: 5263: 5262: 5260: 5242: 5241: 5235: 5225: 5221: 5217: 5213: 5209: 5205: 5200: 5199: 5196:Other sources 5195: 5189: 5184: 5173: 5168: 5164: 5160: 5156: 5152: 5147: 5143: 5137: 5133: 5129: 5125: 5121: 5116: 5112: 5106: 5102: 5098: 5094: 5090: 5085: 5084: 5080: 5067: 5063: 5058: 5048: 5044: 5040: 5034: 5030: 5026: 5022: 5018: 5014: 5010: 5005: 4995: 4989: 4985: 4981: 4977: 4973: 4968: 4967: 4963: 4951: 4947: 4943: 4939: 4935: 4931: 4927: 4923: 4919: 4914: 4904: 4900: 4896: 4892: 4888: 4884: 4880: 4876: 4872: 4867: 4863: 4859: 4855: 4851: 4846: 4842: 4838: 4833: 4828: 4824: 4820: 4815: 4810: 4806: 4802: 4798: 4794: 4790: 4785: 4775: 4771: 4766: 4761: 4757: 4753: 4749: 4745: 4741: 4737: 4733: 4728: 4724: 4720: 4715: 4710: 4706: 4702: 4698: 4694: 4690: 4686: 4683:(3): 200805. 4682: 4678: 4674: 4669: 4658: 4657: 4651: 4647: 4643: 4638: 4633: 4629: 4625: 4620: 4615: 4611: 4607: 4603: 4599: 4595: 4590: 4580: 4576: 4572: 4568: 4564: 4560: 4556: 4551: 4547: 4543: 4538: 4533: 4529: 4525: 4521: 4517: 4513: 4509: 4505: 4501: 4497: 4492: 4488: 4484: 4479: 4474: 4470: 4466: 4462: 4458: 4454: 4450: 4446: 4442: 4438: 4433: 4423: 4419: 4415: 4411: 4406: 4401: 4397: 4393: 4389: 4385: 4381: 4377: 4373: 4367: 4357: 4353: 4349: 4345: 4341: 4337: 4333: 4329: 4325: 4320: 4316: 4312: 4308: 4304: 4299: 4294: 4290: 4286: 4281: 4276: 4272: 4268: 4264: 4259: 4249: 4245: 4241: 4237: 4232: 4227: 4223: 4219: 4215: 4211: 4207: 4203: 4199: 4194: 4190: 4186: 4182: 4178: 4173: 4168: 4164: 4160: 4156: 4151: 4147: 4141: 4127: 4123: 4119: 4115: 4110: 4105: 4101: 4097: 4093: 4089: 4085: 4080: 4070: 4066: 4062: 4058: 4054: 4050: 4046: 4042: 4038: 4033: 4029: 4025: 4021: 4017: 4013: 4009: 4004: 3999: 3995: 3991: 3987: 3982: 3972: 3968: 3964: 3960: 3956: 3952: 3948: 3944: 3940: 3935: 3931: 3918: 3904: 3900: 3896: 3891: 3887: 3883: 3878: 3873: 3869: 3865: 3860: 3855: 3851: 3847: 3843: 3839: 3835: 3830: 3820: 3816: 3812: 3808: 3803: 3798: 3794: 3790: 3786: 3782: 3778: 3773: 3763: 3759: 3755: 3751: 3746: 3741: 3737: 3733: 3729: 3724: 3720: 3716: 3711: 3706: 3702: 3698: 3693: 3688: 3684: 3680: 3676: 3672: 3668: 3663: 3659: 3655: 3650: 3645: 3641: 3637: 3632: 3627: 3623: 3619: 3615: 3610: 3606: 3602: 3597: 3592: 3588: 3584: 3579: 3574: 3570: 3566: 3562: 3557: 3553: 3549: 3544: 3539: 3534: 3529: 3525: 3521: 3517: 3513: 3509: 3504: 3494: 3490: 3486: 3482: 3478: 3474: 3470: 3466: 3462: 3457: 3447: 3443: 3439: 3435: 3431: 3427: 3423: 3419: 3415: 3411: 3407: 3402: 3398: 3394: 3389: 3384: 3380: 3376: 3372: 3368: 3364: 3359: 3349: 3345: 3341: 3337: 3333: 3329: 3325: 3320: 3310: 3306: 3302: 3298: 3294: 3290: 3286: 3282: 3278: 3273: 3269: 3265: 3261: 3257: 3253: 3249: 3245: 3241: 3237: 3233: 3228: 3218: 3214: 3210: 3206: 3202: 3198: 3194: 3190: 3186: 3181: 3177: 3173: 3168: 3163: 3159: 3155: 3150: 3145: 3141: 3137: 3133: 3128: 3124: 3120: 3115: 3110: 3106: 3102: 3098: 3094: 3090: 3085: 3081: 3077: 3072: 3067: 3063: 3059: 3054: 3049: 3045: 3041: 3037: 3032: 3028: 3024: 3019: 3014: 3010: 3006: 3001: 2996: 2992: 2988: 2984: 2979: 2975: 2971: 2967: 2963: 2958: 2953: 2949: 2945: 2941: 2937: 2933: 2928: 2918: 2914: 2909: 2904: 2900: 2896: 2892: 2888: 2884: 2880: 2876: 2871: 2867: 2863: 2858: 2853: 2849: 2845: 2840: 2835: 2831: 2827: 2823: 2818: 2808: 2804: 2800: 2796: 2792: 2788: 2784: 2779: 2775: 2769: 2755: 2751: 2747: 2743: 2739: 2735: 2731: 2727: 2723: 2718: 2714: 2708: 2694: 2690: 2686: 2682: 2678: 2674: 2670: 2666: 2662: 2657: 2647: 2643: 2639: 2635: 2631: 2626: 2616: 2612: 2608: 2604: 2600: 2596: 2592: 2588: 2584: 2579: 2575: 2571: 2566: 2561: 2557: 2553: 2549: 2545: 2541: 2537: 2533: 2529: 2525: 2520: 2510: 2506: 2502: 2498: 2494: 2490: 2486: 2482: 2478: 2474: 2470: 2465: 2461: 2457: 2452: 2447: 2443: 2439: 2434: 2429: 2425: 2421: 2417: 2413: 2409: 2404: 2394: 2390: 2386: 2382: 2378: 2374: 2370: 2365: 2361: 2357: 2352: 2347: 2343: 2339: 2334: 2329: 2325: 2321: 2320:PLOS Medicine 2317: 2312: 2308: 2302: 2298: 2294: 2290: 2285: 2275: 2271: 2267: 2263: 2259: 2255: 2251: 2247: 2243: 2239: 2235: 2230: 2220: 2216: 2212: 2208: 2204: 2200: 2196: 2192: 2188: 2183: 2179: 2175: 2171: 2167: 2163: 2159: 2155: 2151: 2147: 2142: 2141: 2137: 2131: 2125: 2122:. Routledge. 2121: 2116: 2112: 2106: 2102: 2097: 2093: 2087: 2084:. Routledge. 2083: 2078: 2074: 2068: 2064: 2059: 2055: 2049: 2045: 2040: 2036: 2030: 2026: 2021: 2017: 2011: 2007: 2002: 1998: 1992: 1988: 1983: 1982: 1978: 1972: 1966: 1962: 1958: 1957: 1953: 1940: 1939: 1934: 1930: 1926: 1921: 1916: 1912: 1908: 1904: 1900: 1896: 1892: 1888: 1884: 1880: 1874: 1873: 1869: 1863: 1856: 1851: 1848: 1844: 1839: 1836: 1832: 1827: 1824: 1821: 1816: 1813: 1809: 1804: 1801: 1797: 1792: 1789: 1785: 1780: 1778: 1776: 1774: 1772: 1770: 1766: 1763:, p. 77. 1762: 1757: 1755: 1753: 1749: 1746:, p. 26. 1745: 1740: 1738: 1734: 1731:, p. 16. 1730: 1725: 1722: 1718: 1713: 1710: 1706: 1701: 1699: 1695: 1692:, p. 50. 1691: 1686: 1683: 1679: 1674: 1671: 1667: 1662: 1659: 1655: 1650: 1648: 1646: 1644: 1642: 1638: 1634: 1629: 1626: 1622: 1617: 1614: 1610: 1605: 1602: 1598: 1593: 1591: 1589: 1585: 1581: 1576: 1573: 1569: 1564: 1561: 1557: 1552: 1549: 1545: 1540: 1537: 1533: 1528: 1525: 1522:, p. 48. 1521: 1516: 1513: 1510:, p. 63. 1509: 1504: 1502: 1498: 1494: 1489: 1487: 1483: 1479: 1474: 1471: 1467: 1466:Leonelli 2018 1462: 1460: 1456: 1452: 1447: 1444: 1441:, p. 51. 1440: 1435: 1432: 1428: 1423: 1420: 1416: 1411: 1408: 1404: 1399: 1396: 1392: 1387: 1384: 1380: 1375: 1372: 1368: 1363: 1360: 1357:, p. 65. 1356: 1351: 1348: 1344: 1339: 1336: 1332: 1327: 1324: 1320: 1315: 1312: 1308: 1303: 1300: 1296: 1291: 1288: 1285:, p. 83. 1284: 1279: 1276: 1272: 1267: 1265: 1261: 1257: 1252: 1249: 1237: 1233: 1229: 1223: 1220: 1216: 1211: 1208: 1204: 1199: 1196: 1192: 1187: 1184: 1181:, p. 10. 1180: 1175: 1172: 1168: 1167:Drummond 2018 1163: 1160: 1157:, p. 49. 1156: 1151: 1148: 1144: 1139: 1136: 1132: 1127: 1124: 1120: 1115: 1112: 1109:, p. 56. 1108: 1103: 1100: 1096: 1091: 1088: 1085:, p. 46. 1084: 1079: 1077: 1075: 1073: 1071: 1067: 1064:, p. 15. 1063: 1058: 1055: 1052:, p. 45. 1051: 1046: 1043: 1039: 1034: 1031: 1027: 1022: 1019: 1015: 1010: 1007: 1004:, p. 44. 1003: 998: 995: 992:, p. 47. 991: 986: 984: 980: 976: 971: 968: 964: 959: 956: 952: 947: 944: 940: 935: 933: 931: 929: 927: 923: 919: 914: 912: 908: 904: 899: 896: 892: 887: 884: 880: 875: 872: 868: 863: 861: 857: 853: 848: 845: 841: 836: 833: 829: 824: 821: 817: 812: 809: 805: 800: 797: 793: 788: 785: 782:, p. 45. 781: 776: 773: 769: 764: 762: 760: 758: 754: 750: 745: 742: 736: 734: 732: 724: 721: 718: 715: 714: 713: 711: 703: 701: 697: 690: 688: 684: 680: 678: 673: 671: 667: 659: 657: 655: 650: 648: 644: 640: 636: 633:In 2012, the 631: 625: 623: 621: 617: 613: 608: 606: 598: 596: 592: 589: 581: 579: 575: 568: 566: 563: 562:Kevin Elliott 558: 555: 547: 545: 541: 537: 534: 526: 524: 520: 517: 512: 511:replicability 508: 504: 495: 493: 489: 487: 483: 479: 474: 469: 466: 462: 461:replicability 458: 450: 448: 445: 444:Kevin Elliott 436: 434: 430: 427: 422: 418: 416: 412: 403: 401: 397: 394: 386: 382: 380: 376: 372: 367: 364: 360: 359: 354: 353: 348: 347: 342: 338: 334: 333:PLOS Medicine 329: 327: 322: 314: 309: 301: 299: 297: 291: 288: 287:Irving Karten 284: 283:Stanley Schor 280: 276: 275: 274:meta-analysis 269: 266: 258: 256: 254: 250: 244: 241: 235: 233: 228: 224: 220: 216: 212: 208: 199: 194: 192: 190: 186: 182: 178: 173: 169: 166: 161: 159: 154: 150: 142: 140: 136: 134: 128: 125: 121: 116: 109: 107: 104: 100: 95: 93: 92:repeatability 89: 85: 84:United States 80: 78: 74: 70: 66: 65:replicability 60: 54: 49: 47: 45: 41: 35: 33: 27: 25: 21: 5270:Open science 5244:. Retrieved 5239: 5227:. Retrieved 5207: 5190:. RTI Press. 5187: 5176:. Retrieved 5154: 5123: 5092: 5070:. Retrieved 5065: 5050:. Retrieved 5012: 4997:. Retrieved 4975: 4953:. Retrieved 4925: 4921: 4906:. Retrieved 4878: 4874: 4853: 4849: 4796: 4792: 4777:. Retrieved 4739: 4735: 4680: 4676: 4661:. Retrieved 4655: 4601: 4597: 4582:. Retrieved 4562: 4558: 4503: 4499: 4444: 4440: 4425:. Retrieved 4379: 4375: 4359:. Retrieved 4331: 4327: 4270: 4267:PLOS Biology 4266: 4251:. Retrieved 4208:(7810): 32. 4205: 4201: 4162: 4158: 4140:cite journal 4129:. Retrieved 4091: 4087: 4072:. Retrieved 4044: 4040: 3993: 3990:Cell Systems 3989: 3974:. Retrieved 3946: 3942: 3917:cite journal 3906:. Retrieved 3841: 3837: 3822:. Retrieved 3784: 3780: 3765:. Retrieved 3735: 3731: 3674: 3670: 3621: 3617: 3568: 3564: 3515: 3511: 3496:. Retrieved 3468: 3464: 3449:. Retrieved 3413: 3409: 3370: 3366: 3351:. Retrieved 3334:(1): 46–64. 3331: 3327: 3312:. Retrieved 3284: 3280: 3235: 3231: 3220:. Retrieved 3192: 3188: 3139: 3135: 3096: 3092: 3043: 3039: 2990: 2987:PLOS Biology 2986: 2957:10722/230596 2939: 2935: 2920:. Retrieved 2882: 2878: 2829: 2826:PLOS Biology 2825: 2810:. Retrieved 2790: 2786: 2768:cite journal 2757:. Retrieved 2732:(1): 19–42. 2729: 2725: 2707:cite journal 2696:. Retrieved 2671:(1): 48–53. 2668: 2664: 2649:. Retrieved 2637: 2633: 2618:. Retrieved 2590: 2586: 2531: 2527: 2512:. Retrieved 2476: 2472: 2418:(5): –5738. 2415: 2411: 2396:. Retrieved 2376: 2372: 2323: 2319: 2288: 2277:. Retrieved 2241: 2237: 2222:. Retrieved 2194: 2190: 2153: 2149: 2119: 2100: 2081: 2062: 2043: 2024: 2005: 1986: 1943:. Retrieved 1937: 1886: 1882: 1864:Bibliography 1857:, p. 9. 1850: 1838: 1833:, p. 4. 1826: 1815: 1803: 1798:, p. 1. 1791: 1724: 1719:, p. 9. 1712: 1707:, p. 8. 1685: 1673: 1661: 1628: 1616: 1609:Justman 2019 1604: 1599:, p. 2. 1582:, p. 2. 1575: 1570:, p. X. 1568:Elliott 2017 1563: 1556:Douglas 2009 1551: 1544:Douglas 2009 1539: 1532:Douglas 2009 1527: 1515: 1473: 1453:, p. 4. 1446: 1434: 1422: 1410: 1398: 1391:Elliott 2020 1386: 1381:, p. 6. 1379:Elliott 2020 1374: 1362: 1350: 1345:, p. 1. 1338: 1333:, p. 2. 1326: 1314: 1302: 1295:Bausell 2021 1290: 1278: 1251: 1239:. Retrieved 1231: 1222: 1210: 1198: 1186: 1179:Bausell 2021 1174: 1169:, p. 2. 1162: 1150: 1138: 1133:, p. 3. 1126: 1114: 1107:Steinle 2016 1102: 1090: 1057: 1050:Steinle 2016 1045: 1033: 1021: 1009: 1002:Steinle 2016 997: 975:Fanelli 2018 970: 963:Steinle 2016 958: 946: 920:, p. 2. 918:Elliott 2020 898: 893:, p. 5. 886: 874: 867:Fanelli 2009 847: 835: 823: 811: 806:, p. 1. 804:Bausell 2021 799: 794:, p. 2. 792:Plesser 2018 787: 775: 770:, p. 2. 749:Plesser 2018 744: 730: 728: 709: 707: 698: 694: 685: 681: 676: 674: 669: 665: 663: 653: 651: 646: 642: 638: 632: 629: 619: 615: 611: 609: 602: 593: 587: 585: 576: 572: 559: 551: 542: 538: 532: 530: 521: 515: 510: 506: 502: 499: 490: 470: 464: 460: 456: 454: 440: 431: 425: 423: 419: 407: 398: 392: 390: 368: 362: 356: 350: 344: 340: 330: 323: 319: 295: 292: 279:meta-science 278: 272: 270: 262: 252: 249:Robert Doyle 245: 239: 236: 231: 222: 218: 214: 206: 203: 176: 174: 170: 164: 162: 158:open science 157: 149:open science 146: 137: 129: 123: 119: 117: 113: 98: 96: 91: 81: 76: 72: 68: 64: 61: 58: 43: 40:open science 36: 31: 28: 19: 18: 5208:Nature News 5081:Conferences 5029:10871/31336 4659:. MetaArXiv 4447:(1): 5840. 4405:2434/717112 4231:2066/221028 4109:2066/222238 3471:: 428–436. 3416:(1): 1–11. 3373:(1): 0021. 2793:(1–4): 62. 2326:(8): –124. 1717:Romero 2019 1705:Romero 2019 1508:Crosas 2014 1369:, p. 6 1145:, p. 8 816:Pimple 2002 503:replication 468:analysis". 50:Definitions 5259:Categories 5246:2020-02-10 5229:2020-02-10 5178:2022-06-13 5072:2020-02-10 5052:2022-09-10 4999:2020-02-10 4955:2022-06-12 4908:2022-06-12 4779:2022-06-12 4663:2022-02-18 4584:2022-09-10 4427:2022-09-10 4361:2022-06-12 4253:2022-09-10 4165:: 120037. 4131:2022-09-10 4074:2022-09-10 4047:(1): 3–5. 3996:(1): 1–2. 3976:2022-09-10 3908:2020-02-10 3824:2022-09-10 3787:(1): 1–4. 3767:2021-11-11 3498:2021-11-11 3451:2020-02-12 3353:2022-06-10 3314:2020-02-10 3222:2022-02-18 2922:2022-06-12 2885:(1): 5–6. 2879:BioScience 2812:2022-05-15 2759:2022-06-13 2698:2022-06-13 2651:2022-02-12 2620:2020-02-12 2514:2022-06-12 2398:2020-02-11 2279:2022-02-19 2224:2020-02-11 1945:2022-09-10 1493:Brase 2004 1355:Laine 2018 1271:Baker 2016 1241:19 January 1232:www.cos.io 1155:Laine 2018 737:References 5224:114978507 4950:248733109 4942:1572-8587 4903:236815515 4895:1089-2680 4856:(5): 21. 4823:1932-6203 4756:1572-8587 4705:2054-5703 4628:1932-6203 4579:0749-5978 4565:: 18–27. 4528:1551-3777 4469:2041-1723 4422:211265856 4356:225695820 4348:0045-5091 4315:220609403 4289:1545-7885 4248:219175762 4189:216442567 4181:0040-1625 4126:219740617 4118:0953-1513 4069:214414648 4061:0034-6543 4028:198912667 4012:2405-4712 3971:202261836 3963:1747-9991 3868:1932-6203 3811:2055-1045 3762:115161422 3754:1797-9129 3701:1932-6203 3640:1662-5196 3587:0027-8424 3493:158229869 3485:0148-2963 3438:0952-813X 3348:1746-8256 3301:1478-6990 3252:1946-6242 3217:155715556 3209:2213-056X 3158:1664-1078 3062:1664-1078 3009:1544-9173 2974:218065162 2899:0006-3568 2848:1545-7885 2807:0739-1137 2754:154301281 2746:1049-0965 2685:1049-0965 2646:1239-6540 2556:1095-9203 2509:146243575 2493:0165-0106 2442:1932-6203 2393:143487211 2379:: 69–88. 2342:1549-1676 2258:1471-5546 2211:0098-7484 2170:2330-9431 1911:0036-8075 1403:Peng 2011 1062:Bell 1902 903:Lyon 2016 268:process. 223:Accademia 181:open data 124:grey area 5174:(Report) 5047:55353995 4964:Chapters 4841:34242331 4793:PLOS ONE 4774:34720421 4723:34035933 4646:34242331 4598:PLOS ONE 4546:33913617 4487:34611157 4414:32099126 4307:32673304 4240:32488161 4020:31344358 3886:30011289 3838:PLOS ONE 3819:51935690 3719:30011289 3671:PLOS ONE 3658:29403370 3605:29531051 3552:29765001 3446:46838834 3397:33954258 3309:49226080 3260:27252173 3176:27933012 3142:: 1832. 3123:26940865 3080:27933012 3046:: 1832. 3027:26431313 2966:26315443 2917:26955072 2866:25768323 2693:14202765 2607:22508865 2574:22144613 2501:41476727 2460:19478950 2412:PLOS ONE 2360:16060722 2274:25084326 2266:12092490 2178:51639145 2156:: 1–27. 1929:26113702 668:(6) and 618:(3) and 413:and the 211:Florence 153:Liz Lyon 101:(2021), 4832:8270481 4801:Bibcode 4765:8550477 4714:8101540 4685:Bibcode 4637:8270481 4606:Bibcode 4537:8596791 4508:Bibcode 4478:8492806 4449:Bibcode 4384:Bibcode 4298:7365391 4210:Bibcode 3903:3433221 3877:6047784 3846:Bibcode 3710:6047784 3679:Bibcode 3649:5778115 3596:5856498 3543:5960342 3520:Bibcode 3418:Bibcode 3388:7610724 3167:5122713 3101:Bibcode 3093:Science 3071:5122713 3018:4592065 2936:Science 2908:4776714 2857:4359000 2615:8400625 2565:3383002 2536:Bibcode 2528:Science 2451:2685008 2420:Bibcode 2351:1182327 2219:5952081 1954:Reports 1920:4550299 1891:Bibcode 1883:Science 195:History 133:p value 5222: 5138: 5107: 5045: 5035: 4990: 4948: 4940: 4901: 4893: 4839: 4829: 4821: 4772: 4762: 4754: 4721: 4711: 4703: 4644: 4634: 4626: 4577: 4544: 4534: 4526: 4485: 4475: 4467: 4420: 4412: 4376:Nature 4354: 4346: 4313: 4305: 4295: 4287: 4246: 4238: 4202:Nature 4187: 4179: 4124: 4116: 4067: 4059: 4026: 4018: 4010: 3969: 3961: 3901: 3884: 3874: 3866: 3817: 3809: 3760: 3752: 3717: 3707: 3699: 3656: 3646: 3638: 3624:: 76. 3603: 3593: 3585: 3550: 3540: 3491: 3483: 3444: 3436: 3395: 3385: 3346: 3307: 3299: 3268:848096 3266: 3258: 3250: 3215: 3207: 3174: 3164: 3156: 3121: 3078: 3068: 3060: 3025: 3015: 3007: 2972: 2964: 2915: 2905: 2897: 2864: 2854: 2846: 2805: 2752: 2744: 2691: 2683: 2644: 2613: 2605: 2572: 2562: 2554: 2507: 2499: 2491: 2458: 2448: 2440: 2391: 2358: 2348: 2340: 2303: 2272: 2264: 2256: 2217: 2209: 2176: 2168: 2126: 2107: 2088: 2069: 2050: 2031: 2012: 1993: 1967: 1927: 1917: 1909: 411:Scopus 393:Nature 355:, and 5220:S2CID 5043:S2CID 4946:S2CID 4899:S2CID 4418:S2CID 4352:S2CID 4311:S2CID 4244:S2CID 4185:S2CID 4122:S2CID 4065:S2CID 4024:S2CID 3967:S2CID 3815:S2CID 3758:S2CID 3738:(4). 3489:S2CID 3442:S2CID 3305:S2CID 3264:S2CID 3213:S2CID 2970:S2CID 2750:S2CID 2689:S2CID 2640:(1). 2611:S2CID 2505:S2CID 2497:JSTOR 2389:S2CID 2270:S2CID 2174:S2CID 614:(2), 232:salon 5136:ISBN 5105:ISBN 5033:ISBN 4988:ISBN 4938:ISSN 4891:ISSN 4837:PMID 4819:ISSN 4770:PMID 4752:ISSN 4719:PMID 4701:ISSN 4642:PMID 4624:ISSN 4575:ISSN 4542:PMID 4524:ISSN 4483:PMID 4465:ISSN 4410:PMID 4344:ISSN 4303:PMID 4285:ISSN 4236:PMID 4177:ISSN 4146:link 4114:ISSN 4057:ISSN 4016:PMID 4008:ISSN 3959:ISSN 3930:help 3899:SSRN 3882:PMID 3864:ISSN 3807:ISSN 3750:ISSN 3715:PMID 3697:ISSN 3654:PMID 3636:ISSN 3601:PMID 3583:ISSN 3548:PMID 3481:ISSN 3434:ISSN 3393:PMID 3344:ISSN 3297:ISSN 3256:PMID 3248:ISSN 3205:ISSN 3172:PMID 3154:ISSN 3119:PMID 3076:PMID 3058:ISSN 3023:PMID 3005:ISSN 2962:PMID 2913:PMID 2895:ISSN 2862:PMID 2844:ISSN 2803:ISSN 2774:link 2742:ISSN 2713:link 2681:ISSN 2642:ISSN 2603:PMID 2570:PMID 2552:ISSN 2489:ISSN 2456:PMID 2438:ISSN 2356:PMID 2338:ISSN 2301:ISBN 2262:PMID 2254:ISSN 2215:PMID 2207:ISSN 2191:JAMA 2166:ISSN 2124:ISBN 2105:ISBN 2086:ISBN 2067:ISBN 2048:ISBN 2029:ISBN 2010:ISBN 1991:ISBN 1965:ISBN 1925:PMID 1907:ISSN 1243:2022 586:The 285:and 213:the 189:OECD 177:open 67:and 5212:doi 5159:doi 5128:doi 5097:doi 5025:hdl 5017:doi 4980:doi 4930:doi 4883:doi 4858:doi 4827:PMC 4809:doi 4760:PMC 4744:doi 4709:PMC 4693:doi 4632:PMC 4614:doi 4567:doi 4563:167 4532:PMC 4516:doi 4473:PMC 4457:doi 4400:hdl 4392:doi 4380:578 4336:doi 4293:PMC 4275:doi 4226:hdl 4218:doi 4206:582 4167:doi 4163:156 4104:hdl 4096:doi 4049:doi 3998:doi 3951:doi 3872:PMC 3854:doi 3797:hdl 3789:doi 3740:doi 3705:PMC 3687:doi 3644:PMC 3626:doi 3591:PMC 3573:doi 3569:115 3538:PMC 3528:doi 3516:115 3473:doi 3426:doi 3383:PMC 3375:doi 3336:doi 3289:doi 3240:doi 3197:doi 3162:PMC 3144:doi 3109:doi 3097:351 3066:PMC 3048:doi 3013:PMC 2995:doi 2952:hdl 2944:doi 2940:349 2903:PMC 2887:doi 2852:PMC 2834:doi 2795:doi 2734:doi 2673:doi 2595:doi 2560:PMC 2544:doi 2532:334 2481:doi 2446:PMC 2428:doi 2381:doi 2346:PMC 2328:doi 2293:doi 2246:doi 2199:doi 2195:195 2158:doi 2154:XIX 1915:PMC 1899:doi 1887:348 480:or 277:or 183:or 34:). 5261:: 5218:. 5210:. 5206:. 5153:. 5134:. 5122:. 5103:. 5091:. 5041:. 5031:. 5023:. 4986:. 4974:. 4944:. 4936:. 4926:53 4924:. 4920:. 4897:. 4889:. 4879:26 4877:. 4873:. 4854:89 4852:. 4835:. 4825:. 4817:. 4807:. 4797:16 4795:. 4791:. 4768:. 4758:. 4750:. 4740:52 4738:. 4734:. 4717:. 4707:. 4699:. 4691:. 4679:. 4675:. 4640:. 4630:. 4622:. 4612:. 4602:16 4600:. 4596:. 4573:. 4561:. 4557:. 4540:. 4530:. 4522:. 4514:. 4504:17 4502:. 4498:. 4481:. 4471:. 4463:. 4455:. 4445:12 4443:. 4439:. 4416:. 4408:. 4398:. 4390:. 4378:. 4374:. 4350:. 4342:. 4332:52 4330:. 4326:. 4309:. 4301:. 4291:. 4283:. 4271:18 4269:. 4265:. 4242:. 4234:. 4224:. 4216:. 4204:. 4200:. 4183:. 4175:. 4161:. 4157:. 4142:}} 4138:{{ 4120:. 4112:. 4102:. 4092:33 4090:. 4086:. 4063:. 4055:. 4045:90 4043:. 4039:. 4022:. 4014:. 4006:. 3992:. 3988:. 3965:. 3957:. 3947:14 3945:. 3941:. 3921:: 3919:}} 3915:{{ 3880:. 3870:. 3862:. 3852:. 3842:13 3840:. 3836:. 3813:. 3805:. 3795:. 3783:. 3779:. 3756:. 3748:. 3736:37 3734:. 3730:. 3713:. 3703:. 3695:. 3685:. 3675:13 3673:. 3669:. 3652:. 3642:. 3634:. 3622:11 3620:. 3616:. 3599:. 3589:. 3581:. 3567:. 3563:. 3546:. 3536:. 3526:. 3514:. 3510:. 3487:. 3479:. 3469:88 3467:. 3463:. 3440:. 3432:. 3424:. 3414:30 3412:. 3408:. 3391:. 3381:. 3369:. 3365:. 3342:. 3332:12 3330:. 3326:. 3303:. 3295:. 3285:48 3283:. 3279:. 3262:. 3254:. 3246:. 3234:. 3211:. 3203:. 3193:25 3191:. 3187:. 3170:. 3160:. 3152:. 3138:. 3134:. 3117:. 3107:. 3095:. 3091:. 3074:. 3064:. 3056:. 3042:. 3038:. 3021:. 3011:. 3003:. 2991:13 2989:. 2985:. 2968:. 2960:. 2950:. 2938:. 2934:. 2911:. 2901:. 2893:. 2883:65 2881:. 2877:. 2860:. 2850:. 2842:. 2830:13 2828:. 2824:. 2801:. 2791:37 2789:. 2785:. 2770:}} 2766:{{ 2748:. 2730:47 2728:. 2724:. 2709:}} 2705:{{ 2687:. 2669:47 2667:. 2663:. 2638:31 2636:. 2632:. 2609:. 2601:. 2591:23 2589:. 2585:. 2568:. 2558:. 2550:. 2542:. 2530:. 2526:. 2503:. 2495:. 2487:. 2477:75 2475:. 2471:. 2454:. 2444:. 2436:. 2426:. 2414:. 2410:. 2387:. 2377:18 2375:. 2371:. 2354:. 2344:. 2336:. 2322:. 2318:. 2299:. 2268:. 2260:. 2252:. 2240:. 2236:. 2213:. 2205:. 2193:. 2189:. 2172:. 2164:. 2152:. 2148:. 1923:. 1913:. 1905:. 1897:. 1885:. 1881:. 1768:^ 1751:^ 1736:^ 1697:^ 1640:^ 1587:^ 1500:^ 1485:^ 1458:^ 1263:^ 1234:. 1230:. 1069:^ 982:^ 925:^ 910:^ 859:^ 756:^ 349:, 234:. 5249:. 5232:. 5214:: 5181:. 5165:. 5161:: 5144:. 5130:: 5113:. 5099:: 5075:. 5055:. 5027:: 5019:: 5002:. 4982:: 4958:. 4932:: 4911:. 4885:: 4864:. 4860:: 4843:. 4811:: 4803:: 4782:. 4746:: 4725:. 4695:: 4687:: 4681:8 4666:. 4648:. 4616:: 4608:: 4587:. 4569:: 4548:. 4518:: 4510:: 4489:. 4459:: 4451:: 4430:. 4402:: 4394:: 4386:: 4364:. 4338:: 4317:. 4277:: 4256:. 4228:: 4220:: 4212:: 4191:. 4169:: 4148:) 4134:. 4106:: 4098:: 4077:. 4051:: 4030:. 4000:: 3994:9 3979:. 3953:: 3932:) 3928:( 3911:. 3888:. 3856:: 3848:: 3827:. 3799:: 3791:: 3785:4 3770:. 3742:: 3721:. 3689:: 3681:: 3660:. 3628:: 3607:. 3575:: 3554:. 3530:: 3522:: 3501:. 3475:: 3454:. 3428:: 3420:: 3399:. 3377:: 3371:1 3356:. 3338:: 3317:. 3291:: 3270:. 3242:: 3236:8 3225:. 3199:: 3178:. 3146:: 3140:7 3125:. 3111:: 3103:: 3082:. 3050:: 3044:7 3029:. 2997:: 2976:. 2954:: 2946:: 2925:. 2889:: 2868:. 2836:: 2815:. 2797:: 2776:) 2762:. 2736:: 2715:) 2701:. 2675:: 2654:. 2623:. 2597:: 2576:. 2546:: 2538:: 2517:. 2483:: 2462:. 2430:: 2422:: 2416:4 2401:. 2383:: 2362:. 2330:: 2324:2 2309:. 2295:: 2282:. 2248:: 2242:8 2227:. 2201:: 2180:. 2160:: 2132:. 2113:. 2094:. 2075:. 2056:. 2037:. 2018:. 1999:. 1973:. 1948:. 1931:. 1901:: 1893:: 1845:. 1810:. 1786:. 1668:. 1623:. 1611:. 1495:. 1480:. 1429:. 1417:. 1405:. 1321:. 1273:. 1258:. 1245:. 1217:. 1193:. 1016:. 977:. 965:. 881:. 869:. 854:. 842:. 830:. 363:p

Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.

Index