-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathsips2019.html
711 lines (627 loc) · 27.6 KB
/
sips2019.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta charset="utf-8" />
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="pandoc" />
<title>SIPS 2019 Hackathon on Open research documentation</title>
<script src="site_libs/jquery-1.11.3/jquery.min.js"></script>
<meta name="viewport" content="width=device-width, initial-scale=1" />
<link href="site_libs/bootstrap-3.3.5/css/flatly.min.css" rel="stylesheet" />
<script src="site_libs/bootstrap-3.3.5/js/bootstrap.min.js"></script>
<script src="site_libs/bootstrap-3.3.5/shim/html5shiv.min.js"></script>
<script src="site_libs/bootstrap-3.3.5/shim/respond.min.js"></script>
<script src="site_libs/jqueryui-1.11.4/jquery-ui.min.js"></script>
<link href="site_libs/tocify-1.9.1/jquery.tocify.css" rel="stylesheet" />
<script src="site_libs/tocify-1.9.1/jquery.tocify.js"></script>
<script src="site_libs/navigation-1.1/tabsets.js"></script>
<style type="text/css">
h1 {
font-size: 34px;
}
h1.title {
font-size: 38px;
}
h2 {
font-size: 30px;
}
h3 {
font-size: 24px;
}
h4 {
font-size: 18px;
}
h5 {
font-size: 16px;
}
h6 {
font-size: 12px;
}
.table th:not([align]) {
text-align: left;
}
</style>
<style type = "text/css">
.main-container {
max-width: 940px;
margin-left: auto;
margin-right: auto;
}
code {
color: inherit;
background-color: rgba(0, 0, 0, 0.04);
}
img {
max-width:100%;
height: auto;
}
.tabbed-pane {
padding-top: 12px;
}
.html-widget {
margin-bottom: 20px;
}
button.code-folding-btn:focus {
outline: none;
}
summary {
display: list-item;
}
</style>
<style type="text/css">
/* padding for bootstrap navbar */
body {
padding-top: 60px;
padding-bottom: 40px;
}
/* offset scroll position for anchor links (for fixed navbar) */
.section h1 {
padding-top: 65px;
margin-top: -65px;
}
.section h2 {
padding-top: 65px;
margin-top: -65px;
}
.section h3 {
padding-top: 65px;
margin-top: -65px;
}
.section h4 {
padding-top: 65px;
margin-top: -65px;
}
.section h5 {
padding-top: 65px;
margin-top: -65px;
}
.section h6 {
padding-top: 65px;
margin-top: -65px;
}
.dropdown-submenu {
position: relative;
}
.dropdown-submenu>.dropdown-menu {
top: 0;
left: 100%;
margin-top: -6px;
margin-left: -1px;
border-radius: 0 6px 6px 6px;
}
.dropdown-submenu:hover>.dropdown-menu {
display: block;
}
.dropdown-submenu>a:after {
display: block;
content: " ";
float: right;
width: 0;
height: 0;
border-color: transparent;
border-style: solid;
border-width: 5px 0 5px 5px;
border-left-color: #cccccc;
margin-top: 5px;
margin-right: -10px;
}
.dropdown-submenu:hover>a:after {
border-left-color: #ffffff;
}
.dropdown-submenu.pull-left {
float: none;
}
.dropdown-submenu.pull-left>.dropdown-menu {
left: -100%;
margin-left: 10px;
border-radius: 6px 0 6px 6px;
}
</style>
<script>
// manage active state of menu based on current page
$(document).ready(function () {
// active menu anchor
href = window.location.pathname
href = href.substr(href.lastIndexOf('/') + 1)
if (href === "")
href = "index.html";
var menuAnchor = $('a[href="' + href + '"]');
// mark it active
menuAnchor.parent().addClass('active');
// if it's got a parent navbar menu mark it active as well
menuAnchor.closest('li.dropdown').addClass('active');
});
</script>
<!-- tabsets -->
<style type="text/css">
.tabset-dropdown > .nav-tabs {
display: inline-table;
max-height: 500px;
min-height: 44px;
overflow-y: auto;
background: white;
border: 1px solid #ddd;
border-radius: 4px;
}
.tabset-dropdown > .nav-tabs > li.active:before {
content: "";
font-family: 'Glyphicons Halflings';
display: inline-block;
padding: 10px;
border-right: 1px solid #ddd;
}
.tabset-dropdown > .nav-tabs.nav-tabs-open > li.active:before {
content: "";
border: none;
}
.tabset-dropdown > .nav-tabs.nav-tabs-open:before {
content: "";
font-family: 'Glyphicons Halflings';
display: inline-block;
padding: 10px;
border-right: 1px solid #ddd;
}
.tabset-dropdown > .nav-tabs > li.active {
display: block;
}
.tabset-dropdown > .nav-tabs > li > a,
.tabset-dropdown > .nav-tabs > li > a:focus,
.tabset-dropdown > .nav-tabs > li > a:hover {
border: none;
display: inline-block;
border-radius: 4px;
}
.tabset-dropdown > .nav-tabs.nav-tabs-open > li {
display: block;
float: none;
}
.tabset-dropdown > .nav-tabs > li {
display: none;
}
</style>
<script>
$(document).ready(function () {
window.buildTabsets("TOC");
});
$(document).ready(function () {
$('.tabset-dropdown > .nav-tabs > li').click(function () {
$(this).parent().toggleClass('nav-tabs-open')
});
});
</script>
<!-- code folding -->
<script>
$(document).ready(function () {
// move toc-ignore selectors from section div to header
$('div.section.toc-ignore')
.removeClass('toc-ignore')
.children('h1,h2,h3,h4,h5').addClass('toc-ignore');
// establish options
var options = {
selectors: "h1,h2,h3",
theme: "bootstrap3",
context: '.toc-content',
hashGenerator: function (text) {
return text.replace(/[.\\/?&!#<>]/g, '').replace(/\s/g, '_').toLowerCase();
},
ignoreSelector: ".toc-ignore",
scrollTo: 0
};
options.showAndHide = true;
options.smoothScroll = true;
// tocify
var toc = $("#TOC").tocify(options).data("toc-tocify");
});
</script>
<style type="text/css">
#TOC {
margin: 25px 0px 20px 0px;
}
@media (max-width: 768px) {
#TOC {
position: relative;
width: 100%;
}
}
.toc-content {
padding-left: 30px;
padding-right: 40px;
}
div.main-container {
max-width: 1200px;
}
div.tocify {
width: 20%;
max-width: 260px;
max-height: 85%;
}
@media (min-width: 768px) and (max-width: 991px) {
div.tocify {
width: 25%;
}
}
@media (max-width: 767px) {
div.tocify {
width: 100%;
max-width: none;
}
}
.tocify ul, .tocify li {
line-height: 20px;
}
.tocify-subheader .tocify-item {
font-size: 0.90em;
}
.tocify .list-group-item {
border-radius: 0px;
}
</style>
</head>
<body>
<div class="container-fluid main-container">
<!-- setup 3col/9col grid for toc_float and main content -->
<div class="row-fluid">
<div class="col-xs-12 col-sm-4 col-md-3">
<div id="TOC" class="tocify">
</div>
</div>
<div class="toc-content col-xs-12 col-sm-8 col-md-9">
<div class="navbar navbar-default navbar-fixed-top" role="navigation">
<div class="container">
<div class="navbar-header">
<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="index.html">scienceverse</a>
</div>
<div id="navbar" class="navbar-collapse collapse">
<ul class="nav navbar-nav">
</ul>
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">
Menu
<span class="caret"></span>
</a>
<ul class="dropdown-menu" role="menu">
<li>
<a href="scienceverse/index.html">scienceverse R package</a>
</li>
<li>
<a href="sips2019.html">SIPS 2019 Hackathon</a>
</li>
<li>
<a href="metadatalist.html">Other Meta-Data Projects</a>
</li>
</ul>
</li>
</ul>
</div><!--/.nav-collapse -->
</div><!--/.container -->
</div><!--/.navbar -->
<div class="fluid-row" id="header">
<h1 class="title toc-ignore">SIPS 2019 Hackathon on Open research documentation</h1>
<h3 class="subtitle">(or: Scienceverse: A Machine Readable Grammar of Science)</h3>
</div>
<div id="abstract" class="section level2">
<h2>Abstract</h2>
<p>The goal of scienceverse is to generate and process machine-readable study descriptions to facilitate archiving studies, pre-registering studies, finding variables and measures used in other research, meta-analyses of studies, and finding and re-using datasets in other ways. Explicitly documenting hypotheses, materials, methods/procedure, data, and analysis code facilitates cumulative science. It is often difficult to exactly recover some of these aspects, and usually impossible to automate this process. We will present a preliminary version of a framework (with an associated R package and shiny app) for organising this information in a machine-readable way and automatically producing human-readable summaries. This framework can integrate with other projects to document study planning decisions (e.g., justifier), data (e.g., Psych-DS, codebook), and methods (e.g., experimentum, formR).</p>
<p>We have 4 main goals in this project, which we might examine collaboratively, or in small subgroups.</p>
<ol style="list-style-type: decimal">
<li>Imagine a fully developed Grammar of Science exists. All aspects of a study you need are machine readable. What would you want to use it for? Which use-cases can we identify?</li>
<li>What do we need to get to this fully developed Grammar of Science? Which aspects of the research cycle and research output do we need to develop a Grammar for? For example, how can we develop a Grammar for paradigms used in psychology?</li>
<li>What are the potential roadblocks to realizing a Grammar of Science that can be widely implemented? What is needed to resolve these roadblocks?</li>
<li>How does Scienceverse integrate with other ongoing projects at SIPS? Is it possible for Scienceverse to facilitate these other projects? E.g., machine readable codebooks, better preregistration documents, Psych-DS, etc. Throughout SIPS, tag any projects you run into that Scienceverse should interact with!</li>
</ol>
</div>
<div id="resources" class="section level2">
<h2>Resources</h2>
<ul>
<li><a href="https://docs.google.com/document/d/1DKhnypsG__XG9k_16smU3IJDYGgnxFP5LHw4P6Qh50g/edit">Main Google Doc</a></li>
<li><a href="https://docs.google.com/document/d/1eI9rY20f0exxeegZZlnX-D0cNGoZcl36rKYFMtnRqbU/">Projects group Google Doc</a></li>
<li><a href="https://docs.google.com/document/d/1Qoi0YxlGFWqsxSsRtTFZxOPd3LTCn5qyQShvI_NvpAI/">Needs Google Doc</a></li>
<li><a href="https://osf.io/c52yh/">OSF</a></li>
<li><a href="https://sips-2019.slack.com/messages/h_open_documentation">Slack channel</a></li>
</ul>
<p>The links above provide the raw materials created during the hackathon. Below, we summarise the discussion.</p>
</div>
<div id="use-cases" class="section level2">
<h2>Use Cases</h2>
<p>Imagine a fully developed Grammar of Science exists. All aspects of a study you need are machine readable. What would you want to use it for? Which use-cases can we identify?</p>
<div id="contributors" class="section level3">
<h3>Contributors</h3>
<ul>
<li>Anna Szabelska (<a href="mailto:[email protected]" class="email">[email protected]</a>)</li>
<li>Aleksandra Tolopilo (<a href="mailto:[email protected]" class="email">[email protected]</a>)</li>
<li>Bjørn Sætrevik (<a href="mailto:[email protected]" class="email">[email protected]</a>)</li>
<li>Ekaterina (Katia) Damer <a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Michal Olszanowski (<a href="mailto:[email protected]" class="email">[email protected]</a>)</li>
<li>Martin Müller (<a href="mailto:[email protected]" class="email">[email protected]</a>)</li>
<li>Kohinoor Monish Darda (<a href="mailto:[email protected]" class="email">[email protected]</a>)</li>
<li>Constantin Yves Plessen(<a href="mailto:[email protected]" class="email">[email protected]</a>)</li>
<li>Sau-Chin Chen; <a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Ilse Pit; <a href="mailto:[email protected]" class="email">[email protected]</a>; <span class="citation">@ilsepit</span></li>
<li>Pierpaolo Primoceri <a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Ruud Hortensius <a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Zoltan Kekecs (<a href="mailto:[email protected]" class="email">[email protected]</a>)</li>
<li>David Moreau <a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Kristina Wiebels <a href="mailto:[email protected]" class="email">[email protected]</a></li>
</ul>
</div>
<div id="evaluating-literature" class="section level3">
<h3>Evaluating literature:</h3>
<ul>
<li>Simplifying and accelerating review processes</li>
<li>Give machine recommendations for reviewers - e.g.: check this section, because it looks different from the others, or that it looks fine.</li>
<li>Integrating of behavioural/neuroimaging/etc. data, methods, databases etc with published ‘article’</li>
<li>Ease the evaluation of manuscripts, which accelerates the review process</li>
<li>Links between behavioural, neuroimaging and other kinds of data and univariate and multivariate analyses</li>
<li>A standardized taxonomy for keywords, cover all psy research in non-overlapping categories</li>
<li>Who do I need to cite in my paper?</li>
<li>Who should I recommend as a reviewer for my manuscript?</li>
</ul>
</div>
<div id="accelerate-the-writing-of-papers" class="section level3">
<h3>Accelerate the writing of papers</h3>
<ul>
<li>Automatic answers to questions you have, e.g. what is the effect of x on y? And get an answer based on the database, including reliability score</li>
<li>Automated manuscript write-up based on the pre-registered hypotheses and analysis plan (machine-recommended language for the results section of the manuscript)</li>
<li>Similar to “fMRI prep”</li>
</ul>
</div>
<div id="research-synthesis-meta-analysis" class="section level3">
<h3>Research synthesis & meta analysis</h3>
<ul>
<li>Make data (sets) accessible and searchable as a whole by topics/variables</li>
<li>Automated meta-analyses / continuously updating meta-analyses</li>
<li>We can have automated meta-meta-analyses on more meta-subjects</li>
<li>Future proofing meta-analyses / meta-meta analyses</li>
<li>All transformations / formulas need to be attached as meta-data</li>
<li>Decisions rules in case of deviations from the analysis plan need to be specified (i.e. categorical moderator is changed from 4 to 3 categories because not enough observations were obtained in one category)</li>
</ul>
</div>
<div id="collaboration-planning" class="section level3">
<h3>Collaboration / planning</h3>
<ul>
<li>Be able to Google data sets</li>
<li>Who have looked at the relationship between these variables before?
<ul>
<li>When considering new projects:</li>
<li>When evaluating grant proposals for novelty</li>
</ul></li>
<li>Create study blobs - linking up unrelated studies which use the same variables</li>
<li>Including future research ideas/hypotheses that are also accessible / recommended future research questions</li>
<li>Suggesting the design based on the claimed hypotheses</li>
<li>Good for big collaborations</li>
<li>Time stamped workflow (e.g., integrated with workflowr)</li>
<li>Compare semantic information: instructions etc</li>
</ul>
</div>
<div id="scicom" class="section level3">
<h3>Scicom</h3>
<p>Science education & communication, community, stakeholders</p>
<ul>
<li>Better science education, e.g. this could broaden students’ scope / perspective on what data already exists and what hypotheses one might test</li>
<li>Better science communication and dissemination in applied settings, e.g. scienceverse could make it easier for journalists or politicians to interpret evidence and its robustness</li>
<li>One format to teach students</li>
<li>Search enginge where people from the public can enter research questions, and a bot will convert the question to research hypothesis and will search the database for compatible stuides, do a quick meta-analyis and answer the question</li>
</ul>
</div>
<div id="pre-registration" class="section level3">
<h3>Pre-registration</h3>
<ul>
<li>A shiny-app for (quick/simple?) pre-registrations in machine readable form.
<ul>
<li>Start with simple but formalized predictions.</li>
<li>Suggest as one of the OSF pre-reg options?</li>
<li>Does not need to wait for “the perfect future”</li>
</ul></li>
<li>Better comparison between different pre-registrations/hypotheses of similar studies/investigations</li>
<li>Integrate the preregistration templates<br />
</li>
<li>Include power analyses (if not already possible)</li>
</ul>
</div>
<div id="theoretical-development" class="section level3">
<h3>Theoretical development</h3>
<ul>
<li>Work towards “a grand unified theory” understanding of psychology</li>
<li>Machine readable findings would allow us to compile all findings on a given subject, compare contributions</li>
<li>Iterate faster between developing theory and testing hypotheses</li>
</ul>
</div>
<div id="analysis" class="section level3">
<h3>Analysis</h3>
<ul>
<li>Pre-processing:
<ul>
<li>cleaning data</li>
<li>exactly when items and variables removed</li>
<li>outliers</li>
<li>wrong values</li>
<li>Crohnbach’s alpha is wrong</li>
</ul></li>
<li>Containers: standardized ways of analysing (also preprocessing) data</li>
<li>Running other’s hypotheses on your data automatically</li>
</ul>
</div>
<div id="experimental-design" class="section level3">
<h3>Experimental design:</h3>
<ul>
<li>Standardize experimental manipulation methods</li>
<li>Reuse experiments; psychopy (scripts) info etc</li>
<li>Recycle stimuli presentation script</li>
<li>Recycle questionnaires/experiment info</li>
<li>Increases reproducibility</li>
</ul>
</div>
<div id="reliability-robustness" class="section level3">
<h3>Reliability / robustness:</h3>
<ul>
<li>Evaluate how robust were the analyses?</li>
<li>Adding meta-data for multiverse and/or specification curve analysis?</li>
<li>What other decisions/analyses could have been done?</li>
<li>Automated robustness check based on the literature</li>
<li>Limit researcher degrees of freedom and avoid p-hacking</li>
</ul>
</div>
<div id="integrations" class="section level3">
<h3>Integrations</h3>
<ul>
<li>R Markdown *<a href="www.prolific.ac">Prolific</a> reduce research redundancy (so researchers don’t have to collect the same data over and over again, but can see where there are gaps in the data)</li>
</ul>
</div>
</div>
<div id="needs" class="section level2">
<h2>Needs</h2>
<p>What do we need to get to this fully developed Grammar of Science? Which aspects of the research cycle and research output do we need to develop a Grammar for? For example, how can we develop a Grammar for paradigms used in psychology?</p>
<p>This groups’ thoughts are pretty complex and can best be viewed at the <a href="https://docs.google.com/document/d/1Qoi0YxlGFWqsxSsRtTFZxOPd3LTCn5qyQShvI_NvpAI/edit">Google Doc</a> until we have a chance to synthesise things.</p>
<div id="contributors-1" class="section level3">
<h3>Contributors</h3>
<ul>
<li>David Moreau, The University of Auckland, <a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Kristina Wiebels, The University of Auckland, <a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Ian Hussey, Ghent University, <span class="citation">@ianhussey</span> <a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Ginette Lafit, KU Leuven, <a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Liam Satchell <span class="citation">@lpsatchell</span> University of Winchester, <a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Tanja Burgard, ZPID, <a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Elena SIxtus, Uni Potsdam, <a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Daniel Toribio-Florez, <a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Shannon McNee, University of Glasgow, <a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Anna Lohmann, <a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Ellie Hassan, University of Exeter, <a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Gabriela Hofer, University of Graz, <a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Franziska Stanke, University of Münster, <a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Divya Seernani, <a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Elise Gould, <a href="mailto:[email protected]" class="email">[email protected]</a></li>
</ul>
</div>
</div>
<div id="roadblocks" class="section level2">
<h2>Roadblocks</h2>
<p>What are the potential roadblocks to realizing a Grammar of Science that can be widely implemented? What is needed to resolve these roadblocks?</p>
<div id="contributors-2" class="section level3">
<h3>Contributors</h3>
</div>
<div id="motivations" class="section level3">
<h3>Motivations</h3>
<ul>
<li>There needs to be a reason for researchers (not just meta-scientists) to use the scienceverse. This could be if the scienceverse becomes a useful database of results; one that researchers would search before conducting their studies (in addition to, or as part of, Google Scholar and other academic search engines).</li>
</ul>
</div>
<div id="missing-essentials" class="section level3">
<h3>Missing essentials</h3>
<ul>
<li>Registration documents must clearly specify the primary and secondary hypotheses.</li>
<li>The package needs to incorporate the ability to separate and specify changes to preregistered analysis protocols.</li>
<li>Containerisation will be required to make the results from scienceverse hypotheses fully reproducible.</li>
</ul>
</div>
<div id="complexitysimplicity" class="section level3">
<h3>Complexity/simplicity</h3>
<ul>
<li>The scienceverse may not be for everyone at first. Does this limit its utility?</li>
<li>Researchers will find it difficult to specify their hypotheses in advance.</li>
<li>Including the results of more complex statistical analyses may be more difficult, but it is possible: the key parameters just need to be included in a list.</li>
<li>There is an extra step involved for those conducting their analyses outside of R. This could be another barrier to entry, but may be overcome by the use of easy-to-complete online apps.</li>
<li>To work, the specification must be very flexible. This will require a great deal of development.</li>
<li>People use things because of their cool tools! Their extra functionality. If the scienceverse is linked to other useful functionality then people will be more likely to use it.</li>
<li>Integration with other open source software would make the scienceverse easier to use (e.g., JASP, jamovi).</li>
</ul>
</div>
<div id="standarisation-vs.flexibility" class="section level3">
<h3>Standarisation vs. flexibility</h3>
<ul>
<li>The centralisation of the project could make it difficult to implement and difficult to get researchers on board who do not use other open science websites and tools.</li>
<li>Naming analysis parameters in a standardised way will facilitate the use of the scienceverse, including for meta-analysis. But the scienceverse must work well with non-standardised output for it to be widely adopted and easy to use (currently possible but more difficult?). This link may be possible within json files.</li>
</ul>
</div>
<div id="optimism" class="section level3">
<h3>Optimism</h3>
<ul>
<li>Researchers may be motivated to use the scienceverse because it could help them to plan their analyses by helping them to simulate data based on their hypotheses and data structure.</li>
<li>This could help researchers to find and work with the particular measurement tools they are interested in using or analysing data from (can also include delivery variables, e.g. online vs. face-to-face, standard question order vs. a randomised order).</li>
<li>A book on experimental design using the scienceverse (and incorporation in teaching) may be required to get students and academic faculty to use it.</li>
</ul>
</div>
</div>
<div id="aligned-projects" class="section level2">
<h2>Aligned Projects</h2>
<p>How does Scienceverse integrate with other ongoing projects at SIPS? Is it possible for Scienceverse to facilitate these other projects? E.g., machine readable codebooks, better preregistration documents, Psych-DS, etc. Throughout SIPS, tag any projects you run into that Scienceverse should interact with!</p>
<div id="contributors-3" class="section level3">
<h3>Contributors</h3>
<ul>
<li><a href="mailto:[email protected]" class="email">[email protected]</a> <span class="citation">@rt_thibault</span></li>
<li><a href="mailto:[email protected]" class="email">[email protected]</a> <span class="citation">@ebanders</span></li>
<li>Johannes Breuer, <a href="mailto:[email protected]" class="email">[email protected]</a>, <span class="citation">@MattEagle09</span></li>
<li><a href="mailto:[email protected]" class="email">[email protected]</a></li>
<li>Stephan Heunis</li>
<li>Thomas Richardson</li>
<li>Elise Gould</li>
<li>Zita Mayer</li>
</ul>
<p>The list of <a href="metadatlist.html">Meta-data Projects and Tools</a> is derived from the list generated by this group.</p>
</div>
<div id="ideas" class="section level3">
<h3>Ideas</h3>
<ul>
<li>Essential versus encouraged documentation</li>
<li>Are the researchers responsible for inputting all the information? Or, should we involve librarians or others to fill in the grammar field?</li>
<li>What if we still have meta-analyses where those interested in a specific questions fill in all the grammar? This takes the extra workload off the original researchers (find the right balance)</li>
<li>Can we make a common format platform for meta-analysts and systematic reviewers to upload their findings too, so that the reviews can be continued into the future?</li>
<li>This could incentivize the release of old data</li>
<li>Cunningham’s law: the quickest way to find the right answer (online) is to post the wrong answer.</li>
<li>Could journals (esp. community journals) implement checklists to help achieve a grammar of science (eg. OHBM–organization for human brain mapping, is using the COBIDAS checklist)</li>
<li>What about journals like Psychological Science, or APA journals…</li>
<li>Possibility to generate some visual output/summary of the study/project structure/design (that could be included in presentations or publications)?</li>
<li>The specification of all study parts (esp. In the analysis) for complex designs might be challenging (for example, longitudinal studies, complex sampling strategies, multilevel models, etc.)</li>
<li>Documentation ideally for whole research lifecycle (for a nice visualization of the research lifecycle see <a href="https://how-to-open.science/" class="uri">https://how-to-open.science/</a>)</li>
<li>It should “threaten” to constrain exploratory research</li>
</ul>
</div>
</div>
</div>
</div>
</div>
<script>
// add bootstrap table styles to pandoc tables
function bootstrapStylePandocTables() {
$('tr.header').parent('thead').parent('table').addClass('table table-condensed');
}
$(document).ready(function () {
bootstrapStylePandocTables();
});
</script>
<!-- dynamically load mathjax for compatibility with self-contained -->
<script>
(function () {
var script = document.createElement("script");
script.type = "text/javascript";
script.src = "https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML";
document.getElementsByTagName("head")[0].appendChild(script);
})();
</script>
</body>
</html>