644:
imagine that there are 15 good measures of an RfA candidate. If we came up with 10 and had the community rank them, we would be missing 5. If came up with 20 and had the community rank them, when we do the cluster analysis over a large sample, it becomes statistically evident that 5 of them are not good measures, and we throw them out. In your examples above, any one person cannot affect the results so do not worry about skewing anything. If you found 4 or 5 items you thought were poor, place them anywhere randomly, and when we analyze data over 100+ users we will see that people are not consistently placing items in the same bin, and thus they are a poor measure. In addition, when naming measures and categories it was a challenge to make them short enough so you aren't required to read a paragraph every time you made a decision, but long enough to be meaningful. While we spent substantial time improving the survey and piloting it, I'm sure as with any piece of research we make errors in clarity vs. brevity.
586:
study. Then their study just seems, well, odd. There isn't sufficient information provided on the survey page about who is conducting the research and why, and what question(s) they're seeking to answer; the study doesn't appear well-designed or capable of producing meaningful results. There are no open-ended questions. Finally, it seems strange that they failed to take advantage of one aspect of
Knowledge (XXG): the talk page. CMUReasercher could have created a survey discussion subpage where people could actually discuss... whatever it is they're trying to ascertain. Anyway, best of luck to them and all that; perhaps we'll all turn out to have been utterly wrong.
173:
also seems to contain some fundamental misunderstandings of how
Knowledge (XXG) works. For instance, "appearing on watch lists" is so vague that it's hard to know what you mean. Every user appears on watchlists. Maybe you meant the frequency of such appearances; but that is nothing but their "edit count". I also wonder how "Demonstrating NPOV" can be seen as a low-level measurement. NPOV is defined by the interaction of different viewpoints, and is about as hard to define for an individual as the equality of a pig. (That latter point may not be shared by everyone at Knowledge (XXG), but I'd still like to know how you propose to measure NPOV of an individual!) —
65:
possible. I tried very hard to not violate any policies with the survey, I contacted both the Chief
Research Officer, and the the Knowledge (XXG) Research Network for advice and followed it. This survey is not for the promotion of any organization or product, it is an attempt to help the Knowledge (XXG) community, much like the creators of suggestbot or any other technical contribution, but to help the community we need input from the community. Also, personally I would like to get more involved in editing articles, I've been looking through lately on places I can make a contribution, so far issues with Organizational Behavior seem good for me to update.
140:
as though I was dragging phrases from left to right just to finish. I know that the alternative (allowing participants to sort as little as they like) will make it difficult to weed out aborted attempts and result in drastic under-representation of traits, it may be worth considering if your response is lackluster. For what it's worth I wish you luck but I probably won't go back to reattempt it.
648:
agree what category it is for across a large number of people. We're sorry if that has created in frustration in taking the survey because you feel some measures are poor, but if you would complete the survey we would have the data to empirically confirm the measures are poor from community consensus.
87:
So who, exactly, is conducting this study? I see no names whatsoever, whether grad student, doctoral candidate, principal investigator, or faculty advisor. And the outside website used for gathering data -- I would have thought
Carnegie-Mellon, of all places, might have the ability to pull together a
463:
learning how wikipedia works at the same time. We (by that I mean those of us who have plowed considerable amounts of time and effort into the project) know almost intuitively what about wikipedia is important or interesting. We can spot the salient policies, boards and norms from a mile away. We
172:
Same here. This really feels weird. I can't see any value in this. In addition to the concerns raised by others above, I am concerned about the high number of leading questions. Example: When you name an item "Edit summary length", it just begs to be dragged to the box "Edit summary usage". The list
139:
Just leaving a note to inform you folks that I started, but did not complete a survey. The format at present, where I have to categorize every single characteristic (including many that I don't bother checking up on for RfA candidates) is a strong disincentive for completion. After about 10 I felt
536:
I can also vouch for these individuals. The work they are doing here is to understand and improve
Knowledge (XXG). We should not be pushing so strongly against their efforts. Like Wikipedians, they are working to make Knowledge (XXG) better. It is just that they are trying to understand/improve
64:
My apologies
Exploding Boy for the misunderstanding, I didn't realize creating a username for 4 people would be a violation of policy. Truthfully, the account was only ever used by myself, but we thought longer term it would be better to have others be able to edit with it but now I see that is not
585:
I'm not sure anyone's really "pushing against them strongly." I think people are interested, but have some fairly legitimate concerns. To begin with they violated our username policy and got their first account blocked--it's easy to do and happens often, but doesn't bode well for a well-prepared
106:
Good to hear from your Calton! About your comment on using an outside website for the survey, did you have any trouble with the functionality of the webpage? I'm just curious if there is a reason you suggested we should have created it in-house. As for who is conducting the study, the principle
643:
Thank you for your feedback! As with any research there are always tradeoffs to be made when trying to collect data. In this case, we had the option of including less than the true number of measures or more than the true number of measures. I'm not sure what your background in research is, but
155:
Ditto. It also wasn't clear to me the reason for the format and categories. What are you actually trying to find out? My answers might change depending on what the question is. Additionally, many of the left-hand items don't seem to fit with the right-hand items at all in any meaningful way.
647:
To perhaps say the same thing more clearly, in both lists there are good and poor measures, and good and poor categories. Rather than have the research team simply decide what is good and bad, we have asked the community. If a measure is poor, we will find that out because the community cannot
203:
Likewise. Even though I'm inclined to believe it is an opinion survey, it fails even at that since every single unsorted thing has to be tacked, and many of the items are the kinds of things I'd need to be clairvoyant to tack. Also, like
SebastianHelm, I have issues with what seems to me like
188:
I started the survey and then abandoned it. The phrases and categories don't lend themselves to natural matching, so the exercise seems a bit pointless. It's quite baffling - and I say that as a research professional with 19 years' experience in social science research and experimental design.
514:
I know personally Kraut, Kittur, and Burke (though not
Collier himself). They are well-known and highly qualified scientists. What they (and I, as a researcher myself) are trying to do is learn how Knowledge (XXG) works in a scientifically rigorous way, to glean lessons from this unique and
612:
Actually, Wikiversity would probably be more useful to them, since it has a few more bells and whistles (like
DynamicPageList) to hel organize and analyze the data (see my comment below). I'm generally not one to jump to attention when someone says "hi! please take our survey!", but I
417:
and try not to demand that they render unto Caesar, as it were. I'm probably injecting my own bias here--if this were a company selling widgets I would have blocked the account on sight for spamming, but I don't feel too bad giving these folks a pass based on the context.
412:
I don't mind if this account never edits the encyclopedia. Researchers and other curious social scientists are going to pay more attention to wikipedia and a natural path for communication may be the user/user talk pages. We should remember that they are
295:
Please, always sign your entries on
Village Pump or elsewhere. The edit bar has an icon for doing it. Beyond that, I am put off by remarks above about how difficult it is to finish your survey. Do you learn something from those who start but don't finish?
436:
I guess for me it's partly that the survey itself just seems so amateurish and badly designed. Maybe it would be worth thinking up some kind of academic wikiproject that could be used for people who are interested in recruiting research subjects.
568:, and there's plenty to stand on, both for understanding Knowledge (XXG) (e.g. the book "Knowledge (XXG): The Missing Manual by John Broughton" is said to be good) and for how to do surveys. Does CMU not have better standards for their research? —
219:
Yes, but there are certain pages to edit that will make you show up on more peoples' watchlists. Also, the more different articles you edit and the more frequently you do so, the more you show up, as well. Then, people start recognizing your name.
204:
working backwards from an answer to get a solution and very vague responses (i.e. "appearing on watchlists" - ALL editors, good editors and vandals alike, tend to show up on watchlists, so a more narrow definitition is required here). -
563:
Well, if you both know them, would you have an explanation why they are so acting so amateurish - both in in their lack of preparation to understand Knowledge (XXG), and in the way they conduct the survey? Science grows by
464:
can articulate the fundamental frictions easily. A new user can do none of these things. Perhaps this team is experimenting with a novel methods (asking respondents to categorize behaviors), subject matter,
784:
for review by the community and I'm making one last call for cleanup and input. Please give the article a careful read if you have a chance. Unless major flaws are discovered, we'll be adding the
257:
24:
Hi, I notice you describe this as "our talk page" please can you confirm that that was a slip of the tongue and that only one individual has access to the password of this account.
733:
327:
Oh, so I decided to take it - content work and the like were dumped into trust worthiness, and everything else was in "varied experience". I wonder if they will understand.
672:
Hi CMUResearcher. Assuming these aren't deeply personal questions that require anonymous answers, you might have a bit better luck just asking people to take the survey on
515:
successful community/information resource -- which can then be applied to other areas, leading to better communication and better decision-making in a variety of venues.
518:
The claimed affiliation could be falsified, of course, but it seems very unlikely to me that an evildoer would choose this rather obscure fictitious identity.
565:
617:
interested in this sort of research. WV has the added advantage of letting others also take part in the research phase from all sides of the table. --
390:
I kind of agree. Are you planning on actually editing Knowledge (XXG), or is your whole purpose here to promote this ... study (or whatever it is)?
700:
699:
Hi CMU, Based on your earlier survey I wondered if you would be interested in a problem area on the Wiki that would benefit from research - see
773:
737:
726:
119:
who has also published research on Knowledge (XXG) and is a graduate student in HCI. Lastly, myself, Ben Collier, a PhD student. -
755:
156:
Also, it's not clear enough that you don't have to actually sign up. This doesn't strike me as a very well designed study.
815:
An article I wrote was recently deleted by an admin but I need to access the article (or at least the text of the article).
651:
Thank you again for all the feedback, I hope I've addressed the issues you've raised and feel free to continue to comment.
710:
31:
207:
468:
tools (maybe the web based survey bit is new). I feel that we can do little wrong by cutting them some slack.
676:, which is more or less "in house". I'm sure you would find plenty of people willing to help set it up on the
488:
FWIW, I completed the survey. I'm not sure how many other people did, though, and I'd be curious to find out.
591:
442:
395:
161:
51:
705:
26:
332:
194:
781:
762:
689:
703:. Alternatively perhaps your existing research can cast light on our problems with RFA and adminship.
366:
818:
Is there any easy way for me to do this? Can you (or another admin) send me the text of my article?
794:
template to the talk page to start the process on March 2nd. That's one week from this posting. --
824:
681:
618:
526:
301:
276:
828:
654:
587:
569:
438:
391:
316:
174:
157:
120:
66:
47:
473:
423:
145:
414:
354:
777:
496:
328:
315:
In an effort to keep conversations on topic, I moved the following here from other pages: —
228:
190:
111:, he is full professor in Human Computer Interaction. Other team members include professor
97:
766:
361:
795:
741:
550:
546:
542:
538:
522:
297:
272:
788:
469:
419:
141:
673:
491:
223:
93:
89:
832:
802:
780:
and myself are about to start the (poorly documented) process of submitting
719:
662:
626:
595:
572:
554:
530:
503:
477:
446:
427:
399:
373:
336:
319:
305:
280:
235:
214:
198:
177:
165:
149:
128:
115:
who has published several papers on Knowledge (XXG), and a graduate student
100:
74:
55:
40:
537:
Knowledge (XXG) as a system rather than on an article-by-article basis. --
268:
I (and the rest of us here, too) hope you like it here and decide to stay!
459:
I dunno. I suspect the researchers are both learning how to do surveys
772:
I'm messaging you about this because you are listed as a member of
677:
116:
740:
a couple of days ago. You might want to add your name there. --
732:
I saw that you added your name to the list of supporters in the
108:
88:
functional webpage -- gives me enormous pause. I still remember
680:
if you're not familiar with wiki-markup or data processing. --
521:
You can look me up in the real world as Reid Priedhorsky. --
701:
Knowledge (XXG) talk:Requests for adminship#Admin retention
265:
To get started, click on the link that says "welcome".
821:
I greatly appreciate any assistance you can provide,
112:
16:Please feel free to leave comments or suggestions!
46:Hi, not to worry. Glad you plan to contribute.
8:
566:dwarfs standing on the shoulders of giants
734:project proposal for WikiProject Research
667:
291:Always sign your entries on Village Pump
490:
222:
756:_Wikipedia:WikiProject_Research": -->
7:
774:Knowledge (XXG):WikiProject_Research
727:Knowledge (XXG):WikiProject_Research
311:Remarks moved here from other pages
14:
668:If you're hitting difficulties...
765:last call for cleanup before an
357:is starting to come to mind. —
107:investigator for this study is
720:16:22, 17 September 2009 (UTC)
690:23:17, 16 September 2009 (UTC)
627:23:23, 16 September 2009 (UTC)
212:
205:
1:
803:22:32, 23 February 2010 (UTC)
757:16:22, 17 November 2009 (UTC)
796:
742:
725:WikiProject Proposal --: -->
639:Response to Survey Concerns
258:welcome to Knowledge (XXG)!
849:
663:17:55, 6 August 2009 (UTC)
596:17:16, 6 August 2009 (UTC)
573:16:58, 6 August 2009 (UTC)
555:15:24, 6 August 2009 (UTC)
531:14:56, 6 August 2009 (UTC)
504:14:07, 6 August 2009 (UTC)
478:03:05, 6 August 2009 (UTC)
447:02:48, 6 August 2009 (UTC)
428:02:40, 6 August 2009 (UTC)
400:00:40, 6 August 2009 (UTC)
374:22:42, 5 August 2009 (UTC)
337:22:26, 5 August 2009 (UTC)
320:22:46, 5 August 2009 (UTC)
306:12:36, 5 August 2009 (UTC)
281:12:05, 5 August 2009 (UTC)
254:Hello, CMUResearcher, and
236:14:03, 6 August 2009 (UTC)
215:06:13, 6 August 2009 (UTC)
199:00:31, 6 August 2009 (UTC)
178:17:52, 5 August 2009 (UTC)
166:06:38, 5 August 2009 (UTC)
150:05:42, 5 August 2009 (UTC)
833:15:56, 27 July 2010 (UTC)
808:Access to deleted article
129:18:55, 16 July 2009 (UTC)
101:17:23, 16 July 2009 (UTC)
75:18:54, 16 July 2009 (UTC)
56:15:14, 16 July 2009 (UTC)
41:15:07, 16 July 2009 (UTC)
658:
124:
70:
510:These people are legit
738:the actual project
736:. We started the
801:
754:
750:
746:
553:
545:
372:
135:Survey difficulty
840:
800:
798:
793:
787:
753:
751:
748:
744:
717:
713:
708:
695:Research and RFA
686:
623:
549:
541:
501:
499:
494:
371:
369:
358:
349:Interesting ...
271:Happy editing!
233:
231:
226:
213:
211:
38:
34:
29:
848:
847:
843:
842:
841:
839:
838:
837:
810:
791:
785:
770:
730:
715:
711:
706:
697:
682:
670:
641:
619:
512:
497:
492:
367:
359:
353:, but the term
313:
293:
252:
229:
224:
137:
92:, after all. --
85:
36:
32:
27:
22:
12:
11:
5:
846:
844:
809:
806:
769:
760:
729:
723:
696:
693:
669:
666:
640:
637:
636:
635:
634:
633:
632:
631:
630:
629:
603:
602:
601:
600:
599:
598:
578:
577:
576:
575:
558:
557:
511:
508:
507:
506:
485:
484:
483:
482:
481:
480:
452:
451:
450:
449:
431:
430:
409:
408:
407:
406:
405:
404:
403:
402:
381:
380:
379:
378:
377:
376:
342:
341:
340:
339:
312:
309:
292:
289:
288:
287:
286:
285:
284:
283:
251:
248:
247:
246:
245:
244:
243:
242:
241:
240:
239:
238:
181:
180:
169:
168:
136:
133:
132:
131:
84:
81:
80:
79:
78:
77:
59:
58:
21:
18:
13:
10:
9:
6:
4:
3:
2:
845:
836:
834:
830:
826:
822:
819:
816:
813:
807:
805:
804:
799:
790:
783:
779:
775:
768:
764:
761:
759:
758:
752:
739:
735:
728:
724:
722:
721:
718:
714:
709:
702:
694:
692:
691:
687:
685:
679:
675:
665:
664:
660:
656:
655:CMUResearcher
652:
649:
645:
638:
628:
624:
622:
616:
611:
610:
609:
608:
607:
606:
605:
604:
597:
593:
589:
588:Exploding Boy
584:
583:
582:
581:
580:
579:
574:
571:
567:
562:
561:
560:
559:
556:
552:
548:
544:
540:
535:
534:
533:
532:
528:
524:
519:
516:
509:
505:
502:
500:
495:
487:
486:
479:
475:
471:
467:
462:
458:
457:
456:
455:
454:
453:
448:
444:
440:
439:Exploding Boy
435:
434:
433:
432:
429:
425:
421:
416:
411:
410:
401:
397:
393:
392:Exploding Boy
389:
388:
387:
386:
385:
384:
383:
382:
375:
370:
364:
363:
356:
352:
348:
347:
346:
345:
344:
343:
338:
334:
330:
326:
325:
324:
323:
322:
321:
318:
310:
308:
307:
303:
299:
290:
282:
278:
274:
270:
269:
267:
266:
264:
263:
262:
261:
260:
259:
249:
237:
234:
232:
227:
218:
217:
216:
210:
209:
202:
201:
200:
196:
192:
187:
186:
185:
184:
183:
182:
179:
176:
171:
170:
167:
163:
159:
158:Exploding Boy
154:
153:
152:
151:
147:
143:
134:
130:
126:
122:
121:CMUResearcher
118:
114:
110:
105:
104:
103:
102:
99:
95:
91:
82:
76:
72:
68:
67:CMUResearcher
63:
62:
61:
60:
57:
53:
49:
48:Exploding Boy
45:
44:
43:
42:
39:
35:
30:
20:Our talk page
19:
17:
823:
820:
817:
814:
811:
771:
731:
704:
698:
683:
671:
653:
650:
646:
642:
620:
614:
520:
517:
513:
489:
465:
460:
360:
350:
314:
294:
256:
255:
253:
250:Greetings...
221:
206:
138:
109:Robert Kraut
86:
83:Who are you?
25:
23:
15:
782:WP:Research
778:PiperNigrum
763:WP:Research
674:Wikiversity
329:Ottava Rima
191:Majoreditor
117:Moira Burke
113:Niki Kittur
812:Hi Aaron,
678:Colloquium
90:Marty Rimm
835:kbattick
797:EpochFail
684:SB_Johnny
621:SB_Johnny
570:Sebastian
523:R27182818
317:Sebastian
298:DThomsen8
273:DThomsen8
175:Sebastian
825:Kbattick
716:Chequers
415:new here
37:Chequers
470:Protonk
420:Protonk
355:WP:SPAM
142:Protonk
493:hmwith
225:hmwith
208:Jeremy
94:Calton
712:Spiel
351:again
33:Spiel
829:talk
745:POCH
707:Ϣere
659:talk
592:talk
543:poch
527:talk
474:talk
443:talk
424:talk
396:talk
362:Ched
333:talk
302:talk
277:talk
195:talk
162:talk
146:talk
125:talk
98:Talk
71:talk
52:talk
28:Ϣere
789:rfc
776:.
767:RFC
749:AIL
688:|
625:|
551:ail
466:and
461:and
831:)
792:}}
786:{{
661:)
615:am
594:)
529:)
476:)
445:)
426:)
398:)
368:?
365::
335:)
304:)
296:--
279:)
197:)
164:)
148:)
127:)
96:|
73:)
54:)
827:(
747:F
743:E
657:(
590:(
547:F
539:E
525:(
498:t
472:(
441:(
422:(
394:(
331:(
300:(
275:(
230:t
193:(
160:(
144:(
123:(
69:(
50:(
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.