292:(NCLB) was implemented to hold schools accountable for the reported standardized, summative assessment metrics. However, schools already had active internal accountability systems that place high emphasis on an ongoing cycle of instructional improvement based on the use of data including formative assessment results and behavioral information. Therefore, when the high-stakes accountability was implemented, schools naturally go through process of alignment between different types of data different purposes and the corresponding tension. Richard Halverson and his colleagues, employing case study approaches, explore leaders’ effort of coordination and alignment process which occurs between extant “central practices and cultures of schools” and “new accountability pressure” in a pursuit of improving student achievement score.
472:
needs, leading to better achievement”. Summative assessments should not be used to inform the day-to-day teaching and learning that is supported by data-driven instruction. Additional problems associated with perceptions of data driven instruction include the limitations of quantitative data to represent student learning, not considering the social and emotional needs or the context of the data when making instructional decisions, and a hyperfocus on the core areas of literacy and mathematics while ignoring the encore, traditionally high-interest areas such as the arts and humanities.
346:
textbooks and experiments, or programs such as individualized education programs (Intervention). The effect of these intervention artifacts can be evaluated through formative assessments, either commercial or self-created, from the perspective that they had brought intended changes to teaching and learning (Assessment). In the actuation space, educators interpret the assessment results in relation to the initial goals of the intervention, and discuss how to modify the instruction delivery or assessment as measurement tools, which lays groundwork for the new interventions (Actuation).
224:
classroom assessment that is formative in nature, including student observations and discussions, understand pupils’ needs and challenges, and looking at student work. Conversely, summative assessments are designed to determine whether or not a student can transfer their learning to new contexts, as well as for accountability purposes. Formative assessment is the use of information made evident during instruction in order to improve student progress and performance. Summative assessments occur after teaching and learning occurred.
310:
non-academic data like student demographic information, community survey data, curricula, technological capacity, and behavioral records. In order to store such data, some schools develop their own local collection strategies using low-tech printouts and notebooks, whereas other schools rely on high-tech district storage systems, which provide tremendous amounts of reports. School leaders have discussions around which data needs to be reported and how to report the data in a way that they can use to guide teaching practices.
376:“The first is data collection and analysis. Districts and schools must carefully consider what data they need to collect, develop instruments with which to collect the data, and make the data available as soon as possible. The second component is data use. Principals and district leaders must give teachers sufficient time and training to understand the data and learn how to respond to what the data reveal”.
22:
355:(curriculum-embedded activities), assist students to practice or be accustomed to test-taking with similar types of tests (test practice), and establish a favorable test-taking environment (environmental design). Further, teachers communicate with parents and the community members on the topics ranging from test implementation to interpreting the test results (community outreach).
484:
188:. Data-driven instruction works on two levels. One, it provides teachers the ability to be more responsive to students’ needs, and two, it allows students to be in charge of their own learning. Data-driven instruction can be understood through examination of its history, how it is used in the classroom, its attributes, and examples from teachers using this process.
124:
337:
variety of professional development opportunities and coaching focused on professional interaction (faculty-based programs). In addition, educators modify their curriculum as a whole-classroom approach (curriculum-based programs) or develop customized instructional plans taking into account individual students’ needs (student-based programs).
411:
data analysis by students into classroom practices, it requires time, training, and action. The strategies that students use to evaluate their own learning vary in effectiveness. In a meta-analysis, Dunlosky, Rawson, Marsh, Nathan & Willingham ranked ten learning strategies based on the projected impact each would have on achievement:
1259:
Stamper, J., Ed, Pardos, Z., Ed, Mavrikis, M., Ed, McLaren, B. M., Ed, & International
Educational Data Mining Society. (2014). Proceedings of the Seventh International Conference on Educational Data Mining (EDM) (7th, London, United Kingdom, July 4–7, 2014). International Educational Data Mining
354:
This function is not intended for teachers to “teach to the test.” Rather, it points to the following activities: curriculum-embedded activities, test practice, environmental design, and community outreach. Teachers incorporate the content of standardized assessment into their day-to-day instructions
327:
Richard
Halverson and his colleagues states that program alignment function refers to “link the relevant content and performance standards with the actual content taught in classroom.” For example, the benchmark assessment results, as “problem-finding tools,” help educators to identify the curricular
287:
Data-Driven
Instructional Systems refers to a comprehensive system of structures that school leaders and teachers design in order to incorporate the data into their instructions. Building on organizational and school change literature, Richard Halverson, Jeffrey Grigg, Reid Prichett, and Chris Thomas
397:
Data-driven instruction is created and implemented in the classroom. Teachers have the most direct link between student performance and classroom practices. Through the use of data, teachers can make decisions about what and how to teach including how to use time in class, interventions for students
345:
Educators interact with each other around the formative feedback on the local interventions implemented across classrooms and programs. Formative feedback systems are made of three main components: intervention, assessment, and actuation. Intervention artifacts here include curriculum materials like
318:
In the DDIS model, data reflection refers to collectively making sense of the reported data. District-level data retreats provide key opportunities for the schools within districts to identify the school-level strengths and weaknesses in terms of achievement data. Retreats help districts to develop
200:
in education. From the idea of creating standards comes accountability, the idea that schools should report on their ability to meet the designated standards. Late in the last century and in the early 2000s, an increased emphasis on accountability in public organizations made its way into the realm
410:
A significant new growth in data-driven instruction is in having students shape their lessons using data about their own progress. Younger learners who are able to self-report regarding grades and other assessments can experience high levels of achievement and progress within instruction. To embed
401:
To be able to engage in data-driven instruction, teachers must first develop the knowledge, skills, and dispositions required. Working in a school culture and climate in which data-driven instruction is valued and supported, teachers have the ability to increase student achievement and potentially
388:
Schools have a major role in establishing the conditions for data-driven instruction to flourish. Heppen, et al. indicate a need for a clear and consistent focus on using data and a data-rich environment to support teachers’ efforts to use data to drive instruction. When the leadership creates and
269:
Because of the lack of timely feedback regarding the results plus the inability to personalize the approach, summative assessments are not readily used for data driven instruction in the classroom. Instead, a variety of information gleaned from different forms of assessments should be used to make
462:
It is worth noting that the less effective strategies may be more commonly used in K-12 classrooms than the moderately effective and highly effective strategies. The authors suggest that students should be taught how to use more effective techniques and when they are most helpful in guiding their
248:
that tease out this data can be defined as assessment literacy. Building assessment literacy also includes knowing when to use which type of assessment and the resulting data to use to inform instruction. The purpose of data driven instruction is to use information to guide teaching and learning.
219:
Data in the classroom is any information that is visible during instruction that could be used to inform teaching and learning. Types of data include quantitative and qualitative data, although quantitative data is most often used for data-driven instruction. Examples of quantitative data include
471:
A major criticism of data driven instruction is that it focuses too much on test scores, and that not enough attention is given to the results of classroom assessments. Data driven instruction should serve as a “road map through assessment” that helps “teachers plan instruction to meet students’
205:
in 2001 came laws requiring schools to provide information to the public concerning the quality of education provided to students. To be able to provide such data, states were mandated to create accountability measures and yearly assessments to gauge the effectiveness of schools in meeting those
336:
After identifying the main areas in relation to students learning needs and school goals, leaders and teachers design interventions: faculty-based programs; curriculum-based programs; and student-based programs. In an effort to improve the faculty’s data literacy, educators are provided with a
402:
reduce the achievement gap. Additionally, teachers must have access to learning opportunities or professional development which helps them understand the pedagogical framework and technical skills required to obtain, analyze, and use information about students to make instructional decisions.
223:
Quantitative and qualitative data is generally captured through two forms of assessments: formative and summative. Formative assessment is the information that is revealed and shared during instruction and is actionable by the teacher or student. Paul Black and Dylan Wiliam offer examples of
368:
The primary implication for school districts is in ensuring high quality and relevant data is gathered and available. Beyond creating systems to gather and share the data, the school district must provide the expertise, in the form of data expert personnel and/or the access to professional
309:
Data acquisition includes the data collection, data storage, and data reporting functions. “Data” in DDIS model is broadly conceptualized as any type of information that guides teaching and learning. In practice, schools collect academic data standardized assessment test scores, as well as
389:
maintains an environment which promotes collaboration and clearly communicates the urgency to improve student learning, teachers feel supported to engage in data use. The additional scaffold of modeling the use of data at the school level increases teachers’ expertise in the use of data.
372:
Another critical component of the responsibility of the district is to provide the leadership and vision to promulgate the use of information about student performance to improve teaching practice. Zavadsky and Dolejs suggest two areas for school districts to consider:
210:
further pushed states to use data gathering and reporting to demonstrate school’s ability to meet the demands of the public. Embedded in both NCLB and the Race to the Top Act is an assumption that the collection and use of data can lead to increased student performance.
300:
In their article, Richard
Halverson, Jeffrey Grigg, Reid Prichett, and Chris Thomas suggest that the DDIS framework is composed of six organizational functions: data acquisition; data reflection; program alignment; program design; formative feedback; test preparation.
1252:
Shanahan, T., Callison, K., Carriere, C., Duke, N. K., Pearson, P. D., Schatschneider, C., & Torgesen, J. (2010). Improving
Reading Comprehension in Kindergarten through 3rd Grade: IES Practice Guide. NCEE 2010-4038. What Works Clearinghouse. Retrieved from
183:
is an educational approach that relies on information to inform teaching and learning. The idea refers to a method teachers use to improve instruction by looking at the information they have about their students. It takes place within the classroom, compared to
1079:
Heppen, J., Faria, A.-M., Thomsen, K., Sawyer, K., Townsend, M., Kutner, M., ... Casserly, M. (2010). Using Data to
Improve Instruction in the Great City Schools: Key Dimensions of Practice. Urban Data Study. Council of the Great City Schools. Retrieved from
379:
While the literature shows the vital importance of the role of the district in setting the stage for data driven instruction, more of the work of connecting student performance to classroom practices happens at the school and classroom level.
1030:
Eagle, M., Corbett, A., Stamper, J., McLaren, B. M., Baker, R., Wagner, A., ... Mitchell, A. (2016). Predicting
Individual Differences for Learner Modeling in Intelligent Tutors from Previous Learner Activities (pp. 55–63). ACM Press.
463:
learning. When these strategies become internalized, students will have developed techniques in order to learn how to learn. This is critical as they move into the secondary level and are expected to be more independent in their studies.
319:
district-level visions for instruction. In contrast, through local data reflection meetings, teachers have conversations focused on the individual students’ progress by examining each student’s performance on the assessed standards.
220:
test scores, results on a quiz, and levels of performance on a periodic assessment. Examples of qualitative data include field notes, student work/artifacts, interviews, focus groups, digital pictures, video, reflective journals.
993:
Dunlosky, J.; Rawson, K. A.; Marsh, E. J.; Nathan, M. J.; Willingham, D. T. (2013). "Improving students' learning with effective learning techniques promising directions from cognitive and educational psychology".
900:
Dunlosky, J.; Rawson, K. A.; Marsh, E. J.; Nathan, M. J.; Willingham, D. T. (2013). "Improving students' learning with effective learning techniques promising directions from cognitive and educational psychology".
288:
developed a DDIS framework in an attempt to describe how relevant actors manage school-level internal accountability to external accountability. Specifically, high-stakes external accountability policies such as
1069:
Hamilton, L., Halverson, R., Jackson, S. S., Mandinach, E., Supovitz, J. A., Wayman, J. C., ... Steele, J. L. (2009). Using student achievement data to support instructional decision making. Retrieved from
986:
Dana, N. F. & Yendol-Hoppey, D. (2014). The
Reflective Educator’s Guide to Classroom Research: Learning to Teach and Teaching to Learn Through Practitioner Inquiry (3rd Ed.). Thousand Oaks, CA: Corwin.
1060:
Halverson, R., Grigg, J., Prichett, R., & Thomas, C. (2007). The New
Instructional Leadership: Creating Data-Driven Instructional Systems in School. Journal of School Leadership, 17(March), 159–194.
270:
decisions about student progress and performance within data-driven instruction. The use of multiple measures of different forms and at different times to make instructional decisions is referred to as
983:
Boudett, K. P., City, E. A., Murname, R. J. (2013). Data Wise: A Step-by-Step Guide to Using
Assessment Results to Improve Teaching and Learning. Cambridge, MA: Harvard Education Press.
398:
who are not meeting standards, customizing lessons based on real-time information, adapting teaching practice to align to student needs, and making changes to pace, scope and sequence.
816:
Heppen, Jessica; Faria, Ann-Marie; Thomsen, Kerri; Sawyer, Katherine; Townsend, Monika; Kutner, Melissa; Stachel, Suzanne; Lewis, Sharon; Casserly, Michael (December 2010).
149:
1188:
Pella, S (2012). "What Should Count as Data for Data-Driven Instruction? Toward Contextualized Data-Inquiry Models for Teacher Education and Professional Development".
945:
1208:
715:
39:
1269:"Examining data driven decision making via formative assessment: A confluence of technology, data interpretation heuristics and curricular policy"
1200:
Rogers, L. N., & Tyndall, P. D. (2001). Teachers’ Perspectives: Developing Instructional Leadership through Classroom Inquiry. Retrieved from
776:"Examining data driven decision making via formative assessment: A confluence of technology, data interpretation heuristics and curricular policy"
1249:
Schmoker, M. (1996). Results: the key to continuous school improvement. Alexandria, VA: Association for Supervision and Curriculum Development.
134:
592:
Kennedy, Brianna L.; Datnow, Amanda (December 2011). "Student Involvement and Data-Driven Decision Making: Developing a New Typology".
86:
1286:
Zavadsky, H., & Dolejs, A. (2006). DATA: Not Just Another Four-Letter Word. Principal Leadership, Middle Level Ed., 7(2), 32–36.
167:
105:
58:
65:
196:
Prior to the current emphasis on data and accountability in schools, some school leaders and education researchers focused on
145:
271:
43:
649:
The Reflective Educator's Guide to Classroom Research: Learning to Teach and Teaching to Learn Through Practitioner Inquiry
1167:
Mokhtari, K.; Rosemary, C. A.; Edwards, P. A. (2007). "Making Instructional Decisions Based on Data: What, How, and Why".
185:
72:
687:
1106:
Kennedy, B. L.; Datnow, A. (2011). "Student Involvement and Data-Driven Decision Making Developing a New Typology".
54:
32:
1239:
746:
429:
1041:
Elmore, R. F. (2000). Building a new structure for school leadership. Albert Shanker Institute. Retrieved from
497:
289:
202:
1163:
Melucci, L. (2013, August). TEACHER PERCEPTIONS AND USE OF DATA-DRIVEN INSTRUCTION.pdf. Capella University.
197:
819:
Using Data to Improve Instruction in the Great City Schools: Key Dimensions of Practice. Urban Data Study
1063:
Hamilton et al. - 2009 - Using student achievement data to support instruct.pdf. (n.d.). Retrieved from
850:
832:
Hamilton et al. - 2009 - Using student achievement data to support instruct.pdf. (n.d.). Retrieved from
539:
971:
Black, P.; Wiliam, D. (1998). "Inside the Black Box: Raising Standards Through Classroom Assessment".
1086:
Johnson, L. (2009). Randi Weingarten, President Antonia Cortese, Secretary-Treasurer. Retrieved from
1076:
Hattie, J. (2012). Visible Learning for Teachers: Maximizing Impact on Learning. New York: Routledge.
667:
Black, P; Wiliam, D. (1998). "Inside the Black Box: Raising Standards Through Classroom Assessment".
245:
241:
237:
79:
1231:
1152:
1123:
1019:
926:
738:
609:
574:
1299:
1011:
918:
369:
development resources to ensure school building leaders are able to access and use the data.
1223:
1176:
1144:
1115:
1032:
1003:
910:
730:
634:
Data Wise: A Step-by-Step Guide to Using Assessment Results to Improve Teaching and Learning
601:
566:
489:
1209:"The Role of Schooling in Perpetuating Educational Inequality An International Perspective"
847:
Driving classroom instruction with data: From the district to the teachers to the classroom
989:
Datnow, A., & Park, V. (2014). Data-Driven Leadership. San Francisco, CA: Jossey-Bass.
207:
716:"The New Instructional Leadership: Creating Data-Driven Instructional Systems in School"
233:
1135:
Larocque, M (2007). "Closing the Achievement Gap: The Experience of a Middle School".
884:
557:
Larocque, M (2007). "Closing the Achievement Gap: The Experience of a Middle School".
455:
Imagery use for text learning - attempt to form mental images of the things being read
1293:
1156:
1127:
613:
578:
1235:
742:
249:
Dylan Wiliam offers examples of data driven instruction using formative assessment:
1023:
930:
435:
Self-explanation - explaining how new information relates to what is already known
1283:
Wiliam, D. (2011). Embedded Formative Assessment. Bloomington, IN: Solution Tree.
1064:
833:
543:
854:
714:
Halverson, Richard; Grigg, Jeffrey; Prichett, Reid; Thomas, Chris (March 2007).
21:
734:
438:
Interleaved practice - mixing different kinds of problems in a practice session
1148:
886:
Letting Data Lead: How to Design, Analyze, and Respond to Classroom Assessment
570:
479:
1227:
1119:
1007:
914:
605:
328:
standards that are not aligned well with the current instructional programs.
1071:
1036:
797:
Zavadsky, H.; Dolejs, A. (2006). "DATA: Not Just Another Four-Letter Word".
1093:
Jones, A (2005). "The Myths of Data-Driven Schools. Principal Leadership".
1015:
922:
483:
421:
Distributed practice - repetitive practice over specific intervals of time
1261:
1180:
253:
Clarifying, sharing, and understanding learning intentions and criteria
1268:
1207:
Schmidt, W. H.; Burroughs, N. A.; Zoido, P.; Houang, R. T. (2015).
775:
536:
Data-driven decision making: Teachers' use of data in the classroom
1254:
1201:
1087:
1081:
1042:
817:
518:
446:
Summarization - writing summaries of the material being learned
262:
Activating students as instructional resources for one another
117:
15:
870:
Visible Learning for Teachers: Maximizing Impact on Learning
762:
206:
measures. Following NCLB, more recent legislation under the
418:
Practice testing - self-testing, solving practice problems
1273:
Contemporary Issues in Technology and Teacher Education
780:
Contemporary Issues in Technology and Teacher Education
449:
Highlighting - marking parts of text that are important
141:
709:
707:
705:
703:
701:
699:
458:
Rereading - read material again after initial reading
632:
Boudett, K. P.; City, E. A.; Murname, R. J. (2013).
265:
Activating students as owners of their own learning
46:. Unsourced material may be challenged and removed.
946:"Code Red: The Danger of Data-Driven Instruction"
1065:http://files.eric.ed.gov/fulltext/ED506645.pdf
889:. Bloomington, IN: Solution Tree. p. 106.
834:http://files.eric.ed.gov/fulltext/ED506645.pdf
520:Building a New Structure for School Leadership
452:Keyword mnemonic - various mnemonic techniques
259:Providing feedback that moves learning forward
8:
996:Psychological Science in the Public Interest
903:Psychological Science in the Public Interest
845:Furlong-Gordon, Jean Marie (November 2009).
256:Eliciting evidence of learners’ achievement
651:(3rd ed.). Thousand Oaks, CA: Corwin.
1072:http://repository.upenn.edu/gse_pubs/279/
636:. Cambridge, MA: Harvard Education Press.
168:Learn how and when to remove this message
106:Learn how and when to remove this message
647:Dana, N. F.; Yendol-Hoppey, D. (2014).
509:
232:Understanding the differences between
201:of education. With the passing of the
799:Principal Leadership, Middle Level Ed
7:
1262:http://www.educationaldatamining.org
822:. Council of the Great City Schools.
662:
660:
658:
627:
625:
623:
135:research paper or scientific journal
44:adding citations to reliable sources
1048:Gold, S (2005). "DRIVEN by DATA".
944:Neuman, Susan B. (November 2016).
14:
692:. Bloomington, IN: Solution Tree.
425:Moderately Effective Strategies:
278:Data-Driven Instructional Systems
883:Depka, Eileen (March 29, 2019).
482:
122:
20:
1255:http://eric.ed.gov/?id=ED512029
1202:http://eric.ed.gov/?id=ED465596
1088:http://eric.ed.gov/?id=ED511575
1082:http://eric.ed.gov/?id=ED536737
1043:http://eric.ed.gov/?id=ED546618
534:Moriarty, Tammy Wu (May 2013).
203:No Child Left Behind (NCLB) Act
31:needs additional citations for
1190:Middle Grades Research Journal
1:
689:Embedded Formative Assessment
414:Highly Effective Strategies:
1267:Swan, G.; Mazur, J. (2011).
774:Swan, G.; Mazur, J. (2011).
723:Journal of School Leadership
523:. Albert Shanker Institute.
517:Elmore, Richard F. (2000).
442:Less Effective Strategies:
186:data-driven decision making
1316:
735:10.1177/105268460701700202
498:Evidence-based instruction
1149:10.3200/tchs.80.4.157-162
1050:Technology & Learning
571:10.3200/tchs.80.4.157-162
430:Elaborative interrogation
55:"Data-driven instruction"
1228:10.3102/0013189x15603982
1120:10.1177/0044118X10388219
1008:10.1177/1529100612453266
915:10.1177/1529100612453266
606:10.1177/0044118X10388219
290:No Child Left Behind Act
150:overly technical phrases
142:help improve the article
1037:10.1145/2930238.2930255
181:Data-driven instruction
1216:Educational Researcher
950:Educational Leadership
872:. New York: Routledge.
686:Wiliam, Dylan (2011).
432:- explaining the 'why'
283:Background and origins
198:standards-based reform
965:Additional References
763:George Kocher Expert
364:For school districts
246:summative assessment
242:formative assessment
40:improve this article
1169:The Reading Teacher
1108:Youth & Society
868:Hattie, J. (2012).
594:Youth & Society
208:Race to the Top Act
144:by rewriting it in
1181:10.1598/rt.61.4.10
341:Formative Feedback
146:encyclopedic style
133:is written like a
323:Program Alignment
240:data, as well as
178:
177:
170:
116:
115:
108:
90:
1307:
1280:
1246:
1244:
1238:. Archived from
1213:
1197:
1184:
1160:
1131:
1114:(4): 1246–1271.
1102:
1057:
1027:
980:
973:Phi Delta Kappan
958:
957:
941:
935:
934:
897:
891:
890:
880:
874:
873:
865:
859:
858:
842:
836:
830:
824:
823:
813:
807:
806:
794:
788:
787:
771:
765:
760:
754:
753:
751:
745:. Archived from
720:
711:
694:
693:
683:
677:
676:
669:Phi Delta Kappan
664:
653:
652:
644:
638:
637:
629:
618:
617:
600:(4): 1246–1271.
589:
583:
582:
554:
548:
547:
531:
525:
524:
514:
492:
490:Education portal
487:
486:
350:Test Preparation
305:Data Acquisition
173:
166:
162:
159:
153:
126:
125:
118:
111:
104:
100:
97:
91:
89:
48:
24:
16:
1315:
1314:
1310:
1309:
1308:
1306:
1305:
1304:
1290:
1289:
1266:
1242:
1211:
1206:
1187:
1166:
1134:
1105:
1095:Middle Level Ed
1092:
1047:
992:
970:
967:
962:
961:
943:
942:
938:
899:
898:
894:
882:
881:
877:
867:
866:
862:
844:
843:
839:
831:
827:
815:
814:
810:
796:
795:
791:
773:
772:
768:
761:
757:
749:
718:
713:
712:
697:
685:
684:
680:
666:
665:
656:
646:
645:
641:
631:
630:
621:
591:
590:
586:
556:
555:
551:
533:
532:
528:
516:
515:
511:
506:
488:
481:
478:
469:
408:
395:
386:
366:
361:
352:
343:
334:
325:
316:
314:Data Reflection
307:
298:
285:
280:
230:
217:
194:
174:
163:
157:
154:
139:
127:
123:
112:
101:
95:
92:
49:
47:
37:
25:
12:
11:
5:
1313:
1311:
1303:
1302:
1292:
1291:
1288:
1287:
1284:
1281:
1264:
1257:
1250:
1247:
1245:on 2020-02-19.
1222:(7): 371–386.
1204:
1198:
1185:
1175:(4): 354–359.
1164:
1161:
1143:(4): 157–162.
1137:Clearing House
1132:
1103:
1090:
1084:
1077:
1074:
1067:
1061:
1058:
1045:
1039:
1028:
990:
987:
984:
981:
966:
963:
960:
959:
936:
892:
875:
860:
837:
825:
808:
789:
766:
755:
752:on 2020-02-11.
729:(2): 159–194.
695:
678:
654:
639:
619:
584:
565:(4): 157–162.
559:Clearing House
549:
526:
508:
507:
505:
502:
501:
500:
494:
493:
477:
474:
468:
465:
460:
459:
456:
453:
450:
447:
440:
439:
436:
433:
423:
422:
419:
407:
404:
394:
391:
385:
382:
365:
362:
360:
357:
351:
348:
342:
339:
333:
332:Program Design
330:
324:
321:
315:
312:
306:
303:
297:
294:
284:
281:
279:
276:
267:
266:
263:
260:
257:
254:
229:
226:
216:
213:
193:
190:
176:
175:
130:
128:
121:
114:
113:
28:
26:
19:
13:
10:
9:
6:
4:
3:
2:
1312:
1301:
1298:
1297:
1295:
1285:
1282:
1278:
1274:
1270:
1265:
1263:
1258:
1256:
1251:
1248:
1241:
1237:
1233:
1229:
1225:
1221:
1217:
1210:
1205:
1203:
1199:
1195:
1191:
1186:
1182:
1178:
1174:
1170:
1165:
1162:
1158:
1154:
1150:
1146:
1142:
1138:
1133:
1129:
1125:
1121:
1117:
1113:
1109:
1104:
1100:
1096:
1091:
1089:
1085:
1083:
1078:
1075:
1073:
1068:
1066:
1062:
1059:
1056:(11): 6, 8–9.
1055:
1051:
1046:
1044:
1040:
1038:
1034:
1029:
1025:
1021:
1017:
1013:
1009:
1005:
1001:
997:
991:
988:
985:
982:
979:(2): 139–148.
978:
974:
969:
968:
964:
955:
951:
947:
940:
937:
932:
928:
924:
920:
916:
912:
908:
904:
896:
893:
888:
887:
879:
876:
871:
864:
861:
856:
852:
848:
841:
838:
835:
829:
826:
821:
820:
812:
809:
804:
800:
793:
790:
785:
781:
777:
770:
767:
764:
759:
756:
748:
744:
740:
736:
732:
728:
724:
717:
710:
708:
706:
704:
702:
700:
696:
691:
690:
682:
679:
675:(2): 139–148.
674:
670:
663:
661:
659:
655:
650:
643:
640:
635:
628:
626:
624:
620:
615:
611:
607:
603:
599:
595:
588:
585:
580:
576:
572:
568:
564:
560:
553:
550:
545:
541:
537:
530:
527:
522:
521:
513:
510:
503:
499:
496:
495:
491:
485:
480:
475:
473:
466:
464:
457:
454:
451:
448:
445:
444:
443:
437:
434:
431:
428:
427:
426:
420:
417:
416:
415:
412:
405:
403:
399:
392:
390:
383:
381:
377:
374:
370:
363:
358:
356:
349:
347:
340:
338:
331:
329:
322:
320:
313:
311:
304:
302:
295:
293:
291:
282:
277:
275:
273:
272:triangulation
264:
261:
258:
255:
252:
251:
250:
247:
243:
239:
235:
227:
225:
221:
214:
212:
209:
204:
199:
191:
189:
187:
182:
172:
169:
161:
158:February 2017
151:
148:and simplify
147:
143:
137:
136:
131:This article
129:
120:
119:
110:
107:
99:
88:
85:
81:
78:
74:
71:
67:
64:
60:
57: –
56:
52:
51:Find sources:
45:
41:
35:
34:
29:This article
27:
23:
18:
17:
1276:
1272:
1240:the original
1219:
1215:
1193:
1189:
1172:
1168:
1140:
1136:
1111:
1107:
1098:
1094:
1053:
1049:
999:
995:
976:
972:
953:
949:
939:
906:
902:
895:
885:
878:
869:
863:
846:
840:
828:
818:
811:
802:
798:
792:
783:
779:
769:
758:
747:the original
726:
722:
688:
681:
672:
668:
648:
642:
633:
597:
593:
587:
562:
558:
552:
535:
529:
519:
512:
470:
461:
441:
424:
413:
409:
406:For students
400:
396:
393:For teachers
387:
378:
375:
371:
367:
359:Implications
353:
344:
335:
326:
317:
308:
299:
296:Key concepts
286:
268:
234:quantitative
231:
222:
218:
195:
180:
179:
164:
155:
132:
102:
93:
83:
76:
69:
62:
50:
38:Please help
33:verification
30:
1196:(1): 57–75.
1101:(2): 37–39.
1002:(1): 4–58.
956:(3): 24–29.
909:(1): 4–58.
805:(2): 32–36.
384:For schools
238:qualitative
1260:Society.
849:(Thesis).
544:1432373944
538:(Thesis).
467:Criticisms
215:Attributes
66:newspapers
1279:(2): 205.
1157:145741309
1128:145417758
855:250914319
786:(2): 205.
614:145417758
579:145741309
504:Citations
236:data vs.
1300:Teaching
1294:Category
1236:96446917
1016:26173288
923:26173288
851:ProQuest
743:61185505
540:ProQuest
476:See also
228:Examples
96:May 2017
1024:1621081
931:1621081
192:History
140:Please
80:scholar
1234:
1155:
1126:
1022:
1014:
929:
921:
853:
741:
612:
577:
542:
82:
75:
68:
61:
53:
1243:(PDF)
1232:S2CID
1212:(PDF)
1153:S2CID
1124:S2CID
1020:S2CID
927:S2CID
750:(PDF)
739:S2CID
719:(PDF)
610:S2CID
575:S2CID
87:JSTOR
73:books
1012:PMID
919:PMID
244:vs.
59:news
1224:doi
1177:doi
1145:doi
1116:doi
1033:doi
1004:doi
911:doi
731:doi
602:doi
567:doi
42:by
1296::
1277:11
1275:.
1271:.
1230:.
1220:44
1218:.
1214:.
1192:.
1173:61
1171:.
1151:.
1141:80
1139:.
1122:.
1112:43
1110:.
1097:.
1054:25
1052:.
1018:.
1010:.
1000:14
998:.
977:80
975:.
954:74
952:.
948:.
925:.
917:.
907:14
905:.
801:.
784:11
782:.
778:.
737:.
727:17
725:.
721:.
698:^
673:80
671:.
657:^
622:^
608:.
598:43
596:.
573:.
563:80
561:.
274:.
1226::
1194:7
1183:.
1179::
1159:.
1147::
1130:.
1118::
1099:6
1035::
1026:.
1006::
933:.
913::
857:.
803:7
733::
616:.
604::
581:.
569::
546:.
171:)
165:(
160:)
156:(
152:.
138:.
109:)
103:(
98:)
94:(
84:·
77:·
70:·
63:·
36:.
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.
