116:. Some of the major criticisms of the study have been that five of the eight datasets consisted of paragraphs rather than essays, four of the eight data sets were graded by human readers for content only rather than for writing ability, and that rather than measuring human readers and the AES machines against the "true score", the average of the two readers' scores, the study employed an artificial construct, the "resolved score", which in four datasets consisted of the higher of the two human scores if there was a disagreement. This last practice, in particular, gave the machines an unfair advantage by allowing them to round up for these datasets.
108:
The competition also hosted a separate demonstration among nine AES vendors on a subset of the ASAP data. Although the investigators reported that the automated essay scoring was as reliable as human scoring, this claim was not substantiated by any statistical tests because some of the vendors required that no such tests be performed as a precondition for their participation. Moreover, the claim that the
Hewlett Study demonstrated that AES can be as reliable as human raters has since been strongly contested, including by
189:
the prompt's topic, locations of argument components (major claim, claim, premise), errors in the arguments, cohesion in the arguments among various other features. In contrast to the other models mentioned above, this model is closer in duplicating human insight while grading essays. Due to the growing popularity of deep neural networks, deep learning approaches have been adopted for automated essay scoring, generally obtaining superior results, often surpassing inter-human agreement levels.
281:. mention "the over-reliance on surface features of responses, the insensitivity to the content of responses and to creativity, and the vulnerability to new types of cheating and test-taking strategies." Several critics are concerned that students' motivation will be diminished if they know that no human will read their writing. Among the most telling critiques are reports of intentionally gibberish essays being given high scores.
63:. In 1966, he argued for the possibility of scoring essays by computer, and in 1968 he published his successful work with a program called Project Essay Grade (PEG). Using the technology of that time, computerized essay scoring would not have been cost-effective, so Page abated his efforts for about two decades. Eventually, Page sold PEG to
208:
The automated essay scoring task has also been studied in the cross-domain setting using machine learning models, where the models are trained on essays written for one prompt (topic) and tested on essays written for another prompt. Successful approaches in the cross-domain scenario are based on deep
217:
Any method of assessment must be judged on validity, fairness, and reliability. An instrument is valid if it actually measures the trait that it purports to measure. It is fair if it does not, in effect, penalize or privilege any one class of people. It is reliable if its outcome is repeatable, even
119:
In 1966, Page hypothesized that, in the future, the computer-based judge will be better correlated with each human judge than the other human judges are. Despite criticizing the applicability of this approach to essay marking in general, this hypothesis was supported for marking free text answers to
256:
Percent agreement is a simple statistic applicable to grading scales with scores from 1 to n, where usually 4 ā¤ n ā¤ 6. It is reported as three figures, each a percent of the total number of essays scored: exact agreement (the two raters gave the essay the same score), adjacent agreement (the raters
188:
Recently, one such mathematical model was created by Isaac
Persing and Vincent Ng. which not only evaluates essays on the above features, but also on their argument strength. It evaluates various features of the essay, such as the agreement level of the author and reasons for the same, adherence to
107:
called the
Automated Student Assessment Prize (ASAP). 201 challenge participants attempted to predict, using AES, the scores that human raters would give to thousands of essays written to eight different prompts. The intent was to demonstrate that AES can be as reliable as human raters, or more so.
92:
Under the leadership of Howard Mitzel and Sue
Lottridge, Pacific Metrics developed a constructed response automated scoring engine, CRASE. Currently utilized by several state departments of education and in a U.S. Department of Education-funded Enhanced Assessment Grant, Pacific Metricsā technology
260:
Inter-rater agreement can now be applied to measuring the computer's performance. A set of essays is given to two human raters and an AES program. If the computer-assigned scores agree with one of the human raters as well as the raters agree with each other, the AES program is considered reliable.
184:
From the beginning, the basic procedure for AES has been to start with a training set of essays that have been carefully hand-scored. The program evaluates surface features of the text of each essay, such as the total number of words, the number of subordinate clauses, or the ratio of uppercase to
132:
of answers showed that excellent papers and weak papers formed well-defined clusters, and the automated marking rule for these clusters worked well, whereas marks given by human teachers for the third cluster ('mixed') can be controversial, and the reliability of any assessment of works from the
38:
Several factors have contributed to a growing interest in AES. Among them are cost, accountability, standards, and technology. Rising education costs have led to pressure to hold the educational system accountable for results by imposing standards. The advance of information technology promises to
306:
In a detailed summary of research on AES, the petition site notes, "RESEARCH FINDINGS SHOW THAT no oneāstudents, parents, teachers, employers, administrators, legislatorsācan rely on machine scoring of essays ... AND THAT machine scoring does not measure, and therefore does not promote, authentic
264:
Some researchers have reported that their AES systems can, in fact, do better than a human. Page made this claim for PEG in 1994. Scott Elliot said in 2003 that
IntelliMetric typically outperformed human scorers. AES machines, however, appear to be less reliable than human readers for any kind of
84:
Educational
Testing Service offers "e-rater", an automated essay scoring program. It was first used commercially in February 1999. Jill Burstein was the team leader in its development. ETS's Criterion Online Writing Evaluation Service uses the e-rater engine to provide both scores and targeted
77:
developed a system using a scoring engine called the
Intelligent Essay Assessor (IEA). IEA was first used to score essays in 1997 for their undergraduate courses. It is now a product from Pearson Educational Technologies and used for scoring within a number of commercial products and state and
69:
By 1990, desktop computers had become so powerful and so widespread that AES was a practical possibility. As early as 1982, a UNIX program called Writer's
Workbench was able to offer punctuation, spelling and grammar advice. In collaboration with several companies (notably Educational Testing
221:
Before computers entered the picture, high-stakes essays were typically given scores by two trained human raters. If the scores differed by more than one point, a more experienced third rater would settle the disagreement. In this system, there is an easy way to measure reliability: by
257:
differed by at most one point; this includes exact agreement), and extreme disagreement (the raters differed by more than two points). Expert human graders were found to achieve exact agreement on 53% to 81% of all essays, and adjacent agreement on 97% to 100%.
185:
lowercase letters—quantities that can be measured without any human insight. It then constructs a mathematical model that relates these quantities to the scores that the essays received. The same model is then applied to calculate scores of new essays.
670:
88:
Lawrence Rudner has done some work with
Bayesian scoring, and developed a system called BETSY (Bayesian Essay Test Scoring sYstem). Some of his results have been published in print or online, but no commercial system incorporates BETSY as yet.
46:
in education has generated significant backlash, with opponents pointing to research that computers cannot yet grade writing accurately and arguing that their use for such purposes promotes teaching writing in reductive ways (i.e.
31:. Its objective is to classify a large set of textual entities into a small number of discrete categories, corresponding to the possible grades, for example, the numbers 1 to 6. Therefore, it can be considered a problem of
289:
On 12 March 2013, HumanReaders.Org launched an online petition, "Professionals
Against Machine Scoring of Student Essays in High-Stakes Assessment". Within weeks, the petition gained thousands of signatures, including
192:
The various AES programs differ in what specific surface features they measure, how many essays are required in the training set, and most significantly in the mathematical modeling technique. Early attempts used
226:. If raters do not consistently agree within one point, their training may be at fault. If a rater consistently disagrees with how other raters look at the same essays, that rater probably needs extra training.
268:
In current practice, high-stakes assessments such as the GMAT are always scored by at least one human. AES is used in place of a second rater. A human rater resolves any disagreements of more than one point.
261:
Alternatively, each essay is given a "true score" by taking the average of the two human raters' scores, and the two humans and the computer are compared on the basis of their agreement with the true score.
386:- Shermis, Mark D., Jill Burstein, and Claudia Leacock (2006). "Applications of Computers in Assessment and Analysis of Writing", p. 403. In MacArthur, Charles A., Steve Graham, and Jill Fitzgerald, eds.,
653:
141:
According to a recent survey, modern AES systems try to score different dimensions of an essay's quality in order to provide feedback to users. These dimensions include the following items:
919:
Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
1484:
1644:
588:
Burstein, Jill (2003). "The E-rater(R) Scoring Engine: Automated Essay Scoring with Natural Language Processing", p. 113. In Shermis, Mark D., and Jill Burstein, eds.,
242:
2258:
1622:
360:- Larkey, Leah S., and W. Bruce Croft (2003). "A Text Categorization Approach to Automated Essay Grading", p. 55. In Shermis, Mark D., and Jill Burstein, eds.
650:
246:
2033:
1477:
1201:
1447:
1066:
Chung, Gregory K.W.K., and Eva L. Baker (2003). "Issues in the Reliability and Validity of Automated Scoring of Constructed Responses", p. 23. In:
2238:
2202:
303:
The petition describes the use of AES for high-stakes testing as "trivial", "reductive", "inaccurate", "undiagnostic", "unfair" and "secretive".
1331:
197:. Modern systems may use linear regression or other machine learning techniques often in combination with other statistical techniques such as
1305:
998:
882:
634:
2243:
1943:
1634:
1470:
100:
1285:
423:
2248:
2197:
250:
81:
IntelliMetric is Vantage Learning's AES engine. Its development began in 1996. It was first used commercially to score essays in 1998.
2233:
1804:
933:"Enhancing Automated Essay Scoring Performance via Fine-tuning Pre-trained Language Models with Combination of Regression and Ranking"
399:- Attali, Yigal, Brent Bridgeman, and Catherine Trapani (2010). "Performance of a Generic Approach in Automated Essay Scoring", p. 4.
1218:- Ben-Simon, Anat (2007). "Introduction to Automated Essay Scoring (AES)", PowerPoint presentation, Tbilisi, Georgia, September 2007.
1958:
1789:
395:
536:
1729:
373:- Keith, Timothy Z. (2003). "Validity of Automated Essay Scoring Systems", p. 153. In Shermis, Mark D., and Jill Burstein, eds.,
1099:
128:
demonstrate that the automatic systems perform well when marking by different human teachers is in good agreement. Unsupervised
2146:
1799:
409:- Wang, Jinhao, and Michelle Stallone Brown (2007). "Automated Essay Scoring Versus Human Scoring: A Comparative Study", p. 6.
615:
495:
MacDonald, N.H., L.T. Frase, P.S. Gingrich, and S.A. Keenan (1982). "The Writers Workbench: Computer Aids for Text Analysis",
1794:
1539:
1171:
1075:
597:
564:
382:
369:
356:
23:) is the use of specialized computer programs to assign grades to essays written in an educational setting. It is a form of
1229:
2063:
1784:
1254:
1351:
1756:
699:
310:
The petition specifically addresses the use of AES for high-stakes testing and says nothing about other possible uses.
2101:
2086:
2058:
1923:
1918:
1493:
555:
Elliot, Scott (2003). "Intellimetric TM: From Here to Validity", p. 75. In Shermis, Mark D., and Jill Burstein, eds.,
323:
113:
28:
729:"Critique of Mark D. Shermis & Ben Hamner, "Contrasting State-of-the-Art Automated Scoring of Essays: Analysis""
576:
1838:
1809:
1587:
32:
1681:
1534:
1028:
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
2207:
2131:
1863:
1819:
1704:
1602:
983:
Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval
198:
64:
1402:"Research Findings >> Professionals Against Machine Scoring Of Student Essays In High-Stakes Assessment"
2081:
1748:
1129:
McCurry, D. (2010). "Can machine scoring deal with broad and open writing tests as well as human readers?".
1100:"Technology and Writing Assessment: Lessons Learned from the US National Assessment of Educational Progress"
238:
223:
1198:
1968:
1661:
1639:
1629:
1597:
1572:
1444:
229:
Various statistics have been proposed to measure inter-rater agreement. Among them are percent agreement,
24:
1828:
1421:"Works Cited >> Professionals Against Machine Scoring Of Student Essays In High-Stakes Assessment"
1255:"Signatures >> Professionals Against Machine Scoring Of Student Essays In High-Stakes Assessment"
2181:
1857:
1833:
1686:
48:
347:
Page, E.B. (2003). "Project Essay Grade: PEG", p. 43. In Shermis, Mark D., and Jill Burstein, eds.,
2161:
2091:
2048:
2004:
1776:
1766:
1761:
1649:
728:
125:
43:
2171:
2043:
1908:
1671:
1654:
1512:
1049:
1031:
1004:
950:
826:
792:
631:
452:
296:
202:
515:
Page, E.B. (1994). "New Computer Grading of Student Prose, Using Modern Concepts and Software",
2176:
1888:
1696:
1607:
1071:
994:
878:
593:
560:
420:
391:
378:
365:
352:
194:
60:
1070:. Shermis, Mark D., and Jill Burstein, eds. Lawrence Erlbaum Associates, Mahwah, New Jersey,
2053:
1938:
1913:
1714:
1617:
1138:
1041:
986:
940:
868:
836:
784:
234:
129:
109:
2253:
2165:
2126:
2121:
1989:
1719:
1592:
1567:
1549:
1451:
1205:
1175:
945:
865:
Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence
657:
638:
619:
540:
427:
74:
1401:
985:. SIGIR '20. New York, NY, USA: Association for Computing Machinery. pp. 1011ā1020.
93:
has been used in large-scale formative and summative assessment environments since 2007.
1873:
1853:
1577:
1167:
Yang, Yongwei, Chad W. Buckendahl, Piotr J. Juszkiewicz, and Dennison S. Bhola (2002).
533:
2227:
2136:
1948:
1928:
1709:
1110:
1008:
954:
914:
796:
230:
96:
Measurement Inc. acquired the rights to PEG in 2002 and has continued to develop it.
2116:
1420:
1377:"Professionals Against Machine Scoring Of Student Essays In High-Stakes Assessment"
1053:
932:
612:
291:
1168:
857:
775:
Bennett, Randy E. (March 2015). "The Changing Nature of Educational Assessment".
2073:
1953:
1666:
1582:
1559:
1507:
841:
814:
1262:
931:
Yang, Ruosong; Cao, Jiannong; Wen, Zhiyuan; Wu, Youzheng; He, Xiaodong (2020).
686:
Handbook of Automated Essay Evaluation: Current Applications and New Directions
59:
Most historical summaries of AES trace the origins of the field to the work of
1676:
1462:
1142:
873:
70:
Service), Page updated PEG and ran some successful trials in the early 1990s.
788:
1544:
1030:. Melbourne, Australia: Association for Computational Linguistics: 503ā509.
990:
1045:
978:
469:
Page, E.B. (1968). "The Use of the Computer in Analyzing Student Essays",
133:'mixed' cluster can often be questioned (both human and computer-based).
2019:
1999:
1984:
1963:
1933:
1878:
1843:
1724:
456:
2156:
2014:
1994:
1868:
1612:
1527:
815:"Automatic short answer grading and feedback using text mining methods"
757:
Perelman, L. (2014). "When 'the state of the art is counting words'",
443:
Page, E. B. (1966). "The imminence of... grading essays by computer".
1522:
1517:
937:
Findings of the Association for Computational Linguistics: EMNLP 2020
104:
1376:
1023:
151:
Mechanics: following rules for spelling, punctuation, capitalization
1036:
831:
2212:
1848:
1169:"A Review of Strategies for Validating Computer-Automated Scoring"
209:
neural networks or models that combine deep and shallow features.
112:, the Norman O. Frederiksen Chair in Assessment Innovation at the
1024:"Automated essay scoring with string kernels and word embeddings"
1734:
939:. Online: Association for Computational Linguistics: 1560ā1569.
121:
1466:
977:
Cao, Yue; Jin, Hanqi; Wan, Xiaojun; Yu, Zhiwei (25 July 2020).
2009:
1150:
R. Bridgeman (2013). Shermis, Mark D.; Burstein, Jill (eds.).
1022:
Cozma, MÄdÄlina; Butnaru, Andrei; Ionescu, Radu Tudor (2018).
813:
SĆ¼zen, N.; Mirkes, E. M.; Levesley, J; Gorban, A. N. (2020).
318:
Most resources for automated essay scoring are proprietary.
1332:"Petition Against Machine Scoring Essays, HumanReaders.Org"
1230:"Facing a Robo-Grader? Just Keep Obfuscating Mellifluously"
858:"Automated Essay Scoring: A Survey of the State of the Art"
1195:
Wang, Jinhao, and Michelle Stallone Brown (2007), pp. 4-5.
1068:
Automated Essay Scoring: A Cross-Disciplinary Perspective
590:
Automated Essay Scoring: A Cross-Disciplinary Perspective
557:
Automated Essay Scoring: A Cross-Disciplinary Perspective
421:"Toward Theoretically Meaningful Automated Essay Scoring"
375:
Automated Essay Scoring: A Cross-Disciplinary Perspective
362:
Automated Essay Scoring: A Cross-Disciplinary Perspective
349:
Automated Essay Scoring: A Cross-Disciplinary Perspective
967:
Bennett, Randy Elliot, and Anat Ben-Simon (2005), p. 7.
120:
short questions, such as those typical of the British
300:, and on a number of education and technology blogs.
294:, and was cited in a number of newspapers, including
1107:
International Association for Educational Assessment
419:- Bennett, Randy Elliot, and Anat Ben-Simon (2005).
175:
Persuasiveness: convincingness of the major argument
157:
Relevance: how relevant of the content to the prompt
2190:
2145:
2100:
2072:
2032:
1977:
1899:
1887:
1818:
1775:
1747:
1695:
1558:
1500:
641:, Measurement Incorporated. Retrieved 9 March 2012.
613:"Computer Grading using Bayesian Networks-Overview"
592:. Lawrence Erlbaum Associates, Mahwah, New Jersey,
559:. Lawrence Erlbaum Associates, Mahwah, New Jersey,
377:. Lawrence Erlbaum Associates, Mahwah, New Jersey,
364:. Lawrence Erlbaum Associates, Mahwah, New Jersey,
351:. Lawrence Erlbaum Associates, Mahwah, New Jersey,
1286:"Essay-Grading Software Offers Professors a Break"
1306:"Professors angry over essays marked by computer"
277:AES has been criticized on various grounds. Yang
39:measure educational achievement at reduced cost.
979:"Domain-Adaptive Neural Automated Essay Scoring"
722:
720:
671:"Man and machine: Better writers, better grades"
579:", Vantage Learning. Retrieved 28 February 2012.
169:Coherence: appropriate transitions between ideas
1210:Journal of Technology, Learning, and Assessment
411:Journal of Technology, Learning, and Assessment
401:Journal of Technology, Learning, and Assessment
166:Cohesion: appropriate use of transition phrases
163:Development: development of ideas with examples
1352:"Computers Cannot Read, Write or Grade Papers"
915:"Modeling Argument Strength in Student Essays"
607:
605:
551:
549:
218:when irrelevant external factors are altered.
160:Organization: how well the essay is structured
154:Style: word choice, sentence structure variety
1478:
534:"Three prominent writing assessment programs"
8:
1199:"An Overview of Automated Scoring of Essays"
1163:
1161:
808:
806:
684:- Shermis, Mark D., and Jill Burstein, eds.
511:
509:
1896:
1692:
1485:
1471:
1463:
438:
436:
331:Project Essay Grade ā by Measurement, Inc.
1035:
944:
872:
840:
830:
1154:. New York: Routledge. pp. 221ā232.
148:Usage: using of prepositions, word usage
913:Persing, Isaac, and Vincent Ng (2015).
340:
247:Spearman's rank correlation coefficient
145:Grammaticality: following grammar rules
1152:Handbook of Automated Essay Evaluation
1350:Jaffee, Robert David (5 April 2013).
172:Thesis Clarity: clarity of the thesis
137:Different dimensions of essay quality
7:
2259:Tasks of natural language processing
1944:Simple Knowledge Organization System
1336:Teaching & Learning in Higher Ed
673:. University of Akron. 12 April 2012
1330:Corrigan, Paul T. (25 March 2013).
946:10.18653/v1/2020.findings-emnlp.141
497:IEEE Transactions on Communications
328:Intellimetric ā by Vantage Learning
251:concordance correlation coefficient
243:Pearson's correlation coefficient r
1228:Winerip, Michael (22 April 2012).
14:
1959:Thesaurus (information retrieval)
904:Keith, Timothy Z. (2003), p. 149.
517:Journal of Experimental Education
471:International Review of Education
430:, p. 6. Retrieved 19 March 2012-.
1304:Garner, Richard (5 April 2013).
1180:Applied Measurement in Education
1088:- Burstein, Jill (2003), p. 114.
700:"Humans Fight Over Robo-Readers"
777:Review of Research in Education
2239:Educational evaluation methods
1540:Natural language understanding
1284:Markoff, John (4 April 2013).
1098:Bennett, Randy E. (May 2006).
1:
2064:Optical character recognition
733:Journal of Writing Assessment
727:Perelman, Les (August 2013).
611:Rudner, Lawrence (ca. 2002).
486:Page, E.B. (2003), pp. 44-45.
1757:Multi-document summarization
1186:(4). Retrieved 8 March 2012.
1086:Elliot, Scott (2003), p. 77.
856:Ke, Zixuan (9 August 2019).
698:Rivard, Ry (15 March 2013).
577:IntelliMetricĀ®: How it Works
390:. Guilford Press, New York,
388:Handbook of Writing Research
2244:Natural language processing
2087:Latent Dirichlet allocation
2059:Natural language generation
1924:Machine-readable dictionary
1919:Linguistic Linked Open Data
1494:Natural language processing
842:10.1016/j.procs.2020.02.171
324:Educational Testing Service
114:Educational Testing Service
103:sponsored a competition on
29:natural language processing
2275:
2249:Statistical classification
1839:Explicit semantic analysis
1588:Deep linguistic processing
33:statistical classification
2234:Computational linguistics
1682:Word-sense disambiguation
1535:Computational linguistics
1445:"Assessment Technologies"
1143:10.1016/j.asw.2010.04.002
819:Procedia Computer Science
660:. Retrieved 5 March 2012.
632:"Assessment Technologies"
622:. Retrieved 7 March 2012.
543:. Retrieved 6 March 2012.
285:HumanReaders.Org Petition
2208:Natural Language Toolkit
2132:Pronunciation assessment
2034:Automatic identification
1864:Latent semantic analysis
1820:Distributional semantics
1705:Compound-term processing
1603:Named-entity recognition
1450:24 February 2019 at the
1197:- Dikli, Semire (2006).
789:10.3102/0091732X14554179
637:29 December 2011 at the
199:latent semantic analysis
65:Measurement Incorporated
2112:Automated essay scoring
2082:Document classification
1749:Automatic summarization
1174:13 January 2016 at the
991:10.1145/3397271.3401037
921:. Retrieved 2015-10-22.
874:10.24963/ijcai.2019/879
17:Automated essay scoring
1969:Universal Dependencies
1662:Terminology extraction
1645:Semantic decomposition
1640:Semantic role labeling
1630:Part-of-speech tagging
1598:Information extraction
1583:Coreference resolution
1573:Collocation extraction
867:. pp. 6300ā6308.
426:7 October 2007 at the
322:eRater ā published by
265:complex writing test.
27:and an application of
25:educational assessment
1730:Sentence segmentation
656:30 March 2012 at the
224:inter-rater agreement
2182:Voice user interface
1893:datasets and corpora
1834:Document-term matrix
1687:Word-sense induction
1204:8 April 2013 at the
1116:on 24 September 2015
1046:10.18653/v1/P18-2080
618:8 March 2012 at the
539:9 March 2012 at the
445:The Phi Delta Kappan
213:Criteria for success
49:teaching to the test
2162:Interactive fiction
2092:Pachinko allocation
2049:Speech segmentation
2005:Google Ngram Viewer
1777:Machine translation
1767:Text simplification
1762:Sentence extraction
1650:Semantic similarity
1454:, Measurement, Inc.
1265:on 18 November 2019
126:supervised learning
124:system. Results of
44:high-stakes testing
42:The use of AES for
2172:Question answering
2044:Speech recognition
1909:Corpus linguistics
1889:Language resources
1672:Textual entailment
1655:Sentiment analysis
1290:The New York Times
1234:The New York Times
917:, pp. 543-552. In
688:. Routledge, 2013.
532:Rudner, Lawrence.
307:acts of writing."
297:The New York Times
203:Bayesian inference
101:Hewlett Foundation
2221:
2220:
2177:Virtual assistant
2102:Computer-assisted
2028:
2027:
1785:Computer-assisted
1743:
1742:
1735:Word segmentation
1697:Text segmentation
1635:Semantic analysis
1623:Syntactic parsing
1608:Ontology learning
1131:Assessing Writing
1000:978-1-4503-8016-4
884:978-0-9992411-4-1
759:Assessing Writing
195:linear regression
61:Ellis Batten Page
2266:
2198:Formal semantics
2147:Natural language
2054:Speech synthesis
2036:and data capture
1939:Semantic network
1914:Lexical resource
1897:
1715:Lexical analysis
1693:
1618:Semantic parsing
1487:
1480:
1473:
1464:
1455:
1442:
1436:
1435:
1433:
1431:
1425:HumanReaders.Org
1416:
1414:
1412:
1406:HumanReaders.Org
1398:
1392:
1391:
1389:
1387:
1381:HumanReaders.Org
1373:
1367:
1366:
1364:
1362:
1346:
1344:
1342:
1327:
1321:
1320:
1318:
1316:
1300:
1298:
1296:
1281:
1275:
1274:
1272:
1270:
1261:. Archived from
1259:HumanReaders.Org
1251:
1245:
1244:
1242:
1240:
1225:
1219:
1193:
1187:
1165:
1156:
1155:
1146:
1125:
1123:
1121:
1115:
1109:. Archived from
1104:
1095:
1089:
1084:
1078:
1064:
1058:
1057:
1039:
1019:
1013:
1012:
974:
968:
965:
959:
958:
948:
928:
922:
911:
905:
902:
896:
895:
893:
891:
876:
862:
853:
847:
846:
844:
834:
810:
801:
800:
772:
766:
755:
749:
748:
746:
744:
724:
715:
714:
712:
710:
704:Inside Higher Ed
695:
689:
682:
680:
678:
667:
661:
648:
642:
629:
623:
609:
600:
586:
580:
573:
567:
553:
544:
530:
524:
513:
504:
493:
487:
484:
478:
467:
461:
460:
440:
431:
345:
110:Randy E. Bennett
78:national exams.
73:Peter Foltz and
2274:
2273:
2269:
2268:
2267:
2265:
2264:
2263:
2224:
2223:
2222:
2217:
2186:
2166:Syntax guessing
2148:
2141:
2127:Predictive text
2122:Grammar checker
2103:
2096:
2068:
2035:
2024:
1990:Bank of English
1973:
1901:
1892:
1883:
1814:
1771:
1739:
1691:
1593:Distant reading
1568:Argument mining
1554:
1550:Text processing
1496:
1491:
1460:
1458:
1452:Wayback Machine
1443:
1439:
1429:
1427:
1419:
1417:
1410:
1408:
1400:
1399:
1395:
1385:
1383:
1375:
1374:
1370:
1360:
1358:
1356:Huffington Post
1349:
1347:
1340:
1338:
1329:
1328:
1324:
1314:
1312:
1310:The Independent
1303:
1301:
1294:
1292:
1283:
1282:
1278:
1268:
1266:
1253:
1252:
1248:
1238:
1236:
1227:
1226:
1222:
1217:
1206:Wayback Machine
1196:
1194:
1190:
1176:Wayback Machine
1166:
1159:
1149:
1147:
1128:
1126:
1119:
1117:
1113:
1102:
1097:
1096:
1092:
1087:
1085:
1081:
1065:
1061:
1021:
1020:
1016:
1001:
976:
975:
971:
966:
962:
930:
929:
925:
912:
908:
903:
899:
889:
887:
885:
860:
855:
854:
850:
812:
811:
804:
774:
773:
769:
756:
752:
742:
740:
726:
725:
718:
708:
706:
697:
696:
692:
683:
676:
674:
669:
668:
664:
658:Wayback Machine
649:
645:
639:Wayback Machine
630:
626:
620:Wayback Machine
610:
603:
587:
583:
574:
570:
554:
547:
541:Wayback Machine
531:
527:
514:
507:
494:
490:
485:
481:
468:
464:
442:
441:
434:
428:Wayback Machine
418:
408:
398:
385:
372:
359:
346:
342:
338:
316:
287:
275:
239:Krippendorf's Ī±
215:
182:
139:
75:Thomas Landauer
57:
12:
11:
5:
2272:
2270:
2262:
2261:
2256:
2251:
2246:
2241:
2236:
2226:
2225:
2219:
2218:
2216:
2215:
2210:
2205:
2200:
2194:
2192:
2188:
2187:
2185:
2184:
2179:
2174:
2169:
2159:
2153:
2151:
2149:user interface
2143:
2142:
2140:
2139:
2134:
2129:
2124:
2119:
2114:
2108:
2106:
2098:
2097:
2095:
2094:
2089:
2084:
2078:
2076:
2070:
2069:
2067:
2066:
2061:
2056:
2051:
2046:
2040:
2038:
2030:
2029:
2026:
2025:
2023:
2022:
2017:
2012:
2007:
2002:
1997:
1992:
1987:
1981:
1979:
1975:
1974:
1972:
1971:
1966:
1961:
1956:
1951:
1946:
1941:
1936:
1931:
1926:
1921:
1916:
1911:
1905:
1903:
1894:
1885:
1884:
1882:
1881:
1876:
1874:Word embedding
1871:
1866:
1861:
1854:Language model
1851:
1846:
1841:
1836:
1831:
1825:
1823:
1816:
1815:
1813:
1812:
1807:
1805:Transfer-based
1802:
1797:
1792:
1787:
1781:
1779:
1773:
1772:
1770:
1769:
1764:
1759:
1753:
1751:
1745:
1744:
1741:
1740:
1738:
1737:
1732:
1727:
1722:
1717:
1712:
1707:
1701:
1699:
1690:
1689:
1684:
1679:
1674:
1669:
1664:
1658:
1657:
1652:
1647:
1642:
1637:
1632:
1627:
1626:
1625:
1620:
1610:
1605:
1600:
1595:
1590:
1585:
1580:
1578:Concept mining
1575:
1570:
1564:
1562:
1556:
1555:
1553:
1552:
1547:
1542:
1537:
1532:
1531:
1530:
1525:
1515:
1510:
1504:
1502:
1498:
1497:
1492:
1490:
1489:
1482:
1475:
1467:
1457:
1456:
1437:
1393:
1368:
1322:
1276:
1246:
1220:
1188:
1157:
1137:(2): 118ā129.
1090:
1079:
1059:
1014:
999:
969:
960:
923:
906:
897:
883:
848:
802:
783:(1): 370ā407.
767:
750:
716:
690:
662:
651:Hewlett prize"
643:
624:
601:
581:
568:
545:
525:
505:
488:
479:
462:
451:(5): 238ā243.
432:
339:
337:
334:
333:
332:
329:
326:
315:
312:
286:
283:
274:
271:
214:
211:
181:
178:
177:
176:
173:
170:
167:
164:
161:
158:
155:
152:
149:
146:
138:
135:
56:
53:
13:
10:
9:
6:
4:
3:
2:
2271:
2260:
2257:
2255:
2252:
2250:
2247:
2245:
2242:
2240:
2237:
2235:
2232:
2231:
2229:
2214:
2211:
2209:
2206:
2204:
2203:Hallucination
2201:
2199:
2196:
2195:
2193:
2189:
2183:
2180:
2178:
2175:
2173:
2170:
2167:
2163:
2160:
2158:
2155:
2154:
2152:
2150:
2144:
2138:
2137:Spell checker
2135:
2133:
2130:
2128:
2125:
2123:
2120:
2118:
2115:
2113:
2110:
2109:
2107:
2105:
2099:
2093:
2090:
2088:
2085:
2083:
2080:
2079:
2077:
2075:
2071:
2065:
2062:
2060:
2057:
2055:
2052:
2050:
2047:
2045:
2042:
2041:
2039:
2037:
2031:
2021:
2018:
2016:
2013:
2011:
2008:
2006:
2003:
2001:
1998:
1996:
1993:
1991:
1988:
1986:
1983:
1982:
1980:
1976:
1970:
1967:
1965:
1962:
1960:
1957:
1955:
1952:
1950:
1949:Speech corpus
1947:
1945:
1942:
1940:
1937:
1935:
1932:
1930:
1929:Parallel text
1927:
1925:
1922:
1920:
1917:
1915:
1912:
1910:
1907:
1906:
1904:
1898:
1895:
1890:
1886:
1880:
1877:
1875:
1872:
1870:
1867:
1865:
1862:
1859:
1855:
1852:
1850:
1847:
1845:
1842:
1840:
1837:
1835:
1832:
1830:
1827:
1826:
1824:
1821:
1817:
1811:
1808:
1806:
1803:
1801:
1798:
1796:
1793:
1791:
1790:Example-based
1788:
1786:
1783:
1782:
1780:
1778:
1774:
1768:
1765:
1763:
1760:
1758:
1755:
1754:
1752:
1750:
1746:
1736:
1733:
1731:
1728:
1726:
1723:
1721:
1720:Text chunking
1718:
1716:
1713:
1711:
1710:Lemmatisation
1708:
1706:
1703:
1702:
1700:
1698:
1694:
1688:
1685:
1683:
1680:
1678:
1675:
1673:
1670:
1668:
1665:
1663:
1660:
1659:
1656:
1653:
1651:
1648:
1646:
1643:
1641:
1638:
1636:
1633:
1631:
1628:
1624:
1621:
1619:
1616:
1615:
1614:
1611:
1609:
1606:
1604:
1601:
1599:
1596:
1594:
1591:
1589:
1586:
1584:
1581:
1579:
1576:
1574:
1571:
1569:
1566:
1565:
1563:
1561:
1560:Text analysis
1557:
1551:
1548:
1546:
1543:
1541:
1538:
1536:
1533:
1529:
1526:
1524:
1521:
1520:
1519:
1516:
1514:
1511:
1509:
1506:
1505:
1503:
1501:General terms
1499:
1495:
1488:
1483:
1481:
1476:
1474:
1469:
1468:
1465:
1461:
1453:
1449:
1446:
1441:
1438:
1426:
1422:
1407:
1403:
1397:
1394:
1382:
1378:
1372:
1369:
1357:
1353:
1337:
1333:
1326:
1323:
1311:
1307:
1291:
1287:
1280:
1277:
1264:
1260:
1256:
1250:
1247:
1235:
1231:
1224:
1221:
1215:
1211:
1207:
1203:
1200:
1192:
1189:
1185:
1181:
1177:
1173:
1170:
1164:
1162:
1158:
1153:
1144:
1140:
1136:
1132:
1112:
1108:
1101:
1094:
1091:
1083:
1080:
1077:
1073:
1069:
1063:
1060:
1055:
1051:
1047:
1043:
1038:
1033:
1029:
1025:
1018:
1015:
1010:
1006:
1002:
996:
992:
988:
984:
980:
973:
970:
964:
961:
956:
952:
947:
942:
938:
934:
927:
924:
920:
916:
910:
907:
901:
898:
886:
880:
875:
870:
866:
859:
852:
849:
843:
838:
833:
828:
824:
820:
816:
809:
807:
803:
798:
794:
790:
786:
782:
778:
771:
768:
764:
760:
754:
751:
738:
734:
730:
723:
721:
717:
705:
701:
694:
691:
687:
672:
666:
663:
659:
655:
652:
647:
644:
640:
636:
633:
628:
625:
621:
617:
614:
608:
606:
602:
599:
595:
591:
585:
582:
578:
572:
569:
566:
562:
558:
552:
550:
546:
542:
538:
535:
529:
526:
523:(2), 127-142.
522:
518:
512:
510:
506:
503:(1), 105-110.
502:
498:
492:
489:
483:
480:
477:(3), 253-263.
476:
472:
466:
463:
458:
454:
450:
446:
439:
437:
433:
429:
425:
422:
416:
412:
406:
402:
397:
396:1-59385-190-1
393:
389:
384:
380:
376:
371:
367:
363:
358:
354:
350:
344:
341:
335:
330:
327:
325:
321:
320:
319:
313:
311:
308:
304:
301:
299:
298:
293:
284:
282:
280:
272:
270:
266:
262:
258:
254:
252:
249:Ļ, and Lin's
248:
244:
240:
236:
232:
227:
225:
219:
212:
210:
206:
204:
200:
196:
190:
186:
179:
174:
171:
168:
165:
162:
159:
156:
153:
150:
147:
144:
143:
142:
136:
134:
131:
127:
123:
117:
115:
111:
106:
102:
99:In 2012, the
97:
94:
90:
86:
82:
79:
76:
71:
67:
66:
62:
54:
52:
50:
45:
40:
36:
34:
30:
26:
22:
18:
2117:Concordancer
2111:
1513:Bag-of-words
1459:
1440:
1428:. Retrieved
1424:
1409:. Retrieved
1405:
1396:
1384:. Retrieved
1380:
1371:
1359:. Retrieved
1355:
1339:. Retrieved
1335:
1325:
1313:. Retrieved
1309:
1293:. Retrieved
1289:
1279:
1267:. Retrieved
1263:the original
1258:
1249:
1237:. Retrieved
1233:
1223:
1213:
1209:
1191:
1183:
1179:
1151:
1134:
1130:
1118:. Retrieved
1111:the original
1106:
1093:
1082:
1067:
1062:
1027:
1017:
982:
972:
963:
936:
926:
918:
909:
900:
888:. Retrieved
864:
851:
822:
818:
780:
776:
770:
762:
758:
753:
741:. Retrieved
736:
732:
707:. Retrieved
703:
693:
685:
675:. Retrieved
665:
646:
627:
589:
584:
571:
556:
528:
520:
516:
500:
496:
491:
482:
474:
470:
465:
448:
444:
414:
410:
404:
400:
387:
374:
361:
348:
343:
317:
309:
305:
302:
295:
292:Noam Chomsky
288:
278:
276:
267:
263:
259:
255:
228:
220:
216:
207:
191:
187:
183:
140:
118:
98:
95:
91:
87:
83:
80:
72:
68:
58:
41:
37:
20:
16:
15:
2074:Topic model
1954:Text corpus
1800:Statistical
1667:Text mining
1508:AI-complete
825:: 726ā743.
2228:Categories
1795:Rule-based
1677:Truecasing
1545:Stop words
1076:0805839739
1037:1804.07954
832:1807.10543
765:, 104-111.
598:0805839739
565:0805839739
383:0805839739
370:0805839739
357:0805839739
336:References
130:clustering
85:feedback.
2104:reviewing
1902:standards
1900:Types and
1009:220730151
955:226299478
797:145592665
273:Criticism
235:Cohen's Īŗ
231:Scott's Ļ
180:Procedure
2020:Wikidata
2000:FrameNet
1985:BabelNet
1964:Treebank
1934:PropBank
1879:Word2vec
1844:fastText
1725:Stemming
1448:Archived
1202:Archived
1172:Archived
890:11 April
654:Archived
635:Archived
616:Archived
537:Archived
457:20371545
424:Archived
314:Software
2191:Related
2157:Chatbot
2015:WordNet
1995:DBpedia
1869:Seq2seq
1613:Parsing
1528:Trigram
1430:5 April
1411:5 April
1386:5 April
1361:5 April
1341:5 April
1315:5 April
1295:5 April
1269:5 April
1239:5 April
1054:5070986
743:13 June
709:14 June
55:History
2254:Essays
2164:(c.f.
1822:models
1810:Neural
1523:Bigram
1518:n-gram
1120:5 July
1074:
1052:
1007:
997:
953:
881:
795:
677:4 July
596:
563:
455:
394:
381:
368:
355:
105:Kaggle
2213:spaCy
1858:large
1849:GloVe
1114:(PDF)
1103:(PDF)
1050:S2CID
1032:arXiv
1005:S2CID
951:S2CID
861:(PDF)
827:arXiv
793:S2CID
453:JSTOR
279:et al
1978:Data
1829:BERT
1432:2013
1413:2013
1388:2013
1363:2013
1343:2013
1317:2013
1297:2013
1271:2013
1241:2013
1122:2015
1072:ISBN
995:ISBN
892:2020
879:ISBN
745:2015
711:2015
679:2015
594:ISBN
561:ISBN
392:ISBN
379:ISBN
366:ISBN
353:ISBN
201:and
122:GCSE
2010:UBY
1216:(1)
1139:doi
1042:doi
987:doi
941:doi
869:doi
837:doi
823:169
785:doi
739:(1)
417:(2)
407:(3)
51:).
21:AES
2230::
1423:.
1418:-
1404:.
1379:.
1354:.
1348:-
1334:.
1308:.
1302:-
1288:.
1257:.
1232:.
1212:,
1208:,
1184:15
1182:,
1178:,
1160:^
1148:-
1135:15
1133:.
1127:-
1105:.
1048:.
1040:.
1026:.
1003:.
993:.
981:.
949:.
935:.
877:.
863:.
835:.
821:.
817:.
805:^
791:.
781:39
779:.
763:21
761:,
735:.
731:.
719:^
702:.
604:^
548:^
521:62
519:,
508:^
499:,
475:14
473:,
449:47
447:.
435:^
413:,
405:10
403:,
253:.
245:,
241:,
237:,
233:,
205:.
35:.
2168:)
1891:,
1860:)
1856:(
1486:e
1479:t
1472:v
1434:.
1415:.
1390:.
1365:.
1345:.
1319:.
1299:.
1273:.
1243:.
1214:5
1145:.
1141::
1124:.
1056:.
1044::
1034::
1011:.
989::
957:.
943::
894:.
871::
845:.
839::
829::
799:.
787::
747:.
737:6
713:.
681:.
575:"
501:3
459:.
415:6
19:(
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.