John Markoff at the New York Times reports on a fast-moving, back-and-forth exchange where students submit their essays online, receive a grade almost immediately, and improve their grades based on system-generated feedback.
While controversy rages over the reliability of artificial intelligence to grade essays, EdX software is free to any institution that wants to offer its courses online. So far, the program has been adopted by 12 prestigious universities and it is spreading rapidly worldwide.
Proponents of the software argue that instant feedback is an invaluable learning aid to students versus waiting weeks for professor-graded feedback. Moreover, students find it engaging in much the same way as video games and claim they learn better from the process.
Critics counter that even with the best machine learning algorithms in place; computers cannot perform the essentials of assessing written communication. Les Perelman, a researcher at MIT, has tricked such grading systems into awarding high grades with nonsensical submissions.
A group of educators to which he belongs known as Professionals Against Machine Scoring of Student Essays in High-Stakes Assessment, has collected nearly 2,000 signatures and makes the case that “Computers cannot ‘read.’ They cannot measure the essentials of effective written communication: accuracy, reasoning, adequacy of evidence, good sense, ethical stance, convincing argument, meaningful organization, clarity, and veracity, among others.”
The EdX program has human graders assess the first 100 essays or essay questions. From then on, the system uses various machine-learning algorithms to train itself automatically. Once trained, it can grade any number of essays or answers in near real time. The software lets the teacher create the scoring system based on letter grades or numerical rankings.
Dr. Anant Agarwal, president of EdX, believes the program is approaching the capability of human graders. Skeptics point out how formal studies comparing the system against qualified human graders have not been done. Nevertheless, Dr. Agarwal claims the quality of EdX grading is as consistent as that found from one instructor to another.
Instant, automated feedback has its adherents elsewhere as well, including start-ups Coursera and Udacity. Both are funded by Stanford faculty members as part of their mission to create “massive open online courses,” or MOOCs.
Coursera founder, Daphne Koller, believes instant feedback turns learning into a game students feel compelled to master where they resubmit their work until they achieve a certain level of proficiency.
So, if automated grading is possible in academic settings, the general idea of assessing new written content based on previous human assessments of existing content is sure to explode over the next few years.
Applications that mine blogs, social media and forum postings to understand markets and communities come to mind.
What do you see happening in your field once automated interpretation of extended passages of text goes mainstream?