Automated Essay Grading vs Enterprise Resource Planning Software - Case Studies

Case Study 1: Should a Computer Grade Your Essays?

Would you like your college essays graded by a computer? Well, you just might find that happening in your next course. In April 2013, EdX, a Harvard/MIT joint venture to develop massively open online courses (MOOCs), launched an essay-scoring program. Using artificial intelligence technology, essays and short answers are immediately scored and feedback tendered, allowing students to revise, resubmit, and improve their grade as many times as necessary. The non-profit organization is offering the software free to any institution that wants to use it. From a pedagogical standpoint—if the guidance is sound—immediate feedback and the ability to directly act on it is an optimal learning environment. But while proponents trumpet automated essay grading's superiority to students waiting days or weeks for returned papers— which they may or may not have the opportunity to revise—as well as the time-saving benefit for instructors, critics doubt that humans can be replaced.

In, Les Perelman, the former director of writing at MIT, countered a paper touting the proficiency of automated essay scoring (AES) software. University of Akron College of Education dean, Mark Shermis, and co-author, data scientist Ben Hamner used AES programs from nine companies, including Pearson and McGraw-Hill, to rescore over 16,000 middle and high school essays from six different state standardized tests. Their Hewlett Foundation sponsored study found that machine scoring closely tracked human grading, and in some cases, produced a more accurate grade. Perelman, however, found that no direct statistical comparison between the human graders and the programs was performed. While Shermis concedes that regression analysis was not performed—because the software companies imposed this condition in order to allow him and Hamner to test their products—he unsurprisingly accuses Perelman of evaluating their work without performing research of his own.

Perelman has in fact conducted studies on the Electronic Essay Rater (e-rater) developed by the Educational Testing Service (ETS)—the only organization that would allow him access. The e-rater uses syntactic variety, discourse structure (like PEG) and content analysis (like IEA) and is based on natural language processing technology. It applies statistical analysis to linguistic features like argument formation and syntactic variety to determine scores, but also gives weight to vocabulary and topical content. In the month granted him, Perelman analyzed the algorithms and toyed with the e-Rater, confirming his prior critiques. The major problem with AES programs (so far) is that they cannot distinguish fact from fiction. For example, in response to an essay prompt about the causes for the steep rise in the cost of higher education, Perelman wrote that the main driver was greedy teaching assistants whose salaries were six times that of college presidents with exorbitant benefits packages including South Seas vacations, private jets, and movie contracts. He supplemented the argument with a line from Allen Ginsberg's "Howl," and received the top score of. The metrics that merited this score included overall length, paragraph length, number of words per sentence, word length, and the use of conjunctive adverbs such as "however" and "moreover." Since computer programs cannot divine meaning, essay length is a proxy for writing fluency, conjunctive adverb use for complex thinking, and big words for vocabulary aptitude.

Case Study 2: American Water Keeps Data Flowing

1. Identify the kinds of systems described in this case.

2. What are the benefits of automated essay grading? What are the drawbacks?

3. What management, organization, and technology factor should be considered when deciding whether to use AES?

American Water, founded in 1886, is the largest public water utility in the United States. Headquartered in Voorhees, N.J., the company employs more than 7,000 dedicated professionals who provide drinking water, wastewater and other related services to approximately 16 million people in  states, as well as Ontario and Manitoba, Canada. Most of American Water's services support locally managed utility subsidiaries that are regulated by the U.S. state in which each operates as well as the federal government. American Water also owns subsidiaries that manage municipal drinking water and wastewater systems under contract and others that supply businesses and residential communities with water management products and services.

Until recently, American water's systems and business, processes were much localized, and many of these processes were manual. Over time, this information environment became increasingly difficult to manage. Many systems were not integrated, so that running any type of report that had to provide information about more than one region was a heavily manual process. Data had to be extracted from the systems supporting each region and then combined manually to create the desired output. When the company was preparing to hold an initial public offering of its stock in 2006, its software systems could not handle the required regulatory controls, so roughly 80 percent of this work had to be performed manually. It was close to a nightmare.

Management wanted to change the company from a decentralized group of independent regional businesses into a more centralized organization with standard company-wide business processes and enterprise-wide reporting. The first step toward achieving this goal was to implement an enterprise resource planning (ERP) system designed to replace disparate systems with a single integrated software platform. The company selected SAP as its ERP system vendor.

An important step of this project was to migrate the data from American Water's old systems to the new platform. The company's data resided in many different systems in various formats. Each regional business maintained some of its own data in its own systems, and a portion of these data was redundant and inconsistent. For example, there were duplicate pieces of materials master data because a material might be called one thing in the company's Missouri operation and another in its New Jersey business. These names had to be standardized so that every business unit used the same name for a piece of data. American Water's business users had to buy into this new company-wide view of data.

1. How did implementing a data warehouse help American Water move toward a more centralized organization?

2. Give some examples of problems that would have occurred at American Water if its data were not "clean"?

3. How did American Water's data warehouse improve operations and management decision making?