Part 6: Science? Research? An AI Example

At CREST, we looked at evaluating research and developed suggestions for laypeople to evaluate citations provided to us as support for topics that impact our instruction. Below are our notes on evaluating the last two citations on Pearson's website about Automated Scoring.  

• Citation: Landauer, T. K., Laham, R. D. & Foltz, P. W. (2003b). Automated Essay Assessment. Assessment in Education, 10,3, pp. 295-308.
• Authors: Landauer and Foltz direct Research at Pearson’s Knowledge Division
• Source: Journal: Assessment in Education; The Journal is associated with the International Association for Educational Assessment. We can read about their aims and scope here. 
• Question: While the citation on Pearson’s website lists three authors, the actual journal only lists Landauer. Why?
• Type of Research: Survey, description; 
• Validity: Reports on scientific research, but is not scientific research in itself. This article would be a good place to begin by reading the studies referenced
• Something we might consider are works written about the Intelligent Essay Assessor like the one in chapter 5 of Machine Scoring of Student Essays: Truth and Consequences. This chapter describes the user experience with the tool and the outcomes.

• Citation: Landauer, T. K., Laham, D. & Foltz, P. W. (2001). Automated essay scoring. IEEE Intelligent Systems. September/October.
• Authors: We are already familiar with Landauer and Foltz. Laham is a PhD who has written and published 33 articles according to research gate. The publication itself gives additional information about the authors. Laham also works for Knowledge Analysis Technologies - which Pearson has purchased. 
• Source: IEEE Intelligent Systems: This is a journal published by the the IEEE Computer society and is sponsored by the Association for the Advancement of Artificial Intelligence (wikipedia)
• Link to Article: https://www.researchgate.net/publication/256980328_The_Intelligent_Essay_Assessor 
• Validity: The article is basically a report with interviews with Landauer and others like ETS. It includes data from research, but technically isn’t scientific research itself. 
• The journal did give some insightful information about the authors. We even have a face to put with the name.

So it's also important to dig into the types of research and how the research is designed and conducted. But, as we see from the notes, none of the citations are actually the research itself that would help us evaluate the impact of these types of programs. It's like we need a way to know when we are presented with research and where to find it when it's not present. And then, we need a clearinghouse of some type that would explain what all this stuff means. Right now, it feels like a big mess that's hard to untangle and arrive at the actual research. And that's just the beginning - we still have to make sense of the research and how it impacts our classrooms - ultimately, how it impacts the humans we serve.