Part 4: Is it research or science? Tips for Evaluating Citations.

 Just because something is cited, doesn't mean that it is research. It's called a reference. At CREST, I worked with a group of colleagues to establish some advice about how to evaluate citations to determine what research says and what we need to know about it as practitioners - people who work with kids and teachers. 

In this post, we'll evaluate Pearson's third citation about Automated Scoring.

Citation: Foltz, P. W., Streeter, L. A. , Lochbaum, K. E., & Landauer, T. K. (2013). Implementation and applications of the Intelligent Essay Assessor. In Handbook of Automated Essay Evaluation, M. D. Shermis & J. Burstein (Eds.), pp. 68-88. Routledge: New York.

Authors:

Foltz, P; Landauer and Foltz used AI scoring in their classes starting in 1994. Now they both direct research at Pearson’s Knowledge Technologies Group. Lochbaum is VP of Technology Services at Pearson. As before, we've pointed out the possible bias. But I'd also have to say, if I were a company, I'd be all about hiring the best researchers in the field. It's not that the folks are evil - we just need to be aware that they get a paycheck from the company that sells that product.

Source:

Handbook of Automated Essay Evaluation; Published by Routledge. Routeledge is a respected publisher from my limited experience in academia. I found the whole book online and the chapter, Implementation and Applications of the Intelligent Essay Assessor.

Title: 

Maybe this should be obvious, but the title tells you a lot. The chapter is about Implementation and Applications of automated scoring. Does that answer our questions as practitioners? It should also tell us a bit about the audience.  Are they writing for us? 

Abstract: 

On a separate site, The American Psychological Association, I looked for information about the chapter. The abstract of a text gives you an overview of the work. See it here.  Now, just be aware, writers craft their own abstracts. Much of the text in the abstract is also in the introduction of the chapter. In the abstract, we learn about the background of the authors and their use of the technology in their classes during the late 90's and the association with Pearson. The abstract chronologically highlights the activities of developing the technology. We learn about prompts: "Describe the differences between classical and operant conditioning." and "Describe the functioning of the human heart." We learn that NLP, natural language processing (readability, grammar, and spelling) is combined with LSA (latent semantic analysis) to measure content and semantics. 

So is this helpful for grading essays? In the abstract, we learn that the machine can tell if a writer is high school student, an undergraduate, or a medical student describing the heart prompt mentioned above. 

We learn that the ELAR crowd wanted more - stuff about style, grammar, and feedback. So the program algorithms became more robust - up to 60 different criteria, including "trait scores such as organization or conventions." 

Common Core is mentioned to show a focus on what is valued and assessed in terms of "mastery and higher order thinking." Then, the article promises to address how the IEA (Intelligent Essay Assessor) will be used and how it works, even in comparison to human raters. 

So...the article is a history and summary of the project. It will include research, but it's NOT the research. 

The Article Itself:

Note that Write to Learn's essay feedback scoreboard (Pearson's Product) appears on page 70 of the article. Research IS reported and documented in the article. Lots of great information follows about how this stuff works. But, to really know about the quality and meaning of the research, we'd have to actually look at the stuff cited in the article. 

Drilling Down to the Citations

The internal citations throughout the article and the references on pages 86-88 are the places we would head next to validate and search for the actual research. The chapter is a great place to begin, but the chapter itself isn't the research. 

Admission of Conflict of Interest

Scholarly work will show when conflicts of interest or associations are present. It's part of being honest. 

The chapter concludes with such a statement. And it is important that practitioners recognize the involvement that might color the way the research referenced in the article is presented. 

The Book and Table of Contents

Remember, this search is less about the particular authors and text and more about answering our questions and determining the content and quality of the research behind the topic. Sometimes, the book the chapter is in will have the kind of information we seek as practitioners. 

Chapter 11, on Validity and Reliability of Automated Essay Scoring might just be the chapter we need to consider. 

Bottom Line: 

Yep. The article posted on the Pearson website to validate the product is a reference. It's NOT research. But, the article is a bounty of background and a map of places to go in search of the research.