Is it RESEARCH? AI Scoring: A Call For Readers and Thinkers

Think and reason with me? Pretty Please? 

CAVEAT: So, I really do think we are ok for AI scoring of the type of writing that STAAR will assess. CAVEAT: And, I think this will be better than what happened with scoring in the last administration. (I've seen pretty much identical papers get very different scores. And there's lots of confusion out there because of it about what kids need to do, even bad advice like just use less words.) 

But AI scoring is going to be a reality. It's already used effectively with TSI. It's already used with TELPAS. Not sure I can say effectively because I don't have any information or experience about how that's working. 

Now it's time to dig deeper. As I've been discussing things with folks I respect, these are some thoughts we should consider: 

Concerns: 

AI can be: 

• technically flawed
• unresponsive to originality and nuance
• based on predictability formulas

Essential questions: 

In addition, is there consensus in the field about AI's value, validity, and reliability? Are we looking at real research? And what does the research we are given say? What other voices are out there? 

Background: 

Pearson reports that their scoring is "backed by research and [is] unparalleled in the field." At the end of the website are several articles. I'm waiting on access to some of these. In the mean time, let's take a closer look at what's offered as research.

I've started a spreadsheet here that we can use to collaborate. 

Call for research and analysis: 

As Joyce Armstrong Carroll and Abydos state: I'm [researching.] Won't you join me? 

Step One: 

Can you join me in reviewing these texts? We need an annotated copy of the articles. If you have information for the other cells, we need help in evaluating those details. What else should I consider? 

Step Two: 

What other peer reviewed research of excellence should be considered? I've added a second tab for that analysis.