Part 2: Is it research or scientific? Tips for Evaluating Citations

    Here's some things we discovered when analyzing the citations on Pearson's website about Automated Scoring. 

What do people usually do when they see citations on a website? It's easy to see accurately annotated information and think that it's research. That would be a mistake. Some folks might go as far as trying to click on the links. Well, they don't work. 

Evaluating Citations: DOI

Links fail. That's a reality. And some stuff gets posted that's old. Modern research of peer reviewed material often comes with a doi link: Digital object identifier of stuff that's published that has a permanent web address. When webpages change, the DOI doesn't. If you don't have a doi, sometimes you can find the article here. Respectable research is out there that doesn't have a doi, so it's not a deal killer if what' you are looking at doesn't use them. And you might have to reach out to a librarian or professor to get access to some articles. But citations with a doi are preferred when I'm doing research. 

Conference Presentations aren't Research

Pearson lists two conference presentations in the first two citations. 

Presented at the National Conference on Student Assessment (NCSA), San Diego, CA. and Presented at the National Conference on Student Assessment (NCSA), Philadelphia, PA. 

Researchers love to present at these things to share and celebrate their findings. While presentations can contain research, they aren't research in themselves, we don't know what kind, and have no way to evaluate the source and product. Expecially when the conference links are broken. 

Usually, it's tough to be approved to present at national conferences. So even though the presentations aren't research, we do learn that the presenters were actively involved in presenting and refining their professionalism. 

Consider Affiliations and Origins

We also need to note the name of the conference. These folks are all about Student Assessment. That's their focus. We need to keep that in mind. As we/practitioners look at research and researchers, is that our focus? Usually not. Usually, our focus is on people who have to take the assessments. That changes things for us and how we look at their activities in comparison to ours. 

It's probably also important to go look at the website of these conferences and ask some important questions:  

What do they value in proposals?  

Why do they exist? What kind of people belong and attend?  Well - they are associated with Council of Chief State School Officers, state education leaders, and assessment practitioners. They "support states in implementing high-quality assessments and robust accountability systems, ultimately driving better outcomes for all students." That sounds like politics and state agencies to me. 

What kinds of things do they talk about?  

Who gives them money?  Pearson was a Platinum sponsor for 2024. (There is also no evidence that proposals are blind or double blind for selection. This means that those selecting the presenters know who the presenters are. This could cause bias, especially since Pearson hosts the conference - and as you will see later, som of the presenters also work for Pearson.)

Ultimately, we have to ask: Do these answers resonate with our purposes, people, and needs?