Part 3: Is it research or science? Tips for Evaluating Citations

In evaluating citations, you might find out that something really doesn't count as scientific research. But, you can certainly learn a lot. 

Let's take the first two citations from Pearson's Automated Scoring Website. 

As we worked at the 2024 CREST conference session, we looked at the first two citations and developed some important ideas and tips for evaluating citations. 

Part of evaluating the citations comes from looking at the parts of the citation itself: author, venue/source, accessibility of the link. 

We started by trying out the links. As you know, the links don't work. 

Next, we copied and pasted the entire citation to see if we could find it posted or cited elsewhere on an internet or generic scholar search. 

As explained before, we were able to look up the organization/conference. 

Citation One (2018, June)

We found a pdf of the conference handbook. From there, we were able to see information about the presenters and the session itself.  

The Authors: On page 7 of the program, we see that the first presenter is J. Hauger, a member of the conference program planning committee, and member of the New Jersey Department of Education. On page 13 of the program, we see the session description and the remaining authors. 

Lochbaum is listed first in the program, but wasn't listed first in the citation.  (That's a red flag when people change the order of authorship/participation, as that's part of accurate scholarship because it is used for tenure and issues of academic integrity and property.) She's the VP of Technology Services at Pearson and Knowledge Technologies. 

 

Quesen is listed as an Associate Research Scientist at Pearson. 

Zurkowski was listed in Pearson's citation, but not in the conference program. This is not and academic practice of integrity and is dishonest. 

We also see from the program that another person from Pearson is associated with the presentation as a moderator for the session: Trent Workman, Vice President of School Assessment. Since this is a national conference on assessment, members from other states would be in attendance. Meeting potential clients at the conference would be beneficial. And nothing wrong with that. But it's not research. And it does show bias. 

The Session Description: From the session description, we can see that the session is about how Pearson's assessment product benefits "school needs" and saves time. "Consistency, accuracy, and time savings" are offered as powerful enticements, but research is not the focus of this session. 

Citation 2 (2018, June) We already know about Lochbaum. We did find a citation to the conference in another work, but yet again, the order of authorship/participation were different. 

Tips on Evaluating Citations

Find out who the authors are. Where do they work? What else have they written? 

Make sure the citations match when considered/reported in multiple places. 

Consider the dates. Y'all, this stuff is from 2016 and 2018. That's 6 to 8 years ago. ANCIENT history in the technology world. And most scholarship doesn't consider anything older than 5 years unless it's seminal research or philosophy.