During yesterday's PLC, we tried to figure out what was going on with the item analysis for critical items on last year's English I STAAR. Summary and Inference are always problems. Always. We're tired of it. How can we fix this and accelerate the growth for kids on these item types?
Data Analysis:
We started with the item analysis. Question 49, Reporting Category 3: The student will demonstrate an ability to understand and analyze informational texts. 5D: paraphrase and summarize texts in ways that maintain meaning and logical order.
"Why are less than half of us getting this question right?" we asked. And why such large distractors? We did NOT ask the following: Did they read the passage? Did they read all the answer choices? Did they read all of each answer choice? We already know those are problems. That kind of problem requires a response that has nothing to do with reading instruction.
(I use a benchmark of 18% or above to identify distractors that indicate a problem with instruction or student reasoning. We were almost there with three distractors. To me, that means that a large percentage of that 47% that picked the right answer probably weren't thinking right about it either. We have a huge problem here. If kids can't summarize correctly to demonstrate their understanding, then how were they going to be able to make good decisions about harder questions?)
Using Tools from TEA:
Next, we went to the parent/student portal for the learning leader's kid who took the test last year. This was the question:
49. Which sentence best summarizes this article?
A Automobile companies Oldsmobile, Winton, and Packard sponsored road trips like the one that Dr. Horatio Nelson Jackson made across the United States in order to help promote sales during the early years of car manufacturing.
B. When Dr. Horatio Nelson Jackson accepted a bet that he could travel across the United states in three months or less, he did not realize the trip would cost him what was considered a huge amount of money in that time period.
C. Dr. Horatio Nelson decided to take a road trip in a car across the United States, and he did not let the many problems he encountered during the long journey prevent him from reading his goal in less than three months.
D. The road trip that Dr. Horatio Nelson Jackson made across the United States created a tremendous amount of media attention, and the interest from the public caused him significant problems during the journey.
Aha Number One: Narrative vs Informational Strategies and Purposes for Reading
After talking to the kids, we realized they were using a summary strategy for Narrative to make their decisions (Somebody Wanted But So). The passage was an ARTICLE from Cobblestone called 64 Days and $8000 Dollars. Students had selected a strategy unfit for the informational genre or to guide their purposes for reading. In addition, answer choices for informational genres will never follow a plot structure: informational summaries will reflect informational text structures.
Solution: We updated the SWBST strategy anchor chart to include the Something Happened And So Then strategy. We also added a call out that said: "Use your genre bookmarks determine the genre and set your purpose for reading."
Aha Number Two: Looking for Misconceptions
When we looked at the rationale for each item that TEA provided on the portal, we realized that we were missing a critical component. We were teaching how to summarize. We were NOT teaching how to tell if you are correctly summarizing. As Fisher, Frey, and Hattie explain, students must be capable of self-regulation and assessment to know where they are in meeting the success criteria. When I first started teaching, we learned something similar: providing non-examples so you can prevent misconceptions and pseudo-concepts. Samantha Johnson summed up what we discovered: She compared it to a store. You have the front end of a store with the merchandise. But you have the stock area and business end in the back. "Teaching the misconception allows a strategy for the back end. With only the front-end strategies (like SWBST), we are only teaching half of it." We realized we weren't teaching how to reason and discern what common errors look like. After all, on a multiple choice test, there are all kinds of ways to be wrong and only one way to be right.
Here's what we realized about answer choice A:
19% of us didn't realize that there was wrong information that was never stated in the passage. The passage NEVER states that auto companies sponsored road trips like the one Dr. Jackson makes. The passage NEVER states that the reason was to promote car sales. In fact, paragraph 9 states that the Winton Motor company only found out about such a trip until the media got involved. While they did offer to sponsor the trip, Jackson wouldn't take the money. We asked ourselves: Did students realize that they needed to be looking for false information? Did we teach them to do that? Nope.
Here's what we realized about answer choice B:
21A% of us didn't realize that a detail in the passage was incorrectly represented as the main focus of the article. This article does reference the cost of the trip. And it is reasonable that $8000 was a lot of money in 1903. And it is strange that they paid $500 dollars more for that car than it cost new. Yet, there is no evidence that Dr. Jackson didn't realize how much the trip was going to cost. In fact, there is evidence that he didn't even care about the cost. In paragraph 4, the text reports that he spends all that money just for a $50 dollar bet. He didn't really seem to care about how much it cost at all. Did we teach kids that they should be looking for details that misrepresent the gist and focus of the article as a whole? Were we teaching them that no evidence is evidence? Were we teaching them to look for contradictory evidence? Nope. We were teaching them SWBST. Even if we taught them SHAST, we'd still be missing what students needed to run the "summary store" that Samantha explained with her metaphor.
Here's what we realized about answer choice C:
Only 47% of us got this right. Why were less than half of us not getting this? We realized that some students were using the wrong summary strategy. And we realized that if they were using the correct roping and branding strategy, they'd be able to pair each "brand" with a section of the summary.
Uh, ya'll. The rope and brand strategy worked and also reflected the organizational structure of the article that would also help with question #50. (See Jennifer Martin's Original Lesson for Rope and Brand here. See the instructional resources for test taking processes based on her model here.
Here's what we learned about answer choice D:
14% of us didn't realize this was a fake summary that only focused on one part of the text or a detail. Answer choice D is partially correct, but only is represented in paragraphs 8-9. And like answer A, there is also false information. None of the evidence in those paragraphs indicate that the media attention was a problem for Jackson or Crocker. Did we teach kids to look for an answer choice that only summarized one portion of the text? Nope. We were teaching the components of a summary and not what it wasn't. (Almost all of the summary questions for a passage that I have examined have an answer choice like this, regardless of the grade.)
Solution: We started an anchor chart called "Misconceptions about Summary Answer Choices." We felt that this provided success criteria for HOW students were to apply their knowledge of summary in assessment situations. (And one of us may or may not have remarked that this is the kind of stuff that we wondered why no one ever told us about before.)
