EOC I Failures: A BLUNT Analysis

Part of my job involved data analysis to pinpoint solutions. This is the work I just completed as a model of interpreting an item analysis. 

Dear All,

I just analyzed the objective performance of students who took the English I exam at your campus. I have been able to pinpoint several items to guide instruction, test preparation, test taking strategies, etc. Bottom line: In both reading and writing sections of the test it appears

• that the students are reading pieces of the text, but are not reading the whole text
• that students are not reading for comprehension, but to find answers
• that students are not reading for deep thematic links, author’s purpose, or message
• that students have inadequate or inaccurate testing strategies
• that students are not applying concepts in reading or writing to their own compositions
• that students are not utilizing logical reasoning techniques to discard distractors
• that reading and writing instruction are not adequately informing each other in terms of how students compose and comprehend
• that instruction
• has focused on specific texts/novels/assignments as opposed to reading and writing processes and problem solving (Figure 19 and TEKS: SE +KS).
• could also be to misaligned to dual coded assessment items (sometimes items come from F19 and the general TEKS). Lessons have to be taught that combine the TEKS with the appropriate F19 skill, with the appropriate genre, and with  making comparisons using more than one text.
• has misapplied the connection between the Knowledge and Skills statement and the Student Expectation. If teachers teach lessons solely based on the SE, it is quite possible that the rigor from the KS and F19 were not a part of those lessons

I have analyzed each question,  provided a possible explanation of the problem students were having with the question,  and provided a solution. It is quite possible that English II students experience the same struggles. I will meet with you to show you how to complete an analysis for your other grades if you desire.

Your data nerd,

Shona Rose

Writing 2017 Item Analysis

1. Transition,  - 24% chose first sentence. Did they read all the options? Did they only read sentence 5? Did students write in the sample replacement to consider it in context of the overall development and progression of ideas? (Solution: Test Taking Strategy)
2. Word choice, transition, sentence to sentence coherence. 41% chose and answer that inaccurately contradicted the previous sentence in the passage.  Do students know the meaning and purpose for each transition? In fact - gives an example to extend or explain a previous statement.  However - establishes a contrast or alternate considerations to the previous sentence. Did student read the sentences before and after sentence 10? Did students write in the sample replacement to consider it in context of the overall development and progression of ideas? (Solution: Test Taking Strategy; Solution: Reading for Meaning; Solution: Lesson on function and use of transitions in papers authored by students.)
3. Combining sentences: 41% chose the right answer, 24% chose B, 24% chose C, and 12% chose D. This tells me that most of the kids were guessing. About half of those that got it right, still don’t know how to do this properly. The problem is in punctuation of complex sentences and clauses and inattention to logic/reasoning. For example D - Traveling shorter distances between charges don’t CAUSE the batteries to weaken over time. (Solution: Lessons in logical reasoning. Solution: Lesson in punctuation of complex sentences and clauses in context of student authored papers. The STAAR essays indicate similar inaccurate patterns represented by the inaccurate answer choices. Students are writing like the incorrect answers; therefore, they don’t know what’s wrong when they see it in an answer choice.)
4. Revise to add relevant detail; 65% chose the right answer 24% chose H. Students are not reading the DETAILS or thinking about logical progression. In this example, the sentence to be added includes a detail about traveling long distances being inconvenient. Putting the sentence after a sentence about weather is illogical. Weather does not make the batteries inconvenient. Students are not thinking about the passage as a whole in terms of the topic of the paragraph and the author’s purpose. (Solution: Reading for comprehension. Solution: Lessons on logical reasoning. Solution: connect revisions to author’s purpose in context of student authored papers. They don’t encounter errors like this in what they are reading – published pieces. The only place you will see stuff like this is in what students write.)
5. Delete redundant sentence: Those that chose A - 18% of them - chose to delete the main idea sentence. Those that chose C - 18% of them - chose to delete one of the primary reasons that support the main idea sentence. That the percentages are spread so evenly --I’d bet that about 18% of those that got it right were guessing too. (STAAR essays indicated that students are writing MANY redundant sentences in their own papers. Solution: Teach students how to identify redundancies in papers they have written. Solution: STAAR essays indicated that students were not including topic sentences in their own papers. Students should go back and analyze their own writing to identify main ideas. STAAR essays also indicate that students are not connecting supporting evidence back to main ideas. They list main ideas and then move to another one. Students need to learn techniques for developing and supporting a main idea effectively without repetition.)
6. Effective Thesis: 76% got this right. 18% chose the first response. Did they not read the rest of the answers? The first answer has the classic no-no of I’m-going-to-tell-you-about-I-hope-you -like-my-paper statement that is no good in an expository essay or article. (Solution: teach thesis no-nos.)
7. Sentence variety, awkward structures: 59% got this right. 12 % chose an answer that was a runon. 24% chose an answer that had incorrect verb tense and pronoun references. (Solution: STAAR essays indicate that students write sentences just like these incorrect examples. Students are trying to represent speech and complex patterns, but they do not yet know how to punctuate them. This kind of practice has to come directly from transcribed speech – actual student papers. You won’t find this kind of sentence combination on a grammar worksheet.)
8. Supporting Details: More students got this question incorrect than correct. 24 % answered correctly. 47% chose an answer that had nothing to do with the paragraph sentence 22 came from. 24% selected an idea that was not a topic in the paragraph sentence 22 came from. (Solution: Read for meaning. Read the whole piece for comprehension.)
9. Vague pronoun reference: 76% got this right. 12% chose an answer that indicated that advertisements (Commercials) were standing in lines to get Hamm to endorse their products in other ads. I don’t think the author was trying to personify commercials. 12% chose an answer that said that random women were standing in lines to ask Hamm if she would endorse their products in advertisements. (Solution: teach students to evaluate answers for logical reasoning. Solution: teach visualization and contextualization processes.)

Conclusions:

1. Students do not have the appropriate test taking annotation strategies for success.
2. Students are not reading the passages. They are only reading individual sentences. Students are not considering the context of the ideas or the logical progression of ideas and purpose of the author. Basically, they are reading the question, finding the sentence in the passage, and then looking at the answer choices. That won’t work. The test is not a seek and find on a Chili’s menu.
3. Students are not thinking logically or using reasoning to eliminate incorrect answers.
4. Students are not making revisions of these types in their own writing.
5. Teachers are not modeling how to make revisions of these types with class assignments and papers.
6. Some students were flat out guessing.

Reading 2017 Item Analysis:

For this section, I just focused on the items that were below 40% correct.

20. Author’s purpose, simile: 35% chose correctly; Honestly, I think this comes from having kids identify or include similes without having discussions about why you would use them in the first place. It looks like kids that chose different answers were only looking at the sentence the simile was in and didn’t use the contextual features available in the rest of the paragraph it was in, nor the paragraph that followed. There is ample evidence to support the correct answer choice.  (Solution: Read the whole passage for comprehension. Solution: evaluate the purpose and use of figurative language. Solution: teach students that evidence is found in multiple places in a text. Searching solely within the sentence or within the paragraph is never sufficient.)

31: Parenthesis, inference: 25% of the students got this correct. Most people blame this question on the term “asides” - that kids didn’t know the word, especially as it was applied to expository vs. drama. (Solution: Test Taking Strategy – use the dictionary.)

But let’s examine the other choices. 18% said that the data in the parenthesis simplified difficult concepts. One of the parenthesis repeated the word /dozens/. Dozens is not a complicated concept. The other one added a condition to the previous statement, that what you find in someone’s house depends on how people like to decorate as well as by chance. A neighbor’s taste in decorating is hardly a difficult concept. (Solution: Teach logical reasoning and visualization.)

29% chose an answer that /dozens/ and /neighbor’s taste/ added a counterargument. So...do they know what the argument was? Dozens does not counter a thing. Neither does your neighbor's taste. Remember, they can use a dictionary. (Solution: test taking strategy – use a dictionary; Solution: teach the elements of argument and evaluate the effectiveness of statements in context of student authored documents. The problems represented in questions like this do not appear in published, quality texts that we use for instruction. It is only in grappling with these struggles as a writer, that we learn how the elements of argument are to be evaluated and revised.)  

29% decided that /dozens/ and /neighbor’s taste/ added credibility to the article. Descriptive terms are usually not the things that help readers know if the writer is filling the reader full of smoke. Students in English I should know about counterarguments and credibility. These choices show that they don’t know these TEKS. (Solution: teach the components of argument and evaluate them in prewriting/planning, drafting, and revision.)

36: Comparison across two texts, deep thematic links:   35% got this answer correct. 35% chose an answer that didn’t match the stem. The stem asked students to explain the theme about TRADITIONAL libraries in both passages. The incorrect answer choice was talking about the NON-TRADITIONAL libraries. Students who got this wrong did not pay attention to the distinctions in the stem. (Solution: Test taking strategies. Students need help in picking out the distinctions between passages. This is done with a careful annotation of the question stem. Then, students need strategies for marking the passages to distinguish between them.)

37: Comparison across two texts, inference and text evidence: 53% of the students chose the same wrong answer. This is a multi step kind of problem. First, the reader must know what the message is from Hunger for Books - which was the first passage. Next, the reader must consider something that was said in a separate article  - which was the second passage - that might support that message in the first passage. Convoluted. But doable. The first passage’s message is that books are lovely - she want more of  them. The answer that 53% of the kids incorrectly chose talks directly about Little Free Libraries specifically - not the message or theme from the other passage. The first passage was not about being connected to others, nor was it about having conversations with other people. Not even connected to those ideas. As a matter of fact - all of the incorrect answers directly reference Little Free Libraries not the overall message of the first piece. Getting this question correct requires that students read the entire passage and understand it as a whole. Then they have to test each of the answers to see if they help support the message. (Solution: Each passage should be read to identify deep thematic links and author’s purpose/message. Solution: Students need to realize that some questions have several steps. Solution: Test Taking Strategy: Students need strategies for marking the passages and stems to make sure they know where to look for what.)

40: supporting details, text evidence; 41% got this question correct. 29% chose an answer that reference scientific studies and statistics. Which would make sense - writers use scientific studies and statistics to prove their points. But what did this author do after paragraphs 5-7? Paragraphs 8-11 are his personal experiences. Paragraph 11 is a transition. Paragraphs 12-15 share the ideas that come from researchers. Again, kids are reading isolated pieces without connecting them to how we understand the passage as a whole. (Solution: Read for meaning and comprehension. Solution: Understand the development of ideas into paragraph blocs/kernels.)

41: inferences; text evidence, purpose of graphics: only 18% of the students got this question correct.  47% said that the cartoon was important because the author talked about how important it was to prevent boredom. Uh. The author never talked about boredom in the text. The cartoon doesn’t talk about boredom either. The only thing I can think of is that cartoons are fun and not boring. But that has nothing to do with helping me understand the passage or why the author would have used it, especially at the beginning of the essay. (Solution: Read for meaning and comprehension. Solution: Student may not rely on experience or logic separate from the text. Solution: Read for deep thematic links and author’s purpose/message.)

44: author’s position, inference: 35% of the students got the right answer. 41% of them chose an answer that is an opinion related to the topic in the essay. It is probably even true. But the choice they picked has nothing to do with the author’s position - they missed the whole point of the essay! (Solution: Read for meaning and comprehension. Read for deep thematic links and author’s purpose/meaning.)

45. Text evidence, credibility and function: 29% of the students answered correctly. 29% chose an answer that indicated that paragraph was a restatement of the thesis- but it was smack dab in the middle of the section that described scientific studies and statistics! 24% chose and answer that indicated the paragraph was about historical context - it talked about one study, last year. Hardly historical context. (Solution: STAAR essays also indicate that students to not understand the structure of argument. They need to understand bloc/kernels of development and how each serve a purpose in developing the thesis. They can’t identify the right answer in this example because they are doing the same think in their own writing. Solution: Logical reasoning and contextual visualization are needed to discard incorrect answers.)

46. Point of view; perspective: 18% got this correct. The spread of the answer choices suggests that most kids read the first three choices, knew that they didn’t know what those things were, and slapped down an answer. They truly didn’t know this information. (Solution: Actually open the dictionary during the text. Solution: Keep a cumulative chart of all things read in class for students to consult: genre, topic, theme, point of view, message/purpose.)

48: function of dialogue, character development: 12% of the students chose the correct answer. 35% chose an answer that indicated they were not tracking the development of the character. Students read, “It’ll be dead soon” without thinking about the paragraphs above and below. Of course that sounded like a mean thing to say - IF you didn’t understand what was going on in the passage.  29% chose an answer that also indicated lack of focus on the entire context and situation - read alone, the sentences “There’s still a few left. There’s still one left” could be interpreted as sarcasm, especially with the italicized word. But in context - no one was being sarcastic. And no one was trying to be accepted. The character was trying to help Aunt Jessie feel better when the weather got cold. (Solution: read for  meaning – the whole piece. Solution: teach students that evidence is found throughout the passage. Reading one paragraph is never going to give enough context. Solution: Students may not rely on their own personal experiences or opinions without supporting text evidence.)

49: character development, inferences: 18% of the students chose correctly. 65% chose an incorrect answer that Aunt Jessie refuses to admit that she is sad. In isolation, you might get there by only reading the paragraph cited. The roses were planted when her child died...she is sad, but admits it openly with her actions... “each year” she grieves all over again as the roses die. The reader has to understand how this paragraph fits into understanding the trajectory and trail of the entire plot - not just the paragraph by itself. (Solution: Read the whole passage for deep thematic links and author’s message and purpose. One paragraph is never enough.)

50: text evidence, inference: 35% of the students chose correctly. 24% chose an answer that went a little too far away from the text evidence to make the inference. “I snuck it into the rosebushes in the yard, tying it to a branch.” Yes. This sentence shows what the narrator did after she became sensitive to how her aunt was feeling. 24% chose an answer that was also too many steps away from the text evidence. The answer suggested that the girls “wandered” and were “unable to make up their minds.” Some sensitive people have trouble making decisions because they are sensitive to what others might think. But there is not text evidence that sensitivity is causing the wandering and indecision. The correct answer - “It sent a shiver through me”  - is directly related to how the character responds to something. It is a direct cause and effect relationship that reveals how the character responded with sensitivity to a concrete action of her aunt. (Solutions: Teach students to ground their inferences in the action and text evidence. Sometimes they go too far. Solution: Read for meaning and comprehension. Read for deep thematic links, author’s purpose, and message.)

Conclusions:

1. Students are not reading to comprehend.
2. In addition, students appear to be reading the questions first. Or they could be reading the stem and only looking at particular sections of the text. Answer choices must all be considered by support or refutation from text evidence and reasoning based on understanding of the entire text and the author’s purpose, craft, and organization.
3. Students are going past the text to make unsupported inferences. In some instances they are taking isolated evidence and adding it to their experience without consideration of the meaning of the whole text. In other instances, they are relying on their own opinions and experiences as opposed to relying on text evidence and comprehension of the whole text.
4. Students are not attending to nuanced details required by the stem.
5. Students do not know certain concepts or how to apply them past identification.