OCTOBER 2015

Thinking about "Balancing” Procedures, Concepts, and Problem Solving in Summative Mathematics Assessments: What’s in Your Blueprint?

 

Earlier this year, I wrote in my newsletter about six central alignment questions that a thoughtful analysis of an assessment can answer.

 

  1. To what degree is there a strong content match between the test items/tasks (and the test as a whole) and the state’s college and career content standards?

  2. Are the test items/tasks (and the test as a whole) more rigorous, less rigorous, or of comparable rigor and complexity to the intended rigor of the state’s college and career content and performance standards at each grade level?

  3. Is the source of challenge for test items/tasks appropriate?  That is, is the hardest thing about the test items/tasks that which the item/task is targeting for assessment; or is there an underlying factor making the item more difficult to access or comprehend than it should be? (in math assessments, this might be a reading challenge.)

  4. Are the texts/stimuli for reading/writing/literacy assessments of appropriate length and complexity for this grade level? And does the balance between literary and informational texts appropriately reflect the intent of the college and career standards and have coherence with the state’s test blueprint?

  5. To what degree does the content coverage and test design (of this assessment) include assessment of all of the major strands or claims (i.e., evidence-centered designed assessments) as described in grade-level eligible content standards in English language arts/literacy and mathematics at corresponding grade levels?

  6. To what degree does the test blueprint and set of items (test as a whole) emphasize content and performance expectations (e.g., application of mathematical practices to mathematics concepts and procedures; increasing text complexity) to elicit evidence that students are preparing to perform more sophisticated, college and career-type work in the future?

 

In this post, I’ll unpack the concept of balance of emphasis in high-quality mathematics assessments in order to, in part, answer alignment questions 1, 2, and 6.

 

I think most people might interpret “balance” in an assessment as equal emphasis – 1/3 of the assessment on procedures, 1/3 of the assessment on concepts, and 1/3 of the assessment on mathematical applications. This approach to balance of emphasis is far too simplistic. For example, consider that test items or tasks that assess problem solving SHOULD include some routine word problems (DOK 2), as well as more in-depth (non-routine) problem-solving tasks (DOK 3 or DOK 4). However, as I work with schools to develop their local assessment systems, I see many mathematics assessments with routine word problems, but few or no items/tasks that actually require strategic/deeper thinking in order to solve problems. For a summative mathematics assessment to be regarded as “a high quality” assessment, we need to think more deeply about what the appropriate balance of procedures-concepts-applications might look like at each grade level.

 

A starting point is to think about a “balance” that reflects the balance of what the standards for the grade level imply…but I don’t think that is enough to consider, being that mathematics content standards are mostly written at DOK 1 (procedures) and DOK 2 (concepts). We need to also examine to what degree the assessment of mathematical practices are applied to content/concepts.

 

When asked about what a “good balance” in a summative math assessment would be, I recommend a shift that reflects fewer routine and more non-routine applications as grade levels increase. This often can be achieved by replacing some of the existing “lower level” (recall and reproduce) items with items/tasks that uncover student thinking and require students to use mathematical reasoning to support their solutions. These deeper understanding item types generally provide more information about what students actually know and should get more than one score point! In other words, I am talking about the balance of score points in an assessment. Deeper thinking tasks should count more in the final score than the items assessing routine procedures and applications.

 

A suggested balance of emphasis for elementary grades: performing isolated procedures (about 15%), demonstrating conceptual understanding (about 45%), and problem solving (about 40%), broken down in this way…

         35% performing procedures, including performing operations and algorithms in isolation (DOK 1 – 15%)  and application of procedures in the context of routine word problems (DOK 2 – 20%)

         45% demonstrating conceptual understanding (DOK 2) – where explanations/solutions are based on explaining concepts (e.g., identifying structures or patterns observed), selecting an appropriate tool or strategy for the context presented, providing examples/non-examples, development/ use /interpretation of models/representations, etc.

         20% problem solving and reasoning (DOK 3 or DOK 4) where the application of procedures and concepts are in the context of non-routine problems - analyzing reasoning used (critiquing reasoning of others), or developing mathematical arguments – most likely performance tasks at DOK 3, illustrating use/application of concepts and procedures in non-routine problems

 

A suggested balance of emphasis for middle school grades: performing isolated procedures (about 10%), demonstrating conceptual understanding (about 45%), and problem solving (about 40%), broken down in this way…

         30% procedures, including performing operations and algorithms (DOK 1 – 10%) and application of procedures in the context of routine word problems (DOK 2 – 20%)

         45% conceptual understanding (DOK 2) – where explanations are based on explaining concepts (e.g., identifying and explaining structures or patterns observed), development/ use /interpretation of models/representations to demonstrate concepts, etc.

         25% problem solving and reasoning (DOK 3 or DOK 4) where the application of procedures and concepts are in the context of non-routine problems, analyzing reasoning used (critiquing reasoning of others), or developing mathematical arguments

 

A suggested balance of emphasis for high school grades: performing isolated procedures (about 10%), demonstrating conceptual understanding (about 40%), and problem solving (about 50%), broken down in this way…

         30% procedures, including performing operations and algorithms in isolation (DOK 1 – 10%) and application of procedures in the context of routine word problems (DOK 2 – 20%)

         40% conceptual understanding (DOK 2) – where explanations are based on explaining concepts (e.g., identifying structures or patterns observed; appropriate use of models to demonstrate concepts)

         30% problem solving and reasoning (DOK 3 or DOK 4) where the application of procedures and concepts are in the context of non-routine problems, analyzing reasoning used (critiquing reasoning of others), or developing mathematical arguments, etc.

 

Have you been developing local assessments or examining assessments you currently use? I’d love to hear your questions or feedback on this suggested balance of emphasis proposal. I’d also be happy to send you a sample alignment coding tool for mathematics assessments. You can contact me through my website www.karin-hess.com .

 

Karin Hess, Ed.D, is a recognized international leader in developing practical approaches for using cognitive rigor and learning progressions as the foundation for formative, interim, and performance assessments.

 

Read More...

  • Facebook Social Icon
  • Twitter Social Icon
  • LinkedIn Classic

© since July 2014 Karin Hess, Ed.D.