Literacy Measurement: A Holistic Framework

Measurement How-To's

This section outlines general best practices around how to carry out measurement exercises for literacy programs. It provides guidance and considerations how to carry out measurement activities, but is not designed to be a step-by-step guide to carrying out monitoring and evaluation.
Make sure to have a look at our helpful framing question here.

Fit-for-purpose learning assessments

This dimension provides discussion around types of literacy assessments and guidance around how to choose the activitiy to be carried out. Topics covered in the linked resources include benchmarking, progress monitoring, mastery checks, screening, impact evaluations, and formal/informal assessment, among others.

Measurement strategy

This dimension includes resources around designing a broad measurement strategy, including assessment timing, cadence, and when to include specific types of measurement activities.
Why is this important?
Logistical considerations for assessment, as well as an overall cadence of measurement activities are important for ensuring that your measurement exercise gets you the information you need. Proper planning can help make sure your measurement exercise is successful.
Considerations/Limitations
It is essential that data collection procedures are streamlined and efficient. Pilot testing in particular can help to check he readiness of materials, and logistical procedures, including transportation and communication, among assessor teams, field coordinators, and other staff. Ineffective methodologies mean missed data collection opportunities and the loss, or corruption of valuable data.

Adapting to local context

This dimension focuses on considerations for adapting tools to different local contexts, with a particular focus on language.
Why is this important?
Much of the literature on literacy assessment is grounded in English literacy (or more broadly, literacy in alphabetic languages). However the approach to measurement and evaluation may need to be adpated depending on the mother tongue of the learners, language of instruction, or in consideration of multilingual contexts. Other contextual factors-- such as location (urban vs. rural), proportion of out of school children, community literacy rates-- may impact the measurement approach used, and the corresponding level of effort required for carrying out the activity.

Sampling and census approaches

This dimension unpacks approaches to data collection, namely, considerations are census and sampling approaches. A census involves the collection of data from an entire population, whereas sampling involves collecting data from a smaller subset of a population.
Why is this important?
Census approaches ensure that data accurately reflect a population, since most or all of the research subjects are included in data collection, however they may be cost- and time-prohibitive. Sampling approaches that ensure representativeness can produce results that are just as reliable. Constructing a representative sample does require forethought, planning, and some understanding of statistcs. The resources included in this section provide some information and considerations with regards to sampling.

Inclusive measurement

This dimension focuses on strategies for ensuring that assessments are accessible and inclusive, from data collection to analysis. This may include strategies for including students with sensory, physical and learning disabilities, as well as readers of all genders, and linguistic minorities.
Why is this important?
It is not uncommon for hard-to-reach students to be left out of measurement activities.  These may be out of school children, who are hard to access because they don't attend school, individuals with disabilities for whom data collection tools might need to be adapted, and students who do not speak the language of instruction or assessment. When these individuals are left out of assessment activities, the status of their learning is left unknown.  While additional effort may be necessary for making measurement an inclusive activity, it is important for ensuring a complete measurement activity that provides the full picture of learning within a community or institution.

Validity & reliability

This dimension focuses on how to ensure that an assessment or measurement tool measures what it intends to measure (validity) and that it produces consistent results (reliability).
Why is this important?
All assessment tools should be checked by domain and assessment experts to ensure that the test has sufficient items, addressing an appropriate range of content, contexts and skills that are suited to the test takers’ levels of skill, to meet the claim that it assesses a particular domain. For example, a valid reading comprehension test should include a range of comprehension skills, different texts and at least 20 items that require reading the text(s) and cannot be answered correctly by guessing or general knowledge. Quality assessments include documentation and evidence of validity, with a clear definition of the construct and scope of the test. Pilot testing and analysis can check if items are assessing skills related to the relevant domain.

Assessments need to be reliable, so that all students have the same opportunity to demonstrate their skill and are not disadvantaged through unclear instructions, variable administrative procedures, bias in scoring responses or other factors that are not related to the skills being assessed. Administration and scoring procedures need to very clear in order to be consistent.  Training is generally essential. Pilot testing will also allow you to undertake quality assurance checks to ensure that tests are reliable.  Different aspects of reliability can be evaluated such as internal consistency, or inter-rater reliability, using a range of metrics.

Analysis & use of results

This dimension focuses on the interpretation of data and analysis, particularly with regards to understanding what learners do or do not know, as well as considerations around the use of findings from a measurement exercise.
Why is this important?
Numbers only tell us part of the story. Contextualizing your results and making them understandable to key stakeholders is important for operationalizing and learning from your measurement exercise.  Moreover, thinking through the use of your data before you collect it will help ensure its relevance, and ensure that the results of your measurement exercise are used, rather than sitting around collecting dust.

Engaging the community

This dimension covers ways to engage community members in the process of measurement, from community-led assessments (CLAs) to ways to strategies for reporting out on results to communith members and stakeholders.
Why is this important?
Often times community engagement stops at the programmatic level, however there are ample opportunities to include community members in the measurement process. Engaging communities in assessment and measurement activities can help raise awareness around learning levels and help ensure that underrepresented groups are included in assessment and measurement activities.

Framing Questions

What sort of resources do you have to dedicate to your measurement exercise?

While this should not be the primary consideration that influences your measurement approach, it is relevant with regards to the level of effort required, the number of measurement activities carried out and the methodologies used.  For example, more experimental approaches and impact evaluations often require more resources, as do more comprehensive meausrement exercises that include multiple assessment periods.  

What is the level of expertise and comfort level of your staff with regards to assessment, evaluation, measurement, etc?  

Will program staff need to support your measurement activities and if so, what are their comfort levels?   This may help you determine whether you need to bring in outside assistance, as well as what kind of training might be necessary.  More complex, experimental approaches require more expertise.   Understanding this before starting will help you develop a plan for having a runway leading up to your measurement exercise.

Who are your intended audiences and how will you use the data you collect?

Understanding and articulating your goals around usage and dissemination of the data will inform your overall approach.  Additionally, identifying the stakeholders who will be consuming the data and/or involved in the process is key.

Download our handy worksheet to guide you through these questions.
want more updates?

Subscribe to our Newsletter

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.