Choosing What to Measure

Perhaps the most difficult aspect of evaluating any program or intervention, especially your engagement efforts, is deciding what to measure and how.

In some ways the well-known SMART framework is still the most relevant.

Specific - What exactly do you want to do, with or for whom?

Measurable - Can you measure whether or not you have achieved the objective?

Achievable - Is the objective achievable? In particular can you get it done in the time you have available, within your budget and within the prevailing political/institutional climate?

Relevant - Will achieving this objective contribute to the delivery of your overall aim and support your/your funders'/your institution's goals?

Time-bound - When do you want to achieve this objective and/or when do you think you will be able to achieve this objective?

Inputs, Outputs, Outcomes, and Impacts

Increasingly, these distinctions are important to funders of all types (i.e. public, foundations, donors etc.) and in many nations. Knowing the differences, and being able to speak to them is critical to telling your story effectively.

  • Inputs are those things that you do as a group or organization to achieve your goals. Examples include staff time, grant-writing, web development, mailings, survey development, conducting meetings or other events, and evaluation. (formative or process)
  • Outputs are the results of your inputs and activities. How many people participated? Did you reach your intended audience? How many sessions were delivered? What are the tangible products (e.g. fact sheets, curriculum etc.) (formative or process)
  • Outcomes are the overall benefits or changes on the participants you witnessed - in the short and intermediate terms. Did people change behavior or attitudes? Did they acquire new knowledge of skills? (formative and/or summative)
  • Impacts are the overall effect or influence of your program or intervention. These are the long term effects on the social, environmental, or economic conditions you sought to address. (summative evaluation)

There are a wide range of resolveuid/a8feedd3229147449f9859492d458376 to help you think through which of these are most applicable to your program.

Indicators and Metrics

There is a wide range of possible metrics you may want to consider for your engagement efforts and, of course, many will be derived directly from the specific goals and objectives of your particular program. Some of the most common metrics used in engagement evaluation studies include:

Process Metrics

  • Representativeness
  • Interactions
  • Inclusivity
  • Comfort
  • Participation rate
  • Identification of common goals
  • Fairness
  • Satisfaction
  • Perceived openness
  • Effectiveness (process and methods)
  • Transparency
  • Incorporation of values and beliefs into discussion
  • Trust
  • Communication
  • Process flexibility
  • Continuity

Outcome Metrics

  • Policy/decision influence
  • Time to develop solutions or regulations
  • Reduction of legal challenges
  • Agency or organization responsiveness
  • Trust
  • Social, economic, environmental impact
  • Participants values/opinions
  • Conflict resolution
  • Volunteer time and effort
  • Effectiveness and cost effectiveness
  • Savings or resolveuid/a8feedd3229147449f9859492d458376 generated
  • Effect on planning process

(Adapted from Rowe and Frewer, 2004).

It is also important to remember that your choice of metrics or indicators can, and in many cases should, be developed in concert with participants in your engagement efforts. This can often provide you with important insight regarding what's most important to measure from their perspective as well as enhance 'buy-in' and connection between the project and your target audience.

Data Collection Methods

When developing methods for collecting your data, you will do well to consider a number of important questions. Two of the most important are:

  • How suitable is each method to your intended audience?
  • What information is the most critical and how can you make it easiest for your team and respondents to collect this data?

Collection methods

  1. Measuring short-term changes. What changes in attitudes, knowledge, skills, or intentions can be documented?
    1. Pre/post survey
    2. Post event or intervention survey or focus groups
    3. Skill assessment/demonstration
    4. Observations
    5. Practical application exercise
    6. Willingness to participate in follow-up activities
  2. Measuring intermediate changes. What do participants do, what behaviors actually changed after a specified length of time?
    1. Logging behaviors
    2. Monitoring of completion of follow-up activities
    3. Surveys
    4. Focus groups
    5. Observation
  3. Measuring long-term impacts and outcomes.
    1. Secondary data
    2. Surveys/ follow-up with participants
    3. Interviews and focus groups

Regardless of which method - or combination of methods - are appropriate to your project, your task will be easier and in the long-run more effective if you have a well-conceived evaluation plan as early in the process as possible.

References


Rowe, Gene, and Lynn J. Frewer. "Evaluation public-participation exercises: a research agenda." Science, technology & human values 29, no. 4 (204): 512-556.