How do you measure the impact of stronger citizen participation?

12 posts / 0 new
Last post
How do you measure the impact of stronger citizen participation?

To help start the conversation and keep the focus of this discussion thread, please consider the following questions:

  • By implementing your tactics, what has been the impact on systems, policies, governments and communities? 
  • How do you know when you have successfully strengthened the participation of citizens in local governance? 
  • What are the indicators, milestones and measurements that you use to better understand your impact?

Share your experiences, thoughts, ideas and questions by adding a comment below or replying to existing comments!

Measuring impact of stronger citizen participation.

This comment is inteded to start of a conversation. It is not intended to be a paper on impact measurement :)

The first step in measuring impact would be building a theory of change around citizens participation. This implies answering questions like

  1. How do we know that citizens are participating more?
  2. What change do we expect to see in the short term, if citizens participate?
  3. What is the assumption(s) we are making when we say that (change will happen)?
  4. How realistic are these assumptions in the context in which we are working?
  5. How will the short term changes, evolve in the longer term (impact)?
  6. What are the other factors that will come into play at that stage? Over which do we have control? Which can we influence? Who are our allies? Over which we have no control? 

This theory of how change happens  is actually the basis of ones who programming - and in effect also the reason why one is promoting participation by citizens in any case.

Once you have a theory of change, you need to enusre that it remains a 'live' document. That means it needs to be revisited at periodic intervals just to test that the changing context does not / does affect the assumptions.

The next step is to identify the indicators that will tell you whether the short / long term change is happening? Developing indicators, is NOT rocket science. All you need to answer is the question "how will anyone know that change has happened?" For instance,

  • the only way to know whether someone has capacity to speak in public is to see them do so.
  • the only way to know if women have control over assets is to see if they are engaged in the market place in purchase and sale of assets.

With active citizens too, that is what you need to identify - answer to the questions (like)

  1. How will anyone  know that the citizens are participating more?
  2. How does anyone know that changes are happening?
  3. Is there an evident connection between the change and what we have done? 

The best part of citizens participation in governance is that you can never accurately predict what path the process will take. You can almost rarely predict the exact nature of demands that active citizens will make. You may have started the process to get increased participation in budget making, citizens may make demands for better school infrastructure. That means that one need to remain extremely alert to the changes that are happening, positive and negative.

Without intending this to be long, it has gone on a bit. So shall stop. Additions / Challenges welcome 

How to measure the impact of citizens' participation

Hi All,

As some of us mentioned during our teleconference, it is not always simple to measure the impact of most of the initiatives/campaigns in social campaigns.

For example, Fahamu is leading the campaign (in Kenya) of the State of the Union, Africa (http://www.stateoftheunionafrica.net/) coalition and one of the key legal instruments that we are using is the African Charter on Democracy, Elections and Governance. In its various articles, the charter calls for citizens' participation in political process and at al levels i.e. counties/zones/districts, provinces and at the national level.

Some of the desired impact may not be noticed till after many years. For example, if the objective is to increase the number of voters, it may not be possible to verify if the number has gone up or down unless there is another election time (after four to seven years depeding on the country).

Some progress can be noticed in a much shorter time: giving inputs into the drafting process of the constituency/national budget is a process that is repeated every year. By looking at the number of inputs/comments/ or level of participation in this process can help the project coordinator/leader to assess if there is any progress or not.

For every project, there are phases after which an evaluation is conducted (after three or four months). By setting short term goals such as "To motivate citizens to report on going abuses of power by local leaders, police men",.... After three or four months, the project/campaign leader can look back and see if there is  any progress that has been achieved.

In general, in the planning process, there are short/long term objectives that are set and for each objective/goal, there is an indicator/an event/ a sign/a reaction or action/change that will show that that objective/goal has been achieved.

A final example (going back to the State of the Union campaign), we are planning to work with other stakeholders in Kenya to push the Government of Kenya to ratify the African Charter on Democracy, Eletions and Governance. There may not be a ratification in one year (if the goal is to have the charter ratified), but some steps towards that direction can be considered as postive impacts. On the other hand, even if the charter is not ratified, the government can commence some programmes that are in line with what the charter advocates for. In this case, should we say that we did not achieve our objective? Or what do others think? I think that it would be a positive sign from the government.

Thank you,

Yves

 

Measuring the impact of stronger citizen participation

Media , considered as the fifth pillar of the state, is playing a crucial role to create political awareness among the citizens of Pakistan and assumed a role as a watchdog of democracy by providing unbiased information to the viewers through various talk shows, current and international affairs programs and comedy shows on political themes.

In these programs, the viewers are allowed to participate in the live shows through text messages, emails and using social websites. 

Media is playing its positive role in strengthening a democratic institutions in Pakistan.. Every Government has tried their level best to curb media but now it is out of control and now Media is in a good position to defend itself. The success of Long March to restore the Chief Justice of Pakistan was also made possible due to media role. The Restoration of judiciary was the Hall Mark for media in Particular and people in General.#mce_temp_url# 

The major impact of the media, particularly citizen journalism can also be used as a tool or tactics not only to promote human rights but also measures the impact of citizen participation. For instance one of the cases highlighted through a mobile camera video of Wahida Shah, Member of Provincial Assembly, manhandling in by-polls. Shah was captured by camera while slapping a lady returning officer at a polling station. The Supreme court of Pakistan took a suo- moto notice against the ruling party Candidate and hence the Election Commission of Pakistan banned her for two years and defunct her wining in the by-polls. The link to the video is attached

 

Measuring impacts of participation

This post may be more academic than is appropriate for this dialogue, but here goes: 

Studies of participatory budgeting (PB) have documented important effects on the quality of local democracy, on government spending patterns, on living conditions, and on individual participants.  These effects have not been seen in all cities that use PB, but there is growing evidence that where PB is practiced well, over time it has significant, positive, measurable impacts. Several studies have shown a decrease in corruption (number of irregularities; see Zamboni 2007) and clientelism (citizen use of particularistic versus collective strategies; Wampler 2007) and an increase in the number of local civic associations and whether they use internal elections (Abers 2000; Baiocchi 2002).  Other studies show that cities with PB tend to spend more on healthcare and basic sanitation (for Brazil, Gonçalves 2009; for Peru, in Spanish, Banco Mundial 2011); Gonçalves (2009) also finds that cities with PB reduced infant and child mortality rates faster than cities that did not, though Boulding and Wampler (2010) do not find such strong effects.  And Lerner and Schugurensky (2007) and Ford (2009) find -- through surveys and interviews -- that individual participants gained knowledge not only about the budget process and how local goverment works but also but their rights as citizens.

References (only some of these are available on line):

  • Abers, Rebecca. 2000. Inventing Local Democracy: Grassroots Politics in Brazil. Boulder:  Lynne Rienner.
  • Baiocchi, Gianpaolo. 2002. “Synergizing Civil Society: State-Civil Society Regimes in Porto Alegre, Brazil.” Political Power and Social Theory, 15:3-86. 
  • Banco Mundial.  2011.  Evaluación del Presupuesto Participativo y su relación con el presupuesto por resultados.  Documento Resumen.  Lima: Banco Mundial.
  • Boulding, Carew, and Brian Wampler. 2010. “Voice, Votes, and Resources: Evaluating the Effect of Participatory Democracy on Well-being,” World Development Vol. 38, No. 1, pp. 125–135.
  • Ford, Alberto. 2009.  “¿Existen Condiciones Previas del Éxito de un Presupuesto Participativo? Aportes para la Discusión desde la Experiencia de Rosario.” Paper presented at IPSA World Congress of Political Science, Santiago, Chile.  July.
  • Gonçalves, Sónia. 2009.  “Power to the People: The Effects of Participatory Budgeting on Municipal Expenditures and Infant Mortality in Brazil,” London School of Economics.  February.
  • Lerner, Josh y Daniel Schugurensky. 2007. “La dimensión educativa de la democracia local: el caso del presupuesto participativo”. Revista Temas & Debates. Facultad de Ciencia Política y Relaciones Internacionales, Universidad Nacional de Rosario.
  • Wampler, Brian.2007. “Can Participatory Institutions Promote Pluralism? Mobilizing Low-Income Citizens in Brazil.” Studies in Comparative International Development 41 (4).
  • Zamboni, Yves. 2007.  "Participatory Budgeting and Local Governance: An Evidence-Based Evaluation of Participatory Budgeting Experiences in Brazil."  http://siteresources.worldbank.org/INTRANETSOCIALDEVELOPMENT/Resources/Z...
  • Banco Mundial. 2008. “Rumo a um Orçamento Participativo mais inclusive e efetivo em Porto Alegre.”  Washington, DC:  Banco Mundial.  February.
  • 2011.  Evaluación del Presupuesto Participativo y su relación con el presupuesto por resultados.  Documento Resumen.  Lima: Banco Mundial.
  • Boulding, Carew, and Brian Wampler. 2010. “Voice, Votes, and Resources: Evaluating the Effect of Participatory Democracy on Well-being,” World Development Vol. 38, No. 1, pp. 125–135.
  • Banco Mundial. 2008. “Rumo a um Orçamento Participativo mais inclusive e efetivo em Porto Alegre.”  Washington, DC:  Banco Mundial.  February.
  • 2011.  Evaluación del Presupuesto Participativo y su relación con el presupuesto por resultados.  Documento Resumen.  Lima: Banco Mundial.
  • Boulding, Carew, and Brian Wampler. 2010. “Voice, Votes, and Resources: Evaluating the Effect of Participatory Democracy on Well-being,” World Development Vol. 38, No. 1, pp. 125–135.
Citizens learn about their rights by participating

Ben Goldfrank wrote:

And Lerner and Schugurensky (2007) and Ford (2009) find -- through surveys and interviews -- that individual participants gained knowledge not only about the budget process and how local goverment works but also but their rights as citizens.

Fascinating!  Thanks for synthesizing the researching findings in such an accessible way.  I found this last type of impact especially interesting, and perhaps unexpected.  One thing I have learned in this dialogue is that while it is important to educate citizens on their role in local governance (and that this is an important step is helping citizens participate in local governance) - the process of actively participating is perhaps the most effective way of passing along this education/training on their rights and responsibilities in this context.  Thanks for sharing these research findings, Ben!

Outcome mapping as a monitoring approach to active citizenship

One method worth considering to measure whether change is happening around citizen participation is outcome mapping. http://www.outcomemapping.ca/ By measuring behaviour change of a number of different boundary partners to a programme (stakeholders) against a defined outcome statement (vision for a stakeholder's behaviour) at different levels of progress marker (indicators)- expect to see (output level) like to see (change happening), love to see (transformational change), its possible to pick up trends of change and if it is transformational.  Oxfam is using this approach in our Tanzania active citizenship programme, and we have found it really helpful in both measuring indicators that are meaningful to our programme and deepening our understanding of power at local level, what is able to change and what structures or hidden power are stopping it from changing. We have adapted the approach slightly to ensure we are capturing unexpected change- both positive and negative deviations from what we expected to happen. 

Outcome mapping has been really good for measuring behaviour change and we have found it a great tool for integrating monitoring fully into management decisions in what needs to be an adaptive, opportunistic programme to be successful.  Any experiences from others on outcome mapping to share?

However, it doesn't easily capture the impact of the changes in behaviour we are seeing. To that end we intend to commission an impact assessment that will creatively assess what the consequences of increased citizen participation have been. Any experiences from others on what methodologies work best in impact assessment for this type of programme?

Surveying the Participants

Most of the Portuguese municipalities who have been nurturing Participatory Budgets (PB) throughout the last years invite the citizens to participate in the process using the internet. Some of them put really a strong emphasis in e-participation, even considering that this type of approach targets a different group of people than those who usually participate in in-person meetings or regular assemblies. That is the case of Lisbon, the capital city. In more practical terms, citizens are expected to register in a particular portal on the internet. Afterwards they are able to develop many activities and grant access to relevant information. This sort of arrangements allows the municipalities to survey the participants in the end of the season, but also to have a track record of who is registered in the web portal and who is indeed participating (e.g. submitting a proposal, voting in a proposal, etc.).

A wide range of indicators and concepts have been tested so far. These surveys usually are designed to measure items such as the level of satisfaction with the participatory process, the level of trust in politics and in institutions, opinions and attitudes towards the PB, where the information about the PB was accessed or the level of participation demonstrated by every respondent across a range of different yet comparable activities. There are not so many examples of PBs in Portugal if one considers the total number of municipalities. Nevertheless, it is possible to find several successful experiences, not only on municipalities, but also in parishes, a smaller level of the Portuguese local government. In general, often it is possible to read improvements in the level of participation and the PBs are becoming more successful. Another important output is the fact that even in times of austerity and major cuts in the transfers from the central State the amount of resources made available for PBs has been steadily increasing in many cities. In other words, apparently many politicians understood the positive impacts of PBs and they are not able to discontinue these initiatives.

Politicians are unable to discontinue participatory budgeting

ff wrote:

Another important output is the fact that even in times of austerity and major cuts in the transfers from the central State the amount of resources made available for PBs has been steadily increasing in many cities. In other words, apparently many politicians understood the positive impacts of PBs and they are not able to discontinue these initiatives.

Thanks, Francisco, for sharing this information on how you have used an online platform to empower citizens and to track, measure and survey the participation! 

I am curious to learn more about this last line of your comment - that "[politicians] are not able to discontinue these initiatives."  To me, that sounds encredibly empowering and I am interested in learning more about why you think politicians are unable to discontinue the PB initiatives.  Is it because they would lose popularity among their constituents?  Is the participation of citizens absolutely required now to create and implement government budgets?  Thanks, Francisco!

Minimizing losses, fetching opportunities

The starting of a PB in every municipality always constitutes a political decision. In other words, it is something that it has to be decided by the board of a municipality and particularly by its mayor. In this respect, we find often two major ‘pathways’: committed politicians who think of the PB as an instrument to improve local governance, and those who implement PB without any sort of political will, probably because is a trendy or a ‘must have mechanism’, or simply because the surrounding municipalities adopted this type of instruments and they want to be at the front line as well. We all know that political will is a mandatory prerequisite for a PB to flourish. That is why many process collapse after their very first cycle, first of all because they have been conducted by sceptical people. Also in our context there is a strong tradition of top-down management in the municipalities, so what the board decides or thinks about a certain issue is important in terms of the motivation of the local officers. However, even without a strong commitment, sometimes a PB can lift off (e.g. high participation rates) and then they are not able to stop it anymore.

Every single decision is of course balanced in terms of popularity among their constituents, but there are several mayors who realised that the PB can be a very valid tool in terms of their own accountability and in terms of the transparency in local governance, especially because of all those issues on corruption and the image of local governance postulated in the media. Regarding the last question, the participation of citizens is far from being absolutely required for implementing budgets at the local level, but some mayors prefer to use the PB as a way to explain the existent limitations and the options they had to make while dealing with very scarce resources. For other issues than the budgets, the consultation of citizens is written in the law and usually is defined a certain amount of time for that.

PB can help explain realities of gov't budgets to citizens

Thanks for the reply, Francisco!  It makes sense to me that using participatory budgets help politicians explain their limited resources to their constituents.  I hadn't really thought of that benefit for local government officials before.  Thanks!

Elsewhere, PB is often discontinued or weakly implemented

It is great to learn that participatory budgeting is flourishing in Portugal.  Unfortunately, it does not seem to be as consistent in Latin America.  In Brazil, the number of cities with PB has grown over the yeas to a plateau of about 200 cities with PB today, but generally each new mayoral term, about half of the existing PBs are not renewed.  The same rate of (non-)continuation is true in many other countries where PB is not mandatory.  Where it is nationally mandated by law, in Peru and the Dominican Republic, about one-fifth of the local governments do not implement it, and not all of the remaining cities do so effectively.  In a recent paper on PB and "urban sustainable development," I wrote:   "One conclusion that emerges from a comparative look at participatory budgeting in Latin America is that where PB is best able to contribute to sustainable development (Brazil), it is not especially widespread across municipalities and tends not to persist, and that where PB seems to be less effective thus far (Peru, the Dominican Republic), it is often more widespread and sustainable because it is mandated by law.  The two-fold challenge for advocates of urban sustainability is therefore to encourage the expansion and longevity of PB in countries where it is not mandated by law and to promote more effective PB in countries where it is mandated by law."  Lack of municipal resources remains a major challenge in many Latin American cities.

Topic locked