Writing the perfect Scottish referendum question

On Friday the 11^th of May 2012 the Scottish Governments’ consultation on the proposed independence referendum closed. There has been a lot of debate over the currently proposed question so we thought we’d take a look at what makes the perfect question.  We applied our User Research skills to design a suitable alternative question for the referendum.

What makes the perfect referendum question?

There has been a lot of consultation by both the UK Government and the Scottish Government into the principles behind a good referendum question. Their findings, combined with the outputs from the Venice Commission’s report in 2005 which gave a review of referendum practices across Europe, has produced the following set of standards for the referendum question:

1. easy-to-understand
2. concise, unambiguous and to the point
3. legal, fair and decisive
4. must not lead the voter in one direction or another.

Ruth Stevenson, of Ruthless Research, summed up the situation perfectly, in her written evidence to the UK consultation earlier this year. Setting out the professional standards of question design outlined in the Market Research Society’s good practice guidance. She stated that the proposed question should be:

“Short, straightforward and clear […] balanced so as not to lead voters towards a particular response […and] presented in a manner that does not introduce bias.”

So that seems pretty simple on the surface of things, how hard can it be to pose a simple, fair yet decisive question?

What questions have been proposed?

To date there have been two proposed questions for the referendum from the Scottish Government.

The first question was designed in 2010 to both ask the independence question but also to satisfy the legal constraints around the referendum (powers to call and hold a referendum were not devolved to Scotland and so only the UK Government could legally do so at this time). The question that was put forward was:

‘Do you agree/disagree:

1. The Scottish Parliament should have its powers and responsibilities extended as described [on the ballot paper].
2.  The Scottish Government proposes that, in addition to the extension of the powers and responsibilities of the Parliament’s powers should also be extended to enable independence to be achieved.’

Even with the caveat that the question had to be worded in a certain way to make sure that it was legitimate e.g. saying ‘enable independence’ rather than calling for independence these are really poor questions to be put to the Scottish people.

Firstly, this question is really combining two queries in one which happen to be completely different constitutional issues which is not possible for a UK referendum: the devolution of power and independence. Whilst I am not part of the camp that believes that people are only able to answer one question per referendum, I do think that the set up here is both ambiguous and is certainly not easy to understand. In addition, the question does not state explicitly what would happen if the referendum returned a ‘yes’ vote.

The Scottish Government implied the question does not have to be fully explicit as there would be lots of campaigning prior to the referendum so that the population know what all the implications of each option. I, however, see this as a high-risk strategy for both campaigns as you are presuming that:

• parties campaign fairly and scrupulously
• people listen to the campaign messages.

However, the real issue with the question(s) is that there are actually 4 possible outcomes:

1. No additional devolution and no independence.
2. Additional devolution but no independence.
3. Additional devolution and independence.
4. No additional devolution but independence.

The last of the four options does not even make sense but is still a plausible outcome. Therefore, it could be possible to have four different campaigns for the four outcomes. Therefore, each campaign will give a slightly different message and therefore it introduces a higher risk of a mixed and confusing message being delivered to the electorate.

This first question was rejected at consultation stage and the UK Government agreed to remove any of the legitimacy issues so that the Scottish Government could set their question.

In January 2012 as part of the document issued by the Scottish Government called ‘Your Scotland, Your Referendum.’ that outlined the rules for campaigning and contained a draft ballot paper with this question:

Do you agree Scotland should be an independent country? Yes / No

 

On some levels this question is much better than the former issued in 2010, it is certainly simple and much shorter and unambiguous.  However this is a fundamentally unbalanced, loaded and biased question that leads the voter to the positive agreement answer as it asks ‘Do you agree…’ only.  This mistake is against so many statistical and research principles that a question relating to this situation is actually included in a standard GCSE statistics exam! It could be slightly more balanced by offering both options e.g. ‘Do you agree or disagree…’ but this is really just a quick fix.

The second problem with this question is an attitudinal question e.g. it is asking the way the voter thinks about the situation, by using the term agree, rather than what outcome the voter wants which would give a clearer and more definitive result.

Does the question really have that much impact on the result?

You may be wondering why it is so important how this question is phrased. It may seem to be a minor point as there will be plenty of campaigning and so people should have been made fully aware of all the implications prior to the vote taking place. However recent research conducted by Lord Ashcroft, as part of the UK Government Consultation, clearly shows the impact of the wording of the question on the response rate.

Lord Ashcroft polled the same 3090 Scottish adult respondents between the 26th and the 31st January 2012 with three questions and got varying results depending on the variation of the question.

So, you can see from this simple piece of research that by changing the question from an unbiased attitudinal question to a one asking about the direct outcome of the referendum introduces an 8% swing in the vote.  Therefore the phrasing of the question is of the utmost importance and as Ipsos MORI stated, in their written evidence to the UK consultation, earlier this year:

“It must be seen to be fair so that the losing side in the ballot do not try to discredit or challenge the result on the basis of the question wording.”

What is the perfect question?

The best referendum question for independence that I came across during my research was from the referendum of the Republic of Montenegro in 2006 who were proposing independence from Serbia and Montenegro. Interestingly they set the question by taking one pro-independent, one anti-independent and one neutral and asking them to write it together.

This is the question that was posed to the people of Montenegro:

‘Do you want the Republic of Montenegro to be an independent state with a full international and legal personality?

Yes / No ’

This is a good question as it is fair, simple and unambiguous. It does not lead the voter to one particular outcome and it outlines what the outcome of the vote will be. With this question each voter is aware of the implication of the vote that they cast. Also by using the word ‘want’ rather than asking for an agreement it forces the voter to make a choice.

Based on this question and all the other factors we have come up with our own question that we think is fair, simple, legal and unbiased for the Scottish Government to pose in the referendum:

‘Do you want Scotland to become an independent country with a full international and legal personality or do you want Scotland to remain part of the United Kingdom?

I want Scotland to be an independent country / I want Scotland to remain part of the United Kingdom ’

Our reasoning behind this wording:

1. It is simple and easy to understand with clear and explicit responses.
2. It is fair to both arguments.
3. It forces people to understand that they are choosing between independence and remaining part of the UK rather than just choosing independence therefore removing an ambiguity from the question.
4. It adheres to the Market Research best practice in terms of not leading the voter and will give clear results.

We think that this question is far better than the current proposed question and will be interested to read the outcome of the Scottish Government consultancy, due to be released in Summer 2012, as to what the rest of the population think about the question for the referendum.

What do you think of our question or can you write a better one? Please share your thoughts and ideas with us in the comments.