Evaluation

Qualitative and quantitative techniques were used in the Digital Dialogues evaluation to capture:

• Scope for applying different ICT techniques and tools across central public engagement exercises;
• Practice of managing ICT by policy teams to support their public engagement activity;
• Short-term impact of ICT-enabled consultation on policy outcomes;
• Government and public attitudes towards communication and consultation through technology.

Statistical data about site visit and usage traffic was gathered at a server and site level. Demographic data about participants was recorded at registration. As far as possible a set of standardised questions were used but on occasion policy teams adapted the questions to meet their departmental standards [see Draft Guidance 1].

Surveys, incorporating both closed and open-ended questions, were used to capture attitudinal and behavioural data from participating members of the public [see Draft Guidance 2]. For forums, surveys were introduced at the beginning and end of the consultation; for both blogs and webchats a single survey was available for completion toward the end of the evaluation period. All public participants were invited to complete surveys (both the pre- and post-consultation where applicable). Participants were sent a maximum of three requests to complete surveys. Surveys could be completed on- or offline; no zero-response rates were recorded. Samples were self-selecting and thus not representative of the entire UK population.

Feedback from government participants was collected at group and individual training sessions and interviews. At least two government participants from each exercise were obliged to complete questionnaires and participate in interviews, and this was achieved for each of the case studies. Wherever possible, the findings gathered were compared with those from consultation activity conducted by conventional, offline means (for example, opinion polling or citizens juries).

Evaluation periods were set at a minimum of three weeks and a maximum of 20. It was not possible to pre-determine sample sizes and response rates before each exercise began. Ultimately, sample sizes differed for each case study.

The impact of ICT-enabled consultation on policy is a challenging and under-developed area of study. Established practice in online consultation is to define feedback processes and provide responses to contributors soon after the closure of a consultation. Therefore, it was possible in some instances to observe the short-term impact on policies, and their developers, as a result of Digital Dialogues exercises. Practically all of the Digital Dialogues case studies observed consultation undertaken at the developmental stages of policy-making, thus putting evaluation of long term impacts beyond the scope of this particular initiative.