Evaluation Methods
Qualitative and quantitative techniques were used in the Digital Dialogues evaluations to capture:
- Practices developed by case study teams to set up, manage and evaluate their online engagement exercises;
- Demographics, attitudes and behaviours of the sites’ user communities;
- Short-term impacts on policy processes.
Evaluation periods varied depending on the duration of the case studies.
Data about website traffic was gathered by statistical software. This captured routine information for each case study about, for example, hits, unique visitors, return visits, visit duration and popular pages.
User demographics (age, ethnicity, gender, location) were collected where registration was required to participate. As far as possible a set of standardised questions were used [see Appendix A], but on occasion policy teams adapted the questions to meet their departmental standards. In some case studies, submission of demographic data was optional. The blog-based case studies did not require registration, and therefore no demographic data was gathered.
Surveys, incorporating both closed and open-ended questions, were used to capture attitudinal and behavioural data from users [see Appendix A]. For the forums, surveys were introduced at the beginning and end of the exercise; for both blogs and webchats a single survey was available for completion toward the end of the evaluation period.
All users were invited to complete surveys (both the pre- and post-activity where applicable). Users were sent a maximum of three requests to complete surveys. Surveys could be completed on- or offline. Respondents were self-selecting and thus not representative of the entire UK population. Some questions were open-ended, others used scales; some were compulsory and others were optional.
It was not possible to pre-determine sample sizes and response rates before each exercise began. Ultimately, sample sizes differed for each case study, and in some cases were very low; no zero-response rates were recorded.
Feedback from government participants was collected through group and individual training sessions, post-activity surveys and semi-structured interviews. A minimum of two case study team members were required to complete surveys and participate in interviews; and this response rate was achieved for each of the case studies.
In some cases, it was possible to determine the short-term impact of the exercise on policy-making. This assessment was based on the testimony of the case study owners, and is referenced in the individual case study reports. However, given the duration of our evaluation periods, it has not been possible to verify or track direct influence of the online engagement exercise on policy development in this report.
Archived sites are available for viewing at the addresses provided on each case study report. The archived sites will be available for a minimum of six months after publication.
