Measurement is the critical building block for improvement
Help create better benchmarks and evaluations for democratic processes.
There is lots to do from building to researching to adopting, all of which is critically dependent on our ability to learn, understand and improve. We need the tools to measure, evaluate and benchmark democratic processes.
This page includes sections for: What we can’t do today, research questions, and the relevant capabilities.
What we can’t do today
This list contains all the related goals for improving the measurability of deliberative processes.
Table
Dimensions
Measurability
Long-term effects of participation that can be tracked on individuals and their networks
Urgent
Metrics of quality of deliberation that can be measured in real time, enabling facilitators to make adaptive process interventions
Urgent
Deliberations that can be captured faithfully and unobtrusively with full participant consent and ethical protection
Urgent
Preference transformation and participant learning that can be tracked throughout the process
Urgent
Process outcomes that can be empirically measured and compared across contexts, processes and systems, enabling evidence-based improvements
Urgent
Measurability research questions
We’ve outlined the research questions that we think will help improve our ability to measure and evaluate processes.
Table
Dimensions
Measurability
Which evaluation metrics (comparing single-dimension vs. composite indices) are sensitive enough to detect quality differences within similar processes but robust enough for valid comparison across different topics, geographies, and participant populations?
What consent, anonymization, and data governance protocols (comparing opt-in vs. opt-out, persistent vs. temporary storage, restricted vs. open licensing) enable practitioners to balance participant privacy and autonomy against the research value of maintaining rich deliberative records?
What constellation of outcomes (spanning legitimacy, recommendation quality, participant satisfaction, opinion change, and downstream policy impact) must any democratic process achieve to be considered successful, and how do these vary with process purpose?
How do downstream effects from participation systematically vary across different deliberative process formats (comparing citizens' assemblies, deliberative polls, mini-publics, and online forums), and what process features predict effect heterogeneity?
What particular knock-on effects from participation (spanning civic engagement, political efficacy, discussion spillover, network influence, or policy awareness) are most important to measure, and what longitudinal methods best capture them without excessive participant burden?
What observable deliberative quality dimensions (such as turn-taking equity, argument depth, perspective inclusion, or respectfulness) can be reliably measured through automated content analysis or human observation in real time, and what does measurement reveal about facilitator behavior changes?
What measurement approaches (comparing explicit belief statements, semantic mapping, implicit preference tasks, or network analysis of argument adoption) best capture individual and group learning and preference shifts while remaining feasible to administer at deliberation intervals?
How do different methods for measuring preference transformation (pre/post surveys, in-process journaling, exit interviews, or network tracking) correlate with one another and with long-term behavioral change, under different deliberative process formats?
How can process outcomes (spanning legitimacy, recommendation quality, participant satisfaction, opinion change, and downstream policy impact) be operationalized as measurable indicators practitioners can feasibly collect?
What recording modalities (comparing video, audio-only, spatial tracking, or multimodal combinations) most reliably preserve the substance of deliberation while remaining minimally intrusive and respectful of participant discomfort?
Which transcription and annotation approaches (comparing human verbatim, human semantic, hybrid human-AI, or AI-only) best handle cross-talk, non-verbal communication, and emotional valence while maintaining accuracy standards?
How can practitioners balance (through adaptive protocols or meta-evaluation frameworks) universal standards for cross-context learning against context-specific adaptations required by local stakeholder concerns and governance structures?
Relevant capabilities
These are the capabilities for measuring and quickly improving how we learn from process runs.
Table
Dimension
Measurability Learning Speed
Collectivize data
Learning Speed
Ability to make data open and easily available to researchers.
Evaluate processes
Urgent Measurability
Ability to measure desired outcomes to compare methods, processes and systems.
Gather process data
Urgent Measurability
Ability to gather process data in a cheap, reliable, accessible manner.
Simulate prototyping
Urgent Learning Speed
Ability to run trials that are good enough to learn from, and fast enough to enable rapid testing of new methods and process comparisons.