Defining Success & Measuring Regulatory Performance


Tuesday 13th and 27th July, 2021
(9:00am - 12:30pm AEST) 

Masterclass Facilitator:

Professor Malcolm K Sparrow

Professor of the Practice of Public Management John F Kennedy School of Government, Harvard University

During this two-part seminar we will be discussing the meaning of “success” in a regulatory environment and the associated challenges of performance measurement and management.

Professional regulators and law-enforcement officials certainly deal with customers, and so the normal range of customer-service metrics has some relevance for them. But their principal business is to deliver protections to society against various classes of harm, and towards that end they deliver and, where necessary, enforce obligations, influence behaviors, and manage compliance. The nature of “success” to be pursued and assessed is therefore quite different in the “protective” role of government than in service-provision roles.

Many public agencies select three or four metrics they think are particularly important, and designate them as “Key Performance Indicators,” drawing to them an inordinate amount of internal and external attention. Sometimes this focus can have perverse effects and bias operations in a way that is not quite as valuable as executives intended and may even distort operations in a way that is contrary to the public interest.

For regulators, much attention is typically paid to functional outputs (such as inspection or audit rates), to the rates at which agents discover or detect non-compliance, and to process metrics such as timeliness and accuracy of decision making. Connecting these measures to each other, and to valuable social outcomes, remains a complex puzzle.

This seminar provides an opportunity for attendees to assess their own organisation’s traditions in performance measurement and reporting, and—where necessary—to identify opportunities to broaden the range of indicators and analyses in a way that will enable their agency to present a more rounded and meaningful picture of their contributions to society.

Download the Program

Key Learning Outcomes

Day 1 - Core Concepts

  • Understanding distinct types of regulatory work (functional, process-based, risk-based, & crisis-response), and the classes of performance metrics relevant to each.
  • The notion of an “information rich environment,” and making the distinction between management information & performance metrics
  • The multiple purposes of performance measurement
  • Where “customer-service” metrics fit within regulatory operations, & where they do not
  • Classic ambiguities in assessing regulatory performance (e.g. When would finding more violations be good news, or bad? How should we interpret increased volumes of complaints?)
  • What difference it makes to the performance story when regulators turn their attention to “risk-based” strategies

Day 2 - Dive Deeper

  • What it takes to “prove causality”. The meaning of “evidence-based” policy and practice, and how these ideas apply to regulatory & enforcement agencies
  • The role of program evaluation. Where it fits naturally within regulatory operations. Other forms of analysis and pattern recognition that count
  • The distinctive challenges of performance measurement when tackling invisible risks, catastrophic risks, and risks involving conscious opponents (adversaries)
  • The implications of different regulatory structures on performance monitoring and evaluation
  • Seminar participants will also have the chance to submit in advance your own performance measurement puzzles. While we cannot promise to address them all, we will incorporate some, of broad general interest, into the agenda.

Who should attend?

This course is designed for mid to upper level regulatory and enforcement practitioners.

Download the Program

Use the code GC100 and save an additional $100 off the current price.