1% Metrics

Update 25/09/14:

  1. We have simplified these to bring them down from 14 to 10 metrics. We’ve done this by combing some of the metrics (the 3 ‘area perception’ ones, and the 3 ‘sustainability’ ones).
  2. We’ve also changed the ‘Sustainability’ metric to be ‘Sustainability and Resilience’.

The metrics are used when assessing Beta Projects in reports such as this.

 

What’s the issue?

We need a system for measuring our experiments:

  • Evidence of the impact of individual Beta Projects.
  • To allow Dublin City Council to make actionable decisions about each Beta Project.
  • To justify requests for additional Dublin City Council resources to specific Beta Projects.

Measure imageDublin City Council Beta Projects include elements of the Lean Startup process (which we tweak as we adopt them in order to better suit their use with the complex issues and opportunities that cities and local goverment tends to uniquely face).

One of those steps is to follow a Build-Measure-Learn loop when trialling project. Initially we explored the various aspects of ‘building’ (ie putting a physical, but temporary, item onto public streets). Whilst doing that, it became clear that we needed to begin to look at the Measure step.

Dublin City Council often will look at long-term metrics (such as Dublin being the 9th best city in the world for cycling), but currently doesn’t have a standardised system for measuring the impact of prototyping projects – whether that’s a Beta Project macro looking at policy, a Beta Project micro looking at a specific local neighbourhood trial, something like Upstart’s Granby Park, or something such as Intel’s City Watch app.

1% Metrics and 99% Metrics

So we decided to begin to try to develop up a method for the latter type of metric. For simplicity we’re calling  them “1% metrics” (and “99% metrics” for the longterm type).

The 1% metrics would be for learning. Did experiment X work or not. They’re measured in terms of days, weeks, months.

The 99% metrics would be for checking that all Beta Projects (X, Y and Z) plus our ‘business as usual’ actions are all keeping us on the correct path. They’re measured in terms of quarters, years and decades.

Developing a first draft set…

[Update 18/11/13: We employed The Civic Works on this project. They prepared, facilitated, helped with the distilling of the information, and then with the final report – including the below sketches!]

We held a 3.5 hour workshop with City Council staff from across various departments and with the aim of tapping into their expertise. Despite that being a very short amount of time for such an exercise, we were able to very quickly pull very good information together.
(Thanks to Ré, Deirdre, Peter, Siobhan, John, Kevin, Pauline, Jamie, Walter, Seán.)
(For some background links you might like to see The Studio, Digital Dublin, Activating Dublin, Grafton Street Quarter projectsThe Civic Works).

20130903_120628

That work was then summarised over two half-day sessions by one member of Dublin City Beta Projects and of The Civic Works.

photo

A series of metrics and indicators emerged which we will now need to begin to use in order to trial them in the real world! We expect we’ll very quickly learn about how useful they are, whether we need to add some more, or some could be removed.

The aim is that no matter what is being trialled, that the same metrics would always apply. That way people will become used to them, and don’t need to keep familiarising themselves.

These ‘metrics’ will be informed by ‘indicators’. Depending on the project though, the indicators will need to change. Measuring how a website is working is very different to measuring how an onstreet trial is working. Equally you might use 3 different types of indicators in order to give you an overall rating for a metric.

Developing up methods of gathering measurements for these indicators is going to take time, effort and resources (and lots of trial and error!)…but let’s start the process so that we’re beginning to grow the methods and systems required.

A side note: The icons used here are from The Noun Project so can be easily changed.

The 14 initial metrics that we’ll now begin to trial…

1. Dublin City Council costs

beta9

The total reduction in costs across all DCC departments as a result of a Beta Project (target min ~10%)

Potential indicators might include:
•Saving against initial baseline, establish at start of the Beta
•Cost /benefit analysis / full life cycle costs analysis
•Self reported expert hunch

2. Dublin City Council time

beta7

The total reduction in staff time across all DCC departments as a result of a Beta Project

Potential indicators might include:
•Reduction in staff time as reported via weekly returns
•Reduction logged calls / call times to DCC call centre
•Reduction in overtime / time in-lieu
•ROI (time)
•Self reported expert hunch

3. Dublin City Council staff morale

beta11

The change in staff morale a Beta Project enables across engaged DCC Departments

Potential indicators might include:
•Self reported by staff (+/-scale)
•Reduction in sick days / formal complaints
•Reduction in H&S incident reporting

4. Public perception of Dublin City Council

beta14

The change in the public perception of DCC as a result of a Beta Project

Potential indicators might include:
•Advertising Equivalent Value (Beta-related)
•Reduction in complaints
•Increase in positive qualitative comments

5. Perception of the Area

Area Perception

The change in people’s (living/working/playing/visiting) perception of an area as a result of a Beta Project.

  • Potential indicators might include:
  • Qualitative reporting by residents (street conversations/ survey)
  • Reduction in anti-social behaviour
  • Increase in signs of pride & ownership –e.g. flower boxes
  • Qualitative reporting by workers (street conversations/ survey)
  • Reduction in anti-social behaviour
  • Increase in footfall
  • Qualitative reporting by visitors –tourist and people passing through (street conversations/ survey)
  • Increase in tourist photos of area on social media
  • Expert observation

Area perception of residents

beta4_grey

The change in residents’perception of an area as a result of a Beta Project

Potential indicators might include:
•Qualitative reporting by residents (street conversations/ survey)
•Reduction in anti-social behaviour
•Increase in signs of pride & ownership –e.g. flower boxes

Area perception of workers

beta5_grey

The change in workers’perception of an area as a result of a Beta Project

Potential indicators might include:
•Qualitative reporting by workers (street conversations/ survey)
•Reduction in anti-social behaviour

Area perception for visitors

beta8_grey

The change in vistors’perception of an area as a result of a Beta Project

Potential indicators might include:
•Increase in footfall
•Qualitative reporting by visitors –tourist and people passing through (street conversations/ survey)
•Increase in tourist photos of area on social media
•Expert observation

6. Sustainability and Resilience

Sustainability

The change in environmental/social/economic sustainable activity and resilience for engaged participants as a result of a Beta Project.

  • Natural Step laws 1-4
  • Expert rating
  • Increase in bio-diversity / wildlife / biophilia
  • Self-reported expert hunch
  • Self reported % increase in revenue by local enterprise
  • Increase in footfall
  • Reduction in time for DCC enterprise decisions
  • Self reported increase in weak social ties (0-5 scale)
  • % increase in participant income
  • Self reported % increase in revenue by local enterprise

Environmental sustainability

beta6_grey    

The % change in environmentally sustainable activity as a result of a Beta Project

Potential indicators might include:
•Natural Step laws 1-3 (0-5 scale)
•Expert rating
•Increase in bio-diversity / wildlife / biophilia
•Self reported expert hunch

Social sustainability

beta3_grey

The % change in socially sustainable activity for engaged participants as a result of a Beta Project

Potential indicators might include:
•Natural Step law 4 (0-5 scale)
•Expert rating
•Self reported increase in weak social ties (0-5 scale)
•% increase in participant income
•Self-reported expert hunch

Economic sustainability

Beta1_grey

The % change in economic activity as a result of a Beta Project

Potential indicators might include:
•Self reported % increase in revenue by local enterprise
•Increase in footfall
•Reduction in time for DCC enterprise decisions

7. Beta engagement

beta12

The number of people actively involved in a beta project, including their age / gender / ability / socio-economic grouping / nationality / ethnicity

Potential indicators might include:
•Number & details self reported by engaged participants
•Survey via Beta Projects
•Self reported via expert observation

8. Negative Interactions

beta13

The change in negative interactions between stakeholders (including citizens) as a result of a Beta Project

Potential indicators might include:
•+ / -change in enforcements & other forced interventions
•% Change in associated councillor questions
•Self reported via expert observation

9. Beta Demand

beta2

Requests for a solution, or the continuation or expansion of a Beta Project.

Potential indicators might include:
•Concrete internal requests to Beta via email / meetings etc
•Concrete external requests to Beta via email / social media etc

10. Beta transferability

beta10

The potential of a Beta Project to expand at DCC & beyond

Potential indicators might include:
•Self reported scale (by all engaged stakeholders) as below:
1 –Site specific
2 –Area / theme specific
3 –Suitable for multiple themes & areas
4 –DCC Wide / City wide
5 –Beyond DCC (national / international)

How might the ‘1% Metrics’ be used?

Internal (DCC) Report Cards

beta_sketch3-1

On Street Display

beta_sketch4-1

In press

Beta_sketch1-1

By project / city stakeholders

beta_sketch4-2

As supporting measures for international rankings

beta_sketch2-1

Will the full set of metrics always be used for all Beta Projects?

No. The metrics will always be the same set, but perhaps sometimes certain ones wouldn’t apply.

When a particular metric is not being used, we expect that we’ll still show in grey so that it’s really easy for people to understand and stay familiar with.

beta9_grey instead of beta9

Will there be a simple, clear, decision at the end of the Beta Project? 

Yes. We are proposing using these ‘4 Ps’…

  • Pull (wouldn’t appear to be a good solution in any way, time to pull it)
  • Pivot (use what has been learned in order to make a major change in direction)
  • Persevere (on right track, need to  learn more)
  • Policy (would appear to be a good solution, formalise as necessary – eg planning, hard infrastructure, etc)

For example, a Beta Project might undergo 3 iterations before a good solution is found. The first 2 Beta Project Report Cards could report the need to ‘Perseverse’ and then the final Report Card would report the need to move to ‘Policy’.

Feedback and Suggestions please!

As always, feedback and suggestions would be great! Should we add any more? Should we remove some?

You don’t understand what is meant by one of them? Please tell us!! We need to get these to the stage that they’re very simple to understand, and don’t need to be explained.

We will begin to use these at the end of every Beta Project, and each opportunity will all of us a chance to further tweak every aspect!

About Dublin City Council beta

Dublin City Council are trying a new method called DCC Beta Projects...your input, then quick trials 'on the street', then your feedback! We'd LOVE your help!
This entry was posted in Beta Project macro, Beta Project micro, Shane, Siobhan, Strategy. Bookmark the permalink.

6 Responses to 1% Metrics

  1. I am very pleased to hear you are working on this. In fact I am quite excited (I will explain why at the very end below). I do think you could be more succinct and focused and there are suggestions below.

    Before I start, I don’t mean to be negative with what is below and I hope it doesn’t seem that way. I just want to suggest what my experience as a psychological researcher and in a variety of accounting/business roles of all sorts has taught me about these things and the pitfalls to avoid. I’ve run public service type stuff too, so the ground is not completely new to me.

    Also, I think you have to be clear what you are measuring. Are these criteria for business cases for expanding existing projects, or for new projects, or just for evaluating whether a projects should begin at all? These are quite distinct areas. I think these are good at overview level, but I don’t know if these criteria are not necessarily going to be that helpful for the projects you are doing.

    Mainly, I think you are setting the bar very high for what is in reality a very small budget and very small projects. For example, saving 10 percent from the DCC budget is a saving of about €85m/year. I am sure that is not what you meant, but it seems to me that it is what you wrote! Anyway, a more realistic cost saving for any one project would be more like 1 or 2 percent in its area. DCC’s budget is so vast that a saving of this level in any one area, without reducing the level of service, is actually pretty big. A better way to measure it would be using a technique like ‘NPV’ or ‘IRR’ to determine whether savings/gains are worthwhile from a project. If it isn’t just money, I would consider writing a business case (http://whatis.techtarget.com/definition/How-to-write-a-business-case-document) or better still a Toyota-style A3 report (http://sloanreview.mit.edu/article/toyotas-secret-the-a3-report/) to fully articulate the project and what the ‘point’ of it.

    My biggest comment, there is not enough priority on providing a benefit to and solving problems for the punters by which I mean the businesses, other organisations and people who choose to enrich Dublin life by their decision to live, work and have fun in the city. These are the people who matter most. Everything else is really secondary.

    For simplicity, and for emphasis, I would squash the related categories, 5, 6, 7 and 9, 11 and 12 together and put them in one category. And I would put them at the top.

    I would also say that some of the things you want to base the metrics on are very hard to measure. For example, measuring ‘psychological’ things about visitors or inhabitants is in itself a major undertaking, requiring hundreds of sample points. This type of research is very difficult to arrange. If this is just not going to be practical, I would really drop them as measures and concentrate on things that are somehow concrete. As well as the stuff you have mentioned, I would look at things like queueing time and time required to carry out particular tasks.

    I would sharpen 10 and call it something like ‘economic contribution’. I would focus on ways in which it contributed directly to the economy.

    The reality is that you in general, you can’t make an actual saving. You can probably only make a ‘gain in productivity’, at best. By that I mean that just because your project is successful, it won’t mean that costs for DCC will go down. What I mean is that no one will get made redundant (at least not directly because of the project – I know there are other initiatives underway in relation to headcount). In fact, the saved resources will be redeployed to other areas, to improve service. You are making everything more complex by opening up the area of non-money statistics for internal savings/gains in productivity.

    Staff morale improvements are basically productivity gains, or should be expressible into productivity gains. Yes, I know there is more to it than that, but the non-productivity benefits of making DCC staff happier are very soggy ground to build a business case on. In terms of quantifying it, if you can save, say 1 sick day per employee per year at DCC that is an enormous gain in productivity for DCC and that can be quantified easily enough (it would be by my quick estimate worth about €900,000, but it could be more). Another problem is that if you get involved with staff satisfaction and so forth, you risk becoming internally focused. It is not that there is not important work to be done internally in DCC, but it isn’t really the ‘spirit’ of Dublin Beta, in my opinion. Beta should be more externally focused.

    So I would squash those areas together and put areas 1-3 together into one category of ‘productivity’ or something.

    So overall, I would have Dieter Rams ‘less but better’ approach to metrics and business justification.

    So I said I was excited. Here’s why. The metrics, for me, might be suggesting a wider direction for DublinBeta, away from ‘architectural intervention’ projects, to projects of other types which have wider, more direct appeal and benefit to larger swathes of dubliners, and which will have clearer, shorter-term gains.

  2. Will C says:

    Hi All,

    Really think that this initative (Beta Projects generally) is fantastic. Must admit that there is a lot of work done up front in thinking, theorising and explaining. Most of which (including this blog post and the first comment) definitely fall into the TL DNR (too long did not read) bracket. This work is essential to working things out, but got to question does it take away from the essential message of what you are trying to achieve – which I understand to be greater community well-being in times of great change and little available funding. I venture to suggest that having a marketing / communications skill set on board (for free of course!) would help you refine and condense your message to make it more easily communicable and understood across a broader spectrum. Personally I’m a professional in the “urban” area and I’m struggling a bit to get my head around it, I think I like it… but am a bit lost on clarity of message.

    The Yanks have a fairly well established movement in essentially the same area – they call it “Placemaking” and they have the message and metrics fairly well down.. worth reviewing and borrowing?? good links at

    see this interesting recent paper from no less than MIT
    http://dusp.mit.edu/cdd/event/places-making

    Here are a set of open source “tactics” that can be borrowed and employed to try and achieve more strategic goals

    the font of all knowledge in this area are the brilliant PPS
    http://www.pps.org/

    As a Dubliner in exile, I wish ye all the best, and am keen to stay updated.

    Cheers,

  3. Pingback: Dublin City Beta Projects – Status Update | Beta Projects Dublin

  4. Pingback: Beta Project Report Cards | Dublin City Council Beta Projects

  5. Pingback: Rain Box Planters | Dublin City Council Beta Projects

  6. Pingback: LEAP points | Dublin City Council Beta Projects

What do you think? We'd love to hear feedback! :)

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s