How can we offer digital services without understanding what people value?
Andrea Hill
January 29, 2020
Code for Canada fellow Andrea Hill explores some of the tensions and differences between user research and public opinion research in government.
Joining the public service as a Code for Canada fellow after 15 years in the private sector has been a very eye-opening experience. I knew there would be some cultural differences, but theyâve been in areas I never would have expected!
One is in the area of research. User research has been at the core of my entire career, starting as a prototyper in the private sector back in 2003 (!) Iâve conducted hundreds of user interviews, but I knew I had more to learn when I had this doozy of an ethics guide shared with me a few days into my new gig.
Universally, user researchers steer clear of opinion questions. I was thrilled very early on to hear the drumbeat of âFACTS AND BEHAVIOURS, NO OPINIONSâ.
Whatâs surprised me, though, is the definition of what constitutes an opinion, and why itâs so important UX researchers with the government of Canada steer clear of opinion-based questions!
Public Opinion Research
According to the Treasury Board, public opinion research is:
The planned, one-way systematic collection, by or for the Government of Canada, of opinion-based information of any target audience using quantitative or qualitative methods and techniques such as surveys or focus groups. Public opinion research provides insight and supports decision making. The process used for gathering information usually assumes an expectation and guarantee of anonymity for respondents. Public opinion research includes information collected from the public, including private individuals and representatives of businesses or other entities. It involves activities such as the design and testing of collection methods and instruments, data collection, data entry, data coding, and primary data analysis.
Usability research and âbehavioural or factual researchâ are explicitly called out as not generally considered public opinion research. When public opinion research is going to be undertaken, public servants need to refer to the Mandatory Procedures for Public Opinion Research (Appendix C of the Directive on the Management of Communications), and thereâs quite a lengthy process to follow.
So, itâs clear why a researcher would want to try to steer clear from public opinion research. The question then becomes, what are they giving up in doing so?
Can you ask attitudinal questions without it being classified as âpublic opinion researchâ?
There are a wide variety of user research methods. In addition to the qualitative-quantitative spectrum, thereâs also a range between behavioural and attitudinal inquiry. We know that usability and behavioural research is NOT considered POR, but does that mean that anything else IS considered public opinion?
Hereâs the thing.. we can measure behaviour only once we have something to measure. We have to already be in the solution space. At that point, weâre doing evaluation.
As a designer, I need to be operating a few steps before that. I want insights into user needs to design the right thing. I donât want to simply guess, and then measure whether itâs usable. I want to be conducting problem space research. I want to delve into my usersâ mental models, so I can design a solution that fits into their understanding of the world.
If we focus only on what users do, and shy away from the why, weâre severely limiting our ability to serve them. Iâd go so far as to say weâre actually preventing ourselves from developing empathy. We canât call ourselves human-centred designers if we refuse to acknowledge the human experiences that drive behaviour.
Why are attitudinal questions to be avoided?
Where does this aversion come from? Is it externally or internally imposed?
- Are they furthering a political agenda?
- Are they âbadâ or âmisleadingâ from a UX research perspective?
Iâve heard some suggestions of where this may stem from, but nothing official. (Dare I say, Iâve heard opinions, but maybe not facts?). If thereâs an official explanation anywhere, Iâd appreciate the reference!
At some point of trying to understand why things are the way they are, I had to ask myself if it really mattered anyway.
Do user preferences even matter when youâre operating as a monopoly?
In order to win in a free market, a company needs to offer a product or service that meets user needs âbetterâ than alternatives. According to Jobs to be Done, users will âhireâ the products and services that will help them get their jobs done better, faster or cheaper. So how do we find out which alternatives exist, and what users are choosing?
Back to our two research approaches: we can observe them (behavioural), or we can ask them (attitudinal).
Except â weâre talking about social services here. We canât observe usersâ choices, because they donât have one. They have the single service that we offer.
Well, you may be saying, we can observe them using our own service. We can see how they use the service we already have, and go from there.
Yes⌠butâŚwhen youâre being innovative and coming up with something new, thereâs nothing to measure! There may not be any âfactsâ related to how something is done currently, because it may not yet be possible! Not to mention, youâd have to already be well down in the solution space once you get to the point of measuring the usability of a specific proposed solution.
So, itâs possible that as a monopolistic supplier, we actually donât have to worry about providing a solution that meets users expectations. Because they donât have a choice.
We could get away with offering the bare minimum of service for our users, âusableâ, and they would use it, because they lack choice.
I donât know about you, but I want to do better. I believe our users â our citizens and residents â deserve better.
So letâs go above usable, and aspire to offering something useful, desirable, any perhaps even delightful (!). If we aspire to offer a solution that meets our usersâ expectations, we have to do things a different way. First off, we need to figure out what those expectations are. But, is an expectation a fact? Can it be observed?
As it stands now, the path of least resistance is to steer clear of the murky waters of POR and just measure whether somethingâs usable. Can I get my work done through this tedious process? Yes. Is it a mind-numbing, painful process? Maybe â but users donât have choices.
By calling out usability testing as a way to engage with users through the product development process, we draw attention away from other types of research. Usability research is âsafeâ. Itâs measurable. When you measure usersâ abilities to perform tasks, you can report back a nice clean result.
But are we ignoring the âwhyâ behind the âwhatâ? For many designers, that âwhyâ is critical. The âwhyâ helps us better understand the underlying job the user is trying to get done and whatâs important for them to feel successful and confident. If someone can complete a task but the system or process is frustrating and leaves them disappointed, confused or angry, we havenât delivered on our promise to design a good experience for them.
When weâre designing services, we need to put the people weâre designing for before our own processes and policies.
How does research dovetail with inclusive design?
Inclusive design is a bit of a buzzword now in 2019. Inclusive design can be defined as âdesign that considers the full range of human diversity with respect to ability, language, culture, gender, age and other forms of human difference.â
These 6 Principles for Inclusive Design are fantastic:
- Seek out points of exclusion
- Identify situational challenges
- Recognize personal biases
- Offer different ways to engage
- Provide equivalent experiences
- Extend the solution to everyone
When we choose to focus only on âfacts and behavioursâ, we ignore the human experience. Yes, I can park in the back of a dimly lit parking garage, cross railway tracks and walk several blocks to my hotel in the middle of the night, but Iâd prefer not to.
Is my experience invalid?
What if I tell you Iâm a petite woman, or that Iâm in a wheelchair. Does one or the other make my preference (or need) more reasonable?
Do I need to justify my preference by giving you my gender and stature? Or does what I prefer not matter â and does that make me feel like I donât matter?
We get into a slippery slope when we start trying to differentiate between preferences and needs.
Who are we serving when we put the burden on individuals to prove they need something (as a fact â provide a health certificate, etc)? Is it not more respectful and humane to recognize that our human experience is much more than what can be demonstrated, observed and measured?
The sixth principle of inclusive design is to extend the solution to everyone. If we truly believe we are engaging in inclusive design, we need to release ourselves from this tight grasp on facts. If itâs a fact that anyone cannot stay in this hotel, and we have to offer an alternative, then we need to open it up to everyone. At that moment, does it matter whether we designed for the person who couldnât, or the person who preferred not to?
Could we have just started with understanding and empathizing with the concerns, and fostering some goodwill, rather than only offering a solution begrudgingly because someone proved they needed it?
Since starting my fellowship, Iâve been counseled numerous times that âthis isnât the private sectorâ. Constraints foster creativity, and I relish the challenge of figuring out how we can deliver delightful experiences to the public while respecting data protection and other relevant policies and procedures. But this canât be an excuse to offer lackluster experiences at scale simply because âthatâs the way weâve always done it.â
We need to put the people weâre designing for before our own processes and policies.
This post has sat in my âdraftsâ for months, and I found myself sharing some of thoughts at a meetup last night. Iâve been reflecting on my time within the public sector, and I think this has been one of the most jarring aspects. Iâm sharing this in the hope of starting a conversation and hopefully gaining an understanding and acceptance for this train of thought. 🤷ââď¸
Further Reading
- Defining public opinion research (2016â11â08)
https://www.canada.ca/en/treasury-board-secretariat/services/government-communications/public-opinion-research-government.html - Mandatory Procedures for Public Opinion Research (2016â05â11)
https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=30682#appC - Standards for the Conduct of Government of Canada Public Opinion Research â Qualitative Research (2019â04â02)
https://www.tpsgc-pwgsc.gc.ca/rop-por/rechqual-qualres-eng.html - Scaling Design Research in the Government of Canada (2019â06â26)
https://digital.canada.ca/2019/06/26/scaling-design-research-in-the-government-of-canada - A guide to interviewing from the Canadian Digital Service
https://digital.canada.ca/tools-and-resources/guide-interviewing/ - Usability Testing is not Public Opinion Research (2014â07â31)
https://www.akendi.com/blog/usability-testing-is-not-public-opinion-research/