Who Listens to the People?

Generation1.ca,

By Arundati Dandapani

74th AAPOR Annual Conference & 72nd WAPOR Annual Conference 2019, Toronto, Canada

The opportunity to be among important books and those who wrote them fills me with immeasurable delight. So you can picture my sight at the 74th Annual AAPOR Conference from May 16th-19th. Happy to have extended my knowledge expedition (from the previously concluded IIeX) and hoping to keep up this bookish tradition at future events. Big thank you to ASDE Survey Sampler, NORC, RTI, SSRS and especially Randa Bell, Justine A Bulgar-Medina, Kristin Dwan, Marielle Weindorf, Ipek Bilgen, Susan Scherr, Benjamin Schapiro, Kiril Makarov, Stella Park, Mandy Sha and peers, leaders and organizers for making me feel at home, at home, in what really was the conference of conferences in the many roles I got to play as a first-time attendee!

  • QR Codes are the new business cards; for the rest of us there’s always Twitter.

I made it to a handful of sessions in an age where no conference has yet mastered the art of getting around concurrent sessions. Although, AAPOR will be releasing a lot of their sessions’ videos on their YouTube channel to allow us to catch up on missed concurrent tracks and others at leisure! Open access content is the best, given that event attendance has its own value proposition in establishing face-to-face connections that would otherwise be impossible from the remote trenches of a geofenced microchip.

The quality of ideas and publications presented were impressive and specific, resonating with our global realities and digital interoperability. Only two sessions I attended had half their speakers not show up, one of which was on the last day (Sunday 9 am) about immigration (with no replacements). Back-up speakers and moderators sound like a good idea, although ill-luck can strike anyone. And a madly successful karaoke night party of the night before was probably what the audience felt was the reason for the less attended panel.

https://platform.twitter.com/widgets.js

The conference was spread out across four floors including a basement so at the end of Day Three my joints were screeching for CBD. There were numerous well-intentioned jokes (and recipes) about the ChickenXChicken menu, but the food for thought was most provoking.       

Some takeaways from my experiences are capsuled in a few broad themes interrupted by Twitter snippets throughout:

Everything is the same, everything has changed

The quality of ideas and publications presented were impressive and specific. The opening keynote highlighted how digitization had impacted  census response rates (e.g. modes, incentives, messaging effectiveness, respondent safety). In this session, “Conducting a Census in the Digital Age”, Patrice Mathieu of Statistics Canada talked about how the use of internet in the Canadian census (since 2001) has evolved.  Over the past 20 years, response rates have increased using their wave methodology, which includes sending out letters, reminders, and aligned public messages. Tailored and targeted communications have improved messaging efficiency: 88 clusters were organized into 12 metaclusters, for example.  A master control system was adopted, and in 2016 integrated collection was achieved through the collection management portal to deal with non-response.  The goal for the 2021 and 2026 census is to increase research on administrative data use, expand mail out areas, and reduce paper waste. Regarding the much talked about US decennial census, past types of respondent complaints have included: Address Problems, Addressed to Resident, Age/Illness/Death, Behaviour – Field Representative, Behaviour – Telephone Interviewer, Complaint – general, Complaint – government, Confidentiality/Privacy, Constitutionality, Contact Procedures, Decennial confusion, Decline to Participate/Opt-out/Refusal, Feedback – Suggestion, ID Theft, Invasive/Intrusive, Legitimate/Scam, Mail response problem, Mandatory, Online response problem, Personal visit, Phone call, Question – general, Request a questionnaire, Selection (Why Me), and Time to complete, etc.  D. Sunshine Hillygus’ book The Hard Count: The Political and Social Challenges of Census Mobilization offers more insight into how a mobilization campaign can dramatically increase voluntary participation in the decennial headcount, and identifies emerging social and political challenges that may threaten future census counts.  A key issue discussed on the panel was the inclusion of a citizenship question on the census.  Hillygus was very critical of this, owing to the bias it can create from fear and ancillary pressures.  Given that nearly half (8 million) of 19 million Latino immigrants (as an example) in the US are undocumented, the inclusion of this question has become a politically contentious issue.

Conducting a Census in the Digital Age: (Seated from L to R ) Peter Miller (moderator), Patrice Mathieu (panelist), James Treat (panelist), D. Sunshine Hillygus (panelist)

Identity is Fast and Slow

In Canada over a fifth of the population (22%) identifies as a visible minority and, in a hyper diverse city like Toronto, over half the population (52%) identifies as a visible minority.  The 2016 US Census reports the US population at about 77% white and 23% non-white (Black, Latino, American Indian, Asian, Native Hawaiian, biracial, others etc.).

The Issues and Diversity in North America panel reacted to Robert Putnam’s (American, Political Scientist) idea that bringing in immigrants reduces social capital.  Social capital in trust, despite what critics may say, is at an all-time high.  In a city that welcomes over 400,000 newcomers each year, although problems like income inequality, unaffordable housing and traffic congestion persist, there is a relatively low crime rate and over 400 different ethnicities mix peacefully.  Understanding this progress and benchmarking it, is key to understanding how social capital has stayed stable in Toronto and in Canada.  Social capital is used to describe the vibrancy of social networks and the extent to which there is trust and reciprocity within a community and among individuals, according to the Environics Institute.

https://platform.twitter.com/widgets.js

The changing face of electorate demographics and corresponding increase in non-white voters in the US and Canada is disrupting elections.  The panel touched on issues and challenges facing North American societies (cities), including cultural or urban integration, urbanization, wealth, education as well as declining economic outlook and social capital among new immigrants and millennials, partisanship, affective polarization among groups around issues like diverse hiring, workplace cultures, leadership and equity, tolerance to out-groups, contact with out-groups, acknowledgement of inequity between gender groups, racial groups or others, and growing economic despair among new immigrants and millennials.  It was concluded that Toronto was setting a positive model for social integration.

https://platform.twitter.com/widgets.js

Behavioural insights have harnessed the power of inducements to remind respondents to participate in census surveys whether in the US or in Canada.  A US Census panel spoke on the applications of behavioural science and eye-tracking to understand response rates and patterns, and to design effective communications.  The research challenges included:  What kind of survey reminder letters were working and what layout or design conveyed a mix of authority, authenticity, likeability, legibility and direct appeal to action?  Using words like “Open immediately” moves more people to act.  Insertion of playful knowledge-props like data-slides are another tactic that has worked for the American Community Survey, the American Housing Survey and the Census.

https://platform.twitter.com/widgets.js

Response rates in census surveys have slowly declined over the past 15 years.  Research and communications need to accommodate the longitudinal considerations of research: incentives may work, discontinuation of incentives in another wave may or may not matter.  The literature is mixed on this topic.  Established in 1973, the American Housing Survey is conducted every odd-numbered year, primarily through face-to-face interviews.  The core questionnaire is supplemented with rotating modules and every household is treated as a unit.  The movement from no incentive to some incentive had the most impact in raising response rates.  Melissa Cidade revealed it was luxurious to be working with a longitudinal survey panel, with so much potential to predict future response behaviours and to develop a propensity model for incentive allocation.  Modern methods like eye-tracking continue to offer design optimization for better response rates.  Key design elements include using icons, FAQs, and sidebars in the letters/survey invites. Static heatmaps (red is hottest) and dynamic eye tracking help to process attention spans better. Two top reasons people don’t respond to community census or housing surveys are: 1) their legitimacy is not established, and 2) privacy concerns.

For an improved survey experience:

  1. Limit volume of messaging, offer diffused focus
  2. Follow plain language, use average American grade 7 language
  3. Offer clear purpose and incentives to participate
  4. Reinforce US Census Bureau brand

Citizens and consumers respond to good design, and what good design is, develops in many iterations and waves. The US Census Bureau uses strategic messaging to differentiate themselves.

Polarization and its Discontents

Among other panels of note were the ones on Election Flashpoints: Religion and Latino voters, partisanship, ideology and consumer confidence, new voting demographics, designing the right research and asking the right questions to opioid substance abusers, messaging communications on climate change (making it more accessible, clear, personal and urgent to the public mostly), the persistence of anti-Semitism in America and the relationship between violence (mass shootings) and (voter) values presented with the case of Dutch nationalism while examining the relationship between voter proximity to mosques (with and without minarets) and their support for the radical right in the Netherlands.

The Environics Institute’s Black Experience in the GTA Research Study showed community research in action.  The project went through its share of critical questioning: Should we really do this study?  Is it going to be different from previous studies?

“There is one distinction between Toronto and other cities in the US of comparable size: Discrimination can be fought here,” reasoned Wendell Ajetey, entrepreneur, lecturer and a member of the Canadian Studies Advisory Council at Yale University.  “Can research make a difference to improving the lives and policies that affect communities?  Yes, but it needs an equal amount of political will to be effective and sustain long term systemic change, replied the panel. 

The project was also a response to how racial bias is embedded in the police system in dealing with growing income inequality in the city, and the concentration of poverty and increased gun violence.  From a research standpoint, data collection methods among ethnic groups and populations ignore challenges being faced by people of different ethnic or religious origins, pointed out Dr. Keith Neuman, Executive Director of the Environics Institute.  And, this is why they chose/recruited community members to conduct the interviews and be a part of the research process and design, offering skills training opportunities to the community where needed.  As with all data, we need comprehensive clarity by desegregating the data for trends or patterns, said Akwasi Owusu-Bempah, an Assistant Professor in the Department of Sociology at the University of Toronto.  “Political representatives often disregard anecdote in favour of stats or facts especially in the age of fake and alternative news,” said Joseph Smith, reminding everyone of the need to correct for such misrepresentation in research.  Marva Wisden, Director of Community Engagement and Outreach at the Black Experience Project, emphasized the economic and social power of provinces with multigenerational black communities, upholding the province of Nova Scotia, home to the African Nova Scotian Commission.  Although the Black Experience Project was a one-time study, other markets would benefit to scale this project to their jurisdictions using their own resources.

https://platform.twitter.com/widgets.js https://platform.twitter.com/widgets.js https://platform.twitter.com/widgets.js https://platform.twitter.com/widgets.js https://platform.twitter.com/widgets.js https://platform.twitter.com/widgets.js

Accolades

Captured below are prizes won, performances made and the top ten rejected sessions titles courtesy Conference Chair Courtney Kennedy, of the Pew Research Centre (who moaned the lack of interestingly crafted paper titles this year), all offering plenty to marvel at with tear-lit eyes.

  • “Take Sample and Poll!”


Books are political, the oldest technology filled with extraordinary powers of new information, problem-solving, history and healing.  Princeton University Press, Stata, Kogan Page, University of Chicago Press and Cambridge University Press were among the publishers whose books were on sale at AAPOR’s Book Exhibition this year.

Beyond excited to be selling books here at AAPOR! Many familiar and new customers visit us.
Book Exhibition Photo Credits: Jasmeet Kaur, Quorus Consulting Group
At the Book Exhibition with NORC’s Kristin Dwan, who showed me all the ropes for a successful three days of emptying out stock! Amazing titles, and 3 lucky raffle prize winners.

Back in the early noughties, publishers in the developed world were worried sick. Self-publishing online might have changed the livelihoods of a handful of writers, but the physical magic of a book is far from over. Passersby / booksale visitors told me the selection had been vaster in last year’s conference. We could have an AAPOR online bookselling hub, and I’d be beyond happy to host this on Generation1.ca with support from Canadian Market Research firms and Canadian publishers. We would even take this global.

The outgoing AAPOR President’s (Dr. David Dutwin, EVP & Chief Methodologist of SSRS) address was full of advice to the industry at large, particularly with respect to strengthening ties and relationships between the polling industry and the media to incorporate ongoing education, advocacy and grants/funding to tap into the voice of the people.  He committed to polling education (and grants) for the media, who he cited are the biggest spokespeople of the association and prioritizes the role of strategic communications in public opinion research. The voice of the leadership can often give a sense of the health of an organization. Dutwin’s entire speech can be watched here.

https://platform.twitter.com/widgets.js

Global Lens – WAPOR

The WAPOR (The World Association of Public Opinion Research) Conference followed AAPOR and there were joint sessions leading up to the WAPOR only conference. The joint panel on “Populism around the world” kicked off the opening plenary for WAPOR 2019. Pippa Norris took the stage introducing her latest book Cultural Backlash and the Rise of Populism: Trump, Brexit, and the Rise of Authoritarianism Populism‘s central thesis. Laura Silver of the Pew Research Centre talked about measuring populism in Europe, and Gary Langar offered his views on democracy in troubled times; Langar’s previous presentation highlighted the changing face of the electorate and how the biggest challenge facing US Republicans was the rise of Hispanic eligible voters and the “running out of white voters”. The plenary on populism offered a deeper understanding the roots of this populist wave (values shift) and how it can help predict or model voter intent and behaviour.

Norris said that populism is a lingual “facade”, and that we need to understand the conceptual theoretical framework – authoritarian populism and cultural backlash – to truly understand the real challenges facing democracy.  Authoritarian values (fear of out-groups, xenophobia, need for security or tribal protection against risks, conventionalism/solidarity and loyalty towards leaders of one’s tribe) mixed with populist rhetoric led to a sweeping phenomenon in the US, UK, Europe and elections worldwide.  Income is virtually a useless predictor of voting as much as cultural values today.  Populism’s success manifests as in-group conformity and out-group rejection (us vs them); nationalism is one of its forms.  Pegged now as the polar opposite of authoritarianism and populism, post-materialism came about in the 60s and 70s in the post-war era.  Authoritarian populism is a culture and values shift more than anything else, strongly correlated with conservative votes in Europe and in North America.

Cultural backlash is also a response to the concentration of income in the knowledge economy creating its own share of inequities, disproportionately impacting rural and suburban regions that traditionally relied on manufacturing services and blue-collar work.  As Norris’ book’s co-author Ronald Inglehart later put it, the unforeseen aspect of the knowledge society is the disproportionate wins for those at the very top (like Jeff Bezos and Mark Zuckerberg).  The ratio of earnings for the winners to the rest, in a “Winners Take All” knowledge economy, used to be 20:1 but now this has moved to a dramatic 341:1, he said!

Norris commented that the term “crisis” should be banned from use, poking fun at the rise of crisis literatures over time that were born from situations like ungovernability, the decline of social capital (Putnam is the biggest proponent of this), and the recent rise of populist forces that are determined to safeguard the “in-group” (“us”) vs the “out-group” (“others”, foreigners, immigrants, educated elites etc.).  The tribal need to protect oneself (and the in-group) from the perceived threat of others triggers the authoritarian-populist mold of thought.  Trust in these times is at an all-time low, in the media, in pollsters, in politicians, in brands, etc.

A slide from Laura Silver’s deck indicating lowest trust in media in Italy, Spain and the UK

If trust is frail, how is it best measured?  Trust is lower in countries with authoritarian rule than in post-materialist ones.  Although, when looking at the data from the World Values Survey, trust in Vietnam and China is high versus post-materialist countries like Sweden, Canada and Denmark trust, noted Norris.  So, what comprises trust?  Is trust an attitudinal construct, or is it something else?  Norris cites trust in an agent as being a factor of three powerful phenomena: a) the agent is competent (does things/executes action), b) is altruistic and c) is rational/honest.  In fact trust is best graphed along the four regions of credulity, skeptical mistrust, skeptical trust, and cynicism.  Skeptical trust and skeptical mistrust were cited as being healthier than blind mistrust and blind credulity; Blind mistrust and blind credulity are the real problems warned Norris, who observed that the educated are less trusting. The July 2020 release of the World Values Survey will unveil full measures on interpersonal trust, institutional trust and political trust around the world.

https://platform.twitter.com/widgets.js

Public Opinion and Democracy was the theme of the 72nd WAPOR Annual Conference, hosting 200 delegates from 38 countries in Toronto. Delegates discussed to better understand democracy with relation to how political systems are functioning and evolving.  In that context WAPOR also engaged with host country Canada to help Canadian researchers raise the level of discourse on standards and advocacy challenges in public opinion research in Canada in a day of sessions featuring WAPOR-ESOMAR-CRIC panels.  The WAPOR-ESOMAR-CRIC sessions aimed to move the needle on the relationship between pollsters/polling organizations and the news media.  The media have often come under fire for misrepresenting polls in a headline-hungry world where polls are easy to misread or sensationalize, resulting in the maligned reputations of both polling firms and journalists.  Journalistic and polling rigour demands ongoing and open partnership between researchers and journalists, but also budget (paid media), and strong statistical knowledge, complete transparency and ongoing education. The Poynter Institute conducts courses and training for journalists on polling reportage, best practices and principles including training and courses here.

WAPOR-ESOMAR-CRIC Sessions 1, 2, & 3 : Accuracy of Polls in Canada. Challenges of Public Opinion Research in Canada Overcoming Challenges and Assuring the Future of Public Opinion Polls in Canada

The first panel was chaired by Marita Carballo and kicked off with Jon Puleston of Lightspeed Research introducing Canada’s polling in the global context. It is an action-packed time in elections everywhere. But polls are more accurate the closer they get to election day. He talked about how about 88% of polls everywhere are accurate. And how sources of polling error could be traced back to sampling error.

https://platform.twitter.com/widgets.js

With weighting, need skilled statisticians and transparency. Pollsters must be able to measure shy voter effects. Biggest cause of error is the undecided voter. The level of undecided voter indicates the level of error. The more undecided the voter is, the greater the error.

The undecided voter is the major challenge to polling accuracy.

Tactical voting or last minute votes don’t fall out evenly. Polling embargoes create network errors. Other errors are captured in the snapshot below from Puleston’s presentation.

According to Puleston, polling has been relatively stable over the past 40-50 years.

Regional polling has become more challenging, but national election events have been more or less stable especially in the past 1- 2 years, with about 85% of polls being accurate and 88% within the margin of error, he said.

Next, Christopher Adams offered a retrospective look at the 2017 Calgary election campaign that became widely known for the release of inaccurate polls impacting the elections and all the public discourse around it in his presentation Problematic Polling Conducted During the 2017 Calgary Election. Adams has reviewed 43 public polls in the final five days of all federal and provincial elections held from 2015 to 2018 across live telephone, online and Interactive Voice Response (IVR). As one of three independent reviewers hired by the former MRIA to review the campaign and offer a report with their findings and recommendations for the media and polling industry firms, Adams’ talk centred on explaining the three pillars of their joint report that was neither presented at the MRIA Annual Conference nor published by MRIA before disbanding; their report details:

Adams observed that, “The blurring of the lines between social scientist, pundit and brand-building entrepreneur is made more complex when polling companies enter into ‘exclusive’ contractual relationships with media outlets (even when there is no remuneration involved). Currently, consumers of public opinion research cannot easily appreciate when media outlets are reporting on independently conducted polls and when they are actively promoting findings which they have sponsored. In many cases, news outlets suspend their normal critical reporting practices because they are the sponsors of a particular poll.” Traditional advocacy and standards are being disrupted by social media,  which “as a communications tool has eclipsed letter writing, the telephone, e-mails and face-to-face conversations in terms of influence on social affairs discourse” (Adams 2019). This talk offered a strong background into the next panel that enumerated challenges surrounding the polling and media industries with relation to standards and advocacy in Canada.

In The Quebec 2018 election: A failure of the polls, a late campaign swing, or else? Claire Durand analyzed the Quebec 2018 election, highlighting that a missed poll was vastly different from a failed poll. She reiterated the accuracy levels in global polling (84%) and said that we have never had a polling miss in Canadian federal elections even if we have had them regionally. “On October 1st 2018, the Quebec electoral campaign concluded with a majority government of a “new party” that had never been elected before, the Coalition Avenir Québec (CAQ), a right-wing party. The CAQ obtained 37% of the vote, 12 points above the Quebec Liberal Party (QLP) at 25%.

The two other parties – Parti Québécois (PQ) and Québec Solidaire (QS) – received respectively 17% and 16% of the vote. In the days preceding the Quebec 2018 election vote, the polls generally showed a close race between the two major parties. Therefore, the results came as a major surprise to pollsters and to the public. The M5 absolute error for the two major parties reached an average of 10 for the 5 last polls, which is way higher than the preceding polling miss in Quebec in 1998 (7.4), in the UK in 2015 (6.6) and in the Chile 2017 presidential election (7.2). The literature shows that the possible reasons for polling misses are either a late campaign swing, differential participation, a “shy conservative” phenomena, and/or a biased sampling frame. Durand examined these possible explanations using a recontact survey conducted by Ipsos among the respondents of its last pre-electoral survey. The data show that three phenomena combined to explain the polling miss: a late campaign swing towards CAQ, a tendency of “discrete” respondents to finally vote in greater proportion for CAQ and a small differential participation. The presentation concluded on the necessity to find ways to anticipate that such phenomena are likely to occur and to correct the estimates accordingly. It also raised the question of whether we can conclude to a polling miss without the necessary validation coming from a recontact survey” (Source: WAPOR 2019)

https://platform.twitter.com/widgets.js

The second WAPOR ESOMAR CRIC panel chaired by Claire Durand, University of Montreal, about Challenges of Public Opinion Research in Canada, offered healthy discussion on existing standards, advocacy and the challenges ahead in polling. This panel discussed the political events and economic pressures leading up to the MRIA’s bankruptcy and consequences in terms of standards, advocacy and polling industry challenges ahead.

Christian Bourque, of Léger360, said that strategic voting has not been as important as it is recently with low party loyalty. Polls are not as predictable as they used to be. And today, except for Elizabeth May, all party leaders are unpopular. CAPOR was actually first founded because pollsters led by Darell Bricker, were not satisfied with the standards being followed after the polls by journalists and polling firms and abject non-compliance of the Elections Canada Act. Typically a 75-page report is expected to be released to the public including total transparency, weighting, sampling methods, etc after every poll and everyone is expected to abide by the same standards of total transparency. This was not happening.   

Dr. Annie Pettit, of MOSRCanada, reminded the audience that associations may not act on small complaints because concerned companies may cancel their membership and then the association loses that funding which needs to go towards advocacy. Thus, people/firms realize they don’t have to comply with standards for small things such as only using margin of error for probability samples. This particular example is widespread in Canada. She added that associations may not act on large complaints because they cannot afford repercussions such as lawsuits. Thus, complying with standards for major things is also problematic. Associations defer responsibility for making complaints to individuals thereby reducing the chances that complaints will happen – individuals cannot afford legal action (nor can the associations). When an association goes bankrupt, no one is left to enforce compliance with standards at all. The consequence is that people comply less and less with standards, and while companies may know who they can trust, the public doesn’t know which companies they can trust. The line between innovation and ethics is often a fine one, and the majority of Canadian market researchers are ethical said Dr. Pettit.
In a post-MRIA environment, there is no single source of knowing who represents the firms’ best interests.

Christopher Adams talked about the challenges of being an independent reviewer. He released a statement, but their joint report was delayed due to criticisms of the review. The committee went ahead and released the report independently. Page 62 of their report details the committee’s recommendations. He talked about how associations should work like interest groups in America operating on the three-fold value proposition of material benefits, solidarity benefits, and advocacy (lobbying on behalf of firms for the government, and MRIA’s Do Not Call List was cited as an effective example of advocacy that worked for its members). Public opinion research looks to associations that can protect their public spaces with integrity, said Adams who also advised everyone to think about saving your data for those who come after, piling on the collective memory, and creating a “bandwagon effect” on legacy.

Adam Radwanski of The Globe and Mail detailed the challenges with journalism’s role in the polling process and talked about opportunities to train and educate to the rigour of global polling standards. He said hazy coverage has been a legitimate concern in relation to the media and that there has been a shift in recent year to get beyond the horse race numbers and scope more information on attitudes, behaviours, and voter segments, and understanding what explains results by studying closely outcomes and attitudes. Radwanski talked about ways the media could be more conscientious by educating journalists on the science of polling, contextualizing the numbers and data and outcomes. He then reiterated the common sentiment that media and pollsters could work together about being honest about “not knowing things” to avoid traps like false certainty and hyper confidence – “If a poll looks like an outlier it’s ok to admit it,” he said.

Darrell Bricker of Ipsos Public Affairs said “public opinion research has never been weirder – sampling is harder, everything is harder. A lot of modelling is going on and a lot of change is happening.” He pointed out that in the past polling firms would work with dedicated media partners who were not just paid for, but who also offered their expertise and inputs on research and questionnaire design. These days, said Bricker, it’s all about “who has got whose e-mail.” We have a collective memory of how bad polling has been, he said. There have been instances of the media blaming pollsters for inaccuracies when the problem was with the media not taking responsibility for knowing what we are supposed to disclose. Bricker went on to give the history and reasoning behind why CAPOR was originally founded with the intention of achieving compliance of all firms to the stringent industry standards as laid out in the Elections Canada Act, stemming from a deep dissatisfaction with the status quo. He offered that if pollsters cannot govern the way the media operates, they can at least govern how they disclose. His two major learnings about reactions to CAPOR were that most companies were not active enough (a very nominal portion of any MR firm’s business is reliant on polling), and the media and polling firms were not working closely enough. His own company Ipsos donates all of its election polling money to Wilfrid Laurier University and the University of Toronto. Durand added here, that the transparency can be improved a lot over time.

Cautious Pessimism Precedes Canadian Pollsters

Finally, world renowned US pollster, Kathy Frankovic, chaired the concluding afternoon WAPOR-ESOMAR-CRIC session of Frank Graves, EKOS Research Associates, Christian Bourque, Léger360, and Erik Grenier, CBC News/308. They discussed the outlook ahead and polling best practices. Opinions on polling outlook ranged from mildly pessimistic (Grenier) to moderately pessimistic (Graves) and outright worried (Bourque).

https://platform.twitter.com/widgets.js

CBC’s Grenier is mildly pessimistic. People are much more willing to change their minds now and are harder to predict, he said. Because of the way news is spread now, misunderstanding is faster. I wish there was more money for the proper reporting and education on polling coverage, he said.

Bourque of Leger360 is worried. Make voters great again, he said. We don’t have a likely voter model. There are more concerns on the day of election poll accuracy. We want to improve respondent participation.

Graves, who repeatedly tracks trust in pollsters through his firm Ekos Research Associates, believes major media should get funding. The methodological accuracy for consumers of polls (especially governments) should be a focus. Especially, as the federal government’s polling budget has dropped from $32M to less than $12M since 2006 or about a quarter of what it used to be, he said.

While AI has its share of big problems (and especially in something the scale of polling as he noted), Graves has seen experimental work done in AI and remembers the Obama era aggregators, and is open to considering some of it in his methods perhaps at a micro-level, when tested with the scientific rigour and tools that would allow for more intelligent blending.

https://platform.twitter.com/widgets.js

He further allowed some speculation on the Green Party’s streak of wins. “Are the Greens for real this time around,” was the question. All party leaders are polling much lower than their voter intent, said Graves. We will be facing a growing problem with how newcomers vote and if they participate at all, he added. Moreover, lack of polling data will lead to inaccurate predictions/polls. We will need to work on sample early on, reminded Graves.


Grenier  said, “Polls will be able to tell broad outlines. We may not be able to go into election day with certainty but we can analyze what factors can contribute to the misses.” For example of late, the Progressive Conservatives have tended to over perform of late but knowing that is not enough – we have had warnings in the last few elections where we know who the frontrunners are but because we don’t know who the people (voters) are, predictions become difficult.

Frankovic then asked of the panel, “If I were to look at polls in Canada, what will help?”

Grenier said that Justin Trudeau is no more popular and that has dragged down party popularity for the Liberal Democratic Party. Graves added that the relationship between media and pollsters is not working now as used to. “We’d spend 150K and hire statisticians. Now there’s no budget so it’s a bit of an issue.” Graves has seen lots of interesting trends lately with the Green Party gaining popularity reminiscent of the Orange Wave in 2011 and the National Reform Party in between. Seeing some Conservatives falling back and using a wisdom of crowd approach has worked. There’s room more so than ever to vote strategically he said. Graves talked about joint efforts at doing a large exit poll post elections at CRIC involving the pooling of resources. People don’t understand probability and the average person can’t predict/understand polls. This must be understood in the reporting of polls, he said.

Frankovic invited more comment on the totally changed nature of partnerships between news media and polling organizations.

Bourque said that the reporting of polls is affecting campaigns, so reporting that contextualized rather than sensationalized polls was in order, quoting an example where,”leading by 2 points was not necessarily ‘surging’.”

Graves said that funds for polling have declined over time with methodological and statistical fluency of users and consumers. There have been a number of examples where data was manipulated in the polls e.g. Cambridge Analytica or with Cohn and Trump.  The possibility for fictitious respondents or bots is more real now than before cautioned Graves, whose firm uses probability methods of IVR, CATI and the Knowledge Panel.


Frankovic noted the movement to online polling  and asked, “What about social listening?”

Bourque said, “We use chatbots on Facebook messenger for polls, but nothing we could publish about yet.” Graves added that his firm was trying to use AI in these elections, but seeing significant problems. Using it at microlevel polling  to understand sarcasm from reality is a starting point. Graves also was interested in using the X-box platform for polls.  He supported the use of autodiallers.

https://platform.twitter.com/widgets.js

Finally, Frankovic posed the question about what can CRIC do to make polling better?


Borque reminded everyone that CAPOR was first formed because MRIA was not stringent enough to make all pollsters act according to the Elections Act. Graves who was also on the founding board of CAPOR said that efforts are channeled at increasing the level of standards followed. With CRIC in place, different models of non-police but retrospective audits or analyses of standards would benefit the polling industry, he said.

Graves, who did constant testing on electoral reform and democracy and advocated for regularly scientifically tested polls, said the yardstick for polling firms performance right now is how accurate their victory prediction is.  Our industry and public needs a voice and a continuing ballot box. Borque talked about the need for more party funding, and talked about how in current situation pollsters blackbox their models and aren’t transparent about methods or procedures.  Graves cited as a strong model, the IRPP, who benchmark against known methods, and blend methods to produce statistical accuracy. There are lots of probability options, he said. 

Durand brought to attention the fact that there wasn’t a single headline that said a poll had got it right, failure was more sensational, etc. There was general agreement, and Graves pointed out that it was also a time to lower the hubris and work towards real partnerships. Borque added in conclusion, “Be modest in our claims but make them” with reference to an environment where there is so much emphasis on precision polling than the context surrounding it.

Next year’s AAPOR conference will take place in Atlanta, Georgia and  the 2020 WAPOR conference will be held in Malaysia.

https://platform.twitter.com/widgets.js

Published by Generation1.ca

Leave a Reply