{"id":4364,"date":"2017-03-17T20:50:25","date_gmt":"2017-03-18T00:50:25","guid":{"rendered":"https:\/\/digital.hbs.edu\/platform-digit\/submission\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\/"},"modified":"2017-03-17T21:10:14","modified_gmt":"2017-03-18T01:10:14","slug":"taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing","status":"publish","type":"hck-submission","link":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\/","title":{"rendered":"TayTweets with Trolls: Microsoft Research\u2019s Painful Lesson in Conversation Crowdsourcing"},"content":{"rendered":"

A year ago, Microsoft Research launched \u201cTayTweets\u201d (as\u00a0@TayandYou<\/a>), a Twitter \u201cbot\u201d designed to mimic the conversational style of a 19-year-old woman. Tay engaged other Twitter users in conversation, and used their responses to further refine its conversational model in order to better interact with others in the future.<\/p>\n

As DIGIT students have discussed in our examinations of *gramLabs<\/a> and Amazon Alexa<\/a>, effective machine learning applications rely heavily on expansive \u201ctraining\u201d data. The best algorithm in the world is useless without quality data to \u201clearn\u201d from, and obtaining such data for certain applications can be costly or impossible. By engaging other Twitter users in conversation and using their responses to refine Tay\u2019s model, researchers effectively crowdsourced training data for the task, foregoing the cost of manually curating alternative data sources.<\/p>\n

What Happened?<\/h3>\n

Microsoft had previously launched Xiaoice<\/a>\u2014a similar bot that learns from conversations with users\u2014in China to significant fanfare, and anticipated a similarly warm reception for Tay. Of course, by selecting a public medium for holding conversations, researchers left Tay\u2019s learning process exposed to the whims of Twitter\u2019s users and trolls<\/a>. Its conversations quickly went awry.<\/p>\n

\"\"<\/a>
\n(Tay says hello, as its programmers would<\/a>.)<\/small><\/div>\n

Within hours, the bot became inundated with hateful and offensive tweets. As it \u201clearned\u201d indiscriminately from arbitrary conversations, Tay began to respond to certain topics with hateful or controversial remarks that it learned from others. Topics ranged from soundbites cribbed from Donald Trump\u2019s more controversial policy proposals, to outright neo-Nazi and white supremacist rhetoric<\/a>. (The link contains several screenshots of Tay\u2019s more shocking remarks.)<\/p>\n

\"\"<\/a>
\n(One of Tay\u2019s \u201ctamer\u201d learned responses.)<\/small><\/div>\n

Within two days, Microsoft apologized<\/a> for Tay\u2019s behavior and pulled the bot from Twitter, citing the need to perform \u201cupgrades.\u201d Observers speculated<\/a> that, in contrast to China, Americans\u2019 extensive free speech protections may have led Tay to fail where Xiaoice never did. In any case, researchers learned a humbling lesson about the willingness of crowds to supply data not altruistically, but for reasons of malice or amusement.<\/p>\n

In Hindsight<\/h3>\n

Commentators observed that this outcome could have been foreseen<\/a>: users like to \u201ckick the tires\u201d of artificial intelligence systems and understand their limits (or lack thereof). And Twitter\u2019s nature as a public medium provided public payoffs for those with malicious intent.<\/p>\n

With the above in mind, researchers would have been wise to initially train Tay using different data. They might have tried scraping high-traffic Twitter accounts that match the bot\u2019s intended persona, or searching for conversational data elsewhere on the web. To manage Tay\u2019s ongoing learning process, they might have manually tagged users or conversations as appropriate (rather than using all<\/em> data for learning). And to deny trolls the public attention that they crave, designers could have also limited learning to private messages.<\/p>\n

Finally, evidence suggests that researchers programmed some explicit \u201cguardrails\u201d into Tay\u2019s responses, like eschewing discussion of the killing of Eric Garner. The bot\u2019s minders ought to have quickly \u201cblacklisted\u201d similarly contentious or offensive topics (e.g., the Gamergate controversy<\/a>), perhaps with manual filters or by incorporating non-engagement into Tay\u2019s underlying machine learning algorithm.<\/p>\n

The Broader Context<\/h3>\n

It now seems unsurprising, particularly given Twitter\u2019s recent struggles with abuse on its network, that Microsoft\u2019s efforts attracted such vulgar and offensive responses. But Tay\u2019s failure highlights a broader concern about how machine learning and artificial intelligence systems \u201clearn\u201d from data\u2014a concern that will become more salient as such systems increasingly affect peoples\u2019 lives.<\/p>\n

Intuition might suggest that \u201cimpersonal\u201d algorithms represent a way to remove bias in decision-making\u2014that computers will transcend human prejudices and make \u201cmeritorious\u201d and unbiased decisions. But as described above, machine learning is intrinsically reliant on input data. If this data reflects existing human biases, we should not be surprised when trained models reflect those biases. As data collection\u2014an essential step in machine learning\u2014is often expensive, companies that gather data by crowdsourcing or rely on other potentially-biased sources risk repeating Microsoft\u2019s mistake if they uncritically incorporate all data into their models.<\/p>\n

Machine-driven biases have begun to appear in high-stakes, real world decision-making. ProPublica<\/em> reported last year that an algorithm used for \u201cscoring\u201d recidivism risk appeared biased against black offenders<\/a>. That data is not \u201ccrowdsourced\u201d in the same sense that Tay\u2019s was, and the origin of the algorithm\u2019s bias is unclear. But it provides ample evidence that algorithms can affect life-altering decisions, and companies that leverage crowdsourced data ought to think carefully about the impact of bias on their efforts.<\/p>\n","protected":false},"excerpt":{"rendered":"

In March 2016, Microsoft launched Tay, a Twitter bot that could learn from its conversations with others. The experiment quickly unraveled.<\/p>\n","protected":false},"author":444,"featured_media":4365,"comment_status":"open","ping_status":"closed","template":"","categories":[1273,1274,673,366,6,1281,142],"class_list":["post-4364","hck-submission","type-hck-submission","status-publish","has-post-thumbnail","hentry","category-artificial-intelligence","category-chatbots","category-crowdsourcing","category-machine-learning","category-microsoft","category-tay","category-twitter"],"connected_submission_link":"https:\/\/d3.harvard.edu\/platform-digit\/assignment\/managing-digital-crowds\/","yoast_head":"\nTayTweets with Trolls: Microsoft Research\u2019s Painful Lesson in Conversation Crowdsourcing - Digital Innovation and Transformation<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/d3.harvard.edu\/platform-digit\/submission\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"TayTweets with Trolls: Microsoft Research\u2019s Painful Lesson in Conversation Crowdsourcing - Digital Innovation and Transformation\" \/>\n<meta property=\"og:description\" content=\"In March 2016, Microsoft launched Tay, a Twitter bot that could learn from its conversations with others. The experiment quickly unraveled.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/d3.harvard.edu\/platform-digit\/submission\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\/\" \/>\n<meta property=\"og:site_name\" content=\"Digital Innovation and Transformation\" \/>\n<meta property=\"article:modified_time\" content=\"2017-03-18T01:10:14+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/d3.harvard.edu\/platform-digit\/wp-content\/uploads\/sites\/2\/2017\/03\/Tay-Banner.jpeg\" \/>\n\t<meta property=\"og:image:width\" content=\"1500\" \/>\n\t<meta property=\"og:image:height\" content=\"500\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/submission\\\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\\\/\",\"url\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/submission\\\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\\\/\",\"name\":\"TayTweets with Trolls: Microsoft Research\u2019s Painful Lesson in Conversation Crowdsourcing - Digital Innovation and Transformation\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/submission\\\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/submission\\\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2017\\\/03\\\/Tay-Banner.jpeg\",\"datePublished\":\"2017-03-18T00:50:25+00:00\",\"dateModified\":\"2017-03-18T01:10:14+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/submission\\\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/submission\\\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/submission\\\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\\\/#primaryimage\",\"url\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2017\\\/03\\\/Tay-Banner.jpeg\",\"contentUrl\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2017\\\/03\\\/Tay-Banner.jpeg\",\"width\":1500,\"height\":500},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/submission\\\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Submissions\",\"item\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/submission\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"TayTweets with Trolls: Microsoft Research\u2019s Painful Lesson in Conversation Crowdsourcing\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/#website\",\"url\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/\",\"name\":\"Digital Innovation and Transformation\",\"description\":\"MBA Student Perspectives\",\"potentialAction\":[{\"@type\":\"性视界Action\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"TayTweets with Trolls: Microsoft Research\u2019s Painful Lesson in Conversation Crowdsourcing - Digital Innovation and Transformation","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\/","og_locale":"en_US","og_type":"article","og_title":"TayTweets with Trolls: Microsoft Research\u2019s Painful Lesson in Conversation Crowdsourcing - Digital Innovation and Transformation","og_description":"In March 2016, Microsoft launched Tay, a Twitter bot that could learn from its conversations with others. The experiment quickly unraveled.","og_url":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\/","og_site_name":"Digital Innovation and Transformation","article_modified_time":"2017-03-18T01:10:14+00:00","og_image":[{"width":1500,"height":500,"url":"https:\/\/d3.harvard.edu\/platform-digit\/wp-content\/uploads\/sites\/2\/2017\/03\/Tay-Banner.jpeg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\/","url":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\/","name":"TayTweets with Trolls: Microsoft Research\u2019s Painful Lesson in Conversation Crowdsourcing - Digital Innovation and Transformation","isPartOf":{"@id":"https:\/\/d3.harvard.edu\/platform-digit\/#website"},"primaryImageOfPage":{"@id":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\/#primaryimage"},"image":{"@id":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\/#primaryimage"},"thumbnailUrl":"https:\/\/d3.harvard.edu\/platform-digit\/wp-content\/uploads\/sites\/2\/2017\/03\/Tay-Banner.jpeg","datePublished":"2017-03-18T00:50:25+00:00","dateModified":"2017-03-18T01:10:14+00:00","breadcrumb":{"@id":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/d3.harvard.edu\/platform-digit\/submission\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\/#primaryimage","url":"https:\/\/d3.harvard.edu\/platform-digit\/wp-content\/uploads\/sites\/2\/2017\/03\/Tay-Banner.jpeg","contentUrl":"https:\/\/d3.harvard.edu\/platform-digit\/wp-content\/uploads\/sites\/2\/2017\/03\/Tay-Banner.jpeg","width":1500,"height":500},{"@type":"BreadcrumbList","@id":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/taytweets-with-trolls-microsoft-researchs-painful-lesson-in-conversation-crowdsourcing\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/d3.harvard.edu\/platform-digit\/"},{"@type":"ListItem","position":2,"name":"Submissions","item":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/"},{"@type":"ListItem","position":3,"name":"TayTweets with Trolls: Microsoft Research\u2019s Painful Lesson in Conversation Crowdsourcing"}]},{"@type":"WebSite","@id":"https:\/\/d3.harvard.edu\/platform-digit\/#website","url":"https:\/\/d3.harvard.edu\/platform-digit\/","name":"Digital Innovation and Transformation","description":"MBA Student Perspectives","potentialAction":[{"@type":"性视界Action","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/d3.harvard.edu\/platform-digit\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/d3.harvard.edu\/platform-digit\/wp-json\/wp\/v2\/hck-submission\/4364","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/d3.harvard.edu\/platform-digit\/wp-json\/wp\/v2\/hck-submission"}],"about":[{"href":"https:\/\/d3.harvard.edu\/platform-digit\/wp-json\/wp\/v2\/types\/hck-submission"}],"author":[{"embeddable":true,"href":"https:\/\/d3.harvard.edu\/platform-digit\/wp-json\/wp\/v2\/users\/444"}],"replies":[{"embeddable":true,"href":"https:\/\/d3.harvard.edu\/platform-digit\/wp-json\/wp\/v2\/comments?post=4364"}],"version-history":[{"count":0,"href":"https:\/\/d3.harvard.edu\/platform-digit\/wp-json\/wp\/v2\/hck-submission\/4364\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/d3.harvard.edu\/platform-digit\/wp-json\/wp\/v2\/media\/4365"}],"wp:attachment":[{"href":"https:\/\/d3.harvard.edu\/platform-digit\/wp-json\/wp\/v2\/media?parent=4364"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/d3.harvard.edu\/platform-digit\/wp-json\/wp\/v2\/categories?post=4364"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}