{"id":10077,"date":"2019-12-04T05:02:26","date_gmt":"2019-12-04T10:02:26","guid":{"rendered":"https:\/\/digital.hbs.edu\/platform-digit\/submission\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\/"},"modified":"2019-12-04T05:02:26","modified_gmt":"2019-12-04T10:02:26","slug":"putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing","status":"publish","type":"hck-submission","link":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\/","title":{"rendered":"Putting AI bots to the test: Test.ai and the future of app testing"},"content":{"rendered":"

The Evolution of Testing Apps<\/strong><\/p>\n

Test.ai (formerly and legally Appdiff) \u201cprovides an AI-powered test automation platform to help organizations deliver quality mobile app experiences at scale.\u201d[2]<\/a> Founded less than five years ago by software engineers from Google and Microsoft, the company uses AI bots to test mobile apps. A couple decades ago, testing software applications was a manual process performed by testing engineers.[3]<\/a> The engineers developed a test plan based on business requirements, created test cases, and executed the test. Later, automation test engineers would automate parts of the testing process by writing code that replicated the actions a user might take, thereby reducing human effort. Testing has historically been bottleneck for app development teams and one of the most expensive parts of building an app \u2013 each time a change is made to an app\u2019s user interface (UI) \u2013 for example, relabeling an object such as a button \u2013 an engineer might have to rewrite an entire framework for testing.<\/p>\n

Today, testing is evolving even further with AI bots that take on the heavy lifting. In the case of the test.ai platform, bots use computer vision with natural language processing to see and understand everything on the screen and test the app\u2019s workflows as a human would.[4]<\/a> However, because the bots can learn and understand the entirety of the app at superhuman speed, they can run tests for multiple environments or OS\u2019s, multiple devices, and screen sizes \u2013 all simultaneously and continuously. Furthermore, machine learning allows the bots to grow smarter with more iterations and discover things humans may not have thought of. Because the bots look at the visual layer of the app, tests can be written without ever touching the code base of the app. Testing an app that once took months to years, and even still takes days, can now happen in hours or minutes with fewer resources. The key value proposition of test.ai is that it \u201cprovides performance and UI analysis of apps without any upfront setup, code, or overhead\u2026and works in minutes.\u201d[5]<\/a> Additionally, there is a benefit to running the same test on multiple apps. One company\u2019s app can be benchmarked against competitors\u2019 apps, delivering insights on how it performs objectively in certain metrics that drive user experience relative to competitors and opening the door to evaluate and improve quality in new ways.<\/p>\n

https:\/\/www.pinklion.ai\/static\/videos\/app_wall_1min.mp4<\/a><\/video><\/div><\/p>\n

Test.ai bots running tests identify issues with red boxes.<\/p>\n

The founders sought to solve a fundamental problem faced by organizations with digital apps by making testing faster and simpler. As apps grow increasingly complex with new features and integrations and speed to market pressure increases as consumers demand flawless user experiences, organizations run into a coverage gap because they don\u2019t have enough time and labor to test everything through traditional processes.4<\/sup> The business case and value created by the test.ai platform is clear: organizations deploy resources more efficiently and effectively, improve software quality, decrease time to market, and spend less time on quality and focus on more interesting work.<\/p>\n

\"\"<\/a><\/p>\n

\"\"<\/a><\/p>\n

A game-changing opportunity, but with challenges and the need to change itself<\/strong><\/p>\n

It is worth noting that the tech itself is not new \u2013 it has been used by telecom companies and device manufacturers with large app stores such as Google and Apple for years. The opportunity for test.ai is to help organizations improve their software quality at scale by giving them access to such technology. This opportunity poses several challenges. First, without having had exposure to the AI technology, potential enterprise customers have been hesitant to embrace it. As a pioneer in AI-powered testing technology, test.ai must first educate organizations. Moreover, they must help customers identify pain points and help them to understand how AI can be a solution. This surfaces a basic and often seen challenge faced by companies in the AI space at this stage \u2013 shifting the sole focus of the founder team from product\/technology to customer-oriented solutions. In its first few years, test.ai has been focused on the AI technology and less so on marketing or selling a software solution. They must now orient themselves towards the concrete commercial applications of the product.<\/p>\n

Picking up $11M last year in its Series A round, test.ai would do well to hire a leader that can bring marketing, sales, and business development expertise to help the company to create a more customer-oriented culture, build out its marketing\/sales organization, and grow the customer base.[6]<\/a> This is essential for a couple of key reasons: 1) in order for the bots to learn different kinds of applications, they need more test cases to train, and to get more test cases, they need more customers and 2) VC investors are exercising more scrutiny and demanding concrete data and proof that the product is helpful to customers.<\/p>\n

Another way test.ai could increase its marketing capabilities is through partnerships. This is an avenue that is has already begun to pursue through a Partner Program that aims to connect with technology creators, system integrators, and managed service provides. One such partnership is with PinkLion, a full-service AI vendor that provides an AI-first development approach and works with companies to \u201cbuild a custom AI integration layer that ties people, processes, and AI together.\u201d[7]<\/a><\/p>\n

Test.ai also needs to solidify its target market which was initially enterprises with existing test engineering teams that are actively seeking solutions.5<\/sup> This has broadly included any industry. Test.ai must ask itself whether it should continue in this way or whether it might also consider approaching companies that outsource testing and require a full solution. Test.ai could expand is product offering of quality assurance and testing in SaaS model. Currently, customers of test.ai pay an annual subscription fee for each app tested on the platform. While pricing information isn\u2019t publicly available (and, also, likely to be a challenge), the founder has made clear that there are enormous cost savings to be gained by companies using the platform. The suggestion for expanding product offering leads to another recommendation which is to hash out a solid product pipeline.<\/p>\n

Test.ai has a competitive edge from being the leader \u2013 its AI bots are adding to their years of learning as I write this \u2013 but it must make some necessary changes in order to deliver on its goal to automate testing of the world\u2019s apps.<\/p>\n

\"\"<\/a>
AI or not? Unlike test.ai, most testing competitors don’t share their “secret sauce” making it difficult to determine whether there is a true neural network thinking behind the scenes.<\/figcaption><\/figure>\n

[1]<\/a> Tomio Geron, \u201cAppdiff Bags $2.5M for Bots That Test Apps,\u201d Wall Street Journal (Online), September 14, 2016, ABI\/INFORM via ProQuest, accessed December 2019.<\/p>\n

[2]<\/a> \u201cTest.ai Overview,\u201d Crunchbase, https:\/\/www.crunchbase.com\/organization\/testai#section-overview<\/a>, accessed December 2019.<\/p>\n

[3]<\/a> Execute Automation, \u201cAutomating Web Applications with Artificial Intelligence and understand how it works!,\u201d YouTube, uploaded May 19, 2018, https:\/\/www.youtube.com\/watch?v=EcTmKXdYvtM<\/a>, accessed December 2019.<\/p>\n

[4]<\/a> SoftwareTestPro, \u201cRe-shaping the Test Pyramid \u2013 Jennifer Bonine,\u201d YouTube, uploaded August 30, 2019, https:\/\/www.youtube.com\/watch?v=LHyBiR7aR-c<\/a>, accessed December 2019.<\/p>\n

[5]<\/a> Clare McGrane, \u201cAppdiff raises $2.5M for AI-powered automated app testing platform,\u201d GeekWire, September 14, 2016, https:\/\/www.geekwire.com\/2016\/appdiff-raises-2-5m-ai-powered-automated-app-testing-platform\/<\/a>, accessed December 2019.<\/p>\n

[6]<\/a> Lucas Matney, \u201cTest.ai nabs $11M Series A led by Google to put bots to work testing apps,\u201d TechCrunch, July 31, 2018, https:\/\/techcrunch.com\/2018\/07\/31\/test-ai-nabs-11m-series-a-led-by-google-to-put-bots-to-work-testing-apps\/<\/a>, accessed December 2019.<\/p>\n

[7]<\/a> Test.ai, \u201ctest.ai Partner Program,\u201d https:\/\/www.test.ai\/partner\/pinklion<\/a>, accessed December 2019.<\/p>\n","protected":false},"excerpt":{"rendered":"

\u201cIt\u2019s easy to build an app. It\u2019s hard to test it.\u201d \u2013 Liu, co-founder of test.ai[1] <\/p>\n

With 70,000+ apps already tested, Test.ai is on a mission to \u201cimprove the quality of software apps by making app development easier through the power of AI and automation.\u201d<\/p>\n","protected":false},"author":12581,"featured_media":10078,"comment_status":"open","ping_status":"closed","template":"","categories":[877,1273,1207,1535,366,1272,2542],"class_list":["post-10077","hck-submission","type-hck-submission","status-publish","has-post-thumbnail","hentry","category-ai","category-artificial-intelligence","category-bots","category-computer-vision","category-machine-learning","category-ml","category-natural-language-processing","hck-taxonomy-organization-testai","hck-taxonomy-industry-technology","hck-taxonomy-country-united-states"],"connected_submission_link":"https:\/\/d3.harvard.edu\/platform-digit\/assignment\/value-creation-with-ai\/","yoast_head":"\nPutting AI bots to the test: Test.ai and the future of app testing - Digital Innovation and Transformation<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/d3.harvard.edu\/platform-digit\/submission\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Putting AI bots to the test: Test.ai and the future of app testing - Digital Innovation and Transformation\" \/>\n<meta property=\"og:description\" content=\"\u201cIt\u2019s easy to build an app. It\u2019s hard to test it.\u201d \u2013 Liu, co-founder of test.ai[1] With 70,000+ apps already tested, Test.ai is on a mission to \u201cimprove the quality of software apps by making app development easier through the power of AI and automation.\u201d\" \/>\n<meta property=\"og:url\" content=\"https:\/\/d3.harvard.edu\/platform-digit\/submission\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\/\" \/>\n<meta property=\"og:site_name\" content=\"Digital Innovation and Transformation\" \/>\n<meta property=\"og:image\" content=\"https:\/\/d3.harvard.edu\/platform-digit\/wp-content\/uploads\/sites\/2\/2019\/12\/Test.ai_logo_RGB_transp.png\" \/>\n\t<meta property=\"og:image:width\" content=\"2495\" \/>\n\t<meta property=\"og:image:height\" content=\"569\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/submission\\\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\\\/\",\"url\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/submission\\\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\\\/\",\"name\":\"Putting AI bots to the test: Test.ai and the future of app testing - Digital Innovation and Transformation\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/submission\\\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/submission\\\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2019\\\/12\\\/Test.ai_logo_RGB_transp.png\",\"datePublished\":\"2019-12-04T10:02:26+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/submission\\\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/submission\\\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/submission\\\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\\\/#primaryimage\",\"url\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2019\\\/12\\\/Test.ai_logo_RGB_transp.png\",\"contentUrl\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2019\\\/12\\\/Test.ai_logo_RGB_transp.png\",\"width\":2495,\"height\":569},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/submission\\\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Submissions\",\"item\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/submission\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Putting AI bots to the test: Test.ai and the future of app testing\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/#website\",\"url\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/\",\"name\":\"Digital Innovation and Transformation\",\"description\":\"MBA Student Perspectives\",\"potentialAction\":[{\"@type\":\"性视界Action\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/d3.harvard.edu\\\/platform-digit\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Putting AI bots to the test: Test.ai and the future of app testing - Digital Innovation and Transformation","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\/","og_locale":"en_US","og_type":"article","og_title":"Putting AI bots to the test: Test.ai and the future of app testing - Digital Innovation and Transformation","og_description":"\u201cIt\u2019s easy to build an app. It\u2019s hard to test it.\u201d \u2013 Liu, co-founder of test.ai[1] With 70,000+ apps already tested, Test.ai is on a mission to \u201cimprove the quality of software apps by making app development easier through the power of AI and automation.\u201d","og_url":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\/","og_site_name":"Digital Innovation and Transformation","og_image":[{"width":2495,"height":569,"url":"https:\/\/d3.harvard.edu\/platform-digit\/wp-content\/uploads\/sites\/2\/2019\/12\/Test.ai_logo_RGB_transp.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\/","url":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\/","name":"Putting AI bots to the test: Test.ai and the future of app testing - Digital Innovation and Transformation","isPartOf":{"@id":"https:\/\/d3.harvard.edu\/platform-digit\/#website"},"primaryImageOfPage":{"@id":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\/#primaryimage"},"image":{"@id":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\/#primaryimage"},"thumbnailUrl":"https:\/\/d3.harvard.edu\/platform-digit\/wp-content\/uploads\/sites\/2\/2019\/12\/Test.ai_logo_RGB_transp.png","datePublished":"2019-12-04T10:02:26+00:00","breadcrumb":{"@id":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/d3.harvard.edu\/platform-digit\/submission\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\/#primaryimage","url":"https:\/\/d3.harvard.edu\/platform-digit\/wp-content\/uploads\/sites\/2\/2019\/12\/Test.ai_logo_RGB_transp.png","contentUrl":"https:\/\/d3.harvard.edu\/platform-digit\/wp-content\/uploads\/sites\/2\/2019\/12\/Test.ai_logo_RGB_transp.png","width":2495,"height":569},{"@type":"BreadcrumbList","@id":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/putting-ai-bots-to-the-test-test-ai-and-the-future-of-app-testing\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/d3.harvard.edu\/platform-digit\/"},{"@type":"ListItem","position":2,"name":"Submissions","item":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/"},{"@type":"ListItem","position":3,"name":"Putting AI bots to the test: Test.ai and the future of app testing"}]},{"@type":"WebSite","@id":"https:\/\/d3.harvard.edu\/platform-digit\/#website","url":"https:\/\/d3.harvard.edu\/platform-digit\/","name":"Digital Innovation and Transformation","description":"MBA Student Perspectives","potentialAction":[{"@type":"性视界Action","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/d3.harvard.edu\/platform-digit\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/d3.harvard.edu\/platform-digit\/wp-json\/wp\/v2\/hck-submission\/10077","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/d3.harvard.edu\/platform-digit\/wp-json\/wp\/v2\/hck-submission"}],"about":[{"href":"https:\/\/d3.harvard.edu\/platform-digit\/wp-json\/wp\/v2\/types\/hck-submission"}],"author":[{"embeddable":true,"href":"https:\/\/d3.harvard.edu\/platform-digit\/wp-json\/wp\/v2\/users\/12581"}],"replies":[{"embeddable":true,"href":"https:\/\/d3.harvard.edu\/platform-digit\/wp-json\/wp\/v2\/comments?post=10077"}],"version-history":[{"count":0,"href":"https:\/\/d3.harvard.edu\/platform-digit\/wp-json\/wp\/v2\/hck-submission\/10077\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/d3.harvard.edu\/platform-digit\/wp-json\/wp\/v2\/media\/10078"}],"wp:attachment":[{"href":"https:\/\/d3.harvard.edu\/platform-digit\/wp-json\/wp\/v2\/media?parent=10077"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/d3.harvard.edu\/platform-digit\/wp-json\/wp\/v2\/categories?post=10077"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}