Data, data everywhere, but not a byte to crunch.\u00a0 This expertly-crafted opening sentence captures the sentiment of many an aspiring analyst as he or she crawls through the tubes of the internet, desperate for tabular, structured data to feed into a project.\u00a0 While companies might have more structured data than they know how to handle, the opposite is true for most people outside of a firm\u2019s org structure.<\/p>\n
<\/p>\n
This is understandable:\u00a0 data is a competitive advantage, so sharing ought to be done, if at all, judiciously.\u00a0 But in so doing, data owners take the limitless power of the crowd down a peg.<\/p>\n
Until the storied data singularity occurs, then, crawler & scraper technology fills the gap.\u00a0 Borrowing a page* from search engines, these automatons wander the web, downloading & storing whatever data they\u2019re instructed to.\u00a0 With a little python & a lot of moxie, users can build datasets in exponentially shorter amounts of time than manually harvesting.<\/p>\n
Some companies have fed their cash cows on the grass of web scraping tech.\u00a0 In a well-known lawsuit, internet artifact craigslist.org set their legal team on 3Taps, a company that scraped all Craigslist postings & sold to third-parties (most famously, padmapper.com built its reputation on Craigslist data scraped by 3Taps).<\/p>\n
To be clear:\u00a0 scraping is not in-and-of-itself illegal.\u00a0 In fact, the technology does little more than replicate the same requests of servers that a user\u2019s browser might.\u00a0 So long as a scraper doesn\u2019t overload the server with request (a popular weapon used in \u201cDenial of Service\u201d attacks, often used by ne\u2019er-do-wells to bring-down webservers), the entire process is on the up-and-up.<\/p>\n
Where 3Taps went wrong, then, was monetizing the asset on the back-end\u2014data to which Craigslist held exclusive rights*.\u00a0 But the fruit of the data tree proved too sweet for other companies to resist a taste in the wake of the Craigslist fiasco.<\/p>\n
Enter \u201cimport.io.\u201d\u00a0 This young company has two notable components to their business model:\u00a0 the first is a user-friendly, powerful, and graphical utility for crawling the web.\u00a0 Using import.io solutions for the first time, this author was able to harvest thousands of posts from a forum with just 15 minutes of setup–no lines of code, and only a few trips to the documentation.\u00a0 To put that in perspective, using some of the more popular code-based scraper solutions, only an expert user could have built that sort of scraper in so little time.<\/p>\n
https:\/\/youtube.com\/watch?v=cdmsTxu45-c<\/p>\n
The second component of import.io is where the company got especially clever.\u00a0 Much in the same way that Duolingo provides free language lessons in-exchange for crowd-sourced translation services, import.io puts its army of users to work.\u00a0 Their enterprise package boasts that \u201cevery day our powerful infrastructure collects millions of data records from the web,\u201d access to which they sell at a premium.<\/p>\n
For a single company to offer just this second component would be a tremendous burden. Every site\u2019s structure is different, and that typically calls for a custom scraper program.\u00a0 No company could write & maintain scrapers for every site on the web.\u00a0 So, import.io doesn\u2019t.\u00a0 With every user building scrapers, tabularizing & organizing the data, and (most importantly) updating their scraper when a site changes, import.io has an unprecedented army of data harvesters at their fingers.<\/p>\n
For import.io and the few competitors in this space, the future is ostensibly bright.\u00a0 Because their critical mass of data likely doesn\u2019t depend on any one site (like 3Taps & padmapper did), they\u2019re able to adjust to company\u2019s response to scraping. This is clearly on their radar, as their documentation pretty explicitly states that they respect target sites\u2019 wishes.<\/p>\n
The most obvious room for growth for import.io is the flexibility of their scraper solution.\u00a0 Programmatic scraping might be difficult, but it\u2019s almost infinitely extensible\u2014something a graphical tool can rarely claim.\u00a0 Until big-dog open-source solutions like Scrapy & Selenium are no longer necessary, import.io will not be able to address the entire market.<\/p>\n
*Pun.<\/p>\n
**There\u2019s more to it, including 3Taps circumventing of Craigslist\u2019s efforts to block them, but this is sufficient for our purposes.<\/p>\n","protected":false},"excerpt":{"rendered":"
Why should just the big-shots have access to an unlimited flow of data from the unstructured web?<\/p>\n","protected":false},"author":68,"featured_media":2737,"comment_status":"open","ping_status":"closed","template":"","categories":[655,1006],"class_list":["post-2736","hck-submission","type-hck-submission","status-publish","has-post-thumbnail","hentry","category-data","category-web-scraping"],"connected_submission_link":"https:\/\/d3.harvard.edu\/platform-digit\/assignment\/data-driven-value-creation-value-capture-and-operating-models\/","yoast_head":"\n