This plugin will crawl the seed URL you give it (crawling means that it will search all links that the webpage contains) and will visit and extract content from each crawled URL. The crawling process is customizable: you can set the crawling depth, crawling rate, maximum crawled article count, crawl only links with specific class or ID and many more customizations.
New features included in this update:
Check the official documentation of the v2 update, browse through examples and check FAQ for crafting a perfectly optimized web scraper.
For more info on how to configure the plugin, please check also this 1 hour long tutorial video, which covers the full feature set of the plugin.
First version released!Version 1.1 Release Date 2017-08-16
Fixed some small issuesVersion 1.2 Release Date 2017-08-17
Added the ability to crawl page by div class or idVersion 1.2.1 Release Date 2017-08-18
Fixed incompatibility with some WordPress installsVersion 1.2.2 Release Date 2017-08-22
Added a shortcode to display post generated by this pluginVersion 1.2.3 Release Date 2017-08-30
Added an option to crawl the page from Google cache when direct crawling fails (blocked)Version 1.2.4 Release Date 2017-08-31
Added the ability to set proxies for crawling pagesVersion 1.2.5 Release Date 2017-09-04
Added the canonicalization for generated articlesVersion 1.2.6 Release Date 2017-09-13
Made the plugin timezone awareVersion 1.2.7 Release Date 2017-09-14
Fixed post date for non gmt blogsVersion 1.2.8 Release Date 2017-09-23
Added paginated post importing supportVersion 1.2.9 Release Date 2017-09-27
BugfixesVersion 1.3.0 Release Date 2017-09-28
Fixed rule restoreVersion 1.3.1 Release Date 2017-10-20
Fixed featured image generationVersion 1.3.2 Release Date 2017-10-22
Added crawling helperVersion 1.3.3 Release Date 2017-11-06
Fixed a memory issueVersion 1.3.4 Release Date 2017-11-07
BugfixesVersion 1.3.5 Release Date 2017-12-14
Fixed class selector not working in all casesVersion 1.3.6 Release Date 2017-12-18
Added the ability to specify a custom user agent for each crawled webpageVersion 1.3.7 Release Date 2018-01-20
Added a new text spinner service: SpinrewriterVersion 1.3.8 Release Date 2018-01-22
Plugin can now continuously import contentVersion 1.3.9 Release Date 2018-02-02
Fixed issue when multiple crawl classes where specifiedVersion 1.4.0 Release Date 2018-02-22
Major update: added the ability to crawl imported product prices (WooCommerce compatible) Added the ability to crawl serial content (paged crawling - crawling for articles will continue on the next page)Version 1.4.1 Release Date 2018-03-07
BugfixesVersion 1.4.2 Release Date 2018-03-21
Fixed a duplicate posting issueVersion 1.4.3 Release Date 2018-03-22
Fixed a critical issue with multiple rule runningVersion 1.4.4 Release Date 2018-04-04
Added the ability to define multiple proxies. The plugin will select one at random at each page accessVersion 1.4.5 Release Date 2018-07-13
Updated built-in readability moduleVersion 1.4.6 Release Date 2018-07-16
Critical bugfixesVersion 1.4.7 Release Date 2018-07-19
Added the ability to not translate linksVersion 1.4.8 Release Date 2018-09-05
Added JavaScript execution support for crawled pages - requires PhantomJS installed on serverVersion 1.4.9 Release Date 2018-09-18
BugfixesVersion 1.5.0 Release Date 2018-09-24
Added the ability to add custom post taxonomies from crawled content Added the ability to add unlimited crawled variables to posts's content/ meta/ taxonomiesVersion 1.5.1 Release Date 2018-10-16
Fixed issue when importing large pagesVersion 1.5.2 Release Date 2018-10-24
Added the ability to shorten links using Shorte.stVersion 1.5.3 Release Date 2018-10-29
Fixed issue when importing paginated postsVersion 1.5.4 Release Date 2018-11-06
Added the ability to strip HTML elements by tag name (div,a,span,etc.)Version 1.5.5 Release Date 2018-11-07
Added WooCommerce product category creation supportVersion 1.5.6 Release Date 2018-12-16
Added nested importing support - import mixed content into a single post, from multiple plugins created by CodeRevolutionVersion 1.5.7 Release Date 2018-12-16
Added the ability to define a list of URLs to skip from crawling and importingVersion 1.5.8 Release Date 2019-01-08
Added the ability to import royalty free images for created postsVersion 1.5.9 Release Date 2019-01-12
Added Gutenberg blocks supportVersion 1.6.0 Release Date 2019-02-01
Added the ability to make screenshots of scraped pagesVersion 1.6.1 Release Date 2019-02-06
Improved compatibility with some crawled pagesVersion 1.6.2 Release Date 2019-04-19
Security updateVersion 1.6.3 Release Date 2019-05-15
Fixed some recently found bugs with post paginationVersion 1.6.4 Release Date 2019-05-17
Added support for TurkceSpin content spinnerVersion 1.6.5 Release Date 2019-05-27
Added a much demanded new feature: Visual Content Selector for assigning scraped page content Added the ability to scrape pages from bottom to top Added the ability to replace words in scraped content Other minor bug fixes and functionality improvementsVersion 1.6.6 Release Date 2019-07-26
Fixed timeout issue with some crawled pages Many small issues fixed and features improvedVersion 1.6.7 Release Date 2019-08-05
Fixed issue with Google TranslateVersion 1.6.8 Release Date 2019-11-15
WordPress 5.3 compatibility updateVersion 1.6.9 Release Date 2020-05-11
New features added for content templates Bugfix updateVersion 1.7.0 Release Date 2020-07-21
Added support for scraping more sitesVersion 1.7.1 Release Date 2020-09-28
Added the ability to crawl sitemaps and to scrape posts linked in them Added the ability to respect the directives set in the robots.txt filesVersion 2.0.0 Release Date 2020-12-08
Added a new shortcode and Gutenberg block alternative that will enable live scraping of any website Major performance improvement Fixed reported bugsVersion 2.1.0 Release Date 2021-01-02
Added support for using the Tor Browser to crawl dark web sites! Scrape .onion websites like you would scrape any other public website!Version 2.1.1 Release Date 2021-01-04
Added the ability to crawl and scrape pages using POST requests (POST form submission scraping support)Version 2.2.0 Release Date 2021-01-14
Added support for HeadlessBrowserAPI to scrape JavaScript rendered content with easeVersion 2.2.1 Release Date 2021-01-16
PHP 8 compatibility update Added support for crawling links from RSS feedsVersion 2.2.2 Release Date 2021-01-28
Fixed rare issue when saving importing rule settings on some PHP 8 configurationsVersion 2.2.3 Release Date 2021-02-01
Improved content extraction algorithmVersion 2.2.4 Release Date 2021-02-17
Added the ability to not spin posts generated by specific rulesVersion 2.2.5 Release Date 2021-03-07
Added the ability to enter multiple URLs (one per line) to be crawled and scrapedVersion 2.2.6 Release Date 2021-03-07
Visual Selector improvements - now it will be able to use HeadlessBrowserAPI/Puppeteer/PhantomJS/Tor to visualize scrape contentVersion 2.2.7 Release Date 2021-04-02
Fixed rare issues when crawling links with URL parametersVersion 2.2.8 Release Date 2021-04-07
Fixed rare issues with relative URL paths in crawled contentVersion 2.2.9 Release Date 2021-05-03
Added the ability to skip publishing of new posts if not images found (separately, for each rule)Version 2.3.0 Release Date 2021-05-19
Added the ability to make screenshots of websites using the HeadlessBrowserAPI featureVersion 2.3.1 Release Date 2021-06-10
Fixed content extracting/stripping in case of some websites with dynamically generated contentVersion 2.3.2 Release Date 2021-07-15
Added multiple Regex expression support (for content stripping and replacement)Version 2.3.3 Release Date 2021-07-18
Added SpinnerChief to the supported premium text spinners (SpinRewriter, The Best Spinner, WordAI, TurkceSpin)Version 2.3.4 Release Date 2021-07-19
Added Bing Translator support (next to Google Translator and DeepL Translator)Version 2.3.5 Release Date 2021-08-06
Added the ability to execute your own custom JavaScript on scraped pages when using headless browsers (PhantomJS/Puppeteer/Tor) or HeadlessBrowserAPI (XSS - cross site scripting feature) and scrape the resulting HTML contentVersion 2.3.6 Release Date 2021-08-30
Added the ability to set featured images of posts from website screenshots Added the ability to remove HTML content (leave text only) of XPath matched contentVersion 2.3.7 Release Date 2021-09-02
Added the ability to set local storage objects when scraping websites (these are similar to cookies, their usage is supported only when using headless browsers or HeadlessBrowserAPI in conjunction with the plugin)Version 2.3.8 Release Date 2021-09-15
Added the ability to set the WPML language to created postsVersion 2.3.9 Release Date 2021-10-19
WooCommerce product scraping related improvementsVersion 2.4.0 Release Date 2022-02-28
Added support for creating WooCommerce product attributes and assign values to them from scraped dataVersion 2.4.1 Release Date 2022-03-05
Added the ability to scrape image galleries for WooCommerce productsVersion 2.4.1.1 Release Date 2022-03-21
Bugfix updateVersion 2.4.2 Release Date 2022-04-20
Fixed Google Translator problem caused by a recent Google API updateVersion 2.5.0 Release Date 2022-05-01
Crawlomatic now can scrape search engine results from Google and Bing - tutorial video:Version 2.5.1 Release Date 2022-05-06
Added the ability to scrape WooCommerce product variations from Shopify and other WooCommerce products Added the ability to automatically detect product prices Improved readability module Fixes and improvementsVersion 2.5.2 Release Date 2022-06-14
Added the ability to translate posts a third time (acting like a Word Spinner, if the content is translated back to the original languageVersion 2.5.3 Release Date 2022-06-23
Fixed WooCommerce price scraping related issueVersion 2.5.4 Release Date 2022-09-12
Added the ability to scrape links from TXT filesVersion 2.5.5 Release Date 2022-10-14
Major update: post/page/product automatic updating if the scraped source URL changedVersion 2.5.6 Release Date 2022-11-30
Major update: added support for Google News scrapingVersion 2.5.7 Release Date 2023-01-05
Added a new ability to HeadlessBrowserAPI to click on HTML elements by CSS selectors, enabling loading of Ajax content and bypassing Captchas which require a clickVersion 2.5.8 Release Date 2023-01-17
Added product regular price scraping feature to WooCommerce products - the regular price is the price displayed before the discount is applied. You can scrape this full price from the websites or add/multiply the original price to create it automaticallyVersion 2.5.9 Release Date 2023-02-10
Fixed Google News scraping after recent changes
Disclaimer
Through this plugin you are able to grab content from various websites that does not necessary belong to you or which are not under your control. If you grab copyrighted material without the author’s permission, the plugin’s developer does not assume any responsibility for your actions. Also, the plugin’s developer has no control over the nature, content and availability of those sites.