Merge pull request #130 from yujiosaka/browser_cache_option

Browser cache option

CHANGELOG.md

@@ -6,6 +6,9 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+- Support `browserCache` for [crawler.queue()](https://github.com/yujiosaka/headless-chrome-crawler#crawlerqueueoptions)'s options.
+- Support `depthPriority` option again.
+
 ## [1.3.4] - 2018-02-22
 
 ### changed

README.md

@@ -182,7 +182,7 @@ browserWSEndpoint, ignoreHTTPSErrors
 Also, the following options can be set as default values when [crawler.queue()](#crawlerqueueoptions) are executed.
 
 ```
-url, allowedDomains, deniedDomains, timeout, priority, depthPriority, delay, retryCount, retryDelay, jQuery, device, username, password, evaluatePage
+url, allowedDomains, deniedDomains, timeout, priority, depthPriority, delay, retryCount, retryDelay, jQuery, browserCache, device, username, password, evaluatePage
 ```
 
 > **Note**: In practice, setting the options every time you queue equests is redundant. Therefore, it's recommended to set the default values and override them depending on the necessity.
@@ -222,7 +222,7 @@ ignoreHTTPSErrors, headless, executablePath, slowMo, args, ignoreDefaultArgs, ha
 Also, the following options can be set as default values when [crawler.queue()](#crawlerqueueoptions) are executed.
 
 ```
-url, allowedDomains, deniedDomains, timeout, priority, depthPriority, delay, retryCount, retryDelay, jQuery, device, username, password, evaluatePage
+url, allowedDomains, deniedDomains, timeout, priority, depthPriority, delay, retryCount, retryDelay, jQuery, browserCache, device, username, password, evaluatePage
 ```
 
 > **Note**: In practice, setting the options every time you queue the requests is redundant. Therefore, it's recommended to set the default values and override them depending on the necessity.
@@ -251,6 +251,7 @@ url, allowedDomains, deniedDomains, timeout, priority, depthPriority, delay, ret
   * `retryCount` <[number]> Number of limit when retry fails, defaults to `3`.
   * `retryDelay` <[number]> Number of milliseconds after each retry fails, defaults to `10000`.
   * `jQuery` <[boolean]> Whether to automatically add [jQuery](https://jquery.com) tag to page, defaults to `true`.
+  * `browserCache` <[boolean]> Whether to enable browser cache for each request, defaults to `true`.
   * `device` <[string]> Device to emulate. Available devices are listed [here](https://github.com/GoogleChrome/puppeteer/blob/master/DeviceDescriptors.js).
   * `username` <[string]> Username for basic authentication. pass `null` if it's not necessary.
   * `screenshot` <[Object]> Screenshot option, defaults to `null`. This option is passed to [Puppeteer's page.screenshot()](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagescreenshotoptions). Pass `null` or leave default to disable screenshot.

lib/crawler.js

@@ -80,6 +80,7 @@ class Crawler {
       this._preventNewTabs(),
       this._authenticate(),
       this._emulate(),
+      this._setCacheEnabled(),
       this._setUserAgent(),
       this._setExtraHeaders(),
       this._handlePageEvents(),
@@ -118,6 +119,15 @@ class Crawler {
     return this._page.emulate(devices[this._options.device]);
   }
 
+  /**
+   * @return {!Promise}
+   * @private
+   */
+  _setCacheEnabled() {
+    if (this._options.browserCache) return Promise.resolve();
+    return this._page.setCacheEnabled(false);
+  }
+
   /**
    * @return {!Promise}
    * @private

lib/hccrawler.js

@@ -111,6 +111,7 @@ class HCCrawler extends EventEmitter {
       retryCount: 3,
       retryDelay: 10000,
       jQuery: true,
+      browserCache: true,
       persistCache: false,
       skipDuplicates: true,
       depthPriority: true,