se-scraper/TODO.md

### 24.12.2018
    - fix interface to scrape() [DONE]
    - add to Github


### 24.1.2018
    - fix issue #3: add functionality to add keyword file

### 27.1.2019
    - Add functionality to block images and CSS from loading as described here:
        https://www.scrapehero.com/how-to-increase-web-scraping-speed-using-puppeteer/
        https://www.scrapehero.com/how-to-build-a-web-scraper-using-puppeteer-and-node-js/

### 29.1.2019
    - implement proxy support functionality
        - implement proxy check

    - implement scraping more than 1 page
        - do it for google
        - and bing
    - implement duckduckgo scraping


### 30.1.2019
    - modify all scrapers to use the generic class where it makes sense
        - Bing, Baidu, Google, Duckduckgo

### 7.2.2019
    - add num_requests to test cases [done]

### 25.2.2019
    - https://antoinevastel.com/crawler/2018/09/20/parallel-crawler-puppeteer.html
    - add support for browsing with multiple browsers, use this neat library:
    - https://github.com/thomasdondorf/puppeteer-cluster [done]
    
    
### 28.2.2019
    - write test case for multiple browsers/proxies
    - write test case and example for multiple tabs with bing
    - make README.md nicer. https://github.com/thomasdondorf/puppeteer-cluster/blob/master/README.md as template


### 11.6.2019
    - TODO: fix amazon scraping
    - change api of remaining test cases [done]
    - TODO: implement custom search engine parameters on scrape()
    
### 12.6.2019
    - remove unnecessary sleep() calls and replace with waitFor selectors


### 16.7.2019

- resolve issues
    - fix this https://github.com/NikolaiT/se-scraper/issues/37 [done]
    
- use puppeteer stealth plugin: https://www.npmjs.com/package/puppeteer-extra-plugin-stealth

    - we will need to load at the concurrency impl of puppeteer-cluster [no typescript support :(), I will not support this right now]

- user random user agents plugin: https://github.com/intoli/user-agents [done]

- add screenshot capability (make the screen after parsing)
    - store as b64 [done]



### 12.8.2019

- add static test case for bing [done]
- add options that minimize `html_output` flag: 
    `clean_html_output` will remove all JS and CSS from the html 
    `clean_data_images` removes all data images from the html
    [done]
    
    
### 13.8.2019
- Write test case for clean html output [done]
- Consider better compression algorithm. [done] There is the brotli algorithm, but this is only supported
  in very recent versions of nodejs
- what else can we remove from the dom [done] Removing comment nodes now! They are large in BING.
- remove all whitespace and \n and \t from html

### TODO:
1. fix googlenewsscraper waiting for results and parsing. remove the static sleep [done]
2. when using multiple browsers and random user agent, pass a random user agent to each perBrowserOptions

3. dont create a new tab when opening a new scraper