holm/se-scraper
1
0
Fork 0
forked from extern/se-scraper
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Commit Graph

  • 0d7778e36a
         Bump flat and mocha dependabot/npm_and_yarn/flat-and-mocha-5.0.2 dependabot[bot] 2022-12-30 17:55:13 +0000
  • 61ae3d38b9
         Bump express from 4.17.1 to 4.17.3 dependabot/npm_and_yarn/express-4.17.3 dependabot[bot] 2022-12-11 01:49:58 +0000
  • de72ead81d
         Bump qs and express dependabot/npm_and_yarn/qs-and-express-6.11.0 dependabot[bot] 2022-12-06 21:32:16 +0000
  • 078832b2b0
         Bump got from 9.6.0 to 11.8.5 dependabot/npm_and_yarn/got-11.8.5 dependabot[bot] 2022-06-22 05:38:18 +0000
  • cde1ba1128
         Bump async from 2.6.3 to 2.6.4 dependabot/npm_and_yarn/async-2.6.4 dependabot[bot] 2022-04-29 03:31:09 +0000
  • 0aa70dbeff
         Bump pathval from 1.1.0 to 1.1.1 dependabot/npm_and_yarn/pathval-1.1.1 dependabot[bot] 2022-02-11 00:35:41 +0000
  • 8e2d4b7d9d
         Bump merge-deep from 3.0.2 to 3.0.3 dependabot/npm_and_yarn/merge-deep-3.0.3 dependabot[bot] 2021-06-08 02:24:04 +0000
  • 9fb66e1159
         Bump lodash from 4.17.14 to 4.17.21 dependabot/npm_and_yarn/lodash-4.17.21 dependabot[bot] 2021-05-07 23:52:01 +0000
  • 4349c37528
         Bump ua-parser-js from 0.7.21 to 0.7.24 dependabot/npm_and_yarn/ua-parser-js-0.7.24 dependabot[bot] 2021-05-07 20:45:29 +0000
  • 49ba3bef1d
         Bump underscore from 1.9.1 to 1.13.1 dependabot/npm_and_yarn/underscore-1.13.1 dependabot[bot] 2021-05-06 20:55:09 +0000
  • 78cc7f1a2f
         Bump y18n from 4.0.0 to 4.0.1 dependabot/npm_and_yarn/y18n-4.0.1 dependabot[bot] 2021-03-30 18:35:56 +0000
  • 5a0eea201d Merge branch 'master' of github.com:NikolaiT/se-scraper master Nikolai Tschacher 2020-05-17 22:06:57 +0200
  • 0278b24f0d ad Nikolai Tschacher 2020-05-17 22:06:33 +0200
  • 33fa371716
         Merge pull request #62 from aularon/patch-1 Nikolai Tschacher 2020-02-13 20:04:20 +0100
  • 6b806dedfe
         Merge pull request #61 from Monibrand/refactor/use-original-puppeteer-cluster Nikolai Tschacher 2020-02-13 20:03:39 +0100
  • 5633b10e50
         Merge pull request #60 from Monibrand/fix/unusable-proxy-file-option Nikolai Tschacher 2020-02-13 20:02:55 +0100
  • c58d4fa74d fix(proxy): throw on use_proxies_only if no proxies given HugoPoi 2020-01-17 15:55:17 +0100
  • 4f467abf1e fix(scrape-manager): keywords propagated through a clone config for not being re-affected HugoPoi 2020-01-17 15:12:00 +0100
  • 89dc5c3ebb fix(scrape-manager): conflict between proxies and user_agent option HugoPoi 2020-01-17 12:07:12 +0100
  • 4b33ef9b19 fix(duckduckgo): extract correct amount of results, handle pagination HugoPoi 2020-01-15 16:33:55 +0100
  • 28332528ea test(duckduckgo): implement tests for duckduckgo module HugoPoi 2020-01-15 16:33:30 +0100
  • b685fb4def test: working test for html_output HugoPoi 2020-01-10 09:51:54 +0100
  • 394b567db6 test: add user_agent tests, add html_output tests HugoPoi 2020-01-10 09:35:24 +0100
  • cac6b87e92 test: Bing tests working, refactor proxy for tests HugoPoi 2020-01-08 14:40:28 +0100
  • 1c1db88545 test: add config proxy options tests HugoPoi 2020-01-07 16:50:09 +0100
  • 8f6317cea7 style: add debug trace on some file HugoPoi 2020-01-07 16:47:09 +0100
  • f192e4ebb4 test: remove legacy tests HugoPoi 2020-01-07 09:53:59 +0100
  • 3ab8e46126 test: add bing module test HugoPoi 2020-01-07 09:48:46 +0100
  • 392c43390e test(google): add real integration/unit tests for google module HugoPoi 2020-01-03 19:21:34 +0100
  • 77c1bb8372
         Take screenshot before modifying HTML aularon 2020-01-03 11:12:40 +0200
  • 8f40057534 refactor(cluster): use custom concurrency for puppeteer-cluster HugoPoi 2019-12-20 19:44:59 +0100
  • 301695cd2b fix(scrape-manager): proxy_file options can be used with proxies default value HugoPoi 2019-12-20 19:35:23 +0100
  • d362e4ae2c
         Merge pull request #59 from TDenoncin/refactor/logging Nikolai Tschacher 2019-12-20 14:42:09 +0100
  • bcd181111b refactor(log): remove common.js, use winston and debug HugoPoi 2019-12-15 17:56:22 +0100
  • b4a86fcc51 refactor(proxy): remove proxy option not working replace by proxies HugoPoi 2019-12-12 15:16:37 +0100
  • 9e6a555663
         Merge pull request #52 from kujaomega/master Nikolai Tschacher 2019-12-01 22:15:39 +0100
  • ca9f5f7f50 Added post install script to build the puppeteer-cluster, and also added the updated dependencies from puppeteer-cluster David Solé 2019-11-22 00:37:29 +0100
  • 1694ee92d0 updated to puppeteeer 2.0 Nikolai Tschacher 2019-11-08 16:21:16 +0100
  • da69913272 added detected status to metadata Nikolai Tschacher 2019-10-06 15:34:18 +0200
  • 4a3a0e6fd4 better pluggable api Nikolai Tschacher 2019-10-05 19:39:33 +0200
  • 4953d9da7a chaned version Nikolai Tschacher 2019-09-23 23:39:06 +0200
  • 5e47c27c70 too late to find a proper commit description Nikolai Tschacher 2019-09-23 23:38:38 +0200
  • 95a5ee56d8 remove cheerio from parsing Nikolai Tschacher 2019-09-23 21:57:13 +0200
  • 52a2ec7b33 changed README Nikolai Tschacher 2019-09-23 16:50:57 +0200
  • 07f3dceba1 fixed google SERP title, better docker support Nikolai Tschacher 2019-09-23 16:46:22 +0200
  • b25f7a4285 added test to my working tree Nikolai Tschacher 2019-09-13 18:28:19 +0200
  • 4b581bd03f removed static tests because they are too larege Nikolai Tschacher 2019-09-13 18:21:17 +0200
  • 21378dab02 removed some search engines, added tests for existing, added yandex search engines Nikolai Tschacher 2019-09-13 16:15:33 +0200
  • 77d6c4f04a removed some stuff Nikolai Tschacher 2019-09-12 10:43:57 +0200
  • b513bb0f5b Merge branch 'master' of github.com:NikolaiT/se-scraper Nikolai Tschacher 2019-09-04 12:28:05 +0200
  • 855a874f9e some minor changes Nikolai Tschacher 2019-09-04 12:27:53 +0200
  • dde1711d9d
         Merge pull request #45 from slotix/master Nikolai Tschacher 2019-08-29 20:41:42 +0200
  • 7ba7ee9226 add process supervisor for starting server.js slotix 2019-08-19 14:01:37 +0200
  • e661241f6f added some parsing to google Nikolai Tschacher 2019-08-16 20:10:40 +0200
  • 98414259fe docker support added Nikolai Tschacher 2019-08-13 17:35:06 +0200
  • 19a172c654 better tests Nikolai Tschacher 2019-08-13 15:28:30 +0200
  • 0f7e89c272 added little bug in cleaning Nikolai Tschacher 2019-08-12 17:16:37 +0200
  • ca941cee45 added static bing test, added html cleaning when exporting html Nikolai Tschacher 2019-08-12 16:05:17 +0200
  • 4c77aeba76
         Merge pull request #42 from TDenoncin/error-management Nikolai Tschacher 2019-08-12 00:04:40 +0200
  • 0427d9f915
         Merge branch 'master' into error-management Nikolai Tschacher 2019-08-12 00:04:27 +0200
  • 87fcdd35d5 readme in static tests Nikolai Tschacher 2019-08-12 00:01:02 +0200
  • 4ca50ab2b9 added new static test case that runs much faster and tests a lot of behavior Nikolai Tschacher 2019-08-11 23:58:10 +0200
  • 8e629f6266
         Merge pull request #41 from victor9000/master Nikolai Tschacher 2019-08-08 21:57:14 +0200
  • a369bd07f9 Add "use strict" to ensure quality code control HugoPoi 2019-08-06 12:18:51 +0200
  • dde2b14fc0 Remove uneeded try catch block in Google Search module HugoPoi 2019-08-06 11:50:08 +0200
  • 0db6e068da Remove uneeded try catch block Add proper error for ip matching test HugoPoi 2019-08-06 11:46:53 +0200
  • 50bda275a6 Clean integration tests for mocha HugoPoi 2019-08-02 17:41:29 +0200
  • a61fade2c9 Fix broken Google News selectors, fixes #40 Victor 2019-08-04 14:25:13 -0700
  • 78fe12390b better user agents now, added option to include screenshots as base64 in results Nikolai Tschacher 2019-07-18 20:19:15 +0200
  • fcbe66b56b using random user agents now from https://github.com/intoli/user-agents Nikolai Tschacher 2019-07-18 19:34:09 +0200
  • 59154694f2 fixed issue https://github.com/NikolaiT/se-scraper/issues/37 Nikolai Tschacher 2019-07-18 19:14:33 +0200
  • 60a9d52924 add fucking google product information Nikolai Tschacher 2019-07-11 19:23:40 +0200
  • 1fc7f0d1c8 fixed a badboy Nikolai Tschacher 2019-07-11 16:54:32 +0200
  • baaff5824e ... Nikolai Tschacher 2019-07-11 16:43:41 +0200
  • dab25f9068 added google shopping results Nikolai Tschacher 2019-07-11 16:42:01 +0200
  • a413cb54ef parsing ads works for duckduckgo, google, bing. tested. Nikolai Tschacher 2019-07-07 19:38:28 +0200
  • bbebe3ce60 parsing ads is supported now for google, bing and duckduckgo Nikolai Tschacher 2019-07-06 21:42:13 +0200
  • 09c1255400 removed some superflous stuff Nikolai Tschacher 2019-07-02 18:04:01 +0200
  • 5e8ff1cb34 Merge branch 'master' of https://github.com/NikolaiT/se-scraper Nikolai Tschacher 2019-06-29 17:01:25 +0200
  • c1a036e8da removed some stuff Nikolai Tschacher 2019-06-29 17:00:50 +0200
  • d1e9b21269 added google maps scraper Nikolai Tschacher 2019-06-29 17:00:19 +0200
  • 593f3a95e5
         Merge pull request #33 from TDenoncin/add-html-output-rework Nikolai Tschacher 2019-06-26 15:38:38 +0200
  • d9ac9f4162 Add test for html_output, refactor the results return HugoPoi 2019-06-26 12:03:42 +0200
  • a0e63aa4b0 Use bing_setting.bing_domain if defined for startUrl Thomas 2019-04-26 15:13:51 +0200
  • a3ebe357a4 Add html_output fonctionality Pagination support for html output Change return value to keep it compliant to the current version of se-scrapper Thomas 2019-04-18 15:23:01 +0200
  • 0d7f6dcd11 worked on issue #31 Nikolai Tschacher 2019-06-18 22:23:52 +0200
  • 80d23a9d57 users may pass their own user agents, different browsers have random user agents and not the same now Nikolai Tschacher 2019-06-17 21:25:45 +0200
  • ebe9ba8ea9 added option to throw on detection Nikolai Tschacher 2019-06-17 15:02:44 +0200
  • caa93df3b0 random user agent fixed Nikolai Tschacher 2019-06-17 12:01:13 +0200
  • 0c9f353cb2 remove hardcoded sleep() in Google Image Nikolai Tschacher 2019-06-17 00:03:13 +0200
  • 43d5732de7 resolved issue #30, custom scrapers now possible. new npm version Nikolai Tschacher 2019-06-13 12:34:39 +0200
  • 06d500f75c . Nikolai Tschacher 2019-06-12 21:25:40 +0200
  • 784e887787 fixed issue #22 Nikolai Tschacher 2019-06-12 21:25:20 +0200
  • db5fbb23d2 removed unnecessary sleeping times Nikolai Tschacher 2019-06-12 18:14:49 +0200
  • 5bf7c94b9a new version Nikolai Tschacher 2019-06-11 22:01:27 +0200
  • d4d06f7d67 need to edit readme Nikolai Tschacher 2019-06-11 18:34:51 +0200
  • 35943e7449 minor stuff Nikolai Tschacher 2019-06-11 18:33:11 +0200
  • 7e06944fa1 updated README Nikolai Tschacher 2019-06-11 18:27:34 +0200
  • 6825c97790 changed api big time Nikolai Tschacher 2019-06-11 18:16:59 +0200
  • 3d69f4e249 added a proxy test script Nikolai Tschacher 2019-05-06 21:54:23 +0200