extern/se-scraper
1
0
Fork 1
mirror of https://github.com/NikolaiT/se-scraper.git synced 2024-11-21 23:23:07 +01:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • ed346ae9bf
         Merge 0d7778e36a into 5a0eea201d dependabot[bot] 2022-12-30 17:55:15 +0000
  • 0d7778e36a
         Bump flat and mocha dependabot/npm_and_yarn/flat-and-mocha-5.0.2 dependabot[bot] 2022-12-30 17:55:13 +0000
  • 701ffd474f
         Merge 61ae3d38b9 into 5a0eea201d dependabot[bot] 2022-12-11 01:50:00 +0000
  • 61ae3d38b9
         Bump express from 4.17.1 to 4.17.3 dependabot/npm_and_yarn/express-4.17.3 dependabot[bot] 2022-12-11 01:49:58 +0000
  • a3b360e300
         Merge de72ead81d into 5a0eea201d dependabot[bot] 2022-12-06 21:32:18 +0000
  • de72ead81d
         Bump qs and express dependabot/npm_and_yarn/qs-and-express-6.11.0 dependabot[bot] 2022-12-06 21:32:16 +0000
  • eebf3b6de0
         Merge 078832b2b0 into 5a0eea201d dependabot[bot] 2022-06-22 05:38:20 +0000
  • 078832b2b0
         Bump got from 9.6.0 to 11.8.5 dependabot/npm_and_yarn/got-11.8.5 dependabot[bot] 2022-06-22 05:38:18 +0000
  • a3dae8a5a3
         Merge 2d833679f7 into 5a0eea201d Kazerlelutin 2022-05-18 04:34:00 -0400
  • 2d833679f7 fix(*): start urls Benoist Bouteiller 2022-05-16 16:36:47 +0200
  • 1259c837e3 feat(*): add custom url Benoist Bouteiller 2022-05-16 09:18:22 +0200
  • 8bac76ba93
         Merge cde1ba1128 into 5a0eea201d dependabot[bot] 2022-04-29 03:31:10 +0000
  • cde1ba1128
         Bump async from 2.6.3 to 2.6.4 dependabot/npm_and_yarn/async-2.6.4 dependabot[bot] 2022-04-29 03:31:09 +0000
  • 1e76e21681
         Merge 0aa70dbeff into 5a0eea201d dependabot[bot] 2022-02-11 00:35:43 +0000
  • 0aa70dbeff
         Bump pathval from 1.1.0 to 1.1.1 dependabot/npm_and_yarn/pathval-1.1.1 dependabot[bot] 2022-02-11 00:35:41 +0000
  • 92817b3977 watermark fork version in package.json and README.md HugoPoi 2019-08-06 12:44:41 +0200
  • 24daadf0b1 fix: mobile gdpr popup scroll too long todo HugoPoi 2021-07-19 17:14:01 +0200
  • b8a620025e fix(google): gdpr popup on mobile need to be scrolled HugoPoi 2021-07-19 14:47:11 +0200
  • a73ccbce1a fix(google): gdpr auto accept HugoPoi 2021-07-16 17:05:00 +0200
  • 3abb375c07
         Merge 8e2d4b7d9d into 5a0eea201d dependabot[bot] 2021-06-08 02:24:06 +0000
  • 8e2d4b7d9d
         Bump merge-deep from 3.0.2 to 3.0.3 dependabot/npm_and_yarn/merge-deep-3.0.3 dependabot[bot] 2021-06-08 02:24:04 +0000
  • 9d424df3fd
         Merge 9fb66e1159 into 5a0eea201d dependabot[bot] 2021-05-07 23:52:02 +0000
  • 9fb66e1159
         Bump lodash from 4.17.14 to 4.17.21 dependabot/npm_and_yarn/lodash-4.17.21 dependabot[bot] 2021-05-07 23:52:01 +0000
  • 55d128b5d7
         Merge 4349c37528 into 5a0eea201d dependabot[bot] 2021-05-07 20:45:31 +0000
  • 4349c37528
         Bump ua-parser-js from 0.7.21 to 0.7.24 dependabot/npm_and_yarn/ua-parser-js-0.7.24 dependabot[bot] 2021-05-07 20:45:29 +0000
  • 5a2556d317
         Merge 49ba3bef1d into 5a0eea201d dependabot[bot] 2021-05-06 20:55:10 +0000
  • 49ba3bef1d
         Bump underscore from 1.9.1 to 1.13.1 dependabot/npm_and_yarn/underscore-1.13.1 dependabot[bot] 2021-05-06 20:55:09 +0000
  • 04617a0be7
         Merge 78cc7f1a2f into 5a0eea201d dependabot[bot] 2021-03-30 18:35:58 +0000
  • 78cc7f1a2f
         Bump y18n from 4.0.0 to 4.0.1 dependabot/npm_and_yarn/y18n-4.0.1 dependabot[bot] 2021-03-30 18:35:56 +0000
  • 52675092ab
         Merge 5e83c58eaf into 5a0eea201d Przemek Nowicki 2020-10-30 13:25:35 +0100
  • 5e83c58eaf fix query selectors for google organic results Przemek Nowicki 2020-10-30 13:16:40 +0100
  • 3bdefff2df
         Bump lodash from 4.17.14 to 4.17.19 dependabot[bot] 2020-07-16 15:13:20 +0000
  • 6742e37310
         Merge 77b749e3fc into 5a0eea201d Hugo 2020-06-17 17:30:55 +0200
  • 77b749e3fc fix(google): shopping top and right working now HugoPoi 2020-06-17 17:26:29 +0200
  • c132f6e31e fix(google): parser shopping top HugoPoi 2020-06-17 16:40:22 +0200
  • ad8903b001 fix(google): parser for one shopping product right panel HugoPoi 2020-06-17 16:24:11 +0200
  • b9a9227f14 test(google): add test for google pages with shopping HugoPoi 2020-06-17 12:28:04 +0200
  • 59e612a106 test(scrape-manager): add check tests for chrome instances being closed properly HugoPoi 2020-06-16 16:10:15 +0200
  • abbae380e2
         Merge 5e5e265ce3 into 5a0eea201d Hugo 2020-05-18 18:02:13 +0200
  • 5a0eea201d Merge branch 'master' of github.com:NikolaiT/se-scraper master Nikolai Tschacher 2020-05-17 22:06:57 +0200
  • 0278b24f0d ad Nikolai Tschacher 2020-05-17 22:06:33 +0200
  • 5e5e265ce3 test(proxy): fix raise random condition error on first test HugoPoi 2020-04-01 16:29:24 +0200
  • 70753021c4 feat: add keep_html_on_error option HugoPoi 2020-04-01 15:26:22 +0200
  • 2846a50983 feat: add options for timeout HugoPoi 2020-03-31 18:32:12 +0200
  • 33fa371716
         Merge pull request #62 from aularon/patch-1 Nikolai Tschacher 2020-02-13 20:04:20 +0100
  • 6b806dedfe
         Merge pull request #61 from Monibrand/refactor/use-original-puppeteer-cluster Nikolai Tschacher 2020-02-13 20:03:39 +0100
  • 5633b10e50
         Merge pull request #60 from Monibrand/fix/unusable-proxy-file-option Nikolai Tschacher 2020-02-13 20:02:55 +0100
  • c58d4fa74d fix(proxy): throw on use_proxies_only if no proxies given HugoPoi 2020-01-17 15:55:17 +0100
  • 4f467abf1e fix(scrape-manager): keywords propagated through a clone config for not being re-affected HugoPoi 2020-01-17 15:12:00 +0100
  • 89dc5c3ebb fix(scrape-manager): conflict between proxies and user_agent option HugoPoi 2020-01-17 12:07:12 +0100
  • 4b33ef9b19 fix(duckduckgo): extract correct amount of results, handle pagination HugoPoi 2020-01-15 16:33:55 +0100
  • 28332528ea test(duckduckgo): implement tests for duckduckgo module HugoPoi 2020-01-15 16:33:30 +0100
  • b685fb4def test: working test for html_output HugoPoi 2020-01-10 09:51:54 +0100
  • 394b567db6 test: add user_agent tests, add html_output tests HugoPoi 2020-01-10 09:35:24 +0100
  • cac6b87e92 test: Bing tests working, refactor proxy for tests HugoPoi 2020-01-08 14:40:28 +0100
  • 1c1db88545 test: add config proxy options tests HugoPoi 2020-01-07 16:50:09 +0100
  • 8f6317cea7 style: add debug trace on some file HugoPoi 2020-01-07 16:47:09 +0100
  • f192e4ebb4 test: remove legacy tests HugoPoi 2020-01-07 09:53:59 +0100
  • 3ab8e46126 test: add bing module test HugoPoi 2020-01-07 09:48:46 +0100
  • 392c43390e test(google): add real integration/unit tests for google module HugoPoi 2020-01-03 19:21:34 +0100
  • 77c1bb8372
         Take screenshot before modifying HTML aularon 2020-01-03 11:12:40 +0200
  • 8f40057534 refactor(cluster): use custom concurrency for puppeteer-cluster HugoPoi 2019-12-20 19:44:59 +0100
  • 301695cd2b fix(scrape-manager): proxy_file options can be used with proxies default value HugoPoi 2019-12-20 19:35:23 +0100
  • d362e4ae2c
         Merge pull request #59 from TDenoncin/refactor/logging Nikolai Tschacher 2019-12-20 14:42:09 +0100
  • bcd181111b refactor(log): remove common.js, use winston and debug HugoPoi 2019-12-15 17:56:22 +0100
  • b4a86fcc51 refactor(proxy): remove proxy option not working replace by proxies HugoPoi 2019-12-12 15:16:37 +0100
  • 9e6a555663
         Merge pull request #52 from kujaomega/master Nikolai Tschacher 2019-12-01 22:15:39 +0100
  • ca9f5f7f50 Added post install script to build the puppeteer-cluster, and also added the updated dependencies from puppeteer-cluster David Solé 2019-11-22 00:37:29 +0100
  • 1694ee92d0 updated to puppeteeer 2.0 Nikolai Tschacher 2019-11-08 16:21:16 +0100
  • da69913272 added detected status to metadata Nikolai Tschacher 2019-10-06 15:34:18 +0200
  • 4a3a0e6fd4 better pluggable api Nikolai Tschacher 2019-10-05 19:39:33 +0200
  • 4953d9da7a chaned version Nikolai Tschacher 2019-09-23 23:39:06 +0200
  • 5e47c27c70 too late to find a proper commit description Nikolai Tschacher 2019-09-23 23:38:38 +0200
  • 95a5ee56d8 remove cheerio from parsing Nikolai Tschacher 2019-09-23 21:57:13 +0200
  • 52a2ec7b33 changed README Nikolai Tschacher 2019-09-23 16:50:57 +0200
  • 07f3dceba1 fixed google SERP title, better docker support Nikolai Tschacher 2019-09-23 16:46:22 +0200
  • b25f7a4285 added test to my working tree Nikolai Tschacher 2019-09-13 18:28:19 +0200
  • 4b581bd03f removed static tests because they are too larege Nikolai Tschacher 2019-09-13 18:21:17 +0200
  • 21378dab02 removed some search engines, added tests for existing, added yandex search engines Nikolai Tschacher 2019-09-13 16:15:33 +0200
  • 77d6c4f04a removed some stuff Nikolai Tschacher 2019-09-12 10:43:57 +0200
  • b513bb0f5b Merge branch 'master' of github.com:NikolaiT/se-scraper Nikolai Tschacher 2019-09-04 12:28:05 +0200
  • 855a874f9e some minor changes Nikolai Tschacher 2019-09-04 12:27:53 +0200
  • dde1711d9d
         Merge pull request #45 from slotix/master Nikolai Tschacher 2019-08-29 20:41:42 +0200
  • 7ba7ee9226 add process supervisor for starting server.js slotix 2019-08-19 14:01:37 +0200
  • e661241f6f added some parsing to google Nikolai Tschacher 2019-08-16 20:10:40 +0200
  • 98414259fe docker support added Nikolai Tschacher 2019-08-13 17:35:06 +0200
  • 19a172c654 better tests Nikolai Tschacher 2019-08-13 15:28:30 +0200
  • 0f7e89c272 added little bug in cleaning Nikolai Tschacher 2019-08-12 17:16:37 +0200
  • ca941cee45 added static bing test, added html cleaning when exporting html Nikolai Tschacher 2019-08-12 16:05:17 +0200
  • 4c77aeba76
         Merge pull request #42 from TDenoncin/error-management Nikolai Tschacher 2019-08-12 00:04:40 +0200
  • 0427d9f915
         Merge branch 'master' into error-management Nikolai Tschacher 2019-08-12 00:04:27 +0200
  • 87fcdd35d5 readme in static tests Nikolai Tschacher 2019-08-12 00:01:02 +0200
  • 4ca50ab2b9 added new static test case that runs much faster and tests a lot of behavior Nikolai Tschacher 2019-08-11 23:58:10 +0200
  • 8e629f6266
         Merge pull request #41 from victor9000/master Nikolai Tschacher 2019-08-08 21:57:14 +0200
  • a369bd07f9 Add "use strict" to ensure quality code control HugoPoi 2019-08-06 12:18:51 +0200
  • dde2b14fc0 Remove uneeded try catch block in Google Search module HugoPoi 2019-08-06 11:50:08 +0200
  • 0db6e068da Remove uneeded try catch block Add proper error for ip matching test HugoPoi 2019-08-06 11:46:53 +0200
  • 50bda275a6 Clean integration tests for mocha HugoPoi 2019-08-02 17:41:29 +0200
  • a61fade2c9 Fix broken Google News selectors, fixes #40 Victor 2019-08-04 14:25:13 -0700
  • 1ffe33dc72 update dependencies slotix 2019-07-26 22:07:22 +0200