extern/se-scraper
1
0
Fork 1
mirror of https://github.com/NikolaiT/se-scraper.git synced 2024-11-24 08:33:12 +01:00
Code Issues Packages Projects Releases Wiki Activity
131 Commits 14 Branches 0 Tags 12 MiB
dependabot/npm_and_yarn/underscore-1.13.1
Commit Graph

131 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
dependabot[bot]
49ba3bef1d
Bump underscore from 1.9.1 to 1.13.1 
Bumps [underscore](https://github.com/jashkenas/underscore) from 1.9.1 to 1.13.1.
- [Release notes](https://github.com/jashkenas/underscore/releases)
- [Commits](https://github.com/jashkenas/underscore/compare/1.9.1...1.13.1)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-05-06 20:55:09 +00:00
Nikolai Tschacher
5a0eea201d Merge branch 'master' of github.com:NikolaiT/se-scraper 
branchy
 2020-05-17 22:06:57 +02:00
Nikolai Tschacher
0278b24f0d ad 2020-05-17 22:06:33 +02:00
Nikolai Tschacher
33fa371716
Merge pull request #62 from aularon/patch-1 
Take screenshot before modifying HTML
 2020-02-13 20:04:20 +01:00
Nikolai Tschacher
6b806dedfe
Merge pull request #61 from Monibrand/refactor/use-original-puppeteer-cluster 
Refactor/use original puppeteer cluster
 2020-02-13 20:03:39 +01:00
Nikolai Tschacher
5633b10e50
Merge pull request #60 from Monibrand/fix/unusable-proxy-file-option 
fix(scrape-manager): proxy_file options can't be used with proxies default value
 2020-02-13 20:02:55 +01:00
HugoPoi
c58d4fa74d fix(proxy): throw on use_proxies_only if no proxies given 2020-01-17 15:55:17 +01:00
HugoPoi
4f467abf1e fix(scrape-manager): keywords propagated through a clone config for not being re-affected 2020-01-17 15:12:00 +01:00
HugoPoi
89dc5c3ebb fix(scrape-manager): conflict between proxies and user_agent option 2020-01-17 12:07:12 +01:00
HugoPoi
4b33ef9b19 fix(duckduckgo): extract correct amount of results, handle pagination 2020-01-15 16:35:16 +01:00
HugoPoi
28332528ea test(duckduckgo): implement tests for duckduckgo module 2020-01-15 16:33:30 +01:00
HugoPoi
b685fb4def test: working test for html_output 2020-01-10 09:51:54 +01:00
HugoPoi
394b567db6 test: add user_agent tests, add html_output tests 2020-01-10 09:35:24 +01:00
HugoPoi
cac6b87e92 test: Bing tests working, refactor proxy for tests 2020-01-08 14:40:28 +01:00
HugoPoi
1c1db88545 test: add config proxy options tests 2020-01-07 16:50:09 +01:00
HugoPoi
8f6317cea7 style: add debug trace on some file 2020-01-07 16:47:09 +01:00
HugoPoi
f192e4ebb4 test: remove legacy tests 2020-01-07 16:43:17 +01:00
HugoPoi
3ab8e46126 test: add bing module test 2020-01-07 09:48:46 +01:00
HugoPoi
392c43390e test(google): add real integration/unit tests for google module 2020-01-03 19:21:34 +01:00
aularon
77c1bb8372
Take screenshot before modifying HTML 
Otherwise the screenshot will be very messed up
 2020-01-03 11:12:40 +02:00
HugoPoi
8f40057534 refactor(cluster): use custom concurrency for puppeteer-cluster 2019-12-20 19:44:59 +01:00
HugoPoi
301695cd2b fix(scrape-manager): proxy_file options can be used with proxies default value 2019-12-20 19:35:23 +01:00
Nikolai Tschacher
d362e4ae2c
Merge pull request #59 from TDenoncin/refactor/logging 
Refactor logging
 2019-12-20 14:42:09 +01:00
HugoPoi
bcd181111b refactor(log): remove common.js, use winston and debug 2019-12-15 17:56:22 +01:00
HugoPoi
b4a86fcc51 refactor(proxy): remove proxy option not working replace by proxies 2019-12-13 18:02:22 +01:00
Nikolai Tschacher
9e6a555663
Merge pull request #52 from kujaomega/master 
Added post install script to build the puppeteer-cluster, and also ad…
 2019-12-01 22:15:39 +01:00
David Solé
ca9f5f7f50 Added post install script to build the puppeteer-cluster, and also added the updated dependencies from puppeteer-cluster 2019-11-22 00:37:29 +01:00
Nikolai Tschacher
1694ee92d0 updated to puppeteeer 2.0 2019-11-08 16:21:16 +01:00
Nikolai Tschacher
da69913272 added detected status to metadata 2019-10-06 15:34:18 +02:00
Nikolai Tschacher
4a3a0e6fd4 better pluggable api 2019-10-05 19:39:33 +02:00
Nikolai Tschacher
4953d9da7a chaned version 2019-09-23 23:39:06 +02:00
Nikolai Tschacher
5e47c27c70 too late to find a proper commit description 2019-09-23 23:38:38 +02:00
Nikolai Tschacher
95a5ee56d8 remove cheerio from parsing 2019-09-23 21:57:13 +02:00
Nikolai Tschacher
52a2ec7b33 changed README 2019-09-23 16:50:57 +02:00
Nikolai Tschacher
07f3dceba1 fixed google SERP title, better docker support 2019-09-23 16:46:22 +02:00
Nikolai Tschacher
b25f7a4285 added test to my working tree 2019-09-13 18:28:19 +02:00
Nikolai Tschacher
4b581bd03f removed static tests because they are too larege 2019-09-13 18:21:17 +02:00
Nikolai Tschacher
21378dab02 removed some search engines, added tests for existing, added yandex search engines 2019-09-13 16:15:33 +02:00
Nikolai Tschacher
77d6c4f04a removed some stuff 2019-09-12 10:43:57 +02:00
Nikolai Tschacher
b513bb0f5b Merge branch 'master' of github.com:NikolaiT/se-scraper 
server in dockerfile was changed
 2019-09-04 12:28:05 +02:00
Nikolai Tschacher
855a874f9e some minor changes 2019-09-04 12:27:53 +02:00
Nikolai Tschacher
dde1711d9d
Merge pull request #45 from slotix/master 
add process supervisor for starting server.js
 2019-08-29 20:41:42 +02:00
slotix
7ba7ee9226 add process supervisor for starting server.js 2019-08-19 14:01:37 +02:00
Nikolai Tschacher
e661241f6f added some parsing to google 2019-08-16 20:10:40 +02:00
Nikolai Tschacher
98414259fe docker support added 2019-08-13 17:35:06 +02:00
Nikolai Tschacher
19a172c654 better tests 2019-08-13 15:28:30 +02:00
Nikolai Tschacher
0f7e89c272 added little bug in cleaning 2019-08-12 17:16:37 +02:00
Nikolai Tschacher
ca941cee45 added static bing test, added html cleaning when exporting html 2019-08-12 16:05:17 +02:00
Nikolai Tschacher
4c77aeba76
Merge pull request #42 from TDenoncin/error-management 
Clean integration tests with mocha
 2019-08-12 00:04:40 +02:00
Nikolai Tschacher
0427d9f915
Merge branch 'master' into error-management 2019-08-12 00:04:27 +02:00