Nikolai Tschacher
|
5a0eea201d
|
Merge branch 'master' of github.com:NikolaiT/se-scraper
branchy
|
2020-05-17 22:06:57 +02:00 |
|
Nikolai Tschacher
|
0278b24f0d
|
ad
|
2020-05-17 22:06:33 +02:00 |
|
Nikolai Tschacher
|
33fa371716
|
Merge pull request #62 from aularon/patch-1
Take screenshot before modifying HTML
|
2020-02-13 20:04:20 +01:00 |
|
Nikolai Tschacher
|
6b806dedfe
|
Merge pull request #61 from Monibrand/refactor/use-original-puppeteer-cluster
Refactor/use original puppeteer cluster
|
2020-02-13 20:03:39 +01:00 |
|
Nikolai Tschacher
|
5633b10e50
|
Merge pull request #60 from Monibrand/fix/unusable-proxy-file-option
fix(scrape-manager): proxy_file options can't be used with proxies default value
|
2020-02-13 20:02:55 +01:00 |
|
HugoPoi
|
c58d4fa74d
|
fix(proxy): throw on use_proxies_only if no proxies given
|
2020-01-17 15:55:17 +01:00 |
|
HugoPoi
|
4f467abf1e
|
fix(scrape-manager): keywords propagated through a clone config for not being re-affected
|
2020-01-17 15:12:00 +01:00 |
|
HugoPoi
|
89dc5c3ebb
|
fix(scrape-manager): conflict between proxies and user_agent option
|
2020-01-17 12:07:12 +01:00 |
|
HugoPoi
|
4b33ef9b19
|
fix(duckduckgo): extract correct amount of results, handle pagination
|
2020-01-15 16:35:16 +01:00 |
|
HugoPoi
|
28332528ea
|
test(duckduckgo): implement tests for duckduckgo module
|
2020-01-15 16:33:30 +01:00 |
|
HugoPoi
|
b685fb4def
|
test: working test for html_output
|
2020-01-10 09:51:54 +01:00 |
|
HugoPoi
|
394b567db6
|
test: add user_agent tests, add html_output tests
|
2020-01-10 09:35:24 +01:00 |
|
HugoPoi
|
cac6b87e92
|
test: Bing tests working, refactor proxy for tests
|
2020-01-08 14:40:28 +01:00 |
|
HugoPoi
|
1c1db88545
|
test: add config proxy options tests
|
2020-01-07 16:50:09 +01:00 |
|
HugoPoi
|
8f6317cea7
|
style: add debug trace on some file
|
2020-01-07 16:47:09 +01:00 |
|
HugoPoi
|
f192e4ebb4
|
test: remove legacy tests
|
2020-01-07 16:43:17 +01:00 |
|
HugoPoi
|
3ab8e46126
|
test: add bing module test
|
2020-01-07 09:48:46 +01:00 |
|
HugoPoi
|
392c43390e
|
test(google): add real integration/unit tests for google module
|
2020-01-03 19:21:34 +01:00 |
|
aularon
|
77c1bb8372
|
Take screenshot before modifying HTML
Otherwise the screenshot will be very messed up
|
2020-01-03 11:12:40 +02:00 |
|
HugoPoi
|
8f40057534
|
refactor(cluster): use custom concurrency for puppeteer-cluster
|
2019-12-20 19:44:59 +01:00 |
|
HugoPoi
|
301695cd2b
|
fix(scrape-manager): proxy_file options can be used with proxies default value
|
2019-12-20 19:35:23 +01:00 |
|
Nikolai Tschacher
|
d362e4ae2c
|
Merge pull request #59 from TDenoncin/refactor/logging
Refactor logging
|
2019-12-20 14:42:09 +01:00 |
|
HugoPoi
|
bcd181111b
|
refactor(log): remove common.js, use winston and debug
|
2019-12-15 17:56:22 +01:00 |
|
HugoPoi
|
b4a86fcc51
|
refactor(proxy): remove proxy option not working replace by proxies
|
2019-12-13 18:02:22 +01:00 |
|
Nikolai Tschacher
|
9e6a555663
|
Merge pull request #52 from kujaomega/master
Added post install script to build the puppeteer-cluster, and also ad…
|
2019-12-01 22:15:39 +01:00 |
|
David Solé
|
ca9f5f7f50
|
Added post install script to build the puppeteer-cluster, and also added the updated dependencies from puppeteer-cluster
|
2019-11-22 00:37:29 +01:00 |
|
Nikolai Tschacher
|
1694ee92d0
|
updated to puppeteeer 2.0
|
2019-11-08 16:21:16 +01:00 |
|
Nikolai Tschacher
|
da69913272
|
added detected status to metadata
|
2019-10-06 15:34:18 +02:00 |
|
Nikolai Tschacher
|
4a3a0e6fd4
|
better pluggable api
|
2019-10-05 19:39:33 +02:00 |
|
Nikolai Tschacher
|
4953d9da7a
|
chaned version
|
2019-09-23 23:39:06 +02:00 |
|
Nikolai Tschacher
|
5e47c27c70
|
too late to find a proper commit description
|
2019-09-23 23:38:38 +02:00 |
|
Nikolai Tschacher
|
95a5ee56d8
|
remove cheerio from parsing
|
2019-09-23 21:57:13 +02:00 |
|
Nikolai Tschacher
|
52a2ec7b33
|
changed README
|
2019-09-23 16:50:57 +02:00 |
|
Nikolai Tschacher
|
07f3dceba1
|
fixed google SERP title, better docker support
|
2019-09-23 16:46:22 +02:00 |
|
Nikolai Tschacher
|
b25f7a4285
|
added test to my working tree
|
2019-09-13 18:28:19 +02:00 |
|
Nikolai Tschacher
|
4b581bd03f
|
removed static tests because they are too larege
|
2019-09-13 18:21:17 +02:00 |
|
Nikolai Tschacher
|
21378dab02
|
removed some search engines, added tests for existing, added yandex search engines
|
2019-09-13 16:15:33 +02:00 |
|
Nikolai Tschacher
|
77d6c4f04a
|
removed some stuff
|
2019-09-12 10:43:57 +02:00 |
|
Nikolai Tschacher
|
b513bb0f5b
|
Merge branch 'master' of github.com:NikolaiT/se-scraper
server in dockerfile was changed
|
2019-09-04 12:28:05 +02:00 |
|
Nikolai Tschacher
|
855a874f9e
|
some minor changes
|
2019-09-04 12:27:53 +02:00 |
|
Nikolai Tschacher
|
dde1711d9d
|
Merge pull request #45 from slotix/master
add process supervisor for starting server.js
|
2019-08-29 20:41:42 +02:00 |
|
slotix
|
7ba7ee9226
|
add process supervisor for starting server.js
|
2019-08-19 14:01:37 +02:00 |
|
Nikolai Tschacher
|
e661241f6f
|
added some parsing to google
|
2019-08-16 20:10:40 +02:00 |
|
Nikolai Tschacher
|
98414259fe
|
docker support added
|
2019-08-13 17:35:06 +02:00 |
|
Nikolai Tschacher
|
19a172c654
|
better tests
|
2019-08-13 15:28:30 +02:00 |
|
Nikolai Tschacher
|
0f7e89c272
|
added little bug in cleaning
|
2019-08-12 17:16:37 +02:00 |
|
Nikolai Tschacher
|
ca941cee45
|
added static bing test, added html cleaning when exporting html
|
2019-08-12 16:05:17 +02:00 |
|
Nikolai Tschacher
|
4c77aeba76
|
Merge pull request #42 from TDenoncin/error-management
Clean integration tests with mocha
|
2019-08-12 00:04:40 +02:00 |
|
Nikolai Tschacher
|
0427d9f915
|
Merge branch 'master' into error-management
|
2019-08-12 00:04:27 +02:00 |
|
Nikolai Tschacher
|
87fcdd35d5
|
readme in static tests
|
2019-08-12 00:01:02 +02:00 |
|