Nikolai Tschacher
|
33fa371716
|
Merge pull request #62 from aularon/patch-1
Take screenshot before modifying HTML
|
2020-02-13 20:04:20 +01:00 |
|
HugoPoi
|
c58d4fa74d
|
fix(proxy): throw on use_proxies_only if no proxies given
|
2020-01-17 15:55:17 +01:00 |
|
HugoPoi
|
4f467abf1e
|
fix(scrape-manager): keywords propagated through a clone config for not being re-affected
|
2020-01-17 15:12:00 +01:00 |
|
HugoPoi
|
89dc5c3ebb
|
fix(scrape-manager): conflict between proxies and user_agent option
|
2020-01-17 12:07:12 +01:00 |
|
HugoPoi
|
4b33ef9b19
|
fix(duckduckgo): extract correct amount of results, handle pagination
|
2020-01-15 16:35:16 +01:00 |
|
HugoPoi
|
8f6317cea7
|
style: add debug trace on some file
|
2020-01-07 16:47:09 +01:00 |
|
HugoPoi
|
3ab8e46126
|
test: add bing module test
|
2020-01-07 09:48:46 +01:00 |
|
aularon
|
77c1bb8372
|
Take screenshot before modifying HTML
Otherwise the screenshot will be very messed up
|
2020-01-03 11:12:40 +02:00 |
|
HugoPoi
|
8f40057534
|
refactor(cluster): use custom concurrency for puppeteer-cluster
|
2019-12-20 19:44:59 +01:00 |
|
HugoPoi
|
301695cd2b
|
fix(scrape-manager): proxy_file options can be used with proxies default value
|
2019-12-20 19:35:23 +01:00 |
|
HugoPoi
|
bcd181111b
|
refactor(log): remove common.js, use winston and debug
|
2019-12-15 17:56:22 +01:00 |
|
HugoPoi
|
b4a86fcc51
|
refactor(proxy): remove proxy option not working replace by proxies
|
2019-12-13 18:02:22 +01:00 |
|
David Solé
|
ca9f5f7f50
|
Added post install script to build the puppeteer-cluster, and also added the updated dependencies from puppeteer-cluster
|
2019-11-22 00:37:29 +01:00 |
|
Nikolai Tschacher
|
1694ee92d0
|
updated to puppeteeer 2.0
|
2019-11-08 16:21:16 +01:00 |
|
Nikolai Tschacher
|
da69913272
|
added detected status to metadata
|
2019-10-06 15:34:18 +02:00 |
|
Nikolai Tschacher
|
4a3a0e6fd4
|
better pluggable api
|
2019-10-05 19:39:33 +02:00 |
|
Nikolai Tschacher
|
5e47c27c70
|
too late to find a proper commit description
|
2019-09-23 23:38:38 +02:00 |
|
Nikolai Tschacher
|
95a5ee56d8
|
remove cheerio from parsing
|
2019-09-23 21:57:13 +02:00 |
|
Nikolai Tschacher
|
07f3dceba1
|
fixed google SERP title, better docker support
|
2019-09-23 16:46:22 +02:00 |
|
Nikolai Tschacher
|
21378dab02
|
removed some search engines, added tests for existing, added yandex search engines
|
2019-09-13 16:15:33 +02:00 |
|
Nikolai Tschacher
|
77d6c4f04a
|
removed some stuff
|
2019-09-12 10:43:57 +02:00 |
|
Nikolai Tschacher
|
855a874f9e
|
some minor changes
|
2019-09-04 12:27:53 +02:00 |
|
Nikolai Tschacher
|
e661241f6f
|
added some parsing to google
|
2019-08-16 20:10:40 +02:00 |
|
Nikolai Tschacher
|
98414259fe
|
docker support added
|
2019-08-13 17:35:06 +02:00 |
|
Nikolai Tschacher
|
19a172c654
|
better tests
|
2019-08-13 15:28:30 +02:00 |
|
Nikolai Tschacher
|
0f7e89c272
|
added little bug in cleaning
|
2019-08-12 17:16:37 +02:00 |
|
Nikolai Tschacher
|
ca941cee45
|
added static bing test, added html cleaning when exporting html
|
2019-08-12 16:05:17 +02:00 |
|
Nikolai Tschacher
|
0427d9f915
|
Merge branch 'master' into error-management
|
2019-08-12 00:04:27 +02:00 |
|
Nikolai Tschacher
|
4ca50ab2b9
|
added new static test case that runs much faster and tests a lot of behavior
|
2019-08-11 23:58:10 +02:00 |
|
HugoPoi
|
a369bd07f9
|
Add "use strict" to ensure quality code control
|
2019-08-06 12:18:51 +02:00 |
|
HugoPoi
|
dde2b14fc0
|
Remove uneeded try catch block in Google Search module
|
2019-08-06 11:50:08 +02:00 |
|
HugoPoi
|
0db6e068da
|
Remove uneeded try catch block
Add proper error for ip matching test
|
2019-08-06 11:46:53 +02:00 |
|
Victor
|
a61fade2c9
|
Fix broken Google News selectors, fixes #40
|
2019-08-04 14:43:02 -07:00 |
|
Nikolai Tschacher
|
78fe12390b
|
better user agents now, added option to include screenshots as base64 in results
|
2019-07-18 20:19:15 +02:00 |
|
Nikolai Tschacher
|
fcbe66b56b
|
using random user agents now from https://github.com/intoli/user-agents
|
2019-07-18 19:34:09 +02:00 |
|
Nikolai Tschacher
|
59154694f2
|
fixed issue https://github.com/NikolaiT/se-scraper/issues/37
|
2019-07-18 19:14:33 +02:00 |
|
Nikolai Tschacher
|
60a9d52924
|
add fucking google product information
|
2019-07-11 19:23:40 +02:00 |
|
Nikolai Tschacher
|
baaff5824e
|
...
|
2019-07-11 16:43:41 +02:00 |
|
Nikolai Tschacher
|
dab25f9068
|
added google shopping results
|
2019-07-11 16:42:01 +02:00 |
|
Nikolai Tschacher
|
a413cb54ef
|
parsing ads works for duckduckgo, google, bing. tested.
|
2019-07-07 19:38:28 +02:00 |
|
Nikolai Tschacher
|
bbebe3ce60
|
parsing ads is supported now for google, bing and duckduckgo
|
2019-07-06 21:42:13 +02:00 |
|
Nikolai Tschacher
|
09c1255400
|
removed some superflous stuff
|
2019-07-02 18:04:01 +02:00 |
|
Nikolai Tschacher
|
5e8ff1cb34
|
Merge branch 'master' of https://github.com/NikolaiT/se-scraper
|
2019-06-29 17:01:25 +02:00 |
|
Nikolai Tschacher
|
d1e9b21269
|
added google maps scraper
|
2019-06-29 17:00:19 +02:00 |
|
HugoPoi
|
d9ac9f4162
|
Add test for html_output, refactor the results return
|
2019-06-26 12:03:42 +02:00 |
|
Thomas
|
a0e63aa4b0
|
Use bing_setting.bing_domain if defined for startUrl
|
2019-06-25 17:16:17 +02:00 |
|
Thomas
|
a3ebe357a4
|
Add html_output fonctionality
Pagination support for html output
Change return value to keep it compliant to the current version of se-scrapper
|
2019-06-25 17:02:34 +02:00 |
|
Nikolai Tschacher
|
0d7f6dcd11
|
worked on issue #31
|
2019-06-18 22:23:52 +02:00 |
|
Nikolai Tschacher
|
80d23a9d57
|
users may pass their own user agents, different browsers have random user agents and not the same now
|
2019-06-17 21:25:45 +02:00 |
|
Nikolai Tschacher
|
ebe9ba8ea9
|
added option to throw on detection
|
2019-06-17 15:02:44 +02:00 |
|