aularon
|
77c1bb8372
|
Take screenshot before modifying HTML
Otherwise the screenshot will be very messed up
|
2020-01-03 11:12:40 +02:00 |
|
Nikolai Tschacher
|
d362e4ae2c
|
Merge pull request #59 from TDenoncin/refactor/logging
Refactor logging
|
2019-12-20 14:42:09 +01:00 |
|
HugoPoi
|
bcd181111b
|
refactor(log): remove common.js, use winston and debug
|
2019-12-15 17:56:22 +01:00 |
|
HugoPoi
|
b4a86fcc51
|
refactor(proxy): remove proxy option not working replace by proxies
|
2019-12-13 18:02:22 +01:00 |
|
Nikolai Tschacher
|
9e6a555663
|
Merge pull request #52 from kujaomega/master
Added post install script to build the puppeteer-cluster, and also ad…
|
2019-12-01 22:15:39 +01:00 |
|
David Solé
|
ca9f5f7f50
|
Added post install script to build the puppeteer-cluster, and also added the updated dependencies from puppeteer-cluster
|
2019-11-22 00:37:29 +01:00 |
|
Nikolai Tschacher
|
1694ee92d0
|
updated to puppeteeer 2.0
|
2019-11-08 16:21:16 +01:00 |
|
Nikolai Tschacher
|
da69913272
|
added detected status to metadata
|
2019-10-06 15:34:18 +02:00 |
|
Nikolai Tschacher
|
4a3a0e6fd4
|
better pluggable api
|
2019-10-05 19:39:33 +02:00 |
|
Nikolai Tschacher
|
4953d9da7a
|
chaned version
|
2019-09-23 23:39:06 +02:00 |
|
Nikolai Tschacher
|
5e47c27c70
|
too late to find a proper commit description
|
2019-09-23 23:38:38 +02:00 |
|
Nikolai Tschacher
|
95a5ee56d8
|
remove cheerio from parsing
|
2019-09-23 21:57:13 +02:00 |
|
Nikolai Tschacher
|
52a2ec7b33
|
changed README
|
2019-09-23 16:50:57 +02:00 |
|
Nikolai Tschacher
|
07f3dceba1
|
fixed google SERP title, better docker support
|
2019-09-23 16:46:22 +02:00 |
|
Nikolai Tschacher
|
b25f7a4285
|
added test to my working tree
|
2019-09-13 18:28:19 +02:00 |
|
Nikolai Tschacher
|
4b581bd03f
|
removed static tests because they are too larege
|
2019-09-13 18:21:17 +02:00 |
|
Nikolai Tschacher
|
21378dab02
|
removed some search engines, added tests for existing, added yandex search engines
|
2019-09-13 16:15:33 +02:00 |
|
Nikolai Tschacher
|
77d6c4f04a
|
removed some stuff
|
2019-09-12 10:43:57 +02:00 |
|
Nikolai Tschacher
|
b513bb0f5b
|
Merge branch 'master' of github.com:NikolaiT/se-scraper
server in dockerfile was changed
|
2019-09-04 12:28:05 +02:00 |
|
Nikolai Tschacher
|
855a874f9e
|
some minor changes
|
2019-09-04 12:27:53 +02:00 |
|
Nikolai Tschacher
|
dde1711d9d
|
Merge pull request #45 from slotix/master
add process supervisor for starting server.js
|
2019-08-29 20:41:42 +02:00 |
|
slotix
|
7ba7ee9226
|
add process supervisor for starting server.js
|
2019-08-19 14:01:37 +02:00 |
|
Nikolai Tschacher
|
e661241f6f
|
added some parsing to google
|
2019-08-16 20:10:40 +02:00 |
|
Nikolai Tschacher
|
98414259fe
|
docker support added
|
2019-08-13 17:35:06 +02:00 |
|
Nikolai Tschacher
|
19a172c654
|
better tests
|
2019-08-13 15:28:30 +02:00 |
|
Nikolai Tschacher
|
0f7e89c272
|
added little bug in cleaning
|
2019-08-12 17:16:37 +02:00 |
|
Nikolai Tschacher
|
ca941cee45
|
added static bing test, added html cleaning when exporting html
|
2019-08-12 16:05:17 +02:00 |
|
Nikolai Tschacher
|
4c77aeba76
|
Merge pull request #42 from TDenoncin/error-management
Clean integration tests with mocha
|
2019-08-12 00:04:40 +02:00 |
|
Nikolai Tschacher
|
0427d9f915
|
Merge branch 'master' into error-management
|
2019-08-12 00:04:27 +02:00 |
|
Nikolai Tschacher
|
87fcdd35d5
|
readme in static tests
|
2019-08-12 00:01:02 +02:00 |
|
Nikolai Tschacher
|
4ca50ab2b9
|
added new static test case that runs much faster and tests a lot of behavior
|
2019-08-11 23:58:10 +02:00 |
|
Nikolai Tschacher
|
8e629f6266
|
Merge pull request #41 from victor9000/master
Fix broken Google News selectors, fixes #40
|
2019-08-08 21:57:14 +02:00 |
|
HugoPoi
|
a369bd07f9
|
Add "use strict" to ensure quality code control
|
2019-08-06 12:18:51 +02:00 |
|
HugoPoi
|
dde2b14fc0
|
Remove uneeded try catch block in Google Search module
|
2019-08-06 11:50:08 +02:00 |
|
HugoPoi
|
0db6e068da
|
Remove uneeded try catch block
Add proper error for ip matching test
|
2019-08-06 11:46:53 +02:00 |
|
HugoPoi
|
50bda275a6
|
Clean integration tests for mocha
|
2019-08-05 17:01:48 +02:00 |
|
Victor
|
a61fade2c9
|
Fix broken Google News selectors, fixes #40
|
2019-08-04 14:43:02 -07:00 |
|
Nikolai Tschacher
|
78fe12390b
|
better user agents now, added option to include screenshots as base64 in results
|
2019-07-18 20:19:15 +02:00 |
|
Nikolai Tschacher
|
fcbe66b56b
|
using random user agents now from https://github.com/intoli/user-agents
|
2019-07-18 19:34:09 +02:00 |
|
Nikolai Tschacher
|
59154694f2
|
fixed issue https://github.com/NikolaiT/se-scraper/issues/37
|
2019-07-18 19:14:33 +02:00 |
|
Nikolai Tschacher
|
60a9d52924
|
add fucking google product information
|
2019-07-11 19:23:40 +02:00 |
|
Nikolai Tschacher
|
1fc7f0d1c8
|
fixed a badboy
|
2019-07-11 16:54:32 +02:00 |
|
Nikolai Tschacher
|
baaff5824e
|
...
|
2019-07-11 16:43:41 +02:00 |
|
Nikolai Tschacher
|
dab25f9068
|
added google shopping results
|
2019-07-11 16:42:01 +02:00 |
|
Nikolai Tschacher
|
a413cb54ef
|
parsing ads works for duckduckgo, google, bing. tested.
|
2019-07-07 19:38:28 +02:00 |
|
Nikolai Tschacher
|
bbebe3ce60
|
parsing ads is supported now for google, bing and duckduckgo
|
2019-07-06 21:42:13 +02:00 |
|
Nikolai Tschacher
|
09c1255400
|
removed some superflous stuff
|
2019-07-02 18:04:01 +02:00 |
|
Nikolai Tschacher
|
5e8ff1cb34
|
Merge branch 'master' of https://github.com/NikolaiT/se-scraper
|
2019-06-29 17:01:25 +02:00 |
|
Nikolai Tschacher
|
c1a036e8da
|
removed some stuff
|
2019-06-29 17:00:50 +02:00 |
|
Nikolai Tschacher
|
d1e9b21269
|
added google maps scraper
|
2019-06-29 17:00:19 +02:00 |
|