extern/se-scraper
1
0
Fork 1
mirror of https://github.com/NikolaiT/se-scraper.git synced 2024-11-07 08:13:59 +01:00
Code Issues Packages Projects Releases Wiki Activity
130 Commits 14 Branches 0 Tags 12 MiB
master
Commit Graph

22 Commits

Author SHA1 Message Date
HugoPoi
394b567db6 test: add user_agent tests, add html_output tests 2020-01-10 09:35:24 +01:00
HugoPoi
cac6b87e92 test: Bing tests working, refactor proxy for tests 2020-01-08 14:40:28 +01:00
HugoPoi
392c43390e test(google): add real integration/unit tests for google module 2020-01-03 19:21:34 +01:00
HugoPoi
8f40057534 refactor(cluster): use custom concurrency for puppeteer-cluster 2019-12-20 19:44:59 +01:00
HugoPoi
bcd181111b refactor(log): remove common.js, use winston and debug 2019-12-15 17:56:22 +01:00
HugoPoi
b4a86fcc51 refactor(proxy): remove proxy option not working replace by proxies 2019-12-13 18:02:22 +01:00
Nikolai Tschacher
1694ee92d0 updated to puppeteeer 2.0 2019-11-08 16:21:16 +01:00
Nikolai Tschacher
21378dab02 removed some search engines, added tests for existing, added yandex search engines 2019-09-13 16:15:33 +02:00
Nikolai Tschacher
98414259fe docker support added 2019-08-13 17:35:06 +02:00
Nikolai Tschacher
78fe12390b better user agents now, added option to include screenshots as base64 in results 2019-07-18 20:19:15 +02:00
Nikolai Tschacher
fcbe66b56b using random user agents now from https://github.com/intoli/user-agents 2019-07-18 19:34:09 +02:00
Nikolai Tschacher
1fc7f0d1c8 fixed a badboy 2019-07-11 16:54:32 +02:00
Nikolai Tschacher
09c1255400 removed some superflous stuff 2019-07-02 18:04:01 +02:00
HugoPoi
d9ac9f4162 Add test for html_output, refactor the results return 2019-06-26 12:03:42 +02:00
Nikolai Tschacher
7e06944fa1 updated README 2019-06-11 18:27:34 +02:00
Nikolai Tschacher
7b52b4e62f added suport for custom query string parameters 2019-03-06 00:08:25 +01:00
Nikolai Tschacher
089e410ec6 support for multible browsers and proxies 2019-02-27 20:58:13 +01:00
Nikolai Tschacher
7572ebd314 added chrome detection evasion techniques 2019-02-07 16:09:38 +01:00
Nikolai Tschacher
9e62f23451 resolved some issues. proxy possible now. scraping for more than one page possible now 2019-01-29 22:48:08 +01:00
Nikolai Tschacher
c5e3e84e1d minor changes 2019-01-27 22:11:41 +01:00
Nikolai Tschacher
bab902e80a supporting yahoo ticker search for news 2019-01-24 15:50:03 +01:00
Nikolai Tschacher
9af1630e19 initial 2018-12-24 14:25:02 +01:00