extern/se-scraper
1
0
Fork 1
mirror of https://github.com/NikolaiT/se-scraper.git synced 2024-12-01 03:53:30 +01:00
Code Issues Packages Projects Releases Wiki Activity
30 Commits 14 Branches 0 Tags 12 MiB
abf4458e46
Commit Graph

30 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Nikolai Tschacher
abf4458e46 fixed quotes in user agent. this lead to cloudflare detecting the scraper. very bad. 2019-03-01 16:02:30 +01:00
Nikolai Tschacher
79d32a315a fixed some errors and way better README 2019-02-28 15:34:25 +01:00
Nikolai Tschacher
089e410ec6 support for multible browsers and proxies 2019-02-27 20:58:13 +01:00
Nikolai Tschacher
393b9c0450
Merge pull request #8 from NikolaiT/add-license-1 
Create LICENSE
 2019-02-08 00:58:27 +01:00
Nikolai Tschacher
fb3f2836e4
Create LICENSE 2019-02-08 00:58:15 +01:00
Nikolai Tschacher
53c9ebf467
Merge pull request #7 from NikolaiT/add-code-of-conduct-1 
Create CODE_OF_CONDUCT.md
 2019-02-08 00:54:28 +01:00
Nikolai Tschacher
9521c54c77
Create CODE_OF_CONDUCT.md 2019-02-08 00:54:10 +01:00
Nikolai Tschacher
77c332d7c8 updated readme 2019-02-07 16:26:11 +01:00
Nikolai Tschacher
7b5048b8ee num_keywords are counted now. added to pluggable 2019-02-07 16:21:56 +01:00
Nikolai Tschacher
7572ebd314 added chrome detection evasion techniques 2019-02-07 16:09:38 +01:00
Nikolai Tschacher
d5b147296e ticker search OOP now and added tests 2019-01-31 22:13:22 +01:00
Nikolai Tschacher
d35a602994 added clean test cases for bing and duckduckgo 2019-01-31 15:36:27 +01:00
Nikolai Tschacher
7441c57a43 removed generic tests. too complicated 2019-01-31 14:58:07 +01:00
Nikolai Tschacher
c60d0f3528 clean test case for google is passing 2019-01-31 14:57:34 +01:00
Nikolai Tschacher
987e3d7342 tested and works 2019-01-30 23:53:09 +01:00
Nikolai Tschacher
581568ff18 cleaned up google scrapers. All scrapers are classes now. from 600 LOC to 400 LOC. HIGH IQ MOVE 2019-01-30 20:24:03 +01:00
Nikolai Tschacher
4306848657 implemented generic scraping class 2019-01-30 16:05:08 +01:00
Nikolai Tschacher
9e62f23451 resolved some issues. proxy possible now. scraping for more than one page possible now 2019-01-29 22:48:08 +01:00
Nikolai Tschacher
89441070cd before_keyword_scraped() hook supported 2019-01-29 13:29:24 +01:00
Nikolai Tschacher
c5e3e84e1d minor changes 2019-01-27 22:11:41 +01:00
Nikolai Tschacher
a20b6c79a9 added handle_results 2019-01-27 20:08:09 +01:00
Nikolai Tschacher
86a66a09fd pluggable is now class 2019-01-27 19:08:07 +01:00
Nikolai Tschacher
8e695626b6 added pluggable functionality 2019-01-27 15:54:56 +01:00
Nikolai Tschacher
e78d7145b5 faster scraping, added ticker search engines 2019-01-27 01:27:52 +01:00
Nikolai Tschacher
b0e588f916 removed file 2019-01-26 20:15:40 +01:00
Nikolai Tschacher
b354e6918d . 2019-01-26 20:15:19 +01:00
Nikolai Tschacher
bab902e80a supporting yahoo ticker search for news 2019-01-24 15:50:03 +01:00
Nikolai Tschacher
9cfa502851 removed idea stuff 2018-12-24 14:30:36 +01:00
Nikolai Tschacher
a96f14a4f8 second 2018-12-24 14:26:58 +01:00
Nikolai Tschacher
9af1630e19 initial 2018-12-24 14:25:02 +01:00