S’il vous prenait l’envie de gravir un petit millier de marches, comme ça, juste pour le plaisir, on vous conseille de vous rendre au mont Tianmen, en Chine (littéralement « la porte du Paradis », rien que ça). Situé 1 500 mètres au-dessus du niveau de la mer, ce trou dans la montagne, que vous voyez sur l’image d’aujourd’hui, est la plus haute arche naturelle au monde. A l’origine il y avait une grotte mais cette dernière se transforma en arche en 263 avant J.C quand l’arrière de la montagne s’effondra, créant alors cette cavité béante. Pour atteindre le sommet, il vous faudra escalader les 999 marches qui y mènent. Mais la vue en vaut la peine, c’est promis.
In keyword-driven tests, every action to be performed (a mouse click, keystroke, and so on) is described by a keyword. TestComplete supports this kind of tests. This section contains topics that describe how to create and use keyword tests with TestComplete.
Some keyword test operations (for instance, the Log Message, Delay and Comment operations) do not have nodes. If you add such operations to a test, they may break the current group node. To avoid this, TestComplete allows you to specify keyword test operations that do not break operation groups when they are added to a test.
Keyword Tool vous permet d'extraire des mots-clés de 192 domaines Google et d'utiliser 83 langues pour générer des suggestions de mots clés. De cette façon, nous nous assurons que les mots clés générés seront pertinents pour le pays et / ou la langue pour laquelle vous créez votre contenu.
A Keyword or Action Word is a defined combination of actions on a test object which describes how test lines must be executed. An action word contains arguments and is defined by a test analyst. An action word contains arguments and is defined by a test analyst.
Découvrez de nouveaux mots clés Cherchez des termes ou des expressions en rapport avec vos produits ou services. Notre outil de recherche de mots clés vous aidera à trouver les mots clés les plus pertinents pour votre activité.
Un Keyword Tool performant . Vous souhaitez connaître la position exacte de vos mots-clés ? Suivez en détail la position de vos mots-clés, analysez l'historique de vos performances, la popularité des mots-clés que vous avez choisi et comparez vos résultats avec ceux de vos concurrents avec l'outil Keyword Tool. Essayez-le maintenant!
Keywords were created to gives users another level of depth when categorizing test cases. Keyword Creation. At this time keywords can only be created by users with the mgt_modify_key rights. These rights are currently held only by leads. Once a keyword or grouping of keywords have been created users may assign them to test cases.
Type pattern, which tests whether an expression can be converted to a specified type and, if it can be, casts it to a variable of that type. Modèle de constante : teste si une expression correspond à une valeur de constante spécifiée. Constant pattern, which tests whether an expression evaluates to …
Using Keyword Tool, you can choose a specific Google domain out of 192 supported domains and one out of 83 languages that will be used to produce keyword suggestions. The free version of Keyword Tool can generate up to 750+ keywords from Google autocomplete in seconds.
22/06/2011 · This video demonstrates how you can create and use parameters in automated keyword-driven tests. Parameters allow you to pass data into keyword-driven tests or between tests. By replacing hard ...
Faites le test dès maintenant. Cet outil permet de tester si votre page est conforme aux exigences de Google et la compare avec les pages obtenant les meilleurs scores. SEO Page Optimizer vous donne toutes les clés pour vous approcher au maximum des chiffres indiqués. Vous pouvez ainsi obtenir un meilleur positionnement dans Google.
Keyword Position Checker is a tool used to detect the position of a website or URL in the search engine (particularly, Google) for a given keyword as per competing with other websites for the same keyword.
Our keyword research tool gives you insight into how often certain words are searched and how those searches have changed over time. This can help you narrow your …
Keyword Density Checker is a tool built solely for the purpose of calculating the keyword density of any web page. The dev team at Small SEO Tools created the tool after finding out that some marketers were still stuffing their content with loads of keywords even without realizing it.
Keyword-driven testing is an approach to software testing that separates test case design from execution. Each automated UI test is built on a series of operations, specified by keywords, that simulates a user action, such as a mouse click or keystroke.
Keywords Usage Test What is it? This will check if your most common keywords are used in the webpage's title and description. Check your URL: × Would you like to analyze more competitors simultaneously? Sign up for our free trial. Checkup! How do I fix it? First of all, you must make sure that your page is using the title and meta-description tags. Second, you must adjust these tags content ...
The Keyword Cloud is a visual representation of keywords used on your website. This will show you which words are frequently used in the content of your webpage. Keywords having higher density are presented in larger fonts and displayed in alphabetic order.
The keyword @ImposedOpeningDisplacement is not documented yet. The @ImposedStrain keyword. The @ImposedStrain keyword allows the user to impose the evolution of a component of the strains. This keyword may have one option, which is the way the evolution will be defined. Two values are accepted: evolution and function.
La version complète convient aux développeurs en leur offrant la possibilité de coder des tests complets et/ou des keywords directement en Python. Dans tous les cas, il permet l'écriture rapide de jeux de tests, et ses rapports, à la fois complets et explicites ne pourront …
Get 7x more the best YouTube keywords with YouTube Keyword Tool alternative! Explore the step-by-step algorithm for using Kparser for blogging: find new ideas for own videos, generate thousands of long-tail suggestions for the most profitable keywords in YouTube title, description, tags.
Keywords with exact search volumes Get search volumes with historical data. Timing is the key! Be ready and create content based on historical search volumes and long-term trends. Identify seasonal keywords and hot topics 🌶️ that will boost the organic traffic of your website.
The SEO Expert Quiz has 50 action-packed questions and takes 15 minutes to complete. You have nothing to lose and a lot of prestige to gain. Let the games …
Nov 21, 2019In keyword-driven tests, every action to be performed (a mouse click, keystroke, and so on) is described by a keyword. TestComplete supports this kind of tests. This section contains topics that describe how to create and use keywordtests with TestComplete.
You can execute keywordtests in any of the following ways: You can run the test as your project's test item. If a test has one or more tags assigned, TestComplete will run it when you run tests by tags. You can run the test from the Project Explorer panel. You can run the test from the Keyword ...
The keyword-driven testing methodology divides test process execution into several stages: Model basis/prototyping: analysis and assessment of requirements. Test model definition: on the result requirements assessment, approach an own software model. Test data definition: on the basis of the ...
Keyword-driven testing is an approach to software testing that separates test case design from execution. Each automated UI test is built on a series of operations, specified by keywords, that simulates a user action, such as a mouse click or keystroke.
A keyword-driven test consists of high and low-level keywords, including the keyword arguments, which is composed to describe the action of a test case. In Keyword Driven Testing, you first identify a set of keywords and then associate an action (or function) related to these keywords.
Test your website with a Free SEO Check-Up. Make a free SEO online analysis of a website with simple and effective web marketing tools: keyword analysis, complete website crawling, SEO copywriting assistant, page performance, html structure and social media popularity.
Keyword Position Checker is a tool used to detect the position of a website or URL in the search engine (particularly, Google) for a given keyword as per competing with other websites for the same keyword.
Keyword Tool Is The Best Alternative To Google Keyword Planner And Other Keyword Research Tools. Here are a few reasons why: Free version of Keyword Tool generates up to 750+ long-tail keyword suggestions for every search term
"KWFinder is an excellent tool that really simplifies the keyword research process in a way other tools just haven't managed to do." Read the review More than 18,000 paying customers can't be wrong.
The meta keywords tags, comments tags, and other somewhat hidden inputs may be given less weight than page copy. For instance, most large scale hypertext search engines put zero weight on the meta keyword tag. Page copy which is bolded, linked, or in a heading tag is likely given greater weighting than normal text. Weights are relative.
Nov 21, 2019In keyword-driven tests, every action to be performed (a mouse click, keystroke, and so on) is described by a keyword. TestComplete supports this kind of tests. This section contains topics that describe how to create and use keywordtests with TestComplete.
You can execute keywordtests in any of the following ways: You can run the test as your project's test item. If a test has one or more tags assigned, TestComplete will run it when you run tests by tags. You can run the test from the Project Explorer panel. You can run the test from the Keyword ...
The keyword-driven testing methodology divides test process execution into several stages: Model basis/prototyping: analysis and assessment of requirements. Test model definition: on the result requirements assessment, approach an own software model. Test data definition: on the basis of the ...
Keyword-driven testing is an approach to software testing that separates test case design from execution. Each automated UI test is built on a series of operations, specified by keywords, that simulates a user action, such as a mouse click or keystroke.
A keyword-driven test consists of high and low-level keywords, including the keyword arguments, which is composed to describe the action of a test case. In Keyword Driven Testing, you first identify a set of keywords and then associate an action (or function) related to these keywords.
Test your website with a Free SEO Check-Up. Make a free SEO online analysis of a website with simple and effective web marketing tools: keyword analysis, complete website crawling, SEO copywriting assistant, page performance, html structure and social media popularity.
Keyword Position Checker is a tool used to detect the position of a website or URL in the search engine (particularly, Google) for a given keyword as per competing with other websites for the same keyword.
Keyword Tool Is The Best Alternative To Google Keyword Planner And Other Keyword Research Tools. Here are a few reasons why: Free version of Keyword Tool generates up to 750+ long-tail keyword suggestions for every search term
"KWFinder is an excellent tool that really simplifies the keyword research process in a way other tools just haven't managed to do." Read the review More than 18,000 paying customers can't be wrong.
The meta keywords tags, comments tags, and other somewhat hidden inputs may be given less weight than page copy. For instance, most large scale hypertext search engines put zero weight on the meta keyword tag. Page copy which is bolded, linked, or in a heading tag is likely given greater weighting than normal text. Weights are relative.
New Test Guides. Alzheimer's Disease Beta-Amyloid 42/40 Ratio, Plasma This test is used to help assess whether Alzheimer's disease (AD) in adults is the cause of subjective cognitive decline (SCD), mild cognitive impairment (MCI), or dementia.
Research keywords Our keyword research tool gives you insight into how often certain words are searched and how those searches have changed over time. This can help you narrow your keyword list down to the ones you really want.
Value Description; AllFaults: All faults are logged. CompactCSwitch: Used in connection with CSwitch, this reduces the information logged for each CSwitch, as well as uses differential compression and batching.
These tools can help you answer that question, so they'd make great additions to your keyword research arsenal: Moz Keyword Explorer - Input a keyword in Keyword Explorer and get information like monthly search volume and SERP features (like local packs or featured snippets) that are ranking for that term. The tool extracts accurate search ...
Keyword Density Checker is a tool built solely for the purpose of calculating the keyword density of any web page. The dev team at Small SEO Tools created the tool after finding out that some marketers were still stuffing their content with loads of keywords even without realizing it.
Oct 16, 2019Keyword-Driven Testing (KDT) is a battle-tested method to speed up test automation development and cut down scaling costs in the long run. This article gives you the best practices that we've crystallized from our 20+ years of helping clients successfully build and maintain their Keyword-Driven Testing frameworks.
A keyword symbolizes a function and is defined as a set of actions that must be executed on a test object. A keyword-driven test consists of high and low-level keywords, including the keyword arguments, which is composed to describe the action of a test case. 2. Advantages and disadvantages of Keyword-driven testing Advantage:
Keywords are used to categorize test cases. Keywords are ideal for filtering. It is not categorizing and placing the same test cases at one place; keywords help to bring test cases from different folders with the same scope together using the same keywords. For Example − Regression Test Cases, Smoke Test Cases, Solaris and Change Request, etc.
WordStream's Free Keyword Tool makes it fast and easy to find the keywords your business needs to drive traffic through organic and paid search. Each keyword has a competition score, an estimated CPC, and a proprietary Opportunity Score that helps you make the most of your marketing budget. Download ...
Day 3: Testing Keywords with Google AdWords Test Before You Build. Here's a simple tip that could easily save you 12 months of work: Setting up an effective SEO campaign in a competitive market can be a long drawn out and expensive process.
Most Common KeywordsTest What is it? Check the most common keywords in your page and their usage (number of times used). This can help give a quick overview of the keywords and topics that crawlers may associate with your web page.
Keyword and Data Driven Test Cases - The workflow of a test-case can be tested using keyword or data driven style. In case you want to test the workflow with different inputs, the same can be done
7 Excellent Websites for Checking Google Keyword Rankings 19th Dec, 2019 Harsh Agrawal 56 Comments One of the essential elements of SEO is picking the right keyword, making them rank & track the keyword position in search.
Find keywords with a high volume of traffic and CPC for quick ranking of your website or blog. SEO Comparison Tool Insert a keyword to assess the strength of your competitors backlink compared to yours.
Basically Cucumber doesn't care about what Keyword you use to define test steps, all it cares about what code it needs to execute for each step. That code is called a step definition and we will discuss about it in the next section. At this time just remember that all the keywords can be replaced by the * keyword and your test will just work ...
The is keyword supports the following patterns: Type pattern, which tests whether an expression can be converted to a specified type and, if it can be, casts it to a variable of that type. Constant pattern, which tests whether an expression evaluates to a specified constant value.
"I've been using seositecheckup.com over three months now and i would like to congratulate you for your hard work." Thodoris Themelis Analyze and monitor your SEO with our powerful ToolBox.
A keyword-driven test is a sequence of operations, in a keyword format, that simulate user actions on the tested application. Basically, to perform any testing actions, testers simply drag and drop the keyword that corresponds to the desired operation or they can just record their actions and the keyword-driven test is built for them.
Online Keyword Density Checker. Keyword density is the percentage of times a keyword or phrase used in a web page as compare to the total content available on that web page. In SEO term, webmasters calculate it to determine whether a web page is relevant to a specified topic.
Nov 21, 2019In keyword-driven tests, every action to be performed (a mouse click, keystroke, and so on) is described by a keyword. TestComplete supports this kind of tests. This section contains topics that describe how to create and use keywordtests with TestComplete.
You can execute keywordtests in any of the following ways: You can run the test as your project's test item. If a test has one or more tags assigned, TestComplete will run it when you run tests by tags. You can run the test from the Project Explorer panel. You can run the test from the Keyword ...
The keyword-driven testing methodology divides test process execution into several stages: Model basis/prototyping: analysis and assessment of requirements. Test model definition: on the result requirements assessment, approach an own software model. Test data definition: on the basis of the ...
Keyword-driven testing is an approach to software testing that separates test case design from execution. Each automated UI test is built on a series of operations, specified by keywords, that simulates a user action, such as a mouse click or keystroke.
A keyword-driven test consists of high and low-level keywords, including the keyword arguments, which is composed to describe the action of a test case. In Keyword Driven Testing, you first identify a set of keywords and then associate an action (or function) related to these keywords.
Test your website with a Free SEO Check-Up. Make a free SEO online analysis of a website with simple and effective web marketing tools: keyword analysis, complete website crawling, SEO copywriting assistant, page performance, html structure and social media popularity.
Keyword Position Checker is a tool used to detect the position of a website or URL in the search engine (particularly, Google) for a given keyword as per competing with other websites for the same keyword.
Keyword Tool Is The Best Alternative To Google Keyword Planner And Other Keyword Research Tools. Here are a few reasons why: Free version of Keyword Tool generates up to 750+ long-tail keyword suggestions for every search term
"KWFinder is an excellent tool that really simplifies the keyword research process in a way other tools just haven't managed to do." Read the review More than 18,000 paying customers can't be wrong.
The meta keywords tags, comments tags, and other somewhat hidden inputs may be given less weight than page copy. For instance, most large scale hypertext search engines put zero weight on the meta keyword tag. Page copy which is bolded, linked, or in a heading tag is likely given greater weighting than normal text. Weights are relative.
New Test Guides. Alzheimer's Disease Beta-Amyloid 42/40 Ratio, Plasma This test is used to help assess whether Alzheimer's disease (AD) in adults is the cause of subjective cognitive decline (SCD), mild cognitive impairment (MCI), or dementia.
Research keywords Our keyword research tool gives you insight into how often certain words are searched and how those searches have changed over time. This can help you narrow your keyword list down to the ones you really want.
Value Description; AllFaults: All faults are logged. CompactCSwitch: Used in connection with CSwitch, this reduces the information logged for each CSwitch, as well as uses differential compression and batching.
These tools can help you answer that question, so they'd make great additions to your keyword research arsenal: Moz Keyword Explorer - Input a keyword in Keyword Explorer and get information like monthly search volume and SERP features (like local packs or featured snippets) that are ranking for that term. The tool extracts accurate search ...
Keyword Density Checker is a tool built solely for the purpose of calculating the keyword density of any web page. The dev team at Small SEO Tools created the tool after finding out that some marketers were still stuffing their content with loads of keywords even without realizing it.
Oct 16, 2019Keyword-Driven Testing (KDT) is a battle-tested method to speed up test automation development and cut down scaling costs in the long run. This article gives you the best practices that we've crystallized from our 20+ years of helping clients successfully build and maintain their Keyword-Driven Testing frameworks.
A keyword symbolizes a function and is defined as a set of actions that must be executed on a test object. A keyword-driven test consists of high and low-level keywords, including the keyword arguments, which is composed to describe the action of a test case. 2. Advantages and disadvantages of Keyword-driven testing Advantage:
Keywords are used to categorize test cases. Keywords are ideal for filtering. It is not categorizing and placing the same test cases at one place; keywords help to bring test cases from different folders with the same scope together using the same keywords. For Example − Regression Test Cases, Smoke Test Cases, Solaris and Change Request, etc.
WordStream's Free Keyword Tool makes it fast and easy to find the keywords your business needs to drive traffic through organic and paid search. Each keyword has a competition score, an estimated CPC, and a proprietary Opportunity Score that helps you make the most of your marketing budget. Download ...
Day 3: Testing Keywords with Google AdWords Test Before You Build. Here's a simple tip that could easily save you 12 months of work: Setting up an effective SEO campaign in a competitive market can be a long drawn out and expensive process.
Most Common KeywordsTest What is it? Check the most common keywords in your page and their usage (number of times used). This can help give a quick overview of the keywords and topics that crawlers may associate with your web page.
Keyword and Data Driven Test Cases - The workflow of a test-case can be tested using keyword or data driven style. In case you want to test the workflow with different inputs, the same can be done
7 Excellent Websites for Checking Google Keyword Rankings 19th Dec, 2019 Harsh Agrawal 56 Comments One of the essential elements of SEO is picking the right keyword, making them rank & track the keyword position in search.
Find keywords with a high volume of traffic and CPC for quick ranking of your website or blog. SEO Comparison Tool Insert a keyword to assess the strength of your competitors backlink compared to yours.
Basically Cucumber doesn't care about what Keyword you use to define test steps, all it cares about what code it needs to execute for each step. That code is called a step definition and we will discuss about it in the next section. At this time just remember that all the keywords can be replaced by the * keyword and your test will just work ...
The is keyword supports the following patterns: Type pattern, which tests whether an expression can be converted to a specified type and, if it can be, casts it to a variable of that type. Constant pattern, which tests whether an expression evaluates to a specified constant value.
"I've been using seositecheckup.com over three months now and i would like to congratulate you for your hard work." Thodoris Themelis Analyze and monitor your SEO with our powerful ToolBox.
A keyword-driven test is a sequence of operations, in a keyword format, that simulate user actions on the tested application. Basically, to perform any testing actions, testers simply drag and drop the keyword that corresponds to the desired operation or they can just record their actions and the keyword-driven test is built for them.
Online Keyword Density Checker. Keyword density is the percentage of times a keyword or phrase used in a web page as compare to the total content available on that web page. In SEO term, webmasters calculate it to determine whether a web page is relevant to a specified topic.
Apr 30, 2019Can you name the Java keywords? Test your knowledge on this science quiz to see how you do and compare your score to others. play quizzes ad-free. Random Quiz ...
Nov 10, 2019The Keyword driven testing framework is an extension to Data driven Testing Framework in a sense that it not only segregates the test data from the scripts, it also keeps the certain set of code belonging to the test script into an external data file.
Check Search Optimization Automatically. Google Webmaster Guidelines - check for hidden text, single-pixel links, links to bad neighborhoods, sneaky redirects, etc.; Bing/Live Webmaster Guidelines - check for keyword stuffed links and ALT tags, etc.
Python Global Keyword In this article, you'll learn about the global keyword, global variable and when to use global keywords. Before reading this article, make sure you have got some basics of Python Global, Local and Nonlocal Variables .
this keyword in java. There can be a lot of usage of java this keyword.In java, this is a reference variable that refers to the current object.. Usage of java this keyword. Here is given the 6 usage of java this keyword.
Use Wordtracker to reveal 1000s of profitable longtail keywords with up to 10,000 results per search Free Keyword Research Tool from Wordtracker The best FREE alternative to the Keyword Planner.
Keywords Studios are trusted to deliver superior voiced Audio solutions for our clients, every time. Managing actors, studios, voice directors and audio engineers around the world, our Audio Services provide the original langue and/or localized audio for most of the world's leading console and PC games.
Sep 25, 2019ATS keyword optimization This is one of the most important elements of a resume in regards to getting it past an ATS test. Keywords , in this case, refer to words that the ATS may be looking for that match the industry or job description.
LabCorp's test menu provides a comprehensive list of specialty and general laboratory testing services. Search Our Health Care Diagnostics Tests Menu | LabCorp Skip to main content
Keyword research is the process of discovering words and phrases that people use in search engines with the ultimate goal of optimizing content around those terms. Why is Keyword Research Important for SEO? Keyword research impacts every other SEO task that you perform, including finding content topics, on-page SEO, and outreach and promotion.
Whereas the body of a keyword-driven test case is constructed from keywords and their possible arguments, test cases with template contain only the arguments for the template keyword. Instead of repeating the same keyword multiple times per test and/or with all tests in a file, it is possible to use it only per test or just once per file.
May 24, 2017How to Test your Keywords the Right Way. Testing is one area that many people don't do right when it comes to Facebook Ads. To find the best keywords for Facebook ads, you need to create a campaign plan that will split test to find out which keywords give you the better results.
The basic working of the Keyword Driven Framework is to divide the Test Case into four different parts. First is called as Test Step , second is Object of Test Step, third is Action on Test Object and fourth is Data for Test Object.
A keyword-driven testing framework separates test automation implementation from test case design. Each keyword represents a single test action. Keywords can be combined to create sophisticated automated tests by testers regardless of programming expertise.
This module allows a Python program to determine if a string is a keyword. keyword.iskeyword (s) ¶ Return true if s is a Python keyword. keyword.kwlist¶ Sequence containing all the keywords defined for the interpreter. If any keywords are defined to only be active when particular __future__ statements are in effect, these will be included as ...
Keywords is a great place to work for all gamers & provides ample growth opportunities, I have myself been promoted as a Senior Tester in just 2 years. Environment here is friendly, management is open to suggestion & implementing changes if required.
Jan 3, 2018test, tested, tester, testing schedule, scheduled, scheduler, scheduling The best practice for determining which tense or form to use with your resume keywords is to mirror the job description.
When you research keywords for car insurance, you'll certainly not want to miss keywords like auto policy limits, young drivers or hail damage for content and marketing ideas. Our Related Keyword Tool finds thousands of those for you, including search volume, Adwords prices and competition. Find out more about keyword.io PRO
Opening library documentation failed. Verify that you have JavaScript enabled in your browser.; Make sure you are using a modern enough browser.If using Internet Explorer, version 8 or newer is required.
The test files mentioned for the test-cases are similar to those as mentioned in the above example of Simple keyword based test-case. To run the test suite TAF makes use of the data-driven technique provided with TestNG. TAF provides in-built utilities that will parse the suite file and data-driven test object for execution as simple keywordtest.
Using the Keyword Density Analysis Tool. Content matters, not only from an end user perspective, but also from a search engine perspective. The words used on a webpage, including what type (keywords or stop words), how they are used (alone or within phrases) and where they are used (link text or non-link body text), can all influence the value of the page in search.
Apr 15, 2015Hans Buwalda is an internationally recognized expert in test development and testing technology management and a pioneer of keyword-driven test automation. He was the first to present this approach, which is now widely used throughout the testing industry.
Nov 21, 2019Keyword Driven Framework is a framework where keywords drive the test. Here keyword refers to the user-defined functions. In this framework, keywords are created in order to perform a particular test step or a test case. These keywords are then called into the driver test to run several test cases in the same test.
Keywordtest case design requires Application Under Test (AUT) domain expertise and strength in test methodology. Keyword implementation may require technical strength and programming skill in the detail of the AUT interface. The skill level required will depend on the complication / atypical implementation of the AUT.
Oct 30, 2018Keyword density is the percentage calculated based on the number of times a keyword occurs inside the content of webpage divided by the total word count. Keyword density / keyword frequency is still a pretty strong indicator to determine the main focus keywords and keyword phrases for a specific webpage.
Settling on the right SEO keywords is a delicate process, but we're here to walk you through it--from researching what your customers are looking for, to discovering the keywords that will help you rank on a search engine results page (SERP), to putting those keywords to work in your online content.
Keyword Planner will show you historical statistics such as search volume data for these keywords so you can decide whether to add them to your campaigns. This article goes over how you can use Keyword Planner to get search volume data and find new keywords that are relevant to your business.
Learn more and understand better with BrainPOP's animated movies, games, playful assessments, and activities covering Science, Math, History, English, and more!
This module allows a Python program to determine if a string is a keyword. keyword.iskeyword (s) ¶ Return True if s is a Python keyword. keyword.kwlist¶ Sequence containing all the keywords defined for the interpreter. If any keywords are defined to only be active when particular __future__ statements are in effect, these will be included as ...
One test can run the same keyword multiple times to validate multiple related variations; If the keyword is implemented as a user keyword, it typically contains a similar workflow as workflow tests. Unless needed elsewhere, it is a good idea to create it in the same file as tests using it. Recommended to use the test template functionality.
Test your knowledge with Moz's SEO Expert Quiz. 50 action-packed questions cover all of SEO. You have nothing to lose and a lot of prestige to gain. Let the games begin!
Keyword Density Basics. Keyword Density is the percentage of occurrence of your keywords to the text in the rest of your webpage. It is important for your main keywords to have the correct keyword density to rank well in Search Engines. A Keyword Cloud is a visual depiction of keywords used on a website. Keywords that appear more frequently on ...
I have a set of test data that is categorized by test case, and I would like to have a test setup keyword that loads the data for each test case. Something like this would work: *** Keywords *** ...
Also, more general keywords can be more competitive and may require higher bid amounts. Layer broad targeting with Smart Bidding, so machine learning technology can prioritize the best performing searches regardless of their match type. Learn more About Smart Bidding. Test general keywords, and then decide which ones give you better results.
Jul 22, 2017Keyword Driven Testing Framework: It is also known as table-driven testing or action word based testing. In Keyword-driven testing, we use a table format to define keywords or action words for each function or method that we would execute. It performs automation test scripts based on the keywords specified in the excel sheet.
TypingTest.com - Complete a Typing Test in 60 Seconds! Welcome to the #1 typing speed test with over 4 million tests completed every month! 1 minute test 3 minute test 5 minute test
7 days agoAnd finally, discover the keywords that people are already using to discover your site by using Google Analytics and Google Search Console. We describe the process in detail in our keyword research 101 article. Keyword research isn't just useful for creating new content. It's also an important part of improving SEO for existing content.
Nov 5, 2004Some keyword framework designers like to write test cases that use multiple keywords, as in the example above, so the script looks like a simple programming language. Others prefer to have a single keyword that can perform the work of an entire test case, blurring the line between data-driven and keyword-driven testing.
Keyword: Users can filter test cases by keyword. Keywords are set either using the Create/Edit/Delete Test Cases or by the Assign Keywords To Multiple Cases. Keywords can only be created, edited, or deleted by leads but may be assigned to test cases by testers.
Test accounts can be set up in a hierarchy and organized just like production accounts, but provide extra benefits during active development. Particularly, test accounts: Don't require an approved developer token, so you can start experimenting with the API immediately, even before your application is reviewed or approved.
This keyword is also used to declare that a method returns a value of the primitive type float. for The for keyword is used to create a for loop, which specifies a variable initialization, a boolean expression, and an incrementation. The variable initialization is performed first, and then the boolean expression is evaluated.
Nov 22, 2019Re: Keywordtest: Auto-Wait Timeout always used instead of WaitProperty's WaitTime value Check and make sure that your WaitTime is actually of type Integer. You can click the ellipses to the right of the value and see the properties.
Mar 12, 2019This is the ultimate guide to using Google Keyword Planner in 2019. In fact, I've used the Google Keyword Planner (formerly known as Google Keyword Tool) to help grow my site's organic traffic to 176,091 visits per month. And in this guide I'll show you how to get the most SEO value out of this awesome tool.
\ No newline at end of file
diff --git a/test/mocks/google/test keyword_page1.html b/test/mocks/google/test keyword_page1.html
new file mode 100644
index 0000000..975f547
--- /dev/null
+++ b/test/mocks/google/test keyword_page1.html
@@ -0,0 +1,209 @@
+test keyword - Google Search
Outil de mots clés SEMrush : Surpassez vos concurrents - Essai gratuit. Best SEO Suite 2018. 150 bases de données. 3M+ d’utilisateurs. 8 milliards de mots clés. Services: Analyse de mots clés, Audit de site, Analyse du trafic, Suivi de position, Recherche organique.
Nov 21, 2019 - Keyword-driven tests consist of keywords that define the action to be performed by the test. Each keyword corresponds to an individual action ...
In keyword-driven tests, every action to be performed (a mouse click, keystroke, and so on) is described by a keyword. TestComplete supports this kind of tests.
Keyword-driven testing, also known as table-driven testing or action word based testing is a software testing methodology suitable for both manual and ...
Dec 4, 2019 - A keyword-driven framework is a table-driven testing or action word based testing. It is a software testing method suitable for both manual and ...
Keyword-driven testing is a software testing methodology that separates test design from test development and therefore allows the involvement of additional ...
According to ISO/IEC/IEEE 29119-5: Keyword-Driven Testing is a way of describing test cases by using a predefined set of Keywords. These Keywords are ...
May 24, 2019 - This keyword makes it easy for test libraries to interact with other test libraries that have state. This is illustrated by the Python example below:
\ No newline at end of file
diff --git a/test/mocks/google/test keyword_page2.html b/test/mocks/google/test keyword_page2.html
new file mode 100644
index 0000000..ab36724
--- /dev/null
+++ b/test/mocks/google/test keyword_page2.html
@@ -0,0 +1,206 @@
+test keyword - Google Search
Improve Your Quality Score. Boost Your CTR. Reduce Your CPC. 5X Faster Than The Editor. Automatic Negative Keyword Generation, DKI, 24/7 Chat Support & More. Sign Up...
Our approach targets users first because that's what search engines reward. This chapter covers keyword research and other methods to determine what your ...
Dec 4, 2019 - A keyword-driven framework is a table-driven testing or action word based testing. It is a software testing method suitable for both manual and automated testing. ... A keyword-driven testing is a scripting technique that uses data files to contain the keywords related to the ...
Robot Framework is a Python-based, extensible keyword-driven automation framework for acceptance testing, acceptance test driven development (ATDD), ...
According to ISO/IEC/IEEE 29119-5: Keyword-Driven Testing is a way of describing test cases by using a predefined set of Keywords. These Keywords are ...
Introduction[edit]. The hallmark of keyword test design is extensive reuse of keywords to build libraries of focused test cases to test the full scope of functionality ...
After adding keywords to the repository, you can add them to a new or existing script. Keyword-enabled manual test scripts can be composed solely of keywords, ...
For example, all the tests that Rank Math performs won't work well if Rank Math considers all the focus keywords at the same time. Some tests only work well if ...
The tool also shows a preview of a Google search result page for a specific term. This helps you see which ads and extensions are appearing for your keyword.
\ No newline at end of file
diff --git a/test/mocks/google/test keyword_page3.html b/test/mocks/google/test keyword_page3.html
new file mode 100644
index 0000000..b27b638
--- /dev/null
+++ b/test/mocks/google/test keyword_page3.html
@@ -0,0 +1,191 @@
+test keyword - Google Search
A keyword-driven testing framework separates test automation implementation from test case design. Each keyword represents a single test action. Keywords ...
Definition and Usage. The is keyword is used to test if two variables refer to the same object. The test returns True if the two objects are the same object. The test ...
Download Your Competitors' Most Profitable Keywords and Ads For Paid ... See every keyword they've ever bought on Google and every ad test they've run.
... with powerful SEO Tools. Test your website with a Free SEO Check-Up. ... find the best keyword and write SEO-friendly content. Optimize your website now ...
Dec 5, 2019 - Keyword-driven testing is a software testing methodology which uses keywords (or action words) to symbolize a functionality to be tested.
Keyword Driven Testing is the next generation test automation approach that separates the task of automated test case implementation from the automation ...
\ No newline at end of file
diff --git a/test/modules/bing.js b/test/modules/bing.js
new file mode 100644
index 0000000..bbc9dd0
--- /dev/null
+++ b/test/modules/bing.js
@@ -0,0 +1,123 @@
+'use strict';
+const express = require('express');
+const puppeteer = require('puppeteer');
+const { createLogger, transports } = require('winston');
+const http = require('http');
+const https = require('https');
+const assert = require('assert');
+const path = require('path');
+const keyCert = require('key-cert');
+const Promise = require('bluebird');
+const Proxy = require('http-mitm-proxy');
+
+const debug = require('debug')('se-scraper:test');
+const { BingScraper } = require('../../src/modules/bing');
+
+const httpPort = 3012;
+const httpsPort = httpPort + 1;
+const proxyPort = httpPort + 2;
+
+const fakeSearchEngine = express();
+fakeSearchEngine.get('/search', (req, res, next) => {
+ debug('q=%s', req.query.q);
+ const pageNumber = Math.round((req.query.first || 0) /10) + 1;
+ res.sendFile(path.join(__dirname, '../mocks/bing/' + req.query.q + '_page' + pageNumber + '.html'));
+});
+fakeSearchEngine.use(express.static('test/mocks/bing', {extensions: ['html']}));
+
+describe('Module Bing', function(){
+
+ let httpServer, httpsServer, proxy;
+ before(async function(){
+ // Here mount our fake engine in both http and https listen server
+ httpServer = http.createServer(fakeSearchEngine);
+ httpsServer = https.createServer(await keyCert(), fakeSearchEngine);
+
+ proxy = Proxy();
+ proxy.onRequest((ctx, callback) => {
+ ctx.proxyToServerRequestOptions.host = 'localhost';
+ ctx.proxyToServerRequestOptions.port = (ctx.isSSL) ? httpsPort : httpPort;
+ ctx.proxyToServerRequestOptions.headers['X-Forwarded-Host'] = 'ProxiedThroughFakeEngine';
+ debug('connection proxied askedHost=%s toPort=%s', ctx.clientToProxyRequest.headers.host, ctx.proxyToServerRequestOptions.port);
+ return callback();
+ });
+
+ await Promise.promisify(proxy.listen, { context: proxy })({ port: proxyPort });
+ await Promise.promisify(httpServer.listen, {context: httpServer})(httpPort);
+ await Promise.promisify(httpsServer.listen, {context: httpsServer})(httpsPort);
+ debug('Fake http search engine servers started');
+ });
+
+ after(function(){
+ proxy.close();
+ httpsServer.close();
+ httpServer.close();
+ });
+
+ let browser;
+ let page;
+ beforeEach(async function(){
+ debug('Start a new browser');
+ browser = await puppeteer.launch({
+ //dumpio: true,
+ //headless: false,
+ ignoreHTTPSErrors: true,
+ args: [ '--proxy-server=http://localhost:' + proxyPort ]
+ });
+ debug('Open a fresh page');
+ page = await browser.newPage();
+ });
+
+ afterEach(async function(){
+ await browser.close();
+ });
+
+ const testLogger = createLogger({
+ transports: [
+ new transports.Console({
+ level: 'error'
+ })
+ ]
+ });
+
+ it('one keyword one page', function(){
+ const bingScraper = new BingScraper({
+ config: {
+ search_engine_name: 'bing',
+ throw_on_detection: true,
+ keywords: ['test keyword'],
+ logger: testLogger,
+ scrape_from_file: '',
+ }
+ });
+ bingScraper.STANDARD_TIMEOUT = 500;
+ return bingScraper.run({page}).then(({results, metadata, num_requests}) => {
+ assert.strictEqual(num_requests, 1, 'Must do one request');
+ assert.strictEqual(results['test keyword']['1'].results.length, 6, 'Must have 6 organic results parsed');
+ });
+ });
+
+ it('one keyword 3 pages', function () {
+ const bingScraper = new BingScraper({
+ config: {
+ search_engine_name: 'bing',
+ throw_on_detection: true,
+ keywords: ['test keyword'],
+ logger: testLogger,
+ scrape_from_file: '',
+ num_pages: 3,
+ }
+ });
+ bingScraper.STANDARD_TIMEOUT = 500;
+ return bingScraper.run({page}).then(({results, metadata, num_requests}) => {
+ assert.strictEqual(num_requests, 3, 'Must three requests');
+ assert.strictEqual(results['test keyword']['1'].results.length, 6, 'Must have 6 organic results parsed on page 1');
+ assert.strictEqual(results['test keyword']['1'].results[0].title, 'Keyword Tests | TestComplete Documentation', 'Title not matching on first organic result page 1');
+ assert.strictEqual(results['test keyword']['2'].results.length, 10, 'Must have 10 organic results parsed on page 2');
+ assert.strictEqual(results['test keyword']['2'].results[0].title, 'Keywords - TestLink', 'Title not matching on first organic result page 2');
+ assert.strictEqual(results['test keyword']['3'].results.length, 10, 'Must have 10 organic results parsed on page 3');
+ assert.strictEqual(results['test keyword']['3'].results[0].title, 'Keyword Driven Testing | TestComplete', 'Title not matching on first organic result page 3');
+ });
+ });
+
+});
\ No newline at end of file
diff --git a/test/modules/duckduckgo.js b/test/modules/duckduckgo.js
new file mode 100644
index 0000000..0997200
--- /dev/null
+++ b/test/modules/duckduckgo.js
@@ -0,0 +1,140 @@
+'use strict';
+const express = require('express');
+const puppeteer = require('puppeteer');
+const { createLogger, transports } = require('winston');
+const http = require('http');
+const https = require('https');
+const assert = require('assert');
+const path = require('path');
+const keyCert = require('key-cert');
+const Promise = require('bluebird');
+const Proxy = require('http-mitm-proxy');
+
+const debug = require('debug')('se-scraper:test');
+const { DuckduckgoScraper } = require('../../src/modules/duckduckgo');
+
+const httpPort = 3012;
+const httpsPort = httpPort + 1;
+const proxyPort = httpPort + 2;
+
+const fakeSearchEngine = express();
+fakeSearchEngine.use(express.urlencoded({ extended: true }))
+fakeSearchEngine.get('/', (req, res, next) => {
+ if(!req.query.q){
+ return next();
+ }
+ debug('q=%s page=%d', req.query.q, req.query.page);
+ const pageNumber = req.query.page;
+ res.sendFile(path.join(__dirname, '../mocks/duckduckgo/' + req.query.q + '_page' + pageNumber + '.html'));
+});
+fakeSearchEngine.post('/html', (req, res) => {
+ debug('body=%o', req.body);
+ const pageNumber = 1;
+ res.sendFile(path.join(__dirname, '../mocks/duckduckgo/' + req.body.q + '_page' + pageNumber + '.html'));
+});
+fakeSearchEngine.use(express.static('test/mocks/duckduckgo', {extensions: ['html']}));
+
+describe('Module DuckDuckGo', function(){
+
+ let httpServer, httpsServer, proxy;
+ before(async function(){
+ // Here mount our fake engine in both http and https listen server
+ httpServer = http.createServer(fakeSearchEngine);
+ httpsServer = https.createServer(await keyCert(), fakeSearchEngine);
+
+ proxy = Proxy();
+ proxy.onRequest((ctx, callback) => {
+ ctx.proxyToServerRequestOptions.host = 'localhost';
+ ctx.proxyToServerRequestOptions.port = (ctx.isSSL) ? httpsPort : httpPort;
+ ctx.proxyToServerRequestOptions.headers['X-Forwarded-Host'] = 'ProxiedThroughFakeEngine';
+ debug('proxy askedHost=%s method=%s url=%s toPort=%s',
+ ctx.clientToProxyRequest.headers.host,
+ ctx.clientToProxyRequest.method,
+ ctx.clientToProxyRequest.url,
+ ctx.proxyToServerRequestOptions.port
+ );
+ return callback();
+ });
+
+ await Promise.promisify(proxy.listen, { context: proxy })({ port: proxyPort });
+ await Promise.promisify(httpServer.listen, {context: httpServer})(httpPort);
+ await Promise.promisify(httpsServer.listen, {context: httpsServer})(httpsPort);
+ debug('Fake http search engine servers started');
+ });
+
+ after(function(){
+ proxy.close();
+ httpsServer.close();
+ httpServer.close();
+ });
+
+ let browser;
+ let page;
+ beforeEach(async function(){
+ debug('Start a new browser');
+ browser = await puppeteer.launch({
+ //dumpio: true,
+ //headless: false,
+ ignoreHTTPSErrors: true,
+ args: [ '--proxy-server=http://localhost:' + proxyPort ]
+ });
+ debug('Open a fresh page');
+ page = await browser.newPage();
+ });
+
+ afterEach(async function(){
+ await browser.close();
+ });
+
+ const testLogger = createLogger({
+ transports: [
+ new transports.Console({
+ level: 'error'
+ })
+ ]
+ });
+
+ it('one keyword one page', function(){
+ const duckduckgoScraper = new DuckduckgoScraper({
+ config: {
+ search_engine_name: 'duckduckgo',
+ throw_on_detection: true,
+ keywords: ['test keyword'],
+ logger: testLogger,
+ scrape_from_file: '',
+ }
+ });
+ duckduckgoScraper.STANDARD_TIMEOUT = 1000;
+ return duckduckgoScraper.run({page}).then(({results, metadata, num_requests}) => {
+ assert.strictEqual(num_requests, 1, 'Must do one request');
+ assert.strictEqual(results['test keyword']['1'].results.length, 10, 'Must have 10 organic results parsed');
+ });
+ });
+
+ it('one keyword 3 pages', function () {
+ this.timeout(4000);
+ const duckduckgoScraper = new DuckduckgoScraper({
+ config: {
+ search_engine_name: 'google',
+ throw_on_detection: true,
+ keywords: ['test keyword'],
+ logger: testLogger,
+ scrape_from_file: '',
+ num_pages: 3,
+ }
+ });
+ duckduckgoScraper.STANDARD_TIMEOUT = 1000;
+ return duckduckgoScraper.run({page}).then(({results, metadata, num_requests}) => {
+ assert.strictEqual(num_requests, 3, 'Must three requests');
+ assert.strictEqual(results['test keyword']['1'].results.length, 10, 'Must have 10 organic results parsed on page 1');
+ assert.strictEqual(results['test keyword']['1'].results[0].title, 'Keyword Tests | TestComplete Documentation', 'Title not matching on first organic result page 1');
+ debug('results page 1 %O',results['test keyword']['1'].results);
+ debug('results page 2 %O', results['test keyword']['2'].results);
+ assert.strictEqual(results['test keyword']['2'].results.length, 19, 'Must have 19 organic results parsed on page 2');
+ assert.strictEqual(results['test keyword']['2'].results[0].title, 'Quest Diagnostics: Test Directory', 'Title not matching on first organic result page 1');
+ assert.strictEqual(results['test keyword']['3'].results.length, 48, 'Must have 48 organic results parsed on page 3');
+ assert.strictEqual(results['test keyword']['3'].results[0].title, 'Java Keywords Quiz - Sporcle', 'Title not matching on first organic result page 1');
+ });
+ });
+
+});
\ No newline at end of file
diff --git a/test/modules/google.js b/test/modules/google.js
new file mode 100644
index 0000000..83c2ae3
--- /dev/null
+++ b/test/modules/google.js
@@ -0,0 +1,123 @@
+'use strict';
+const express = require('express');
+const puppeteer = require('puppeteer');
+const { createLogger, transports } = require('winston');
+const http = require('http');
+const https = require('https');
+const assert = require('assert');
+const path = require('path');
+const keyCert = require('key-cert');
+const Promise = require('bluebird');
+const Proxy = require('http-mitm-proxy');
+
+const debug = require('debug')('se-scraper:test');
+const { GoogleScraper } = require('../../src/modules/google');
+
+const httpPort = 3012;
+const httpsPort = httpPort + 1;
+const proxyPort = httpPort + 2;
+
+const fakeSearchEngine = express();
+fakeSearchEngine.get('/search', (req, res) => {
+ debug('q=%s', req.query.q);
+ const pageNumber = ((req.query.start/10) || 0) + 1;
+ res.sendFile(path.join(__dirname, '../mocks/google/' + req.query.q + '_page' + pageNumber + '.html'));
+});
+fakeSearchEngine.use(express.static('test/mocks/google', {extensions: ['html']}));
+
+describe('Module Google', function(){
+
+ let httpServer, httpsServer, proxy;
+ before(async function(){
+ // Here mount our fake engine in both http and https listen server
+ httpServer = http.createServer(fakeSearchEngine);
+ httpsServer = https.createServer(await keyCert(), fakeSearchEngine);
+
+ proxy = Proxy();
+ proxy.onRequest((ctx, callback) => {
+ ctx.proxyToServerRequestOptions.host = 'localhost';
+ ctx.proxyToServerRequestOptions.port = (ctx.isSSL) ? httpsPort : httpPort;
+ ctx.proxyToServerRequestOptions.headers['X-Forwarded-Host'] = 'ProxiedThroughFakeEngine';
+ debug('connection proxied askedHost=%s toPort=%s', ctx.clientToProxyRequest.headers.host, ctx.proxyToServerRequestOptions.port);
+ return callback();
+ });
+
+ await Promise.promisify(proxy.listen, { context: proxy })({ port: proxyPort });
+ await Promise.promisify(httpServer.listen, {context: httpServer})(httpPort);
+ await Promise.promisify(httpsServer.listen, {context: httpsServer})(httpsPort);
+ debug('Fake http search engine servers started');
+ });
+
+ after(function(){
+ proxy.close();
+ httpsServer.close();
+ httpServer.close();
+ });
+
+ let browser;
+ let page;
+ beforeEach(async function(){
+ debug('Start a new browser');
+ browser = await puppeteer.launch({
+ //dumpio: true,
+ //headless: false,
+ ignoreHTTPSErrors: true,
+ args: [ '--proxy-server=http://localhost:' + proxyPort ]
+ });
+ debug('Open a fresh page');
+ page = await browser.newPage();
+ });
+
+ afterEach(async function(){
+ await browser.close();
+ });
+
+ const testLogger = createLogger({
+ transports: [
+ new transports.Console({
+ level: 'error'
+ })
+ ]
+ });
+
+ it('one keyword one page', function(){
+ const googleScraper = new GoogleScraper({
+ config: {
+ search_engine_name: 'google',
+ throw_on_detection: true,
+ keywords: ['test keyword'],
+ logger: testLogger,
+ scrape_from_file: '',
+ }
+ });
+ googleScraper.STANDARD_TIMEOUT = 500;
+ return googleScraper.run({page}).then(({results, metadata, num_requests}) => {
+ assert.strictEqual(num_requests, 1, 'Must do one request');
+ assert.strictEqual(results['test keyword']['1'].results.length, 10, 'Must have 10 organic results parsed');
+ });
+ });
+
+ it('one keyword 3 pages', function () {
+ const googleScraper = new GoogleScraper({
+ config: {
+ search_engine_name: 'google',
+ throw_on_detection: true,
+ keywords: ['test keyword'],
+ logger: testLogger,
+ scrape_from_file: '',
+ num_pages: 3,
+ }
+ });
+ googleScraper.STANDARD_TIMEOUT = 500;
+ return googleScraper.run({page}).then(({results, metadata, num_requests}) => {
+ assert.strictEqual(num_requests, 3, 'Must three requests');
+ assert.strictEqual(results['test keyword']['1'].results.length, 10, 'Must have 10 organic results parsed on page 1');
+ assert.strictEqual(results['test keyword']['1'].results[0].title, 'Keyword Tool (FREE) ᐈ #1 Google Keyword Planner Alternative', 'Title not matching on first organic result page 1');
+ assert.strictEqual(results['test keyword']['2'].results.length, 10, 'Must have 10 organic results parsed on page 2');
+ assert.strictEqual(results['test keyword']['2'].results[0].title, 'Keyword Research | The Beginner\'s Guide to SEO - Moz', 'Title not matching on first organic result page 1');
+ assert.strictEqual(results['test keyword']['3'].results.length, 10, 'Must have 10 organic results parsed on page 3');
+ assert.strictEqual(results['test keyword']['3'].results[0].title, 'The ACT Keyword Study Plan — NerdCoach', 'Title not matching on first organic result page 1');
+ });
+ });
+
+});
\ No newline at end of file
diff --git a/test/proxy.js b/test/proxy.js
new file mode 100644
index 0000000..c1092ea
--- /dev/null
+++ b/test/proxy.js
@@ -0,0 +1,161 @@
+'use strict';
+const express = require('express');
+const { createLogger, transports } = require('winston');
+const http = require('http');
+const https = require('https');
+const assert = require('assert');
+const keyCert = require('key-cert');
+const Promise = require('bluebird');
+const Proxy = require('http-mitm-proxy');
+
+const debug = require('debug')('se-scraper:test');
+const se_scraper = require('../');
+const Scraper = require('../src/modules/se_scraper');
+
+const httpPort = 3012;
+const httpsPort = httpPort + 1;
+const proxyPort = httpPort + 2;
+
+const fakeSearchEngine = express();
+fakeSearchEngine.set('trust proxy', 'loopback');
+fakeSearchEngine.get('/test-proxy', (req, res) => {
+ debug('fake-search-engine req.hostname=%s', req.hostname);
+ //debug('req to', req.socket.localAddress, req.socket.localPort);
+ res.send(req.hostname);
+});
+
+describe('Config', function(){
+
+ let httpServer, httpsServer, proxy;
+ before(async function(){
+ // Here mount our fake engine in both http and https listen server
+ httpServer = http.createServer(fakeSearchEngine);
+ httpsServer = https.createServer(await keyCert(), fakeSearchEngine);
+
+ proxy = Proxy();
+ proxy.onRequest((ctx, callback) => {
+ ctx.proxyToServerRequestOptions.host = 'localhost';
+ ctx.proxyToServerRequestOptions.port = (ctx.isSSL) ? httpsPort : httpPort;
+ ctx.proxyToServerRequestOptions.headers['X-Forwarded-Host'] = 'ProxiedThroughFakeEngine';
+ debug('Proxy request to %s', ctx.clientToProxyRequest.headers.host);
+ return callback();
+ });
+
+ await Promise.promisify(proxy.listen, {context: proxy})({port: proxyPort});
+ await Promise.promisify(httpServer.listen, {context: httpServer})(httpPort);
+ await Promise.promisify(httpsServer.listen, {context: httpsServer})(httpsPort);
+ debug('Fake http search engine servers started');
+ });
+
+ after(function(){
+ httpsServer.close();
+ httpServer.close();
+ proxy.close();
+ });
+
+ describe('proxies', function(){
+
+ class MockScraperTestProxy extends Scraper {
+
+ async load_start_page(){
+ return true;
+ }
+
+ async search_keyword(){
+ await this.page.goto('http://test.local:' + httpPort + '/test-proxy');
+ }
+
+ async parse_async(){
+ const bodyHandle = await this.page.$('body');
+ return await this.page.evaluate(body => body.innerHTML, bodyHandle);
+ }
+ }
+
+ const testLogger = createLogger({
+ transports: [
+ new transports.Console({
+ level: 'error'
+ })
+ ]
+ });
+
+ /**
+ * Jobs will be executed 2 by 2 through the proxy and direct connection
+ * THIS TEST NEED TO HAVE test.local 127.0.0.1 in /etc/hosts because chrome bypass localhost even with proxy set
+ */
+ it('one proxy given, use_proxies_only=false', async function () {
+
+ const scrape_job = {
+ search_engine: MockScraperTestProxy,
+ keywords: ['news', 'some stuff', 'i work too much', 'what to do?', 'javascript is hard'],
+ };
+
+ var scraper = new se_scraper.ScrapeManager({
+ throw_on_detection: true,
+ proxies: ['http://localhost:' + proxyPort],
+ // default is use_proxies_only: false,
+ logger: testLogger,
+ });
+ await scraper.start();
+
+ const { results } = await scraper.scrape(scrape_job);
+ assert.strictEqual(results['news']['1'], 'test.local');
+ assert.strictEqual(results['some stuff']['1'], 'ProxiedThroughFakeEngine');
+ assert.strictEqual(results['i work too much']['1'], 'test.local');
+ assert.strictEqual(results['what to do?']['1'], 'ProxiedThroughFakeEngine');
+ assert.strictEqual(results['javascript is hard']['1'], 'test.local');
+
+ await scraper.quit();
+ });
+
+ /**
+ * Jobs will be executed 1 by 1 through the proxy
+ */
+ it('one proxy given, use_proxies_only=true', async function () {
+
+ const scrape_job = {
+ search_engine: MockScraperTestProxy,
+ keywords: ['news', 'some stuff', 'i work too much', 'what to do?', 'javascript is hard'],
+ };
+
+ var scraper = new se_scraper.ScrapeManager({
+ throw_on_detection: true,
+ proxies: ['http://localhost:' + proxyPort],
+ use_proxies_only: true,
+ logger: testLogger,
+ });
+ await scraper.start();
+
+ const { results } = await scraper.scrape(scrape_job);
+ assert.strictEqual(results['news']['1'], 'ProxiedThroughFakeEngine');
+ assert.strictEqual(results['some stuff']['1'], 'ProxiedThroughFakeEngine');
+ assert.strictEqual(results['i work too much']['1'], 'ProxiedThroughFakeEngine');
+ assert.strictEqual(results['what to do?']['1'], 'ProxiedThroughFakeEngine');
+ assert.strictEqual(results['javascript is hard']['1'], 'ProxiedThroughFakeEngine');
+
+ await scraper.quit();
+ });
+
+ it('zero proxy given, use_proxies_only=true', async function () {
+
+ const scrape_job = {
+ search_engine: MockScraperTestProxy,
+ keywords: ['news', 'some stuff', 'i work too much', 'what to do?', 'javascript is hard'],
+ };
+
+ await assert.rejects(async () => {
+ var scraper = new se_scraper.ScrapeManager({
+ throw_on_detection: true,
+ use_proxies_only: true,
+ logger: testLogger,
+ });
+ await scraper.start();
+ const { results } = await scraper.scrape(scrape_job);
+ await scraper.quit();
+ }, /Must provide at least one proxy in proxies if you enable use_proxies_only/);
+
+ });
+
+ });
+
+});
\ No newline at end of file
diff --git a/test/static_tests/README.md b/test/static_tests/README.md
deleted file mode 100644
index 140c6d2..0000000
--- a/test/static_tests/README.md
+++ /dev/null
@@ -1,15 +0,0 @@
-## Test with static HTML
-
-Dynamic testing of se-scraper takes too much time.
-
-Save some html and initialize se-scraper by loading the search from disk.
-
-### Disadvantage
-
-static html gets outdated after some time
-
-### Advantages
-
-1. Let's us test corner cases that are missed easily
-2. Testing is not reliable, since search engines do not always return the same results for the same query
-3. As said, much faster
\ No newline at end of file
diff --git a/test/static_tests/bing.js b/test/static_tests/bing.js
deleted file mode 100644
index ae0b127..0000000
--- a/test/static_tests/bing.js
+++ /dev/null
@@ -1,222 +0,0 @@
-const se_scraper = require('./../../index.js');
-const chai = require('chai');
-chai.use(require('chai-string'));
-const assert = chai.assert;
-const path = require('path');
-
-async function bing_ads() {
- let config = {
- compress: false,
- debug_level: 1,
- headless: true,
- };
-
- let scrape_config = {
- search_engine: 'bing',
- keywords: ['kaffeemaschine kaufen'],
- num_pages: 1,
- scrape_from_file: 'file://' + path.join(__dirname, './html/bing.html'),
- };
-
- var scraper = new se_scraper.ScrapeManager(config);
-
- await scraper.start();
-
- bing_search_with_ads( await scraper.scrape(scrape_config) );
-
- scrape_config.keywords = ['best cloud services'];
- scrape_config.scrape_from_file = 'file://' + path.join(__dirname, './html/bing2.html');
-
- bing_search_with_ads2( await scraper.scrape(scrape_config) );
-
- scrape_config.keywords = ['car tires cheap'];
- scrape_config.scrape_from_file = 'file://' + path.join(__dirname, './html/bing3.html');
-
- bing_search_with_ads3( await scraper.scrape(scrape_config) );
-
- scrape_config.keywords = ['service auto garage'];
- scrape_config.scrape_from_file = 'file://' + path.join(__dirname, './html/bing4.html');
-
- bing_search_with_ads4( await scraper.scrape(scrape_config) );
-
- await scraper.quit();
-}
-
-// we test with a callback function to our handler
-function bing_search_with_ads(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- for (let query in response.results) {
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.include(obj.num_results, '1’100’000', 'num results not included');
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query', 'ads'], 'not all keys are in the object');
- assert.isAtLeast(obj.results.length, 6, 'results must have at least 6 SERP objects');
- assert.isAtLeast(obj.ads.length, 7, 'there are 7 ads');
-
- assert.isAtLeast(obj.right_side_ads.length, 5, 'there are 5 ads');
-
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- confirm_results_ok(obj);
- }
- }
-}
-
-
-function bing_search_with_ads2(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- for (let query in response.results) {
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.include(obj.num_results, '44’300’000', 'num results not included');
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query', 'ads'], 'not all keys are in the object');
- assert.isAtLeast(obj.results.length, 6, 'results must have at least 6 SERP objects');
- assert.isAtLeast(obj.ads.length, 7, 'there are 7 ads');
-
- assert.isAtLeast(obj.right_side_ads.length, 5, 'there are 5 ads');
-
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- confirm_results_ok(obj);
- }
- }
-}
-
-function bing_search_with_ads3(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- for (let query in response.results) {
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.include(obj.num_results, '65.500.000 Results', 'num results not included');
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query', 'ads'], 'not all keys are in the object');
- assert.isAtLeast(obj.results.length, 10, 'results must have at least 10 SERP objects');
- assert.isAtLeast(obj.ads.length, 3, 'there are 3 ads');
-
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- confirm_results_ok(obj);
- }
- }
-}
-
-function bing_search_with_ads4(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- for (let query in response.results) {
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.include(obj.num_results, '4.200.000 Ergebnisse', 'num results not included');
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query', 'ads'], 'not all keys are in the object');
- assert.isAtLeast(obj.results.length, 9, 'results must have at least 9 SERP objects');
- assert.isAtLeast(obj.ads.length, 3, 'there are 3 ads');
-
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- confirm_results_ok(obj);
- }
- }
-}
-
-
-function confirm_results_ok(obj) {
-
- for (let res of obj.results) {
- assert.containsAllKeys(res, ['link', 'title', 'rank', 'visible_link', 'snippet'], 'not all keys are in the SERP object');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'visible_link must be ok');
- assert.typeOf(res.visible_link, 'string', 'visible_link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'visible_link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 8, 'title must have at least 8 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
-
- assert.isNumber(res.rank, 'rank must be integer');
- }
-
- for (let res of obj.ads) {
-
- assert.isOk(res.tracking_link, 'link must be ok');
- assert.typeOf(res.tracking_link, 'string', 'link must be string');
- assert.isAtLeast(res.tracking_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'link must be ok');
- assert.typeOf(res.visible_link, 'string', 'link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 8, 'title must have at least 8 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
- }
-
- for (let res of obj.right_side_ads) {
-
- assert.isOk(res.tracking_link, 'link must be ok');
- assert.typeOf(res.tracking_link, 'string', 'link must be string');
- assert.isAtLeast(res.tracking_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'link must be ok');
- assert.typeOf(res.visible_link, 'string', 'link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 8, 'title must have at least 8 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
- }
-}
-
-describe('Bing', function(){
- this.timeout(15000);
- it('static bing searches with ads', bing_ads);
-});
\ No newline at end of file
diff --git a/test/static_tests/clean_html_test.js b/test/static_tests/clean_html_test.js
deleted file mode 100644
index 6bbe4dc..0000000
--- a/test/static_tests/clean_html_test.js
+++ /dev/null
@@ -1,173 +0,0 @@
-const se_scraper = require('./../../index.js');
-const chai = require('chai');
-chai.use(require('chai-string'));
-const assert = chai.assert;
-const path = require('path');
-const cheerio = require('cheerio');
-
-
-async function test_html_output() {
- let config = {
- debug_level: 1,
- headless: true,
- html_output: true,
- // whether to strip JS and CSS from the html_output
- // has only an effect if `html_output` is true
- clean_html_output: true,
- // remove all data images from the html
- clean_data_images: true,
- // test compression
- compress: false,
- };
-
- let scrape_config = {
- search_engine: 'bing',
- keywords: ['kaffeemaschine kaufen'],
- num_pages: 1,
- scrape_from_file: 'file://' + path.join(__dirname, './html/bing.html'),
- };
-
- var scraper = new se_scraper.ScrapeManager(config);
-
- await scraper.start();
-
- var response = await scraper.scrape(scrape_config);
-
- scrape_config.clean_html_output = false;
- scrape_config.clean_data_images = false;
-
- var response_no_cleaned = await scraper.scrape(scrape_config);
-
- test(response, response_no_cleaned, 'bing');
-
- scrape_config.search_engine = 'google';
- scrape_config.keywords = ['rückspiegel schwarz'];
- scrape_config.scrape_from_file = 'file://' + path.join(__dirname, './html/google.html');
- scrape_config.clean_html_output = true;
- scrape_config.clean_data_images = true;
-
- var responseGoogle = await scraper.scrape(scrape_config);
-
- scrape_config.clean_html_output = false;
- scrape_config.clean_data_images = false;
-
- var response_no_cleanedGoogle = await scraper.scrape(scrape_config);
-
- test(responseGoogle, response_no_cleanedGoogle, 'google');
-
-
- scrape_config.keywords = ['cloud services'];
- scrape_config.scrape_from_file = 'file://' + path.join(__dirname, './html/googleLarge.html');
- scrape_config.clean_html_output = true;
- scrape_config.clean_data_images = true;
-
- var responseGoogle = await scraper.scrape(scrape_config);
-
- scrape_config.clean_html_output = false;
- scrape_config.clean_data_images = false;
-
- var response_no_cleanedGoogle = await scraper.scrape(scrape_config);
-
- test(responseGoogle, response_no_cleanedGoogle, 'google');
-
- await scraper.quit();
-}
-
-function test(response, response_no_cleaned, se='google') {
- for (let query in response.results) {
- for (let page_number in response.results[query]) {
- let obj = response.results[query][page_number];
- let obj_no_cleaned = response_no_cleaned.results[query][page_number];
-
- console.log('html length of no cleaned SERP: ' + obj_no_cleaned.html.length);
- console.log('html length of cleaned SERP: ' + obj.html.length);
-
- assert.isOk(obj.html, 'Html must be ok!');
- assert.isAtLeast(obj.html.length, 100, 'html must be a length string');
-
- assert.isOk(obj_no_cleaned.html, 'Html must be ok!');
- assert.isAtLeast(obj_no_cleaned.html.length, 100, 'html must be a length string');
-
- assert.isBelow(obj.html.length, obj_no_cleaned.html.length, 'cleaned html must be smaller');
-
- // test that we can parse the html of both the cleaned and no cleaned versions
- // with cheerio and that serp results are roughly the same
-
- const cleaned$ = cheerio.load(obj.html);
- const no_cleaned$ = cheerio.load(obj_no_cleaned.html);
-
- var resCleaned = parseResults(cleaned$, se);
- var resNoCleaned = parseResults(no_cleaned$, se);
-
- assert.equal(resCleaned.length, resNoCleaned.length);
- assert.equal(resCleaned.length, obj.results.length);
- assert.equal(resNoCleaned.length, obj.results.length);
-
- // unset the rank
- resCleaned = resCleaned.map((el) => el.rank = undefined);
- resNoCleaned = resNoCleaned.map((el) => el.rank = undefined);
- obj.results = obj.results.map((el) => el.rank = undefined);
-
- assert.deepEqual(resCleaned, resNoCleaned, 'parsed results should be equal, even if html is cleaned');
- assert.deepEqual(resCleaned, obj.results, 'parsed results from cleaned html should be equal to se-scraper results');
- assert.deepEqual(resNoCleaned, obj.results, 'parsed results from non-cleaned html should be equal to se-scraper results');
- }
- }
-}
-
-
-function parseResults(s$, se) {
-
- var results = [];
-
- if (se === 'google') {
- s$('#center_col .g').each((i, link) => {
- results.push({
- link: s$(link).find('.r a').attr('href'),
- title: s$(link).find('.r a').text(),
- snippet: s$(link).find('span.st').text(),
- visible_link: s$(link).find('.r cite').text(),
- date: s$(link).find('span.f').text() || '',
- })
- });
-
- } else if (se === 'bing') {
- s$('#b_content #b_results .b_algo').each((i, link) => {
- results.push({
- link: s$(link).find('h2 a').attr('href'),
- title: s$(link).find('h2').text(),
- snippet: s$(link).find('.b_caption p').text(),
- visible_link: s$(link).find('cite').text(),
- })
- });
- } else {
- throw "no such search engine";
- }
-
- results = clean_results(results, ['title', 'link', 'snippet']);
- return results;
-}
-
-function clean_results(results, attributes) {
- const cleaned = [];
- var rank = 1;
- for (var res of results) {
- let goodboy = true;
- for (var attr of attributes) {
- if (!res[attr] || !res[attr].trim()) {
- goodboy = false;
- break;
- }
- }
- if (goodboy) {
- res.rank = rank++;
- cleaned.push(res);
- }
- }
- return cleaned;
-}
-
-describe('html output', function(){
- this.timeout(15000);
- it('static html output test', test_html_output);
-});
\ No newline at end of file
diff --git a/test/static_tests/compression.js b/test/static_tests/compression.js
deleted file mode 100644
index a41dba8..0000000
--- a/test/static_tests/compression.js
+++ /dev/null
@@ -1,24 +0,0 @@
-'use strict';
-const zlib = require('zlib');
-const fs = require('fs');
-const path = require('path');
-
-var files = ['google.html', 'google2.html', 'google3.html', 'bing.html', 'bing2.html'];
-
-for (var file of files) {
- var html = fs.readFileSync(path.resolve(__dirname, './html/' + file));
-
- var compressed = zlib.gzipSync(html);
- var deflated = zlib.deflateSync(html);
-
- var compressed_encoded = compressed.toString('base64');
- var deflated_encoded = deflated.toString('base64');
-
- console.log(file)
- console.log('Normal length: ' + html.length/1000);
- console.log('GZIP Compressed length: ' + compressed.length/1000);
- console.log('Deflate Compressed length: ' + deflated.length/1000);
- console.log('Encoded GZIP Compressed length: ' + compressed_encoded.length/1000);
- console.log('Encoded Deflate Compressed length: ' + deflated_encoded.length/1000);
- console.log('------\n')
-}
diff --git a/test/static_tests/duckduckgo.js b/test/static_tests/duckduckgo.js
deleted file mode 100644
index f0f0834..0000000
--- a/test/static_tests/duckduckgo.js
+++ /dev/null
@@ -1,99 +0,0 @@
-const se_scraper = require('./../../index.js');
-const chai = require('chai');
-chai.use(require('chai-string'));
-const assert = chai.assert;
-const path = require('path');
-
-async function duckduckgo() {
- let config = {
- compress: false,
- debug_level: 1,
- headless: true,
- };
-
- let scrape_config = {
- search_engine: 'duckduckgo',
- keywords: ['cloud service'],
- num_pages: 1,
- scrape_from_file: 'file://' + path.join(__dirname, './html/duckduckgo1.html'),
- };
-
- var scraper = new se_scraper.ScrapeManager(config);
-
- await scraper.start();
-
- duckduckgo_normal( await scraper.scrape(scrape_config) );
-
- await scraper.quit();
-}
-
-function duckduckgo_normal(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- for (let query in response.results) {
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time', 'ads',], 'not all keys are in the object');
- assert.isAtLeast(obj.results.length, 10, 'results must have at least 10 SERP objects');
- assert.isAtLeast(obj.ads.length, 2, 'ads must have at least 2 SERP objects');
-
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- confirm_results_ok(obj);
- }
- }
-}
-
-function confirm_results_ok(obj) {
-
- for (let res of obj.results) {
- assert.containsAllKeys(res, ['link', 'title', 'rank', 'visible_link', 'snippet'], 'not all keys are in the SERP object');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'visible_link must be ok');
- assert.typeOf(res.visible_link, 'string', 'visible_link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'visible_link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
-
- assert.isNumber(res.rank, 'rank must be integer');
- }
-
- for (let res of obj.ads) {
-
- assert.isOk(res.tracking_link, 'link must be ok');
- assert.typeOf(res.tracking_link, 'string', 'link must be string');
- assert.isAtLeast(res.tracking_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'link must be ok');
- assert.typeOf(res.visible_link, 'string', 'link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
- }
-}
-
-describe('Duckduckgo', function(){
- this.timeout(10000);
- it('static duckduckgo sarch', duckduckgo);
-});
\ No newline at end of file
diff --git a/test/static_tests/google.js b/test/static_tests/google.js
deleted file mode 100644
index fd9e154..0000000
--- a/test/static_tests/google.js
+++ /dev/null
@@ -1,410 +0,0 @@
-const se_scraper = require('./../../index.js');
-const chai = require('chai');
-chai.use(require('chai-string'));
-const assert = chai.assert;
-const path = require('path');
-
-async function normal_search_test() {
- let config = {
- compress: false,
- debug_level: 1,
- headless: true,
- };
-
- let scrape_config = {
- search_engine: 'google',
- keywords: ['rückspiegel schwarz'],
- num_pages: 1,
- scrape_from_file: 'file://' + path.join(__dirname, './html/google.html'),
- };
-
- var scraper = new se_scraper.ScrapeManager(config);
-
- await scraper.start();
-
- google_search_with_products( await scraper.scrape(scrape_config) );
-
- scrape_config.scrape_from_file = 'file://' + path.join(__dirname, './html/google2.html');
- scrape_config.keywords = ['autoreifen mercedes c-klasse'];
-
- google_search_with_products2( await scraper.scrape(scrape_config) );
-
- scrape_config.scrape_from_file = 'file://' + path.join(__dirname, './html/google3.html');
- scrape_config.keywords = ['kaffeemaschine kaufen'];
-
- google_places( await scraper.scrape(scrape_config) );
-
- scrape_config.scrape_from_file = 'file://' + path.join(__dirname, './html/google4.html');
- scrape_config.keywords = ['MODEL MARKET SW18 4ES'];
-
- right_side_info_text( await scraper.scrape(scrape_config) );
-
- scrape_config.scrape_from_file = 'file://' + path.join(__dirname, './html/google5.html');
- scrape_config.keywords = ['BRANDON MOTORS HP13 6NR'];
-
- right_side_info_text2( await scraper.scrape(scrape_config) );
-
- scrape_config.scrape_from_file = 'file://' + path.join(__dirname, './html/google6.html');
- scrape_config.keywords = ['car tires for sale'];
-
- google_places_and_ads( await scraper.scrape(scrape_config) );
-
- scrape_config.scrape_from_file = 'file://' + path.join(__dirname, './html/google_bmw_felgen.html');
- scrape_config.keywords = ['bmw felgen'];
-
- google_ads2( await scraper.scrape(scrape_config) );
-
- await scraper.quit();
-}
-
-// we test with a callback function to our handler
-function google_search_with_products(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- for (let query in response.results) {
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.include(obj.num_results, '1’780’000', 'num results not included');
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query', 'top_ads', 'bottom_ads', 'places'], 'not all keys are in the object');
- assert.isAtLeast(obj.results.length, 9, 'results must have at least 8 SERP objects');
- assert.isAtLeast(obj.top_ads.length, 0, 'there are no top ads');
- assert.isAtLeast(obj.bottom_ads.length, 3, 'there are 3 bottom ads');
- assert.isAtLeast(obj.top_products.length, 15, 'there are 15 top products');
- assert.equal(obj.right_products.length, 0, 'there are 0 right products');
-
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- confirm_results_ok(obj);
-
- }
- }
-}
-
-
-function google_search_with_products2(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- for (let query in response.results) {
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.include(obj.num_results, '437’000 Ergebnisse (0.41 Sekunden)', 'num results not included');
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query', 'top_ads', 'bottom_ads', 'places'], 'not all keys are in the object');
- assert.isAtLeast(obj.results.length, 9, 'results must have at least 8 SERP objects');
- assert.isAtLeast(obj.top_ads.length, 0, 'there are no top ads');
- assert.isAtLeast(obj.bottom_ads.length, 1, 'there are 1 bottom ads');
- assert.isAtLeast(obj.top_products.length, 0, 'there are 0 top products');
- assert.equal(obj.right_products.length, 4, 'there are 4 right products');
-
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- confirm_results_ok(obj);
-
- }
- }
-}
-
-function google_places(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- for (let query in response.results) {
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.include(obj.num_results, '6’750’000 Ergebnisse (0.52 Sekunden)', 'num results not included');
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query', 'top_ads', 'bottom_ads', 'places'], 'not all keys are in the object');
- assert.isAtLeast(obj.results.length, 10, 'results must have at least 9 SERP objects');
- assert.equal(obj.top_ads.length, 0, 'there are no top ads');
- assert.equal(obj.bottom_ads.length, 0, 'there are 0 bottom ads');
- assert.equal(obj.top_products.length, 0, 'there are 0 top products');
- assert.equal(obj.right_products.length, 0, 'there are 0 right products');
- assert.equal(obj.places.length, 3, 'there are 3 places');
-
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- confirm_results_ok(obj);
- }
- }
-}
-
-function right_side_info_text(response) {
- assert.equal(response.metadata.num_requests, 1);
- for (let query in response.results) {
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.include(obj.num_results, '6 Ergebnisse', 'num results not included');
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results',
- 'effective_query', 'top_ads', 'bottom_ads', 'right_side_info_text'], 'not all keys are in the object');
-
- assert.isAtLeast(obj.results.length, 7, 'results must have at least 7 SERP objects');
-
- assert.isOk(obj.right_side_info_text.length, 'right_side_info_text should have some data');
- assert.isAtLeast(obj.right_side_info_text.length, 50, 'right_side_info_text should have some data');
-
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- confirm_results_ok(obj);
- }
- }
-}
-
-function right_side_info_text2(response) {
- assert.equal(response.metadata.num_requests, 1);
- for (let query in response.results) {
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.include(obj.num_results, '5 Ergebnisse', 'num results not included');
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results',
- 'effective_query', 'top_ads', 'bottom_ads', 'right_side_info_text'], 'not all keys are in the object');
-
- assert.isAtLeast(obj.results.length, 5, 'results must have at least 5 SERP objects');
- assert.isOk(obj.right_side_info_text.length, 'right_side_info_text should have some data');
- assert.isAtLeast(obj.right_side_info_text.length, 50, 'right_side_info_text should have some data');
-
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- confirm_results_ok(obj);
- }
- }
-}
-
-function google_places_and_ads(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- for (let query in response.results) {
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.include(obj.num_results, '439.000.000 Ergebnisse (0,64 Sekunden)', 'num results not included');
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query', 'top_ads', 'bottom_ads', 'places'], 'not all keys are in the object');
- assert.isAtLeast(obj.results.length, 10, 'results must have at least 10 SERP objects');
- assert.equal(obj.top_ads.length, 0, 'there are no top ads');
- assert.equal(obj.bottom_ads.length, 0, 'there are 0 bottom ads');
- assert.isAtLeast(obj.top_products.length, 13, 'there are 13 top products');
- assert.equal(obj.right_products.length, 0, 'there are 0 right products');
- assert.equal(obj.places.length, 2, 'there are 2 places');
-
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- confirm_results_ok(obj);
- }
- }
-}
-
-
-function google_ads2(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- for (let query in response.results) {
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.include(obj.num_results, 'Ungefähr 23.200.000 Ergebnisse (0,29 Sekunden)', 'num results not included');
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query', 'top_ads', 'bottom_ads', 'places'], 'not all keys are in the object');
- assert.isAtLeast(obj.results.length, 10, 'results must have at least 10 SERP objects');
- assert.equal(obj.top_ads.length, 3, 'there are no top ads');
- assert.equal(obj.bottom_ads.length, 0, 'there are 0 bottom ads');
- assert.isAtLeast(obj.top_products.length, 0, 'there must be 0 top products');
- assert.equal(obj.right_products.length, 9, 'there are 9 right products');
- assert.equal(obj.places.length, 0, 'there are 0 places');
-
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- confirm_results_ok(obj);
- }
- }
-}
-
-
-function confirm_results_ok(obj) {
-
- for (let res of obj.results) {
- assert.containsAllKeys(res, ['link', 'title', 'rank', 'visible_link', 'snippet'], 'not all keys are in the SERP object');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'visible_link must be ok');
- assert.typeOf(res.visible_link, 'string', 'visible_link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'visible_link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
-
- assert.isNumber(res.rank, 'rank must be integer');
- }
-
- for (let res of obj.top_ads) {
-
- assert.isOk(res.tracking_link, 'link must be ok');
- assert.typeOf(res.tracking_link, 'string', 'link must be string');
- assert.isAtLeast(res.tracking_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'link must be ok');
- assert.typeOf(res.visible_link, 'string', 'link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.link, 'visible_link must be ok');
- assert.typeOf(res.link, 'string', 'visible_link must be string');
- assert.isAtLeast(res.link.length, 5, 'visible_link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
-
- assert.typeOf(res.links, 'array', 'links must be array');
- }
-
- for (let res of obj.bottom_ads) {
- assert.isOk(res.tracking_link, 'link must be ok');
- assert.typeOf(res.tracking_link, 'string', 'link must be string');
- assert.isAtLeast(res.tracking_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'link must be ok');
- assert.typeOf(res.visible_link, 'string', 'link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.link, 'visible_link must be ok');
- assert.typeOf(res.link, 'string', 'visible_link must be string');
- assert.isAtLeast(res.link.length, 5, 'visible_link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
-
- assert.typeOf(res.links, 'array', 'links must be array');
- }
-
- for (let res of obj.top_products) {
-
- assert.isOk(res.tracking_link, 'link must be ok');
- assert.typeOf(res.tracking_link, 'string', 'link must be string');
- assert.isAtLeast(res.tracking_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.price, 'price must be ok');
- assert.typeOf(res.price, 'string', 'price must be string');
- assert.isAtLeast(res.price.length, 5, 'price must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.vendor_link, 'vendor_link must be ok');
- assert.typeOf(res.vendor_link, 'string', 'vendor_link must be string');
- assert.isAtLeast(res.vendor_link.length, 8, 'vendor_link must have at least 10 chars');
- }
-
- for (let res of obj.right_products) {
- assert.isOk(res.tracking_link, 'link must be ok');
- assert.typeOf(res.tracking_link, 'string', 'link must be string');
- assert.isAtLeast(res.tracking_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.price, 'price must be ok');
- assert.typeOf(res.price, 'string', 'price must be string');
- assert.isAtLeast(res.price.length, 5, 'price must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.vendor_link, 'vendor_link must be ok');
- assert.typeOf(res.vendor_link, 'string', 'vendor_link must be string');
- assert.isAtLeast(res.vendor_link.length, 8, 'vendor_link must have at least 10 chars');
- }
-
- for (let res of obj.places) {
- assert.isOk(res.heading, 'heading must be ok');
- assert.typeOf(res.heading, 'string', 'heading must be string');
- assert.isAtLeast(res.heading.length, 5, 'heading must have at least 5 chars');
-
- assert.isOk(res.rating, 'rating must be ok');
- assert.typeOf(res.rating, 'string', 'rating must be string');
- assert.isAtLeast(res.rating.length, 5, 'rating must have at least 5 chars');
-
- assert.isOk(res.contact, 'contact must be ok');
- assert.typeOf(res.contact, 'string', 'contact must be string');
- assert.isAtLeast(res.contact.length, 5, 'contact must have at least 5 chars');
-
- assert.typeOf(res.hours, 'string', 'hours must be string');
- if (res.hours) {
- assert.isAtLeast(res.hours.length, 10, 'hours must have at least 10 chars');
- }
- }
-}
-
-describe('Google', function() {
- this.timeout(25000);
- it('static google searches with products,ads and places', normal_search_test);
-});
\ No newline at end of file
diff --git a/test/static_tests/second_google.js b/test/static_tests/second_google.js
deleted file mode 100644
index 9fd95b8..0000000
--- a/test/static_tests/second_google.js
+++ /dev/null
@@ -1,213 +0,0 @@
-const se_scraper = require('./../../index.js');
-const chai = require('chai');
-chai.use(require('chai-string'));
-const assert = chai.assert;
-const path = require('path');
-
-async function normal_search_test() {
- let config = {
- compress: false,
- debug_level: 1,
- headless: true,
- };
-
- let scrape_config = {
- search_engine: 'google',
- keywords: ['in.linkedin.com/in/altanai'],
- num_pages: 1,
- scrape_from_file: 'file://' + path.join(__dirname, './html/google7.html'),
- };
-
- var scraper = new se_scraper.ScrapeManager(config);
-
- await scraper.start();
-
- google_test_title( await scraper.scrape(scrape_config) );
-
- await scraper.quit();
-}
-
-// we test with a callback function to our handler
-function google_test_title(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- for (let query in response.results) {
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.include(obj.num_results, '7.600', 'num results not included');
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query', 'top_ads', 'bottom_ads', 'places'], 'not all keys are in the object');
- assert.isAtLeast(obj.results.length, 9, 'results must have at least 9 SERP objects');
- assert.isAtLeast(obj.top_ads.length, 0, 'there are no top ads');
- assert.isAtLeast(obj.bottom_ads.length, 0, 'there are 0 bottom ads');
- assert.isAtLeast(obj.top_products.length, 0, 'there are 0 top products');
- assert.equal(obj.right_products.length, 0, 'there are 0 right products');
-
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- confirm_results_ok(obj);
-
- assert.equal( obj.results[0].title, 'ALTANAI BISHT - SD2 at Voice Engineering - Plivo | LinkedIn' );
- assert.equal( obj.results[1].title, 'ALTANAI BISHT | LinkedIn' );
- assert.equal( obj.results[2].title, 'ALTANAI BISHT – SD2 at Voice Engineering – Plivo | LinkedIn' );
- assert.equal( obj.results[3].title, 'AI AT - South Delhi, Delhi, India | Professional Profile | LinkedIn' );
- assert.equal( obj.results[4].title, 'ALTANAI BISHT | LinkedIn' );
- assert.equal( obj.results[9].title, 'Phani Kumar Parasaram - VOIP Expert - Infinite ... - LinkedIn');
-
- assert.equal (obj.results[0].date, '27.07.2016');
- assert.equal( obj.results[0].snippet, '27.07.2016 - View ALTANAI BISHT\'S profile on LinkedIn, the world\'s largest professional community. ALTANAI has 6 jobs listed on their profile. See the ...');
-
- assert.equal (obj.results[2].date, '27.07.2016');
- }
- }
-}
-
-function confirm_results_ok(obj) {
-
- for (let res of obj.results) {
- assert.containsAllKeys(res, ['link', 'title', 'rank', 'visible_link', 'snippet'], 'not all keys are in the SERP object');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'visible_link must be ok');
- assert.typeOf(res.visible_link, 'string', 'visible_link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'visible_link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
-
- assert.isNumber(res.rank, 'rank must be integer');
- }
-
- for (let res of obj.top_ads) {
-
- assert.isOk(res.tracking_link, 'link must be ok');
- assert.typeOf(res.tracking_link, 'string', 'link must be string');
- assert.isAtLeast(res.tracking_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'link must be ok');
- assert.typeOf(res.visible_link, 'string', 'link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.link, 'visible_link must be ok');
- assert.typeOf(res.link, 'string', 'visible_link must be string');
- assert.isAtLeast(res.link.length, 5, 'visible_link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
-
- assert.typeOf(res.links, 'array', 'links must be array');
- }
-
- for (let res of obj.bottom_ads) {
- assert.isOk(res.tracking_link, 'link must be ok');
- assert.typeOf(res.tracking_link, 'string', 'link must be string');
- assert.isAtLeast(res.tracking_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'link must be ok');
- assert.typeOf(res.visible_link, 'string', 'link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.link, 'visible_link must be ok');
- assert.typeOf(res.link, 'string', 'visible_link must be string');
- assert.isAtLeast(res.link.length, 5, 'visible_link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
-
- assert.typeOf(res.links, 'array', 'links must be array');
- }
-
- for (let res of obj.top_products) {
-
- assert.isOk(res.tracking_link, 'link must be ok');
- assert.typeOf(res.tracking_link, 'string', 'link must be string');
- assert.isAtLeast(res.tracking_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.price, 'price must be ok');
- assert.typeOf(res.price, 'string', 'price must be string');
- assert.isAtLeast(res.price.length, 5, 'price must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.vendor_link, 'vendor_link must be ok');
- assert.typeOf(res.vendor_link, 'string', 'vendor_link must be string');
- assert.isAtLeast(res.vendor_link.length, 10, 'vendor_link must have at least 10 chars');
- }
-
- for (let res of obj.right_products) {
- assert.isOk(res.tracking_link, 'link must be ok');
- assert.typeOf(res.tracking_link, 'string', 'link must be string');
- assert.isAtLeast(res.tracking_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.price, 'price must be ok');
- assert.typeOf(res.price, 'string', 'price must be string');
- assert.isAtLeast(res.price.length, 5, 'price must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.vendor_link, 'vendor_link must be ok');
- assert.typeOf(res.vendor_link, 'string', 'vendor_link must be string');
- assert.isAtLeast(res.vendor_link.length, 10, 'vendor_link must have at least 10 chars');
- }
-
- for (let res of obj.places) {
- assert.isOk(res.heading, 'heading must be ok');
- assert.typeOf(res.heading, 'string', 'heading must be string');
- assert.isAtLeast(res.heading.length, 5, 'heading must have at least 5 chars');
-
- assert.isOk(res.rating, 'rating must be ok');
- assert.typeOf(res.rating, 'string', 'rating must be string');
- assert.isAtLeast(res.rating.length, 5, 'rating must have at least 5 chars');
-
- assert.isOk(res.contact, 'contact must be ok');
- assert.typeOf(res.contact, 'string', 'contact must be string');
- assert.isAtLeast(res.contact.length, 5, 'contact must have at least 5 chars');
-
- assert.typeOf(res.hours, 'string', 'hours must be string');
- if (res.hours) {
- assert.isAtLeast(res.hours.length, 10, 'hours must have at least 10 chars');
- }
- }
-}
-
-describe('Google2', function(){
- this.timeout(10000);
- it('static google searches testing various details', normal_search_test);
-});
\ No newline at end of file
diff --git a/test/static_tests/yandex.js b/test/static_tests/yandex.js
deleted file mode 100644
index 9cb806c..0000000
--- a/test/static_tests/yandex.js
+++ /dev/null
@@ -1,152 +0,0 @@
-const se_scraper = require('./../../index.js');
-const chai = require('chai');
-chai.use(require('chai-string'));
-const assert = chai.assert;
-const path = require('path');
-
-async function yandex_ads() {
- let config = {
- compress: false,
- debug_level: 1,
- headless: true,
- };
-
- let scrape_config = {
- search_engine: 'yandex',
- keywords: ['cloud service'],
- num_pages: 1,
- scrape_from_file: 'file://' + path.join(__dirname, './html/yandex1.html'),
- };
-
- var scraper = new se_scraper.ScrapeManager(config);
-
- await scraper.start();
-
- yandex_search_with_ads( await scraper.scrape(scrape_config) );
-
- scrape_config.keywords = ['car tires cheap'];
- scrape_config.scrape_from_file = 'file://' + path.join(__dirname, './html/yandex2.html');
-
- yandex_search_with_ads2( await scraper.scrape(scrape_config) );
-
- scrape_config.keywords = ['купить деревянные окна'];
- scrape_config.scrape_from_file = 'file://' + path.join(__dirname, './html/yandex3.html');
-
- yandex_search_with_ads3( await scraper.scrape(scrape_config) );
-
- await scraper.quit();
-}
-
-// we test with a callback function to our handler
-function yandex_search_with_ads(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- for (let query in response.results) {
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.include(obj.num_results, '2 million results', 'num results not included');
- assert.containsAllKeys(obj, ['results', 'time', 'num_results',], 'not all keys are in the object');
- assert.isAtLeast(obj.results.length, 12, 'results must have at least 12 SERP objects');
-
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- confirm_results_ok(obj);
- }
- }
-}
-
-
-function yandex_search_with_ads2(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- for (let query in response.results) {
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.include(obj.num_results, '5 million results', 'num results not included');
- assert.containsAllKeys(obj, ['results', 'time', 'num_results',], 'not all keys are in the object');
- assert.isAtLeast(obj.results.length, 11, 'results must have at least 12 SERP objects');
-
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- confirm_results_ok(obj);
- }
- }
-}
-
-
-function yandex_search_with_ads3(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- for (let query in response.results) {
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- // console.dir(obj.results, {depth: null, colors: true});
-
- assert.containsAllKeys(obj, ['results', 'time', 'num_results',], 'not all keys are in the object');
- assert.isAtLeast(obj.results.length, 14, 'results must have at least 14 SERP objects');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- // at least 4 ads
- let cnt = 0;
- obj.results.forEach((res) => {
- if (res.is_ad) {
- cnt++;
- }
- });
-
- assert.isAtLeast(cnt, 4, 'there should be at least 4 ads in the results');
-
- confirm_results_ok(obj);
- }
- }
-}
-
-
-function confirm_results_ok(obj) {
-
- for (let res of obj.results) {
- assert.containsAllKeys(res, ['link', 'title', 'rank', 'visible_link', 'snippet'], 'not all keys are in the SERP object');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'visible_link must be ok');
- assert.typeOf(res.visible_link, 'string', 'visible_link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'visible_link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
-
- assert.isNumber(res.rank, 'rank must be integer');
- }
-}
-
-describe('Yandex', function(){
- this.timeout(10000);
- it('static yandex searches with ads', yandex_ads);
-});
\ No newline at end of file
diff --git a/test/test_amazon.js b/test/test_amazon.js
deleted file mode 100644
index aa38e82..0000000
--- a/test/test_amazon.js
+++ /dev/null
@@ -1,141 +0,0 @@
-'use strict';
-const se_scraper = require('./../index.js');
-const assert = require('chai').assert;
-
-/*
- * Use chai and mocha for tests.
- * https://mochajs.org/#installation
- */
-
-const normal_search_keywords = ['iphone', 'clock'];
-
-async function normal_search_test() {
- let config = {
- compress: false,
- keyword_file: '',
- headless: true,
- output_file: '',
- block_assets: true,
- random_user_agent: false,
- };
-
- let scrape_config = {
- search_engine: 'amazon',
- num_pages: 1,
- keywords: normal_search_keywords,
- };
-
- console.log('normal_search_test()');
- normal_search_test_case( await se_scraper.scrape(config, scrape_config) );
-}
-
-// we test with a callback function to our handler
-function normal_search_test_case(response) {
- assert.equal(response.metadata.num_requests, 2);
-
- for (let query in response.results) {
- let total_rank = 1;
- assert.containsAllKeys(response.results, normal_search_keywords, 'not all keywords were scraped.');
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query'], 'not all keys are in the object');
-
- assert.isAtLeast(obj.results.length, 7, 'results must have at least 7 SERP objects');
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- for (let res of obj.results) {
-
- assert.containsAllKeys(res, ['link', 'title', 'rank', 'image', 'seller', 'stars', 'num_reviews', 'price', 'oldprice'], 'not all keys are in the SERP object');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 5, 'title must have at least 5 chars');
-
- assert.isOk(res.seller, 'seller must be ok');
- assert.typeOf(res.seller, 'string', 'seller must be string');
- assert.isAtLeast(res.seller.length, 5, 'seller must have at least 10 chars');
-
- assert.isOk(res.stars, 'stars be ok');
- assert.typeOf(res.stars, 'string', 'stars must be string');
- assert.isAtLeast(res.stars.length, 5, 'stars must have at least 6 chars');
- assert.include(res.stars, ' out of ', 'stars must include " out of "');
-
- assert.isOk(res.num_reviews, 'num_reviews be ok');
- assert.typeOf(res.num_reviews, 'string', 'num_reviews must be string');
- assert.isAtLeast(res.num_reviews.length, 1, 'num_reviews must have at least 1 chars');
-
- assert.isOk(res.price, 'price be ok');
- assert.typeOf(res.price, 'string', 'price must be string');
- assert.isAtLeast(res.price.length, 5, 'price must have at least 5 chars');
-
- assert.isNumber(res.rank, 'rank must be integer');
- assert.equal(res.rank, total_rank++, 'rank ist wrong');
- }
- }
- }
-}
-
-const keywords_no_results = ['2342kljp;fj9834u40abJ54634344023safkl34a44dsflkjaQQuBBdfk',];
-
-async function no_results_test() {
- let config = {
- compress: false,
- debug_level: 1,
- keyword_file: '',
- headless: true,
- output_file: '',
- block_assets: true,
- random_user_agent: false,
- };
-
- let scrape_config = {
- search_engine: 'amazon',
- num_pages: 1,
- keywords: keywords_no_results,
- };
-
- console.log('no_results_test()');
- test_case_no_results( await se_scraper.scrape(config, scrape_config) );
-}
-
-// we test with a callback function to our handler
-function test_case_no_results(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- for (let query in response.results) {
-
- assert.containsAllKeys(response.results, keywords_no_results, 'not all keywords were scraped.');
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query'], 'not all keys are in the object');
-
- assert(obj.results.length === 0, 'results must have 0 SERP objects');
- assert.equal(obj.no_results, true, 'no results should be true');
- assert.isEmpty(obj.num_results, 'no results should be a empty string');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
- }
- }
-}
-
-describe('Amazon', function(){
- this.timeout(30000);
- it('normal search test', normal_search_test);
- it('no results test', no_results_test);
-});
\ No newline at end of file
diff --git a/test/test_baidu.js b/test/test_baidu.js
deleted file mode 100644
index 887d7d1..0000000
--- a/test/test_baidu.js
+++ /dev/null
@@ -1,87 +0,0 @@
-'use strict';
-const se_scraper = require('./../index.js');
-const assert = require('chai').assert;
-
-/*
- * Use chai and mocha for tests.
- * https://mochajs.org/#installation
- */
-
-const normal_search_keywords = ['mouse', 'cat'];
-
-async function normal_search_test() {
- let config = {
- compress: false,
- debug_level: 1,
- keyword_file: '',
- headless: true,
- block_assets: true,
- user_agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36',
- };
-
- let scrape_config = {
- search_engine: 'baidu',
- keywords: normal_search_keywords,
- num_pages: 2,
- };
-
- console.log('normal_search_test()');
- normal_search_test_case( await se_scraper.scrape(config, scrape_config) );
-}
-
-// we test with a callback function to our handler
-function normal_search_test_case(response) {
- assert.equal(response.metadata.num_requests, 4);
-
- for (let query in response.results) {
- let total_rank = 1;
-
- assert.containsAllKeys(response.results, normal_search_keywords, 'not all keywords were scraped.');
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time', 'num_results', 'no_results'], 'not all keys are in the object');
-
- assert.isAtLeast(obj.results.length, 7, 'results must have at least 7 SERP objects');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
-
- assert.equal(obj.no_results, false, 'no results should be false');
-
- for (let res of obj.results) {
-
- assert.containsAllKeys(res, ['link', 'title', 'rank', 'visible_link', 'snippet'], 'not all keys are in the SERP object');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'visible_link must be ok');
- assert.typeOf(res.visible_link, 'string', 'visible_link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'visible_link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 5, 'title must have at least 5 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
-
- assert.isNumber(res.rank, 'rank must be integer');
- assert.equal(res.rank, total_rank++, 'rank ist wrong');
- }
- }
- }
-}
-
-describe('Baidu', function(){
- this.timeout(30000);
- it('normal search test', normal_search_test);
-});
\ No newline at end of file
diff --git a/test/test_bing.js b/test/test_bing.js
deleted file mode 100644
index f62c56a..0000000
--- a/test/test_bing.js
+++ /dev/null
@@ -1,271 +0,0 @@
-'use strict';
-const se_scraper = require('./../index.js');
-const chai = require('chai');
-chai.use(require('chai-string'));
-const assert = chai.assert;
-/*
- * Use chai and mocha for tests.
- * https://mochajs.org/#installation
- */
-
-const normal_search_keywords = ['apple tree', 'weather tomorrow'];
-
-async function normal_search_test() {
- let config = {
- search_engine: 'bing',
- compress: false,
- debug_level: 1,
- keywords: normal_search_keywords,
- keyword_file: '',
- num_pages: 3,
- headless: true,
- output_file: '',
- block_assets: true,
- user_agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36',
- random_user_agent: false,
- };
-
- let scrape_config = {
- search_engine: 'bing',
- keywords: normal_search_keywords,
- num_pages: 3,
- };
-
- console.log('normal_search_test()');
- normal_search_test_case( await se_scraper.scrape(config, scrape_config) );
-}
-
-// we test with a callback function to our handler
-function normal_search_test_case(response) {
- assert.equal(response.metadata.num_requests, 6);
-
- for (let query in response.results) {
- let total_rank = 1;
- assert.containsAllKeys(response.results, normal_search_keywords, 'not all keywords were scraped.');
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query'], 'not all keys are in the object');
-
- assert.isAtLeast(obj.results.length, 7, 'results must have at least 7 SERP objects');
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- for (let res of obj.results) {
-
- assert.containsAllKeys(res, ['link', 'title', 'rank', 'visible_link', 'rank'], 'not all keys are in the SERP object');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'visible_link must be ok');
- assert.typeOf(res.visible_link, 'string', 'visible_link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'visible_link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 5, 'title must have at least 5 chars');
-
- if (res.snippet) {
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
- }
-
- assert.isNumber(res.rank, 'rank must be integer');
- assert.equal(res.rank, total_rank++, 'rank ist wrong');
- }
- }
- }
-}
-
-const keywords_no_results = ['2342kljp;fj9834u40abJAkasdlfkjsladfkjasfdas;lk3453-934023safkl34a44dsflkjaQQuBBdfk',];
-
-async function no_results_test() {
- let config = {
- search_engine: 'bing',
- compress: false,
- debug_level: 1,
- keywords: keywords_no_results,
- keyword_file: '',
- num_pages: 1,
- headless: true,
- output_file: '',
- block_assets: true,
- user_agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36',
- random_user_agent: false,
- };
-
- let scrape_config = {
- search_engine: 'bing',
- keywords: keywords_no_results,
- num_pages: 1,
- };
-
- console.log('no_results_test()');
- test_case_no_results( await se_scraper.scrape(config, scrape_config) );
-}
-
-// we test with a callback function to our handler
-function test_case_no_results(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- for (let query in response.results) {
-
- assert.containsAllKeys(response.results, keywords_no_results, 'not all keywords were scraped.');
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query'], 'not all keys are in the object');
-
- assert(obj.results.length === 0, 'results must have 0 SERP objects');
- assert.equal(obj.no_results, true, 'no results should be true');
- assert.isEmpty(obj.num_results, 'no results should be a empty string');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
- }
- }
-}
-
-const effective_query_keywords = ['mount everrest'];
-
-async function effective_query_test() {
- let config = {
- compress: false,
- debug_level: 1,
- keyword_file: '',
- headless: true,
- output_file: '',
- block_assets: true,
- user_agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36',
- random_user_agent: false,
- };
-
- let scrape_config = {
- search_engine: 'bing',
- keywords: effective_query_keywords,
- num_pages: 1,
- };
-
- console.log('effective_query_test()');
- test_case_effective_query( await se_scraper.scrape(config, scrape_config) );
-}
-
-// we test with a callback function to our handler
-function test_case_effective_query(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- for (let query in response.results) {
-
- assert.containsAllKeys(response.results, effective_query_keywords, 'not all keywords were scraped.');
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query'], 'not all keys are in the object');
-
- // effective query must be different to the original keyword
- assert.isOk(obj.effective_query, 'effective query must be ok');
- assert.isNotEmpty(obj.effective_query, 'effective query must be valid');
- assert(obj.effective_query !== query, 'effective query must be different from keyword');
-
- assert.isAtLeast(obj.results.length, 7, 'results must have at least 7 SERP objects');
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
- }
- }
-}
-
-
-const ads_keywords = ['cloud services', 'buy shoes'];
-
-async function ads_test() {
- let config = {
- compress: false,
- debug_level: 1,
- headless: true,
- block_assets: false,
- random_user_agent: true,
- };
-
- let scrape_config = {
- search_engine: 'bing',
- keywords: ads_keywords,
- num_pages: 1,
- };
-
- console.log('ads_test()');
- test_case_ads_test( await se_scraper.scrape(config, scrape_config) );
-}
-
-function test_case_ads_test(response) {
- assert.equal(response.metadata.num_requests, 2);
-
- for (let query in response.results) {
-
- assert.containsAllKeys(response.results, ads_keywords, 'not all keywords were scraped.');
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query', 'ads'], 'not all keys are in the object');
-
- assert.isAtLeast(obj.results.length, 5, 'results must have at least 5 SERP objects');
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- assert.isAtLeast(obj.ads.length, 2, 'ads must have at least 2 SERP objects');
-
- for (let res of obj.ads) {
-
- assert.isOk(res.tracking_link, 'link must be ok');
- assert.typeOf(res.tracking_link, 'string', 'link must be string');
- assert.isAtLeast(res.tracking_link.length, 5, 'link must have at least 5 chars');
-
- // assert.isOk(res.link, 'link must be ok');
- // assert.typeOf(res.link, 'string', 'link must be string');
- // assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'visible_link must be ok');
- assert.typeOf(res.visible_link, 'string', 'visible_link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'visible_link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
- }
- }
- }
-}
-
-describe('Bing', function(){
- this.timeout(30000);
- it('normal search', normal_search_test);
- it('no results', no_results_test);
- it('effective query', effective_query_test);
- it('finds ads', ads_test);
-});
diff --git a/test/test_duckduckgo.js b/test/test_duckduckgo.js
deleted file mode 100644
index 39efe6e..0000000
--- a/test/test_duckduckgo.js
+++ /dev/null
@@ -1,192 +0,0 @@
-'use strict';
-const se_scraper = require('./../index.js');
-const chai = require('chai');
-chai.use(require('chai-string'));
-const assert = chai.assert;
-
-const normal_search_keywords = ['apple tree', 'weather tomorrow'];
-
-async function normal_search_test() {
- let config = {
- compress: false,
- debug_level: 1,
- headless: true,
- block_assets: false,
- random_user_agent: true,
- };
-
- let scrape_config = {
- search_engine: 'duckduckgo',
- keywords: normal_search_keywords,
- num_pages: 2,
- };
-
- console.log('normal_search_test()');
- normal_search_test_case( await se_scraper.scrape(config, scrape_config) );
-}
-
-// we test with a callback function to our handler
-function normal_search_test_case(response) {
- assert.equal(response.metadata.num_requests, 2);
-
- for (let query in response.results) {
- let total_rank = 1;
-
- assert.containsAllKeys(response.results, normal_search_keywords, 'not all keywords were scraped.');
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time', 'effective_query'], 'not all keys are in the object');
-
- assert.isAtLeast(obj.results.length, 7, 'results must have at least 7 SERP objects');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- for (let res of obj.results) {
-
- assert.containsAllKeys(res, ['link', 'title', 'rank', 'visible_link', 'snippet'], 'not all keys are in the SERP object');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'visible_link must be ok');
- assert.typeOf(res.visible_link, 'string', 'visible_link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'visible_link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 5, 'title must have at least 5 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
-
- assert.isNumber(res.rank, 'rank must be integer');
- assert.equal(res.rank, total_rank++, 'rank ist wrong');
- }
- }
- }
-}
-
-const effective_query_keywords = ['mount everrest'];
-
-async function effective_query_test() {
- let config = {
- compress: false,
- debug_level: 1,
- headless: true,
- block_assets: true,
- random_user_agent: true,
- };
-
- let scrape_config = {
- search_engine: 'duckduckgo',
- keywords: effective_query_keywords,
- num_pages: 1,
- };
-
- console.log('test_case_effective_query()');
- test_case_effective_query( await se_scraper.scrape(config, scrape_config) );
-}
-
-// we test with a callback function to our handler
-function test_case_effective_query(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- results = response.results;
- for (let query in response.results) {
-
- assert.containsAllKeys(response.results, effective_query_keywords, 'not all keywords were scraped.');
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time', 'effective_query'], 'not all keys are in the object');
-
- // effective query must be different to the original keyword
- assert.isOk(obj.effective_query, 'effective query must be ok');
- assert.isNotEmpty(obj.effective_query, 'effective query must be valid');
- assert(obj.effective_query !== query, 'effective query must be different from keyword');
-
- assert.isAtLeast(obj.results.length, 7, 'results must have at least 7 SERP objects');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
- }
- }
-}
-
-const ads_keywords = ['cloud services', 'buy shoes'];
-
-async function ads_test() {
- let config = {
- compress: false,
- debug_level: 1,
- headless: true,
- block_assets: false,
- random_user_agent: false,
- };
-
- let scrape_config = {
- search_engine: 'duckduckgo',
- keywords: ads_keywords,
- num_pages: 1,
- };
-
- console.log('ads_test()');
- test_case_ads_test( await se_scraper.scrape(config, scrape_config) );
-}
-
-function test_case_ads_test(response) {
- assert.equal(response.metadata.num_requests, 2);
-
- for (let query in response.results) {
-
- assert.containsAllKeys(response.results, ads_keywords, 'not all keywords were scraped.');
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time', 'effective_query', 'ads'], 'not all keys are in the object');
-
- assert.isAtLeast(obj.results.length, 6, 'results must have at least 6 SERP objects');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
- assert.isAtLeast(obj.ads.length, 2, 'ads must have at least 2 SERP object');
-
- for (let res of obj.ads) {
-
- assert.isOk(res.tracking_link, 'link must be ok');
- assert.typeOf(res.tracking_link, 'string', 'link must be string');
- assert.isAtLeast(res.tracking_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'visible_link must be ok');
- assert.typeOf(res.visible_link, 'string', 'visible_link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'visible_link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
- }
- }
- }
-}
-
-
-describe('Duckduckgo', function(){
- this.timeout(30000);
- it('normal search', normal_search_test);
- it('effective query', effective_query_test);
- it('finds ads', ads_test);
-});
\ No newline at end of file
diff --git a/test/test_google.js b/test/test_google.js
deleted file mode 100644
index c21ef4b..0000000
--- a/test/test_google.js
+++ /dev/null
@@ -1,424 +0,0 @@
-'use strict';
-const se_scraper = require('./../index.js');
-const chai = require('chai');
-chai.use(require('chai-string'));
-const assert = chai.assert;
-
-/*
- * Use chai and mocha for tests.
- * https://mochajs.org/#installation
- */
-
-const normal_search_keywords = ['apple tree', 'weather tomorrow'];
-
-async function normal_search_test() {
- let config = {
- compress: false,
- debug_level: 1,
- keyword_file: '',
- headless: true,
- output_file: '',
- block_assets: true,
- user_agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36',
- random_user_agent: false,
- };
-
- let scrape_config = {
- search_engine: 'google',
- keywords: normal_search_keywords,
- num_pages: 3,
- };
-
- console.log('normal_search_test()');
- normal_search_test_case( await se_scraper.scrape(config, scrape_config) );
-}
-
-// we test with a callback function to our handler
-function normal_search_test_case(response) {
- assert.equal(response.metadata.num_requests, 6);
-
- for (let query in response.results) {
- let total_rank = 1;
-
- assert.containsAllKeys(response.results, normal_search_keywords, 'not all keywords were scraped.');
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query'], 'not all keys are in the object');
-
- assert.isAtLeast(obj.results.length, 7, 'results must have at least 8 SERP objects');
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- for (let res of obj.results) {
-
- assert.containsAllKeys(res, ['link', 'title', 'rank', 'visible_link'], 'not all keys are in the SERP object');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'visible_link must be ok');
- assert.typeOf(res.visible_link, 'string', 'visible_link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'visible_link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
-
- assert.isNumber(res.rank, 'rank must be integer');
- assert.equal(res.rank, total_rank++, 'rank ist wrong');
- }
- }
- }
-}
-
-const keywords_no_results = ['fgskl34440abJAksfs4353534a3l34AVGFDFflkjaQQuBBdfk',];
-
-async function no_results_test() {
- let config = {
- compress: false,
- debug_level: 1,
- keyword_file: '',
- headless: true,
- output_file: '',
- block_assets: true,
- user_agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36',
- random_user_agent: false,
- };
-
- let scrape_config = {
- search_engine: 'google',
- keywords: keywords_no_results,
- num_pages: 1,
- };
-
- console.log('no_results_test()');
- test_case_no_results( await se_scraper.scrape(config, scrape_config) );
-}
-
-// we test with a callback function to our handler
-function test_case_no_results(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- for (let query in response.results) {
-
- assert.containsAllKeys(response.results, keywords_no_results, 'not all keywords were scraped.');
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query'], 'not all keys are in the object');
-
- assert.strictEqual(obj.results.length, 0, 'results must have 0 SERP objects');
- assert.equal(obj.no_results, true, 'no results should be true');
- assert.isEmpty(obj.num_results, 'num_results should be a empty string');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
- }
- }
-}
-
-const effective_query_keywords = ['mount evverrest'];
-
-async function effective_query_test() {
- let config = {
- compress: false,
- debug_level: 1,
- keyword_file: '',
- headless: true,
- output_file: '',
- block_assets: true,
- user_agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36',
- random_user_agent: false,
- };
-
- let scrape_config = {
- search_engine: 'google',
- keywords: effective_query_keywords,
- num_pages: 1,
- };
-
- console.log('effective_query_test()');
- test_case_effective_query( await se_scraper.scrape(config, scrape_config) );
-}
-
-// we test with a callback function to our handler
-function test_case_effective_query(response) {
- assert.equal(response.metadata.num_requests, 1);
-
- for (let query in response.results) {
-
- assert.containsAllKeys(response.results, effective_query_keywords, 'not all keywords were scraped.');
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query'], 'not all keys are in the object');
-
- // effective query must be different to the original keyword
- assert.isOk(obj.effective_query, 'effective query must be ok');
- assert.isNotEmpty(obj.effective_query, 'effective query must be valid');
- assert(obj.effective_query !== query, 'effective query must be different from keyword');
-
- assert.isAtLeast(obj.results.length, 7, 'results must have at least 8 SERP objects');
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
- }
- }
-}
-
-async function html_output_query_test() {
- let config = {
- compress: false,
- debug_level: 1,
- keyword_file: '',
- headless: true,
- output_file: '',
- block_assets: true,
- user_agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36',
- random_user_agent: false,
- };
-
- let scrape_config = {
- search_engine: 'google',
- keywords: normal_search_keywords,
- num_pages: 3,
- html_output: true,
- };
-
- let output = await se_scraper.scrape(config, scrape_config);
- normal_search_test_case( output );
- check_html_output_test_case( output );
-}
-
-function check_html_output_test_case( response ) {
- for (let query in response.html_output) {
-
- assert.containsAllKeys(response.html_output, normal_search_keywords, 'not all keywords were scraped.');
-
- for (let page_number in response.html_output[query]) {
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
- assert.startsWith(response.html_output[query][page_number], '= 1 || obj.bottom_ads.length >= 1, 'top_ads or bottom_ads must have at least 1 SERP object');
-
- for (let res of obj.top_ads) {
-
- assert.isOk(res.tracking_link, 'link must be ok');
- assert.typeOf(res.tracking_link, 'string', 'link must be string');
- assert.isAtLeast(res.tracking_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'link must be ok');
- assert.typeOf(res.visible_link, 'string', 'link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.link, 'visible_link must be ok');
- assert.typeOf(res.link, 'string', 'visible_link must be string');
- assert.isAtLeast(res.link.length, 5, 'visible_link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
-
- assert.typeOf(res.links, 'array', 'links must be array');
- }
-
- for (let res of obj.bottom_ads) {
- assert.isOk(res.tracking_link, 'link must be ok');
- assert.typeOf(res.tracking_link, 'string', 'link must be string');
- assert.isAtLeast(res.tracking_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'link must be ok');
- assert.typeOf(res.visible_link, 'string', 'link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.link, 'visible_link must be ok');
- assert.typeOf(res.link, 'string', 'visible_link must be string');
- assert.isAtLeast(res.link.length, 5, 'visible_link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
-
- assert.typeOf(res.links, 'array', 'links must be array');
- }
-
- }
- }
-}
-
-
-
-const product_keywords = ['autoreifen bmw'];
-
-async function products_test() {
- let config = {
- compress: false,
- debug_level: 1,
- headless: true,
- block_assets: false,
- random_user_agent: false, // dont try to trick google with ads
- };
-
- let scrape_config = {
- search_engine: 'google',
- keywords: ads_keywords,
- num_pages: 1,
- };
-
- console.log('products_test()');
- test_case_products_test( await se_scraper.scrape(config, scrape_config) );
-}
-
-function test_case_products_test(response) {
- assert.equal(response.metadata.num_requests, 2);
-
- for (let query in response.results) {
-
- assert.containsAllKeys(response.results, ads_keywords, 'not all keywords were scraped.');
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query', 'top_ads', 'bottom_ads', 'places'], 'not all keys are in the object');
-
- assert.isAtLeast(obj.results.length, 7, 'results must have at least 7 SERP objects');
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- assert(obj.top_products.length >= 1 || obj.right_products.length >= 1, 'top_products or right_products must have at least 1 SERP object');
-
- for (let res of obj.top_products) {
-
- assert.isOk(res.tracking_link, 'link must be ok');
- assert.typeOf(res.tracking_link, 'string', 'link must be string');
- assert.isAtLeast(res.tracking_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.price, 'price must be ok');
- assert.typeOf(res.price, 'string', 'price must be string');
- assert.isAtLeast(res.price.length, 5, 'price must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.vendor_link, 'vendor_link must be ok');
- assert.typeOf(res.vendor_link, 'string', 'vendor_link must be string');
- assert.isAtLeast(res.vendor_link.length, 10, 'vendor_link must have at least 10 chars');
- }
-
- for (let res of obj.right_products) {
- assert.isOk(res.tracking_link, 'link must be ok');
- assert.typeOf(res.tracking_link, 'string', 'link must be string');
- assert.isAtLeast(res.tracking_link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.price, 'price must be ok');
- assert.typeOf(res.price, 'string', 'price must be string');
- assert.isAtLeast(res.price.length, 5, 'price must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.vendor_link, 'vendor_link must be ok');
- assert.typeOf(res.vendor_link, 'string', 'vendor_link must be string');
- assert.isAtLeast(res.vendor_link.length, 10, 'vendor_link must have at least 10 chars');
- }
-
- }
- }
-}
-
-describe('Google', function(){
- this.timeout(30000);
- it('normal search', normal_search_test);
- it('no results', no_results_test);
- it('effective query', effective_query_test);
- it('html output query', html_output_query_test);
- it('ads', ads_test);
- it('products test', products_test);
-});
diff --git a/test/test_googleimage.js b/test/test_googleimage.js
deleted file mode 100644
index 31f00e9..0000000
--- a/test/test_googleimage.js
+++ /dev/null
@@ -1,80 +0,0 @@
-'use strict';
-const se_scraper = require('./../index.js');
-const assert = require('chai').assert;
-
-/*
- * Use chai and mocha for tests.
- * https://mochajs.org/#installation
- */
-
-const normal_search_keywords = ['apple', 'rain'];
-
-async function normal_image_search_test() {
- let config = {
- compress: false,
- debug_level: 0,
- headless: true,
- block_assets: true,
- user_agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36',
- random_user_agent: false,
- };
-
- let scrape_config = {
- search_engine: 'google_image',
- keywords: normal_search_keywords,
- num_pages: 2,
- };
-
- console.log('normal_image_search_test()');
- normal_image_search_test_case( await se_scraper.scrape(config, scrape_config) );
-}
-
-// we test with a callback function to our handler
-function normal_image_search_test_case(response) {
- assert.equal(response.metadata.num_requests, 2);
-
- for (let query in response.results) {
-
- let total_rank = 1;
-
- assert.containsAllKeys(response.results, normal_search_keywords, 'not all keywords were scraped.');
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'effective_query'], 'not all keys are in the object');
-
- assert.isAtLeast(obj.results.length, 15, 'results must have at least 15 SERP objects');
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- for (let res of obj.results) {
-
- assert.containsAllKeys(res, ['link', 'snippet', 'rank', 'clean_link'], 'not all keys are in the SERP object');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.clean_link, 'clean_link must be ok');
- assert.typeOf(res.clean_link, 'string', 'clean_link must be string');
- assert.isAtLeast(res.clean_link.length, 5, 'clean_link must have at least 5 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
-
- assert.isNumber(res.rank, 'rank must be integer');
- assert.equal(res.rank, total_rank++, 'rank ist wrong');
- }
- }
- }
-}
-
-describe('Google Image', function(){
- this.timeout(30000);
- it('normal image search test', normal_image_search_test);
-});
\ No newline at end of file
diff --git a/test/test_queryargs_google.js b/test/test_queryargs_google.js
deleted file mode 100644
index d4b51db..0000000
--- a/test/test_queryargs_google.js
+++ /dev/null
@@ -1,91 +0,0 @@
-'use strict';
-const se_scraper = require('./../index.js');
-const assert = require('chai').assert;
-
-const normal_search_keywords = ['apple juice'];
-
-async function queryargs_search_test() {
- let config = {
- search_engine: 'google',
- compress: false,
- debug: true,
- verbose: true,
- keywords: normal_search_keywords,
- keyword_file: '',
- num_pages: 2,
- headless: true,
- output_file: '',
- block_assets: true,
- // use specific search engine parameters for various search engines
- google_settings: {
- google_domain: 'google.com',
- gl: 'fr', // The gl parameter determines the Google country to use for the query.
- hl: 'fr', // The hl parameter determines the Google UI language to return results.
- start: 30, // Determines the results offset to use, defaults to 0.
- num: 100, // Determines the number of results to show, defaults to 10. Maximum is 100.
- },
- };
-
- console.log('queryargs_search_test()');
- await se_scraper.scrape(config, queryargs_search_test_case);
-}
-
-// we test with a callback function to our handler
-function queryargs_search_test_case(err, response) {
-
- if (err) {
- console.error(err);
- } else {
- assert.equal(response.metadata.num_requests, 2);
-
- for (let query in response.results) {
- let total_rank = 1;
-
- assert.containsAllKeys(response.results, normal_search_keywords, 'not all keywords were scraped.');
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time', 'no_results', 'num_results', 'effective_query'], 'not all keys are in the object');
-
- assert.isAtLeast(obj.results.length, 90, 'results must have at least 80 SERP objects');
- assert.equal(obj.no_results, false, 'no results should be false');
- assert.typeOf(obj.num_results, 'string', 'num_results must be a string');
- assert.isAtLeast(obj.num_results.length, 5, 'num_results should be a string of at least 5 chars');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- for (let res of obj.results) {
-
- assert.containsAllKeys(res, ['link', 'title', 'rank', 'visible_link'], 'not all keys are in the SERP object');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.visible_link, 'visible_link must be ok');
- assert.typeOf(res.visible_link, 'string', 'visible_link must be string');
- assert.isAtLeast(res.visible_link.length, 5, 'visible_link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 10, 'title must have at least 10 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
-
- assert.isNumber(res.rank, 'rank must be integer');
- assert.equal(res.rank, total_rank++, 'rank ist wrong');
- }
- }
- }
- }
-}
-
-describe('Google with query arguments', function(){
- this.timeout(30000);
- it('query args search test', queryargs_search_test);
-});
diff --git a/test/test_ticker_search.js b/test/test_ticker_search.js
deleted file mode 100644
index f8a379f..0000000
--- a/test/test_ticker_search.js
+++ /dev/null
@@ -1,217 +0,0 @@
-'use strict';
-const se_scraper = require('./../index.js');
-const assert = require('chai').assert;
-
-/*
- * Use chai and mocha for tests.
- * https://mochajs.org/#installation
- */
-
-const quote_search_keywords = ['MSFT', 'AAPL'];
-
-async function reuters_search_test() {
- let config = {
- search_engine: 'reuters',
- compress: false,
- debug: false,
- verbose: false,
- keywords: quote_search_keywords,
- keyword_file: '',
- num_pages: 1,
- headless: true,
- output_file: '',
- block_assets: true,
- user_agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36',
- random_user_agent: false,
- };
-
- console.log('reuters_search_test()');
- await se_scraper.scrape(config, reuters_search_test_case);
-}
-
-// we test with a callback function to our handler
-function reuters_search_test_case(err, response) {
-
- if (err) {
- console.error(err);
- } else {
-
- for (let query in response.results) {
- let total_rank = 1;
- assert.containsAllKeys(response.results, quote_search_keywords, 'not all keywords were scraped.');
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time'], 'not all keys are in the object');
-
- assert.isAtLeast(obj.results.length, 7, 'results must have at least 7 SERP objects');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- for (let res of obj.results) {
-
- assert.containsAllKeys(res, ['link', 'title', 'date', 'snippet'], 'not all keys are in the SERP object');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 5, 'title must have at least 5 chars');
-
- assert.isOk(res.snippet, 'snippet must be ok');
- assert.typeOf(res.snippet, 'string', 'snippet must be string');
- assert.isAtLeast(res.snippet.length, 10, 'snippet must have at least 10 chars');
-
- assert.isOk(res.date, 'date must be ok');
- assert.typeOf(res.date, 'string', 'date must be string');
- assert.isAtLeast(res.date.length, 5, 'date must have at least 5 chars');
- }
- }
- }
- }
-}
-
-async function cnbc_search_test() {
- let config = {
- search_engine: 'cnbc',
- compress: false,
- debug: false,
- verbose: false,
- keywords: quote_search_keywords,
- keyword_file: '',
- num_pages: 1,
- headless: true,
- output_file: '',
- block_assets: true,
- user_agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36',
- random_user_agent: false,
- };
-
- console.log('cnbc_search_test()');
- await se_scraper.scrape(config, cnbc_search_test_case);
-}
-
-// we test with a callback function to our handler
-function cnbc_search_test_case(err, response) {
-
- if (err) {
- console.error(err);
- } else {
-
- for (let query in response.results) {
- let total_rank = 1;
- assert.containsAllKeys(response.results, quote_search_keywords, 'not all keywords were scraped.');
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time'], 'not all keys are in the object');
-
- assert.isAtLeast(obj.results.length, 7, 'results must have at least 7 SERP objects');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- for (let res of obj.results) {
-
- assert.containsAllKeys(res, ['link', 'title', 'date'], 'not all keys are in the SERP object');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 5, 'title must have at least 5 chars');
-
- assert.isOk(res.date, 'date must be ok');
- assert.typeOf(res.date, 'string', 'date must be string');
- assert.isAtLeast(res.date.length, 5, 'date must have at least 5 chars');
- }
- }
- }
- }
-}
-
-const marketwatch_search_keywords = ['MSFT'];
-
-async function marketwatch_search_test() {
- let config = {
- search_engine: 'marketwatch',
- compress: false,
- debug: false,
- verbose: false,
- keywords: marketwatch_search_keywords,
- keyword_file: '',
- num_pages: 1,
- headless: true,
- output_file: '',
- block_assets: true,
- user_agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36',
- random_user_agent: false,
- };
-
- console.log('marketwatch_search_test()');
- await se_scraper.scrape(config, marketwatch_search_test_case);
-}
-
-// we test with a callback function to our handler
-function marketwatch_search_test_case(err, response) {
-
- if (err) {
- console.error(err);
- } else {
-
- for (let query in response.results) {
- let total_rank = 1;
- assert.containsAllKeys(response.results, marketwatch_search_keywords, 'not all keywords were scraped.');
-
- for (let page_number in response.results[query]) {
-
- assert.isNumber(parseInt(page_number), 'page_number must be numeric');
-
- let obj = response.results[query][page_number];
-
- assert.containsAllKeys(obj, ['results', 'time'], 'not all keys are in the object');
-
- assert.isAtLeast(obj.results.length, 7, 'results must have at least 7 SERP objects');
- assert.typeOf(Date.parse(obj.time), 'number', 'time should be a valid date');
-
- for (let res of obj.results) {
-
- assert.containsAllKeys(res, ['link', 'title', 'date', 'author'], 'not all keys are in the SERP object');
-
- assert.isOk(res.link, 'link must be ok');
- assert.typeOf(res.link, 'string', 'link must be string');
- assert.isAtLeast(res.link.length, 5, 'link must have at least 5 chars');
-
- assert.isOk(res.title, 'title must be ok');
- assert.typeOf(res.title, 'string', 'title must be string');
- assert.isAtLeast(res.title.length, 5, 'title must have at least 5 chars');
-
- assert.isOk(res.author, 'author must be ok');
- assert.typeOf(res.author, 'string', 'author must be string');
- assert.isAtLeast(res.author.length, 5, 'author must have at least 5 chars');
-
- assert.isOk(res.date, 'date must be ok');
- assert.typeOf(res.date, 'string', 'date must be string');
- assert.isAtLeast(res.date.length, 5, 'date must have at least 5 chars');
- }
- }
- }
- }
-}
-
-
-describe('Ticker', function(){
- this.timeout(30000);
- it('Reuters search test', reuters_search_test);
- it('CNBC search test', cnbc_search_test);
- it('Marketwatch search test', marketwatch_search_test);
-});
\ No newline at end of file
diff --git a/test/user_agent.js b/test/user_agent.js
new file mode 100644
index 0000000..b4ddc68
--- /dev/null
+++ b/test/user_agent.js
@@ -0,0 +1,144 @@
+'use strict';
+const express = require('express');
+const { createLogger, transports } = require('winston');
+const http = require('http');
+const https = require('https');
+const assert = require('assert');
+const keyCert = require('key-cert');
+const Promise = require('bluebird');
+const Proxy = require('http-mitm-proxy');
+const UAParser = require('ua-parser-js');
+const _ = require('lodash');
+
+const debug = require('debug')('se-scraper:test');
+const se_scraper = require('../');
+const Scraper = require('../src/modules/se_scraper');
+
+const httpPort = 3012;
+const httpsPort = httpPort + 1;
+const proxyPort = httpPort + 2;
+
+const fakeSearchEngine = express();
+fakeSearchEngine.set('trust proxy', 'loopback');
+fakeSearchEngine.get('/test-user_agent', (req, res) => {
+ debug('fake-search-engine req.headers.user-agent=%s', req.headers['user-agent']);
+ res.send(req.headers['user-agent']);
+});
+
+describe('Config', function(){
+
+ let httpServer, httpsServer, proxy;
+ before(async function(){
+ // Here mount our fake engine in both http and https listen server
+ httpServer = http.createServer(fakeSearchEngine);
+ httpsServer = https.createServer(await keyCert(), fakeSearchEngine);
+
+ proxy = Proxy();
+ proxy.onRequest((ctx, callback) => {
+ ctx.proxyToServerRequestOptions.host = 'localhost';
+ ctx.proxyToServerRequestOptions.port = (ctx.isSSL) ? httpsPort : httpPort;
+ ctx.proxyToServerRequestOptions.headers['X-Forwarded-Host'] = 'ProxiedThroughFakeEngine';
+ debug('Proxy request to %s', ctx.clientToProxyRequest.headers.host);
+ return callback();
+ });
+
+ await Promise.promisify(proxy.listen, {context: proxy})({port: proxyPort});
+ await Promise.promisify(httpServer.listen, {context: httpServer})(httpPort);
+ await Promise.promisify(httpsServer.listen, {context: httpsServer})(httpsPort);
+ debug('Fake http search engine servers started');
+ });
+
+ after(function(){
+ httpsServer.close();
+ httpServer.close();
+ proxy.close();
+ });
+
+ describe('user_agent', function(){
+
+ class MockScraperTestUserAgent extends Scraper {
+
+ async load_start_page(){
+ return true;
+ }
+
+ async search_keyword(){
+ await this.page.goto('http://localhost:' + httpPort + '/test-user_agent');
+ }
+
+ async parse_async(){
+ const bodyHandle = await this.page.$('body');
+ return await this.page.evaluate(body => body.innerHTML, bodyHandle);
+ }
+ }
+
+ const testLogger = createLogger({
+ transports: [
+ new transports.Console({
+ level: 'error'
+ })
+ ]
+ });
+
+ /**
+ * Test user_agent option
+ */
+ it('fixed user_agent', async function () {
+
+ const scrape_job = {
+ search_engine: MockScraperTestUserAgent,
+ keywords: ['javascript is hard'],
+ };
+
+ var scraper = new se_scraper.ScrapeManager({
+ throw_on_detection: true,
+ logger: testLogger,
+ user_agent: 'THIS IS A USERAGENT 42.0'
+ });
+ await scraper.start();
+
+ const { results } = await scraper.scrape(scrape_job);
+ assert.strictEqual(results['javascript is hard']['1'], 'THIS IS A USERAGENT 42.0');
+
+ await scraper.quit();
+ });
+
+ /**
+ * Test random_user_agent option
+ * TODO generated user_agent should be different for each keyword
+ * TODO this test will sometimes fail because user_agent not very random :-(
+ */
+ it('random_user_agent', async function () {
+
+ const scrape_job = {
+ search_engine: MockScraperTestUserAgent,
+ keywords: ['news'],
+ };
+
+ const NUMBER_OF_EXEC = 10;
+
+ const uaList = await Promise.map(_.range(NUMBER_OF_EXEC), async (i) => {
+ const scraper = new se_scraper.ScrapeManager({
+ throw_on_detection: true,
+ logger: testLogger,
+ random_user_agent: true,
+ });
+ await scraper.start();
+ const { results: { news } } = await scraper.scrape(scrape_job);
+ await scraper.quit();
+ return news['1'];
+ });
+
+ uaList.forEach((userAgent) => {
+ const uaParsed = UAParser(userAgent);
+ assert(uaParsed.browser.name, 'UserAgent should have a browser name detected');
+ assert(uaParsed.os.name, 'UserAgent should have a os name detected');
+ });
+
+ assert( _.chain(uaList).countBy().toPairs().sortBy(e => e[1]).last().value()[1] < (NUMBER_OF_EXEC * 0.4), 'Each user agent should appear less than 40% of the time' );
+
+ });
+
+ });
+
+});
\ No newline at end of file