diff options
| author | Markus Heiser <markus.heiser@darmarit.de> | 2024-12-15 09:59:50 +0100 |
|---|---|---|
| committer | Markus Heiser <markus.heiser@darmarIT.de> | 2025-01-28 07:07:08 +0100 |
| commit | edfbf1e1183815cea3b723f3b66260bc55679f32 (patch) | |
| tree | 46ad7b01bcb3acbeff64a848a8b5f100e03e2ce4 /tests/unit | |
| parent | 9079d0cac0156139952446f0fdc8b37b94c10756 (diff) | |
[refactor] typification of SearXNG (initial) / result items (part 1)
Typification of SearXNG
=======================
This patch introduces the typing of the results. The why and how is described
in the documentation, please generate the documentation ..
$ make docs.clean docs.live
and read the following articles in the "Developer documentation":
- result types --> http://0.0.0.0:8000/dev/result_types/index.html
The result types are available from the `searx.result_types` module. The
following have been implemented so far:
- base result type: `searx.result_type.Result`
--> http://0.0.0.0:8000/dev/result_types/base_result.html
- answer results
--> http://0.0.0.0:8000/dev/result_types/answer.html
including the type for translations (inspired by #3925). For all other
types (which still need to be set up in subsequent PRs), template documentation
has been created for the transition period.
Doc of the fields used in Templates
===================================
The template documentation is the basis for the typing and is the first complete
documentation of the results (needed for engine development). It is the
"working paper" (the plan) with which further typifications can be implemented
in subsequent PRs.
- https://github.com/searxng/searxng/issues/357
Answer Templates
================
With the new (sub) types for `Answer`, the templates for the answers have also
been revised, `Translation` are now displayed with collapsible entries (inspired
by #3925).
!en-de dog
Plugins & Answerer
==================
The implementation for `Plugin` and `Answer` has been revised, see
documentation:
- Plugin: http://0.0.0.0:8000/dev/plugins/index.html
- Answerer: http://0.0.0.0:8000/dev/answerers/index.html
With `AnswerStorage` and `AnswerStorage` to manage those items (in follow up
PRs, `ArticleStorage`, `InfoStorage` and .. will be implemented)
Autocomplete
============
The autocompletion had a bug where the results from `Answer` had not been shown
in the past. To test activate autocompletion and try search terms for which we
have answerers
- statistics: type `min 1 2 3` .. in the completion list you should find an
entry like `[de] min(1, 2, 3) = 1`
- random: type `random uuid` .. in the completion list, the first item is a
random UUID
Extended Types
==============
SearXNG extends e.g. the request and response types of flask and httpx, a module
has been set up for type extensions:
- Extended Types
--> http://0.0.0.0:8000/dev/extended_types.html
Unit-Tests
==========
The unit tests have been completely revised. In the previous implementation,
the runtime (the global variables such as `searx.settings`) was not initialized
before each test, so the runtime environment with which a test ran was always
determined by the tests that ran before it. This was also the reason why we
sometimes had to observe non-deterministic errors in the tests in the past:
- https://github.com/searxng/searxng/issues/2988 is one example for the Runtime
issues, with non-deterministic behavior ..
- https://github.com/searxng/searxng/pull/3650
- https://github.com/searxng/searxng/pull/3654
- https://github.com/searxng/searxng/pull/3642#issuecomment-2226884469
- https://github.com/searxng/searxng/pull/3746#issuecomment-2300965005
Why msgspec.Struct
==================
We have already discussed typing based on e.g. `TypeDict` or `dataclass` in the past:
- https://github.com/searxng/searxng/pull/1562/files
- https://gist.github.com/dalf/972eb05e7a9bee161487132a7de244d2
- https://github.com/searxng/searxng/pull/1412/files
- https://github.com/searxng/searxng/pull/1356
In my opinion, TypeDict is unsuitable because the objects are still dictionaries
and not instances of classes / the `dataclass` are classes but ...
The `msgspec.Struct` combine the advantages of typing, runtime behaviour and
also offer the option of (fast) serializing (incl. type check) the objects.
Currently not possible but conceivable with `msgspec`: Outsourcing the engines
into separate processes, what possibilities this opens up in the future is left
to the imagination!
Internally, we have already defined that it is desirable to decouple the
development of the engines from the development of the SearXNG core / The
serialization of the `Result` objects is a prerequisite for this.
HINT: The threads listed above were the template for this PR, even though the
implementation here is based on msgspec. They should also be an inspiration for
the following PRs of typification, as the models and implementations can provide
a good direction.
Why just one commit?
====================
I tried to create several (thematically separated) commits, but gave up at some
point ... there are too many things to tackle at once / The comprehensibility of
the commits would not be improved by a thematic separation. On the contrary, we
would have to make multiple changes at the same places and the goal of a change
would be vaguely recognizable in the fog of the commits.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Diffstat (limited to 'tests/unit')
33 files changed, 552 insertions, 499 deletions
diff --git a/tests/unit/__init__.py b/tests/unit/__init__.py index e43c93f1c..0f13c536d 100644 --- a/tests/unit/__init__.py +++ b/tests/unit/__init__.py @@ -1,8 +1,10 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name import os -from os.path import dirname, sep, abspath +from pathlib import Path -# In unit tests the user settings from unit/settings/test_settings.yml are used. -os.environ['SEARXNG_SETTINGS_PATH'] = abspath(dirname(__file__) + sep + 'settings' + sep + 'test_settings.yml') +# By default, in unit tests the user settings from +# unit/settings/test_settings.yml are used. + +os.environ['SEARXNG_SETTINGS_PATH'] = str(Path(__file__).parent / "settings" / "test_settings.yml") diff --git a/tests/unit/engines/__init__.py b/tests/unit/engines/__init__.py index 9ed59c825..f78f6acdd 100644 --- a/tests/unit/engines/__init__.py +++ b/tests/unit/engines/__init__.py @@ -1,2 +1,2 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name diff --git a/tests/unit/engines/test_command.py b/tests/unit/engines/test_command.py index 2123ab168..e9a88ffe1 100644 --- a/tests/unit/engines/test_command.py +++ b/tests/unit/engines/test_command.py @@ -1,27 +1,12 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring - -''' -searx is free software: you can redistribute it and/or modify -it under the terms of the GNU Affero General Public License as published by -the Free Software Foundation, either version 3 of the License, or -(at your option) any later version. - -searx is distributed in the hope that it will be useful, -but WITHOUT ANY WARRANTY; without even the implied warranty of -MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -GNU Affero General Public License for more details. - -You should have received a copy of the GNU Affero General Public License -along with searx. If not, see < http://www.gnu.org/licenses/ >. - -''' +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name from searx.engines import command as command_engine from tests import SearxTestCase -class TestCommandEngine(SearxTestCase): # pylint: disable=missing-class-docstring +class TestCommandEngine(SearxTestCase): + def test_basic_seq_command_engine(self): ls_engine = command_engine ls_engine.command = ['seq', '{{QUERY}}'] diff --git a/tests/unit/engines/test_xpath.py b/tests/unit/engines/test_xpath.py index e80ef96e2..81af052b9 100644 --- a/tests/unit/engines/test_xpath.py +++ b/tests/unit/engines/test_xpath.py @@ -1,5 +1,5 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name from collections import defaultdict import mock @@ -12,7 +12,7 @@ from tests import SearxTestCase logger = logger.getChild('engines') -class TestXpathEngine(SearxTestCase): # pylint: disable=missing-class-docstring +class TestXpathEngine(SearxTestCase): html = """ <div> <div class="search_result"> @@ -29,6 +29,7 @@ class TestXpathEngine(SearxTestCase): # pylint: disable=missing-class-docstring """ def setUp(self): + super().setUp() xpath.logger = logger.getChild('test_xpath') def test_request(self): diff --git a/tests/unit/network/__init__.py b/tests/unit/network/__init__.py index 9ed59c825..f78f6acdd 100644 --- a/tests/unit/network/__init__.py +++ b/tests/unit/network/__init__.py @@ -1,2 +1,2 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name diff --git a/tests/unit/network/test_network.py b/tests/unit/network/test_network.py index eabb23082..46dc62ad9 100644 --- a/tests/unit/network/test_network.py +++ b/tests/unit/network/test_network.py @@ -1,17 +1,15 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring, protected-access - -from mock import patch +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name import httpx +from mock import patch -from searx.network.network import Network, NETWORKS, initialize +from searx.network.network import Network, NETWORKS from tests import SearxTestCase -class TestNetwork(SearxTestCase): # pylint: disable=missing-class-docstring - def setUp(self): - initialize() +class TestNetwork(SearxTestCase): + # pylint: disable=protected-access def test_simple(self): network = Network() @@ -122,10 +120,13 @@ class TestNetwork(SearxTestCase): # pylint: disable=missing-class-docstring await network.aclose() -class TestNetworkRequestRetries(SearxTestCase): # pylint: disable=missing-class-docstring +class TestNetworkRequestRetries(SearxTestCase): TEXT = 'Lorem Ipsum' + def setUp(self): + self.init_test_settings() + @classmethod def get_response_404_then_200(cls): first = True @@ -195,10 +196,13 @@ class TestNetworkRequestRetries(SearxTestCase): # pylint: disable=missing-class await network.aclose() -class TestNetworkStreamRetries(SearxTestCase): # pylint: disable=missing-class-docstring +class TestNetworkStreamRetries(SearxTestCase): TEXT = 'Lorem Ipsum' + def setUp(self): + self.init_test_settings() + @classmethod def get_response_exception_then_200(cls): first = True diff --git a/tests/unit/processors/__init__.py b/tests/unit/processors/__init__.py index 9ed59c825..f78f6acdd 100644 --- a/tests/unit/processors/__init__.py +++ b/tests/unit/processors/__init__.py @@ -1,2 +1,2 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name diff --git a/tests/unit/processors/test_online.py b/tests/unit/processors/test_online.py index fcb01587d..b447533c2 100644 --- a/tests/unit/processors/test_online.py +++ b/tests/unit/processors/test_online.py @@ -1,31 +1,16 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name from searx.search import SearchQuery, EngineRef from searx.search.processors import online -import searx.search from searx import engines from tests import SearxTestCase -TEST_ENGINE_NAME = 'dummy engine' -TEST_ENGINE = { - 'name': TEST_ENGINE_NAME, - 'engine': 'dummy', - 'categories': 'general', - 'shortcut': 'du', - 'timeout': 3.0, - 'tokens': [], -} +TEST_ENGINE_NAME = "dummy engine" # from the ./settings/test_settings.yml -class TestOnlineProcessor(SearxTestCase): # pylint: disable=missing-class-docstring - - def setUp(self): - searx.search.initialize([TEST_ENGINE]) - - def tearDown(self): - searx.search.load_engines([]) +class TestOnlineProcessor(SearxTestCase): def _get_params(self, online_processor, search_query, engine_category): params = online_processor.get_params(search_query, engine_category) diff --git a/tests/unit/settings/limiter.toml b/tests/unit/settings/limiter.toml new file mode 100644 index 000000000..e8be2c70b --- /dev/null +++ b/tests/unit/settings/limiter.toml @@ -0,0 +1,2 @@ +[botdetection.ip_limit] +link_token = true diff --git a/tests/unit/settings/test_result_container.yml b/tests/unit/settings/test_result_container.yml new file mode 100644 index 000000000..c88a55861 --- /dev/null +++ b/tests/unit/settings/test_result_container.yml @@ -0,0 +1,8 @@ +# This SearXNG setup is used in unit tests + +use_default_settings: + + engines: + keep_only: + - google + - duckduckgo diff --git a/tests/unit/settings/test_settings.yml b/tests/unit/settings/test_settings.yml index e08105853..0836a165b 100644 --- a/tests/unit/settings/test_settings.yml +++ b/tests/unit/settings/test_settings.yml @@ -1,10 +1,30 @@ # This SearXNG setup is used in unit tests -use_default_settings: true +use_default_settings: + + engines: + # remove all engines + keep_only: [] + search: + formats: [html, csv, json, rss] +server: + + secret_key: "user_secret_key" + engines: - - name: general dummy + + - name: dummy engine + engine: demo_offline + categories: ["general"] + shortcut: "gd" + timeout: 3 + + - name: dummy private engine engine: demo_offline + categories: ["general"] + shortcut: "gdp" timeout: 3 + tokens: ["my-token"] diff --git a/tests/unit/settings/test_tineye.yml b/tests/unit/settings/test_tineye.yml new file mode 100644 index 000000000..7b6f37919 --- /dev/null +++ b/tests/unit/settings/test_tineye.yml @@ -0,0 +1,16 @@ +# This SearXNG setup is used in unit tests + +use_default_settings: + + engines: + # remove all engines + keep_only: [] + +engines: + + - name: tineye + engine: tineye + categories: ["general"] + shortcut: "tin" + timeout: 9.0 + disabled: true diff --git a/tests/unit/test_answerers.py b/tests/unit/test_answerers.py index 716b544de..4b7809796 100644 --- a/tests/unit/test_answerers.py +++ b/tests/unit/test_answerers.py @@ -1,17 +1,34 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name -from mock import Mock from parameterized import parameterized -from searx.answerers import answerers +import searx.plugins +import searx.answerers +import searx.preferences + +from searx.extended_types import sxng_request + from tests import SearxTestCase -class AnswererTest(SearxTestCase): # pylint: disable=missing-class-docstring - @parameterized.expand(answerers) - def test_unicode_input(self, answerer): - query = Mock() - unicode_payload = 'árvíztűrő tükörfúrógép' - query.query = '{} {}'.format(answerer.keywords[0], unicode_payload) - self.assertIsInstance(answerer.answer(query), list) +class AnswererTest(SearxTestCase): + + def setUp(self): + super().setUp() + + self.storage = searx.plugins.PluginStorage() + engines = {} + self.pref = searx.preferences.Preferences(["simple"], ["general"], engines, self.storage) + self.pref.parse_dict({"locale": "en"}) + + @parameterized.expand(searx.answerers.STORAGE.answerer_list) + def test_unicode_input(self, answerer_obj: searx.answerers.Answerer): + + with self.app.test_request_context(): + sxng_request.preferences = self.pref + + unicode_payload = "árvíztűrő tükörfúrógép" + for keyword in answerer_obj.keywords: + query = f"{keyword} {unicode_payload}" + self.assertIsInstance(answerer_obj.answer(query), list) diff --git a/tests/unit/test_engine_mariadb_server.py b/tests/unit/test_engine_mariadb_server.py index c4144a601..983b0ea06 100644 --- a/tests/unit/test_engine_mariadb_server.py +++ b/tests/unit/test_engine_mariadb_server.py @@ -1,12 +1,13 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name from unittest.mock import MagicMock, Mock + from searx.engines import mariadb_server from tests import SearxTestCase -class MariadbServerTests(SearxTestCase): # pylint: disable=missing-class-docstring +class MariadbServerTests(SearxTestCase): def test_init_no_query_str_raises(self): self.assertRaises(ValueError, lambda: mariadb_server.init({})) diff --git a/tests/unit/test_engine_tineye.py b/tests/unit/test_engine_tineye.py index 7dc8233d4..5aa8c3e82 100644 --- a/tests/unit/test_engine_tineye.py +++ b/tests/unit/test_engine_tineye.py @@ -1,25 +1,25 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring import logging from datetime import datetime from unittest.mock import Mock from requests import HTTPError from parameterized import parameterized + import searx.search import searx.engines from tests import SearxTestCase -class TinEyeTests(SearxTestCase): # pylint: disable=missing-class-docstring +class TinEyeTests(SearxTestCase): - def setUp(self): - searx.search.initialize( - [{'name': 'tineye', 'engine': 'tineye', 'shortcut': 'tin', 'timeout': 9.0, 'disabled': True}] - ) + TEST_SETTINGS = "test_tineye.yml" + def setUp(self): + super().setUp() self.tineye = searx.engines.engines['tineye'] - self.tineye.logger.setLevel(logging.CRITICAL) + self.tineye.logger.setLevel(logging.INFO) def tearDown(self): searx.search.load_engines([]) @@ -33,11 +33,12 @@ class TinEyeTests(SearxTestCase): # pylint: disable=missing-class-docstring @parameterized.expand([(400), (422)]) def test_returns_empty_list(self, status_code): response = Mock() - response.json.return_value = {} + response.json.return_value = {"suggestions": {"key": "Download Error"}} response.status_code = status_code response.raise_for_status.side_effect = HTTPError() - results = self.tineye.response(response) - self.assertEqual(0, len(results)) + with self.assertLogs(self.tineye.logger): + results = self.tineye.response(response) + self.assertEqual(0, len(results)) def test_logs_format_for_422(self): response = Mock() diff --git a/tests/unit/test_engines_init.py b/tests/unit/test_engines_init.py index e2445160a..b91b67162 100644 --- a/tests/unit/test_engines_init.py +++ b/tests/unit/test_engines_init.py @@ -1,16 +1,11 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name from searx import settings, engines from tests import SearxTestCase -class TestEnginesInit(SearxTestCase): # pylint: disable=missing-class-docstring - @classmethod - def tearDownClass(cls): - settings['outgoing']['using_tor_proxy'] = False - settings['outgoing']['extra_proxy_timeout'] = 0 - engines.load_engines([]) +class TestEnginesInit(SearxTestCase): def test_initialize_engines_default(self): engine_list = [ @@ -23,7 +18,7 @@ class TestEnginesInit(SearxTestCase): # pylint: disable=missing-class-docstring self.assertIn('engine1', engines.engines) self.assertIn('engine2', engines.engines) - def test_initialize_engines_exclude_onions(self): # pylint: disable=invalid-name + def test_initialize_engines_exclude_onions(self): settings['outgoing']['using_tor_proxy'] = False engine_list = [ {'engine': 'dummy', 'name': 'engine1', 'shortcut': 'e1', 'categories': 'general'}, @@ -35,7 +30,7 @@ class TestEnginesInit(SearxTestCase): # pylint: disable=missing-class-docstring self.assertIn('engine1', engines.engines) self.assertNotIn('onions', engines.categories) - def test_initialize_engines_include_onions(self): # pylint: disable=invalid-name + def test_initialize_engines_include_onions(self): settings['outgoing']['using_tor_proxy'] = True settings['outgoing']['extra_proxy_timeout'] = 100.0 engine_list = [ diff --git a/tests/unit/test_exceptions.py b/tests/unit/test_exceptions.py index fb2834058..bf9fa08c9 100644 --- a/tests/unit/test_exceptions.py +++ b/tests/unit/test_exceptions.py @@ -1,5 +1,5 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name from parameterized import parameterized from tests import SearxTestCase @@ -7,7 +7,8 @@ import searx.exceptions from searx import get_setting -class TestExceptions(SearxTestCase): # pylint: disable=missing-class-docstring +class TestExceptions(SearxTestCase): + @parameterized.expand( [ searx.exceptions.SearxEngineAccessDeniedException, diff --git a/tests/unit/test_external_bangs.py b/tests/unit/test_external_bangs.py index 153521599..0a911a2f9 100644 --- a/tests/unit/test_external_bangs.py +++ b/tests/unit/test_external_bangs.py @@ -1,5 +1,5 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name from searx.external_bang import ( get_node, @@ -34,7 +34,7 @@ TEST_DB = { } -class TestGetNode(SearxTestCase): # pylint: disable=missing-class-docstring +class TestGetNode(SearxTestCase): DB = { # pylint:disable=invalid-name 'trie': { @@ -65,7 +65,8 @@ class TestGetNode(SearxTestCase): # pylint: disable=missing-class-docstring self.assertEqual(after, 's') -class TestResolveBangDefinition(SearxTestCase): # pylint:disable=missing-class-docstring +class TestResolveBangDefinition(SearxTestCase): + def test_https(self): url, rank = resolve_bang_definition('//example.com/' + chr(2) + chr(1) + '42', 'query') self.assertEqual(url, 'https://example.com/query') @@ -77,7 +78,8 @@ class TestResolveBangDefinition(SearxTestCase): # pylint:disable=missing-class- self.assertEqual(rank, 0) -class TestGetBangDefinitionAndAutocomplete(SearxTestCase): # pylint:disable=missing-class-docstring +class TestGetBangDefinitionAndAutocomplete(SearxTestCase): + def test_found(self): bang_definition, new_autocomplete = get_bang_definition_and_autocomplete('exam', external_bangs_db=TEST_DB) self.assertEqual(bang_definition, TEST_DB['trie']['exam'][LEAF_KEY]) @@ -109,7 +111,8 @@ class TestGetBangDefinitionAndAutocomplete(SearxTestCase): # pylint:disable=mis self.assertEqual(new_autocomplete, []) -class TestExternalBangJson(SearxTestCase): # pylint:disable=missing-class-docstring +class TestExternalBangJson(SearxTestCase): + def test_no_external_bang_query(self): result = get_bang_url(SearchQuery('test', engineref_list=[EngineRef('wikipedia', 'general')])) self.assertIsNone(result) diff --git a/tests/unit/test_locales.py b/tests/unit/test_locales.py index cf22563e0..34b721806 100644 --- a/tests/unit/test_locales.py +++ b/tests/unit/test_locales.py @@ -1,5 +1,6 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name + """Test some code from module :py:obj:`searx.locales`""" from __future__ import annotations diff --git a/tests/unit/test_plugin_calculator.py b/tests/unit/test_plugin_calculator.py index caca0dfe2..f75a565db 100644 --- a/tests/unit/test_plugin_calculator.py +++ b/tests/unit/test_plugin_calculator.py @@ -1,57 +1,57 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name -import flask from parameterized.parameterized import parameterized -from searx import plugins -from searx import preferences + +import searx.plugins +import searx.preferences + +from searx.extended_types import sxng_request +from searx.plugins._core import _default, ModulePlugin +from searx.result_types import Answer +from searx.utils import load_module + from tests import SearxTestCase from .test_utils import random_string - -from .test_plugins import get_search_mock +from .test_plugins import do_post_search -class PluginCalculator(SearxTestCase): # pylint: disable=missing-class-docstring +class PluginCalculator(SearxTestCase): def setUp(self): - from searx import webapp # pylint: disable=import-outside-toplevel + super().setUp() - self.webapp = webapp - self.store = plugins.PluginStore() - plugin = plugins.load_and_initialize_plugin('searx.plugins.calculator', False, (None, {})) - self.store.register(plugin) - self.preferences = preferences.Preferences(["simple"], ["general"], {}, self.store) - self.preferences.parse_dict({"locale": "en"}) + f = _default / "calculator.py" + mod = load_module(f.name, str(f.parent)) + engines = {} - def test_plugin_store_init(self): - self.assertEqual(1, len(self.store.plugins)) - - def test_single_page_number_true(self): - with self.webapp.app.test_request_context(): - flask.request.preferences = self.preferences - search = get_search_mock(query=random_string(10), pageno=2) - result = self.store.call(self.store.plugins, 'post_search', flask.request, search) + self.storage = searx.plugins.PluginStorage() + self.storage.register(ModulePlugin(mod)) + self.storage.init(self.app) + self.pref = searx.preferences.Preferences(["simple"], ["general"], engines, self.storage) + self.pref.parse_dict({"locale": "en"}) - self.assertTrue(result) - self.assertNotIn('calculate', search.result_container.answers) + def test_plugin_store_init(self): + self.assertEqual(1, len(self.storage)) - def test_long_query_true(self): - with self.webapp.app.test_request_context(): - flask.request.preferences = self.preferences - search = get_search_mock(query=random_string(101), pageno=1) - result = self.store.call(self.store.plugins, 'post_search', flask.request, search) + def test_pageno_1_2(self): + with self.app.test_request_context(): + sxng_request.preferences = self.pref + query = "1+1" + answer = Answer(results=[], answer=f"{query} = {eval(query)}") # pylint: disable=eval-used - self.assertTrue(result) - self.assertNotIn('calculate', search.result_container.answers) + search = do_post_search(query, self.storage, pageno=1) + self.assertIn(answer, search.result_container.answers) - def test_alpha_true(self): - with self.webapp.app.test_request_context(): - flask.request.preferences = self.preferences - search = get_search_mock(query=random_string(10), pageno=1) - result = self.store.call(self.store.plugins, 'post_search', flask.request, search) + search = do_post_search(query, self.storage, pageno=2) + self.assertEqual(list(search.result_container.answers), []) - self.assertTrue(result) - self.assertNotIn('calculate', search.result_container.answers) + def test_long_query_ignored(self): + with self.app.test_request_context(): + sxng_request.preferences = self.pref + query = f"1+1 {random_string(101)}" + search = do_post_search(query, self.storage) + self.assertEqual(list(search.result_container.answers), []) @parameterized.expand( [ @@ -77,27 +77,22 @@ class PluginCalculator(SearxTestCase): # pylint: disable=missing-class-docstrin ("1,0^1,0", "1", "de"), ] ) - def test_localized_query(self, operation: str, contains_result: str, lang: str): - with self.webapp.app.test_request_context(): - self.preferences.parse_dict({"locale": lang}) - flask.request.preferences = self.preferences - search = get_search_mock(query=operation, lang=lang, pageno=1) - result = self.store.call(self.store.plugins, 'post_search', flask.request, search) + def test_localized_query(self, query: str, res: str, lang: str): + with self.app.test_request_context(): + self.pref.parse_dict({"locale": lang}) + sxng_request.preferences = self.pref + answer = Answer(results=[], answer=f"{query} = {res}") - self.assertTrue(result) - self.assertIn('calculate', search.result_container.answers) - self.assertIn(contains_result, search.result_container.answers['calculate']['answer']) + search = do_post_search(query, self.storage) + self.assertIn(answer, search.result_container.answers) @parameterized.expand( [ "1/0", ] ) - def test_invalid_operations(self, operation): - with self.webapp.app.test_request_context(): - flask.request.preferences = self.preferences - search = get_search_mock(query=operation, pageno=1) - result = self.store.call(self.store.plugins, 'post_search', flask.request, search) - - self.assertTrue(result) - self.assertNotIn('calculate', search.result_container.answers) + def test_invalid_operations(self, query): + with self.app.test_request_context(): + sxng_request.preferences = self.pref + search = do_post_search(query, self.storage) + self.assertEqual(list(search.result_container.answers), []) diff --git a/tests/unit/test_plugin_hash.py b/tests/unit/test_plugin_hash.py index 5c55d7ff1..f7da3c67f 100644 --- a/tests/unit/test_plugin_hash.py +++ b/tests/unit/test_plugin_hash.py @@ -1,51 +1,69 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring, invalid-name +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name -from mock import Mock from parameterized.parameterized import parameterized -from searx import plugins + +import searx.plugins +import searx.preferences + +from searx.extended_types import sxng_request +from searx.result_types import Answer + from tests import SearxTestCase +from .test_plugins import do_post_search -from .test_plugins import get_search_mock +query_res = [ + ("md5 test", "md5 hash digest: 098f6bcd4621d373cade4e832627b4f6"), + ("sha1 test", "sha1 hash digest: a94a8fe5ccb19ba61c4c0873d391e987982fbbd3"), + ("sha224 test", "sha224 hash digest: 90a3ed9e32b2aaf4c61c410eb925426119e1a9dc53d4286ade99a809"), + ("sha256 test", "sha256 hash digest: 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"), + ( + "sha384 test", + "sha384 hash digest: 768412320f7b0aa5812fce428dc4706b3c" + "ae50e02a64caa16a782249bfe8efc4b7ef1ccb126255d196047dfedf1" + "7a0a9", + ), + ( + "sha512 test", + "sha512 hash digest: ee26b0dd4af7e749aa1a8ee3c10ae9923f6" + "18980772e473f8819a5d4940e0db27ac185f8a0e1d5f84f88bc887fd67b143732c304cc5" + "fa9ad8e6f57f50028a8ff", + ), +] -class PluginHashTest(SearxTestCase): # pylint: disable=missing-class-docstring +class PluginHashTest(SearxTestCase): + def setUp(self): - self.store = plugins.PluginStore() - plugin = plugins.load_and_initialize_plugin('searx.plugins.hash_plugin', False, (None, {})) - self.store.register(plugin) + super().setUp() + engines = {} + + self.storage = searx.plugins.PluginStorage() + self.storage.register_by_fqn("searx.plugins.hash_plugin.SXNGPlugin") + self.storage.init(self.app) + self.pref = searx.preferences.Preferences(["simple"], ["general"], engines, self.storage) + self.pref.parse_dict({"locale": "en"}) def test_plugin_store_init(self): - self.assertEqual(1, len(self.store.plugins)) - - @parameterized.expand( - [ - ('md5 test', 'md5 hash digest: 098f6bcd4621d373cade4e832627b4f6'), - ('sha1 test', 'sha1 hash digest: a94a8fe5ccb19ba61c4c0873d391e987982fbbd3'), - ('sha224 test', 'sha224 hash digest: 90a3ed9e32b2aaf4c61c410eb925426119e1a9dc53d4286ade99a809'), - ('sha256 test', 'sha256 hash digest: 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08'), - ( - 'sha384 test', - 'sha384 hash digest: 768412320f7b0aa5812fce428dc4706b3c' - 'ae50e02a64caa16a782249bfe8efc4b7ef1ccb126255d196047dfedf1' - '7a0a9', - ), - ( - 'sha512 test', - 'sha512 hash digest: ee26b0dd4af7e749aa1a8ee3c10ae9923f6' - '18980772e473f8819a5d4940e0db27ac185f8a0e1d5f84f88bc887fd67b143732c304cc5' - 'fa9ad8e6f57f50028a8ff', - ), - ] - ) - def test_hash_digest_new(self, query: str, hash_str: str): - request = Mock(remote_addr='127.0.0.1') - search = get_search_mock(query=query, pageno=1) - self.store.call(self.store.plugins, 'post_search', request, search) - self.assertIn(hash_str, search.result_container.answers['hash']['answer']) - - def test_md5_bytes_no_answer(self): - request = Mock(remote_addr='127.0.0.1') - search = get_search_mock(query=b'md5 test', pageno=2) - self.store.call(self.store.plugins, 'post_search', request, search) - self.assertNotIn('hash', search.result_container.answers) + self.assertEqual(1, len(self.storage)) + + @parameterized.expand(query_res) + def test_hash_digest_new(self, query: str, res: str): + with self.app.test_request_context(): + sxng_request.preferences = self.pref + answer = Answer(results=[], answer=res) + + search = do_post_search(query, self.storage) + self.assertIn(answer, search.result_container.answers) + + def test_pageno_1_2(self): + with self.app.test_request_context(): + sxng_request.preferences = self.pref + query, res = query_res[0] + answer = Answer(results=[], answer=res) + + search = do_post_search(query, self.storage, pageno=1) + self.assertIn(answer, search.result_container.answers) + + search = do_post_search(query, self.storage, pageno=2) + self.assertEqual(list(search.result_container.answers), []) diff --git a/tests/unit/test_plugin_self_info.py b/tests/unit/test_plugin_self_info.py index 1196e908c..701d02c18 100644 --- a/tests/unit/test_plugin_self_info.py +++ b/tests/unit/test_plugin_self_info.py @@ -1,65 +1,69 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring, invalid-name +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name -from mock import Mock from parameterized.parameterized import parameterized -from searx import ( - plugins, - limiter, - botdetection, -) +from flask_babel import gettext + +import searx.plugins +import searx.preferences +import searx.limiter +import searx.botdetection + +from searx.extended_types import sxng_request +from searx.result_types import Answer + from tests import SearxTestCase -from .test_plugins import get_search_mock +from .test_plugins import do_post_search + +class PluginIPSelfInfo(SearxTestCase): -class PluginIPSelfInfo(SearxTestCase): # pylint: disable=missing-class-docstring def setUp(self): - plugin = plugins.load_and_initialize_plugin('searx.plugins.self_info', False, (None, {})) - self.store = plugins.PluginStore() - self.store.register(plugin) - cfg = limiter.get_cfg() - botdetection.init(cfg, None) + super().setUp() + + self.storage = searx.plugins.PluginStorage() + self.storage.register_by_fqn("searx.plugins.self_info.SXNGPlugin") + self.storage.init(self.app) + self.pref = searx.preferences.Preferences(["simple"], ["general"], {}, self.storage) + self.pref.parse_dict({"locale": "en"}) + cfg = searx.limiter.get_cfg() + searx.botdetection.init(cfg, None) def test_plugin_store_init(self): - self.assertEqual(1, len(self.store.plugins)) - - def test_ip_in_answer(self): - request = Mock() - request.remote_addr = '127.0.0.1' - request.headers = {'X-Forwarded-For': '1.2.3.4, 127.0.0.1', 'X-Real-IP': '127.0.0.1'} - search = get_search_mock(query='ip', pageno=1) - self.store.call(self.store.plugins, 'post_search', request, search) - self.assertIn('127.0.0.1', search.result_container.answers["ip"]["answer"]) - - def test_ip_not_in_answer(self): - request = Mock() - request.remote_addr = '127.0.0.1' - request.headers = {'X-Forwarded-For': '1.2.3.4, 127.0.0.1', 'X-Real-IP': '127.0.0.1'} - search = get_search_mock(query='ip', pageno=2) - self.store.call(self.store.plugins, 'post_search', request, search) - self.assertNotIn('ip', search.result_container.answers) + self.assertEqual(1, len(self.storage)) + + def test_pageno_1_2(self): + + with self.app.test_request_context(): + sxng_request.preferences = self.pref + sxng_request.remote_addr = "127.0.0.1" + sxng_request.headers = {"X-Forwarded-For": "1.2.3.4, 127.0.0.1", "X-Real-IP": "127.0.0.1"} # type: ignore + answer = Answer(results=[], answer=gettext("Your IP is: ") + "127.0.0.1") + + search = do_post_search("ip", self.storage, pageno=1) + self.assertIn(answer, search.result_container.answers) + + search = do_post_search("ip", self.storage, pageno=2) + self.assertEqual(list(search.result_container.answers), []) @parameterized.expand( [ - 'user-agent', - 'What is my User-Agent?', + "user-agent", + "USER-AgenT lorem ipsum", ] ) def test_user_agent_in_answer(self, query: str): - request = Mock(user_agent=Mock(string='Mock')) - search = get_search_mock(query=query, pageno=1) - self.store.call(self.store.plugins, 'post_search', request, search) - self.assertIn('Mock', search.result_container.answers["user-agent"]["answer"]) - @parameterized.expand( - [ - 'user-agent', - 'What is my User-Agent?', - ] - ) - def test_user_agent_not_in_answer(self, query: str): - request = Mock(user_agent=Mock(string='Mock')) - search = get_search_mock(query=query, pageno=2) - self.store.call(self.store.plugins, 'post_search', request, search) - self.assertNotIn('user-agent', search.result_container.answers) + query = "user-agent" + + with self.app.test_request_context(): + sxng_request.preferences = self.pref + sxng_request.user_agent = "Dummy agent" # type: ignore + answer = Answer(results=[], answer=gettext("Your user-agent is: ") + "Dummy agent") + + search = do_post_search(query, self.storage, pageno=1) + self.assertIn(answer, search.result_container.answers) + + search = do_post_search(query, self.storage, pageno=2) + self.assertEqual(list(search.result_container.answers), []) diff --git a/tests/unit/test_plugins.py b/tests/unit/test_plugins.py index 86b5ce930..1096894b4 100644 --- a/tests/unit/test_plugins.py +++ b/tests/unit/test_plugins.py @@ -1,50 +1,106 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name import babel from mock import Mock -from searx import plugins + +import searx.plugins +import searx.preferences +import searx.results + +from searx.result_types import Result +from searx.extended_types import sxng_request + from tests import SearxTestCase +plg_store = searx.plugins.PluginStorage() +plg_store.load_builtins() + def get_search_mock(query, **kwargs): + lang = kwargs.get("lang", "en-US") + kwargs["pageno"] = kwargs.get("pageno", 1) kwargs["locale"] = babel.Locale.parse(lang, sep="-") - return Mock(search_query=Mock(query=query, **kwargs), result_container=Mock(answers={})) + user_plugins = kwargs.pop("user_plugins", [x.id for x in plg_store]) + + return Mock( + search_query=Mock(query=query, **kwargs), + user_plugins=user_plugins, + result_container=searx.results.ResultContainer(), + ) + + +def do_pre_search(query, storage, **kwargs) -> bool: + + search = get_search_mock(query, **kwargs) + ret = storage.pre_search(sxng_request, search) + return ret + + +def do_post_search(query, storage, **kwargs) -> Mock: + + search = get_search_mock(query, **kwargs) + storage.post_search(sxng_request, search) + return search -class PluginMock: # pylint: disable=missing-class-docstring, too-few-public-methods - default_on = False - name = 'Default plugin' - description = 'Default plugin description' +class PluginMock(searx.plugins.Plugin): + def __init__(self, _id: str, name: str, default_on: bool): + self.id = _id + self.default_on = default_on + self._name = name + super().__init__() + + # pylint: disable= unused-argument + def pre_search(self, request, search) -> bool: + return True + + def post_search(self, request, search) -> None: + return None + + def on_result(self, request, search, result) -> bool: + return False + + def info(self): + return searx.plugins.PluginInfo( + id=self.id, + name=self._name, + description=f"Dummy plugin: {self.id}", + preference_section="general", + ) + + +class PluginStorage(SearxTestCase): -class PluginStoreTest(SearxTestCase): # pylint: disable=missing-class-docstring def setUp(self): - self.store = plugins.PluginStore() + super().setUp() + engines = {} + + self.storage = searx.plugins.PluginStorage() + self.storage.register(PluginMock("plg001", "first plugin", True)) + self.storage.register(PluginMock("plg002", "second plugin", True)) + self.storage.init(self.app) + self.pref = searx.preferences.Preferences(["simple"], ["general"], engines, self.storage) + self.pref.parse_dict({"locale": "en"}) def test_init(self): - self.assertEqual(0, len(self.store.plugins)) - self.assertIsInstance(self.store.plugins, list) - - def test_register(self): - testplugin = PluginMock() - self.store.register(testplugin) - self.assertEqual(1, len(self.store.plugins)) - - def test_call_empty(self): - testplugin = PluginMock() - self.store.register(testplugin) - setattr(testplugin, 'asdf', Mock()) - request = Mock() - self.store.call([], 'asdf', request, Mock()) - self.assertFalse(getattr(testplugin, 'asdf').called) # pylint: disable=E1101 - - def test_call_with_plugin(self): - store = plugins.PluginStore() - testplugin = PluginMock() - store.register(testplugin) - setattr(testplugin, 'asdf', Mock()) - request = Mock() - store.call([testplugin], 'asdf', request, Mock()) - self.assertTrue(getattr(testplugin, 'asdf').called) # pylint: disable=E1101 + + self.assertEqual(2, len(self.storage)) + + def test_hooks(self): + + with self.app.test_request_context(): + sxng_request.preferences = self.pref + query = "" + + ret = do_pre_search(query, self.storage, pageno=1) + self.assertTrue(ret is True) + + ret = self.storage.on_result( + sxng_request, + get_search_mock("lorem ipsum", user_plugins=["plg001", "plg002"]), + Result(results=[]), + ) + self.assertFalse(ret) diff --git a/tests/unit/test_preferences.py b/tests/unit/test_preferences.py index 92abe81ee..d7009cbfd 100644 --- a/tests/unit/test_preferences.py +++ b/tests/unit/test_preferences.py @@ -1,9 +1,9 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring, invalid-name +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name import flask from mock import Mock -from tests import SearxTestCase + from searx import favicons from searx.locales import locales_initialize from searx.preferences import ( @@ -15,20 +15,19 @@ from searx.preferences import ( PluginsSetting, ValidationException, ) -from searx.plugins import Plugin +import searx.plugins from searx.preferences import Preferences +from tests import SearxTestCase +from .test_plugins import PluginMock + + locales_initialize() favicons.init() -class PluginStub(Plugin): # pylint: disable=missing-class-docstring, too-few-public-methods - def __init__(self, plugin_id, default_on): - self.id = plugin_id - self.default_on = default_on +class TestSettings(SearxTestCase): - -class TestSettings(SearxTestCase): # pylint: disable=missing-class-docstring # map settings def test_map_setting_invalid_default_value(self): @@ -93,6 +92,7 @@ class TestSettings(SearxTestCase): # pylint: disable=missing-class-docstring self.assertEqual(setting.get_value(), ['2']) # search language settings + def test_lang_setting_valid_choice(self): setting = SearchLanguageSetting('all', choices=['all', 'de', 'en']) setting.parse('de') @@ -114,23 +114,30 @@ class TestSettings(SearxTestCase): # pylint: disable=missing-class-docstring self.assertEqual(setting.get_value(), 'es-ES') # plugins settings + def test_plugins_setting_all_default_enabled(self): - plugin1 = PluginStub('plugin1', True) - plugin2 = PluginStub('plugin2', True) - setting = PluginsSetting(['3'], plugins=[plugin1, plugin2]) - self.assertEqual(set(setting.get_enabled()), set(['plugin1', 'plugin2'])) + storage = searx.plugins.PluginStorage() + storage.register(PluginMock("plg001", "first plugin", True)) + storage.register(PluginMock("plg002", "second plugin", True)) + plgs_settings = PluginsSetting(False, storage) + self.assertEqual(set(plgs_settings.get_enabled()), {"plg001", "plg002"}) def test_plugins_setting_few_default_enabled(self): - plugin1 = PluginStub('plugin1', True) - plugin2 = PluginStub('plugin2', False) - plugin3 = PluginStub('plugin3', True) - setting = PluginsSetting('name', plugins=[plugin1, plugin2, plugin3]) - self.assertEqual(set(setting.get_enabled()), set(['plugin1', 'plugin3'])) + storage = searx.plugins.PluginStorage() + storage.register(PluginMock("plg001", "first plugin", True)) + storage.register(PluginMock("plg002", "second plugin", False)) + storage.register(PluginMock("plg003", "third plugin", True)) + plgs_settings = PluginsSetting(False, storage) + self.assertEqual(set(plgs_settings.get_enabled()), set(['plg001', 'plg003'])) -class TestPreferences(SearxTestCase): # pylint: disable=missing-class-docstring +class TestPreferences(SearxTestCase): + def setUp(self): - self.preferences = Preferences(['simple'], ['general'], {}, []) + super().setUp() + + storage = searx.plugins.PluginStorage() + self.preferences = Preferences(['simple'], ['general'], {}, storage) def test_encode(self): url_params = ( diff --git a/tests/unit/test_query.py b/tests/unit/test_query.py index 00c53edc7..c979a8493 100644 --- a/tests/unit/test_query.py +++ b/tests/unit/test_query.py @@ -1,25 +1,13 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name from parameterized.parameterized import parameterized -import searx.search from searx.query import RawTextQuery from tests import SearxTestCase -TEST_ENGINES = [ - { - 'name': 'dummy engine', - 'engine': 'dummy', - 'categories': 'general', - 'shortcut': 'du', - 'timeout': 3.0, - 'tokens': [], - }, -] +class TestQuery(SearxTestCase): - -class TestQuery(SearxTestCase): # pylint:disable=missing-class-docstring def test_simple_query(self): query_text = 'the query' query = RawTextQuery(query_text, []) @@ -59,7 +47,8 @@ class TestQuery(SearxTestCase): # pylint:disable=missing-class-docstring self.assertEqual(query.getFullQuery(), '<8 another text') -class TestLanguageParser(SearxTestCase): # pylint:disable=missing-class-docstring +class TestLanguageParser(SearxTestCase): + def test_language_code(self): language = 'es-ES' query_text = 'the query' @@ -143,7 +132,8 @@ class TestLanguageParser(SearxTestCase): # pylint:disable=missing-class-docstri self.assertEqual(query.autocomplete_list, autocomplete_list) -class TestTimeoutParser(SearxTestCase): # pylint:disable=missing-class-docstring +class TestTimeoutParser(SearxTestCase): + @parameterized.expand( [ ('<3 the query', 3), @@ -182,7 +172,8 @@ class TestTimeoutParser(SearxTestCase): # pylint:disable=missing-class-docstrin self.assertEqual(query.autocomplete_list, ['<3', '<850']) -class TestExternalBangParser(SearxTestCase): # pylint:disable=missing-class-docstring +class TestExternalBangParser(SearxTestCase): + def test_external_bang(self): query_text = '!!ddg the query' query = RawTextQuery(query_text, []) @@ -212,17 +203,11 @@ class TestExternalBangParser(SearxTestCase): # pylint:disable=missing-class-doc self.assertEqual(query.get_autocomplete_full_query(a), a + ' the query') -class TestBang(SearxTestCase): # pylint:disable=missing-class-docstring +class TestBang(SearxTestCase): - SPECIFIC_BANGS = ['!dummy_engine', '!du', '!general'] + SPECIFIC_BANGS = ['!dummy_engine', '!gd', '!general'] THE_QUERY = 'the query' - def setUp(self): - searx.search.initialize(TEST_ENGINES) - - def tearDown(self): - searx.search.load_engines([]) - @parameterized.expand(SPECIFIC_BANGS) def test_bang(self, bang: str): with self.subTest(msg="Check bang", bang=bang): @@ -246,7 +231,7 @@ class TestBang(SearxTestCase): # pylint:disable=missing-class-docstring def test_bang_autocomplete(self): query = RawTextQuery('the query !dum', []) - self.assertEqual(query.autocomplete_list, ['!dummy_engine']) + self.assertEqual(query.autocomplete_list, ['!dummy_engine', '!dummy_private_engine']) query = RawTextQuery('!dum the query', []) self.assertEqual(query.autocomplete_list, []) diff --git a/tests/unit/test_results.py b/tests/unit/test_results.py index 740d36a03..6560f7b5c 100644 --- a/tests/unit/test_results.py +++ b/tests/unit/test_results.py @@ -1,75 +1,56 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name + +from searx.result_types import LegacyResult from searx.results import ResultContainer -import searx.search from tests import SearxTestCase -def make_test_engine_dict(**kwargs) -> dict: - test_engine = { - # fmt: off - 'name': None, - 'engine': None, - 'categories': 'general', - 'shortcut': 'dummy', - 'timeout': 3.0, - 'tokens': [], - # fmt: on - } - - test_engine.update(**kwargs) - return test_engine - - -def fake_result(url='https://aa.bb/cc?dd=ee#ff', title='aaa', content='bbb', engine='wikipedia', **kwargs): - result = { - # fmt: off - 'url': url, - 'title': title, - 'content': content, - 'engine': engine, - # fmt: on - } - result.update(kwargs) - return result - - -class ResultContainerTestCase(SearxTestCase): # pylint: disable=missing-class-docstring - - def setUp(self) -> None: - stract_engine = make_test_engine_dict(name="stract", engine="stract", shortcut="stra") - duckduckgo_engine = make_test_engine_dict(name="duckduckgo", engine="duckduckgo", shortcut="ddg") - mojeek_engine = make_test_engine_dict(name="mojeek", engine="mojeek", shortcut="mjk") - searx.search.initialize([stract_engine, duckduckgo_engine, mojeek_engine]) - self.container = ResultContainer() +class ResultContainerTestCase(SearxTestCase): + # pylint: disable=use-dict-literal - def tearDown(self): - searx.search.load_engines([]) + TEST_SETTINGS = "test_result_container.yml" def test_empty(self): - self.assertEqual(self.container.get_ordered_results(), []) + container = ResultContainer() + self.assertEqual(container.get_ordered_results(), []) def test_one_result(self): - self.container.extend('wikipedia', [fake_result()]) + result = dict(url="https://example.org", title="title ..", content="Lorem ..") - self.assertEqual(self.container.results_length(), 1) + container = ResultContainer() + container.extend("google", [result]) + container.close() - def test_one_suggestion(self): - self.container.extend('wikipedia', [fake_result(suggestion=True)]) - - self.assertEqual(len(self.container.suggestions), 1) - self.assertEqual(self.container.results_length(), 0) - - def test_result_merge(self): - self.container.extend('wikipedia', [fake_result()]) - self.container.extend('wikidata', [fake_result(), fake_result(url='https://example.com/')]) + self.assertEqual(container.results_length(), 1) + self.assertIn(LegacyResult(result), container.get_ordered_results()) - self.assertEqual(self.container.results_length(), 2) - - def test_result_merge_by_title(self): - self.container.extend('stract', [fake_result(engine='stract', title='short title')]) - self.container.extend('duckduckgo', [fake_result(engine='duckduckgo', title='normal title')]) - self.container.extend('mojeek', [fake_result(engine='mojeek', title='this long long title')]) - - self.assertEqual(self.container.get_ordered_results()[0].get('title', ''), 'this long long title') + def test_one_suggestion(self): + result = dict(suggestion="lorem ipsum ..") + + container = ResultContainer() + container.extend("duckduckgo", [result]) + container.close() + + self.assertEqual(container.results_length(), 0) + self.assertEqual(len(container.suggestions), 1) + self.assertIn(result["suggestion"], container.suggestions) + + def test_merge_url_result(self): + # from the merge of eng1 and eng2 we expect this result + result = LegacyResult( + url="https://example.org", title="very long title, lorem ipsum", content="Lorem ipsum dolor sit amet .." + ) + eng1 = dict(url=result.url, title="short title", content=result.content, engine="google") + eng2 = dict(url="http://example.org", title=result.title, content="lorem ipsum", engine="duckduckgo") + + container = ResultContainer() + container.extend(None, [eng1, eng2]) + container.close() + + result_list = container.get_ordered_results() + self.assertEqual(container.results_length(), 1) + self.assertIn(result, result_list) + self.assertEqual(result_list[0].title, result.title) + self.assertEqual(result_list[0].content, result.content) diff --git a/tests/unit/test_search.py b/tests/unit/test_search.py index be95fb08e..69b997c89 100644 --- a/tests/unit/test_search.py +++ b/tests/unit/test_search.py @@ -1,8 +1,7 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring, invalid-name +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name from copy import copy -import logging import searx.search from searx.search import SearchQuery, EngineRef @@ -12,20 +11,11 @@ from tests import SearxTestCase SAFESEARCH = 0 PAGENO = 1 -PUBLIC_ENGINE_NAME = 'general dummy' -TEST_ENGINES = [ - { - 'name': PUBLIC_ENGINE_NAME, - 'engine': 'dummy', - 'categories': 'general', - 'shortcut': 'gd', - 'timeout': 3.0, - 'tokens': [], - }, -] - - -class SearchQueryTestCase(SearxTestCase): # pylint: disable=missing-class-docstring +PUBLIC_ENGINE_NAME = "dummy engine" # from the ./settings/test_settings.yml + + +class SearchQueryTestCase(SearxTestCase): + def test_repr(self): s = SearchQuery('test', [EngineRef('bing', 'general')], 'all', 0, 1, '1', 5.0, 'g') self.assertEqual( @@ -44,21 +34,7 @@ class SearchQueryTestCase(SearxTestCase): # pylint: disable=missing-class-docst self.assertEqual(s, t) -class SearchTestCase(SearxTestCase): # pylint: disable=missing-class-docstring - def setUp(self): - - log = logging.getLogger("searx") - log_lev = log.level - log.setLevel(logging.ERROR) - from searx import webapp # pylint: disable=import-outside-toplevel - - log.setLevel(log_lev) - - self.app = webapp.app - - @classmethod - def setUpClass(cls): - searx.search.initialize(TEST_ENGINES) +class SearchTestCase(SearxTestCase): def test_timeout_simple(self): settings['outgoing']['max_request_timeout'] = None diff --git a/tests/unit/test_settings_loader.py b/tests/unit/test_settings_loader.py index 99baee1ca..e6ede27aa 100644 --- a/tests/unit/test_settings_loader.py +++ b/tests/unit/test_settings_loader.py @@ -1,5 +1,5 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name from pathlib import Path @@ -17,7 +17,8 @@ def _settings(f_name): return str(Path(__file__).parent.absolute() / "settings" / f_name) -class TestLoad(SearxTestCase): # pylint: disable=missing-class-docstring +class TestLoad(SearxTestCase): + def test_load_zero(self): with self.assertRaises(SearxSettingsException): settings_loader.load_yaml('/dev/zero') @@ -28,7 +29,8 @@ class TestLoad(SearxTestCase): # pylint: disable=missing-class-docstring self.assertEqual(settings_loader.load_yaml(_settings("empty_settings.yml")), {}) -class TestDefaultSettings(SearxTestCase): # pylint: disable=missing-class-docstring +class TestDefaultSettings(SearxTestCase): + def test_load(self): settings, msg = settings_loader.load_settings(load_user_settings=False) self.assertTrue(msg.startswith('load the default settings from')) @@ -42,7 +44,8 @@ class TestDefaultSettings(SearxTestCase): # pylint: disable=missing-class-docst self.assertIsInstance(settings['default_doi_resolver'], str) -class TestUserSettings(SearxTestCase): # pylint: disable=missing-class-docstring +class TestUserSettings(SearxTestCase): + def test_is_use_default_settings(self): self.assertFalse(settings_loader.is_use_default_settings({})) self.assertTrue(settings_loader.is_use_default_settings({'use_default_settings': True})) diff --git a/tests/unit/test_toml.py b/tests/unit/test_toml.py index 766a2bb97..7a298502b 100644 --- a/tests/unit/test_toml.py +++ b/tests/unit/test_toml.py @@ -1,12 +1,12 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name from tests import SearxTestCase from searx import compat from searx.favicons.config import DEFAULT_CFG_TOML_PATH -class CompatTest(SearxTestCase): # pylint: disable=missing-class-docstring +class CompatTest(SearxTestCase): def test_toml(self): with DEFAULT_CFG_TOML_PATH.open("rb") as f: diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py index 4c7ebaec7..55352914b 100644 --- a/tests/unit/test_utils.py +++ b/tests/unit/test_utils.py @@ -1,5 +1,5 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring, invalid-name +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name import random import string @@ -16,7 +16,8 @@ def random_string(length, choices=string.ascii_letters): return ''.join(random.choice(choices) for _ in range(length)) -class TestUtils(SearxTestCase): # pylint: disable=missing-class-docstring +class TestUtils(SearxTestCase): + def test_gen_useragent(self): self.assertIsInstance(utils.gen_useragent(), str) self.assertIsNotNone(utils.gen_useragent()) @@ -109,7 +110,10 @@ class TestUtils(SearxTestCase): # pylint: disable=missing-class-docstring class TestHTMLTextExtractor(SearxTestCase): # pylint: disable=missing-class-docstring + def setUp(self): + super().setUp() + self.html_text_extractor = utils._HTMLTextExtractor() # pylint: disable=protected-access def test__init__(self): diff --git a/tests/unit/test_webadapter.py b/tests/unit/test_webadapter.py index 493020859..4519600a9 100644 --- a/tests/unit/test_webadapter.py +++ b/tests/unit/test_webadapter.py @@ -1,52 +1,38 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name -from searx.preferences import Preferences -from searx.engines import engines +import searx.plugins -import searx.search +from searx.engines import engines +from searx.preferences import Preferences from searx.search import EngineRef from searx.webadapter import validate_engineref_list + from tests import SearxTestCase +PRIVATE_ENGINE_NAME = "dummy private engine" # from the ./settings/test_settings.yml +SEARCHQUERY = [EngineRef(PRIVATE_ENGINE_NAME, "general")] + + +class ValidateQueryCase(SearxTestCase): -PRIVATE_ENGINE_NAME = 'general private offline' -TEST_ENGINES = [ - { - 'name': PRIVATE_ENGINE_NAME, - 'engine': 'dummy-offline', - 'categories': 'general', - 'shortcut': 'do', - 'timeout': 3.0, - 'engine_type': 'offline', - 'tokens': ['my-token'], - }, -] -SEARCHQUERY = [EngineRef(PRIVATE_ENGINE_NAME, 'general')] - - -class ValidateQueryCase(SearxTestCase): # pylint: disable=missing-class-docstring - @classmethod - def setUpClass(cls): - searx.search.initialize(TEST_ENGINES) - - def test_query_private_engine_without_token(self): # pylint:disable=invalid-name - preferences = Preferences(['simple'], ['general'], engines, []) + def test_without_token(self): + preferences = Preferences(['simple'], ['general'], engines, searx.plugins.STORAGE) valid, unknown, invalid_token = validate_engineref_list(SEARCHQUERY, preferences) self.assertEqual(len(valid), 0) self.assertEqual(len(unknown), 0) self.assertEqual(len(invalid_token), 1) - def test_query_private_engine_with_incorrect_token(self): # pylint:disable=invalid-name - preferences_with_tokens = Preferences(['simple'], ['general'], engines, []) + def test_with_incorrect_token(self): + preferences_with_tokens = Preferences(['simple'], ['general'], engines, searx.plugins.STORAGE) preferences_with_tokens.parse_dict({'tokens': 'bad-token'}) valid, unknown, invalid_token = validate_engineref_list(SEARCHQUERY, preferences_with_tokens) self.assertEqual(len(valid), 0) self.assertEqual(len(unknown), 0) self.assertEqual(len(invalid_token), 1) - def test_query_private_engine_with_correct_token(self): # pylint:disable=invalid-name - preferences_with_tokens = Preferences(['simple'], ['general'], engines, []) + def test_with_correct_token(self): + preferences_with_tokens = Preferences(['simple'], ['general'], engines, searx.plugins.STORAGE) preferences_with_tokens.parse_dict({'tokens': 'my-token'}) valid, unknown, invalid_token = validate_engineref_list(SEARCHQUERY, preferences_with_tokens) self.assertEqual(len(valid), 1) diff --git a/tests/unit/test_webapp.py b/tests/unit/test_webapp.py index 65705d164..12c86e4f0 100644 --- a/tests/unit/test_webapp.py +++ b/tests/unit/test_webapp.py @@ -1,41 +1,33 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name -import logging import json from urllib.parse import ParseResult import babel from mock import Mock -from searx.results import Timing +import searx.webapp +import searx.search import searx.search.processors -from searx.search import Search + +from searx.results import Timing from searx.preferences import Preferences from tests import SearxTestCase -class ViewsTestCase(SearxTestCase): # pylint: disable=missing-class-docstring, too-many-public-methods +class ViewsTestCase(SearxTestCase): # pylint: disable=too-many-public-methods + def setUp(self): + super().setUp() + # skip init function (no external HTTP request) def dummy(*args, **kwargs): # pylint: disable=unused-argument pass self.setattr4test(searx.search.processors, 'initialize_processor', dummy) - - log = logging.getLogger("searx") - log_lev = log.level - log.setLevel(logging.ERROR) - from searx import webapp # pylint: disable=import-outside-toplevel - - log.setLevel(log_lev) - - webapp.app.config['TESTING'] = True # to get better error messages - self.app = webapp.app.test_client() - - # remove sha for the static file - # so the tests don't have to care about the changing URLs - for k in webapp.static_files: - webapp.static_files[k] = None + # remove sha for the static file so the tests don't have to care about + # the changing URLs + self.setattr4test(searx.webapp, 'static_files', {}) # set some defaults test_results = [ @@ -85,7 +77,7 @@ class ViewsTestCase(SearxTestCase): # pylint: disable=missing-class-docstring, ) search_self.search_query.locale = babel.Locale.parse("en-US", sep='-') - self.setattr4test(Search, 'search', search_mock) + self.setattr4test(searx.search.Search, 'search', search_mock) original_preferences_get_value = Preferences.get_value @@ -100,7 +92,7 @@ class ViewsTestCase(SearxTestCase): # pylint: disable=missing-class-docstring, self.maxDiff = None # pylint: disable=invalid-name def test_index_empty(self): - result = self.app.post('/') + result = self.client.post('/') self.assertEqual(result.status_code, 200) self.assertIn( b'<div class="title"><h1>SearXNG</h1></div>', @@ -108,34 +100,34 @@ class ViewsTestCase(SearxTestCase): # pylint: disable=missing-class-docstring, ) def test_index_html_post(self): - result = self.app.post('/', data={'q': 'test'}) + result = self.client.post('/', data={'q': 'test'}) self.assertEqual(result.status_code, 308) self.assertEqual(result.location, '/search') def test_index_html_get(self): - result = self.app.post('/?q=test') + result = self.client.post('/?q=test') self.assertEqual(result.status_code, 308) self.assertEqual(result.location, '/search?q=test') def test_search_empty_html(self): - result = self.app.post('/search', data={'q': ''}) + result = self.client.post('/search', data={'q': ''}) self.assertEqual(result.status_code, 200) self.assertIn(b'<div class="title"><h1>SearXNG</h1></div>', result.data) def test_search_empty_json(self): - result = self.app.post('/search', data={'q': '', 'format': 'json'}) + result = self.client.post('/search', data={'q': '', 'format': 'json'}) self.assertEqual(result.status_code, 400) def test_search_empty_csv(self): - result = self.app.post('/search', data={'q': '', 'format': 'csv'}) + result = self.client.post('/search', data={'q': '', 'format': 'csv'}) self.assertEqual(result.status_code, 400) def test_search_empty_rss(self): - result = self.app.post('/search', data={'q': '', 'format': 'rss'}) + result = self.client.post('/search', data={'q': '', 'format': 'rss'}) self.assertEqual(result.status_code, 400) def test_search_html(self): - result = self.app.post('/search', data={'q': 'test'}) + result = self.client.post('/search', data={'q': 'test'}) self.assertIn( b'<span class="url_o1"><span class="url_i1">http://second.test.xyz</span></span>', @@ -147,11 +139,11 @@ class ViewsTestCase(SearxTestCase): # pylint: disable=missing-class-docstring, ) def test_index_json(self): - result = self.app.post('/', data={'q': 'test', 'format': 'json'}) + result = self.client.post('/', data={'q': 'test', 'format': 'json'}) self.assertEqual(result.status_code, 308) def test_search_json(self): - result = self.app.post('/search', data={'q': 'test', 'format': 'json'}) + result = self.client.post('/search', data={'q': 'test', 'format': 'json'}) result_dict = json.loads(result.data.decode()) self.assertEqual('test', result_dict['query']) @@ -160,11 +152,11 @@ class ViewsTestCase(SearxTestCase): # pylint: disable=missing-class-docstring, self.assertEqual(result_dict['results'][0]['url'], 'http://first.test.xyz') def test_index_csv(self): - result = self.app.post('/', data={'q': 'test', 'format': 'csv'}) + result = self.client.post('/', data={'q': 'test', 'format': 'csv'}) self.assertEqual(result.status_code, 308) def test_search_csv(self): - result = self.app.post('/search', data={'q': 'test', 'format': 'csv'}) + result = self.client.post('/search', data={'q': 'test', 'format': 'csv'}) self.assertEqual( b'title,url,content,host,engine,score,type\r\n' @@ -174,11 +166,11 @@ class ViewsTestCase(SearxTestCase): # pylint: disable=missing-class-docstring, ) def test_index_rss(self): - result = self.app.post('/', data={'q': 'test', 'format': 'rss'}) + result = self.client.post('/', data={'q': 'test', 'format': 'rss'}) self.assertEqual(result.status_code, 308) def test_search_rss(self): - result = self.app.post('/search', data={'q': 'test', 'format': 'rss'}) + result = self.client.post('/search', data={'q': 'test', 'format': 'rss'}) self.assertIn(b'<description>Search results for "test" - SearXNG</description>', result.data) @@ -191,28 +183,28 @@ class ViewsTestCase(SearxTestCase): # pylint: disable=missing-class-docstring, self.assertIn(b'<description>first test content</description>', result.data) def test_redirect_about(self): - result = self.app.get('/about') + result = self.client.get('/about') self.assertEqual(result.status_code, 302) def test_info_page(self): - result = self.app.get('/info/en/search-syntax') + result = self.client.get('/info/en/search-syntax') self.assertEqual(result.status_code, 200) self.assertIn(b'<h1>Search syntax</h1>', result.data) def test_health(self): - result = self.app.get('/healthz') + result = self.client.get('/healthz') self.assertEqual(result.status_code, 200) self.assertIn(b'OK', result.data) def test_preferences(self): - result = self.app.get('/preferences') + result = self.client.get('/preferences') self.assertEqual(result.status_code, 200) self.assertIn(b'<form id="search_form" method="post" action="/preferences"', result.data) self.assertIn(b'<div id="categories_container">', result.data) self.assertIn(b'<legend id="pref_ui_locale">Interface language</legend>', result.data) def test_browser_locale(self): - result = self.app.get('/preferences', headers={'Accept-Language': 'zh-tw;q=0.8'}) + result = self.client.get('/preferences', headers={'Accept-Language': 'zh-tw;q=0.8'}) self.assertEqual(result.status_code, 200) self.assertIn( b'<option value="zh-Hant-TW" selected="selected">', @@ -226,42 +218,43 @@ class ViewsTestCase(SearxTestCase): # pylint: disable=missing-class-docstring, ) def test_browser_empty_locale(self): - result = self.app.get('/preferences', headers={'Accept-Language': ''}) + result = self.client.get('/preferences', headers={'Accept-Language': ''}) self.assertEqual(result.status_code, 200) self.assertIn( b'<option value="en" selected="selected">', result.data, 'Interface locale ignored browser preference.' ) def test_locale_occitan(self): - result = self.app.get('/preferences?locale=oc') + result = self.client.get('/preferences?locale=oc') self.assertEqual(result.status_code, 200) self.assertIn( b'<option value="oc" selected="selected">', result.data, 'Interface locale ignored browser preference.' ) def test_stats(self): - result = self.app.get('/stats') + result = self.client.get('/stats') self.assertEqual(result.status_code, 200) self.assertIn(b'<h1>Engine stats</h1>', result.data) def test_robots_txt(self): - result = self.app.get('/robots.txt') + result = self.client.get('/robots.txt') self.assertEqual(result.status_code, 200) self.assertIn(b'Allow: /', result.data) def test_opensearch_xml(self): - result = self.app.get('/opensearch.xml') + result = self.client.get('/opensearch.xml') self.assertEqual(result.status_code, 200) self.assertIn( b'<Description>SearXNG is a metasearch engine that respects your privacy.</Description>', result.data ) def test_favicon(self): - result = self.app.get('/favicon.ico') + result = self.client.get('/favicon.ico') + result.close() self.assertEqual(result.status_code, 200) def test_config(self): - result = self.app.get('/config') + result = self.client.get('/config') self.assertEqual(result.status_code, 200) json_result = result.get_json() self.assertTrue(json_result) diff --git a/tests/unit/test_webutils.py b/tests/unit/test_webutils.py index cf9da5f10..8d52045c5 100644 --- a/tests/unit/test_webutils.py +++ b/tests/unit/test_webutils.py @@ -1,5 +1,5 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -# pylint: disable=missing-module-docstring +# pylint: disable=missing-module-docstring,disable=missing-class-docstring,invalid-name import mock from parameterized.parameterized import parameterized @@ -7,7 +7,7 @@ from searx import webutils from tests import SearxTestCase -class TestWebUtils(SearxTestCase): # pylint: disable=missing-class-docstring +class TestWebUtils(SearxTestCase): @parameterized.expand( [ @@ -78,8 +78,10 @@ class TestWebUtils(SearxTestCase): # pylint: disable=missing-class-docstring self.assertEqual(webutils.highlight_content(content, query), expected) -class TestUnicodeWriter(SearxTestCase): # pylint: disable=missing-class-docstring +class TestUnicodeWriter(SearxTestCase): + def setUp(self): + super().setUp() self.unicode_writer = webutils.CSVWriter(mock.MagicMock()) def test_write_row(self): @@ -93,7 +95,8 @@ class TestUnicodeWriter(SearxTestCase): # pylint: disable=missing-class-docstri self.assertEqual(self.unicode_writer.writerow.call_count, len(rows)) -class TestNewHmac(SearxTestCase): # pylint: disable=missing-class-docstring +class TestNewHmac(SearxTestCase): + @parameterized.expand( [ b'secret', |