summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2023-03-29Merge pull request #2291 from searxng/update_data_update_currencies.pyMarkus Heiser
Update searx.data - update_currencies.py
2023-03-29Merge pull request #2292 from searxng/update_data_update_engine_descriptions.pyMarkus Heiser
Update searx.data - update_engine_descriptions.py
2023-03-29Merge pull request #2269 from return42/locale-revisionMarkus Heiser
Revision of the locale- and language- handling in SearXNG
2023-03-29Update searx.data - update_engine_descriptions.pydalf
2023-03-29Update searx.data - update_currencies.pydalf
2023-03-28[fix] engine: google has changed the layout of its responseMarkus Heiser
Since 28. March google has changed its response, this patch fixes the google engine to scrap out the results & images from the new designed response. closes: https://github.com/searxng/searxng/issues/2287 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24Update searx.data - update_engine_traits.pydalf
2023-03-24Bump babel from 2.11.0 to 2.12.1dependabot[bot]
Bumps [babel](https://github.com/python-babel/babel) from 2.11.0 to 2.12.1. - [Release notes](https://github.com/python-babel/babel/releases) - [Changelog](https://github.com/python-babel/babel/blob/master/CHANGES.rst) - [Commits](https://github.com/python-babel/babel/compare/v2.11.0...v2.12.1) --- updated-dependencies: - dependency-name: babel dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
2023-03-24[mod] replace utils.match_language by locales.match_localeMarkus Heiser
This patch replaces the *full of magic* ``utils.match_language`` function by a ``locales.match_locale``. The ``locales.match_locale`` function is based on the ``locales.build_engine_locales`` introduced in 9ae409a0 [1]. In the past SearXNG did only support a search by a language but not in a region. This has been changed a long time ago and regions have been added to SearXNG core but not to the engines. The ``utils.match_language`` was the function to handle the different aspects of language/regions in SearXNG core and the supported *languages* in the engine. The ``utils.match_language`` did it with some magic and works good for most use cases but fails in some edge case. To replace the concurrence of languages and regions in the SearXNG core the ``locales.build_engine_locales`` was introduced in 9ae409a0 [1]. With the last patches all engines has been migrated to a ``fetch_traits`` and a language/region concept that is based on ``locales.build_engine_locales``. To summarize: there is no longer a need for the ``locales.match_language``. [1] https://github.com/searxng/searxng/pull/1652 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] remove obsolete EngineTraits.supported_languagesMarkus Heiser
All engines has been migrated from ``supported_languages`` to the ``fetch_traits`` concept. There is no longer a need for the obsolete code that implements the ``supported_languages`` concept. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] Archlinux Wiki: improved request API & upgrade to data_type: traits_v1Markus Heiser
re-implementation of the Archlinux Wiki: - fetch_traits(): fetch languages, wiki URLs and title arguments - add content field to the result list - add documentation Wikis from wiki.archlinux.fr, wiki.archlinux.ro, archtr.org/wiki do no longer exists (has been merged in the main wiki). Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] SepiaSearch: re-engineered & upgrade to data_type: traits_v1Markus Heiser
- fetch_traits() SepiaSearch and Peertube are using identical languages. Replace module's dictionary `supported_languages` by `engine.traits.languages` (data_type: `traits_v1`). - fixed code to pass pylint - request(): add argument boostLanguages - response(): is replaced by peertube's video_response() function, which adds metadata from channel name, host & tags Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] Dailymotion: improved request API & upgrade to data_type: traits_v1Markus Heiser
- fetch_traits(): fetch locales (and languages) from dailymotion API - removed obsolete data-type "supported_languages" - add documentation - improved argument list of the HTTP request: - add argument: family_filter_map - add conditional argument: localization Don't add localization and country arguments if the user does select a language (:de, :en, ..) - improve code quality (mainly improve readability) Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] Google: reversed engineered & upgrade to data_type: traits_v1Markus Heiser
Partial reverse engineering of the Google engines including a improved language and region handling based on the engine.traits_v1 data. When ever possible the implementations of the Google engines try to make use of the async REST APIs. The get_lang_info() has been generalized to a get_google_info() function / especially the region handling has been improved by adding the cr parameter. searx/data/engine_traits.json Add data type "traits_v1" generated by the fetch_traits() functions from: - Google (WEB), - Google images, - Google news, - Google scholar and - Google videos and remove data from obsolete data type "supported_languages". A traits.custom type that maps region codes to *supported_domains* is fetched from https://www.google.com/supported_domains searx/autocomplete.py: Reversed engineered autocomplete from Google WEB. Supports Google's languages and subdomains. The old API suggestqueries.google.com/complete has been replaced by the async REST API: https://{subdomain}/complete/search?{args} searx/engines/google.py Reverse engineering and extensive testing .. - fetch_traits(): Fetch languages & regions from Google properties. - always use the async REST API (formally known as 'use_mobile_ui') - use *supported_domains* from traits - improved the result list by fetching './/div[@data-content-feature]' and parsing the type of the various *content features* --> thumbnails are added searx/engines/google_images.py Reverse engineering and extensive testing .. - fetch_traits(): Fetch languages & regions from Google properties. - use *supported_domains* from traits - if exists, freshness_date is added to the result - issue 1864: result list has been improved a lot (due to the new cr parameter) searx/engines/google_news.py Reverse engineering and extensive testing .. - fetch_traits(): Fetch languages & regions from Google properties. *supported_domains* is not needed but a ceid list has been added. - different region handling compared to Google WEB - fixed for various languages & regions (due to the new ceid parameter) / avoid CONSENT page - Google News do no longer support time range - result list has been fixed: XPath of pub_date and pub_origin searx/engines/google_videos.py - fetch_traits(): Fetch languages & regions from Google properties. - use *supported_domains* from traits - add paging support - implement a async request ('asearch': 'arc' & 'async': 'use_ac:true,_fmt:html') - simplified code (thanks to '_fmt:html' request) - issue 1359: fixed xpath of video length data searx/engines/google_scholar.py - fetch_traits(): Fetch languages & regions from Google properties. - use *supported_domains* from traits - request(): include patents & citations - response(): fixed CAPTCHA detection (Scholar has its own CATCHA manager) - hardening XPath to iterate over results - fixed XPath of pub_type (has been change from gs_ct1 to gs_cgt2 class) - issue 1769 fixed: new request implementation is no longer incompatible Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] DuckDuckGo: reversed engineered & upgrade to data_type: traits_v1Markus Heiser
Partial reverse engineering of the DuckDuckGo (DDG) engines including a improved language and region handling based on the enigne.traits_v1 data. - DDG Lite - DDG Instant Answer API - DDG Images - DDG Weather docs/src/searx.engine.duckduckgo.rst: Online documentation of the DDG engines (make docs.live) searx/data/engine_traits.json Add data type "traits_v1" generated by the fetch_traits() functions from: - "duckduckgo" (WEB), - "duckduckgo images" and - "duckduckgo weather" and remove data from obsolete data type "supported_languages". searx/autocomplete.py: Reversed engineered Autocomplete from DDG. Supports DDG's languages. searx/engines/duckduckgo.py: - fetch_traits(): Fetch languages & regions from DDG. - get_ddg_lang(): Get DDG's language identifier from SearXNG's locale. DDG defines its languages by region codes. DDG-Lite does not offer a language selection to the user, only a region can be selected by the user. - Cache ``vqd`` value: The vqd value depends on the query string and is needed for the follow up pages or the images loaded by a XMLHttpRequest (DDG images). The ``vqd`` value of a search term is stored for 10min in the redis DB. - DDG Lite engine: reversed engineered request method with improved Language and region support and better ``vqd`` handling. searx/engines/duckduckgo_definitions.py: DDG Instant Answer API The *instant answers* API does not support languages, or at least we could not find out how language support should work. It seems that most of the features are based on English terms. searx/engines/duckduckgo_images.py: DDG Images Reversed engineered request method. Improved language and region handling based on cookies and the enigne.traits_v1 data. Response: add image format to the result list searx/engines/duckduckgo_weather.py: DDG Weather Improved language and region handling based on cookies and the enigne.traits_v1 data. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] Startpage: reversed engineered & upgrade to data_type: traits_v1Markus Heiser
One reason for the often seen CAPTCHA of the Startpage requests are the incomplete requests SearXNG sends to startpage.com: this patch is a complete new implementation of the ``request()`` function, reversed engineered from the Startpage's search form. The new implementation: - use traits of data_type: traits_v1 and drop deprecated data_type: supported_languages - adds time-range support - adds save-search support - fix searxng/searxng/issues 1884 - fix searxng/searxng/issues 1081 --> improvements to avoid CAPTCHA In preparation for more categories (News, Images, Videos ..) from Startpage, the variable ``startpage_categ`` was set up. The default value is ``web`` and other categories from Startpage are not yet implemented. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] wikipedia & wikidata: upgrade to data_type: traits_v1Markus Heiser
BTW this fix an issue in wikipedia: SearXNG's locales zh-TW and zh-HK are now using language `zh-classical` from wikipedia (and not `zh`). Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[doc] add a description of bing engines (web, news, video, images)Markus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] bing_news: use async API & upgrade to data_type: traits_v1Markus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] bing_images: use async API & upgrade to data_type: traits_v1Markus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] bing_videos: use async API & upgrade to data_type: traits_v1Markus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] bing: add time_range support & upgrade to data_type: traits_v1Markus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] replace searx.languages by searx.sxng_localesMarkus Heiser
With the language and region tags from the EngineTraitsMap the handling of SearXNG's tags of languages and regions has been normalized and is no longer a *mystery*. The "languages" became "locales" that are supported by babel and by this, the update_engine_traits.py can be simplified a lot. Other code places can be simplified as well, but these simplifications should (respectively can) only be done when none of the engines work with the deprecated EngineTraits.supported_languages interface anymore. This commit replaces searx.languages by searx.sxng_locales and fix the naming of some names from "language" to "locale" (e.g. language_codes --> sxng_locales). Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] Wikipedia: fetch engine traits (data_type: supported_languages)Markus Heiser
Implements a fetch_traits function for the Wikipedia engines. .. note:: Does not include migration of the request methode from 'supported_languages' to 'traits' (EngineTraits) object! Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] Google: fetch engine traits (data_type: supported_languages)Markus Heiser
Implements a fetch_traits function for the Google engines. .. note:: Does not include migration of the request methode from 'supported_languages' to 'traits' (EngineTraits) object! Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] DuckDuckGo: fetch engine traits (data_type: supported_languages)Markus Heiser
Implements a fetch_traits function for the DuckDuckGo engines. .. note:: Does not include migration of the request methode from 'supported_languages' to 'traits' (EngineTraits) object! Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] yahoo: fetch engine traits (data_type: traits_v1)Markus Heiser
Implements a fetch_traits function for the Yahoo engine. .. note:: Includes migration of the request methode from 'supported_languages' to 'traits' (EngineTraits) object! Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] qwant: fetch engine traits (data_type: traits_v1)Markus Heiser
Implements a fetch_traits function for the Qwant engines. .. note:: Includes migration of the request methode from 'supported_languages' to 'traits' (EngineTraits) object! Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] Dailymotion: fetch engine traits (data_type: supported_languages)Markus Heiser
Implements a fetch_traits function for the Dailymotion engine. .. note:: Does not include migration of the request methode from 'supported_languages' to 'traits' (EngineTraits) object! Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] Startpage: fetch engine traits (data_type: supported_languages)Markus Heiser
Implements a fetch_traits function for the Startpage engine. .. note:: Does not include migration of the request methode from 'supported_languages' to 'traits' (EngineTraits) object! Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] bing: fetch engine traits (data_type: supported_languages)Markus Heiser
Implements a fetch_traits function for the Bing engines. .. note:: Does not include migration of the request methode from 'supported_languages' to 'traits' (EngineTraits) object! Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] Peertube: re-engineered & upgrade to data_type: traits_v1Markus Heiser
- fetch_traits(): Fetch languages from peertube's search-index source code. [mod] Include migration of the request methode from 'supported_languages' to 'traits' (EngineTraits) object. [fix] old supported_languages_url is no longer valid since the sources has been moved to a different path. - fixed code to pass pylint - request(): complete re-implementation based on the API docs [1] - response(): complete re-implementation, adds serveral fields missed before - add source code documentation [1] https://docs.joinpeertube.org/api-rest-reference.html#tag/Search/operation/searchVideos Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] replace engines_languages.json by engines_traits.jsonMarkus Heiser
Implementations of the *traits* of the engines. Engine's traits are fetched from the origin engine and stored in a JSON file in the *data folder*. Most often traits are languages and region codes and their mapping from SearXNG's representation to the representation in the origin search engine. To load traits from the persistence:: searx.enginelib.traits.EngineTraitsMap.from_data() For new traits new properties can be added to the class:: searx.enginelib.traits.EngineTraits .. hint:: Implementation is downward compatible to the deprecated *supported_languages method* from the vintage implementation. The vintage code is tagged as *deprecated* an can be removed when all engines has been ported to the *traits method*. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24Merge pull request #2280 from searxng/dependabot/pip/master/linuxdoc-20230321Markus Heiser
Bump linuxdoc from 20221127 to 20230321
2023-03-24Merge pull request #2283 from ↵Markus Heiser
searxng/revert-2279-dependabot/pip/master/redis-4.5.3 Revert "Bump redis from 4.5.1 to 4.5.3"
2023-03-24Revert "Bump redis from 4.5.1 to 4.5.3"Markus Heiser
2023-03-24Merge pull request #2279 from searxng/dependabot/pip/master/redis-4.5.3Markus Heiser
Bump redis from 4.5.1 to 4.5.3
2023-03-24Merge pull request #2281 from searxng/dependabot/pip/master/pylint-2.17.1Markus Heiser
Bump pylint from 2.17.0 to 2.17.1
2023-03-24Bump linuxdoc from 20221127 to 20230321dependabot[bot]
Bumps [linuxdoc](https://github.com/return42/linuxdoc) from 20221127 to 20230321. - [Release notes](https://github.com/return42/linuxdoc/releases) - [Commits](https://github.com/return42/linuxdoc/commits) --- updated-dependencies: - dependency-name: linuxdoc dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
2023-03-24Merge pull request #2278 from searxng/dependabot/pip/master/yamllint-1.30.0Markus Heiser
Bump yamllint from 1.29.0 to 1.30.0
2023-03-24Bump pylint from 2.17.0 to 2.17.1dependabot[bot]
Bumps [pylint](https://github.com/PyCQA/pylint) from 2.17.0 to 2.17.1. - [Release notes](https://github.com/PyCQA/pylint/releases) - [Commits](https://github.com/PyCQA/pylint/compare/v2.17.0...v2.17.1) --- updated-dependencies: - dependency-name: pylint dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
2023-03-24Bump redis from 4.5.1 to 4.5.3dependabot[bot]
Bumps [redis](https://github.com/redis/redis-py) from 4.5.1 to 4.5.3. - [Release notes](https://github.com/redis/redis-py/releases) - [Changelog](https://github.com/redis/redis-py/blob/master/CHANGES) - [Commits](https://github.com/redis/redis-py/compare/v4.5.1...v4.5.3) --- updated-dependencies: - dependency-name: redis dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
2023-03-24Bump yamllint from 1.29.0 to 1.30.0dependabot[bot]
Bumps [yamllint](https://github.com/adrienverge/yamllint) from 1.29.0 to 1.30.0. - [Release notes](https://github.com/adrienverge/yamllint/releases) - [Changelog](https://github.com/adrienverge/yamllint/blob/master/CHANGELOG.rst) - [Commits](https://github.com/adrienverge/yamllint/compare/v1.29.0...v1.30.0) --- updated-dependencies: - dependency-name: yamllint dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
2023-03-24Merge pull request #2277 from searxng/translations_updateMarkus Heiser
Update translations
2023-03-24[translations] update from Weblatesearxng-bot
abfec8f4 - 2023-03-23 - return42 <markus.heiser@darmarit.de> f02ea21c - 2023-03-23 - return42 <markus.heiser@darmarit.de> 3fc6c653 - 2023-03-20 - chenghui-lee <chlee9926@gmail.com> 342bbf46 - 2023-03-20 - return42 <markus.heiser@darmarit.de>
2023-03-22Merge pull request #2274 from searxng/codespacesAlexandre Flament
Add GitHub Codespaces configuration
2023-03-22Merge pull request #2272 from Solirs/Wikidata_descMarkus Heiser
wikidata: Add description for results
2023-03-21Add GitHub Codespaces configurationAlexandre Flament
2023-03-21Pass black formatting testSolirs
2023-03-21WIKIDATA: Add description for resultsSolirs
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-12 15:44:08 +0000