summaryrefslogtreecommitdiff
path: root/docs/src
AgeCommit message (Collapse)Author
2023-03-24[mod] DuckDuckGo: reversed engineered & upgrade to data_type: traits_v1Markus Heiser
Partial reverse engineering of the DuckDuckGo (DDG) engines including a improved language and region handling based on the enigne.traits_v1 data. - DDG Lite - DDG Instant Answer API - DDG Images - DDG Weather docs/src/searx.engine.duckduckgo.rst: Online documentation of the DDG engines (make docs.live) searx/data/engine_traits.json Add data type "traits_v1" generated by the fetch_traits() functions from: - "duckduckgo" (WEB), - "duckduckgo images" and - "duckduckgo weather" and remove data from obsolete data type "supported_languages". searx/autocomplete.py: Reversed engineered Autocomplete from DDG. Supports DDG's languages. searx/engines/duckduckgo.py: - fetch_traits(): Fetch languages & regions from DDG. - get_ddg_lang(): Get DDG's language identifier from SearXNG's locale. DDG defines its languages by region codes. DDG-Lite does not offer a language selection to the user, only a region can be selected by the user. - Cache ``vqd`` value: The vqd value depends on the query string and is needed for the follow up pages or the images loaded by a XMLHttpRequest (DDG images). The ``vqd`` value of a search term is stored for 10min in the redis DB. - DDG Lite engine: reversed engineered request method with improved Language and region support and better ``vqd`` handling. searx/engines/duckduckgo_definitions.py: DDG Instant Answer API The *instant answers* API does not support languages, or at least we could not find out how language support should work. It seems that most of the features are based on English terms. searx/engines/duckduckgo_images.py: DDG Images Reversed engineered request method. Improved language and region handling based on cookies and the enigne.traits_v1 data. Response: add image format to the result list searx/engines/duckduckgo_weather.py: DDG Weather Improved language and region handling based on cookies and the enigne.traits_v1 data. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] Startpage: reversed engineered & upgrade to data_type: traits_v1Markus Heiser
One reason for the often seen CAPTCHA of the Startpage requests are the incomplete requests SearXNG sends to startpage.com: this patch is a complete new implementation of the ``request()`` function, reversed engineered from the Startpage's search form. The new implementation: - use traits of data_type: traits_v1 and drop deprecated data_type: supported_languages - adds time-range support - adds save-search support - fix searxng/searxng/issues 1884 - fix searxng/searxng/issues 1081 --> improvements to avoid CAPTCHA In preparation for more categories (News, Images, Videos ..) from Startpage, the variable ``startpage_categ`` was set up. The default value is ``web`` and other categories from Startpage are not yet implemented. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] wikipedia & wikidata: upgrade to data_type: traits_v1Markus Heiser
BTW this fix an issue in wikipedia: SearXNG's locales zh-TW and zh-HK are now using language `zh-classical` from wikipedia (and not `zh`). Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[doc] add a description of bing engines (web, news, video, images)Markus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] replace searx.languages by searx.sxng_localesMarkus Heiser
With the language and region tags from the EngineTraitsMap the handling of SearXNG's tags of languages and regions has been normalized and is no longer a *mystery*. The "languages" became "locales" that are supported by babel and by this, the update_engine_traits.py can be simplified a lot. Other code places can be simplified as well, but these simplifications should (respectively can) only be done when none of the engines work with the deprecated EngineTraits.supported_languages interface anymore. This commit replaces searx.languages by searx.sxng_locales and fix the naming of some names from "language" to "locale" (e.g. language_codes --> sxng_locales). Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] Startpage: fetch engine traits (data_type: supported_languages)Markus Heiser
Implements a fetch_traits function for the Startpage engine. .. note:: Does not include migration of the request methode from 'supported_languages' to 'traits' (EngineTraits) object! Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] Peertube: re-engineered & upgrade to data_type: traits_v1Markus Heiser
- fetch_traits(): Fetch languages from peertube's search-index source code. [mod] Include migration of the request methode from 'supported_languages' to 'traits' (EngineTraits) object. [fix] old supported_languages_url is no longer valid since the sources has been moved to a different path. - fixed code to pass pylint - request(): complete re-implementation based on the API docs [1] - response(): complete re-implementation, adds serveral fields missed before - add source code documentation [1] https://docs.joinpeertube.org/api-rest-reference.html#tag/Search/operation/searchVideos Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] replace engines_languages.json by engines_traits.jsonMarkus Heiser
Implementations of the *traits* of the engines. Engine's traits are fetched from the origin engine and stored in a JSON file in the *data folder*. Most often traits are languages and region codes and their mapping from SearXNG's representation to the representation in the origin search engine. To load traits from the persistence:: searx.enginelib.traits.EngineTraitsMap.from_data() For new traits new properties can be added to the class:: searx.enginelib.traits.EngineTraits .. hint:: Implementation is downward compatible to the deprecated *supported_languages method* from the vintage implementation. The vintage code is tagged as *deprecated* an can be removed when all engines has been ported to the *traits method*. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-17Add "Auto-detected" as a language.Alexandre Flament
When the user choose "Auto-detected", the choice remains on the following queries. The detected language is displayed. For example "Auto-detected (en)": * the next query language is going to be auto detected * for the current query, the detected language is English. This replace the autodetect_search_language plugin.
2023-01-29[doc] Add doc-strings to searx.exceptionsMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-01-04Correct my small mistakeJulian
2023-01-04Correct a small mistakeJulian
2022-12-14Merge pull request #1954 from dalf/fix.redis.init.2Alexandre Flament
[fix] follow up of PR-1856
2022-12-11[mod] improve 'Autodetect search language' pluginMarkus Heiser
- Add documentation to the plugin - Harmonize FastText language model with SearXNG's language model Reosurces:: import fasttext # --> +10 MB fasttext.load_model(str(data_dir / 'lid.176.ftz')) # --> +4MB Suggested-by: @dalf - To speed up and simplify the deployment use fasttext-wheel instead of fasttext - Building numpy on the Alpine Linux of docker-images takes ages --> install py3-numpy from Alpines package manager (apk) - Alpine Linux on docker-images (musl libc) do not support fasttext-wheel (gnu libc) --> patch Dockerfile and build from fastetxt: sed -i s/fasttext-wheel/fasttext/ requirements.txt Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-12-10move searx.shared.redisdb to searx.redisdbAlexandre Flament
2022-07-19[mod] add tor_check plugin - convenient tor checking trough searxngSolirs
2022-05-30[mod] collection of redis functions and lua scripts (initial)Markus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-25[doc] add some documentation about the limiter plugin (and redis)Markus Heiser
Requested-by: https://github.com/searxng/searxng/discussions/993#discussioncomment-2396914 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-16[mod] add i18n infrastructure for SearXNG message files (searxng.msg)Markus Heiser
With this patch ``searxng.msg`` files can be added to SearXNG. In ``searxng.msg`` files messages can be defined which are not captured by babel's gettext, like the generic names of the categories or messages that are stored in constants. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-12[mod] replace /help by /info pages and include pages in project docsMarkus Heiser
This patch implements a bolierplate to share content from info-pages of the SearXNG instance (URL /info) with the project documentation (path /docs/user). The info pages are using Markdown (CommonMark), to include them in the project documentation (reST) the myst-parser [1] is used in the Sphinx-doc build chain. If base_url is known (defined in settings.yml) links to the instance are also inserted into the project documentation:: searxng_extra/docs_prebuild [1] https://www.sphinx-doc.org/en/master/usage/markdown.html Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-04Merge pull request #839 from dalf/docs_searx_utilsMarkus Heiser
[mod] add documentation about searx.utils
2022-02-02[enh] limiter pluginAlexandre Flament
can replace filtron: * rate limite the number of request per IP and per (IP, User-Agent) * block some bots use Redis data stored in Redis never contains the IP addresses, only HMAC using the secret_key Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-29[mod] add documentation about searx.utilsAlexandre Flament
This module is a toolbox for the engines. Is should be documented. In addition, searx/utils.py is checked by pylint.
2022-01-28[mod] tineye - add some documentationMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-03[enh] add redis connector searx/shared/redisdb.pyMarkus Heiser
Add a redis connector, the default DB connector is a socket at:: unix:///usr/local/searxng-redis/run/redis.sock?db=0 To set up a redis instance simply use:: $ ./manage redis.build $ sudo -H ./manage redis.install A hint for developers: To get access rights to this instance, your developer account needs to be added to the *searxng-redis* group:: $ sudo -H ./manage redis.addgrp "${USER}" # don't forget to logout & login to get member of group Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-16[fix] engine - yahoo: rewrite and fix issuesMarkus Heiser
Languages are supported by mapping the language to a domain. If domain is not found in :py:obj:`lang2domain` URL ``<lang>.search.yahoo.com`` is used. BTW: fix issue reported at https://github.com/searx/searx/issues/3020 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-10[doc] update docs/dev/plugins.rstAlexandre Flament
2021-08-04[mod] pylint & document searx.locales (settings.yml: remove locales)Markus Heiser
- Add ``# lint: pylint`` header to pylint this python file. - Fix issues reported by pylint. - Add source code documentation of modul searx.locales Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-21[docs] add documentation from the sources of the google enginesMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-04[docs] add engine "Demo Online Engine"Markus Heiser
This engine just exists for documentation purpose. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-04[docs] move article "XPath Engine" to to admin/engines/Markus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-04[docs] add engine "Demo Offline Engine"Markus Heiser
This engine just exists for documentation purpose. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-01[docs] add new rubric 'Source-Code'Markus Heiser
- document namespace: searx.engines - move docs/dev/xpath_engine.rst -> docs/src/searx.engines.xpath.rst Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-11 11:46:29 +0000