From 5720844fcdc8601798e10544e2fd25ce4f2ad099 Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Fri, 30 Jun 2023 18:07:02 +0200 Subject: [doc] rearranges Settings & Engines docs for better readability We have built up detailed documentation of the *settings* and the *engines* over the past few years. However, this documentation was still spread over various chapters and was difficult to navigate in its entirety. This patch rearranges the Settings & Engines documentation for better readability. To review new ordered docs:: make docs.clean docs.live Signed-off-by: Markus Heiser --- docs/dev/contribution_guide.rst | 2 +- docs/dev/engine_overview.rst | 432 ------------------- docs/dev/engines/demo/demo_offline.rst | 14 + docs/dev/engines/demo/demo_online.rst | 14 + docs/dev/engines/engine_overview.rst | 468 +++++++++++++++++++++ docs/dev/engines/enginelib.rst | 22 + docs/dev/engines/engines.rst | 9 + docs/dev/engines/index.rst | 97 +++++ docs/dev/engines/offline/command-line-engines.rst | 23 + docs/dev/engines/offline/nosql-engines.rst | 97 +++++ .../dev/engines/offline/search-indexer-engines.rst | 62 +++ docs/dev/engines/offline/sql-engines.rst | 121 ++++++ docs/dev/engines/offline_concept.rst | 69 +++ docs/dev/engines/online/annas_archive.rst | 13 + docs/dev/engines/online/archlinux.rst | 14 + docs/dev/engines/online/bing.rst | 43 ++ docs/dev/engines/online/dailymotion.rst | 13 + docs/dev/engines/online/duckduckgo.rst | 22 + docs/dev/engines/online/google.rst | 76 ++++ docs/dev/engines/online/peertube.rst | 27 ++ docs/dev/engines/online/recoll.rst | 13 + docs/dev/engines/online/startpage.rst | 13 + docs/dev/engines/online/torznab.rst | 13 + docs/dev/engines/online/wikipedia.rst | 27 ++ docs/dev/engines/online/yahoo.rst | 13 + docs/dev/engines/online_url_search/tineye.rst | 14 + docs/dev/engines/xpath.rst | 14 + docs/dev/index.rst | 4 +- docs/dev/lxcdev.rst | 2 +- docs/dev/makefile.rst | 6 +- docs/dev/offline_engines.rst | 78 ---- docs/dev/reST.rst | 2 +- docs/dev/searxng_extra/index.rst | 1 - 33 files changed, 1318 insertions(+), 520 deletions(-) delete mode 100644 docs/dev/engine_overview.rst create mode 100644 docs/dev/engines/demo/demo_offline.rst create mode 100644 docs/dev/engines/demo/demo_online.rst create mode 100644 docs/dev/engines/engine_overview.rst create mode 100644 docs/dev/engines/enginelib.rst create mode 100644 docs/dev/engines/engines.rst create mode 100644 docs/dev/engines/index.rst create mode 100644 docs/dev/engines/offline/command-line-engines.rst create mode 100644 docs/dev/engines/offline/nosql-engines.rst create mode 100644 docs/dev/engines/offline/search-indexer-engines.rst create mode 100644 docs/dev/engines/offline/sql-engines.rst create mode 100644 docs/dev/engines/offline_concept.rst create mode 100644 docs/dev/engines/online/annas_archive.rst create mode 100644 docs/dev/engines/online/archlinux.rst create mode 100644 docs/dev/engines/online/bing.rst create mode 100644 docs/dev/engines/online/dailymotion.rst create mode 100644 docs/dev/engines/online/duckduckgo.rst create mode 100644 docs/dev/engines/online/google.rst create mode 100644 docs/dev/engines/online/peertube.rst create mode 100644 docs/dev/engines/online/recoll.rst create mode 100644 docs/dev/engines/online/startpage.rst create mode 100644 docs/dev/engines/online/torznab.rst create mode 100644 docs/dev/engines/online/wikipedia.rst create mode 100644 docs/dev/engines/online/yahoo.rst create mode 100644 docs/dev/engines/online_url_search/tineye.rst create mode 100644 docs/dev/engines/xpath.rst delete mode 100644 docs/dev/offline_engines.rst (limited to 'docs/dev') diff --git a/docs/dev/contribution_guide.rst b/docs/dev/contribution_guide.rst index 0a2f07e40..67133b6b3 100644 --- a/docs/dev/contribution_guide.rst +++ b/docs/dev/contribution_guide.rst @@ -4,7 +4,7 @@ How to contribute ================= -.. contents:: Contents +.. contents:: :depth: 2 :local: :backlinks: entry diff --git a/docs/dev/engine_overview.rst b/docs/dev/engine_overview.rst deleted file mode 100644 index 5fd31c01d..000000000 --- a/docs/dev/engine_overview.rst +++ /dev/null @@ -1,432 +0,0 @@ -.. _engines-dev: - -=============== -Engine Overview -=============== - -.. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine - -.. sidebar:: Further reading .. - - - :ref:`configured engines` - - :ref:`settings engine` - -.. contents:: - :depth: 3 - :backlinks: entry - -SearXNG is a metasearch-engine_, so it uses different search engines to provide -better results. - -Because there is no general search API which could be used for every search -engine, an adapter has to be built between SearXNG and the external search -engines. Adapters are stored under the folder :origin:`searx/engines`. - -.. _general engine configuration: - -General Engine Configuration -============================ - -It is required to tell SearXNG the type of results the engine provides. The -arguments can be set in the engine file or in the settings file (normally -``settings.yml``). The arguments in the settings file override the ones in the -engine file. - -It does not matter if an option is stored in the engine file or in the settings. -However, the standard way is the following: - -.. _engine file: - -Engine File ------------ - -.. table:: Common options in the engine module - :width: 100% - - ======================= =========== ======================================================== - argument type information - ======================= =========== ======================================================== - categories list categories, in which the engine is working - paging boolean support multiple pages - time_range_support boolean support search time range - engine_type str - ``online`` :ref:`[ref] ` by - default, other possibles values are: - - ``offline`` :ref:`[ref] ` - - ``online_dictionary`` - - ``online_currency`` - - ``online_url_search`` - ======================= =========== ======================================================== - -.. _engine settings: - -Engine ``settings.yml`` ------------------------ - -For a more detailed description, see :ref:`settings engine` in the :ref:`settings.yml`. - -.. table:: Common options in the engine setup (``settings.yml``) - :width: 100% - - ======================= =========== ================================================== - argument type information - ======================= =========== ================================================== - name string name of search-engine - engine string name of searxng-engine (file name without ``.py``) - enable_http bool enable HTTP (by default only HTTPS is enabled). - shortcut string shortcut of search-engine - timeout string specific timeout for search-engine - display_error_messages boolean display error messages on the web UI - proxies dict set proxies for a specific engine - (e.g. ``proxies : {http: socks5://proxy:port, - https: socks5://proxy:port}``) - ======================= =========== ================================================== - -.. _engine overrides: - -Overrides ---------- - -A few of the options have default values in the namespace of engine's python -modul, but are often overwritten by the settings. If ``None`` is assigned to an -option in the engine file, it has to be redefined in the settings, otherwise -SearXNG will not start with that engine (global names with a leading underline can -be ``None``). - -Here is an very simple example of the global names in the namespace of engine's -module: - -.. code:: python - - # engine dependent config - categories = ['general'] - paging = True - _non_overwritten_global = 'foo' - - -.. table:: The naming of overrides is arbitrary / recommended overrides are: - :width: 100% - - ======================= =========== =========================================== - argument type information - ======================= =========== =========================================== - base_url string base-url, can be overwritten to use same - engine on other URL - number_of_results int maximum number of results per request - language string ISO code of language and country like en_US - api_key string api-key if required by engine - ======================= =========== =========================================== - -.. _engine request: - -Making a Request -================ - -To perform a search an URL have to be specified. In addition to specifying an -URL, arguments can be passed to the query. - -.. _engine request arguments: - -Passed Arguments (request) --------------------------- - -These arguments can be used to construct the search query. Furthermore, -parameters with default value can be redefined for special purposes. - -.. _engine request online: - -.. table:: If the ``engine_type`` is :py:obj:`online - ` - :width: 100% - - ====================== ============== ======================================================================== - argument type default-value, information - ====================== ============== ======================================================================== - url str ``''`` - method str ``'GET'`` - headers set ``{}`` - data set ``{}`` - cookies set ``{}`` - verify bool ``True`` - headers.User-Agent str a random User-Agent - category str current category, like ``'general'`` - safesearch int ``0``, between ``0`` and ``2`` (normal, moderate, strict) - time_range Optional[str] ``None``, can be ``day``, ``week``, ``month``, ``year`` - pageno int current pagenumber - searxng_locale str SearXNG's locale selected by user. Specific language code like - ``'en'``, ``'en-US'``, or ``'all'`` if unspecified. - ====================== ============== ======================================================================== - - -.. _engine request online_dictionary: - -.. table:: If the ``engine_type`` is :py:obj:`online_dictionary - `, - in addition to the :ref:`online ` arguments: - :width: 100% - - ====================== ============== ======================================================================== - argument type default-value, information - ====================== ============== ======================================================================== - from_lang str specific language code like ``'en_US'`` - to_lang str specific language code like ``'en_US'`` - query str the text query without the languages - ====================== ============== ======================================================================== - -.. _engine request online_currency: - -.. table:: If the ``engine_type`` is :py:obj:`online_currency - `, - in addition to the :ref:`online ` arguments: - :width: 100% - - ====================== ============== ======================================================================== - argument type default-value, information - ====================== ============== ======================================================================== - amount float the amount to convert - from str ISO 4217 code - to str ISO 4217 code - from_name str currency name - to_name str currency name - ====================== ============== ======================================================================== - -.. _engine request online_url_search: - -.. table:: If the ``engine_type`` is :py:obj:`online_url_search - `, - in addition to the :ref:`online ` arguments: - :width: 100% - - ====================== ============== ======================================================================== - argument type default-value, information - ====================== ============== ======================================================================== - search_url dict URLs from the search query: - - .. code:: python - - { - 'http': str, - 'ftp': str, - 'data:image': str - } - ====================== ============== ======================================================================== - -Specify Request ---------------- - -The function :py:func:`def request(query, params): -` always returns the ``params`` variable, the -following parameters can be used to specify a search request: - -.. table:: - :width: 100% - - =================== =========== ========================================================================== - argument type information - =================== =========== ========================================================================== - url str requested url - method str HTTP request method - headers set HTTP header information - data set HTTP data information - cookies set HTTP cookies - verify bool Performing SSL-Validity check - allow_redirects bool Follow redirects - max_redirects int maximum redirects, hard limit - soft_max_redirects int maximum redirects, soft limit. Record an error but don't stop the engine - raise_for_httperror bool True by default: raise an exception if the HTTP code of response is >= 300 - =================== =========== ========================================================================== - - -.. _engine results: -.. _engine media types: - -Media Types -=========== - -Each result item of an engine can be of different media-types. Currently the -following media-types are supported. To set another media-type as ``default``, -the parameter ``template`` must be set to the desired type. - -.. table:: Parameter of the **default** media type: - :width: 100% - - ========================= ===================================================== - result-parameter information - ========================= ===================================================== - url string, url of the result - title string, title of the result - content string, general result-text - publishedDate :py:class:`datetime.datetime`, time of publish - ========================= ===================================================== - - -.. table:: Parameter of the **images** media type: - :width: 100% - - ========================= ===================================================== - result-parameter information - ------------------------- ----------------------------------------------------- - template is set to ``images.html`` - ========================= ===================================================== - url string, url to the result site - title string, title of the result *(partly implemented)* - content *(partly implemented)* - publishedDate :py:class:`datetime.datetime`, - time of publish *(partly implemented)* - img\_src string, url to the result image - thumbnail\_src string, url to a small-preview image - ========================= ===================================================== - - -.. table:: Parameter of the **videos** media type: - :width: 100% - - ========================= ===================================================== - result-parameter information - ------------------------- ----------------------------------------------------- - template is set to ``videos.html`` - ========================= ===================================================== - url string, url of the result - title string, title of the result - content *(not implemented yet)* - publishedDate :py:class:`datetime.datetime`, time of publish - thumbnail string, url to a small-preview image - ========================= ===================================================== - -.. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme - -.. table:: Parameter of the **torrent** media type: - :width: 100% - - ========================= ===================================================== - result-parameter information - ------------------------- ----------------------------------------------------- - template is set to ``torrent.html`` - ========================= ===================================================== - url string, url of the result - title string, title of the result - content string, general result-text - publishedDate :py:class:`datetime.datetime`, - time of publish *(not implemented yet)* - seed int, number of seeder - leech int, number of leecher - filesize int, size of file in bytes - files int, number of files - magnetlink string, magnetlink_ of the result - torrentfile string, torrentfile of the result - ========================= ===================================================== - -.. table:: Parameter of the **map** media type: - :width: 100% - - ========================= ===================================================== - result-parameter information - ------------------------- ----------------------------------------------------- - template is set to ``map.html`` - ========================= ===================================================== - url string, url of the result - title string, title of the result - content string, general result-text - publishedDate :py:class:`datetime.datetime`, time of publish - latitude latitude of result (in decimal format) - longitude longitude of result (in decimal format) - boundingbox boundingbox of result (array of 4. values - ``[lat-min, lat-max, lon-min, lon-max]``) - geojson geojson of result (https://geojson.org/) - osm.type type of osm-object (if OSM-Result) - osm.id id of osm-object (if OSM-Result) - address.name name of object - address.road street name of object - address.house_number house number of object - address.locality city, place of object - address.postcode postcode of object - address.country country of object - ========================= ===================================================== - -.. _BibTeX format: https://www.bibtex.com/g/bibtex-format/ -.. _BibTeX field types: https://en.wikipedia.org/wiki/BibTeX#Field_types - -.. list-table:: Parameter of the **paper** media type / - see `BibTeX field types`_ and `BibTeX format`_ - :header-rows: 2 - :width: 100% - - * - result-parameter - - Python type - - information - - * - template - - :py:class:`str` - - is set to ``paper.html`` - - * - title - - :py:class:`str` - - title of the result - - * - content - - :py:class:`str` - - abstract - - * - comments - - :py:class:`str` - - free text display in italic below the content - - * - tags - - :py:class:`List `\ [\ :py:class:`str`\ ] - - free tag list - - * - publishedDate - - :py:class:`datetime ` - - last publication date - - * - type - - :py:class:`str` - - short description of medium type, e.g. *book*, *pdf* or *html* ... - - * - authors - - :py:class:`List `\ [\ :py:class:`str`\ ] - - list of authors of the work (authors with a "s") - - * - editor - - :py:class:`str` - - list of editors of a book - - * - publisher - - :py:class:`str` - - name of the publisher - - * - journal - - :py:class:`str` - - name of the journal or magazine the article was - published in - - * - volume - - :py:class:`str` - - volume number - - * - pages - - :py:class:`str` - - page range where the article is - - * - number - - :py:class:`str` - - number of the report or the issue number for a journal article - - * - doi - - :py:class:`str` - - DOI number (like ``10.1038/d41586-018-07848-2``) - - * - issn - - :py:class:`List `\ [\ :py:class:`str`\ ] - - ISSN number like ``1476-4687`` - - * - isbn - - :py:class:`List `\ [\ :py:class:`str`\ ] - - ISBN number like ``9780201896831`` - - * - pdf_url - - :py:class:`str` - - URL to the full article, the PDF version - - * - html_url - - :py:class:`str` - - URL to full article, HTML version diff --git a/docs/dev/engines/demo/demo_offline.rst b/docs/dev/engines/demo/demo_offline.rst new file mode 100644 index 000000000..1b4cb887f --- /dev/null +++ b/docs/dev/engines/demo/demo_offline.rst @@ -0,0 +1,14 @@ +.. _demo offline engine: + +=================== +Demo Offline Engine +=================== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.demo_offline + :members: + diff --git a/docs/dev/engines/demo/demo_online.rst b/docs/dev/engines/demo/demo_online.rst new file mode 100644 index 000000000..9b94207f7 --- /dev/null +++ b/docs/dev/engines/demo/demo_online.rst @@ -0,0 +1,14 @@ +.. _demo online engine: + +================== +Demo Online Engine +================== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.demo_online + :members: + diff --git a/docs/dev/engines/engine_overview.rst b/docs/dev/engines/engine_overview.rst new file mode 100644 index 000000000..8052a8bea --- /dev/null +++ b/docs/dev/engines/engine_overview.rst @@ -0,0 +1,468 @@ +.. _engines-dev: + +=============== +Engine Overview +=============== + +.. contents:: + :depth: 3 + :local: + :backlinks: entry + +.. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine + +.. sidebar:: Further reading .. + + - :ref:`configured engines` + - :ref:`settings engine` + +SearXNG is a metasearch-engine_, so it uses different search engines to provide +better results. + +Because there is no general search API which could be used for every search +engine, an adapter has to be built between SearXNG and the external search +engines. Adapters are stored under the folder :origin:`searx/engines`. + +.. _general engine configuration: + +General Engine Configuration +============================ + +It is required to tell SearXNG the type of results the engine provides. The +arguments can be set in the engine file or in the settings file (normally +``settings.yml``). The arguments in the settings file override the ones in the +engine file. + +It does not matter if an option is stored in the engine file or in the settings. +However, the standard way is the following: + +.. _engine file: + +Engine File +----------- + +.. table:: Common options in the engine module + :width: 100% + + ======================= =========== ======================================================== + argument type information + ======================= =========== ======================================================== + categories list categories, in which the engine is working + paging boolean support multiple pages + time_range_support boolean support search time range + engine_type str - ``online`` :ref:`[ref] ` by + default, other possibles values are: + - ``offline`` :ref:`[ref] ` + - ``online_dictionary`` :ref:`[ref] ` + - ``online_currency`` :ref:`[ref] ` + - ``online_url_search`` :ref:`[ref] ` + ======================= =========== ======================================================== + +.. _engine settings: + +Engine ``settings.yml`` +----------------------- + +For a more detailed description, see :ref:`settings engine` in the :ref:`settings.yml`. + +.. table:: Common options in the engine setup (``settings.yml``) + :width: 100% + + ======================= =========== ================================================== + argument type information + ======================= =========== ================================================== + name string name of search-engine + engine string name of searxng-engine (file name without ``.py``) + enable_http bool enable HTTP (by default only HTTPS is enabled). + shortcut string shortcut of search-engine + timeout string specific timeout for search-engine + display_error_messages boolean display error messages on the web UI + proxies dict set proxies for a specific engine + (e.g. ``proxies : {http: socks5://proxy:port, + https: socks5://proxy:port}``) + ======================= =========== ================================================== + +.. _engine overrides: + +Overrides +--------- + +A few of the options have default values in the namespace of engine's python +modul, but are often overwritten by the settings. If ``None`` is assigned to an +option in the engine file, it has to be redefined in the settings, otherwise +SearXNG will not start with that engine (global names with a leading underline can +be ``None``). + +Here is an very simple example of the global names in the namespace of engine's +module: + +.. code:: python + + # engine dependent config + categories = ['general'] + paging = True + _non_overwritten_global = 'foo' + + +.. table:: The naming of overrides is arbitrary / recommended overrides are: + :width: 100% + + ======================= =========== =========================================== + argument type information + ======================= =========== =========================================== + base_url string base-url, can be overwritten to use same + engine on other URL + number_of_results int maximum number of results per request + language string ISO code of language and country like en_US + api_key string api-key if required by engine + ======================= =========== =========================================== + +.. _engine request: + +Making a Request +================ + +To perform a search an URL have to be specified. In addition to specifying an +URL, arguments can be passed to the query. + +.. _engine request arguments: + +Passed Arguments (request) +-------------------------- + +These arguments can be used to construct the search query. Furthermore, +parameters with default value can be redefined for special purposes. + +.. _engine request online: + +.. table:: If the ``engine_type`` is :py:obj:`online + ` + :width: 100% + + ====================== ============== ======================================================================== + argument type default-value, information + ====================== ============== ======================================================================== + url str ``''`` + method str ``'GET'`` + headers set ``{}`` + data set ``{}`` + cookies set ``{}`` + verify bool ``True`` + headers.User-Agent str a random User-Agent + category str current category, like ``'general'`` + safesearch int ``0``, between ``0`` and ``2`` (normal, moderate, strict) + time_range Optional[str] ``None``, can be ``day``, ``week``, ``month``, ``year`` + pageno int current pagenumber + searxng_locale str SearXNG's locale selected by user. Specific language code like + ``'en'``, ``'en-US'``, or ``'all'`` if unspecified. + ====================== ============== ======================================================================== + + +.. _engine request online_dictionary: + +.. table:: If the ``engine_type`` is :py:obj:`online_dictionary + `, + in addition to the :ref:`online ` arguments: + :width: 100% + + ====================== ============== ======================================================================== + argument type default-value, information + ====================== ============== ======================================================================== + from_lang str specific language code like ``'en_US'`` + to_lang str specific language code like ``'en_US'`` + query str the text query without the languages + ====================== ============== ======================================================================== + +.. _engine request online_currency: + +.. table:: If the ``engine_type`` is :py:obj:`online_currency + `, + in addition to the :ref:`online ` arguments: + :width: 100% + + ====================== ============== ======================================================================== + argument type default-value, information + ====================== ============== ======================================================================== + amount float the amount to convert + from str ISO 4217 code + to str ISO 4217 code + from_name str currency name + to_name str currency name + ====================== ============== ======================================================================== + +.. _engine request online_url_search: + +.. table:: If the ``engine_type`` is :py:obj:`online_url_search + `, + in addition to the :ref:`online ` arguments: + :width: 100% + + ====================== ============== ======================================================================== + argument type default-value, information + ====================== ============== ======================================================================== + search_url dict URLs from the search query: + + .. code:: python + + { + 'http': str, + 'ftp': str, + 'data:image': str + } + ====================== ============== ======================================================================== + +Specify Request +--------------- + +The function :py:func:`def request(query, params): +` always returns the ``params`` variable, the +following parameters can be used to specify a search request: + +.. table:: + :width: 100% + + =================== =========== ========================================================================== + argument type information + =================== =========== ========================================================================== + url str requested url + method str HTTP request method + headers set HTTP header information + data set HTTP data information + cookies set HTTP cookies + verify bool Performing SSL-Validity check + allow_redirects bool Follow redirects + max_redirects int maximum redirects, hard limit + soft_max_redirects int maximum redirects, soft limit. Record an error but don't stop the engine + raise_for_httperror bool True by default: raise an exception if the HTTP code of response is >= 300 + =================== =========== ========================================================================== + + +.. _engine results: +.. _engine media types: + +Result Types (``template``) +=========================== + +Each result item of an engine can be of different media-types. Currently the +following media-types are supported. To set another media-type as +:ref:`template default`, the parameter ``template`` must be set to the desired +type. + +.. _template default: + +``default`` +----------- + +.. table:: Parameter of the **default** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ========================= ===================================================== + url string, url of the result + title string, title of the result + content string, general result-text + publishedDate :py:class:`datetime.datetime`, time of publish + ========================= ===================================================== + + +.. _template images: + +``images`` +---------- + +.. table:: Parameter of the **images** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ------------------------- ----------------------------------------------------- + template is set to ``images.html`` + ========================= ===================================================== + url string, url to the result site + title string, title of the result *(partly implemented)* + content *(partly implemented)* + publishedDate :py:class:`datetime.datetime`, + time of publish *(partly implemented)* + img\_src string, url to the result image + thumbnail\_src string, url to a small-preview image + ========================= ===================================================== + + +.. _template videos: + +``videos`` +---------- + +.. table:: Parameter of the **videos** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ------------------------- ----------------------------------------------------- + template is set to ``videos.html`` + ========================= ===================================================== + url string, url of the result + title string, title of the result + content *(not implemented yet)* + publishedDate :py:class:`datetime.datetime`, time of publish + thumbnail string, url to a small-preview image + ========================= ===================================================== + + +.. _template torrent: + +``torrent`` +----------- + +.. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme + +.. table:: Parameter of the **torrent** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ------------------------- ----------------------------------------------------- + template is set to ``torrent.html`` + ========================= ===================================================== + url string, url of the result + title string, title of the result + content string, general result-text + publishedDate :py:class:`datetime.datetime`, + time of publish *(not implemented yet)* + seed int, number of seeder + leech int, number of leecher + filesize int, size of file in bytes + files int, number of files + magnetlink string, magnetlink_ of the result + torrentfile string, torrentfile of the result + ========================= ===================================================== + + +.. _template map: + +``map`` +------- + +.. table:: Parameter of the **map** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ------------------------- ----------------------------------------------------- + template is set to ``map.html`` + ========================= ===================================================== + url string, url of the result + title string, title of the result + content string, general result-text + publishedDate :py:class:`datetime.datetime`, time of publish + latitude latitude of result (in decimal format) + longitude longitude of result (in decimal format) + boundingbox boundingbox of result (array of 4. values + ``[lat-min, lat-max, lon-min, lon-max]``) + geojson geojson of result (https://geojson.org/) + osm.type type of osm-object (if OSM-Result) + osm.id id of osm-object (if OSM-Result) + address.name name of object + address.road street name of object + address.house_number house number of object + address.locality city, place of object + address.postcode postcode of object + address.country country of object + ========================= ===================================================== + + +.. _template paper: + +``paper`` +--------- + +.. _BibTeX format: https://www.bibtex.com/g/bibtex-format/ +.. _BibTeX field types: https://en.wikipedia.org/wiki/BibTeX#Field_types + +.. list-table:: Parameter of the **paper** media type / + see `BibTeX field types`_ and `BibTeX format`_ + :header-rows: 2 + :width: 100% + + * - result-parameter + - Python type + - information + + * - template + - :py:class:`str` + - is set to ``paper.html`` + + * - title + - :py:class:`str` + - title of the result + + * - content + - :py:class:`str` + - abstract + + * - comments + - :py:class:`str` + - free text display in italic below the content + + * - tags + - :py:class:`List `\ [\ :py:class:`str`\ ] + - free tag list + + * - publishedDate + - :py:class:`datetime ` + - last publication date + + * - type + - :py:class:`str` + - short description of medium type, e.g. *book*, *pdf* or *html* ... + + * - authors + - :py:class:`List `\ [\ :py:class:`str`\ ] + - list of authors of the work (authors with a "s") + + * - editor + - :py:class:`str` + - list of editors of a book + + * - publisher + - :py:class:`str` + - name of the publisher + + * - journal + - :py:class:`str` + - name of the journal or magazine the article was + published in + + * - volume + - :py:class:`str` + - volume number + + * - pages + - :py:class:`str` + - page range where the article is + + * - number + - :py:class:`str` + - number of the report or the issue number for a journal article + + * - doi + - :py:class:`str` + - DOI number (like ``10.1038/d41586-018-07848-2``) + + * - issn + - :py:class:`List `\ [\ :py:class:`str`\ ] + - ISSN number like ``1476-4687`` + + * - isbn + - :py:class:`List `\ [\ :py:class:`str`\ ] + - ISBN number like ``9780201896831`` + + * - pdf_url + - :py:class:`str` + - URL to the full article, the PDF version + + * - html_url + - :py:class:`str` + - URL to full article, HTML version + diff --git a/docs/dev/engines/enginelib.rst b/docs/dev/engines/enginelib.rst new file mode 100644 index 000000000..34e3250da --- /dev/null +++ b/docs/dev/engines/enginelib.rst @@ -0,0 +1,22 @@ +.. _searx.enginelib: + +============== +Engine Library +============== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.enginelib + :members: + +.. _searx.enginelib.traits: + + +Engine traits +============= + +.. automodule:: searx.enginelib.traits + :members: diff --git a/docs/dev/engines/engines.rst b/docs/dev/engines/engines.rst new file mode 100644 index 000000000..0f2cb1f22 --- /dev/null +++ b/docs/dev/engines/engines.rst @@ -0,0 +1,9 @@ +.. _searx.engines loader: + +======================== +SearXNG's engines loader +======================== + +.. automodule:: searx.engines + :members: + diff --git a/docs/dev/engines/index.rst b/docs/dev/engines/index.rst new file mode 100644 index 000000000..ca8f87b3c --- /dev/null +++ b/docs/dev/engines/index.rst @@ -0,0 +1,97 @@ +.. _engine implementations: + +====================== +Engine Implementations +====================== + +Framework Components +==================== + +.. toctree:: + :maxdepth: 2 + + enginelib + engines + engine_overview + + +Engine Types +============ + +The :py:obj:`engine_type ` of an engine +determines which :ref:`search processor ` is used by +the engine. + +In this section a list of the enignes that are documented is given, a complete +list of the engines can be found in the source under: :origin:`searx/engines`. + +.. _online engines: + +Online Engines +-------------- + +.. sidebar:: info + + - :py:obj:`processors.online ` + +.. toctree:: + :maxdepth: 1 + :glob: + + demo/demo_online + xpath + online/* + +.. _offline engines: + +Offline Engines +--------------- + +.. sidebar:: info + + - :py:obj:`processors.offline ` + +.. toctree:: + :maxdepth: 1 + :glob: + + offline_concept + demo/demo_offline + offline/* + +.. _online url search: + +Online URL Search +----------------- + +.. sidebar:: info + + - :py:obj:`processors.online_url_search ` + +.. toctree:: + :maxdepth: 1 + :glob: + + online_url_search/* + +.. _online currency: + +Online Currency +--------------- + +.. sidebar:: info + + - :py:obj:`processors.online_currency ` + +*no engine of this type is documented yet / comming soon* + +.. _online dictionary: + +Online Dictionary +----------------- + +.. sidebar:: info + + - :py:obj:`processors.online_dictionary ` + +*no engine of this type is documented yet / comming soon* diff --git a/docs/dev/engines/offline/command-line-engines.rst b/docs/dev/engines/offline/command-line-engines.rst new file mode 100644 index 000000000..0a80d698e --- /dev/null +++ b/docs/dev/engines/offline/command-line-engines.rst @@ -0,0 +1,23 @@ +.. _engine command: + +==================== +Command Line Engines +==================== + +.. sidebar:: info + + - :origin:`command.py ` + - :ref:`offline engines` + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. sidebar:: info + + Initial sponsored by `Search and Discovery Fund + `_ of `NLnet Foundation `_. + +.. automodule:: searx.engines.command + :members: diff --git a/docs/dev/engines/offline/nosql-engines.rst b/docs/dev/engines/offline/nosql-engines.rst new file mode 100644 index 000000000..76f5cfb61 --- /dev/null +++ b/docs/dev/engines/offline/nosql-engines.rst @@ -0,0 +1,97 @@ +.. _nosql engines: + +=============== +NoSQL databases +=============== + +.. sidebar:: further read + + - `NoSQL databases `_ + - `redis.io `_ + - `MongoDB `_ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. sidebar:: info + + Initial sponsored by `Search and Discovery Fund + `_ of `NLnet Foundation `_. + +The following `NoSQL databases`_ are supported: + +- :ref:`engine redis_server` +- :ref:`engine mongodb` + +All of the engines above are just commented out in the :origin:`settings.yml +`, as you have to set various options and install +dependencies before using them. + +By default, the engines use the ``key-value`` template for displaying results / +see :origin:`simple ` +theme. If you are not satisfied with the original result layout, you can use +your own template, set ``result_template`` attribute to ``{template_name}`` and +place the templates at:: + + searx/templates/{theme_name}/result_templates/{template_name} + +Furthermore, if you do not wish to expose these engines on a public instance, you +can still add them and limit the access by setting ``tokens`` as described in +section :ref:`private engines`. + + +Extra Dependencies +================== + +For using :ref:`engine redis_server` or :ref:`engine mongodb` you need to +install additional packages in Python's Virtual Environment of your SearXNG +instance. To switch into the environment (:ref:`searxng-src`) you can use +:ref:`searxng.sh`:: + + $ sudo utils/searxng.sh instance cmd bash + (searxng-pyenv)$ pip install ... + + +Configure the engines +===================== + +`NoSQL databases`_ are used for storing arbitrary data without first defining +their structure. + + +.. _engine redis_server: + +Redis Server +------------ + +.. _redis: https://github.com/andymccurdy/redis-py#installation + +.. sidebar:: info + + - ``pip install`` redis_ + - redis.io_ + - :origin:`redis_server.py ` + +.. automodule:: searx.engines.redis_server + :members: + + +.. _engine mongodb: + +MongoDB +------- + +.. _pymongo: https://github.com/mongodb/mongo-python-driver#installation + +.. sidebar:: info + + - ``pip install`` pymongo_ + - MongoDB_ + - :origin:`mongodb.py ` + + +.. automodule:: searx.engines.mongodb + :members: + diff --git a/docs/dev/engines/offline/search-indexer-engines.rst b/docs/dev/engines/offline/search-indexer-engines.rst new file mode 100644 index 000000000..fa92798cb --- /dev/null +++ b/docs/dev/engines/offline/search-indexer-engines.rst @@ -0,0 +1,62 @@ +================= +Local Search APIs +================= + +.. sidebar:: further read + + - `Comparison to alternatives + `_ + +.. contents:: + :depth: 1 + :local: + :backlinks: entry + +.. sidebar:: info + + Initial sponsored by `Search and Discovery Fund + `_ of `NLnet Foundation `_. + +Administrators might find themselves wanting to integrate locally running search +engines. The following ones are supported for now: + +* `Elasticsearch`_ +* `Meilisearch`_ +* `Solr`_ + +Each search engine is powerful, capable of full-text search. All of the engines +above are added to ``settings.yml`` just commented out, as you have to +``base_url`` for all them. + +Please note that if you are not using HTTPS to access these engines, you have to +enable HTTP requests by setting ``enable_http`` to ``True``. + +Furthermore, if you do not want to expose these engines on a public instance, +you can still add them and limit the access by setting ``tokens`` as described +in section :ref:`private engines`. + +.. _engine meilisearch: + +MeiliSearch +=========== + +.. automodule:: searx.engines.meilisearch + :members: + + +.. _engine elasticsearch: + +Elasticsearch +============= + +.. automodule:: searx.engines.elasticsearch + :members: + +.. _engine solr: + +Solr +==== + +.. automodule:: searx.engines.solr + :members: + diff --git a/docs/dev/engines/offline/sql-engines.rst b/docs/dev/engines/offline/sql-engines.rst new file mode 100644 index 000000000..f0f5add0b --- /dev/null +++ b/docs/dev/engines/offline/sql-engines.rst @@ -0,0 +1,121 @@ +.. _sql engines: + +=========== +SQL Engines +=========== + +.. sidebar:: further read + + - `SQLite `_ + - `PostgreSQL `_ + - `MySQL `_ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. sidebar:: info + + Initial sponsored by `Search and Discovery Fund + `_ of `NLnet Foundation `_. + +With the *SQL engines* you can bind SQL databases into SearXNG. The following +Relational Database Management System (RDBMS) are supported: + +- :ref:`engine sqlite` +- :ref:`engine postgresql` +- :ref:`engine mysql_server` + +All of the engines above are just commented out in the :origin:`settings.yml +`, as you have to set the required attributes for the +engines, e.g. ``database:`` ... + +.. code:: yaml + + - name: ... + engine: {sqlite|postgresql|mysql_server} + database: ... + result_template: {template_name} + query_str: ... + +By default, the engines use the ``key-value`` template for displaying results / +see :origin:`simple ` +theme. If you are not satisfied with the original result layout, you can use +your own template, set ``result_template`` attribute to ``{template_name}`` and +place the templates at:: + + searx/templates/{theme_name}/result_templates/{template_name} + +If you do not wish to expose these engines on a public instance, you can still +add them and limit the access by setting ``tokens`` as described in section +:ref:`private engines`. + + +Extra Dependencies +================== + +For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to +install additional packages in Python's Virtual Environment of your SearXNG +instance. To switch into the environment (:ref:`searxng-src`) you can use +:ref:`searxng.sh`:: + + $ sudo utils/searxng.sh instance cmd bash + (searxng-pyenv)$ pip install ... + + +Configure the engines +===================== + +The configuration of the new database engines are similar. You must put a valid +SQL-SELECT query in ``query_str``. At the moment you can only bind at most one +parameter in your query. By setting the attribute ``limit`` you can define how +many results you want from the SQL server. Basically, it is the same as the +``LIMIT`` keyword in SQL. + +Please, do not include ``LIMIT`` or ``OFFSET`` in your SQL query as the engines +rely on these keywords during paging. If you want to configure the number of +returned results use the option ``limit``. + +.. _engine sqlite: + +SQLite +------ + +.. sidebar:: info + + - :origin:`sqlite.py ` + +.. automodule:: searx.engines.sqlite + :members: + + +.. _engine postgresql: + +PostgreSQL +---------- + +.. _psycopg2: https://www.psycopg.org/install + +.. sidebar:: info + + - :origin:`postgresql.py ` + - ``pip install`` `psycopg2-binary `_ + +.. automodule:: searx.engines.postgresql + :members: + +.. _engine mysql_server: + +MySQL +----- + +.. sidebar:: info + + - :origin:`mysql_server.py ` + - ``pip install`` :pypi:`mysql-connector-python ` + + +.. automodule:: searx.engines.mysql_server + :members: + diff --git a/docs/dev/engines/offline_concept.rst b/docs/dev/engines/offline_concept.rst new file mode 100644 index 000000000..ddb34fc60 --- /dev/null +++ b/docs/dev/engines/offline_concept.rst @@ -0,0 +1,69 @@ +=============== +Offline Concept +=============== + +.. sidebar:: offline engines + + - :ref:`demo offline engine` + - :ref:`engine command` + - :ref:`sql engines` + - :ref:`nosql engines` + - :py:obj:`searx.search.processors.offline` + +To extend the functionality of SearXNG, offline engines are going to be +introduced. An offline engine is an engine which does not need Internet +connection to perform a search and does not use HTTP to communicate. + +Offline engines can be configured, by adding those to the `engines` list of +:origin:`settings.yml `. An example skeleton for offline +engines can be found in :ref:`demo offline engine` (:origin:`demo_offline.py +`). + + +Programming Interface +===================== + +:py:func:`init(engine_settings=None) ` + All offline engines can have their own init function to setup the engine before + accepting requests. The function gets the settings from settings.yml as a + parameter. This function can be omitted, if there is no need to setup anything + in advance. + +:py:func:`search(query, params) ` + Each offline engine has a function named ``search``. This function is + responsible to perform a search and return the results in a presentable + format. (Where *presentable* means presentable by the selected result + template.) + + The return value is a list of results retrieved by the engine. + +Engine representation in ``/config`` + If an engine is offline, the attribute ``offline`` is set to ``True``. + +.. _offline requirements: + +Extra Dependencies +================== + +If an offline engine depends on an external tool, SearXNG does not install it by +default. When an administrator configures such engine and starts the instance, +the process returns an error with the list of missing dependencies. Also, +required dependencies will be added to the comment/description of the engine, so +admins can install packages in advance. + +If there is a need to install additional packages in *Python's Virtual +Environment* of your SearXNG instance you need to switch into the environment +(:ref:`searxng-src`) first, for this you can use :ref:`searxng.sh`:: + + $ sudo utils/searxng.sh instance cmd bash + (searxng-pyenv)$ pip install ... + + +Private engines (Security) +========================== + +To limit the access to offline engines, if an instance is available publicly, +administrators can set token(s) for each of the :ref:`private engines`. If a +query contains a valid token, then SearXNG performs the requested private +search. If not, requests from an offline engines return errors. + diff --git a/docs/dev/engines/online/annas_archive.rst b/docs/dev/engines/online/annas_archive.rst new file mode 100644 index 000000000..db88e5069 --- /dev/null +++ b/docs/dev/engines/online/annas_archive.rst @@ -0,0 +1,13 @@ +.. _annas_archive engine: + +============== +Anna's Archive +============== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.annas_archive + :members: diff --git a/docs/dev/engines/online/archlinux.rst b/docs/dev/engines/online/archlinux.rst new file mode 100644 index 000000000..834fffa43 --- /dev/null +++ b/docs/dev/engines/online/archlinux.rst @@ -0,0 +1,14 @@ +.. _archlinux engine: + +========== +Arch Linux +========== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.archlinux + :members: + diff --git a/docs/dev/engines/online/bing.rst b/docs/dev/engines/online/bing.rst new file mode 100644 index 000000000..19c31aa80 --- /dev/null +++ b/docs/dev/engines/online/bing.rst @@ -0,0 +1,43 @@ +.. _bing engines: + +============ +Bing Engines +============ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _bing web engine: + +Bing WEB +======== + +.. automodule:: searx.engines.bing + :members: + +.. _bing images engine: + +Bing Images +=========== + +.. automodule:: searx.engines.bing_images + :members: + +.. _bing videos engine: + +Bing Videos +=========== + +.. automodule:: searx.engines.bing_videos + :members: + +.. _bing news engine: + +Bing News +========= + +.. automodule:: searx.engines.bing_news + :members: diff --git a/docs/dev/engines/online/dailymotion.rst b/docs/dev/engines/online/dailymotion.rst new file mode 100644 index 000000000..c661172e5 --- /dev/null +++ b/docs/dev/engines/online/dailymotion.rst @@ -0,0 +1,13 @@ +.. _dailymotion engine: + +=========== +Dailymotion +=========== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.dailymotion + :members: diff --git a/docs/dev/engines/online/duckduckgo.rst b/docs/dev/engines/online/duckduckgo.rst new file mode 100644 index 000000000..d0d18eb8a --- /dev/null +++ b/docs/dev/engines/online/duckduckgo.rst @@ -0,0 +1,22 @@ +.. _duckduckgo engines: + +================= +DukcDukGo Engines +================= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.duckduckgo + :members: + +.. automodule:: searx.engines.duckduckgo_images + :members: + +.. automodule:: searx.engines.duckduckgo_definitions + :members: + +.. automodule:: searx.engines.duckduckgo_weather + :members: diff --git a/docs/dev/engines/online/google.rst b/docs/dev/engines/online/google.rst new file mode 100644 index 000000000..9085070bd --- /dev/null +++ b/docs/dev/engines/online/google.rst @@ -0,0 +1,76 @@ +.. _google engines: + +============== +Google Engines +============== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _google API: + +Google API +========== + +.. _Query Parameter Definitions: + https://developers.google.com/custom-search/docs/xml_results#WebSearch_Query_Parameter_Definitions + +SearXNG's implementation of the Google API is mainly done in +:py:obj:`get_google_info `. + +For detailed description of the *REST-full* API see: `Query Parameter +Definitions`_. The linked API documentation can sometimes be helpful during +reverse engineering. However, we cannot use it in the freely accessible WEB +services; not all parameters can be applied and some engines are more *special* +than other (e.g. :ref:`google news engine`). + + +.. _google web engine: + +Google WEB +========== + +.. automodule:: searx.engines.google + :members: + +.. _google autocomplete: + +Google Autocomplete +==================== + +.. autofunction:: searx.autocomplete.google_complete + +.. _google images engine: + +Google Images +============= + +.. automodule:: searx.engines.google_images + :members: + +.. _google videos engine: + +Google Videos +============= + +.. automodule:: searx.engines.google_videos + :members: + +.. _google news engine: + +Google News +=========== + +.. automodule:: searx.engines.google_news + :members: + +.. _google scholar engine: + +Google Scholar +============== + +.. automodule:: searx.engines.google_scholar + :members: diff --git a/docs/dev/engines/online/peertube.rst b/docs/dev/engines/online/peertube.rst new file mode 100644 index 000000000..bedf055fb --- /dev/null +++ b/docs/dev/engines/online/peertube.rst @@ -0,0 +1,27 @@ +.. _peertube engines: + +================ +Peertube Engines +================ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _peertube video engine: + +Peertube Video +============== + +.. automodule:: searx.engines.peertube + :members: + +.. _sepiasearch engine: + +SepiaSearch +=========== + +.. automodule:: searx.engines.sepiasearch + :members: diff --git a/docs/dev/engines/online/recoll.rst b/docs/dev/engines/online/recoll.rst new file mode 100644 index 000000000..2f1a1e4df --- /dev/null +++ b/docs/dev/engines/online/recoll.rst @@ -0,0 +1,13 @@ +.. _engine recoll: + +============= +Recoll Engine +============= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.recoll + :members: diff --git a/docs/dev/engines/online/startpage.rst b/docs/dev/engines/online/startpage.rst new file mode 100644 index 000000000..89e3ad10b --- /dev/null +++ b/docs/dev/engines/online/startpage.rst @@ -0,0 +1,13 @@ +.. _startpage engines: + +================= +Startpage Engines +================= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.startpage + :members: diff --git a/docs/dev/engines/online/torznab.rst b/docs/dev/engines/online/torznab.rst new file mode 100644 index 000000000..9056b60d0 --- /dev/null +++ b/docs/dev/engines/online/torznab.rst @@ -0,0 +1,13 @@ +.. _torznab engine: + +============== +Torznab WebAPI +============== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.torznab + :members: diff --git a/docs/dev/engines/online/wikipedia.rst b/docs/dev/engines/online/wikipedia.rst new file mode 100644 index 000000000..d4920f0f6 --- /dev/null +++ b/docs/dev/engines/online/wikipedia.rst @@ -0,0 +1,27 @@ +.. _wikimedia engines: + +========= +Wikimedia +========= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _wikipedia engine: + +Wikipedia +========= + +.. automodule:: searx.engines.wikipedia + :members: + +.. _wikidata engine: + +Wikidata +========= + +.. automodule:: searx.engines.wikidata + :members: diff --git a/docs/dev/engines/online/yahoo.rst b/docs/dev/engines/online/yahoo.rst new file mode 100644 index 000000000..96c1e2774 --- /dev/null +++ b/docs/dev/engines/online/yahoo.rst @@ -0,0 +1,13 @@ +.. _yahoo engine: + +============ +Yahoo Engine +============ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.yahoo + :members: diff --git a/docs/dev/engines/online_url_search/tineye.rst b/docs/dev/engines/online_url_search/tineye.rst new file mode 100644 index 000000000..3f4db7e0e --- /dev/null +++ b/docs/dev/engines/online_url_search/tineye.rst @@ -0,0 +1,14 @@ +.. _tineye engine: + +====== +Tineye +====== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.tineye + :members: + diff --git a/docs/dev/engines/xpath.rst b/docs/dev/engines/xpath.rst new file mode 100644 index 000000000..42c4d47b6 --- /dev/null +++ b/docs/dev/engines/xpath.rst @@ -0,0 +1,14 @@ +.. _xpath engine: + +============ +XPath Engine +============ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.xpath + :members: + diff --git a/docs/dev/index.rst b/docs/dev/index.rst index 39be0885c..aa0667c58 100644 --- a/docs/dev/index.rst +++ b/docs/dev/index.rst @@ -4,12 +4,10 @@ Developer documentation .. toctree:: :maxdepth: 2 - :caption: Contents quickstart contribution_guide - engine_overview - offline_engines + engines/index search_api plugins translation diff --git a/docs/dev/lxcdev.rst b/docs/dev/lxcdev.rst index ef603e9fd..854b193ff 100644 --- a/docs/dev/lxcdev.rst +++ b/docs/dev/lxcdev.rst @@ -22,7 +22,7 @@ In this article we will show, how you can make use of Linux Containers (LXC_) in section :ref:`internet connectivity docker`. -.. contents:: Contents +.. contents:: :depth: 2 :local: :backlinks: entry diff --git a/docs/dev/makefile.rst b/docs/dev/makefile.rst index bdf7f0f49..ad5665de7 100644 --- a/docs/dev/makefile.rst +++ b/docs/dev/makefile.rst @@ -22,7 +22,7 @@ Calling the ``help`` target gives a first overview (``make help``): .. program-output:: bash -c "cd ..; make --no-print-directory help" -.. contents:: Contents +.. contents:: :depth: 2 :local: :backlinks: entry @@ -243,14 +243,14 @@ calling ``make clean`` stop all processes using the :ref:`make install` or We describe the usage of the ``doc.*`` targets in the :ref:`How to contribute / Documentation ` section. If you want to edit the documentation read our :ref:`make docs.live` section. If you are working in your own brand, -adjust your :ref:`settings global`. +adjust your :ref:`settings brand`. .. _make docs.gh-pages: ``make docs.gh-pages`` ====================== -To deploy on github.io first adjust your :ref:`settings global`. For any +To deploy on github.io first adjust your :ref:`settings brand`. For any further read :ref:`deploy on github.io`. .. _make test: diff --git a/docs/dev/offline_engines.rst b/docs/dev/offline_engines.rst deleted file mode 100644 index bfb2664f5..000000000 --- a/docs/dev/offline_engines.rst +++ /dev/null @@ -1,78 +0,0 @@ -.. _offline engines: - -=============== -Offline Engines -=============== - -.. sidebar:: offline engines - - - :ref:`demo offline engine` - - :ref:`sql engines` - - :ref:`engine command` - - :origin:`Redis ` - -To extend the functionality of SearXNG, offline engines are going to be -introduced. An offline engine is an engine which does not need Internet -connection to perform a search and does not use HTTP to communicate. - -Offline engines can be configured, by adding those to the `engines` list of -:origin:`settings.yml `. An example skeleton for offline -engines can be found in :ref:`demo offline engine` (:origin:`demo_offline.py -`). - - -Programming Interface -===================== - -:py:func:`init(engine_settings=None) ` - All offline engines can have their own init function to setup the engine before - accepting requests. The function gets the settings from settings.yml as a - parameter. This function can be omitted, if there is no need to setup anything - in advance. - -:py:func:`search(query, params) ` - - Each offline engine has a function named ``search``. This function is - responsible to perform a search and return the results in a presentable - format. (Where *presentable* means presentable by the selected result - template.) - - The return value is a list of results retrieved by the engine. - -Engine representation in ``/config`` - If an engine is offline, the attribute ``offline`` is set to ``True``. - -.. _offline requirements: - -Extra Dependencies -================== - -If an offline engine depends on an external tool, SearXNG does not install it by -default. When an administrator configures such engine and starts the instance, -the process returns an error with the list of missing dependencies. Also, -required dependencies will be added to the comment/description of the engine, so -admins can install packages in advance. - -If there is a need to install additional packages in *Python's Virtual -Environment* of your SearXNG instance you need to switch into the environment -(:ref:`searxng-src`) first, for this you can use :ref:`searxng.sh`:: - - $ sudo utils/searxng.sh instance cmd bash - (searxng-pyenv)$ pip install ... - - -Private engines (Security) -========================== - -To limit the access to offline engines, if an instance is available publicly, -administrators can set token(s) for each of the :ref:`private engines`. If a -query contains a valid token, then SearXNG performs the requested private -search. If not, requests from an offline engines return errors. - - -Acknowledgement -=============== - -This development was sponsored by `Search and Discovery Fund -`_ of `NLnet Foundation `_ . - diff --git a/docs/dev/reST.rst b/docs/dev/reST.rst index b19a7096e..47af0b130 100644 --- a/docs/dev/reST.rst +++ b/docs/dev/reST.rst @@ -37,7 +37,7 @@ docs.live ` to build HTML while editing. - DOT_, `Graphviz's dot`_, Graphviz_ -.. contents:: Contents +.. contents:: :depth: 3 :local: :backlinks: entry diff --git a/docs/dev/searxng_extra/index.rst b/docs/dev/searxng_extra/index.rst index c2b5c312b..7bca3c0d4 100644 --- a/docs/dev/searxng_extra/index.rst +++ b/docs/dev/searxng_extra/index.rst @@ -9,7 +9,6 @@ developers. .. toctree:: :maxdepth: 2 - :caption: Contents update standalone_searx.py -- cgit v1.2.3