diff options
Diffstat (limited to 'docs')
32 files changed, 1046 insertions, 464 deletions
diff --git a/docs/admin/plugins.rst b/docs/admin/plugins.rst index d97b3dada..d3d137fcf 100644 --- a/docs/admin/plugins.rst +++ b/docs/admin/plugins.rst @@ -1,12 +1,14 @@ -.. _plugins generic: +.. _plugins admin: =============== -Plugins builtin +List of plugins =============== .. sidebar:: Further reading .. + - :ref:`SearXNG settings <settings plugins>` - :ref:`dev plugin` + - :ref:`builtin plugins` Configuration defaults (at built time): @@ -25,15 +27,10 @@ Configuration defaults (at built time): - DO - Description - JS & CSS dependencies + {% for plg in plugins %} - {% for plgin in plugins %} - - * - {{plgin.name}} - - {{(plgin.default_on and "y") or ""}} - - {{plgin.description}} - - {% for dep in (plgin.js_dependencies + plgin.css_dependencies) %} - | ``{{dep}}`` {% endfor %} + * - {{plg.info.name}} + - {{(plg.default_on and "y") or ""}} + - {{plg.info.description}} {% endfor %} diff --git a/docs/admin/settings/index.rst b/docs/admin/settings/index.rst index acc91dbdd..d1f808f3d 100644 --- a/docs/admin/settings/index.rst +++ b/docs/admin/settings/index.rst @@ -22,6 +22,6 @@ Settings settings_redis settings_outgoing settings_categories_as_tabs - + settings_plugins diff --git a/docs/admin/settings/settings_plugins.rst b/docs/admin/settings/settings_plugins.rst new file mode 100644 index 000000000..991e7ee53 --- /dev/null +++ b/docs/admin/settings/settings_plugins.rst @@ -0,0 +1,67 @@ +.. _settings plugins: + +======= +Plugins +======= + +.. sidebar:: Further reading .. + + - :ref:`plugins admin` + - :ref:`dev plugin` + - :ref:`builtin plugins` + + +The built-in plugins can be activated or deactivated via the settings +(:ref:`settings enabled_plugins`) and external plugins can be integrated into +SearXNG (:ref:`settings external_plugins`). + + +.. _settings enabled_plugins: + +``enabled_plugins:`` (internal) +=============================== + +In :ref:`plugins admin` you find a complete list of all plugins, the default +configuration looks like: + +.. code:: yaml + + enabled_plugins: + - 'Basic Calculator' + - 'Hash plugin' + - 'Self Information' + - 'Tracker URL remover' + - 'Unit converter plugin' + - 'Ahmia blacklist' + + +.. _settings external_plugins: + +``plugins:`` (external) +======================= + +SearXNG supports *external plugins* / there is no need to install one, SearXNG +runs out of the box. But to demonstrate; in the example below we install the +SearXNG plugins from *The Green Web Foundation* `[ref] +<https://www.thegreenwebfoundation.org/news/searching-the-green-web-with-searx/>`__: + +.. code:: bash + + $ sudo utils/searxng.sh instance cmd bash -c + (searxng-pyenv)$ pip install git+https://github.com/return42/tgwf-searx-plugins + +In the :ref:`settings.yml` activate the ``plugins:`` section and add module +``only_show_green_results`` from ``tgwf-searx-plugins``. + +.. code:: yaml + + plugins: + - only_show_green_results + # - mypackage.mymodule.MyPlugin + # - mypackage.mymodule.MyOtherPlugin + +.. hint:: + + ``only_show_green_results`` is an old plugin that was still implemented in + the old style. There is a legacy treatment for backward compatibility, but + new plugins should be implemented as a :py:obj:`searx.plugins.Plugin` class. diff --git a/docs/conf.py b/docs/conf.py index fec9eb64b..aeb62e337 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -54,7 +54,7 @@ searx.engines.load_engines(searx.settings['engines']) jinja_contexts = { 'searx': { 'engines': searx.engines.engines, - 'plugins': searx.plugins.plugins, + 'plugins': searx.plugins.STORAGE, 'version': { 'node': os.getenv('NODE_MINIMUM_VERSION') }, @@ -129,8 +129,9 @@ extensions = [ 'notfound.extension', # https://github.com/readthedocs/sphinx-notfound-page ] +# autodoc_typehints = "description" autodoc_default_options = { - 'member-order': 'groupwise', + 'member-order': 'bysource', } myst_enable_extensions = [ diff --git a/docs/dev/answerers/builtins.rst b/docs/dev/answerers/builtins.rst new file mode 100644 index 000000000..cca020a7a --- /dev/null +++ b/docs/dev/answerers/builtins.rst @@ -0,0 +1,11 @@ +.. _builtin answerers: + +================== +Built-in Answerers +================== + +.. toctree:: + :maxdepth: 1 + + random + statistics diff --git a/docs/dev/answerers/development.rst b/docs/dev/answerers/development.rst new file mode 100644 index 000000000..04a1da0bb --- /dev/null +++ b/docs/dev/answerers/development.rst @@ -0,0 +1,7 @@ +.. _dev answerers: + +==================== +Answerer Development +==================== + +.. automodule:: searx.answerers diff --git a/docs/dev/answerers/index.rst b/docs/dev/answerers/index.rst new file mode 100644 index 000000000..02af238bf --- /dev/null +++ b/docs/dev/answerers/index.rst @@ -0,0 +1,9 @@ +========= +Answerers +========= + +.. toctree:: + :maxdepth: 2 + + development + builtins diff --git a/docs/dev/answerers/random.rst b/docs/dev/answerers/random.rst new file mode 100644 index 000000000..4489d2a34 --- /dev/null +++ b/docs/dev/answerers/random.rst @@ -0,0 +1,8 @@ +.. _answerer.random: + +====== +Random +====== + +.. autoclass:: searx.answerers.random.SXNGAnswerer + :members: diff --git a/docs/dev/answerers/statistics.rst b/docs/dev/answerers/statistics.rst new file mode 100644 index 000000000..86ae6861e --- /dev/null +++ b/docs/dev/answerers/statistics.rst @@ -0,0 +1,8 @@ +.. _answerer.statistics: + +========== +Statistics +========== + +.. autoclass:: searx.answerers.statistics.SXNGAnswerer + :members: diff --git a/docs/dev/engines/engine_overview.rst b/docs/dev/engines/engine_overview.rst index 1a6270a3b..4aeb5b624 100644 --- a/docs/dev/engines/engine_overview.rst +++ b/docs/dev/engines/engine_overview.rst @@ -237,335 +237,18 @@ following parameters can be used to specify a search request: =================== =========== ========================================================================== -.. _engine results: -.. _engine media types: +Making a Response +================= -Result Types (``template``) -=========================== +In the ``response`` function of the engine, the HTTP response (``resp``) is +parsed and a list of results is returned. -Each result item of an engine can be of different media-types. Currently the -following media-types are supported. To set another media-type as -:ref:`template default`, the parameter ``template`` must be set to the desired -type. +A engine can append result-items of different media-types and different +result-types to the result list. The list of the result items is render to HTML +by templates. For more details read section: -.. _template default: +- :ref:`simple theme templates` +- :ref:`result types` -``default`` ------------ - -.. table:: Parameter of the **default** media type: - :width: 100% - - ========================= ===================================================== - result-parameter information - ========================= ===================================================== - url string, url of the result - title string, title of the result - content string, general result-text - publishedDate :py:class:`datetime.datetime`, time of publish - ========================= ===================================================== - - -.. _template images: - -``images`` ----------- - -.. list-table:: Parameter of the **images** media type - :header-rows: 2 - :width: 100% - - * - result-parameter - - Python type - - information - - * - template - - :py:class:`str` - - is set to ``images.html`` - - * - url - - :py:class:`str` - - url to the result site - - * - title - - :py:class:`str` - - title of the result - - * - content - - :py:class:`str` - - description of the image - - * - publishedDate - - :py:class:`datetime <datetime.datetime>` - - time of publish - - * - img_src - - :py:class:`str` - - url to the result image - - * - thumbnail_src - - :py:class:`str` - - url to a small-preview image - - * - resolution - - :py:class:`str` - - the resolution of the image (e.g. ``1920 x 1080`` pixel) - - * - img_format - - :py:class:`str` - - the format of the image (e.g. ``png``) - - * - filesize - - :py:class:`str` - - size of bytes in :py:obj:`human readable <searx.humanize_bytes>` notation - (e.g. ``MB`` for 1024 \* 1024 Bytes filesize). - - -.. _template videos: - -``videos`` ----------- - -.. table:: Parameter of the **videos** media type: - :width: 100% - - ========================= ===================================================== - result-parameter information - ------------------------- ----------------------------------------------------- - template is set to ``videos.html`` - ========================= ===================================================== - url string, url of the result - title string, title of the result - content *(not implemented yet)* - publishedDate :py:class:`datetime.datetime`, time of publish - thumbnail string, url to a small-preview image - length :py:class:`datetime.timedelta`, duration of result - views string, view count in humanized number format - ========================= ===================================================== - - -.. _template torrent: - -``torrent`` ------------ - -.. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme - -.. table:: Parameter of the **torrent** media type: - :width: 100% - - ========================= ===================================================== - result-parameter information - ------------------------- ----------------------------------------------------- - template is set to ``torrent.html`` - ========================= ===================================================== - url string, url of the result - title string, title of the result - content string, general result-text - publishedDate :py:class:`datetime.datetime`, - time of publish *(not implemented yet)* - seed int, number of seeder - leech int, number of leecher - filesize int, size of file in bytes - files int, number of files - magnetlink string, magnetlink_ of the result - torrentfile string, torrentfile of the result - ========================= ===================================================== - - -.. _template map: - -``map`` -------- - -.. table:: Parameter of the **map** media type: - :width: 100% - - ========================= ===================================================== - result-parameter information - ------------------------- ----------------------------------------------------- - template is set to ``map.html`` - ========================= ===================================================== - url string, url of the result - title string, title of the result - content string, general result-text - publishedDate :py:class:`datetime.datetime`, time of publish - latitude latitude of result (in decimal format) - longitude longitude of result (in decimal format) - boundingbox boundingbox of result (array of 4. values - ``[lat-min, lat-max, lon-min, lon-max]``) - geojson geojson of result (https://geojson.org/) - osm.type type of osm-object (if OSM-Result) - osm.id id of osm-object (if OSM-Result) - address.name name of object - address.road street name of object - address.house_number house number of object - address.locality city, place of object - address.postcode postcode of object - address.country country of object - ========================= ===================================================== - - -.. _template paper: - -``paper`` ---------- - -.. _BibTeX format: https://www.bibtex.com/g/bibtex-format/ -.. _BibTeX field types: https://en.wikipedia.org/wiki/BibTeX#Field_types - -.. list-table:: Parameter of the **paper** media type / - see `BibTeX field types`_ and `BibTeX format`_ - :header-rows: 2 - :width: 100% - - * - result-parameter - - Python type - - information - - * - template - - :py:class:`str` - - is set to ``paper.html`` - - * - title - - :py:class:`str` - - title of the result - - * - content - - :py:class:`str` - - abstract - - * - comments - - :py:class:`str` - - free text display in italic below the content - - * - tags - - :py:class:`List <list>`\ [\ :py:class:`str`\ ] - - free tag list - - * - publishedDate - - :py:class:`datetime <datetime.datetime>` - - last publication date - - * - type - - :py:class:`str` - - short description of medium type, e.g. *book*, *pdf* or *html* ... - - * - authors - - :py:class:`List <list>`\ [\ :py:class:`str`\ ] - - list of authors of the work (authors with a "s") - - * - editor - - :py:class:`str` - - list of editors of a book - - * - publisher - - :py:class:`str` - - name of the publisher - - * - journal - - :py:class:`str` - - name of the journal or magazine the article was - published in - - * - volume - - :py:class:`str` - - volume number - - * - pages - - :py:class:`str` - - page range where the article is - - * - number - - :py:class:`str` - - number of the report or the issue number for a journal article - - * - doi - - :py:class:`str` - - DOI number (like ``10.1038/d41586-018-07848-2``) - - * - issn - - :py:class:`List <list>`\ [\ :py:class:`str`\ ] - - ISSN number like ``1476-4687`` - - * - isbn - - :py:class:`List <list>`\ [\ :py:class:`str`\ ] - - ISBN number like ``9780201896831`` - - * - pdf_url - - :py:class:`str` - - URL to the full article, the PDF version - - * - html_url - - :py:class:`str` - - URL to full article, HTML version - - -.. _template packages: - -``packages`` ------------- - -.. list-table:: Parameter of the **packages** media type - :header-rows: 2 - :width: 100% - - * - result-parameter - - Python type - - information - - * - template - - :py:class:`str` - - is set to ``packages.html`` - - * - title - - :py:class:`str` - - title of the result - - * - content - - :py:class:`str` - - abstract - - * - package_name - - :py:class:`str` - - the name of the package - - * - version - - :py:class:`str` - - the current version of the package - - * - maintainer - - :py:class:`str` - - the maintainer or author of the project - - * - publishedDate - - :py:class:`datetime <datetime.datetime>` - - date of latest update or release - - * - tags - - :py:class:`List <list>`\ [\ :py:class:`str`\ ] - - free tag list - - * - popularity - - :py:class:`str` - - the popularity of the package, e.g. rating or download count - - * - license_name - - :py:class:`str` - - the name of the license - - * - license_url - - :py:class:`str` - - the web location of a license copy - - * - homepage - - :py:class:`str` - - the url of the project's homepage - * - source_code_url - - :py:class:`str` - - the location of the project's source code - * - links - - :py:class:`dict` - - additional links in the form of ``{'link_name': 'http://example.com'}`` diff --git a/docs/dev/extended_types.rst b/docs/dev/extended_types.rst new file mode 100644 index 000000000..59013eef2 --- /dev/null +++ b/docs/dev/extended_types.rst @@ -0,0 +1,7 @@ +.. _extended_types.: + +============== +Extended Types +============== + +.. automodule:: searx.extended_types diff --git a/docs/dev/index.rst b/docs/dev/index.rst index 09be9de5e..01a16ba80 100644 --- a/docs/dev/index.rst +++ b/docs/dev/index.rst @@ -8,9 +8,13 @@ Developer documentation quickstart rtm_asdf contribution_guide + extended_types engines/index + result_types/index + templates search_api - plugins + plugins/index + answerers/index translation lxcdev makefile diff --git a/docs/dev/plugins.rst b/docs/dev/plugins.rst deleted file mode 100644 index fb3201e66..000000000 --- a/docs/dev/plugins.rst +++ /dev/null @@ -1,106 +0,0 @@ -.. _dev plugin: - -======= -Plugins -======= - -.. sidebar:: Further reading .. - - - :ref:`plugins generic` - -Plugins can extend or replace functionality of various components of searx. - -Example plugin -============== - -.. code:: python - - name = 'Example plugin' - description = 'This plugin extends the suggestions with the word "example"' - default_on = False # disabled by default - - # attach callback to the post search hook - # request: flask request object - # ctx: the whole local context of the post search hook - def post_search(request, search): - search.result_container.suggestions.add('example') - return True - -External plugins -================ - -SearXNG supports *external plugins* / there is no need to install one, SearXNG -runs out of the box. But to demonstrate; in the example below we install the -SearXNG plugins from *The Green Web Foundation* `[ref] -<https://www.thegreenwebfoundation.org/news/searching-the-green-web-with-searx/>`__: - -.. code:: bash - - $ sudo utils/searxng.sh instance cmd bash -c - (searxng-pyenv)$ pip install git+https://github.com/return42/tgwf-searx-plugins - -In the :ref:`settings.yml` activate the ``plugins:`` section and add module -``only_show_green_results`` from ``tgwf-searx-plugins``. - -.. code:: yaml - - plugins: - ... - - only_show_green_results - ... - - -Plugin entry points -=================== - -Entry points (hooks) define when a plugin runs. Right now only three hooks are -implemented. So feel free to implement a hook if it fits the behaviour of your -plugin. A plugin doesn't need to implement all the hooks. - - -.. py:function:: pre_search(request, search) -> bool - - Runs BEFORE the search request. - - `search.result_container` can be changed. - - Return a boolean: - - * True to continue the search - * False to stop the search - - :param flask.request request: - :param searx.search.SearchWithPlugins search: - :return: False to stop the search - :rtype: bool - - -.. py:function:: post_search(request, search) -> None - - Runs AFTER the search request. - - :param flask.request request: Flask request. - :param searx.search.SearchWithPlugins search: Context. - - -.. py:function:: on_result(request, search, result) -> bool - - Runs for each result of each engine. - - `result` can be changed. - - If `result["url"]` is defined, then `result["parsed_url"] = urlparse(result['url'])` - - .. warning:: - `result["url"]` can be changed, but `result["parsed_url"]` must be updated too. - - Return a boolean: - - * True to keep the result - * False to remove the result - - :param flask.request request: - :param searx.search.SearchWithPlugins search: - :param typing.Dict result: Result, see - :ref:`engine results` - :return: True to keep the result - :rtype: bool diff --git a/docs/dev/plugins/builtins.rst b/docs/dev/plugins/builtins.rst new file mode 100644 index 000000000..902092e11 --- /dev/null +++ b/docs/dev/plugins/builtins.rst @@ -0,0 +1,15 @@ +.. _builtin plugins: + +================ +Built-in Plugins +================ + +.. toctree:: + :maxdepth: 1 + + calculator + hash_plugin + hostnames + self_info + tor_check + unit_converter diff --git a/docs/dev/plugins/calculator.rst b/docs/dev/plugins/calculator.rst new file mode 100644 index 000000000..7ca9e32ed --- /dev/null +++ b/docs/dev/plugins/calculator.rst @@ -0,0 +1,8 @@ +.. _plugins.calculator: + +========== +Calculator +========== + +.. automodule:: searx.plugins.calculator + :members: diff --git a/docs/dev/plugins/development.rst b/docs/dev/plugins/development.rst new file mode 100644 index 000000000..6bece72c0 --- /dev/null +++ b/docs/dev/plugins/development.rst @@ -0,0 +1,7 @@ +.. _dev plugin: + +================== +Plugin Development +================== + +.. automodule:: searx.plugins diff --git a/docs/dev/plugins/hash_plugin.rst b/docs/dev/plugins/hash_plugin.rst new file mode 100644 index 000000000..a5ac0728f --- /dev/null +++ b/docs/dev/plugins/hash_plugin.rst @@ -0,0 +1,8 @@ +.. _hash_plugin plugin: + +=========== +Hash Values +=========== + +.. autoclass:: searx.plugins.hash_plugin.SXNGPlugin + :members: diff --git a/docs/src/searx.plugins.hostnames.rst b/docs/dev/plugins/hostnames.rst index 5d0365536..434f170ea 100644 --- a/docs/src/searx.plugins.hostnames.rst +++ b/docs/dev/plugins/hostnames.rst @@ -1,9 +1,8 @@ .. _hostnames plugin: -================ -Hostnames plugin -================ +========= +Hostnames +========= .. automodule:: searx.plugins.hostnames - :members: - + :members: diff --git a/docs/dev/plugins/index.rst b/docs/dev/plugins/index.rst new file mode 100644 index 000000000..bed8095e2 --- /dev/null +++ b/docs/dev/plugins/index.rst @@ -0,0 +1,9 @@ +======= +Plugins +======= + +.. toctree:: + :maxdepth: 2 + + development + builtins diff --git a/docs/dev/plugins/self_info.rst b/docs/dev/plugins/self_info.rst new file mode 100644 index 000000000..a1dd66b17 --- /dev/null +++ b/docs/dev/plugins/self_info.rst @@ -0,0 +1,8 @@ +.. _self_info plugin: + +========= +Self-Info +========= + +.. autoclass:: searx.plugins.self_info.SXNGPlugin + :members: diff --git a/docs/src/searx.plugins.tor_check.rst b/docs/dev/plugins/tor_check.rst index 905328ebf..f1a02381e 100644 --- a/docs/src/searx.plugins.tor_check.rst +++ b/docs/dev/plugins/tor_check.rst @@ -1,9 +1,8 @@ .. _tor check plugin: -================ -Tor check plugin -================ +========= +Tor check +========= .. automodule:: searx.plugins.tor_check - :members: - + :members: diff --git a/docs/src/searx.plugins.unit_converter.rst b/docs/dev/plugins/unit_converter.rst index 48d495d80..2e19bc38e 100644 --- a/docs/src/searx.plugins.unit_converter.rst +++ b/docs/dev/plugins/unit_converter.rst @@ -1,9 +1,8 @@ .. _unit converter plugin: -===================== -Unit converter plugin -===================== +============== +Unit Converter +============== .. automodule:: searx.plugins.unit_converter :members: - diff --git a/docs/dev/result_types/answer.rst b/docs/dev/result_types/answer.rst new file mode 100644 index 000000000..72368be39 --- /dev/null +++ b/docs/dev/result_types/answer.rst @@ -0,0 +1,7 @@ +.. _result_types.answer: + +============== +Answer Results +============== + +.. automodule:: searx.result_types.answer diff --git a/docs/dev/result_types/base_result.rst b/docs/dev/result_types/base_result.rst new file mode 100644 index 000000000..24fd96a9d --- /dev/null +++ b/docs/dev/result_types/base_result.rst @@ -0,0 +1,5 @@ +====== +Result +====== + +.. automodule:: searx.result_types._base diff --git a/docs/dev/result_types/correction.rst b/docs/dev/result_types/correction.rst new file mode 100644 index 000000000..51cdd45d2 --- /dev/null +++ b/docs/dev/result_types/correction.rst @@ -0,0 +1,34 @@ +.. _result_types.corrections: + +================== +Correction Results +================== + +.. hint:: + + There is still no typing for these result items. The templates can be used as + orientation until the final typing is complete. + +The corrections area shows the user alternative search terms. + +A result of this type is a very simple dictionary with only one key/value pair + +.. code:: python + + {"correction" : "lorem ipsum .."} + +From this simple dict another dict is build up: + +.. code:: python + + # use RawTextQuery to get the corrections URLs with the same bang + {"url" : "!bang lorem ipsum ..", "title": "lorem ipsum .." } + +and used in the template :origin:`corrections.html +<searx/templates/simple/elements/corrections.html>`: + +title : :py:class:`str` + Corrected search term. + +url : :py:class:`str` + Not really an URL, its the value to insert in a HTML form for a SearXNG query. diff --git a/docs/dev/result_types/index.rst b/docs/dev/result_types/index.rst new file mode 100644 index 000000000..e90dba849 --- /dev/null +++ b/docs/dev/result_types/index.rst @@ -0,0 +1,95 @@ +.. _result types: + +============ +Result Types +============ + +To understand the typification of the results, let's take a brief look at the +structure of SearXNG .. At its core, SearXNG is nothing more than an aggregator +that aggregates the results from various sources, renders them via templates and +displays them to the user. + +The **sources** can be: + +1. :ref:`engines <engine implementations>` +2. :ref:`plugins <dev plugin>` +3. :ref:`answerers <dev answerers>` + +The sources provide the results, which are displayed in different **areas** +depending on the type of result. The areas are: + +main results: + It is the main area in which -- as is typical for search engines -- the + results that a search engine has found for the search term are displayed. + +answers: + This area displays short answers that could be found for the search term. + +info box: + An area in which additional information can be displayed, e.g. excerpts from + wikipedia or other sources such as maps. + +suggestions: + Suggestions for alternative search terms can be found in this area. These can + be clicked on and a search is carried out with these search terms. + +corrections: + Results in this area are like the suggestion of alternative search terms, + which usually result from spelling corrections + +At this point it is important to note that all **sources** can contribute +results to all of the areas mentioned above. + +In most cases, however, the :ref:`engines <engine implementations>` will fill +the *main results* and the :ref:`answerers <dev answerers>` will generally +provide the contributions for the *answer* area. Not necessary to mention here +but for a better understanding: the plugins can also filter out or change +results from the main results area (e.g. the URL of the link). + +The result items are organized in the :py:obj:`results.ResultContainer` and +after all sources have delivered their results, this container is passed to the +templating to build a HTML output. The output is usually HTML, but it is also +possible to output the result lists as JSON or RSS feed. Thats quite all we need +to know before we dive into typification of result items. + +.. hint:: + + Typification of result items: we are at the very first beginng! + +The first thing we have to realize is that there is no typification of the +result items so far, we have to build it up first .. and that is quite a big +task, which we will only be able to accomplish gradually. + +The foundation for the typeless results was laid back in 2013 in the very first +commit :commit:`ae9fb1d7d`, and the principle has not changed since then. At +the time, the approach was perfectly adequate, but we have since evolved and the +demands on SearXNG increase with every feature request. + +**Motivation:** in the meantime, it has become very difficult to develop new +features that require structural changes and it is especially hard for newcomers +to find their way in this typeless world. As long as the results are only +simple key/value dictionaries, it is not even possible for the IDEs to support +the application developer in his work. + +**Planning:** The procedure for subsequent typing will have to be based on the +circumstances .. + +.. attention:: + + As long as there is no type defined for a kind of result the HTML template + specify what the properties of a type are. + + In this sense, you will either find a type definition here in the + documentation or, if this does not yet exist, a description of the HTML + template. + + +.. toctree:: + :maxdepth: 2 + + base_result + main_result + answer + correction + suggestion + infobox diff --git a/docs/dev/result_types/infobox.rst b/docs/dev/result_types/infobox.rst new file mode 100644 index 000000000..428dc8db7 --- /dev/null +++ b/docs/dev/result_types/infobox.rst @@ -0,0 +1,60 @@ +.. _result_types.infobox: + +=============== +Infobox Results +=============== + +.. hint:: + + There is still no typing for these result items. The templates can be used as + orientation until the final typing is complete. + +The infobox is an area where addtional infos shown to the user. + +Fields used in the :origin:`infobox.html +<searx/templates/simple/elements/infobox.html>`: + +img_src: :py:class:`str` + URL of a image or thumbnail that is displayed in the infobox. + +infobox: :py:class:`str` + Title of the info box. + +content: :py:class:`str` + Text of the info box. + +The infobox has additional subsections for *attributes*, *urls* and +*relatedTopics*: + +attributes: :py:class:`List <list>`\ [\ :py:class:`dict`\ ] + A list of attributes. An *attribute* is a dictionary with keys: + + - label :py:class:`str`: (mandatory) + + - value :py:class:`str`: (mandatory) + + - image :py:class:`List <list>`\ [\ :py:class:`dict`\ ] (optional) + + A list of images. An *image* is a dictionary with keys: + + - src :py:class:`str`: URL of an image/thumbnail (mandatory) + - alt :py:class:`str`: alternative text for the image (mandatory) + +urls: :py:class:`List <list>`\ [\ :py:class:`dict`\ ] + A list of links. An *link* is a dictionary with keys: + + - url :py:class:`str`: URL of the link (mandatory) + - title :py:class:`str`: Title of the link (mandatory) + +relatedTopics: :py:class:`List <list>`\ [\ :py:class:`dict`\ ] + A list of topics. An *topic* is a dictionary with keys: + + - name: :py:class:`str`: (mandatory) + + - suggestions: :py:class:`List <list>`\ [\ :py:class:`dict`\ ] (optional) + + A list of suggestions. A *suggestion* is simple dictionary with just one + key/value pair: + + - suggestion: :py:class:`str`: suggested search term (mandatory) + diff --git a/docs/dev/result_types/main_result.rst b/docs/dev/result_types/main_result.rst new file mode 100644 index 000000000..1f178cbd1 --- /dev/null +++ b/docs/dev/result_types/main_result.rst @@ -0,0 +1,17 @@ +============ +Main Results +============ + +There is still no typing for the items in the :ref:`main result list`. The +templates can be used as orientation until the final typing is complete. + +- :ref:`template default` +- :ref:`template images` +- :ref:`template videos` +- :ref:`template torrent` +- :ref:`template map` +- :ref:`template paper` +- :ref:`template packages` +- :ref:`template code` +- :ref:`template files` +- :ref:`template products` diff --git a/docs/dev/result_types/suggestion.rst b/docs/dev/result_types/suggestion.rst new file mode 100644 index 000000000..52e8a05a8 --- /dev/null +++ b/docs/dev/result_types/suggestion.rst @@ -0,0 +1,38 @@ +.. _result_types.suggestion: + +================== +Suggestion Results +================== + +.. hint:: + + There is still no typing for these result items. The templates can be used as + orientation until the final typing is complete. + +The suggestions area shows the user alternative search terms. + +A result of this type is a very simple dictionary with only one key/value pair + +.. code:: python + + {"suggestion" : "lorem ipsum .."} + +From this simple dict another dict is build up: + +.. code:: python + + {"url" : "!bang lorem ipsum ..", "title": "lorem ipsum" } + +and used in the template :origin:`suggestions.html +<searx/templates/simple/elements/suggestions.html>`: + +.. code:: python + + # use RawTextQuery to get the suggestion URLs with the same bang + {"url" : "!bang lorem ipsum ..", "title": "lorem ipsum" } + +title : :py:class:`str` + Suggested search term + +url : :py:class:`str` + Not really an URL, its the value to insert in a HTML form for a SearXNG query. diff --git a/docs/dev/searxng_extra/update.rst b/docs/dev/searxng_extra/update.rst index dc3b06744..000085970 100644 --- a/docs/dev/searxng_extra/update.rst +++ b/docs/dev/searxng_extra/update.rst @@ -60,6 +60,7 @@ Scripts to update static data in :origin:`searx/data/` .. automodule:: searxng_extra.update.update_engine_traits :members: +.. _update_osm_keys_tags.py: ``update_osm_keys_tags.py`` =========================== diff --git a/docs/dev/templates.rst b/docs/dev/templates.rst new file mode 100644 index 000000000..b62c6b732 --- /dev/null +++ b/docs/dev/templates.rst @@ -0,0 +1,577 @@ +.. _simple theme templates: + +====================== +Simple Theme Templates +====================== + +The simple template is complex, it consists of many different elements and also +uses macros and include statements. The following is a rough overview that we +would like to give the developerat hand, details must still be taken from the +:origin:`sources <searx/templates/simple/>`. + +A :ref:`result item <result types>` can be of different media types. The media +type of a result is defined by the :py:obj:`result_type.Result.template`. To +set another media-type as :ref:`template default`, the field ``template`` +in the result item must be set to the desired type. + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + + +.. _result template macros: + +Result template macros +====================== + +.. _macro result_header: + +``result_header`` +----------------- + +Execpt ``image.html`` and some others this macro is used in nearly all result +types in the :ref:`main result list`. + +Fields used in the template :origin:`macro result_header +<searx/templates/simple/macros.html>`: + +url : :py:class:`str` + Link URL of the result item. + +title : :py:class:`str` + Link title of the result item. + +img_src, thumbnail : :py:class:`str` + URL of a image or thumbnail that is displayed in the result item. + + +.. _macro result_sub_header: + +``result_sub_header`` +--------------------- + +Execpt ``image.html`` and some others this macro is used in nearly all result +types in the :ref:`main result list`. + +Fields used in the template :origin:`macro result_sub_header +<searx/templates/simple/macros.html>`: + +publishedDate : :py:obj:`datetime.datetime` + The date on which the object was published. + +length: :py:obj:`time.struct_time` + Playing duration in seconds. + +views: :py:class:`str` + View count in humanized number format. + +author : :py:class:`str` + Author of the title. + +metadata : :py:class:`str` + Miscellaneous metadata. + + +.. _engine_data: + +``engine_data_form`` +-------------------- + +The ``engine_data_form`` macro is used in :origin:`results,html +<searx/templates/simple/results.html>` in a HTML ``<form/>`` element. The +intention of this macro is to pass data of a engine from one :py:obj:`response +<searx.engines.demo_online.response>` to the :py:obj:`searx.search.SearchQuery` +of the next :py:obj:`request <searx.engines.demo_online.request>`. + +To pass data, engine's response handler can append result items of typ +``engine_data``. This is by example used to pass a token from the response to +the next request: + +.. code:: python + + def response(resp): + ... + results.append({ + 'engine_data': token, + 'key': 'next_page_token', + }) + ... + return results + + def request(query, params): + page_token = params['engine_data'].get('next_page_token') + + +.. _main result list: + +Main Result List +================ + +The **media types** of the **main result type** are the template files in +the :origin:`result_templates <searx/templates/simple/result_templates>`. + +.. _template default: + +``default.html`` +---------------- + +Displays result fields from: + +- :ref:`macro result_header` and +- :ref:`macro result_sub_header` + +Additional fields used in the :origin:`default.html +<searx/templates/simple/result_templates/default.html>`: + +content : :py:class:`str` + General text of the result item. + +iframe_src : :py:class:`str` + URL of an embedded ``<iframe>`` / the frame is collapsible. + +audio_src : uri, + URL of an embedded ``<audio controls>``. + + +.. _template images: + +``images.html`` +--------------- + +The images are displayed as small thumbnails in the main results list. + +title : :py:class:`str` + Title of the image. + +thumbnail_src : :py:class:`str` + URL of a preview of the image. + +resolution :py:class:`str` + The resolution of the image (e.g. ``1920 x 1080`` pixel) + + +Image labels +~~~~~~~~~~~~ + +Clicking on the preview opens a gallery view in which all further metadata for +the image is displayed. Addition fields used in the :origin:`images.html +<searx/templates/simple/result_templates/images.html>`: + +img_src : :py:class:`str` + URL of the full size image. + +content: :py:class:`str` + Description of the image. + +author: :py:class:`str` + Name of the author of the image. + +img_format : :py:class:`str` + The format of the image (e.g. ``png``). + +source : :py:class:`str` + Source of the image. + +filesize: :py:class:`str` + Size of bytes in :py:obj:`human readable <searx.humanize_bytes>` notation + (e.g. ``MB`` for 1024 \* 1024 Bytes filesize). + +url : :py:class:`str` + URL of the page from where the images comes from (source). + + +.. _template videos: + +``videos.html`` +--------------- + +Displays result fields from: + +- :ref:`macro result_header` and +- :ref:`macro result_sub_header` + +Additional fields used in the :origin:`videos.html +<searx/templates/simple/result_templates/videos.html>`: + +iframe_src : :py:class:`str` + URL of an embedded ``<iframe>`` / the frame is collapsible. + + The videos are displayed as small thumbnails in the main results list, there + is an additional button to collaps/open the embeded video. + +content : :py:class:`str` + Description of the code fragment. + + +.. _template torrent: + +``torrent.html`` +---------------- + +.. _magnet link: https://en.wikipedia.org/wiki/Magnet_URI_scheme +.. _torrent file: https://en.wikipedia.org/wiki/Torrent_file + +Displays result fields from: + +- :ref:`macro result_header` and +- :ref:`macro result_sub_header` + +Additional fields used in the :origin:`torrent.html +<searx/templates/simple/result_templates/torrent.html>`: + +magnetlink: + URL of the `magnet link`_. + +torrentfile + URL of the `torrent file`_. + +seed : ``int`` + Number of seeders. + +leech : ``int`` + Number of leecher + +filesize : ``int`` + Size in Bytes (rendered to human readable unit of measurement). + +files : ``int`` + Number of files. + + +.. _template map: + +``map.html`` +------------ + +.. _GeoJSON: https://en.wikipedia.org/wiki/GeoJSON +.. _Leaflet: https://github.com/Leaflet/Leaflet +.. _bbox: https://wiki.openstreetmap.org/wiki/Bounding_Box +.. _HTMLElement.dataset: https://developer.mozilla.org/en-US/docs/Web/API/HTMLElement/dataset +.. _Nominatim: https://nominatim.org/release-docs/latest/ +.. _Lookup: https://nominatim.org/release-docs/latest/api/Lookup/ +.. _place_id is not a persistent id: + https://nominatim.org/release-docs/latest/api/Output/#place_id-is-not-a-persistent-id +.. _perma_id: https://wiki.openstreetmap.org/wiki/Permanent_ID +.. _country code: https://wiki.openstreetmap.org/wiki/Country_code + +Displays result fields from: + +- :ref:`macro result_header` and +- :ref:`macro result_sub_header` + +Additional fields used in the :origin:`map.html +<searx/templates/simple/result_templates/map.html>`: + +content : :py:class:`str` + Description of the item. + +address_label : :py:class:`str` + Label of the address / default ``_('address')``. + +geojson : GeoJSON_ + Geometries mapped to HTMLElement.dataset_ (``data-map-geojson``) and used by + Leaflet_. + +boundingbox : ``[ min-lon, min-lat, max-lon, max-lat]`` + A bbox_ area defined by min longitude , min latitude , max longitude and max + latitude. The bounding box is mapped to HTMLElement.dataset_ + (``data-map-boundingbox``) and is used by Leaflet_. + +longitude, latitude : :py:class:`str` + Geographical coordinates, mapped to HTMLElement.dataset_ (``data-map-lon``, + ``data-map-lat``) and is used by Leaflet_. + +address : ``{...}`` + A dicticonary with the address data: + + .. code:: python + + address = { + 'name' : str, # name of object + 'road' : str, # street name of object + 'house_number' : str, # house number of object + 'postcode' : str, # postcode of object + 'country' : str, # country of object + 'country_code' : str, + 'locality' : str, + } + + country_code : :py:class:`str` + `Country code`_ of the object. + + locality : :py:class:`str` + The name of the city, town, township, village, borough, etc. in which this + object is located. + +links : ``[link1, link2, ...]`` + A list of links with labels: + + .. code:: python + + links.append({ + 'label' : str, + 'url' : str, + 'url_label' : str, # set by some engines but unused (oscar) + }) + +data : ``[data1, data2, ...]`` + A list of additional data, shown in two columns and containing a label and + value. + + .. code:: python + + data.append({ + 'label' : str, + 'value' : str, + 'key' : str, # set by some engines but unused + }) + +type : :py:class:`str` # set by some engines but unused (oscar) + Tag label from :ref:`OSM_KEYS_TAGS['tags'] <update_osm_keys_tags.py>`. + +type_icon : :py:class:`str` # set by some engines but unused (oscar) + Type's icon. + +osm : ``{...}`` + OSM-type and OSM-ID, can be used to Lookup_ OSM data (Nominatim_). There is + also a discussion about "`place_id is not a persistent id`_" and the + perma_id_. + + .. code:: python + + osm = { + 'type': str, + 'id': str, + } + + type : :py:class:`str` + Type of osm-object (if OSM-Result). + + id : + ID of osm-object (if OSM-Result). + + .. hint:: + + The ``osm`` property is set by engine ``openstreetmap.py``, but it is not + used in the ``map.html`` template yet. + + + +.. _template paper: + +``paper.html`` +-------------- + +.. _BibTeX format: https://www.bibtex.com/g/bibtex-format/ +.. _BibTeX field types: https://en.wikipedia.org/wiki/BibTeX#Field_types + +Displays result fields from: + +- :ref:`macro result_header` + +Additional fields used in the :origin:`paper.html +<searx/templates/simple/result_templates/paper.html>`: + +content : :py:class:`str` + An abstract or excerpt from the document. + +comments : :py:class:`str` + Free text display in italic below the content. + +tags : :py:class:`List <list>`\ [\ :py:class:`str`\ ] + Free tag list. + +type : :py:class:`str` + Short description of medium type, e.g. *book*, *pdf* or *html* ... + +authors : :py:class:`List <list>`\ [\ :py:class:`str`\ ] + List of authors of the work (authors with a "s" suffix, the "author" is in the + :ref:`macro result_sub_header`). + +editor : :py:class:`str` + Editor of the book/paper. + +publisher : :py:class:`str` + Name of the publisher. + +journal : :py:class:`str` + Name of the journal or magazine the article was published in. + +volume : :py:class:`str` + Volume number. + +pages : :py:class:`str` + Page range where the article is. + +number : :py:class:`str` + Number of the report or the issue number for a journal article. + +doi : :py:class:`str` + DOI number (like ``10.1038/d41586-018-07848-2``). + +issn : :py:class:`List <list>`\ [\ :py:class:`str`\ ] + ISSN number like ``1476-4687`` + +isbn : :py:class:`List <list>`\ [\ :py:class:`str`\ ] + ISBN number like ``9780201896831`` + +pdf_url : :py:class:`str` + URL to the full article, the PDF version + +html_url : :py:class:`str` + URL to full article, HTML version + + +.. _template packages: + +``packages`` +------------ + +Displays result fields from: + +- :ref:`macro result_header` + +Additional fields used in the :origin:`packages.html +<searx/templates/simple/result_templates/packages.html>`: + +package_name : :py:class:`str` + The name of the package. + +version : :py:class:`str` + The current version of the package. + +maintainer : :py:class:`str` + The maintainer or author of the project. + +publishedDate : :py:class:`datetime <datetime.datetime>` + Date of latest update or release. + +tags : :py:class:`List <list>`\ [\ :py:class:`str`\ ] + Free tag list. + +popularity : :py:class:`str` + The popularity of the package, e.g. rating or download count. + +license_name : :py:class:`str` + The name of the license. + +license_url : :py:class:`str` + The web location of a license copy. + +homepage : :py:class:`str` + The url of the project's homepage. + +source_code_url: :py:class:`str` + The location of the project's source code. + +links : :py:class:`dict` + Additional links in the form of ``{'link_name': 'http://example.com'}`` + + +.. _template code: + +``code.html`` +------------- + +Displays result fields from: + +- :ref:`macro result_header` and +- :ref:`macro result_sub_header` + +Additional fields used in the :origin:`code.html +<searx/templates/simple/result_templates/code.html>`: + +content : :py:class:`str` + Description of the code fragment. + +codelines : ``[line1, line2, ...]`` + Lines of the code fragment. + +code_language : :py:class:`str` + Name of the code language, the value is passed to + :py:obj:`pygments.lexers.get_lexer_by_name`. + +repository : :py:class:`str` + URL of the repository of the code fragment. + + +.. _template files: + +``files.html`` +-------------- + +Displays result fields from: + +- :ref:`macro result_header` and +- :ref:`macro result_sub_header` + +Additional fields used in the :origin:`code.html +<searx/templates/simple/result_templates/files.html>`: + +filename, size, time: :py:class:`str` + Filename, Filesize and Date of the file. + +mtype : ``audio`` | ``video`` | :py:class:`str` + Mimetype type of the file. + +subtype : :py:class:`str` + Mimetype / subtype of the file. + +abstract : :py:class:`str` + Abstract of the file. + +author : :py:class:`str` + Name of the author of the file + +embedded : :py:class:`str` + URL of an embedded media type (``audio`` or ``video``) / is collapsible. + + +.. _template products: + +``products.html`` +----------------- + +Displays result fields from: + +- :ref:`macro result_header` and +- :ref:`macro result_sub_header` + +Additional fields used in the :origin:`products.html +<searx/templates/simple/result_templates/products.html>`: + +content : :py:class:`str` + Description of the product. + +price : :py:class:`str` + The price must include the currency. + +shipping : :py:class:`str` + Shipping details. + +source_country : :py:class:`str` + Place from which the shipment is made. + + +.. _template answer results: + +Answer results +============== + +See :ref:`result_types.answer` + +Suggestion results +================== + +See :ref:`result_types.suggestion` + +Correction results +================== + +See :ref:`result_types.corrections` + +Infobox results +=============== + +See :ref:`result_types.infobox` diff --git a/docs/src/index.rst b/docs/src/index.rst index fd16e7635..7806a9c83 100644 --- a/docs/src/index.rst +++ b/docs/src/index.rst @@ -2,9 +2,9 @@ Source-Code =========== -This is a partial documentation of our source code. We are not aiming to document -every item from the source code, but we will add documentation when requested. - +This is a partial documentation of our source code. We are not aiming to +document every item from the source code, but we will add documentation when +requested. .. toctree:: :maxdepth: 2 |