From edfbf1e1183815cea3b723f3b66260bc55679f32 Mon Sep 17 00:00:00 2001
From: Markus Heiser <markus.heiser@darmarit.de>
Date: Sun, 15 Dec 2024 09:59:50 +0100
Subject: [refactor] typification of SearXNG (initial) / result items (part 1)

Typification of SearXNG
=======================

This patch introduces the typing of the results.  The why and how is described
in the documentation, please generate the documentation ..

    $ make docs.clean docs.live

and read the following articles in the "Developer documentation":

- result types --> http://0.0.0.0:8000/dev/result_types/index.html

The result types are available from the `searx.result_types` module.  The
following have been implemented so far:

- base result type: `searx.result_type.Result`
  --> http://0.0.0.0:8000/dev/result_types/base_result.html

- answer results
  --> http://0.0.0.0:8000/dev/result_types/answer.html

including the type for translations (inspired by #3925).  For all other
types (which still need to be set up in subsequent PRs), template documentation
has been created for the transition period.

Doc of the fields used in Templates
===================================

The template documentation is the basis for the typing and is the first complete
documentation of the results (needed for engine development).  It is the
"working paper" (the plan) with which further typifications can be implemented
in subsequent PRs.

- https://github.com/searxng/searxng/issues/357

Answer Templates
================

With the new (sub) types for `Answer`, the templates for the answers have also
been revised, `Translation` are now displayed with collapsible entries (inspired
by #3925).

    !en-de dog

Plugins & Answerer
==================

The implementation for `Plugin` and `Answer` has been revised, see
documentation:

- Plugin: http://0.0.0.0:8000/dev/plugins/index.html
- Answerer: http://0.0.0.0:8000/dev/answerers/index.html

With `AnswerStorage` and `AnswerStorage` to manage those items (in follow up
PRs, `ArticleStorage`, `InfoStorage` and .. will be implemented)

Autocomplete
============

The autocompletion had a bug where the results from `Answer` had not been shown
in the past.  To test activate autocompletion and try search terms for which we
have answerers

- statistics: type `min 1 2 3` .. in the completion list you should find an
  entry like `[de] min(1, 2, 3) = 1`

- random: type `random uuid` .. in the completion list, the first item is a
  random UUID

Extended Types
==============

SearXNG extends e.g. the request and response types of flask and httpx, a module
has been set up for type extensions:

- Extended Types
  --> http://0.0.0.0:8000/dev/extended_types.html

Unit-Tests
==========

The unit tests have been completely revised.  In the previous implementation,
the runtime (the global variables such as `searx.settings`) was not initialized
before each test, so the runtime environment with which a test ran was always
determined by the tests that ran before it.  This was also the reason why we
sometimes had to observe non-deterministic errors in the tests in the past:

- https://github.com/searxng/searxng/issues/2988 is one example for the Runtime
  issues, with non-deterministic behavior ..

- https://github.com/searxng/searxng/pull/3650
- https://github.com/searxng/searxng/pull/3654
- https://github.com/searxng/searxng/pull/3642#issuecomment-2226884469
- https://github.com/searxng/searxng/pull/3746#issuecomment-2300965005

Why msgspec.Struct
==================

We have already discussed typing based on e.g. `TypeDict` or `dataclass` in the past:

- https://github.com/searxng/searxng/pull/1562/files
- https://gist.github.com/dalf/972eb05e7a9bee161487132a7de244d2
- https://github.com/searxng/searxng/pull/1412/files
- https://github.com/searxng/searxng/pull/1356

In my opinion, TypeDict is unsuitable because the objects are still dictionaries
and not instances of classes / the `dataclass` are classes but ...

The `msgspec.Struct` combine the advantages of typing, runtime behaviour and
also offer the option of (fast) serializing (incl. type check) the objects.

Currently not possible but conceivable with `msgspec`: Outsourcing the engines
into separate processes, what possibilities this opens up in the future is left
to the imagination!

Internally, we have already defined that it is desirable to decouple the
development of the engines from the development of the SearXNG core / The
serialization of the `Result` objects is a prerequisite for this.

HINT: The threads listed above were the template for this PR, even though the
implementation here is based on msgspec.  They should also be an inspiration for
the following PRs of typification, as the models and implementations can provide
a good direction.

Why just one commit?
====================

I tried to create several (thematically separated) commits, but gave up at some
point ... there are too many things to tackle at once / The comprehensibility of
the commits would not be improved by a thematic separation. On the contrary, we
would have to make multiple changes at the same places and the goal of a change
would be vaguely recognizable in the fog of the commits.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
---
 docs/dev/templates.rst | 577 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 577 insertions(+)
 create mode 100644 docs/dev/templates.rst

(limited to 'docs/dev/templates.rst')

diff --git a/docs/dev/templates.rst b/docs/dev/templates.rst
new file mode 100644
index 000000000..b62c6b732
--- /dev/null
+++ b/docs/dev/templates.rst
@@ -0,0 +1,577 @@
+.. _simple theme templates:
+
+======================
+Simple Theme Templates
+======================
+
+The simple template is complex, it consists of many different elements and also
+uses macros and include statements.  The following is a rough overview that we
+would like to give the developerat hand, details must still be taken from the
+:origin:`sources <searx/templates/simple/>`.
+
+A :ref:`result item <result types>` can be of different media types.  The media
+type of a result is defined by the :py:obj:`result_type.Result.template`.  To
+set another media-type as :ref:`template default`, the field ``template``
+in the result item must be set to the desired type.
+
+.. contents:: Contents
+   :depth: 2
+   :local:
+   :backlinks: entry
+
+
+.. _result template macros:
+
+Result template macros
+======================
+
+.. _macro result_header:
+
+``result_header``
+-----------------
+
+Execpt ``image.html`` and some others this macro is used in nearly all result
+types in the :ref:`main result list`.
+
+Fields used in the template :origin:`macro result_header
+<searx/templates/simple/macros.html>`:
+
+url :  :py:class:`str`
+  Link URL of the result item.
+
+title :  :py:class:`str`
+  Link title of the result item.
+
+img_src, thumbnail : :py:class:`str`
+  URL of a image or thumbnail that is displayed in the result item.
+
+
+.. _macro result_sub_header:
+
+``result_sub_header``
+---------------------
+
+Execpt ``image.html`` and some others this macro is used in nearly all result
+types in the :ref:`main result list`.
+
+Fields used in the template :origin:`macro result_sub_header
+<searx/templates/simple/macros.html>`:
+
+publishedDate : :py:obj:`datetime.datetime`
+  The date on which the object was published.
+
+length: :py:obj:`time.struct_time`
+  Playing duration in seconds.
+
+views: :py:class:`str`
+  View count in humanized number format.
+
+author : :py:class:`str`
+  Author of the title.
+
+metadata : :py:class:`str`
+  Miscellaneous metadata.
+
+
+.. _engine_data:
+
+``engine_data_form``
+--------------------
+
+The ``engine_data_form`` macro is used in :origin:`results,html
+<searx/templates/simple/results.html>` in a HTML ``<form/>`` element.  The
+intention of this macro is to pass data of a engine from one :py:obj:`response
+<searx.engines.demo_online.response>` to the :py:obj:`searx.search.SearchQuery`
+of the next :py:obj:`request <searx.engines.demo_online.request>`.
+
+To pass data, engine's response handler can append result items of typ
+``engine_data``.  This is by example used to pass a token from the response to
+the next request:
+
+.. code:: python
+
+   def response(resp):
+       ...
+       results.append({
+          'engine_data': token,
+          'key': 'next_page_token',
+       })
+       ...
+       return results
+
+   def request(query, params):
+       page_token = params['engine_data'].get('next_page_token')
+
+
+.. _main result list:
+
+Main Result List
+================
+
+The **media types** of the **main result type** are the template files in
+the :origin:`result_templates <searx/templates/simple/result_templates>`.
+
+.. _template default:
+
+``default.html``
+----------------
+
+Displays result fields from:
+
+- :ref:`macro result_header` and
+- :ref:`macro result_sub_header`
+
+Additional fields used in the :origin:`default.html
+<searx/templates/simple/result_templates/default.html>`:
+
+content :  :py:class:`str`
+  General text of the result item.
+
+iframe_src : :py:class:`str`
+  URL of an embedded ``<iframe>`` / the frame is collapsible.
+
+audio_src : uri,
+  URL of an embedded ``<audio controls>``.
+
+
+.. _template images:
+
+``images.html``
+---------------
+
+The images are displayed as small thumbnails in the main results list.
+
+title :  :py:class:`str`
+  Title of the image.
+
+thumbnail_src : :py:class:`str`
+  URL of a preview of the image.
+
+resolution :py:class:`str`
+  The resolution of the image (e.g. ``1920 x 1080`` pixel)
+
+
+Image labels
+~~~~~~~~~~~~
+
+Clicking on the preview opens a gallery view in which all further metadata for
+the image is displayed.  Addition fields used in the :origin:`images.html
+<searx/templates/simple/result_templates/images.html>`:
+
+img_src : :py:class:`str`
+  URL of the full size image.
+
+content:  :py:class:`str`
+  Description of the image.
+
+author:  :py:class:`str`
+  Name of the author of the image.
+
+img_format : :py:class:`str`
+  The format of the image (e.g. ``png``).
+
+source : :py:class:`str`
+  Source of the image.
+
+filesize: :py:class:`str`
+  Size of bytes in :py:obj:`human readable <searx.humanize_bytes>` notation
+  (e.g. ``MB`` for 1024 \* 1024 Bytes filesize).
+
+url : :py:class:`str`
+  URL of the page from where the images comes from (source).
+
+
+.. _template videos:
+
+``videos.html``
+---------------
+
+Displays result fields from:
+
+- :ref:`macro result_header` and
+- :ref:`macro result_sub_header`
+
+Additional fields used in the :origin:`videos.html
+<searx/templates/simple/result_templates/videos.html>`:
+
+iframe_src : :py:class:`str`
+  URL of an embedded ``<iframe>`` / the frame is collapsible.
+
+  The videos are displayed as small thumbnails in the main results list, there
+  is an additional button to collaps/open the embeded video.
+
+content :  :py:class:`str`
+  Description of the code fragment.
+
+
+.. _template torrent:
+
+``torrent.html``
+----------------
+
+.. _magnet link: https://en.wikipedia.org/wiki/Magnet_URI_scheme
+.. _torrent file: https://en.wikipedia.org/wiki/Torrent_file
+
+Displays result fields from:
+
+- :ref:`macro result_header` and
+- :ref:`macro result_sub_header`
+
+Additional fields used in the :origin:`torrent.html
+<searx/templates/simple/result_templates/torrent.html>`:
+
+magnetlink:
+  URL of the `magnet link`_.
+
+torrentfile
+  URL of the `torrent file`_.
+
+seed : ``int``
+  Number of seeders.
+
+leech : ``int``
+  Number of leecher
+
+filesize : ``int``
+  Size in Bytes (rendered to human readable unit of measurement).
+
+files : ``int``
+  Number of files.
+
+
+.. _template map:
+
+``map.html``
+------------
+
+.. _GeoJSON: https://en.wikipedia.org/wiki/GeoJSON
+.. _Leaflet: https://github.com/Leaflet/Leaflet
+.. _bbox: https://wiki.openstreetmap.org/wiki/Bounding_Box
+.. _HTMLElement.dataset: https://developer.mozilla.org/en-US/docs/Web/API/HTMLElement/dataset
+.. _Nominatim: https://nominatim.org/release-docs/latest/
+.. _Lookup: https://nominatim.org/release-docs/latest/api/Lookup/
+.. _place_id is not a persistent id:
+    https://nominatim.org/release-docs/latest/api/Output/#place_id-is-not-a-persistent-id
+.. _perma_id: https://wiki.openstreetmap.org/wiki/Permanent_ID
+.. _country code: https://wiki.openstreetmap.org/wiki/Country_code
+
+Displays result fields from:
+
+- :ref:`macro result_header` and
+- :ref:`macro result_sub_header`
+
+Additional fields used in the :origin:`map.html
+<searx/templates/simple/result_templates/map.html>`:
+
+content :  :py:class:`str`
+  Description of the item.
+
+address_label : :py:class:`str`
+  Label of the address / default ``_('address')``.
+
+geojson : GeoJSON_
+  Geometries mapped to HTMLElement.dataset_ (``data-map-geojson``) and used by
+  Leaflet_.
+
+boundingbox : ``[ min-lon, min-lat, max-lon, max-lat]``
+  A bbox_ area defined by min longitude , min latitude , max longitude and max
+  latitude.  The bounding box is mapped to HTMLElement.dataset_
+  (``data-map-boundingbox``) and is used by Leaflet_.
+
+longitude, latitude : :py:class:`str`
+  Geographical coordinates, mapped to HTMLElement.dataset_ (``data-map-lon``,
+  ``data-map-lat``) and is used by Leaflet_.
+
+address : ``{...}``
+  A dicticonary with the address data:
+
+  .. code:: python
+
+     address = {
+         'name'          : str,  # name of object
+         'road'          : str,  # street name of object
+         'house_number'  : str,  # house number of object
+         'postcode'      : str,  # postcode of object
+         'country'       : str,  # country of object
+         'country_code'  : str,
+         'locality'      : str,
+     }
+
+  country_code : :py:class:`str`
+    `Country code`_ of the object.
+
+  locality : :py:class:`str`
+    The name of the city, town, township, village, borough, etc. in which this
+    object is located.
+
+links : ``[link1, link2, ...]``
+  A list of links with labels:
+
+  .. code:: python
+
+     links.append({
+         'label'       : str,
+         'url'         : str,
+         'url_label'   : str,  # set by some engines but unused (oscar)
+     })
+
+data : ``[data1, data2, ...]``
+  A list of additional data, shown in two columns and containing a label and
+  value.
+
+  .. code:: python
+
+     data.append({
+        'label'   : str,
+        'value'   : str,
+        'key'     : str,  # set by some engines but unused
+     })
+
+type : :py:class:`str`  # set by some engines but unused (oscar)
+  Tag label from :ref:`OSM_KEYS_TAGS['tags'] <update_osm_keys_tags.py>`.
+
+type_icon : :py:class:`str`  # set by some engines but unused (oscar)
+  Type's icon.
+
+osm : ``{...}``
+  OSM-type and OSM-ID, can be used to Lookup_ OSM data (Nominatim_). There is
+  also a discussion about "`place_id is not a persistent id`_" and the
+  perma_id_.
+
+  .. code:: python
+
+     osm = {
+         'type': str,
+         'id':   str,
+     }
+
+  type : :py:class:`str`
+    Type of osm-object (if OSM-Result).
+
+  id :
+    ID of osm-object (if OSM-Result).
+
+  .. hint::
+
+     The ``osm`` property is set by engine ``openstreetmap.py``, but it is not
+     used in the ``map.html`` template yet.
+
+
+
+.. _template paper:
+
+``paper.html``
+--------------
+
+.. _BibTeX format: https://www.bibtex.com/g/bibtex-format/
+.. _BibTeX field types: https://en.wikipedia.org/wiki/BibTeX#Field_types
+
+Displays result fields from:
+
+- :ref:`macro result_header`
+
+Additional fields used in the :origin:`paper.html
+<searx/templates/simple/result_templates/paper.html>`:
+
+content :  :py:class:`str`
+  An abstract or excerpt from the document.
+
+comments : :py:class:`str`
+  Free text display in italic below the content.
+
+tags : :py:class:`List <list>`\ [\ :py:class:`str`\ ]
+  Free tag list.
+
+type : :py:class:`str`
+  Short description of medium type, e.g. *book*, *pdf* or *html* ...
+
+authors : :py:class:`List <list>`\ [\ :py:class:`str`\ ]
+  List of authors of the work (authors with a "s" suffix, the "author" is in the
+  :ref:`macro result_sub_header`).
+
+editor : :py:class:`str`
+  Editor of the book/paper.
+
+publisher : :py:class:`str`
+  Name of the publisher.
+
+journal : :py:class:`str`
+  Name of the journal or magazine the article was published in.
+
+volume : :py:class:`str`
+  Volume number.
+
+pages : :py:class:`str`
+  Page range where the article is.
+
+number : :py:class:`str`
+  Number of the report or the issue number for a journal article.
+
+doi : :py:class:`str`
+  DOI number (like ``10.1038/d41586-018-07848-2``).
+
+issn : :py:class:`List <list>`\ [\ :py:class:`str`\ ]
+  ISSN number like ``1476-4687``
+
+isbn : :py:class:`List <list>`\ [\ :py:class:`str`\ ]
+  ISBN number like ``9780201896831``
+
+pdf_url : :py:class:`str`
+  URL to the full article, the PDF version
+
+html_url : :py:class:`str`
+  URL to full article, HTML version
+
+
+.. _template packages:
+
+``packages``
+------------
+
+Displays result fields from:
+
+- :ref:`macro result_header`
+
+Additional fields used in the :origin:`packages.html
+<searx/templates/simple/result_templates/packages.html>`:
+
+package_name : :py:class:`str`
+  The name of the package.
+
+version : :py:class:`str`
+  The current version of the package.
+
+maintainer : :py:class:`str`
+  The maintainer or author of the project.
+
+publishedDate : :py:class:`datetime <datetime.datetime>`
+  Date of latest update or release.
+
+tags : :py:class:`List <list>`\ [\ :py:class:`str`\ ]
+  Free tag list.
+
+popularity : :py:class:`str`
+  The popularity of the package, e.g. rating or download count.
+
+license_name : :py:class:`str`
+  The name of the license.
+
+license_url : :py:class:`str`
+  The web location of a license copy.
+
+homepage : :py:class:`str`
+  The url of the project's homepage.
+
+source_code_url: :py:class:`str`
+  The location of the project's source code.
+
+links : :py:class:`dict`
+  Additional links in the form of ``{'link_name': 'http://example.com'}``
+
+
+.. _template code:
+
+``code.html``
+-------------
+
+Displays result fields from:
+
+- :ref:`macro result_header` and
+- :ref:`macro result_sub_header`
+
+Additional fields used in the :origin:`code.html
+<searx/templates/simple/result_templates/code.html>`:
+
+content :  :py:class:`str`
+  Description of the code fragment.
+
+codelines : ``[line1, line2, ...]``
+  Lines of the code fragment.
+
+code_language : :py:class:`str`
+  Name of the code language, the value is passed to
+  :py:obj:`pygments.lexers.get_lexer_by_name`.
+
+repository : :py:class:`str`
+  URL of the repository of the code fragment.
+
+
+.. _template files:
+
+``files.html``
+--------------
+
+Displays result fields from:
+
+- :ref:`macro result_header` and
+- :ref:`macro result_sub_header`
+
+Additional fields used in the :origin:`code.html
+<searx/templates/simple/result_templates/files.html>`:
+
+filename, size, time: :py:class:`str`
+  Filename, Filesize and Date of the file.
+
+mtype : ``audio`` | ``video`` | :py:class:`str`
+  Mimetype type of the file.
+
+subtype : :py:class:`str`
+  Mimetype / subtype of the file.
+
+abstract : :py:class:`str`
+  Abstract of the file.
+
+author : :py:class:`str`
+  Name of the author of the file
+
+embedded : :py:class:`str`
+  URL of an embedded media type (``audio`` or ``video``) / is collapsible.
+
+
+.. _template products:
+
+``products.html``
+-----------------
+
+Displays result fields from:
+
+- :ref:`macro result_header` and
+- :ref:`macro result_sub_header`
+
+Additional fields used in the :origin:`products.html
+<searx/templates/simple/result_templates/products.html>`:
+
+content :  :py:class:`str`
+  Description of the product.
+
+price : :py:class:`str`
+  The price must include the currency.
+
+shipping : :py:class:`str`
+  Shipping details.
+
+source_country : :py:class:`str`
+  Place from which the shipment is made.
+
+
+.. _template answer results:
+
+Answer results
+==============
+
+See :ref:`result_types.answer`
+
+Suggestion results
+==================
+
+See :ref:`result_types.suggestion`
+
+Correction results
+==================
+
+See :ref:`result_types.corrections`
+
+Infobox results
+===============
+
+See :ref:`result_types.infobox`
-- 
cgit v1.2.3