Merge pull request #2269 from return42/locale-revision

Revision of the locale- and language- handling in SearXNG

17 files changed, 304 insertions, 20 deletions

diff --git a/docs/admin/engines/configured_engines.rst b/docs/admin/engines/configured_engines.rst
index c7b6a1f52..fa1e5a4b0 100644
--- a/docs/admin/engines/configured_engines.rst
+++ b/docs/admin/engines/configured_engines.rst
@@ -42,7 +42,7 @@ Explanation of the :ref:`general engine configuration` shown in the table
         - Timeout
         - Weight
         - Paging
-        - Language
+        - Language, Region
         - Safe search
         - Time range
 
diff --git a/docs/admin/engines/settings.rst b/docs/admin/engines/settings.rst
index 099e449e0..0d9e14e57 100644
--- a/docs/admin/engines/settings.rst
+++ b/docs/admin/engines/settings.rst
@@ -569,10 +569,13 @@ engine is shown.  Most of the options have a default value or even are optional.
   To disable by default the engine, but not deleting it.  It will allow the user
   to manually activate it in the settings.
 
+``inactive``: optional
+  Remove the engine from the settings (*disabled & removed*).
+
 ``language`` : optional
   If you want to use another language for a specific engine, you can define it
-  by using the full ISO code of language and country, like ``fr_FR``, ``en_US``,
-  ``de_DE``.
+  by using the ISO code of language (and region), like ``fr``, ``en-US``,
+  ``de-DE``.
 
 ``tokens`` : optional
   A list of secret tokens to make this engine *private*, more details see
diff --git a/docs/conf.py b/docs/conf.py
index 8e0c3ab1b..1d71b7f8a 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -127,6 +127,10 @@ extensions = [
     'notfound.extension',  # https://github.com/readthedocs/sphinx-notfound-page
 ]
 
+autodoc_default_options = {
+    'member-order': 'groupwise',
+}
+
 myst_enable_extensions = [
   "replacements", "smartquotes"
 ]
@@ -135,6 +139,7 @@ suppress_warnings = ['myst.domains']
 
 intersphinx_mapping = {
     "python": ("https://docs.python.org/3/", None),
+    "babel" : ("https://babel.readthedocs.io/en/latest/", None),
     "flask": ("https://flask.palletsprojects.com/", None),
     "flask_babel": ("https://python-babel.github.io/flask-babel/", None),
     # "werkzeug": ("https://werkzeug.palletsprojects.com/", None),
diff --git a/docs/dev/engine_overview.rst b/docs/dev/engine_overview.rst
index 95ed267e2..930fd0813 100644
--- a/docs/dev/engine_overview.rst
+++ b/docs/dev/engine_overview.rst
@@ -54,6 +54,7 @@ Engine File
                                        - ``offline`` :ref:`[ref] <offline engines>`
                                        - ``online_dictionary``
                                        - ``online_currency``
+                                       - ``online_url_search``
    ======================= =========== ========================================================
 
 .. _engine settings:
@@ -131,8 +132,10 @@ Passed Arguments (request)
 These arguments can be used to construct the search query.  Furthermore,
 parameters with default value can be redefined for special purposes.
 
+.. _engine request online:
 
-.. table:: If the ``engine_type`` is ``online``
+.. table:: If the ``engine_type`` is :py:obj:`online
+           <searx.search.processors.online.OnlineProcessor.get_params>`
    :width: 100%
 
    ====================== ============== ========================================================================
@@ -149,12 +152,16 @@ parameters with default value can be redefined for special purposes.
    safesearch             int            ``0``, between ``0`` and ``2`` (normal, moderate, strict)
    time_range             Optional[str]  ``None``, can be ``day``, ``week``, ``month``, ``year``
    pageno                 int            current pagenumber
-   language               str            specific language code like ``'en_US'``, or ``'all'`` if unspecified
+   searxng_locale         str            SearXNG's locale selected by user.  Specific language code like
+                                         ``'en'``, ``'en-US'``, or ``'all'`` if unspecified.
    ====================== ============== ========================================================================
 
 
-.. table:: If the ``engine_type`` is ``online_dictionary``, in addition to the
-           ``online`` arguments:
+.. _engine request online_dictionary:
+
+.. table:: If the ``engine_type`` is :py:obj:`online_dictionary
+           <searx.search.processors.online_dictionary.OnlineDictionaryProcessor.get_params>`,
+           in addition to the :ref:`online <engine request online>` arguments:
    :width: 100%
 
    ====================== ============== ========================================================================
@@ -165,8 +172,11 @@ parameters with default value can be redefined for special purposes.
    query                  str            the text query without the languages
    ====================== ============== ========================================================================
 
-.. table:: If the ``engine_type`` is ``online_currency```, in addition to the
-           ``online`` arguments:
+.. _engine request online_currency:
+
+.. table:: If the ``engine_type`` is :py:obj:`online_currency
+           <searx.search.processors.online_currency.OnlineCurrencyProcessor.get_params>`,
+           in addition to the :ref:`online <engine request online>` arguments:
    :width: 100%
 
    ====================== ============== ========================================================================
@@ -179,6 +189,26 @@ parameters with default value can be redefined for special purposes.
    to_name                str            currency name
    ====================== ============== ========================================================================
 
+.. _engine request online_url_search:
+
+.. table:: If the ``engine_type`` is :py:obj:`online_url_search
+           <searx.search.processors.online_url_search.OnlineUrlSearchProcessor.get_params>`,
+           in addition to the :ref:`online <engine request online>` arguments:
+   :width: 100%
+
+   ====================== ============== ========================================================================
+   argument               type           default-value, information
+   ====================== ============== ========================================================================
+   search_url             dict           URLs from the search query:
+
+                                         .. code:: python
+
+                                            {
+                                              'http': str,
+                                              'ftp': str,
+                                              'data:image': str
+                                            }
+   ====================== ============== ========================================================================
 
 Specify Request
 ---------------
diff --git a/docs/dev/searxng_extra/update.rst b/docs/dev/searxng_extra/update.rst
index d05c81409..a125303e0 100644
--- a/docs/dev/searxng_extra/update.rst
+++ b/docs/dev/searxng_extra/update.rst
@@ -52,12 +52,12 @@ Scripts to update static data in :origin:`searx/data/`
   :members:
 
 
-``update_languages.py``
-=======================
+``update_engine_traits.py``
+===========================
 
-:origin:`[source] <searxng_extra/update/update_languages.py>`
+:origin:`[source] <searxng_extra/update/update_engine_traits.py>`
 
-.. automodule:: searxng_extra.update.update_languages
+.. automodule:: searxng_extra.update.update_engine_traits
   :members:
 
 
diff --git a/docs/src/searx.engine.archlinux.rst b/docs/src/searx.engine.archlinux.rst
new file mode 100644
index 000000000..be48b1859
--- /dev/null
+++ b/docs/src/searx.engine.archlinux.rst
@@ -0,0 +1,9 @@
+.. _archlinux engine:
+
+==========
+Arch Linux
+==========
+
+.. automodule:: searx.engines.archlinux
+  :members:
+
diff --git a/docs/src/searx.engine.dailymotion.rst b/docs/src/searx.engine.dailymotion.rst
new file mode 100644
index 000000000..84348e2d0
--- /dev/null
+++ b/docs/src/searx.engine.dailymotion.rst
@@ -0,0 +1,8 @@
+.. _dailymotion engine:
+
+===========
+Dailymotion
+===========
+
+.. automodule:: searx.engines.dailymotion
+  :members:
diff --git a/docs/src/searx.engine.duckduckgo.rst b/docs/src/searx.engine.duckduckgo.rst
new file mode 100644
index 000000000..1646d4984
--- /dev/null
+++ b/docs/src/searx.engine.duckduckgo.rst
@@ -0,0 +1,22 @@
+.. _duckduckgo engines:
+
+=================
+DukcDukGo engines
+=================
+
+.. contents:: Contents
+   :depth: 2
+   :local:
+   :backlinks: entry
+
+.. automodule:: searx.engines.duckduckgo
+   :members:
+
+.. automodule:: searx.engines.duckduckgo_images
+   :members:
+
+.. automodule:: searx.engines.duckduckgo_definitions
+   :members:
+
+.. automodule:: searx.engines.duckduckgo_weather
+   :members:
diff --git a/docs/src/searx.enginelib.rst b/docs/src/searx.enginelib.rst
new file mode 100644
index 000000000..651a04e68
--- /dev/null
+++ b/docs/src/searx.enginelib.rst
@@ -0,0 +1,17 @@
+.. _searx.enginelib:
+
+============
+Engine model
+============
+
+.. automodule:: searx.enginelib
+  :members:
+
+.. _searx.enginelib.traits:
+
+=============
+Engine traits
+=============
+
+.. automodule:: searx.enginelib.traits
+  :members:
diff --git a/docs/src/searx.engines.bing.rst b/docs/src/searx.engines.bing.rst
new file mode 100644
index 000000000..6b7bba8f2
--- /dev/null
+++ b/docs/src/searx.engines.bing.rst
@@ -0,0 +1,43 @@
+.. _bing engines:
+
+============
+Bing Engines
+============
+
+.. contents:: Contents
+   :depth: 2
+   :local:
+   :backlinks: entry
+
+
+.. _bing web engine:
+
+Bing WEB
+========
+
+.. automodule:: searx.engines.bing
+  :members:
+
+.. _bing images engine:
+
+Bing Images
+===========
+
+.. automodule:: searx.engines.bing_images
+  :members:
+
+.. _bing videos engine:
+
+Bing Videos
+===========
+
+.. automodule:: searx.engines.bing_videos
+  :members:
+
+.. _bing news engine:
+
+Bing News
+=========
+
+.. automodule:: searx.engines.bing_news
+  :members:
diff --git a/docs/src/searx.engines.google.rst b/docs/src/searx.engines.google.rst
index 2d10b5eea..9c15325f8 100644
--- a/docs/src/searx.engines.google.rst
+++ b/docs/src/searx.engines.google.rst
@@ -12,15 +12,21 @@ Google Engines
 
 .. _google API:
 
-google API
+Google API
 ==========
 
 .. _Query Parameter Definitions:
    https://developers.google.com/custom-search/docs/xml_results#WebSearch_Query_Parameter_Definitions
 
+SearXNG's implementation of the Google API is mainly done in
+:py:obj:`get_google_info <searx.engines.google.get_google_info>`.
+
 For detailed description of the *REST-full* API see: `Query Parameter
-Definitions`_.  Not all parameters can be appied and some engines are *special*
-(e.g. :ref:`google news engine`).
+Definitions`_.  The linked API documentation can sometimes be helpful during
+reverse engineering.  However, we cannot use it in the freely accessible WEB
+services; not all parameters can be applied and some engines are more *special*
+than other (e.g. :ref:`google news engine`).
+
 
 .. _google web engine:
 
@@ -30,6 +36,13 @@ Google WEB
 .. automodule:: searx.engines.google
   :members:
 
+.. _google autocomplete:
+
+Google Autocomplete
+====================
+
+.. autofunction:: searx.autocomplete.google_complete
+
 .. _google images engine:
 
 Google Images
@@ -53,3 +66,11 @@ Google News
 
 .. automodule:: searx.engines.google_news
   :members:
+
+.. _google scholar engine:
+
+Google Scholar
+==============
+
+.. automodule:: searx.engines.google_scholar
+  :members:
diff --git a/docs/src/searx.engines.peertube.rst b/docs/src/searx.engines.peertube.rst
new file mode 100644
index 000000000..8e1576ea0
--- /dev/null
+++ b/docs/src/searx.engines.peertube.rst
@@ -0,0 +1,27 @@
+.. _peertube engines:
+
+================
+Peertube Engines
+================
+
+.. contents:: Contents
+   :depth: 2
+   :local:
+   :backlinks: entry
+
+
+.. _peertube video engine:
+
+Peertube Video
+==============
+
+.. automodule:: searx.engines.peertube
+  :members:
+
+.. _sepiasearch engine:
+
+SepiaSearch
+===========
+
+.. automodule:: searx.engines.sepiasearch
+  :members:
diff --git a/docs/src/searx.engines.rst b/docs/src/searx.engines.rst
index 687fdb0b2..4ce96b27e 100644
--- a/docs/src/searx.engines.rst
+++ b/docs/src/searx.engines.rst
@@ -1,8 +1,8 @@
-.. _load_engines:
+.. _searx.engines:
 
-============
-Load Engines
-============
+=================
+SearXNG's engines
+=================
 
 .. automodule:: searx.engines
   :members:
diff --git a/docs/src/searx.engines.startpage.rst b/docs/src/searx.engines.startpage.rst
new file mode 100644
index 000000000..c885d8f1b
--- /dev/null
+++ b/docs/src/searx.engines.startpage.rst
@@ -0,0 +1,13 @@
+.. _startpage engines:
+
+=================
+Startpage engines
+=================
+
+.. contents:: Contents
+   :depth: 2
+   :local:
+   :backlinks: entry
+
+.. automodule:: searx.engines.startpage
+   :members:
diff --git a/docs/src/searx.engines.wikipedia.rst b/docs/src/searx.engines.wikipedia.rst
new file mode 100644
index 000000000..e644cd645
--- /dev/null
+++ b/docs/src/searx.engines.wikipedia.rst
@@ -0,0 +1,27 @@
+.. _wikimedia engines:
+
+=========
+Wikimedia
+=========
+
+.. contents:: Contents
+   :depth: 2
+   :local:
+   :backlinks: entry
+
+
+.. _wikipedia engine:
+
+Wikipedia
+=========
+
+.. automodule:: searx.engines.wikipedia
+  :members:
+
+.. _wikidata engine:
+
+Wikidata
+=========
+
+.. automodule:: searx.engines.wikidata
+  :members:
diff --git a/docs/src/searx.locales.rst b/docs/src/searx.locales.rst
index 579247aff..2f13bfca1 100644
--- a/docs/src/searx.locales.rst
+++ b/docs/src/searx.locales.rst
@@ -4,5 +4,17 @@
 Locales
 =======
 
+.. contents:: Contents
+   :depth: 2
+   :local:
+   :backlinks: entry
+
 .. automodule:: searx.locales
   :members:
+
+
+SearXNG's locale codes
+======================
+
+.. automodule:: searx.sxng_locales
+  :members:
diff --git a/docs/src/searx.search.processors.rst b/docs/src/searx.search.processors.rst
new file mode 100644
index 000000000..390680657
--- /dev/null
+++ b/docs/src/searx.search.processors.rst
@@ -0,0 +1,47 @@
+.. _searx.search.processors:
+
+=================
+Search processors
+=================
+
+.. contents:: Contents
+   :depth: 2
+   :local:
+   :backlinks: entry
+
+
+Abstract processor class
+========================
+
+.. automodule:: searx.search.processors.abstract
+  :members:
+
+Offline processor
+=================
+
+.. automodule:: searx.search.processors.offline
+  :members:
+
+Online processor
+================
+
+.. automodule:: searx.search.processors.online
+  :members:
+
+Online currency processor
+=========================
+
+.. automodule:: searx.search.processors.online_currency
+  :members:
+
+Online Dictionary processor
+===========================
+
+.. automodule:: searx.search.processors.online_dictionary
+  :members:
+
+Online URL search processor
+===========================
+
+.. automodule:: searx.search.processors.online_url_search
+  :members: