diff options
Diffstat (limited to 'docs/dev')
| -rw-r--r-- | docs/dev/engine_overview.rst | 509 | ||||
| -rw-r--r-- | docs/dev/index.rst | 2 | ||||
| -rw-r--r-- | docs/dev/lxcdev.rst | 416 | ||||
| -rw-r--r-- | docs/dev/makefile.rst | 4 | ||||
| -rw-r--r-- | docs/dev/offline_engines.rst | 78 | ||||
| -rw-r--r-- | docs/dev/reST.rst | 4 | ||||
| -rw-r--r-- | docs/dev/search_api.rst | 2 |
7 files changed, 759 insertions, 256 deletions
diff --git a/docs/dev/engine_overview.rst b/docs/dev/engine_overview.rst index 2a661ec40..d79f662b8 100644 --- a/docs/dev/engine_overview.rst +++ b/docs/dev/engine_overview.rst @@ -1,310 +1,317 @@ - .. _engines-dev: =============== -Engine overview +Engine Overview =============== .. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine -searx is a metasearch-engine_, so it uses different search engines to provide -better results. +.. sidebar:: Further reading .. -Because there is no general search API which could be used for every search -engine, an adapter has to be built between searx and the external search -engines. Adapters are stored under the folder :origin:`searx/engines`. + - :ref:`configured engines` + - :ref:`settings engine` .. contents:: :depth: 3 :backlinks: entry +searx is a metasearch-engine_, so it uses different search engines to provide +better results. + +Because there is no general search API which could be used for every search +engine, an adapter has to be built between searx and the external search +engines. Adapters are stored under the folder :origin:`searx/engines`. .. _general engine configuration: -general engine configuration +General Engine Configuration ============================ It is required to tell searx the type of results the engine provides. The -arguments can be set in the engine file or in the settings file -(normally ``settings.yml``). The arguments in the settings file override -the ones in the engine file. +arguments can be set in the engine file or in the settings file (normally +``settings.yml``). The arguments in the settings file override the ones in the +engine file. -It does not matter if an option is stored in the engine file or in the -settings. However, the standard way is the following: +It does not matter if an option is stored in the engine file or in the settings. +However, the standard way is the following: .. _engine file: -engine file +Engine File ----------- -======================= =========== ======================================================== -argument type information -======================= =========== ======================================================== -categories list pages, in which the engine is working -paging boolean support multible pages -time_range_support boolean support search time range -engine_type str ``online`` by default, other possibles values are - ``offline``, ``online_dictionnary``, ``online_currency`` -======================= =========== ======================================================== +.. table:: Common options in the engine module + :width: 100% + + ======================= =========== ======================================================== + argument type information + ======================= =========== ======================================================== + categories list pages, in which the engine is working + paging boolean support multible pages + time_range_support boolean support search time range + engine_type str - ``online`` :ref:`[ref] <demo online engine>` by + default, other possibles values are: + - ``offline`` :ref:`[ref] <offline engines>` + - ``online_dictionary`` + - ``online_currency`` + ======================= =========== ======================================================== .. _engine settings: -settings.yml ------------- - -======================= =========== ============================================= -argument type information -======================= =========== ============================================= -name string name of search-engine -engine string name of searx-engine - (filename without ``.py``) -enable_http bool enable HTTP - (by default only HTTPS is enabled). -shortcut string shortcut of search-engine -timeout string specific timeout for search-engine -display_error_messages boolean display error messages on the web UI -proxies dict set proxies for a specific engine - (e.g. ``proxies : {http: socks5://proxy:port, - https: socks5://proxy:port}``) -======================= =========== ============================================= - - -overrides +Engine ``settings.yml`` +----------------------- + +For a more detailed description, see :ref:`settings engine` in the :ref:`settings.yml`. + +.. table:: Common options in the engine setup (``settings.yml``) + :width: 100% + + ======================= =========== =============================================== + argument type information + ======================= =========== =============================================== + name string name of search-engine + engine string name of searx-engine (filename without ``.py``) + enable_http bool enable HTTP (by default only HTTPS is enabled). + shortcut string shortcut of search-engine + timeout string specific timeout for search-engine + display_error_messages boolean display error messages on the web UI + proxies dict set proxies for a specific engine + (e.g. ``proxies : {http: socks5://proxy:port, + https: socks5://proxy:port}``) + ======================= =========== =============================================== + +.. _engine overrides: + +Overrides --------- -A few of the options have default values in the engine, but are often -overwritten by the settings. If ``None`` is assigned to an option in the engine -file, it has to be redefined in the settings, otherwise searx will not start -with that engine. +.. sidebar:: engine's global names -The naming of overrides is arbitrary. But the recommended overrides are the -following: + Global names with a leading underline are *private to the engine* and will + not be overwritten. -======================= =========== =========================================== -argument type information -======================= =========== =========================================== -base_url string base-url, can be overwritten to use same - engine on other URL -number_of_results int maximum number of results per request -language string ISO code of language and country like en_US -api_key string api-key if required by engine -======================= =========== =========================================== +A few of the options have default values in the namespace of engine's python +modul, but are often overwritten by the settings. If ``None`` is assigned to an +option in the engine file, it has to be redefined in the settings, otherwise +searx will not start with that engine. -example code ------------- +Here is an very simple example of the global names in the namespace of engine's +module: .. code:: python # engine dependent config categories = ['general'] paging = True + _non_overwritten_global = 'foo' + +.. table:: The naming of overrides is arbitrary / recommended overrides are: + :width: 100% + + ======================= =========== =========================================== + argument type information + ======================= =========== =========================================== + base_url string base-url, can be overwritten to use same + engine on other URL + number_of_results int maximum number of results per request + language string ISO code of language and country like en_US + api_key string api-key if required by engine + ======================= =========== =========================================== .. _engine request: -making a request +Making a Request ================ To perform a search an URL have to be specified. In addition to specifying an URL, arguments can be passed to the query. -passed arguments ----------------- +.. _engine request arguments: + +Passed Arguments (request) +-------------------------- These arguments can be used to construct the search query. Furthermore, parameters with default value can be redefined for special purposes. -If the ``engine_type`` is ``online```: - -====================== ============== ======================================================================== -argument type default-value, information -====================== ============== ======================================================================== -url str ``''`` -method str ``'GET'`` -headers set ``{}`` -data set ``{}`` -cookies set ``{}`` -verify bool ``True`` -headers.User-Agent str a random User-Agent -category str current category, like ``'general'`` -safesearch int ``0``, between ``0`` and ``2`` (normal, moderate, strict) -time_range Optional[str] ``None``, can be ``day``, ``week``, ``month``, ``year`` -pageno int current pagenumber -language str specific language code like ``'en_US'``, or ``'all'`` if unspecified -====================== ============== ======================================================================== - - -If the ``engine_type`` is ``online_dictionnary```, in addition to the ``online`` arguments: - -====================== ============ ======================================================================== -argument type default-value, information -====================== ============ ======================================================================== -from_lang str specific language code like ``'en_US'`` -to_lang str specific language code like ``'en_US'`` -query str the text query without the languages -====================== ============ ======================================================================== - -If the ``engine_type`` is ``online_currency```, in addition to the ``online`` arguments: - -====================== ============ ======================================================================== -argument type default-value, information -====================== ============ ======================================================================== -amount float the amount to convert -from str ISO 4217 code -to str ISO 4217 code -from_name str currency name -to_name str currency name -====================== ============ ======================================================================== - - -parsed arguments ----------------- - -The function ``def request(query, params):`` always returns the ``params`` -variable. Inside searx, the following paramters can be used to specify a search -request: - -=================== =========== ========================================================================== -argument type information -=================== =========== ========================================================================== -url str requested url -method str HTTP request method -headers set HTTP header information -data set HTTP data information -cookies set HTTP cookies -verify bool Performing SSL-Validity check -allow_redirects bool Follow redirects -max_redirects int maximum redirects, hard limit -soft_max_redirects int maximum redirects, soft limit. Record an error but don't stop the engine -raise_for_httperror bool True by default: raise an exception if the HTTP code of response is >= 300 -=================== =========== ========================================================================== - - -example code ------------- - -.. code:: python - - # search-url - base_url = 'https://example.com/' - search_string = 'search?{query}&page={page}' - - # do search-request - def request(query, params): - search_path = search_string.format( - query=urlencode({'q': query}), - page=params['pageno']) - - params['url'] = base_url + search_path - return params +.. table:: If the ``engine_type`` is ``online`` + :width: 100% + + ====================== ============== ======================================================================== + argument type default-value, information + ====================== ============== ======================================================================== + url str ``''`` + method str ``'GET'`` + headers set ``{}`` + data set ``{}`` + cookies set ``{}`` + verify bool ``True`` + headers.User-Agent str a random User-Agent + category str current category, like ``'general'`` + safesearch int ``0``, between ``0`` and ``2`` (normal, moderate, strict) + time_range Optional[str] ``None``, can be ``day``, ``week``, ``month``, ``year`` + pageno int current pagenumber + language str specific language code like ``'en_US'``, or ``'all'`` if unspecified + ====================== ============== ======================================================================== + + +.. table:: If the ``engine_type`` is ``online_dictionary``, in addition to the + ``online`` arguments: + :width: 100% + + ====================== ============== ======================================================================== + argument type default-value, information + ====================== ============== ======================================================================== + from_lang str specific language code like ``'en_US'`` + to_lang str specific language code like ``'en_US'`` + query str the text query without the languages + ====================== ============== ======================================================================== + +.. table:: If the ``engine_type`` is ``online_currency```, in addition to the + ``online`` arguments: + :width: 100% + + ====================== ============== ======================================================================== + argument type default-value, information + ====================== ============== ======================================================================== + amount float the amount to convert + from str ISO 4217 code + to str ISO 4217 code + from_name str currency name + to_name str currency name + ====================== ============== ======================================================================== + + +Specify Request +--------------- + +The function :py:func:`def request(query, params): +<searx.engines.demo_online.request>` always returns the ``params`` variable, the +following parameters can be used to specify a search request: + +.. table:: + :width: 100% + + =================== =========== ========================================================================== + argument type information + =================== =========== ========================================================================== + url str requested url + method str HTTP request method + headers set HTTP header information + data set HTTP data information + cookies set HTTP cookies + verify bool Performing SSL-Validity check + allow_redirects bool Follow redirects + max_redirects int maximum redirects, hard limit + soft_max_redirects int maximum redirects, soft limit. Record an error but don't stop the engine + raise_for_httperror bool True by default: raise an exception if the HTTP code of response is >= 300 + =================== =========== ========================================================================== .. _engine results: - -returned results -================ - -Searx is able to return results of different media-types. Currently the -following media-types are supported: - -- default_ -- images_ -- videos_ -- torrent_ -- map_ - -To set another media-type as default, the parameter ``template`` must be set to -the desired type. - -default -------- - -========================= ===================================================== -result-parameter information -========================= ===================================================== -url string, url of the result -title string, title of the result -content string, general result-text -publishedDate :py:class:`datetime.datetime`, time of publish -========================= ===================================================== - -images ------- - -To use this template, the parameter: - -========================= ===================================================== -result-parameter information -========================= ===================================================== -template is set to ``images.html`` -url string, url to the result site -title string, title of the result *(partly implemented)* -content *(partly implemented)* -publishedDate :py:class:`datetime.datetime`, - time of publish *(partly implemented)* -img\_src string, url to the result image -thumbnail\_src string, url to a small-preview image -========================= ===================================================== - -videos ------- - -========================= ===================================================== -result-parameter information -========================= ===================================================== -template is set to ``videos.html`` -url string, url of the result -title string, title of the result -content *(not implemented yet)* -publishedDate :py:class:`datetime.datetime`, time of publish -thumbnail string, url to a small-preview image -========================= ===================================================== - -torrent -------- +.. _engine media types: + +Media Types +=========== + +Each result item of an engine can be of different media-types. Currently the +following media-types are supported. To set another media-type as ``default``, +the parameter ``template`` must be set to the desired type. + +.. table:: Parameter of the **default** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ========================= ===================================================== + url string, url of the result + title string, title of the result + content string, general result-text + publishedDate :py:class:`datetime.datetime`, time of publish + ========================= ===================================================== + + +.. table:: Parameter of the **images** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ------------------------- ----------------------------------------------------- + template is set to ``images.html`` + ========================= ===================================================== + url string, url to the result site + title string, title of the result *(partly implemented)* + content *(partly implemented)* + publishedDate :py:class:`datetime.datetime`, + time of publish *(partly implemented)* + img\_src string, url to the result image + thumbnail\_src string, url to a small-preview image + ========================= ===================================================== + + +.. table:: Parameter of the **videos** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ------------------------- ----------------------------------------------------- + template is set to ``videos.html`` + ========================= ===================================================== + url string, url of the result + title string, title of the result + content *(not implemented yet)* + publishedDate :py:class:`datetime.datetime`, time of publish + thumbnail string, url to a small-preview image + ========================= ===================================================== .. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme -========================= ===================================================== -result-parameter information -========================= ===================================================== -template is set to ``torrent.html`` -url string, url of the result -title string, title of the result -content string, general result-text -publishedDate :py:class:`datetime.datetime`, - time of publish *(not implemented yet)* -seed int, number of seeder -leech int, number of leecher -filesize int, size of file in bytes -files int, number of files -magnetlink string, magnetlink_ of the result -torrentfile string, torrentfile of the result -========================= ===================================================== - - -map ---- - -========================= ===================================================== -result-parameter information -========================= ===================================================== -url string, url of the result -title string, title of the result -content string, general result-text -publishedDate :py:class:`datetime.datetime`, time of publish -latitude latitude of result (in decimal format) -longitude longitude of result (in decimal format) -boundingbox boundingbox of result (array of 4. values - ``[lat-min, lat-max, lon-min, lon-max]``) -geojson geojson of result (https://geojson.org/) -osm.type type of osm-object (if OSM-Result) -osm.id id of osm-object (if OSM-Result) -address.name name of object -address.road street name of object -address.house_number house number of object -address.locality city, place of object -address.postcode postcode of object -address.country country of object -========================= ===================================================== +.. table:: Parameter of the **torrent** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ------------------------- ----------------------------------------------------- + template is set to ``torrent.html`` + ========================= ===================================================== + url string, url of the result + title string, title of the result + content string, general result-text + publishedDate :py:class:`datetime.datetime`, + time of publish *(not implemented yet)* + seed int, number of seeder + leech int, number of leecher + filesize int, size of file in bytes + files int, number of files + magnetlink string, magnetlink_ of the result + torrentfile string, torrentfile of the result + ========================= ===================================================== + +.. table:: Parameter of the **map** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ------------------------- ----------------------------------------------------- + template is set to ``map.html`` + ========================= ===================================================== + url string, url of the result + title string, title of the result + content string, general result-text + publishedDate :py:class:`datetime.datetime`, time of publish + latitude latitude of result (in decimal format) + longitude longitude of result (in decimal format) + boundingbox boundingbox of result (array of 4. values + ``[lat-min, lat-max, lon-min, lon-max]``) + geojson geojson of result (https://geojson.org/) + osm.type type of osm-object (if OSM-Result) + osm.id id of osm-object (if OSM-Result) + address.name name of object + address.road street name of object + address.house_number house number of object + address.locality city, place of object + address.postcode postcode of object + address.country country of object + ========================= ===================================================== diff --git a/docs/dev/index.rst b/docs/dev/index.rst index ba0a25a9c..93c914bbb 100644 --- a/docs/dev/index.rst +++ b/docs/dev/index.rst @@ -9,8 +9,10 @@ Developer documentation quickstart contribution_guide engine_overview + offline_engines search_api plugins translation + lxcdev makefile reST diff --git a/docs/dev/lxcdev.rst b/docs/dev/lxcdev.rst new file mode 100644 index 000000000..ef26e3734 --- /dev/null +++ b/docs/dev/lxcdev.rst @@ -0,0 +1,416 @@ +.. _lxcdev: + +============================== +Developing in Linux Containers +============================== + +.. _LXC: https://linuxcontainers.org/lxc/introduction/ + +In this article we will show, how you can make use of Linux Containers (LXC_) in +*distributed and heterogeneous development cycles* (TL;DR; jump to the +:ref:`lxcdev summary`). + +.. sidebar:: Audience + + This blog post is written for experienced admins and developers. Readers + should have a serious meaning about the terms: *distributed*, *merge* and + *linux container*. + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + + +Motivation +========== + +Usually in our development cycle, we edit the sources and run some test and/or +builds by using ``make`` :ref:`[ref] <makefile>` before we commit. This cycle +is simple and perfect but might fail in some aspects we should not overlook. + + **The environment in which we run all our development processes matters!** + +The :ref:`makefile` and the :ref:`make install` encapsulate a lot for us, but +they do not have access to all prerequisites. For example, there may have +dependencies on packages that are installed on the developer's desktop, but +usually are not preinstalled on a server or client system. Another example is; +settings have been made to the software on developer's desktop that would never +be set on a *production* system. + + **Linux Containers are isolate environments and not to mix up all the + prerequisites from various projects on developer's desktop is always a good + choice.** + +The scripts from :ref:`searx_utils` can divide in those to install and maintain +software: + +- :ref:`searx.sh` +- :ref:`filtron.sh` +- :ref:`morty.sh` + +and the script :ref:`lxc.sh`, with we can scale our installation, maintenance or +even development tasks over a stack of isolated containers / what we call the: + + **searxNG LXC suite** + + +Gentlemen, start your engines! +============================== + +.. _LXD: https://linuxcontainers.org/lxd/introduction/ +.. _archlinux: https://www.archlinux.org/ + +Before you can start with containers, you need to install and initiate LXD_ +once: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ snap install lxd + $ lxd init --auto + +And you need to clone from origin or if you have your own fork, clone from your +fork: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ cd ~/Downloads + $ git clone https://github.com/searxng/searxng.git searx + $ cd searx + +The :ref:`lxc-searx.env` consists of several images, see ``export +LXC_SUITE=(...`` near by :origin:`utils/lxc-searx.env#L19`. For this blog post +we exercise on a archlinux_ image. The container of this image is named +``searx-archlinux``. Lets build the container, but be sure that this container +does not already exists, so first lets remove possible old one: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh remove searx-archlinux + $ sudo -H ./utils/lxc.sh build searx-archlinux + +.. sidebar:: The ``searx-archlinux`` container + + is the base of all our exercises here. + +In this container we install all services :ref:`including searx, morty & filtron +<lxc.sh install suite>` in once: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh install suite searx-archlinux + +To proxy HTTP from filtron and morty in the container to the outside of the +container, install nginx into the container. Once for the bot blocker filtron: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/filtron.sh nginx install + ... + INFO: got 429 from http://10.174.184.156/searx + +and once for the content sanitizer (content proxy morty): + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/morty.sh nginx install + ... + INFO: got 200 from http://10.174.184.156/morty/ + +.. sidebar:: Fully functional searXNG suite + + From here on you have a fully functional searXNG suite running with bot + blocker (filtron) and WEB content sanitizer (content proxy morty), both are + needed for a *privacy protecting* search engine. + +On your system, the IP of your ``searx-archlinux`` container differs from +http://10.174.184.156/searx, just open the URL reported in your installation +protocol in your WEB browser from the desktop to test the instance from outside +of the container. + +In such a searXNG suite admins can maintain and access the debug log of the +different services quite easy. + +.. _working in containers: + +In containers, work as usual +============================ + +Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers +open the root-bash in the container using ``./utils/lxc.sh cmd +searx-archlinux``: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux bash + INFO: [searx-archlinux] bash + [root@searx-archlinux searx]# pwd + /share/searx + +The prompt ``[root@searx-archlinux ...]`` signals, that you are the root user in +the searx-container. To debug the running searXNG instance use: + +.. tabs:: + + .. group-tab:: root@searx-archlinux + + .. code:: sh + + $ ./utils/searx.sh inspect service + ... + use [CTRL-C] to stop monitoring the log + ... + +Back in the browser on your desktop open the service http://10.174.184.156/searx +and run your application tests while the debug log is shown in the terminal from +above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug +option"* in searXNG's settings file and restarts the searXNG uwsgi application. +To debug services from filtron and morty analogous use: + +.. tabs:: + + .. group-tab:: root@searx-archlinux + + .. code:: sh + + $ ./utils/filtron.sh inspect service + $ ./utils/morty.sh inspect service + +Another point we have to notice is that each service (:ref:`searx <searx.sh>`, +:ref:`filtron <filtron.sh>` and :ref:`morty <morty.sh>`) runs under dedicated +system user account with the same name (compare :ref:`create searx user`). To +get a shell from theses accounts, simply call one of the scripts: + +.. tabs:: + + .. group-tab:: root@searx-archlinux + + .. code:: sh + + $ ./utils/searx.sh shell + $ ./utils/filtron.sh shell + $ ./utils/morty.sh shell + +To get in touch, open a shell from the service user (searx@searx-archlinux): + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/searx.sh shell + // exit with [CTRL-D] + (searx-pyenv) [searx@searx-archlinux ~]$ ... + +The prompt ``[searx@searx-archlinux]`` signals that you are logged in as system +user ``searx`` in the ``searx-archlinux`` container and the python *virtualenv* +``(searx-pyenv)`` environment is activated. + +.. tabs:: + + .. group-tab:: searx@searx-archlinux + + .. code:: sh + + (searx-pyenv) [searx@searx-archlinux ~]$ pwd + /usr/local/searx + + + +Wrap production into developer suite +==================================== + +In this section we will see how to change the *"Fully functional searXNG suite"* +from a LXC container (which is quite ready for production) into a developer +suite. For this, we have to keep an eye on the :ref:`installation basic`: + +- searXNG setup in: ``/etc/searx/settings.yml`` +- searXNG user's home: ``/usr/local/searx`` +- virtualenv in: ``/usr/local/searx/searx-pyenv`` +- searXNG software in: ``/usr/local/searx/searx-src`` + +The searXNG software is a clone of the ``git_url`` (see :ref:`settings global`) +and the working tree is checked out from the ``git_branch``. With the use of +the :ref:`searx.sh` the searx service was installed as :ref:`uWSGI application +<searx uwsgi>`. To maintain this service, we can use ``systemctl`` (compare +:ref:`service architectures on distributions <uwsgi configuration>`). + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + systemctl stop uwsgi@searx + +With the command above, we stopped the searx uWSGI-App in the archlinux +container. + +The uWSGI-App for the archlinux dsitros is configured in +:origin:`utils/templates/etc/uwsgi/apps-archlinux/searx.ini`, from where at +least you should attend the settings of ``uid``, ``chdir``, ``env`` and +``http``:: + + env = SEARX_SETTINGS_PATH=/etc/searx/settings.yml + http = 127.0.0.1:8888 + + chdir = /usr/local/searx/searx-src/searx + virtualenv = /usr/local/searx/searx-pyenv + pythonpath = /usr/local/searx/searx-src + +If you have read the :ref:`"Good to know section" <lxc.sh>` you remember, that +each container shares the root folder of the repository and the command +``utils/lxc.sh cmd`` handles relative path names **transparent**. To wrap the +searXNG installation into a developer one, we simple have to create a smylink to +the **transparent** reposetory from the desktop. Now lets replace the +repository at ``searx-src`` in the container with the working tree from outside +of the container: + +.. tabs:: + + .. group-tab:: container becomes a developer suite + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ln -s /share/searx/ /usr/local/searx/searx-src + +Now we can develop as usual in the working tree of our desktop system. Every +time the software was changed, you have to restart the searx service (in the +conatiner): + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + systemctl restart uwsgi@searx + + +Remember: :ref:`working in containers` .. here are just some examples from my +daily usage: + +.. tabs:: + + .. group-tab:: desktop + + To *inspect* the searXNG instance (already described above): + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/searx.sh inspect service + + Run :ref:`makefile`, e.g. to test inside the container: + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + make test + + To install all prerequisites needed for a :ref:`buildhosts`: + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/searx.sh install buildhost + + To build the docs on a buildhost :ref:`buildhosts`: + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + make docs.html + +.. _lxcdev summary: + +Summary +======= + +We build up a fully functional searXNG suite in a archlinux container: + +.. code:: sh + + $ sudo -H ./utils/lxc.sh install suite searx-archlinux + +To access HTTP from the desktop we installed nginx for the services inside the +conatiner: + +.. tabs:: + + .. group-tab:: [root@searx-archlinux] + + .. code:: sh + + $ ./utils/filtron.sh nginx install + $ ./utils/morty.sh nginx install + +To wrap the suite into a developer one, we created a symbolic link to the +repository which is shared **transparent** from the desktop's file system into +the container : + +.. tabs:: + + .. group-tab:: [root@searx-archlinux] + + .. code:: sh + + $ mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old + $ ln -s /share/searx/ /usr/local/searx/searx-src + $ systemctl restart uwsgi@searx + +To get information about the searxNG suite in the archlinux container we can +use: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh show suite searx-archlinux + ... + [searx-archlinux] INFO: (eth0) filtron: http://10.174.184.156:4004/ http://10.174.184.156/searx + [searx-archlinux] INFO: (eth0) morty: http://10.174.184.156:3000/ + [searx-archlinux] INFO: (eth0) docs.live: http://10.174.184.156:8080/ + [searx-archlinux] INFO: (eth0) IPv6: http://[fd42:573b:e0b3:e97e:216:3eff:fea5:9b65] + ... + diff --git a/docs/dev/makefile.rst b/docs/dev/makefile.rst index 870b5d49c..b7472dad7 100644 --- a/docs/dev/makefile.rst +++ b/docs/dev/makefile.rst @@ -29,8 +29,8 @@ Calling the ``help`` target gives a first overview (``make help``): .. _make install: -Python environment -================== +Python Environment (``make install``) +===================================== .. sidebar:: activate environment diff --git a/docs/dev/offline_engines.rst b/docs/dev/offline_engines.rst new file mode 100644 index 000000000..ce6924542 --- /dev/null +++ b/docs/dev/offline_engines.rst @@ -0,0 +1,78 @@ +.. _offline engines: + +=============== +Offline Engines +=============== + +.. sidebar:: offline engines + + - :ref:`demo offline engine` + - :ref:`sql engines` + - :ref:`engine command` + - :origin:`Redis <searx/engines/redis_server.py>` + +To extend the functionality of SearxNG, offline engines are going to be +introduced. An offline engine is an engine which does not need Internet +connection to perform a search and does not use HTTP to communicate. + +Offline engines can be configured, by adding those to the `engines` list of +:origin:`settings.yml <searx/settings.yml>`. An example skeleton for offline +engines can be found in :ref:`demo offline engine` (:origin:`demo_offline.py +<searx/engines/demo_offline.py>`). + + +Programming Interface +===================== + +:py:func:`init(engine_settings=None) <searx.engines.demo_offline.init>` + All offline engines can have their own init function to setup the engine before + accepting requests. The function gets the settings from settings.yml as a + parameter. This function can be omitted, if there is no need to setup anything + in advance. + +:py:func:`search(query, params) <searx.engines.demo_offline.searc>` + + Each offline engine has a function named ``search``. This function is + responsible to perform a search and return the results in a presentable + format. (Where *presentable* means presentable by the selected result + template.) + + The return value is a list of results retrieved by the engine. + +Engine representation in ``/config`` + If an engine is offline, the attribute ``offline`` is set to ``True``. + +.. _offline requirements: + +Extra Dependencies +================== + +If an offline engine depends on an external tool, SearxNG does not install it by +default. When an administrator configures such engine and starts the instance, +the process returns an error with the list of missing dependencies. Also, +required dependencies will be added to the comment/description of the engine, so +admins can install packages in advance. + +If there is a need to install additional packages in *Python's Virtual +Environment* of your SearxNG instance you need to switch into the environment +(:ref:`searx-src`) first, for this you can use :ref:`searx.sh`:: + + $ sudo utils/searx.sh shell + (searx-pyenv)$ pip install ... + + +Private engines (Security) +========================== + +To limit the access to offline engines, if an instance is available publicly, +administrators can set token(s) for each of the :ref:`private engines`. If a +query contains a valid token, then SearxNG performs the requested private +search. If not, requests from an offline engines return errors. + + +Acknowledgement +=============== + +This development was sponsored by `Search and Discovery Fund +<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_ . + diff --git a/docs/dev/reST.rst b/docs/dev/reST.rst index 181d9829d..230c92a78 100644 --- a/docs/dev/reST.rst +++ b/docs/dev/reST.rst @@ -1281,10 +1281,10 @@ Templating Templating is suitable for documentation which is created generic at the build time. The sphinx-jinja_ extension evaluates jinja_ templates in the :ref:`make install` (with searx modules installed). We use this e.g. to build chapter: -:ref:`engines generic`. Below the jinja directive from the +:ref:`configured engines`. Below the jinja directive from the :origin:`docs/admin/engines.rst` is shown: -.. literalinclude:: ../admin/engines.rst +.. literalinclude:: ../admin/engines/configured_engines.rst :language: reST :start-after: .. _configured engines: diff --git a/docs/dev/search_api.rst b/docs/dev/search_api.rst index 68fee94bf..5fcdc4560 100644 --- a/docs/dev/search_api.rst +++ b/docs/dev/search_api.rst @@ -20,7 +20,7 @@ Parameters - :ref:`engines-dev` - :ref:`settings.yml` - - :ref:`engines generic` + - :ref:`configured engines` ``q`` : required The search query. This string is passed to external search services. Thus, |