Merge branch 'master' into about-opensearch

5 files changed, 400 insertions, 0 deletions

diff --git a/docs/utils/filtron.sh.rst b/docs/utils/filtron.sh.rst
new file mode 100644
index 000000000..86e3fa88a
--- /dev/null
+++ b/docs/utils/filtron.sh.rst
@@ -0,0 +1,80 @@
+
+.. _filtron.sh:
+
+====================
+``utils/filtron.sh``
+====================
+
+.. sidebar:: further reading
+
+   - :ref:`searx filtron`
+   - :ref:`architecture`
+   - :ref:`installation` (:ref:`nginx <installation nginx>` & :ref:`apache
+     <installation apache>`)
+
+.. _Go: https://golang.org/
+.. _filtron: https://github.com/asciimoo/filtron
+.. _filtron README: https://github.com/asciimoo/filtron/blob/master/README.md
+
+To simplify installation and maintenance of a filtron instance you can use the
+script :origin:`utils/filtron.sh`.  In most cases you will install filtron_
+simply by running the command:
+
+.. code::  bash
+
+   sudo -H ./utils/filtron.sh install all
+
+The script adds a ``${SERVICE_USER}`` (default:``filtron``) and installs filtron_
+into this user account:
+
+#. Create a separated user account (``filtron``).
+#. Download and install Go_ binary in user's $HOME (``~filtron``).
+#. Install filtron with the package management from Go_ (``go get -v -u
+   github.com/asciimoo/filtron``)
+#. Setup a proper rule configuration :origin:`[ref]
+   <utils/templates/etc/filtron/rules.json>` (``/etc/filtron/rules.json``).
+#. Setup a systemd service unit :origin:`[ref]
+   <utils/templates/lib/systemd/system/filtron.service>`
+   (``/lib/systemd/system/filtron.service``).
+
+
+Create user
+===========
+
+.. kernel-include:: $DOCS_BUILD/includes/filtron.rst
+   :start-after: START create user
+   :end-before: END create user
+
+
+Install go
+==========
+
+.. kernel-include:: $DOCS_BUILD/includes/filtron.rst
+   :start-after: START install go
+   :end-before: END install go
+
+
+Install filtron
+===============
+
+Install :origin:`rules.json <utils/templates/etc/filtron/rules.json>` at
+``/etc/filtron/rules.json`` (see :ref:`Sample configuration of filtron`) and
+install filtron software and systemd unit:
+
+.. kernel-include:: $DOCS_BUILD/includes/filtron.rst
+   :start-after: START install filtron
+   :end-before: END install filtron
+
+.. kernel-include:: $DOCS_BUILD/includes/filtron.rst
+   :start-after: START install systemd unit
+   :end-before: END install systemd unit
+
+.. _filtron.sh overview:
+
+Overview
+========
+
+The ``--help`` output of the script is largely self-explanatory
+(:ref:`toolboxing common`):
+
+.. program-output:: ../utils/filtron.sh --help
diff --git a/docs/utils/index.rst b/docs/utils/index.rst
new file mode 100644
index 000000000..13914af28
--- /dev/null
+++ b/docs/utils/index.rst
@@ -0,0 +1,53 @@
+.. _searx_utils:
+.. _toolboxing:
+
+=======================
+Tooling box ``utils/*``
+=======================
+
+In the folder :origin:`utils/` we maintain some tools useful for admins and
+developers.
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents
+
+   searx.sh
+   filtron.sh
+   morty.sh
+   lxc.sh
+
+.. _toolboxing common:
+
+Common commands & environment
+=============================
+
+Scripts to maintain services often dispose of common commands and environments.
+
+``shell`` : command
+  Opens a shell from the service user ``${SERVICE_USSR}``, very helpful for
+  troubleshooting.
+
+``inspect service`` : command
+  Shows status and log of the service, most often you have a option to enable
+  more verbose debug logs.  Very helpful for debugging, but be careful not to
+  enable debugging in a production environment!
+
+``FORCE_TIMEOUT`` : environment
+  Sets timeout for interactive prompts. If you want to run a script in batch
+  job, with defaults choices, set ``FORCE_TIMEOUT=0``.  By example; to install a
+  reverse proxy for filtron on all containers of the :ref:`searx suite
+  <lxc-searx.env>` use ::
+
+    sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/filtron.sh apache install
+ 
+.. _toolboxing setup:
+
+Tooling box setup
+=================
+
+The main setup is done in the :origin:`.config.sh` (read also :ref:`makefile
+setup`).
+
+.. literalinclude:: ../../.config.sh
+   :language: bash
diff --git a/docs/utils/lxc.sh.rst b/docs/utils/lxc.sh.rst
new file mode 100644
index 000000000..ae1412870
--- /dev/null
+++ b/docs/utils/lxc.sh.rst
@@ -0,0 +1,148 @@
+
+.. _snap: https://snapcraft.io
+.. _snapcraft LXD: https://snapcraft.io/lxd
+.. _LXC/LXD Image Server: https://uk.images.linuxcontainers.org/
+.. _LXC: https://linuxcontainers.org/lxc/introduction/
+.. _LXD: https://linuxcontainers.org/lxd/introduction/
+.. _`LXD@github`: https://github.com/lxc/lxd
+
+.. _archlinux: https://www.archlinux.org/
+
+.. _lxc.sh:
+
+================
+``utils/lxc.sh``
+================
+
+.. sidebar:: further reading
+
+   - snap_, `snapcraft LXD`_
+   - LXC_,  LXD_
+   - `LXC/LXD Image Server`_
+   - `LXD@github`_
+
+With the use of *Linux Containers* (LXC_) we can scale our tasks over a stack of
+containers, what we call the: *lxc suite*.  The *searx suite*
+(:origin:`lxc-searx.env <utils/lxc-searx.env>`) is loaded by default, every time
+you start the ``lxc.sh`` script (*you do not need to care about*).
+
+Before you can start with containers, you need to install and initiate LXD_
+once::
+
+  $ snap install lxd
+  $ lxd init --auto
+
+To make use of the containers from the *searx suite*, you have to build the
+:ref:`LXC suite containers <lxc.sh help>` initial.  But be warned, **this might
+take some time**::
+
+  $ sudo -H ./utils/lxc.sh build
+
+A cup of coffee later, your LXC suite is build up and you can run whatever task
+you want / in a selected or even in all :ref:`LXC suite containers <lxc.sh
+help>`.  If you do not want to build all containers, **you can build just
+one**::
+
+  $ sudo -H ./utils/lxc.sh build searx-ubu1804
+
+*Good to know ...*
+
+Eeach container shares the root folder of the repository and the
+command ``utils/lxc.sh cmd`` **handles relative path names transparent**,
+compare output of::
+
+  $ sudo -H ./utils/lxc.sh cmd -- ls -la Makefile
+  ...
+
+In the containers, you can run what ever you want, e.g. to start a bash use::
+
+  $ sudo -H ./utils/lxc.sh cmd searx-ubu1804 bash
+  INFO:  [searx-ubu1804] bash
+  root@searx-ubu1804:/share/searx#
+
+If there comes the time you want to **get rid off all** the containers and
+**clean up local images** just type::
+
+  $ sudo -H ./utils/lxc.sh remove
+  $ sudo -H ./utils/lxc.sh remove images
+
+
+Install suite
+=============
+
+To install the complete :ref:`searx suite (includes searx, morty & filtron)
+<lxc-searx.env>` into all LXC_ use::
+
+  $ sudo -H ./utils/lxc.sh install suite
+
+The command above installs a searx suite (see :ref:`installation scripts`).  To
+get the IP (URL) of the filtron service in the containers use ``show suite``
+command.  To test instances from containers just open the URLs in your
+WEB-Browser::
+
+  $ sudo ./utils/lxc.sh show suite | grep filtron
+  [searx-ubu1604]  INFO:  (eth0) filtron:    http://n.n.n.246:4004/ http://n.n.n.246/searx
+  [searx-ubu1804]  INFO:  (eth0) filtron:    http://n.n.n.147:4004/ http://n.n.n.147/searx
+  [searx-ubu1910]  INFO:  (eth0) filtron:    http://n.n.n.140:4004/ http://n.n.n.140/searx
+  [searx-ubu2004]  INFO:  (eth0) filtron:    http://n.n.n.18:4004/ http://n.n.n.18/searx
+  [searx-fedora31]  INFO:  (eth0) filtron:    http://n.n.n.46:4004/ http://n.n.n.46/searx
+  [searx-archlinux]  INFO:  (eth0) filtron:    http://n.n.n.32:4004/ http://n.n.n.32/searx
+
+To :ref:`install a nginx <installation nginx>` reverse proxy for filtron and
+morty use (or alternatively use :ref:`apache <installation apache>`)::
+
+    sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/filtron.sh nginx install
+    sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/morty.sh nginx install
+
+
+Running commands
+================
+
+**Inside containers, you can use make or run scripts** from the
+:ref:`toolboxing`.  By example: to setup a :ref:`buildhosts` and run the
+Makefile target ``test`` in the archlinux_ container::
+
+  sudo -H ./utils/lxc.sh cmd searx-archlinux ./utils/searx.sh install buildhost
+  sudo -H ./utils/lxc.sh cmd searx-archlinux make test
+
+
+Setup searx buildhost
+=====================
+
+You can **install the searx buildhost environment** into one or all containers.
+The installation procedure to set up a :ref:`build host<buildhosts>` takes its
+time.  Installation in all containers will take more time (time for another cup
+of coffee).::
+
+  sudo -H ./utils/lxc.sh cmd -- ./utils/searx.sh install buildhost
+
+To build (live) documentation inside a archlinux_ container::
+
+  sudo -H ./utils/lxc.sh cmd searx-archlinux make docs-clean docs-live
+  ...
+  [I 200331 15:00:42 server:296] Serving on http://0.0.0.0:8080
+
+To get IP of the container and the port number *live docs* is listening::
+
+  $ sudo ./utils/lxc.sh show suite | grep docs-live
+  ...
+  [searx-archlinux]  INFO:  (eth0) docs-live:  http://n.n.n.12:8080/
+
+
+.. _lxc.sh help:
+
+Overview
+========
+
+The ``--help`` output of the script is largely self-explanatory:
+
+.. program-output:: ../utils/lxc.sh --help
+
+
+.. _lxc-searx.env:
+
+searx suite
+===========
+
+.. literalinclude:: ../../utils/lxc-searx.env
+   :language: bash
diff --git a/docs/utils/morty.sh.rst b/docs/utils/morty.sh.rst
new file mode 100644
index 000000000..3bdf9bdd3
--- /dev/null
+++ b/docs/utils/morty.sh.rst
@@ -0,0 +1,80 @@
+
+.. _morty: https://github.com/asciimoo/morty
+.. _morty's README: https://github.com/asciimoo/morty
+.. _Go: https://golang.org/
+
+.. _morty.sh:
+
+==================
+``utils/morty.sh``
+==================
+
+.. sidebar:: further reading
+
+   - :ref:`architecture`
+   - :ref:`installation` (:ref:`nginx <installation nginx>` & :ref:`apache
+     <installation apache>`)
+   - :ref:`searx morty`
+
+To simplify installation and maintenance of a morty_ instance you can use the
+script :origin:`utils/morty.sh`.  In most cases you will install morty_ simply by
+running the command:
+
+.. code::  bash
+
+   sudo -H ./utils/morty.sh install all
+
+The script adds a ``${SERVICE_USER}`` (default:``morty``) and installs morty_
+into this user account:
+
+#. Create a separated user account (``morty``).
+#. Download and install Go_ binary in user's $HOME (``~morty``).
+#. Install morty_ with the package management from Go_ (``go get -v -u
+   github.com/asciimoo/morty``)
+#. Setup a systemd service unit :origin:`[ref]
+   <utils/templates/lib/systemd/system/morty.service>`
+   (``/lib/systemd/system/morty.service``).
+
+.. hint::
+
+   To add morty to your searx instance read chapter :ref:`searx morty`.
+
+Create user
+===========
+
+.. kernel-include:: $DOCS_BUILD/includes/morty.rst
+   :start-after: START create user
+   :end-before: END create user
+
+
+Install go
+==========
+
+.. kernel-include:: $DOCS_BUILD/includes/morty.rst
+   :start-after: START install go
+   :end-before: END install go
+
+
+Install morty
+=============
+
+Install morty software and systemd unit:
+
+.. kernel-include:: $DOCS_BUILD/includes/morty.rst
+   :start-after: START install morty
+   :end-before: END install morty
+
+.. kernel-include:: $DOCS_BUILD/includes/morty.rst
+   :start-after: START install systemd unit
+   :end-before: END install systemd unit
+
+.. _morty.sh overview:
+
+Overview
+========
+
+The ``--help`` output of the script is largely self-explanatory
+(:ref:`toolboxing common`):
+
+.. program-output:: ../utils/morty.sh --help
+
diff --git a/docs/utils/searx.sh.rst b/docs/utils/searx.sh.rst
new file mode 100644
index 000000000..dd4442f94
--- /dev/null
+++ b/docs/utils/searx.sh.rst
@@ -0,0 +1,39 @@
+
+.. _searx.sh:
+
+==================
+``utils/searx.sh``
+==================
+
+.. sidebar:: further reading
+
+   - :ref:`architecture`
+   - :ref:`installation`
+   - :ref:`installation nginx`
+   - :ref:`installation apache`
+
+To simplify installation and maintenance of a searx instance you can use the
+script :origin:`utils/searx.sh`.
+
+Install
+=======
+
+In most cases you will install searx simply by running the command:
+
+.. code::  bash
+
+   sudo -H ./utils/searx.sh install all
+
+The script adds a ``${SERVICE_USER}`` (default:``searx``) and installs searx
+into this user account.  The installation is described in chapter
+:ref:`installation basic`.
+
+.. _intranet reverse proxy:
+
+Overview
+========
+
+The ``--help`` output of the script is largely self-explanatory
+(:ref:`toolboxing common`):
+
+.. program-output:: ../utils/searx.sh --help