diff options
| author | Markus Heiser <markus.heiser@darmarit.de> | 2025-07-16 15:47:36 +0200 |
|---|---|---|
| committer | Markus Heiser <markus.heiser@darmarIT.de> | 2025-07-28 12:53:26 +0200 |
| commit | ae0fcf3a4208401f5dc0296c29227a55f63163da (patch) | |
| tree | 3f3612b982c1180064e944e1a83060f0161af57e /docs/dev | |
| parent | 54a2b553f46df6dc98a4b8410788e35165e0fa3e (diff) | |
[chore] overdue maintenance of shell scripts
Removes obsolete scripts and fix various leftovers.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Diffstat (limited to 'docs/dev')
| -rw-r--r-- | docs/dev/index.rst | 1 | ||||
| -rw-r--r-- | docs/dev/lxcdev.rst | 437 |
2 files changed, 0 insertions, 438 deletions
diff --git a/docs/dev/index.rst b/docs/dev/index.rst index 6d601d166..82670284b 100644 --- a/docs/dev/index.rst +++ b/docs/dev/index.rst @@ -17,7 +17,6 @@ Developer documentation plugins/index answerers/index translation - lxcdev makefile reST searxng_extra/index diff --git a/docs/dev/lxcdev.rst b/docs/dev/lxcdev.rst deleted file mode 100644 index 22c16ff0d..000000000 --- a/docs/dev/lxcdev.rst +++ /dev/null @@ -1,437 +0,0 @@ -.. _lxcdev: - -============================== -Developing in Linux Containers -============================== - -.. _LXC: https://linuxcontainers.org/lxc/introduction/ - -In this article we will show, how you can make use of Linux Containers (LXC_) in -*distributed and heterogeneous development cycles* (TL;DR; jump to the -:ref:`lxcdev summary`). - -.. sidebar:: Audience - - This blog post is written for experienced admins and developers. Readers - should have a serious meaning about the terms: *distributed*, *merge* and - *linux container*. - - **hint** - - If you have issues with the internet connectivity of your containers read - section :ref:`internet connectivity docker`. - - -.. contents:: - :depth: 2 - :local: - :backlinks: entry - - -Motivation -========== - -Most often in our development cycle, we edit the sources and run some test -and/or builds by using ``make`` :ref:`[ref] <makefile>` before we commit. This -cycle is simple and perfect but might fail in some aspects we should not -overlook. - - **The environment in which we run all our development processes matters!** - -The :ref:`makefile` and the :ref:`make install` encapsulate a lot for us, but -these tools do not have access to all prerequisites. For example, there may -have dependencies on packages that are installed on developer's desktop, but -usually are not preinstalled on a server or client system. Another example is; -settings have been made to the software on developer's desktop that would never -be set on a *production* system. - - **Linux Containers are isolate environments**, we use them to not mix up all - the prerequisites from various projects on developer's desktop. - -The scripts from :ref:`searx_utils` can divide in those to install and maintain -software - -- :ref:`searxng.sh` - -and the script - -- :ref:`lxc.sh` - -with we can scale our installation, maintenance or even development tasks over a -stack of isolated containers / what we call the: - -- :ref:`searxng lxc suite` - -.. _lxcdev install searxng: - -Gentlemen, start your engines! -============================== - -.. _LXD: https://linuxcontainers.org/lxd/introduction/ -.. _archlinux: https://www.archlinux.org/ - -Before you can start with containers, you need to install and initiate LXD_ -once: - -.. tabs:: - - .. group-tab:: desktop (HOST) - - .. code:: bash - - $ snap install lxd - $ lxd init --auto - -And you need to clone from origin or if you have your own fork, clone from your -fork: - -.. tabs:: - - .. group-tab:: desktop (HOST) - - .. code:: bash - - $ cd ~/Downloads - $ git clone https://github.com/searxng/searxng.git searxng - $ cd searxng - -.. sidebar:: The ``searxng-archlinux`` container - - is the base of all our exercises here. - -The :ref:`lxc-searxng.env` consists of several images, see ``export -LXC_SUITE=(...`` near by :origin:`utils/lxc-searxng.env#L19`. -For this blog post we exercise on a archlinux_ image. The container of this -image is named ``searxng-archlinux``. - -Lets build the container, but be sure that this container does not already -exists, so first lets remove possible old one: - -.. tabs:: - - .. group-tab:: desktop (HOST) - - .. code:: bash - - $ sudo -H ./utils/lxc.sh remove searxng-archlinux - $ sudo -H ./utils/lxc.sh build searxng-archlinux - - -.. sidebar:: further read - - - :ref:`lxc.sh install suite` - - :ref:`installation nginx` - -To install the complete :ref:`SearXNG suite <searxng lxc suite>` and the HTTP -proxy :ref:`installation nginx` into the archlinux container run: - -.. tabs:: - - .. group-tab:: desktop (HOST) - - .. code:: bash - - $ sudo -H ./utils/lxc.sh install suite searxng-archlinux - $ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx - $ sudo ./utils/lxc.sh show suite | grep SEARXNG_URL - ... - [searxng-archlinux] SEARXNG_URL : http://n.n.n.140/searxng - -.. sidebar:: Fully functional SearXNG suite - - From here on you have a fully functional SearXNG suite (including a - :ref:`valkey db`). - -In such a SearXNG suite admins can maintain and access the debug log of the -services quite easy. - -In the example above the SearXNG instance in the container is wrapped to -``http://n.n.n.140/searxng`` to the HOST system. Note, on your HOST system, the -IP of your ``searxng-archlinux`` container is different to this example. To -test the instance in the container from outside of the container, in your WEB -browser on your desktop just open the URL reported in your installation - -.. _working in containers: - -In containers, work as usual -============================ - -Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers -open the root-bash in the container is done by the ``./utils/lxc.sh cmd -searxng-archlinux`` command: - -.. tabs:: - - .. group-tab:: desktop (HOST) - - .. code:: bash - - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux bash - INFO: [searxng-archlinux] bash - [root@searxng-archlinux SearXNG]$ - -The prompt ``[root@searxng-archlinux ...]`` signals, that you are the root user -in the container (GUEST). To debug the running SearXNG instance use: - -.. tabs:: - - .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) - - .. code:: bash - - $ ./utils/searxng.sh instance inspect - ... - use [CTRL-C] to stop monitoring the log - ... - - .. group-tab:: desktop (HOST) - - .. code:: bash - - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh instance inspect - ... - use [CTRL-C] to stop monitoring the log - ... - - -Back in the browser on your desktop open the service http://n.n.n.140/searxng -and run your application tests while the debug log is shown in the terminal from -above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug -option"* in SearXNG's settings file and restarts the SearXNG uwsgi application. - -Another point we have to notice is that the service :ref:`SearXNG <searxng.sh>` -runs under dedicated system user account with the same name (compare -:ref:`create searxng user`). To get a login shell from these accounts, simply -call: - -.. tabs:: - - .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) - - .. code:: bash - - $ ./utils/searxng.sh instance cmd bash -l - (searx-pyenv) [searxng@searxng-archlinux ~]$ pwd - /usr/local/searxng - - .. group-tab:: desktop (HOST) - - .. code:: bash - - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh instance cmd bash -l - INFO: [searxng-archlinux] ./utils/searxng.sh instance cmd bash -l - (searx-pyenv) [searxng@searxng-archlinux ~]$ pwd - /usr/local/searxng - -The prompt ``[searxng@searxng-archlinux]`` signals that you are logged in as system -user ``searxng`` in the ``searxng-archlinux`` container and the python *virtualenv* -``(searxng-pyenv)`` environment is activated. - - -Wrap production into developer suite -==================================== - -In this section we will see how to change the *"Fully functional SearXNG suite"* -from a LXC container (which is quite ready for production) into a developer -suite. For this, we have to keep an eye on the :ref:`installation basic`: - -- SearXNG setup in: ``/etc/searxng/settings.yml`` -- SearXNG user's home: ``/usr/local/searxng`` -- virtualenv in: ``/usr/local/searxng/searxng-pyenv`` -- SearXNG software in: ``/usr/local/searxng/searxng-src`` - -With the use of the :ref:`searxng.sh` the SearXNG service was installed as -:ref:`uWSGI application <searxng uwsgi>`. To maintain this service, we can use -``systemctl`` (compare :ref:`uWSGI maintenance`). - -.. tabs:: - - .. group-tab:: uwsgi@searxng - - .. code:: bash - - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl stop uwsgi@searxng - -With the command above, we stopped the SearXNG uWSGI-App in the archlinux -container. - -The uWSGI-App for the archlinux distros is configured in -:origin:`utils/templates/etc/uwsgi/apps-archlinux/searxng.ini`, from where at -least you should attend the settings of ``uid``, ``chdir``, ``env`` and -``http``:: - - env = SEARXNG_SETTINGS_PATH=/etc/searxng/settings.yml - http = 127.0.0.1:8888 - - chdir = /usr/local/searxng/searxng-src/searx - virtualenv = /usr/local/searxng/searxng-pyenv - pythonpath = /usr/local/searxng/searxng-src - -If you have read the :ref:`Good to know` you remember, that each container -shares the root folder of the repository and the command ``utils/lxc.sh cmd`` -handles relative path names **transparent**. - -To wrap the SearXNG installation in the container into a developer one, we -simple have to create a symlink to the **transparent** repository from the -desktop. Now lets replace the repository at ``searxng-src`` in the container -with the working tree from outside of the container: - -.. tabs:: - - .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) - - .. code:: bash - - $ mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old - $ ln -s /share/SearXNG/ /usr/local/searxng/searxng-src - - .. group-tab:: desktop (HOST) - - .. code:: bash - - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ - mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old - - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ - ln -s /share/SearXNG/ /usr/local/searxng/searxng-src - -Now we can develop as usual in the working tree of our desktop system. Every -time the software was changed, you have to restart the SearXNG service (in the -container): - -.. tabs:: - - .. group-tab:: uwsgi@searxng - - .. code:: bash - - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng - - -Remember: :ref:`working in containers` .. here are just some examples from my -daily usage: - -To *inspect* the SearXNG instance (already described above): - -.. tabs:: - - .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) - - .. code:: bash - - $ ./utils/searxng.sh inspect service - - .. group-tab:: desktop (HOST) - - .. code:: bash - - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh inspect service - -Run :ref:`makefile`, e.g. to test inside the container: - -.. tabs:: - - .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) - - .. code:: bash - - $ make test - - .. group-tab:: desktop (HOST) - - .. code:: bash - - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux make test - - - -To install all prerequisites needed for a :ref:`buildhosts`: - -.. tabs:: - - .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) - - .. code:: bash - - $ ./utils/searxng.sh install buildhost - - .. group-tab:: desktop (HOST) - - .. code:: bash - - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh install buildhost - - -To build the docs on a buildhost :ref:`buildhosts`: - -.. tabs:: - - .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) - - .. code:: bash - - $ make docs.html - - .. group-tab:: desktop (HOST) - - .. code:: bash - - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux make docs.html - - -.. _lxcdev summary: - -Summary -======= - -We build up a fully functional SearXNG suite in a archlinux container: - -.. code:: bash - - $ sudo -H ./utils/lxc.sh build searxng-archlinux - $ sudo -H ./utils/lxc.sh install suite searxng-archlinux - ... - Developer install? (wraps source from HOST into the running instance) [YES/no] - -To wrap the suite into a developer one answer ``YES`` (or press Enter). - -.. code:: text - - link SearXNG's sources to: /share/SearXNG - ========================================= - - mv -f "/usr/local/searxng/searxng-src" "/usr/local/searxng/searxng-src.backup" - ln -s "/share/SearXNG" "/usr/local/searxng/searxng-src" - ls -ld /usr/local/searxng/searxng-src - |searxng| lrwxrwxrwx 1 searxng searxng ... /usr/local/searxng/searxng-src -> /share/SearXNG - -On code modification the instance has to be restarted (see :ref:`uWSGI -maintenance`): - -.. code:: bash - - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng - -To access HTTP from the desktop we installed nginx for the services inside the -container: - -.. code:: bash - - $ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx - -To get information about the SearxNG suite in the archlinux container we can -use: - -.. code:: text - - $ sudo -H ./utils/lxc.sh show suite searxng-archlinux - [searxng-archlinux] INFO: (eth0) docs-live: http:///n.n.n.140:8080/ - [searxng-archlinux] INFO: (eth0) IPv6: http://[fd42:555b:2af9:e121:216:3eff:fe5b:1744] - [searxng-archlinux] uWSGI: - [searxng-archlinux] SEARXNG_UWSGI_SOCKET : /usr/local/searxng/run/socket - [searxng-archlinux] environment /usr/local/searxng/searxng-src/utils/brand.env: - [searxng-archlinux] GIT_URL : https://github.com/searxng/searxng - [searxng-archlinux] GIT_BRANCH : master - [searxng-archlinux] SEARXNG_URL : http:///n.n.n.140/searxng - [searxng-archlinux] SEARXNG_PORT : 8888 - [searxng-archlinux] SEARXNG_BIND_ADDRESS : 127.0.0.1 |