diff options
| author | Markus Heiser <markus.heiser@darmarIT.de> | 2021-06-08 10:56:18 +0000 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2021-06-08 10:56:18 +0000 |
| commit | 5c5db719d2039d34aa1426bf7eb3e57055d4a54a (patch) | |
| tree | 15a7a5e1a811f1366cdb49b69e35633afaf56114 /docs/blog | |
| parent | 5f76238d5cd6f9c850d5237fe639ba2900fd3e6c (diff) | |
| parent | 6f7b0d72c084845ea073a82a357c5e99cd41a85f (diff) | |
Merge pull request #97 from return42/drop-searx-admin
[docs] reorder blog articles
Diffstat (limited to 'docs/blog')
| -rw-r--r-- | docs/blog/admin.rst | 43 | ||||
| -rw-r--r-- | docs/blog/command-line-engines.rst | 65 | ||||
| -rw-r--r-- | docs/blog/index.rst | 16 | ||||
| -rw-r--r-- | docs/blog/intro-offline.rst | 77 | ||||
| -rw-r--r-- | docs/blog/lxcdev-202006.rst | 413 | ||||
| -rw-r--r-- | docs/blog/private-engines.rst | 65 | ||||
| -rw-r--r-- | docs/blog/python3.rst | 65 | ||||
| -rw-r--r-- | docs/blog/search-indexer-engines.rst | 114 | ||||
| -rw-r--r-- | docs/blog/searx-admin-engines.png | bin | 50840 -> 0 bytes | |||
| -rw-r--r-- | docs/blog/searxpy3.png | bin | 30947 -> 0 bytes | |||
| -rw-r--r-- | docs/blog/sql-engines.rst | 160 |
11 files changed, 0 insertions, 1018 deletions
diff --git a/docs/blog/admin.rst b/docs/blog/admin.rst deleted file mode 100644 index e95316192..000000000 --- a/docs/blog/admin.rst +++ /dev/null @@ -1,43 +0,0 @@ -============================================================= -Searx admin interface -============================================================= - -.. _searx-admin: https://github.com/kvch/searx-admin#searx-admin -.. _NLnet Foundation: https://nlnet.nl/ - - manage your instance from your browser - -.. sidebar:: Installation - - Installation guide can be found in the repository of searx-admin_. - -One of the biggest advantages of searx is being extremely customizable. But at -first it can be daunting to newcomers. A barrier of taking advantage of this -feature is our ugly settings file which is sometimes hard to understand and -edit. - -To make self-hosting searx more accessible a new tool is introduced, called -``searx-admin``. It is a web application which is capable of managing your -instance and manipulating its settings via a web UI. It aims to replace editing -of ``settings.yml`` for less experienced administrators or people who prefer -graphical admin interfaces. - -.. figure:: searx-admin-engines.png - :alt: Screenshot of engine list - - Configuration page of engines - -Since ``searx-admin`` acts as a supervisor for searx, we have decided to -implement it as a standalone tool instead of part of searx. Another reason for -making it a standalone tool is that the codebase and dependencies of searx -should not grow because of a fully optional feature, which does not affect -existing instances. - - -Acknowledgements -================ - -This development was sponsored by `NLnet Foundation`_. - -| Happy hacking. -| kvch // 2017.08.22 21:25 diff --git a/docs/blog/command-line-engines.rst b/docs/blog/command-line-engines.rst deleted file mode 100644 index 09eb84fb4..000000000 --- a/docs/blog/command-line-engines.rst +++ /dev/null @@ -1,65 +0,0 @@ -======================================== -Running shell commands to fetch results -======================================== - -Previously, with searx you could search over the Internet on other people's -computers. Now it is possible to fetch results from your local machine without -connecting to any networks from the same graphical user interface. - - -Command line engines -==================== - -In :pull-searx:`2128` a new type of engine has been introduced called ``command``. -This engine lets administrators add engines which run arbitrary shell commands -and show its output on the web UI of searx. - -When creating and enabling a ``command`` engine on a public searx instance, -you must be careful to avoid leaking private data. The easiest solution -is to add tokens to the engine. Thus, only those who have the appropriate token -can retrieve results from the it. - -The engine base is flexible. Only your imagination can limit the power of this engine. (And -maybe security concerns.) The following options are available: - -* ``command``: A comma separated list of the elements of the command. A special token {{QUERY}} tells searx where to put the search terms of the user. Example: ``['ls', '-l', '-h', '{{QUERY}}']`` -* ``delimiter``: A dict containing a delimiter char and the "titles" of each element in keys. -* ``parse_regex``: A dict containing the regular expressions for each result key. -* ``query_type``: The expected type of user search terms. Possible values: ``path`` and ``enum``. ``path`` checks if the uesr provided path is inside the working directory. If not the query is not executed. ``enum`` is a list of allowed search terms. If the user submits something which is not included in the list, the query returns an error. -* ``query_enum``: A list containing allowed search terms if ``query_type`` is set to ``enum``. -* ``working_dir``: The directory where the command has to be executed. Default: ``.`` -* ``result_separator``: The character that separates results. Default: ``\n`` - - -The example engine below can be used to find files with a specific name in the configured -working directory. - -.. code:: yaml - - - name: find - engine: command - command: ['find', '.', '-name', '{{QUERY}}'] - query_type: path - shortcut: fnd - delimiter: - chars: ' ' - keys: ['line'] - - -Next steps -========== - -In the next milestone, support for local search engines and indexers (e.g. Elasticsearch) -are going to be added. This way, you will be able to query your own databases/indexers. - -Acknowledgement -=============== - -This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ . - -.. _Search and Discovery Fund: https://nlnet.nl/discovery -.. _NLnet Foundation: https://nlnet.nl/ - - -| Happy hacking. -| kvch // 2020.09.28 21:26 diff --git a/docs/blog/index.rst b/docs/blog/index.rst deleted file mode 100644 index a396ecaf6..000000000 --- a/docs/blog/index.rst +++ /dev/null @@ -1,16 +0,0 @@ -==== -Blog -==== - -.. toctree:: - :maxdepth: 2 - :caption: Contents - - lxcdev-202006 - python3 - admin - intro-offline - private-engines - command-line-engines - search-indexer-engines - sql-engines diff --git a/docs/blog/intro-offline.rst b/docs/blog/intro-offline.rst deleted file mode 100644 index 3a706d038..000000000 --- a/docs/blog/intro-offline.rst +++ /dev/null @@ -1,77 +0,0 @@ -=============================== -Preparation for offline engines -=============================== - -Offline engines -=============== - -To extend the functionality of searx, offline engines are going to be -introduced. An offline engine is an engine which does not need Internet -connection to perform a search and does not use HTTP to communicate. - -Offline engines can be configured as online engines, by adding those to the -`engines` list of :origin:`settings.yml <searx/settings.yml>`. Thus, searx -finds the engine file and imports it. - -Example skeleton for the new engines: - -.. code:: python - - from subprocess import PIPE, Popen - - categories = ['general'] - offline = True - - def init(settings): - pass - - def search(query, params): - process = Popen(['ls', query], stdout=PIPE) - return_code = process.wait() - if return_code != 0: - raise RuntimeError('non-zero return code', return_code) - - results = [] - line = process.stdout.readline() - while line: - result = parse_line(line) - results.append(results) - - line = process.stdout.readline() - - return results - - -Development progress -==================== - -First, a proposal has been created as a Github issue. Then it was moved to the -wiki as a design document. You can read it here: :wiki:`Offline-engines`. - -In this development step, searx core was prepared to accept and perform offline -searches. Offline search requests are scheduled together with regular offline -requests. - -As offline searches can return arbitrary results depending on the engine, the -current result templates were insufficient to present such results. Thus, a new -template is introduced which is caplable of presenting arbitrary key value pairs -as a table. You can check out the pull request for more details see -:pull-searx:`1700`. - -Next steps -========== - -Today, it is possible to create/run an offline engine. However, it is going to be publicly available for everyone who knows the searx instance. So the next step is to introduce token based access for engines. This way administrators are able to limit the access to private engines. - -Acknowledgement -=============== - -This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ . - -.. _Search and Discovery Fund: https://nlnet.nl/discovery -.. _NLnet Foundation: https://nlnet.nl/ - - -| Happy hacking. -| kvch // 2019.10.21 17:03 - diff --git a/docs/blog/lxcdev-202006.rst b/docs/blog/lxcdev-202006.rst deleted file mode 100644 index 24d9028c2..000000000 --- a/docs/blog/lxcdev-202006.rst +++ /dev/null @@ -1,413 +0,0 @@ -.. _blog-lxcdev-202006: - -======================================= -Developing in Linux containers [202006] -======================================= - -.. _LXC: https://linuxcontainers.org/lxc/introduction/ - -.. sidebar:: Audience - - This blog post is written for experienced admins and developers / readers - should have a serious meaning about: *distributed*, *merge* and *linux - container*. - -.. contents:: Contents - :depth: 2 - :local: - :backlinks: entry - -In PR :PR:`1803` we added a lot of scripts to Searx's boilerplate. In this blog -post I will show you, how you can make use of them in *distributed and -heterogeneous development cycles* (TL;DR; jump to the :ref:`blog-lxcdev-202006 -abstract`). - -Motivation -========== - -Normally in our development cycle, we edit the sources and run some test and/or -builds by using ``make`` before we commit. This cycle is simple and perfect but -might fail in some aspects we should not overlook. - - The environment in which we run all our development processes matters! - -The :ref:`makefile` and the :ref:`make install` encapsulate a lot for us, but they -do not have access to all prerequisites. For example, there may have -dependencies on packages that are installed on the developer's desktop, but -usually are not preinstalled on a server or client system. Another examples -are; settings have been made to the software on the developer's host that would -never be set on a *production* system. - -*Linux Containers* (LXC_) are isolate environments and not to mix up on -developer's all the prerequisites of all the projects he contribute to, is -always a good choice. - -The scripts from PR :PR:`1803` can divide in those to install and maintain -software: - -- :ref:`searx.sh` -- :ref:`filtron.sh` -- :ref:`morty.sh` - -and the script :ref:`lxc.sh`, with we can scale our installation, maintenance or -even development tasks over a stack of containers, what we call: *Searx's lxc -suite*. - -Gentlemen, start your engines! -============================== - -.. _LXD: https://linuxcontainers.org/lxd/introduction/ -.. _archlinux: https://www.archlinux.org/ - -Before you can start with containers, you need to install and initiate LXD_ -once: - -.. tabs:: - - .. group-tab:: desktop - - .. code:: sh - - $ snap install lxd - $ lxd init --auto - -And you need to clone from origin or if you have your own fork, clone from your -fork: - -.. tabs:: - - .. group-tab:: desktop - - .. code:: sh - - $ cd ~/Downloads - $ git clone https://github.com/searxng/searxng.git searx - $ cd searx - -The :ref:`lxc-searx.env` consists of several images, see ``export -LXC_SUITE=(...`` near by :origin:`utils/lxc-searx.env#L19`. For this blog post -we exercise on a archlinux_ image. The container of this image is named -``searx-archlinux``. Lets build the container, but be sure that this container -does not already exists, so first lets remove possible old one: - -.. tabs:: - - .. group-tab:: desktop - - .. code:: sh - - $ sudo -H ./utils/lxc.sh remove searx-archlinux - $ sudo -H ./utils/lxc.sh build searx-archlinux - -.. sidebar:: The ``searx-archlinux`` container - - is the base of all our exercises here. - -In this container we install all services :ref:`including searx, morty & filtron -<lxc.sh install suite>` in once: - -.. tabs:: - - .. group-tab:: desktop - - .. code:: sh - - $ sudo -H ./utils/lxc.sh install suite searx-archlinux - -To proxy HTTP from filtron and morty in the container to the outside of the -container, install nginx into the container. Once for the bot blocker filtron: - -.. tabs:: - - .. group-tab:: desktop - - .. code:: sh - - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ - ./utils/filtron.sh nginx install - ... - INFO: got 429 from http://10.174.184.156/searx - -and once for the content sanitizer (content proxy morty): - -.. tabs:: - - .. group-tab:: desktop - - .. code:: sh - - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ - ./utils/morty.sh nginx install - ... - INFO: got 200 from http://10.174.184.156/morty/ - -.. sidebar:: Fully functional searx suite - - From here on you have a fully functional searx suite running with bot blocker - (filtron) and Web content sanitizer (content proxy morty) needed for a - *privacy protecting* search engine. - -On your system, the IP of your ``searx-archlinux`` container differs from -http://10.174.184.156/searx, just open the URL reported in your installation -protocol in your WEB browser from the desktop to test the instance from outside -of the container. - -In such a searx suite admins can maintain and access the debug log of the -different services quite easy. - -.. _working in containers: - -In containers, work as usual -============================ - -Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers -open the root-bash in the container using ``./utils/lxc.sh cmd -searx-archlinux``: - -.. tabs:: - - .. group-tab:: desktop - - .. code:: sh - - $ sudo -H ./utils/lxc.sh cmd searx-archlinux bash - INFO: [searx-archlinux] bash - [root@searx-archlinux searx]# pwd - /share/searx - -The prompt ``[root@searx-archlinux ...]`` signals, that you are the root user in -the searx-container. To debug the running searx instance use: - -.. tabs:: - - .. group-tab:: root@searx-archlinux - - .. code:: sh - - $ ./utils/searx.sh inspect service - ... - use [CTRL-C] to stop monitoring the log - ... - -Back in the browser on your desktop open the service http://10.174.184.156/searx -and run your application tests while the debug log is shown in the terminal from -above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug -option"* in searx's settings file and restarts the searx uwsgi application. To -debug services from filtron and morty analogous use: - -.. tabs:: - - .. group-tab:: root@searx-archlinux - - .. code:: sh - - $ ./utils/filtron.sh inspect service - $ ./utils/morty.sh inspect service - -Another point we have to notice is that each service (:ref:`searx <searx.sh>`, -:ref:`filtron <filtron.sh>` and :ref:`morty <morty.sh>`) runs under dedicated -system user account with the same name (compare :ref:`create searx user`). To -get a shell from theses accounts, simply call one of the scripts: - -.. tabs:: - - .. group-tab:: root@searx-archlinux - - .. code:: sh - - $ ./utils/searx.sh shell - $ ./utils/filtron.sh shell - $ ./utils/morty.sh shell - -To get in touch, open a shell from the service user (searx@searx-archlinux): - -.. tabs:: - - .. group-tab:: desktop - - .. code:: sh - - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ - ./utils/searx.sh shell - // exit with [CTRL-D] - (searx-pyenv) [searx@searx-archlinux ~]$ ... - -The prompt ``[searx@searx-archlinux]`` signals that you are logged in as system -user ``searx`` in the ``searx-archlinux`` container and the python *virtualenv* -``(searx-pyenv)`` environment is activated. - -.. tabs:: - - .. group-tab:: searx@searx-archlinux - - .. code:: sh - - (searx-pyenv) [searx@searx-archlinux ~]$ pwd - /usr/local/searx - - - -Wrap production into developer suite -==================================== - -In this section we will see how to change the *"Fully functional searx suite"* -from a LXC container (which is quite ready for production) into a developer -suite. For this, we have to keep an eye on the :ref:`installation basic`: - -- searx setup in: ``/etc/searx/settings.yml`` -- searx user's home: ``/usr/local/searx`` -- virtualenv in: ``/usr/local/searx/searx-pyenv`` -- searx software in: ``/usr/local/searx/searx-src`` - -The searx software is a clone of the ``git_url`` (see :ref:`settings global`) and -the working tree is checked out from the ``git_branch``. With the use of the -:ref:`searx.sh` the searx service was installed as :ref:`uWSGI application -<searx uwsgi>`. To maintain this service, we can use ``systemctl`` (compare -:ref:`service architectures on distributions <uwsgi configuration>`). - -.. tabs:: - - .. group-tab:: desktop - - .. code:: sh - - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ - systemctl stop uwsgi@searx - -With the command above, we stopped the searx uWSGI-App in the archlinux -container. - -The uWSGI-App for the archlinux dsitros is configured in -:origin:`utils/templates/etc/uwsgi/apps-archlinux/searx.ini`, from where at -least you should attend the settings of ``uid``, ``chdir``, ``env`` and -``http``:: - - env = SEARX_SETTINGS_PATH=/etc/searx/settings.yml - http = 127.0.0.1:8888 - - chdir = /usr/local/searx/searx-src/searx - virtualenv = /usr/local/searx/searx-pyenv - pythonpath = /usr/local/searx/searx-src - -If you have read the :ref:`"Good to know section" <lxc.sh>` you remember, that -each container shares the root folder of the repository and the command -``utils/lxc.sh cmd`` handles relative path names **transparent**. To wrap the -searx installation into a developer one, we simple have to create a smylink to -the **transparent** reposetory from the desktop. Now lets replace the -repository at ``searx-src`` in the container with the working tree from outside -of the container: - -.. tabs:: - - .. group-tab:: container becomes a developer suite - - .. code:: sh - - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ - mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old - - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ - ln -s /share/searx/ /usr/local/searx/searx-src - -Now we can develop as usual in the working tree of our desktop system. Every -time the software was changed, you have to restart the searx service (in the -conatiner): - -.. tabs:: - - .. group-tab:: desktop - - .. code:: sh - - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ - systemctl restart uwsgi@searx - - -Remember: :ref:`working in containers` .. here are just some examples from my -daily usage: - -.. tabs:: - - .. group-tab:: desktop - - To *inspect* the searx instance (already described above): - - .. code:: sh - - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ - ./utils/searx.sh inspect service - - Run :ref:`makefile`, e.g. to test inside the container: - - .. code:: sh - - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ - make test - - To install all prerequisites needed for a :ref:`buildhosts`: - - .. code:: sh - - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ - ./utils/searx.sh install buildhost - - To build the docs on a buildhost :ref:`buildhosts`: - - .. code:: sh - - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ - make docs.html - -.. _blog-lxcdev-202006 abstract: - -Abstract -======== - -We build up a fully functional searx suite in a archlinux container: - -.. code:: sh - - $ sudo -H ./utils/lxc.sh install suite searx-archlinux - -To access HTTP from the desktop we installed nginx for the services inside the -conatiner: - -.. tabs:: - - .. group-tab:: [root@searx-archlinux] - - .. code:: sh - - $ ./utils/filtron.sh nginx install - $ ./utils/morty.sh nginx install - -To wrap the suite into a developer one, we created a symbolic link to the -repository which is shared **transparent** from the desktop's file system into -the container : - -.. tabs:: - - .. group-tab:: [root@searx-archlinux] - - .. code:: sh - - $ mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old - $ ln -s /share/searx/ /usr/local/searx/searx-src - $ systemctl restart uwsgi@searx - -To get remarks from the suite of the archlinux container we can use: - -.. tabs:: - - .. group-tab:: desktop - - .. code:: sh - - $ sudo -H ./utils/lxc.sh show suite searx-archlinux - ... - [searx-archlinux] INFO: (eth0) filtron: http://10.174.184.156:4004/ http://10.174.184.156/searx - [searx-archlinux] INFO: (eth0) morty: http://10.174.184.156:3000/ - [searx-archlinux] INFO: (eth0) docs.live: http://10.174.184.156:8080/ - [searx-archlinux] INFO: (eth0) IPv6: http://[fd42:573b:e0b3:e97e:216:3eff:fea5:9b65] - ... - diff --git a/docs/blog/private-engines.rst b/docs/blog/private-engines.rst deleted file mode 100644 index 37b1d4cc5..000000000 --- a/docs/blog/private-engines.rst +++ /dev/null @@ -1,65 +0,0 @@ -================================== -Limit access to your searx engines -================================== - -Administrators might find themselves wanting to limit access to some of the -enabled engines on their instances. It might be because they do not want to -expose some private information through an offline engine. Or they -would rather share engines only with their trusted friends or colleagues. - -.. _private engines: - -Private engines -=============== - -To solve this issue private engines were introduced in :pull-searx:`1823`. -A new option was added to engines named `tokens`. It expects a list -of strings. If the user making a request presents one of the tokens -of an engine, they can access information about the engine -and make search requests. - -Example configuration to restrict access to the Arch Linux Wiki engine: - -.. code:: yaml - - - name : arch linux wiki - engine : archlinux - shortcut : al - tokens : [ 'my-secret-token' ] - - -Unless a user has configured the right token, the engine is going -to be hidden from him/her. It is not going to be included in the -list of engines on the Preferences page and in the output of -`/config` REST API call. - -Tokens can be added to one's configuration on the Preferences page -under "Engine tokens". The input expects a comma separated list of -strings. - -The distribution of the tokens from the administrator to the users -is not carved in stone. As providing access to such engines -implies that the admin knows and trusts the user, we do not see -necessary to come up with a strict process. Instead, -we would like to add guidelines to the documentation of the feature. - -Next steps -========== - -Now that searx has support for both offline engines and private engines, -it is possible to add concrete engines which benefit from these features. -For example engines which search on the local host running the instance. -Be it searching your file system or querying a private database. Be creative -and come up with new solutions which fit your use case. - -Acknowledgement -=============== - -This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ . - -.. _Search and Discovery Fund: https://nlnet.nl/discovery -.. _NLnet Foundation: https://nlnet.nl/ - - -| Happy hacking. -| kvch // 2020.02.28 22:26 diff --git a/docs/blog/python3.rst b/docs/blog/python3.rst deleted file mode 100644 index 62d7052fc..000000000 --- a/docs/blog/python3.rst +++ /dev/null @@ -1,65 +0,0 @@ -============================ -Introducing Python 3 support -============================ - -.. _Python 2.7 clock: https://pythonclock.org/ - -.. sidebar:: Python 2.7 to 3 upgrade - - This chapter exists of historical reasons. Python 2.7 release schedule ends - (`Python 2.7 clock`_) after 11 years Python 3 exists - -As most operation systems are coming with Python3 installed by default. So it is -time for searx to support Python3. But don't worry support of Python2.7 won't be -dropped. - -.. image:: searxpy3.png - :scale: 50 % - :alt: hurray - :align: center - - -How to run searx using Python 3 -=============================== - -Please make sure that you run at least Python 3.5. - -To run searx, first a Python3 virtualenv should be created. After entering the -virtualenv, dependencies and searx must be installed. Then run searx from the -command line. - -.. code:: sh - - python3 -m venv venv3 - source venv3/bin/activate - pip install -U pip setuptools wheel pyyaml - pip install -e . - searx-run - -Fun facts -========= - -- 115 files were changed when implementing the support for both Python versions. - -- All of the dependencies was compatible except for the robotframework used for - browser tests. Thus, these tests were migrated to splinter. So from now on - both versions are being tested on Travis and can be tested locally. - -If you found bugs -================= - -Please open an issue on `GitHub`_. Make sure that you mention your Python -version in your issue, so we can investigate it properly. - -.. _GitHub: https://github.com/searxng/searxng/issues - -Acknowledgment -============== - -This development was sponsored by `NLnet Foundation`_. - -.. _NLnet Foundation: https://nlnet.nl/ - - -| Happy hacking. -| kvch // 2017.05.13 22:57 diff --git a/docs/blog/search-indexer-engines.rst b/docs/blog/search-indexer-engines.rst deleted file mode 100644 index ca4dd3c88..000000000 --- a/docs/blog/search-indexer-engines.rst +++ /dev/null @@ -1,114 +0,0 @@ -=============================== -Query your local search engines -=============================== - -From now on, searx lets you to query your locally running search engines. The following -ones are supported now: - -* `Elasticsearch`_ -* `Meilisearch`_ -* `Solr`_ - -All of the engines above are added to ``settings.yml`` just commented out, as you have to -``base_url`` for all them. - -Please note that if you are not using HTTPS to access these engines, you have to enable -HTTP requests by setting ``enable_http`` to ``True``. - -Futhermore, if you do not want to expose these engines on a public instance, you can -still add them and limit the access by setting ``tokens`` as described in the `blog post about -private engines`_. - -Configuring searx for search engines -==================================== - -Each search engine is powerful, capable of full-text search. - -Elasticsearch -------------- - -Elasticsearch supports numerous ways to query the data it is storing. At the moment -the engine supports the most popular search methods: ``match``, ``simple_query_string``, ``term`` and ``terms``. - -If none of the methods fit your use case, you can select ``custom`` query type and provide the JSON payload -searx has to submit to Elasticsearch in ``custom_query_json``. - -The following is an example configuration for an Elasticsearch instance with authentication -configured to read from ``my-index`` index. - -.. code:: yaml - - - name : elasticsearch - shortcut : es - engine : elasticsearch - base_url : http://localhost:9200 - username : elastic - password : changeme - index : my-index - query_type : match - enable_http : True - - -Meilisearch ------------ - -This search engine is aimed at individuals and small companies. It is designed for -small-scale (less than 10 million documents) data collections. E.g. it is great for storing -web pages you have visited and searching in the contents later. - -The engine supports faceted search, so you can search in a subset of documents of the collection. -Futhermore, you can search in Meilisearch instances that require authentication by setting ``auth_token``. - -Here is a simple example to query a Meilisearch instance: - -.. code:: yaml - - - name : meilisearch - engine : meilisearch - shortcut: mes - base_url : http://localhost:7700 - index : my-index - enable_http: True - - -Solr ----- - -Solr is a popular search engine based on Lucene, just like Elasticsearch. -But instead of searching in indices, you can search in collections. - -This is an example configuration for searching in the collection ``my-collection`` and get -the results in ascending order. - -.. code:: yaml - - - name : solr - engine : solr - shortcut : slr - base_url : http://localhost:8983 - collection : my-collection - sort : asc - enable_http : True - - -Next steps -========== - -The next step is to add support for various SQL databases. - -Acknowledgement -=============== - -This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ . - -.. _blog post about private engines: private-engines.html#private-engines -.. _Elasticsearch: https://www.elastic.co/elasticsearch/ -.. _Meilisearch: https://www.meilisearch.com/ -.. _Solr: https://solr.apache.org/ -.. _Search and Discovery Fund: https://nlnet.nl/discovery -.. _NLnet Foundation: https://nlnet.nl/ - - -| Happy hacking. -| kvch // 2021.04.07 23:16 - diff --git a/docs/blog/searx-admin-engines.png b/docs/blog/searx-admin-engines.png Binary files differdeleted file mode 100644 index 610bacdf7..000000000 --- a/docs/blog/searx-admin-engines.png +++ /dev/null diff --git a/docs/blog/searxpy3.png b/docs/blog/searxpy3.png Binary files differdeleted file mode 100644 index 8eeaeec55..000000000 --- a/docs/blog/searxpy3.png +++ /dev/null diff --git a/docs/blog/sql-engines.rst b/docs/blog/sql-engines.rst deleted file mode 100644 index 413316054..000000000 --- a/docs/blog/sql-engines.rst +++ /dev/null @@ -1,160 +0,0 @@ -=========== -SQL engines -=========== - -.. sidebar:: further read - - - `SQLite <https://www.sqlite.org/index.html>`_ - - `PostgreSQL <https://www.postgresql.org>`_ - - `MySQL <https://www.mysql.com>`_ - -With the *SQL engines* you can bind SQL databases into SearxNG. The following -Relational Database Management System (RDBMS) are supported: - -- :ref:`engine sqlite` -- :ref:`engine postgresql` -- :ref:`engine mysql_server` - -All of the engines above are just commented out in the :origin:`settings.yml -<searx/settings.yml>`, as you have to set the required attributes for the -engines, e.g. ``database:`` ... - -.. code:: yaml - - - name: ... - engine: {sqlite|postgresql|mysql_server} - database: ... - result_template: {template_name} - query_str: ... - -By default, the engines use the ``key-value`` template for displaying results / -see :origin:`oscar <searx/templates/oscar/result_templates/key-value.html>` & -:origin:`simple <searx/templates/simple/result_templates/key-value.html>` -themes. If you are not satisfied with the original result layout, you can use -your own template, set ``result_template`` attribute to ``{template_name}`` and -place the templates at:: - - searx/templates/{theme_name}/result_templates/{template_name} - -As mentioned in previous blog posts, if you do not wish to expose these engines -on a public instance, you can still add them and limit the access by setting -``tokens`` as described in section :ref:`private engines`. - -Configure the engines -===================== - -The configuration of the new database engines are similar. You must put a valid -SQL-SELECT query in ``query_str``. At the moment you can only bind at most one -parameter in your query. By setting the attribute ``limit`` you can define how -many results you want from the SQL server. Basically, it is the same as the -``LIMIT`` keyword in SQL. - -Please, do not include ``LIMIT`` or ``OFFSET`` in your SQL query as the engines -rely on these keywords during paging. If you want to configure the number of -returned results use the option ``limit``. - -.. _engine sqlite: - -SQLite ------- - -.. _MediathekView: https://mediathekview.de/ - -SQLite is a small, fast and reliable SQL database engine. It does not require -any extra dependency. To demonstrate the power of database engines, here is a -more complex example which reads from a MediathekView_ (DE) movie database. For -this example of the SQlite engine download the database: - -- https://liste.mediathekview.de/filmliste-v2.db.bz2 - -and unpack into ``searx/data/filmliste-v2.db``. To search the database use e.g -Query to test: ``!mediathekview concert`` - -.. code:: yaml - - - name: mediathekview - engine: sqlite - disabled: False - categories: general - result_template: default.html - database: searx/data/filmliste-v2.db - query_str: >- - SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title, - COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url, - description AS content - FROM film - WHERE title LIKE :wildcard OR description LIKE :wildcard - ORDER BY duration DESC - - -Extra Dependencies ------------------- - -For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to -install additional packages in Python's Virtual Environment of your SearxNG -instance. To switch into the environment (:ref:`searx-src`) you can use -:ref:`searx.sh`:: - - $ sudo utils/searx.sh shell - (searx-pyenv)$ pip install ... - - -.. _engine postgresql: - -PostgreSQL ----------- - -.. _psycopg2: https://www.psycopg.org/install - -.. sidebar:: requirements - - ``pip install`` psycopg2_ - -PostgreSQL is a powerful and robust open source database. Before configuring -the PostgreSQL engine, you must install the dependency ``psychopg2``. You can -find an example configuration below: - -.. code:: yaml - - - name: my_database - engine: postgresql - database: my_database - username: searx - password: password - query_str: 'SELECT * from my_table WHERE my_column = %(query)s' - -.. _engine mysql_server: - -MySQL ------ - -.. _mysql-connector-python: https://pypi.org/project/mysql-connector-python - -.. sidebar:: requirements - - ``pip install`` mysql-connector-python_ - -MySQL is said to be the most popular open source database. Before enabling MySQL -engine, you must install the package ``mysql-connector-python``. - -The authentication plugin is configurable by setting ``auth_plugin`` in the -attributes. By default it is set to ``caching_sha2_password``. This is an -example configuration for quering a MySQL server: - -.. code:: yaml - - - name: my_database - engine: mysql_server - database: my_database - username: searx - password: password - limit: 5 - query_str: 'SELECT * from my_table WHERE my_column=%(query)s' - - -Acknowledgement -=============== - -This development was sponsored by `Search and Discovery Fund -<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_ . - |