diff options
Diffstat (limited to 'docs/admin')
| -rw-r--r-- | docs/admin/engines.rst | 88 | ||||
| -rw-r--r-- | docs/admin/engines/command-line-engines.rst | 79 | ||||
| -rw-r--r-- | docs/admin/engines/configured_engines.rst | 77 | ||||
| -rw-r--r-- | docs/admin/engines/index.rst | 22 | ||||
| -rw-r--r-- | docs/admin/engines/private-engines.rst | 49 | ||||
| -rw-r--r-- | docs/admin/engines/recoll.rst | 12 | ||||
| -rw-r--r-- | docs/admin/engines/search-indexer-engines.rst | 136 | ||||
| -rw-r--r-- | docs/admin/engines/searx.engines.xpath.rst | 9 | ||||
| -rw-r--r-- | docs/admin/engines/settings.rst (renamed from docs/admin/settings.rst) | 227 | ||||
| -rw-r--r-- | docs/admin/engines/sql-engines.rst | 169 | ||||
| -rw-r--r-- | docs/admin/index.rst | 3 |
11 files changed, 674 insertions, 197 deletions
diff --git a/docs/admin/engines.rst b/docs/admin/engines.rst deleted file mode 100644 index 3ad206303..000000000 --- a/docs/admin/engines.rst +++ /dev/null @@ -1,88 +0,0 @@ -======= -Engines -======= - -Special Engine Settings -======================= - -.. sidebar:: Further reading .. - - - :ref:`settings engine` - - :ref:`engine settings` & :ref:`engine file` - -.. toctree:: - :maxdepth: 1 - - engines/recoll.rst - - -.. _engines generic: - -General Engine Settings -======================= - -Explanation of the :ref:`general engine configuration` shown in the table -:ref:`configured engines`. - -============= =========== ==================== ============ -:ref:`engine settings` :ref:`engine file` -------------------------- --------------------------------- -Name (cfg) Categories -------------------------- --------------------------------- -Engine .. Paging support **P** -------------------------- -------------------- ------------ -Shortcut **S** Language support **L** -Timeout **TO** Time range support **TR** -Disabled **D** Engine type **ET** -------------- ----------- -------------------- ------------ -Safe search **SS** -------------- ----------- --------------------------------- -Weigth **W** -------------- ----------- --------------------------------- -Disabled **D** -------------- ----------- --------------------------------- -Show errors **DE** -============= =========== ================================= - -.. _configured engines: - -.. jinja:: searx - - .. flat-table:: Engines configured at built time (defaults) - :header-rows: 1 - :stub-columns: 2 - - * - Name (cfg) - - S - - Engine - - TO - - Categories - - P - - L - - SS - - D - - TR - - ET - - W - - D - - DE - - {% for name, mod in engines.items() %} - - * - {{name}} - - !{{mod.shortcut}} - - {{mod.__name__}} - - {{mod.timeout}} - - {{", ".join(mod.categories)}} - - {{(mod.paging and "y") or ""}} - - {{(mod.language_support and "y") or ""}} - - {{(mod.safesearch and "y") or ""}} - - {{(mod.disabled and "y") or ""}} - - {{(mod.time_range_support and "y") or ""}} - - {{mod.engine_type or ""}} - - {{mod.weight or 1 }} - - {{(mod.disabled and "y") or ""}} - - {{(mod.display_error_messages and "y") or ""}} - - {% endfor %} - diff --git a/docs/admin/engines/command-line-engines.rst b/docs/admin/engines/command-line-engines.rst new file mode 100644 index 000000000..e9535e74f --- /dev/null +++ b/docs/admin/engines/command-line-engines.rst @@ -0,0 +1,79 @@ +.. _engine command: + +==================== +Command Line Engines +==================== + +.. sidebar:: info + + - :origin:`command.py <searx/engines/command.py>` + - :ref:`offline engines` + +With *command engines* administrators can run engines to integrate arbitrary +shell commands. + +When creating and enabling a ``command`` engine on a public instance, you must +be careful to avoid leaking private data. The easiest solution is to limit the +access by setting ``tokens`` as described in section :ref:`private engines`. + +The engine base is flexible. Only your imagination can limit the power of this +engine (and maybe security concerns). The following options are available: + +``command``: + A comma separated list of the elements of the command. A special token + ``{{QUERY}}`` tells where to put the search terms of the user. Example: + + .. code:: yaml + + ['ls', '-l', '-h', '{{QUERY}}'] + +``delimiter``: + A mapping containing a delimiter ``char`` and the *titles* of each element in + ``keys``. + +``parse_regex``: + A dict containing the regular expressions for each result key. + +``query_type``: + + The expected type of user search terms. Possible values: ``path`` and + ``enum``. + + ``path``: + Checks if the user provided path is inside the working directory. If not, + the query is not executed. + + ``enum``: + Is a list of allowed search terms. If the user submits something which is + not included in the list, the query returns an error. + +``query_enum``: + A list containing allowed search terms if ``query_type`` is set to ``enum``. + +``working_dir``: + + The directory where the command has to be executed. Default: ``./`` + +``result_separator``: + The character that separates results. Default: ``\n`` + +The example engine below can be used to find files with a specific name in the +configured working directory: + +.. code:: yaml + + - name: find + engine: command + command: ['find', '.', '-name', '{{QUERY}}'] + query_type: path + shortcut: fnd + delimiter: + chars: ' ' + keys: ['line'] + + +Acknowledgment +============== + +This development was sponsored by `Search and Discovery Fund +<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_. diff --git a/docs/admin/engines/configured_engines.rst b/docs/admin/engines/configured_engines.rst new file mode 100644 index 000000000..af20e5611 --- /dev/null +++ b/docs/admin/engines/configured_engines.rst @@ -0,0 +1,77 @@ +.. _configured engines: + +================== +Configured Engines +================== + +.. sidebar:: Further reading .. + + - :ref:`engines-dev` + - :ref:`settings engine` + +Explanation of the :ref:`general engine configuration` shown in the table +:ref:`configured engines`. + +.. table:: The legend for the following table + :width: 100% + + ============= =========== ==================== ============ + :ref:`engine settings` :ref:`engine file` + ------------------------- --------------------------------- + Name (cfg) .. Categories + ------------- ----------- -------------------- ------------ + Engine .. Paging support **P** + ------------- ----------- -------------------- ------------ + Shortcut **S** Language support **L** + Timeout **TO** Time range support **TR** + Disabled **D** Engine type **ET** + ------------- ----------- -------------------- ------------ + Safe search **SS** + ------------- ----------- --------------------------------- + Weigth **W** + ------------- ----------- --------------------------------- + Disabled **D** + ------------- ----------- --------------------------------- + Show errors **DE** + ============= =========== ================================= + +.. jinja:: searx + + .. flat-table:: Engines configured at built time (defaults) + :header-rows: 1 + :stub-columns: 2 + + * - Name (cfg) + - S + - Engine + - TO + - Categories + - P + - L + - SS + - D + - TR + - ET + - W + - D + - DE + + {% for name, mod in engines.items() %} + + * - {{name}} + - !{{mod.shortcut}} + - {{mod.__name__}} + - {{mod.timeout}} + - {{", ".join(mod.categories)}} + - {{(mod.paging and "y") or ""}} + - {{(mod.language_support and "y") or ""}} + - {{(mod.safesearch and "y") or ""}} + - {{(mod.disabled and "y") or ""}} + - {{(mod.time_range_support and "y") or ""}} + - {{mod.engine_type or ""}} + - {{mod.weight or 1 }} + - {{(mod.disabled and "y") or ""}} + - {{(mod.display_error_messages and "y") or ""}} + + {% endfor %} + diff --git a/docs/admin/engines/index.rst b/docs/admin/engines/index.rst new file mode 100644 index 000000000..80f4120a5 --- /dev/null +++ b/docs/admin/engines/index.rst @@ -0,0 +1,22 @@ +.. _engines and settings: + +================== +Engines & Settings +================== + +.. sidebar:: Further reading .. + + - :ref:`settings engine` + - :ref:`engine settings` & :ref:`engine file` + +.. toctree:: + :maxdepth: 1 + + settings + configured_engines + private-engines + recoll + sql-engines + search-indexer-engines + command-line-engines + searx.engines.xpath diff --git a/docs/admin/engines/private-engines.rst b/docs/admin/engines/private-engines.rst new file mode 100644 index 000000000..cc6ab2565 --- /dev/null +++ b/docs/admin/engines/private-engines.rst @@ -0,0 +1,49 @@ +.. _private engines: + +============================ +Private Engines (``tokens``) +============================ + +Administrators might find themselves wanting to limit access to some of the +enabled engines on their instances. It might be because they do not want to +expose some private information through :ref:`offline engines`. Or they would +rather share engines only with their trusted friends or colleagues. + +To solve this issue the concept of *private engines* exists. + + +A new option was added to engines named `tokens`. It expects a list of +strings. If the user making a request presents one of the tokens of an engine, +they can access information about the engine and make search requests. + +Example configuration to restrict access to the Arch Linux Wiki engine: + +.. code:: yaml + + - name: arch linux wiki + engine: archlinux + shortcut: al + tokens: [ 'my-secret-token' ] + + +Unless a user has configured the right token, the engine is going +to be hidden from him/her. It is not going to be included in the +list of engines on the Preferences page and in the output of +`/config` REST API call. + +Tokens can be added to one's configuration on the Preferences page +under "Engine tokens". The input expects a comma separated list of +strings. + +The distribution of the tokens from the administrator to the users +is not carved in stone. As providing access to such engines +implies that the admin knows and trusts the user, we do not see +necessary to come up with a strict process. Instead, +we would like to add guidelines to the documentation of the feature. + + +Acknowledgment +============== + +This development was sponsored by `Search and Discovery Fund +<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_. diff --git a/docs/admin/engines/recoll.rst b/docs/admin/engines/recoll.rst index cba2e81f7..9b198db7e 100644 --- a/docs/admin/engines/recoll.rst +++ b/docs/admin/engines/recoll.rst @@ -1,17 +1,17 @@ .. _engine recoll: -====== -Recoll -====== +============= +Recoll Engine +============= .. sidebar:: info - `Recoll <https://www.lesbonscomptes.com/recoll/>`_ - `recoll-webui <https://framagit.org/medoc92/recollwebui.git>`_ + - :origin:`searx/engines/recoll.py` -Recoll_ is a desktop full-text search tool based on Xapian. By itself Recoll_ -does not offer web or API access, this can be achieved using recoll-webui_ - +Recoll_ is a desktop full-text search tool based on Xapian. By itself Recoll_ +does not offer WEB or API access, this can be achieved using recoll-webui_ Configuration diff --git a/docs/admin/engines/search-indexer-engines.rst b/docs/admin/engines/search-indexer-engines.rst new file mode 100644 index 000000000..8d7c10d67 --- /dev/null +++ b/docs/admin/engines/search-indexer-engines.rst @@ -0,0 +1,136 @@ +==================== +Local Search Engines +==================== + +.. sidebar:: further read + + - `Comparison to alternatives + <https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html>`_ + +Administrators might find themselves wanting to integrate locally running search +engines. The following ones are supported for now: + +* `Elasticsearch`_ +* `Meilisearch`_ +* `Solr`_ + +Each search engine is powerful, capable of full-text search. All of the engines +above are added to ``settings.yml`` just commented out, as you have to +``base_url`` for all them. + +Please note that if you are not using HTTPS to access these engines, you have to enable +HTTP requests by setting ``enable_http`` to ``True``. + +Futhermore, if you do not want to expose these engines on a public instance, you +can still add them and limit the access by setting ``tokens`` as described in +section :ref:`private engines`. + +.. _engine meilisearch: + +MeiliSearch +=========== + +.. sidebar:: info + + - :origin:`meilisearch.py <searx/engines/meilisearch.py>` + - `MeiliSearch <https://www.meilisearch.com>`_ + - `MeiliSearch Documentation <https://docs.meilisearch.com/>`_ + - `Install MeiliSearch + <https://docs.meilisearch.com/learn/getting_started/installation.html>`_ + +MeiliSearch_ is aimed at individuals and small companies. It is designed for +small-scale (less than 10 million documents) data collections. E.g. it is great +for storing web pages you have visited and searching in the contents later. + +The engine supports faceted search, so you can search in a subset of documents +of the collection. Furthermore, you can search in MeiliSearch_ instances that +require authentication by setting ``auth_token``. + +Here is a simple example to query a Meilisearch instance: + +.. code:: yaml + + - name: meilisearch + engine: meilisearch + shortcut: mes + base_url: http://localhost:7700 + index: my-index + enable_http: true + + +.. _engine elasticsearch: + +Elasticsearch +============= + +.. sidebar:: info + + - :origin:`elasticsearch.py <searx/engines/elasticsearch.py>` + - `Elasticsearch <https://www.elastic.co/elasticsearch/>`_ + - `Elasticsearch Guide + <https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html>`_ + - `Install Elasticsearch + <https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html>`_ + +Elasticsearch_ supports numerous ways to query the data it is storing. At the +moment the engine supports the most popular search methods (``query_type``): + +- ``match``, +- ``simple_query_string``, +- ``term`` and +- ``terms``. + +If none of the methods fit your use case, you can select ``custom`` query type +and provide the JSON payload to submit to Elasticsearch in +``custom_query_json``. + +The following is an example configuration for an Elasticsearch_ instance with +authentication configured to read from ``my-index`` index. + +.. code:: yaml + + - name: elasticsearch + shortcut: es + engine: elasticsearch + base_url: http://localhost:9200 + username: elastic + password: changeme + index: my-index + query_type: match + # custom_query_json: '{ ... }' + enable_http: true + +.. _engine solr: + +Solr +==== + +.. sidebar:: info + + - :origin:`solr.py <searx/engines/solr.py>` + - `Solr <https://solr.apache.org>`_ + - `Solr Resources <https://solr.apache.org/resources.html>`_ + - `Install Solr <https://solr.apache.org/guide/installing-solr.html>`_ + +Solr_ is a popular search engine based on Lucene, just like Elasticsearch_. But +instead of searching in indices, you can search in collections. + +This is an example configuration for searching in the collection +``my-collection`` and get the results in ascending order. + +.. code:: yaml + + - name: solr + engine: solr + shortcut: slr + base_url: http://localhost:8983 + collection: my-collection + sort: asc + enable_http: true + + +Acknowledgment +============== + +This development was sponsored by `Search and Discovery Fund +<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_. diff --git a/docs/admin/engines/searx.engines.xpath.rst b/docs/admin/engines/searx.engines.xpath.rst new file mode 100644 index 000000000..695aa5224 --- /dev/null +++ b/docs/admin/engines/searx.engines.xpath.rst @@ -0,0 +1,9 @@ +.. _xpath engine: + +============ +XPath Engine +============ + +.. automodule:: searx.engines.xpath + :members: + diff --git a/docs/admin/settings.rst b/docs/admin/engines/settings.rst index d0773467d..1fe083a97 100644 --- a/docs/admin/settings.rst +++ b/docs/admin/engines/settings.rst @@ -22,13 +22,14 @@ file. settings.yml location ===================== -First, searx will try to load settings.yml from these locations: +The initial ``settings.yml`` we be load from these locations: 1. the full path specified in the ``SEARX_SETTINGS_PATH`` environment variable. 2. ``/etc/searx/settings.yml`` If these files don't exist (or are empty or can't be read), searx uses the -:origin:`searx/settings.yml` file. +:origin:`searx/settings.yml` file. Read :ref:`settings use_default_settings` to +see how you can simplify your *user defined* ``settings.yml``. .. _settings global: @@ -42,17 +43,19 @@ Global Settings .. code:: yaml general: - debug : False # Debug mode, only for development - instance_name : "searxng" # displayed name - contact_url: False # mailto:contact@example.com + debug: false # Debug mode, only for development + instance_name: "searxng" # displayed name + contact_url: false # mailto:contact@example.com + +.. code:: yaml brand: - git_url: https://github.com/searxng/searxng - git_branch: master - issue_url: https://github.com/searxng/searxng/issues - docs_url: https://searxng/searxng.github.io/searxng - public_instances: https://searx.space - wiki_url: https://github.com/searxng/searxng/wiki + git_url: https://github.com/searxng/searxng + git_branch: master + issue_url: https://github.com/searxng/searxng/issues + docs_url: https://searxng/searxng.github.io/searxng + public_instances: https://searx.space + wiki_url: https://github.com/searxng/searxng/wiki ``debug`` : Allow a more detailed log if you run searx directly. Display *detailed* error @@ -68,10 +71,10 @@ Global Settings If you host your own documentation, change this URL. ``wiki_url``: - Link to your wiki (or ``False``) + Link to your wiki (or ``false``) ``twitter_url``: - Link to your tweets (or ``False``) + Link to your tweets (or ``false``) ``server:`` @@ -80,19 +83,19 @@ Global Settings .. code:: yaml server: - port : 8888 - bind_address : "127.0.0.1" # address to listen on - secret_key : "ultrasecretkey" # change this! - base_url : False # set custom base_url (or False) - image_proxy : False # proxying image results through searx - default_locale : "" # default interface locale - default_theme : oscar # ui theme + port: 8888 + bind_address: "127.0.0.1" # address to listen on + secret_key: "ultrasecretkey" # change this! + base_url: false # set custom base_url (or false) + image_proxy: false # proxying image results through searx + default_locale: "" # default interface locale + default_theme: oscar # ui theme default_http_headers: - X-Content-Type-Options : nosniff - X-XSS-Protection : 1; mode=block - X-Download-Options : noopen - X-Robots-Tag : noindex, nofollow - Referrer-Policy : no-referrer + X-Content-Type-Options : nosniff + X-XSS-Protection : 1; mode=block + X-Download-Options : noopen + X-Robots-Tag : noindex, nofollow + Referrer-Policy : no-referrer ``port`` & ``bind_address``: Port number and *bind address* of the searx web application if you run it @@ -125,26 +128,30 @@ Global Settings ``outgoing:`` ------------- +Communication with search engines. + .. code:: yaml - outgoing: # communication with search engines - request_timeout : 2.0 # default timeout in seconds, can be override by engine - # max_request_timeout: 10.0 # the maximum timeout in seconds - useragent_suffix : "" # informations like an email address to the administrator - pool_connections : 100 # Maximum number of allowable connections, or None for no limits. The default is 100. - pool_maxsize : 10 # Number of allowable keep-alive connections, or None to always allow. The default is 10. - enable_http2: True # See https://www.python-httpx.org/http2/ - # uncomment below section if you want to use a proxy - # proxies: - # all://: - # - http://proxy1:8080 - # - http://proxy2:8080 - # uncomment below section only if you have more than one network interface - # which can be the source of outgoing search requests - # source_ips: - # - 1.1.1.1 - # - 1.1.1.2 - # - fe80::/126 + outgoing: + request_timeout: 2.0 # default timeout in seconds, can be override by engine + max_request_timeout: 10.0 # the maximum timeout in seconds + useragent_suffix: "" # informations like an email address to the administrator + pool_connections: 100 # Maximum number of allowable connections, or null + # for no limits. The default is 100. + pool_maxsize: 10 # Number of allowable keep-alive connections, or null + # to always allow. The default is 10. + enable_http2: true # See https://www.python-httpx.org/http2/ + # uncomment below section if you want to use a proxy + # proxies: + # all://: + # - http://proxy1:8080 + # - http://proxy2:8080 + # uncomment below section only if you have more than one network interface + # which can be the source of outgoing search requests + # source_ips: + # - 1.1.1.1 + # - 1.1.1.2 + # - fe80::/126 ``request_timeout`` : @@ -180,8 +187,8 @@ Global Settings * ``[ 192.168.0.1, fe80::/126 ]`` ``retries`` : - Number of retry in case of an HTTP error. - On each retry, searx uses an different proxy and source ip. + Number of retry in case of an HTTP error. On each retry, searx uses an + different proxy and source ip. ``retry_on_http_error`` : Retry request on some HTTP status code. @@ -193,7 +200,7 @@ Global Settings * ``[403, 429]``: on HTTP status code 403 and 429. ``enable_http2`` : - Enable by default. Set to ``False`` to disable HTTP/2. + Enable by default. Set to ``false`` to disable HTTP/2. ``max_redirects`` : 30 by default. Maximum redirect before it is an error. @@ -205,18 +212,18 @@ Global Settings .. code:: yaml locales: - en : English - de : Deutsch - he : Hebrew - hu : Magyar - fr : Français - es : Español - it : Italiano - nl : Nederlands - ja : 日本語 (Japanese) - tr : Türkçe - ru : Russian - ro : Romanian + en: English + de: Deutsch + he: Hebrew + hu: Magyar + fr: Français + es: Español + it: Italiano + nl: Nederlands + ja: 日本語 (Japanese) + tr: Türkçe + ru: Russian + ro: Romanian ``locales`` : Locales codes and their names. Available translations of searx interface. @@ -229,35 +236,49 @@ Engine settings .. sidebar:: Further reading .. + - :ref:`configured engines` - :ref:`engines-dev` +In the code example below a *full fledged* example of a YAML setup from a dummy +engine is shown. Most of the options have a default value or even are optional. + .. code:: yaml - - name : bing - engine : bing - shortcut : bi - base_url : 'https://{language}.wikipedia.org/' - categories : general - timeout : 3.0 - api_key : 'apikey' - disabled : True - language : en_US - #enable_http: False - #enable_http2: False - #retries: 1 - #retry_on_http_error: True # or 403 or [404, 429] - #max_connections: 100 - #max_keepalive_connections: 10 - #keepalive_expiry: 5.0 - #proxies: - # http: - # - http://proxy1:8080 - # - http://proxy2:8080 - # https: - # - http://proxy1:8080 - # - http://proxy2:8080 - # - socks5://user:password@proxy3:1080 - # - socks5h://user:password@proxy4:1080 + - name: example engine + engine: example + shortcut: demo + base_url: 'https://{language}.example.com/' + categories: general + timeout: 3.0 + api_key: 'apikey' + disabled: false + language: en_US + tokens: [ 'my-secret-token' ] + weigth: 1 + display_error_messages: true + about: + website: https://example.com + wikidata_id: Q306656 + official_api_documentation: https://example.com/api-doc + use_official_api: true + require_api_key: true + results: HTML + enable_http: false + enable_http2: false + retries: 1 + retry_on_http_error: true # or 403 or [404, 429] + max_connections: 100 + max_keepalive_connections: 10 + keepalive_expiry: 5.0 + proxies: + http: + - http://proxy1:8080 + - http://proxy2:8080 + https: + - http://proxy1:8080 + - http://proxy2:8080 + - socks5://user:password@proxy3:1080 + - socks5h://user:password@proxy4:1080 ``name`` : Name that will be used across searx to define this engine. In settings, on @@ -297,10 +318,14 @@ Engine settings by using the full ISO code of language and country, like ``fr_FR``, ``en_US``, ``de_DE``. +``tokens`` : optional + A list of secret tokens to make this engine *private*, more details see + :ref:`private engines`. + ``weigth`` : default ``1`` Weighting of the results of this engine. -``display_error_messages`` : default ``True`` +``display_error_messages`` : default ``true`` When an engine returns an error, the message is displayed on the user interface. ``network``: optional @@ -320,7 +345,7 @@ Engine settings use_default_settings ==================== -.. sidebar:: ``use_default_settings: True`` +.. sidebar:: ``use_default_settings: true`` - :ref:`settings location` - :ref:`use_default_settings.yml` @@ -329,7 +354,7 @@ use_default_settings The user defined ``settings.yml`` is loaded from the :ref:`settings location` and can relied on the default configuration :origin:`searx/settings.yml` using: - ``use_default_settings: True`` + ``use_default_settings: true`` ``server:`` In the following example, the actual settings are the default settings defined @@ -338,22 +363,22 @@ and can relied on the default configuration :origin:`searx/settings.yml` using: .. code-block:: yaml - use_default_settings: True + use_default_settings: true server: - secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA" + secret_key: "ultrasecretkey" # change this! bind_address: "0.0.0.0" ``engines:`` - With ``use_default_settings: True``, each settings can be override in a + With ``use_default_settings: true``, each settings can be override in a similar way, the ``engines`` section is merged according to the engine ``name``. In this example, searx will load all the engine and the arch linux - wiki engine has a :ref:`token<private engines>`: + wiki engine has a :ref:`token <private engines>`: .. code-block:: yaml - use_default_settings: True + use_default_settings: true server: - secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA" + secret_key: "ultrasecretkey" # change this! engines: - name: arch linux wiki tokens: ['$ecretValue'] @@ -366,11 +391,11 @@ and can relied on the default configuration :origin:`searx/settings.yml` using: .. code-block:: yaml use_default_settings: - engines: - remove: - - google + engines: + remove: + - google server: - secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA" + secret_key: "ultrasecretkey" # change this! engines: - name: arch linux wiki tokens: ['$ecretValue'] @@ -382,12 +407,12 @@ and can relied on the default configuration :origin:`searx/settings.yml` using: .. code-block:: yaml use_default_settings: - engines: - keep_only: - - google - - duckduckgo + engines: + keep_only: + - google + - duckduckgo server: - secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA" + secret_key: "ultrasecretkey" # change this! engines: - name: google tokens: ['$ecretValue'] diff --git a/docs/admin/engines/sql-engines.rst b/docs/admin/engines/sql-engines.rst new file mode 100644 index 000000000..ee7f31c27 --- /dev/null +++ b/docs/admin/engines/sql-engines.rst @@ -0,0 +1,169 @@ +.. _sql engines: + +=========== +SQL Engines +=========== + +.. sidebar:: further read + + - `SQLite <https://www.sqlite.org/index.html>`_ + - `PostgreSQL <https://www.postgresql.org>`_ + - `MySQL <https://www.mysql.com>`_ + +With the *SQL engines* you can bind SQL databases into SearxNG. The following +Relational Database Management System (RDBMS) are supported: + +- :ref:`engine sqlite` +- :ref:`engine postgresql` +- :ref:`engine mysql_server` + +All of the engines above are just commented out in the :origin:`settings.yml +<searx/settings.yml>`, as you have to set the required attributes for the +engines, e.g. ``database:`` ... + +.. code:: yaml + + - name: ... + engine: {sqlite|postgresql|mysql_server} + database: ... + result_template: {template_name} + query_str: ... + +By default, the engines use the ``key-value`` template for displaying results / +see :origin:`oscar <searx/templates/oscar/result_templates/key-value.html>` & +:origin:`simple <searx/templates/simple/result_templates/key-value.html>` +themes. If you are not satisfied with the original result layout, you can use +your own template, set ``result_template`` attribute to ``{template_name}`` and +place the templates at:: + + searx/templates/{theme_name}/result_templates/{template_name} + +If you do not wish to expose these engines on a public instance, you can still +add them and limit the access by setting ``tokens`` as described in section +:ref:`private engines`. + + +Configure the engines +===================== + +The configuration of the new database engines are similar. You must put a valid +SQL-SELECT query in ``query_str``. At the moment you can only bind at most one +parameter in your query. By setting the attribute ``limit`` you can define how +many results you want from the SQL server. Basically, it is the same as the +``LIMIT`` keyword in SQL. + +Please, do not include ``LIMIT`` or ``OFFSET`` in your SQL query as the engines +rely on these keywords during paging. If you want to configure the number of +returned results use the option ``limit``. + +.. _engine sqlite: + +SQLite +------ + +.. sidebar:: info + + - :origin:`sqlite.py <searx/engines/sqlite.py>` + +.. _MediathekView: https://mediathekview.de/ + +SQLite is a small, fast and reliable SQL database engine. It does not require +any extra dependency. To demonstrate the power of database engines, here is a +more complex example which reads from a MediathekView_ (DE) movie database. For +this example of the SQlite engine download the database: + +- https://liste.mediathekview.de/filmliste-v2.db.bz2 + +and unpack into ``searx/data/filmliste-v2.db``. To search the database use e.g +Query to test: ``!mediathekview concert`` + +.. code:: yaml + + - name: mediathekview + engine: sqlite + disabled: False + categories: general + result_template: default.html + database: searx/data/filmliste-v2.db + query_str: >- + SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title, + COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url, + description AS content + FROM film + WHERE title LIKE :wildcard OR description LIKE :wildcard + ORDER BY duration DESC + + +Extra Dependencies +------------------ + +For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to +install additional packages in Python's Virtual Environment of your SearxNG +instance. To switch into the environment (:ref:`searx-src`) you can use +:ref:`searx.sh`:: + + $ sudo utils/searx.sh shell + (searx-pyenv)$ pip install ... + + +.. _engine postgresql: + +PostgreSQL +---------- + +.. _psycopg2: https://www.psycopg.org/install + +.. sidebar:: info + + - :origin:`postgresql.py <searx/engines/postgresql.py>` + - ``pip install`` psycopg2_ + +PostgreSQL is a powerful and robust open source database. Before configuring +the PostgreSQL engine, you must install the dependency ``psychopg2``. You can +find an example configuration below: + +.. code:: yaml + + - name: my_database + engine: postgresql + database: my_database + username: searx + password: password + query_str: 'SELECT * from my_table WHERE my_column = %(query)s' + +.. _engine mysql_server: + +MySQL +----- + +.. _mysql-connector-python: https://pypi.org/project/mysql-connector-python + +.. sidebar:: info + + - :origin:`mysql_server.py <searx/engines/mysql_server.py>` + - ``pip install`` mysql-connector-python_ + +MySQL is said to be the most popular open source database. Before enabling MySQL +engine, you must install the package ``mysql-connector-python``. + +The authentication plugin is configurable by setting ``auth_plugin`` in the +attributes. By default it is set to ``caching_sha2_password``. This is an +example configuration for quering a MySQL server: + +.. code:: yaml + + - name: my_database + engine: mysql_server + database: my_database + username: searx + password: password + limit: 5 + query_str: 'SELECT * from my_table WHERE my_column=%(query)s' + + +Acknowledgment +============== + +This development was sponsored by `Search and Discovery Fund +<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_. + diff --git a/docs/admin/index.rst b/docs/admin/index.rst index c708c4ffa..3139db99c 100644 --- a/docs/admin/index.rst +++ b/docs/admin/index.rst @@ -13,11 +13,10 @@ Administrator documentation installation-apache installation-docker update-searx - settings + engines/index api architecture filtron morty - engines plugins buildhosts |