From a5610835377470b0ce19d8e40f91e48ce4aae4cc Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Thu, 3 Jun 2021 09:48:50 +0200 Subject: [docs] move blog article "Command line engines" to admin/engines/ The article "Command line engines" should be in admin's engine documentation (like the recoll engine). Signed-off-by: Markus Heiser --- docs/admin/engines.rst | 3 +- docs/admin/engines/command-line-engines.rst | 64 +++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+), 1 deletion(-) create mode 100644 docs/admin/engines/command-line-engines.rst (limited to 'docs/admin') diff --git a/docs/admin/engines.rst b/docs/admin/engines.rst index 3ad206303..520fda35c 100644 --- a/docs/admin/engines.rst +++ b/docs/admin/engines.rst @@ -13,7 +13,8 @@ Special Engine Settings .. toctree:: :maxdepth: 1 - engines/recoll.rst + engines/recoll + engines/command-line-engines .. _engines generic: diff --git a/docs/admin/engines/command-line-engines.rst b/docs/admin/engines/command-line-engines.rst new file mode 100644 index 000000000..b4568f7d5 --- /dev/null +++ b/docs/admin/engines/command-line-engines.rst @@ -0,0 +1,64 @@ +.. _engine command: + +============================== +Fetch results from commandline +============================== + +Previously, with searx you could search over the Internet on other people's +computers. Now it is possible to fetch results from your local machine without +connecting to any networks from the same graphical user interface. + +.. _command line engines: + +Command line engines +==================== + +In :pull-searx:`2128` a new type of engine has been introduced called ``command``. +This engine lets administrators add engines which run arbitrary shell commands +and show its output on the web UI of searx. + +When creating and enabling a ``command`` engine on a public searx instance, +you must be careful to avoid leaking private data. The easiest solution +is to add tokens to the engine. Thus, only those who have the appropriate token +can retrieve results from the it. + +The engine base is flexible. Only your imagination can limit the power of this engine. (And +maybe security concerns.) The following options are available: + +* ``command``: A comma separated list of the elements of the command. A special token {{QUERY}} tells searx where to put the search terms of the user. Example: ``['ls', '-l', '-h', '{{QUERY}}']`` +* ``delimiter``: A dict containing a delimiter char and the "titles" of each element in keys. +* ``parse_regex``: A dict containing the regular expressions for each result key. +* ``query_type``: The expected type of user search terms. Possible values: ``path`` and ``enum``. ``path`` checks if the uesr provided path is inside the working directory. If not the query is not executed. ``enum`` is a list of allowed search terms. If the user submits something which is not included in the list, the query returns an error. +* ``query_enum``: A list containing allowed search terms if ``query_type`` is set to ``enum``. +* ``working_dir``: The directory where the command has to be executed. Default: ``.`` +* ``result_separator``: The character that separates results. Default: ``\n`` + + +The example engine below can be used to find files with a specific name in the configured +working directory. + +.. code:: yaml + + - name: find + engine: command + command: ['find', '.', '-name', '{{QUERY}}'] + query_type: path + shortcut: fnd + delimiter: + chars: ' ' + keys: ['line'] + + +Next steps +========== + +In the next milestone, support for local search engines and indexers (e.g. Elasticsearch) +are going to be added. This way, you will be able to query your own databases/indexers. + +Acknowledgement +=============== + +This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ . + +.. _Search and Discovery Fund: https://nlnet.nl/discovery +.. _NLnet Foundation: https://nlnet.nl/ -- cgit v1.2.3 From 1b3f0c848e9e9785635f19e4c935a9a0a8d4f7f9 Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Thu, 3 Jun 2021 09:53:28 +0200 Subject: [docs] move blog article "SQL engines" to admin/engines/ The article "SQL engines" should be in admin's engine documentation (like the recoll engine). Signed-off-by: Markus Heiser --- docs/admin/engines.rst | 1 + docs/admin/engines/sql-engines.rst | 169 +++++++++++++++++++++++++++++++++++++ 2 files changed, 170 insertions(+) create mode 100644 docs/admin/engines/sql-engines.rst (limited to 'docs/admin') diff --git a/docs/admin/engines.rst b/docs/admin/engines.rst index 520fda35c..c7461d3fd 100644 --- a/docs/admin/engines.rst +++ b/docs/admin/engines.rst @@ -14,6 +14,7 @@ Special Engine Settings :maxdepth: 1 engines/recoll + engines/sql-engines engines/command-line-engines diff --git a/docs/admin/engines/sql-engines.rst b/docs/admin/engines/sql-engines.rst new file mode 100644 index 000000000..ee7f31c27 --- /dev/null +++ b/docs/admin/engines/sql-engines.rst @@ -0,0 +1,169 @@ +.. _sql engines: + +=========== +SQL Engines +=========== + +.. sidebar:: further read + + - `SQLite `_ + - `PostgreSQL `_ + - `MySQL `_ + +With the *SQL engines* you can bind SQL databases into SearxNG. The following +Relational Database Management System (RDBMS) are supported: + +- :ref:`engine sqlite` +- :ref:`engine postgresql` +- :ref:`engine mysql_server` + +All of the engines above are just commented out in the :origin:`settings.yml +`, as you have to set the required attributes for the +engines, e.g. ``database:`` ... + +.. code:: yaml + + - name: ... + engine: {sqlite|postgresql|mysql_server} + database: ... + result_template: {template_name} + query_str: ... + +By default, the engines use the ``key-value`` template for displaying results / +see :origin:`oscar ` & +:origin:`simple ` +themes. If you are not satisfied with the original result layout, you can use +your own template, set ``result_template`` attribute to ``{template_name}`` and +place the templates at:: + + searx/templates/{theme_name}/result_templates/{template_name} + +If you do not wish to expose these engines on a public instance, you can still +add them and limit the access by setting ``tokens`` as described in section +:ref:`private engines`. + + +Configure the engines +===================== + +The configuration of the new database engines are similar. You must put a valid +SQL-SELECT query in ``query_str``. At the moment you can only bind at most one +parameter in your query. By setting the attribute ``limit`` you can define how +many results you want from the SQL server. Basically, it is the same as the +``LIMIT`` keyword in SQL. + +Please, do not include ``LIMIT`` or ``OFFSET`` in your SQL query as the engines +rely on these keywords during paging. If you want to configure the number of +returned results use the option ``limit``. + +.. _engine sqlite: + +SQLite +------ + +.. sidebar:: info + + - :origin:`sqlite.py ` + +.. _MediathekView: https://mediathekview.de/ + +SQLite is a small, fast and reliable SQL database engine. It does not require +any extra dependency. To demonstrate the power of database engines, here is a +more complex example which reads from a MediathekView_ (DE) movie database. For +this example of the SQlite engine download the database: + +- https://liste.mediathekview.de/filmliste-v2.db.bz2 + +and unpack into ``searx/data/filmliste-v2.db``. To search the database use e.g +Query to test: ``!mediathekview concert`` + +.. code:: yaml + + - name: mediathekview + engine: sqlite + disabled: False + categories: general + result_template: default.html + database: searx/data/filmliste-v2.db + query_str: >- + SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title, + COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url, + description AS content + FROM film + WHERE title LIKE :wildcard OR description LIKE :wildcard + ORDER BY duration DESC + + +Extra Dependencies +------------------ + +For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to +install additional packages in Python's Virtual Environment of your SearxNG +instance. To switch into the environment (:ref:`searx-src`) you can use +:ref:`searx.sh`:: + + $ sudo utils/searx.sh shell + (searx-pyenv)$ pip install ... + + +.. _engine postgresql: + +PostgreSQL +---------- + +.. _psycopg2: https://www.psycopg.org/install + +.. sidebar:: info + + - :origin:`postgresql.py ` + - ``pip install`` psycopg2_ + +PostgreSQL is a powerful and robust open source database. Before configuring +the PostgreSQL engine, you must install the dependency ``psychopg2``. You can +find an example configuration below: + +.. code:: yaml + + - name: my_database + engine: postgresql + database: my_database + username: searx + password: password + query_str: 'SELECT * from my_table WHERE my_column = %(query)s' + +.. _engine mysql_server: + +MySQL +----- + +.. _mysql-connector-python: https://pypi.org/project/mysql-connector-python + +.. sidebar:: info + + - :origin:`mysql_server.py ` + - ``pip install`` mysql-connector-python_ + +MySQL is said to be the most popular open source database. Before enabling MySQL +engine, you must install the package ``mysql-connector-python``. + +The authentication plugin is configurable by setting ``auth_plugin`` in the +attributes. By default it is set to ``caching_sha2_password``. This is an +example configuration for quering a MySQL server: + +.. code:: yaml + + - name: my_database + engine: mysql_server + database: my_database + username: searx + password: password + limit: 5 + query_str: 'SELECT * from my_table WHERE my_column=%(query)s' + + +Acknowledgment +============== + +This development was sponsored by `Search and Discovery Fund +`_ of `NLnet Foundation `_. + -- cgit v1.2.3 From 0d31740cfd20fae749277bb7604253aa0c86c898 Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Thu, 3 Jun 2021 14:19:32 +0200 Subject: [docs] move article "XPath Engine" to to admin/engines/ Signed-off-by: Markus Heiser --- docs/admin/engines.rst | 1 + docs/admin/engines/searx.engines.xpath.rst | 9 +++++++++ 2 files changed, 10 insertions(+) create mode 100644 docs/admin/engines/searx.engines.xpath.rst (limited to 'docs/admin') diff --git a/docs/admin/engines.rst b/docs/admin/engines.rst index c7461d3fd..87613f1ee 100644 --- a/docs/admin/engines.rst +++ b/docs/admin/engines.rst @@ -16,6 +16,7 @@ Special Engine Settings engines/recoll engines/sql-engines engines/command-line-engines + engines/searx.engines.xpath .. _engines generic: diff --git a/docs/admin/engines/searx.engines.xpath.rst b/docs/admin/engines/searx.engines.xpath.rst new file mode 100644 index 000000000..695aa5224 --- /dev/null +++ b/docs/admin/engines/searx.engines.xpath.rst @@ -0,0 +1,9 @@ +.. _xpath engine: + +============ +XPath Engine +============ + +.. automodule:: searx.engines.xpath + :members: + -- cgit v1.2.3 From d965c634297fc1b6010e45b2050422611d06193b Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Thu, 3 Jun 2021 14:54:20 +0200 Subject: [docs] reorder chapter "Engines & Settings" - Split chapter "Engines" and rename it into "Engines & Settings" - Move docs/admin/engines.rst -> docs/admin/engines/engine_settings.rst Signed-off-by: Markus Heiser --- docs/admin/engines.rst | 91 ---------------------------------- docs/admin/engines/engine_settings.rst | 76 ++++++++++++++++++++++++++++ docs/admin/engines/index.rst | 21 ++++++++ docs/admin/index.rst | 2 +- docs/admin/settings.rst | 1 + 5 files changed, 99 insertions(+), 92 deletions(-) delete mode 100644 docs/admin/engines.rst create mode 100644 docs/admin/engines/engine_settings.rst create mode 100644 docs/admin/engines/index.rst (limited to 'docs/admin') diff --git a/docs/admin/engines.rst b/docs/admin/engines.rst deleted file mode 100644 index 87613f1ee..000000000 --- a/docs/admin/engines.rst +++ /dev/null @@ -1,91 +0,0 @@ -======= -Engines -======= - -Special Engine Settings -======================= - -.. sidebar:: Further reading .. - - - :ref:`settings engine` - - :ref:`engine settings` & :ref:`engine file` - -.. toctree:: - :maxdepth: 1 - - engines/recoll - engines/sql-engines - engines/command-line-engines - engines/searx.engines.xpath - - -.. _engines generic: - -General Engine Settings -======================= - -Explanation of the :ref:`general engine configuration` shown in the table -:ref:`configured engines`. - -============= =========== ==================== ============ -:ref:`engine settings` :ref:`engine file` -------------------------- --------------------------------- -Name (cfg) Categories -------------------------- --------------------------------- -Engine .. Paging support **P** -------------------------- -------------------- ------------ -Shortcut **S** Language support **L** -Timeout **TO** Time range support **TR** -Disabled **D** Engine type **ET** -------------- ----------- -------------------- ------------ -Safe search **SS** -------------- ----------- --------------------------------- -Weigth **W** -------------- ----------- --------------------------------- -Disabled **D** -------------- ----------- --------------------------------- -Show errors **DE** -============= =========== ================================= - -.. _configured engines: - -.. jinja:: searx - - .. flat-table:: Engines configured at built time (defaults) - :header-rows: 1 - :stub-columns: 2 - - * - Name (cfg) - - S - - Engine - - TO - - Categories - - P - - L - - SS - - D - - TR - - ET - - W - - D - - DE - - {% for name, mod in engines.items() %} - - * - {{name}} - - !{{mod.shortcut}} - - {{mod.__name__}} - - {{mod.timeout}} - - {{", ".join(mod.categories)}} - - {{(mod.paging and "y") or ""}} - - {{(mod.language_support and "y") or ""}} - - {{(mod.safesearch and "y") or ""}} - - {{(mod.disabled and "y") or ""}} - - {{(mod.time_range_support and "y") or ""}} - - {{mod.engine_type or ""}} - - {{mod.weight or 1 }} - - {{(mod.disabled and "y") or ""}} - - {{(mod.display_error_messages and "y") or ""}} - - {% endfor %} - diff --git a/docs/admin/engines/engine_settings.rst b/docs/admin/engines/engine_settings.rst new file mode 100644 index 000000000..9ca5620e2 --- /dev/null +++ b/docs/admin/engines/engine_settings.rst @@ -0,0 +1,76 @@ +.. _general engine settings: + +======================= +General Engine Settings +======================= + +.. sidebar:: Further reading .. + + - :ref:`engines-dev` + - :ref:`settings engine` + +Explanation of the :ref:`general engine configuration` shown in the table +:ref:`configured engines`. + +============= =========== ==================== ============ +:ref:`engine settings` :ref:`engine file` +------------------------- --------------------------------- +Name (cfg) Categories +------------------------- --------------------------------- +Engine .. Paging support **P** +------------------------- -------------------- ------------ +Shortcut **S** Language support **L** +Timeout **TO** Time range support **TR** +Disabled **D** Engine type **ET** +------------- ----------- -------------------- ------------ +Safe search **SS** +------------- ----------- --------------------------------- +Weigth **W** +------------- ----------- --------------------------------- +Disabled **D** +------------- ----------- --------------------------------- +Show errors **DE** +============= =========== ================================= + +.. _configured engines: + +.. jinja:: searx + + .. flat-table:: Engines configured at built time (defaults) + :header-rows: 1 + :stub-columns: 2 + + * - Name (cfg) + - S + - Engine + - TO + - Categories + - P + - L + - SS + - D + - TR + - ET + - W + - D + - DE + + {% for name, mod in engines.items() %} + + * - {{name}} + - !{{mod.shortcut}} + - {{mod.__name__}} + - {{mod.timeout}} + - {{", ".join(mod.categories)}} + - {{(mod.paging and "y") or ""}} + - {{(mod.language_support and "y") or ""}} + - {{(mod.safesearch and "y") or ""}} + - {{(mod.disabled and "y") or ""}} + - {{(mod.time_range_support and "y") or ""}} + - {{mod.engine_type or ""}} + - {{mod.weight or 1 }} + - {{(mod.disabled and "y") or ""}} + - {{(mod.display_error_messages and "y") or ""}} + + {% endfor %} + diff --git a/docs/admin/engines/index.rst b/docs/admin/engines/index.rst new file mode 100644 index 000000000..923b20a2b --- /dev/null +++ b/docs/admin/engines/index.rst @@ -0,0 +1,21 @@ +.. _engines and settings: + +================== +Engines & Settings +================== + +.. sidebar:: Further reading .. + + - :ref:`settings engine` + - :ref:`engine settings` & :ref:`engine file` + +.. toctree:: + :maxdepth: 1 + + engine_settings + recoll + sql-engines + command-line-engines + searx.engines.xpath + + diff --git a/docs/admin/index.rst b/docs/admin/index.rst index c708c4ffa..9be5b6a37 100644 --- a/docs/admin/index.rst +++ b/docs/admin/index.rst @@ -14,10 +14,10 @@ Administrator documentation installation-docker update-searx settings + engines/index api architecture filtron morty - engines plugins buildhosts diff --git a/docs/admin/settings.rst b/docs/admin/settings.rst index d0773467d..3baa83365 100644 --- a/docs/admin/settings.rst +++ b/docs/admin/settings.rst @@ -229,6 +229,7 @@ Engine settings .. sidebar:: Further reading .. + - :ref:`general engine settings` - :ref:`engines-dev` .. code:: yaml -- cgit v1.2.3 From f8442820890f723e0139db0058156452ec735c34 Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Thu, 3 Jun 2021 16:15:45 +0200 Subject: [docs] move blog article "private engines" to admin/engines/ Signed-off-by: Markus Heiser --- docs/admin/engines/index.rst | 1 + docs/admin/engines/private-engines.rst | 49 ++++++++++++++++++++++++++++++++++ 2 files changed, 50 insertions(+) create mode 100644 docs/admin/engines/private-engines.rst (limited to 'docs/admin') diff --git a/docs/admin/engines/index.rst b/docs/admin/engines/index.rst index 923b20a2b..02d6e604b 100644 --- a/docs/admin/engines/index.rst +++ b/docs/admin/engines/index.rst @@ -13,6 +13,7 @@ Engines & Settings :maxdepth: 1 engine_settings + private-engines recoll sql-engines command-line-engines diff --git a/docs/admin/engines/private-engines.rst b/docs/admin/engines/private-engines.rst new file mode 100644 index 000000000..cc6ab2565 --- /dev/null +++ b/docs/admin/engines/private-engines.rst @@ -0,0 +1,49 @@ +.. _private engines: + +============================ +Private Engines (``tokens``) +============================ + +Administrators might find themselves wanting to limit access to some of the +enabled engines on their instances. It might be because they do not want to +expose some private information through :ref:`offline engines`. Or they would +rather share engines only with their trusted friends or colleagues. + +To solve this issue the concept of *private engines* exists. + + +A new option was added to engines named `tokens`. It expects a list of +strings. If the user making a request presents one of the tokens of an engine, +they can access information about the engine and make search requests. + +Example configuration to restrict access to the Arch Linux Wiki engine: + +.. code:: yaml + + - name: arch linux wiki + engine: archlinux + shortcut: al + tokens: [ 'my-secret-token' ] + + +Unless a user has configured the right token, the engine is going +to be hidden from him/her. It is not going to be included in the +list of engines on the Preferences page and in the output of +`/config` REST API call. + +Tokens can be added to one's configuration on the Preferences page +under "Engine tokens". The input expects a comma separated list of +strings. + +The distribution of the tokens from the administrator to the users +is not carved in stone. As providing access to such engines +implies that the admin knows and trusts the user, we do not see +necessary to come up with a strict process. Instead, +we would like to add guidelines to the documentation of the feature. + + +Acknowledgment +============== + +This development was sponsored by `Search and Discovery Fund +`_ of `NLnet Foundation `_. -- cgit v1.2.3 From c647c8b7a594864a3cfbc881a9bb80efde9d73cf Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Thu, 3 Jun 2021 18:56:39 +0200 Subject: [docs] revision of the article "settings.yml" This patch is a marginal revision of the article "settings.yml", most changes are from normalizing the YAML syntax. Signed-off-by: Markus Heiser --- docs/admin/settings.rst | 226 ++++++++++++++++++++++++++---------------------- 1 file changed, 125 insertions(+), 101 deletions(-) (limited to 'docs/admin') diff --git a/docs/admin/settings.rst b/docs/admin/settings.rst index 3baa83365..fd99d9587 100644 --- a/docs/admin/settings.rst +++ b/docs/admin/settings.rst @@ -22,13 +22,14 @@ file. settings.yml location ===================== -First, searx will try to load settings.yml from these locations: +The initial ``settings.yml`` we be load from these locations: 1. the full path specified in the ``SEARX_SETTINGS_PATH`` environment variable. 2. ``/etc/searx/settings.yml`` If these files don't exist (or are empty or can't be read), searx uses the -:origin:`searx/settings.yml` file. +:origin:`searx/settings.yml` file. Read :ref:`settings use_default_settings` to +see how you can simplify your *user defined* ``settings.yml``. .. _settings global: @@ -42,17 +43,19 @@ Global Settings .. code:: yaml general: - debug : False # Debug mode, only for development - instance_name : "searxng" # displayed name - contact_url: False # mailto:contact@example.com + debug: false # Debug mode, only for development + instance_name: "searxng" # displayed name + contact_url: false # mailto:contact@example.com + +.. code:: yaml brand: - git_url: https://github.com/searxng/searxng - git_branch: master - issue_url: https://github.com/searxng/searxng/issues - docs_url: https://searxng/searxng.github.io/searxng - public_instances: https://searx.space - wiki_url: https://github.com/searxng/searxng/wiki + git_url: https://github.com/searxng/searxng + git_branch: master + issue_url: https://github.com/searxng/searxng/issues + docs_url: https://searxng/searxng.github.io/searxng + public_instances: https://searx.space + wiki_url: https://github.com/searxng/searxng/wiki ``debug`` : Allow a more detailed log if you run searx directly. Display *detailed* error @@ -68,10 +71,10 @@ Global Settings If you host your own documentation, change this URL. ``wiki_url``: - Link to your wiki (or ``False``) + Link to your wiki (or ``false``) ``twitter_url``: - Link to your tweets (or ``False``) + Link to your tweets (or ``false``) ``server:`` @@ -80,19 +83,19 @@ Global Settings .. code:: yaml server: - port : 8888 - bind_address : "127.0.0.1" # address to listen on - secret_key : "ultrasecretkey" # change this! - base_url : False # set custom base_url (or False) - image_proxy : False # proxying image results through searx - default_locale : "" # default interface locale - default_theme : oscar # ui theme + port: 8888 + bind_address: "127.0.0.1" # address to listen on + secret_key: "ultrasecretkey" # change this! + base_url: false # set custom base_url (or false) + image_proxy: false # proxying image results through searx + default_locale: "" # default interface locale + default_theme: oscar # ui theme default_http_headers: - X-Content-Type-Options : nosniff - X-XSS-Protection : 1; mode=block - X-Download-Options : noopen - X-Robots-Tag : noindex, nofollow - Referrer-Policy : no-referrer + X-Content-Type-Options : nosniff + X-XSS-Protection : 1; mode=block + X-Download-Options : noopen + X-Robots-Tag : noindex, nofollow + Referrer-Policy : no-referrer ``port`` & ``bind_address``: Port number and *bind address* of the searx web application if you run it @@ -125,26 +128,30 @@ Global Settings ``outgoing:`` ------------- +Communication with search engines. + .. code:: yaml - outgoing: # communication with search engines - request_timeout : 2.0 # default timeout in seconds, can be override by engine - # max_request_timeout: 10.0 # the maximum timeout in seconds - useragent_suffix : "" # informations like an email address to the administrator - pool_connections : 100 # Maximum number of allowable connections, or None for no limits. The default is 100. - pool_maxsize : 10 # Number of allowable keep-alive connections, or None to always allow. The default is 10. - enable_http2: True # See https://www.python-httpx.org/http2/ - # uncomment below section if you want to use a proxy - # proxies: - # all://: - # - http://proxy1:8080 - # - http://proxy2:8080 - # uncomment below section only if you have more than one network interface - # which can be the source of outgoing search requests - # source_ips: - # - 1.1.1.1 - # - 1.1.1.2 - # - fe80::/126 + outgoing: + request_timeout: 2.0 # default timeout in seconds, can be override by engine + max_request_timeout: 10.0 # the maximum timeout in seconds + useragent_suffix: "" # informations like an email address to the administrator + pool_connections: 100 # Maximum number of allowable connections, or null + # for no limits. The default is 100. + pool_maxsize: 10 # Number of allowable keep-alive connections, or null + # to always allow. The default is 10. + enable_http2: true # See https://www.python-httpx.org/http2/ + # uncomment below section if you want to use a proxy + # proxies: + # all://: + # - http://proxy1:8080 + # - http://proxy2:8080 + # uncomment below section only if you have more than one network interface + # which can be the source of outgoing search requests + # source_ips: + # - 1.1.1.1 + # - 1.1.1.2 + # - fe80::/126 ``request_timeout`` : @@ -180,8 +187,8 @@ Global Settings * ``[ 192.168.0.1, fe80::/126 ]`` ``retries`` : - Number of retry in case of an HTTP error. - On each retry, searx uses an different proxy and source ip. + Number of retry in case of an HTTP error. On each retry, searx uses an + different proxy and source ip. ``retry_on_http_error`` : Retry request on some HTTP status code. @@ -193,7 +200,7 @@ Global Settings * ``[403, 429]``: on HTTP status code 403 and 429. ``enable_http2`` : - Enable by default. Set to ``False`` to disable HTTP/2. + Enable by default. Set to ``false`` to disable HTTP/2. ``max_redirects`` : 30 by default. Maximum redirect before it is an error. @@ -205,18 +212,18 @@ Global Settings .. code:: yaml locales: - en : English - de : Deutsch - he : Hebrew - hu : Magyar - fr : Français - es : Español - it : Italiano - nl : Nederlands - ja : 日本語 (Japanese) - tr : Türkçe - ru : Russian - ro : Romanian + en: English + de: Deutsch + he: Hebrew + hu: Magyar + fr: Français + es: Español + it: Italiano + nl: Nederlands + ja: 日本語 (Japanese) + tr: Türkçe + ru: Russian + ro: Romanian ``locales`` : Locales codes and their names. Available translations of searx interface. @@ -232,33 +239,46 @@ Engine settings - :ref:`general engine settings` - :ref:`engines-dev` +In the code example below a *full fledged* example of a YAML setup from a dummy +engine is shown. Most of the options have a default value or even are optional. + .. code:: yaml - - name : bing - engine : bing - shortcut : bi - base_url : 'https://{language}.wikipedia.org/' - categories : general - timeout : 3.0 - api_key : 'apikey' - disabled : True - language : en_US - #enable_http: False - #enable_http2: False - #retries: 1 - #retry_on_http_error: True # or 403 or [404, 429] - #max_connections: 100 - #max_keepalive_connections: 10 - #keepalive_expiry: 5.0 - #proxies: - # http: - # - http://proxy1:8080 - # - http://proxy2:8080 - # https: - # - http://proxy1:8080 - # - http://proxy2:8080 - # - socks5://user:password@proxy3:1080 - # - socks5h://user:password@proxy4:1080 + - name: example engine + engine: example + shortcut: demo + base_url: 'https://{language}.example.com/' + categories: general + timeout: 3.0 + api_key: 'apikey' + disabled: false + language: en_US + tokens: [ 'my-secret-token' ] + weigth: 1 + display_error_messages: true + about: + website: https://example.com + wikidata_id: Q306656 + official_api_documentation: https://example.com/api-doc + use_official_api: true + require_api_key: true + results: HTML + enable_http: false + enable_http2: false + retries: 1 + retry_on_http_error: true # or 403 or [404, 429] + max_connections: 100 + max_keepalive_connections: 10 + keepalive_expiry: 5.0 + proxies: + http: + - http://proxy1:8080 + - http://proxy2:8080 + https: + - http://proxy1:8080 + - http://proxy2:8080 + - socks5://user:password@proxy3:1080 + - socks5h://user:password@proxy4:1080 ``name`` : Name that will be used across searx to define this engine. In settings, on @@ -298,10 +318,14 @@ Engine settings by using the full ISO code of language and country, like ``fr_FR``, ``en_US``, ``de_DE``. +``tokens`` : optional + A list of secret tokens to make this engine *private*, more details see + :ref:`private engines`. + ``weigth`` : default ``1`` Weighting of the results of this engine. -``display_error_messages`` : default ``True`` +``display_error_messages`` : default ``true`` When an engine returns an error, the message is displayed on the user interface. ``network``: optional @@ -321,7 +345,7 @@ Engine settings use_default_settings ==================== -.. sidebar:: ``use_default_settings: True`` +.. sidebar:: ``use_default_settings: true`` - :ref:`settings location` - :ref:`use_default_settings.yml` @@ -330,7 +354,7 @@ use_default_settings The user defined ``settings.yml`` is loaded from the :ref:`settings location` and can relied on the default configuration :origin:`searx/settings.yml` using: - ``use_default_settings: True`` + ``use_default_settings: true`` ``server:`` In the following example, the actual settings are the default settings defined @@ -339,22 +363,22 @@ and can relied on the default configuration :origin:`searx/settings.yml` using: .. code-block:: yaml - use_default_settings: True + use_default_settings: true server: - secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA" + secret_key: "ultrasecretkey" # change this! bind_address: "0.0.0.0" ``engines:`` - With ``use_default_settings: True``, each settings can be override in a + With ``use_default_settings: true``, each settings can be override in a similar way, the ``engines`` section is merged according to the engine ``name``. In this example, searx will load all the engine and the arch linux - wiki engine has a :ref:`token`: + wiki engine has a :ref:`token `: .. code-block:: yaml - use_default_settings: True + use_default_settings: true server: - secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA" + secret_key: "ultrasecretkey" # change this! engines: - name: arch linux wiki tokens: ['$ecretValue'] @@ -367,11 +391,11 @@ and can relied on the default configuration :origin:`searx/settings.yml` using: .. code-block:: yaml use_default_settings: - engines: - remove: - - google + engines: + remove: + - google server: - secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA" + secret_key: "ultrasecretkey" # change this! engines: - name: arch linux wiki tokens: ['$ecretValue'] @@ -383,12 +407,12 @@ and can relied on the default configuration :origin:`searx/settings.yml` using: .. code-block:: yaml use_default_settings: - engines: - keep_only: - - google - - duckduckgo + engines: + keep_only: + - google + - duckduckgo server: - secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA" + secret_key: "ultrasecretkey" # change this! engines: - name: google tokens: ['$ecretValue'] -- cgit v1.2.3 From 340d25b19b53582ee6465796d83eb419d2862dcc Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Fri, 4 Jun 2021 11:29:58 +0200 Subject: [docs] rename 'General Engine Settings' to 'Configured Engines' Signed-off-by: Markus Heiser --- docs/admin/engines/configured_engines.rst | 77 +++++++++++++++++++++++++++++++ docs/admin/engines/engine_settings.rst | 76 ------------------------------ docs/admin/engines/index.rst | 2 +- docs/admin/settings.rst | 2 +- 4 files changed, 79 insertions(+), 78 deletions(-) create mode 100644 docs/admin/engines/configured_engines.rst delete mode 100644 docs/admin/engines/engine_settings.rst (limited to 'docs/admin') diff --git a/docs/admin/engines/configured_engines.rst b/docs/admin/engines/configured_engines.rst new file mode 100644 index 000000000..af20e5611 --- /dev/null +++ b/docs/admin/engines/configured_engines.rst @@ -0,0 +1,77 @@ +.. _configured engines: + +================== +Configured Engines +================== + +.. sidebar:: Further reading .. + + - :ref:`engines-dev` + - :ref:`settings engine` + +Explanation of the :ref:`general engine configuration` shown in the table +:ref:`configured engines`. + +.. table:: The legend for the following table + :width: 100% + + ============= =========== ==================== ============ + :ref:`engine settings` :ref:`engine file` + ------------------------- --------------------------------- + Name (cfg) .. Categories + ------------- ----------- -------------------- ------------ + Engine .. Paging support **P** + ------------- ----------- -------------------- ------------ + Shortcut **S** Language support **L** + Timeout **TO** Time range support **TR** + Disabled **D** Engine type **ET** + ------------- ----------- -------------------- ------------ + Safe search **SS** + ------------- ----------- --------------------------------- + Weigth **W** + ------------- ----------- --------------------------------- + Disabled **D** + ------------- ----------- --------------------------------- + Show errors **DE** + ============= =========== ================================= + +.. jinja:: searx + + .. flat-table:: Engines configured at built time (defaults) + :header-rows: 1 + :stub-columns: 2 + + * - Name (cfg) + - S + - Engine + - TO + - Categories + - P + - L + - SS + - D + - TR + - ET + - W + - D + - DE + + {% for name, mod in engines.items() %} + + * - {{name}} + - !{{mod.shortcut}} + - {{mod.__name__}} + - {{mod.timeout}} + - {{", ".join(mod.categories)}} + - {{(mod.paging and "y") or ""}} + - {{(mod.language_support and "y") or ""}} + - {{(mod.safesearch and "y") or ""}} + - {{(mod.disabled and "y") or ""}} + - {{(mod.time_range_support and "y") or ""}} + - {{mod.engine_type or ""}} + - {{mod.weight or 1 }} + - {{(mod.disabled and "y") or ""}} + - {{(mod.display_error_messages and "y") or ""}} + + {% endfor %} + diff --git a/docs/admin/engines/engine_settings.rst b/docs/admin/engines/engine_settings.rst deleted file mode 100644 index 9ca5620e2..000000000 --- a/docs/admin/engines/engine_settings.rst +++ /dev/null @@ -1,76 +0,0 @@ -.. _general engine settings: - -======================= -General Engine Settings -======================= - -.. sidebar:: Further reading .. - - - :ref:`engines-dev` - - :ref:`settings engine` - -Explanation of the :ref:`general engine configuration` shown in the table -:ref:`configured engines`. - -============= =========== ==================== ============ -:ref:`engine settings` :ref:`engine file` -------------------------- --------------------------------- -Name (cfg) Categories -------------------------- --------------------------------- -Engine .. Paging support **P** -------------------------- -------------------- ------------ -Shortcut **S** Language support **L** -Timeout **TO** Time range support **TR** -Disabled **D** Engine type **ET** -------------- ----------- -------------------- ------------ -Safe search **SS** -------------- ----------- --------------------------------- -Weigth **W** -------------- ----------- --------------------------------- -Disabled **D** -------------- ----------- --------------------------------- -Show errors **DE** -============= =========== ================================= - -.. _configured engines: - -.. jinja:: searx - - .. flat-table:: Engines configured at built time (defaults) - :header-rows: 1 - :stub-columns: 2 - - * - Name (cfg) - - S - - Engine - - TO - - Categories - - P - - L - - SS - - D - - TR - - ET - - W - - D - - DE - - {% for name, mod in engines.items() %} - - * - {{name}} - - !{{mod.shortcut}} - - {{mod.__name__}} - - {{mod.timeout}} - - {{", ".join(mod.categories)}} - - {{(mod.paging and "y") or ""}} - - {{(mod.language_support and "y") or ""}} - - {{(mod.safesearch and "y") or ""}} - - {{(mod.disabled and "y") or ""}} - - {{(mod.time_range_support and "y") or ""}} - - {{mod.engine_type or ""}} - - {{mod.weight or 1 }} - - {{(mod.disabled and "y") or ""}} - - {{(mod.display_error_messages and "y") or ""}} - - {% endfor %} - diff --git a/docs/admin/engines/index.rst b/docs/admin/engines/index.rst index 02d6e604b..2be4d0e08 100644 --- a/docs/admin/engines/index.rst +++ b/docs/admin/engines/index.rst @@ -12,7 +12,7 @@ Engines & Settings .. toctree:: :maxdepth: 1 - engine_settings + configured_engines private-engines recoll sql-engines diff --git a/docs/admin/settings.rst b/docs/admin/settings.rst index fd99d9587..1fe083a97 100644 --- a/docs/admin/settings.rst +++ b/docs/admin/settings.rst @@ -236,7 +236,7 @@ Engine settings .. sidebar:: Further reading .. - - :ref:`general engine settings` + - :ref:`configured engines` - :ref:`engines-dev` In the code example below a *full fledged* example of a YAML setup from a dummy -- cgit v1.2.3 From e34dc7818f65f1c2fd2b9625e5446fd8278f1939 Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Fri, 4 Jun 2021 11:35:31 +0200 Subject: [docs] move 'settings.yml' into admin's 'Engines & Settings' Signed-off-by: Markus Heiser --- docs/admin/engines/index.rst | 1 + docs/admin/engines/settings.rst | 420 ++++++++++++++++++++++++++++++++++++++++ docs/admin/index.rst | 1 - docs/admin/settings.rst | 420 ---------------------------------------- 4 files changed, 421 insertions(+), 421 deletions(-) create mode 100644 docs/admin/engines/settings.rst delete mode 100644 docs/admin/settings.rst (limited to 'docs/admin') diff --git a/docs/admin/engines/index.rst b/docs/admin/engines/index.rst index 2be4d0e08..51707b792 100644 --- a/docs/admin/engines/index.rst +++ b/docs/admin/engines/index.rst @@ -12,6 +12,7 @@ Engines & Settings .. toctree:: :maxdepth: 1 + settings configured_engines private-engines recoll diff --git a/docs/admin/engines/settings.rst b/docs/admin/engines/settings.rst new file mode 100644 index 000000000..1fe083a97 --- /dev/null +++ b/docs/admin/engines/settings.rst @@ -0,0 +1,420 @@ +.. _settings.yml: + +================ +``settings.yml`` +================ + +This page describe the options possibilities of the :origin:`searx/settings.yml` +file. + +.. sidebar:: Further reading .. + + - :ref:`use_default_settings.yml` + - :ref:`search API` + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. _settings location: + +settings.yml location +===================== + +The initial ``settings.yml`` we be load from these locations: + +1. the full path specified in the ``SEARX_SETTINGS_PATH`` environment variable. +2. ``/etc/searx/settings.yml`` + +If these files don't exist (or are empty or can't be read), searx uses the +:origin:`searx/settings.yml` file. Read :ref:`settings use_default_settings` to +see how you can simplify your *user defined* ``settings.yml``. + + +.. _settings global: + +Global Settings +=============== + +``general:`` +------------ + +.. code:: yaml + + general: + debug: false # Debug mode, only for development + instance_name: "searxng" # displayed name + contact_url: false # mailto:contact@example.com + +.. code:: yaml + + brand: + git_url: https://github.com/searxng/searxng + git_branch: master + issue_url: https://github.com/searxng/searxng/issues + docs_url: https://searxng/searxng.github.io/searxng + public_instances: https://searx.space + wiki_url: https://github.com/searxng/searxng/wiki + +``debug`` : + Allow a more detailed log if you run searx directly. Display *detailed* error + messages in the browser too, so this must be deactivated in production. + +``contact_url``: + Contact ``mailto:`` address or WEB form. + +``git_url`` and ``git_branch``: + Changes this, to point to your searx fork (branch). + +``docs_url`` + If you host your own documentation, change this URL. + +``wiki_url``: + Link to your wiki (or ``false``) + +``twitter_url``: + Link to your tweets (or ``false``) + + +``server:`` +----------- + +.. code:: yaml + + server: + port: 8888 + bind_address: "127.0.0.1" # address to listen on + secret_key: "ultrasecretkey" # change this! + base_url: false # set custom base_url (or false) + image_proxy: false # proxying image results through searx + default_locale: "" # default interface locale + default_theme: oscar # ui theme + default_http_headers: + X-Content-Type-Options : nosniff + X-XSS-Protection : 1; mode=block + X-Download-Options : noopen + X-Robots-Tag : noindex, nofollow + Referrer-Policy : no-referrer + +``port`` & ``bind_address``: + Port number and *bind address* of the searx web application if you run it + directly using ``python searx/webapp.py``. Doesn't apply to searx running on + Apache or Nginx. + +``secret_key`` : + Used for cryptography purpose. + +``base_url`` : + The base URL where searx is deployed. Used to create correct inbound links. + +``image_proxy`` : + Allow your instance of searx of being able to proxy images. Uses memory space. + +``default_locale`` : + Searx interface language. If blank, the locale is detected by using the + browser language. If it doesn't work, or you are deploying a language + specific instance of searx, a locale can be defined using an ISO language + code, like ``fr``, ``en``, ``de``. + +``default_theme`` : + Name of the theme you want to use by default on your searx instance. + +.. _HTTP headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers + +``default_http_headers``: + Set additional HTTP headers, see `#755 `__ + +``outgoing:`` +------------- + +Communication with search engines. + +.. code:: yaml + + outgoing: + request_timeout: 2.0 # default timeout in seconds, can be override by engine + max_request_timeout: 10.0 # the maximum timeout in seconds + useragent_suffix: "" # informations like an email address to the administrator + pool_connections: 100 # Maximum number of allowable connections, or null + # for no limits. The default is 100. + pool_maxsize: 10 # Number of allowable keep-alive connections, or null + # to always allow. The default is 10. + enable_http2: true # See https://www.python-httpx.org/http2/ + # uncomment below section if you want to use a proxy + # proxies: + # all://: + # - http://proxy1:8080 + # - http://proxy2:8080 + # uncomment below section only if you have more than one network interface + # which can be the source of outgoing search requests + # source_ips: + # - 1.1.1.1 + # - 1.1.1.2 + # - fe80::/126 + + +``request_timeout`` : + Global timeout of the requests made to others engines in seconds. A bigger + timeout will allow to wait for answers from slow engines, but in consequence + will slow searx reactivity (the result page may take the time specified in the + timeout to load). Can be override by :ref:`settings engine` + +``useragent_suffix`` : + Suffix to the user-agent searx uses to send requests to others engines. If an + engine wish to block you, a contact info here may be useful to avoid that. + +``keepalive_expiry``: + Number of seconds to keep a connection in the pool. By default 5.0 seconds. + +.. _httpx proxies: https://www.python-httpx.org/advanced/#http-proxying + +``proxies`` : + Define one or more proxies you wish to use, see `httpx proxies`_. + If there are more than one proxy for one protocol (http, https), + requests to the engines are distributed in a round-robin fashion. + +``source_ips`` : + If you use multiple network interfaces, define from which IP the requests must + be made. Example: + + * ``0.0.0.0`` any local IPv4 address. + * ``::`` any local IPv6 address. + * ``192.168.0.1`` + * ``[ 192.168.0.1, 192.168.0.2 ]`` these two specific IP addresses + * ``fe80::60a2:1691:e5a2:ee1f`` + * ``fe80::60a2:1691:e5a2:ee1f/126`` all IP addresses in this network. + * ``[ 192.168.0.1, fe80::/126 ]`` + +``retries`` : + Number of retry in case of an HTTP error. On each retry, searx uses an + different proxy and source ip. + +``retry_on_http_error`` : + Retry request on some HTTP status code. + + Example: + + * ``true`` : on HTTP status code between 400 and 599. + * ``403`` : on HTTP status code 403. + * ``[403, 429]``: on HTTP status code 403 and 429. + +``enable_http2`` : + Enable by default. Set to ``false`` to disable HTTP/2. + +``max_redirects`` : + 30 by default. Maximum redirect before it is an error. + + +``locales:`` +------------ + +.. code:: yaml + + locales: + en: English + de: Deutsch + he: Hebrew + hu: Magyar + fr: Français + es: Español + it: Italiano + nl: Nederlands + ja: 日本語 (Japanese) + tr: Türkçe + ru: Russian + ro: Romanian + +``locales`` : + Locales codes and their names. Available translations of searx interface. + + +.. _settings engine: + +Engine settings +=============== + +.. sidebar:: Further reading .. + + - :ref:`configured engines` + - :ref:`engines-dev` + +In the code example below a *full fledged* example of a YAML setup from a dummy +engine is shown. Most of the options have a default value or even are optional. + +.. code:: yaml + + - name: example engine + engine: example + shortcut: demo + base_url: 'https://{language}.example.com/' + categories: general + timeout: 3.0 + api_key: 'apikey' + disabled: false + language: en_US + tokens: [ 'my-secret-token' ] + weigth: 1 + display_error_messages: true + about: + website: https://example.com + wikidata_id: Q306656 + official_api_documentation: https://example.com/api-doc + use_official_api: true + require_api_key: true + results: HTML + enable_http: false + enable_http2: false + retries: 1 + retry_on_http_error: true # or 403 or [404, 429] + max_connections: 100 + max_keepalive_connections: 10 + keepalive_expiry: 5.0 + proxies: + http: + - http://proxy1:8080 + - http://proxy2:8080 + https: + - http://proxy1:8080 + - http://proxy2:8080 + - socks5://user:password@proxy3:1080 + - socks5h://user:password@proxy4:1080 + +``name`` : + Name that will be used across searx to define this engine. In settings, on + the result page... + +``engine`` : + Name of the python file used to handle requests and responses to and from this + search engine. + +``shortcut`` : + Code used to execute bang requests (in this case using ``!bi`` or ``?bi``) + +``base_url`` : optional + Part of the URL that should be stable across every request. Can be useful to + use multiple sites using only one engine, or updating the site URL without + touching at the code. + +``categories`` : optional + Define in which categories this engine will be active. Most of the time, it is + defined in the code of the engine, but in a few cases it is useful, like when + describing multiple search engine using the same code. + +``timeout`` : optional + Timeout of the search with the current search engine. **Be careful, it will + modify the global timeout of searx.** + +``api_key`` : optional + In a few cases, using an API needs the use of a secret key. How to obtain them + is described in the file. + +``disabled`` : optional + To disable by default the engine, but not deleting it. It will allow the user + to manually activate it in the settings. + +``language`` : optional + If you want to use another language for a specific engine, you can define it + by using the full ISO code of language and country, like ``fr_FR``, ``en_US``, + ``de_DE``. + +``tokens`` : optional + A list of secret tokens to make this engine *private*, more details see + :ref:`private engines`. + +``weigth`` : default ``1`` + Weighting of the results of this engine. + +``display_error_messages`` : default ``true`` + When an engine returns an error, the message is displayed on the user interface. + +``network``: optional + Use the network configuration from another engine. + In addition, there are two default networks: + * ``ipv4`` set ``local_addresses`` to ``0.0.0.0`` (use only IPv4 local addresses) + * ``ipv6`` set ``local_addresses`` to ``::`` (use only IPv6 local addresses) + +.. note:: + + A few more options are possible, but they are pretty specific to some + engines, and so won't be described here. + + +.. _settings use_default_settings: + +use_default_settings +==================== + +.. sidebar:: ``use_default_settings: true`` + + - :ref:`settings location` + - :ref:`use_default_settings.yml` + - :origin:`/etc/searx/settings.yml ` + +The user defined ``settings.yml`` is loaded from the :ref:`settings location` +and can relied on the default configuration :origin:`searx/settings.yml` using: + + ``use_default_settings: true`` + +``server:`` + In the following example, the actual settings are the default settings defined + in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and + the ``bind_address``: + + .. code-block:: yaml + + use_default_settings: true + server: + secret_key: "ultrasecretkey" # change this! + bind_address: "0.0.0.0" + +``engines:`` + With ``use_default_settings: true``, each settings can be override in a + similar way, the ``engines`` section is merged according to the engine + ``name``. In this example, searx will load all the engine and the arch linux + wiki engine has a :ref:`token `: + + .. code-block:: yaml + + use_default_settings: true + server: + secret_key: "ultrasecretkey" # change this! + engines: + - name: arch linux wiki + tokens: ['$ecretValue'] + +``engines:`` / ``remove:`` + It is possible to remove some engines from the default settings. The following + example is similar to the above one, but searx doesn't load the the google + engine: + + .. code-block:: yaml + + use_default_settings: + engines: + remove: + - google + server: + secret_key: "ultrasecretkey" # change this! + engines: + - name: arch linux wiki + tokens: ['$ecretValue'] + +``engines:`` / ``keep_only:`` + As an alternative, it is possible to specify the engines to keep. In the + following example, searx has only two engines: + + .. code-block:: yaml + + use_default_settings: + engines: + keep_only: + - google + - duckduckgo + server: + secret_key: "ultrasecretkey" # change this! + engines: + - name: google + tokens: ['$ecretValue'] + - name: duckduckgo + tokens: ['$ecretValue'] diff --git a/docs/admin/index.rst b/docs/admin/index.rst index 9be5b6a37..3139db99c 100644 --- a/docs/admin/index.rst +++ b/docs/admin/index.rst @@ -13,7 +13,6 @@ Administrator documentation installation-apache installation-docker update-searx - settings engines/index api architecture diff --git a/docs/admin/settings.rst b/docs/admin/settings.rst deleted file mode 100644 index 1fe083a97..000000000 --- a/docs/admin/settings.rst +++ /dev/null @@ -1,420 +0,0 @@ -.. _settings.yml: - -================ -``settings.yml`` -================ - -This page describe the options possibilities of the :origin:`searx/settings.yml` -file. - -.. sidebar:: Further reading .. - - - :ref:`use_default_settings.yml` - - :ref:`search API` - -.. contents:: Contents - :depth: 2 - :local: - :backlinks: entry - -.. _settings location: - -settings.yml location -===================== - -The initial ``settings.yml`` we be load from these locations: - -1. the full path specified in the ``SEARX_SETTINGS_PATH`` environment variable. -2. ``/etc/searx/settings.yml`` - -If these files don't exist (or are empty or can't be read), searx uses the -:origin:`searx/settings.yml` file. Read :ref:`settings use_default_settings` to -see how you can simplify your *user defined* ``settings.yml``. - - -.. _settings global: - -Global Settings -=============== - -``general:`` ------------- - -.. code:: yaml - - general: - debug: false # Debug mode, only for development - instance_name: "searxng" # displayed name - contact_url: false # mailto:contact@example.com - -.. code:: yaml - - brand: - git_url: https://github.com/searxng/searxng - git_branch: master - issue_url: https://github.com/searxng/searxng/issues - docs_url: https://searxng/searxng.github.io/searxng - public_instances: https://searx.space - wiki_url: https://github.com/searxng/searxng/wiki - -``debug`` : - Allow a more detailed log if you run searx directly. Display *detailed* error - messages in the browser too, so this must be deactivated in production. - -``contact_url``: - Contact ``mailto:`` address or WEB form. - -``git_url`` and ``git_branch``: - Changes this, to point to your searx fork (branch). - -``docs_url`` - If you host your own documentation, change this URL. - -``wiki_url``: - Link to your wiki (or ``false``) - -``twitter_url``: - Link to your tweets (or ``false``) - - -``server:`` ------------ - -.. code:: yaml - - server: - port: 8888 - bind_address: "127.0.0.1" # address to listen on - secret_key: "ultrasecretkey" # change this! - base_url: false # set custom base_url (or false) - image_proxy: false # proxying image results through searx - default_locale: "" # default interface locale - default_theme: oscar # ui theme - default_http_headers: - X-Content-Type-Options : nosniff - X-XSS-Protection : 1; mode=block - X-Download-Options : noopen - X-Robots-Tag : noindex, nofollow - Referrer-Policy : no-referrer - -``port`` & ``bind_address``: - Port number and *bind address* of the searx web application if you run it - directly using ``python searx/webapp.py``. Doesn't apply to searx running on - Apache or Nginx. - -``secret_key`` : - Used for cryptography purpose. - -``base_url`` : - The base URL where searx is deployed. Used to create correct inbound links. - -``image_proxy`` : - Allow your instance of searx of being able to proxy images. Uses memory space. - -``default_locale`` : - Searx interface language. If blank, the locale is detected by using the - browser language. If it doesn't work, or you are deploying a language - specific instance of searx, a locale can be defined using an ISO language - code, like ``fr``, ``en``, ``de``. - -``default_theme`` : - Name of the theme you want to use by default on your searx instance. - -.. _HTTP headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers - -``default_http_headers``: - Set additional HTTP headers, see `#755 `__ - -``outgoing:`` -------------- - -Communication with search engines. - -.. code:: yaml - - outgoing: - request_timeout: 2.0 # default timeout in seconds, can be override by engine - max_request_timeout: 10.0 # the maximum timeout in seconds - useragent_suffix: "" # informations like an email address to the administrator - pool_connections: 100 # Maximum number of allowable connections, or null - # for no limits. The default is 100. - pool_maxsize: 10 # Number of allowable keep-alive connections, or null - # to always allow. The default is 10. - enable_http2: true # See https://www.python-httpx.org/http2/ - # uncomment below section if you want to use a proxy - # proxies: - # all://: - # - http://proxy1:8080 - # - http://proxy2:8080 - # uncomment below section only if you have more than one network interface - # which can be the source of outgoing search requests - # source_ips: - # - 1.1.1.1 - # - 1.1.1.2 - # - fe80::/126 - - -``request_timeout`` : - Global timeout of the requests made to others engines in seconds. A bigger - timeout will allow to wait for answers from slow engines, but in consequence - will slow searx reactivity (the result page may take the time specified in the - timeout to load). Can be override by :ref:`settings engine` - -``useragent_suffix`` : - Suffix to the user-agent searx uses to send requests to others engines. If an - engine wish to block you, a contact info here may be useful to avoid that. - -``keepalive_expiry``: - Number of seconds to keep a connection in the pool. By default 5.0 seconds. - -.. _httpx proxies: https://www.python-httpx.org/advanced/#http-proxying - -``proxies`` : - Define one or more proxies you wish to use, see `httpx proxies`_. - If there are more than one proxy for one protocol (http, https), - requests to the engines are distributed in a round-robin fashion. - -``source_ips`` : - If you use multiple network interfaces, define from which IP the requests must - be made. Example: - - * ``0.0.0.0`` any local IPv4 address. - * ``::`` any local IPv6 address. - * ``192.168.0.1`` - * ``[ 192.168.0.1, 192.168.0.2 ]`` these two specific IP addresses - * ``fe80::60a2:1691:e5a2:ee1f`` - * ``fe80::60a2:1691:e5a2:ee1f/126`` all IP addresses in this network. - * ``[ 192.168.0.1, fe80::/126 ]`` - -``retries`` : - Number of retry in case of an HTTP error. On each retry, searx uses an - different proxy and source ip. - -``retry_on_http_error`` : - Retry request on some HTTP status code. - - Example: - - * ``true`` : on HTTP status code between 400 and 599. - * ``403`` : on HTTP status code 403. - * ``[403, 429]``: on HTTP status code 403 and 429. - -``enable_http2`` : - Enable by default. Set to ``false`` to disable HTTP/2. - -``max_redirects`` : - 30 by default. Maximum redirect before it is an error. - - -``locales:`` ------------- - -.. code:: yaml - - locales: - en: English - de: Deutsch - he: Hebrew - hu: Magyar - fr: Français - es: Español - it: Italiano - nl: Nederlands - ja: 日本語 (Japanese) - tr: Türkçe - ru: Russian - ro: Romanian - -``locales`` : - Locales codes and their names. Available translations of searx interface. - - -.. _settings engine: - -Engine settings -=============== - -.. sidebar:: Further reading .. - - - :ref:`configured engines` - - :ref:`engines-dev` - -In the code example below a *full fledged* example of a YAML setup from a dummy -engine is shown. Most of the options have a default value or even are optional. - -.. code:: yaml - - - name: example engine - engine: example - shortcut: demo - base_url: 'https://{language}.example.com/' - categories: general - timeout: 3.0 - api_key: 'apikey' - disabled: false - language: en_US - tokens: [ 'my-secret-token' ] - weigth: 1 - display_error_messages: true - about: - website: https://example.com - wikidata_id: Q306656 - official_api_documentation: https://example.com/api-doc - use_official_api: true - require_api_key: true - results: HTML - enable_http: false - enable_http2: false - retries: 1 - retry_on_http_error: true # or 403 or [404, 429] - max_connections: 100 - max_keepalive_connections: 10 - keepalive_expiry: 5.0 - proxies: - http: - - http://proxy1:8080 - - http://proxy2:8080 - https: - - http://proxy1:8080 - - http://proxy2:8080 - - socks5://user:password@proxy3:1080 - - socks5h://user:password@proxy4:1080 - -``name`` : - Name that will be used across searx to define this engine. In settings, on - the result page... - -``engine`` : - Name of the python file used to handle requests and responses to and from this - search engine. - -``shortcut`` : - Code used to execute bang requests (in this case using ``!bi`` or ``?bi``) - -``base_url`` : optional - Part of the URL that should be stable across every request. Can be useful to - use multiple sites using only one engine, or updating the site URL without - touching at the code. - -``categories`` : optional - Define in which categories this engine will be active. Most of the time, it is - defined in the code of the engine, but in a few cases it is useful, like when - describing multiple search engine using the same code. - -``timeout`` : optional - Timeout of the search with the current search engine. **Be careful, it will - modify the global timeout of searx.** - -``api_key`` : optional - In a few cases, using an API needs the use of a secret key. How to obtain them - is described in the file. - -``disabled`` : optional - To disable by default the engine, but not deleting it. It will allow the user - to manually activate it in the settings. - -``language`` : optional - If you want to use another language for a specific engine, you can define it - by using the full ISO code of language and country, like ``fr_FR``, ``en_US``, - ``de_DE``. - -``tokens`` : optional - A list of secret tokens to make this engine *private*, more details see - :ref:`private engines`. - -``weigth`` : default ``1`` - Weighting of the results of this engine. - -``display_error_messages`` : default ``true`` - When an engine returns an error, the message is displayed on the user interface. - -``network``: optional - Use the network configuration from another engine. - In addition, there are two default networks: - * ``ipv4`` set ``local_addresses`` to ``0.0.0.0`` (use only IPv4 local addresses) - * ``ipv6`` set ``local_addresses`` to ``::`` (use only IPv6 local addresses) - -.. note:: - - A few more options are possible, but they are pretty specific to some - engines, and so won't be described here. - - -.. _settings use_default_settings: - -use_default_settings -==================== - -.. sidebar:: ``use_default_settings: true`` - - - :ref:`settings location` - - :ref:`use_default_settings.yml` - - :origin:`/etc/searx/settings.yml ` - -The user defined ``settings.yml`` is loaded from the :ref:`settings location` -and can relied on the default configuration :origin:`searx/settings.yml` using: - - ``use_default_settings: true`` - -``server:`` - In the following example, the actual settings are the default settings defined - in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and - the ``bind_address``: - - .. code-block:: yaml - - use_default_settings: true - server: - secret_key: "ultrasecretkey" # change this! - bind_address: "0.0.0.0" - -``engines:`` - With ``use_default_settings: true``, each settings can be override in a - similar way, the ``engines`` section is merged according to the engine - ``name``. In this example, searx will load all the engine and the arch linux - wiki engine has a :ref:`token `: - - .. code-block:: yaml - - use_default_settings: true - server: - secret_key: "ultrasecretkey" # change this! - engines: - - name: arch linux wiki - tokens: ['$ecretValue'] - -``engines:`` / ``remove:`` - It is possible to remove some engines from the default settings. The following - example is similar to the above one, but searx doesn't load the the google - engine: - - .. code-block:: yaml - - use_default_settings: - engines: - remove: - - google - server: - secret_key: "ultrasecretkey" # change this! - engines: - - name: arch linux wiki - tokens: ['$ecretValue'] - -``engines:`` / ``keep_only:`` - As an alternative, it is possible to specify the engines to keep. In the - following example, searx has only two engines: - - .. code-block:: yaml - - use_default_settings: - engines: - keep_only: - - google - - duckduckgo - server: - secret_key: "ultrasecretkey" # change this! - engines: - - name: google - tokens: ['$ecretValue'] - - name: duckduckgo - tokens: ['$ecretValue'] -- cgit v1.2.3 From 27e5a7437a7af51f38a076a86deb9115c9a345b0 Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Fri, 4 Jun 2021 11:48:48 +0200 Subject: [docs] rename 'Recoll' to 'Recoll Engine' Signed-off-by: Markus Heiser --- docs/admin/engines/recoll.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) (limited to 'docs/admin') diff --git a/docs/admin/engines/recoll.rst b/docs/admin/engines/recoll.rst index cba2e81f7..9b198db7e 100644 --- a/docs/admin/engines/recoll.rst +++ b/docs/admin/engines/recoll.rst @@ -1,17 +1,17 @@ .. _engine recoll: -====== -Recoll -====== +============= +Recoll Engine +============= .. sidebar:: info - `Recoll `_ - `recoll-webui `_ + - :origin:`searx/engines/recoll.py` -Recoll_ is a desktop full-text search tool based on Xapian. By itself Recoll_ -does not offer web or API access, this can be achieved using recoll-webui_ - +Recoll_ is a desktop full-text search tool based on Xapian. By itself Recoll_ +does not offer WEB or API access, this can be achieved using recoll-webui_ Configuration -- cgit v1.2.3 From c5ccd50ffaa4254b6ad9902f6eb89c34e4f22f01 Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Fri, 4 Jun 2021 13:09:26 +0200 Subject: [docs] revision of the blog article about local search engines The blog article 'Query your local search engines' has been renamed 'Local Search Engines', revised and moved into admin's chapter 'Engine & Settings'. Signed-off-by: Markus Heiser --- docs/admin/engines/index.rst | 3 +- docs/admin/engines/search-indexer-engines.rst | 136 ++++++++++++++++++++++++++ 2 files changed, 137 insertions(+), 2 deletions(-) create mode 100644 docs/admin/engines/search-indexer-engines.rst (limited to 'docs/admin') diff --git a/docs/admin/engines/index.rst b/docs/admin/engines/index.rst index 51707b792..80f4120a5 100644 --- a/docs/admin/engines/index.rst +++ b/docs/admin/engines/index.rst @@ -17,7 +17,6 @@ Engines & Settings private-engines recoll sql-engines + search-indexer-engines command-line-engines searx.engines.xpath - - diff --git a/docs/admin/engines/search-indexer-engines.rst b/docs/admin/engines/search-indexer-engines.rst new file mode 100644 index 000000000..8d7c10d67 --- /dev/null +++ b/docs/admin/engines/search-indexer-engines.rst @@ -0,0 +1,136 @@ +==================== +Local Search Engines +==================== + +.. sidebar:: further read + + - `Comparison to alternatives + `_ + +Administrators might find themselves wanting to integrate locally running search +engines. The following ones are supported for now: + +* `Elasticsearch`_ +* `Meilisearch`_ +* `Solr`_ + +Each search engine is powerful, capable of full-text search. All of the engines +above are added to ``settings.yml`` just commented out, as you have to +``base_url`` for all them. + +Please note that if you are not using HTTPS to access these engines, you have to enable +HTTP requests by setting ``enable_http`` to ``True``. + +Futhermore, if you do not want to expose these engines on a public instance, you +can still add them and limit the access by setting ``tokens`` as described in +section :ref:`private engines`. + +.. _engine meilisearch: + +MeiliSearch +=========== + +.. sidebar:: info + + - :origin:`meilisearch.py ` + - `MeiliSearch `_ + - `MeiliSearch Documentation `_ + - `Install MeiliSearch + `_ + +MeiliSearch_ is aimed at individuals and small companies. It is designed for +small-scale (less than 10 million documents) data collections. E.g. it is great +for storing web pages you have visited and searching in the contents later. + +The engine supports faceted search, so you can search in a subset of documents +of the collection. Furthermore, you can search in MeiliSearch_ instances that +require authentication by setting ``auth_token``. + +Here is a simple example to query a Meilisearch instance: + +.. code:: yaml + + - name: meilisearch + engine: meilisearch + shortcut: mes + base_url: http://localhost:7700 + index: my-index + enable_http: true + + +.. _engine elasticsearch: + +Elasticsearch +============= + +.. sidebar:: info + + - :origin:`elasticsearch.py ` + - `Elasticsearch `_ + - `Elasticsearch Guide + `_ + - `Install Elasticsearch + `_ + +Elasticsearch_ supports numerous ways to query the data it is storing. At the +moment the engine supports the most popular search methods (``query_type``): + +- ``match``, +- ``simple_query_string``, +- ``term`` and +- ``terms``. + +If none of the methods fit your use case, you can select ``custom`` query type +and provide the JSON payload to submit to Elasticsearch in +``custom_query_json``. + +The following is an example configuration for an Elasticsearch_ instance with +authentication configured to read from ``my-index`` index. + +.. code:: yaml + + - name: elasticsearch + shortcut: es + engine: elasticsearch + base_url: http://localhost:9200 + username: elastic + password: changeme + index: my-index + query_type: match + # custom_query_json: '{ ... }' + enable_http: true + +.. _engine solr: + +Solr +==== + +.. sidebar:: info + + - :origin:`solr.py ` + - `Solr `_ + - `Solr Resources `_ + - `Install Solr `_ + +Solr_ is a popular search engine based on Lucene, just like Elasticsearch_. But +instead of searching in indices, you can search in collections. + +This is an example configuration for searching in the collection +``my-collection`` and get the results in ascending order. + +.. code:: yaml + + - name: solr + engine: solr + shortcut: slr + base_url: http://localhost:8983 + collection: my-collection + sort: asc + enable_http: true + + +Acknowledgment +============== + +This development was sponsored by `Search and Discovery Fund +`_ of `NLnet Foundation `_. -- cgit v1.2.3 From 6f7b0d72c084845ea073a82a357c5e99cd41a85f Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Fri, 4 Jun 2021 15:02:47 +0200 Subject: [docs] revision of the section 'Command Line Engines' Signed-off-by: Markus Heiser --- docs/admin/engines/command-line-engines.rst | 95 +++++++++++++++++------------ 1 file changed, 55 insertions(+), 40 deletions(-) (limited to 'docs/admin') diff --git a/docs/admin/engines/command-line-engines.rst b/docs/admin/engines/command-line-engines.rst index b4568f7d5..e9535e74f 100644 --- a/docs/admin/engines/command-line-engines.rst +++ b/docs/admin/engines/command-line-engines.rst @@ -1,41 +1,64 @@ .. _engine command: -============================== -Fetch results from commandline -============================== +==================== +Command Line Engines +==================== -Previously, with searx you could search over the Internet on other people's -computers. Now it is possible to fetch results from your local machine without -connecting to any networks from the same graphical user interface. +.. sidebar:: info -.. _command line engines: + - :origin:`command.py ` + - :ref:`offline engines` -Command line engines -==================== +With *command engines* administrators can run engines to integrate arbitrary +shell commands. + +When creating and enabling a ``command`` engine on a public instance, you must +be careful to avoid leaking private data. The easiest solution is to limit the +access by setting ``tokens`` as described in section :ref:`private engines`. + +The engine base is flexible. Only your imagination can limit the power of this +engine (and maybe security concerns). The following options are available: + +``command``: + A comma separated list of the elements of the command. A special token + ``{{QUERY}}`` tells where to put the search terms of the user. Example: + + .. code:: yaml + + ['ls', '-l', '-h', '{{QUERY}}'] + +``delimiter``: + A mapping containing a delimiter ``char`` and the *titles* of each element in + ``keys``. -In :pull-searx:`2128` a new type of engine has been introduced called ``command``. -This engine lets administrators add engines which run arbitrary shell commands -and show its output on the web UI of searx. +``parse_regex``: + A dict containing the regular expressions for each result key. -When creating and enabling a ``command`` engine on a public searx instance, -you must be careful to avoid leaking private data. The easiest solution -is to add tokens to the engine. Thus, only those who have the appropriate token -can retrieve results from the it. +``query_type``: -The engine base is flexible. Only your imagination can limit the power of this engine. (And -maybe security concerns.) The following options are available: + The expected type of user search terms. Possible values: ``path`` and + ``enum``. -* ``command``: A comma separated list of the elements of the command. A special token {{QUERY}} tells searx where to put the search terms of the user. Example: ``['ls', '-l', '-h', '{{QUERY}}']`` -* ``delimiter``: A dict containing a delimiter char and the "titles" of each element in keys. -* ``parse_regex``: A dict containing the regular expressions for each result key. -* ``query_type``: The expected type of user search terms. Possible values: ``path`` and ``enum``. ``path`` checks if the uesr provided path is inside the working directory. If not the query is not executed. ``enum`` is a list of allowed search terms. If the user submits something which is not included in the list, the query returns an error. -* ``query_enum``: A list containing allowed search terms if ``query_type`` is set to ``enum``. -* ``working_dir``: The directory where the command has to be executed. Default: ``.`` -* ``result_separator``: The character that separates results. Default: ``\n`` - + ``path``: + Checks if the user provided path is inside the working directory. If not, + the query is not executed. -The example engine below can be used to find files with a specific name in the configured -working directory. + ``enum``: + Is a list of allowed search terms. If the user submits something which is + not included in the list, the query returns an error. + +``query_enum``: + A list containing allowed search terms if ``query_type`` is set to ``enum``. + +``working_dir``: + + The directory where the command has to be executed. Default: ``./`` + +``result_separator``: + The character that separates results. Default: ``\n`` + +The example engine below can be used to find files with a specific name in the +configured working directory: .. code:: yaml @@ -49,16 +72,8 @@ working directory. keys: ['line'] -Next steps -========== - -In the next milestone, support for local search engines and indexers (e.g. Elasticsearch) -are going to be added. This way, you will be able to query your own databases/indexers. - -Acknowledgement -=============== - -This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ . +Acknowledgment +============== -.. _Search and Discovery Fund: https://nlnet.nl/discovery -.. _NLnet Foundation: https://nlnet.nl/ +This development was sponsored by `Search and Discovery Fund +`_ of `NLnet Foundation `_. -- cgit v1.2.3