summaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
Diffstat (limited to 'docs')
-rw-r--r--docs/_themes/searx/static/searx.css12
-rw-r--r--docs/admin/arch_public.dot8
-rw-r--r--docs/admin/architecture.rst12
-rw-r--r--docs/admin/buildhosts.rst59
-rw-r--r--docs/admin/filtron.rst224
-rw-r--r--docs/admin/index.rst5
-rw-r--r--docs/admin/installation-apache.rst94
-rw-r--r--docs/admin/installation-nginx.rst141
-rw-r--r--docs/admin/installation.rst324
-rw-r--r--docs/admin/morty.rst3
-rw-r--r--docs/admin/settings.rst8
-rw-r--r--docs/blog/index.rst3
-rw-r--r--docs/conf.py4
-rw-r--r--docs/dev/contribution_guide.rst5
-rw-r--r--docs/dev/index.rst3
-rw-r--r--docs/dev/makefile.rst14
-rw-r--r--docs/dev/reST.rst7
-rw-r--r--docs/index.rst18
-rw-r--r--docs/user/index.rst3
-rw-r--r--docs/user/own-instance.rst6
-rw-r--r--docs/user/public_instances.rst3
-rw-r--r--docs/utils/filtron.sh.rst75
-rw-r--r--docs/utils/index.rst53
-rw-r--r--docs/utils/morty.sh.rst39
-rw-r--r--docs/utils/searx.sh.rst73
25 files changed, 832 insertions, 364 deletions
diff --git a/docs/_themes/searx/static/searx.css b/docs/_themes/searx/static/searx.css
index d6a664f0f..ae742a4e0 100644
--- a/docs/_themes/searx/static/searx.css
+++ b/docs/_themes/searx/static/searx.css
@@ -33,7 +33,7 @@ p.sidebar-title, .sidebar p {
/* admonitions
*/
-div.admonition, div.topic {
+div.admonition, div.topic, div.toctree-wrapper {
background-color: #fafafa;
margin: 8px 0px;
padding: 1em;
@@ -42,6 +42,16 @@ div.admonition, div.topic {
border-right: none;
border-bottom: none;
border-left: 5pt solid #ccc;
+ list-style-type: disclosure-closed;
+}
+
+div.toctree-wrapper p.caption {
+ font-weight: normal;
+ font-size: 24px;
+ margin: 0 0 10px 0;
+ padding: 0;
+ line-height: 1;
+ display: inline;
}
p.admonition-title:after {
diff --git a/docs/admin/arch_public.dot b/docs/admin/arch_public.dot
index a46c96de3..5018225c3 100644
--- a/docs/admin/arch_public.dot
+++ b/docs/admin/arch_public.dot
@@ -4,11 +4,11 @@ digraph G {
edge [fontname="Sans"];
browser [label="Browser", shape=Mdiamond];
- rp [label="Reverse Proxy", href="url to configure reverse proxy"];
- filtron [label="Filtron", href="https://github.com/asciimoo/filtron"];
- morty [label="Morty", href="https://github.com/asciimoo/morty"];
+ rp [label="Reverse Proxy", href="https://asciimoo.github.io/searx/utils/filtron.sh.html#public-reverse-proxy"];
+ filtron [label="Filtron", href="https://asciimoo.github.io/searx/utils/filtron.sh.html"];
+ morty [label="Morty", href="https://asciimoo.github.io/searx/utils/morty.sh.html"];
static [label="Static files", href="url to configure static files"];
- uwsgi [label="uwsgi", href="url to configure uwsgi"]
+ uwsgi [label="uwsgi", href="https://asciimoo.github.io/searx/utils/searx.sh.html"]
searx1 [label="Searx #1"];
searx2 [label="Searx #2"];
searx3 [label="Searx #3"];
diff --git a/docs/admin/architecture.rst b/docs/admin/architecture.rst
index 7064a294b..fbc3bf483 100644
--- a/docs/admin/architecture.rst
+++ b/docs/admin/architecture.rst
@@ -4,17 +4,19 @@
Architecture
============
-.. sidebar:: Needs work!
+.. sidebar:: Tooling box
- This article needs some work / Searx is a collaborative effort. If you have
- any contribution, feel welcome to send us your :pull:`PR <../pulls>`, see
- :ref:`how to contribute`.
+ - :ref:`searx & uwsgi <searx.sh>`
+ - :ref:`filtron <filtron.sh>`
+ - :ref:`reverse proxy`
+ - :ref:`morty <morty.sh>`
Herein you will find some hints and suggestions about typical architectures of
searx infrastructures.
We start with a contribution from :pull:`@dalf <1776#issuecomment-567917320>`.
-It shows a *reference* setup for public searx instances.
+It shows a *reference* setup for public searx instances which can build up and
+maintained by the scripts from our :ref:`toolboxing`.
.. _arch public:
diff --git a/docs/admin/buildhosts.rst b/docs/admin/buildhosts.rst
index 5260da033..27784764f 100644
--- a/docs/admin/buildhosts.rst
+++ b/docs/admin/buildhosts.rst
@@ -4,13 +4,18 @@
Buildhosts
==========
+To get best results from build, its recommend to install additional packages
+on build hosts.
+
.. sidebar:: This article needs some work
If you have any contribution send us your :pull:`PR <../pulls>`, see
:ref:`how to contribute`.
-To get best results from build, its recommend to install additional packages
-on build hosts.
+.. contents:: Contents
+ :depth: 2
+ :local:
+ :backlinks: entry
.. _docs build:
@@ -35,8 +40,17 @@ processing additional packages are needed. The XeTeX_ needed not only for PDF
creation, its also needed for :ref:`math` when HTML output is build.
To be able to do :ref:`sphinx:math-support` without CDNs, the math are rendered
-as images (``sphinx.ext.imgmath`` extension). If your docs build (``make
-docs``) shows warnings like this::
+as images (``sphinx.ext.imgmath`` extension).
+
+Here is the extract from the :origin:`docs/conf.py` file, setting math renderer
+to ``imgmath``:
+
+.. literalinclude:: ../conf.py
+ :language: python
+ :start-after: # sphinx.ext.imgmath setup
+ :end-before: # sphinx.ext.imgmath setup END
+
+If your docs build (``make docs``) shows warnings like this::
WARNING: dot(1) not found, for better output quality install \
graphviz from http://www.graphviz.org
@@ -47,8 +61,6 @@ docs``) shows warnings like this::
you need to install additional packages on your build host, to get better HTML
output.
-.. _system requirements:
-
.. tabs::
.. group-tab:: Ubuntu / debian
@@ -94,10 +106,35 @@ For PDF output you also need:
texlive-collection-fontsrecommended texlive-collection-latex \
dejavu-sans-fonts dejavu-serif-fonts dejavu-sans-mono-fonts
-.. _system requirements END:
+.. _sh lint:
-.. literalinclude:: ../conf.py
- :language: python
- :start-after: # sphinx.ext.imgmath setup
- :end-before: # sphinx.ext.imgmath setup END
+Lint shell scripts
+==================
+
+.. _ShellCheck: https://github.com/koalaman/shellcheck
+
+To lint shell scripts, we use ShellCheck_ - A shell script static analysis tool.
+
+.. SNIP sh lint requirements
+
+.. tabs::
+
+ .. group-tab:: Ubuntu / debian
+
+ .. code-block:: sh
+
+ $ sudo apt install shellcheck
+
+ .. group-tab:: Arch Linux
+
+ .. code-block:: sh
+
+ $ sudo pacman -S shellcheck
+
+ .. group-tab:: Fedora / RHEL
+
+ .. code-block:: sh
+
+ $ sudo dnf install ShellCheck
+.. SNAP sh lint requirements
diff --git a/docs/admin/filtron.rst b/docs/admin/filtron.rst
index 07dcb9bc5..2bc663411 100644
--- a/docs/admin/filtron.rst
+++ b/docs/admin/filtron.rst
@@ -1,18 +1,48 @@
+
+.. _searx_filtron:
+
==========================
How to protect an instance
==========================
+.. sidebar:: further reading
+
+ - :ref:`filtron.sh`
+
+.. contents:: Contents
+ :depth: 2
+ :local:
+ :backlinks: entry
+
+.. _filtron: https://github.com/asciimoo/filtron
+
Searx depens on external search services. To avoid the abuse of these services
it is advised to limit the number of requests processed by searx.
-An application firewall, ``filtron`` solves exactly this problem. Information
-on how to install it can be found at the `project page of filtron
-<https://github.com/asciimoo/filtron>`__.
+An application firewall, filtron_ solves exactly this problem. Filtron is just
+a middleware between your web server (nginx, apache, ...) and searx, we describe
+such infratructures in chapter: :ref:`architecture`.
+
+
+filtron & go
+============
+
+.. _Go: https://golang.org/
+.. _filtron README: https://github.com/asciimoo/filtron/blob/master/README.md
+
+Filtron needs Go_ installed. If Go_ is preinstalled, filtron_ is simply
+installed by ``go get`` package management (see `filtron README`_). If you use
+filtron as middleware, a more isolated setup is recommended. To simplify such
+an installation and the maintenance of, use our script :ref:`filtron.sh`.
Sample configuration of filtron
===============================
+.. sidebar:: Tooling box
+
+ - :origin:`/etc/filtron/rules.json <utils/templates/etc/filtron/rules.json>`
+
An example configuration can be find below. This configuration limits the access
of:
@@ -24,100 +54,100 @@ of:
.. code:: json
- [{
- "name":"search request",
- "filters":[
- "Param:q",
- "Path=^(/|/search)$"
- ],
- "interval":"<time-interval-in-sec (int)>",
- "limit":"<max-request-number-in-interval (int)>",
- "subrules":[
- {
- "name":"roboagent limit",
- "interval":"<time-interval-in-sec (int)>",
- "limit":"<max-request-number-in-interval (int)>",
- "filters":[
- "Header:User-Agent=(curl|cURL|Wget|python-requests|Scrapy|FeedFetcher|Go-http-client)"
- ],
- "actions":[
- {
- "name":"block",
- "params":{
- "message":"Rate limit exceeded"
- }
- }
- ]
- },
- {
- "name":"botlimit",
- "limit":0,
- "stop":true,
- "filters":[
- "Header:User-Agent=(Googlebot|bingbot|Baiduspider|yacybot|YandexMobileBot|YandexBot|Yahoo! Slurp|MJ12bot|AhrefsBot|archive.org_bot|msnbot|MJ12bot|SeznamBot|linkdexbot|Netvibes|SMTBot|zgrab|James BOT)"
- ],
- "actions":[
- {
- "name":"block",
- "params":{
- "message":"Rate limit exceeded"
- }
- }
- ]
- },
- {
- "name":"IP limit",
- "interval":"<time-interval-in-sec (int)>",
- "limit":"<max-request-number-in-interval (int)>",
- "stop":true,
- "aggregations":[
- "Header:X-Forwarded-For"
- ],
- "actions":[
- {
- "name":"block",
- "params":{
- "message":"Rate limit exceeded"
- }
- }
- ]
- },
- {
- "name":"rss/json limit",
- "interval":"<time-interval-in-sec (int)>",
- "limit":"<max-request-number-in-interval (int)>",
- "stop":true,
- "filters":[
- "Param:format=(csv|json|rss)"
- ],
- "actions":[
- {
- "name":"block",
- "params":{
- "message":"Rate limit exceeded"
- }
- }
- ]
- },
- {
- "name":"useragent limit",
- "interval":"<time-interval-in-sec (int)>",
- "limit":"<max-request-number-in-interval (int)>",
- "aggregations":[
- "Header:User-Agent"
- ],
- "actions":[
- {
- "name":"block",
- "params":{
- "message":"Rate limit exceeded"
- }
- }
- ]
- }
- ]
- }]
-
+ [
+ { "name": "search request",
+ "filters": [
+ "Param:q",
+ "Path=^(/|/search)$"
+ ],
+ "interval": "<time-interval-in-sec (int)>",
+ "limit": "<max-request-number-in-interval (int)>",
+ "subrules": [
+ {
+ "name": "roboagent limit",
+ "interval": "<time-interval-in-sec (int)>",
+ "limit": "<max-request-number-in-interval (int)>",
+ "filters": [
+ "Header:User-Agent=(curl|cURL|Wget|python-requests|Scrapy|FeedFetcher|Go-http-client)"
+ ],
+ "actions": [
+ { "name": "log"},
+ { "name": "block",
+ "params": {
+ "message": "Rate limit exceeded"
+ }
+ }
+ ]
+ },
+ {
+ "name": "botlimit",
+ "limit": 0,
+ "stop": true,
+ "filters": [
+ "Header:User-Agent=(Googlebot|bingbot|Baiduspider|yacybot|YandexMobileBot|YandexBot|Yahoo! Slurp|MJ12bot|AhrefsBot|archive.org_bot|msnbot|MJ12bot|SeznamBot|linkdexbot|Netvibes|SMTBot|zgrab|James BOT)"
+ ],
+ "actions": [
+ { "name": "log"},
+ { "name": "block",
+ "params": {
+ "message": "Rate limit exceeded"
+ }
+ }
+ ]
+ },
+ {
+ "name": "IP limit",
+ "interval": "<time-interval-in-sec (int)>",
+ "limit": "<max-request-number-in-interval (int)>",
+ "stop": true,
+ "aggregations": [
+ "Header:X-Forwarded-For"
+ ],
+ "actions": [
+ { "name": "log"},
+ { "name": "block",
+ "params": {
+ "message": "Rate limit exceeded"
+ }
+ }
+ ]
+ },
+ {
+ "name": "rss/json limit",
+ "interval": "<time-interval-in-sec (int)>",
+ "limit": "<max-request-number-in-interval (int)>",
+ "stop": true,
+ "filters": [
+ "Param:format=(csv|json|rss)"
+ ],
+ "actions": [
+ { "name": "log"},
+ { "name": "block",
+ "params": {
+ "message": "Rate limit exceeded"
+ }
+ }
+ ]
+ },
+ {
+ "name": "useragent limit",
+ "interval": "<time-interval-in-sec (int)>",
+ "limit": "<max-request-number-in-interval (int)>",
+ "aggregations": [
+ "Header:User-Agent"
+ ],
+ "actions": [
+ { "name": "log"},
+ { "name": "block",
+ "params": {
+ "message": "Rate limit exceeded"
+ }
+ }
+ ]
+ }
+ ]
+ }
+ ]
Route request through filtron
diff --git a/docs/admin/index.rst b/docs/admin/index.rst
index b3c7f5119..4190bf24b 100644
--- a/docs/admin/index.rst
+++ b/docs/admin/index.rst
@@ -3,9 +3,12 @@ Administrator documentation
===========================
.. toctree::
- :maxdepth: 1
+ :maxdepth: 2
+ :caption: Contents
installation
+ installation-nginx
+ installation-apache
settings
api
architecture
diff --git a/docs/admin/installation-apache.rst b/docs/admin/installation-apache.rst
new file mode 100644
index 000000000..7f92e47d5
--- /dev/null
+++ b/docs/admin/installation-apache.rst
@@ -0,0 +1,94 @@
+.. _installation apache:
+
+===================
+Install with apache
+===================
+
+.. sidebar:: public HTTP servers
+
+ On public searx instances use an application firewall (:ref:`filtron
+ <filtron.sh>`).
+
+.. contents:: Contents
+ :depth: 2
+ :local:
+ :backlinks: entry
+
+Add wsgi mod
+============
+
+.. tabs::
+
+ .. group-tab:: Ubuntu / debian
+
+ .. code:: sh
+
+ sudo -H apt-get install libapache2-mod-uwsgi
+ sudo -H a2enmod uwsgi
+
+Add this configuration in the file ``/etc/apache2/apache2.conf``. To limit
+acces to your intranet replace ``Allow from all`` directive and replace
+``192.168.0.0/16`` with your subnet IP/class.
+
+.. _inranet apache site:
+
+Note that if your instance of searx is not at the root, you should change
+``<Location />`` by the location of your instance, like ``<Location /searx>``:
+
+.. code:: apache
+
+ # CustomLog /dev/null combined
+
+ <IfModule mod_uwsgi.c>
+
+ <Location />
+
+ Options FollowSymLinks Indexes
+ SetHandler uwsgi-handler
+ uWSGISocket /run/uwsgi/app/searx/socket
+
+ Order deny,allow
+ Deny from all
+ # Allow from fd00::/8 192.168.0.0/16 fe80::/10 127.0.0.0/8 ::1
+ Allow from all
+
+ </Location>
+
+ </IfModule>
+
+Enable apache mod_uwsgi and restart apache:
+
+.. tabs::
+
+ .. group-tab:: Ubuntu / debian
+
+ .. code:: sh
+
+ a2enmod uwsgi
+ sudo -H systemctl restart apache2
+
+disable logs
+============
+
+For better privacy you can disable Apache logs. Go back to
+``/etc/apache2/apache2.conf`` :ref:`[example] <inranet apache site>` and above
+``<Location />`` activate directive:
+
+.. code:: apache
+
+ CustomLog /dev/null combined
+
+Restart apache:
+
+.. tabs::
+
+ .. group-tab:: Ubuntu / debian
+
+ .. code:: sh
+
+ sudo -H systemctl restart apache2
+
+.. warning::
+
+ You can only disable logs for the whole (virtual) server not for a specific
+ path.
diff --git a/docs/admin/installation-nginx.rst b/docs/admin/installation-nginx.rst
new file mode 100644
index 000000000..37d3e7532
--- /dev/null
+++ b/docs/admin/installation-nginx.rst
@@ -0,0 +1,141 @@
+.. _installation nginx:
+
+==================
+Install with nginx
+==================
+
+.. sidebar:: public HTTP servers
+
+ On public searx instances use an application firewall (:ref:`filtron
+ <filtron.sh>`).
+
+.. contents:: Contents
+ :depth: 2
+ :local:
+ :backlinks: entry
+
+If nginx is not installed (uwsgi will not work with the package
+nginx-light):
+
+.. tabs::
+
+ .. group-tab:: Ubuntu / debian
+
+ .. code:: sh
+
+ sudo -H apt-get install nginx
+
+Hosted at ``/``
+===============
+
+Create the configuration file ``/etc/nginx/sites-available/searx`` with this
+content:
+
+.. code:: nginx
+
+ server {
+ listen 80;
+ server_name searx.example.com;
+ root /usr/local/searx/searx;
+
+ location /static {
+ }
+
+ location / {
+ include uwsgi_params;
+ uwsgi_pass unix:/run/uwsgi/app/searx/socket;
+ }
+ }
+
+Create a symlink to sites-enabled:
+
+.. code:: sh
+
+ sudo -H ln -s /etc/nginx/sites-available/searx /etc/nginx/sites-enabled/searx
+
+Restart service:
+
+.. tabs::
+
+ .. group-tab:: Ubuntu / debian
+
+ .. code:: sh
+
+ sudo -H systemctl restart nginx
+ sudo -H systemctl restart uwsgi
+
+from subdirectory URL (``/searx``)
+==================================
+
+Add this configuration in the server config file
+``/etc/nginx/sites-enabled/default``:
+
+.. code:: nginx
+
+ location /searx/static {
+ alias /usr/local/searx/searx/static;
+ }
+
+ location /searx {
+ uwsgi_param SCRIPT_NAME /searx;
+ include uwsgi_params;
+ uwsgi_pass unix:/run/uwsgi/app/searx/socket;
+ }
+
+
+**OR** using reverse proxy (Please, note that reverse proxy advised to be used
+in case of single-user or low-traffic instances.)
+
+.. code:: nginx
+
+ location /searx/static {
+ alias /usr/local/searx/searx/static;
+ }
+
+ location /searx {
+ proxy_pass http://127.0.0.1:8888;
+ proxy_set_header Host $host;
+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+ proxy_set_header X-Scheme $scheme;
+ proxy_set_header X-Script-Name /searx;
+ proxy_buffering off;
+ }
+
+Enable ``base_url`` in ``searx/settings.yml``
+
+.. code:: yaml
+
+ base_url : http://your.domain.tld/searx/
+
+Restart service:
+
+.. tabs::
+
+ .. group-tab:: Ubuntu / debian
+
+ .. code:: sh
+
+ sudo -H systemctl restart nginx
+ sudo -H systemctl restart uwsgi
+
+
+disable logs
+============
+
+For better privacy you can disable nginx logs about searx. How to proceed:
+below ``uwsgi_pass`` in ``/etc/nginx/sites-available/default`` add:
+
+.. code:: nginx
+
+ access_log /dev/null;
+ error_log /dev/null;
+
+Restart service:
+
+.. tabs::
+
+ .. group-tab:: Ubuntu / debian
+
+ .. code:: sh
+
+ sudo -H systemctl restart nginx
diff --git a/docs/admin/installation.rst b/docs/admin/installation.rst
index 15800fc01..a4c7eb8b3 100644
--- a/docs/admin/installation.rst
+++ b/docs/admin/installation.rst
@@ -4,44 +4,64 @@
Installation
============
-.. contents::
- :depth: 3
+.. sidebar:: Searx server setup
+
+ - :ref:`installation nginx`
+ - :ref:`installation apache`
+
+ If you do not have any special preferences, it is recommend to use
+ :ref:`searx.sh`.
+
+.. contents:: Contents
+ :depth: 2
+ :local:
+ :backlinks: entry
+
+.. _installation basic:
Basic installation
==================
-Step by step installation for Debian/Ubuntu with virtualenv. For Ubuntu, be sure
-to have enable universe repository.
+Step by step installation with virtualenv. For Ubuntu, be sure to have enable
+universe repository.
Install packages:
-.. code:: sh
+.. tabs::
+
+ .. group-tab:: Ubuntu / debian
- $ sudo -H apt-get install \
- git build-essential libxslt-dev \
- python-dev python-virtualenv python-babel \
- zlib1g-dev libffi-dev libssl-dev
+ .. code-block:: sh
+
+ $ sudo -H apt-get install -m \
+ git build-essential
+ libxslt-dev python3-dev python3-babel python3-venv \
+ zlib1g-dev libffi-dev libssl-dev
Install searx:
.. code:: sh
- cd /usr/local
- sudo -H git clone https://github.com/asciimoo/searx.git
- sudo -H useradd searx -d /usr/local/searx
+ sudo -H useradd searx --system --disabled-password -d /usr/local/searx
+ sudo -H usermod -a -G shadow searx
+ cd /usr/local/searx
+ sudo -H git clone https://github.com/asciimoo/searx.git searx-src
sudo -H chown searx:searx -R /usr/local/searx
-Install dependencies in a virtualenv:
+Install virtualenv:
.. code:: sh
- cd /usr/local/searx
sudo -H -u searx -i
+ (searx)$ python3 -m venv searx-pyenv
+ (searx)$ echo 'source ~/searx-pyenv/bin/activate' > ~/.profile
+
+Exit the searx bash and restart a new to install the searx dependencies:
.. code:: sh
- (searx)$ virtualenv searx-ve
- (searx)$ . ./searx-ve/bin/activate
+ sudo -H -u searx -i
+ (searx)$ cd searx-src
(searx)$ ./manage.sh update_packages
Configuration
@@ -49,7 +69,9 @@ Configuration
.. code:: sh
- sed -i -e "s/ultrasecretkey/`openssl rand -hex 16`/g" searx/settings.yml
+ sudo -H -u searx -i
+ (searx)$ cd searx-src
+ (searx)$ sed -i -e "s/ultrasecretkey/`openssl rand -hex 16`/g" searx/settings.yml
Edit searx/settings.yml if necessary.
@@ -60,7 +82,9 @@ Start searx:
.. code:: sh
- python searx/webapp.py
+ sudo -H -u searx -i
+ (searx)$ cd searx-src
+ (searx)$ python3 searx/webapp.py
Go to http://localhost:8888
@@ -70,254 +94,118 @@ If everything works fine, disable the debug option in settings.yml:
sed -i -e "s/debug : True/debug : False/g" searx/settings.yml
-At this point searx is not demonized ; uwsgi allows this.
-
-You can exit the virtualenv and the searx user bash (enter exit command
-twice).
+At this point searx is not demonized ; uwsgi allows this. You can exit the
+virtualenv and the searx user bash (enter exit command twice).
uwsgi
=====
Install packages:
-.. code:: sh
-
- sudo -H apt-get install \
- uwsgi uwsgi-plugin-python
-
-Create the configuration file ``/etc/uwsgi/apps-available/searx.ini`` with this
-content:
-
-.. code:: ini
-
- [uwsgi]
- # Who will run the code
- uid = searx
- gid = searx
-
- # disable logging for privacy
- disable-logging = true
-
- # Number of workers (usually CPU count)
- workers = 4
+.. tabs::
- # The right granted on the created socket
- chmod-socket = 666
+ .. group-tab:: Ubuntu / debian
- # Plugin to use and interpretor config
- single-interpreter = true
- master = true
- plugin = python
- lazy-apps = true
- enable-threads = true
+ .. code-block:: bash
- # Module to import
- module = searx.webapp
+ sudo -H apt-get install uwsgi uwsgi-plugin-python3
- # Support running the module from a webserver subdirectory.
- route-run = fixpathinfo:
-
- # Virtualenv and python path
- virtualenv = /usr/local/searx/searx-ve/
- pythonpath = /usr/local/searx/
- chdir = /usr/local/searx/searx/
-
-Activate the uwsgi application and restart:
-
-.. code:: sh
-
- cd /etc/uwsgi/apps-enabled
- ln -s ../apps-available/searx.ini
- /etc/init.d/uwsgi restart
-
-Web server
-==========
-
-with nginx
-----------
-
-If nginx is not installed (uwsgi will not work with the package
-nginx-light):
-
-.. code:: sh
-
- sudo -H apt-get install nginx
-
-Hosted at /
-~~~~~~~~~~~
-
-Create the configuration file ``/etc/nginx/sites-available/searx`` with this
+Create the configuration file ``/etc/uwsgi/apps-available/searx.ini`` with this
content:
-.. code:: nginx
-
- server {
- listen 80;
- server_name searx.example.com;
- root /usr/local/searx/searx;
-
- location /static {
- }
-
- location / {
- include uwsgi_params;
- uwsgi_pass unix:/run/uwsgi/app/searx/socket;
- }
- }
-
-Create a symlink to sites-enabled:
-
-.. code:: sh
-
- sudo -H ln -s /etc/nginx/sites-available/searx /etc/nginx/sites-enabled/searx
-
-Restart service:
-
-.. code:: sh
-
- sudo -H service nginx restart
- sudo -H service uwsgi restart
-
-from subdirectory URL (/searx)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Add this configuration in the server config file
-``/etc/nginx/sites-enabled/default``:
-
-.. code:: nginx
-
- location /searx/static {
- alias /usr/local/searx/searx/static;
- }
-
- location /searx {
- uwsgi_param SCRIPT_NAME /searx;
- include uwsgi_params;
- uwsgi_pass unix:/run/uwsgi/app/searx/socket;
- }
-
-
-**OR** using reverse proxy (Please, note that reverse proxy advised to be used
-in case of single-user or low-traffic instances.)
-
-.. code:: nginx
-
- location /searx/static {
- alias /usr/local/searx/searx/static;
- }
-
- location /searx {
- proxy_pass http://127.0.0.1:8888;
- proxy_set_header Host $host;
- proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
- proxy_set_header X-Scheme $scheme;
- proxy_set_header X-Script-Name /searx;
- proxy_buffering off;
- }
-
-
-Enable ``base_url`` in ``searx/settings.yml``
-
-.. code:: yaml
-
- base_url : http://your.domain.tld/searx/
+.. code:: ini
-Restart service:
+ [uwsgi]
-.. code:: sh
+ # uWSGI core
+ # ----------
+ #
+ # https://uwsgi-docs.readthedocs.io/en/latest/Options.html#uwsgi-core
- sudo -H service nginx restart
- sudo -H service uwsgi restart
+ # Who will run the code
+ uid = searx
+ gid = searx
-disable logs
-^^^^^^^^^^^^
+ # chdir to specified directory before apps loading
+ chdir = /usr/local/searx/searx-src/searx
-for better privacy you can disable nginx logs about searx.
+ # disable logging for privacy
+ disable-logging = true
-how to proceed: below ``uwsgi_pass`` in ``/etc/nginx/sites-available/default``
-add:
+ # The right granted on the created socket
+ chmod-socket = 666
-.. code:: nginx
+ # Plugin to use and interpretor config
+ single-interpreter = true
- access_log /dev/null;
- error_log /dev/null;
+ # enable master process
+ master = true
-Restart service:
+ # load apps in each worker instead of the master
+ lazy-apps = true
-.. code:: sh
+ # load uWSGI plugins
+ plugin = python3,http
- sudo -H service nginx restart
+ # By default the Python plugin does not initialize the GIL. This means your
+ # app-generated threads will not run. If you need threads, remember to enable
+ # them with enable-threads. Running uWSGI in multithreading mode (with the
+ # threads options) will automatically enable threading support. This *strange*
+ # default behaviour is for performance reasons.
+ enable-threads = true
-with apache
------------
+ # plugin: python
+ # --------------
+ #
+ # https://uwsgi-docs.readthedocs.io/en/latest/Options.html#plugin-python
-Add wsgi mod:
+ # load a WSGI module
+ module = searx.webapp
-.. code:: sh
+ # set PYTHONHOME/virtualenv
+ virtualenv = /usr/local/searx/searx-pyenv
- sudo -H apt-get install libapache2-mod-uwsgi
- sudo -H a2enmod uwsgi
+ # add directory (or glob) to pythonpath
+ pythonpath = /usr/local/searx/searx-src
-Add this configuration in the file ``/etc/apache2/apache2.conf``:
-.. code:: apache
+ # plugin http
+ # -----------
+ #
+ # https://uwsgi-docs.readthedocs.io/en/latest/Options.html#plugin-http
- <Location />
- Options FollowSymLinks Indexes
- SetHandler uwsgi-handler
- uWSGISocket /run/uwsgi/app/searx/socket
- </Location>
+ # Native HTTP support: https://uwsgi-docs.readthedocs.io/en/latest/HTTP.html
+ http = 127.0.0.1:8888
-Note that if your instance of searx is not at the root, you should change
-``<Location />`` by the location of your instance, like ``<Location /searx>``.
-
-Restart Apache:
+Activate the uwsgi application and restart:
.. code:: sh
- sudo -H /etc/init.d/apache2 restart
-
-disable logs
-~~~~~~~~~~~~
-
-For better privacy you can disable Apache logs.
-
-.. warning::
-
- You can only disable logs for the whole (virtual) server not for a specific
- path.
-
-Go back to ``/etc/apache2/apache2.conf`` and above ``<Location />`` add:
-
-.. code:: apache
-
- CustomLog /dev/null combined
-
-Restart Apache:
-
-.. code:: sh
+ cd /etc/uwsgi/apps-enabled
+ ln -s ../apps-available/searx.ini
+ /etc/init.d/uwsgi restart
- sudo -H /etc/init.d/apache2 restart
How to update
=============
.. code:: sh
- cd /usr/local/searx
sudo -H -u searx -i
-
-.. code:: sh
-
- (searx)$ . ./searx-ve/bin/activate
(searx)$ git stash
(searx)$ git pull origin master
(searx)$ git stash apply
(searx)$ ./manage.sh update_packages
-.. code:: sh
+Restart uwsgi:
+
+.. tabs::
+
+ .. group-tab:: Ubuntu / debian
+
+ .. code:: sh
- sudo -H service uwsgi restart
+ sudo -H systemctl restart uwsgi
Docker
======
diff --git a/docs/admin/morty.rst b/docs/admin/morty.rst
index 7d7b34492..9af9b6ae9 100644
--- a/docs/admin/morty.rst
+++ b/docs/admin/morty.rst
@@ -1,3 +1,6 @@
+
+.. _searx_morty:
+
=========================
How to setup result proxy
=========================
diff --git a/docs/admin/settings.rst b/docs/admin/settings.rst
index 0bfdcc6cb..59a406a28 100644
--- a/docs/admin/settings.rst
+++ b/docs/admin/settings.rst
@@ -4,11 +4,17 @@
``settings.yml``
================
+This page describe the options possibilities of the :origin:`searx/settings.yml`
+file.
+
.. sidebar:: Further reading ..
- :ref:`search API`
-This page describe the options possibilities of the settings.yml file.
+.. contents:: Contents
+ :depth: 2
+ :local:
+ :backlinks: entry
.. _settings global:
diff --git a/docs/blog/index.rst b/docs/blog/index.rst
index 52fa3f126..bc90f4502 100644
--- a/docs/blog/index.rst
+++ b/docs/blog/index.rst
@@ -3,7 +3,8 @@ Blog
====
.. toctree::
- :maxdepth: 1
+ :maxdepth: 2
+ :caption: Contents
python3
admin
diff --git a/docs/conf.py b/docs/conf.py
index ebcd102b2..8f11b1c6c 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -5,6 +5,7 @@ from searx.version import VERSION_STRING
from pallets_sphinx_themes import ProjectLink
GIT_URL = os.environ.get("GIT_URL", "https://github.com/asciimoo/searx")
+GIT_BRANCH =os.environ.get("GIT_BRANCH", "master")
SEARX_URL = os.environ.get("SEARX_URL", "https://searx.me")
DOCS_URL = os.environ.get("DOCS_URL", "https://asciimoo.github.io/searx/")
@@ -35,7 +36,7 @@ extlinks['wiki'] = ('https://github.com/asciimoo/searx/wiki/%s', ' ')
extlinks['pull'] = ('https://github.com/asciimoo/searx/pull/%s', 'PR ')
# links to custom brand
-extlinks['origin'] = (GIT_URL + '/blob/master/%s', 'git://')
+extlinks['origin'] = (GIT_URL + '/blob/' + GIT_BRANCH + '/%s', 'git://')
extlinks['patch'] = (GIT_URL + '/commit/%s', '#')
extlinks['search'] = (SEARX_URL + '/%s', '#')
extlinks['docs'] = (DOCS_URL + '/%s', 'docs: ')
@@ -61,6 +62,7 @@ extensions = [
"pallets_sphinx_themes",
"sphinx_issues", # https://github.com/sloria/sphinx-issues/blob/master/README.rst
"sphinxcontrib.jinja", # https://github.com/tardyp/sphinx-jinja
+ "sphinxcontrib.programoutput", # https://github.com/NextThought/sphinxcontrib-programoutput
'linuxdoc.rstFlatTable', # Implementation of the 'flat-table' reST-directive.
'linuxdoc.kfigure', # Sphinx extension which implements scalable image handling.
"sphinx_tabs.tabs", # https://github.com/djungelorm/sphinx-tabs
diff --git a/docs/dev/contribution_guide.rst b/docs/dev/contribution_guide.rst
index 459dfb448..f9e9569f9 100644
--- a/docs/dev/contribution_guide.rst
+++ b/docs/dev/contribution_guide.rst
@@ -4,6 +4,11 @@
How to contribute
=================
+.. contents:: Contents
+ :depth: 2
+ :local:
+ :backlinks: entry
+
Prime directives: Privacy, Hackability
======================================
diff --git a/docs/dev/index.rst b/docs/dev/index.rst
index cb913a82b..ba0a25a9c 100644
--- a/docs/dev/index.rst
+++ b/docs/dev/index.rst
@@ -3,7 +3,8 @@ Developer documentation
=======================
.. toctree::
- :maxdepth: 1
+ :maxdepth: 2
+ :caption: Contents
quickstart
contribution_guide
diff --git a/docs/dev/makefile.rst b/docs/dev/makefile.rst
index f5957001c..ed9942f4c 100644
--- a/docs/dev/makefile.rst
+++ b/docs/dev/makefile.rst
@@ -11,6 +11,8 @@ Makefile Targets
Before looking deeper at the targets, first read about :ref:`makefile setup`
and :ref:`make pyenv`.
+ To install system requirements follow :ref:`buildhosts`.
+
With the aim to simplify development cycles, started with :pull:`1756` a
``Makefile`` based boilerplate was added. If you are not familiar with
Makefiles, we recommend to read gnu-make_ introduction.
@@ -37,15 +39,17 @@ Calling the ``help`` target gives a first overview::
.. _makefile setup:
-Setup
-=====
+Makefile setup
+==============
.. _git stash: https://git-scm.com/docs/git-stash
-The main setup is done in the :origin:`Makefile`::
+The main setup is done in the :origin:`.config.sh` (read :ref:`toolboxing
+setup`)::
- export GIT_URL=https://github.com/asciimoo/searx
export SEARX_URL=https://searx.me
+
+ export GIT_URL=https://github.com/asciimoo/searx
export DOCS_URL=https://asciimoo.github.io/searx
.. sidebar:: fork & upstream
@@ -170,7 +174,7 @@ e.g.:
.. code:: sh
- $ make test.pep8 test.unit
+ $ make test.pep8 test.unit test.sh
. ./local/py3/bin/activate; ./manage.sh pep8_check
[!] Running pep8 check
. ./local/py3/bin/activate; ./manage.sh unit_tests
diff --git a/docs/dev/reST.rst b/docs/dev/reST.rst
index 4835cbbcf..7b253d9ec 100644
--- a/docs/dev/reST.rst
+++ b/docs/dev/reST.rst
@@ -326,7 +326,7 @@ Literal blocks
The simplest form of :duref:`literal-blocks` is a indented block introduced by
two colons (``::``). For highlighting use :dudir:`highlight` or :ref:`reST
code` directive. To include literals from external files use directive
-:dudir:`literalinclude`.
+:rst:dir:`literalinclude`.
.. _reST literal:
@@ -1312,9 +1312,8 @@ others are basic-tabs_ and code-tabs_. Below a *group-tab* example from
.. literalinclude:: ../admin/buildhosts.rst
:language: reST
- :start-after: .. _system requirements:
- :end-before: .. _system requirements END:
-
+ :start-after: .. SNIP sh lint requirements
+ :end-before: .. SNAP sh lint requirements
.. _math:
diff --git a/docs/index.rst b/docs/index.rst
index b62f4dc5d..9e590867c 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -2,7 +2,14 @@
Welcome to searx
================
-Search without being tracked.
+ *Search without being tracked.*
+
+Searx is a free internet metasearch engine which aggregates results from more
+than 70 search services. Users are neither tracked nor profiled. Additionally,
+searx can be used over Tor for online anonymity.
+
+Get started with searx by using one of the Searx-instances_. If you don't trust
+anyone, you can set up your own, see :ref:`installation`.
.. sidebar:: Features
@@ -16,19 +23,14 @@ Search without being tracked.
- Hosted by organizations, such as *La Quadrature du Net*, which promote
digital rights
-Searx is a free internet metasearch engine which aggregates results from more
-than 70 search services. Users are neither tracked nor profiled. Additionally,
-searx can be used over Tor for online anonymity.
-
-Get started with searx by using one of the Searx-instances_. If you don't trust
-anyone, you can set up your own, see :ref:`installation`.
-
.. toctree::
:maxdepth: 2
+ :caption: Contents
user/index
admin/index
dev/index
+ utils/index
blog/index
.. _Searx-instances: https://searx.space
diff --git a/docs/user/index.rst b/docs/user/index.rst
index b13aca216..96d11bf8a 100644
--- a/docs/user/index.rst
+++ b/docs/user/index.rst
@@ -3,7 +3,8 @@ User documentation
==================
.. toctree::
- :maxdepth: 1
+ :maxdepth: 2
+ :caption: Contents
search_syntax
own-instance
diff --git a/docs/user/own-instance.rst b/docs/user/own-instance.rst
index a2f736562..4876abe99 100644
--- a/docs/user/own-instance.rst
+++ b/docs/user/own-instance.rst
@@ -2,8 +2,10 @@
Why use a private instance?
===========================
-"Is it worth to run my own instance?" is a common question among searx users.
-Before answering this question, see what options a searx user has.
+ *"Is it worth to run my own instance?"*
+
+\.\. is a common question among searx users. Before answering this question,
+see what options a searx user has.
Public instances are open to everyone who has access to its URL. Usually, these
are operated by unknown parties (from the users' point of view). Private
diff --git a/docs/user/public_instances.rst b/docs/user/public_instances.rst
deleted file mode 100644
index f58ab7a02..000000000
--- a/docs/user/public_instances.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-:orphan:
-
-This page page has been moved to `searx.space <https://searx.space/>`__
diff --git a/docs/utils/filtron.sh.rst b/docs/utils/filtron.sh.rst
new file mode 100644
index 000000000..1bba1b3e4
--- /dev/null
+++ b/docs/utils/filtron.sh.rst
@@ -0,0 +1,75 @@
+
+.. _filtron.sh:
+
+====================
+``utils/filtron.sh``
+====================
+
+.. sidebar:: further reading
+
+ - :ref:`installation`
+ - :ref:`searx_filtron`
+ - :ref:`architecture`
+
+.. _Go: https://golang.org/
+.. _filtron: https://github.com/asciimoo/filtron
+.. _filtron README: https://github.com/asciimoo/filtron/blob/master/README.md
+
+To simplify installation and maintenance of a filtron instance you can use the
+script :origin:`utils/filtron.sh`. In most cases you will install filtron_
+simply by running the command:
+
+.. code:: bash
+
+ sudo -H ./utils/filtron.sh install all
+
+The script adds a ``${SERVICE_USER}`` (default:``filtron``) and installs filtron_
+into this user account:
+
+#. Create a separated user account (``filtron``).
+#. Download and install Go_ binary in users $HOME (``~filtron``).
+#. Install filtron with the package management of Go_ (``go get -v -u
+ github.com/asciimoo/filtron``)
+#. Setup a proper rule configuration :origin:`[ref]
+ <utils/templates/etc/filtron/rules.json>` (``/etc/filtron/rules.json``).
+#. Setup a systemd service unit :origin:`[ref]
+ <utils/templates/lib/systemd/system/filtron.service>`
+ (``/lib/systemd/system/filtron.service``).
+
+.. _reverse proxy:
+
+Public Reverse Proxy
+====================
+
+To install searx in your public HTTP server use:
+
+.. code:: bash
+
+ sudo -H ./utils/filtron.sh apache install
+
+.. tabs::
+
+ .. group-tab:: apache
+
+ .. literalinclude:: ../../utils/templates/etc/apache2/sites-available/searx.conf:filtron
+ :language: apache
+
+ .. tabs::
+
+ .. group-tab:: Ubuntu / debian
+
+ .. code-block:: sh
+
+ $ sudo -H a2enmod headers
+ $ sudo -H a2enmod proxy
+ $ sudo -H a2enmod proxy_http
+
+
+Overview
+========
+
+The ``--help`` output of the script is largely self-explanatory
+(:ref:`toolboxing common`):
+
+.. program-output:: ../utils/filtron.sh --help
+
diff --git a/docs/utils/index.rst b/docs/utils/index.rst
new file mode 100644
index 000000000..338914715
--- /dev/null
+++ b/docs/utils/index.rst
@@ -0,0 +1,53 @@
+
+.. _searx_utils:
+.. _toolboxing:
+
+=======================
+Tooling box ``utils/*``
+=======================
+
+In the folder :origin:`utils/` we maintain some tools useful for admins and
+developers.
+
+.. sidebar:: Work needed!
+
+ Our scripts to maintain services do most support only systemd init process
+ used by debian, ubuntu and many other dists. In general our scripts are only
+ partially usable on debian systems. We are working on this limitation, if
+ you have any contribution, please send us your :pull:`PR <../pulls>`, see
+ :ref:`how to contribute`.
+
+.. toctree::
+ :maxdepth: 2
+ :caption: Contents
+
+ searx.sh
+ filtron.sh
+ morty.sh
+
+.. _toolboxing common:
+
+Common commands
+===============
+
+Scripts to maintain services often dispose of common commands and environments.
+
+``shell``:
+ Opens a shell from the service user ``${SERVICE_USSR}``, very helpful for
+ troubleshooting.
+
+``inspect service``:
+ Shows status and log of the service, most often you have a option to enable
+ more verbose debug logs. Very helpful for debugging, but be careful not to
+ enable debugging in a production environment!
+
+.. _toolboxing setup:
+
+Tooling box setup
+=================
+
+The main setup is done in the :origin:`.config.sh` (read also :ref:`makefile
+setup`).
+
+.. literalinclude:: ../../.config.sh
+ :language: bash
diff --git a/docs/utils/morty.sh.rst b/docs/utils/morty.sh.rst
new file mode 100644
index 000000000..9997ffec5
--- /dev/null
+++ b/docs/utils/morty.sh.rst
@@ -0,0 +1,39 @@
+
+.. _morty: https://github.com/asciimoo/morty
+.. _morty's README: https://github.com/asciimoo/morty
+
+.. _morty.sh:
+
+==================
+``utils/morty.sh``
+==================
+
+.. sidebar:: further reading
+
+ - :ref:`installation`
+ - :ref:`architecture`
+
+To simplify installation and maintenance of a morty_ instance you can use the
+script :origin:`utils/morty.sh`. In most cases you will install morty_ simply by
+running the command:
+
+.. code:: bash
+
+ sudo -H ./utils/morty.sh install all
+
+The script adds a ``${SERVICE_USER}`` (default:``morty``) and installs morty_
+into this user account.
+
+.. hint::
+
+ To add morty to your searx instance read chapter :reF:`searx_morty`.
+
+
+Overview
+========
+
+The ``--help`` output of the script is largely self-explanatory
+(:ref:`toolboxing common`):
+
+.. program-output:: ../utils/morty.sh --help
+
diff --git a/docs/utils/searx.sh.rst b/docs/utils/searx.sh.rst
new file mode 100644
index 000000000..988c08438
--- /dev/null
+++ b/docs/utils/searx.sh.rst
@@ -0,0 +1,73 @@
+
+.. _searx.sh:
+
+==================
+``utils/searx.sh``
+==================
+
+.. sidebar:: further reading
+
+ - :ref:`installation`
+ - :ref:`architecture`
+ - :ref:`filtron.sh`
+
+To simplify installation and maintenance of a searx instance you can use the
+script :origin:`utils/searx.sh`.
+
+Install
+=======
+
+In most cases you will install searx simply by running the command:
+
+.. code:: bash
+
+ sudo -H ./utils/searx.sh install all
+
+The script adds a ``${SERVICE_USER}`` (default:``searx``) and installs searx
+into this user account. The installation is described in chapter
+:ref:`installation basic`.
+
+.. _intranet reverse proxy:
+
+Intranet Reverse Proxy
+======================
+
+To install searx in your intranet HTTP server use:
+
+.. code:: bash
+
+ sudo -H ./utils/searx.sh apache install
+
+.. warning::
+
+ This setup is **not** suitable **for public instances**, go on with
+ :ref:`reverse proxy`!
+
+.. tabs::
+
+ .. group-tab:: apache
+
+ .. literalinclude:: ../../utils/templates/etc/apache2/sites-available/searx.conf:uwsgi
+ :language: apache
+
+ .. tabs::
+
+ .. group-tab:: Ubuntu / debian
+
+ .. code-block:: sh
+
+ $ sudo -H apt install libapache2-mod-uwsgi
+
+ .. group-tab:: Arch Linux
+
+ .. code-block:: sh
+
+ $ sudo pacman -S uwsgi
+
+Overview
+========
+
+The ``--help`` output of the script is largely self-explanatory
+(:ref:`toolboxing common`):
+
+.. program-output:: ../utils/searx.sh --help