diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/_themes/searx/static/searx.css | 12 | ||||
| -rw-r--r-- | docs/admin/arch_public.dot | 8 | ||||
| -rw-r--r-- | docs/admin/architecture.rst | 12 | ||||
| -rw-r--r-- | docs/admin/buildhosts.rst | 59 | ||||
| -rw-r--r-- | docs/admin/filtron.rst | 224 | ||||
| -rw-r--r-- | docs/admin/index.rst | 5 | ||||
| -rw-r--r-- | docs/admin/installation-apache.rst | 94 | ||||
| -rw-r--r-- | docs/admin/installation-nginx.rst | 141 | ||||
| -rw-r--r-- | docs/admin/installation.rst | 324 | ||||
| -rw-r--r-- | docs/admin/morty.rst | 3 | ||||
| -rw-r--r-- | docs/admin/settings.rst | 8 | ||||
| -rw-r--r-- | docs/blog/index.rst | 3 | ||||
| -rw-r--r-- | docs/conf.py | 4 | ||||
| -rw-r--r-- | docs/dev/contribution_guide.rst | 5 | ||||
| -rw-r--r-- | docs/dev/index.rst | 3 | ||||
| -rw-r--r-- | docs/dev/makefile.rst | 14 | ||||
| -rw-r--r-- | docs/dev/reST.rst | 7 | ||||
| -rw-r--r-- | docs/index.rst | 18 | ||||
| -rw-r--r-- | docs/user/index.rst | 3 | ||||
| -rw-r--r-- | docs/user/own-instance.rst | 6 | ||||
| -rw-r--r-- | docs/user/public_instances.rst | 3 | ||||
| -rw-r--r-- | docs/utils/filtron.sh.rst | 75 | ||||
| -rw-r--r-- | docs/utils/index.rst | 53 | ||||
| -rw-r--r-- | docs/utils/morty.sh.rst | 39 | ||||
| -rw-r--r-- | docs/utils/searx.sh.rst | 73 |
25 files changed, 832 insertions, 364 deletions
diff --git a/docs/_themes/searx/static/searx.css b/docs/_themes/searx/static/searx.css index d6a664f0f..ae742a4e0 100644 --- a/docs/_themes/searx/static/searx.css +++ b/docs/_themes/searx/static/searx.css @@ -33,7 +33,7 @@ p.sidebar-title, .sidebar p { /* admonitions */ -div.admonition, div.topic { +div.admonition, div.topic, div.toctree-wrapper { background-color: #fafafa; margin: 8px 0px; padding: 1em; @@ -42,6 +42,16 @@ div.admonition, div.topic { border-right: none; border-bottom: none; border-left: 5pt solid #ccc; + list-style-type: disclosure-closed; +} + +div.toctree-wrapper p.caption { + font-weight: normal; + font-size: 24px; + margin: 0 0 10px 0; + padding: 0; + line-height: 1; + display: inline; } p.admonition-title:after { diff --git a/docs/admin/arch_public.dot b/docs/admin/arch_public.dot index a46c96de3..5018225c3 100644 --- a/docs/admin/arch_public.dot +++ b/docs/admin/arch_public.dot @@ -4,11 +4,11 @@ digraph G { edge [fontname="Sans"]; browser [label="Browser", shape=Mdiamond]; - rp [label="Reverse Proxy", href="url to configure reverse proxy"]; - filtron [label="Filtron", href="https://github.com/asciimoo/filtron"]; - morty [label="Morty", href="https://github.com/asciimoo/morty"]; + rp [label="Reverse Proxy", href="https://asciimoo.github.io/searx/utils/filtron.sh.html#public-reverse-proxy"]; + filtron [label="Filtron", href="https://asciimoo.github.io/searx/utils/filtron.sh.html"]; + morty [label="Morty", href="https://asciimoo.github.io/searx/utils/morty.sh.html"]; static [label="Static files", href="url to configure static files"]; - uwsgi [label="uwsgi", href="url to configure uwsgi"] + uwsgi [label="uwsgi", href="https://asciimoo.github.io/searx/utils/searx.sh.html"] searx1 [label="Searx #1"]; searx2 [label="Searx #2"]; searx3 [label="Searx #3"]; diff --git a/docs/admin/architecture.rst b/docs/admin/architecture.rst index 7064a294b..fbc3bf483 100644 --- a/docs/admin/architecture.rst +++ b/docs/admin/architecture.rst @@ -4,17 +4,19 @@ Architecture ============ -.. sidebar:: Needs work! +.. sidebar:: Tooling box - This article needs some work / Searx is a collaborative effort. If you have - any contribution, feel welcome to send us your :pull:`PR <../pulls>`, see - :ref:`how to contribute`. + - :ref:`searx & uwsgi <searx.sh>` + - :ref:`filtron <filtron.sh>` + - :ref:`reverse proxy` + - :ref:`morty <morty.sh>` Herein you will find some hints and suggestions about typical architectures of searx infrastructures. We start with a contribution from :pull:`@dalf <1776#issuecomment-567917320>`. -It shows a *reference* setup for public searx instances. +It shows a *reference* setup for public searx instances which can build up and +maintained by the scripts from our :ref:`toolboxing`. .. _arch public: diff --git a/docs/admin/buildhosts.rst b/docs/admin/buildhosts.rst index 5260da033..27784764f 100644 --- a/docs/admin/buildhosts.rst +++ b/docs/admin/buildhosts.rst @@ -4,13 +4,18 @@ Buildhosts ========== +To get best results from build, its recommend to install additional packages +on build hosts. + .. sidebar:: This article needs some work If you have any contribution send us your :pull:`PR <../pulls>`, see :ref:`how to contribute`. -To get best results from build, its recommend to install additional packages -on build hosts. +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry .. _docs build: @@ -35,8 +40,17 @@ processing additional packages are needed. The XeTeX_ needed not only for PDF creation, its also needed for :ref:`math` when HTML output is build. To be able to do :ref:`sphinx:math-support` without CDNs, the math are rendered -as images (``sphinx.ext.imgmath`` extension). If your docs build (``make -docs``) shows warnings like this:: +as images (``sphinx.ext.imgmath`` extension). + +Here is the extract from the :origin:`docs/conf.py` file, setting math renderer +to ``imgmath``: + +.. literalinclude:: ../conf.py + :language: python + :start-after: # sphinx.ext.imgmath setup + :end-before: # sphinx.ext.imgmath setup END + +If your docs build (``make docs``) shows warnings like this:: WARNING: dot(1) not found, for better output quality install \ graphviz from http://www.graphviz.org @@ -47,8 +61,6 @@ docs``) shows warnings like this:: you need to install additional packages on your build host, to get better HTML output. -.. _system requirements: - .. tabs:: .. group-tab:: Ubuntu / debian @@ -94,10 +106,35 @@ For PDF output you also need: texlive-collection-fontsrecommended texlive-collection-latex \ dejavu-sans-fonts dejavu-serif-fonts dejavu-sans-mono-fonts -.. _system requirements END: +.. _sh lint: -.. literalinclude:: ../conf.py - :language: python - :start-after: # sphinx.ext.imgmath setup - :end-before: # sphinx.ext.imgmath setup END +Lint shell scripts +================== + +.. _ShellCheck: https://github.com/koalaman/shellcheck + +To lint shell scripts, we use ShellCheck_ - A shell script static analysis tool. + +.. SNIP sh lint requirements + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code-block:: sh + + $ sudo apt install shellcheck + + .. group-tab:: Arch Linux + + .. code-block:: sh + + $ sudo pacman -S shellcheck + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + $ sudo dnf install ShellCheck +.. SNAP sh lint requirements diff --git a/docs/admin/filtron.rst b/docs/admin/filtron.rst index 07dcb9bc5..2bc663411 100644 --- a/docs/admin/filtron.rst +++ b/docs/admin/filtron.rst @@ -1,18 +1,48 @@ + +.. _searx_filtron: + ========================== How to protect an instance ========================== +.. sidebar:: further reading + + - :ref:`filtron.sh` + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. _filtron: https://github.com/asciimoo/filtron + Searx depens on external search services. To avoid the abuse of these services it is advised to limit the number of requests processed by searx. -An application firewall, ``filtron`` solves exactly this problem. Information -on how to install it can be found at the `project page of filtron -<https://github.com/asciimoo/filtron>`__. +An application firewall, filtron_ solves exactly this problem. Filtron is just +a middleware between your web server (nginx, apache, ...) and searx, we describe +such infratructures in chapter: :ref:`architecture`. + + +filtron & go +============ + +.. _Go: https://golang.org/ +.. _filtron README: https://github.com/asciimoo/filtron/blob/master/README.md + +Filtron needs Go_ installed. If Go_ is preinstalled, filtron_ is simply +installed by ``go get`` package management (see `filtron README`_). If you use +filtron as middleware, a more isolated setup is recommended. To simplify such +an installation and the maintenance of, use our script :ref:`filtron.sh`. Sample configuration of filtron =============================== +.. sidebar:: Tooling box + + - :origin:`/etc/filtron/rules.json <utils/templates/etc/filtron/rules.json>` + An example configuration can be find below. This configuration limits the access of: @@ -24,100 +54,100 @@ of: .. code:: json - [{ - "name":"search request", - "filters":[ - "Param:q", - "Path=^(/|/search)$" - ], - "interval":"<time-interval-in-sec (int)>", - "limit":"<max-request-number-in-interval (int)>", - "subrules":[ - { - "name":"roboagent limit", - "interval":"<time-interval-in-sec (int)>", - "limit":"<max-request-number-in-interval (int)>", - "filters":[ - "Header:User-Agent=(curl|cURL|Wget|python-requests|Scrapy|FeedFetcher|Go-http-client)" - ], - "actions":[ - { - "name":"block", - "params":{ - "message":"Rate limit exceeded" - } - } - ] - }, - { - "name":"botlimit", - "limit":0, - "stop":true, - "filters":[ - "Header:User-Agent=(Googlebot|bingbot|Baiduspider|yacybot|YandexMobileBot|YandexBot|Yahoo! Slurp|MJ12bot|AhrefsBot|archive.org_bot|msnbot|MJ12bot|SeznamBot|linkdexbot|Netvibes|SMTBot|zgrab|James BOT)" - ], - "actions":[ - { - "name":"block", - "params":{ - "message":"Rate limit exceeded" - } - } - ] - }, - { - "name":"IP limit", - "interval":"<time-interval-in-sec (int)>", - "limit":"<max-request-number-in-interval (int)>", - "stop":true, - "aggregations":[ - "Header:X-Forwarded-For" - ], - "actions":[ - { - "name":"block", - "params":{ - "message":"Rate limit exceeded" - } - } - ] - }, - { - "name":"rss/json limit", - "interval":"<time-interval-in-sec (int)>", - "limit":"<max-request-number-in-interval (int)>", - "stop":true, - "filters":[ - "Param:format=(csv|json|rss)" - ], - "actions":[ - { - "name":"block", - "params":{ - "message":"Rate limit exceeded" - } - } - ] - }, - { - "name":"useragent limit", - "interval":"<time-interval-in-sec (int)>", - "limit":"<max-request-number-in-interval (int)>", - "aggregations":[ - "Header:User-Agent" - ], - "actions":[ - { - "name":"block", - "params":{ - "message":"Rate limit exceeded" - } - } - ] - } - ] - }] - + [ + { "name": "search request", + "filters": [ + "Param:q", + "Path=^(/|/search)$" + ], + "interval": "<time-interval-in-sec (int)>", + "limit": "<max-request-number-in-interval (int)>", + "subrules": [ + { + "name": "roboagent limit", + "interval": "<time-interval-in-sec (int)>", + "limit": "<max-request-number-in-interval (int)>", + "filters": [ + "Header:User-Agent=(curl|cURL|Wget|python-requests|Scrapy|FeedFetcher|Go-http-client)" + ], + "actions": [ + { "name": "log"}, + { "name": "block", + "params": { + "message": "Rate limit exceeded" + } + } + ] + }, + { + "name": "botlimit", + "limit": 0, + "stop": true, + "filters": [ + "Header:User-Agent=(Googlebot|bingbot|Baiduspider|yacybot|YandexMobileBot|YandexBot|Yahoo! Slurp|MJ12bot|AhrefsBot|archive.org_bot|msnbot|MJ12bot|SeznamBot|linkdexbot|Netvibes|SMTBot|zgrab|James BOT)" + ], + "actions": [ + { "name": "log"}, + { "name": "block", + "params": { + "message": "Rate limit exceeded" + } + } + ] + }, + { + "name": "IP limit", + "interval": "<time-interval-in-sec (int)>", + "limit": "<max-request-number-in-interval (int)>", + "stop": true, + "aggregations": [ + "Header:X-Forwarded-For" + ], + "actions": [ + { "name": "log"}, + { "name": "block", + "params": { + "message": "Rate limit exceeded" + } + } + ] + }, + { + "name": "rss/json limit", + "interval": "<time-interval-in-sec (int)>", + "limit": "<max-request-number-in-interval (int)>", + "stop": true, + "filters": [ + "Param:format=(csv|json|rss)" + ], + "actions": [ + { "name": "log"}, + { "name": "block", + "params": { + "message": "Rate limit exceeded" + } + } + ] + }, + { + "name": "useragent limit", + "interval": "<time-interval-in-sec (int)>", + "limit": "<max-request-number-in-interval (int)>", + "aggregations": [ + "Header:User-Agent" + ], + "actions": [ + { "name": "log"}, + { "name": "block", + "params": { + "message": "Rate limit exceeded" + } + } + ] + } + ] + } + ] Route request through filtron diff --git a/docs/admin/index.rst b/docs/admin/index.rst index b3c7f5119..4190bf24b 100644 --- a/docs/admin/index.rst +++ b/docs/admin/index.rst @@ -3,9 +3,12 @@ Administrator documentation =========================== .. toctree:: - :maxdepth: 1 + :maxdepth: 2 + :caption: Contents installation + installation-nginx + installation-apache settings api architecture diff --git a/docs/admin/installation-apache.rst b/docs/admin/installation-apache.rst new file mode 100644 index 000000000..7f92e47d5 --- /dev/null +++ b/docs/admin/installation-apache.rst @@ -0,0 +1,94 @@ +.. _installation apache: + +=================== +Install with apache +=================== + +.. sidebar:: public HTTP servers + + On public searx instances use an application firewall (:ref:`filtron + <filtron.sh>`). + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +Add wsgi mod +============ + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: sh + + sudo -H apt-get install libapache2-mod-uwsgi + sudo -H a2enmod uwsgi + +Add this configuration in the file ``/etc/apache2/apache2.conf``. To limit +acces to your intranet replace ``Allow from all`` directive and replace +``192.168.0.0/16`` with your subnet IP/class. + +.. _inranet apache site: + +Note that if your instance of searx is not at the root, you should change +``<Location />`` by the location of your instance, like ``<Location /searx>``: + +.. code:: apache + + # CustomLog /dev/null combined + + <IfModule mod_uwsgi.c> + + <Location /> + + Options FollowSymLinks Indexes + SetHandler uwsgi-handler + uWSGISocket /run/uwsgi/app/searx/socket + + Order deny,allow + Deny from all + # Allow from fd00::/8 192.168.0.0/16 fe80::/10 127.0.0.0/8 ::1 + Allow from all + + </Location> + + </IfModule> + +Enable apache mod_uwsgi and restart apache: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: sh + + a2enmod uwsgi + sudo -H systemctl restart apache2 + +disable logs +============ + +For better privacy you can disable Apache logs. Go back to +``/etc/apache2/apache2.conf`` :ref:`[example] <inranet apache site>` and above +``<Location />`` activate directive: + +.. code:: apache + + CustomLog /dev/null combined + +Restart apache: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: sh + + sudo -H systemctl restart apache2 + +.. warning:: + + You can only disable logs for the whole (virtual) server not for a specific + path. diff --git a/docs/admin/installation-nginx.rst b/docs/admin/installation-nginx.rst new file mode 100644 index 000000000..37d3e7532 --- /dev/null +++ b/docs/admin/installation-nginx.rst @@ -0,0 +1,141 @@ +.. _installation nginx: + +================== +Install with nginx +================== + +.. sidebar:: public HTTP servers + + On public searx instances use an application firewall (:ref:`filtron + <filtron.sh>`). + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +If nginx is not installed (uwsgi will not work with the package +nginx-light): + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: sh + + sudo -H apt-get install nginx + +Hosted at ``/`` +=============== + +Create the configuration file ``/etc/nginx/sites-available/searx`` with this +content: + +.. code:: nginx + + server { + listen 80; + server_name searx.example.com; + root /usr/local/searx/searx; + + location /static { + } + + location / { + include uwsgi_params; + uwsgi_pass unix:/run/uwsgi/app/searx/socket; + } + } + +Create a symlink to sites-enabled: + +.. code:: sh + + sudo -H ln -s /etc/nginx/sites-available/searx /etc/nginx/sites-enabled/searx + +Restart service: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: sh + + sudo -H systemctl restart nginx + sudo -H systemctl restart uwsgi + +from subdirectory URL (``/searx``) +================================== + +Add this configuration in the server config file +``/etc/nginx/sites-enabled/default``: + +.. code:: nginx + + location /searx/static { + alias /usr/local/searx/searx/static; + } + + location /searx { + uwsgi_param SCRIPT_NAME /searx; + include uwsgi_params; + uwsgi_pass unix:/run/uwsgi/app/searx/socket; + } + + +**OR** using reverse proxy (Please, note that reverse proxy advised to be used +in case of single-user or low-traffic instances.) + +.. code:: nginx + + location /searx/static { + alias /usr/local/searx/searx/static; + } + + location /searx { + proxy_pass http://127.0.0.1:8888; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Scheme $scheme; + proxy_set_header X-Script-Name /searx; + proxy_buffering off; + } + +Enable ``base_url`` in ``searx/settings.yml`` + +.. code:: yaml + + base_url : http://your.domain.tld/searx/ + +Restart service: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: sh + + sudo -H systemctl restart nginx + sudo -H systemctl restart uwsgi + + +disable logs +============ + +For better privacy you can disable nginx logs about searx. How to proceed: +below ``uwsgi_pass`` in ``/etc/nginx/sites-available/default`` add: + +.. code:: nginx + + access_log /dev/null; + error_log /dev/null; + +Restart service: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: sh + + sudo -H systemctl restart nginx diff --git a/docs/admin/installation.rst b/docs/admin/installation.rst index 15800fc01..a4c7eb8b3 100644 --- a/docs/admin/installation.rst +++ b/docs/admin/installation.rst @@ -4,44 +4,64 @@ Installation ============ -.. contents:: - :depth: 3 +.. sidebar:: Searx server setup + + - :ref:`installation nginx` + - :ref:`installation apache` + + If you do not have any special preferences, it is recommend to use + :ref:`searx.sh`. + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. _installation basic: Basic installation ================== -Step by step installation for Debian/Ubuntu with virtualenv. For Ubuntu, be sure -to have enable universe repository. +Step by step installation with virtualenv. For Ubuntu, be sure to have enable +universe repository. Install packages: -.. code:: sh +.. tabs:: + + .. group-tab:: Ubuntu / debian - $ sudo -H apt-get install \ - git build-essential libxslt-dev \ - python-dev python-virtualenv python-babel \ - zlib1g-dev libffi-dev libssl-dev + .. code-block:: sh + + $ sudo -H apt-get install -m \ + git build-essential + libxslt-dev python3-dev python3-babel python3-venv \ + zlib1g-dev libffi-dev libssl-dev Install searx: .. code:: sh - cd /usr/local - sudo -H git clone https://github.com/asciimoo/searx.git - sudo -H useradd searx -d /usr/local/searx + sudo -H useradd searx --system --disabled-password -d /usr/local/searx + sudo -H usermod -a -G shadow searx + cd /usr/local/searx + sudo -H git clone https://github.com/asciimoo/searx.git searx-src sudo -H chown searx:searx -R /usr/local/searx -Install dependencies in a virtualenv: +Install virtualenv: .. code:: sh - cd /usr/local/searx sudo -H -u searx -i + (searx)$ python3 -m venv searx-pyenv + (searx)$ echo 'source ~/searx-pyenv/bin/activate' > ~/.profile + +Exit the searx bash and restart a new to install the searx dependencies: .. code:: sh - (searx)$ virtualenv searx-ve - (searx)$ . ./searx-ve/bin/activate + sudo -H -u searx -i + (searx)$ cd searx-src (searx)$ ./manage.sh update_packages Configuration @@ -49,7 +69,9 @@ Configuration .. code:: sh - sed -i -e "s/ultrasecretkey/`openssl rand -hex 16`/g" searx/settings.yml + sudo -H -u searx -i + (searx)$ cd searx-src + (searx)$ sed -i -e "s/ultrasecretkey/`openssl rand -hex 16`/g" searx/settings.yml Edit searx/settings.yml if necessary. @@ -60,7 +82,9 @@ Start searx: .. code:: sh - python searx/webapp.py + sudo -H -u searx -i + (searx)$ cd searx-src + (searx)$ python3 searx/webapp.py Go to http://localhost:8888 @@ -70,254 +94,118 @@ If everything works fine, disable the debug option in settings.yml: sed -i -e "s/debug : True/debug : False/g" searx/settings.yml -At this point searx is not demonized ; uwsgi allows this. - -You can exit the virtualenv and the searx user bash (enter exit command -twice). +At this point searx is not demonized ; uwsgi allows this. You can exit the +virtualenv and the searx user bash (enter exit command twice). uwsgi ===== Install packages: -.. code:: sh - - sudo -H apt-get install \ - uwsgi uwsgi-plugin-python - -Create the configuration file ``/etc/uwsgi/apps-available/searx.ini`` with this -content: - -.. code:: ini - - [uwsgi] - # Who will run the code - uid = searx - gid = searx - - # disable logging for privacy - disable-logging = true - - # Number of workers (usually CPU count) - workers = 4 +.. tabs:: - # The right granted on the created socket - chmod-socket = 666 + .. group-tab:: Ubuntu / debian - # Plugin to use and interpretor config - single-interpreter = true - master = true - plugin = python - lazy-apps = true - enable-threads = true + .. code-block:: bash - # Module to import - module = searx.webapp + sudo -H apt-get install uwsgi uwsgi-plugin-python3 - # Support running the module from a webserver subdirectory. - route-run = fixpathinfo: - - # Virtualenv and python path - virtualenv = /usr/local/searx/searx-ve/ - pythonpath = /usr/local/searx/ - chdir = /usr/local/searx/searx/ - -Activate the uwsgi application and restart: - -.. code:: sh - - cd /etc/uwsgi/apps-enabled - ln -s ../apps-available/searx.ini - /etc/init.d/uwsgi restart - -Web server -========== - -with nginx ----------- - -If nginx is not installed (uwsgi will not work with the package -nginx-light): - -.. code:: sh - - sudo -H apt-get install nginx - -Hosted at / -~~~~~~~~~~~ - -Create the configuration file ``/etc/nginx/sites-available/searx`` with this +Create the configuration file ``/etc/uwsgi/apps-available/searx.ini`` with this content: -.. code:: nginx - - server { - listen 80; - server_name searx.example.com; - root /usr/local/searx/searx; - - location /static { - } - - location / { - include uwsgi_params; - uwsgi_pass unix:/run/uwsgi/app/searx/socket; - } - } - -Create a symlink to sites-enabled: - -.. code:: sh - - sudo -H ln -s /etc/nginx/sites-available/searx /etc/nginx/sites-enabled/searx - -Restart service: - -.. code:: sh - - sudo -H service nginx restart - sudo -H service uwsgi restart - -from subdirectory URL (/searx) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Add this configuration in the server config file -``/etc/nginx/sites-enabled/default``: - -.. code:: nginx - - location /searx/static { - alias /usr/local/searx/searx/static; - } - - location /searx { - uwsgi_param SCRIPT_NAME /searx; - include uwsgi_params; - uwsgi_pass unix:/run/uwsgi/app/searx/socket; - } - - -**OR** using reverse proxy (Please, note that reverse proxy advised to be used -in case of single-user or low-traffic instances.) - -.. code:: nginx - - location /searx/static { - alias /usr/local/searx/searx/static; - } - - location /searx { - proxy_pass http://127.0.0.1:8888; - proxy_set_header Host $host; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Scheme $scheme; - proxy_set_header X-Script-Name /searx; - proxy_buffering off; - } - - -Enable ``base_url`` in ``searx/settings.yml`` - -.. code:: yaml - - base_url : http://your.domain.tld/searx/ +.. code:: ini -Restart service: + [uwsgi] -.. code:: sh + # uWSGI core + # ---------- + # + # https://uwsgi-docs.readthedocs.io/en/latest/Options.html#uwsgi-core - sudo -H service nginx restart - sudo -H service uwsgi restart + # Who will run the code + uid = searx + gid = searx -disable logs -^^^^^^^^^^^^ + # chdir to specified directory before apps loading + chdir = /usr/local/searx/searx-src/searx -for better privacy you can disable nginx logs about searx. + # disable logging for privacy + disable-logging = true -how to proceed: below ``uwsgi_pass`` in ``/etc/nginx/sites-available/default`` -add: + # The right granted on the created socket + chmod-socket = 666 -.. code:: nginx + # Plugin to use and interpretor config + single-interpreter = true - access_log /dev/null; - error_log /dev/null; + # enable master process + master = true -Restart service: + # load apps in each worker instead of the master + lazy-apps = true -.. code:: sh + # load uWSGI plugins + plugin = python3,http - sudo -H service nginx restart + # By default the Python plugin does not initialize the GIL. This means your + # app-generated threads will not run. If you need threads, remember to enable + # them with enable-threads. Running uWSGI in multithreading mode (with the + # threads options) will automatically enable threading support. This *strange* + # default behaviour is for performance reasons. + enable-threads = true -with apache ------------ + # plugin: python + # -------------- + # + # https://uwsgi-docs.readthedocs.io/en/latest/Options.html#plugin-python -Add wsgi mod: + # load a WSGI module + module = searx.webapp -.. code:: sh + # set PYTHONHOME/virtualenv + virtualenv = /usr/local/searx/searx-pyenv - sudo -H apt-get install libapache2-mod-uwsgi - sudo -H a2enmod uwsgi + # add directory (or glob) to pythonpath + pythonpath = /usr/local/searx/searx-src -Add this configuration in the file ``/etc/apache2/apache2.conf``: -.. code:: apache + # plugin http + # ----------- + # + # https://uwsgi-docs.readthedocs.io/en/latest/Options.html#plugin-http - <Location /> - Options FollowSymLinks Indexes - SetHandler uwsgi-handler - uWSGISocket /run/uwsgi/app/searx/socket - </Location> + # Native HTTP support: https://uwsgi-docs.readthedocs.io/en/latest/HTTP.html + http = 127.0.0.1:8888 -Note that if your instance of searx is not at the root, you should change -``<Location />`` by the location of your instance, like ``<Location /searx>``. - -Restart Apache: +Activate the uwsgi application and restart: .. code:: sh - sudo -H /etc/init.d/apache2 restart - -disable logs -~~~~~~~~~~~~ - -For better privacy you can disable Apache logs. - -.. warning:: - - You can only disable logs for the whole (virtual) server not for a specific - path. - -Go back to ``/etc/apache2/apache2.conf`` and above ``<Location />`` add: - -.. code:: apache - - CustomLog /dev/null combined - -Restart Apache: - -.. code:: sh + cd /etc/uwsgi/apps-enabled + ln -s ../apps-available/searx.ini + /etc/init.d/uwsgi restart - sudo -H /etc/init.d/apache2 restart How to update ============= .. code:: sh - cd /usr/local/searx sudo -H -u searx -i - -.. code:: sh - - (searx)$ . ./searx-ve/bin/activate (searx)$ git stash (searx)$ git pull origin master (searx)$ git stash apply (searx)$ ./manage.sh update_packages -.. code:: sh +Restart uwsgi: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: sh - sudo -H service uwsgi restart + sudo -H systemctl restart uwsgi Docker ====== diff --git a/docs/admin/morty.rst b/docs/admin/morty.rst index 7d7b34492..9af9b6ae9 100644 --- a/docs/admin/morty.rst +++ b/docs/admin/morty.rst @@ -1,3 +1,6 @@ + +.. _searx_morty: + ========================= How to setup result proxy ========================= diff --git a/docs/admin/settings.rst b/docs/admin/settings.rst index 0bfdcc6cb..59a406a28 100644 --- a/docs/admin/settings.rst +++ b/docs/admin/settings.rst @@ -4,11 +4,17 @@ ``settings.yml`` ================ +This page describe the options possibilities of the :origin:`searx/settings.yml` +file. + .. sidebar:: Further reading .. - :ref:`search API` -This page describe the options possibilities of the settings.yml file. +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry .. _settings global: diff --git a/docs/blog/index.rst b/docs/blog/index.rst index 52fa3f126..bc90f4502 100644 --- a/docs/blog/index.rst +++ b/docs/blog/index.rst @@ -3,7 +3,8 @@ Blog ==== .. toctree:: - :maxdepth: 1 + :maxdepth: 2 + :caption: Contents python3 admin diff --git a/docs/conf.py b/docs/conf.py index ebcd102b2..8f11b1c6c 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -5,6 +5,7 @@ from searx.version import VERSION_STRING from pallets_sphinx_themes import ProjectLink GIT_URL = os.environ.get("GIT_URL", "https://github.com/asciimoo/searx") +GIT_BRANCH =os.environ.get("GIT_BRANCH", "master") SEARX_URL = os.environ.get("SEARX_URL", "https://searx.me") DOCS_URL = os.environ.get("DOCS_URL", "https://asciimoo.github.io/searx/") @@ -35,7 +36,7 @@ extlinks['wiki'] = ('https://github.com/asciimoo/searx/wiki/%s', ' ') extlinks['pull'] = ('https://github.com/asciimoo/searx/pull/%s', 'PR ') # links to custom brand -extlinks['origin'] = (GIT_URL + '/blob/master/%s', 'git://') +extlinks['origin'] = (GIT_URL + '/blob/' + GIT_BRANCH + '/%s', 'git://') extlinks['patch'] = (GIT_URL + '/commit/%s', '#') extlinks['search'] = (SEARX_URL + '/%s', '#') extlinks['docs'] = (DOCS_URL + '/%s', 'docs: ') @@ -61,6 +62,7 @@ extensions = [ "pallets_sphinx_themes", "sphinx_issues", # https://github.com/sloria/sphinx-issues/blob/master/README.rst "sphinxcontrib.jinja", # https://github.com/tardyp/sphinx-jinja + "sphinxcontrib.programoutput", # https://github.com/NextThought/sphinxcontrib-programoutput 'linuxdoc.rstFlatTable', # Implementation of the 'flat-table' reST-directive. 'linuxdoc.kfigure', # Sphinx extension which implements scalable image handling. "sphinx_tabs.tabs", # https://github.com/djungelorm/sphinx-tabs diff --git a/docs/dev/contribution_guide.rst b/docs/dev/contribution_guide.rst index 459dfb448..f9e9569f9 100644 --- a/docs/dev/contribution_guide.rst +++ b/docs/dev/contribution_guide.rst @@ -4,6 +4,11 @@ How to contribute ================= +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + Prime directives: Privacy, Hackability ====================================== diff --git a/docs/dev/index.rst b/docs/dev/index.rst index cb913a82b..ba0a25a9c 100644 --- a/docs/dev/index.rst +++ b/docs/dev/index.rst @@ -3,7 +3,8 @@ Developer documentation ======================= .. toctree:: - :maxdepth: 1 + :maxdepth: 2 + :caption: Contents quickstart contribution_guide diff --git a/docs/dev/makefile.rst b/docs/dev/makefile.rst index f5957001c..ed9942f4c 100644 --- a/docs/dev/makefile.rst +++ b/docs/dev/makefile.rst @@ -11,6 +11,8 @@ Makefile Targets Before looking deeper at the targets, first read about :ref:`makefile setup` and :ref:`make pyenv`. + To install system requirements follow :ref:`buildhosts`. + With the aim to simplify development cycles, started with :pull:`1756` a ``Makefile`` based boilerplate was added. If you are not familiar with Makefiles, we recommend to read gnu-make_ introduction. @@ -37,15 +39,17 @@ Calling the ``help`` target gives a first overview:: .. _makefile setup: -Setup -===== +Makefile setup +============== .. _git stash: https://git-scm.com/docs/git-stash -The main setup is done in the :origin:`Makefile`:: +The main setup is done in the :origin:`.config.sh` (read :ref:`toolboxing +setup`):: - export GIT_URL=https://github.com/asciimoo/searx export SEARX_URL=https://searx.me + + export GIT_URL=https://github.com/asciimoo/searx export DOCS_URL=https://asciimoo.github.io/searx .. sidebar:: fork & upstream @@ -170,7 +174,7 @@ e.g.: .. code:: sh - $ make test.pep8 test.unit + $ make test.pep8 test.unit test.sh . ./local/py3/bin/activate; ./manage.sh pep8_check [!] Running pep8 check . ./local/py3/bin/activate; ./manage.sh unit_tests diff --git a/docs/dev/reST.rst b/docs/dev/reST.rst index 4835cbbcf..7b253d9ec 100644 --- a/docs/dev/reST.rst +++ b/docs/dev/reST.rst @@ -326,7 +326,7 @@ Literal blocks The simplest form of :duref:`literal-blocks` is a indented block introduced by two colons (``::``). For highlighting use :dudir:`highlight` or :ref:`reST code` directive. To include literals from external files use directive -:dudir:`literalinclude`. +:rst:dir:`literalinclude`. .. _reST literal: @@ -1312,9 +1312,8 @@ others are basic-tabs_ and code-tabs_. Below a *group-tab* example from .. literalinclude:: ../admin/buildhosts.rst :language: reST - :start-after: .. _system requirements: - :end-before: .. _system requirements END: - + :start-after: .. SNIP sh lint requirements + :end-before: .. SNAP sh lint requirements .. _math: diff --git a/docs/index.rst b/docs/index.rst index b62f4dc5d..9e590867c 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -2,7 +2,14 @@ Welcome to searx ================ -Search without being tracked. + *Search without being tracked.* + +Searx is a free internet metasearch engine which aggregates results from more +than 70 search services. Users are neither tracked nor profiled. Additionally, +searx can be used over Tor for online anonymity. + +Get started with searx by using one of the Searx-instances_. If you don't trust +anyone, you can set up your own, see :ref:`installation`. .. sidebar:: Features @@ -16,19 +23,14 @@ Search without being tracked. - Hosted by organizations, such as *La Quadrature du Net*, which promote digital rights -Searx is a free internet metasearch engine which aggregates results from more -than 70 search services. Users are neither tracked nor profiled. Additionally, -searx can be used over Tor for online anonymity. - -Get started with searx by using one of the Searx-instances_. If you don't trust -anyone, you can set up your own, see :ref:`installation`. - .. toctree:: :maxdepth: 2 + :caption: Contents user/index admin/index dev/index + utils/index blog/index .. _Searx-instances: https://searx.space diff --git a/docs/user/index.rst b/docs/user/index.rst index b13aca216..96d11bf8a 100644 --- a/docs/user/index.rst +++ b/docs/user/index.rst @@ -3,7 +3,8 @@ User documentation ================== .. toctree:: - :maxdepth: 1 + :maxdepth: 2 + :caption: Contents search_syntax own-instance diff --git a/docs/user/own-instance.rst b/docs/user/own-instance.rst index a2f736562..4876abe99 100644 --- a/docs/user/own-instance.rst +++ b/docs/user/own-instance.rst @@ -2,8 +2,10 @@ Why use a private instance? =========================== -"Is it worth to run my own instance?" is a common question among searx users. -Before answering this question, see what options a searx user has. + *"Is it worth to run my own instance?"* + +\.\. is a common question among searx users. Before answering this question, +see what options a searx user has. Public instances are open to everyone who has access to its URL. Usually, these are operated by unknown parties (from the users' point of view). Private diff --git a/docs/user/public_instances.rst b/docs/user/public_instances.rst deleted file mode 100644 index f58ab7a02..000000000 --- a/docs/user/public_instances.rst +++ /dev/null @@ -1,3 +0,0 @@ -:orphan: - -This page page has been moved to `searx.space <https://searx.space/>`__ diff --git a/docs/utils/filtron.sh.rst b/docs/utils/filtron.sh.rst new file mode 100644 index 000000000..1bba1b3e4 --- /dev/null +++ b/docs/utils/filtron.sh.rst @@ -0,0 +1,75 @@ + +.. _filtron.sh: + +==================== +``utils/filtron.sh`` +==================== + +.. sidebar:: further reading + + - :ref:`installation` + - :ref:`searx_filtron` + - :ref:`architecture` + +.. _Go: https://golang.org/ +.. _filtron: https://github.com/asciimoo/filtron +.. _filtron README: https://github.com/asciimoo/filtron/blob/master/README.md + +To simplify installation and maintenance of a filtron instance you can use the +script :origin:`utils/filtron.sh`. In most cases you will install filtron_ +simply by running the command: + +.. code:: bash + + sudo -H ./utils/filtron.sh install all + +The script adds a ``${SERVICE_USER}`` (default:``filtron``) and installs filtron_ +into this user account: + +#. Create a separated user account (``filtron``). +#. Download and install Go_ binary in users $HOME (``~filtron``). +#. Install filtron with the package management of Go_ (``go get -v -u + github.com/asciimoo/filtron``) +#. Setup a proper rule configuration :origin:`[ref] + <utils/templates/etc/filtron/rules.json>` (``/etc/filtron/rules.json``). +#. Setup a systemd service unit :origin:`[ref] + <utils/templates/lib/systemd/system/filtron.service>` + (``/lib/systemd/system/filtron.service``). + +.. _reverse proxy: + +Public Reverse Proxy +==================== + +To install searx in your public HTTP server use: + +.. code:: bash + + sudo -H ./utils/filtron.sh apache install + +.. tabs:: + + .. group-tab:: apache + + .. literalinclude:: ../../utils/templates/etc/apache2/sites-available/searx.conf:filtron + :language: apache + + .. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code-block:: sh + + $ sudo -H a2enmod headers + $ sudo -H a2enmod proxy + $ sudo -H a2enmod proxy_http + + +Overview +======== + +The ``--help`` output of the script is largely self-explanatory +(:ref:`toolboxing common`): + +.. program-output:: ../utils/filtron.sh --help + diff --git a/docs/utils/index.rst b/docs/utils/index.rst new file mode 100644 index 000000000..338914715 --- /dev/null +++ b/docs/utils/index.rst @@ -0,0 +1,53 @@ + +.. _searx_utils: +.. _toolboxing: + +======================= +Tooling box ``utils/*`` +======================= + +In the folder :origin:`utils/` we maintain some tools useful for admins and +developers. + +.. sidebar:: Work needed! + + Our scripts to maintain services do most support only systemd init process + used by debian, ubuntu and many other dists. In general our scripts are only + partially usable on debian systems. We are working on this limitation, if + you have any contribution, please send us your :pull:`PR <../pulls>`, see + :ref:`how to contribute`. + +.. toctree:: + :maxdepth: 2 + :caption: Contents + + searx.sh + filtron.sh + morty.sh + +.. _toolboxing common: + +Common commands +=============== + +Scripts to maintain services often dispose of common commands and environments. + +``shell``: + Opens a shell from the service user ``${SERVICE_USSR}``, very helpful for + troubleshooting. + +``inspect service``: + Shows status and log of the service, most often you have a option to enable + more verbose debug logs. Very helpful for debugging, but be careful not to + enable debugging in a production environment! + +.. _toolboxing setup: + +Tooling box setup +================= + +The main setup is done in the :origin:`.config.sh` (read also :ref:`makefile +setup`). + +.. literalinclude:: ../../.config.sh + :language: bash diff --git a/docs/utils/morty.sh.rst b/docs/utils/morty.sh.rst new file mode 100644 index 000000000..9997ffec5 --- /dev/null +++ b/docs/utils/morty.sh.rst @@ -0,0 +1,39 @@ + +.. _morty: https://github.com/asciimoo/morty +.. _morty's README: https://github.com/asciimoo/morty + +.. _morty.sh: + +================== +``utils/morty.sh`` +================== + +.. sidebar:: further reading + + - :ref:`installation` + - :ref:`architecture` + +To simplify installation and maintenance of a morty_ instance you can use the +script :origin:`utils/morty.sh`. In most cases you will install morty_ simply by +running the command: + +.. code:: bash + + sudo -H ./utils/morty.sh install all + +The script adds a ``${SERVICE_USER}`` (default:``morty``) and installs morty_ +into this user account. + +.. hint:: + + To add morty to your searx instance read chapter :reF:`searx_morty`. + + +Overview +======== + +The ``--help`` output of the script is largely self-explanatory +(:ref:`toolboxing common`): + +.. program-output:: ../utils/morty.sh --help + diff --git a/docs/utils/searx.sh.rst b/docs/utils/searx.sh.rst new file mode 100644 index 000000000..988c08438 --- /dev/null +++ b/docs/utils/searx.sh.rst @@ -0,0 +1,73 @@ + +.. _searx.sh: + +================== +``utils/searx.sh`` +================== + +.. sidebar:: further reading + + - :ref:`installation` + - :ref:`architecture` + - :ref:`filtron.sh` + +To simplify installation and maintenance of a searx instance you can use the +script :origin:`utils/searx.sh`. + +Install +======= + +In most cases you will install searx simply by running the command: + +.. code:: bash + + sudo -H ./utils/searx.sh install all + +The script adds a ``${SERVICE_USER}`` (default:``searx``) and installs searx +into this user account. The installation is described in chapter +:ref:`installation basic`. + +.. _intranet reverse proxy: + +Intranet Reverse Proxy +====================== + +To install searx in your intranet HTTP server use: + +.. code:: bash + + sudo -H ./utils/searx.sh apache install + +.. warning:: + + This setup is **not** suitable **for public instances**, go on with + :ref:`reverse proxy`! + +.. tabs:: + + .. group-tab:: apache + + .. literalinclude:: ../../utils/templates/etc/apache2/sites-available/searx.conf:uwsgi + :language: apache + + .. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code-block:: sh + + $ sudo -H apt install libapache2-mod-uwsgi + + .. group-tab:: Arch Linux + + .. code-block:: sh + + $ sudo pacman -S uwsgi + +Overview +======== + +The ``--help`` output of the script is largely self-explanatory +(:ref:`toolboxing common`): + +.. program-output:: ../utils/searx.sh --help |