From 1d6597b40773b688ca317e1ab1ffe133b370afb1 Mon Sep 17 00:00:00 2001 From: Adam Tauber Date: Tue, 17 Nov 2015 23:13:30 +0100 Subject: [enh] initial structure --- docs/dev/contribution_guide.rst | 71 ++++++++++ docs/dev/install/installation.rst | 277 ++++++++++++++++++++++++++++++++++++++ docs/dev/plugins.rst | 45 +++++++ docs/dev/search_api.rst | 24 ++++ docs/dev/translation.rst | 74 ++++++++++ 5 files changed, 491 insertions(+) create mode 100644 docs/dev/contribution_guide.rst create mode 100644 docs/dev/install/installation.rst create mode 100644 docs/dev/plugins.rst create mode 100644 docs/dev/search_api.rst create mode 100644 docs/dev/translation.rst (limited to 'docs/dev') diff --git a/docs/dev/contribution_guide.rst b/docs/dev/contribution_guide.rst new file mode 100644 index 000000000..5685549eb --- /dev/null +++ b/docs/dev/contribution_guide.rst @@ -0,0 +1,71 @@ +How to contribute +----------------- + +Prime directives: Privacy, Hackability +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Searx has 2 prime directives, privacy-by-design and hackability. The +hackability comes in at least 3 levels: + +- support for search engines +- plugins for altering search behaviour +- hacking searx itself. + +Happy hacking. Observe the lack of "world domination" among the +directives, searx has no intentions for wide mass-adoption, rounded +corners, etc. The prime directive: "privacy" - deserves a seperate +chapter, as it's quite uncommon unfortunately, here it goes: + +Privacy-by-design +^^^^^^^^^^^^^^^^^ + +Searx is a privacy-respecting, hackable meta-search engine. It was born +out of the need for a privacy-respecing search facility that can be +expanded easily to maximise both its search and it's privacy protecting +capabilities. + +Consequences of Privacy-by-design are that some widely used features +work differently or not by default or at all. If some feature reduces +the privacy perserving aspects of searx, it should by default be +switched of, if implemented at all. There is enough search engines +already out there providing such features. = Since privacy-preservation +is a prime goal, if some feature does reduce the protection of searx and +is implemented, care should be taken to educate the user about the +consequences of choosing to enable this. Further features which +implement widely known features in a manner that protects privacy but +thus deviate from the users expectations should also be explained to the +user. Also if you think that something works weird with searx, maybe +it's because of the tool you use is designed in a way to interfere with +privacy respect, submiting a bugreport to the vendor of the tool that +misbehaves might be a good feedback for the vendor to reconsider his +disrespect towards his customers (e.g. GET vs POST requests in various +browsers). + +Remember the other prime directive of searx is to be hackable, so if the +above privacy concerns do not fancy you, simply fork it. + +Code +~~~~ + +Code modifications are accepted in pull requests, don't forget to add +yourself to the AUTHORS file. + +Python code follows all the pep8 standards except maximum line width +which is 120 char. + +Please be sure that the submitted code doesn't break existing tests and +follows coding conventions. + +If new functionality implemented, tests are highly appreciated. + +Translation +~~~~~~~~~~~ + +Translation currently happens on +`transifex `__. Please do not +update translation files in the repo. + +Documentation +~~~~~~~~~~~~~ + +The main place of the documentation is this wiki, updates are welcome. diff --git a/docs/dev/install/installation.rst b/docs/dev/install/installation.rst new file mode 100644 index 000000000..2ce6a6b54 --- /dev/null +++ b/docs/dev/install/installation.rst @@ -0,0 +1,277 @@ +Installation +============ + +Step by step installation for Debian / Ubuntu with virtualenv. + +Source: https://about.okhin.fr/posts/Searx/ with some additions + +How to: `Setup searx in a couple of hours with a free SSL +certificate `__ + +Basic installation +------------------ + +For Ubuntu, be sure to have enable universe repository. + +Install packages : + +.. code:: sh + + sudo apt-get install git build-essential libxslt-dev python-dev python-virtualenv python-pybabel zlib1g-dev libffi-dev libssl-dev + +Install searx : + +.. code:: sh + + cd /usr/local + sudo git clone https://github.com/asciimoo/searx.git + sudo useradd searx -d /usr/local/searx + sudo chown searx:searx -R /usr/local/searx + +Install dependencies in a virtualenv : + +.. code:: sh + + sudo -u searx -i + cd /usr/local/searx + virtualenv searx-ve + . ./searx-ve/bin/activate + pip install -r requirements.txt + python setup.py install + +Configuration +------------- + +.. code:: sh + + sed -i -e "s/ultrasecretkey/`openssl rand -hex 16`/g" searx/settings.yml + +Edit searx/settings.yml if necessary. + +Check +----- + +Start searx : + +.. code:: sh + + python searx/webapp.py + +Go to http://localhost:8888 + +If everything works fine, disable the debug option in settings.yml : + +.. code:: sh + + sed -i -e "s/debug : True/debug : False/g" searx/settings.yml + +At this point searx is not demonized ; uwsgi allows this. + +You can exit the virtualenv and the searx user bash (enter exit command +twice). + +uwsgi +----- + +Install packages : + +.. code:: sh + + sudo apt-get install uwsgi uwsgi-plugin-python + +Create the configuration file /etc/uwsgi/apps-available/searx.ini with +this content : + +:: + + [uwsgi] + # Who will run the code + uid = searx + gid = searx + + # disable logging for privacy + disable-logging = true + + # Number of workers (usually CPU count) + workers = 4 + + # The right granted on the created socket + chmod-socket = 666 + + # Plugin to use and interpretor config + single-interpreter = true + master = true + plugin = python + + # Module to import + module = searx.webapp + + # Virtualenv and python path + virtualenv = /usr/local/searx/searx-ve/ + pythonpath = /usr/local/searx/ + chdir = /usr/local/searx/searx/ + +Activate the uwsgi application and restart : + +.. code:: sh + + cd /etc/uwsgi/apps-enabled + ln -s ../apps-available/searx.ini + /etc/init.d/uwsgi restart + +Web server +---------- + +with nginx +^^^^^^^^^^ + +If nginx is not installed (uwsgi will not work with the package +nginx-light) : + +.. code:: sh + + sudo apt-get install nginx + +Hosted at / +""""""""""" + +Create the configuration file /etc/nginx/sites-available/searx with this +content : + +.. code:: nginx + + server { + listen 80; + server_name searx.example.com; + root /usr/local/searx; + + location / { + include uwsgi_params; + uwsgi_pass unix:/run/uwsgi/app/searx/socket; + } + } + +Restart service : + +.. code:: sh + + sudo service nginx restart + sudo service uwsgi restart + +from subdirectory URL (/searx) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Add this configuration in the server config file +/etc/nginx/sites-available/default : + +.. code:: nginx + + location = /searx { rewrite ^ /searx/; } + location /searx { + try_files $uri @searx; + } + location @searx { + uwsgi_param SCRIPT_NAME /searx; + include uwsgi_params; + uwsgi_modifier1 30; + uwsgi_pass unix:/run/uwsgi/app/searx/socket; + } + +Enable base\_url in searx/settings.yml + +:: + + base_url : http://your.domain.tld/searx/ + +Restart service : + +.. code:: sh + + sudo service nginx restart + sudo service uwsgi restart + +disable logs +~~~~~~~~~~~~ + +for better privacy you can disable nginx logs about searx. + +how to proceed : below ``uwsgi_pass`` in +/etc/nginx/sites-available/default add + +:: + + access_log /dev/null; + error_log /dev/null; + +Restart service : + +.. code:: sh + + sudo service nginx restart + +with apache +----------- + +Add wsgi mod : + +.. code:: sh + + sudo apt-get install libapache2-mod-uwsgi + sudo a2enmod uwsgi + +Add this configuration in the file /etc/apache2/apache2.conf : + +.. code:: apache + + + Options FollowSymLinks Indexes + SetHandler uwsgi-handler + uWSGISocket /run/uwsgi/app/searx/socket + + +Note that if your instance of searx is not at the root, you should +change ```` by the location of your instance, like +````. + +Restart Apache : + +.. code:: sh + + sudo /etc/init.d/apache2 restart + +disable logs +------------ + +For better privacy you can disable Apache logs. + +WARNING : not tested + +WARNING : you can only disable logs for the whole (virtual) server not +for a specific path. + +Go back to /etc/apache2/apache2.conf and above ```` add : + +.. code:: apache + + CustomLog /dev/null combined + +Restart Apache : + +.. code:: sh + + sudo /etc/init.d/apache2 restart + +How to update +------------- + +.. code:: sh + + cd /usr/local/searx + sudo -u searx -i + . ./searx-ve/bin/activate + git stash + git pull origin master + git stash apply + pip install --upgrade -r requirements.txt + sudo service uwsgi restart + diff --git a/docs/dev/plugins.rst b/docs/dev/plugins.rst new file mode 100644 index 000000000..c51fdf0eb --- /dev/null +++ b/docs/dev/plugins.rst @@ -0,0 +1,45 @@ +Plugins +------- + +Plugins can extend/replace functionality of various components inside +searx. + +example\_plugin.py +~~~~~~~~~~~~~~~~~~ + +.. code:: python + + name = 'Example plugin' + description = 'This plugin extends the suggestions with the word "example"' + default_on = False # disable by default + + js_dependencies = tuple() # optional, list of static js files + css_dependencies = tuple() # optional, list of static css files + + + # attach callback to the post search hook + # request: flask request object + # ctx: the whole local context of the post search hook + def post_search(request, ctx): + ctx['search'].suggestions.add('example') + return True + +Currently implemented plugin entry points (a.k.a hooks) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- Pre search hook (``pre_search``) +- Post search hook (``post_search``) +- Result hook (``on_result``) (is called if a new result is added (see + https\_rewrite plugin)) + +Feel free to add more hooks to the code if it is required by a plugin. + +TODO +~~~~ + +- Better documentation +- More hooks +- search hook (is called while searx is requesting results (for + example: things like math-solver), the different hooks are running + parallel) + diff --git a/docs/dev/search_api.rst b/docs/dev/search_api.rst new file mode 100644 index 000000000..cd86fd71c --- /dev/null +++ b/docs/dev/search_api.rst @@ -0,0 +1,24 @@ +Search API +========== + +Search API endpoints: ``/``, ``/search`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Endpoints have equivalent functionality. + +Parameters +^^^^^^^^^^ + ++------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ +| Name | Description | | ++==================+====================================================================================================+=============================+ +| ``q`` | The search query, see :doc:`/user/search_syntax` | required | ++------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ +| ``categories`` | Comma separated list, specifies the active search categories | optional | ++------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ +| ``engines`` | Comma separated list, specifies the active search engines | optional | ++------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ +| ``pageno`` | Search page number | optional (default: ``1``) | ++------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ + +Both ``GET`` and ``POST`` methods are supported. diff --git a/docs/dev/translation.rst b/docs/dev/translation.rst new file mode 100644 index 000000000..0fc02a7de --- /dev/null +++ b/docs/dev/translation.rst @@ -0,0 +1,74 @@ +Translation +=========== + +run these commands in the root directory of searx + +Add new language +~~~~~~~~~~~~~~~~ + +``pybabel init -i messages.pot -d searx/translations -l it`` + +Update .po files +~~~~~~~~~~~~~~~~ + +``./utils/update-translations.sh`` + +You may have errors here. In that case, edit the +``update-translations.sh`` script to change ``pybabel`` to +``pybabel-python2`` + +After this step, you can modify the .po files. + +Compile translations +~~~~~~~~~~~~~~~~~~~~ + +``pybabel compile -d searx/translations`` + +Transifex stuff +~~~~~~~~~~~~~~~ + +Init Project +^^^^^^^^^^^^ + +.. code:: shell + + tx set --auto-local -r searx.messagespo 'searx/translations//LC_MESSAGES/messages.po' \ + --source-lang en --type PO --source-file messages.pot --execute + +http://docs.transifex.com/developer/client/set + +*TODO: mapping between transifex and searx* + +Get translations +^^^^^^^^^^^^^^^^ + +.. code:: shell + + tx pull -a + +http://docs.transifex.com/developer/client/pull + +Upload source File +^^^^^^^^^^^^^^^^^^ + +:: + + tx push -s + +Upload all Translation +^^^^^^^^^^^^^^^^^^^^^^ + +:: + + tx push -s -t + +upload specifc Translation (only for admins) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + tx push -t -l tr + +http://docs.transifex.com/developer/client/push + +*TODO: upload empty files? (new translations)* -- cgit v1.2.3 From 11d6d80d8831459161ac08ad0587ab8775ffd3e3 Mon Sep 17 00:00:00 2001 From: Adam Tauber Date: Sat, 28 Nov 2015 19:26:45 +0100 Subject: [doc] engines overview added --- docs/dev/engine_overview.rst | 314 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 314 insertions(+) create mode 100644 docs/dev/engine_overview.rst (limited to 'docs/dev') diff --git a/docs/dev/engine_overview.rst b/docs/dev/engine_overview.rst new file mode 100644 index 000000000..7f170e9d6 --- /dev/null +++ b/docs/dev/engine_overview.rst @@ -0,0 +1,314 @@ +Engine overview +=============== + + +searx is a `metasearch-engine `__, +so it is using different search engines to provide better results. + +Because there is no general search-api which can be used for every +search-engine, there must be build an adapter between searx and the +external search-engine. This adapters are stored in the folder +`*searx/engines* `__, +and this site is build to make an general documentation about this +engines + + +.. contents:: + :depth: 3 + +general engine configuration +---------------------------- + +It is required to tell searx what results can the engine provide. The +arguments can be inserted in the engine file, or in the settings file +(normally ``settings.yml``). The arguments in the settings file override +the one in the engine file. + +Really, it is for most options no difference if there are contained in +the engine-file or in the settings. But there is a standard where to +place specific arguments by default. + + +engine-file +~~~~~~~~~~~ + ++---------------------+-----------+-----------------------------------------+ +| argument | type | information | ++=====================+===========+=========================================+ +| categories | list | pages, in which the engine is working | ++---------------------+-----------+-----------------------------------------+ +| paging | boolean | support multible pages | ++---------------------+-----------+-----------------------------------------+ +| language\_support | boolean | support language choosing | ++---------------------+-----------+-----------------------------------------+ + +settings.yml +~~~~~~~~~~~~ + ++------------+----------+-----------------------------------------------+ +| argument | type | information | ++============+==========+===============================================+ +| name | string | name of search-engine | ++------------+----------+-----------------------------------------------+ +| engine | string | name of searx-engine (filename without .py) | ++------------+----------+-----------------------------------------------+ +| shortcut | string | shortcut of search-engine | ++------------+----------+-----------------------------------------------+ +| timeout | string | specific timeout for search-engine | ++------------+----------+-----------------------------------------------+ + +overrides +~~~~~~~~~ + +There are some options, with have default values in the engine, but are +often overwritten by the settings. If the option is assigned in the +engine-file with ``None`` it has to be redefined in the settings, +otherwise searx is not starting with that engine. + +The naming of that overrides can be wathever you want. But we recommend +the using of already used overrides if possible: + ++-----------------------+----------+--------------------------------------------------------------+ +| argument | type | information | ++=======================+==========+==============================================================+ +| base\_url | string | base-url, can be overwrite to use same engine on other url | ++-----------------------+----------+--------------------------------------------------------------+ +| number\_of\_results | int | maximum number of results per request | ++-----------------------+----------+--------------------------------------------------------------+ +| language | string | ISO code of language and country like en\_US | ++-----------------------+----------+--------------------------------------------------------------+ +| api\_key | string | api-key if required by engine | ++-----------------------+----------+--------------------------------------------------------------+ + +example-code +~~~~~~~~~~~~ + +.. code:: python + + # engine dependent config + categories = ['general'] + paging = True + language_support = True + +doing request +------------- + +To perform a search you have to specific at least a url on which the +request is performing + +passed arguments +~~~~~~~~~~~~~~~~ + +This arguments can be used to calculate the search-query. Furthermore, +some of that parameters are filled with default values which can be +changed for special purpose. + ++----------------------+------------+------------------------------------------------------------------------+ +| argument | type | default-value, informations | ++======================+============+========================================================================+ +| url | string | ``''`` | ++----------------------+------------+------------------------------------------------------------------------+ +| method | string | ``'GET'`` | ++----------------------+------------+------------------------------------------------------------------------+ +| headers | set | ``{}`` | ++----------------------+------------+------------------------------------------------------------------------+ +| data | set | ``{}`` | ++----------------------+------------+------------------------------------------------------------------------+ +| cookies | set | ``{}`` | ++----------------------+------------+------------------------------------------------------------------------+ +| verify | boolean | ``True`` | ++----------------------+------------+------------------------------------------------------------------------+ +| headers.User-Agent | string | a random User-Agent | ++----------------------+------------+------------------------------------------------------------------------+ +| category | string | current category, like ``'general'`` | ++----------------------+------------+------------------------------------------------------------------------+ +| started | datetime | current date-time | ++----------------------+------------+------------------------------------------------------------------------+ +| pageno | int | current pagenumber | ++----------------------+------------+------------------------------------------------------------------------+ +| language | string | specific language code like ``'en_US'``, or ``'all'`` if unspecified | ++----------------------+------------+------------------------------------------------------------------------+ + +parsed arguments +~~~~~~~~~~~~~~~~ + +The function ``def request(query, params):`` is always returning the +``params`` variable back. Inside searx, the following paramters can be +used to specific a search-request: + ++------------+-----------+----------------------------------------------------------+ +| argument | type | information | ++============+===========+==========================================================+ +| url | string | requested url | ++------------+-----------+----------------------------------------------------------+ +| method | string | HTTP request methode | ++------------+-----------+----------------------------------------------------------+ +| headers | set | HTTP header informations | ++------------+-----------+----------------------------------------------------------+ +| data | set | HTTP data informations (parsed if ``method != 'GET'``) | ++------------+-----------+----------------------------------------------------------+ +| cookies | set | HTTP cookies | ++------------+-----------+----------------------------------------------------------+ +| verify | boolean | Performing SSL-Validity check | ++------------+-----------+----------------------------------------------------------+ + +example-code +~~~~~~~~~~~~ + +.. code:: python + + # search-url + base_url = 'https://example.com/' + search_string = 'search?{query}&page={page}' + + # do search-request + def request(query, params): + search_path = search_string.format( + query=urlencode({'q': query}), + page=params['pageno']) + + params['url'] = base_url + search_path + + return params + +returning results +----------------- + +Searx has the possiblity to return results in different media-types. +Currently the following media-types are supported: + +- default +- images +- videos +- torrent +- map + +to set another media-type as default, you must set the parameter +``template`` to the required type. + +default +~~~~~~~ + ++--------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| result-parameter | information | ++====================+====================================================================================================================================+ +| url | string, which is representing the url of the result | ++--------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| title | string, which is representing the title of the result | ++--------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| content | string, which is giving a general result-text | ++--------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| publishedDate | `datetime.datetime `__, represent when the result is published | ++--------------------+------------------------------------------------------------------------------------------------------------------------------------+ + +images +~~~~~~ + +to use this template, the parameter + ++--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+ +| result-parameter | information | ++====================+===========================================================================================================================================================+ +| template | is set to ``images.html`` | ++--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+ +| url | string, which is representing the url to the result site | ++--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+ +| title | string, which is representing the title of the result *(partly implemented)* | ++--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+ +| content | *(partly implemented)* | ++--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+ +| publishedDate | `datetime.datetime `__, represent when the result is published *(partly implemented)* | ++--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+ +| img\_src | string, which is representing the url to the result image | ++--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+ +| thumbnail\_src | string, which is representing the url to a small-preview image | ++--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+ + +videos +~~~~~~ + ++--------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| result-parameter | information | ++====================+====================================================================================================================================+ +| template | is set to ``videos.html`` | ++--------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| url | string, which is representing the url of the result | ++--------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| title | string, which is representing the title of the result | ++--------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| content | *(not implemented yet)* | ++--------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| publishedDate | `datetime.datetime `__, represent when the result is published | ++--------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| thumbnail | string, which is representing the url to a small-preview image | ++--------------------+------------------------------------------------------------------------------------------------------------------------------------+ + +torrent +~~~~~~~ + ++------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ +| result-parameter | information | ++------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ +| template | is set to ```torrent.html``` | ++------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ +| url | string, which is representing the url of the result | ++------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ +| title | string, which is representing the title of the result | ++------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ +| content | string, which is giving a general result-text | ++------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ +| publishedDate | [datetime.datetime](https://docs.python.org/2/library/datetime.html#datetime-objects), represent when the result is published _(not implemented yet)_ | ++------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ +| seed | int, number of seeder | ++------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ +| leech | int, number of leecher | ++------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ +| filesize | int, size of file in bytes | ++------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ +| files | int, number of files | ++------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ +| magnetlink | string, which is the [magnetlink](https://en.wikipedia.org/wiki/Magnet_URI_scheme) of the result | ++------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ +| torrentfile | string, which is the torrentfile of the result | ++------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ + + +map +~~~ + ++-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| result-parameter | information | ++=========================+====================================================================================================================================+ +| url | string, which is representing the url of the result | ++-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| title | string, which is representing the title of the result | ++-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| content | string, which is giving a general result-text | ++-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| publishedDate | `datetime.datetime `__, represent when the result is published | ++-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| latitude | latitude of result (in decimal format) | ++-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| longitude | longitude of result (in decimal format) | ++-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| boundingbox | boundingbox of result (array of 4. values ``[lat-min, lat-max, lon-min, lon-max]``) | ++-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| geojson | geojson of result (http://geojson.org) | ++-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| osm.type | type of osm-object (if OSM-Result) | ++-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| osm.id | id of osm-object (if OSM-Result) | ++-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| address.name | name of object | ++-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| address.road | street adress of object | ++-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| address.house\_number | house number of object | ++-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| address.locality | city, place of object | ++-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| address.postcode | postcode of object | ++-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ +| address.country | country of object | ++-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ + -- cgit v1.2.3 From ae69a58089cc195d211cd195c79cd60bb49a474f Mon Sep 17 00:00:00 2001 From: Noemi Vanyi Date: Tue, 1 Dec 2015 15:32:06 +0100 Subject: [doc] grammar & format fixes in engine overview --- docs/dev/engine_overview.rst | 185 +++++++++++++++++++++---------------------- 1 file changed, 91 insertions(+), 94 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/engine_overview.rst b/docs/dev/engine_overview.rst index 7f170e9d6..05a54b641 100644 --- a/docs/dev/engine_overview.rst +++ b/docs/dev/engine_overview.rst @@ -3,14 +3,13 @@ Engine overview searx is a `metasearch-engine `__, -so it is using different search engines to provide better results. +so it uses different search engines to provide better results. -Because there is no general search-api which can be used for every -search-engine, there must be build an adapter between searx and the -external search-engine. This adapters are stored in the folder -`*searx/engines* `__, -and this site is build to make an general documentation about this -engines +Because there is no general search API which could be used for every +search engine, an adapter has to be built between searx and the +external search engines. Adapters are stored under the folder +`searx/engines +`__. .. contents:: @@ -19,17 +18,16 @@ engines general engine configuration ---------------------------- -It is required to tell searx what results can the engine provide. The -arguments can be inserted in the engine file, or in the settings file +It is required to tell searx the type of results the engine provides. The +arguments can be set in the engine file or in the settings file (normally ``settings.yml``). The arguments in the settings file override -the one in the engine file. +the ones in the engine file. -Really, it is for most options no difference if there are contained in -the engine-file or in the settings. But there is a standard where to -place specific arguments by default. +It does not matter if an options is stored in the engine file or in the +settings. However, the standard way is the following: -engine-file +engine file ~~~~~~~~~~~ +---------------------+-----------+-----------------------------------------+ @@ -60,27 +58,27 @@ settings.yml overrides ~~~~~~~~~ -There are some options, with have default values in the engine, but are -often overwritten by the settings. If the option is assigned in the -engine-file with ``None`` it has to be redefined in the settings, -otherwise searx is not starting with that engine. - -The naming of that overrides can be wathever you want. But we recommend -the using of already used overrides if possible: - -+-----------------------+----------+--------------------------------------------------------------+ -| argument | type | information | -+=======================+==========+==============================================================+ -| base\_url | string | base-url, can be overwrite to use same engine on other url | -+-----------------------+----------+--------------------------------------------------------------+ -| number\_of\_results | int | maximum number of results per request | -+-----------------------+----------+--------------------------------------------------------------+ -| language | string | ISO code of language and country like en\_US | -+-----------------------+----------+--------------------------------------------------------------+ -| api\_key | string | api-key if required by engine | -+-----------------------+----------+--------------------------------------------------------------+ - -example-code +A few of the options have default values in the engine, but are +often overwritten by the settings. If ``None`` is assigned to an option +in the engine file, it has to be redefined in the settings, +otherwise searx will not start with that engine. + +The naming of that overrides is arbitrary. But the recommended +overrides are the following: + ++-----------------------+----------+----------------------------------------------------------------+ +| argument | type | information | ++=======================+==========+================================================================+ +| base\_url | string | base-url, can be overwritten to use same engine on other URL | ++-----------------------+----------+----------------------------------------------------------------+ +| number\_of\_results | int | maximum number of results per request | ++-----------------------+----------+----------------------------------------------------------------+ +| language | string | ISO code of language and country like en\_US | ++-----------------------+----------+----------------------------------------------------------------+ +| api\_key | string | api-key if required by engine | ++-----------------------+----------+----------------------------------------------------------------+ + +example code ~~~~~~~~~~~~ .. code:: python @@ -90,21 +88,20 @@ example-code paging = True language_support = True -doing request -------------- +making a request +---------------- -To perform a search you have to specific at least a url on which the -request is performing +To perform a search an URL have to be specified. In addition to +specifying an URL, arguments can be passed to the query. passed arguments ~~~~~~~~~~~~~~~~ -This arguments can be used to calculate the search-query. Furthermore, -some of that parameters are filled with default values which can be -changed for special purpose. +These arguments can be used to construct the search query. Furthermore, +parameters with default value can be redefined for special purposes. +----------------------+------------+------------------------------------------------------------------------+ -| argument | type | default-value, informations | +| argument | type | default-value, information | +======================+============+========================================================================+ | url | string | ``''`` | +----------------------+------------+------------------------------------------------------------------------+ @@ -132,27 +129,27 @@ changed for special purpose. parsed arguments ~~~~~~~~~~~~~~~~ -The function ``def request(query, params):`` is always returning the -``params`` variable back. Inside searx, the following paramters can be -used to specific a search-request: - -+------------+-----------+----------------------------------------------------------+ -| argument | type | information | -+============+===========+==========================================================+ -| url | string | requested url | -+------------+-----------+----------------------------------------------------------+ -| method | string | HTTP request methode | -+------------+-----------+----------------------------------------------------------+ -| headers | set | HTTP header informations | -+------------+-----------+----------------------------------------------------------+ -| data | set | HTTP data informations (parsed if ``method != 'GET'``) | -+------------+-----------+----------------------------------------------------------+ -| cookies | set | HTTP cookies | -+------------+-----------+----------------------------------------------------------+ -| verify | boolean | Performing SSL-Validity check | -+------------+-----------+----------------------------------------------------------+ - -example-code +The function ``def request(query, params):`` always returns the +``params`` variable. Inside searx, the following paramters can be +used to specify a search request: + ++------------+-----------+---------------------------------------------------------+ +| argument | type | information | ++============+===========+=========================================================+ +| url | string | requested url | ++------------+-----------+---------------------------------------------------------+ +| method | string | HTTP request method | ++------------+-----------+---------------------------------------------------------+ +| headers | set | HTTP header information | ++------------+-----------+---------------------------------------------------------+ +| data | set | HTTP data information (parsed if ``method != 'GET'``) | ++------------+-----------+---------------------------------------------------------+ +| cookies | set | HTTP cookies | ++------------+-----------+---------------------------------------------------------+ +| verify | boolean | Performing SSL-Validity check | ++------------+-----------+---------------------------------------------------------+ + +example code ~~~~~~~~~~~~ .. code:: python @@ -171,10 +168,10 @@ example-code return params -returning results ------------------ +returned results +---------------- -Searx has the possiblity to return results in different media-types. +Searx is able to return results of different media-types. Currently the following media-types are supported: - default @@ -183,8 +180,8 @@ Currently the following media-types are supported: - torrent - map -to set another media-type as default, you must set the parameter -``template`` to the required type. +To set another media-type as default, the parameter +``template`` must be set to the desired type. default ~~~~~~~ @@ -246,31 +243,31 @@ videos torrent ~~~~~~~ -+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ -| result-parameter | information | -+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ -| template | is set to ```torrent.html``` | -+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ -| url | string, which is representing the url of the result | -+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ -| title | string, which is representing the title of the result | -+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ -| content | string, which is giving a general result-text | -+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ -| publishedDate | [datetime.datetime](https://docs.python.org/2/library/datetime.html#datetime-objects), represent when the result is published _(not implemented yet)_ | -+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ -| seed | int, number of seeder | -+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ -| leech | int, number of leecher | -+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ -| filesize | int, size of file in bytes | -+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ -| files | int, number of files | -+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ -| magnetlink | string, which is the [magnetlink](https://en.wikipedia.org/wiki/Magnet_URI_scheme) of the result | -+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ -| torrentfile | string, which is the torrentfile of the result | -+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ ++------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ +| result-parameter | information | ++------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ +| template | is set to ```torrent.html``` | ++------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ +| url | string, which is representing the url of the result | ++------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ +| title | string, which is representing the title of the result | ++------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ +| content | string, which is giving a general result-text | ++------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ +| publishedDate | `datetime.datetime `__, represent when the result is published *(not implemented yet)* | ++------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ +| seed | int, number of seeder | ++------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ +| leech | int, number of leecher | ++------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ +| filesize | int, size of file in bytes | ++------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ +| files | int, number of files | ++------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ +| magnetlink | string, which is the `magnetlink `__ of the result | ++------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ +| torrentfile | string, which is the torrentfile of the result | ++------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ map -- cgit v1.2.3 From 240802003ee98e93e084788665445d81a9623c7d Mon Sep 17 00:00:00 2001 From: Noemi Vanyi Date: Thu, 10 Dec 2015 21:38:39 +0100 Subject: [doc] fix small mistakes in engine overview --- docs/dev/engine_overview.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/engine_overview.rst b/docs/dev/engine_overview.rst index 05a54b641..186cc0f94 100644 --- a/docs/dev/engine_overview.rst +++ b/docs/dev/engine_overview.rst @@ -23,7 +23,7 @@ arguments can be set in the engine file or in the settings file (normally ``settings.yml``). The arguments in the settings file override the ones in the engine file. -It does not matter if an options is stored in the engine file or in the +It does not matter if an option is stored in the engine file or in the settings. However, the standard way is the following: @@ -63,7 +63,7 @@ often overwritten by the settings. If ``None`` is assigned to an option in the engine file, it has to be redefined in the settings, otherwise searx will not start with that engine. -The naming of that overrides is arbitrary. But the recommended +The naming of overrides is arbitrary. But the recommended overrides are the following: +-----------------------+----------+----------------------------------------------------------------+ -- cgit v1.2.3 From 3b068141928c17b6bb7b343d6e926fc3c443a022 Mon Sep 17 00:00:00 2001 From: Noemi Vanyi Date: Thu, 10 Dec 2015 21:40:22 +0100 Subject: [doc] rewrite contribution guide --- docs/dev/contribution_guide.rst | 107 +++++++++++++++++++++++++--------------- 1 file changed, 66 insertions(+), 41 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/contribution_guide.rst b/docs/dev/contribution_guide.rst index 5685549eb..73f4ea332 100644 --- a/docs/dev/contribution_guide.rst +++ b/docs/dev/contribution_guide.rst @@ -4,68 +4,93 @@ How to contribute Prime directives: Privacy, Hackability ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Searx has 2 prime directives, privacy-by-design and hackability. The -hackability comes in at least 3 levels: +Searx has two prime directives, privacy-by-design and hackability. The +hackability comes in three levels: -- support for search engines -- plugins for altering search behaviour -- hacking searx itself. +- support of search engines +- plugins to alter search behaviour +- hacking searx itself -Happy hacking. Observe the lack of "world domination" among the -directives, searx has no intentions for wide mass-adoption, rounded -corners, etc. The prime directive: "privacy" - deserves a seperate -chapter, as it's quite uncommon unfortunately, here it goes: +Note the lack of "world domination" among the directives. +Searx has no intention of wide mass-adoption, rounded +corners, etc. The prime directive "privacy" deserves a separate +chapter, as it's quite uncommon unfortunately. Privacy-by-design ^^^^^^^^^^^^^^^^^ -Searx is a privacy-respecting, hackable meta-search engine. It was born -out of the need for a privacy-respecing search facility that can be -expanded easily to maximise both its search and it's privacy protecting -capabilities. - -Consequences of Privacy-by-design are that some widely used features -work differently or not by default or at all. If some feature reduces -the privacy perserving aspects of searx, it should by default be -switched of, if implemented at all. There is enough search engines -already out there providing such features. = Since privacy-preservation -is a prime goal, if some feature does reduce the protection of searx and -is implemented, care should be taken to educate the user about the -consequences of choosing to enable this. Further features which -implement widely known features in a manner that protects privacy but -thus deviate from the users expectations should also be explained to the -user. Also if you think that something works weird with searx, maybe -it's because of the tool you use is designed in a way to interfere with -privacy respect, submiting a bugreport to the vendor of the tool that -misbehaves might be a good feedback for the vendor to reconsider his -disrespect towards his customers (e.g. GET vs POST requests in various -browsers). +Searx was born out of the need for a privacy-respecting search tool +which can be extended easily to maximize both its search and its +privacy protecting capabilities. + +A few widely used features work differently or turned off by default or not implemented +at all as a consequence of privacy-by-design. + +If a feature reduces the privacy preserving aspects of searx, it +should be switched off by default or should not implemented at all. +There are plenty of search engines already providing such features. +If a feature reduces the protection of searx, users must be +informed about the effect of choosing to enable it. Features +that protect privacy but differ from the expectations of the +user should also be explained. + +Also, if you think that something works weird with searx, +it's might be because of the tool you use is designed in a way to interfere with +the privacy respect. Submitting a bugreport to the vendor of the tool that +misbehaves might be a good feedback to reconsider the disrespect to +its customers (e.g. GET vs POST requests in various browsers). Remember the other prime directive of searx is to be hackable, so if the above privacy concerns do not fancy you, simply fork it. +Happy hacking. + Code ~~~~ -Code modifications are accepted in pull requests, don't forget to add -yourself to the AUTHORS file. +In order to submit a patch, please follow the steps below: + +- Follow coding conventions. + + - PEP8 standards apply, except the convention of line length + + - Maximum line length is 120 characters + +- Check if your code breaks existing tests. If so, update the tests or fix your code. + +- If your code can be unit-tested, add unit tests. -Python code follows all the pep8 standards except maximum line width -which is 120 char. +- Add yourself to the AUTHORS file. -Please be sure that the submitted code doesn't break existing tests and -follows coding conventions. +- Create a pull request. -If new functionality implemented, tests are highly appreciated. Translation ~~~~~~~~~~~ -Translation currently happens on -`transifex `__. Please do not -update translation files in the repo. +Translation currently takes place on +`transifex `__. + +**Please, do not update translation files in the repo.** Documentation ~~~~~~~~~~~~~ -The main place of the documentation is this wiki, updates are welcome. +The documentation is built using Sphinx. So in order to be able to generate the required +files, you have to install it on your system. (It can be installed easily using pip.) + +1. Checkout the gh-pages branch. + +2. Edit the rst file you wish to update. Or create a new rst file and place it under the appropriate folder. + +3. Build the documentation using Sphinx. + +4. Add the updated and created files of these extension: + + - .rst + + - .html + + - .txt + +6. Create a pull request. -- cgit v1.2.3 From 08d26609a5cffa1c70d6aa22cfa4455e1feb2002 Mon Sep 17 00:00:00 2001 From: Noemi Vanyi Date: Sun, 10 Jan 2016 20:10:53 +0100 Subject: update & fix install guide after install refactor --- docs/dev/install/installation.rst | 81 +++++++++++++++++++++++++-------------- 1 file changed, 52 insertions(+), 29 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/install/installation.rst b/docs/dev/install/installation.rst index 2ce6a6b54..e15ff8b25 100644 --- a/docs/dev/install/installation.rst +++ b/docs/dev/install/installation.rst @@ -1,25 +1,28 @@ Installation ============ -Step by step installation for Debian / Ubuntu with virtualenv. +Step by step installation for Debian/Ubuntu with virtualenv. Source: https://about.okhin.fr/posts/Searx/ with some additions How to: `Setup searx in a couple of hours with a free SSL certificate `__ +.. contents:: + :depth: 3 + Basic installation ------------------ For Ubuntu, be sure to have enable universe repository. -Install packages : +Install packages: .. code:: sh sudo apt-get install git build-essential libxslt-dev python-dev python-virtualenv python-pybabel zlib1g-dev libffi-dev libssl-dev -Install searx : +Install searx: .. code:: sh @@ -28,7 +31,7 @@ Install searx : sudo useradd searx -d /usr/local/searx sudo chown searx:searx -R /usr/local/searx -Install dependencies in a virtualenv : +Install dependencies in a virtualenv: .. code:: sh @@ -36,8 +39,7 @@ Install dependencies in a virtualenv : cd /usr/local/searx virtualenv searx-ve . ./searx-ve/bin/activate - pip install -r requirements.txt - python setup.py install + ./manage.sh update_packages Configuration ------------- @@ -51,7 +53,7 @@ Edit searx/settings.yml if necessary. Check ----- -Start searx : +Start searx: .. code:: sh @@ -59,7 +61,7 @@ Start searx : Go to http://localhost:8888 -If everything works fine, disable the debug option in settings.yml : +If everything works fine, disable the debug option in settings.yml: .. code:: sh @@ -73,14 +75,14 @@ twice). uwsgi ----- -Install packages : +Install packages: .. code:: sh sudo apt-get install uwsgi uwsgi-plugin-python Create the configuration file /etc/uwsgi/apps-available/searx.ini with -this content : +this content: :: @@ -111,7 +113,7 @@ this content : pythonpath = /usr/local/searx/ chdir = /usr/local/searx/searx/ -Activate the uwsgi application and restart : +Activate the uwsgi application and restart: .. code:: sh @@ -126,7 +128,7 @@ with nginx ^^^^^^^^^^ If nginx is not installed (uwsgi will not work with the package -nginx-light) : +nginx-light): .. code:: sh @@ -136,7 +138,7 @@ Hosted at / """"""""""" Create the configuration file /etc/nginx/sites-available/searx with this -content : +content: .. code:: nginx @@ -151,7 +153,7 @@ content : } } -Restart service : +Restart service: .. code:: sh @@ -159,10 +161,10 @@ Restart service : sudo service uwsgi restart from subdirectory URL (/searx) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +"""""""""""""""""""""""""""""" Add this configuration in the server config file -/etc/nginx/sites-available/default : +/etc/nginx/sites-available/default: .. code:: nginx @@ -183,7 +185,7 @@ Enable base\_url in searx/settings.yml base_url : http://your.domain.tld/searx/ -Restart service : +Restart service: .. code:: sh @@ -195,7 +197,7 @@ disable logs for better privacy you can disable nginx logs about searx. -how to proceed : below ``uwsgi_pass`` in +how to proceed: below ``uwsgi_pass`` in /etc/nginx/sites-available/default add :: @@ -203,23 +205,23 @@ how to proceed : below ``uwsgi_pass`` in access_log /dev/null; error_log /dev/null; -Restart service : +Restart service: .. code:: sh sudo service nginx restart with apache ------------ +^^^^^^^^^^^ -Add wsgi mod : +Add wsgi mod: .. code:: sh sudo apt-get install libapache2-mod-uwsgi sudo a2enmod uwsgi -Add this configuration in the file /etc/apache2/apache2.conf : +Add this configuration in the file /etc/apache2/apache2.conf: .. code:: apache @@ -233,29 +235,29 @@ Note that if your instance of searx is not at the root, you should change ```` by the location of your instance, like ````. -Restart Apache : +Restart Apache: .. code:: sh sudo /etc/init.d/apache2 restart disable logs ------------- +"""""""""""" For better privacy you can disable Apache logs. -WARNING : not tested +WARNING: not tested -WARNING : you can only disable logs for the whole (virtual) server not +WARNING: you can only disable logs for the whole (virtual) server not for a specific path. -Go back to /etc/apache2/apache2.conf and above ```` add : +Go back to /etc/apache2/apache2.conf and above ```` add: .. code:: apache CustomLog /dev/null combined -Restart Apache : +Restart Apache: .. code:: sh @@ -272,6 +274,27 @@ How to update git stash git pull origin master git stash apply - pip install --upgrade -r requirements.txt + ./manage.sh update_packages sudo service uwsgi restart +Docker +------ + +Make sure you have installed Docker. For instance, you can deploy searx like this: + +.. code:: sh + + docker pull wonderfall/searx + docker run -d --name searx -p $PORT:8888 wonderfall/searx + +Go to http://localhost:$PORT. + +See https://hub.docker.com/r/wonderfall/searx/ for more informations. + +It's also possible to build searx from the embedded Dockerfile. + +.. code:: sh + + git clone https://github.com/asciimoo/searx.git + cd searx + docker build -t whatever/searx . -- cgit v1.2.3 From fb62d01194bc6049a0dc1e5ffbc3ca2f9536f26c Mon Sep 17 00:00:00 2001 From: Thomas Pointhuber Date: Thu, 21 Jan 2016 16:25:40 +0100 Subject: [enh] improve translation documentation --- docs/dev/translation.rst | 36 +++++++++++++++++++++--------------- 1 file changed, 21 insertions(+), 15 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/translation.rst b/docs/dev/translation.rst index 0fc02a7de..c070c36ea 100644 --- a/docs/dev/translation.rst +++ b/docs/dev/translation.rst @@ -6,23 +6,29 @@ run these commands in the root directory of searx Add new language ~~~~~~~~~~~~~~~~ -``pybabel init -i messages.pot -d searx/translations -l it`` +.. code:: shell + + pybabel init -i messages.pot -d searx/translations -l it Update .po files ~~~~~~~~~~~~~~~~ -``./utils/update-translations.sh`` +.. code:: shell + + ./utils/update-translations.sh You may have errors here. In that case, edit the ``update-translations.sh`` script to change ``pybabel`` to -``pybabel-python2`` +``pybabel-python2 or pybabel2`` After this step, you can modify the .po files. Compile translations ~~~~~~~~~~~~~~~~~~~~ -``pybabel compile -d searx/translations`` +.. code:: shell + + pybabel compile -d searx/translations Transifex stuff ~~~~~~~~~~~~~~~ @@ -32,12 +38,14 @@ Init Project .. code:: shell + tx init # Transifex instance: https://www.transifex.com/asciimoo/searx/ + tx set --auto-local -r searx.messagespo 'searx/translations//LC_MESSAGES/messages.po' \ --source-lang en --type PO --source-file messages.pot --execute -http://docs.transifex.com/developer/client/set +http://docs.transifex.com/client/init/ -*TODO: mapping between transifex and searx* +http://docs.transifex.com/client/set/ Get translations ^^^^^^^^^^^^^^^^ @@ -46,29 +54,27 @@ Get translations tx pull -a -http://docs.transifex.com/developer/client/pull +http://docs.transifex.com/client/pull Upload source File ^^^^^^^^^^^^^^^^^^ -:: +.. code:: shell tx push -s Upload all Translation ^^^^^^^^^^^^^^^^^^^^^^ -:: +.. code:: shell tx push -s -t -upload specifc Translation (only for admins) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +upload specifc Translation +~~~~~~~~~~~~~~~~~~~~~~~~~~ -:: +.. code:: shell tx push -t -l tr -http://docs.transifex.com/developer/client/push - -*TODO: upload empty files? (new translations)* +http://docs.transifex.com/client/push -- cgit v1.2.3 From cf9bd14a7d8a34c08af5f7a6370525c559d2ed96 Mon Sep 17 00:00:00 2001 From: Thomas Pointhuber Date: Thu, 21 Jan 2016 16:30:42 +0100 Subject: [fix] little typo --- docs/dev/translation.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'docs/dev') diff --git a/docs/dev/translation.rst b/docs/dev/translation.rst index c070c36ea..bc32eda90 100644 --- a/docs/dev/translation.rst +++ b/docs/dev/translation.rst @@ -19,7 +19,7 @@ Update .po files You may have errors here. In that case, edit the ``update-translations.sh`` script to change ``pybabel`` to -``pybabel-python2 or pybabel2`` +``pybabel-python2`` or ``pybabel2`` After this step, you can modify the .po files. -- cgit v1.2.3 From 016ab5c1e87fa86b5e618616e1dd9d49d880c506 Mon Sep 17 00:00:00 2001 From: Noemi Vanyi Date: Mon, 1 Feb 2016 21:28:13 +0100 Subject: add development quickstart --- docs/dev/quickstart.rst | 90 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 docs/dev/quickstart.rst (limited to 'docs/dev') diff --git a/docs/dev/quickstart.rst b/docs/dev/quickstart.rst new file mode 100644 index 000000000..edca8af5e --- /dev/null +++ b/docs/dev/quickstart.rst @@ -0,0 +1,90 @@ +Development Quickstart +---------------------- + +This quickstart guide gets your environment set up with searx. Furthermore, it gives a +short introduction to the new manage.sh script. + +How to setup your development environment +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +First, clone the source code of searx to the desired folder. In this case the source +is cloned to ~/myprojects/searx. Then create and activate the searx-ve +virtualenv and install the required packages using manage.sh. + +.. code:: sh + + cd ~/myprojects + git clone https://github.com/asciimoo/searx.git + cd searx + virtualenv searx-ve + . ./searx-ve/bin/activate + ./manage.sh update_dev_packages + + +How to run tests +~~~~~~~~~~~~~~~~ + +Tests can be run using the manage.sh script. + +Following tests and checks are available: + +- Unit tests + +- Selenium tests + +- PEP8 validation + +- Unit test coverage check + +For example unit tests are run with the command below: + +.. code:: sh + + ./manage.sh unit_tests + +For further test options, please consult the help of the manage.sh script. + + +How to compile styles and javascript +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +How to build styles +^^^^^^^^^^^^^^^^^^^ + +Less is required to build the styles of searx. Less can be installed using either NodeJS or Apt. + +.. code:: sh + + sudo apt-get install nodejs + sudo npm install -g less + + +OR + +.. code:: sh + + sudo apt-get install node-less + +After satisfying the requirements styles can be build using manage.sh + +.. code:: sh + + ./manage.sh styles + + +How to build the source of the oscar theme +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Grunt must be installed in order to build the javascript sources. It depends on NodeJS, so first +Node has to be installed. + +.. code:: sh + + sudo apt-get install nodejs + sudo npm install -g grunt-cli + +After installing grunt, the files can be built using the following command: + +.. code:: sh + + ./manage.sh build_grunt -- cgit v1.2.3 From 9c56154cfe121c6b89f48dd6b9af4e5e029d8a6e Mon Sep 17 00:00:00 2001 From: Noemi Vanyi Date: Thu, 21 Apr 2016 13:15:04 +0200 Subject: add dev tips to quickstart --- docs/dev/contribution_guide.rst | 2 ++ docs/dev/quickstart.rst | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+) (limited to 'docs/dev') diff --git a/docs/dev/contribution_guide.rst b/docs/dev/contribution_guide.rst index 73f4ea332..65e587294 100644 --- a/docs/dev/contribution_guide.rst +++ b/docs/dev/contribution_guide.rst @@ -64,6 +64,8 @@ In order to submit a patch, please follow the steps below: - Create a pull request. +For more help on getting started with searx development, see :ref:`devquickstart`. + Translation ~~~~~~~~~~~ diff --git a/docs/dev/quickstart.rst b/docs/dev/quickstart.rst index edca8af5e..92ef62385 100644 --- a/docs/dev/quickstart.rst +++ b/docs/dev/quickstart.rst @@ -1,3 +1,5 @@ +.. _devquickstart: + Development Quickstart ---------------------- @@ -88,3 +90,19 @@ After installing grunt, the files can be built using the following command: .. code:: sh ./manage.sh build_grunt + + + +Tips for debugging/development +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +1. Turn on debug logging + Whether you are working on a new engine or trying to eliminate a bug, it is always a good idea + to turn on debug logging. When debug logging is enabled a stack trace appears, + instead of the cryptic ``Internal Server Error`` message. It can be turned on by setting + ``debug: False`` to ``debug: True`` in settings.yml. + +2. Run ``./manage.sh tests`` before creating a PR. + Failing build on Travis is common because of PEP8 checks. So a new commit must be created + containing these format fixes. This phase can be skipped if ``./manage.sh tests`` is run + locally before creating a PR. -- cgit v1.2.3 From 2430baa1fd6e4ecba4bc72a8e71c49a38b424216 Mon Sep 17 00:00:00 2001 From: Noemi Vanyi Date: Thu, 21 Apr 2016 13:16:18 +0200 Subject: update install docs on reverse proxy --- docs/dev/install/installation.rst | 3 +++ 1 file changed, 3 insertions(+) (limited to 'docs/dev') diff --git a/docs/dev/install/installation.rst b/docs/dev/install/installation.rst index e15ff8b25..d69f30e8f 100644 --- a/docs/dev/install/installation.rst +++ b/docs/dev/install/installation.rst @@ -171,6 +171,8 @@ Add this configuration in the server config file location = /searx { rewrite ^ /searx/; } location /searx { try_files $uri @searx; + proxy_pass http://localhost:9999/; + proxy_set_header X-Script-Name /searx; } location @searx { uwsgi_param SCRIPT_NAME /searx; @@ -179,6 +181,7 @@ Add this configuration in the server config file uwsgi_pass unix:/run/uwsgi/app/searx/socket; } + Enable base\_url in searx/settings.yml :: -- cgit v1.2.3 From b40d7a2b36014f8db825c06350ed84db7eed67d5 Mon Sep 17 00:00:00 2001 From: Noemi Vanyi Date: Thu, 21 Apr 2016 14:30:17 +0200 Subject: add reverse proxy configuration --- docs/dev/install/installation.rst | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/install/installation.rst b/docs/dev/install/installation.rst index d69f30e8f..cbbd25f57 100644 --- a/docs/dev/install/installation.rst +++ b/docs/dev/install/installation.rst @@ -171,8 +171,6 @@ Add this configuration in the server config file location = /searx { rewrite ^ /searx/; } location /searx { try_files $uri @searx; - proxy_pass http://localhost:9999/; - proxy_set_header X-Script-Name /searx; } location @searx { uwsgi_param SCRIPT_NAME /searx; @@ -182,6 +180,23 @@ Add this configuration in the server config file } +OR + +using reverse proxy +(Please, note that reverse proxy advised to be used in case of single-user or low-traffic instances.) + +.. code:: nginx + + location /searx { + proxy_pass http://127.0.0.1:8888; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Scheme $scheme; + proxy_set_header X-Script-Name /searx; + proxy_buffering off; + } + + Enable base\_url in searx/settings.yml :: -- cgit v1.2.3 From 2ad8715b3282557d39da21721487ff85683b1219 Mon Sep 17 00:00:00 2001 From: Noemi Vanyi Date: Sat, 9 Jul 2016 22:31:21 +0200 Subject: better API docs && more features in list --- docs/dev/plugins.rst | 34 +++++++++++++++++----------------- docs/dev/search_api.rst | 19 +++++++++++++------ 2 files changed, 30 insertions(+), 23 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/plugins.rst b/docs/dev/plugins.rst index c51fdf0eb..0f52af0d8 100644 --- a/docs/dev/plugins.rst +++ b/docs/dev/plugins.rst @@ -1,11 +1,11 @@ Plugins ------- -Plugins can extend/replace functionality of various components inside +Plugins can extend or replace functionality of various components of searx. -example\_plugin.py -~~~~~~~~~~~~~~~~~~ +Example plugin +~~~~~~~~~~~~~~ .. code:: python @@ -24,22 +24,22 @@ example\_plugin.py ctx['search'].suggestions.add('example') return True -Currently implemented plugin entry points (a.k.a hooks) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Plugin entry points +~~~~~~~~~~~~~~~~~~~ -- Pre search hook (``pre_search``) -- Post search hook (``post_search``) -- Result hook (``on_result``) (is called if a new result is added (see - https\_rewrite plugin)) +Entry points (hooks) define when a plugin runs. Right now only three hooks are implemented. So feel free to implement a hook if it fits the behaviour of your plugin. -Feel free to add more hooks to the code if it is required by a plugin. +Pre search hook +``````````````` -TODO -~~~~ +Runs BEFORE the search request. Function to implement: ``pre_search`` -- Better documentation -- More hooks -- search hook (is called while searx is requesting results (for - example: things like math-solver), the different hooks are running - parallel) +Post search hook +```````````````` +Runs AFTER the search request. Function to implement: ``post_search`` + +Result hook +``````````` + +Runs when a new result is added to the result list. Function to implement: ``on_result`` diff --git a/docs/dev/search_api.rst b/docs/dev/search_api.rst index cd86fd71c..59d3c825f 100644 --- a/docs/dev/search_api.rst +++ b/docs/dev/search_api.rst @@ -1,13 +1,20 @@ Search API ========== -Search API endpoints: ``/``, ``/search`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +The search supports both ``GET`` and ``POST``. However, using ``GET`` the parameters of the request remain hidden. So it is advised to use ``GET`` for querying. -Endpoints have equivalent functionality. +Furthermore, two enpoints ``/`` and ``/search`` are available for querying. + +.. code:: sh + + GET / + +.. code:: sh + + GET /search Parameters -^^^^^^^^^^ +`````````` +------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ | Name | Description | | @@ -18,7 +25,7 @@ Parameters +------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ | ``engines`` | Comma separated list, specifies the active search engines | optional | +------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ +| ``lang`` | Code of the language | optional (default: ``all``) | ++------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ | ``pageno`` | Search page number | optional (default: ``1``) | +------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ - -Both ``GET`` and ``POST`` methods are supported. -- cgit v1.2.3 From 9e135f9fab3894cc86264cb7552252ef0cf10581 Mon Sep 17 00:00:00 2001 From: Noemi Vanyi Date: Fri, 15 Jul 2016 18:41:05 +0200 Subject: simplify index page --- docs/dev/install/installation.rst | 2 ++ 1 file changed, 2 insertions(+) (limited to 'docs/dev') diff --git a/docs/dev/install/installation.rst b/docs/dev/install/installation.rst index cbbd25f57..b641a628e 100644 --- a/docs/dev/install/installation.rst +++ b/docs/dev/install/installation.rst @@ -1,3 +1,5 @@ +.. _installation: + Installation ============ -- cgit v1.2.3 From 6d99db7af33f82c00dbb9d11e17a9c6a907591cd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?No=C3=A9mi=20V=C3=A1nyi?= Date: Tue, 6 Sep 2016 18:13:20 +0200 Subject: add new search parameter time_range --- docs/dev/search_api.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) (limited to 'docs/dev') diff --git a/docs/dev/search_api.rst b/docs/dev/search_api.rst index 59d3c825f..c1bc6a139 100644 --- a/docs/dev/search_api.rst +++ b/docs/dev/search_api.rst @@ -1,7 +1,7 @@ Search API ========== -The search supports both ``GET`` and ``POST``. However, using ``GET`` the parameters of the request remain hidden. So it is advised to use ``GET`` for querying. +The search supports both ``GET`` and ``POST``. Furthermore, two enpoints ``/`` and ``/search`` are available for querying. @@ -29,3 +29,5 @@ Parameters +------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ | ``pageno`` | Search page number | optional (default: ``1``) | +------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ +| ``time_range`` | Time range of search | optional | ++------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ -- cgit v1.2.3 From daf736f1dda06507b63520d37e1160de44ba6636 Mon Sep 17 00:00:00 2001 From: Noemi Vanyi Date: Sat, 10 Sep 2016 15:12:11 +0200 Subject: add possible values of time_range param --- docs/dev/search_api.rst | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/search_api.rst b/docs/dev/search_api.rst index c1bc6a139..03bb6f3fc 100644 --- a/docs/dev/search_api.rst +++ b/docs/dev/search_api.rst @@ -16,18 +16,18 @@ Furthermore, two enpoints ``/`` and ``/search`` are available for querying. Parameters `````````` -+------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ -| Name | Description | | -+==================+====================================================================================================+=============================+ -| ``q`` | The search query, see :doc:`/user/search_syntax` | required | -+------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ -| ``categories`` | Comma separated list, specifies the active search categories | optional | -+------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ -| ``engines`` | Comma separated list, specifies the active search engines | optional | -+------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ -| ``lang`` | Code of the language | optional (default: ``all``) | -+------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ -| ``pageno`` | Search page number | optional (default: ``1``) | -+------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ -| ``time_range`` | Time range of search | optional | -+------------------+----------------------------------------------------------------------------------------------------+-----------------------------+ ++------------------+----------------------------------------------------------------------------------------------------+--------------------------------------------------+ +| Name | Description | | ++==================+====================================================================================================+==================================================+ +| ``q`` | The search query, see :doc:`/user/search_syntax` | required | ++------------------+----------------------------------------------------------------------------------------------------+--------------------------------------------------+ +| ``categories`` | Comma separated list, specifies the active search categories | optional | ++------------------+----------------------------------------------------------------------------------------------------+--------------------------------------------------+ +| ``engines`` | Comma separated list, specifies the active search engines | optional | ++------------------+----------------------------------------------------------------------------------------------------+--------------------------------------------------+ +| ``lang`` | Code of the language | optional (default: ``all``) | ++------------------+----------------------------------------------------------------------------------------------------+--------------------------------------------------+ +| ``pageno`` | Search page number | optional (default: ``1``) | ++------------------+----------------------------------------------------------------------------------------------------+--------------------------------------------------+ +| ``time_range`` | Time range of search | optional (possible: ``day``, ``month``, ``year``)| ++------------------+----------------------------------------------------------------------------------------------------+--------------------------------------------------+ -- cgit v1.2.3 From c2cf46092e7e805587fde191ea9c620ee2bcb158 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?No=C3=A9mi=20V=C3=A1nyi?= Date: Fri, 4 Nov 2016 18:38:07 +0100 Subject: docs: fix typos based on #653 --- docs/dev/engine_overview.rst | 226 ++++++++++++++++++++++--------------------- docs/dev/plugins.rst | 2 +- 2 files changed, 115 insertions(+), 113 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/engine_overview.rst b/docs/dev/engine_overview.rst index 186cc0f94..da7ee2710 100644 --- a/docs/dev/engine_overview.rst +++ b/docs/dev/engine_overview.rst @@ -30,15 +30,17 @@ settings. However, the standard way is the following: engine file ~~~~~~~~~~~ -+---------------------+-----------+-----------------------------------------+ -| argument | type | information | -+=====================+===========+=========================================+ -| categories | list | pages, in which the engine is working | -+---------------------+-----------+-----------------------------------------+ -| paging | boolean | support multible pages | -+---------------------+-----------+-----------------------------------------+ -| language\_support | boolean | support language choosing | -+---------------------+-----------+-----------------------------------------+ ++----------------------+-----------+-----------------------------------------+ +| argument | type | information | ++======================+===========+=========================================+ +| categories | list | pages, in which the engine is working | ++----------------------+-----------+-----------------------------------------+ +| paging | boolean | support multible pages | ++----------------------+-----------+-----------------------------------------+ +| language\_support | boolean | support language choosing | ++----------------------+-----------+-----------------------------------------+ +| time\_range\_support | boolean | support search time range | ++----------------------+-----------+-----------------------------------------+ settings.yml ~~~~~~~~~~~~ @@ -186,126 +188,126 @@ To set another media-type as default, the parameter default ~~~~~~~ -+--------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| result-parameter | information | -+====================+====================================================================================================================================+ -| url | string, which is representing the url of the result | -+--------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| title | string, which is representing the title of the result | -+--------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| content | string, which is giving a general result-text | -+--------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| publishedDate | `datetime.datetime `__, represent when the result is published | -+--------------------+------------------------------------------------------------------------------------------------------------------------------------+ ++--------------------+---------------------------------------------------------------------------------------------------------------+ +| result-parameter | information | ++====================+===============================================================================================================+ +| url | string, url of the result | ++--------------------+---------------------------------------------------------------------------------------------------------------+ +| title | string, title of the result | ++--------------------+---------------------------------------------------------------------------------------------------------------+ +| content | string, general result-text | ++--------------------+---------------------------------------------------------------------------------------------------------------+ +| publishedDate | `datetime.datetime `__, time of publish | ++--------------------+---------------------------------------------------------------------------------------------------------------+ images ~~~~~~ to use this template, the parameter -+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+ -| result-parameter | information | -+====================+===========================================================================================================================================================+ -| template | is set to ``images.html`` | -+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+ -| url | string, which is representing the url to the result site | -+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+ -| title | string, which is representing the title of the result *(partly implemented)* | -+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+ -| content | *(partly implemented)* | -+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+ -| publishedDate | `datetime.datetime `__, represent when the result is published *(partly implemented)* | -+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+ -| img\_src | string, which is representing the url to the result image | -+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+ -| thumbnail\_src | string, which is representing the url to a small-preview image | -+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+ ++--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| result-parameter | information | ++====================+=======================================================================================================================================+ +| template | is set to ``images.html`` | ++--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| url | string, url to the result site | ++--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| title | string, title of the result *(partly implemented)* | ++--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| content | *(partly implemented)* | ++--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| publishedDate | `datetime.datetime `__, time of publish *(partly implemented)* | ++--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| img\_src | string, url to the result image | ++--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| thumbnail\_src | string, url to a small-preview image | ++--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ videos ~~~~~~ -+--------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| result-parameter | information | -+====================+====================================================================================================================================+ -| template | is set to ``videos.html`` | -+--------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| url | string, which is representing the url of the result | -+--------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| title | string, which is representing the title of the result | -+--------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| content | *(not implemented yet)* | -+--------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| publishedDate | `datetime.datetime `__, represent when the result is published | -+--------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| thumbnail | string, which is representing the url to a small-preview image | -+--------------------+------------------------------------------------------------------------------------------------------------------------------------+ ++--------------------+--------------------------------------------------------------------------------------------------------------+ +| result-parameter | information | ++====================+==============================================================================================================+ +| template | is set to ``videos.html`` | ++--------------------+--------------------------------------------------------------------------------------------------------------+ +| url | string, url of the result | ++--------------------+--------------------------------------------------------------------------------------------------------------+ +| title | string, title of the result | ++--------------------+--------------------------------------------------------------------------------------------------------------+ +| content | *(not implemented yet)* | ++--------------------+--------------------------------------------------------------------------------------------------------------+ +| publishedDate | `datetime.datetime `__, time of publish | ++--------------------+--------------------------------------------------------------------------------------------------------------+ +| thumbnail | string, url to a small-preview image | ++--------------------+--------------------------------------------------------------------------------------------------------------+ torrent ~~~~~~~ -+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ -| result-parameter | information | -+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ -| template | is set to ```torrent.html``` | -+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ -| url | string, which is representing the url of the result | -+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ -| title | string, which is representing the title of the result | -+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ -| content | string, which is giving a general result-text | -+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ -| publishedDate | `datetime.datetime `__, represent when the result is published *(not implemented yet)* | -+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ -| seed | int, number of seeder | -+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ -| leech | int, number of leecher | -+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ -| filesize | int, size of file in bytes | -+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ -| files | int, number of files | -+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ -| magnetlink | string, which is the `magnetlink `__ of the result | -+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ -| torrentfile | string, which is the torrentfile of the result | -+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ ++------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| result-parameter | information | ++------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| template | is set to ``torrent.html`` | ++------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| url | string, url of the result | ++------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| title | string, title of the result | ++------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| content | string, general result-text | ++------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| publishedDate | `datetime.datetime `__, time of publish *(not implemented yet)* | ++------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| seed | int, number of seeder | ++------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| leech | int, number of leecher | ++------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| filesize | int, size of file in bytes | ++------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| files | int, number of files | ++------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| magnetlink | string, `magnetlink `__ of the result | ++------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| torrentfile | string, torrentfile of the result | ++------------------+---------------------------------------------------------------------------------------------------------------------------------------+ map ~~~ -+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| result-parameter | information | -+=========================+====================================================================================================================================+ -| url | string, which is representing the url of the result | -+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| title | string, which is representing the title of the result | -+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| content | string, which is giving a general result-text | -+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| publishedDate | `datetime.datetime `__, represent when the result is published | -+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| latitude | latitude of result (in decimal format) | -+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| longitude | longitude of result (in decimal format) | -+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| boundingbox | boundingbox of result (array of 4. values ``[lat-min, lat-max, lon-min, lon-max]``) | -+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| geojson | geojson of result (http://geojson.org) | -+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| osm.type | type of osm-object (if OSM-Result) | -+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| osm.id | id of osm-object (if OSM-Result) | -+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| address.name | name of object | -+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| address.road | street adress of object | -+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| address.house\_number | house number of object | -+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| address.locality | city, place of object | -+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| address.postcode | postcode of object | -+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ -| address.country | country of object | -+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+ ++-------------------------+--------------------------------------------------------------------------------------------------------------+ +| result-parameter | information | ++=========================+==============================================================================================================+ +| url | string, url of the result | ++-------------------------+--------------------------------------------------------------------------------------------------------------+ +| title | string, title of the result | ++-------------------------+--------------------------------------------------------------------------------------------------------------+ +| content | string, general result-text | ++-------------------------+--------------------------------------------------------------------------------------------------------------+ +| publishedDate | `datetime.datetime `__, time of publish | ++-------------------------+--------------------------------------------------------------------------------------------------------------+ +| latitude | latitude of result (in decimal format) | ++-------------------------+--------------------------------------------------------------------------------------------------------------+ +| longitude | longitude of result (in decimal format) | ++-------------------------+--------------------------------------------------------------------------------------------------------------+ +| boundingbox | boundingbox of result (array of 4. values ``[lat-min, lat-max, lon-min, lon-max]``) | ++-------------------------+--------------------------------------------------------------------------------------------------------------+ +| geojson | geojson of result (http://geojson.org) | ++-------------------------+--------------------------------------------------------------------------------------------------------------+ +| osm.type | type of osm-object (if OSM-Result) | ++-------------------------+--------------------------------------------------------------------------------------------------------------+ +| osm.id | id of osm-object (if OSM-Result) | ++-------------------------+--------------------------------------------------------------------------------------------------------------+ +| address.name | name of object | ++-------------------------+--------------------------------------------------------------------------------------------------------------+ +| address.road | street name of object | ++-------------------------+--------------------------------------------------------------------------------------------------------------+ +| address.house\_number | house number of object | ++-------------------------+--------------------------------------------------------------------------------------------------------------+ +| address.locality | city, place of object | ++-------------------------+--------------------------------------------------------------------------------------------------------------+ +| address.postcode | postcode of object | ++-------------------------+--------------------------------------------------------------------------------------------------------------+ +| address.country | country of object | ++-------------------------+--------------------------------------------------------------------------------------------------------------+ diff --git a/docs/dev/plugins.rst b/docs/dev/plugins.rst index 0f52af0d8..1d77f8611 100644 --- a/docs/dev/plugins.rst +++ b/docs/dev/plugins.rst @@ -11,7 +11,7 @@ Example plugin name = 'Example plugin' description = 'This plugin extends the suggestions with the word "example"' - default_on = False # disable by default + default_on = False # disabled by default js_dependencies = tuple() # optional, list of static js files css_dependencies = tuple() # optional, list of static css files -- cgit v1.2.3 From 50f25193c17707157c905f455b1d3a27c3edc001 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?No=C3=A9mi=20V=C3=A1nyi?= Date: Sat, 5 Nov 2016 16:22:29 +0100 Subject: engine overview fix torrent table --- docs/dev/engine_overview.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'docs/dev') diff --git a/docs/dev/engine_overview.rst b/docs/dev/engine_overview.rst index da7ee2710..d57ca5b38 100644 --- a/docs/dev/engine_overview.rst +++ b/docs/dev/engine_overview.rst @@ -247,7 +247,7 @@ torrent +------------------+---------------------------------------------------------------------------------------------------------------------------------------+ | result-parameter | information | -+------------------+---------------------------------------------------------------------------------------------------------------------------------------+ ++==================+=======================================================================================================================================+ | template | is set to ``torrent.html`` | +------------------+---------------------------------------------------------------------------------------------------------------------------------------+ | url | string, url of the result | -- cgit v1.2.3 From 6ecd8e3bd72df43d903800ee05e33185efe22ee6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?No=C3=A9mi=20V=C3=A1nyi?= Date: Sun, 20 Nov 2016 21:12:52 +0100 Subject: installation: nginx mod to sites-enabled --- docs/dev/install/installation.rst | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/install/installation.rst b/docs/dev/install/installation.rst index b641a628e..b02718fe3 100644 --- a/docs/dev/install/installation.rst +++ b/docs/dev/install/installation.rst @@ -3,20 +3,13 @@ Installation ============ -Step by step installation for Debian/Ubuntu with virtualenv. - -Source: https://about.okhin.fr/posts/Searx/ with some additions - -How to: `Setup searx in a couple of hours with a free SSL -certificate `__ - .. contents:: :depth: 3 Basic installation ------------------ -For Ubuntu, be sure to have enable universe repository. +Step by step installation for Debian/Ubuntu with virtualenv. For Ubuntu, be sure to have enable universe repository. Install packages: @@ -166,7 +159,7 @@ from subdirectory URL (/searx) """""""""""""""""""""""""""""" Add this configuration in the server config file -/etc/nginx/sites-available/default: +/etc/nginx/sites-enabled/default: .. code:: nginx @@ -318,3 +311,11 @@ It's also possible to build searx from the embedded Dockerfile. git clone https://github.com/asciimoo/searx.git cd searx docker build -t whatever/searx . + +References +========== + + * https://about.okhin.fr/posts/Searx/ with some additions + + * How to: `Setup searx in a couple of hours with a free SSL certificate `__ + -- cgit v1.2.3 From 1bd51c6de2f31ad15e5689ee7ff80f2283baa817 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?No=C3=A9mi=20V=C3=A1nyi?= Date: Sat, 18 Mar 2017 23:57:19 +0100 Subject: rewrite Translation page --- docs/dev/translation.rst | 69 ++++++++++++++++++------------------------------ 1 file changed, 25 insertions(+), 44 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/translation.rst b/docs/dev/translation.rst index bc32eda90..ab4dd98cb 100644 --- a/docs/dev/translation.rst +++ b/docs/dev/translation.rst @@ -1,80 +1,61 @@ Translation =========== -run these commands in the root directory of searx +Requirements +------------ -Add new language -~~~~~~~~~~~~~~~~ + * Transifex account -.. code:: shell + * Installed CLI tool of Transifex - pybabel init -i messages.pot -d searx/translations -l it +Init Transifex project +---------------------- -Update .po files -~~~~~~~~~~~~~~~~ +After installing ``transifex`` using pip, run the following command to initialize the project. .. code:: shell - ./utils/update-translations.sh + tx init # Transifex instance: https://www.transifex.com/asciimoo/searx/ -You may have errors here. In that case, edit the -``update-translations.sh`` script to change ``pybabel`` to -``pybabel-python2`` or ``pybabel2`` -After this step, you can modify the .po files. +After ``$HOME/.transifexrc`` is created, get a Transifex API key and insert it into the configuration file. -Compile translations -~~~~~~~~~~~~~~~~~~~~ +Create a configuration file for ``tx`` named ``$HOME/.tx/config``. .. code:: shell - pybabel compile -d searx/translations + [main] + host = https://www.transifex.com + [searx.messagespo] + file_filter = searx/translations//LC_MESSAGES/messages.po + source_file = messages.pot + source_lang = en + type = PO -Transifex stuff -~~~~~~~~~~~~~~~ -Init Project -^^^^^^^^^^^^ +Then run ``tx set``: .. code:: shell - tx init # Transifex instance: https://www.transifex.com/asciimoo/searx/ - tx set --auto-local -r searx.messagespo 'searx/translations//LC_MESSAGES/messages.po' \ --source-lang en --type PO --source-file messages.pot --execute -http://docs.transifex.com/client/init/ -http://docs.transifex.com/client/set/ +Update translations +------------------- -Get translations -^^^^^^^^^^^^^^^^ +To retrieve the latest translations, pull it from Transifex. .. code:: shell tx pull -a -http://docs.transifex.com/client/pull - -Upload source File -^^^^^^^^^^^^^^^^^^ - -.. code:: shell - - tx push -s - -Upload all Translation -^^^^^^^^^^^^^^^^^^^^^^ +Then check the new languages. If strings translated are not enough, delete those folders, because +those should not be compiled. Call the command below to compile the ``.po`` files. .. code:: shell - tx push -s -t - -upload specifc Translation -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. code:: shell + pybabel compile -d searx/translations - tx push -t -l tr -http://docs.transifex.com/client/push +After the compilation is finished commit the ``.po`` and ``.mo`` files and create a PR. -- cgit v1.2.3 From b92631bf8c36a8b7fa1fada219428af8e2abb566 Mon Sep 17 00:00:00 2001 From: Adam Tauber Date: Mon, 12 Jun 2017 12:34:30 +0200 Subject: various mods: uwsgi setup and version update ++ py3 img src fix ++ new sphinx version - closes #939 --- docs/dev/install/installation.rst | 2 ++ 1 file changed, 2 insertions(+) (limited to 'docs/dev') diff --git a/docs/dev/install/installation.rst b/docs/dev/install/installation.rst index b02718fe3..e1ae3f2a1 100644 --- a/docs/dev/install/installation.rst +++ b/docs/dev/install/installation.rst @@ -99,6 +99,8 @@ this content: single-interpreter = true master = true plugin = python + lazy-apps = true + enable-threads = true # Module to import module = searx.webapp -- cgit v1.2.3 From 74d8ed284dfb5363df4280350a8e29125f67a10a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?No=C3=A9mi=20V=C3=A1nyi?= Date: Sun, 9 Jul 2017 22:02:29 +0200 Subject: add format param to search api --- docs/dev/search_api.rst | 2 ++ 1 file changed, 2 insertions(+) (limited to 'docs/dev') diff --git a/docs/dev/search_api.rst b/docs/dev/search_api.rst index 03bb6f3fc..331bc4eb9 100644 --- a/docs/dev/search_api.rst +++ b/docs/dev/search_api.rst @@ -31,3 +31,5 @@ Parameters +------------------+----------------------------------------------------------------------------------------------------+--------------------------------------------------+ | ``time_range`` | Time range of search | optional (possible: ``day``, ``month``, ``year``)| +------------------+----------------------------------------------------------------------------------------------------+--------------------------------------------------+ +| ``format`` | Output format of results | optional (possible: ``json``, ``csv``, ``rss``) | ++------------------+----------------------------------------------------------------------------------------------------+--------------------------------------------------+ -- cgit v1.2.3 From 07ef4a3e4afc390ac08fe2ac4b6b2e1195604cd1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?No=C3=A9mi=20V=C3=A1nyi?= Date: Mon, 23 Oct 2017 16:29:43 +0200 Subject: update search api with params and description on syntax --- docs/dev/search_api.rst | 213 +++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 191 insertions(+), 22 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/search_api.rst b/docs/dev/search_api.rst index 331bc4eb9..ea3b0501a 100644 --- a/docs/dev/search_api.rst +++ b/docs/dev/search_api.rst @@ -5,31 +5,200 @@ The search supports both ``GET`` and ``POST``. Furthermore, two enpoints ``/`` and ``/search`` are available for querying. + +``GET /`` + +``GET /search`` + +Parameters +~~~~~~~~~~ + .. code:: sh - GET / + q + +The search query. This string is passed to external search services. +Thus, searx supports syntax of each search service. For example, ``site:github.com searx`` is a valid +query for Google. However, if simply the query above is passed to any search engine which does not filter its +results based on this syntax, you might not get the results you wanted. + + +See more at :doc:`/user/search_syntax` + +Required. .. code:: sh - GET /search + categories -Parameters -`````````` - -+------------------+----------------------------------------------------------------------------------------------------+--------------------------------------------------+ -| Name | Description | | -+==================+====================================================================================================+==================================================+ -| ``q`` | The search query, see :doc:`/user/search_syntax` | required | -+------------------+----------------------------------------------------------------------------------------------------+--------------------------------------------------+ -| ``categories`` | Comma separated list, specifies the active search categories | optional | -+------------------+----------------------------------------------------------------------------------------------------+--------------------------------------------------+ -| ``engines`` | Comma separated list, specifies the active search engines | optional | -+------------------+----------------------------------------------------------------------------------------------------+--------------------------------------------------+ -| ``lang`` | Code of the language | optional (default: ``all``) | -+------------------+----------------------------------------------------------------------------------------------------+--------------------------------------------------+ -| ``pageno`` | Search page number | optional (default: ``1``) | -+------------------+----------------------------------------------------------------------------------------------------+--------------------------------------------------+ -| ``time_range`` | Time range of search | optional (possible: ``day``, ``month``, ``year``)| -+------------------+----------------------------------------------------------------------------------------------------+--------------------------------------------------+ -| ``format`` | Output format of results | optional (possible: ``json``, ``csv``, ``rss``) | -+------------------+----------------------------------------------------------------------------------------------------+--------------------------------------------------+ +Comma separated list, specifies the active search categories + +Optional. + +.. code:: sh + + engines + +Comma separated list, specifies the active search engines. + +Optional. + +.. code:: sh + + lang + +Code of the language. + +Optional. + +Default: ``all`` + +.. code:: sh + + pageno + +Search page number. + +Optional. + +Default: ``1`` + +.. code:: sh + + time_range + +Time range of search for engines which support it. See if an engine supports time range search in the preferences page of an instance. + +Optional. + +Possible: ``day``, ``month``, ``year`` + +.. code:: sh + + format + +Output format of results. + +Optional. + +Possible: ``json``, ``csv``, ``rss`` + +.. code:: sh + + results_on_new_tab + +Open search results on new tab. + +Optional. + +Default: ``0`` + +Possible: ``0``, ``1`` + +.. code:: sh + + image_proxy + +Proxy image results through searx. + +Optional. + +Default: ``False`` + +Possible: ``True``, ``False`` + +.. code:: sh + + autocomplete + +Service which completes words as you type. + +Optional. + +Default: empty + +Possible: ``google``, ``dbpedia``, ``duckduckgo``, ``startpage``, ``wikipedia`` + +.. code:: sh + + safesearch + +Filter search results of engines which support safe search. See if an engine supports safe search in the preferences page of an instance. + +Optional. + +Default: ``None`` + +Possible: ``0``, ``1``, ``None`` + +.. code:: sh + + theme + +Theme of instance. + +Optional. + +Default: ``oscar`` + +Possible: ``oscar``, ``simple``, ``legacy``, ``pix-art``, ``courgette`` + +Please note, available themes depend on an instance. It is possible that an instance administrator deleted, created or renamed themes on his/her instance. See the available options in the preferences page of the instance. + +.. code:: sh + + oscar-style + +Style of Oscar theme. It is only parsed if the theme of an instance is ``oscar``. + +Optional. + +Default: ``logicodev`` + +Possible: ``pointhi``, ``logicodev`` + +Please note, available styles depend on an instance. It is possible that an instance administrator deleted, created or renamed styles on his/her instance. See the available options in the preferences page of the instance. + +.. code:: sh + + enabled_plugins + +List of enabled plugins. + +Optional. + +Default: ``HTTPS_rewrite``, ``Self_Informations``, ``Search_on_category_select``, ``Tracker_URL_remover`` + +Possible: ``DOAI_rewrite``, ``HTTPS_rewrite``, ``Infinite_scroll``, ``Vim-like_hotkeys``, ``Self_Informations``, ``Tracker_URL_remover``, ``Search_on_category_select`` + +.. code:: sh + + disabled_plugins + +List of disabled plugins. + +Optional. + +Default: ``DOAI_rewrite``, ``Infinite_scroll``, ``Vim-like_hotkeys`` + +Possible: ``DOAI_rewrite``, ``HTTPS_rewrite``, ``Infinite_scroll``, ``Vim-like_hotkeys``, ``Self_Informations``, ``Tracker_URL_remover``, ``Search_on_category_select`` + +.. code:: sh + + enabled_engines + +List of enabled engines. + +Optional. + +Possible: all engines + +.. code:: sh + + disabled_engines + +List of disabled engines. + +Optional. + +Possible: all engines -- cgit v1.2.3 From 2134c7c6939ce4f20fbb6ffca330ad7af0ff673a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?No=C3=A9mi=20V=C3=A1nyi?= Date: Wed, 17 Jan 2018 22:05:10 +0100 Subject: change python-pybabel to python-babel in package requirements Closes #1147 --- docs/dev/install/installation.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'docs/dev') diff --git a/docs/dev/install/installation.rst b/docs/dev/install/installation.rst index e1ae3f2a1..8b6bb69b0 100644 --- a/docs/dev/install/installation.rst +++ b/docs/dev/install/installation.rst @@ -15,7 +15,7 @@ Install packages: .. code:: sh - sudo apt-get install git build-essential libxslt-dev python-dev python-virtualenv python-pybabel zlib1g-dev libffi-dev libssl-dev + sudo apt-get install git build-essential libxslt-dev python-dev python-virtualenv python-babel zlib1g-dev libffi-dev libssl-dev Install searx: -- cgit v1.2.3 From 94af140318338ab6e7b064b3b26b5fb45523d17a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9o=20Bourrel?= Date: Thu, 30 Aug 2018 11:14:03 +0200 Subject: Update quickstart.rst Rename `build_grunt` to `grunt_build` as in manage.sh --- docs/dev/quickstart.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'docs/dev') diff --git a/docs/dev/quickstart.rst b/docs/dev/quickstart.rst index 92ef62385..e7d4baf46 100644 --- a/docs/dev/quickstart.rst +++ b/docs/dev/quickstart.rst @@ -89,7 +89,7 @@ After installing grunt, the files can be built using the following command: .. code:: sh - ./manage.sh build_grunt + ./manage.sh grunt_build -- cgit v1.2.3 From 10ebe8cec9d926ca1c7c19336bf60c1ee88d9300 Mon Sep 17 00:00:00 2001 From: tmikaeld Date: Sun, 21 Oct 2018 11:36:11 +0200 Subject: Added missing symlink instruction --- docs/dev/install/installation.rst | 4 ++++ 1 file changed, 4 insertions(+) (limited to 'docs/dev') diff --git a/docs/dev/install/installation.rst b/docs/dev/install/installation.rst index 8b6bb69b0..6c3f773a5 100644 --- a/docs/dev/install/installation.rst +++ b/docs/dev/install/installation.rst @@ -150,6 +150,10 @@ content: } } +Create a symlink to sites-enabled: + +sudo ln -s /etc/nginx/sites-available/searx /etc/nginx/sites-enabled/searx + Restart service: .. code:: sh -- cgit v1.2.3 From 77494166d2c61dcc5e62447e88e6d3bb4e5254c2 Mon Sep 17 00:00:00 2001 From: tmikaeld Date: Sun, 21 Oct 2018 11:42:00 +0200 Subject: Formatting... --- docs/dev/install/installation.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) (limited to 'docs/dev') diff --git a/docs/dev/install/installation.rst b/docs/dev/install/installation.rst index 6c3f773a5..50b33a324 100644 --- a/docs/dev/install/installation.rst +++ b/docs/dev/install/installation.rst @@ -152,7 +152,9 @@ content: Create a symlink to sites-enabled: -sudo ln -s /etc/nginx/sites-available/searx /etc/nginx/sites-enabled/searx +.. code:: sh + + sudo ln -s /etc/nginx/sites-available/searx /etc/nginx/sites-enabled/searx Restart service: -- cgit v1.2.3 From 021e1da4c9a8eca7293ad62608cc55da5dfde73f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?No=C3=A9mi=20V=C3=A1nyi?= Date: Wed, 23 Oct 2019 13:06:19 +0200 Subject: add post about introducing offline engines --- docs/dev/engine_overview.rst | 2 ++ 1 file changed, 2 insertions(+) (limited to 'docs/dev') diff --git a/docs/dev/engine_overview.rst b/docs/dev/engine_overview.rst index d57ca5b38..a6867b5d0 100644 --- a/docs/dev/engine_overview.rst +++ b/docs/dev/engine_overview.rst @@ -41,6 +41,8 @@ engine file +----------------------+-----------+-----------------------------------------+ | time\_range\_support | boolean | support search time range | +----------------------+-----------+-----------------------------------------+ +| offline | boolean | engine runs offline | ++----------------------+-----------+-----------------------------------------+ settings.yml ~~~~~~~~~~~~ -- cgit v1.2.3 From af2cae6d1d71859e867a9fbc5da604ef6a898794 Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Wed, 4 Dec 2019 16:48:36 +0100 Subject: doc: make use of sphinx.ext.extlinks & sphinx.ext.intersphinx - add sphinx extensions - patch documentation to make use of These modules help to simplify the reST markup of external references. BTW it helps to write more readable reST and form custom brands. Signed-off-by: Markus Heiser --- docs/dev/engine_overview.rst | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/engine_overview.rst b/docs/dev/engine_overview.rst index a6867b5d0..937ca8880 100644 --- a/docs/dev/engine_overview.rst +++ b/docs/dev/engine_overview.rst @@ -8,8 +8,7 @@ so it uses different search engines to provide better results. Because there is no general search API which could be used for every search engine, an adapter has to be built between searx and the external search engines. Adapters are stored under the folder -`searx/engines -`__. +:origin:`searx/engines`. .. contents:: @@ -199,7 +198,7 @@ default +--------------------+---------------------------------------------------------------------------------------------------------------+ | content | string, general result-text | +--------------------+---------------------------------------------------------------------------------------------------------------+ -| publishedDate | `datetime.datetime `__, time of publish | +| publishedDate | :py:class:`datetime.datetime`, time of publish | +--------------------+---------------------------------------------------------------------------------------------------------------+ images @@ -218,7 +217,7 @@ to use this template, the parameter +--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ | content | *(partly implemented)* | +--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| publishedDate | `datetime.datetime `__, time of publish *(partly implemented)* | +| publishedDate | :py:class:`datetime.datetime`, time of publish *(partly implemented)* | +--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ | img\_src | string, url to the result image | +--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ @@ -239,7 +238,7 @@ videos +--------------------+--------------------------------------------------------------------------------------------------------------+ | content | *(not implemented yet)* | +--------------------+--------------------------------------------------------------------------------------------------------------+ -| publishedDate | `datetime.datetime `__, time of publish | +| publishedDate | :py:class:`datetime.datetime`, time of publish | +--------------------+--------------------------------------------------------------------------------------------------------------+ | thumbnail | string, url to a small-preview image | +--------------------+--------------------------------------------------------------------------------------------------------------+ @@ -258,7 +257,7 @@ torrent +------------------+---------------------------------------------------------------------------------------------------------------------------------------+ | content | string, general result-text | +------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| publishedDate | `datetime.datetime `__, time of publish *(not implemented yet)* | +| publishedDate | :py:class:`datetime.datetime`, time of publish *(not implemented yet)* | +------------------+---------------------------------------------------------------------------------------------------------------------------------------+ | seed | int, number of seeder | +------------------+---------------------------------------------------------------------------------------------------------------------------------------+ @@ -286,7 +285,7 @@ map +-------------------------+--------------------------------------------------------------------------------------------------------------+ | content | string, general result-text | +-------------------------+--------------------------------------------------------------------------------------------------------------+ -| publishedDate | `datetime.datetime `__, time of publish | +| publishedDate | :py:class:`datetime.datetime`, time of publish | +-------------------------+--------------------------------------------------------------------------------------------------------------+ | latitude | latitude of result (in decimal format) | +-------------------------+--------------------------------------------------------------------------------------------------------------+ -- cgit v1.2.3 From 5b14772b203fa32a9a2c7a6951a3c17ff32b6d88 Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Wed, 4 Dec 2019 17:54:41 +0100 Subject: doc: installation.rst improve reST markup - normalize section header markup - limit all lines to a maximum of 79 characters - add option -H to the sudo command used in code blocks - place *WARNING* lines into a '.. warning::' admonition block - make use of ``literal`` markup for monospace rendering - drop useless indentation of lists [1] https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html Signed-off-by: Markus Heiser --- docs/dev/install/installation.rst | 156 ++++++++++++++++++++------------------ 1 file changed, 84 insertions(+), 72 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/install/installation.rst b/docs/dev/install/installation.rst index 50b33a324..239ce0704 100644 --- a/docs/dev/install/installation.rst +++ b/docs/dev/install/installation.rst @@ -1,5 +1,6 @@ .. _installation: +============ Installation ============ @@ -7,37 +8,44 @@ Installation :depth: 3 Basic installation ------------------- +================== -Step by step installation for Debian/Ubuntu with virtualenv. For Ubuntu, be sure to have enable universe repository. +Step by step installation for Debian/Ubuntu with virtualenv. For Ubuntu, be sure +to have enable universe repository. Install packages: .. code:: sh - sudo apt-get install git build-essential libxslt-dev python-dev python-virtualenv python-babel zlib1g-dev libffi-dev libssl-dev + $ sudo -H apt-get install \ + git build-essential libxslt-dev \ + python-dev python-virtualenv python-babel \ + zlib1g-dev libffi-dev libssl-dev Install searx: .. code:: sh cd /usr/local - sudo git clone https://github.com/asciimoo/searx.git - sudo useradd searx -d /usr/local/searx - sudo chown searx:searx -R /usr/local/searx + sudo -H git clone https://github.com/asciimoo/searx.git + sudo -H useradd searx -d /usr/local/searx + sudo -H chown searx:searx -R /usr/local/searx Install dependencies in a virtualenv: .. code:: sh - sudo -u searx -i cd /usr/local/searx - virtualenv searx-ve - . ./searx-ve/bin/activate - ./manage.sh update_packages + sudo -H -u searx -i + +.. code:: sh + + (searx)$ virtualenv searx-ve + (searx)$ . ./searx-ve/bin/activate + (searx)$ ./manage.sh update_packages Configuration -------------- +============== .. code:: sh @@ -46,7 +54,7 @@ Configuration Edit searx/settings.yml if necessary. Check ------ +===== Start searx: @@ -68,18 +76,19 @@ You can exit the virtualenv and the searx user bash (enter exit command twice). uwsgi ------ +===== Install packages: .. code:: sh - sudo apt-get install uwsgi uwsgi-plugin-python + sudo -H apt-get install \ + uwsgi uwsgi-plugin-python -Create the configuration file /etc/uwsgi/apps-available/searx.ini with -this content: +Create the configuration file ``/etc/uwsgi/apps-available/searx.ini`` with this +content: -:: +.. code:: ini [uwsgi] # Who will run the code @@ -119,22 +128,22 @@ Activate the uwsgi application and restart: /etc/init.d/uwsgi restart Web server ----------- +========== with nginx -^^^^^^^^^^ +---------- If nginx is not installed (uwsgi will not work with the package nginx-light): .. code:: sh - sudo apt-get install nginx + sudo -H apt-get install nginx Hosted at / -""""""""""" +~~~~~~~~~~~ -Create the configuration file /etc/nginx/sites-available/searx with this +Create the configuration file ``/etc/nginx/sites-available/searx`` with this content: .. code:: nginx @@ -154,20 +163,20 @@ Create a symlink to sites-enabled: .. code:: sh - sudo ln -s /etc/nginx/sites-available/searx /etc/nginx/sites-enabled/searx + sudo -H ln -s /etc/nginx/sites-available/searx /etc/nginx/sites-enabled/searx Restart service: .. code:: sh - sudo service nginx restart - sudo service uwsgi restart + sudo -H service nginx restart + sudo -H service uwsgi restart from subdirectory URL (/searx) -"""""""""""""""""""""""""""""" +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Add this configuration in the server config file -/etc/nginx/sites-enabled/default: +``/etc/nginx/sites-enabled/default``: .. code:: nginx @@ -183,10 +192,8 @@ Add this configuration in the server config file } -OR - -using reverse proxy -(Please, note that reverse proxy advised to be used in case of single-user or low-traffic instances.) +**OR** using reverse proxy (Please, note that reverse proxy advised to be used +in case of single-user or low-traffic instances.) .. code:: nginx @@ -200,9 +207,9 @@ using reverse proxy } -Enable base\_url in searx/settings.yml +Enable ``base_url`` in ``searx/settings.yml`` -:: +.. code:: yaml base_url : http://your.domain.tld/searx/ @@ -210,18 +217,18 @@ Restart service: .. code:: sh - sudo service nginx restart - sudo service uwsgi restart + sudo -H service nginx restart + sudo -H service uwsgi restart disable logs -~~~~~~~~~~~~ +^^^^^^^^^^^^ for better privacy you can disable nginx logs about searx. -how to proceed: below ``uwsgi_pass`` in -/etc/nginx/sites-available/default add +how to proceed: below ``uwsgi_pass`` in ``/etc/nginx/sites-available/default`` +add: -:: +.. code:: nginx access_log /dev/null; error_log /dev/null; @@ -230,19 +237,19 @@ Restart service: .. code:: sh - sudo service nginx restart + sudo -H service nginx restart with apache -^^^^^^^^^^^ +----------- Add wsgi mod: .. code:: sh - sudo apt-get install libapache2-mod-uwsgi - sudo a2enmod uwsgi + sudo -H apt-get install libapache2-mod-uwsgi + sudo -H a2enmod uwsgi -Add this configuration in the file /etc/apache2/apache2.conf: +Add this configuration in the file ``/etc/apache2/apache2.conf``: .. code:: apache @@ -252,27 +259,26 @@ Add this configuration in the file /etc/apache2/apache2.conf: uWSGISocket /run/uwsgi/app/searx/socket -Note that if your instance of searx is not at the root, you should -change ```` by the location of your instance, like -````. +Note that if your instance of searx is not at the root, you should change +```` by the location of your instance, like ````. Restart Apache: .. code:: sh - sudo /etc/init.d/apache2 restart + sudo -H /etc/init.d/apache2 restart disable logs -"""""""""""" +~~~~~~~~~~~~ For better privacy you can disable Apache logs. -WARNING: not tested +.. warning:: -WARNING: you can only disable logs for the whole (virtual) server not -for a specific path. + You can only disable logs for the whole (virtual) server not for a specific + path. -Go back to /etc/apache2/apache2.conf and above ```` add: +Go back to ``/etc/apache2/apache2.conf`` and above ```` add: .. code:: apache @@ -282,24 +288,30 @@ Restart Apache: .. code:: sh - sudo /etc/init.d/apache2 restart + sudo -H /etc/init.d/apache2 restart How to update -------------- +============= .. code:: sh cd /usr/local/searx - sudo -u searx -i - . ./searx-ve/bin/activate - git stash - git pull origin master - git stash apply - ./manage.sh update_packages - sudo service uwsgi restart + sudo -H -u searx -i + +.. code:: sh + + (searx)$ . ./searx-ve/bin/activate + (searx)$ git stash + (searx)$ git pull origin master + (searx)$ git stash apply + (searx)$ ./manage.sh update_packages + +.. code:: sh + + sudo -H service uwsgi restart Docker ------- +====== Make sure you have installed Docker. For instance, you can deploy searx like this: @@ -308,22 +320,22 @@ Make sure you have installed Docker. For instance, you can deploy searx like thi docker pull wonderfall/searx docker run -d --name searx -p $PORT:8888 wonderfall/searx -Go to http://localhost:$PORT. - -See https://hub.docker.com/r/wonderfall/searx/ for more informations. +Go to ``http://localhost:$PORT``. -It's also possible to build searx from the embedded Dockerfile. +See https://hub.docker.com/r/wonderfall/searx/ for more informations. It's also +possible to build searx from the embedded Dockerfile. .. code:: sh - git clone https://github.com/asciimoo/searx.git - cd searx - docker build -t whatever/searx . + git clone https://github.com/asciimoo/searx.git + cd searx + docker build -t whatever/searx . References ========== - * https://about.okhin.fr/posts/Searx/ with some additions +* https://about.okhin.fr/posts/Searx/ with some additions - * How to: `Setup searx in a couple of hours with a free SSL certificate `__ +* How to: `Setup searx in a couple of hours with a free SSL certificate + `__ -- cgit v1.2.3 From e9fff4fde6d7a8bec3fae087d2afe1fce2145f22 Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Thu, 12 Dec 2019 19:20:56 +0100 Subject: doc: proofread of the all reST sources (no content change) Normalize reST sources with best practice and KISS in mind. to name a few points: - simplify reST tables - make use of ``literal`` markup for monospace rendering - fix code-blocks for better rendering in HTML - normalize section header markup - limit all lines to a maximum of 79 characters - add option -H to the sudo command used in code blocks - drop useless indentation of lists - ... [1] https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html Signed-off-by: Markus Heiser --- docs/dev/contribution_guide.rst | 103 +++++---- docs/dev/engine_overview.rst | 453 ++++++++++++++++++---------------------- docs/dev/plugins.rst | 45 ++-- docs/dev/quickstart.rst | 92 ++++---- docs/dev/search_api.rst | 226 ++++++-------------- docs/dev/translation.rst | 44 ++-- 6 files changed, 414 insertions(+), 549 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/contribution_guide.rst b/docs/dev/contribution_guide.rst index 65e587294..9414de2c9 100644 --- a/docs/dev/contribution_guide.rst +++ b/docs/dev/contribution_guide.rst @@ -1,66 +1,68 @@ +================= How to contribute ------------------ +================= Prime directives: Privacy, Hackability -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +====================================== -Searx has two prime directives, privacy-by-design and hackability. The +Searx has two prime directives, **privacy-by-design and hackability** . The hackability comes in three levels: -- support of search engines -- plugins to alter search behaviour -- hacking searx itself +- support of search engines +- plugins to alter search behaviour +- hacking searx itself -Note the lack of "world domination" among the directives. -Searx has no intention of wide mass-adoption, rounded -corners, etc. The prime directive "privacy" deserves a separate -chapter, as it's quite uncommon unfortunately. +Note the lack of "world domination" among the directives. Searx has no +intention of wide mass-adoption, rounded corners, etc. The prime directive +"privacy" deserves a separate chapter, as it's quite uncommon unfortunately. Privacy-by-design -^^^^^^^^^^^^^^^^^ +----------------- -Searx was born out of the need for a privacy-respecting search tool -which can be extended easily to maximize both its search and its -privacy protecting capabilities. +Searx was born out of the need for a **privacy-respecting** search tool which +can be extended easily to maximize both, its search and its privacy protecting +capabilities. -A few widely used features work differently or turned off by default or not implemented -at all as a consequence of privacy-by-design. +A few widely used features work differently or turned off by default or not +implemented at all **as a consequence of privacy-by-design**. -If a feature reduces the privacy preserving aspects of searx, it -should be switched off by default or should not implemented at all. -There are plenty of search engines already providing such features. -If a feature reduces the protection of searx, users must be -informed about the effect of choosing to enable it. Features -that protect privacy but differ from the expectations of the -user should also be explained. +If a feature reduces the privacy preserving aspects of searx, it should be +switched off by default or should not implemented at all. There are plenty of +search engines already providing such features. If a feature reduces the +protection of searx, users must be informed about the effect of choosing to +enable it. Features that protect privacy but differ from the expectations of +the user should also be explained. -Also, if you think that something works weird with searx, -it's might be because of the tool you use is designed in a way to interfere with -the privacy respect. Submitting a bugreport to the vendor of the tool that -misbehaves might be a good feedback to reconsider the disrespect to -its customers (e.g. GET vs POST requests in various browsers). +Also, if you think that something works weird with searx, it's might be because +of the tool you use is designed in a way to interfere with the privacy respect. +Submitting a bugreport to the vendor of the tool that misbehaves might be a good +feedback to reconsider the disrespect to its customers (e.g. ``GET`` vs ``POST`` +requests in various browsers). -Remember the other prime directive of searx is to be hackable, so if the -above privacy concerns do not fancy you, simply fork it. +Remember the other prime directive of searx is to be hackable, so if the above +privacy concerns do not fancy you, simply fork it. -Happy hacking. + *Happy hacking.* Code -~~~~ +==== + +.. _PEP8: https://www.python.org/dev/peps/pep-0008/ + In order to submit a patch, please follow the steps below: - Follow coding conventions. - - PEP8 standards apply, except the convention of line length - + - PEP8_ standards apply, except the convention of line length - Maximum line length is 120 characters -- Check if your code breaks existing tests. If so, update the tests or fix your code. +- Check if your code breaks existing tests. If so, update the tests or fix your + code. - If your code can be unit-tested, add unit tests. -- Add yourself to the AUTHORS file. +- Add yourself to the :origin:`AUTHORS.rst` file. - Create a pull request. @@ -68,31 +70,20 @@ For more help on getting started with searx development, see :ref:`devquickstart Translation -~~~~~~~~~~~ +=========== -Translation currently takes place on -`transifex `__. +Translation currently takes place on :ref:`transifex `. -**Please, do not update translation files in the repo.** +.. caution:: -Documentation -~~~~~~~~~~~~~ + Please, do not update translation files in the repo. -The documentation is built using Sphinx. So in order to be able to generate the required -files, you have to install it on your system. (It can be installed easily using pip.) -1. Checkout the gh-pages branch. - -2. Edit the rst file you wish to update. Or create a new rst file and place it under the appropriate folder. - -3. Build the documentation using Sphinx. - -4. Add the updated and created files of these extension: - - - .rst +Documentation +============= - - .html +.. admonition:: ToDo - - .txt + docs are no longer in gh-branch -6. Create a pull request. + Update this chapter and document the /docs workflow!! diff --git a/docs/dev/engine_overview.rst b/docs/dev/engine_overview.rst index 937ca8880..92405dc64 100644 --- a/docs/dev/engine_overview.rst +++ b/docs/dev/engine_overview.rst @@ -1,21 +1,25 @@ + +.. _engines-dev: + +=============== Engine overview =============== +.. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine -searx is a `metasearch-engine `__, -so it uses different search engines to provide better results. - -Because there is no general search API which could be used for every -search engine, an adapter has to be built between searx and the -external search engines. Adapters are stored under the folder -:origin:`searx/engines`. +searx is a metasearch-engine_, so it uses different search engines to provide +better results. +Because there is no general search API which could be used for every search +engine, an adapter has to be built between searx and the external search +engines. Adapters are stored under the folder :origin:`searx/engines`. .. contents:: :depth: 3 + :backlinks: entry general engine configuration ----------------------------- +============================ It is required to tell searx the type of results the engine provides. The arguments can be set in the engine file or in the settings file @@ -23,292 +27,239 @@ arguments can be set in the engine file or in the settings file the ones in the engine file. It does not matter if an option is stored in the engine file or in the -settings. However, the standard way is the following: +settings. However, the standard way is the following: engine file -~~~~~~~~~~~ - -+----------------------+-----------+-----------------------------------------+ -| argument | type | information | -+======================+===========+=========================================+ -| categories | list | pages, in which the engine is working | -+----------------------+-----------+-----------------------------------------+ -| paging | boolean | support multible pages | -+----------------------+-----------+-----------------------------------------+ -| language\_support | boolean | support language choosing | -+----------------------+-----------+-----------------------------------------+ -| time\_range\_support | boolean | support search time range | -+----------------------+-----------+-----------------------------------------+ -| offline | boolean | engine runs offline | -+----------------------+-----------+-----------------------------------------+ +----------- + +======================= =========== =========================================== +argument type information +======================= =========== =========================================== +categories list pages, in which the engine is working +paging boolean support multible pages +language_support boolean support language choosing +time_range_support boolean support search time range +offline boolean engine runs offline +======================= =========== =========================================== + settings.yml -~~~~~~~~~~~~ - -+------------+----------+-----------------------------------------------+ -| argument | type | information | -+============+==========+===============================================+ -| name | string | name of search-engine | -+------------+----------+-----------------------------------------------+ -| engine | string | name of searx-engine (filename without .py) | -+------------+----------+-----------------------------------------------+ -| shortcut | string | shortcut of search-engine | -+------------+----------+-----------------------------------------------+ -| timeout | string | specific timeout for search-engine | -+------------+----------+-----------------------------------------------+ +------------ + +======================= =========== =========================================== +argument type information +======================= =========== =========================================== +name string name of search-engine +engine string name of searx-engine + (filename without ``.py``) +shortcut string shortcut of search-engine +timeout string specific timeout for search-engine +======================= =========== =========================================== + overrides -~~~~~~~~~ - -A few of the options have default values in the engine, but are -often overwritten by the settings. If ``None`` is assigned to an option -in the engine file, it has to be redefined in the settings, -otherwise searx will not start with that engine. - -The naming of overrides is arbitrary. But the recommended -overrides are the following: - -+-----------------------+----------+----------------------------------------------------------------+ -| argument | type | information | -+=======================+==========+================================================================+ -| base\_url | string | base-url, can be overwritten to use same engine on other URL | -+-----------------------+----------+----------------------------------------------------------------+ -| number\_of\_results | int | maximum number of results per request | -+-----------------------+----------+----------------------------------------------------------------+ -| language | string | ISO code of language and country like en\_US | -+-----------------------+----------+----------------------------------------------------------------+ -| api\_key | string | api-key if required by engine | -+-----------------------+----------+----------------------------------------------------------------+ +--------- + +A few of the options have default values in the engine, but are often +overwritten by the settings. If ``None`` is assigned to an option in the engine +file, it has to be redefined in the settings, otherwise searx will not start +with that engine. + +The naming of overrides is arbitrary. But the recommended overrides are the +following: + +======================= =========== =========================================== +argument type information +======================= =========== =========================================== +base_url string base-url, can be overwritten to use same + engine on other URL +number_of_results int maximum number of results per request +language string ISO code of language and country like en_US +api_key string api-key if required by engine +======================= =========== =========================================== example code -~~~~~~~~~~~~ +------------ .. code:: python - # engine dependent config - categories = ['general'] - paging = True - language_support = True + # engine dependent config + categories = ['general'] + paging = True + language_support = True + making a request ----------------- +================ -To perform a search an URL have to be specified. In addition to -specifying an URL, arguments can be passed to the query. +To perform a search an URL have to be specified. In addition to specifying an +URL, arguments can be passed to the query. passed arguments -~~~~~~~~~~~~~~~~ +---------------- -These arguments can be used to construct the search query. Furthermore, +These arguments can be used to construct the search query. Furthermore, parameters with default value can be redefined for special purposes. -+----------------------+------------+------------------------------------------------------------------------+ -| argument | type | default-value, information | -+======================+============+========================================================================+ -| url | string | ``''`` | -+----------------------+------------+------------------------------------------------------------------------+ -| method | string | ``'GET'`` | -+----------------------+------------+------------------------------------------------------------------------+ -| headers | set | ``{}`` | -+----------------------+------------+------------------------------------------------------------------------+ -| data | set | ``{}`` | -+----------------------+------------+------------------------------------------------------------------------+ -| cookies | set | ``{}`` | -+----------------------+------------+------------------------------------------------------------------------+ -| verify | boolean | ``True`` | -+----------------------+------------+------------------------------------------------------------------------+ -| headers.User-Agent | string | a random User-Agent | -+----------------------+------------+------------------------------------------------------------------------+ -| category | string | current category, like ``'general'`` | -+----------------------+------------+------------------------------------------------------------------------+ -| started | datetime | current date-time | -+----------------------+------------+------------------------------------------------------------------------+ -| pageno | int | current pagenumber | -+----------------------+------------+------------------------------------------------------------------------+ -| language | string | specific language code like ``'en_US'``, or ``'all'`` if unspecified | -+----------------------+------------+------------------------------------------------------------------------+ +====================== ============ ======================================================================== +argument type default-value, information +====================== ============ ======================================================================== +url string ``''`` +method string ``'GET'`` +headers set ``{}`` +data set ``{}`` +cookies set ``{}`` +verify boolean ``True`` +headers.User-Agent string a random User-Agent +category string current category, like ``'general'`` +started datetime current date-time +pageno int current pagenumber +language string specific language code like ``'en_US'``, or ``'all'`` if unspecified +====================== ============ ======================================================================== parsed arguments -~~~~~~~~~~~~~~~~ - -The function ``def request(query, params):`` always returns the -``params`` variable. Inside searx, the following paramters can be -used to specify a search request: - -+------------+-----------+---------------------------------------------------------+ -| argument | type | information | -+============+===========+=========================================================+ -| url | string | requested url | -+------------+-----------+---------------------------------------------------------+ -| method | string | HTTP request method | -+------------+-----------+---------------------------------------------------------+ -| headers | set | HTTP header information | -+------------+-----------+---------------------------------------------------------+ -| data | set | HTTP data information (parsed if ``method != 'GET'``) | -+------------+-----------+---------------------------------------------------------+ -| cookies | set | HTTP cookies | -+------------+-----------+---------------------------------------------------------+ -| verify | boolean | Performing SSL-Validity check | -+------------+-----------+---------------------------------------------------------+ +---------------- + +The function ``def request(query, params):`` always returns the ``params`` +variable. Inside searx, the following paramters can be used to specify a search +request: + +============ =========== ========================================================= +argument type information +============ =========== ========================================================= +url string requested url +method string HTTP request method +headers set HTTP header information +data set HTTP data information (parsed if ``method != 'GET'``) +cookies set HTTP cookies +verify boolean Performing SSL-Validity check +============ =========== ========================================================= + example code -~~~~~~~~~~~~ +------------ .. code:: python - # search-url - base_url = 'https://example.com/' - search_string = 'search?{query}&page={page}' + # search-url + base_url = 'https://example.com/' + search_string = 'search?{query}&page={page}' - # do search-request - def request(query, params): - search_path = search_string.format( - query=urlencode({'q': query}), - page=params['pageno']) + # do search-request + def request(query, params): + search_path = search_string.format( + query=urlencode({'q': query}), + page=params['pageno']) - params['url'] = base_url + search_path + params['url'] = base_url + search_path + + return params - return params returned results ----------------- +================ -Searx is able to return results of different media-types. -Currently the following media-types are supported: +Searx is able to return results of different media-types. Currently the +following media-types are supported: -- default -- images -- videos -- torrent -- map +- default_ +- images_ +- videos_ +- torrent_ +- map_ -To set another media-type as default, the parameter -``template`` must be set to the desired type. +To set another media-type as default, the parameter ``template`` must be set to +the desired type. default -~~~~~~~ - -+--------------------+---------------------------------------------------------------------------------------------------------------+ -| result-parameter | information | -+====================+===============================================================================================================+ -| url | string, url of the result | -+--------------------+---------------------------------------------------------------------------------------------------------------+ -| title | string, title of the result | -+--------------------+---------------------------------------------------------------------------------------------------------------+ -| content | string, general result-text | -+--------------------+---------------------------------------------------------------------------------------------------------------+ -| publishedDate | :py:class:`datetime.datetime`, time of publish | -+--------------------+---------------------------------------------------------------------------------------------------------------+ +------- + +========================= ===================================================== +result-parameter information +========================= ===================================================== +url string, url of the result +title string, title of the result +content string, general result-text +publishedDate :py:class:`datetime.datetime`, time of publish +========================= ===================================================== images -~~~~~~ - -to use this template, the parameter - -+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| result-parameter | information | -+====================+=======================================================================================================================================+ -| template | is set to ``images.html`` | -+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| url | string, url to the result site | -+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| title | string, title of the result *(partly implemented)* | -+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| content | *(partly implemented)* | -+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| publishedDate | :py:class:`datetime.datetime`, time of publish *(partly implemented)* | -+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| img\_src | string, url to the result image | -+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| thumbnail\_src | string, url to a small-preview image | -+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +------ + +To use this template, the parameter: + +========================= ===================================================== +result-parameter information +========================= ===================================================== +template is set to ``images.html`` +url string, url to the result site +title string, title of the result *(partly implemented)* +content *(partly implemented)* +publishedDate :py:class:`datetime.datetime`, + time of publish *(partly implemented)* +img\_src string, url to the result image +thumbnail\_src string, url to a small-preview image +========================= ===================================================== videos -~~~~~~ - -+--------------------+--------------------------------------------------------------------------------------------------------------+ -| result-parameter | information | -+====================+==============================================================================================================+ -| template | is set to ``videos.html`` | -+--------------------+--------------------------------------------------------------------------------------------------------------+ -| url | string, url of the result | -+--------------------+--------------------------------------------------------------------------------------------------------------+ -| title | string, title of the result | -+--------------------+--------------------------------------------------------------------------------------------------------------+ -| content | *(not implemented yet)* | -+--------------------+--------------------------------------------------------------------------------------------------------------+ -| publishedDate | :py:class:`datetime.datetime`, time of publish | -+--------------------+--------------------------------------------------------------------------------------------------------------+ -| thumbnail | string, url to a small-preview image | -+--------------------+--------------------------------------------------------------------------------------------------------------+ +------ + +========================= ===================================================== +result-parameter information +========================= ===================================================== +template is set to ``videos.html`` +url string, url of the result +title string, title of the result +content *(not implemented yet)* +publishedDate :py:class:`datetime.datetime`, time of publish +thumbnail string, url to a small-preview image +========================= ===================================================== torrent -~~~~~~~ - -+------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| result-parameter | information | -+==================+=======================================================================================================================================+ -| template | is set to ``torrent.html`` | -+------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| url | string, url of the result | -+------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| title | string, title of the result | -+------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| content | string, general result-text | -+------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| publishedDate | :py:class:`datetime.datetime`, time of publish *(not implemented yet)* | -+------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| seed | int, number of seeder | -+------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| leech | int, number of leecher | -+------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| filesize | int, size of file in bytes | -+------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| files | int, number of files | -+------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| magnetlink | string, `magnetlink `__ of the result | -+------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| torrentfile | string, torrentfile of the result | -+------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +------- + +.. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme + +========================= ===================================================== +result-parameter information +========================= ===================================================== +template is set to ``torrent.html`` +url string, url of the result +title string, title of the result +content string, general result-text +publishedDate :py:class:`datetime.datetime`, + time of publish *(not implemented yet)* +seed int, number of seeder +leech int, number of leecher +filesize int, size of file in bytes +files int, number of files +magnetlink string, magnetlink_ of the result +torrentfile string, torrentfile of the result +========================= ===================================================== map -~~~ - -+-------------------------+--------------------------------------------------------------------------------------------------------------+ -| result-parameter | information | -+=========================+==============================================================================================================+ -| url | string, url of the result | -+-------------------------+--------------------------------------------------------------------------------------------------------------+ -| title | string, title of the result | -+-------------------------+--------------------------------------------------------------------------------------------------------------+ -| content | string, general result-text | -+-------------------------+--------------------------------------------------------------------------------------------------------------+ -| publishedDate | :py:class:`datetime.datetime`, time of publish | -+-------------------------+--------------------------------------------------------------------------------------------------------------+ -| latitude | latitude of result (in decimal format) | -+-------------------------+--------------------------------------------------------------------------------------------------------------+ -| longitude | longitude of result (in decimal format) | -+-------------------------+--------------------------------------------------------------------------------------------------------------+ -| boundingbox | boundingbox of result (array of 4. values ``[lat-min, lat-max, lon-min, lon-max]``) | -+-------------------------+--------------------------------------------------------------------------------------------------------------+ -| geojson | geojson of result (http://geojson.org) | -+-------------------------+--------------------------------------------------------------------------------------------------------------+ -| osm.type | type of osm-object (if OSM-Result) | -+-------------------------+--------------------------------------------------------------------------------------------------------------+ -| osm.id | id of osm-object (if OSM-Result) | -+-------------------------+--------------------------------------------------------------------------------------------------------------+ -| address.name | name of object | -+-------------------------+--------------------------------------------------------------------------------------------------------------+ -| address.road | street name of object | -+-------------------------+--------------------------------------------------------------------------------------------------------------+ -| address.house\_number | house number of object | -+-------------------------+--------------------------------------------------------------------------------------------------------------+ -| address.locality | city, place of object | -+-------------------------+--------------------------------------------------------------------------------------------------------------+ -| address.postcode | postcode of object | -+-------------------------+--------------------------------------------------------------------------------------------------------------+ -| address.country | country of object | -+-------------------------+--------------------------------------------------------------------------------------------------------------+ - +--- + +========================= ===================================================== +result-parameter information +========================= ===================================================== +url string, url of the result +title string, title of the result +content string, general result-text +publishedDate :py:class:`datetime.datetime`, time of publish +latitude latitude of result (in decimal format) +longitude longitude of result (in decimal format) +boundingbox boundingbox of result (array of 4. values + ``[lat-min, lat-max, lon-min, lon-max]``) +geojson geojson of result (http://geojson.org) +osm.type type of osm-object (if OSM-Result) +osm.id id of osm-object (if OSM-Result) +address.name name of object +address.road street name of object +address.house_number house number of object +address.locality city, place of object +address.postcode postcode of object +address.country country of object +========================= ===================================================== diff --git a/docs/dev/plugins.rst b/docs/dev/plugins.rst index 1d77f8611..e97bbeb4a 100644 --- a/docs/dev/plugins.rst +++ b/docs/dev/plugins.rst @@ -1,45 +1,48 @@ +======= Plugins -------- +======= -Plugins can extend or replace functionality of various components of -searx. +Plugins can extend or replace functionality of various components of searx. Example plugin -~~~~~~~~~~~~~~ +============== .. code:: python - name = 'Example plugin' - description = 'This plugin extends the suggestions with the word "example"' - default_on = False # disabled by default + name = 'Example plugin' + description = 'This plugin extends the suggestions with the word "example"' + default_on = False # disabled by default - js_dependencies = tuple() # optional, list of static js files - css_dependencies = tuple() # optional, list of static css files + js_dependencies = tuple() # optional, list of static js files + css_dependencies = tuple() # optional, list of static css files - # attach callback to the post search hook - # request: flask request object - # ctx: the whole local context of the post search hook - def post_search(request, ctx): - ctx['search'].suggestions.add('example') - return True + # attach callback to the post search hook + # request: flask request object + # ctx: the whole local context of the post search hook + def post_search(request, ctx): + ctx['search'].suggestions.add('example') + return True Plugin entry points -~~~~~~~~~~~~~~~~~~~ +=================== -Entry points (hooks) define when a plugin runs. Right now only three hooks are implemented. So feel free to implement a hook if it fits the behaviour of your plugin. +Entry points (hooks) define when a plugin runs. Right now only three hooks are +implemented. So feel free to implement a hook if it fits the behaviour of your +plugin. Pre search hook -``````````````` +--------------- Runs BEFORE the search request. Function to implement: ``pre_search`` Post search hook -```````````````` +---------------- Runs AFTER the search request. Function to implement: ``post_search`` Result hook -``````````` +----------- -Runs when a new result is added to the result list. Function to implement: ``on_result`` +Runs when a new result is added to the result list. Function to implement: +``on_result`` diff --git a/docs/dev/quickstart.rst b/docs/dev/quickstart.rst index e7d4baf46..a4a37a266 100644 --- a/docs/dev/quickstart.rst +++ b/docs/dev/quickstart.rst @@ -1,17 +1,18 @@ .. _devquickstart: +====================== Development Quickstart ----------------------- +====================== -This quickstart guide gets your environment set up with searx. Furthermore, it gives a -short introduction to the new manage.sh script. +This quickstart guide gets your environment set up with searx. Furthermore, it +gives a short introduction to the ``manage.sh`` script. How to setup your development environment -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +========================================= -First, clone the source code of searx to the desired folder. In this case the source -is cloned to ~/myprojects/searx. Then create and activate the searx-ve -virtualenv and install the required packages using manage.sh. +First, clone the source code of searx to the desired folder. In this case the +source is cloned to ``~/myprojects/searx``. Then create and activate the +searx-ve virtualenv and install the required packages using manage.sh. .. code:: sh @@ -24,85 +25,86 @@ virtualenv and install the required packages using manage.sh. How to run tests -~~~~~~~~~~~~~~~~ +================ -Tests can be run using the manage.sh script. - -Following tests and checks are available: +Tests can be run using the ``manage.sh`` script. Following tests and checks are +available: - Unit tests - - Selenium tests - - PEP8 validation - - Unit test coverage check For example unit tests are run with the command below: .. code:: sh - ./manage.sh unit_tests + ./manage.sh unit_tests -For further test options, please consult the help of the manage.sh script. +For further test options, please consult the help of the ``manage.sh`` script. How to compile styles and javascript -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +==================================== + +.. _less: http://lesscss.org/ +.. _NodeJS: https://nodejs.org How to build styles -^^^^^^^^^^^^^^^^^^^ +------------------- -Less is required to build the styles of searx. Less can be installed using either NodeJS or Apt. +Less_ is required to build the styles of searx. Less_ can be installed using +either NodeJS_ or Apt. .. code:: sh - sudo apt-get install nodejs - sudo npm install -g less - + sudo -H apt-get install nodejs + sudo -H npm install -g less OR .. code:: sh - sudo apt-get install node-less + sudo -H apt-get install node-less -After satisfying the requirements styles can be build using manage.sh +After satisfying the requirements styles can be build using ``manage.sh`` .. code:: sh - ./manage.sh styles + ./manage.sh styles How to build the source of the oscar theme -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +========================================== -Grunt must be installed in order to build the javascript sources. It depends on NodeJS, so first -Node has to be installed. +.. _grunt: https://gruntjs.com/ + +Grunt_ must be installed in order to build the javascript sources. It depends on +NodeJS, so first Node has to be installed. .. code:: sh - sudo apt-get install nodejs - sudo npm install -g grunt-cli + sudo -H apt-get install nodejs + sudo -H npm install -g grunt-cli -After installing grunt, the files can be built using the following command: +After installing grunt, the files can be built using the following command: .. code:: sh - ./manage.sh grunt_build - + ./manage.sh grunt_build Tips for debugging/development -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -1. Turn on debug logging - Whether you are working on a new engine or trying to eliminate a bug, it is always a good idea - to turn on debug logging. When debug logging is enabled a stack trace appears, - instead of the cryptic ``Internal Server Error`` message. It can be turned on by setting - ``debug: False`` to ``debug: True`` in settings.yml. - -2. Run ``./manage.sh tests`` before creating a PR. - Failing build on Travis is common because of PEP8 checks. So a new commit must be created - containing these format fixes. This phase can be skipped if ``./manage.sh tests`` is run - locally before creating a PR. +============================== + +Turn on debug logging + Whether you are working on a new engine or trying to eliminate a bug, it is + always a good idea to turn on debug logging. When debug logging is enabled a + stack trace appears, instead of the cryptic ``Internal Server Error`` + message. It can be turned on by setting ``debug: False`` to ``debug: True`` in + :origin:`settings.yml `. + +Run ``./manage.sh tests`` before creating a PR. + Failing build on Travis is common because of PEP8 checks. So a new commit + must be created containing these format fixes. This phase can be skipped if + ``./manage.sh tests`` is run locally before creating a PR. diff --git a/docs/dev/search_api.rst b/docs/dev/search_api.rst index ea3b0501a..158cab7c5 100644 --- a/docs/dev/search_api.rst +++ b/docs/dev/search_api.rst @@ -1,3 +1,4 @@ +========== Search API ========== @@ -11,194 +12,101 @@ Furthermore, two enpoints ``/`` and ``/search`` are available for querying. ``GET /search`` Parameters -~~~~~~~~~~ - -.. code:: sh - - q - -The search query. This string is passed to external search services. -Thus, searx supports syntax of each search service. For example, ``site:github.com searx`` is a valid -query for Google. However, if simply the query above is passed to any search engine which does not filter its -results based on this syntax, you might not get the results you wanted. - - -See more at :doc:`/user/search_syntax` - -Required. - -.. code:: sh - - categories - -Comma separated list, specifies the active search categories - -Optional. - -.. code:: sh - - engines - -Comma separated list, specifies the active search engines. - -Optional. - -.. code:: sh - - lang - -Code of the language. - -Optional. - -Default: ``all`` - -.. code:: sh - - pageno - -Search page number. - -Optional. - -Default: ``1`` - -.. code:: sh - - time_range - -Time range of search for engines which support it. See if an engine supports time range search in the preferences page of an instance. - -Optional. - -Possible: ``day``, ``month``, ``year`` - -.. code:: sh - - format - -Output format of results. - -Optional. - -Possible: ``json``, ``csv``, ``rss`` - -.. code:: sh - - results_on_new_tab - -Open search results on new tab. - -Optional. - -Default: ``0`` - -Possible: ``0``, ``1`` - -.. code:: sh - - image_proxy - -Proxy image results through searx. - -Optional. - -Default: ``False`` - -Possible: ``True``, ``False`` - -.. code:: sh - - autocomplete - -Service which completes words as you type. - -Optional. - -Default: empty - -Possible: ``google``, ``dbpedia``, ``duckduckgo``, ``startpage``, ``wikipedia`` - -.. code:: sh - - safesearch - -Filter search results of engines which support safe search. See if an engine supports safe search in the preferences page of an instance. - -Optional. - -Default: ``None`` - -Possible: ``0``, ``1``, ``None`` - -.. code:: sh - - theme - -Theme of instance. - -Optional. +========== -Default: ``oscar`` +``q`` : required + The search query. This string is passed to external search services. Thus, + searx supports syntax of each search service. For example, ``site:github.com + searx`` is a valid query for Google. However, if simply the query above is + passed to any search engine which does not filter its results based on this + syntax, you might not get the results you wanted. -Possible: ``oscar``, ``simple``, ``legacy``, ``pix-art``, ``courgette`` + See more at :ref:`search-syntax` -Please note, available themes depend on an instance. It is possible that an instance administrator deleted, created or renamed themes on his/her instance. See the available options in the preferences page of the instance. +``categories`` : optional + Comma separated list, specifies the active search categories -.. code:: sh +``engines``: optional + Comma separated list, specifies the active search engines. - oscar-style +``lang``: default ``all`` + Code of the language. -Style of Oscar theme. It is only parsed if the theme of an instance is ``oscar``. +``pageno``: default ``1`` + Search page number. -Optional. +``time_range``: optional + [ ``day``, ``month``, ``year`` ] -Default: ``logicodev`` + Time range of search for engines which support it. See if an engine supports + time range search in the preferences page of an instance. -Possible: ``pointhi``, ``logicodev`` +``format``: optional + [ ``json``, ``csv``, ``rss`` ] -Please note, available styles depend on an instance. It is possible that an instance administrator deleted, created or renamed styles on his/her instance. See the available options in the preferences page of the instance. + Output format of results. -.. code:: sh +``results_on_new_tab``: default ``0`` + [ ``0``, ``1`` ] - enabled_plugins + Open search results on new tab. -List of enabled plugins. +``image_proxy``: default ``False`` + [ ``True``, ``False`` ] -Optional. + Proxy image results through searx. -Default: ``HTTPS_rewrite``, ``Self_Informations``, ``Search_on_category_select``, ``Tracker_URL_remover`` +``autocomplete``: default *empty* + [ ``google``, ``dbpedia``, ``duckduckgo``, ``startpage``, ``wikipedia`` ] -Possible: ``DOAI_rewrite``, ``HTTPS_rewrite``, ``Infinite_scroll``, ``Vim-like_hotkeys``, ``Self_Informations``, ``Tracker_URL_remover``, ``Search_on_category_select`` + Service which completes words as you type. -.. code:: sh +``safesearch``: default ``None`` + [ ``0``, ``1``, ``None`` ] - disabled_plugins + Filter search results of engines which support safe search. See if an engine + supports safe search in the preferences page of an instance. -List of disabled plugins. +``theme``: default ``oscar`` + [ ``oscar``, ``simple``, ``legacy``, ``pix-art``, ``courgette`` ] -Optional. + Theme of instance. -Default: ``DOAI_rewrite``, ``Infinite_scroll``, ``Vim-like_hotkeys`` + Please note, available themes depend on an instance. It is possible that an + instance administrator deleted, created or renamed themes on his/her instance. + See the available options in the preferences page of the instance. -Possible: ``DOAI_rewrite``, ``HTTPS_rewrite``, ``Infinite_scroll``, ``Vim-like_hotkeys``, ``Self_Informations``, ``Tracker_URL_remover``, ``Search_on_category_select`` +``oscar-style``: default ``logicodev`` + [ ``pointhi``, ``logicodev`` ] -.. code:: sh + Style of Oscar theme. It is only parsed if the theme of an instance is + ``oscar``. - enabled_engines + Please note, available styles depend on an instance. It is possible that an + instance administrator deleted, created or renamed styles on his/her + instance. See the available options in the preferences page of the instance. -List of enabled engines. +``enabled_plugins``: optional + List of enabled plugins. -Optional. + :default: ``HTTPS_rewrite``, ``Self_Informations``, + ``Search_on_category_select``, ``Tracker_URL_remover`` -Possible: all engines + :values: [ ``DOAI_rewrite``, ``HTTPS_rewrite``, ``Infinite_scroll``, + ``Vim-like_hotkeys``, ``Self_Informations``, ``Tracker_URL_remover``, + ``Search_on_category_select`` ] -.. code:: sh +``disabled_plugins``: optional + List of disabled plugins. - disabled_engines + :default: ``DOAI_rewrite``, ``Infinite_scroll``, ``Vim-like_hotkeys`` + :values: ``DOAI_rewrite``, ``HTTPS_rewrite``, ``Infinite_scroll``, + ``Vim-like_hotkeys``, ``Self_Informations``, ``Tracker_URL_remover``, + ``Search_on_category_select`` -List of disabled engines. +``enabled_engines``: optional : *all* :origin:`engines ` + List of enabled engines. -Optional. +``disabled_engines``: optional : *all* :origin:`engines ` + List of disabled engines. -Possible: all engines diff --git a/docs/dev/translation.rst b/docs/dev/translation.rst index ab4dd98cb..86c4c843b 100644 --- a/docs/dev/translation.rst +++ b/docs/dev/translation.rst @@ -1,28 +1,36 @@ +.. _translation: + +=========== Translation =========== -Requirements ------------- +.. _searx@transifex: https://www.transifex.com/asciimoo/searx/ + +Translation currently takes place on `searx@transifex`_ - * Transifex account +Requirements +============ - * Installed CLI tool of Transifex +* Transifex account +* Installed CLI tool of Transifex Init Transifex project ----------------------- +====================== -After installing ``transifex`` using pip, run the following command to initialize the project. +After installing ``transifex`` using pip, run the following command to +initialize the project. -.. code:: shell +.. code:: sh - tx init # Transifex instance: https://www.transifex.com/asciimoo/searx/ + tx init # Transifex instance: https://www.transifex.com/asciimoo/searx/ -After ``$HOME/.transifexrc`` is created, get a Transifex API key and insert it into the configuration file. +After ``$HOME/.transifexrc`` is created, get a Transifex API key and insert it +into the configuration file. Create a configuration file for ``tx`` named ``$HOME/.tx/config``. -.. code:: shell +.. code:: ini [main] host = https://www.transifex.com @@ -42,20 +50,22 @@ Then run ``tx set``: Update translations -------------------- +=================== To retrieve the latest translations, pull it from Transifex. -.. code:: shell +.. code:: sh - tx pull -a + tx pull -a -Then check the new languages. If strings translated are not enough, delete those folders, because -those should not be compiled. Call the command below to compile the ``.po`` files. +Then check the new languages. If strings translated are not enough, delete those +folders, because those should not be compiled. Call the command below to compile +the ``.po`` files. .. code:: shell - pybabel compile -d searx/translations + pybabel compile -d searx/translations -After the compilation is finished commit the ``.po`` and ``.mo`` files and create a PR. +After the compilation is finished commit the ``.po`` and ``.mo`` files and +create a PR. -- cgit v1.2.3 From 04eeeb53a1215315d8027ac99978b560f45d5521 Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Thu, 12 Dec 2019 19:48:42 +0100 Subject: doc: moved reST sources in the right folder (much clearer) Signed-off-by: Markus Heiser --- docs/dev/index.rst | 13 ++ docs/dev/install/installation.rst | 341 -------------------------------------- 2 files changed, 13 insertions(+), 341 deletions(-) create mode 100644 docs/dev/index.rst delete mode 100644 docs/dev/install/installation.rst (limited to 'docs/dev') diff --git a/docs/dev/index.rst b/docs/dev/index.rst new file mode 100644 index 000000000..8e18066ca --- /dev/null +++ b/docs/dev/index.rst @@ -0,0 +1,13 @@ +======================= +Developer documentation +======================= + +.. toctree:: + :maxdepth: 1 + + quickstart + contribution_guide + engine_overview + search_api + plugins + translation diff --git a/docs/dev/install/installation.rst b/docs/dev/install/installation.rst deleted file mode 100644 index 239ce0704..000000000 --- a/docs/dev/install/installation.rst +++ /dev/null @@ -1,341 +0,0 @@ -.. _installation: - -============ -Installation -============ - -.. contents:: - :depth: 3 - -Basic installation -================== - -Step by step installation for Debian/Ubuntu with virtualenv. For Ubuntu, be sure -to have enable universe repository. - -Install packages: - -.. code:: sh - - $ sudo -H apt-get install \ - git build-essential libxslt-dev \ - python-dev python-virtualenv python-babel \ - zlib1g-dev libffi-dev libssl-dev - -Install searx: - -.. code:: sh - - cd /usr/local - sudo -H git clone https://github.com/asciimoo/searx.git - sudo -H useradd searx -d /usr/local/searx - sudo -H chown searx:searx -R /usr/local/searx - -Install dependencies in a virtualenv: - -.. code:: sh - - cd /usr/local/searx - sudo -H -u searx -i - -.. code:: sh - - (searx)$ virtualenv searx-ve - (searx)$ . ./searx-ve/bin/activate - (searx)$ ./manage.sh update_packages - -Configuration -============== - -.. code:: sh - - sed -i -e "s/ultrasecretkey/`openssl rand -hex 16`/g" searx/settings.yml - -Edit searx/settings.yml if necessary. - -Check -===== - -Start searx: - -.. code:: sh - - python searx/webapp.py - -Go to http://localhost:8888 - -If everything works fine, disable the debug option in settings.yml: - -.. code:: sh - - sed -i -e "s/debug : True/debug : False/g" searx/settings.yml - -At this point searx is not demonized ; uwsgi allows this. - -You can exit the virtualenv and the searx user bash (enter exit command -twice). - -uwsgi -===== - -Install packages: - -.. code:: sh - - sudo -H apt-get install \ - uwsgi uwsgi-plugin-python - -Create the configuration file ``/etc/uwsgi/apps-available/searx.ini`` with this -content: - -.. code:: ini - - [uwsgi] - # Who will run the code - uid = searx - gid = searx - - # disable logging for privacy - disable-logging = true - - # Number of workers (usually CPU count) - workers = 4 - - # The right granted on the created socket - chmod-socket = 666 - - # Plugin to use and interpretor config - single-interpreter = true - master = true - plugin = python - lazy-apps = true - enable-threads = true - - # Module to import - module = searx.webapp - - # Virtualenv and python path - virtualenv = /usr/local/searx/searx-ve/ - pythonpath = /usr/local/searx/ - chdir = /usr/local/searx/searx/ - -Activate the uwsgi application and restart: - -.. code:: sh - - cd /etc/uwsgi/apps-enabled - ln -s ../apps-available/searx.ini - /etc/init.d/uwsgi restart - -Web server -========== - -with nginx ----------- - -If nginx is not installed (uwsgi will not work with the package -nginx-light): - -.. code:: sh - - sudo -H apt-get install nginx - -Hosted at / -~~~~~~~~~~~ - -Create the configuration file ``/etc/nginx/sites-available/searx`` with this -content: - -.. code:: nginx - - server { - listen 80; - server_name searx.example.com; - root /usr/local/searx; - - location / { - include uwsgi_params; - uwsgi_pass unix:/run/uwsgi/app/searx/socket; - } - } - -Create a symlink to sites-enabled: - -.. code:: sh - - sudo -H ln -s /etc/nginx/sites-available/searx /etc/nginx/sites-enabled/searx - -Restart service: - -.. code:: sh - - sudo -H service nginx restart - sudo -H service uwsgi restart - -from subdirectory URL (/searx) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Add this configuration in the server config file -``/etc/nginx/sites-enabled/default``: - -.. code:: nginx - - location = /searx { rewrite ^ /searx/; } - location /searx { - try_files $uri @searx; - } - location @searx { - uwsgi_param SCRIPT_NAME /searx; - include uwsgi_params; - uwsgi_modifier1 30; - uwsgi_pass unix:/run/uwsgi/app/searx/socket; - } - - -**OR** using reverse proxy (Please, note that reverse proxy advised to be used -in case of single-user or low-traffic instances.) - -.. code:: nginx - - location /searx { - proxy_pass http://127.0.0.1:8888; - proxy_set_header Host $host; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Scheme $scheme; - proxy_set_header X-Script-Name /searx; - proxy_buffering off; - } - - -Enable ``base_url`` in ``searx/settings.yml`` - -.. code:: yaml - - base_url : http://your.domain.tld/searx/ - -Restart service: - -.. code:: sh - - sudo -H service nginx restart - sudo -H service uwsgi restart - -disable logs -^^^^^^^^^^^^ - -for better privacy you can disable nginx logs about searx. - -how to proceed: below ``uwsgi_pass`` in ``/etc/nginx/sites-available/default`` -add: - -.. code:: nginx - - access_log /dev/null; - error_log /dev/null; - -Restart service: - -.. code:: sh - - sudo -H service nginx restart - -with apache ------------ - -Add wsgi mod: - -.. code:: sh - - sudo -H apt-get install libapache2-mod-uwsgi - sudo -H a2enmod uwsgi - -Add this configuration in the file ``/etc/apache2/apache2.conf``: - -.. code:: apache - - - Options FollowSymLinks Indexes - SetHandler uwsgi-handler - uWSGISocket /run/uwsgi/app/searx/socket - - -Note that if your instance of searx is not at the root, you should change -```` by the location of your instance, like ````. - -Restart Apache: - -.. code:: sh - - sudo -H /etc/init.d/apache2 restart - -disable logs -~~~~~~~~~~~~ - -For better privacy you can disable Apache logs. - -.. warning:: - - You can only disable logs for the whole (virtual) server not for a specific - path. - -Go back to ``/etc/apache2/apache2.conf`` and above ```` add: - -.. code:: apache - - CustomLog /dev/null combined - -Restart Apache: - -.. code:: sh - - sudo -H /etc/init.d/apache2 restart - -How to update -============= - -.. code:: sh - - cd /usr/local/searx - sudo -H -u searx -i - -.. code:: sh - - (searx)$ . ./searx-ve/bin/activate - (searx)$ git stash - (searx)$ git pull origin master - (searx)$ git stash apply - (searx)$ ./manage.sh update_packages - -.. code:: sh - - sudo -H service uwsgi restart - -Docker -====== - -Make sure you have installed Docker. For instance, you can deploy searx like this: - -.. code:: sh - - docker pull wonderfall/searx - docker run -d --name searx -p $PORT:8888 wonderfall/searx - -Go to ``http://localhost:$PORT``. - -See https://hub.docker.com/r/wonderfall/searx/ for more informations. It's also -possible to build searx from the embedded Dockerfile. - -.. code:: sh - - git clone https://github.com/asciimoo/searx.git - cd searx - docker build -t whatever/searx . - -References -========== - -* https://about.okhin.fr/posts/Searx/ with some additions - -* How to: `Setup searx in a couple of hours with a free SSL certificate - `__ - -- cgit v1.2.3 From 97b86fd184e81263b818dfd2168385c2e60f327a Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Fri, 13 Dec 2019 14:57:07 +0100 Subject: doc: document the new reST & gh-pages workflow Signed-off-by: Markus Heiser --- docs/dev/contribution_guide.rst | 59 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 56 insertions(+), 3 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/contribution_guide.rst b/docs/dev/contribution_guide.rst index 9414de2c9..22d690033 100644 --- a/docs/dev/contribution_guide.rst +++ b/docs/dev/contribution_guide.rst @@ -82,8 +82,61 @@ Translation currently takes place on :ref:`transifex `. Documentation ============= -.. admonition:: ToDo +.. _Sphinx: http://www.sphinx-doc.org +.. _reST: http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html - docs are no longer in gh-branch +.. sidebar:: The reST sources - Update this chapter and document the /docs workflow!! + has been moved from ``gh-branch`` into ``master`` (:origin:`docs`). + +The documentation is built using Sphinx_. So in order to be able to generate +the required files, you have to install it on your system. Much easier, use +Makefile our targets. + +Here is an example which makes complete rebuild: + +.. code:: sh + + $ make docs-clean docs + ... + The HTML pages are in dist/docs. + + +live build +---------- + +Live build is like WYSIWYG, If you want to edit the documentation, its +recommended to use. The Makefile target ``docs-live`` builds the docs, opens URL +in your favorite browser and rebuilds every time a reST file has been changed. + +.. code:: sh + + $ make docs-clean docs-clean + ... + The HTML pages are in dist/docs. + ... Serving on http://0.0.0.0:8080 + ... Start watching changes + + + +deploy on github.io +------------------- + +To deploy documentation at :docs:`github.io <.>` use Makefile target +``gh-pages``, which will builds the documentation, clones searx into a sub +folder ``gh-pages``, cleans it, copies the doc build into and runs all the +needed git add, commit and push: + +.. code:: sh + + $ make gh-pages + ... + SPHINX docs --> file://<...>/dist/docs + The HTML pages are in dist/docs. + ... + Cloning into 'gh-pages' ... + ... + cd gh-pages; git checkout gh-pages >/dev/null + Switched to a new branch 'gh-pages' + ... + doc available at --> https://asciimoo.github.io/searx -- cgit v1.2.3 From fcd7f4cd1205db48c8796a1ba12bcb1d358a554e Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Fri, 13 Dec 2019 15:22:10 +0100 Subject: doc: add note about docs-clean target BTW: fix some typos Signed-off-by: Markus Heiser --- docs/dev/contribution_guide.rst | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/contribution_guide.rst b/docs/dev/contribution_guide.rst index 22d690033..064f28e63 100644 --- a/docs/dev/contribution_guide.rst +++ b/docs/dev/contribution_guide.rst @@ -93,7 +93,7 @@ The documentation is built using Sphinx_. So in order to be able to generate the required files, you have to install it on your system. Much easier, use Makefile our targets. -Here is an example which makes complete rebuild: +Here is an example which makes a complete rebuild: .. code:: sh @@ -105,13 +105,18 @@ Here is an example which makes complete rebuild: live build ---------- +.. sidebar:: docs-clean + + It is recommended to assert a complete rebuild before deploying (use + ``docs-clean``). + Live build is like WYSIWYG, If you want to edit the documentation, its recommended to use. The Makefile target ``docs-live`` builds the docs, opens URL in your favorite browser and rebuilds every time a reST file has been changed. .. code:: sh - $ make docs-clean docs-clean + $ make docs-live ... The HTML pages are in dist/docs. ... Serving on http://0.0.0.0:8080 @@ -129,7 +134,7 @@ needed git add, commit and push: .. code:: sh - $ make gh-pages + $ make docs-clean gh-pages ... SPHINX docs --> file://<...>/dist/docs The HTML pages are in dist/docs. -- cgit v1.2.3