| Age | Commit message (Collapse) | Author |
|---|
|
see searx.search.processors.abstract.EngineProcessor
First the method searx call the get_params method.
If the return value is not None, then the searx call the method search.
|
|
check HTTP response:
* detect some comme CAPTCHA challenge (no solving). In this case the engine is suspended for long a time.
* otherwise raise HTTPError as before
the check is done in poolrequests.py (was before in search.py).
update qwant, wikipedia, wikidata to use raise_for_httperror instead of raise_for_status
|
|
Closes #302
|
|
add an new API /stats/errors
|
|
This bug exists since the PR https://github.com/searx/searx/pull/751
|
|
credits go to @bauruine see https://github.com/searx/searx/pull/1958
|
|
Xpath engine and results template changed to account for the fact that
archive.org doesn't cache .onions, though some onion engines migth have
their own cache.
Disabled by default. Can be enabled by setting the SOCKS proxies to
wherever Tor is listening and setting using_tor_proxy as True.
Requires Tor and updating packages.
To avoid manually adding the timeout on each engine, you can set
extra_proxy_timeout to account for Tor's (or whatever proxy used) extra
time.
|
|
Instead of loading the data/*.json in different location,
load these files in the new searx.data module.
|
|
|
|
and some other exceptions:
* KeyboardInterrupt
* SystemExit
* RuntimeError
* SystemError
* ImportError: an engine with an unmet dependency will stop everything.
|
|
This reverts commit 4fb3ed2c6335b68f6b28ebc68d5d22f2fd621648.
|
|
|
|
A new option is added to engines to hide error messages from users. It
is called `display_error_messages` and by default it is set to `True`.
If it is set to `False` error messages do not show up on the UI.
Keep in mind that engines are still suspended if needed regardless of
this setting.
Closes #1828
|
|
This PR adds a new setting to engines named `tokens`.
It expects a list of tokens which lets searx validate
if the request should be accepted or not.
|
|
|
|
|
|
Minor fix: "%s engine initialized" display the right engine name
|
|
|
|
Add match_language function in utils to match any user given
language code with a list of engine's supported languages.
Also add language_aliases dict on each engine to translate
standard language codes into the custom codes used by the engine.
|
|
|
|
Related to discussion in #1124
The io.open import is necessary for python2
|
|
SearX currently doesn't start up when run with Python 3 as it tries to parse the
settings.yml file with ASCII codecs.
There are similar problems with engines_languages.json and currencies.json
Python 3 requires that files with Unicode characters be read with a 'b' flag.
This also works with Python 2 and hence can be integrated into the main source
code.
Tested with the latest Python 3.6.4rc1 on Debian unstable.
Signed-off-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
|
|
|
|
This modification allows us to deactivate engines in settings.yml
without commenting them out
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
and refactor method to make it testable without making requests
|
|
utils/fetch_languages.py gets languages supported by each engine and
generates engines_languages.json with each engine's supported language.
|
|
|
|
|
|
|
|
The timeouts in settings.yml is about the total time (not only the HTTP request but also the prepare the request and parsing the response)
It was more or less the case before since the threaded_requests function ignores the thread after the timeout even the HTTP request is ended.
New / changed stats :
* page_load_time : record the HTTP request time
* page_load_count: the number of HTTP request
* engine_time : the execution total time of an engine
* engine_time_count : the number of "engine_time" measure
The avg response times in the preferences are the engine response time (engine_load_time / engine_load_count)
To sum up :
* Search.search() filters the engines that can't process the request
* Search.search() call search_multiple_requests function
* search_multiple_requests creates one thread per engine, each thread runs the search_one_request function
* search_one_request calls the request function, make the HTTP request, calls the response function, extends the result_container
* search_multiple_requests waits for the the thread to finish (or timeout)
|
|
|
|
support time range search
|
|
|
|
|
|
WARNING: dependency changes
|
|
The duration is based on the number of continuous errors, but maximized in one minute
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Lots of Flake8 corrections
Maybe we should change the rule to allow lines of 120 chars. It seems more usable.
Big twitter correction : now it outputs the words in right order...
|
