summaryrefslogtreecommitdiff
path: root/searx/plugins
AgeCommit message (Collapse)Author
2023-06-01[fix] limiter: replace real_ip by IPv4/v6 networkMarkus Heiser
Closes: https://github.com/searxng/searxng/issues/2477 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-01[fix] correct determination of the IP for the requestMarkus Heiser
For correct determination of the IP to the request the function botdetection.get_real_ip() is implemented. This fonction is used in the ip_limit and link_token method of the botdetection and it is used in the self_info plugin. A documentation about the X-Forwarded-For header has been added. [1] https://github.com/searxng/searxng/pull/2357#issuecomment-1566211059 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-01[mod] botdetection - improve ip_limit and link_token methodsMarkus Heiser
- counting requests in LONG_WINDOW and BURST_WINDOW is not needed when the request is validated by the link_token method [1] - renew a ping-key on validation [2], this is needed for infinite scrolling, where no new token (CSS) is loaded. / this does not fix the BURST_MAX issue in the vanilla limiter - normalize the counter names of the ip_limit method to 'ip_limit.*' - just integrate the ip_limit method straight forward in the limiter plugin / non intermediate code --> ip_limit now returns None or a werkzeug.Response object that can be passed by the plugin to the flask application / non intermediate code that returns a tuple [1] https://github.com/searxng/searxng/pull/2357#issuecomment-1566113277 [2] https://github.com/searxng/searxng/pull/2357#discussion_r1208542206 [3] https://github.com/searxng/searxng/pull/2357#issuecomment-1566125979 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-01[mod] limiter: add config file /etc/searxng/limiter.tomlMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-29[mod] limiter -> botdetection: modularization and documentationMarkus Heiser
In order to be able to meet the outstanding requirements, the implementation is modularized and supplemented with documentation. This patch does not contain functional change, except it fixes issue #2455 ---- Aktivate limiter in the settings.yml and simulate a bot request by:: curl -H 'Accept-Language: de-DE,en-US;q=0.7,en;q=0.3' \ -H 'Accept: text/html' -H 'User-Agent: xyz' \ -H 'Accept-Encoding: gzip' \ 'http://127.0.0.1:8888/search?q=foo' In the LOG: DEBUG searx.botdetection.link_token : missing ping for this request: ..... Since ``BURST_MAX_SUSPICIOUS = 2`` you can repeat the query above two time before you get a "Too Many Requests" response. Closes: https://github.com/searxng/searxng/issues/2455 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-29[mod] limiter: add random token to the limiter URLMarkus Heiser
By adding a random component in the limiter URL a bot can no longer send a ping by request a static URL. Related: https://github.com/searxng/searxng/pull/2357#issuecomment-1518525094 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-29[mod] limiter: reduce request rates for requests without a pingMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-30[mod] limiter: block requests from PetalBotMarkus Heiser
Block requests from PetalBlock. Normally robots.txt is enough to stop PetalBlock from making requests [1]. However, if SearXNG is offered below a path (example.org/search), then the robots.txt is not available in the root paths of the domain / subdomain. [1] https://webmaster.petalsearch.com/site/petalbot Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-13[mod] limiter: block unmaintained Farside instancesMarkus Heiser
Since [bb3a01f8] has been merged to the Farside project, Farside instances do no longer need to send requests to SearXNG instances [1]. There are some old unmaintained Farside instances on the web that continue to query SearXNG instances --> we can safely block their requests. [1] https://github.com/benbusby/farside/issues/95 [bb3a01f8] https://github.com/benbusby/farside/commit/bb3a01f8 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-03[fix] limiter: never block a /healthz requestMarkus Heiser
Related: https://github.com/searxng/searxng/issues/2310#issuecomment-1494417531 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-01[mod] limiter: minor improvementsMarkus Heiser
- requests without HTTP header 'Connection' or missing 'User-Agent' will be blocked by the limiter - re_bot is related to 'User-Agent' and has been renamed to block_user_agent Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-01[mod] plugin limiter: improve the log messagesMarkus Heiser
In debug mode more detailed logging is needed to evaluate if an access should have been blocked by the limiter. BTW: remove duplicate code checking bot signature ``re_bot.match(user_agent)`` Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-17Add "Auto-detected" as a language.Alexandre Flament
When the user choose "Auto-detected", the choice remains on the following queries. The detected language is displayed. For example "Auto-detected (en)": * the next query language is going to be auto detected * for the current query, the detected language is English. This replace the autodetect_search_language plugin.
2023-02-17[fix] typo in searx/plugins/tor_check.pyMarkus Heiser
Related: https://github.com/searxng/searxng/pull/2189 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-14[fix] spelling: Tor, SearXNGAllan Nordhøy
2022-12-16Merge branch 'master' into fasttextArtikusHG
2022-12-16Replace langdetect with fasttextArtikusHG
2022-12-14Merge pull request #1954 from dalf/fix.redis.init.2Alexandre Flament
[fix] follow up of PR-1856
2022-12-11[mod] improve 'Autodetect search language' pluginMarkus Heiser
- Add documentation to the plugin - Harmonize FastText language model with SearXNG's language model Reosurces:: import fasttext # --> +10 MB fasttext.load_model(str(data_dir / 'lid.176.ftz')) # --> +4MB Suggested-by: @dalf - To speed up and simplify the deployment use fasttext-wheel instead of fasttext - Building numpy on the Alpine Linux of docker-images takes ages --> install py3-numpy from Alpines package manager (apk) - Alpine Linux on docker-images (musl libc) do not support fasttext-wheel (gnu libc) --> patch Dockerfile and build from fastetxt: sed -i s/fasttext-wheel/fasttext/ requirements.txt Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-12-10[mod] new plugin: Autodetect search languageArtikusHG
2022-12-10move searx.shared.redisdb to searx.redisdbAlexandre Flament
2022-11-05The checker requires RedisAlexandre Flament
Remove the abstraction in searx.shared.SharedDict. Implement a basic and dedicated scheduler for the checker using a Redis script.
2022-09-27[fix] typos / reported by @kianmeng in searx PR-3366Markus Heiser
[PR-3366] https://github.com/searx/searx/pull/3366 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-23oa_doi_rewrite: add the doi to the result when it is found.Alexandre FLAMENT
Currentty, when oa_doi_rewrite find a DOI in the result URL, it replace the URL. In this commit, the plugin adds the key "doi" to the result, so the paper.html can show it.
2022-08-25[mod] limiter plugin: Accept-Encoding handlingLéon Tiekötter
Only raise "suspicious Accept-Encoding" when both "gzip" and "deflate" are missing from Accept-Encoding. Prevent Browsers which only implement one compression solution from being blocked by the limiter plugin. Example Browser which is currently blocked: Lynx Browser (https://lynx.invisible-island.net)
2022-07-19[mod] add tor_check plugin - convenient tor checking trough searxngSolirs
2022-07-05fix: return body on limiter block so there is not just a blank pagemrpaulblack
2022-06-06Merge pull request #1047 from return42/redis-libAlexandre Flament
Add a redis library to generalize DB functions we need in SearXNG.
2022-06-03[fix] prepare for pylint 2.14.0Markus Heiser
Remove issue reported by Pylint 2.14.0: - no-self-use: has been moved to optional extension [1] - The refactoring checker now also raises 'consider-using-generator' messages for max(), min() and sum(). [2] .pylintrc: - <option name>-hint has been removed since long, Pylint 2.14.0 raises an error on invalid options - bad-continuation and bad-whitespace have been removed [3] [1] https://pylint.pycqa.org/en/latest/whatsnew/2/2.14/summary.html#removed-checkers [2] https://pylint.pycqa.org/en/latest/whatsnew/2/2.14/full.html#what-s-new-in-pylint-2-14-0 [2] https://pylint.pycqa.org/en/latest/whatsnew/2/2.6/summary.html#summary-release-highlights Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-05-30[mod] plugins/limiter.py - use searx.redislib.incr_sliding_windowMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-05-10Fix typoÉmilien Devos
2022-05-07theme: remove __common__Alexandre Flament
2022-03-25[doc] add some documentation about the limiter plugin (and redis)Markus Heiser
Requested-by: https://github.com/searxng/searxng/discussions/993#discussioncomment-2396914 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-20[mod] infinite_scroll as preferenceAlexandre Flament
* oscar theme: code from searx/plugins/infinite_scroll.py * simple theme: new implementation Co-authored-by: Markus Heiser <markus.heiser@darmarIT.de>
2022-02-18Merge pull request #899 from dalf/limiter_updateAlexandre Flament
[limiter] update
2022-02-18[mod] templates: rename field for <iframe> URL to iframe_srcMarkus Heiser
Rename result field data_src to iframe_src Suggested-by: @dalf https://github.com/searxng/searxng/pull/882#issuecomment-1037997402 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-18[fix] hostname_replace.py: don't stop replace URL in fieldsMarkus Heiser
This is a rewrite of the hostname_replace.py that: - don't stop to replace URL in fields ('data_src', 'audio_src') if there isn't a 'parsed_url', - adds a comment about keep or remove a result from the result list - adds a loop over ['data_src', 'audio_src'] instead of doubling code lines Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-17[limiter] updateAlexandre Flament
2022-02-13[mod] result_templates/default.html replace embedded HTML by data_src audio_srcMarkus Heiser
Embedded HTML breaks SearXNG architecture. To modularize, HTML is generated in the templates (oscar & simple) and result parameter 'embedded' is replaced by 'data_src' (and 'audio_src'), an URL for embedded content (<iframe>). Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-13[mod] hostname_replace: replace hostnames in result's data_src paramMarkus Heiser
To test you need to redirect embeded videos (e.g.) from youtube to a invidios instance. Search for videos using engine `!youtube lebowski`. The result URLs and the embeded videos should link to the invidios instance. Here is an example of such a `hostname_replace` configuration:: hostname_replace: # youtube --> Invidious '(.*\.)?youtube-nocookie\.com': 'invidio.xamh.de' '(.*\.)?youtube\.com$': 'invidio.xamh.de' '(.*\.)?invidious\.snopyta\.org$': 'invidio.xamh.de' '(.*\.)?vid\.puffyan\.us': 'invidio.xamh.de' '(.*\.)?invidious\.kavin\.rocks$': 'invidio.xamh.de' '(.*\.)?inv\.riverside\.rocks$': 'invidio.xamh.de' Closes: https://github.com/searxng/searxng/issues/873 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-12plugin limiter: check only /image_proxy and /searchAlexandre Flament
also adjust the number of req/time
2022-02-02[enh] limiter pluginAlexandre Flament
can replace filtron: * rate limite the number of request per IP and per (IP, User-Agent) * block some bots use Redis data stored in Redis never contains the IP addresses, only HMAC using the secret_key Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-17[typing] add optional attrs to PluginMartin Fischer
2022-01-06[refactor] add type hints & remove Setting._post_initMartin Fischer
Previously the Setting classes used a horrible _post_init hack that prevented proper type checking.
2021-12-27[format.python] initial formatting of the python codeMarkus Heiser
This patch was generated by black [1]:: make format.python [1] https://github.com/psf/black Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-27[format.python] disable py code formatting for some hunks of codeMarkus Heiser
Disable the python code formatting from python-black, where the readability of code suffers by formatting. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-07[mod] searx.plugins.prepare_package_resources() - use generatorsMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-07[fix] searx.plugins.initialize() - don't miss module & module-nameMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-06plugins: refactor initializationAlexandre Flament
add a new function "init" call when the app starts. The function can: * return False to disable the plugin. * modify the Flask app.
2021-09-11[enh] add hostname_replace pluginAlexandre Flament
* backport of https://github.com/searx/searx/pull/2724 * allow to remove result if the replacement is the boolean value false
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-06 14:31:13 +0000