summaryrefslogtreecommitdiff
path: root/searx/plugins
AgeCommit message (Collapse)Author
2024-05-28[fix] unit converter operating backwards (from_si <-> to_si)Markus Heiser
The factors for from_si and to_si were reversed. Closes: https://github.com/searxng/searxng/issues/3497 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-05-09[fix] unit converter plugin: can't be disabled in settingsBnyro
2024-05-09[feat] plugins: new calculator pluginBnyro
2024-05-09[mod] improve unit converter pluginMarkus Heiser
- l10n support: parse and format decimal numbers by babel - ability to add additional units - improved unit detection (symbols are not unique) - support for alias units (0,010C to F --> 32,018 °F) Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-04-27[feat] plugins: new unit converter pluginBnyro
2024-03-11[mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTIONMarkus Heiser
In the past, some files were tested with the standard profile, others with a profile in which most of the messages were switched off ... some files were not checked at all. - ``PYLINT_SEARXNG_DISABLE_OPTION`` has been abolished - the distinction ``# lint: pylint`` is no longer necessary - the pylint tasks have been reduced from three to two 1. ./searx/engines -> lint engines with additional builtins 2. ./searx ./searxng_extra ./tests -> lint all other python files Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-03-09[fix] issues reported by pylint 3.1.0Markus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-03-09[black] upgrade black 22.12.0 --> 24.2.0Markus Heiser
The issue discussed in [1] has been solved since [2] has been merged into black / now we can upgrade without touching 69 files as it was needed with black 23.1.0 [3]. [1] https://github.com/searxng/searxng/pull/2159#issuecomment-1425723977 [2] https://github.com/psf/black/pull/4060 [3] https://github.com/searxng/searxng/pull/2159/files Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-11-01[mod] isolation of botdetection from the limiterMarkus Heiser
This patch was inspired by the discussion around PR-2882 [2]. The goals of this patch are: 1. Convert plugin searx.plugin.limiter to normal code [1] 2. isolation of botdetection from the limiter [2] 3. searx/{tools => botdetection}/config.py and drop searx.tools 4. in URL /config, 'limiter.enabled' is true only if the limiter is really enabled (Redis is available). This patch moves all the code that belongs to botdetection into namespace searx.botdetection and code that belongs to limiter is placed in namespace searx.limiter. Tthe limiter used to be a plugin at some point botdetection was added, it was not a plugin. The modularization of these two components was long overdue. With the clear modularization, the documentation could then also be organized according to the architecture. [1] https://github.com/searxng/searxng/pull/2882 [2] https://github.com/searxng/searxng/pull/2882#issuecomment-1741716891 To test: - check the app works without the limiter, check `/config` - check the app works with the limiter and with the token, check `/config` - make docs.live .. and read - http://0.0.0.0:8000/admin/searx.limiter.html - http://0.0.0.0:8000/src/searx.botdetection.html#botdetection Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-25add new parameter called server.public_instanceEmilien Devos
for enabling by default advanced limiter functions in the future allow us to add features just for the public instances
2023-09-19[mod] preferences: implement drop-down menu for hotkeys (default, vim)Markus Heiser
Replace the on/off checkbox of the vim-hotkeys in the preferences by a drop-down menu. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-18[feat] search on category select without JSBnyro
Co-authored-by: Alexandre Flament <alex@al-f.net>
2023-09-18[fix] spellingjazzzooo
2023-06-01[fix] limiter: replace real_ip by IPv4/v6 networkMarkus Heiser
Closes: https://github.com/searxng/searxng/issues/2477 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-01[fix] correct determination of the IP for the requestMarkus Heiser
For correct determination of the IP to the request the function botdetection.get_real_ip() is implemented. This fonction is used in the ip_limit and link_token method of the botdetection and it is used in the self_info plugin. A documentation about the X-Forwarded-For header has been added. [1] https://github.com/searxng/searxng/pull/2357#issuecomment-1566211059 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-01[mod] botdetection - improve ip_limit and link_token methodsMarkus Heiser
- counting requests in LONG_WINDOW and BURST_WINDOW is not needed when the request is validated by the link_token method [1] - renew a ping-key on validation [2], this is needed for infinite scrolling, where no new token (CSS) is loaded. / this does not fix the BURST_MAX issue in the vanilla limiter - normalize the counter names of the ip_limit method to 'ip_limit.*' - just integrate the ip_limit method straight forward in the limiter plugin / non intermediate code --> ip_limit now returns None or a werkzeug.Response object that can be passed by the plugin to the flask application / non intermediate code that returns a tuple [1] https://github.com/searxng/searxng/pull/2357#issuecomment-1566113277 [2] https://github.com/searxng/searxng/pull/2357#discussion_r1208542206 [3] https://github.com/searxng/searxng/pull/2357#issuecomment-1566125979 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-01[mod] limiter: add config file /etc/searxng/limiter.tomlMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-29[mod] limiter -> botdetection: modularization and documentationMarkus Heiser
In order to be able to meet the outstanding requirements, the implementation is modularized and supplemented with documentation. This patch does not contain functional change, except it fixes issue #2455 ---- Aktivate limiter in the settings.yml and simulate a bot request by:: curl -H 'Accept-Language: de-DE,en-US;q=0.7,en;q=0.3' \ -H 'Accept: text/html' -H 'User-Agent: xyz' \ -H 'Accept-Encoding: gzip' \ 'http://127.0.0.1:8888/search?q=foo' In the LOG: DEBUG searx.botdetection.link_token : missing ping for this request: ..... Since ``BURST_MAX_SUSPICIOUS = 2`` you can repeat the query above two time before you get a "Too Many Requests" response. Closes: https://github.com/searxng/searxng/issues/2455 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-29[mod] limiter: add random token to the limiter URLMarkus Heiser
By adding a random component in the limiter URL a bot can no longer send a ping by request a static URL. Related: https://github.com/searxng/searxng/pull/2357#issuecomment-1518525094 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-29[mod] limiter: reduce request rates for requests without a pingMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-30[mod] limiter: block requests from PetalBotMarkus Heiser
Block requests from PetalBlock. Normally robots.txt is enough to stop PetalBlock from making requests [1]. However, if SearXNG is offered below a path (example.org/search), then the robots.txt is not available in the root paths of the domain / subdomain. [1] https://webmaster.petalsearch.com/site/petalbot Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-13[mod] limiter: block unmaintained Farside instancesMarkus Heiser
Since [bb3a01f8] has been merged to the Farside project, Farside instances do no longer need to send requests to SearXNG instances [1]. There are some old unmaintained Farside instances on the web that continue to query SearXNG instances --> we can safely block their requests. [1] https://github.com/benbusby/farside/issues/95 [bb3a01f8] https://github.com/benbusby/farside/commit/bb3a01f8 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-03[fix] limiter: never block a /healthz requestMarkus Heiser
Related: https://github.com/searxng/searxng/issues/2310#issuecomment-1494417531 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-01[mod] limiter: minor improvementsMarkus Heiser
- requests without HTTP header 'Connection' or missing 'User-Agent' will be blocked by the limiter - re_bot is related to 'User-Agent' and has been renamed to block_user_agent Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-01[mod] plugin limiter: improve the log messagesMarkus Heiser
In debug mode more detailed logging is needed to evaluate if an access should have been blocked by the limiter. BTW: remove duplicate code checking bot signature ``re_bot.match(user_agent)`` Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-17Add "Auto-detected" as a language.Alexandre Flament
When the user choose "Auto-detected", the choice remains on the following queries. The detected language is displayed. For example "Auto-detected (en)": * the next query language is going to be auto detected * for the current query, the detected language is English. This replace the autodetect_search_language plugin.
2023-02-17[fix] typo in searx/plugins/tor_check.pyMarkus Heiser
Related: https://github.com/searxng/searxng/pull/2189 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-14[fix] spelling: Tor, SearXNGAllan Nordhøy
2022-12-16Merge branch 'master' into fasttextArtikusHG
2022-12-16Replace langdetect with fasttextArtikusHG
2022-12-14Merge pull request #1954 from dalf/fix.redis.init.2Alexandre Flament
[fix] follow up of PR-1856
2022-12-11[mod] improve 'Autodetect search language' pluginMarkus Heiser
- Add documentation to the plugin - Harmonize FastText language model with SearXNG's language model Reosurces:: import fasttext # --> +10 MB fasttext.load_model(str(data_dir / 'lid.176.ftz')) # --> +4MB Suggested-by: @dalf - To speed up and simplify the deployment use fasttext-wheel instead of fasttext - Building numpy on the Alpine Linux of docker-images takes ages --> install py3-numpy from Alpines package manager (apk) - Alpine Linux on docker-images (musl libc) do not support fasttext-wheel (gnu libc) --> patch Dockerfile and build from fastetxt: sed -i s/fasttext-wheel/fasttext/ requirements.txt Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-12-10[mod] new plugin: Autodetect search languageArtikusHG
2022-12-10move searx.shared.redisdb to searx.redisdbAlexandre Flament
2022-11-05The checker requires RedisAlexandre Flament
Remove the abstraction in searx.shared.SharedDict. Implement a basic and dedicated scheduler for the checker using a Redis script.
2022-09-27[fix] typos / reported by @kianmeng in searx PR-3366Markus Heiser
[PR-3366] https://github.com/searx/searx/pull/3366 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-23oa_doi_rewrite: add the doi to the result when it is found.Alexandre FLAMENT
Currentty, when oa_doi_rewrite find a DOI in the result URL, it replace the URL. In this commit, the plugin adds the key "doi" to the result, so the paper.html can show it.
2022-08-25[mod] limiter plugin: Accept-Encoding handlingLéon Tiekötter
Only raise "suspicious Accept-Encoding" when both "gzip" and "deflate" are missing from Accept-Encoding. Prevent Browsers which only implement one compression solution from being blocked by the limiter plugin. Example Browser which is currently blocked: Lynx Browser (https://lynx.invisible-island.net)
2022-07-19[mod] add tor_check plugin - convenient tor checking trough searxngSolirs
2022-07-05fix: return body on limiter block so there is not just a blank pagemrpaulblack
2022-06-06Merge pull request #1047 from return42/redis-libAlexandre Flament
Add a redis library to generalize DB functions we need in SearXNG.
2022-06-03[fix] prepare for pylint 2.14.0Markus Heiser
Remove issue reported by Pylint 2.14.0: - no-self-use: has been moved to optional extension [1] - The refactoring checker now also raises 'consider-using-generator' messages for max(), min() and sum(). [2] .pylintrc: - <option name>-hint has been removed since long, Pylint 2.14.0 raises an error on invalid options - bad-continuation and bad-whitespace have been removed [3] [1] https://pylint.pycqa.org/en/latest/whatsnew/2/2.14/summary.html#removed-checkers [2] https://pylint.pycqa.org/en/latest/whatsnew/2/2.14/full.html#what-s-new-in-pylint-2-14-0 [2] https://pylint.pycqa.org/en/latest/whatsnew/2/2.6/summary.html#summary-release-highlights Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-05-30[mod] plugins/limiter.py - use searx.redislib.incr_sliding_windowMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-05-10Fix typoÉmilien Devos
2022-05-07theme: remove __common__Alexandre Flament
2022-03-25[doc] add some documentation about the limiter plugin (and redis)Markus Heiser
Requested-by: https://github.com/searxng/searxng/discussions/993#discussioncomment-2396914 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-20[mod] infinite_scroll as preferenceAlexandre Flament
* oscar theme: code from searx/plugins/infinite_scroll.py * simple theme: new implementation Co-authored-by: Markus Heiser <markus.heiser@darmarIT.de>
2022-02-18Merge pull request #899 from dalf/limiter_updateAlexandre Flament
[limiter] update
2022-02-18[mod] templates: rename field for <iframe> URL to iframe_srcMarkus Heiser
Rename result field data_src to iframe_src Suggested-by: @dalf https://github.com/searxng/searxng/pull/882#issuecomment-1037997402 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-18[fix] hostname_replace.py: don't stop replace URL in fieldsMarkus Heiser
This is a rewrite of the hostname_replace.py that: - don't stop to replace URL in fields ('data_src', 'audio_src') if there isn't a 'parsed_url', - adds a comment about keep or remove a result from the result list - adds a loop over ['data_src', 'audio_src'] instead of doubling code lines Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-04 16:17:36 +0000