| Age | Commit message (Collapse) | Author |
|---|
|
including version (#5414)
|
|
The User-Agent header recently become required to fetch blacklist from URL
https://ahmia.fi/blacklist/
[1] https://github.com/searxng/searxng/actions/runs/18892940199/job/53924400294
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
This is an old and grumpy hack / SearXNG is a Flask application with
client/server structure, which can't be turned into a command line tool the way
it was done here.
Maintaining this hack is becoming increasingly complex the more we try to
remodel the core code, which is why we should now remove the hack from SearXNG.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
In 25.1.0 [2] an old bug has been fixed: "Docstring formatting does not apply to
module docstrings" [3].
[1] https://github.com/psf/black/blob/main/CHANGES.md#2590
[2] https://github.com/psf/black/blob/main/CHANGES.md#2510
[3] https://github.com/psf/black/issues/4094
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Continuation of #5147 .. typification of the engine processors.
BTW:
- removed obsolete engine property https_support
- fixed & improved currency_convert
- engine instances can now implement a engine.setup method
[#5147] https://github.com/searxng/searxng/pull/5147
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
- pyright configuration [1]_
- stub files: types-lxml [2]_
- addition of various type hints
- enable use of new type system features on older Python versions [3]_
- ``.tool-versions`` - set python to lowest version we support (3.10.18) [4]_:
Older versions typically lack some typing features found in newer Python
versions. Therefore, for local type checking (before commit), it is necessary
to use the older Python interpreter.
.. [1] https://docs.basedpyright.com/v1.20.0/configuration/config-files/
.. [2] https://pypi.org/project/types-lxml/
.. [3] https://typing-extensions.readthedocs.io/en/latest/#
.. [4] https://mise.jdx.dev/configuration.html#tool-versions
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Format: reST
|
|
Compared to ``lightbulb`` theme we used in the past for the dark theme, the
``monokai`` has a better contrast [1].
BTW, the result list of SearXNG should not act as a code-checker: The border &
color of class ``.err`` (used for syntax errors) has been removed / code
snippets are often not well formed nor valid code and the rendering of such code
fragments as errors obscures the view on the code snippet.
[1] https://pygments.org/styles/
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
SPDX short-form identifiers to communicate license information in a simple,
efficient, portable and machine-readable manner [1]
[1] https://spdx.dev/learn/handling-license-info/
|
|
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Using plain `httpx` directly doesn't use SearXNG's additional network config, including proxies, http2 config, ...
Related issues:
- https://github.com/searxng/searxng/issues/5027
|
|
On demand, the tracker data is loaded directly into the cache, so that the
maintenance of this data via PRs is no longer necessary.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
|
|
|
|
To reduce the memory footprint, this patch no longer loads the JSON data
completely into memory. Instead, there is an SQL database based on
`ExpireCacheSQLite`.
The class CurrenciesDB is a simple DB application that encapsulates the
DB (queries and initialization) and provides convenient methods like
`name_to_iso4217` and `iso4217_to_name`.
Related:
- https://github.com/searxng/searxng/discussions/1892
- https://github.com/searxng/searxng/pull/3458#issuecomment-2900807671
- https://github.com/searxng/searxng/pull/4650
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
The entries in the catalog of search languages are build up from the "Engine
Traits" [1] and which entries are included in the catalog is controlled qby two
threshold values [2].
If possible, the values should ensure that no languages or regions disappear
from the catalog of search languages [3].
The threshold values should have been adjusted in commit 30a8204:
- ``min_eng_per_region = 18``
- ``min_eng_per_lang = 22``
Because the threshold values were not adjusted, many entries were missing in the
search language catalog. This bug has been fixed with this patch: the threshold
values have been adjusted and the catalog of search languages has been completed
again.
[1] https://docs.searxng.org/dev/engines/enginelib.html#module-searx.enginelib.traits
[2] https://github.com/searxng/searxng/blob/96a6e3dcb2283fa7ad9db4172a00582073a166d7/searxng_extra/update/update_engine_traits.py#L104-L105
[3] https://github.com/searxng/searxng/blob/master/searx/sxng_locales.py
Closes: https://github.com/searxng/searxng/issues/4519
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
The big queries for initializing and updating the currencies take longer than
the default of the wikidata engine, which is only 3sec.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Vite is a build tool, it consists of two major parts [1]:
- A dev server that provides rich feature enhancements over native ES modules,
for example extremely fast Hot Module Replacement (HMR).
- A build command that bundles your code with Rollup, pre-configured to output
highly optimized static assets for production.
[1] https://vite.dev/guide/
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Github action Update data - update_currencies [1]:
./manage pyenv.cmd python ./searxng_extra/update/update_currencies.py
fails with ``httpcore.ReadTimeout`` / the default timeout is 3sec.
[1] https://github.com/searxng/searxng/actions/runs/12076864366/job/33703464399
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
|
|
Improve region and language detection / all locale
Testing has shown the following behaviour for the different
default and empty values of Mojeeks parameters:
| param | idx | value | behaviour |
| -------- | --- | ------ | ------------------------- |
| region | 0 | '' | detect region based on IP |
| region | 1 | 'none' | all regions |
| language | 0 | '' | all languages |
|
|
Sometimes the URL prefix switches from a http to a https, this patch harden the
code that removes the URL prefix from wikidata Q-name, issue has been reported
in [1].
[1] https://github.com/searxng/searxng/pull/3437#issuecomment-2082121730
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
|
|
In the past, some files were tested with the standard profile, others with a
profile in which most of the messages were switched off ... some files were not
checked at all.
- ``PYLINT_SEARXNG_DISABLE_OPTION`` has been abolished
- the distinction ``# lint: pylint`` is no longer necessary
- the pylint tasks have been reduced from three to two
1. ./searx/engines -> lint engines with additional builtins
2. ./searx ./searxng_extra ./tests -> lint all other python files
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
- pylint all scripts
- fix some errors reported by pyright
- from searx.data import data_dir (Path.open)
- fix import from pygments.formatters.html
NOTE: none functional changes!
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
JSON file which contains the bangs / there is no longer a versioning of this
file.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
To avoid unnecessary changes to the file, the list should be sorted before it is
written to the file.
You can test it by calling multiple times::
make data.locales
and searx/data/locales.json should be unchanged.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
babel.Locale.parse loads more than 60MB in RAM. The only purpose is to get:
LOCALE_NAMES - searx.data.LOCALES["LOCALE_NAMES"]
RTL_LOCALES - searx.data.LOCALES["RTL_LOCALES"]
This commit calls babel.Locale.parse when the translations are update from
weblate and stored in::
searx/data/locales.json
This file can be build by::
./manage data.locales
By store these variables in searx.data when the translations are updated we save
round about 65MB (usually 4 worker = 260MB of RAM saved.
Suggested-by: https://github.com/searxng/searxng/discussions/2633#discussioncomment-8490494
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
|
|
This reverts commit 3c6549a17f886b90a8e700810f9e1aea86581966.
Related:
- https://github.com/searxng/searxng/pull/2826
|
|
|
|
|
|
Closes: https://github.com/searxng/searxng/issues/1354
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
incident:
flask_babel.gettext() does not work in the engine modules.
cause:
the request() and response() functions of the engine modules run in the
processor, whose search() method runs in a thread and in the threads the
context of the Flask app does not exist. The context of the Flask app is
needed by the gettext() function for the L10n.
Solution:
copy context of the Flask app into the threads. [1]
special case:
We cannot equip the search() method of the processors with the decorator [1],
because the decorator requires a context (Flask app) that does not yet exist
at the time of the initialization of the processors (the initialization of the
processors is part of the initialization of the Flask app).
[1] https://flask.palletsprojects.com/en/2.3.x/api/#flask.copy_current_request_context
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
'insert'
Replace lists with one item by the item, not before last currency has been
added. In this traceback 'MXN' is added to 'pesos' while pesos is no longer a
list as the optimization was carried out too early.
$ ./local/py3/bin/python searxng_extra/update/update_currencies.py
Traceback (most recent call last):
File "searxng_extra/update/update_currencies.py", line 164, in <module>
main()
File "searxng_extra/update/update_currencies.py", line 157, in main
add_currency_name(db, "pesos", 'MXN')
File "searxng_extra/update/update_currencies.py", line 89, in add_currency_name
iso4217_set.insert(0, iso4217)
AttributeError: 'str' object has no attribute 'insert'
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
change)
Over the years the webapp module became more and more a mess. To improve the
modulaization a little this patch moves some implementations from the webapp
module to webutils module.
HINT: this patch brings non functional change
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
🇮🇱 https://emojipedia.org/flag-israel/
|
|
Wikipedia description are fetched without the help the wikipedia engine:
* the SQPARL query return the wikipedia URL of the article
|
|
Follow up of #2269
The script to update the descriptions of the engines does no longer work since
PR #2269 has been merged.
searx/engines/wikipedia.py
==========================
1. There was a misusage of zh-classical.wikipedia.org:
- `zh-classical` is dedicate to classical Chinese [1] which is not
traditional Chinese [2].
- zh.wikipedia.org has LanguageConverter enabled [3] and is going to
dynamically show simplified or traditional Chinese according to the
HTTP Accept-Language header.
2. The update_engine_descriptions.py needs a list of all wikipedias. The
implementation from #2269 included only a reduced list:
- https://meta.wikimedia.org/wiki/Wikipedia_article_depth
- https://meta.wikimedia.org/wiki/List_of_Wikipedias
searxng_extra/update/update_engine_descriptions.py
==================================================
Before PR #2269 there was a match_language() function that did an approximation
using various methods. With PR #2269 there are only the types in the data model
of the languages, which can be recognized by babel. The approximation methods,
which are needed (only here) in the determination of the descriptions, must be
replaced by other methods.
[1] https://en.wikipedia.org/wiki/Classical_Chinese
[2] https://en.wikipedia.org/wiki/Traditional_Chinese_characters
[3] https://www.mediawiki.org/wiki/Writing_systems#LanguageConverter
Closes: https://github.com/searxng/searxng/issues/2330
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
This patch replaces the *full of magic* ``utils.match_language`` function by a
``locales.match_locale``. The ``locales.match_locale`` function is based on the
``locales.build_engine_locales`` introduced in 9ae409a0 [1].
In the past SearXNG did only support a search by a language but not in a region.
This has been changed a long time ago and regions have been added to SearXNG
core but not to the engines. The ``utils.match_language`` was the function to
handle the different aspects of language/regions in SearXNG core and the
supported *languages* in the engine. The ``utils.match_language`` did it with
some magic and works good for most use cases but fails in some edge case.
To replace the concurrence of languages and regions in the SearXNG core the
``locales.build_engine_locales`` was introduced in 9ae409a0 [1]. With the last
patches all engines has been migrated to a ``fetch_traits`` and a
language/region concept that is based on ``locales.build_engine_locales``.
To summarize: there is no longer a need for the ``locales.match_language``.
[1] https://github.com/searxng/searxng/pull/1652
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
With the language and region tags from the EngineTraitsMap the handling of
SearXNG's tags of languages and regions has been normalized and is no longer
a *mystery*. The "languages" became "locales" that are supported by babel and
by this, the update_engine_traits.py can be simplified a lot.
Other code places can be simplified as well, but these simplifications
should (respectively can) only be done when none of the engines work with the
deprecated EngineTraits.supported_languages interface anymore.
This commit replaces searx.languages by searx.sxng_locales and fix the naming of
some names from "language" to "locale" (e.g. language_codes --> sxng_locales).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Implementations of the *traits* of the engines.
Engine's traits are fetched from the origin engine and stored in a JSON file in
the *data folder*. Most often traits are languages and region codes and their
mapping from SearXNG's representation to the representation in the origin search
engine.
To load traits from the persistence::
searx.enginelib.traits.EngineTraitsMap.from_data()
For new traits new properties can be added to the class::
searx.enginelib.traits.EngineTraits
.. hint::
Implementation is downward compatible to the deprecated *supported_languages
method* from the vintage implementation.
The vintage code is tagged as *deprecated* an can be removed when all engines
has been ported to the *traits method*.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
To reproduce the issue set base_url::
diff --git a/searx/settings.yml b/searx/settings.yml
index 841457b5e..4e282cb61 100644
--- a/searx/settings.yml
+++ b/searx/settings.yml
@@ -72,7 +72,7 @@ server:
bind_address: "127.0.0.1"
# public URL of the instance, to ensure correct inbound links. Is overwritten
# by ${SEARXNG_URL}.
- base_url: false # "http://example.com/location"
+ base_url: "http://example.com/location"
limiter: false # rate limit the number of request on the instance, block some bots
and build the docs::
$ make docs
SPHINX HTML ./docs --> file:///800GBPCIex4/share/SearXNG/dist/docs
DOCS build build/docs/includes
Traceback (most recent call last):
File "searxng_extra/docs_prebuild", line 85, in <module>
sys.exit(main())
File "searxng_extra/docs_prebuild", line 31, in main
f.write(page.content)
File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
next(self.gen)
File "searxng_extra/docs_prebuild", line 81, in _instance_infosetset_ctx
return DOC
NameError: name 'DOC' is not defined
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Closes: https://github.com/searxng/searxng/issues/2168
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
|
|
The Firefox version in the user agent doesn't include the patch version: 106.0 not 106.0.2
Close #1914
|
