summaryrefslogtreecommitdiff
path: root/searx/data
AgeCommit message (Collapse)Author
2025-10-20[fix] TrackerPatternsDB.clean_url: don't delete query argument from new_url ↵Markus Heiser
(#5339) The query argument for URLs like: - 'http://example.org?q=' --> query_str is 'q=' - 'http://example.org?/foo/bar' --> query_str is 'foo/bar' is a *simple string* and not a key/value dict. This string may only be removed from the URL if one of the patterns matches. BTW get_pretty_url(): keep such a *simple string* in the path element. Closes: https://github.com/searxng/searxng/issues/5299 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2025-09-30[data] update searx.data - update_engine_traits.pysearxng-bot
2025-09-29[data] update searx.data - update_currencies.py (#5261)github-actions[bot]
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
2025-09-29[data] update searx.data - update_engine_descriptions.py (#5262)github-actions[bot]
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
2025-09-29[data] update searx.data - update_wikidata_units.py (#5259)github-actions[bot]
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
2025-09-29[data] update searx.data - update_ahmia_blacklist.py (#5257)github-actions[bot]
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
2025-09-29[data] update searx.data - update_firefox_version.pysearxng-bot
2025-09-28[mod] ExpireCacheSQLite - implement .setmany() for bulk loadingMarkus Heiser
[1] https://github.com/searxng/searxng/issues/5223#issuecomment-3328597147 Suggested-by: Ivan G <igabaldon@inetol.net> [1] Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2025-09-28[mod] debug-log the bootload of the CURRENCIES caches (CurrenciesDB)Markus Heiser
The time for the bootload is measured and recorded. To have an eye on the bootload time is motivated by: In [1] it can be seen that the bootload of the CURRENCIES cache takes about 2/3 of the time required to initialize SearXNG:: 6.068 <module> searx/webapp.py:1 └─ 6.068 wrapper pyinstrument/context_manager.py:52 ├─ 5.538 init searx/webapp.py:1373 │ ├─ 4.822 initialize searx/search/__init__.py:34 │ │ ├─ 4.631 ProcessorMap.init searx/search/processors/__init__.py:47 │ │ │ ├─ 4.607 OnlineCurrencyProcessor.initialize searx/search/processors/online_currency.py:55 │ │ │ │ └─ 4.607 CurrenciesDB.init searx/data/currencies.py:25 │ │ │ │ ├─ 4.601 CurrenciesDB.load searx/data/currencies.py:34 │ │ │ │ │ ├─ 4.572 ExpireCacheSQLite.set searx/cache.py:334 In the example, the CurrenciesDB.init call takes 4.6 seconds... on my laptop, CurrenciesDB.init takes only 0.7 seconds. The absolute numerical values depend on external conditions, where I already find 4-5 seconds very long. Test:: $ rm /tmp/sxng_cache_*.db* $ make run 2>&1 | grep "searx.data.CURRENCIES" DEBUG searx.data : init searx.data.CURRENCIES DEBUG searx.data : init searx.data.CURRENCIES added 9089 items in 0.7623255252838135 sec. [1] https://github.com/searxng/searxng/issues/5223#issuecomment-3323083411 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2025-09-26[upd] pypi: Bump black from 24.3.0 to 25.9.0 (#5251)Markus Heiser
In 25.1.0 [2] an old bug has been fixed: "Docstring formatting does not apply to module docstrings" [3]. [1] https://github.com/psf/black/blob/main/CHANGES.md#2590 [2] https://github.com/psf/black/blob/main/CHANGES.md#2510 [3] https://github.com/psf/black/issues/4094 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2025-09-18[mod] addition of various type hints / engine processorsMarkus Heiser
Continuation of #5147 .. typification of the engine processors. BTW: - removed obsolete engine property https_support - fixed & improved currency_convert - engine instances can now implement a engine.setup method [#5147] https://github.com/searxng/searxng/pull/5147 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2025-09-03[mod] drop: from __future__ import annotationsMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2025-09-03[mod] addition of various type hints / tbcMarkus Heiser
- pyright configuration [1]_ - stub files: types-lxml [2]_ - addition of various type hints - enable use of new type system features on older Python versions [3]_ - ``.tool-versions`` - set python to lowest version we support (3.10.18) [4]_: Older versions typically lack some typing features found in newer Python versions. Therefore, for local type checking (before commit), it is necessary to use the older Python interpreter. .. [1] https://docs.basedpyright.com/v1.20.0/configuration/config-files/ .. [2] https://pypi.org/project/types-lxml/ .. [3] https://typing-extensions.readthedocs.io/en/latest/# .. [4] https://mise.jdx.dev/configuration.html#tool-versions Signed-off-by: Markus Heiser <markus.heiser@darmarit.de> Format: reST
2025-08-29[data] update searx.data - update_external_bangs.py (#5158)github-actions[bot]
2025-08-29[data] update searx.data - update_wikidata_units.py (#5159)github-actions[bot]
2025-08-29[data] update searx.data - update_firefox_version.py (#5160)github-actions[bot]
2025-08-29[data] update searx.data - update_ahmia_blacklist.py (#5161)github-actions[bot]
2025-08-29[data] update searx.data - update_currencies.py (#5162)github-actions[bot]
2025-08-29[data] update searx.data - update_engine_traits.py (#5163)github-actions[bot]
2025-08-29[data] update searx.data - update_engine_descriptions.py (#5164)github-actions[bot]
2025-08-01[fix] CI task "update_engine_traits.py" fails (#5069)Markus Heiser
* [fix] CI task "update_engine_traits.py" fails To catch all problems with an HTTP request, the more general class ``httpx.HTTPError`` must be caught, for your test use:: $ ./manage dev.env $ python ./searxng_extra/update/update_engine_traits.py Closes: https://github.com/searxng/searxng/issues/5068 * [data] update searx.data - update_engine_traits.py
2025-07-29[data] update searx.data - update_wikidata_units.py (#5062)github-actions[bot]
2025-07-29[data] update searx.data - update_external_bangs.py (#5061)github-actions[bot]
2025-07-29[data] update searx.data - update_ahmia_blacklist.py (#5064)github-actions[bot]
2025-07-29[data] update searx.data - update_currencies.py (#5065)github-actions[bot]
2025-07-29[data] update searx.data - update_engine_descriptions.py (#5066)github-actions[bot]
2025-07-29[data] update searx.data - update_firefox_version.py (#5063)github-actions[bot]
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
2025-07-28[fix] tracker pattern: let startup continue if url fetch fails (#5055)mggh0139
Use Python exception to prevent startup crash in case of fetch ClearURL failure. Also add some logs. Closes: https://github.com/searxng/searxng/issues/5054
2025-07-22[fix] tracker url remover + external bangs: use standard network configBnyro
Using plain `httpx` directly doesn't use SearXNG's additional network config, including proxies, http2 config, ... Related issues: - https://github.com/searxng/searxng/issues/5027
2025-07-09[fix] prevent multiple, parallel initializations of tables in the cache DB ↵Markus Heiser
(#4991) Depending on the respective runtime behavior, it could happen that the initial loading of the DB tables in the cache was performed multiple times and in parallel. The concurrent accesses then led to the `sqlite3.OperationalError: database is locked` exception as in #4951. Since this problem depends significantly on the runtimes (e.g., how long it takes to retrieve the content for a table), this error could not be observed in all installations. Closes: https://github.com/searxng/searxng/issues/4951 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2025-06-29[data] update searx.data - update_firefox_version.py (#4954)github-actions[bot]
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
2025-06-29[data] update searx.data - update_external_bangs.py (#4955)github-actions[bot]
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
2025-06-29[data] update searx.data - update_wikidata_units.py (#4956)github-actions[bot]
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
2025-06-29[data] update searx.data - update_ahmia_blacklist.py (#4957)github-actions[bot]
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
2025-06-29[data] update searx.data - update_currencies.py (#4958)github-actions[bot]
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
2025-06-29[data] update searx.data - update_engine_traits.py (#4959)github-actions[bot]
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
2025-06-29[data] update searx.data - update_engine_descriptions.py (#4960)github-actions[bot]
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
2025-06-23[mod] data: implement a simple tracker URL (SQL) databaseMarkus Heiser
On demand, the tracker data is loaded directly into the cache, so that the maintenance of this data via PRs is no longer necessary. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2025-06-23[data] update searx.data - add tracker_patterns.jsonBnyro
2025-06-23[feat] tracker url plugin: use ClearURL tracking param listBnyro
2025-05-29[fix] plugins: currency processor bugMarkus Heiser
Fix C&P typo from https://github.com/searxng/searxng/pull/4836 Closes: https://github.com/searxng/searxng/issues/4861
2025-05-29[data] update searx.data - update_ahmia_blacklist.pysearxng-bot
2025-05-29[data] update searx.data - update_firefox_version.pysearxng-bot
2025-05-29[data] update searx.data - update_wikidata_units.pysearxng-bot
2025-05-29[data] update searx.data - update_currencies.pysearxng-bot
2025-05-29[data] update searx.data - update_engine_traits.pysearxng-bot
2025-05-29[data] update searx.data - update_engine_descriptions.pysearxng-bot
2025-05-25[mod] data: implement a simple currencies (SQL) database (#4836)Markus Heiser
To reduce the memory footprint, this patch no longer loads the JSON data completely into memory. Instead, there is an SQL database based on `ExpireCacheSQLite`. The class CurrenciesDB is a simple DB application that encapsulates the DB (queries and initialization) and provides convenient methods like `name_to_iso4217` and `iso4217_to_name`. Related: - https://github.com/searxng/searxng/discussions/1892 - https://github.com/searxng/searxng/pull/3458#issuecomment-2900807671 - https://github.com/searxng/searxng/pull/4650 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2025-05-22[mod] searx.data: lazy load of the data objects (databases) (#4834)Markus Heiser
In the previous implementation, all databases were loaded into memory when importing the searx.data package, regardless of whether they were ever needed. Regardless of this, it is an antipattern to load entire databases into memory when importing a package or module; databases should be loaded when needed. Lazy loading is a first step toward improving memory usage and also improves performance when setting up the runtime environment. Building on this, subsequent PRs will be able to further optimize memory behavior, e.g., by using a real database application such as the one already available via searx.cache.ExpireCache Related: - https://github.com/searxng/searxng/discussions/1892 - https://github.com/searxng/searxng/pull/3458 - https://github.com/searxng/searxng/pull/4650 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2025-05-09[data] update searx.data - update_wikidata_units.py (#4738)github-actions[bot]
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-06 21:08:51 +0000