| Age | Commit message (Collapse) | Author |
|---|
|
Set minor versioning for most of the packages that iterate fast or we know won't
cause problems, and fixed versioning for the rest.
Packages going into bundles should be placed in "dependencies".
The inspection of prod bundles is necessary. Although it does not make a lot of
sense right now, it will be useful in later PR and will give us a reference to
start with.
|
|
Pyright has been installed twice so far, once via `package.json` and once in the
test script via `npx --no-install`. Separating the type checks in the CI and on
the developer desktop is also not necessary.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Implement rules and functions to format shell scripts:
$ make format.shell
or alternatively to format all source code:
$ make format
The formatter `shfmt` reads the rules from the editorconfig[1]
If any EditorConfig files are found, they will be used to apply formatting
options. If any parser or printer flags are given to the tool, no
EditorConfig files will be used.
[1] https://github.com/patrickvane/shfmt?tab=readme-ov-file#description
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Removes obsolete scripts and fix various leftovers.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
This patch migrates from `redis==5.2.1` [1] to `valkey==6.1.0` [2].
The migration to valkey is necessary because the company behind Redis has decided
to abandon the open source license. After experiencing a drop in user numbers,
they now want to run it under a dual license again. But this move demonstrates
once again how unreliable the company is and how it treats open source
developers.
To review first, read the docs::
$ make docs.live
Follow the instructions to remove redis:
- http://0.0.0.0:8000/admin/settings/settings_redis.html
Config and install a local valkey DB:
- http://0.0.0.0:8000/admin/settings/settings_valkey.html
[1] https://pypi.org/project/redis/
[2] https://pypi.org/project/valkey/
Co-authored-by: HLFH <gaspard@dhautefeuille.eu>
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
This is one of various PR to refactor the simple theme internally.
Replace eslint tool with Biome. I have been using this for quite some time, and
it will help us to have more consistent and valid code without extending on
other third party plugins.
Removes unused dependencies.
|
|
That entrypoint is prone to screw things up, especially with permission handling. The new script handles initialization better and fixes some issues like delayed settings update via ENVs and timestamp overwriting, also adjusts what should be copied into the container.
Related https://github.com/searxng/searxng/pull/4721#issuecomment-2850272129
|
|
Everyone should have already switched from legacy methods
|
|
container.yml will run after integration.yml COMPLETES successfully and in master branch.
Style changes, cleanup and improved integration with CI by leveraging the use of
shared cache between all workflows.
* Podman is now supported to build the container images (Docker also received a refactor, merging both build and buildx)
* Container images are being built by Buildah instead of Docker BuildKit.
* Container images are tested before release.
* Splitting "modern" (amd64 & arm64) and "legacy" (armv7) arches on different Dockerfiles allowing future optimizations.
|
|
The big queries for initializing and updating the currencies take longer than
the default of the wikidata engine, which is only 3sec.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
- upgrade to ubuntu-24.04 to get NodeJS v20
- remove DEBUG (V=1) environment / in a ViteJS env the DEBUG environment is
reserved for interactive debugging tasks (not for verbose build messages).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Typification of SearXNG
=======================
This patch introduces the typing of the results. The why and how is described
in the documentation, please generate the documentation ..
$ make docs.clean docs.live
and read the following articles in the "Developer documentation":
- result types --> http://0.0.0.0:8000/dev/result_types/index.html
The result types are available from the `searx.result_types` module. The
following have been implemented so far:
- base result type: `searx.result_type.Result`
--> http://0.0.0.0:8000/dev/result_types/base_result.html
- answer results
--> http://0.0.0.0:8000/dev/result_types/answer.html
including the type for translations (inspired by #3925). For all other
types (which still need to be set up in subsequent PRs), template documentation
has been created for the transition period.
Doc of the fields used in Templates
===================================
The template documentation is the basis for the typing and is the first complete
documentation of the results (needed for engine development). It is the
"working paper" (the plan) with which further typifications can be implemented
in subsequent PRs.
- https://github.com/searxng/searxng/issues/357
Answer Templates
================
With the new (sub) types for `Answer`, the templates for the answers have also
been revised, `Translation` are now displayed with collapsible entries (inspired
by #3925).
!en-de dog
Plugins & Answerer
==================
The implementation for `Plugin` and `Answer` has been revised, see
documentation:
- Plugin: http://0.0.0.0:8000/dev/plugins/index.html
- Answerer: http://0.0.0.0:8000/dev/answerers/index.html
With `AnswerStorage` and `AnswerStorage` to manage those items (in follow up
PRs, `ArticleStorage`, `InfoStorage` and .. will be implemented)
Autocomplete
============
The autocompletion had a bug where the results from `Answer` had not been shown
in the past. To test activate autocompletion and try search terms for which we
have answerers
- statistics: type `min 1 2 3` .. in the completion list you should find an
entry like `[de] min(1, 2, 3) = 1`
- random: type `random uuid` .. in the completion list, the first item is a
random UUID
Extended Types
==============
SearXNG extends e.g. the request and response types of flask and httpx, a module
has been set up for type extensions:
- Extended Types
--> http://0.0.0.0:8000/dev/extended_types.html
Unit-Tests
==========
The unit tests have been completely revised. In the previous implementation,
the runtime (the global variables such as `searx.settings`) was not initialized
before each test, so the runtime environment with which a test ran was always
determined by the tests that ran before it. This was also the reason why we
sometimes had to observe non-deterministic errors in the tests in the past:
- https://github.com/searxng/searxng/issues/2988 is one example for the Runtime
issues, with non-deterministic behavior ..
- https://github.com/searxng/searxng/pull/3650
- https://github.com/searxng/searxng/pull/3654
- https://github.com/searxng/searxng/pull/3642#issuecomment-2226884469
- https://github.com/searxng/searxng/pull/3746#issuecomment-2300965005
Why msgspec.Struct
==================
We have already discussed typing based on e.g. `TypeDict` or `dataclass` in the past:
- https://github.com/searxng/searxng/pull/1562/files
- https://gist.github.com/dalf/972eb05e7a9bee161487132a7de244d2
- https://github.com/searxng/searxng/pull/1412/files
- https://github.com/searxng/searxng/pull/1356
In my opinion, TypeDict is unsuitable because the objects are still dictionaries
and not instances of classes / the `dataclass` are classes but ...
The `msgspec.Struct` combine the advantages of typing, runtime behaviour and
also offer the option of (fast) serializing (incl. type check) the objects.
Currently not possible but conceivable with `msgspec`: Outsourcing the engines
into separate processes, what possibilities this opens up in the future is left
to the imagination!
Internally, we have already defined that it is desirable to decouple the
development of the engines from the development of the SearXNG core / The
serialization of the `Result` objects is a prerequisite for this.
HINT: The threads listed above were the template for this PR, even though the
implementation here is based on msgspec. They should also be an inspiration for
the following PRs of typification, as the models and implementations can provide
a good direction.
Why just one commit?
====================
I tried to create several (thematically separated) commits, but gave up at some
point ... there are too many things to tackle at once / The comprehensibility of
the commits would not be improved by a thematic separation. On the contrary, we
would have to make multiple changes at the same places and the goal of a change
would be vaguely recognizable in the fog of the commits.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
|
|
To avoid unnecessary changes to the file, the list should be sorted before it is
written to the file.
You can test it by calling multiple times::
make data.locales
and searx/data/locales.json should be unchanged.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
SearXNG is a rolling release / we do not deploy packages on PyPi
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
All the environments defined in ./utils/brand.env are generated on the fly, so
there is no longer a need to define the brand environment in this file and all
the workflows to handle this file.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
BTW force modularization of the ./mange script into sub modules:
- utils/lib_sxng_data.sh
- utils/lib_sxng_node.sh
- utils/lib_sxng_static.sh
- utils/lib_sxng_test.sh
- utils/lib_sxng_themes.sh
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Follow up of https://github.com/searxng/searxng/pull/2269
Related:
- https://github.com/searxng/searxng/issues/2314
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
- rename ``./manage node.env.devtools`` to ``./manage node.env.dev``
- add ``make node.env.dev``
- remove local developer tools by ``make node.clean``
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Since ./utils/searxng.sh is implemented, the old installation procedures from
filtron, morty and searx can be removed.
For users who want to upgrade, the procedures for removing old installations
have still been retained.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
This is the first step of removing oscar theme
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
We need to cover issue related to pybabel/translation tasks. By example there
was an issue [1] (fixed [2]) when upgrading jinja2 from 3.0.3 to 3.1.0 [3] that
has not been covered by the CI tests.
[1] https://github.com/searxng/searxng/runs/5688624325?check_suite_focus=true#step:6:348
[2] https://github.com/searxng/searxng/pull/1011
[3] https://github.com/searxng/searxng/pull/1008
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Since we currently have many type checking errors,
we for now only test with typeCheckingMode: off
which makes pyright only check files that contain a comment:
# pyright: basic
to enable basic type checking, or
# pyright: strict
to enable strict type checking.
|
|
This patch implements the command and Makefile target::
./manage test.pyright
make test.pyright
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Add redis DB and connector
|
|
[mod] manage: add themes.live command (rebuild on modification)
|
|
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Environment variable to get live builds while modifying CSS & JS of a theme::
LIVE_THEME=simple make run
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
"make test.black" checks for the code style
"make format.python" format the python code
|
|
Add the 'SearXNG JavaScript Style Guide' as one quality gate in the build chain
of the simple theme::
make themes.simple
make themes.simple.test
In the github CI the `themes.all` target enforce a `themes.simple`.
BTW: Remove 'jshint' left overs from 0ee316f3d
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
[mod] Tools to install and maintain golang binaries & packages.
|
|
Remove targets marked as deprecated and convert selected to short hands [1].
[1] https://github.com/searxng/searxng/pull/511#issuecomment-971324265
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
[1] https://github.com/nvm-sh/nvm
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
[1] https://golang.org/doc/devel/release#policy
A simple *helloworld* test with user 'my_user' :
sudo -H adduser my_user
./manage go.golang go1.17.3 my_user
./manage go.install github.com/go-training/helloworld@latest my_user
./manage go.bash my_user
$ helloword
Hello World!!
Don't forget to remove 'my_user': sudo -H deluser --remove-home my_user
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
|
|
|
|
in debug mode, add some padding to make the output more readable
in procution mode, add the timestamp
|
|
Functions implemented to run *Weblate* workflows should use prefix 'weblate.':
- babel.setup.translations.worktree --> weblate.translations.worktree
Create git worktree ${TRANSLATIONS_WORKTREE} and checkout branch
'translations' from Weblate's counterpart (weblate) of the SearXNG
(origin)::
remote weblate https://weblate.bubu1.eu/git/searxng/searxng/
- babel.weblate.to.translations --> weblate.to.translations
Update 'translations' branch of SearXNG (origin) with last additions from
Weblate.
- babel.translations.to.master --> weblate.translations.commit
Update 'translations' branch of SearXNG (origin) with last additions from
Weblate. Copy the changes to the master branch, compile translations and
create a commit in the local branch (master)
- babel.master.to.translations --> weblate.push.translations
Push *translation changes* from SearXNG (origin) to Weblate's
counterpart (weblate).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
* make babel.translations.to.master: pull weblate updates
* make babel.master.to.translations: push .pot and .po files to weblate
|
|
In ./manage implement babel.*:
- extract : extract messages from source files and generate POT file
- update : update existing message catalogs from POT file
- compile : compile translation catalogs into binary MO files
Replace searx_extra/update/update_translations.sh by command:
- ci.babel.update
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
**new** utils/lib_install.sh:
Used to initialize installation procedures
- Modified source_dot_config function that
- loads .config.sh from an existing installation (at SEARX_SRC).
- initialize **SEARX_SRC_INIT_FILES**
- functions like:
- install_log_searx_instance()
- install_searx_get_state()
**modified** utils/searx.sh
- obsolete environment SEARX_INSTANCE_NAME has been replaced
by string 'SearXNG'.
**modified** utils/filtron.sh, utils/morty.sh, utils/searx.sh
- source utils/lib_install.sh
- normalize logging of environment variables using new function
install_log_searx_instance()
**modified** utils/lib.sh
- fix marginal typos
**Installation scripts**
The utils/lib_install.sh is sourced by the installations scripts:
- utils/searx.sh
- utils/morty.sh
- utils/filtron.sh
If '${SEARX_SRC}/.config.sh' exists, the modified source_dot_config() function
loads this configuration (instead of './.config.sh').
**SEARX_SRC_INIT_FILES**
Array of file names to sync into a installation at $SEARX_SRC. The file names
are relative to the $REPO_ROOT. Set by function init_SEARX_SRC_INIT_FILES().
Most often theses are files like:
- .config.sh
- searx/settings.yml
- utils/brand.env
- ...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
The functions:
- static.build.commit
- static.build.commit.drop
- static.build.restore
are imported into the ./manage script. To avoid name collisions some variables
and fucntions has been renamed by adding the prefix *static_*.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
- add script to shellcheck,
- fix error messages from shellcheck and
- moved global variables to local variables (lower case)
No functional change!
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
