| Age | Commit message (Collapse) | Author |
|
The engine configuration of marginalia [2][3][4][5] spams marginalia.nu with
requests from SearXNG instances [1]. It is not in the interest of SearXNG to
disturb other FOSS projects, so the engine will be removed::
- name: marginalia
engine: json_engine
shortcut: mar
categories: general
paging: false
# Key and license: https://www.marginalia.nu/marginalia-search/api/
# index: 0 popular, 1 blogs, 2 big_sites, 3 default, 4 experimental
search_url: https://api.marginalia.nu/<insert your key here>/search/{query}?index=4&count=20
results_query: results
url_query: url
title_query: title
content_query: description
timeout: 1.5
disabled: true
about:
website: https://www.marginalia.nu/
official_api_documentation: https://api.marginalia.nu/
use_official_api: true
require_api_key: true
results: JSON
[1] https://github.com/searxng/searxng/issues/1673
[2] https://github.com/searxng/searxng/pull/1627
[3] https://github.com/searxng/searxng/issues/1620
[4] https://news.ycombinator.com/item?id=35874640
[5] https://github.com/MarginaliaSearch/MarginaliaSearch/blob/d82a8584915c9d416921cc9f1c0637efedea664f/code/services-satellite/api-service/src/main/java/nu/marginalia/api/svc/ResponseCache.java#L12-L20
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
SearXNG's donation campaign has been ended.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
If language recognition fails use the Accept-Language
|
|
Closes: https://github.com/searxng/searxng/pull/1746
Closes: https://github.com/searxng/searxng/issues/1599
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
[1] https://www.reddit.com/r/searchengines/comments/128wdcp/gigablastcom_has_been_erased/
Closes: https://github.com/searxng/searxng/issues/2321
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Gigablast's API does block unauthorized request[1].
[1] https://gigablast.com/searchfeed.html
Closes: https://github.com/searxng/searxng/issues/1454
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Partial reverse engineering of the Google engines including a improved language
and region handling based on the engine.traits_v1 data.
When ever possible the implementations of the Google engines try to make use of
the async REST APIs. The get_lang_info() has been generalized to a
get_google_info() function / especially the region handling has been improved by
adding the cr parameter.
searx/data/engine_traits.json
Add data type "traits_v1" generated by the fetch_traits() functions from:
- Google (WEB),
- Google images,
- Google news,
- Google scholar and
- Google videos
and remove data from obsolete data type "supported_languages".
A traits.custom type that maps region codes to *supported_domains* is fetched
from https://www.google.com/supported_domains
searx/autocomplete.py:
Reversed engineered autocomplete from Google WEB. Supports Google's languages and
subdomains. The old API suggestqueries.google.com/complete has been replaced
by the async REST API: https://{subdomain}/complete/search?{args}
searx/engines/google.py
Reverse engineering and extensive testing ..
- fetch_traits(): Fetch languages & regions from Google properties.
- always use the async REST API (formally known as 'use_mobile_ui')
- use *supported_domains* from traits
- improved the result list by fetching './/div[@data-content-feature]'
and parsing the type of the various *content features* --> thumbnails are
added
searx/engines/google_images.py
Reverse engineering and extensive testing ..
- fetch_traits(): Fetch languages & regions from Google properties.
- use *supported_domains* from traits
- if exists, freshness_date is added to the result
- issue 1864: result list has been improved a lot (due to the new cr parameter)
searx/engines/google_news.py
Reverse engineering and extensive testing ..
- fetch_traits(): Fetch languages & regions from Google properties.
*supported_domains* is not needed but a ceid list has been added.
- different region handling compared to Google WEB
- fixed for various languages & regions (due to the new ceid parameter) /
avoid CONSENT page
- Google News do no longer support time range
- result list has been fixed: XPath of pub_date and pub_origin
searx/engines/google_videos.py
- fetch_traits(): Fetch languages & regions from Google properties.
- use *supported_domains* from traits
- add paging support
- implement a async request ('asearch': 'arc' & 'async':
'use_ac:true,_fmt:html')
- simplified code (thanks to '_fmt:html' request)
- issue 1359: fixed xpath of video length data
searx/engines/google_scholar.py
- fetch_traits(): Fetch languages & regions from Google properties.
- use *supported_domains* from traits
- request(): include patents & citations
- response(): fixed CAPTCHA detection (Scholar has its own CATCHA manager)
- hardening XPath to iterate over results
- fixed XPath of pub_type (has been change from gs_ct1 to gs_cgt2 class)
- issue 1769 fixed: new request implementation is no longer incompatible
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
- fetch_traits(): Fetch languages from peertube's search-index source code.
[mod] Include migration of the request methode from 'supported_languages'
to 'traits' (EngineTraits) object.
[fix] old supported_languages_url is no longer valid since the sources
has been moved to a different path.
- fixed code to pass pylint
- request(): complete re-implementation based on the API docs [1]
- response(): complete re-implementation, adds serveral fields missed before
- add source code documentation
[1] https://docs.joinpeertube.org/api-rest-reference.html#tag/Search/operation/searchVideos
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
|
|
Closes: https://github.com/searxng/searxng/issues/2190
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Fix the petalsearch engine
|
|
|
|
Tineye becomes active as soon as a https:// signature is found in the search
term, but most of the time a reverse image search is not requested when a URL is
specified, often the URL is just from a C&P.
The frequent requests to tineye lead in the end to the SearXNG instance being
blocked by tineye and the user seeing unexpected error messages.
BTW: many maintainers have disabled this engine in their local SearXNG settings.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Engine is broken and can't by used any longer as a simple XPath engine.
@allendema tested a engines/neeva.py version using json from the dom, but
without luck: There was some kind of captcha for pagination.
[1] https://github.com/searxng/searxng/issues/2007#issuecomment-1426061698
Closes: https://github.com/searxng/searxng/issues/2007
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
Suggested by: @wibyweb in searx https://github.com/searx/searx/pull/3465
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
* fix type in settings.yml: replace suspend_times by suspended_times
* always use delay defined in settings.yml:
* HTTP status 402 and 403: read the value from settings.yml instead of using the hardcoded value of 1 day.
* startpage engine: CAPTCHA suspend the engine for one day instead of one week
|
|
Add goo engine
|
|
Make suspended_time changeable in settings.yml
Allow different values to be set for different exceptions.
Co-authored-by: Alexandre Flament <alex@al-f.net>
|
|
|
|
|
|
- Add documentation to the plugin
- Harmonize FastText language model with SearXNG's language model
Reosurces::
import fasttext # --> +10 MB
fasttext.load_model(str(data_dir / 'lid.176.ftz')) # --> +4MB
Suggested-by: @dalf
- To speed up and simplify the deployment use fasttext-wheel instead of fasttext
- Building numpy on the Alpine Linux of docker-images takes ages --> install
py3-numpy from Alpines package manager (apk)
- Alpine Linux on docker-images (musl libc) do not support fasttext-wheel (gnu
libc) --> patch Dockerfile and build from fastetxt:
sed -i s/fasttext-wheel/fasttext/ requirements.txt
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
|
|
|
|
Add MetaCPAN engine
|
|
|
|
settings.yml:
* The default URL was unix:///usr/local/searxng-redis/run/redis.sock?db=0
* The default URL is now "false"
The default URL makes the log difficult to deal with:
if the admin didn't install a Redis instance, the logs record a false error.
It worked before because SearXNG initialized the Redis connection when the limiter started.
In this commit, SearXNG initializes Redis in searx/webapp.py
so various components can use Redis without taking care of the initialization step.
|
|
Each network can define a verify option:
* false to disable certificate verification
* a path to existing certificate.
SearXNG uses SSL_CERT_FILE and SSL_CERT_DIR when they are defined
see https://www.python-httpx.org/environment_variables/#ssl_cert_file
|
|
|
|
[PR-3366] https://github.com/searx/searx/pull/3366
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
|
|
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
|
|
* use the paper.html template
* fetch more data from the engines
* add crossref.py
|
|
add duckduckgo weather engine
|
|
Add wttr.in engine
|
|
|
|
|
|
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
no_result_for_http_status contains a list of HTTP status.
These HTTP status are seen an empty result list.
In other cases an exception is thrown as usual.
Previously raise_for_httperror were ignoring all HTTP error,
which make defective engines invisible in the stats.
|
|
Woxikon expects a word in German, so with query "foo" the site finds nothing and
respons a 404:
httpx.HTTPStatusError: Client error '404 Not Found' \
for url 'https://synonyme.woxikon.de/synonyme/foo.php'
[1] https://github.com/searxng/searxng/issues/1543#issuecomment-1193317054
Closes: https://github.com/searxng/searxng/issues/1543
Suggested-by: @allendema [1]
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
|
|
[mod] add deepl translation engine
|
|
Brave is too unstable and will often not work by default. As seen in many issues: https://github.com/searxng/searxng/issues?q=is%3Aissue++sort%3Aupdated-desc+brave+label%3Abug+
|
|
|
|
* initialize result_proxy with searx/settings_defaults.py
* allow result_proxy.key to be a string
this commit supersedes #1522
|
|
add apple app store engine
|
|
add apple maps engine
|
|
|
|
The Apple App Store is the digital app distribution platform for iOS & iPadOS.
|
|
9GAG is a social media website where users upload and share user-generated images and videos
|