summaryrefslogtreecommitdiff
path: root/searx/engines
AgeCommit message (Collapse)Author
2021-07-11[drop] engine torrentz: torrentz2.eu and torrentz2.is are offlineMarkus Heiser
[1] https://torrentfreak.com/torrentz2-eu-domain-suspended-by-registry-on-public-prosecutors-order-200628/ Suggested-by: @rasos https://github.com/searx/searx/issues/1875#issuecomment-877755872 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-10Fix google imagesÉmilien Devos
Proposed fix in https://github.com/searx/searx/pull/2115#issuecomment-876716010
2021-06-21[docs] add documentation from the sources of the google enginesMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-21[fix] google answers: normalize space of the answers.Markus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-21[mod] google engine: reduce mobile UI parameters to what is neededMarkus Heiser
Reverse engineering shows that not all of the parameters used by google's mobile UI (aka "more results" button) are needed [1]. [1] https://github.com/searxng/searxng/pull/160#issuecomment-865013625 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-21[mod] google: add "use_mobile_ui" parameter to use mobile endpoint.Alexandre Flament
disable by default, it has to be enabled in settings.yml related to #159
2021-06-18[fix] google news - send CONSENT Cookie to not be redirectedMarkus Heiser
In the EU there exists a "General Data Protection Regulation" [1] aka GDPR (BTW: very user friendly!) which requires consent to tracking. To get the consent from the user, google-news requests are redirected to confirm and get a CONSENT Cookie from https://consent.google.de/s?continue=... This patch adds a CONSENT Cookie to the google-news request to avoid redirection. The behavior of the CONTENTS cookies over all google engines seems similar but the pattern is not yet fully clear to me, here are some random samples from my analysis .. Using common google search from different domains:: google.com: CONSENT=YES+cb.{{date}}-14-p0.de+FX+816 google.de: CONSENT=YES+cb.{{date}}-14-p0.de+FX+333 google.fr: CONSENT=YES+srp.gws-{{date}}-0-RC2.fr+FX+826 When searching about videos (google-videos):: google.es: CONSENT=YES+srp.gws-{{date}}-0-RC2.es+FX+076 google.de: CONSENT=YES+srp.gws-{{date}}-0-RC2.de+FX+171 Google news has only one domain for all languages:: news.google.com: CONSENT=YES+cb.{{date}}-14-p0.de+FX+816 Using google-scholar search from different domains:: scholar.google.de: CONSENT=YES+cb.{{date}}-14-p0.de+FX+333 scholar.google.fr: does not use such a cookie / did not ask the user scholar.google.es: does not use such a cookie / did not ask the user Interim summary: Pattern is unclear and I won't apply the CONSENT cookie to all google engines. More experience is need before we generalize the CONSENT cookies over all google engines. Related: - e9a6ab401 [fix] youtube - send CONSENT Cookie to not be redirected - https://github.com/benbusby/whoogle-search/issues/311 - https://github.com/benbusby/whoogle-search/issues/243 [1] https://en.wikipedia.org/wiki/General_Data_Protection_Regulation Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-18[fix] google-news engine - KeyError: 'hl in requestMarkus Heiser
Since we added - 1c67b6aec [enh] google engine: supports "default language" there is a KeyError: 'hl in request,error pattern:: ERROR:searx.searx.search.processor.online:engine google news : exception : 'hl' Traceback (most recent call last): File "searx/search/processors/online.py", line 144, in search search_results = self._search_basic(query, params) File "searx/search/processors/online.py", line 118, in _search_basic self.engine.request(query, params) File "searx/engines/google_news.py", line 97, in request if lang_info['hl'] == 'en': KeyError: 'hl' Closes: https://github.com/searxng/searxng/issues/154 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-14[pylint] searx/engines/duckduckgo_definitions.pyMarkus Heiser
BTW: normalize indentations Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-11[fix] log messages from: google- images, news, scholar, videosMarkus Heiser
- HTTP header Accept-Language --> lang_info['headers']['Accept-Language'] - remove obsolete query_url log messages which is already logged by httpx._client:HTTP request Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-11[mod] google - get_lang_info add documentataion & commentsMarkus Heiser
BTW: remove obsolete log messages from google engine Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-10[enh] google engine: supports "default language"Alexandre Flament
Same behaviour behaviour than Whoogle [1]. Only the google engine with the "Default language" choice "(all)"" is changed by this patch. When searching for a locate place, the result are in the expect language, without missing results [2]: > When a language is not specified, the language interpretation is left up to > Google to decide how the search results should be delivered. The query parameters are copied from Whoogle. With the ``all`` language: - add parameter ``source=lnt`` - don't use parameter ``lr`` - don't add a ``Accept-Language`` HTTP header. The new signature of function ``get_lang_info()`` is: lang_info = get_lang_info(params, lang_list, custom_aliases, supported_any_language) Argument ``supported_any_language`` is True for google.py and False for the other google engines. With this patch the function now returns: - query parameters: ``lang_info['params']`` - HTTP headers: ``lang_info['headers']`` - and as before this patch: - ``lang_info['subdomain']`` - ``lang_info['country']`` - ``lang_info['language']`` [1] https://github.com/benbusby/whoogle-search [2] https://github.com/benbusby/whoogle-search/releases/tag/v0.5.4
2021-06-09[fix] openstreetmap - fix some minor whitespace & indentation issuesMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-09[enh] openstreetmap / map template: improve resultsAlexandre Flament
implements ideas described in #69 * update the engine * use wikidata * update map.html template
2021-06-08Merge pull request #97 from return42/drop-searx-adminMarkus Heiser
[docs] reorder blog articles
2021-06-04[fix] peertube fetch supported languagesAlexandre Flament
close #127
2021-06-04[fix] typo: online_dictionnary --> online_dictionaryMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-04[docs] add engine "Demo Online Engine"Markus Heiser
This engine just exists for documentation purpose. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-04[docs] add engine "Demo Offline Engine"Markus Heiser
This engine just exists for documentation purpose. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-02Merge pull request #124 from return42/searx-mergeAlexandre Flament
merge redis offline engine from searx
2021-06-02[fix] enigine redis - avoid error when the engine is loadedMarkus Heiser
Should be _redis_client to avoid an error when the engine is loaded. Suggested-by: @dalf https://github.com/searxng/searxng/pull/124#pullrequestreview-673885664 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-01[fix] sys.exit(1) when there is duplicate engine nameAlexandre Flament
2021-06-01[mod] searx.engines.load_engine return None instead of sys.exit(1)Markus Heiser
Loading an engine should not exit the application (*). Instead of exit, return None. (*) RuntimeError still exit the application: syntax error, etc... BTW: add documentation and normalize indentation (no functional change) Suggested-by: @dalf https://github.com/searxng/searxng/pull/116#issuecomment-851865627 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-01[mod] searx.engines.__init__: refactoringAlexandre Flament
2021-06-01[enh] add redis offline engine / https://redis.io/Adam Tauber
Slightly modified merge of commit [97269be6], [01a8a5814a] and [c8d2b5eb] from searx. [97269be6] https://github.com/searx/searx/commit/97269be6 [01a8a581] https://github.com/searx/searx/commit/01a8a581 [c8d2b5eb] https://github.com/searx/searx/commit/c8d2b5eb Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-01[mod] move all default settings into searx.settings_defaultsAlexandre Flament
2021-05-30[enh] add engine - Docker HubKyle Anthony Williams
Slightly modified merge of commit [1cb1d3ac] from searx [PR 2543]: This adds Docker Hub .. as a search engine .. the engine's favicon was downloaded from the Docker Hub website with wget and converted to a PNG with ImageMagick .. It supports the parsing of URLs, titles, content, published dates, and thumbnails of Docker images. [1cb1d3ac] https://github.com/searx/searx/pull/2543/commits/1cb1d3ac [PR 2543] https://github.com/searx/searx/pull/2543 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-29[mod] the bittorent search engines are available only in the files categoryAlexandre Flament
related to #101
2021-05-28[enh] add MySQL engineNoémi Ványi
Slightly modified merge of [c00a33fe] from searx. [c00a33fe] https://github.com/searx/searx/commit/c00a33feee0886dc2b42f4c7b43731210597e5b7 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-28[enh] add PostgreSQL engineNoémi Ványi
Slightly modified merge of [22079ff] from searx. [22079ff] https://github.com/searx/searx/commit/22079ffdefb07cf69486a232e823797e737014e9 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-27Merge pull request #93 from return42/genius-miscMarkus Heiser
Some minor Genius improvements
2021-05-27Merge pull request #103 from searxng/add-sqlite-engine2Markus Heiser
[enh] add offline engine for sqlite database
2021-05-27[enh] add offline engine for sqlite databaseAlexandre Flament
To test & demonstrate this implementation download: https://liste.mediathekview.de/filmliste-v2.db.bz2 and unpack into searx/data/filmliste-v2.db, in your settings.yml define a sqlite engine named "demo":: - name : demo engine : sqlite shortcut: demo categories: general result_template: default.html database : searx/data/filmliste-v2.db query_str : >- SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title, COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url, description AS content FROM film WHERE title LIKE :wildcard OR description LIKE :wildcard ORDER BY duration DESC disabled : False Query to test: "!demo concert" This is a rewrite of the implementation from commit [1] [1] searx/searx@8e90a21 Suggested-by: @virtadpt searx/searx#2808
2021-05-25[fix] unsplash engine - 'searx:result: invalid title:'Markus Heiser
- Use result 'alt_description' as title, if not given use default title 'unknown'. - Use result 'description' from unsplash as 'content' Fix error:: DEBUG:searx:result: invalid title: {..., 'title': None, 'content': '', 'engine': 'unsplash'} Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-25[pylint] searx/engines/unsplash.py, add logger & norm indentationMarkus Heiser
- fix messages from pylint - add logger and log request URL - normalized various indentation Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24[fix] engine genius should not use the video templateMarkus Heiser
Remove 'template' from result. Engine genius should not use the video template. BTW: fix indentations Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24[pylint] searx/engines/genius.py, add logger & normalized indentationMarkus Heiser
- pylint searx/engines/genius.py - add logger and log ignored exceptions - normalized various indentation Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-23[enh] XPath engine - add time safe-search supportMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-23[enh] XPath engine - add time range supportMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-23[enh] XPath engine - add ISO 639-1 {lang} replacement to search-URLMarkus Heiser
BTW: remove obsolte params['query'] and not needed paging condition. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-23[doc] add documentation about the XPath engineMarkus Heiser
- pylint searx/engines/xpath.py - fix indentation of some long lines - add logging - add doc-strings Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17[enh] xpath engine - add request parameter 'soft_max_redirects'Markus Heiser
Make 'soft_max_redirects' configurable per Xpath engine:: - name : <engine-name> engine : xpath soft_max_redirects: 1 ... Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-05[mod] multithreading only in searx.search.* packagesAlexandre Flament
it prepares the new architecture change, everything about multithreading in moved in the searx.search.* packages previously the call to the "init" function of the engines was done in searx.engines: * the network was not set (request not sent using the defined proxy) * it requires to monkey patch the code to avoid HTTP requests during the tests
2021-05-02fix Qwant's fetch_languages functionMarc Abonce Seguin
2021-04-29[fix] URL to solidtorrent result pageMichael Ilsaas
Reported-by: https://github.com/searx/searx/pull/2786
2021-04-26[pylint] tag PYLINT_FILES by comment `# lint: pylint`Markus Heiser
These py files are linted by `test.pylint`, all other files are linted by `test.pep8`. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-25[brand] searxng -- fix links to issue tracker & WEB-GUIMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-24[mod] core.ac.uk engineMarkus Heiser
- add to list of pylint scripts - add debug log messages - move API key int `settings.yml` - improved readability - add some metadata to results Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-24add core.ac.uk enginespongebob33
2021-04-22Merge pull request #15 from return42/add-springerAlexandre Flament
Add a search engine for Springer Nature
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-21 05:35:45 +0000