summaryrefslogtreecommitdiff
path: root/searx/engines/duckduckgo.py
AgeCommit message (Collapse)Author
2024-05-24[enh] add instant answers from ddgallendema_searxng_pi
2024-04-08[fix] ddg engine: if no vqd value can be determined, don't save NoneMarkus Heiser
Closes: https://github.com/searxng/searxng/issues/3370 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-03-11[mod] pylint all engines without PYLINT_SEARXNG_DISABLE_OPTIONMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-03-10[fix] duckduckgo.fetch_traist - URL of region definitions has changedMarkus Heiser
- https://duckduckgo.com/dist/util/u.7669f071a13a7daa57cb.js updated from u661.js to u.7669f071a13a7daa57cb / should be updated automatically? The last change was on March 23rd in dba8977b098 [1] - [1] https://github.com/searxng/searxng/pull/2269 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-03-05[fix] ddg engines (get_vqd) - the vqd value is no longer in the formMarkus Heiser
Closes: https://github.com/searxng/searxng/issues/3276 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-12[fix] ddg-lite & ddg-extra: don't send empty vqd valueMarkus Heiser
DDG's bot detection is sensitive to the vqd value. For some search terms (such as extremely long search terms that are often sent by bots), no vqd value can be determined. If SearXNG cannot determine a vqd value, then no request should go out to DDG (WEB): a request with a wrong vqd value leads to DDG temporarily putting SearXNG's IP on a block list. Requests from IPs in this block list run into timeouts. Not sure, but it seems the block list is a sliding window: to get my IP rid from the bot list I had to cool down my IP for 1h (send no requests from that IP to DDG). Since such issues can't reproduce in a local instance I tested this patch 24h on my public SearXNG instance: There are still errors (rare), but the reliability is still 100%. Related: - https://github.com/searxng/searxng/pull/2922 - https://github.com/searxng/searxng/pull/2923 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-10[fix] ddg-lite vqd value: some search terms do not have a vqd valueMarkus Heiser
Some search terms do not have results and therefore no vqd value BTW: remove a leftover from 9197efa Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-10[fix] duckduckgo lite engine: set HTTP header 'Referer'Markus Heiser
We have had problems with this before, the bot protection from ddg-lite seems to have included this referer in the rating [1][2]. From reverse engineering: - The Referer ``https://google.com/`` was set in commt 257dc7d6c4 --> DDG lite does not like this referer anymore! - The 'Referer' header is only set on second and follow up pages but not on the first page - The vqd value is not needed on the first page, the ddg-lite client sets this value only on follow up pages / this can help to reduce the vqd requests from SearXNG. Related to 'Referer' header & ddg requests: [1] https://github.com/searxng/searxng/pull/2161 [2] https://github.com/searxng/searxng/pull/2081 Closes: https://github.com/searxng/searxng/issues/2796 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-09[feat] duckduckgo: support for videos and newsBnyro
2023-09-22Revert "[fix] engine - duckduckgo vqd edge-case"Markus Heiser
This reverts commit 102502a4f09e78682cd4f030605be394bc33282c.
2023-09-20[fix] engine - duckduckgo vqd edge-casejazzzooo
2023-09-18[fix] spellingjazzzooo
2023-09-05[fix] engine - duckduckgo_images / determination of vqd value incorrectMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-25[fix] engine & network issues / documentation and type annotationsMarkus Heiser
This patch fixes some quirks and issues related to the engines and the network. Each engine has its own network and this network was broken for the following engines[1]: - archlinux - bing - dailymotion - duckduckgo - google - peertube - startpage - wikipedia Since the files have been touched anyway, the type annotaions of the engine modules has also been completed so that error messages from the type checker are no longer reported. Related and (partial) fixed issue: - [1] https://github.com/searxng/searxng/issues/762#issuecomment-1605323861 - [2] https://github.com/searxng/searxng/issues/2513 - [3] https://github.com/searxng/searxng/issues/2515 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-12[fix] engine ddg: minor change in the API of ddgMarkus Heiser
Closes: https://github.com/searxng/searxng/issues/2419 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-03[fix] engine ddg: quote !bangs in a request send to ddgMarkus Heiser
Closes: https://github.com/searxng/searxng/issues/392 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] DuckDuckGo: reversed engineered & upgrade to data_type: traits_v1Markus Heiser
Partial reverse engineering of the DuckDuckGo (DDG) engines including a improved language and region handling based on the enigne.traits_v1 data. - DDG Lite - DDG Instant Answer API - DDG Images - DDG Weather docs/src/searx.engine.duckduckgo.rst: Online documentation of the DDG engines (make docs.live) searx/data/engine_traits.json Add data type "traits_v1" generated by the fetch_traits() functions from: - "duckduckgo" (WEB), - "duckduckgo images" and - "duckduckgo weather" and remove data from obsolete data type "supported_languages". searx/autocomplete.py: Reversed engineered Autocomplete from DDG. Supports DDG's languages. searx/engines/duckduckgo.py: - fetch_traits(): Fetch languages & regions from DDG. - get_ddg_lang(): Get DDG's language identifier from SearXNG's locale. DDG defines its languages by region codes. DDG-Lite does not offer a language selection to the user, only a region can be selected by the user. - Cache ``vqd`` value: The vqd value depends on the query string and is needed for the follow up pages or the images loaded by a XMLHttpRequest (DDG images). The ``vqd`` value of a search term is stored for 10min in the redis DB. - DDG Lite engine: reversed engineered request method with improved Language and region support and better ``vqd`` handling. searx/engines/duckduckgo_definitions.py: DDG Instant Answer API The *instant answers* API does not support languages, or at least we could not find out how language support should work. It seems that most of the features are based on English terms. searx/engines/duckduckgo_images.py: DDG Images Reversed engineered request method. Improved language and region handling based on cookies and the enigne.traits_v1 data. Response: add image format to the result list searx/engines/duckduckgo_weather.py: DDG Weather Improved language and region handling based on cookies and the enigne.traits_v1 data. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] DuckDuckGo: fetch engine traits (data_type: supported_languages)Markus Heiser
Implements a fetch_traits function for the DuckDuckGo engines. .. note:: Does not include migration of the request methode from 'supported_languages' to 'traits' (EngineTraits) object! Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-03[fix-2146] set different HTTP Referer header to DuckDuckGo requestsMarkus Heiser
For what ever reasons, ddg-lite don't like the Referer https://lite.duckduckgo.com/ In an interactive session in the WEB browser the the Reverer has exactly this value, but ddg-lite don't like this value when the request is build up by SearXNG. The new value is: https://google.com/ What fakes a user comes from a google link. Related: https://github.com/searxng/searxng/pull/2081 Closes: https://github.com/searxng/searxng/issues/2146 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-01-06Add HTTP Referer header to DuckDuckGo requestsRudis Muiznieks
closes #2080
2022-12-22Fix: add trailing slash to duckduckgo urlRudis Muiznieks
Close #1854
2022-08-01[mod] add 'Accept-Language' HTTP header to online processoresMarkus Heiser
Most engines that support languages (and regions) use the Accept-Language from the WEB browser to build a response that fits to the language (and region). - add new engine option: send_accept_language_header Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-05[enh] add more categoriesMartin Fischer
2021-12-27[format.python] initial formatting of the python codeMarkus Heiser
This patch was generated by black [1]:: make format.python [1] https://github.com/psf/black Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-01[mod] engine duckduckgo - update supported_languages_urlMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-01[mod] engine duckduckgo - use DuckDuckGo-LiteMarkus Heiser
Implement a scrapper for DuckDuckGo-Lite [1]. The existing DuckDuckGo [2] engine does not support paging. DuckDuckgo-Lite is much faster, less verbose and does have a paging option (reversed engineered from the input form of [1]). [1] https://lite.duckduckgo.com/lite [2] https://duckduckgo.com/ Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-12[httpx] replace searx.poolrequests by searx.networkAlexandre Flament
settings.yml: * outgoing.networks: * can contains network definition * propertiers: enable_http, verify, http2, max_connections, max_keepalive_connections, keepalive_expiry, local_addresses, support_ipv4, support_ipv6, proxies, max_redirects, retries * retries: 0 by default, number of times searx retries to send the HTTP request (using different IP & proxy each time) * local_addresses can be "192.168.0.1/24" (it supports IPv6) * support_ipv4 & support_ipv6: both True by default see https://github.com/searx/searx/pull/1034 * each engine can define a "network" section: * either a full network description * either reference an existing network * all HTTP requests of engine use the same HTTP configuration (it was not the case before, see proxy configuration in master)
2021-03-25[enh] add year filter to duckduckgoAdam Tauber
2021-02-12[fix] duckduckgo engine: "!ddg !g" do not redirect to googleAlexandre Flament
* searx understand "!ddg !g time" as : send "!g time" to DDG * !g a DDG bang for Google: DDG return a HTTP redirect to Google This commit adds a the allows_redirect param not to follow HTTP redirect. The DDG engine returns a empty result as before without HTTP redirect.
2021-02-09[mod] duckduckgo engine: better support of the language preferenceAlexandre Flament
After the main request, send a second to https://duckduckgo.com/t/sl_h See https://github.com/searx/searx/issues/2259
2021-02-01[mod] dynamically set language_support variableAlexandre Flament
The language_support variable is set to True by default, and set to False in only 5 engines. Except the documentation and the /config URL, this variable is not used. This commit remove the variable definition in the engines, and set value according to supported_languages length: False when the length is 0, True otherwise. Close #2485
2021-01-14[enh] engines: add about variableAlexandre Flament
move meta information from comment to the about variable so the preferences, the documentation can show these information
2020-12-01[mod] pylint: numerous minor code fixesAlexandre Flament
2020-11-14[mod] remove unused importAlexandre Flament
use from searx.engines.duckduckgo import _fetch_supported_languages, supported_languages_url # NOQA so it is possible to easily remove all unused import using autoflake: autoflake --in-place --recursive --remove-all-unused-imports searx tests
2020-10-09[fix] fix duckduckgo engineAlexandre Flament
- remove paging support: a "vqd" parameter is required between each request. This parameter is uniq for each request - update the URL (no redirect), use the POST method - language support: works if there is no more than request per minute, otherwise it is ignored !
2020-10-02[mod] move extract_text, extract_url to searx.utilsAlexandre Flament
2020-09-10Drop Python 2 (1/n): remove unicode string and url_utilsDalf
2020-06-13add correction support for duckduckgoGordon Quad
2019-11-15[mod] speed optimizationDalf
compile XPath only once avoid redundant call to urlparse get_locale(webapp.py): avoid useless call to request.accept_languages.best_match
2019-10-14[fix] pep8Adam Tauber
2019-10-14[fix] duckduckgo paging - closes #1677Adam Tauber
2019-01-07Revert "remove 'all' option from search languages"Noémi Ványi
This reverts commit 4d1770398a6af8902e75c0bd885781584d39e796.
2018-03-27refactor engine's search language handlingMarc Abonce Seguin
Add match_language function in utils to match any user given language code with a list of engine's supported languages. Also add language_aliases dict on each engine to translate standard language codes into the custom codes used by the engine.
2018-03-20fix fetch_langauges to be more accurateMarc Abonce Seguin
Add languages supported by either all default general engines or 10 engines.
2018-03-14[fix] change duckduckgo url to avoid error responseMarc Abonce Seguin
2017-12-06remove 'all' option from search languagesmarc
2017-10-10update engines_languages.json and languages.pymarc
Also, fix fetch_languages.py so it can run on python3.
2017-07-21[fix] use poolrequests in ddg enginesAdam Tauber
2017-07-10[fix] duckduckgo images doesn't fail with countryless languagemarc
2017-05-27[fix] fix duckduckgo's offsetmarc
First page now starts with 0 offset, rather than starting on the 30th result. DuckDuckGo returns 30 results on each page.
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-23 19:38:39 +0000