[fix] ddg-lite & ddg-extra: don't send empty vqd value

DDG's bot detection is sensitive to the vqd value.  For some search terms (such
as extremely long search terms that are often sent by bots), no vqd value can be
determined.

If SearXNG cannot determine a vqd value, then no request should go out to
DDG (WEB): a request with a wrong vqd value leads to DDG temporarily putting
SearXNG's IP on a block list.

Requests from IPs in this block list run into timeouts.

Not sure, but it seems the block list is a sliding window: to get my IP rid from
the bot list I had to cool down my IP for 1h (send no requests from that IP to
DDG).

Since such issues can't reproduce in a local instance I tested this patch 24h on
my public SearXNG instance: There are still errors (rare), but the reliability
is still 100%.

Related:

- https://github.com/searxng/searxng/pull/2922
- https://github.com/searxng/searxng/pull/2923

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

1 files changed, 8 insertions, 1 deletions

diff --git a/searx/engines/duckduckgo_extra.py b/searx/engines/duckduckgo_extra.py
index 7e3a3282d..25692add7 100644
--- a/searx/engines/duckduckgo_extra.py
+++ b/searx/engines/duckduckgo_extra.py
@@ -48,6 +48,13 @@ search_path_map = {'images': 'i', 'videos': 'v', 'news': 'news'}
 
 def request(query, params):
 
+    # request needs a vqd argument
+    vqd = get_vqd(query)
+    if not vqd:
+        # some search terms do not have results and therefore no vqd value
+        params['url'] = None
+        return params
+
     eng_region = traits.get_region(params['searxng_locale'], traits.all_locale)
     eng_lang = get_ddg_lang(traits, params['searxng_locale'])
 
@@ -57,7 +64,7 @@ def request(query, params):
         # 'u': 'bing',
         'l': eng_region,
         'f': ',,,,,',
-        'vqd': get_vqd(query),
+        'vqd': vqd,
     }
 
     if params['pageno'] > 1: