diff options
| -rw-r--r-- | AUTHORS.rst | 25 | ||||
| -rw-r--r-- | docs/dev/contribution_guide.rst | 2 | ||||
| -rw-r--r-- | docs/dev/quickstart.rst | 127 | ||||
| -rw-r--r-- | searx/engines/searchcode_doc.py | 49 | ||||
| -rw-r--r-- | searx/engines/startpage.py | 4 | ||||
| -rw-r--r-- | searx/engines/twitter.py | 87 | ||||
| -rw-r--r-- | searx/settings.yml | 8 |
7 files changed, 52 insertions, 250 deletions
diff --git a/AUTHORS.rst b/AUTHORS.rst index 0de17dbb0..036ae0fe1 100644 --- a/AUTHORS.rst +++ b/AUTHORS.rst @@ -11,6 +11,7 @@ Major contributing authors: - Marc Abonce Seguin @a01200356 - @pofilo - Markus Heiser @return42 +- Émilien Devos @unixfox People who have submitted patches/translates, reported bugs, consulted features or generally made searx better: @@ -110,7 +111,6 @@ generally made searx better: - Nick Espig @nachtalb - Rachmadani Haryono @rachmadaniHaryono - Frank de Lange @yetangitu -- Émilien Devos @unixfox - Nicolas Gelot @nfk - @volth - Mathieu Brunot @madmath03 @@ -134,3 +134,26 @@ generally made searx better: - Sophie Tauchert @999eagle - @bauruine - Michael Ilsaas `<https://mikeri.net>`_ +- @renyhp +- rachmadani haryono @rachmadaniHaryono +- Mohamad Safadieh @msafadieh +- @gardouille +- @resynth1943 +- @Eliesemoule +- @gardouille +- @GazoilKerozen +- Lukáš Kucharczyk @KucharczykL +- Lynda Lopez @lyndalopez544 +- M. Efe Çetin @efectn +- Nícholas Kegler @nicholasks +- @pierrechtux +- Scott Wallace @scottwallacesh +- @Singustromo +- @TheEvilSkeleton +- @Wonderfall +- @mrwormo +- Xiaoyu WEI @xywei +- @joshu9h + + + diff --git a/docs/dev/contribution_guide.rst b/docs/dev/contribution_guide.rst index 90b22670c..cffbbcae8 100644 --- a/docs/dev/contribution_guide.rst +++ b/docs/dev/contribution_guide.rst @@ -87,7 +87,7 @@ In order to submit a patch, please follow the steps below: - Add yourself to the :origin:`AUTHORS.rst` file. -- Choose meaning full commit messages, read `Conventional Commits`_ +- Choose meaningful commit messages, read `Conventional Commits`_ .. code:: diff --git a/docs/dev/quickstart.rst b/docs/dev/quickstart.rst index 14af03fa6..bb9f4d67c 100644 --- a/docs/dev/quickstart.rst +++ b/docs/dev/quickstart.rst @@ -4,129 +4,52 @@ Development Quickstart ====================== -.. sidebar:: :ref:`makefile` +.. _npm: https://www.npmjs.com/ - For additional developer purpose there are :ref:`makefile`. - -This quickstart guide gets your environment set up with searx. Furthermore, it -gives a short introduction to the ``manage.sh`` script. - -How to setup your development environment -========================================= - -.. sidebar:: :ref:`make pyenv <make pyenv>` - - Alternatively use the :ref:`make pyenv`. - -First, clone the source code of searx to the desired folder. In this case the -source is cloned to ``~/myprojects/searx``. Then create and activate the -searx-ve virtualenv and install the required packages using ``manage.sh``. +Searx loves developers, just clone and start hacking. All the rest is done for +you simply by using :ref:`make <makefile>`. .. code:: sh - cd ~/myprojects git clone https://github.com/searx/searx.git - cd searx - python3 -m venv searx-ve - . ./searx-ve/bin/activate - ./manage.sh update_dev_packages - - -How to run tests -================ - -.. sidebar:: :ref:`make test.unit <make test>` - - Alternatively use the ``test.pep8``, ``test.unit``, ``test.robot`` targets. - -Tests can be run using the ``manage.sh`` script. Following tests and checks are -available: - -- Unit tests -- Selenium tests -- PEP8 validation -- Unit test coverage check - -For example unit tests are run with the command below: - -.. code:: sh - - ./manage.sh unit_tests - -For further test options, please consult the help of the ``manage.sh`` script or -read :ref:`make test`. - -How to compile styles and javascript -==================================== +Here is how a minimal workflow looks like: -.. _less: https://lesscss.org/ -.. _NodeJS: https://nodejs.org +1. *start* hacking +2. *run* your code: :ref:`make run` +3. *test* your code: :ref:`make test` -How to build styles -------------------- +If you think at some point something fails, go back to *start*. Otherwise, +choose a meaningful commit message and we are happy to receive your pull +request. To not end in *wild west* we have some directives, please pay attention +to our ":ref:`how to contribute`" guideline. -Less_ is required to build the styles of searx. Less_ can be installed using -either NodeJS_ or Apt. +If you implement themes, you will need to compile styles and JavaScript before +*run*. .. code:: sh - sudo -H apt-get install nodejs - sudo -H npm install -g less - -OR - -.. code:: sh - - sudo -H apt-get install node-less - -After satisfying the requirements styles can be build using ``manage.sh`` - -.. code:: sh - - ./manage.sh styles - - -How to build the source of the themes -===================================== - -.. _grunt: https://gruntjs.com/ - -Grunt_ must be installed in order to build the javascript sources. It depends on -NodeJS, so first Node has to be installed. - -.. code:: sh + make themes - sudo -H apt-get install nodejs - make node.env +Don't forget to install npm_ first. -After installing grunt, the files can be built using the following command: +.. tabs:: -.. code:: sh + .. group-tab:: Ubuntu / debian - make themes + .. code:: sh + sudo -H apt-get install npm -Tips for debugging/development -============================== + .. group-tab:: Arch Linux -.. sidebar:: :ref:`make run` + .. code-block:: sh - Makefile target ``run`` already enables debug option for your developer - session / see :ref:`make run`. + sudo -H pacman -S npm -Turn on debug logging - Whether you are working on a new engine or trying to eliminate a bug, it is - always a good idea to turn on debug logging. When debug logging is enabled a - stack trace appears, instead of the cryptic ``Internal Server Error`` - message. It can be turned on by setting ``debug: False`` to ``debug: True`` in - :origin:`settings.yml <searx/settings.yml>`. + .. group-tab:: Fedora / RHEL -.. sidebar:: :ref:`make test` + .. code-block:: sh - Alternatively use the :ref:`make test` targets. + sudo -H dnf install npm -Run ``./manage.sh tests`` before creating a PR. - Failing build on Travis is common because of PEP8 checks. So a new commit - must be created containing these format fixes. This phase can be skipped if - ``./manage.sh tests`` is run locally before creating a PR. diff --git a/searx/engines/searchcode_doc.py b/searx/engines/searchcode_doc.py deleted file mode 100644 index 878d2e792..000000000 --- a/searx/engines/searchcode_doc.py +++ /dev/null @@ -1,49 +0,0 @@ -""" - Searchcode (It) - - @website https://searchcode.com/ - @provide-api yes (https://searchcode.com/api/) - - @using-api yes - @results JSON - @stable yes - @parse url, title, content -""" - -from json import loads -from urllib.parse import urlencode - -# engine dependent config -categories = ['it'] -paging = True - -# search-url -url = 'https://searchcode.com/' -search_url = url + 'api/search_IV/?{query}&p={pageno}' - - -# do search-request -def request(query, params): - params['url'] = search_url.format(query=urlencode({'q': query}), pageno=params['pageno'] - 1) - - return params - - -# get response from search-request -def response(resp): - results = [] - - search_results = loads(resp.text) - - # parse results - for result in search_results.get('results', []): - href = result['url'] - title = "[{}] {} {}".format(result['type'], result['namespace'], result['name']) - - # append result - results.append({'url': href, - 'title': title, - 'content': result['description']}) - - # return results - return results diff --git a/searx/engines/startpage.py b/searx/engines/startpage.py index 90ed0a7a1..16e846a14 100644 --- a/searx/engines/startpage.py +++ b/searx/engines/startpage.py @@ -36,8 +36,8 @@ search_url = base_url + 'do/search' # specific xpath variables # ads xpath //div[@id="results"]/div[@id="sponsored"]//div[@class="result"] # not ads: div[@class="result"] are the direct childs of div[@id="results"] -results_xpath = '//div[@class="w-gl__result"]' -link_xpath = './/a[@class="w-gl__result-title"]' +results_xpath = '//div[@class="w-gl__result__main"]' +link_xpath = './/a[@class="w-gl__result-url result-link"]' content_xpath = './/p[@class="w-gl__description"]' diff --git a/searx/engines/twitter.py b/searx/engines/twitter.py deleted file mode 100644 index 6d9bdbb5c..000000000 --- a/searx/engines/twitter.py +++ /dev/null @@ -1,87 +0,0 @@ -""" - Twitter (Social media) - - @website https://twitter.com/ - @provide-api yes (https://dev.twitter.com/docs/using-search) - - @using-api no - @results HTML (using search portal) - @stable no (HTML can change) - @parse url, title, content - - @todo publishedDate -""" - -from urllib.parse import urlencode, urljoin -from lxml import html -from datetime import datetime -from searx.utils import extract_text - -# engine dependent config -categories = ['social media'] -language_support = True - -# search-url -base_url = 'https://twitter.com/' -search_url = base_url + 'search?' - -# specific xpath variables -results_xpath = '//li[@data-item-type="tweet"]' -avatar_xpath = './/img[contains(@class, "avatar")]/@src' -link_xpath = './/small[@class="time"]//a' -title_xpath = './/span[contains(@class, "username")]' -content_xpath = './/p[contains(@class, "tweet-text")]' -timestamp_xpath = './/span[contains(@class,"_timestamp")]' - - -# do search-request -def request(query, params): - params['url'] = search_url + urlencode({'q': query}) - - # set language if specified - if params['language'] != 'all': - params['cookies']['lang'] = params['language'].split('-')[0] - else: - params['cookies']['lang'] = 'en' - - return params - - -# get response from search-request -def response(resp): - results = [] - - dom = html.fromstring(resp.text) - - # parse results - for tweet in dom.xpath(results_xpath): - try: - link = tweet.xpath(link_xpath)[0] - content = extract_text(tweet.xpath(content_xpath)[0]) - img_src = tweet.xpath(avatar_xpath)[0] - img_src = img_src.replace('_bigger', '_normal') - except Exception: - continue - - url = urljoin(base_url, link.attrib.get('href')) - title = extract_text(tweet.xpath(title_xpath)) - - pubdate = tweet.xpath(timestamp_xpath) - if len(pubdate) > 0: - timestamp = float(pubdate[0].attrib.get('data-time')) - publishedDate = datetime.fromtimestamp(timestamp, None) - # append result - results.append({'url': url, - 'title': title, - 'content': content, - 'img_src': img_src, - 'publishedDate': publishedDate}) - else: - # append result - results.append({'url': url, - 'title': title, - 'content': content, - 'img_src': img_src}) - - # return results - return results diff --git a/searx/settings.yml b/searx/settings.yml index 3ba9b745f..e263e3ad4 100644 --- a/searx/settings.yml +++ b/searx/settings.yml @@ -694,10 +694,6 @@ engines: engine : stackoverflow shortcut : st - - name : searchcode doc - engine : searchcode_doc - shortcut : scd - - name : searchcode code engine : searchcode_code shortcut : scc @@ -764,10 +760,6 @@ engines: categories : onions shortcut : tch - - name : twitter - engine : twitter - shortcut : tw - # maybe in a fun category # - name : uncyclopedia # engine : mediawiki |