Add install/update instructions database (#1160)

* Add install/update instructions database

* Update the database

* Revert README changes

They will be overwritten later.

* Revert

* Tweak

* Tweaks

* Upgrade database

* Complete commands

Still not sure about Spack upgrades.

* Sort

* Doc generation script draft

* Remove OS names from tool names

* Fix Linuxbrew name

* `wheel` already installs `setuptools`

* Gen docs

* Update

* Tweak

* Add a GitHub workflow to check for outdated installation instructions

* Fix return value

* Test

* Delete test

* Rename the script

* Add `make doc-install-inst`

* Add missing dev requirement

* The first tool is the primary we want to display

Then they are simply sorted by `tool.title`.

* Sort OSes by name

* Refactoring, jinja template, etc.

* Add tool title uniqueness `assert`, fix platform list extra `\n`

* Rebuild docs

* Update generate.py

* Update README.md

* Update methods.yml

* Update distros derived, more assertions

* Tweaks

* Add workflow to auto-update the docs

* Do not hide the command

* Tweaks

Co-authored-by: Jakub Roztocil <jakub@roztocil.co>

.github/workflows/code-style.yml

@@ -1,5 +1,3 @@
-name: Code style
-
 on:
   pull_request:
     paths:

.github/workflows/coverage.yml

@@ -1,5 +1,3 @@
-name: Coverage
-
 on:
   pull_request:
     paths:

.github/workflows/docs-check-markdown.yml

@@ -1,5 +1,3 @@
-name: Check markdown
-
 on:
   pull_request:
     paths:

.github/workflows/docs-deploy.yml

@@ -1,5 +1,3 @@
-name: Update documentation
-
 on:
   push:
     branches:

.github/workflows/docs-update-install.yml

@@ -0,0 +1,26 @@
+on:
+  push:
+    branches:
+      - master
+    paths:
+      - docs/installation/*
+
+  # Allow to call the workflow manually
+  workflow_dispatch:
+
+jobs:
+  doc:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+        with:
+          python-version: 3.9
+      - run: make install
+      - run: make doc-update-install
+      - uses: Automattic/action-commit-to-branch@master
+        with:
+          branch: master
+          commit_message: Auto-update installation instructions in the docs
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/release.yml

@@ -1,5 +1,3 @@
-name: Release
-
 on:
   # Add a "Trigger" button to manually start the workflow.
   workflow_dispatch:

.github/workflows/test-package-linux-snap.yml

@@ -1,5 +1,3 @@
-name: Linux snap
-
 on:
   pull_request:
     paths:

.github/workflows/tests.yml

@@ -1,5 +1,3 @@
-name: Tests
-
 on:
   push:
     branches:

Makefile

@@ -144,7 +144,12 @@ codecov-upload:
 
 doc-check:
 	@echo $(H1)Running documentations checks$(H1END)
-	mdl --git-recurse --style docs/linter/mdl-styles.rb .
+	mdl --git-recurse --style docs/markdownlint.rb .
+
+
+doc-update-install:
+	@echo $(H1)Updating installation instructions in the docs$(H1END)
+	$(VENV_PYTHON) docs/installation/generate.py
 
 
 ###############################################################################

docs/README.md

@@ -41,99 +41,283 @@ You are invited to submit fixes and improvements to the docs by editing [this fi
 
 ## Installation
 
-### macOS
+<div data-installation-instructions>
 
-On macOS, HTTPie can also be installed via [Homebrew](https://brew.sh/):
+<!--
+THE INSTALLATION SECTION IS GENERATED
+
+Do not edit here, but in docs/installation/.
+
+-->
+
+- [Universal](#universal)
+- [macOS](#macos)
+- [Windows](#windows)
+- [Linux](#linux)
+- [FreeBSD](#freebsd)
+
+### Universal
+
+#### PyPi
+
+Please make sure you have Python 3.6 or newer (`python --version`).
 
 ```bash
+# Install
+$ python -m pip install --upgrade pip wheel
+$ python -m pip install httpie
+```
+
+```bash
+# Upgrade
+$ python -m pip install --upgrade pip wheel
+$ python -m pip install --upgrade httpie
+```
+
+### macOS
+
+#### Homebrew
+
+To install [Homebrew](https://brew.sh/) follow [installation instructions](https://docs.brew.sh/Installation).
+
+```bash
+# Install
+$ brew update
 $ brew install httpie
 ```
 
-A MacPorts *port* is also available:
+```bash
+# Upgrade
+$ brew update
+$ brew upgrade httpie
+```
+
+#### MacPorts
+
+To install [MacPorts](https://www.macports.org/) follow [installation instructions](https://www.macports.org/install.php).
 
 ```bash
+# Install
+$ port selfupdate
 $ port install httpie
 ```
 
+```bash
+# Upgrade
+$ port selfupdate
+$ port upgrade httpie
+```
+
+#### Snapcraft (macOS)
+
+To install [Snapcraft](https://snapcraft.io/) follow [installation instructions](https://snapcraft.io/docs/installing-snapd).
+
+```bash
+# Install
+$ snap install httpie
+```
+
+```bash
+# Upgrade
+$ snap refresh httpie
+```
+
+#### Spack (macOS)
+
+To install [Spack](https://spack.readthedocs.io/en/latest/index.html) follow [installation instructions](https://spack.readthedocs.io/en/latest/getting_started.html#installation).
+
+```bash
+# Install
+$ spack install httpie
+```
+
+```bash
+# Upgrade
+$ spack install httpie
+```
+
+### Windows
+
+#### Chocolatey
+
+To install [Chocolatey](https://chocolatey.org/) follow [installation instructions](https://chocolatey.org/install).
+
+```bash
+# Install
+$ choco install httpie
+```
+
+```bash
+# Upgrade
+$ choco upgrade httpie
+```
+
 ### Linux
 
-HTTPie is available on the [Snap Store](https://snapcraft.io/httpie):
+#### Snapcraft (Linux)
+
+To install [Snapcraft](https://snapcraft.io/) follow [installation instructions](https://snapcraft.io/docs/installing-snapd).
 
 ```bash
+# Install
 $ snap install httpie
 ```
 
-And most Linux distributions provide a package that can be installed using the
+```bash
-system package manager, for example:
+# Upgrade
+$ snap refresh httpie
+```
+
+#### Linuxbrew
+
+To install [Linuxbrew](https://docs.brew.sh/Homebrew-on-Linux) follow [installation instructions](https://docs.brew.sh/Homebrew-on-Linux#install).
 
 ```bash
-# Debian, Ubuntu, etc.
+# Install
+$ brew update
+$ brew install httpie
+```
+
+```bash
+# Upgrade
+$ brew update
+$ brew upgrade httpie
+```
+
+#### Debian and Ubuntu
+
+Also works for other Debian-derived distributions like MX Linux, Linux Mint, deepin, Pop!_OS, KDE neon, Zorin OS, elementary OS, Kubuntu, Devuan, Linux Lite, Peppermint OS, Lubuntu, antiX, Xubuntu, etc.
+
+```bash
+# Install
+$ apt update
 $ apt install httpie
 ```
 
 ```bash
-# Fedora
+# Upgrade
+$ apt update
+$ apt upgrade httpie
+```
+
+#### Fedora
+
+```bash
+# Install
+$ dnf update
 $ dnf install httpie
 ```
 
 ```bash
-# CentOS, RHEL, ...
+# Upgrade
+$ dnf update
+$ dnf upgrade httpie
+```
+
+#### CentOS and RHEL
+
+Also works for other RHEL-derived distributions like ClearOS, Oracle Linux, etc.
+
+```bash
+# Install
+$ yum update
 $ yum install epel-release
 $ yum install httpie
 ```
 
 ```bash
-# Gentoo
+# Upgrade
-$ emerge httpie
+$ yum update
+$ yum upgrade httpie
 ```
 
-```bash
+#### Alpine Linux
-# Arch Linux
-$ pacman -S httpie
-```
 
 ```bash
-# Alpine Linux
+# Install
+$ apk update
 $ apk add httpie
 ```
 
 ```bash
-# Solus
+# Upgrade
-$ eopkg install httpie
+$ apk update
+$ apk add --upgrade httpie
+```
+
+#### Gentoo
+
+```bash
+# Install
+$ emerge --sync
+$ emerge httpie
+```
+
+```bash
+# Upgrade
+$ emerge --sync
+$ emerge --update httpie
+```
+
+#### Arch Linux
+
+Also works for other Arch-derived distributions like ArcoLinux, EndeavourOS, Artix Linux, etc.
+
+```bash
+# Install
+$ pacman -Sy httpie
+```
+
+```bash
+# Upgrade
+$ pacman -Syu httpie
+```
+
+#### Void Linux
+
+```bash
+# Install
+$ xbps-install -Su
+$ xbps-install -S httpie
+```
+
+```bash
+# Upgrade
+$ xbps-install -Su
+$ xbps-install -Su httpie
+```
+
+#### Spack (Linux)
+
+To install [Spack](https://spack.readthedocs.io/en/latest/index.html) follow [installation instructions](https://spack.readthedocs.io/en/latest/getting_started.html#installation).
+
+```bash
+# Install
+$ spack install httpie
+```
+
+```bash
+# Upgrade
+$ spack install httpie
 ```
 
 ### FreeBSD
 
-On FreeBSD, HTTPie is available in the ports collection. A prebuilt package
+#### FreshPorts
-can be installed via [pkg(8)](https://man.freebsd.org/pkg/8>):
 
 ```bash
+# Install
 $ pkg install www/py-httpie
 ```
 
-### Windows, universal
-
-A universal installation method (that works on Linux, macOS, Windows, FreeBSD, and always provides the latest version) is to use [pip](https://pypi.org/project/pip/):
-
 ```bash
-# Make sure we have an up-to-date version of pip and setuptools:
+# Upgrade
-$ python -m pip install --upgrade pip setuptools
+$ pkg upgrade www/py-httpie
-
-$ python -m pip install --upgrade httpie
 ```
 
-(If `pip` installation fails for some reason, you can try
+<!-- /GENERATED SECTION -->
-`easy_install httpie` as a fallback.)
 
-Windows users can also install HTTPie with [Chocolatey](https://chocolatey.org):
+</div>
-
-```bash
-$ choco upgrade httpie
-```
-
-### Python version
-
-Python version 3.6 or greater is required.
 
 ### Unstable version
 
@@ -160,7 +344,7 @@ $ snap remove httpie
 $ snap install httpie --edge
 ```
 
-Verify that now you have the [current development version identifier](https://github.com/httpie/httpie/blob/master/httpie__init__.py#L6) with the `-dev` suffix, for example:
+Verify that now you have the [current development version identifier](https://github.com/httpie/httpie/blob/master/httpie/__init__.py#L6) with the `.dev0` suffix, for example:
 
 ```bash
 $ http --version

docs/installation/README.md

@@ -0,0 +1,5 @@
+Here we maintain a database of installation methods, from which we generate
+the installation section in docs. If you’d like add or update an installation method,
+edit [methods.yml](./methods.yml), do not edit the main docs directly.
+
+For HTTPie installation instructions see: <https://httpie.io/docs#installation>.

docs/installation/generate.py

@@ -0,0 +1,85 @@
+import re
+import sys
+from pathlib import Path
+from typing import Dict
+
+import yaml
+from jinja2 import Template
+
+Database = Dict[str, dict]
+
+# Files
+HERE = Path(__file__).parent
+DB_FILE = HERE / 'methods.yml'
+DOC_FILE = HERE.parent / 'README.md'
+TPL_FILE = HERE / 'installation.jinja2'
+
+# Database keys
+KEY_DOC_STRUCTURE = 'docs-structure'
+KEY_TOOLS = 'tools'
+
+# Markers in-between content will be put.
+MARKER_START = '<div data-installation-instructions>'
+MARKER_END = '</div>'
+
+
+def generate_documentation() -> str:
+    database = load_database()
+    structure = build_docs_structure(database)
+    template = Template(source=TPL_FILE.read_text(encoding='utf-8'))
+    output = template.render(structure=structure)
+    output = clean_template_output(output)
+    return output
+
+
+def save_doc_file(content: str) -> None:
+    current_doc = load_doc_file()
+    marker_start = current_doc.find(MARKER_START) + len(MARKER_START)
+    assert marker_start > 0, 'cannot find the start marker'
+    marker_end = current_doc.find(MARKER_END, marker_start)
+    assert marker_start < marker_end, f'{marker_end=} < {marker_start=}'
+    updated_doc = (
+        current_doc[:marker_start]
+        + '\n\n'
+        + content
+        + '\n\n'
+        + current_doc[marker_end:]
+    )
+    if current_doc != updated_doc:
+        DOC_FILE.write_text(updated_doc, encoding='utf-8')
+
+
+def build_docs_structure(database: Database):
+    tools = database[KEY_TOOLS]
+    assert len(tools) == len({tool['title'] for tool in tools.values()}), 'tool titles need to be unique'
+    tree = database[KEY_DOC_STRUCTURE]
+    structure = []
+    for platform, tools_ids in tree.items():
+        assert platform.isalnum(), f'{platform=} must be alpha-numeric for generated links to work'
+        platform_tools = [tools[tool_id] for tool_id in tools_ids]
+        structure.append((platform, platform_tools))
+    return structure
+
+
+def clean_template_output(output):
+    output = '\n'.join(line.strip() for line in output.strip().splitlines())
+    output = re.sub('\n{3,}', '\n\n', output)
+    return output
+
+
+def load_database() -> Database:
+    return yaml.safe_load(DB_FILE.read_text(encoding='utf-8'))
+
+
+def load_doc_file() -> str:
+    return DOC_FILE.read_text(encoding='utf-8')
+
+
+def main() -> int:
+    content = generate_documentation()
+    save_doc_file(content)
+    return 0
+
+
+if __name__ == '__main__':
+    sys.exit(main())

docs/installation/installation.jinja2

@@ -0,0 +1,37 @@
+<!--
+THE INSTALLATION SECTION IS GENERATED
+
+Do not edit here, but in docs/installation/.
+
+-->
+{% for platform, tools in structure %}
+    - [{{ platform }}](#{{ platform.lower() }}){% endfor %} {# <= keep `endfor` here to prevent unwanted `\n` #}
+
+{% for platform, tools in structure %}
+
+    ### {{ platform }}
+
+    {% for tool in tools %}
+        #### {{ tool.title }}
+
+        {% if tool.note %}
+            {{ tool.note }}
+        {% endif %}
+
+        {% if tool.links.setup %}
+            To install [{{ tool.name }}]({{ tool.links.homepage }}) follow [installation instructions]({{ tool.links.setup }}).
+        {% endif %}
+
+        ```bash
+        # Install
+        $ {{ tool.commands.install|join('\n$ ') }}
+        ```
+
+        ```bash
+        # Upgrade
+        $ {{ tool.commands.upgrade|join('\n$ ') }}
+        ```
+    {% endfor %}
+
+{% endfor %}
+<!-- /GENERATED SECTION -->

docs/installation/methods.yml

@@ -0,0 +1,269 @@
+# Database of HTTPie installation methods. Used to build the docs.
+#
+# We currently only include here methods for popular systems where we take care of the package,
+# or have a good relationship with the maintainers.
+#
+# Each tool name should be unique (it becomes a linkable header).
+# If a tools have `links.setup`, it also needs `links.homepage`.
+# Some tools are available on multiple platforms, take into account when editing.
+#
+
+docs-structure:
+  Universal:
+    - pypi
+  macOS:
+    - brew-mac
+    - port
+    - snap-mac
+    - spack-mac
+  Windows:
+    - chocolatey
+  Linux:
+    - snap-linux
+    - brew-linux
+    - apt
+    - dnf
+    - yum
+    - apk
+    - emerge
+    - pacman
+    - xbps-install
+    - spack-linux
+  FreeBSD:
+    - pkg
+
+tools:
+  apk:
+    title: Alpine Linux
+    name: apk
+    links:
+      homepage: https://wiki.alpinelinux.org/wiki/Alpine_Linux_package_management
+      package: https://pkgs.alpinelinux.org/package/edge/community/x86/httpie
+    commands:
+      install:
+        - apk update
+        - apk add httpie
+      upgrade:
+        - apk update
+        - apk add --upgrade httpie
+
+  apt:
+    title: Debian and Ubuntu
+    note: Also works for other Debian-derived distributions like MX Linux, Linux Mint, deepin, Pop!_OS, KDE neon, Zorin OS, elementary OS, Kubuntu, Devuan, Linux Lite, Peppermint OS, Lubuntu, antiX, Xubuntu, etc.
+    name: APT
+    links:
+      homepage: https://en.wikipedia.org/wiki/APT_(software)
+      package: https://packages.debian.org/sid/web/httpie
+    commands:
+      install:
+        - apt update
+        - apt install httpie
+      upgrade:
+        - apt update
+        - apt upgrade httpie
+
+  brew-mac:
+    title: Homebrew
+    name: Homebrew
+    links:
+      homepage: https://brew.sh/
+      setup: https://docs.brew.sh/Installation
+      package: https://formulae.brew.sh/formula/httpie
+    commands:
+      install:
+        - brew update
+        - brew install httpie
+      upgrade:
+        - brew update
+        - brew upgrade httpie
+
+  brew-linux:
+    title: Linuxbrew
+    name: Linuxbrew
+    links:
+      homepage: https://docs.brew.sh/Homebrew-on-Linux
+      setup: https://docs.brew.sh/Homebrew-on-Linux#install
+      package: https://formulae.brew.sh/formula/httpie
+    commands:
+      install:
+        - brew update
+        - brew install httpie
+      upgrade:
+        - brew update
+        - brew upgrade httpie
+
+  chocolatey:
+    title: Chocolatey
+    name: Chocolatey
+    links:
+      homepage: https://chocolatey.org/
+      setup: https://chocolatey.org/install
+      package: https://community.chocolatey.org/packages/httpie/
+    commands:
+      install:
+        - choco install httpie
+      upgrade:
+        - choco upgrade httpie
+
+  dnf:
+    title: Fedora
+    name: DNF
+    links:
+      homepage: https://fedoraproject.org/wiki/DNF
+      package: https://src.fedoraproject.org/rpms/httpie
+    commands:
+      install:
+        - dnf update
+        - dnf install httpie
+      upgrade:
+        - dnf update
+        - dnf upgrade httpie
+
+  emerge:
+    title: Gentoo
+    name: Portage
+    links:
+      homepage: https://wiki.gentoo.org/wiki/Portage
+      package: https://packages.gentoo.org/packages/net-misc/httpie
+    commands:
+      install:
+        - emerge --sync
+        - emerge httpie
+      upgrade:
+        - emerge --sync
+        - emerge --update httpie
+
+  pacman:
+    title: Arch Linux
+    name: pacman
+    note: Also works for other Arch-derived distributions like ArcoLinux, EndeavourOS, Artix Linux, etc.
+    links:
+      homepage: https://archlinux.org/pacman/
+      package: https://archlinux.org/packages/community/any/httpie/
+    commands:
+      install:
+        - pacman -Sy httpie
+      upgrade:
+        - pacman -Syu httpie
+
+  pkg:
+    title: FreshPorts
+    name: FreshPorts
+    links:
+      homepage: https://www.freebsd.org/cgi/man.cgi?query=pkg&sektion=8&n=1
+      package: https://www.freshports.org/www/py-httpie/
+    commands:
+      install:
+        - pkg install www/py-httpie
+      upgrade:
+        - pkg upgrade www/py-httpie
+
+  port:
+    title: MacPorts
+    name: MacPorts
+    links:
+      homepage: https://www.macports.org/
+      setup: https://www.macports.org/install.php
+      package: https://ports.macports.org/port/httpie/
+    commands:
+      install:
+        - port selfupdate
+        - port install httpie
+      upgrade:
+        - port selfupdate
+        - port upgrade httpie
+
+  pypi:
+    title: PyPi
+    name: pip
+    note: Please make sure you have Python 3.6 or newer (`python --version`).
+    links:
+      homepage: https://pypi.org/
+      # setup: https://pip.pypa.io/en/stable/installation/
+      package: https://pypi.org/project/httpie/
+    commands:
+      install:
+        - python -m pip install --upgrade pip wheel
+        - python -m pip install httpie
+      upgrade:
+        - python -m pip install --upgrade pip wheel
+        - python -m pip install --upgrade httpie
+
+  snap-linux:
+    title: Snapcraft (Linux)
+    name: Snapcraft
+    links:
+      homepage: https://snapcraft.io/
+      setup: https://snapcraft.io/docs/installing-snapd
+      package: https://snapcraft.io/httpie
+    commands:
+      install:
+        - snap install httpie
+      upgrade:
+        - snap refresh httpie
+
+  snap-mac:
+    title: Snapcraft (macOS)
+    name: Snapcraft
+    links:
+      homepage: https://snapcraft.io/
+      setup: https://snapcraft.io/docs/installing-snapd
+      package: https://snapcraft.io/httpie
+    commands:
+      install:
+        - snap install httpie
+      upgrade:
+        - snap refresh httpie
+
+  spack-linux:
+    title: Spack (Linux)
+    name: Spack
+    links:
+      homepage: https://spack.readthedocs.io/en/latest/index.html
+      setup: https://spack.readthedocs.io/en/latest/getting_started.html#installation
+    commands:
+      install:
+        - spack install httpie
+      upgrade:
+        - spack install httpie
+
+  spack-mac:
+    title: Spack (macOS)
+    name: Spack
+    links:
+      homepage: https://spack.readthedocs.io/en/latest/index.html
+      setup: https://spack.readthedocs.io/en/latest/getting_started.html#installation
+    commands:
+      install:
+        - spack install httpie
+      upgrade:
+        - spack install httpie
+
+  xbps-install:
+    title: Void Linux
+    name: XBPS
+    links:
+      homepage: https://docs.voidlinux.org/xbps/index.html
+    commands:
+      install:
+        - xbps-install -Su
+        - xbps-install -S httpie
+      upgrade:
+        - xbps-install -Su
+        - xbps-install -Su httpie
+
+  yum:
+    title: CentOS and RHEL
+    name: Yum
+    note: Also works for other RHEL-derived distributions like ClearOS, Oracle Linux, etc.
+    links:
+      homepage: http://yum.baseurl.org/
+      package: https://src.fedoraproject.org/rpms/httpie
+    commands:
+      install:
+        - yum update
+        - yum install epel-release
+        - yum install httpie
+      upgrade:
+        - yum update
+        - yum upgrade httpie

docs/markdownlint.rb

@@ -1,3 +1,5 @@
+# Rules for <https://github.com/markdownlint/markdownlint>
+
 # Load all rules by default
 all
 

setup.py

@@ -22,8 +22,10 @@ dev_require = [
     'flake8-tuple',
     'pyopenssl',
     'pytest-cov',
+    'pyyaml',
     'twine',
     'wheel',
+    'Jinja2'
 ]
 install_requires = [
     'charset_normalizer>=2.0.0',