release v0.1.10

Without this, I get errors when running "podman-compose -p podname run".

README.md

@@ -1,10 +1,13 @@
 # Podman Compose
 
-An implementation of [Compose Spec](https://compose-spec.io/) with [Podman](https://podman.io/) backend.
+An implementation of `docker-compose` with [Podman](https://podman.io/) backend.
-This project focus on:
+The main objective of this project is to be able to run `docker-compose.yml` unmodified and rootless.
+This project is aimed to provide drop-in replacement for `docker-compose`,
+and it's very useful for certain cases because:
 
-* rootless
+- can run rootless
-* daemon-less process model, we directly execute podman, no running daemon.
+- no daemon, no setup.
+- can be used by developers to run single-machine containerized stacks using single familiar YAML file
 
 This project only depend on:
 
@@ -15,23 +18,16 @@ This project only depend on:
 
 And it's formed as a single python file script that you can drop into your PATH and run.
 
-## References:
-
-* [spec.md](https://github.com/compose-spec/compose-spec/blob/master/spec.md)
-* [docker-compose compose-file-v3](https://docs.docker.com/compose/compose-file/compose-file-v3/)
-* [docker-compose compose-file-v2](https://docs.docker.com/compose/compose-file/compose-file-v2/)
-
-## Alternatives
-
-As in [this article](https://fedoramagazine.org/use-docker-compose-with-podman-to-orchestrate-containers-on-fedora/) you can setup a `podman.socket` and use unmodified `docker-compose` that talks to that socket but in this case you lose the process-model (ex. `docker-compose build` will send a possibly large context tarball to the daemon)
 
 For production-like single-machine containerized environment consider
 
 - [k3s](https://k3s.io) | [k3s github](https://github.com/rancher/k3s)
 - [MiniKube](https://minikube.sigs.k8s.io/)
+- [MiniShift](https://www.okd.io/minishift/)
+
 
 For the real thing (multi-node clusters) check any production
-OpenShift/Kubernetes distribution like [OKD](https://www.okd.io/).
+OpenShift/Kubernetes distribution like [OKD](https://www.okd.io/minishift/).
 
 ## Versions
 
@@ -63,7 +59,7 @@ curl -o /usr/local/bin/podman-compose https://raw.githubusercontent.com/containe
 chmod +x /usr/local/bin/podman-compose
 ```
 
-or inside your home
+or 
 
 ```
 curl -o ~/.local/bin/podman-compose https://raw.githubusercontent.com/containers/podman-compose/devel/podman_compose.py
@@ -107,4 +103,11 @@ When testing the `AWX3` example, if you got errors just wait for db migrations t
 Inside `tests/` directory we have many useless docker-compose stacks
 that are meant to test as much cases as we can to make sure we are compatible
 
+## How it works
+
+The default mapping `1podfw` creates a single pod and attach all containers to
+its network namespace so that all containers talk via localhost.
+For more information see [docs/Mappings.md](docs/Mappings.md).
+
+If you are running as root, you might use identity mapping.
 

podman_compose.py

@@ -35,7 +35,7 @@ except ImportError:
 import yaml
 from dotenv import dotenv_values
 
-__version__ = '1.0.2'
+__version__ = '0.1.10'
 
 PY3 = sys.version_info[0] == 3
 if PY3:
@@ -256,21 +256,149 @@ def norm_ulimit(inner_value):
     # if int or string return as is
     return inner_value
 
-#def tr_identity(project_name, given_containers):
-#    pod_name = f'pod_{project_name}'
-#    pod = dict(name=pod_name)
-#    containers = []
-#    for cnt in given_containers:
-#        containers.append(dict(cnt, pod=pod_name))
-#    return [pod], containers
 
-def tr_identity(project_name, given_containers):
+# transformation helpers
+
+def adj_hosts(services, cnt, dst="127.0.0.1"):
+    """
+    adjust container cnt in-place to add hosts pointing to dst for services
+    """
+    common_extra_hosts = []
+    for srv, cnts in services.items():
+        common_extra_hosts.append("{}:{}".format(srv, dst))
+        for cnt0 in cnts:
+            common_extra_hosts.append("{}:{}".format(cnt0, dst))
+    extra_hosts = list(cnt.get("extra_hosts", []))
+    extra_hosts.extend(common_extra_hosts)
+    # link aliases
+    for link in cnt.get("links", []):
+        a = link.strip().split(':', 1)
+        if len(a) == 2:
+            alias = a[1].strip()
+            extra_hosts.append("{}:{}".format(alias, dst))
+    cnt["extra_hosts"] = extra_hosts
+
+
+def move_list(dst, containers, key):
+    """
+    move key (like port forwarding) from containers to dst (a pod or a infra container)
+    """
+    a = set(dst.get(key, None) or [])
+    for cnt in containers:
+        a0 = cnt.get(key, None)
+        if a0:
+            a.update(a0)
+            del cnt[key]
+    if a:
+        dst[key] = list(a)
+
+
+def move_port_fw(dst, containers):
+    """
+    move port forwarding from containers to dst (a pod or a infra container)
+    """
+    move_list(dst, containers, "ports")
+
+
+def move_extra_hosts(dst, containers):
+    """
+    move port forwarding from containers to dst (a pod or a infra container)
+    """
+    move_list(dst, containers, "extra_hosts")
+
+
+# transformations
+
+transformations = {}
+
+
+def trans(func):
+    transformations[func.__name__.replace("tr_", "")] = func
+    return func
+
+
+@trans
+def tr_identity(project_name, services, given_containers):
     containers = []
     for cnt in given_containers:
         containers.append(dict(cnt))
     return [], containers
 
 
+@trans
+def tr_publishall(project_name, services, given_containers):
+    containers = []
+    for cnt0 in given_containers:
+        cnt = dict(cnt0, publishall=True)
+        # adjust hosts to point to the gateway, TODO: adjust host env
+        adj_hosts(services, cnt, '10.0.2.2')
+        containers.append(cnt)
+    return [], containers
+
+
+@trans
+def tr_hostnet(project_name, services, given_containers):
+    containers = []
+    for cnt0 in given_containers:
+        cnt = dict(cnt0, network_mode="host")
+        # adjust hosts to point to localhost, TODO: adjust host env
+        adj_hosts(services, cnt, '127.0.0.1')
+        containers.append(cnt)
+    return [], containers
+
+
+@trans
+def tr_cntnet(project_name, services, given_containers):
+    containers = []
+    infra_name = project_name + "_infra"
+    infra = dict(
+        name=infra_name,
+        image="k8s.gcr.io/pause:3.1",
+        _service=None,
+        service_name=None
+    )
+    for cnt0 in given_containers:
+        cnt = dict(cnt0, network_mode="container:"+infra_name)
+        deps = cnt.get("depends_on", None) or []
+        deps.append(infra_name)
+        cnt["depends_on"] = deps
+        # adjust hosts to point to localhost, TODO: adjust host env
+        adj_hosts(services, cnt, '127.0.0.1')
+        if "hostname" in cnt:
+            del cnt["hostname"]
+        containers.append(cnt)
+    move_port_fw(infra, containers)
+    move_extra_hosts(infra, containers)
+    containers.insert(0, infra)
+    return [], containers
+
+
+@trans
+def tr_1pod(project_name, services, given_containers):
+    """
+    project_name:
+    services: {service_name: ["container_name1", "..."]}, currently only one is supported
+    given_containers: [{}, ...]
+    """
+    pod = dict(name=project_name)
+    containers = []
+    for cnt0 in given_containers:
+        cnt = dict(cnt0, pod=project_name)
+        # services can be accessed as localhost because they are on one pod
+        # adjust hosts to point to localhost, TODO: adjust host env
+        adj_hosts(services, cnt, '127.0.0.1')
+        containers.append(cnt)
+    return [pod], containers
+
+
+@trans
+def tr_1podfw(project_name, services, given_containers):
+    pods, containers = tr_1pod(project_name, services, given_containers)
+    pod = pods[0]
+    move_port_fw(pod, containers)
+    return pods, containers
+
+
 def assert_volume(compose, mount_dict):
     """
     inspect volume to get directory
@@ -544,47 +672,6 @@ def norm_ports(ports_in):
         ports_out.append(port)
     return ports_out
 
-def assert_cnt_nets(compose, cnt):
-    """
-    create missing networks
-    """
-    proj_name = compose.project_name
-    nets = compose.networks
-    default_net = compose.default_net
-    cnt_nets = norm_as_list(cnt.get("networks", None) or default_net)
-    for net in cnt_nets:
-        net_desc = nets[net] or {}
-        net_name = net_desc.get("name", None) or f"{proj_name}_{net}"
-        print(f"podman network exists '{net_name}' || podman network create '{net_name}'")
-        try: compose.podman.output([], "network", ["exists", net_name])
-        except subprocess.CalledProcessError:
-            args = [
-                "create",
-                "--label", "io.podman.compose.project={}".format(proj_name),
-                "--label", "com.docker.compose.project={}".format(proj_name),
-            ]
-            # TODO: add more options here, like driver, internal, ..etc
-            labels = net_desc.get("labels", None) or []
-            for item in norm_as_list(labels):
-                args.extend(["--label", item])
-            args.append(net_name)
-            compose.podman.output([], "network", args)
-            compose.podman.output([], "network", ["exists", net_name])
-
-def get_net_args(compose, cnt):
-    service_name = cnt["service_name"]
-    project_name = compose.project_name
-    default_net = compose.default_net
-    nets = compose.networks
-    cnt_nets = norm_as_list(cnt.get("networks", None) or default_net)
-    net_names = set()
-    for net in cnt_nets:
-        net_desc = nets[net] or {}
-        net_name = net_desc.get("name", None) or f"{project_name}_{net}"
-        net_names.add(net_name)
-    net_names_str = ",".join(net_names)
-    return ["--net", net_names_str, "--network-alias", service_name]
-
 def container_to_args(compose, cnt, detached=True):
     # TODO: double check -e , --add-host, -v, --read-only
     dirname = compose.dirname
@@ -631,8 +718,6 @@ def container_to_args(compose, cnt, detached=True):
         podman_args.extend(['--tmpfs', i])
     for volume in cnt.get('volumes', []):
         podman_args.extend(get_mount_args(compose, cnt, volume))
-    assert_cnt_nets(compose, cnt)
-    podman_args.extend(get_net_args(compose, cnt))
     log = cnt.get('logging')
     if log is not None:
         podman_args.append(f'--log-driver={log.get("driver", "k8s-file")}')
@@ -655,7 +740,7 @@ def container_to_args(compose, cnt, detached=True):
         elif not isinstance(port, str):
             raise TypeError("port should be either string or dict")
         podman_args.extend(['-p', port])
-
+        
     user = cnt.get('user', None)
     if user is not None:
         podman_args.extend(['-u', user])
@@ -962,8 +1047,6 @@ class PodmanCompose:
         self.pods = None
         self.containers = None
         self.vols = None
-        self.networks = {}
-        self.default_net = "default"
         self.declared_secrets = None
         self.container_names_by_service = None
         self.container_by_name = None
@@ -1036,6 +1119,7 @@ class PodmanCompose:
         no_ansi = args.no_ansi
         no_cleanup = args.no_cleanup
         dry_run = args.dry_run
+        transform_policy = args.transform_policy
         host_env = None
         dirname = os.path.realpath(os.path.dirname(filename))
         dir_basename = os.path.basename(dirname)
@@ -1091,26 +1175,6 @@ class PodmanCompose:
         flat_deps(services)
         service_names = sorted([ (len(srv["_deps"]), name) for name, srv in services.items() ])
         service_names = [ name for _, name in service_names]
-        nets = compose.get("networks", None) or {}
-        if not nets:
-            nets["default"] = None
-        self.networks = nets
-        if len(self.networks)==1:
-            self.default_net = list(nets.keys())[0]
-        elif "default" in nets:
-            self.default_net = "default"
-        else:
-            self.default_net = None
-        default_net = self.default_net
-        allnets = set()
-        for name, srv in services.items():
-            srv_nets = norm_as_list(srv.get("networks", None) or default_net)
-            allnets.update(srv_nets)
-        given_nets = set(nets.keys())
-        missing_nets = given_nets - allnets
-        if len(missing_nets):
-            missing_nets_str= ",".join(missing_nets)
-            raise RuntimeError(f"missing networks: {missing_nets_str}")
         # volumes: [...]
         self.vols = compose.get('volumes', {})
         podman_compose_labels = [
@@ -1167,11 +1231,14 @@ class PodmanCompose:
         given_containers = list(container_by_name.values())
         given_containers.sort(key=lambda c: len(c.get('_deps', None) or []))
         #print("sorted:", [c["name"] for c in given_containers])
-        pods, containers = tr_identity(project_name, given_containers)
+        tr = transformations[transform_policy]
+        pods, containers = tr(
+            project_name, container_names_by_service, given_containers)
         self.pods = pods
         self.containers = containers
         self.container_by_name = dict([ (c["name"], c) for c in containers])
 
+
     def _parse_args(self):
         parser = argparse.ArgumentParser(
             formatter_class=argparse.RawTextHelpFormatter
@@ -1216,6 +1283,17 @@ class PodmanCompose:
                             help="Do not stop and remove existing pod & containers", action='store_true')
         parser.add_argument("--dry-run",
                             help="No action; perform a simulation of commands", action='store_true')
+        parser.add_argument("-t", "--transform_policy",
+                            help=textwrap.dedent("""\
+                            how to translate docker compose to podman (default: 1podfw)
+                                1podfw - create all containers in one pod (inter-container communication is done via localhost), doing port mapping in that pod
+                                1pod - create all containers in one pod, doing port mapping in each container (does not work)
+                                identity - no mapping
+                                hostnet - use host network, and inter-container communication is done via host gateway and published ports
+                                cntnet - create a container and use it via --network container:name (inter-container communication via localhost)
+                                publishall - publish all ports to host (using -P) and communicate via gateway
+                            """),
+                            choices=['1pod', '1podfw', 'hostnet', 'cntnet', 'publishall', 'identity'], default='1podfw')
 
 podman_compose = PodmanCompose()
 
@@ -1344,9 +1422,10 @@ def create_pods(compose, args):
         podman_args = [
             "create",
             "--name={}".format(pod["name"]),
+            "--share", "net",
         ]
-        #if compose.podman_version and not strverscmp_lt(compose.podman_version, "3.4.0"):
+        if compose.podman_version and not strverscmp_lt(compose.podman_version, "3.4.0"):
-        #    podman_args.append("--infra-name={}_infra".format(pod["name"]))
+            podman_args.append("--infra-name={}_infra".format(pod["name"]))
         ports = pod.get("ports", None) or []
         if isinstance(ports, str):
             ports = [ports]

tests/multicompose/d1/docker-compose.yml

@@ -4,7 +4,7 @@ services:
     image: busybox
     command: busybox httpd -h /var/www/html/ -f -p 8001
     volumes:
-      - ./1.env:/var/www/html/index.txt:z
+    - ./1.env:/var/www/html/index.txt
     env_file: ./1.env
     labels:
       l1: v1

tests/multicompose/d2/docker-compose.yml

@@ -1,11 +1,10 @@
 version: '3'
 services:
   web1:
-    image: busybox
     env_file: ./12.env
     labels:
-      - l1=v2
+    - l1=v2
-      - l2=v2
+    - l2=v2
     environment:
         mykey1: myval2
         mykey2: myval2
@@ -14,6 +13,6 @@ services:
     image: busybox
     command: busybox httpd -h /var/www/html/ -f -p 8002
     volumes:
-      - ./2.env:/var/www/html/index.txt:z
+    - ./2.env:/var/www/html/index.txt
     env_file: ./2.env
 

tests/nets_test1/docker-compose.yml

@@ -1,21 +0,0 @@
-version: "3"
-services:
-    web1:
-      image: busybox
-      hostname: web1
-      command: ["/bin/busybox", "httpd", "-f", "-h", "/var/www/html", "-p", "8001"]
-      working_dir: /var/www/html
-      ports:
-        - 8001:8001
-      volumes:
-        - ./test1.txt:/var/www/html/index.txt:ro,z
-    web2:
-      image: busybox
-      hostname: web1
-      command: ["/bin/busybox", "httpd", "-f", "-h", "/var/www/html", "-p", "8001"]
-      working_dir: /var/www/html
-      ports:
-        - 8002:8001
-      volumes:
-        - ./test2.txt:/var/www/html/index.txt:ro,z
-

tests/nets_test1/test1.txt

@@ -1 +0,0 @@
-test1

tests/nets_test1/test2.txt

@@ -1 +0,0 @@
-test2

tests/nets_test2/docker-compose.yml

@@ -1,23 +0,0 @@
-version: "3"
-networks:
-    mystack:
-services:
-    web1:
-      image: busybox
-      hostname: web1
-      command: ["/bin/busybox", "httpd", "-f", "-h", "/var/www/html", "-p", "8001"]
-      working_dir: /var/www/html
-      ports:
-        - 8001:8001
-      volumes:
-        - ./test1.txt:/var/www/html/index.txt:ro,z
-    web2:
-      image: busybox
-      hostname: web2
-      command: ["/bin/busybox", "httpd", "-f", "-h", "/var/www/html", "-p", "8001"]
-      working_dir: /var/www/html
-      ports:
-        - 8002:8001
-      volumes:
-        - ./test2.txt:/var/www/html/index.txt:ro,z
-

tests/nets_test2/test1.txt

@@ -1 +0,0 @@
-test1

tests/nets_test2/test2.txt

@@ -1 +0,0 @@
-test2

tests/nets_test3/docker-compose.yml

@@ -1,31 +0,0 @@
-version: "3"
-networks:
-    net1:
-    net2:
-services:
-    web1:
-      image: busybox
-      #container_name: web1
-      hostname: web1
-      command: ["/bin/busybox", "httpd", "-f", "-h", "/var/www/html", "-p", "8001"]
-      working_dir: /var/www/html
-      networks:
-      - net1
-      ports:
-        - 8001:8001
-      volumes:
-        - ./test1.txt:/var/www/html/index.txt:ro,z
-    web2:
-      image: busybox
-      #container_name: web2
-      hostname: web2
-      command: ["/bin/busybox", "httpd", "-f", "-h", "/var/www/html", "-p", "8001"]
-      working_dir: /var/www/html
-      networks:
-      - net1
-      - net2
-      ports:
-        - 8002:8001
-      volumes:
-        - ./test2.txt:/var/www/html/index.txt:ro,z
-

tests/nets_test3/test1.txt

@@ -1 +0,0 @@
-test1

tests/nets_test3/test2.txt

@@ -1 +0,0 @@
-test2

tests/ports/docker-compose.yml

@@ -2,34 +2,32 @@ version: "3"
 services:
     web1:
       image: busybox
-      hostname: web1
       command: ["/bin/busybox", "httpd", "-f", "-h", "/var/www/html", "-p", "8001"]
       working_dir: /var/www/html
       ports:
-        - 8001:8001
+      - 8001:8001
       volumes:
-        - ./test1.txt:/var/www/html/index.txt:ro,z
+      - ./test1.txt:/var/www/html/index.txt:ro
     web2:
       image: busybox
-      hostname: web2
       command: ["/bin/busybox", "httpd", "-f", "-h", "/var/www/html", "-p", "8002"]
       working_dir: /var/www/html
       ports:
-        - 8002:8002
+      - 8002:8002
-        - target: 8003
+      - target: 8003
-          host_ip: 127.0.0.1
+        host_ip: 127.0.0.1
-          published: 8003
+        published: 8003
-          protocol: udp
+        protocol: udp
-        - target: 8004
+      - target: 8004
-          host_ip: 127.0.0.1
+        host_ip: 127.0.0.1
-          published: 8004
+        published: 8004
-          protocol: tcp
+        protocol: tcp
-        - target: 8005
+      - target: 8005
-          published: 8005
+        published: 8005
-        - target: 8006
+      - target: 8006
-          protocol: udp
+        protocol: udp
-        - target: 8007
+      - target: 8007
-          host_ip: 127.0.0.1
+        host_ip: 127.0.0.1
       volumes:
-        - ./test2.txt:/var/www/html/index.txt:ro,z
+      - ./test2.txt:/var/www/html/index.txt:ro
 

tests/secrets/docker-compose.yaml

@@ -8,7 +8,7 @@ services:
         - /run
         - /tmp
       volumes:
-        - ./print_secrets.sh:/tmp/print_secrets.sh:z
+        - ./print_secrets.sh:/tmp/print_secrets.sh
       secrets:
         - my_secret
         - my_secret_2

tests/short/docker-compose.yaml

@@ -4,7 +4,7 @@ services:
       image: redis:alpine
       command: ["redis-server", "--appendonly yes", "--notify-keyspace-events", "Ex"]
       volumes:
-        - ./data/redis:/data:z
+        - ./data/redis:/data
       tmpfs: /run1
       ports:
         - "6379"
@@ -25,16 +25,16 @@ services:
       command: ["/bin/busybox", "httpd", "-f", "-h", "/var/www/html", "-p", "8001"]
       working_dir: /var/www/html
       volumes:
-        - ./data/web:/var/www/html:ro,z
+        - ./data/web:/var/www/html:ro
     web2:
       image: busybox
       command: ["/bin/busybox", "httpd", "-f", "-h", "/var/www/html", "-p", "8002"]
       working_dir: /var/www/html
       volumes:
-        - ~/Downloads/www:/var/www/html:ro,z
+        - ~/Downloads/www:/var/www/html:ro
     web3:
       image: busybox
       command: ["/bin/busybox", "httpd", "-f", "-h", "/var/www/html", "-p", "8003"]
       working_dir: /var/www/html
       volumes:
-        - /var/www/html:/var/www/html:ro,z
+        - /var/www/html:/var/www/html:ro

tests/vol/docker-compose.yaml

@@ -14,7 +14,7 @@ services:
       command: ["/bin/busybox", "httpd", "-f", "-h", "/var/www/html", "-p", "8001"]
       working_dir: /var/www/html
       volumes:
-        - myvol1:/var/www/html:ro,z
+        - myvol1:/var/www/html:ro
     web2:
       image: busybox
       command: ["/bin/busybox", "httpd", "-f", "-h", "/var/www/html", "-p", "8002"]