Requirements:
- prefer popular and maintained open source products
- a virtual machine manager so everything is abstracted from hardware.
- virtual machine manager should be able to do high availability
- virtual machine manager should be able to do backups with disaster recovery.
- the infrastructure backups should be encrypted and duplicated offsite.
- virtual machine manager should offer a graphical (web) interface to manage its settings.
- (web) applications should be dockerised
- dockers should auto update and restart
- web applications should be available over https/TLS only.
- todo: virtual private gateway, online ingress controller for ipv4 and ipv6
software component choices based on requirements:
- vm manger : proxmox https://www.proxmox.com/en/
- docker manager: portainer https://www.portainer.io/
- docker updater: watchtower Watchtower (containrrr.dev)
- ssl terminating reverse proxy: https://nginxproxymanager.com/
- oracle cloud offers 100GB free-tier storage for offsite backups
- optional: OC could possibly also run the ingress gateway to shield the main host and route ipv6 and manage DDOS. or maybe use cloudflare. Not sure about the implications for the dependency this creates.
design choices
considerations (proxmox):
- to keep the proxmox host as clean and reproducable as possible, we dont want to run any additional software directly on the portainer host machine.
- proxmox is able to run either LXC containers or fully virtualised machines using KVM.
- LXC, while more secure then docker by design, it is less popular and it would likely require manual adaptations in the images to use the maintained docker software stacks on dockerhub and the likes.
- while docker within LXC is possible, it is would be using the shared kernel from the host machine, so this is not adequate from an security/isolation perspective.
- for this reason proxmox advises to run docker within a fully virtualised machine.
- docker still requires root access. While there is non-root docker, its likely that some containers expect the privileged environment.
- to keep the proxmox host as clean and reproducable as possible, we dont want to run any additional software directly on the portainer host machine.
considerations (portainer and docker)
while installing i had a learning curve using the docker networks. for dockers to be able to reach each other they need to be connected to the same network, to be defined in the compose file. It matters if a docker is started from the commandline using compose of started via portainer, they use different defaults for networking. While docker compose on the command line uses a ‘default’ virtual network of the bridge type, if no networks are defined. But portainer creates and uses ‘portainer_default’ as a default virtual bridge. Causing communication problems.
To prevent portainer from prepending ‘portainer_’ for every dockername, volumes and networks, we have to use quotes and define a specific name: for every object where it matters.
Portainer cant manage stacks, edit compose files of dockers that it didnt start itself, so we will bring this to a minimum by only starting portainer itself from the commandline.
flows
software stacking: proxmox –> KVM virtual machine (ubuntu) –> docker and compose –> portainer –> Nginx proxy manager(NPM) and all other dockerised apps.
network flow: internet –> router port forward 80 & 443 onto –> proxmox machine hosting ubuntu hosting docker, hosting NPM –> npm redirects to https and terminates ssl –>then forwards internally over http in the internal docker network using its docker application name
implementation
- install proxmox and create a virtual machine with ubuntu
- configure the router to forward 22 to the virtual machine.
- configure the router to forward 80 and 443 to the virtual machine
- update and install docker and docker-compose using apt
- run portainer using compose:
merijn@ubuntu-server:~$ cat /home/merijn/docker-compose/portainer/compose.yml
version: '3.8'
services:
portainer:
container_name: 'portainer'
image: portainer/portainer-ce:latest
privileged: true
environment:
- VIRTUAL_HOST=portainer.koehoorn.net
- VIRTUAL_PORT=9000
volumes:
- 'data:/data'
- '/var/run/docker.sock:/var/run/docker.sock'
restart: unless-stopped
ports:
- 9000:9000
networks:
- 'dockernet'
volumes:
data:
networks:
dockernet:
driver: bridge
name: 'dockernet'note thath the app is specifically connected to networks: – ‘dockernet’ and this network is also defined at the bottom, along with its quoted name as a bridge-type. the ports section defines an exposed port 9000, it is only handy for the next step and can be removed later.
then just run ‘docker compose up -d’ within the portainer directory.
for now, lets access this portainer instance from your pc outside using ssh tunneling:
ssh <username>@<hostname> -p 22 -L 81:localhost:9000
this would connect your localhost port 81 on the remote vurtual machines’s localhost port 9000. so you can access portainer with a browser to finish the installation, set up a password.
the rest of the applications can be created using portainer, ‘new stack’, but we need one more part, the reverse proxy. create a new stack within portainer and paste this compose file:
version: '3.8'
services:
app:
container_name: 'npm'
image: 'jc21/nginx-proxy-manager:latest'
restart: unless-stopped
environment:
PUID: 1000
PGID: 1000
ports:
- '80:80'
- '81:81'
- '443:443'
networks:
- "dockernet"
volumes:
- data:/data
- letsencrypt:/etc/letsencrypt
volumes:
data:
letsencrypt:
networks:
dockernet:
driver: bridge
name: "dockernet"note that all apps that need reverse proxying will use same ‘dockernet’ network. This compose file exposes a few ports onto the ubuntu host machine, the ports section.
In the DNS server i created a wildcard record * to redirect to my machine. This adds a little security, one needs to know the subdomain to access the web app.
the admin interface for npm is on port 81 so use the same ssl tunneling trick to connect to this port.
add a new proxy host for portainer : portainer.yourdomain.dns to the internal docker dns name http://portainer:9000, along with the option to request new letsencypt ssl cert and force ssl. also the proxy manager to http://npm:81 and maybe also add a proxyhost entry to the proxmox interface via its lan-ip and put some additional security on these interfaces.
NPM also allows you to add access lists for sensitive apps. only allow connections from specific ip’s and/or a basic authentication at the frontend. This is recommended for administrative apps ofcourse, to only allow connections from within the docker network, while connected with a VPN container for example.
todo:
- centralised sys/access/error logging
- log analytics and alerts
- kicktoban and/or ip blacklists
- a vpn container
- proxmox clustering, HA.