ISN Innovations Stack

ISN Innovations looks after a software development at ISN.

Machines

We have the following machines:

SHIVA - a TrueNAS physical machine:
- physical location: 115 Cotham Rd
- local IP: 192.168.40.20
- Wireguard VPN IP: 10.13.13.8
KRISHNA - a TrueNAS physical machine:
- physical location: 35 Cotham Rd
- local IP: 192.168.20.145
- Wireguard VPN IP: 10.13.13.9
INDRA - a physical machine:
- physical location: 115 Cotham Rd
- local IP: 192.168.40.40
- Wireguard VPN IP: 10.13.13.6
- Has 8 L40S GPUs
KRYPTON - a physical machine:
- physical location: 115 Cotham Rd
- local IP: 192.168.40.60
- Wireguard VPN IP: 10.13.13.3
- Has 4 L40S GPUs
RAMA - a GCP virtual machine
- GCP IP: 10.152.0.3
- public IP: 34.116.65.212
- Wireguard VPN IP: 10.13.13.7
GANESHA a GCP virtual machine:
- GCP IP: 10.152.0.2
- public IP: 34.40.255.123
- Wireguard VPN IP: 10.13.13.1

How to Connect

PEER=nathan
ssh ganesha "docker exec wireguard /app/show-peer $PEER"
ssh ganesha "sudo cat /var/lib/docker/volumes/infrastructure-setup_wireguard_config/_data/peer_$PEER/peer_$PEER.conf"

$PEER = "nathan"
ssh ganesha "docker exec wireguard /app/show-peer $PEER"
ssh ganesha "sudo cat /var/lib/docker/volumes/infrastructure-setup_wireguard_config/_data/peer_$PEER/peer_$PEER.conf"

cmd
set PEER=nathan
ssh ganesha "docker exec wireguard /app/show-peer %PEER%"
ssh ganesha "sudo cat /var/lib/docker/volumes/infrastructure-setup_wireguard_config/_data/peer_%PEER%/peer_%PEER%.conf"

Recommended Developer SSH Configuration

Assuming your ~/.ssh/config includes the entries below, you will be able to simply run:

ssh indra
ssh krypton
ssh rama
ssh ganesha
ssh shiva
ssh krishna
git add remote origin git@10.152.0.2:2222:REPO-OWNER/REPO-NAME.git
git add remote origin git@isn-gitea:REPO-OWNER/REPO-NAME.git

Host indra
    HostName 10.13.13.6
    User YOUR_USER_ON_INDRA
    IdentityFile ~/.ssh/id_ed25519_indra
    IdentitiesOnly yes

Host krypton
    HostName 10.13.13.3
    User YOUR_USER_ON_KRYPTON
    IdentityFile ~/.ssh/id_ed25519_krypton
    IdentitiesOnly yes

Host ganesha
    HostName 10.13.13.1
    User YOUR_USER_ON_GANESHA
    IdentityFile ~/.ssh/id_ed25519_ganesha
    IdentitiesOnly yes

Host rama
    HostName 10.13.13.6
    User YOUR_USER_ON_RAMA
    IdentityFile ~/.ssh/id_ed25519_rama
    IdentitiesOnly yes

Host isn-gitea
    HostName 10.13.13.1
    Port 2222
    User YOUR_USER_ON_GANESHA
    IdentityFile ~/.ssh/id_ed25519_ganesha
    IdentitiesOnly yes

Host shiva
    HostName 10.13.13.8
    User mtamassia
    IdentityFile ~/.ssh/id_ed25519_shiva
    IdentitiesOnly yes

Host krishna
    HostName 10.13.13.9
    User mtamassia
    IdentityFile ~/.ssh/id_ed25519_krishna
    IdentitiesOnly yes

Apps

We look after the following software:

VISNA, an AI chat app for students:
- Backend in Kotlin + Spring Boot 3, frontend in Typescript + Vue 3
- It is deployed on RAMA in Docker containers
- RAMA runs VISNA's staging, sandbox and production environments
- The deployment uses blue-green deployment, with routing handled via a containerized Nginx instance running on GANESHA
- The source code is on this machine at https://gitea.dev.isn.edu.au/ISN-Innovations/Visna
- The pipeline runs on a Gitea instance hosted on GANESHA (origin: ganesha)
- The public URL is https://visna.isn.edu.au, which reaches GANESHA
- A Nginx instance running on GANESHA routes requests to the correct port on RAMA
- It is deployed by simply pushing the branches production, sandbox and staging
THRIVE, a course generation app for staff:
- Backend in Typescript + Express 4, frontend in Typescript + React 19
- It is deployed on RAMA in Docker containers
- RAMA runs THRIVE's staging, sandbox and production environments
- The deployment uses blue-green deployment, with routing handled via a containerized Nginx instance running on GANESHA
- The source code is on this machine at https://gitea.dev.isn.edu.au/ISN-Innovations/Thrive
- The deployment pipeline is in the relative folder .gitea/
- The pipeline runs on a Gitea instance hosted on GANESHA(origin: origin)
- The public URL is https://thrive.isn.edu.au, which reaches GANESHA
- A Nginx instance running on GANESHA routes requests to the correct port on RAMA
- It is deployed by simply pushing the branches production, sandbox and staging
Modelix, an AI gateway app used behind the scenes by THRIVE and soon by VISNA
- Backend in Rust + Axum + Tower, frontend in Typescript + Vue 3
- It is deployed on GANESHA, in a Docker container
- The deployment is configured together with all the other services running on GANESHA (see below)
- The source code is on this machine at https://gitea.dev.isn.edu.au/ISN-Innovations/Modelix
- The deployment pipeline is in the relative folder .gitea/
- The pipeline runs on a Gitea instance hosted on GANESHA (origin: origin)
- This is only reachable using local IPs, so I use a VPN, while THRIVE and VISNA are in the same network
- Pushing the branch main builds an image, stored in Gitea's imnage registry
- Deployment on GANESHA is done via git pull ganesha main && docker-compose pull modelix && docker-compose up -d --force-recreate modelix-caddy

Internal Services

GANESHA runs a host of services:

The services are set up using a long Docker Compose stack defined at https://gitea.dev.isn.edu.au/ISN-Innovations/Infrastructure-Setup
This lives on GANESHA in /Users/mtamassia/Projects/Infrastructure-Setup
The folder containes a .env file, too
GANESHA uses Docker Compose v1, so the right commands start with docker-compose (with a dash)
Among the services, there are:
- Grafana and Kibana, which I use to monitor VISNA and THRIVE.
- Gitea, where we host all of our code, is reachable at https://gitea.dev.isn.edu.au
- Keycloak, which THRIVE and Modelix use for authorisation and identity management, and is reachable at https://iam.isn.edu.au
- Garage, which offers a S3-compatible API, which THRIVE uses to store assets, and is reachable at https://s3.isn.edu.au
- Otterwiki, where we collect our knowledge, is reachable at https://wiki.dev.isn.edu.au
- Prometheus and Elasticsearch, where the data used by Grafana and Kibana transits
- Various other utilities, which we ourseleves use, such as NextCloud, Infisical, Homarr, Portainer.
- Nginx, which is used to route requests to VISNA, THRIVE (on RAMA) but also to Keycloak and Garage
- Modelix, our AI gateway, is reachable at https://ai.isn.edu.au
- Open WebUI, an AI chat app, is reachable at https://chat.staff.isn.edu.au

Setup Client

Build config

ssh ganesha "sudo bash -c 'V=/var/lib/docker/volumes/infrastructure-setup_wireguard_config/_data && printf \"[Interface]\nAddress = 10.13.13.9\nPrivateKey = \$(cat \$V/peer_$PEER/privatekey-peer_$PEER)\n\n[Peer]\nPublicKey = jI0naHt99LeIcbZUBXW4Q25Lhksxsjl7fdoWiOlMXF0=\nPresharedKey = RPEETrKWpDcJsFiMahBYJY5+nOisRavqaNGoaN81xNM=\nEndpoint = 34.40.255.123:51820\nAllowedIPs = 10.152.0.0/24, 10.13.13.0/24\nPersistentKeepalive = 25\n\"'" > /tmp/$PEER_wg0.conf

Copy to /tmp

scp -i ~/.ssh/id_ed25519_$PEER /tmp/$PEER_wg0.conf $SSH_USER@$SSH_HOST:/tmp/wg0.conf && rm /tmp/$PEER_wg0.conf

Install a. Server Setup

ssh -i ~/.ssh/id_ed25519_$PEER -t $SSH_USER@$SSH_HOST "sudo install -m 600 -o root -g root /tmp/wg0.conf /etc/wireguard/wg0.conf && rm /tmp/wg0.conf && sudo systemctl enable --now wg-quick@wg0"

b. Desktop Setup

ssh ganesha "sudo cat /var/lib/docker/volumes/infrastructure-setup_wireguard_config/_data/peer_$PEER/peer_$PEER.conf"

d. Mobile Setup

ssh ganesha "docker exec wireguard /app/show-peer $PEER"