Indra startup checklist

SSH into INDRA (192.168.40.40) as the user isnai. If you SSH with another user, change users with sudo -u isnai bash (assuming you are among the sudoers)

Sanity Checks

After booting check that all GPUs are working with nvtop.

If they are not working or nvtop is not working, please reinstall nvtop, nvidia-smi and nvidia drivers in that order: try after each.

You may have to restart again after the installation.

Restart the LLM servers:

cd /home/isnai/dev/tgi
nohup ./run-qwen32.sh > /home/isnai/qwen32.log 2>&1 &
nohup ./roleplay.sh > /home/isnai/roleplay.log 2>&1 &

cd /home/isnai/dev/sglang/sglang
source env/bin/activate
nohup ./sglangrun.sh > /home/isnai/sglang.log 2>&1 &

Deploy VISNA

cd /home/isnai/testing/visna
source env/bin/activate
export CUDA_VISIBLE_DEVICES=6,7
nohup ./gunicorn_backend.sh > backend.log 2>&1 &
nohup ./gunicorn_frontend.sh > frontend.log 2>&1 &

Troubleshooting

You should be able to see VISNA. If notm here are other possible things you may have to do.

1. Restart nginx with sudo systemctl restart nginx
2. Check if mongodb is running with sudo systemctl status mongod
3. If is not running, run sudo systemctl start mongod