I don't recall which plugin, but it said something in console like "Config file saved, reloading" Does anyone know how to replicate this?
There wouldn't be any way to monitor it via a plugin, System.IO is restricted. It is an open issue though to add auto-reloading for config files.