There is a BR! provided GUIMode environmental variable (case insensitive). It's value is either ON or OFF depending on whether BR is in GUI mode.
x$=env$("GUIMode")
See also:
Env$ GUI Mode