Yes, doing regular expressions can be tricky anyway. Doing so inside tagger script can become really tricky due to the double escaping needed.
Try this script:
$set(title,$rreplace(%title%,\\\([^\)]*\\\)\$,))
This will only replace anything “(…)” at the end of the title, not something in between. If you want this differently, remove the \$ at the end of the search expression.
To break this down a bit more, the actual regular expression here is \([^)]*\)$. The braces need escaping with a backslash, as they are special characters for regular expressions. See regexr.com/8l28d for a demonstration of this regex.
But when using this inside a tagger script, you need to escape \, (, ) and $ all with a backslash again, as they are special characters in tagger script. So you end up with \\\([^\)]*\\\)\$.