Ah, yes. I see it now 
The problem are the braces used in the expression. They are special characters in regular expression, and hence have to be escaped. But they also are special characters in tagger script, so this needs another level of escaping.
The regular expression we want is \s+\(.*\)$. But when using this as part of tagger script the backslash, the braces and the dollar sign all are special characters there as well and need another escaping. So we actually get \\s+\\\(.*\\\)\$.
Full script:
%artist% - $strip($rreplace(%title%,\\s+\\\(.*\\\)\$,))
In the original the braces had only a single backslash before them. So this gave you a regular expression of \s+(.*)$, where the braces just form a matching group.
Yeah, combining two separate languages into one can lead to some real mess with escape characters.