Thanks for reporting this problem. I could replicate this locally.
It turns out that if the resulting members list is supposed to be empty after operation completion, the changes are not being saved to LDAP. That's why you were only able to do partial removals.
I opened an [issue](https://github.com/GluuFederation/oxTrust/issues/1372) for it. I think it's solved now but it will be closed upon further testing. The fix will be part of 3.1.5 release.
By now you can apply the hot fix yourself (you can ask how), or deal with the inability of having an empty group, say by having a dummy user part of your groups.