Gluu has been verified to be able to handle hundreds of thousands of user entries, so 11k should be nothing for it. Please try to diagnose it a bit thoroughly, you could set connections to both internal and external directories to use unencrypted channels, and then check with Wireshark what is being sent back and forth.