>> We have set default authentication mode to "Default" and default authentication level to "Default".
'Default' is the 'basic' method.
>> But now, when try to connect with admin username, Gluu tries to use the custom authentication script, it failed off course and doesn't try any other authentication mode.
Try keep your 'basic' method enabled.
>> What is the exact configuration to integrate a custom authentication script and still be able to connect with admin user ?
Well, the purpose of custom authentication script is achieve special authentication. We do have few techniques which is helping couple of our customers to use mutiple authentication method ( both basic + custom authn ) but that is in commercial version.
You can try to write your own custom authentication script which will act like:
If ( custom_authn ) success:
login with custom_authn.
else use 'basic method'