>> in case, user directly tries to login to gluu(idp) instead of coming via SP. Meaning, if User doesn't come via SP and directly comes to gluu url then, after authentication, i want user to be redirected to some custom url(SP), instead of showing gluu home page.
Not possible. If user direct come to IDP without redirection from SP; it won't have any idea where to go.
However.. it 'might' be possible with bunch of custom works if you have just one SP connected with your Gluu Server; but generally there are multiple to 100s of SPs connected with every Gluu Server.