Just to elaborate on this a bit. The root cause here is that we're unable to add a redirect url containing a custom URL scheme to the OAuth client in the Gluu UI. Whenever we try to use a custom URL scheme, an error message pops up containing "The url is malformed".
Is our assumption is that custom URL schemes should be supported correct? If so, we've spent a little bit of time tracking this down. It seems like this PR https://github.com/GluuFederation/oxTrust/issues/1892 introduced checking for "https" schemes for non-native clients. In order to perform that check, a java.net.URL object is instantiated with the redirect url (see [here](https://github.com/GluuFederation/oxTrust/blob/e133aedf2fc6c9c1408516994551f85e087fc5de/server/src/main/java/org/gluu/oxtrust/action/UpdateClientAction.java#L691)). In cases of custom URL schemes, [java.net.URL](https://docs.oracle.com/javase/7/docs/api/java/net/URL.html#URL(java.lang.String)) throws a MalformedURLException. This leads to the error message being shown.