I mean I knew the horse mechanics in the Witcher 3 was bad, but I almost entirely forgot about Bethesda...
Here is the secret; whatever direction you're going, turn your back to it and then whistle and he should be coming from the direction you wish to go.