A clear impediment to your right of way is the garden office, which you say has been built across part of it. Whoever owns that building may argue that the fact it is there, and has been for some time ...