It'll be fine. Somewhere "below" the pole (in your drawing) put a curved turnout in your outside track. Then you would curve the diverging leg back the other way and head down the spur/yard. Yep, that would make an S-curve and some people poo-poo those - I am not one. You'll just have to play around with the positioning of the turnout off the main so that the radii in the S aren't too tight. Once the main leg is figured out, add turnouts and... small yard.On a seeking advice point I was thinking of maybe doing a small yard off of the big loop that goes under the stairs for maybe a programing yard but not sure how a few lines off a big loop would look?
PECO has turnout templates (plans) that you can download and print actual size. Along with a few pieces of flex track and some T-pins, they're great for transferring plans from paper to baseboard to see how things are really going to fit... before buying.