Closest I've ever heard was that Musashi made a call on Yagyu Muneyoshi Sekishusai. That night Sekishusai sent a flower to Muashi's room, Musashi examined the stem of the flower and saw that it had been cut by a sword, tried to mimic the cut, couldn't, and left. I've read that Musashi later went to Edo a few times and rumors abounded about a match between him and Sekishusai's son Munenori, but nothing ever came out of that either. Far as I know Musashi never crossed swords with any of the Yagyu swordsmen, at least none of the Yagyu family members anyway.
Increase My Killing Power, Eh?
-Homer Simpson