According to William Scott Wilson in his translation "The Life-Giving Sword" (Japanese title:
Heiho Kadensho): "Just as important perhaps was that Sekishusai's wife, Shunto Gozen, was a daughter of the Okuhara family, a powerful clan in the nearby area of Iga, famous for the warriors trained in the arts of espionage and assassination called
ninja."(2)
Mr. Wilson goes on to discuss the death of Muneyoshi: "It is interesting in this regard that upon his death, Sekishusai passed the title of head of family on to Munenori, due to his reasonable confidence that the clan would continue under shogunal protection. But he passed the
inka and secret written materials he had received from Kamiizumi on to his grandson, Yagyu Hyogonosuke Toshiyoshi (1577-1650), the son of his eldest son, Shinjiro Yoshikatsu, who had been wounded and crippled so many years before."(2) He goes on to talk about how Toshiyoshi went to Nagoya, in Owari, to teach Tokugawa Yoshinao, and how this tradition became the Owari Yagyu Shinkage Ryu.
I've checked Hatsumi Masaaki's "Ninjutsu: History and Tradition"(3), and I do not see anything about Yagyu being influenced by or influencing the Togakure school. However, lack of evidence is not evidence of lack, as they say.