You found what he kept disappearing to. There was a shrine that was just barely in the kingdom walls but also in the forest that looked well taken care of. Same routine; he started by visiting it before going to do work in the forest. Following that, he returned to tend to the shrine before heading back to the kingdom.

He was also really quiet, always lurking in the shadows and never let you get close. So you had to start moving close.