Where I draw the line is feeling guilty about everything I’m not doing. I refuse to feel guilty if I miss visiting teaching for a month or two. But if I don’t go for a year, yes, I feel guilty and I deserve to feel guilty. I do genealogy, but I don’t do it all the time. I go in spurts. I don’t feel guilty when I’m not doing it, because I know I have done a good job and will continue to do so. There are bigger things to feel guilty about.
My kids happen to be perfect (in my unbiased opinion), but if they weren’t, I wouldn’t feel guilty about it. You can’t feel guilty for things you can’t completely control. When there are other forces working against you, and free agency in the mix, you have to do your best and hope for the best, but the cards will fall where they may and you are not to blame.
The good thing about feeling guilty is that it validates your belief system. If you don’t believe what you’re doing is wrong, you don’t feel guilty. So when you do feel guilty, do something about it.