This is a cool topic. I'm going to become a momma this summer, so this whole idea is now becoming relevant to me. xD
I guess I have never worried too much about it, though; obviously I'm going to keep my children away from anime stuff that isn't appropriate for them at tender ages, but the fact is... I'm a nerd. I like nerdy things including anime, any I am just nerdy enough to include many aspects of this into daily life. (I am working on a fluffy Lolita picture to hang in my baby's bedroom when she's born. And I will be making her a Perry the Platypus outfit for Halloween; oh yes.) I'm not so much into anime or games as I used to be, but I suppose my baby will be fairly indoctrinated with the cute and the funny nerd things from a tender age. My husband's not really into anime at all, but he understands well enough and has his own nerdiness he'll be introducing to the baby.
So yeah... If it's something you're interested in and want to include any degree of it in your life, I say go for it. Family should come first, but that shouldn't be at the complete exclusion of who you are and what you want to do. And really, everyone's a nerd in one way or another. Nerdiness can be a healthy adult coping mechanism. ;3