I don't have a good memory of my childhood is possible, but I think it's quite an unlikely sentence for a native speaker to produce. If the speaker has an imperfect recollection, then it might be something like I don’t remember my childhood very well. On the other hand, I have good memories of my childhood would, as you suggest, mean that the childhood was a happy one, just as I have bad memories of my childhood would mean that it was the opposite.