But then I don't read the Constitution as trying to enumerate all of our rights.
It doesn't try. The Bill of Rights is a list of areas that are to remain free of governmental interference. Most are designed to defend against a tyranical government. (Much more of a danger, especially when you consider the logistics of invading across an ocean and finding an armed and hostile populace.)
One of the Federalists major concerns, in fact, was that by enumerating natural rights in a Bill of Rights, those rights not so enumerated could be more easily trampled by the State.