First set of refactorings to lazy-load some information and optimize serialized objects.

First set of refactorings to lazy-load some information and optimize serialized objects.

Previously we had no filter on serializing these objects for caching, and a lot
of them have recursive references. Not serializing those saves a lot of time and
memory.

This probably has some bugs left, definitely needs to be updated for the CVS
driver, and certainly can be improved further.