This is why I'm confused. My tent is pretty light, and I'm honestly not gonna get a ton of weight savings with a bivy. I've never used a bivy, though, other than in the Marines. and I'm wondering how they are to sleep in the rain. I think that will probably keep me in my tent.
I've never done a race like the TD, but just the bikepacking that I have done, I would think a bivy would be better for an actual race. Most of the bivys I or my mates have used weigh almost 2 lbs less then your tent and are much easier to deal with at the end of 12+ hours on the saddle. I use a very lightweight tent for my bikepacking adventures that takes less then 6 minutes to set up, but the people that I have camped with that bivy spend less time getting set up. In a race like the TD I can only imagine after days and weeks of pushing yourself that the little differences (like 2lbs saved weight with a bivy and less time to setup) start to become much bigger.
But don't let me discourage you from doing/trying anything. From all the reading and following on this forum, it seems there are many ways to skin the TD cat.