Are you sure you want panniers? If you use light camping gear, you should be able to fit everything into a frame bag, handlebar bag and seat bag, possibly complemented by a small backpack. No need for panniers. This light camping gear is also a plus when backpacking.
You could also use a backpack only if you go with really light and compact gear, at least outside the winter season. A 35 liter backpack should suffice. Any more and it is too big and heavy for biking and this does limit how many days you can go unsupported. But even such a backpack loaded with 20 lbs of gear is suboptimal when riding singletrack.
Of course, this is only my opinion. But at least it is clear that a traditional large hiking backpack full with gear is incompatible with bike riding due to the weight.
