Week 17 QB & Receiver Question?

What is your opinion on starting a receiver whose quarterback is starting for your opponent? For example I have Texans Kevin Walter and the guy I'm in the finals with is starting Schaub. Another example...would you start Colston if playing against Drew Brees? I worry about the fact that any points these receivers get their QB is also likely to get as well. What do you think????