You could solve that problem of changing decks rather easily: Store all single ratings ever given including the list of cards that the deck was using at that time. When the deck changes, figure out the relative change (with 5 new cards in 60, that's 8.3%), and then recalculate the ratings as weighted average, with the weight getting decided from this value. In my example, that would give all old ratings only 91.7% weight, and all newer ratings 100% weight. It is not as perfect as it could be, but still decent. If more than half the cards change, delete the rating entirely.
For performance reasons, you could just discard the oldest 100 ratings, because those are probably wrong anyway by now. And allow people to re-vote when their rating was deleted.