I downvoted an answer, and it was deleted.
Why don't (or shouldn't) I get the reputation back?
If I don't get it back right away, then how come the Community User owns that many downvotes?
You will when your reputation is recalculated. But note, almost everyone loses rep overall when their rep is recalced because you also lose reputation gained from answers to now deleted questions.
You will now get the reputation back within 5 minutes of the deletion. In this case we update the rep of those affected as part of a background job, this is part of the recent reputation changes.