The Short Version
The historical record supports the claim. Reliable academic and official sources describe major rebuilding or expansion phases under Byzantine, medieval Serbian, Ottoman, and Habsburg rule. The only notable caveat is that the Serbian contribution is most specifically documented for the Serbian Despotate and rulers such as Stefan Lazarević.