The Short Version
The claim that physics is "generally considered" harder than math is misleading. While some students and forum commenters report finding physics harder, the available evidence shows no broad consensus. The most comprehensive source in the evidence pool explicitly states there is "no clear consensus" on which major is more difficult. The supporting evidence consists entirely of anecdotal forum posts, with no academic studies, curriculum analyses, or graduation data to back up the claim of a general view. Difficulty varies greatly by institution, level, and individual strengths.