I wonder if we are "going through a phase" right now where the "diversity of content" is taking a back seat while technology and production refinements are being explored. More specifically, it seems like the current technology lends itself to producing one type of content over another.
I think the original premise that Top-40 lists used to have a wider range of styles, sounds, tempo, meter, etc. is undeniable. Whether or not the hits that rose to the tippy top were "dumb songs" or not.
Over the years, I have transcribed hundreds of pop songs for school bands. When I created a lyric/chord sheet, it was a puzzle challenge to get it all on one page. For some songs I had to create a separate chord chart because every verse and/or chorus had little variations in them, or the rhythm of the accompaniment was an important part of the song. More recently, I've noticed pop songs have either one chord progression for the whole song (Demons, Shake it Off, Despacito), or the chord progression will alter slightly for a chorus or bridge (Rise, Heathens).
It's just kind of funny now when a kid requests to learn a song, I can just hand him a post-it (that's a sticky note, kids) with 4 chords on it-- some combination of I, vi, IV, V, and iii.
Next time you're listening a pop station, notice how many songs have 2 or 4 beats (as opposed to 3 or 6), notice how many songs have ternary meters (6/8, 12/8). I'm not saying that one is better or worse than another, but just noticing that whole worlds of rhythm are just closed off.
I'm no Andrew Lloyd Weber fan, but as a kid, I remember the first time I heard "Everything's Alright", it was fascinating, and even soothing to me.
https://www.youtube.com/watch?v=1Vx8KpqTVCk
It's a swung 5/4 (3+2), you could call it 15/8, but it really is played as a swing tune with all those trappings.
I'm not saying that pop music needs to be like Tool, or clever, or even with tons of odd meters, etc. But it does seem a little sad that mainstream culture doesn't stretch much. Even film scores (which have more room to stretch) don't often go beyond the safety of 4/4 w/ 16ths. I've noticed that some music theater works will push boundaries in a satisfying way.
This ain't a fair, apples-to-apples comparision, but listen to tracks from Rent or Wicked, then listen to tracks from West Side Story or Sunday in the Park with George. Some people will have a strong preference for the former-- but if they listened to the latter more and more, I believe they would grow into it and develop a deeper appreciate of the marvelous craftsmanship and inspired artistry of those works.