
To me, notes have a pitch and a duration, maybe some dynamics.
That could be part of the problem: time signatures aren’t a property of the notes, they’re a property of the song. The time signature is the rhythm and flow of the music. If you try to do a waltz in 4/4, it’s probably not going to sound like a waltz unless you do some really screwy things.
Plus, knowing the time signature helps you figure out why that guy is waving his stick like that while you’re playing.

It’s funny how stripping out the fourth beat just sounds like a band who doesn’t quite know how Take 5 goes and they’re just winging it: https://youtu.be/t3u8GwMlJaw