What actually happens when a video becomes an MP3
YouTube never stores an MP3. Every upload is transcoded into AAC audio streams — usually 128kbps for standard uploads and around 160kbps for higher-tier ones — which are delivered separately from the video. Converting to MP3 means decoding that AAC track and re-encoding it, and every re-encode of a lossy format costs a little quality.
That is why a 320kbps MP3 made from a 128kbps AAC source is not better than the source; it is the same audio in a bigger container. If your player supports it, taking the M4A/AAC track untouched gives you the cleanest possible copy of what YouTube actually holds.