MP3 usage in HiMD

[Berke]

Hi,

The HiMD Units' incapability of drag&dropping of the mp3 files directly raised this question in me: What does the unit do to the mp3 file to make it playable? and Is there "any" change in the data after downloading to the md and uploading it back to the computer?

I'm asking this because if the mp3 files are transferred as "bit by bit" exact copies, I'd like to free up some space on my HDD by deleting my whole library. (It's a bit of a too many copies of the same music under one roof: The Cds, the mp3s on the hdd and the mp3s on the MD).

I asked this also because there were talks about how not very good at playing mp3 files the hi-md units are. (Is this sentence even readable let alone under understandable?)

Thank you in advance for the nice answers.

[raintheory]

Understand your question perfectly.

Interestingly enough, WAV files that are transferred to HiMD then back to a PC retain their bit-by-bit information (MD5 & CRC32)... However, I just tested with an mp3 and the MD5 and CRC32 values were different...

Let me test this with at least one more mp3 and post back...

This is very interesting because SonicStage does not appear to be changing mp3's when it transfers them to 2nd gen HiMD recorders, whereas when you transfer a WAV file, it converts it (to PCM)...

EDIT: Retested this and the MD5 and CRC32 values are indeed different. Strange.

Edited February 8, 2006 by raintheory

[Berke]

Strange, very strange indeed.

I'm very curious about the detailed technical changes in the file. The bitrate and the filesize doesn't change I guess, but I wish I knew a way to compare the sound qualities of the both versions of the files (by a program with meticulous detail, rather than just hearing them by ear for comparison).

[Sony_Fan]

Hi,

The HiMD Units' incapability of drag&dropping of the mp3 files directly raised this question in me: What does the unit do to the mp3 file to make it playable? and Is there "any" change in the data after downloading to the md and uploading it back to the computer?

I'm asking this because if the mp3 files are transferred as "bit by bit" exact copies, I'd like to free up some space on my HDD by deleting my whole library. (It's a bit of a too many copies of the same music under one roof: The Cds, the mp3s on the hdd and the mp3s on the MD).

I asked this also because there were talks about how not very good at playing mp3 files the hi-md units are. (Is this sentence even readable let alone under understandable?)

Thank you in advance for the nice answers.

Direct playback of MP3s on Hi-MD units will result in poor sound quality. Unfortunately, I've had to convert my MP3s to ATRAC3+ and this improves the sound quality. If you want to just store the MP3s as files, you can drag and drop them into the unit, but the unit will not recognize them as audio files, only as data files. Of course, you can transfer the MP3s directly, but like I said the sound quality isn't worth it.

[raintheory]

Strange, very strange indeed.

I'm very curious about the detailed technical changes in the file. The bitrate and the filesize doesn't change I guess, but I wish I knew a way to compare the sound qualities of the both versions of the files (by a program with meticulous detail, rather than just hearing them by ear for comparison).

Actually, the size of the file does change slightly. It is slightly smaller when transferred back to the PC. Also the header information is moved. See my post regarding this here: http://forums.minidisc.org/index.php?s=&sh...indpost&p=86843

[samplehunter]

...This is very interesting because SonicStage does not appear to be changing mp3's when it transfers them to 2nd gen HiMD recorders, whereas when you transfer a WAV file, it converts it (to PCM)...

EDIT: Retested this and the MD5 and CRC32 values are indeed different. Strange.

This "Conversion" to pcm isn't really one if you use uncompressed WAV files.

There is a Header at the beginning which is lost at DL and reconstructed at UL. Try saving a WAV with additional Chunks with META-Data or Sampler Infos like Loop-Points or Cuelists etc. You can do this with e.g. CoolEdit. If you transfer the File and get it back to convert to wav again, I'm sure you will lose the Meta-Data and therefor get a different Hash.

Maybe mp3s are re-padded, or the ID3 Tags are stripped out before transfer. So the file might be physically different after back transfer even if they contain the same mpeg stream.

[raintheory]

This "Conversion" to pcm isn't really one if you use uncompressed WAV files.

There is a Header at the beginning which is lost at DL and reconstructed at UL. Try saving a WAV with additional Chunks with META-Data or Sampler Infos like Loop-Points or Cuelists etc. You can do this with e.g. CoolEdit. If you transfer the File and get it back to convert to wav again, I'm sure you will lose the Meta-Data and therefor get a different Hash.

Maybe mp3s are re-padded, or the ID3 Tags are stripped out before transfer. So the file might be physically different after back transfer even if they contain the same mpeg stream.

Right, the WAV header thing I understand. Also, when I did the mp3 testing I stripped the tags from the files completelty. Before and after conversion.

I just find it odd, because SonicStage appears to just transfer the mp3 as-is (without anything converting on the file) whereas it does actually change/add data to the WAV file prior to transferring.

[Berke]

But how can we be sure that it's the same mpeg stream? Given the fact that the lesser sound quality himd units on mp3s have, how do we know it doesn't change the file in order to make it more managable for the unit? Hearing the not so decent quality on the md units make me think that.

[samplehunter]

...

I just find it odd, because SonicStage appears to just transfer the mp3 as-is (without anything converting on the file) whereas it does actually change/add data to the WAV file prior to transferring.

I think it's possible to re-upload the mp3s from the unit to the pc, right?

If not, you couldn't haave done crc checks on both versions :-)

ok, to check if the data itself differs you can do the following:

decompress 1.mp3 to 1.WAV

transfer it to the unit and back to the pc as 2.mp3

check, if 1.mp3 and 2.mp3 have exactly the same bytesize

decompress 2.mp3 to 2.WAV

in an audio editor you can now substract one file from the other. (invert the phase of one file and mix them)

If they are identical, there will be nothing left but silence. else you'll get (hear) the difference. This would be unlikely because then the file must have been transcoded which costs cpu power.

IF the decompressed files are identical, the mp3s may differ only in the kind of padding the data or different flag bits like "original" or "pre emphasis".

mp3s can store the frames either with no padding at all or with iso compatible or cd-rom compatible padding.

You can also try:

transfer a mp3 with ID3v1 or v2 tag. does it have the tag after transfer back?

transfer one without any tags. give it a title on the md. does it have a tag after transfer back?

[raintheory]

I think it's possible to re-upload the mp3s from the unit to the pc, right?

If not, you couldn't haave done crc checks on both versions :-)

- Yes, they are re-uploaded fine.

ok, to check if the data itself differs you can do the following:

decompress 1.mp3 to 1.WAV

transfer it to the unit and back to the pc as 2.mp3

check, if 1.mp3 and 2.mp3 have exactly the same bytesize

decompress 2.mp3 to 2.WAV

in an audio editor you can now substract one file from the other. (invert the phase of one file and mix them)

If they are identical, there will be nothing left but silence. else you'll get (hear) the difference. This would be unlikely because then the file must have been transcoded which costs cpu power.

IF the decompressed files are identical, the mp3s may differ only in the kind of padding the data or different flag bits like "original" or "pre emphasis".

mp3s can store the frames either with no padding at all or with iso compatible or cd-rom compatible padding.

- The mp3 that is transferred back to the PC from HiMD is slightly smaller than the original. I tested this with an mp3 that I had previously uploaded from HiMD and the file size was slightly smaller again.

- I did also test by decompressing to the original to WAV, as well as the uploaded mp3 to WAV. Still changed.

- The phase inversion test equals silence...

You can also try:

transfer a mp3 with ID3v1 or v2 tag. does it have the tag after transfer back?

transfer one without any tags. give it a title on the md. does it have a tag after transfer back?

- the tags are intact if they exist in the original mp3

- I stripped tags from the mp3 and after uploading back it added tags for the title and album (which i stripped before comparing to the original.

Edited February 8, 2006 by raintheory

[demod]

i'm into reverse engineering a little bit,

so i actually couldn't resist and took my time

to figure it out how mp3s are transformed, right

the next day i got my device

I've got rh10 here, but i believe it's all the same

for every unit out there.

First of all, all the id3v1/id3v2 data is cut off,

along with other stuff ( if there's any, don't really

know mp3 format that much ) which is not related to the

actual sound data.

Then the most interesting part comes.

The leftovers of a song are basically XORed with a

32-bit key which is constant for the very first

song being transferred onto device, but for subsequent

ones it does change and it somehow depends on the size

of already transferred data.

I couldn't find any obvious pattern of key changes,

but most likely it could be unambiguously restored

for every single song from its byte offset in the

main storage file. ( cant get my hands on the device

to tell exact file name now, sorry ).

One could clear the situation by *cough* disassembling

ss binaries, but you all know that you're breaking the law

by doing that. So do i. Furthermore, i didn't want to waste my

time

cheers.

i have to add that i did that mainly to make sure

those mp3 playback drawbacks ( modified db/freq ratio )

were because of the hardware, not some sonicstage intrigues

[raintheory]

Strange, very strange indeed.

I'm very curious about the detailed technical changes in the file. The bitrate and the filesize doesn't change I guess, but I wish I knew a way to compare the sound qualities of the both versions of the files (by a program with meticulous detail, rather than just hearing them by ear for comparison).

There doesn't appear to be any changes in the sound of the mp3 (tested this by inverting the phase of the original, and then mixing the transferred file into it. The result was complete silence, leading me to beleive the changes aren't being made to the audio portions of the file.

[Berke]

It's good then that there is no change in the actual music data of the file. But what happens if the same mp3 is ULed and DLed back and forth a hundred times per say? Would the tiny file decrease ever continue?

Edited February 8, 2006 by Berke

[samplehunter]

It's good then that there is no change in the actual music data of the file. But what happens if the same mp3 is ULed and DLed back and forth a hundred times per say? Would the tiny file decrease ever continue?

i think mp3 frames are normally filled up with zeros or junk data to match sector size better or to be more compatible when used as live stream. But it's not necessary. some mp3 encoders allow you to save just the raw stream without these fillings. i suppose that if you upload a transferred mp3, transfer it to the unit and back to the pc the size will not decrease further. Otherwise they must drop some of the actual mpeg data and then the signal would not be identical anymore.

btw are there any audio demos of the "bad" mp3 playback quality available here?

[dex Otaku]

btw are there any audio demos of the "bad" mp3 playback quality available here?

http://forums.minidisc.org/index.php?showtopic=10621

Scroll down to "MP3 playback test"

I don't think an audio demo is really necessary. You can see exactly what it's doing, there.

[raintheory]

i think mp3 frames are normally filled up with zeros or junk data to match sector size better or to be more compatible when used as live stream. But it's not necessary. some mp3 encoders allow you to save just the raw stream without these fillings. i suppose that if you upload a transferred mp3, transfer it to the unit and back to the pc the size will not decrease further. Otherwise they must drop some of the actual mpeg data and then the signal would not be identical anymore.

btw are there any audio demos of the "bad" mp3 playback quality available here?

Actually from what I've tested, it does still decrease slightly after another subsequent transfer/upload (after the initial transfer which reduced the size). Really though, it would be great if someone else with a 2nd gen recorder could try this as well.

[Sparky191]

A discussion about MP3 vs ATRAC sound quality is pointless unless you give details of the encoder used, the settings and the bitrates you used.

If you can do a blind listening test ABX between MP3 and ATRAC of comparitive bitrates and encodings and get over 80% correct over a large sample of tests then you can notice a difference. Otherwise its not a valid test, and is probably just a placebo effect. The mind plays funny tricks.

Edited February 9, 2006 by Sparky191

[Tom Ellard]

For what it is worth when you drag and drop an MP3 to a PlayStation Portable and also use Sonic Stage to do the transfer again the two sound identical. They sound different to the track on Hi-MD.

I really think that Hi-MDs just can't play back MP3 very well, rather than any Sony weirdness.

[Sparky191]

I'm confused Tom. Sony made the HiMD. So how is that not Sony "weirdness"?

[raintheory]

The initial question wasn't so much sound quality differnces in ATRAC3+ & MP3... It was if the Mp3 Data is changed in any way after transferring to HiMD and back to PC again...

[Sparky191]

"not very good at playing mp3" would indicate a query re: SQ to me?

[1kyle]

Any operation (apart from a non destructive READ or Bitwise copy) you do to a compressed lossy file will reduce its quality (and make it smaller). -- Some compressed files .ZIP for example are Lossless -- these can be worked on without loss of data or quality.

MP3's are of course compressed Lossy files.

For example you are probably familiar with jpegs (jpgs) --picture / camera files.

Open a Jpeg with say microsoft photo editor or photoshop or whatever.

Save the original.

Open a copy.

Do a small edit on it.

Save.

Open again and do some more editing.

After say 3 or 4 edits compare it with your original file -- you will see the quality of your processed file considerably degraded from the original one.

The same with an MP3 file --it's a compressed file so each time you do an operation on it and save you are throwing away more information.

If you MUST use MP3's --start with a WAV file -- do all the editing you need FIRST then save as MP3 otherwise you'll get a really horibble sounding file when compared with the original source.

I'd also store the mp3 at the highest bit rate possible -- if you've got a large bit rate then the degradation on conversions will be less than with smaller bit rates.

If you started with ATRAC3 Lossless then I'd go first to WAV (you won't lose any info) then EDIT then save as mp3.

I'm not sure about ATRAC3 lossless direct to mp3 you could try that as an experiment --but ensure the original source is UNCOMPRESSED.

ATRAC(lossy) to Mp3 (another lossy format) will result in quite a bit of degradation --even going from ATRAC (lossy) to WAV first and then back to mp3. This could explain the poor quality of MP3 playback on some MD units as it probably has to convert the MP3 to ATRAC in the device before playing / up / downloading.

I doubt if the MD unit natively handles MP3 without doing some conversion in the hardware / firmware --so a pure Computer Hack is unlikely to yield a solution.

Other devices probably can just read your MP3 without any conversion hence no (more) degradation.

I'm not a fan of MP3 so I can't say --however the general principle of degradation of lossy files always applies whether it's pictures, audio or any thing else so always keep the number of conversion operations to an absolute minimum when working with lossy files.

Cheers

-K

Edited February 12, 2006 by 1kyle

[Sparky191]

What? The whole point of feature of the 2nd is that it IS native MP3 playback.Whats effecting the sound is that Sony have tweaked the firmware to that the EQ is different for MP3's vs ATRAC files. Why? So that people who don't realise this will say ATRAC is better than MP3. There simply is no other reason to do this. Most people don't even realise (or can't hear) the different in SQ between low and high bitrate file, in either ATRAC or MP3. So basically Sony are treating the consumer as a moron, by making this tweak. They assumed the consumer would be too dumb to realise what they were doing.

This topic has been discussed at length before in these forums. At the end of the day theres no reason for Sony to muck about with the MP3 file as its transferred (not converted or transcoded). It just annoys the consumer.

Edited February 12, 2006 by Sparky191

[Berke]

Both 1kyle's and Sparky191's comments sound feasible to me, although I wouldn't express my thoughts on Sony's possible manipulation with the consumers in the same exact manner as Sparky did.

I don't reject Sparky's notion only because Sony hasn't made an announcement explaining the inner workings of the native mp3 playing that there is. Otherwise I totally would lean onto the hardware/firmware limitations on mp3 playback on himd units (but hey! doesn't that make them "un"native?).

I'm currently very happy with the mp3 playback as long as the bitrate is high. (128kbps is sure unpleasant to listen).

Unless you'll be working on detailed audio-making or whatever, the SQ is not a huge issue IMHO. Just off-topic, I couldn't find the Atrac3plus Hi-LP so much different than the Atrac3plus Hi-SP playback, so for me, putting 20+ CDs of mine in one single MD does far outweigh putting only 7 albums on one disc (with the "slightly" better Hi-SP SQ).

Edited February 12, 2006 by Berke

[Sparky191]

....

I'm currently very happy with the mp3 playback as long as the bitrate is high. (128kbps is sure unpleasant to listen). ...I couldn't find the Atrac3plus Hi-LP so much different than the Atrac3plus Hi-SP playback...

Well thats a good example.

You reckon 64kps ATRAC (HiLP) is ok but 128kps MP3 isn't. But then you say you can't hear much difference between 64kps (HiLP) and 256kps (HiSP). The different between 64kps HiLP vs 128kps MP3 should be a lot less and the difference between HiLP and HiSP should be more. Basically theres something not right with that.

If HIMD couldn't play back MP3's natively then it would have to trancode/convert the MP3 file to ATRAC to transfer it. The transfer times simply do not reflect this. So the assumption therefore is that they files are not being trancoded and therefore it IS native playback.

This is all a pretty pointless though. Since if you can't hear the difference you'll use MP3's regardless. If you can hear the difference you'll use ATRAC on HiMD and MP3 on another devices. The fact that 1st gen HIMD couldn't play MP3's meant most of the market with MP3 collections has moved to other devices which could play them. I don't think theres many left to care at this point.

...I'm asking this because if the mp3 files are transferred as "bit by bit" exact copies, I'd like to free up some space on my HDD by deleting my whole library. (It's a bit of a too many copies of the same music under one roof: The Cds, the mp3s on the hdd and the mp3s on the MD)...

People do it in different ways.

I use an external USB HD as the main library. I copy from that onto what ever device I want to use. CD's are ripped into this library then they go back on the shelf for use with the HiFi. Once ripped the CD's are effectively my backup. I don't use MD's as my main portable. I use a MP3 player. My MD's are only used for recording, once thats transferred to the library I reuse the MD. The Library on the HD is backed up to another HD (along with all my other PC data) on a regular basis.

If I used the HiMD as my main portable, I think I'd still use the HD as my main library.

[Berke]

Well thats a good example.

You reckon 64kps ATRAC (HiLP) is ok but 128kps MP3 isn't. But then you say you can't hear much difference between 64kps (HiLP) and 256kps (HiSP). The different between 64kps HiLP vs 128kps MP3 should be a lot less and the difference between HiLP and HiSP should be more. Basically theres something not right with that.

Well, I had a feeling someone would mention the inconsistency of my utterances. The point is that I only said what I heard. Truly 64kbps hi-LP sounded clearer to me whereas my other 128kbps mp3s sounded foggy and shallow in some way. I didn't particularly put the three formats all side by side and compare them. Firstly, I knew I wasn't happy with the 128k mp3 files, and then When I was DL'ing some CDs onto my walkman on SB, I noticed maybe hiSP wasn't worth it when I could cramp up 20 or more CDs on one MD after checking and comparing SP and LP. I thought I wouldn't mind listening to the music at this (undisturbing) quality (this is for casual daily music that I listen without too mch focus)

But the conclusion: Still no to 128 mp3 and yes to Hi-LP. And that is, you can say, my subjective hearing. SOme other folk might think differently. (Even I might change my mind after some time).

Also on the Main library usage: I think a portable Music library on a USB disk sounds very good. After I transfer all my CDs to my MDs, I'll try to move my library somewhere else than my HDD. Cheers.

Edited February 12, 2006 by Berke

[Sparky191]

How did you create the MP3's?

[Berke]

Good question! I'm sure it also has a big effect on how they sound too. Not all the same bitrate mp3s sound the same. The thing is that I don't remember how I encoded the files. Most of them I know I used Lame, and some I used iTunes' mp3 encoder. But I don't know which ones which.

[Sparky191]

Good question! I'm sure it also has a big effect on how they sound too. Not all the same bitrate mp3s sound the same. The thing is that I don't remember how I encoded the files. Most of them I know I used Lame, and some I used iTunes' mp3 encoder. But I don't know which ones which.

Makes all the difference. Personally I can't stand low bit rates ATRAC or MP3. So maybe I'm biased...

[1kyle]

I'm still not sure whether SONY plays mp3's in Native mode or does "in line" conversions.

Note that as any conversion will be done by HARDWARE it can be incredibly quick unlike software transcoding -- and as the MD device has something like a 10 second buffer (or bigger) you won't percieve any delay due to "In line" transcoding.

One would probably have to get at the design manual to see exactly where transcoding is done -- if any.

I doubt whether Sony would deliberately add artifacts or intentionally make the sound worse. What probably is taking place is that Sony's transcoding algorithm is probably using less of the file to keep the transcoding speed as high as possible. This means that unless you have a very high bit rate your mp3 files won't sound very good.

I'd be interested to know exactly where the transcoding is done however since it seems totally unrealistic for SONY to include 2 DAC (Digital to Analog) engines --one for ATRAC and one for MP3.

In all cases of course the final output has to be analog --that's what drives speakers and of course our ears.

Cheers

-K

[Sparky191]

This is old ground. Scroll down and look at the freq analysis.

http://forums.minidisc.org/index.php?showtopic=10304

Edited February 14, 2006 by Sparky191