bug80 Posted February 17, 2005 Report Share Posted February 17, 2005 (edited) I've read a couple of times on this forum that the ATRAC encoders in Sony's hardware units perform better in terms of sound quality than the encoder used by SonicStage. I tought it would be a good idea to do a listening test to find out if this is true. I'll present the results here.Used test methodI used an ABX test. For the people who don't know what that is: you can find more info here. For pval, I used a tolerance of 5% in this test.Used equipmentThe following hardware was used:* Sony MZ-N510 type S portable MD recorder* Sony DVP-S735D CD/DVD player* Alecto PRO-147 mixing desk* AKAI AM-A302 amplifier* Creative Live! 5.1 soundcard* Sennheiser EH 1430 headphone (attached to mixing desk)The following software was used:* SonicStage 2.3* ABX Comparator 1.7.4Collection of samplesAs a test sample, I used a .WAV with a length of 30 seconds of the song Prayer for the gun from the Dutch band Solo. The sample contains a drumcomputer with a sharp attack, a short hi-hat in the left channel and an open hi-hat in the right. It also contains a piece with piano, vocals and acoustic guitar. I recorded the sample in SP, LP2 and LP4 format using the optical out of the CD player and the optical in on the MD unit. From now on these are called hardware samples. I recorded these hardware samples to my computer in an analogue way using a connection on the mixing desk, which is attached to my Live! soundcard via the amplifier.Furthermore, I encoded the original WAV file to LP2 and LP4 using SonicStage. Because I had to make a fair comparison, I transferred these samples (which will be called software samples from now on) to MD and recorded them back using the above analogue method.ResultsWith these samples I performed an ABX test, with 'A' being the original WAV and 'B' being an encoded sample. I used the headphone to listen to the samples.Hardware SP: 19 out of 20 (pval < 0.05%)Hardware LP2: 20/20 (pval < 0.05%)Hardware LP4: 20/20 (pval < 0.05%)Software LP2: 20/20 (pval < 0.05%)Software LP4: 20/20 (pval < 0.05%)Because this test is all about comparing hardware with software encoding, I also did an ABX test between hardware/software LP2/4:Hardware LP2 versus Software LP2: 14/20 (pval = 7.5%)Hardware LP4 versus Software LP4: 18/20 (pval < 0.05%)ConclusionsFirst of all, I clearly heard the difference between all encoded files and the original WAV. The difference in the case of LP files was evident, these samples sounded compressed with a lot of warbling and pre-echo in the hi-hat sound.In the case of SP the difference was more subtle, but hearable. Very high frequencies were missing and some acoustic strums that were played simultaniously with a snare drum were attenuated (most probably because of the masking algorithm).In the case of LP4, I could tell the difference between the hardware and the software encoded files. And here comes the weird part: the file encoded in SonicStage sounded better!!. The hardware sample suffered from slightly more artifacts in the hi-hat sounds.In the case of LP2, I couldn't tell the difference. The result of 14 out of 20 isn't relevant, because pval is greater than the tolerance of 5%.RemarksI'm still quite shocked that I could hear the difference between SP and WAV. I thought SP should be transparent to most people with most music. It would be very accidental if the first sample I've picked would be a problem sample. What I think, is that the very high frequenies were missing because of the analogue chain MD > cable > mixing desk > cable > amplifier > cable > soundcard. Because of this, I can't really draw the conclusion that SP isn't transparent to me. Still, there seems to be a failing "masking effect" that shouldn't have to do with the way the sample is recorded. Maybe I should do a test in the future with my MD recorder directly connected to my soundcard.Second, these results are valid for this sample only. Maybe there's another sample with which I can tell the difference between hardware and software LP2, who knows. Also, the headphone I used isn't Class A. Better headphones may yield other results.* EDIT *I recorded a new WAV sample by playing the original one back on the computer, while recording it in the same way as I did from MD. If I ABX this new sample with Hardware SP, the result is 12/20 (pval = 40%), so the difference was indeed due to the signal path instead of the ATRAC encoding. I still can manage to ABX this WAV file with the hardware LP2 file with a result of 19/20 (pval < 0.05%), so that hasn't changed. Edited February 17, 2005 by bug80 Quote Link to comment Share on other sites More sharing options...
Michael1980 Posted February 17, 2005 Report Share Posted February 17, 2005 Interesting topic. IMO the encoder (hardware) gets better and better with each revision. You have the *10 model which is only one generation behind the HiMD units. So it is pretty upto date.At the same time SS's encoder is revised (supposedly) with each rev as well, and you have the latest one there as well.Interesting results (SS being better than hardware). Quote Link to comment Share on other sites More sharing options...
jadeclaw Posted February 19, 2005 Report Share Posted February 19, 2005 In the case of LP4, I could tell the difference between the hardware and the software encoded files. And here comes the weird part: the file encoded in SonicStage sounded better!!. The hardware sample suffered from slightly more artifacts in the hi-hat sounds.In the case of LP2, I couldn't tell the difference. The result of 14 out of 20 isn't relevant, because pval is greater than the tolerance of 5%.Well, that confirms my tests, with Version 2.3 SonicStage has taken the lead over my NH700.As with your test, the difference with LP2 was small, but audible, with LP4 and Hi-LP, it was quite audible. Quote Link to comment Share on other sites More sharing options...
dex Otaku Posted February 19, 2005 Report Share Posted February 19, 2005 I haven't done any similar testing with SS 2.3, but back with 2.2 the difference between hardware and software LP2 with basically anything I put through [except for mono recordings] was like night and day. Note that my LP2 hardware encodes were done via analogue, and sounded far better than SS's rips directly from CD. I'd like to hear the results of a similar test done with Hi-SP, really. SP is pretty much out of the game since SS simply can't do it.And indeed, the best-known problems [to me at least] with all of the ATRAC encoders are pre-echo and ringing, which basically any electronic music featuring drum machines and sounds with sharp attacks will basically cause hell with. In terms of this, you've probably chosen *good* material to test the differences with, since such material will plainly show the difference in this deficiency of the encoders. Quote Link to comment Share on other sites More sharing options...
jadeclaw Posted February 19, 2005 Report Share Posted February 19, 2005 I haven't done any similar testing with SS 2.3, but back with 2.2 the difference between hardware and software LP2 with basically anything I put through [except for mono recordings] was like night and day. Note that my LP2 hardware encodes were done via analogue, and sounded far better than SS's rips directly from CD. That was my experience with SS2.2 as well, 2.3 reversed it.I'd like to hear the results of a similar test done with Hi-SP, really. I have to disappoint you, my ears aren't good enough to discern reliably between PCM and Hi-SP.So I leave that to the young'uns...And indeed, the best-known problems [to me at least] with all of the ATRAC encoders are pre-echo and ringing, which basically any electronic music featuring drum machines and sounds with sharp attacks will basically cause hell with. In terms of this, you've probably chosen *good* material to test the differences with, since such material will plainly show the difference in this deficiency of the encoders.←The infamous castanets come to mind... Seriously, I test it with Jazz and classical music and of course high quality digital recordings only.When Marcus Millers' razor sharp bass riffs get mudded down, then you know, something is wrong... Quote Link to comment Share on other sites More sharing options...
bug80 Posted February 19, 2005 Author Report Share Posted February 19, 2005 I haven't done any similar testing with SS 2.3, but back with 2.2 the difference between hardware and software LP2 with basically anything I put through [except for mono recordings] was like night and day. Note that my LP2 hardware encodes were done via analogue, and sounded far better than SS's rips directly from CD. Of course, there are two options:1) The encoder in SS 2.3 has improved considerably compared to 2.22) The encoder in your unit is better than in mineNote that I didn't rip directly from CD. I ripped the track to a .wav file using CDeX. Sorry, I should have mentioned that. The .wav file was then encoded to ATRAC using SonicStage.I'd like to hear the results of a similar test done with Hi-SP, really. SP is pretty much out of the game since SS simply can't do it.I'd love to do it, but I don't own a HiMD unit, unfortunately. Quote Link to comment Share on other sites More sharing options...
jadeclaw Posted February 21, 2005 Report Share Posted February 21, 2005 Note that I didn't rip directly from CD. I ripped the track to a .wav file using CDeX. Sorry, I should have mentioned that. The .wav file was then encoded to ATRAC using SonicStage.←Should not make any difference, as long as CDeX doesn't change the data, e.g. Normalizing. Quote Link to comment Share on other sites More sharing options...
bug80 Posted February 22, 2005 Author Report Share Posted February 22, 2005 Should not make any difference, as long as CDeX doesn't change the data, e.g. Normalizing.←Yes I know, but when you present your method & results you should mention such things Quote Link to comment Share on other sites More sharing options...
Latexxx Posted February 22, 2005 Report Share Posted February 22, 2005 I have to disappoint you, my ears aren't good enough to discern reliably between PCM and Hi-SP.←http://www.hydrogenaudio.org/forums/index....5&hl=atrac3plus Quote Link to comment Share on other sites More sharing options...
Michael1980 Posted February 22, 2005 Report Share Posted February 22, 2005 Wow, some MDCF bashing going on there. Quote Link to comment Share on other sites More sharing options...
bug80 Posted February 22, 2005 Author Report Share Posted February 22, 2005 Cygnus X1 didn't give a full explanation of his method. For instance, I showed that the signal path can be as much of influence on the quality as the codec. Quote Link to comment Share on other sites More sharing options...
Guest NRen2k5 Posted February 28, 2005 Report Share Posted February 28, 2005 (edited) I'd like to see a meaningful test:Hi-SP, Hi-LP, LP2 and LP4 as encoded by SonicStage 2.3vsHi-SP, Hi-LP, LP2 and LP4 as recorded on the NH700 from an optical source.(Keep in mind that timing and volume can affect ABX results, so make sure the length and volume of the competing samples is the same.)Anyway, my first guess - or my *opinion* if you'd prefer - is that the encoding engine between SS 2.2 and 2.3 is identical and something between your ears is influencing the test results. Edited February 28, 2005 by NRen2k5 Quote Link to comment Share on other sites More sharing options...
bug80 Posted February 28, 2005 Author Report Share Posted February 28, 2005 I'd like to see a meaningful test:Hi-SP, Hi-LP, LP2 and LP4 as encoded by SonicStage 2.3vsHi-SP, Hi-LP, LP2 and LP4 as recorded on the NH700 from an optical source.Well, go ahead! I'm curious too.(Keep in mind that timing and volume can affect ABX results, so make sure the length and volume of the competing samples is the same.)Yes, I took that into account.Anyway, my first guess - or my *opinion* if you'd prefer - is that the encoding engine between SS 2.2 and 2.3 is identical and something between your ears is influencing the test results. I didn't complare 2.2 and 2.3. Furthermore, it was a blind test, so no user-bias was present in the tests. Quote Link to comment Share on other sites More sharing options...
bug80 Posted March 3, 2005 Author Report Share Posted March 3, 2005 (edited) UPDATEI installed the new SS version (3.0) and did a quick test to see if the encoder has improved, and it seems it did (a little)! I converted the same sample I used before with SS 3.0 to LP2 and did an ABX test between this one and the SS 2.3 version:SS 2.3 (LP2) versus SS 3.0 (LP2): 15/16Original WAV versus SS 3.0 (LP2): 16/16So, I could tell the difference between the two SS versions and SS 3.0 sounded better in fact, with slightly less artifacts in the hi-hat and even the vocals were more smooth (some distortion is present in the vocals with SS 2.3).As expected, the "new" LP2 still differs a lot from WAV. Sorry HiMD people, I still don't own a HiMD unit, so I can't test higher bitrates. Edited March 3, 2005 by bug80 Quote Link to comment Share on other sites More sharing options...
ROMBUSTERS Posted March 4, 2005 Report Share Posted March 4, 2005 has anyone conducted similar tests with version 3.0? Quote Link to comment Share on other sites More sharing options...
Daijoubu Posted March 5, 2005 Report Share Posted March 5, 2005 Dude, you made me want to wipe and reencode everything >_< Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.