Lossless

[Batman]

Yo,

yesterday, just for fun, I tried to send a WAV (PCM) file to my unit (MZ-NH900), using ATRAC Lossless format. So far so good, the unit got it.

Then I deleted it from the computer, and reimported _as_ WAV.

Then , of course, I went through checking whether the files were equal. I mean, an _exact_ copy, and with great frustration I discovered they were not.

First, the file uploaded from the MD was longer than the original, and this makes me guess that SS arbitrarily adds a silent gap, then, also brutally removing the gap (I assumed it was at the end...) by cutting it with a hex editor, the files comparison was not different only at the very beginning (the few bytes in the header, I mean). Ok, I could be wrong assuming the extra space was added only at the end, as it could have been split half at the beginning as well, but... if it is called LOSSLESS, why the hell does it get modified???

I also thought that perhaps my unit was not able to accept the ATRAC Lossless format, but out of my knowledge (little, thou') I know that the ATRAC decoder may not be aware of new encoders, and always decode the data properly, no matter what level of compression and/or other funny stuff was used to generate the ATRAC file.

Anyone like to shed some light on this?

I am considering buying a Sony PCM-D50 or, even better, any unit than can play FLAC files gplassly (there are a few out there, but the D50 unfortunately isn't among them) but the thing is that I love too much the media feeling... a memory card is not the same as a Minidisk... but if I cannot be sure of getting true lossless quality out of my unit, I really shall move over to a different product (the D50 doesn't play FLAC, but offers at least 4GB internal and 4GB on the MemoryStick, which with a bunch of, nowadays cheap, 4GB sticks may suffice my needs).

Thank you, a MD nostalgic... ;-)

[sfbp]

I seem to recall that even the Redbook spec for CD's is kinda iffy. Always amazed me how the drivers could decode all that info and reconstitute it by means of CRC, lead in and the like, especially when there's only one chance to write it. I well remember writing the low level drivers, one byte at a time, for floppy disks, and how you had to allow for the fact that there is no guaranteed moment during the spin when data starts arriving - it is necessary to "sync" with the bytes until you see a pattern.

Same with serial and Ethernet communication. Underneath the nice veneer of ordered frames and guaranteed delivery there is a mess.

Sound and video data is the same - ultimately analog sound discretised into bits. The major difference is that we are accustomed to a lot of compression techniques to get manageable data sizes. Unlike (say) LZH compression (zipfiles), we generally don't expect perfect reproduction of what was compressed. For example I was collecting up some old episodes of a show I like, scattered over multiple DVD's, and one of them had awful errors on it. The solution, make a digital copy using the computer, and ignore the errors. Even though the copy I made was not correct (the source simply had irrecoverable bad data in it in about 20 places), once it had a good checksum, it played back in the DVD player, and there was little or no hint to the eye and ear that something was wrong. Listener/watcher doesn't really care.

Ultimately that is the test - is the quality of what we hear/see good enough to listen/watch?

BTW, there's a well documented thing with SS rearranging data and it involves 0.5 second at the end and beginning of tracks - a "bug", and I am sure you have read the threads (but for others reading this, I mention it - look for "gapless"). There's a reason for the 2 seconds usually added between sound tracks during CD creation. It's similar to the space between bands on an LP - the idea is just what I was talking about at the beginning, time for the reading of data to synch up.

We are incredibly lucky to have all this technology handed to us by the engineers whoi worked so hard. I don't grumble much about the issues you raise - if there were perfect sound, as with video, it could be measured compared, fingerprinted, and locked up. You wouldn't want that?

[Batman]

Ultimately that is the test - is the quality of what we hear/see good enough to listen/watch?

We are incredibly lucky to have all this technology handed to us by the engineers whoi worked so hard. I don't grumble much about the issues you raise - if there were perfect sound, as with video, it could be measured compared, fingerprinted, and locked up. You wouldn't want that?

Actually, if it was only an issue of byte-differences, I could have skipped it (not too much, thou'...) but the additional gap is something I really don't like... How can you listen to a suite (and no matter if it is from Marillion, Pink Floyd or Bach) with even only half-a-second gap between the movements???

Just as a comparison, how can you call it lossless if the original -> compressed -> decompressed aren't the same (I am not talking about the sensations listening to the two files)?

If I FLAC a WAV, and then WAV it back, the two files are _exactly_ the bloo*y same, and this should be, as it _is_ a lossless codec.

So this allows me to think that the so called ATRAC Lossless, may be called lossless in the compression stage, but absolutely not in the decompression one...

Just like LZH _is_ lossless for binary files.

This was my point.

Anyway, thank you for taking your time.

Cheers, A.

[sfbp]

I think it's very simple: the Sony stuff was done 1/2 an era earlier (on the bleeding edge) when there were no standards or design requirements such as this. The gapless probably arose by accident, and the glitch in the decoding/encoding is a bug, nothing more.

The fact that they sold millions of these and that the technology is so solid (every SP disc is playable in every MD device, bar a few MZ-1's I believe) is why we are all here at MDCF.

If I want to control exactly what I hear, I use SP (good AtoD on the decks) and then edit the WAV files, finally burning them to CD. Then and only then do I consider portable audio. We're still way ahead of the I**d crowd in sound quality, IMO.

[Batman]

I think it's very simple: the Sony stuff was done 1/2 an era earlier (on the bleeding edge) when there were no standards or design requirements such as this. The gapless probably arose by accident, and the glitch in the decoding/encoding is a bug, nothing more.

This (as I said) may be called lossless, but it's not the very same as the original, which in my own knowledge is not acceptable, as a concept. If I want cheap audio, then I'll go through MP3s, that's it.

If I want to control exactly what I hear, I use SP (good AtoD on the decks) and then edit the WAV files, finally burning them to CD. Then and only then do I consider portable audio. We're still way ahead of the I**d crowd in sound quality, IMO.

I do agree with you here, but I'm pretty much considering portable audio, since I do own the original CDs, I don't even need to burn them again; I just want to hear to some high quality thing, which could be worth listening to with my fabulous (and reasonably quite expensive, allow me...) Shure SE-420. All the other stuff doesn't fall in my perception, nor in my interest.

If I cannot be sure of converting and getting the same sound quality (again, using "compressed", no matter how, audio, in the range of lossless, anyway i.e. FLAC) without any stupid, arbitrary mute-gap at either the beginning or at the end of a song (why should the encoder decide itself that there is the need to add some "silence" to let you realize the song is over???)

When I say lossless I mean listening to the original CD, or the converted, ATRAC lossless file, should make no difference at all; first in sound quality (and may my ears accept even lower quality, I should be happy as well) and second in, at least, leaving the song as long as the original was, without compromising (yes, it's compromising, or even worst, ruining) the result as a whole.

If you have any clue, or explanation, or suggestion to better make things in order to achieve what I am looking for (same quality, same gapless) you are more than welcome, otherwise i do appreciate having had a chat with you, but remain in my frustration... ;-)

I will be reading you all in the next days, and really would like to get more skilled about the whole thing; I know there are some guys here with valuable experience about the formats, who I would love to hear something from.

Cheers, A.

[sfbp]

If you have any clue, or explanation, or suggestion to better make things in order to achieve what I am looking for (same quality, same gapless) you are more than welcome, otherwise i do appreciate having had a chat with you, but remain in my frustration... ;-)

Stick to SP, get devices with digi-out, and a sound card with Digi-In. Give up on USB. Once the sound you have is on CD, then burn it to wherever you want bearing in mind it's not going to get any better.

One of the advantages of SP is there are no silly restrictions on editing. I have been quite unable to find out if HiMD in a deck removes those silly restrictions - I rather suspect not, as I found even my MXD-D400 deck refuses to edit certain tracks, unlike the JE640 which doesn't care.

[jupitreas]

The Hi-MD unit is not able to play ATRAC-Lossless files. It only plays the ordinary ATRAC portion and ignores the remaining data present in the codec.

For this reason, what you actually transferred to your player wasnt lossless - it was the 256 kbps ATRAC file. Naturally, transcoding it into WAV again resulted in it being a very different file.

The only way to play lossless on the Hi-MD unit is using PCM and this would most likely result in the files not being changed at all, or at least only slightly by adding tags etc.

[Batman]

The Hi-MD unit is not able to play ATRAC-Lossless files. It only plays the ordinary ATRAC portion and ignores the remaining data present in the codec.

For this reason, what you actually transferred to your player wasnt lossless - it was the 256 kbps ATRAC file. Naturally, transcoding it into WAV again resulted in it being a very different file.

Now, this actually makes sense. I thought the units were able to play any type or version of ATRAC, including this lossless.

This tells why the files were different.

The only way to play lossless on the Hi-MD unit is using PCM and this would most likely result in the files not being changed at all, or at least only slightly by adding tags etc.

Yes, this I know, of course...

Thank you very much for the explanation.

Cheers, A.

[sfbp]

Sorry - completely missed the ball on this one.

It has been noted in the past that even the simplest transfers won't generate binary copies, just as Jupitreas says. Atrac Lossless is just a container, it's not actually useful except as a better way to store (and regenerate) wave files. And of course when something goes from 650MB to 300 (as you will find with AAL) then something got thrown away. We all know that zipping sound files doesn't reduce their size. The Lossless here means "just a better form of compression", as explained by Sony in the Help file, I thought adequately.

"Lossless" is a relative word. My own take is that 292kbps(SP, original Atrac) and 256kbps (Atrac3+/HiMD) are as good as you are going to get. The major reason it's worth recording things in 1411 (CD) resolution is that if you get the dynamic range wrong in a live recording, there are still enough bits there that you can amplify it or normalize it, and it still sounds ok.

Beware salesmen bandying words around......

[MDane]

The Hi-MD unit is not able to play ATRAC-Lossless files. It only plays the ordinary ATRAC portion and ignores the remaining data present in the codec. For this reason, what you actually transferred to your player wasnt lossless - it was the 256 kbps ATRAC file. Naturally, transcoding it into WAV again resulted in it being a very different file.

Beware salesmen bandying words around......

I was thinking the same thing when I read post #1. The reason it came to me so quickly is because when I first started messing with SS, I also was misled by the term "atrac lossless". I initially didn't realize this was basically just a glorified way of saying it is a splitable file...meaning it is virtually the same file as the original yet can be split off (immediately) into the desired (pre-determined, pre-tagged) compression for transfers to portable devices. But after some experiments, and reading info on this site and a few others places, it all made sense. So I had already had the dilema the OP had a few months ago, and had already worked it out for myself.

1. One of the advantages of SP is there are no silly restrictions on editing. 2. We're still way ahead of the I**d crowd in sound quality, IMO.

1. I agree, as my JE440 only allows SFE editing in SP. Due to this, those of us with RH1/M200's are stuck with using SP at some stage for all live recordings that inevitably require SFE editing (unless there is some deck or software I don't know about that allows such editing in all MD modes). Of course we can get by recoding in LP modes for material that only requires divides...but fade in and out, etc. sure would be nice there too.

The rub is that to get either long duration live recordings or pure sound quality benefit from an RH1/M200, it needs to be used in Atrac 3+ or PCM in Hi-MD mode. Sure, one can record in PCM, and then record that music to SP the old school way, and start from there - but that is a whole lot of extra steps...and then the newly edited (SP) file has to be re-uploaded and compressed (if not saving in WAV) just to have a PC useable file. This almost makes me want to go back to a SP recording portable, and a simple SP deck...but then I would loose my uploading ability. Sheesh!

2. For what it's worth, using say 256-352k Atrac 3+ is probably acoustically transparent to the human ear on almost all music...and even 292 (SP) is nearly equal to it when recorded in real time using a quality (Type-R) home deck. In other words, they both fall under the category of "good enough", but I do realize the desire to have a perfect file to satisfy the OCD in all of us.

[sfbp]

those of us with RH1/M200's are stuck with using SP at some stage for all live recordings that inevitably require SFE editing (unless there is some deck or software I don't know about that allows such editing in all MD modes). Of course we can get by recoding in LP modes for material that only requires divides...but fade in and out, etc. sure would be nice there too.

Don't you do your SFE using a tool on the computer? It's as quick or quicker, and reversible too. Once I have got the recorded sound split into tracks I want, I upload with RH1, and then work on the wave files. Then I delete the oma from My Library, and reimport the .wav files. At this point they can be compressed for the purpose of saving space, to AAL. Some weird thing prevents wave files from being transcoded to AAL sometimes, but I haven't discovered what it is. The above sequence seems to work reliably.

Alternatively, if I have major problems with the raw sound, I may upload it as one file, or via digi-opti-IO, so that whatever I do it's done consistently to the whole thing (eg removing a whine, or normalizing L/R balance). Then I split it up myself from CoolEdit (aka Audition) into individual wave files and import them into SS to do with as I wish.

On another topic (that you touch on), I did record in MDCF my finding that creating groups on a HiMD disk after recording affects the ability to do editing operations such as creating track marks...... as long as you don't move things around, you can divide and combine to your heart's content.

[MDane]

Don't you do your SFE using a tool on the computer? It's as quick or quicker, and reversible too. Once I have got the recorded sound split into tracks I want, I upload with RH1, and then work on the wave files. Then I delete the oma from My Library, and reimport the .wav files. At this point they can be compressed for the purpose of saving space, to AAL. Some weird thing prevents wave files from being transcoded to AAL sometimes, but I haven't discovered what it is. The above sequence seems to work reliably.

I didn't know I could...assuming you are speaking of an option in SS. If so, do I have to work primarily with an uploaded file, or can you do this with a file contained on a disc when the RH1/M200 is connected via USB. I will experiment a little when I hear back from you.

Either way, from my experience over the years, deck editing is so much tighter than via software as I found when I had a PC3 a few years ago. But sure, having the ability to use SFE options on non-SP recordings in any form would be great.

[sfbp]

I didn't know I could...assuming you are speaking of an option in SS. If so, do I have to work primarily with an uploaded file, or can you do this with a file contained on a disc when the RH1/M200 is connected via USB. I will experiment a little when I hear back from you.

Either way, from my experience over the years, deck editing is so much tighter than via software as I found when I had a PC3 a few years ago. But sure, having the ability to use SFE options on non-SP recordings in any form would be great.

Sorry to disappoint you - it's all done with WAV files. I have SS set to always produce WAV on upload (saves a lotta accidents!) and when I need to play with them, there they are.

The neatest thing I can do with CoolEdit (aside from FFT noise reduction which is a neat trick if you do it right), is to REMOVE the characteristic crackles from LP recordings. It always makes me laugh when I upload a recording from some cassette that I own, only to see the telltale signs of life as an LP that even the manufacturers didnt get rid of, that I suffering from audiophile OCD feel obliged to get rid of.

(hint: when you delete 3 msec of sound, no ear will ever tell the difference, and most crackles are way smaller, well below 1msec)

[Avrin]

And of course when something goes from 650MB to 300 (as you will find with AAL) then something got thrown away. We all know that zipping sound files doesn't reduce their size.

WHAT???!!!

[sfbp]

Yep. Took wave files and they reduce to approx 40-50% of the original when coded to AAL. Of course this corresponds to 256 or 352 kbps - but I noticed there was little difference between those two at least.

If you run LZH compression on a WAV file you get maybe 5% compression. Check?

Maybe it's late in Moscow......

[Avrin]

I don't currently have an LZH archiver configured in my system. But I do have some others. Let's take the most famous Brian Eno track, called "The Microsoft Sound.wav", which is 135,876 bytes long, and compress it using various archivers from the command line with their default settings. That's what we get:

The Microsoft Sound.ace - 33,959 bytes

The Microsoft Sound.bz2 - 45,634 bytes

The Microsoft Sound.cab - 60,556 bytes

The Microsoft Sound.gz - 63,272 bytes

the microsoft sound.ha - 45,113 bytes

The Microsoft Sound.imp - 61,377 bytes

The Microsoft Sound.j - 64,921 bytes

The Microsoft Sound.rar - 35,049 bytes

The Microsoft Sound.z - 73,702 bytes

The Microsoft Sound.zip - 62,015 bytes.

The CAB here is Microsoft CAB, and the Z is InstallShield 3.00. And I only used archivers supporting long file names.

All the above files perfectly decompress to the original WAV file.

Now for some lossless audio stuff (APE using Normal compression, and FLAC using default settings, both from the command line):

The Microsoft Sound.ape - 38,703 bytes

The Microsoft Sounf.flac - 46,324 bytes

These two files perfectly decompress to the audio part of the original WAV file, but lose the final 826 bytes containing copyright, author, and other non-audio information.

And there is absolutely no way to compress this file to AAL directly, since it is 22,050 Hz, 8-bit, Mono, while AAL only supports 44,100 Hz, 16-bit, Stereo.

Edited November 26, 2008 by Avrin

[Strungup]

Ooopps , he done ticked of Da Russian , bes be watchin da back door mon !

Avrin be knowing a ting or 2 now . Ya know !

das ma rasta mon .......

[sfbp]

Ticked off? Wouldn't you expect a computer-generated sound to be more compressible? Avrin is just playing devil's advocate shoorley? :)

Funnily enough I did just get a Winzip upgrade that promises to do better on jpegs, maybe it does better on sound too?

added: I have a hunch this is something to do with the sound being in Mono. I just took some white noise (actually a background I used to subtract out from a particularly noisy recording) and it goes from 9.8M to 8.5. Ok, more like 10 or even 15% but nowhere near half. Real music seems to compress hardly at all)

[Avrin]

The compression ratio doesn't depend on files being mono or stereo. A mono file is already half the size of a "stereo" file in which both channels are the same.

The actual lossless compression ratio (and also the quality of lossy compression) depends entirely on the complexity of audio material. Styles like jazz or trance are quite hard to compress, while most classical music compresses much better.

For example, "Take Ten" by Paul Desmond (the track I'm using to test almost everything) occupies 33,833,564 bytes as a WAV file, 22,825,780 bytes as a RAR archive containing the WAV file (compressed with default settings), and 20,117,246 bytes as a FLAC file (also compressed with default settings).

[sfbp]

Fair enough. But I have continued to sample "randomly selected" wave files from my (almost 100% classical) collection - and the average is only 10% reduction in size.

The point about mono is that if both channels are exactly the same (ie dual stereo), it will compress more. I just observed this. Mono cannot be burned to CD at least for most current CD players to read (I still have a busted CD670 from Phillips that did) so you have to provide a stereo file to Nero. Sure enough, the files zip to about 50%. Granted I didn't actually NEED to have a WAV file that was 2x as big, but that is the default I think, for most tools, and indeed SonicStage itself.

Which all tends to confirm my point. I just manufactured a mono wave file and it was exactly half the size of the dual stereo one. On (Winzip) compression, it went down by only about 9.8%. Whereas the stereo file was only a little bit bigger than the mono file uncompressed.

[Avrin]

WinZip (like the entire ZIP family) is not the most efficient compressor. Try WinRAR from http://www.rarsoft.com. I have seen it providing a 10 times better compression in its native RAR format, than WinZip on simple Microsoft Office documents (200 kb v. 2 Mb). It can also create ZIP files, in addition to its native RAR format, plus it opens and decompresses 12 more archive formats WITHOUT the need for any external programs.

Edited November 27, 2008 by Avrin

[sfbp]

WinZip (like the entire ZIP family) is not the most efficient compressor. Try WinRAR from http://www.rarsoft.com. I have seen it providing a 10 times better compression in its native RAR format, than WinZip on simple Microsoft Office documents (200 kb v. 2 Mb). It can also create ZIP files, in addition to its native RAR format, plus it opens and decompresses 12 more archive formats WITHOUT the need for any external programs.

It also happens to be the only utility I have ever caught any kind of trojan/virus from. So I have it but will only use to decode things that I cannot any other way, and it's only on a single machine.

The clue to the suggestion in my previous post was the word "mono" in yours. Go on, try it......

Meanwhile I do agree with you that WinRar does compress ordinary (i.e. not dual-mono) wave files quite a bit better, somewhere between 30 and 40%. I hate the interface, which at least in my current version is poorly integrated with Windows (compared to WinZip). It also appears to be quite a bit slower - you don't get something for nothing. Maybe it's time to install the newest Winzip, I have it but am just very slow to change everyday tools.

Cheers

Stephen

[Strungup]

I hate compression , I just go get a BIGGER hard drive LOL

documents , pix , a Folder of pix , ....... yes compress the heck out of it so I can email it .

But My Music is on MD ...... so why store it on my Drive ???

stopped counting my discs 6 years ago , toooo many .

[MDane]

But My Music is on MD ...... so why store it on my Drive ???

stopped counting my discs 6 years ago , toooo many .

That (in bold) is exactly the reason. Plus it gets a little messy for car use when you try to carry many discs at time. Also, somehow various discs would always dissapear like socks in the dryer - often my favorites. Finally, the advent of the stopwatch (Psyc) flash units a couple years ago sealed the deal for me...an actual excercize portable that plays most of the current Atracs that can be had for under $20. So having one's whole music / MD collection on the HD opens up many new doors.

Once everything is stored on the HD, you can then dole out files for all of your various equipment, even back to MD if desired, back it up, share between computers, etc. In other words, with the right Sony components, you can actually have the same flexibilty as those Smipod guys, but with actual harware based editing/recording, and better sound quality.

Don't get me wrong, there is nothing wrong with keeping it simple having, say a home deck and car deck (or portable) and lots of discs (I did that for nearly a decade). But for very little additional money, one can increase their music flexibility while still staying with true to the Atrac/MD format to some extent.