emeb

Hear Hear! Also, if you could post before/after upload copies of the trkidx??.hma file from the hmdhifi directory that would help us understand any new formats & flags that the new units support.

I haven't had any luck whatsoever booting off of my MZ-NHF800. I've got two systems: * an AthlonXP with a 3-year old mobo that has USB-ZIP as a boot option * an Athlon64 with a 2-week old mobo that has USB-ZIP, USB-CDROM and USB-Floppy as boot options. Neither of these systems can boot off of the MD with any of the USB options, although I've reformatted the disc and tried installing both a bootable MS-DOS and a bootable Linux system. They don't even seem to see the disc - on one of the systems it doesn't even start spinning until after the system has given up. I tried following the instructions on the weethet.nl site without any noticable differences.

I've only got a 1st-gen Hi-MD so I can't be certain, but I doubt that would do anything interesting. In order to load MP3 data in, you'd need to store it in the atdata??.hma file and create a track record for it in the trkidx??.hma file. There are enough checksums/keys in the track record which we don't know how to compute that the MD device is almost certain to recognise the data as invalid and refuse to play it. One thing which would be useful: The information we currently have on track types/sources doesn't include the MP3 formats from 2nd-gen devices. Anyone who has a 2nd-gen device and could post a copy of the trkidx??.hma file from a disc with MP3 tracks along with a text description of the tracks could help out the general knowledge. (FWIW, this is one of the posts that got deleted in the great dex Otaku incident of 2005-07-13. I've restored it pretty much as I remembered it was with some extra content)

No doubt! I'd like to think that I don't have any illusions about how hard that would be. As I mentioned in the very first post of this topic, the only chance we have is if the encryption that's used on LPCM is somehow weaker because it originated in the MD recorder instead of on the PC. It's a long shot, but possible if you consider that Sonicstage appears to re-do the encryption when you upload. If it were as strong as the PC-based encryption they might not have needed to reencrypt.

I didn't have to - I recorded silence, so the data in the WAV file was 00 00 00 01 ff ff ff fe (basically just slight variations +/- a few LSBs about 0), etc. whereas the data in the LPCM blocks was completely random. It's pretty easy to tell the difference without bringing endianess into the picture. If you'd like I could post some text snapshots of the hex dumps and everyone can have a looksie.

Yes, it's that easy. For example, follow this link: Patent # 5796839 You can see the entire text of the patent, as well as any figures that were part of the submission. Beware that it's written in legalese (you'll get used to reading the phrase "skilled in the art") and patents in general try to be vague (you don't want to give the competition too much help). No, we don't know what's used in Hi-MD, although it's pretty safe to assume it's MagicGate since that's what Sony knows and uses elsewhere. I have no experience with attacks on encryption so I can't speak to the process or the likelihood of getting results. It's probably safe to assume that it won't fall over at the touch of a feather though - the folks who put this together aren't stupid and have been working on it for a while.

dex Otaku's comment about patents turned a light on in my head. The US Patent database is online and searchable at USPTO. Here's what turns up when you search for Sony and Encryption: PAT. NO. Title 1 6,504,930 Full-Text Encryption and decryption method and apparatus using a work key which is generated by executing a decryption algorithm 2 6,363,148 Full-Text Method, apparatus and computer program for activating an alternate encryption using an identifier embedded in data 3 6,223,285 Full-Text Method and system for transferring information using an encryption mode indicator 4 5,796,839 Full-Text Encryption method, encryption apparatus, recording method, decoding method, decoding apparatus and recording medium These might be good starting points for finding out more details. #4 looks particularly germane to our topic.

Thank you for keeping an eye out and raising the issue. It's fun to talk about a wide range of subjects here, but keeping the forum open is primary and we have to make sure that we stay safe. This isn't the EFF/ACLU and I expect that neither the minidisc.org operators, nor we participants can afford to defend against any legal objections to what we do here. We're not blackhats and don't want to come across as such. Just enthusiasts with healthy curiosity. Let us know if you think we're getting too close to the line.

I agree that the copyright/DRM issues are tricky and we'll have to exercise some good judgement in that area. As far as the patented algorithms are concerned, I think we're safe there. Even in these United States, reverse engineering isn't illegal (yet). Exposing the workings of patented algorithms is also OK - that's what patents do! It's when someone uses that information for commercial gain without proper licenses that the liabilities open up. For example, the MP3 encoding/decoding process is patented by Philips/Fraunhofer, yet publishing details of how MP3 works is not illegal. Using that information without license however is, which is why the LAME encoder is sometimes disputed.

<IANAL> MagicGate applied to content copyright by someone else would be specifically covered by the DMCA, and so under US law cracking it to get access to someone else's content is illegal. It gets into a kinda grey area if you're cracking it to get access to material you have copyright on - specifically recordings you have made of sounds that you have permission to record (or which you needed no permission to record). However, if in the process of cracking it for potentially legal access, you also make illegal access possible you'd probably be liable. </IANAL> As you noted this is a tricky area.

rombusters: Regarding checksums - I think the checksums for each track's data are done only over the idividual data record for that track and is stored with the data record in the trkidx??.hma file. There are still a number of data fields in the trkidx??.hma file that aren't understood and which change when the track is uploaded, so those are the ones I suspect. ksandbergfl: I did do a statistical analysis of the raw PCM in the WAV file (excluding the non-data chunks) - it's very correlated whereas the OMA and LPCM data is almost completely uncorrelated. rombusters and dex Otaku are correct - once you strip off the header from a WAV file, it's just binary data that can be viewed as an array of L/R alternating ?-endian data. Very simple to pull apart, and very easy to recognise as PCM, even by eyeball within a hex editor. The same cannot be said for the OMA files and LPCM data in the atdata??.hma file, both of which are highly randomized. This gives us three challenges: 1) Figure out the checksum/hash algorithm used to validate the trkidx??.hma data records. If you can figure that out then you can reset the upload status of LPCM tracks. 2) Figure out the session validation handshake that's used to protect the key exchange between SonicStage and the MD device. If you're interested in what this means, google for 'Diffie-Hellman' and skim through what you find. 3) Figure out what the MagicGate encryption algorithm is. Just looking at the data blocks in the atdata??.hma file isn't particularly revealing, except that it seems to work on ~4kB blocks, uses an 8-byte state vector that's preserved from block to block and appears to have a number of additional 8-byte public keys. Sony hasn't publicly revealed much about it (security through obscurity), so this is a pretty big hurdle. Looks like a full plate

Not so much error correction as validation of the track data. By validating the checkout status with checksums computed by some keyed algorithm they can ensure that users don't fiddle with the track database and grant themselves additional uploads, etc. Error correction is about identifying mistakes and fixing them. This is about DRM where the goal is to identify differences and deny access. Similar processes with different goals. DRM assumes that the underlying data channel is reliable enough (ie doesn't require error correction) that users won't be denied access unless they're trying to do something they're not allowed to do.

No, you're right - it's all in the trkidx??.hma which contains a database of information for the disc, including the following (that we know of): * Number of tracks * Disc name * group names and per track: * track codec (LPCM, various ATRACs, etc) * track source (pc, analog in, optical in, etc) * Which data blocks in the atdata??.hma are part of this track * which group this track is in * title * artist * album * Keys, checksums & checkout status

Nope - trkidx??.hma is not encrypted, although it does appear to contain keys/hashes which are used to control access to the audio data. The various listing programs out there (three that I know of: HIMD-Lister, HIMD-Xtract and one other) all parse the trkidx??.hma file for details of the tracks and no fancy coding is applied.

Not all of the comms between sw & device are encrypted, just the initial key exchange. Once that handshake is complete & the link is established, the actual data transfer is clear. Except of course that most of the data that's being uploaded is _already_ encrypted (the contents of the atdata??.hma file). Looking at the USB logs I can see the trkidx??.hma file being read multiple times, as well as several of the other files too. Curious why they need to read trkidx??.hma so often (probably half-dozen times in one session!) Your conclusion is correct though - they're definitely belt-n-suspenders when it comes to the DRM.

Yes, it would. I don't think it's unit-specific, but rather disc-specific. There's some sort of media key that is stored in an invisible area (eg non-FAT) of the Hi-MD disk and is sent via USB to SonicStage when the LPCM data is uploaded. SonicStage uses that media key to decrypt the LPCM data and then re-encrypts it when it saves it as an OMA file on your PC. The LPCM data on the Hi-MD disc is marked as 'uploaded' (but otherwise not changed), so you can still play the disc in any Hi-MD unit. The re-encrypted OMA file remains on the PC and isn't played on the Hi-MD unit, so that doesn't break anything. The only unit-specific encryption that's taking place is in the key-exchange handshake between SonicStage and the Hi-MD recorder, and that never alters the audio data - it just validates the communication channel between the PC and the Hi-MD unit.

Thanks for the encouragement. I'm not super skilled at this either, but if we can keep banging on it who knows what will turn up. Re: resetting the upload count - that would be a great development. I guess the trick is to grab a copy of the trkidx??.hma file before uploading and afterwards try resetting the fields that changed. My guess is that they're checksummed/hashed in some way that will allow SS to notice that you've hacked it. Since we don't know what the exact authentication hash is it will be hard to set them back. I remember reading some earlier postings where folks tried this and failed, but then they weren't hitting all the changes either. Latest developements: I grabbed a copy of usbsnoop and logged an LPCM upload session. Not surprisingly, you can see the same LPCM blocks in the USB transactions that are visible in the atdata??.hma file - byte-for-byte. The trkidx??.hma file is in there too, so that's a pretty big clue that there's no decryption of the data taking place in the recorder during the upload process - it's all handled in the PC by SonicStage. I was peripherally involved with some of the libNetMD hacking that took place back in the 505/707 days. Those interfaces used completely proprietary commands. Hi-MD is different though - it seems to be USB-Storage centric. Next I need to parse through the detailed handshake that takes place during authentication to see what kind of setup there is. I'm curious if the key exchange & handshaking takes place within the context of a USB-Storage device, or if they have some proprietary commands. It would probably be worthwhile to log the USB transactions that happen while playing from the MD to PC - I'm betting that the data is still going over the bus encrypted. I did a little web research on Sony's MagicGate DRM. There's not a lot of technical detail out there, but they do claim that it is a hardware block that's built into their players and memory sticks. The PSP supports both AES and MagicGate (named separately) so I don't think that MagicGate is necessarily AES. What we need is some participation from the guys who have experience with the WinXX DLLs (ie marcnet and fishstyc) to give some hints on where to look for the LPCM decryption stuff.

bjsilva: Thanks! A MacOS X app would be cool (I've got a G4 iBook). Hopefully we can get enough eyeballs looking at this problem that some ideas will come out and we can make some progress. For now I'm just poking at it and hoping to stimulate some interest. I know there's some folks out there that have worked on this recently (fishstyc, ksandberg, fuz, marcnet, streaml1ne, ksandbergfl, etc) - hopefully they're not burned out yet! emeb

I did some statistical analysis on the first LPCM data block from an analog recording using the 'ent' program (find it here). I took the data portion from the first block of the track from the atdata??.hma file, the first 16320 bytes from the WAV data chunk and the first 16320 bytes of encrypted data from the corresponding .oma file and saved all three to separate binary files. Ent reported the following statistics: atdata???.hma: Entropy = 7.988250 bits per byte. Optimum compression would reduce the size of this 16320 byte file by 0 percent. Chi square distribution for 16320 samples is 268.05, and randomly would exceed this value 50.00 percent of the times. Arithmetic mean value of data bytes is 127.9270 (127.5 = random). Monte Carlo value for Pi is 3.130882353 (error 0.34 percent). Serial correlation coefficient is -0.002103 (totally uncorrelated = 0.0). .oma file: Entropy = 7.988036 bits per byte. Optimum compression would reduce the size of this 16320 byte file by 0 percent. Chi square distribution for 16320 samples is 272.09, and randomly would exceed this value 25.00 percent of the times. Arithmetic mean value of data bytes is 127.4705 (127.5 = random). Monte Carlo value for Pi is 3.161764706 (error 0.64 percent). Serial correlation coefficient is 0.020758 (totally uncorrelated = 0.0). .wav file: Entropy = 4.477646 bits per byte. Optimum compression would reduce the size of this 16320 byte file by 44 percent. Chi square distribution for 16320 samples is 1201580.33, and randomly would exceed this value 0.01 percent of the times. Arithmetic mean value of data bytes is 60.4253 (127.5 = random). Monte Carlo value for Pi is 3.319117647 (error 5.65 percent). Serial correlation coefficient is 0.497866 (totally uncorrelated = 0.0). Basically what this is telling us is that the randomization on the raw LPCM blocks read from the atdata??.hma file or the encrypted data in the .oma file is pretty damned good. There's not a lot of structure in the byte stream to get any analysis hooks into.

Here's some additional detail based on inspection of the OMA and WAV files derived from uploading LPCM tracks: Hi-MD LPCM Behavior ------------------- * trkidx??.hma changes when uploading: - in HEADER_INFO_LIST_ADDR (0x8050) area: > SOURCE: (uint32 @ offset 0x00) + Tracks that haven't been uploaded have source = 0x5DEF8DDD + Tracks that have been uploaded have source = 0x5F9E0000 > Unknown field (unint64 @ offset 0x18) + changes > Unknown field (uint16 @ offset 0x4e) + Tracks that haven't been uploaded = 0x0308 + Tracks that have been uploaded = 0x0044 * No changes are made to the ATDATA??.HMA file as a result of uploading * OMA files derived from LPCM tracks bear no resemblance to raw data in the ATDATA??.HMA file. It's suggested earlier that LPCM is decrypted/re-encrypted during conversion to OMA. * OMA LPCM files are exactly 3124 (0xc34) bytes longer than corresponding WAV files. - WAV RIFF header is 44 (0x2c) bytes to start of data so OMA header is 3168 (0xc60) bytes. - At offset (0xc60) in OMA file, randomized data begins. - Dividing WAV data length by block count from trkidx??.hma file gives 16320 bytes/block. > [AA BB CC DD EE FF GG II] sequence appears to be real data that is repeated in header of following block to initialize encryption state. * data chunk of WAV files derived from LPMC OMA files begins with 8712 (0x2208) bytes of 0x00 data. Possibly startup delay of recording process. Make of that what you will. Interesting to note that many encryption related fields are 8 or 16 bytes. 16 byte fields suggest 128-bit algorithm (AES?) but I don't know what 8 means.

1kyle: Using something like Linuxant's NDIS wrapper is an interesting idea. There are a few challenges with it though. The NDIS network card interface under WinXX is a well known API intended to allow cross-vendor compatibility. Sony's driver spec for Hi-MD devices is entirely proprietary and unpublished, so getting their API working on Linux would require a massive software reverse-engineering effort, on the order of Wine. If you'll recall the huge number of files that are scattered across you WinXX system when installing SonicStage I'm sure you understand the scope of this effort. Obviously, the means to decrypt the the LPCM stream exists somewhere in that mass of code, but teasing out where and how it happens could take years. Marcnet has figured out a lot of this background and would be much better qualified to comment on it than I. The thing we have going for us on Hi-MD is that much of the information we're interested in is available through the open standard access method of the USB-Storage & FAT filesystem. Simply mounting a Hi-MD device gives us the encrypted LPCM data. The trick of course is that there are apparently side-channel data streams that aren't available just by reading the contents of the hmdhifi directory. Others on this site have done USB traffic snoops and there appears to be a fairly complex 'secret handshake' going on when the Sony software talks to the Hi-MD device. It seems clear that these hidden exchanges of data are necessary to get all the information needed to decrypt the LPCM stream, and could be essential to any non-Sony/non-WinXX access solution. If we are able to puzzle this out, it may be easier to use something like libusb on Linux to recreate the essential parts of this handshake without requiring the Sony drivers.

This is a bit technical... I've been spending the afternoon playing with fishstyc's HIMD-Xtract code. I've converted it over to POSIX compliant C and have it running on my Linux box. My (probably hopeless) goal is to get a way to load LPCM recordings from an MZ-NHF800 onto my Linux PC _without_ going through SonicStage and WinXX. The rationale is this: 1) I don't care about ATRAC encoded tracks since those originate entirely from sources I already have as cleartext (CDs). I don't use Sony's Connect site to buy tracks and have no interest in reverse-engineering the ATRAC formats. 2) LPCM Analog recordings never went through SonicStage, so the encryption applied to them has to originate entirely in the MD recorder. There's a chance that the encryption used in this case is not as strong. 3) Since it's possible to convert LPCM to WAV in SonicStage and PCM is a fairly simple data structure, it gives us a chance to see both the ciphertext and cleartext. That is one of the holy grails of any code breaking process. 4) I suspect (but can't be certain - IANAL) that this activity is not a violation of the DMCA since I'm only trying to get access to recordings that I have personally made. I'm not interested in violation of copyrights. fishstyc's code has made some assumptions about the structure of the data in the atdata??.hma file which are sufficient to accurately recreate OMA files that can be played by SonicStage on the originating computer. I'm not sure if he's had any success with analog LPCM files, but I do disagree with the details of the LPCM blocks as described in the HIMD-Xtract code. Here's what I've puzzled out: Hi-MD LPCM Block Structure -------------------------- 16384 Bytes/Block (0x4000) 32 Bytes Header: ---------------- 4c 50 43 4d 00 00 01 22 00 00 00 00 00 00 HH LL | LPCM..."......hl MM NN OO PP QQ RR SS TT AA BB CC DD EE FF GG II | mnopqrstabcdefgh [16312 Bytes Data] 40 Bytes Footer: ---------------- ?? ?? ?? ?? ?? ?? ?? ?? AA BB CC DD EE FF GG II | ????????abcdefgh MM NN OO PP QQ RR SS TT 00 00 00 00 00 00 00 00 | mnopqrst........ 4c 50 43 4d 00 00 01 22 VV WW XX YY 00 00 HH LL | LPCM..."vwxy..hl Where: [HH LL] is a 2-byte block counter (Note that this gives an upper bound on the longest possible Hi-MD track of 1 GB). [AA BB CC DD EE FF GG II] is an 8-byte sequence that is identical in the footer of one block and the header of the following block. It is all 0 for the first block in a track. Possibly some sort of encryption initial state. [MM NN OO PP QQ RR SS TT] is an 8-byte sequence that is common to all blocks in a track, but is different from one track to another. [VV WW XX YY] is the last 4 bytes of the Key found in the trkidx??.hma file. Comments & criticism appreciated. PS: I'm in _awe_ of those of you have figured out the format of the trkidx??.hma file. Thanks to fishstyc and anyone else who contributed to puzzling that out. Is there any wisdom out there about the contents of the other files in the hmdhifi directory?

bri: I tried the full download from the Downloads section of the forum - it aborted complaining that I needed to download the WMP9 codecs. (I already have WMP9 installed). The online installer gets through the download, extraction and part of the component install (OpenMG) before terminating during the SonicStage app install. kurisu: I've got an AthlonXP1700 with 640MB DRAM on an ECS mobo running Win98SE. Plenty of HD space in both C: and D: partitions. Latest patches from MS, WMP9, Directx9. I am able to install and run Sonicstage 2.1 and SimpleBurner from CD w/o trouble. Could it be that the SS3.1 update is looking for a previous install of SonicStage? I had uninstalled everything and cleaned the registry prior to trying to install it.

Well, Sonicstage 3.1 has become unreliable. Running the application causes all manner of errors dialogs to appear, so I've uninstalled it. I tried re-installing 3.1, but the online installer terminates before completing. I've dropped back to the version that came on the original installation CD with the MD and that seems to work fine. Also, now HiMD Lister cause my machine to bluescreen. Meh! Eric

The timestamps in the zipfile I sent you are identical because of the way I prepared the data (copying to the hard-drive & editing before zipping). On the actual HiMD disk the timestamps are as follows: -rwxr-xr-x 1 ericb ericb 32768 Jan 1 2004 _mdhifi.hma -rwxr-xr-x 1 ericb ericb 32768 Jan 1 2004 mclist01.hma -rwxr-xr-x 1 ericb ericb 172 Jan 1 2004 _0010012.hma -rwxr-xr-x 1 ericb ericb 172 Jan 1 2004 00010012.hma -rwxr-xr-x 1 ericb ericb 65536 Jun 25 15:10 text_g01.hma -rwxr-xr-x 1 ericb ericb 327680 Jun 25 15:11 trkidx01.hma -rwxr-xr-x 1 ericb ericb 32768 Jun 25 15:12 mclist02.hma -rwxr-xr-x 1 ericb ericb 327680 Jun 25 15:12 trkidx02.hma -rwxr-xr-x 1 ericb ericb 36044800 Jun 25 15:12 atdata02.hma So there's 1 minute difference between the two trkidx??.hma files I'll try forcing the lister to use the newer one on my end to see what happens. Eric