HiMD Transfer for Linux, working

[cbmuser]

Hi folks,

we have unveiled the encrpyption of MP3s on Sony HiMD-Walkman, soon we'll have a transfer software for linux. We're already able to transfer tracks off the HiMD to the PC, download to HiMD won't take so much anymore (we already know how). Please see our wiki:

<https://wiki.physik.fu-berlin.de/linux-minidisc/>

Oh, and no, it's not a fake, it's really working .

Edit: It would be nice, if anyone here with some programming knowledge would be willing to join and help us !!

Adrian

Edited April 9, 2009 by cbmuser

[sfbp]

Verrry interesting. I might be willing. I am intrigued by several properties of the way Sony "protects" tracks. In particular one oddity is that changing the owning group (using SS) prevents further editing on the HiMD portable.

Another thing I would like to know is about the NetMD - everyone swears it cannot be done, but we manage to alter headers no problem even on a no-upload disc. So if we can read headers why not the data too?

Looking forward to reading your wiki

[cbmuser]

Another thing I would like to know is about the NetMD - everyone swears it cannot be done, but we manage to alter headers no problem even on a no-upload disc. So if we can read headers why not the data too?

NetMD is using its own proprietary USB-protocol whereas HiMD uses USB mass-storage. The whole code for communicating with the NetMD device is protected against debugging/disassembling (residing in the .ocm-files in Sony Shared-Folder), the audio-data itself is encrypted with 3DES. One of the guys in our project has been working on this for years, we're concentrating on HiMD first though.

Looking forward to reading your wiki

Please drop me an email (see link in the wiki) to request a wiki-account to get access to all information, including those on NetMD.

Adrian

[cbmuser]

Hi,

just to show you the current status of the Qt HiMD application, I attach one screenshot. Feel free to retrieve the source from git and build and test the things yourself.

Adrian

[cbmuser]

Hi folks,

the software "QHiMDTransfer" supports uploading of MP3s on all newer HiMD-models. Please test as much as you can.

All infos can be found on the linux-minidisc wiki.

Adrian

[bobt]

Hi folks,

the software "QHiMDTransfer" supports uploading of MP3s on all newer HiMD-models. Please test as much as you can.

All infos can be found on the linux-minidisc wiki.

Adrian

This sounds great, is there a version for windoz?

A of of people thank you for your work

Bob

[cbmuser]

This sounds great, is there a version for windoz?

A of of people thank you for your work

Bob

Oh, you want a Windows-version ?! . Well, I haven't compiled a version for Windows yet. I have working versions for Linux and MacOSX (Intel/PPC) so far. I can upload everything tomorrow if I find the time to.

On Linux (Ubuntu/Debian) it's very easy to compile everything from source:

1. Install necessary packages (together with their dependencies):

apt-get install git-core build-essential automake autoconf gcc  libqt4-dev libglib2.0-dev libmad0-dev

2. Fetch full source-code from linux-minidisc:

git clone git://z6.physik.fu-berlin.de/linux-minidisc

3. Build libhimd and QHiMDTransfer:

cd linux-minidisc/libhimd
./autogen.sh
make

cd ../QHiMDTransfer
qmake
make

Building on MacOS-X is slightly more difficult, you'll need Qt-Creator, Qt Open Source Edition 4.5.0 for Mac, Macports with glib2-devel, libmad-devel and git-core.

Adrian

Edited April 21, 2009 by cbmuser

[bobt]

Thanks for all your work, again, you have made a lot of people happy

Bob

[cbmuser]

Seems nice; how do I connect my Hi-MD Walkman? I.e. in which directory am I supposed to find it? It has connected to the computer in some way, as the screen displays the connected to the PC animation & then screen.

Sorry if it is obvious, but I am quite new to Linux - Ubuntu.

When you connect the Walkman, a windows should pop up showing the directory contents of the Walkman. Then start QHiMDTransfer in the directory it was compiled with "./QHiMDTransfer". In the software, click the button connect and choose the main-directory of the Walkman (it ususally lists as "1 GB Storage Device"). You'll have to play around a bit until you understand everything if you're new to Ubuntu, but it's actually quite easy. Future versions of the software will detect the Walkman automatically and do all the work for you, but we're still "Work in Progress".

Adrian

[tekdroid]

This excites me.

If a data-recovery tool can be built from something like this, all the better.

Keep up the good work

[cbmuser]

This excites me.

If a data-recovery tool can be built from something like this, all the better.

Keep up the good work

Yeah, the ultimate goal is to provide all possible upload/download-functionality provided by the (Hi-)MD-Walkman. We already know how many of the audio-data is encrypted on the discs, but we need more support from other people to write code or reverse-engineer things.

If you want to know how backup could be possible, read the pages on our wiki.

Adrian

[Taliesin]

Hi! I have became excited with this project. Sadly, I haven't any program knowledge, so my contribution will be just to encourage

I have a question that isn't very clear to me: you have gotten transfer only mp3 format files or also atrac ones?

Keep the flame on!

Yeah, the ultimate goal is to provide all possible upload/download-functionality provided by the (Hi-)MD-Walkman. We already know how many of the audio-data is encrypted on the discs, but we need more support from other people to write code or reverse-engineer things.

If you want to know how backup could be possible, read the pages on our wiki.

Adrian

[Taliesin]

I hope LPCM transfers will be supported asap!

Uau, that would be a great option too!!!

[cbmuser]

Hi folks,

at the moment we only support uploading of MP3s from HiMD-Walkman. We also already know how to format an HiMD from Linux and setting time/date (we have reverse-engineered all Sony SCSI-commands for the HiMD).

Next will be support for uploading PCMs, then downloading MP3s and PCMs. We have started with the MP3s because they use a very weak encryption (XOR-based) while PCMs are encrypted using 3DES. But we have the master-key for 3DES, so expect PCM-upload capability soon. Although we know how to encrypt MP3s for the MD-Walkman, we haven't started on downloading tracks to the device yet, since we do not want to provide code yet which can destroy your audio-data . But we have the knowledge, we just need manpower for coding.

We're still working on the encryption of the PCMs, they are less encrypted on some HiMD-devices. For example, the Sony MZ-RH1 uses a very simple encryption for PCMs recorded with the device itself whereas it uses strong encryption for PCMs transfered to the RH1 with SonicStage. That's why you can only upload PCM-tracks with the Mac-Software which were recorded with the RH1 itself but not with SonicStage.

I cannot say anything about ATRAC yet, but I will keep you informed as soon as I know more.

Adrian

PS: You may also support with fundings if you want, we have a PayPal-donate button .

[Taliesin]

Hi cbmuser. Thanks for your answer. I hope ATRAC support could be a reality sooner or later cause this is the only format I use except PCM , but seeing mp3 support is a great thing too!

[baturjan]

Thank you! If I had the technical skill to help, I definitely would.

Looking forward to its development!

[cbmuser]

Hi,

just for a short update. Our project has received its first donation over 10 Euros .

We are also now working on NetMD transfer, studying several Sony patents for that and writing sample/test code.

Also PCM-Upload for HiMD is work-in-progress now.

Adrian

[cbmuser]

Hi guys,

just a small update:

We have 3 people now working on Reverse Engineering Sony's OpenMG which manages all encryption and content-management on the NetMD/Hi-MD-Walkman. The OpenMG-Modules come in a special binary-coded format named OCM, whose files reisde in C:\Program Files\Common Files\Sony Shared\OpenMG. Once we have understood the OCM-format, we can disassemble all the code for enrcryption and download/upload to/from the MD-Walkmans. This Reverse Engineering process takes more time, but in the end we will be able to support all possible transfers with the devices.

Btw, one of the ffmpeg-developers who wrote the ATRAC3-decoder for ffmpeg has been in contact with us. He wants us to share code with him later on and vice versa, so we can integrate full ATRAC1/3-support.

Stay tuned and thanks for all your supprt. We have collected 30 Euros of donations already,

Adrian

[Strungup]

hmmm , This is getting very interesting , I cant wait for a DMG, cased version for the Mac ( I am on Tiger , Macbook Pro C2D ) I would be willing to test it . this is indeed interesting.

[baturjan]

If you need testers for the Mac, I'll do it.

MacBook 2.4, OS Leopard 10.5.6.

[cbmuser]

Hi,

I have a working version for MacOS which I compiled on my Mac Mini (Leopard 10.5.6/PPC). I can upload it for you to test it, but it works already (except for some bugs).

EDIT:

I have uploaded the version here:

http://users.physik.fu-berlin.de/~glaubitz/qhimd_mac_ppc.zip

You will need to install Qt 4.5.0 or higher and Macports with libglib2 and libmad for this application to work at the moment.

Qt for Mac: ftp://ftp.trolltech.com/qt/source/qt-mac-...ource-4.5.1.dmg

Macports: http://www.macports.org/install.php

Once both is installed, open a terminal and type:

sudo bash

<Password>

port install libmad glib2

Sorry for the inconvience at the moment, but everything is still in development and I am not so much into Mac development yet. On Linux, the compilation and the creation of ready-made packages is much more simple. I have to figure out, how to link the application statically, so you won't need to install the additional software. Any experts on MacOS-development are therefore welcome.

EDIT2:

I have written a short howto for compiling the software on Ubuntu and MacOS. Compiling on MacOS is still a bit tricky, but on Ubuntu it's very easy if you're following the instructions:

Ubuntu: https://wiki.physik.fu-berlin.de/linux-mini...ompilingonlinux

MacOS: https://wiki.physik.fu-berlin.de/linux-mini...=compilingonmac

Adrian

Edited May 10, 2009 by cbmuser

[cbmuser]

Hello,

update:

Upload of weak-encrypted PCMs (like they are recorded with the MZ-RH1) is now possible. First you need to build the latest version of libhimd from source, see: https://wiki.physik.fu-berlin.de/linux-mini...ompilingonlinux

Then you can upload tracks with:

./himdtest /mnt dumplpcm 1

to upload track no.1 of the HiMD which is mounted to /mnt. However, since the audio-data is in RAW-PCM, you'll have to convert it to WAVE:

sox -t raw -b 16 -e signed-integer -r 44100 -B -c2 stream.pcm stream.wav

PCM-Upload is not yet implemented in QHiMDTransfer. However, QHiMDTransfer will directly output WAV-files so you won't have to convert the files after upload.

PCM-Upload for older models and ATRAC3-upload will still take some time in development, as OpenMG needs to be reverse-engineered.

Adrian

[sfbp]

I'm totally up for the challenge of NetMD, my biggest problem is that I don't live in your linux/IRC world so the guys I want to talk to are never on line when I finally get around to **hobbling** over there. Also I would almost certainly want to develop/port back under Windows, which is good and bad.....

- bad because all the porting between OS's would have to be done (and adjusting for differences in C libraries and compilers)

- good because the majority of folks here and everywhere use Windows.

So I am not in a terrible hurry, as the porting of what you have done may be easier with something a bit more stable.

[cbmuser]

I'm totally up for the challenge of NetMD, my biggest problem is that I don't live in your linux/IRC world so the guys I want to talk to are never on line when I finally get around to **hobbling** over there. Also I would almost certainly want to develop/port back under Windows, which is good and bad.....

- bad because all the porting between OS's would have to be done (and adjusting for differences in C libraries and compilers)

- good because the majority of folks here and everywhere use Windows.

So I am not in a terrible hurry, as the porting of what you have done may be easier with something a bit more stable.

The code doesn't have to be ported, it is portable already. However, I am somewhat too stupid to setup the build-environment under Windows. You can try it yourself, if you're more into Windows-development.

Here is what you need:

- CygWin

- Qt 4.5.1 OpenSource for Windows (see ftp.trolltech.com/qt/source)

- Qt Creator 1.1.0 for Windows (see www.trolltech.com)

- libmad

- libmcrypt

- libglib

- git to checkout the code

You can install git, libglib and libmcrypt with the setup-program of CygWin, building is analog to Linux (same commands) in CygWin.

Once you really managed to setup CygWin with Qt properly in Windows, you should be able to compile everything. It's just QHiMDTransfer which I can't manage to compile, libhimd together with himdtest already compile. I will upload himdtest.exe.

Adrian

Edited May 10, 2009 by cbmuser

[sfbp]

Great. Today is impossible for me with commitments, but hope I can get going in about 24 hours from now.

[cbmuser]

Hi,

ok, I managed to build himdtest in Windows, get it here:

http://users.physik.fu-berlin.de/~glaubitz...imdtest-win.zip

Here a short howto which explains how to build libhimd on Windows with CygWin:

https://wiki.physik.fu-berlin.de/linux-mini...pilingonwindows

Adrian

[sfbp]

tyvm

Just a question: do you know if it would be better to try running on a machine that has NO SonicStage, USB drivers for MD etc. installed? Or would the presence of certain things in the environment put there (by Sony) actually make things easier for running the new stuff?

[cbmuser]

tyvm

Just a question: do you know if it would be better to try running on a machine that has NO SonicStage, USB drivers for MD etc. installed? Or would the presence of certain things in the environment put there (by Sony) actually make things easier for running the new stuff?

No, it does not make any difference, whether SonicStage and drivers are installed or not. Our software connects to the walkman over USB-Mass-Storage, it directly reads the files from the drive-letter, where Windows has mounted it to. You could also copy the files off a HiMD and put them elsewhere, libhimd/himdtest/QHiMDTransfer will still be able to upload (extract) files from the HMA-files.

Adrian

[cbmuser]

Hi folks,

as baturjan has pointed out to me in a mail, I'd say you guts what support we could need. Actually, we are collecting the money to buy an MZ-RH1 for testing. At the moment, we only have some older NetMD-Walkman and a Sony MZ-RH10. For full testing, however, we will need an RH1. Since we are more than one developer and we only have one HiMD-model at the moment, we'd be happy if anyone of you collectors is willing to share a HiMD-model or you could donate some money for the RH1 we want to buy (a used one off eBay is fine ).

Ah, and yes, our software will allow maximum upload/download-functionality provided that the hardware supports it. It will not impose any of the restrictions which SonicStage brings when touching any audio on the MD (disabling of Editing etc), whenever possible.

Adrian

Edited May 10, 2009 by cbmuser

[cbmuser]

Hi folks,

QHiMDTransfer (the GUI-application) can also upload unencrypted PCMs off a HiMD already. However those PCMs are in RAW-format and need to be converted afterwards:

sox -t raw -b 16 -e signed-integer -r 44100 -B -c2 stream.raw stream.wav

Upload of encrypted material is being worked on.

Adrian

[sfbp]

At the moment, we only have some older NetMD-Walkman and a Sony MZ-RH10. For full testing, however, we will need an RH1.

Adrian, if there is something I could test on my RH1, I would be happy to. I'm not in a position to send the unit itself, sadly.

[Syrius]

Hi folks,

as baturjan has pointed out to me in a mail, I'd say you guts what support we could need. Actually, we are collecting the money to buy an MZ-RH1 for testing. At the moment, we only have some older NetMD-Walkman and a Sony MZ-RH10. For full testing, however, we will need an RH1. Since we are more than one developer and we only have one HiMD-model at the moment, we'd be happy if anyone of you collectors is willing to share a HiMD-model or you could donate some money for the RH1 we want to buy (a used one off eBay is fine ).

Ah, and yes, our software will allow maximum upload/download-functionality provided that the hardware supports it. It will not impose any of the restrictions which SonicStage brings when touching any audio on the MD (disabling of Editing etc), whenever possible.

Adrian

How about we pool the money, send it to one of the members who is selling any of their RH1s, forward the money to them, and kill two birds with one stone that way?

Meaning, I pledge 10 dollars.

SFBP for example, pledges 5.

GuitarFXR promises another 7, and so on...

then we forward the money to whoever is selling an RH1, and everyone is happy?

[Strungup]

How about we pool the money, send it to one of the members who is selling any of their RH1s, forward the money to them, and kill two birds with one stone that way?

Meaning, I pledge 10 dollars.

SFBP for example, pledges 5.

GuitarFXR promises another 7, and so on...

then we forward the money to whoever is selling an RH1, and everyone is happy?

not a bad idea , at all .

[Taliesin]

Hey! Count me too! I have 10$ with the name of this project!

[sinus]

I'd spend another 10 euros!

[Syrius]

I officially pledge 10 dollars.

Which seller will have the honor of sending his/her RH1 to be used for this project?

[cbmuser]

Hi folks,

thats so great off all you guys. We have 40 euros already, so I think I will give the 40 euros to the guy then who is selling his/her MZ-RH1. Please just let me know once you have found someone.

Project is still making progress, however reverse engineering of OpenMG still takes some time. We are at a progressed point however.

http://techon.nikkeibp.co.jp/article/HONSH...427/169423/?P=2

"Sony hasn't taken open technology very seriously in the past. Its CONNECT music download service was a failure. It

was based on OpenMG, a proprietary digital rights management (DRM) technology. At the time, we thought we would make

more money that way than with open technology, because we could manage the customers and their downloads. "

Adrian

[Taliesin]

Uuuups! When I posted 10$ I meant 10€!!!!

Luckly you have understood me :

[cbmuser]

Hi folks,

there are 2 MZ-RH1 recorders for sale on eBay Germany at the moment. One of these units is for sale to local pickup only (seller claims to be an old grandpa, not being able to walk to the post-office). Anyway, we have collected 50 Euros already and would only need some more 50 Euros to get that walkman I think, I will pay the difference from my own pocket then. Please use our donate button on the frontpage:

https://wiki.physik.fu-berlin.de/linux-minidisc/

The MZ-RH1 will be made available to *all* developers involved. It will be shipped to any of us on demand .

PS: One guy from the German MiniDisc-Forum is working on building the software on Windows.

Adrian

[Syrius]

I'm donating my 10 USD today. Less than 10 Euros, but I'm glad to help.

Donated 10 USD.

CBMUser, I have a question. I don't use Linux. How will this benefit those of us who prefer or are stuck with Windows?