Questions/Comments for Marc's uploading utility.

[dex Otaku]

Himdrenderer seems to be getting better and better, although I noticed this afternoon (having rendered two 'test' tracks (about 1 minute each) of me playing the piano, that each contained a small electrical-interference-type click about 30 secs into the track. It doesn't seem to be there on the .oma file when I play it in SS. Is this a known problem? I have read mention of 'clicks' relating to himdrenderer, but am not sure if this is the same kind of thing.

That 'click' is the one-sample error I mentioned in my post.

[dex Otaku]

I was just wondering if there was another way of transferring the audio over and converting into a .wav (the speed doesnt really bother me all that much, but I really dont want to risk losing any of the material I recorded over the weekend) via the USB rather than doing a jack-to-jack via the mic input on the PC & then recording with goldwave (what I was doing with my old MD). As I say speed is not really an issue but file integrity is.

See http://forums.minidisc.org/viewtopic.php?t=5858

Using that method you never have to upload [and risk trashing] your tracks to the computer at all if you prefer not to.

[lyndon1904]

That 'click' is the one-sample error I mentioned in my post.

What is strange (to my limited knowledge) is that it doesn't seem to happen every time: I had it happen twice in a one-minute track and not at all in the twenty-minute test I did last night. However, the short track was of me playing the piano and the long one was of nothing in particular - just background sounds around the house. This may be a stupid question, but does the intensity/density of the recorded sound make any difference to the incidence of a 'one-sample error'?

[petegab]

Super nice for me !

Sound quality good. No click, no repetition in "Allegri Misere" 20 minutes track.

Only pb : the converter left me a oma.wav.tmp temporary file (200 Mo) that I have to delete myself on the drive.

[dex Otaku]

What is strange (to my limited knowledge) is that it doesn't seem to happen every time: I had it happen twice in a one-minute track and not at all in the twenty-minute test I did last night. However, the short track was of me playing the piano and the long one was of nothing in particular - just background sounds around the house. This may be a stupid question, but does the intensity/density of the recorded sound make any difference to the incidence of a 'one-sample error'?

This might seem like an evasive answer, but chances are that a single-sample error will be more noticeable in certain situations [among certain sounds] than others. This is related to how adaptive compression [like atrac] works, incidentally, since it relies on masking.

Another thing is that since the single-sample error is related to the boundary/overlap of the segments used to decode the OMA file, the error would likely be larger if the amplitude at the point of overlap is high.

That's a guess, actually, but going by the files I was having the problem with [and like you, it wasn't with every file] it seems at least partly correct.

Ergo, if the amplitude is really small [quiet recording], chances are the error will be much smaller as well, to the point that you possibly can't hear it.

edit:

Hey marc.. wouldn't it be funny if Sony rang you up and asked to buy your program to use as their wave convertor software?

edit:

Here's why I refer to it as a single-sample error.

Interesting points about this:

* it doesn't do it with every file,

* if it does it with a file, it will do it every time; if it doesn't, it won't every time,

* it doesn't seem to matter whether the source is atrac3+ or PCM, and

* it doesn't always do it in both channels [it seems to like the right channel a lot more, for some reason]

[marcnet]

Yes. Another version. No core changes, i'm afraid, but there is a reason for this release....

There are now 2 executables. One of them (himdrender.exe) is the standard command-line program. The only change to this version is the version number itself

The other executable (himdrenderwin.exe) is the gui version of the same program.

The core is IDENTICAL (the same source file) between the command-line and gui versions.

http://www.marcnetsystem.co.uk/himdrenderer005.zip

As for the click... I still cannot replicate it :/ If someone can experiment more with the overlap & block size settings. This will (hopefully) give me a better idea of what's going on

Alternativly look at the source yourself and figure it out ...

[Rumz]

You know why I'm stoked about this? Digital Linear PCM recordings uploaded and converted to WAV. This will allow me to bypass copy protection on such CDs (usually imports-- I have one in the mail so this will come in handy right in time).

I did get a very small pop or 2 in the track I tested. So are you suggesting playing with the settings (I usted 005 GUI version) to see if I can eliminate the pop from happening?

[Rumz]

Originally I heard the click at about 28 seconds into the song. I re-converted it using a 50 second block size rather than 30 and there was no more click at that part. Whether there were other clicks introduced elsewhere I could not hear in this one samplel track... there were a couple clicks near the end but they were also present in the oma file.

[[StrangeByte]]

You know why I'm stoked about this?  Digital Linear PCM recordings uploaded and converted to WAV.  This will allow me to bypass copy protection on such CDs

:laugh: :laugh: :laugh: :laugh: :laugh: :laugh:

someone really using hi-md for what sony is most afraid of? loooool!

[lyndon1904]

This might seem like an evasive answer

Not at all - it was most informative.

This evening I carried out the following test on a file of 1 minute 17 secs duration - the same 'piano' file on which I'd noticed the clicks. I used the latest version of Himdrenderer with the Windows GUI and played the files back in Windows Media Player.

Block Size=5 - clicks and repeats every 5 seconds

Block Size=10 - clicks and repeats every 10 seconds

Block Size=20 - clicks only (i.e. no repeats) at 38 and 58 seconds

Block Size=30 - clicks only at 28 and 58 seconds

Block Size=40 - clicks only at 38 seconds

Block Size=45 - NO CLICKS OR REPEATS ANYWHERE

Block Size=48 - click at 47 seconds

Block Size=49 - click at 48 seconds

Block Size=50 - click at 49 seconds

Block Size=45 (second test) NO CLICKS OR REPEATS ANYWHERE

Haven't yet tried 45-second block size on a longer file.

[lyndon1904]

it occurs to me that the 'click' I (and others) have reported corresponds exactly to the kind of sound experienced in Cool Edit if you perform an edit, such as a cut or a deletion, without first selecting 'zero crossings' and expanding or contracting the selection to be deleted or cut so that the ends are at points (please excuse non-technical language) where both the left and right channel wave patterns together bisect the 'neutral' line in the middle. I don't understand how 'block sizes' work, but I wonder if there's any way to stop the next 'pass' from occuring (or 'block' from starting) until a zero-crossing point is reached. This may, of course, be a complete load of rubbish...

[lyndon1904]

Please excuse the third post in a row from the same person, but I just wanted to add, especially after my constant carping about 'clicks'...

Last Friday I accompanied a former singing pupil in a lunchtime concert (Vaughan Williams 'Songs of Travel' and 'Let us Garlands Bring' by Gerald Finzi). This was the first time I'd used the MZ-NH1 to record a concert. Before I'd heard of Himdrenderer, I uploaded the concert in the normal way, via analogue cable to Cool Edit - and was rather disappointed with the result (still the same rather '2-D' sound, despite having recorded in PCM). I then joined this forum, discovered Himdrenderer and did the whole thing again (via the woeful SonicStage) and used Mark's program to the .oma files to .wavs. There is no comparison between my original effort and the real 'presence' of the unspoilt digital recording (I think there's just one 'click' in the whole hour). If it wasn't for Mark's efforts, I'd feel I'd completely wasted nearly £300 on the MZ-NH1.

[Rumz]

lyndon-- that's helpful. I also was getting a click at 28 seconds when having the block size set to 30...

In your last post you mention the hour-long recording. Did you try that one with a size 45 block?

I guess most people would have shorter tracks, but it might help.

[dex Otaku]

As for the click... I still cannot replicate it :/  If someone can experiment more with the overlap & block size settings. This will (hopefully) give me a better idea of what's going on

As I haven't done any programming in 10 years the source would not help me much.

I have been playing with the settings and have found that:

* various settings [no consistency from file to file] make it possible to introduce the glitch in any file, but not to get rid of it from files that always do it

* as a note: yes, the files are defragged

* It is more likely to happen with PCM sources than atrac3+

* It is also more likely to happen with longer files

[petegab]

Sorry,

I cannot hear the glitch you're speaking of as my record was made an old analogic vinyl of Allegri Misere, full of "Clicks". But I confirme, that, with this 20 mn track, the conversion leaves a 222 temporary file on my hard drive. Any body has the same thing ? Is it possible to correct it or not ?

Have a nice day to all.

[lyndon1904]

I’ve spent a couple of hours experimenting with Himdrenderer in an attempt to eliminate the ‘clicks’ issue. It appears to me that changing the overlap size makes no difference to the clicks: the default setting of 1000 milliseconds seems to be the ideal one in terms of avoiding skips (overlap size too low, and occurred when I set it at only 100 milliseconds) or repeated chords (overlap size too high, and occurred when I set it at 2500 milliseconds).

The only way to reduce the number of clicks is to increase the block size to the maximum, and they then occur once ever 50 seconds.

What follows is pure hypothesis – I can only guess at how Himdrenderer (Hmdr) works. Am I correct in saying that Hmdr takes each block (of whatever the length set) from the .oma file, ‘processes’ it into .wav form, stores it in a temporary file, deals with the next block in the same way until it reaches the end, at which point all the blocks are, effectively, crossfaded together to the extent specified by the amount of overlap set, and the final .wav file is then created?

If so, the problem would be eliminated at a stroke if it were possible to set the block size to the entire length of the track concerned. Is that possible?

The problem appears to me to be that there is a ‘visible seam’ (well, an audible one!) between the blocks, despite the overlap. I can recreate exactly the same ‘click’ in Cool Edit. If I am editing a concert file and want to take out the audience coughs and splutters between pieces, I select the part of the file concerned, go to Edit, Zero Crossings and ‘Adjust Selection Outwards’. The selection expands to the next point on either side of it where the waveform on both channels crosses the centre ‘zero’ line. The result is an absolutely seamless edit. If I don’t do this and simply select and delete the coughs, I get a click exactly the same as the one I’m getting with Hmdr at the point at which the edit was made.

Is there any possibility (please bear in mind that I have no idea at all of the programming complexity this may or may not involve) that Hmdr could be altered to begin each new block at the point closest to the block size selected where there is a zero crossing?

[lyndon1904]

lyndon-- that's helpful.  I also was getting a click at 28 seconds when having the block size set to 30...  

In your last post you mention the hour-long recording.  Did you try that one with a size 45 block?

I had actually split the hour-long recording into tracks on the recorder before uploading it - each track was about 5 minutes long. I think that there are, actually, a few clicks here and there, but since the concert was in a large church with the mic placed about 50 feet from the singer, the average recording level is quite low, although very clear: as previously mentioned by someone else (dex Otaku?) they tend to be less obvious when the amplitude is lower.

[marcnet]

1) Direct show (the thing I use to do the .oma to temporarry .wav) has the annoying limit of 1 minute (or less) .wav files. Even if you try to render a 2 minute track in one go you will only get the first minute of audio in .wav form. I got around this limit by processing blocks of x seconds of audio instead of just processing the whole track in one go. Thats kinda one of the main reasons for my program to exist.....

2a) If you simply render x seconds to a .wav file, stop and resume rendering another x seconds you will get missing bits of audio in the .wav.

This is bad.

2b) If you render x seconds to a .wav file, stop, seek back y seconds (the overlap time) and resume you will get repeated bits of audio in the .wav. This is not so bad. You can then erase repeated blocks of audio by looking at cetain parts of the .wav (every x seconds - the blocksize ) and finding equal (or near enough) values furthur on. Once the end of the repeat is found ... look for the beginning... Then cut (or rewrite the .wav and miss the repeated bits out)

My program simply does 2b. No crossfading. Simple :smile:

3) It might be difficult to cut "where the start & end cross zero" because sample data (a) and its repeat ( may do this....

a) 20 -75 -195 -100 30 -50 50 100 75  ... 

b) 22 -73 -194 -102 30 -51 49 101 76  ... 

[/code]

Its the way the .oma decompression works....  the PCM values at the end of one block will not be exact to the repeated values at the beginning of the next block. 

And, as by my example, you can see that there is no zero byte to do the cutting at. 

Ive got 2 things to try:
1) Smooth the transition from the start and end of the cut by either repeating or averaging values.

2) I could be talking bollocks and I could try cutting at equal or simular values instead of doing a dumb "repeat is here... so cut"

[dex Otaku]

What if:

xxxxxxx0xxxxxxxx0xxxxxx0xxxx0xxxxxxxx0xxxxxx0xxxxxx0

was the end of the last segment [where '0' is a zero crossing]

and

xx0xxxx0xxxxxxxx0xxxxxx0xxxxxx0|xxxxx0xxxx0xxxxx0

[where | is the end-of-overlap point] was the beginning of the next

If you know exactly where the end of the last seg is, which you really should, or you could perhaps mark by a code of sorts that is unlikely to occur in actual audio..

then, using the last short chunk of the prev seg's end [up to the signature] start looking at the beginning on the next segment for the same pattern of 0 crossings...

then cut the next seg at its first zero crossing,

0xxxx0xxxxxxxx0xxxxxx0xxxxxx0|xxxxx0xxxx0xxxxx0

and cut back to the beginning of where the patterns overlap and [overwriting the signature as well] remove the space in between

xxxxxxx0xxxxxxxx0xxxxxx|0xxxx0xxxxxxxx0xxxxxx0xxxxxx0xxxxx0xxxx0xxxxx0

[where | is the beginning of the word-accurate overlap point]

Looking for 0 crossings seems like a simple pattern search routine, definitely simpler than trying to match entire segments of audio [since there's a lot less to search for, and the search is for something altogether distinct].

Maybe I'm talking out my arse here, but it just seems like a simpler way to do things, and would not require crossfading [which precludes any possibility of bit accuracy] and should also prevent glitches since only zero crossings would be used for searching and concatenating.

Note that you'd only have to do the match with one channel. If the accuracy of the concatenation is actually down to a single word, both should be aligned correctly.

Shoot it down if you will.

[marcnet]

I dont have direct access to sonys wav writer fitler - so I cant pump in a known sequence of zeros, or whatever, in the .oma -> temp .wav stage.

And the accuracy of DirectShow is not perfect (its quite rubbish when dealing with compressed file)... so I dont exactly know where the repeated audio is - hence I have to search.

[poorlyconditioned]

Dear MarC,

You asked about writing a pattern zeros. What I you used an audio program (like Audacity) to make "silence", then use an optical cable to write

this to minidisc, where you record in PCM. Would this work?

Another interesting one would be to generate sinusoids or something

so we could see (gradually) increasing or decreasing waveforms.

If you want such a file, please let me know. I can generate one.

Richard

[marcnet]

It's ok ... I can do that myself.... but its a good idea... ill hook up Buzz (http://www.buzzmachines.com/ - which I use for music making) to make simple sine wave / saw wave/ etc sounds to my HI-MD via my sound cards optical out.

I can also have Buzz render the same sounds direct to a .wav so I have something to compare with.

[dex Otaku]

My point was that you can analyse the decoded chunks in order to match them. I.E. decode 45 seconds, pad the end with a signature so you know where the end is, mark the zero crossings in the last few seconds, decode 45 more seconds, mark the zero crossings in the first 3 seconds, match the pattern there with what's in the end of the last chunk, erase the bit in between, move to the end of that segment and start over.. nothing to do with the wave writer, nothing to do with directshow, all just done in the temp file with the wav data already decoded.

[classicalnut]

How easy is this program to use for those of us who have very little working knowledge of computers. Most of what I do is just click this and then click that. My method of record from MD has been from line out to line in and then recording to Sound Forge. Is your method as easy as this? Hopefully it is or maybe you can post everything needed to know in qreat detail. Thanks for taking the time to read my post. Classicalnut

[marcnet]

Version 006....

* Fixes a small GUI bug where multiple instances of save window / open window could be displayed

* Added smoother option to smooth between block borders. This *should* remove clicks .... at least it did on my test file . Use -m to turn off in command-line version.

http://www.marcnetsystem.co.uk/himdrenderer006.zip

[MZ-NH1]

marcnet, HiMDRenderer WIN, is exactly the same as the HIMDRENDER, but in FULL GUI format, because in the DOS format, if you click on the icon, it auto loads with the 'select input' window. What I am asking is, both these programs do exactly the same right... just the WIN version, is complete GUI and the other is more DOS then GUI... That right? :S

Hope that all makes sense.

[marcnet]

That is correct. The Win version uses exactly the same code as the DOS version. The only difference is that the Win version has a clickable interface and the DOS version displays an "Open File" dialog

Both versions even use the same code to display the "Open file" dialog (the Win version does it when "Browse" is pressed)

Hope that clears things up :smile:

[lyndon1904]

'Smooth over block borders' has definitely improved the click problem, although it hasn't eradicated it altogether. I did some more 'road testing' this morning. The best result I obtained (on a file 20m 26s long) was with a block size of 45 seconds. There were clicks at 13.11 (none at all before that, curiously) 13.55, 15.23, 16.51, 17.35, 18.19 and 19.47.

A block size of 30 seconds gave the following result - some clicks have been eradicated, as can be seen:

0.28, 1.55, 2.25, 2.53, 3.22, 4.20, 5.18, 6.16, 6.45, 7.43, 9.10, 10.37, 12.04, 12.33, 13.31, 15.27, 16.54, 17.23, 17.52, 18.21, 18.50, 19.19 and 20.17.

A block size of 50 seconds resulted in a click approximately every 49 seconds on a three-minute section of the same file when I played it back.

I also tested another file (5m 3s) using a block size of 30 secs. In this case, clicks were heard as follows:

1.55 (click-free until then), 2.25, 3.22 and 4.49 - but again, not on every block as was the case yesterday.

I also tried a block size of 40 seconds (on the same file) but I only listened to enough to confirm that there were some clicks (at 16.14 and 17.32 on the selection I heard).

So, 45-seconds seems to be the optimum figure at the moment, but I've not tried finer tweaking (e.g. 46 or 44s).

Looks as though Marc's on the right track, though. :smile:

[lyndon1904]

As Himdrenderer is still under development, I've bought Total Recorder to use in the meantime. Now, I realise that sound files are either digital or analogue, but are files recorded with Total Recorder via SonicStage *as* digital - i.e., as 'pure' - as files rendered using Himdrenderer? MarC's program creates (I assume) an exact copy of the original .oma file in .wav format. Does TR do this, or is the file subjected to further processing (thus becoming a little further removed from the original sound)? I have listened to both types (Himdrenderer and TR) of recordings on my hi-fi and must say I can't really tell the difference., but I'd still like to know.

(In TR I have recording source set to 'software', and recording level set to '100% of original', so the sound card is bypassed TR's own driver is used.)

[Latexxx]

It is likely that Total Recorder preserves the original waveform better than himdrenderer because himdrenderer adds pops and clicks etc. which aren't in the original source.

[dex Otaku]

TR records the stream exactly as it comes from whatever you're playing from. This means that if you use the volume control, EQ, effects of any kind, compression, stereo enhancement etc. from your player, TR will record with all that DSP. If you play with no DSP on, volume all the way up, etc. then you should get a bit-for bit copy.

Himdrender is not currently capable of bit-for-bit copies because of the overlap glitch.

Ergo, TR is more accurate at this time.

[teelar]

To check out how reliable the Total Recorder method works, I did the following test:

- transferred a CD track to the HiMD recorder, using SonicStage 2.1 in PCM mode

- uploaded the track from the recorder back to a WAV file, using the Total Recorder method

- ripped the CD track directly to a WAV file, this time without using SonicStage

A comparision of the 2 resulting WAV files using 'windiff' showed, that they were absolutely identical. The only difference were some additional zero samples at the beginning of the file ripped directly from the CD drive, which had to be cropped off manually before executing 'windiff'.

This is a clear indication, that the TR method DOES deliver a bit-for-bit copy, at least for tracks previously downloaded via USB. I assume this applies as well for recordings made from the mic port of the Hi-MD device, although this cannot be checked for obvious reasons.

Since one should principally never remove write protection from a disc containing important mic footage, the TR method is probably still the best and safest approach, even if it’s only realtime.

[marcnet]

I appologise for the stupid size of this thread. But anyway it seems like only yesterday I released 0.06... and here we go again.

Version 0.07:

http://www.marcnetsystem.co.uk/himdrenderer007.zip

* GUI version has a batch mode. It might be a bit flakey, so be nice to it. Simply press the "Batch mode" button to access it. It's also why the GUI options has that "all the way over there" feel.

* Re-wrote the smoother code in 0.06. It was horrible and had a sample corruption bug in there. Come on, it was gone midnight when I wrote it

* There are 2 new options. The smoother analysis window (-a in command-line version) is used to calculate what is a spike and what isnt and the smoother averaging window (-g in command-line version) is a smaller window where the actual smoothing is done.

* Added seperate enable/disable option for smoother. The -m option has been modified to simply to this for the command-line version

By the way... Is it just me or has my program gained a more "Its good, but we dont need it" feel..... Just seems things have gone quiet on the himdrenderer reaction front....

[Christopher]

I don't feel that way; I think it should be announced on the front page of minidisc.org and then you'll realize how much of a star you are.

[dex Otaku]

By the way... Is it just me or has my program gained a more "Its good, but we dont need it" feel..... Just seems things have gone quiet on the himdrenderer reaction front....

I don't think so, at least. I'm still testing every revision you post, I'm just not using it for making my actual "to edit with" files yet because of 1) the glitch problem, and 2) SS's bugs.

You're doing excellent work and I believe that everyone here really appreciate the effort you're putting into it.

Now if we could only get Sony to fix SS so the trashing of tracks never occurs, all would be well and your prog would become -the- standard tool for conversions.

[mmilovan]

Now if we could only get Sony to fix SS so the trashing of tracks never occurs, all would be well and your prog would become -the- standard tool for conversions.

Yes, it could be the similar way EAC became standard tool for audio extraction.

[zinereem]

I definitely prefer your program, Marc. Realtime encoding is a pain. And we definitely need it. So keep it up.

[marcnet]

okay. Version 0.07 is back up. Thanks for your positive comments too... they keep me going :smile:

[jadeclaw]

By the way... Is it just me or has my program gained a more "Its good, but we dont need it" feel..... Just seems things have gone quiet on the himdrenderer reaction front....

The traffic went down in general, I see that at the number of threads marked with new posts.

Other forums I frequent show this too, the traffic went down there as well, so I say, don't worry.

When all the clicks and kinks are ironed out, the number of downloads will surge considerably...

[lyndon1904]

okay. Version 0.07 is back up. Thanks for your positive comments too... they keep me going :smile:

I hope you don't regard mine, with all my carping about clicks etc., as negative - I greatly admire all you have done so far, and most particularly the speed with which you post new versions. Version 0.06 was a huge improvement on 0.05, and so tantalisingly nearly there. I'm just about to download version 0.07. Keep up the good work. :smile: