The Problem With Time Mark & Oma Conversion

[himd_user]

The live recording "faq" says this:

>>>>

2) TIME MARK: To automatically place a (silent, gapless) track mark at fixed intervals from 5 to 60 minutes. Turn to ON, pick your interval. Also useful for interviews. This overrides Sync Recording.

TIP: If you are recording through Line In, and want to defeat Sync Recording, set Time Mark to 60 minutes and you will get one track mark every hour rather than marks at silences.

<<<<

TIME MARK causes the following problem, after uploading tracks, when converting

oma's to any other format

(either from within Sony s/w / sonicstage 2.x/3.0 or using himdrenderer):

Each time there is a track transition, there will be either:

a) silence inserted at the beginning/end of track

missing samples (!!! non-continuous audio!) at the beginning/end of track.

To test this problem, do the following.

Set TIME MARK to 1 minute.

Record using the Mic.

Count off quickly, "1 2 3 4 .." just before and after Time Mark is hit

(better yet is to create a real test scenerio, i.e. tone sequence with 5ms tones).

Stop recording, there will be 2 tracks.

Upload the audio to SonicStage.

Convert the audio tracks to WAV (or mp3 using himdrenderer).

Load the wav's into audio editing s/w.

Concatenate the two files.

At the transition from Track1 to Track2, there will be audio corruption (missing

samples or silent samples).

I have measured this missing audio time to be about 20ms to 100 ms on my PC.

In practice, this means a single phoneme in a recorded interview may be lost.

This 20-100ms time may have to do with PC performance, hard drive performance,

etc, because it may be a file buffering problem in the s/w conversion

process. Or, it could be a block encoding problem, in which case PC performance

does not matter.

There is only one workaround to this problem that I have found.

If anyone has suggestions, please reply.

The only workaround (preserving all-digitial transfer) is to use Sonicstage to play

the oma files in real time.

Then, capture from the sound card via Audio Out Mix while recording in real time

from audio editing s/w.

This will create continuous audio, without any sample loss.

But, obviously, the recording is done in real time, so it takes FOREVER, and

the levels need adjustment prior to record.

This means direct oma -> mp3 conversion is not possible (himdrenderer), this may

mean lots of hard drive space is needed for the intermediate WAV recording.

This workaround also loses the TIME MARKS during the recording so any info

gained from using TIME MARKS is pretty much useless.

This is a seriously annoying problem for me, because it means I can not use

TIME MARK the way I want to: With a short time (like 2 minutes), to delete

intermediate "junk" tracks I am not interested in (example: during breaks in a

lecture or breaks during live recording), in conjunction with saving successive

tracks as continuous audio.

This problem may be a symptom inherent in OMA conversion process. Maybe it exists on all converted audio files (end of every file has last 10-50 ms as silence or

missing data?) and I have not noticed it.

Conclusion:

TIME MARK is not "gapless" on the PC side!!!

In my opinion a Time Mark should never affect the continuity of audio. It does!

[ozpeter]

What format are you using to record in? In my experience, PCM format does not appear to cause any loss or addition of material at track marks. Hi-SP causes (here) duplication of material at track marks. Hi-LP I don't use so can't comment.

Another factor is how the track marks are inserted. Mine mostly arise from using line input and the tracks being inserted at points where the level rises from close to silence. Maybe pressing the button manually, or inserting them automatically on a timed basis, causes different problems.

What surprises me is that these problems are starting to be discussed only now. Maybe this is to do with SS 3.x or a recently manufactured batch of recorders?

Additional thought - you know how it's difficult to edit mpeg video apart from at certain frames? I wonder whether this is related to these atrac encoded files - maybe you simply can't have accurate joins exactly where the track marks are.

Edited May 24, 2005 by ozpeter

[matheis]

over in this thread i talk about something similar. i recorded a concert in pcm with auto track-marking on. after hearing about problems with track-marks and glitches between tracks, i combined the tracks in ss3.0 before editing in an external program. i then saved the file as several tracks and imported them as wav files into ss3.1 (upgraded) and transferred to md as hi-sp files. this introduced glitches between tracks. i also tried this with the good-ol' dino's nero-simpleburner method and found that when i created a cd image of the tracks and loaded that into simpleburner i didn't get any glitches at all. i could see a difference in the two different methods in the album times, as well. even though they were he same album, the ss3.1 transferred album was 2 seconds longer due to repetition of material. so, i'd suggest loading the split tracks into nero and making an image file and then using simpleburner to get them back on md as necessary for glitch-free playback

hope this helps!

nikolaus

What format are you using to record in?  In my experience, PCM format does not appear to cause any loss or addition of material at track marks.  Hi-SP causes (here) duplication of material at track marks.  Hi-LP I don't use so can't comment.

Another factor is how the track marks are inserted.  Mine mostly arise from using line input and the tracks being inserted at points where the level rises from close to silence.  Maybe pressing the button manually, or inserting them automatically on a timed basis, causes different problems.

What surprises me is that these problems are starting to be discussed only now.  Maybe this is to do with SS 3.x or a recently manufactured batch of recorders?

Additional thought - you know how it's difficult to edit mpeg video apart from at certain frames?  I wonder whether this is related to these atrac encoded files - maybe you simply can't have accurate joins exactly where the track marks are.

←

[dex Otaku]

I regularly use time mark to divide large programmes into chunks. The first thing I do after uploading is use SS's combine function to recombine all obviously contiguous tracks. I never get gaps, I never get dropped samples, and if anything catastrophic should happen during an upload the largest piece I will be missing is the length of the time mark setting.

In fact, I have never experienced gaps/lost samples, whether using manual track marks, time mark, or auto trackmarking with the line-in.

The trick to avoiding this is SS's combine feature, IMO.

The likely reason this is happening is that all discrete-transform [fft, mdct, etc.] lossy formats use frames that overlap. This is part of why MP3 is not gapless [aside from its using, by default, a framesize not related to CD-DA's 588 stereo samples/frame, aka 75fps], among other things.

SS is slightly smarter than you think, as is how all of the atrac variants are set up. Combine and divide are there for a reason. Use them, and use them well. Dont', and do so at your peril.

ozpeter: MPEG2 video [and almost all lossily-compressed formats] is/are difficult to edit with because they use keyframes, which happen either at a maximum interval apart [like 300 frames by default for most mpeg4 codec variants] or as automatically detected at scene changes. Audio recording does not work the same way, and the problem being experienced here is completely unrelated.

matheis: as I've pointed out before, the only method that retains true gapless editing/burning/playback is to edit all contiguous tracks as a single file, inserting trackmarks with your CD burning software, not by dividing them before or during editing.

[himd_user]

To followup other asked points:

1. The problem occurs with either SS 2.1 or SS 3.0

2. The problem occurs when converting from OMA -> WAV or OMA -> himdrenderer -> any format. When playing OMA within SS, the audio is gapless. When playing from the deck, audio is gapless. Convert on the PC to any other format, and there are missing/corrupt/blank samples at the end of the track.

3. I was using Hi-LP for audio interviews.

4. The method I used to avoid problem #2 was to re-record in real time into

a single file for editing (ugh!!).

I agree, from #2 above, it seems it may be a frame length/algorithmic problem in

PC s/w.

I did try SS "combine" function without success I think. It would be good to

hear your experience here. It seems you are suggesting a method like this:

a) Use Time Mark while recording as desired.

Upload tracks to SS

c) Combine all tracks in SS into one track

d) Convert single track to WAV/MP3

e) Edit WAV/MP3 as single file (including re-splitting tracks)

f) Save edited file in desired format

Unfortunately step © removes the time marks (obviously) so notes taken regarding

different tracks is lost, while this is not desirable, it is still better than not using time

marks at all!

Also unfortunately, step © means there is one huge file to edit. Kind of a strain on the PC sometimes, especially for WAV's. (example: 2 hour voice interview)

So while slightly more time consuming, I prefer to edit using smaller audio files

(individual tracks) sometimes, or concatenate in some other cases.

I regularly use time mark to divide large programmes into chunks.  The first thing I do after uploading is use SS's combine function to recombine all obviously contiguous tracks.  I never get gaps, I never get dropped samples, and if anything catastrophic should happen during an upload the largest piece I will be missing is the length of the time mark setting.

In fact, I have never experienced gaps/lost samples, whether using manual track marks, time mark, or auto trackmarking with the line-in. 

The trick to avoiding this is SS's combine feature, IMO. 

The likely reason this is happening is that all discrete-transform [fft, mdct, etc.] lossy formats use frames that overlap.  This is part of why MP3 is not gapless [aside from its using, by default, a framesize not related to CD-DA's 588 stereo samples/frame, aka 75fps], among other things. 

SS is slightly smarter than you think, as is how all of the atrac variants are set up.  Combine and divide are there for a reason.  Use them, and use them well.  Dont', and do so at your peril. 

ozpeter: MPEG2 video [and almost all lossily-compressed formats] is/are difficult to edit with because they use keyframes, which happen either at a maximum interval apart [like 300 frames by default for most mpeg4 codec variants] or as automatically detected at scene changes.  Audio recording does not work the same way, and the problem being experienced here is completely unrelated.

matheis: as I've pointed out before, the only method that retains true gapless editing/burning/playback is to edit all contiguous tracks as a single file, inserting trackmarks with your CD burning software, not by dividing them before or during editing.

←

[ozpeter]

The likely reason this is happening is that all discrete-transform [fft, mdct, etc.] lossy formats use frames that overlap.

Well, that's handy to know - in essence, that's what I'm hearing, overlaps at track marks. If that's inherent in the format, then it's simply something one has to live with, or deal with by the various possible and somewhat tedious methods in instances where it's clearly audible. In my experience (with Hi-SP) so far, it's not something that jumps out at you very often.

[dex Otaku]

Well, that's handy to know - in essence, that's what I'm hearing, overlaps at track marks.  If that's inherent in the format, then it's simply something one has to live with, or deal with by the various possible and somewhat tedious methods in instances where it's clearly audible.  In my experience (with Hi-SP) so far, it's not something that jumps out at you very often.

←

I haven't had any problems with HiSP, however - people have reported bugs in how HiLP splits and/or combines on 1st-gen units themselves, so perhaps this is a problem inherent to HiLP.

[ozpeter]

Sorry to return to this, but curiosity is my middle name...

Never being one to accept what I'm told without verifying it myself, I did a test line-in recording using Hi-SP of a CD track, and while it was recording for about 90 secs, I put in about ten track marks manually. I then loaded it onto the PC with SS 3.0 and converted to a set of wave files. I combined the waves into a single file using two alternative methods, and checked that they were identical (to ensure my method of combining to one file isn't faulty). On replay, I could hear tiny bits of audio were missing at most of the joins.

Following the advice given by dex, I then combined the files within SS, converted the SS-combined files to wave, and compared that to the version combined outside SS. At each track mark the SS combined version played a little further behind the version combined outside SS, and the total difference in file length was around half a second in the 90 second recording.

So, at least here, it's clear that the problem lies in the conversion to wave of uncombined .oma files. If you do as advised by dex Otaku, namely let SS combine the .oma files into one, then convert that to wave, you'll have a transfer without overlaps or missing material at track marks (but see next section below ==== lines).

Now, this of course means that (if you wanted them at all) your track marks have gone from the SS-combined version. Here at least I've got a reasonable workaround, involving a freeware tool called CueListTool. This has the ability to copy cues in a wave file from one file to another. So, assuming you are working with Hi-SP format (haven't tested Hi-LP, and PCM seems not to have the problem) in combination with Cool Edit or Audition (or possibly other programs), what you can do is....

1. Transfer from MD to PC and to wave without combining, thus creating a set of wave files, one file per marker.

2. If need be use the method I've described elsewhere to automatically rename the wave files to ensure correct chronological order.

3. Use the "Open Append" command in Cool Edit / Audition to create a single large wave file, which Cool Edit / Audition will automatically mark with wave file cue marks where the original files begin and end.

4. Save that big file and delete the original set of wave files (if you like).

5. The file that you have just created will have bits missed or duplicated at the cue marks.

6. Now in SS, combine the .oma files into one.

7. Convert that large .oma to wave

8. In CueListTool, open the Cool Edit wave file, and read in its cues.

9. In CueListTool, open the SS-combined wave file, and write the cues you just read from the other file.

10. Eureka. You have a file with no glitches, but you have cues, and you didn't have to do anything manually.

But - of course due to the fact that the cues come from a file with bits missing, then they will be in slightly the wrong place in the file you've written them to - the further you go into the file, the more inaccurate they will be. But the difference will not be that big, and to correct that (by dragging them in Cool Edit, or even using CueListTool ) is easier that trying to work out where they would have been from scratch.

As with any typed out procedure, it seems long winded, but in fact it's no big deal in practice, particularly if you are reasonably familiar with the programs concerned, and each process is pretty automatic and brain-out-of-gear.

And what's the alternative?

=================================

Now for part two. Yup, there's more.

Using the above, I think I have proved that converting to wave from a SS combined set of Hi-SP files leads to a wave file with no problems where the orignal track marks were, ok?

Well, that's only true if you put the track marks in manually.

Days back I commented on how I was getting duplicated material at track marks when inserted automatically during a line-in recording at points where the volume comes back up past the -40dB threshold (or whatever the figure is). Doing the above test tonight, I wasn't getting that problem. So I repeated the whole procedure above but this time, by playing the CD track at low level and recording it at low level, but bring the level up and down a bit, I caused automatic track marking to be triggered. This time, whether combining the tracks in SS or externally, little bits of repeated audio could be clearly heard where the original track marks were. You can in fact hear these repetitions when replaying the SS-combined .oma file in SS - it's not caused by the conversion to wave.

So in this scenario - Hi-SP with auto track marks - there's no way of getting a transfer without repeated bits (overlaps) at the track marks, unless you use an analogue realtime transfer, because for reasons I can't begin to understand, playback on the NH900 itself is free of these overlaps. For some unfathomable reason, SS3.0 can't accurately determine where the track marks are if the level is low at the point they occur (which is where they are when placed automatically - they seem to be inserted just before the increase in level rather than right on it).

===========================

For many users, these problems won't matter a jot. But if you are recording material with a wide dynamic range using hi-SP via line in, (eg classical music as I do), then you should be aware of the issues at least. Whether these issues affect other versions of SS or whether they affect other models I don't know of course. It would be interesting if someone could test a 2nd gen model in due course.

[Edit:- just for completeness, a test of PCM shows none of these problems occur and there should be no need to use SS to combine PCM tracks to obtain a glitchless file.]

Edited May 25, 2005 by ozpeter