Elektor copybit stripper

[NGY]

I must confess I need someone to explain this subframe / status block structure. I cannot visualize that "at Bit 30" there is another block (of another 32 bits), of which "Bit 2 is the copy bit". Apologies for the dumb question, but how can one "split a bit" into another 32 bits? I am missing here something bit big time (no pun was intended :-) ).

(For reading the .bin file I am using this free hex editor, it's a neat little program that helped a lot in visualizing the general structure of the ROM FSM, as well as finding the "interesting"' parts.)

[kgallen]

It's not a dumb question at all, I need to get my head around that too, it's complex.

Separately: Teasing apart more info from the circuit description: On the "block" of memory, our FSM "state" is the top address bits.  The SR (SPDIF data) is the bottom 8 bits, so SPDIF data values move us around the least-significant locations within a 256-byte block. Our state machine states are the upper 7 bits of address, so a movement in the state machine hops us around between 256-byte blocks. So the "detail" within a 256-byte block will be a "response" to a moving low-level SPDIF bit pattern, and the FSM will "jump" us to another state in the machine by moving us to another 256-bytes block. This probably makes little sense, but as we start to get the hang of how this machine "interprets" the SPDIF stream, it might help! (Maybe...!)

[zedstarr]

As I read the file in Linux (on the Raspberry Pi) there's hexdump and xxd - tools that can format & decode the binary into ASCII hex. (both seem to be present on MacOS too tho you'll need to open the terminal to use these command-line utilities.)

[kgallen]

19 hours ago, NGY said:

I must confess I need someone to explain this subframe / status block structure. I cannot visualize that "at Bit 30" there is another block (of another 32 bits), of which "Bit 2 is the copy bit". Apologies for teh dumb question, but how can one "split a bit" into another 32 bits? I am missing here something bit time.

(For reading the .bin file I am using this free hex editor, it's a neat little program that helped a lot in visualizing the general structure of the ROM FSM, as well as finding the "interesting"' parts.)

OK, one needs to read the article desciption a few times looking at the frame diagram.

This structure is carrying more data that just the audio data. There are 192 "frames" and each frame has two "subframes" which will be the left and right audio channels. A subframe is 32 bits: 4 bits of preamble, 4 bits of "aux data" (supports 24-bit audio samples), 20 bit audio sample, the 4 other bits, 2 of which are of interest - bit 30 which is one bit of a multi-byte "Consumer Channel Status" and a Parity bit (even or odd?).

Before each audio data sample is this preamble burst, which serves several purposes like telling us if we're at the start of the 192-frame sequence - we look for preamble "Z" (4 bits, but biphase code=11101000), then we know we are starting Frame 0. Frame 2 is special for us here, because the non-audio information for that frame contains byte 0/bit 2 of the "Consumer Channel Status" which is the COPY bit we are looking for. It seems this same data is replicated in both subframes.

Re the "bit 2"/"bit 30" thing: every subframe has 32 bits. Bit 30 is the Consumer Channel Status. But the CCS is multiple bytes long, so it's conveyed over the frames, 1 bit at a time - 1 bit per frame (repeated twice - in each subframe). As there are 192 frames, this would mean the CCS could be 192/8=24 bytes long. COPY is CCS byte 0/bit 2, so we need to wait to frame *2* to get byte 0/bit 2 of the CCS, since the CCS is conveyed 1 bit per frame in bit 30.

Make sense?

CCS is multiple bytes long! The second byte for example has that Category Code field. We're lucky our COPY bit is in the first byte otherwise we'd need to look further into the frame counts!

So loosely, the process for the machine will be:

- Find preamble Z - SR will hold [D7]11101000[D0]. This means we're at frame 0.

- We then have to count through a load of bits (actually 2x due to biphase) to skip past F0/SF0,F0/SF1,F1/SF0,F1/SF1

- On arrival at what we think is Frame 2, we should see preamble X, so the SR should hold [D7]11100010[D0]. Then we have to skip along to bit 30 (biphase 60/61) to find our COPY bit.

- Then we have to check the COPY bit. If it's already a 1 (=COPY allowed) we have nothing to change, just keep on passing the SPDIF through unchanged - COPY and PARITY bits. If the COPY bit is 0, then we need to flippity-doo-dah the COPY bit and flippity the PARITY bit too (if the COPY bit was already "correct" (i.e. a 1) then we don't change the PARITY bit otherwise we'd make it wrong). So actually we don't care what whether the parity is even or odd, we just flip it if we flip the COPY bit. If the parity was errored due to a transmission issue, then it's still errored!

Lost the will to live yet? We're learning together (=me waffling on at random!).

[kgallen]

Looking back at some of the references I posted before, the SPDIF format is:

- biphase mark for all bits except the preamble (sync) bits.

The Preamble biphase bit sequences (4 bits, 8 bits of biphase) are shown in the diagram excerpts above from @zedstarr

So for the non-sync bits: binary 1 = transition halfway through bit, so Shift Register will have 01 or 10 in even/odd bit-pairs; binary 0 = no transition, so Shift Register will have 00 or 11 in even/odd bit-pairs.

[NGY]

2 hours ago, kgallen said:

the CCS is multiple bytes long, so it's conveyed over the frames, 1 bit at a time - 1 bit per frame

Thanks Kevin! This was the key sentence for me. And yes, everything you wrote does make sense.

Now I can go back and break my head over the mechanism of the FSM at those "interesting" addresses/blocks. (Those minor, singular oddities might still be just pure errors of an aged silicon, although ~20 years time is not that much for an EPROM)

[kgallen]

One other horror I’ve just thought of, is the output stream is also biphase. This means that for every input bit pair we need to output a biphase bit pair. This is fair enough, once we see preamble Z we just use the ROM contents to replicate every bit on the output. However there is a complication if we flip a bit: converting a COPY bit 0 to a 1 requires us to output 10 or 01 in biphase. The style will depend on the polarity that has come before. Likewise if we invert the PARITY bit we might need to output a binary 0 or a 1 which in biphase could be any of the patterns 10, 01 (for a 1) or 00 or 11 (for the 0). However flipping the COPY bit will flip the biphase polarity and the PARITY bit by chance is the very next bit in the subframe, which will flip the polarity back, so we don’t get into the situation of needing to biphase invert the rest of the data stream. 

[kgallen]

From the beginning the SR is reset (0x00).  Then imagine biphase bits coming in. We need to replicate SR bit 0 on the output whilst holding our state machine in state 0 (so A[14:8]=0). This means we’ll rattle around in EPROM addresses 0x0000-0x00FF with EPROM D0 generating the output where ROM contents D0=A0 and D[7:1]=0x00 (although looking at the bin, bit 1 is a 1 as we see 0x02 and 0x03 in there. Hmmm... so I think this means we’ll end up rattling around ROM block 1 at 0x0100-0x01FF)

However when we see the preamble Z pattern on ROM A[7:0] then our FSM will spring into action and we should jump to another 256-byte block (==state in this context) to start hunting for the COPY and PARITY bits...

Luckly preamble Z has one fixed pattern as it’s not in true biphase (it seems?).

 

ETA: it’s all coming together now. I wondered why the User Bit is included in the table from @zedstarr above. It’s the bit before the COPY bit. Thus it gives us the biphase polarity information we need to determine how we form our replacement COPY bit in biphase. Clever...

[kgallen]

8 hours ago, NGY said:

Thanks Kevin! This was the key sentence for me. And yes, everything you wrote does make sense.

Now I can go back and break my head over the mechanism of the FSM at those "interesting" addresses/blocks. (Those minor, singular oddities might still be just pure errors of an aged silicon, although ~20 years time is not that much for an EPROM)

Those minor singular oddities are the state changes (or output polarity flips if D0). Key to the operation of this FSM :-)

[NGY]

1 hour ago, kgallen said:

Those minor singular oddities are the state changes (or output polarity flips if D0). Key to the operation of this FSM :-)

Now you need to explain it to me :-) . In theory, I can imagine, that this is key.

Seeing the pattern and compare it to the periodicity of the SPDIF serial data stream (with your kind explanations above) though I still cannot see, how.

So here are those exceptions are at 0x0717, 0x07e8, 0x081d, 0x08e2, 0x091d, 0x09e2, 0x7f17 and 0x7fe8 - that is eight in total, that goes well with the "bit management" description part of the original article.

More precisely these addresses with the respective values there:

0x0717 - 0x11
0x07e8 - 0x10
0x081d - 0x13
0x08e2 - 0x12
0x091d - 0x21
0x09e2 - 0x20
0x7f17 - 0x11
0x7fe8 - 0x10

OK, I see something in the address, these are in pairs (sort of), like ...17, ...e8, ...1d, ...e2 - this is cool (even if I don't yet understand the other part of the addresses, i.e., 0x07... to 0x09... but then jumps to 0x7f...). As for the values, I could explain (I think) most, but these ones: 0x13, 0x12, 0x21.

[kgallen]

On 2/20/2021 at 9:49 AM, NGY said:

Now you need to explain it to me :-) . In theory, I can imagine, that this is key.

I'm still working on this too, but this is my hypothesis.

The background fill in the ROM (i.e. all locations before we do any "magic") should be 0x00 in even addresses and 0x01 in odd addresses (at least the bottom 256 anyway). If you programmed a ROM like this and plugged it into the circuit, then it would just replicate the SPDIF input onto the output. This is because ROM A[7:0] would just "point" at one of the bottom 256 locations. An value in the SR of %???????0 (where ?=don't care) would output 0 on D0 (the SPDIF output) and %???????1 would output a 1. Since in the ROM, D[7:1]=0x00, our "state" stays at "state 0" - so ROM A[14:8]=0x00.

So like this we are "stuck" in one 256-byte block of the ROM. When we want to do something special, we need to "jump out" of this boring 256-byte table into another 256-byte table that has more interesting numbers in it (well it will still be mostly "background" as usually we still want to replicate SPDIF input to output). So to do this "jump" we have a more "interesting" value in one or more entries of our 256-byte "block". This will move our "state" (A[14:8]=previous clock cycle ROM Q[7:1]) to somewhere else where more interesting things can happen.

For example, an interesting number in the SR would be the Z preamble. This is binary pattern %11101000. In hex this is E8. So at some "interesting" addresses 0x??E8 (??=don't know yet!) we will find a more interesting number that takes us somewhere else.

We might also be interested in the X preamble which is 0xE2.

With that little teaser I'll let you continue... :-) (compare to the address list in your previous post).

[NGY]

5 minutes ago, kgallen said:

With that little teaser on E8 I'll let you continue... :-)

Haha, you expect too much from me :-) - but yes, I will spin on this.

Sadly my brain has trouble with thinking and "seeing" in hex (I had a mate at the university, who actually did assembly level programming in hex, all in his head - I mean he did not take any notes, only jotted down the final code when he was ready ... - I am quite the opposite).

[kgallen]

Regarding the Z preamble, I think I got slightly mislead on the polarity. It seems that whilst these preambles are not "true biphase" they still adhere to the ongoing polarity of the biphase data. So 0xE8 where the bits are flipped is 0x17. So it seems our machine needs to "look" for both SR=0xE8 and SR=0x17 for Z preamble.

Similarly X preamble is 0xE2 or 0x1D.

Your address numbers are starting to take some shape, huh?! :-)

You ok with hex to/from binary? Convert one hex digit to 4 binary digits. (Why we use hex!)

ps, hope I'm not coming across as "cocky", I don't know the answer either, but with your info it's also helping me piece together this thing. I would not be this far without the input from you both. As I'm typing these replies, I too have some "aha!" moments of further discovery!

[kgallen]

I'm not sure about how to represent this information. For now, I'm using an Excel sheet in a "traditional" FSM format that is a bit like one would use when designing the next state logic for an FSM (ROM or otherwise). I'm not sure if this is helping me yet, but at least I can capture some of the data (there are some bin2hex formula behind some cells). Note the "exception" addresses quoted by @NGY yield a flip in the SPDIF output polarity.

 

[kgallen]

Note there are many ‘state chains’ of boring 256-byte block numbers where the ‘state’ part of the number just increments. These will be to ‘step through’ all of the other data in the frame (like the audio data, frame 1 etc). e.g. states 0x10 to 0x43 (0x44 too, but we'll come to that below, because it's "special" whilst not being special...)

[kgallen]

For example, the first 7 address blocks (0x00??-0x06??) just copy SPDIF in to out. This is so we can shift in 7 biphase bits of SPDIF. At address block 7 (0x07??) is where things get interesting, because we can now start looking for the Z preamble as we've shifted in 8 bits. And yay, at 0x0717 and 0x07e8 we have our first "exceptions" which will move us on to track towards the COPY bit. Also note in block 7 that the background fill is the same in block 6 (0e and 0f). This means we loop in state 7 waiting for the Z preamble to appear before we jump out to the next phase.

So our state machine so far is:
                                                                                                                                  v--no match--|
reset -> state0 -> state1 -> state2 -> state3 -> state4 -> state5 -> state6 -> state7 ----------|--match Z preamble--> state8 -> TBD

Matching the Z preamble is the process of "gaining frame alignment" (that I may have mentioned early on in this thread). The trick here is that the preamble sequences are carefully designed so that they can't be "simulated" anywhere else in the data stream - noting that of course the audio data can take any binary value. This will be why they've not used true biphase for the preamble signatures - because the preamble patterns can't be emulated in the rest of the data stream that is true biphase (excepting crosstalk errors of course, we won't go there for this project, but I bet the chips in the MD machines do account for this). If it were, we could gain "false frame alignment" by seeing the preamble pattern in what is actually payload data.

Being from a telecoms background, I bet @zedstarr can relate to this (e.g. ITU-T G.706, 2048kb/s PDH basic frame alignment).

[kgallen]

        v--no match--|                                                        v-no match--|                                                        v-no match--|
    -> state7 ----------|--match Frame 0 Z preamble--> state8 ---------|--match Frame 1 X preamble--> state9 ---------|--match Frame 2 X preamble--> state10 -> TBD 

[kgallen]

I'm a little lost now as I expected in block 0x42 to find some operation related to the COPY bit. 0x42 is state 0x10 plus 50 (dec) biphase clock cycles which would put us pointing at the U bit prior to the COPY bit in Frame 2 Subframe X. Also since this same operation needs to be performed in the "Y" subframe I though the designer might hunt for preamble Y then run this same count-50 loop again. But I don't see, yet, any code that matches preamble Y. Maybe it does a brute-force count over the bits to do this - probably so as there is no mechanism in this FSM style to flag that Y (versus X) has been done and jump back to the preamble Z hunt phase.

However blocks 0x45 and 0x46 look more interesting...

The quest continues...

My story so far:

[kgallen]

(Edited to correct that the COPY bit is handled by states 0x44 and 0x45).

For those playing along at home, this is what I'm currently considering in states 0x44/0x45. In state 0x44, the first biphase bit of COPY is in bit 0 of the SR.

In the incoming frame, COPY could be a binary 0 or a 1. We want it to be a 1 in the output SPDIF. Since the bits are biphase, we can have the following scenarios as we clock in each half of the COPY bit. Remember each biphase bit of COPY needs to go to the output on the next clock cycle - we can't wait to see both halves of the COPY bit to make a decision. Seems it all works nicely for us...

Let's call the the biphase parts of the COPY bit 'a' and 'b'. At state 0x44, we have 'a' in bit 0 of the SR.

We have 4 scenarios for the biphase bits of COPY, but at this point in time we only have 'a' to look at. The polarity of the (incoming) COPY bit 'a' phase will depend on the polarity of the 'b' phase of the previous bit, the U bit, since biphase mark requires the encoding to transition the bit polarity. We thus have the following scenarios:

The cute thing we see from the above diagram, is that if we have just shifted in 'a', we can shift out the same polarity to the output regardless of whether we need to change COPY or not. We can make our invert decision on the 'b' phase of the COPY bit (state 0x45). Convenient!!! Bear in mind that in SR bit 1 we have the previous biphase bit in time - the 'b' bit of the U bit.

Block 0x44 does a plain out=in as we require - so it's "special" in that it's dealing with 'a' of the COPY bit, but not special in that it is just replicating in to out :-)

So considering SR[1]=U 'b' and SR[0]=COPY 'a', we can see that valid combinations for biphase - which requires a transition - on SR[1:0] are 01 and 10. The values SR[1:0]=11 and SR[1:0]=00 would not be valid for the 'b' phase of U and the 'a' phase of COPY to be compliant biphase. So the addresses of interest are the "odd" pairs of 0x4401 and 0x4402 (and the same for the rest of the 256-byte block since we don't know/care what SR[7:2] bits are, but we want the same behaviour whatever). So we get the values of: SR[1:0]=01 -> 0x8b - thus a 1 will be output on ROM Q[0] as required and SR[1:0]=10 -> 0x8a - thus a 0 will be output on ROM Q[0] again as required and these will be the 'a' phase of the COPY bit clocked to the SPDIF output.

That's block 0x44. Phew! On to block 0x45 where we actually determine the binary value of the COPY bit by determining the polarity of the 'b' bit of COPY. Block 0x45 has a different data pattern to what we've seen before: 8d 8d 8c 8c ...

[kgallen]

Changing the COPY to a binary 1: block 0x45 is probably best summarised by sharing the relevant portion of my ever-growing table, with comments:

So that means blocks 0x46 and 0x47 will deal with correcting the PARITY bit if needed (if we changed the COPY bit). What is also "clever" here is we don't need to remember what went out as the COPY bit (we have no way of recording this) - but the information we need is there in the historical biphase information. Remember we have an 8 bit shift register which still has the incoming COPY biphase bits in there...

 

[zedstarr]

3 hours ago, kgallen said:

Being from a telecoms background, I bet @zedstarr can relate to this (e.g. ITU-T G.706, 2048kb/s PDH basic frame alignment).

Aye I was thinking about that last night :-) "Frame alignment words in timeslot 0"... The 2048kb/s interface (well the G.703 & G.704 realisations) are still in use in most (if not all) telecom networks today as synchronisation signals. Even big Cisco etc. switches that support Synchronous Ethernet and PTP (IEEE1588-2008) still have "sync out"/"sync in" interfaces that can handle the old PDH "2 Meg" signals :-) 

[kgallen]

8 minutes ago, zedstarr said:

The 2048kb/s interface (well the G.703 & G.704 realisations) are still in use in most (if not all) telecom networks today as synchronisation signals.

Indeed - only the other week I added new logic to generate a 2048kHz network time reference output from our current chip in development...

[kgallen]

OK, so blocks 0x46 and 0x47 for the PARITY were a bit more hairy. But we're getting the hang of this now, so this is getting easier! Also finally we bump into the Y preamble detection! Remember we have to do the operation of blocks 0x44 thru 0x47 again on Sub-frame Y. And you all thought this stuff was easy! We're only 2/3 of the way there yet!

You get another bit - a big bit! -  of my table as explanation for blocks 0x46 and 0x47 handling PARITY. Note we'll stay in state47 pushing in to out until we detect the Y preamble pattern (which should come along in the next 8 bi-phase bits). Note some of the 0x47 entries I've commented as "illegal" will be used to propagate the incoming bits of the Y preamble through to the output unchanged. After matching the Y preamble we'll exit to state48.

(Edit to correct a couple of the state47 COPY/PARITY action comments).

Hold on, I'm sure there must be something more important I should be doing...!

[zedstarr]

3 hours ago, kgallen said:

For those playing along at home

This is the most fun I've had in a long time :-D

Quote

Hold on, I'm sure there must be something more important I should be doing...!

Isn't that often the way? Out of lockdown boredom I retrieved my MiniDisc kit out of the cellar and look where we are now :-)

[kgallen]

Right I'm being lazy now and jumping to the endgame from what I see in the 0x7? blocks. This is what I "promised" you right? I might finish the entries in my table later just for completeness, but I need food and to remind myself what the rest of the family look like...

Here is my take on the state diagram. Not much of a surprise there, huh! Apologies for the brevity but with the above information I'm sure you can work out what I'm saying...

(Sorry this is so large, using a thumbnail and expansion leaves the text unreadable.)

Table:

elektor_scms_rom_fsm.pdf

Overall I think I'd comment that both the design of the SPDIF frame is smart and the ROM-based design of this "Copy-bit Killer" is also very neat and well implemented. I'll note also from the FSM state diagram above that the "illegal states" (0x0a-0x0f) are also handled. If the machine ended up by error in one of those blocks, if will self-recover back to the initialisation state where it will look for Z preamble again. Good robust design. Actually it's required since IC9, the register chip, doesn't have a reset, so in theory it could come up (from power-on) in any state (any binary value) and thus dump us unceremoniously into any state of the FSM, including the "illegal" ones.

Hex dump of the bin file if anyone else wants a play. Open in a text editor.

elektor_scms_rom.hex

Please post your own homework below :-)

[NGY]

Thanks for your enormous work Kevin! You "lit the light" so to speak - no, actually your analysis above is way more than that.

It will take some time for me to read and digest everything, but way less time if I had to figure out all this on my own (if I ever would have been able to). So thanks again, and please forward our apologies to your family, for stealing you from them (or maybe they were happy that Dad was busy with some crazy stuff and kept quiet the whole day :-P ?)

[kgallen]

3 minutes ago, NGY said:

It will take some time for me to read and digest everything, but way less time if I had to figure out all this on my own (if I ever would have been able to).

You have to "fill in the blanks" for me on the 0x7? stages to make sure I didn't miss anything! :-D

Just look at the technical insight in your other posts - you would have got there.

[zedstarr]

That is truly an epic piece of work Kevin! Many many thanks for the insight  :-)

[kgallen]

Fixed a couple of the Comments on state47 COPY/PARITY and improved a couple of other Comments in the Table pdf above if you've already downloaded it and are scratching your head on some of my documentation errors...

[kgallen]

Here is some C code to generate a hex and bin version of the ROM. I'm no C coder and this C code is awful. Someone, please rewrite!

I compiled this with gcc on Linux. I've no idea how to compile C on Windows, sorry. So use this code as inspiration for your own more worthy efforts!

elektor_scms_rom.c

 

I used tkd iff to check the hex output from this code matches the hex dump I generated from the bin file uploaded above by @zedstarr .

[kgallen]

If you can make use of it, here is the same thing in the scripting language Tcl. It's interpreted rather than compiled like C. 

You can get a Tcl interpretter here, but you have to "register" with ActiveState these days: Tcl Download

The script: it's just a rehash of the C code provided above.

elektor_scms_rom.tcl

 

If you're on Linux you can almost certainly use wish or tclsh.

[kgallen]

@zedstarr - looks like you could knock us up a Python version of same... we'll find a language that is portable enough for everyone to have a play, sooner or later!

Thanks for the 'bump' on the Blog :-D

[zedstarr]

Disclaimer: I'm not a software engineer...

Requires python3 - I'm sure there's a more elegant way that will support both python2 and python3 but it's late and I'm tired. I'll look again tomorrow after some sleep and a chat with an actual software engineer :-D
(also there's a kludgy hack to define a fprintf function in python so I could get around converting the formatting of the ASCII hex output :-) )

Tested OK on Fedora 33, RaspiOS (buster) & MacOS Catalina.

execute as:

python3 elektor_scms_rom.py

Enjoy!

@kgallen I've just spotted an error in a comment in your original code which I've propagated through to the python: your line 72 "0x00 to 0x07" should be "0x00 to 0x06". Obviously doesnt affect the output files at all 

 

elektor_scms_rom.py

[kgallen]

7 hours ago, zedstarr said:

I've just spotted an error in a comment in your original code which I've propagated through to the python: your line 72 "0x00 to 0x07" should be "0x00 to 0x06". Obviously doesnt affect the output files at all.

@zedstarr Thanks for the py, I like it!

There were a couple of comment errors - also "...4-byte fill..." should have been "8-byte fill" (it changed!). Should be fixed in the uploaded C (and Tcl) above. Any more, let me know, I'm a stickler for correctness! :-D

[zedstarr]

@kgallen Seeing as the EEVBlog forum post on this same topic was the first one I came across (and the first one the OP posted to?) do you mind if I share the source code over there? I've zipped the C/Tcl/Python together

[kgallen]

@zedstarr - sure, go for it. No pride in that code!

[zedstarr]

Never one to miss an opportunity to revive an old thread...

 

Another ROM found in the wild and imaged: https://github.com/andydoswell/Elektor-976516-1

 

md5sums the same so yay/meh but at least it's on the internet somewhere else 😉

 

[kgallen]

5 hours ago, zedstarr said:

Another ROM found in the wild and imaged: https://github.com/andydoswell/Elektor-976516-1

I’ll leave you to do the decode as I ‘demo’d’ on the last one!

 

(Think I still owe you some data on the last one? Sorry!)

[kgallen]

Adding a zip file of the data/docs used/generated in this reverse engineering.

 

 

Elektor_Copy_Bit_Stripper.zip