I sat down with a fresh set of eyes and worked through the problem of receiving multiple packets. Part of the issue was that I was starting a new packet before appending the current segment to the prior packet. This ended up pushing the end of the prior packet data into the next packet. Another issue was timing. I was recording the end of a packet as the start time + packet duration. This would make sense… but that’s exactly when the next packet starts. I ended up subtracting a millisecond to the end-time. There were a few other tweaks and “gotchas” that I addressed, but those are the two primary ones that caused the most havoc. In the end, I now have fully decoded messages!

The error percent is the next thing to tackle along with the dimly lit zeros and dots. Due to how the packets are at a fixed size, I’m receiving padded data at the end of the message. This translates as a null character. Since null is not a printable character, I show a dot instead.

My packet needs to send in a value that indicates how long the data is over the packetized transmission. This is separate from a packet size. It’s a transmission size. The question is – what is the maximum length that I want to allow for a transmission? This affects the number of bytes in the first packet, as now the extra space is used for a header. – yep, we are back to headers again, but this time we only need it in the initial packet. Having an 8 bit number gives us a maximum of 256 bytes. That’s not ideal as I am allowing a packet size up to 64kb. I think a data length with a 16 bit number may be good enough. It gets us up to the 64kb of data of the largest packet size. That’s a crazy high number for audio and would take a long time to transmit.

Basically I have two functions to convert between an unsigned 16 bit integer and a binary array.

function bitsToInt(bits, bitLength) {
  // only grab the bits we need
  const bitString = bits.slice(0, bitLength)
    // combine into string
    .join('')
    // Assume missing bits were zeros
    .padEnd(bitLength, '0');
  // parse as int
  return parseInt(bitString, 2);
}
function intToBits(int, bitLength) {
  // max number for bit length
  const max = Math.pow(2, bitLength) - 1;
  // overflow numbers too high
  const value = (int & max);
  return value
    // convert to bit string
    .toString(2)
    // prefix with zeros to fill bit length
    .padStart(bitLength, '0')
    // convert to array
    .split('')
    // convert strings to numbers
    .map(Number);
}

I prefixed the bits to the packet before I determine how large the packet can be.

  bits.unshift(...intToBits(bytes.length, MAXIMUM_PACKETIZATION_SIZE_BITS));
  const bitCount = bits.length;

Now I need to decode the data length that I received, and cut off the extra bits.

function removeDecodedHeadersAndPadding(bits) {
  const sizeBits = MAXIMUM_PACKETIZATION_SIZE_BITS;
  let bitCount = bits.length / 8;
  if(bits.length >= sizeBits) {
    bitCount = bitsToInt(bits.slice(0, sizeBits), sizeBits);
  }
  // remove size header
  bits.splice(0, sizeBits);
  // remove excessive bits
  bits.splice(bitCount * 8);
  return bits;
}

Now that is much better! Both the decoded bits and decoded text are no longer padded. Without all of the extra padding, the error percent is down to zero for the decoded bits. Now I need to address the encoded bits received. This is a bit tricky since the data size is encoded within those blocks to prevent noise interference from corrupting the transmission.

Removing the padding from encoding was much harder. Although I already have a value of “Sent Encoded Bits” in memory, I need to depend only on the data received. I had to check if error correction was on, calculate the blocks needed for the data size, correct those blocks, then pull out only the bits representing the number, and then remove the padding.

Yea, the code is ugly, but it gets it done.

function removeEncodedPadding(bits) {
  const sizeBits = MAXIMUM_PACKETIZATION_SIZE_BITS;
  const dataSize = ERROR_CORRECTION_DATA_SIZE;
  const blockSize = ERROR_CORRECTION_BLOCK_SIZE;
  let bitsNeeded = sizeBits;
  let blocksNeeded = sizeBits;
  // need to calc max bits
  if(HAMMING_ERROR_CORRECTION) {
    blocksNeeded = Math.ceil(sizeBits / dataSize);
    bitsNeeded = blocksNeeded * blockSize;
  }
  if(bits.length < bitsNeeded) {
    // unable to parse size just yet
    return bits;
  }

  // get bits representing the size
  let dataSizeBits = [];
  if(HAMMING_ERROR_CORRECTION) {
    for(i = 0; i < blocksNeeded; i++) {
      const block = bits.slice(i * blockSize, (i + 1) * blockSize);
      dataSizeBits.push(...hammingToNibble(block));
    }
    dataSizeBits.length = sizeBits;
  } else {
    dataSizeBits = bits.slice(0, sizeBits);
  }
  // decode the size
  const dataByteCount = bitsToInt(dataSizeBits, sizeBits);

  // determine how many decoded bits need to be sent (including the size)
  const totalBits = (dataByteCount * 8) + sizeBits;
  let encodingBitCount = totalBits;
  if(HAMMING_ERROR_CORRECTION) {
    const blocks = Math.ceil(encodingBitCount / dataSize);
    encodingBitCount = blocks * blockSize;
  }

  // bits are padded
  if(bits.length > encodingBitCount) {
    // remove padding
    bits = bits.slice();
    bits.length = encodingBitCount;
  }
  return bits;
}

I have half a mind to remove the padded segments in my last packet, or at least make them dim to represent unused segments. Removing the segments would hide any transmission errors. For now, all is well.

We now have another problem. The information about the packet itself is off. It’s easier to see when sending one character.

Although we only sent one byte, the “Data Bytes” is showing that we sent 3. That’s due to the packetization size prefixing the data with two bytes. I think I need to list the data bytes and added header bytes separately.

Our next problem is trust. Can I trust that the size is correct? Error correction only helps if one bit fails out of every seven bits. Having a 16 bit number representing the size means that we need 28 bits to represent that number – 4 blocks, 4 failures at max as long as they are in different blocks. Although I have error correction in place, the receiver is not aware if the packet has an error. I need error detection. This is often done with a checksum, parity bit, or cycling redundancy check value.

Not only do I need to put the type of error detection, but I would like to have it available as soon as possible for the packetization length itself. We can get into error detection on individual packets later. If just one bit is bad in the packet – I still want to be able to get the total length if at all possible. It’s pretty important. Since this is just for a 16 bit number, the detection can be fairly small. A parity bit could help – but their is a trust of bits when error correction fails. I think I need to work in at least 8 bits. If I go up to 16 bits, then it’s a bit pointless to do a check since I could just duplicate the number and write it out twice. My target is 8 bits to verify a 16 bit number was decoded/received properly.

Looking at Geeks for Geeks: Difference between Checksum and CRC, they list a few differences between the two. From my take, the CRC algorithm is capable of detecting more errors as a hash where a checksum algorithm is just a little bit better than a parity check. If a bit is off in both the checksum and original data, it may not be as easily detectable as a cyclic redundancy check. Checksums are used for software validation where CRC are used for transmission integrity.

So… what is a cycling redundancy check, and how to we do it?

• Initialize a check
• Loop through each byte
• Xor the check with the current byte
• shift the bits to the left
• If the first bit was a 1, xor the polynomial
• return the check

In a nutshell, that’s it. Different configurations start off with different initializations where it’s usually 0 or the max value (0xFF for 8 bit, 0xFFFF for 16 bit). In some cases, you’ll need to reverse the bits in your byte before you do an xor with the check (reflect in). In other cases, you may need to reverse the bits of the check itself (reflect out).

It’s not perfect, but here is my implementation.

function calcCrc(
  bytes,
  size,
  polynomial,
  {
    initialization = 0,
    reflectIn = false,
    reflectOut = false,
    xorOut = 0
  } = {}
) {
  if(bytes.length === 0) return 0;
  const validBits = (1 << size) - 1;
  const mostSignificantBit = 1 << size - 1;
  const bitsBeforeLastByte = size - 8;

  // setup our initial value
  let crc = initialization;

  function reverseBits(value, size) {
    let reversed = 0;
    for(let i = 0; i < size; i++) {
      // if bit position is on
      if(value & (1<<i)) {
        // turn on bit in reverse order
        reversed |= 1 << (size - 1 - i);
      }
    }
    return reversed;
  }

  for(let byte of bytes) {
    // reflect incoming bits?
    if(reflectIn){
      byte = reverseBits(byte, 8);
    }
    // xor current byte against first byte of crc
    crc ^= byte << bitsBeforeLastByte;
    // loop through the first 8 bits of the crc
    for(let i = 0; i < 8; i++) {
      // is first bit 1?
      const isFlagged = crc & mostSignificantBit;
      // if flagged, xor the first bit to prevent overflow
      if(isFlagged) crc ^= mostSignificantBit;
      // shift bits left
      crc <<= 1;
      // remove invalid bits
      crc &= validBits;
      // xor the polynomial
      if(isFlagged) crc ^= polynomial;
    }
  }

  // We only want the last [size] bits
  crc &= validBits;

  // reflect final bits?
  if(reflectOut) crc = reverseBits(crc, size);

  // xor the final value going out
  crc ^= xorOut;

  // remove sign
  if(size >= 32 && crc & mostSignificantBit) {
    crc >>>= 0;
  }
  return crc;
}

For the most part, it works when comparing results with an online CRC Calculator. I run into trouble with 32 bit CRC codes. I believe it’s due to the negative sign and that I’m working with 32 bit signed integers in JavaScript.

We have our check! Now we need to apply the 8 bit version to our packetization. I thin I have something, but I don’t believe it can be this easy.

// packetization headers
// data length
const dataLengthBits = numberToBits(bytes.length, MAXIMUM_PACKETIZATION_SIZE_BITS);
bits.unshift(...dataLengthBits);
// crc on data length
const dataLengthCrc = crc8(bitsToBytes(dataLengthBits));
bits.unshift(...numberToBits(dataLengthCrc, 8));

No… it wasn’t that easy. I was popping the CRC in front of the data transmission size. I need the size ASAP to determine how to truncate the data.

// packetization headers
// data length
const dataLengthBits = numberToBits(bytes.length, MAXIMUM_PACKETIZATION_SIZE_BITS);
// crc on data length
const dataLengthCrc = crc8(bitsToBytes(dataLengthBits));
// crc second
bits.unshift(...numberToBits(dataLengthCrc, 8));
// data length first
bits.unshift(...dataLengthBits);

After some head scratching, I finally got the actual number of bytes to display with a matching CRC value. Once I have the length of bytes, I can verify that the byte length is accurate using the CRC check. If the numbers match, I’m good to go!

Even if the first packet is corrupt, I have a higher confidence that the length is correct, which gives me an idea of how many packets I will receive, and how long it will take to transfer all of the packets.

Now that I have the expected data length, I can now create a progress bar!

I don’t initially have enough information to know what the data size is until the first 16 decoded bits are available. At first, I decode what I can, and fill in the rest of the bits with 1’s for the maximum size. As more bits arrive, the size is narrowed down to be more accurate until all 16 bits are available. From that point on, most of the calculations involve functions that I’ve already setup to estimate the number of bits that I as going to send earlier.

const totalBitsTransferring = parseTotalBitsTransferring(allDecodedBits);
const receivedProgess = document.getElementById('received-progress');
let percentReceived = allRawBits.length / totalBitsTransferring;
receivedProgess.style.width = `${Math.floor(Math.min(1, percentReceived) * 100)}%`;
function parseTotalBitsTransferring(bits) {
  const dataByteCount = parseTransmissionByteCount(bits);
  const bitCount = getPacketizationBitCount(dataByteCount * 8);
  const segments = getTotalSegmentCount(bitCount);
  return segments * getChannels().length;
}
function parseTransmissionByteCount(bits) {
  bits = bits.slice(0, MAXIMUM_PACKETIZATION_SIZE_BITS);
  while(bits.length < MAXIMUM_PACKETIZATION_SIZE_BITS) {
    // assume maximum value possible
    // until we have enough bits to find the real size
    bits.push(1);
  }
  return bitsToInt(bits, MAXIMUM_PACKETIZATION_SIZE_BITS);
}

It’s fun to watch the progress bar…

It’s getting late. I need to get some rest before the birds wake up.

The latest state of the application is getting a bit messy.