Tokenisation is very important. When working with a stream of data, as provided by TCP and UDP, you most certainly want to
Wait for a complete response before processing
Want to process one response at a time
Whilst this might seem to occur naturally most of the time, network contention, network errors and high data rates will trip you up. CBus is one system where I’ve often see back to back messages returned in a single IO read.
ACA Engine ships with two tokenisers to help you break up the incoming data. The default buffered tokeniser and the more advanced abstract tokeniser.
Usage:
class Example::Driver# Device driver helpertokenize delimiter: "\x0D"end
Options:
Option | Description |
| sequence to detect the end of message. Supports strings and regular expressions |
| sequence to detect the start of a message |
| can be used with an indicator if messages are always a fixed length |
| prevents buffering from using all your memory if the end of a message is never detected |
| can help prevent false positives |
| defaults to |
Example:
tokenize indicator: "\x02", delimiter: "\x03"
and data: "yu\x03\x02hello\x03\x02world\x03\x02how"
Would have the following result:
yu\x03
would be discarded
hello
would be returned
world
would be returned
\x02how
would be buffered
The primary use case for this tokeniser is variable length messages, where length can be determined by the message contents. (commonly a length field in the header)
Usage:
class Samsung::Displays::MdSeriestokenize indicator: "\xAA", callback: :check_length# Called by the Abstract Tokenizerdef check_length(byte_str)# Check for minimum lengthreturn false if byte_str.bytesize <= 3response = str_to_array(byte_str)# data length byte + (header + checksum) == message lengthlen = response[2] + 4if response.length >= len# return the length of this message (any excess will be buffered)return lenelse# false if the complete message hasn't arrived yetreturn falseendendend
Options:
Option | Description |
| callable code, proc, lambda, method etc that will return an integer or false |
| sequence to detect the start of a message (string or regex) |
| prevents buffering from using all your memory if the end of a message is never detected |
| defaults to |
For a detailed overview of what these tokenisers are capable of, it is worth looking at their tests.