Parallel-to-serial-converter

[bitrex]

I’ve been working with variadic patch nodes and the custom output types to try to implement a parallel to serial converter/byte serializer patch. The idea is that you connect up any number of sources of bytes to a parallel set of inputs, and then there is a serial output which allows you to send them one at a time over say the UART or i2c. this is the general idea:

There is a variadic parallel to serial input that takes in bytes and emits a custom “self” type which is the internal “List” type. The nodes store an input byte in the state, converts it to a single element array in the state which holds a pointer to that byte, turns that into a single element list of a pointer to byte and emits that. The next stage then concatenates that single element with the list of X size coming in on the bus from the left.

Code for byte to list conversion is like:

using Type = List<uint8_t*>;

struct State {
    uint8_t byte;
    uint8_t* byte_arr[1];
    PlainListView<uint8_t*>* byte_view;
};

{{ GENERATED_CODE }}

void evaluate(Context ctx) {
    State* state = getState(ctx);
    if (isSettingUp())
    {
        state->byte_view = new PlainListView<uint8_t*>(state->byte_arr, 1);
    }

    state->byte = getValue<input_IN>(ctx);
    state->byte_arr[0] = &state->byte;
    Type out = Type::List(state->byte_view);
    emitValue<output_OUT>(ctx, out);
}

Code for the concatenation is like:

struct State {
    ConcatListView<uint8_t*> view;
};

{{ GENERATED_CODE }}

void evaluate(Context ctx) {
     State* state = getState(ctx);
     auto list1 = getValue<input_IN1>(ctx);
     auto list2 = getValue<input_IN2>(ctx);
     state->view = ConcatListView<uint8_t*>(list1, list2);
     emitValue<output_OUT>(ctx, List<uint8_t*>::List(&state->view));
}

And then code for the serialization is like:

struct State {
    Iterator<uint8_t*>* it;
};

{{ GENERATED_CODE }}

void evaluate(Context ctx) {
    if (!isInputDirty<input_UPD>(ctx))
        return;

     State* state = getState(ctx);

    if (isSettingUp())
    {
        state->it = new Iterator<uint8_t*>(Iterator<uint8_t*>::nil());
        auto byte_list = getValue<input_P2S>(ctx);
        *state->it = byte_list.iterate();
    }

    if (!(*state->it))
    {
        auto byte_list = getValue<input_P2S>(ctx);
        *state->it = byte_list.iterate();
    } else {
         //dereference pointer-to-iterator stored in the State, dereference 
         //iterator to get byte pointer and post-increment the iterator, dereference 
         //byte pointer and emit a byte. Yuck!
         emitValue<output_BYTE>(ctx, *(*(*state->it)++)); 
         emitValue<output_DONE>(ctx, true);
    }
}

Edit: The problem is in the code above, an Iterator type doesn’t have a postfix increment operator overload, only a prefix increment overload. Surprised it even kindof worked at all! Instead of that pointer-mess I think I’ll rewrite to do like auto& it = *(state->foo); to take a reference to the iterator (Iterators are non-copy-assignable) to make the code easier to reason about

Which generates an iterator for the “bus” list input and pushes bytes one at a time, then loops around. So ideally if I have a 4 input parallel-to-serial node up top with input values for the bytes in hex 0x41, 0x42, 0x43, 0x44 the UART at the bottom will spit out “ABCDABCDABCD…” etc. over and over.

So this kind of works but there are also some problems/improvements that could be made. There’s a bug such that the UART seems to be printing an extra character e.g. “ABCDDABCDDABCDD…” etc. instead of what’s expected, not sure what the problem is exactly. The method I’m using doesn’t seem to be particularly memory-efficient, the example patch in the image with a 4 byte “bus” and a UART takes over 5kb of program memory and nearly 400 bytes of SRAM on an Arduino Nano, the memory usage grows rapidly with increasing variadic byte inputs.

There is probably a better way to use the various list functions available like the one for list folding to accomplish this sort of thing and the fact that nodes already store their inputs in memory, or are constexpr, so I don’t think I should also have to store the input byte in the state, but I don’t really understand how to use that function! Also would be nice to make the inputs generic rather than just bytes. Additionally feeding parallel-to-serial-inputs from the single byte output of another parallel-to-serial-output doesn’t really work at all. You can connect up these “buses” and nodes in interesting ways “on paper” at least but anything other than just an input block into an output block you’re most likely to just get a processor reset.

Any suggestions on how to address primarily the off-by-one error or other improvements/issue would be appreciated.

ParallelToSerial.xodball (11.4 KB)

[bitrex]

I think the problem I’m having with this arrangement might be from trying to store an Iterator type in the heap, though from looking at the Iterator code it does say they are supposed to be on the stack but I don’t see immediately what the problem with putting one in the node State would be.

I think I’ll just try not using a stored Iterator at all and instead cache the result of list::length for the serial output node on startup, dump the incoming List of bytes to a flat array of appropriate size on the stack on each transaction cycle and hold an integer-type index in the State into that array between transaction cycles to serialize the list output.

[bitrex]

Ok, so I’ve revised the code and figured out the cause of one last issue with it, after I figure out how to resolve that this parallel-to-serial converter should work OK. Using “new” to create an Iterator on the heap and storing its pointer in a State is fine, there doesn’t appear to be any problem there.

Again three not-implemented-in-xod nodes using a custom output type for a list of uint8_t*, List<uint8_t*>. There are X number of pointers to bytes stored at the “top” of the variadic patch input node, generated from X byte inputs, and that’s concatenated down to the final output of a list of pointers to all of them. Then that’s handed off to the serializer node, which iterates through them and sends them out one at a time in response to pulses.

Patch node that stores an input byte, keeps its address into a single-element array, and makes it into a single element List of a pointer-to-byte (cheap-to-copy wrapper around a single-element ListView, which must be created on the heap via “new” and exist for the lifetime of the program):

using Type = List<uint8_t*>;

struct State {
    uint8_t byte;
    uint8_t* byte_arr[1];
    PlainListView<uint8_t*>* byte_view;
};

{{ GENERATED_CODE }}

void evaluate(Context ctx) {
    State* state = getState(ctx);
    if (isSettingUp())
    {
        state->byte_view = new PlainListView<uint8_t*>(state->byte_arr, 1);
    }

    state->byte = getValue<input_IN>(ctx);
    state->byte_arr[0] = &state->byte;
    Type out = Type::List(state->byte_view);
    emitValue<output_OUT>(ctx, out);
}

Concatenation code, that takes two Lists of pointers, concatenates them, and spits out a new List. In a variadic patch node the input on the right will be a single element List from the last byte in the set of X, the input on the left will be a list of all the elements that came before and were themselves concatenated:

struct State {
    ConcatListView<uint8_t*> view;
};

{{ GENERATED_CODE }}

void evaluate(Context ctx) {
     State* state = getState(ctx);
     auto list1 = getValue<input_IN1>(ctx);
     auto list2 = getValue<input_IN2>(ctx);
     state->view = ConcatListView<uint8_t*>(list1, list2);
     emitValue<output_OUT>(ctx, List<uint8_t*>::List(&state->view));
}

And the serializer, which takes the final List of all the pointers to the input byte data and iterates thru them in response to input pulses:

struct State {
    Iterator<uint8_t*>* it;
};

{{ GENERATED_CODE }}

void evaluate(Context ctx) {
    if (!isInputDirty<input_UPD>(ctx))
        return;

     State* state = getState(ctx);

    if (isSettingUp())
    {
        state->it = new Iterator<uint8_t*>(Iterator<uint8_t*>::nil());
        auto byte_list = getValue<input_P2S>(ctx);
        *state->it = byte_list.iterate();
    }

    auto& it = *(state->it);
    emitValue<output_BYTE>(ctx, **it);
    emitValue<output_DONE>(ctx, true);

    if (!++it)
    {
        auto byte_list = getValue<input_P2S>(ctx);
        *state->it = byte_list.iterate();
    }
}

So this works essentially OK, I can take any number of input bytes I want in parallel, variadically, and then spit them out to the serial port one by one:

The final problem is the base case in the concatenation. If I have “A,B,C,D” in ASCII hex representation as input bytes the serial port is sending out " ABCD ABCD ABCD" in a loop rather than “ABCDABCDABCD” which is what it should be. It’s because of the unconnected List input up at the top left in the screenshot here, the concatenation code adds the “default” List generated by the XOD transpiler for that unconnected input to the concatenation, which is a nil List, at the first level of the variadic expansion, and that seems to add a blank space to the stream that shouldn’t be there.

I need to find a way to strip that nil-List out if that first List custom type input is unconnected, but I haven’t found a way that works yet. Any suggestions from the devs would be appreciated!

ParallelToSerialRevised.xodball (10.8 KB)

[bitrex]

A proper base-case check for the concatenation node solved this problem.

void evaluate(Context ctx) {
     State* state = getState(ctx);

     auto list1 = getValue<input_IN1>(ctx);
     auto list2 = getValue<input_IN2>(ctx);
     auto it = list1.iterate();

    //if the list custom type coming in from the left holds 0x0 and is of length 1,
    //then it's the uninitialized base case custom input. Concatenate the right side
    //list with an empty list instead.

     if ((**it == 0x0) && (length(list1) == 1)) { 
          state->view = ConcatListView<uint8_t*>(List<uint8_t*>(), list2);
     } else {
          state->view = ConcatListView<uint8_t*>(list1, list2);
     }
     emitValue<output_OUT>(ctx, List<uint8_t*>{&state->view});
}

I’ve edited some of the state constructors and tided it up a bit, attached is the first working revision.

ParallelToSerial-working.xodball (11.7 KB)

[bitrex]

An insight I had tonight is that since I’m just passing out a list of pointers to bytes, most of the work I’m doing on each transaction re: concatenation etc. can be done just once on setup. The input data can change but the List type that’s output at the end of the variadic chain only contains pointers, those pointers can be const-qualified and the output List of them itself can be const also as the number of inputs is fixed at compile-time.

It’s probably possible to simplify that even further, I should be able to construct a List of function pointers instead of data pointers as well. getValue<input_IN>(ctx) is a free function taking type Context as argument and what particular node’s input I’m referring to is uniquely identified by the particular Context.

So I should just be able to construct a List of const structures holding a function pointer and the appropriate Context at setup and pass that down to the final output and iterate through that. I think ideally that should allow the whole thing to be essentially stateless, not using any more memory than just what’s used already by the input and output node’s buffers. Cool! I think I’ll try that approach next.

[bitrex]

Alright, so some evenings and head-scratching later I’ve got a set of P2S converter nodes that I think is worthy of a “1.0” release. It’s essentially a more refined version of my first approach using templates to make it generic and a factory method for packaging up data.

The insight I had before didn’t seem like a bad idea at the time, unfortunately because of the way XOD stores each node’s context data on the stack as each node executes, and per-node ContextObject structures are not maintained on the heap between node transactions, trying to make a list of Context variables and call some other node’s getValue method on its “evaluate” function’s context argument, from a different context, using some kind of lambda/wrapper seems a non-starter, the function pointer called with the former ContextObject pointer as argument will return garbage because that ContextObject was on the stack previously and no longer exists. so some type of buffer variable to capture and store an input value on the heap in the State of an input node and hold it there until the output node is executed seems mandatory.

I noticed that in the code for the “Iterator” class a dynamic allocation and release (new/delete) is used in the constructor and destructor, and I use an Iterator in a somewhat unusual way here, but in testing the way I’m using it doesn’t seem to lead to those calls creating memory fragmentation or blowing up the heap (it seems there are definitely wrong ways to use it that will, I unfortunately discovered… )

ParallelToSerial-1.0-8-29-18.xodball (17.6 KB)