Another month another post? I wish… This time, I got sucked into an almost-month long black hole of a trip and when I was spewed back into regular Warsaw reality it turned out I needed to dedicate 200% of my time to my day job. Scraping some free time I had (it’s Corpus Christi holiday in Poland as I am writing this) - I hence publish this blogpost where you can find some key discoveries of mine from Embedded Open Source Summit 2024, the Linux Foundation’s ONE Summit 2024 and a meetup talk of mine from Digital Futures 2024 happening here in Warsaw. All this is sprinkled with pictures from the beautiful Washington State which I started discovering during my week-long roadtrip!

Buckle up, here we go!

Embedded Open Source Summit 2024

This was my second EOSS and also the second one where I had the pleasure of being a speaker. You probably already guessed the topic! 😁 I was delighted to see quite the audience of makers/hackers and tinkering enthusiasts, and discussing all kind of tangents to the project after the presentation. Hopefully, the presentation and the project will serve as a good source of tips/ways of work when coming up with a great project idea and reduce the struggles in realizing it.

As for the rest of the conference - I mostly attended Zephyr/Embedded Linux topics although one could get lost in the myriad halls of co-located Open Source Summit and other smaller Summits. Prevailing topics were focused around supply chain safety and healthy contributor ecosystem (looking at you XZ backdoor), some other notable highlights include:

• the widespread adoption of SPDX as the de-facto SBOM tool which was not the case a few years ago
• we have a lot of Linux in Space - 80 running copies more every time a SpaceX’s Starlink is commissioned to work!
• Zephyr is becoming even more mature and widely adopted - the long awaited HTTP server support has been added recently forcing me to rewrite parts of ZLED Frame 🤦

ZLED Frame - how to Illuminate your artwork

In case you have not been following my project (yet! 👀) or missed my talk (it was streamed online 😁) - here is the video! After long chats with my audience after the talk I got some words of wisdom and encouragement (that I oh so need sometimes 😄). What I really enjoyed was to see that people always stumble upon similar issues, like measurements errors, messing the cabling up and other blunders - but that is something worth appreciating or even loving about making/hacking and computer science in general! I also got some helpful tips regarding a nasty bug where LEDs would not be lighted to a proper color/configuration - apparently there is a driver that is very picky about proper timing and whenever something disrupts it (interrupt request?) - utter rubbish will be displayed on the strip.

Link to the repository and project page

WA travelling

I won’t bore you with very succinct descriptions of my trip around the Washington State (although maybe I should) - rather, grab some best-of pictures from the trip, along descriptions of locations and marvel at them 😁

After leaving Seattle, I headed for Mt. Rainier, but since winter was still abound (apparently so), I ended in Packwood and explored nearby trails (with nobody but game and bears around yikes). From left to right: Lake Glacier, Vista of Mt. St. Adams and marvelous Lake Wallupt!

Afterwards I had to travel a couple of hours to Wenatchee to see some more praire’ish vistas. Lake Packwood, A great vista of Mt. Rainier (which I will visit one day) and the rocky way to Wenatchee.

In Wenatchee and then Chelan, I went hiking and marvelling at the landscapes of great-nothing until the horizon. First picture - Saddle Rock, then two more of the Chelan Lake Trail.

The next leg was a fairly long and beautiful loop through the Cascadia Mountains which took me just over 3 hours to cross! The weather ranged from full sun, through rain and snow to finally end with a misty and gloomy view on the Western side of the mountains. You can see the stark difference in the pictures below.

Right after crossing Cascadia, I headed for Anacortes where I took some strolls around the coastline and caught a ferry to the south-western part of WA - ending my voyage in Port Angeles.

If you are not already astonished by the sheer variety of weather conditions this state had to offer at this time of the yer, here comes the big punch! During the same day I waded through knee-deep snow and at the same time was under Olympic Rainforest’s torrential rain (kudos to me for buying the raincoat a couple of days before). Below - views from the melting Olympic Mountains.

It was really soaking wet there!

Glad I managed to hike a long way to see the Deer Lake where I met some folks who were so nice to offer me something warm to drink! Fishing, cooking and living their life, waist-deep in the snow but warm and cozy in their tent!

LF ONE Summit

Visiting the venue of this year’s NVIDIA GTC (which I unfortunately missed) left me in awe at how big these conference centres can be (and most of the time just a fraction of them is filled with people). This was a slightly more niche Summit, mostly focused around Core Network and futureproofing it against the 6G with widespread adoption of Network APIs, through even wider usage of Open Source to clever and safe usage of AI to achieve tasks. To my disappointment - not much revolved around Baseband there - so I can share with you only two projects that I have been keeping tabs on: SRSRAN Project an open source L1/2/3 implementation on commodity Hardware, and the project EvenStar which was initially conceived by Meta to allow anyone to rollout their own Open Source and Hardware Radio Unit (currently stewarded by the Open Compute Project).

Digital Futures 2024

This edition of the Digital Futures Meetup was happening in Warsaw so this time all I had to do was spend a short time commuting 😄 Compare that with 20/24+ hours of flight when travelling for overseas conferences and you will see why I am mentioning it!

Topics covered included LLMs, their usage, limitations and implications for other jobs, such as data processing and collection; this was paired with my (longer-than-intended) talk on Telecommunications and the role Open Source plays in shaping its future. Krzysztof and Tomasz delved deep into the various roles LLMs have inside Tietoevry and why they are only a complement to other more traditional ways of data collection and processing. This resonates heavily with my take on them - yet another (fast, amazing, powerful) tool in my engineering toolbelt.

My talk

In my presentation I tried merging what I talked about in Košice about FlexRAN and it’s importance in bringing the OpenRAN revolution to the Baseband side of 5G/LTE(FOSDEM 2024 talk), while showcasing how Open Source developments in the Core Network and Baseband helped bring the long-awaited shift in productivity and quality of code to these areas. Time proved to be my enemy and I had to skip a big part of what I wanted to present, but the avid discussion and genuine curiosity of the audience after the talk proved that I managed to sneak just the content that would provoke such discussions 😁 The big question is: What do the telecom vendors have to gain from Open Sourcing their Baseband parts of code? - My answer to that is clear: open just the most crucial parts and innovate in terms of hardware/algorithms or unique value propositions and take a look at how creating projects like ONAP or CNTi helped make the Core Network a more unified and resilient part of the 5G/LTE system.

Summary

As always, thank for staying with me until now - I hope you learned something and we that might see each other on upcoming events/travels 😁 Like I promised - the ZLED Frame updates are on their way and since there were some exciting updates to the Zephyr project - I will be busy rewriting the project to adhere to them! See you next time!

P.S. I plan to be writing some recommendations regarding what I have been (quite intensively) recently reading. Also, a small fastfoody reward (In-N-Out Animal Style!) for you who stayed for so long here 😄

This is the second part of the ZLED Frame series:

• First part - Project introduction and early prototyping
• Third part - Coming soon! - FreeCAD + 3D printing
• Fourth part - Coming soon! - Extras

Now have a nice early March Austrian sunset to accompany your reading:

Zephyr How-To and project set-up

We got so far without really digging into what exactly Zephyr is doing in this project. First of all, what is Zephyr? To me it has always been akin to a younger brother of Linux having equally strong virality and momentum but targeted more for embedded devices. The thing that really bought me is that you usually can get any off-the-shelf development board and it will work assuming the example is supported for this board(most of which are available for a plethora of devices). Compare that to installing dubious toolchains 🤨 or binaries accessible only after you sell your data, kidney and a cut on your profits 😄

(Yes I am biased towards proprietary software, why do you ask?) - no, but really? Licensed compiler??? - yes - companies do that, yes - you know the big names

I won’t go into details on how to obtain the code itself (come on, we are all grown-up here), rather I will point out my preferred ways of working with this project. I usually checkout the latest stable branch and optionally create a new branch for some modifications to the project’s code that might arise during hacking around. Always be sure to be working in the virtual environment in which west has all the paths and variables set up properly - refer to the official guide on how to set up that.

Since I am using an out-of-tree project structure basing on existing project repository where I specify overlays for my board, the best course of action for you is to follow my step. Trust me, this way you will probably save yourself from painful decoupling if you would like to move your project to be something bigger. What I don’t recommend though, is working outside the Zephyr’s virtual environment - don’t go this path of pain…

Once we have our project set up, we should create an app.overlay file with following contents:

/*
 * Copyright (c) 2024 Jakub Duchniewicz
 *
 * SPDX-License-Identifier: Apache-2.0
 */

#include <zephyr/dt-bindings/led/led.h>

&spi2 {
	/* Workaround to support WS2812 driver */
	line-idle-low;

        led_strip: ws2812@0 {
                compatible = "worldsemi,ws2812-spi";

                /* SPI */
                reg = <0>; /* ignored, but necessary for SPI bindings */
                spi-max-frequency = <6400000>;

                /* WS2812 */
                chain-length = <256>; /* arbitrary; change at will */
                spi-cpha;
                spi-one-frame = <0xf0>; /* 11110000: 625 ns high and 625 ns low */
                spi-zero-frame = <0xc0>; /* 11000000: 312.5 ns high and 937.5 ns low */
                color-mapping = <LED_COLOR_ID_GREEN
                                 LED_COLOR_ID_RED
                                 LED_COLOR_ID_BLUE>;
        };
};

$wifi {
    status = "okay";
};

/ {
        aliases {
                led-strip = &led_strip;
        };
};

In case you are not familiar with the Device Tree Structure files I recommend(as always) great Bootlin’s presentations that very concisely describe what’s what.

The overlay above initializes the SPI connected Data pin of the LED strip, enables the Wi-Fi and creates an alias for the LED strip node in the Device Tree. Overlays are special entities that are used to add or alter nodes of the Device Tree that are added during compilation. In our case we have a standalone ESP32-Wroom board that has some fixed peripherals and settings specified first in the SoC’s device tree file and then the board itself and we simply add additional external peripheral that we have to configure during compilation. (There have been recently some changes since Espressif introduced a new hardware model which I still need to go over - so expect me explaining that in a future post).

Assuming you want to use the same LED strip with ESP32 you will simply need to adjust the chain-length parameter at will.

Device tree is just one part of the configuration - we also need to add several basic config defines to augment those included in the default app:

CONFIG_LOG=y
CONFIG_SPI=y
CONFIG_LED_STRIP=y
CONFIG_WS2812_STRIP=y

## Copied over from the sample - to reduce
# POSIX options
CONFIG_POSIX_MAX_FDS=20

# Networking config
CONFIG_NETWORKING=y
CONFIG_NET_IPV4=y
CONFIG_NET_IPV6=n # it is set to yes either way
CONFIG_NET_TCP=y
CONFIG_NET_UDP=n
CONFIG_NET_SOCKETS=y
CONFIG_NET_SOCKETS_POSIX_NAMES=y
CONFIG_NET_MAX_CONN=20
CONFIG_NET_MAX_CONTEXTS=20
CONFIG_NET_STATISTICS=y
CONFIG_NET_CONNECTION_MANAGER=y

# Network buffers
CONFIG_NET_PKT_RX_COUNT=96
CONFIG_NET_PKT_TX_COUNT=96
CONFIG_NET_BUF_RX_COUNT=128
CONFIG_NET_BUF_TX_COUNT=128
CONFIG_NET_CONTEXT_NET_PKT_POOL=y
CONFIG_NET_BUF_POOL_USAGE=y

# IP address options
CONFIG_NET_IF_UNICAST_IPV6_ADDR_COUNT=3
CONFIG_NET_IF_MCAST_IPV6_ADDR_COUNT=4

# Network address config
CONFIG_NET_CONFIG_SETTINGS=y
CONFIG_NET_CONFIG_NEED_IPV4=y
CONFIG_NET_CONFIG_MY_IPV4_ADDR="192.168.0.230"
CONFIG_NET_CONFIG_PEER_IPV4_ADDR="192.168.0.231"
CONFIG_NET_CONFIG_NEED_IPV6=y
CONFIG_NET_CONFIG_MY_IPV6_ADDR="2001:db8::1"
CONFIG_NET_CONFIG_PEER_IPV6_ADDR="2001:db8::2"

# Network debug config
CONFIG_NET_LOG=y
CONFIG_NET_SHELL=y

# This is needed so that the sample app can respond to queries
# as fast as possible.
CONFIG_NET_TCP_TIME_WAIT_DELAY=0

# This is needed for Wi-Fi connections
CONFIG_NET_L2_WIFI_SHELL=y

CONFIG_ESP_HEAP_MEM_POOL_REGION_1_SIZE=1024
# is this necessary?
CONFIG_HEAP_MEM_POOL_SIZE=98304


## overlays for ESP32
CONFIG_WIFI=y
CONFIG_NET_L2_ETHERNET=y
CONFIG_NET_DHCPV4=y
CONFIG_ESP32_WIFI_STA_AUTO_DHCPV4=y

Most importantly, we need to enable logging, Wi-Fi, IPv4 capabilities, the WS2812 LED strip support, and change the CONFIG_ESP_HEAP_MEM_POOL_REGION_1_SIZE to be 1024 otherwise we will be hitting errors binary creation due to too small memory regions. Lastly, we need to set up the IPv4 address that we will be connecting to from our client (be careful not to overlap with any existing one on your local network).

Application logical split

Even though it is not a grand project, I would be neglecting my computer science background without partitioning it into encapsulated components, so here they are:

• main - self-explanatory, calls the interface functions and spawns threads
• pixels - arranging the pixels, creating frames and pushing them to the LED driver
• network - basing on the dumb_http_server sample, establishes the HTTP server and responds to API calls, also parses the incoming data and informs the pixels module that the data was obtained

As you can see it is really basic but allows us to freely add more components if we wish without cluttering the main file too much. Thread-wise we also don’t have much going on and we only need to synchronize the threads whenever a new image is provided over the network so that proper data is displayed on the screen.

Interfacing WS2812

As we have already seen above, we are using the LED strip containing the WS2812 LEDs having a very tiny datasheet. Even though the datasheet is tiny (and does not really tell us what is the factual power consumption of a single LED) it shows that we have just 3 wires, namely VCC(Voltage input), GND, and a Data in pin that is interfaced from our ESP.

The driver is also fairly simple, uses bit banging? (verify that) - we won’t go into details, and that’s also one thing that is so great about Zephyr - when you have drivers, it just works! When you don’t have the drivers, well then that’s a different story, but then usually you are experienced enough to at least tackle it on your own 😁

For the prototype I used a battery pack that initially supplied 6V (4 * 1.5V battery) so I had to use several voltage dropping diodes (those I had laying around were of the Schottky type so they had slightly lower voltage drop). One thing that I must warn you about when you would like to use such a battery pack is that the batteries yield lower voltage over time (as they discharge), so after several hours of debugging I stumbled into a very weird error: I could not turn the LEDs blue…

Dumb as it may sound, I was thinking that maybe there was a mistake in the driver or the hardware itself was faulty, only after several hours of digging around and learning about blue LED production difficulties - I realized that maybe the voltage is simply too low for the blue LED to light up. Removing several of the voltage dropping diodes solved the problem 🤦

There is a lesson in this for all of us - never trust your batteries 😁

After making sure the LEDs can shine in their entirety of the spectrum, one can start giving them something more lofty than just test patterns, but before we go into displaying images from memory, let’s go briefly over the code of how to interface them. As I said before the beauty of working with a full-fledged system is that you usually have most things abstracted in the form of API (as long as it works 😄)

In my case it was as simple as getting the device from the device tree, initializing it and calling led_strip_update_rgb with the data I want to display.

   #define STRIP_NODE DT_ALIAS(led_strip)

   static const struct device *const strip = DEVICE_DT_GET(STRIP_NODE);

Then we call a simple calibration routine:

int display_preset_pattern(const struct device *const strip)
{
    int rc;

    memset(&pixels, 0x00, sizeof(pixels));
    memcpy(&pixels[cursor], &colors[color], sizeof(struct led_rgb));
    rc = led_strip_update_rgb(strip, pixels, STRIP_NUM_PIXELS);

    if (rc)
    {
        return rc;
    }

    cursor++;
    if (cursor >= STRIP_NUM_PIXELS)
    {
        cursor = 0;
        color++;
        if (color == ARRAY_SIZE(colors))
        {
            color = 0;
        }
    }
    return 0;
}

My code is slightly more complex because I am running it in a thread and waiting for the data to arrive from the client and then I need to reverse the data as the LED is soldered in a snake pattern like in the below figure while the data is ordered sequentially in the memory buffer.

This can be also done on the client side to reduce the workload on the MCU 😁

int display_network_image(const struct device *const strip)
{
    int rc;

    // Wait for the semaphore indicating a new image is ready
    if (k_sem_take(&image_semaphore, K_NO_WAIT) == 0)
    {
        LOG_ERR("Received image - semaphore taken");
        //  Copy the received image to the pixels array
        memset(pixels, 0x00, sizeof(pixels));
        // convert pixel data to led_rgb
        size_t index = 0;
        for (int i = 0; i < STRIP_LINE_LENGTH; ++i)
        {
            for (int j = 0; j < STRIP_LINE_LENGTH; ++j)
            {
                // since the led strip is sequential, we have to reverse every second line
                if (i % 2 == 0)
                {
                    pixels[index] = RGB(received_image[index * 3], received_image[index * 3 + 1], received_image[index * 3 + 2]);
                }
                else
                {
                    size_t reversed_index = i * STRIP_LINE_LENGTH + STRIP_LINE_LENGTH - j - 1;
                    pixels[index] = RGB(received_image[reversed_index * 3], received_image[reversed_index * 3 + 1], received_image[reversed_index * 3 + 2]);
                }
                ++index;
            }
        }

        // Update the LED strip with the new image
        rc = led_strip_update_rgb(strip, pixels, STRIP_NUM_PIXELS);

        if (rc)
        {
            return rc;
        }
        LOG_ERR("Finished image drawing");
    }

    return 0;

As you can see the pixel displaying code is really simple, we will extend (and optimize 😄) it in the near future.

Networking primer

Networking-wise, the ESP works as a server - meaning that it provides the HTML + CSS + Javascript code needed to display the website on the client which may be your PC, mobile phone or even another microcontroller (of course assuming it has a browser capabilities). The server is running on the Wi-Fi network with a static IP so everyone on the network can access it (yes, it is not the most secure thing but since my network is fairly well isolated from the outside world I am not really afraid - besides it is not a drug injector). Also, implementing full network security is beyond this blogpost, but you are always encouraged to read up on it 😄.

When I connect to the server, I make a request to the root endpoint - / and the server transfers the binary blob containing the code to be run on the client. The HTML code is rendered by the browser and the Javascript code executes either immediately or when special action is undertaken (i.e) button press. The code runs on the client so we can leverage a more powerful machine to do our dirty data processing 😁

In our case we offload image pre-processing and massaging it to be a raw-byte array that we receive on the server via an API-call to the /api/image endpoint. There are some caveats that we have to consider when we are implementing networking ourselves - such as, parsing (or ignoring) message headers and knowing when we receive desired data.

Overall, the flow looks like that: (basic functionality for now)

Server Side

We talked high-level so let’s now get more concrete and see how it is implemented in code. Everything starts by spawning a thread for TCP IPv4 processing (somehow I got weird crashes when not enabling IPv6) - this thread creates a BSD socket that will be used as a high-level abstraction for communication with the client.

static void process_tcp4(void)
{
	struct sockaddr_in addr4;
	int ret;

	(void)memset(&addr4, 0, sizeof(addr4));
	addr4.sin_family = AF_INET;
	addr4.sin_port = htons(MY_PORT);

	ret = setup(&tcp4_listen_sock, (struct sockaddr *)&addr4,
				sizeof(addr4));
	if (ret < 0)
	{
		return;
	}

	LOG_DBG("Waiting for IPv4 HTTP connections on port %d, sock %d",
			MY_PORT, tcp4_listen_sock);

	while (ret == 0 || !want_to_quit)
	{
		ret = process_tcp(&tcp4_listen_sock, tcp4_accepted);
		if (ret < 0)
		{
			return;
		}
	}
}

Then we accept and create a thread for receiving the data:

static int process_tcp(int *sock, int *accepted)
{
	static int counter;
	int client;
	int slot;
	struct sockaddr_in6 client_addr;
	socklen_t client_addr_len = sizeof(client_addr);

	client = accept(*sock, (struct sockaddr *)&client_addr,
					&client_addr_len);

	/* ommited */

#if defined(CONFIG_NET_IPV4)
	if (client_addr.sin6_family == AF_INET)
	{
		tcp4_handler_tid[slot] = k_thread_create(
			&tcp4_handler_thread[slot],
			tcp4_handler_stack[slot],
			K_THREAD_STACK_SIZEOF(tcp4_handler_stack[slot]),
			client_conn_handler,
			INT_TO_POINTER(slot),
			&accepted[slot],
			&tcp4_handler_tid[slot],
			THREAD_PRIORITY,
			0, K_NO_WAIT);
	}
#endif

	if (LOG_LEVEL >= LOG_LEVEL_DBG)
	{
		char addr_str[INET6_ADDRSTRLEN];

		net_addr_ntop(client_addr.sin6_family,
					  &client_addr.sin6_addr,
					  addr_str, sizeof(addr_str));

		LOG_DBG("[%d] Connection #%d from %s",
				client, ++counter,
				addr_str);
	}

	return 0;
}

Then whenever a request is made we read the data from the socket using the recv API and then match the received endpoint against the list of allowed endpoints (that we specify) - note the parse_header trick I added as the recv reads data in batches and we might have already read the header of the request. It is probably not the most robust way of doing things but for my limited application it works just fine.

static void client_conn_handler(void *ptr1, void *ptr2, void *ptr3)
{
	ARG_UNUSED(ptr1);
	int *sock = ptr2;
	k_tid_t *in_use = ptr3;
	int client;
	int received;
	int ret;
	char buf[RECEIVE_BUF_LEN];

	char endpoint_buf[20];
	bool parsed_header = false;
	method_t method;

	client = *sock;

	do
	{
		received = recv(client, buf, sizeof(buf), 0);

		if (received == 0)
		{
			/* Connection closed */
			LOG_DBG("[%d] Connection closed by peer", client);
			break;
		}
		else if (received < 0)
		{
			/* Socket error */
			ret = -errno;
			LOG_ERR("[%d] Connection error %d", client, ret);
			break;
		}

		LOG_DBG("[%d] Received data: %.*s", client, received, buf);

		if (!parsed_header)
		{
			if (parse_header(buf, sizeof(buf), endpoint_buf, sizeof(endpoint_buf), &method) != 0)
			{
				LOG_ERR("[%d] Could not parse header", client);
				break;
			}
			parsed_header = true;
		}

		if (handle_endpoint(client, endpoint_buf, method, buf, RECEIVE_BUF_LEN) == 0)
		{
			LOG_ERR("Handled the endpoint - exiting.");
			break;
		}
	} while (true);

	(void)close(client);

	*sock = -1;
	*in_use = NULL;
}

We first parse_header and allow only ones that we desire - note that this is a VERY incomplete HTTP server 😁

static int parse_header(char *buf, int buf_size, char *endpoint, int endpoint_size, method_t *method)
{
	char *ptr = buf, *delim_pos = NULL;
	int len = 0;
	if (strstr(ptr, "GET"))
	{
		*method = GET;
		ptr += 4; // +1 for whitespace
	}
	else if (strstr(ptr, "POST"))
	{
		*method = POST;
		ptr += 5;
	}
	else
	{
		LOG_ERR("Unknown method found!");
		return -1;
	}

	delim_pos = strchr(ptr, ' ');
	len = delim_pos - ptr;
	if (len + 1 > endpoint_size)
	{
		LOG_ERR("Too long endpoint name! Max allowed: %d", endpoint_size);
		return -1;
	}

	strncpy(endpoint, ptr, len);
	*(endpoint + len) = '\0'; // null-terminate it

	LOG_DBG("Found method of type: %d with endpoint name: %s", *method, endpoint);

	return 0;
}

Then we handle the endpoint using handle_endpoint the function:

static endpoint_t valid_endpoints[NUMBER_OF_ENDPOINTS] = {
	{"/", GET}, // TODO :add handling for favicon :)
	{"/api/image", POST},
};

static int handle_endpoint(int client, char *endpoint_buf, method_t method, char *buf, int buf_len)
{
	//  GET /
	if (strncmp(valid_endpoints[0].name, endpoint_buf, strlen(endpoint_buf)) == 0)
	{
		// here we don't need to check any return values and can directly return the page to render
		LOG_ERR("Sending the data to the client!");
		(void)sendall(client, content, sizeof(content));
		return 0;
	}
	else if (strncmp(valid_endpoints[1].name, endpoint_buf, strlen(endpoint_buf)) == 0)
	{
		LOG_ERR("Handling POST /api/image");
		static bool read_header = false;
		uint8_t length = 0u;

		if (read_header)
		{
			LOG_ERR("rcv_img_offset + buf_len: %d MAX_IMAGE_SIZE: %d", rcv_img_offset + buf_len, MAX_IMAGE_SIZE);
			// for now this will inform that we handled all
			if (rcv_img_offset + buf_len >= MAX_IMAGE_SIZE)
			{
				LOG_ERR("Stop accepting - Trying to send more data than Image can accept.");
				length = MAX_IMAGE_SIZE - rcv_img_offset;
				if (length > 0)
				{
					// this contains raw binary data - 0 is null terminator there :)
					memcpy(received_image + rcv_img_offset, buf, length);
				}
				rcv_img_offset = 0;
				read_header = false;

				// Give the semaphore to notify that a new image is ready
				k_sem_give(&image_semaphore);

				// sleep a short amount so that the waiting thread can read the data
				k_sleep(K_MSEC(20));
				// Wait for the image to be processed
				k_sem_take(&image_semaphore, K_FOREVER);
				return 0;
			}

			memcpy(received_image + rcv_img_offset, buf, buf_len);

			rcv_img_offset += buf_len;
			return -1;
		}

		// until get the magic number, keep reading the header 0xBADAD00B
		// there is a slight chance that the magic number will be split between two packets - but for now we don't handle it
		char *pos = strstr(buf, magic_number);
		if (pos)
		{
			LOG_ERR("Found the magic number!");
			read_header = true;

			// read the remainder as the image data
			rcv_img_offset = 0;
			length = buf_len - (pos - buf) - strlen(magic_number);
			memcpy(received_image, pos + strlen(magic_number), length);
			rcv_img_offset += length;
		}
		LOG_ERR("Still parsing the header");
		return -1;
	}

	LOG_ERR("Unknown endpoint and header!");
	return 0;
}

You can see that this function is fairly long while not doing that much. The main issue here is that we need to handle each distinct API endpoint differently and also parse/ignore the header fields, I added a magic value of 0xBADAB00B to know when my desired payload begins. This magic value is added on the client side as well. Astute readers might note that the magic value might be split between two consecutive buffers - this will be fixed soon but for now we don’t really need to care as the header length is fixed - but it should be fixed nonetheless.

When the entirety of the image payload is read the semaphore is kicked and the image is displayed on the LED strip - you might say our job here is done 😁

Client Side

Client-wise we have a very basic HTML + CS + Javascript code. For now it is really basic, but we will put our frontend magic skills to use in a moment so let’s go over the full code in a future post. For now just a small snippet that downsamples the uploaded image and calls the /api/image endpoint:

function processImage() {
  var fileInput = document.getElementById('fileInput');

  // Check if a file is selected
  if (fileInput.files.length > 0) {
    var file = fileInput.files[0];

    // Read the file as a data URL
    var reader = new FileReader();
    reader.onload = function(event) {
      var dataURL = event.target.result;

      // Display the original image preview
      var originalPreview = document.getElementById('originalPreview');
      originalPreview.innerHTML = '<img src="' + dataURL + '" width="200" alt="Original Image"/>';

      // Create an image element
      var image = new Image();
      image.onload = function() {
        // Create a canvas element
        var canvas = document.createElement('canvas');
        canvas.width = 16;
        canvas.height = 16;
        var ctx = canvas.getContext('2d');

        // Draw the image onto the canvas (resizing it to 16x16)
        ctx.drawImage(image, 0, 0, 16, 16);

        // Get the pixel data from the canvas
        var imageData = ctx.getImageData(0, 0, 16, 16);
        var pixelData = imageData.data;
        console.log(pixelData);

        // Create a Uint8Array to store the raw binary data
        var binaryData = new Uint8Array(16 * 16 * 3);
        var newIdx = 0;
        for (var i = 0; i < pixelData.length; i += 4, newIdx += 3) {
          binaryData[newIdx] = pixelData[i];           // Red
          binaryData[newIdx + 1] = pixelData[i + 1];   // Green
          binaryData[newIdx + 2] = pixelData[i + 2];   // Blue
        }

        processedImage = binaryData;

        // Display the converted image preview
        var convertedPreview = document.getElementById('convertedPreview');
        convertedPreview.innerHTML = '<img src="' + canvas.toDataURL() + '" width="200" alt="Converted Image"/>';

        // Log the binary data to the console
        console.log('Binary Data:', binaryData);
      };

      // Set the image source to the data URL
      image.src = dataURL;
    };

    // Read the file as a data URL
    reader.readAsDataURL(file);
  } else {
    alert("Please select a file before processing.");
  }
}

function uploadProcessedData() {
  // Create a new Uint8Array to hold the combined data
  var combinedArray = new Uint8Array(processedImage.length + 8); // 8 bytes for the "BADAB00D" string

  // Set the first 8 bytes to the "BADAB00D" string
  var magicHeader = new TextEncoder().encode("BADAB00D");
  combinedArray.set(magicHeader, 0);

  // Set the remaining bytes to the processed image data
  combinedArray.set(processedImage, 8);

  // Create a binary blob from the combined data
  var blob = new Blob([combinedArray]);

  console.log('Sending image + magic:', combinedArray);

  // Add your server endpoint URL here
  var endpointUrl = '/api/image';

  // Upload data to server
  fetch(endpointUrl, {
    method: 'POST',
    body: blob,
    headers: {
      'Content-Type': 'application/octet-stream' // Specify the content type as binary
    }
  })
  .then(response => response.json())
  .then(data => {
    // Handle server response if needed
    console.log('Server response:', data);
  })
  .catch(error => {
    console.error('Error:', error);
  });
}

Please note that we convert the RGBA input array to RGB values (this was confusing me because I assumed the input was RGB not RGBA so imagine the rubbish I got on the output - and even displayed!). Also, you can see that we are prepending the magic value of BADAB00D so that we may know when we receive the payload.

Closing thoughts

We have covered most important code that powers this project, still, in a future post I will go over several other additions to this project, such as animations and prettifying the UI. The aim of this blogpost series is for you to understand the most important concepts and be able to follow through by yourself - all code and design files are on GitHub so all you really need is to follow my tracks and maybe build something even better 😄

Next up, me learning FreeCAD, discovering yet again that I hate when programmers think cryptic error messages help anyone, 3D printing, re-calibrating my 3D printer, fighting with first-layer adhesion issues and then 3D printing again so stay tuned 😁

I am leaving you with a picture of my happy face with something working after I assembled and resoldered it:

Until later! (Yes I know that 2 weeks between blogposts is not feasible - probably one per month is the most often I can)

Brussels daytrip

I arrived in Brussels early on Friday morning and was forced to travel all the way from Charelroi airport (including standing in 1,5 hour queue just to get on a bus) where said bus would drive me around Brussels’ suburbs for yet another 1,5 hour until we arrived in the Brussels Midi station. Thankfully, these were the only inconveniences of the upcoming day.

I won’t bore you about all the usual tourist crap, instead let me just leave you with several pictures and the recommendation to eat Belgian fries, drink a Belgian Dubbel or Trippel (or a Blonde) and visit the Magritte museum to get a dessert in form of abstract art.

Also some works of Magritte that caught my eye (pissed pig or pre-Minecraft voxels):

FOSDEM in general

I have been encouraged to come to FOSDEM for several years already and heard only superlatives about this conference - mainly because it is Free as the F in FOSS and because it has Beer as .* in FOSS 😁. I must say I was not disenchanted by a single bit (or byte) as during my entire stay at the premises of the ULB where it took place, people have been more than helpful and everyone was in the networking and learning mood that I all so enjoy.

Szymon was supposed to join me but was forced to cancel his talk due to blockers that he faced in the course of his thesis development (the talk was based on his thesis progress) - but worry not, he will come next year (even if he does not know that yet :D). Also Michał, who is not really a developer of FOSS software but rather its user, considered coming, since travelling to Brussels from London is fairly quick and still not very expensive (by train).

I must say I am quite envious of my Western European friends who happen to travel from Germany or France to Brussels by train instead of an airplane, but then we have other benefits by being slightly more remote 😄 Let’s not get sidetracked though and let’s get to the main course of this blogpost - my talk on the role of Open Source in the development of 5G and LTE networks.

There were hallmarks of FOSS everywhere!

My talk

Even though it was quite short (around 20 minutes as the Network track was very packed this year), my talk garnered the audience of around 300 people that is the biggest so far. This has also been a talk that was more day-job related than my previous ones so I could finally boast of my experience in FlexRAN’s development for Intel or other work around 5G and ORAN networks. If you were to remember one thing from the talk here it is: “It’s great that we have some established projects in the Core Network like ONAP or Sylva, but we really need a solid open source foundation in the lowest layer of the processing chain!”

After this call to action you know the gist of my talk, though re-watching it could give you more insight about the current state of the telecom industry and the problems and opportunities that it is currently facing. Apart from the said call-to-action, I explain how 5G (and especially L1) works and why we care about opening of the ecosystem. I also go over several foundational projects like DPDK, ARM RAL or CESNET’s ORAN MPlane libraries that all start to form the open source core of L1 in 5G and soon 6G.

I am leaving you here with 2 pictures related to the 5G protocol stack and network topology so that you may at least have a glimpse into the complex world of 5G. The picture below depicts the overall structure of the network with the main components depicted:

• Radio Unit - the most physical part of the network, think very complex physical calculations being done in real-time, so quick that it is implemented on ASICs/FPGAs most of the time.
• Distributed Unit - depending on the network split, more or less calculations are performed here - this is the interface to the UPF. Usually up to L2/L3 layers are implemented here.
• User Plane Function - tasked with higher-level functions of the network, such as packet filtering or quality of service.
• Core Network - here be dragons - so many distinct things happening here, from security, to network management, updates and other things that are deemed important by the network operators. Also makes sure your packets are routed to the outside world and that you receive your data 😁

(If you cannot see some parts of the images, please toggle the Dark/Light theme settings in the context menu - they are .svg so have a transparent background)

Now please take a look at the protocol stack and where exactly is L1 in all this mess:

In all honesty, if you are at least slightly interested in 5G networks and about their current state, my talk will give you at least some knowledge and point you to proper resources to learn more. And if said topics are something that resonate with you, we will be more than happy than work with you (we meaning my current employer - Tietoevry).

Zephyr, Emulators and Retrocomputing

Zephyr

Having talked more than 2 hours with folks after my talk I managed to catch up with people from the Zephyr project and we even went for a dinner where we discussed our recent experiences with various chips (I could prattle about my struggles with ESP32C6 and lack of support for it - yet). If any of you is into embedded development and is currently searching for a great and friendly project - search no more - Zephyr got you covered. Meeting with them reminded me to push the PRs that enhance the documentation to the HTTP server samples which I have currently been porting to my newest project.

I managed to catch several hours of sleep (next to the European Council - the Schumann station) and went for another busy day which I spent almost entirely on two tracks - Emulators and then Retrocomputing. Every time I meet folks that do these things I get a small impostor syndrome - how can you be an electronics, system, and game developer at the same time?! Well, it turns out that with dedication and commitment (and a ton of curiosity) such people are molded. Needless to say, mark my words, my next project will be in the emulation area (but more on that in future blogposts 😁)…

Emulators

I won’t cover all talks from these tracks but rather focus on the most eyebrow raising ones:

The Emulator track opened with talks on my beloved GameBoy emulation and getting started with that. Firstly we had a talk on Nintendo 3DS emulator - Panda3DS including a very in-depth discussion about various 3DS internals and peripherals. Probably the most important useless fact is that Nintendo did software SHA calculation even though they had a hardware chip to EXACTLY DO THAT 😁

Next up, there was a talk on approaching emulation and reverse-engineering Android operating system including the oh-so-dreaded Primary Bootloader, which turns out to be a very complex code when disassembled. I really enjoyed the talk because we could have a glimpse how a national institution approaches such problems - and believe me they do it very professionally. Also, it included using Ghidra and live patching the binary with scripts, Unicorn and Qiling to make the program behave as the researcher deemed it. I also learned a new word (or a neologism) Concolic execution - which is a conjunction of concrete and symbolic. Basically it means that we choose heuristics to progress with the program execution to cover all of our desired paths (though I know it is still complex…)

There was also a talk on how Microsoft recently started supporting ARM64EC (which stands for emulated code), where I could learn about the pleasures (or lack of them) in marrying x86 instruction set with ARM64 one with both of them having different register/flag and caller/callee register preserving conventions. I highly recommend watching it if you like to listen how Microsoft chose the proper way to approach a problem that is solved by the rest of us in a totally different way 😁

Retrocomputing

The next half of the day was dedicated to retrocomputing machines, such as Z80, GameBoy Advanced one more time and the mysterious Sanco 6003 computer. The hero of the first talk was a Commodore CBM II. I don’t stop to be amazed by the ingenuity of older computer designs - and I was not disappointed now - this machine had a possibility for connecting a secondary CPU to be working either as the main one or a co-processor!

If you are curious about other such examples, I highly recommend The Bald Engineer and his talk from last year’s Hackaday Supercon about bringing the Apple ][ machine to life. (It’s a link to his video on that topic - could not find the actual recording…)

The talk on GBA mostly tackled tooling and resources for folks who want to hack around with this retro piece of console (like me) that I will shamelessly plug in here:

• gbdev.io - community page for GBA hackers
• CTR - a Complete Technical Reference for GBA
• romhacking - a collection of hacked ROMs that are free to use and peruse (and to contribute of course 😄)
• assembly - GB Assembly for modern Game Developer
• GB ASM tutorial - how to make GB games from scratch

Lastly, I really enjoyed and kept grinning when 2 guys were presenting their thrash-prize - Sanco computer that turned out to be Franco-Japanese machine that (no surprise here) had scarce documentation. They reverse engineered the schematic (**from the traces on the board!!!*) and continue to add support and documentation for it. When they were talking about solutions on printing the characters to the screen and how the characters are stretched vertically or horizontally using very simple combinatorial circuits, for a moment I wished I was back at Uni doing Boolean tables and digital circuit optimization. But worry not, this moment passed as quickly as it arrived 😁

Summary

Although this was the first FOSDEM I ever attended, it surely won’t be the last - Michal and Szymon, shame on you if you don’t come next year! The sheer number of people attending this event made me feel a part of something greater even more so than during my last year’s GDC. What’s better, this event was all about FREE technologies that happen to power most of our modern world and are ever so important while, sadly, often overlooked. Being part of these conferences always makes me want more and gives me food for thought and new projects ideas 😁 Speaking of, I need to get back to the ZLED project that already started gathering some dust as I didn’t shine these LEDs while being sucked in by the Zephyr’s networking stack. Worry not, we will have a shiny 3D printed case for the frame soon . Until then!

Thus, after thinking what project would I really like to have around that would be a base for my further reverse-engineering/retro-gaming efforts, I came up with the LED-frame concept that has already had several previous renditions, like Raspberry-Pi Frame, or another similar one. Since in previous attempts some things were left out and, to my knowledge, none of previous efforts utilized Zephyr and less powerful MCUs like ESP32, I decided to share my work on this project here, and maybe on some other social media platforms (not really an avid user of any of them). Of course, all of the project will be shared on my GitHub as I live by the rules of Free Open Source Software (and hardware), in fact I am travelling to this year’s FOSDEM so expect some updates from there too!

Goal and how to get there?

The goal here is quite straightforward - I want to have a physical LED frame that I can hang in my living room that would display static/animated pictures of my choice on a 16x16 pixel grid. As with all ideas, it is too vague so let’s flesh it out then:

• Physical LED frame - meaning that I will have to design it and then either build from regular materials like wood or plastic, or 3D-print it. In my case it will be the latter, but I am not restricting anyone here 😁.
• 16x16 pixel grid - sounds like I need to have either separate pixels or use a LED strip, or even have an entire programmable LED array - I chose the LED strip for its cost and malleability.
• display static/animated pictures of my choice - this means I need a way of communicating with the frame, so it definitely needs to be smart (the question is how smart).

Now that it sounds less like a wish and more like engineering project, let’s address the rest of the requirements that were not voiced anywhere:

• Controller - implied in the design is of course the fact that something needs to drive the LEDs and decide what to display there, in my case it will be ESP32 (tried the new one based on RISC-V, more on that later).
• Power management - since LEDs are powered from 5V voltage source and ESP32 is capable of outputting only 3.3V we need either a logic-level shifter or separate voltage source.
• Data processing - displaying stuff on such a small resolution requires some image pre-processing and we are speaking about embedded devices that might not have the processing power.
• Communication - well, how to interface the Controller? We have mediums/protocols of choice, such as HTTP over Wi-Fi or using BLE for simpler communication.

Steps

Okay, now it seems like we have something tangible. Let me present you the plan to get there - by taking small steps and documenting my progress so people can follow along:

• First, there is this blogpost that introduces the project and the prototype.
• Then, we will have either one or two that dig into the details of setting up a HTTP server on Zephyr, connecting the LEDs and displaying the picture of choice on them.
• We also need to talk about assembly and some hardware stuff, like 3D design and printing (here I am a layman and I might request Szymon’s help.
• Lastly, we should talk about powering it up, having some Li-Po (or other) battery and a charging circuit.
• (there might be changes to the schedule along the way, since we need to design the Frontend for the controlling website, dig around Zephyr, etc.) - but expect a blogpost roughly every 2 weeks!

After the last blogpost we should have a solid base for further experiments and extensions to this project, but I am will not jump ahead and first focus on actually assembling it!

Prototyping

Here I won’t go into very technical details (this will be done in the second blogpost), rather I will focus on why even do prototyping and why not do it The Proper Way(TM). If any of you worked on a big software/hardware (and got forbid a mix of both) engineering project, you probably know that the path from A (conception) to B (the product doing something useful) is loooooooooooong. I think the complexity is logarithmic here (but don’t quote me on that 😁), the more people and components are involved, the more things can go wrong and therefore will go wrong thanks to our old friend - statistics.

So, how to quickly ascertain that an idea is tangible and can be realized? By rushing to have something very basic ASAP - this means prototyping (there even is an earlier form of it called pretotyping, that validates the idea even earlier!). Following this ideology, my way forward with this project was simple - have something displayed on a basic LED grid ASAP. This usually means doing nasty hackish things, such as copy-pasting code from samples, putting everything into main.c and using cardboard/paper scraps for assembly. And, that’s exactly what I did and am totally not ashamed to showcase in the pictures below 😁.

The exact steps of connecting and powering the LED matrix are coming soon and for now please accept some teasers!

After some more assembly:

And finally the classic blinky!

ESP32C6

While I promised not to get very technical in this post, I should present my reasoning of choosing ESP32 for this task. Previous year, I attended the RISC-V Summit NA where I obtained a cute ESP32C6 prototype board (thanks Tiffany!) and decided to use it for this project to further my RISC-V knowledge. However, as it turns out, the board is not yet supported in the Zephyr ecosystem, and basing on my previous experience with adding more support for a particular board to Zephyr, I decided to focus on the project first and then invest my time into that (if it were my daily job, then sure why not now). So rest assured, I will go back to this board and give you more updates on that 😄

Summary

As usual, I hope you gained something from this post and maybe will be motivated to follow along or create something amazing of your own! Warm wintry hugs from me, and see you around!

Leaving you with beautiful Austrian Alp view 😁

In this blog post you will get a glimpse of what all the conferences were about and what are latest market trends and concerns related to today’s geopolitical and economic situation.

Also, I will talk a tiny bit about surfing 😁

Monterey

The first step on our journey was Monterey, CA where we both attended the Linux Foundation Member Summit 2023 which was a first for us. Summit was mostly community and business oriented where major topics such as Open Source Project Office or Open Source community building and management or Open Source project lifecycles were discussed.

My interest was piqued by the talk about Kernel Maintaining by Johnatan Corbet from LWN.net. There, he presented the broad ecosystem that the Linux Kernel is and he pointed out some major challenges that its maintainers are currently facing. Namely there are some boxes in the dark cellar that have been mostly lying dormant and not receiving that much attention as, for example, device drivers or features that majority of users needs. These areas include documentation, build system being old KConfig and Makefile based (looking at you sweet and shiny meson) and some core kernel areas and drivers that are in a dire need of a maintainer (this article is slightly older, so you may refer to the actual talk).

There was a lot of discussion on open Artificial Intelligence models and how to properly and safely develop them so that they do not become evil rogues with intent of murdering all of humanity. Lastly, there was a talk about licensing and I finally understood how many flavors of GPL there exist(hint - 6) and about SPDX (Software Package Data Exchange) - a standard introduced to make licensing code less convolved. Check it out if you still don’t know what GPL or MIT stand for (ok the latter was quite easy).

Hackaday Supercon 2023

After our brief respite in Encinitas, CA, (Tony Hawk lives there 😁) where we surfed and rested a little-bit we hit the road and arrived in beautiful Pasadena, CA. Apart from being greeted by a gigantic traffic, we were also greeted by the most diverse and welcoming crowd of people I ever met in my life. I cannot speak any bad word about this conference and community because the passion for hacking and tinkering was literally seeping from these people.

It was also my first conference with a non-conventional badge, which instead of being a paper or plastic rectangle was a full-blown Vectorscope! My amazement had no boundaries when I learned that during the 3-day conference we could hack and tinker with it and we finally presented it at the closing gala where we almost won in our category (losing only to Pluto which apparently regained its status of a planet 🤪

Even though the badge was initially slightly unstable (both the software and hardware had been made by the folks at Hackaday!), people created AMAZING creations using it, think DOOM on Vectroscope, MPEG-1 video player and others!

Our game was quite simple yet fun to hack around and tinker with in the corridors and workbenches of the conference. We added a parallax background (scrolling behind the main scene), some simple controls and game loop. Oh and we also managed to play some audio 😄 In fact, this reminded me that I need to resurrect my GameBoy game porting and design side project!

RISC-V Summit

Right after the closing gala we hit the road and travelled over 400 miles to Santa Clara, where the next conference would start the next morning. While writing this I just remembered that the time change occurred in the US so we lost an hour of sleeping which might explain why we were so tired after Supercon in the first place.

The RISC-V Summit was much more business-oriented and was a stark contrast to the geeky and casual Supercon. In fact, I felt slightly underdressed without a tuxedo and in my Vans 😁 Nonetheless, the conference was mostly focusing at current challenges that the chip industry is trying to solve and how RISC-V is a good answer to them. What bothers the industry the most is effective architecture for LLMs (Large Language Models, think GPT-3) or other Generative AI models like Stable Diffusion. RISC-V is a good answer to that, and Meta has used a fabric of 2 RISC-V cores comprising one processing-element (one core for processing and another for vector operations) in their MTIA accelerator.

Other topics included governing and growth of the RISC-V foundation that is in a stark contrast to chip architectures like Intel or ARM, where this architecture is created in an Open Source way instead of being proprietary and centrally governed. Imagine a world where it is YOU who can request a new feature to be added/improved in your everyday hardware, well, we might just as well be living it 😄

We also reunited with some of our friends at beagleboard.org who recently released yet another RISC-V based board - BeagleV-Fire!

Surfing

Probably your inner geek has already been satiated by this point so now let’s discuss how’s the surfing in California. Answer? Can’t get any better! 🌊

SoCal surfing

Let’s first clarify how big California really is. Sources do not agree, but its coast is of more-less 1200 miles long (that’s around 2000 km) and we have 3 distinct regions: Southern California spanning from Mexican border to around Santa Maria. Then we have Central (wild) California where we now have more sharks and golfers than surfers but also breathtaking surfing vistas and waves. Lastly, we have Northern California starting at San Francisco up to Oregon. This last stretch of the Coastline is starkly different from its southern counterpart, but this time we had no pleasure surfing there.

The surf culture north of San Diego is pervasive, locals are more-less cultured and crowds are mostly only on weekends. My personal favorite is Encinitas but for concrete spot recommendations you would have to ask me personally 😁

Santa Cruz

Santa Cruz was a reality check for us, as we are not used to surfing in crowds and in such densely populated areas. Although the city itself is beautiful and the vistas are marvelous, you simply cannot surf near the weekends and after regular working hours. That’s a shame because the legendary spot Steamer Lane provides great waves DAILY!. Here is a picture of it at its regular potential 😆

Overall, Santa Cruz is a great place to live, albeit quite expensive one. Locals are friendly and the surfing vibe is magnetic (wishing I did my studies at UCSC).

Summary

This is definitely not the last time we visited California, we still have some unfinished businesses both in the literal meaning of it and with the surfing spots!

Stay tuned for some goodies from Sticky Piston Studios and if you still have not followed us on LinkedIn, then please do!

Link to our talk

Link to Szymon’s talk on Green Software for Embedded Devices

By the end of June, me and Szymon travelled to Prague, Czech Republic for the Embedded Open Source Summit 2023 where we participated in the conference both as speakers and attendees. We also attended Xen Developer Summit and the Yocto Project Dev Day!

Our talk

Our talk titled “Porting an AI Powered Wearable Health Monitor to Zephyr on Open Hardware boasted almost full room by which we were very flattered. Warm thanks to everyone who engaged with us after the talk and hope you took away something useful from this presentation.

In short, we presented a war-story where we worked on adding support for QuickLogic’s EOS S3 SoC running the Zephyr real-time operating system, while deploying a Blood Pressure prediction model that you may already know from this post. The model was deployed using TensorFlow Lite for Microcontrollers which proved to be very easy and accurate.

Here you may see the breadboard prototype as we didn’t have time to remake the original case Szymon envisioned for the previous iteration of this project.

Beagleboard.org

The highlights from this conference include finally meeting Jason Kridner - the founder of beagleboard.org for whom we had the pleasure to create projects during Google Summer of Code 2021 where I did GPGPU computing with BBB and Szymon started porting Zephyr support for Wio Terminal to act as a Greybus host for BeagleConnect Freedom. We also met other folks from the beagleboard.org organization including Kathy Giori and Drew Fustini who conducted a panel regarding future of beagleboard.org foundation.

BTW - fresh RISC-V board dropped from beagleboard.org - BeagleV Ahead based on Alibaba T-Head chip!

Zephyr

We also met some Zephyr people who helped us a lot during development of this project - Chris Friedt or Henrik Brix Andersen as well as Benjamin Cabé. Actually, there are already three PR’s that were the direct result of our participation in this conference and in this project 1,2, 3. Being an Open Source contributor to such an important project is surely satisfying 😎.

We plan to continue extending our support for Zephyr for this board by adding support for more peripherals and trying to decouple the Zephyr subsystem implementation from the Hardware Abstraction Layer from Quicklogic. Also, since I have always been eager to learn PCB design we might create a single PCB for this solution (but that’s a stretch and the day has only 24 hours, eh?).

I now have a beautiful Zephyr kite 🪁 (that I wish I could surf on 😁) in my living/hacking room.

Oh we also had the opportunity to see Josef’s Prusa HQ in Holešovice Praha.

What’s next?

Since we are only getting started, expect us on further conferences around Autumn (Hackaday Supercon and RISC-V Summit 😉 - not yet confirmed), and now we will enjoy some well deserved rest and vacation 😄 (Windsurfing 🏄 time for Szymon and some gym 🏋️ and cycling 🚴 for me). Enjoy the Summer folks!

Also, stay tuned for something from Sticky Piston Studios as even though we will be mostly resting, something nice is en-route 😁

Why go there?

We received tickets to Digital Dragons during HackYeah 2022 on which we also won with our game The Deluge that you are encouraged to try and enjoy! Below you can see us in the moment of triumph:

Who are we?

We are Sticky Piston Studios and we are a team of 3 passionate young people who have been working in the industry for a while already. Having attended more than 15 gamejams in total we feel that it is time to transition to game publishing so stay tuned for some of our new productions 😁!

Below you may see the holy trinity: COO, CEO and CTO in that order. Szymon Duchniewicz (Willmish), Me (Celeborth) and Mateusz Szymoński (Hist0r)

Quick update from Digital Dragons 2023

Digital dragons were filled to the brim with good laughs, people on crutches, talks about building a healthy company and how to best capitalize on the market during tough times. We attended different tracks and while I focused mostly on the teambuilding and mobile tracks, our CTO attended more technical talks and some post-mortems, such as House Flipper or Castle Crumble. Szymon, on the other hand, participated in talks about market development and brand construction. During the Indie Expo we met with several publishers with whom we plan our future releases and we also had pleasure to play innovative Indie games that were nominated to Indie Awards.

Lastly, here is a picture of us with our badges and the shiny Digital Dragons logo 😁

Next stop, Embedded Open Source Summit 2023 in Prague with my and Szymon’s talk on embedded health blood-pressure measuring device based on Zephyr OS and open hardware. Until next time!

Introduction

Hi! My name is Jakub Duchniewicz and welcome to Brother Industry Band of Power.

In this post series we plan to describe our progress with the BIBoP project and shed some light on various issues we had to tackle along the way.

Be sure to check the [official repository] of this project!

Here is a small sneak-peek of how the device will look like once assembled:

Why?

Since I haven’t posted in a while, a small update on the reasons behind this blog post series and why ultimately my last project ([Envidrawer]) was not documented here.

The reason is quite obvious: I am already documenting the project on the [official contest website] and I am bringing here the best nitbits from there.

Also, this project touches on so many domains and digs into several obscure areas (SAMD21 MCU programming, BP estimation from PPG, AWS Lambda over TLS/SSL…), it would be a shame not to document them!

Envidrawer

As for the [Envidrawer] project, I finished it in quite busy time of my life (changing countries, being sick) so I did not move it to this website. Maybe one day…

Nevertheless, be sure to check [some] [cool] [concepts] we came up with during development of the Envidrawer project.

Project overview

Similarly as with Envidrawer, I teamed up with Michał and Szymon to create a solution to an important problem which the humanity is currently facing or will be facing in near future. Unfortunately, the project is during their exam time on universities and it is mostly my responsibility 😄 (not that I am complaining!).

Our project is a wearable band which that will measure oxygen saturation, pulse, and body temperature of the potential patient. It will automatically connect to Wi-Fi and on reaching certain thresholds call in the emergency team for hospitalization. Being a cheap and easy to manufacture device, it could be distributed to the patients from high-risk groups which upon falling sick could make use of it. The true beauty lies in the fact that it could stay after the pandemic, once again moving healthcare closer to its recipients.

In other words, we want to provide people with affordable, easy-to-use smart-bands which will measure their vital signals and report any abnormalities to medical staff and to the users themselves.

In the picture below you can see the high-level overview of the project (the real photos will come in as soon as the 3D printed case is ready!).

Every project has to be powered by [Arduino Nano 33 IoT], thus I decided to make use of some ready made Arduino libraries, such as WiFiNINA or BearSSL which are necessary for secure wireless communication. At the moment of writing this post I am waist-deep in the SAMD21 datasheet and trying to enable external interrupts in low power modes, since Arduino development framework does not provide all conceivable solutions.

Because of shortages in the developer team, the project may not achieve all of its goals on-time, but it definitely is worth spending some time on after the contest ends. After all, having your own smart-band is quite exciting 😁.

I - First steps

Setting up the development environment is one of the first steps every developer has to undertake, so that is what I also had to do. The SAMD boards require their own Board Support Package which is best downloaded using the official Arduino IDE.

First of all we need to setup our build and editing environment.

Enter Makefiles.

DISCLAIMER - This is fairly technical paragraph so feel free to scroll down for the Machine Learning related content

Why use Makefiles, when we have Arduino IDE?

We have Arduino IDE, right. But apart from providing users with handy building scripts and libraries + toolchains management, it failed to meet the IDE standards in term of code editing. Since I am heavy vim user, I wanted to write my code in familiar environment and with handy shortcuts and handy tools.

This is just one reason behind it, the other being code extensibility, compiler choice and language version freedom. Assuming you want to have your own libraries as a part of a bigger project, Arduino IDE does not present you with an easy way to do it - with a custom Makefile you can structure your project as desired. This unfortunately is by no means out of the box solution, but I was able to polish some rough edges and possibly make it easier to fall in step for whoever reads this post.

Cloning and setting up

Assuming you have read this far and are willing to create your own Makefile-based Arduino project, you need to create the folder structure or simply fork/clone this project. You have a bunch of unnecessary files to delete, such as exemplary Makefiles and mock libraries. Instead what you need to do is follow the INSTALL.md which guides you by hand mostly.

Once you have the Makefile in the project directory (src/ProjectName/Makefile), you have to fill it with some necessary variables which are used during the process of building the binary and during the flashing and debugging the code. Exemplary Makefile can be found below:

# For now makefile will be in this file, we will change it later
PROJECT_DIR       = $(shell dirname $(shell dirname $(shell pwd)))

ARDMK_DIR         = $(PROJECT_DIR)/Arduino-Makefile
ARDUINO_PACKAGE_DIR = $(HOME)/.arduino15/packages
ARDUINO_DIR       = /usr/share/arduino
USER_LIB_PATH     :=  $(realpath $(PROJECT_DIR)/lib)
BOARD_TAG         = nano_33_iot
MONITOR_PORT      = /dev/ttyACM*
MONITOR_BAUDRATE  = 115200
AVR_TOOLS_DIR     = /usr
AVRDUDE           = /usr/bin/avrdude
CFLAGS_STD        = -std=gnu11
CXXFLAGS_STD      = -std=gnu++17
CXXFLAGS         += -pedantic -Wall -Wextra
LDFLAGS          += -fdiagnostics-color

### OBJDIR
### Don't touch this!
### This is were you put the binaries you just compile using 'make'
CURRENT_DIR       = $(shell basename $(CURDIR))
OBJDIR            = $(PROJECT_DIR)/build/$(CURRENT_DIR)/$(BOARD_TAG)

include $(ARDMK_DIR)/Sam.mk

As you can see, not much is needed to make all of this work (most of the variables are pre-filled and just have to be tweaked by you - hopefully you won’t need to know the Makefile content by heart like I did while repairing it ). Remember to point the USER_LIB_PATH to the ProjectRoot/lib path and to symlink/copy the libraries you plan to use to this directory.

The GitHub repository README mentions that you have to set the BOARD_TAG and BOARD_SUB if you have boards which have these two parameters, but for the Arduino Nano 33 IoT we just need to set the BOARD_TAG to nano_33_iot

Lastly, because the original Makefiles are for the regular AVR Arduinos, you need to change the last line which includes the main Makefile from include $(ARDMK_DIR)/Arduino.mk to include $(ARDMK_DIR)/Sam.mk

Polishing the rough edges

If you tried compiling right now, you would be met by some errors about missing LTO plugins during running the binary linking step. The reason for this was fixed upstream but is not ported to the Makefile you have in the project, so I suggest changing the git submodule to the upstream. You can do it with following commands:

git submodule deinit Arduino-Makefile
git rm Arduino-Makefile
# now add the gitmodule
git submodule add https://github.com/sudar/Arduino-Makefile Arduino-Makefile
git submodule update --init --recursive

The --recursive flag was needed in case the submodule includes other submodules.

Having the upstream Makefile, we have some problems fixed. Unfortunately this does not fix the libraries you create by yourself so I just add the LTO plugin by myself by adding the following line to the Arduino.mk:

# SPECIAL WORKAROUND FOR LIBRARIES
LTO_PLUGIN_DIR = $(HOME)/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/lib/gcc/arm-none-eabi/7.2.1/liblto_plugin.so

Now we should be able to add our files to the lib directory and compile successfully.

Building and flashing

In order to compile the program you simply run make in the ProjectName/src/AnotherName directory and wait for the magic to happen. If everything went fine it should have built the binary and left the build artifacts in the build folder (check which one this is!).

In order to flash the binary on the target just run make upload and be sure to put the proper port in the MONITOR_PORT variable. It is quite picky about the port being used so be sure to close all other serial connections on this port even if they are already dead.

You can also monitor and debug the code using a serial connection and GDB. You know the magic spells by now!

• The monitor command will simply output your Serial messages (or other printing method you have set on the target) using the screen program so be sure to install it!

• The debug command will run GDB (if you don’t know how it works, be sure to learn some basics and refer to this [cheatsheet]).

The Arduino Nano 33 IoT exposes SWD debug pads, but does not provide a built-in debugger - you need to provide it on your own ([like this one])

II - Machine Learning

The second section focuses on going step-by-step over a [Jupyter notebook] where a ML model for Blood Pressure (both systolic and diastolic) estimation is created and trained. First, the data is obtained and cleaned, followed by features extraction and evaluation, to finally end with model construction, training and testing.

The resultant models are serialized and saved for deployment.

If you want more in depth information - the [original post] covers that.

Data obtaining and cleaning

The data which will be the basis of the model was obtained from a publicly available data from [UCI Machine Learning Repository] which contained PPG, ECG and invasive BP measurements. The ground truth for our estimation were extracted SBP and DBP (systolic and diastolic BP), that were extracted from the ABP (Arterial Blood Pressure). The sampling frequency used was 125 Hz and this is the frequency with which we will be collecting our data by the BIBoP. The signals were of course not ideal and they had to be cleaned beforehand (this was done by the creators of the dataset).

After downloading the dataset, extracting it and loading to memory, the process of cleaning and data reshaping can commence!

This is visible in this chunk of code:

# download and load the data

chunk_size = 4096
url = "https://archive.ics.uci.edu/ml/machine-learning-databases/00340/data.zip"
req = requests.get(url, stream = True)
total_size = int(req.headers['content-length'])

# Faster downloads in chunks
with open("data.zip", "wb") as file:
    for data in tqdm(iterable=req.iter_content(chunk_size=chunk_size), total = total_size/chunk_size, unit='KB'):
        file.write(data)

# extract the data
if not os.path.exists(DATA_FOLDER):
    os.mkdir(DATA_FOLDER)

import zipfile
with zipfile.ZipFile("data.zip", "r") as ref:
    ref.extractall(DATA_FOLDER)

test_samples = mat73.loadmat(f"{DATA_FOLDER}/Part_1.mat")['Part_1']

We want to divide the data into 125 samples segments (or longer depending on the segment length - more on that later!):

# the sampling speed - 125 Hz
FS = 125
# SAMPLE_SIZE = 125 # the frequency in Hz (1 second samples)
# segments of 10 seconds
SAMPLE_SIZE = 125 # or longer!
NUM_PERIODS = SAMPLE_SIZE // FS
# partition the data into equal length pgg segments
ppg = []
for i in range(len(test_samples)):
    l = test_samples[i][0].size
    for j in range(l // SAMPLE_SIZE):
        ppg.append(test_samples[i][0][j * SAMPLE_SIZE : (j + 1) * SAMPLE_SIZE])

Since machine learning (regression) in a BIG approximation is “Given x calculate y, for y = f(x)”, we are preparing our x’s (independent variables) - our y’s (dependent variables) will be predicted and tested against ground truth -> blood pressure already in the dataset.

(regression is in smart words prediction)

We now have to extract both systolic and diastolic blood pressure, and we do it by obtaining minimums and maximums of consecutive periods:

sbp = []
dbp = []
bp = []

for i in range(len(test_samples)):
    l = test_samples[i][1].size
    for j in range(l // SAMPLE_SIZE):
        temp_bp = test_samples[i][1][j * SAMPLE_SIZE : (j + 1) * SAMPLE_SIZE]
        tmp_sbp = []
        tmp_dbp = []
        for k in range(NUM_PERIODS):
          tmp_bp_small = temp_bp[k * FS : (k + 1) * FS]
          tmp_sbp.append(np.max(tmp_bp_small))
          tmp_dbp.append(np.min(tmp_bp_small))
        # SBP will be maximum and DBP will be minimum of 1 such sampling period (or averaged if more periods)
        bp.append(temp_bp)
        sbp.append(np.mean(tmp_sbp))
        dbp.append(np.mean(tmp_dbp))

A comparison of the ABP and PPG signal is visible below:

Inspecting the SBP and DBP shows they are good enough:

The PPG signal is surely non-ideal and that is why we will need to perform some further cleaning:

For now we can create a baseline model and see how it fares. We did not extract any features yet so we do the prediction based only on the PPG signal:

# since we want to predict both SBP and DBP we will pack them for each sample
target_bp = []
for i in range(len(ppg)):
    target_bp.append((sbp[i], dbp[i]))

x_train, x_test, y_train, y_test = train_test_split(ppg, target_bp, test_size=0.3)

model = LinearRegression()
model.fit(x_train, y_train)
y_pred = model.predict(x_test)
mean_absolute_error(y_test, y_pred)

The achieved Mean Absolute Error was 13.8 which gives us some room for improvement. We need to extract features from the signal which may help us identify potential important characteristics for models.

Temporal and Spectral features extraction

Temporal

Since classic Machine Learning usually benefits from hand-picking some features from the input data, some features are extracted from the signal periods and assessed for validity. Features were chosen based on most popular ones in the recent scientific papers.

These include:

• Cycle duration time
• Time from cycle start to systolic peak
• Time from systolic peak to cycle end
• Time from systolic peak to dicrotic notch
• Time from dicrotic notch to end
• Ratio between systolic and diastolic amplitude

These features are explained in more detail in the graph underneath:

The code for extraction of both features is available in the [Jupyter notebook].

Spectral

Additionally, we can extract spectral features, which could help the models:

• Three largest magnitudes (both values and frequencies)
• Normalized energy
• Entropy
• Histogram - Binned distribution from 0 to 60 Hz (10 bins)
• Skewness
• Kurtosis

These features require some periodicity (remember our old friend Joseph Fourier?), so they have to be taken from ~10 periods of PPG.

Exploratory Data Analysis

After we obtained the features, we assemble the dataframes and clean them slightly.

rows = []

for i in range(len(ppg)):
    cycle_len, t_start_sys, t_sys_end, t_sys_dicr, t_dicr_end, ratio = extract_features_long_seg(ppg[i], ppg_ii[i])
    freq_1, mag_1, freq_2, mag_2, freq_3, mag_3, energy, entro, bins, skewness, kurt = extract_spectral_features(ppg[i])
    rows.append((cycle_len, t_start_sys, t_sys_end, t_sys_dicr, t_dicr_end, ratio,
                freq_1, mag_1, freq_2, mag_2, freq_3, mag_3, energy, entro, skewness, kurt, *bins))

rows = np.array(rows)

# make a df from the data to clean it
bins = [f"bin_{i}" for i in range(10)]
col = ["cycle_len", "t_start_sys", "t_sys_end", "t_sys_dicr", "t_dicr_end", "ratio",
       "freq_1", "mag_1", "freq_2", "mag_2", "freq_3", "mag_3", "energy", "entropy", "skewness", "kurtosis", *bins]
df = pd.DataFrame(rows, columns=col)

# drop rows with wrong values
idxs = df.loc[df['t_sys_end'] < 0.].index
inf_idxs = df.loc[df.values >= np.finfo(np.float64).max].index
indices = idxs.append(inf_idxs)
df = df.drop(indices)

# fill all NaN's
df.fillna(0, inplace=True)


# clean also the target variables
col = ["SBP" , "DBP"]
df_target = pd.DataFrame(target_bp, columns=col)
df_target= df_target.drop(indices)

Let’s take a quick look at the df.describe() which will tell us how the dataframe fares:

Briefly exploring the data, it can be seen there are some outliers which should be eliminated in order to make the models’ predictions better. For this we will use IQR elimination - removing all the values which lie outside of the Q1 and Q3 percentile quarters in the normal distribution of the data:

# only some columns are striking, remove only rows where outliers are present in these columns
sus = ["ratio", "mag_1", "mag_2", "mag_3", "energy", "entropy", "skewness", "kurtosis"]
to_remove = set()

indices = set()
for x in sus:
  q25, q75 = np.percentile(df.loc[:,x], [25, 75])
  intra = q75 - q25

  max = q75 + intra * 1.5
  min = q25 - intra * 1.5

  idxs_1 = df.loc[df[x] < min, x].index
  idxs_2 = df.loc[df[x] > max, x].index
  to_remove = to_remove.union(idxs_1).union(idxs_2)

df.drop(to_remove, inplace=True)
df_target.drop(to_remove, inplace=True)

Now we can finally plot our data and see if there are any striking relations we should be aware of:

# correlation plots
plt.rcParams['figure.dpi'] = 100
plt.rcParams["figure.figsize"] = (20,15)
scatter_var = ["cycle_len", "t_start_sys", "t_sys_end", "t_sys_dicr", "t_dicr_end", "ratio",
               "freq_1", "mag_1", "freq_2", "mag_2", "freq_3", "mag_3", "energy", "entropy", "skewness", "kurtosis"]
correlation_matrix = df[scatter_var].corr()
sns.heatmap(correlation_matrix, annot=True)

It can be seen that time features are quite correlated, which is expected. Also the energy is somewhat correlated with the time features. There are strong anticorrelations in frequency domain features - frequency/magnitude associations.

Apart from confirmation, this plot did not give as much insight as it can in other cases, so let’s move on!

Machine Learning and Results

Having the data prepared, we can now proceed with the training of various models to assess which one is the best. The most promising models in the literature are a Random Forest and Linear Regression so we are using them here as well. I won’t describe these two models here, because there is plenty of good sources on [RF] and [LR].

The data is further split using [K-Fold cross-validation], in order to train the model for a given number of epochs. Then it is finally used to predict on the intact test set.

folds = KFold(n_splits=10, shuffle=False)
# resplit the data after processing
x_train, x_test, y_train, y_test = train_test_split(df, df_target, test_size=0.3)

Finally, the model training can commence:

linear = LinearRegression()
errors_sbp = []
errors_dbp = []

for i, (train_idx, val_idx) in enumerate(folds.split(x_train, y_train)):
    train_data, train_target = x_train.iloc[train_idx], y_train.iloc[train_idx]
    val_data, val_target = x_train.iloc[val_idx], y_train.iloc[val_idx]

    linear.fit(train_data, train_target)
    predictions = linear.predict(val_data)
    error_sbp = mean_absolute_error(predictions[:,0], val_target["SBP"].values)
    error_dbp = mean_absolute_error(predictions[:,1], val_target["DBP"].values)

    print(f"Train fold {i} MAE SBP: {error_sbp} MAE DBP: {error_dbp}")
    errors_sbp.append(error_sbp)
    errors_dbp.append(error_dbp)

print(f"Average MAE SBP: {np.mean(errors_sbp)} MAE DBP: {np.mean(errors_dbp)}")

The results are visible in the picture below:

It is visible that the rightmost model is the most accurate - Random Forest with 10s segment lengths and 100 estimators. The results are far from ideal but they are not much worse than recent State of The Art (we have still much to improve in this area).

The most important problem we are now facing is making the data captured from the PPG with the Arduino fit our model (be normalized and rescaled). Also the PPG reading often wanders and has varying amplitude between readings, so we will have to rescale the data dynamically (averaging a period of 10 seconds or more).

Summary

This was the first of posts describing my endeavours with BIBoP, with several more to come. The next one will be focusing on AWS and setting up a Lambda which will make use of our model. I hope this post was informative enough and you learned something valuable.

As always, if you found anything unclear or want to provide feedback, reach out to me, be it below this post or personally 😄.

Also, if you like what I’m doing and you would like to see more of it - consider buying me a [coffee] ☕ [coffee]: https://www.buymeacoffee.com/jduchniewicz [Envidrawer]: https://www.element14.com/community/community/design-challenges/1-meter-of-pi/blog/2021/01/06/envidrawer-final-post [official repository]: https://github.com/JDuchniewicz/BIBoP [official contest website]: https://www.element14.com/community/community/design-challenges/design-for-a-cause-2021 [some]: https://www.element14.com/community/community/design-challenges/1-meter-of-pi/blog/2021/01/02/envidrawer-blog-7-ride-the-lightning [cool]: https://www.element14.com/community/community/design-challenges/1-meter-of-pi/blog/2021/01/03/envidrawer-4-3d-modelling-printing-and-further-assembly [concepts]: https://www.element14.com/community/community/design-challenges/1-meter-of-pi/blog/2020/11/02/envidrawer-2-materials-and-casing-assembly [Arduino Nano 33 IoT]: https://store.arduino.cc/arduino-nano-33-iot [cheatsheet]: https://gabriellesc.github.io/teaching/resources/GDB-cheat-sheet.pdf [like this one]: https://1bitsquared.de/products/black-magic-probe [Jupyter notebook]: https://github.com/JDuchniewicz/BloodPressurePPG [original post]: https://www.element14.com/community/community/design-challenges/design-for-a-cause-2021/blog/2021/05/13/bibop-3-blood-pressure-inference-machine-learning [UCI Machine Learning Repository]: https://archive.ics.uci.edu/ml/datasets/Cuff-Less+Blood+Pressure+Estimation [RF]: https://towardsai.net/p/machine-learning/why-choose-random-forest-and-not-decision-trees [LR]: https://www.cs.toronto.edu/~frossard/post/linear_regression/ [K-Fold cross-validation]: https://machinelearningmastery.com/k-fold-cross-validation/

Original Chapter 1

Update

Thank you wholeheartedly for the support and the comments on this post. Some mistakes were fixed and some things are now better clarified. Also thanks to soruh for the optimization PR to the repository (already merged). The relevant benchmarks are mentioned there and the code is parallelized with rayon. If you are interested in the discussion take a look here. The main branch contains some improvements over the original code in this post, so check it out for some cool things such as Rust macros or the aforementioned parallelization.

Every programmer wants to feel loved (yes I am looking at you!), be it by others or yourself. Usually you really love yourself when you accomplish something you are proud of. That is why from time to time programmers tend to learn languages (be it programming or spoken ones - unless you can talk to your fridge in assembly of course) or challenge themselves and write tough and unintelligible pieces of code which do something amazing. If you are like me and were always amazed by how the computer can render something resembling real life instead of just 2D graphics, you came to the right place!

Cutting the slack, I will reimplement the amazing tutorial on Ray Tracing in One Weekend in the Rust programming language. This post is aimed at people who are interested in the subject of rendering and want to try Rust, or are simply curious about how things are done in this language. I will not go through all content, but only focus on parts which are starkly different from the original implementation. The code for this project is available on this GitHub repo.

We aim to obtain such a render at the end of this tutorial.

Although you can just read through the whole thing and see how things are done differently in Rust compared to C++ or C, I recommend reading through the original tutorial and implementing the code yourself! Nevertheless, be prepared to learn a great deal about why and how Rust does some things the other way (the modern one?). Of course, do read The Rust Programming Language, in which you can find a comprehensive intro to Rust, or if you prefer less reading and more code look no further than Rust by example.

This is yet another blog post in the style of RIIR but with educational aims (don’t hang me for it, please). The target audience should have some knowledge of programming (especially in C or C++). Assuming you are the target audience, Rust knowledge is not required but as stated before, do read up the official tutorials - this one is for those who want to have a sense of accomplishment and a pretty solid infant renderer.

Views expressed here are my own only… you know the rest.

I promise, this is the last paragraph that keeps you from writing actual code. I will link relevant paragraphs from the original tutorial so you can see where the code differs so much it was worth me rambling on it.

Outputting an image

Original Chapter 2

Time to get our hands dirty and code something! Rendering something is most fun if we can actually see the result, so we need to create a function which will save our rendered image into a PPM image format (probably due to its simplicity).

Each time there is code to compare, I will paste both the C++ code and its Rust counterpart so you can spot the differences. I will only attach some of the images from the original post and instead provide direct links to them.

The C++ code:

#include <iostream>

int main() {

    // Image

    const int image_width = 256;
    const int image_height = 256;

    // Render

    std::cout << "P3\n" << image_width << ' ' << image_height << "\n255\n";

    for (int j = image_height-1; j >= 0; --j) {
        for (int i = 0; i < image_width; ++i) {
            auto r = double(i) / (image_width-1);
            auto g = double(j) / (image_height-1);
            auto b = 0.25;

            int ir = static_cast<int>(255.999 * r);
            int ig = static_cast<int>(255.999 * g);
            int ib = static_cast<int>(255.999 * b);

            std::cout << ir << ' ' << ig << ' ' << ib << '\n';
        }
    }
}

Rust:

fn main() {
    let image_width = 256;
    let image_height = 256;

    println!("P3\n{} {}\n255", image_height, image_width);

    for j in (0..image_height).rev() {
        for i in 0..image_width {
            let r = i as f64 / (image_width - 1) as f64;
            let g = j as f64 / (image_height - 1) as f64;
            let b = 0.25;

            let ir = (255.999 * r) as i32;
            let ig = (255.999 * g) as i32;
            let ib = (255.999 * b) as i32;

            println!("{} {} {}", ir, ig, ib);
        }
    }
}

This should result in an image looking like this - you can view these images with most viewers as this is quite common image format for ASCII encoded images.

The first stark difference is the for loop - Rust uses syntax similar to Python and supports looping though iterable objects thanks to the Iterator trait (we will get to traits soon enough, for now it is a kind of interface). Looping forwards is easy, as you just specify the range of iteration like this:

for i in 1..10 {
    println!("looping: {}", i);
}

You can also loop in steps similar to the C++ i += 2 by using this syntax: for i in (1..10).step_by(2), but how do you loop backward? I probably spoiled the fun as the answer is visible above, a range already implements the Iterator trait, and for other types you create an iterator from them by yet another trait called Into which is a reciprocal of From - in short it allows the programmer to specify legal conversions between types in Rust. So, we take a range of values, call the rev() function on it (provided by the Iterator trait) and voila, we got our reverse loop: for i in (1..10).rev().

Barring from some syntax differences, the program is quite similar to the original version, we use as instead of C-style (unsafe) casts and static_cast<T>s. This cast will of course detect any mismatch at compile time.

Building the code

You will of course need a way to build the code and run it, and this is a good opportunity to introduce you to the first key selling point of Rust: Cargo. This is both a build system and a package manage (think like Python’s pip but with Makefiles on top of it). Forget about annoying CMake or writing Makefiles by hand - finally we have something with an easy-to-read syntax: TOML. Cargo, of course allows for creating targets and managing compiler and linker flags but removes all the nitty-gritty details of including files and setting up export options.

In order to create a new project you just need to run cargo new coolprojectname and if you want to build it and run you may run cargo run or cargo build if you want to build it only (and have a brief lesson in Rust compiler messages). For the release builds, just pass the flag --release to the compiler (and be sure to run this project with this command, otherwise be prepared for long trips to kitchen to kill the time while the scene renders).

Vec3 helper class

Original Chapter 3

Because we will be using some heavy 3D maths, we will need a helper class capable of performing some operations automatically instead of writing them by hand. This is where our code starts to diverge (rather strongly I would say). The original class relies on standard C++ features such as constructor and operator overloading with a sprinkle of friend functions on top, while Rust has no notion of overloading and instead achieves these things with the power of traits and generics which are also present in C++ albeit wear a cover of templates. I will not go into much detail on the topic of run-time vs compile-time polymorphism, but if you are eager for a read then I leave one.

C++:

#ifndef VEC3_H
#define VEC3_H

#include <cmath>
#include <iostream>

using std::sqrt;

class vec3 {
    public:
        vec3() : e{0,0,0} {}
        vec3(double e0, double e1, double e2) : e{e0, e1, e2} {}

        double x() const { return e[0]; }
        double y() const { return e[1]; }
        double z() const { return e[2]; }

        vec3 operator-() const { return vec3(-e[0], -e[1], -e[2]); }
        double operator[](int i) const { return e[i]; }
        double& operator[](int i) { return e[i]; }

        vec3& operator+=(const vec3 &v) {
            e[0] += v.e[0];
            e[1] += v.e[1];
            e[2] += v.e[2];
            return *this;
        }

        vec3& operator*=(const double t) {
            e[0] *= t;
            e[1] *= t;
            e[2] *= t;
            return *this;
        }

        vec3& operator/=(const double t) {
            return *this *= 1/t;
        }

        double length() const {
            return sqrt(length_squared());
        }

        double length_squared() const {
            return e[0]*e[0] + e[1]*e[1] + e[2]*e[2];
        }

    public:
        double e[3];
};

// Type aliases for vec3
using point3 = vec3;   // 3D point
using color = vec3;    // RGB color

#endif

Rust

use std::fmt;
use std::ops;

#[derive(Clone, Copy, Debug, Default)]
pub struct Vec3 {
    pub x: f64,
    pub y: f64,
    pub z: f64,
}

pub use Vec3 as Point3;
pub use Vec3 as Color;

impl Vec3 {
    pub fn new(x: f64, y: f64, z: f64) -> Vec3 {
        Vec3 { x, y, z }
    }

    pub fn length(&self) -> f64 {
        self.length_squared().sqrt()
    }

    pub fn length_squared(&self) -> f64 {
        self.x * self.x + self.y * self.y + self.z * self.z
    }
}

impl ops::AddAssign<&Vec3> for Vec3 {
    fn add_assign(&mut self, rhs: &Vec3) {
        self.x += rhs.x;
        self.y += rhs.y;
        self.z += rhs.z;
    }
}

impl ops::MulAssign<f64> for Vec3 {
    fn mul_assign(&mut self, rhs: f64) {
        self.x *= rhs;
        self.y *= rhs;
        self.z *= rhs;
    }
}

impl ops::DivAssign<f64> for Vec3 {
    fn div_assign(&mut self, rhs: f64) {
        self.x /= rhs;
        self.y /= rhs;
        self.z /= rhs;
    }
}

First important thing to note is that Rust provides no constructor overloading and achieves similar goals with the Builder Pattern. This pattern is quite popular and has been with us for quite some time now and it neatly fits in the assumed immutability philosophy of Rust. So instead of providing an overloaded constructor for every type, you call the builder and chain functions like this:

let vector: Vec3 = Vec3::new(1.0, 2.0, 3.0).frobnicate().build();

Quite simple, isn’t it?

And it removes much of the noise related to the class (rule of the 5? anyone?).

However, we will go even simpler route an simply provide various constructor functions. Using builder here would be a pretty big overkill. Rust goes even further so that we don’t need to provide a default constructor when we derive a Default trait. It manages all the initialization for us and does this properly.

You may have noticed the self and mut self arguments to the functions of this class (yes I know, it is a struct) - these are the indicators that this is a method compared to ordinary associated function (you may know them as static methods from C++).

Probably quite important design decision worth mentioning now is that I am storing the x, y, z components separately instead of a static array of this form: [1.0, 2.0, 3.0];. You could do both and just benchmark it later, but remember that you cannot modify elements of this array without a &mut, you also cannot resize this array (treat it as a statically allocated C array with normal Rust ownership rules). If you need resizable arrays on the spot then look no further than a vector or Vec in Rust terms. There are some situation in which you need more than one mutable reference to an object. In these circumstances, wrap them in Cell or bring upon thyself wrath of the Rust gods for using unsafe code where it can be avoided. It is often just a matter of preference, and what is more optimal for you!

This brings us to the most glaring difference - operators or the lack thereof. Rust handles them via traits and requires to provide the impl block if one wishes to use them with custom types. Thus, we have types such as Add, AddAssign and others, where all we as implementors have to do is provide the similarly named function implementation.

The next step is quite similar so I am not including the code for it here - we have to implement the friend functions for adding two Vec3s and some utility functions. Implementing the std{io,err} printing is worth looking at though, so here it is: C++:

inline std::ostream& operator<<(std::ostream &out, const vec3 &v) {
    return out << v.e[0] << ' ' << v.e[1] << ' ' << v.e[2];
}

Rust:

impl fmt::Display for Vec3 {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "{} {} {}", self.x, self.y, self.z)
    }
}

While in C++ you define the operator<< for standard stream operations, in Rust you need to only #[derive(Debug)] in order to have a debug printing of your custom struct. You may be now wondering what is the difference between Debug and Display and the question is quite simple. Debugging printing uses some heuristics to format our struct and print it in a predefined form with {:?} or {:#?} for a pretty-printed form, whereas using the Display trait allows the struct implementor to specify a custom form of printing. Because we have a need for a custom printing format we go ahead and implement it. Profit?

Structuring our project

This is a slight detour from our project development, although an important one. In C and C++ the custom is to have one header and one source file per class (or functionality), this is encouraged by the notion of Translation Unit - compiler-friendly name for a single source file and all necessary stuff from headers for this file. In Rust, however, there is no customary split between headers and source files (in fact all crate is a single TU), so how on earth the compiler understands how to resolve the symbols and what are the dependencies between them?

The answer is: namespaces. In Rust, we write all code in source files only and can write either a binary application or a library (in our case it is a binary). Structuring the code properly is a challenge but boils down to either:

• creating a namespace inside the same source file
• splitting the source file into modules
• moving our code to a library crate

These steps are illustrated somewhat accurately in this blog post, and serve as a complement to the official docs.

Hence, in order to see our Vec3 struct in our main.rs we need to first inform the compiler we are using another module - mod vec3; and then import the necessary types with use vec3::Vec3;. These types need to have a pub keyword next to them for the import to work. I will keep doing that for each new file that is created in the original tutorial, and if one is not necessary, it will be boldly stated here. In fact, renderer could be a library and the binary would only be using the functions from it to tell it what it wants to get rendered.

Who owns who?

Original Chapter 5

So far, so good. Assuming you went and read the original tutorial and implemented the code you were probably faced with an error similar to this:

error[E0382]: borrow of moved value: `ray`
    --> src/main.rs:23:50
       |
    19 | fn ray_color(ray: Ray) -> Color {
       |              --- move occurs because `ray` has type `Ray`, which does not implement the `Copy` trait
    20 |     if hit_sphere(&Point3::with_values(0.0, 0.0, -1.0), 0.5, ray) {
       |                                                              --- value moved here
      ...
    23 |     let unit_direction: Vec3 = Vec3::unit_vector(ray.direction);
       |                                                  ^^^^^^^^^^^^^^ value used here after move

What on earth is going on here? Remember the time we added a #[derive(Copy)] statement to the Vec3 class? Our Ray class requires such a statement to inform the compiler it is Copyable. But wait, this now has two Vec3s and they each have 3 doubles(f64) and this starts to amount to a significant overhead when passing on the stack - 2 * 3 * 8 = 48 bytes!

Of course we may pretend such puny numbers do not bother us, but if we wanted to implement this renderer on anything slightly less powerful than our PC, we should strongly consider changing this approach. I guess I need not explain this further to a C++ dev *shrugs*.

What we do instead is pass the value by a reference or in Rust terms borrow it. Borrows come in two flavors: immutable (the default ones, also called shared) - & and mutable - mut&. There are two rules which cannot be broken

• You can have either one mutable reference or any number of immutable references
• References must always be valid

This is all, no more rules? Yes, this is the cornerstone of Rust’s ownership model and it makes much easier to understand how it all fits together. Be sure to read up on this chapter of the Rust Book in order to be on the same page here.

So, now we are borrowing things instead of copying them like this:

// function
fn hit_sphere(center: &Point3, radius: f64, ray: &Ray) -> bool {…}

// call site
if hit_sphere(&Point3::with_values(0.0, 0.0, -1.0), 0.5, &ray) {…}

As you can see we are passing in an immutable reference borrowing the Ray for the time of the hit_sphere call. Keep in mind that it is in stark contrast to C++ where passing an object without a & or const& tag did a copy of the object and in Rust we have either a copy (if it satisfies the Copy trait) or a move, similar to C++ std::move.

Traits, Box and Rc

Original Chapter 6

So far, we were using only structs and things associated with them to achieve our goals. Now, it is a good time to learn about traits and trait objects. trait is like an interface in C++ (okay, you got me, there is no such thing in C++, there are pure virtual classes), allowing for dynamic dispatch of the function call. They in fact allow for much more, but this time we are considering them only as trait objects. This means, that their size is evaluated at run-time rather than at compile-time. For this we need to store them on the heap rather than on the stack.

In order to be stored on the heap they have to be wrapped in a special Rust’s built-in - Box. I won’t dig into details on this type, except mentioning it allocates the object it boxes on the heap (or whatever the custom allocator you provide for it #77187. Because of that we are conforming to Rust’s rules on types with a known size and can happily store such trait objects like this: Box<dyn Hittable>. The dyn keyword the key here - it is the marker telling this is not a regular trait. [Polymorphism in Rust] elaborates on this topic and provides further references.

Thus, being familiar with this concept you try to implement the HittableList class in Rust and you encounter std::vector and std::shared_ptr. While you already know about Vec, you need a replacement for std::shared_ptr - Rc. This type is a Reference Counted pointer, so behaves similarly to its C++ sibling. It is not thread-safe however, so remember to use its cross-thread counterpart - Arc. Both of them also, obviously store their contents on the stack - otherwise would be quite difficult to be shared between threads or different parts of the project.

Great, knowing all this, you finally translate the code (solve several problems with references and ownership on the way) and meet this brow-raising error:

error[E0382]: use of moved value: `temp_rec`
  --> src/hittable_list.rs:42:34
   |
35 |         let mut temp_rec: HitRecord;
   |             ------------ move occurs because `temp_rec` has type `HitRecord`, which does not implement the `Copy` trait
...
42 |                 closest_so_far = temp_rec.t;
   |                                  ^^^^^^^^^^ value used here after move
43 |                 *rec = temp_rec;
   |                        -------- value moved here, in previous iteration of loop

The code for this part looks like this in C++:

bool hittable_list::hit(const ray& r, double t_min, double t_max, hit_record& rec) const {
    hit_record temp_rec;
    bool hit_anything = false;
    auto closest_so_far = t_max;

    for (const auto& object : objects) {
        if (object->hit(r, t_min, closest_so_far, temp_rec)) {
            hit_anything = true;
            closest_so_far = temp_rec.t;
            rec = temp_rec;
        }
    }

    return hit_anything;
}

Rust attempt:

impl Hittable for HittableList {
    fn hit(&self, ray: &Ray, t_min: f64, t_max: f64, rec: &mut HitRecord) -> bool {
        let mut temp_rec: HitRecord;
        let mut hit_anything: bool = false;
        let mut closest_so_far = t_max;

        for object in &self.objects {
            if object.hit(ray, t_min, closest_so_far, &mut temp_rec) {
                hit_anything = true;
                closest_so_far = temp_rec.t;
                *rec = temp_rec;
            }
        }

        hit_anything
    }
}

And now, before you start doubting your whole existence, let me explain why (and how) not to translate C++’s return-by-reference code. Remember that Rust is fond of moving rather than coping (unless the type is marked Copy), and here the temp_rec variable gets set(otherwise Rust will nag you about the uninitialized variable) and then is read from in the loop to be finally assigned to the outbound reference. And this is where all the trouble happens… Here, the temp_rec is moved out and Rust cannot trust you that it will be initialized again in the successive iterations. After some clarifications on reddit the problem boils down to passing uninitialized memory to a Rust function. This can be done with the unsafe code or a special wrapper - core::mem::MaybeUninit

Since this guide is meant to introduce you to Rust and how it achieves its goals, this is a perfect opportunity for some rustacean culture!

Enter Option - Rust’s type for values that may or may not have a value (std::optional). hit function is a good candidate as this type’s user as it both returns by reference and returns a success value. So let’s see how this code would look like with this type instead:

impl Hittable for HittableList {
    fn hit(&self, ray: &Ray, t_min: f64, t_max: f64) -> Option<HitRecord> {
        let mut temp_rec: Option<HitRecord> = None;
        let mut closest_so_far = t_max;

        for object in &self.objects {
            match object.hit(ray, t_min, closest_so_far) {
                Some(rec) => {
                    closest_so_far = rec.t;
                    temp_rec.replace(rec);
                }
                None => {}
            }
        }

        temp_rec
    }
}

Notice how the bool variable is gone and the code seems to be more understandable (at least for me). Gone is the misleading out-reference, and the function returns a Option<HitRecord> which is more idiomatic. What you may have noticed is that this type takes either a None or Some(value) for its contents. We initialize it with None and then if the hit function returned a Some we do our true branch. Otherwise we do nothing as indicated by None => {}. The match clause is a powerful tool which was one of killer features when my colleague sold this language to me a while ago. If you forget about the mut modifier next to the variable’s type, no worries - Rust will notify you about it 😁

There are small changes that need to be made in the sphere.rs file:

let mut rec: Option<HitRecord> = Some(HitRecord::with_values(ray.at(root), root));
let outward_normal: Vec3 = (rec.as_ref().unwrap().p - self.center) / self.radius;
rec.as_mut().unwrap().set_face_normal(ray, &outward_normal);

The HitRecord variable is now initialized with Some and is modified later with as_ref().unwrap() combo. Be careful with using unwrap() though - it’s like telling the compiler: I know then value is not None, just give it to me!, so when it is indeed None, the code will panic. Similarly, if you wish to modify the object stored in the Option, call as_mut() before doing so. Finally we return the Option<HitRecord> as before.

Adding external crates and global variables

Original Chapter 7

In this chapter, we implement antialiasing using (not so) random capabilities of our hardware. For that we need a random number generator and while C++ has its #include <random>, Rust does not have a built-in for random numbers. Thus we will need to make use of our build and package manager - Cargo. As you already know, it handles most of the irritating details for us and lets us focus on the actual programming (while still allowing for plumbing the build machinery).

Adding a dependency to Cargo is as easy as including it in the Cargo.toml like this:

[dependencies]
rand = "0.8.3"

This follows the semver versioning system, which is quite easy to grasp. At this point I would also like to propose an alternative to manually adding the dependency with the command cargo add. It is not a built-in and requires you to install cargo-edit, which is a set of tools endorsed by the Rust language.

With the new rand package added, we may create our random-generating function. Honestly, I was not sure if it was worth creating a new function for this as the call is simply:

use rand::prelude::*;

#[inline]
pub fn random_f64() -> f64 {
    rand::thread_rng().gen()
}

While the original uses static function variables, meaning they are shared through all function calls. In Rust, however, there are no such things, and the global mutable shared state is strongly discouraged. There are some alternatives, like the singleton pattern, but this would be too much for now. Thus, we remain steadfast in our initial implementation, knowing the tread_rng object is created each time we call random_f64.

Handling Errors

The rest of the original tutorial will guide you through the complex maths of rendering different materials and making the camera much more flexible. Notable differences include implementing a trait for Materials and matching on Options instead of usual branch conditions. Feel free to look up the repo if you are stuck - but try to push it yourself and let the Rust compiler guide you 😄

Although this tutorial does not cover the language extensively (there is only so much one can do with a short project), it strives to show you some of most prominent features of the Rust programming language (copyright). Leaving out error handling would be a disgrace, so here they are: recoverable errors and unrecoverable errors. The latter were discussed briefly when introducing unwraps on the Option type, while the former were not mentioned at all yet.

Recoverable errors are those, which signal an improper condition in the program, but do not abort it, simply try to mitigate (if possible) any consequences induced by such an error. For example, we had our function, which saved data to a impl std::io::Write buffer, and this write is not guaranteed to succeed. For instance, we are writing it to a file descriptor that is already closed by a different thread, or it is being written over the network and there was failure delivering the packet (this is of course quite stretched reason but you get the point). In such case we should handle this failure and probably reprocess this chunk of data, or at least signal this to the user.

So in our case we can change the signature of our function to be:

pub fn write_color(
    stream: &mut impl Write,
    pixel_color: Color,
    samples_per_pixel: i32,
) -> Result<(), io::Error> {
    let mut r = pixel_color.x;
    let mut g = pixel_color.y;
    let mut b = pixel_color.z;

    // Divide the color by the number of samples
    let scale = 1.0 / samples_per_pixel as f64;
    r = (scale * r).sqrt();
    g = (scale * g).sqrt();
    b = (scale * b).sqrt();

    match stream.write_fmt(format_args!(
        "{} {} {}\n",
        (256.0 * r.clamp(0.0, 0.999)) as i32,
        (256.0 * g.clamp(0.0, 0.999)) as i32,
        (256.0 * b.clamp(0.0, 0.999)) as i32
    )) {
        Ok(_) => Ok(()),
        Err(e) => Err(e),
    }
}

You can see, that the ubiquitous match construct is used once again here. Of course at the callee side we should do something with our error, and for this simple problem I will simply report it to the stderr like this:

match write_color(&mut handle, pixel_color, SAMPLES_PER_PIXEL) {
    Ok(_) => continue,
    Err(e) => eprint!(
        "Oops, error {} saving pixel {} for indices i {} j {}",
        e, pixel_color, i, j
    ),
}

You can see the obvious downside to this? Yes, while verbosity is often a desired feature and makes understanding code easier, being over-verbose counters this effect, effectively (no pun intended) bloating the code - rendering it tedious to comprehend. Thankfully we have a syntactic sugar made just for this occasion - ? operator. This allows for propagating any error that arose until it is finally handled in the outer scope. So now we simply write:

stream.write_fmt(format_args!(
      "{} {} {}\n",
      (256.0 * r.clamp(0.0, 0.999)) as i32,
      (256.0 * g.clamp(0.0, 0.999)) as i32,
      (256.0 * b.clamp(0.0, 0.999)) as i32
  ))?;

and we handle the error in the main function. Of course we need to change mains signature to also return a Result<(), io::Error>.

This is by no means complete introduction to error handling and this subject is quite broad. I recommend of course reading the chapter on error handling in the official book and then follow various resources liked there for building on top of it.

Conclusions

Whew, that was quite a lot of information to digest. I hope you took your time and learned some valuable knowledge about the language and how it differs from the C++ or other languages you are familiar with. I skipped over some subjects, like unsafe Rust or embedded Rust applications. There is also much more you can do with the trait system, concurrency, async programming and closures. There are also many other functional topics and of course data structures, under the guise of collections.

All credit for the raytracer algorithms + the guide I based this post on, goes to the author - Peter Shirley. Once again, thank you for sharing with us your expertise about computer graphics and providing the readers with valuable insights and tricks. If you are interested in deepening your knowledge on this topic - check out his other books.

Also, thank you kind reader for the patience to read this thing through. Don’t hesitate to leave review below or reach to me directly via my mail. Enjoy the newly met Rust language and let it help you in your programming adventures!

Ending words

As of today, the subject of the error handling is so important, there is a whole project group in the Rust community, of which I am a proud member. Come over and chat with us if something is bothering you about the current state of error handling or you would like to get involved in the process.

If you like what I’m doing and you would like to see more of it - consider buying me a [coffee] ☕ [coffee]: https://www.buymeacoffee.com/jduchniewicz

Update 28.02

I am already writing with almost 70 wpm constant. So this is quite productive layout all in all.

This entry is a special one! It is dedicated to people brave or crazy enough to do a keyboard layout switch. I won’t go into details on why you should consider a different layout if you are like me, spending rather big part of your life going clack-clack and pretending you are working and not browsing reddit and social media. I curated some materials you may want to study in order to make up your mind:

• original - The official Workman layout
• icyphox - an opinionated blog describing similar experience to mine
• Keyboard Layout Analyzer - website that analyzes the text you input and spews out the most optimal layout for it
• carpalx - two layouts in one, a popular alternative to Workman and Colemak
• keyboard-design - a good source of info from a person experimenting a lot with kb layouts

There seem to be a shortage of relevant study on this topic, which was surprising to me, as nowadays we all use keyboards daily (especially these awful laptop ones). If you, dear reader are a specialist, please consider doing such research and help us all find the perfect layout. My hope is that we will be soon having personalized layouts so that our hands do not strain so much.

I decided to switch layouts because I started to feel fatigue after prolonged sessions of writing journals or even ordinary code. Foreseeing future and being fond of my hands I knew something had to be done!

Enter Workman

My interest was piqued by several layouts, namely Colemak, Colemak-DH and classic and well-known Dvorak. However, I was sold on Workman as the upgraded Colemak (which is also repaired by the DH mod). I may try the DH mod in some time and decide it is superior to Workman but this is unlikely. Having spent just 12 hours practicing I am able to type with 50 WPM and found most keys easy to learn and replace.

As for the practice website, I can highly recommend keybr, which guides you quite well through all keys in a reasonable order, gives you neat graphs and even some competition! Additionally if you want to have some fun while you are at it, try ztype - a small typing space shooter game.

Installation (X11 setup)

As for the X11 (main system) keyboard setup and for other systems, in the basic form it requires only installation of the layout from official workman repo - system specific instructions are provided there. In my case I wanted to have both German and Polish special signs and followed this great tutorial by Michał and ended up with such a configuration.

Remember to create a fresh file, otherwise you won’t be able to turn the layout on with the command:

setxkbmap -v workman_pl_de && xset r 66

Where xset part takes care of proper autorepeat rate of keys.

Setup Vim

Vanilla

Assuming you need a text editor, you probably do not want to alter your muscle memory and relearn where each special key is after the switch. Thankfully, vim provides users with a method langmap which allows for easy remapping of all key combinations to a new layout leaving your muscle memory intact.

I found this to be quite tempting but soon fell into the habit of striking the Workman keys more and more and turned it off. Here is the relevant plugin, but it is in my opinion not that developed (shortcuts like Ctrl-WW are missing).

In order to make my life with hjkl bearable (too much muscle memory) I remapped them toAlt-[yneo] so they are in the same place as before with just the modifier key added. Hopefully, in time I will become true vim ninja not needing these puny keys and relying only on motions and commands, but today I am still too used to corrections and off-by-one movements. The bindings can be seen here (the funny ^[ sign is the escape sign for the Alt key and you obtain it by pressing Ctrl-V in insert mode and pressing the key you want to input).

""" Workman
" Map to Alt-yneo instead of hjkl
nnoremap ^[y h
nnoremap ^[n j
nnoremap ^[e k
nnoremap ^[o l

vnoremap ^[y h
vnoremap ^[n j
vnoremap ^[e k
vnoremap ^[o l

In case you are not familiar with vim remappings, nnoremap is for the normal mode and vnoremap is for the visual. This way I can use my long-learned habits of hjkl with only a slight complication. Of course now we come to the topic of plugins but this is only slightly more convolved.

Plugins

This post is not dedicated to advocating any particular plugins, but if you are curious which I am using, take a look at my vimrc. Because my muscle memory was not that tied to particular keys and instead remembered the combination by name, I did not change anything here. You, however might want to do some jumping around here, depending on your usage of non-vanilla vim.

Setup Fish

This setting applies to every shell there is and I will focus on the fish shell (which you should try out if you never did!). Most probably you already know about going forwards and backward through the shell history commands (Ctrl-p and Ctrl-n - previous and next) and if you didn’t they try it out and don’t thank me :) These keys are actually in quite handy place on the keyboard so no need to change them for me, even more, they are now handier to use in my opinion!

The keys I am having most trouble with are Ctrl-c and Ctrl-v which are shifted one column to the right. Also, moving around the command parts in the shell prompt is non-intuitive and I might remap this later via fish-bind.

The last of inconveniences are related to a plugin I am using systemwise (fzf - smart searching and directory changing). This plugin is a lifesaver but getting used to misplaced r, t and c is annoying. Of course it can be remapped, but not worth it anymore.

Lastly, browsing diffs, manpages and all other utilities basing on the emacs/vim bindings is a stretch, but as possible one and even as I was browsing manpages today, I did not have as much trouble as I thought.

Setup Tmux

Setting up tmux is fairly easy, as since several versions it has become highly customizable. Below you may find the most important settings along their QWERTY counterparts.

### QWERTY ###
# Use vim type keys for navigating between windows
#bind h select-pane -L
#bind l select-pane -R
#bind k select-pane -U
#bind j select-pane -D

# Use vim type keys for re-sizing panes
bind -r < resize-pane -L 1
bind -r > resize-pane -R 1
bind -r - resize-pane -D 1
bind -r + resize-pane -U 1

# Copying with vim shortcuts
#bind-key -T copy-mode-vi v send-keys -X begin-selection
#bind-key -T copy-mode-vi y send-keys -X copy-selection
#bind-key -T copy-mode-vi r send-keys -X rectangle-toggle
#bind P paste-buffer
#set-option -s set-clipboard off
#
## copy to the xclip
#bind-key -T copy-mode-vi Enter send-keys -X copy-pipe-and-cancel "xclip -sel clip -i"
#bind-key -T copy-mode-vi MouseDragEnd1Pane send-keys -X copy-pipe-and-cancel "xclip -sel clip -i"
#
#bind c source-file ~/.tmux/dev

### Workman ####
bind y select-pane -L
bind i select-pane -R
bind e select-pane -U
bind n select-pane -D

bind-key -T copy-mode-vi m send-keys -X begin-selection
bind-key -T copy-mode-vi j send-keys -X copy-selection
bind-key -T copy-mode-vi r send-keys -X rectangle-toggle
bind P paste-buffer
set-option -s set-clipboard off

# copy to the xclip
bind-key -T copy-mode-vi Enter send-keys -X copy-pipe-and-cancel "xclip -sel clip -i"
bind-key -T copy-mode-vi MouseDragEnd1Pane send-keys -X copy-pipe-and-cancel "xclip -sel clip -i"

bind m source-file ~/.tmux/dev

With these binding handy, I can do most tasks as before. The only problems being the vim mode bindings which seem to be hardcoded in the tmux source code. After using them for a while I can say it that they are mostly comfortable and I do not plan to change them.

Browser

Like most modern people, I am using a GUI browser (Firefox) and sometime don’t want to get distracted by removing my hands from the keyboard. I tried using some plugins to automate that, starting from saka key, to finish on Vimium.It allows for key rebinding, so my precious hjkl is safe. Because I am terrible at keeping it clean, my tabbar has over 350! tabs (don’t ask) - thus last year I decided to write a small plugin which would automate my switching to them and searching in different windows only - find-tab. Now, with Vimium I got most of its functionality already there, but Vimium is slightly over-dumbing it and I prefer using my plugin.

Conclusions

While the transition to the new layout is not smooth and without its hindrances, I am glad I could rebind almost everything as wanted and achieve a portable layout without any system hacking and rebinding keys in the Xorg configs. Typing with some speed this soon after switching is a pleasure for my ego (I improved 5 WPM since starting to write this post - of course with some training in between :)

I am keyed up (no pun intended) with the speeds and pleasure I will have from mastering this layout soon. I also hope I will be able to switch to QWERTY from time to time and not be totally baffled by it (reddit users claim this is not a big deal after some time so I remain positive).

I hope this post will help you with choosing the proper layout, and if it is workman, then transitioning to it as well. We should take great care of our bodies and especially hands, since most of us are earning their lives with their help. Thank you for reading and stay safe! Please comment below and help me make this blog a better place.

Btw. I changed the comment engine to utterances as it is more responsible in terms of privacy than disqus.

Considering that you like what I’m doing and you would like to see more of it - consider buying me a [coffee] ☕ [coffee]: https://www.buymeacoffee.com/jduchniewicz

I am bursting with excitement as I am writing this - I can finally share with the world something I have been working very hard for the first half of this overly eventful year!

Today I have uploaded my Bachelor’s thesis for open access - it can be viewed at this link. If this was something not worth your time or digital space (already polluted), I would not blog about it :)

This project was probably the most ambitious one in my whole life, and what is most important - successful. Back in my previous job, my colleagues who have already graduated presented me with their theses, one of which utilized an FPGA for megahertz signal synthesis. Much as I like tackling challenges, black magic was not something I was eager to dab in. However, my curiosity on the subject of FPGAs was born, and soon I begun development of my thesis.

In case you are a busy person and don’t want to dig into the thesis too much, here is the link to my about subpage containing the abstract of the paper, and here are some sounds generated on the FPGA and recorded on the Linux side of the SoC.

Sine wave

Square wave

Sawtooth wave

Triangle wave

Polyphonic capabilities

Beatles - Let it be intro (tragic cover by me…)

Favourite song of Pope JP II - Barka

Line-up

Barring some digital circuits courses at WUT, I was on uncharted waters and had to quickly get a grasp on the subject. In this series of posts I wish to give you brief introduction to the subject of FPGAs and sound synthesis, along with some tips I wish somebody told me when I started learning it. The listing below may be changed in future, so do not take it for granted, additional topics may yet arise.

Agenda

• FPGAs for software engineers
• Digital Synthesis basics and tips
• Digital Filtering basics and State Variable Filter Overview
• FPGAs and Linux
• Linux Device Drivers basics
• Linux ALSA and DMA Drivers

Update 02.2021

I did not write these blog entries yet as the end of the year time was rife with other project (Envidrawer, Robotics) and I will be coming back to the topic pretty soon from a different angle (I hope so!). If somebody cannot wait that long, please message me and I will be glad to share my experience.

The code (with detailed guides etc.) will be available on GitHub soon (although astute readers will find almost all necessities in the paper already).

Other updates

I recently got chosen to participate in a project on element14 - a sustainable closed-space smart gardening solution utilizing less than 1m³ of space Envidrawer. I will probably post some more updates regarding this once the project is worth showing off :)

I am gradually falling in love with Rust and will be trying some embedded solutions, probably mixing some freshly learned Machine Learning tricks to spice it up. Meanwhile, stay tuned for first post of FPGA series, coming this month!

Thanks for reading!

If you like what I’m doing and you would like to see more of it - consider buying me a [coffee] ☕ [coffee]: https://www.buymeacoffee.com/jduchniewicz

This is my second post, glaringly differing from the previous one in terms of topics presented. Right now I am working on a visualizer-equalizer project in Rust language (for hands-on learning experience!). As the name implies, it concerns Linux Audio and specifically Advanced Linux Audio System, which is one of more popular low-level frameworks on this OS. Since Rust audio frameworks are mostly in their infancy at the time of writing this, I decided to use [cpal] library which interfaces several other backends for portability of the application.

Right now the application is utilizing GTK for the GUI part and is not yet ready for show-off (unfortunately), but it will be soon!

Goal

Because in principle I want to display frequency components of the audio signal being played in my headphones, I need to split the sound source for recording. This will be achieved by modifying ALSA’s .asoundrc file, or any other configuration file found in /etc/alsa/conf.d/ (note that ones with higher index will overwrite variables set in these with lower). Once, the configuration is proper, I want to see the device listed with either aplay -L or arecord -L and make sure I can use it as desired. Astute readers may notice that I use device for something not residing in /dev/, which is because ALSA allows for virtual devices that do not have 1 to 1 mapping with these from /dev/.

Small ALSA introduction

I have mentioned ALSA too many times before without explaining why does one even need to use it in the first place. In the days of yore, there was the Open Sound System, which was the go-to audio backend for Unices. Once we entered more modern days, something more robust and thread-safe was required, and thus ALSA was born. Inside Linux kernel, all audio devices are served via the ALSA API (which is not the most pleasant API to work with).

Whenever any audio-related functionality is performed on Linux, it goes through ALSA eventually (or OSS if it is a dinosaur kernel). Other sound servers such as JACK or PulseAudio are a kind of extension of ALSA’s functionality and they are two most popular audio servers. JACK is favoured by people enamored with good audio quality and low latency, whereas PulseAudio is distributed with most major Linux distros, such as Ubuntu or Fedora. Therefore, getting a grasp on basics of ALSA is often useful when you would like to modify audio related settings in your system.

ALSA config

The [minimal configuration file] that satisfied my needs took me a while to find. After understanding the syntax of the config files (which is explained nicely [here]), I was able to understand and tweak the .asoundrc but still no device would show up when queried. After hours of searching to and fro, I decided to dig into cpal’s code and then into Rust’s ALSA bindings. It turns out that ALSA devices can have hints which are used to describe it more verbosely and as it turned out: virtual devices need hints. Below you may see what does the proper config file look like to allow for simultaneous sound playback and recording.

defaults.pcm.dmix.!rate 48000
defaults.pcm.dmix.!format S32_LE
pcm.multi {
    type multi
    slaves.a.pcm "dmix:PCH"
    slaves.a.channels 2
    slaves.b.pcm "dmix:Loopback"
    slaves.b.channels 2
    bindings.0 { slave a; channel 0; } # bind channels to slave devices
    bindings.1 { slave a; channel 1; }
    bindings.2 { slave b; channel 0; }
    bindings.3 { slave b; channel 1; }
}
pcm.both { # duplicates channels for 'multi' module 
    type route
    slave.pcm "multi"
    ttable.0.0 1
    ttable.1.1 1
    ttable.0.2 1
    ttable.1.3 1
}
pcm.!default { # override default PCM
    type asym # type asymmetrical
    playback.pcm "plug:both"
    capture.pcm "plug:dsnoop:PCH"
    # Need a hint, else aplay -L won't show this device
    hint.description "Default output device forwarding captured data to Loopback loop" # THIS IS THE OFFENDER (rather lack of it)
}

#pcm.loop "plug:\"dsnoop:Loopback,1\"" # this is the same as below - just compact
pcm.loop {
    type plug
    slave.pcm "dsnoop:Loopback,1"
    hint.description "Loopback device which captures from default PCM"
}

Brief explanation, this config overrides default sampling rate and format for this devices streams. It defines several PCM modules (kind of ALSA’s processing entities), and connects them, multiplexing the playback to both a Loopback device and audio device down the chain. This SO [anwser] goes into much detail about it if you wish to understand more about it.

PulseAudio config

As most users won’t need to dabble in ALSA, instead relying on PA or JACK, I will present below another .asoundrc which captures the monitors(duplicates) of desired PulseAudio devices (sorry, no JACK config - I am not familiar with it yet).

pcm.steelseries_game {
    type pulse
    device "alsa_output.usb-SteelSeries_SteelSeries_Arctis_5_00000000-00.analog-game.monitor"
    hint.description "PulseAudio Game Monitor"
}

pcm.steelseries_chat {
    type pulse
    device "alsa_output.usb-SteelSeries_SteelSeries_Arctis_5_00000000-00.analog-chat.monitor"
    hint.description "PulseAudio Chat Monitor"
}

Again, without the hint.description tag, they won’t show up in the aplay’s listings.

Conclusions

Having read this post you will be able to create your own ALSA configuration for simultaenous capture and playback for both ALSA and PulseAudio. This is not the end of my ALSA adventures, but the next time I will write about it will be from the device driver perspective, so we will be looking at some kernel topics, such as DMA and timers. I hope you will find this post useful and educating :) I also hope to be posting some more in the upcoming weeks, as currently I have some free time due to current world situation. So stay productive and stay safe.

As always, if you like what I’m doing and you would like to see more of it - consider buying me a [coffee] ☕ [coffee]: https://www.buymeacoffee.com/jduchniewicz [cpal]: https://docs.rs/cpal/0.12.1/cpal/index.html [here]: https://wiki.archlinux.org/index.php/Advanced_Linux_Sound_Architecture#Configuration [minimal configuration file]: https://ristovski.github.io/posts/master-alsa-config/ [anwser]: https://stackoverflow.com/a/44217925/7092926

This post will make you familiar with some basic features of GitHub Actions featuring code for cross-platform Continous Integration. This is my first post so any feedback is appreciated :)

Why?

Until a few years ago, managing continuous integration and deployment of a software product was quite sophisticated and required much toilsome work. I do not claim it is easy now, but it is certainly much easier thanks to tools like GitHub Actions. While [PolyEngine] had pretty good CI and CD already running it was divided between Travis(Linux and macOS) and Appveyor(Windows). Thankfully GitHub Actions try to make developer’s life as easy as possible and allow for builds for all 3 platforms in one place. I decided to give it a go while doing our side project [PolyDock] which is a standalone plugin for managing windows in a Firefox/Chrome-like manner.

The basics

The language in which it is written is YAML, which is a recursive acronym for “YAML Ain’t Markup Language”, the language itself is quite punishing and required usage of spaces instead of tabs(no bias here at all). However, when editing it in GitHub IDE you get nice suggestions and autocompletion - it would still benefit much from interpreting it on-the-go. The painful truth you have to accept is that your commit count will bloat once you start configuring GH Actions. There is a tool([Act]) which allows for testing them offline, but only Linux builds are supported :/

This said, it is time to show some code:

# You start with the workflow name
name: CI

# This workflow is triggered on pushes and PR's to the repository
on: [push, pull_request]

# And you specify jobs it has to run
jobs:
  ci: # in this case CI

Every job is composed of steps, which specify what will happen sequentially when it is run - take this into account when writing your own workflows! You have to specify a name and the commands it should run. Something I struggled to find is - how to run given step conditionally. Well, nothing simpler! Just bash an if statement at the end of particular step and v’oila - it triggers only on particular conditions.

  steps:
  - name: Install Dependencies (Linux) # notice that each step begins with '-' followed by a space
    run: | # pipe operator allows you to chain commands in several lines
        sudo apt-get update
        sudo apt-get install -y -qq cmake # qq for limiting verbosity
    if: matrix.os == 'ubuntu-latest' # for now ignore the 'matrix' part
  - name: Echo Something
    run: echo "Hey!"

Remember not to include new-lines between entries in jobs, otherwise GH Actions will complain about YAML issues and refuse to run your job - prepare a next commit…

Going cross-platform

Now for the matrix part:

jobs:
  ci:
   runs-on: ${{ matrix.os }}
   
   strategy:
    matrix:
        os: [ubuntu-latest, macOS-latest, windows-latest]

Matrix is the specified configuration you would like to run the job on. Apart from specifying the OS, you can also set desired package versions (but keep in mind that some packages may be not present in pre-built OS images), and customize settings for any of the operating systems of choice by include or exclude keywords. It is worth noting that if you are running several jobs in parallel(multiple OS compilation), if one fails, all will fail too. If you wish to run all the jobs till the end even if one fails specify fail-fast: false in the strategy settings.

Additional commands

If you wish to use external actions - no problem! Just specify a uses clause and add the path of the action you want to use:

  - name: Install Qt
    uses: jurplel/install-qt-action@v2 # specify the user and repository along with the version of this action

If you want to run a series of commands and do not want to specify a directory each time (or directory structures vary between systems) - use working-directory:

   - name: Setup CMake
     run: |
        mkdir "${{ runner.workspace }}/PolyDock/Build" # runner.workspace is the variable of the current runner 
        cd "${{ runner.workspace }}/PolyDock/Build" 
        cmake ../PolyDock

Probably the most used action is the checkout action which checks out your repository on the runner, basically performing a git pull. However be careful when you have submodules! It won’t initialize them automatically, so you have to specify it by hand like so:

   - name: Get Sources
     uses: actions/checkout@v2
     with:
       submodules: 'true'    

Conclusion

While I covered some basic features, and some that I simply needed, your configuration will be probably much more complex. Thus, I recommend consulting the official docs (they present the basics pretty well, but obscure some details like if statements in steps) and/or this excellent [post series] by Edward Thomson, which dig deeper into this subject. I hope that now you have some knowledge about the amazing world of GitHub Actions and will automate your own repositories :)

On closing words, if you like what I’m doing and you would like to see more of it - consider buying me a [coffee] ☕ [coffee]: https://www.buymeacoffee.com/jduchniewicz [PolyEngine]: https://github.com/PolyEngineTeam/PolyEngine/ [PolyDock]: https://github.com/PolyEngineTeam/PolyDock/ [Act]: https://github.com/nektos/act [post series]: https://www.edwardthomson.com/blog/github_actions_advent_calendar.html

I always wanted to have my own personal place for sharing my both technical and geeky content with the general public. Well, here it is and I hope it will be of some use to you. In the upcoming days I am mostly doing my thesis project and cannot publish it until it is done and defended… However, I will post some other posts related to some side projects of my own and of course PolyEngine.