From fd2f137a9b7a5f95b8cb9524e46747844f7180b4 Mon Sep 17 00:00:00 2001 From: Norman Feske Date: Tue, 25 May 2021 17:17:06 +0200 Subject: [PATCH] Release notes for version 21.05 --- doc/release_notes/21-05.txt | 817 ++++++++++++++++++++++++++++++++++++ 1 file changed, 817 insertions(+) create mode 100644 doc/release_notes/21-05.txt diff --git a/doc/release_notes/21-05.txt b/doc/release_notes/21-05.txt new file mode 100644 index 0000000000..d53fd31efc --- /dev/null +++ b/doc/release_notes/21-05.txt @@ -0,0 +1,817 @@ + + + =============================================== + Release notes for the Genode OS Framework 21.05 + =============================================== + + Genode Labs + + + +The most prominent user-visible features of Genode 21.05 are the support for +webcams and an easy-to-use component for file encryption on +[https://genode.org/download/sculpt - Sculpt OS]. Both topics greatly benefit +from Genode's component architecture. The video-conferencing scenario +described in Section [Webcam support] sandboxes the webcam driver in a +disposable Genode component while using a second instance of the nitpicker GUI +server as a video bridge. This design strikes a beautiful combination of +simplicity, robustness, and flexibility. + +The new file vault described in Section +[File vault based on the CBE block encrypter] leverages Genode's dynamic +sandboxing capabilities to manage the creation and operation of an encrypted +file store. Even though the underpinnings can be described as nothing less +than sophisticated machinery, the package presented to the user combines ease +of use with a great sense of control. + +The second focus of the current release are the manifold improvements of +Genode's driver and platform support as described in Sections [Device drivers] +and [Platforms]. Our USB support received the attention needed to accommodate +the webcam scenario, the arsenal of i.MX8 drivers got enriched with I2C and +power-domain control, the Pine-A64 board support is growing, Genode has become +able to run on 64-bit ARM Linux, and we enabled principle networking for +RISC-V. + +Speaking of platforms, this release features the first version of a new +"Genode Platforms" documentation (Section [Updated and new documentation]) +that aids the porting of Genode to new ARM SoCs. With this document, we share +our former in-house know-how and methodology about the porting and development +of drivers with developers outside of Genode Labs. + +The release is rounded up by several performance optimizations +(Section [Performance optimizations]) to the benefit of most Genode system +scenarios. Furthermore, it is accompanied with an updated tool chain, +following our established two-years rhythm +(Section [Tool-chain update to GCC 10.3 and binutils 2.36]). + + +Webcam support +############## + +During 2020, the amount of home office and remote work took an unexpected turn. +Video conferences and video chats have become the norm, which people and +companies rely upon. Even though, not to be found on our +[https://genode.org/about/road-map - road map] for 2021, this development +prompted the Genode team to explore the field of webcam and video chat support +on Genode. + +Webcams are generally connected via USB to a host device and implement the USB +video device class +([https://www.usb.org/sites/default/files/USB_Video_Class_1_5.zip - UVC spec]). +Therefore, it is possible to drive many different webcam devices using the +same USB interface. To support this protocol, we enabled +[https://ken.tossell.net/libuvc/doc - libuvc], which offers fine-grained control +over UVC exporting USB devices. In order to enable _libuvc_ on Genode, we +simply integrated the library into Genode's port system with no further +changes required. _libuvc_ depends on [https://libusb.info - libusb] as a back +end to access the actual webcam device. While there exists a port of _libusb_ +for Genode - that connects through Genode's USB session interface to the USB +host controller - the port still lacked support for isochronous USB transfers +as required by UVC devices. Isochronous transfers represent a continuous +stream of data (either input or output) with a constant rate without delivery +guarantees. We extended _libusb_ to handle isochronous transfers, which were +already supported by Genode's USB session. Observing that this kind of +transfers can cause high load within the USB host driver, we optimized +isochronous transfer support at the host driver level (Section [USB]). + +At the front-end side, we created a small _usb_webcam_ component that uses +_libuvc_ in order to enable, disable, and configure the camera. The component +connects to a GUI session, and thus, can be interfaced directly, for example, +to the Nitpicker component for rendering webcam images natively on screen. +Whereas Genode's pixel format is 32 bit RGB, webcams stream data in the YUV2, +MJPEG, or H.264 formats. To handle the conversion of these formats to Genode's +pixel format, we utilize the +[https://chromium.googlesource.com/libyuv/libyuv - libyuv] library and thereby +support the YUV2 as well as the MJPEG pixel format for webcams. + +Additionally, we wanted to be able to transfer the webcam data directly into +our VirtualBox port, thus enabling, sophisticated video conference systems +like Jitsi or Skype. + +[image webcam] + +Our USB host-controller support for VirtualBox is based on the ported Qemu USB +3.0 (XHCI) controller model. Since no USB webcam device model is available for +Qemu, we were required to develop a one from scratch. The new USB webcam model +is attached to the QEMU USB XHCI controller and operates as a bulk endpoint. +In contrast to an isochronous endpoint, the model causes less CPU load and +fewer virtual interrupts. The supported formats offered to the guest are YUV2 +and BGR3. By enabling the USB webcam model within the Genode VirtualBox +configuration, a _Capture_ session is used to capture pictures at the rate of +a configured _fps_ value. The following snippet shows the default values of +the supported configuration attributes. + +! +! ... +! +! ... +! + +If the _screen_size_ attribute is set to _true_, the device model determines +the resolution from the established capture session. Otherwise, the specified +_width_ and _height_ values are used. The _vertical_flip_ attribute is useful +for the BGR3 format, which is - when interpreted by Linux guests - flipped +vertically and can be flipped back by setting the attribute to _true_. + +If the _report_ attribute is set to _true_, a report will be generated +whenever the guest changes the state of the webcam model, either by switching +capturing on/off or by changing the pixel format. + +! + +[image webcam_chat] + +Finally, our developers, croc and lion, setup the Webcam scenario in Sculpt +and test drive the new feature fascinated. The picture shows a session via +Jitsi, on the right side croc participates at the meeting via a Win10 VM on +Sculpt and lion sitting left joined via an Android tablet. + + +Performance optimizations +######################### + +One of the overarching topics of this year's +[https://genode.org/about/road-map - roadmap] is optimization. +As part of working on the Sculpt OS +[https://genode.org/news/sculpt-os-21.03-boots-now-in-2.5-seconds - version 21.03], +we identified several optimization vectors with the potential for user-visible +improvements. In particular, while interacting with the system, a few effects +made us curious. + +Operations that involved changes to the runtime subsystem, e.g., adding or +reconfiguring a component, seemed to interfere with multi-media workloads. +When running a graphical animation, we could see it stutter in such +situations. Another direction of our curiosity was the boot time of the +system. The boot time of Sculpt OS has always been relatively quick compared +to commodity operating systems. E.g., on a 5-years old laptop like a Lenovo +x260, the system used to boot in about 5 seconds to the graphical user +interface. However, with the anticipation of Sculpt OS on lower-end platforms +like the Pinephone and with the vision of instant-on systems, we wondered +about the potential for improvement. + +While gathering a CPU-load profile of the boot process using the top tool, we +learned that the boot time was bounded not by I/O but by the CPU load (the +kernel's idle thread did not appear in the profile). Interestingly, a +significant portion of the cycles were consumed by various instances of the +init component, which prompted us to turn our attention to the implementation +of init. + + +Clock-cycle measurements +------------------------ + +The next natural step was the benchmarking of various code paths of init using +a cycle-accurate time-stamp counter (TSC). Even though Genode has a +'Trace::timestamp' utility readily available, it remains barely used for +manual instrumentation because such instrumentations require too much labor: +allocation of state variables for gathering the statistics, computing time +differences, traffic-shaping of the debug noise (needed whenever investigating +highly frequently called code). These tasks should better be covered by a +utility so that friction-less performance analysis can become a regular part +of our development work. As a side effect of our investigation, we came up +with a new utility called GENODE_LOG_TSC. This utility is covered by a +dedicated article. + +:Performance analysis made easy: + + [https://genodians.org/nfeske/2021-04-07-performance] + +Thanks to GENODE_LOG_TSC, we were able to identify three concrete +opportunities for optimization in a course of one evening. First, the dynamic +reconfiguration of init apparently did not scale well with a growing number of +components. The code for analysing differences of configuration versions +relied on doubly nested loops in order to stay as simple as possible. With the +typical number of 30 or more components/subsystems hosted in Sculpt's runtime, +we passed a tipping point where quadratic time complexity is justifiable. +Second, during a configuration update, the XML data is evaluated in multiple +passes, which puts pressure on the efficiency of Genode's XML parser. This +pressure could in principle be relieved. Third, the process of taking +session-routing decisions involved XML parsing. In scenarios as sophisticated +as Sculpt, the routing rules can become quite elaborate. Since the rules are +consulted for each session route, the costs for the rule evaluations stack up. + + +Init optimizations +------------------ + +These realizations motivated us to replace the hand-crafted configuration +processing by the use of Genode's generic 'List_model' utility. This way, the +parsing follows a common formalism that makes the code easier to maintain and +to understand while reducing the XML parsing to a single pass. The increased +formality cleared the way for further optimizations. In particular, init +became able to skip the re-evaluation of the session routing whenever no +service is affected by the configuration change. This is actually the common +case in Sculpt. + +To alleviate the costs for evaluating session routes, we introduced an +internal data model for the routing rules that is optimized for the matching +of routes. With this model, the detection of a definite mismatch (the common +case) comes down to a comparison of a single numeric value. + +Combined, those optimizations yield a great effect. In a typical Sculpt +system, the time of a dynamic reconfiguration got reduced by factor 10 to the +order of 10 to 20 milliseconds. Hence, the visual stuttering we observed +during structural changes of the runtime are completely eliminated. + +Besides the major optimization of init, we were able to shave off a few +milliseconds from the boot procedure here and there. For example, by deferring +the initialization of the real-time clock driver to its first use, we avoid a +potentially expensive active polling loop during the highly contended boot +phase. Another obvious heuristic improvement is the skipping of the GUI +handling until the framebuffer driver is up because all the nice pixels would +not be visible anyway. + +Combined, these optimizations were able to reduce the boot time of Sculpt from +the entering of the kernel up to the graphical user interface down to only 2.3 +seconds. The improved performance of init is impactful beyond Sculpt OS +because it is a central component of all Genode systems large and small. + + +Updated and new documentation +############################# + +Genode Platforms +---------------- + +We are proud to introduce the first version of a new "Genode Platforms" +document, which complements the existing Genode Foundations book with +low-level hardware-related topics. It is primarily intended for integrators +and developers of device drivers. + +:
+:

+:

+: +: +: +:
+:

+ +In this first edition, the document features a practical guide for the steps +needed to bring Genode to a new ARM SoC. The content is based on the ongoing +Pine Fun article series at [https://genodians.org - Genodians.org]. +We plan to continuously extend it with further practical topics as we go. + +:Initial revision of the Genode Platforms document: + + [https://genode.org/documentation/genode-platforms-21-05.pdf] + + +Genode Foundations +------------------ + +The "Genode Foundations" book received its annual update. It is available at +the [https://genode.org] website as a PDF document and an online version. +The most noteworthy additions and changes are: + +:
+:

+:

+: +: +: +:
+:

+ +* Adaptation to the re-stacked GUI stack introduced in + [https://genode.org/documentation/release-notes/20.08#The_GUI_stack__restacked - version 20.08] +* Coverage of the new uplink, capture, and event session interfaces +* Updated API documentation + +:
+ +To examine the changes in detail, please refer to the book's +[https://github.com/nfeske/genode-manual/commits/master - revision history]. + + +Base framework and OS-level infrastructure +########################################## + +API refinements +=============== + +VFS-access utilities +-------------------- + +Low-complexity native Genode components do not depend on a C runtime. To allow +such components to still enjoy the power and flexibility of the Genode's VFS +infrastructure, we provide an evolving front-end API +[https://github.com/genodelabs/genode/blob/master/repos/os/include/os/vfs.h - os/vfs.h] +first introduced in version +[https://genode.org/documentation/release-notes/19.11#Virtual_file-system_infrastructure - 19.11]. + +The API is tailored and refined according to the relatively simple use cases +of low-complexity Genode components. The current release introduces a new +utility for the creation of new files, appropriately named 'New_file'. The +change is accompanied by a new 'Directory::create_sub_directory' method for +the easy creation of directory hierarchies. + + +Safeguarded arrays +------------------ + +To handle arrays in a safe and C++-like fashion, a new helper class has become +available at _base/include/util/array.h_. It accommodates an increasingly used +pattern where elements are dynamically added at construction time but stay the +same once the array is constructed. + + +Cosmetic changes +---------------- + +We refined the 'Range_allocator::alloc_aligned' interface to make it more +safe. The former 'from' and 'to' arguments are replaced by a single 'range' +argument. The distinction of the use cases of regular allocations vs. +address-constrained allocations is now covered by a dedicated overload instead +of relying on a default argument. The 'align' argument has been changed from +'int' to 'unsigned' to be better compatible with 'addr_t' and 'size_t'. + +The 'Cache_attribute' type has been renamed to 'Cache'. + + +Input-event handling +==================== + +A central component for Genode's input-event handling functionality is the +event filter. It merges input events from multiple event sources and passes +them to the event sink (typically the GUI server). In between, it performs +low-level key remapping and applies character mapping rules. Character mapping +rules are essential for supporting different keyboard layouts (including +dead-key sequences). Low-level key remapping is, for instance, used for +changing the emitted key codes of the Num Pad keys according to the Num Lock +state. The different filter functionalities can be arbitrarily assembled into +a filter chain and provided as a dynamic config ROM to the event filter +component. The event sink then receives and processes the filtered events. + +Some input devices emit unusual and/or extra key codes in certain situations, +which impedes the event sink's ability to detect key combos correctly. We +therefore added the functionality to completely mute certain key codes. In +order to ignore all unknown key codes for instance, we can now add an +'' node to the config of the event filter. + +! +! +! ... +! + +Note, that '' is part of the '' filter. The name attribute +refers to the low-level key name before any remapping rule has been applied. + +As a second addition, we implemented a '' filter that allows low-level +debugging of the event-filter component and its configuration. The '' +filter can appear at each stage in the filter chain. For instance, we can log +the input events before and after the remap filter as follows. + +! +! +! +! ... +! +! +! + +The optional 'prefix' attribute thereby helps to distinguish the log output +from different stages. + + +File-system helpers +=================== + +The +[https://genode.org/documentation/release-notes/18.08#New_component_for_querying_information_from_a_file_system - fs_query] +component is a simple helper to query information from a file system. E.g., it +is used by the file browser of Sculpt OS to obtain the directory structure. +The component received two welcomed improvements. First, directory content is +now reported in alphabetic order. Thereby, all consumers of the reports become +able to rely on deterministic output. For example, the file browser of Sculpt +OS, the launcher menu items, and the depot-selection items will appear in a +predictable way. Second, the size of files can be queried now. By adding an +attribute 'size="yes"' to a query, fs_query is instructed to report the size +of each queried file as attribute 'size' of the corresponding 'file' node. + +Whereas fs_query inspects a file system without changing it, its sister +component fs_tool is able to perform file-system modifications. The new +version adds a '' operation, which writes the content of +the XML node into the file specified as 'path' attribute. The directory +structure leading to the file is implicitly created if needed. Should a file +with the specified name already exist, the original file will be overwritten. + + +Applications +############ + +File vault based on the CBE block encrypter +=========================================== + +Over several releases +([https://genode.org/documentation/release-notes/19.11#Preliminary_block-device_encrypter - 19.11], +[https://genode.org/documentation/release-notes/20.05#Feature-completeness_of_the_consistent_block_encrypter - 20.05], +[https://genode.org/documentation/release-notes/20.08#Consistent_Block_Encrypter - 20.08], +[https://genode.org/documentation/release-notes/20.11#Consistent_Block_Encrypter__CBE_ - 20.11]), +we persistently worked at a native solution for modern block encryption - the +SPARK-based CBE-library - and its integration into Genode's VFS. Even though, +this work was already suitable for real-world scenarios like +[https://genodians.org/m-stein/2020-06-12-cbe-linux-vm - hosting a Linux VM on top of an encrypted block device], +it still lacked stress-testing by a regular user base because its integration +into an end-user system - like Sculpt - required tedious low-level wizardry. + +This situation had to change because we want to encourage as many people as +possible to expose the codebase around the CBE to their workflows and let it +mature. Therefore, we came up with a new package called file vault that can be +readily deployed on Sculpt OS. It is a graphical front end that aims at making +the creation, use, and maintenance of a CBE-based encrypted file store as +intuitive and secure as possible. + +:Introducing the file vault: + + [https://genodians.org/m-stein/2021-05-17-introducing-the-file-vault] + +[image file_vault_setup] + +The file vault only requires two file-system sessions from you (the trust +anchor is stored separately from the payload data). With that, it will +automatically create and connect a trust anchor, set up a CBE image, prepare +an ext2 FS on top of the CBE image and provide it through a file system +service - ready to be used like a simple directory. The directory can be +locked by closing the file vault and unlocked by starting the file vault on +the same trust anchor and entering the correct user passphrase. All controls +for the file vault's underlying CBE encrypter - like for its re-sizing and +re-keying functionality - are presented through a simple and guiding UI that +also provides you with the most relevant status information of your vault. + +The file vault package is accompanied by some notable improvements regarding +CBE's key management. Whereas in the previous release, this aspect was still +merely a prototype with almost no protective value, the current implementation +embraces well-known algorithms to generate and encrypt the keys used within +the CBE respectively the file vault. This is explained in detail in the +aforementioned article. + +As a note of caution, the primary purpose of the current version of the file +vault is to lift native block encryption in Genode from the development stage +to product quality. At the current stage, it is neither time-tested nor +reviewed by independent cryptography experts. Consequently, you should use it +with a healthy dose of suspicion, for non-critical data only! We would be more +than happy to receive feedback on your experience with the file vault. + + +VirtualBox +========== + +Since the previous release, we continued the enablement of VirtualBox 6 on +Genode and put efforts into stabilizing the port. Therefore, we updated to +version 6.1.18 and reorganized the internal structure for a more +comprehensible execution model with fewer threads. Further, we improved +synchronization in multi-processor use cases and added a Sculpt runtime +package for vbox6. + +Finally, as a little treat, our ports of VirtualBox now support to pass extra +buttons of five-button mice to the guest. + + +Device drivers +############## + +Platform driver on ARM +====================== + +The current release streamlines Genode's API for interacting with the platform +driver on ARM platforms. It eases the access to memory-mapped I/O registers +and interrupts by introducing the notions of + +:'Platform::Device': one device obtained from a platform session + +:'Platform::Device::Mmio': locally-mapped MMIO registers of a device + +:'Platform::Device::Irq': interface for receiving device interrupts + +The API is covered in detail by the following article. + +:One Platform driver to rule them all: + + [https://genodians.org/nfeske/2021-04-29-platform-driver] + +It goes without saying that this change touches most ARM-specific drivers. +Closely related, we also revised the concept of the XML based device-info +mechanism provided by the platform driver to accommodate both complex drivers +operating on multiple devices simultaneously such as driver stacks ported from +Linux as well as low-complexity drivers for simple devices. In the new +version, the device XML-information dataspace is only provided if the client's +session policy states 'info="yes"'. The format of the XML information got +refined to include the physical resource names (I/O memory and IRQ addresses) +instead of virtual IDs and page offsets and by using a 'type' attribute +instead of a '' node to uniquely identify devices. + + +Changes specific to i.MX8 +------------------------- + +The platform driver incarnation specific to i.MX8 got slightly improved. It +can handle the configuration of reset-pins now. Analogously to the already +existent power domains, one can assign reset domains per device. Whenever a +device with a reset domain gets acquired, its reset-pins are de-asserted. When +the device gets released again, its reset-pins are asserted to put it into +reset state. A sample configuration looks as follows: + +! +! ... +! + +Technically, those reset domains map to pin settings of the System Reset +Controller (SRC) that is part of the i.MX8 SoC. The SRC is under control of +the platform driver now. Currently, only the pins for the MIPI DSI Phy get +exported. They are used by the graphical subsystem to handle panels connected +via MIPI DSI. + + +I2C driver for i.MX8 +==================== + +Thanks to Jean-Adrien Domage from [https://www.gapfruit.com - gapfruit], an +API for I2C bus transactions and a new I2C bus driver for the i.MX8 SoC +entered our framework. Coincidentally, the need to use the new I2C API more +intensively arose soon after his initial contribution. As a consequence, the +API got extended a bit. The result is a nice joint venture, and looks like the +following: + +! void transmit(Transaction & t); + +Hereby a 'Transaction' is a simple array of 'Message' objects, and a 'Message' +is an array of bytes that are either read or written. For very simple +use-cases, e.g., a client that polls single bytes from a temperature sensor, +some convenience utilities are incorporated into the 'I2c::Connection'. + + +USB +=== + +The USB-driver system has received quite a few refinements, performance +improvements, and robustness handling efforts during the current release +cycle. The HID subsystem is now capable of handling devices where the HID USB +interface is at an arbitrary location within the device descriptors - as +opposed to the assumption that the HID interface is always at the first +position in the interface list of the device. Also, the HID driver now handles +session destruction more gracefully and supports unlimited plug and unplug +events of an associated HID device. + +For the USB host driver, various fixes of newer Linux kernel versions have +been back ported, which concern the handling of DMA memory. Error code and +timeout handling have been improved in order to support more corner cases, and +the USB session handles outstanding USB requests (synchronous and +asynchronous) on sudden session disconnects gracefully now. + +The CPU usage of the host driver for isochronous transfers has been reduced +significantly for Intel XHCI controllers by adding a fix that reduces the +triggering of an interrupt for every completed isochronous packet to one +interrupt per eight packets, bringing the worst case scenario down to 1000 +interrupts per second from a possible 8000 IRQs before. + + +NIC drivers +=========== + +Drivers for iPXE-supported Ethernet devices, Wifi adapters, and Linux TAP +devices now support the reporting of the MAC address of detected adapters. +The feature can be enabled by a '' node in the driver configuration as +follows, prompting the driver to request a report session with the label +_devices_. + +! +! +! + +The resulting report is depicted below. + +! +! +! + + +Platforms +######### + +Genode/Linux on 64-bit ARM +========================== + +The release introduces the support for running the Linux version of Genode on +64-bit ARM platforms. As a part of this line of work, Genode's system call +bindings for Linux underwent a modernization to harmonize the system calls +across the supported CPU architectures. Furthermore, we took the opportunity +to simplify the use of the clone system call by eliminating the need for +passing a TLS pointer. + +Expecting that the 64-bit Genode/Linux version will remain a niche use case of +Genode in the foreseeable future, we do not provide a pre-built tool chain. +Hence, as a preparatory step for using this version of Genode, the tool chain +must be built manually via Genode's _tool/tool_chain_ script. + +As a known limitation, Genode's 'Trace::timestamp' function is not available +on this version of Genode because Linux prevents the user land from accessing +the cycle counter (pmccntr_el0). So the accuracy of timing is somewhat impeded +to the order of milliseconds. Also, the jitterentropy random-number generator +cannot be used. + +Those limitations notwithstanding, one can successfully execute scenarios as +complex as _leitzentrale.run_. When using AARCH64 Linux as host, run scripts +can be executed with the same convenience as on Linux on a PC. + +! $ make run/