denshooter/genode
1
0
Fork 0
mirror of https://github.com/mmueller41/genode.git synced 2026-01-22 04:52:56 +01:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: denshooter:sculpt_vc
Branches Tags
denshooter:ealan denshooter:feature/habitats denshooter:ealan-v2 denshooter:gpgpu denshooter:pc-ixgbe denshooter:staging denshooter:genode-24 denshooter:master denshooter:tasking-profiler denshooter:gpgpu-bench
denshooter:22.05 denshooter:sculpt-22.04 denshooter:22.02 denshooter:21.11 denshooter:sculpt-21.10 denshooter:21.08 denshooter:21.05 denshooter:sculpt-21.03 denshooter:21.02 denshooter:20.11 denshooter:sculpt-20.08 denshooter:20.08 denshooter:20.05 denshooter:sculpt-20.02 denshooter:20.02 denshooter:19.11 denshooter:19.08 denshooter:sculpt-19.07 denshooter:19.05 denshooter:sculpt_ce denshooter:19.02 denshooter:18.11 denshooter:sculpt_vc denshooter:18.08 denshooter:18.05 denshooter:18.02 denshooter:17.11 denshooter:17.08 denshooter:17.05 denshooter:17.02 denshooter:16.11 denshooter:16.08 denshooter:16.05 denshooter:16.02 denshooter:15.11 denshooter:15.08 denshooter:15.05 denshooter:15.02 denshooter:14.11 denshooter:14.08 denshooter:14.05 denshooter:14.02 denshooter:13.11 denshooter:13.08 denshooter:13.05 denshooter:13.02 denshooter:12.11 denshooter:12.08 denshooter:12.05 denshooter:12.02 denshooter:11.11
...
compare: denshooter:sculpt-21.03
Branches Tags
denshooter:feature/habitats denshooter:ealan-v2 denshooter:gpgpu denshooter:pc-ixgbe denshooter:staging denshooter:genode-24 denshooter:master denshooter:ealan denshooter:tasking-profiler denshooter:gpgpu-bench
denshooter:22.05 denshooter:sculpt-22.04 denshooter:22.02 denshooter:21.11 denshooter:sculpt-21.10 denshooter:21.08 denshooter:21.05 denshooter:sculpt-21.03 denshooter:21.02 denshooter:20.11 denshooter:sculpt-20.08 denshooter:20.08 denshooter:20.05 denshooter:sculpt-20.02 denshooter:20.02 denshooter:19.11 denshooter:19.08 denshooter:sculpt-19.07 denshooter:19.05 denshooter:sculpt_ce denshooter:19.02 denshooter:18.11 denshooter:sculpt_vc denshooter:18.08 denshooter:18.05 denshooter:18.02 denshooter:17.11 denshooter:17.08 denshooter:17.05 denshooter:17.02 denshooter:16.11 denshooter:16.08 denshooter:16.05 denshooter:16.02 denshooter:15.11 denshooter:15.08 denshooter:15.05 denshooter:15.02 denshooter:14.11 denshooter:14.08 denshooter:14.05 denshooter:14.02 denshooter:13.11 denshooter:13.08 denshooter:13.05 denshooter:13.02 denshooter:12.11 denshooter:12.08 denshooter:12.05 denshooter:12.02 denshooter:11.11

2454 Commits
sculpt_vc ... sculpt-21.

Author SHA1 Message Date
Norman Feske
0f9cb72cfa depot: update recipe hashes 2021-03-29 13:46:49 +02:00
Norman Feske
27527bf165 wm: allow clients to upgrade session caps 
This patch adds the handling neccessary for clients to upgrade the cap
quota of their GUI session. Until now, the upgrade mechanism supported
only RAM quota.

The limitation became visible once a client - in the particular case the
motif_decorator - attempted the cap upgrade of its GUI session. The cap
quota would arrive at the wm, but the wm would keep it instead of
forwarding the quota to nitpicker. This resulted in an infinite retry
loop at the motif_decorator, ultimatedly depleting all its cap quota.
 2021-03-29 13:45:25 +02:00
Norman Feske
f839b3ecba wm: make hover handling robust against input races 
The window manager infers the overall state from the intercepted input
events for the decorator and all GUI clients. However, each of those
parties have an independent input-event stream. Whereas the order of
events within one GUI session is strict, the order of events between GUI
sessions is arbitrary. The window manager wrongly relied on a global
event ordering to track the pointed-at GUI session.

The patch removes the assumption of a global event order by tracking the
relevant pointer state for each GUI session independently and evaluating
these states when propagating the pointer position to the decorator.

Fixes #4059
 2021-03-29 13:45:24 +02:00
Norman Feske
bfea27a258 Tweak to support delete key in bash 
This commit modified bash's builtin mapping of character sequences to
readline-functionality to detect the sequence of the delete symbol as
generated by the terminal.

Fixes #4058
 2021-03-29 13:45:24 +02:00
Christian Helmuth
4f91d71cf9 zynq_qemu: disable fb_bench and nic_router_flood tests 2021-03-29 13:45:24 +02:00
Jean-Adrien DOMAGE
32169cd137 add I2c driver for imx8q based platform 
This commit introduces a new i2c driver for the imx8q_evk based platform.

Fixes #4052
 2021-03-29 13:45:24 +02:00
Norman Feske
eb89b13327 depot: update recipe hashes 2021-03-24 11:32:53 +01:00
Norman Feske
b51c1a0fe3 News item for Sculpt 21.03 2021-03-24 11:32:53 +01:00
Norman Feske
f90cd542cb sculpt: documentation update 2021-03-24 11:32:53 +01:00
Christian Helmuth
dce272ba8f tool/xkb2ifcfg: update README to event_filter 2021-03-24 11:09:08 +01:00
Alexander Boettcher
141af733aa vbox5: remove priority bands from vbox5-* packages 2021-03-24 10:34:55 +01:00
Alexander Boettcher
945b4760ef sculpt: remove vfs.lib and init from common rules 
to support packages using own build versions and to detect packages missing
to specify dependency on init.
 2021-03-24 10:34:55 +01:00
Alexander Boettcher
53041f4cd8 usb_host: support to disable usb controller types 
The commits adds the same configuration values as supported up to now by
the monolithic usb driver. In contrast to the original, by default all drivers
are started. Disabling a controller type is used by Sculpt, e.g. for OHCI if
running Sculpt inside Virtualbox.
 2021-03-23 11:50:37 +01:00
Alexander Boettcher
521f61b9e0 usb_hid: disable wacom driver 
Issue #3997
 2021-03-23 11:32:24 +01:00
Norman Feske
ca50a41d28 depot: update recipe hashes 2021-03-23 11:28:53 +01:00
Norman Feske
b29f1497bf depot_query: sort depot-user scan results 
Related to issue #4054
 2021-03-23 11:28:21 +01:00
Norman Feske
ca5522d4d9 sculpt: move chargen files to config/keyboard/ 
The addition of further keyboard layouts would otherwise pollute the
config/ directory too much.

Caution: When updating Sculpt OS, don't reuse your existing
config/event_filter file unmodified! You need to replace the chargen
includes, pointing to the new location, e.g.,

  <include rom="en_us.chargen"/>
  <include rom="special.chargen"/>

must be changed to

  <include rom="keyboard/en_us"/>
  <include rom="keyboard/special"/>

Issue #4055
 2021-03-23 11:28:21 +01:00
Norman Feske
36ef41626a sculpt: keyboard-layout selection dialog 
This patch extends the settings dialog with the ability to select the
keyboard layout between the options that are included in the sculpt
image. The manual configuration is of course still possible by editing
the /config/event_filter directly.

If both the fonts configuration and the event-filter configuration are
managed manually, the settings button and window are not displayed.

Fixes #4055
 2021-03-23 11:28:21 +01:00
Norman Feske
e9ac14ed49 fs_query: report dir content in alphabetic order 
By sorting the reported output, all consumers of the reports become able
to rely on a deterministic order. For example, the file browser of
Sculpt OS, the launcher menu items, and the depot-selection items will
appear in a predictable order.

Fixes #4054
 2021-03-23 11:28:20 +01:00
Norman Feske
8f1db47c26 sculpt: screen capturing and event injection 
This patch adds session-routing options to connect components to the
capture/event interfaces of the global nitpicker GUI server or the
leitzentrale GUI server. It thereby enables the implementation of screen
capturing components, remote management tools, or virtual keyboards.

Since those services are very powerful, they are subsumed under the
"hardware" category of the component graph as opposed to the GUI
category, which guarantees the separation of clients.

Fixes #4053
 2021-03-23 11:28:20 +01:00
Norman Feske
d2fc834bfa os: depot recipe for pkg/test-capture 
This package allows for experimenting with the capture-session
interface.
 2021-03-18 17:36:35 +01:00
Christian Prochaska
3d432331b9 libc: mmap: use VFS handle as reference for shared mappings 
Fixes #4050
 2021-03-17 16:30:41 +01:00
Christian Prochaska
446df00d0d libc: support 'RLIMIT_DATA' resource in 'getrlimit()' 
Fixes #4049
 2021-03-16 17:45:40 +01:00
Tomasz Gajewski
2f0898d2a9 dde_linux: fixes in makefiles 
Issue #3972
 2021-03-16 11:25:25 +01:00
Christian Prochaska
9a0217f21a imx8: use 'snps,dis_u2_susphy_quirk' 
Fixes #4048
 2021-03-16 11:19:32 +01:00
Christian Prochaska
0cfafa1c8f usb_host_drv: support command timeouts and bool properties 
Fixes #4047
 2021-03-16 11:19:17 +01:00
Norman Feske
2c85e48a0d Revert "run: tweak CRLF translation in log scripts" 
The substitution eats carriage-return characters in some cases such as
the Pine-A64-LTS board connected via an TTL-232R-RPi cable.
 2021-03-12 15:08:11 +01:00
Norman Feske
15780a657c sd_card_bench.run: allow timer to reach IO_MEM 
This is needed on seL4 on i.MX6 where the user-level timer driver
accesses memory-mapped I/O registers directly.
 2021-03-12 12:09:17 +01:00
Norman Feske
5c5b56d1e0 depot: update recipe hashes 2021-03-12 12:08:24 +01:00
Norman Feske
00900d82b5 sculpt: tweak window motions in leitzentrale 2021-03-12 12:01:39 +01:00
Alexander Boettcher
18182b11da base: exit loop on cpu session close 
if session can't be closed immediately, e.g. if the service is provided by
a child (cpu_balancer) and not by the parent.

Issue #4029
 2021-03-12 12:01:39 +01:00
Norman Feske
8eb514d6b5 sculpt: interactive priority configuration 
This patch reworks the resource-assignment dialog to allow the user to
select the scheduling priority in addition to the CPU affinity.

Fixes #4045
 2021-03-12 12:01:39 +01:00
Alexander Boettcher
8a8de970a5 sculpt: add system clock package to index 
Issue #4046
 2021-03-12 12:01:39 +01:00
Sid Hussmann
cae3e447d6 vfs/pipe: fix pagefault during congested pipes 
Issue #4030
 2021-03-12 12:01:39 +01:00
Alexander Boettcher
f98d10a3f3 depot_deploy: support packages providing Rtc 
Fixes #4046
 2021-03-12 12:01:39 +01:00
Alexander Boettcher
521663c6de system_rtc: add standalone runtime package 
Package contains no rtc driver, which must be provided separately.

Issue #4046
 2021-03-12 12:01:39 +01:00
Alexander Boettcher
9b5bedefc7 expanding_pd_session: remove cap warning message 
which are caused often when a component runs a bit longer and has dynamic
resource usage, e.g. running Genode's tool chain.
 2021-03-12 12:01:39 +01:00
Norman Feske
8ecc258d3f sculpt: introduce runtime priorities 
This patch adds 4 priority levels to the runtime subsystem. The highest
priority is used for components that are critical for the operation of
Sculpt, in particular the Leitzentrale GUI. All regularly deployed
components are assigned the lowest priority by default.

With priorities available in the runtime subsystem, this patch flattens
the priority levels at the top-level init to only two levels and
overlays the priority bands of the drivers, leitzentrale, and runtime
subsystems into one priority band. This has three benenfits:

- This change prevents the starvation of the Leitzentrale GUI from a
  spinning high-priority driver (issue #3997).

- The change will also ease the hosting of latency-critical components
  in the runtime subsystem that are prioritized higher than regular
  components, the storage stack, and the network stack.

- The Leitzentrale GUI remains always perfectly responsive regardless
  of the workloads deployed from packages. In the previous version,
  the runtime graph was sometimes stuttering on high system load.

Issue #4045
 2021-03-12 12:01:39 +01:00
Sebastian Sumpf
7bbd050f25 riscv: adjust timeout run script to RISC-V 
* Limit required memory for RISC-V to 15 MB per buffer

issue #4021
 2021-03-12 12:01:39 +01:00
Norman Feske
7e7c10e66c Add sandbox test to depot autopilot 
Issue #4044
 2021-03-12 12:01:39 +01:00
Norman Feske
d5d3b3c3a4 test.run: support binaries other than init 
This change enables the use of runtime pkgs that specify the test binary
directly as attribute in the <runtime> node.
 2021-03-12 12:01:39 +01:00
Norman Feske
2baa283d87 vfs_lwip: reduce repeated warnings 
This patch avoids the repeated warning "read blocked until lwIP
interface is ready" by printing the message only once. Otherwise,
the log is flooded with those warnings when falkon web browser
is started on Sculpt OS without network connectivity.
 2021-03-12 12:01:39 +01:00
Josef Söntgen
4a12b5c653 mixer: fix README with regard to boolean values 
Issue #4043.
 2021-03-12 12:01:39 +01:00
Josef Söntgen
ba6c4a664f mixer: use audible default settings 
In case there is no valid configuration the component used to mute
output by default. This, however, might lead to the assumption that
audio is not working at all. Instead set the master output volume to
50% and the per application volume to 100%.

Fixes #4043.
 2021-03-12 12:01:38 +01:00
Norman Feske
9093c293cb sandbox: destroy 'Session_state' of local services 
This patch adds the missing destruction of session-state objects of
local services when closing a session. Because of the missing
destruction, those session-state object remained part of the server
ID space. This becomes a problem once the backing store of the session
state object vanishes, that is when the client child gets removed from
the sandbox. Hence, the removal of a child with an open session to a
local service would lead to the corruption of the server ID space.

This patch adds the missing session.destroy() call.

Fixes #4044
 2021-03-12 12:01:38 +01:00
Norman Feske
935bb36fe4 base: fix child destruction while close requested 
This patch fixes a corner case where a child is destructed while a
asynchronous close request to a sibling server is still pending.

The child immediately discarded the session ID as the end of the
close-session processing, assuming that this ID is never to be needed
again. The session-state continues to exist to handle asynchrous close
protocol with the server.

However, if the child is destructed at this point (before the server
responded to the session request), the destruction of the child would
not cover the discharging of the session state because the session state
was no longer be part of the client's ID space. So once the asynchronous
close response from the server came in, the session state contained
stale information, in particular a stale closed_callback pointer.

The patch fixes the problem by deferring the discarding of the client ID
to the point where the session state is actually destructed. So the
session of a pending close response is covered by the child destructor.

Thanks to Pirmin Duss for reporting this issue along with a test
scenario for reproducing it!

Fixes #4039
 2021-03-12 12:01:38 +01:00
Norman Feske
755aed7cb2 dde_linux: don't spin when panicking 
Issue #3997
 2021-03-12 12:01:38 +01:00
Sebastian Sumpf
6223ae4413 riscv: Interrupt controller support 
- Enable the "platform-level interrupt controller" PLIC on base-hw
- The RISC-V specification offers only a register description, but no
  layout for the register set. This implies the layout is platform
  dependent, and therefore, implemented separately for Qemu

issue #4042
 2021-03-12 12:01:38 +01:00
Sebastian Sumpf
bebba3876e run boot_dir hw: build debug ELF binaries again 
- build debug version of core and bootstrap found under
  _var/run/<runscript>.core_ (.bootstrap). This is a regression when
  switching from object to archive library linkage of core.

fixes #4027
 2021-03-12 12:01:38 +01:00
Norman Feske
aa0a98bd43 base-foc/hw: avoid use of placement new operator 
Fixes #2106
 2021-03-12 12:01:38 +01:00
Josef Söntgen
42f3d2eccd qemu-usb: reset completion pointer 
Reset the completion pointer to prevent a double free in case the
packet is processed during ISOC IN request handling.

Fixes #4041.
 2021-03-12 12:01:38 +01:00
Josef Söntgen
c03534e355 boot_dir/nova: enable intel_hwp 
Instruct bender to enable Intel HWP on supported CPUs unconditionally
to gain more insight by using the feature most prominently in Sculpt.

Fixes #4040.
 2021-03-12 12:01:38 +01:00
Norman Feske
1e0d843464 libc: avoid using placement new operator 
Issue #2106
 2021-03-12 12:01:38 +01:00
Norman Feske
8c7d34ff21 rpi_platform_drv: remove placement new operator 
Issue #2106
 2021-03-12 12:01:38 +01:00
Norman Feske
d6a312f438 tool/builddir: pass --ccache to run tool 
With this patch, the build directory's CCACHE configuration is applied
to the building of depot archives when using the --depot-auto-update
feature.

Note that this is effective only when resetting the hash files before
building so that each depot-auto-update results in the same-named
versions. Otherwise, the include paths (that include the version name as
part of the path) make cache hits impossible.

Issue #4004
 2021-03-12 12:01:38 +01:00
Norman Feske
6544cca320 tool/run: add --ccache argument 
If this argument is combined with --depot-auto-update, the run tool
enables the use of ccache when invoking the depot/create tool.

Issue #4004
 2021-03-12 12:01:38 +01:00
Norman Feske
3d0ed5992d tool/depot/create: add CCACHE=1 option 
This commit complements issue #4004 with support for the depot/create
tool. If CCACHE=1 is specified at the command line, the build directory
for building a binary archive is configured with 'CCACHE := yes'.
 2021-03-12 12:01:38 +01:00
Norman Feske
366fda0e47 sculpt: release part_block after device discovery 
This patch triggers the immediate removal of part_block once the
discovery of a fresh inserted USB stick has completed. Without the
patch, part_block was retained until the runtime was reconfigured the
next time for other reasons (e.g., toggling an inspect view). Until the
next reconfituration, part_block tended to stand in the way of directly
assigning the USB device to a VM.
 2021-03-12 12:01:38 +01:00
Christian Helmuth
7ce1f8e92d Enable en_us chargen in drivers_interactive-imx8q_evk 
Issue #4038
 2021-03-12 12:01:37 +01:00
Christian Helmuth
6e9843bd05 test-libc_fifo_pipe: more cap quota required for sel4/x86_64 
Issue #4030
 2021-03-12 12:01:37 +01:00
Norman Feske
2ff252360d sculpt: avoid flickering of leitzentrale 
This patch increases the RAM quota of the nitpicker instance for the
leitzentrale to make it suffice for the buffering of content during
resize operations. This fixes a flickering artifact when having an
inspect view open while entering a WLAN passphrase. When the connect
button appears or disappears (depending on the number of written
characters), some parts of the inspect terminal would flicker sometimes.
 2021-03-12 12:01:37 +01:00
Norman Feske
9de61e7014 base: detect truncated session args 
This patch reflects the truncation of session arguments (e.g., by very
long session labels) by denying the session request.

Fixes #4037
 2021-03-12 12:01:37 +01:00
Norman Feske
6712eac7e6 platform driver: use same label for all device PDs 
Fixes #4037
 2021-03-12 12:01:37 +01:00
Martin Stein
25a212aa24 nic_router: fix uncaught bit alloc exception 
* catch every possible path of the Bit_allocator_dynamic::Out_of_indices
  exception
* add unconditional log output in the new "catch" directives, so, we will be
  able to debug the problem a bit more in detail next time

Fixes #4036
 2021-03-12 12:01:37 +01:00
Alexander Boettcher
89ffc48576 depot_deploy: integrate shim and adapt sculpt 
Issue #3837
Issue #4029
 2021-03-12 12:01:37 +01:00
Norman Feske
9a5bc9caf0 cpu_balancer.run: use shim for hosting cpu_burner 
This way, the redirected env sessions for the CPU burner are no longer
called directly by init. Init stays independent from the behavior of the
CPU balancer.

Issue #3837
Issue #4029
 2021-03-12 12:01:37 +01:00
Norman Feske
c0a7565c21 os: shim helper for safe routing of env sessions 
This shim component can be used in case where env sessions of child
(i.e., child's PD session) must be routed to another child of init.

Without the shim, init would directly need to interact with these
sessions and would thereby make itself dependent of the server's
behavior. RPC calls to a server hosted as a child lead to all kinds of
problems such as livelock situations, and putting the robustness of init
at the whim of its child.

With the shim, init merely needs to bootstrap the shim component by
routing the shim's env sessions to core as usual. The server is only
used for the sessions for the actual application hosted atop the shim.

Issue #3837
Issue #4029
 2021-03-12 12:01:37 +01:00
Alexander Boettcher
a02ec07e49 sculpt_manager: add PD as service 
Issue #3837
 2021-03-12 12:01:37 +01:00
Alexander Boettcher
1f29055927 depot_deploy: support providing CPU and PD service 
Issue #3837
 2021-03-12 12:01:37 +01:00
Alexander Boettcher
7af276ac81 cpu_balancer: add support to provide PD service 
Issue #3837
 2021-03-12 12:01:37 +01:00
Tomasz Gajewski
de62582905 prepare_port: simple cache for downloads 
Fixes #4028
 2021-03-12 12:01:37 +01:00
Norman Feske
ba567f4ba8 Avoid double slashes in select_from_repositories 
Issue #4019
 2021-03-12 12:01:37 +01:00
Alexander Boettcher
ee0ed273e6 vm_stress: increase cap quota for 3 & 4 vcpu test 
due to changes by

   base: refactor VM/vCPU API

   Issue #3554
 2021-03-12 12:01:37 +01:00
Norman Feske
e1bb0e8e15 usb_host: refine class heuristics on device report 
This patch is an interim fix for using HID devices that offer a HID
interface as not the first interface. It also supplements the
interface classes as supplemental information to the USB-devices
report.

Fixes #4035
 2021-03-12 12:01:37 +01:00
Norman Feske
2e4ccc1459 usb_host: reflect ENOSPC as STALL 
Issue #4035
 2021-03-12 12:01:37 +01:00
Norman Feske
80522fadf6 usb_hid: don't touch non-HID interfaces 
Issue #4035
 2021-03-12 12:01:36 +01:00
Norman Feske
2ce4a3b400 sculpt: version 21.03 2021-03-12 12:01:36 +01:00
Norman Feske
c68443e2eb base-hw: support packaging of multi-arch boards 
With this patch, the board/<name>/arch property file can have multiple
lines where each line denotes an instruction-set architecture supported
for the board.

Issue #4019
Fixes #4034
 2021-03-12 12:01:36 +01:00
Stefan Kalkowski
9685a8b60d drivers_managed-imx8q_evk: add dummy_rtc_drv 
Ref #4033
 2021-03-12 12:01:36 +01:00
Tomasz Gajewski
23e3079f46 tool/run: option to copy tftp served files 
Fixes #3386
 2021-03-12 12:01:36 +01:00
Stefan Kalkowski
10b56afff0 rtc_drv: add dummy driver 
* Add dummy driver implementation and depot package
* Extend configuration to allow setting an initial RTC value

Fix #4033
 2021-03-12 12:01:36 +01:00
Sebastian Sumpf
d4b58b689c base-hw: fix RISC-V duration calculation 
Simplify calculation of Timer::_duration, the old implementation caused
the time running backwards sometimes. This makes
'nic_router_dhcp_*' and 'event_filter' run scripts succeed.

issue #4021
 2021-03-12 12:01:36 +01:00
Norman Feske
1826ff8a59 sculpt: constrain initial log-window size 
This prevents the log terminal from adjusting itself to the whole screen
size before the leitzentrale window layout is applied. This intermediate
size can otherwise exhaust the 8 MiB of RAM quota of the log terminal.

Issue #3970
 2021-03-12 12:01:31 +01:00
Christian Helmuth
86ad4ed17f version: 21.02 2021-02-25 11:26:18 +01:00
Norman Feske
1d1b5b88c5 News item for version 21.02 2021-02-25 11:24:28 +01:00
Norman Feske
4f1a3a8000 Release notes for version 21.02 2021-02-25 11:24:28 +01:00
Christian Helmuth
0afd3db894 depot: update recipe hashes 2021-02-25 11:24:23 +01:00
Christian Helmuth
cbe81d35b9 autopilot: disable run script not supported by riscv 
Issue #4021
 2021-02-25 11:22:59 +01:00
Josef Söntgen
1d551bd967 vbox6: add QEMU xHCI device-model 2021-02-25 11:22:59 +01:00
Alexander Boettcher
812c3599de vbox6: add network backend using the nic session 
as used in vbox5 already.
 2021-02-25 11:22:59 +01:00
Christian Helmuth
20caac5f3b Port of VirtualBox 6.1.14 2021-02-25 11:22:59 +01:00
Christian Helmuth
a47b374905 vim.run: import clipboard archive only once 2021-02-23 13:22:50 +01:00
Christian Helmuth
7a3dc68f34 vbox5: fix typo 2021-02-23 13:22:48 +01:00
Christian Prochaska
dd92ab126b libc: test TLS support in pthread test 
Fixes #4023
 2021-02-23 12:19:25 +01:00
Sebastian Sumpf
f68e655312 mk: link core to library archive instead of .o file 
Do not link base and core libraries into on large relocatable .o file,
which is linked later to core - causing long link times. Create an
independent library archive out of the base and core libraries that can
be linked faster.

issue #4027
 2021-02-23 12:17:04 +01:00
Norman Feske
64165d829e depot: update recipe hashes 2021-02-23 12:07:18 +01:00
Norman Feske
c2feba065f Reduce pointers to doc/ in top-level README 
Avoid referencing the content of doc/ for topics that are covered by the
Genode Foundations book.
 2021-02-23 12:07:18 +01:00
Christian Helmuth
219809ffed base: refactor VM/vCPU API 
Issue #3554
 2021-02-23 12:07:18 +01:00
Sid Hussmann
6e8728f2d3 vfs/pipe: use local signal handling 
This commit improves the performance of the pipe plugin by using local
signal handling and avoiding sending a signal during write when pipe
buffer is full.

Issue #3583
 2021-02-23 12:07:17 +01:00
Stefan Kalkowski
90d9470dfd vmm: add virtio block device model 
* Add new virtio device model
* Extend test run-script with vfat block test image
* Add vmm depot src recipe
* Use packages in test run-script

Fix #4025
 2021-02-23 12:07:17 +01:00
Stefan Kalkowski
2879aa003b vmm: cleanup virtio model, make it extensible 
* Introduce different index types for ring counters and descriptor arrays
  within the Virtio split queue implementation
* Be more accurate in reporting the queue number supported, and raise it
  to 512
* Introduce abstractions for mmio register access, where several values
  are stored at the same place, and selector registers exist.
* Turn Virtio_device into a template to define the Virio queue type,
  and its numbers per device model (needed for e.g., future block model)

Issue #4025
 2021-02-23 12:07:17 +01:00
Stefan Kalkowski
83c2309710 run: don't rebuild core/bootstrap when available 
Fix #4026
 2021-02-23 12:07:17 +01:00
Christian Prochaska
59459e60e7 libc: pthread TLS optimizations 
Fixes #4024
 2021-02-23 12:07:17 +01:00
Sid Hussmann
8d13121e84 vfs/pipe: refactor read/write synchronization 
All pipe-ends were notified at the same time which leads to dead-locks.
This commit mitigates this by having a signal handler for each pipe and
each pipe-end respectively.

Issue #3583
 2021-02-23 12:07:17 +01:00
Sid Hussmann
3ff0efd627 vfs/pipe: add fifo feature to pipe plugin 
The vfs pipe plugin can now be used as named pipe which anables data
transfer via file handles from one component to another. E.g. if one
would like to send data from component A to stdin of a libc component B,
one can do so by simply writing to that fifo file.

Issue #3583
 2021-02-23 12:07:16 +01:00
Norman Feske
10605a6903 Spelling fix 2021-02-23 12:07:16 +01:00
Martin Stein
6937eb7d94 drivers_nic-rpi: use usb_net and usb_host 
Ref #3961
 2021-02-23 12:07:16 +01:00
Martin Stein
a462a8e741 usb_host: compile errors with DEBUG_LINUX_PRINTK 
Ref #3961
 2021-02-23 12:07:16 +01:00
Martin Stein
3485282909 drivers/platform/rpi: remove dead code 
Ref #3961
 2021-02-23 12:07:16 +01:00
Sebastian Sumpf
b6d20b4742 autopilot: disable run script not supported by riscv 
- libc and nic driver are currently not available on RISC-V

issue #4021
 2021-02-23 12:07:16 +01:00
Sebastian Sumpf
7318ca6084 depot: add 'riscv' to depot spec 
issue #4021
 2021-02-23 12:07:16 +01:00
Sebastian Sumpf
ca777fe93f riscv: add riscv_qemu base-hw recipe 
issue #4021
 2021-02-23 12:07:16 +01:00
Sebastian Sumpf
ccd9ba4161 riscv: move CPU header from base-hw to base 
issue #4021
 2021-02-23 12:07:16 +01:00
Christian Helmuth
954f03257d Update curl to version 7.70.0 
Issue #3773
 2021-02-23 12:07:16 +01:00
Pirmin Duss
190b4784c5 openssl: do not depend on ARM CPU probing 
With this commit libcrypto does not use ARM NEON extension as long as
SPECS includes "neon". arm_v7a does declare "neon" per default while
arm_v8a does.

Issue #3773
 2021-02-23 12:07:16 +01:00
Christian Helmuth
f23e302475 Update OpenSSL to version 1.1.1i 
Note, OpenSSL now comes as one combined depot archive *openssl* that
replaces the former *libssl* and *libcrypto* archives. The libraries are
still separate binaries for compatibility with legacy software.

Issue #3773
 2021-02-23 12:07:16 +01:00
Sebastian Sumpf
f5cd12dcf9 base/base-hw: unify startup library builds 
- Use 'select_from_repositories' for crt0.s of startup library
- Move RISC-V crt0.s to base (where the other archs are located)

issue #4019
 2021-02-23 12:07:15 +01:00
Martin Stein
ce31c90bc3 run/netperf.inc: fix problem with reused ports 
It seems that with the nigthly autopilot the data port (that is choosen by the
client) may stay some time (minutes) allocated after the a test run. This
causes successive tests to fail when trying to reuse the port.

* Use a unique data port for each target platform in the range of 18000-18099.
* Add documentation on how to prepare for running the test on a Linux target.
* Remove Qemu-specific code paths as Qemu isn't supported anymore by the script
  since the introduction of the Uplink session (running with a NIC router on
  Qemu was never supported and now we always need a NIC router).

Ref #3961
 2021-02-23 12:02:46 +01:00
Martin Stein
f9c258a372 lwip_lx.run: typo in doc 
Ref #3961
 2021-02-23 12:02:45 +01:00
Norman Feske
048a4625c5 base: remove stale spec/fpu_vfpv3.mk file 2021-02-23 12:02:45 +01:00
Norman Feske
db3f86d603 base-hw: avoid double slashes in path names 2021-02-23 12:02:45 +01:00
Norman Feske
fa68325a57 base-hw: streamline extraction of depot archives 
This patch simplifies the packaging of the base-hw kernel for a given
board, in particular when the board support is hosted as an external
repository such as genode-world. Regardless of where the board-support
is hosted, the content.mk file becomes as simple as:

  include $(GENODE_DIR)/repos/base-hw/recipes/src/base-hw_content.inc

The board name is automatically inferred from the path of the src
recipe. The architecture is determined from board/<name>/arch files,
following the pattern of the image_link_address. The attempt to build a
base-hw-<board> binary archive for the wrong architecture is now
gracefully handled by skipping all targets (using the REQUIRES
mechanism).

Besides the improved convenience, the patch results in depot archives
that are much closer tailored to the actual board by omitting files for
architectures that are not used by the board. E.g., the src/base-hw-pc
archive does not contain any ARM-related content.

The patch also restores the package builds for core/bootstrap object
files as a follow-up fix of "base: remove SPEC variables of boards",
which happened to left 'BOARD' undefined in the src archives.

Fixes #4019
 2021-02-23 12:02:45 +01:00
Josef Söntgen
1b77cb3832 qemu-usb: add ring-buffer for ISOCH OUT transfers 
Be more in line with the original host implementation in Qemu and
buffer transfers. Having transfers in-flight helps to smoothen playback
in case other components utilize the CPU.

Issue #4018.
 2021-02-23 12:02:44 +01:00
Josef Söntgen
19d9409a34 qemu-usb: adapt ISOCH IN handling for audio 
Linux as well as Windows guests want queue to a varying number of
transfers, where each transfer only covers one transaction (iso frame).
The best results were obtained by following that behaviour rather
than queuing multiple iso frames per requests (like is done with OUT
transfers).

The number of requests queued is increased to 32 while the number of
packets per reques is decreased to 1.

Issue #4018.
 2021-02-23 12:02:44 +01:00
Josef Söntgen
9918a8f88d qemu-usb: update port to 5.2.0 
The contrib code is updated from 2.4.1 to version 5.2.0 and the used
device-model is changed to QEMU xHCI. Due to this change older guests
OSes, namely Windows 7, that relied on the NEC xHCI device-model will
not work anymore.

The 'Qemu::Controller' interface was extended by an 'info' method, which
returns the vendor and product ID. This allows for removing the
hard-coded values in the VirtualBox glue code.

Issue #4018.
 2021-02-23 12:02:44 +01:00
Josef Söntgen
a6f0b05834 usb_host: properly handle ISOC read requests 
* Update the 'packet_size' information with the actual length for
  each isoc frame to be able to handle short reads at the client side.

* Copy the whole transfer buffer because the host controller stores
  the data at the original offsets, i.e., the buffer is not densely
  packed.

Fixes #4018.
 2021-02-23 12:02:44 +01:00
Josef Söntgen
b51ae104c2 qemu-usb: use bounce buffer to access DMA memory 
The former implemention assumed that the guest physical memory is
mapped continously. This, however, is not true. Writing larger
files to an USB stick with a Windows 10 guest would therefore lead
to data corruption.

The current implementation uses a bounce buffer to copy the data
to and from the guest physical memory and leaves dealing with the
memory mappings entirely up to the VMM.

Fixes #4017.
 2021-02-23 12:02:44 +01:00
Martin Stein
23620942bf usb_net: allow calling netif_stop_queue dummy 
When the usb_net_drv was introduced in ports/run/netperf.inc, the
netperf_lxip_usb test on x86_64/hw/pc triggered the calling of the
netif_stop_queue dummy at
contrib/<DDE_LINUX>/src/drivers/usb_net/drivers/net/usb/usbnet.c:1464. As
netif_start_queue was also a dummy and allowed to be called, we tried allowing
calls to the netif_stop_queue dummy as well which fixed the
netperf_lxip_usb test on x86_64/hw/pc.

Ref #3961
 2021-02-23 12:02:44 +01:00
Norman Feske
a99f6a81b6 libc: clone 'environ' pointer value after fork 
The 'environ' pointer is a global variable that must be considered as
part of the application state. It must be copied from the parent to the
child process during fork. Otherwise, a child returning from fork is
unable to access environment variables before invoking execve. The
actual environment variables and their values are already captured
because they reside at the application heap, which is cloned from the
parent. So the copied 'environ' pointer refers to valid data.

Fixes #4015
 2021-02-23 12:02:44 +01:00
Sebastian Sumpf
fd0e6685fc riscv: Use Qemu and update to ISA v1.10 
- remove Spike/BBL support in favour of Qemu (>=4.2.1)
- add 'riscv_qemu' board, remove 'spike' board'
- update to privileged ISA v1.10 (from v1.9.1)
- use direct system calls for privileged core threads (they call into
  the kernel and don't use mode changing system calls, i.e. 'ecall',
  semantics)
- use 'OpenSBI' semtantics for SBI calls (to machine mode) instead of
  BBL

issue #4012
 2021-02-23 12:02:44 +01:00
Alexander Boettcher
18e282ab8a allocator_avl: avoid false dangling warnings 
By first removing unused ranges, implicitly meta data allocations are freed
up. This leads to more unused slab blocks and freed up meta data allocations
in the avl tree.

Issue #4014
 2021-02-23 12:02:44 +01:00
Alexander Boettcher
1e84b46c3f slab: add option to free empty blocks 
explictly by a method

Issue #4014
 2021-02-23 12:02:44 +01:00
Martin Stein
19d0142e10 ports/recipes/src: usb_net_drv 
Ref #3961
 2021-02-23 12:02:44 +01:00
Martin Stein
983a18d06e ports: rename *_usb30.run *_usb.run 
Ref #3961
 2021-02-23 12:02:43 +01:00
Christian Prochaska
f654e6f02d usb_host_drv: dwc_otg stability improvements 
- improve NYET error handling
- use split interrupt transaction hack
- clear TT buffers in error situations

Issue #3247
 2021-02-23 12:02:43 +01:00
Sebastian Sumpf
cb2e27f8e4 riscv: Fix link errors for Genode nightly 
- add .got filter rule for trace policies, otherwise ld raises a
  segmentation fault
- force soft-float for binary sources (SRC_BIN)

issue #4012
 2021-02-23 12:02:43 +01:00
Alexander Boettcher
c58acd0b2b allocator_avl: avoid memory leakage 
Fixes #4013
 2021-02-23 12:02:43 +01:00
Alexander Boettcher
26506673c4 seoul-auto: limit to one CPU on Qemu 
Issue #3965
 2021-02-23 12:02:43 +01:00
Norman Feske
df38140ed6 base-pistachio: coding style 2021-02-23 12:02:43 +01:00
Norman Feske
9633a0a524 wm.run: increase RAM quota for wm_backdrop 
Overcomes a startup issue of the wm_backdrop component on base-linux:

[init] child "wm_backdrop" requests resources: ram_quota=10855936
 2021-02-23 12:02:43 +01:00
Norman Feske
7d568247e3 wm: fix deadlock in destroy-view handling 
The deadlock reported in #3236 could be reproduced via the wm.run script
and the modification of test/nitpicker in commit "nitpicker: fix destroy
with invalid handle" by clicking on the testnit entry of the launchpad.

This patch fixes the deadlock by releasing the locked pointer early in
the destruction path, which is legitimate as the wm is single-threaded.

Fixes #3236
 2021-02-23 12:02:43 +01:00
Norman Feske
b5fb37ddee nitpicker: invalidate dangling hovered ptr 
This patch adds the missing invalidation of the _hovered pointer to
User_state::forget, which is required to cover the situation where the
owner of the hovered view vanishes.

Fixes #4011
 2021-02-23 12:02:43 +01:00
Alexander Boettcher
d29b843a0f dde_ipxe: improve throughput for Realtek devices 
By increasing the ring descriptor count the throughput increased from
few kB/s to the range of MB/s.

Issue #3939
 2021-02-23 12:02:43 +01:00
Alexander Boettcher
8958c769ab dde_ipxe: poll once again on irq without rx packets 
This quirk is required for Realtek devices.

Fixes #3939
 2021-02-23 12:02:43 +01:00
Christian Helmuth
210f5073e3 platform_drv.inc: check board=pc not spec=x86 
Fixes run scripts for Linux/x86.
 2021-02-23 12:02:43 +01:00
Martin Stein
ef88d05f2b os: api/net archive 
Ref #3961
 2021-02-23 12:02:43 +01:00
Alexander Boettcher
d6a5a66623 vbox5: enable 64bit SVM support 
Fixes #3965
 2021-02-23 12:02:42 +01:00
Alexander Boettcher
d186e4361e Measure TSC variance of CPUs in test-smp 
Issue #3965
 2021-02-23 12:02:42 +01:00
Emery Hemingway
2acfacb639 genode_dyn.ld: do not emit PHDR segment 
Recent versions of binutils check that PHDR segments are covered
by a LOAD segment. In this case the unloaded PHDR segment can be
ommited.

Fix #4003
 2021-02-23 12:02:42 +01:00
Alexander Boettcher
696d8f030f libgmp: avoid undefined gmpn_*_2 symbols (x86_64) 
__gmpn_addmul_2
__gmpn_mul_2

Fixes #4001
 2021-02-23 12:02:42 +01:00
Alexander Boettcher
e3233a4824 ldso: turn deadlock message into Linker message 
Issue #3991

[init -> ...] Error: LD: jump slot relocation failed for symbol: 'XYZ'
[init -> ...] Error: deadlock ahead, mutex=0x161860, return ip=0xeb870

turns into:

[init -> ...] Error: LD: jump slot relocation failed for symbol: 'XYZ'
[init -> ...] Error: Uncaught exception of type 'Linker::Not_found'
[init -> ...] Warning: abort called - thread: ep

Fixes #4002
 2021-02-23 12:02:42 +01:00
Christian Prochaska
5c5d16f524 ports: add 'vbox5_genode_usb_hid_raw' run script 
Fixes #4009
 2021-02-23 12:02:42 +01:00
Christian Prochaska
c16611dff2 usb_hid_raw.run: add "Timer" service route 
Fixes #4008
 2021-02-23 12:02:42 +01:00
Christian Prochaska
33406940f3 usb_hid_drv: stability improvements 
Fixes #4007
 2021-02-23 12:02:42 +01:00
Christian Prochaska
e1698cf200 dde_linux: fix possible memory leak in wait queue implementation 
Fixes #4006
 2021-02-23 12:02:42 +01:00
Christian Prochaska
2670ae399b usb_host_drv: improve error handling 
Fixes #4005
 2021-02-23 12:02:42 +01:00
Christian Helmuth
91a7fb1da7 cxx: localize operator new(unsigned long) in supc++.o 
Otherwise, base code in ld.lib.so may accidentally use 'new' without
passing an allocator.
 2021-02-23 12:02:41 +01:00
Martin Stein
a9c4ebc9e9 drivers/nic/mode: exception on malformed values 
Ref #3961
 2021-02-23 12:02:41 +01:00
Norman Feske
e3783b00bb coding_style.txt: initializers, control flow 2021-02-23 12:02:41 +01:00
Norman Feske
493924a35e base-hw: coding style 
Improve consistency with the other base repositories, in particular

- Indentation of class initializers
- Vertical whitespace around control-flow statements
- Preferably place control-flow statements (return, break, continue) at
  beginning of a line
- Placing the opening brace of a namespace at the end of line
- Placing the opening brace of a class at a new line
- Removing superfluous braces around single statements
- Two empty lines between methods/functions in implementation files
 2021-02-23 12:02:41 +01:00
Norman Feske
cbae9bc1c8 Add ccache support to build system 
This patch simplifies the use of ccache with the build system. Up until
now, each developer had to set up the ccache hooks manually, adjust the
PATH variable, and customize the etc/tools.conf in each build directory.
With the patch, ccache can be enabled by un-commenting a single line in
the etc/build.conf file.

Fixes #4004
 2021-02-23 12:02:41 +01:00
Norman Feske
8cc2662aac Remove support for the Muen separation kernel 
Fixes #3995
Fixes #3994
 2021-02-23 11:55:44 +01:00
Christian Helmuth
af9ab9190b sd_card_bench.run: add missing braces 
This is a follow-up fix for "base: remove SPEC variables of boards (fix #3971)".
 2021-02-23 11:53:59 +01:00
Christian Helmuth
14db22c77c nic_router.run: disable test 2 UDP 
Issue #3992 is still pending but has low priority, so disable the test
to reduce the noise and increase the chance to spot independent
regressions.
 2021-02-23 11:53:42 +01:00
Martin Stein
691be92046 Don't use the NIC server mode of NIC drivers 
Issue #3961
 2021-02-23 11:53:41 +01:00
Christian Helmuth
9f3c5d92b3 base: cleanup Noncopyable declarations/coding style 2021-02-23 11:51:57 +01:00
Josef Söntgen
36b55e065a usb_host: check ep pointer before using it 
The Usb session allows for submitting packets even when the interface
in question is not yet enabled. Enabling an interface will configure
the udev members properly and is normally done implicitly during
processing of an 'ALT_SETTING' packet.

In case the interface was not enabled this leads to a page-fault in
the USB host-controller driver as 'ep' is NULL.

Fixes #3999.
 2021-02-23 11:51:57 +01:00
Alexander Boettcher
6789ce8b83 platform_drv(x86): support dynamic reconfiguration 
Fixes #3989
 2021-02-23 11:51:39 +01:00
Norman Feske
a981fb864c depot: update recipe hashes 2021-01-25 14:00:43 +01:00
Norman Feske
c4cf9b6e6d base-hw: introduce board-property directories 
This patch moves the bootstrap-link-address information from the
tool/run/boot_dir/hw file to board-specific property files that can be
accessed by using the board as key. This eliminates the need to
customize boot_dir/hw when hosting board-support in an external
repository.

Fixes #3998
 2021-01-25 14:00:43 +01:00
Norman Feske
4bc9b9a2ef tool/run: coding style 2021-01-25 14:00:43 +01:00
Piotr Tworek
ad4211ae2c os: Make Redistributor class inherit Genode::Interface. 
The Vmm::Gic::Gicd_banked::Redistributor is used with Constructible, but
does not have a virtual destructor even though it has virtual methods.
This prompts clang to issue the following warning:

  reconstructible.h:122:4: warning: destructor called on non-final
  'Vmm::Gic::Gicd_banked::Redistributor' that has virtual functions
  but non-virtual destructor [-Wdelete-non-abstract-non-virtual-dtor]

Fix this by inheriting Genode::Interface.

Issue #3984
 2021-01-25 14:00:43 +01:00
Norman Feske
ff28ed0f8c base: avoid superfluous postprocessing of ldso 
Even though the binary patching of ldso must be performed only once,
this postprocessing step was executed on each run because the
postprocess.tag rule is phony (the tag file is never created).
This patch removes this phony behavior by creating the tag file.

Issue #3974
 2021-01-25 14:00:43 +01:00
Tomasz Gajewski
693a4d78dd lib.mk: avoid checking abi on every build 
Fixes #3974
 2021-01-25 14:00:43 +01:00
Piotr Tworek
8f6b934caa base-hw: Fully define _crt0_start_stack value on arm_64 
The _crt0_start_stack label points to a memory location containing the
size of the bootstrap stack. On AArch64 this should be an 8 byte value,
but the code only only defines half of those using asm .long statement.
The other half is expected to be 0, which is true when using GNU as.
This is not the case when using clang's integrated as however. Since
_crt0_stack_size is defined inside .text section clang uses 0xd503201f
value (aarch64 nop instruction) to fill the extra 4 bytes.

Fix this minor incompatibility by explicitly defining both halfs of
this 8 byte quantity.

Fixes #3987
 2021-01-25 14:00:43 +01:00
Piotr Tworek
384cf14bee libc: Remove unused variables from Libc::Kernel. 
The _dispatch_pending_io_signals and _original_suspended_callback member
variables are not used anywhere in the code. This prompts clang to produce
a warning message about it. Remove both variables to fix it.

Issue #3985
 2021-01-25 14:00:42 +01:00
Piotr Tworek
90b20b4daf nitpicker: Fix unsigned long to unsigned int cast. 
The "unsigned Nitpicker::Gui_session::layer()" function returns
~0UL. This prompts clang to produce a warning about implicit unsigned
long to unsigned int conversion. Fix it by returning ~0U instead of
~0UL.

Issue #3985
 2021-01-25 14:00:42 +01:00
Piotr Tworek
80318b9ae0 libc: Add semicolons after [[fallthrough]] 
GCC doesn't care, but clang complains if [[fallthrough]] is not followed
by a semicolon. Existing Genode code is also not consitent in this regard.
This patch adds the extra semicolons since it works in both GCC and
clang.

Issue #3985
 2021-01-25 14:00:42 +01:00
Piotr Tworek
fce5c249c2 gems: Drop unused variables from Window_list class 
No code in this class uses this private member variable. This prompts
clang to produce a warning message about it. Fix it by dropping the
variable.

Issue #3985
 2021-01-25 14:00:42 +01:00
Piotr Tworek
71abfb3b4f libc: Mark Libc::Vfs_plugin class as final. 
The class has final destructor, but is not itself final. This prompts
clang to produce the following warning message:

class with destructor marked 'final' cannot be inherited from [-Wfinal-dtor-non-final-class]
                ~Vfs_plugin() final { }
                              ^
vfs_plugin.h:39:13: note: mark 'Libc::Vfs_plugin' as 'final' to silence this warning

Issue #3985
 2021-01-25 14:00:42 +01:00
Piotr Tworek
395a9b5bf5 libc: Remove unused print function from select.cc. 
This static inline function is not used anywhere. GCC does not care,
but clang warns about this. Remove the function to allow the code to
compile cleanly with both clang and GCC.

Issue #3985
 2021-01-25 14:00:42 +01:00
Piotr Tworek
53081ac6b3 libc: Dont't use local_addr in try/catch block. 
The code in Libc::Cloned_malloc_heap_range initializes its local_addr
member variable by calling Region_map::attach_at. This function can
throw Region_conflict exception. The handler for this exception uses the
local_addr to produce an error message. Such error log is IHMO
useless, or even incorrect since the value of local_addr is undefined
in such case.

Its also worth noting that clang 12 produces the following warning for
this code:
  "cannot refer to a non-static member from the handler of a constructor
   function try block"

Issue #3985
 2021-01-25 14:00:42 +01:00
Piotr Tworek
c6d5b98227 decorator: Remove unused _topped_cnt variable. 
This private member variable is not used anywhere in the code. This
produces compilation warning when using clang instead of GCC. Drop the
unused variable.

Issue #3985
 2021-01-25 14:00:42 +01:00
Piotr Tworek
c402cc1045 window_layouter: Drop unused code. 
The State enum and _state private member variable are not used anywhere.
This produces an extra warning when building the code with clang instead
of GCC. Remove this dead code.

Issue #3985
 2021-01-25 14:00:42 +01:00
Piotr Tworek
1edac9730c scout: Instantiate templates in their namespace. 
When building the code with clang the following warning message is
prodiced:

  "explicit instantiation of 'Scout::Browser_window' must occur in namespace 'Scout'
   template class Browser_window<Genode::Pixel_rgb888>"

 This happens for several different types. This patch fixes the problem
 by instantiating all those templates using their explicit full name.

Issue #3985
 2021-01-25 14:00:14 +01:00
Piotr Tworek
d475015ada launchpad: Fix _spacer member initialization. 
The Scout::Spacer constructor requires two arguments. The initialization
of the type in Launchpad_window declaration does not specify them. The
variable is however initialized a second time in class constructor. This is
most likely why GCC accepts this code. Clang on the other hand
complains about it.

Fix this by properly initializing both _spacer and _docview only once at
declaration time.

Issue #3985
 2021-01-25 13:58:11 +01:00
Piotr Tworek
ffb931f8b1 nano3d: Fix header guard in sqrt.h 
The code fails to build with clang due to the following warning/error:

  error: '_INCLUDE__NANO3D__SQRT_H_' is used as a header guard here,
  followed by #define of a different macro [-Werror,-Wheader-guard]

Fix this by removing the extra underscore from this header guard
definition.

Issue #3985
 2021-01-25 13:58:11 +01:00
Piotr Tworek
b4d294f62e os: Drop unused Net::Arp_cache::_init member. 
Not used anywhere in the code. Clang 11 complains about this.

Issue #3984
 2021-01-25 13:58:11 +01:00
Piotr Tworek
052f678225 os: Add missing override keyword. 
The _device_specific_features() implements part of the Virtio_device
interface. Decorate it with override keyword to make this clear and
also satisfy clang which produces warning regarding this.

Issue #3984
 2021-01-25 13:58:11 +01:00
Piotr Tworek
3fdf323e6e os: Drop unused Net::Dhcp_client::_alloc member. 
Not used for anything so technically a dead code. Clang 11 complains
about this.

Issue #3984
 2021-01-25 13:58:11 +01:00
Piotr Tworek
05c36d67ce os: Fix clang constant warning in Port_allocator. 
Clang 11 produces the following warning when building port_allocator.cc:

port_allocator.cc:27:21: error: result of comparison of constant 65536 with
expression of type 'const Genode::uint16_t' (aka 'const unsigned short') is
always true [-Werror,-Wtautological-constant-out-of-range-compare]
               (port.value <  (unsigned)(Port_allocator::FIRST +

Basically the code compares Port::value (uint16_t) against a constant
65536 which is larger than UINT16_MAX (65535). This comparison will always
be true.

Issue #3984
 2021-01-25 13:58:11 +01:00
Piotr Tworek
ffc2a2f306 os: Disambiguate File_system in ram_file_system.h. 
According to clang there are two symbols which File_system can refer to:
1. namespace File_system from ram_fs/chunk.h.
2. Vfs::File_system class from  vfs/file_system.h.
Make it clear we refer to the File_system namespace in this case.

Issue #3984
 2021-01-25 13:58:11 +01:00
Piotr Tworek
fc089a1673 nic_dump: Remove unused Net::Interface::_alloc 
This private member variable is not used anywhere in the code. This
breaks builds using clang instead of GCC. Drop the variable.

Issue #3984
 2021-01-25 13:58:11 +01:00
Piotr Tworek
428de89f9a nic_dump: Disambiguate "Interface" in component.cc. 
Clang 12 complains that Interface can both refer to Genode::Interface
and Net::Interface in this case. Explicitly state the code refers to
the latter.

Issue #3984
 2021-01-25 13:58:11 +01:00
Piotr Tworek
30429a5228 nic_flood: Drop Dhcp_client::_alloc member. 
This private member variable is not used anywhere in the code. This
breaks builds using clang instead of GCC. Drop it.

Issue #3984
 2021-01-25 13:58:11 +01:00
Piotr Tworek
e6a9e06f62 base-linux: Drop parentheses from .cfi_undefined. 
According to GNU as manual the syntax of this directive is:
  .cfi_undefined register

The manual does not mention the register should be in parentheses.
This works in GNU as even when those are present, but unfortunately
clang integrated-as does not parse this correctly. Both GNU and
clang's integrated assembler work fine when the extra parentheses
are omitted.

Fixes #3986
 2021-01-25 13:58:10 +01:00
Piotr Tworek
8b172bf22e base: Explicitly state ELF segment flags 
The code in base-hw/src/bootstrap/platform.cc uses segment flags for
identification purposes. Based on this information the code decides
what to do with each segment. Unfortunately the linker script does
not actually ensure the flags for a specific named segment match
expectations. The code relies on implicit linker behaviour.
This implicit behaviour can vary between linkers. This breaks
arm_v7a base-hw builds linked with LLVM's lld linker. The segment
named "ro" ends up having writeable flag set when using LLD.

This patch ensures that all ELF segments in genode.ld have their
required perimssion flags set explicitly.

Fixes #3988
 2021-01-25 13:58:10 +01:00
Piotr Tworek
80e8cf99e2 base: Make Genode::List clang friendly. 
Clang is generally fine with Genode::List and compiles code using it
without emitting any warnings. There is however one exception. Clang
fails hard when building base-hw/src/core/kernel/object.cc.
This is due to a call to Genode::List::remove made from
Object_identity::invalidate function. The error message clang
produces is:

  list.h:96:33: error: 'Genode::List<Kernel::Object_identity_reference>::Element::_next'
  is not a member of class 'const Kernel::Object_identity'
                              _first = le->List::Element::_next;
                                           ~~~~~~~~~~~~~~~^

When we look at the declaration of the Kernel::Object class on which
the remove method is called. as expected it does inherit Genode::List:

using Object_identity_list
	= Genode::List<Kernel::Object_identity>;

class Kernel::Object : private Object_identity_list
{
...
}

Given the error message we see that List::Element should be resolved to
Genode::List<Kernel::Object_identity>::Element, and not
Genode::List<Kernel::Object_identity_reference>::Element. But how does
clang manage to figure out we're talking about Object_identity_refecence
list here? Well, I admit I don't know the exact steps it takes to arrive
at this conclusion, but it is not entirely wrong. If we take a look at
what Kernel::Object_identity is we'll see:

class Kernel::Object_identity
: public Object_identity_list::Element,
  public Kernel::Object_identity_reference_list
{
...
}

Where as one can guess Object_identity_reference_list is defined as:

using Object_identity_reference_list
	= Genode::List<Object_identity_reference>;

Long story short Kernel::Object has Genode::List of both Kernel::Object_identity
and Kernel::Object_identity_reference in its inheritance chain and clang
is not really sure to which of those the code refers to in
Genode::List::remove method by using List::Element::.

The fix for this is relatively simple, explicitly state the full type of
the base class the code intends to refer to. Replacing List::Element,
with List<LT>::Element makes the code buildable with both clang and GCC.

Fixes #3990
 2021-01-25 13:58:10 +01:00
Alexander Boettcher
9d239957bc libc: update seek state on opening O_APPEND fds 
If a fd is opened in append mode and just is to be used (so never written by
the parent component) for the to be forked child, the seek state was not
pointing to the end of the file.  The wrong seek value then was used in
File_descriptor_allocator::generate_info().

Issue #3991
 2021-01-25 13:58:10 +01:00
Alexander Boettcher
5fa91c573b libc: update O_APPEND fds on joined child 
The fds in O_APPEND state may have changed by the child. The local seek
state needs to be valid/current for the next to be forked child, which seek
state is set by File_descriptor_allocator::generate_info().

Issue #3991
 2021-01-25 13:58:10 +01:00
Norman Feske
9bfd812a88 tool/run: search repos for run-tool plugins 
This patch allows for supplementing the existing run-tool plugins with
additional plugins hosted inside a repository. E.g., it enables the run
tool to find world/tool/run/boot_dir/hw automatically.

Fixes #3993
 2021-01-25 13:58:10 +01:00
Norman Feske
1ccf8a280c base-hw: simplify board support in external repo 
Avoid use of REP_DIR in *.mk files to simplify the use of these files as
templates for a board hosted in a separate repository.

Use REP_INC_DIR for searching headers, thereby considering headers
hosted in an external repository.

Issue #3168
 2021-01-25 13:58:10 +01:00
Stefan Kalkowski
f034f560be create_builddir: remove deprecated platforms 
Fix #3996
 2021-01-25 13:58:10 +01:00
Martin Stein
f45aa85e9f nic_router.run: fix redundant IPv4 subnets 2021-01-25 13:58:10 +01:00
Norman Feske
84443d6548 os: div_zero trace policy 
This trace policy can be used as a fault-injection mechanism.
Once installed, the next time the traced thread passes a trace point
(e.g., RPC call, RPC request, ...), the policy produces a divide-by-zero
exception.
 2021-01-25 13:58:10 +01:00
Norman Feske
a6a923c31b driver manager: restart intel_fb when frozen 
This patch introduces heartbeat monitoring and automated restarting
of the intel framebuffer driver.
 2021-01-25 13:58:10 +01:00
Norman Feske
f687d4824b sculpt: set lower bound for font size 2021-01-25 13:58:10 +01:00
Norman Feske
0a478dac7f pdf_view: tweaks for showing 16:10 slides 
- Shave off two pixels on right, which are rendered white for no reason.
- Adjust resolution to make the slides fit vertically by default.
 2021-01-25 13:58:10 +01:00
Norman Feske
5905e0a4a0 Road map for 2021 2021-01-25 13:58:09 +01:00
Martin Stein
d0ac8a6036 cbe_tester.run: use /tmp for image on autopilot 2021-01-25 13:58:09 +01:00
Stefan Kalkowski
bdd923406f base: remove SPEC variables of boards (fix #3971) 
* Remove SPEC declarations from mk/spec
* Remove all board-specific REQUIRE declaratiions left
* Replace [have_spec <board>] run-script declarations with have_board where necessary
* Remove addition of BOARD variable to SPECS in toplevel Makefile
* Move board-specific directories in base-hw out of specs
 2021-01-25 13:58:09 +01:00
Stefan Kalkowski
5a123e37c9 run: introduce have_board helper function 
The new helper function returns a boolean value analoque to 'have_spec' if the
BOARD variable corresponds to the given value. It shall replace [have_spec <board>]
declarations in run-scripts.

Ref #3971
 2021-01-25 13:58:09 +01:00
Christian Helmuth
6cfaac182a Remove Cpu_session::Native_cpu definition from API 
This type can be a forward declaration in the public API because its
definition is required only in kernel-specific code.

Related to #3979
 2021-01-25 13:58:09 +01:00
Christian Helmuth
3e73d8d7b6 drivers_nic-muen: permit Timer access in platform_drv 2021-01-25 13:58:09 +01:00
Pirmin Duss
a4d5687510 base-hw: allow out of tree imx8 platforms 
Issue #3911
 2021-01-25 13:58:09 +01:00
Alexander Boettcher
2b0170fb6a base-hw: ack thread capability on construction 
Fixes #3982
 2021-01-25 13:58:09 +01:00
Martin Stein
2d21d04c76 cbe_tester.run: raise timeout to 6 minutes 2021-01-25 13:58:09 +01:00
Martin Stein
f6d195a9de nic drivers: provide optional Uplink-client mode 
In order to perform a smooth transition from NIC drivers that act only as NIC
session clients to NIC drivers that act only as Uplink session clients, this
commit introduces an intermediate state in which all NIC drivers support both
modes. That said, a NIC drivers mode is now statically determined through a new
optional 'mode' attribute in the drivers <config> tag that can be set to either
'nic_server' (default value) or 'uplink_client'. Reconfiguring this attribute
at a driver doesn't have any effects. Whithout this attribute being set, all
NIC drivers will behave the same as they did before the commit. When set to
'uplink_client', however, instead of providing a Nic service, they request
an Uplink session whenever their network interface becomes "UP" and close the
session whenever their network interface becomes "DOWN".

Ref #3961
 2021-01-25 13:58:09 +01:00
Martin Stein
1d2649b49a nic_router: act as "Uplink" server 
Let the NIC router provide an Uplink service besides the Nic service that it
already provided. Requests for an Uplink session towards the NIC router are
assigned to Domains using the same <policy> configuration tags that are used in
order to assign Nic session requests. The MAC addresses of Uplink session
components are _NOT_ considered during the allocation of MAC addresses for NIC
session components at the same Domain. The task of avoiding MAC address clashes
between Uplink session components and Nic session components is therefore left
to the integrator. Apart from that, Uplink session components are treated by
the NIC router like any other interface.

Ref #3961
 2021-01-25 13:57:43 +01:00
Martin Stein
cf72d1aac3 os: introduce new "Uplink" session 
Adds new Uplink session interface, the corresponding client side (Client,
Connection), and the corresponding API archives. An Uplink session is almost
the same as a NIC session with the difference that the roles of the end points
are swapped. An Uplink client is the one that provides a network interface
(for instance, a NIC driver) whereas an Uplink server is the one that uses
that network interface (for instance, a networking stack).

Therefore, in contrast to the NIC session, MAC address and link state come from
the Uplink client. The link state is reflected through the lifetime of an
Uplink session: The client requests the session only when the link state is
"UP" and closes it whenever the link state becomes "DOWN" again. The MAC
address is transmitted from the Uplink client to the Uplink server as an
argument of the session request.

Ref #3961
 2021-01-25 13:48:09 +01:00
Martin Stein
9222463565 run/ping: support running manually on Linux 
* Adds documentation how to prepare and finalize a Linux for running the
  scenario ontop of it
* Adds consideration of env variable 'ON_LINUX_WITH_DST_IP' that, if set,
  adapts the run script to running on Linux with the given ping destination IP

Ref #3961
 2021-01-25 13:48:09 +01:00
Norman Feske
8ff75346dd sticks_blue_backdrop: inherit roms from backdrop 
This patch moves the <content> <rom/> ... </content> declarations that
refer to pkg/backdrop to the runtime file of pkg/backdrop.

Issue #3981
 2021-01-25 13:48:09 +01:00
Norman Feske
cae5d380c4 depot_query: inherit pkg rom content from deps 
This patch relaxes the need to specify the rom content of all pkg dependencies
in each runtime file. Whenever a dependency features a runtime file, the
contained <content> <rom/> ... </content> nodes are implicitely included.

Fixes #3981
 2021-01-25 13:48:09 +01:00
Alexander Boettcher
14d8627186 platform_drv: avoid using Register in Bdf type 
Issue #3963
 2021-01-25 13:48:09 +01:00
Alexander Boettcher
f358fcbda6 nova: flush Intel IOMMU caches 
- when memory is detached

Issue #3963
 2021-01-25 13:48:09 +01:00
Alexander Boettcher
b185f3fac1 platform_drv: add support to power PCI devices 
- depending on available PCI power cap power off and on
- save and restore PCI bars if required
- reset PCI devices after power on if supported

Fixes #3963
 2021-01-25 13:48:09 +01:00
Alexander Boettcher
5f7fe7498f platform_drv: add mmio delayer support 
required after power on and function level reset

Issue #3963
 2021-01-25 13:48:08 +01:00
Alexander Boettcher
c89864c830 platform_drv: add typesafe PCI config access 
using Register_set && Register. It is derived from Genode::Mmio, however uses
as backend the Config_access implementation which is still used widely in
the platform driver.

Issue #3963
 2021-01-25 12:51:54 +01:00
Alexander Boettcher
59fafac4d6 platform_drv: increase readability 
by adding convenience functions to make code easier readable

Issue #3963
 2021-01-25 12:51:54 +01:00
Alexander Boettcher
ebf7f8f599 platform_drv: introduce structured PCI BDF type 
Replace explicit usage of bus, device, function arguments to methods or
variables all over the code by a single data type. It eases the reading of and
shorten the code.

Issue #3963
 2021-01-25 12:51:54 +01:00
Norman Feske
f57519397b Remove Pd_session::Native_pd definition from API 
This type can be a forward declaration in the public API because its
definition is required only in kernel-specific code.

Fixes #3979
 2021-01-25 12:51:54 +01:00
Christian Helmuth
5ca3847c89 cbe_tester requires x86_64 currently 2021-01-25 12:51:53 +01:00
Norman Feske
eee8f64fd4 sculpt: fix dialog sizes with manual fonts config 
Fixes #3976
 2021-01-25 12:51:53 +01:00
Alexander Boettcher
0a5741f076 acpica.run: add acpi_hid evaluation support 
Evaluate acpi_hid report in acpi_event

Showcase feature in acpica.run for FUJITSU FUJ02E3 and KEY_FN_F4.
 2021-01-25 12:51:53 +01:00
Alexander Boettcher
1147f35972 acpica: add limited support for FUJ02E3 ACPI dev 
The keys, mute, touchpad toggle (Fn-F4) and rfkill may be reported by some
Fujitsu machines via the ACPI FUJ02E3 ACPI device. With this commit limited
support to detect the 3 keys are added and will be reported as Genode report.
 2021-01-25 12:50:55 +01:00
Norman Feske
d698e0876d window_layouter/decorator: drag-state corner cases 
This patch complements the commit "nitpicker: defer hover changed while
dragging" with fixes of the window layouter and motif decorator.

- Handling of empty pointer model (after unhovering a decoration)
  instead of not updating it.

- Re-applying the hover model to the window layout when leaving the
  drag state. This addresses the corner case that the hover model
  changed during the drag operation (which is rightfully not handled
  while dragging).

- Letting the window layouter enter the drag state only if a dragged
  window is defined.

Issue #3973
 2021-01-25 12:50:55 +01:00
Tomasz Gajewski
98798f18b5 Fix minor inconsistencies in mk files 
Fixes #3972
 2021-01-25 12:50:54 +01:00
Christian Helmuth
8bed4c1d54 base: support hexa-decimal values in ascii_to(long) 
This aligns the behavior of ascii_to(long&) with the behavior of
unsigned integer types, e.g., when parsing XML nodes into signed long
variables.
 2021-01-25 12:50:54 +01:00
Norman Feske
72801975cd nitpicker: defer hover changed while dragging 
When holding at least one key or button, the hovering should never
change. In the previous version, this invariant was not enforced,
but the symptoms for eventual inconsistencies were masked by two
checkes: one when generating the hover report, and one when submitting
the leave event. This patch enforces the invariant by suppressing any
change of 'User_state::_hovered' while in dragged state.

Fixes #3973
 2021-01-25 12:50:54 +01:00
Norman Feske
7266f29491 decorator: disable sync handling when idle 
When idle, the decorator still consumes a bit of CPU time when triggered
by the GUI session's sync signal. This patch installs the sync handler
each time the decorator goes idle.
 2021-01-25 12:50:38 +01:00
Norman Feske
2c82636a98 terminal: respond to font-size changes 
Issue #3970
 2020-12-23 13:38:39 +01:00
Norman Feske
d47f87a768 terminal: use GUI session directly 
Issue #3196
Fixes #3970
 2020-12-23 13:38:31 +01:00
Norman Feske
887fcecf63 base-linux: disable initramfs build on 32-bit 
The build tries to access the host header asm/unistd.h, which is
apparently not present on 64-bit hosts (in particular our CI).
 2020-12-18 09:11:25 +01:00
Christian Prochaska
0428e5e8b9 qt5: improve quickcontrols2 module support 
Fixes #3968
 2020-12-18 09:11:25 +01:00
Alexander Boettcher
0359ee6a76 sandbox: fix position scaling in affinity filter 
During position scaling use the location width/height which is the permitted
dimension within the child space, instead of using the whole possible
child space. (Fixup for "init: scale affinity location" Issue #1412)

Thanks to the report by Peter.
 2020-12-18 09:11:25 +01:00
Christian Helmuth
1bef11accf depot: update recipe hashes 2020-12-18 09:10:52 +01:00
Alexander Boettcher
c5de2acf57 vm_session(x86): support cstar register 
Fixes #3964
 2020-12-09 14:02:12 +01:00
Norman Feske
9189342b77 base-okl4: coding style 2020-12-09 14:02:11 +01:00
Norman Feske
abd688097a base-fiasco: coding style 2020-12-09 14:02:11 +01:00
Norman Feske
6930372d55 base-foc: remove stale README content 2020-12-09 14:02:11 +01:00
Martin Stein
a124f5b88d nic_router: support and show-case disabling arp at a domain 
* Introduce 'use_arp' configuration flag at the NIC router in order to disable
  requesting IP-to-MAC address resolutions via ARP for domains.
* Add automated run/nic_router_disable_arp test to the autopilot that tests
  the effect of the new flag
* Document the new flag and the new test in the NIC router README

Fixes #3935
 2020-12-09 14:02:11 +01:00
Sebastian Sumpf
0beda6bca4 dde_linux: USB MBIM modem driver support 
Thise driver supports USB LTE modems for Huawais' ME906s through MBIM
and provides a traditional Nic session. The "control" interface is a
Terminal session, which can be used via libmbim/mbimcli.

issue #3822
 2020-12-09 14:02:11 +01:00
Norman Feske
a0fb944721 Propagate session diag flag to core 
This commit restores the diag feature for selecting diagnostic output of
services provided by core. This feature became unavailable with commit
"base: remove dependency from deprecated APIs", which hard-wired the
diag flag for core services to false.

To control this feature, three possible policies can be expressed in a
routing target of init's configuration:

* Forcing silence by specifying 'diag="no"'
* Enabling diagnostics by specifying 'diag="yes"'
* Forwarding the preference of the client by omitting the 'diag'
  attribute

Fixes #3962
 2020-12-09 14:02:11 +01:00
Sebastian Sumpf
36eeab6df2 dde_linux: usb support extra data and zero config 
- Vendor devices add addtional data to the config descriptor, read and
  added to the usb session
- allow '0' configuration within the usb session

issue #3822
 2020-12-09 14:02:11 +01:00
Sebastian Sumpf
537472e9af libports: ffi for ARM_64 
Required by 'glib'

issue #3822
 2020-12-09 14:02:10 +01:00
Christian Helmuth
496dc5508f usb_hid: support for Apple HID devices 
Issue #3953
 2020-12-09 14:02:10 +01:00
Christian Helmuth
2a659cb750 usb_hid: support for Holtek-based mice fixups 
This enables USB mice based on the Holtek chipsets, which need USB
descriptor fixups. An example mouse is the Sharkoon Drakonia.

Issue #3953
 2020-12-09 14:02:10 +01:00
Alexander Boettcher
b097e598f1 usb_hid: use heap instead of dma/mem allocator 
The usb_hid driver does not need to distinguish between
normal memory and DMA capable memory, since all requests are routed via the
USB raw session to the usb host driver. The default Malloc implementation
implements this distinction, however exposes restrictions on the size of
allocations. As seen now by several USB HID devices, the size of device
driver allocations depend on read out hardware features and can be
larger than we support with our specialized default Malloc implementation.

Since we don't need this functionality, switching to an well
tested allocator (Heap) which can cope with varying sizes of allocation,
we can mitigate the size restriction.

Fixes #3953
 2020-12-09 14:02:10 +01:00
Alexander Boettcher
2c639169fd dde_linux: use linux memory primitives in impl/usb 
Issue #3953
 2020-12-09 14:02:10 +01:00
Martin Stein
bad8caee3f nic_router: support multiple DHCP option 6 entries 
* The NIC router now considers, memorizes, and, if configured, reports
  multiple DHCP option 6 entries from DHCP replies that it received as DHCP
  client
* A DHCP server at the NIC router can now be configured statically with
  multiple DNS server addresses to propagate
* The 'dns_server_from' attribute of the DHCP server of the NIC router now
  supports the forwarding of multiple DNS server addresses
* The automated run/nic_router_dhcp test tests all the above mentioned new
  functionality and reconfiguring it at runtime. The test was added to the
  autopilot.
* All run scripts were adapted to fit the new NIC router configuration
  interface

Fixes #3952
 2020-12-09 14:02:10 +01:00
Emery Hemingway
306466fc60 libc: always set argv and envp to valid arrays 
The arrays passed to main(…) must always be valid null-terminated arrays.

Fix #3955
 2020-12-09 14:02:10 +01:00
Martin Stein
063e4bd072 remove remnants of 'virtualbox_nic_router' test 
Ref #3952
 2020-12-09 14:02:10 +01:00
Martin Stein
e14b58a82c net/dhcp.h: more specific include guards 
Ref #3952
 2020-12-09 14:02:10 +01:00
Martin Stein
8d8edaea5d nic_router: fix link state on dhcp server reconfig 
When reconfiguring the NIC router modifies the information that DHCP clients
received through a DHCP server of the NIC router, the link state of the
correspodning interface has to do a "down-up" sequence. This provides a signal
to the DHCP clients to redo DHCP, receive the new information and update
accordingly.

Fixes #3931
 2020-12-09 14:02:10 +01:00
Martin Stein
b0327d0544 nic_router: preserve session link state until read 
See the NIC router README paragraph "Behavior regarding the NIC-session link
state" for further information.

Ref #3931
 2020-12-09 14:01:42 +01:00
Martin Stein
a7b878cbb5 nic_router: allow for config values of 0 seconds 
Ref #3931
 2020-12-09 14:01:42 +01:00
Martin Stein
7ac6f93838 nic_router: clean up init of configuration members 
Ref #3931
 2020-12-09 14:01:42 +01:00
Norman Feske
70ff3d9c90 base-foc: coding style 
- Cosmetic adjustments according to
  https://genode.org/documentation/developer-resources/coding_style

- Replace manual inclusion of kernel headers by one new compound
  header foc/syscall.h

- Rename namespace Fiasco to Foc
 2020-12-09 14:01:42 +01:00
Norman Feske
0209a2465d base-foc: update cap_integrity.run 2020-11-28 19:37:15 +01:00
Alexander Boettcher
b6408cec1c ssh_terminal: close terminal fd in pthread context 
Issue #3682
 2020-11-27 14:40:41 +01:00
Christian Helmuth
3fac8b106d version: 20.11 2020-11-27 14:01:47 +01:00
Norman Feske
5c27270b17 News item for version 20.11 2020-11-27 13:57:55 +01:00
Norman Feske
3f15d18392 Release notes for version 20.11 2020-11-27 13:57:54 +01:00
Christian Helmuth
f2e0c164c2 depot: update recipe hashes 2020-11-27 09:21:06 +01:00
Norman Feske
d672e95090 doc: update components.txt 2020-11-27 09:19:09 +01:00
Norman Feske
98211db63d doc: move release notes to sub directory 
This keeps the doc/ directory tidy and neat.
 2020-11-27 09:19:09 +01:00
Sebastian Sumpf
722254f864 dde_rump: Set '-fcommon' for rump host tools 
GCC 10 has `-fno-common` as default which leads to multiple definitions of
of symbols.

Fixes #3960
 2020-11-27 09:19:09 +01:00
Stefan Kalkowski
b907629341 sculpt: add basic support for i.MX8 Quad EVK 
* Introduce CPU quota for driver subsytem (needed by sd_card_drv)
* Introduce CPU quota for runtime subsytem and nic_drv (needed by fec_nic_drv)
* Increase CAP quota for inspect terminal slightly
* Add sculpt packages for imx8q_evk

Fix #3958
 2020-11-27 09:19:09 +01:00
Stefan Kalkowski
22852f2e50 sculpt: move pc-specifics away from generic part 
* Introduce sculpt-[board] specific package
* Move rtc driver into managed drivers sub-system
* Name nic_drv in a generic fashion in sculpt_manager
* Copy over pc-specific config files only when building for PC

Ref #3958
 2020-11-27 09:19:09 +01:00
Christian Prochaska
e22e2540ee qt5: add Qt5Xml library 
Fixes #3959
 2020-11-27 09:19:09 +01:00
Alexander Boettcher
78ab3c8db5 usb: avoid memset on failed memory allocation 
Issue #3953
 2020-11-27 09:19:09 +01:00
Alexander Boettcher
ffdd49f9ce platform_drv: free ever non PCI devices on close 
Fixes #3957
 2020-11-27 09:19:09 +01:00
Stefan Kalkowski
0cbd1d1b7c os: adjust sd_card_drv to use platform API 
Fix #3956
 2020-11-27 09:19:09 +01:00
Stefan Kalkowski
f4ac642f64 platform_drv: reset pointer when releasing device 
Ref #3956
 2020-11-27 09:19:09 +01:00
Martin Stein
955afd8837 ada runtime: support image of enums 
Ref #3937
 2020-11-27 09:19:09 +01:00
Norman Feske
9b544787bd base: make Affinity::Space::location_of_index const 2020-11-27 09:19:09 +01:00
Alexander Boettcher
774b1f4277 base: remove Thread_deprecated 
Fixes #3954
 2020-11-27 09:19:08 +01:00
Josef Söntgen
dbcb1ff480 ssh_terminal: flush before closing connection 
Originally the outgoing buffer was flush when the terminal was
detached. Since we cannot do that at this point, make the flushing
part of the common cleanup procedure.

Issue #3682.
 2020-11-27 09:19:08 +01:00
Josef Söntgen
551b17591c ssh_terminal: handle detached term in event loop 
Rather than calling 'ssh_disconnect' from within the Terminal session,
flag the detached terminal in the session and let the event loop do the
cleanup. Otherwise it might happen that the 'ep' (handling the Terminal
session) as well as the 'pthread.0' (executing the ssh event loop) end
up both triggering the cleanup concurrently.

Issue #3682.
 2020-11-27 09:19:08 +01:00
Josef Söntgen
51a50ece60 libc: guard select call in socket_fs with monitor 
Issue #3682.
 2020-11-27 09:19:08 +01:00
Norman Feske
0dcb526ae5 base-linux: coding style 2020-11-27 09:19:08 +01:00
Christian Helmuth
dc016cbd5c ldso: log linker-area info on 'ld_verbose' 2020-11-27 09:19:08 +01:00
Christian Prochaska
e5f442f2d3 qt5: use integrated virtual keyboard in example application 
Fixes #3949
 2020-11-27 09:19:08 +01:00
Piotr Tworek
5db2971903 base-hw/arm_v7a: Mark r1 as out in Kernel::call64 
In case of arm_v7a Kernel::call64 the syscall will use both r0 and r1 as
output registers. Unfortunately the inline asm does not reflect this and
only r0 is explicitly specified as output. GCC manages to produce output
which we'd like to see. Clang on the other hand takes adventage of the fact
r1 should not be touched by swi and produces fewer instructions which do
what the code describes, but not what we actually want.

Basically the code which we want and is generated by GCC is:
  svc     0
  mov     r2, r0
  mov     r0, r1
  mov     r1, r2
  bx      lr

Clang on the other hand generates correct assembly given the code, but
incorrect given what the function is supposed to do:
  svc     0
  mov     r1, r0
  mov     r0, #0
  bx      lr

Both GCC and clang generate the same, expected assembly output when r1
is marked as output register from the inline asm swi call.

Fixes #3951
 2020-11-27 09:19:08 +01:00
Alexander Boettcher
aa7f5bc95f os: add cpu balancer component 
Issue #3843
 2020-11-27 09:19:08 +01:00
Alexander Boettcher
6872fdb0de cpu_burner: utilize all assigned CPUs 
Issue #3843
 2020-11-27 09:19:08 +01:00
Piotr Tworek
48220dfd9b os: Add missing {} around sub-object initializers. 
Clang complains when a sub-object is initialized without {}.

Issue #3950
 2020-11-27 09:19:08 +01:00
Piotr Tworek
50ab86cd72 base-hw: Remove unused rdtsc function for x86_64 
This function has no callers. Clang warns about this.

Issue #3950
 2020-11-27 09:19:08 +01:00
Piotr Tworek
cc7de65c9e nitpicker: Remove unused hover_reported variables 
Fix for unused private member variable warning reported by clang.

Issue #3950
 2020-11-27 09:19:08 +01:00
Piotr Tworek
cc193a9155 base: drop Trace::Session_component::_parent_levels 
Clang correctly asserts this private member variable is not used
anywhere in the code. I'm not sure what the intention of the code is,
might be this is a part of some unfinished feature. This patch just does
the minimum amount of work to allow the code to build with clang. If
required I can also drop the parent_levels constructor argument and
clean up the call sites.

Issue #3950
 2020-11-27 09:19:08 +01:00
Piotr Tworek
c0309a634e base: Silence unused arg warning in rpc_server.h 
The msg argument in Genode::Rpc_dispatcher::_read_arg is not used. GCC
does not care about this, but clang does and prints a warning regaring
this. Silence it by removing unused argument name.

fixup! base: Silence unused arg warning in rpc_server.h
 2020-11-27 09:19:08 +01:00
Martin Stein
30b8f4efc8 gems: import Genode-specific code of the CBE 
The CBE repository contained a lot of Genode-specific code despite the fact
that the CBE core logic is not bound to Genode in any way. Therefore the
Genode-specific CBE code is moved to the 'gems' repository to form part of
Genode mainline. The remaining CBE code becomes a port in Genode instead of
being invoked as sub-repository.

The commit combines the following work steps:

* add all files removed from CBE repository
* add CBE port files
* make all CBE libs and targets build again
* make all CBE run scripts succeed again
* make all CBE recipes build again
* make CBE autopilot succeed again
* let CBE autopilot use 'libsparcrypto' contrib dir and Genode build dir
  instead of '.ci' dir in CBE contrib dir (remove '.ci' dir from CBE repo)
* let CBE autopilot always check for all ports
* make CBE autopilot directly executable
* fix license headers in all Genode CBE files
* remove unused VFS replay component
* remove unused CBE test
* remove unused external crypto
* remove unused files in run dir
* remove unused external trust anchor
* add cbe_tester test to autopilot list
* get rid of directories 'include/cbe_*' and 'include/utils'

Fixes #3937
 2020-11-27 09:19:08 +01:00
Alexander Boettcher
24181f2bf6 driver_manager: use componentized usb driver 
Fixes #3943
 2020-11-23 12:03:00 +01:00
Alexander Boettcher
fae3c12366 usb: skip reporting of incomplete init. usb device 
Fixes #3944
 2020-11-23 12:03:00 +01:00
Stefan Kalkowski
4e90dc4512 dde_linux: adjust fec_nic_drv to use platform_drv 
Fix #3947
 2020-11-23 12:03:00 +01:00
Alexander Boettcher
a4c7837fb3 Update to bender with optional microcode and HWP 
plugin support.

Fixes #3871
 2020-11-23 12:03:00 +01:00
Christian Prochaska
764ab3be20 qt5: use libc 'posix_memalign()' in qtwebengine 
Fixes #3916
 2020-11-23 12:03:00 +01:00
Alexander Boettcher
c6a2e287d0 trace: forward exceptions during construction 
The control area is constructed during session creation and the caller can
handle the Out_of_* exception by increasing the quota by the next attempt.

Fixes #3917
 2020-11-23 12:03:00 +01:00
Emery Hemingway
d16a1bd922 vfs_lwip: log warning when blocking application 
Log a warning when the lwIP VFS plugin queues a read while waiting
for an interface to become ready. By comparision the Lxip plugin
has the same behavior but a more verbose initialization process.

Fix #3920
 2020-11-23 12:03:00 +01:00
Alexander Boettcher
b7ba508110 nova: update to kernel supporting AMD IO-MMU 
Fixes #3928
 2020-11-23 12:03:00 +01:00
Alexander Boettcher
d9cde328cb acpi: support amd iommu detection 
Issue #3928
 2020-11-23 12:03:00 +01:00
Alexander Boettcher
6b20a6bc7c tool: add hint how to enable vga console in nova 
Issue #3928
 2020-11-23 12:03:00 +01:00
Alexander Boettcher
95c2e5beb3 run: extend pci test to show more information 
Add some more vendors and classcode decoding. Additionally, walk PCI cap
space of each device and decode some of the types.

Issue #3928
 2020-11-23 12:03:00 +01:00
Josef Söntgen
194305a8bb dde_bsd: apply HP sense quirk for Fujitsu S398 
This quirk is needed for properly switching between the internal and
an external headphone.

Thanks to Peter for the patch.

Issue #3929.
 2020-11-23 12:03:00 +01:00
Josef Söntgen
b6912a3d87 dde_bsd: query initial head-phone state 
Account for the situation where the driver is started while the HP
is already plugged in and configure the mixer accordingly.

Thanks to Peter for the patch.

Issue #3929.
 2020-11-23 12:03:00 +01:00
Josef Söntgen
1b4444ce9e dde_bsd: use right member for iterating mixer set 
Due to what seems like a copy-and-paste error, the wrong union member
was used to compare the label. Fortunately, as both structs have the
same memory layout that did not result in any issue.

Issue #3929.
 2020-11-23 12:03:00 +01:00
Josef Söntgen
b9869b666a dde_bsd: use bsd_task to handle I/O 
Originally, the kernel code should have been executed within the
context of the main task like it is done in dde_linux. The initial
port of the HDA driver, however, did not required doing so and the
session called code directly.

When using USB device, that is no longer possible as we have to
suspend the execution during the execution of the kernel code. So
we pass in the audio data and schedule the kernel.

Fixes #3929.
 2020-11-23 12:03:00 +01:00
Josef Söntgen
cd7c99afdc dde_bsd: implemented delayed session announcement 
This becomes necessary in case of the USB audio driver where we have
to wait for and query the function first.

That being said, alternatively we could also announce the session but
ignore any request until there is a device we can use.

Issue #3929.
 2020-11-23 12:02:59 +01:00
Josef Söntgen
2ec398e550 dde_bsd: use timeout framework for time handling 
With this commit the timer back-end uses the timeout framework to
schedule any occuring timeouts and for providing the current time.
For now there is only one timeout, the unsolicited azlia codec event
and therefore the timeout queue consists of solely one timeout object.

In addition a timer session is used for implementing 'delay()' where
we have to block until the delay is completed.

Issue #3929.
 2020-11-23 12:02:59 +01:00
Josef Söntgen
bdb71d94c2 dde_bsd: pull strlcpy in via contrib code 
For historical reason the 'strlcpy' implemention was directly pull in
into the emulation environment. There is, however, no reason not to
use the contrib sources in the usual fashion.

Issue #3929.
 2020-11-23 12:02:59 +01:00
Josef Söntgen
7193902cc0 dde_bsd: properly name PCI audio driver component 
Instead of the generic name, call the PCI driver 'pci_audio_drv'.

This is preliminary clean-up work before introducing the USB audio
driver.

Issue #3929.
 2020-11-23 12:02:59 +01:00
Josef Söntgen
3faf5c43a8 dde_bsd: seperate PCI back end 
The supported drivers so far exclusively used PCI and for practical
reasons the emulation environment was to tied to. To make future
addition of drivers that employ other transport busses easier, split
the bus handling into its own backend.

This is preliminary clean-up work before introducing the USB audio
driver.

Issue #3929.
 2020-11-23 12:02:59 +01:00
Josef Söntgen
6c7f0cb7cc dde_bsd: properly enable recording in run script 
* Updating the run script was missed in commit "dde_bsd: remove
    configration attributes".

  * Decrease output and input volume to "more sane" levels.

Issue #3929.
 2020-11-23 12:02:59 +01:00
Josef Söntgen
54d36a7d1b mixer: apply config on new sessions 
Updating the config and using its settings instead of only reporting
the new session allows to apply stored channel settings immediately.

In case of Sculpt this is useful when an existing launcher already
contains a valid config that sets the volume levels appropiately or
the 'recall_fs' is used.

Fixes #3930.
 2020-11-23 12:02:59 +01:00
Martin Stein
9b164d20fd vfs directory service: name enum Open_mode 
Fixes #3936
 2020-11-23 12:02:59 +01:00
Alexander Boettcher
cd8b436566 usb_hid: avoid destruction of used signal handler 
Fixes #3945
 2020-11-23 12:02:59 +01:00
Piotr Tworek
87e90d640f base-linux: Drop unused member variables 
The _root, _gid and _uid private member variables are not used for
anything. GCC does not care, but clang likes to complain about things
like this.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
db71cb8c63 base-linux: Drop ununsed Message::_addr 
Not used anywhere in the current codebase. GCC does not care but clang
warns about this.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
a892018926 base-hw: Define Entry function as noreturn. 
Right now the code marks specific instance of this function as noreturn.
It then tries to initialize it using a value that has the same type,
except for the noreturn part. GCC does not care, but clang complains this
technically assigns the value of entry from incompatible pointer type.

Fix this by defining Entry function as no return.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
1643d623e4 base-hw: Fix warning about unused lambda capture 
The lambda function passed to Board::Vm_page_table_array does not
actually use "this" for anything. GCC does not warn about this, but
clang does. Remove the extra capture to make clang happy.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
9b84a8a402 base-hw: Add semicolons after [[fallthrough]] 
GCC doesn't care, but clang complains if [[fallthrough]] is not followed
by a semicolon. Existing code is also not consitent in this regard. Lets
just fix it to make clang happy.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
db17d51ff1 os: Drop unused _tree member in Root_clock_divider 
Its not used anywhere in the code. Clang produces warning due to this.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
736b000c19 os: Drop ununsed Main::_id_id variable in ping 
The variable is not used anywhere in the code. This produces warning
when building with clang.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
187b8ece27 os: Drop ununsed Virtdev_rom::Root::_env member 
Set but not accessed from anywhare. This triggers compilation warning
when building the code with clang.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
93288bccb3 os: Refer to ::Sandbox::Child explicitly. 
When compiling this code, clang complains reference to 'Sandbox' is
ambiguous. It can either mean ::Sandbox namespace from
os/src/lib/sandbox/types.h, or Genode::Sandbox declared in
repos/os/include/os/sandbox.h. Since the code refers to ::Sandbox::Child
and its already in ::Sandbox namespace we can just drop "Sandbox" and
refer to just Child, which makes clang happy.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
444bc18fcf base: Drop unused Cpu_root::_md_alloc member 
This private variable has no uses in the context of the class. Spotted
when building the code with clang.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
18be6315cb base: Drop unused _ram member variable 
Its initialized in constructor but never used. Spotted when building the
code with clang.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
9c3ce58e57 base: Drop unused "verbose" variable 
Spotted when building the code with clang. This is not used anywhere in
the file.

Issue #3938
 2020-11-23 12:02:59 +01:00
Piotr Tworek
d4a3aa7eda base: explicit copy constructor for Rpc_in_buffer 
According to the "rule of three" [1] and C++11 [2] Rpc_in_buffer needs
to have an explicit copy constructor since it also has user defined
copy assignment operator. Both clang and newer versions of GCC complain
about this.

[1] https://en.cppreference.com/w/cpp/language/rule_of_three
[2] https://www.ece.uvic.ca/~frodo/cppdraft/n4659/html/depr.impldec

Issue #3938
 2020-11-23 12:02:58 +01:00
Piotr Tworek
8d6ca9556f base: fix clang warning for void cast of retval 
This path fixes a void cast used to silence unused return value warning.
Its a common pattern to use void cast to do that. The code uses void *
cast instead. It works for GCC, but clang complains about this.

Issue #3938
 2020-11-23 12:02:58 +01:00
Piotr Tworek
81a49bffee base: exception specification for operator delete 
Clang is rather picky about this and prints the following warning when
compiling new_delete.cc:

  error: function previously declared with an explicit exception
  specification redeclared with an implicit exception specification
  [-Werror,-Wimplicit-exception-spec-mismatch]

Issue #3938
 2020-11-23 12:02:58 +01:00
Piotr Tworek
53a990579b base: Fix UAF in Genode::Pd_session_component::free 
This was discovered when building the code with clang instead of GCC. In
this setup the run/ping on base-hw/arm_v8a/virt_qemu would crash
on shutdown due to uncaught Deref_unconstructed_object exception thrown
for Genode::Reconstructible<Genode::Account<Genode::Ram_quota>>. The
specific instance throwing this exception was
Pd_session_component::_ram_account. My investigation exposed the
following problem:

1. The Pd_session_component has a _sliced_heap member backed by
   _constrained_ram_alloc which in turn uses Pd_session_component itself
   as its Ram_allocator.
2. When ~Pd_session_component is called it first destroys _ram_account,
   followed by _signal_broker.
3. The signal broker holds a reference to
   Pd_session_component::_sliced_heap as Signal_broker::_md_alloc.
4. The base-hw implementation of ~Signal_broker destroys some contexts
   and does this by calling Genode::destroy on some slabs using the
   _md_alloc (ref to Pd_session_component::_sliced_heap).
5. The Genode::Slab calls the Ram_allocator::free which ends up calling
   Pd_session_component::free.
6. The Pd_session_component::free can among other things call replenish
   method on Pd_session_component::_ram_account which has already been
   freed at this point.

From my POV calling replenish at this point is basically an undefined
behavior. The Genode::Constructible holding the Genode::Account was
already detroyed at this point. GCC builds happen to somehow manage to
go through the -> operator call without raising any alarms, while clang
builds trip on the _check_constructed() call.

This fix moves the _ram_account a bit higher in class declaration to
ensure its destroyed after _sliced_heap. This seems like the simpliest
solution for this problem.

Fixes #3941
 2020-11-23 12:02:58 +01:00
Christian Helmuth
a8d3cd9b15 libc: open socket files early on socket creation 
This prevents later file-descriptor shortage when opening files on
demand, which can't be reflected to the application in a sane manner.

The real fix is to open socket files not on libc level but on VFS level
only effectively consume one libc file descriptor for one socket.
 2020-11-23 12:02:58 +01:00
Christian Helmuth
5dfca79bcc libc: use pthread mutex in getifaddrs() 2020-11-23 12:02:58 +01:00
Christian Prochaska
ff429a8056 base-linux: build 'lx_iopl()' function on x86 only 
Fixes #3934
 2020-11-23 12:02:58 +01:00
Christian Prochaska
eafbfb8edf base-linux: socket descriptor caps for RPC (BOARD=pc) 
Fixes #3932
 2020-11-23 12:02:58 +01:00
Christian Prochaska
b72503e581 base-linux: remove 'lx_ioperm()' function 
Fixes #3933
 2020-11-23 12:02:58 +01:00
Stefan Kalkowski
429cd8d37a os: enable smp support for ARM vmm 
* Fix GIC model to support priority and cpu target settings correspondingly
* Fix semantic of SGIR register for GICv2
* Minor GIC model IRQ state fix
* Introduce synchronization for VirtIO and GIC models
* Enable multiple CPUs in test run-script for ARMv8

Fix #3926
 2020-11-23 12:02:58 +01:00
Stefan Kalkowski
6be09a27ca os: minor fix terminal_expect_send test-component 
When an empty line was send, it could occur
that a previous line got printed instead.

Ref #3926
 2020-11-23 12:02:58 +01:00
Stefan Kalkowski
7298b00013 base-hw: make ARMv7/v8 hypervisor smp ready 
* Introduce hypervisor-stack per CPU
* Introduce host world context per CPU
* Mark EL2 translation table memory as inner shareable
* The VMID is not bound to a single VCPU, but to the Vm_session as a whole
* Set affinity of the VCPU accordingly
* Add VMPIDR to VM state

Ref #3926
 2020-11-23 12:02:58 +01:00
Stefan Kalkowski
1d826a2c48 base-hw: do direct syscall when run/pause a VCPU 
Instead of calling core to run/pause a VCPU, go directly to the kernel.
Apart from the performance win, it would otherwise involve a more complex
protocol, when a VCPU on another core has to be removed from the scheduler.
Core's entrypoint handling those request runs on the boot-cpu only.

Ref #3926
 2020-11-23 12:02:58 +01:00
Stefan Kalkowski
40445d7011 base: extend vm_session API with native vcpu cap 
To enable the interaction of a VMM with the kernel directly,
a hidden RPC gets introduced. It allows a kernel-specific
base-library implementation of the Vm_session::Client to request
a kernel-specific capability to address a VCPU, e.g., to
run/stop it.

Ref #3926
 2020-11-23 12:02:58 +01:00
Alexander Boettcher
11e261ada4 platform_drv(x86): use 32bit writes for MSI-X 
Fixes #3927
 2020-11-23 12:02:57 +01:00
Christian Helmuth
c93f3a1136 libusb: establish/terminate USB session 
Now, the USB connection is established on backend initialization and
terminated on backend exit triggered by high-level libusb code.

Thanks to Peter for the patch.
 2020-11-23 12:02:57 +01:00
Christian Helmuth
e339dd542c Prevent warning in test-pthread 2020-11-23 12:02:57 +01:00
Christian Helmuth
3d23c8c419 libports: update curl download location 2020-11-23 12:02:57 +01:00
Martin Stein
89d28c8222 timeout: no volatile stackvars, better warnings 2020-11-23 12:02:57 +01:00
Christian Prochaska
dff3bac441 libc: submit monitor execution signals locally 
Issue #3924
 2020-11-23 12:02:55 +01:00
Christian Prochaska
798beab30e base: support process-local signal submission 
Issue #3923
 2020-11-23 12:02:53 +01:00
Christian Prochaska
50e0f3b977 base: don't throw exceptions in 'Signal_receiver::pending_signal()' 
Issue #3922
 2020-11-23 12:02:49 +01:00
Josef Söntgen
f754e2a7d7 stdcxx: add symbols needed by testsuite 
Fixes #3921
 2020-10-23 15:12:06 +02:00
Josef Söntgen
1dd1bfe692 stdcxx: add new/delete aligned variants (C++17) 
Issue #3921
 2020-10-23 15:08:40 +02:00
Josef Söntgen
a74b572e1f stdcxx: force symlinks to allow for re-preparing 
Issue #3921
 2020-10-23 14:59:56 +02:00
Josef Söntgen
a24911296a stdcxx: adapt header files for testsuite 
Issue #3921
 2020-10-23 14:59:49 +02:00
Josef Söntgen
563cc07cb0 libports: posix src archive implements posix API 
Issue #3921
 2020-10-23 14:58:27 +02:00
Josef Söntgen
59f562f627 libc: add symbols needed by testsuite 
Issue #3921
 2020-10-23 14:58:04 +02:00
Josef Söntgen
4981eb425e ld: add symbols needed by testsuite 
Issue #3921
 2020-10-23 14:58:01 +02:00
Josef Söntgen
de8411a5e1 cxx: add missing low-level symbols 
Issue #3921
 2020-10-23 14:56:36 +02:00
Christian Helmuth
5be1c793a5 depot: update recipe hashes 2020-10-23 14:16:38 +02:00
Emery Hemingway
b4076e762c libc: log a message and exit for raise(...) 
Fix #3919
 2020-10-21 09:14:55 +02:00
Christian Helmuth
6ea628195f dde_bsd: update mirror URL 
It seems RWTH Aachen mirrors only ancient versions currently.
 2020-10-20 09:14:40 +02:00
Martin Stein
64487ded7c timeout: don't warn "timestamp value too big" too often 
Fixes #3657
 2020-10-19 14:26:56 +02:00
Christian Prochaska
405955eaef libc: implement 'posix_memalign()' 
Fixes #3915
 2020-10-15 15:41:55 +02:00
Alexander Boettcher
0aaed47652 nova: re-/store ep fpu state in vm_session 
Fixes #3913
 2020-10-13 14:42:06 +02:00
Alexander Boettcher
20606bc6de foc: save/restore fpu state in vm_session 
Issue #3913
 2020-10-13 14:42:03 +02:00
Christian Prochaska
9cd38a6846 libc: rwlock cleanup 
Issue #3912
 2020-10-13 14:38:43 +02:00
Christian Prochaska
bf4afefaa1 libc: use semaphore in rwlock implementation 
Fixes #3912
 2020-10-13 08:21:04 +02:00
Christian Helmuth
f09b0dc224 Improve synchronization in lwip.run 
The lynx HTTP GET test was started before the HTTP server was up and
therefore failed on Qemu/PBXA9.

Issue #3874
 2020-10-09 16:13:14 +02:00
Christian Prochaska
658030ef49 qt5: update port for qtwebengine improvements 
- unlink shared memory files
- lower maximum number of socket pool sockets to reduce chance of file
  descriptor exhaustion
- fix a build dependency which caused sporadic parallel build errors

Fixes #3910
 2020-10-09 13:51:26 +02:00
Christian Prochaska
4e8bfed5b1 libc: duplicate the file descriptor on shared 'mmap()' mappings 
Fixes #3909
 2020-10-09 13:50:35 +02:00
Christian Prochaska
5c47fa0d41 libc: reduce file descriptor lookup exceptions in 'select()' 
Fixes #3908
 2020-10-09 13:48:23 +02:00
Christian Prochaska
058f2e687c libc: make 'mmap()' address alignment configurable 
With this commit, the alignment of anonymous 'mmap()' allocations can be
configured like this:

<config>
  <libc>
    <mmap align_log2="21"/>
  </libc>
</config>

Fixes #3907
 2020-10-09 13:47:33 +02:00
Josef Söntgen
7d21335ac9 vfs/oss: VFS plugin for Audio_out access via files 
This plugin gives access to the Audio_out session by roughly
implementing a OSS pseudo-device. It merely wrapps the session and does
not provide any resampling or re-coding.

Fixes #3891.
 2020-10-09 13:44:27 +02:00
Josef Söntgen
3d2b0cab93 libc: implement SNDCTL I/O control handling 
In the same vein as the terminal and block I/O controls, the sound
controls are implemented via poperty files and match the OSS
API ([1] features a nice overview while [2] is v3 and [3] gives
in-depth information on the current v4.x API we eventually might want
to implement).

  [1] https://wiki.freebsd.org/RyanBeasley/ioctlref/
  [2] http://www.opensound.com/pguide/oss.pdf
  [3] http://manuals.opensound.com/developer/

The controls currently implemented are the ones used by the cmus OSS
output plugin, which was the driving factor behind the implementation.
It uses the obsolete (v3) API and does not check if the requested
parameter was actually set, which should be done according to the
official OSS documentation.

At the moment it is not possible to set or rather change any
parameters. In case the requested setting differs from the parameters
of the underlying Audio_out session - in contrast to the suggestion in
the OSS manual - we do not silently adjust the parameters returned
to the callee but outright fail the I/O control operation.

The following list contains all currently handled I/O controls.

  * SNDCTL_DSP_CHANNELS sets the number of channels. We return the
    available channels here and return ENOTSUP if it differs from
    the requested number of channels.

  * SNDCTL_DSP_GETOSPACE returns amount of playback data that can
    be written without blocking. For now it amounts the space left
    in the Audio_out packet-stream.

  * SNDCTL_DSP_POST forces playback to start. We do nothing and return
    success.

  * SNDCTL_DSP_RESET is supposed to reset the device when it is
    active before any parameters are changed. We do nothing and return
    success.

  * SNDCTL_DSP_SAMPLESIZE sets the sample size. We return the
    sample size of the underlying Audio_out session and return ENOTSUP
    if it differs from the requested number of channels.

  * SNDCTL_DSP_SETFRAGMENT sets the buffer size hint. We ignore the
    hint and return success.

  * SNDCTL_DSP_SPEED sets the samplerate. For now, we always return
    the rate of the underlying Audio_out session and return ENOTSUP
    if it differs from the requested one.

This commit serves as a starting point for further implementing the
OSS API by exploring more users, e.g. as VirtualBox/Qt5/SDL2 audio
backend or a more sophisticated progam like sndiod.

Issue #3891.
 2020-10-09 13:44:14 +02:00
Christian Helmuth
bcf1cc6397 Ensure high priority for timer in test.run 2020-10-09 13:37:17 +02:00
Martin Stein
bff624c75a test/entrypoint: extend timeout interval 
At least on some PIT-based platforms (x86_32 + pistachio/okl4/sel4), we run
into trouble with the reworked timeout framework that now proccesses all
pending timeouts before calling their handlers. This order change leads to a
higher rate of handling of short periodic timeouts in the timer driver which
can cause lower prioritized components to starve. Especially, if submitting
signals (from timer to client) isn't cheap (as is the case on qemu + pistachio
for example).

Issue #3884
 2020-10-09 13:37:14 +02:00
Martin Stein
512be0a52a test/timer_rate: determine lowest accurate period 2020-10-09 13:37:10 +02:00
Christian Helmuth
91f8281618 depot: update recipe hashes 2020-10-09 13:35:57 +02:00
Christian Prochaska
0e01729d77 libc: handle file descriptor allocation errors 
Fixes #3906
 2020-10-09 13:35:57 +02:00
Sebastian Sumpf
fe1ee05186 recipe: imx53_qsb_drivers 
Do not copy imx53 specfic source code since it has been unified within
the GPIO driver.

follow up for issue #3900
 2020-10-09 13:35:57 +02:00
Christian Helmuth
ec957739e9 Adapt some RAM quotas for 2560x1080 screens 2020-10-09 13:35:57 +02:00
Piotr Tworek
8d5005e03a os: add VirtIO nic driver 
The driver is faily simple and does not support fancy features like
TCP checksum offloading or vlan filtering, but it is fully capable of
running every Genode network based scenario I've tried. Its currently
known to work on virt_qemu arm platforms and x86_64.

Fix #3825
 2020-10-09 13:35:57 +02:00
Piotr Tworek
7fbb245710 os: add VirtIO device helper classes. 
To simplify writing native VirtIO drivers for Genode add helper classes
representing VirtIO device and queue. The queue implementation should
be platform independant. The device abstraction however is closely tied
to the VirtIO transport being used (PCI/MMIO). Both PCI and MMIO
implementations expose the same public API so the actual driver logic
should be the same regardless of which transport is used.

Its also important to note that the PCI version of Virtio::Device
currently does not support MSI-X interrupts. Unfortunately my kowledge
about PCI bus is very limited and my main area of interest was to get
VirtIO drivers working on virt_qemu ARM/Aarch64 platform. As such all
the VirtIO drivers I plan to submit will work with PCI bus, but might
not use some extended capabilities.

Ref #3825
 2020-10-09 13:35:57 +02:00
Piotr Tworek
9bd548c4bd os: add platform_drv config generator for virt_qemu 
The VirtIO device configuration on Qemu is dynamic. The
order and presence of different command line switches affects
base address and interrupt assignment of each device. One could
probably hard-code the necessary switches and resulting XML ARM
platform driver configuration in each run script, but this seems
like troublesome and hard to maintain solution.

This patch explores an alternative approach to the problem.
It implements a ROM driver which probes the address space region
Qemu virt machines assign to VirtIO MMIO devices and exposes the
result as XML via a ROM session. This XML output can be fed directly
as config to the generic ARM platform driver.

Ref #3825
 2020-10-09 13:35:57 +02:00
Piotr Tworek
fe0ad0addb tool: Consolidate qemu nic setup. 
Right now the same code dealing with nic setup on qemu is duplicated
in many different run scripts. It makes it unnecesarily complex to
change the existing config or add support for new nic types. Lets move
all this common code to qemu.inc.

Ref #3825
 2020-10-09 13:35:57 +02:00
Stefan Kalkowski
aa2511e209 platform_drv: make it available for arm_64 generic 
Ref #3825
 2020-10-09 13:35:57 +02:00
Stefan Kalkowski
3cf3344fa3 sel4 + foc: extend port's tool check rules 2020-10-09 13:35:56 +02:00
Sebastian Sumpf
c79687f5f4 gpio: introduce Pin and '_with_gpio' 
- make GPIO server more robust on imx by not throwing exceptions for
  unknown pins, use '_with_gpio' instead
- use 'Gpio::Pin' data type instead of POD 'unsigned'

issue #3900
 2020-10-09 13:35:56 +02:00
Sebastian Sumpf
b9bd179e54 gpio_drv: remove specs for imx53 and imx6 
Also cleanup run scripts and recipes were gpio driver is not required,
update the ones were it is.

issue #3900
 2020-10-09 13:35:56 +02:00
Sebastian Sumpf
6c6deb7e8b dde_linux: add touch to i.MX8 recipe 
Add touch screen and event filter to drivers interactive package.

issue #3900
 2020-10-09 13:35:56 +02:00
Sebastian Sumpf
d387eba0ba os: driver recipe for i.MX8 drivers 
Includes touch and gpio driver

issue #3900
 2020-10-09 13:35:56 +02:00
Sebastian Sumpf
96eb83f19a os: Synaptics DSX touch screen driver 
Provides touch event service for i.MX8M Quad only

issue #3900
 2020-10-09 13:35:56 +02:00
Sebastian Sumpf
89972b11b7 gpio_drv: use platform session for i.MX 
This makes the driver work for all i.MX boards. A platform driver with
the board specific configuration is required.

issue #3900
 2020-10-09 13:35:56 +02:00
Sebastian Sumpf
664b861f9d imx8_fb_drv: MIPI DSI display support 
- support for Northwest Logic MIPI DSI bridge
- support for Radium panel (NXP MX8_DSI_OLED1)
- new 'DSI-1' connector in driver config file

issue #3900
 2020-10-09 13:35:56 +02:00
Sebastian Sumpf
27f705bc48 bootstrap: iomux adaptions for i.MX8 MIPI 
These settings were obtained with a JTAG debugger from a running Linux
system. They work for MIPI as well as for HDMI.

issue #3900
 2020-10-09 13:35:56 +02:00
Alexander Boettcher
325e9cb9fa platform_drv(x86): avoid memory leak 2020-10-09 13:35:56 +02:00
Alexander Boettcher
50b10ef4a5 aes_cbc_4k: extend test and run in autopilot 
- to compare crypted text with host openssl

Fixes #3905
 2020-10-09 13:35:56 +02:00
Alexander Boettcher
c0f8022a78 aes_cbc_4k: use libcrypto 
Issue #3905
 2020-10-09 13:35:56 +02:00
Alexander Boettcher
5d808cdc01 platform_drv: extend MSI-X supports 
The patch handles the case, that the memory for the MSI-X table is part
of one of the Pci::Resource Memory BARs, which got allocated beforehand already.

With this commit, the platform driver will not fall back to use legacy IRQs or MSI, whereby MSI-X is available actually. Additionally, this patch avoids a lot of red
messages about non available IO-MEM printed by the roottask.

Fixes #3904
 2020-10-09 13:35:56 +02:00
Christian Helmuth
abefca500b libc: fix deadlock in pthread_cond_timedwait/signal() 
The deadlock occured with three concurrently running threads: two
waiters calling pthread_cond_timedwait() and one signaller calling
pthread_cond_signal().

If waiter W1 hits its timeout, the signaller may have called
pthread_cond_signal(), detected this waiter and posted the internal
'signal_sem' concurrently. Then, the signaller waits for 'handshake_sem'
to ensure the waiter got woken up.

Waiter W1 can't consume the 'signal_sem' post by
'sem_wait(&c->signal_sem)' because another waiter W2 may have consumed
the post already above in sem_wait/timedwait(). Waiting for a post on
'signal_sem' would block the waiter W1 in perfect deadlock with
signaller on 'handshake_sem'. As W1 also owns 'counter_mutex' in this
situation, waiter W2 would block when trying to aquire 'counter_mutex'
and can't resolve the situation.

So, W1 does nothing in this case and we accept the spurious wakeup on
next pthread_cond_wait/timedwait().
 2020-10-09 13:35:56 +02:00
Martin Stein
7feea78991 timeout: rework timeout framework 
* get rid of alarm abstraction
* get rid of Timeout::Time type
* get rid of pointer arguments
* get rid of _discard_timeout indirection
* get rid of 65th bit in stored time values
* get rid of Timeout_scheduler interface
* get rid of uninitialized deadlines
* get rid of default arguments
* get rid of Timeout::_periodic
* get rid of Timeout::Raw
* use list abstraction
* only one interface for timeout handlers
* rework locking scheme to be smp safe
* move all method definitions to CC file
* name mutexes more accurate
* fix when & how to set time-source timeout
* fix deadlocks

Fixes #3884
 2020-10-09 13:35:56 +02:00
Martin Stein
9e5d479d03 timeout: test smp support 
Ref #3884
 2020-10-09 13:35:56 +02:00
Christian Helmuth
26011a7151 libc: update status for component select handlers 
In case of contexts blocked in select() the monitor updates the
file-descriptor status, but if the entrypoint is just blocked for the
select handler, the status must be updated explicitly on
dispatch_select().
 2020-10-09 13:35:56 +02:00
Stefan Kalkowski
bbb017dc24 muen: update to Community 2019 toolchain version 
Fix #3903
 2020-10-09 13:35:55 +02:00
Alexander Boettcher
04d3c9e750 nova: re-enable DMAR super pages in kernel 
Fixes #3902
 2020-10-09 13:35:55 +02:00
Alexander Boettcher
e5fe9c6fc7 qemu-usb: catch exception on already gone devices 
Fixes #3893
 2020-10-09 13:35:55 +02:00
Norman Feske
04821b1abc nitpicker: apply focus change w/o input events 
This patch fixes the corner case where the keyboard focus is defined
independently from user interactivity, e.g., the activation of a
screensaver or lock screen.

In this case, nitpicker would update its internal focus state not before
the next input event is handled. Should this input event be a press
event, this event would wrongly be delivered to the prior focused
session. Another problematic situation is the initial state before the
very first input event occurs. Since the focus remains undefined until
the first input event is handled, an initial key press event would not
be delivered.

This is a regression caused be the transition to the event-session
interface and the removal of the nitpicker's periodic way of operation.
The patch fixes the problem by applying pending focus changes not only
at the input processing but also on the code path that responds to focus
changes (e.g., focus-rom update).

Issue #3812
 2020-10-09 13:35:55 +02:00
Alexander Boettcher
afab15f1a4 linux: round up dataspace to 4k sizes 
Issue #3901
 2020-10-09 13:35:55 +02:00
Alexander Boettcher
e61f6cfd38 base: add thread migration test 
Issue #3842
 2020-10-09 13:35:55 +02:00
Alexander Boettcher
90bea1499e core: store new affinity on successful migration 
Adjust the base-* platforms to acknowledge new thread location solely if
migration is supported and succeeded. Otherwise the wrong thread
locations are observed via the trace session and utilization time calculation
get wrong.

Issue #3842
 2020-10-09 13:33:36 +02:00
Alexander Boettcher
99fa203673 nova: add migration support for global threads 
Fixes #3842
 2020-10-09 13:33:36 +02:00
Martin Stein
1b41d9db90 base: remove alarm library from base 
Ref #3884
 2020-10-09 13:33:36 +02:00
Martin Stein
c1d0179194 dde_rump: move private headers to src/include 
Ref #3884
 2020-10-09 13:33:36 +02:00
Martin Stein
04463806a8 dde_rump: remove unused rump_cgd 
Ref #3884
 2020-10-09 13:33:36 +02:00
Martin Stein
af01370cc1 run/rump_fat: raise ram quota 
Ref #3884
 2020-10-09 13:33:35 +02:00
Martin Stein
4eb4bd6f96 trace_logger: warn on multiple buffer wraps 
Ref #3884
 2020-10-09 13:33:35 +02:00
Christian Prochaska
d2d74cc5fa tool chain: fix build error on Ubuntu 20.04 (GCC 9) 
Fixes #3899
 2020-10-09 13:33:35 +02:00
Josef Söntgen
f53df495db libc: provide 'sys/soundcard.h' header file 
Issue #3891.
 2020-10-09 13:33:35 +02:00
Josef Söntgen
f3268cade6 libc: split ioctl method 
Use one ioctl method for each type of I/O control because by now the
general method will become increasingly long.

Fixes #3890.
 2020-10-09 13:33:35 +02:00
Josef Söntgen
1a54ee895e vfs: use compound ioctl in Block-VFS plugin 
Instead of using the old 'ioctl' Vfs::File_io_services API implement
the I/O control functionality in a buch of files. This is similar to
the terminal-VFS plugin.

Fixes #3889.
 2020-10-09 13:33:35 +02:00
Josef Söntgen
27d4cb871f libc: use property files for block ioctl 
Like already done for terminal I/O controls use collect the information
by reading property files instead of using the old VFS ioctl interface.

Fixes #3888.
 2020-10-09 13:33:35 +02:00
Josef Söntgen
2312ad35dd libc: match ioctl request type to contrib type 
There is a type mismatch as in the FreeBSD contrib code the type of the
request is 'unsigned long'. So far, only I/O controls where the request
falls into the signed range where used and this was not a problem.

Some of the SNDCTL requests, however, have the bit set.

Fixes #3887.
 2020-10-09 13:33:35 +02:00
Christian Prochaska
85a84f5042 qt5: generate enter events related to popup menus 
Fixes #3894
 2020-09-18 15:23:45 +02:00
Sebastian Sumpf
0fd979b147 imx8_fb_drv: make 'Capture' constructible 
Reconstruct capture connection on config update in order to receive
update of complete screen data from GUI session.

fixes #3878
 2020-09-18 15:21:10 +02:00
Christian Helmuth
ad595d2701 sculpt: version 20.09 2020-09-18 15:20:18 +02:00
Christian Helmuth
f6337a6446 depot: update recipe hashes 2020-09-18 14:04:56 +02:00
Stefan Kalkowski
f1b3e826d5 sculpt: update building section of sculpt README 2020-09-18 14:04:56 +02:00
Alexander Boettcher
2afba3c137 driver_manager: provide more memory to boot_fb 
Fixes #3896
 2020-09-18 14:04:56 +02:00
Norman Feske
e0d9a04f67 libc: resolve circular dependency in fork 
This patch is a follow-up for the commit "libc: use monitor for fork".
It removes the use of the monitor mechanism from the
'Local_clone_service::close' RPC function because the fork_ep must stay
responsive for the destruction and creation of 'Child' objects.

Issue #3874
 2020-09-18 14:04:56 +02:00
Norman Feske
274f306315 release_notes-20-08.txt: minor style fixes 2020-09-18 14:04:56 +02:00
Christian Helmuth
c59c266afc depot: update recipe hashes 2020-09-17 14:23:14 +02:00
Norman Feske
9aca1ac775 News item for Sculpt 20.08 2020-09-17 14:01:19 +02:00
Norman Feske
3e83b4b39e Update Sculpt documentation for version 20.08 2020-09-17 14:01:19 +02:00
Stefan Kalkowski
d7eb174c88 muen: fix building on modern devel distros 
* Use python2 interpreter not available by default explicitely
* Update ada-bfd binding library to meet binutils-dev >= 2.34
 2020-09-17 10:14:07 +02:00
Stefan Kalkowski
c48b3ca16f okl4: use python2 explicitely in elfweaver tool 
Fix #3892
 2020-09-17 10:14:07 +02:00
Norman Feske
951409f14b intel_fb: reinit capture session on mode change 
The current version of the driver leaves the screen in a black state
after a mode change, e.g., when connecting an external display.
With this patch, the framebuffer content is fetched for the entire
screen after a mode change. Thanks to Sebastian Sumpf for investigating.

Issue #3878
 2020-09-17 10:14:07 +02:00
Norman Feske
2075b119ac sculpt: usability tweaks in popup dialog 
- Show "..." for resource-assignment menu entry because it leads
  to a sub menu.

- Avoid dialog changes when clicking on "Add component" while a
  routing or resource dialog is unfolded.
 2020-09-17 10:14:07 +02:00
Norman Feske
0d61029d7e depot_deploy: move affinity to <affinity> sub node 
The initial implementation of the affinity configuration in
"depot_deploy: support affinity configuration" added the affinity
location attributes to the <start> node of the deploy config. This patch
moves the information into a dedicated <affinity> sub node as done by
the init configuration. So the context of the attributes 'xpos', 'ypos',
'width' and 'height' becomes clear.

It also fixes a usability issue in Sculpt that occurred during testing:
When configuring multiple components with custom affinities, the
resources dialog of later components would wrongly display the state of
earlier components instead of displaying the fresh (default) state. The
resulting configuration would then not match the displayed information.
This is fixed by resetting the dialog state.

As another minor cosmetic change, the patch adds a line break in front
of copied <config> or <heartbeat> nodes.

Issue #3597
 2020-09-17 10:14:07 +02:00
Christian Helmuth
2eb8c5e21a net: move ascii_to() into Net namespace 
The combination of Net::Mac_address and
Genode::ascii_to(Net::Mac_address) required shaky quirks in several
places because GCC is not able to resolve the ascii_to overload if
base/xml_node.h was included to early. The current solution moves the
several ascii_to overloads "closer" to the Net types by putting them
into the Net namespace, where GCC reliably picks them up.

Hence, co-locating the ascii_to() utility with the overload type in the
same scope/namespace is good practice.

This patch removes the now obsolete <nic/xml_node.h> header file.
 2020-09-17 10:13:22 +02:00
Norman Feske
c0f9f2c6d7 sculpt: restore toggleable format button 
With the move of the storage-management dialogs to the graph in Sculpt
version 20.02, the ability to unselect the buttons for destructive
storage operations such as format or expand disappeared. This patch
restores the original behavior.
 2020-09-17 10:13:21 +02:00
Alexander Boettcher
f0f8d0e0ca base: add affinity to session creation request 
Fixes #3838
 2020-09-17 10:13:21 +02:00
Norman Feske
5dc8e330b6 sculpt: add button for restarting a component 
Fixes #3885
 2020-09-17 10:13:21 +02:00
Alexander Boettcher
c9f2847420 foc: make serial esc functional 2020-09-17 10:13:21 +02:00
Stefan Kalkowski
fcb0ca305c dde_linux: don't zero out framebuffer of imx8 
* On this platform there is no need to zero out the framebuffer
  dataspace, which is already blanked by core
* But it might happen that the GUI server is sending a capture
  event before the zero out happens. Thereby screen content can get
  lost

Fix #3878
 2020-09-17 10:13:21 +02:00
Stefan Kalkowski
fd161cd814 dde_linux: turn imx8_fb_drv into Platform client 
* Requests its I/O resources and clock settings from the platform driver

Fix #3877
 2020-09-17 10:13:21 +02:00
Stefan Kalkowski
5913cdae89 os: fix clock settings for i.MX 8M platform_drv 
* Fixes faulty algorithm in fractional pll rate calculation
* Enables clocks that are set as reference clock to prevent system freeze
* Enables/disables root clocks of gates implicitely

Fix #3876
 2020-09-17 10:13:21 +02:00
Norman Feske
8dde14f93e bash: prevent aliasing of libc's 'getenv' 
By default, bash brings its own version of 'getenv', named exactly like
the libc function. This becomes a problem in fork/execve scenarios if
the dynamic linker resolves 'getenv' to the bash binary instead of the
libc.

This patch fixes the generate step of the genodians.org scenario.

Issue #3882
 2020-09-17 10:13:20 +02:00
Norman Feske
f5dc71ed35 os: remove obsolete single_session_service.h 2020-09-17 10:13:20 +02:00
Christian Prochaska
e0ca250232 libc: support detached pthreads 
Fixes #3880
 2020-09-17 10:13:20 +02:00
Christian Helmuth
75ba52a52b libc: remove mutex parameter from monitor 
Issue #3874
 2020-09-17 10:13:20 +02:00
Christian Helmuth
a0a112ffe7 libc: use monitor for fork 
Issue #3874
 2020-09-17 10:13:20 +02:00
Christian Helmuth
a891b3832c libc: use monitor for pthread join/cancel 
Issue #3874
 2020-09-17 10:13:20 +02:00
Christian Helmuth
d6f89b285d libc: use monitor for sleep() 
Issue #3874
 2020-09-17 10:13:20 +02:00
Norman Feske
e9a3f0f095 libc: use monitor for select 
This patch simplifies the control flow within the libc kernel by
facilitating the monitor mechanism for the implementation of 'select'.

Issue #3874
 2020-09-17 10:13:20 +02:00
Christian Helmuth
52a6cf1412 libc: support creat() 2020-09-09 16:57:34 +02:00
Norman Feske
aeeade53f9 sculpt: don't restart menu view if font changes 
This patch enables the menu view's new ability to respond to font-size
changes in Sculpt so that the menu view instanced no longer need to be
restarted whenever the screen resolution or font size is changed.

Fixes #3875
 2020-09-09 16:57:34 +02:00
Norman Feske
c05f716478 menu view: respond to font configuration changes 
Issue #3875
 2020-09-09 16:57:34 +02:00
Norman Feske
4aca94b08b vfs/ttf: support for watching font-size changes 
This patch allows the use of the VFS watch mechanism for the glyph file
of the TTF VFS plugin so that clients become able to dynamically respond
to font reconfigurations.

Issue #3875
 2020-09-09 16:57:34 +02:00
Norman Feske
3031fd2a7d os/vfs.h: const 'Directory' arg for 'Watch_handler' 
The watch mechanism should not require a reference to a mutable
directory.
 2020-09-09 16:57:34 +02:00
Norman Feske
7de2b040f8 vfs: deliver ACKs after config change 
This is needed to enable VFS plugins to notify VFS clients about file
changes that depend on the plugin configuration, E.g., whenever the vfs_ttf
plugin responds to a font-size change, it generates a watch notification
for the glyphs file. Since the change is independent from I/O, we need
to manually call 'handle_io_progress'.
 2020-09-09 16:57:34 +02:00
Norman Feske
0af969543d nitpicker: preserve alpha channel during resize 
This patch reduces flickering artifacts that appear during the
resizing of clients that use an alpha channel.

Issue #3812
 2020-09-09 16:57:34 +02:00
Josef Söntgen
9e97393e3a os: check path for dataspace in ROM VFS plugin 
Fixes #3879.
 2020-09-09 16:57:34 +02:00
Norman Feske
be14e68a83 sculpt/vimrc: 'set nowritebackup' 
By default, Vim renames a written file to a backup file suffixed with
"~" before writing the current buffer to a new file. Consequently, there
exists an intermediate state when no file exists. Should a client watch
such a file to obtain dynamic configuration info, it observes the empty
state.

Some components have builtin heuristics for such a situation. In
particular, the window layouter falls back to a predefined default
'rules' if no rules are provided as a file. So when interactively
editing window-layouter rules using Vim, it can happen that the manually
maintained rules get overwritten by the default rules.

By setting 'set nowritebackup', we can sidestep this issue by preventing
Vim from producing the bad intermediate state.
 2020-09-09 16:57:34 +02:00
Stefan Kalkowski
c1d99630c2 base: reset discarded Alarm object appropriatedly 
Fix #3881
 2020-09-09 16:57:34 +02:00
Norman Feske
328a4fa644 nitpicker: update hovering on view-stack changes 
With the change of nitpicker to the event session interface, the
formerly periodic hovering updates moved to the - now sporadic - input
processing. This has the unfortunate side effect that hovering changes
caused by non-user-input, in particular view-stack changes issued by the
GUI clients that change the view under the current pointer position,
would no longer be reported immediately but only after receiving the
next incoming input event.

This patch reworks the hover handling such that potential hovering
changes due to view-stack operations are evaluated immediately by
those operations, covering the update of the hover report and the
generation of artificial enter/motion events.

Issue #3812
 2020-09-09 16:57:34 +02:00
Christian Helmuth
ff82dc1ad5 doc: activate --depot-auto-update in getting_started.txt 
Also be more explicit about the command to "execute" if depot archives
are missing in depot.inc.
 2020-09-09 16:57:34 +02:00
Norman Feske
17a6318ad6 vfs: process all acks in each iteration 
This patch solves a corner case where one long-active job (e.g.,
read-ready request) stays at the beginning of the '_active_jobs' queue
without an ack. In this case, the '_try_acknowledge_jobs' method would
wrongly stop processing the subsequent acknowledgements. In practice,
this can lead to a delayed sending of acknowledgements until new I/O or
client requests occur. In particular, Vim in Sculpt's inspect window
sometimes did not immediately respond to key presses during tab
completion. Here, the read-ready request of the terminal prevented the
acknowledgement for read of directory entry from being delivered until
the next key was pressed.

Fixes #3873
 2020-09-09 16:57:34 +02:00
Christian Helmuth
0605180a61 libc: remove legacy kernel functions 2020-09-09 16:57:34 +02:00
Christian Helmuth
ab953a534c libc: remove residues of noux fork support 2020-09-09 16:57:34 +02:00
Christian Helmuth
0977574372 vfs/rtc: return timestamp size on stat() 2020-09-01 16:48:50 +02:00
Alexander Boettcher
99bdfbe36f trace: don't account argument_buffer 2x 
The quota for the argument buffer is already accounted by using the
Attached_ram_dataspace _argument_buffer, which uses the Constraint_ram_allocator
_ram, which uses the Ram_quota_guard from the Session_object. Running on
Sculpt with more than 1000 Subject_info objects/trace IDs the memory
waste become noticeable.
 2020-09-01 16:48:29 +02:00
Christian Helmuth
f70b02ae3b vbox: remove logger libc plugin 
Fixes #3870
 2020-08-28 14:48:05 +02:00
Christian Helmuth
574e41119a sculpt: version 20.08 2020-08-28 14:34:22 +02:00
Christian Helmuth
8d31590a66 version: 20.08 2020-08-28 10:31:12 +02:00
Norman Feske
8a5277d53f News item for version 20.08 2020-08-28 10:18:07 +02:00
Norman Feske
fb6549a092 Release notes for version 20.08 2020-08-28 10:18:07 +02:00
Christian Helmuth
c649307720 depot: update recipe hashes 2020-08-28 08:29:12 +02:00
Christian Helmuth
bf0f3b65b4 Replace remaining 'constrain_phys' by 'managing_system' 2020-08-28 08:28:13 +02:00
Norman Feske
b9a21ee3ae nitpicker: updated README 2020-08-28 08:28:13 +02:00
Piotr Tworek
8d790010bf Xml_generator: Remove unused member variable 
This triggers a warning when building the code with clang.

Fixes #3868
 2020-08-28 08:28:13 +02:00
Stefan Kalkowski
e29485fa59 dde_linux: decouple Lx_kit::Irq and Platform::Device 
Follow-up commit to fix the old usb_drv. Under rpi one and the
same irq can be requested several times. Therefore, we've to track
the Irq_connection objects.

Ref #3865
 2020-08-28 08:28:13 +02:00
Christian Prochaska
ace172ebf3 qt5: prerequisites for qtwebengine support 
Fixes #3867
 2020-08-28 08:28:13 +02:00
Norman Feske
bbfda019df vfs/terminal: minor const improvement 
Issue #3860
 2020-08-25 12:03:31 +02:00
Norman Feske
5cc2adb421 pkg/sculpt: update README 
Adaptation to the change of the input filter to the event filter.
 2020-08-25 12:02:37 +02:00
Norman Feske
90b3b72a91 doc: update components.txt 
This patch adjusts the components.txt to the recent changes of the GUI
stack.

Issue #3812
 2020-08-25 12:02:36 +02:00
Christian Helmuth
dd4a3b0263 vbox: direct screen updates via NotifyUpdate() 2020-08-25 12:02:36 +02:00
Norman Feske
7d0cb9620b depot: update recipe hashes 2020-08-25 11:50:41 +02:00
Christian Helmuth
2c29bf5a21 tool_chain_auto: depend on zlib from depot 
This prevents run-script failures in case hashes are out-of-date.
 2020-08-25 11:50:12 +02:00
Piotr Tworek
80bc1cff5f run: Use Qemu 5.0.0 compatile gic-version param. 
It seems Qemu 5.0.0 no longer supports gic_version parameter.
Apparently this only worked due to some compat feature that
was dropped in Qemu 5.0.0. The replacement which works on both
old and new Qemu versions is gic-version.

Fixes #3823
 2020-08-25 11:50:12 +02:00
Stefan Kalkowski
a5f6d0f081 dde_linux: use new platform API for usb_host_drv 
Introduce a common platform device initialization routine for all
ARM SoCs by using the new platform driver API.

Fix #3865
 2020-08-25 11:50:12 +02:00
Stefan Kalkowski
e223be32ce dde_linux: decouple Lx_kit::Irq and Platform::Device 
The Lx_kit::Irq abstraction of DDE Linux was always using the very first
interrupt of a Platform::Device stored in it. Instead of handing over the
platform device it is much more flexible and sane to hand over the actual
interrupt capability.

Ref #3865
 2020-08-25 11:50:12 +02:00
Stefan Kalkowski
73d3698e2f os: add new platform driver for Raspberry Pi 
Fix #3864
 2020-08-25 11:50:12 +02:00
Stefan Kalkowski
5f5ad41ad3 hw: unify irq enumeration for Raspberri Pi 
By now, the enumeration of peripheral interrupts on Raspberry Pi 1 was
different in between base-hw kernel and Fiasco.OC. Therefore, hacks were
needed in every driver to request the correct interrupt number dependent
on the kernel. Before reproducing the same in the platform driver for rpi,
we can more easily use the same enumeration with base-hw.

Ref #3864
 2020-08-25 11:50:12 +02:00
Stefan Kalkowski
7ba31d4447 os: disable initialization of i.MX8MQ platform_drv 
This is a temporary workaround to not harm drivers, which aren't
converted yet to request all device resources including power and
clocks from this new platform driver.

Ref #3863
 2020-08-25 11:50:12 +02:00
Stefan Kalkowski
224f5907b2 os: add platform driver for i.MX 8M Quad SoC 
The new platform driver supports clock and power-domain settings per device.
IOMUX constroller settings are not part of the driver yet.

Fix #3863
 2020-08-25 11:50:12 +02:00
Stefan Kalkowski
70f98fcc44 hw: implement Pd_session::managing_system for ATF 
To access the ARM Trusted Firmware from the platform driver
fill the new `managing_system` call of the PD session with life resp.
do a SMC call on behalf of the client.

Fix #3816
 2020-08-25 11:50:11 +02:00
Stefan Kalkowski
06edc0d52b base: extend PD session with managing_system call 
Introduce the managing_system privilege for components like the
platform_driver to allow it to call system management functionality
that is reserved by kernel or special firmware, e.g., ARM Trusted Firmware.

The former RAM resource configuration attribute `constrain_phys`,
which enabled to constrain the region of physical RAM to be used,
gets replaced by the new, broader managing_system configuration
attribute of a `start` node. It gets enforced by the sandbox library.

Ref #3816
 2020-08-25 11:50:11 +02:00
Christian Prochaska
af6d2a8c54 stdcxx: add abi symbols needed by qtwebengine 
Fixes #3862
 2020-08-25 11:50:11 +02:00
Josef Söntgen
157f4b1270 pthread: implement condattr setclock 
Normally CLOCK_REALTIME is used. However libraries, like glib, want to
use CLOCK_MONOTONIC. To make those users happy add setting the clock.

Note, the pthread_cond implementation uses the POSIX semaphore API
internally that does not have means to set the clock. For this reason,
the private 'sem_set_clock' function is introduced.

Fixes #3846.
 2020-08-25 11:50:11 +02:00
Martin Stein
66ff18a53e libc: support pthread mutex type "Adaptive NP" 
The initial motivation for enabling support for this pthread mutex type was
that it is required by the Glib test in genode/world.

Fixes #3817
 2020-08-25 11:50:11 +02:00
Norman Feske
2a4adc895c Remove dithering from menu_view, themed decorator 
With the change from RGB565 to RGB888, the application of dithering has
become obsolete.

Issue #3784
 2020-08-25 11:50:11 +02:00
Josef Söntgen
d21bc8268b vfs: add raw option to terminal plugin 
When the 'raw' attribute is set to 'yes' the terminal VFS plugin will
ignore control characters. This in necessary for terminal connections
that transport data that contain such characters as part of message.

Fixes #3860.
 2020-08-25 11:50:11 +02:00
Norman Feske
8b556a9435 usb_block: fix upper block-number range check 
The driver wrongly rejected a block request for the very last block of
the device, which prevented part_block from successfully parsing the
partition table (when attempting to access the GPT backup).

Fixes #3861
 2020-08-25 11:50:11 +02:00
Norman Feske
a68a6665ac sculpt: remove part_block upon failure 
With this patch, sculpt uses init's heartbeat-monitoring mechanism to
detect the failure of part_block instances during storage disovery.
If part_block gets stuck, the device is released and can thereby
be accessed at the whole-device level.

Issue #3861
 2020-08-25 11:50:11 +02:00
Norman Feske
1e7c94759d part_block: spelling fix 2020-08-25 11:50:11 +02:00
Christian Prochaska
9ec2a19cc0 libc: add wrapper function for 'Genode::cache_coherent()' 
Fixes #3858
 2020-08-25 11:50:11 +02:00
Christian Prochaska
9bc6b8be5a libc: handle 'MAP_SHARED' flag in 'mmap()' 
Fixes #3857
 2020-08-25 11:50:11 +02:00
Norman Feske
f8f8b665c3 vfs: don't accept new job before old job is acked 
Fixes #3859
 2020-08-25 11:50:11 +02:00
Christian Prochaska
31397d67ae base: increase linker area size 
Fixes #3856
 2020-08-25 11:50:11 +02:00
Christian Prochaska
dd8777093d libc: don't treat 'mmap()' address hint w/o MAP_FIXED flag as error 
Fixes #3855
 2020-08-25 11:50:10 +02:00
Christian Prochaska
2f55ffdf20 libc: zero-initialize anonymous memory from 'mmap()' 
Fixes #3854
 2020-08-25 11:50:10 +02:00
Josef Söntgen
7c1a4522d6 virtualbox5: enable storage device flushing 
The patches disable VBOX_IGNORE_FLUSH and a sanity check in the VMDK
backend. This enables passing an explicit flush request by the guest
down to the VFS.

Fixes #3743.
 2020-08-25 11:50:10 +02:00
Norman Feske
3e8824908d system_shell: increase gui_fb quota 
With this change, the window can be maximized when using a 1920x1080
resolution with 32 bits per pixel.

Issue #3784
 2020-08-25 11:50:10 +02:00
Norman Feske
28714979bd sculpt: drop storage-detect-cancellation feature 
Fixes #3853
 2020-08-25 11:50:10 +02:00
Norman Feske
658091bfad ttf_font: add sanity check for invalid scale value 
The check prevents the Ttf_font from violating the bounding box in the
presence of very small scale values. This can happen during the startup
of Sculpt. Before the framebuffer driver is up, Sculpt bases its dynamic
font-size setting on a screen resolution of 1x1.

Issue #3812
 2020-08-25 11:50:10 +02:00
Norman Feske
e8e14ad1bf test-libc_execve: adjust timeout for pistachio 2020-08-25 11:50:10 +02:00
Martin Stein
e544464354 libsparkcrypto: src recipe, contrib alis, mk fixes 
* add libsparkcrypto source-recipe
* provide ALI files through a new repository by mstein and add the repo
  download to the libsparcrypto port-file
* remove dependencies to non-existent contrib ADB files from the library make
  files

Fixes: #3852
 2020-08-25 11:50:10 +02:00
Norman Feske
969e59c599 terminal: consider border in size calculation 2020-08-25 11:50:10 +02:00
Norman Feske
f2d7f7aa6e sculpt: remove decor frame from panel view 2020-08-25 11:50:10 +02:00
Norman Feske
a59b69758b sculpt: remove deferred GUI initialization 
The sculpt manager used to defer the initialization of the GUI until
nitpicker's first display report became ready. This way, Sculpt was able
to run headlessly even if the framebuffer driver failed to start up.
Thanks to #3827, nitpicker no longer depends on a working framebuffer
driver. So the sculpt manager is safe to rely on nitpicker in any case,
simplifying the code.

Issue #3827
 2020-08-25 11:50:10 +02:00
Norman Feske
c7d9b2ca92 nitpicker: fix mode change for requested fb 
This is a follow-up commit to "nitpicker: make framebuffer and input
optional". It restores the dynamic mode-change support when using
'request_framebuffer="yes"' as needed in scenarios where multiple
nitpicker instances are used in a cascaded way. E.g., Sculpt's
Leitzentrale. The previous version missed to reconstruct the
'_fb_screen' on mode changes.

Issue #3812
 2020-08-25 11:50:10 +02:00
Norman Feske
2437d759b6 Xml_node: remove deprecated methods 
Issue #3755
 2020-08-25 11:50:10 +02:00
Norman Feske
0434cb6fd6 gdb_monitor.run: cap-quota adjustment 2020-08-25 11:50:10 +02:00
Norman Feske
556a5c8086 libc: clean up 'clock_gettime' handling 
This patch untangles the dependency of VFS operations that need RTC
information from the 'clock_gettime' libc function that must never be
called from the libc kernel context.

- The 'Rtc' class uses the VFS directly for reading the rtc file instead
  of relying on libc functions.

- The 'Rtc' instance has become part of the 'Kernel' instead of
  being construced as a side effect of the first call of
  'clock_gettime'.

- Changed 'Rtc::read' to return a timespec value, which has a higher
  precision than the formerly used time_t value.

- The 'Rtc::read' returns a value with the relative 'current_time'
  already applied. The former handling of subsequent rtc-value
  updates has been rewritten to become more logical.

- The 'Vfs_plugin' no longer calls 'clock_gettime' but the new
  kernel-level 'Current_real_time' interface.

Issue #2635
 2020-08-25 11:50:10 +02:00
Norman Feske
852ab79359 Move atexit handling from base lib to libc 
This patch untangles the interplay of the base library and the libc
during the exit handling.

- The CXA ABI for the atexit handling is now provided by the libc.
  For plain Genode components without libc dependency, __cxa_atexit
  is a no-op, which is consistent with Genode's notion of components.

- The 'abort' implementation of the base library no longer calls
  'genode_exit' but merely 'sleep_forever'. This way, the cxx library
  no longer depends on a 'genode_exit' implementation.

- The libc provides 'atexit' support by storing metadata on the
  libc kernel's heap now, thereby eliminating the former bounded
  maximum number of atexit handlers.

- Shared-library dtors are no longer called via the atexit mechanism
  by explicitly by the dynamic linker. This slightly changes the
  call order of destructors (adjustment of the ldso test). Functions
  marked as destructors are called after the atexit handlers now.

- The libc executes atexit handlers in the application context,
  which supports the I/O operations in those handles, in particular
  the closing of file descriptors.

Fixes #3851
 2020-08-25 11:50:09 +02:00
Christian Prochaska
66063e5137 qt5_component: support arguments and environment from config 
Fixes #3049
 2020-08-25 11:50:09 +02:00
Josef Söntgen
f03917ab7c aes_cbc_4k: api recipe 
Fixes #3850
 2020-08-25 11:50:09 +02:00
Martin Stein
5ed528a2ad aes_cbc_4k: import file for include directories 
Ref #3850
 2020-08-25 11:50:09 +02:00
Josef Söntgen
81ae4599ae libsparkcypto: api recipe 
For the time being, we put all spec and body files into the recipe although
there might be a (so far to us unknown) way to have only the API relevant files
in it.

Fixes #3849
 2020-08-25 11:50:09 +02:00
Martin Stein
82090d2ea1 libsparkcrypto: mark some units pure 
* switch to a libsparkcrypto fork of m-stein
* switch to a libsparkcrypto state that makes some units pure that are used by
  the consistent block encrypter, in order that the latter can become
  completely pure

Ref #3849
 2020-08-25 11:50:09 +02:00
Martin Stein
2e3b11b354 ada-runtime: add s-expint.ad* 
* switch to fork of the Ada runtime provided by m-stein
* switch to a Ada-runtime state that provides the Exp_Int package (exponential
  function on integers)
* adapt spark lib, symbols, and recipes to incorporate the Exp_Int package

Fixes #3848
 2020-08-25 11:50:09 +02:00
Martin Stein
f3eaeb08ef ada/spark: all warnings, warn strict, style checks 
* enable all common warnings through default value of CC_ADA_WARN
* treat warnings like errors through default value of CC_ADA_WARN_STRICT
* enable almost all style checks through default value of CC_ADA_WARN_STRICT
* style fixes for aes_cbc_4k
* disable strict warnings and style checks for libsparkcrypto and spark lib

Ref #3848
 2020-08-25 11:50:09 +02:00
Alexander Boettcher
3f97269988 sculpt_manager: add graphical CPU affinity config 
Fixes #3820
 2020-08-25 11:50:09 +02:00
Norman Feske
1aed881313 Remove drivers/gpio/spec directory 
Issue #2190
 2020-08-25 11:50:09 +02:00
Norman Feske
3c78265e66 Remove drivers/framebuffer/spec directory 
Thanks to the unification of build directories, the sub directories
(rpi, sdl, imx53) do not depend on SPEC values.

Issue #2190
 2020-08-25 11:50:09 +02:00
Norman Feske
fcb21732e0 Move drivers/input/spec/ps2 to drivers/ps2 
This simplifies the directory structure.

Issue #2190
 2020-08-25 11:50:09 +02:00
Norman Feske
db0e86fa41 Remove drivers/input/dummy pseudo driver 
With the transition from the input-session to the event-session
interface, the use case for this component has vanished.

Issue #3827
 2020-08-25 11:50:09 +02:00
Norman Feske
2e22498e5a nitpicker: no framebuffer and input by default 
This patch prevents nitpicker from requesting a framebuffer and input
session by default because the regular use of nitpicker relies on the
capture-session and event-session interfaces by now.

For supporting the nested use of nitpicker via the gui_fb component, it
is still possible to enable the traditional behavior by explicitely
setting the 'request_input' and 'request_framebuffer' config attributes
to "yes".

Issue #3812
 2020-08-25 11:50:09 +02:00
Norman Feske
264f695373 Remove transitionary input_event_client component 
With the transition of the input servers to event clients completed,
this helper component is no longer needed.

Issue #3845
Issue #3827
 2020-08-25 11:50:09 +02:00
Norman Feske
a2381c7e4c Turn USB HID drivers into event-session clients 
Issue #3845
 2020-08-25 11:50:09 +02:00
Norman Feske
974118acec Turn PS/2 driver into event-session client 
Issue #3845
 2020-08-25 11:49:45 +02:00
Christian Prochaska
08ef528577 stdcxx: enable thread features 
Issue #2442
 2020-08-25 11:49:45 +02:00
Norman Feske
50b367c076 Remove dde_zircon 
Fixes #3847
 2020-08-25 11:49:44 +02:00
Christian Helmuth
36b6ebc030 libc: process VFS operations in monitor 
Fixes #2635
 2020-08-25 11:49:44 +02:00
Christian Helmuth
a89d61acf2 base: call Io_progress_handler only in signal RPC 
This remove the call to Io_progress_handler::handle_io_progress() from
wait_and_dispatch_one_io_signal() to prevent unexpected nesting
I/O-progress handling in case of custom dispatch loops (e.g., in libc).
The original intention of Io_progress_handler was to inform the
entrypoint just before blocking in the entrypoint loop.

Issue #2635
 2020-08-25 11:49:44 +02:00
Norman Feske
9e42a9ac7e Replace server/acpi_input with app/acpi_event 
Fixes #3827
 2020-08-25 11:49:44 +02:00
Norman Feske
19690193a4 Remove obsolete use of input_session/connection.h 2020-08-25 11:49:44 +02:00
Norman Feske
78047b5bd8 Rename fb_boot_drv to boot_fb_drv 
to be consistent with the naming of the other drivers.
 2020-08-25 11:49:43 +02:00
Norman Feske
9662d89cfb Replace input filter with event filter 
This commit applies the transition from the "Input" session to the "Event"
session to the event-filtering mechansim. The functionality of the
input_filter is now provided by the event_filter. The event filter
requests only one "Event" session as destination for the filter result,
which is usually routed to the nitpicker GUI server. It provides an
"Event" service to which any number of event sources can connect.

The configuration of the filter chain remains almost the same. Only the
declaration of the <input> nodes is no longer needed. Instead, the
configuration must specify <policy> nodes, which define the mapping of
"Event" clients (event sources) to the inputs used in the filter chain.

The patch adjusts all uses of the nitpicker GUI server accordingly such
that the event filter reports events to nitpicker's event service
instead of having nitpicker request an "Input" session. This dissolves
the dependency of nitpicker from input drivers.

Issue #3827
 2020-08-25 11:49:43 +02:00
Norman Feske
bc5b161260 os: event_dump server for printing events 
This component takes over the role of the test-input component.

Issue #3827
 2020-08-25 11:49:43 +02:00
Norman Feske
c502e1d095 input_filter: prepare for refactorization 
This patch brings the input filter into a shape that is easier to
re-mold into an event filter, reversing the client/server roles of
the component.

* The 'Sink &destination' is no longer passed as constructor argument
  to the individual filters but passed as argument to the 'generate'
  method. This way, the final destination does not need to exist at
  the construction time of the filter chain but can be created on
  the fly (clearing the way for using 'Event::Client::with_batch').

* A new 'Source::Filter' interface with the 'apply' method aids the
  cascading of filters during 'generate'. The modules now implement
  the 'Source::Filter::filter_event' interface instead of the
  'Source::Sink::submit_event' interface.

* Since the 'Sink &destination' is no longer a member of the filter
  modules, character-repeat events can no longer be emitted in an
  ad-hoc way. Instead, the character-repeat mechanism now invokes
  a new 'Trigger::trigger_generate' hook that prompts the execution
  of the regular 'generate' mechanism by the main program.

This patch is supposed to leave the semantics of the input filter
unchanged (validated by the input_filter.run script).

Issue #3827
 2020-08-25 11:49:43 +02:00
Norman Feske
6c4ce86a34 event_session: make Session_client::Batch abstract 
This way, there can exist multiple implementations of the interface such
as event filters.

Issue #3812
 2020-08-25 11:49:43 +02:00
Norman Feske
7bac1ec7e5 Adapter for connecting input and event servers 
This is an intermediate helper component for the transition of input
servers into event clients. It works analogously to the
input_event_bridge but it connects servers instead of clients.
In particular, it can act as glue between a traditional input driver
that provides an input service and the event filter that provides an
event service. Thanks to this component, each driver can be migrated to
the event-client interface individually.

Issue #3827
 2020-08-25 11:49:43 +02:00
Norman Feske
7932c76d85 nitpicker: add event service 
Issue #3812
 2020-08-25 11:49:43 +02:00
Norman Feske
66c520cdae Remove stale virtualbox_nic_router.run script 
The run script is not regularly tested and thereby has become
nonoperational since a long time.
 2020-08-25 11:49:43 +02:00
Christian Helmuth
9453287a6b libc: move current working directory into kernel 2020-08-25 11:49:43 +02:00
Christian Helmuth
40e936911f libc: remove unused member from Errno 2020-08-25 11:49:43 +02:00
Christian Helmuth
d9121e50c3 libc: translate file-descriptor exhaustion into EMFILE 
Fixes #3841
 2020-08-25 11:49:43 +02:00
Christian Helmuth
cb15f40028 Remove remnants of libc_lxip 
Issue #2960
 2020-08-25 11:49:42 +02:00
Alexander Boettcher
ce1d0464b8 ssh_terminal: destroy libc fd within user context 
Fixes #3840
 2020-08-25 11:49:42 +02:00
Alexander Boettcher
bed531b604 libc: initialize tv struct in pselect 
Fixes #3839
 2020-08-25 11:49:42 +02:00
Christian Helmuth
257b3b6775 libc: cleanup monitor implementation 
- Explicit types for function and monitor execution results
- Remove pending flag and mutex (pending flag was moved to kernel)
 2020-08-25 11:49:42 +02:00
Christian Helmuth
420ed91480 linux_nic_drv: stop receiver thread on congested queue 
Otherwise, the thread spins on and drops packets.
 2020-08-25 11:49:42 +02:00
Christian Prochaska
c6b17be744 vfs/lwip: clear the packet queue in '~Udp_socket_dir()' 
Fixes #3836
 2020-08-25 11:49:42 +02:00
Christian Helmuth
d266f44ef3 libc: free resources on socket creation failure 
Issue #3766
 2020-08-25 11:49:42 +02:00
Christian Helmuth
5a2ac73b69 vfs/lwip: prevent pbuf leakage on ENOTCONN 
Issue #3766
 2020-08-25 11:49:42 +02:00
Christian Helmuth
7996fc45f3 vfs/lwip: signal ENOTCONN via EOF on remote file 
Issue #3766
 2020-08-25 11:49:42 +02:00
Alexander Boettcher
a03a37b1d9 vfs/lwip: avoid leaking of pbuf objects 
Issue #3766
 2020-08-25 11:42:37 +02:00
Christian Helmuth
76ac94f4f2 vfs/lwip: fix pbuf slab-block size 
Issue #3766
 2020-08-25 11:42:37 +02:00
Alexander Boettcher
f4d3231034 cpu_sampler: handle all quota upgrades 
Beforehand cap_quota upgrades got ignored which leads to non working clients.

Issue #3826
 2020-08-25 11:42:37 +02:00
Alexander Boettcher
b7ffeb51aa cpu_sampler: avoid spinning on unavailable state 
Fixes #3826
 2020-08-25 11:42:37 +02:00
Alexander Boettcher
ed15a46ca4 base: construct child process only once 
Fixes #3821
 2020-08-25 11:42:36 +02:00
Alexander Boettcher
c55a499009 base: remove delayed dispatch from Rpc_entrypoint 
Fixes #3833
 2020-08-25 11:42:36 +02:00
Alexander Boettcher
d1be1281bc os: use whole memory in Packet_allocator 2020-08-25 11:42:36 +02:00
Christian Prochaska
377dadd461 libc_getaddrinfo.run: fix run script 
Fixes #3832
 2020-08-25 11:42:36 +02:00
Christian Prochaska
353baa9251 libc: add pthread alias functions with underscore 
The libc calls pthread functions with underscore internally.

Issue #725
 2020-08-25 11:42:36 +02:00
Alexander Boettcher
60106ac2c8 base: avoid deadlock and page fault 
Fixes #3830
 2020-08-25 11:42:29 +02:00
Christian Helmuth
de7d4a5523 Replace libc_pipe plugin by VFS pipe plugin 2020-07-30 08:49:24 +02:00
Christian Prochaska
9b6cc75f1c tool/seccomp: allow 'restart_syscall' on x86_64 
Fixes #3831
 2020-07-30 08:49:23 +02:00
Christian Helmuth
18b26ff595 vfs/pipe: never report read_ready on pipe/out handle 2020-07-30 08:49:23 +02:00
Christian Helmuth
e44ffa02c9 libc: remove redundant dummies (fstatat, stat) 2020-07-30 08:49:23 +02:00
Christian Helmuth
5122917d62 os: declare Path::max_len() const 2020-07-30 08:49:23 +02:00
Alexander Boettcher
3639a1af80 acpi_drv: avoid out-of-bound access to FADT table 
Fixes #3829
 2020-07-30 08:49:23 +02:00
Alexander Boettcher
851b842033 vbox5-generic: use pthread_cond_timedwait for halt 
use similar implementation as introduced for vbox5-nova

vbox5-nova: avoid Blocking_canceled exception

Related to #3810
 2020-07-30 08:49:23 +02:00
Alexander Boettcher
072a00ba18 vbox5: remove ubuntu 14_04 test 2020-07-30 08:49:23 +02:00
Alexander Boettcher
7e0b66835b vbox5: show serial for ubuntu VMs during autopilot 2020-07-30 08:49:23 +02:00
Alexander Boettcher
73e671893b nova: remove obsolete local vcpu case 
Seoul was the last user of Vcpu_same_pd
 2020-07-30 08:49:23 +02:00
Christian Prochaska
fd682cd470 hw: check descriptor validity in LPAE lookup 
Fixes #3828
 2020-07-30 08:49:23 +02:00
Norman Feske
bfd94d64ba pbxa9_drivers: add blit api 
This is a follow-up fix of "Turn framebuffer drivers into capture
clients".

Fixes #3813
 2020-07-30 08:49:23 +02:00
Piotr Tworek
d8539af412 platform_session: Add missing include. 
The code uses Genode::Attached_dataspace but fails to include header
defining this type.

Fixes #3824
 2020-07-30 08:49:23 +02:00
Norman Feske
cd7a6fc9fe pkg/drivers_interactive-muen: use capture session 
This is follow-up commit to "Turn framebuffer drivers into capture
client".

Issue #3813
 2020-07-30 08:49:23 +02:00
Alexander Boettcher
41380ff769 base: remove Cancelable_lock 
- base/cancelable_lock.h becomes base/lock.h
- all members become private within base/lock.h
- solely Mutex and Blockade are friends to use base/lock.h

Fixes #3819
 2020-07-30 08:49:23 +02:00
Alexander Boettcher
0ed7367c97 vbox5-nova: avoid using semaphore used by Lock 
Issue #3819
 2020-07-20 15:59:47 +02:00
Alexander Boettcher
eab09a2f7c nova: avoid Blocking_canceled exception in timer 
Issue #3819
 2020-07-20 15:59:47 +02:00
Norman Feske
de795b1a6e depot: update recipe hashes 2020-07-13 11:33:53 +02:00
Alexander Boettcher
73546a135a seoul: increase VMM memory for vm_stress* 
The memory was not sufficient if more than 2 vcpus are used. The test starts
up to 4 vCPUs.
 2020-07-13 11:33:15 +02:00
Alexander Boettcher
c888d856ee vbox5*.run: disable in autopilot usb usage 
to avoid to spam the log with usb messages about our usb dongle sending
regularly input events
 2020-07-13 11:33:15 +02:00
Norman Feske
c81af531a3 Turn framebuffer drivers into capture clients 
This patch replaces the use of the "Framebuffer" session interface by
the new "Capture" session interface in all framebuffer drivers. Thanks
to this change, those drivers have become mere clients of the nitpicker
GUI server now, and are no longer critical for the liveliness of the GUI
server.

The patch touches the following areas:

- The actual driver components. The new versions of all drivers have
  been tested on the respective hardware. Generally, the drivers
  have become simpler.

- The drivers_interactive packages for various boards. The drivers
  subsystem no longer provides a "Framebuffer" service but needs a
  valid route to the "Capture" service provided by nitpicker.

- The driver manager of Sculpt OS.

- This patch changes the role of the test-framebuffer component from a
  framebuffer client to a capture server so that drivers (capture clients)
  can be directly connected to the test component without the nitpicker
  server.

- Framebuffer driver no longer support the unbuffered mode.

- The fb_bench.run script is no longer very meaningful because it
  interplays solely with nitpicker, not with the driver directly.

- All run scripts for graphical scenarios and the related depot
  archives got adapted to the change.

Fixes #3813
 2020-07-13 11:33:15 +02:00
Norman Feske
6743669ab8 nitpicker: report no mode smaller than 1x1 pixels 
Issue #3812
 2020-07-13 11:33:14 +02:00
Norman Feske
320387db89 nitpicker: improve GUI client resize handling 
This patch replaces meta-data allocation during the resize handling by a
new 'Resizeable_texture' type that has all meta data preallocated.
It also replaces the use of pointer return values with the
'Resizeable_texture::with_texture' method.

Issue #3812
 2020-07-13 11:33:14 +02:00
Norman Feske
4cad1a87df nitpicker: make framebuffer and input optional 
Nitpicker used to unconditionally request a framebuffer and input
session. With the transition to the new capture/event session
interfaces, this built-in policy does no longer suffice.

This patch introduces the attributes 'request_framebuffer="yes"' and
'request_input="yes"' (with the default values shown) to nitpicker's
<config> node. If setting those attributes to "no", nitpicker won't
request a "Framebuffer" and "Input" session respectively.

Issue #3812
 2020-07-13 11:33:14 +02:00
Norman Feske
22cb6dded7 nitpicker: add capture service 
Issue #3812
 2020-07-13 11:33:14 +02:00
Norman Feske
537b317273 nitpicker: handle failed try_alloc_texture 
Fixes #3782
 2020-07-13 11:33:14 +02:00
Norman Feske
795a817a33 nitpicker: remove dirty_rect state from view stack 
In the presence of potentially multiple output back ends, this
dirty_rect state must be maintained individually per back end. Instead
of storing the dirty_rect as view-stack member, the view stack now calls
a new 'Damage::mark_as_damaged' interface, which allows nitpicker to
propagate this information to multiple back ends. Unfortunately, the
patch must remove the per-view dirty_rect state.

Issue #3812
 2020-07-13 11:33:14 +02:00
Norman Feske
067a7ad7e9 nitpicker: internal cleanup 
This patch simplifies the internal naming of the nitpicker GUI server as
a preparatory step for adding support for the capture session interface.

Issue #3812
 2020-07-13 11:33:14 +02:00
Norman Feske
3863de9589 Input-event-bridging service 
The sole purpose of this service is the migration path from the use of
input session interface to the event session interface.

Issue #3812
 2020-07-13 11:33:14 +02:00
Norman Feske
443d3c98dd Event session interface 
Issue #3812
 2020-07-13 11:33:13 +02:00
Norman Feske
602def9bdd Simple example for using a capture connection 
Issue #3812
 2020-07-13 11:33:13 +02:00
Norman Feske
3d1d1b439d Capture session interface 
Issue #3812
 2020-07-13 11:33:13 +02:00
Josef Söntgen
49e907e5f6 libc: remove trailing slash in lstat and stat 
Fixes #3814.
 2020-07-13 11:33:13 +02:00
Alexander Boettcher
c4a8c6798b top: react on too small trace argument buffer size 
Issue #3811
 2020-07-13 11:33:13 +02:00
Alexander Boettcher
21e48a8e12 trace: extend for_each_subject_infos return value 
Fixes #3811
 2020-07-13 11:33:13 +02:00
Norman Feske
a7d170adda os: add blit/painter.h 
This utility eases the application of the 'blit' function when using the
'Texture' and 'Surface' types.
 2020-07-13 11:33:13 +02:00
Norman Feske
91b71f7a9f run: add drivers_interactive for panda and arndale 2020-07-13 11:33:13 +02:00
Alexander Boettcher
641679f7e7 vbox5-nova: avoid Blocking_canceled exception 
using pthread primitives to implement block/wakeup of EMT thread

Fixes #3810
 2020-07-13 11:33:13 +02:00
Christian Helmuth
ed4594c76b Fix success/failure detection in cpu_bench testsuite 
Issue #3785
 2020-07-13 11:33:12 +02:00
Alexander Boettcher
997a77b3de foc: use Mutex 
Issue #3809
 2020-07-13 11:33:12 +02:00
Alexander Boettcher
6fa4307005 foc: keep core internal pager state separately 
beforehand the Lock object and more was accessible
via FOC native cpu thread RPC

Fixes #3809
 2020-07-13 11:33:12 +02:00
Alexander Boettcher
f3efbe50bb base: remove deprecated cancel_blocking() support 
for threads.

Fixes #3806
 2020-07-13 11:33:12 +02:00
Alexander Boettcher
ce6f4dffe5 linux: use Mutex 
Fixes #3807
 2020-07-13 11:33:12 +02:00
Norman Feske
a0b0892df3 UEFI/boot_fb: constrain Qemu to 1920x1080 
Fixes #3808
 2020-07-13 11:33:12 +02:00
Christian Helmuth
e163055f6a ram fs: Release node mutex on unlink 
This is needed if an open file is unlinked.

Fixes #3800
 2020-07-13 11:33:12 +02:00
Alexander Boettcher
7acc34b48b dde_zircon: use Mutex 
Fixes #3805
 2020-07-13 11:33:12 +02:00
Alexander Boettcher
a9827c662e dde_ipxe: use Mutex 
Issue #3805
 2020-07-13 11:33:12 +02:00
Alexander Boettcher
9a8a42e819 dde_rump: use Mutex/Blockade 
Issue #3805
 2020-07-13 11:33:12 +02:00
Alexander Boettcher
5516dbcb1f dde_linux: use Mutex/Blockade 
Issue #3805
 2020-07-13 11:33:11 +02:00
Alexander Boettcher
b7cdb5840a dde_bsd: remove unused Lock 
Issue #3805
 2020-07-06 14:43:05 +02:00
Alexander Boettcher
61d798c629 launchpad: use Mutex 
Fixes #3804
 2020-07-06 14:43:05 +02:00
Alexander Boettcher
973fbc98be nitlog: use Mutex 
Issue #3804
 2020-07-06 14:43:05 +02:00
Alexander Boettcher
3d634df34d gems: remove report_rom_slave.h 
Fixes #3803
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
ad284491e6 mixer_gui_qt: use Blockade 
Issue #3803
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
f946de4450 tcp_terminal: use Mutex 
Issue #3803
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
377c6ed0ce ssh_terminal: use Mutex 
Isse #3803
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
1609d8a92f gdb_server: use Mutex 
Fixes #3802
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
169cad6059 seoul: use Mutex 
Issue #3802
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
bba4790002 virtualbox5: use Mutex/Blockade 
Issue #3802
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
c943e26913 lib/drm: use Blockade 
Fixes #3801
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
7d1fd0b0a3 lwip: use Mutex 
Issue #3801
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
6d653b3c0f qemu-usb: use Mutex 
Issue #3801
 2020-07-06 14:43:04 +02:00
Alexander Boettcher
5936d00b5e acpica: use Mutex 
Issue #3801
 2020-07-06 14:43:04 +02:00
Christian Prochaska
5af4552511 qt5: build Qt libraries and applications with qmake 
Fixes #3795
 2020-07-03 13:48:55 +02:00
Norman Feske
0310c733d5 base-linux: let seccomp permit 'read' 
This is needed for using the 'wait_for_continue' debug mechanism.

Fixes #3798
 2020-07-03 11:58:31 +02:00
Christian Helmuth
0e6c32f75e libc: use Mutex/Blockade 
Fixes #3796
 2020-07-03 11:16:20 +02:00
Christian Helmuth
93ab972ddc tool/seccomp: allow sigreturn on x86 
Fixes #3799
 2020-07-03 11:15:33 +02:00
Norman Feske
35c3acdf05 depot: update recipe hashes 2020-07-03 11:13:59 +02:00
Alexander Boettcher
103236fdca nitpicker: fix memory accounting during realloc 
Fixes #3794
 2020-07-03 11:11:24 +02:00
Christian Prochaska
62848b1a68 mk: set soname in abi libraries 
Fixes #3793
 2020-07-03 11:11:23 +02:00
Alexander Boettcher
64ef651d4d libc: handle MSG_WAITALL in socket_fs_plugin 
Fixes #3774
 2020-07-03 11:11:23 +02:00
Emery Hemingway
1a620acc17 libc: add newlocale and freelocale dummies 
Fix #3777
 2020-07-03 11:11:23 +02:00
Christian Prochaska
6e71208db3 libc: fix timeout calculation in 'ppoll()' 
Fixes #3791
 2020-06-30 09:15:53 +02:00
Christian Helmuth
8e205e0324 sel4: assignment of BOARD as effective override 
Issue #3792
 2020-06-30 09:11:57 +02:00
Christian Prochaska
9578fadae2 libc: handle nullptr timeout in 'ppoll()' 
Fixes #3790
 2020-06-29 16:59:35 +02:00
Alexander Boettcher
1ec823bf5e libc: support manual pthread configuration 
Fixes #3787
 2020-06-29 16:56:47 +02:00
Alexander Boettcher
2471410fe5 libc: provide number of CPUs for sysctl&sysconf 
Fixes #3786
 2020-06-29 16:55:56 +02:00
Alexander Boettcher
3fb5ae4fdc vm_stress: fix resource request by seoul-debian32 
+ increase VM RAM to same size as used by vbox5
 2020-06-29 16:54:43 +02:00
Norman Feske
4450b37ff5 depot: update recipe hashes 2020-06-29 14:25:28 +02:00
Alexander Boettcher
309597bbda timeout fw: be robust against past _timestamp() 
_timestamp() returns CPU local values which may not be in sync with _ts
taken from another CPU. Be robust and don't produce wraparound/negative
timeout values.

Issue #3657
 2020-06-29 14:25:27 +02:00
Norman Feske
7dbf836217 Move stubby to genode-world 
Fixes #3789
 2020-06-29 14:25:27 +02:00
Norman Feske
5249f3358f Move Solo5 to genode-world 
Fixes #3761
 2020-06-29 14:25:27 +02:00
Stefan Kalkowski
1f91fd3f7d hw: scale imx8q_evk cpu to 1.5 GHz 2020-06-29 14:25:27 +02:00
Alexander Boettcher
01676717e2 tool: remove outdated tianocore image 
It is known to work up to Qemu 2.9 solely.

Instead use ovmf package provided by Linux distributions.

Issue #3784
 2020-06-29 14:25:27 +02:00
Stefan Kalkowski
1cfb1af56e os: automate cpu_bench testsuite 
* Differentiate in between different architectures with assembler routines
  for correct measures
* Automate first step measuring of 10G bogomips across different hardware

Fix #3785
 2020-06-29 14:25:27 +02:00
Norman Feske
ef741ef80d Change pixel format to 32 bits per pixel 
Until now, Genode's framebuffer session interface was based on the
RGB565 pixel format. This patch changes the pixel format to 32-bit
XRGB where the X part is ignored. It adapts all graphical applications
and device drivers accordingly.

The patch also adjusts the users of the drivers_interactive packages,
assigning 64 MiB RAM and 1500 caps to the drivers subsystem, which is
sufficient for covering high resolutions at 32 bits per pixel and to
accommodate multi-component USB HID input stacks.

Fixes #3784
 2020-06-29 14:22:29 +02:00
Alexander Boettcher
6119e03081 grub2: avoid hardcoding boot disc 
see alex-ab/g2fg#1
 2020-06-29 14:22:28 +02:00
Josef Söntgen
6d945e6a61 os: remove rom_block server component 
This component is superceeded by vfs_block.

Issue #3781.
 2020-06-29 14:22:28 +02:00
Josef Söntgen
6fc7ed55cf Replace 'rom_block' with 'vfs_block' 
Issue #3781.
 2020-06-29 14:22:28 +02:00
Josef Söntgen
8b590e2330 os: remove ram_block server component 
This component is superceeded by vfs_block.

Issue #3781.
 2020-06-29 14:22:28 +02:00
Josef Söntgen
ec711b008e Replace 'ram_block' with 'vfs_block' 
Issue #3781.
 2020-06-29 14:22:28 +02:00
Josef Söntgen
e5b00d89fb depot_autopilot: test case for vfs_block 
Issue #3781.
 2020-06-22 09:50:51 +02:00
Josef Söntgen
be55f080e4 os: add test-vfs_block pkg recipe 
Issue #3781.
 2020-06-22 09:50:51 +02:00
Josef Söntgen
e1aab829ca os: Block session server backed by VFS library 
The 'vfs_block' component will give access to a VFS file through a
Block session. For more detailed information please look at its
README.

(On a technical note, the server currently only allows for one
active session and has only one pending back end request but can
easily be extended in the future.)

Fixes #3781.
 2020-06-22 09:50:51 +02:00
Josef Söntgen
e56dd15a4b vfs: extend <zero> fs to limit amount of zeros 
Setting the 'size' attribute will restrict a reader to consume
0s up to the specified amount per VFS handle. The attribute is
parsed as 'Number_of_bytes', which will limit the amount to the
maximum of the addressable memory on the platform.

This addition comes in handy when needing to provide a synthetic
empty file with a fixed size.

Issue #3781.
 2020-06-22 09:50:51 +02:00
Josef Söntgen
103ae9df4a block_tester: limit batching in sequential test 
So far the condition whether to spawn a new job or not depended on
the amount of data already processed. This could lead to spawning
more jobs than necessary if batching is used and in return could
result in creating invalid requests in case the tested block session
is not large enough.

In addition to checking the amount of data the test now stores the
number of the last block and checks if the current request is in
range. This properly limits the total amount of requests.

Issue #3781.
 2020-06-22 09:50:51 +02:00
Josef Söntgen
b5f0c07eb3 os: add block_tester src recipe 
Issue #3781.
 2020-06-22 09:50:51 +02:00
Norman Feske
eb3a81a874 depot: update recipe hashes 2020-06-22 09:39:41 +02:00
Martin Stein
1e96510815 server/vfs: set seek offset before executing write 
The seek offset has to be updated on partial writes according to the
current write position. Otherwise writes will override the content
at the initial offset.

Fixes #3780.
 2020-06-22 09:39:40 +02:00
Stefan Kalkowski
04d8c859d8 hw: enable branch predictor on rpi 
* Z-bit was never enabled for rpi1
* Write buffer, and TCM bits are deprecated in this processor generation
  so do not use them

Ref #3247
 2020-06-22 09:39:40 +02:00
Stefan Kalkowski
b11da67679 usb_hid: introduce cpu quota 
Ref #3247
 2020-06-22 09:39:40 +02:00
Norman Feske
c8322ffd2a decorator: fix cap-upgrade handling 
This commit fixes the 'Out_of_caps' handling. The previous version
mistakenly upgraded the RAM instead the caps as response. The problem
could be triggered by the 'decorator_stress.run' script.
 2020-06-22 09:39:40 +02:00
Norman Feske
f63713694c api/blit: package arch-specific optimizations 
The blit API archive missed the CPU-architecture-specific variants of
the blit library such that all depot binaries worked with the generic
(slow) fallback implementation. This patch adds the missing pieces to
the blit API archive.
 2020-06-22 09:39:40 +02:00
Norman Feske
7a97cd70aa Rename nit_fader to gui_fader 
Issue #3778
 2020-06-22 09:39:40 +02:00
Norman Feske
48b4891f6e Rename nit_fb to gui_fb 
Issue #3778
 2020-06-22 09:39:40 +02:00
Norman Feske
5d40c0c1ce Rename 'Nitpicker' namespace to 'Gui' 
Issue #3778
 2020-06-22 09:39:40 +02:00
Norman Feske
e8f5706382 Rename <nitpicker> in runtime files to <gui> 
Issue #3778
 2020-06-22 09:39:39 +02:00
Norman Feske
98f39c698f Rename "Nitpicker" service name to "Gui" 
Issue #3778
 2020-06-22 09:39:39 +02:00
Norman Feske
c6eda9bd80 Rename include/nitpicker_session to gui_session 
This patch also renames the depot api archive accordingly.

Issue #3778
 2020-06-22 09:39:39 +02:00
Alexander Boettcher
87b08d6c7f libc: support pthread placement configuration 
Up to now all pthreads get placed on one CPU.

The patch adds support to evaluate a libc pthread configuration specifying
a placement strategy.

The default strategy is now to use all available CPUs as imposed by Genode's
affinity configuration for the pthread application.

The old behaviour, putting all pthreads on one single cpu can be still
configured by:

<libc>
 <pthread placement="single-cpu"/>
 ...
</libc>

Fixes #3775
 2020-06-22 09:39:39 +02:00
Alexander Boettcher
5f7e670ebc release_notes: fix Sculpt ARM 64 topic branch link 2020-06-22 09:39:39 +02:00
Josef Söntgen
e8fec3eed6 terminal_log: remove session size check 
The amount of RAM required by the session object is already accounted
for by the Root_component.

Fixes #3776.
 2020-06-22 09:39:38 +02:00
Emery Hemingway
dc8b4eeb40 cached_fs_rom: diagnosis of delivered ROM sessions 
To enable logging of ROM session deliveries, set the "diag" on
session routes:

<route>
  <service name="ROM" label_prefix="/bin/">
    <child name="cached_fs_rom" diag="yes"/>
  </service>
  …
<route>

Fix #3772
 2020-06-22 09:39:38 +02:00
Christian Helmuth
4f87fbd5ae Support linux and mod_openssl in lighttpd test 2020-06-22 09:39:38 +02:00
Christian Helmuth
df3f7dc1bc Support linux in manual fetchurl test 2020-06-22 09:39:38 +02:00
Christian Helmuth
8e85d889f1 libc: generate osreldate.h 2020-06-22 09:39:38 +02:00
Norman Feske
f687b0f3b9 News item for Sculpt 20.05 2020-06-22 09:39:38 +02:00
Christian Helmuth
5b87f68900 version: 20.05 2020-05-28 11:59:30 +02:00
Norman Feske
7221199f74 Release notes for version 20.05 2020-05-28 11:50:55 +02:00
Christian Helmuth
6006051fb9 depot: update recipe hashes 2020-05-27 11:56:47 +02:00
Christian Helmuth
63048fb89f doc: replace http by https in links where possible 
This also fixes some mixed content pages on genode.org and, thus,
removes the ugly browser warning, e.g., on
https://genode.org/documentation/release-notes/17.05.
 2020-05-27 11:56:47 +02:00
Norman Feske
a90aa78c6e doc: update components.txt 2020-05-27 11:56:46 +02:00
Stefan Kalkowski
70acd4b2d5 os: pl050, pl11x, lan9118 use ARM platform driver 
Make the framebuffer driver for pl11x chipsets,
the ps2 input driver for pl050, and the lan9116 NIC driver independent from
the pbxa9 board by using the newly introduced common ARM platform driver API.

Ref #3299
 2020-05-27 11:56:46 +02:00
Stefan Kalkowski
1a80f166c5 os: introduce common platform driver for ARM 
Ref #3299
 2020-05-27 11:56:46 +02:00
Stefan Kalkowski
935dcf8b18 usb_drv: include specific rpi platform_session API 
The rpi_usb_drv uses the rpi-specific variant of the platform_session API.
Therefore, it has to include it excplicitly.

Ref #3299
 2020-05-27 11:56:46 +02:00
Stefan Kalkowski
b915b0adc4 Annotate irq session interface with RAM_QUOTA 
Ref #3299
 2020-05-27 11:56:46 +02:00
Stefan Kalkowski
6e6b671a66 Annotate io_mem session interface with RAM_QUOTA 
Ref #3299
 2020-05-27 11:56:46 +02:00
Stefan Kalkowski
e1333c9421 os: refactor i.MX53 interactive drivers (ref #3299) 
* Remove input driver specific to i.MX53 tablet board from QSB driver pkg
* Move GPIO settings for QSB LVDS backlight out of framebuffer driver into
  GPIO driver config
* Move PWM driver functionality out of framebuffer driver
* Make framebuffer driver configureable, and less dependent on
  platform driver i.MX53 specifics
 2020-05-27 11:56:46 +02:00
Norman Feske
ab8ef5750d doc: tweaks for updated Genode Foundations book 2020-05-27 11:56:46 +02:00
Norman Feske
7c20ba84e4 tool/parse_cxx: consider __attribute__((...)) 
This change is needed to parse the __attribute__((deprecated))
annotations present in 'util/xml_node.h'.
 2020-05-27 11:56:46 +02:00
Stefan Thöni
78497c03ca base-linux: enabled seccomp 
base-linux uses seccomp to reduce the available system calls
to the minimum set needed to run base-linux. There are still
some syscalls that allow accessing global state which should
be further reduced.

The combination of seccomp and socket descriptor caps should
provide an intermediate level of security for base-linux
thereby enabling base-linux as a migration path from using
the Linux kernel to the use of microkernel-based Genode.

Fixes #3581
 2020-05-27 11:56:46 +02:00
Norman Feske
128ba65109 doc: fix autopilot command in release notes 19.08 
Fixes #3764
 2020-05-27 11:56:46 +02:00
Alexander Boettcher
f80b4b9fd9 vm_stress.run: update numbers 
due to script changes to stabilize/minimize variants in runs

Issue #3683
 2020-05-27 11:56:46 +02:00
Pirmin Duss
7f6b6d7e13 core-linux: make RAM quota configurable 
Add the option to configure the RAM quota for core on base-linux.
The environment variable GENODE_RAM_QUOTA if it exists is interpreted
as number of bytes to provide to the first init.

Issue #3762
 2020-05-27 11:56:46 +02:00
Pirmin Duss
51104454aa depot_deploy: copy all attributes of <report> node 
This allows the configuration of all report attributes of the <report>
node for the sub init instead of only `delay_ms`.

Issue #3760
 2020-05-27 11:56:46 +02:00
Christian Prochaska
2f47bbde30 qt5: use VFS pipe plugin 
Fixes #3759
 2020-05-27 11:56:46 +02:00
Norman Feske
b078224753 Replace Genode::strncpy by Genode::copy_cstring 
- Since Genode::strncpy is not 100% compatible with the POSIX
  strncpy function, better use a distinct name.

- Remove bogus return value from the function, easing the potential
  enforcement of mandatory return-value checks later.

Fixes #3752
 2020-05-27 11:56:45 +02:00
Norman Feske
0f27d139bd depot: update recipe hashes 2020-05-18 10:16:59 +02:00
Norman Feske
77a875735a fs_query.run: adjust quota for seL4 2020-05-18 10:16:15 +02:00
Christian Helmuth
32753c3f69 Use HTTPS in fetchurl test 
HTTPS is standard today and HTTP the exception.
 2020-05-18 10:16:15 +02:00
Christian Helmuth
7d2eb71094 openssl: update to current version 1.0.2u 2020-05-18 10:16:15 +02:00
Norman Feske
0de54cddaa gdb_monitor: hide exceptions during create_thread 
This patch is a workaround for the missing implementation of
'Pd_session::transfer_quota' interface by the GDB monitor's PD service.
The missing implementation becomes problematic with the changes of #3750
that enabled the cap-quota accounting for core's CPU service.

In regular scenarios without the GDB monitor, the client of
'Cpu_session::create_thread' deals with Out_of_caps or Out_of_ram by
upgrading the CPU session's cap and RAM quotas. This, in turn, results
in a sequence of 'transfer_quota' operations at the parent.

Since GDB monitor implements a custom PD service, these 'transfer_quota'
calls try to transfer quota between sessions provided by core and those
provided by the GDB monitor. This does of course not work. To fix this
issue, the GDB monitor needs a major overhaul. This patch side-steps
the problem by handing Out_of_caps and Out_of_ram from the debuging
target.
 2020-05-18 10:16:15 +02:00
Norman Feske
4002653334 qt5: fix audio driver support on Linux 
The lx_hybrid audio driver must be started with the 'ld="no"' argument.
 2020-05-18 10:16:15 +02:00
Emery Hemingway
adb48b5c9e Libc: MSG_PEEK support 
As discovered by Johannes Kliemann, peeking at buffered socket data
using 'recv' and 'MSG_PEEK' is not supported. Read a "peek" control file
from the socket directory to attempt to peek into buffers at the
socket_fs. Support for every feature of POSIX sockets cannot be
expected, but this one is trivial to implement.

Fix #2875
 2020-05-18 10:16:15 +02:00
Emery Hemingway
7f928a6573 LxIP "peek" file 
Add a control file for peeking at socket buffers.

Ref #2875
 2020-05-18 10:16:15 +02:00
Josef Söntgen
f1427fc02a dde_bsd: remove configration attributes 
The driver always opens the audio device in duplex mode, i.e.,
playback and recording. Setting the 'playback' or 'recording' attribute
only influnces the service announcement. Due to changes made in a more
recent OpenBSD release recording must be set enabled explicitly anyway.
Since we already provide the interface that mirrors the one used by
OpenBSD 1:1 in the configuration use that and the remove the additional
config attributes.

Fixes #3757.
 2020-05-18 10:16:14 +02:00
Josef Söntgen
8ce798abed dde_bsd: document audio driver configuration 
So far the configuration was only document in the commit message that
introduce the change.

Issue #3757.
 2020-05-18 10:16:14 +02:00
Norman Feske
dd0c1575f5 Xml_node: mark deprecated methods 
Fixes #3755
 2020-05-18 10:16:14 +02:00
Norman Feske
4b9c1d8f2b solo5: tolerate use of deprecated Xml_node API 
This patch is an interim fix to allow building solo5 despite of the use
of deprecated parts of the Xml_node API.

Issue #3755
 2020-05-18 10:16:14 +02:00
Norman Feske
be65c4acd2 Avoid use of deprecated Xml_node methods 
Issue #3755
 2020-05-18 10:16:14 +02:00
Norman Feske
d22b95ded3 test-fs_rom_update_ram: increase timeout for foc 
The timeout triggered on foc/x86_32 in Qemu.
 2020-05-18 10:16:13 +02:00
Christian Helmuth
c11d9b7b5c Remove false API dependencies from libc/posix components 
Fixes #3720
 2020-05-18 10:16:13 +02:00
Christian Helmuth
42fddf8390 Cleanup shared-object support mechanics 
The former ldso-startup static library (now called ldso_so_support) is
used to spice each shared object/library with local support code for the
dynamic linker (execution of static constructors and ARM-EABI).
Therefore, the library must be statically linked to each dynamic
library.

As a result recipes for dynamic libraries must always depend on the "so"
API, which makes ldso_so_support.mk and so_support.c available
independent of "base". Additionally, ldso_so_support is also provided in
the libc API to cut the dependency early for libc/posix libraries.

Issue #3720
 2020-05-18 10:16:13 +02:00
Christian Helmuth
589b416ca8 Remove misleading "pipe" utility and test 
If a "cat" tool as simple as the removed one is needed, it should be
implemented with Goa, only depend on libc/posix (*not* base), and be
named "simple_cat". Up to today, the misleading naming of the removed
tool, tricked me into believing "test-pipe" tests POSIX pipes, which it
did not!
 2020-05-18 10:16:13 +02:00
Christian Helmuth
4738d77c88 libc: fix CC_C_OPT in libm 2020-05-18 10:16:13 +02:00
Norman Feske
64bc008c3a core: fix inconsistent state after failed 'trace' 
This patch fixes the handling of the corner case where the allocation of
a trace buffer throws 'Out_of_caps' or 'Out_of_ram'. Under this
circumstance, the '_buffer' would still be flagged with the 'size',
which prevented any subsequent allocation attempt. This patch fixes the
problem by initializing the 'size' after the potentially throwing
allocation.

The problem triggered with the test-trace_logger after the accounting of
core's TRACE service (replacing the 'Allocator_guard' by
'Constrained_ram_allocator') became more accurate.

Related to issue #3750
 2020-05-18 10:16:13 +02:00
Christian Helmuth
236e02a2dd linux: remove initramfs from base-linux src archive 2020-05-18 10:16:13 +02:00
Norman Feske
cb5b688eb9 Remove <configfile> feature from init 
Fixes #3753
 2020-05-18 10:16:13 +02:00
Norman Feske
c39a342fe5 Remove use of <configfile> feature of init 
Issue #3753
 2020-05-18 10:16:13 +02:00
Norman Feske
07502ce6bb Remove Python2 
Fixes #3751
 2020-05-18 10:16:13 +02:00
Norman Feske
48b2456845 util/token.h: fix possible out-of-bounds read 
The 'WHITESPACE' case of the _calc_len method wrongly accessed the
character before checking upper bound of the token. The problem is fixed
by switching the order of both conditions.

Fixes #3756
 2020-05-18 10:16:12 +02:00
Christian Helmuth
894c7411e7 Remove obsolete vmm_utils test 2020-05-18 10:16:12 +02:00
Sebastian Sumpf
ea78d85d35 ahci: SYNC with empty NCQ queue only 
Fixes #3746
 2020-05-18 10:16:12 +02:00
Norman Feske
04aeaa25e5 Remove dead code 2020-05-18 10:16:12 +02:00
Norman Feske
9d67f9fc8e Remove Allocator_guard 
This patch removes old 'Allocator_guard' utility and replaces its use
with the modern 'Constrained_ram_allocator'.

The adjustment of core in this respect has the side effect of a more
accurate capability accounting in core's CPU, TRACE, and RM services.
In particular, the dataspace capabilities needed for core-internal
allocations via the 'Sliced_heap' are accounted to the client now.
The same goes for nitpicker and nic_dump as other former users of the
allocator guard. Hence, the patch also touches code at the client and
server sides related to these services.

The only remaining user of the 'Allocator_guard' is the Intel GPU
driver. As the adaptation of this component would be too invasive
without testing, this patch leaves this component unchanged by keeping a
copy of the 'allocator_guard.h' locally at the component.

Fixes #3750
 2020-05-18 10:16:12 +02:00
Christian Prochaska
1cc7277996 qt5: don't build webkit-related libraries for arm_v8a 
Fixes #3747
 2020-05-05 13:53:09 +02:00
Norman Feske
a9f0e47ea3 Remove return value of Log_session::write 
Fixes #3749
 2020-05-05 13:51:05 +02:00
Norman Feske
bbc21cf063 util/bit_array.h: remove use of memset and memcpy 
This makes the code less dependent on functions considered unsafe.

Fixes #3748
 2020-05-05 13:30:15 +02:00
Christian Helmuth
f722aa3325 os: src recipe for lx_fs 2020-05-04 15:59:29 +02:00
Norman Feske
eb0a33302a libc: strip trailing '/' for directory operations 
Fixes #2686
 2020-05-04 15:28:47 +02:00
Christian Prochaska
151828752c vfs: handle read error in fs file system 
Issue #3742
 2020-04-28 11:43:12 +02:00
Christian Prochaska
6a8fb48c13 vfs server: keep ack packet intact on failure 
Issue #3742
 2020-04-28 11:42:52 +02:00
Christian Prochaska
07b87f6f1f pthread: initialize static condition variables 
Fixes #3741
 2020-04-28 11:42:16 +02:00
Christian Prochaska
905b0c4aef libc: remove log2.patch 
Fixes #3740
 2020-04-28 11:41:38 +02:00
Christian Prochaska
6505ce47ae libc vfs plugin: report non-zero link count in 'stat()' 
Fixes #3739
 2020-04-28 11:40:58 +02:00
Christian Prochaska
31d7b3eb97 vfs_lwip: guard 'blocked_handles' Fifo 
Fixes #3738
 2020-04-28 11:37:56 +02:00
Alexander Boettcher
80d89c20fd top: use batched subject_infos of trace session 
Issue #3610
 2020-04-28 08:58:03 +02:00
Alexander Boettcher
1a94338389 trace: support to request subject infos batched 
Optimize requesting the Subject_infos from O(n) to O(1) RPC call.

Issue #3610
 2020-04-28 08:57:57 +02:00
Norman Feske
dd899fde29 depot: update recipe hashes 2020-04-24 14:37:57 +02:00
Norman Feske
1bf796d69a Remove Rust support 
Fixes #3488
 2020-04-24 14:37:57 +02:00
Norman Feske
c12d76686e Move OpenVPN to genode-world repository 
Fixes #3737
 2020-04-24 14:37:57 +02:00
Stefan Kalkowski
199821a247 os: re-enable drivers_interactive for i.MX53 
* Fix IPU hardware structure layout
* Turn Signal_handler into Io_signal_handler
* Eliminate the faulty msleep in I2C initalization sequence
* Fix routes and resources for tablet board (i.MX53 SMD)

Fix #3735
 2020-04-24 14:37:57 +02:00
Norman Feske
840f383e46 Remove input_merger component 
Fixes #3736
 2020-04-24 14:37:57 +02:00
Norman Feske
1459085a4d mk: prevent duplicates in REPOSITORIES definition 
Fixes #3731
 2020-04-24 14:37:56 +02:00
Norman Feske
6f6340644b Hide binary_ready_hook_for_platform from public 
The hook is meant to be internal to the framework. So better keep its
declaration in the framework-internal globals.h header.

Issue #3581
 2020-04-24 14:37:47 +02:00
Norman Feske
b134867f31 Remove Rpc_entrypoint::Native_context 
This patch largely reverts the commit "base: lay groundwork for
base-linux caps change" because the use of 'epoll' instead of 'select'
alleviated the need to allocate large FD sets, which motivated the
introduction of the 'Native_context' hook.

Related to issue #3581
 2020-04-21 16:50:37 +02:00
Norman Feske
01bf32b998 Remove ram_fs server 
Fixes #3734
 2020-04-21 16:27:07 +02:00
Norman Feske
f14cc2edab Replace use of ram_fs by VFS server 
Issue #3734
 2020-04-21 16:24:37 +02:00
Norman Feske
f0c4fc1e22 sandbox: defer requests for upcoming server 
The sandbox library supports the forwarding of session requests from the
outside to one of the hosted children according to a policy. This patch
introduces the distinction between two cases, which previously triggered
the denial of the session request.

- There exists no matching policy for the requested session
- There exists a matching policy but the referred server child
  does not exist (yet)

Whereas the proper response to the first case is the denial of the
request, the second case can occur in situation where a dynamic init is
used to implement a staged startup, for example via the deploy
mechanism. In such cases, a policy may exist as a static rule while the
server has not been started yet. This patch changes the behavior such
that such requests are stalled.

The patch is accompanied with test cases for exercising both situations.

Fixes #3733
 2020-04-21 14:52:29 +02:00
Alexander Boettcher
f891f4c963 vm_stress: change output message for ci scripts 
so that it detect it correctly as not available instead of a failure on
normal work days

Issue #3683
 2020-04-20 12:36:00 +02:00
Stefan Kalkowski
9f28f4f803 hw: avoid BASE_DIR relative path definitions 
Fix #3728
 2020-04-17 12:55:13 +02:00
Stefan Kalkowski
0e49336b96 Retire Exynos 5 support (fix #3725) 2020-04-17 12:53:57 +02:00
Stefan Kalkowski
941e918b46 vmm: unify armv7/v8 virtualization 
Fix #3638
 2020-04-17 12:52:02 +02:00
Stefan Kalkowski
74e75d7fbc hw: enable virtualization support for virt_qemu 
Ref #3638
 2020-04-17 12:51:53 +02:00
Stefan Kalkowski
58db6542f8 hw: check for security extension support in GICv2 
Ref #3638
 2020-04-17 12:51:44 +02:00
Tomasz Gajewski
fca2a05adf trace: extend test for log_output 
Issue #3714
 2020-04-17 12:48:17 +02:00
Tomasz Gajewski
4639978b3a trace_logger: avoid output of empty lines 
It avoids output of empty lines in output buffer if traced messages end
with \n (what is common with messages logged from dde_linux).

Issue #3714
 2020-04-17 12:48:13 +02:00
Tomasz Gajewski
870d348d77 trace: redirect logs to trace based on policy 
If trace is enabled for component than an attempt to put message into
trace buffer is performed using log_output policy. If it succeeds than
message is not put to logs using log service.

Fixes #3714
 2020-04-17 12:47:48 +02:00
Tomasz Gajewski
db8ec81e9f trace: log_output policy that redirects logs to trace 
Issue #3714
 2020-04-17 12:47:35 +02:00
Tomasz Gajewski
1d9a2dce94 trace: extend policy with log_output 
Issue #3714
 2020-04-17 12:47:19 +02:00
Roman Iten
12ea494477 vfs/File_content: fix end condition in for_each_line 
Fixes #3729
 2020-04-17 12:45:10 +02:00
Christian Helmuth
b60b591d06 depot: update recipe hashes 2020-04-17 12:40:13 +02:00
Christian Helmuth
9fbc68bda1 libports: icu library moved to GitHub 2020-04-17 12:40:13 +02:00
Norman Feske
132569d12b base-linux: socket descriptor caps for RPC 
On Linux, Genode used to represent each RPC object by a socket
descriptor of the receiving thread (entrypoint) and a globally-unique
value that identifies the object. Because the latter was transferred as
plain message payload, clients had to be trusted to not forge the
values. For this reason, Linux could not be considered as a productive
Genode base platform but remained merely a development vehicle.

This patch changes the RPC mechanism such that each RPC object is
represented by a dedicated socket pair. Entrypoints wait on a set of
the local ends of the socket pairs of all RPC objects managed by the
respective entrypoint. The epoll kernel interface is used as the
underlying mechanism to wait for a set of socket descriptors at the
server side.

When delegating a capability, the remote end of the socket pair is
transferred to the recipient along with a plaintext copy of the
socket-descriptor value of the local end. The latter value serves as a
hint for re-identifiying a capability whenever it is delegated back to
its origin. Note that the client is not trusted to preserve this
information. The integrity of the hint value is protected by comparing
the inode values of incoming and already present capablities at the
originating site (whenever the capability is invoked or presented to the
owner of the RPC object).

The new mechanism effectively equips base-linux with Genode's capablity
model as described in the Chapter 3 of the Genode Foundations book.
That said, the sandboxing of components cannot be assumed at this point
because each component has still direct access to the Linux system-call
interface.

This patch is based on the extensive exploration work conducted by
Stefan Thoeni who strongly motivated the inclusion of this feature into
Genode.

Issue #3581
 2020-04-17 12:40:13 +02:00
Alexander Boettcher
319d2be1af seoul: add vm_stress script version 
Issue #3683
 2020-04-17 12:40:13 +02:00
Alexander Boettcher
29911cf114 seoul: update contrib branch 
- reactivate vga update, range was to small
- remove diagnostic messages
- add support to ignore machine check MSR MCG CAP

Issue #3683
 2020-04-17 12:40:13 +02:00
Alexander Boettcher
8be72b0be1 vbox5: add testing VMs causing load 
Issue #3683
 2020-04-17 12:40:13 +02:00
Emery Hemingway
6359445a8e Add <exit propagate="…"/> to Init config schema 
Fix #3724
 2020-04-17 12:40:13 +02:00
Christian Helmuth
c783764d0b Region-map attach/detach stress depot_autopilot test 
Issue #3715
 2020-04-17 12:40:13 +02:00
Stefan Kalkowski
ff378a8c5b hw: cache maintainance on mapping removal too 
Older ARM processors like ARMv6, or Cortex A8 need to write back changes
of the page-tables to physical ram because the MMU does not use the cache.
This naturally needs to be done not only when adding a mapping,
but on removal too.

Fix #3715
 2020-04-17 12:40:13 +02:00
Norman Feske
03c3040a1d Remove stale documentation 
The topics are either covered by the Genode Founations book for by our
tools, in particular the integration of the prepare_port mechanism with
the run tool.
 2020-04-17 12:40:13 +02:00
Christian Helmuth
cc4e21e7a7 Fix some details in run scripts 
- don't use 'qemu -serial mon:stdio' anymore as it no longer works as
  expected
- use "bash -l" with [terminal] to read user's profile configuration,
  e.g., PATH settings
- added missing boot modules and cap quotas
 2020-04-17 12:40:13 +02:00
Christian Helmuth
bb34aafa45 run: use x-terminal-emulator in [terminal] 
The environment variables TERM and COLORTERM do not contain an
executable names.
 2020-04-17 12:40:13 +02:00
Norman Feske
6dfd268ef1 base/registry.h: remove misleading comment 2020-04-17 12:40:13 +02:00
Christian Helmuth
838c5ba7de ieee754: enable output check for ARM64 
Issue #3723
 2020-04-17 12:40:13 +02:00
Christian Helmuth
4ab990ad5b libc: provide C++ runtime symbols in ABI 
This commit puts all C++ runtime/support symbols of ld.lib.so in a
dedicated section of base/lib/symbols/ld and mirrors the section to
libports/lib/symbols/libc. So, the libc ABI resolves potential C++
runtime dependencies of base-ABI-agnostic components at link time. The
runtime resolution is done by the linker by symbol lookup in ld.lib.so.

Issue #3720
 2020-04-17 12:40:13 +02:00
Alexander Boettcher
13cd25e7b0 vfs/ram: avoid 2x mutex releases in guard 
Issue #3612
 2020-04-17 12:40:12 +02:00
Alexander Boettcher
2b41323fe6 test-* recipes: fix xmllint errors 
config_0.xml.tmp:26: element config: Schemas validity error :
  Element 'config': Character content other than whitespace is
  not allowed because the content type is 'element-only'

Issue #3612
 2020-04-17 12:40:12 +02:00
Norman Feske
7bf47b1982 base-linux: boost max open fds to the hard rlimit 
This patch increases the default limit of the maximum number of open
file descriptors to the hard limit of the system. This is needed for
complex scenarios, which require more FDs than the default of 1024
at core.

Related to issue #3581

Fixes #3721
 2020-04-17 12:40:12 +02:00
Alexander Boettcher
22d71d5a8b os: use Mutex/Blockade 
Issue #3612
 2020-04-17 12:40:12 +02:00
Alexander Boettcher
3956530634 base: use Mutex/Blockade 
Issue #3612
 2020-04-17 12:40:12 +02:00
Alexander Boettcher
e87d60ddf7 core: use Mutex/Blockade 
Issue #3612
 2020-04-17 12:40:12 +02:00
Norman Feske
85a1f91f59 Sanitize Framebuffer::Mode member access 
Fixes #3719
 2020-04-17 12:40:12 +02:00
Alexander Boettcher
dc90740549 nic_router: accept DHCP reply with missing option 
The options are not always set, which throws exception. This leads to not
accepting the selected IP address.

Fixes #3716
 2020-04-17 12:40:12 +02:00
Tomasz Gajewski
d111af922a trace documentation: fix reference to events definitions 2020-04-17 12:40:12 +02:00
Norman Feske
40c21b6d0f window layouter: restore screen tiling 
This commit fixes a regression introduced by "window_layouter: add
dynamic screen handling", rendering the sub-division of screens in
columns and row unusable. The said commit removed a condition needed
for the correct window placement. This patch restores the condition.

Issues #3646
 2020-04-17 12:40:12 +02:00
Norman Feske
4dd110ce5e Remove noux runtime 
The feature set of noux is fully covered by the regular C runtime now.

Fixes #3696
 2020-04-17 12:39:33 +02:00
Norman Feske
d6bdeed38f report_dump: replace noux by plain bash 
This patch also introduces the use of the RTC for creating directory
names and increases the rate of snapshot creation from 10 seconds to
one minute to make sure that directories are named differently when
reading the RTC clock at granularity of minutes.

Issue #3696
 2020-04-17 12:39:33 +02:00
Stefan Thöni
e8fb7ad470 report_rom: add package recipe 2020-04-17 12:39:33 +02:00
Stefan Thöni
5b68286e07 fs_report: add package recipe 2020-04-17 12:39:33 +02:00
Stefan Thöni
bb12d79ae9 vfs: add package recipe 2020-04-17 12:39:33 +02:00
Alexander Boettcher
d657b61f1b x86: support msi-x by platform driver 
Fixes #3699
 2020-04-17 12:39:33 +02:00
Norman Feske
42a77c531a sculpt: remove noux runtime from inspect view 
Issue #3696
 2020-04-17 12:39:32 +02:00
Norman Feske
5347278136 sculpt: perform prepare step without noux 
Issue #3696
 2020-04-17 12:39:32 +02:00
Norman Feske
ca5c25ac04 sculpt: replace log-noux instance by stdin2out 
Issue #3696
 2020-04-17 12:39:32 +02:00
Norman Feske
bb35b997b8 New stdin2out utility as replacement for tail -f 
This little utility allows us to remove the noux instance for the log
view of Sculpt OS.

Issue #3696
 2020-04-17 12:39:32 +02:00
Norman Feske
9c95e4bb4f sculpt: replace noux-based e2fsprogs by fs_utils 
With this patch, Sculpt executes the e2fs file-system utilities
directly instead of hosting them within the noux runtime.

Issue #3696
 2020-04-17 12:39:32 +02:00
Norman Feske
e63195a940 e2fsprogs: disable HAVE_GETMNTINFO 
This patch avoids calls of '__sys_getstatfs' (indirectly via
'getmntinfo'), which does not return any meaningful values in Genode's
libc. Otherwise, the libc's dummy implementation prompts resize2fs to
back out.

Issue #3696
 2020-04-17 12:39:32 +02:00
Norman Feske
0836293d1a libports: src/fs_utils depot archive 
The depot archive contains the utilities of e2fsprogs that are needed
for Sculpt OS as standalone components.
 2020-04-17 12:39:32 +02:00
Norman Feske
330672e030 libports: mke2fs and resize2fs as standalone apps 
This patch makes the e2fsprogs 'mke2fs' and 'resize2fs' available as
standalone components that can be used by Sculpt OS directly without
a Unix emulation environment.
 2020-04-17 12:39:32 +02:00
Josef Söntgen
150d143755 os: use Request_stream API in NVMe driver 
As a result of the API change the memory handling could be simplified.
Since the Block session dataspace is now directly used for DMA, we
actually only have to provide the memory for setting up PRP lists for
large requests (for the moment more than 8 KiB of data).

As we limit the maximum data transfer length to 2 MiB, we get by with
just a page per request. Those memory is allocated beforehand for the
maximum number of I/O requests, which got bumbed to 512 entries. Since
not all NVMe controllers support such large a maximum data transfer
length and this many entries, especially older ones, the values are
capped according to the properties of the controller during
initialization. (The memory demands of the component are around 3 MiB
due to setting up for the common case, even if a particular controller
is only able to make use of less.)

(Although there are controllers whose maximum memory page size is more
than 4K, the driver is hardcoded to solely use 4K pages.)

In addition to those changes, the driver now supports the 'SYNC' and
'TRIM' operations of the Block session by using the NVMe 'FLUSH' and
'WRITE_ZEROS' commands.

Fixes #3702.
 2020-04-17 12:39:32 +02:00
Christian Helmuth
e63c5e6c69 base: rework interplay of signal-proxy and entrypoint 
The former scheme left open a race window between
_process_incoming_signals() and wait_and_dispatch_one_io_signal()
resulting in both threads calling block_for_signal() and blocking
forever with one unprocessed signal.

Fixes #3704
 2020-04-17 12:39:32 +02:00
Christian Helmuth
69080014b0 depot_autopilot: test case for entrypoint 
Test for entrypoint destruction and signalling.

Issue #3704
 2020-04-17 12:39:32 +02:00
Stefan Kalkowski
f051065582 Retire OMAP4 support 
Fix #2850
 2020-04-17 12:39:32 +02:00
Piotr Tworek
585c4b8c69 hw: Add support for Qemu virt arm platforms. 
This patch adds support for booting base-hw kernel on qemu-arm virt
machines. The arm_virt machine has 2GB of RAM, 2 Cortex A15 cores and
uses GICv2 interrupt controller. The arm_64_virt machine also has 2GB of
RAM, but has 4 Cortex A53 cores and uses GICv3. Both machines use PSCI
to boot additional CPU cores.

Fixes #3673
 2020-04-17 12:39:32 +02:00
Piotr Tworek
dad1de1865 hw: Add interface for making basic ARM PSCI calls. 
So far this only exposes two functions allowing the code to turn on
additional CPUs. There is much more functionality defined by PSCI spec
but so far its not needed by Genode.

Fixes #3672
 2020-04-17 12:39:32 +02:00
Pirmin Duss
6f1357c6f8 libc: add symbol used by protobuf 
Issue #3692
 2020-04-17 12:39:32 +02:00
Josef Söntgen
336228f357 os: handle SYNC packet in lx_fs 
Call 'fsync(2)' on files and directories.

Fixes #3711.
 2020-04-17 12:39:32 +02:00
Josef Söntgen
df553e9360 os: remove catch-all from FS VFS update mtime 
Only 'alloc_packet' is supposed to throw an exception, which is already
caught. The 'submit_packet' call does not need further handling.

Issue #3713.
 2020-04-17 12:39:32 +02:00
Josef Söntgen
ddc0caa605 os: remove error messages from FS VFS update mtime 
The update modification timestamp is implemented as one-shot where
the caller is expected to try again if the first attempt failed (see
current libc implementation). So the message is misleading as further
attempts might succeed.

Issue #3713.
 2020-04-17 12:39:32 +02:00
Christian Helmuth
0df358a550 nova: early message on successful boot 
This patch fixes a regression with run/log_core introduced by

  2064ffd64b nova: support multidimensional affinity space

The run tool uses a log message 'run_boot_string' to detect successful
boot as well as to detect unexpected reboots. This message should never
be logged twice and, thus, should not be part of the core_log. The patch
mentioned above moved the former run_boot_string after the core_log
initialization.
 2020-04-17 12:39:31 +02:00
Pirmin Duss
9da428dccf vfs_import: implement write loop for copy_file 
This is required to allow import of large file.

Fixes #3655
 2020-04-17 12:37:26 +02:00
Norman Feske
46fa8197a2 depot: update recipe hashes 2020-03-26 11:39:02 +01:00
Christian Helmuth
52d798ff40 init: support heartbeat in config.xsd 2020-03-26 11:39:02 +01:00
Stefan Kalkowski
256b2fa3e1 hw: cortex a9 diagnostic register fixup 
This is a follow-up fix for commit 202333c881.
It checks for the diagnostic registers being already setup correctly.
Otherwise on platforms with secure firmware, like Pandaboard it will stuck.

Ref #3639
 2020-03-26 11:39:02 +01:00
Norman Feske
d8bcaa4fa4 base-linux: disable ASLR for child processes 
ASLR collides with the libc's fork mechanism on 32-bit. In particular,
the tool_chain_auto.run scenario would sporadically fail while mirroring
the parent's address space.

Fixes #3710
 2020-03-26 11:39:02 +01:00
Emery Hemingway
7a5841b637 ahci: driver improve diagnostic messages for empty ports 
Fix #3709
 2020-03-26 11:39:02 +01:00
Tomasz Gajewski
358fdd54b9 run: multiple power-on-command-param values 
Fixes #3708
 2020-03-26 11:39:02 +01:00
Emery Hemingway
45a74023a7 part_block: support for up to 128 sessions 
Allow sessions for each of the 128 sessions supported by GPT.

Fix #3705
 2020-03-26 11:38:59 +01:00
Alexander Boettcher
6f3f46c7ba top: support affinity space height up to 4 
Issue #3635
 2020-03-26 11:38:59 +01:00
Alexander Boettcher
2064ffd64b nova: support multidimensional affinity space 
Fixes #3635
 2020-03-26 11:38:59 +01:00
Norman Feske
685bd763f9 ldso: place dynamic allocations within linker area 
This is needed to execute the tool-chain scenario on base-nova.
Otherwise, the fork mechanism stumbles upon a region conflict
between ldso allocations and the application heap.

Fixes #3706
 2020-03-26 11:38:59 +01:00
Norman Feske
0f3ac4cb9f base-hw: let upgrade_cap_slab reflect Out_of_caps 
This patch reflects the Out_of_caps exception from core to the client,
in addition to the already covered Out_of_ram exception. It thereby
eliminates a potential abort in core, which I observed with the
tool_chain_auto.run script:

  Kernel: RPC upgrade_cap_slab
  Error: Uncaught exception of type 'Genode::Quota_guard<Genode::Cap_quota>::Limit_exceeded'
  Warning: abort called - thread: entrypoint

In addition to propagating the exception, the patch add the client-side
exception-handling code to the base library.

Fixes #3703
 2020-03-26 11:38:59 +01:00
Norman Feske
208294b0d3 Turn pkg/noux-system into pkg/system_shell 
The new package does no longer require noux.

Issue #3696
 2020-03-26 11:38:59 +01:00
Norman Feske
cf052996d3 Remove pkg/noux-build-x86 
Use-case-specific Unix runtimes should better be realized via Goa now.

Issue #3696
 2020-03-26 11:38:58 +01:00
Norman Feske
4961c84f12 Remove noux from vbox_share.inc 
This snippet is used by the vbox5_win7_64_share.run script.

Issue #3696
 2020-03-26 11:38:58 +01:00
Norman Feske
2f598e5680 Remove noux_uname.run script 
This run script is outdated and not regularly tested. In fact, the
tested mechanism (obtaining values from a '.sysctl/' directory) is
nowhere to be found, neither in noux nor the libc. The test still
returns success as it merely checks for the completion of the sequence,
not the printed results. Hence, it is not worth preserving.

Issue #3696
 2020-03-26 11:38:58 +01:00
Norman Feske
951f484fad Remove noux_gdb.run script 
This interactive run script is not regularly tested nor used.

Issue #3696
 2020-03-26 11:38:58 +01:00
Norman Feske
c086bba71a Remove noux_fs.run 
The test exercised the mounting of an ext2 file system in the noux
runtime. This feature is nowadays routinely stressed by Sculpt OS.

Issue #3696
 2020-03-26 11:38:57 +01:00
Norman Feske
5a02b5fd23 Remove noux_fork.run script 
This run script is superseded by pkg/test-fork.

Issue #3696
 2020-03-26 11:38:57 +01:00
Norman Feske
50a1e540a0 Remove noux_bash.run script 
This run script is superseded by the bash.run script.

Issue #3696
 2020-03-26 11:38:57 +01:00
Norman Feske
b8738dee90 Remove noux_terminal_fs.run script 
This run script tested the terminal VFS plugin as a new feature of noux.
In the meantime, the plugin remains as the only way for the interaction
of command-line-based applications with the terminal session. Hence,
the mechanism is now stressed by many other scenarios.

Issue #3696
 2020-03-26 11:38:57 +01:00
Norman Feske
a7e7eeec38 Remove noux from Vim example scenario 
Issue #3696
 2020-03-26 11:38:57 +01:00
Norman Feske
1d480e3329 Remove noux from tool-chain test 
Issue #3696
 2020-03-26 11:38:57 +01:00
Norman Feske
3b427824a7 ports/make: avoid aliasing with libc symbols 
This patch prefixes a few symbols in the make binary that are offered by
both the libc and the make binary. The clash of symbol names produces
confusing runtime linkage otherwise. This becomes a problem during
fork/evecve.
 2020-03-26 11:38:57 +01:00
Norman Feske
3a1f5bc13a tool/check_abi: prevent use of temporary files 
Issue #3696
 2020-03-26 11:38:57 +01:00
Norman Feske
923c38f7cd libc: allocate errno of main thread statically 
The thread-local errno instance for the main thread must not be
allocated on the application heap because it must survive 'execve'.

Fixes #3701
 2020-03-26 11:38:57 +01:00
Norman Feske
ee6d38a770 libc: prune session labels of forked processes 
This patch reduces the size of session labels for all services other
than LOG, keeping only the last element. This avoids exceeding the
maximum label length in the presence of deep fork hierarchies, e.g., for
running the tool chain.

Fixes #3700
 2020-03-26 11:38:57 +01:00
Christian Helmuth
dd524b56fa libc: propagate socket config to child processes 
Fixes #3698
 2020-03-26 11:38:57 +01:00
Norman Feske
4fe4c9aa63 vfs/pipe: trigger I/O-progress handler on EOF 
When the last writer closes the pipe, we must trigger the I/O progress
handler to deliver the final EOF to the client (READ_OK with out count
of 0).

Fixes #3697
 2020-03-26 11:38:56 +01:00
Norman Feske
e3bc77c386 Remove noux from SSH exec terminal 
This patch replaces the former Noux instance by a sub init that hosts
the combination of VFS + fs_rom + bash. If bash exits, the whole sub
init exits, which implicitly tears down the terminal session used by the
sub init. The latter is expected by the test sequence.

Issue #3696
 2020-03-26 11:38:56 +01:00
Norman Feske
cacbad1bd8 Remove noux from ssh_terminal.run 
Issue #3696
 2020-03-26 11:38:56 +01:00
Norman Feske
cc8552f81e Remove noux_vfs_audit.run 
This run script was merely a development vehicle.

Issue #3696
 2020-03-26 11:38:56 +01:00
Norman Feske
34a60b56f6 Remove noux from fs_query.run 
Issue #3696
 2020-03-26 11:38:56 +01:00
Norman Feske
25f7fdcb40 Remove noux from vfs_import.run 
Issue #3696
 2020-03-26 11:38:56 +01:00
Norman Feske
b6d3063ada Remove noux and cpu_sampler_noux from autopilot 
Issue #3696
 2020-03-26 11:38:56 +01:00
Norman Feske
c2cf0ae8c6 os/vfs.h: fix possible nullptr dereference 
Fixes #3694
 2020-03-26 11:38:56 +01:00
Norman Feske
7f5186d489 vfs/tar: return inode numbers of null records 
The tar vfs plugin returns an inode value of zero and a type directory
for null records, which wrongly triggers the file-system loop detection
of the find utility. This patch returns the node pointer as inode value
instead, which is a unique value.

Fixes #3695
 2020-03-26 11:38:56 +01:00
Norman Feske
e67f389da1 Add fork test to depot autopilot 2020-03-26 11:38:56 +01:00
Norman Feske
4ebc164c11 Add execve test to depot autopilot 2020-03-26 11:38:56 +01:00
Martin Stein
886d0da57c ping_nic_router.run: test dhcp renew 
Ref #3681
 2020-03-26 11:38:56 +01:00
Norman Feske
2912096f6e vfs: distinguish NO_PERM from NO_ENTRY in unlink 
This patch changes the 'Single_file_system' to return NO_PERM only if
the to-be-unlinked file corresponds to the single file. This way, a
<rom> co-mounted with a <ram> file-system does not stand in the way of
unlinking files from the <ram>. The concrete symptom occurred the
following scenario:

  <vfs>
    <dir name="home">
      <ram/>
      <rom name="..."/>
    </dir>
  </vfs>

The following sequence of commands wrongly resulted in "Operation not
permitted":

 $ mkdir -p /home/a/b/c
 $ rm -f /home/a/b/c/d

In this case, rm should not fail (unlink should return ENOENT)

Fixes #3690
 2020-03-26 11:38:55 +01:00
Martin Stein
ba55409c86 nic_router: keep dyn IP when updating domain obj 
When updating the domain object of interfaces that stay with the same domain
during a reconfiguration, until now, the normal "detach raw" function was used.
This caused the old domain object to discard a dynamic IP config as all
interfaces detached. This caused interfaces also to discard network links
established with the old configuration although it wasn't necessary. Thus, now
we use a dedicated "detach" in case that an interface actually stays with its
domain. This new "detach" doesn't decrease the interface counter of the domain,
so, it'll not discard its dynamic IP config. If, however, during a
reconfiguration, there's no interface calling this function (all interfaces
move to another or no domain), the dynamic IP config is still discarded as
expected.

Fixes #3686
 2020-03-26 11:38:55 +01:00
Martin Stein
f3185de7f5 nic_stress: further reduce nr of rounds on sel4 
Otherwise, the limit of caps in Capability_space_sel4 in the NIC router is
reached.

Fixes #3684
 2020-03-26 11:38:55 +01:00
Martin Stein
c1b6000248 nic_router: don't call DHCP client on requests 
Until now, the DHCP client was called also for DHCP requests when an interface
had a domain but yet no IP config. Now, an interface distinguishes between DHCP
requests and replies first and then accordingly calls the DHCP server or the
DHCP client if they're available. This also prevents that the DHCP client has
to handle packet headers other than that of DHCP.

Fixes #3681
 2020-03-26 11:38:55 +01:00
Martin Stein
0a468a07a7 nic_router: fix lifetime of DHCP clients 
Let the DHCP client be a constructible member of Interface that is constructed
only as long as the interface is attached to a domain with a dynamic IP config.
This prevents DHCP client timeouts from a period with dynamic IP config to
trigger after a reconfiguration to a static IP config. Furthermore, handle
DHCP-reply packets at an interface only when the DHCP client its constructed.
Otherwise drop such packets.

Ref #3681
 2020-03-26 11:38:55 +01:00
Stefan Kalkowski
6d48b5484d hw: correct the ARM cache maintainance operations 
This commit fixes the following issues regarding cache maintainance
under ARM:

* read out I-, and D-cache line size at runtime and use the correct one
* remove 'update_data_region' call from unprivileged syscalls
* rename 'update_instr_region' syscall to 'cache_coherent_region' to
  reflect what it doing, namely make I-, and D-cache coherent
* restrict 'cache_coherent_region' syscall to one page at a time
* lookup the region given in a 'cache_coherent_region' syscall in the
  page-table of the PD to prevent machine exceptions in the kernel
* only clean D-cache lines, do not invalidate them when pages where
  added on Cortex-A8 and ARMv6 (MMU sees phys. memory here)
* remove unused code relicts of cache maintainance

In addition it introduces per architecture memory clearance functions
used by core, when preparing new dataspaces. Thereby, it optimizes:

* on ARMv7 using per-word assignments
* on ARMv8 using cacheline zeroing
* on x86_64 using 'rept stosq' assembler instruction

Fix #3685
 2020-03-26 11:38:55 +01:00
Stefan Kalkowski
3a862334fc depot_autopilot: re-enable test-fs_packet on rpi 
Fixes #3249
 2020-03-26 11:38:55 +01:00
Alexander Boettcher
cd1c0aace3 nova: remove specific stack area location 
and use instead the generic one of base

Fixes #3654
 2020-03-26 11:38:55 +01:00
Sebastian Sumpf
6e54cad44c usb_block: switch to Request_stream API 
- remove old 'Driver' interface
- implement Request stream front end
- fix some namespacing

issue #3661
 2020-03-26 11:38:55 +01:00
Josef Söntgen
a71ef16423 vfs: implement 'sync' in Block VFS plugin 
Fixes #3659.
 2020-03-26 11:38:55 +01:00
Sebastian Sumpf
2495a86aff block_tester: add 'sync' command to replay 
issue #3636
 2020-03-26 11:38:55 +01:00
Sebastian Sumpf
5225d12381 ahci: add sync operation support 
- implemement 'flush cache ext' for ATA
- enable sync operation for request stream

issue #3636
 2020-03-26 11:38:54 +01:00
Alexander Boettcher
e096202b1f hw/x86: use PAT for write combining support 
Fixes #3623
 2020-03-26 11:38:54 +01:00
Alexander Boettcher
44ace04b95 hw/x86: add cpuid register macro 
Issue #3623
 2020-03-26 11:38:54 +01:00
Alexander Boettcher
507a34b3d0 hw/x86: implement wrmsr in macro 
Issue #3623
 2020-03-26 11:38:54 +01:00
Christian Helmuth
677d9abd07 rtc: system_rtc pkg for PC and Linux 
This depot package is the runtime for rtc_drv and system_rtc server with
coordinated update of system and hardware RTC via reports. It replaces
drivers_rtc which was never freestanding drivers package and, therefore,
has to be accompanied with a running platform_drv etc.

Fixes #3680
 2020-03-26 11:38:54 +01:00
Christian Helmuth
e66726e931 rtc: use depot archives in test 
Issue #3680
 2020-03-26 11:38:54 +01:00
Christian Helmuth
cda778a94d rtc: recipe for linux_rtc_drv 
Issue #3680
 2020-03-26 11:38:54 +01:00
Sebastian Sumpf
a799a5f978 os: adjust tests to changed output of block client 
issue #3671
 2020-03-26 11:38:54 +01:00
Sebastian Sumpf
f5a6d85bc2 os: expand block_tester.run 
- use two disks with two partitions each
- one disk uses MBR and one GPT
- connect one 'part_block' to each disk
- run 'block_tester' and 'test-block-client' connecting to one partiion
  on each disk
- x86 uses 'ahci_drv' as disk back-end, Linux 'lx_block'
- on autopilot enable for Qemu only

issue #3671
 2020-03-26 11:38:54 +01:00
Christian Prochaska
8535688605 qt5: handle Genode signals as Qt signals in QPA plugin 
Fixes #3688
 2020-03-26 11:38:54 +01:00
Johannes Schlatow
8788f13e11 Qt5 tool chain: always build without zstd 
Fixes #3687
 2020-03-26 11:38:54 +01:00
Johannes Schlatow
aee5b59c51 Backport changeset 82268 from virtualbox 
Fixes #3689
 2020-03-26 11:38:54 +01:00
Christian Helmuth
66a7e749e8 Update URLs in LICENSE 
The official AGPLv3 text from https://www.gnu.org/licenses/agpl-3.0.txt
now uses https:// consistently.
 2020-03-26 11:38:54 +01:00
Norman Feske
e2b9308b11 lx_hybrid_errno.run: add missing parent services 2020-03-26 11:38:54 +01:00
Alexander Boettcher
97bbad5aea nova: avoid dying thread in platform.run 
Avoids timely interference by dying thread.

Fixes #3670
 2020-03-26 11:38:54 +01:00
Christian Helmuth
d452094f43 Only rewind next-free pointer in Bit_allocator 
Formerly, _next was always updated to the last free'd allocation, which
left large gaps on alloc/free bursts. Now, we try keep allocation
density high from the start of the array and ensure that allocations
happen at the lowest available index in the bit array.

Fixes #3679
 2020-03-26 11:38:53 +01:00
Piotr Tworek
139f390317 libc: Expose timespec_get symbol. 
The function is exposed in libc headers provided by Genode, the code for
the function is being compiled and actually works, but the symbol is
missing from the symbols file resulting in linking failures. Add it to
the libc symbols file.

Fixes #3676
Fixes #3677
 2020-03-26 11:38:53 +01:00
Emery Hemingway
9ee1045aad Remove SVN build dependency from libc port 
Fixes #3678
 2020-03-26 11:38:53 +01:00
Norman Feske
25c85fab18 sculpt: version 20.03 
This patch increases the Sculpt version to avoid the risk of binary
compatibility clashes with index files published for the just-released
Sculpt 20.02 version.
 2020-03-26 11:38:53 +01:00
Norman Feske
095a0a6439 News item for Sculpt 20.02 2020-03-10 11:06:10 +01:00
Christian Helmuth
bf36d9eb48 depot: update recipe hashes 2020-03-09 16:02:56 +01:00
Norman Feske
9d24e906a8 text_area: fix interplay of shift with selection 
Without this patch, the text area would mistakenly delete the characters
types after placing the cursor with the mouse as soon as shift is used
as modifier.
 2020-03-09 15:41:29 +01:00
Norman Feske
44e4d1bd6c Update Sculpt documentation for version 20.02 2020-03-09 15:41:29 +01:00
Norman Feske
ab5770c492 sculpt: improve USB hotplug handling 
- Don't reset sculpt partition on USB changes whenever ram fs is
  selected for use.

- Reconfigure runtime immediately when plugging USB devices.
  This prevents the usb-block driver and part_block from
  dangling in the graph. They disappear immediately now.
 2020-03-09 15:36:21 +01:00
Norman Feske
a90eab8b9a menu_view: reduce frame transparency 
This makes the dialog content better readable.
 2020-03-09 15:36:21 +01:00
Norman Feske
e54e4dd532 sculpt: anchor popup position at '+' button 
This patch restores the anchoring of the popup menu to the original
behavior, placing the popup to the right of the '+' button. The
anchoring got lost due to the recent structural changes of the GUI,
which affect the hover XML structure.
 2020-03-09 15:36:21 +01:00
Sebastian Sumpf
c856ba2a49 ahci: report ports correctly 
Only use one report for devices report, not one per device

fixes #3671
 2020-03-09 15:36:21 +01:00
Christian Prochaska
1087e3f59e doc: minor tweaks of release notes 20.02 
Thanks to Jeroen van Gelderen for the feedback.
 2020-03-09 15:36:04 +01:00
Christian Helmuth
cfb49c7316 version: 20.02 2020-02-28 12:19:49 +01:00
Norman Feske
a3fad2e171 News item for version 20.02 2020-02-28 12:19:49 +01:00
Norman Feske
dd5db8484a Release notes for version 20.02 2020-02-28 12:19:49 +01:00
Christian Helmuth
5affd51250 depot: update recipe hashes 2020-02-28 08:59:43 +01:00
Alexander Boettcher
ce27b5ebce tool: support hyphen in depot-user name (depot.inc) 
Fixes #3668
 2020-02-28 08:58:28 +01:00
Norman Feske
fce9cd8c22 sculpt manager: use runtime state report max of 1M 
With complex scenarios, supplementing the <report> node with the
attribute 'requested' or 'provides' can easily result in a report of
more than 64K.
 2020-02-27 14:47:00 +01:00
Christian Prochaska
8faa916d93 gdb_monitor.run: increase RAM quota 
Fixes #3667
 2020-02-27 14:47:00 +01:00
Christian Helmuth
e52802162c libc: reimplement synchronization primitives 
The new implementation relieves the main entrypoint from monitor jobs
for contended lock primitives and is based on custom applicant data
structures, per-lock resp. per-semaphore applicant lists, and a
libc-internal blockade with timeouts based on libc kernel primitives.
 2020-02-27 14:47:00 +01:00
Josef Söntgen
f3ec246b67 base: remove diagnostic message in destructor 
The LOG message in the Signal_handler_thread's destructor is just an
diagnostic leftover.

Fixes #3666.
 2020-02-27 14:46:59 +01:00
Christian Prochaska
434c9ceb5d vbox5-nova: document possible 'Blocking_canceled' exception 
Fixes #3664
 2020-02-27 14:46:59 +01:00
Norman Feske
c3fb81d1a1 window_layouter: improve rules formatting 
This is a follow-up patch for "window_layouter: add dynamic screen
handling".

Issue #3646
 2020-02-27 14:46:59 +01:00
Norman Feske
c340f57207 Xml_node: skip leading whitespace in with_raw_node 
This patch makes the 'with_raw_node' method more useful in situations
where content of an Xml_node is fed into an Xml_generator, i.e., the
rules report/rom mechanism of the window layouter.
 2020-02-27 14:46:59 +01:00
Norman Feske
bbe1bf9c3a fs_query.run: increase timeout 
This is apparently needed for base-foc on Qemu/x86.
 2020-02-27 14:46:59 +01:00
Alexander Boettcher
accda1211b mutex: reinit Applicant state in Mutex::acquire() 
The applicant_to_wake_up() member must be initialized to 0 to keep the
same semantic as before the change by

"base: add mutex as derivate of lock"

Issue #3662
 2020-02-27 14:46:59 +01:00
Norman Feske
daee1f4cb8 timer/nova: prevent potential division by zero 
This case triggered with the leitzentrale.run script on Qemu.
The frequency value must never initialized with zero.

Fixes #3663
 2020-02-27 14:46:59 +01:00
Stefan Kalkowski
87cb10c558 sculpt_manager: accept uplink from "local" network 
Be less rigid when checking for available network to support
setups where the uplink or depot repositories are provided by
another source than "wired" or "wifi" NIC services,
e.g., a virtual machine with pass-through network device.
 2020-02-27 14:46:59 +01:00
Norman Feske
904651ada9 sculpt: version 20.02 2020-02-27 14:46:59 +01:00
Norman Feske
1d3ce93107 sculpt: leitzentrale user-interface redesign 
Issue #3650
 2020-02-27 14:46:59 +01:00
Sebastian Sumpf
103dcdeea8 vmm: ARMv8 virtio improvements 
net:
* increase queue size to 1024 (more stable on Linux)
* use mac address from Nic session instead of random one
* handle data that is larger than rx descriptor correctly (copy less)
* clear descriptor header (12 bytes) on rx

generic:
* always use 'avail_idx' (tx and rx)
* added barriers when reading/writing queues (TMP)

Ref #3620
 2020-02-27 14:46:59 +01:00
Sebastian Sumpf
f77531138a vmm: ARMv8 virtio for network and console 
Simple console and network implementation of the Virtio protocol.

Fix #3620
 2020-02-27 14:46:59 +01:00
Sebastian Sumpf
c8b3b060aa vmm: ARMv8 add 'local_address' to Ram_allocator 
Ref #3620
 2020-02-27 14:46:59 +01:00
Stefan Kalkowski
7780ee6a34 vmm: do not lock inside VMM 
We do not use the VMM with several CPUs until now.
On the other hand there is a dead-lock due to the
generic timer calling the cpu handler function explicitly,
which already holds the lock. For the time-being remove the lock.

Ref #3620
 2020-02-27 14:46:59 +01:00
Stefan Kalkowski
2e2625e952 vmm: check ring-buffer capacity in pl011 model 
Ref #3620
 2020-02-27 14:46:58 +01:00
Stefan Kalkowski
55c3eb7c14 vmm: fix GIC interrupt register bit shifts 
Ref #3620
 2020-02-27 14:46:58 +01:00
Stefan Thöni
a7a9855493 base: lay groundwork for base-linux caps change 
Include the necessary hooks to introduce file descriptor
based capabilities in base linux.

Issue #3581
 2020-02-27 14:46:58 +01:00
Norman Feske
5eaaee0dbe text_area: a simple text viewer / editor 
The new text_area component is able to view and edit files.
Internally, it employs a menu_view for the graphics output.

Only basic notepad-like text-editing functions are supported.

At the current time, it is solely meant as a companion of the Sculpt
manager.

Issue #3650
 2020-02-27 14:46:58 +01:00
Norman Feske
b2bc718c1f fs_query: report 'writeable' attribute for files 2020-02-27 14:46:58 +01:00
Norman Feske
7118ad494c os/vfs.h: add Directory::Entry::rwx accessor 2020-02-27 14:46:58 +01:00
Norman Feske
582e0e718c menu_view: make texture handling more robust 
This patch improves the robustness of menu_view when encounting missing
textures, which can happen during development when using styled buttons
and frames. With the patch, menu_view outputs diagnostic messages,
pinpointing the problem.

The patch also updates the texture handling to use the 'File_content'
utility and the VFS for obtaining PNG images.

Issue #3629
 2020-02-27 14:46:58 +01:00
Norman Feske
1713583a19 menu_view: basic support for styling labels 
This patch allows for the customization of the text color and alpha
value of the label widget by the means of a style-definition file.
The mechanism is exemplified with the new "invisible" label style
that sets the alpha value to zero.

Issue #3629
 2020-02-27 14:46:57 +01:00
Norman Feske
38aef49428 menu_view: invisible button style 2020-02-27 14:46:57 +01:00
Alexander Boettcher
a9caf3fbe4 base: avoid wrong warning by Mutex::acquire 
A _owner is only valid if lock is actually locked.

Issue #3612
 2020-02-27 14:46:57 +01:00
Josef Söntgen
80ff844dc2 depot: update public key for cnuke 2020-02-27 14:46:57 +01:00
Josef Söntgen
c53be5a3fb os: omit size check on payloadless Block packets 
The check is already performed in 'packet_valid' where the conditions
for payloadless packets are already considered.

Fixes #3660.
 2020-02-27 14:46:57 +01:00
Josef Söntgen
6addd6cf1e os: fix Node_rwx::rwx 
Fixes #3658.
 2020-02-27 14:46:57 +01:00
Sebastian Sumpf
3995d2f4a2 part_block: switch to Request_stream and Job APIs 
- use Job API as block connection back end
- use Request_stream API as front end
- use Mmio framework for gpt and mbr parsing
- implement sync correctly

fixes #3652
 2020-02-27 14:46:57 +01:00
Sebastian Sumpf
b95dc611d6 gmp: use architecture specific 'config.h' 
Generated a separate 'config.h' for arm, arm_64, x86_32, x86_64 for the
current version (6.1.2) of GMP. This became necessary because
configurations differ for each architecture.

'config.h' generaton on x86_64 host in'gmp-6.1.2' directory:

for x86_64 (native):
! configure

for x86_32:
! configure --host=x86-pc-linux-gnu --build=x86_64-pc-linux-gnu

for arm:
! configure --host=arm-linux-gnu --build=x86_64-pc-linux-gnu \
!  CC=/usr/local/gcc-linaro-arm/bin/arm-linux-gnueabi-gcc ABI=32

for arm_v8:
! configure --host=aarch64-linux-gnu --build=x86_64-pc-linux-gnu \
!  CC=/usr/local/gcc-linaro/bin/aarch64-linux-gnu-gcc ABI=64

issue #3598
 2020-02-27 14:46:57 +01:00
Christian Helmuth
4cccf74664 depot: update recipe hashes 2020-02-20 12:11:23 +01:00
Stefan Kalkowski
8cc48d5688 hw: be more accurate in synchronizing ASID/Pages 
Fix #3651
 2020-02-20 12:11:23 +01:00
Stefan Kalkowski
b76bd57ed1 window_layouter: add dynamic screen handling 
This commit adds dynamic switching in between different defined
screens via keyboard shortcuts, or by editing the rules file.
In the window_layouter package it defines per default ten screens,
reachable by pressing the screen key along with numbers 0-9.

Fix #3646
 2020-02-20 12:11:23 +01:00
Alexander Boettcher
2afc218767 test/registry: measure time used for iteration 
Issue #3612
 2020-02-20 12:11:22 +01:00
Alexander Boettcher
5bbaa30655 base: use mutex in registry 
Issue #3612
 2020-02-20 12:11:22 +01:00
Alexander Boettcher
5440cd4b50 base: use blockade in thread 
Issue #3612
 2020-02-20 12:11:22 +01:00
Alexander Boettcher
e686ff78e9 base: add blockade as derivate of lock 
Issue #3612
 2020-02-20 12:11:22 +01:00
Alexander Boettcher
2bd77722c7 base: use mutex in heap 
Issue #3612
 2020-02-20 12:11:22 +01:00
Alexander Boettcher
00f69bc70d base: add mutex as derivate of lock 
The mutex class is more restrictive in usage compared to
Genode::Lock.

- At initialiation time it is ever unlocked.
- No thread is permitted to lock twice. Warn about it
  in case it happens.
- Only the lock onwer is permitted to unlock the mutex.
  Warn about it and don't unlock the mutex in case it happens.

Issue #3612
 2020-02-20 12:11:22 +01:00
Stefan Kalkowski
d1609e771a pdf_view: fix typo in pkg runtime requires 
Fix #3648
 2020-02-20 12:11:22 +01:00
Norman Feske
89f813f113 os: add sandbox lib to api/os 2020-02-20 12:11:22 +01:00
Norman Feske
9b0fbf000e sandbox: support for local state-report handling 
Fixes #3647
 2020-02-20 12:11:22 +01:00
Stefan Kalkowski
725d16e18e hw: optimize ipc capability allocation 
In preparation of ipc receive, by now a bunch of capabilities got
allocated to be ready iin case of capability receiption. After that
unuse slots were freed again. This overhead of senseless (de-)allocations
is replaced in this commit by just restock capability slots that got
used by the last receive.

Fix #3640
 2020-02-20 12:11:22 +01:00
Martin Stein
e42a205a51 base-hw: Ada-compliant scheduler method signatures 
Ref #3308
 2020-02-20 12:11:22 +01:00
Martin Stein
0d5f185267 base-hw: no pointers in scheduler interface 
Ref #3308
 2020-02-20 12:11:22 +01:00
Martin Stein
c146a215fb base-hw: cpu scheduler without inheritance 
Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
eef7b5e168 base-hw: double list without inheritance 
Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
a753b6ce46 base-hw: fix run scripts 
Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
793e12f8f3 base-hw: no error return codes in signaling 
Error return codes are used with non-const functions in the signaling modules.
This would have been impractical for the in-place translation of the module to
Ada in the context of the Spunky project. Besides, it is easy to get rid of
them and it makes the execution flow more clear.

Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
751e6430fa base-hw: obsolete Kernel::Thread::_signal_receiver 
Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
9eb20c2be7 base-hw: no pointer arguments in signaling 
Besides reducing the use of pointers in base-hw, this prepares for the in-place
translation of the signaling module to Ada in the context of the Spunky
project.

Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
5e460394d2 base-hw: do not inherit Kernel::Object 
This prevents the use of Kernel::Object as base class for the specific kernel
objects and instead makes it a member of the kernel objects. Besides
simplifying inheritance hierarchies in base-hw, this prepares for the in-place
translation of the signaling module (and others) to Ada in the context of the
Spunky project.

Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
88043e144a base-hw: no virtual functions in signaling 
Besides simplifying the execution flow in the signaling module, this prepares
for the in-place translation of the signaling module to Ada in the context of
the Spunky project.

Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
3cc7774fe4 base-hw: Ada/SPARK-friendly Ipc_node interface 
This prevents the use of C++ features in the public method interface of the
synchronous-IPC module that would be impractical for the in-place
translation of the module into Ada in the context of the Spunky project.

* Get rid of thread accessor.
* Get rid of non-const functions with return values.
* Get rid of pointer return-values.

Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
a04243aaf4 base-hw: remove unused Ipc_node accessor methods 
Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
5a95183c3e base-hw: avoid that Ipc_node inherits 
Besides simplifying inheritance hierarchies in base-hw, this prepares for the
in-place translation of the synchronous-IPC module to Ada in the context of the
Spunky project.

Ref #3308
 2020-02-20 12:11:21 +01:00
Martin Stein
6a5aa18a7b base-hw: no PD code / virt methods in Ipc_node 
This is a simplication of the asyncronous-IPC module of the base-hw kernel.
Besides structuring the code in a cleaner way, it prepares for the in-place
translation of the module into Ada in the context of the Spunky project.

* Get rid of virtual methods in Ipc_node.
* Move all stuff related to protection domains, capabilities, and UTCBs to
  the Thread class. this code might later be moved to a dedicated module, but
  for now it's just fine to have it done by the thread module.

Ref #3308
 2020-02-20 12:11:21 +01:00
Christian Prochaska
4b3c40f35b test-tcp_bulk: increase resources 
Fixes #3645
 2020-02-20 12:11:21 +01:00
Stefan Kalkowski
79fba6c2ac core: add kernel-specific UTCB quota bookkeeping 
Fix #859
 2020-02-20 12:11:21 +01:00
Tomasz Gajewski
4f217b19a9 hw: add CPU wake up code for rpi3 
Moved code waking up processors for Cortex A53 before changing privilege
level because sending events to higher privilege levels is not allowed.

Fixed enable_mmu for Cortex A53 to properly return cpu id.

Fixed starting code for secondary cores to properly initialize stacks.

Added code to wake up secondary cores on rpi3.

Ref #3573
 2020-02-20 12:08:17 +01:00
Stefan Kalkowski
202333c881 hw: cortex a9 diagnostic register initialization 
We cannot count on the correct initialization of the diagnostic register
of the secondary cores. But the boot core gets initialized by the bootchain,
so we can copy over those values.

Fix #3639
 2020-02-20 12:08:17 +01:00
Christian Prochaska
2ce0395fd8 arora.run: fix 'focus' route 
Fixes #3637
 2020-02-20 12:08:17 +01:00
Emery Hemingway
bbdf181828 core: add support for unlabeled LOG sessions 
If the root child requests a LOG service with the label "unlabeled" then
return a LOG session that logs unprefixed messages. This allows a external
test controller to recognize log messages produced by a blessed component.
 2020-02-20 12:08:17 +01:00
Norman Feske
0181c6025a menu_view: add <label min_ex=".."> attribute 
This new attribute can be used to enforce a minimum width in the unit of
the size of the character 'x'. Furthermore, the patch sets the minimum
height of a label to 0 if no 'text' attribute is present. By combining
both features, the label becomes usable as a horizontal spacer.

Issue #3629
 2020-02-20 12:08:17 +01:00
Norman Feske
6eff83c1eb menu_view: invisible frame 
Issue #3629
 2020-02-20 12:08:17 +01:00
Norman Feske
5aae0f2379 menu_view: support multiple floats within a frame 
This patch refines the hover handling such that a float widget never
responds to hovering unless a child is hovered. This way, it becomes
possible to stack multiple float widgets within one frame and still
reach all child widgets.

Issue #3629
 2020-02-20 12:08:17 +01:00
Norman Feske
49ae4a834f menu_view: support for displaying text selections 
Similarly to the way of how a <cursor> can be defined for a <label>
widget, a selection can be expressed as follows:

  <label ...>
    <selection at="2" length="12"/>
  </label>

Issue #3607
Issue #3629
 2020-02-20 12:08:17 +01:00
Norman Feske
5b650434b0 menu_view: unquote label text attribute value 
This patch enables the menu view to display '"' characters supplied via
the 'text' attribute of the <label> widget.

Issue #3607
Issue #3629
 2020-02-20 12:08:16 +01:00
Norman Feske
e612f7cd7d menu_view: add character position to hover report 
When setting the <dialog> attribute 'hover_details' to "yes", the hover
report features the character position of a hovered label.

Issue #3607
Issue #3629
 2020-02-20 12:08:16 +01:00
Norman Feske
7cc4aa2a28 menu_view: text cursor support 
This patch equips the label widget with the ability to display a text
cursor, as illustrated by the following example:

  <label text="...">
    <cursor at="10"/>
  </label>

Issue #3607
Issue #3629
 2020-02-20 12:08:16 +01:00
Norman Feske
60f5d0e34a gems/animated_geometry.h: increase value range 
The value range supported by the 'Animated_rect' utility is constrained
to 2^11 by the used 32-bit integer type. The interpolation is performed
on fixpoint numbers with a fractional precision of 10 bits. The
non-linear interpolation as performed by the 'Lazy_value' utility
involves squaring the values. Hence, the interpolated integral values
must not exceed 2^11 (2*11 bits for the square plus 10 bit for the
fractional part require 32 bits). This range is too small for
high-resolution displays. Hence this patch replaces the 32-bit integer
type by a 64-bit integer type.

Fixes #3627
 2020-02-20 12:08:16 +01:00
Norman Feske
46c5a90ba1 nitpicker_gfx: Text_painter::Font::index_at_xpos 
This method is useful for finding the character under a mouse position.
 2020-02-20 12:08:16 +01:00
Stefan Kalkowski
1273c573b6 wm: destroy view before newly creating it 2020-02-20 12:08:16 +01:00
Norman Feske
3a2895af19 wm: reset pointer position on leave event 
The wm used to remember the last pointer position when observing a leave
event. With such a stale pointer position, the wm would eventually
wrongly create an artifical motion event when a supposedly hovered
window changes its position (during an animated move). This can lead to
the propagation of the stale pointer position to the wm client, which,
in turn, may interpret the outdated position (e.g., the menu view would
report a wrong hover state).

This patch removes the stale state by resetting the '_pointer_pos' when
observing a leave event.

Fixes #3632
 2020-02-20 12:08:16 +01:00
Norman Feske
0bffac6c98 nit_fader: make fading steps configurable 
Fixes #3633
 2020-02-20 12:08:16 +01:00
Norman Feske
c25de5dba3 backdrop: make width and height customizable 2020-02-20 12:08:16 +01:00
Norman Feske
60edfa4d77 fs_query: add directory entries to listing report 
Fixes #3631
 2020-02-20 12:08:16 +01:00
Norman Feske
52e582132f os/vfs.h: handle corner case in join function 
When specifying "/" or "" as rel_path to the 'Directory' constructor,
the constructed directory should refer to the same directory. The
implementation of the join utility did not consider this corner case. It
occurred during the attempt to use fs_query with "/" given as path.

This patch also adds a Directory::Entry::dir accessor that returns true
if the entry is a directory.

Fixes #3630
 2020-02-20 12:08:16 +01:00
Norman Feske
a888041ba4 cached_fs_rom: fix file-handle leak 
Fixes #3634
 2020-02-20 12:08:16 +01:00
Norman Feske
844af06782 base/buffered_output.h: flush at destruction time 
By writing out all dangling characters at destruction time, the
'Buffered_output' utility can be used as a local variable rather
than a long-living object.
 2020-02-20 12:08:16 +01:00
Norman Feske
7da3404bd0 pkg/noux-system: increase nit_fb quota 
With this change, the window can be maximized on a full-HD screen.
 2020-02-20 12:08:16 +01:00
Sid Hussmann
7676f47540 depot_deploy: support heartbeat in deploy config 
Issue #3575
 2020-02-20 12:08:16 +01:00
Josef Söntgen
9d7a58f6a7 dde_bsd: update audio driver to OpenBSD 6.6 
Besides updating to a newer version the update adds the AC97 ICH driver
and addresses shortcomings with the OpenBSD emulation environment:

* Fix 'delay(9)' implementation - it now properly uses 'us' as unit,
  which results on faster initialization of the drivers.

* Fix LOG output that got lost during commit f23579532 and bring over
  the printf implementation from dde_linux for more structured
  printing.

* As said the driver now recognizes the AUICH devices. So far it was
  only tested with the device model in VirtualBox where it produces
  stuttering audio, investigating the cause is still ongoing.

Fixes #3641.
 2020-02-20 12:08:16 +01:00
Christian Prochaska
9bd3d2aa5c qt5: use main entrypoint in QPA plugin 
Fixes #3644
 2020-02-20 12:08:16 +01:00
Christian Prochaska
28e782dda5 qt5: use pthread backend for QThreads 
Fixes #3643
 2020-02-20 12:08:16 +01:00
Christian Prochaska
597098845c libc: support pthread cleanup handlers 
Fixes #3642
 2020-02-20 12:08:16 +01:00
Sebastian Sumpf
8a7deae238 ahci: switch to request stream API + structural changes 
- Remove dated 'Block::Driver' front end and implement
  'Block::Request_stream' front end
- Remove all dynamic memory allocations but DMA memory allocations
- Remove 'Platform_hba' and implement platform specific functions in
  'spec/<platform>/*'
- Ata and Atapi don't inherit from 'Port' any more, but are a member of
  'Port' as a protocol implementation
- Use platform driver for MMIO mappings (x86)
- Exchange stateful initialization of Ata/Atapi with a sequential
  initialization using 'wait_for_any' and 'retry' patterns
- Fix Atapi initialization by setting the byte count limit
- Set FIS receive base only when Cmd::FRE is disabled and Cmd::FR is 0
- Put everything in namespaces ('Ahci', 'Ata', or 'Atapi')
- Ata decides during read/write operations to use native-command queuing
  or normal DMA requests
- Remove port claiming logic (is now done via 'Constructibles')

fixes #3636
 2020-02-20 12:08:16 +01:00
Sebastian Sumpf
73f2c7043c ahci: remove Exynos5 support 
Exynos5 support has ceased on Genode

issue #3636
 2020-02-10 16:00:41 +01:00
Sebastian Sumpf
de24035066 base: add 'wait_for_any' to 'Register_set' 
This is the same as 'wait_for' with the difference that 'wait_for_any'
succeeds when *one* condition is met.

issue #3636
 2020-02-10 16:00:40 +01:00
Norman Feske
57ea1dbdd3 depot: update recipe hashes 2020-02-10 14:29:06 +01:00
Christian Prochaska
9f73f09cec tcp bulk test: add VFS server tests 
Fixes #3611
 2020-02-10 14:21:48 +01:00
Martin Stein
d56b21d329 nic_router: do not change ethernet header as hub 
When forwarding domain-local traffic like a hub, the NIC router shouldn't
modify packets at all.

Fixes #3625
 2020-02-10 14:21:47 +01:00
Christian Helmuth
8d60bc11b5 vbox: periodic timer based on absolute timestamps 
Before timestamps where calculated as durations since current wakeup
which introduces a systematical drift.
 2020-02-10 14:21:47 +01:00
Christian Helmuth
604f4c666b Remove Timed_semaphore from libc incl test package 
Issue #3550
 2020-02-10 14:21:47 +01:00
Christian Helmuth
ff5175ec76 libc: synchronization primitives based on monitor 
The libc monitor facility enables the execution of monitor jobs by the
main thread when the monitor pool was charged. In comparison to the
current suspend/resume_all mechanism the main thread iterates over all
job functions in contrast to waking up all threads to check their
conditions by themselves. Threads are only woken up if the completion
condition was met.

This commit is the result of a collaboration with Christian Prochaska.
Many thanks for your support, Christian.

Fixes #3550
 2020-02-10 14:21:47 +01:00
Christian Helmuth
6aebd5dd95 libc: append serial number to pthread name 
... in pthread_create()

Issue #3550
 2020-02-10 14:21:47 +01:00
Christian Helmuth
3d4bed3374 libc: unify semantic of sleep functions 
sleep(), usleep(), and nanosleep() now return immediately on
zero-timeout. Also, non-zero timeouts sleep at least 1 ms (the current
minimal timeout in libc), which compensates rounding errors.

Issue #3550
 2020-02-10 14:21:47 +01:00
Alexander Boettcher
ee7a77643e depot_deploy: support affinity configuration 
Fixes #3597
 2020-02-10 14:21:47 +01:00
Alexander Boettcher
646d6c368c core: fixup cpu weight on failed thread creation 
Fixes #3613
 2020-02-07 14:14:20 +01:00
Alexander Boettcher
d96e14fe16 sandbox: adjust location size scaling 
by incorporating session location

Issue #1412
 2020-02-07 14:14:20 +01:00
Alexander Boettcher
3a9d450106 trace: avoid page fault on client calling subjects 
Fixes #3608
 2020-02-07 14:14:20 +01:00
Martin Stein
a73ef9fc06 app/ping: fix destination MAC on ARP replies 
The ARP destination MAC was set to the Ethernet source MAC which is not
necessarily the MAC of the other ARP peer. Instead use the ARP source MAC of
the ARP request.
 2020-02-04 16:12:04 +01:00
Stefan Kalkowski
3016b64fac mesa: enable ARM-64 port 
Fix #3622
 2020-02-04 16:10:28 +01:00
Stefan Kalkowski
22498e0b09 os: add backtrace utility for arm_64 
Fix #3621
 2020-02-04 16:10:22 +01:00
Stefan Kalkowski
79dff674fd gems: add connect facility to tcp_terminal 
Instead of listening for new TCP/IP connections only,
the TCP terminal connects to a server when an IP address is
configured in the policy for a terminal client.

Fix #3619
 2020-02-04 16:10:09 +01:00
Stefan Kalkowski
56ef7ca9e7 hw: enable ARMv8 performance counter 
Fix #3618
 2020-02-04 16:05:01 +01:00
Stefan Kalkowski
9db50753f1 libports: add ARMv8 support for libpng 
Fix #3617
 2020-02-04 16:04:54 +01:00
Stefan Kalkowski
e84e1bbf36 dde_linux: optimize refresh routine in imx8_fb_drv 
Fix #3616
 2020-02-04 16:03:08 +01:00
Stefan Kalkowski
fda337a1c0 os: increase throughput of blit for ARM 64-bit 
Fix #3615
 2020-02-04 16:01:35 +01:00
Stefan Kalkowski
f49f91da08 os: increase RAM quota for drivers in fb_bench.run 
Enables fb_bench.run script for new imx8 framebuffer driver.

Ref #3615
 2020-02-04 15:59:33 +01:00
Christian Helmuth
90535a1401 depot: update recipe hashes 2020-02-04 15:51:10 +01:00
Christian Helmuth
43719b5fd1 libc: fix returned timeout-left value 
If the suspend method for the main thread detects that the suspend
condition is false it must return the passed timeout value (not always
0). Otherwise, the caller may incorrectly assume the timeout expired.
 2020-02-04 15:51:10 +01:00
Stefan Kalkowski
2a94f8cdb4 depot: update public key for skalk 2020-02-04 15:51:10 +01:00
Christian Helmuth
1e578f1a50 Log key count in input test 2020-02-04 15:51:10 +01:00
Alexander Boettcher
a036d2373a x86/ps2: disable devices during commands 
Especially the initial reset ended up with an unusable keyboard in rare
cases before. This commit disables keyboard and mouse handling of i8042
during command sequences according to a note from

  https://www.win.tue.nl/~aeb/linux/kbd/scancodes-12.html
 2020-02-04 15:51:10 +01:00
Norman Feske
a2b303e95a Road map for 2020 2020-02-04 15:51:10 +01:00
Alexander Boettcher
7b964fa700 base: add affinity space info to platform info 
Issue #3599
 2020-02-04 15:51:10 +01:00
Alexander Boettcher
72f5f9d133 sculpt_manager: apply affinity-space to runtime 
Issue #3599
 2020-02-04 15:51:10 +01:00
Christian Prochaska
186a6bc080 test-tcp_bulk_lxip: increase timeout value 
Fixes #3609
 2020-02-04 15:51:10 +01:00
Norman Feske
567d9f7910 Xml_node::Unquoted utility for attribute values 
The new utility can be used to revert quoted XML attribute values.
Such quoting is needed whenever an attribute value can contain '"'
characters. E.g., in the menu_view's <label text="..."> widget.

Issue #1757
 2020-02-04 15:51:10 +01:00
Norman Feske
d70cf314d8 Never throw in 'Attached_rom_dataspace::size' 
If the ROM service returned in invalid dataspace, reflect this condition
via a size of zero instead of triggering an exception of type
'Reconstructible<Attached_dataspace>::Deref_unconstructed_object'.

Issue #3606
 2020-02-04 15:51:10 +01:00
Norman Feske
c6445da654 vfs: watch support for <rom> file system 
Fixes #3606
 2020-02-04 15:51:10 +01:00
Norman Feske
96cde52838 vfs: make <rom> fs robust against invalid ROMs 
An invalid ROM dataspace can occur, for example, when requesting a ROM
from the report_rom service before the first report was posted. Such a
<rom> node can still be useful if it's dynamic. E.g., it can be watched.
Hence, the corner case of an invalid ROM dataspace should be gracefully
handled instead of causing an abort.

Issue #3606
 2020-02-04 15:51:09 +01:00
Norman Feske
c67a0d3dd8 os/dynamic_rom_session.h: support Xml_generator 
This patch adds a special variant of a 'Content_producer' called
'Xml_producer', which allows the direct use of an 'Xml_generator'
for generating the ROM-module content.
 2020-02-04 15:51:09 +01:00
Norman Feske
78c0e5f6b6 New sandbox library extracted from init component 
This patch extracts the child-management functionality from the init
component into a new library called "sandbox". The library API is
located at 'os/include/os/sandbox.h'.

The sandbox API allows for the interaction of the component with the
sandboxed children by providing locally implemented services. This
mechanism is illustrated by the new test at os/src/test/sandbox.

Issue #3601
 2020-02-04 15:51:09 +01:00
Norman Feske
f82e7df0ba os/buffered_xml.h: support for generating XML 
This patch extends the 'Buffered_xml' utility with a new constructor
that fills the buffer with the output of an 'Xml_generator'. It thereby
presents an easy way to generate XML to be consumed locally.

The patch also add a deprecation mark to the original 'xml' accessor
because copying 'Xml_node' objects (here as return value) is dangerous.
The new 'with_xml_node' method should instead be used to access the XML
content stored in the buffer.

Fixes #3602
 2020-02-04 15:51:09 +01:00
Norman Feske
640a001ab6 util/xml_node.h: avoid internal use of exceptions 
The XML parser used to rely in C++ exceptions while parsing, which is an
artifact from the initial implementation. This patch reworks the code such
that exceptions are avoided in the common cases.

Fixes #3605
 2020-02-04 15:51:09 +01:00
Alexander Boettcher
c5c5f8754c test-init: add test to check for affinity warning 
Issue #1412
 2020-02-04 15:51:09 +01:00
Alexander Boettcher
5be3bf4f26 init: warn about missing affinity-space 
if affinity is set by children

Issue #1412
 2020-02-04 15:51:09 +01:00
Alexander Boettcher
d132fc0a73 init: scale affinity location 
Issue #1412
 2020-02-04 15:51:09 +01:00
Christian Helmuth
285a33c97d dde_linux: add missing white space in log message 2020-02-04 15:51:09 +01:00
Alexander Boettcher
f09ac23144 base: Timestamp type on arm_64 is 64bit 2020-02-04 15:51:09 +01:00
Alexander Boettcher
734752d6b5 base-nova: show size of framebuffer correctly 2020-02-04 15:51:09 +01:00
Christian Prochaska
4fc6c4ff5c test-tcp_bulk_lwip: increase timeout value 
Fixes #3604
 2020-02-04 15:51:09 +01:00
Christian Prochaska
746d373362 vfs_lxip: classify 'data' file as continuous 
Fixes #3603
 2020-02-04 15:51:09 +01:00
Alexander Boettcher
2256f5fb4b dde_linux: add dma_wmb/rmb barriers for arm_v6 2020-02-04 15:51:09 +01:00
Christian Prochaska
d8e2c95597 vfs server: ensure retry for stalled session 
Fixes #3600
 2020-02-04 15:51:09 +01:00
Alexander Boettcher
6506240642 dde_linux: add dma_wmb/rmb barriers for arm_v7 
and remove re-defined dma_wmb in fec driver
 2020-02-04 15:51:09 +01:00
Sebastian Sumpf
bd284347da libports: fix undefined reference for gmp on arm_v8 
* enable native 'invert_limb' implemenation
* disable native 'udiv_qrnnd' implemenation

fixes #3598
 2020-02-04 15:51:08 +01:00
Sebastian Sumpf
3813f9772a libports: add '__gmpn_invert_limb' for arm_v7 
Implementation was missing.

issue #3598
 2020-02-04 15:51:08 +01:00
Sebastian Sumpf
1902d1a06b hw: do not map boot modules in bootstrap 
This commit safes virtual address space.
 2020-02-04 15:51:08 +01:00
Josef Söntgen
7ecabb25eb ssh_terminal: address ambigouity of return values 
Replace return values with appropiate bool and document two-staged
publickey authentication. This fixes a bug where wrong authentication
attempts are not properly denied.

Issue #3590.
 2020-02-04 15:51:08 +01:00
Alexander Boettcher
5b633a83df core: dissolve thread object on failed creation 
Fixes #3596
 2020-02-04 15:51:08 +01:00
Norman Feske
beb8bf498c base-hw: add explicit array-bounds check 
This patch rules out out-of-bounds array accesses without inspecting the
caller. It is not a bug fix but adds clarity.
 2020-02-04 15:51:08 +01:00
Norman Feske
de764d8490 base-hw/muen: add explicit array-bounds check 2020-02-04 15:51:08 +01:00
Norman Feske
5635c1318c base-hw: explicit bounds check in store_apic_id 
Avoid relying on the caller regarding the CPU index argument to ease the
reasoning about the code.
 2020-02-04 15:51:08 +01:00
Norman Feske
01713c74f9 base-foc: explicit array-index bounds check 
Adding an explicit check clarifies at the first glance that the array
bounds are respected without needing knowledge about MAX_CAPS_PER_MSG
and L4_UTCB_GENERIC_DATA_SIZE (used for dimensioning 'mr').
 2020-02-04 15:51:08 +01:00
Norman Feske
9ec66f0594 allocator_avl: simplify structure 
This patch changes the code of '_alloc_two_blocks_metadata' to not
leak the result of a partial allocation in the out parameters of
the method. This eases the reasoning about the absence of a
use-atfer-free problem (there was none).
 2020-02-04 15:51:08 +01:00
Norman Feske
6947bddd3f base: avoid use of memcpy 2020-02-04 15:51:08 +01:00
Norman Feske
37ec636018 test/smp: add missing destruct call 2020-02-04 15:51:08 +01:00
Norman Feske
9bba6613e7 base/affinity.h: simplify member initialization 
This is just a minor style improvement.
 2020-02-04 15:51:08 +01:00
Christian Prochaska
d4f246517c depot: recipe for drivers_interactive-imx8q_evk 
Fixes #3595
 2020-02-04 15:51:08 +01:00
Stefan Kalkowski
5bfebe7a3f depot: recipe for usb_hid_drv 
Fixes #3594
 2020-02-04 15:47:27 +01:00
Stefan Kalkowski
3df67362b4 depot: recipe for usb_host_drv 
Fixes #3593
 2020-02-04 15:47:27 +01:00
Christian Prochaska
f1042e7fb1 depot: recipe for imx8_fb_drv 
Fixes #3592
 2020-02-04 15:47:27 +01:00
Christian Prochaska
b29112efdf dde_linux: framebuffer driver for i.MX8 EVK 
Fixes #3591
 2020-02-04 15:47:27 +01:00
Stefan Kalkowski
fe899eecc7 base-linux: fix configurable host toolchain 
With the '-x' argument of the 'g++' tool the language gets specified,
therefore it has to be 'c++' furthermore, not CUSTOM_HOST_CXX. Moreover,
when import-lx_hybrid.mk gets evaluated first, global.mk is not included
yet, therefore we cannot eagerly evaluate the CXX_LINK_OPT_NO_PIE variable.

Issue #3466
 2020-02-04 15:47:27 +01:00
Norman Feske
c2a2ec121f libc-net/resolv: silence warning in 3rd-party code 2020-02-04 15:47:27 +01:00
Norman Feske
e1e1fa23b7 zlib: fix compile warning 
This patch includes 'unistd.h' to avoid warnings about the implicit
declaration of the functions ‘read’, 'close', and 'write.
 2020-02-04 15:47:27 +01:00
Norman Feske
aee8d35dc4 libcrypto: suppress warning about unused value 2020-02-04 15:47:27 +01:00
Christian Helmuth
4bbbf5d2e3 dde_linux: reduce compiler warnings 2020-02-04 15:47:27 +01:00
Norman Feske
ba7e832c5d ssh_terminal: fix compile warning 
server.h:51:12: warning: ‘int write_avail_cb(socket_t, int, void*)’
                declared ‘static’ but never defined [-Wunused-function]

This patch fixes the warning by moving the user of 'write_avail_cb' to
the compliation unit that defines it.
 2020-01-03 14:44:31 +01:00
Norman Feske
bbfc092a31 core: avoid use of C array as buffer in Log_root 2020-01-03 14:44:15 +01:00
Norman Feske
de52cf1cdd core: check return value of map.metadata 
Even though the call of map.metadata is known to always return a valid
pointer (because the meta data is assigned in the code just above),
better add an explicit nullptr check.
 2020-01-03 14:44:15 +01:00
Norman Feske
783c05fd6c uart/exynos: be explicit about unused return value 2020-01-03 14:44:15 +01:00
Norman Feske
6ae98e2e6d base: mark Parent::Close_result with [[nodiscard]] 2020-01-03 14:44:15 +01:00
Norman Feske
ffc099eb54 cxx: add nullptr check in calloc 
Even though malloc via the cxx_heap never fails, the implementation of
calloc should better not rely on this assumption.
 2020-01-03 14:44:15 +01:00
Christian Helmuth
9321067b68 Remove unused/redundant longlong.h 
This was used by an ancient libgmp port - longlong.h is now part of the
GMP sources.
 2020-01-03 14:21:41 +01:00
Christian Prochaska
0eaa1f7a08 lx_kit: support enable/disable IRQ and pass IRQ number to handler 
Fixes #3589
 2020-01-02 15:53:46 +01:00
Christian Helmuth
18f90ca1e3 nova: request stack pointer on page fault 
The SP value is part of the Page_fault_info object but was not requested
from the kernel on portal creation.
 2020-01-02 15:30:28 +01:00
Christian Helmuth
9a35743df6 Mark packet-stream descriptor head/tail as volatile 
`volatile` effectively prevents instruction reordering by the compiler
and fixes an issue with -O3 compiled components.

Note, this commit does not address further arguments regarding memory
barriers and volatile voiced in issue #693.

Issue #693
 2019-12-20 14:34:22 +01:00
Christian Helmuth
8d63a3c1f3 Increase timeout of RTC test 
It takes its time on sel4 and Qemu.
 2019-12-20 14:28:18 +01:00
Alexander Boettcher
1ac33caa90 nova: handle rpc_entrypoint destruction better 
Fixes #1921
 2019-12-20 11:31:39 +01:00
Alexander Boettcher
1c361bf545 base: add test to destruct entrypoint 
Issue #1921
 2019-12-20 11:31:27 +01:00
Alexander Boettcher
a41dd48986 nova: avoid invalid capability counting overhead 
- handle it the same way as done for other kernels

Fixes #3579
 2019-12-20 11:29:21 +01:00
Christian Helmuth
b931b67cba depot: update recipe hashes 2019-12-19 17:01:43 +01:00
Christian Helmuth
24435e9ca1 Run TCP bulk tests on all Qemu platforms 
This explicitly includes ARM32/64.
 2019-12-19 17:01:43 +01:00
Alexander Boettcher
e54ff599ca base-hw: trace execution time of core threads 
Fixes #3572
 2019-12-19 17:01:43 +01:00
Sebastian Sumpf
04969b6be0 base-hw: add trace execution time support 
This enables the 'top' program on base-hw for debugging issue #3247 on
rpi.

Fixes #3572
 2019-12-19 17:01:42 +01:00
Christian Helmuth
1ddf1dbc25 Use ram-block in libc vfs tests on Rpi3 2019-12-19 17:01:42 +01:00
Christian Helmuth
8699f5592f wifi: run wpa_main in pthread 2019-12-19 17:01:42 +01:00
Norman Feske
73d089da36 Quota tweaks for base-sel4 2019-12-19 17:01:42 +01:00
Christian Prochaska
c8cd09e72c qt5: don't build qt5_jscore and qt5_webcore for arm_v8a 
Fixes #3585
 2019-12-19 17:01:42 +01:00
Christian Prochaska
be1ef01f10 stdcxx: add symbols needed by dosbox 
Fixes #3588
 2019-12-19 17:01:42 +01:00
Alexander Boettcher
0a1bc1f4b7 nova: add requesting time of killed SCs 
Fixes #3587
 2019-12-19 17:01:42 +01:00
Alexander Boettcher
bcb7f45201 core: avoid dangling ptr in cpu thread component 
if exception is triggered during construction of a thread
 2019-12-19 17:01:42 +01:00
Alexander Boettcher
33db0e0d4d base: add uboot for imx8 as port 
that contains scripts to download and build uboot

Fixes #3584
 2019-12-19 17:01:42 +01:00
Christian Helmuth
504539ad1e vfs/lwip: notify I/O progress on sent packets 
Incoming ACK packets for sent data packets may be the only unblocker for
suspended write/send loops. This patch informs VFS users about I/O of
VFS handle on successfully sent packets.
 2019-12-19 17:01:42 +01:00
Christian Prochaska
a62fce8dc5 sd_card_drv: i.MX8 support 
Fixes #3580
 2019-12-19 17:01:42 +01:00
Christian Prochaska
81a78cf1d0 usb_host_drv: fix 'matches->compatible' string access 
Fixes #3582
 2019-12-19 17:01:42 +01:00
Christian Helmuth
283135c9cd test-libc_getenv: allow unrelated content in log pattern 2019-12-19 17:01:42 +01:00
Sebastian Sumpf
22d4d5c1c1 ldso: update link map during respawn 
move binary to front of link map after respawn. This is required by GDB
in order to load symbol files correctly.

issue #3481
 2019-12-19 17:01:42 +01:00
Sebastian Sumpf
9c372c36c1 libc: thread local errno support 
Store errno in pthread objects, return member upon call to '__error()'.
This became necessary in order to make errno thread-safe.

Note, any call to libc code from a non-pthread (beside the first
entrypoint) is not supported.

issue #3568
 2019-12-19 17:01:42 +01:00
Sebastian Sumpf
9767c4db0e vfs_lwip/lwip: Lwip::lock for concurrent access 
This is required in case the send and receive of IP packets is
performed by multiple thread because lwip is not thread safe.

issue #3568
 2019-12-19 17:01:42 +01:00
Alexander Boettcher
9812799b24 seoul: reduce CPU load in graphic mode 
- Stop refreshing if all CPUs go to sleep, also in text mode.
- Any input resets update rate to 100Hz in graphic mode, which gets
  decreased step by step down to 25Hz if no input is available (looking video
  without input by user).

Fixes #3576
 2019-12-19 17:01:42 +01:00
Alexander Boettcher
d385749ead seoul: make vmm memory configurable 
- document Genode specific configuration option
 2019-12-19 17:01:42 +01:00
Christian Helmuth
23ed5d3936 base-linux: configurable host toolchain 
This commit uses CUSTOM_HOST_CC/CUSTOM_HOST_CXX instead of hardcoded
commands and introduces HOST_DEV_PREFIX.

Original patch by Roman Iten and Pirmin Duss.

Issue #3466
 2019-12-19 17:01:42 +01:00
Christian Helmuth
cebc963396 Cleanup linux_rtc_drv and lx_fs target.mk 
Original patch by Roman Iten and Pirmin Duss.

Issue #3466
 2019-12-19 17:01:42 +01:00
Norman Feske
0c8ec41c21 vfs/pipe: let num_dirent return 0 
This prevents the original value ~0UL from messing up the output
of 'ls -l /dev'.

Issue #3578
 2019-12-19 17:01:42 +01:00
Norman Feske
9f7b8c1a17 vfs/log: allow output redirection via '> /dev/log' 
Issue #3578
 2019-12-19 17:01:42 +01:00
Norman Feske
cd92b32622 libc: close all open FDs on exit 
This is important to issue sync requests for written-to files.

As the closing must be performed by an atexit handler, it happens at a
time _after_ libc plugins are destructed. Consequently an FD allocated
by such a plugin results in a close error, which in turn, does not
destruct the FD. We ultimatedly end up in an infinte loop of
re-attempting the close. For this reason, the patch changes 'close' to
be robust against this special case.

This is generally not a problem because libc plugins are phased out.
However, at present, the libc_noux plugin is still important. With the
changed 'close' in place, there occurred an error message "Error: close:
close not implemented" at the exit of each noux program. This patch
removes the error printing from the libc plugin mechansim to avoid this
noise. The error messages are not important anyway because the
deprecation of the libc plugin interface.

Issue #3578
 2019-12-19 17:00:48 +01:00
Norman Feske
5853a68904 libc: silence calls of getpeername w/o socket fs 
The getpeername function is provided only by the socket fs.
In the case where the socket fs is not configured, return an appropriate
errno instead probing for a libc plugin (there is none).

Issue #3578
 2019-12-19 17:00:48 +01:00
Norman Feske
ae64830bd5 libc: silence get(e)gid, get(e)uid, getppid 
These dummies spam the log when running bash + make, like in the
genodians.org scenario.

Issue #3578
 2019-12-19 17:00:48 +01:00
Norman Feske
e8878eee8a ldso: increase initial heap block 
This is needed to enable the fork of 'make' in the genodians.org
scenario on NOVA.

Issue #3578
 2019-12-19 17:00:48 +01:00
Norman Feske
3897ddea03 cxx: don't rely on global ctors 
This patch removes the global variable 'blocker', which was expected to
be constructed via the global ctors. This mechanism, however, is not
used for the base library, which resulted in the use of an unconstructed
object. Specifically, the spinlock of the 'Lock' of the 'Registry'
defaulted to the LOCKED state (value 0), which eventually would lead to
a deadlock in the contention case of the cxa guard.

I could observe this deadlock once on during the component startup on
base-linux during the construction of the 'startup_lock'.

This patch fixes the problem by explicitly initializing the registry
of blockers via an init function.

Issue #2299
Issue #3578
 2019-12-19 17:00:48 +01:00
Norman Feske
6858270517 base: disarm atexit handling for signal thread 
The signal thread is not supposed to be destructed in any scenario other
than the noux fork mechanism (where no signals occur).

Issue #3578
 2019-12-19 17:00:48 +01:00
Norman Feske
4e57b6eceb core: fix destruction of dependent CPU sessions 
Issue #3578
 2019-12-19 17:00:48 +01:00
Norman Feske
298f317f44 core: remove quota-related diagnostic messages 
Issue #3578
 2019-12-19 17:00:48 +01:00
Norman Feske
5820ad8309 fiasco: silence ipc_send error messages 
These diagnostic messages spam the log but can occur during the regular
component-destruction procedure.

Issue #3578
 2019-12-19 17:00:48 +01:00
Norman Feske
b7fbe65ff2 libc: fork/execve improvements 
- Close FDs marked with the close-on-execve flag
  (needed for 'make', which sets the flag for the pipe-in
  FD of forked children)
- Update binary name on execve to use as ROM for subsequent fork
- Enable vfork as an alias for fork (needed by make)
- Purge line buffers for output streams during execve because they
  may be allocated at the allocation heap, which does not survive
  the execve call.
- Consider short-lived processes that may exit while the parent still
  blocks in the fork call.

With these changes, the website generator of genodians.org works without
the need for the Noux runtime.

Issue #3578
 2019-12-19 17:00:47 +01:00
Norman Feske
d1cf216384 libc: propagate pipe config to child processes 
Issue #3578
 2019-12-19 17:00:47 +01:00
Alexander Boettcher
3011dc5876 sculpt: add vbox5 generic package as option 
Fixes #3574
 2019-12-19 17:00:47 +01:00
Alexander Boettcher
6e99f00f5c usb_hid_raw: enable imx8q_evk for nightly testing 
Issue #3571
 2019-12-19 17:00:47 +01:00
Alexander Boettcher
beb1e084a6 base: add imx8 evk usb driver to platform config 
Issue #3571
 2019-12-19 17:00:47 +01:00
Alexander Boettcher
e34b443c29 usb: add imx8q_evk support to usb_host 
Issue #3571
 2019-12-19 17:00:47 +01:00
Alexander Boettcher
6b17bb647e dde_linux: add barriers to readl/writel 
Issue #3571
 2019-12-19 17:00:47 +01:00
Christian Helmuth
4299b85cdb base-linux: support invalid dataspace for RM client 
This restores behavior introduced in

  commit cb232891bf
  Author: Norman Feske <norman.feske@genode-labs.com>
  Date:   Tue May 10 11:55:25 2016 +0200

    Fix noux.run on base-linux

but removed by issue #2829.

Related to issue #1938
 2019-12-19 17:00:47 +01:00
Sebastian Sumpf
6dae147785 libc: limit fd id allocations to FD_SETSIZE 
fd > FD_SETSIZE cannot use 'select' or 'poll' within our libc.
Therefore, we added a bit allocator in order to allocate fd < FD_SETSIZE
(1024).

fixes #3568
 2019-12-19 17:00:47 +01:00
Sebastian Sumpf
e4255e4c8b vfs/lxip: wait for i/o progress on connect 
Since connect is asynchronous, add a waiter during connect

issue #3568
 2019-12-19 17:00:47 +01:00
Christian Helmuth
161274f785 pistachio: check for 'ed' tool on port prepare 
Some debian systems seems to come without 'ed' in the base installation.
The tool is needed when building the pistachio kernel.
 2019-12-19 17:00:47 +01:00
Roman Iten
6145cdcf37 run: close connection to spawned processes on kill 
From the man page of expect:

> Both expect and interact will detect when the current process exits
> and implicitly do a close.  But if you kill the process by, say, "exec
> kill  $pid",  you  will need to explicitly call close.

Fixes #3569
 2019-12-19 17:00:47 +01:00
Tomasz Gajewski
b57a4c98cf tool/run: run external command plugin for power_on subsystem 
Fixes #3385
 2019-12-19 16:59:03 +01:00
Emery Hemingway
a3e43aca87 Clang: undeclared identifier __UINT64_C 
Issue #3564
 2019-12-19 16:59:03 +01:00
Norman Feske
6b6915e304 ttf_font: fix artifacts with large font sizes 
This patch is a follow-up fix for "ttf_font: prevent out-of-bounds
access", which produced sporadic single-pixel artifacts with large font
sizes, e.g., with the monospaced font set to 24px in Sculpt.

The artifacts occurred only for some horizontal sub-pixel positions, in
combination with the font cache, and after the rendering of '>'
characters. They were ultimately caused by the missed clearning of the
first pixel of a glyph where x0 is 1 (e.g., the character 'd'). In this
case, a pixel from the previously evicted cache entry (the '>') shined
through. The patch fixes the problem by clearing the glyph starting from
the first, not the x0's, pixel.

Fixes #3567
 2019-12-19 16:59:03 +01:00
Alexander Boettcher
3f83ac5580 tool: add imx8_evk board variable hint 2019-12-19 16:59:03 +01:00
Emery Hemingway
7f57de1b74 VFS/terminal: fix clang warning 
Variable-sized object may not be initialized.

Issue #3564
 2019-12-19 16:59:03 +01:00
Emery Hemingway
648382db74 Align after template expansion 
Clang Cannot align template structs. Also, cannot cast void* to addr_t
in constexpr function.

Issue #3564
 2019-12-19 16:59:03 +01:00
Emery Hemingway
2c510bb7f9 Remove unused lamba capture to fix clang warning 
Issue #3564
 2019-12-19 16:59:03 +01:00
Emery Hemingway
23710dff5e Remove dead code 
Issue #3564
 2019-12-19 16:59:03 +01:00
Emery Hemingway
ff0436357b nic_stress: keep signal-context cap by value 
Fixes clang warning about binding reference to stack allocated
parameter.

Issue #3564
 2019-12-19 16:59:03 +01:00
Emery Hemingway
091e5157aa vfs server: fix File_system namespace warnings 
Issue #3564
 2019-12-19 16:59:03 +01:00
Alexander Boettcher
c1e181a407 vbox: handle invalid dataspace exception 
Fixes #3566
 2019-12-19 16:59:03 +01:00
Stefan Kalkowski
8f71c90ca8 dde_rump: backport aarch64 (Fix #3565) 2019-12-19 16:59:03 +01:00
Norman Feske
f85ec313de os/vfs.h: handle zero-sized files 
The 'File_content' class failed to consider zero-sized files.

Fixes #3557
 2019-12-19 16:59:03 +01:00
Norman Feske
2aa6471608 test/depot_query: add test for zero-sized archives 
This test covers the situation where depot_query evaluates depot content
that is incompletely extracted. In particular, if the 'archives' file
already exists but has a size of zero, depot_query would produce a
page fault. This situation can be manually provoked by deliberately
creating a zero-sized archives file for any otherwise correct pkg.

This patch also fixes the success indicator of the test. It wrongly
matched an early line of the log output.

Issue #3557
 2019-12-19 16:59:02 +01:00
Stefan Kalkowski
9814fc5447 libports: enable arm_64 libssl, libcrypto variant 
Fix #3563
 2019-12-19 16:59:02 +01:00
Martin Stein
3655ea77a3 depot_autopilot: rework log matching 
* The log history of a test is stored as a whole for the lifetime of the test.
* Matching of the log history against log patterns is done correctly now
  (previously, a pattern like "AAB" on an input like "AAAB" wouldn't have
  triggered).
* Use memcmp, memcpy, memmove instead of the former character-wise operations.
* Sanitizing of log input and log patterns now works more generic through the
  new Filter class for all replacements/removals.
* Sanitizing is done as soon as a string is available and remains for the
  lifetime of the test.
* Sanitizing doesn't interfer with the matching algorithm.
* Decomposing into small clearly named functions.

Ref #3555
 2019-12-19 16:59:02 +01:00
Johannes Kliemann
89d35bc41e tool: boot Genode on bare metal Linux 
Fixes #2829
 2019-12-19 16:59:02 +01:00
Johannes Kliemann
93639532f0 base-linux: core session support (IO_PORT, IO_MEM, IRQ) 2019-12-19 16:59:02 +01:00
Christian Helmuth
8cf7aaad65 version: 19.11 2019-11-28 11:49:10 +01:00
Norman Feske
25a8ef3b7c News item for version 19.11 2019-11-28 11:44:29 +01:00
Norman Feske
b18a56c2c4 Release notes for version 19.11 2019-11-28 11:44:29 +01:00
Christian Helmuth
9d42e3f69b release notes: fix example in 19.08 input section 2019-11-28 11:44:29 +01:00
Christian Helmuth
11ef8e1ff2 depot: update recipe hashes 2019-11-28 09:06:39 +01:00
Christian Helmuth
1deab4a67f Disable noux_tool_chain_auto on foc/imx7d_sabre 
The bootstrapper always fails with

  fatal: could not find free RAM region for modules

in this scenario.
 2019-11-28 09:06:27 +01:00
Christian Helmuth
f23c70e068 sculpt_manager: use Codepoint::valid() 2019-11-28 09:06:26 +01:00
Alexander Boettcher
f7c818d303 vbox5: add recipe of kernel independent version 2019-11-25 16:02:40 +01:00
Stefan Kalkowski
7996cf06ab os: remove jitterentropy from fs_packet test 
Fixes #3562
 2019-11-25 15:44:57 +01:00
Stefan Kalkowski
4800bcf5a0 hw: correct the i.MX6 Sabrelite timer settings 
Fix #3561
 2019-11-25 15:43:59 +01:00
Christian Prochaska
4c74f4792c ahci_drv: set 'Cmd::St' to 1 before writing 'Sact' register 
Issue #3058
 2019-11-25 14:15:57 +01:00
Christian Prochaska
57d080d4f8 hw: use correct type on IRQ kernel object destruction 
Fixes #3560
 2019-11-25 14:15:39 +01:00
Alexander Boettcher
2778debc29 ram_fs: close open nodes on session close 
Fixes #3559
 2019-11-22 15:59:13 +01:00
Martin Stein
7182c10c90 test/nic_stress: test bad session args 
Fixes #3525
 2019-11-22 14:16:24 +01:00
Christian Prochaska
7309bcf4b5 gdb_monitor: add platform checks in run scripts 
Fixes #3558
 2019-11-22 14:12:06 +01:00
Sebastian Sumpf
309bc2083e vbox5: raise rrtimer priority 
Make sure timers run at the same priority as component threads, otherwise
no timer progress can be made. See 'rtTimeNanoTSInternalRef' (timesupref.h)
and 'rtTimerLRThread' (timerlr-generic.cpp).
 2019-11-22 13:33:15 +01:00
Norman Feske
6e098a9d17 ram_fs: apply session policy to write / mtime ops 
This patch extends the distinction between read-only and writeable
sessions to packet-stream operations for WRITE and WRITE_TIMESTAMP.
 2019-11-21 16:21:19 +01:00
Norman Feske
573b6d3345 noux: fix condition for mtime update 
This is a follow-up patch for "noux: don't update mtimes for read-only
files". It eliminates warning messages during Sculpt's prepare step when
'cp' tries to update the mtimes of the source files.
 2019-11-21 16:21:19 +01:00
Norman Feske
077fa355ce pkg/wm: increase RAM quota 
This patch avoids a warning during the wm startup in Sculpt.
 2019-11-21 15:13:20 +01:00
Norman Feske
e76ce05844 sculpt manager: ignore empty deploy configs 
When copying config/managed/deploy to config/deploy, the latter may
temporarily result in an empty configuration. Such an intermediate
state should be ignored to keep the currently running scenario in tact
instead of restarting it.
 2019-11-21 15:07:51 +01:00
Norman Feske
4622ddb46f nitpicker: add version attribute in clicked report 
This patch addresses a corner case where the nitpicker focus is not
solely defined by mouse clicks or (exclusively) by a window manager, but
by a policy component that takes mouse clicks and other policy (e.g.,
a lock screen) into account. It ensures that each click that follows a
focus change (however initiated) results in a new "clicked" report even
when the report looks the same. To allow the policy component to
uniquely distiguish subsequent reports, the report features a new
'version' attribute.

Fixes #3493
 2019-11-21 14:31:52 +01:00
Stefan Kalkowski
972e1893c9 os: introduce vmm component for ARMv8 
Fix #3553
 2019-11-21 14:29:36 +01:00
Stefan Kalkowski
af29dcf557 hw: introduce virtualization support for ARMv8 
Ref #3553
 2019-11-21 14:29:36 +01:00
Stefan Kalkowski
f82714f341 vm_session: return vcpu id when creating vcpu 
Ref #3553
 2019-11-21 14:29:36 +01:00
Stefan Kalkowski
02d68fdb97 hw: move arm virtualization to generic place 
Ref #3553
 2019-11-21 14:29:36 +01:00
Stefan Kalkowski
065b9fdb46 base-hw: extend syscalls to five arguments 
Ref #3553
 2019-11-21 14:29:36 +01:00
Stefan Kalkowski
18dbd75860 base: explictly target entrypoint in timer session 
Fix #3548
 2019-11-21 14:29:36 +01:00
Norman Feske
3ac970ac1d Streamline Timestamp::INVALID values 
This is a follow-up patch for issue #1784 that solves two
inconsistencies.

- The Vfs::Timestamp::INVALID matches File_system::Timestamp::INVALID
- The Noux libc plugin tests for Timestamp::INVALID instead of a
  positive value.

The patch fixes the mtime info as shown in directory listings in
Sculpt's inspect window.
 2019-11-21 14:26:27 +01:00
Norman Feske
3aaed7188f ram_fs: tie writeable bit to session policy 
Related to issue #3507
 2019-11-21 14:26:01 +01:00
Norman Feske
ee64e29e77 vfs server: tie writeable bit to session policy 
This patch let the VFS server reflect the session policy via the
writeable bit in directory entries instead of merely forwarding the bit
from the respective VFS plugin. This way, all files originating from a
read-only file-system session automatically appear in directory listings
as read-only files.

Related to issue #3507
 2019-11-21 14:25:43 +01:00
Norman Feske
cfba429c15 vfs/rump: offer all files as read-writeable 
The distinction between read-only and read-writeable is the
responsibility of the front end, i.e., the vfs server.
 2019-11-21 14:25:05 +01:00
Josef Söntgen
25aa25c6a0 wifi_drv: enable loading of 9000 series FW 
Tested with a 'Intel(R) Wireless-AC 9462' device.

Fixes #3556.
 2019-11-20 12:56:10 +01:00
Josef Söntgen
2afc02051c sculpt_manager: increase Rom_name size 
The current value of 32 is too short for accessing the
iwlwifi 9000 ucode ROM.

Issue #3556.
 2019-11-20 12:56:03 +01:00
Christian Helmuth
ce1b813105 depot: update recipe hashes (vbox, sculpt) 2019-11-19 16:08:19 +01:00
Christian Helmuth
7ed1d7f11d depot: update recipe hashes 2019-11-19 14:54:14 +01:00
Norman Feske
6ccd65bd8e sculpt_manager: invoke mkfs.ext2 with -F 
The block-vfs plugin is no longer a special file. So we have to
explicitly discharge the sanity check of the mkfs tool.

Issue #3507
 2019-11-19 14:54:14 +01:00
Norman Feske
18b621c2fe sculpt: version 19.11 2019-11-19 14:54:14 +01:00
Alexander Boettcher
cb61a28362 vbox5: use with_libc when call-in into vbox code 
required due to pthreads becoming/are an integral part of the libc now.
Commit avoids libc error:

Error: libc suspend() called from non-user context (0xaf5c784) - aborting

Issue #3550
 2019-11-19 14:54:14 +01:00
Norman Feske
4d7d208940 test-libc_vfs_fs_chained: fix XML validity error 
This problem was spotted by the test.run script.
 2019-11-19 14:54:14 +01:00
Sebastian Sumpf
f7509a5b78 libports: libc: extract file name from path in dlopen 
'dlopen' causes the ldso to open ROM connections, right now we only
support single file names for these ROM not paths. Therefore, we extract
the file name from path within libc's 'dlopen'

fixes #3551
 2019-11-19 14:54:14 +01:00
Christian Helmuth
a54c04d247 libc: return EPIPE on send to shut down sockets 
This intermediate fix still lacks the generation of SIGPIPE and return
of EINTR if the MSG_NOSIGNAL flag is not set in the call to send().
 2019-11-19 14:54:14 +01:00
Christian Helmuth
e70c04ef86 lwip: shut down TCP sockets return WRITE_ERR_IO on write 2019-11-19 14:54:13 +01:00
Christian Helmuth
6410e88698 netperf: handle EPIPE / silence warnings 
This is an intermediate fix to handle sends to sockets shut down by the
peer already. A final fix in the libc would add support for the
MSG_NOSIGNAL send flag, generate SIGPIPE, and return the expected EINTR.
Then this patch is no longer required.

This commit also drops the obsolete patch for missing
sysconf(_SC_NPROCESSORS_ONLN), which is now part of libc.
 2019-11-19 14:54:13 +01:00
Christian Helmuth
a3cb9d9897 lwip: prevent pointer-compare compilation warnings 2019-11-19 14:54:13 +01:00
Alexander Boettcher
b2a7ac2996 depot_deploy: support cpu quota configuration 
the same as the ram and cap quota
 2019-11-19 14:54:13 +01:00
Norman Feske
5f350adb57 vfs/lxip: don't misinterpret EINPROGRESS as error 2019-11-19 14:54:13 +01:00
Norman Feske
1485cd9d24 noux: don't update mtimes for read-only files 
The VFS server as reworked in issue #3547 responds to such an attempt
with a failure as acknowledgement, triggering the client-side error
"Error: packet operation=2 failed" at noux.
 2019-11-19 14:54:13 +01:00
Norman Feske
65d72fb07a drivers/gpu/intel: upgrade platform session on demand 2019-11-19 14:54:13 +01:00
Norman Feske
4871c7bba0 drivers/gpu/intel: reflect insuff. quota to client 
This patch prevents the driver from translating Insufficient_ram_quota
and Insufficient_cap_quota exceptions to Service_denied exceptions at
session-creation time. This enables the client to respond to such a
condition by retrying the session creation with increased session quota.
 2019-11-19 14:54:13 +01:00
Norman Feske
97e2968986 pkg/test-tcp_bulk_lwip: remove lwip.lib.so 
The lwip library is not shared object.

This change is needed to execute the test with the test.run script.
 2019-11-19 14:54:13 +01:00
Norman Feske
eb9a9bf23d bash.run: support using pipes 2019-11-19 14:54:13 +01:00
Emery Hemingway
c51b4b5742 New VFS plugin for emulating POSIX pipes 
Add a new plugin for creating pipes between pairs of VFS handles. It is
intended to replace the libc_pipe plugin, one of the last remaining libc
plugins.

In contrast to the libc_pipe plugin, this plugin defers cross-handle
notification until I/O signal handling rather than block and unblock
readers using a semaphore. This is a performance regression in the case
of multiple threads blocking on a pipe, but shall be an intermediate
mechanism pending renovations within the libc VFS and threading layers.
As a side effect, threads blocked on a pipe might not be resumed until
the main thread suspends and dispatches I/O signals.

The "test-libc_pipe" test has been adjusted to use the VFS pipe plugin
and tests both local pipes and pipes hosted remotely in the VFS server.

Merge adaptations (such as EOF handling, adjustment to VFS/libc
interface changes) by Norman Feske.

Fix #2303
 2019-11-19 14:54:13 +01:00
Norman Feske
9b7915facb vfs server: redesigned internal control flow 
This patch reworks the VFS server to become easier to understand.

- The packet-stream handling is hidden from the node types.

- Introduced the notion of a "job" as an abstraction from the raw packet
  stream.

- The processing of requests is split into three phases:
  (1) accepting new jobs,
  (2) the execution of jobs, and
  (3) the delivery of acknowledgements.

- There is no longer a global fifo of pending nodes. Instead, each
  session has a fifo of "active" nodes, which are nodes that wait
  for the completion of jobs.

- The Io_node type no longer uses virtual functions as hooks for
  the derrived classes. Instead, it provides a number of utility
  functions.

- The control flow of the 'Io_progress_handler' is now the same as
  for the session-local packet-stream handler.

- Removed dependency from 'file_system/node.h', which is a relic
  from the ram_fs.

While reworking the code, the following semantic changes were made
additionally.

- The handling of SEEK_TAIL no longer overwrites the file content at the
  end of the file.

- Removed diagnostic message at session-creation time.

- Write loop for continuous files

Issue #3547
Issue #2303
 2019-11-19 14:54:13 +01:00
Norman Feske
f0de187bbb libc: increase default quota of forked processes 
This patch reduces the debug noise for the prominent case of executing
bash with coreutils. Without it, the forked process will always ask for
more RAM immediately when starting up.
 2019-11-19 14:54:13 +01:00
Norman Feske
539110c0b1 base: make 'Fifo::Element::enqueued' const 
Issue #3547
 2019-11-19 14:54:13 +01:00
Norman Feske
d7b1a89087 file-system session: use same dirent size as VFS 
This change allows for the conversion of directory entries in place
(i.e., in the VFS server) and anticipates the potential future merge of
both types into one.

Issue #3547
 2019-11-19 14:54:13 +01:00
Christian Helmuth
c50252fb35 libc: suspend/resume in pthread mutex lock/unlock 
Issue #3550
 2019-11-19 14:54:13 +01:00
Christian Helmuth
54002e6e6b test-pthread: refactor and cleanup test cases 
Issue #3550
 2019-11-19 14:54:13 +01:00
Sebastian Sumpf
16994d637b libports: update gcov port 
removes bogus include from generated file libgcc_tm.h

issue #3537
 2019-11-19 14:54:12 +01:00
Martin Stein
b541a0d448 smbios.h: move to os/include/smbios 
Fixes #3489
 2019-11-19 14:54:12 +01:00
Norman Feske
d4e0d2f578 libc: defer clone ack after FD initialization 
Issue #3478
Issue #2303
 2019-11-19 14:54:12 +01:00
Norman Feske
fab2fc874f libc: write loop for continuous files 
This patch improves the libc's write operation to iterate on partial
writes to continuous files until the original write count is reached.
The split of large write operations into small partial writes as
dictated by the VFS infrastructure (e.g., constained by I/O buffer
sizes) becomes invisible to the libc-using application.

Issue #3507
Issue #2303
 2019-11-19 14:54:04 +01:00
Sebastian Sumpf
54643d6878 base: add entry to .dynanmic section in genode_dyn.ld 
Make sure that at least entry is present in the .dynamic section, so the
PT_DYNAMIC segment points to something valid in case there are no
dynamic symbols.

issue #3537
 2019-11-19 14:45:40 +01:00
Sebastian Sumpf
2653fad0c4 ports: noux_tool_chain_auto for ARMv8 
issue #3537
 2019-11-19 14:45:40 +01:00
Sebastian Sumpf
b752d22a77 libports: checkout ARMv8 version of gcov 
issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
ad12b42d1c gems: depot autopilot enable gcov for ARMv8 
issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
c845a2d943 libprots: libgcov ARMv8 support 
issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
b7e06a0b5b recipe: gmp for ARMv8 
issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
417dd59b22 libports: gcov for ARMv8 
issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
3024720656 ports: update make to version 4.2.1 
Required for ARMv8 support

issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
f23eab735b ports: binutils target for ARMv8 
issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
112c32eb54 ports: noux-pkg/gcc_arm_64 
Added ARMv8 as a native gcc target. This requires a rebuild of the tool
chain.

issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
3103ce1fa8 libports: update mpc to 1.1.0 
issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
db18fc42fe libports: update mpfr to 4.0.2 
issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
d28fe9e938 libports: enable gmp 6.1.2 for ARMv8 
issue #3537
 2019-11-19 14:45:39 +01:00
Sebastian Sumpf
31a035a907 libports: update gmp from 4.3.2 to 6.1.2 
issue #3537
 2019-11-19 14:45:39 +01:00
Norman Feske
91412c6c52 libc: trigger SIGCHLD for when forked child exits 2019-11-19 14:45:39 +01:00
Norman Feske
068324536c libc: eliminate atexit handler for 'Rtc' 
We never want to destruct the 'Rtc' because other atexit handlers may
depend on it (e.g., for updating modification times when closing files).
 2019-11-19 14:45:38 +01:00
Norman Feske
bb6eb0f6ea libc: local signal delivery via 'kill' 
This patch adds the ability to call 'kill' with the own PID to trigger
the execution of the handler of the specified POSIX signal. This is used
by 'bash', e.g., when cancelling the input of a command via control-c.

Related to issue #3546
 2019-11-19 14:45:38 +01:00
Norman Feske
c1012e6a45 libc: translate terminal user interrupts to SIGINT 
Issue #3546
 2019-11-19 14:45:38 +01:00
Norman Feske
23d21d77e9 vfs/terminal: detect user interrupts (control-c) 
This patch equips the VFS terminal file system with the ability to
detect user interrupts in the incoming data stream, and propagate this
information via the new pseudo file '.terminal/interrupts'. Each time,
the user presses control-c in the terminal, the value stored in this
pseudo file is increased. Thereby, a VFS client can watch this file to
get notified about the occurrences of user interrupts.

Fixes #3546
 2019-11-19 14:45:38 +01:00
Norman Feske
8c44b17e86 libc: shebang handling for execve 
Fixes #3545
 2019-11-19 14:45:38 +01:00
Norman Feske
636e0f6444 libc: cancel select when POSIX signal occurs 
With this patch, Vim running via the 'bash.run' script becomes able to
adopt itself to changed window dimensions.

Issue #3544
 2019-11-19 14:45:36 +01:00
Josef Söntgen
fafa409cf9 block_tester: use dynamic shared scratch buffer 
The scratch buffer is now allocated from the heap and is shared
between the test as they are executed in a serial fashion. This
change saves memory as the test are constructed at start-up.

Fixes #3539.
 2019-11-19 14:44:02 +01:00
Josef Söntgen
dbecceec09 expanding_reporter: configure initial buffer size 
Fixes #3543.
 2019-11-19 14:44:02 +01:00
Josef Söntgen
60f390ddf8 noux: prevent accessing invalid leaf path in open 
Fixes #3542.
 2019-11-19 14:44:02 +01:00
Josef Söntgen
c79ebc93a2 libc_noux: try read again on EAGAIN/EINTR 
Issue #3541.
 2019-11-19 14:44:02 +01:00
Josef Söntgen
55ab694d79 libc_noux: leave write loop on error 
Fixes #3541.
 2019-11-19 14:44:02 +01:00
Josef Söntgen
f5c5479faa ram_block: use Number_of_bytes to parse block size 
Using this helper rather than parsing as size_t directly allows use to
use size suffixes, e.g. K, M, G.

Fixes #3540.
 2019-11-19 14:44:02 +01:00
Josef Söntgen
ba9b612c4f libc_noux: close fd on failed O_TRUNC attempt 
Fixes #3524.
 2019-11-19 14:44:02 +01:00
Norman Feske
7b0771659e libc: trigger SIGWINCH by watching .terminal/info 
Issue #3544
 2019-11-19 14:43:43 +01:00
Norman Feske
e9762ee25f vfs: 'Rtc_file_system::close' for watch handle 
The default implementation of this method throws an exception.
 2019-11-19 14:42:23 +01:00
Norman Feske
7549189f88 vfs: watch support for Readonly_value_file_system 
This patch adds support for watch notifications for the
'Readonly_value_file_system', which is often used by VFS plugins to
implement pseudo files. It thereby enables VFS clients to respond to
VFS-plugin events (think of terminal resize) dynamically.

Fixes #3523
 2019-11-19 14:42:23 +01:00
Oleg Girko
a5bc031cca genode_org: Fix download URL. 
As genode.org site is now protected by TLS and unencrypted site
is redirected to enctypted one, download URL should be modified
to reflect this change.

Without this change, wget downloads just index.html file.

Signed-off-by: Oleg Girko <ol@infoserver.lv>

Fixes #3538
 2019-11-19 14:42:23 +01:00
Oleg Girko
5cd684997a sntp_client: No need to always limit this scenario to x86. 
It runs pretty well on Raspberry Pi, for example.

Leaving this scenario limited to x86 for autopilot runs only.

Signed-off-by: Oleg Girko <ol@infoserver.lv>

Fixes #3536
 2019-11-19 14:42:23 +01:00
Christian Helmuth
93d3a0848a Remove unused libm/posix.lib.so from lwip_lx.run 
Prevents integration failures like follows if the libraries were not
built already in the build directory.

  Missing boot modules: libm.lib.so posix.lib.so
  Makefile:323: recipe for target 'run/lwip_lx' failed
 2019-11-19 14:42:23 +01:00
Christian Helmuth
f1b1dd26cf run: tweak CRLF translation in log scripts 
Ideally, I'd like to disable expects's translation here, which does not
work. The current setting is the result of testing all combinations.
 2019-11-19 14:42:23 +01:00
Christian Helmuth
cd5e906bd0 imx: do not convert LF to CRLF in UART driver 
This is done on higher level already.
 2019-11-19 14:42:23 +01:00
Oleg Girko
c589660182 Replace ENV make variable with MKENV. 
ENV variable conflicts with Environment Modules.

Signed-off-by: Oleg Girko <ol@infoserver.lv>

Fixes #3535
 2019-11-19 14:42:23 +01:00
Stefan Kalkowski
8a8aa85726 hw: initialize iomux, ccm and gpc for i.MX8 EVK 
Fix #3534
 2019-11-19 14:42:23 +01:00
Stefan Kalkowski
105b2c9b7a hw: fix gicv3 implementation of clear/set regs 
In addition use uniformly enums for interupt count in register declarations.

Fix #3532
 2019-11-19 14:42:23 +01:00
Stefan Kalkowski
f6435d91fc hw: turn Kernel_object into Genode::Constructible 
Fix #3531
 2019-11-19 14:42:23 +01:00
Stefan Kalkowski
3e3fb63863 hw: enable Genode::raw for bootstrap 
Ref #3531
 2019-11-19 14:42:23 +01:00
Oleg Girko
4007cee852 Fix MAC address printing in dde_linux USB net drivers. 
Passing array of unsigned chars to Genode::log() function
makes it converted to void pointer, resulting in printing its address.

Wrapping this array into Genode::Cstring solves this problem
and makes it being printed properly as zero-terminaled string.

Signed-off-by: Oleg Girko <ol@infoserver.lv>

Fixes #3530
 2019-11-19 14:42:23 +01:00
Christian Helmuth
b622a5a788 vbox: use RTThreadCreate for periodic_gip thread 
This prevents the following error message.

  [init -> vbox1] Error: pthread_self() called from alien thread named 'periodic_gip'
 2019-11-19 14:42:23 +01:00
Christian Helmuth
ec9e40695d vbox: use PIIX3 instead of ICH9 in VM config 
Prevents #3533
 2019-11-19 14:42:23 +01:00
Martin Stein
0ad0153626 nic_router: fix broken uplink label on re-config 
Uplink labels were stored in the uplink objects as reference. As uplink objects
may be re-used on re-configuration to avoid unnecessary closing and re-opening
of NIC sessions, the label reference could get invalid as the object that
stored the label content didn't get re-used. Thus, the label is now copied to
the uplink object.

Fixes #3529
 2019-11-19 14:42:23 +01:00
Martin Stein
cd37bff514 nic_router: do not report while re-configuring 
Fixes #3528
 2019-11-19 14:42:23 +01:00
Martin Stein
27c2a66bbd nic_router: report link state 
This adds two new boolean attributes to the <report> tag of the NIC router
configuration 'link_state' and 'link_state_triggers'. The former decides
whether to report the link state of each NIC interface (downlink, uplinks) at
the NIC router. The other decides whether to trigger reporting each time the
link state of an interface changes.

Fixes #3527
 2019-11-19 14:42:23 +01:00
Martin Stein
58247737fd nic_router: free MAC on session-creation exception 
If a MAC address was once allocated for a downlink and during the further
creation of the downlink an exception caused the creation to be aborted, the
NIC router didn't free the MAC address again.

Ref #3525
 2019-11-19 14:42:23 +01:00
Martin Stein
ebcca179ed nic_router: fix 'catch' paths in session creation 
The handling of exceptions during session creation implies freeing and
detaching the RAM DS behind the session env and session component. But this
was done using the session env located in the same RAM DS, which must
lead to problems. Now, the session env from the RAM DS is copied to the stack
and the operations are done on this temporary instance instead.

Ref #3525
 2019-11-19 14:42:22 +01:00
Martin Stein
60d37f690c packet_stream_*: fix missing dissolve on exception 
When the construction of a member of Packet_stream_*::Rpc_object after
the _cap member threw an exception, the object was not dissolved from
the entrypoint although the Rpc_object vanished at this point. This was
because the call to 'manage()' happened in the initializer list (for the
_cap member instantiation). The destruction of the _cap member then did
not dissolve the object.

This first fix moves the call to 'manage()' into the constructor body
after the instantiation of all other members. A more sophisticated fix
would use some kind of 'Managed_object' life-time guard that manages an
object on construction and dissolves on destruction.

Ref #3525
 2019-11-19 14:42:22 +01:00
Martin Stein
0ed5655086 nic_router: do DHCP when re-config to dynamic IP 
When re-configuring a domain from using a static IP config to using a dynamic
IP config, attached interfaces didn't react with doing DHCP discover. This bug
is fixed now and the nic_router_flood test was adapted in a way that it tests
the described behavior.

Fix #3518
 2019-11-19 14:42:22 +01:00
Stefan Kalkowski
87a6368ba1 hw: implement multi-processor support for rpi3 
Fix #3522
 2019-11-19 14:42:22 +01:00
Stefan Kalkowski
1cbd77c806 hw: implement multi-processor support for i.MX8 
Fix #3520
 2019-11-19 14:42:22 +01:00
Stefan Kalkowski
e3f82b09d7 hw: instantiate pic object per cpu 
Ref #3520
 2019-11-19 14:42:22 +01:00
Christian Helmuth
d4a3db22bd solo5: migrate snapshot archive to genode.org 2019-11-19 14:42:22 +01:00
Christian Helmuth
43f28e0451 stdcxx: add more missing symbols to ABI 
Fixes #3506
 2019-11-19 14:42:22 +01:00
Norman Feske
d516515c7a vfs/terminal: provide meta info as pseudo files 
Issue #3519
Fixes #3521
 2019-11-19 14:39:09 +01:00
Norman Feske
7ac32ea60c libc: support for ioctls via ioctl directory 
This patch introduces a new scheme of handling ioctl operations that
maps ioctls to pseudo-file accesses, similar to how the libc maps socket
calls to socket-fs operations.

A device file can be accompanied with a (hidden) directory that is named
after the device file and hosts pseudo files for triggering the various
device operations. For example, for accessing a terminal, the directory
structure looks like this:

  /dev/terminal
  /dev/.terminal/info

The 'info' file contains device information in XML format. The type of
the XML node corresponds to the device type. E.g., If the libc receives
a 'TIOCGWINSZ' ioctl for /dev/terminal, it reads the content of
/dev/.terminal/info to obtain the terminal-size information. In this
case, the 'info' file looks as follows:

  <terminal rows="25" columns="80/>

Following this scheme, VFS plugins can support ioctl operations by
providing an ioctl directory in addition to the actual device file.

Internally, the mechanism uses the 'os/vfs.h' API to access pseudo
files. Hence, we need to propagate the Vfs::Env to 'vfs_plugin.cc' to
create an instance of a 'Directory' for the root for the VFS.

Issue #3519
 2019-11-19 14:39:09 +01:00
Christian Prochaska
07a40d028a libc: fix and cleanup pthread mutexes 
Issue #3503
Fixes #3504
 2019-11-19 14:23:57 +01:00
Christian Prochaska
a47adecdcd test-pthread: add mutex tests 
Add tests for recursive and errorcheck type mutexes.

Fixes #3503
 2019-11-19 14:23:57 +01:00
Christian Helmuth
355d94f5df Use NTP server from pool.ntp.org in sntp_client test 2019-11-19 14:23:57 +01:00
Pirmin Duss
c0789a6c0e depot_deploy: status report 
The idea is, that other components may know, when a valid
init.config is available.

Issue #3482
 2019-11-19 14:23:57 +01:00
Norman Feske
530144b040 doc/challenges.txt: add Puppet topic 2019-11-19 14:23:56 +01:00
Norman Feske
c85bc38802 Move include/gems/vfs.h to include/os/vfs.h 
Fixes #3515
 2019-11-19 14:23:56 +01:00
Norman Feske
a8dd7dd2fa Remove rump_fs server component 
This patch replaces the use of the rump_fs server by the VFS server with
the Rump VFS plugin.

Issue #3512
 2019-11-19 14:23:56 +01:00
Norman Feske
6e86d6d699 Remove server/fatfs_fs and libc_fatfs plugin 
Issue #3512
 2019-11-19 14:23:56 +01:00
Norman Feske
2954abb58a Simplify qt5_textedit.run 
The run script used to be the only user of the fatfs_fs server, which
we're going to remove. This patch removes the components for accessing a
real storage device and file system from the run script. The new version
just uses ram_fs.

Issue #3512
 2019-11-19 14:23:56 +01:00
Norman Feske
5bb366513b Remove os/src/app/cli_monitor 
Nowadays, we use standard command-line tools like vim to edit init
configurations dynamically, which alleviates the need for a custom CLI.
The CLI-monitor component was too limited for use cases like Sculpt
anyway.

The patch also removes the ancient (and untested for long time)
terminal_mux.run script, which used to be the only remaining user of the
CLI monitor.

Issue #3512
 2019-11-19 14:23:56 +01:00
Norman Feske
4bcc75365c Remove gems/src/app/launcher 
The graphical launcher remained unused for a few years now. It is not
suitable for systems as flexible as Sculpt OS.

Issue #3512
 2019-11-19 14:23:56 +01:00
Emery Hemingway
288f79270d test/vfs_stress: annotate switch label fall-throughs 
This test was written for speed, not correctness.

Fixes #3516
 2019-11-19 14:23:56 +01:00
Christian Prochaska
3c62a33a25 qt5: add NEON-related files and symbols to qt5_gui lib 
Fixes #3517
 2019-11-19 14:23:56 +01:00
Piotr Tworek
80a84bef26 Fix ram session quota accounting in lan9118 
This driver manually checks if the RAM quota is big enough for the
Lan9118 nic session component. The problem is Root_component::_create
from which Root_component::_create_session gets called does already
check this. No need to account for it twice.

Fixes #3514
 2019-11-19 14:23:56 +01:00
Johannes Kliemann
022fac0d37 dde_zircon: use fork by jklmnn 
Fixes #3513
 2019-11-19 14:23:56 +01:00
Norman Feske
5ab1505d43 file system: enhanced file status info 
This patch extends the 'File_system::Status',
'File_system::Directory_entry', and the related 'Vfs' types with
the following additional information:

- Distinction between continuous and transactional files (Node_type)
  (issue #3507)
- Readable, writeable, and executable attributes (Node_rwx),
  replacing the former 'mode' bits
  (issue #3030)

The types 'Node_rwx', 'Node_type' are defined twice,
once for the VFS (vfs/types.h) and once for the 'File_system'
session (file_system_session/file_system_session.h).
Similarly, there is a direct correspondance between
'Vfs::Directory_service::Dirent' and 'File_system::Directory_entry'.

This duplication of types follows the existing pattern of keeping the
VFS and file-system session independent from each other.
 2019-11-19 14:23:56 +01:00
Alexander Boettcher
1297a8fb57 trace: skip subject_id 0 in subject id allocator 
Issue #3510
 2019-11-19 14:23:55 +01:00
Christian Helmuth
732215a83f libc: limit fcntl(F_SETFL) to file-status flags 
This fixes unintended (and unpermitted) changes of O_ACCMODE bits.
 2019-11-19 14:23:55 +01:00
Christian Helmuth
e11addec7d Cleanup vfs dev directory in vbox5_genode_usb_hid 2019-11-19 14:23:55 +01:00
Christian Helmuth
2166a4b17f dde_linux: prevent warning about 'register' keyword 2019-11-19 14:23:55 +01:00
Christian Helmuth
291587f545 libc: honor poll() event flags POLLRDNORM etc. 
Fixes empty read file-descriptor sets in fetchurl_lwip/lxip.

Issue #3499
 2019-11-19 14:23:55 +01:00
Christian Helmuth
97df705e53 dde_rump: eliminate global ctors 2019-11-19 14:23:55 +01:00
Josef Söntgen
e281174dae libc: bring back old 'poll()' emulation 
Issue #3499
 2019-11-19 14:23:55 +01:00
Alexander Boettcher
ed73feddc5 vbox5: react on host size changes immediately 
The guest may take some time to adjust nevertheless.

Fixes #3498
 2019-11-19 14:23:55 +01:00
Josef Söntgen
c5706e8f4a wifi_drv: enable loading of 5000 series FW 
Fixes #3502.
 2019-11-19 14:23:55 +01:00
Christian Helmuth
1782c6be79 qt5: rework keyboard handling 
Fixes #3483
 2019-11-19 14:23:54 +01:00
Christian Helmuth
cccfd0719d qt5: prevent deprecated warnings 
Issue #3483
 2019-11-19 14:23:54 +01:00
Christian Helmuth
edc9545229 input: move control-modifier handling into clients 
Moving the handling into the input-session clients enables more
sophisticated implementations (like Qt5) to apply key-symbol based
handling of those modifiers like correct CTRL-A with QWERTY and AZERTY
layouts and distinction of CTRL-J and Return.

Issue #3483
 2019-11-19 14:23:54 +01:00
Christian Helmuth
cc611834c9 input: repeat events require valid codepoints 
Issue #3483
 2019-11-19 14:23:54 +01:00
Christian Helmuth
bbd27a54d3 input_filter: fix de_de/fr_fr character generators 
Issue #3483
 2019-11-19 14:23:54 +01:00
Christian Helmuth
5a06751242 Increase input_filter RAM quotas to 1280K 
Issue #3483
 2019-11-19 14:23:54 +01:00
Christian Helmuth
6df8b44616 platform/x86: support 64-bit base address registers 
The API still exports 32-bit address and size values only, which works
as the actual MMIO resources are allocated in platform_drv internally.

Fixes #3494
 2019-11-19 14:23:54 +01:00
Christian Helmuth
e0af9c2d8b platform_drv.inc: reduce acpi_drv caps to 350 
This value equals the configuration in drivers_managed-pc, i.e., Sculpt.
 2019-11-19 14:23:54 +01:00
Christian Helmuth
11a7ac0536 acpi_drv: reduce capability quota requirements 
Map ACPI tables entirely as one dataspaces (not page-by-page). Note, the
current approach does only merge overlapping but not consecutive I/O
memory regions, which would reduce the amount of capabilities required
even further.

Fixes #3495
 2019-11-19 14:23:54 +01:00
Christian Prochaska
5c25e0bdb0 heap: fix typo in alloc(0) diagnostic message 
Fixes #3501
 2019-11-19 14:19:34 +01:00
Norman Feske
222f214341 extract.run: suppress rtc warnings 2019-11-19 14:19:34 +01:00
Norman Feske
eefe91ee41 depot_download_manager: no mtime update in extract 
With the added modification-time support in the libc, the extract tool
requires a timer session, which is not plausible for the purpose of the
program.

This behavior stems from the fact that the libc implicitly writes the
mtime when closing a written file. For this update, it implicitly calls
'clock_gettime', which in turn initializes the timer subsystem within
the libc (creating a timer session).

For the extract tool, the implicitly updated mtime is useless because
the extract tool overwrites this modification time with the mtime stored
in the archive anyway. However, the dependency from a timer service
remains.

This patch explicitly disables the libc's implicit updating of the
file-modification when closing a written file.

Issue #1784
 2019-11-19 14:19:34 +01:00
Norman Feske
979d823d85 libc: make mtime update configurable 
By specifying <libc update_mtime="no"...>, the modification-time update
on VFS-sync operations (as issued whenever a written file is closed)
can explicitly be disabled.

Issue #1784
 2019-11-19 14:19:34 +01:00
Christian Helmuth
76438a3f85 dde_linux: refactor usb arch-dependent includes 
Also added arm_64/arm_v8, which fixes building board-independent usb_hid
etc.
 2019-11-19 14:19:34 +01:00
Christian Helmuth
b0271ae5e1 Fix hard-disk image creation in vfs server tests 
The hard-disk image must be created before boot-image creation as it is
used as boot module by variants of the test (use_ram_block=true).
 2019-11-19 14:17:30 +01:00
Norman Feske
6a063364da fs_rom: adaptation to added file-modification time 
Issue #1784
 2019-11-19 14:17:30 +01:00
Josef Söntgen
99b632f86c server/vfs: add modification time 
Issue #1784.
 2019-11-19 14:17:30 +01:00
Josef Söntgen
180f9e6384 libc: handle modified files 
Issue #1784.
 2019-11-19 14:17:30 +01:00
Josef Söntgen
94b63924ed noux: add time modification 
... and set initial time by using RTC session.

Up to now Noux used a monotic clock whose initial start value always
was '0' (which correlates to the start of the UNIX epoch) to provide
a notion of time. In addition it is now possible to use the RTC
session to set the initial value for use cases where having a proper
real-world time matters.

To use the RTC session the 'rtc' attribute of the '<config>' node
must be set to 'yes'. Thereby the session becomes a mandatory
dependency as Noux will not start without it.

Issue #1784
 2019-11-19 14:17:30 +01:00
Josef Söntgen
d0bf6d2b52 libc: add modification time 
Issue #1784.
 2019-11-19 14:17:30 +01:00
Josef Söntgen
9a82bbb54d vfs: add modification time member to directory_service 
Issue #1784.
 2019-11-19 14:17:30 +01:00
Josef Söntgen
400039e1b6 rump_fs: add modification time 
Issue #1784.
 2019-11-19 14:17:30 +01:00
Josef Söntgen
2e5166efd7 lx_fs: add modification time 
Issue #1784.
 2019-11-19 14:17:30 +01:00
Josef Söntgen
2ec3aaf639 file_system_session: add modification time 
* add modification_time member
* add WRITE_TIMESTAMP packet
 2019-11-19 14:17:30 +01:00
Norman Feske
ab5187d673 libc: resolve symlinks in execve 
This patch complements the commit "libc: execve" with the ability to
execute files stored at arbitrary sub directories of the file system.

Issue #3481
Issue #3500
 2019-11-19 14:17:29 +01:00
Norman Feske
697d496093 ldso: don't strip leading path from ROM name 
This patch removes ldso's builtin policy of removing any path elements
prepending the ROM module name. Instead, the ROM name is used as is.
This clears the way to access different ROM modules that share the same
name but are stored at different directories behind an fs_rom (e.g.,
/bin/bash vs. /usr/local/bin/bash).

Issue #3500
 2019-11-19 14:17:29 +01:00
Norman Feske
a17c5e30b7 mk/gnu_build.mk: avoid absolute lib paths 
This patch changes the way how libraries are specified at the linker
command line from /abs/path/to/library.lib.so

Issue #3500
 2019-11-19 14:17:29 +01:00
Christian Helmuth
8d6285927b tools.conf: fix check for arm_64 
Before arm matched first resulting in the wrong tool-chain binaries
being used.
 2019-11-19 14:17:29 +01:00
Norman Feske
90a91f3536 dde_linux: eliminate global ctors in lx_kit 
As the lx_kit library is used by the vfs_lxip plugin, it must not
contain any static global constructor.

Related to issue #3487
 2019-11-19 14:17:29 +01:00
Norman Feske
c8b7710e5d libc: improve dup/dup2 in vfs_plugin 
This patch replaces the naive dup2 implementation (that merely
duplicated the context pointer) by the replication of the original
FD state by re-opening the same file with the same flags and seek
position. This prevents a potential double release of the VFS handle
(the FD context). It also implements 'dup'.

Fixes #3505
Fixes #3477
 2019-11-19 14:10:55 +01:00
Norman Feske
59c60b8031 test-libc: test double dup2, dup 
Issue #3477
Issue #3505
 2019-11-19 14:10:55 +01:00
Norman Feske
9500e8b6e1 lwip: remove use of global constructor 
Related to issue #3487
 2019-11-19 14:10:55 +01:00
Norman Feske
e0ee56275e vfs_lwip: fix switch fall-through 
Pointed out by a compiler warning.
 2019-11-19 14:10:55 +01:00
Norman Feske
418ac4c560 libc: remove global watch() function 
This patch replaces the function with a 'Watch' interface to be
explicitly passed to the caller (currently only time.cc).

Issue #3497
 2019-11-19 14:10:55 +01:00
Norman Feske
5f5d709c07 libc: remove global libc_config() 
Issue #3497
 2019-11-19 14:10:55 +01:00
Norman Feske
89a38723bd libc: reimplement passwd handling 
- Eliminate call of global libc_config()
- Remove dynamic memory allocation, const cast
- Prepare for moving the state from compilation unit to header
- Fix run/libc_getpwent.run

Issue #3497
 2019-11-19 14:10:55 +01:00
Norman Feske
648bcd1505 libc: unify use of namespaces 
This patch unifies the patterns of using the 'Genode' and 'Libc'
namespaces.

Types defined in the 'internal/' headers reside in the 'Libc'
namespace. The code in the headers does not need to use the
'Libc::' prefix.

Compilation units import the 'Libc' namespace after the definition of
local types. Local types reside in the 'Libc' namespace (and should
eventually move to an 'internal/' header).

Since the 'Libc' namespace imports the 'Genode' namespace, there is
no need to use the 'Genode::' prefix. Consequently, code in the
compilation units rarely need to qualify the 'Genode' or 'Libc'
namespaces.

There are a few cases where the 'Libc', the 'Genode', and the global
(libc) namespaces are ambigious. In these cases, an explicit
clarification is needed:

- 'Genode::Allocator' differs from 'Libc::Allocator'.
- 'Genode::Env' differs from 'Libc::Env'.
- Genode's string functions (strcmp, memcpy, strcpy) conflict
  with the names of the (global) libc functions.
- There exist both 'Genode::uint64_t' and the libc'c 'uint64_t'.

Issue #3497
 2019-11-19 14:10:55 +01:00
Norman Feske
bf92232698 libc: split task.cc into multiple files 
This patch is the first step of re-organizing the internal structure of
the libc. The original version involved many direct calls of global
functions (often with side effects) across compilation units, which
made the control flow (e.g., the initialization sequence) hard to
follow.

The new version replaces those ad-hoc interactions with dedicated
interfaces (like suspend.h, resume.h, select.h, current_time.h). The
underlying facilities are provided by the central Libc::Kernel and
selectively propagated to the various compilation units. The latter is
done by a sequence of 'init_*' calls, which eventually will be replaced
by constructor calls.

The addition of new headers increases the chance for name clashes with
existing (public) headers. To disambiguate libc-internal header files
from public headers, this patch moves the former into a new 'internal/'
subdirectory. This makes the include directives easier to follow and the
libc's source-tree structure more tidy.

There are still a few legacies left, which cannot easily be removed
right now (e.g., because noux relies on them). However, the patch moves
those bad apples to legacy.h and legacy.cc, which highlights the
deprecation of those functions.

Issue #3497
 2019-11-19 14:10:55 +01:00
Norman Feske
aec1178ab1 libc: simplify initialization sequence 
This patch reverts the interim patch "libc: init malloc_heap before
VFS", which is no longer needed with the solved issue #3496 and
issue #3487.
 2019-09-20 14:16:07 +02:00
Norman Feske
efe7f5172d ldso: prevent premature execution of ctors 
Fixes #3487
 2019-09-20 14:16:05 +02:00
Norman Feske
0aedabd245 libc: remove global constructors 
- readv_writev: move 'rw_lock' instance into a function scope,
  constructing the instance on the first access.
- select: move 'select_cb_list' instance into function scope.
- thread: move 'key_list_lock' and 'keys' into function scope.
- rwlock, semaphore, socket_fs_plugin, thread, thread_create:
  instantiate 'Libc::Allocator' per use, alleviating the need for a
  global instance.

Issue #3496
 2019-09-20 14:16:01 +02:00
Christian Helmuth
4a7b0e99a6 depot: update recipe hashes 2019-09-20 14:14:16 +02:00
Norman Feske
614f7fa56a mesa.run: omit depot content from boot modules 2019-09-17 14:38:07 +02:00
Norman Feske
6d230134cb doc: update challenges 2019-09-13 12:14:19 +02:00
Pirmin Duss
d953030c0e lx_fs: respect writeable session argument 
Fixes #3492
 2019-09-12 12:28:41 +02:00
Alexander Boettcher
6fe80c3cc7 sculpt: add lock screen/GUI support 
Fixes #3491
 2019-09-09 15:25:45 +02:00
Christian Prochaska
afa0e26a6a qt5: update virtual keyboard example to Qt 5.13 
Fixes #3490
 2019-09-09 15:08:13 +02:00
Christian Prochaska
1d3bbde70a qt5: add 'lupdate' and 'lrelease' to host tools 
Fixes #1736
 2019-09-09 15:05:50 +02:00
Alexander Boettcher
6f0c6501f2 acpica: catch Io_mem exceptions 
Issue #2998
 2019-09-09 15:04:53 +02:00
Christian Helmuth
e448022f23 Fix hard-disk image creation in libc_fatfs 
The hard-disk image must be created before boot-image creation as it is
used as boot module by variants of the test (use_ram_block=true).
 2019-09-09 14:58:25 +02:00
Christian Helmuth
664d858e63 Fix sntp_client test for hw/muen 2019-09-09 12:15:07 +02:00
Christian Helmuth
ede009edf9 Fix expected output in USB HID tests 
Invalid characters in press events are no longer 0 but 0xfffe since

  a97b8043b5 utf8: non-character U+fffe as invalid codepoint

Issue #3483
 2019-09-09 11:35:42 +02:00
Christian Helmuth
08aa7d310a Increase stack size in thread test for arm_v8a 
It seems throwing an exception on arm_v8a needs a significant amount of
stack space (> 2048 bytes).
 2019-09-03 12:12:46 +02:00
Christian Helmuth
19bf0fdeb8 xkb2ifcfg: detect and log initialization errors 2019-09-03 09:00:31 +02:00
Emery Hemingway
36111a2edf Libc: implement getifaddrs 
Implement getifaddrs and freeifaddrs within the libc using socket
control files at the VFS. Add an "address" and "netmask" file to the
lwIP plugin.

Only a single IPv4 address is initially supported, and the broadcast
address returned will never be valid.

Fixes #3439
 2019-09-02 16:36:07 +02:00
Christian Prochaska
ab017607a2 tool chain: TLS support 
Fixes #3418
 2019-09-02 16:29:33 +02:00
Christian Helmuth
2c47cd5c94 version: 19.08 2019-08-28 14:36:57 +02:00
Norman Feske
44ce1b37c1 News item for version 19.08 2019-08-28 14:36:57 +02:00
Norman Feske
9a52a93c24 Release notes for version 19.08 2019-08-28 14:36:57 +02:00
Christian Helmuth
b2c59576ae depot: update recipe hashes 2019-08-28 14:36:56 +02:00
Norman Feske
7f8fe00cdc drivers_managed-pc: route for smbios_table report 2019-08-28 14:22:33 +02:00
Norman Feske
c0e8336e98 acpi_drv: report SMBIOS after ACPI info 
This order of execution reduces the boot time of Sculpt.
 2019-08-28 14:22:33 +02:00
Norman Feske
155e214a69 libc: init malloc_heap before VFS 
This is an interim fix for issue #3487.
 2019-08-28 14:22:33 +02:00
Christian Helmuth
4911acedf5 sculpt: version 19.08 2019-08-28 14:22:33 +02:00
Christian Helmuth
01bd87e90b tool/port: re-enable check_tool in install.mk 
The check was accidentally in

  66d44289e1 tool/ports: streamline hash tool usage

because $(call VAR, ...) just expands to nothing without errors if VAR
is undefined.
 2019-08-28 14:22:33 +02:00
Christian Prochaska
65f402807f qt5: update to version 5.13.0 
Fixes #3485
 2019-08-28 14:22:33 +02:00
Christian Prochaska
d00cfd7cff stdcxx: add missing symbol for qt5 update 
Fixes #3486
 2019-08-28 14:22:33 +02:00
Christian Helmuth
701b1d41e8 input_filter: update character generators 
This adds complete character-generator configurations for English (US),
German (Germany and Switzerland), and French (France and Switzerland).
The configs are manually amended and stripped-down versions of
xkb2ifcfg generated configs.

Issue #3483
 2019-08-28 14:22:33 +02:00
Sebastian Sumpf
8ae5c906d0 noux_tool_chain_auto: use absolute paths for command 
issue #3481
 2019-08-28 14:22:32 +02:00
Christian Helmuth
43a8118d2e tool: generate input_filter config from XKB 
Issue #3483
 2019-08-28 14:22:32 +02:00
Christian Helmuth
ca850c787f input_filter: dead-key sequence support 
Issue #3483
 2019-08-28 14:22:32 +02:00
Christian Helmuth
4491c070be input: fix Codepoint::INVALID in Press event 
The Press event is actually a Press_char event with a default codepoint.
The default codepoint is now

  Codepoint { Codepoint::INVALID } /* value 0xfffe */

in contrast to

  Codepoint { Input::Event::INVALID } /* value 0 */

Issue #3483
 2019-08-28 14:22:32 +02:00
Christian Helmuth
a97b8043b5 utf8: non-character U+fffe as invalid codepoint 
Unicode non-characters [1] are guaranteed to never be used for a
character. The formerly used U+fffd however is a valid character - the
replacement character [2] correctly displayed by Qt5 as <?>.

[1] https://en.wikipedia.org/wiki/Universal_Character_Set_characters#Non-characters
[2] https://en.wikipedia.org/wiki/Specials_(Unicode_block)#Replacement_character

Issue #3483
 2019-08-28 14:22:32 +02:00
Christian Helmuth
4967166811 run: exit if boot modules are missing on linux 2019-08-28 14:22:32 +02:00
Martin Stein
cc2828cf3a sntp_client: improve README, clean source code 
Fixes #3448
 2019-08-28 14:21:36 +02:00
Norman Feske
1dd68ce04b Adjust quotas for seL4 2019-08-28 14:19:45 +02:00
Norman Feske
23f3112e3e ports: bash.run script using the libc fork/execve 
Issue #3481
 2019-08-28 14:19:45 +02:00
Norman Feske
6894ced63b libc: execve 
This patch implements 'execve' in Genode's libc.

The mechanism relies on the dynamic linker's ability to replace the
loaded binary while keeping crucial libraries - in particular the libc -
intact. The state outside the libc is wiped. For this reason, all libc
internal state needed beyond the 'execve' call must be allocated on a
heap separate from the application-owned malloc heap. E.g.,
libc-internal file-descriptor objects must not be allocated or refer to
any memory object allocated from the malloc heap.

Issue #3481
 2019-08-28 14:19:45 +02:00
Alexander Senier
2a3cebdd6e libc: Allow private writable mappings 
Fixes #3480
 2019-08-28 14:18:45 +02:00
Norman Feske
66d5359d75 ldso: heuristics for libs needed during execve 
The allocation of regions within the linker area is normally left to the
best-fit 'Allocator_avl', which happens to populate the linker area
starting with the binary followed by all loaded libraried with no gaps
in between.

When replacing the binary during execve, however, we need to ensure that
the new binary does not conflict with any library that stays resident
during execve. This patch tweaks the linker's region allocation scheme
such that these libraries are placed at the end of the linker area.

Issue #3481
 2019-08-28 14:18:45 +02:00
Norman Feske
fa48054959 ldso: Dynamic_linker::respawn mechanism 
This patch extends the interface of the dynamic linker with the ability
to replace the running binary executable by another one. It is
designated for the implementation of execve. The interface consists of
two new functions.

'Dynamic_linker::keep' marks the specified shared object as unloadable.
This can be used to pin a set of libraries (i.e., the libc) within the
local address space while replacing the binary and other higher-level
libraries.

'Dynamic_linker::respawn' unloads the current binary, loads the one
specifed as first argument, and looks up the entry point symbol of the
new binary, which would be "main" for POSIX programs.

In addition to implementing the new interface, the patch adjusts the
linker at various places that previously assumed the binary to be
constant over runtime.

Issue #3481
 2019-08-28 14:18:45 +02:00
Norman Feske
808ff3714e rust: don't export cxx syms from libunwind-rust 2019-08-28 14:18:45 +02:00
Norman Feske
a98f78afd9 ldso: avoid 'dd' output in build log 
This patch is a follow-up commit for "ld: load dynamic linker at static
address on Linux". It suppresses the stderr output of 'dd' when marking
the ELF binary as executable.

Issue #3479
 2019-08-28 14:18:45 +02:00
Sebastian Sumpf
4c14af4d8a noux-pkg: enable noux packages for ARMv8 
* add -fPIC
* disable 'stdbuf' for coreutils
* works for bash as well
 2019-08-28 14:18:45 +02:00
Norman Feske
817ff6ca68 posix: improve config error diagnostics 2019-08-28 14:18:45 +02:00
Norman Feske
3c6fe6e741 ldso: statically allocate initial heap block 
This patch enables the fork.run script to run on base-linux. It should
be regarded as an interim solution, however, because the randomization
performed by the Linux kernel may still - by chance - produce a
situation where one of the libc's malloc heap regions intersects with
another dataspace dynamically attached to the child.

The better solution would be to make the 'Region_map_mmap'
implementation not depend on the kernel's allocation policy by using a
locally implemented allocator.

Issue #3478
 2019-08-28 14:18:45 +02:00
Norman Feske
bb5827b4e3 libc: fork, getpid, and wait4 
This patch complements the C runtime with support for fork, getpid, and
wait4 (and its cousin 'waitpid').

Fixes #3478
 2019-08-28 14:18:45 +02:00
Norman Feske
581785a48f Extend fork test 
This patch extends the fork test with explicit checks for the cloned
content of the heap and RW segment as well as the seek position of an
open file descriptor. It adds the new libports/run/fork.run script
that exercises the fork mechanism implemented by the libc. It is based
on noux_fork.run, which tests the mechansim provided by noux. The
test program has been moved from ports to libports.

Issue #3478
 2019-08-28 14:18:44 +02:00
Norman Feske
65f75589e9 libc: configurable initial FDs 
The libc already supports the configuration of 'stdin', 'stdout', and
'stderr' using '<libc>' config attributes. This patch equips the libc
with the additional ability to pre-initialize any other file descriptor.
A file descriptor is configured as follows:

<config>
  ...
  <libc ...>
    <fd id="3" path="/dev/log" writeable="yes" readable="no" seek="10"/>
    ...
  </libc>
</config>

Furthermore, this patch moves the FD initialization code from the VFS
plugin to the libc kernel initialization because opening the FDs
depends on 'malloc' ('strdup'), which should not be used at early
'Libc::Kernel' initialization time.

Issue #3478
 2019-08-28 14:18:44 +02:00
Norman Feske
6e38b53001 libc: use Id_space for FD allocator 
This patch replaces the former use of an Allocator_avl with the Id_space
utility, which is safer to use and allows for the iteration of all
elements. The iteration over open file descriptors is needed for
implementing 'fork'.

Issue #3478
 2019-08-28 14:18:44 +02:00
Norman Feske
354e310c87 heap: interface for obtaining backing-store info 
The new 'Heap::for_each_region' method provides information about the
heap's used virtual-memory regions. This method allows for the
mirroring of the heap state as needed by 'fork'.

Issue #3478
 2019-08-28 14:18:44 +02:00
Norman Feske
2cff12e1fb base: remove debug message 2019-08-28 14:18:44 +02:00
Norman Feske
55e99e16ef base: add missing include in base/id_space.h 2019-08-28 14:18:44 +02:00
Norman Feske
e499a04de7 ldso: config option to disarm ctors check 
By specifying the config attribute 'check_ctors="no"', the dynamic
linker won't abort the program on a missing call of
'Env::exec_static_constructors'. This is the case for forked programs
where the ctors were already executed by the forking program prior the
fork operation.

Issue #3478
 2019-08-28 14:18:44 +02:00
Norman Feske
abdf422681 ldso: interface to get infos about loaded objects 
This patch enhances the 'base/shared_object.h' interface of the dynamic
linker with the function 'for_each_loaded_object', which allows the
caller to obtain information about the currently loaded binary and
shared libraries.

The new interface is a base mechanism needed for implementing 'fork' in
the libc.

Issue #3478
 2019-08-21 14:08:18 +02:00
Norman Feske
fd8a209da2 libc: fix warning in socket_fs_plugin.cc 
By using Genode::strncpy instead of the libc's strncpy, we cannot end up
in the situation where the result lacks the zero termination (where the
number of charactors equals the destination buffer size).
 2019-08-21 14:01:54 +02:00
Sebastian Sumpf
15b27a1e9d drivers_nic-imx7d_sabre: increase caps for sel4 2019-08-21 14:00:47 +02:00
Norman Feske
b113f869bb Use I/O signals for resource-request responses 2019-08-21 13:39:44 +02:00
Norman Feske
6fb7022508 Cap-resource request on Pd_session::transfer_quota 
This patch handles the case where a PD's cap quota becomes exhausted
on the attempt to transfer caps via 'Env::pd().transfer_quota'. The
solution mirrors the existing code for RAM quota.

Prior this patch, the 'transfer_quota' operation would trigger an
'Out_of_caps' exception.
 2019-08-21 13:38:24 +02:00
Norman Feske
cc437a5eca Spelling fix in os/single_session_service.h 2019-08-21 13:35:52 +02:00
Christian Helmuth
312f801f8a depot: update recipe hashes 2019-08-21 13:25:26 +02:00
Sebastian Sumpf
8509687c8d system_rtc.run: use packages 
issue #3450
 2019-08-21 13:25:25 +02:00
Martin Stein
ce633c0bba gems: recipes/pkg/drivers_rtc-pc 
Ref #3450
 2019-08-21 13:25:25 +02:00
Sebastian Sumpf
2fc6cedcc0 ld: load dynamic linker at static address on Linux 
* move stack area to static address at beginning of binary
* set ELF type of linker from shared to executable

fixes #3479
 2019-08-21 13:25:25 +02:00
Christian Helmuth
468270f6e9 Enable libusb-internal error login smartcard test 2019-08-21 13:25:25 +02:00
Alexander Boettcher
32323abe8e vfs: handle unsupported watch exception 2019-08-21 13:25:25 +02:00
Martin Stein
1113c4f6a2 depot: mstein download and pubkey 2019-08-21 13:25:25 +02:00
Sebastian Sumpf
e855638266 hw: add system call for irq mode setting 
Core is not allowd to access the kernel's Pic implementation directly.

fixes #3474
 2019-08-21 13:25:25 +02:00
Sebastian Sumpf
f3a7d3750f base: Add Linux and GOT relative startup code on ARM 
* Linux specific startup code for base-linux ARM
* load symbols global offset table relative in order to avoid text
  relocations

issue #3466
 2019-08-21 13:25:25 +02:00
Johannes Kliemann
1bdd18a196 ada-runtime: update to 1.1 2019-08-21 13:25:25 +02:00
Christian Helmuth
0c20bb6ab9 doc: mention BOARD in getting_started.txt 2019-08-21 13:25:25 +02:00
Christian Helmuth
d01cc3bf41 Configurable set_rtc values in RTC test 
Fixes #3450
 2019-08-21 13:25:25 +02:00
Josef Söntgen
9cf5da85ef libc: change time back end to re-read RTC 
Issue #3450
 2019-08-21 13:25:25 +02:00
Josef Söntgen
316f9e4df3 libc: add VFS watch handle accessor 
Issue #3450
 2019-08-21 12:37:03 +02:00
Josef Söntgen
18e586daed libc: merge RTC and time backends 
Issue #3450
 2019-08-21 12:37:03 +02:00
Christian Helmuth
1fdd5b636b vfs: watch set-rtc signal in RTC file system 
Issue #3450
 2019-08-21 12:37:02 +02:00
Christian Helmuth
b8ed80b7dd Simple libc-based RTC test 
Issue #3450
 2019-08-21 12:37:02 +02:00
Martin Stein
cb6377355e sntp_client: report received transmit timestamp 
For now in libports because it relies on contrib sources from musl doing
the time data conversion.

Ref #3448
Ref #3450
 2019-08-21 12:37:02 +02:00
Martin Stein
322bacd380 timeout_types.xsd: Minutes type 
Ref #3448
Ref #3450
 2019-08-21 12:37:02 +02:00
Josef Söntgen
99cb585b6e libports: system RTC server and test 
For now in libports because it relies on contrib sources from musl
doing the time data conversion.

Issue #3450
 2019-08-21 12:37:02 +02:00
Christian Helmuth
0037edfeee libports: minimal time-type conversion from musl 
These are time-type conversion functions (including leap years and
seconds) taken from musl-libc git rev
c82d3bada30cb27e14abda7859da5d2e784830ff with some adaptions. Musl is
licensed under permissive MIT license.

Issue #3450
 2019-08-21 12:37:02 +02:00
Josef Söntgen
a7fe4a502d rtc_drv: implement set signal handling 
Issue #3450
 2019-08-21 12:37:01 +02:00
Josef Söntgen
ea2b330158 rtc: extended session with set-signal 
A client may register a signal handler to be notified whenever the
RTC value was changed, i.e., a mis-configured clock was synchronized,
by calling 'set_sigh()'.

Issue #3450
 2019-08-21 12:37:01 +02:00
Josef Söntgen
86cacd23bb rtc: add documentation comments 
Issue #3450
 2019-08-21 12:37:01 +02:00
Christian Prochaska
85b1563e57 mk: generate .data symbols with distinct values/addresses 
Prevent the static linker from assuming weak object symbols to be alias
symbols for unrelated non-weak symbols with the same value/address.

Fixes #3458
 2019-08-21 12:37:01 +02:00
Christian Prochaska
35c724512d stdcxx: revert '_ZTISt16invalid_argument' symbol type to 'V' 
Issue #3458
 2019-08-21 12:37:01 +02:00
Norman Feske
ff07654560 noux: rm explicit dependency from libc_noux.lib.so 
Issue #3472
 2019-08-21 12:37:01 +02:00
Norman Feske
695a212877 noux: preload libc_noux.lib.so for noux processes 
Issue #3472
 2019-08-21 12:37:01 +02:00
Norman Feske
686dd8affd ldso: ability to preload shared objects 
Fixes #3472
 2019-08-21 12:37:00 +02:00
Christian Prochaska
3acb509b9e vbox: add 'RTMemDupExTag()' implementation 
Fixes #3473
 2019-08-21 12:37:00 +02:00
Norman Feske
874172ca76 sculpt manager: support GEMDOS file systems 
Pass the distinction between GEMDOS and MSDOS file systems from
part-block reports to the vfs_rump plugin.

Fixes #3471
 2019-08-21 12:37:00 +02:00
Norman Feske
114de7721f rump_fs: support the access of GEMDOS file systems 
The GEMDOS variant is supported by NetBSD's msdos file-system driver.
However, it must explicitly be enabled by a mount flag. This patch
adds the principle ability for passing mount flags to file system
drivers and enables the MSDOSFSMNT_GEMDOSFS flag based on the
config attribute 'gemdos="yes"'.

Issue #3471
 2019-08-21 12:37:00 +02:00
Norman Feske
ceae637416 part_block: support for AHDI partition scheme 
This patch enhances part_block with support for parsing the AHDI
partition scheme, and the detection of the GEMDOS variant of FAT as used
by Atari TOS. As a side effect of the implementation, the patch improves
the MBR parsing code by avoiding pointers and using const qualifiers.

Fixes #3470
 2019-08-21 12:37:00 +02:00
Norman Feske
23c2606ce0 usb_block: support iomega zip drives 
This patch enhances the initialization sequence by issueing a START_STOP
SCSI command when needed.

Fixes #3469
 2019-08-21 12:36:59 +02:00
Norman Feske
67a3c2ea4b sculpt: handle failures of usb_block_drv 
This patch handles the situation where the usb_block_drv exits for any
reason, in particular when the driver fails to initialize the device. In
such cases, the usb_block_drv used to stay stale in the system,
effectively preventing the device from being passed to a VM. With the
patch, the USB storage device gets flagged as failed, the usb_block_drv
is removed from the runtime, and the condition is reflected at the user
interface.

This situation occurred on the attempt to access an iomega zip drive
with a version of the usb_block_drv without support for the START-STOP
command, but it may potentially also occur in other circumstances.

Fixes #3468
 2019-08-21 12:36:59 +02:00
Norman Feske
2d03e622f1 usb_block: exit on initialization failure 
Issue #3468
 2019-08-21 12:36:59 +02:00
Christian Prochaska
b9f0318ab8 mk: print symbol assembler file lines only once 
Fixes #3461
 2019-08-21 12:36:59 +02:00
Christian Helmuth
08ac64bba9 libports: download bzip2 port from sourceware.org 
fossies.org removed the archive and Linux From Scratch uses
sourceware.org.

http://www.linuxfromscratch.org/lfs/view/development/chapter03/packages.html
 2019-08-21 12:36:59 +02:00
Stefan Thöni
f018dac506 util/fifo: Change empty() method of fifo to const. 
Fixes #3464
 2019-08-13 12:02:27 +02:00
Christian Helmuth
607fe83c63 autopilot: align output for wide run-script names 2019-08-13 12:02:27 +02:00
Christian Prochaska
f73c63900f abi_symbols: drop weak local symbols 
Fixes #3457
 2019-08-13 12:02:27 +02:00
Christian Prochaska
2fad5eff95 abi_symbols: handle symbols without value 
Fixes #3456
 2019-08-13 12:02:27 +02:00
Christian Prochaska
1f56ffa51a abi_symbols: better error message on failed regexp match 
Fixes #3455
 2019-08-13 12:02:27 +02:00
Christian Helmuth
ce149397ec acpica: retrieve static battery info only once 
On Fujitsu S936, evaluating the _BIF method on each battery-info update
successively increased the RAM usage, while  _BST (dynamic battery
status) and _STA (generic status) did not show this behavior. Therefore
this commit retrieves only dynamic information periodically (resp. on
SCI IRQ). Now, acpica hast static RAM usage in idle state for 24+ hours.

The root cause for the increased RAM usage is still shady. While it
could just be normal that it grows until a certain yet unknown limit,
there may also be memory leak in contrib code or some strange AML on the
designated notebook.

Issue #3454
 2019-08-13 12:02:27 +02:00
Christian Helmuth
a7835650e8 acpica: prevent contrib code warnings 2019-08-13 12:02:27 +02:00
Christian Helmuth
83ead086a1 acpica: enable interpreter-slack mode 
This mode is used on Linux (if acpi=strict is not set on boot cmdline)
and Windows. The mode ignores certain errors and/or bad AML constructs.

  1) Allow "implicit return" of last value in a control method
  2) Allow access beyond the end of an operation region
  3) Allow access to uninitialized locals/args (auto-init to integer 0)
  4) Allow ANY object type to be a source operand for the Store() operator
  5) Allow unresolved references (invalid target name) in package objects
  6) Enable warning messages for behavior that is not ACPI spec compliant
 2019-08-13 12:02:27 +02:00
Christian Prochaska
91c8e70bef depot: add recipe for drivers_nic-imx8q_evk 
Fixes #3452
 2019-08-13 12:02:27 +02:00
Christian Prochaska
38dcdeeb04 fec_nic_drv: arm_64 support 
Fixes #3446
 2019-08-13 12:02:27 +02:00
Christian Helmuth
1c77ea2b03 base-hw: remove other board libs from recipes 2019-08-13 12:02:27 +02:00
Christian Helmuth
82d50912f6 foc: exclude .git directories from src archives 2019-08-13 12:02:27 +02:00
Christian Prochaska
4c113182b0 depot: add recipe for base-hw-imx8q_evk 
Issue #3426
 2019-08-13 12:02:27 +02:00
Stefan Kalkowski
7ced122ddc hw: support for i.MX8M Quad EVK 
Fix #3426
 2019-08-13 12:02:27 +02:00
Stefan Kalkowski
6b09ac59f0 hw: enable performance counter for ARMv8 
Ref #3426
 2019-08-13 12:02:27 +02:00
Stefan Kalkowski
ee38504d81 hw: implement update_data_region for ARMv8 
Ref #3426
 2019-08-13 12:02:26 +02:00
Sebastian Sumpf
dd505edd19 hw: GICv3 implementation 
* modern GICv3 implementation
* distributor
* redistributor
* MMIO cpu interface

Ref #3426
 2019-08-13 12:02:26 +02:00
Stefan Kalkowski
fa1aa33f83 hw: sanitize arm trustzone/virtualization services 
Those services are not SoC specific and have to reside at a generic place.

Fix #3445
 2019-08-13 12:02:26 +02:00
Stefan Kalkowski
907de9d37f hw: move timer into board.h 
Unify the generic timer implementation for ARMv7 and ARMv8.

Ref #3445
 2019-08-13 12:02:26 +02:00
Stefan Kalkowski
5c7436bf10 hw: remove SMP variable from board.h 
Whether an SoC has the multiprocessing extensions can be read out
from the identification registers, and does not need to be specified
in each board header.

Ref #3445
 2019-08-13 12:02:26 +02:00
Stefan Kalkowski
0b77e8ea62 hw: consistently move cpu into board namespace 
Ref #3445
 2019-08-13 12:02:26 +02:00
Stefan Kalkowski
875858b2cc hw: integrate interrupt controllers into board.h 
Additionally, unify more implementation details in between different
usage patterns of ARM's generic interrupt controller (v2)

Ref #3445
 2019-08-13 12:02:26 +02:00
Martin Stein
fe426e6f8f smbios_decoder: XML report from SMBIOS table ROM 
Fixes #3430
 2019-08-13 12:02:26 +02:00
Martin Stein
1e379cb3a9 drivers/acpi: provide plain SMBIOS table as report 
Ref #3430
 2019-08-13 12:02:03 +02:00
Martin Stein
ead385dd17 base-nova: EFI sys-table pointer in platform info 
Ref #3430
 2019-08-13 12:02:03 +02:00
Martin Stein
b87e21a392 base-hw: EFI sys-table pointer in platform info 
Ref #3430
 2019-08-13 12:02:03 +02:00
Christian Prochaska
04e8ba716c ps2_drv: wait longer for ACK after RESET command 
On some machines, more time passes until the acknowledgement arrives.

Fixes #3442
 2019-08-13 12:02:03 +02:00
Christian Prochaska
193a401097 Increase RAM quota of ACPI driver to 4M 
On some machines, more than 3 MiB are needed. This finally equalizes all
occurences of RAM quotas with drivers_managed.

Fixes #3443
 2019-08-13 12:02:03 +02:00
Alexander Boettcher
53a83fb76e nova: update kernel branch 
- Avoid DoS against kernel, reported by Julian Stecklina
- Provide EFI systab pointer through HIP
- Fix PCID bug

Fixes #3440
 2019-08-13 12:02:03 +02:00
Josef Söntgen
47e6d72bf2 rtc_drv: add support for setting RTC 
The rtc_drv on x86 can now by used to also set the RTC. If the config
attribute 'allow_setting_rtc' is set to 'yes' the driver will update
the RTC from the content of the 'set-rtc' ROM module. A valid ROM must
contain a top node with the following attributes: 'year', 'month',
'day', 'hour', 'minute' and 'second'.

* Only rudimentary checking of the provided values is done.
* '12H' mode is not supported.

Fixes #3438.
 2019-08-13 12:02:03 +02:00
Josef Söntgen
91ce57848c rtc_session: make timestamp printable 
Issue #3438.
 2019-08-13 12:02:03 +02:00
Martin Stein
22d7871e1d test/init: raise log performance 
Fixes #3429
 2019-08-13 12:02:03 +02:00
Tomasz Gajewski
4189157d10 cpu_bench: fixes to make it work reliably 
Added missing build dependency.

Eliminated race condition when processing test output.

Fixes #3412
 2019-08-13 12:02:03 +02:00
Sid Hussmann
eaefcc2c6f ssh: add exec channel and exit on interactive 
This commit implements the ssh exec channel request. It also handles
some shortcommings on the interactive channel like exit and concurrent
session establishments.

Pipes into the channel do not work yet. E.g.:
echo foobar | ssh noux@localhost -p 5555 "cat > /rw/test.txt"

The issue described with FIXME in Ssh::Server::incoming_connection()
could not be reproduced and might have been fixed with the improved
file descriptor handling.

Fixes #3401
 2019-08-13 12:02:03 +02:00
Christian Helmuth
92bdcbf1fe ps2: initially set all keyboard LEDs off 
This ensures a safe initial state and also informs the BIOS (resp.
notebook keyboards with dual-use keys for numpad) about our
assumptions.
 2019-07-09 15:57:32 +02:00
Norman Feske
c011d54158 sculpt/README: refer to cnuke's updated article 2019-07-09 14:18:31 +02:00
Norman Feske
d782499541 sculpt/README: specify BOARD for building 2019-07-09 13:59:59 +02:00
Norman Feske
59f9af23b9 News item for Sculpt 19.07 2019-07-09 12:20:06 +02:00
Norman Feske
70a236cccd sculpt/README: adjust git tag to version 19.07 2019-07-09 11:59:44 +02:00
Norman Feske
d33bef2e49 Update documentation for Sculpt 19.07 2019-07-09 11:46:20 +02:00
Christian Helmuth
fbae7767b4 sculpt: version 19.07 2019-07-09 09:06:54 +02:00
Christian Helmuth
17d32b3e15 depot: update recipe hashes 2019-07-09 09:06:54 +02:00
Norman Feske
6380ad1c80 pkg/vbox5-nova-sculpt: clipboard support 
To enable the clipboard for a VM, add the following node to the
<Hardware> sub node of your machine.vbox configuration:

  <Clipboard mode="Bidirectional"/>

Issue #3437
 2019-07-09 08:58:39 +02:00
Norman Feske
0c3d3e7c8f pkg/qt5_textedit: clipboard support 
Issue #3437
 2019-07-09 08:58:39 +02:00
Norman Feske
3b0d694f42 pkg/noux-system: clipboard support 
Issue #3437
 2019-07-09 08:58:39 +02:00
Norman Feske
c7e79030dd sculpt: limit rate of depot queries 
The triggering of a new depot query can happen more than once per
activation of the sculpt manager if multiple conditions call for updated
information about the depot. When this happens, the depot-query
component produces intermediate results, which are not consumed by the
sculpt manager. By deferring depot queries for a few milliseconds, we
avoid such intermediate queries, relieving the workload of the
depot-query component at system boot time.

Issue #3436
 2019-07-09 08:58:39 +02:00
Norman Feske
58a0f5c30b sculpt: break possible depot-query feedback loop 
Fixes #3436
 2019-07-09 08:58:39 +02:00
Norman Feske
c92a9ce591 sculpt: show names of unsatisfied launchers 
The diagnostic messages presented in the runtime dialog lacked the name
if the subsystem was created from a launcher, e.g., the 'vm'. Instead of
determining the subsystem name from the start-XML-node (a launcher has
no 'name' attribute but the name corresponds to the launcher's file
name), the name is now passed as a dedicated argument.
 2019-07-09 08:58:39 +02:00
Norman Feske
605f9abf96 sculpt: integrate global clipboard 
Issue #3437
 2019-07-09 08:58:39 +02:00
Norman Feske
e4412a3e56 noux_vim.run: clipboard integration with wm 
This patch enhances the noux_vim.run scenario to make it usable as a
testing ground of the terminal's and window manager's resize and
clipboard handling.

Issue #2079
 2019-07-09 08:58:39 +02:00
Norman Feske
1fe9b07c99 wm: forwarding of clipboard ROM/Report sessions 
The labels of clipboard ROM and clipboard report sessions of WM clients
must be consistent with the client's nitpicker label. Hence, we must
route those sessions through the window manager, analogously to the
approach taken for shape reports in #3165.

Issue #3437
 2019-07-09 08:58:38 +02:00
Norman Feske
2da604a6e2 terminal: make color 4 less green, less saturated 
Issue #3406
 2019-07-09 08:58:38 +02:00
Norman Feske
fc7b983a40 terminal: clipboard support 
Fixes #2079
 2019-07-09 08:58:38 +02:00
Norman Feske
6399fc12ac clipboard: allow for dynamic reconfiguration 2019-07-09 08:58:38 +02:00
Norman Feske
db6e013577 clipboard: match client labels against focus 
When setting the config attribute 'match_labels="yes"', the clipboard
performs plausibility checks for copy and paste operations against the
nitpicker focus.

Issue #3437
 2019-07-09 08:58:38 +02:00
Christian Prochaska
71ce1442c8 run: enable 'imx7d_sabre' in network tests 
Fixes #3434
 2019-07-09 08:58:38 +02:00
Christian Prochaska
5dd1abcc2d depot: add recipe for drivers_nic-imx7d_sabre 
Fixes #3433
 2019-07-09 08:58:38 +02:00
Christian Prochaska
4dafdbd5b7 dde_linux: update 'fec' driver to version 4.16.3 
Fixes #3432
 2019-07-09 08:58:38 +02:00
Christian Helmuth
949130d80e Remove warning exec_static_constructors() warning 
The warning falsely detected cases where shared objects where loaded before
exec_static_constructors() was called as unneeded even in cases were the
binary itself contained static globals.

The commit also removes one redundant call to exec_static_constructors()
from the block tester.
 2019-07-09 08:58:38 +02:00
Josef Söntgen
4c8369ab1b tool/depot/publish: support multiple threads in xz 
Introduce 'XZ_THREADS' to override the default number of threads used
for compression, which is still set to '1'. As using multiple threads
will increase the amount of memory needed during the compression and
potentially influences the size of the created archive, setting the
variable limits the impact.

Fixes #3431.
 2019-07-09 08:58:38 +02:00
Christian Helmuth
fcdad00044 test.run: build vfs plugins of in rom list 2019-07-09 08:58:38 +02:00
Norman Feske
417191bdbd wm.run: build window_layouter 
This change makes the wm.run more convenient as testbed for developing
the window layouter.
 2019-07-09 08:58:38 +02:00
Norman Feske
c00666b7e8 depot_deploy.run: integrate recall_fs 
This is needed to deploy the current version of the window_layouter pkg.
 2019-07-09 08:58:38 +02:00
Norman Feske
854a154fb4 depot_query: cache file-system accesses 
This patch introduces two caches to the depot-query tool.

- A stat cache remembers the results of 'Directory::file_exists'
  calls.

- The 'Cached_rom_query' caches the result of scanning the depot
  for a given ROM module and pkg path. To elminates the need to
  parse 'archive' files of pkgs referenced from other pkgs or
  for the repeated instantation of the same pkg.

Both caches are bypassed whenever referring to the 'local' depot user.

Fixes #3427
 2019-07-09 08:58:38 +02:00
Norman Feske
7e7eff0eb7 window layouter: reduce snap-back artifact 
When resizing windows of clients that respond very slowly to resize
requests, the window's size sometimes snapped back to its original size
immediately after finishing the drag operation.

The problem was caused by the interplay of the layout rules (obtained
via the 'rules' ROM, generated by the 'rules' report) and the
temporary interactive state that occurs during drag operations.
The rules are updated only at the time of releasing the button to keep
the overhead while dragging the window low. However, when releasing the
mouse, the (now outdated) rules kicked back into effect, triggering
resize requests for the window to its old size.

The patch solves this problem by decoupling the dragged state of a
window from the physical release of the button. The button release
triggers a transition from a DRAGGING to a SETTLING state and programs
a timer. In the SETTLING state, the windows behave as in DRAGGING state,
giving the interactive geometry precedence over the rules-dictated
geometry. During this state, further responses of window-resize requests
may come in and are handled like dragging was still in progress. After a
timeout, however, the current window layout is conserved as a new rules
report and the state goes back to IDLE.

For clients that takes a very long time (in particular, VirtualBox when
resizing the desktop, which takes sometimes multiple seconds), the
snap-back artifact can still occur, but the effect is reduced.
 2019-07-09 08:58:38 +02:00
Norman Feske
48a361107f gpt_write: fix false-positive fallthrough warning 2019-07-09 08:58:38 +02:00
Norman Feske
716453aaa1 gpt_write: fix pointer to local variable 
This problem surfaced with the new tool chain that changes the stack
layout. A pointer to the the config XML data was kept in the main object
but pointed to a stack variable. This patch fixes it by removing the
pointer.

Fixes #3416
 2019-07-09 08:58:38 +02:00
Christian Helmuth
63e69d35e0 Disable unsupported autopilot tests on rpi3 2019-07-09 08:58:38 +02:00
Christian Helmuth
a0c71cffd4 Solo5 tests require x86_64 2019-07-09 08:58:38 +02:00
Norman Feske
d1da5b9e16 libc_noux: override __libc_sigaction 
This is a follow-up commit to the libc update. The FreeBSD
implementation of sigset or sigaction calls the __libc_sigaction now.

The patch fixes the SIGWINCH handling for Vim when interactively
resizing the terminal window.

Fixes #3424
 2019-07-09 08:58:38 +02:00
Norman Feske
52ec4526e1 libc: discharge use of hidden symbols 
Issue #3424
 2019-07-09 08:58:38 +02:00
Martin Stein
e199acca05 util/print_lines: don't skip end of long lines 
Print the end of oversized lines (> 200 chars) to new lines instead of
skipping it

Fixes #3422
 2019-07-09 08:55:24 +02:00
Christian Helmuth
cea842b256 autopilot: append RUN_OPT also with --keep 
In case RUN_OPT_AUTOPILOT is changed on consecutive calls of "autopilot
--keep", the changes are now effectively propagated to the run tool.

I also documented the RUN_OPT_AUTOPILOT environment variable.
 2019-07-09 08:55:24 +02:00
Josef Söntgen
984bddcedd libc: cleanup port preparation 
- make port able to be prepared repeatedly
  Rpcgen refuses to overwrite an already existing header, so try to
  always remove it beforehand
- get rid of 'cp' warning 'cp: warning: source file
  'src/lib/libc/sys/sys/time.h' specified more than once'
- silence patch message 'patch unexpectedly ends in middle of line'

Fixes #3420.
 2019-07-09 08:55:24 +02:00
Josef Söntgen
d32e8390c2 os: remove trace_fs 
This component is superseded by the 'vfs_trace' plugin.

Issue #3294.
 2019-07-09 08:55:24 +02:00
Sebastian Sumpf
57e650ef2c gems: recipes for vfs_trace pluging 
* source recipe for vfs_trace
* pkg for 'trace_fs' with runtime

issue #3294
 2019-07-09 08:55:24 +02:00
Sebastian Sumpf
c34176580c os: recipe for 'trace_policy' 
* source packet for all trace policies

issue #3294
 2019-07-09 08:55:24 +02:00
Emery Hemingway
807f5d9af1 Use pthread_join in test/libc_pipe 
Ref #2303
 2019-07-09 08:55:24 +02:00
Norman Feske
d18f8eea2b window layouter: control maximize by editing rules 
This patch adds the missing propagation of the maximized state from the
layout rules to the internal representation of a window. Without this
patch this state could be toggled by clicking on the maximizer button
only.
 2019-07-09 08:55:24 +02:00
Christian Helmuth
8d62f21b40 autopilot: support arch-specific build directories 
Replace the notion of board-specific platforms by clear definition
architecture and board.

Also added new command line switch for "targets"

  -t <target>         test target as triple of architecture-board-kernel,
                      e.g., arm_v7a-pbxa9-hw

Related to #3316
Fixes #3417
 2019-07-09 08:55:24 +02:00
Norman Feske
7b1e3a9d63 window layouter: resize resizeable windows only 
Initiate the drag operation of a resize handle only if the client is
known to be resizable.

Issue #3303
 2019-07-09 08:55:24 +02:00
Norman Feske
070a24956d themed_decorator: use Attached_rom_dataspace::xml 
This simplifies the code a bit.
 2019-07-09 08:55:24 +02:00
Norman Feske
dec2d897c3 themed_decorator: fix jumping windows 
The decorator's detection of the hovered window element was inaccurate,
which resulted in "jumping" windows in some situations, ultimately
caused by a combination of three different mechanisms.

First, when moving the pointer into the area of a window, the decorator
would detect the hovering of the left border whenever the distance of
the pointer from the border was less than the half of the theme texture
(e.g., 64x64 pixels for the default theme). However, if the left border
margin is set to a small value (e.g., 1), there is an overlap of the
sensitive resize border area and the content. Hence, chances were quite
high that - when moving the pointer from the left into the window - the
hover report would contain the hovering of the left border.

Second, the window manager tries to hide pointer movements from the
decorator if possible. It informs the decorator of the pointer position
if any decoration is hovered or if a new window is hovered. But it does
not expose pointer movements within a window to the decorator. For this
reason, the decorator would not update the hover report as long as the
pointer stays within a once hovered window. In the situation described
above, the hover report would still contain the stale information about
the hovering of the left resize border.

Third, when the user clicks on the window, the decorator examines the
most recent hover report and - in the situation described above - finds
the left border hovered. Consequently, it initiates a window-drag
operation. While resizing the window with the left border, the window
layouter pins the right border of the window to its current position.
All window-size changes of the client will be applied towards the left
(dragged) border. In the case of the top view, which continuously
resizes the window by itself, the window would "jump". In reality, it
actually tries to respond an interactive resize operation. The window
layouter cannot guess that the client is not responding to window
layouter's resize request but is acting independently.

This patch fixes the jumping window problem for the case where the
pointer hovers the overlapping area of the resize border and the
content. However, when trying the to interactively resize the top window
via the bottom-left corner, the "jumping" can still be observed.

Fixes #3303
 2019-07-09 08:55:24 +02:00
Norman Feske
8625146011 sculpt: use secondary CPU for leitzentrale 
By decoupling the leitzentrale from the (contended) boot CPU, the fading
on F12 interferes much less with animations like nano3d deployed in the
runtime.

Fixes #3268
 2019-07-09 08:55:24 +02:00
Sebastian Sumpf
24eea0b653 base: add 'trace' function to base/log.h 
The 'Genode::trace' convenience function prints messages to Genode's
trace buffer (if tracing is enabled).

issue #3294
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
18b3253cac vfs_trace: VFS plugin that offers trace buffer access 
The plugin creates a file-system hierarchy that enabled the access of
trace buffers for each component and its threads.

issue #3294
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
25484f870e trace: make trace buffer resizeable 
Trace buffers a re-allocated during subsequent calls to 'trace'.

issue #3294
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
ac0ecdf855 base: make trace resumable 
revert relocations so a paused subject can resume without changing its
policy.

issue #3294
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
399487c85d gems: add direct fs constructor to Watch_handler 
issue #3294
 2019-07-09 08:55:23 +02:00
Alexander Boettcher
bc14a32998 bomb: handle failing child creation 
Issue #3107
 2019-07-09 08:55:23 +02:00
Christian Helmuth
cf2c9f8d88 run: abort with error if BOARD was not set 2019-07-09 08:55:23 +02:00
Stefan Kalkowski
186c35bb09 depot: recipe for base-hw-rpi3 
Ref #3407
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
3a2de0d9a3 depot_autopilot: limit dynamic to 286 MB 
* also set test-libc to 256 MB (seems to be enough)
* fixes memory issues on platforms with 512 MB RAM

issue #3407
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
1b489820a9 base: timestamp support for ARM 64-Bit 
issue #3407
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
ef8140cce5 lxip: ARM 64-Bit support 
issue #3407
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
790a57041c base: add ARM 64-Bit to base API 
issue #3407
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
e43b848ac8 jitterentropy: ARM 64-Bit version 
issue #3407
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
59347d2560 stdcxx: make linkable for ARM 64-Bit 
'typeinfo for std::invalid_argument' was not found when using 'V' (weak
object)

issue #3407
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
2b14284e2b libc: add ARM-64 bit to recipe 
issue #3407
 2019-07-09 08:55:23 +02:00
Sebastian Sumpf
c107aba913 depot_autopilot: add ARM 64-bit support 
issue #3407
 2019-07-09 08:55:22 +02:00
Sebastian Sumpf
e337b844e5 depot: support for ARM 64-bit for base-foc on rpi3 
issue #3407
 2019-07-09 08:55:22 +02:00
Stefan Kalkowski
90d07741aa hw: support for ARM64 Raspberry Pi 3 
Restriction: enables only cpu core 0 and the timer interrupt by now.

Fix #3405
 2019-07-09 08:55:22 +02:00
Stefan Kalkowski
87015df66c hw: change update_pd to invalidate_tlb 
In the past, the core-only privileged syscall `update_pd` was used only
to invalidate the TLB after removal of page-table entries.
By now, the whole TLB at least for one protection domain got invalidated,
but in preparation for optimization and upcomingARM v8 support,
it is necessary to deliver the virtual memory region that needs to get
invalidated. Moreover, the name of the call shall represent explicitely
that it is used to invalidate the TLB.

Ref #3405
 2019-07-09 08:55:22 +02:00
Stefan Kalkowski
d9a0f76e7a hw: extend long-descriptor page table format 
Ref #3405
 2019-07-09 08:55:22 +02:00
Tomasz Gajewski
0ecc48e6de Mini UART driver for Rpi3 
Ref #3405
 2019-07-09 08:55:22 +02:00
Christian Helmuth
efe0887247 tool: fix BOARD_RUN_OPT for arm_v8 2019-07-09 08:41:53 +02:00
Christian Helmuth
3c4c460f82 depot: update recipe hashes 2019-06-13 13:40:37 +02:00
Christian Helmuth
ab3efc0a33 depot_autopilot: skip test-fs_packet on rpi 2019-06-13 12:22:50 +02:00
Christian Helmuth
e1e67d8330 Increase timeouts in test-libc* pkgs 
These tests often fail with Qemu on aged hosts because of the tight
timeouts.
 2019-06-13 12:22:50 +02:00
Christian Helmuth
de4c31e557 Additional boot time (10 s) in test.run 2019-06-13 12:22:50 +02:00
Martin Stein
5a59d295a4 depot_autopilot.run: relax test timeout 
There are still nightly tests like test-tcp_bulk_lxip on sel4 x86_64 qemu
that manage to hit the test timeout of the run script although the test was
successful. So, raise the extra time added by the run script to 30 seconds.

Ref #3411
 2019-06-13 12:22:49 +02:00
Martin Stein
fe5ada85cb depot_autopilot.run: max nr of tests per boot 
Some platforms (sel4 imx6/imx7) cannot manage to execute all tests in a single
boot. Thus, we re-boot them periodically after a given maximum number of tests
to avoid that arbitrary tests always fail due to the long uptime and not due to
the tests themselves.

If the maximum number of tests is set to 0, no limit is applied.

Fixes #3411
 2019-06-13 12:22:49 +02:00
Johannes Kliemann
b55358c535 ada-runtime: fix exception handling on ARM 
ref #3395
 2019-06-13 12:22:49 +02:00
Christian Helmuth
1b059649ff depot_autopilot: skip test-spark_exception on ARM 
Issue #3395
 2019-06-13 12:22:49 +02:00
Christian Helmuth
45f3ccc42a vfs: fix empty-node handling in inline fs 
Now <inline></inline> as well as <inline/> are supported.
 2019-06-13 12:22:49 +02:00
Christian Helmuth
4d34f9f9f4 Reduce third-party kernel build warnings 2019-06-13 12:22:49 +02:00
Christian Prochaska
62232f9a35 fs_rom: retry watching a path element after watching its parent dir 
If a path element cannot be watched because it does not exist yet, the next
available parent directory is watched to detect the creation of the missing
path element. If a missing path element got created just between the failed
watch request for the element and the successful watch request for its parent
directory, the creation of the path element could remain unnoticed.

With this commit, 'fs_rom' tries to watch a path element again if the watch
request for its parent directory was successful.

Fixes #3398
 2019-06-13 12:22:49 +02:00
Alexander Boettcher
a25c656b1a image/disk: increase estimated disk size 
Fixes #3410
 2019-06-13 12:22:49 +02:00
Christian Helmuth
ed0d76552b Relax timing requirements in lazy timer test 
The lazy-timer test depends on the faster-timer handler to be executed
before the fast timeout occurs, which was pretty hard to achieve on Qemu
and a busy host machine. Therefore, I increased the fast-to-faster
timeout ratio from 50/25 ms to 200/25 ms and set the test runtime to
4000 ms.
 2019-06-13 12:22:49 +02:00
Emery Hemingway
43315444a8 Libc: wrap VFS sync's in VFS_THREAD_SAFE 
Move the _vfs_sync implementation from vfs_plugin.h to vfs_plugin.cc and
wrap all VFS access in the VFS_THREAD_SAFE macro.

Syncing must be made thread safe because it often happens when closing
files and may require suspending the current thread for I/O signal
dispatching.

Ref #3409
 2019-06-13 12:22:49 +02:00
Alexander Boettcher
c055986e38 vbox5*.run: enable VM tests for Fiasco.OC 
Issue #3111
 2019-06-13 12:22:49 +02:00
Alexander Boettcher
7ec37b2d8d vbox5: adjust to run with Fiasco.OC 
- use priorities solely with nova
- handle exceptions during vCPU creation
- avoid assertion in vga_vbda
- silence compiler warnings about outdated register keyword

Issue #3111
 2019-06-13 12:22:49 +02:00
Alexander Boettcher
64fac6cee7 foc: extend vm_session impl. for vbox5 
- transfer more guest registers
- print warning once for unsupported guest registers
- improve synchronization of vm state transfer

Issue #3111
 2019-06-13 12:22:49 +02:00
Emery Hemingway
12127a7bd2 Construct Terminal_session within with_libc 
The "pipe" procedure can only be safely called within with_libc.

Fix #3408
 2019-06-13 12:22:49 +02:00
Josef Söntgen
eaed28f68d sculpt: document version string constraint 
Issue #3403.
 2019-06-13 12:22:49 +02:00
Tomasz Gajewski
4a102fb58c sculpt doc: be more precise about the use of GPT 
Information about GPT being required for permanent sculpt
customizations

Fixes #3381
 2019-06-13 12:22:48 +02:00
Christian Helmuth
ae7fbd6b1a terminal: modernize color palette 
The terminal now got a configurable palette for 16 colors (8 normal, 8
bright/bold).

  <config>
    <palette>
      <color index="0" value="#000000"/> <!-- black is real black -->
      <color index="8" value="#101010"/> <!-- bright black stands out a bit -->
    </palette>
  </config>

Note, the old (undocumented) <color index="..." bg="..."> configuration
scheme is no longer supported.

Also, this commit adds a pleasing default palette that ensures
readability of ViM's standard hightlighting.

Fixes #3406
 2019-06-13 12:22:48 +02:00
Christian Helmuth
72478a9e31 nova: provide missing Rpc_entrypoint::is_myself() 
The symbol is implemented by the common implementation but was missing
from the nova-specific fork of rpc_entrypoint.cc.
 2019-06-13 12:22:48 +02:00
Stefan Kalkowski
811410fdbb foc: update to current github version (fix #3402) 
Instead of hosting our outdated own fork, this commit now uses kernkonzept's
nowadays available github source repositories. This hopefully simplifies
updates, or cherry-picking single fixes.
 2019-06-13 12:22:48 +02:00
Christian Helmuth
5738e53be2 Warn about questionable call of wait_and_dispatch_one_io_signal 
Issue #2399
 2019-06-13 12:17:32 +02:00
Christian Helmuth
be4d8b24e2 run: improve error message on missing recipe 2019-06-13 12:17:32 +02:00
Christian Helmuth
1f77f37310 qt5: handle keypad-enter key 2019-06-13 12:17:32 +02:00
Norman Feske
7e174e73be ttf_font: prevent out-of-bounds access 
Thanks Alexander Böttcher for investigating.

Fixes #3393
 2019-06-13 12:17:31 +02:00
Emery Hemingway
ff23d665c6 depot: move ehmry depot URL 
Fix #3400
 2019-06-13 12:17:31 +02:00
Christian Helmuth
00daecf838 qt5: enable capslock/numlock in test run scripts 2019-06-13 12:17:31 +02:00
Christian Helmuth
f8f9f710e5 vbox5: rewrite (truncate) nitpicker session label 
The shape-report label matching depends on the truncated label, i.e.,
Nitpicker label "wm -> vm" matches, while "wm -> vm -> vbox" does not.
 2019-06-13 12:17:31 +02:00
Alexander Boettcher
e77362e7b9 driver_manager: fix vesa_fb_drv binary usage 
Fixes #3394
 2019-06-13 12:17:31 +02:00
Alexander Boettcher
c58acbd444 sculpt: add VM service 
Fixes #3396
 2019-06-13 12:17:31 +02:00
Alexander Boettcher
e2c1ce2add foc: add core threads as trace source 
Fixes #3390
 2019-06-13 12:17:31 +02:00
Alexander Boettcher
fc1b7a626c foc: add idle times as trace source 
Issue #3390
 2019-06-13 12:17:31 +02:00
Christian Prochaska
a7825ef292 gdb_monitor: disable pagination earlier 
Fixes #3397
 2019-06-13 12:17:31 +02:00
Christian Prochaska
8dd391371c libusb: show "device is plugged" message only when plugged 
Fixes #3399
 2019-06-13 12:17:31 +02:00
Josef Söntgen
737e9580b8 nvme_drv: only set the first INTMS bit on reset 
Since we only set the first INTMC bit a check in QEMU will fail
and we don't get interrupts.

Fixes #3369.
 2019-06-13 12:17:31 +02:00
Josef Söntgen
a2abc13425 nvme_drv: only wait for first admin cmd 
Leave the loop if we processed the cmd we were looking for. That
should fix the delay users notice in Sculpt when NVMe is used.

Issue #3369.
 2019-06-13 12:17:31 +02:00
Josef Söntgen
1daf179db9 nvme_drv: adapt run script to block_tester config 
Issue #3369.
 2019-06-13 12:17:31 +02:00
Josef Söntgen
6b48a08697 libc: silence noncritical dummy implementations 
* isatty

  For the moment it is not possible to determine if the fd belongs to a
  tty. For whatever reasons, the check is done multiple times, e.g.
  'tclsh', which will spam the LOG.

* sysctl(HW_FLOATINGPT)

  FPU is enabled on all our platforms, so return true.

Fixes #3389.
 2019-06-13 12:17:31 +02:00
Emery Hemingway
dc8a2ec523 Remove libc_terminal 
Fix #3378
 2019-06-13 12:13:40 +02:00
Christian Prochaska
46f89d143b libports: ncurses: disable relative cursor movement 
Fixes #3380
 2019-06-13 12:13:40 +02:00
Emery Hemingway
8a4adceaf0 Add note on PCAP dump to run/fetchurl.inc 
Ref #3388
 2019-06-13 12:13:40 +02:00
Emery Hemingway
bbf7a6230e VFS lwIP: restart DHCP with link-state changes 
Start and stop the DHCP state machine as the Nic link-state changes.
Invoke the link state handler during configuration rather than assume
that in the case of a downed link lwIP will defer DHCP until the link
comes up.

Additionally, support static DNS configuration via the "nameserver"
configuration attribute.

Fix #3388
 2019-06-13 12:13:39 +02:00
Christian Helmuth
24a3eea89e version: 19.05 2019-05-29 11:03:29 +02:00
Norman Feske
65b9a46d6c News item for version 19.05 2019-05-29 11:02:24 +02:00
Norman Feske
4dbb18bf5e Release notes for version 19.05 2019-05-29 11:02:24 +02:00
Norman Feske
44181fedf7 doc: update tool_chain.txt 
This commit updates the tool-chain information, pointing at the
versioned install path now.

It also removes historic information that is no longer of interest.
 2019-05-29 11:02:24 +02:00
Christian Helmuth
2b183f9497 depot: update recipe hashes 2019-05-29 10:20:52 +02:00
Martin Stein
a09150e6b4 depot_autopilot.run: relax reboot timeout 
It might happen, in CPU intensive tests (like TCP bulk lxip), or when
printing debugging output after a failed test (as done currently on
staging), that the run script on the host gives up and reboots the
platform too early. Thus, we raise the buffer time. A reboot should
be necessary only in rare cases anyway.

Fixes #3387
 2019-05-29 10:20:52 +02:00
Martin Stein
da3c52668d depot_autopilot.run: fix insufficient Qemu RAM 
Fixes an error output in init at the beginning of the test and insufficient
transfer of RAM to the runtime of RAM-intensive test packages.

Ref #3387
 2019-05-29 10:20:52 +02:00
Martin Stein
5ba19dd49b test-nic_loopback: fix RAM resource request 
Ref #3387
 2019-05-29 10:20:52 +02:00
Sebastian Sumpf
4b72bbaa57 libc: mktime signed overflow 
issue #3289
 2019-05-29 10:20:52 +02:00
Alexander Boettcher
9f02151d0b seoul-auto: disable autopilot test on qemu x86_32 
Issue #3373
 2019-05-29 10:20:52 +02:00
Sebastian Sumpf
f75f477c2b nic_router: [[fallthrough]] annotation 
issue #3377
 2019-05-29 10:20:52 +02:00
Sebastian Sumpf
cec050983a hw: [[fallthrough]] annotations for ARM/RISC-V 
* double checked

issue #3377
 2019-05-29 10:20:52 +02:00
Christian Prochaska
ca39a9ea61 tool_chain: versioned install location 
Issue #3307
 2019-05-29 10:20:52 +02:00
Martin Stein
ca95621db5 run/nic_router_uplinks: do not run on muen 
On muen, we don't have configured access to wifi so far. Thus, the wifi
driver doesn't come up and we should skip tests using this driver for
the platform.

Fix #3384
 2019-05-29 10:20:52 +02:00
Tomasz Gajewski
27e9d1fcf0 tool/run/README: Fixed tftp related argument names in example 
Fixes #3382
 2019-05-29 10:20:52 +02:00
Christian Prochaska
f99a1d78bb os: fix 'CUU' terminal command 
Decrement the y position to move the cursor up.

Fixes #3379
 2019-05-29 10:20:52 +02:00
Sebastian Sumpf
d417d26ce8 hw: fix calculation of CPU count on x86_64 
On x86 the CPU count is determined through ACPI's MADT by counting the
local APICs reported there. Some platforms report more APICs
than there are actual CPUs. These might be physically disabled CPUs.
Therefore, a check if the LAPIC is actually physically enabled in
hardware fixes this issue.

Thanks to Alex Boettcher

fixes #3376
 2019-05-29 10:20:52 +02:00
Sebastian Sumpf
5611020f33 hw: fix stack alignment in bootstrap for x86_64 
Fix initial stack pointer alignment for x86_64 in crt0.s startup code of
bootstrap. SysV ABI states that upon function entry (rsp + 8) % 16 = 0.
There, we have to align the stack to 16 bytes before all 'call'
instruction not 8. Otherwise FPU (GP) exception might be raised later on
because of unaligned FPU accesses.

issue #3365
 2019-05-29 10:20:52 +02:00
Emery Hemingway
fc68e97e4a Sculpt: redirect "nic_drv" binary to "ipxe_nic_drv" 
Ref #2190
Fix #3374
 2019-05-29 10:20:52 +02:00
Alexander Boettcher
b3a9ad2215 vbox5: fpu support via new vm_session 
Issue #3111
 2019-05-29 10:20:52 +02:00
Alexander Boettcher
ab6315d6b4 vm_session: add fpu state for x86 
Issue #3111
 2019-05-29 10:20:52 +02:00
Alexander Boettcher
5673c163fb vbox5: use vm_session interface 
Issue #3111
 2019-05-29 10:20:52 +02:00
Alexander Boettcher
07ee9654e4 vm_session: consume spurious wakeups on nova 
Issue #3111
 2019-05-27 14:53:32 +02:00
Alexander Boettcher
9f70084524 base: add vm session API recipe 
Issue #3111
 2019-05-27 14:53:32 +02:00
Sebastian Sumpf
da17f2cbd3 hw: eager FPU switching for x86_64 
Since gcc 8.3.0 generates SSE instructions into kernel code, the
kernel itself may raise FPU exceptions and/or corrupt user level FPU
contexts thereby. Both things are not feasible, and therefore, lazy FPU
switching becomes a no go for base-hw because we cannot avoid FPU
instructions because of the entanglement of base-hw, base, and the tool
chain (libgcc_eh.a).

issue #3365
 2019-05-27 14:53:32 +02:00
Norman Feske
34a711b439 tool/parse_cxx: const var decl, initializers 
This patch handles a few corner cases that appear in the headers of
Genode 19.05, in particular the new client-side block API in
'block_session/connection.h'.
 2019-05-27 14:53:32 +02:00
Christian Helmuth
b359ec14a8 autopilot: align output for wide board names 2019-05-27 14:53:32 +02:00
Alexander Boettcher
68d5293f01 platform_drv: adjust to g++ 8.3.0 
Issue #3307
 2019-05-27 14:53:31 +02:00
Alexander Boettcher
91146a3a70 acpi_drv: adjust to g++ 8.3.0 
Issue #3307
 2019-05-27 14:53:31 +02:00
Norman Feske
405a9d2144 Refinements for updated "Genode Foundations" book 
- Improve API descriptions
- Remove obsolete Xml_node::value method (fixes #3323)
- Follow coding style 'const char' -> 'char const'
- Avoid '>>' when nesting templates (limitation of parse_cxx)
 2019-05-27 14:52:53 +02:00
Christian Helmuth
6c42bd4dd3 libc: use non-anonymous 'operator new' and destroy() 
This removes implementations of and also references to anonymous new and
delete operators from the libc implementation. As allocators for
new/delete Libc::Allocator instances are used, which (paradoxically) map
to libc malloc/free.
 2019-05-27 14:52:53 +02:00
Christian Helmuth
9288fe63ad arg_string: return default value on parsing error 2019-05-27 14:52:53 +02:00
Pirmin Duss
bd045c65a2 util/string.h: suppress case fallthrough warnings 2019-05-27 14:52:53 +02:00
Johannes Kliemann
35b1440c97 ada-runtime: update to GCC 8.3.0 
ref #3362
fixes #3371
 2019-05-27 14:52:53 +02:00
Martin Stein
ba9ffc6243 nic_dump: do not depend on libports repo 
Fixes #3372
 2019-05-27 14:52:52 +02:00
Martin Stein
95ece89cf8 tool/depot: improve handling of missing ports 
* The extract tool determines and reports all missing ports at once.
* The extract tool automatically prepares all missing ports if PREPARE_PORTS=1.
* The missing_ports tool prints a list of missing ports for given archives.

Fixes #3353
 2019-05-27 14:52:52 +02:00
Stefan Kalkowski
6d8d6b5552 hw: disable alignment checking at earliest 
Fix #3370
 2019-05-27 14:52:52 +02:00
Sebastian Sumpf
f18285205c hw: enable FPU during CPU startup on x86 
Also disable TS (task switch) flag in cr0 during kernel initialization,
so FPU faults are not raised. This became necessary since GCC lately
aggressively generates FPU instructions at arbitrary places and also at
early kernel-bootstrapping stages.

fixes #3365
 2019-05-27 14:52:52 +02:00
Christian Helmuth
e60b597af5 linux: 32-bit compatibility with GCC versions before 7 
GCC version 7 and above generate calls to __divmoddi for 64bit integer
division on 32-bit. Unfortunately, libgcc liberaries of older compilers
lack this symbol and are still in use by Debian/Ubuntu LTS at least.
 2019-05-27 14:52:52 +02:00
Christian Helmuth
96b29c6f8a Update lwip_lx test (vfs plugin, ld="no") 2019-05-27 14:52:52 +02:00
Christian Prochaska
a5d6cbf44d tool_chain: strip gdb 
Issue #3307
 2019-05-27 14:52:52 +02:00
Martin Stein
79d1c4f83a nic_router_flood: adapt test timeout 
Fixes #3368
 2019-05-27 14:52:52 +02:00
Christian Prochaska
4dd9d5eb6c dde_zircon: fix compile error with GCC 8.3.0 
Fixes #3366
 2019-05-27 14:52:52 +02:00
Alexander Boettcher
753e78d122 seoul: adjust to g++ 8.3.0 
Issue #3307
 2019-05-27 14:52:52 +02:00
Alexander Boettcher
b50e54b0a8 nova: adjust to g++ 8.3.0 
Issue #3307
 2019-05-27 14:52:52 +02:00
Christian Prochaska
7fadfbbd9f Treat 'implicit fallthrough' errors as warnings 
After fixing the warnings, the compiler option should be removed.

Issue #3307
 2019-05-27 14:52:52 +02:00
Christian Prochaska
b4649d84ee ada-runtime: adjustments for Genode tool chain 19.05 
Issue #3307
 2019-05-27 14:52:52 +02:00
Christian Prochaska
eb4d431e76 stdcxx: update to version 8.3.0 
Issue #3307
 2019-05-27 14:52:52 +02:00
Christian Prochaska
8e2e4374f5 sanitizer: update to version 8.3.0 
Issue #3307
 2019-05-27 14:52:52 +02:00
Christian Prochaska
720919bc14 gcov: update to version 8.3.0 
Issue #3307
 2019-05-27 14:52:52 +02:00
Christian Prochaska
61140380ee base: update 'ld' symbols for tool chain 19.05 
Issue #3307
 2019-05-27 14:52:52 +02:00
Sebastian Sumpf
a8d856fb65 ldso: dynamic linking support for ARM 64-bit 
* added relocation support
* added assembler invocation path for jump slot relocations

fixes issue #3260
 2019-05-27 14:52:52 +02:00
Sebastian Sumpf
7dc875e8c7 base: dynamic linking support of crt0.s on ARM 64-bit 
* added global offset table relative loading for global symbols
* removed 'initial_sp' and 'initial_x0' because they are currently not
  used on this platform. If required they are easy to resurrect.

issue #3260
 2019-05-27 14:52:51 +02:00
Stefan Kalkowski
f0d28eeca7 foc: add support for Raspberry Pi 3 
Ref #3260
 2019-05-27 14:52:51 +02:00
Stefan Kalkowski
71a48c0a26 base: add initial support for ARM 64-bit 
Ref #3260
 2019-05-27 14:52:51 +02:00
Stefan Kalkowski
c98597a2c0 base: increase initial stack 
Ref #3260
 2019-05-27 14:52:51 +02:00
Christian Prochaska
6af3899bcb Enable C++17 by default 
Issue #3307
 2019-05-27 14:52:51 +02:00
Christian Prochaska
74260c96bf tool_chain: integrate 'ali2dep' tool 
Fixes #3361
 2019-05-27 14:52:51 +02:00
Stefan Kalkowski
fc922d263c tool_chain: add aarch64 compiler target 
Ref #3260
 2019-05-27 14:52:51 +02:00
Sebastian Sumpf
36f90e57f3 riscv: tool chain 19.05 
issue #3273
 2019-05-27 14:52:51 +02:00
Christian Prochaska
f369da741d tool_chain: update GDB to version 8.2.1 
Issue #3307
 2019-05-27 14:52:51 +02:00
Christian Prochaska
1720d2d86d tool_chain: update GCC to version 8.3.0 
Issue #3307
 2019-05-27 14:46:54 +02:00
Christian Prochaska
96ed3c8db0 tool_chain: update binutils to version 2.32 
Issue #3307
 2019-05-27 14:46:54 +02:00
Christian Prochaska
fe878e65de base: add copy constructor to 'Genode::Session_label' 
Fixes #3333
 2019-05-27 14:46:54 +02:00
Josef Söntgen
d223539165 wifi_drv: ignore socket flags on socket creation 
The libc features support for SOCK_CLOEXEC now which gets set via
the type argument in 'libnl'. Since we are only interested in the
actual type, i.e. if it is SOCK_RAW, just ignore the flags.

Issue #3289.
 2019-05-27 14:46:54 +02:00
Sebastian Sumpf
a71253fa58 ieee754: adjust to libc/libm update 
* Adjust expected pow(-1/1, inf/-inf/nan) output to current libm
  behavior (as in FreeBSD, glibc, and OpenLibm) and return 1 on
  x86_64/x86_32/arm

* Add 'double float' outputs for ARM which are defined to float

issue #3289
 2019-05-27 14:46:54 +02:00
Christian Helmuth
6db43d2c19 libc_noux target depends on POSIX runtime 
This prevents

  error copying "bin/libm.lib.so": no such file or directory
  error copying "bin/posix.lib.so": no such file or directory

when running "make run/noux" from an empty build directory.
 2019-05-27 14:46:54 +02:00
Christian Prochaska
4fc3eca4aa base-hw: fix compile errors with GCC 8.3.0 
Fixes #3326
 2019-05-27 14:46:54 +02:00
Stefan Kalkowski
5c77ebb1fb hw: factor out x86 specific bootinfo 
Ref #3326
 2019-05-27 14:46:54 +02:00
Stefan Kalkowski
054df95ea4 hw: unify board definitions of bootstrap/core 
Ref #3326
 2019-05-27 14:46:54 +02:00
Martin Stein
8eecb39792 test/timeout: configurable fast-polling buffers 
On some platforms (foc+pbxa9, hw+imx53_qsb_tz, hw+rpi) the default buffer size
is to much for the RAM available on the board. Thus, decrease the buffer size
and therefore the number of polls for these platforms only.

Fixes #3354
 2019-05-27 14:46:54 +02:00
Christian Prochaska
8af81668ea rump_fs: fix run time errors when built with GCC 8.3.0 
Fixes #3360
 2019-05-27 14:46:54 +02:00
Christian Prochaska
c9b5dcafbd vbox: save FPU state in 'EPT violation' exit handler 
Fixes #3359
 2019-05-27 14:46:54 +02:00
Christian Prochaska
f596595c98 vbox: use 'Abi::stack_align()' for exit handler stack 
This ensures proper stack alignment for FPU instructions on x86_64.

Fixes #3358
 2019-05-27 14:46:53 +02:00
Christian Prochaska
3fa994a7a4 dde_bsd: update 'platform_execute()' functions based on libc 
This ensures proper stack alignment for FPU instructions on x86_64.

Fixes #3357
 2019-05-27 14:46:53 +02:00
Christian Prochaska
c7d9df6350 lx_kit: update 'arch_execute()' functions based on libc 
This ensures proper stack alignment for FPU instructions on x86_64.

Fixes #3356
 2019-05-27 14:46:53 +02:00
Christian Prochaska
c2c33d6808 base: save FPU registers in '_jmp_slot' function (x86_64) 
Issue #3355
 2019-05-27 14:46:53 +02:00
Christian Prochaska
9923a1bf50 ports: fix 'seoul' compile errors with GCC 8.3.0 
Fixes #3352
 2019-05-27 14:46:53 +02:00
Emery Hemingway
863654d188 Libc: update port to Freebsd 12 
Fix #3289
 2019-05-27 14:46:53 +02:00
Emery Hemingway
777d92f6de Port of OpenLibm 
Replace the FreeBSD libm with OpenLibm, which is easier to port.
OpenLibm is used by Mirage's freestanding Ocaml runtime (sin POSIX).

https://openlibm.org/

Ref #3289
 2019-05-27 14:46:53 +02:00
Christian Prochaska
382371d9e4 nova: remove '-Wabi' compiler option 
Fixes #3351
 2019-05-27 14:46:53 +02:00
Christian Prochaska
3556a40f81 seoul: adjustments to compile with GCC 8.3.0 
Issue #3352
 2019-05-27 14:46:53 +02:00
Emery Hemingway
4caffd79db Ethernet multicast support at nic_bridge 
Fix #3282
 2019-05-27 14:46:53 +02:00
Stefan Kalkowski
3460444d84 os: name tz_vmm unambigously (ref #2190) 
Moreover, express current requirement to hw API, which was missing until now.

Originally, in the board-specific build directory of imx53_qsb the
"KERNEL" variable was preset with "hw". Therefore, it was not perceived
that this dependency is required.

Ref #3316
 2019-05-27 14:46:53 +02:00
Alexander Boettcher
faee97dd1e sel4: let seoul-kernelbuild.run succeed 
Issue #3111
 2019-05-27 14:46:53 +02:00
Alexander Boettcher
c6ec2c1dd7 libc_terminal plugin: increase stack 
Issue #3111
 2019-05-27 14:46:53 +02:00
Christian Prochaska
92c314d3c8 intel_gpu_drv: fix compile error with GCC 8.3.0 
Fixes #3332
 2019-05-27 14:46:53 +02:00
Christian Prochaska
934ada72fa nova: remove deprecated 'register' keyword 
Fixes #3350
 2019-05-27 14:46:53 +02:00
Christian Prochaska
05a382b1a8 ports: fix 'noux' compile error with GCC 8.3.0 
Fixes #3348
 2019-05-27 14:46:53 +02:00
Christian Helmuth
0777d16e78 Revert "ieee754: disable test for muen (fix #3305)" 
This reverts commit 918b9a9fa4.

The Muen debug console buffer was increased by the recent update, which
alleviates the issue with many log messages as in the ieee754 test.
 2019-05-27 14:46:53 +02:00
Adrian-Ken Rueegsegger
d131e537e9 Update Muen port 
- Drop unnecessary patch
- Improved build speed/parallelization
- Increased log channel size
- Fix path in base-hw/Muen documentation
 2019-05-27 14:46:53 +02:00
Christian Prochaska
37ff9e1e23 gems: fix 'sculpt_manager' compile errors with GCC 8.3.0 
Fixes #3347
 2019-05-27 14:46:52 +02:00
Josef Söntgen
c8c354d1e6 fetchurl: add progress timeout handling 
The component will now abort an ongoing download attempt if it stalls
for given amount of time, the default is 10 seconds.

Fixes #3346.
 2019-05-27 14:46:52 +02:00
Christian Prochaska
4e6216bced libports: fix 'qt5_webcore' compile error with GCC 8.3.0 
Fixes #3345
 2019-05-27 14:46:52 +02:00
Christian Prochaska
5b232df503 libports: fix 'solo5' compile error with GCC 8.3.0 
Fixes #3344
 2019-05-27 14:46:52 +02:00
Stefan Kalkowski
a1e70b9ba4 kernel: differentiate board-specific components 
Components like kernel, core, and bootstrap that are built for a
specific board need to reside inside the same architectural dependent
build directory. For instance there are sel4, foc, and hw kernel builds
for imx6q_sabrelite and imx7d_sabre, which have to reside inside the same
arm_v7 build directory.
This commit names those components explicitely, and adapts the run-tool to it.

Fix #3316
 2019-05-27 14:46:52 +02:00
Stefan Kalkowski
c43267dbaa tool: deprecate board-specific build directories 
* Introduces BOARD variable to determine actual board
* Removes formerly deprecated kernel-specific build directories

The following boards are available:

arm_v6: rpi
arm_v7a: arndale, imx53_qsb, imx53_qsb_tz, imx6q_sabrelite, imx7d_sabre,
         nit6_solox, odroid_x2, odroid_xu, panda, pbxa9, usb_armory,
         wand_quad, zynq_qemu
x86_64: pc, linux, muen
x86_32: pc, linux
riscv:  spike

Ref #3316
 2019-05-27 14:46:29 +02:00
Christian Prochaska
02afb04b7d base: fix 'test-sanitizer' compile error with GCC 8.3.0 
Fixes #3343
 2019-05-16 13:11:03 +02:00
Christian Prochaska
e9fcbace61 libports: fix 'extract' compile errors with GCC 8.3.0 
Fixes #3342
 2019-05-16 13:11:03 +02:00
Christian Prochaska
2b21f41495 demo: fix 'mini_c' compile errors with GCC 8.3.0 
Fixes #3341
 2019-05-16 13:11:03 +02:00
Josef Söntgen
dd6bd0f880 sequence: add keep-going feature 
When enabling the 'keep_going' config attribute, the component will
carry on in case a child exited with an error. In addition, if the
'restart' attribute is set it will start executing the children from
the beginning.
 2019-05-16 13:11:03 +02:00
Martin Stein
1583782446 Revert "nic_router_flood: reworked to stress/analyze more" 
This reverts commit ae55954919.
 2019-05-16 13:11:03 +02:00
Christian Helmuth
22af4436f7 nova: log sp/bp on unresolvable exception 2019-05-16 13:11:03 +02:00
Martin Stein
8fb0d668e0 heap: fix exception handling in _allocate_dataspace 
Previously, only Invalid_dataspace, Region_conflict, and Out_of_ram were
handled for both allocate and attach with the same handlers. However,
both operations can also throw Out_of_caps and for all exceptions during
attach, the dataspace must be freed again whereas this is not the case
when the exception occured during allocate.

Issue #2953
 2019-05-16 13:11:02 +02:00
Sebastian Sumpf
467b96abf4 ldso: lazy binding support for RISC-V 
* added assembler invocation path for jump slot relocations
* fix GOT initialization (jmp_slot pointer goes to GOT[0] not GOT[2] on
  RISC-V)

Fixes #3339
 2019-05-16 13:11:02 +02:00
Christian Helmuth
4f0b17a4dc dde_ipxe: update Intel NIC driver to latest upstream 
This patch is motivated by sporadic hangs during link down/up on i219
NICs handling and the fix implemented upstream in

  https://git.ipxe.org/ipxe.git/commit/546dd51de8459d4d09958891f426fa2c73ff090d

Issue #1220
 2019-05-16 13:11:02 +02:00
Christian Helmuth
a633b5e36e drivers_interactive-pc: quotas (ps2_drv, input_filter) 2019-05-16 13:11:02 +02:00
Christian Helmuth
ea954e7e15 demo: support more platforms in test / drivers RAM 2019-05-16 13:11:02 +02:00
Christian Helmuth
3033496fa4 run: honor DEPOT_DIR for --depot-auto-update 
Issue #3270
 2019-05-16 13:11:02 +02:00
Stefan Kalkowski
13fb51eecf hw: remove implicit SoC-specific include path 
Fix #3336
 2019-05-16 13:11:02 +02:00
Christian Prochaska
9c17c83bf1 os: fix 'test-cpu' compile error with GCC 8.3.0 
Fixes #3335
 2019-05-16 13:11:02 +02:00
Christian Prochaska
2191ff656e os: fix 'test-block_request_stream' compile error with GCC 8.3.0 
Fixes #3334
 2019-05-16 13:11:02 +02:00
Christian Prochaska
2aeb1a70ea os: remove 'register' keyword from 'pixel_rgba.h' 
Fixes #3331
 2019-05-16 13:11:02 +02:00
Christian Prochaska
aa63628536 pistachio: fix compile error with GCC 8.3.0 
Fixes #3330
 2019-05-16 13:11:02 +02:00
Christian Prochaska
5a2e7a8d66 base-sel4: fix compile errors with GCC 8.3.0 
Fixes #3328
 2019-05-16 13:11:02 +02:00
Christian Prochaska
a41d46e193 base-pistachio: fix compile error with GCC 8.3.0 
Fixes #3327
 2019-05-16 13:11:02 +02:00
Christian Prochaska
940ba9ba95 base-nova: fix compile errors with GCC 8.3.0 
Fixes #3325
 2019-05-16 13:11:02 +02:00
Christian Prochaska
96627df4d4 base: fix cxx library compile errors with GCC 8.3.0 
Fixes #3322
 2019-05-16 13:11:02 +02:00
Christian Prochaska
47a2ad604c base: fix xml_node test compile error with GCC 8.3.0 
Fixes #3324
 2019-05-16 13:11:02 +02:00
Alexander Boettcher
70e0514a02 seoul: handle late timeouts 
either
- due to poor signal performance of base platform
- due to being to less prioritized
- due to schedule overload
- due to using time sources of different physical CPUs

Issue #3111
 2019-05-16 13:11:02 +02:00
Stefan Kalkowski
9135be8d5f fb_bench: disable automated test for imx7d_sabre 
This platform hasn't a fb_drv yet.
 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
c5fee20286 sel4: build elfloader on demand (fix #3329) 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
817eb4f23c okl4: create elfweaver tools on demand (ref #3329) 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
4fa34190de platform_drv: check acpi ability by platform_info 
Instead of retieving the information about the underlying platform from
the configuration, check the running kernel from the platform_info. This
commit removes the undocumented "acpi" config attribute.
 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
be053ed257 depot: add recipe for drivers_interactive-muen 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
ae3a6fe270 depot: add recipe for drivers_nic-muen pkg 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
da498af74e imx7d_sabre: disable automated network test 
Until there is no network driver present for this board it is needless to
let the tests fail each night.
 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
1e986fade8 os: name pbxa9 ps2_drv explicitely (ref #2190) 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
d9143f805e os: name rtc_drv unambigously (ref #2190) 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
2e68fae2ec os: name ahci_drv unambigously (ref #2190) 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
d18d2d0d9b os: name sd_card_drv unambigously (ref #2190) 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
4e6b571a36 os: rename uart_drv unambigously (ref #2190) 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
e6f83d4df2 depot: add recipe for drivers_interactive-rpi 
Ref #2190
 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
a623a66019 depot: add recipe for drivers_interactive-imx53_qsb 
Ref #2190
 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
35e73b1a2d os: name imx53 input_drv unambigously (ref #2190) 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
8d1cfce15e os: name fb_drv unambigously (ref #2190) 2019-05-16 13:11:01 +02:00
Stefan Kalkowski
4550056de7 run: use drivers_interactive-* pkg where possible 
Ref #2190
 2019-05-16 13:11:01 +02:00
Johannes Schlatow
0451d3bbed Simple cache performance test 
ref #3321
 2019-05-16 13:11:00 +02:00
Johannes Schlatow
9097c80269 zynq: improve nic driver error handling 
also be more verbose about detected errors

fixes #3320
 2019-05-16 13:11:00 +02:00
Christian Prochaska
739317a83f noux: support non-blocking pipes 
Needed for 'noux_gdb.run' with newer gdb versions.

Fixes #3319
 2019-05-16 13:11:00 +02:00
Alexander Boettcher
92510af9d4 foc: working VM session support for AMD 
- kernel: propagate cr0 to VMM on exit in nested paging case
- kernel: disable forceful VM exit on task switch
- vm_session: adjust to kernel changes

Issue #3111
 2019-05-16 13:11:00 +02:00
Alexander Boettcher
b3f288c035 seoul: avoid corrupted cr0 and GP on debug wrmsr 
- fix bug in instruction emulator on clts - mark cr0 as changed
- don't cause GP on debug control wrmsr

Issue #3111
 2019-05-16 13:11:00 +02:00
Emery Hemingway
761e312219 Vbox: compile VMM/VMMR3/VMReq permissive 
Ref #3289
 2019-05-16 13:11:00 +02:00
Christian Prochaska
c0f03a28e9 gdb_monitor: use terminal VFS plugin 
... instead of the deprecated libc terminal plugin.

Fixes #3318
 2019-05-16 13:11:00 +02:00
Emery Hemingway
d2ab699cd5 Remove Post_signal_hook from Entrypoint 
The Post_signal_hook mechanism has been completely replaced by
Io_progress_handler and can be removed.

Ref #3132
Fix #3302
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
5d4064fed4 depot: add recipe for drivers_nic-pbxa9 
Ref #3180
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
5182224c2b depot: add recipe for drivers_nic-zynq 
Ref #3180
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
8dd618d67f depot: add recipe for drivers_nic-imx6q_sabrelite 
Ref #3180
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
82693ba5b5 depot: add recipe for drivers_nic-imx53_qsb 
Ref #3180
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
ff031d792c depot: add recipe for driver_nic-rpi 
Ref #3180
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
5b5e8a9bd4 depot: add recipe for drivers_nic-linux 
Ref #3180
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
4fdbb1b1ad depot: add recipe for linux_nic_drv 
Ref #3180
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
bf5a631a14 dde_linux: name usb_drv unambigously 
* Make package buildable for ARM too
* Move usb library to src targets for explicitly named targets
* adapt remaining run-scripts to use the correctly named usb drivers

Ref #2190
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
d5104aca05 dde_linux: make rpi usb drivers kernel independent 
Ref #3180
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
d00bcac941 dde_linux: exclude non-base API from fec_nic_drv 
Ref #3180
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
5bc498c812 dde_linux: name FEC nic_drv unambigously 
Ref #2190
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
312499a1ef os: name lan9118 nic driver explicitly 
* it is not dependent on pbxa9 anymore, but configureable
* rename it to lan9118_nic_drv

Ref #2190
 2019-05-16 13:11:00 +02:00
Stefan Kalkowski
dcc28b65cb run: use driver_nic-* pkg where possible (fix #3180) 2019-05-16 13:10:06 +02:00
Stefan Kalkowski
b6d14d9960 dde_ipxe: name ipxe nic_drv unambigously 
Ref #2190
 2019-05-16 12:52:59 +02:00
Stefan Kalkowski
7c1e3c84ba os: name gpio_drv unambigously (ref #2190) 2019-05-16 12:52:59 +02:00
Stefan Kalkowski
911f4ada0a os: make platform_drv package ready for ARM 
* Make target binaries independent of board SPECS
* Name binaries of one architecture unambigously
* Extend include path to match board specifics
* Adapt run-scripts to use the right binary

Ref #2190
Ref #3180
 2019-05-16 12:52:59 +02:00
Stefan Kalkowski
c55663b923 os: extend regulator session api package 
Ref #2190
Ref #3180
 2019-05-16 12:52:59 +02:00
Stefan Kalkowski
cac0f44194 os: extend platform_session api package 
Ref #2190
Ref #3180
 2019-05-16 12:52:59 +02:00
Christian Prochaska
5bc9082fb7 noux_terminal_fs.run: increase noux capability quota 
Fixes #3309
 2019-05-16 12:52:59 +02:00
Emery Hemingway
62a4d1de0e Remove Nim tests 
Nim is now built independently of the Genode repository using an SDK and
Nimble.

Fix #3300
 2019-05-16 12:52:59 +02:00
Ben Larson
5f0aa16184 platform_drv/x86: fix report for many PCI devices 
When there are too many PCI devices, the Expanding_reporter regenerates
the report. However, this doesn't reset the BDF counter used to iterate
over the devices. This results in starting the new report after the PCI
device that triggered the report buffer overflow. This commit fixes the
issue by putting the BDF counter initialization inside the lambda
function used to generate the report.

Fixes #3317
 2019-05-16 12:52:59 +02:00
Christian Helmuth
ba51800b31 depot: update recipe hashes 2019-05-06 16:15:27 +02:00
Stefan Kalkowski
b495799d31 sel4: change ELF loader link address on imx7d 
Fix #3311
 2019-05-06 16:15:27 +02:00
Alexander Boettcher
c0a00019c0 foc/sel4: place vcpu thread on same cpu as ep 
Issue #3111
 2019-05-06 16:15:27 +02:00
Alexander Boettcher
85d98195d5 top: show priority and quota 
Issue #3192
 2019-05-06 16:15:27 +02:00
Emery Hemingway
3e848dff10 solo5: update bindings 
Update the Solo5 upstream to accomadate changes to the Block session and
stack protection support in the base library.

Ref #3275
Ref #3283
 2019-05-06 16:15:27 +02:00
Stefan Kalkowski
b85071174a base: wait a bit at the end of the smp test 
Fix #3306
 2019-05-06 16:15:27 +02:00
Stefan Kalkowski
918b9a9fa4 ieee754: disable test for muen (fix #3305) 2019-05-06 16:15:26 +02:00
Stefan Kalkowski
089cb0f536 foc: change core link address for imx7d_sabre 
Fix #3298
 2019-05-06 16:15:26 +02:00
Stefan Kalkowski
f42c21f16b platform_info: add kernel information (fix #3295) 2019-05-06 16:15:26 +02:00
Norman Feske
c38c80fd43 menu_view: midway re-targeting of geometry motion 
This patch fixes the corner case where an animated geometry changes its
destination mid-way while an animation is already in progress. The
'_trigger_animated_geometry' method used to back out early in this case,
which was intended as an optimization.

Fixes #3296
 2019-05-06 16:15:26 +02:00
Norman Feske
3f8dfa346c sculpt_manager: remove double name attributes 
The 'name' attribute was generated by both the call of 'gen_named_node'
and 'Hoverable_item::gen_button_attr'. Use only the former.
 2019-05-06 16:15:26 +02:00
Alexander Boettcher
05fa063068 vm_session: support to trace vCPU 
Issue #3111
 2019-05-06 16:15:26 +02:00
Christian Prochaska
dca144f2ee noux: fix noux_gdb run script 
Fixes #3287
 2019-05-06 16:15:26 +02:00
Christian Prochaska
4672fdb9f0 gdb_monitor.run: remove broken stderr redirection 
Fixes #3290
 2019-05-06 16:15:26 +02:00
Stefan Kalkowski
d6332f1767 zynq: make nic driver independent of hw kernel 
Use uncached ram dataspace for RX and TX communication buffers
instead of cache maintainance functions that are not part of the
Genode API.

Fix #3291
 2019-05-06 16:15:26 +02:00
Stefan Kalkowski
c767f6ccf1 os: explicit cache policy in Nic::Session_component 
To enable the use of uncached DMA buffers as RX and TX communication
buffers in between driver (service) and client, introduce a cache
attribute in the constructor of Nic::Session_component

Ref #3291
 2019-05-06 16:15:26 +02:00
Martin Stein
0b9916cae2 Timer::Connection: simplify curr_time 
With the new fact that plain time values are always 64 bit unsigned, the
timestamp type is never bigger than the plain time type. Therefore, a code path
in the curr_time interpolation that treated this condition is not neccessary
anymore.

Ref #3208
 2019-05-06 16:15:26 +02:00
Martin Stein
181c78d482 timeout: use uint64_t for all plain time values 
This enforces the use of unsigned 64-bit values for time in the duration type,
the timeout framework, the timer session, the userland timer-drivers, and the
alarm framework on all platforms. The commit also adapts the code that uses
these tools accross all basic repositories (base, base-*, os. gems, libports,
ports, dde_*) to use unsigned 64-bit values for time as well as far as this
does not imply profound modifications.

Fixes #3208
 2019-05-06 16:15:26 +02:00
Stefan Kalkowski
e072ee480b depot: add recipe for base-hw-imx7d_sabre 
Fix #3293
 2019-05-06 16:15:26 +02:00
Stefan Kalkowski
eb933af40a depot: add recipe for base-foc-imx7d_sabre 
Fix #3293
 2019-05-06 16:15:26 +02:00
Stefan Kalkowski
6d0a271308 depot: add recipe for base-sel4-imx7d_sabre 
Fix #3293
 2019-05-06 16:15:26 +02:00
Stefan Kalkowski
4bcd9169c0 sel4: add timer for imx7d_sabre 
Fix #3292
 2019-05-06 16:15:26 +02:00
Stefan Kalkowski
95b9e0a0e2 sel4: fix configuration for imx7d sabre 
Ref #3292
 2019-05-06 16:15:26 +02:00
Alexander Boettcher
cc64c43758 vm_session: adjustments to work with seoul vmm 
for foc, nova, sel4

Issue #3111
 2019-05-06 16:15:26 +02:00
Alexander Boettcher
8950de5a89 base: support whole addressable range in allocator 
Issue #3111
 2019-05-06 16:15:26 +02:00
Alexander Boettcher
8840ca96a9 seoul: use vm_session interface 
and remove any dependency on the NOVA kernel interface

Issue #3111
 2019-05-06 16:15:26 +02:00
Alexander Boettcher
35d46dca72 lib/launchpad: add VM session forwarding support 
Issue #3111
 2019-05-06 16:15:25 +02:00
Alexander Boettcher
393643515c base: extend attach of vm_session 
by offset, size, writeable and executable parameter

Issue #3111
 2019-05-06 16:15:25 +02:00
Norman Feske
17fda73ca1 block_tester: update to new block-client API 
- Added 'io_buffer' attribute, default is 4M
- Added 'batch' attribute, specifying the number of jobs used
  in parallel, default is 1 (sequential)
- Removed 'synchronous' attribute (use batch of 1 instead)
- Added 'copy' attribute (default "yes")
- Print number of signals ("triggered")

Issue #3283
 2019-05-06 16:15:25 +02:00
Norman Feske
784206c85c block_session: new client-side API 
This patch equips the 'Block::Connection' with a framework API for the
implementation of robust block-session clients that perform block I/O in
an asynchronous fashion.

An application-defined 'JOB' type, inherited from 'Connection::Job',
encapsulates the application's context information associated with a
block operation.

The lifecycle of the jobs is implemented by the 'Connection' and driven
by the application's invokation of 'Connection::update_jobs'. The
'update_jobs' mechanism takes three hook functions as arguments, which
implement the applications-defined policy for producing and consuming
data, and for the completion of jobs.

Issue #3283
 2019-05-06 16:15:25 +02:00
Norman Feske
976f67eee6 os: make 'Block::Operation' printable 
Issue #3283
 2019-05-06 16:15:25 +02:00
Norman Feske
e5f27b44b5 block_session: turn 'Connection' into template 
As a preparatory step for introducing the new block-client API, we have
to turn the 'Block::Connection' into a class template. The template
argument will be used to tie an application-defined job type to the
block connection.

Issue #3283
 2019-05-06 16:15:25 +02:00
Norman Feske
6c0dd9fe3b block_session: use types from block/request.h 
Issue #3283
 2019-05-06 16:15:25 +02:00
Stefan Kalkowski
7a45867841 vmm_arm: support for i.MX7 Dual SABRE board 
Fix #3285
 2019-05-06 16:15:25 +02:00
Stefan Kalkowski
0e94d7410d noux_tool_chain_auto: enable imx6 and imx7 boards 
Fix #3288
 2019-05-06 16:15:25 +02:00
Stefan Kalkowski
f074954d3d hw: use eret in hyp mode to switch mode 
Instead of using `cps` instruction, use an exception return
instruction to switch from `hyp` mode to `svc` mode.
Otherwise it causes unpredicted behaviour on ARM.

Fix #3284
 2019-05-06 16:15:25 +02:00
Norman Feske
c433f87000 os: batching of packet-stream source operations 
This patch adds support for manually triggering the wakeup of the packet
sink by the source. This way, a packet source becomes able to marshal
batches of submissions or unmarshal batches of acknowledgements before
yielding the control over to the sink.

Issue #3283
 2019-05-06 16:15:25 +02:00
Norman Feske
ea6b1c255e base: add const version of 'Fifo::Element::object' 2019-05-06 16:15:25 +02:00
Alexander Boettcher
35cf804471 base: free up all blocks on avl destruction 
Issue #3111

remove_range may deny to the job on memory pressure or insane ranges,
which ends up in an endless loop when the Avl allocator is in destruction.

Since the Avl gets destructed, solely the memory free up is of importance,
not the correct range adjustments during remove_range.
 2019-05-06 16:15:25 +02:00
Alexander Boettcher
526554741f vm_session: evaluate priority (nova) 
Issue #3111
 2019-05-06 16:15:25 +02:00
Alexander Boettcher
d2229ab381 vm_session: evaluate priority 
Issue #3111
 2019-05-06 16:15:25 +02:00
Alexander Boettcher
d459432c45 init: scale priority of vm_session 
Issue #3111
 2019-05-06 16:15:25 +02:00
Alexander Boettcher
450c8dc149 vm_session: track dataspaces used by attach 
Track the dataspaces used by attach and add handling of flushing VM space
when dataspace gets destroyed (not triggered via the vm_session interface).

Issue #3111
 2019-05-06 16:15:25 +02:00
Alexander Boettcher
169c51d50d base: remove error message in slab block 
Issue #3111
 2019-05-06 16:15:25 +02:00
Alexander Boettcher
d1f37e66eb base: fix _sum_in_range in avl structure 
Handles corner case when addr + size becomes exactly 0. Before the commit
the function returned that sum is not part of the range, which is wrong.

Issue #3111
 2019-05-06 16:15:24 +02:00
Alexander Boettcher
812149ed29 core: add Region_map_detach interface 
Issue #3111
 2019-05-06 16:15:24 +02:00
Norman Feske
593e971121 block_session: SYNC and TRIM as async operations 
This patch removes the blocking Block::Session::sync RPC function and
adds the asynchronous operations SYNC and TRIM to the block session's
packet-stream interface.

Even though the patch adjusts all block components to the interface
change, the components keep the former blocking handling of sync
internally for now because of the design of the 'Block::Driver'
interface. This old interface is not worth changing. We should instead
migrate the block servers step by step to the new
'Block::Request_stream' API.

Fixes #3274
 2019-05-06 16:15:24 +02:00
Norman Feske
e058adc4fe block_session: add client-defined request tag 
The new request tag allows a block-session client to uniquely correlate
acknowledgements with outstanding requests. Until now, this was possible
for read and write operations by taking the value of the request's
packet-stream offset. However, SYNC and TRIM requests do not carry any
packet-stream payload and thereby lack meaningful offset values. By
introducing the notion of a 'tag', we can support multiple outstanding
requests of any type and don't need to overload the meaning of the
'offset' value.

Issue #3274
 2019-05-06 16:15:24 +02:00
Norman Feske
a39474a245 block-request stream: split Operation from Request 
This patch splits the 'Request' definition into smaller types that are
suitable for the client-side API too.

The new 'Operation' type comprises the block operation's type (opcode)
and the operation's arguments (block number, block count).
The former 'Request::operation_defined' is now 'Operation::valid'.
The 'Request' aggregates an 'Operation', which changes its object
layout.

Note that this commit relaxes the bit-precise definition of 'Request' to
facilitate the use of 'unsigned long' where appropriate, in particular
for the request tag (which should correspond to an 'Id_space::Id'). The
originally bit-precise definition was pursued to allow the sharing of
the 'Request' type between SPARK and C++ code. However, it turns out
that defining a native type in each language and a (set of) converting
constructors is a more natural approach.

Issue #3283
 2019-05-06 16:15:24 +02:00
Christian Prochaska
ed246d152b tool_chain: build in <genode-dir>/build/tool_chain-<version> 
Fixes #2608
 2019-05-06 16:15:24 +02:00
Stefan Kalkowski
7f965dad29 vmm_arm: do not use uart driver in automated test 
Fix #3278
 2019-05-06 16:15:24 +02:00
Stefan Kalkowski
0a470e66f2 vmm_arm: uart device model needs to take all chars 
When receiving a terminal signal, the uart device model has to
take all characters out of the stream. Otherwise, characters
might never arrive at the VM. This was not recognized before,
because it was used with a quite slow UART only, which obviously
never achieved to send more characters at once.

Ref #3278
 2019-05-06 16:15:24 +02:00
Stefan Kalkowski
94881e757a test-terminal_expect_send: expect-like helper tool 
A small terminal-client tool, which expects a specific line(-start),
and then sends a specified line to the other side. Optionally,
it prints all received lines to its LOG service.

Ref #3278
 2019-05-06 16:15:24 +02:00
Alexander Boettcher
26796c8f7a nova: update kernel branch 
- sorting of initializer by Christian Prochaska - issue #3253
- leak of FPU register state reported by Julian Stecklina - thanks !
- fix page table synchronization bug - by Julian Stecklina
- add a second dynamic buddy if the memory requirements could not be fulfilled
  by the first dynamic buddy
- disable vtlb peek if EPT/NPT is enabled
 2019-05-06 16:15:16 +02:00
Norman Feske
51f51c18af block/request_stream: rename wakeup_client 
This patch renames 'wakeup_client' to 'wakeup_client_if_needed' to
clarify that the method triggers signals only when needed, not on every
call.

The name 'wakeup_client' is prone to misguide users to call the function
conditionally as an optimization, thereby complicating the code, but to
no effect.

Fixes #3279
 2019-05-03 13:53:12 +02:00
Norman Feske
34fd5f626a test/block_request_stream: private Request_stream 
Avoid public inheritance to make the code simpler to understand.

Issue #3279
 2019-05-03 13:53:12 +02:00
Norman Feske
bbe3ee8dc5 block_session: server-defined payload alignment 
This patch replaces the formerly fixed 2 KiB data alignment within the
packet-stream buffer by a server-defined alignment. This has two
benefits.

First, when using block servers that provide small block sizes like 512
bytes, we avoid fragmenting the packet-stream buffer, which occurs when
aligning 512-byte requests at 2 KiB boundaries. This reduces meta data
costs for the packet-stream allocator and also allows fitting more
requests into the buffer.

Second, block drivers with alignment constraints dictated by the
hardware can now pass those constraints to the client, thereby easing
the use of zero-copy DMA directly into the packet stream.

The alignment is determined by the Block::Session_client at construction
time and applied by the Block::Session_client::alloc_packet method.
Block-session clients should always use this method, not the 'alloc_packet'
method of the packet stream (tx source) directly. The latter merely
applies a default alignment of 2 KiB.

At the server side, the alignment is automatically checked by
block/component.h (old API) and block/request_stream.h (new API).

Issue #3274
 2019-05-03 13:53:12 +02:00
Norman Feske
2208220c12 block session: remove Block::Session::Operations 
This patch modernizes the 'Block::Session::info' interface. Instead of
using out parameters, the 'init' RPC function returns a compound 'Info'
object now. The rather complicated 'Operations' struct is replaced by
a 'writeable' attribute in the 'Info' object.

Fixes #3275
 2019-05-03 13:53:12 +02:00
Norman Feske
10c567daee base: add 'aligned' function to util/misc_math.h 
This function simplifies the sanity checking of values that are expected
to be aligned, e.g., data offsets within packet streams.
 2019-05-03 13:31:39 +02:00
Christian Helmuth
6ea1179145 depot: update recipe hashes 2019-04-09 12:30:35 +02:00
Christian Helmuth
71a8dcca08 tool/depot: use nullglob on stale archive removal 
nullglob is needed to get an empty shell glob in case no matching binary
archive (path) exists. Otherwise, the original glob string including the
* is returned and used in the for loop.
 2019-04-09 12:30:35 +02:00
Stefan Kalkowski
a147bdf406 hw: silent warning about unknown signal context 
Triggering of an invalidated signal seems to be no real exception,
but something that occurs regularily. Therefore, the kernel warning
is of no use to developers anymore.

Ref #3277
 2019-04-09 12:30:35 +02:00
Stefan Kalkowski
0ca199f89a hw: replace lock-safe log variants in kernel 
As far as possible remove usage of warning/error/log in the kernel,
otherwise the kernel context might try to take a lock hold by a core
thread, which results in a syscall to block.

Fix #3277
 2019-04-09 12:30:35 +02:00
Norman Feske
9e238d624e extract: discharge dependency from timer session 
This patch removes the reliance of the extract tool from the libc's
behavior regarding the access of time and timing.

The extract tool is not expected to need time. However, unfortunately,
libarchive calls the 'time' function unconditionally. By adding a
dummy for 'time', we avoid bothering the libc, which would otherwise
need to obtain a time source.

Issue #3204
 2019-04-09 12:30:35 +02:00
Norman Feske
6d11591d83 test-spark: rename number_of_bits -> object_size 
In the original version, I used 'number_of_bits' because Ada's 'Size
returns the size in bits, not bytes. But the values (for objects) are
always a multiple of 8. On the C++ side, performing size checks at the
granularity of bits is just awkward. The term 'object size' is more
natural.
 2019-04-09 12:30:35 +02:00
Sebastian Sumpf
5937492e93 base-foc: map debug cap through the correct index 
* necessary for enabling the kernel debugging facilities on base-foc
  (outstring, fiasco_tbuf_log_3val and friends)
* disabled by default

related to issue #3260
 2019-04-09 12:30:35 +02:00
Norman Feske
ec70081258 test-ldso: refine log pattern 
By adding a wildcard at the beginning of (expected) error messages, the
test.run tool becomes able to match the lines (ignoring the characters
of the color escape sequence).
 2019-04-09 12:30:35 +02:00
Christian Helmuth
9948a77558 spark: provide rcheck symbols in ABI 2019-04-09 12:30:35 +02:00
Norman Feske
724761565d libsparkcrypto: don't override CUSTOM_ADA_OPT 
CUSTOM_ADA_OPT must not be defined outside <build-dir>/etc/tools.conf
 2019-04-09 12:30:35 +02:00
Martin Stein
9d236e8e03 block/request: fixed enum values 2019-04-09 12:30:35 +02:00
Martin Stein
a9851aca55 block/request: add member 'tag' 
Issue #3274
 2019-04-09 12:30:35 +02:00
Norman Feske
a2743dcaeb Library for the AES-CBC en/decryption of 4K blocks 
The 'aes_cbc_4k' library is simple wrapper around libsparkcrypto to
serve as a backend for storage encryption. It operates on data chunks of
4 KiB and uses AES-CBC while incorporating the block number and the
private key as salt values.
 2019-04-09 12:30:35 +02:00
Pirmin Duss
8b4e2a21e4 e2fsprogs: disarm the mount-point check 
Discovered while creating an ext2 file system on an sd_card (no
partitions, imx6), which failed with an error message that stated the
device is in use. Genode's libc/posix layer has no notion of mount
points.

Fixes #3271
 2019-04-09 12:30:35 +02:00
Pirmin Duss
642c2ab4bc sd_card_drv: increase multiblock transfer timeout 
Disconnecting a client and connecting an other to the sd_card_drv
on imx6 results in a "Completion host signal timed out" error in
the newly connected client.

Fixes #3272
 2019-04-09 12:30:35 +02:00
Alexander Boettcher
93ba870b2d top: show execution times per SC and per EC 
Issue #3192
 2019-04-09 12:30:35 +02:00
Alexander Boettcher
36fac8e22b run/test: support TRACE service 
Issue #3192
 2019-04-09 12:30:35 +02:00
Alexander Boettcher
ae16edf1d6 trace: support more facets of execution time 
- execution time per thread context
- execution time per scheduling context
- quantum and priority

Issue #3192
 2019-04-09 12:30:34 +02:00
Emery Hemingway
66f49e6c42 VFS lwIP: support opening a handle on the root directory 
The "nameserver" file cannot be opened through a VFS File_system client
if the plugin does not support opening the parent directory of
"/nameserver", which would be "/".

Ref #3269
 2019-04-09 12:30:34 +02:00
Emery Hemingway
a635873568 VFS: Replace global response handlers with local handlers 
Replace the I/O response handler that is passed to the VFS at
construction with an object that is dynamically attached to handles.
This object shall also accept read-ready notifications, and plugins are
encouraged to keep handles awaiting ready-ready notifications separate
from handles that await I/O progress.

Replace the use of handle lists in plugins with handle queues, this
makes the code easier to understand and the ordering of notifications to
the application more explicit.

These changes replace the use of the Post_signal_hook from all VFS
plugins, applications must assume that read-ready and I/O notifications
occur during I/O signal dispatch and use an Io_progress_handler at its
entrypoints to defer response until after signal dispatching.

Fix #3257
 2019-04-09 12:30:34 +02:00
Emery Hemingway
e2ff776b35 Print error locations in test-libc_connect 
Ref #3257
 2019-04-09 12:30:34 +02:00
Alexander Boettcher
356526d610 foc: increase USER_BASE_CAP 
due to THREAD_AREA_SLOT increase by

foc/x86: implement vm_session interface

Add additional static assertion check to detect misconfiguration earlier.
 2019-04-09 12:30:34 +02:00
Josef Söntgen
38a10c92d3 pthread: initialize static rwlocks/conds 
Make sure that the rwlock is allocated before a lock operation is
performed. This case occurs if a static rwlock was create by using
PTHREAD_RWLOCK_INITIALIZER. Same goes for PTHREAD_CONDS_INITIALIZER.

Fixes #3262.
 2019-04-09 12:30:34 +02:00
Josef Söntgen
3c8714ed5a dde_bsd: make HDMI/DP HDA check stricter 
Since QEMU might put the audio device at 00:03.0, also check if the
vendor is Intel. Hopefully we do not render HDA on real machines
useless with this changes (so far I have not encountered one).

Fixes #3263.
 2019-04-09 12:30:34 +02:00
Josef Söntgen
3362216b66 os: clear remaining samples in Audio packet 
... instead of potentially producing out-of-bounds write in the source
buffer.

Fixes #3264.
 2019-04-09 12:30:34 +02:00
Josef Söntgen
5ceba11982 os: make Audio_out content filler data arg const 
Issue #3264.
 2019-04-09 12:30:34 +02:00
Josef Söntgen
b9e12d7c23 libc: always set O_RDWR in fcntl on a socket 
For better or worse we have no proper way of handling this right now
but contrib libraries, e.g. glib, use it to determine if they can use
the underlying fd.

Fxies #3265.
 2019-04-09 12:30:34 +02:00
Josef Söntgen
5af0162b3f libc: fix storing TIOCGWINSZ result 
Fixes #3266.
 2019-04-09 12:30:34 +02:00
Norman Feske
a3411c8e96 run: add --depot-auto-update feature 
If enabled, this option triggers the automated management of depot
content according to the needs of a run script.

Fixes #3270
 2019-04-09 12:30:34 +02:00
Norman Feske
d027f12764 depot: update recipe hashes 2019-04-02 09:36:39 +02:00
Norman Feske
4b0b4928f5 tool/depot: rm binary archives if versions collide 
This patch handles version collisions of binary archives. If a binary
archive for an (automatically) updated src-archive version already
exists in the depot, the extract tool removes the binary archive because
its existing content may stem from another src content (a version from a
different topic branch). This ensures that the new version is always
built, not skipped, when 'UPDATE_VERSIONS=1 REBUILD=' is specified.

Fixes #3267
 2019-04-01 19:33:52 +02:00
Alexander Boettcher
ca32c11f4f nitpicker: send mode change on domain update 
Fixes #3259
 2019-04-01 19:33:52 +02:00
Christian Prochaska
38de57be76 tool_chain: make sudo command configurable 
Fixes #3255
 2019-04-01 19:33:52 +02:00
Norman Feske
2b8ea7845f smartcard.run: increase caps for usb_drv 
This is needed for running the test on seL4.
 2019-04-01 19:33:52 +02:00
Alexander Boettcher
483de40c96 base: remove message in sliced_heap.alloc() 
Issue #3111
 2019-04-01 19:33:52 +02:00
Norman Feske
a224d828f6 test.run: integrate content from raw archives 
This patch enables the test.run tool to execute the test-init package.

Issue #3256
 2019-04-01 19:33:52 +02:00
Alexander Boettcher
ef130a3bf9 sel4/x86: implement vm_session interface 
Issue #3111

- enable vt-x in kernel configuration

Kernel patches:

- add unrestricted guest support
- avoid kernel boot failure when vt-x is not available
- avoid nullpointer in kernel when vcpu is not fully setup
- avoid vcpu scheduling bug which causes starvation on same/below prio level
- save efer register correctly from guest
 2019-04-01 19:33:52 +02:00
Alexander Boettcher
b75307b070 foc/x86: implement vm_session interface 
Issue #3111
 2019-04-01 19:33:51 +02:00
Alexander Boettcher
3a1bbfad28 nova: implement vm_session interface 
Issue #3111
 2019-04-01 19:33:51 +02:00
Alexander Boettcher
ae31b761f4 hw: move vmm.run to vmm_arm.run 
Issue #3111
 2019-04-01 19:33:51 +02:00
Alexander Boettcher
f927684eb8 os: add vmm_x86 interface test 
Issue #3111
 2019-04-01 19:33:51 +02:00
Alexander Boettcher
5fa91a1bcc base: add x86 vm_state 
Issue #3111
 2019-04-01 19:33:51 +02:00
Norman Feske
aeb7e7cd7c sculpt manager: prevent double destruction 
The '_currently_constructed' pointer caches the information about which
'Launched_child' is currently configured in the menu. When discarding
the runtime (e.g., when un-using a file system) at this point, this
cached pointer was not invalidated while all 'Launched_child' objects
would be freed (including the currently constructed one). On the next
attempt to construct a new child, the sculpt manager attempted to
destruct the 'Launched_child' referred by the (now outdated)
'_currently_constructed' again.

Fixes #3240
 2019-04-01 19:33:51 +02:00
Stefan Kalkowski
935abb55b7 hw: move src/lib/hw header to src/include/hw 
* Remove bad style of using `src/lib` as include path

Fix #3244
 2019-04-01 19:33:51 +02:00
Alexander Boettcher
00953e39f4 vbox: allocate REM memory dense 
Fixes #2116
 2019-04-01 19:33:50 +02:00
Alexander Boettcher
3b36673e03 seoul: use nitpicker session 
Issue #3196
 2019-04-01 19:33:50 +02:00
Alexander Boettcher
3b36542212 vbox5: use nitpicker session 
Issue #3196
 2019-04-01 19:33:50 +02:00
Josef Söntgen
af97a581e3 bsd_audio_drv: fix runtime for Sculpt CE 
Fixes #3250.
 2019-04-01 19:33:49 +02:00
Stefan Kalkowski
baf815d099 hw: add support for i.MX7 Dual SABRE board 
Fix #3251
 2019-04-01 19:33:49 +02:00
Stefan Kalkowski
8f28f884ee hw: name vm_state header explicitely 
Ref #3251
 2019-04-01 19:33:49 +02:00
Stefan Kalkowski
4c492a3be7 imx7d_sabre: add board for sel4 and foc 
Ref #3251
 2019-04-01 19:33:49 +02:00
Stefan Kalkowski
3d727a403f sel4: update tools to currently used version 9.0.x 
Until now, Genode referenced a fork of the outdated elfloader-tool
to enable bootstrapping of sel4 on ARM platforms. Because the
elfloader is inherently dependent on the used platforms newer
ARM versions supported by the kernel could not be loaded by the
outdated elfloader. This commit uses a fresh fork of the nowadays
used sel4_tools repository.

Ref #3251
 2019-04-01 19:33:49 +02:00
Alexander Boettcher
c19c5ed0a4 wm: fix destroy with invalid handle 
Issue #3232
 2019-04-01 19:33:49 +02:00
Roman Iten
1caa2ff3cb prepare_port: fix race condition in safety rule 
The safe guard Git repository in $(CONTRIB_DIR) may only be initialized
once, regardless of the number of ports to be prepared.

Issue #3235
 2019-04-01 19:33:49 +02:00
Norman Feske
ed33cf08c7 test-fs_tool: make output deterministic 
Issue #3238
 2019-04-01 19:33:48 +02:00
Norman Feske
a67b23eb2d test.run: handle '*' wildcards 2019-04-01 19:33:48 +02:00
Norman Feske
01cff3ce84 sculpt: inform user about incomplete pkg install 
This commit handles the corner case where a package could be installed
successfully but the package's runtime definition is inconsistent with
the content delivered by the package's dependencies, i.e., the <content>
of the runtime file lists ROM modules that do not exist.

With this patch, the '+' menu shows the message "installed but
incomplete" whenever a package is in such a state.

Issue #3241
 2019-04-01 19:33:48 +02:00
Stefan Kalkowski
d6376f8188 rustc: select ARM target correctly (fix #3087) 2019-04-01 19:33:48 +02:00
Christian Helmuth
f0a3377247 Remove obsolete symbols from symbols/ld ABI 2019-04-01 19:33:48 +02:00
Christian Helmuth
9debc0fa4b Mention sel4 kernel as option for imx6q_sabrelite 2019-04-01 19:33:48 +02:00
Christian Helmuth
37a93f53c4 Provide Genode::cache_coherent() consistently 
It got lost on base-foc and also was not put into Genode namespace in
the default implementation.
 2019-04-01 19:33:48 +02:00
Alexander Boettcher
c3fd572a52 init: add report/buffer attribute to xsd 
Issue #3216
 2019-04-01 19:33:48 +02:00
Martin Stein
bcc049ceeb nic_router: manipulate port-forwarding port 
The new configuration attributes <tcp-forward to_port="123" /> and
<udp-forward to_port="123" /> enable manipulating the destination port of
port-forwarded packets.

Fixes #3237
 2019-04-01 19:33:48 +02:00
Martin Stein
3a59f11708 nic_router/nic_bridge: free MAC addresses 
Free a MAC address of a session as soon as the session gets destructed.
Adds also according tests to the autopilot list.

Fixes #2470
 2019-04-01 19:33:48 +02:00
Alexander Boettcher
bafe3c3fc3 vbox5: check memory configuration stricter 
The required memory of the VMM depends on the VM memory configuration (vbox)
and the available screen resolution. Check for insane configurations in the
beginning to avoid late surprises.

Issue #3216
 2019-04-01 19:33:47 +02:00
Alexander Boettcher
ce1f245952 vbox5: free bitmap source during resize 
Fixes #3216
 2019-04-01 19:33:47 +02:00
Emery Hemingway
800b4e44b1 Reduce Vfs::Vfs_handle::Context to empty struct type 
The "Vfs::Vfs_handle" type should not contain any public members that
can be initialized by the VFS internally and by the application, so
remove inheritance from the "Genode::list::Element" class. The VFS
plugins must instead use lists of "Vfs::Vfs_handle" sub-classes, the
lifetime of which are always managed by the plugin.

Ref #3036
 2019-04-01 19:33:47 +02:00
Emery Hemingway
260fc30be3 lxip: consolidate lx timers and timeouts 
Use a single timeout scheduler passed during lxip instantiation for
both timers and 'schedule_timeout' facilites rather than instantiate
two timer sessions and signal handlers. This reduces the library's
capability cost and initialization time.

Fix #2961
 2019-04-01 19:33:47 +02:00
Stefan Kalkowski
3725e91603 hw: implement power-saving kernel lock for ARM smp 
Thanks to former work of Martin Stein this commit finally incorporates a
non-spinning kernel lock on multi-core ARM platforms.

Fix #1313
 2019-04-01 19:33:47 +02:00
Martin Stein
b04a70177b base & arm: simplify and optimize cmpxchg 
The old inline assembly provided two output operands that were afterwards
combined for a return value. However, the second output operand isn't
necessary when using the "Acquiring a Lock" example-code of the ARM manual
"Barrier Litmus Tests and Cookbook". This saves two logical operations
per acquisition try. Additionally better documentation is now provided.

Fixes #1292
 2019-04-01 19:33:47 +02:00
Stefan Kalkowski
e9b3569f44 hw: remove overall cache maintainance from core 
This functionality is only needed in bootstrap now that kernel and
userland share the same address-space.

Fix #2699
 2019-04-01 19:33:46 +02:00
Christian Helmuth
9bbc91bb52 Add missing newline to pipe test 
Original patch by Ben Larson <hungryninja101@gmail.com>.

Fixes #3055
 2019-04-01 19:33:46 +02:00
Emery Hemingway
5efa6d5273 Stack smashing test 
A test to check if -fstack-protector can be enabled and is effective.

Ref #3066
 2019-04-01 19:33:46 +02:00
Emery Hemingway
6819c43a05 Add stack protector storage to base library 
This patch adds the items necessary for building Genode components with
stack protection enabled, but it is not initialized at runtime. They are
provided at the moment as a convenience and do not implement a security
feature.

Fix #3066
 2019-04-01 19:33:46 +02:00
Tomasz Gajewski
35c17ced72 Trivial doc fix for Register_set_plain_access::read 
Fix #3229
 2019-04-01 19:33:46 +02:00
Sebastian Sumpf
1705aab290 seoul: PS2 mouse wheel support 
fixes #3234
 2019-04-01 19:33:46 +02:00
Christian Prochaska
9f50b8897a Update Arora default bookmarks 
Fixes #3198
 2019-04-01 19:33:46 +02:00
Pirmin Duss
1c746df82c fetchurl: update README 
Fix the configuration example in the README.

Fix #3213
 2019-04-01 19:33:46 +02:00
Pirmin Duss
8456ffa231 bomb: prevent Timer-session creation errors 
Fix #3212
 2019-04-01 19:33:46 +02:00
Christian Helmuth
ebe71a12ed depot: update recipe hashes 2019-03-19 11:12:36 +01:00
Norman Feske
0872406e3c News item for Sculpt CE 2019-03-19 11:00:01 +01:00
Norman Feske
cffe847778 Update documentation for Sculpt CE 2019-03-19 11:00:01 +01:00
Norman Feske
4353c9e0f1 os: make Block::Request_stream::Ack noncopyable 
This is a safeguard against accidentally taking the 'Ack' interface as a
value instead of a reference.

The 'Payload' interface should also not be copied (and potentially
stored) because it contains a pointer.
 2019-03-19 11:00:01 +01:00
Norman Feske
6763ac29d9 depot_download_manager: fetchurl watchdog 
If fetchurl the amount of bytes downloaded by fetchurl does not change
for 5 seconds, respawn it.
 2019-03-19 11:00:01 +01:00
Norman Feske
4f99224255 ram_fs: increase max file size on 64 bit to 8 GiB 
Fixes #2315
 2019-03-18 15:57:00 +01:00
Norman Feske
c76f7c0c2a sculpt: allow graph to overlap the log 2019-03-18 15:56:59 +01:00
Norman Feske
806dc5de36 window layouter: improved maximize handling 
It turns out that the commit "window layouter: allow floating apps to
resize" interplays badly with the interactive toggling of the maximize
state of windows. In contrast to the window geometry and stacking, which
is always updated through the rules-feedback mechanism, interactive
changes of the maximize state omitted this loop and instead took a local
shortcut. Because of this shortcut, the maximized geometry eventually
ended up as window size in the window's assign rule. So unmaximizing the
window failed to revert the geometry to its original state.

This patch removes this inconsistency. The maximize state adheres to the
official chain of commands through the rules mechanism now. The state is
now maintained internally without affecting the window's geometry and is
evaluated while generating the window layout only.

As a minor loosely related improvement, this patch prevents the
highlighting of resize handles for non-resizable windows.

Issue #3200
 2019-03-18 15:56:59 +01:00
Alexander Boettcher
104def8e51 acpica: update to sculpt ce 2019-03-18 15:56:59 +01:00
Alexander Boettcher
dc4513f965 sculpt_manager: support system state ROM 2019-03-18 15:56:59 +01:00
Stefan Kalkowski
822a6e7c5f hw_riscv: strictly separate machine and syscall ids 
Fix #3230
 2019-03-18 15:56:59 +01:00
Norman Feske
91197804ac nitpicker: fix destroy with invalid handle 
This patch reworks the 'Session_component::destroy' to cope become
robust against a client-provided invalid view handle. The code did not
consider that 'Handle_registry::has_handle' may throw.

Thanks to Alexander Boettcher for reporting and the initial fix.

Fixes #3232
 2019-03-18 15:56:59 +01:00
Martin Stein
ff2516deb2 hw: fix documentation of Kernel::update_pd 
Add note that the calling thread must not be destroyed while in the syscall.

Fixes #1253
 2019-03-18 15:56:59 +01:00
Martin Stein
a74ae75680 run/nic_router foc x86_64: raise test timeout 
Fixes #2908
 2019-03-18 15:56:59 +01:00
Norman Feske
9438caa6a3 base: never throw in Xml_node::for_each_sub_node 
Fixes #3231
 2019-03-18 15:56:59 +01:00
Norman Feske
d75c5f6722 window layouter: fix glitch after window resize 
This patch improves the transition from an interactive window geometry
change (dragging a window element) to the point where the resulting
new layout rules come into effect. During this short time, no resize
request must be issued because such a resize request would be based on
stale rules.

Fixes #3227
 2019-03-18 15:56:59 +01:00
Norman Feske
baf46db287 Adjust wm -> focus label 
This is a follow-up commit to "Update <provides> info in pkg runtimes",
which adapts the users of the wm pkg to the changed label of the "focus"
nitpicker session.
 2019-03-18 15:56:59 +01:00
Christian Prochaska
67fd77d10a base: catch 'Ipc_error' in 'Expanding_parent_client::exit()' 
If a component is being destroyed just before it calls `exit()` at its
parent, the `exit()` call causes an `Ipc_error` exception, which leads to
an `abort()` loop with repeated error messages, because `abort()` calls
`exit()` too. Catching the exception in `Expanding_parent_client::exit()`
avoids this problem.

Fixes #3228
 2019-03-18 15:56:59 +01:00
Sebastian Sumpf
98e2f91036 qemu-usb: fix multiple device support 
bus or device should be different (not both) when comparing a device to
already present devices. Because of this the second USB device was
marked as existent when the bus matched or the bus did not but the
device number.
 2019-03-18 15:56:59 +01:00
Sebastian Sumpf
326deb14fe sculpt: add global KEY_SCREEN for themed_wm 
fixes switching windows by global key
 2019-03-18 15:56:59 +01:00
Norman Feske
591e9457e6 window layouter: allow floating apps to resize 
This patch gives applications the ability to control the size of their
window whenever the window is floating, not tiled or maximized. See the
comment in the code for the rationale.

Fixes #3200
 2019-03-18 15:56:59 +01:00
Norman Feske
1f3f7282f3 depot_download: limit rate of update-state reports 
The default rate of 100 ms keeps Sculpt too busy because the menu that
displays the percentage values is drawn completely on each update.
Limiting the rate to 1/4 seconds relieves the effect.
 2019-03-18 15:56:59 +01:00
Norman Feske
1ead0ea3a7 sculpt: allow for removal of index files 
By clicking on a yellow checkbox in the depot selection dialog, the
corresponding index files are removed. This way, index files can
be update by removing and downloading them again.

This patch also filters out sculpt-managed components from the graph to
avoid erratic graph-position changes while the '+' menu is open.

Fixes #3193
 2019-03-18 15:56:59 +01:00
Norman Feske
f3d4ee4d4c init: always abandon child on version change 
Without this patch, a version change of an already exited child would not
trigger the restart of the child because the version is evaluated as
late as the child configuration, but only if the child has not exited
yet.

This patch evaluates the version at the earlier stage where the identity
of the child (its name) is checked against the new configuration.

Fixes #3226
 2019-03-18 15:56:58 +01:00
Josef Söntgen
8ccfe4361c part_block: properly indent code 
Issue #3223.
 2019-03-18 15:56:58 +01:00
Josef Söntgen
cb2ebd0bf7 part_block: fix handling of chained EBR's 
Fixes #3223.
 2019-03-18 15:56:58 +01:00
Josef Söntgen
1c2dabc6d7 test-part_block_mbr: update disk image 
The disk image contains an extended partition table now that would
have been parsed incorrectly in the past. While there remove the unused
gpt BLOB (test-part_block_gpt has its own raw archive).

Issue #3223.
 2019-03-18 15:56:58 +01:00
Norman Feske
cac3f3b24e gems: initial version of fs_tool 
The fs_tool component performs file operations according to its
configuration. This initial version implements only the operation
<remove-file> as needed for Sculpt CE.

Issue #3222
Issue #3193
 2019-03-18 15:56:58 +01:00
Norman Feske
73f79ed5b1 gems/vfs.h: add Directory::unlink 
This is needed to accommodate the fs_tool component.

Issue #3222
 2019-03-18 15:56:25 +01:00
Norman Feske
4264de05f9 sculpt: avoid duplicated diagnostic messages 
Related to #3190
 2019-03-18 15:56:25 +01:00
Norman Feske
6528f50bfc sculpt: NIC ready only with IP other than 0.0.0.0 
This patch refines the criterion of when the networking is considered as
ready to use. Until now, any IP reported by the NIC router was taken as
an indicator for connectivity. But as the NIC router reports an IP
0.0.0.0/32 when no network cable is plugged at the uplink, the condition
was too loose.
 2019-03-18 15:56:25 +01:00
Norman Feske
94e2a64f60 Sculpt: defer config dialog until pkg is complete 
Fixes #3189
 2019-03-18 15:56:25 +01:00
Stefan Kalkowski
330692350e hw: introduce non-blocking signal checking 
* Introduces pending_signal syscall to check for new signals for the
  calling thread without blocking
* Implements pending_signal in the base-library specific for hw to use the
  new syscall

Fix #3217
 2019-03-18 15:56:24 +01:00
Stefan Kalkowski
3e70b53165 vfs_lxip: fix return code in blocking accept case 
Ref #3217
 2019-03-18 15:56:24 +01:00
Norman Feske
00eb100114 sculpt: gracefully handle missing depot-user info 
This patch improves the error handling of depot-download manager for the
case where a download is requested but the corresponding software
provider information is absent from the depot. Without this patch, the
update mechanism would get stuck in the failed depot-query step and
won't attempt to perform subsequent download jobs.

Fixes #3224
 2019-03-18 15:56:24 +01:00
Norman Feske
41575ad60d sculpt: update menu when folding condition changed 
The storage dialog is folded when activating the runtime view (e.g., by
clicking on the Genode Logo). This should happen immediately as response
of the mouse click.
 2019-03-18 15:56:24 +01:00
Norman Feske
122c404883 menu view: cleanly separate update, layout phases 
This patch improves the separation of the update and layout phases to
avoid superfluous geometry animations of its child widgets. Prior this
patch, 'Widget::geometry' was called in both phases, potentially
triggering geometry animations with intermediate values at the update
phase.

Related to issue #3221
 2019-03-18 15:56:24 +01:00
Norman Feske
ba5de21db4 menu view: fade dependency changes 
This patch applies fading when the visibility of dependencies changes.

Issue #3221
 2019-03-18 15:56:24 +01:00
Norman Feske
b3219fc3a4 sculpt: fine-tune motion of graph positioning 
By using 30 motion steps, the positioning of the graph window fits
better with the geometry animations of the graph nodes inside the
window.
 2019-03-18 15:56:24 +01:00
Norman Feske
905da4b0cc menu_view: fade between button styles 
The button widget already supported an animated transition between
hovered and unhovered states. This patch generalizes the mechanism to
allow animated transitions between arbitrary button states, including
style changes.

This way, the fade-out of non-TCB components in Sculpt CE happens not
abruptly but smooth.

Fixes #3221
 2019-03-18 15:56:24 +01:00
Norman Feske
116bfab449 icon_painter.h: customizable pixel transfer 
This patch makes the application of color/alpha from the icon's texture
to the target surface customizable by replacing the formerly built-in
'_transfer_pixel' function by calls to the new 'Pixel_rgba::transfer'
interface.

Issue #3221
 2019-03-18 15:56:24 +01:00
Norman Feske
ebd9f36b0d os: Pixel_rgba::transfer interface 
The new 'transfer' function interface defines how pixel/alpha values
sampled from texture are applied to a destination pixel, similar to the
role of a fragment shader in GPU-based rendering. The transfer function
can be customized by defining custom pixel types, which may be (but
don't need to be) derived from 'Pixel_rgba'.

Issue #3221
 2019-03-18 15:56:24 +01:00
Norman Feske
b00e1f4e4b menu view: position widgets initially at (0, 0) 
The default 'Rect' constructor constructs an invalid rectangle where the
p1 coordinates are lower than the p2 coordinates. In particular, p1 is
set to (1, 1). The 'Widget' implementation uses the points individually
as input into the 'Animated_rect' mechanism. This way, widgets end up
being positioned at (1, 1) initially and are moved to (0, 0) once the
first layout update is applied. By explicitly initializing the
'_geometry' to (0x0+0+0), we avoid this initial artifact.
 2019-03-18 15:56:24 +01:00
Christian Prochaska
f4542b378c arora: update Nitpicker and shape report routes 
Fixes #3219
 2019-03-18 15:56:24 +01:00
Alexander Boettcher
0c6548fda0 sculpt: increase leitzentrale memory 
to avoid hanging popup and warning:

[leitzentrale -> gui] Warning: popup assigned RAM (12M) exceeds available RAM (6596441)

Issue #3218
 2019-03-18 15:56:24 +01:00
Norman Feske
669aed0ac0 sculpt: don't reset hover on click on '+' menu 
When entering/leaving sub menus of Sculpt's '+' menu, some parts of the
menu sometimes remain unchanged, in particular the back button.
Originally, a click would reset the hovering on clicks in the
expectation that any click would eventually result in a completely new
situation where the old hovering information does not make sense and
would only (potentially) confuse the menu. But this was apparently
overzealous. With the patch a once hovered back button stays hovered
even when actitivated and the back button of the upper-level menu
happens to stay under the current pointer position.

Issue #3209
 2019-03-18 15:56:23 +01:00
Norman Feske
014e800e8a menu view: immediate hover update on dialog change 
This patch improves the hover handling in situations where the dialog
changes under the pointer. Previously, hover changes were reported
as response to user input only, which failed to cover this case. This
became a problem with Sculpt CE's '+' menu, which changes on the fly
when entering/leaving sub menus.

The patch also cleanly separates the hover handling from the focus
handling. Originally, the hovering was reset when the menu view got
unfocused. In situations like Sculpt's '+' menu where the menu view
receives a transient focus only while clicked and gets unfocused on the
button-release event (aka clack), each clack would invalidate the hover
information until a new input event comes in.

Finally, the patch introduces the clear distinction between situations
where the entire dialog is hovered or not. Previously, this state was
somehow implicitly kept by issuing an invalid hover report whenever a
leave event was observed.

Issue #3209
 2019-03-18 15:56:23 +01:00
Norman Feske
a61c5b6be3 wm: move virtual pointer pos on window movement 
When a window is moved, the virtual pointer position must be updated,
taking the changed input coordinate into account. This patch propagates
such changes via absolute motion events to the client.

Without this patch, Sculpt CE's '+' menu wouldn't update the hovered
item correctly when entering/leaving sub menus (which happen to trigger
the repositioning of the menu on screen).

Issue #3209
 2019-03-18 15:56:23 +01:00
Norman Feske
ac3509d308 News item about Genodians.org 2019-03-18 15:56:23 +01:00
Norman Feske
30505c7977 test-spark: add object-size assertions 2019-03-18 15:56:23 +01:00
Christian Prochaska
2a71c8fa82 pcsc-lite: read vendor id and product id from USB device 
Fixes #3211
 2019-03-18 15:56:23 +01:00
Stefan Kalkowski
4796f761b8 base: initialize Cpu_state consistently (Fix #1452) 2019-03-18 15:56:23 +01:00
Christian Helmuth
18546ea16d Use https URLs for qt5 downloads 
The site download.qt.io seems to apply some mirror-via-HTTP-redirect
scheme which, unfortunately, seems broken (and therefore annoyingly
slow) for non-SSL-secured URLs.
 2019-03-18 15:56:23 +01:00
Josef Söntgen
0ef784b8fd mixer: only provide Audio_out in pkg 
The mixer only supports Audio_out.

Fixes #3210.
 2019-03-18 15:56:23 +01:00
Stefan Kalkowski
80fa23da5e hw: increase timing accuracy of kernel (fix #3081) 
* Introduce 64-bit tick counter
* Let the timer always count when possible, also if it already fired
* Simplify the kernel syscall API to have one current time call,
  which returns the elapsed microseconds since boot
 2019-03-18 15:56:23 +01:00
Stefan Kalkowski
2ecf1d887b hw: schedule on demand (Fix #3157) 2019-03-18 15:56:23 +01:00
Alexander Senier
a58fcc3b1e Update runtime ALIs 2019-03-18 15:56:23 +01:00
Alexander Senier
4c8d787918 Port libsparkcrypto 2019-03-18 15:56:23 +01:00
Emery Hemingway
127e5a2726 Add more features and symbols to libcrypto 
Fix #3203
 2019-03-18 15:56:23 +01:00
Johannes Schlatow
7bd506b913 virtualbox: fix compiler error with newer iasl versions 
I also added another sed command to suppress warnings that appear
because the include-guards contain dashes.

fixes #3147
 2019-03-18 15:56:23 +01:00
Norman Feske
bd09783438 sculpt: dim non-TCB when selecting a component 2019-03-18 15:56:22 +01:00
Norman Feske
ef0a3f5be1 menu_view: hidden dependencies in <depgraph> 
When specifying the attribute 'dep_visible="false"' for a primary
dependency or the attribute 'visible="false"' for a secondary
dependency, the dependency is used for the layout calculation but not
displayed in the graph.
 2019-03-18 15:56:22 +01:00
Norman Feske
653f653c67 menu_view: style for unimportant frame and button 2019-03-18 15:56:22 +01:00
Norman Feske
ac37991b34 sculpt: minor graph improvements 
- Omit showing routes to uninteresting ROMs obtained from the parent,
  i.e., the binaries requested by the sculpt-managed subsystems.
- Change the routes for the inspect subsystem such that the inspect-noux
  instance is anchored at the config node (critical!) and the nit_fb
  instance anchored at the used GUI.
 2019-03-18 15:56:22 +01:00
Norman Feske
6ee7049736 sculpt: add whitespace around parent resources 2019-03-18 15:56:22 +01:00
Emery Hemingway
57fd4e9148 Add Io_progress_handler to Entrypoint interface 
The "schedule_post_signal_hook" method of the Genode::Entrypoint class
is problematic because the signal hook can be scheduled and replaced
multiple times during the signal dispatch cycle. Add an alternative to
this method with "register_io_progress_handler" and the "Post_signal_
hook" class with "Io_progress_handler". The difference being an
"Io_progress_handler" may be registered once during the lifetime of an
entrypoint to prevent arbitrary libraries from replacing a pending hook.

The "register_io_progress_handler" remains as a deprecated API, and is
now invoked for every I/O signal received and only for I/O signals
rather than for any signal.

Ref #3132
 2019-03-18 15:56:22 +01:00
Emery Hemingway
93fb79f357 Check for existing directories at Vfs::Dir_file_system 
Return OPENDIR_ERR_NODE_ALREADY_EXISTS for existing directories.
Remove deprecated 'is_directory' method.

Fix #3083
 2019-03-04 10:57:04 +01:00
Norman Feske
c41fbe9ea5 sculpt: don't mark in-construction pkgs as present 
This patch excludes the current "Construction" from the list of
"present" components in the runtime. Without the patch, a missing "wm"
would go unmissing once when the routing dialog of a new wm instance
appears. Now an already present window layouter that had a broken route
would prematurely re-appear in the config, which should not happen
because the new wm does not exist yet.
 2019-03-04 10:53:37 +01:00
Alexander Boettcher
1cc3472e1d dde_ipxe: enable Intel i211 network card 
Fixes #3201
 2019-03-04 09:20:24 +01:00
Norman Feske
00ac29e53d Sculpt: avoid superfluous vertical space w/o info 
When a <pkg> node of an index lacks an 'info' attribute, omit the
corresponding fields in the pkg dialog to avoid blank vertical space.
 2019-03-01 19:27:40 +01:00
Norman Feske
434783da4d Sculpt: replace '_' by whitespace in GUI 
Fixes #3190
 2019-03-01 19:27:23 +01:00
Norman Feske
71019eccbd doc: minor tweaks of release notes 19.02 2019-03-01 16:45:24 +01:00
Norman Feske
06d68e87ba sculpt: list all depot users in selection menu 
Fixes #3188
 2019-03-01 16:45:24 +01:00
Christian Helmuth
8c2235c9a7 depot: delete signature file if signing failed 
In cases where the signing failed (maybe the passphrase for the key was
not available at this time), there remained an empty .sig file, which is
newer than the to-be-signed file and, therefore, prevents subsequent
signing processes.
 2019-03-01 15:43:50 +01:00
Norman Feske
cd244c2077 sculpt: don't install any index by default 
When buiding the sculpt image, the sculpt.run script used to integrate
the current version of the index of the 'depot_user' into the boot
image. At runtime, when the Sculpt partition is selected for "use" this
index - along with the 'pubkey' and 'download' files of the known
depot users - is written to the Sculpt partition.

This has the undesirable effect that a later version of the index
(published some time after the sculpt image was created) would always be
overwritten by the outdated index shipped with the boot image.

The built-in default index was actually a stop-gap solution needed
during the development of Sculpt CE, introduced when the downloading of
index files was not yet supported. Now, with the working download
mechanism, it is no longer needed. Hence, this patch removes the default
index from the sculpt image.
 2019-03-01 14:06:22 +01:00
Alexander Boettcher
30ddae0f91 nova: evaluate second kernel buddy earlier 
Fixes #3195
 2019-02-28 16:37:53 +01:00
Alexander Boettcher
38d3577c10 pkg: limit vbox5-nova-sculpt runtime to 4G 
Fixes #3197
 2019-02-28 16:36:12 +01:00
Emery Hemingway
bffcffe072 Skip Solo5 timer test for auto-QEMU 
Fix #3191
 2019-02-28 15:38:01 +01:00
Christian Helmuth
18ce901746 version: 19.02 2019-02-28 14:19:59 +01:00
Norman Feske
4afab58739 News item for version 19.02 2019-02-28 13:56:15 +01:00
Norman Feske
ca7bc5913f Release notes for version 19.02 2019-02-28 13:56:14 +01:00
Christian Helmuth
36adbef3f9 depot: update recipe hashes 2019-02-28 11:34:45 +01:00
Norman Feske
d6030a16b3 sculpt: tweaks for the component graph 
- Hide depot_rom and dynamic_depot_rom
- Reset selection when removing the selected component
 2019-02-28 11:34:08 +01:00
Norman Feske
3840ea9537 sculpt: anchor nic_drv, wifi_drv to hardware node 2019-02-28 11:34:08 +01:00
Christian Prochaska
2c253edda9 wm: forward 'buffer_size' argument of shape report session 
Fixes #3186
 2019-02-28 11:34:08 +01:00
Alexander Boettcher
47730f9a6e init: avoid upgrade of resources on exited child 
Fixes #3184
 2019-02-28 11:34:08 +01:00
Christian Helmuth
a5d0f6a2af ada: fix ada-runtime-alis for current changes 
Note, this also adapts the spark depot recipe to preserve source-file
time stamps.
 2019-02-28 11:34:07 +01:00
Christian Helmuth
fb155b95c7 Clarify documentation of Block::Session::info() 
Also cleanup some proscribed abbrevations.

Fixes #3185
 2019-02-28 11:34:07 +01:00
Christian Helmuth
28c25f120e run: save [run_dir].config also on hw and linux 
This was missing as boot_dir/hw and boot_dir/linux do not use
`proc build_core_image`.
 2019-02-28 11:34:07 +01:00
Christian Helmuth
b06ec370d1 Use platform-specific NIC driver in lwip test 
Follow-up to "zynq: restructure nic_drv spec structure"

Issue #3179
 2019-02-28 11:34:07 +01:00
Sebastian Sumpf
2f7fa3b905 depot: pubkey and download for ssumpf 2019-02-28 11:34:07 +01:00
Christian Prochaska
23220a98b9 qt5: don't use 'HEADERS' variable in library makefiles 
The 'HEADERS' variable is currently only supported for applications with
qmake project files.

Fixes #3183
 2019-02-28 11:34:07 +01:00
Norman Feske
e679d55f67 Update <provides> info in pkg runtimes 
This information is now used by Sculpt's '+' menu for the interactive
routing.
 2019-02-28 11:34:07 +01:00
Norman Feske
7921834b1d sculpt: update default launchers 
This commit removes most of the default launchers, which are now
superseded by the interactive component addition feature of the '+'
menu.

We keep the chroot components because we cannot easily create chroot
instances interactively yet.

The usb_devices_rom is still needed because its configuration is meant
to be edited at runtime.

It also adds a 'themed_wm' launcher to make the initial sculpt
experience easier. For knowledgeable users, the index contains all
ingredients needed to build a multi-component window manager manually.
 2019-02-28 11:34:07 +01:00
Norman Feske
c0b93190d0 sculpt: browse depot index files in '+' menu 
This commit turns the '+' menu into a tool for the following tasks:

- Selecting and downloading of depot index files
- Browsing of the hierarchical depot index files
- Installation of packages found in the index files
- Interactive routing configuration of a selected package
- Deployment of configured component
 2019-02-28 11:34:07 +01:00
Norman Feske
b7f5aae64a sculpt: don't constrain '+' menu to avail space 
Sculpt used to restrict the size of leitzentrale windows to the screen
area that is not obstructed by the menu and log. This is useful for the
runtime view and the inspect window. However, the menu should be allowed
to use the entire screen because it overlays the other content.

Before this patch, the menu wouldn't be displayed completely on small
resolutions (e.g., 1024x768 when using the VESA driver) because the log
at the bottom of the screen imposed the size constraint on the menu.
With the patch, the menu is able to overlay the log window.
 2019-02-28 11:34:07 +01:00
Norman Feske
2163ce25f6 sculpt: show TCB dependencies of selection 
This patch enhances the runtime view such that not only immediate
dependencies but also all transitive dependencies of the selected
component are displayed. This way, the graph nicely reveals the
trusted computing base of the selection.
 2019-02-28 11:34:07 +01:00
Norman Feske
21968d35bb sculpt: display parent roles in graph 2019-02-28 11:34:07 +01:00
Norman Feske
d0e1ddb8c2 sculpt: cache runtime-config info for GUI 
Instead of parsing the runtime's configuration each time when generating
the graph dialog (e.g., when changing the hover state), extract the
relevant information only on configuration changes.
 2019-02-28 11:34:06 +01:00
Norman Feske
010caa6236 sculpt: omit the sculpt helpers from the graph 
The runtime view, launcher query, and depot query increase the
complexity of the graph without providing a tangible value to the user.
This patch omits those components from the runtime view to make the
graph less confusing.
 2019-02-28 11:34:06 +01:00
Norman Feske
e9f40e9d68 sculpt: let depot/public_rw depend on update state 
By running those components only when needed, the graph stays simpler in
the normal state.
 2019-02-28 11:34:06 +01:00
Norman Feske
1afec11dfc sculpt: append "..." to non-immediate buttons 
Append "..." to button labels whenever the button does not perform an
immediate action but merely toggles user-interface elements. This
tells the user that the button can be pressed without risk.
 2019-02-28 11:34:06 +01:00
Norman Feske
852c319399 sculpt: use radio button for APs and launchers 2019-02-28 11:34:06 +01:00
Norman Feske
6110e6561c sculpt: show message when network is needed 
Whenever Sculpt needs to download depot content but no network
connectivity is provided, a message is displayed in the "Diagnostics"
part of menu.
 2019-02-28 11:34:06 +01:00
Norman Feske
ac68073ffc depot_download: support downloading index files 
With this commit, the 'installation' input of the depot-download
subsystem accepts <index> nodes in addition to <archive> nodes. Each
index node refers to one index file specified via the 'path' attribute.

This commit also improves the tracking of failure states. Once an
installation job failed (due to a download of verification error),
it won't get re-scheduled. In the past, such failure states were not kept
across subsequent import iterations, which could result in infinite
re-attempts when an installation contained archives from multiple users.
The the progress of the download process is now reflected by the
"progress" attribute on the download manager's state report, which
allows the final report to contain the list of installed/failed archives
along with the overall progress/completed state. The detection of the
latter is important for the sculpt manager for reattempting the
deployment of the completed packages.

The patch enhances the depot_download.run script to stress the new
abilities. In particular, the scenario downloads a mix of index files
(one present, one missing) and archives, from two different depot users
(genodelabs and nfeske).

Issue #3172
 2019-02-28 11:34:06 +01:00
Norman Feske
1b518965cc depot/sculpt: support for index files 
The input for the pkg index is located at gems/run/sculpt/index.

The sculpt.run script uses this input for generating the depot index
file at depot/<user>/index/<version>.

The tool/depot/publish tool support arguments of the form
<user>/index/<version> where <version> corresponds to the Sculpt
version.

Issue #3172
 2019-02-28 11:34:06 +01:00
Norman Feske
0d5b8cb0fe sculpt: version 19.02 2019-02-28 11:34:06 +01:00
Norman Feske
de26d3c099 sculpt: show '+' button only when partition 'used' 
This prevents the situation where the user has booted the system, has
not yet selected a storage target to "use" for Sculpt, yet clicks on the
'+' menu. Such clicks show no immediate response because Sculpt cannot
know where to deploy the selected package. But since the user is not
guided towards resolving this prerequisite, it's better to not present
the menu in the first place. The '+' appears as soon as a storage target
is selected for "use".
 2019-02-28 11:34:06 +01:00
Norman Feske
db8b054fa4 extract: support the extraction of raw xz files 
Issue #3172
 2019-02-28 11:34:06 +01:00
Norman Feske
edef2df21b depot_query: scan depot for users 2019-02-28 11:34:06 +01:00
Norman Feske
c25fbc010d depot_query: support for querying index files 
Issue #3172
 2019-02-28 11:34:06 +01:00
Norman Feske
90709fc4d3 menu_view: remove "failed to construct" message 
This error message may occur during the startup of a multi-component
application when the very first dialog is generated just after the menu
view is ready. It is not an error.
 2019-02-28 11:34:06 +01:00
Norman Feske
74c31c60c6 menu_view: improved buffer alloc and view update 
This patch improves the consistency of the view size with the dialog
size for situations where the dialog shrinks.
 2019-02-28 11:34:06 +01:00
Norman Feske
cb163bfbe1 menu_view: prevent superfluous relinking 2019-02-28 11:34:06 +01:00
Norman Feske
40b986bcc4 menu_view: button and frame styles for sculpt 
This commit adds the following styles:

button/enter    - for entering a sub menu
button/back     - for returning from a sub menu
button/radio    - for picking one item of a list
button/checkbox - for making a selection
frame/transient - for temporary GUI elements
 2019-02-28 11:34:06 +01:00
Norman Feske
ca51692164 wm: remove "no focus model available" warning 
This message is diagnostic, but also occurs in legitimate situations
such as the wm in Sculpt's Leitzentrale where the focus is managed
completely outside the wm.
 2019-02-28 11:34:06 +01:00
Norman Feske
c51508a47c lx_fs: return host inode value for dir entries 
With this patch, bash running in noux becomes able to list the content
of directories (via 'echo *') hosted in lx_fs.
 2019-02-28 11:34:06 +01:00
Johannes Kliemann
fa5de776a6 ada: add arit64 to runtime 2019-02-28 11:34:06 +01:00
Johannes Kliemann
fd6047f5d8 ada: add Interfaces.C 
extensively in auto-generated bindings
 2019-02-28 11:34:05 +01:00
Alexander Senier
f30a82d599 ada: provide runtime symbols for elaboration code 2019-02-28 11:32:53 +01:00
Christian Helmuth
46a29532a9 depot: update recipe hashes 2019-02-26 14:47:02 +01:00
Johannes Schlatow
11d8f97845 zynq: restructure nic_drv spec structure 
The zynq nic_drv also depends on hw, we therefore adapted the folder
structure for clarity. Also renamed the binary to 'zynq_nic_drv' to
prevent conflicts and to allow removing the cadence_gem spec.

Issue #3179
 2019-02-26 14:47:02 +01:00
Johannes Schlatow
c1caeb7e70 zynq: improve cache handling in nic_drv 
Clean and invalidate caches in nic_drv before/after triggering DMA.

Issue #3179
 2019-02-26 14:47:02 +01:00
Norman Feske
1d0a1e7937 depot/extract: feed file names into version hash 
Fixes #3178
 2019-02-26 14:47:02 +01:00
Johannes Schlatow
d9043057a2 lx_fs: implement move() 
Fixes #3176
 2019-02-26 14:47:02 +01:00
Johannes Schlatow
65a662f5b3 lx_fs: close file on destruction 
Fixes #3177
 2019-02-26 14:47:02 +01:00
Emery Hemingway
d273129d9d Disambiguate noux-system config 2019-02-26 14:45:31 +01:00
Christian Prochaska
a85c47134b qt5: remove 'tablet' QEMU argument from run scripts 
Fixes #3175
 2019-02-26 14:45:31 +01:00
Stefan Kalkowski
2cf4e5a6de hw: fix regression of smp kernel initialization 
In commit "hw: improve cross-cpu synchronization" the implicit safe
initialization of the global kernel lock gets unsafe.
It is a static object, which is protected by the cxx library regarding
its initialization. But our cxx library uses a Genode::semaphore in
the contention case of object construction, which implicitly leads
to kernel syscalls for blocking the corresponding thread. This behaviour
is unacceptable for the kernel code.
Therefore, this fix guards the initialization of the kernel code with
a simple static boolean value explicitely.

Ref #3042
Ref #3043
 2019-02-26 14:45:31 +01:00
Stefan Kalkowski
1ff36965f4 sd_card_drv: support for Nitrogen6 SoloX 2019-02-26 14:45:31 +01:00
Stefan Kalkowski
55b0dff795 hw: add board support for Nitrogen6 SoloX 2019-02-26 14:45:31 +01:00
Stefan Kalkowski
7ffc480db8 hw: avoid code duplication for imx6 initialization 
This is preliminary work to move the Wandboard to and add the
Nitrogen6 SX board to the world repository.

Ref #3168
 2019-02-26 14:45:31 +01:00
Emery Hemingway
20f7f5b64f VFS Rump: print file-system usage stats at mount time 
Fix #3174
 2019-02-26 14:45:20 +01:00
Emery Hemingway
6c7356072d Catch session exceptions during VFS File_system stat 
Fix #3173
 2019-02-26 14:45:13 +01:00
Norman Feske
3de673a8ca dde_linux/usb: remove deprecated APIs on ARM 
Issue #1987
 2019-02-26 14:45:07 +01:00
Christian Helmuth
bce4fa9531 sel4: user virtual-memory size of 8 GiB on 64-bit 2019-02-26 14:45:07 +01:00
Norman Feske
5335563a4b tiled_wm.run: message about fd limit on Linux 
Taken from leitzentrale.run
 2019-02-26 14:45:07 +01:00
Christian Prochaska
dcdd3bff94 qt5_drivers.inc: don't call 'append_platform_drv_config' 
'append_platform_drv_config' appends to the global 'config' variable,
which seems to cause problems with the order of appending when the
'config' variable does not contain the '<config>' start node yet.
'[platform_drv_config]' can be used instead to append to a local variable.

Fixes #3171
 2019-02-26 14:45:07 +01:00
Christian Helmuth
2211d20c1d Increase CAP quota in wifi tests (sel4) 2019-02-26 14:45:07 +01:00
Norman Feske
d3d6b643f1 wm: shape report interception mechanism 
This patch simplifies the propagation of pointer shapes from
window-manager clients to the pointer. The "shape" report is routed to
the wm server, which, in turn, reports it to the pointer. This way, the
pointer can easily correlate the label of the application's "shape"
report with the label of the application's Nitpicker session. The
formerly used manual rewriting of the "shape" label is not needed
anymore.

Since the wm server provides a "Report" service now, its <provides>
declaration must cover "Report" in addition to "Nitpicker" to avoid
runtime error messages. Vice versa, the wm is now expected to request
"shape" reports, which should be routed to the pointer (using the
'label_last' routing attribute).

Fixes #3165
 2019-02-26 14:45:07 +01:00
Christian Helmuth
5fa0de95f6 Disable ping_nic_router test on linux 
The test depends on working DHCP for the uplink currently.
 2019-02-26 14:45:07 +01:00
Christian Helmuth
30b47eab37 Fix IP address matching in vbox share tests 2019-02-26 14:45:07 +01:00
Emery Hemingway
bebbc87c3a Replace static "genodelabs" depot user with "[depot_user]" call 
Use "depot_user" TCL procedure may be used in run scripts for tests not
related to downloading specific packages.

Fix #3170
 2019-02-26 14:44:31 +01:00
Emery Hemingway
024630d6ad Add Unicode to terminal_crosslink test 
Ref #3148
 2019-02-26 14:44:31 +01:00
Emery Hemingway
2fa7964a39 VFS lwIP: fix TCP pbuf unchaining 
The lwIP VFS plugin uses lwIP pbuf chains to queue recieved TCP data and
must rechain them when the application dequeues data. Remove an
"pbuf_realloc" call which is not needed for updating pbuf metadata when
dequeuing the head of the chain.

Fix #3169
 2019-02-26 14:44:31 +01:00
Norman Feske
8ca887a25c decorator pkgs: add missing default policy 
This prevents spamming the log with false-positive warning messages.
It is related to commit "os/session_policy: warn if no policy exists".
 2019-02-26 14:44:31 +01:00
Christian Helmuth
2a41f50661 Remove obsolete usb_hid_panda test 2019-02-26 14:44:31 +01:00
Christian Helmuth
c749f577e6 Do not run usb_hid test on imx6q_sabrelite 
We do not support the monolithic usb_drv on this platform.
 2019-02-26 14:44:31 +01:00
Christian Helmuth
a9d58e47cb Adapt run scripts to nic_bridge default policy 2019-02-26 14:44:31 +01:00
Norman Feske
5ed5fddb7c base/os: remove deprecated APIs 
This commit removes APIs that were previously marked as deprecated. This
change has the following implications:

- The use of the global 'env()' accessor is not possible anymore.
- Boolean accessor methods are no longer prefixed with 'is_'. E.g.,
  instead of 'is_valid()', use 'valid()'.
- The last traces of 'Ram_session' are gone now. The 'Env::ram()'
  accessor returns the 'Ram_allocator' interface, which is a subset of
  the 'Pd_session' interface.
- All connection constructors need the 'Env' as argument.
- The 'Reporter' constructor needs an 'Env' argument now because the
  reporter creates a report connection.
- The old overload 'Child_policy::resolve_session_request' that returned
  a 'Service' does not exist anymore.
- The base/printf.h header has been removed, use base/log.h instead.
- The old notion of 'Signal_dispatcher' is gone. Use 'Signal_handler'.
- Transitional headers like os/server.h, cap_session/,
  volatile_object.h, os/attached*_dataspace.h, signal_rpc_dispatcher.h
  have been removed.
- The distinction between 'Thread_state' and 'Thread_state_base' does
  not exist anymore.
- The header cpu_thread/capability.h along with the type definition of
  'Cpu_thread_capability' has been removed. Use the type
  'Thread_capability' define in cpu_session/cpu_session.h instead.
- Several XML utilities (i.e., at os/include/decorator) could be removed
  because their functionality is nowadays covered by util/xml_node.h.
- The 'os/ram_session_guard.h' has been removed.
  Use 'Constrained_ram_allocator' provided by base/ram_allocator.h instead.

Issue #1987
 2019-02-26 14:44:15 +01:00
Emery Hemingway
ffeb8e2af0 Chargen description of French keyboard 
Add an input_filter character description file for the French AZERTY
keyboard layout and include it with Sculpt.

Fix #3149
 2019-02-26 14:42:55 +01:00
Christian Prochaska
ee423e5bf1 qt5: remove deprecated APIs 
Issue #3162
 2019-02-26 14:38:03 +01:00
Norman Feske
0e0b6bdde7 depot: update recipe hashes 2019-02-19 11:13:56 +01:00
Christian Helmuth
dabddd4c89 Remove outdated affinity test 
Thanks Tomasz Gajewski for the hint.

Related to #3041
 2019-02-19 11:13:45 +01:00
Josef Söntgen
5bab5f4cca libc: extent vbox specific pthread_create API 
Make it possible to adopt normal Genode::Threads for use within in a
pthread environment.

Issue #3164.
 2019-02-19 11:12:13 +01:00
Emery Hemingway
2f33d44713 VFS: Configure File_system client buffer at runtime 
Make the size of the packet buffer at the VFS File_system client
configurable. This allows larger reads to be completed in a single
packet cycle.

Example: <vfs> <fs buffer_size="512K"/> </vfs>

Fix #3167
 2019-02-19 11:12:12 +01:00
Christian Prochaska
4921388732 usb_hid.run: adapt for rpi and arndale 
- add 'Regulator' route and adapt string unification rule for arndale
- add 'IRQ' route for rpi

Fixes #3166
 2019-02-19 11:12:12 +01:00
Christian Prochaska
fb18e7e95d qt5: use 'qt_nanosleep()' for sleeping 
Fixes #3162
 2019-02-19 11:12:12 +01:00
Norman Feske
aa662f939a Remove OMAP4 gpio_drv test 
This (interactive) test has not been used for a long time and remained
not updated to the current Genode API.
 2019-02-19 11:12:12 +01:00
Emery Hemingway
b5bd6e0114 Write abritrary printable objects into Xml_generator 
Fix #3161
 2019-02-19 11:12:12 +01:00
Norman Feske
b24edc1633 Remove residual uses of deprecated APIs 
Issue #1987, related also to issue #3163 and issue #3164.
 2019-02-19 11:12:12 +01:00
Christian Helmuth
c4c19f885f base: synchronize signal-proxy thread destruction 2019-02-19 11:12:12 +01:00
Christian Helmuth
1c1cbb9b95 Synchronize thread destruction in signal test 2019-02-19 11:12:12 +01:00
Norman Feske
3d6f4979c3 tz_vmm: remove use of deprecated APIs 
Issue #1987
 2019-02-19 11:12:12 +01:00
Norman Feske
630cb96a54 omap4 fb_drv: remove use of deprecated API 
Issue #1987
 2019-02-19 11:12:12 +01:00
Norman Feske
cf70f65bec exynos5 fb_drv: remove use of deprecated API 
Issue #1987
 2019-02-19 11:12:12 +01:00
Norman Feske
8f2212ea1e nova: disable -Wsuggest-override for the kernel 
Issue #3159
 2019-02-19 11:12:12 +01:00
Norman Feske
d9feaa1875 Add -Wsuggest-override to default strict warnings 
Issue #3159
 2019-02-19 11:12:12 +01:00
Norman Feske
b3727a9b46 Add missing override annotations 
Issue #3159
 2019-02-19 11:12:11 +01:00
Christian Prochaska
1f47e2823a foc: always enable user mode access for performance monitors 
QEMU implements the cycle count register read by
'Genode::Trace::timestamp()', but does not report a supported debug model
version for Cortex-A9.

Fixes #3154
 2019-02-19 11:12:11 +01:00
Stefan Kalkowski
89f0717df6 base: timeout framework calibration exit condition 
Fix #3156
 2019-02-19 11:12:11 +01:00
Stefan Kalkowski
5136d9ddc7 base: fix timer_ticks_to_us casting error 
Fix #3155
 2019-02-19 11:12:11 +01:00
Martin Stein
10b6b88b01 nic_bridge: simplified session request handling 
1) A session request gets denied if there is no matching session policy.
   (The <defaul-policy/> tag can be used for the former default behavior)
2) A session request gets denied if the MAC address is given through the
   matching policy but this address cannot be allocated.
3) A session request gets denied if the MAC address is not given through the
   matching policy and it is also not possible to allocate one.

Issue #3040
 2019-02-19 11:12:11 +01:00
Norman Feske
801aa46c46 leitzentrale.run: tweak quota for log_noux 2019-02-19 11:12:11 +01:00
Norman Feske
c232d703ca sculpt: remove policy warnings 
The warnings were introduced with commit "os/session_policy: warn if no
policy exists".
 2019-02-19 11:12:11 +01:00
Norman Feske
924e5c54eb core: fix RM-session upgrade mechanism 
This is a follow-up commit of "base/core: use references instead of
pointers". Because the 'Rm_root::_upgrade_session' implementation
lacked the 'override' keyword, my overzealous change of the pointer
argument went unnoticed.

This commit fixes the depot_rom (cached_fs_rom) failure in Sculpt.
When cached_fs_rom attempted to create a new managed dataspace while the
RM session quota was depleted, it tried to upgrade the session (via
Rm_connection::create). However, the upgraded resources never reached
the actual session because the default 'Root_component::_upgrade_session'
was called instead of 'Rm_root::_upgrade_session'.

Issue #3135
 2019-02-19 11:12:11 +01:00
Norman Feske
5e9102f031 Run script to execute a single test w/o the depot 2019-02-19 11:12:11 +01:00
Christian Helmuth
3208c4f2cb Increase netperf test timeout 2019-02-19 11:12:11 +01:00
Christian Helmuth
abdd1d7715 Adapt network run scripts for recent Qemu (3.1) 
-netdev and -device should work with Qemu versions >= 2.5 at least but
3.0 dropped support for legacy -net completely.
 2019-02-19 11:12:10 +01:00
Norman Feske
00fa48a886 os: new Block::Request_stream API 
Issue #3092
 2019-02-19 11:12:10 +01:00
Josef Söntgen
f9523c32d5 os: accessor for addr and size of packet stream 2019-02-19 11:08:18 +01:00
Norman Feske
3858e1df51 os: support for deferred packet-stream signals 
This patch enhances the packet-stream API with the principle ability to
side-step the built-in implicity data-flow signals and manage the
signals manually. This allows for a more efficient batching of packet
processing.

Issue #3092
 2019-02-19 11:08:18 +01:00
Norman Feske
69d6145f5a os: don't hide tx_cap from block-session interface 
The 'tx_cap' RPC function is only used at session-creation time. For
this reason, it was not listed in the "official" RPC interface in
'block_session.h'. However, this makes the interface more obscure than
it needs to be. So this patch promotes it to a regular RPC function.

Issue #3092
 2019-02-19 11:08:18 +01:00
Norman Feske
aa66b5d62f base: remove dependency from deprecated APIs 
This patch adjusts the implementation of the base library and core such
that the code no longer relies on deprecated APIs except for very few
cases, mainly to keep those deprecated APIs in tact for now.

The most prominent changes are:

- Removing the use of base/printf.h

- Removing of the log backend for printf. The 'Console' with the
  format-string parser is still there along with 'snprintf.h' because
  the latter is still used at a few places, most prominently the
  'Connection' classes.

- Removing the notion of a RAM session, which does not exist in
  Genode anymore. Still the types were preserved (by typedefs to
  PD session) to keep up compatibility. But this transition should
  come to an end now.

- Slight rennovation of core's tracing service, e.g., the use of an
  Attached_dataspace as the Argument_buffer.

- Reducing the reliance on global accessors like deprecated_env() or
  core_env(). Still there is a longish way to go to eliminate all such
  calls. A useful pattern (or at least a stop-gap solution) is to
  pass the 'Env' to the individual compilation units via init functions.

- Avoiding the use of the old 'Child_policy::resolve_session_request'
  interface that returned a 'Service' instead of a 'Route'.

Issue #1987
 2019-02-19 11:08:17 +01:00
Norman Feske
c629a92aa2 base: init_env_ram_session -> init_env_pd_session 
The notion of a RAM session does not exist anymore. Hence, we have to
adjust the name of 'init_env_ram_session'. Since this change modifies
the ABI, it comes as separate commit.

Issue #2407
 2019-02-19 11:08:17 +01:00
Emery Hemingway
22327b43ae Refactor terminal for intrinsic Unicode support 
Refactor the graphical terminal server to internally represent
characters as 16-bit codepoints and handle the duplex terminal stream as
UTF-8.

- Make the Codepoint class printable to the Output interface
- Decode data received at the Terminal session from UTF-8 to a 16-bit
  character
- Pass 16-bit characters through terminal decoder and char-cell arrays
- Send Unicode through terminal session in a burst of UTF-8 bytes

Fix #3148
 2019-02-19 11:08:17 +01:00
Alexander Boettcher
0c24e1efdc vm_session: extensions 
- support to create multiple vCPUs
- support to implement Vm_session methods client side within base library
- adjust muen specific virtualbox4 version to compile/link

Issue #3111
 2019-02-19 11:08:17 +01:00
Emery Hemingway
c5786b212b Update Seoul contrib code for new Fifo accessors 
Ref #3135
 2019-02-19 11:08:17 +01:00
Emery Hemingway
38ab456c78 Remove pointers from Genode::Fifo interface 
Replace methods of Genode::Fifo returning pointers with methods which
call lambdas with references.

Ref #3135
 2019-02-19 11:08:17 +01:00
Stefan Kalkowski
328c1ad96e foc: prevent region overlap on i.MX6 Sabrelite 2019-02-19 11:08:17 +01:00
Stefan Kalkowski
161f39f7af imx6q_sabrelite: enable sd_card_drv 2019-02-19 11:08:17 +01:00
Stefan Kalkowski
da502dd036 muen: disable SMP test on Muen 2019-02-19 11:08:17 +01:00
Alexander Boettcher
6bb145bdd8 Remove virtualbox 4 vmm 
Fixes #3141
 2019-02-19 11:08:17 +01:00
Christian Prochaska
87d526968c usb_hid: test both USB host drivers 
Fixes #3152
 2019-02-19 11:08:17 +01:00
Alexander Senier
5ccae43552 Rename Ada runtime to SPARK runtime 
The minimal-footprint Ada runtime for implementing library-like
functionality in SPARK is now called "spark" runtime.

The full Ada runtime for entire components written in Ada and using the
libc as glue to the underlying system will move to the world repository
as "ada" runtime.

Issue #3144
 2019-02-19 11:08:17 +01:00
Alexander Senier
3b41e02ea5 Create make dependencies using ali2dep 2019-02-19 10:50:51 +01:00
Alexander Senier
80a607ee0c Abandon gnatmake and generate elaboration code 2019-02-19 10:50:51 +01:00
Emery Hemingway
253d6b0b92 Runtime package of clipboard service 
Fix #3150
 2019-02-19 10:50:51 +01:00
Christian Helmuth
271e2398f9 depot: update recipe hashes 2019-02-12 14:24:12 +01:00
Christian Prochaska
106cd86375 smartcard.run: increase cap quota of test component 
Fixes #3151
 2019-02-12 10:33:32 +01:00
Norman Feske
5d66bfbd19 netperf.run: increase caps of netserver_genode 
This is needed for seL4/x86_64.
 2019-02-12 10:33:32 +01:00
Christian Helmuth
193c1aa533 wifi: refactor config generation in run script 
Now sophisticated test scripts can be easily generated like documented.
 2019-02-12 10:33:32 +01:00
Christian Helmuth
2b1732bdec Adapt usb_drv RAM quota to 12M 
All autopilot scripts use 12 or more MiB for the usb_drv and my personal
tests never succeeded with less on x86 test machines.
 2019-02-12 10:33:32 +01:00
Christian Helmuth
dbac453a39 smartcard: declare vendor/product IDs at one place 
Also added a report_rom for USB devices and let the driver report like
in a real scenario. Can also be used for debugging by setting
verbose="yes" in report_rom.
 2019-02-12 10:33:31 +01:00
Josef Söntgen
3ada4f4733 noux: increase argv buffer to 16KiB 
And most importantly: use the same size everywhere.

Note, this commit also configures the stack size of noux-process threads
to 64 KiB independently of the CPU architecture. The reason is the
increased stack usage because of the additional argument space in
Execve_child_env in syscall(SYSCALL_EXECVE), which crafts a child
environment on stack for the creation of the new child.

Fixes #3145.
 2019-02-12 10:33:16 +01:00
Christian Helmuth
afc95e36e1 netperf: 'times_up' mechanism based on pthread 2019-02-12 10:33:16 +01:00
Norman Feske
9e9614af13 test-init: increase timeout 
The original timeout is one second too short to let the test pass on
NOVA/32bit. This patch increases the timeout by 10 seconds.
 2019-02-12 10:33:16 +01:00
Christian Prochaska
ddacb1400c Do not disable USB BIOS handoff in tests 
Fixes #3140
 2019-02-12 10:33:15 +01:00
Norman Feske
2e1e7b8270 recipes/src/base-hw-muen: add static ACPI ROM 
As we don't execute the acpi_drv on Muen, we have to supply a static
'acpi' info as boot module. This is normally done by the
base/run/platform.inc include. However, when using base-hw-muen kernel
from a depot archive - as done by modern run scripts like
depot_download.run - the platform.inc magic is not applied.

This patch enhances the src archive of base-hw-muen with a mechanism
that creates a pre-defined acpi info at the bin directory via an
artificial src/acpi/target.mk file. This way, the static acpi ROM ends
up as boot module when importing the base-hw-muen archive into a
run script.
 2019-02-12 10:33:15 +01:00
Christian Helmuth
03da438fa4 foc: fix parallel build of kernel libuart 
This incorporates a patch from upstream.
 2019-02-12 10:33:15 +01:00
Josef Söntgen
ac59b68257 wifi_drv: split front end notification 
Issue #3139.
 2019-02-12 10:33:15 +01:00
Josef Söntgen
8f47609024 wifi_drv: fix explicit SCAN request handling 
* Increase the CMD length to accomodate the limit of the wpa_supplicant
* Fix case where multiple SSIDs with 32 bytes are used
* Use Expanding_reporter for accesspoints report

Fixes #3139.
 2019-02-12 10:33:15 +01:00
Josef Söntgen
1c0541e7cb run: add power cycle procedure 
The 'run_genode_until' procedure only called 'run_power_on' to reset
the target machine. That works will with the softreset module, which
is used by all x86-based test system but falls short regarding ARM
boards. The way those boards are connected requires turning the power
off and on for a complete cycle.
 2019-02-12 10:33:15 +01:00
Christian Prochaska
369d60bc21 dde_linux: iterate over PCI devices in ascending BDF order 
On a Lenovo ThinkCentre M57p, the system locks up when the UHCI controller
BIOS handoff (disabling bit 4 in the LEGSUP register) for the controller
with PCI BDF 00:1d:2 is attempted before the handoff for the controller
with BDF 00:1a:0.

Fixes #3138
 2019-02-12 10:33:15 +01:00
Christian Prochaska
e9e048c4ea dde_linux: fix timer callback compatibility with older drivers 
Fixes #3137
 2019-02-12 10:33:15 +01:00
Christian Helmuth
845253af3a dde_linux: prevent truncation in update_jiffies() 
Before, jiffies wrapped after 2**32 microseconds (71.5 minutes) to 0.
 2019-02-12 10:33:15 +01:00
Christian Helmuth
bcef3aeb1e dde_linux: fix calculation of ktime from jiffies 2019-02-12 10:33:15 +01:00
Emery Hemingway
47c6377ac0 Remove libc_lxip 
The libc_lxip library is superceded by vfs_lwip.

Fix #2960
Fix #2535
 2019-02-12 10:33:14 +01:00
Christian Prochaska
db9ff821a2 libc: sync 'listen' socket fs control file 
Fixes #3124
 2019-02-12 10:33:14 +01:00
Emery Hemingway
968e220fd2 Update jbig2dec to version 0.15 
Ref #3131
 2019-02-12 10:33:14 +01:00
Emery Hemingway
13268ec401 Update libpng to version 1.6.36 
Fix #3131
 2019-02-12 10:33:14 +01:00
Alexander Boettcher
2c969f1167 nova: skip smp revoke test in Qemu 2019-02-12 10:33:14 +01:00
Norman Feske
bcb24e316c base: fix warnings reported by LLVM analyzer 
The warnings were false positives though.
 2019-02-12 10:33:14 +01:00
Norman Feske
6b289a1423 base/core: use references instead of pointers 
This patch replaces the former prominent use of pointers by references
wherever feasible. This has the following benefits:

* The contract between caller and callee becomes more obvious. When
  passing a reference, the contract says that the argument cannot be
  a null pointer. The caller is responsible to ensure that. Therefore,
  the use of reference eliminates the need to add defensive null-pointer
  checks at the callee site, which sometimes merely exist to be on the
  safe side. The bottom line is that the code becomes easier to follow.

* Reference members must be initialized via an object initializer,
  which promotes a programming style that avoids intermediate object-
  construction states. Within core, there are still a few pointers
  as member variables left though. E.g., caused by the late association
  of 'Platform_thread' objects with their 'Platform_pd' objects.

* If no pointers are present as member variables, we don't need to
  manually provide declarations of a private copy constructor and
  an assignment operator to avoid -Weffc++ errors "class ... has
  pointer data members [-Werror=effc++]".

This patch also changes a few system bindings on NOVA and Fiasco.OC,
e.g., the return value of the global 'cap_map' accessor has become a
reference. Hence, the patch touches a few places outside of core.

Fixes #3135
 2019-02-12 10:33:13 +01:00
Norman Feske
f9373b4430 base: new Allocator_avl::construct_metadata method 
The new method allows for the construction of a meta-data object inside
the reserved space of the allocator's meta data. It thereby alleviates
the need to copy the meta data object (via the assignment operator) as
done by the traditional 'metadata' setter method. This, in turn, allows
one to use non-copyable objects (like objects with constant member
variables) as meta data.
 2019-02-12 10:33:12 +01:00
Norman Feske
437e6c5653 core: make ASSERT_NEVER_CALLED a single statement 
This way, the macro can be used as a body of an if statement with no
curly braces.
 2019-02-12 10:33:12 +01:00
Norman Feske
3a169d3a78 depot: update recipe hashes 2019-01-30 13:55:20 +01:00
Norman Feske
237d2bff3a base: fix deadlock during signal-context dissolve 
This patch moves the removal of the signal context from the
'_platform_finish_dissolve' to the '_platform_begin_dissolve'
method. This is needed because the removal involves taking
the signal-registry lock. The latter must adhere the same
locking order as the code path used for signal delivery.

Fixes #3109
 2019-01-30 13:55:19 +01:00
Norman Feske
158650ff01 base-nova: remove signal-delivery failure warning 
Fixes #3136
 2019-01-30 13:55:19 +01:00
Norman Feske
9859df8c6c test-fs_report: increase timeout (foc/x86_64/qemu) 2019-01-30 13:55:19 +01:00
Christian Prochaska
210fbcc4c2 gdb_monitor: fix possible page fault on thread removal 
Fixes #3133
 2019-01-30 13:54:54 +01:00
Emery Hemingway
4f88b664e2 Improve VFS server session quota management 
Move the allocation of the session packet buffer into a subclass that is
constructed before the File_system RPC object. This allows the buffer
allocation to be wrapped in a constructor/destructor class and clearly
accounted for.

Fix #3134
 2019-01-30 13:54:54 +01:00
Christian Helmuth
37b2aaa051 Equalize RAM quantum in acpi_drv configs 
While the managed drivers already used 4M, interactive and nic failed on
hardware with larger ACPI tables.
 2019-01-30 13:54:54 +01:00
Christian Helmuth
87c9186efd ports: enable basic clock support in tclsh 2019-01-30 13:54:53 +01:00
Christian Helmuth
f7f1509a27 Re-enable network tests on all platforms 2019-01-30 13:54:53 +01:00
Norman Feske
27bb44d39c launcher.run: adaptation to new window layouter 2019-01-30 13:54:53 +01:00
Norman Feske
431c80bbca Add missing 'root' attribute to vfs policies 
The commit "os: avoid using deprecated APIs" tightens the policy
configuration of the VFS server such that the 'root' attribute is no
longer optional.
 2019-01-30 13:54:53 +01:00
Martin Stein
f3dfe88477 build system: show full paths on ADA errors 
Add '-gnatef' to the default 'CUSTOM_ADA_OPT' in generic.mk to ensure that, on
compile errors, the full source path is shown.
 2019-01-30 13:49:55 +01:00
Christian Prochaska
342ddcf71a libsanitizer: replace use of 'Genode::printf()' 
Fixes #3129
 2019-01-30 13:49:55 +01:00
Christian Prochaska
4b805ccde9 base: move 'Buffered_output' class into public header 
Fixes #3128
 2019-01-30 13:49:55 +01:00
Norman Feske
d3759811b6 os/session_policy: warn if no policy exists 
The situation where a 'Session_policy' is constructed for a label with
no matching policy is in almost all cases a configuration problem.
A diagnostic message eases pin-pointing such mistaks. By adding the
message to the 'Session_policy', servers don't need to manually handle
the exception to provide diagnostic information. This simplifies the
server code in many components.
 2019-01-30 13:49:55 +01:00
Emery Hemingway
f529871162 Add [depot_user] to run tool builtins 
Now that the depot_user procedure has found use in the depot_autopilot
it has become a convention.

Fix #3127
 2019-01-30 13:49:55 +01:00
Norman Feske
6154d9067e libports: avoid using deprecated APIs 
Issue #1987
Issue #3125
 2019-01-30 13:49:55 +01:00
Norman Feske
ba2b0b8360 gems: remove the use of deprecated APIs 
This patch also updates os/slave.h because the app/launcher cannot be
reasonably updated without it.

Issue #1987
Issue #3125
 2019-01-30 13:49:54 +01:00
Norman Feske
f23579532e dde_*: remove the use of deprecated APIs 
Issue #1987
Issue #3125
 2019-01-30 13:49:54 +01:00
Norman Feske
954aff7002 demo: avoid using deprecated APIs 
Issue #1987
Issue #3125
 2019-01-30 13:49:54 +01:00
Norman Feske
6b94e65a95 os: avoid using deprecated APIs 
Issue #1987
Issue #3125
 2019-01-30 13:49:54 +01:00
Norman Feske
cb36d96569 netperf: remove mechanism to update 'times_up' 
The mechanism relied on deprecated Genode APIs. To revive the mechanism,
we could use pthread_create.
 2019-01-30 13:49:54 +01:00
Norman Feske
486e534df0 gdb_monitor: remove use of deprecated APIs 
Issue #1987
 2019-01-30 13:49:54 +01:00
Norman Feske
5f1f67153b Xml_node: safe alternatives to unsafe accessors 
Issue #3125
 2019-01-30 13:49:54 +01:00
Norman Feske
8cc11d6cc6 libc_terminal: remove use of deprecated APIs 
Issue #1987
 2019-01-30 13:49:54 +01:00
Norman Feske
fdbf30fc4c Xml_node: replace value(T *out) by value(T &out) 
Promote the use of references instead of pointers.

Issue #1987
 2019-01-30 13:49:54 +01:00
Norman Feske
98a75b1a78 noux: remove use of deprecated APIs 
Issue #1987
 2019-01-30 13:49:54 +01:00
Norman Feske
d7e552a169 libusb: remove use of deprecated APIs 
Issue #1987
 2019-01-30 13:49:53 +01:00
Norman Feske
3da6aab353 noux: enable strict warnings 2019-01-30 13:35:29 +01:00
Norman Feske
e889c58bbc noux: remove dependency from alarm library 
Fixes #3123
 2019-01-30 13:35:29 +01:00
Martin Stein
ffd4e231d7 run/depot_autopilot: less interactive mode 
In less interactive mode, the run script doesn't give up on missing test
archives but instead removes the corresponding tests and marks them "missing".
This mode avoids total failure of a platform in automated test infrastructures
when only a few archives are missing.

Fixes #3120
 2019-01-30 13:35:29 +01:00
Norman Feske
af146e7dcd Remove base/timed_semaphore.h from API 
The former 'Genode::Timed_semaphore' mechanism is moved to the private
part of the two remaining users, namely dde_rump and the libc. Note
there are now two private copies of 'timed_semaphore.h'. This should be
regarded as an interim step until the use of this mechanism is removed
from both users.

This patch also cleans up the mechanism from legacy Genode API calls and
global side effects (alarm-thread singleton). The test/timed_semaphore
is now located at the libports repository as it now tests a mechanism of
the libc. The former timed_semaphore library is no more.

Fixes #3121
 2019-01-30 13:35:29 +01:00
Norman Feske
7b37546a4d tool/abi_symbols: omit known internal symbols 
This patch prevents the abi_symbols tool from generating symbols that
are known to occur in shared objects but must not be part of a library
ABI. This saves a bit of time during library-porting work.

However, to avoid the accidental use of ABI symbol definitions that lack
any form of manual curation, the abi_symbols tool outputs a special
message, which is explicitly checked-for by the check_abi tool.

Fixes #3112
 2019-01-30 13:35:29 +01:00
Christian Prochaska
a155d0e531 qt5: fix parallel build issues related to generated files 
Store all files generated by moc and rcc in the application's build
directory to prevent the use of unfinished generated files for other
applications built at the same time.

Issue #3115
 2019-01-30 13:35:28 +01:00
Martin Stein
e0af049124 depot_autopilot: fix regression with nr. of tests 
The output of the number of tests to run was removed accidentally with the
commit "depot_autopilot: provide repeat mode".
 2019-01-30 13:35:28 +01:00
Martin Stein
f69d3020fd remove unused repos/app-file_cache* files 
These files sneaked in accidentally with the commit "Support GNAT.IO in Ada
programs".
 2019-01-30 13:35:28 +01:00
Alexander Boettcher
9a2bdf8798 vm_session: move from base-hw to base 
Issue #3111
 2019-01-30 13:35:28 +01:00
Alexander Boettcher
9f8198d946 hw: deny to attach managed dataspaces to VMs 
Issue #3111
 2019-01-30 13:35:28 +01:00
Stefan Kalkowski
5b8e1cdbbb foc: add recipe for imx6q_sabrelite base lib 2019-01-30 13:35:28 +01:00
Stefan Kalkowski
b765cef359 replace Wandboard by i.MX6 reference board 
Instead of using the Wandboard Quad, the reference hardware from NXP
i.MX6 Quad Sabrelite will be used by Fiasco.OC, sel4 and hw by default.
 2019-01-30 13:35:28 +01:00
Stefan Kalkowski
c65860ee53 enable i.MX6 Quad Sabrelite board for hw and foc 2019-01-30 13:35:28 +01:00
Norman Feske
2f9da1c7c8 Road map for 2019 2019-01-30 13:35:28 +01:00
Norman Feske
3bd4197951 gems: enable strict warnings for more components 
Issue #465
 2019-01-30 13:35:28 +01:00
Christian Prochaska
21ed41da9a usb_hid_drv: wait for keyboard LED registry in 'led_connect()' 
It can happen that a keyboard gets plugged in and 'led_connect()' is
called while the keyboard LED of another keyboard is just being updated
(and the registry is locked).

Fixes #3118
 2019-01-30 13:35:28 +01:00
Norman Feske
ece5cbbbb5 gems: utility for implementing LRU-based caches 
This patch extracts the LRU cache implementation from the 'Cached_font'
so that it becomes reusable for other applications.

Fixes #3117
 2019-01-30 13:35:28 +01:00
Stefan Kalkowski
6b4d76739e base: remove Signal_receiver::pending() method 
Fix #1864
 2019-01-30 13:35:28 +01:00
Christian Prochaska
e8ea28d897 socket fs: support non-blocking 'connect()' 
Fixes #3113
 2019-01-30 13:35:28 +01:00
Christian Prochaska
12c10dbcd1 libc: 'connect()' test 
Fixes #3114
 2019-01-30 13:35:28 +01:00
Norman Feske
ed65267bc5 depot: update recipe hashes 2019-01-14 12:34:41 +01:00
Christian Helmuth
9dfad6d9da run: plugin for Allnet MSR powerplug 
The plugin was tested with ALL4176 but should support all modern Allnet
MSR devices according to the documentation

  https://service.allnet.de/image-ftp/ftp/pub/allnet/MSR/JSON/JSON_MSR.zip

It works just like the existing powerplug plugins by

  RUN_OPT += --include power_on/allnet --include power_off/allnet

and uses the following parameters

  --power-on-allnet-host        network address of device
  --power-on-allnet-user        user for device
  --power-on-allnet-password    password for device
  --power-on-allnet-port        target port/socket of device
  --power-off-allnet-host       network address of device
  --power-off-allnet-user       user for device
  --power-off-allnet-password   password for device
  --power-off-allnet-port       target port/socket of device
 2019-01-14 12:34:41 +01:00
Norman Feske
97e3d05f37 base-foc: fix placement of utcb area stack area 
This commit ensures that UTCB areas of PDs are positioned relative to
the stack areas of regular components, not the one of core.

Fixes #3108
 2019-01-14 12:34:41 +01:00
Martin Stein
3f60dcfae8 sel4 run/depot_autopilot: handle resource leak 
After a certain number of tests, presumably some resource in core is exceeded
and loading the successive test fails. This quickfix looks out for the
characteristic Core error and then reboots to avoid that all successive tests
are marked as failed.
 2019-01-14 12:34:41 +01:00
Christian Prochaska
8aaffe829a usb_hid.run: enable more platforms for autopilot test 
Fixes #3105
 2019-01-14 12:34:41 +01:00
Norman Feske
81fb10daaa Consistently name block components 
This patch replaces abbreviations like "blk", "cli", and "srv" by their
full forms "block", "client", and "server".

Fixes #1258
 2019-01-14 12:34:39 +01:00
Norman Feske
bf62d6b896 Move timer from os to base repository 
Since the timer and timeout handling is part of the base library (the
dynamic linker), it belongs to the base repository.

Besides moving the timer and its related infrastructure (alarm, timeout
libs, tests) to the base repository, this patch also moves the timer
from the 'drivers' subdirectory directly to 'src' and disamibuates the
timer's build locations for the various kernels. Otherwise the different
timer implementations could interfere with each other when using one
build directory with multiple kernels.

Note that this patch changes the include paths for the former os/timer,
os/alarm.h, os/duration.h, and os/timed_semaphore.h to base/.

Issue #3101
 2019-01-14 12:33:57 +01:00
Alexander Senier
14cd115c82 Support GNAT.IO in Ada programs 2019-01-14 12:21:10 +01:00
Christian Prochaska
119a12cba5 qt5: support relative paths in qmake project files 
Support paths like '../util.h' or 'resources/panel.qrc' in the HEADERS and
RESOURCES variables in qmake project files.

Fixes #3115
 2019-01-14 12:21:09 +01:00
Christian Helmuth
745ee04583 Window manager test for tiled-console scenario 2019-01-14 12:21:09 +01:00
Christian Helmuth
21a61cd583 depot: update recipe hashes 2019-01-07 12:43:37 +01:00
Norman Feske
60dc783399 motif decorator: enable closer in pkg runtime 2019-01-07 12:43:37 +01:00
Josef Söntgen
f113460348 Move FUSE to world 
Fixes #3104.
 2019-01-07 12:43:37 +01:00
Norman Feske
39085f08fc Move libav, avplay, and qt_avplay to genode-world 
Fixes #3103
 2019-01-07 12:43:37 +01:00
Norman Feske
a15b825418 Move libsdl and companion libs to genode-world 
Fixes #3100
 2019-01-07 12:43:37 +01:00
Norman Feske
48aed0ea46 Move Dosbox to genode-world 
Fixes #3099
 2019-01-07 12:43:37 +01:00
Norman Feske
76e96e92cb nitpicker: avoid color bleeding 
This patch improves the output of opaque pixels in the presence of an
alpha channel by adding a special case for the maximum alpha value.

Fixes #2831
 2019-01-07 12:43:23 +01:00
Norman Feske
97bfc13237 sculpt: slight visual improvements of leitzentrale 
This patch improves the appearance of the leitzentrale by eliminating
the (hardly visible) decorations from the GUI and graph views, and by
animating the motion of the graph position. The latter is meant to
remove the stuttering effect when the graph's size changes (and
re-centered).
 2019-01-07 12:38:46 +01:00
Norman Feske
a636f90240 wm: update hover after drag operation 
This patch improves the consistency of the hover handling after
finishing a drag operation. Normally, the window manager hides pointer
updates while the user is performing a drag operation with the mouse
from the decorator. However, in the special case where a drag operation
results in a window-layout change, the decorator's hover model may be
affected. Hence, the window manager must supply the current pointer
position when leaving the drag state.

Issue #3097
 2019-01-07 12:38:46 +01:00
Norman Feske
7c79bfc903 motif decorator: visual feedback to mouse clicks 
Issue #3097
 2019-01-07 12:38:46 +01:00
Norman Feske
49e0036471 window layouter: avoid superfluous layout updates 2019-01-07 12:38:45 +01:00
Norman Feske
9efb957059 window_layouter: propagate pressed window controls 
This patch supplements the dragging state of window controls to the
window layout so that decorators become able to visually reflect this
state, i.e., pressing the title bar while moving a window.

Issue #3097
 2019-01-07 12:38:45 +01:00
Norman Feske
2d95c4dc1c themed_decorator: new 'motion' policy attribute 
This commit adds the optional 'motion=<number>' attribute to the
decorator's <policy> nodes. The default value is 0. If a value higher
than 0 is specified, window-geometry changes are applied as an animation
where the <number> denotes the number of animation steps.

Issue #3096
 2019-01-07 12:38:45 +01:00
Norman Feske
a3bbef5f21 themed_decorator: optionally disable decorations 
This patch adds the boolean policy attribute "decoration", which
controls whether window decorations are presented or not. It is enabled
by default. By setting the attribute to "no", matching windows appear
without any border, which is desireable for Sculpt's component graph.

Issue #3096
 2019-01-07 12:33:57 +01:00
Norman Feske
296a409a29 gems: make menu_view's animated_geomerty.h public 
This commit moves menu_view's utility for animating rectangles available
at 'include/gems/animated_geometry.h'.

Issue #3096
 2019-01-07 12:33:57 +01:00
Norman Feske
b188a4dbc9 lazy_value.h: improve handling of low steps value 
Don't just clamp the speed value to a positive number but immediately
assign the destination value. This is needed in situations where the
number of steps is too low for proper acceleration and deceleration.

Issue #3096
 2019-01-07 12:33:57 +01:00
Norman Feske
2565928495 fb_sdl: support the resizing of the SDL window 
Fixes #3095
 2019-01-07 12:33:57 +01:00
Norman Feske
56cb1885bb decorator: make window-layout updates more robust 
This patch improves the window decorators in the following respects:

* Strict warnings are enabled now.
* The use of the 'List_model' makes the application of window-
  layout changes more robust. This is particularly the case for
  the restacking of windows.
* Display-mode changes are now supported by both decorators.

Issue #3094
 2019-01-07 12:33:57 +01:00
Norman Feske
f7d33010e5 gems: strict warning fixes 
Issue #3094
 2019-01-07 12:33:57 +01:00
Norman Feske
cd29ca3c40 base: add List_model::apply_first 
The new 'apply_first' method enables users of the list model to manually
traverse the list model via the 'Element::next' method instead of
iterating via 'for_each'. This is needed in situations where the
list-model elements are visited via recursion, not via a loop.

Issue #3094
 2019-01-07 12:33:56 +01:00
Norman Feske
554a100407 wm.run: build decorator, use wm.config from repo 
With these little tweaks, the run script becomes more convenient as a
testing ground.

Issue #3094
 2019-01-07 12:33:56 +01:00
Norman Feske
dc9cd38189 wm: support TO_BACK command issued by decorator 
Until now, decorators used to rely only on the TO_FRONT command for
propagating the window stacking to nitpicker. However, as the additional
use of the TO_BACK command allows for a more robust procedure, this
patch enhances the wm to handle both view stacking commands.

Issue #3094
 2019-01-07 12:33:56 +01:00
Norman Feske
b3fa7b0650 wm: enable strict warning level 
Issue #3094
 2019-01-07 12:33:56 +01:00
Stefan Kalkowski
7f1692b3ca core: support unmap of managed dataspace generally 
This commit solves several issues:

* correct calculation of overlap region when detaching regions
  in managed dataspaces
* prevent unmap of Fiasco.OC's core log buffer
* calculate the core-local address of regions in managed dataspaces
  if possible at all and use it to unmap on kernels where this is
  needed

Fix #976
Fix #3082
 2019-01-07 12:33:56 +01:00
Stefan Kalkowski
e31ded2198 foc: silence mapping warnings 
Ref #3082
 2019-01-07 12:33:56 +01:00
Christian Helmuth
56650e7f79 depot: prevent warning with newer GPG versions 
gpg: WARNING: no command supplied.  Trying to guess what you mean ...
 2019-01-07 12:33:56 +01:00
Alexander Boettcher
af710cdd7f sculpt: update browser VMs for 18.11 
use Firefox 64.0
 2019-01-07 12:33:56 +01:00
Martin Stein
f0842a27c5 depot_autopilot: provide repeat mode 
Adds an config attribute to the Depot Autopilot component:

:<config repeat>:

  Can be one of

    "false"         - process the given test list only once,
    "until_forever" - endlessly repeat processing the given test list,
    "until_failed"  - repeat processing the given test list until it fails.

Adds an environment variable to the Depot Autopilot Run script:

:TEST_REPEAT:

  Same as the <config repeat> attribute of the Depot Autopilot.

This is useful when having to debug very sporadic errors during one test
or a series of tests.
 2019-01-07 12:33:56 +01:00
Norman Feske
b7cb5839eb test/vfs_stress: prevent division by zero 
Issue #3090
 2019-01-07 12:33:56 +01:00
Norman Feske
ca6ee2d91a block_tester.run: fix use of ld attribute 2019-01-07 12:33:55 +01:00
Norman Feske
9818237918 test-ada: showcase Ada/SPARK object construction 2019-01-07 12:33:55 +01:00
Norman Feske
4f316cffbc init: update state reports on heartbeat responses 
Init's state reports are updated whenever an interesting part of init's
internal state changes (e.g., when sessions are established, or when
children are started/removed). However, until now, a change of a skipped
heartbeat counter was not taken as trigger for state-report updates.

In scenarios where no other intersting event happened, the last reported
state did no reflect the current heartbeat state. In particular, when
the last report was issued during the construction of a new child just
before the child became able to respond to heartbeat requests, the stale
report hinted at heartbeat problems that were just an initialization
artifact. This problem became visible on some Qemu platform where the
child startup takes a long time.

The patch tracks the observed skipped-heartbeat counter and triggers a
report whenever the counter value changes.

Issue #3079
 2019-01-07 12:33:55 +01:00
Martin Stein
68f6b9443e test/trace_logger: don't use Trace::timestamp 
At least on foc pbxa9, Trace::timestamp gets stuck.
 2019-01-07 12:33:55 +01:00
Stefan Kalkowski
c20c643b66 depot_autopilot: skip test-libc on rpi and sel4 
Fix #3086
 2019-01-07 12:33:55 +01:00
Norman Feske
ed7dfddc5d Increase timeout of init test 
The increased timeout takes the use of NOVA or seL4 on Qemu into
account.

Issue #3079
 2019-01-07 12:33:55 +01:00
Norman Feske
8f1f4f2652 Improve robustness of init test 
By adding an additional synchronization point in the form of a matched
log message, this patch makes the timing behavior of the "test changing
provided services" step more deterministic. Without it, the scheduling
of OKL4 and base-hw resulted in a merge of two config updates into one.

Issue #3079
 2019-01-07 12:33:55 +01:00
Christian Prochaska
46b68b0e66 gmp: search headers in REP_DIR first 
Fixes #3068
 2019-01-07 12:33:55 +01:00
Alexander Boettcher
035439c710 sel4: enable fpu for wand_quad 2019-01-07 12:33:55 +01:00
Norman Feske
89883a6988 Depot recipe for src/lighttpd 2019-01-07 12:33:55 +01:00
Norman Feske
89935e7308 gems: add pkg/drivers_nic-pc 2019-01-07 12:33:55 +01:00
Josef Söntgen
c43723abdd ports: update lighttpd to 1.4.52 
In addition enable TLS.

Fixes #3069.
 2019-01-07 12:33:55 +01:00
Josef Söntgen
0139fa20ff libports: add tcp_fsm header to libc includes 
Needed by lighttpd-1.4.52

Issue #3069.
 2019-01-07 12:33:54 +01:00
Josef Söntgen
c58ad11f2f libports: update OpenSSL to 1.0.2q 
And enable SSL_CONF_* in libssl, needed by lighttpd's mod_openssl.

Issue #3069.
 2019-01-07 12:33:54 +01:00
Stefan Kalkowski
5147c71fdf qt5: update qtscriptclassic archive path 2019-01-07 12:33:54 +01:00
Norman Feske
9ca42448c7 doc: grammar fix in the release notes 18.11 
Thanks to Jeroen van Gelderen for reporting!
 2019-01-07 12:33:54 +01:00
Christian Prochaska
a5547e5b1d base: improve floating point output 
Fixes #2876
 2019-01-07 12:33:54 +01:00
Sebastian Sumpf
3347d08b79 ldso: cleanup if loading of 'Shared_object' fails 
This can happen, for example, during 'dlopen' if unresolved symbols are
present.

* Unload already loaded shared libraries
* Delete dependencies
* Flush initializer list (ctors)

fixes #3073
 2019-01-07 12:33:54 +01:00
Josef Söntgen
7c4986bd83 dde_rump: remove out-dated CGD test 2019-01-07 12:33:54 +01:00
Josef Söntgen
108034b050 packet_stream: packets w/o payload are still valid 
Packets whose data is stored within the Packet_descriptor itself
but not as payload, .e.g Usb::Packet_descriptor, are valid packets
after all. So loosen the packet valid check for zero-sized packets
is reasonable.

Fixes #3076.
 2019-01-07 12:33:54 +01:00
Robin Eklind
6b35b9de00 doc: prevent line break between the* and *health* 
Fixes #3077
 2019-01-07 12:33:54 +01:00
Christian Prochaska
9c8e76b190 base: 'sanitizer' test 
Issue #3072
 2019-01-07 12:33:54 +01:00
Christian Prochaska
5569d2ddc2 mk: add 'SANITIZE_UNDEFINED' option 
'SANITIZE_UNDEFINED = yes' in 'target.mk' adds the '-fsanitize=undefined'
compiler flag and links the program with libubsan and libsanitizer_common.

Issue #3072
 2019-01-07 12:33:54 +01:00
Christian Prochaska
c2884a6e63 libports: libubsan and libsanitizer_common 
Issue #3072
 2019-01-07 12:33:54 +01:00
Norman Feske
03cbd4257f sequence: add 'repeat="yes"' config option 
Fixes #3075
 2019-01-07 12:33:54 +01:00
Norman Feske
d46b63a18c extract: allow stripping of leading path elements 
Fixes #3074
 2019-01-07 12:33:54 +01:00
Christian Helmuth
0dc4335fbc run: fix custom image disk size calculation 2019-01-07 12:33:53 +01:00
Martin Stein
f213a07c83 run/depot_autopilot: always log available results 
This ensures that the depot_autopilot.run script, when exiting, always prints
a result overview of the so far available test results, except the Depot
Autopilot component has managed to print the result overview up to this point.
 2019-01-07 12:33:50 +01:00
Stefan Kalkowski
59a8856773 depot_autopilot: sanitize XML results 
Unequal numbers of double quotes let the XML parser of init fail,
therefore replace all double quotes when transfering previous
results after a reboot.

Ref #3027
 2019-01-07 12:30:40 +01:00
Stefan Kalkowski
88b704db19 depot_autopilot: reduce timeouts of test pkgs 
Our overall nightly test time greatly decreases when the timeouts for
the single tests are not that over-pessimistic. Using the slowest
platforms as reference, this commit reduces the test timeouts.

Ref #3027
 2019-01-07 12:30:40 +01:00
Martin Stein
19e928271b run/depot_autopilot: reboot on some kernel faults 2019-01-07 12:30:37 +01:00
Martin Stein
54d1a676b7 run/depot_autopilot: reboot on log EOF 2019-01-07 12:25:46 +01:00
Martin Stein
f3a520e75d run/depot_autopilot: do not run on foc+panda 2019-01-07 12:25:46 +01:00
Martin Stein
7ad00d1152 base-hw/recipes: src/base-hw-muen 2019-01-07 12:25:46 +01:00
Christian Helmuth
600a5ecdaf hw: log stack pointer on x86 CPU exception 2019-01-07 12:25:46 +01:00
Emery Hemingway
bd53e5b496 VFS lwIP: send application data immediately 
Call 'tcp_output' if application data has been successfully queued. This
sends data immediately that may otherwise remain queued until the next
periodic TCP timer event.

This reverts a change made in 3e31e2ba53.

Fix #3067
 2019-01-07 12:25:46 +01:00
Martin Stein
a79bfad08e base-linux: init trace control 2019-01-07 12:25:46 +01:00
Martin Stein
f8a27e6acf pkg/test-init_loop: raise timeout to 150s 
At least autopilot qemu x86_64 foc needs more time.
 2019-01-07 12:25:46 +01:00
Christian Prochaska
73e3ed0bd0 gcov: print annotated source only on incomplete coverage 
Fixes #3071
 2019-01-07 12:25:46 +01:00
Christian Prochaska
97d8bea5ec gcov: print selected annotated source files only 
Fixes #3070
 2019-01-07 12:25:46 +01:00
Martin Stein
abe80a4bfe run/depot_autopilot: raise initial timeout to 40s 
At least sel4 on qemu/x86_32 on autopilot needs more time.
 2019-01-07 12:25:45 +01:00
Martin Stein
6a8a5d2167 run/depot_autopilot: print nr of tests to run 
The number of tests to run is the number of test package-archives minus the
the those that are skipped for the given platform. The number is printed
directly after checking if the given platform is supported by the run script.
It helps the surrounding test infrastructure to ensure that, for instance, a
result graph always reflects the same total number of tests, even though there
is a sporadic problem with booting the platform.
 2019-01-07 12:25:45 +01:00
Martin Stein
da3e5fd3d6 run/depot_autopilot: reset qemu args correctly 2019-01-07 12:25:45 +01:00
Alexander Boettcher
0a0a6ef591 nova: de-schedule SCs in destruction earlier 
Fixes issues detected during destruction in

Issue #3041
 2019-01-07 12:25:45 +01:00
Alexander Boettcher
85b998a4af os: nullpointer check in os/path.h 
Issue #3022
 2019-01-07 12:25:45 +01:00
Alexander Boettcher
67fd3333e2 base: avoid null pointer reference warning 
Issue #3022
 2019-01-07 12:25:45 +01:00
Alexander Boettcher
518d157f76 os: avoid null pointer acces in alarm lib 
Issue #3022
 2019-01-07 12:25:45 +01:00
Alexander Boettcher
1ec0619b3e base: avoid warnings in fifo.h 
error: member 'Element' found in multiple base classes of different types

Issue #3022
 2019-01-07 12:25:45 +01:00
Alexander Boettcher
dbeb1b82a9 os: avoid warning in ethernet.h 
issued by clang/llvm static analyzer
 2019-01-07 12:25:45 +01:00
Alexander Boettcher
aa03c4ce9f os: avoid garbage warnings in app/painter 
Issue #3022
 2019-01-07 12:25:45 +01:00
Alexander Boettcher
5572430ba5 os: avoid ambiguous warnings for vfs/server 
between File_system and Vfs::File_system

Issue #3022
 2019-01-07 12:25:45 +01:00
Alexander Boettcher
9bb0e10eec os: avoid warning in platform driver 
calling wrong destructor issued by clang/llvm static analyzer

Issue #3022
 2019-01-07 12:25:45 +01:00
Alexander Boettcher
c2d54aaede base: avoid null pointer warning in avl tree 
Issue #3022
 2019-01-07 12:25:45 +01:00
Alexander Boettcher
7536b665f1 core: avoid null pointer reference warning 
Issue #3022
 2019-01-07 12:25:44 +01:00
Martin Stein
8db02b0a39 run/depot_autopilot: env-variables user-interface 2019-01-07 12:25:44 +01:00
Stefan Kalkowski
8e13b376b0 hw: improve cross-cpu synchronization 
This commit addresses several multiprocessing issues in base-hw:

* it reworks cross-cpu maintainance work for TLB invalidation by
  introducing a generic Inter_processor_work and removes the so
  called Cpu_domain_update
* thereby it solves the cross-cpu thread destruction, when the
  corresponding thread is active on another cpu (fix #3043)
* it adds the missing TLB shootdown for x86 (fix #3042)
* on ARM it removes the TLB shootdown via IPIs, because this
  is not needed on the multiprocessing ARM platforms we support
* it enables the per-cpu initialization of the kernel's cpu
  objects, which means those object initialization is executed
  by the proper cpu
* it rollbacks prior decision to make multiprocessing an aspect,
  but puts back certain 'smp' mechanisms (like cross-cpu lock)
  into the generic code base for simplicity reasons
 2019-01-07 12:25:44 +01:00
Christian Helmuth
8236a18260 Revert "Use strictly-typed Microseconds for Libc timeout scheduling" 
This reverts commit 4808565a28afe9ff248fb5c98aceb6f8d3e791c1.
 2019-01-07 12:25:44 +01:00
Christian Helmuth
e1b27885f9 solo5: needs IRQ for timer on sel4 2019-01-07 12:25:44 +01:00
Stefan Kalkowski
1d0e063f49 base-sel4: recipe for Wandboard platform 
* Allow depot_autopilot to be run on top of sel4/wand_quad too (Ref #3027)
 2019-01-07 12:25:44 +01:00
Stefan Kalkowski
76966ed61a depot_autopilot: add route to IO_MEM for timer 
* some timer drivers like epit for i.MX* need I/O memory access

Ref #3027
 2019-01-07 12:25:44 +01:00
Christian Helmuth
6315e4503e dde_linux: adapt to changes in os/duration.h 
The Linux emulation library provides preprocessor macros for min() and
max(), which now clash with implementation in duration.h. So, we disable
those macros in the delay implementation.
 2019-01-07 12:25:44 +01:00
Alexander Boettcher
eead1af140 sel4: flush tlb on x86 
Issue #3041
 2019-01-07 12:25:43 +01:00
Stefan Kalkowski
6fb9c802d3 depot: enable package building for armv6 
* Allow depot_autopilot to be run on top of arm_v6 too (Ref #3027)
 2019-01-07 12:25:43 +01:00
Stefan Kalkowski
42c5f2e91e hw: add src package for base-hw-zynq_qemu 
* Needed for nightly depot_autopilot tests (Ref #3027)
 2019-01-07 12:25:43 +01:00
Stefan Kalkowski
216dc49741 depot_autopilot: skip test-libc with low memory 
* Skip test-libc on top of foc_pbxa9 and hw_imx53_tz as both platforms
  have to few memory in their configuration
 2019-01-07 12:25:43 +01:00
Martin Stein
271fd0ba06 run/depot_autopilot: show depot size 2019-01-07 12:25:43 +01:00
Martin Stein
858a7823fb init: print label on "no route to service" warning 
It can be hard to interpret "no route to service" warnings if Init doesn't
print the session label
 2019-01-07 12:25:43 +01:00
Martin Stein
79ca4e1718 init: show args on "denied env session" error 
It can be hard to resolve "denied env session" errors when you don't have the
session label.
 2019-01-07 12:25:43 +01:00
Martin Stein
7f959a06f6 depot_autopilot: tune for multi-platform support 
Issue #3027
 2019-01-07 12:25:43 +01:00
Christian Prochaska
ac8d030855 gcov: open files in buffered mode 
Fixes #3065
 2019-01-07 12:25:43 +01:00
Emery Hemingway
f2df40f58b Add simple Solo5 tests to depot autopilot 
Add the following Solo5 tests to Autopilot: hello, fpu, globals, quiet,
blk. The remaining tests require a Rtc service or IP routing.

Ref #3027
 2019-01-07 12:25:43 +01:00
Emery Hemingway
5c2599e24e Remove 'clock()' implementation, print warning 
FreeBSD implements 'clock' with an accuracy of 128 ticks-per-second for
compatibility reasons, Linux uses 1000000 per-second. Remove 'clock' and
print an error because it is unlikely that this is the resolution
expected by the application.

Fix #3057
 2019-01-07 12:25:43 +01:00
Martin Stein
82ded858aa nic_bridge: fix bug when reading MAC address 
For reading the MAC address we try first to read it from the <policy> tag, and
when it is not defined in the <policy> tag, we allocate a MAC. But there was
no handling of the case that there is no appropriate <policy> tag. In this
case we want to create the session with an allocated MAC also.
 2019-01-07 12:25:42 +01:00
Emery Hemingway
8cb8082206 Remove Nim support from toolchain 
Nim components are best build externally using the Nimble tools and a
Genode SDK.

See https://github.com/ehmry/nim-genode

Fix #2949
 2019-01-07 12:25:42 +01:00
Stefan Kalkowski
aeb7ab4774 hw: prevent potential dead-lock in signal destruction 
Fix #3063
 2019-01-07 12:25:42 +01:00
Emery Hemingway
8a3b0ebea9 Use strictly-typed Microseconds for Libc timeout scheduling 
Fix #3050
 2019-01-07 12:25:42 +01:00
Emery Hemingway
9c7d5b2a66 Implement print and min/max for Microseconds and Milliseconds 
Ref #3050
 2019-01-07 12:25:42 +01:00
Norman Feske
8f43a1303b News item for version 18.11 2018-11-29 14:48:01 +01:00
Christian Helmuth
d69cb1ba7b version: 18.11 2018-11-29 14:15:25 +01:00
Norman Feske
456dda9ac0 Release notes for version 18.11 2018-11-29 14:12:41 +01:00
Christian Helmuth
11eecdc7bd depot: update recipe hashes 2018-11-29 11:54:31 +01:00
Stefan Kalkowski
34480c9269 test-rm_fault: use ld.lib.so to test read-only mem 
Due to the changing environment this test is executed (like depot_autopilot)
the binary test-rm_fault used to check ROM dataspace's read-only property
does not suit anymore. This commit changes the binary to ld.lib.so that is
normally still provided as a ROM dataspace.

Ref #3027
 2018-11-29 11:54:31 +01:00
Stefan Kalkowski
fa67ec52ae cpu_sampler_noux: do not test platforms w/o pkg 
Platform without driver-interactive package will fail. Therefore,
whitelist which platforms do work in the run-script
 2018-11-29 11:54:31 +01:00
Stefan Kalkowski
41dfc51beb test-expat: use zero-filled read buffer 
Fix #3062
 2018-11-29 11:54:31 +01:00
Stefan Kalkowski
9857a0c956 depot_autopilot: disable test-python for ARM 
The test for python requires x86 to be built. Therefore, there is no test
binary available when trying to execute that test on ARM with depot_autopilot.
 2018-11-29 11:54:31 +01:00
Johannes Kliemann
db162477a4 ada: fix exception handling 
Fixes #3061
 2018-11-29 11:54:31 +01:00
Christian Prochaska
3b1653dad7 sculpt: update Arora launcher for release 18.11 2018-11-29 11:54:31 +01:00
Stefan Kalkowski
d7fa4cfb8b hw: enable eager FPU context switch for ARM 
* Add an ieee754 FPU test
* Remove simple fpu test

Fix #2822
 2018-11-29 11:54:31 +01:00
Norman Feske
4b4247f412 window layouter: limit maximized size to client 
This patch constraints the window size of the generated layout to the
minimum of the client's real window size and the wanted window size
(both may differ when resizing or maximizing windows).
 2018-11-29 11:54:31 +01:00
Norman Feske
08bb689ae7 window layouter: bring unknown windows to front 
This patch improves the handing of new appearing windows for which only
a wildcard assignment - but no exact assignment - rule exists. In the
prior version, an interactively raised window would stay in front of
such a window, which is unintuitive. The new version applies the
to-front mechanism to unknown new windows. For known new windows (with
an exact assignment rule) their original stacking position is preserved.
 2018-11-29 11:54:30 +01:00
Norman Feske
4145417f67 window_layouter: improve window-resize handling 
This patch solves an off-by-one problem in the window-size calculation,
which resulted in sporadic artificial resize requests. In Sculpt, this
glitch caused flickering artifacts in VirtualBox windows caused by
superfluous guest desktop-resize handling.

Furthermore, the patch introduces the dropping of resize requests with
unchanged content.
 2018-11-29 11:54:30 +01:00
Josef Söntgen
af5869cd07 sculpt: update download_debian launcher for 18.11 2018-11-29 11:54:30 +01:00
Alexander Boettcher
876f60169c sculpt: update top_view to 18.11 2018-11-29 11:54:30 +01:00
Alexander Boettcher
4ea98dfee4 sculpt: update browser VMs for 18.11 2018-11-29 11:54:30 +01:00
Emery Hemingway
376f086ec1 Split off graphical Mirage run into mirage_pretty 
Ref #2945
 2018-11-29 11:54:30 +01:00
Norman Feske
005d6d6b35 sculpt: trim config/deploy default 
The commented-out <start> nodes are prone to become inconsistent with
the launchers. Hence, this patch removes them. Start nodes should better
be added by the '+' menu, at least initially.

Furthermore, the patch directs requests for the vfs.lib.so ROM to core's
ROM service to reduce the impact of low-level ABI changes (i.e., the
packet-stream layout) on existing vfs/libc-based packages.
 2018-11-29 11:54:30 +01:00
Martin Stein
89020bc3c0 depot_autopilot: move RTC test to autopilot.list 2018-11-29 11:54:30 +01:00
Norman Feske
538c3a6692 init: consider lacking service name attribute 
This patch makes init robust against invalid routing rules that lack a
name attribute in the '<service>' node. This situation may occur when
interactively editing routes in Sculpt.
 2018-11-29 11:54:30 +01:00
Sebastian Sumpf
8015dbe8b7 Move OpenJDK to the Genode world repository 
This reverts commit 58fb5ed722.

fixes #3060
 2018-11-29 11:54:30 +01:00
Sebastian Sumpf
d3dbdae395 libc: add SO_LINGER and TCP_NODELY to socket fs plugin 
issue #3060
 2018-11-29 11:54:30 +01:00
Sebastian Sumpf
48e07d7dff vfs: support files names of with exactly 100 characters 
GNU tar does not null terminate or create a long name for file names of
exactly 100 characters.

issue #3060
 2018-11-29 11:54:30 +01:00
Emery Hemingway
2d17af9f28 Libc: use caller integer width for return value of sysctl PHYSMEM 
Return a value in the same width as provided by the caller of sysctl for
PHYSMEM and USERMEM. This is to ensure that if a caller provides a
64-bit integer, a 64-bit value will be returned for 32-bit machines.

issue #3060
 2018-11-29 11:54:29 +01:00
Norman Feske
98518e39cd drivers_interactive-pc: increase ps2_drv caps 
This is needed because the added heartbeat monitoring.
 2018-11-29 11:54:29 +01:00
Christian Helmuth
af49ec7583 Support custom DEPOT_DIR in noux_tool_chain_auto.run 2018-11-29 11:54:29 +01:00
Emery Hemingway
a20d37c50a Add [depot_user] hook to depot_autopilot 2018-11-29 11:54:29 +01:00
Emery Hemingway
a2bdcc68c2 Throw exception for invalid packets at packet streams 
Some application code is dereferencing the pointer returned by
'packet_content' at packet streams without checking that it is valid.
Throw an exception rather than return a null pointer, except for
zero-length packets, which have somewhat implicit invalid content and
that we believe to be properly handled in all current cases.

The client-side of a packet stream cannot take corrective action if the
server-side is sending packets with invalid content, but the servers
that provide packet streams should catch this exception to detect
misbehaving clients.

Ref #3059
 2018-11-29 11:54:29 +01:00
Norman Feske
e5d1d26535 sculpt: version 18.11 2018-11-29 11:54:29 +01:00
Sebastian Sumpf
91225fbcca qt5: forward window title to nitpicker 
This enables Qt5 applications to set a Genode label via 'setWindowTitle'
from within Qt5 applications, and thus, making them identifiable to
other Genode components, like a layout manager.

fixes #3046
 2018-11-29 11:54:29 +01:00
Norman Feske
13be339d81 sculpt: let window layouter recall its prior state 
Issue #3024
Issue #3031
 2018-11-29 11:46:02 +01:00
Norman Feske
b89cfa95e7 sculpt: launcher for recall_fs 
The recall file system is a place where components can remember state.
E.g., to allow the window layouter to recall the window layout of the
previous session.
 2018-11-29 11:46:02 +01:00
Norman Feske
6c70a51d28 sculpt_manager: improved popup toggling 
In a corner case, the toggling of the popup menu entered a state
where the menu could not be opened anymore by the user. Specifically
the following input sequence triggered this problem.

1. The user opens the menu
2. The user clicks on the menu and holds the button
3. While holding the button, the user moves the pointer to the
   outside of the popup (e.g., to the '+' button)
4. The user releases the button.

In this situation, the popup is closed but the hover information for the
popup contains still the original clicked-on item. Hence, all subsequent
clicks on the '+' appear as both a click on the '+' (opening the popup)
and a click on the "hovered" popup entry (closing the popup).

The patch explicitely clears the popup's hover information when closing
the popup.
 2018-11-29 11:46:02 +01:00
Emery Hemingway
7f1974e9e8 Update lwIP to 2.1.2 
Fix #3035
 2018-11-29 11:46:02 +01:00
Alexander Boettcher
9f4801363c tool: use static analyzer with depot creation tool 
Convince wrapper to make static analyzer working with tool/depot/* tools.

Issue #3022
 2018-11-29 11:46:02 +01:00
Alexander Boettcher
8199b3e685 tool: support static clang analyzer in build.mk 
Issue #3022
 2018-11-29 11:46:02 +01:00
Emery Hemingway
918281b01f Genode SDK 
Makefile for generating a Genode SDK.

Fix #2948
 2018-11-29 11:46:02 +01:00
Emery Hemingway
447329eaee Chroot: change root to explicit prefix and label sub-dirs 
Change the root of a session request into an explicit path and apply the
label-to-path conversion using the "path_prefix" policy attribute. This
is in addition to only applying a root change with a "path" attribute.

Ref #3031
Fix #3056
 2018-11-29 11:46:02 +01:00
Johannes Schlatow
74f2954013 packet_stream: improve buffer alignment 
The bulk buffer is now 64Byte-aligned so that the allocated
packets get aligned likewise (assumed the packet allocator uses an
appropriately aligned block size). This ensures that each packet
starts at a new cache line on common platforms.

Issue #3053
 2018-11-29 11:46:02 +01:00
Johannes Schlatow
4e375ec6df zynq: move RAM size to board defs 2018-11-29 11:46:02 +01:00
Johannes Schlatow
dc0bfd7008 zynq: zero-copy implementation of nic_drv 
- Packet stream buffers are directly passed to DMA.
- Also enables pause frames and checksum offloading.

Issue #3053
 2018-11-29 11:46:01 +01:00
Emery Hemingway
8ad56a6c0e Move libc headers to conform to x86_32, x86_64, and arm SPECS 
Move the libc-i386, libc-amd64, and libc-arm include directories into
the standard "include/spec" directory. This allows the platform specific
headers in the libc API package to be detected in a generic manner.

Ref #3051
 2018-11-29 11:46:01 +01:00
Emery Hemingway
7a11384177 Merge pthread into libc library 
The pthread API is considered a standard feature of libc so better to
simply merge it with the libc. Pthreads are in fact already a part of
the libc in the form of weak symbols. This merger is also a prerequisite
for better integrating pthreads with the libc I/O task.

Fix #3054
 2018-11-29 11:46:01 +01:00
Christian Prochaska
fe322b8e82 test-xml_generator: enable code coverage analysis 
Issue #3048
 2018-11-29 11:46:01 +01:00
Christian Prochaska
5639f31295 depot_autopilot.run: integrate gcov tool 
Issue #3048
 2018-11-29 11:46:01 +01:00
Christian Prochaska
651d9f587f depot.inc: 'append_src_and_api_depot_packages_to_tar' function 
This function adds the src and api depot packages for the given pkg or src
packages to the given tar archive.

Issue #3048
 2018-11-29 11:46:01 +01:00
Christian Prochaska
1f985dec38 mk: add 'COVERAGE' option 
'COVERAGE=yes' in 'target.mk' adds gcov-specific compiler flags and links the
program with libgcov.

Issue #3048
 2018-11-29 11:46:01 +01:00
Christian Prochaska
e74771e047 libports: add libgcov and gcov tool 
Issue #3048
 2018-11-29 11:46:01 +01:00
Emery Hemingway
18e855e4d5 Genode toolchain depot package 
Fix #3028
 2018-11-27 11:38:13 +01:00
Emery Hemingway
15039f3ae8 Update MPFR port URL 
Ref #3028
 2018-11-27 11:38:13 +01:00
Emery Hemingway
63b6e04dae Move GMP spec directories to standard locations 
Ref #3028
 2018-11-27 11:38:13 +01:00
Christian Prochaska
fd7ab79fe0 vfs: implement 'complete_sync()' in 'Single_file_system' 
Fixes #3047
 2018-11-27 11:38:13 +01:00
Ben Larson
cde542c37c libc: read whole file into buffer 
Fix #3023
 2018-11-27 11:36:36 +01:00
Martin Stein
789d908cee depot_autopilot: show result statistic 
Print a line like "succeeded: 35 failed: 11 skipped: 2" below the list of test
results. Adds further attributes to <previous-results> to communicate also the
previous statistics.
 2018-11-27 11:36:36 +01:00
Martin Stein
c405ec19ce pkg/test-rm_fault: event "Error: could modify ROM" 2018-11-27 11:36:36 +01:00
Martin Stein
c65a13b3a4 run/depot_autopilot: power off before reboot 2018-11-27 11:36:36 +01:00
Johannes Kliemann
eb7c367e25 ada: replace local runtime implementation with port 
Fixes #3044
 2018-11-27 11:36:36 +01:00
Christian Helmuth
dee4d43eb9 Use common abort message in solo5 run scripts 
This enables our nightly tools to detect this abort is not fatal but
because of an unsupported platform.
 2018-11-27 11:36:35 +01:00
Ben Larson
1039ef7a65 qt5: avoid unresponsive window when aborting close 
Fixes #2997
 2018-11-27 11:36:35 +01:00
Norman Feske
c6fd0055b1 sculpt: split window manager into multiple pkgs 
This commit moves the window layouter and window decorator into
dedicated packages that can now be combined with the "wm" server at
runtime and restarted/reconfigured/swapped-out independently.

To use the window manager, one must start the 'wm', 'window_layouter',
and one of the 'motif_decorator' or 'themed_decorator' subsystems.

Fixes #3024
 2018-11-27 11:36:35 +01:00
Martin Stein
eab7f54139 depot_autopilot: skip libc_getenv on foc x86 
When doing the libc_getenv test on autopilot+foc+x86 and one of the
subsequent tests crashes the system so it gets rebooted by the run
script, the system doesn't come up again. It gets stuck after core
initialization.

Issue #3027
 2018-11-27 11:36:35 +01:00
Martin Stein
abd3855161 test/rm_nested: fix EP double dissolve 
Fixes #3045
 2018-11-27 11:36:35 +01:00
Christian Helmuth
712df01341 libports: update hash of downloaded fatfs sources 
The author updated http://www.elm-chan.org/fsw/ff/arc/ff13b.zip recently
with a marginal change of documents/doc/appnote.html.
 2018-11-27 11:36:35 +01:00
Emery Hemingway
e63a870bce Update lwIP to 2.1.1 
Fix #3035
 2018-11-27 11:36:35 +01:00
Martin Stein
e0b7fb1929 nic_bridge: fixed MAC addresses 
Enable configuration of a fixed MAC address for each client.

Fixes #3040
 2018-11-27 11:36:35 +01:00
Stefan Kalkowski
a062ba6dd2 base: add tlb shootdown test metric to smp test 
Fix #3041
 2018-11-27 11:36:35 +01:00
Stefan Kalkowski
36fe50ebad base: unify mp_server and affinity test 
* remove outdated cpufreq test for Arndale
* execute new SMP test on hardware not in Qemu in nightly tests

Ref #3041
 2018-11-27 11:36:35 +01:00
Norman Feske
19d7a488de init: health monitoring of child components 
Fixes #3039
 2018-11-27 11:36:34 +01:00
Stefan Kalkowski
d56a7beadc hw: increase cpu frequency on Wandboard Quad 
Ref #1807
 2018-11-16 15:17:06 +01:00
Stefan Kalkowski
8c460b3ea5 hw: enable l2-cache on Wandboard Quad (fix #1807) 2018-11-16 15:17:06 +01:00
Norman Feske
4cffefe1dd News item about Genode Component Public License 2018-11-16 15:07:53 +01:00
Christian Helmuth
70e7499e48 depot: update recipe hashes 2018-11-16 15:07:53 +01:00
Martin Stein
2d041ac0fc depot_autopilot: evaluate sets of test packages 2018-11-16 15:07:53 +01:00
Martin Stein
146f45f3d4 Archive recipes for autopilot tests 2018-11-16 15:07:53 +01:00
Martin Stein
6a540652fa pkg/test-fs_report: add autopilot events 2018-11-16 15:07:53 +01:00
Martin Stein
0c60b312be liports/recipes: api/ and src/python 2018-11-16 15:07:53 +01:00
Martin Stein
21efcf9e45 os/recipes: src/nic_bridge 2018-11-16 15:07:53 +01:00
Emery Hemingway
6c8f1c8796 Add Terminal provider rules to depot deploy and depot autopilot 
Support the '<terminal/>' tag in the depot package runtime '<provides/>'
declaration.

Fix #3034
 2018-11-16 15:07:52 +01:00
Emery Hemingway
bc539ce892 Native Solo5 bindings 
A shared library implementation of the unikernel middleware.

https://github.com/Solo5/solo5

Fix #2945
 2018-11-16 15:07:52 +01:00
Emery Hemingway
5a4dab88d6 PDF viewer: revert changes to framebuffer model 
The PDF viewer can crash under a number of conditions due to changes to
the internal framebuffer model that were made to support mouse wheel
panning. These changes are reverted until the model can be reimplemented
to explicitly support page panning.

Fix #3021
 2018-11-16 15:07:52 +01:00
Norman Feske
1a75c5227e Rename wm pkg to motif_wm 
This way, we can subsequently introduce a new 'wm' pkg that contains the
window manager only, w/o any hard-wired decorator and layouter.

Issue #3024
 2018-11-16 15:07:52 +01:00
Christian Helmuth
61863e2ffb qt5: configurable layouter/decorator in run scripts 2018-11-16 15:07:52 +01:00
Norman Feske
c60604062c decorator: improve robustness of window restacking 
This patch improves the detection of new appearing top-most windows.
Such a window should prompt the decorator to bring the corresponding
nitpicker view(s) to the front of the view stack. The original
implementation relied on hints provided by the layouter (the 'topped'
attribute). With the patch, the decorator tracks the top-most window by
itself, which improves the robustness.

As a second improvement, the patch defers the destruction of windows to
the point when all other window operations are completed. This hides
intermediate states when replacing one window by another in one step,
which is typical for console-like scenarios. Hence, this patch should
eliminate flickering artifacts when switching from one virtual console
to another.

Issue #3031
 2018-11-16 14:53:26 +01:00
Norman Feske
a973d9902b gems: flexible window layouter 
This commit replaces the former floating_window_layouter with a new
window_layouter component that supports the subdivision of screen space
into columns and rows, the concept of layers, and the principle ability
to store window layout information across reboots. The latter is
accomplished by reflecting the component's internal state as a 'rules'
report to the outside.

Fixes #3031
 2018-11-16 14:53:20 +01:00
Martin Stein
5bb5a62d37 timeout_types.xsd: allow 0 seconds 
The previous range was from 1..X because we only used it for durations. But
with the Depot Autopilot we want to use it also for a timestamp.
 2018-11-16 14:37:47 +01:00
Martin Stein
a1ff3cc317 base-foc: recipes for ARMv7a platforms 
* In base-foc/recipes/src/ replace base-foc with base-foc-pc
* To base-foc/recipes/src add base-foc-arndale, base-foc-pbxa9
* Ensure that the correct base-foc recipe is choosen by the run module
  'boot_dir/foc'
 2018-11-16 14:37:47 +01:00
Martin Stein
655fbbd984 base-hw: recipes for ARMv7a platforms 
* To base-hw/recipes/src add base-hw-arndale, base-hw-imx53_qsb,
  base-hw-imx53_qsb_tz, base-hw-odroid_xu, base-hw-panda, base-hw-rpi,
  base-hw-wand_quad
* Ensure that the correct base-hw recipe is choosen by the run module
  'boot_dir/hw'
 2018-11-16 14:37:20 +01:00
Norman Feske
f93294975b base-linux: avoid double define of __always_inline 
This patch fixes the following build problem that occurs on recent
GNU/Linux systems:

  /usr/include/x86_64-linux-gnu/sys/cdefs.h:307:0: error: "__always_inline" redefined
  ...
  /usr/include/linux/stddef.h:5:0: note: this is the location of the previous definition

The code in 'cdefs.h' does not check if the '__always_inline' is already
defined. The patch works around the problem by including the 'sys/cdefs.h'
first.
 2018-11-16 14:37:20 +01:00
Martin Stein
ee91d7339c run: prevent duplicates in missing archive warning 2018-11-16 14:37:20 +01:00
Martin Stein
941f785b7f Add tests to autopilot.list 2018-11-16 14:37:20 +01:00
Martin Stein
f64ec500bf run/rtc: assert spec x86 and !linux 2018-11-16 14:37:20 +01:00
Martin Stein
36932ee129 run/fs_packet: raise test timeout 2018-11-16 14:37:20 +01:00
Martin Stein
0c70a69618 run/dynamic_config_slave: fix cap quota 2018-11-16 14:37:20 +01:00
Martin Stein
0b3351991a run/dynamic_config_loader: fix cap quota 2018-11-16 14:37:20 +01:00
Martin Stein
99d9830968 run/dynamic_config: fix caps attribute 2018-11-16 14:37:20 +01:00
Martin Stein
719848d59a run/trace_logger: remove unnecessary stuff 2018-11-16 14:37:20 +01:00
Martin Stein
fff80bfe6c run/terminal_crosslink: fix cap quota 2018-11-16 14:37:20 +01:00
Martin Stein
4e29c1ca40 test/xml_node: move to base 2018-11-16 14:37:19 +01:00
Martin Stein
ed1cec2efa test/xml_generator: move to base 2018-11-16 14:37:19 +01:00
Martin Stein
510d7644fa test/weak_ptr: move to os/ as it uses a timer 2018-11-16 14:37:19 +01:00
Martin Stein
51cae1f0a3 Add base lib to tests 
For being able to build the tests via source archives, the 'base'
library must be denoted as dependency explicitely in their 'target.mk' file.
This is because when building archives, the API dependencies of used APIs are
not taken into account.
 2018-11-16 14:37:19 +01:00
Martin Stein
0b8ea50589 test/slab: move to os/ as it uses timer driver 2018-11-16 14:37:19 +01:00
Martin Stein
ebfd49661e test/init: remove unused _expected_log_message 2018-11-16 14:37:19 +01:00
Martin Stein
0eef5b506c test-util_mmio: rename test-mmio 2018-11-16 14:37:19 +01:00
Martin Stein
590dc1ac59 test/util_mmio: simplify success conditions 2018-11-16 14:37:19 +01:00
Martin Stein
f439cf0de1 test/rm_nested: destroy all objects before finish 
Explicit destruction of test objects (esp. RM connection) tests for
regressions in the service life-time implementation.

Issue #3012
 2018-11-16 14:37:19 +01:00
Martin Stein
ebdb3c4c32 ada lib: prepare for creation of packages 2018-11-16 14:37:19 +01:00
Martin Stein
5e8c53f61c base_types.xsd: allow session labels of length 0 2018-11-16 14:37:19 +01:00
Stefan Kalkowski
fe3f67f712 test: add simple cpu benchmark test (fix #3026) 2018-11-16 14:37:19 +01:00
Stefan Kalkowski
24e6a5ee73 test/memcpy: add Genode's memset to the suite 
Ref #3016
 2018-11-16 14:37:18 +01:00
Alexander Boettcher
f53a56982c clang: use template keyword on template methods 
avoids tons of warnings

Issue #3022
 2018-11-16 14:37:18 +01:00
Alexander Boettcher
886619f63e base: set parent info in child_process solely once 
clang:
 warning: Value stored to 'parent_info' is never read

Issue #3022
 2018-11-16 14:37:18 +01:00
Alexander Boettcher
5120e5138c nova: avoid null pointer warnings 
Issue #3022
 2018-11-16 14:37:18 +01:00
Alexander Boettcher
f7364d8463 base: avoid warnings in list.h 
clang:

error: member 'Element' found in multiple base classes of different types

Issue #3022
 2018-11-16 14:37:18 +01:00
Alexander Boettcher
fc0dbc3f70 base: avoid null pointer warnings 
warning: Called C++ object pointer is null

Issue #3022
 2018-11-16 14:37:18 +01:00
Alexander Boettcher
17f7147ac1 base: avoid warnings about shift operations 
clang:
 warning: The result of the '<<' expression is undefined
 2018-11-16 14:37:18 +01:00
Alexander Boettcher
4b62f091a9 base: fix accessibility of enum in bit_array.h 
avoids warning by clang:

error: 'BITS_PER_WORD' is a private member of 'Genode::Bit_array_base'

Issue #3022
 2018-11-16 14:37:18 +01:00
Alexander Boettcher
9ee3843f35 timer: fix conversion error in timeout handling 
clang complains: comparison is always false due to limited range of data type

Issue #3022
 2018-11-16 14:37:18 +01:00
Christian Prochaska
7c9e850235 qt5: add typeinfo and vtable symbols to symbol files 
Fixes #3025
 2018-11-16 14:37:18 +01:00
Christian Helmuth
0867da28a2 depot: update recipe hashes 2018-10-29 09:36:23 +01:00
Emery Hemingway
3e5ac64ee2 Libc: support monotonic time without RTC 
Use the Timer session duration for CLOCK_MONOTONIC and  CLOCK_UPTIME.
Use the Genode::Duration object for passing internal time, it supports
sub-millisecond time and helps disambiguate units of time.

Fix #3018
 2018-10-29 09:36:23 +01:00
Emery Hemingway
6178e378c1 Test libc sleep and gettime in run/libc 
Ref #3018
 2018-10-29 09:36:22 +01:00
Emery Hemingway
3e31e2ba53 VFS LwIP: blocking TCP write 
Block and loop until application writes are sent or buffered in their
entirety. Do not call "tcp_output" directly, LwIP calls this procedure
internally and calling it again appears to disrupt the LwIP TCP state
machine.

Fix #3017
 2018-10-29 09:36:22 +01:00
Emery Hemingway
3958ea50a0 Allocator_avl: apply_any method 
Provide an `apply_any` method for accessing any member of the allocator,
this is provided for destructing members of the allocator.

Ref #2996
 2018-10-29 09:36:22 +01:00
Christian Helmuth
9ca214eee8 usb: quickfix pointer-emulation for touch screen 
Move button-event reporting behind pointer-position update.

This is just a quickfix to restore correct pointer emulation with
dde_linux usb_drv. It does not change usb_hid_drv because the approach
is a questionable workaround of the current input event handling, which
reports events promptly on occurrence of inputs events and ignores
synchronization events completely. The original contrib code reported
the button press before absolute x and y positioning followed by a sync
event.

Issue #3019
 2018-10-29 09:36:22 +01:00
Josef Söntgen
9557e64822 gems: add interactive SSH Terminal component 
This component allows access to Terminal sessions via interactive SSH
sessions. Please read _repos/gems/src/server/ssh_terminal/README_ for
more detailed information.

Fixes #3014.
 2018-10-29 09:36:22 +01:00
Josef Söntgen
82fb76c142 libssh: add async event bind patch 
Issue #3014.
 2018-10-29 09:36:22 +01:00
Josef Söntgen
9ab288d8e3 libports: update libssh to 0.8.4 
Issue #3014.
 2018-10-29 09:36:22 +01:00
Alexander Boettcher
5446c52c43 base: make Entrypoint placeable on specific cpu 
Fixes #3008
 2018-10-29 09:36:22 +01:00
Pirmin Duss
b112b7b4ce mesa: fix port_prepare on ARCH Linux 
Issue #3003
 2018-10-29 09:36:22 +01:00
Norman Feske
f21493272d test for measuring memcpy throughput (fix #3016) 2018-10-29 09:36:22 +01:00
Johannes Kliemann
c2d85ff554 Ada: allow separate GCC for gnatmake 
Fixes #3011
 2018-10-29 09:36:22 +01:00
Alexander Boettcher
b8cc468f02 base: support assignment of invalid weak_ptr 
This fixes the region-map component implementation in core, which uses a
'Genode::Weak_ptr<Genode::Region_map_component> _faulting_region_map'
member. This member is assigned a valid weak_ptr or an invalid weak_ptr
according to the state machine.

Fixes #3012
 2018-10-29 09:36:22 +01:00
Alexander Boettcher
e6ddffb93c nova: exclude framebuffer range from buddy memory 
Fixes #3013
 2018-10-29 09:36:21 +01:00
Alexander Boettcher
ca727ea3d9 nova: update to latest r10 branch 
- which avoids some false postive overmap mappings
- and contains more accurate kernel memory accounting.

Fixes #3009
 2018-10-29 09:36:21 +01:00
Norman Feske
7d641d5f1f base: add Reconstructible::conditional method 
The new 'conditional' method simplifies the typical use case for
'Constructible' objects where the constructed/destructed state depends
on a configuration parameter. The method alleviates the need to
re-implement the logic again and again.

The patch also removes the 'Reconstructible' constructor arguments
because they are unused.

Fixes #3006
 2018-10-29 09:36:21 +01:00
Martin Stein
fe303f0e46 init/config.xsd: add <service> specification 
Fixes #3007
 2018-10-29 09:36:21 +01:00
Alexander Boettcher
5473a36f81 seoul: produce hash stable port 
Remove line printing the user name and the date in generated source code.

Fixes #3004
 2018-10-29 09:36:21 +01:00
Emery Hemingway
cf7e23f0d6 Tool/run: configure power_on/amt and log/amt separately 
Fix #3005
 2018-10-29 09:36:21 +01:00
Martin Stein
ae55954919 nic_router_flood: reworked to stress/analyze more 
Issue #2953
 2018-10-29 09:36:21 +01:00
Martin Stein
7b47c8f0c6 net_flood: fix CRC error, make more precise 
Issue #2953
 2018-10-29 09:36:21 +01:00
Martin Stein
304cb290d9 nic_router: rework quota accounting 
* Account all RAM/CAP quota of a session except quota for metadata used in
  core. The latter is considered when asking if a session can afford to make
  an operation but it does not get accounted to always be able to pay back all
  quota when a session closes. The general accounting mechanism is moved from
  atop of the allocators down to the level of RAM/RM session operations.
* report statistics about session objects and quota if <report stats="yes"
  quota="yes"/> is configured. (default is yes if <report> is present)

Issue #2953
 2018-10-29 09:36:21 +01:00
Martin Stein
bd77bb41df nic_router: improve handling of TCP termination 
Like suggested by RFC 2663, reprogram the dissolve timeout of a TCP link
state to 2 times the maximum segment lifetime (by default 1 minute) when
receiving a matching packet with the FIN flag set, or with the ACK flag
set to acknowledge a FIN of the remote side.

Mark a link state as closed (no further reprogramming of the dissolve
timeout) and set the dissolve timeout to 2 times the maximum segment
lifetime when receiving a packet with the RESET flag set.

Issue #2953
 2018-10-29 09:36:21 +01:00
Martin Stein
3db7181104 nic_router: limit packets handled per signal 
Make it configurable how many packets get handled at a max per signal to
prevent DoS attacks by clients.

Issue #2953
 2018-10-29 09:36:21 +01:00
Martin Stein
b48c917984 nic_router: allow ld_verbose attribute 
Issue #2953
 2018-10-29 09:36:21 +01:00
Martin Stein
d6c6549354 nic_router: destroy links on insufficient resource 
If the NIC router has insufficient CAP or RAM quota for the creation of
a state object for an interface, it tries to destroy a certain amount of
existing state objects of this interface to free resources. Afterwards,
it retries handling the current packet once. If it does fail again, the
router drops the packet.

Issue #2953
 2018-10-29 09:36:21 +01:00
Martin Stein
4442c79526 nic_router: "packet alloc" error only when verbose 
Issue #2953
 2018-10-29 09:36:21 +01:00
Martin Stein
364f69edad Xml_generator: fix and test missing '\0' 
Ensure that a '\0' always appears at the end of the ouput of the XML
generator.  Previously, exceptions during the Node(...) constructor
might have prevented this. This commit also extends the xml_generator
test to drive a harder test on exceptions in the Xml_generator.

Issue #2953
 2018-10-29 09:36:20 +01:00
Martin Stein
85d589a49c Xml_generator: fix exception handling in Node(...) 
When the functor provided to the Node constructor throws an exception,
do revert all changes in reverse order. Previously, the changes made
to the parent node were not considered by the exception handler which
caused unnecessary characters to remain in the out buffer for each
reverted node.

Issue #2953
 2018-10-29 09:36:20 +01:00
Christian Helmuth
e88081a454 depot: update recipe hashes 2018-10-01 11:25:03 +02:00
Christian Prochaska
31ca9d9ad7 libusb: fix page fault after failed USB transfer 
Fixes #3002
 2018-10-01 10:41:46 +02:00
Christian Prochaska
9c6120ccad usb_drv: call 'mod_timer()' with absolute timeout value 
Fixes #3001
 2018-10-01 10:41:46 +02:00
Sebastian Sumpf
0cc87d3c85 ldso: check for DYNAMIC segment in ELF files 
If the DYNAMIC segment cannot be located the ELF file may be statically
linked. In this case an error is raised.

Fixes #3000
 2018-10-01 10:41:46 +02:00
Alexander Boettcher
a8ed11e75b sel4/arm: make alignment faults visible 
Fixes #2993
 2018-10-01 10:41:46 +02:00
Edgard Schmidt
24a2b15eec tutorial: document required KERNEL argument 2018-10-01 10:41:46 +02:00
Edgard Schmidt
0a74f35062 tutorial: remove blocking run_genode_until line 2018-09-24 11:48:16 +02:00
Edgard Schmidt
a3fdefa6f9 tutorial: remove unused variable 2018-09-24 11:44:37 +02:00
Josef Söntgen
d56454f153 dde_bsd: update mirror URL 2018-09-24 11:18:23 +02:00
Emery Hemingway
c697fb6345 Simple bulk TCP test 
Send 8 MiB from one socket to another, anticipating short writes and
short reads from sockets.

Fix #2989
 2018-09-24 11:18:23 +02:00
Roman Iten
f65a7650c5 depot: allow to inject version command 
Issue #2991
 2018-09-24 11:18:23 +02:00
Josef Söntgen
018aebc0c8 wifi_drv: re-arm scan timer when enabled again 
Issue #2988.
 2018-09-21 15:51:58 +02:00
Josef Söntgen
f888c70982 wifi_drv: use dynamic allocations for AP list 
Issue #2988.
 2018-09-21 15:51:33 +02:00
Alexander Boettcher
9eabe316bf depot: support for gpg command override 
Fixes #2981
 2018-09-21 15:49:52 +02:00
7452 changed files with 331555 additions and 272531 deletions
Expand all files Collapse all files

6
LICENSE
View File

@@ -41,7 +41,7 @@
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
@@ -683,7 +683,7 @@ the "copyright" line and a pointer to where the full notice is found.
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
@@ -698,4 +698,4 @@ specific requirements.
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU AGPL, see
<http://www.gnu.org/licenses/>.
<https://www.gnu.org/licenses/>.

26
README
View File

@@ -65,34 +65,24 @@ The source tree is composed of the following subdirectories:
:'doc':
This directory contains general documentation. Please consider the following
document for a quick guide to get started with the framework:
! doc/getting_started.txt
If you are curious about the ready-to-use components that come with the
framework, please review the components overview:
! doc/components.txt
This directory contains general documentation along with a comprehensive
collection of release notes.
:'repos':
This directory contains the so-called source-code repositories of Genode.
Please refer to the README file in the 'repos' directory to learn more
about the roles of the individual repositories.
This directory contains the source code, organized in so-called source-code
repositories. Please refer to the README file in the 'repos' directory to
learn more about the roles of the individual repositories.
:'tool':
Source-code management tools and scripts. Please refer to the README file
contained in the directory.
:'depot' and 'public':
:'depot':
Local depot and public archive of Genode packages. Please refer to
! doc/depot.txt
for more details.
Directory used by Genode's package-management tools. It contains the public
keys and download locations of software providers.
Additional community-maintained components

2
VERSION
View File

@@ -1 +1 @@
18.08
21.02

54
depot/cnuke/pubkey
View File

@@ -12,19 +12,19 @@ sCaRvwPZdRCDDdhObkgkMlYROVNdC7ju8jZmB4n5O/5N7Ll7/RVhUWD7KeJu1UTM
oNEmhxEMrEcYcHFt8N8YVtJleRMVnsrZBNxOkFnpsPZr02XIQKfYi5tqSaBQZ47h
TTtXP3+FEaU+EoJprWqH55Sh97Fg6vuBJEmcGJeMGudFrypTzwqnM22DHwARAQAB
tCdKb3NlZiBTw7ZudGdlbiA8Y251a2VAZGVwb3QuZ2Vub2RlLm9yZz6JAlQEEwEI
AD4WIQSFq6G0496kOfRE2qnpSz5BlOa1+AUCWpAOsgIbAwUJA8JnAAULCQgHAgYV
CAkKCwIEFgIDAQIeAQIXgAAKCRDpSz5BlOa1+NGTEACg69FezAbc9Rzeb7j66NwW
3x/Zpi/jbmezMEtCZqAOcR8HJ7C0DN8gmCWPPB6oxAEeyL/i2cUb+9F0fTD6N7OL
TSOH75mNlyB9b8D2HdDILnLy4ClEitEOHFLMnlP58PGVtUNgbmsiM6cLLQtlJKvg
1xHlBTG9Ic8qgBcd808lLSC1XWN+nufVYBTE2RHNZqGeIWqPG5z1eW/JJO4M3V3j
MAw1p5Rpf9G6iWNxURutQOl/ii+97IKmHXX7N6ZRzawMGkOCAGNk1xeI71wCrALv
i3m7NmXb6XIJjD88hO5xfjjRO/0Gcw3vdmTXcVNzM1TvWj8ZsU6XjnMvNxa4LHqh
MxWuHlX0cAeCOzo1LJZ54f13hJ3YlTCrv68UIoNGRVi1LEbpE0sU0Ycdbw7ur2Gt
8Brx5WXv3hXTL8s3fmJzYU3/cFDfXjnaRiMWcNYdvckdHy6R7zPTk1NOvmfkolnO
W73ivdbonXpn0YqRGo2L76GdahLaVYn2gEeisqHnyeBCvI2snI+dMlEK0YNFrEkP
wgtWnF131Oorn0HMS6mpjcq4su4s5d84KCMGevFnW69oBRG27Bk/fJQiv2dSqkZg
+3TnyNApg1G7Bp5uD83mOvJWKdTWAntCJSQGbg1dmEsOBG+Bu7EE5LVBoMH3h/LQ
GB7c5NpgNIFQ2DZy2vkOI7kCDQRakA6yARAAws48PKN3BQPEM0M4kTO57/OqwGrA
AD4CGwMFCwkIBwIGFQgJCgsCBBYCAwECHgECF4AWIQSFq6G0496kOfRE2qnpSz5B
lOa1+AUCXk6dFAUJB4D1YgAKCRDpSz5BlOa1+JHXEACyB6vfzfuyo8fu1IaGrloC
qUUuSfhdleF1tPhZQn9W92bEEoXPfqFUuiNaxeFIXonyiD7wJYdcowdIb6CiNWrx
B8yjDQ7yb08sDLlL821I93kByy24Er5hY3QEMuEL3qv8YD/6Bh36Q0aoTXqQXMuQ
3AcHp0JI932Oez3q+gEdW7CUe40N9LaTx0A7ZTHe0i8IELkUPFtDg8xibOjZhBMR
vlxNUtzQVvC5kT5Wt0w6ZFKPGJguFylaE+TcsTNtiLNS4Z8PmRPgygrWotmprELo
2/py/VMxgQb7p4LFj+sNmU9x3ulFO3Efb9cVZVjBMYx5xh83ZiLZ3PvK43Eo+7nm
kYQhPinAjNmqng4/ZdQFrayFMkZc4oKCqV6BaYnybbsSn2ZEwVQqpDMeswWXTf9k
qC6DONz3ETtyzBEF8cHDpy+BwHRXx+pysGLhIvHS9C3M91j8oXvogZRbw/dEKGX2
eWes7nsdHeTbyRJunCCnsXDm0Js8cZcqj3Eu3Kpeu2CbgJGD+ajOel2RSMo2mFtg
OoRWjCibWHDSeUS+LGtspDpry+4VYwm4KBdcyEgiXnE8WDb/xk/hZgJCr2814AJC
RPpXoIZdFiwSP46vgoAnHynPbGTmE/Wt+F9Y2cn/dz6rmg5ynS8AKuTOUSoAKEUV
J20mBJpkSx7Cc95AGE8K/LkCDQRakA6yARAAws48PKN3BQPEM0M4kTO57/OqwGrA
AMlxnB3n+AP7rtZEC2L548+3pfg7Ub45Oq1X0ySxgPhV15CDUEK69pgs0JhphMbM
DyaUp04DyQWFhdueAqQVFnKvg6nZm/WIJF1VYEc0/q6Ramsdv2cWdxFBkFHYsH3P
G2bXCZZAEO6eJI8cdmBH6+bVptky/AybzBLztp1ruWlj+rm79qULmW7zJfxYwL1U
@@ -35,18 +35,18 @@ kAVN0Y0H9EGSEoVyKwS1DErzcBgajPKWedKlmjU1uIzupRWn5oqetVbNfZ3Z/EJ/
HdMCX0PN1kxg4WoI9mkP/moly1yopVOTTLJwvC2C85NQUmxyZeb4h9O7toFczBxn
7pV9Eprm/xRcO3ZEEpmdM7gR3+R3PpxkjgPIZAn9il32VYzdT3eQmDZ8sKC0qASG
ik2if34YnAggGmsrVB0nzT9fZR7CQq2IR7eohczLJRl+n7USQMgRweF+sl9PcbDT
aC0b0i+VveSEBOEAEQEAAYkCPAQYAQgAJhYhBIWrobTj3qQ59ETaqelLPkGU5rX4
BQJakA6yAhsMBQkDwmcAAAoJEOlLPkGU5rX4QjEQAIJF78XjvGomhafMsWlcd7fP
/j45B+KIpyruc7wHtLqWibi+YoDuvtG4m1/4Ckcz5qNrV0f5nQojEAeCSCh7Sl+s
yAJ9tmP1XET29DJq1t0iMsu+RDCLhdOfL/Wi/YJARtDloYbDlD8Rc/JnL2aOx2W/
Ybajj2lloxSDxKCnzCh1aZixie1YaQSm8ErshT2k4qTx48D3mRoBLAYyzdEbLkl3
ZBGDQy3Xk/miJ/hsj6L3w3G2YjMywZZZEjgDSUgSJ6MazBTgMmbCy1/0YGh7rNF7
iUoWyDZq4qiDGNAI695I6tas7s4X2dhZn10xgbJoa5Dq2rqQLek+ErEsr3DFYN53
7e9H0vJ9ydXB7piRJ9bxVFBng5CX9677IN5k9T/0lvcZWAFnDcHiMkIjIrTQ1Y04
WkG6DMVvzLD+PKA07cZLf692rsrbfpP4sGj8X6baWh58mGAMMwZ2Dy07grsi73qI
p/H4r4c/xRsH6nTbvb40zLfqcCz1xOTEcDtflYGMoeshu/H2EoruNMFi3c/9WO2G
3zrSOUenWFSiacbsh8HVE9HUFGddBTB3SrLddO5vvvi15OvGI4aWpnkW+9s4107h
jE0d0+c/Xm4AS0T59OGj/cd45k/vRe5vaPhQobZ6hXBzkKWPVt5uH8xbROiGiqdR
jMuaeWIldeJ60aeu39Dh
=JE0L
aC0b0i+VveSEBOEAEQEAAYkCPAQYAQgAJgIbDBYhBIWrobTj3qQ59ETaqelLPkGU
5rX4BQJeTp05BQkHgPWHAAoJEOlLPkGU5rX4RRAQAKSsr//mEXy6iQyoqaxJ796s
ljaoqsmT4ly/+0k9diX1+/wkulSkj4/fL2fDvRQK2cU1i6mYJVugN5swqXvp8WF8
scnDS/H75v/3Z+clTlsMjZ4qI0PmXYDpcvp5+zU85USEjOxpHmIKvQGflZ0NMvkI
uo2ZC6F4xnB5iizx9Ebm0gGAiKKEJ8ID/LLhJmOtidnGc3322SUVXIl4+ue9KgpO
hTBjx5UdIQ+21uSRf+zv3ZQH/KA/08iNyD20fIiUZKfEoeovlW8SipoO/pf/+1DE
WSiJqL3K7T3ixkC4oWSp7tbo2MWU247PItxTnnGF+KjwbkzpPwyPYPHT5Ed3XEZV
e39r9xUzdvYKKDCNDToDFsh4bm4hq7xXgnTwAbF6hw0iyBK02ZB6FKJsF/ZJi8gC
Ni4d3Lb54CV/Z7nAqMFbTa6KT8o9fc0Bw6tgX2ch7SRWU70yVGYmQzrdRsOtDkIV
iYA05JozwwsIcJQS7hBVnsqgD/huYLd+y7iCUgw35NdQ5764jzVlUpYtFO0NFrM6
QSKnnFFRiwr3+ZqIja2SGW455uP1hsl2ccA4Rdp/JRuEI6ojIo0R+JBpcMeqdv9d
HcNsAoZkrm/cQfIwlyKsU4/dEFL09kEtv7x/aO2ImqjAZ5xT3ml+4kFGEn7M7rsc
M6R6fgkYrVo1GhjXTlCL
=4bj4
-----END PGP PUBLIC KEY BLOCK-----

2
depot/ehmry/download
View File

@@ -1 +1 @@
https://depot.fuzzlabs.org
https://depot.1337.cx

1
depot/mstein/download Normal file
View File

@@ -0,0 +1 @@
https://depot.genode.org

51
depot/mstein/pubkey Normal file
View File

@@ -0,0 +1,51 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBFuhfGYBEADOj0EW/U2O3koeGIuNGDuBDSzkYjncoBrrPN1z69hgK349xiP7
ZwrPBA0kJ3ctLcRQcOEDF5XJzjByuNk3JV6MuKFqwy2i9dDTHkv5pv7Na/R3L2CM
k79weVahjPllt9nnbSMMtYH0eL187EhLW9owd0CY9wDj1lhLISHpyKvb9vwiUAwQ
i+xypGgU+Qvf8BmfTDdjfP0n+Iw31z+R3xiHhWmhrs9gID2XVtGZIy9NQQrijJYS
1Hp8hjgH9Fzmj0GkxUI+2mEqLeEp6wBmmo+7PRzc+iRYJNocyRxDbmdE0eyh3P93
QABzZiWTU+Feq0MYa1FK55uFCFHe6MSFLq33CGYMzQ9oFQHjuEX1KqJT/OoumPgN
iEb8Bt3XmqiI8GejafLKIW8gO8tYLABdrEpFelnyXy12IIOpOwb8GrxFUEu8+bDb
Fc0YkV7cZ4NH2Kq/6/Or30gCA2XUpqhDNSxA6jer35a3ZzvgXvcRkNR/oOt+qt/3
u9ew+e6Xjque0PRWzdavVlKdYWzSV8umVw5DGgQj/rSn+8spog/5w1lElyUa/5Yx
Z+1bvOIyXPg+tTLimWtGRZQorak1sMsLMDqQck9SwOQWpSmVGsJ14uWXSMaZjdj9
4rHSSymYlcQWP8t9lfPf6/ZSH8574PbbJSe71UXcWWznSGprSt2QQAQ7NwARAQAB
tCtNYXJ0aW4gU3RlaW4gPG1hcnRpbi5zdGVpbkBnZW5vZGUtbGFicy5jb20+iQI+
BBMBAgAoBQJboXxmAhsDBQkDwmcABgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAK
CRDJZVRAR8NPkiTSD/9xiUVNgKTkmH9WRdCcaHuyhKoQI6CsWnMOW5OZ+gVXPc9U
LapQ5Wymb+yGE8Uyn4EJnwdEa8TdWztMs3TNtK5vh1Pk1lKmtKcblBiDrUPEURhR
mUBoUkMe2VDXyN27e0lkbxcFJ0HrYnhvb72Yfnw6B3XMa/NIhbOU3ttB/tVR6Kjl
YVDtTokfILQc07Ai+TqkqNARtPGntHAn9qFvPLNKI31c+Xw/QGZbVPNFW1QAYBiO
5dqsSwKaouapHAb0GBmU0fAfJZNcvwt9Ph7WDXU9Ce3LZmIshGW7Gzm7EGdf4VS6
k4OtEWZhZHRnl9/sBJ2UlQU0tFs3ow53aDZQIa4FKd4JKabBeRgDUKCGEyfoNKCq
NKkLZaR/kFA9EqTPZSIXPLYIzIlSFoL3LVDOc5O7hDT9vXaubdZeAfE9VcHDdm3u
usW034F0zbsuu4G83qZ6QhDOp0UhujfSU+BDkH0I+OfFyVkQAmy69J3byi7Iluno
OcZM13rbe30jzyyyWAnjHicAt8tNwCLMmnIRvYT9ELqywID0Zzxkct61eJQ6L6aq
/DnUD9Kq0OwdxC1wMGtimX5eUnN7S7y5+TMRepJnWcTtgrLBVqwRrvSLnsKVd4Gb
Bu3dSs+BxMV1UcyU0KWwDPY2D7k8SGgeH7IEPo/JiGO8RrWYJRac9qel9egMYbkC
DQRboXxmARAAoaRH/xgKvp0QrNFJZBW12SdXT4h1nXglbpZd+bUl738LqM2JSIC5
Qq7t2DUtOiL9UxqvRgAIjkVLR+hHJBZyuPOP5Ebaf3Mr7eAdmCx2TMsHZPKBLnTy
EqeUcnPwThNKcUaGsQYlkxIQAJtPzy3YBuDCYnzzSi6KzP3VC8szHsXcPo2DeKWI
m9BbMMSQkB3m0fz8W3ar/64RDoubRWRJ07b7q4WUb4jPvql+Tz+Gv5x3qm8g6m1j
uD+U5OhGCe3m5/MsI2JYdWXEwif8YD59gylsj3LJIu3i4SkR8aS+x6n3LIvdSCOt
P3To87wfoSqBWWObVe/vOHRXqpzEF6T/ih/rkhUFzrHNMziTPsujBazm82AUZ6xi
2TddbdSAK+ZpSPQjyyRflNyNgv0iu6P3BnBgv0Xcuf2RsDMBDjdvHtX0lXkPMB+r
6jgxl0faV6UktH5R2d4Zt9i0/vWlZZd0MDylPNQDiKs+qSIX3ntfUUaPdEJqYAwM
gloXr3cZtsVh+Yoqt8pa4x0x6kqez+oyBUFf6y1FCKYq2H/QfPRkF/DrfZS+7jXW
iAX8PcrDceCN6Dr10/g5LPWXQOHUWxCu7FEYoYJsVXst4N0FnCOCe/OGYIhMteSV
IJJvNS4qwXZKKZ+cmuT9skEQ6zsLV9+VDYMwVqXB4wTp4EwT2pCp6LEAEQEAAYkC
JQQYAQIADwUCW6F8ZgIbDAUJA8JnAAAKCRDJZVRAR8NPktRiD/9qjH6uuT1y5uNA
4E8u6/vxZGsyBG57dtnQ0KQbcsZSrUxMiDTtMMrklsl2vtut/XDroajm+d2eaQ4K
j0h285VRkT5xJ79JaIQ8oyzRWBtpENy0iLUm7CmqCMuH8VFifZyEBYZgR/K+9ago
WwF3sk0QP0WJTFqIZajKK0Lo5Juk8DCXjvtkvGNytHkZ2vEkiZ9m7Eq0AjQnts5T
PdHkob7Z8fu0KfUMoCUtb+TzNyMYOVGE6uTfo8wKvbhwzUWSymZkrecx73XUUmew
MdZ2WzwGdUMXwcBZ4LvgrZVg+LKxR1lyJ+qbIUpDKXCZGGH2VZVaQspgnx2Yt0EI
Zl2ELMrTq5IcOkxtblEZpQkwaXszqUDNO43ezp/PmbTRWChoM/k3RAbp9rvoYx93
UChyRXpJRaqDxK502k3BSnOfmolpQvxTMKtLZO50URa0F0F4qD2csns4QrqOWsGr
qPqVKcLn8Q/yMW9fm5+xX+ZHICKk64GAB0YyHcw9lOWekLhGzZV1KnD5ID6+5E0B
nsWZB6PDqLM+/ykXh0rcN8DWwZvhimI5/dOEIWIkADKT8WGwh+2JnRyimhxYdaiH
ul1BvZAYqF1FVXAHd0u48MssLywGGbW4fH8D9SM+LFpP4+8oYFLNgH1OreMENuag
i47EaxpnQUgIt8PZ/gL3LeiGs8SNpw==
=flyM
-----END PGP PUBLIC KEY BLOCK-----

34
depot/skalk/pubkey
View File

@@ -6,26 +6,26 @@ spm0DdNCEVfUbIQUstbSNt1kDGIlqPvZmoRj5YwDECW6ULG8jTWhaAz9SybJCE86
jLPzZHjKtVhCVO1X0ogSxvvYGemUpqVCLfKcIb1TucieKKCrnjiHwp0XZ41DWqSb
kBwWW2YQMEsw8JNGNTfCCUFf6+1l4mMixtv2sEqFiH1wQJ0c4gSa4iSYZ+eNmzJ0
B13K8kONOctSg3NYZz/dC1aUzLOTkgJHqEPHABEBAAG0KVN0ZWZhbiBLYWxrb3dz
a2kgPHNrYWxrQGRlcG90Lmdlbm9kZS5vcmc+iQFUBBMBCAA+FiEEI2X1/oP+p+Ls
2L28qrC4CLnp/P8FAlrQZGACGwMFCQPCZwAFCwkIBwIGFQgJCgsCBBYCAwECHgEC
F4AACgkQqrC4CLnp/P9kKQgAxc11eDhYkMVg9cuipFoqtV5lY9aT2qkocZ28IhbF
LXhy9lcn5FlxZSVdkzJQ5tUf4nSuhhMb6z3r8edaOebcYAyFk0DTymNpcEyT2XHd
lsOcInhfU423m1tCNHdmxtv9HERtj2zS1KNkjWxY6xsqfEw1eDfUvdS3K9KpUqc6
vWZwsPd9EHxW3mzWJS3lrSAnNsCwtdmiqB9045Yss4KednMcN6qxE+uHppQ+25Ib
5ZICpiVqOJ+eQXeY3kRx84lfZJr3uFdn00RSU5fn0uol8ZZYX9tQd9SC1GIlkyYj
HxNLKNVaYzF1nnmR1s7cpY6PUAYbt0im6kd4VJ1wQcWTnrkBDQRaU+QuAQgAuame
a2kgPHNrYWxrQGRlcG90Lmdlbm9kZS5vcmc+iQFUBBMBCAA+AhsDBQsJCAcCBhUI
CQoLAgQWAgMBAh4BAheAFiEEI2X1/oP+p+Ls2L28qrC4CLnp/P8FAl4XHPUFCQ0p
OkcACgkQqrC4CLnp/P8bAgf/ToUKzXMGdRXpB+xIBbOxuu8xVdLRj+5cm7cjZRiy
N0ulrzAlaZqrfqb5rTujPDjakmVSBVRIi8zWlzU7y67fevTFdi/Q9xeLgbqP3cWU
UgUJs03cdacFIamSIO//I9gaImZ2kpUcA6Nj+N+jPepRgUBzljsJG8678ndicA0e
Yio4uAvZY9JmsWGn48rmkqshPf9lag4GUnXn+R/AFKCts55kOiPekux3pJBhmrLJ
TdLgdAzlzGQfHxQbz3oVK4yE19ReAd6rnpW06oK4KfiWrgCmaEk1quirDNfIMDfr
nwe6571TY5is0f9zJjJgP8ogLUMI3n54OvwMezUMz+bLbbkBDQRaU+QuAQgAuame
2VDCgNmiwu+QmWyNN4jzbE7+VNmDr37HO9lZRIROC4eACPOGfUL03jLGvUn7rrxQ
JK+Pit6gXXCoIWDhCMNRSZKho416KJZWxF2jxKBKGQ7DtWaTR3YOzSf0ka9DZSrp
wG22xS0Uf1U/0ZBIf24LbyUDFLc8zt4eey2D9AHm+9vCf7wnf8TV6SNIXRz3wj7d
VqoZLXRTT7twBSaNahLhNtg7fhS8Nu6/THuwXpNKPAvAsgJRTGk7kmrKj5P7rOZA
rNHC1pvK1EWJJi2onHOmCzBccKRp9SeQlj+ddqG6seZhEnRYG6l7uhkyNZoyvKxO
Fguc5g8Xf37VyuRMfwARAQABiQE8BBgBCAAmFiEEI2X1/oP+p+Ls2L28qrC4CLnp
/P8FAlpT5C4CGwwFCQPCZwAACgkQqrC4CLnp/P/IOAf+LnQVtU7aHh4AZDsi1wXq
KBo5l6r3G8tC/S0HEf8nnWMUio2/mwVrkbuTvBeKrcQ/mXFHHAG8YCAIHPgR7T0y
2L6l2PL4HoXiLD8EwJ0sWZu2waxuxcTX+bb1i3Xm3squnqDtCX3pyoXWx0GVgrz9
7I/zitxeER+35ScaZ+JAAcNW59LpiV1SdIXqbtrw5QBJBuZUp0bvnzCNvdZLhnhb
gWfwPEfcXFt5K87iTmfMFJOJpbUrEz/NWE9gOBCBjqxW0wVb+IWr0oFWvfxjuBq7
IW9DezwkN1wAavP6g7+B4esCD6SRq+3CCzbT1By3X2h3SevU8tHkCSA3cIKgWdyD
2A==
=IsaH
Fguc5g8Xf37VyuRMfwARAQABiQE8BBgBCAAmAhsMFiEEI2X1/oP+p+Ls2L28qrC4
CLnp/P8FAl4XHW4FCQ0pOsAACgkQqrC4CLnp/P/xGggAhMz6NH1dYiFtc9+3vkAW
bZ10MDvxdgyU+G1a0BVYVo2ZXkhJlb2yg5MEeL1MZ6cUtM7MR1OL0U1jayfdcTfr
kkUi2a35BdvKRpxc3hcc4i2CdDxdx+IGrBa75P+vIKFj8WIvwABgRJNVzY1Oq6Lq
zKPUpVPug80Hch++Kv6l1DEpcoXlGaI8Z1dWfAjYQ7HJF/YJNHloKwxa1mXo/l/S
qQSFHB23IN682ANm8iEUphuvJVbOw4mubpCEpmJvIhEBeesgXfB503nniOsshq+9
1bNZ3CFv/ylnj5sy5VzU0lfW/TrHiW4OOOSbX8mCSvcqN2mhjcu6r4olV0fWPrrV
Lg==
=H+8g
-----END PGP PUBLIC KEY BLOCK-----

1
depot/ssumpf/download Normal file
View File

@@ -0,0 +1 @@
https://depot.genode.org

52
depot/ssumpf/pubkey Normal file
View File

@@ -0,0 +1,52 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBFxvvIkBEACdb88mQFNBM7IG6i52BgpBqfkn4HNrlb5xOBgV90btReZWUfgs
2Fas7kg69yyYuuPkdsWHG5d06tj21WVuuB9r2I0aFvNOIFOc0uSDUCl+YZGiwYjB
UvRg+oKFmWO2f62ttpsrAKjCKAIuQBJBnPsDXtvcRzquV8C/bJTMOTLQ1NfgS9Ay
wSOdvooG8/gfuC0+ZgbdPgpDcDMQcS2o6Q0aAa3dbTksWKtrnqzE/m7wA4CgZWTd
6LzD8hGayVjLcTUzsqBRqixPSACi4CbOnohWIuXPU6FZWD2YgOKfRf3w3OvX+o/k
9SuOv8fR72kQoRrDhz6D3Fq5koST1977oEYc6Ns/yCdzV5xzCYBQ6D30LrkyGj6y
4anEkaljeCqGXf1S/Dg+1lQ7lzbII/BHJIAn18303XBeg29mLSXPLpNtrpb7UFMX
cjNlN4nG2N7Oa2N73lD+FK56mVwQfb5Xoj5kZCPMLvxicfRHmf1UAVE47Y0p0eE3
B2SqqGflcSCZbpGK/o21jn16Vme2vJqIXbs+mGdYTBHn/9O312XWs8apNqWW92u3
5pVcxc3JK1fLpLUZo/ohUneM2V8rXPdVCg7h5c84dHUrPXfOQH8e9s24Inq5l9vx
zYcf1QUzeFQspO3ECKvuNmKyQTAWmrcD5vSx7Epa3wynsXOx6bttR2/OyQARAQAB
tDREZXBvdCBTdW1wZiAoZGVwb3QgcGFja2FnZXMpIDxkZXBvdEBnZW5vZGUtbGFi
cy5jb20+iQJOBBMBCAA4FiEE0Tow76OCSsNBYjoWmSWROw/a3bUFAlxvvIkCGwMF
CwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQmSWROw/a3bWmcxAAiKJRk+Sp4wmP
PVCztw0h4bwu6/AHWM34qgWLFIf7pX+uUp7NZ8vnMEft5+XXLJdCnIdCeMCgSfo/
5FeD2uNHrLLI33PMCixAidg0sZ0ENpopY48SHdlADCMUH0c7/kFWdX63UznyhAIe
KsUyglY9j6oCNSEEna3lUsFGQ2Z7Cl4b3OWSkBAMyBe7NUs3pnyy2XdCP5uUREuB
PakqS2HuQ6zvdFLU0h2PBq4KT7nzBHPUMnkrw0ulM+4JX29GFk1WzNpJYCZW50vx
kD8bEfzwqyZPpo12M4Ica9su7mMFpVSMeTVhI5vdQ6AxaCRRNojgfVNn1TjCRWUD
91PxKg+CbHiAcEdFRNOH+nVZpqQ/sspSKigS5QrGmRvr2dwP8KFOyjN1hNLp3x1R
SkGqPtThlpqAJo1YEsuy0gfetdSaLPXtkEFuAIynt861C/bBYfB9vaD8dSEbolQj
CI2k6sWv8p9Ym+/2vNvYOgKe6PlVtoXOydRR1i4uWuVsJIFlIc2NGyiUXIZ0AvFp
X2KAMjxGAvcfLN6MTzCwSSN5Zk0oOV3NRqbdqOWRDVhCZIpGpjov/E9LKjh4X4xf
ybdrU4vRgiTRk4WriY0tiWOqgJrOo+wc+1VgJl6YaP4fY7eocGS7NvZVnWhq2JNY
s8oiNYbdd72NVQYdTAJzDhpnjJ4USSC5Ag0EXG+8iQEQAMVEBe9AOUrsbqn6dvhS
vgs/TsTx1R7hWA/RgP95nDZDnzQg+QVCs2rqp+pVgqNgaz9eJhlT7NCCLE95LCKY
F0lWkSwRRWxMTQzdou0/o132UBSUKD2IAkY1wsH86foUZY7UDENm7Ht23Uvy2dCY
fwNVaNOvqs7FGDWIPmb6IehJqrJgFUEfgJYUAqTr4R/q3MIYceC5Llnm6NqptlSy
BN7Ixyvvij56MsWM9dasm/lHTYJuVWmRjo0gvlvL3eTD/NMSkRqzGw1O6RRbmQxX
yfo19ERZ/ALY6q6/cmOSQh2Fi8vd+NgJ308PthwPAnAsnMz82sKnJAoZ6PNmk5GT
dbQe+lQCwBreozGfLhP8hYlR27N6oydc0t2nf8h/GGQONcX6ySAiRaGt3Ryc59d0
H+S1JTJoUxaFBBsLFJWe1Ci93ueg0370nHtZbjtP/TG/4pS+A/hDiRMCBIMOP1eh
tQRjO6bMgPgxX0i1Lv6aHDf3cJcg0dqjAxfLaTVXYWtnDcqtwARnbcJYqaj6Umgm
l1kOFN4HsqTSKViXeTrP0DGZ+NQn9fo0E0qyyY6/YwKw0/FAa3xQu1oYH+w2inzJ
RbCHfSf4z49O8M3T7e6dskBj/YoKlRhYmazQrhZa3ixt2zwQuMH1L0DAnPY02U1y
BeSEAlLpFI8vCSk5S0LEVOntABEBAAGJAjYEGAEIACAWIQTROjDvo4JKw0FiOhaZ
JZE7D9rdtQUCXG+8iQIbDAAKCRCZJZE7D9rdtcY+D/9xykYgmOd09n6E2BjrV7H+
CtTq4iF/Y+HhbMtmlon1F2nNmWP0i2l0HonTaQRfCXcpXJYjQQX9SDD3Bb8Qkdxr
Bl3pMiixsbWyupQ8UxgEdK3mAfgnOxEilwlHB0vAhB2OX6zZVVH6g37+gJMiwOZf
zISDQH1FI6feheYAqU8AVM3RE33K16fhslQEg5Gi87f8jXyVT01qwPNysOfRBB2u
j8uJoeSKXwbHaTzzzlK1XzpPz3jCvy3QNZPMVi53KvEIzCKtBmrlgYbKKRLH2rMr
C9DxbXihvnNyIyQT6VQCbC9m7qNwrcriE/TahCPy1GP9ZUT4RiF3oy7lDDlk+9Jo
qh+LxbOw/UlEwgu7gtfcjfLK4yyLN1EjkCtGZIk5RHrA0fHYaRmjP82/h9QHIp8J
bBO08eDanPC64PwLMLymW7GoU+afetjn/LzP02TjxuO55ZHn8UaG2Cxx0nlipm0n
FXSect9nr/MiLRb+C3o0M4dZ/51IPl3BuhhBirKPe3li3ieaVQkZYnY4KhML2aj0
A9H479fStEg1AlsISFH2wol7Gw3WP8zk58NqRF7l1h5XtMQ8sOsVVzqyOHwIr482
UaY2AgdLgCCiVrHk4KzXxi9jA4vY5xWyU5SU2kEYJha5qAi8AzBWBvLrD/3YEdy/
4oe2jMt7+puZ672u9RNjYg==
=tNAP
-----END PGP PUBLIC KEY BLOCK-----

253
doc/challenges.txt
View File

@@ -16,17 +16,6 @@ research projects on Genode.
Applications and library infrastructure
#######################################
:GNU Privacy Guard:
The [https://gnupg.org/ - GNU Privacy Guard] (GNUPG) is the most widely
used Free-Software implementation of the OpenGPG standard. It comprises a
rich set of tools for encryption and key management. For many forthcoming
application scenarios of Genode such as package management and email
communication, GNUPG is crucial. Hence, it should be ported to Genode. Such
a port may leverage Genode's fine-grained component architecture to strongly
separate network-exposed functionality, the storage of key material, and the
cryptographic functions.
:VNC server implementing Genode's framebuffer session interface:
With 'Input' and 'Framebuffer', Genode provides two low-level interfaces
@@ -50,24 +39,6 @@ Applications and library infrastructure
integrated in the operating system, i.e., in the form of Genode components
or a set of Genode VFS plugins.
:Tiled window manager:
At Genode Labs, we pursue the goal to shape Genode into an general-purpose
operating system suitable for productive work. The feature set needed to
achieve this goal largely depends on the tools and applications daily used by
the Genode engineers. As one particularly important tool for being highly
productive, we identified a tiled user interface. Currently, all developers
at Genode Labs embrace either the Ion3 window manager or the tiled Terminator
terminal emulator. Hence, we desire to have a similar mode of user
interaction on Genode as well. The goal of this challenge is to identify the
most important usage patters and the implementation of a tiled GUI that
multiplexes the framebuffer into a set of tiled and tabbed virtual
framebuffers.
Related to this work, the low-level 'Framebuffer' and 'Input' interfaces
should be subject to a revision, for example for enabling the flexible change
of framebuffer sizes as needed by a tiled user interface.
:Interactive sound switchbox based on Genode's Audio_out session interface:
Since version 10.05, Genode features a highly flexible configuration concept
@@ -116,16 +87,33 @@ Applications and library infrastructure
of communicating threads as captured on the running system. The tool should
work on a selected kernel that provides a facility for tracing IPC messages.
The underlying light-weight tracing infrastructure is
[https://genode.org/documentation/release-notes/19.08#Tracinghttps://genode.org/documentation/release-notes/19.08#Tracing - already in place].
The Qt-based tracing tools would complement this infrastructure with
an interactive front end.
:Ports of popular software:
Genode features a ports mechanism to cleanly integrate 3rd-party software.
Thanks to the C runtime, the flexible per-component VFS, the standard
C++ library, and the Noux runtime (for UNIX software), porting software
to Genode is relatively straight forward. The
[http://genode.org/documentation/developer-resources/porting - porting guide]
[https://genode.org/documentation/developer-resources/porting - porting guide]
explains the typical steps. A wish list of software that we'd like to
have available on Genode is available at
[http://usr.sysret.de/jws/genode/porting_wishlist.html].
[https://usr.sysret.de/jws/genode/porting_wishlist.html].
:Native Open-Street-Maps (OSM) client:
When using Sculpt OS, we regularly need to spawn a fully fledged web
browser in a virtual machine for using OSM or Google maps. The goal
of this project would be a native component that makes maps functionality
directly available on Genode, alleviating the urge to reach for a SaaS
product. The work would include a review of existing OSM clients regarding
their feature sets and the feasibility of porting them to Genode.
Depending on the outcome of this review, an existing application could
be ported or a new component could be developed, e.g., leveraging Genode's
Qt support.
Application frameworks and runtime environments
@@ -133,20 +121,20 @@ Application frameworks and runtime environments
:OpenJDK:
[http://openjdk.java.net/ - OpenJDK] is the reference implementation of the
[https://openjdk.java.net/ - OpenJDK] is the reference implementation of the
Java programming language and hosts an enormous ecosystem of application
software. The goal of this line of work is the ability to run this
software directly on Genode. The centerpiece of OpenJDK is Hotspot - the
Java virtual machine implementation, which must be ported to Genode.
The initial port should suffice to execute simple example programs that
operate on textual input. Since Genode has the FreeBSD libc readily
available, OpenJDK's existing POSIX backends can be reused. The next step
is the creation of Genode-specific native classes that bridge the gap
between the Java world and Genode, in particular the glue code to
run graphical applications as clients of Genode's GUI server. Since
OpenJDK has been ported to numerous platforms (such as Haiku), there
exists a comforting number of implementations that can be taken as
reference.
software.
Since
[https://genode.org/documentation/release-notes/19.02#Showcase_of_a_Java-based_network_appliance - version 19.02],
Genode features a port of OpenJDK that allows the use of Java for networking
applications.
The next step would be the creation of Genode-specific native classes that
bridge the gap between the Java world and Genode, in particular the glue
code to run graphical applications as clients of Genode's GUI server. Since
OpenJDK has been ported to numerous platforms (such as Haiku), there exists
a comforting number of implementations that can be taken as reference.
:Android's ART VM natively on Genode:
@@ -155,22 +143,6 @@ Application frameworks and runtime environments
removed from the trusted computing base of Android, facilitating the use of
this mobile OS in high-assurance settings.
:Rust bindings for the Genode API:
Rust is a low-level systems programming language that ensures memory
safety without employing a garbage collector. It thereby challenges C++
as the go-to programming language for high-performance and low-level code.
Since
[http://genode.org/documentation/release-notes/16.05#New_support_for_the_Rust_programming_language - version 16.05],
Genode supports the use of the Rust programming language within
components. However, to unleash the potential of this combination,
Genode's API must become available to native Rust code. The intermediate goal
of this project is the implementation of an example server, e.g., a
component that provides a terminal-session interface. Thereby, we
will encounter the problems of bootstrapping and configuration of the
component, the provisioning of signal handlers and session objects, and
memory management.
:Go language runtime:
Go is a popular language in particular for web applications. In the past,
@@ -218,10 +190,37 @@ Application frameworks and runtime environments
analyzing such components with formal methods.
The use of Haskell for systems development was pioneered by the
[http://programatica.cs.pdx.edu/House/ - House Project]. A more recent
development is [http://halvm.org - HalVM] - a light-weight OS runtime for
[https://programatica.cs.pdx.edu/House/ - House Project]. A more recent
development is [https://halvm.org - HalVM] - a light-weight OS runtime for
Xen that is based on Haskell.
:Xlib compatibility:
Developments like Wayland notwithstanding, most application software on
GNU/Linux systems is built on top of the Xlib programming interface.
However, only a few parts of this wide interface are actually used today.
I.e., modern applications generally deal with pixel buffers instead of
relying on graphical drawing primitives of the X protocol. Hence, it seems
feasible to reimplement the most important parts of the Xlib interface to
target Genode's native GUI interfaces (nitpicker) directly. This would
allow us to port popular application software to Sculpt OS without
changing the application code.
:Bump-in-the-wire components for visualizing session interfaces:
Genode's session interfaces bear the potential for monitoring and
visualizing their use by plugging a graphical application
in-between any two components. For example, by intercepting block
requests issued by a block-session client to a block-device driver,
such a bump-in-the-wire component could visualize
the access patterns of a block device. Similar ideas could be pursued for
other session interfaces, like the audio-out (sound visualization) or NIC
session (live visualization of network communication).
The visualization of system behavior would offer valuable insights,
e.g., new opportunities for optimization. But more importantly, they
would be extremely fun to play with.
Virtualization
##############
@@ -237,21 +236,6 @@ Virtualization
is normally not possible. Also, complex Genode scenarios (like Turmvilla)
could be prototyped on GNU/Linux.
:VirtualBox on top of seL4:
The [https://sel4.systems - seL4 microkernel] is a modern microkernel that
undergoes formal verification to prove the absence of bugs. Since version
4.0, the kernel supports virtualization support on x86-based hardware.
Genode has experimental support for seL4 that allows almost all Genode
components to be used on top of this kernel. VirtualBox is an exception
because it closely interacts with the underlying kernel (like NOVA) to
attain good performance. We have shown that VirtualBox can be executed
within a protection domain of the NOVA microhypervisor. The goal of this
project is the application of this approach to the virtualization
interface of seL4. The result will be a VM hosting environment that
ensures the separation of virtual machines via the formally verified
seL4 kernel.
:Xen as kernel for Genode:
Using Xen as kernel for Genode would clear the way to remove the
@@ -294,22 +278,25 @@ Virtualization
the project bears the opportunity to explore the provisioning of the
KVM interface based on Genode's VFS plugin concept.
:Hardware-accelerated graphics for virtual machines:
In
[https://genode.org/documentation/release-notes/17.08#Hardware-accelerated_graphics_for_Intel_Gen-8_GPUs - Genode 17.08],
we introduced a GPU multiplexer for Intel Broadwell along with support
for Mesa-based 3D-accelerated applications.
While designing Genode's GPU-session interface, we also aimed at supporting
the hardware-accelerated graphics for Genode's virtual machine monitors like
VirtualBox or Seoul, but until now, we did not took the practical steps of
implementing a virtual GPU device model.
The goal of this project is the offering of a virtual GPU to a Linux guest
OS running on top of Genode's existing virtualization and driver
infrastructure.
Device drivers
##############
:Isochronous USB devices:
Genode's USB driver supports bulk and interrupt endpoints. Thereby, most
USB devices like USB storage, user input, printers, and networking devices
can be used. However, multi-media devices such as cameras or audio
equipment use isochronous endpoints, which are not supported. The goal
of this line of work is the support of these devices in Genode. The topic
touches the USB driver, the USB session interface, an example implementation
of a USB client driver (using the session interface) for a device of choice,
and - potentially - the enhancement of Genode's USB-pass-through mechanism
for VirtualBox.
:Sound on the Raspberry Pi:
The goal of this project is a component that uses the Raspberry Pi's
@@ -318,24 +305,12 @@ Device drivers
backend, the new driver will make the sound of all SDL-based games
available on the Raspberry Pi.
:Framebuffer for UEFI and Coreboot:
By moving away from the legacy BIOS boot mechanism, it is time to
reconsider closely related traditional approaches such as the use of
the VESA BIOS extensions for accessing the frame buffer. On UEFI or
Coreboot systems, there exist alternative ways to initialize and
access the framebuffer in a hardware-independent way. On the course of
this project, we will explore the available options and create dedicated
Genode driver components that use the modern mechanisms.
For reference, the current state of Genode's UEFI support is documented
in [https://github.com/genodelabs/genode/issues/2242 - Issue 2242].
:Data Plane Development Kit (DPDK):
Genode utilizes the network device drivers of the iPXE project, which
perform reasonably well for everyday use cases but are obviously not
designated for high-performance networking.
The [http://dpdk.org/ - DPDK] is a vendor-supported suite of network device
The [https://dpdk.org/ - DPDK] is a vendor-supported suite of network device
drivers that is specifically developed for high-performance applications.
It presents an attractive alternative to iPXE-based drivers. This project
has the goal to make DPDK drivers available as a Genode component.
@@ -357,8 +332,22 @@ Platforms
Genode functionality such as its native GUI, lwIP, and Noux, many protocol
stacks can effectively be removed from the Linux kernel.
The goal of this project is to evaluate how small the Linux kernel can get
when used as a microkernel.
In 2018, Johannes Kliemann pursued this topic to a state where Genode
could be used as init process atop a customized Linux kernel.
[https://lists.genode.org/pipermail/users/2018-May/006066.html - His work]
included the execution of Genode's regular device drivers for VESA and
PS/2 as regular Genode components so that Genode's interactive demo
scenario ran happily on a laptop. At this time, however, only parts of
his results were merged into Genode's mainline.
The goal of this project is to follow up on Johannes' work, bring the
[https://github.com/genodelabs/genode/pull/2829 - remaining parts] into
shape for the inclusion into Genode, and address outstanding topics, in
particular the handling of DMA by user-level device drivers. Further down
the road, it would be tempting to explore the use of
[https://en.wikipedia.org/wiki/Seccomp - seccomp] as sandboxing mechanism
for Genode on Linux and the improvement of the Linux-specific implementation
of Genode's object-capability model.
:Support for the HelenOS/SPARTAN kernel:
@@ -381,34 +370,46 @@ Platforms
kernel is used for Mac OS X, it could represent an industry-strength
base platform for Genode supporting all CPU features as used by Mac OS X.
:Linux process containers for supporting Genode`s resource trading:
:Genode on the Librem5 phone hardware:
Even though the Linux version of Genode is primarily meant as a development
platform, there exist interesting opportunities to explore when combining
Genode with Linux, in particular Linux' process containers.
Linux process containers provide a mechanism to partition physical resources,
foremost CPU time, between Linux processes. This raises the interesting
question of whether this mechanism could be used for a proper implementation
of Genode's resource trading on Linux.
[http://lwn.net/Articles/236038/ - Process containers introduction...]
Even though there exists a great variety of ARM-based SoCs, Genode
primarily focuses on the NXP i.MX family because it is - in contrast
to most SoCs in the consumer space - very liberal in terms of
good-quality public documentation and reference code, and it scales
from industrial to end-user-facing use cases (multi-media).
The [https://puri.sm/products/librem-5/ - Librem5] project - with its
mission to build a trustworthy mobile phone - has chosen the i.MX family as
the basis for their product for likely the same reasons that attract us.
To goal of this work is bringing Genode to the Librem5 hardware.
For the Librem5 project, Genode could pave the ground towards new use cases
like high-security markets where a regular Linux-based OS would not be
accepted. For the Genode community, the Librem5 hardware could become an
attractive mobile platform for everyday use, similar to how we developers
use our Genode-based [https://genode.org/download/sculpt - Sculpt OS] on our
laptops.
System management
#################
:Remote management of Sculpt OS via Puppet:
[https://en.wikipedia.org/wiki/Puppet_(company)#Puppet - Puppet] is a
software-configuration management tool for administering a large amount
of machines from one central place. Genode's
[https://genode.org/download/sculpt - Sculpt OS] lends itself to such
an approach of remote configuration management by the means of the
"config" file system (for configuring components and deployments) and
the "report" file system (for obtaining the runtime state of components).
The project would explore the application of the Puppet approach and tools
to Sculpt OS.
Optimizations
#############
:Low-latency audio streaming:
Genode comes with an audio streaming interface called 'Audio_out' session.
It is based on a shared-memory packet stream accompanied with asynchronous
data-flow signals. For real-time audio processing involving chains of Genode
components, streams of audio data must be carried at low latency, imposing
constraints to buffer sizes and the modes of operation of the audio mixer and
audio drivers. The goal of this project is to create a holistic design of the
whole chain of audio processing, taking thread-scheduling into account. A
particular challenge is the mixed output of real-time (small buffer, low
latency) and non-real-time (larger buffer to compensate jitter, higher
latency) audio sources.
:De-privileging the VESA graphics driver:
The VESA graphics driver executes the graphics initialization code provided

18
doc/coding_style.txt
View File

@@ -93,6 +93,24 @@ and the source code always looks good.
_Hint:_ In VIM, use the 'set list' and 'set listchars' commands to make tabs
and spaces visible.
* If class initializers span multiple lines, put the colon on a separate
line and indent the initializers using one tab. For example:
! Complicated_machinery(Material &material, Deadline deadline)
! :
! <tab>_material(material),
! <tab>_deadline(deadline),
! <tab>...
! {
! ...
! }
* Preferably place statements that alter the control flow - such as
'break', 'continue', or 'return' - at the beginning of a separate line,
followed by vertical space (a blank line or the closing brace of the
surrounding scope).
! if (early_return_possible)
! return;
Switch statements
~~~~~~~~~~~~~~~~~

495
doc/components.txt
View File

@@ -14,6 +14,11 @@ Genode comes with a growing number of components apparently scattered across
various repositories. This document provides an overview of these components
and outlines the systematics behind them.
The scope of this document is limited to the Genode main repository maintained
by Genode Labs. Many additional components and device drivers can be found in
the community-maintained
[https://github.com/genodelabs/genode-world/ - Genode-World] repository.
Categorization of components
############################
@@ -26,11 +31,11 @@ of them is briefly characterized as follows:
session interfaces. Naturally, a device driver is specific to a
particular hardware platform. The hardware resources are accessed
via core's IO_MEM, IO_PORT, and IRQ services. The functionality of
the driver is made available to other system components by announcing
the driver is made available to other system components via
one of Genode's device-independent session interfaces, which are
'platform_session', 'framebuffer_session', 'input_session', 'block_session',
'platform_session', 'capture_session', 'event_session', 'block_session',
'audio_out_session', 'log_session', 'nic_session', and 'timer_session'
(see 'os/include/' for the interface definitions). Those interfaces are
(see _os/include/_ for the interface definitions). Those interfaces are
uniform across hardware platforms and kernel base platforms. Usually,
each device driver can accommodate only one client at a time.
@@ -59,7 +64,7 @@ of them is briefly characterized as follows:
Device drivers
##############
Device drivers usually reside in the 'src/drivers' subdirectory of source-code
Device drivers usually reside in the _src/drivers/_ subdirectory of source-code
repositories. The most predominant repositories hosting device drivers are
'os', 'dde_ipxe', 'dde_linux'.
@@ -67,19 +72,23 @@ repositories. The most predominant repositories hosting device drivers are
Platform devices
================
:'os/src/drivers/platform/': Platform drivers for various platforms.
:_os/src/drivers/platform/_: Platform drivers for various platforms.
On x86, the platform driver uses the PCI controller as found on x86 PC
hardware. A client can probe for a particular device and request information
about physical device resources (using the 'platform_device' interface). I/O
resources for MMIO regions, I/O ports, and interrupts can be requested by the
provided device abstraction.
:'os/src/drivers/acpi':
:_os/src/drivers/acpi/_:
On x86 platforms that use the APIC (namely Fiasco.OC, NOVA, and hw_x86_64)
this simple ACPI parser traverses the ACPI tables and reports device-resource
information (e.g., interrupt lines of PCI devices).
:'libports/src/app/acpica':
:_os/src/app/smbios_decoder/_:
A component that parses SMBIOS information on x86 platforms and makes the
result available as a report.
:_libports/src/app/acpica/_:
In addition to our ACPI base driver, the acpica component uses the
ACPICA library to provide access to dynamic functions like battery
states, events (e.g., notebook lid close and power buttons), as well
@@ -93,91 +102,75 @@ UART devices
The UART device drivers implement the UART-session interface.
:'os/src/drivers/uart/spec/pl011':
:_os/src/drivers/uart/spec/pbxa9/_:
Driver for the PL011 UART as found on many ARM-based platforms.
:'os/src/drivers/uart/spec/i8250':
:_os/src/drivers/uart/spec/x86/_:
Driver for the i8250 UART as found on PC hardware.
:'os/src/drivers/uart/spec/omap4':
Driver for the UART as found on OMAP4-based hardware.
:'os/src/drivers/uart/spec/exynos5':
Driver for the UART as found on Exynos-5-based hardware.
Framebuffer and input drivers
=============================
Framebuffer and input drivers implement the framebuffer-session interface and
input-session interfaces respectively.
Framebuffer and input drivers are implemented as clients of the
capture-session and event-session interfaces respectively.
:'os/src/drivers/input/dummy':
Pseudo input driver without accessing any hardware. This component is useful
to resolve a dependency from an input session for scenarios where no user
input is required.
:'os/src/drivers/input/spec/ps2/x86':
:_os/src/drivers/ps2/x86/_:
Driver for the 'i8042' PS/2 controller as found in x86 PCs. It supports both
mouse (including ImPS/2, ExPS/2) and keyboard.
:'os/src/drivers/input/spec/ps2/pl050':
:_os/src/drivers/ps2/pl050/_:
Driver for the PL050 PS/2 controller as found on ARM platforms such as
VersatilePB. The physical base address used by the driver is obtained at
compile time from a header file called 'pl050_defs.h'. The version of the
VersatilePB platform can be found at 'os/include/platform/vpb926/' and
compile time from a header file called _pl050_defs.h_. The version of the
VersatilePB platform can be found at _os/include/platform/vpb926/_ and
is made available to the driver via the SPECS machinery of the Genode build
system.
:'os/src/drivers/input/spec/imx53':
Input driver for Egalaxy touchscreen and Freescale's MPR121
capacitative touch buttons on i.MX53.
:'libports/src/drivers/framebuffer/vesa':
:_libports/src/drivers/framebuffer/vesa/_:
Driver using VESA mode setting on x86 PCs. For more information, please refer
to the README file in the driver directory.
:'libports/src/drivers/framebuffer/boot':
:_libports/src/drivers/framebuffer/boot/_:
Driver for boot-time initialized framebuffers (e.g., UEFI GOP)
discovered from the 'platform_info' ROM
:'os/src/drivers/framebuffer/spec/pl11x':
:_os/src/drivers/framebuffer/pl11x/_:
Driver for the PL110/PL111 LCD display.
:'os/src/drivers/framebuffer/spec/omap4':
Driver for HDMI output on OMAP4 SoCs.
:'os/src/drivers/framebuffer/spec/exynos5':
Driver for HDMI output on Exynos-5 SoCs.
:'os/src/drivers/framebuffer/spec/imx53':
:_os/src/drivers/framebuffer/imx53/_:
Driver for LCD output on i.MX53 SoCs.
:'os/src/drivers/framebuffer/spec/rpi':
:_os/src/drivers/framebuffer/rpi/_:
Driver for the HDMI output of the Raspberry Pi.
:'os/src/drivers/framebuffer/spec/sdl':
:_os/src/drivers/framebuffer/sdl/_:
Serves as both framebuffer and input driver on Linux using libSDL. This
driver is only usable on the Linux base platform.
:'os/src/drivers/gpu/intel':
Intel Graphics GPU multiplexer for Broadwell and newer.
:_os/src/drivers/gpu/intel/_:
An experimental Intel Graphics GPU multiplexer for Broadwell and newer.
:'dde_linux/src/drivers/framebuffer/intel':
:_dde_linux/src/drivers/framebuffer/intel/_:
Framebuffer driver for Intel i915 compatible graphic cards based on
the Linux Intel KMS driver.
:'dde_linux/src/drivers/usb':
USB driver that makes USB HID and USB storage devices available as input
sessions and block session respectively. For examples of using this driver,
refer to the run scripts at 'dde_linux/run/usb_hid' and
'dde_linux/run/usb_storage'.
:_dde_linux/src/drivers/usb_host/_:
USB host-controller driver that provides an USB session interface to
USB drivers.
:_dde_linux/src/drivers/usb_hid/_:
USB Human Interface Device driver using the USB session interface.
:_os/src/drivers/usb_block/_:
USB storage driver that uses the USB session interface and provides
a block-session interface.
Timer drivers
=============
The timer driver located at 'os/src/drivers/timer' implements the timer-session
The timer driver located at _base/src/timer/_ implements the timer-session
interface. Technically, it is both a device driver (accessing a timer
device) and a resource multiplexer (supporting multiple timer-session clients
at the same time). Depending on the base platform, the implementation uses
@@ -198,13 +191,13 @@ Audio drivers
=============
Audio drivers implement the Audio_out session interface defined at
'os/include/audio_out_session/' for playback and optionally the audio_in
_os/include/audio_out_session/_ for playback and optionally the audio_in
interface for recording.
:'os/src/drivers/audio/spec/linux':
:_os/src/drivers/audio/spec/linux/_:
Uses ALSA as back-end on the Linux base platform and supports only playback.
:'dde_bsd/src/drivers/audio':
:_dde_bsd/src/drivers/audio/_:
Sound drivers ported from OpenBSD. Currently, the repository
includes support for Intel HD Audio as well as for Ensoniq AudioPCI
(ES1370) compatible sound cards.
@@ -214,34 +207,31 @@ Block drivers
=============
All block drivers implement the block-session interface defined at
'os/include/block_session/'.
_os/include/block_session/_.
:'os/src/drivers/sd_card/spec/pl180':
:_os/src/drivers/sd_card/spec/pl180/_:
Driver for SD-cards connected via the PL180 device as found on the PBX-A9
platform.
:'os/src/drivers/sd_card/spec/omap4':
Driver for SD-cards connected to the SD-card controller of the OMAP4 SoC.
:'os/src/drivers/sd_card/spec/exynos5':
Driver for SD-cards and eMMC connected to Exynos-5-based platforms.
:'os/src/drivers/sd_card/spec/imx53':
:_os/src/drivers/sd_card/spec/imx53/_:
Driver for SD-cards connected to the Freescale i.MX53 platform like the
Quick Start Board or the USB armory device.
:'os/src/drivers/sd_card/spec/rpi':
:_os/src/drivers/sd_card/spec/rpi/_:
Driver for SD-cards connected to the Raspberry Pi.
:'dde_linux/src/drivers/usb':
:_dde_linux/src/drivers/usb/_:
USB driver that makes USB storage devices available as block sessions.
For an example of using this driver, refer to the run script at
'dde_linux/run/usb_storage'.
_dde_linux/run/usb_storage.run_.
:'os/src/drivers/ahci':
:_os/src/drivers/ahci/_:
Driver for SATA disks and CD-ROMs on x86 PCs.
:'os/src/drivers/usb_block':
:_os/src/drivers/nvme/_:
Driver for NVMe block devices on x86 PCs.
:_os/src/drivers/usb_block/_:
USB Mass Storage Bulk-Only driver using the USB session interface.
@@ -249,86 +239,83 @@ Network interface drivers
=========================
All network interface drivers implement the NIC session interface
defined at 'os/include/nic_session'.
defined at _os/include/nic_session/_.
:'os/src/drivers/nic/spec/linux':
:_os/src/drivers/nic/spec/linux/_:
Driver that uses a Linux tap device as back end. It is only useful on the
Linux base platform.
:'os/src/drivers/nic/spec/lan9118':
:_os/src/drivers/nic/spec/lan9118/_:
Native device driver for the LAN9118 network adaptor as featured on the
PBX-A9 platform.
:'os/src/drivers/nic/gem':
Device driver for Cadence EMAC PS network adaptor as featured on the
Xilinx Zynq.
:'dde_ipxe/src/drivers/nic':
:_dde_ipxe/src/drivers/nic/_:
Device drivers ported from the iPXE project. Supported devices are Intel
E1000 and pcnet32.
:'dde_linux/src/drivers/wifi':
:_dde_linux/src/drivers/wifi/_:
The wifi_drv component is a port of the Linux mac802.11 stack, including the
iwlwifi driver. It enables the use of Intel Wireless 6xxx and 7xxx cards.
:'dde_linux/src/drivers/usb':
:_dde_linux/src/drivers/usb/_:
For the OMAP4 platform, the USB driver contains the networking driver.
:'dde_linux/src/drivers/nic/fec':
:_dde_linux/src/drivers/nic/fec/_:
Driver for ethernet NICs of the i.MX SoC family.
General-purpose I/O drivers
===========================
:'os/src/drivers/gpio/spec/omap4':
Driver for accessing the GPIO pins of OMAP4 platforms.
:'os/src/drivers/gpio/spec/imx53':
:_os/src/drivers/gpio/spec/imx53/_:
Driver for accessing the GPIO pins of i.MX53 platforms.
:'os/src/drivers/gpio/spec/rpi':
:_os/src/drivers/gpio/spec/rpi/_:
Driver for accessing the GPIO pins of Raspberry Pi platforms.
:'os/src/drivers/gpio/spec/exynos5':
Driver for accessing the GPIO pins of Exynos4 platforms, e.g.,
Odroid-X2.
Resource multiplexers
#####################
By convention, resource multiplexers are located at the 'src/server'
By convention, resource multiplexers are located at the _src/server/_
subdirectory of a source repository.
:Framebuffer and input: The framebuffer and input session interfaces can be
multiplexed using the Nitpicker GUI server, which allows multiple clients to
create and manage rectangular areas on screen. Nitpicker uses one input
session and one framebuffer session as back end and, in turn, provides
so-called nitpicker sessions to one or multiple clients. Each nitpicker
session contains a virtual framebuffer and a virtual input session. Nitpicker
(including a README file) is located at 'os/src/server/nitpicker'.
:Framebuffer and input: Framebuffer and input devices can be multiplexed using
the Nitpicker GUI server, which allows multiple clients to create and manage
rectangular areas on screen. Nitpicker serves as broker between input
devices, output devices, and graphical applications. It provides an event
service for input drivers, a capture service for output drivers, and a GUI
service for the applications. Each GUI session contains a virtual
framebuffer and a virtual input interface. Nitpicker (including a README
file) is located at _os/src/server/nitpicker/_.
:Audio output: The audio mixer located at 'os/src/server/mixer' enables
:Audio output: The audio mixer located at _os/src/server/mixer/_ enables
multiple clients to use the audio-out interface. The mixing is done by simply
adding and clamping the signals of all present clients.
:Networking: The NIC bridge located at 'os/src/server/nic_bridge' multiplexes
:Networking: The NIC bridge located at _os/src/server/nic_bridge/_ multiplexes
one NIC session to multiple virtual NIC sessions using a proxy-ARP
implementation. Each client has to obtain a dedicated IP address visible to
the physical network. DHCP requests originating from the virtual NIC sessions
are delegated to the physical network.
:Block: The block-device partition server at 'os/src/server/part_blk' reads
The NIC router located at _os/src/server/nic_router/_ multiplexes one NIC
session to multiple virtual NIC sessions by applying network address
translation (NAT).
:Block: The block-device partition server at _os/src/server/part_block/_ reads
the partition table of a block session and exports each partition found as
separate block session. For using this server, please refer to the run
script at 'os/run/part_blk'.
script at _os/run/part_block.run_.
:File system: The FAT file-system service allows multiple clients to
concurrently access the same FAT-formatted block device. It is located
at 'libports/src/server/fatfs_fs' and supports FAT, FAT32, and exFAT.
:File system: The VFS file-system server allows multiple clients to
concurrently access the same virtual file system. It is located at
_os/src/server/vfs/_. The VFS can be assembled out of several builtin
file-system types (like a RAM file system, or pseudo file systems for
various Genode session interfaces) as well as external plugins such as rump
(mounting file systems supported by the NetBSD kernel).
:Terminal: The terminal_mux service located at gems/src/server/terminal_mux
:Terminal: The terminal_mux service located at _gems/src/server/terminal_mux/_
is able to provide multiple terminal sessions over one terminal-client
session. The user can switch between the different sessions using a keyboard
shortcut, which brings up an ncurses-based menu.
@@ -340,214 +327,211 @@ Protocol stacks
Protocol stacks come either in the form of separate components that translate
one session interface to another, or in the form of libraries.
Separate components:
Separate components
===================
:'os/src/server/nit_fb':
Translates a nitpicker session to a pair of framebuffer and input sessions.
Each 'nit_fb' instance is visible as a rectangular area on screen presenting
:_os/src/server/gui_fb/_:
Translates a GUI session to a pair of framebuffer and input sessions.
Each 'gui_fb' instance is visible as a rectangular area on screen presenting
a virtual frame buffer. The area is statically positioned. For more
information, please refer to 'os/src/server/nit_fb/README'.
information, please refer to _os/src/server/gui_fb/README_.
:'gems/src/server/wm':
Window manager that implements the nitpicker session interface but manages
:_gems/src/server/wm/_:
Window manager that implements the GUI session interface but manages
each client view as a separate window. The window decorations are provided
by a so-called decorator (e.g., 'gems/src/app/decorator'). The behaviour
by a so-called decorator (e.g., _gems/src/app/decorator/_). The behaviour
is defined by a so-called window layouter such as the floating window
layouter located at 'gems/src/app/floating_window_layouter/'.
layouter located at _gems/src/app/floating_window_layouter/_.
:'demo/src/server/liquid_framebuffer':
Implements the same translation as 'nit_fb' but by presenting an interactive
:_demo/src/server/liquid_framebuffer/_:
Implements the same translation as 'gui_fb' but by presenting an interactive
window rather than a statically positioned screen area.
:'os/src/server/tar_rom':
:_os/src/server/tar_rom/_:
Provides each file contained in a tar file obtained via Genode's ROM session
as separate ROM session.
:'os/src/server/iso9660':
:_os/src/server/iso9660/_:
Provides each file of an ISO9660 file system accessed via a block session as
separate ROM session.
:'os/src/server/ram_fs':
A file-system implementation that keeps all data in memory.
:'dde_rump/src/server/rump_fs':
A file-system server that contains various file-systems ported from the
NetBSD kernel.
:'os/src/server/lx_fs':
:_os/src/server/lx_fs/_:
A file system server that makes the file system of a Linux base platform
available to Genode.
:'os/src/server/trace_fs':
A pseudo file system that can be used as a front end to core's TRACE
service.
:_os/src/server/vfs_block/_:
Provides the content of a file obtained from a VFS as a block session,
similar to the loop-mount mechanism on Linux
:'os/src/server/rom_blk':
Provides the content of a ROM file as a block session, similar to the
loop-mount mechanism on Linux
:'os/src/server/ram_blk':
Provides the content of a RAM dataspace as a block session. In contrast
to 'rom_blk', this server provides a writeable block device.
:'os/src/server/terminal_log':
:_os/src/server/terminal_log/_:
Adapter for forwarding LOG messages to a terminal session.
:'os/src/server/log_terminal':
:_os/src/server/log_terminal/_:
Adapter for forwarding terminal output to a LOG session.
:'os/src/server/fs_log':
:_os/src/server/fs_log/_:
Adapter that writes LOG messages to files on a file system.
:'demo/src/server/nitlog':
Provides a LOG session, printing log output on screen via a nitpicker
session.
:_demo/src/server/nitlog/_:
Provides a LOG session, printing log output on screen via a GUI session.
:'os/src/app/rom_logger':
:_os/src/app/rom_logger/_:
The rom_logger component requests a ROM session and writes the
content of the ROM dataspace to the LOG.
:'os/src/server/rom_filter':
:_os/src/server/rom_filter/_:
The ROM filter provides a ROM module that depends on the content of
other ROM modules steered by the filter configuration, e.g., dynamic
switching between configuration variants dependent on the state of
the system.
:'os/src/server/vfs':
A file-system server using the VFS library and plugins as backend.
:'os/src/server/log_terminal':
:_os/src/server/log_terminal/_:
Forwards terminal output to a LOG session.
:'gems/src/server/file_terminal':
:_gems/src/server/file_terminal/_:
Provides terminal sessions that target files on a file system.
:'gems/src/server/terminal':
:_gems/src/server/terminal/_:
Provides a terminal session via a graphical terminal using a framebuffer
session and an input session.
:'gems/src/server/tcp_terminal':
:_gems/src/server/tcp_terminal/_:
Provides one or multiple terminal sessions over TCP connections.
For further information, refer to 'gems/src/server/tcp_terminal/README'.
For further information, refer to _gems/src/server/tcp_terminal/README_.
:'os/src/server/terminal_crosslink':
:_os/src/server/terminal_crosslink/_:
The terminal crosslink service allows to terminal clients to talk to each
other.
:'gems/src/server/http_blk':
:_gems/src/server/http_block/_:
A block service that fetches a virtual block device over the network from
a HTTP server.
:'os/src/server/fs_rom':
:_os/src/server/fs_rom/_:
A ROM service that translates the 'File_system' session interface to the
'ROM' session' interface. Each request for a ROM file is handled by looking
up an equally named file on the file system.
Please refer to 'os/src/server/fs_rom' for more information.
Please refer to _os/src/server/fs_rom/_ for more information.
:'os/src/server/dynamic_rom':
For use cases where ROMs are known to be static, the
_os/src/server/cached_fs_rom/_ can be considered as a faster alternative to
the regular 'fs_rom' server. Note that 'cached_fs_rom' is not supported
in base-linux though.
:_os/src/server/chroot/_:
An intermediate file-system server that makes a sub directory of a file
system available as the root of a file system handed out to its client.
:_os/src/server/dynamic_rom/_:
A simple ROM service that provides ROM modules that change in time according
to a configured timeline.
:'os/src/server/report_rom':
:_os/src/server/report_rom/_:
A service that implements both the report session interface and the ROM
session interface. It reflects incoming reports as ROM modules.
:'os/src/server/fs_report':
:_os/src/server/fs_report/_:
Report server that writes reports to file-systems
:'os/src/server/clipboard':
:_os/src/server/clipboard/_:
This component is both a report service and a ROM service. The
clients of the report service can issue new clipboard content, which
is then propagated to the clients of the ROM service according to a
configurable information-flow policy.
:'ports/src/app/openvpn':
OpenVPN enables access to remote network resources through a secure tunnel
by providing an encrypted connection to a remote host. It is plugged between
NIC server (such as a network driver) and NIC client.
:_os/src/server/event_filter/_:
A component that transforms and merges input events from multiple sources
into a single event stream.
:'os/src/server/input_merger':
A component that merges input events from multiple sources into a single
stream.
:'libports/src/server/acpi_input':
:_libports/src/app/acpi_event/_:
A component that transforms ACPI events into Genode input events.
:'gems/src/server/nit_fader':
A wrapper for nitpicker's session interface that applies alpha-blending to
the of views a nitpicker client.
:_gems/src/server/gui_fader/_:
A wrapper for nitpicker's GUI session interface that applies alpha-blending
to the of views a GUI client.
Libraries:
:'libports/lib/mk/libc':
VFS plugins
===========
VFS plugins are file-system drivers in the form of shared libraries that
implement the VFS-plugin interface. They can be combined with any application
based on Genode's C runtime, with the VFS server, and with non-POSIX
components that use the Genode's VFS library directly.
:_gems/src/lib/vfs/trace/_:
A VFS plugin that makes core's TRACE service accessible as a pseudo
file system.
:_gems/src/lib/vfs/import/_:
A VFS plugin that pre-populates a VFS with initial content.
:_gems/src/lib/vfs/pipe/_:
A VFS plugin that provides bi-directional pipes for exchanging streamed
data between components.
:_gems/src/lib/vfs/ttf/_:
A VFS plugin that makes rendered pixel data of the glyphs of Truetype fonts
available as a pseudo file system.
:_libports/src/lib/vfs/jitterentropy/_:
A VFS plugin that provides random numbers based on the jitter of executing
CPU instructions.
:_libports/src/lib/vfs/lwip/_:
A VFS plugin that uses the light-weight IP (lwIP) stack to provide a
network socket interface as a pseudo file system.
:_dde_linux/src/lib/vfs/lxip/_:
A VFS plugin that uses the TCP/IP stack ported from the Linux kernel to
provide a network socket interface as a pseudo file system.
:_libports/src/lib/vfs/fatfs/_:
A VFS plugin that allows for the mounting of FAT-formatted block devices.
:_dde_rump/src/lib/vfs/rump/_:
A VFS plugin that enables the use of NetBSD's file-system drivers such
as ext2 or msdos.
Libraries
=========
:_libports/lib/mk/libc/_:
C runtime ported from FreeBSD.
:'libports/lib/mk/libc_lwip_nic_dhcp':
Translates the BSD socket API to a NIC session using the lwIP stack.
:'dde_linux/lib/mk/libc_lxip':
Translates the BSD socket API to a NIC session using the Linux TCP/IP stack.
:'libports/lib/mk/libc_ffat':
Accesses files on a block device that contains a FAT32 file system.
:'libports/lib/mk/libc_fuse_exfat':
Accesses files on a block device that contains an exFAT file system.
:'libports/lib/mk/libc_fuse_ext2':
Accesses files on a block device that contains an ext2 file system.
:'libports/lib/mk/libc_terminal':
Connects the standard input and output from/to Genode's terminal session
interface.
:'libports/lib/mk/stdcxx':
:_libports/lib/mk/stdcxx/_:
Standard C++ library
:'libports/lib/mk/mesa_api':
:_libports/lib/mk/mesa_api/_:
Mesa OpenGL API with backends for software rasterization (egl_swrast)
and Intel Graphics (egl_i965)
:'libports/lib/mk/pthread':
Subset of the POSIX thread and semaphore API.
:'libports/lib/mk/python':
Runtime of the Python scripting language.
:'libports/lib/mk/mupdf':
:_libports/lib/mk/mupdf/_:
PDF rendering engine.
:'libports/lib/mk/sdl':
Translates the libSDL API to framebuffer and input sessions.
:'libports/lib/mk/ncurses':
:_libports/lib/mk/ncurses/_:
Library for implementing pseudo-graphical applications (i.e., VIM) that
run on a text terminal.
:'libports/lib/mk/avcodec':
A library for video decoding, conversion, and streaming.
:_libports/lib/mk/qt5_*/_:
Qt5 framework, using GUI session and NIC session as back end.
:'libports/lib/mk/lua':
Runtime for the Lua scripting language.
:'libports/lib/mk/qt5_*':
Qt5 framework, using nitpicker session and NIC session as back end.
:'libports/lib/mk/vfs_jitterentropy.mk':
:_libports/lib/mk/vfs_jitterentropy.mk_:
A VFS plugin that makes a jitter-based random-number generator available
as a file within the process-local VFS.
:'libports/lib/mk/libarchive.mk':
:_libports/lib/mk/libarchive.mk_:
Library providing a common interface to a variety of archive
formats.
:'libports/lib/mk/lz4.mk':
:_libports/lib/mk/lz4.mk_:
Library for processing LZ4 lossless compression archives.
:'libports/lib/mk/liblzma.mk':
:_libports/lib/mk/liblzma.mk_:
Library for processing LZMA archives.
:'libports/lib/mk/libgcrypt.mk':
:_libports/lib/mk/libgcrypt.mk_:
GnuPG library for OpenPGP processing, e.g., signature verification.
@@ -555,103 +539,96 @@ Applications
############
Applications are Genode components that use other component's services but
usually do not provide services. They are typically located in the 'src/app/'
usually do not provide services. They are typically located in the _src/app/_
subdirectory of a repository. Most applications come with README files
located in their respective directory.
:'gems/src/app/backdrop':
Nitpicker client application that sets a composition of PNG images as
desktop background.
:_gems/src/app/backdrop/_:
GUI client application that sets a composition of PNG images as desktop
background.
:'demo/src/app/launchpad':
:_demo/src/app/launchpad/_:
Graphical application for interactively starting and killing subsystems.
:'gems/app/launcher': Graphical launcher of Genode subsystems.
:'os/app/cli_monitor': Command-line-based launcher of Genode subsystems.
:'demo/src/app/scout':
:_demo/src/app/scout/_:
Graphical hypertext browser used for Genode's default demonstration scenario.
:'libports/src/test/mesa_demo':
Example programs for using the Mesa OpenGL graphics stack.
:'ports/src/app/arora':
Arora is a Qt-based web browser using the Webkit engine.
:'ports/src/app/gdb_monitor':
:_ports/src/app/gdb_monitor/_:
Application that allows the debugging of a process via GDB over a remote
connection.
:'libports/src/app/qt5/qt_launchpad':
:_libports/src/app/qt5/qt_launchpad/_:
Graphical application starter implemented using Qt.
:'libports/src/app/qt5/examples/':
:_libports/src/app/qt5/examples/_:
Several example applications that come with Qt.
:'os/src/app/sequence':
:_os/src/app/sequence/_:
Simple utility to serialize the execution of multiple components
:'ports/src/noux-pkg':
:_ports/src/noux-pkg/_:
Ports of popular commandline-based Unix software such as VIM, bash,
coreutils, binutils, gcc, findutils, and netcat. The programs are supposed
to be executed within the Noux runtime environment.
:'ports/src/app/lighttpd':
:_ports/src/app/lighttpd/_:
Lighttpd is a fast and feature-rich web server. The port of lighttpd uses
a file-system session to access the website content and the web-server
configuration.
:'os/src/app/trace_logger':
:_os/src/app/trace_logger/_:
Convenient, runtime-configurable frontend to the tracing facility.
:'os/src/app/rom_reporter':
:_os/src/app/rom_reporter/_:
The ROM-reporter component requests a ROM session and reports the
content of the ROM dataspace to a report session with the same label
as the ROM session.
:'os/src/app/log_core':
:_os/src/app/log_core/_:
Component transforming core and kernel output to Genode LOG output.
Package-management components
=============================
:'gems/src/app/depot_query':
:_gems/src/app/depot_query/_:
Tool for querying subsystem information from a depot.
:'gems/src/app/depot_download_manager':
:_gems/src/app/depot_download_manager/_:
Tool for managing the download of depot content.
:'gems/src/app/depot_deploy':
:_gems/src/app/depot_deploy/_:
Subsystem init configuration generator based on blueprints.
:'libports/src/app/fetchurl':
:_libports/src/app/fetchurl/_:
A runtime-configurable frontend to the libcURL library for
downloading content.
:'libports/src/app/extract':
:_libports/src/app/extract/_:
Tool for extracting archives using libarchive.
:'ports/src/app/verify':
:_ports/src/app/verify/_:
This component verifies detached OpenPGP signatures using libgcrypt.
Runtime environments
####################
:'ports/src/noux': Noux is an experimental implementation of a UNIX-like API
that enables the use of unmodified command-line based GNU software. For using
noux, refer to the run script 'ports/run/noux.run'.
:'ports/src/app/seoul': Seoul is a virtual-machine monitor developed for
:_ports/src/app/seoul/_: Seoul is a virtual-machine monitor developed for
the use with the NOVA platform. It virtualizes 32bit x86 PC hardware
including various peripherals.
:'os/src/server/loader': A service that allows the creation and destruction
:_os/src/server/loader/_: A service that allows the creation and destruction
of Genode subsystems via a session interface. For further information,
refer to 'os/src/server/loader/README'.
refer to _os/src/server/loader/README_.
:'ports/src/app/dosbox': A port of DosBox for executing DOS software.
:_ports/src/virtualbox5/_: VirtualBox running on top of the NOVA hypervisor.
:'ports/src/virtualbox': VirtualBox running on top of the NOVA hypervisor.
:_os/src/server/vmm/_: A virtual machine monitor that is based on
hardware-assisted virtualization of ARM platforms. It is supported on
the base-hw kernel only.
:_os/src/server/cpu_balancer/_: The CPU balancer intercepts the interaction
of components with core's low-level services to migrate threads dynamically
between CPU cores.

2
doc/contributions.txt
View File

@@ -140,7 +140,7 @@ permission to let Genode Labs redistribute his contributions under non-AGPLv3
licenses. This permission is granted by signing the Genode Contributors
Agreement:
:[http:gca.pdf - Genode Contributor's Agreement]:
:[https://genode.org/community/gca.pdf - Genode Contributor's Agreement]:
Genode Contributor's Agreement (GCA)
By signing the GCA, you don't lose any rights for your contribution. However,

21
doc/getting_started.txt
View File

@@ -50,17 +50,26 @@ This creates a new build directory for building x86_64 binaries in './build'.
The build system creates unified binaries that work on the given
architecture independent from the underlying base platform, in this case Linux.
To give Genode a try, build and execute a simple demo scenario via:
Now change into the fresh build directory:
! cd build/x86_64
! make KERNEL=linux run/demo
Please uncomment the following line in 'etc/build.conf' to make the
build process as smooth as possible.
! RUN_OPT += --depot-auto-update
To give Genode a try, build and execute a simple demo scenario via:
! make KERNEL=linux BOARD=linux run/demo
By invoking 'make' with the 'run/demo' argument, all components needed by the
demo scenario are built and the demo is executed. This includes all components
which are implicitly needed by the base platform. The base platform that the
components will be executed upon on is selected via the 'KERNEL' variable. If
you are interested in looking behind the scenes of the demo scenario, please
refer to 'doc/build_system.txt' and the run script at 'os/run/demo.run'.
components will be executed upon on is selected via the 'KERNEL' and 'BOARD'
variables. If you are interested in looking behind the scenes of the demo
scenario, please refer to 'doc/build_system.txt' and the run script at
'os/run/demo.run'.
Using platforms other than Linux
@@ -112,7 +121,7 @@ steps are required:
# Uncomment the following line in 'x86_32/etc/build.conf'
! REPOSITORIES += $(GENODE_DIR)/repos/libports
# Build and execute the demo using Qemu
! make -C build/x86_32 KERNEL=okl4 run/demo
! make -C build/x86_32 KERNEL=okl4 BOARD=pc run/demo
The procedure works analogously for the other base platforms. You can, however,
reuse the already created build directory and skip its creation step if the

668
doc/news.txt
View File

@@ -4,6 +4,644 @@
===========
Sculpt OS release 21.03 | 2021-03-24
####################################
| Version 21.03 of the Sculpt operating system makes the system resilient
| against classes of driver failures, adds configurable real-time priorities,
| and introduces interfaces for screen capturing and user-event injection.
Sculpt OS 21.03 incorporates the many improvements of the latest two Genode
releases. Thanks to Genode's concept of
[https://genode.org/documentation/release-notes/21.02#Pluggable_network_device_drivers - pluggable device drivers],
the system has reached a new level of robustness against malfunctioning
drivers. For example, if the Intel graphics driver trips over an unsupported
external display, the driver gets automatically restarted while all graphical
applications keep running. Or as another example, should the overly complex
Wifi driver have a hick-up, it can be restarted with a simple mouse click
without harming the networking stacks running on top.
Even though Genode supports static-priority scheduling since more than a
decade, Sculpt did not make this feature available to end users so far. The
new version changes that. For each component, the user can now take a
deliberate decision about the hard scheduling priority, e.g., prioritizing
latency-critical multi-media applications over computational workloads or
virtual machines.
Speaking of workloads, to push the limits of what is possible with Sculpt OS,
the new version introduces additional interfaces that can be assigned to
components. First, it has become possible to redirect the interaction of a
component with the kernel through another component, thereby enabling features
like dynamic CPU-load balancing to be implemented as plain user-level
services. Second, there are new interfaces for capturing the screen and for
injecting input events. The latter interfaces pave the ground for virtual
keyboards, screen-sharing application, or remote administration scenarios.
Under the hood, there are plenty of improvements that make the life of
Sculpt users better. The keyboard layout can now be picked from a menu.
The Chromium-based Falkon web browser runs circles around the previous
version. Menu items and file lists appear nicely sorted. Terminal windows
immediately respond to global font-size changes. On modern Intel machines,
Sculpt leverages Intel Hardware P-states (HWP) for power and thermal
management now. You can find an illustrated tour of these and more changes in
a dedicated
[https://genodians.org/nfeske/2021-03-24-sculpt-os - article at Genodians.org].
The updated [https://genode.org/documentation/articles/sculpt-21-03 - manual]
goes into detail about the use of the new system.
The ready-to-use system image for version 21.03 is available at the
[https://genode.org/download/sculpt - Sculpt download page].
Genode OS Framework release 21.02 | 2021-02-25
##############################################
| The highlights of version 21.02 are the addition of VirtualBox 6,
| mobile-data connectivity via LTE, pluggable network drivers, initial
| support for the Pine-A64-LTS board, and revived work on RISC-V.
Many topics of the current release draw a connection to our overarching goal
to use Genode on the Pinephone by the end of the year. Besides the obvious
steps of enabling the hardware - starting with the Pine-A64-LTS board - the
release introduces mobile-data connectivity as a Genode feature, and changes
the network-driver architecture in anticipation of dynamic power-management
schemes.
For PC hardware, the flagship feature of version 21.02 is the addition of
VirtualBox 6, giving us the prospect to eventually replace the aging port of
VirtualBox 5. Speaking of VirtualBox, the release comes with profound
improvements of the USB-device pass-through abilities, most importantly
covering audio headsets.
Besides these prominent features, the new version comes with many further
improvements. Just to name a few, virtual machines on ARM have become
able to provide VirtIO-block devices to guests, named pipes can now
be used to connect components, Genode's RISC-V support received an
update to ISA spec 1.10, and OpenSSL has been bumped to version 1.1.1.
For the full story, please refer to the
[https:/documentation/release-notes/21.02 - release documentation of version 21.02...]
Road Map for 2021 | 2021-01-15
##############################
| In 2021, we plan to bring Genode to the Pinephone, advance the framework's
| GPU support, and focus on development workflows.
During the annual road-map discussion on Genode's public
[https://genode.org/community/mailing-lists - mailing list],
the following hot topics for this year emerged.
First and most inspiring for many Genode developers, we aspire to have
Genode running on the Pinephone with basic feature-phone functionality by the
end of the year. Since this will involve substantial device-driver-related
developments, the team will take this line of work as an opportunity to
advance the tooling and workflows for carrying out such tasks. This, in turn,
will hopefully ease the on-boarding of new driver developers in the future.
Closely related to the Pinephone scenario, the project will make optimizations
a top priority this year. The opportunities are plenty, ranging from
micro-optimizations, over API refinements, to architectural changes if
needed.
Another recurring topic is the request for GPU support, which is required
by many modern workloads such as video conferencing or streaming on mobile
device. Therefore, we will revamp our past developments of GPU multiplexing
on Intel hardware while also starting the investigation of GPUs on ARM-based
devices.
More information about our review of the past year, this year's focus, and a
rough schedule are presented at our official
[https:/about/road-map - road-map page].
Genode OS Framework release 20.11 | 2020-11-27
##############################################
| Genode 20.11 brings Sculpt OS to 64-bit ARM hardware, introduces dynamic
| CPU-load balancing, and enables multicore virtualization on ARM. Driver-wise,
| the release improves audio on PC hardware, and adds VirtIO networking support.
ARM 64-bit has been a recurring theme of the Genode releases this year and the
just released version 20.11 is no exception. We are proud to announce that our
Genode-based custom general-purpose OS called Sculpt has come to life on
64-bit ARM hardware, namely the NXP i.MX8 EVK board. This is the result of
intensive work on the framework's driver architecture for ARM and several
SoC-specific device drivers. Closely related to this line of work is the new
ability to run multicore virtual machines on ARM.
Another highlight of version 20.11 is a new CPU balancing mechanism, which
automates the dynamic assignment of threads to CPU cores for complex
workloads. With traditional operating systems, such policies are normally part
of the OS kernel. Thanks to Genode's component architecture, we are able to
implement such potentially complex policies in the form of an optional
component, which offers ultimate flexibility while keeping the kernel
untainted by complex heuristics.
Further topics of the current release are improved power management and audio
support on PC hardware, a new OSS API emulation that allows for the
reuse of popular audio applications on Genode, and new support for VirtIO
networking. The full picture is given by the
[https:/documentation/release-notes/20.11 - release documentation of version 20.11...]
Sculpt OS release 20.08 | 2020-09-17
####################################
| Version 20.08 of the Sculpt operating system refines the user experience
| and becomes able to host the Chromium-based Falkon web browser.
The new version of Sculpt OS is based on the latest Genode release
[https://genode.org/documentation/release-notes/20.08 - 20.08].
In particular, it incorporates the
[https://genode.org/documentation/release-notes/20.08#The_GUI_stack__restacked - redesigned GUI stack]
to the benefit of quicker boot times, improved interactive responsiveness, and
better pixel output quality.
It also removes the last traces of the
[https://genode.org/documentation/release-notes/20.05#Retired_Noux_runtime_environment - noux runtime].
Fortunately, these massive under-the-hood changes do not disrupt the
user-visible surface of Sculpt. Most users will feel right at home.
Upon closer inspection, there are couple of new features to appreciate. The
CPU-affinity of each component can now be restricted interactively by the user,
components can be easily restarted via a click on a button, font-size changes
have an immediate effect now, and the VESA driver (used when running Sculpt in
a virtual machine) can dynamically change the screen resolution.
Thanks to our continuous efforts of strengthening of the base system, Sculpt
OS has become able to host a first version of the Chromium-based Falkon web
browser from an installable package. Even though this version is still rough
around the edges and unoptimized, it already enables Sculpt users to casually
browse the modern web without the need for a virtual machine.
The updated [https://genode.org/documentation/articles/sculpt-20-08 - manual]
goes into detail about the use of the new system.
The ready-to-use system image for version 20.08 is available at the
[https://genode.org/download/sculpt - Sculpt download page].
Genode OS Framework release 20.08 | 2020-08-28
##############################################
| With Genode 20.08, the low-level GUI stack underwent a profound redesign,
| the Chromium web engine comes to life, the i.MX8 support covers clock and
| power management, and the CBE block encrypter becomes highly modular.
The most stunning feature of Genode 20.08 is most certainly the new ability
to host the Chromium web engine as native component in the form of the Falkon
web browser. This long-time project involved overcoming countless road
blocks along the way to the great benefit of Genode's users at large: building
bridges between 3rd-party build systems and Genode, covering seemingly obscure
corner cases of POSIX, solving instruction-cache invalidation issues on ARM -
just to name a few.
Under the hood, we took the release cycle as opportunity to tackle a major
surgery of the low-level GUI stack that we planned for more than two years.
The architectural change lays the groundwork for swapping out graphics and
input drivers on the fly without reboot. It also paves the ground for features
like screen capturing and remote desktop scenarios in a privacy-protecting
way.
On our mission of bringing the driver support for the 64-bit ARM-based i.MX8
SoC on par with our driver coverage on Intel PCs, the release introduces a
platform driver specifically for this SoC that covers clock and power
management. One step closer to using Sculpt OS on the MNT Reform laptop.
Furthermore, Genode's custom block encrypter called CBE received continuous
development. In particular, the cryptographic algorithm and trust anchor
have become pluggable modules. This will allow for tailoring the CBE to
custom products - like hardware trust anchors - without changing its
implementation.
These and many more improvements are covered in detail in the
[https:/documentation/release-notes/20.08 - release documentation of version 20.08...]
Genode OS Framework release 20.05 | 2020-05-28
##############################################
| Version 20.05 wraps up the consolidation of Noux with the C runtime,
| advances the device-driver infrastructure and virtualization support on ARM,
| reaches feature-completion of our block encryptor, and leverages seccomp for
| sandboxing components on top of Linux.
It has become almost a tradition to dedicate the spring release to topics
under the hood of the Genode OS framework, taking the time for careful
consolidation, and architectural improvements.
On the latter account, the Linux version gained an architectural revamp of its
inter-component communication model, leveraging the combination of the
seccomp and epoll kernel mechanisms to enable Genode's capability-based
security concept on this kernel. The new version applies strict sandboxing
to each component individually. In particular, the host file system is
shielded from Genode components and Genode's least-privilege access-control
scheme comes into effect.
As the second prominent architectural topic, the release features new
device-driver infrastructure for the ARM universe along with a unified
version of the formerly distinct virtual machine monitors for ARMv7 and
ARMv8. These are important steps to bring Genode on ARM to parity with the
x86 version.
Further highlights of the release are the first feature-complete version
of our custom block-encryption component, the improved management of CPU
affinities on x86, and new tracing utilities. Last but not least, the
release is accompanied with an updated version of the Genode Foundations
book, reflecting the changes of the framework since one year ago.
Read the details of the new release in the
[https:/documentation/release-notes/20.05 - release documentation of version 20.05...]
Sculpt OS release 20.02 | 2020-03-10
####################################
| Version 20.02 of the Sculpt operating system revisits the administrative
| user interface for a more intuitive and logical user experience.
With the release of Sculpt version 20.02, we follow our
[https://genode.org/about/road-map - roadmap's] mission to make Sculpt OS
easier to approach. In particular, we
[https://genodians.org/nfeske/2020-01-06-pending-sculpt-ui - identified] the
reliance on a command-line interface as a potential barrier of entry. As
Sculpt OS is not a Unix-like system, it should not require any Unix know-how
from the user. To relieve users from this burden, Sculpt 20.02 introduces
a custom graphical file browser and editor that can be used for interactively
inspecting and tweaking the state of the system. The traditional command-line
interface is still present as a fallback for advanced tasks though.
The updated [https://genode.org/documentation/articles/sculpt-20-02 - manual]
goes into detail about the use of the new system.
Thanks to the work of seasoned Genode developers, many software packages are
already available for the new version. These include virtual machine monitors
like VirtualBox, performance-monitoring tools, GUI components, Genode's custom
Unix runtime, and several applications and games. In particular, the software
depots offered by alex-ab, cnuke, skalk are worth exploring.
The ready-to-use system image for version 20.02 can be obtained from the
[https://genode.org/download/sculpt - Sculpt download page] and is
accompanied by matching
[https://genode.org/documentation/articles/sculpt-20-02 - documentation].
Genode OS Framework release 20.02 | 2020-02-28
##############################################
| With version 20.02, Genode makes Sculpt OS fit for running on i.MX 64-bit
| ARM hardware, optimizes the performance throughout the entire software stack,
| and takes the next evolutionary step of the user-facing side of Sculpt OS.
Without any doubt, Sculpt OS has been the driving motivation behind most
working topics featured by the new release. One particularly exciting line
of work is the enabling of Sculpt on i.MX-based 64-bit ARM hardware, which
touched the framework on all levels, from the boot loader, over the kernel,
device drivers, libraries, system management, up to the application level.
The work goes as far as supporting Sculpt OS as a hypervisor platform for
hosting Linux in a virtual machine.
As a second Sculpt-related development, we strive to make the user-visible
side of the operating system better approachable and more logical. With this
background, the current release comes with a profound redesign of the
administrative user interface of Sculpt OS. An updated downloadable system
image will follow soon.
Also related to Sculpt are an updated audio driver based on OpenBSD 6.6,
the support of virtual desktops, and performance optimization of the
Seoul virtual machine monitor on x86 hardware.
Regarding the framework API, the release introduces a new library for
building multi-component applications. It aims to bring the benefits of
Genode's unique security architecture from the operating-system level to the
application level.
These topics are only the tip of the iceberg. For the complete picture,
please consult the
[https:/documentation/release-notes/20.02 - release documentation of version 20.02...]
Road Map for 2020 | 2020-01-20
##############################
| In 2020, we will be concerned about dwarfing the barrier of entry into
| the Genode world.
Following the last year's leitmotif of "bridging worlds", we turn our
attention to the removal of the hurdles faced by aspiring developers and
users. During the annual road-map
[https://lists.genode.org/pipermail/users/2019-December/006987.html - discussion]
on our mailing list, we identified four tangible approaches towards that
goal. First, making Sculpt OS more user friendly. Second, reinforcing trust in
Genode by fostering the framework's high quality. Third, making the tooling
around Genode a joy to use. And finally, the illustration of Genode's
versatility in the form practical use cases.
Besides this overall theme, we plan to continue our commitment to the
NXP i.MX SoC family, revisit Genode's low-latency audio support, and
extend the cultivation of Ada/SPARK within (and on top of) Genode.
More background information about the new road map and a rough schedule are
presented at our official [https:/about/road-map - road-map page].
Genode OS Framework release 19.11 | 2019-11-28
##############################################
| Following this year's theme of "bridging worlds", Genode 19.11 adds the
| ability to use popular build tools like CMake for application development,
| introduces a new virtual-machine monitor for 64-bit ARM, and enhances
| POSIX compatibility. As another highlight, it features the first version
| of our custom block-device encrypter.
Block-device encryption is a feature often requested by users of our Sculpt OS.
Until now, we deliberately left this topic unaddressed because we felt that a
profound answer was beyond our expertise. However, during the past year, we
dived deep into it. The result is the prototype for a new block encrypter that
encrypts data but also protects integrity and freshness. For us, the
implementation of the encrypter is especially intriguing because - with about
7000 lines of code - it is Genode's first non-trivial component written in the
[https://en.wikipedia.org/wiki/SPARK_(programming_language) - SPARK]
programming language.
The second major addition is a new virtual machine monitor (VMM) for 64-bit
ARM platforms such as the NXP i.MX8. It leverages the
[https://genode.org/documentation/articles/arm_virtualization - proof of concept]
we developed in 2015 for ARMv7, which we pursued as a technology exploration.
In contrast, our aspiration with the new VMM is a product-quality solution.
In our [https://genode.org/about/road-map - road map] for 2019, we stated
the "bridging of worlds" as our overall theme for this year. On that account,
the current release moves the project forward on two levels. First, by
successively increasing the scope of POSIX compatibility, we reduce the
friction when porting existing application software to Genode. We managed
to bridge several gaps in our POSIX support that we considered as impossible
to cover some years ago. In particular, we identified ways to emulate certain
POSIX signals, ioctl calls, and fork/execve semantics. This way, popular
software such as bash, coreutils, or Vim can now be executed as regular
Genode components with no additional runtime environment (like Noux or a VMM)
required.
At a higher level, the current release introduces new tooling especially
geared at the development and porting of application software. Compared to
Genode's regular development tools, which were designed for whole-system
development, the new tool called Goa relieves the developer from the
complexity of Genode's custom build system and instead promotes the use of
popular commodity solutions like CMake.
These and more topics are described in the
[https:/documentation/release-notes/19.11 - release documentation of version 19.11...]
Genode OS Framework release 19.08 | 2019-08-28
##############################################
| Genode 19.08 puts emphasis on practical concerns ranging from
| keyboard layouts, over system-time management, to remote system
| administration. It also continues our commitment to the 64-bit ARM
| i.MX8 SoC, comes with Qt5 version 5.13, and improves POSIX compatibility.
The summer release of Genode addresses a variety of topics when using Genode
and Sculpt OS in practice. The confrontation with the real world prompted us
to develop new concepts for managing system time, keyboards layouts, and
copy-and-paste. For using Sculpt OS on the road, a new application VM for
accessing captive portals smoothes the experience of connecting to public WiFi
networks.
Besides the practical focus, the new release continues our commitment to the
64-bit ARM i.MX8 platform through new kernel support, device drivers, and test
coverage. Further topics include SMBIOS support for commodity PC hardware, a
new tracing tool, enhanced POSIX compatibility, and a major update of Qt5 to
version 5.13.
The complete picture is presented in the
[https:/documentation/release-notes/19.08 - release documentation of version 19.08...]
Sculpt OS release 19.07 | 2019-07-09
####################################
| Version 19.07 of the Sculpt operating system improves overall performance
| and introduces copy and paste between terminals, virtual machines, and
| graphical applications.
The most prominent user-visible feature of Sculpt OS 19.07 is the ability
of copy and paste text between terminals, graphical applications, and
virtual machines. Our unique take on this feature is described in
a [https://genodians.org/nfeske/2019-07-03-copy-paste - dedicated article].
Under the hood, Sculpt 19.07 benefits from the massive infrastructure
improvements that came with
[https://genode.org/documentation/release-notes/19.05 - Genode 19.05],
yielding a smoother user experience compared to earlier versions.
The new release can be obtained from the
[https://genode.org/download/sculpt - Sculpt download page] and is
accompanied by updated
[https://genode.org/documentation/articles/sculpt-19-07 - documentation].
Genode OS Framework release 19.05 | 2019-05-29
##############################################
| The highlights of version 19.05 are a new kernel-agnostic virtualization
| interface, initial support for the 64-bit ARM architecture, the use of
| C++17 by default, a new tool chain based on GCC 8.3, updated C and SPARK
| runtimes, and the consolidation of build directories across boards.
We dedicated the release cycle of Genode 19.05 to platform topics at various
levels. The flagship feature is certainly the introduction of our
kernel-agnostic virtualization interface. It has been in the works for more
than a half year and gives us the prospect of running virtual machine monitors
like Seoul and VirtualBox seamlessly across Genode's supported kernels.
The second major theme is the extension of Genode's CPU-architecture support
to 64-bit ARM (AARCH64). This step motivated the update of many parts of the
framework's fundamental infrastructure, ranging from the tool chain (updated
to GCC 8.3), over the C runtime (updated to FreeBSD 12 libc), to the dynamic
linker. The new tool chain, in turn, paved the ground for enabling C++17 by
default.
With the diversity of kernels, CPU architectures, and boards growing, we are
constantly striving to remove friction and redundancies between Genode's
underlying platforms. The current release eventually consolidates the build
directories not only across kernels but also across all boards of a given
CPU architecture. This vastly increases the velocity of Genode-based system
scenarios when targeting multiple boards or emulators at the same time.
Further details about these and many more improvements are given in the
[https:/documentation/release-notes/19.05 - release documentation of version 19.05...]
Sculpt as a Community Experience | 2019-03-19
#############################################
| The fourth stage of Sculpt OS introduces a new federated software
| provisioning model while giving the user full control over the
| component deployment via a novel graphical user interface.
With Sculpt CE, we enter the final stage of the evolution of Sculpt OS as
envisioned roughly one year ago. Initially geared towards die-hard enthusiasts
only, each revision became more and more user friendly. The previous version
Sculpt VC already offered a glimpse of Sculpt's unique user interface in the
form of an interactive component graph.
The just released Sculpt OS "as a community experience" (CE) combines this
tangible notion of component compositions with a completely federated software
provisioning model that cuts out middlemen like an app store or a
distribution. With Sculpt CE, components can be offered by a federation of
independent software providers selectable by the user. The software
installation is sandboxed and protected via digital signatures. The
integration of components with the rest of the system is completely under
control by the user. With the principle of least privilege at the heart
of Sculpt's architecture, you - the user - can fearlessly install and run
software without the need to ultimately trust the software providers.
Sculpt CE is intended to work in tandem with the new community blog
[https://genodians.org - Genodians.org] where developers and users exchange
experiences and announce new software. The best way to watch how the
Sculpt story continues is the RSS feed of Genodians.org.
To dive into the new world of Sculpt CE,
[https://genode.org/download/sculpt - download Sculpt OS...]
Genodians.org | 2019-03-08
##########################
| Genodians.org is the new place to be for getting the latest news and stories
| around Genode. It is a federated blog by and for developers and users
| alike.
With [https://genodians.org - Genodians.org], the Genode community has gained
a new place for exchanging ideas, announcing current developments, giving
tutorials, and sharing experience stories. In contrast to the formal character
of Genode's regular release notes, the articles at Genodians.org are raw and
personal, authored by individuals with no editorial process. For feedback
about the articles, readers are invited to the new
[https://reddit.com/r/genode - /r/genode] subreddit.
As written in the
[https://genodians.org/nfeske/2019-01-07-welcome - initial posting],
Genode users and developers are warmly invited to join the authors at
Genodians.org!
As a side note, the blogging platform is based on the Genode OS framework and
is of course open source
([https://github.com/genodelabs/genodians.org - GitHub repository]).
[https://genodians.org - Visit Genodians.org...]
Genode OS Framework release 19.02 | 2019-02-28
##############################################
| Version 19.02 enhances Sculpt OS with a federated software provisioning
| model, showcases the use of Java for an IoT network appliance and the
| creation of a component-based web service, improves the runtime support
| for Ada and SPARK, and adds board support for i.MX6 Quad Sabrelite and
| Nitrogen6 SoloX.
Our first release of 2019 pays tribute to this year's
[https://genode.org/about/road-map - road map] topic
of making Genode relevant and attractive for a broader community.
First, it enhances Sculpt OS with an easy-to-use way to discover, install,
and integrate software originating from different providers into a running
Sculpt system. Conversely, software providers get a distribution channel
directly to the user, secured by cryptographic signatures. Unlike commodity
OSes that rely on app stores or distributions, there is no middleman between
software providers and users in Sculpt OS.
Second, it makes the world's most popular programming language - Java -
available. Our port of OpenJDK facilitates just-in-time compilation on both
32-bit ARM and 64-bit x86 architectures. The use of Java within a Genode
system is nicely showcased by an exemplary IoT network appliance.
Third, to foster a strong sense of community, the release introduces a
Genode-based federated blogging platform, which enables users and
developers alike to share ideas, practical tips and tricks, and announcements.
[https://genodians.org - Genodians.org] is open for everyone to participate.
Other highlights of Genode 19.02 are the improved runtimes for the
Ada/SPARK and OCaml programming languages, and the added support for the i.MX6
Quad Sabrelite and Nitrogen6 SoloX boards.
For more details, please refer to the
[https:/documentation/release-notes/19.02 - release documentation of version 19.02...]
Road Map for 2019 | 2019-01-15
##############################
| In 2019, we will focus on practical use cases, on interoperability, and
| on harmonizing Genode with existing applications and programming languages.
The past Year of Sculpt was dedicated to bringing Genode to the desktop on
commodity PC hardware. Now is a good time to focus on making the Sculpt OS
relevant and appealing for a broader community. The road map for 2019 features
three major ambitions towards that goal. First, making Genode easier
approachable and usable by presenting practical use cases while fostering a
stronger sense of community among users and developers. Second, simplifying
the use of existing applications and programming languages in Genode-based
systems. And third, improving the interoperability of Genode with existing
protocols and systems.
The complete story behind the new road map is presented at the
[https:/about/road-map - road-map page].
Genode OS Framework release 18.11 | 2018-11-29
##############################################
| Genode 18.11 is focused on improving quality assurance on various fronts,
| including static code analysis, on-target test orchestration, and
| code-coverage measurements. Furthermore, it introduces support for Mirage-OS
| unikernels, a new health-monitoring mechanism, a Genode SDK, an SSH server,
| and a new window layouter.
On our road map for 2018, we identified software quality and resilience
as one of the major topics for this year. With the current release, we fulfil
this promise on various levels, ranging from static code analysis, over
the gathering of test-coverage metrics, a new Genode-based test-automation
framework, over to the health monitoring of components at runtime.
The second theme of the current release is the use of Genode for network
appliances and server applications. On that account, the new ability of
hosting Mirage-OS unikernels directly on top of Genode as well as a new SSH
server component clear the way to entirely new application areas.
Further highlights of the current release are the enhanced flexibility of the
GUI stack of Sculpt OS, the increased network performance on Xilinx Zynq, the
initial version of a Genode SDK, performance improvements of the base-hw
kernel on NXP i.MX platforms, and the updated language support for Ada and
Java.
These and many more topics of the new version are covered by the
[https:/documentation/release-notes/18.11 - release documentation of version 18.11...]
Dual licensing of 3rd-party Genode components | 2018-11-16
##########################################################
| To nurture a sustainable ecosystem around the Genode OS framework, we
| introduce a new approach for conducting dual-licensing businesses enabled
| by Genode.
Since founded ten years ago, Genode Labs pursues the Genode project based on a
dual-licensing business model, which allows us to fund the development of
Genode as an independent team. The licensing business is enabled by the
combination of the AGPLv3 as a strong copyleft license with the library-like
nature of Genode. Until today, this model is applicable to our framework but
impractical for 3rd-party component developers. To foster a sustainable
ecosystem around Genode, we wish to enable others to pursue a similar business
model while maintaining the spirit of open collaboration and free software.
We eventually crafted a new license called "Genode Component Public License"
(Genode CPL) specifically for components developed by 3rd parties, outside of
Genode Labs. The article
[https://genode.org/documentation/articles/component_public_license - Dual licensing of 3rd-party Genode components]
provides the rationale, license text, and FAQ of this software license.
Sculpt with Visual Composition | 2018-09-21
###########################################
@@ -689,7 +1327,7 @@ The story behind Genode's TrustZone demo on the USB Armory | 2015-12-08
| Our latest article provides a look behind the scenes of the
| development of Genode's support for the USB Armory platform.
The [http://inversepath.com/usbarmory - USB Armory] is a computer in the form
The [https://inversepath.com/usbarmory - USB Armory] is a computer in the form
of a USB stick. It normally runs Linux. But thanks to the ARM TrustZone
capabilities of the device, it is possible to run Genode behind the back of
Linux. This is useful for shielding sensitive information like cryptographic
@@ -703,7 +1341,7 @@ was splitting the hardware platform into two worlds while maintaining the
full functionality of Linux. The article goes on to explain the interplay
between the secure world (Genode) and the normal world (Linux). Furthermore,
it provides all the pointers needed to reproduce the scenario.
[http:/documentation/articles/usb_armory - Read the article...]
[https://genode.org/documentation/articles/usb_armory - Read the article...]
Genode OS Framework release 15.11 | 2015-11-30
@@ -1882,7 +2520,7 @@ for our dual-licensing business model. If we made Genode available under the
BSD license, there would be not point in pursuing this model. However, hiding
the development process from the public is not only poor-spirited but it
creates an artificial barrier for people who want to participate. The book
"Producing Open Source Software" (http://producingoss.com) by Karl Fogel was an
"Producing Open Source Software" (https://producingoss.com) by Karl Fogel was an
eye opener to us.
Regarding the efficiency of collaboration, I have to admit that the statement
@@ -1980,7 +2618,7 @@ new support of L4Android on Genode, Android can be used on Genode on the
Fiasco.OC kernel on the IA32 architecture. For those of you who are eager to
experiment with L4Android on Genode, please find further information at the
top-level
[http://genode.svn.sourceforge.net/viewvc/genode/trunk/ports-foc/README - README]
[https://genode.svn.sourceforge.net/viewvc/genode/trunk/ports-foc/README - README]
file of the 'ports-foc' repository and share your results with us at the
[https:/community/mailing-lists - Genode mailing list].
@@ -2095,17 +2733,17 @@ concepts live. The talk was recorded at the
Amsterdam. Thanks to Bas the Lange for publishing the material.
: <object height="385" width="480">
: <param name="movie" value="http://www.youtube.com/v/Z1IMV3FJO7Q" />
: <param name="movie" value="https://www.youtube.com/v/Z1IMV3FJO7Q" />
: <param name="allowFullScreen" value="true" />
: <param name="allowscriptaccess" value="always" />
: <embed width="480" height="385" allowfullscreen="true"
: allowscriptaccess="always"
: type="application/x-shockwave-flash"
: src="http://www.youtube.com/v/Z1IMV3FJO7Q">
: src="https://www.youtube.com/v/Z1IMV3FJO7Q">
: </embed>
: </object>
[http://www.youtube.com/watch?v=Z1IMV3FJO7Q - Visit the YouTube page...]
[https://www.youtube.com/watch?v=Z1IMV3FJO7Q - Visit the YouTube page...]
Genode OS Framework release 11.02 | 2011-02-24
@@ -2162,17 +2800,17 @@ prepared the following screencast with a guided walk-through. Enjoy!
: <object height="385" width="480">
: <param name="movie"
: value="http://www.youtube.com/v/CJdWOmajo_8?fs=1&amp;hl=en_US" />
: value="https://www.youtube.com/v/CJdWOmajo_8?fs=1&amp;hl=en_US" />
: <param name="allowFullScreen" value="true" />
: <param name="allowscriptaccess" value="always" />
: <embed width="480" height="385"
: src="http://www.youtube.com/v/CJdWOmajo_8?fs=1&amp;hl=en_US"
: src="https://www.youtube.com/v/CJdWOmajo_8?fs=1&amp;hl=en_US"
: type="application/x-shockwave-flash" allowscriptaccess="always"
: allowfullscreen="true">
: </embed>
: </object>
[http://www.youtube.com/watch?v=CJdWOmajo_8 - Visit the YouTube page...]
[https://www.youtube.com/watch?v=CJdWOmajo_8 - Visit the YouTube page...]
[https:/download/live-cds - Download the real thing...]
@@ -2428,7 +3066,7 @@ applications, most prominently, it serves as the foundation of the KDE project.
Since the release 9.05, the official distribution of Genode supports Qt4 as a
regular feature. The document "Portierung von Qt auf Genode" _(german)_
describes the challenging endeavor of porting this high-complexity C++
framework to Genode. Major problems to overcome had been the missing C libary
framework to Genode. Major problems to overcome had been the missing C library
(at the time when the project started), the integration of the Qt4 project
files with Genode's build system, the adaption of Qt4 to the basic primitives
provided by Genode, and the integration of Qt4 with Genode's GUI. In addition
@@ -2571,13 +3209,13 @@ for graphics, input devices, and sound. It is often used as back end for games,
emulators, and media players. Also the Linux version of Genode relies on the
hardware abstractions provided by libSDL.
As [http://sourceforge.net/mailarchive/message.php?msg_id=21406424 - announced on the Genode mailing list],
As [https://sourceforge.net/mailarchive/message.php?msg_id=21406424 - announced on the Genode mailing list],
libSDL has been ported to Genode. At the current stage, the port supports
the video subsystem and the input handling for mouse and keyboard. With
libSDL now becoming available for Genode, it becomes much easier to make the
wealth of libSDL-based applications available on our platform.
[http://sourceforge.net/mailarchive/message.php?msg_id=21406424 - Read the announcement...]
[https://sourceforge.net/mailarchive/message.php?msg_id=21406424 - Read the announcement...]
Genode on the L4ka::Pistachio kernel | 2008-12-18
@@ -2729,7 +3367,7 @@ Project website launched | 2008-07-29
| Genode OS framework is scheduled for the 6th of August.
Today, we proudly launched the website of the Genode project
[https://www.genode.org - https://www.genode.org]. This website is the central
[https://genode.org - https://genode.org]. This website is the central
resource for people using or developing the Genode OS framework. It covers
the latest news about our progress, architectural and technical documentation,
a community-maintained wiki, mailing lists, information on accessing the
@@ -2738,7 +3376,7 @@ attention of people who want bring forward the project together with us.
We have scheduled the first official release of the Genode OS framework for the
6th August. Until then, we invite you to test-drive the beta-version of the
framework as provided at the [https://www.genode.org/download - download].
framework as provided at the [https://genode.org/download - download].
Genode Labs founded | 2008-07-17

830
doc/release_notes-08-11.txt
View File

@@ -1,830 +0,0 @@
==============================================
Release notes for the Genode OS Framework 8.11
==============================================
Genode Labs
Summary
#######
This document presents the new features and major changes introduced
in version 8.11 of the Genode OS Framework. It is geared towards
people interested in closely following the progress of the Genode
project and to developers who want to adopt their software to our
mainline development. The document aggregates important fragments
of the updated documentation such that you won't need to scan existing
documents for the new bits. Furthermore, it attempts to provide our
rationale behind the taken design decisions.
The general theme for the release 8.11 is enabling the use of the
Genode OS framework for real-world applications. Because we regard
the presence of device drivers and a way to reuse existing library
code as fundamental prerequisites for achieving this goal, the major
new additions are an API for device drivers written in C, an API for
handling asynchronous notifications, and a C runtime. Other noteworthy
improvements are the typification of capabilities at the C++-language
level, a way for receiving and handling application faults, the
introduction of managed dataspaces, and a new API for scheduling
timed events.
Base framework
##############
This section documents the new features and changes affecting the
'base' repository, in particular the base API.
New features
============
Connection handling
~~~~~~~~~~~~~~~~~~~
The interaction of a client with a server involves the definition of
session-construction arguments, the request of the session creation via
its parent, the initialization of the matching RPC-client stub code
with the received session capability, the actual use of the session
interface, and the closure of the session. A typical procedure of
using a service looks like this:
!#include <rom_session/client.h>
!...
!
!/* construct session-argument string and create session */
!char *args = "filename=config, ram_quota=4K");
!Capability session_cap = env()->parent()->session("ROM", args);
!
!/* initialize RPC stub code */
!Rom_session_client rsc(session_cap);
!
!/* invoke remote procedures, 'dataspace' is a RPC function */
!Capability ds_csp = rsc.dataspace();
!...
!
!/* call parent to close the session */
!env()->parent()->close(session_cap);
Even though this procedure does not seem to be overly complicated,
is has raised the following questions and criticism:
* The quota-donation argument is specific for each server. Most services
use client-donated RAM quota only for holding little meta data and,
thus, are happy with a donation of 4KB. Other services maintain larger
client-specific state and require higher RAM-quota donations. The
developer of a client has to be aware about the quota requirements for
each service used by his application.
* There exists no formalism for documenting session arguments.
* Because session arguments are passed to the 'session'-call as a plain
string, there are no syntax checks for the assembled string performed
at compile time. For example, a missing comma would go undetected until
a runtime test is performed.
* There are multiple lines of client code needed to open a session to
a service and the session capability must be maintained manually for
closing the session later on.
The new 'Connection' template provides a way to greatly simplify the
handling of session arguments, session creation, and destruction on the
client side. By implementing a service-specific connection class
inherited from 'Connection', session arguments become plain constructor
arguments, session functions can be called directly on the 'Connection'
object, and the session gets properly closed when destructing the
'Connection'. By convention, the 'Connection' class corresponding to a
service resides in a file called 'connection.h' in the directory of the
service's RPC interface. For each service, a corresponding 'Connection'
class becomes the natural place where session arguments and quota
donations are documented. With this new mechanism in place, the example
above becomes as simple as:
!#include <rom_session/connection.h>
!...
!
!/* create connection to the ROM service */
!Rom_connection rom("config");
!
!/* invoke remote procedure */
!Capability ds_csp = rom.dataspace();
[http://genode.org/documentation/api/base_index#Connecting_to_services - See the API documentation for the connection template...]
Typed capabilities
~~~~~~~~~~~~~~~~~~
A plain 'Capability' is an untyped reference to a remote object of any
type. For example, a capability can reference a thread object or a
session to a service. It is loosely similar to a C void pointer, for which
the programmer maintains the knowledge about which data type is actually
referenced. To facilitate the type-safe use of RPC interfaces at the C++
language level, we introduced a template for creating specialized
capability types ('Typed_capability' in 'base/typed_capability.h') and
the convention that each RPC interface declares a dedicated capability
type. Note that type-safety is not maintained across RPC interfaces. As
illustrated in Figure [layered_ipc], typification is done at the
object-framework level on the server side and via in the 'Connection'
classes at the client side.
[image layered_ipc]
From the application-developer's perspective, working with capabilities
has now become type-safe, making the produced code more readable and robust.
[http://genode.org/documentation/api/base_index#Capability_representation - See the updated API documentation for the capability representation...]
Fifo data structure
~~~~~~~~~~~~~~~~~~~
Because the 'List' data type inserts new list elements at the list head,
it cannot be used for implementing wait queues requiring first-in
first-out semantics. For such use cases, we introduced a dedicated
'Fifo' template. The main motivation for introducing 'Fifo' into the
base API is the new semaphore described below.
[http://genode.org/documentation/api/base_index#Structured_data_types - See the new API documentation for the fifo template...]
Semaphore
~~~~~~~~~
Alongside lock-based mutual exclusion of entering critical sections,
organizing threads in a producer-consumer relationship via a semaphore
is a common design pattern for thread synchronization. Prior versions
of Genode provided a preliminary semaphore implementation as part of
the 'os' repository. This implementation, however, supported only one
consumer thread (caller of the semaphore's 'down' function). We have
now enhanced our implementation to support multiple consumer threads
and added the semaphore to Genode's official base API. We have made
the wake-up policy in the presence of multiple consumers configurable
via a template argument. The default policy is first-in-first-out.
[http://genode.org/documentation/api/base_index#Synchronization - See the new API documentation for the semaphore...]
Thanks to Christian Prochaska for his valuable contributions to the new
semaphore design.
Asynchronous notifications
~~~~~~~~~~~~~~~~~~~~~~~~~~
Inter-process communication via remote procedure calls requires both
communication partners to operate in a synchronous fashion. The caller
of an RPC blocks as long as the RPC is not answered by the called
server. In order to receive the call, the server has to explicitly
wait for incoming messages. There are a number of situations where
synchronous communication is not suited.
For example, a GUI server wants to deliver a notification to one of its
clients about new input events being available. It does not want to
block on a RPC to one specific client because it has work to do for
other clients. Instead, the GUI server wants to deliver this
_notification_ with _fire-and-forget_ semantics and continue with
its operation immediately, regardless of whether the client received
the notification or not. The client, in turn, does not want to poll
for new input events at the GUI server but it wants to be _waken_up_
when something interesting happens. Another example is a block-device
driver that accepts many requests for read/write operations at once.
The operations may be processed out of order and may take a long time.
When having only synchronous communication available, the client and
the block device driver would have to employ one distinct thread for
each request, which is complicated and a waste of resources. Instead,
the block device driver just wants to acknowledge the completeness of
an operation _asynchronously_.
Because there are many more use cases for asynchronous inter-process
communication, we introduced a new signalling framework that complements
the existing synchronous RPC mode of communication with an interface for
issuing and receiving asynchronous notifications. It defines interfaces
for signal transmitters and signal receivers. A signal receiver can
receive signals from multiple sources, whereas the sources of incoming
signals are clearly distinguishable. One or multiple threads can either
poll or block for incoming signals. Each signal receiver is addressable
via a capability. The signal transmitter provides fire-and-forget
semantics for submitting signals to exactly one signal receiver. Signals
are communicated in a reliable fashion, which means that the exact number
of signals submitted to a signal transmitter is communicated to the
corresponding signal receiver. If notifications are generated at a higher
rate than as they can be processed at the receiver, the transmitter
counts the notifications and delivers the total amount with the next
signal transmission. This way, the total number of notifications gets
properly communicated to the receiver even if the receiver is not highly
responsive. Notifications do not carry any payload because this payload
would have to be queued at the transmitter.
[image signals]
Image [signals] illustrates the roles of signaller thread,
transmitter, receiver, and signal-handler thread.
[http://genode.org/documentation/api/base_index#Asynchronous_notifications - See the new API documentation for asynchronous notifications...]
The current generic implementation of the signalling API employs one
thread at each transmitter and one thread at each receiver. Because
the used threads are pretty heavy weight with regard to resource usage,
ports of Genode should replace this implementation with platform-
specific variants, for example by using inter-process semaphores or
native kernel support for signals.
Region-manager faults
~~~~~~~~~~~~~~~~~~~~~
In Genode, region-manager (RM) sessions are used to manage the
address-space layout for processes. A RM session is an address-space
layout that can be populated by attaching (portions of) dataspaces to
(regions of) the RM session. Normally, the RM session of a process is
first configured by the parent when decoding the process' ELF binary.
During the lifetime of the process, the process itself may attach
further dataspaces to its RM session to access the dataspace's content.
Core as the provider of the RM service uses this information for
resolving page faults raised by the process. In prior versions of
Genode, core ignored unresolvable page faults, printed a debug message
and halted the page-faulted thread. However, this condition may be of
interest, in particular to the process' parent for reacting on the
condition of a crashed child process. Therefore, we enhanced the RM
interface by a fault-handling mechanism. For each RM session, a fault
handler can be installed by registering a signal receiver capability.
If an unresolvable page fault occurs, core delivers a signal to the
registered fault handler. The fault handler, in turn, can request the
actual state of the RM session (page-fault address) and react upon
the fault. One possible reaction is attaching a new dataspace at the
fault address and thereby implicitly resolving the fault. If core
detects that a fault is resolved this way, it resumes the operation
of the faulted thread.
This mechanism works analogously to how page faults are handled by
CPUs, but on a more abstract level. A (n-level) page table corresponds
to a RM session, a page-table entry corresponds to a dataspace-
attachment, the RM-fault handler corresponds to a page-fault
exception handler, and the resolution of page-faults (RM fault)
follows the same basic scheme:
# Application accesses memory address with no valid page-table-entry
(RM fault)
# CPU generates page-fault exception (core delivers signal to fault
handler)
# Kernel reads exception-stack frame or special register to determine
fault address (RM-fault handler reads RM state)
# Kernel adds a valid page-table entry and returns from exception
(RM-fault handler attaches dataspace to RM session, core resumes
faulted thread)
The RM-fault mechanism is not only useful for detecting crashing child
processes but it enables a straight-forward implementation of growing
stacks and heap transparently for a child process. An example for
using RM-faults is provided at 'base/src/test/rm_fault'.
Note that this mechanism is only available on platforms on which core
resolves page faults. This is the case for kernels of the L4 family.
On Linux however, the Linux kernel resolves page faults and suspends
processes performing unresolvable memory accesses (segmentation fault).
Managed dataspaces (experimental)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The RM-fault mechanism clears the way for an exciting new feature
of Genode 8.11 called managed dataspaces. In prior versions of Genode,
each dataspace referred to a contiguous area of physical memory (or
memory-mapped I/O) obtained by one of core's RAM, ROM, or IO_MEM
services, hence we call them physical dataspaces. We have now added
a second type of dataspaces called managed dataspaces. In contrast
to a physical dataspace, a managed dataspace is backed by the content
described by an RM session. In fact, each RM session can be used as
dataspace and can thereby be attached to other RM sessions.
Combined with the RM fault mechanism described above, managed
dataspaces enable a new realm of applications such as dataspaces
entirely managed by user-level services, copy-on-write dataspaces,
non-contiguous large memory dataspaces that are immune to physical
memory fragmentation, process-local RM fault handlers (e.g., managing
the own thread-stack area as a sub-RM-session), and sparsely populated
dataspaces.
Current limitations
-------------------
Currently, managed dataspaces still have two major limitations. First,
this mechanism allows for creating cycles of RM sessions. Core must
detect such cycles during page-fault resolution. Although, a design for
an appropriate algorithm exists, cycle-detection is not yet implemented.
The missing cycle detection would enable a malicious process to force
core into an infinite loop. Second, RM faults are implemented using the
new signalling framework. With the current generic implementation, RM
sessions are far more resource-demanding than they should be. Once the
signalling framework is optimized for L4, RM sessions and thereby
managed dataspaces will become cheap. Until then, we do not recommend
to put this mechanism to heavy use.
Because of these current limitations, managed dataspaces are marked as
an experimental feature. When building Genode, experimental features are
disabled by default. To enable them, add a file called 'specs.conf'
with the following content to the 'etc/' subdirectory of your build
directory:
! SPECS += experimental
For an example of how to use the new mechanism to manage a part of a
process' own address space by itself, you may take a look at
'base/src/test/rm_nested'.
Changes
=======
Besides the addition of the new features described above, the following
parts of the base framework underwent changes worth describing.
Consistent use of typed capabilities and connection classes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We applied capability typification to all interfaces of Genode including
the base API and the interfaces defined in the 'os' repository. Figure
[base_cap_types] provides an overview about the capability types
provided by the base API.
[image base_cap_types]
Overview about the capability types provided by the base API
Furthermore, we have complemented all session interfaces with
appropriate 'Connection' classes taking service-specific session
arguments into account.
For session-interface classes, we introduced the convention to declare
the service name as part of the session-interface via a static member
function:
! static const char *service_name();
Allocator refinements
~~~~~~~~~~~~~~~~~~~~~
Throughout Genode, allocators are not only used for allocating memory
but also for managing address-space layouts and ranges of physical
resources such as I/O-port ranges or IRQ ranges. In these cases, the
address '0' may be a valid value. Consequently, this value cannot be
used to signal allocation errors as done in prior versions of Genode.
Furthermore, because managed dataspaces use the RM session interface to
define the dataspace layout, the address-'0' problem applies here as
well. We have now refined our allocator interfaces and the RM-session
interface to make them fit better for problems other than managing
virtual memory.
Misc changes
~~~~~~~~~~~~
We revised all interfaces to consistently use _exceptions_ to signal
error conditions rather than delivering error codes as return values.
This way, error codes become exception types that have a meaningful
name and, in contrast to global 'errno' definitions, an error exception
type can be defined local to the interface it applies to. Furthermore,
the use of exceptions allows for creating much cleaner looking interfaces.
Traditionally, we have provided our custom _printf_ implementation as C
symbol to make this function available from both C and C++ code. However,
we observed that we never called this function from C code and that the
'printf' symbol conflicts with the libc. Hence, we turned 'printf'
into a C++ symbol residing in the 'Genode' namespace.
Operating-system services and libraries
#######################################
This section documents the new features and changes affecting
the 'os' repository.
New Features
============
Device-driver framework for C device drivers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Genode's base API features everything needed to create user-level device
drivers. For example, the 'os' repository's PS/2 input driver and the
PCI bus driver are using Genode's C++ base API directly. However, most of
today's device drivers are written in C. To ease the reuse of existing
drivers on Genode, we have introduced a C API for device drivers into
Genode's 'os' repository. The API is called DDE kit (DDE is an acronym
for device-driver environment) and it is located at 'os/include/dde_kit'.
The DDE kit API is the result of long-year experiences with porting device
drivers from Linux and FreeBSD to custom OS environments. The following
references are the most significant contributions to the development of
the API.
;
Christian Helmuth created the initial version of the Linux device-driver
environment for L4. He describes his effort of reusing unmodified sound
drivers on the L4 platform in his thesis
[http://os.inf.tu-dresden.de/papers_ps/helmuth-diplom.pdf - Generische Portierung von Linux-Gerätetreibern auf die DROPS-Architektur].
;
Gerd Griessbach approached the problem of re-using Linux USB drivers
by following the DDE approach in his diploma thesis
[http://os.inf.tu-dresden.de/papers_ps/griessbach-diplom.pdf - USB for DROPS].
;
Marek Menzer adapted Linux DDE to Linux 2.6 and explored the DDE
approach for block-device drivers in his student research project
[http://os.inf.tu-dresden.de/papers_ps/menzer-beleg.pdf - Portierung des DROPS Device Driver Environment (DDE) für Linux 2.6 am Beispiel des IDE-Treibers ]
and his diploma thesis
[http://os.inf.tu-dresden.de/papers_ps/menzer-diplom.pdf - Entwicklung eines Blockgeräte-Frameworks für DROPS].
;
Thomas Friebel generalized the DDE approach and introduced the DDE kit
API to enable the re-use of device driver from other platforms than
Linux. In particular, he experimented with the block-device drivers of
FreeBSD in his diploma thesis
[http://os.inf.tu-dresden.de/papers_ps/friebel-diplom.pdf - Übertragung des Device-Driver-Environment-Ansatzes auf Subsysteme des BSD-Betriebssystemkerns].
;
Dirk Vogt successfully re-approached the port of USB device drivers
from the Linux kernel to L4 in his student research project
[http://os.inf.tu-dresden.de/papers_ps/beleg-vogt.pdf - USB for the L4 Environment].
The current incarnation of the DDE kit API provides the following
features:
* General infrastructure such as init calls, assertions, debug output
* Interrupt handling (attach, detach, disable, enable)
* Locks, semaphores
* Memory management (slabs, malloc)
* PCI access (find device, access device config space)
* Virtual page tables (translation between physical and virtual
addresses)
* Memory-mapped I/O, port I/O
* Multi-threading (create, exit, thread-local storage, sleep)
* Timers, jiffies
For Genode, we have created a complete reimplementation of the DDE kit
API from scratch by fully utilizing the existing Genode infrastructure
such as the available structured data types, core's I/O services,
the synchronization primitives, and the thread API.
[image dde_kit]
Figure [dde_kit] illustrates the role of DDE kit when re-using an
unmodified device driver taken from the Linux kernel. DDE kit translates
Genode's C++ base API to the DDE kit C API. The DDE kit API, in turn, is
used as back end by the Linux driver environment, which translates Linux
kernel interfaces to calls into DDE kit. With this translation in place,
an unmodified Linux device driver can be embedded into the Linux driver
environment. The device API is specific for a class of devices such as
NICs, block devices, or input devices. It can either be used directly as
a function interface by an application that is using the device driver
as a library, or it can be made accessible to external processes via an
RPC interface.
Limitations
-----------
The PCI sub system is not completely implemented, yet.
Alarm API providing a timed event scheduler
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The scheduling of timed events is a recurring pattern found in device
drivers, application frameworks such as Qt4 ('qeventdispatcher'), and
applications. Therefore, we have added a timed event scheduler to the
'os' repository. The new alarm API ('os/include/os/alarm.h') allows
for the scheduling of both one-shot alarms and periodic alarms.
Changes
=======
PS/2 input driver
~~~~~~~~~~~~~~~~~
The original PS/2 driver tried to switch the PS/2 keyboard to
scan-code set 2 and assumed that all modern keyboards support this
mode of operation. However, this assumption was wrong. We observed
that the legacy PS/2 support of some USB keyboards covers only the
emulated (xlate) scan-code set 1 mode. This is also case for the PS/2
emulation in VirtualBox. Therefore, we changed our PS/2 driver to
never touch the keyboard mode but to only detect the current mode
of operation. The driver has now to support both, scan-code set 1 and
scan-code set 2. This change comes along with a slightly more complex
state machine in the driver. Hence, we moved the state machine from
the IRQ handler to a distinct class and changed the control flow of
the driver to fetch only one value from the i8042 PS/2 controller
per received interrupt.
PCI bus driver
~~~~~~~~~~~~~~
Until now, Genode's PCI bus driver was only used for experimentation
purposes. With the forthcoming driver framework however, the PCI bus
driver will play a central role in the system. Therefore, we adapted
the interface of the PCI driver to these requirements. Specifically,
the scanning of the PCI bus can now be performed without constraining
the results by a specific vendor ID.
Nitpicker GUI server
~~~~~~~~~~~~~~~~~~~~
We improved the _output_latency_ of the Nitpicker GUI server by flushing
pixels eagerly and deferring the next periodically scheduled flush.
This change has a positive effect on the responsiveness of the GUI to
user input.
Misc changes
~~~~~~~~~~~~
Prior versions of the 'os' repository came with a custom 'os/include/base'
directory with interfaces extending the base API. To avoid confusion
between the 'base' repository and the 'os' repository, 'os'-local API
extensions are now located at 'os/include/os'. This way, the folder
prefix of include statements indicates well from which repository the
included header files comes from.
C runtime
#########
Most of existing libraries rely on the presence of a C library. For
making the reuse of this software on Genode possible, we have now
made a complete C library available for Genode. It comes as a separate
source-code repository called 'libc' and is based on the code of FreeBSD.
The original code is available at the official FreeBSD website.
:FreeBSD website:
[http://www.freebsd.org/developers/cvs.html]
Our libc port comprises the libraries 'gdtoa', 'gen', 'locale', 'stdio',
'stdlib', 'stdtime', 'string', and 'msun'. Currently, it supports the
x86 architecture. Support for other architectures is planned as future
addition. At the current stage, our back end is very basic and most of
its functions are dummy stubs. We used Christian Prochaska's forthcoming
Genode port of Qt4 as test case and successfully used the new libc as
foundation for building graphical Qt4 applications. We will further
extend the back end in correspondence to the growing feature set of the
Genode OS framework.
:Usage:
To use the libc in your application, just add 'libc' to the 'LIBS'
declaration in your build-description file. This declaration will make
the libc headers available for the include path of your target and link
the C library. When building, make sure that the 'libc' repository is
included in your build configuration ('etc/build.conf').
:Limitations:
The current version of the C library is not thread-safe. For most
string and math functions, this is not a problem (as these functions
do not modify global state) but be careful with using more complex
functions such as 'malloc' from multiple threads. Also, 'errno' may
become meaningless when calling libc functions from multiple threads.
We have left out the following files from the Genode port of the
FreeBSD libc: gdtoa 'strtodnrp.c' (gdtoa), 'getosreldate.c' (gen),
'strcoll.c', 'strxfrm.c', 'wcscoll.c', 'wcsxfrm.c' (string),
's_exp2l.c' ('msun').
The current back end is quite simplistic and it may help you to revisit
the current state of the implementation in the 'libc/src/lib/libc'
directory. If one of the functions in 'dummies.c' is called, you will
see the debug message:
! "<function-name> called, not yet implemented!"
However, some of the back-end function implemented in the other files
have dummy semantics but have to remain quiet because they are called
from low-level libc code.
Build infrastructure
####################
Build-directory creation tool
=============================
Because we think that each Genode developer benefits from knowing the
basics about the functioning of the build system, the manual creation of
build directories is described in Genode's getting-started document.
However, for regular developers, creating build directories becomes a
repetitive task. Hence, it should be automated. We have now added a
simple build-directory creation tool that creates pre-configured build
directories for some supported platforms. The tool is located at
'tool/builddir/create_builddir'. To print its usage information, just
execute the tool without arguments.
Improved linking of binary files
================================
For linking binary data, binary file have to be converted to object
files. Over the time, we have used different mechanisms for this
purpose. Originally, we used 'ld -r -b binary'. Unfortunately, these
linker options are not portable. Therefore, the mechanism was changed
to a 'hexdump' and 'sed' magic that generated a C array from binary data.
This solution however, is complicated and slow. Now, we have adopted
an idea of Ludwig Hähne to use the 'incbin' directive of the GNU
assembler, which is a very clean, flexible, and fast solution.
Lib-import mechanism
====================
Libraries often require specific include files to be available at the
default include search location. For example, users of a C library
expect 'stdio.h' to be available at the root of the include search
location. Placing the library's include files in the root of the
default search location would pollute the include name space for
all applications, regardless if they use the library or not. To
keep library-include files well separated from each other, we have
enhanced our build system by a new mechanism called lib-import.
For each library specified in the 'LIBS' declaration of a build
description file, the build system incorporates a corresponding
'import-<libname>.mk' file into the build process. Such as file
defines library-specific compiler options, in particular additional
include-search locations. The build system searches for lib-import
files in the 'lib/import/' subdirectories of all used repositories.
Using 'ar' for creating libraries
=================================
The previous versions of Genode relied on incremental linking ('ld -r')
for building libraries. This approach is convenient because the linker
resolves all cross-dependencies between libraries regardless of the
order of how libraries are specified at the linker's command line.
However, incremental linking prevents the linker from effectively
detecting dead code. In contrast, when linking '.a' files, the linker
detects unneeded object files. Traditionally, we have only linked our
own framework containing no dead code. This changed with the new 'libc'
support. When linking the 'libc', the presence of dead code becomes
the normal case rather than the exception. Consequently, our old
incremental-linking approach produced exceedingly large binaries
including all functions that come with the 'libc'. We have now adopted
the classic 'ar' mechanism for assembling libraries and use the linker's
'start-group' 'end-group' feature to resolve inter-library-dependencies.
This way, dead code gets eliminated at the granularity of object files.
In the future, we will possible look into the '-ffunction-sections' and
'-gc-sections' features of the GNU tool chain to further improve the
granularity to function level.
If your build-description files rely on custom rules referring to
'lib.o' files, these rules must be adapted to refer to 'lib.a' files
instead.
Misc changes
============
* Added sanity check for build-description files overriding 'INC_DIR'
instead of extending it.
* Restrict inclusion of dependency files to those that actually matter
when building libraries within 'var/libcache'. This change significantly
speeds up the build process in the presence of large libraries such as
Qt4 and libc.
* Added rule for building 'cpp' files analogously to the 'cc' rule.
Within Genode, we name all C++ implementation files with the 'cc'
suffix. However, Qt4 uses 'cpp' as file extension so we have to
support both.
* Build-description files do no longer need the declaration
'REQUIRES = genode'. Genode's include search locations are now
incorporated into the build process by default.
Applications
############
This section refers to the example applications contained in Genode's
'demo' repository.
We have enhanced the _Scout_widgets_ as used by the launchpad and the
Scout tutorial browser to perform all graphical output double-buffered,
which effectively eliminates drawing artifacts that could occur when
exposing intermediate drawing states via direct (unbuffered) output.
Furthermore, we have added a way to constrain the maximum size of
windows to perform pixel-buffer allocations on realistic window sizes.
Both launchpad and Scout can now start child applications. In Scout
this functionality is realized by special "execute" links. We have
generalized the underlying application logic for creating and
maintaining child processes between both applications and placed
the unification into a separate 'launchpad' library.
We have replaced the default document presented in Scout with an
_interactive_walk-through_guide_ explaining the basic features of Genode.
The document uses the new "execute" link facility to let the user start
a launchpad instance by clicking on a link.
Platform-specific changes
#########################
Genode used to define _fixed-width_integer_types_ in a file 'stdint.h'
placed in a directory corresponding to bit-width of the platform, for
example 'include/32bit/stdint.h'. When building for a 32bit platform,
the build system included the appropriate directory into the
include-search path and thereby made 'stdint.h' available at the root
of the include location. Unfortunately, this clashes with the 'stdint.h'
file that comes with the C library. To avoid conflict with libc header
files, we moved the definition of fixed-width integer types to
'32bit/base/fixed_stdint.h'.
For the L4/Fiasco version of Genode, there existed some x86-specific
header files that did not specifically depend on L4/Fiasco, for example
atomic operations. Because these files are not L4/Fiasco-specific and
may become handy for other platforms as well, we moved them to the
generic 'base' repository.
Linux 32bit
===========
:Dissolving Genode's dependency from the glibc:
The port of the C runtime to Genode posed an interesting challenge to
the Linux version of Genode. This version used to rely on certain
functions provided by the underlying glibc:
* For creating and destroying threads, we used to rely on POSIX threads
as provided by the 'pthread' library
* The lock implementation was based on the POSIX semaphore functions
'sem_init', 'sem_wait', and 'sem_post'
* Shared memory was realized by using files ('open', 'close',
'ftruncate') and the 'mmap' interface
* Starting and killing processes was implemented using 'fork' and 'kill'
* Inter-process communication used the glibc's socket functions
For our custom C runtime, we want to override the glibc functionality
with our own implementation. For example, we want to provide the 'mmap'
interface to a Genode application by implementing 'mmap' with
functions of our base API. On Linux, however, this base API, in turn,
used to rely on 'mmap'. This is just an example. The problem applies
also for the other categories mentioned above. We realized that we cannot
rely on the glibc on one hand but at the same time replace it by a custom
C runtime (in fact, we believe that such a thing is possible by using
awkward linker magic but we desire a clean solution). Consequently, we
have to remove the dependency of Genode from the glibc on Linux. Step
by step, we replaced the used glibc functions by custom Linux system-call
bindings. Each binding function has a prefix 'lx_' such that the symbol
won't collide with 'libc' symbols. The new bindings are located at the file
'base-linux/src/platform/linux_syscalls.h'. It consist of 20 functions,
most of them resembling the original interface ('socket', 'connect',
'bind', 'getsockname', 'recvfrom', 'write', 'close', 'open', 'fork',
'execve', 'mmap', 'ftruncate', 'unlink', 'tkill', 'nanosleep').
For other functions, we simplified the semantics for our use case
('sigaction', 'sigpending', 'sigsetmask', 'create_thread'). The most
noteworthy changes are the creation and destruction of threads by
directly using the 'clone' and 'tkill' system calls, and the lock
implementation. Because we cannot anymore rely on the convenience of
using futexes indirectly through the POSIX semaphore interface, we
have adopted the simple locking approach that we already use for the
L4/Fiasco version. This lock implementation is a simple sleeping
spinlock.
:Compromises:
The introduction of custom Linux system-call bindings for Genode has
several pros and cons. With this change, The Linux version of Genode is
not anymore easy to port to other POSIX platforms such as the Darwin
kernel. For each POSIX kernel used as Genode platform, a custom
implementation of our system-call bindings must be created. The
original POSIX variant could still be reanimated, but this version
would inherently lack support for Genode's C runtime, and thus would
have limited value. A positive side effect of this solution, however,
is that 'linux_syscalls.h' documents well the subset of the Linux'
kernel interface that we are actually using.
The replacement of POSIX semaphores with sleeping spinlocks decreases
locking performance quite significantly. In the contention case, the
wakeup from sleeping introduces a high latency of up to one millisecond.
Furthermore, fairness is not guaranteed and the spinning produces a bit
of system load. If this approach turns out to become a serious performance
bottleneck, we will consider creating custom bindings for Linux' futexes.
L4/Fiasco
=========
The concepts of _RM_faults_ and _managed_dataspaces_ as described in
Section [Base framework], had been implemented into the L4/Fiasco
version of core. Although the introduction of these concepts involved
only minimal changes at the API level, the required core-internal
changes had been quite invasive, affecting major parts of the pager
and RM-session implementations.
Prior versions of the L4/Fiasco version of core did not implement
the _cancel-blocking_mechanism_ as specified by the CPU-session API.
The missing implementation resulted in lock-ups when destructing a
thread that blocks for lock. With the new implementation based on
L4/Fiasco's inter-task ex-regs system call, such threads can now
be gracefully destructed.

460
doc/release_notes-09-02.txt
View File

@@ -1,460 +0,0 @@
==============================================
Release notes for the Genode OS Framework 9.02
==============================================
Genode Labs
Summary
#######
Whereas the focus of the previous release 8.11 was the refinement of
Genode's base API and the creation of the infrastructure needed to build
real-world applications, the release 9.02 is focused on functional
enhancements in two directions. The first direction is broadening the
number of possible base platforms for the framework. At present, most
microkernels bring along a custom user land, which is closely tied to the
particular kernel. Our vision is to establish Genode as a common ground for
developing applications, protocol stacks, and device drivers in such a way
that the software becomes easily portable among different kernels. This
release makes Genode available on the L4ka::Pistachio kernel. Hence,
software developed with the Genode API can now run unmodified on
Linux/x86, L4/Fiasco, and L4ka::Pistachio. In the second direction, we
are steadily advancing the functionality available on top of Genode. With
this release, we introduce a basic networking facility and support for
native Qt4 applications as major new features. Thanks to Genode's
portability, these features become automatically available on all
supported base platforms.
Our original plan for the release 9.02 also comprised the support of a
Linux-on-Genode (para-)virtualization solution. Initially, we intended to
make [http://os.inf.tu-dresden.de/L4/LinuxOnL4/ - L4Linux] available on
the L4/Fiasco version of Genode. However, we identified several downsides
with this approach. Apparently, the development of the officially available
version of L4/Fiasco has become slow and long-known issues remain unfixed.
L4Linux, however, is closely tied to L4/Fiasco and the L4 environment. For
us at Genode Labs, maintaining both a custom port of L4Linux for Genode
and L4/Fiasco by ourself in addition to developing Genode is unfeasible.
In contrast, the Pistachio kernel features more advanced options for
virtualization ([http://l4ka.org/projects/virtualization/afterburn/ - Afterburner]
and VT support) that we want to explore. Furthermore, there exists another
version of L4Linux called OKLinux for the OKL4 kernel developed at
[http://ok-labs.com - OK-Labs], which is very interesting as well.
Therefore, we decided against an ad-hoc solution and deferred this feature
to the next release. [http:/about/road-map - See our updated road map...]
Major new Features
##################
Genode on L4ka::Pistachio
=========================
From the very beginning, the base API of the Genode OS Framework was
designed for portability. We put a lot of effort into finding API
abstractions that are both implementable on a wide range of kernels and as
close to the hardware as possible to keep the abstraction overhead low. For
this reason, we developed the framework in parallel for the Linux kernel and
the L4/Fiasco kernel. To validate our claim that Genode is highly portable,
Julian Stecklina ported the framework to another member of the L4 family,
namely the [http://l4ka.org/projects/pistachio/ - L4ka::Pistachio kernel].
This high-performance kernel implements the latest official L4 API called
L4.x2 and has a number of advanced features such as multi-processor support
and virtualization support.
After Julian successfully created the first Pistachio version of Genode,
we successively refined his work and conducted further unifications among
the platform-dependent code for the different kernels. The result of this
effort is included in this release and comes in the form of the
'base-pistachio' source-code repository.
;Interesting technical notes:
* The IRQ handling on Pistachio is slightly different from L4/Fiasco.
On L4/Fiasco, an IRQ becomes unmasked only when the user-level IRQ
handler thread blocks for an IRQ by issuing an IPC call to the
kernel's corresponding IRQ thread. In contrast, Pistachio unmasks an
IRQ as soon as the user-level IRQ handler associates itself with an
IRQ. Thus, an IRQ message may occur not only when the user-level IRQ
handler blocks for any IRQ but anytime. In particular, IRQ messages
may interfere with the IRQ handler's IPC communication with other
threads. To ensure that IRQ messages do only occur when expecting
them, we lazily associate the IRQ handler thread to the IRQ the
first time we wait for an IRQ and issue an unmasking IPC call
subsequent times.
* Genode provides a mechanism for gracefully destructing threads that
are in a blocking state, for example waiting for an IPC message.
Such a thread may hold locks or other resources that would not
get properly freed when just killing the thread by force. Therefore,
Genode provides a way to issue the cancellation of a blocking
operation by another thread (e.g., the thread calling the destructor).
Once, a blocking operation got canceled, a C++ exception
('Blocking_canceled') is raised such the thread can fall back into
a defined state and then be destructed. On L4ka::Pistachio, we use
Pistachio's pager-exchange-registers feature in combination with
the user-defined UTCB handle for cancelling blocking operations and
detecting cancellations. The interesting code bits can be found in
'src/base/ipc/ipc.cc', 'src/base/lock/lock.cc',
'src/core/platform_thread.cc', and in the Pistachio-specific
timer-service implementation.
* During the refinement of the Pistachio version, we were able to further
generalize code that was previously specific for L4/Fiasco and
L4ka::Pistachio respectively. Now, the platform-specific code comprises
less than 3,000 lines of code (LOC) for L4/Pistachio, circa 2,000 LOC
for L4/Fiasco, and circa 1,000 LOC for Linux. Hence, we expect that
porting the framework to further kernels is possible at reasonable
engineering costs.
:Current limitations:
* The current version does not use superpages (4M mappings) because we
experienced problems with mapping 4K pages out of 4M pages. This is an
issue that we like to investigate further because using 4M mappings
would improve the boot time and reduce the kernel-memory usage.
* Currently, we use a simple sleeping spinlock for synchronization, which
is not optimal for several reasons. There are no fairness guarantees,
the spinning consumes CPU time, and threads that got blocked in the
contention case are woken up at the coarse granularity of the kernel's
timer tick, which is typically one millisecond.
* Nested RM sessions as introduced as an experimental feature in the
Genode release 8.11 are not yet supported.
:Further details:
You can find further technical details and usage instructions at this
dedicated [http://genode.org/documentation/platforms/pistachio - page].
Qt4 on Genode
=============
The minimalism of the Genode OS Framework with regard to its code
complexity raised the question of whether this framework is feasible
for hosting real-world applications and widely used runtime environments.
Christian Prochaska took the challenge to port Trolltech's Qt4 application
framework, which serves as the basis for the popular KDE desktop, to Genode.
Because Christian started his work more than a year ago at a time when no
C library was available on Genode, several intermediate steps were needed.
The first step was the integration of the Qt4 tools such as the meta-object
compiler (moc) and resource compiler properly into the our build systion.
With the tools in place, the Linux version of Genode came to an advantage.
In this environment, a Genode application is able to use glibc functionality.
So the problem of a missing C library could be deferred and Christian was
able to focus on interfacing Qt with the existing Genode services such as
the Nitpicker GUI sever. Next, the glibc dependencies were successively
replaced by custom implementations or simple dummy stubs. Thereby, all
needed functionalities such as timed semaphores and thread-local storage
had to be mapped to existing Genode API calls. Once, all glibc dependencies
had been dissolved, Qt could be compiled for the L4/Fiasco version.
Since a C library has become available in Genode 8.11, we were able to
replace Christian's intermediate stub codes with a real C library. We also
utilize recently added features of Genode such as its alarm framework to
simplify the Qt4 port. Furthermore, we were able to remove all
platform-specific bits such that the Qt4 port has now become completely
generic with regard to the underlying kernel. Qt4 can be executed on Linux,
L4/Fiasco, and L4ka::Pistachio without any changes. Figure [qt4_screenshot]
shows a screenshot of Qt's Tetrix example running side-by-side with native
Genode applications.
[image qt4_screenshot]
:Current state:
* The Qt4 port comes in the form of a source-code repository, which contains
all Qt source codes, and some example programs such as Tetrix. You can
download the Qt4 repository as a separate archive at the download page of
the Genode release 9.2. For the next release, we plan to separate the
Genode-specific parts from Qt original code and make the Genode-specific
parts a regular component of the Genode main line.
* The Qt4 port consists of Qt's Core library, GUI library, Script library,
XML library, and the UI tools library. Other libraries such as Webkit
are not ported yet.
* This first version of Qt4 on Genode is not to be considered as stable.
There are several known issues yet to be addressed. In particular,
the 'QEventDispatcher' is still work in progress and not fully stabilized.
* Because, we use to statically link programs, the binaries of Qt
applications are exceedingly large. For example the Tetrix binary is
100MB including debug information and 11MB in the stripped form. For
employing Qt on Genode at a larger scale, Genode should be enhanced with
shared-library support.
Networking
==========
With Genode 8.11, we introduced the Device-Driver-Environment Kit (DDE Kit)
API, which is a C API specifically designed for implementing and porting
device drivers written in plain C. We have now complemented DDE Kit with an
environment for executing Linux device drivers natively on Genode. This
library is called 'dde_linux26' and contained in our new 'linux_drivers'
source-code repository. The environment consists of several parts, which
correspond to the different sub systems of the Linux kernel 2.6, such as
'arch', 'drivers', 'kernel'.
The first class of device-drivers supported by DDE Linux 2.6 is networking.
At the current stage, the DDE Linux network library comprises general
network-device infrastructure as well as an exemplary driver for the PCnet32
network device.
Based on this library, we have created a basic TCP/IP test utilizing the
uIP stack, which uses the DDE Linux network library as back end. The test
program implements a basic web server displaying uIP packet statistics.
When executed on Qemu, you can use your host's web browser to connect to
the web server running on Genode:
For booting Genode on L4/Fiasco with the web-server demo, use a GRUB
entry in your 'menu.lst' file as follows.
! title Genode: DDE Linux 2.6 NET on L4/Fiasco
! kernel /fiasco/bootstrap -maxmem=64 -modaddr=0x02000000
! module /fiasco/fiasco -nokd -serial -serial_esc
! module /fiasco/sigma0
! module /genode/core
! module /genode/init
! module /config
! module /genode/timer
! module /genode/pci_drv
! module /genode/test-dde_linux26_net
The first four lines are L4/Fiasco specific. When using L4ka::Pistachio,
the 'menu.lst' entry looks like this:
! title Genode: DDE Linux 2.6 NET on L4/Pistachio
! kernel /pistachio/kickstart
! module /pistachio/x86-kernel
! module /pistachio/sigma0
! module /genode/core
! module /genode/init
! module /config
! module /genode/timer
! module /genode/pci_drv
! module /genode/test-dde_linux26_net
The web-server test requires the PCI bus driver and the timer service.
Therefore, the 'config' file for Genode's init should have following
content:
! <config>
! <start>
! <filename>timer</filename>
! <ram_quota>512K</ram_quota>
! </start>
! <start>
! <filename>pci_drv</filename>
! <ram_quota>512K</ram_quota>
! </start>
! <start>
! <filename>test-dde_linux26_net</filename>
! <ram_quota>16M</ram_quota>
! </start>
! </config>
Now, its time to create an ISO image from all files specified in
the GRUB configuration. For this, the new utility 'tool/create_iso'
becomes handy. The ISO image can then be booted on Qemu using the
following arguments:
! qemu -m 64 -serial stdio -no-kqemu -cdrom <iso-image> \
! -net nic,model=pcnet -net user -redir tcp:5555::80
The '-redir' argument tells qemu to redirect TCP connections with
localhost:5555 to the guest OS at port 80. After having booted
up Genode on Qemu, you can use your host's web browser to access
the web server:
! firefox http://localhost:5555
:Notes about using the TAP version:
* Preparations
* You must be permitted to sudo and have installed the tunctl
utility. Under Debian/Ubuntu execute
! sudo apt-get install uml-utilities
* Create TAP device
! TAPDEV=$(sudo tunctl -b -u $USER)
! sudo /sbin/ifconfig $TAPDEV 10.0.0.1
* setup DHCP server on $TAPDEV and 10.0.0.0/8
* Run qemu
! qemu -m 64 -serial stdio -no-kqemu -cdrom dde.iso \
! -net nic,model=pcnet \
! -net tap,ifname=$TAPDEV,script=no,downscript=no
* Ping
* Cleanup
* Stop DHCP server
* Remove TAP device
! sudo tunctl -d $TAPDEV
Operating-system services and libraries
#######################################
C Runtime
=========
We have replaced the 'malloc' implementation of the original FreeBSD C
library with a custom implementation, which relies on Genode's 'Heap' as
allocator. The FreeBSD libc reserves a default memory pool of 1MB, which
is no problem on FreeBSD because virtual memory is backed lazily with
physical pages on demand. On Genode however, we immediately account the
allocated memory, which implicates high quota requirements even for
applications that use little memory. In contrast, Genode's heap allocates
and accounts its backing store in relatively small chunks of a few KB.
Therefore, the quota accounting for applications is much more in line with
the actual memory usage. Furthermore, our custom 'malloc' implementation
has the additional benefit of being thread safe.
* Added i386-specific parts of gen lib, in particular longjmp, setjmp.
Device-Driver-Environment Kit
=============================
* The DDE Kit uses our alarm framework (located in the 'os' repository) now
rather than its own event-scheduler implementation formerly called 'Tick'.
* We refined the DDE Kit API and reduced the number of custom types. For
example, we removed the custom 'dde_kit_lock_t' and using
'struct dde_kit_lock' instead, and replaced 'dde_kit_thread_t' with
'struct dde_kit_thread'.
Because of the apparent stabilization of the DDE Kit API, we have now added
this API to Genode's official API reference.
[http://genode.org/documentation/api/dde_kit_index - See the documentation of the DDE Kit API...]
PS/2 input driver
=================
We improved the PS/2 keyboard driver by adding missing scan-code translations
for the scan code set 1, in particular the cursor keys.
Applications
############
Launchpad configuration
=======================
Launchpad is a graphical application for interactively starting and killing
programs. It is used for the default demonstration of Genode. By default,
launchpad displays a preconfigured list of programs and their respective
default memory quotas. The user can tweak the memory quota for each entry
with mouse and then start a program by clicking on its name. As an
alternative to using the default list, you can now define the list manually
by supplying a configuration to Launchpad. The following example tells
launchpad to display a list of two launcher entries:
!<config>
! <launcher>
! <filename>sdl_pathfind</filename>
! <ram_quota>10M</ram_quota>
! </launcher>
! <launcher>
! <filename>liquid_fb</filename>
! <ram_quota>10M</ram_quota>
! </launcher>
!</config>
To use this configuration for a Launchpad started via init, you can simply
insert the launchpad configuration into the '<start>' node of the launchpad
entry in init's 'config' file.
Platform-specific changes
#########################
L4/Fiasco
=========
* Raise 'Blocking_canceled' exceptions on canceled IPC calls
32-bit Linux
============
* We continued dissolving the dependency of Genode from the glibc by using
a custom 'getenv' implementation used during process creation.
* By default, we compile now with '-nostdinc' and explicitly specify
'/usr/include' as include search directory only when needed. Previously,
a Genode application, which included a host include file by mistake, has
not raised any compilation error when compiled for the Linux version of
Genode. Now, all Genode platforms behave equally with regard to include
search directories.
* We enforce using the actual compiler's C++ support libraries rather than
the default libraries installed on the host.
Tools and build infrastructure
##############################
Official tool chain
===================
At the download section of our website, we used to provide a crosstool-based
tool chain as pre-compiled binaries. Since we got several requests about
how to build such a tool chain from scratch, we created custom utility for
downloading, building, and installing the official Genode tool chain. You
can find the utility at 'tool/tool_chain'. For usage instructions, just
start 'tool_chain' without arguments. Because this utility is a plain script,
you can follow and verify each step that we use for creating the Genode tool
chain. Currently, this official tool chain is based on binutils 2.18 and
gcc 4.2.4.
As an alternative to installing the tool chain from source, we also
provide pre-compiled binaries at the download section of our website.
[http://genode.org/download/tool-chain - Visit our tool-chain download website...]
For the Linux version of Genode, we still use the host's default gcc
as tool chain. This way, we spare the hassle of downloading and installing
a custom tool chain for somebody who wants to give Genode a quick try.
With this is mind, we have fixes several small issues with gcc 4.3.2:
* Fixed dependency generation for gcc-4.3.2. Older version of gcc used to
append a '.o' dependency at the target of '.d'-files. However, gcc-4.3.2
seems to handle the option '-MT' differently, resulting in a rule that
contains only the '.d' as target. Now, we explicitly specify both the
'.o' file and the '.d' file as target. Consequently, on older gcc
versions, the '.o' file appears twice but that is no problem.
* Fixed assembler issue with the 'fnstsw' instruction in the C library.
This instruction does not accept eax but only ax as argument.
Build-directory creation tool
=============================
We added a rule for creating a pre-configured build directory for the
Pistachio version to our build-directory creation tool
('tool/builddir/create_builddir'). Furthermore, we changed the default
build configuration such that the official Genode tool chain is used for
L4/Fiasco and L4ka::Pistachio.
Build system
============
* Improved clean rule - visit each target directory only once
* Stop the build process on the first error by default, for continuing
the build process depite of an error, you can use the '-i' argument
of make.
* Compiler flags can now be set specific for compiling C and C++ sources.
This is needed because both variants allow different sets of warning
options. The new variables are called 'CC_CXX_OPT' and 'CC_C_OPT'.
ISO image creation tool
=======================
We have created a convenient front end for 'genisoimage', which we
use for testing Genode on Qemu. You can find this ISO-image-creation
tool at 'tool/create_iso'. For usage instructions, simply start the
tool without arguments.

585
doc/release_notes-09-05.txt
View File

@@ -1,585 +0,0 @@
==============================================
Release notes for the Genode OS Framework 9.05
==============================================
Genode Labs
Shortly after including support for the L4ka::Pistachio kernel in the
previous release, the Genode version 9.05 gives a further evidence of
Genode's high portability by making the framework available on top of
the OKL4 kernel. This kernel is a commercial-grade microkernel
developed by [http://ok-labs.com - Open Kernel Labs]. In Section
[Supporting the OKL4 kernel as new base platform], we elaborate on the
new base platform and summarize the experiences made during our porting
work.
The previous Genode release was accompanied by a source-code archive containing
the initial version of Qt4 for Genode. Our approach is to make the Qt4
framework available for building Genode applications running natively on the
microkernel rather than within a virtualization environment. As advertised in
our [http://genode.org/about/road-map - road map], we have now seamlessly
integrated the Qt4 framework into our mainline source tree. Furthermore, we
have adapted our port to the Qt4 version 4.5.1. Section [Integration of Qt4
into the mainline repository] gives a rough overview of the changes and an
introduction on how to use the Qt4 framework with Genode.
The third major feature of the release is the addition of preliminary USB
support. We have been able to port major parts of Linux' USB infrastructure
to Genode using the DDE Kit introduced in autumn 2008. Section [USB support]
presents an overview about the pursued design and the current state of
implementation.
Section [OKLinux on Genode] outlines our ongoing efforts of running Linux
as a node in Genode's process tree.
Supporting the OKL4 kernel as new base platform
###############################################
The OKL4 kernel developed by Open Kernel Labs started as a fork of the
L4ka::Pistachio kernel. Whereas L4ka::Pistachio remained true to the L4
x.2 specification, OKL4 was subject of major API changes geared towards high
performance on the ARM architecture. OKL4 earned much fame for executing a
user-level variant of Linux (OKLinux) on top the microkernel, which turned out
to be faster than executing Linux natively on the ARM9 architecture. Even
though OKL4 is primary targeted at the ARM architecture, we wanted to go for
the x86 variant because of two reasons. First, there exists the just mentioned
user-level port of Linux for OKL4, which looks like an attractive way to execute
Linux on Genode once Genode runs on OKL4. Second, we think that distributing
Genode in the form of ISO images bootable on plain PC hardware is the best
way to reach the OS community. Therefore, we decided to use OKL4 version 2.1 as
the base for our work. In contrast to later releases, this version supports
both x86 and ARM. The following section reviews the unique features of the
OKL4 kernel from our perspective.
OKL4 viewed from the angle of a Genode developer
================================================
On the kernel-API level, OKL4 has several interesting properties that had been
both welcome and challenging. We want to highlight the following points:
In contrast to prior L4 kernels, OKL4 has *removed wall-clock timeouts* from
the kernel interface. On L4, timeouts were used as arguments for for blocking
IPC operations serving two purposes. First, specifying IPC timeouts allowed the
IPC caller for regaining control over the blocking thread after the specified
time elapsed. This is considered as important in the case that the called
thread misbehaves and never answers the call. However, the problem of choosing
an appropriate timeout was never properly resolved. When dimensioning the
timeout too small, the called thread may miss the timeout just because it had
no chance to be selected by the scheduler in time. Such timeouts rely on the
presumption that there is low load on the system. On the other hand, when
dimensioning the timeout too high, the system will become sluggish when the
called thread misbehaves. For example, a simple GUI server may want to send
input events to its clients with a timeout to be robust against misbehaving
clients that never wait for events. When choosing a timeout too small, chances
are high that an event will occur at a time when the receiver is handling a
previous event. The timeout would trigger and the event would get lost. When
choosing the timeout too large, say 1 second, any misbehaving client could make
the GUI server freeze for 1 second. Therefore, timeouts for regaining control
over a blocked thread seem to be a bad idea. So we welcome their absence in
OKL4. The second use of timeouts is their use as user-level time source. On L4,
sleep is typically implemented as a blocking IPC with a timeout set to the
sleep value. For this purpose, a system built on top of OKL4 has to employ a
user level device driver accessing a timer device. In Genode, we already have a
timer service for this purpose. So we won't miss timeouts at all.
Classical L4 kernels provide two variants of *synchronous IPC*. So called long
IPC could copy any amount of memory from the sending to the receiving address
space. This is complicated operation because either communication partner may
specify communication buffers that contain unmapped pages. Hence, page faults
may occur during long-IPC operations. On L4, page faults, in turn, are handled
by the user land. Not until a user-level pager thread resolves the page fault
by establishing a mapping at the faulting address, the kernel can proceed the
IPC operation. This sounds pretty complicated, and it is. The second IPC
variant is called short IPC. It constrains the transferable payload to CPU
registers. Hence, these IPC operations should only be used for messages with a
payload of a maximum of 2 machine words. Because short IPCs are never touching
user-level memory pages, no page faults can occur.
On OKL4, there is only one IPC operation, which copies payload from the
sender's user-level thread-control block (UTCB) to the receiver's UTCB. An
UTCB is an always-mapped memory region. Hence no page faults can occur during
IPC operations. On Genode, the UTCB size of 256 bytes may become a limitation
when RPC messages get large. For example, session requests may include large
session-argument strings specifying session-constructor arguments. Current
services rely only on a few arguments so the size limitation is not an
apparent problem. But that may change for future services. Furthermore, in
contrast to L4 x.2, OKL4 does not allow for transferring payload other than
plain data. In particular, OKL4 does not support the transfer of memory
mappings via IPC. Removing memory mappings from the IPC operation is a very
good idea. On Genode, only roottask (core) establishes mappings and shared
memory is implemented as a user-level protocol (data spaces). There is no need
to allow arbitrary processes to establish memory mapping via IPC.
The *boot procedure* of OKL4 largely differs from other L4 kernels. This is
attributed to Open Kernel Labs' focus on embedded systems, which mostly rely on
single-image boot loading. OKL4 employs a tool (elfweaver) for creating a
bootable image from a bunch of files and an XML configuration file. Among the
declarations about which processes to be loaded and which policies to enforce,
the configuration file contains platform parameters such as the amount of
physical memory of the machine. This static approach to configure a system is
certainly useful for embedded systems but PC hardware uses to vary a lot. In
this case, evaluating boot-time memory descriptors would be the preferred
solution.
OKL4 introduces kernel support for *user-level synchronization*. Prior L4
kernels facilitated user-level synchronization through a combination of
synchronous IPC operations with either priorities or delayed preemption.
OKL4's mutexes can make the life in the user land much easier. However, we have
not looked into OKL4 mutexes yet.
There does not exist a recursive *map operation* as the source operand of the
map operation is a physical memory descriptor rather than a virtual address in
the mapper's address space. Consequently, this design eliminates the need for
having a recursive unmap operation and thereby, the need to maintain a mapping
data base in the kernel. This is cool because Genode keeps track of the
mappings at the user level anyway (within core). From our perspective, there is
no need to maintain mapping relationships in the kernel. Removing the mapping
database effectively discards a lot of much-discussed problems about how to
manage the mapping database in a clever way.
There exists *no root memory manager* (sigma0). Because the map operation
takes a physical memory descriptor as argument instead of a virtual address
in the mapper's address space. The mapper does not need to have the mapped
page locally mapped within its own address space. In fact, core (as the only
mapper in a Genode system) does only have very little memory mapped locally.
This design accelerates the boot time because there is no need to map each
physical page in core at startup as performed when running core on the other
L4 kernels.
These differences of OKL4 compared with the microkernels already supported
by Genode posed a number of interesting challenges and opportunities. We have
thoroughly documented the process in
[http://genode.org/documentation/articles/genode-on-okl4 - Bringing Genode to OKL4].
Usage
=====
For using Genode with OKL4, please refer to the following dedicated page:
:[http://genode.org/documentation/platforms/okl4 - Genode on the OKL4 microkernel]:
Site about building and using Genode with the OKL4 kernel.
Limitations of the current implementation
=========================================
The current implementation is able to execute the complete Genode demonstration
scenario on OKL4. This means, we can build and destroy arbitrary trees of
processes, have all the needed infrastructure in place to execute user-level
device drivers such as VESA and PS/2, perform inter-process communication
via RPC and shared memory, and have all basic framework API functions available.
We regard the current state as the first functional version. However, there are
the following points that need improvement and are subject to our future work.
:Incomplete timer driver:
On x86, the timer driver should program the PIT to dispatch sleep requests.
However, the I/O ports of the PIT can only by made available to one party in
the system (which naturally would be the timer driver). Unfortunately, there
are some VESA BIOSes around, which try using the PIT directly. The current
version of our VESA driver does not virtualize these accesses. It rather
tries to gain direct access to the I/O ports from core. This would not work
if the timer already uses this device resource. Our plan is to supplement
our VESA driver with a virtual PIT that uses the timer service as back end.
Then we can safely use the PIT by the timer driver.
:Signalling framework not yet implemented:
We have not yet implemented Genode's API for asynchronous notifications
in the OKL4 version. In fact, the goal of the initial version of the
OKL4 support was running the default demonstration scenario, which does
not rely on signals. The second and more technical reason is that we
consider exploiting OKL4's event mechanism for implementing the signalling
API but have not finalized the design. The generic implementation as used
on the other platforms cannot be used on OKL4 because this implementation
utilizes one helper thread per signal transmitter. Within core, each RM
session is a potential signal transmitter, which means that we need to
create a helper thread per process. Unfortunately, by default, OKL4
limits the number of threads within roottask (core) to only 8 threads,
which would impose a severe limit on the number of processes we could
run on OKL4.
:OKL4's kernel mutexes yet to be used:
We have not yet explored the use of mutexes provided by the OKL4 kernel
for implementing Genode synchronization APIs but we rather rely on a
yielding spin lock for now. This has a number of drawbacks such as high
wake-up latencies in the contention case (depending on the scheduling
time slice), no support for priorities, and no fairness. Although it
is a simple and robust solution to start with, we plan to facilitate
the OKL4 kernel feature with our upcoming work.
:Overly simplistic UTCB allocation:
Right now, we allocate a fixed amount of 32 UTCBs per address space and
thereby limit the maximum number of threads per process. In the future,
this limit should be made configurable.
:Managed dataspaces not yet supported:
The support of managed dataspaces relies on the signal API, which is
not yet available for OKL4.
:Message buffers are limited to 256 bytes:
Because OKL4 performs message-based inter-process communication by
copying data between the UTCBs of the communicating threads, the
UTCB size constaints the maximum message size. Therefore, message
must not exceed 256 bytes. This is not a huge problem for the currently
available Genode programs but we can imagine session argument-lists
to become larger in the future.
:Advanced thread functions are incomplete:
Thread functions such as querying registers of remote threads are not yet
implemented.
Integration of Qt4 into the mainline repository
###############################################
Qt4 is a tool kit for developing platform-independent applications. It
comprises a complete platform-abstraction layer and a rich GUI tool kit
widely used for commercial and open-source applications. It is particularly
known as the technical foundation of the KDE project. The previous Genode
release was accompanied by a snapshot of our initial port of Qt4 to Genode. For
the current release, we have turned this proof-of-concept implementation into a
properly integrated part of the Genode mainline development. This enables Qt4
applications to be executed natively on the full range of kernels supported by
Genode.
Usage
=====
We complemented Genode's source tree with the new 'qt4' source-code repository,
which contains the Genode-specific parts of the Qt4 framework. The most
portions for the Qt4 framework are used unmodified and thereby have not been
made part of the Genode source tree. Instead, we fetch the original Qt4 source
code from Trolltech's FTP server. This way, our source tree remains tidy and
neat.
For using Qt4 for your Genode applications, you first need to download and
prepare the original Qt4 source codes and build a few Qt4 tools such as the
meta-object compiler and the resource compiler. The makefile found in the
top-level directory of the 'qt4' repository automates this task:
! make prepare
To include the 'qt4' repository into the Genode build process, just add the
'qt4' directory to the 'REPOSITORIES' declaration of the 'etc/build.conf' file
within your build directory. Make sure that the repositories 'demo' and 'libc'
are included as well. The 'qt4' repository comes with a couple of demo applications.
The 'qt_launchpad' is especially interesting because it makes use of both the
Qt4 framework and the Genode framework in one application.
Features and limitations
========================
The Qt4 port comprises Qt's Core library, GUI library, Script library, XML
library, and the UI tools library.
For using Qt4 on the Linux version of Genode, we recommend using the Genode
tool chain rather than your host's tool chain. Qt4 makes use of a lot of libc
functionality, supplied by Genode's 'libc' repository. However, on Linux we
still link against your host's libc. This becomes a problem if your host
compiler's C++ support code references libc functionality that is not part of
Genode's libc. Thereby the linker will silently create references to glibc
symbols, making both libraries collide. So if using Qt4, we recommend using the
Genode tool chain:
:[http://genode.org/download/tool-chain]:
Information about downloading and using the Genode tool chain
USB support
###########
This release introduces the first fragments of USB support to Genode, taking
the USB human-interface device (HID) class as starting point. With this work,
we follow our approach of reusing unmodified Linux device drivers executed
within a device-driver environment called DDE Linux. In the previous release,
we already utilized this approach for realizing basic networking on Genode.
With this release, we complement DDE Linux with support required by USB
drivers. We are grateful for being able to base our implementation on the
excellent foundation laid by Dirk Vogt. He described his work in
[http://os.inf.tu-dresden.de/papers_ps/beleg-vogt.pdf - USB for the L4 environment].
For USB HID support, we added the Linux USB and input subsystems to the DDE
Linux 2.6 framework. Besides the 'dde_linux26/net.h' API for network drivers
added in Genode 9.02, the current version also includes APIs for input
('dde_linux26/input.h') and USB ('dde_linux26/usb.h'). We intend these
interfaces to mature towards generic driver-library APIs in the future. For
example, BSD-based drivers shall transparently provide the same functionality
as the current Linux drivers, which permits the simple reuse of driver server
implementations.
[image usb_current]
Image [usb_current] illustrates the current implementation of the USB-based
human-interface device (HID) driver. In this monolithic setup, all parts of the
USB stack and the device API are executed within one address space. These parts
are
* Input server glue code
* HID driver and input subsystem
* Core functions for management of USB request buffers (URBs),
attached devices, and registered drivers
* Host controller drivers for UHCI, OHCI, and EHCI
[image usb_aspired]
We regard this as an intermediate step towards our goal to decompose the USB
stack. Image [usb_aspired] shows our aspired design. In this design, the
USB server and one or more USB gadget drivers run in dedicated address spaces.
The USB server provides two interfaces called USB session interface and USB
device interface. A USB session interface corresponds to a virtual root hub,
from which USB devices can be queried. The client of the USB session interface
is usually an USB gadget driver that uses the USB device interface. Because
this interface is used for transferring the actual payload at a potentially
high bandwidth, it is based on shared memory and signals. The USB server
consists of the following components:
* USB server glue code
* Virtual USB device driver managing all attached devices
* Core functions including hardware hub management
* Host controller drivers
The USB server presents a virtual USB hub to each USB gadget driver. Such
a driver consists of:
* Device interface, e.g., input server glue code
* Gadget driver, e.g., HID driver and input subsystem
* Core functions
* Virtual host controller
* USB client glue code
The HID driver uses the USB session API to monitor ports of its virtual root
hub and submit URBs to attached devices. The session interface facilitates the
signalling framework for event notification and a shared-memory dataspace for
URB transmission.
The 'os' repository already contains the USB session and USB device interfaces.
However, the decomposition is not yet in a functional state.
:Current limitations:
The current monolithic implementation of the USB HID service can already be
used as a replacement of the PS/2 driver. However, both drivers cannot be used
at the same time, yet. To enable the use of both USB HID and PS/2, we plan to
create a further component that merges multiple streams of input events and
passes the result to the GUI server.
OKLinux on Genode
#################
According to our road map, we pursued the goal to run Linux as a node in
Genode's process tree. We explored two approaches:
:Reanimating the Afterburner project conducted by the [http://l4ka.org - L4Ka group]:
This approach is the result of the L4Ka groups's long-year experience with
manually supporting L4Linux on top of the L4ka::Pistachio kernel. Because of
the high costs of maintaining the paravirtualized Linux kernel, a
semiautomatic paravirtualization technique was created. According to the
impressive results presented in
[http://www.l4ka.org/l4ka/publ_2006_levasseur-ua_soft-layering.pdf - Pre-Virtualization: Soft Layering for Virtual Machines],
this approach is able to drastically reduce maintenance costs while retaining
good performance. Furthermore, the approach was applied not only to Linux
running on the L4 kernel but also for using Xen or Linux as underlying
host operating systems.
:Porting the OKL4-specific version of L4Linux to Genode:
Open Kernel Labs maintain a custom version of L4Linux that runs on OKL4. This
version is mostly referred to as OKLinux aka Wombat. Since Genode can now use OKL4
as base platform, the reuse of OKLinux in combination with Genode has become
a feasible option.
Both approaches have pros and cons. Whereas Afterburner is a intriguing
approach, this project seems to be stalled. It relies on a rather old tool
chain, and recent Linux features such as thread-local storage support are not
considered, yet. To pick up this solution for Genode will require us to fully
understand the mechanisms and the code. So we consider this as a mid-term
solution. In short term, running OKLinux on Genode is more feasible. We were
already able to create a prototype version of OKLinux running on Genode. This
version starts up the kernel including all Linux kernel threads, mounts the
boot partition supplied as memory image, and starts the init process. The
engineering costs had been rather low. We replaced the Iguana user land
libraries as originally used by Wombat by a Genode-specific reimplementation to
keep our manual adaptions of the Linux kernel code as small as possible.
Our custom reimplementation of the needed Iguana APIs consists of less than
1,000 lines of code (SLOC). The diff for our changes to the OKLinux kernel code
comprises less than 1,000 lines. We plan to make a snapshot of this prototype
publicly available soon.
Operating-system services and libraries
#######################################
Nitpicker GUI server
====================
We optimized the performance of the 'refresh' call, which updates all views of
a session, which display a given buffer portion. The new implementation restricts
the redraw operations to the fragment of each view that displays the specified
buffer portion. The performance improvement becomes most visible when updating
only small parts of a buffer.
USB session interface
=====================
Genode's emerging USB support introduces two new interfaces to the 'os' repository,
which are called USB session and USB device.
An _USB_session_ is a virtual USB bus with just one root hub with 'MAX_PORTS'
downstream ports. The client of such as session submits USB request blocks
(URBs) and is, in turn, informed about port changes on the root hub as well as
request completion. Connected USB devices can be referenced by USB device
capabilities and are associated with one port at the virtual root hub on
server side.
An _USB_device_ references a hardware device connected to a virtual USB bus's
root hub. The device capability enables the client to send USB request
blocks to the hardware device.
Input interface
===============
We updated the key codes of the input interface in response to recent changes
of Linux' 'dev/event' definitions.
VESA driver
===========
Until now, there existed different processes that tried to access the PCI bus
via I/O ports, in particular the VESA framebuffer driver and the PCI bus
driver.
Since core enforces that each I/O port can only be assigned exclusively to one
component in the system, multiple processes that need access to the same I/O
ports cannot run at the same time. For our default demonstration scenario, we
had been able to allow the VESA driver to use the PCI I/O ports because nobody
else needed them. However, our growing base of device drivers relies on the
PCI bus driver. To be able to use the VESA driver together with other drivers,
we virtualized the access to the PCI bus from within the VESA driver.
Our current PCI virtualization code is pretty limited. The VESA driver sees a
virtual PCI bus with only the VGA card attached. For now, we only allow reading
the PCI configuration space of this device, but not writing to it. Apparently,
this simple approach is sufficient to run the VESA BIOS of Qemu. However, other
VESA BIOS implementations may need further access to the PCI device's
configuration space. For example, for querying the size of a PCI resource,
write access to base address registers is required. In such an event, the VESA
driver will print a message about the missing virtualization feature:
! writing data register not supported
If you see such a message, we are very interested to see your log output such
that we can enhance our PCI virtualization code as needed. Please contact us!
Base framework
##############
In the process of bringing Genode to the OKL4 kernel, we have generalized much
of former platform-specific code:
* The initialization of C++ exception handling has now become part of the
generic 'cxx' library in the 'base' repository. All platforms except
Linux are using this generic library now.
* The 'server' library used to contain a platform-specific part that
implemented the 'manage' function of a 'Server_entrypoint'. The
generalized version of this library is now being used on all platforms
other than Linux.
* We unified core-internal interfaces and their implementations such as
'Dataspace_component', 'Cap_session_component', 'Rm_session_component',
and 'Irq_session_component'. The result has become part of the 'base'
repository.
* On OKL4, threads need to execute small startup code for querying their
own thread IDs. Therefore, we have extended the 'Thread_base' interface
with a platform-specific hook function called '_thread_bootstrap'.
* The types defined in 'base/native_types.h' had been complemented by a
new 'Native_thread_id' type. This type is exclusively used by core and the
framework libraries. For using the Genode API, this type is meaningless.
* For the 64bit support, we slightly refined the interfaces of some utility
template functions in 'util/misc_math.h'. Furthermore, parts of the generic
marshalling code of the IPC framework needed refinement, but no API changes
were needed.
Linux-specific changes
######################
Adaptation to 64 bit
====================
Because most Genode developers tend to work with the Linux version of Genode,
supporting 64-bit Linux becomes increasingly important. With the current release,
we start to officially support 64-bit Linux as base platform. This comes
along with the following changes:
* We replaced the 'spec-x86.mk' file with new 'spec-x86_32.mk' and 'spec-x86_64.mk'
files. The default version of 'base-linux/etc/specs.conf' automatically
chooses the right spec file according to the output of 'uname -m'. Therefore,
output of the build processes matches your host architecture. This behaviour
can be changed by placing a customized 'spec.conf' file in your build directory's
'etc/' subdirectory.
* We added type definitions for 64-bit-specific fixed-size integers in the form
of a 64-bit-specific 'fixed_stdint.h' file.
* Because using 64 bit instead of 32 bit changes the payload size of RPC
messages, we had to adjust several message buffers such as 'Ram_session_client'
and 'Input::Session_client', and adapted the used stack sizes.
* Towards the goal of completely dissolving Genode's dependency on the Linux' glibc,
we implemented custom system-call bindings. Apparently, Linux' syscall interface
differs between 32 bit and 64 bit. For example, the 32-bit version handles
all socket-related calls via a compound 'socketcall' whereas the 64-bit
version uses distinct syscalls. Another difference is the handling of the
'mmap' syscall and different behaviour of 'tkill'. The latter problem was
resolved by replacing 'tkill' with 'tgkill' and setting the thread-group
argument of the corresponding PID. Therefore, a 'Native_thread_id' on Linux
now comprises both the TID and the PID.
* The 'Platform_env' on Linux contains a local implementation of the 'Rm_session'
interface, which uses 'mmap' to attach dataspaces to the process' address
space and maintains the region list in the form of a static array. This array
was dimensioned to 256 entries, which constrained the maximum amount of
usable memory when allocating a lot of small blocks via Genode's heap. Since
the heap allocates backing store at the granularity of only 16KB, the worst
case for reaching this limit was about 4MB. This was OK for our simple test
applications. But for using Qt4, in particular on 64 bit, this has become a
serious limitation. For now, we increased the region limit to 4096 and plan
to replace the static array with a dynamically growing data structure.
Furthermore, we made the heap granularity depend on the actual machine-word
size. Therefore, the heap allocates its backing store in 32KB blocks when
running on 64 bit.
Debugging hooks
===============
On Linux, we use gdb for debugging Genode. This is feasible as long as the
targeted process is running. However, during low-level debugging, we had the
recurring problem of a thread dying shortly after starting up. So we added a hook
for halting a thread at the startup in order to be able to attach gdb to the
thread before it dies. This simple hook lets the thread wait for a key press by
directly calling the 'read' syscall. We also added a simple debug facility for
printing debug messages bypassing Genode's LOG service by calling the 'write'
syscall directly. Both hooks are now part of the Linux version of the 'env'
library (see 'base-linux/src/base/env/debug.cc'). Note that these hooks are not
part of the Genode API. There exists no header file.

1016
doc/release_notes-09-11.txt
View File

File diff suppressed because it is too large Load Diff

1224
doc/release_notes-10-02.txt
View File

File diff suppressed because it is too large Load Diff

1211
doc/release_notes-10-05.txt
View File

File diff suppressed because it is too large Load Diff

871
doc/release_notes-10-11.txt
View File

@@ -1,871 +0,0 @@
===============================================
Release notes for the Genode OS Framework 10.11
===============================================
Genode Labs
During the past three months, the Genode project was primarily driven by our
desire to create a bigger picture out of the rich set of components that we
introduced over time, in particular over the last year. Looking back at the
progress made since mid 2009, there were many functional additions to the
framework, waiting to get combined. To name a few, we added support for
networking, audio output, real-time priorities, mandatory access control,
USB, ATAPI block devices, Python, hardware-accelerated 3D graphics, Qt4,
the WebKit-based Arora browser, and the paravirtualized OKLinux kernel.
So many wonderful toys waiting to get played with. This is how the idea of
creating [http://genode.org/download/live-cds - the new Genode Live CD] was
born. In the past, Genode was mostly used in settings with a relatively static
configuration consisting of several components orchestrated to fulfill
a few special-purpose functions. Now, the time has come for the next step,
creating one dynamic setup that allows for the selection of different subsystems
at runtime rather than at boot time.
This step is challenging in several ways. First, the processes that form
the base system have to run during the entire time of all demo setups. If
any of those processes contained stability problems or leaked memory, it would
subvert the complete system. Second, the components of all subsystems combined
are far too complex to be loaded into memory at boot time. This would not
only take too long but would consume a lot of RAM. Instead, those components
and their data had to be fetched from disk (CDROM) on demand. Third, because
multiple demo subsystems can be active at a time, low-level resources such as
networking and audio output must be multiplexed to prevent different
subsystems from interfering with each other. Finally, we had to create a
single boot and configuration concept that is able to align the needs of all
demos, yet staying manageable.
Alongside these challenges, we came up with a lot of ideas about how Genode's
components could be composed in new creative ways. Some of these ideas such
as the browser-plugin concept and the http-based block server made it onto
the Live CD. So for producing the Live CD, we not only faced the said
technical challenges but also invested substantial development effort in new
components, which contributed to our overall goal. Two weeks ago, we released
the Live CD. This release-notes document is the story about how we got there.
To keep ourself focused on the mission described above, we deferred the
original roadmap goal for this release, which was the creation of a Unix-like
runtime environment to enable compiling Genode on Genode. This will be the
primary goal for the next release.
Execution environment for gPXE drivers
######################################
Up to now, DDE Linux provided Genode with drivers for hardware devices
ranging from USB HID to WLAN. In preparation of the live CD, we
noticed the demand for support of a broader selection of ethernet
devices. Intel's e1000 PCI and PCIe cards seemed to mark the bottom
line of what we had to support. The major advantage of NIC drivers
from Linux is their optimization for maximum performance. This emerges
a major downside if DDE Linux comes into play: We have to provide all
the nifty interfaces used by the driver in our emulation framework. To
achieve our short-term goal of a great live CD experience, we had to
walk a different path.
[http://gpxe.org/ - gPXE] is a lovely network boot loader / open-source
PXE ROM project and the successor of the famous Etherboot
implementation. Besides support for DNS, HTTP, iSCSI and AoE, gPXE
includes dozens of NIC drivers and applies a plain driver framework.
As we were also itching to evaluate DDE kit and the DDE approach at
large with this special _donator OS_, we went for implementing the
device-driver environment for gPXE (DDE gPXE).
The current version provides drivers for e1000, e1000e, and pcnet
devices. The emulation framework comprises just about 600 lines of
code compared to more than 22,000 LOC reused unmodified from gPXE.
Benchmarks with the PCNet32 driver showed that DDE gPXE's performance
is comparable to DDE Linux.
The gPXE driver environment comes in the form of the new 'dde_gpxe'
repository. For building DDE gPXE, you first need to download and patch
the original sources. The top-level makefile of this repository automates
this task. Just issue:
! make prepare
Now, you need to include the DDE gPXE repository into your Genode
build process. Just add the path to this directory to the
'REPOSITORIES' declaration of the 'etc/build.conf' file within your
build directory, for example
! REPOSITORIES += $(GENODE_DIR)/dde_gpxe
After successful build the DDE gPXE based ethernet driver is located
at 'bin/gpxe_nic_drv'.
On-demand paging
################
In the [http://genode.org/documentation/release-notes/8.11#section-8 - release 8.11],
we laid the foundation for implementing user-level dataspace managers.
But so far, the facility remained largely unused except for managing thread
contexts. This changed with this release.
So what is a user-level dataspace manager and who needs it? In short,
Genode's memory management is based on dataspaces. A dataspace is a
container for memory. Normally, it is created via core's RAM or ROM
services. The RAM service hands out dataspaces containing contiguous
physical memory. After allocating such a RAM dataspace, the creator can
attach the dataspace to its own address space to access the dataspace
content. In addition, it can pass a dataspace reference (called dataspace
capability) to other processes, which, in turn, can than attach the same
dataspace to their local address space, thereby establishing shared memory.
Similarly, core's ROM service hands out boot-time binary data as dataspaces.
For the most use cases of Genode so far, these two core services were the
only dataspace providers needed. However, there are use cases that require
more sophisticated memory management. For example, to implement swapping,
the content of a dataspace must be transferred to disk in a way that
is transparent to the users of the dataspace. In monolithic kernels, such
functionality is implemented in the kernel. But on a multi-server OS
such as Genode, this is no option. Implementing such a feature into
core would increase the trusted computing base of all applications
including those who do not need swapping. Core would need a hard-disk
driver, effectively subverting the Genode concept. Other examples for
advanced memory-management facilities are copy-on-write memory and
non-contiguous memory - complexity we wish to avoid at the root of the
process tree. Instead of implementing such memory management facilities
by itself, core provides a mechanism to let any process manage dataspaces.
This technique is also called user-level page-fault handling.
For the Live CD, we decided to give Genode's user-level page-fault handling
facility a go. The incentive was accessing files stored on CDROM in an
elegant way. We wanted to make the CDROM access completely transparent to
the applications. An application should be able to use a ROM session as
if the file was stored at core's ROM service. But instead of being
provided by core, the session request would be delegated to an
alternative ROM service implementation that reads the data from disk
as needed. Some of the files stored in the CDROM are large. For example,
the disk image that we use for the Linux demo is 160MB. So reading
this file at once and keeping it in memory is not an option. Instead, only
those parts of the file should be read from disk, which are actually
needed. To uphold the illusion of dealing with plain ROM files for
the client, we need to employ on-demand-paging in the CDROM server.
Here is how it works.
# The dataspace manager creates an empty managed dataspace. Core
already provides a tool for managing address spaces called
region manager (RM service). A RM session is an address space,
to which dataspaces can be attached. This is exactly what is
needed for a managed dataspace. So a dataspace manager uses the
same core service to define the layout of a managed dataspace
as is used to manage the address space of a process. In fact,
any RM session can be converted into a managed dataspace.
! enum { MANAGED_DS_SIZE = 64*1024*1024 };
! Rm_connection rm(0, MANAGED_DS_SIZE);
This code creates a RM session with the size of 64MB. This is an empty
address space. A dataspace capability that corresponds to this address
space can then be requested via
! Dataspace_capability ds = rm.dataspace();
# The dataspace capability can be passed to a client, which may
attach the dataspace to its local address space. Because the
managed dataspace is not populated by any backing store, however,
an access would trigger a page fault, halting the execution of
the client. Here, the page-fault protocol comes into play.
# The dataspace manager registers itself for receiving a signal each time
a fault occurs:
! Signal_receiver rec;
! Signal_context client;
! Signal_context_capability sig_cap = rec.manage(client);
! rm.fault_handler(sig_cap);
When an empty part of the managed dataspace is accessed by any
process, a signal is delivered. The dataspace manager can then
retrieve the fault information (access type, fault address) and
dispatch the page fault by attaching a real dataspace at the
fault address of the managed dataspace. In a simple case, the code
looks as follows:
! while (true) {
! Signal signal = rec.wait_for_signal();
! for (int i = 0; i < signal.num(); i++) {
! Rm_session::State state = rm.state();
! ds = alloc_backing_store_dataspace(PAGE_SIZE);
! rm.attach_at(ds, state.addr & PAGE_MASK);
! }
! }
This simple page-fault handler would lazily allocate a page of
backing store memory each time a fault occurs. When the backing
store is attached to the managed dataspace, core will automatically
wake up the faulted client.
# The example above has the problem that the dataspace manager has
to pay for the backing store that is indirectly used by the client.
To prevent the client from exhausting the dataspace manager's memory,
the dataspace manager may choose to use a limited pool of backing
store only. If this pool is exceeded, the dataspace manager can reuse
an already used backing-store block by first revoking it from its
current managed dataspace:
! rm.detach(addr);
This will flush all mappings referring to the specified address
from all users of the managed dataspace. The next time, this
address region is accessed, a new signal will be delivered.
This page-fault protocol has the following unique properties. First,
because core is used as a broker between client and dataspace manager, the
dataspace manager remains completely unaware of the identity of its client.
It does not even need to possess the communication right to the client. In
contrast, all other user-level page-fault protocols that we are aware of
require direct communication between client and dataspace manager. Second,
because dataspaces are used as first-level objects to resolve page faults,
page faults can be handed at an arbitrary granularity (of course, a multiple
of the physical page size). For example, a dataspace manager may decide to
attach backing-store dataspaces of 64K to the managed dataspace. So the
overhead produced by user-level page-fault handler can be traded for the
page-fault granularity. But most importantly, the API is the same across
all kernels that support user-level page fault handling. Thus the low-level
page-fault handling code becomes inherently portable.
Having said that, we have completed the implementation of the described
core mechanisms, in particular the 'detach' facility, for OKL4. The ISO9660
driver as featured on the Live CD implements the 'ROM' interface and
reads the contents of those files from CDROM on demand. It uses a
fixed pool of backing store, operates at a page-fault granularity of
64KB, and implements a simple fifo replacement strategy.
Base framework
##############
There had been only a few changes to the base framework described as
follows.
We unified the core-specific console implementation among all
base platforms and added synchronization of 'vprintf' calls.
The kernel-specific code resides now in the respective
'base-<platform>/src/base/console/core_console.h' files.
We removed the argument-less constructor from 'Allocator_avl_tpl'.
This constructor created an allocator that uses itself for
meta-data allocation, which is the usual case when creating
local memory allocators. However, on Genode, this code is typically
used to build non-memory allocators such as address-space regions.
For these use cases, the default policy is dangerous. Hence, we
decided to remove the default policy.
The 'printf' helper macros have been unified and simplified. The
available macros are 'PINF' for status information, 'PWRN' for warnings,
'PLOG' for log messages, and 'PERR' for errors. By default, the message
types are colored differently to make them easily distinguishable.
In addition to normal messages, there is the 'PDBG' for debugging
purposes. It remains to be the only macro that prints the function name
as message prefix and is meant for temporary messages, to be removed
before finalizing the code.
Genode's on-demand-paging mechanism relies on the signalling framework.
Each managed dataspace is assigned to a distinct signal context.
Hence, signal contexts need to be created and disposed alongside
with managed dataspaces. We complemented the signalling framework
with a 'dissolve' function to enable the destruction of signal
contexts.
Operating-system services and libraries
#######################################
Finished transition to new init concept
=======================================
With the release 10.05, we introduced the
[http://genode.org/documentation/release-notes/10.05#section-0 - current configuration concept of init].
This concept supports mandatory access control and provides flexible
ways for defining client-server relationships. Until now, we maintained
the old init concept. With the current release, the transition to the
new concept is finished and we removed the traditional init.
We retained the support for loading configurations for individual subsystems
from different files but adopted the syntax to the use of attributes.
Instead of
! <configfile>subsystem.config</configfile>
the new syntax is
! <configfile name="subsystem.config"/>
Virtual network bridge (Proxy ARP)
==================================
Since we originally added networking support to Genode, only one program could
use the networking facilities at a time. In the simplest form, such a program
included the network driver, protocol stack, and the actual application. For
example, the uIP stack featured with release 9.02 followed this approach.
In release 9.11 we added the 'Nic_session' interface to decouple the network
driver from the TCP/IP protocol stack. But the 1-to-1 relation between
application and network interface remained. With the current release, we
introduce the 'nic_bridge' server, which is able to multiplex the 'Nic_session'
interface.
The implementation is roughly based on the proxy ARP RFC 1027. At startup, the
'nic_bridge' creates a 'Nic_session' to the real network driver and, in turn,
announces a 'Nic' service at its parent. But in contrast to a network driver
implementing this interface, 'nic_bridge' supports an arbitrary number of
'Nic_sessions' to be opened. From the client's perspective, such a session
looks like a real network adaptor.
This way, it has become possible to run multiple TCP/IP stacks in
parallel, each obtaining a distinct IP address via DHCP. For example,
is has become possible to run multiple paravirtualized Linux kernels
alongside an lwIP-based web browser, each accessing the network via a
distinct IP address.
As a side effect for developing the 'nic_bridge', we created a set
of utilities for implementing network protocols. The utilities are
located at 'os/include/net' and comprise protocol definitions for
ethernet, IPv4, UDP, ARP, and DHCP.
Nitpicker GUI server
====================
Our work on the Live CD motivated several improvements of the Nitpicker
GUI server.
Alpha blending
~~~~~~~~~~~~~~
In addition to nitpicker's plain pixel buffer interface that is compatible
with a normal framebuffer session, each nitpicker session can now have
an optional alpha channel as well as an corresponding input-mask channel
associated. Both the alpha channel and the input mask are contained in the
same dataspace as the pixel buffer. The pixel buffer is followed by
the 8-bit alpha values, which are then followed by the input-mask values.
This way, the presence of an alpha channel does not interfere with the
actual pixel format. Each 8-bit input mask value specifies the user-input
policy for the respective pixel. If the value is zero, user input
referring to the pixel is not handled by the client but "falls through"
the view that is visible in the background of the pixel. This is typically
the case for drop shadows. If the input-mask value is '1', the input
is handled by the client.
With the input-mask mechanism in place, we no longer have a definitive
assignment of each pixel to a single client anymore. In principle, an
invisible client is able to track mouse movements by creating a full-screen
view with all alpha values set to '0' and all input-mask values set to '1'.
Once, the user clicks on this invisible view, the user input gets routed
to the invisible client instead of the actually visible view. This
security risk can be addressed at two levels:
* In X-Ray mode, nitpicker completely disables alpha blending
and the input-mask mechanism such that the user can identify the
client that is responsible for each pixel on screen.
* The use of the alpha channel is a session argument, which is specified
by nitpicker clients at session-creation time. Consequently, this
session argument is subjected to the policy of all processes involved
with routing the session request to nitpicker. Such a policy may permit
the use of an alpha channel only for trusted applications.
_Caution:_ The use of alpha channels implies read operations from
the frame buffer. On typical PC graphics hardware, such operations are
extremely slow. For this reason, the VESA driver should operate in
buffered mode when using alpha blending in Nitpicker.
Tinted views in X-Ray mode
~~~~~~~~~~~~~~~~~~~~~~~~~~
We added support for tinting individual clients or groups of clients
with different colors based on their label as reported at session-creation
time. By using session colors, nitpicker assists the user to tell apart
different security domains without reading textual information. In
addition to the tinting effect, the title bar presents the session
color of the currently focused session.
The following nitpicker configuration tints all views of the launchpad
subsystem in blue except for those views that belong to the testnit
child of launchpad. Those are tinted red.
! <config>
! <policy label="launchpad" color="#0000ff"/>
! <policy label="launchpad -> testnit" color="#ff0000"/>
! </config>
Misc Nitpicker changes
~~~~~~~~~~~~~~~~~~~~~~
We introduced a so-called 'stay-top' session argument, which declares
that views created via this session should stay on top of other views.
This function is useful for menus that should always remain accessible
or banner images as used for Live CD.
Nitpicker's reserved region at the top of the screen used to cover up
the screen area as seen by the clients. We have now excluded this area
from the coordinate system of the clients.
We implemented the 'kill' mode that can be activated by the 'kill' key.
(typically this is the 'Print Screen' key) This feature allows the user
to select a client to be removed from the GUI. The client is not
actually killed but only locked out. The 'kill' mode is meant as an
emergency brake if an application behaves in ways not wanted by the
user.
ISO9660 server
==============
As outlined in Section [On-demand paging], we revisited the ISO9660 server
to implement on-demand-paged dataspaces. It is the first real-world
use case for Genode's user-level page-fault protocol. The memory pool
to be used as backing store for managed dataspaces is dimensioned according
to the RAM assigned to the iso9660 server. The server divides this backing
store into blocks of 64KB and assigns those blocks to the managed dataspaces
in a fifo fashion. We found that using a granularity of 64KB improved the
performance over smaller block sizes because this way, we profit from reading
data ahead for each block request. This is particularly beneficial for
CDROM drives because of their extremely long seek times.
Audio mixer
===========
We added a new *channel synchronization* facility to the 'Audio_out_session'
interface. An 'Audio_out_session' refers to a single channel. For stereo
playback, two sessions must be created. At session-creation time, the
client can provide a hint about the channel type such as "front-left" as
session-construction argument. This design principally allows for supporting
setups with an arbitrary amount of channels. However, those channels must
be synchronized. For this reason, we introduced the 'sync_session' function
to the 'Audio_out_session' interface. It takes the session capability of
another 'Audio_out_session' as argument. The specified session is then
used as synchronization reference.
To reduce the latency when stopping audio replay, we introduced a new *flush*
function to the 'Audio_out_session' interface. By calling this function,
a client can express that it is willing to discard all audio data already
submitted to the mixer.
Furthermore, we improved the audio mixer to support both long-running
streams of audio and sporadic sounds. For the latter use case, low latency
is particularly critical. In this regard, the current implementation is a
vast improvement over the initial version. However, orchestrating the
mixer with audio drivers as well as with different clients (in particular
ALSA programs running on a paravirtualized Linux) is not trivial. In the
process, we learned a lot, which will eventually prompt us to further
optimize the current solution.
Nitpicker-based virtual Framebuffer
===================================
To support the browser-plugin demo, we introduced 'nit_fb', which is a
framebuffer service that uses the nitpicker GUI server as back end. It
is similar to the liquid framebuffer as featured in the 'demo' repository
but in contrast to liquid framebuffer, 'nit_fb' is non-interactive.
It has a fixed screen position and size. Furthermore, it does not
virtualize the framebuffer but passes through the framebuffer portion of
the nitpicker session, yielding better performance and lower latency.
If instantiated multiple times, 'nit_fb' can be used to statically arrange
multiple virtual frame buffers on one physical screen. The size and screen
position of each 'nit_fb' instance can be defined via Genode's configuration
mechanism using the following attributes of the 'nit_fb' config node:
! <config xpos="100" ypos="150"
! width="300" height="200"
! refresh_rate="25"/>
If 'refresh_rate' isn't set, the server will not trigger any refresh
operations by itself.
On the Live CD, each browser plugin instantiates a separate instance of
'nit_fb' to present the plugin's content on screen. In this case, the
view position is not fixed because the view is further virtualized by the
loader, which imposes its policy onto 'nit_fb' - Genode's nested
policies at work!
TAR ROM service
===============
For large setups, listing individual files as boot modules in single-image
creation tools (e.g., elfweaver) or multiboot boot loaders can be
cumbersome, especially when many data files or shared libraries are
involved. To facilitate the grouping of files, 'tar_rom' is an
implementation of the 'ROM' interface that operates on a 'tar' file.
The name of the TAR archive must be specified via the 'name' attribute of
an 'archive' tag, for example:
! <config>
! <archive name="archive.tar"/>
! </config>
The backing store for the dataspaces exported via ROM sessions is accounted
on the 'rom_tar' service (not on its clients) to make the use of 'rom_tar'
transparent to the regular users of core's ROM service. Hence, this service
must not be used by multiple clients that do not trust each other.
Typically, 'tar_rom' is instantiated per client.
The Live CD uses the 'tar_rom' service for the browser demo. Each plugin
is fetched from the web as a tar file containing the config file of the
plugin subsystem as well as supplemental binary files that are provided
to the plugin subsystem as ROM files. This way, a plugin can carry along
multiple components and data that form a complete Genode subsystem.
DDE Kit
=======
The DDE kit underwent slight modifications since the previous release.
It now provides 64-bit integer types and a revised virtual PCI bus
implementation.
Device drivers
##############
PCI bus
=======
Genode was tested on several hardware platforms in preparation of the
current release. This revealed some deficiencies with the PCI bus
driver implementation. The revised driver now efficiently supports
platforms with many PCI busses (as PCIe demands) and correctly handles
multi-function devices.
VESA framebuffer
================
We updated the configuration syntax of the VESA driver to better match
the style of new init syntax, preferring the use of attributes rather than
XML sub nodes. Please refer to the updated documentation at
'os/src/drivers/framebuffer/vesa/README'.
:Buffered output:
To accommodate framebuffer clients that need to read from the frame buffer,
in particular the nitpicker GUI server operating with alpha channels, we
introduced a buffered mode to the VESA driver. If enabled, the VESA driver
will hand out a plain memory dataspace to the client rather than the
physical framebuffer. Each time, the client issues as 'refresh' operation
on the framebuffer-session interface, the VESA driver copies the corresponding
screen region from the client-side virtual framebuffer to the physical
framebuffer. Note that the VESA driver will require additional RAM quota
to allocate the client buffer. If the quota is insufficient, the driver will
fall back to non-buffered output.
:Preinitialized video modes:
As an alternative to letting the VESA driver set up a screen mode, the
driver has become able to reuse an already initialized mode, which is useful
if the VESA mode is already initialized by the boot loader. If the screen
is initialized that way, the 'preinit' attribute of the 'config' node can
be set to '"yes"' to prevent the driver from changing the mode. This way,
the driver will just query the current mode and make the already
initialized framebuffer available to its client.
Audio
=====
We observed certain hardware platforms (in particular VirtualBox) to
behave strangely after ALSA buffer-underrun conditions. It seems that the
VirtualBox audio driver plays actually more frames than requested by
ALSA's 'writei' function, resulting in recurring replay of data that
was in the buffer at underrun time. As a work-around for this problem,
we zero-out the sound-hardware buffer in the condition of an ALSA buffer
underrun. This way, the recurring replay is still there, but it is
replaying silence.
To improve the support for sporadic audio output, we added a check for the PCM
state for buffer underruns prior issuing the actual playback. In the event of
an underrun, we re-prepare the sound card before starting the playback.
Furthermore, we implemented the new flush and channel-synchronization
abilities of the 'Audio_out_session' interface for the DDE Linux driver.
Paravirtualized Linux
#####################
To support the demo scenarios that showcase the paravirtualized Linux kernel,
we enhanced our custom stub drivers of the OKLinux kernel. Thereby, we have
reached a high level of integration of OKLinux with native Genode services,
including audio output, block devices, framebuffer output, seamless integration
with the Nitpicker GUI, and networking. All stub drivers are compiled in by
default and are ready to use by specifying a device configuration in the config
node for the Linux kernel. This way, one Linux kernel image can be easily used
in different scenarios.
:Integration with the Nitpicker GUI:
We enhanced our fbdev stub driver with a mechanism to merge view reposition
events. If a X11 window is moved, a lot of subsequent events of this type are
generated. Using the new optimization, only the most recent state gets
reported to Nitpicker, making the X11 GUI more responsive.
:UnionFS:
As we noticed that unionfs is required by all our Linux scenarios, we decided
to include and enable the patch by default.
:Network support:
With the introduction of the 'nic_bridge', multiple networking stacks can run
on Genode at the same time, which paves the way for new use cases. We have now
added a stub driver using Genode's 'Nic_session' interface to make the new
facility available to Linux.
:Audio output:
We adapted the ALSA stub driver to the changes of the 'Audio_out_session'
interface, using the new channel synchronization and flush functions.
Thereby, we optimized the stub driver to keep latency and seek times of
Linux userland applications reasonably low.
:Removed ROM file driver:
With the addition of the 'Block_session' stub driver, the original ROM file
driver is no longer required. So we removed the stub. For using ROM files as
disk images for Linux, there is the 'rom_loopdev' server, which provides a
block session that operates on a ROM file.
:Asynchronous block interface:
To improve performance, we changed the block stub driver to facilitate the
asynchronous mode of operation as provided by the 'Block_session' interface.
This way, multiple block requests can be issued at once, thereby shadowing
the round trip times for individual requests.
Protocol stacks and libraries
#############################
Gallium3D / Intel GEM
=====================
We improved the cache handling of our DRM emulation code (implementing
'drm_clflush_pages') and our EGL driver, thereby fixing caching
artifacts on i945 GPUs. Furthermore, we added a temporary work-around
for the currently dysfunctional sequence-number tracking with i945 GPUs.
On this chipset, issuing the 'MI_STORE_DWORD_INDEX' GPU command used
for tracking sequence numbers apparently halts the processing the command
stream. This condition is normally handled by an interrupt. However,
we have not enabled interrupts yet.
To prepare the future support for more Gallium drivers than i915, we
implemented a driver-selection facility in the EGL driver. The code
scans the PCI bus for a supported GPU and returns the name of the
corresponding driver library. If no driver library could be found,
the EGL driver falls back to softpipe rendering.
lwIP
====
We revised our port of the lwIP TCP/IP stack, and thereby improved its
stability and performance.
* The lwIP library is now built as shared object, following the convention
for libraries contained in the 'libports' repository.
* By default (when using the 'libc_lwip_nic_dhcp' library), lwIP will
issue a DHCP request at startup. If this request times out, the loopback
device is set as default.
* If there is no 'Nic' service available, the lwIP stack will fall back to
the loopback device.
* We increased the default number of PCBs in lwIP to 64.
* We removed a corner case of the timed semaphore that could occur
when a timeout was triggered at the same time ,'up' was called.
In this case, the semaphore was unblocked but the timeout condition
was not reflected at the caller of 'down'. However, the lwIP code
relies on detecting those timeouts.
Qt4
====
We implemented a custom *nitpicker plugin widget*, which allows for the
seamless integration of arbitrary nitpicker clients into a Qt4 application.
The primary use case is the browser plugin mechanism presented at
the Live CD. In principle, the 'QNitpickerViewWidget' allows for creating
mash-up allocations consisting of multiple native Genode programs. As shown
by the browser plugin demo, a Qt4 application can even integrate other
programs that run isolated from the Qt4 application, and thereby depend on
on a significantly less complex trusted computing base than the Qt4
application itself.
[image nitpicker_plugin]
The image above illustrates the use of the 'QNitpickerViewWidget' in the
scenario presented on the Live CD. The browser obtains the Nitpicker view to be
embedded into the website from the loader service, which virtualizes the
Nitpicker session interface for the loaded plugin subsystem. The browser then
tells the loader about where to present the plugin view on screen. But it has
neither control over the plugin's execution nor can it observe any user
interaction with the plugin.
New Gems repository with HTTP-based block server
################################################
To give the web-browser demo of our Live CD a special twist, and to show off
the possibilities of a real multi-server OS, we decided to implement the
somewhat crazy idea of letting a Linux OS run on a disk image fetched at
runtime from a web server. This way, the Linux OS would start right away and
disk blocks would be streamed over the network as needed. Implementing this
idea was especially attractive because such a feature would be extremely hard
to implement on a classical OS but is a breeze to realize on Genode where all
device drivers and protocol stacks are running as distinct user-level
components. The following figure illustrates the idea:
[image http_block]
The block stub driver of the Linux kernel gets connected to a special block
driver called 'http_block', which does not access a real block device but
rather uses TCP/IP and HTTP to fetch disk blocks from a web server.
Because the 'http_block' server is both user of high-level functionality (the
lwIP stack) and provider of a low-level interface ('Block_session'), the
program does not fit well into one of the existing source-code repositories.
The 'os' repository, which is normally hosting servers for low-level interfaces
is the wrong place for 'http_block' because this program would make the 'os'
repository depend on the higher-level 'libports' repository where the 'lwip'
stack is located. On the other hand, placing 'http_block' into the 'libports'
repository is also wrong because the program is not a ported library. It merely
uses libraries provided by 'libports'. In the future, we expect that native
Genode components that use both low-level and high-level repositories will
become rather the norm than an exception. Therefore, we introduced a new
repository called 'gems' for hosting such programs.
Tools
#####
Automated coding-style checker
==============================
As Genode's code base grows and new developers start to get involved,
we noticed recurring questions regarding coding style. There is a
[http://genode.org/documentation/developer-resources/coding_style - document]
describing our coding style but for people just starting to get involved,
adhering all the rules can become tedious. However, we stress the importance
of a consistent coding style for the project. Not only does a consistent style
make the framework more approachable for users, but it also eases the work
of all regular developers, who can feel right at home at any part of
the code.
To avoid wasting precious developer time with coding-style fixes, we
have created a tool for the automated checking and (if possible) fixing
the adherence of source code to Genode's coding style. The tool is
located at 'tool/beautify'. It takes a source file as argument and
reports coding-style violations. The checks are fairly elaborative:
* Placement of braces and parenthesis
* Indentation and alignment, trailing spaces
* Vertical spacing (e.g., between member functions, above comments)
* Naming of member variables and functions (e.g., private members start with '_')
* Use of upper and lower case
* Presence of a file header with the mandatory fields
* Policy for function-header comments (comment at declaration, not
at implementation)
* Style of single-line comments, function-header comments, multi-line comments
The user of 'beautify' may opt to let the tool fix most of the violations
automatically by specifying the command line arguments '-fix' and '-write'.
With only the '-fix' argument, the tool will output the fixed version of
the code via stdout. By specifying the '-write' argument, the changes will
be written back to the original file. In any case, we strongly recommend
to manually inspect all changes made by the tool.
Under the hood, the tool consists of two parts. A custom C++ parser called
'parse_cxx' reads the source code and converts it to a syntax tree. In the
syntax tree, all formating information such as whitespaces are preserved.
The C++ parser is a separate command-line tool, which we also use for
other purposes (e.g., generating the API documentation at the website).
The actual 'beautify' tool calls 'parse_cxx', and applies its checks and
fixes to the output of 'parse_cxx'. For this reason, both tools have to
reside in the same directory.
Platform-specific changes
#########################
OKL4
====
:Added support for shared interrupts:
The Genode Live CD operates on a large number of devices that trigger
interrupts (USB, keyboard, mouse, ATAPI, timer, network). On most
platforms, the chances are extremely high that some of them use
the same IRQ line. Therefore, we enhanced core's IRQ service to
allow multiple clients to request the same IRQ. If the interrupt occurs,
all clients referring to this interrupt are notified. The interrupt
gets cleared after all of those clients responded. Even though, we regard
PIC interrupts as a legacy, the support of shared interrupts enables
us to use OKL4 with such complex usage scenarios.
:Revised page-fault handling:
If a page fault occurs, the OKL4 kernel delivers a message to the page-fault
handler. The message contains the page-fault address and type as well as the
space ID where the fault happened. However, the identity of the faulting
thread is not delivered. Instead, the sender ID of the page fault message
contains the KTCB index of the faulting thread, which is only meaningful
within the kernel. This KTCB index is used as a reply token for answering the
page fault message. We wondered about why OKL4 choose to deliver the KTCB
index rather then the global thread ID as done for plain IPC messages. The
only reasonable answer is that by using the KTCB index directly in OKL4's
page-fault protocol, one lookup from the userland-defined thread ID to the
KTCB index can be avoided. However, this comes at the cost of losing the
identity of the faulting thread. We used to take the space ID as a key for
the fault context within core. However, with Genode's user-level page-fault
mechanism, this simplification does not suffice anymore. We have to know the
faulting thread as a page fault may not be answered immediately but at a
later time. During that time, the page-fault state has to be stored at core's
representation of the faulting thread. Our solution is reverting OKL4's
page-fault protocol to operate with global thread IDs only and to never make
kernel-internal KTCB indices visible at the user land. You can find the patch
for the OKL4 kernel at 'base-okl4/patches/reply_tid.patch'.
:Reboot via kernel debugger:
We fixed the reboot code of OKL4's kernel debugger to improve our work
flow. The patch can be found at 'base-okl4/patches/kdb_reboot.patch'.
:Relieved conflict with libc 'limits.h':
For some reason, the OKL4 kernel bindings provide definitions
normally found in libc headers. This circumstance ultimately leads
to trouble when combining OKL4 with a real C runtime. We have
relieved the problem with the patch 'base-okl4/patches/char_bit.patch'.
:Exception handling:
We added a diagnostic message to core that reports about exceptions
such as division by zero.
Pistachio
=========
Our revised syscall bindings for supporting position-independent code
on L4ka::Pistachio have been integrated into the mainline development
of the kernel. Therefore, the patch is not needed anymore when using
a kernel revision newer than 'r791:0d25c1f65a3a'.
Linux
=====
On Linux, we let the kernel manage all virtual address spaces for us,
except for the thread-context area. Because the kernel does not know
about the special meaning of the thread-context area, it may choose to
use this part of the virtual address space as target for 'mmap'. This
may lead to memory corruption. Fortunately, there is a way to tell the
kernel about virtual address regions that should be reserved. The
trick is to pre-populate the said region with anonymous memory using
the 'mmap' arguments 'MAP_PRIVATE', 'MAP_FIXED', 'MAP_ANONYMOUS', and
'PROT_NONE'. The kernel will still accept a fixed-address mapping
within such a reserved region (overmap) but won't consider using the
region by itself. The reservation must be done at the startup of each
process and each time when detaching a dataspace from the thread
context area. For the process startup, we use the hook function
'main_thread_bootstrap' in 'src/platform/_main_helper.h'. For reverting
detached dataspaces to a reserved region within the context area, we
added as special case to 'src/base/env/rm_session_mmap.cc'.
For hybrid programs (Genode processes that link against native
shared libraries of the Linux system), which are loaded by the dynamic
linker of Linux, we must further prevent the dynamic linker from
populating the thread-context area. This is achieved by adding a
special program segment at the linking stage of all elf binaries.

876
doc/release_notes-11-02.txt
View File

@@ -1,876 +0,0 @@
===============================================
Release notes for the Genode OS Framework 11.02
===============================================
Genode Labs
One year ago, the release 10.02 was our break-through with regard to the support
of multiple kernels as base platform for Genode. With the added support for
the NOVA hypervisor and the Codezero kernel, Genode applications could be executed
on 6 different kernels. With the current release, we take our commitment to
kernel platform support even further. With the added support for the Fiasco.OC
kernel, we make Genode available on one of the most feature-rich modern microkernels.
Additionally, we entered the realms of kernel design with our new platform support
for the Xilinx MicroBlaze architecture. This platform support comes in the shape
of a custom kernel specifically targeted to the MicroBlaze CPU architecture.
Furthermore, we updated our support for the NOVA Hypervisor to the bleeding-edge
version 0.3, which has been released earlier this month.
With the current support for 8 different kernel platforms (L4/Fiasco, Linux,
L4ka::Pistachio, OKL4, NOVA, Codezero, Fiasco.OC, and native MicroBlaze), testing
and integrating application scenarios across all platforms becomes increasingly
challenging. Therefore, we introduce a new framework for automating such tasks.
Thanks to the tight integration of the automation tool with Genode's build system,
going back and forth between different kernels becomes an almost seamless
experience.
Functionality-wise, the release carries on our vision to create a highly secure
yet easy to use general-purpose operating system. Because the Genode framework
is developed on Linux using the wonderful GNU tools, we consider the
availability of the GNU user land on Genode as crucial for using the system by
ourself. This motivation drives the creation of a custom execution environment
for GNU software on top of Genode. With the current release, we are proud to
present the first pieces of this execution environment. Even though not fully
functional yet, it clearly shows the direction of where we are heading.
Support for Fiasco.OC
#####################
The OC in the name of the Fiasco.OC kernel stands for "object capability", hinting
at the most significant feature that sets current-generation microkernels such as
NOVA, seL4, and Fiasco.OC apart from their predecessors. Whereas previous L4 kernels
succeeded in protecting subsystems from each other, the new generation of kernels
is geared towards strict security policies. Traditionally, two protection domains
were able to communicate with each other if they both agreed. Communication partners
were typically globally known via their respective thread/task IDs. Obviously, this
policy is not able to guarantee the separation of subsystems. If two subsystems
conspire, they could always share information. Object-capability-based kernels
are taking the separation much further by prohibiting any communication between
protection domains by default. Two protection domains can communicate only if
a common acquaintance of both agrees. This default-deny policy facilitates the
creation of least-privilege security policies. From the ground up, Genode has
been designed as a capability-based system which is naturally capable of leveraging
kernel-based object-capability support if present. After NOVA, Fiasc.OC is the
second of Genode's base platforms that provides this feature.
Apart from being a capability-based kernel, Fiasco.OC has a number of compelling
features such as thorough support for ARM platforms and the x86 32/64 bit
architectures. It supports SMP, hardware virtualization, and provides special
optimizations for running paravirtualized operating systems.
Technically, Fiasco.OC is the successor of the L4/Fiasco kernel developed by
the OS group of the TU-Dresden. However, the kernel interface of Fiasco.OC has
not much in common with L4/Fiasco. Some heritages are still there (e.g., IPC
timeouts) but the kernel API has evolved to a fully object-oriented model.
:Thanks:
We are indebted to the main developer of Fiasco.OC Alexander Warg for being
very reponsive to our inquiries while doing the porting work. Thanks to his
support, the adaptation of Genode to this kernel has been an almost smooth
ride.
Prerequisites
=============
You need GNU C & C++ Compilers, GNU Binutils, GNU Make, and Perl to use the
Fiasco.OC build system. On Debian/Ubuntu systems, you have to install the
following packages:
! apt-get install make gawk g++ binutils pkg-config subversion
Moreover, you need to download and install the tool-chain used by Genode. Have
a look at this page:
:[http://genode.org/download/tool-chain]:
Genode tool-chain
Downloading and building Fiasco.OC
==================================
Checkout the Fiasco.OC sources and tool-chain to an appropriated directory:
! export REPOMGR_SVN_REV=27
! svn cat http://svn.tudos.org/repos/oc/tudos/trunk/repomgr |\
! perl - init http://svn.tudos.org/repos/oc/tudos fiasco l4re
Building the kernel
~~~~~~~~~~~~~~~~~~~
Create the build directory for the kernel:
! cd <path_to_fiasco_src_dir>/src/kernel/fiasco
! make BUILDDIR=<path_to_kernel_build_dir>
Go to the build directory, configure the kernel:
! cd mybuild
! make config
This will launch the configuration menu. Here you can configure your kernel.
The default config is just fine to test the Genode port. It will build a
uniprocessor IA32 kernel with debugging features enabled. You can exit the menu and
save the configuration by simply typing 'x'.
Now, build Fiasco.OC by invoking:
! make
Building necessary tools
~~~~~~~~~~~~~~~~~~~~~~~~
To practically use Fiasco.OC, you need in addition to the kernel a tool to
bootstrap it, and the initial pager of the system, namely 'sigma0'. Both tools
can be found in the L4 runtime environment's base directory. Outgoing from
the directory where you checked out the sources, you have to change to the
following directory:
! cd <path_to_fiasco_src_dir>/src/l4
Create another build directory:
! make B=<path_to_l4re_build_dir>
Again, you might want to tweak the configuration:
! make O=<path_to_l4re_build_dir> config
Finally, build the tools:
! make O=<path_to_l4re_build_dir>
Building the Fiasco.OC version of Genode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The Fiasco.OC version of Genode is available at the Genode public subversion repository:
:http://genode.org/download/subversion-repository:
Information about accessing the Genode public subversion repository
Go to a directory where you want the Genode/Fiasco.OC build directory to remain. Use
the helper script in the 'tool/builddir' directory of the Genode source tree to
create the initial build environment. You need to state the absolute path to the
build directory of the L4 runtime environment as 'L4_DIR', as it contains the kernel
bindings needed by the Genode port.
! <path_to_genode_src_dir>/tool/builddir/create_builddir foc_x86_32 \
! L4_DIR=<path_to_l4re_build_dir> \
! GENODE_DIR=<path_to_genode_src_dir> \
! BUILD_DIR=<path_to_genode_build_dir>
Now, go to the newly created build directory and type make.
! cd <path_to_genode_build_dir>
! make
Booting Genode on top of Fiasco.OC
==================================
Example GRUB configuration entry:
! timeout 0
! default 0
!
! title Genode on Fiasco.OC
! kernel /bootstrap -modaddr=0x01100000
! module /fiasco -serial_esc
! module /sigma0
! module /core
! module /init
! module /config
! module /pci_drv
! module /vesa_drv
! module /ps2_drv
! module /timer
! module /nitpicker
! module /launchpad
! module /liquid_fb
! module /scout
! module /testnit
! module /nitlog
For an example of a matching Genode 'config' file, please take a look
at 'os/config/demo'.
The Genode binaries are located in '<path_to_genode_build_dir>/bin',
the 'fiasco' kernel in '<path_to_kernel_build_dir>'. Assuming you compiled
for x86/586 (the default), you can find the 'bootstrap' binary in
'bin/x86_586' and 'sigma0' in 'bin/x86_586/l4f' within the
'<path_to_l4re_build_dir>' directory.
Current state
=============
The adaptation of Genode to Fiasco.OC covers most parts of the Genode API
including advanced semantics such as cancelable locks and support for
real-time priorities. So far, it has been tested on the x86 architecture.
Because 'base-foc' does not contain x86-specific code, we expect no major
roadblocks for running Genode on Fiasco.OC on ARM. However, we have not
exercised tests in this regard.
As of today, there exist the following limitations of the Fiasco.OC support:
* The dynamic linker is not yet adapted to Fiasco.OC. Special care must
be taken for handling the parent capability for dynamically loaded
programs. We have already covered this issue for the NOVA version but
the adaptation to Fiasco.OC remains yet to be done.
* The destruction of sub systems is not yet fully stable. Because Genode
forms a more dynamic workload than the original userland accompanied with
the kernel, the usage pattern of the kernel API triggers different
effects. We are working with the Fiasco.OC developers to remedy this
issue.
* The signalling framework is not yet supported. A design exist but it is
not implemented yet.
We believe however that none of these limitations are a significant hurdle for
starting to use Genode with this kernel. Please expect this issues to be
resolved with the upcoming Genode release.
Technical details about 'base-foc'
==================================
The following technical bits are worth noting when exploring the use of
Genode with the 'base-foc' platform.
* The timer implementation uses a one thread-per-client mode of operation.
We use IPC timeouts as time source. Hence, the timer driver is hardware
independent and should work out of the box on all hardware platforms
supported by Fiasco.OC.
* Each 'Server_object' of Genode corresponds to a so-called IPC gate,
which is the Fiasco.OC kernel object used for capability invocation.
Therefore, protection and object integrity is provided at the fine
granularity of single 'Server_objects'. This is in line with our
support for NOVA's implementation of capability-based security.
* In contrast to the lock implementation that we used with the original
L4/Fiasco kernel, the 'base-foc' lock is a fully-featured Genode lock
with support for lock cancellation and blocking. For blocking and
waking up lock applicants, we use Fiasco.OC's IRQ objects.
* The allocator used for managing process-local capability selectors
does not yet support the reuse of capability selectors.
Further Information
===================
:genode/tool/builddir/README:
Reference manual for the 'create_builddir' script
:[http://os.inf.tu-dresden.de/fiasco]:
Official website for the Fiasco.OC microkernel.
Noux - an execution environment for the GNU userland
####################################################
Even though Genode is currently mainly geared to the classical special-purpose
application domains for microkernel-based systems, the main property that sets
Genode apart from traditional systems is the thorough support for dynamic
workloads and the powerful mechanisms for handling hardware resources and
security policies in highly dynamic setting. We are convinced that Genode's
architecture scales far beyond static special-purpose domains and believe in
the feasibility of Genode as a solid foundation for a fully-fledged general
purpose operating system. Internally at Genode Labs, we set up the ultimate
goal to switch from Linux to Genode for our day-to-day work. We identified
several functionalities that we could not live without and systematically try
to bring those features to Genode. Of course, the most fundamental programs
are the tools needed to develop and build Genode. Currently we are developing
on Linux and enjoy using the GNU userland.
Consequently, we require a solution for using this rich tool set on Genode.
The straight-forward way for making these tools available on Genode would be
running them within a virtualized Linux instance (e.g., using OKLinux on OKL4).
However, this approach would defeat our actual goal to create a highly secure
yet easy to use working environment because adding Linux to the picture would
involve administering the virtualized Linux system. We would prefer a native
solution that makes the overall system less, not more, complicated. This way
the idea for a native execution environment for the GNU userland on Genode
was born. The implementation is called Noux and the first bits of code are
featured in the 'ports' repository. Noux consists of two parts, a build
environment for compiling GNU programs such that they can be run as Genode
processes and an execution environment that provides the classical UNIX
functionality to these programs.
Noux build environment
======================
From our experience, porting existing UNIX applications to a non-UNIX system
tends to be a task of manual and time-consuming labour. One has to loosely
understand the build system and the relationship of the involved source codes,
implement dummy functions for unresolved references, and develop custom glue
code that interfaces the ported application to the actual system. Taking the
shortcut of changing the original code has to be avoided at any cost because
this produces recurring costs in the future when updating the application. In
short, this long-winding process does not scale. For porting a tool set such as
the GNU userland consisting of far more than a three-digit number of individual
programs, this manual approach becomes unfeasible. Therefore, we have created
a build environment that facilitates the use of the original procedure of
invoking './configure && make'. The challenge is to supply configure with
the right arguments and environment variables ('CFLAGS' and the like) such that
the package is configured against the Genode environment. The following
considerations must be taken:
* Configure must not detect any global headers (e.g., '/usr/include/')
or libraries (e.g., '/usr/lib/'). This can be achieved by the '-nostdinc' and
'-nostdlib' options
* Configure has to use the same include-search paths as used for compiling
normal libc-using Genode programs
* Configure must use the Genode tool chain
* The final linking stage must use the Genode linker script, the Genode
base libraries, and other Genode-specific linker arguments.
Thanks to the power of the GNU build system, all this can be achieved by
supplying arguments to './configure' and indirectly to the 'make' process via
environment variables. The new Noux build environment takes care of these
precautions. It comes in the form of the 'ports/mk/noux.mk' file which enables
the seamless integration of GNU packages with the Genode build system. To
compile a GNU package, the manual steps needed are reduced to the creation of a
'target.mk' file representing the package. This 'target.mk' defines the name
of the package (by default, the basename of the 'target.mk' enclosing directory
is assumed) and the location of the source package. With this approach, we
managed to build 'coreutils' (over 100 small UNIX utilities such as 'ls', 'cp',
'sort'), 'binutils' (GNU linker, assembler, object-file tools), 'findutils'
('find', 'xargs'), 'bash', 'dash', GNU make, and finally the GNU compiler
collection including 'g++'. The resulting binaries are ready to be executed as
native Genode processes. However, without the right environment that presents
the program the needed UNIX functionality, those programs won't do much.
This leads us to the Noux execution environment.
Noux execution environment
==========================
The Noux execution environment plays the role of a UNIX kernel for programs
built via the Noux build environment. In contrast to a real kernel, the Noux
environment is a plain Genode user-level process that plays the role of being
the parent of one or multiple Noux processes. In addition of providing the
'Genode::Parent' interface, Noux also provides a locally implemented service called
'Noux::Session' that offers UNIX-like system-calls via an RPC interface. Each
hosted program is linked against a special Noux libc plugin that catches all
libc calls that would normally result in a system call. It then transparently
forwards this function call to the 'Noux::Session' interface.
Currently the Noux execution environment implements the following
system calls: 'getcwd', 'write', 'stat', 'fstat', 'fcntl', 'open',
'close', 'dirent', 'fchdir', 'read', and 'execve'.
The execution environment submits arguments (argc, argv, environment) to the
hosted program, manages its current working directory and receives its exit
code. File operations are targeted to a custom VFS infrastructure, which
principally allows a flexible configuration of the virtual file system visible
to the hosted programs. At the current stage, Noux supports mounting plain tar
archives obtained from core's ROM service as read-only file system. On startup,
the Noux environment starts one process (the init process) and connects the
file descriptor 1 (stdout) to Genode's LOG service.
State of the implementation
~~~~~~~~~~~~~~~~~~~~~~~~~~~
The infrastructure implemented so far already allows the execution of many simple
UNIX tools such as 'ls -lRa', 'echo', 'seq', 'find'. The 'execve' system call
is implemented such that a new process is started that inherits the file
descriptors and the PID of the calling process. This allows using the exec
functionality of the 'bash' shell. However, because 'fork' is not implemented
yet, there is currently no way to start multiple programs hosted in a single
Noux execution environment.
As of today, the Noux environment is not considered to be usable for practical
purposes. However, it clearly shows the feasibility of the path we are walking.
With the foundation laid, we are looking forward to expanding Noux to a capable
solution for running our beloved GNU userland tools on Genode.
Vision
~~~~~~
The most significant intermediate result of pursuing the development of Noux is
the realization that such an environment is not exceedingly complex. Because of
the combination with Genode, we only need to provide a comfortable runtime as
expected by user processes but we can leave much of intricate parts of UNIX out
of the picture. For example, because we handle device drivers on Genode, we do
not need to consider device-user interaction in Noux. As another example,
because the problem of bootstrapping the OS is already solved by Genode, there
is no need to run an 'init' process within Noux. Our vision foresees that Noux
runtimes are to be created on demand for individual tasks such as editing a
file (starting a custom Noux instance containing only the file to edit and the
text editor), compiling source code (starting a custom Noux instance with only
the source code and the build tools). Because Noux is so simple, we expect the
runtime overhead of starting a Noux instance to be not more than the time
needed to spawn a shell in a normal UNIX-like system.
Test drive
~~~~~~~~~~
To give Noux a spin, we recommend using Linux as base platform as this is
the platform we use for developing it. First, you will need to download the
source code of the GNU packages. From within the 'ports' repository,
use the following command:
! make prepare PKG=coreutils
This command will download the source code of the GNU coreutils. You may
also like to give the other packages a try. To see what is available,
just call 'make' without any argument.
Create a build directory (e.g., using tool/builddir/create_builddir).
Change to the build directory and issue the command
! make run/noux
This command will execute the run script provided at 'ports/run/noux.run'.
First it builds core, init, and coreutils. Then it creates a tar archive
containing the installed coreutils. Finally, it starts the Noux environment on
Genode. Noux then mounts the TAR archive as file system and executes 'ls -laR',
showing the directory tree.
Approaching platform support for Xilinx MicroBlaze
##################################################
With the release 11.02, we are excited to include the first version of our
custom platform support for the Xilinx MicroBlaze CPU architecture. MicroBlaze
is a so-called softcore CPU, which is commonly used as part of FPGA-based
System-on-Chip designs. At Genode Labs, we are regularly using this IP core,
in particular for our Genode FPGA Graphics Project, which is a GUI software stack
and a set of IP cores for implementing fully-fledged windowed GUIs on FPGAs:
:Website of the Genode FPGA Graphics Project:
[http://genode-labs.com/products/fpga-graphics]
Ever since we first released the Genode FPGA project, we envisioned to combine
it with the Genode OS Framework. In Spring 2010, Martin Stein joined our team
at Genode Labs and accepted the challenge to bring the Genode OS Framework to
the realms of FPGA-based SoCs. Technically, this implies porting the framework
to the MicroBlaze CPU architecture. In contrast to most softcore CPUs such as
the popular Lattice Mico32, the MicroBlaze features a MMU, which is a fundamental
requirement for implementing a microkernel-based system. Architecturally-wise
MicroBlaze is a RISC CPU similar to MIPS. Many system parameters of the CPU
(caches, certain arithmetic and shift instructions) can be parametrized at
synthesizing time of the SoC. We found that the relatively simple architecture
of this CPU provides a perfect playground for pursuing some of our ideas about
kernel design that go beyond the scope of current microkernels. So instead of
adding MicroBlaze support into one of the existing microkernels already
supported by Genode, we went for a new kernel design. Deviating from the typical
microkernel, which is a self-sufficient program running in kernel mode that
executes user-level processes on top, our design regards the kernel as a part of
Genode's core. It is not a separate program but a library that implements the
glue between user-level core and the raw CPU. Specifically, it provides the
entrypoint for hardware exceptions, a thread scheduler, an IPC mechanism, and
functions to manipulate virtual address spaces (loading and flushing entries
from the CPU's software-loaded TLB). It does not manage any physical memory
resources or the relationship between processes. This is the job of core.
From the kernel-developer's point of view, the kernel part can be summarized as
follows:
* The kernel provides user-level threads that are scheduled in a round-robin
fashion.
* Threads can communicate via synchronous IPC.
* There is a mechanism for blocking and waking up threads. This mechanism
can be used by Genode to implement locking as well as asynchronous
inter-process communication.
* There is a single kernel thread, which never blocks in the kernel code paths.
So the kernel acts as a state machine. Naturally, there is no concurrency in the
execution paths traversed in kernel mode, vastly simplifying these code parts.
However, all code paths are extremely short and bounded with regard to
execution time. Hence, we expect the interference with interrupt latencies
to be low.
* The IPC operation transfers payload between UTCBs only. Each thread has a
so-called user-level thread control block which is mapped transparently by
the kernel. Because of this mapping, user-level page faults cannot occur
during IPC transfers.
* There is no mapping database. Virtual address spaces are manipulated by
loading and flushing physical TLB entries. There is no caching of mappings
done in the kernel. All higher-level information about the interrelationship
of memory and processes is managed by the user-level core.
* Core runs in user mode, mapped 1-to-1 from the physical address space
except for its virtual thread-context area.
* The kernel paths are executed in physical address space (MicroBlaze).
Because both kernel code and user-level core code are observing the same
address-space layout, both worlds appear to run within a single address
space.
* User processes can use the entire virtual address space (4G) except for a
helper page for invoking syscalls and a page containing atomic operations.
There is no reservation used for the kernel.
* The MicroBlaze architecture lacks an atomic compare-and-swap instruction. On
user-level, this functionality is emulated via delayed preemption. A kernel-
provided page holds the sequence of operations to be executed atomically and
prevents (actually delays) the preemption of a thread that is currently
executing instructions at that page.
* The MicroBlaze MMU supports several different page sizes (1K up to 16MB).
Genode fully supports this feature for page sizes >= 4K. This way, the TLB
footprint can be minimized by choosing sensible alignments of memory
objects.
Current state
=============
The MicroBlaze platform support resides in the 'base-mb' repository. At the
current stage, core is able to successfully start multiple nested instances of
the init process. Most of the critical kernel functionality is working. This
includes inter-process communication, address-space creation, multi-threading,
thread synchronization, page-fault handling, and TLB eviction.
This simple scenario already illustrates the vast advantage of
using different page sizes supported by the MicroBlaze CPU. If using
4KB pages only, a scenario with three nested init processes produces more than
300.000 page faults. There is an extremely high pressure on the TLB, which
only contains 64 entries. Those entries are constantly evicted so that
threshing effects are likely to occur. By making use of flexible page
sizes (4K, 16K, 64K, 256K, 1M, 4M, 16M), the number of page faults gets
slashed to only 1.800, speeding up the boot time by factor 10.
Currently, there is no restriction of IPC communication rights. Threads are
addressed using their global thread IDs (in fact, using their respective
indices in the KTCB array). For the future, we are planning to add
capabilty-based delegation of communication rights.
Building and using Genode on MicroBlaze
=======================================
For building Genode for the MicroBlaze platform, you need the MicroBlaze
tool chain as it comes with the Xilinx EDK. The tool chain is typically
prefixed with 'mb-'. Please make sure that the tool chain's 'bin/' directory
is included in your 'PATH' environment variable.
For building and starting Genode on MicroBlaze, you first need to create
a build directory using the build-directory creation tool:
! tool/builddir/create_builddir microblaze \
! BUILD_DIR=</path/to/build/dir> \
! GENODE_DIR=</path/to/genode/dir>
The 'base-mb' repository comes with support for Genode's run tool. In order to
use it, you will first need to declare the location of your qemu binary using
the 'QEMU=/path/to/qemu' variable in the '<build-dir>/etc/microblaze.conf'
file. Then you will be able to start an example scenario by issuing the
following command from within your build directory:
! make run/nested_init
Thereby, the 'run' tool will attempt to start core using the microblaze version
of qemu.
You can also find a simple hello-world example at 'base-mb/src/test/hello'.
The corresponding run script is located at 'base-mb/run/hello.run'. You can
execute it via 'make run/hello' from the build directory.
Note that currently, all boot modules are linked against the core binary.
To change the boot modules, the file 'base-mb/src/core/boot_modules.s' must
be modified.
For reference, we are using the following tools:
* mb-g++ (GCC) 4.1.1 20060524 (Xilinx 11.2 Build EDK_LS2.2
20 Apr 2009 Xilinx 11.2 Build EDK_LS2.2 23 Apr 2009)
* GNU ld version 2.16 Xilinx 11.2 Build EDK_LS2.2 23 Apr 2009
* GNU assembler 2.16 Xilinx 11.2 Build EDK_LS2.2 23 Apr 2009
* QEMU emulator version 0.14.50, Copyright (c) 2003-2008 Fabrice Bellard
Petalogix linux reference design targeting Xilinx Spartan 3ADSP-1800 boards.
Supporting the NOVA hypervisor version 0.3
##########################################
NOVA is a so called microhypervisor - a modern capability-based microkernel
with special support for hardware-based virtualization and IOMMUs. Since we
incorporated the initial support for the NOVA hypervisor in Genode one year
ago, this kernel underwent multiple revisions. The latest version was released
earlier this month. To our delight, much of the features that we missed from
the initial release had been implemented during the course of the last year. We
are especially happy about the fully functional 'revoke' system call and the
support for remote kernel-object creation.
With the Genode release 11.02, we officially support the latest NOVA version.
The update of Genode to the new version required two steps. First, because many
details of the kernel interface were changed between version 0.1 and version
0.3, we had to revisit our syscall bindings and adapting our code to changed
kernel semantics. Second, we filled our 'base-nova' code related to object
destruction and unmapping with life to benefit from NOVA's 'revoke' system
call. Consequently, we are now able to run the complete Genode software stack
including the dynamic linker on NOVA.
Note that for using Genode on NOVA, you will need to apply a small patch to the
NOVA source code. This patch enables the re-use of user-level thread control
blocks in the kernel. The patch can be found at 'base-nova/patches/utcb.patch'.
When executing NOVA on qemu, please specify the '-cpu coreduo' argument to the
qemu command line. When using Genode 'run' tool, you may assign this argument
to the 'QEMU_OPT' variable in '<build-dir>/etc/build.conf'.
:Thanks:
We are grateful for the ongoing very pleasant collaboration with Udo Steinberg
who is the driving force behind NOVA. Thanks for the ultra-fast responses to our
questions and for considering our suggestions regarding the feature set of
NOVA's kernel interface!
Base framework
##############
Upgrading existing sessions
===========================
Genode enables a client of a service to lend parts of its own resources to
the service when opening a session. This way, servers do not need to allocate
own resources on behalf of their clients and become inherently robust against
resource-exhaustion-based denial-of-service attacks.
However, there are cases when the client can not decide about the amount of
resources to lend at session-creation time. In such cases, we used to devise an
overly generous client policy. Now, we have added a new 'upgrade' function to
the 'Parent' and 'Root' interfaces that enables a client to upgrade the
resources of an existing session.
For the 'env()->rm_session()' and 'env()->ram_session()' of processes using
the Genode 'env' library, we implemented a transparent quota upgrade that kicks in
in the event of an exceeded metadata backing store.
Comprehensive accounting of core resources
==========================================
We changed all services of core to limit their respective resource usage
specifically for each individual session. For example, the number of dataspaces
that can be handled by a particular region-manager (RM) session depends on the
resource donation attached to the session. To implement this accounting scheme
throughout core, we added a generic 'Allocator_guard' utility to
'base/include/'. We recommend using this utility when implementing resource
multiplexers, in particular multi-level services. Thanks to this change in
core, the need for a slack memory reservation in core has vanished.
Various changes
===============
The remaining parts of the base API underwent no fundamental revision. The
changes are summarized as follows.
:C++ Support:
We removed 'libgcc' from our C++ support library ('cxx') and link
it to each individual final target and shared library instead. This change alleviates
the need to abuse the 'KEEP_SYMBOLS' mechanism that we used in 'cxx' to
keep libc-dependencies of GCC's support libraries local to the 'cxx'
library. Besides the benefit of reducing heuristics, this change improves
the compatibility with recent cross-compiling tool chains.
Furthermore, we added 'realloc' to the local libc support of the 'cxx'
library because recent ARM tool chains tend to use this function.
:Argument handling for 'main()':
We added a hook to the startup code to enable the implementation of
custom facilities for passing arguments to the main function. The
hook uses the global variables 'genode_argc' and 'genode_argv'.
:Child-exit policy hook:
We enhanced the 'Child_policy' with a new policy interface that allows
a simplified implementation of policies related to program termination.
:Changed API of 'Range_allocator':
We changed the return value of 'alloc_addr' to distinguish different error
conditions. Note that the boolean meaning of the return value is inverted.
Please check your uses of 'alloc_addr'!
Operating-system services and libraries
#######################################
C Runtime
=========
In conjunction with our work on Noux, we improved Genode's C runtime at many
places. First, we added libstdtime and some previously missing bits of libgdtoa
to the libc. These additions largely alleviate the need for dummy stubs, in
particular time-related functions. Second, we added the following functions to
our libc plugin interface: 'dup2', 'fchdir', 'fcntl', 'fstat', 'stat', and
'write'. This enables the creation of advanced libc plugins simulating a whole
file system as done with Noux. Still, there are a number of dummy stubs found
at 'libc/src/lib/libc/dummy.cc'. However, those stubs are now all defined as
weak symbols such that they can be overridden by libc plugins. Finally, we have
replaced the original 'exit' implementation that comes with the libc with a
Genode-specific version. The new version reports the exit code of the
application to the parent process via an 'Parent::exit()' RPC call.
Until now, Genode's libc magically handled output to stdout and stderr by
printing messages via Genode's LOG interface. We have now replaced this
hard-wired interface by an optional libc plugin called 'libc_log'. If present, write
operations to stdout are caught at the libc plugin interface and delegated to
the plugin, which implements the output to the LOG interface. If you have an
application using Genode's libc, you might consider adding the 'libc_log'
library to your 'target.mk' file.
Support for big numbers by the means of libgmp and libmpfr
==========================================================
We have now include both the GNU Multiple Precision Arithmetic Library and
(GMP) and MPFR to the 'ports' repository. This work was specifically motivated
by our port of GCC to Genode as GCC version 4.4.5 requires both libraries.
Because we intend to use those libraries primarily on x86_32, the current port
covers only this architecture. However, expanding the port to
further CPU architectures should be straight-forward if needed.
Furthermore, you can now also find GCC's 'longlong.h' header at
'libports/include/gcc'.
Qt4 updated to version 4.7.1
############################
The current release bumps the supported Qt4 version from 4.6.2 to 4.7.1 and the
Arora web browser (located at the ports repository) from version 0.10.2 to
version 0.11. Of course, we updated our custom additions such as our custom
Nitpicker plugin widget that enables the seamless integration of native
Nitpicker GUI clients into Qt4 applications to work with the new Qt4 version.
Tools
#####
Tool chain update to GCC 4.4.5 and Binutils 2.21
================================================
We upgraded the official Genode tool chain from gcc 4.2.4 to gcc 4.4.5. Please
update your tool chain by downloading the new binary archive (available for x86_32)
or building the tool chain from source using our 'tool/tool_chain' utility.
New support for automated integration and testing
=================================================
With the growing number of supported base platforms, the integration and testing
of Genode application scenarios across all kernels becomes
increasingly challenging. Each kernel has a different boot mechanism and
specific requirements such as the module order of multiboot modules (Fiasco's
bootstrap, Pistachio's sigma0 and kickstart), kernel parameters, or the
invocation of a single-image creation tool (OKL4's elfweaver). To make our
life supporting all those platforms easier, we have created a tool called
'run', which is tightly integrated within Genode's build system. In short 'run'
gathers the intrinsics in the form of a 'run/env' file specific for the
platform used by the current build directory from the respective
'base-<platform>' repository. It then executes a so-called run script, which
contains all steps needed to configure, build, and integrate an application
scenario. For example, a typical run script for building and running a test
case resides in a file called '<any-repository>/run/<run-script-name>.run' and
looks as follows:
! build "core init test/exception"
! create_boot_directory
! install_config {
! <config>
! <parent-provides>
! <!--<service name="ROM"/>-->
! <service name="LOG"/>
! </parent-provides>
! <default-route>
! <any-service> <parent/> </any-service>
! </default-route>
! <start name="test-exception">
! <resource name="RAM" quantum="1M"/>
! </start>
! </config>
! }
! build_boot_image "core init test-exception"
! append qemu_args "-nographic -m 64"
! run_genode_until {.*Exception \(label 0xffb0\) occured.*} 10
First, the build system is instructed to create the targets specified as argument
for the 'build' function. Next, for the integration part, a new boot directory is
created. On most kernel platform, the respective location of the boot directory
is '<build-dir>/var/run/<run-script-name>'. Initially, this directory is empty.
It gets populated with a 'config' file specified as argument of the 'install_config'
command, and by the boot modules specified at the 'build_boot_image' command.
Now that the integration is complete, the scenario is executed via the
'run_genode_until' command. This command takes a regular expression as
argument, which determines the successful termination of the test case. The
second argument is a timeout (is seconds). In the example, the test case will
fail if its output does not match the regular expression within the execution
time of 10 seconds.
The command 'append qemu_args' specifies run-script-specific qemu arguments in
the case that qemu is used to execute the scenario. This is the case for most
kernel platforms (except for Linux where core gets executed directly on the host).
Additional build-directory-specific qemu arguments can be specified in the
'etc/build.conf' file by defining the 'QEMU_OPT' variable. For example, to
prevent KVM being used on Ubuntu Linux, specify:
! QEMU_OPT = -no-kvm
To execute the run script from with build directory, you need to have Expect
installed. Typically, the Linux package is called 'expect'. Simply issue
the following command from within your build directory:
! make run/<run-script>
Note that you will need to have a GRUB 'stage2_eltorito' binary available
at '<genode-dir>/tool/grub' on base platforms that use an ISO image as boot
stategy.
Because the whole chain of actions, building, integrating, executing, and
validating an application scenario is now at the fingertips of issuing a
single command with no kernel-specific considerations needed, it has never
been easier to run the same scenario on a wide range of different kernels.
Please find further instructive examples at 'os/run/'. The 'ldso' run
script executes the test of the dynamic linker. It is completely generic.
The 'demo' run script starts Genode's default demo scenario and shows how
platform-specific considerations (e.g., which device drivers to use) can be
taken into account.
We found that the 'run' tool significantly boosted our productivity not
only for testing purposes but also for accelerating the development-test
cycle during our day-to-day work.
:Technical notes:
The 'run' tool uses Expect as automation tool. Expect is a Tcl interpreter,
which is accompanied by special functionality for automating interactive
command-line applications. Technically, a run script is an Expect script
which gets included by the 'tool/run' script. For the reference of
run-specific functions, please revise the documentation in the 'tool/run'
script. Because each run script is actual Expect source code, it is possible
to use all Tcl and Expect scripting features in a run script.
In particular, a run script may issue shell commands using Tcl's 'exec'
function. This way, even complex integration tasks can be accomplished.
For example, the integration of the Genode Live CD was done via a single
run script.
Build system
============
To facilitate the integration of 3rd-party build systems into the Genode build
process, we added support for pseudo targets that do not require any 'SRC'
declaration. Such 'target.mk' may contain custom rules that will be executed
when the target is revisited by the build system. The bindings are as follows:
! build_3rd_party:
! ...custom commands...
!
! $(TARGET): build_3rd_party
!
! clean_3rd_party:
! ...custom commands...
!
! clean_prg_objects: clean_3rd_party:

1289
doc/release_notes-11-05.txt
View File

File diff suppressed because it is too large Load Diff

703
doc/release_notes-11-08.txt
View File

@@ -1,703 +0,0 @@
===============================================
Release notes for the Genode OS Framework 11.08
===============================================
Genode Labs
One of Genode's most distinctive properties is its support for various
different kernels as base platforms. Each of the 8 currently supported kernels
differs with regard to features, security, hardware support, complexity, and
resource management. Even though different applications call for different
kernel properties, through Genode, those properties can be leveraged using a
unified API. The growing number of supported base platforms, however, poses two
challenges, which are the comprehension of the large diversity of tools and
boot concepts, and capturing of the semantic differences of all the kernels.
With the version 11.08, the framework mitigates the former challenge by
introducing a unified way to download, build, and use each of the
kernels with Genode's user-level infrastructure. The new tools empower users of
the framework to instantly change the underlying kernel without the need to know
the peculiarities of the respective kernels. Using microkernels has never been
easier.
The second challenge of translating each kernel's specific behaviour to the
framework's unified API longs for an automated testing infrastructure that
systematically exercises all the various facets of the API on all base
platforms. The new version introduces the tooling support especially designed
for conducting such quality-assurance measures. These tools largely remove the
burden of manual testing while helping us to uphold the stability and quality
of the framework as it grows in terms of functional complexity and number of
base platforms.
Speaking of functional enhancements, the work on version 11.08 was focused
on our block-device infrastructure and ARM support. The block-device-related
work is primarily motivated by our fundamental goal to scale Genode to a
general-purpose computing platform. The additions comprise new drivers for
SD-cards, IDE, SATA, USB storage as well as a new partition server. All those
components provide Genode's generic block interface, which is meant to be used
as back end for file systems. On file-system level, a new libc plugin utilizes
libffat to enable the straight-forward use of VFAT partitions by libc-using
programs.
The current release comes with far-reaching improvements with respect to
ARM-based platforms. The paravirtualized L4Linux kernel has been updated to
Linux version 2.6.39 running on both x86_32 and ARM. Also, Qt4 including Webkit
has become functional on ARMv6-based platforms.
Among the further improvements are many new examples in the form of
ready-to-use run scripts as well as a comprehensive documentation update.
Originally, we had planned to complement the Noux runtime environment to
support interactive command-line applications by the time of the current
release. However, we realized that the current users of the framework would
value the new streamlined tooling support, the enhanced documentation, and the
new quality-assurance infrastructure over such a functional addition. Hence, we
prioritized the topics accordingly. Even though you will find the first bits of
interactive GNU application support in this release, we deferred working on
this topic in full steam to the upcoming version 11.11.
Blurring the boundaries between different kernels
#################################################
Before the Genode project was born, each microkernel carried along its own
userland. For example, the L4/Fiasco kernel came with the L4 environment, the
OKL4 kernel came with Iguana, or the L4ka::Pistachio kernel came with a small
set of example components. Those user-level counterparts of the kernel
complemented their respective kernels with a runtime for user-level
applications and components while exposing significant parts of the kernel
interface at API level. Consequently, most if not all applications developed
against these APIs were tied to a particular kernel. On the one hand, this
approach enabled developers to fine-tune their programs using kernel-specific
features. On the other hand, much effort was wasted by duplicating other
people's work. Eventually, all of the mentioned userlands stayed limited to
special purposes - for the most part the purposes of operating-systems
researchers. Consequently, none of the microkernels gained much attention in
general-purpose computing. Another consequence of the highly fragmented
microkernel community was the lack of a common ground to compare different
kernels in an unbiased way because each userland provided a different set of
components and libraries.
Different application areas call for different kernel features such as
security mechanisms, scheduling, resource management, and hardware support.
Naturally, each kernel exhibits a specific profile of these parameters
depending on its primary purpose. If one microkernel attempted to accommodate
too many features, it would certainly sacrifice the fundamental idea of being
minimally complex. Consequently, kernels happen to be vastly different. During
the past three years, however, Genode has demonstrated that one carefully
crafted API can target highly diverse kernels, and thereby enables users of
the framework to select the kernel that fits best with the requirements
dictated by each application scenario individually. For us Genode developers,
it was extremely gratifying to see that kernels as different as Linux and NOVA
can be reconciled at the programming-interface level. Still, each kernel comes
with different tools, configuration mechanisms, and boot concepts. Even though
Genode programs can be developed in a kernel-independent way, the deployment of
such programs still required profound insights into the peculiarities of the
respective kernel.
With the current release, we introduce a fundamentally new way of using
different microkernels by unifying the procedures of downloading and building
kernels as well as integrating and running Genode programs with each of them.
Existing Genode application scenarios can be ported between kernels in an
instant without the need for deep insights into the kernel's technicalities. As
a teaser, consider the following commands for building and running Genode's
graphical demo scenario on the OKL4 microkernel:
! # check out Genode
! svn co https://genode.svn.sourceforge.net/svnroot/genode/trunk genode
!
! # download the kernel, e.g., OKL4
! make -C genode/base-okl4 prepare
!
! # create Genode build directory
! genode/tool/create_builddir \
! okl4_x86 BUILD_DIR=build
!
! # build everything and execute the interactive demo
! make -C build run/demo
The same principle steps can be used for any of the OKL4, NOVA,
L4/Fiasco, Fiasco.OC, L4ka::Pistachio, or Codezero kernels. You should
nevertheless consult the documentation at 'base-<platform>/doc/' before
starting to use a specific kernel because some base platforms require
the installation of additional tools.
Under the hood, this seamless way of dealing with different kernels is made
possible by the following considerations:
:Repository preparation:
Each kernel comes from a different source such as a Git/SVN/Mercurial
repository or a packaged archive. Some kernels require additional patches. For
example, OKL4 needs to be patched to overcome problems with modern tool chains.
Now, each 'base-<platform>' repository hosts a 'Makefile' that automates the
download and patch procedure. To download the source code of a kernel,
issue 'make prepare' from within the kernel's 'base-<platform>' directory. The
3rd-party source code will be located at 'base-<platform>/contrib/'.
:Building the kernel:
Each kernel has a different approach when it comes to configuration and
compilation. For example, NOVA comes with a simple 'Makefile', OKL4 relies on a
complex SCons-based build system, L4ka::Pistachio uses CML2 and autoconf (for
the userland tools). Furthermore, some kernels require the setting of specific
configuration values. We have streamlined all these procedures into the Genode
build process by the means of a 'kernel' pseudo target and a 'platform' pseudo
library. The kernel can be compiled directly from the Genode build directory by
issuing 'make kernel'. The 'platform' pseudo library takes care of making the
kernel headers available to Genode. For some kernels such as OKL4 and NOVA, we
replaced the original build mechanism with a Genode target. For other kernels
such as L4ka::Pistachio or Fiasco.OC, we invoke the kernel's build system.
:Genode build directory:
Genode build directories are created via the 'tool/create_builddir' tool.
This tool used to require certain kernel-specific arguments such as the
location of the kernel source tree. Thanks to the unified way of preparing
kernels, the need for such arguments has vanished. Now, the only remaining
arguments to 'create_builddir' are the actual platform and the location
of the build directory to create.
:System integration and booting:
As diverse the build systems of the kernels are, so are the boot concepts. Some
kernels rely on a multiboot-compliant boot loader whereas others have special
tools for creating boot images. Thankfully, Genode's run concept allows us to
hide the peculiarities of booting behind a neat and easy-to-use facade. For
each platform we have crafted a dedicated run environment located at
'base-<platform>/run/env', which contains the rules for system integration and
booting. Therefore, one and the same run script can be used to build and
execute one application scenario across various different kernels. For an
illustrative example, the 'os/src/run/demo.run' script can be executed on all
base platforms (except for base-mb) by issuing 'make run/demo' from within the
build directory.
Emerging block-device infrastructure
####################################
Since version 10.08, Genode is equipped with a block-session interface. Its
primary use cases so far were the supply of the paravirtualized OKLinux kernel
with backing store, and the access of the content of a bootable Live CD.
However, for our mission to use Genode as general-purpose computing platform,
disk device access is crucial. Therefore, we dedicated our attention to
various aspects of Genode's block-device infrastructure, reaching from
programming APIs for block drivers, over partition handling, to file-system
access.
:Block session interface:
The glue that holds all block-device-related components together is the generic
block interface 'os/include/block_session'. It is based on the framework's
packet-stream facility, which allows the communication of bulk data via shared
memory and a data-flow protocol using asynchronous notifications. The interface
supports arbitrary allocation schemes and the use of multiple outstanding
requests. Hence, it is generally suited for scatter-gather DMA and the use of
command queuing as offered by the firmware of modern block-device controllers.
(albeit the current drivers do not exploit this potential yet)
:Block component framework:
Our observation that components implementing the block session interface share
similar code patterns prompted us to design a framework API for implementing
this family of components. The set of classes located at 'os/include/block'
facilitate the separation of device-specific code from application logic.
Whereas 'component.h' provides the application logic needed to implement the
block service, the 'driver.h' is an abstract interface to be implemented by the
actual device driver. This new infrastructure significantly reduces code
duplication among new block-device drivers.
:Device-driver implementations:
The new block-device drivers introduced with the current release address
common types of block devices:
* By adding ATA read/write support to the ATAPI driver ('os/src/drivers/atapi'),
this driver can be used to access IDE disks now.
* The new fully-functional SD-card driver ('os/src/drivers/sdcard') enables the
use of SD-cards connected via the PL180 controller.
* The USB storage driver ('linux_drivers/src/drivers/usb') has been adapted
to the block-session interface and can be used on PC hardware.
* The new AHCI driver ('os/src/drivers/ahci') enables the access of disks
connected via SATA on PC hardware.
Because all drivers are providing the generic block-session interfaces, they
can be arbitrarily combined with components that use this interface as back
end, for example, the partition server and file systems.
:Partition manager as resource multiplexer:
The new partition manager ('os/src/server/part_blk') multiplexes one back-end
block session to multiple block sessions, each accessing a different partition.
Its natural role is being "plugged" between a block-device driver and a file
system.
:File-system access:
Even though a session interface for file systems does not exist yet, we
enabled the use of VFAT partitions through a libc plugin. This libc plugin uses
the ffat library to access files stored on a block device. An
application using this plugin can be directly connected to a block session.
New documentation
#################
The new way of dealing with different kernels motivated us to revisit and
complement our exiting documentation. The following documents are new or
have received considerable attention:
:[http://genode.org/documentation/developer-resources/getting_started - Getting started]:
The revised guide of how to explore Genode provides a quick way to
test drive Genode's graphical demo scenario with a kernel of your
choice and gives pointers to documents needed to proceed your
exploration.
:[http://genode.org/documentation/developer-resources/build_system - Build system manual]:
The new build-system manual explains the concepts behind Genode's
build system, provides guidance with creating custom programs and
libraries, and covers the tool support for the automated integration
and testing of application scenarios.
:[http://genode.org/documentation/components - Components overview]:
The new components-overview document explains the categorization of
Genode's components and lists all components that come with the framework.
:[http://genode.org/documentation/developer-resources/init - Configuration of the init process]:
The document describes Genode's configuration concept, the routing of
service requests, and the expression of mandatory access-control policies.
:[http://genode.org/community/wiki - Wiki]:
The platform-specific Wiki pages for L4/Fiasco, L4ka::Pistachio, NOVA,
Codezero, Fiasco.OC, and OKL4 have been updated to reflect the new flows of
working with the respective base platforms.
Base framework
##############
The RPC API for performing procedure calls across process boundaries
introduced with the version 11.05 was the most significant API change
in Genode's history. To make the transition from the old client-server
API to the new RPC API as smooth as possible, we temporarily upheld
compatibility to the old API. Now, the time has come to put the old
API at rest. The changes that are visible at API level are as follows:
* The old client-server API in the form of 'base/server.h' is no more.
The functionality of the original classes 'Server_entrypoint' and
'Server_activation' is contained in the 'Rpc_entrypoint' class provided
via 'base/rpc_server.h'.
* When introducing the RPC API, we intentionally left the actual session
interfaces as unmodified as possible to proof the versatility of the new
facility. However, it became apparent that some of the original interfaces
could profit from using a less C-ish style. For example, some interfaces used
to pass null-terminated strings as 'char const *' rather than via a dedicated
type. The methodology of using the new RPC API while leaving the original
interfaces intact was to implement such old-style functions as wrappers
around new-style RPC functions. These wrappers were contained in
'rpc_object.h' files, e.g. for 'linux_dataspace', 'parent', 'root',
'signal_session', 'cpu_session'. Now, we have taken the chance to modernise
the API by disposing said wrappers. Thereby, the need for 'rpc_object.h'
files has (almost) vanished.
* The remaining users of the old client-server API have been adapted to the
new RPC API, most prominently, the packet-stream-related interfaces such as
'block_session', 'nic_session', and 'audio_session'.
* We removed 'Typed_capability' and the second argument of the 'Capability'
template. The latter was an artifact that was only used to support the
transition from the old to the new API.
* The 'ipc_client' has no longer an 'operator int'. The result of an IPC can
be requested via the 'result' function.
* We refined the accessors of 'Rpc_in_buffer' in 'base/rpc_args.h'. The
'addr()' has been renamed to 'base()', 'is_valid_string()' considers the
buffer's capacity, and the new 'string()' function is guaranteed to return a
null-terminated string.
* We introduced a new 'Rm_session::Local_addr' class, which serves two
purposes. It allows the transfer of the bit representation of pointers across
RPC calls and effectively removes the need for casting the return type of
'Rm_session::attach' to the type needed at the caller side.
* The 'Connection' class template has been simplified, taking the session
interface as template argument (rather than the capability type). This change
simplified the 'Connection' classes of most session interfaces.
* The never-used return value of 'Parent::announce' has been removed. From the
child's perspective, an announcement always succeeds. The way of how the
announcement is treated is entirely up to the parent. The client should never
act differently depending on the parent's policy anyway.
* The new 'Thread_base::cap()' accessor function allows obtaining the thread's
capability as used for the argument to CPU-session operations.
Operating-system services and libraries
#######################################
Dynamic linker
==============
As a follow-up to the major revision of the dynamic linker that was featured
with the previous release, we addressed several corner cases related to
exception handling and improved the handling of global symbols.
The dynamic linker used to resolve requests for global symbols by handing out
its own symbols if present. However, in some cases, this behaviour is
undesired. For example, the dynamic linker contains a small set of libc
emulation functions specifically for the ported linker code. In the presence of
the real libc, however, these symbols should never be considered at all. To
avoid such ambiguities during symbol resolution, the set of symbols to be
exported is now explicitly declared by the white-list contained in the
'os/src/lib/ldso/symbol.map' file.
We changed the linkage of the C++ support library ('cxx') against dynamic
binaries to be consistent with the other base libraries. Originally, the 'cxx'
library was linked to both the dynamic linker and the dynamic binary, which
resulted in subtle problems caused by the duplication of cxx-internal data
structures. By linking 'cxx' only to the dynamic linker and exporting the
'__cxa' ABI as global symbols, these issues have been resolved. As a positive
side effect, this change reduces the size of dynamic binaries.
C++ exception handling in the presence of shared libraries turned out to be
more challenging than we originally anticipated. For example, the
'_Unwind_Resume' symbol is exported by the compiler's 'libsupc++' as a hidden
global symbol, which can only be resolved when linking this library to the
binary but is not seen by the dynamic linker. This was the actual reason of why
we used to link 'cxx' against both dynamic binaries and shared libraries
causing the problem mentioned in the previous paragraph. Normally, this problem
is addressed by a shared library called 'libgcc_s.so' that comes with the
compiler. However, this library depends on glibc, which prevents us from using
it on Genode. Our solution is renaming the hidden global symbol using a
'_cxx__' prefix and introducing a non-hidden global wrapper function
('__cxx__Unwind_Resume' in 'unwind.cc'), which is resolved at runtime by the
dynamic linker.
Another corner case we identified is throwing exceptions from within the
dynamic linker. In contrast to the original FreeBSD version of the dynamic
linker, which is a plain C program that can never throw a C++ exception,
Genode's version relies on C++ code that makes use of exceptions. To support
C++ exceptions from within the dynamic linker, we have to relocate the
linkers's global symbols again after having loaded the dynamic binary. This
way, type information that is also present within the dynamic binary becomes
relocated to the correct positions.
Block partition server
======================
The new block-partition server uses Genode's block-session interfaces as both
front and back end, leading to the most common use case where this server will
reside between a block driver and a higher level component like a file-system
server.
At startup, the partition server will try to parse the master boot record (MBR)
of its back-end block session. If no partition table is found, the whole block
device is exported as partition '0'. In the other case, the MBR and possible
extended boot records (EBRs) are parsed and offered as separate block sessions
to the front-end clients. The four primary partitions will receive partition
numbers '1' to '4' whereas the first logical partition will be assigned to '5'.
The policy of which partition is exposed to which client can be expressed
in the config supplied to the 'part_blk' server. Please refer to the
documentation at 'os/src/server/part_blk/README' for further details. As an
illustration of the practical use of the 'part_blk' server, you can find a run
script at 'os/run/part_blk.run'.
Skeleton of text terminal
=========================
As part of the ongoing work towards using interactive text-based GNU software
on Genode, we created the first bits of the infrastructure required for
pursuing this quest:
The new terminal-session interface at 'os/include/terminal_session/' is the
designated interface to be implemented by terminal programs.
After investigating the pros and cons of various terminal protocols and
terminal emulators, we settled for implementing a custom terminal emulator
implementing the Linux termcap. This termcap offers a reasonable small set of
commands while providing all essential features such as function-key support
and mouse support. Thanks to Peter Persson for pointing us to the right
direction! The preliminary code for parsing the escape sequences for the Linux
termcap is located at 'gems/include/terminal/'.
We have created a simplistic terminal service that implements the
terminal-session interface using a built-in font. Please note that the
implementation at 'gems/src/server/terminal/' is at an early stage. It is
accompanied by a simple echo program located at 'gems/src/test/terminal_echo'.
Device drivers
##############
USB HID and USB storage
=======================
We replaced the former DDE-Linux-based USB-related driver libraries (at the
'linux_drivers/' repository) by a single USB driver server that offers the
'Input' and 'Block' services. This enables us to use both USB HID and USB
storage at the same time. The new USB driver is located at
'linux_drivers/src/drivers/usb/'.
For using the USB driver as input service (supporting USB HID), add the
'<hid/>' tag to the 'usb_drv' configuration. Analogously, for using the driver
as block service, add the '<storage/>' tag. Both tags can be combined.
For testing the USB stack, the 'linux_drivers' repository comes with the run
scripts 'usb_hid.run' and 'usb_storage.run'.
ATA read/write support
======================
The ATAPI driver has been extended to support IDE block devices for both
read and write transactions. To use the new facility, supply 'ata="yes"'
as XML attribute to the config node of 'atapi_drv'. Please note that this
driver was primarily tested on Qemu. Use it with caution.
SATA driver
===========
The new SATA driver at 'os/src/drivers/ahci/' implements the block-driver
API ('os/include/block'), thus exposing the block-session interface as
front-end. AHCI depends on Genode's PCI driver as well as the timer server. For
a usage example see: 'os/run/ahci.run'.
Limitations and known issues
----------------------------
Currently, the server scans the PCI bus at startup and retrieves the first available
AHCI controller, scans the controller ports and uses the first non-ATAPI port
where a device is present.
On real hardware and on kernels taking advantage of I/O APICs (namely NOVA and
Fiasco.OC) we still lack support for ACPI parsing and thus for interrupts,
leading to a non-working driver.
SD-card driver
==============
The first fragments of our SD-card driver that we introduced with the previous
release have been complemented. The new SD-card driver located at
'os/src/drivers/sd_card/' implements the block-session interface by using
MMC/SD-cards and the PL180 controller as back end. Currently the driver
supports single-capacity SD cards. Therefore, the block file for Qemu should
not exceed 512 MB. Because the driver provides the generic block-session
interface, it can be combined with the new 'libc_ffat' plugin in a
straight-forward way. To give the driver a quick spin, you may give the
'libports/run/libc_ffat.run' script on the 'foc_pbxa9' platform a try.
ARM Realview PL011 UART driver
==============================
The new PL011 UART driver at 'os/src/drivers/uart/' implements the LOG session
interface using the PL011 device. Up to 4 UARTs are supported. The assignment
of UARTs to clients can be defined via a policy supplied to the driver's config
node. For further information, please refer to the README file within the
'uart' directory.
Libraries and applications
##########################
Hello tutorial
==============
The 'hello_tutorial/' repository contains a step-by-step guide for building
a simple client-server scenario. The tutorial has been rewritten for the new
RPC API and is now complemented by a run script for testing the final scenario
on various base platforms.
C and C++ runtimes
==================
:Support for standard C++ headers:
Triggered by public demand for using standard C++ headers for Genode applications,
we introduced a generally usable solution in the form of the 'stdcxx' library
to the 'libc' repository. The new 'stdcxx' library is not a real library. (you
will find the corresponding 'lib/mk/stdcxx.mk' file empty) However, it comes
with a 'lib/import/import-stdcxx.mk' file that adds the compiler's C++ includes
to the default include-search path for any target that has 'stdcxx' listed in
its 'LIBS' declaration.
:Libc back end for accessing VFAT partitions:
The new 'libc_ffat' libc plugin uses a block session via the ffat library. It
can be used by a Genode application to access a VFAT file system via the libc
file API. The file-system access is performed via the 'ffat' library. To
download this library and integrate it with Genode, change to the 'libports'
repository and issue the following command:
! make prepare PKG=ffat
For an example of how to use the libc-ffat plugin, please refer to the run
script 'libports/run/libc_ffat.run'. The source code of the test program can be
found at 'libports/src/test/libc_ffat/'.
Qt4
===
Qt4 version 4.7.1 has been enabled on ARMv6-based platforms, i.e., PBX-A9 on
Fiasco.OC. The support comprises the entire Qt4 framework including qt_webcore
(Webkit).
L4Linux
=======
L4Linux enables the use of one or multiple instances of Linux-based operating
systems as subsystems running on the Fiasco.OC kernel. The Genode version of
L4Linux has seen the following improvements:
:Kernel version: has been updated to Linux 2.6.39.
:ARM support: The L4Linux kernel can be used on ARM-based platforms now.
The PBX-A9 platform is supported via the 'l4linux.run' script as found
at 'ports-foc/run/'. Please find more information at 'ports-foc/README'.
:Genode-specific stub drivers outside the kernel tree:
The stub drivers that enable the use of Genode's services as virtual
devices for L4Linux have been moved outside the kernel patch, which
makes them much easier to maintain. These stub drivers are located
under 'ports-foc/src/drivers/'.
Platform support
################
All base platforms are now handled in a unified fashion. Downloading 3rd-party
source code is performed using the 'prepare' rule of the 'Makefile' provided by
the respective kernel's 'base-<platform>' repository. Once, the platform's base
repository is prepared, the kernel can be built directly from the Genode
build directory using 'make kernel'. All base platforms are now supported by
Genode's run mechanism that automates the tasks of system integration and
testing. For more details about each specific kernel, please revisit the
updated documentation within the respective 'base-<platform>/doc/' directory.
:L4/Fiasco:
The kernel has been updated to revision 472, enabling the use of recent
GNU tool chains.
:Fiasco.OC:
The kernel as been updated to revision 36, which remedies stability problems
related to interaction of the IPC path with thread destruction. The new version
improves the stability of highly dynamic workloads that involve the frequent
creation and destruction of subsystems. However, we experienced the new kernel
version to behave instable on the x86_64 architecture. If you depend on x86_64,
we recommend to temporarily stick with Genode 11.05 and Fiasco.OC revision 31.
:L4ka::Pistachio:
The kernel has been updated to revision 803, enabling the use of recent
versions of binutils.
:OKL4:
OKL4v2 is showing its age. Apparently, the use of the original distribution
requires tools (i.e., python 2.4) that do not ship with current Linux
distributions anymore. This makes it increasingly difficult to use this kernel.
Still, we find ourselves frequently using it for our day-to-day development. To
streamline the use of OKL4v2, we have now incorporated the kernel compilation
into the Genode build system and thereby weakened the kernel's dependency on
ancient tools. However, we decided to drop support for OKL4/ARM for now. We
figured that the supported GTA01 platform is hardly used anymore and hard to
test because it is unsupported by Qemu. Newer ARM platforms are supported by
other kernels anyway.
:Codezero:
Even though B-Labs apparently abandoned the idea of developing the Codezero
kernel in the open, we adapted Genode to the kernel's most recent Open-Source
version that is still available at the official Git repository. Furthermore,
the kernel is now fully supported by Genode's new 'make prepare' procedure and
run environment. Therefore, run scripts such as 'run/demo' can now easily be
executed on Codezero without the need to manually configure the kernel.
Note that, for now, we have disabled Codezero's capabilities because they do
not allow the assignment of device resources. Consequently, 'sys_map' fails for
MMIO regions when performing the capability check (calling 'cap_map_check').
Furthermore, the current version of the kernel requires a workaround for a
current limitation regarding the definition of a thread's pager. At some point,
Codezero abandoned the facility to define the pager for a given thread via the
exregs system call. Instead, the kernel hard-wires the creator of the thread as
the thread's pager. This is conflicting with Genode's way of creating and
paging threads. In the current version of Genode for this kernel, all threads
are paged by one thread (thread 3 happens to be the global pager) within core.
As a workaround to Codezero's current limitation, we define thread 3 to be the
pager of all threads. The patch of the upstream code is automatically being
applied by the 'make prepare' mechanism.
Build system and tools
######################
In addition to the major change with respect to the integration of the various
base platforms, Genode's tool support received the following incremental
improvements:
Build system
============
:Simplification of 'create_builddir' tool:
The 'create_builddir' tool has been relocated from
'tool/builddir/create_builddir' to 'tool/create_builddir' to make it more
readily accessible. Furthermore, we simplified the usage of the tool by
removing the mandatory 'GENODE_DIR' argument. If not explicitly specified, the
tool deduces 'GENODE_DIR' from the its known location within the Genode source
tree.
:Booting from USB sticks:
For most x86-based base platforms, their respective run environments execute
Genode from an ISO image via Qemu. Naturally, such an ISO image can be burned
onto a CD-ROM to be used to boot a real machine. However, booting from CD-ROM
is slow and optical drives are becoming scarce. Therefore we changed the
procedure of creating ISO images to support writing the resulting images to a
USB stick. Under the hood, the boot mechanism chain-loads GRUB via ISOLinux.
The files to implement the boot concept are located at 'tool/boot/'.
:Support for source files in target sub directories:
Until now, the 'SRC_*' declarations in target description files contained
a list of plain file names. The location of the files within the directory
tree had to be defined via 'vpath'. This led to inconveniences when building
3rd-party code that contains files with the same name at different subdirectories.
To resolve such an ambiguity, the target had to be decomposed into multiple
libraries each building a different set of subdirectories. To make the
build system more convenient to use, we have now added support for specifying
source codes with a relative pathname. For example, instead of using
! SRC_CC = main.cc addon.cc
! vpath addon.cc $(PRG_DIR)/contrib
we can now use
! SRC_CC = main.cc contrib/addon.cc
Automated testing across multiple kernels
=========================================
To execute one or multiple test cases on more than one base platform, we
introduced a dedicated tool located at 'tool/autopilot'. Its primary purpose is
the nightly execution of test cases. The tool takes a list of platforms and a
list of run scripts as arguments and executes each run script on each platform.
The build directory for each platform is created at
'/tmp/autopilot.<username>/<platform>' and the output of each run script is
written to a file called '<platform>.<run-script>.log'. On stderr, autopilot
prints the statistics about whether or not each run script executed
successfully on each platform. If at least one run script failed, autopilot
returns a non-zero exit code, which makes it straight forward to include
autopilot into an automated build-and-test environment.

1008
doc/release_notes-11-11.txt
View File

File diff suppressed because it is too large Load Diff

862
doc/release_notes-12-02.txt
View File

@@ -1,862 +0,0 @@
===============================================
Release notes for the Genode OS Framework 12.02
===============================================
Genode Labs
The release of Genode 12.02 marks an exciting point in the history of the
project as it is the first version developed in the open rather than within the
chambers of Genode Labs. Thereby, we have embraced GitHub as central facility
for discussion and source-code management. This change has benefits for users
and developers of the framework alike. For users, it has become possible to get
hold of the latest developments using the official 'genodelabs/master' branch and
get involved with discussing the current activities. For regular Genode
developers, the public Git repository replaces a former mix of public
Subversion and company-internal Mercurial repositories, making life much
easier. In Section [Liberation of the development process], we outline the
motivation behind this change and give pointers to the new resources.
The major new additions to the base system are a new framework API for accessing
memory-mapped I/O resources, special support for using Genode as user-level
component framework on Linux, and API support for the reuse of existing
components in the form of sandboxed libraries. These changes are accompanied
with new device-driver infrastructure such as the first version of a device
driver manager and a new ACPI parser.
Feature-wise, the current release takes the first steps towards the goal of the
[http://genode.org/about/road-map - Roadmap for 2012], turning Genode into a
general-purpose OS ready for everyday use by its developers. According to the
roadmap, we enhanced the Noux runtime with fork semantics so that we can run
command-line based GNU programs such as the bash shell and coreutils unmodified
and natively on various microkernels. Furthermore, the library infrastructure
has been enhanced by porting and updating libraries such as Qt 4.7.4 and the
MuPDF PDF rendering engine.
Liberation of the development process
#####################################
In summer 2011, we started a discussion within Genode Labs about changing the
mode of how Genode is developed. Until then, most design discussions and the
actual development work took place locally at the company. At quarterly
intervals, we used to publish our work in the form of official Genode
releases. This way of development seemed to work quite well for us, we were
satisfied about the pace of development, and with each release, our project got
more recognition.
However, the excellent book [http://producingoss.com/ - Producing Open Source Software]
made us realize that even though we released our work under an Open-Source
license, our development process was actually far from being open and may have
discouraged participation of people outside the inner circle of developers.
Because we believe that the framework has reached a state where it becomes
interesting for a wider audience of users and developers, the idea was born
to liberate the project from its closed fashion of development.
In the beginning of December, the vague idea has become a plan. So we finally
brought the topic to our mailing list
([http://genode.org/news/steps-towards-an-open-development-process - Steps towards an open development process]).
We decided to take the release cycle for Genode 12.02 as the opportunity to put
our plan to practice. The central element of this endeavour was moving the
project over to GitHub and adapt our workflows and tooling support accordingly.
First, we started to embrace GitHub's issue tracker for the management of
working topics:
:[http://github.com/genodelabs/genode/issues]: Issue Tracker
The most significant step was leaving our Genode-Labs-internal code
repositories behind and starting a completely public Git repository instead:
:[https://github.com/genodelabs]: Genode Labs at GitHub
With the code repository going public, the way was cleared to opening up design
discussions as well. Instead of having such discussions internally at Genode
Labs, we try to increasingly take them to our mailing list and issue tracker.
With this new way of development, we hope to make the project much more
approachable for people who want to get involved and let Genode reach far out
beyond the reach of our little company.
The changes mentioned above are actually just the tip of the iceberg. For
example, the transition phase required us to rethink the way the project
website is maintained. From now on, almost all of the content of genode.org
comes directly from the project's Git repository. So maintaining website
content is done in the same coherent and transparent way as working on Genode's
code base. So we could finally put the old Wiki to rest. In the process, we
largely revisited the existing content. For example, we rewrote the
[http://genode.org/community/contributions - contributions] document in a
tutorial-like style and incorporated several practical hints, in particular
related to the recommended use of Git.
Although it is probably too early to judge the outcome of our transition, we
are excited how smooth this massive change went. We attribute this pleasant
experience mostly to the excellent GitHub hosting platform, which instantly
ignited a spirit of open collaboration among us. We are excited to see new
people approaching us and showing their interest for teaming up, and we are
curious about where this new model of development will take Genode in the
future.
Base framework, low-level OS infrastructure
###########################################
RPC framework refinements
=========================
Until now, the RPC framework did not support const RPC functions. Rather than
being a limitation inherent to the concept, const RPC functions plainly did not
exist. So supporting them was not deemed too important. However, there are uses
of RPC interfaces that would benefit from a way to declare an RPC function as
const. Candidates are functions like 'Framebuffer::Session::mode()' and
'Input::Session::is_pending()'.
With the current version, we clear the way towards declaring such functions as
const. Even though the change is pretty straight-forward, the thorough support
for const-qualified RPC functions would double the number of overloads for the
'call_member' function template (in 'base/include/util/meta.h'). For this
reason, as of now, the support of const functions is limited to typical getter
functions with no arguments. This appears to be the most common use of such
functions.
API support for enslaving services
==================================
While evolving and using the framework, we always keep an eye on recurring
patterns of how its API is used. Once such a pattern becomes obvious, we try
to take a step back, generalize the observed pattern, and come up with a new
building block that unifies the former repetitive code fragments.
One of those patterns that was far from obvious when we designed Genode years
ago is the use of a service running as child of its own client. At the first
glance, this idea seems counter-intuitive because normally, services are
understood as components that operate independently and protected from their
(untrusted) clients. But there is a class of problems where this approach
becomes extremely useful: The reuse of protocol implementations as a
library-like building block. Most services are actually protocol stacks that
translate a low-level protocol to a more abstract API. For example, a block
device driver translates a specific device API to the generic 'Block_session'
interface. Or the 'iso9660' service translates the 'Block_session' interface to
the 'Rom_session' interface by parsing the ISO9660 file system. Or similarly,
the 'tar_rom' service parses the tar file format to make its content available
via the 'Rom_session' interface.
If a particular functionality is needed by multiple programs, it is common
practice to move this functionality into a dedicated library to avoid the
duplication of the same code at many places. For example, if a program would
need to parse a tar archive, it might be tempting to move the tar-parsing code
from the 'tar_rom' service into a dedicated library, which can then be used by
both the 'tar_rom' service and the new program. An alternative approach is to
just re-use the 'tar_rom' service as a black box and treat it like it was a
library. That is, start the 'tar_rom' service as a child process, supply the
resources the component needs to operate and, in turn, use its API (now in the
form of an RPC interface) to get work done. Because the service is started as a
mere tool at the discretion of its client, we call it *slave*. It turns out
that this idea works exceedingly well in many cases. In a way, it resembles the
Unix philosophy to solve complex problems by combining many small tools each
with a specific purpose. In contrast to the use of libraries, the reuse of
entire components has benefits with regard to fault isolation. Because the
reused functionality is sandboxed within a distinct process, the environment
exposed to this code can be tailored to a rigid subset of the host program's
environment. In the event of a fault within the reused component, the reach of
problem is therefore limited.
On the other hand, we observed that even though this idea works as intended,
implementing the idea for a particular use case involved a good deal of
boiler-plate code where most of this code is needed to define the slave's
environment and resources. Hence, we reviewed the existing occurrences of the
slave pattern and condensed their common concerns into the 'Slave_policy' and
'Slave' classes residing in 'os/include/os/slave.h'. The 'Slave' class is meant
to be used as is. It is merely a convenience wrapper for a child process and
its basic resources. The 'Slave_policy' is meant as a hook for service-specific
customizations. The best showcase is the new 'd3m' component located at
'gems/src/server/d3m'. D3m extensively uses the slave pattern by instantiating
and destroying drivers and file-system instances on-the-fly. A further instance
of this pattern can be found in the new ACPI driver introduced with the current
release.
Support for resizable framebuffers
==================================
The framebuffer-session interface has remained largely untouched since the
original release of Genode in 2008. Back then, we were used to rely on C-style
out parameters in RPC functions. The current RPC framework, however, promotes
the use of a more object-oriented style. So the time has come to revisit the
framebuffer session interface. Instead of using C-style out parameters, the new
'mode()' RPC call returns the mode information as an object of type 'Mode'.
Consequently, mode-specific functions such as 'bytes_per_pixel()' have been
moved to the new 'Framebuffer::Mode' class. The former 'info()' function is
gone.
In addition to the overhaul of the RPC interface, we introduced basic support
for resizable framebuffers. The new 'mode_sigh()' function enables a client to
register a signal handler at the framebuffer session. This signal handler gets
notified in the event of server-side mode changes. Via the new 'release()'
function, the client is able to acknowledge a mode change. By calling it, the
client tells the framebuffer service that it no longer uses the original
framebuffer dataspace. So the server can replace it by a new one. After having
called 'release()', the client can obtain the dataspace for the new mode by
calling 'dataspace()' again.
MMIO access framework
=====================
As the arsenal of native device drivers for Genode grows, we are observing
an increased demand to formalize the style of how drivers are written to
foster code consistency. One particular cause of inconsistency used to be
the way of how memory-mapped I/O registers are accessed. C++ has poor support
for defining bit-accurate register layouts in memory. Therefore, driver code
usually carries along a custom set of convenience functions for reading and
writing registers of different widths as well as a list of bit definitions in
the form of enum values or '#define' statements. To access parts of a register,
the usual pattern is similar to the following example (taken from the pl011
UART driver:
! enum {
! UARTCR = 0x030, /* control register */
! UARTCR_UARTEN = 0x0001, /* enable bit in control register */
! ...
! }
! ...
!
! /* enable UART */
! _write_reg(UARTCR, _read_reg(UARTCR) | UARTCR_UARTEN);
This example showcases two inconveniences: The way the register layout is
expressed and the manual labour needed to access parts of registers. In the
general case, a driver needs to also consider 'MASK' and 'SHIFT' values to
implement access to partial registers properly. This is not just inconvenient
but also error prone. For lazy programmers as ourselves, it's just too easy to
overwrite "undefined" bits in a register instead of explicitly masking the
access to the actually targeted bits. Consequently, the driver may work fine
with the current generation of devices but break with the next generation.
So the idea was born to introduce an easy-to-use formalism for this problem. We
had two goals: First, we wanted to cleanly separate the declaration of register
layouts from the program logic of the driver. The actual driver program should
be free from any intrinsics in the form of bit-masking operations. Second, we
wanted to promote uncluttered driver code that uses names (i.e., in the form of
type names) rather than values to express its operations. The latter goal is
actually achieved by the example above by the use of enum values, but this is
only achieved through discipline. We would prefer to have an API that
facilitates the use of proper names as the most convenient way to express an
operation.
The resulting MMIO API comes in the form of two new header files located at
'base/include/util/register.h' and 'base/include/util/mmio.h'.
Register declarations
~~~~~~~~~~~~~~~~~~~~~
The class templates found in 'util/register.h' provide a means to express
register layouts using C++ types. In a way, these templates make up for
C++'s missing facility to define accurate bitfields. Let's take a look at
a simple example of the 'Register' class template that can be used to define
a register as well as a bitfield within this register:
! struct Vaporizer : Register<16>
! {
! struct Enable : Bitfield<2,1> { };
! struct State : Bitfield<3,3> {
! enum{ SOLID = 1, LIQUID = 2, GASSY = 3 };
! };
!
! static void write (access_t value);
! static access_t read ();
! };
In the example, 'Vaporizer' is a 16-bit register, which is expressed via the
'Register' template argument. The 'Register' class template allows for
accessing register content at a finer granularity than the whole register
width. To give a specific part of the register a name, the 'Register::Bitfield'
class template is used. It describes a bit region within the range of the
compound register. The bit 2 corresponds to true if the device is enabled and
bits 3 to 5 encode the 'State'. To access the actual register, the methods
'read()' and 'write()' must be provided as backend, which performs the access
of the whole register. Once defined, the 'Vaporizer' offers a handy way to
access the individual parts of the register, for example:
! /* read the whole register content */
! Vaporizer::access_t r = Vaporizer::read();
!
! /* clear a bit field */
! Vaporizer::Enable::clear(r);
!
! /* read a bit field value */
! unsigned old_state = Vaporizer::State::get(r);
!
! /* assign new bit field value */
! Vaporizer::State::set(r, Vaporizer::State::LIQUID);
!
! /* write whole register */
! Vaporizer::write(r);
Memory-mapped I/O
~~~~~~~~~~~~~~~~~
The utilities provided by 'util/mmio.h' use the 'Register' template class as
a building block to provide easy-to-use access to memory-mapped I/O registers.
The 'Mmio' class represents a memory-mapped I/O region taking its local base
address as constructor argument. Let's take a simple example to see how it is
supposed to be used:
! class Timer : Mmio
! {
! struct Value : Register<0x0, 32> { };
! struct Control : Register<0x4, 8> {
! struct Enable : Bitfield<0,1> { };
! struct Irq : Bitfield<3,1> { };
! struct Method : Bitfield<1,2>
! {
! enum { ONCE = 1, RELOAD = 2, CYCLE = 3 };
! };
! };
!
! public:
!
! Timer(addr_t base) : Mmio(base) { }
!
! void enable();
! void set_timeout(Value::access_t duration);
! bool irq_raised();
! };
The memory-mapped timer device consists of two registers: The 32-bit 'Value'
register and the 8-bit 'Control' register. They are located at the MMIO offsets
0x0 and 0x4, respectively. Some parts of the 'Control' register have specific
meanings as expressed by the 'Bitfield' definitions within the 'Control'
struct.
Using these declarations, accessing the individual bits becomes almost a
verbatim description of how the device is used. For example:
! void enable()
! {
! /* access an individual bitfield */
! write<Control::Enable>(true);
! }
!
! void set_timeout(Value::access_t duration)
! {
! /* write complete content of a register */
! write<Value>(duration);
!
! /* write all bitfields as one transaction */
! write<Control>(Control::Enable::bits(1) |
! Control::Method::bits(Control::Method::ONCE) |
! Control::Irq::bits(0));
! }
!
! bool irq_raised()
! {
! return read<Control::Irq>();
! }
In addition to those basic facilities, further noteworthy features of the new
API are the support for register arrays and the graceful overflow handling
with respect to register and bitfield boundaries.
C Runtime
=========
We extended our FreeBSD-based C runtime to accommodate the needs of the Noux
runtime environment and our port of the MuPDF application.
* The dummy implementation of '_ioctl()' has been removed. This function is
called internally within the libc, i.e., by 'tcgetattr()'. For running
libreadline in Noux, we need to hook into those ioctl operations via the
libc plugin interface.
* The 'libc/regex' and 'libc/compat' modules have been added to the libc.
These libraries are needed by the forthcoming port of Slashem to Noux.
* We added a default implementation of 'chdir()'. It relies on the sequence of
'open()', 'fchdir()', 'close()'.
* The new libc plugin 'libc_rom' enables the use of libc file I/O functions
to access ROM files as provided by Genode ROM session.
* We changed the libc dummy implementations to always return ENOSYS. Prior
this change, 'errno' used to remain untouched by those functions causing
indeterministic behaviour of code that calls those functions, observes the
error return value (as returned by most dummies), and evaluates the error
condition reported by errno.
* If using the libc for Noux programs, the default implementations of
time-related functions such as 'gettimeofday()' cannot be used because they
rely on a dedicated timeout-scheduler thread. Noux programs, however, are
expected to contain only the main thread. By turning the functions into weak
symbols, we enabled the noux libc-plugin to provide custom implementations.
DDE Kit
=======
Linux DDE used to implement Linux spin locks based on 'dde_kit_lock'. This
works fine if a spin lock is initialized only once and used from then on. But
if spin locks are initialized on-the-fly at a high rate, each initialization
causes the allocation of a new 'dde_kit_lock'. Because in contrast to normal
locks, spinlocks cannot be explicitly destroyed, the spin-lock emulating locks
are never freed. To solve the leakage of locks, we complemented DDE Kit with
the new 'os/include/dde_kit/spin_lock.h' API providing the semantics as
expected by Linux drivers.
Libraries and applications
##########################
New and updated libraries
=========================
:Qt4 updated to version 4.7.4:
We updated Qt4 from version 4.7.1 to version 4.7.4. For the most part, the
update contains bug fixes as detailed in the release notes for the versions
[http://qt.nokia.com/products/changes/changes-4.7.2 - 4.7.2],
[http://qt.nokia.com/products/changes/changes-4.7.3 - 4.7.3], and
[http://labs.qt.nokia.com/2011/09/01/qt-4-7-4-released - 4.7.4].
:Update of zlib to version 1.2.6:
Because zlib 1.2.5 is no more available at zlib.net, we bumped the zlib
version to 1.2.6.
:New ports of openjpeg, jbig2dec, and mupdf:
MuPDF is a fast and versatile PDF rendering library with only a few
dependencies. It depends on openjpeg (JPEG2000 codec) and jbig2dec (b/w image
compression library). With the current version, we integrated those libraries
alongside the MuPDF library to the 'libports' repository.
GDB monitor refinements and automated test
==========================================
We improved the support for GDB-based user-level debugging as introduced with
the previous release.
For the x86 architecture, we fixed a corner-case problem with using the
two-byte 'INT 0' instruction for breakpoints. The fix changes the breakpoint
instruction to the single-byte 'HLT'. 'HLT' is a privileged instruction and
triggers an exception when executed in user mode.
The new 'gdb_monitor_interactive.run' script extends the original
'gdb_monitor.run' script with a startup sequence that automates the
initial break-in at the 'main()' function of a dynamically linked binary.
The revised 'gdb_monitor.run' script has been improved to become a full
automated test case for GDB functionalities. It exercises the following
features (currently on Fiasco.OC only):
* Breakpoint in 'main()'
* Breakpoint in a shared-library function
* Stack trace when not in a syscall
* Thread info
* Single stepping
* Handling of segmentation-fault exception
* Stack trace when in a syscall
PDF viewer
==========
According to our road map for 2012, we pursued the port of an existing PDF
viewer as native application to Genode.
We first looked at the [http://poppler.freedesktop.org - libpoppler],
which seems to be the most popular PDF rendering engine in the world of
freedesktop.org. To get a grasp on what the porting effort of this engine may
be, we looked at projects using this library as well as the library source
code. By examining applications such as the light-weight epdfview
application, we observed that libpoppler's primary design goal is to integrate
well with existing freedesktop.org infrastructure rather than to reimplement
functionality that is provided by another library. For example, fontconfig is
used to obtain font information and Cairo is used as rendering backend. In the
context of freedesktop.org, this makes perfect sense. But in our context,
porting libpoppler would require us to port all this infrastructure to Genode
as well. To illustrate the order of magnitude of the effort needed, epdfview
depends on 65 shared libraries. Of course, at some point in the future, we will
be faced to carry out this porting work. But for the immediate goal to have a
PDF rendering engine available on Genode, it seems overly involved. Another
criterion to evaluate the feasibility of integrating libpoppler with Genode is
its API. By glancing at the API, it seems to be extremely feature rich and
complex - certainly not a thing to conquer in one evening with a glass of wine.
The Qt4 backend of the library comprises circa 8000 lines of code. This value
can be taken as a vague hint at the amount of work needed to create a custom
backend, i.e., for Genode's framebuffer-session interface.
Fortunately for us, there exists an alternative PDF rendering engine named
MuPDF. The concept of MuPDF is quite the opposite of that of libpoppler.
MuPDF tries to be as self-sufficient as possible in order to be suitable
for embedded systems without comprehensive OS infrastructure. It comes with a
custom vector-graphics library (instead of using an existing library such as
Cairo) and it even has statically linked-in all font information needed to
display PDF files that come with no embedded fonts. That said, it does not
try to reinvent the wheel in every regard. For example, it relies on
common libraries such as zlib, libpng, jpeg, freetype, and openjpeg. Most
of them are already available on Genode. And the remaining libraries are rather
free-standing and easy to port. To illustrate the low degree of dependencies,
the MuPDF application on GNU/Linux depends on only 15 shared libraries. The
best thing about MuPDF from our perspective however, is its lean and clean API,
and the wonderfully simple example application. Thanks to this example, it was
a breeze to integrate the MuPDF engine with Genode's native framebuffer-session
and input-session interfaces. The effort needed for this integration work lies
in the order of less than 300 lines of code.
At the current stage, the MuPDF rendering engine successfully runs on Genode
in the form of a simple interactive test program, which can be started
via the 'libports/run/mupdf' run script. The program supports the basic key
handling required to browse through a multi-page PDF document
(page-up or enter -> next page, page-down or backspace -> previous page).
Improved terminal performance
=============================
The terminal component used to make all intermediate states visible to the
framebuffer in a fully synchronous fashion. This is an unfortunate behaviour
when scrolling through large text outputs. By decoupling the conversion of the
terminal state to pixels from the 'Terminal::write()' RPC function,
intermediate terminal states produced by sub sequential write operations do not
end up on screen one by one but only the final state becomes visible. This
improvement drastically improves the speed in situations with a lot of
intermediate states.
Noux support for fork semantics
===============================
Genode proclaims to be a framework out of which operating systems can be built.
There is no better way of putting this claim to the test than to use the
framework for building a Unix-like OS. This is the role of the Noux runtime
environment.
During the past releases, Noux evolved into a runtime environment that is able
to execute complex command-line-based GNU software such as VIM with no
modification. However, we cannot talk of Unix without talking about its
fundamental concept embodied in the form of the 'fork()' system call. We did
not entirely dismiss the idea of implementing 'fork()' into Noux but up to now,
it was something that we willingly overlooked. However, the primary goal of
Noux is to run the GNU userland natively on Genode. This includes a good deal
of programs that rely on fork semantics. We could either try to change all the
programs to use a Genode-specific way of starting programs or bite in the
bullet and implement fork. With the current release, we did the latter.
The biggest challenge of implementing fork was to find a solution that is not
tied to one kernel but one that works across all the different base platforms.
The principle problem of starting a new process in a platform-independent
manner is already solved by Genode in the form of the 'Process' API. But this
startup procedure is entirely different from the semantics of fork. The key to
the solution was Genode's natural ability to virtualize the access to low-level
platform resources. To implement fork semantics, all Noux has to do is to
provide local implementations of core's RAM, RM, and CPU session interfaces.
The custom implementation of the CPU session interface is used to tweak the
startup procedure as performed by the 'Process' class. Normally, processes
start execution immediately at creation time at the ELF entry point. For
implementing fork semantics, however, this default behavior does not work.
Instead, we need to defer the start of the main thread until we have finished
copying the address space of the forking process. Furthermore, we need to start
the main thread at a custom trampoline function rather than at the ELF entry
point. Those customizations are possible by wrapping core's CPU service.
The custom implementation of the RAM session interface provides a pool of RAM
shared by Noux and all Noux processes. The use of a shared pool alleviates the
need to assign RAM quota to individual Noux processes. Furthermore, the custom
implementation is needed to get hold of the RAM dataspaces allocated by each
Noux process. When forking a process, the acquired information is used to
create a shadow copy of the forking address space.
Finally, a custom RM service implementation is used for recording all RM
regions attached to the region-manager session of a Noux process. Using the
recorded information, the address-space layout can then be replayed onto a new
process created via fork.
With the virtualized platform resources in place, the only puzzle piece that
is missing is the bootstrapping of the new process. When its main thread is
started, it has an identical address-space content as the forking process but
it has to talk to a different parent entrypoint and a different Noux session.
The procedure of re-establishing the relationship of the new process to its
parent is achieved via a small trampoline function that re-initializes the
process environment and then branches to the original forking point via
setjmp/longjmp. This mechanism is implemented in the libc_noux plugin.
As the immediate result of this work, a simple fork test can be executed across
all base platforms except for Linux (Linux is not supported yet). The test
program is located at 'ports/src/test/noux_fork' and can be started with the
'ports/run/noux_fork.run' script.
Furthermore, as a slightly more exciting example, there is a run script for
running a bash shell on a tar file system that contains coreutils. By starting
the 'ports/run/noux_bash.run' script, you get presented an interactive bash
shell. The shell is displayed via the terminal service and accepts user input.
It allows you to start one of the coreutils programs such as ls or cat. Please
note that the current state is still largely untested, incomplete, and flaky.
But considering that Noux is comprised of less than 2500 lines of code, we are
quite surprised of how far one can get with such little effort.
Device drivers
##############
Driver improvements to accommodate dynamic (re-)loading
=======================================================
To support the dynamic probing of devices as performed by the new d3m
component, the ATAPI and USB device drivers have been enhanced to support the
subsequent closing and re-opening of sessions.
First bits of the d3m device-driver manager
===========================================
The abbreviation d3m stands for demo device-driver manager. It is our current
solution for the automated loading of suitable drivers as needed for running
Genode from a Live CD or USB stick. Because of the current narrow focus of d3m,
it is not a generic driver-management solution but a first step in this
direction. We hope that in the long run, d3m will evolve to become a generic
driver-management facility so that we can remove one of the "D"s from its name.
In the current form d3m solves the problems of merging input-event streams,
selecting the boot device, and dealing with failing network drivers.
When using the live CD, we expect user input to come from USB HID devices or
from a PS/2 mouse and keyboard. The live system should be operational if at
least one of those sources of input is available. In the presence of multiple
sources, we want to accumulate the events of all of them.
The live CD should come in the form of a single ISO image that can be burned onto a CDROM or
alternatively copied to an USB stick. The live system should boot fine in both
cases. The first boot stage is accommodated by syslinux and the GRUB boot
loader using BIOS functions. But once Genode takes over control, it needs to
figure out on its own from where to fetch data. A priori, there is no way to
guess whether the ATAPI driver or the USB storage driver should be used.
[image d3m_what_next]
Therefore, d3m implements a probing mechanism that starts each of the drivers,
probes for the presence of a particular file on an iso9660 file system.
[image d3m_probing]
Once d3m observes a drivers that is able to successfully access the magic file,
it keeps the driver and announces the driver's service to its own parent. For
the system outside of d3m, the probing procedure is completely transparent. D3m
appears to be just a service that always provides the valid block session for
the boot medium.
[image d3m_ready]
The network device drivers that we ported from the iPXE project cover the
range of most common wired network adaptors, in particular the E1000 family.
But we cannot presume that a computer running the live system comes equipped
with one of the supported devices. If no supported network card could be
detected the driver would simply fail. Applications requesting a NIC session
would block until a NIC service becomes available, which won't happen. To
prevent this situation, d3m wraps the NIC driver and provides a dummy NIC
service in the event the drivers fails. This way, the client application won't
block infinitely but receive an error on the first attempt to use the NIC.
ACPI support
============
To accommodate kernels like Fiasco.OC or NOVA that take advantage of x86's
APIC, we have introduced a simple ACPI parser located at 'os/src/drivers/acpi'.
The server traverses the ACPI tables and sets the interrupt line of devices
within the PCI config space to the GSIs found in the ACPI tables. Internally it
uses Genode's existing PCI driver as a child process for performing PCI access
and, in turn, announces a PCI service itself.
For obtaining the IRQ routing information from the ACPI tables without
employing a full-blown ACPI interpreter, the ACPI driver uses an ingenious
technique invented by Bernhard Kauer, which is described in the following
paper:
:[http://os.inf.tu-dresden.de/papers_ps/tr-atare-2009.pdf - ATARE - ACPI Tables and Regular Expressions]:
_TU Dresden technical report TUD-FI09-09, Dresden, Germany, August 2009_
:Usage:
Start the 'acpi_drv' in your Genode environment. Do not start the 'pci_drv'
since this will be used as a slave of the 'acpi_drv'. You still must load the
'pci_drv' in your boot loader. To integrate the ACPI driver into your boot
configuration, you may take the following snippet as reference:
!<start name="acpi">
! <resource name="RAM" quantum="2M"/>
! <binary name="acpi_drv"/>
! <provides><service name="PCI"/></provides>
! <route>
! <service name="ROM"> <parent/> </service>
! <any-service> <any-child/> <parent/> </any-service>
! </route>
!</start>
:Limitations and known issues:
Currently there is no interface to set the interrupt mode for core's IRQ
sessions (e.g., level or edge triggered). This is required by Fiasco.OCs kernel
interface. We regard this as future work.
Platform support
################
Fiasco.OC microkernel
=====================
The support for the Fiasco.OC base platform is still lacking proper handling
for releasing resources such as kernel capabilities. Although this is a known
issue, we underestimated the reach of the problem when Genode's signal API is
used. Each emitted signal happens to consume one kernel capability within core,
ultimately leading to a resource outage when executing signal-intensive code
such as the packet-stream interface. The current release comes with an interim
solution. To remedy the acute problem, we extended the 'Capability_allocator'
class with the ability to register the global ID of a Genode capability so
that the ID gets associated with a process-local kernel capability. Whenever
a Genode capability gets unmarshalled from an IPC message, the
capability-allocator is asked, with the global ID as key, whether the
kernel-cap already exists. This significantly reduces the waste of
kernel-capability slots.
To circumvent problems of having one and the same ID for different kernel
objects, the following problems had to be solved:
* Replace pseudo IDs with unique ones from core's badge allocator
* When freeing a session object, free the global ID _after_ unmapping
the kernel object, otherwise the global ID might get re-used in some
process and the registry will find a valid but wrong capability
for the ID
Because core aggregates all capabilities of all different processes, its
capability registry needs much more memory compared to a regular process.
By parametrizing capability allocators differently for core and non-core
processes, the global memory overhead for capability registries is kept
at a reasonable level.
Please note that this solution is meant as an interim fix until we have
resolved the root of the problem, which is the proper tracking and releasing
of capability selectors.
Linux
=====
Linux is one of the two original base platforms of Genode. The original
intension behind supporting Linux besides a microkernel was to facilitate
portability of the API design and to have a convenient testing environment for
platform-independent code. Running Genode in the form of a bunch of plain Linux
processes has become an invaluable feature for our fast-paced development.
To our delight, we lately discovered that the use of running Genode on Linux
can actually go far beyond this original incentive. Apparently, on Linux, the
framework represents an equally powerful component framework as on the other
platforms. Hence, Genode has the potential to become an attractive option for
creating complex component-based user-level software on Linux.
For this intended use, however, the framework has to fulfill the following
additional requirements:
* Developers on Linux expect that their components integrate seamlessly with
their existing library infrastructure including all shared libraries
installed on Linux.
* The use of a custom tool chain is hard to justify to developers who regard
Genode merely as an application framework. Hence, a way to use the normal
tool chain as installed on the Linux host system is desired.
* Application developers are accustomed with using GDB for debugging and expect
that GDB can be attached to an arbitrary Genode program in an intuitive way.
Genode's original support for Linux as base platform did not meet those
expectations. Because Genode's libc would ultimately collide with the Linux
glibc, Genode is built with no glibc dependency at all. It talks to the kernel
directly using custom kernel bindings. In particular, Genode threads are created
directly via the 'clone()' system call and thread-local storage (TLS) is managed
in the same way as for the other base platforms. This has two implications.
First, because Genode's TLS mechanism is different than the Linux TLS
mechanism, Genode cannot be built with the normal host tool chain. This
compiler would generate code that would simply break on the first attempt to
use TLS. We solved this problem with our custom tool chain, which is configured
for Genode's needs. The second implication is that GDB is not able to handle
threads created differently than those created via the pthread library. Even
though GDB can be attached to each thread individually, the debugger would not
correctly handle a multi-threaded Genode process as a multi-threaded Linux
program. With regard to the use of Linux shared libraries, Genode used to
support a few special programs that used both the Genode API and Linux
libraries. Those programs (called hybrid Linux/Genode programs) were typically
pseudo device drivers that translate a Linux API to a Genode service. For
example, there exists a framebuffer service that uses libSDL as back end.
Because those programs were a rarity, the support by the build system for such
hybrid targets was rather poor.
Fortunately, all the problems outlined above could be remedied pretty easily.
It turns out that our custom libc is simply not relevant when Genode is
used as plain application framework on Linux. For this intended use, we always
want to use the host's normal libc. This way, the sole reason for using plain
system calls instead of the pthread library vanishes, which, in turn,
alleviates the need for a custom tool chain. Genode threads are then simply
pthreads compatible with the TLS code as emitted by the host compiler and
perfectly recognised by GDB. With the surprisingly little effort of creating a
new implementation of Genode's thread API to target pthreads instead of using
syscalls, we managed to provide a decent level of support for using Genode as
user-level component framework on Linux.
These technical changes alone, however, are not sufficient to make Genode
practical for real-world use. As stated above, the few hybrid Linux/Genode
programs used to be regarded as some leprous artifacts. When using Genode as
Linux application framework, however, this kind of programs are becoming the
norm rather than an exception. For this reason, we introduced new support for
such hybrid programs into the build system. By listing the 'lx_hybrid'
library in the 'LIBS' declaration of a target, this target magically becomes a
hybrid Linux/Genode program. It gets linked to the glibc and uses pthreads
instead of direct syscalls. Furthermore, host libraries can be linked to the
program by stating their respective names in the 'LX_LIBS' variable. For an
example, please refer to the libSDL-based framebuffer at
'os/src/drivers/framebuffer/sdl/target.mk'.
To enforce the build of all targets as hybrid Linux/Genode programs, the build
system features the 'alyways_hybrid' 'SPEC' value. To make it easy to create a
build directory with all targets forced to be hybrid, we have added the new
'lx_hybrid_x86' platform to the 'create_builddir' tool.
OKL4
====
:Sending an invalid-opcode exception IPC on OKL4:
When an invalid opcode gets executed, OKL4 switches to the kernel debugger
console instead of sending an exception IPC to the userland. We fixed this
problem by removing the code that invokes the debugger console from the kernel.
:Hard-wire OKL4 elfweaver to Python 2:
Recent Linux distributions use Python version 3 by default. But OKL4's
elfweaver is not compatible with this version of Python. For this reason, we
introduced a patch for pinning the Python version used by elfweaver to
version 2.
Both patches get automatically applied when preparing the 'base-okl4'
repository via 'make prepare'.
Build system and tools
######################
:Facility for explicitly building all libraries:
During its normal operation, the build system creates libraries as mere side
effects of building targets. There is no way to explicitly trigger the build of
libraries only. However, in some circumstances (for example for testing the
thorough build of all libraries), a mechanism for explicitly building libraries
would be convenient. Hence we introduced this feature in the form of the pseudo
target located at 'base/src/lib/target.mk'. By issuing 'make lib' in a build
directory, this target triggers the build of all libraries available for the
given platform.

1029
doc/release_notes-12-05.txt
View File

File diff suppressed because it is too large Load Diff

665
doc/release_notes-12-08.txt
View File

@@ -1,665 +0,0 @@
===============================================
Release notes for the Genode OS Framework 12.08
===============================================
Genode Labs
With Genode 12.08, the project focused on platform support. It enters the world
of OMAP4-based ARM platforms, revived and vastly enhanced the support for the
NOVA hypervisor, and becomes able to run directly on ARM platforms without the
need for an underlying kernel.
The new 'base-hw' platform is a deviation from Genode's traditional approach to
complement existing kernels with user-land infrastructure. It completely leaves
the separate kernel out of the picture and thereby dwarfs the base line of the
trusted computing base of Genode-based systems to approximately the half. The
new base platform is described in Section [Genode on naked ARM hardware].
Speaking of base platforms, we are happy to have promoted the NOVA hypervisor
to a first-class citizen among the base platforms. During the last months, this
kernel underwent fundamental changes regarding its mode of development and its
feature set. This prompted us to vastly improve Genode's support for this
platform and leverage its unique features. If considering the use of Genode on
x86-based hardware, NOVA has become a very attractive foundation. Section
[Embracing the NOVA Hypervisor] describes the NOVA-specific changes.
The improvement of platform support with the current release does not entail
the base platforms only but extends to profound additions of device drivers, in
particular for the ARM-based OMAP4 SoC as used on the popular Pandaboard. We
are proud to announce the availability of device drivers for HDMI output,
SD-card, USB HID, and networking for this platform.
Beyond the low-level platform improvements, the new version comes with several
new services, optimizations of existing components, and new ported libraries.
In particular, the Noux runtime has reached a point where we can principally
execute serious networking applications such as the Lynx web browser natively
on Genode. Another example is the new FFAT-based file-system service, which
makes persistent storage available via Genode's file-system interface. By
combining this new service with existing components such as the partition
service, Noux, or the file-system plugin of the libc, a lot of new application
scenarios become available. Thanks to these new components, the framework has
become able to perform on-target debugging via GDB running in Noux, or host
the genode.org website via the lighttpd web server,
:What about the road map?:
Those of you who track the milestones laid out in our [http:/about/road-map - road map]
may wonder how Genode 12.08 relates to the stated goals. In fact, several
points of the road map haven't received the attention as originally planned.
As an explanation, let us quote the paragraph right atop of the road-map page:
"The road map is not fixed. If there is commercial interest of pushing the
Genode technology to a certain direction, we are willing to revisit our plans."
Well, this is what happened. So we traded the work on the tiled window manager,
the Intel wireless driver, and SMP support for the work on the platform topics
outlined above. Nevertheless, we stick to our overall plan to turn Genode into
a general-purpose OS that is fit for use by its developers by the end of the
year. If looking closely at the additions that come with the current release,
it will become apparent how well they fit into the big picture.
Genode on naked ARM hardware
############################
One of Genode's most distinguishing properties is the ability to use the framework
on top of a range of different kernels. This way, users of the framework
benefit from the wide variety of features provided by those kernels while
only dealing with a single API and configuration concept. For example, we
frequently find ourselves using the Linux kernel as base platform while
developing services, interfaces, and protocol stacks. By being able to start
Genode as a regular program, we effectively eliminate the reboot-time for each
test run and enjoy using commodity debugging and profiling tools. On the other
hand, if high security is a concern, NOVA and Fiasco.OC provide
capability-based security at kernel-level. So the use of one of those kernels
is desirable. Genode allows for switching between those vastly different
kernels almost seamlessly.
In general, a Genode system consists of a kernel, Genode's core, and the
largely generic components on top of core. Core abstracts away the
peculiarities of the respective kernel and provides a unified API to the
components on top. From the application's point of view both kernel and core
are always at the root of the process tree and thereby are a inherent part of
the application's trusted computing base (TCB). The distinction of both
programs is almost superficial.
Since both the kernel and core must be ultimately trusted, the complexity of
both programs is critical for each Genode-based system. On our quest for
minimizing the TCB complexity so far, however, we did not question the role of
the kernel as an inherent part of the TCB and focused our attention to the
software stack on top. However, with more and more kernels entering the
picture, we identified that there is typically a considerable overlap in
functionality between kernel and core. For example, both need to know about
address spaces and their relationship to physical memory objects. Most kernels
keep track of memory mappings in an in-kernel database. Core also needs to keep
track of this information. Consequently, we found several information
replicated without a clear benefit. With this comes a seemingly significant
redundancy of code for data structures, allocators, and utility functions.
Furthermore, there exists a class of problems that must be solved by the kernel
and core alike. In particular the resource management of dynamically allocated
in-kernel objects respectively in-core objects. Whereas core uses Genode's
resource-trading concept to solve this problem, most kernels lack a good
solution for the management of in-kernel resources and are consequently prone
to resource exhaustion problems.
Out of these observations, the idea was born to explore the opportunities of
merging both programs into one and thereby eliminating the redundancies. Our
first attempt to go into this direction was the 'base-mb' platform, which
enabled us to run Genode on the Xilinx MicroBlaze softcore CPU. With this
experiment, we gained confidence that the approach is generally feasible. So we
took on the challenge to implement the idea of a hybrid kernel/core on a more
complex architecture namely ARM Cortex-A9.
The 'base-hw' platform introduced with the current release is the intermediate
result of our experiment. With this base platform, core plays the role of core
and the kernel within one program. A few code paths that require execution in
privileged mode are executed in kernel mode whereas most code paths are
executed in user mode. Both user mode code and kernel mode code run in the same
address space. The kernel portion merely provides a few basic mechanisms
without performing complex operations such as dynamic memory allocations. For
example, if core is requested to create a new thread via core's CPU session
interface, the user-level code within core allocates a KTCB (kernel thread
control block) and UTCB (user-level thread-control block) from the physical
memory allocator and passes both physical addresses to the kernel function that
spawns the actual thread. This way, we can employ Genode's resource-trading
concept for managing typical kernel resources.
The experiment turned out to be a great success. Traditionally, we would account
at least 10,000 lines of code (LOC) for the kernel. Most kernels are actually
much larger than that. Core comes at a complexity of another 10,000 LOC. So
both kernel and core make up a base line of TCB complexity of more than 20,000
LOC. By co-locating core with the kernel, we end up with a program of just
about 13,000 LOC. The vast reduction of TCB complexity compared to having
kernel and core as separate programs strikes us.
The 'base-hw' version of core supports the complete Genode API covering support
for user-level device drivers, synchronous RPCs, asynchronous notifications,
shared memory, and managed dataspaces. It is thereby able to execute the
sophisticated Genode scenarios on top including the GUI, the dynamic linker,
and user-level device drivers. That said, we regard the current version still
as work in progress. We successfully use it as an experimentation platform for
ongoing research activities (i.e., for exploring ARM TrustZone) but some
important features such as capability-based security are not yet implemented.
:Using the base-hw platform:
The new base platform is fully integrated with Genode's build system.
When listing the supported base platforms via the 'tool/create_builddir' tool,
you will see the new 'hw_panda_a2', 'hw_vea9x4', 'hw_pbxa9' choices of
build-directory templates. The latter platform enables you to run a
'base-hw' Genode system on Qemu.
[http://genode.org/documentation/platforms/hw - Learn more about using the new base-hw platform...]
For running Genode directly on the Pandaboard, please refer to the
[http://genode.org/documentation/platforms/hw_panda_a2 - Pandaboard-specific documentation...]
Embracing the NOVA Hypervisor
#############################
NOVA is a so-called microhypervisor for the x86 architecture. It combines the
principles of microkernels with capability-based security and hardware-assisted
virtualization. Among the various base platforms supported by Genode, NOVA's
kernel interface stands out for being extremely minimalistic and orthogonal,
even by microkernel standards.
Genode has supported NOVA as base platform since 2010. But because we used NOVA
solely for sporadic research activities and NOVA's lack of a regular release
schedule, the framework's platform support received only little attention. This
has changed now. NOVA's main developer Udo Steinberg moved from TU Dresden to
Intel Labs where he leads the development of NOVA as a true Open-Source
project. In fact, the code is now being hosted at GitHub:
:[https://github.com/IntelLabs/NOVA]:
NOVA hypervisor at GitHub
Since its move to GitHub, the hypervisor has already seen significant
improvements. The repository is continuously updated, which enables us to stay
in a close feedback loop with the NOVA developers. This change of how NOVA's
development is conducted ignited our renewed interest in promoting this
platform to a first-level citizen of our framework. The first noteworthy
improvement is the recently added 64-bit support of NOVA. We enabled Genode to
work with both variants of the kernel - 32 bit and 64 bit.
But this was just the first step. The second major change addresses the
allocation of kernel resources. Early versions of the hypervisor allowed each
process to create kernel objects and thereby indirectly consume the limited
memory resources of the kernel. This is perfectly fine for a research project
but it becomes a potential denial-of-service problem in real-world use cases.
For this reason, newer versions introduced the ability to retain the allocation
of kernel objects within a trusted component only. In the Genode world, this
component is naturally core. Even though NOVA still lacks a flexible concept for
kernel-resource management as of now, Genode has become able to use NOVA
without suffering the inherent resource management limitation. This is achieved
because core is able to arbitrate the allocation of kernel resources.
The third fundamental change is the abolishment of the last traces of global
names in a NOVA-based Genode system. There are no thread IDs, object IDs, or
any other kind of globally meaningful names. Each process has a local view on
(a small part of) the system only. If a process interacts with another process,
the kernel translates the references to remote objects from one namespace to
the other. The security implications are eminent. First, a process can only
interact with or refer to objects for which it has a name, which vastly reduces
problems of ambient authority. Second, because the kernel translates names, it
becomes impossible to forge object identities. If a process tried to pass a
forged object reference to another process, the translation would simply fail,
rendering the attack ineffective.
The described changes do not come without issues, though. To make the NOVA
kernel fit with Genode's requirements, minor patches of the hypervisor are
needed. The patches are located at 'base-nova/patches/'. However, those patches
are meant as interim solutions until we find mechanisms that fit well with the
design of the hypervisor and also fulfil our requirements.
So far, we greatly enjoyed the revived collaboration with the NOVA developers
and congratulate Udo Steinberg for the new mode of development of the
hypervisor.
Base framework
##############
In the following, we describe changes of the base API that may affect users of
the framework.
:Allocation of DMA buffers:
We extended the RAM session interface with the ability to allocate DMA buffers.
The client specifies the type of RAM dataspace to allocate via the new 'cached'
argument of the 'Ram_session::alloc()' function. By default, 'cached' is true,
which corresponds to the common case and the original behavior. When setting
'cached' to 'false', core takes the precautions needed to register the memory
as uncached in the page table of each process that has the dataspace attached.
Currently, the support for allocating DMA buffers is implemented for Fiasco.OC
only. On x86 platforms, it is generally not needed. But on platforms with more
relaxed cache coherence (such as ARM), user-level device drivers should always
use uncacheable memory for DMA transactions.
:MMIO framework improvements:
As we find ourselves increasingly using the 'Register' and 'Mmio' templates
provided by 'util/register.h' and 'util/mmio.h' for dealing with memory-mapped
devices, we extended the utilities with support for 64-bit registers and a new
interface for polling bit states. The latter functionality is provided by the
new 'wait_for' function template. To decouple the MMIO-related utility code
from an actual timer facility, the function takes a so-called 'delayer' functor
as argument. This way the user of the MMIO framework is able to pick a timer
facility that fits best with the device.
:New 'memcpy' implementation:
The memory-copy functions provided by 'util/string.h' are extremely simple
and arguably slow, particularly on platforms where byte-wise copy operations
are not supported by the CPU (i.e., ARM). Hence, we have added a CPU-specific
memcpy function ('memcpy_cpu') to 'cpu/string.h', which enables us to
provide optimized implementations. So far, we did so for the ARM architecture.
Low-level OS infrastructure
###########################
FFat-based file-system service
==============================
With the previous release, we introduced Genode's file-system interface
accompanied with a simple in-memory file-system service. With the addition of
'ffat_fs', the current release adds the first persistent file system to the
framework. The service is located at 'libports/src/server/ffat_fs'. It uses
Genode's 'Block::Session' interface as back end. Therefore, it can be combined
with any of Genode's block-device drivers and the partition service called
'part_blk'. To see the new 'ffat_fs' service in action, please refer to the new
'libports/run/libc_ffat_fs.run' script.
On the course of our work on the 'ffat_fs' service, we enabled support for long
file names in libffat and added 'lseek' support to the 'libc_ffat' plugin.
TAR-based file-system service
=============================
The new 'tar_fs' service located at 'os/src/server/tar_fs' provides a read-only
file-system session interface by reading data from a TAR archive, which, in
turn, is fetched from a ROM service. By combining 'tar_fs' with the 'libc_fs'
plugin, it becomes easy to provide customized pseudo file systems to individual
Genode programs. For example, one instance of 'tar_fs' containing a static
website and a web-server configuration can be provided as file system to a web
server. The configuration is similar to the patterns known from the 'tar_rom'
and 'ram_fs' servers:
! <config>
! <archive name="tar_archive.tar" />
! <policy label="label_of_client" root="/rootdir/for/client" />
! </config>
The policy node allows for assigning different parts of one TAR archive to
different clients. For a practical usage example of 'tar_fs', please refer to
the 'libports/run/libc_fs_tar_fs.run' script.
Terminal improvements
=====================
Our work on running a growing number of command-line-based Unix programs via
Noux prompted us to improve our terminal implementation as needed. To ease
debugging for terminal colors, we changed the previous default color scheme to
fully saturated combinations of red, green, and blue. Albeit this looks quite
painful on the eyes, it is easier to spot wrong colors when using a program
that uses ncurses, for example Lynx. Furthermore, we added the handling of
sgr0 and sgr escape sequences and thereby enabled Lynx to become almost
usable when running within Noux.
Terminal cross-link service
===========================
The 'Terminal::Session' interface gets increasingly popular within Genode.
It is used by the UART drivers, the graphical terminal, GDB monitor, the TCP
terminal, and Noux. For most of these programs, their respective client or
server role is quite clear but we find ourselves tempted to combine components
in unusual ways. For example, to let Noux run an instance of GDB, which operates
on a terminal via a virtual character device. For remote debugging, GDB plays
the role of a terminal client and the UART driver plays the role of the server.
But when running GDB monitor on the same machine, GDB monitor will also
expect to play the role of the client. In order to connect GDB monitor
to a local instance of GDB, both of them being terminal clients, we need an
adapter component. This is where the new terminal cross-link service enters
the picture. It plays the role of a terminal server between exactly two
clients. The output of one client ends up as input to the other and vice
versa. Data sent to the server gets stored in a buffer of 4096 bytes (one
buffer per client). As long as the data to be written fits into the buffer, the
'write()' call returns immediately. If no more data fits into the buffer, the
'write()' call blocks until the other client has consumed some of the data from
the buffer via the 'read()' call. The 'read()' call never blocks. A signal
receiver can be used to block until new data is ready for reading.
The new terminal crosslink can be tested via the 'os/run/terminal_crosslink.run'
script. It is also used for the just mentioned on-target debugging scenario
demonstrated by the 'ports/run/noux_gdb.run' script.
DMA-aware and optimized packet streams
======================================
Motivated by our work on OMAP4 platform support, we introduced API extensions
for handling of DMA buffers to the following interfaces:
:'Attached_ram_dataspace':
The convenience utility for allocating and locally mapping a RAM dataspace
has been enhanced with the 'cached' constructor argument, which is true
by default. When using 'Attached_ram_dataspace' for allocating DMA buffers,
this argument should be set to false.
:Block and network packet stream:
The 'Block::Session' and 'Nic::Session' interfaces use Genode's packet stream
facility for transferring bulk payload between processes. A packet stream
combines shared memory with asynchronous notifications and thereby facilitates
the use of batched packet processing. To principally enable zero-copy semantics
for device drivers, the packet-stream buffer is now explicitly allocated as DMA
buffer. This clears the way to let the SD-card driver direct DMA transactions
right into the packet stream buffer. Consequently, when attaching the SD-card
driver directly to a file system, there is no copy of payload needed.
The 'Nic::Session' interface has further been improved by using a fast
bitmap allocator for allocations within the packet-stream buffer. This is
possible because networking packets have the MTU size as an upper limit.
In contrast to the 'Block::Session' interface where requests are relatively
large, 'Nic::Session' packets are tiny, and thus, greatly benefit from the
optimized allocator.
Libraries and applications
##########################
C runtime
=========
:File I/O:
We complemented our C runtime with support for the 'pread', 'pwrite', 'readv',
and 'writev' functions. The 'pread' and 'pwrite' functions are shortcuts for
randomly accessing different parts of a file. Under the hood, the functions are
implemented via 'lseek' and 'read/write'. To provide the atomicity of the
functions, a lock guard prevents the parallel execution of either or both
functions if called concurrently by multiple threads. The 'readv' and 'writev'
functions principally enable the chaining of multiple I/O requests.
Furthermore, we added 'ftruncate', 'poll', and basic support for (read-only)
mmapped files to the C runtime.
:Libc RPC framework headers:
Certain RPC headers of the libc are needed for compiling 'getaddrinfo.c'.
Unfortunately that means we have to generate a few header files, which we do
when we prepare the libc.
New and updated 3rd-party libraries
===================================
:Expat:
[http://expat.sourceforge.net - Expat] is an XML parsing library. The port of
this library was motivated by our goal to use the GNU debugger for on-target
debugging. GDB depends on this library.
:MPC and GMP:
We complemented our existing port of the
[http://gmplib.org - GNU multiple precision arithmetic library (libgmp)] with
support for the x86_64 and ARM architectures. This change combined with the
port of the [http://www.multiprecision.org/index.php?prog=mpc - MPC library]
enables us to build the Genode tool chain for these architectures.
:OpenSSL:
Our port of OpenSSL has been updated to version 1.0.1c. Because libcrypto
provides certain optimized assembler functions, which unfortunately are not
expressed with position-independent code, we removed this assembler code and
build libcrypto with '-DOPENSSL_NO_ASM'. Because the assembler code is not
needed anymore, its generation is also removed from 'openssl.mk'.
:Light-weight IP stack (lwIP):
We enabled the lwIP TCP/IP stack for 64-bit machines and updated the library to
version 1.4.1-rc1. With the new version, the call of 'lwip_loopback_init' is
not needed anymore because lwIP always creates a loopback device. Hence, we
will be able to remove the 'libc_lwip_loopback' in the future. For now, we keep
it around so we currently do not need to update the other targets that depend
on it.
:PCRE:
[http://www.pcre.org/ - PCRE] is a library for parsing regular rexpressions. We
require this library for our ongoing work on porting the lighttpd webserver.
Lighttpd web server
===================
The [http://www.lighttpd.net/ - Lighttpd] web server has been added to the
'ports' repository. The port runs as a native Genode application and ultimately
clears the way to hosting the genode.org website on Genode. To test drive this
scenario, please give the 'ports/run/genode_org.run' script a try.
At the current stage, the port is still quite limited. For example, it does not
make use of non-blocking sockets yet. But the 'genode_org.run' run script
already showcases very well how simple a Genode-based web-server appliance can
look like.
Device drivers
##############
OMAP4 platform drivers
======================
:HDMI output:
The new HDMI driver at 'os/src/drivers/framebuffer/omap4' implements Genode's
'Framebuffer::Session' interface by using the HDMI output of OMAP4. The current
version sets up a fixed XGA screen mode of 1024x768 with the RGB565 pixel
format.
:SD-card:
The new SD card driver at 'os/src/drivers/sd_card/omap4' allows the use of a
HDSD card with the Pandaboard as block service. The driver can be tested using
the 'os/run/sd_card.run' script. Because it implements the generic
'Block::Session' interface, it can be combined with a variety of other
components such as 'part_blk' (for accessing individual partitions) or
'ffat_fs' for accessing a VFAT file system on the SD card.
The driver uses the master DMA facility of the OMAP4 SD-card controller, which
yields to good performance at low CPU utilization. The throughput matches (and
in some cases outperforms) the Linux kernel driver. In the current version,
both modes of operation PIO and DMA are functional. However, PIO mode is
retained for benchmarking purposes only and will possibly be removed to further
simplify the driver.
:USB HID:
The OMAP4-based Pandaboard relies on USB for attaching input devices.
Therefore, we need a complete USB stack to enable the interactive use of the
board. Instead of implementing a USB driver from scratch, we built upon the USB
driver introduced with the Genode release 12.05. This driver was ported from the
Linux kernel.
:Networking:
The Pandaboard realizes network connectivity via the SMSC95xx chip attached to
the USB controller. Therefore, we enhanced our USB driver with support for USB
net and the smsc95xx driver. In addition to enabling the actual device-driver
functionality, the USB stack has received much attention concerning performance
optimizations. To speed up the allocation of SKBs, we replaced the former
AVL-tree based allocator with a fast bitmap allocator. For anonymous
allocations, we introduced a slab-based allocator. Furthermore, we introduced
the distinction between memory objects that are subjected to DMA operations
from non-DMA memory objects. The most profound conceptual optimization is the
use of transmit bursts by the driver. The Linux kernel, which our driver
originates from, does not provide an API for transmitting multiple packets as a
burst. For our driver, however, this optimization opportunity opened up thanks
to Genode's packet stream interface, which naturally facilitates the batching
of networking packets. So the driver has all the information needed to create
burst transactions.
USB driver
==========
By testing our new USB driver on a variety of real PC hardware, we discovered
several shortcomings, which we resolved. In particular, we added support for
more than one UHCI controller, make sure that the 'PIRQ' bit in the legacy
support register (PCI config space) of the UHCI controller is enabled and that
the 'Trap on IRQ' bit is disabled.
With those modifications in place, the USB driver works reliably on the tested
platforms.
Runtime environments
####################
Noux
====
Noux enables the easy reuse of unmodified GNU software on Genode by providing
a Unix-like kernel interface as user-level service. Because Noux is pivotal for
our plan to use Genode for productive work, we significantly enhanced and
complemented its feature set.
:Noux on ARM and x86_64:
For keeping the scope of the development manageable, the initial version of
Noux was tied to the x86_32 platform. This was not a principal limitation of
the approach but rather an artificial restriction to keep us focused on
functionality first. Now that Noux reaches a usable state, we desire to use it
on platforms other than x86_32. The current release enables Noux for the 64-bit
x86 and ARM architectures.
The level of support is pretty far-reaching and even includes the building and
execution of the Genode tool chain on those platforms. In the process of
enabling these platforms, we updated the Noux package for GCC to version 4.6.1,
which matches the version of the current Genode tool chain.
:Terminal file system:
Noux supports the concept of stacked file systems. The virtual file system
is defined at the start of a Noux instance driven by the static Noux
configuration. This way, arbitrary directory structures can be composed out
of file-system sessions and TAR archives. The VFS concept allows for the
easy addition of new file system types. To allow programs running in a Noux
instance to communicate over a dedicated terminal session, we added a new
file-system type that corresponds to a virtual character device node attached
to a terminal session.
:GDB running in the Noux environment:
With the terminal file system in place, we are ready to execute GDB within
Noux and let it talk to a GDB monitor instance over the terminal session
interface. From GDB's point of view, the setup looks like a remote debugging
session. But in reality both the debugging target and GDB reside in different
subtrees of the same Genode system.
:Executing shell scripts:
By inspecting the program specified to the execve system call, Noux has become
able to spawn scripts that use the '#!' syntax. If such a file is detected, it
executes the specified interpreter instead and passes the arguments specified
after the '#!' marker, followed by command-line arguments.
:Networking support:
Our work on porting various networking tools to Noux triggers us to steadily
improve the networking support introduced with Genode 12.05. In particular, we
added proper support for DNS resolving, which enables us to execute the
command-line based Lynx web browser within Noux.
:User information:
Because there are certain programs, which need the information that is stored
in 'struct passwd', we introduced configurable user information support to
Noux. One can set the user information via the '<user>' node in the Noux
config:
! <config>
! <user name="baron" uid="1" gid="1">
! <shell name="/bin/bash" />
! <home name="/home" />
! </user>
! ...
! </config>
When '<user>' is not specified, default values are used. Currently these
are 'root', 0, 0, '/bin/bash', '/'. Note that this is just a single user
implementation because each Noux instance has only one user or rather one
identity and there will be no complete multi-user support in Noux. If you need
different users, just start new Noux instances for each of them.
:New '/dev/null' and '/dev/zero' pseudo devices:
These device are mandatory for most programs (well, at least null is required
to be present for a POSIX compliant OS, which Noux is actually not). But for
proper shell-script support we will need them anyway. Under the hood, both
pseudo devices are implemented as individual file-systems and facilitate Noux's
support for stacked file systems. The following example configuration snippet
creates the pseudo devices under the '/dev' directory.
! <config>
! <fstab>
! <dir name="dev" >
! <null /> <zero />
! </dir>
! ...
! <fstab>
! ...
! </config>
Vancouver
=========
The comprehensive rework of the NOVA base platform affected the Genode version
of the Vancouver virtual machine monitor as this program used to speak directly
to the NOVA kernel. Since no kernel objects can be created outside of core
anymore, the Vancouver port had to be adjusted to solely use Genode interfaces.
L4Linux
=======
To improve the stability and performance of L4Linux on OMAP4 platforms, we
reworked parts of the Genode-specific stub drivers, in particular the
networking code. Among the improvements are the use of a high-performance
allocator for networking packets, improved IRQ safety of IPC calls (to
the Genode world), and tweaks of the TCP rmem and wmem buffer sizes to
achieve good TCP performance when running Linux with little memory.
Furthermore, we added two ready-to-use run scripts residing within
'ports-foc/run' as examples for executing L4Linux on the OMAP4-based
Pandaboard. The 'linux_panda.run' script is meant as a blue print for
experimentation. It integrates one instance of L4Linux with the native SD-card
driver, the HDMI driver, and the USB HID input driver. The
'two_linux_panda.run' script is a more elaborative example that executes two
instances of L4Linux, a block-device test, and a simple web server. Each of
the L4Linux instances accesses a different SD-card partition whereas the
block-device test operates on a third partition.

735
doc/release_notes-12-11.txt
View File

@@ -1,735 +0,0 @@
===============================================
Release notes for the Genode OS Framework 12.11
===============================================
Genode Labs
The central theme of version 12.11 of the Genode OS Framework is
self-hosting Genode on Genode. With self-hosting, we understand the execution of the
entire Genode build system within the Genode environment. There are two motivations
for pursing this line of work. First, it is a fundamental prerequisite for the
Genode developers to move towards using Genode as a day-to-day OS. Of course,
this prerequisite could be realized using one of the available virtualization
solutions. For example, we could run L4Linux on top of Genode on the Fiasco.OC
kernel and use the Genode build system from within an L4Linux instance.
However, this defeats the primary incentive behind Genode to reduce system
complexity. By having both Genode and L4Linux in the picture, we would indeed
increase the overall complexity in configuring, maintaining, and using the
system. Therefore, we would largely prefer to remove the complex Linux user
land from the picture. The second motivation is to prove that the framework and
underlying base platforms are suited and stable enough for real-world use.
If the system is not able to handle a workload like the build system,
there is little point in arguing about the added value of having a
microkernel-based system over current commodity OSes such as GNU/Linux.
We are happy to have reached the state where we can execute the unmodified
Genode build system directly on Genode running on a microkernel. As the
build system is based on GNU utilities and the GNU compiler collection,
significant effort went into the glue between those tools and the Genode API.
Section [Building Genode on Genode] provides insights into the way we achieved
the goal and the current state of affairs.
Along with the work on bringing the build system to Genode came numerous
stability improvements and optimizations all over the place, reaching from the
respective kernels, over the C runtime, the file-system implementations, memory
allocators, up to the actual programs the tool chain is composed of. Speaking
of the tool chain, the official Genode tool chain has been updated from GCC
version 4.6.1 to version 4.7.2. Thereby, all 3rd-party code packages were
subjected to testing and fixing activities.
For running the build system, the project currently focuses on NOVA and
Fiasco.OC as base platforms. However, our custom kernel platform for the ARM
architecture has also received significant improvements. With added support for
Freescale i.MX and Texas Instruments OMAP4, this platform proved to be very
well adaptable to new SoCs whereas new cache handling brings welcome
performance improvements. Furthermore, we have added experimental support for
ARM TrustZone technology, which principally enables the execution of Genode in
the so-called secure world of TrustZone while executing Linux in the so-called
normal world.
As we discovered the increasing interest in using Genode as a middleware
solution on Linux, we largely revisited the support for this kernel platform
and discovered amazing new ways to align the concept of Genode with the
mechanisms provided by the Linux kernel. Section [Linux] provides a summary
of the new approaches taken for supporting this platform.
Functionality-wise, the new version introduces support for audio drivers of
the Open Sound System, a new OMAP4 GPIO driver, improvements of the graphical
terminal, and the initial port of an SSH client.
Building Genode on Genode
#########################
On the Genode developer's way towards using Genode as a day-to-day OS, the
ability to execute the Genode build system within the Genode environment is a
pivotal step - a step that is highly challenging because the build system is
based on the tight interplay of many GNU programs. Among those
programs are GNU make, coreutils, findutils, binutils, gcc, and bash. Even
though there is a large track record of individual programs and libraries ported
to the environment, those programs used to be self-sustaining applications that
require only little interaction with other programs. In contrast, the build
system relies on many utilities working together using mechanisms such as
files, pipes, output redirection, and execve. The Genode base system does not
come with any of those mechanisms let alone the subtle semantics of the POSIX
interface as expected by those utilities. Being true to microkernel principles,
Genode's API has a far lower abstraction level and is much more rigid in scope.
To fill the gap between the requirements of the build system and the bare
Genode mechanisms, the Noux runtime environment was created. Noux is a Genode
process that acts like a Unix kernel. When started, it creates a child process,
which plays a similar role as the init process of Unix. This process communicates
via RPC messages to Noux. Using those messages, the process can perform all the
operations normally provided by a classical Unix kernel. When executed under
Noux, a process can even invoke functionalities such as fork and execve, which
would normally contradict with Genode's principles of resource management.
Over the course of the past year, more and more programs have been ported to
the Noux environment. Thereby, the semantics provided by Noux have been
successively refined so that those program behave as expected. This was an
iterative process. For example, at the beginning, Noux did not consider the
differences between 'lstat' and 'stat' as they did not matter for the first
batch of GNU programs ported to Noux. As soon as the programs got more
sophisticated, such shortcuts had to be replaced by the correct semantics. The
Genode build system is by far the most complex scenario exposed to Noux so far.
It revealed many shortcomings by both functionality implemented in Noux or the
C runtime as well as the underlying base platforms. So it proved to be a great
testing ground for analysing and improving those platform details. Therefore,
the secondary effects of self-hosting Genode on Genode in terms of stability
turned out to be extremely valuable.
The release comes with two ready-to-use run scripts for building bootable
system images that are able to execute the Genode tool chain, one for targeting
NOVA and one for targeting Fiasco.OC. Those run scripts are located at
'ports/run/' and called 'noux_tool_chain_nova.run' and 'noux_tool_chain_foc.run'
respectively. Each of those run scripts can be executed on either of those base
platforms. For example, by executing 'noux_tool_chain_nova' on Fiasco.OC, the
image will run Genode on Fiasco.OC and the tool chain will build binaries for
NOVA. When started, a build directory will be created at '/home/build'.
The Genode source code is located at '/genode'. In the '/bin' directory,
there are all the GNU programs needed to execute the tool chain. For
taking a look into the source code, 'vim' is available. To build core,
change to the build directory '/home/build' and issue 'make core'.
On Fiasco.OC, the complete Genode demo scenario can be compiled. On NOVA, the
incomplete life-time management of kernel objects will still result in an
out-of-memory error of the kernel. This kernel issue is currently being worked
on. Executing the tool chain on either of those platforms is still relatively
slow as extensive trace output is being generated and no actions have been taken to
optimize the performance so far. There are many opportunities for such
optimizations, which will be taken on as the next step.
Base framework
##############
Genode's base framework has received new support for extending session
interfaces and gained improvements with regard to interrupt handling on the x86
platform. At the API level, there are minor changes related to the CPU session
and 'Range_allocator' interfaces.
Support for specializing session interfaces
===========================================
With increasingly sophisticated application scenarios comes the desire to
extend Genode's existing session interface with new functionality. For example,
the 'Terminal::Session' interface covers plain read and write operations. It is
implemented by services such as a graphical terminal, the telnet-like TCP
terminal, or UART drivers. However, for the latter category, the breadth of the
interface is severely limited as UART drivers tend to supplement the read / write
interface with additional control functions, e.g., for setting the baud rate.
One way to go would be to extend the existing 'Terminal::Session' interface
with those control functions. However, these functions would be meaningless for
most implementations. Some of those other implementations may even desire their
own share of additions. In the longer term, this approach might successively
broaden the interface and each implementation will cover a subset only.
Because Genode aspires to keep interfaces as low-complex as possible while, at
the same time, it wants to accommodate the growing sophistication of usage
scenarios, we need a solution that scales. The solution turns out to be
strikingly simple. The RPC framework already supports the inheritance of RPC
interfaces. So it is possible to model the problem such that a new
'Uart::Session' interface derived from the existing 'Terminal::Session' will
be the host of UART-specific functionality. The only piece missing is the
propagation of both 'Uart' and 'Terminal' through the parent interface while
announcing the service. To spare the work of manually announcing the chain of
inherited interfaces from the implementor, the 'Parent::announce()' function has
been enhanced to automatically announce all service types implemented by the
announced interface. This way, a UART driver will always announce a "Uart"
and a "Terminal" service.
Improved interrupt handling
===========================
To accommodate modern x86 platforms, the session arguments of core's IRQ
service have been supplemented with the IRQ mode. There are two degrees of
freedom, namely the trigger (level / edge) and polarity (high / low). Thanks to
this addition, device drivers have become able to supply their knowledge of
devices to core.
In system scenarios with many peripherals, in particular when using the USB
driver, IRQ lines are shared between devices. Until now, Genode supported
shared interrupts for the OKL4 base platform only. To also cover the other
x86 kernels, we have generalized the interrupt sharing code and enabled this
feature on Fiasco.OC and NOVA.
Revised CPU session interface
=============================
We revisited the CPU session interface, removed no-longer used functions and
added support for assigning threads to CPUs.
The original CPU session interface contained functions for iterating through
the threads of a session. This interface was originally motivated by an
experimental statistical profiling tool that was developed at an early stage of
Genode. In the meanwhile, we discovered that the virtualization of the CPU
session interface is much more elegant to cover this use case than the
thread-iterator interface. Because the iteration has no transactional
semantics, it was unsafe to use it anyway.
To enable the use of multiple CPUs on multi-processor systems, the CPU
session interface has been enhanced with two functions, namely 'affinity'
and 'num_cpus'. The interface extension principally allows the assignment of
individual threads to CPUs. It is currently implemented on Fiasco.OC only.
On all other base platforms, 'num_cpus' returns one CPU. Note that on
the Linux platform, multiple CPUs will be used transparently.
The 'Cpu_session::state' function has been split into two functions, one
for retrieving information and one for propagating state information. The
prior interface was less explicit about the semantics of the 'state' function
as it took a non-const pointer to a 'Thread_state' object as argument.
Platform-tailored protection domains
====================================
Genode tries to provide a uniform API across all the different base platforms.
Yet, it also strives to make genuine platform features available to the
users of the framework. Examples for such features are the virtualization
support of the NOVA hypervisor or the special support for paravirtualizing
Linux on Fiasco.OC. Another example is the security model as found on the Linux
platform. Even though the security mechanisms of plain Linux are not as strong
as Genode's capability concept on a conceptual level, we still want to leverage
the available facilities such as user IDs and chroot as far as possible.
Consequently, we need a way to assign platform-specific properties to PD
sessions. With the new 'Native_pd_args' type introduced into
'base/native_types.h', there is now a way to express those platform-specific
concerns. This type is now used at all the places that deal with the creation
of protection domains such as 'Process', 'Child', and the loader.
Revised 'Range_allocator' interface
===================================
The handling of allocation errors has been refined in order to distinguish
different error conditions, in particular out-of-metadata and out-of-memory
conditions. The user of the allocator might want to handle both cases
differently. Hence we return an 'Alloc_return' value as result. In prior
versions, this type was just an enum value. With the new version, the type has
been changed to a class. This makes the differentiation of error conditions at
the caller side more robust because, in contrast to enum values, typed objects
don't get implicitly converted to bool values.
Low-level OS infrastructure
###########################
New UART session interface
==========================
To accommodate UART specific extensions of the 'Terminal::Session' interface,
in particular setting the baud rate, we introduced the new 'Uart::Session'
interface and changed the existing UART drivers to implement this
interface instead of the 'Terminal::Session' interface. Because 'Uart::Session'
inherits the 'Terminal::Session' interface, 'Uart' services announce both
"Uart" and "Terminal" at their parent.
New GPIO session interface
==========================
Embedded SoCs such as OMAP4 provide many general-purpose I/O pins, which can be
used for different purposes depending on the board where they are soldered on.
For example, the Pandaboard uses such GPIO pins to detect the presence of a
HDMI plug or control the power supply for the USB. If only one driver deals
with GPIO pins, the GPIO programming can reside in the driver. However, if
multiple drivers are used, the GPIO device resources cannot be handed out to
more than one driver. This scenario calls for the creation of a GPIO driver as
a separate component, which intermediates (and potentially multiplexes) the
access to the physical GPIO pins. The new 'Gpio::Session' interface allows one
or multiple clients to configure I/O pins, request states, as well as to
register for events happening on the pins.
Terminal
========
The graphical terminal has been enhanced with support for different built-in
font sizes and background-color handling.
In addition to those functional changes, the implementation has been decomposed
into several parts that thereby became reusable. Those parts comprise the
handling of key mappings, decoding the VT character stream, and the handling of
the character array. These functionalities are now available at
'gems/include/terminal'.
Libraries and applications
##########################
C runtime
=========
:Allocator optimized for small-object allocations:
To optimize the performance of workloads that depend on a large number of small
dynamic memory allocations, in particular the lwIP TCP/IP stack, we replaced
the memory allocator of the libc with a more sophisticated strategy. Until now,
the libc used 'Genode::Heap' as allocator. This implementation is an
AVL-tree-based best-fit allocator that is optimized for low code complexity
rather than performance for small allocations. The observation of the allocator
usage pattern of lwIP prompted us to replace the original libc malloc/free with
a version that uses slab allocators for small objects and relies on the
'Genode::Heap' for large objects only.
:Symbolic links:
Because part of our ongoing refinements of the Noux runtime is the provision of
symbolic links, support for symbolic links was added in the libc, libc plugins,
and file system servers.
lwIP
====
We updated the light-weight IP stack to version STABLE-1.4.1. Additionally,
the following optimizations were conducted to improve its performance and
robustness.
We reduced the maximum segment lifetime from one minute to one second to avoid
queuing up PCBs in TIME-WAIT state. This is the state, PCBs end up after
closing a TCP connection socket at the server side. The number of PCBs in this
state is apparently not limited by the value of 'MEMP_NUM_TCP_PCB'. One
allocation costs around 160 bytes. If clients connect to the server at a high
rate, those allocations accumulate quickly and thereby may exhaust the memory
of the server. By reducing the segment lifetime, PCBs in TIME-WAIT state are
cleaned up from the 'tcp_tw_pcbs' queue in a more timely fashion (by
'tcp_slowtmr()').
To prevent the TCP/IP stack from artificially throttling TCP throughput,
we adjusted lwIP's TCP_SND_BUF size.
From our work on optimizing the NIC stub-code performance of L4Linux as
described [http://genode.org/documentation/articles/pandaboard - here],
we learned that the use of a NIC-specific packet allocator for the
packet-stream interface is beneficial. At the lwIP back end, we still relied on
the original general-purpose allocator. Hence, we improved the lwIP back-end
code by using the bitmap-based 'Nic::Packet_allocator' allocator instead.
Standard C++ library
====================
Genode used to rely on the standard C++ library that comes with the tool chain.
However, this mechanism was prone to inconsistencies of the types defined in
the header files used at compile time of the tool chain and the types provided
by our libc. By building the C++ standard library as part of the Genode build
process, such inconsistencies cannot happen anymore. The current version of the
C++ standard library corresponds to GCC 4.7.2.
Note that the patch changes the meaning of the 'stdcxx' library for users that
happened to rely on 'stdcxx' for hybrid Linux/Genode applications. For such
uses, the original mechanism is still available, in the renamed form of
'toolchain_stdcxx'.
Device drivers
##############
Open Sound System
=================
Genode tries to re-use existing device drivers as much as possible using an
approach called device-driver environment (DDE). A DDE is a library that
emulates the environment of the original driver by translating device accesses
to the Genode API. There are many success stories of drivers successfully ported
to the framework this way. For example, using DDE-Linux, we are able to use the
Linux USB stack. Using DDE-ipxe, we are able to use iPXE networking drivers.
With Genode 12.11 we extend our arsenal of DDEs with DDE-OSS, which is a
device-driver environment for the audio drivers of the Open Sound System (OSS).
:Website of the Open Sound System:
[http://http://www.4front-tech.com]
The new 'dde_oss' contains all the pieces needed to use Intel HDA, AC97, and
ES1370 audio cards on Genode. On first use, the 3rd-party code can be
downloaded by issuing 'make prepare' from within the 'dde_oss' source-code
repository. Also, you need to make sure to add the 'dde_oss' repository to your
'REPOSITORIES' variable in 'etc/build.conf'.
An OSS demo configuration can be found under 'run/oss.run' and can be started
via 'make run/oss' from a Genode build directory. Be sure to adjust the
'filename' tag of the 'audio0' program. The file has to reside under
'<build-dir>/bin/'. The file format is header-less two-channel float-32 at
44100 Hz. You may use the 'sox' utility to create these audio files:
! sox -c 2 -r 44100 foo.mp3 foo.f32
OMAP4 GPIO driver
=================
The new OMAP4 GPIO driver is the first implementation of the just introduced
'Gpio::Session' interface. The driver supports two ways of interacting
with GPIO pins, by providing a static configuration, or by interacting with a
session interface at runtime. An example for a static configuration looks as
follows:
! <config>
! <gpio num="121" mode="I"/>
! <gpio num="7" mode="O" value="0"/>
! <gpio num="8" mode="O" value="0"/>
! </config>
The driver is located at 'os/src/drivers/gpio/omap4'. As reference for using
the driver, please refer to the 'os/run/gpio_drv.run' script.
Thanks to Ivan Loskutov of Ksys-Labs for contributing the session interface
and the driver!
iPXE networking drivers
=======================
We updated our device-driver environment for iPXE networking drivers to a
recent git revision and enabled support for the x86_64 architecture.
Currently, the driver covers Intel gigabit ethernet (e1000, e1000e, igb),
Intel eepro100, and Realtek 8139/8169.
Runtime environments
####################
Noux
====
The Noux runtime environment has received plenty of love thanks to the
aspiration to execute the Genode build system.
:Time:
The build system uses GNU make, which depends on time stamps of files. We do
not necessarily need a real clock. A monotonic increasing virtual time is
enough. To provide such a virtual time, the libc was enhanced with basic
support for functions like 'gettimeofday', 'clock_gettime', and 'utimes'. As
there is currently no interface to obtain the real-world time in Genode, Noux
simulates a pseudo real-time clock using a jiffies-counting thread. This
limited degree of support for time is apparently sufficient to trick tools like
ping, find, and make into working as desired.
:Improved networking support:
The Noux/net version of Noux extends the Noux runtime with the BSD-socket
interface by using the lwIP stack. This version of Noux multiplexes the
BSD-socket interface of lwIP to multiple Noux programs, each having a different
socket-descriptor name space and the principal ability to use blocking calls
such as 'select'. The code for multiplexing the lwIP stack among multiple Noux
processes has been improved to cover corner cases exposed by sophisticated
network clients, i.e., openssh.
:Directory cache for the TAR file system:
The original version of the TAR file system required a search in all TAR
records for each file lookup. This takes a long time when composing a large
directory tree out of multiple TAR archives stacked together. This is the case
for the Genode build-system scenario where we have all the files of the GNU
tools as well as the Genode source tree. Searching through thousands of records
for each call of 'stat' quickly becomes a scalability issue. Therefore, we
introduced a TAR indexing mechanism that scans each TAR file only once at the
startup of Noux and generates a tree structure representing the directory
layout. Looking up files using this index is quick.
:New packages:
With Genode-12.11, new 3rd-party packages have become available, namely
OpenSSH, the 'which' command, and all tool-chain components in their current
version. OpenSSH is still at an experimental stage. The run script at
'ports/run/noux_net_openssh_interactive.run' demonstrates how SSH can be used
to login into a remote machine.
:New pseudo file systems:
The new 'stdio' and 'random' file systems are intended to represent the pseudo
devices '/dev/random' and '/dev/tty' on Noux. Both are needed to run OpenSSH.
Note that the 'Arc4random' class, on which the random file system is based on,
currently _does not collect enough_ random bytes! It should not be used for
security-critical applications.
L4Linux
=======
The paravirtualized L4Linux kernel for the Fiasco.OC platform was updated to
SVN revision 25, which matches the Fiasco.OC SVN revision 40. We further
improved the integration of L4Linux with Genode by optimizing the stub drivers
for block devices and networking, and added principal support for running
L4Linux on SMP platforms.
Platforms
#########
NOVA
====
Genode follows the steady development of the NOVA microhypervisor very closely.
The kernel used by the framework corresponds to the current state of the master
branch of IntelLabs/NOVA.
:Improvements towards GDB support:
The NOVA-specific implementation of the CPU session interface has been improved
to accommodate the requirements posed by GDB. In particular, the 'pause',
'resume', 'state', and 'single_step' functions have been implemented. Those
functions can be used to manipulate the execution and register state of
threads. Under the hood, NOVA's 'recall' feature is used to implement these
mechanisms. By issuing a 'recall' for a given thread, the targeted thread is
forced into an exception. In the exception, the current state of the thread can
be obtained and its execution can be halted/paused.
:Maximizing contiguous virtual space:
To enable the Vancouver virtual machine monitor to hand out large amounts of
guest memory, we optimized core's virtual address space to retain large and
naturally aligned contiguous memory regions. For non-core processes, the
thread-context area that contains the stacks of Genode threads has been moved
to the end of the available virtual address space.
:Life-time management of kernel resources:
We improved the life-time management of kernel resources, in particular
capabilities, within Genode. Still the management of such kernel resources
is not on par with the Fiasco.OC version, partially because of missing
kernel functionality. This is an ongoing topic that is being worked on.
:Using the BIOS data area (BDA) to get serial I/O ports on x86:
If the I/O ports for the comport are non default (default is 0x3f8), we had to
specify manually the correct I/O ports in the source code. To avoid the need
for source-code modifications when changing test machines, we changed the core
console to read the BDA and use the first serial interface that is available.
If no serial interface is available, no device configuration will be
undertaken. The BDA can be populated via a multi-boot chain loader. Bender is
such a chain loader that can detect serial ports accessible via PCI and writes
the I/O ports to the Bios Data area (BDA). These values get then picked up by
core.
Fiasco.OC
=========
The Fiasco.OC kernel has been updated to the SVN revision 40. The update improves
SMP support and comes with various bug fixes. There is no noteworthy change
with regard to the kernel interface. We extended the number of supported
Fiasco.OC-based platforms for Genode by including the Freescale i.MX53.
To enable the use of multiple CPUs by Genode processes, the CPU session
interface has been enhanced to support configuring the affinity of threads with
CPUs. We changed the default kernel configuration for x86 and ARM to
enable SMP support and adapted L4Linux to use the new interface.
Execution on bare hardware (base-hw)
====================================
The development of our custom platform for executing Genode directly on bare
hardware with no kernel underneath went full steam ahead during the release
cycle.
:Pandaboard:
The in-kernel drivers needed to accommodate the Pandaboard, more specifically
the timer and interrupt controller, are now supported. So the Pandaboard can be
used with both 'base-hw' and 'base-foc'. Also, the higher-level platform
drivers for USB, HDMI, and SD-card that were introduced with the previous
release, are equally functional on both platforms.
:Freescale i.MX31:
We added principal support for the Freescale i.MX line of SoCs taking the
ARMv6-based i.MX31 as starting point. As of now, the degree of support is
limited to the devices needed by the kernel to operate. Pure software-based
scenarios are able to work, i.e., the nested init run script executes
successfully.
:TrustZone support:
The new VM session interface of core provides a way to execute software
in the normal world of a TrustZone system whereas Genode runs in the secure
world. From Genode's point of view, the normal world looks like a virtual
machine. Each time, the normal world produces a fault or issues a secure
monitor call, control gets transferred to the virtual machine monitor, which is
a normal user-level Genode process. The base-hw kernel has been enhanced to
perform world switches between the secure and normal world and with the ability
to handle fast interrupts (FIQs) in addition to normal interrupts. The latter
extension is needed to assign a subset of devices to either of both worlds.
Currently, the only TrustZone capable platform is the ARM CoreTile Express
CA9x4 for the Versatile Express board. For a virtual machine working properly
on top, some platform resources must be reserved. Therefore, there exist two
flavours of this platform now, one with the 'trustzone' spec-variable enabled
and one without. If 'trustzone' is specified, most platform resources (DDR-RAM,
and most IRQs) are reserved for the normal world and not available to the
secure Genode world.
:Memory attributes and caching:
We successively activated various levels of caching and improved the handling
of caching attributes propagated into the page tables. These changes resulted
in a significant boost in performance on non-emulated platforms.
Linux
=====
The Linux version of Genode was originally meant as a vehicle for rapid
development. It allows the framework components including core to be executed
as plain Linux processes. But in contrast to normal Linux programs, which
use the glibc, Genode's components interact with the kernel directly without
any C runtime other than what comes with Genode. We use the Linux version on a
regular basis to implement platform-agnostic functionality and protocols. Most
of Genode's code (except for device drivers) falls in this category. Because
the Linux version was meant as a mere tool, however, we haven't put much
thought into the principle way to implementing Genode's security concept on
this platform. Threads used to communicate over globally accessible Unix-domain
sockets and memory objects were represented as globally accessible files within
'/tmp'.
That said, even though Linux was not meant as a primary platform for Genode in
the first place, Genode can bring additional value to Linux. When considering
the implementation of a component-based system on Linux, there are several
possible approaches to take. For example, components may use DBus to
communicate, or components could pick from the manifold Unix mechanisms such as
named pipes, files, sysv-shared memory, signals, and others. Unfortunately
those mechanisms are not orthogonal and most of them live in the global name
space of the virtual file system. Whereas those mechanisms are principally able
to let processes communicate, questions about how processes get to know each
other, access-control policy, synchronization of the startup of processes are
left to the developer.
Genode, on the other hand, does provide an API for letting components
communicate but also answers those tricky questions concerning the composition
of components. This makes Genode an interesting option to build component based
applications, even on Linux. However, when used in such a context, the
limitations of the original Linux support need resolutions. Therefore, the
current release comes with a largely revised platform support for the Linux
base platform.
The changes can be summarized as follows:
:Using file descriptors as communication addresses:
Genode's synchronous RPC framework was using Unix domain sockets. Each RPC
entrypoint was represented by a pair of named files, one for sending and one
for receiving messages. In the new version, inter-process communication is
performed via file descriptors only.
:Transfer of communication rights via RPC only:
Capabilities used to be represented as a pair of the destination thread ID and
a global object ID. The thread ID has been replaced by a file descriptor that
points to the corresponding RPC entrypoint. When capabilities are transferred
as RPC arguments, those file descriptors are transferred via SCM rights
messages. This is in line with Genode's way of capability-based delegation of
access rights.
:Core-only creation of communication channels:
Communication channels used to be created locally by each process. The naming
of those channels was a mere convention. In contrast, now, communication
channels are created by core only and do not reside on the Linux virtual file
system. When creating an RPC entrypoint, core creates a socket pair and hands
out both ends to the creator of the entrypoint.
:Restricted access to memory objects:
Access to dataspace content was performed by mmap'ing a file. For a given
dataspace, the file name could be requested at core via a Linux-specific RPC
call. Now, core holds the file descriptors of all dataspaces, which are
actually unlinked files. A process that is in possession of a dataspace
capability can request the file descriptor for the content from core and mmap
the file locally. This way, access to memory objects is subjected to the
delegation of dataspace capabilities.
:Core-local process creation:
Genode used to create new processes by directly forking from the respective
Genode parent using the process library. The forking process created a PD
session at core merely for propagating the PID of the new process into core
(for later destruction). This traditional mechanism has the following
disadvantages:
First, the PID reported by the creating process to core cannot easily be
validated by core. Therefore core has to trust the PD client to not specify a
PID of an existing process, which would happen to be killed once the PD session
gets destructed. Second, there is no way for a Genode process to detect the
failure of any of its grandchildren. The immediate parent of a faulting process
could use the SIGCHLD-and-waitpid mechanism to observe its children but this
mechanism does not work transitively.
By performing the process creation exclusively within core, all Genode
processes become immediate child processes of core. Hence, core can respond to
failures of any of those processes and reflect such conditions via core's
session interfaces. Furthermore, the PID associated to a PD session is locally
known within core and cannot be forged anymore. In fact, there is actually no
need at all to make processes aware of any PIDs of other processes.
:Handling of chroot, user IDs, and group IDs:
With the move of the process creation into core, the original chroot trampoline
mechanism implemented in 'os/src/app/chroot' does not work anymore. A process
could simply escape the chroot environment by spawning a new process via core's
PD service. Therefore, chroot support has been integrated into core and the
chroot policy becomes a mandatory part of the process creation. For each process
created by core, core checks for a 'root' argument of the PD session. If a path
is present, core takes the precautions needed to execute the new process in the
specified chroot environment.
This conceptual change implies minor changes with respect to the Genode API and
the configuration of the init process. The API changes are the enhancement of
the 'Genode::Child' and 'Genode::Process' constructors to take the root path as
argument. Init supports the specification of a chroot per process by specifying
the new 'root' attribute to the '<start>' node of the process. In line with
these changes, the 'Loader::Session::start' function has been enhanced with the
additional (optional) PD argument.
In line with how the chroot path can be propagated into core, core has become
able to assign customized UIDs and GIDs to individual Genode processes or whole
Genode subsystems. The new 'base-linux/run/lx_uid.run' script contains an
example of how to use the feature.
Build system and tools
######################
The current release comes with a new tool chain based on GCC 4.7.2 and binutils
2.22. The tool-chain upgrade involved adapting the Genode code base and fixing
various issues in 3rd-party software. To obtain the new tool chain, please
refer to the tool-chain website:
:Genode tool chain:
[http://genode.org/download/tool-chain]

941
doc/release_notes-13-02.txt
View File

@@ -1,941 +0,0 @@
===============================================
Release notes for the Genode OS Framework 13.02
===============================================
Genode Labs
Traditionally, the February release of Genode is focused on platform support.
The version 13.02 follows this tradition by vastly improving Genode for the
NOVA base platform and the extending the range of ARM SoCs supported by
both our custom kernel platform and the Fiasco.OC kernel.
The NOVA-specific improvements concern three major topics, namely the added
support for running dynamic workloads on this kernel, the use of IOMMUs, and
the profound integration of the Vancouver virtual machine monitor with the
Genode environment. The latter point is particularly exciting to us because
this substantial work is the first contribution by Intel Labs to the Genode
code base. Thanks to Udo Steinberg and Markus Partheymüller for making that
possible.
Beyond the x86 architecture, the new version comes with principal support for
the ARM Cortex-A15-based Exynos 5250 SoC and the Freescale i.MX53 SoC. The
Exynos 5250 SoC has been enabled for our custom kernel as well as for the
Fiasco.OC kernel. The most significant functional improvements are a new
facility to detect faulting processes and a new mechanism for file-system
notifications.
Besides those added functionalities, the release cycle was taken as an
opportunity to revisit several aspects under the hood of the framework. A few
examples are the reworked synchronization primitives, the simplified base
library structure, the completely redesigned audio-output interface, and a
modernized timer interface.
DMA protection via IOMMU
########################
Direct memory access (DMA) of devices is universally considered as the Achilles
heel of microkernel-based operating systems. The most compelling argument in
favour of using microkernels is that by encapsulating each system component
within a dedicated user-level address space, the system as a whole becomes more
robust and secure compared to a monolithic operating-system kernel. In the
event that one component fails due to a bug or an attack, other components
remain unaffected. The prime example for such buggy components are device
drivers. By empirical evidence, those remain the most prominent trouble makers
in today's operating systems. Unfortunately however, most commodity hardware
used to render this nice argumentation moot because it left one giant loophole
open, namely bus-master DMA.
Via bus-master DMA, a device attached to the system bus is able to directly
access the RAM without involving the CPU. This mechanism is crucial for all
devices that process large amounts of data such as network adapters, disk
controllers, or USB controllers. Because those devices can issue bus requests
targeting the RAM directly and not involving the CPU altogether, such requests
are naturally not subjected by the virtual-memory mechanism implemented in the
CPU in the form of an MMU. From the device's point of view there is just
physical memory. Hence, if a driver sets up a DMA transaction, let's say a disk
driver wants to read a block from the disk, the driver tells the device about
the address and size of a physical-memory buffer where the it wants to receive
the data. If the driver lives in a user-level process, as is the case for a
microkernel-based system, it still needs to know the physical address to
program the device correctly. Unfortunately, there is nothing to prevent the
driver from specifying any physical address to the device. Consequently, a
malicious driver could misuse the device to read and manipulate all parts of
the memory, including the kernel.
[image no_iommu]
Traditional machine without IOMMU. Direct memory accesses issued by the
disk controller are not subjected to the MMU. The disk controller can
access the entity of memory present in the system.
So - does this loop hole render the micro-kernel approach useless? Of course not.
Putting each driver in a dedicated address space is still beneficial in two
ways. First, classes of bugs that are unrelated to DMA remain confined in the
driver's address space. In practice most driver issues arise from issues like
memory leaks, synchronization problems, deadlocks, flawed driver logic, wrong
state machines, or incorrect device-initialization sequences. For those classes
of problems, the microkernel argument still applies. Second, executing a driver
largely isolated from other operating-system code minimizes the attack surface
of the driver. If the driver interface is rigidly small and well-defined, it is
hard to compromise the driver by exploiting its interface.
Still the DMA issue remains to be addressed. Fortunately, modern PC hardware
has closed the bus-master-DMA loophole by incorporating a so-called IOMMU into
the system. As depicted in the following figure, the IOMMU sits between the RAM
and the system bus where the devices are attached to. So each DMA request has
to pass the IOMMU, which is not only able to arbitrate the access of DMA
requests to the RAM but also able to virtualize the address space per device.
Similar to how a MMU confines each process running on the CPU within a distinct
virtual address space, the IOMMU is able to confine each device within a
dedicated virtual address space. To tell the different devices apart, the IOMMU
uses the PCI device's bus-device-function triplet as unique identification.
[image iommu]
An IOMMU arbitrates and virtualizes DMA accesses issued by a device to the
RAM. Only if a valid IOMMU mapping exists for a given DMA access, the memory
access is performed.
Of the microkernels supported by Genode, NOVA is the first kernel that supports
the IOMMU. NOVAs interface to the IOMMU is quite elegant. The kernel simply
applies a subset of the (MMU) address space of a process (aka protection domain
in NOVA speak) to the (IOMMU) address space of a device. So the device's
address space can be managed in the same way as we normally manage the address
space of a process. The only missing link is the assignment of device address
spaces to process address spaces. This link is provided by the dedicated system
call "assign_pci" that takes a process identifier and a device identifier as
arguments. Of course, both arguments must be subjected to a security policy.
Otherwise, any process could assign any device to any other process. To enforce
security, the process identifier is a capability to the respective protection
domain and the device identifier is a virtual address where the extended PCI
configuration space of the device is mapped in the specified protection domain.
Only if a user-level device driver got access to the extended PCI configuration
space of the device, it is able to get the assignment in place.
To make NOVA's IOMMU support available to Genode programs, we enhanced the
ACPI/PCI driver with the ability to hand out the extended PCI configuration
space of a device and added a NOVA-specific extension to the PD session
interface. The new 'assign_pci' function allows the assignment of a PCI device
to the protection domain.
[image iommu_aware]
NOVAs management of the IOMMU address spaces facilities the use of
driver-local virtual addresses as DMA addresses.
Even though these mechanisms combined principally
suffice to let drivers operate with the IOMMU enabled, in practice, the
situation is a bit more complicated. Because NOVA uses the same
virtual-to-physical mappings for the device as it uses for the process, the DMA
addresses the driver needs to supply to the device must be virtual addresses
rather than physical addresses. Consequently, to be able to make a device
driver usable on systems without IOMMU as well as on systems with IOMMU, the
driver needs to be IOMMU-aware and distinguish both cases. This is an
unfortunate consequence of the otherwise elegant mechanism provided by NOVA. To
relieve the device drivers from caring about both cases, we came up with a
solution that preserves the original device interface, which expects physical
addresses. The solution comes in the form of so called device PDs. A device PD
represents the address space of a device as a Genode process. Its sole purpose
is to hold mappings of DMA buffers that are accessible by the associated
device. By using one-to-one physical-to-virtual mappings for those buffers
within the device PD, each device PD contains a subset of the physical address
space. The ACPI/PCI server performs the assignment of device PDs to PCI
devices. If a device driver intends to use DMA, it asks the ACPI/PCI driver for
a new DMA buffer. The ACPI/PCI driver allocates a RAM dataspace at core,
attaches it to the device PD using the dataspace's physical address as virtual
address, and hands out the dataspace capability to the driver. If the driver
requests the physical address of the dataspace, the returned address will be a
valid virtual address in the associated device PD. From this design follows
that a device driver must allocate DMA buffers at the ACPI/PCI server (while
specifying the PCI device the buffer is intended for) instead of using core's
RAM service to allocate buffers anonymously. The current implementation of the
ACPI/PCI server assigns all PCI devices to only one device PD. However, the
design devises a natural way to partition devices into different device PDs.
[image iommu_agnostic]
By modelling a device address space as a dedicated process (device PD),
the traditional way of programming DMA transactions can be maintained,
even with the IOMMU enabled.
Because the changed way of how DMA buffers are allocated, our existing drivers
such as the AHCI disk driver, the OSS sound driver, the iPXE network driver,
and the USB driver had to be slightly modified. We also extended DDE Kit with
the new 'dde_kit_pci_alloc_dma_buffer' function for allocating DMA buffers.
With those changes, the complete Genode user land can be used on systems with
IOMMU enabled. Hence, we switched on the IOMMU on NOVA by default.
Full virtualization on NOVA/x86
###############################
Vancouver is a x86 virtual machine monitor that is designed to run as
user-level process on top of the NOVA hypervisor. In
[http://genode.org/documentation/release-notes/11.11#Faithful_x86_PC_Virtualization_enabled_by_the_Vancouver_VMM - Genode version 11.11],
we introduced the preliminary adaptation of Vancouver to Genode. This version
was meant as a mere proof of concept, which allowed the bootup of small Guest
OSes (such as Fiasco.OC or Pistachio) inside the VMM. However, it did not
support any glue code to Genode's session interface, which limited the
usefulness of this virtualization solution at that point. We had planned to
continue the integration of Vancouver with Genode once we observed public
demand.
The move of NOVA's development to Intel Labs apparently created this demand.
It is undeniable that combining the rich user land provided by Genode with the
capabilities of the Vancouver VMM poses an attractive work load for NOVA. So
the stalled line of the integration work of Vancouver with Genode was picked up
within Intel Labs, more specifically by Markus Partheymüller. We are delighted
to be able to merge the outcome of this undertaking into the mainline Genode
development. Thanks to Intel Labs and Markus in particular for this substantial
contribution!
The features added to the new version of Vancouver for Genode are as follows:
:VMX support:
Our initial version supported AMD's SVM technology only because this was
readily supported by Qemu. With the added support for Intel VMX, Vancouver
has become able to operate on both Intel and AMD processors with hardware
virtualization support.
:Timer support:
With added support for timer interrupts, the VMM has become able to
boot a complete Linux system.
:Console support:
With this addition, the guest VM can be provided with a frame buffer and
keyboard input.
For the frame-buffer size in Vancouver, the configuration value in the
machine XML node is used. It is possible to map the corresponding memory
area directly to the guest regardless if it comes from nitpicker, a virtual
frame buffer, or the VESA driver. The guest is provided with two modes (text
mode 3 and graphics mode 0x114 (0x314 in Linux).
Pressing LWIN+END while a VM has focus resets the virtual machine. Also,
RESET and DEBUG key presses will not be forwarded to the VM anymore.
It is possible to dump a VM's state by pressing LWIN+INS keys.
The text console is able to detect idle mode, unmaps the buffer from the
guest and stops interpreting. Upon the next page fault in this area, it
resumes operation again. The code uses a simple checksum mechanism instead
of a large buffer and 'memcmp' to detect an idle text console. False
positives don't matter very much.
:Network support:
The VMM has become able to use the Intel 82576 device model from the NUL
user land to give VMs access to the network via Genode's NIC bridge service
or a NIC driver.
:Disk support:
The VMM can now assign block devices to guests using Genode's block-session
interface. The machine has to be configured to use a specified drive, which
could be theoretically routed to different partitions or services via policy
definitions. Currently the USB driver only supports one device. Genode's AHCI
driver is untested.
:Real-time clock:
By using the new RTC session interface, Vancouver is able to provide the
wall-clock time to guest OSes.
To explore the new version of the Vancouver VMM, there exists a ready-to-use
run script at 'ports/run/vancouver.run'. Only the guest OS binaries such as a
Linux kernel image and a RAM disk must be manually supplied in the
'<build-dir>/bin' directory. The run script is able to start one or multiple
instances of the VMM using the graphical launchpad.
Low-latency audio output
########################
In version 10.05, we introduced an interface for the playback of audio data
along with an audio mixer component and ALSA-based sound drivers ported from
the Linux kernel. The original 'Audio_out' session interface was based on
Genode's packet stream facility, which allows the communication of bulk data
across address spaces via a combination of shared memory and signals. Whereas
shared memory is used to transfer the payload in an efficient manner without
the need to copy data via the kernel, signals are used to manage the data flow
between the information source and sink.
[image packet_stream]
Figure [packet_stream] displays the life cycle of a packet of information
transferred from the source to the sink. The original intent behind the
packet-stream facility was the transmission of networking packets and blocks
of block devices. At the time when we first introduced the 'Audio_out'
interface, the packet stream seemed like a good fit for audio, too. However, in
the meanwhile, we came to the conclusion that this is not the case when trying
to accommodate streamed audio data and sporadic audio output at the same time.
For the output of streamed audio data, a codec typically decodes a relatively
large portion of an audio stream and submits the sample data to the mixer. The
mixer, in turn, mixes the samples of multiple sources and forwards the result
to the audio driver. Each of those components the codec, the mixer, and the
audio driver live in a separate process. By using large buffer sizes between
them, the context-switching overhead is hardly a concern. Also, the driver can
submit large buffers of sample data to the sound device without any further
intervention needed.
In contrast, sporadic sounds are used to inform the user about an immediate
event. It is ultimately expected that such sounds are played back without much
latency. Otherwise the interactive experience (e.g., of games) would suffer.
Hence, using large buffers between the audio source, the mixer, and the driver
is not an option. By using the packet stream concept, we have to settle on a
specific buffer size. A too small buffer increases CPU load caused by many
context switches and the driver, which has to feed small chunks of sample data
to the sound device. A too large buffer, however, makes sporadic sounds at low
latencies impossible. We figured out that the necessity to find a sweet spot
for picking a buffer size is a severe drawback. This observation triggered us
to replace the packet-stream-based communication mechanism of the 'Audio_out'
session interface by a new solution that we specifically designed to
accommodate both corner cases of audio output.
[image audio_out]
Similarly to the packet-stream mechanism, the new interface is based on a
combination of shared memory and signals. However, we dropped the notion of
ownership of packets. When using the packet-stream protocol depicted as above,
either the source or the sink is in charge of handling a given packet at a
given time, not both. At the points 1, 2, and 4, the packet is owned by the
source. At the points 3 and 4, the packet is owned by the sink. By putting a
packet descriptor in the submit queue or acknowledgement queue, there is a
handover of responsibility. The new interface weakens this notion of ownership
by letting the source update once submitted audio frames even after submitting
them. If there are solely continuous streams of audio arriving at the mixer,
the mixer can mix those large batches of audio samples at once and pass the
result to the driver.
[image mixer_streaming]
The mixer processes incoming data from multiple streaming sources as batches.
Now, if a sporadic sound comes in, the mixer checks the
current output position reported by the audio driver, and re-mixes those
portions that haven't been played back yet by incorporating the sporadic sound.
So the buffer consumed by the driver gets updated with new data.
[image mixer_sporadic]
A sporadic occuring sound prompts the mixer to remix packets that are
already submitted in the output queue.
Besides changing the way of how packets are populated with data, the second
major change is turning the interface into a time-triggered concept. The
driver produces periodic signals that indicate the completeness of a
played-back audio packet. This signal triggers the mixer to become active,
which in turn serves as a time base for its clients. The current playback
position is denoted alongside the sample data as a field in the memory buffer
shared between source and sink.
The new 'Audio_out' interface has the potential to align the requirements of
both streamed audio with those of sporadic sounds. As a side benefit, the now
domain-specific interface has become simpler than the original packet-stream
based solution. This becomes nowhere as evident as in the implementation of the
mixer, which has become much simpler (30% less code). The interface change
is accompanied with updates of components related to audio output, in
particular the OSS sound drivers contained in 'dde_oss', the ALSA audio driver
for Linux, the avplay media player, and the libSDL audio back-end.
Base framework
##############
Signalling API improvements
===========================
The signalling API provided by 'base/signal.h' is fairly low level. For
employing the provided mechanism by application software, we used to craft
additional glue code that translates incoming signals to C++ method
invocations. Because the pattern turned out to be not only useful but a good
practice, we added the so called 'Signal_dispatcher' class template to the
signalling API.
In addition to being a 'Signal_context', a 'Signal_dispatcher' associates a
member function with the signal context. It is intended to be used as a member
variable of the class that handles incoming signals of a certain type. The
constructor takes a pointer-to-member to the signal handling function as
argument. If a signal is received at the common signal reception code, this
function will be invoked by calling 'Signal_dispatcher_base::dispatch'. This
pattern can be observed in the implementation of RAM file system
('os/src/server/ram_fs').
Under the hood, the signalling implementation received a major improvement with
regard to the life-time management of signal contexts. Based on the observation
that 'Signal' objects are often referring to non-trivial objects derived from
'Signal_context', it is important to defer the destruction of such objects to a
point when no signal referring to the context is in flight anymore. We solved
this problem by modelling 'Signal' type as a shared pointer that operates on a
reference counter embedded in the corresponding 'Signal_context'. Based on
reference counter, the 'Signal_receiver::dissolve()' function does not return
as long as the signal context to be dissolved is still referenced by one or
more 'Signal' objects.
Trimmed and unified framework API
=================================
A though-provoking
[http://sourceforge.net/mailarchive/forum.php?thread_name=CAGQ-%3Dq27%2B_UooBiJmz9RdTE1gDmVcg9v0w-8TNgEH5fzHYiA%2BQ%40mail.gmail.com&forum_name=genode-main - posting]
on our mailing list prompted us to explore the idea to make shared libraries
and dynamically linked executables binary compatible among different kernels.
This sounds a bit crazy at first but it is not downright infeasible.
As a baby step into this direction, we unified several public headers of the
Genode API and tried to make headers private to the framework where possible.
The latter is the case for the 'base/platform_env.h' header, which is actually
not part of the generic Genode API. Hence, it was moved to the
framework-internal 'src/base/env'. Another step was the removal of
platform-specific types that are not necessarily platform-dependent. We could
remove the 'Native_lock' type without any problems. Also, we were able to unify
the IPC API, which was formerly split into the two parts 'base/ipc_generic.h'
and 'base/ipc.h' respectively. Whereas 'base/ipc_generic.h' was shared among
all platforms, the 'base/ipc.h' header used to contain platform-specific IPC
marshalling and unmarshalling code. But by moving this code from the header to
the corresponding (platform-specific) IPC library, we were able to discard the
content of 'base/ipc.h' altogether. Consequently, the former
'base/ipc_generic.h' could be renamed to 'base/ipc.h'. These changes imply no
changes at the API level.
Simplified structure of base libraries
======================================
The Genode base API used to come in the form of many small libraries, each
covering a dedicated topic. Those libraries were 'allocator_avl', 'avl_tree',
'console', 'env', 'cxx', 'elf', 'env', 'heap', 'server', 'signal', 'slab',
'thread', 'ipc', and 'lock'. The term "library" for those bits of code was
hardly justified as most of the libraries consisted of only a few .cc files.
Still the build system had to check for their inter-dependencies on each run of
the build process. Furthermore, for Genode developers, specifying the list of
base libraries in their 'target.mk' files tended to be an inconvenience. Also,
the number of libraries and their roles (core only, non-core only, shared by
both core and non-core) were not easy to capture. Hence, we simplified the way
of how those base libraries are organized. They have been reduced to the
following few libraries:
* 'cxx.mk' contains the C++ support library
* 'startup.mk' contains the startup code for normal Genode processes
On some platform, core is able to use the library as well.
* 'base-common.mk' contains the parts of the base library that are
identical by core and non-core processes.
* 'base.mk' contains the complete base API implementation for non-core
processes
Consequently, the 'LIBS' declaration in 'target.mk' files becomes simpler as
well. In the normal case, only the 'base' library must be mentioned.
New fault-detection mechanism
=============================
Until now, it was hardly possible for a parent process to respond to crashes of
child processes in a meaningful way. If a child process crashed, the parent
would normally just not take notice. Even though some special use cases such as
GDB monitor could be accommodated by the existing
'Cpu_session::exception_handler' facility, this mechanism requires the
virtualization of the 'Cpu_session interface' because an exception handler used
to refer to an individual thread rather than the whole process. For ordinary
parents, this mechanism is too cumbersome to use. However, there are several
situations where a parent process would like to actively respond to crashing
children. For example, the parent might like to restart a crashed component
automatically, or enter a special failsafe mode.
To ease the implementation of such scenarios, we enhanced the existing
'Cpu_session::exception_handler' mechanism with the provision of a
default signal handler that is used if no thread-specific handler is installed.
The default signal handler can be set by specifying an invalid thread
capability and a valid signal-context capability. So for registering a signal
handler to all threads of a process, no virtualization of the 'Cpu_session'
interface is needed anymore. The new mechanism is best illustrated by the
'os/run/failsafe.run' script, which creates a system that repeatedly spawns a
crashing child process.
Reworked synchronization primitives
===================================
We reworked the framework-internal lock interface in order to be able to use
the 'futex' syscall on the Linux base platform. Previously, the lock
implementation on Linux was based on Unix signals. In the contention case, the
applicant for a contended lock would issue a blocking system call, which gets
canceled by the occurrence of a signal. We used 'nanosleep' for this purpose.
Once the current owner of the lock releases the lock, it sends a signal to the
next applicant of the lock. Because signals are buffered by the kernel, they
are guaranteed to be received by the targeted thread. However, in situations
with much lock contention, we observed the case where the signal was delivered
just before the to-be-blocked thread could enter the 'nanosleep' syscall. In
this case, the signal was not delivered at the next entrance into the kernel
(when entering 'nanosleep') but earlier. Even though the signal handler was
invoked, we found no elegant way to handle the signal such that the subsequent
'nanosleep' call would get skipped. So we decided to leave our signal-based
solution behind and went for the mainstream 'futex' mechanism instead.
Using this mechanism required us to re-design the internal lock API, which was
originally designed with the notion of thread IDs. The 'Native_thread_id' type,
which was previously used in the lock-internal 'Applicant' class to identify a
thread to be woken up, was not suitable anymore for implementing this change.
Hence, we replaced it with the 'Thread_base*' type, which also has the positive
effect of making the public 'base/cancelable_lock.h' header file
platform-independent.
In addition to reworking the basic locking primitives, we changed the
'Object_pool' data structure to become safer to use. The former 'obj_by_*'
functions have been replaced by 'lookup_and_lock' that looks up an object and
locks it in one atomic operation. Additionally, the case that an object may
already be in destruction is handled gracefully. In this case, the lookup will
return that the object is not available anymore.
Low-level OS infrastructure
###########################
Notification mechanism for the file-system interface
====================================================
To support dynamic system scenarios, we extended Genode's file-system interface
with the ability to monitor changes of files or directories, similar to the
inotify mechanism on Linux but simpler. The new 'File_system::sigh' function
can be used to install a signal handler for an open file node. When a node is
closed after a write operation, a prior registered signal handler for this file
gets notified. Signal handlers can also be installed for directories. In this
case, the signal handler gets informed about changes of immediate nodes hosted
in the directory, i.e., the addition, renaming, or removal of nodes.
The 'ram_fs' server has been enhanced to support the new interface. So any file
or directory change can now be observed by 'ram_fs' clients.
New adapter from file-system to ROM session interface
=====================================================
The new 'fs_rom' server translates the 'File_system' session interface to the
'ROM' session' interface. Each request for a ROM file is handled by looking
up an equally named file on the file system. If no such file can be found,
then the server will monitor the file system for the creation of the
corresponding file. Furthermore, the server reflects file changes as signals
to the ROM session.
There currently exist two limitations: First, symbolic links are not handled.
Second, the server needs to allocate RAM for each requested file. The RAM is
always allocated from the RAM session of the server. Thereby, the RAM quota
consumed by the server depends on the client requests and the size of the
requested files. Therefore, one instance of the server should not be used by
untrusted clients and trusted clients at the same time. In such situations,
multiple instances of the server could be used.
The most interesting feature of the 'fs_rom' server is the propagation of
file-system changes as ROM module changes. This clears the way to use this
service to supply dynamic configurations to Genode programs.
Dynamic re-configuration of the init process
============================================
The init process has become able to respond to configuration changes by
restarting the scenario using the new configuration. To make this feature
useful in practice, init must not fail under any circumstances. Even on
conditions that were considered previously as fatal and led to the abort of
init (such as ambiguous names of the children or misconfiguration in general),
init must stay alive and responsive to configuration changes.
With this change, the init process is one of the first use cases of the dynamic
configuration feature enabled via the 'fs_rom' service and the new file-system
notifications. By supplying the configuration of an init instance via the
'fs_rom' and 'ram_fs' services, the configuration of this instance gets fetched
from a file of the 'ram_fs' service. Each time, this file is changed, for
example via VIM running within a Noux runtime environment, the init process
re-evaluates its configuration.
In addition to the support for dynamic re-configurations, we simplified the use
of conditional session routing, namely the '<if-args>' mechanism. When matching
the 'label' session argument using '<if-args>' in a routing table, we can omit
the child name prefix because it is always the same for all sessions
originating from the child anyway. By handling the matching of session labels
as a special case, the expression of label-specific routing
becomes more intuitive.
Timer interface turned into asynchronous mode of operation
==========================================================
The 'msleep' function of 'Timer::Session' interface is one of the last relics
of blocking RPC interfaces present in Genode. As we try to part away from
blocking RPC calls inside servers and as a means to unify the timer
implementation across the many different platforms supported by Genode, we
changed the interface to an asynchronous mode of operation.
Synchronous blocking RPC interfaces turned out to be constant sources of
trouble and code complexity. E.g., a timer client that also wants to respond to
non-timer events was forced to be a multi-threaded process. Now, the blocking
'msleep' call has been replaced by a mechanism for programming timeouts and
receiving wakeup signals in an asynchronous fashion. Thereby signals
originating from the timer can be handled, along with signals from other signal
sources, by a single thread. Once a timer client has registered a signal
handler using the 'Timer::sigh' function, it can program timeouts using the
functions 'trigger_once' and 'trigger_periodic', which take an amount of
microseconds as argument. For maintaining compatibility and convenience, the
interface still contains the virtual 'msleep' function. However, it is not an
RPC function anymore but a mere client-side wrapper around the 'sigh' and
'trigger_once' functions. For use cases where sleeping at the granularity of
milliseconds is too coarse (such as udelay calls by device drivers), we added
a new 'usleep' call, which takes a number of microseconds as argument.
As a nice side effect of the interface changes, the platform-specific
implementations could be vastly unified. On NOVA and Fiasco.OC, the need to use
one thread per client has vanished. As a further simplification, we changed the
timer to use the build system's library-selection mechanism instead of
providing many timer targets with different 'REQUIRES' declarations. This
reduces the noise of the build system. For all platforms, the target at
'os/src/drivers/timer' is built. The target, in turn, depends on a 'timer'
library, which is platform-specific. The various library description files are
located under 'os/lib/mk/<platform>'. The common bits are contained in
'os/lib/mk/timer.inc'.
Since the 'msleep' call is still available from the client's perspective,
the change of the timer interface does not imply an API incompatibility.
However, it provides the opportunity to simplify clients in cases that required
the maintenance of a separate thread for the sole purpose of
periodic signal generation.
Loader
======
The loader is a service that enables its clients to dynamically create Genode
subsystems. Leveraging the new fault-detection support described in section
[New fault-detection mechanism], we enabled loader clients to respond to
failures that occur inside the spawned subsystem. This is useful for scenarios
where subsystems should be automatically restarted or in situations where the
system should enter a designated failsafe mode once an unexpected fault
happens.
The loader provides this feature by installing an optional client-provided
fault handler as default CPU exception handler and a RM fault handler for all
CPU and RM sessions of the loaded subsystem. This way, the failure of any
process within the subsystem gets reflected to the loader client as a signal.
The new 'os/run/failsafe.run' test illustrate this mechanism. It covers two
cases related to the loader, which are faults produced by the immediate child
of the loader and faults produced by indirect children.
Focus events for the nitpicker GUI server
=========================================
To enable a way for applications to provide visual feedback to changed keyboard
focus, we added a new 'FOCUS' event type to the 'Input::Event' structure. To
encode whether the focus was entered or left, the former 'keycode' member is
used (value 0 for leaving, value 1 for entering). Because 'keycode' is
misleading in this context, the former 'Input::Event::keycode' function was
renamed to 'Input::Event::code'. The nitpicker GUI server has been adapted to
deliver focus events to its clients.
NIC bridge with support for static IP configuration
===================================================
NIC bridge is a service that presents one physical network adaptor as many
virtual network adaptors to its clients. Up to now, it required each client
to obtain an IP address from a DHCP server at the physical network. However,
there are situations where the use of static IPs for virtual NICs is useful.
For example, when using the NIC bridge to create a virtual network between
the lighttpd web server and the Arora web browser, both running as Genode
processes without real network connectivity.
The static IP can be configured per client of the NIC bridge using a '<policy>'
node of the configuration. For example, the following policy assigns a static
address to a client with the session label "lighttpd".
!<start name="nic_bridge">
! ...
! <config>
! <policy label="lighttpd" ip_addr="10.0.2.55"/>
! </config>
!</start>
Of course, the client needs to configure its TCP/IP stack to use the assigned
IP address. This can be done via configuration arguments examined by the
'lwip_nic_dhcp' libc plugin. For the given example, the configuration for the
lighttpd process would look as follows.
!<start name="lighttpd">
! <config>
! <interface ip_addr="10.0.2.55"
! netmask="255.255.255.0"
! gateway="10.0.2.1"/>
! </config>
!</start>
Libraries and applications
##########################
New terminal multiplexer
========================
The new 'terminal_mux' server located at 'gems/src/server/terminal_mux' is able
to provide multiple terminal sessions over one terminal-client session. The
user can switch between the different sessions using a keyboard shortcut, which
brings up an ncurses-based menu.
The terminal sessions provided by terminal_mux implement (a subset of) the
Linux terminal capabilities. By implementing those capabilities, the server
is interchangeable with the graphical terminal ('gems/src/server/terminal').
The terminal session used by the server is expected to by VT102 compliant.
This way, terminal_mux can be connected via an UART driver with terminal
programs such as minicom, which typically implement VT102 rather than the Linux
terminal capabilities.
When started, terminal_mux displays a menu with a list of currently present
terminal sessions. The first line presents status information, in particular
the label of the currently visible session. A terminal session can be selected
by using the cursor keys and pressing return. Once selected, the user is able
to interact with the corresponding terminal session. Returning to the menu is
possible at any time by pressing control-x.
For trying out the new terminal_mux component, the 'gems/run/termina_mux.run'
script sets up a system with three terminal sessions, two instances of Noux
executing VIM and a terminal_log service that shows the log output of both Noux
instances.
New ported 3rd-party libraries
==============================
To support our forthcoming port of Git to the Noux runtime environment, we
have made the following libraries available via the libports repository:
* libssh-0.5.4
* curl-7.29.0 (for now the port is x86_* only because it depends on libcrypto,
which is currently not tested on ARM)
* iconv-1.14
Device drivers
##############
Besides the changes concerning the use of IOMMUs, the following device driver
have received improvements:
:UART drivers:
The OMAP4 platform support has been extended by a new UART driver, which
enables the use of up to 4 UART interfaces. The new driver is located at
'os/src/drivers/uart/omap4'.
All UART drivers implement the 'Terminal::Session' interface, which
provides read/write functionality accompanied by a function to determine
the terminal size. The generic UART driver code shared among the various
implementations has been enhanced to support the detection of the terminal
size using a protocol of escape sequences. This feature can be enabled by
including the attribute 'detect_size="yes"' in the policy of a UART client.
This is useful for combining UART drivers with the new 'terminal_mux'
server.
:ACPI support for 64-bit machines:
In addition to IOMMU-related modifications, the ACPI driver has been enhanced
to support 64-bit machines and MCFG table parsing has been added.
:PCI support for IOMMUs:
With the added support of IOMMUs, the 'Pci::Session' interface has been
complemented with a way to obtain the extended PCI configuration space in the
form of a 'Genode::Dataspace'. Also, the interface provides a way to allocate
DMA buffers for a given PCI device. Device drivers that are meant to be used
on system with and without IOMMUs should use this interface rather than
core's RAM session interface to allocate DMA buffers.
:Real-time clock on x86:
Up to now, the x86 real-time clock driver served as a mere example for
accessing I/O ports on x86 machines but the driver did not expose any service
interface. With the newly added 'os/include/rtc_session' interface and the
added support of this interface in the RTC driver, Genode programs have now
become able to read the real-time clock. Currently, the interface is used by
the Vancouver VMM.
:USB driver restructured, support for Arndale board added:
While adding support for the Exynos-5-based Arndale board, we took the
chance to restructure the driver to improve portability to new
platforms. The most part of the driver has become a library, which is
built in a platform-specific way. The build system automatically selects
the library that fits for the platform as set up for the build directory.
Platforms
#########
NOVA
====
The NOVA base platform received major improvements that address the kernel
as well as Genode's NOVA-specific code. We pursued two goals with this line
of work. The first goal was the use of NOVA in highly dynamic settings, which
was not possible before, mainly due to lacking kernel features. The second
goal was the use of IOMMUs.
NOVA is ultimately designed for accommodating dynamic workloads on top of the
kernel. But we found that the implementation of crucial functionality was
missing. In particular, the kernel lacked the ability to destroy all kinds of
kernel objects and to reuse memory of kernel objects that had been destroyed.
Consequently, when successively creating and destroying kernel objects such as
threads and protection domains, the kernel would eventually run out of memory.
This issue became a show stopper for running the Genode tool chain on NOVA
because this scenario spawns and destroys hundreds of processes. For this
reason, we complemented the kernel with the missing functionality. This step
involved substantial changes in the kernel code. So our approach of using the
upstream kernel and applying a hand full of custom patches started to show its
limitations.
To streamline our work flow and to track the upstream kernel in a structured
way, we decided to fork NOVA's Git repository and maintain our patches in our
fork. For each upstream kernel revision that involves kernel ABI changes, we
create a separate branch called "r<number>". This branch corresponds to the
upstream kernel with our series of custom patches applied (actually rebased) on
top. This way, our additions to the upstream kernel are well documented. The
'make prepare' mechanism in the base-nova repository automates the task of
checking out the right branch. So from the Genode user's point of view, this
change is transparent.
The highly dynamic application scenarios executed on NOVA triggered several
synchronization issues in Genode's core process that had not been present on
other base platforms. The reason for those issues to occur specifically on NOVA
lies in the concurrent page fault handling as employed on this base platform.
For all classical L4-like kernels and Fiasco.OC, we use one global pager thread
to resolve all page faults that occur in the whole Genode system. In contrast,
on NOVA we use one pager thread per user thread. Consequently, proper
fine-grained synchronization between those pager threads and the other parts of
core is mandated. Even though the immediate beneficiary of these changes is the
NOVA platform, many of the improvements refer to generic code. This paves the
ground for scaling the page-fault handling on other base platforms (such as
Fiasco.OC) to multiple threads. With these improvements in place, we are able
to successfully execute the 'noux_tool_chain_nova' scenario on the NOVA kernel
and build Genode's core on NOVA. That said, however, not all issues are covered
yet. So there is still a way left to go to turn base-nova into a base platform
that is suitable for highly dynamic scenarios.
The second goal was the use of NOVA's IOMMU support on Genode. This topic is
covered in detail in section [DMA protection via IOMMU].
To be able to use and debug Genode on NOVA on modern machines that lack legacy
comports, we either use UART PCI cards or the Intel's Active Management
Technology (AMT) mechanism. In both cases, the I/O ports to access the serial
interfaces differ from the legacy comports. To avoid the need for adjusting the
I/O port base addresses per platform, we started using the chain-boot-loader
called "bender" developed by the Operating Systems Group of TU Dresden,
Germany. This boot loader is started prior the kernel, searches the PCI bus for
the first suitable device and registers the corresponding I/O port base address
at the bios data area (BDA). Genode's core, in turn, picks the I/O port base
address up from the BDA and uses the registered i8250 serial controller for its
LOG service.
Execution on bare hardware (base-hw)
====================================
The base-hw platform enables the use of Genode on ARM-based hardware without
the need for a 3rd-party kernel.
With the new release, the range of supported ARM-based hardware has been
extended to cover the following additional platforms. With the previous
release, we introduced the support for Freescale i.MX family of SoC, starting
with i.MX31. The current release adds support for the i.MX53 SoC and adds
a user-level timer driver for this platform. With the Samsung Exynos 5, the
first Cortex-A15-based SoC has entered the list of supported SoCs. Thanks to
this addition, Genode has become able to run on the
[http://www.arndaleboard.org - Howchip Arndale board]. At the current state,
core and multiple instances of init can be executed but drivers for peripherals
are largely missing. Those will be covered by our ongoing work with this SoC.
The added platforms are readily available via the 'create_builddir' tool.
To make base-hw practically usable on real hardware (i.e., the Pandaboard),
support for caches has been implemented. Furthermore, the implementation of the
signalling API underwent a redesign, which leverage the opportunities that
arise with tailoring a kernel specifically to the Genode API. As a side-benefit
of this endeavour, we could unify the 'base/signal.h' header with the generic
version and thereby took another step towards the unification of the Genode
headers across different kernels.
Microblaze platform removed
===========================
The 'base-mb' platform has been removed because it is no longer maintained.
This platform enabled Genode to run directly on the Xilinx Microblaze softcore
CPU. For supporting the Microblaze CPU architecture in the future, we might
consider integrating support for this architecture into base-hw. Currently
though, there does not seem to be any demand for it.
Fiasco.OC forked, support for Exynos 5 SoC added
================================================
In the last release cycle, we went beyond just using the Fiasco.OC kernel and
started to engage with the kernel code more intensively. To avoid that the
management of a growing number of kernel patches goes out of hand, we forked
the Fiasco.OC kernel and conduct our development in our Fiasco.OC Git
repository. When using the 'make prepare' mechanism in the 'base-foc'
repository, the new Git repository will be used automatically. There exists a
dedicated branch for each upstream SVN revision that we use. We started with
updating Fiasco.OC to the current revision 47. Hence, the current branch used
by Genode is named "r47". The branch contains the unmodified state of the
upstream SVN repository with our modifications appearing as individual commits
on top. This makes it easy to keep track of the Genode-specific modifications.
Please note that the update to Fiasco.OC requires minor adaptations inside
the 'ports-foc' repository. So for using L4Linux, "make prepare" must be
issued in both repositories 'base-foc' and 'ports-foc'.
Speaking of engaging with the kernel code, the most profound improvement is
the support for the Samsung Exynos-5-based Arndale board that we added to the
kernel. This goes hand in hand with the addition of this platform to Genode.
For creating a build directory targeting the Arndale board, just specify
"foc_arndale" to the 'create_builddir' tool. At the time of the release,
several basic scenarios including the timer driver and the USB driver are
working. Also, both Cortex-A15 CPUs of the Exynos 5 SoC are operational.
However, drivers for most of the peripherals of the Exynos-5 SoC are missing,
which limits the current scope of Genode on this platform.
Linux
=====
Since the base-linux platform became used for more than a mere development
vehicle, we are revisiting several aspects of this base platform. In the last
release, we changed the synchronous inter-process-communication mechanism to
the use of SCM rights. For the current release, it was time to have a closer
look at the memory management within core. The Linux version of core used a
part of the BSS to simulate access to physical memory. All dataspaces would
refer to a portion of 'some_mem'. So each time when core would access the
dataspace contents, it would access its local BSS. For all processes outside of
core, dataspaces were represented as files. We have now removed the distinction
between core and non-core processes. Now, core uses the same 'Rm_session_mmap'
implementation as regular processes. This way, the 'some_mem' could be
abandoned. We still use a BSS variable for allocating core-local meta data
though. The major benefit of this change is the removal of the artificial
quota restriction that was imposed by the predefined size of the 'some_mem'
array. Now, the Linux base platform can use as much memory as it likes. Because
the Linux kernel implements virtual memory, we are not bound by the physical
memory. Hence, the available quota assigned to the init process is almost
without bounds.
To implement the fault-detection mechanism described in section
[New fault-detection mechanism] on Linux, we let core catch SIGCHLD signals of
all Genode processes. If such a signal occurs, core determines the process that
produced the signal by using 'wait_pid', looks up the CPU session that belongs
to the process and delivers an exception signal to the registered exception
handler. This way, abnormal terminations of Genode processes are reflected to
the Genode API in a clean way and Genode processes become able to respond to
terminating Genode child processes.
OKL4
====
The audio stub driver has been removed from OKLinux. Because of the changed
'Audio_out::Session' interface, we needed to decide on whether to adapt the
OKLinux stub driver to the changed interface or to remove the stub driver.
Given the fact that OKLinux is not actively used, we decided for the latter.

943
doc/release_notes-13-05.txt
View File

@@ -1,943 +0,0 @@
===============================================
Release notes for the Genode OS Framework 13.05
===============================================
Genode Labs
With Genode 13.05, we have diverged quite a bit from the feature-laden plans
laid out in our [http://genode.org/about/road-map road map] as we realized
that consolidating and optimizing the current feature set will have a more
sustainable effect than functional enhancements at this point. In particular,
we addressed the problem that the ever growing diversity of platforms imposes
on the quality and coverage of testing. We also desired to extend our
systematic testing efforts to real hardware platforms, and to have a mechanism
for detecting performance regressions. Section
[Automated quality-assurance testing] details how we approached these
challenges, and how we went on analyzing Genode's network performance in
particular.
That said, we haven't completely restrained ourself from implementing new
features. Closely related to test automation but very useful in other
situations, we improved the terminal infrastructure in order to enable the
interactive use of dynamic system scenarios in headless situations. Section
[Terminal infrastructure] introduces a new command-line interface for managing
Genode subsystems.
With regard to platform support, the current release follows up on the
hardware support added in the previous releases. For Samsung Exynos-5-based
platforms, drivers for USB-3, fast-ethernet networking, gigabit networking,
eMMC, and SATA have been added. For Freescale i.MX53-based devices, new
drivers for display, touchscreen, and GPIO have become available. The
OMAP4 display driver has been enhanced to cover both LCD displays and HDMI.
Our custom base-hw kernel has been enabled on the Raspberry Pi
board. Finally, Linux/ARM was added to accompany Linux/x86 as a fully usable
Genode base platform.
Automated quality-assurance testing
###################################
One of the greatest challenges of the Genode OS Framework is preventing
regressions in the face of the growing number of supported platforms.
The challenge stems from the fact that the space of Genode scenarios grow
two-dimensional. On one axis, the software stack on top of Genode gets more
and more complex, which calls for contiguous testing. On the other axis, there
is a growing number of kernel and hardware platforms to support.
In principle, there are even more dimensions, for example the diversity
of tool chains or the diversity of the OS used on the development machine.
Luckily, the problem of tool-chain diversity could be mitigated with the
introduction of the Genode tool chain since version 11.11, which was a huge
relief. However, the mentioned two dimensions cannot be avoided. Because
manual testing of manifold scenarios of component compositions on top of many
different kernels became infeasible, we automated the task of building and
testing years ago.
The automated builder checks out the staging branch of Genode, prepares
the repositories that integrate 3rd-party code, and builds the software
for 12 different kernel/platform combinations. Not all 3rd-party software
packages are built for each combination though. But we make sure that each
piece of software is exposed to different combinations of CPU architectures
and kernels.
The build test is accompanied with automated runtime tests of various
run scripts on Qemu. Each run script listed in 'tool/autopilot.lst' is
executed on each kernel using the autopilot tool. The tests range from
stimulating low-level mechanisms (such as signal, timer, and ldso) to complex
scenarios (such as testing networking with L4Linux, or running Noux).
Both build and runtime tests are executed daily. If any of the
tests fail, the Genode developers receive a notification email.
Once all tests are passed, the staging branch can be merged into the master
branch. This way, we spare the users of Genode to deal with intermediate
problems introduced in the staging branch.
The build and runtime tests have become a fundamental tool for our
development work. With the growing variety of real hardware
(as opposed to hardware emulated via Qemu), however, our existing solution
was falling short. Even though our tests confirm that Genode is running
happily on Qemu, they won't help us to detect regressions in our device
drivers for non-Qemu hardware such as Pandaboard, Arndale, or modern PC
hardware. Furthermore, we are increasingly focussing on performance
considerations. In order to be a viable OS platform, Genode does not only need
to be able to do networking, but networking performance must be on par with
mainstream OSes. This raises the new challenge to extend our
continuous-testing tools to become continuous-benchmarking tools. The ultimate
goal is to monitor the performance of Genode on real hardware over long
periods of development.
In this release cycle, we attacked this problem in two steps. First, we
enabled Genode's run tool to target not only Qemu but real hardware, with the
premise that existing run scripts must not be changed. The second step is the
creation of new run scripts that perform benchmarks in an automated fashion.
By aggregating the results of this automatically executed benchmarks, we can
correlate performance effects with commits in our code repository.
Targeting real hardware via the run tool
========================================
In the following, we briefly describe the procedure to execute run scripts
on native hardware, for both Intel-based x86 machines and ARM-based platforms.
TFTP boot x86
~~~~~~~~~~~~~
The following description uses NOVA as an example to illustrate the usage.
Other base platforms are supported as well and can be configured analogously.
[http://os.inf.tu-dresden.de/~us15/pulsar/ - Pulsar] is a tiny boot loader
that uses PXE to fetch boot images via TFTP over the network. On the x86
architecture, Genode supports the automatic generation of Pulsar configuration
files, which can be placed directly onto a TFTP server. Genode can be booted
via Pulsar using the following steps:
* On the x86 test machine, enable "PXE boot feature" in the BIOS.
* When booting, the machine will look for a DHCP server announcing a TFTP server.
So you need to make sure to have both the DHCP server and the TFTP server
configured such that the 'pulsar' binary will be loaded as PXE binary.
* After the PXE BIOS of the test machine has loaded and started the pulsar
binary, Pulsar will look on the TFTP server for a file called
'config-XX-XX-XX-XX-XX-XX', where the sequence of 'XX' corresponds to the
MAC address of the test machine.
For example, if the MAC of the network card is 01:02:03:04:05:06, Pulsar
would request a file called 'config-01-02-03-04-05-06'.
* Using this configuration file, we direct Pulsar to the configuration
generated by the run tool. I.e., it should look as follows
! root /tftpboot/nova
! config config-00-00-00-00-00-00
The lines above tell pulsar to load another config file, which contains the
actual configuration. To instruct the run script to actually generate the
'config-00-00-00-00-00-00' file, set the following environment variables in
your shell prior executing the run script:
! export PXE_TFTP_DIR_BASE=/tftpboot
! export PXE_TFTP_DIR_OFFSET=/nova
The two-staged configuration of Pulsar may look overly complicated at first
sight but has the benefit that the run tool does not need to know the MAC
address of the test machine in order to generate the Pulsar configuration
file.
* Create a symbolic link '/tftpboot/nova' pointing to the corresponding
Genode build directory.
* The next time 'make run/printf' is invoked,
the run script will generate the 'config-00-00-00-00-00-00' in
'/tftpboot/nova'.
* When rebooting the test machine, it will load and start the printf test.
TFTP boot using U-Boot
~~~~~~~~~~~~~~~~~~~~~~
Configure your U-Boot boot loader to load the images via TFTP.
The remainder of the procedure is similar to the description for x86 above.
On ARM platforms, the run tool automatically generates the uBoot image and
creates a symbolic link into the TFTP directory.
* Pandaboard:
! export PXE_TFTP_DIR_BASE=/tftpboot
! export PXE_TFTP_DIR_OFFSET=/panda
! ln -s <genode-build-dir> /tftpboot/panda
! RUN_OPT="--target uboot" make run/printf
* Arndale board:
! export PXE_TFTP_DIR_BASE=/tftpboot
! export PXE_TFTP_DIR_OFFSET=/arndale
! ln -s <genode-build-dir> /tftpboot/panda
! RUN_OPT="--target uboot" make run/printf
Output and reset with Intel's AMT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Most modern x86-based machines lack a COM port, which is normally used for
kernel debug messages as well as LOG messages printed by Genode's core.
However, Intel's Advanced Management Technology (AMT) can be used to obtain
the serial output of the test machine and to reset the test machine. To use
AMT with Genode's run tool, install the 'amtterm' package (version 1.3 is
known to work well) and set the following environment variables, specifying
the IP address of the test machine and the AMT password.
! export AMT_TEST_MACHINE_IP=XXX.XXX.XXX.XXX
! export AMT_TEST_MACHINE_PWD=XXXXXXXXX
Via setting the RUN_OPT environment variable, we instruct the run tool to use
AMT instead of Qemu. The following command will reset the test machine, the test
machine will load the binaries of the printf run script via PXE, and we will be
able to see the serial output of the test machine through Intel's AMT Serial
Over Line (SOL),
! RUN_OPT="--target amt" make run/printf
Output via a COM port (UART)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If the x86 test machine, Pandaboard or Arndale test board is connected
via UART, the run tool can use a specified command to interact with it.
For example, if the UART interface of the test machine is connected directly
to the host machine at /dev/ttyUSB3, and the picocom tool is available,
the following command can be used to establish a connection:
! RUN_OPT="--target serial --serial-cmd \"picocom -b 115200 /dev/ttyUSB3\"" make run/printf
Alternatively, if the board is connected to some remote machine, which exports
the corresponding serial line via TCP/IP, the socat tool can be used for
communicating with the remote test machine:
! RUN_OPT="--target serial --serial-cmd \"socat - tcp:10.0.0.1:2000\"" make run/printf
Reset via a IP power plug NETIO-230B from Koukaam
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
At Genode Labs, we use a NETIO-230B power plug to automate power-cycling ARM
boards. This power plug can be controlled over the network. For example, if
the Pandaboard is connected to power port 3, the following command will
automatically turn on the board when the run script is started:
! RUN_OPT="--target uboot --target reset --reset-port 2 --reset-ip 10.0.0.1 --reset-user admin --reset-passwd secret" make run/printf
The '--target reset' option can be combined with '--target uboot' to
instruct the run tool to boot via TFTP (as described above) and take care
of power cycling. When the run script has finished, the specified port
will be automatically switched off by the run tool.
Of course, the IP address settings, as well as the actual user name and
password, to access the NETIO-230B power plug, have to be adjusted accordingly.
Automated benchmarking
======================
With the '--target' features added to the run tool, the road is paved to
obtain benchmark results in an automated fashion. Currently, we are most
interested in exploring the network-performance characteristics of Genode.
Network performance can be explored at different levels. We started with
looking at raw driver performance, then looked at the overhead of separating
the network application from the device driver (and thereby introducing
inter-process communication overhead), and finally explored the effects
of the TCP/IP stack.
For pursuing the packet-level performance measurements, we crafted a library
called 'net-stat', which contains the application logic of a low-level
benchmark operating at network-packet level. This library has been
successively incorporated into the 'dde_ipxe' NIC driver and the 'usb_drv'
(NIC driver via ethernet-over-USB) to measure the raw driver performance
without any microkernel overhead or TCP/IP protocol overhead.
To see the influence of the inter-process communication, namely the
packet-stream interface employed by Genode's NIC-session interface,
we implanted the same net-stat library into a NIC-session client. This
experiment enables us to compare the operation of the NIC driver
with the operation of a NIC driver separated from the NIC
application.
The raw networking tests can be executed automatically using the set
of 'network_test_nic*.run' scripts located at 'os/run'.
The scenario sends raw ethernet packets from the host machine to the
target machine. Three tests are provided:
The 'network_test_nic_raw.run' test measures the net-stat-instrumented driver
(of usb_drv and net_drv respectively) to observe the raw receive performance.
The 'network_test_nic_raw_client.run' test implements the benchmark in a
NIC-session client connected to the NIC driver running as a separate
component whereas the NIC driver is not instrumented.
The 'network_test_nic_raw_bridge_client.run' test further adds a NIC bridge
in-between the driver and the NIC-session client.
In addition to analyzing the performance on a low level, we investigated
the effects of TCP/IP for the application performance. This topic is
covered in more detail in Section [TCP/IP performance].
Terminal infrastructure
#######################
Closely related to the quality-assurance measures detailed in the previous
section, there is the arising need to interact with increasingly complex system
scenarios in headless settings. In particular when executing tests remotely on a
development board, manual user-interaction via a GUI
becomes impractical. We vastly prefer a low-bandwidth textual interface
in such situations. But how should a textual user interface for dynamic
systems comprised of many components look like? This is particularly difficult
because most development boards are equipped with merely a single UART
connector.
On a normal Genode system, the UART connector is typically used
by the kernel debugger to print debugging output, or for the interactive
use of a debugger. This leaves no interface for interacting with Genode
components. So how can we expose complex scenarios, such as concurrently
running several instances of Genode subsystems, to the user?
Our solution consists of three parts: A pseudo UART driver for Genode
that uses the kernel debugger as back end, a terminal-multiplexing
facility running on the reference platform, and a command-line based
tool for interacting with Genode. By combining those, the user
can interact with the kernel debugger, a Genode command line, and the
consoles of executed Linux instances over a single serial connection.
The pseudo UART driver called kdb_uart_drv is a Genode service that
implements the 'Uart::Session' interface. Therefore, it can be combined
with all components that use the 'Uart::Session' or the 'Terminal::Session'
interfaces, for example the Noux runtime environment, the terminal_log
service (for displaying LOG messages via the terminal interface), L4Linux, or
programs linked against the 'libc_terminal' plugin. The kdb_uart_drv component
is located at 'os/src/drivers/uart/kdb'. It does not access a real UART device
but rather uses the user-level bindings of the kernel debugger to indirectly
read and write data over the UART interface.
[image kdb_uart_drv 65%]
The kdb_uart_drv driver used for sharing one UART among the kernel
debugger, core's LOG service, and a terminal client application
running on Genode.
Figure [kdb_uart_drv] illustrates the relationship between the kernel
debugger, core's LOG service, and kdb_uart_drv. Because write operations
target the kernel debugger directly, core's LOG service gets bypassed. Output
written to the kdb_uart_drv will directly appear at the terminal program of
the host system. Because kdb_uart_drv has
direct access to the host terminal, it can leverage all facilities of the host
terminal, in particular various escape sequences for terminal manipulations.
For reading from the kernel debugger, there is no way to block for UART input.
Hence, the kdb_uart_drv periodically polls for new input with a period of 20
milliseconds. If new input is available, the driver reads as many characters as
available at once. So the runtime overhead of polling is negligible. To test
kdb_uart_drv as individual component, there is a run script provided at
'os/run/kdb_uart_drv.run'.
Thanks to kdb_uart_drv, both the kernel debugger and Genode can
share one single UART connection. So we have a principal way to let the user
interact with a Genode component that uses the 'Terminal::Session' interface.
However, typical system scenarios should accommodate not just a single program
but multiple Linux instances and native Genode applications simultaneously,
each requiring a dedicated 'Terminal::Session'. Hence, we need a way to
multiplex the 'Terminal::Session' interface between those clients. Our
multiplexing solution comes in the form of a component called terminal_mux,
which we just introduced in the
[http://genode.org/documentation/release-notes/13.02#New_terminal_multiplexer - previous release].
It uses a single terminal connection to implement a text-based user interface
to multiple virtual terminal consoles.
[image terminal_mux 40%]
Operation of the terminal_mux service.
Figure [terminal_mux] depicts the basic functioning of this component. For
terminal_mux clients, the service implements the Linux terminal capabilities.
For doing that, it shares large parts of the implementation of the existing
Genode terminal program. For each client, terminal_mux renders the client
output into a client-specific text-screen buffer. So any number of clients can
perform output on terminal_mux concurrently. According to the selection by
the user, terminal_mux periodically translates one client buffer (the
foreground buffer) to escape sequences as understood by the host terminal. This
translation is performed using the ncurses library. The user can pick the
foreground buffer using an interactive menu that can be activated via the
keyboard shortcut _Control-x_.
By combining kdb_uart_drv with terminal_mux, we created a flexible way
to let the user interact with many Genode applications. The last part
missing for a real dynamic system is a text-based command interface to
start and stop Genode subsystems. This functionality is provided by the
new cli_monitor component located at 'os/src/app/cli_monitor'.
It uses the 'Terminal::Session' interface to present a simple interactive
command line with commands for starting and stopping Genode subsystems,
entering the kernel debugger, and showing status information. It provides
tab completion and inline help to make it easily explorable. The cli_monitor
component is integrated in the scenario of the 'terminal_mux.run' script
mentioned above. Because cli_command is a 'Terminal::Session' client, it can
be interfaced with terminal_mux. This composition is illustrated by Figure
[uart_overview].
[image uart_overview 100%]
Overview of the terminal infrastructure as employed in the
demonstration scenario.
Note that in some situations, e.g., when killing subsystems, the kernel, core,
or the init process may print LOG messages. Because those messages are
naturally not routed through terminal_log, they will interfere with the
operation of terminal_mux and thereby result in visible inconsistencies.
Pressing _Control-x_ will clear such artifacts. This will bring up the
terminal_mux menu, which implicitly triggers the redraw of the entire terminal.
Base framework
##############
The current release comes with incremental improvements of the MMIO framework
API and a new utility to ease the synchronized accesses to otherwise
unsynchronized class interfaces.
:MMIO framework improvements:
For native Genode device drivers, we consistently use our
[http://genode.org/documentation/release-notes/12.02#MMIO_access_framework - MMIO framework API].
These utilities help us to safeguard the access to individual bit fields of
memory-mapped device registers and cleanly separate the declaration of device
registers from the driver logic. During the increased use of the API, we
observe that the 'Genode::Mmio' class template operates mostly on addresses
that belong to dataspaces provided by core's IO_MEM service. Those dataspaces
are typically obtained via the 'Attached_io_mem_dataspace' convenience class,
which requests the dataspace and attaches it to the local address
space at once. To further reduce repetitive code, we introduced the new
'Attached_mmio' class (located at 'os/attached_mmio.h'), which handles the
common case of making the content of a IO_MEM dataspace available through
register definitions using the 'Mmio' utility. Furthermore, the MMIO framework
API has been enhanced with a variant of the 'Mmio::wait_for()' function that
waits for whole register values rather than bits.
:Synchronized interfaces:
Most Genode programs are multi-threaded, which makes the proper use of locks
inevitable. For most data structures, Genode does not implicitly manage the
locking but expects the user of the data structures to know what he is doing.
This way, we can avoid the locking overhead if a data structure is known to be
accessed by a single thread only. If accessed by multiple threads, we usually
wrap such data structures within an accessor interface that takes care of the
locking. For example, for the 'Allocator' interface, there exists a
corresponding 'Synchronized_allocator' interface wrapper. This technique works
well as long as the number of interfaces is low -- as is the case for Genode's
base API. However, as the wrapper code is for the most part pretty dumb, we'd
like to avoid it. Also, when using the Genode API to implement programs on
top, we do not anticipate manually creating such accessor wrappers. To ease
the creation of synchronized interfaces, we introduced the new
'Synced_interface' class template. It takes a pointer to an existing interface
and a lock as arguments. An instance of a 'Synced_interface' provides
synchronized access to the wrapped interface functions via the 'operator ()'.
Because the 'Synced_interface' does not provide any means to obtain the
unsynchronized version of the interface, once wrapped, the interface cannot be
misused by subsystems that get handed over a reference to a
'Synced_interface'. To see how to employ this utility, please have a look of
how we realize the synchronization within the Vancouver VMM (in particular,
the access to the motherboard).
Low-level OS infrastructure
###########################
TCP/IP performance
==================
On the course of the automated benchmarking described in Section
[Automated quality-assurance testing], we conducted the following steps
to enable benchmarks and to improve performance at the TCP/IP level.
At application level, we desire to compare our network performance with the
performance on GNU/Linux using commodity benchmarks. For this reason, netperf
has been ported to run as native Genode using the lwIP stack. This benchmark
allows us to systematically compare our results with those achieved by Linux.
The port of netperf is available in the ports repository.
In addition to running a commodity benchmark, we pursue synthetic benchmarks
that model the behaviour of typical application scenarios, for example, a
web server that receive many small requests. This is where the added
'test-ping_client' and 'test-ping_server' tests come into play. The test
is located at 'libports/src/test/lwip/pingpong'. It is used by the
series of 'network_test_*.run' scripts located at 'libports/run'. The
run scripts exercise the test in various scenarios and thereby allow us to
systematically explore the impact of the libc and NIC bridge on the
application performance.
# Using raw lwIP without the libc
# Like the first test, but with an instance of the NIC bridge in between the
test program and the driver.
# Using lwIP with the libc socket bindings
# Like the third test, but with NIC bridge added
To keep track of the lwIP development more closely, we switched to the
Git version of lwIP instead of using a source snapshot.
Furthermore, we incorporated "window scaling" support (RFC 1323) into our
version of lwIP as we identify the TCP window size as a limiting factor
of the TCP throughput achieved via lwIP.
C runtime
=========
We added support for "resolv" functionality to the libc_lwip_nic_dhcp plugin.
Normally, a file called 'resolv.conf' is expected to be located at '/etc'.
On Genode, however, we don't have a global file system, which makes this
way of configuration cumbersome. To ease the provision of a simple default
'resolv.conf' configuration, the plugin hands out the file as a virtual file.
The configuration automatically provides the DNS server address acquired by
lwIP via DHCP. If, for some reason, this policy is not desired, the feature
can be disabled via:
! <libc resolv="no" />
*Note that the configuration of the C runtime has changed*
To foster consistency of the libc configuration, we moved the static
network "interface" attributes into the 'libc' XML node. A new configuration
of static networking would look as follows:
! <libc ip_addr="..." netmask="..." gateway="..." />
Terminal
========
Genode's custom terminal implementation has been improved to better handle
widely used escape sequences.
The new version is able to handle two-argument SGR commands with
attribute/color arguments in any order, and supports the ED, EL0, and
CUB commands.
Because the terminal classes do not rely on any 3rd-party code, they
have been moved to the os repository at 'os/include/terminal'. This way,
we can use those classes by other components of the os repository such
as the new CLI monitor.
FS-LOG service
==============
Using the new FS-LOG service residing at 'libports/os/src/server/fs_log', log
messages of different processes can be redirected to files on a file-system
service. The assignment of processes to files can be expressed in the
configuration as follows:
! <start name="fs_log">
! <resource name="RAM" quantum="2M"/>
! <provides><service name="LOG"/></provides>
! <config>
! <policy label="noux" file="/noux.log" />
! <policy label="noux ->" file="/noux_process.log" />
! </config>
! </start>
In this example, all messages originating from the noux process are directed
to the file '/noux.log'. All messages originating from children of the noux
process end up in the file '/noux_process.log'.
Liquid FB
=========
Liquid FB is a virtual framebuffer service that uses the nitpicker GUI
server as back end. The virtual framebuffer is presented as a movable
window with a title bar. Until now, we used it primarily for demonstration
purposes, i.e., it is part of Genode's default demo scenario.
Thanks to our forthcoming adaptation of Qt5 to Genode, which requires a
very similar solution to interface Qt5's platform-abstraction layer (QPA) to
Genode, liquid FB got in the spotlight of this release.
First, we took the chance to update its configuration parameters to
become more consistent with similar services such as nit_fb. As liquid_fb was
originally conceived at a time when Genode's XML parser did not support
XML attributes, its configuration syntax used to be a bit arcane. This
has changed now. Apart from this cosmetic refinement, there are two prominent
new features: Support for resizing the framebuffer window with the
mouse and support for dynamic reconfiguration of the virtual framebuffer
via Genode's configuration mechanism.
When the liquid FB window gets resized by the user, the virtual framebuffer
emits a mode-changed signal to its client, which, in turn can handle the
event by re-acquiring the frame-buffer dataspace.
The added support for dynamic reconfiguration allows for changing the
properties of a liquid FB instance via Genode's configuration mechanism.
For example, the window position and size can be manipulated this way.
Furthermore, two new configuration options have been added. The
'resize_handle' option shows or hides the resize handle widget at the
lower-right window corner (by default, it is hidden). The 'decoration' option
defines whether window decorations should be visible (default is yes). Both
options can have the values "on" or "off".
3rd-party libraries
###################
The following 3rd-party libraries have been added or updated:
* To complement libSDL, we have added ports of SDL_ttf, SDL_image,
SDL_image, SDL_mixer, and SDL_loadso. Those additions to libSDL
are used by popular libSDL-based applications such as Tuxpaint.
They are now available at the libports repository.
* GNU FriBidi 0.19.5 added to the libports repository
* Qt4 updated to version 4.8.4
* zlib updated to version 1.2.8
Device drivers
##############
Unified driver names
====================
The growing diversity of supported hardware platforms calls for improved
conventions of how to name device drivers. Otherwise, run scripts that are
meant to support a wide range of platforms will eventually become more
and more complicated due to platform-dependent conditional configuration
snippets. For example, the default framebuffer drivers of the respective
platforms used to be called "vesa_drv" (for x86), "omap4_fb_drv", or "pl11x_drv".
In order to support the different platforms, run scripts that were otherwise
platform-agnostic had to explicitly deal with those differences.
To solve this issue, we introduced a generic SPEC values for device types, for
which a default driver is expected to exist. If a platform features a
framebuffer driver, it includes the SPEC value "framebuffer". On each
platform, the default driver for the respective device has the same name. So
each of "vesa_drv", "pl11x_drv", and "omap4_fb_drv" had been renamed to
"fb_drv". This is possible because the use of those drivers is mutually
exclusive.
The same convention has been applied to GPIO drivers as well. The
corresponding SPEC value is called "gpio". The driver binaries are called
"gpio_drv".
ATAPI
=====
LBA48 support has been added to the ATAPI driver. Thanks to Ivan Loskutov!
KDB UART driver for L4/Fiasco and Fiasco.OC
===========================================
The new KDB UART driver at 'os/src/drivera/uart/kdb' uses the kernel debugger
console as backend for input and output. This is useful in the case that only
one UART is available as described in Section [Terminal infrastructure].
Examples for using the kdb_uart_drv are available in the form of the run scripts
'ports-foc/run/l4linux.run' and 'os/run/kdb_uart_drv.run'.
Revised GPIO session interface
==============================
The original design of the GPIO session interface enabled the client of a
single session to interact with any number GPIO pins. Each function of the
interface took a GPIO number as first argument, which addressed the GPIO pin.
To simplify the interface and to enable fine-grained GPIO-assignment policies,
the interface has been changed to provide access to a single GPIO pin per
session only. At session creation time, the client specifies a single GPIO
pin, to which the session refers. This information can be evaluated for the
session routing. So access-control policies can be easily implemented per GPIO
pin. The server stores the pin as part of the session context and implicitly
uses the pin for operations on the session interface.
Furthermore, a generic driver interface for GPIO-class-device drivers
has been introduced. The new interface at 'os/include/gpio' alleviates the
need to implement the boilerplate code to interface the driver with Genode.
The existing GPIO drivers for OMAP4 and i.MX53 are the first beneficiaries of
these changes.
Exynos 5 SoC
============
After principally enabling the Exynos 5 SoC platform in the previous
release, we moved on with extending the device-driver coverage of this SoC. In
particular, we addressed USB networking, XHCI (USB-3), Gigabit networking over
USB-3, eMMC, and SATA.
The development of those device drivers follows our rationale that guided our
[http://genode.org/documentation/articles/pandaboard - previous work on the OMAP4 platform].
For the USB driver, we employed the device-driver-environment (DDE) approach
for reusing the Linux USB stack and the host controller drivers. In contrast,
the eMMC and SATA drivers are built as genuine Genode drivers with no
3rd-party code used.
Technically, the addition of Exynos-5 support to our USB driver was
an evolutionary step. It required us to add the corresponding EHCI
controller and to supply a few additions to the device-driver
environment. To simplify the driver, we decided to let the driver
rely on the platform initialization as performed by the U-Boot boot
loader. Since the initialization is performed during the boot process
already, there is no need to do this work twice. Because the platforms
supported by the USB driver become more and more diverse, we re-organized the
internal structure of the 'dde_linux' repository to keep those platforms well
separated. Furthermore, we reworked the memory management of the USB driver to
improve the utilization of the available RAM. The new solution employs Genode's
concept of managed dataspaces to manage a part of the local address-space
layout manually. This helps us to implement a fast translation of driver-local
virtual addresses to physical addresses as needed for issuing DMA requests.
The eMMC driver builds upon our protocol implementation for the SD-card
protocol, which was originally developed for the OMAP4 SD-card driver.
Because we kept the SD-card protocol implementation well separated
from the host-controller driver, it was possible to leverage parts of our
existing work for the eMMC driver. Because the eMMC protocol is an extension
of the SD-card protocol, however, we needed to enhance the protocol
implementation accordingly. The extension comprises support for the
MMC_SEND_EXT_CSD, MMC_SEND_OP_COND, and STOP_TRANSMISSION commands as well as
the MMC detection. The host controller driver was implemented from scratch
with the help of I/O access traces gathered from instrumenting the U-Boot boot
loader and the Linux kernel. The driver operates the eMMC in high-speed, 8-bit
mode at 52 MHz using DMA. The implementation can be found at
'os/src/drivers/sd_card/exynos5'.
The initial version of our new SATA driver for Exynos 5 has been implemented
from the ground up. Even though it is at an early stage, it has been
successfully tested with a UDMA-133 disk, e.g., our generic block test
is passed and the disk can be attached as a block device to an instance of
L4Linux.
Freescale i.MX SoC
==================
The support for the Freescale i.MX53 SoC has been extended by a number of
devices. All drivers reside in the os repository under the 'os/src/drivers'
subdirectory.
The general-purpose I/O (GPIO) driver located at 'gpio/imx53' implements the
revised GPIO-session interface.
The i.MX53 input driver provides support for the input devices featured on the
i.MX53 SABRE tablet. The tablet uses an Egalaxy touchscreen and Freescale's
MPR121 capacitative touch buttons. Both are supported by the new driver. The
driver is located at 'input/imx53'.
The new framebuffer driver for the i.MX53 quick-start board (QSB) as well as
the SABRE tablet comes with special support for using the
hardware overlay feature provided by the i.MX53 image processing unit (IPU)
Access to the overlay is implemented via an IPU-specific extension
of the framebuffer-session interface. To combine the driver well with
nitpicker using alpha-channels, optional support for double-buffering
is provided. The driver is located at 'framebuffer/imx53'.
As an abstraction of platform features that need to be accessed by
multiple drivers, a so-called platform driver has been introduced.
The platform driver safeguards the access to global resources such
as clocks and system-configuration bits. It can be found at 'platform/imx53'.
OMAP4 SoC
=========
The OMAP4 framebuffer driver used to support HDMI only, which was used
for connecting a display to the Pandaboard. To make the driver usable on
phones and tablets, the driver has been enhanced to support LCD output. Thanks
to Alexander Tarasikov for the patch and the insightful story about
[http://allsoftwaresucks.blogspot.com/2013/05/porting-genode-to-commercial-hardware.html - porting Genode to the B&N Nook HD+ tablet]!
USB
===
The USB driver of the 'dde_linux' repository has received substantial
improvements both feature-wise and under the hood.
First and foremost, the Linux device-driver environment, on which the
driver is based on, has been updated from kernel version 3.2 to version
3.9 as the latter version includes drivers for recent host controllers
such as DWC3 out of the box.
DWC3 is the host controller employed on the Exynos-5-based
Arndale platform for USB 3. We added the support needed to operate this
controller in XHCI mode and added support for Gigabit networking through
the ASIX AX88179 Gigabit-Ethernet Adapter as well as USB storage support.
Apart from extending the device-driver coverage, we revised the driver
internally. The back-end allocators for DMA buffers and normal memory have been
rewritten to allocate RAM more sparingly. Furthermore, we enabled the USB
driver for 64-bit x86 machines and improved the support for HID keyboards,
including the application of quirks to cherry keyboards.
*Note the change of the USB configuration*
With the addition of XHCI, the USB driver supports a growing number
of host controllers. In some situations, it is desirable to constrain the
driver to a subset of controllers only. For example, on the Arndale platform,
we desire to use a dedicated USB stack for XHCI, which operates completely
independent from the USB stack accessing USB-2. This way, gigabit networking
over USB-3 won't interfere with the operation of USB-2. To make this
possible, we added new configuration options to the USB driver.
With the new scheme, host controllers must be explicitly enabled in the
configuration. Supported config attributes are: 'uhci', 'ehci', and 'xhci'.
For example, a configuration snippet to enable UHCI and EHCI looks as
follows:
! <config uhci="yes" ehci="yes">
Updated iPXE device-driver environment
======================================
The iPXE device-driver environment was update to the most recent
iPXE upstream Git version in order to benefit from upstream improvements
of the Intel E1000 NIC driver.
Runtime environments
####################
Vancouver VMM on NOVA
=====================
Vancouver is the user-level virtual-machine monitor that accompanies the
NOVA hypervisor for hosting unmodified guest operating systems.
The most active line of development is led by Julian Stecklina at TU Dresden
via a fork called Seoul. In contrast to the original version of Vancouver,
this fork is open for outside contributions. Hence, it represents an ideal
platform for those parties with a stake in Vancouver to collaborate, i.e.,
the NUL userland, the NOVA runtime environment of TUD, and Genode.
In the current state of the transition, the Hip structure from Genode
is reused. String functions, which were formerly taken from NUL are now
provided by a stripped-down version of the C library called
'seoul_libc_support'. The nul/config.h is replaced by just using a constant
value in the one place where the file was needed.
The Genode-specific back ends of Vancouver, as largely introduced with the
previous Genode release, have been improved in several respects:
* CPUID 0x40000000: This instruction is issued by Linux when the KVM
guest support is compiled in. We have to return deterministic values to let
the Linux kernel survive.
* Replaced busy thread startup synchronization by proper locking.
* New locking scheme: We replaced the error-prone manual locking with the
use of the freshly introduced 'Synced_interface' for the motherboard and the
VCPU dispatcher. Also, all globally visible locks have been removed. They are
explicitly passed to subsystems only when needed.
* Improved PS/2 mouse back-end:
The previous version of the PS/2 mouse back end managed mouse-motion
events in a strange way, effectively throwing away most information
about the motion vector. Furthermore, the tracking of the mouse-button
states were missing. So drag-and-drop in a guest OS won't work. The new
version fixes those issues. For the transformation of input events to
PS/2 packets, the 'Genode::Register' facility is used, which greatly
simplifies the code.
L4Linux on Fiasco.OC
====================
We improved the memory management of L4Linux on Genode in two ways.
The first improvement is concerned about the upper limit of memory per Linux
instance. The corresponding discussion can be found at
[https://github.com/genodelabs/genode/issues/414 - issue #414].
We changed our L4Re emulation library to match the semantics of the original
L4Re more closely. Furthermore, we removed a heuristic in the L4Linux kernel,
which assumed that all kernel-local addresses above 0x8000000 refer to device
resources. In our version of L4Linux, there exist no MMIO resources. In
contrary, the virtual addresses above this addresses are used for normal
memory. By removing this artificial restriction with regard to the virtual
memory layout of the L4Linux kernel, we can host a larger kernel memory area.
The second improvement is concerned with the allocation of L4Linux
memory at Genode's core. Until now, L4Linux used to allocate its memory
as one contiguous RAM dataspace at core's RAM service. Core tries to
naturally align the allocation to improve the likelihood for large-page
mappings. So a dataspace is likely to be physically located at a
power-of-two boundary larger or equal than the dataspace size. For example,
the allocation of a 100 MiB RAM dataspace for a Linux instance will
be located at a 128 MiB boundary. If multiple of such allocations happen
sub-sequentially, this allocation strategy results in 28 MiB gaps between
100 MiB dataspaces. This memory cannot be used for large contiguous
allocations anymore. So even if the available memory capacity is far
larger than 100 MiB, an allocation of a 100 MiB block may fail.
To relieve this problem, we weakened the requirement for contiguous memory
by assembling L4Linux memory from multiple chunks of small dataspaces.
For example, by using a chunk size of 16 MiB, core's best-fit allocator
will have a better chance to find a more suited position for allocation
when aligning the block to a 16 MiB boundary compared to the allocation
of a larger block. Furthermore, slack memory can be used more efficiently
because smaller gaps (such as a 20 MiB gap) remain to be usable for L4Linux.
The discussion of this topic and the individual patch can be found at
[https://github.com/genodelabs/genode/issues/695 - issue #695].
Furthermore, the L4Linux block driver has been improved to support large
partitions.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Raspberry Pi
~~~~~~~~~~~~
Principal support for the Raspberry Pi platform has been added to the base-hw
kernel. The popular Raspberry Pi board is based on an ARMv6 Broadcom BCM2835
SoC. The current scope of the platform support comprises:
* IRQ controller driver: Because the interrupt controller uses a cascade of
registers, we settled on the following IRQ enumeration scheme.
IRQ numbers 0..7 refer to the basic IRQs.
IRQ numbers 8..39 refer to GPU IRQs 0..31.
IRQ numbers 40..71 refer to GPU IRQs 32..63.
* The kernel employs the so-called system timer for the preemptive scheduling.
* Core's LOG messages are printed over the PL011-based UART.
* The user-level timer driver uses the so-called ARM timer, which is a
slightly modified SP804 timer device.
Up to this point, a few device driver are missing to use Genode on the
Raspberry Pi in practice, most notably USB.
To build and run Genode on the Raspberry Pi, create a new build directory
via the 'create_builddir' tool, specifying 'hw_rpi' as platform.
User-level timer driver for Arndale platform
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
By adding our new Exynos 5250 PWM timer driver, the base-hw kernel can now
be used for executing meaningful scenarios on the Arndale board including
the USB stack and networking.
Linux
=====
Until now, Genode on Linux supported x86-based platforms only.
The newly added 'linux_arm' platform clears the way to run Genode directly on
Linux-based ARM platforms. Genode's entire software stack is supported,
including the dynamic linker, graphical applications, and Qt4.
As a known limitations, the libc 'setjmp()'/'longjmp()' doesn't currently
save/restore floating point registers.
Build system and tools
######################
The run tool has been enhanced as detailed in Section
[Automated quality-assurance testing].

995
doc/release_notes-13-08.txt
View File

@@ -1,995 +0,0 @@
===============================================
Release notes for the Genode OS Framework 13.08
===============================================
Genode Labs
The release of version 13.08 marks the 5th anniversary of the Genode OS
framework. We celebrate this anniversary with the addition of three major
features that we have much longed for, namely the port of Qt5 to Genode,
profound multi-processor support, and a light-weight event tracing
framework. Additionally, the new version comes with new device drivers for
SATA 3.0 and power management for the Exynos-5 SoC, improved virtualization
support on NOVA on x86, updated kernels, and integrity checks for
downloaded 3rd-party source code.
Over the course of the past five years, Genode's development was primarily
motivated by adding and cultivating features to make the framework fit for as
many application areas as possible. Now that we have a critical mass of
features, the focus on mere functionality does not suffice anymore. The
question of what Genode can do ultimately turns into the question of how well
Genode can do something: How stable is a certain workload? How does networking
perform? How does it scale to multi-processor systems? Because we are lacking
concise answers to these kind of questions, we have to investigate.
When talking about stability, our recently introduced automated testing
infrastructure makes us more confident than ever. Each night, over 200
automated tests are performed, covering various kernels and several hardware
platforms. All those tests are publicly available in the form of so-called run
scripts and are under continues development.
Regarding performance investigations, recently we have begun to benchmark
application performance focusing on network throughput. Interestingly, our
measurements reveal significant differences between the used kernels, but
also shortcomings in our software stack. For example, currently we see that
our version of lwIP performs poorly with gigabit networking. To thoroughly
investigate such performance issues, the current version adds support
for tracing the behaviour of Genode components. This will allow us to get a
profound understanding of all inter-component interaction that are on the
critical path for the performance of complex application-level workloads.
Thanks to the Genode architecture, we could come up with a strikingly simple,
yet powerful design for a tracing facility. Section
[Light-weight event tracing] explains how it works.
When it comes to multi-processor scalability, we used to shy away from such
inquiries because, honestly, we haven't paid much consideration to it. This
view has changed by now. With the current release, we implemented the
management of CPU affinities right into the heart of the framework, i.e.,
Genode's session concept. Additionally, we cracked a damn hard nut by enabling
Genode to use multiple CPUs on the NOVA hypervisor. This kernel is by far the
most advanced Open-Source microkernel for the x86 architecture. However,
NOVA's MP model seemed to inherently contradict with the API design of Genode.
Fortunately, we found a fairly elegant way to go forward and we're able to
tame the beast. Section [Enhanced multi-processor support] goes into more
detail.
Functionality-wise, we always considered the availability of Qt on Genode as a
big asset. With the current release, we are happy to announce that we finally
made the switch from Qt4 to Qt5. Section [Qt5 available on all kernels] gives
insights into the challenges that we faced during porting work.
In addition to those highlights, the new version comes with improvements all
over the place. To name a few, there are improved support for POSIX threads,
updated device drivers, an updated version of the Fiasco.OC kernel and
L4Linux, and new device drivers for Exynos-5. Finally, the problem of
verifying the integrity of downloaded 3rd-party source codes has been
addressed.
Qt5 available on all kernels
############################
Since its integration with Genode version 9.02, Qt4 is regarded as
one of the most prominent features of the framework. For users, combining Qt
with Genode makes the world of sophisticated GUI-based end-user applications
available on various microkernels. For Genode developers, Qt represents by far
the most complex work load natively executed on top of the framework API,
thereby stressing the underlying system in any way imaginable. We have been
keeping an eye on Qt version 5 for a while and highly anticipate the direction
where Qt is heading. We think that the time is right to leave Qt4 behind to
embrace Qt5 for Genode.
For the time being, both Qt4 and Qt5 are available for Genode, but Qt4 is
declared as deprecated and will be removed with the upcoming version 13.11.
Since Qt5 is almost API compatible to Qt4, the migration path is relatively
smooth. So we recommend to move your applications over to Qt5 during the
next release cycle.
In the following, we briefly describe the challenges we faced while adding Qt5
support to Genode, point you to the place where to find Qt5 in the source
tree, and give quick-start instructions for getting a Qt5 application
scenario running.
We found that the biggest architectural difference between version 4 and
version 5 is the use of the so-called Qt Platform Abstraction (QPA) interface,
which replaces the former Qt Window System (QWS).
Moving from QWS to QPA
======================
With Qt4, we relied on QWS
to perform the window handling. A Qt4 application used to create a session to
Genode's GUI server (called nitpicker) and applied its graphical output onto a
virtual framebuffer. The virtual framebuffer was not visible per se. To make
portions of the virtual framebuffer visible on screen, the application had to
create so-called nitpicker views. A view is a rectangular area of the physical
screen that displays a (portion of) the virtual framebuffer. The position,
size, and stacking order of views is managed by the application. For each Qt
window, the application would simply create a corresponding nitpicker view and
maintain the consistency of the view with the geometry of the Qt window. Even
though each Qt application seemingly operated as a full-screen application
with the windows managed by the application-local QWS, the use of nitpicker
views still allowed the integration of any number of Qt applications into one
windowed environment.
With the advent of compositing window managers, the typical way of how an
application interacts with the window system of the OS changed. Whereas
old-generation GUIs relied on a tight interplay of the application with the
window server in order to re-generate newly exposed window regions whenever
needed (e.g., revealing the window content previously covered by another
window), the modern model of a GUI server keeps all pixels of all windows in
memory regardless of whether the window is visible or covered by other
windows. The use of one pixel buffer per window seems wasteful with respect to
memory usage when many large windows are overlapping each other. On the other
hand, this technique largely simplifies GUI servers and makes the
implementation of fancy effects, like translucent windows, straight forward.
Since memory is cheap, the Qt developers abandoned the old method and fully
embraced the buffer-per-window approach by the means of QPA.
For Genode, we faced the challenge that we don't have a window server
in the usual sense. With nitpicker, we have a GUI server, but with a more
radical design. In particular, nitpicker leaves the management of window
geometries and stacking to the client. In contrast, QPA expects the window
system to provide both means for a user to interactively change the window
layout and a way for an application to define the properties (such as the
geometry, title, and visibility) of its windows.
The obviously missing piece was the software component that deals with window
controls. Fortunately, we already have a bunch of native nitpicker applications
that come with client-side window controls, in particular the so-called liquid
framebuffer (liquid_fb). This nitpicker client presents a virtual framebuffer
in form of a proper window on screen and, in turn, provides a framebuffer and
input service. These services can be used by other Genode processes, for
example, another nested instance of nitpicker.
This way, liquid_fb lends itself to be the interface between the nitpicker
GUI server and QPA.
For each QPA window, the application creates a new liquid_fb instance as a
child process. The liquid_fb instance will request a dedicated nitpicker
session, which gets routed through the application towards the parent of the
application, which eventually routes the request to the nitpicker GUI server.
Finally, the liquid_fb instance announces its input and framebuffer services
to its parent, which happens to be the application. Now, the application is
able to use those services in order to access the window. Because the
liquid_fb instances are children of the application, the application can
impose control over those. In particular, it can update the liquid_fb
configuration including the window geometry and title at any time. Thanks to
Genode's dynamic reconfiguration mechanism, the liquid_fb instances are able
to promptly respond to such reconfigurations.
Combined, those mechanisms give the application a way to receive user input
(via the input services provided by the liquid_fb instances), perform
graphical output (via the virtual framebuffers provided by the liquid_fb
instances), and define window properties (by dynamically changing the
respective liquid_fb configurations). At the same time, the user can use
liquid_fb's window controls to move, stack, and resize application windows as
expected.
[image qt5_screenshot]
Steps of porting Qt5
====================
Besides the switch to QPA, the second major change was related to the build
system. For the porting work, we use a Linux host system to obtain the
starting point for the build rules. The Qt4 build system would initially
generate all Makefiles, which could be inspected and processed at once. In
contrast, Qt5 generates Makefiles during the build process whenever needed.
When having configured Qt for Genode, however, the build on Linux will
ultimately fail. So the much-desired intermediate Makefiles won't be created.
The solution was to have 'configure' invoke 'qmake -r' instead of 'qmake'.
This way, qmake project files will be processed recursively. A few additional
tweaks were needed to avoid qmake from backing out because of missing
dependencies (qt5_configuration.patch). To enable the build of the Qt tools
out of tree, qmake-specific files had to be slightly adapted
(qt5_tools.patch). Furthermore, qtwebkit turned out to use code-generation
tools quite extensively during the build process. On Genode, we perform this
step during the 'make prepare' phase when downloading and integrating the Qt
source code with the Genode source tree.
For building Qt5 on Genode, we hit two problems. First, qtwebkit depends on
the ICU (International Components for Unicode) library, which was promptly
ported and can be found in the libports repository. Second, qtwebkit
apparently dropped the support of the 'QThread' API in favor of
POSIX-thread support only. For this reason, we had to extend the coverage
of Genode's pthread library to fulfill the needs of qtwebkit.
Once built, we entered the territory of debugging problems at runtime.
* We hit a memory-corruption problem caused by an assumption of 'QArrayData'
with regard to the alignment of memory allocated via malloc. As a
work-around, we weakened the assumptions to 4-byte alignment
(qt5_qarraydata.patch).
* Page faults in QWidgetAnimator caused by
use-after-free problems. Those could be alleviated by adding pointer
checks (qt5_qwidgetanimator.patch).
* Page faults caused by the slot function 'QWidgetWindow::updateObjectName()'
with a 'this' pointer of an incompatible type 'QDesktopWidget*'.
As a workaround, we avoid this condition by delegating the
'QWidgetWindow::event()' that happened to trigger the slot method to
'QWindow' (base class of 'QWidgetWindow') rather than to a
'QDesktopWidget' member (qt5_qwidgetwindow.patch).
* We observed that Arora presented web sites incomplete, or including HTTP
headers. During the evaluation of HTTP data, a signal was sent to another
thread, which activated a "user provided download buffer" for optimization
purposes. On Linux, the receiving thread was immediately scheduled and
everything went fine. However, on some kernels used by Genode, scheduling
is different, so that the original thread continued to execute a bit longer,
ultimately triggering a race condition. As a workaround, we disabled the
"user provided download buffer" optimization.
* Page faults in the JavaScript engine of Webkit. The JavaScript
'RegExp.exec()' function returned invalid string objects. We worked around
this issue by deactivating the JIT compiler for the processing of
regular expressions (ENABLE_YARR_JIT).
The current state of the Qt5 port is fairly complete. It covers the core, gui,
jscore, network, script, scriptclassic, sql, ui, webcore, webkit, widgets,
wtf, and xml modules. That said, there are a few known limitations and
differences compared to Qt4. First, the use of one liquid_fb instance per
window consumes more memory compared to the use of QWS in Qt4. Furthermore,
external window movements are not recognized by our QPA implementation yet.
This can cause popup menus to appear at unexpected positions. Key repeat is
not yet handled. The 'QNitpickerViewWidget' is not yet adapted to Qt5. For this
reason, qt_avplay is not working yet.
Test drive
==========
Unlike Qt4, which was hosted in the dedicated 'qt4' repository, Qt5 is
integrated in the libports repository. It can be downloaded and integrated
into the Genode build system by issuing 'make prepare' from within the
libports repository. The Qt5 versions of the known Qt examples are located at
libports/src/app/qt5. Ready-to-use run scripts for those examples are available
at libports/run.
Migration away from Qt4 to Qt5
==============================
The support for Qt4 for Genode has been declared as deprecated. By default,
it's use is inhibited to avoid name aliasing problems between both versions.
Any attempt to build a qt4-based target will result in a message:
!Skip target app/qt_launchpad because it requires qt4_deprecated
To re-enable the use of Qt4, the SPEC value qt4_deprecated must be defined
manually for the build directory:
!echo "SPECS += qt4_deprecated" >> etc/specs.conf
We will keep the qt4 repository in the source tree during the current
release cycle. It will be removed with version 13.11.
Light-weight event tracing
##########################
With Genode application scenarios getting increasingly sophisticated,
the need for thorough performance analysis has come into spotlight.
Such scenarios entail the interaction of many components.
For example, with our recent work on optimizing network performance, we
have to consider several possible attack points:
* Device driver: Is the device operating in the proper mode? Are there
CPU-intensive operations such as allocations within the critical path?
* Interface to the device driver: How frequent are context switches between
client and device driver? Is the interface designed appropriately for
the access patterns?
* TCP/IP stack: How does the data flow from the raw packet level to the
socket level? How dominant are synchronization costs between the involved
threads? Are there costly in-band operations performed, e.g., dynamic
memory allocations per packet?
* C runtime: How does integration of the TCP/IP stack with the C runtime
work, for example how does the socket API interfere with timeout
handling during 'select' calls?
* Networking application
* Timer server: How often is the timer consulted by the involved components?
What is the granularity of timeouts and thereby the associated costs for
handling them?
* Interaction with core: What is the profile of the component's interaction
with core's low-level services?
This example is just an illustration. Most real-world performance-critical
scenarios have a similar or even larger scope. With our traditional
tools, it is hardly possible to gather a holistic view of the scenario. Hence,
finding performance bottlenecks tends to be a series of hit-and-miss
experiments, which is a tiresome and costly endeavour.
To overcome this situation, we need the ability to gather traces of component
interactions. Therefore, we started investigating the design of a tracing
facility for Genode one year ago while posing the following requirements:
* Negligible impact on the performance, no side effects:
For example, performing a system call per traced event
is out of question because this would severely influence the flow of
control (as the system call may trigger the kernel to take a scheduling
decision) and the execution time of the traced code, not to speak of
the TLB and cache footprint.
* Kernel independence: We want to use the same tracing facility across
all supported base platforms.
* Accountability of resources: It must be clearly defined where the
resources for trace buffers come from. Ideally, the tracing tool should be
able to dimension the buffers according to its needs and, in turn, pay for
the buffers.
* Suitable level of abstraction: Only if the trace contains information at
the right level of abstraction, it can be interpreted for large scenarios.
A counter example is the in-kernel trace buffer of the Fiasco.OC kernel,
which logs kernel-internal object names and a few message words when tracing
IPC messages, but makes it almost impossible to map this low-level
information to the abstraction of the control flow of RPC calls. In
contrast, we'd like to capture the names of invoked RPC calls (which is an
abstraction level the kernel is not aware of). This requirement implies the
need to have useful trace points generated automatically. Ideally those
trace points should cover all interactions of a component with the outside
world.
* (Re-)definition of tracing policies at runtime: The
question of which information to gather when a trace point is passed
should not be solely defined at compile time. Instead of changing static
instrumentations in the code, we'd prefer to have a way to configure
the level of detail and possible conditions for capturing events at runtime,
similar to dtrace. This way, a series of different hypotheses could be
tested by just changing the tracing policy instead of re-building and
rebooting the entire scenario.
* Straight-forward implementation: We found that most existing tracing
solutions are complicated. For example, dtrace comes with a virtual
machine for the sandboxed interpretation of policy code. Another typical
source of complexity is the synchronization of trace-buffer accesses.
Because for Genode, low TCB complexity is of utmost importance, the
simplicity of the implementation is the prerequisite to make it an
integral part of the base system.
* Support for both online and offline analysis of traces.
We are happy to report to have come up with a design that meets all those
requirements thanks to the architecture of Genode. In the following, we
present the key aspects of the design.
The tracing facility comes in the form of a new TRACE service implemented
in core. Using this service, a TRACE client can gather information about
available tracing subjects (existing or no-longer existing threads),
define trace buffers and policies and assign those to tracing subjects,
obtain access to trace-buffer contents, and control the tracing state
of tracing subjects. When a new thread is created via a CPU session, the
thread gets registered at a global registry of potential tracing sources. Each
TRACE service manages a session-local registry of so-called trace subjects.
When requested by the TRACE client, it queries new tracing sources from the
source registry and obtains references to the corresponding threads. This way,
the TRACE session becomes able to control the thread's tracing state.
To keep the tracing overhead as low as possible, we assign a separate trace
buffer to each individually traced thread. The trace buffer is a shared memory
block mapped to the virtual address space of the thread's process. Capturing
an event comes down to a write operation into the thread-local buffer. Because
of the use of shared memory for the trace buffer, no system call is needed and
because the buffer is local to the traced thread, there is no need for
synchronizing the access to the buffer. When no tracing is active, a thread
has no trace buffer. The buffer gets installed only when tracing is started.
The buffer is not installed magically from the outside of the traced process
but from the traced thread itself when passing a trace point. To detect
whether to install a new trace buffer, there exists a so-called trace-control
dataspace shared between the traced process and its CPU service. This
dataspace contains control bits for each thread created via the CPU session.
The control bits are evaluated each time a trace point is passed by the
thread. When the thread detects a change of the tracing state, it actively
requests the new trace buffer from the CPU session and installs it into its
address space. The same technique is used for loading the code for tracing
policies into the traced process. The traced thread actively checks for
policy-version updates by evaluating the trace-control bits. If an update is
detected, the new policy code is requested from the CPU session. The policy
code comes in the form of position-independent code, which gets mapped into
the traced thread's address space by the traced thread itself. Once mapped,
a trace point will call the policy code. When called, the policy
module code returns the data to be captured into the trace buffer. The
relationship between the trace monitor (the client of TRACE service), core's
TRACE service, core's CPU service, and the traced process is depicted in
Figure [trace_control].
[image trace_control]
There is one trace-control dataspace per CPU session, which gets accounted
to the CPU session's quota. The resources needed for the
trace-buffer dataspaces and the policy dataspaces are paid-for by the
TRACE client. On session creation, the TRACE client can specify the amount
of its own RAM quota to be donated to the TRACE service in core. This
enables the TRACE client to define trace buffers and policies of arbitrary
sizes, limited only by its own RAM quota.
In Genode, the interaction of a process with its outside world is
characterized by its use of inter-process communication, namely synchronous
RPC, signals, and access to shared memory. For the former two types of
inter-process communication, Genode generates trace points automatically. RPC
clients generate trace points when an RPC call is issued and when a call
returned. RPC servers generate trace points in the RPC dispatcher, capturing
incoming RPC requests as well as RPC replies. Thanks to Genode's RPC
framework, we are able to capture the names of the RPC functions in the RPC
trace points. This information is obtained from the declarations of the RPC
interfaces. For signals, trace points are generated for submitting and
receiving signals. Those trace points form a useful base line for gathering
tracing data. In addition, manual trace points can be inserted into the code.
State of the implementation
===========================
The implementation of Genode's tracing facility is surprisingly low complex.
The addition to the base system (core as well as the base library) are
merely 1500 lines of code. The mechanism works across all base platforms.
Because the TRACE client provides the policy code and trace buffer to the
traced thread, the TRACE client imposes ultimate control over the traced
thread. In contrast to dtrace, which sandboxes the trace policy, we express
the policy module in the form of code executed in the context of the traced
thread. However, in contrast to dtrace, such code is never loaded into a large
monolithic kernel, but solely into the individually traced processes. So the
risk of a misbehaving policy is constrained to the traced process.
In the current form, the TRACE service of core should be considered as a
privileged service because the trace-subject namespace of each session
contains all threads of the system. Therefore, TRACE sessions should be routed
only for trusted processes. In the future, we plan to constrain the
namespaces for tracing subjects per TRACE session.
The TRACE session interface is located at base/include/trace_session/.
A simple example for using the service is available at os/src/test/trace/
and is accompanied with the run script os/run/trace.run. The test
demonstrates the TRACE session interface by gathering a trace of a thread
running locally in its address space.
Enhanced multi-processor support
################################
Multi-processor (MP) support is one of those features that most users take for
granted. MP systems are so ubiquitous, even on mobile platforms, that a
limitation to utilizing a single CPU only is almost a fallacy.
That said, MP support in operating systems is hard to get right. For this
reason, we successively deferred the topic on the agenda of Genode's
road map.
For some base platforms such as the Linux or Codezero kernels, Genode
always used to support SMP because the kernel would manage the affinity
of threads to CPU cores transparently to the user-level process. So on
these kernels, there was no need to add special support into the framework.
However, on most microkernels, the situation is vastly different. The
developers of such kernels try hard to avoid complexity in the kernel and
rightfully argue that in-kernel affinity management would contribute to kernel
complexity. Another argument is that, in contrast to monolithic kernels that
have a global view on the system and an "understanding" of the concerns of the
user processes, a microkernel is pretty clueless when it comes to the roles
and behaviours of individual user-level threads. Not knowing whether a thread
works as a device driver, an interactive program, or a batch process, the
microkernel is not in the position to form a reasonably useful model of the
world, onto which it could intelligently apply scheduling and affinity
heuristics. In fact, from the perspective of a microkernel, each thread does
nothing else than sending and receiving messages, and causing page faults.
For these reasons, microkernel developers tend to provide the bootstrapping
procedure for the physical CPUs and a basic mechanism to assign
threads to CPUs but push the rest of the problem to the user space, i.e.,
Genode. The most straight-forward way would make all physical CPUs visible
to all processes and require the user or system integrator to assign
physical CPUs when a thread is created. However, on the recursively
structured Genode system, we virtualize resources at each level, which
calls for a different approach. Section [Management of CPU affinities]
explains our solution.
When it comes to inter-process communication on MP systems, there is a
certain diversity among the kernel designs. Some kernels allow the user land
to use synchronous IPC between arbitrary threads, regardless of whether both
communication partners reside on the same CPU or on two different CPUs. This
convenient model is provided by Fiasco.OC. However, other kernels do not offer
any synchronous IPC mechanism across CPU cores at all, NOVA being a poster
child of this school of thought. If a user land is specifically designed for
a particular kernel, those peculiarities can be just delegated to the
application developers. For example, the NOVA user land called NUL is designed
such that a recipient of IPC messages spawns a dedicated thread on each
physical CPU. In contrast, Genode is meant to provide a unified API that
works well across various different kernels. To go forward, we had four
options:
# Not fully supporting the entity of API semantics across all base platforms.
For example, we could stick with the RPC API for synchronous communication
between threads. Programs just would happen to fail on some base platforms
when the called server resides on a different CPU. This would effectively
push the problem to the system integrator. The downside would be the
sacrifice of Genode's nice feature that a program developed
on one kernel usually works well on other kernels without any changes.
# Impose the semantics provided by the most restrictive kernel onto all
users of the Genode API. Whereas this approach would facilitate that
programs behave consistently across all base platforms, the restrictions
would be artificially imposed onto all Genode users, in particular the
users of kernels with less restrictions. Of course, we don't change
the Genode API lightheartedly, which attributes to our hesitance to go
into this direction.
# Hiding kernel restrictions behind the Genode API. This approach could come
in many different shapes. For example, Genode could transparently spawn a
thread on each CPU when a single RPC entrypoint gets created, following
the model of NUL. Or Genode could emulate synchronous IPC using the core
process as a proxy.
# Adapting the kernel to the requirements of Genode. That is, persuading
kernel developers to implement the features we find convenient, i.e., adding
a cross-CPU IPC feature to NOVA. History shows that our track record in
doing that is not stellar.
Because each of those options is like opening a different can of worms, we
used to defer the solution of the problem. Fortunately, however, we finally
kicked off a series of practical experiments, which led to a fairly elegant
solution, which is detailed in Section
[Adding multi-processor support to Genode on NOVA].
Management of CPU affinities
============================
In line with our experience of supporting
[http://www.genode.org/documentation/release-notes/10.02#Real-time_priorities - real-time priorities]
in version 10.02, we were seeking a way to express CPU affinities such that
Genode's recursive nature gets preserved and facilitated. Dealing with
physical CPU numbers would contradict with this mission. Our solution
is based on the observation that most MP systems have topologies that can
be represented on a two-dimensional coordinate system. CPU nodes close
to each other are expected to have closer relationship than distant nodes.
In a large-scale MP system, it is natural to assign clusters of closely
related nodes to a given workload. Genode's architecture is based on the
idea to recursively virtualize resources and thereby lends itself to the
idea to apply this successive virtualization to the problem of clustering
CPU nodes.
In our solution, each process has a process-local view on a so-called affinity
space, which is a two-dimensional coordinate space. If the process creates a
new subsystem, it can assign a portion of its own affinity space to the new
subsystem by imposing a rectangular affinity location to the subsystem's CPU
session. Figure [affinity_space] illustrates the idea.
[image affinity_space]
Following from the expression of affinities as a rectangular location within a
process-local affinity space, the assignment of subsystems to CPU nodes
consists of two parts, the definition of the affinity space dimensions as used
for the process and the association of sub systems with affinity locations
(relative to the affinity space). For the init process, the affinity space is
configured as a sub node of the config node. For example, the following
declaration describes an affinity space of 4x2:
! <config>
! ...
! <affinity-space width="4" height="2" />
! ...
! </config>
Subsystems can be constrained to parts of the affinity space using the
'<affinity>' sub node of a '<start>' entry:
! <config>
! ...
! <start name="loader">
! <affinity xpos="0" ypos="1" width="2" height="1" />
! ...
! </start>
! ...
! </config>
As illustrated by this example, the numbers used in the declarations for this
instance of the init process are not directly related to physical CPUs. If
the machine has just two cores, init's affinity space would be mapped
to the range [0,1] of physical CPUs. However, in a machine
with 16x16 CPUs, the loader would obtain 8x8 CPUs with the upper-left
CPU at position (4,0). Once a CPU session got created, the CPU client can
request the physical affinity space that was assigned to the CPU session
via the 'Cpu_session::affinity()' function. Threads of this CPU session
can be assigned to those physical CPUs via the 'Cpu_session::affinity()'
function, specifying a location relative to the CPU-session's affinity space.
Adding multi-processor support to Genode on NOVA
================================================
The NOVA kernel has been supporting MP systems for a long time. However
Genode did not leverage this capability until now. The main reason was that
the kernel does not provide - intentionally by the kernel developer - the
possibility to perform synchronous IPC between threads residing on different
CPUs.
To cope with this situation, Genode servers and clients would need to make
sure to have at least one thread on a common CPU in order to communicate.
Additionally, shared memory and semaphores could be used to communicate across
CPU cores. Both options would require rather fundamental changes to the Genode
base framework and the API. An exploration of this direction should in any
case be pursued in evolutionary steps rather than as one big change, also
taking into account that other kernels do not impose such hard requirements on
inter-CPU communication. To tackle the challenge, we conducted a series of
experiments to add some kind of cross-CPU IPC support to Genode/NOVA.
As a general implication of the missing inter-CPU IPC, messages between
communication partners that use disjoint CPUs must take an indirection through
a proxy process that has threads running on both CPUs involved. The sender
would send the message to a proxy thread on its local CPU, the proxy process
would transfer the message locally to the CPU of the receiver by using
process-local communication, and the proxy thread on the receiving CPU would
deliver the message to the actual destination. We came up with three options
to implement this idea prototypically:
# Core plays the role of the proxy because it naturally has access to all
CPUs and emulates cross-CPU IPC using the thread abstractions of the
Genode API.
# Core plays the role of the proxy but uses NOVA system calls directly
rather than Genode's thread abstraction.
# The NOVA kernel acts as the proxy and emulates cross-CPU IPC directly
in the kernel.
After having implemented the first prototypes, we reached the following
conclusions.
For options 1 and 2 where core provides this service: If a client can not
issue a local CPU IPC, it asks core - actually the pager of the client
thread - to perform the IPC request. Core then spawns or reuses a proxy thread
on the target CPU and performs the actual IPC on behalf of the client. Option
1 and 2 only differ in respect to code size and the question to whom to
account the required resources - since a proxy thread needs a stack and some
capability selectors.
As one big issue for option 1 and 2, we found that in order to delegate
capabilities during the cross-CPU IPC, core has to receive capability mappings
to delegate them to the target thread. However, core has no means to know
whether the capabilities must be maintained in core or not. If a capability is
already present in the target process, the kernel would just translate the
capability to the target's capability name space. So core wouldn't need to
keep it. In the other case where the target receives a prior unknown
capability, the kernel creates a new mapping. Because the mapping gets
established by the proxy in core, core must not free the capability.
Otherwise, the mapping would disappear in the target process. This means that
the use of core as a proxy ultimately leads to leaking kernel resources
because core needs to keep all transferred capabilities, just for the case a
new mapping got established.
For option 3, the same general functionality as for option 1 and 2 is
implemented in the kernel instead of core. If a local CPU IPC call
fails because of a BAD_CPU kernel error code, the cross-CPU IPC extension
will be used. The kernel extension creates - similar as to option 1 and 2 - a
semaphore (SM), a thread (EC), and a scheduling context (SC) on the remote
CPU and lets it run on behalf of the caller thread. The caller thread
gets suspended by blocking on the created semaphore until the remote EC has
finished the IPC. The remote proxy EC reuses the UTCB of the suspended caller
thread as is and issues the IPC call. When the proxy EC returns, it wakes up
the caller via the semaphore. Finally, the proxy EC and SC de-schedule
themselves and the resources get to be destroyed later on by the kernel's RCU
mechanism. Finally, when the caller thread got woken up, it takes care to
initiate the deconstruction of the semaphore.
The main advantage of option 3 compared to options 1 and 2 is that we don't
have to keep and track the capability delegations during a cross-CPU IPC.
Furthermore, we do not have potentially up to two additional address space
switches per cross-CPU IPC (from client to core and core to the server).
Additionally, the UTCB of the caller is reused by the proxy EC and does not
need to be allocated separately as for option 1 and 2.
For these reasons, we decided to go for the third option. From Genode's API
point of view, the use of cross-CPU IPC is completely transparent. Combined
with the affinity management described in the previous section, Genode/NOVA
just works on MP systems.
As a simple example for using Genode on MP systems, there is a ready-to-use
run script available at base/run/affinity.run.
Base framework
##############
Affinity propagation in parent, root, and RPC entrypoint interfaces
===================================================================
To support the propagation of CPU affinities with session requests, the
parent and root interfaces had to be changed. The 'Parent::Session'
and 'Root::Session' take the affinity of the session as a new argument.
The affinity argument contains both the dimensions of the affinity space
used by the session and the session's designated affinity location within
the space. The corresponding type definitions can be found at
base/affinity.h.
Normally, the 'Parent::Session' function is not used directly but indirectly
through the construction of a so-called connection object, which represents an
open session. For each session type there is a corresponding connection type,
which takes care of assembling the session-argument string by using the
'Connection::session()' convenience function. To maintain API compatibility,
we kept the signature of the existing 'Connection::session()' function using a
default affinity and added a new overload that takes the affinity as
additional argument. Currently, this overload is used in
cpu_session/connection.h.
For expressing the affinities of RPC entrypoints to CPUs within the affinity
space of the server process, the 'Rpc_entrypoint' takes the desired affinity
location of the entrypoint as additional argument. For upholding API
compatibility, the affinity argument is optional.
CPU session interface
=====================
The CPU session interface underwent changes to accommodate the new event
tracing infrastructure and the CPU affinity management.
Originally the 'Cpu_session::num_cpus()' function could be used to
determine the number of CPUs available to the session. This function
has been replaced by the new 'affinity_space' function, which returns the
bounds of the CPU session's physical affinity space. In the simplest case of
an SMP machine, the affinity space is one-dimensional where the width
corresponds to the number of CPUs. The 'affinity' function, which is used to
bind a thread to a specified CPU, has been changed to take an affinity
location as argument. This way, the caller can principally express the
affiliation of the thread with multiple CPUs to guide load-balancing in a CPU
service.
New TRACE session interface
===========================
The new event tracing mechanism as described in Section
[Light-weight event tracing] is exposed to Genode processes in the form
of the TRACE service provided by core. The new session interface
is located under base/include/trace_session/. In addition to the new session
interface, the CPU session interface has been extended with functions for
obtaining the trace-control dataspace for the session as well as the trace
buffer and trace policy for a given thread.
Low-level OS infrastructure
###########################
Event-driven operation of NIC bridge
====================================
The NIC bridge component multiplexes one physical network device among
multiple clients. It enables us to multiplex networking on the network-packet
level rather than the socket level and thereby take TCP/IP out of the
critical software stack for isolating network applications. As it
represents an indirection in the flow of all networking packets, its
performance is important.
The original version of NIC bridge was heavily multi-threaded. In addition to
the main thread, a timer thread, and a thread for interacting with the NIC
driver, it employed one dedicated thread per client. By merging those flows of
control into a single thread, we were able to significantly reduce the number
of context switches and improve data locality. These changes reduced the
impact of the NIC bridge on the packet throughput from 25% to 10%.
Improved POSIX thread support
=============================
To accommodate qtwebkit, we had to extend Genode's pthread library with
working implementations of condition variables, mutexes, and thread-local
storage. The implemented functions are attr_init, attr_destroy, attr_getstack,
attr_get_np, equal, mutex_attr, mutexattr_init, mutexattr_destroy,
mutexattr_settype, mutex_init, mutex_destroy, mutex_lock, mutex_unlock,
cond_init, cond_timedwait, cond_wait, cond_signal, cond_broadcast, key_create,
setspecific, and getspecific.
Device drivers
##############
SATA 3.0 on Exynos 5
====================
The previous release featured the initial version of our SATA 3.0 driver for
the Exynos 5 platform. This driver located at os/src/drivers/ahci/exynos5 has
reached a fully functional state by now. It supports UDMA-133 with up to
6 GBit/s.
For driver development, we set the goal to reach a performance equal to
the Linux kernel. To achieve that goal, we had to make sure to
operate the controller and the disks in the same ways as Linux does.
For this reason, we modeled our driver closely after the behaviour of the
Linux driver. That is, we gathered traces of I/O transactions to determine the
initialization steps and the request patterns that Linux performs to access
the device, and used those behavioral traces as a guide for our
implementation. Through step-by-step analysis of the traces, we not only
succeeded to operate the device in the proper modes, but we also found
opportunities for further optimization, in particular regarding the error
recovery implementation.
This approach turned out to be successful. We measured that our driver
generally operates as fast (and in some cases even significantly faster)
than the Linux driver on solid-state disks as well as on hard disks.
Dynamic CPU frequency scaling for Exynos 5
==========================================
As the Samsung Exynos-5 SoC is primarily targeted at mobile platforms,
power management is an inherent concern. Until now, Genode did not pay
much attention to power management though. For example, we completely
left out the topic from the scope of the OMAP4 support. With the current
release, we took the first steps towards proper power management on ARM-based
platforms in general, and the Exynos-5-based Arndale platform in particular.
First, we introduced a general interface to regulate clocks and voltages.
Priorly, each driver did its own part of configuring clock and power control
registers. The more device drivers were developed, the higher were the chances
that they interfere when accessing those clock, or power units.
The newly introduced "Regulator" interface provides the possibility to enable
or disable, and to set or get the level of a regulator. A regulator might be
a clock for a specific device (such as a CPU) or a voltage regulator.
For the Arndale board, an exemplary implementation of the regulator interface
exists in the form of the platform driver. It can be found at
os/src/drivers/platform/arndale. Currently, the driver implements
clock regulators for the CPU, the USB 2.0 and USB 3.0 host controller, the
eMMC controller, and the SATA controller. Moreover, it provides power
regulators for SATA, USB 2.0, and USB 3.0 host controllers. The selection of
regulators is dependent on the availability of drivers for the platform.
Otherwise it wouldn't be possible to test that clock and power state doesn't
affect the device.
Apart from providing regulators needed by certain device drivers, we
implemented a clock regulator for the CPU that allows changing the CPU
frequency dynamically and thereby giving the opportunity to scale down
voltage and power consumption. The possible values range from 200 MHz to
1.7 GHz whereby the last value isn't recommended and might provoke system
crashes due to overheating. When using Genode's platform driver for Arndale
it sets CPU clock speed to 1.6 GHz by default. When reducing
the clock speed to the lowest level, we observed a power consumption
reduction of approximately 3 Watt. Besides reducing dynamic power consumption
by regulating the CPU clock frequency, we also explored the gating of the clock
management and power management to further reduce power consumption.
With the CPU frequency scaling in place, we started to close all clock gates
not currently in use. When the platform driver for the Arndale board gets
initialized, it closes everything. If a device driver enables its clock
regulator, all necessary clock gates for the device's clock are opened. This
action saves about 0.7 Watt. The initial closing of all unnecessary power
gates was much more effective. Again, everything not essential for the working
of the kernel is disabled on startup. When a driver enables its power
regulator, all necessary power gates for the device are opened. Closing all
power gates saves about 2.6 Watt.
If we consider all measures taken to save power, we were able to reduce power
consumption to about 59% without performance degradation. When measuring power
consumption after boot up, setting the CPU clock to 1.6 GHz, and fully load
both CPU cores without the described changes, we measured about 8 Watt. With
the described power saving provisions enabled, we measured about 4.7 Watt.
When further reducing the CPU clock frequency to 200 MHz, only 1.7 Watt were
measured.
VESA driver moved to libports
=============================
The VESA framebuffer driver executes the initialization code located
in VESA BIOS of the graphics card. As the BIOS code is for real mode,
the driver uses the x86emu library from X11 as emulation environment.
We updated x86emu to version 1.20 and moved the driver from the 'os'
repository to the 'libports' repository as the library is third-party
code. Therefore, if you want to use the driver, the 'libports'
repository has to be prepared
('make -C <genode-dir>/libports prepare PKG=x86emu') and enabled in
your 'etc/build.conf'.
Runtime environments
####################
Seoul (aka Vancouver) VMM on NOVA
=================================
Since we repeatedly received requests for using the Seoul respectively
Vancouver VMM on NOVA, we improved the support for this virtualization
solution on Genode. Seoul now supports booting from raw hard disk images
provided via Genode's block session interface. Whether this image is actually
a file located in memory, or it is coming directly from the hard disk, or just
from a partition of the hard disk using Genode's part_blk service, is
completely transparent thanks to Genode's architecture.
Additionally, we split up the one large Vancouver run script into several
smaller Seoul run scripts for easier usage - e.g. one for disk, one for
network testing, one for automated testing, and one we call "fancy". The
latter resembles the former vancouver.run script using Genode's GUI to let the
user start VMs interactively. The run scripts prefixed with 'seoul-' can be
found at ports/run. For the fancy and network scripts, ready-to-use VM images
are provided. Those images are downloaded automatically when executing the run
script for the first time.
L4Linux on Fiasco.OC
====================
L4Linux has been updated from version 3.5.0 to Linux kernel version 3.9.0 thus
providing support for contemporary user lands running on top of L4Linux on both
x86 (32bit) and ARM platforms.
Noux runtime for Unix software
==============================
Noux is our way to use the GNU software stack natively on Genode. To improve
its performance, we revisited the address-space management of the runtime to
avoid redundant revocations of memory mappings when Noux processes are cleaned
up.
Furthermore, we complemented the support for the Genode tool chain to
cover GNU sed and GNU grep as well. Both packages are available at the ports
repository.
Platforms
#########
Fiasco.OC updated to revision r56
=================================
Fiasco.OC and the required L4RE parts have been updated to the current SVN
revision (r56). For us, the major new feature is the support of Exynos SOCs in
the mainline version of Fiasco.OC (www.tudos.org). Therefore Genode's
implementation of the Exynos5250 platform could be abandoned leading to less
maintenance overhead of Genode on Fiasco.OC.
Furthermore, Genode's multi-processor support for this kernel has been
improved so that Fiasco.OC users benefit from the additions described in
Section [Enhanced multi-processor support].
NOVA updated
============
In the process of our work on the multi-processor support on NOVA, we updated
the kernel to the current upstream version. Additionally, our customized branch
(called r3) comes with the added cross-CPU IPC system call and improvements
regarding the release of kernel resources.
Integrity checks for downloaded 3rd-party software
##################################################
Even though Genode supports a large variety of 3rd-party software, its
source-code repository contains hardly any 3rd-party source code. Whenever
3rd-party source code is needed, Genode provides automated tools for
downloading the code and integrating it with the Genode environment. As of
now, there exists support for circa 70 software packages, including the
tool chain, various kernels, libraries, drivers, and a few applications. Of
those packages, the code for 13 packages comes directly from their respective
Git repositories. The remaining 57 packages are downloaded in the form of tar
archives from public servers via HTTP or FTP. Whereas we are confident with
the integrity of the code that comes from Git repositories, we are less so
about the archives downloaded from HTTP or FTP servers.
Fortunately, most Open-Source projects provide signature files that allow
the user to verify the origin of the archive. For example, archives of
GNU software are signed with the private key of the GNU project. So the
integrity of the archive can be tested with the corresponding public key.
We used to ignore the signature files for many years but
this has changed now. If there is a signature file available for a package,
the package gets verified right after downloading. If only a hash-sum file
is provided, we check it against a known-good hash sum.
The solution required three steps, the creation of tools for validating
signatures and hashes, the integration of those tools into Genode's
infrastructure for downloading the 3rd-party code, and the definition of
verification rules for the individual packages.
First, new tools for downloading and validating hash sums and signatures were
added in the form of the shell scripts download_hashver (verify hash sum) and
download_sigver (verify signature) found at the tool/ directory. Under the
hood, download_sigver uses GNU GPG, and download_hashver uses the tools
md5sum, sha1sum, and sha256sum provided by coreutils.
Second, hooks for invoking the verification tools were added to the
tool-chain build script as well as the ports and the libports repositories.
The third and the most elaborative step, was going through all the packages,
looking for publicly available signature files, and adding corresponding
package rules. As of now, this manual process has been carried out for 30
packages, thereby covering the half of the archives.
Thanks to Stephan Mueller for pushing us into the right direction, kicking off
the work on this valuable feature, and for the manual labour of revisiting all
the 3rd-party packages!

791
doc/release_notes-14-02.txt
View File

@@ -1,791 +0,0 @@
===============================================
Release notes for the Genode OS Framework 14.02
===============================================
Genode Labs
During the release cycle of version 14.02, our development has been focused on
storage and virtualization. It goes without saying that proper support for
block-device access and file systems is fundamental for the use of
Genode as general-purpose OS. Virtualization is relevant as well because
it bridges the gap between the functionality we need and the features
natively available on Genode today.
Our work on the storage topic involved changes of the block-driver APIs to an
asynchronous mode of operation, overhauling most of the existing block-level
components, as well as the creation of new block services, most importantly a
block cache. At file-system level, we continued our line of work on FUSE-based
file systems, adding support for NTFS-3g. A new highlight, however, is a new
file-system service that makes the file systems of the NetBSD kernel available
to Genode. This is made possible by using rump kernels as described in Section
[NetBSD file systems using rump kernels].
Virtualization on Genode has a long history, starting with the original
support of OKLinux on the OKL4 kernel (OKLinux is no longer supported), over
the support of L4Linux on top of the Fiasco.OC kernel, to the support of the
Vancouver VMM on top of NOVA. However, whereas each of those variants has
different technical merits, all of them were developed in the context of
university research projects and were never exposed to real-world scenarios.
We were longing for a solution that meets the general expectations from a
virtualization product, namely the support for a wide range of guest OSes,
guest-host integration features, ease of use, and an active development.
VirtualBox is one of the most popular commodity virtualization products as of
today. With the current release, we are happy to announce the availability of
VirtualBox on top of Genode/NOVA. Section
[VirtualBox on top of the NOVA microhypervisor] gives insights into the
background of this development, the technical challenges we had to overcome,
and the current state of the implementation.
In addition to addressing storage and virtualization, the current release
comes with a new pseudo file system called trace_fs that allows the
interactive use of Genode's tracing facilities via Unix commands,
a profound unification of the various graphics back ends used throughout
the framework, a new facility for propagating status reports, and
improvements of the Noux runtime for executing Unix software on Genode.
VirtualBox on top of the NOVA microhypervisor
#############################################
Virtualization is an important topic for Genode for two distinct reasons.
It is repeatedly requested by users of the framework who consider
Genode as a microkernel-based hosting platform for virtual machines,
and it provides a smooth migration path from using Linux-based systems
towards using Genode as day-to-day OS.
Why do people consider Genode as a hosting platform for virtual machines
if there is an abundance of mature virtualization solutions on the market?
What all existing popular solutions have in common is the staggering complexity
of their respective trusted-computing base (TCB). The user of a virtual
machine on a commodity hosting platform has to trust millions of lines of
code. For example, with Xen, the TCB comprises the hypervisor and the Linux
system running as DOM0. For security-sensitive application areas, it is
almost painful to trust such a complex foundation. In contrast, the TCB of a
hosting platform based on Genode/NOVA is two orders of magnitude less complex.
Lowering the complexity reduces the likelihood for vulnerabilities and thereby
mitigates the attack surface of the system. It also enables the assessment of
security properties by thorough evaluation or even formal verification. In the
light of the large-scale privacy issues of today, the desire for systems that
are resilient against malware and zero-day exploits has never been higher.
Microkernel-based operating systems promise a solution. Virtualization enables
compatibility to existing software. Combining both seems natural. This is what
Genode/NOVA stands for.
From the perspective of us Genode developers who are in the process of
migrating from Linux-based OSes to Genode as day-to-day OS, we consider
virtualization as a stop-gap solution for all those applications that
do not exist natively on Genode, yet. Virtualization makes our transition
an evolutionary process.
Until now, NOVA was typically accompanied with a co-developed virtual machine
monitor called Seoul (formerly called Vancouver), which is executed as a
regular user-level process on top of NOVA. In contrast to conventional wisdom
about the performance of microkernel-based systems, the Seoul VMM on top of
NOVA is extremely fast, actually faster then most (if not all) commonly used
virtualization solutions. However, originating from a research project, Seoul
is quite challenging to use and not as mature as commodity VMMs that were
developed as real-world products. For example, there is a good chance that an
attempt to boot an arbitrary version of a modern Linux distribution might just
fail. In our experience, it takes a few days to investigate the issues, modify
the guest OS configuration, and tweak the VMM here and there, to run the OS
inside the Seoul VMM. That is certainly not a show stopper in appliance-like
scenarios, but it rules out Seoul as a general solution. Running Windows
OS as guest is not supported at all, which further reduces the application
areas of Seoul. With this in mind, it is unrealistic to propose the use
of Genode/NOVA as an alternative for popular VM hosting solutions.
Out of this realization, the idea was born to combine NOVA's virtualization
interface with a time-tested and fully-featured commodity VMM. Out of the
available Open-Source virtualization solutions, we decided to take a closer
look at VirtualBox, which attracted us for several reasons: First, it is
portable, supporting various host OSes such as Solaris, Windows OS, Linux,
and Mac OS X. Second, it has all the guest-integration features we could
wish for. There are extensive so-called guest additions for popular guest
OSes that vastly improve the guest-OS performance and allow a tight
integration with the host OS using shared folders or a shared clipboard.
Third, it comes with sophisticated device models that support all
important popular guest OSes. And finally, it is actively developed and
commercially supported.
However, moving VirtualBox over to NOVA presented us with a number of
problems. As a precondition, we needed to gain a profound understanding
of the VirtualBox architecture and the code base. To illustrate the challenge,
the source-code distribution of VirtualBox comprises 2.8 million lines of
code. This code contains build tools, the VMM, management tools, several
3rd-party libraries, middleware, the guest additions, and tests. The pieces
that are relevant for the actual VMM amount to 700 thousand lines. By
reviewing the architecture, we found that the part of VirtualBox that
implements the hypervisor functionality (the world switch) runs in the
kernel of the host OS (it is loaded on demand by the user-level VM process
through the _/dev/vboxdrv_ interface into the host OS kernel). It is
appropriately named VMMR0. Once installed into the host OS kernel, it
takes over the control over the machine. To put it blatantly simple, it runs
"underneath" the host OS. The VMMR0 code is kernel agnostic, which explains
the good portability of VirtualBox across various host OSes. Porting
VirtualBox to a new host OS comes down to finding a hook for installing the
VMMR0 code into the host OS kernel and adapting the VirtualBox runtime API
to the new host OS.
In the context of microkernel-based systems, however, it becomes clear that
this classical approach of porting VirtualBox would subvert the microkernel
architecture. Not only would we need to punch a hole into NOVA for loading
additional kernel code, but also the VMMR0 code would inflate the amount of
code executed in privileged mode by more than factor 20. Both implications
are gross violations of the microkernel principle. Consequently, we needed to
find a different way to marry NOVA with VirtualBox.
Our solution was the creation of a drop-in replacement of the VMMR0 code that
runs solely at user level and interacts with NOVA's virtualization
interface. Our VMMR0 emulation code is co-located with the VirtualBox
VM process. Architecturally, the resulting solution is identical to the
use of Seoul on top of NOVA. There is one VM process per virtual machine,
and each VM process is isolated from others by the NOVA kernel. In
addition to creating the VMMR0 emulation code, we needed to replace some parts
of the VirtualBox VMMR3 code with custom implementations because they
overlapped with functionality provided by NOVA's virtualization interface,
in particular the provisioning of guest-physical memory. Finally, we needed
to interface the VM process with Genode's API to let the VM process
interact with Genode's input, file-system, and framebuffer services.
The result of this undertaking is available at the _ports_ repository.
VirtualBox can be downloaded and integrated with Genode via the following
command issued from within the repository:
! make prepare PKG=virtualbox
To illustrate the integration of VirtualBox into a Genode system, there
is run script located at _ports/run/virtualbox.run_. It expects a
bootable ISO image containing a guest OS at _<build-dir>/bin/test.iso_.
The configuration of the VirtualBox process is as simple as
! <config>
! <image type="iso" file="/iso/test.iso" />
! </config>
VirtualBox will try to obtain the specified ISO file via a file-system
session. Furthermore, it will open a framebuffer session and an input session.
The memory assigned to the guest OS depends on the RAM quota assigned to the
VirtualBox process. Booting a guest OS stored in a VDI file is supported. The
image type must be changed to "vdi" accordingly.
Please note that this first version of VirtualBox is far from being complete
as it lacks many features (SMP, guest-addition support, networking), is not
optimized, and must be considered as experimental. However, we could
successfully run GNU/Linux, Android, Windows XP, Windows 7, HelenOS, Minix-3,
GNU Hurd, and of course Genode inside VirtualBox.
One point we are pretty excited about is that the porting effort to
Genode/NOVA did not require any change of Genode. From Genode's point of
view, VirtualBox is just an ordinary leaf node of the process tree, which
can happily co-exist with other processes - even if it is the Seoul VMM.
[image seoul-vbox-win7-tinycore]
In the screenshot above, VirtualBox is running besides the Seoul VMM on top of
Genode/NOVA. Seoul executes Tinycore Linux as guest OS. VirtualBox executes MS
Windows 7. Both VMMs are using hardware virtualization (VT-x) but are plain
user-level programs with no special privileges.
NetBSD file systems using rump kernels
######################################
In the previous release, we made FUSE-based file systems available to Genode
via a custom implementation of the FUSE API. Even though this step made
several popular file systems available, we found that the file systems most
important to us (such as ext) are actually not well supported by FUSE. For
example, write support on ext2 is declared as an experimental feature. In
hindsight it is clear why: FUSE is primarily being used for accessing file
systems not found in the Linux kernel. So it shines with supporting NTFS
but less so with file systems that are well supported by the Linux kernel.
Coincidentally, when we came to this realization, we stumbled upon the
wonderful work of Antti Kantee on so-called rump kernels:
:[http://wiki.netbsd.org/rumpkernel/]:
Rump kernel Wiki
The motivation behind the rump kernels was the development of
NetBSD kernel subsystems (referred to as "drivers") in the NetBSD user land.
Such subsystems like file systems, device drivers, or the TCP/IP stack are
linked against a stripped-down version of the NetBSD kernel that can be
executed in user mode and uses a fairly small "hypercall" interface to
interact with the outside world. A rump kernel contains everything needed to
execute NetBSD kernel subsystems but hardly anything else. In particular, it
does not support the execution of programs on top. From our perspective,
having crafted device-driver environments (DDEs) for Linux, iPXE, and OSS over
the years, a rump kernel sounded pretty much like a DDE for NetBSD. So we
started exploring rump kernels with the immediate goal of making time-tested
NetBSD file systems available to Genode.
To our delight, the integration of rump kernels into the Genode system went
fairly smooth. The most difficult part was the integration of the NetBSD build
infrastructure with Genode's build system. The glue between rump kernels and
Genode is less than 3,000 lines of code. This code enables us to reuse all
NetBSD file systems on Genode. A rump kernel instance that contains several
file systems such as ext2, iso9660, msdos, and ffs takes about 8 MiB of memory
when executed on Genode.
The support for rump kernels comes in the form of the dedicated _dde_rump_
repository. For downloading and integrating the required NetBSD source code,
the repository contains a Makefile providing the usual 'make prepare'
mechanism. To build the file-system server, make sure to add the _dde_rump_
repository to the 'REPOSITORIES' declaration of your _etc/build.conf_ file
within your build directory. The server then can be built via
! make server/rump_fs
There is a run script located at _dde_rump/run/rump_ext2.run_ to execute
a simple test scenario:
! make run/rump_ext2
The server can be configured as follows:
!<start name="rump_fs">
! <resource name="RAM" quantum="8M" />
! <provides><service name="File_system"/></provides>
! <config fs="ext2fs"><policy label="" root="/" writeable="yes"/></config>
!</start>
On startup, it requests a service that provides a block session. If
there is more than one block session in the system, the block session must be
routed to the right block-session server. The value of the _fs_ attribute of
the '<config>' node can be one of the following: _ext2fs_ for EXT2, _cd9660_ for
ISO-9660, or _msdos_ for FAT file-system support. _root_ defines the directory
of the file system as seen as root directory by the client. The server hands
most of its RAM quota to the rump kernel. This means the larger the quota is,
the larger the internal block caches of the rump kernel will be.
Base framework
##############
The base API has not underwent major changes apart from the addition of
a few new utilities and minor refinements. Under the hood, however, the inner
workings of the framework received much attention, including an extensive
unification of the startup code and stack management.
New 'construct_at' utility
==========================
A new utility located at 'base/include/util/construct_at.h' allows for the
manual placement of objects without the need to have a global placement new
operation nor the need for type-specific new operators.
New utility for managing volatile objects
=========================================
Throughout Genode, we maintain a programming style that largely avoids dynamic
memory allocations. For the most part, higher-level objects aggregate
lower-level objects as class members. For example, the nitpicker GUI server
is actually a compound of such aggregations (see
[https://github.com/genodelabs/genode/blob/master/os/src/server/nitpicker/main.cc#L803 - Nitpicker::Main]).
This functional programming style leads to robust programs but it poses a
problem for programs that are expected to adopt their behaviour at runtime.
For the example of nitpicker, the graphics back end of the GUI server takes
the size of the screen as constructor argument. If the screen size changes,
the once constructed graphics back end becomes inconsistent with the new
screen size. We desire a way to selectively replace an aggregated object by a
new version with updated constructor arguments. The new utilities found in
'os/include/util/volatile_object.h' solve this problem. A so-called
'Volatile_object' wraps an object of the type specified as template argument.
In contrast of a regular object, a 'Volatile_object' can be re-constructed any
number of times by calling 'construct' with the constructor arguments. It is
accompanied with a so-called 'Lazy_volatile_object', which remains
unconstructed until 'construct' is called the first time.
Changed interface of 'Signal_rpc_member'
========================================
We unified the 'Signal_rpc_member' interface to be more consistent with the
'Signal_rpc_dispatcher'. The new version takes an entrypoint as argument and
cares for dissolving itself from the entrypoint when destructed.
Filename as default label for ROM connections
=============================================
Since the first version of Genode, ROM services used to rely on a "filename"
provided as session argument. In the meanwhile, we established the use of the
session label to select routing policies as well as server-side policies.
Strictly speaking, the name of a ROM module is used as a key to a server-side
policy of ROM services. So why not to use the session label to express the
key as we do with other services? By assigning the file name as label for ROM
sessions, we may become able to remove the filename argument in the future by
just interpreting the last part of the label as filename. By keeping only the
label, we won't need to consider conditional routing (via '<if-arg>') based on
session arguments other than the label anymore, which would simplify Genode
configurations in the long run. This change is transparent at API level but
may be taken into consideration when configuring Genode systems.
New 'Genode::Deallocator' interface
===================================
By splitting the new 'Genode::Deallocator' interface from the former
'Genode::Allocator' interface, we become able to restrict the accessible
operations for code that is only supposed to release memory, but not
perform any allocations.
Closely related to the allocator interface, we introduced variants of the
'new' operator that take a reference (as opposed to a pointer) to a
'Genode::Allocator' as argument.
Unified main-stack management and startup code among all platforms
==================================================================
In contrast to the stacks of regular threads, which are located within a
dedicated virtual-address region called thread-context area, the stack of
the main thread of a Genode program used to be located within the BSS
segment. If the stack of a normal thread overflows, the program produces
an unresolvable page fault, which can be easily debugged. However,
an overflowing main stack would silently corrupt the BSS segment. With
the current release, we finally resolved this long-standing problem by
moving the main stack to the context area, too. The tricky part was that
the context area is created by the main thread. So we hit a hen-and-egg
problem. We overcame this problem by splitting the process startup
into two stages, both called from the crt0 assembly code. The first
stage runs on a small stack within the BSS and has the sole purpose
of creating the context area and a thread object for the main thread.
This code path (and thereby the stack usage) is the same for all programs.
So we can safely dimension the stage-1 stack. Once the first stage
returns to the crt0 assembly code, the stack pointer is loaded with the
stack that is now located within the context area. Equipped with the
new stack, the actual startup code ('_main') including the global
constructors of the program is executed.
This change paved the ground for several further code unifications and
simplifications, in particular related to the dynamic linker.
Low-level OS infrastructure
###########################
Revised block-driver framework
==============================
Whereas Genode's block-session interface was designed to work asynchronously
and supports the out-of-order processing of requests, those capabilities
remained unused by the existing block services as those services used to
operate synchronously to keep their implementation simple. However, this
simplicity came at the prize of two disadvantages: First, it prevented us
to fully utilize native command queuing of modern disk controllers. Second,
when chaining components such as a block driver, the part_blk server, and
a file system, latencies accumulated along the chain of services. This
hurts the performance of random access patterns.
To overcome this limitation, we changed the block-component framework to work
asynchronously and to facilitate the recently introduced server API.
Consequently, all users of the API underwent an update. The affected
components are rom_loopdev, atapi_drv, fb_block_adapter, http_block, usb_drv,
and part_blk. For some components, in particular part_blk, this step led to a
complete redesign.
Besides the change of the block-component framework, the block-session
interface got extended to support logical block addresses greater than
32bit (LBA48). Thereby, the block component framework can now support
devices that exceed 2 TiB in size.
Block cache
===========
The provisioning of a block cache was one of the primary motivations behind the
[http://www.genode.org/documentation/release-notes/13.11#Dynamic_resource_balancing - dynamic resource balancing]
concept that was introduced in Genode 13.11. We are now introducing the first
version of such a cache.
The new block cache component located at _os/src/server/blk_cache/_ is both
a block-session client as well as a block-session server serving a single
client. It is meant to sit between a block-device driver and a file-system
server. When accessing the block device, it issues requests at a granularity
of 4K and thereby implicitly reads ahead whenever a client requests a smaller
amount of blocks. Blocks obtained from the device or written by the client
are kept in memory. If memory becomes scarce, the block cache first tries
to request further memory resources from its parent. If the request
gets denied, the cache evicts blocks from memory to the block device following
a least-recently-used replacement strategy. As of now, the block cache supports
dynamic resource requests to grow on demand but support for handling yield
requests is not yet implemented. So memory once handed out to the block cache
cannot be regained. Adding support for yielding memory on demand will be
complemented in the next version.
To see how to integrate the block cache in a Genode scenario, there is a
ready-to-use run script available at _os/run/blk_cache.run_.
File-system infrastructure
==========================
In addition to the integration of NetBSD's file systems, there are
file-system-related improvements all over the place.
First, the 'File_system::Session' interface has been extended with a 'sync'
RPC function. This function allows the client of a file system to force
the file system to write back its internal caches.
Second, we extended the FUSE implementation introduced with the previous
release.
Since file systems tend to have a built-in caching mechanism, we need to
sync these caches at the end of a session when using the fuse_fs server.
Therefore, each FUSE file system port has to implement a 'Fuse::sync_fs()'
function that executes the necessary actions if requested. Further
improvements are related to the handling of symbolic links and error
handling. Finally, we added a libc plugin for accessing NTFS file systems
via the ntfs-3g library.
Third, we complemented the family of FUSE-based libc plugins with a family of
FUSE-based file-system servers. To utilize a FUSE file system, there is a
dedicated binary (e.g., _os/src/server/fuse_fs/ext2_) for each FUSE
file-system server.
Note that write support is possible but considered to be experimental at this
point. For now, using it is not recommended.
To use the ext2_fuse_fs server in Noux, the following configuration snippet
may be used:
! <start name="ext2_fuse_fs">
! <resource name="RAM" quantum="8M"/>
! <provides> <service name="File_system"/> </provides>
! <config>
! <policy label="noux -> fuse" root="/" writeable="no" />
! </config>
! </start>
Finally, the libc file-system plugin has been extended to support 'unlink'.
Trace file system
=================
The new _trace_fs_ server provides access to a trace session by providing a
file-system session as front end. Combined with Noux, it allows for the
interactive exploration and tracing of Genode's process tree using
traditional Unix tools.
Each trace subject is represented by a directory ('thread_name.subject') that
contains specific files, which are used to control the tracing process of the
thread as well as storing the content of its trace buffer:
:'enable': The tracing of a thread is activated if there is a valid policy
installed and the intend to trace the subject was made clear by writing '1'
to the 'enable' file. The tracing of a thread may be deactivated by writing a
'0' to this file.
:'policy': A policy may be changed by overwriting the currently used one in the
'policy' file. In this case, the old policy is replaced by the new one and
automatically used by the framework.
:'buffer_size': Writing a value to the 'buffer_size' file changes the size of
the trace buffer. This value is evaluated only when reactivating the tracing
of the thread.
:'events': The trace-buffer contents may be accessed by reading from the
'events' file. New trace events are appended to this file.
:'active': Reading the file will return whether the tracing is active (1) or
not (0).
:'cleanup': Nodes of untraced subjects are kept as long as they do not change
their tracing state to dead. Dead untraced nodes are automatically removed
from the file system. Subjects that were traced before and are now untraced
can be removed by writing '1' to the 'cleanup' file.
To use the trace_fs, a configuration similar to the following may be used:
! <start name="trace_fs">
! <resource name="RAM" quantum="128M"/>
! <provides><service name="File_system"/></provides>
! <config>
! <policy label="noux -> trace"
! interval="1000"
! subject_limit="512"
! trace_quota="64M" />
! </config>
! </start>
:'interval': sets the period the Trace_session is polled. The
time is given in milliseconds.
:'subject_limit': specifies how many trace subjects should by acquired at
max when the Trace_session is polled.
:'trace_quota': is the amount of quota the trace_fs should use for the
Trace_session connection. The remaining amount of RAM quota will be used
for the actual nodes of the file system and the 'policy' as well as the
'events' files.
In addition, there are 'buffer_size' and 'buffer_size_limit' that define
the initial and the upper limit of the size of a trace buffer.
A ready-to-use run script can by found in 'ports/run/noux_trace_fs.run'.
Unified interfaces for graphics
===============================
Genode comes with several programs that perform software-based graphics
operations. A few noteworthy examples are the nitpicker GUI server,
the launchpad, the scout tutorial browser, or the terminal. Most of those
programs were equipped with their custom graphics back end. In some
cases such as the terminal, nitpicker's graphics back end was re-used.
But this back end is severely limited because its sole purpose is the
accommodation of the minimalistic (almost invisible) nitpicker GUI server.
The ongoing work on Genode's new user interface involves the creation of
new components that rely on a graphics back end. Instead of further
diversifying the zoo of graphics back ends, we took the intermediate step
to consolidate the existing back ends into one unified concept such that
application-specific graphics back ends can be created and extended using
modular building blocks. The new versions of nitpicker, scout, launchpad,
liquid_fb, nitlog, and terminal have been changed to use the new common
interfaces:
:os/include/util/geometry.h: Basic data structures and operations needed
for 2D graphics.
:os/include/util/color.h: Common color representation and utilities.
:os/include/os/pixel_rgba.h: Class template for representing a pixel.
:os/include/os/pixel_rgb565.h: Template specializations for RGB565 pixels.
:os/include/os/surface.h: Target surface, onto which graphics operations
can be applied.
:os/include/os/texture.h: Source texture for graphics operations that
transfer 2D pixel data to a surface.
The former _os/include/nitpicker_gfx/_ directory is almost deserted. The only
remainders are functors for the few graphics operations actually required by
nitpicker. For the scout widgets, the corresponding functors have become
available at the public headers at _demo/include/scout_gfx/_.
Because the scout widget set is used by at least three programs and will
most certainly play a role in new GUI components, we undertook a major
cleanup of the parts worth reusing. The result can be found at
_demo/include/scout/_.
New session interface for status reporting
==========================================
Genode has a uniform way of how configuration information is passed from
parents to children within the process tree by the means of "config" ROM
modules. Using this mechanism, a parent is able to steer the behaviour of
its children, not just at their start time but also during runtime.
Until now, however, there was no counterpart to the config mechanism, which
would allow a child to propagate runtime information to its parent. There
are many use cases for such a mechanism. For example, a bus-controller driver
might want to propagate a list of devices attached to the bus. When a new
device gets plugged in, this list should be updated to let the parent
take the new device resource into consideration. Another use case would be the
propagation of status information such as the feature set of a plugin.
Taken to the extreme, a process might expose its entire internal state to its
parent in order to allow the parent to kill and restart the process, and
feed the saved state back to the new process instance.
To cover these use cases, we introduced the new report-session interface. When
a client opens a report session, it transfers a part of its RAM quota to the
report server. In return, the report server hands out a dataspace dimensioned
according to the donated quota. Upon reception of the dataspace, the client
can write its status reports into the dataspace and inform the server about
the update via the 'submit' function. In addition to the mere reporting of
status information, the report-session interface is designed to allow the
server to respond to reports. For example, if the report mechanism is used to
implement a desktop notification facility, the user may interactively respond
to an incoming notification. This response can be reflected to the originator
of the notification via the 'response_sigh' and 'obtain_response' functions.
The new _report_rom_ component is both a report service and a ROM service. It
reflects incoming reports as ROM modules. The ROM modules are named
after the label of the corresponding report session.
Configuration
-------------
The report-ROM server hands out ROM modules only if explicitly permitted by a
configured policy. For example:
! <config>
! <rom>
! <policy label="decorator -> pointer" report="nitpicker -> pointer"/>
! <policy ... />
! ...
! </rom>
! </config>
The label of an incoming ROM session is matched against the 'label' attribute
of all '<policy>' nodes. If the session label matches a policy label, the
client obtains the data from the report client with the label specified in the
'report' attribute. In the example above, the nitpicker GUI server sends
reports about the pointer position to the report-ROM service. Those reports
are handed out to a window decorator (labeled "decorator") as ROM module.
XML generator utility
=====================
With the new report-session interface in place, comes the increased
need to produce XML data. The new XML generator utility located at
_os/include/util/xml_generator.h_ makes this extremely easy, thanks to
C++11 language features. For an example application, refer to
_os/src/test/xml_generator/_ and the corresponding run script at
_os/run/xml_generator.run_.
Dynamic ROM service for automated testing
=========================================
The new _dynamic_rom_ service provides ROM modules that change during the
lifetime of a ROM session according to a timeline. The main purpose of this
service is the automated testing of programs that are able to respond to ROM
module changes, for example configuration changes.
The configuration of the dynamic ROM server contains a '<rom>' sub node per
ROM module provided by the service. Each '<rom>' node hosts a 'name' attribute
and contains a sequence of sub nodes that define the timeline of the ROM
module. The possible sub nodes are:
:'<inline>': The content of the '<inline>' node is assigned to the content
of the ROM module.
:'<sleep>': Sleeps a number of milliseconds as specified via the 'milliseconds'
attribute.
:'<empty>': Removes the ROM module.
At the end of the timeline, it re-starts at the beginning.
Nitpicker GUI server
====================
The nitpicker GUI server has been enhanced to support dynamic screen
resizing. This is needed to let nitpicker respond to screen-resolution
changes, or when using a nested version of nitpicker within a resizable
virtual framebuffer window.
To accommodate Genode's upcoming user-interface concept, we introduced the
notion of a parent-child relationship between nitpicker views. If an existing
view is specified as parent at construction time of a new view, the parent
view's position is taken as the origin of the child view's coordinate space.
This allows for the grouping of views, which can be atomically repositioned by
moving their common parent view. Another use case is the handling of popup
menus in Qt5, which can now be positioned relative to their corresponding
top-level window. The relative position is maintained transparently to Qt when
the top-level window gets repositioned.
Libraries and applications
##########################
Noux runtime for executing Unix software
========================================
Noux plays an increasingly important role for Genode as it allows the use
of the GNU software stack. Even though it already supported a variety of
packages including bash, gcc, binutils, coreutils, make, and vim, some
programs were still limited by Noux' not fully complete POSIX semantics,
in particular with regard to signal handling. For example, it was not
possible to cancel the execution of a long-running process via Control-C.
To overcome those limitations, we enhanced Noux by adding the _kill_ syscall,
reworking the _wait_ and _execve_ syscalls, as well as adding
signal-dispatching code to the Noux libc. Special attention had to be paid to
the preservation of pending signals during the process creation via _fork_ and
_execve_.
The current implementation delivers signals each time a Noux syscall
returns. Signal handlers are executed as part of the normal control flow. This
is in contrast to traditional Unix implementations, which allow the
asynchronous invocation of signal handlers out of band with the regular
program flow. The obvious downside of our solution is that a program that got
stuck in a busy loop (and thereby not issuing any system calls) won't respond
to signals. However, as we regard the Unix interface just as a runtime and not
as the glue that holds the system together, we think that this compromise is
justified to keep the implementation simple and kernel-agnostic. In the worst
case, if a Noux process gets stuck because of such a bug, we certainly can
live with the inconvenience of restarting the corresponding Noux subsystem.
To complement our current activities on the block and file-system levels,
the e2fsprogs-v1.42.9 package as been ported to Noux. To allow the
block-device utilities to operate on Genode's block sessions, we added a new
"block" file system to Noux. Such a block file system can be mounted using a
'<block>' node within the '<fstab>'. By specifying a label attribute, each
block session request can be routed to the proper block session provider:
! <fstab>
! ...
! <dir name="dev">
! <block name="blkdev0" label="block_session_0" />
! </dir>
! ...
! </fstab>
In addition to this file system, support for the DIOCGMEDIASIZE ioctl
request was added. This request is used by FreeBSD and therefore by our
libc to query the size of the block device in bytes.
Qt5 refinements
===============
Our port of Qt5 used to rely on custom versions of synchronization
primitives such as 'QWaitCondition' and 'QMutex'. However, since most of the
usual pthread synchronization functions as relied on by Qt5's regular POSIX
back end have been added to Genode's pthread library by now, we could replace
our custom implementations by Qt5's POSIX version.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
The development of our base-hw kernel platform during this release cycle was
primarily geared towards adding multi-processor support. However, as we
haven't exposed the code to thorough testing yet, we deferred the integration
of this feature for the current release.
We increased the number of usable ARM platforms by adding basic support for
the ODROID XU board.
NOVA microhypervisor
====================
The port of VirtualBox to Genode prompted us to improve the NOVA platform in
the following respects.
NOVA used to omit the propagation of the FPU state of the guest OS to the
virtual machine monitor (VMM) during the world switch between the guest OS and
the VMM. With the Vancouver VMM, which is traditionally used on NOVA, this
omission did not pose any problem because Vancouver would never touch the FPU
state of the guest. So the FPU context of the guest was always preserved
throughout the handling of virtualization events. However, in contrast to the
Vancouver VMM, VirtualBox relies on the propagation of the FPU state between
the guest running in VT-X non-root mode and the guest running within the
VirtualBox recompiler. Without properly propagating the FPU state between both
virtualization back ends, both the guest OS in non-root mode and VirtualBox's
recompiler would corrupt each other's FPU state. After first implementing an
interim solution in our custom version of the kernel, the missing FPU context
propagation had been implemented in the upstream version of NOVA as well.
In contrast to most kernels, NOVA did not allow a thread to yield its current
time slice to another thread. The only way to yield CPU time was to block on
a semaphore or to perform an RPC call. Unfortunately both of those instruments
require the time-receiving threads to explicitly unblock the yielding thread
(by releasing the semaphore or replying to the RPC call). However, there are
situations where the progress of a thread may depend on an external
condition or a side effect produced by another (unknown) thread. One
particular example is the spin lock used to protect (an extremely short)
critical section of Genode's lock metadata. Apparently VirtualBox presented
us with several more use cases for thread-yield semantics. Therefore, we
decided to extend NOVA's kernel interface with a new 'YIELD' opcode to the
'ec_control' system call.

1123
doc/release_notes-14-05.txt
View File

File diff suppressed because it is too large Load Diff

1136
doc/release_notes-14-08.txt
View File

File diff suppressed because it is too large Load Diff

1047
doc/release_notes-14-11.txt
View File

File diff suppressed because it is too large Load Diff

899
doc/release_notes-15-02.txt
View File

@@ -1,899 +0,0 @@
===============================================
Release notes for the Genode OS Framework 15.02
===============================================
Genode Labs
Genode's [http://genode.org/about/road-map - roadmap] for this year puts a
strong emphasis on the consolidation and cultivation of the existing feature
set. With the first release of the year, version 15.02 pays tribute to this
mission by stepping up to extensive and systematic automated testing. As
a precondition for scaling up Genode's test infrastructure, the release
features a highly modular tool kit for exercising system scenarios on a growing zoo
of test machines. Section [Modular tool kit for automated testing] explains
the new tools in detail. In the spirit of improving the existing feature
set, Genode 15.02 vastly improves the performance and stability of our version of
VirtualBox running on the NOVA microhypervisor, solves long-standing shortcomings
of memory management on machines with a lot of RAM, addresses NOVA-related
scalability limitations, stabilizes our Rump-kernel-based file-system server,
and refines the configuration interface of the Intel wireless driver.
As the most significant new feature, the new version introduces virtualization
support for ARM to our custom base-hw kernel. Section [Virtualization on ARM]
outlines the design and implementation of this feature, which was greatly
inspired by NOVA's virtualization architecture and has been developed over the
time span of more than a year.
With respect to platform support, we are happy to accommodate the upcoming
USB-Armory board, which is a computer in the form factor of a USB
stick especially geared towards security applications. Section
[Support for the USB-Armory board] covers the background and the current
state of this line of work.
Virtualization on ARM
#####################
The ARMv7 architecture of recent processors like Cortex-A7, Cortex-A15, or
Cortex-A17 CPUs support hardware extensions to facilitate virtualization of
guest operating systems. With the current release, we enable the use of these
virtualization extensions in our custom base-hw kernel when running on the
Cortex-A15-based Arndale board.
While integrating ARM's virtualization extension, we aimed to strictly follow
microkernel-construction principles. The primary design is inspired by the
[http://hypervisor.org/ - NOVA OS Virtualization Architecture]. It is based on a
microhypervisor that provides essential microkernel mechanisms along with
basic primitives to switch between virtual machines (VMs). On top of the
microhypervisor, classical OS services are implemented as
ordinary, unprivileged user-level components. Those services can be used by other
applications. Services may be shared between applications or instantiated
separately, according to security and safety needs. Correspondingly,
following the NOVA principles, each VM has its own associated virtual-machine
monitor (VMM) that runs as an unprivileged user-level component. VMM implementations
can range from simple ones that just emulate primary device requirements to highly
complex monitors including sophisticated device models, like VirtualBox. The
NOVA approach allows to decouple the TCB complexity of one VM with respect to
another, as well as with respect to all components not related to
virtualization at all.
Along those lines, we extended the base-hw kernel/core conglomerate with API
extensions that enable user-level VMM components to create and control virtual
machines.
Design
======
The ARM virtualization extensions are based on the so-called security
extensions, commonly known as
[http://genode.org/documentation/articles/trustzone - TrustZone].
The ARM designers did not follow the
Intel approach to split the CPU into a "root" and a "guest" world while having all prior
existing CPU modes available in both worlds. Instead, ARM added a new privilege level
to the non-secure side of TrustZone that sits underneath the ordinary kernel
and userland privilege levels. It is subjected to a hypervisor-like kernel. All
instructions used to prepare a VM's environment have to be executed in this so
called "hyp" mode. In hyp mode, some instructions
differ from their regular behaviour on the kernel-privilege level.
For this reason, prior-existing kernel code cannot simply be reused in
hyp mode without modifications.
The base-hw kernel is meant to execute Genode's core component on bare hardware.
Core, which is an ordinary user-level component, is
linked together with a slim kernel library that is executed in privileged kernel
mode. To enable ARM hardware virtualization, we pushed this approach
even further by executing core in three different privilege levels. Thereby,
core shares the same view on hardware resources and virtual memory across all
levels. A code path is executed on a higher privilege level only if the code
would fail to execute on a lower privilege level.
Following this approach, we were able to keep most of the existing kernel code
with no modifications.
[image avirt_overview]
Genode's ARM kernel (core) runs across all privilege levels
The hypervisor part of core is solely responsible to switch between VMs and the
host system. Therefore, it needs to load/store additional CPU state that
normally remains untouched during context switches of ordinary tasks. It also needs to
configure the VM's guest-physical to host-physical memory translations. Moreover, the
virtualization extensions of the ARMv7 architecture are not related to the CPU
cores only. The interrupt controller and the CPU-local timers are also
virtualization-aware. Therefore, the hypervisor has to load/store state specific
to those devices, too. Nevertheless, the hypervisor merely reloads those
devices. It does not interpret their state.
In contrast to the low-complexity hypervisor, a user-level VMM can be complex
without putting the system's security at risk. It contains potentially complex
device-emulation code and assigns hardware resources such as memory and
interrupts to the VM. The VMM is an ordinary user-level component running
unprivileged. Of course, as a plain user-level component, it is not able to
directly access hardware resources. Hence an interface between VMMs and the
kernel is needed to share the state of a virtual machine. In the past, we faced a similar
problem when building a VMM for our former TrustZone experiments. It was natural
to build upon the available solution and to extend it where necessary. Core
provides a so-called VM service. Each VM corresponds to a session of this
service. The session provides the following extended interface:
:CPU state:
The CPU-state function returns a dataspace containing the virtual machine's
state. The state is initialized by the VMM before bootstrapping the VM, gets updated
by the hypervisor whenever it switches away from the VM, and can be used by
the VMM to interpret the behavior of the guest OS. Moreover, the CPU state can be
updated after the virtual machine monitor emulated instructions
for the VM.
:Exception handler:
The second function is used to register a signal handler that gets informed
whenever the VM produces a virtualization fault.
:Run:
The run function starts or resumes the execution of the VM.
:Pause:
The pause function removes the VM from the kernel's scheduler.
:Attach:
This function attaches a given RAM dataspace to a designated area of the
guest-physical address space.
:Detach:
The detach function invalidates a designated area of the guest-physical
address space.
:Attach_pic: Tells the hypervisor to attach the CPU's virtual interface of the
virtualization-aware interrupt controller to a designated area of the
guest-physical address space.
Implementation
==============
By strictly following the micro-kernel construction principles when integrating the
hypervisor into the base-hw kernel, we reached a minimally invasive solution. In
doing so, we took the time to separate TrustZone-specific code that was formerly
an inherent part of the kernel on ARMv7 platforms. Now, TrustZone- and
virtualization-specific aspects are incorporated into the kernel only if
actually used. The change in complexity of the whole core component expressed in
lines of code is shown in the table below. As can be seen, the additional code in
the root of the trusted computing base when using virtualization is about 700-800
LOC.
Platform | with TrustZone, no VT | TrustZone/VT optional
-----------------------------------------------------------------
hw_arndale | 17970 LOC | 18730 LOC
----------------------------------------------------------------
hw_imx53_qsb | 17900 LOC | 17760 LOC
----------------------------------------------------------------
hw_imx53_qsb_tz | 18260 LOC | 18320 LOC
----------------------------------------------------------------
hw_rpi | 17500 LOC | 17430 LOC
----------------------------------------------------------------
hw_panda | 18040 LOC | 17880 LOC
----------------------------------------------------------------
hw_odroid_xu | 17980 LOC | 18050 LOC
Besides the VM world switch, we enabled support for the so-called "large
physical address extension" (LPAE), which is obligatory when using
virtualization. It allows for addressing a 40-bit instead of only 32-bit physical
address space. Moreover, to execute in hypervisor mode, the bootstrap code of
the kernel had to be set up properly. Hence, when booting on the Arndale board,
the kernel now prepares the non-secure TrustZone world first, and finally leaves the
secure world forever.
To test and showcase the ARM virtualization features integrated in base-hw, we
implemented a minimal, exemplary VMM. It can be found in
_repos/os/src/server/vmm_. The VMM emulates a simplified variant of ARM's
Versatile Express Cortex-A15 development platform. Currently, it only comprises
support for the CPU, the timer, the interrupt controller, and a UART device. It is
written in 1100 lines of C++ in addition to the base Genode libraries. The VMM
is able to boot a vanilla Linux kernel compiled with a slightly modified
standard configuration (no-SMP), and a device tree description stripped down to
the devices provided by the VMM. This release includes an automated run test that
executes the Linux kernel on top of the VMM on Genode. It can be started via:
! make run/vmm
[image avirt_screen]
Three Linux serial consoles running in parallel on top of Genode
Modular tool kit for automated testing
######################################
In
[http://genode.org/documentation/release-notes/13.05#Automated_quality-assurance_testing - Genode version 13.05],
we already introduced comprehensive support for the automated testing of
Genode scenarios. Since then, Genode Labs has significantly widened the scope
of its internal test infrastructure, both in terms of the coverage of the test
scenarios as well as the variety of the used hardware platforms.
The centerpiece of our test infrastructure is the so-called run tool. Steered
by a script (run script), it performs all the steps necessary to test drive
a Genode system scenario. Those steps are:
# *Building* the components of a scenario
# *Configuration* of the init component
# Assembly of the *boot directory*
# Creation of the *boot image*
# *Powering-on* the test machine
# *Loading* of the boot image
# Capturing the *LOG output*
# *Validation* of the scenario behavior
# *Powering-off* the test machine
Each of those steps depends on various parameters such as the
used kernel, the hardware platform used to run the scenario, the
way the test hardware is connected to the test infrastructure
(e.g., UART, AMT, JTAG, network), the way the test hardware is powered or
reseted, or the way of how the scenario is loaded into the test hardware.
Naturally, to accommodate the growing variety of combinations of those
parameters, the complexity of the run tool increased over time.
This growth of complexity prompted us to eventually turn the run tool into a
highly modular and extensible tool kit.
Originally, the run tool consisted of built-in rules that could be
extended and tweaked by a kernel-specific supplement called run environment.
The execution of a run script used to depend on the policies built into
the run tool, the used run environment, and optional configuration
parameters (run opts).
The new run tool kit replaces most of the formerly built-in policies by the
ability to select and configure different modules for the various steps.
The selection and configuration of the modules is expressed in the run-tool
configuration. There exist the following types of modules:
:boot-dir modules:
These modules contain the functionality to populate the boot directory
and are specific to each kernel. It is mandatory to always include the
module corresponding to the used kernel.
_(the available modules are: linux, hw, okl4, fiasco, pistachio, nova,_
_codezero, foc)_
:image modules:
These modules are used to wrap up all components used by the run script
in a specific format and thereby prepare them for execution.
Depending on the used kernel, different formats can be used. With these
modules, the creation of ISO and disk images is also handled.
_(the available modules are: uboot, disk, iso)_
:load modules:
These modules handle the way the components are transfered to the
target system. Depending on the used kernel there are various options
to pass on the components. For example, loading from TFTP or via JTAG is handled
by the modules of this category.
_(the available modules are: tftp, jtag, fastboot)_
:log modules:
These modules handle how the output of a currently executed run script
is captured.
_(the available modules are: qemu, linux, serial, amt)_
:power_on modules:
These modules are used for bringing the target system into a defined
state, e.g., by starting or rebooting the system.
_(the available modules are: qemu, linux, softreset, powerplug, amt)_
:power_off modules:
These modules are used for turning the target system off after the
execution of a run script.
_(the available modules are: powerplug)_
When executing a run script, only one module of each category must be used.
Each module has the form of a script snippet located under the
_tool/run/<step>/_
directory where _<step>_ is a subdirectory named after the module type.
Further instructions about the use of each module (e.g., additional
configuration arguments) can be found in the form of comments inside the
respective script snippets.
Thanks to this modular structure,
the extension of the tool kit comes down to adding a file at the corresponding
module-type subdirectory. This way, custom work flows (such as tunneling JTAG
over SSH) can be accommodated fairly easily.
Usage examples
==============
To execute a run script, a combination of modules may be used. The combination
is controlled via the RUN_OPT variable used by the build framework. Here are a
few common exemplary combinations:
Executing NOVA in Qemu:
!RUN_OPT = --include boot_dir/nova \
! --include power_on/qemu --include log/qemu --include image/iso
Executing NOVA on a real x86 machine using AMT for resetting the target system
and for capturing the serial output while loading the files via TFTP:
!RUN_OPT = --include boot_dir/nova \
! --include power_on/amt --power-on-amt-host 10.23.42.13 \
! --power-on-amt-password 'foo!' \
! --include load/tftp --load-tftp-base-dir /var/lib/tftpboot \
! --load-tftp-offset-dir /x86 \
! --include log/amt --log-amt-host 10.23.42.13 \
! --log-amt-password 'foo!'
Executing Fiasco.OC on a real x86 machine using AMT for resetting, USB serial
for output while loading the files via TFTP:
!RUN_OPT = --include boot_dir/foc \
! --include power_on/amt --amt-host 10.23.42.13 --amt-password 'foo!' \
! --include load/tftp --tftp-base-dir /var/lib/tftpboot \
! --tftp-offset-dir /x86 \
! --include log/serial --log-serial-cmd 'picocom -b 115200 /dev/ttyUSB0'
Executing base-hw on a Raspberry Pi using powerplug to reset the hardware,
JTAG to load the image and USB serial to capture the output:
!RUN_OPT = --include boot_dir/hw \
! --include power_on/powerplug --power-on-powerplug-ip 10.23.42.5 \
! --power-on-powerplug-user admin \
! --power-on-powerplug-password secret \
! --power-on-powerplug-port 1
! --include power_off/powerplug --power-off-powerplug-ip 10.23.42.5 \
! --power-off-powerplug-user admin \
! --power-off-powerplug-password secret \
! --power-off-powerplug-port 1
! --include load/jtag \
! --load-jtag-debugger /usr/share/openocd/scripts/interface/flyswatter2.cfg \
! --load-jtag-board /usr/share/openocd/scripts/interface/raspberrypi.cfg \
! --include log/serial --log-serial-cmd 'picocom -b 115200 /dev/ttyUSB0'
After the run script was executed successfully, the run tool will print the
string 'Run script execution successful.". This message can be used to check
for the successful completion of the run script when doing automated testing.
Meaningful default behaviour
============================
To maintain the ease of use of creating and using a build directory, the
'create_builddir' tool equips a freshly created build directory with a meaningful
default configuration that depends on the selected platform. For example, if
creating a build directory for the Linux base platform, RUN_OPT
is initially defined as
! RUN_OPT = --include boot_dir/linux \
! --include power_on/linux --include log/linux
Low-level OS infrastructure
###########################
Improved management of physical memory
======================================
On machines with a lot of memory, there exist constraints with regard to
the physical address ranges of memory:
* On platforms with a non-uniform memory architecture, subsystems should
preferably use memory that is local to the CPU cores the subsystem is using.
Otherwise the performance is impeded by costly memory accesses to
the memory of remote computing nodes.
* Unless an IOMMU is used, device drivers program physical addresses
into device registers to perform DMA operations. Legacy devices such as
USB UHCI controllers expect a 32-bit address. Consequently, the memory
used as DMA buffers for those devices must not be allocated above 4 GiB.
* When using an IOMMU on NOVA, Genode represents the address space
accessible by devices (by the means of DMA) using a so-called device PD
([http://genode.org/documentation/release-notes/13.02#DMA_protection_via_IOMMU]).
DMA transactions originating from PCI devices are subjected to the virtual
address space of the device PD.
All DMA buffers are identity-mapped with their physical addresses within
the device PD. On 32-bit systems with more than 3 GiB of memory, this
creates a problem. Because the device PD is a regular user-level component, the
upper 1 GiB of its virtual address space is preserved for the kernel. Since
no user-level memory objects can be attached to this
area, the physical address range to be used for DMA buffers is limited
to the lower 3 GiB.
Up to now, Genode components had no way to influence the allocation of
memory with respect to physical address ranges. To solve the problems outlined
above, we extended core's RAM services to take allocation constraints
as session arguments when a RAM session is created. All dataspaces created
from such a session are subjected to the specified constraints. In particular,
this change enables the AHCI/PCI driver to allocate DMA buffers at suitable
physical address ranges.
This innocent looking feature to constrain RAM allocations raises a problem
though: If any component is able to constrain RAM allocations in
arbitrary ways, it would become able to scan the physical address space for
allocated memory by successively opening RAM sessions with the constraints set
to an individual page and observe whether an allocation succeeds or not. Two
conspiring components could use this information to construct a covert storage
channel.
To prevent such an abuse, the init component filters out allocations
constrains from RAM-session requests unless explicitly permitted. The
permission is granted by supplementing the RAM resource assignment of
a component with a new 'constrain_phys' attribute. For example:
! <resource name="RAM" quantum="3M" constrain_phys="yes"/>
Init component
==============
Most of Genode's example scenarios in the form of run scripts support
different platforms. However, as the platform details vary, the run scripts
have to tweak the configuration of the init component according to the
features of the platform.
For example, when declaring an explicit route to a framebuffer driver named
"fb_drv", the run script won't work on Linux because on this platform, the
framebuffer driver is called "fb_sdl".
Another example is the role of the USB driver. Depending on the platform, the
USB driver is an input driver, a block driver, a networking driver, or a
combination of those.
Consequently, run scripts with support
for a great variety of platforms tend to become convoluted with
platform-specific conditionals.
To counter this problem, we enhanced init to support aliases for component
names. By defining the following aliases in the init configuration
! <alias name="nic_drv" child="usb_drv"/>
! <alias name="input_drv" child="usb_drv"/>
! <alias name="block_drv" child="usb_drv"/>
the USB driver becomes reachable for session requests routed to either "usb_drv",
"nic_drv", "input_drv", and "block_drv". Consequently, the routing
configuration of components that use either of those drivers does no longer
depend on any platform-intrinsic knowledge.
RTC session interface
=====================
Until now, the RTC session interface used an integer to return the current
time. Although this is preferable when performing time-related
calculations, a structured representation is more convenient to use, i.e., if
the whole purpose is showing the current time. This interface change is only
visible to components that use the RTC session directly.
Since the current OS API of Genode lacks time-related functions, most users
end up using the libc, which already converts the structured time stamp
internally, or provide their own time related functions.
Update of rump-kernel-based file systems
========================================
We updated the rump-kernel support to a newer rump-kernel version (as of mid of
January 2015). This way, Genode is able to benefit from upstream stability
improvements related to the memory management. Furthermore, we revised the
Genode backend to allow the rump_fs server to cope well with a large amount of
memory assigned to it. The latter is useful to utilize the block cache of the
NetBSD kernel.
Libraries and applications
##########################
As a stepping stone in the
[https://github.com/genodelabs/genode/issues/1399 - forthcoming community effort]
to bring the Nix package manager to Genode, ports of libbz2 and sqlite have
been added to the _repos/libports/_ repository.
Runtime environments
####################
VirtualBox on NOVA
==================
Whereas our previous efforts to run VirtualBox on Genode/NOVA were mostly
concerned with enabling principal functionality and with the addition of
features, we took the release cycle of Genode 15.02 as a chance to focus
on performance and stability improvements.
:Performance:
Our goal with VirtualBox on NOVA is to achieve a user experience
comparable to running VirtualBox on Linux. Our initial port of VirtualBox used
to cut a lot of corners with regards to performance and timing accuracy
because we had to concentrate on more fundamental issues of the porting
work first. Now, with the feature set settled, it was time to revisit
and solidify our interim solutions.
The first category of performance improvements is the handling of timing,
and virtual guest time in particular. In our original version,
we could observe a substantial drift of the guest time compared to the host time.
The drift is not merely inconvenient but may even irritate the guest OS
because it violates its assumptions about the behaviour of certain virtual devices.
The drift was caused by basing the timing on a simple jiffies counter
that was incremented by a thread after sleeping for a fixed period. Even
though the thread almost never executes, there is still a chance that it gets
preempted by the kernel and resumed only after the time slices of
concurrently running threads have elapsed. This can take tens of milliseconds.
During this time, the jiffies counter remains unchanged. We could
significantly reduce the drift by basing the timing on absolute time values
requested from the timer driver. Depending on the used guest OS, however,
there is still a residual inaccuracy left, which is subject to ongoing
investigations.
The second type of improvements is related to the handling of virtual
interrupts. In its original habitat, VirtualBox relies on so-called
external-interrupt virtualization events. If a host interrupt occurs while the
virtual machine is active, the virtualization event is forwarded by the
VirtualBox hypervisor to the virtual machine monitor (VMM).
On NOVA, however, the kernel does not propagate this
condition to the user-level VMM because the occurrence of host interrupts should
be of no matter to the VMM. In the event of a host interrupt, NOVA takes
a normal scheduling decision (eventually activating the user-level device driver
the interrupt belongs to) and leaves the virtual CPU (vCPU) in a runnable
state - to be rescheduled later. Once the interrupt is handled, the vCPU gets
resumed. The VMM remains out of the loop. Because the update of the VirtualBox
device models ultimately relies on the delivery of external-interrupt
virtualization events, the lack of this kind of event introduced huge delays
with respect to the update of device models and the injection of virtual
interrupts. We solved this problem by exploiting a VirtualBox-internal
mechanism called POKE. By setting the so-called POKE flag, an I/O thread is
able to express its wish to force the virtual machine into the VMM. We only
needed to find the right spots to set the POKE flag.
Another performance-related optimization is the caching of RTC time
information inside VirtualBox. The original version of the gettimeofday
function used by VirtualBox contacted the RTC server for obtaining the
wall-clock time on each call. After the update to VirtualBox 4.3, the rate of those
calls increased significantly. To reduce the costs of these calls, our
new version of gettimeofday combines infrequent calls to the RTC driver
with a component-local time source based on the jiffies mechanism mentioned above.
With these optimizations in place,
simple benchmarks like measuring the boot time of Window 7 or the time of
compiling Genode within a Debian VM suggest that our version of VirtualBox
has reached a performance that is roughly on par with the Linux version.
:Stability:
Since the upgrade to VirtualBox 4.3.16 in release 14.11, we fixed several
regression issues caused by the upgrade. Beside that, we completed the
support to route serial output of guests to Genode, lifted the restriction
to use just one fixed VESA mode, and enabled support for 32-bit Windows 8
guests on 64-bit Genode/NOVA. The 64-bit host restriction stems from
the fact that Windows 8 requires support for the non-executable bit (NX)
feature of page tables. The 32-bit version of the NOVA kernel does not leverage
the physical address extension (PAE) feature, which is a pre-requisite for
using NX on 32-bit.
In the course of the adaptation, our port of VirtualBox now evaluates the
PAE and HardwareVirtExUX XML tags of .vbox files:
!<VirtualBox xmlns=...>
! <Machine uuid=...>
! <Hardware ..>
! <CPU ...>
! <HardwareVirtExUX enabled="true"/>
! <PAE enabled="true"/>
! ...
The PAE tag specifies whether to report PAE capabilities to the guest
or not. The HardwareVirtExUx tag is used by our port to decide whether to stay
for non-paged x86 modes in Virtualbox's recompiler (REM) or not. Until now, we used REM
to emulate execution when the guest was running in real mode and protected mode
with paging disabled. However, newer Intel machines support the unrestricted guest
feature, which makes the usage of REM in non-paged modes not strictly
necessary anymore. Setting the HardwareVirtExUx tag to false accommodates
older machines with no support for the unrestricted-guest feature.
Device drivers
##############
iPXE-based network drivers
==========================
We enabled and tested the driver with Intel I218-LM and I218-V PCI devices.
Intel wireless stack
====================
In this release, several small issues regarding the wireless stack are fixed.
From now on, the driver only probes devices on the PCI bus that correspond to
the PCI_CLASS_NETWORK_OTHER device class. Prior to that, the driver probed all
devices attached to the bus resulting in problems with other devices, e.g.
the GPU, when accessing their extended PCI config space.
Since the driver uses cooperative scheduling internally, it must never block
or, in case it blocks, must schedule another task. Various sleep functions
lacked this scheduling call and are now fixed. Furthermore, a bug in the timer
implementation has been corrected, which caused the scheduling of wrong timeouts.
In addition to these fixes, patches for enabling the support for
Intel 7260 cards were incorporated.
Up to now, the configuration of the wireless driver was rather inconvenient because
it did not export any information to the system. The driver now creates two
distinct reports to communicate its state and information about the wireless
infrastructure to other components. The first one is a list of all available
access points. The following exemplary report shows its structure:
!<wlan_accesspoints>
! <accesspoint ssid="skynet" bssid="00:01:02:03:04:05" quality="40"/>
! <accesspoint ssid="foobar" bssid="01:02:03:04:05:06" quality="70" protection="WPA-PSK"/>
! <accesspoint ssid="foobar" bssid="01:02:03:04:05:07" quality="10" protection="WPA-PSK"/>
!</wlan_accesspoints>
Each '<accesspoint>' node has attributes that contain the SSID and the BSSID
of the access point as well as the link quality (signal strength). These
attributes are mandatory. If the network is protected, the node will also
have an attribute describing the type of protection in addition.
The second report provides information about the state of the connection
with the currently associated access point:
!<wlan_state>
! <accesspoint ssid="foobar" bssid="01:02:03:04:05:06" quality="70"
! protection="WPA-PSK" state="connected"/>
!</wlan_state>
Valid state values are 'connected', 'disconnected', 'connecting' and
'disconnecting'.
The driver obtains its configuration via a ROM module. This ROM
module contains the selected access point and can be updated during runtime.
To connect to an access point, a configuration like the following is used:
!<selected_accesspoint ssid="foobar" bssid="01:02:03:04:05:06"
! protection="WPA-PSK" psk="foobar123!"/>
To disconnect from an access point, an empty configuration can be set:
!<selected_accesspoint/>
For now, the prevalent WPA/WPA2 protection using a pre-shared key is supported.
Improved UART driver for Exynos5
================================
The UART driver for the Exynos5 SoC has been enhanced by enabling the RX
channel. This improvement was motivated by automated tests, where a run script
needs to interact with some component via a terminal connection.
Touchscreen support
===================
We enabled support of Wacom USB touchscreen devices via dde_linux - a port of
Linux USB driver to Genode. In order to make touchscreen coordinates
usable by Genode's input services, they must be calibrated
to screen-absolute coordinates. The screen resolution is not determined
automatically by the USB driver. It can, however, be configured as a sub
node of the '<hid>' XML tag of the USB driver's configuration:
!<start name="usb_drv">
! ...
! <config uhci=... ohci=... xhci=...>
! <hid>
! <screen width="1024" height="768"/>
! </hid>
! ...
USB session interface
=====================
We enhanced our USB driver with the support of remote USB sessions. This
feature makes it possible to implement USB-device drivers outside the USB
server using a native Genode API. The new USB session can be found under
_repos/os/include/usb_session_ and can be used to communicate with the USB
server, which merely acts as a host controller and HUB driver in this scenario.
Under _repos/os/include/usb_, there are a number of convenience
and wrapper functions that operate directly on top of a USB session. These
functions are meant to ease the task of USB-device-driver programming by hiding
most of the USB session management, like packet-stream handling.
We also added a USB terminal server, which exposes a Genode terminal session to
its clients and drives the popular PL2303 USB to UART adapters using the new
USB-session interface.
A practical use case for this component is the transmission of logging data on
systems where neither UART, AMT, nor JTAG are available. A run script
showcasing this feature can be found at _repos/dde_linux/run/usb_terminal.run_.
RTC proxy driver for Linux
==========================
There are a handful of run scripts that depend on the RTC service. So far,
it was not possible to run these tests on Linux due to the lack of an RTC
driver on this platform. To address this problem, we created a proxy driver
that uses the time() system call to provide a
reasonable base period on Linux.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Support for the USB-Armory board
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
With [https://www.crowdsupply.com/inverse-path/usb-armory - USB Armory],
there is an intriguing hardware platform for Genode on the horizon.
In short, USB Armory is a computer in the form factor of a USB
stick. It is meant for security applications such as VPNs,
authentication tokens, and encrypted storage. It is based on the
FreeScale i.MX53 SoC, which is well supported by Genode, i.e.,
Genode can be used as secure-world OS besides Linux running in the
normal world.
Apart from introducing a novel form factor, this project is
interesting because it strives to be an 100% open platform, which
includes hardware, software, and firmware. This motivated us to
bring Genode to this platform.
The underlying idea is to facilitate
[http://genode.org/documentation/articles/trustzone - ARM TrustZone] to
use Genode as a companion to a Linux-based OS on the platform.
Whereas Linux would run in the normal world of TrustZone, Genode runs
in the secure world. With Linux, the normal world will control the
communication over USB and provide a familiar environment to implement
USB-Armory applications. However, security-critical functions and data like
cryptographic keys will reside exclusively in the secure world. Even in
the event that Linux gets compromised, the credentials of the user
will stay protected.
The support of the USB Armory platform was added in two steps:
First, we enabled our base-hw kernel to run as TrustZone monitor with
Genode on the "secure side". Since the USB Armory is based on the
FreeScale i.MX53 SoC, which Genode already supported, this step went
relatively straight-forward.
Second, we enabled a recent version of the Linux kernel (3.18) to run in the
normal world. The normal world is supervised by a user-level Genode component
called tz_vmm (TrustZone Virtual Machine Monitor). The tz_vmm is, among
others, responsible for providing startup and hardware information to the
non-secure guest. The Linux kernel version we used previously as TrustZone
guest on i.MX53 boards expected this information to be communicated via
so-called ATAGs. The new version, however, expects this to be done via a
device tree blob. As a consequence, the tz_vmm had to be adapted to properly
load this blob into the non-secure RAM. The original USB-Armory device tree
was modified to blind out the RAM regions that get protected by the TrustZone
hardware. This way, Linux won't attempt to access them. Furthermore,
to keep basic user interaction simple, our device tree tells Linux to use the
same non-secure UART as Genode for console I/O.
The kernel itself received some modifications, for two reasons. First,
we don't want Linux to rely on resources that are protected to keep
the secure world secure. This is why the driver for the interrupt controller
that originally made use of the TrustZone interrupt configuration, had to be
adapted. Second, to prevent Linux from disturbing Genode activities, we
disabled most of the dynamic clock and power management as it may sporadically
gear down or even disable hardware that Genode relies on. Furthermore, we
disabled the Linux drivers for I2C interfaces and the GPIO configuration as
these are reserved for Genode.
IPC helping
~~~~~~~~~~~
In traditional L4 microkernels, scheduling parameters (like time-slice
length and priority) used to be bound to threads. Usually, those parameters
are defined at thread creation time. The initial version
of base-hw followed this traditional approach. However, it has a few problems:
* For most threads, the proper *choice of scheduling parameters* is very
difficult if not impossible. For example, the CPU-time demands of a
server thread may depend on the usage patterns of its clients. Most
theoretical work in the domain of scheduling presumes the knowledge of
job lengths in advance of computing a schedule. But in practice and in
particular in general-purpose computing, job lengths are hardly known a priori.
As a consequence, in most scenarios, scheduling parameters are
set to default values.
* With each thread being represented as an independent schedulable entity,
the kernel has to take a scheduling decision each time a thread performs an
IPC call because the calling thread gets blocked and the called thread
may get unblocked. In a microkernel-based system, those events occur at a
much higher rate than the duration of typical time slices, which puts the
scheduler in a *performance-critical* position.
* Regarding IPC calls, a synchronous flow of control along IPC call chains is
desired. Ideally, an IPC call should have the same characteristics as
a function call with respect to scheduling. When a client thread performs an
IPC call, it expects the server to immediately become active to
handle the request. But if the kernel treats each thread independently,
it may pick any other thread and thereby introduce *high latencies* into
IPC operations.
To counter those problems, the NOVA microhypervisor introduced a new approach
that decouples scheduling parameters from threads. Instead of selecting
threads for execution, the scheduler selects so-called scheduling contexts.
For a selected scheduling context, the kernel dynamically determines a
thread to execute by taking IPC relationships into account. When a thread
performs an IPC, the thread's scheduling context will be used to execute
the called server. In principle, a server does not need CPU time on its own
but always works with CPU resources provided by clients.
The new version of the base-hw kernel adapts NOVA's approach with slight
modifications. Each thread owns exactly one scheduling context for its entire
lifetime. However, by the means of "helping" during an IPC call, the caller
lends its scheduling context to the callee. Even if the callee is still busy
and cannot handle the IPC request right away, the caller helps because it
wants the callee to become available for its request as soon as
possible. Consequently, a thread has potentially many scheduling contexts at
its disposal, its own scheduling context plus all scheduling contexts
provisioned by helpers. This works transitively.
Purged outdated platforms
~~~~~~~~~~~~~~~~~~~~~~~~~
We removed the support for two stale platforms that remained unused for
more than a year, namely FreeScale i.MX31 and the TrustZone variant
of the Coretile Versatile Express board.
NOVA
====
On Genode/NOVA, we used to employ one pager thread in core for each thread
in the system. We were forced to do so because not every page
fault can be resolved immediately. In some situations, core asynchronously
propagates the fault to an external component for the resolution.
In the meantime, the
pager thread leaves the page fault unanswered. Unfortunately, the kernel
provides no mechanism to support this scenario besides just blocking the
pager thread using a semaphore. This, in turn, means that the pager thread is not
available for other page-fault requests. Ultimately, we had to setup a
dedicated pager per thread.
This implementation has the downside of "wasting" memory for a lot of
pager threads. Moreover, it becomes a denial-of-service vector as soon as more
threads get created than core can accommodate. The number of threads is
limited per address space - also for core - by the size of Genode's context
area, which typically means 256 threads.
To avoid the downsides mentioned, we extended the NOVA IPC reply syscall to
specify an optional semaphore capability. The NOVA kernel validates the
capability and blocks the faulting thread in the semaphore. The faulted thread
remains blocked even after the pager has replied to the fault message. But
the pager immediately becomes available for other
page-fault requests. With this change, it suffices to maintain only one pager
thread per CPU for all client threads.
The benefits are manifold. First, the base-nova implementation converges more
closely to other Genode base platforms. Second, core can not run out of threads
anymore as the number of threads in core is fixed for a given setup. And the
third benefit is that the helping mechanism of NOVA can be leveraged for
concurrently faulting threads.
Build system and tools
######################
Tools for convenient handling of port contrib directories
=========================================================
We supplemented our tools for the ports mechanism with two convenient
scripts:
:_tool/ports/shortcut_:
Creates a symbolic link from _contrib/<port-name>-<hash>_ to
_contrib/<port-name>_. This is useful when working on the third-party
code contained in the _contrib_ directory.
:_tool/ports/current_:
Prints the current contrib directory of a port. When switching
branches back and forth, the hash of the used port might change.
The script provides a shortcut to looking up the hash file for a
specific port within the repositories and printing its content.

1216
doc/release_notes-15-05.txt
View File

File diff suppressed because it is too large Load Diff

791
doc/release_notes-15-08.txt
View File

@@ -1,791 +0,0 @@
===============================================
Release notes for the Genode OS Framework 15.08
===============================================
Genode Labs
The version 15.08 marks the beginning of Genode as day-to-day OS as one of the
project's core developers switched to using Genode/NOVA on his machine,
stressing the OS infrastructure we created over the course of the last seven
years. Thanks to components like VirtualBox, the Noux runtime for GNU software,
the Linux wireless stack and Rump-kernel-based file systems, the transition
went actually much smoother than expected. So other members of the team plan
to follow soon. Section [Genode as day-to-day operating system] gives an
overview of the taken approach. Genode's use as general-purpose OS provided
the incentive for most of the improvements featured by the current release,
starting with the addressing of the long-standing kernel-memory management
deficiencies of the NOVA kernel (Section [NOVA kernel-resource management]),
over enhancements of Genode's tracing and file-system facilities, to vast
improvements of the guest-host integration of VirtualBox when running on
Genode.
The release is accompanied with a second line of work led by our friends
at Codelabs: Enabling Genode to run on top of their Muen separation
kernel as described in Section [Genode on top of the Muen Separation Kernel].
Muen is a low-complexity kernel for the 64-bit x86 architecture that
statically partitions the machine into multiple domains. In contrast to
microkernels like the ones already supported by Genode, the assignment
of physical resources (such as memory, CPU time, and devices) happens at
system-integration time. Since an isolation kernel does not have to deal
with dynamic resource management at runtime, it is less complex than
a general-purpose microkernel. This makes it relatively easy to reason about
its strong isolation properties, which, in turn, makes it attractive for
high-assurance computing. With Genode being able to run within a Muen
domain, the rich component infrastructure of Genode can be combined with
the strong isolation guarantees of Muen.
Genode on top of the Muen Separation Kernel
###########################################
_This section was written by Adrian-Ken Rueegsegger and Reto Buerki who_
_conducted the described line of work independent from Genode Labs._
After completing our x86_64 port of the Genode base-hw kernel, which was
featured in the
[http://genode.org/documentation/release-notes/15.05#Principal_support_for_the_64-bit_x86_architecture - previous release (15.05)],
we immediately started working on our main goal: running a Genode system as
guest on the Muen Separation Kernel (SK). This would enable the Muen platform
to benefit from the rich ecosystem of Genode.
For those who have not read the 15.05 Genode release notes, [http://muen.sk - Muen]
is an Open-Source microkernel, which uses the [http://spark-2014.org/ - SPARK]
programming language to enable light-weight formal methods for high assurance.
The 64-bit x86 kernel, currently consisting of a little over 5'000 LOC, makes
extensive use of the latest Intel virtualization features and has been formally
proven to contain no runtime errors at the source-code level.
The new 'hw_x86_64_muen' platform, as the name implies, extends the 'hw_x86_64'
base-hw kernel by replacing the PIC and timer drivers with paravirtualized
variants.
In contrast to other kernels supported by Genode, the architecture with Muen is
different in the sense that the entire 'hw_x86_64_muen' Genode system runs as
guest VM in VMX non-root mode on the SK. From the perspective of Muen, Genode
is executed on top of the kernel like any other guest OS without special
privileges.
[image muen_system_overview]
Genode running on top of the Muen Separation Kernel alongside other subjects
This loose coupling of Muen and Genode base-hw enables the robust combination
of a static, low-complexity SK with a feature-rich and extensive OS framework.
The result is a flexible platform for the construction of component-based
high-assurance systems.
People interested in giving the 'hw_x86_64_muen' platform a spin can find a
small tutorial at _repos/base-hw/doc/x86_64_muen.txt_.
NOVA kernel-resource management
###############################
For several years, the NOVA kernel has served as Genode's primary base
platform on x86. The main reasons for this choice are: the kernel provides -
among the supported x86 kernels - the richest feature set like the support of
IOMMUs, virtualization, and SMP. It also offers a clean design and a stable
kernel interface. The available kernel-interface specification and the
readable and modern source base are a pleasure to work with. Hence, Genode
Labs is able to fully commit to the maintenance and further evolution of this
kernel.
Nevertheless, since the beginning, the vanilla kernel lacks one essential
feature to reliably host Genode as user-land, namely the proper management of
the memory used by the kernel itself (in short kernel-memory management). In
the past, we already extended the kernel to free up kernel resources when
destroying kernel objects, e.g., protection domains and page-tables, threads,
semaphores, and portals. Still, on Genode/NOVA, a component may trigger
arbitrary kernel-memory consumption during RPC by delegating memory,
capabilities, or by creating other components via Genode's core component. If
the kernel memory gets depleted, the kernel panics with an "Out of memory"
message and the entire Genode scenario stops.
In principal, the consumption of kernel memory can be deliberately provoked by
a misbehaving (greedy) component. But also during the regular day-to-day usage
of Genode, can such a situation occur when the system is used in a highly
dynamic fashion. For example, compiling and linking source code within the
noux environment constantly creates and destroys protection domains, threads,
and memory mappings. Our nightly test of compiling Genode within noux triggers
this condition every once in a while.
The main issue here is that the consumption of kernel memory is not accounted
by Genode. The kernel interface does not support such a feature. Kernels like
seL4 as well as Genode's custom base-hw kernel show how this problem can be
solved.
To improve the current situation - where the overall kernel memory is a fixed
amount - we extended NOVA in the following ways: First, the NOVA kernel
accounts any kernel memory consumption per protection domain. Second, each
process has a limited amount of kernel-memory quota it can use. Last, the
kernel detects when the quota limit of a protection domain is reached.
If the third condition occurs, the kernel stops the offending thread and
(optionally) notifies a handler thread. This so called out-of-memory (OOM)
handler thread receives information about the current situation and may
respond to it in the following ways:
* Stop the thread of the depleted protection domain, or
* Transfer kernel-memory quota between protection domains (upgrading the limit
if desired), or
* Free up kernel memory if possible, e.g., revoke memory delegations, which
can be re-created.
We implemented the steps above inside the NOVA kernel and extended Genode's
core component to handle such OOM situations. All system calls beside the IPC
call/reply may now return an error code upon depletion of the quota. Most of
these system calls can solely be performed by core and are handled inside
core's NOVA-specific platform code.
In the case of IPC call/reply operations, we desired to handle OOM cases
transparently to Genode user-level components. Therefore, each thread in
Genode/NOVA now gets constructed with an OOM IPC portal attached. This portal
is served by the pager thread in core and is traversed on OOM occurrences
during IPC operations. If a pager thread receives such an OOM IPC, it decodes
the involved IPC sender and IPC receiver and locates the appropriate
core-internal paging objects. The currently implemented out-of-memory policy
tries to upgrade the quota. If this is not possible, an attempt to revoke
memory mappings from the OOM-causing protection domain is made. This
implicitly frees-up some kernel memory (e.g., mapping nodes). If none of the
responses suffices, the handler stops the OOM-causing thread and writes a
message to the system log.
The current policy implementation constitutes a rather rough heuristic, which
may not suffice under all circumstances. In the future, we would like to
specify a distinct policy per component, e.g. depending on prior known memory
usage patterns. For example, some components follow well-known usage patterns
and therefore a fixed upper quota limit can be specified. Other components are
highly dynamic and desire quota upgrades on demand. There are many more
combinations imaginable.
Our current plan is to collect more experience over the next months with this
new kernel mechanism. Based on our observations, we may externalize such
policy decisions and possibly make them configurable per component.
The current implementation however, already avoids the situation that the
kernel goes out of service if a single component misbehaves
kernel-memory-wise.
Genode as day-to-day operating system
#####################################
At the beginning of June, Genode reached the probably most symbolic milestone
in the project's history: Norman - one of the core developers - replaced his
Linux-based working environment with a Genode-based system. This system is
composed of the following ingredients:
[image turmvilla_scenario]
The machine used is a Lenovo Thinkpad X201. We settled on this five-year-old
machine for several reasons. First, it is a very solid platform with a nice
form factor. Second, it features Intel's AMT (Active Management Technology),
which is handy to obtain low-level system logs in the case something goes
wrong. Third, refurbished machines of this type can be obtained for as little
as 200 EUR. Finally, an older machine reinforces the need for good performance
of the operating system. So it creates a natural incentive for Norman to find
and address performance bottlenecks.
Our modified version of the NOVA microhypervisor is the used kernel.
The user interface is based on our custom GUI stack including the nitpicker
GUI server as well as the window manager and its companion components
(decorator, layouter, pointer) we introduced in
[http://genode.org/documentation/release-notes/14.08#New_GUI_architecture - version 14.08].
The display is driven by the VESA driver. User input is handled by the PS/2
driver for handling the laptop keyboard and trackpoint, and the USB driver for
handling an externally connected keyboard and mouse.
Network connectivity is provided by our port of the Intel Wireless stack that
we introduced with the version
[http://genode.org/documentation/release-notes/14.11#Intel_wireless_stack - 14.11].
Our custom AHCI driver provides access to the physical hard disk. File-system
access is provided by our
[http://genode.org/documentation/release-notes/14.02#NetBSD_file_systems_using_rump_kernels - Rump-kernel-based file-system server].
A simple Genode shell called CLI monitor allows the user to start and kill
subsystems dynamically. Initially, the two most important subsystems are
VirtualBox and Noux.
VirtualBox executes a GNU/Linux-based guest OS that we refer to as "rich OS".
The rich OS serves as a migration path from GNU/Linux to Genode. It is used
for all tasks that cannot be accomplished directly on Genode yet. At the
beginning of the transition, the daily routine still very much depends on the
rich OS. By moving more and more functionality over to the Genode world, we
will eventually be able to make the rich OS obsolete step by step. Thanks to
VirtualBox' excellent host-guest-integration features, the VirtualBox window
can be dynamically resized and the guest mouse cursor integrates seamlessly
with Genode's pointer. VirtualBox is directly connected to the wireless
network driver. So common applications like Firefox can be used.
The noux runtime allows us to use command-line-based GNU software directly on
Genode. Coreutils and Bash are used for managing files. Vim is used for
editing files. Unlike the rich OS, the noux environment has access to the
Genode partition of the hard disk. In particular, it can be used to update the
Genode system. It has access to a number of pseudo files that contain status
information of the underlying components, e.g., the list of wireless access
points. Furthermore, it has limited access to the configuration interfaces of
the base components. For example, it can point the wireless driver to the
access point to use, or change the configuration of the nitpicker GUI server
at runtime.
As a bridge between the rich OS and the Genode world, we combine VirtualBox'
shared-folder mechanism with Genode's VFS infrastructure. The shared folder is
represented by a dedicated instance of a RAM file system, which is mounted in
both the VFS of VirtualBox and the VFS of noux.
As evidenced by Norman's use since June, the described system setup is
sufficient to be productive. So other members of the Genode team plan to
follow in his footsteps soon. At the same time, the continued use of the
system from day to day revealed a number of shortcomings, performance
limitations, and rough edges, which we eventually eliminated. It goes without
saying that this is an ongoing effort. Eating our own dog food forces us to
address the right issues to make the daily life more comfortable.
Feature-wise the switch to Genode motivated three developments, namely the
enhancement of Genode's CLI monitor, the improvement of the window manager,
and the creation of a CPU-load monitoring tool.
Interactive management of subsystem configurations
==================================================
The original version of CLI monitor obtained the configuration data of its
subsystems at start time via the Genode::config mechanism. But for managing
complex scenarios, the config node becomes very complex. Hence, it is
preferable to have a distinct file for each subsystem configuration.
The new version of CLI monitor scans the directory '/subsystems' for files
ending with ".subsystem". Each file has the same syntax as the formerly used
subsystem nodes. This change has the welcome implication that subsystem
configurations can be changed during the runtime of the CLI monitor, e.g., by
using a concurrently running instance of noux with access to the _subsystems/_
directory. This procedure has become an essential part of the daily work flow
as it enables the interactive evolution of the Genode system.
Window-management improvements
==============================
To make the window manager more flexible while reducing its complexity at the
same time, we removed the formerly built-in policy hosting the decorator and
layout components as children of the window manager. Those components are no
longer child components but siblings. The relationship of the components is
now solely expressed by the configuration of their common parent, i.e., init.
This change clears the way to dynamically replace those components during
runtime (e.g., switching between different decorators).
To improve the usability of the windowed GUI, we enabled the layouter to
raise windows on click and to let the keyboard focus follow the pointer.
Furthermore, the window manager, the decorator, and the floating window
layouter became able to propagate the usage of an alpha channel from the
client application to the decorator. This way, the decorator can paint the
decoration elements behind the affected windows, which would otherwise be
skipped. Consequently, partially transparent windows can be properly displayed.
CPU-load monitoring
===================
During daily system use, we started to wish to know in detail where the CPU
cycles are spent. For example, the access of a file by the rich OS involves
several components, including the guest OS itself, VirtualBox, rump_fs (file
system), part_blk (partition access), ahci_drv (SATA device access), core, and
NOVA. Investigating performance issues requires a holistic view of all those
components. For this reason, we enhanced our existing tracing infrastructure
(Section [Enhanced tracing facilities]) to allow the creation of CPU-load
monitoring tools. The first tool in this category is the graphical CPU-load
monitor located at _gems/app/cpu_load_display/_, which displays a timeline of
the CPU load where each thread is depicted with a different color. Thanks to
this tool, we have become able to explore performance issues in an interactive
way. In particular, it helped us to identify and resolve a long-standing
inaccuracy problem in our low-level timer service.
Base framework and low-level OS infrastructure
##############################################
Improved audio support
======================
In the previous release, we replaced our old audio driver with a new one that
provided the same audio-out session interface. Complementing the audio-out
session, we are now introducing a new audio-in session interface that can be
used to record audio frames. It is modeled after the audio-out interface in
the way how it handles the communication between the client and the server. It
uses shared memory in the form of the Audio_in::Stream to transport the frames
between the components. A server component captures frames and puts them into
a packet queue, which is embedded in the Audio_in::Stream. The server
allocates packets from this queue to store the recorded audio frames. If the
queue is already full, the server will override already allocated packets and
will notify the client by submitting an 'overrun' signal. The client has to
cope with this situation, e.g., by consuming packets more frequently. A client
can install a signal handler to respond to a progress signal, which is sent by
the server when a new Audio_in::Packet has been submitted to the packet queue.
For now, all audio-in server components only support one channel (left)
although the audio-in session interface principally supports multiple
channels.
The _dde_bsd_ audio_drv is the first and currently only audio driver component
that was extended to provide the audio-in session. To express this fact, the
driver was renamed from _audio_out_drv_ to _audio_drv_. In contrast to its
playback functionality, which is enabled by default, recording has to be
enabled explicitly by setting the configuration attribute 'recording' to
'yes'. If the need arises, playback may be disabled by setting 'playback' to
'no'. In addition, it is now possible to configure the driver by adjusting the
mixer in the driver's configuration node. For the time being, the interface as
employed by the original OpenBSD mixer utility is used.
The following snippet shows how to enable and configure recording on a
Thinkpad X220 where the headset instead of the internal microphone is used as
source:
! <start name="audio_drv">
! <resource name="RAM" quantum="8M"/>
! <provides>
! <service name="Audio_out"/>
! <service name="Audio_in"/>
! </provides>
! <config recording="yes">
! <mixer field="outputs.master" value="255"/>
! <mixer field="record.adc-0:1_source" value="sel2"/>
! <mixer field="record.adc-0:1" value="255"/>
! </config>
! </start>
In addition to selecting the recording source, the playback as well as the
recording volume are raised to the maximum. Information about all available
mixers and settings in general may be obtained by specifying the 'verbose'
attribute in the config node.
The enriched driver is accompanied by a simple monitor application, which
directly plays back all recorded audio frames and shows how to use the
audio-in session. It can be tested by executing the
_repos/dde_bsd/run/audio_in.run_ run script.
There are also changes to the audio-out session itself. The length of a period
was reduced from 2048 to 512 samples to accommodate for a lower latency when
mixing audio-out packets. A method for invalidating all packets in the queue
was also added.
File-system infrastructure
==========================
Unlike traditional operating systems that rely on a global name space for
files, each Genode component has a distinct view on files. Many low-level
components do not even have the notion of files. Whereas traditional operating
systems rely on a virtual file system (VFS) implemented in the OS kernel,
Genode's VFS has the form of a library that can optionally be linked to a
component. The implementation of this library originated from the noux runtime
introduced in version
[http://genode.org/documentation/release-notes/11.02#Noux_-_an_execution_environment_for_the_GNU_userland - 11.02],
and was later integrated into our C runtime in version
[http://genode.org/documentation/release-notes/14.05#Per-process_virtual_file_systems - 14.05].
With the current release, we take the VFS a step further by making it
available to components without a C runtime. Thereby, low-complexity
security-sensitive components such as CLI monitor become able to benefit from
the powerful VFS infrastructure.
The VFS itself received a welcome improvement in the form of private RAM file
systems. A need for process-local storage motivated a conversion of the
existing ram_fs server component to an embeddable VFS file system. This
addition to the set of VFS plugins enables components to use temporary file
systems without relying on the resources of an external component.
Unified networking components
=============================
Having had a good experience with our Block::Driver implementation, which
wraps the block-session interface and takes care of the packet-stream
handling, thus easing the implementation of driver and other block components,
we observed that this approach did not provide enough flexibility for
NIC-session servers. For example, NIC servers are bi-directional and when a
network packet arrives the server has to make sure that there are enough
resources available to dispatch the network packet to the client. This has to
be done because the server must never block, e.g., by waiting for allocations
to succeed or for an empty spot in the packet queue of a client. Therefore,
such a non-blocking NIC server needs to validate all preconditions for
dispatching the packet in advance and, if they cannot be met, drop the network
packet.
In order to implement this kind of behavior, NIC-session servers must have
direct access to the actual NIC session. For this reason, we removed the
Nic::Driver interface from Genode and added a Nic::Session_component that
offers common basic packet-stream-signal dispatch functionality. Servers may
now inherit from this component and implement their own policy.
We adjusted all servers that implement NIC sessions to the new interface
(dde_ipxe, wifi, usb, nic_bridge, OpenVPN, ...), and thereby unified all
networking components within Genode.
Enhanced tracing facilities
===========================
Recent Genode-based system scenarios like the one described in Section
[Genode as day-to-day operating system] consist of dozens of components that
interact with each other. For reasoning about the behaviour of such scenarios
and identifying effective optimization vectors, tools for gathering a holistic
view of the system are highly desired.
With the introduction of our light-weight
[http://genode.org/documentation/release-notes/13.08#Light-weight_event_tracing - event-tracing facility]
in version 13.08, we laid the foundation for such tools. The current release
extends core's TRACE service with the ability to obtain statistics about CPU
utilization. More specifically, it enables clients of core's TRACE service to
obtain the execution times of trace subjects (i.e., threads). The execution
time is delivered as part of the 'Subject_info' structure. In addition to the
execution time, the structure delivers the information about the affinity of
the subject with a physical CPU.
At the current stage, the feature is available solely on NOVA since this is
our kernel of choice for using Genode as our day-to-day OS. On all other base
platforms, the returned execution times are 0. To give a complete picture of
the system's threads, the kernel's idle threads (one per CPU) are featured as
trace subjects as well. Of course, idle threads cannot be traced but their
corresponding trace subjects allow TRACE clients to obtain the idle time of
each CPU.
By obtaining the trace-subject information in periodic intervals, a TRACE
client is able to gather statistics about the CPU utilization attributed to
the individual threads present (or no longer present) in the system. One
instance of such a tool is the new trace-subject reporter located at
_os/src/app/trace_subject_reporter_. It acts as a TRACE client, which delivers
the gathered trace-subject information in the form of XML-formatted data to a
report session. This information, in turn, can be consumed by a separate
component that analyses the data. In contrast to the low-complexity
trace-subject reporter, which requires access to the privileged TRACE services
of core, the (potentially complex) analysing component does not require access
to core's TRACE service. So it isn't as critical as the trace-subject monitor.
The first representative of a consumer of trace-subject reports is the
CPU-load display mentioned in Section [CPU-load monitoring] and depicted in
Figure [nano3d].
In addition to the CPU-monitoring additions, the tracing facilities received
minor refinements. Up to now, it was not possible to trace threads that use a
CPU session other than the component's initial one. A specific example is
VirtualBox, which employs several CPU sessions, one for each priority. This
problem has been solved by associating the event logger of each thread with
its actual CPU session. Consequently, the tracing mechanism has become able to
trace VirtualBox, which is pivotal for our further optimizations.
Low-complexity software rendering functions
===========================================
Our ambition to use Genode as our day-to-day OS raises the need for custom
graphical applications. Granted, it is principally possible to base such
applications on Qt5, which is readily available to native Genode components.
However, for certain applications like status displays, we prefer to avoid the
dependency on an overly complex GUI tool kit. To accommodate such
applications, Genode hosts a small collection of low-complexity graphics
functions called painters. All of Genode's low-complexity graphical components
such as nitpicker, launchpad, window decorator, or the terminal are based on
this infrastructure.
With the current release, we extend the collection with two new painters
located at _gems/include/polygon_gfx_. Both draw convex polygons with an
arbitrary number of points. The shaded-polygon painter interpolates the color
and alpha values whereas the textured-polygon painter applies a texture to the
polygon. The painters are accompanied by simplistic 3D routines located at
_gems/include/nano3d/_ and a corresponding example (_gems/run/nano3d.run_).
[image nano3d]
With the nano3d demo and our new CPU load display, the screenshot above shows
two applications that make use of the new graphics operations.
Device drivers
##############
Completing the transition to the new platform driver
====================================================
Until now, the platform driver on x86-based machines was formed by the ACPI
and PCI drivers. The ACPI driver originally executed the PCI driver as a slave
(child) service. The ACPI driver parsed the ACPI tables and provided the
relevant information as configuration during the PCI-driver startup. We
changed this close coupling to the more modern and commonly used
[http://genode.org/documentation/release-notes/14.02#New_session_interface_for_status_reporting - report_rom mechanism].
When the new ACPI driver finishes the ACPI table parsing, it provides the
information via a report to any interested and registered components. The
report contains among other the IRQ re-routing information. The PCI driver is
a component, which - according to its session routing configuration - plays
the role of a consumer of the ACPI report.
With this change of interaction of ACPI and PCI driver, the policy for devices
must be configured solely at the PCI driver and not at the ACPI driver. The
syntax, however, stayed the same as introduced with release 15.05.
Finally, the PCI driver 'pci_drv' got renamed to 'platform_drv' as already
used on most ARM platforms. All files and session interfaces containing
PCI/pci in the names were renamed to Platform/platform. The x86 platform
interfaces moved to _repos/os/include/platform/x86/_ and the implementation of
the platform driver to _repos/os/src/drivers/platform/x86/_.
An example x86 platform configuration snippet looks like this:
!<start name="acpi_drv" >
! <resource .../>
! <route>
! ...
! <service name="Report"> <child name="acpi_report_rom"/> </service>
! </route>
!</start>
!
!<start name="acpi_report_rom" >
! <binary name="report_rom"/>
! <resource .../>
! <provides> <service name="ROM" /> <service name="Report" /> </provides>
! <config>
! <rom> <policy label="platform_drv -> acpi" report="acpi_drv -> acpi"/> </rom>
! </config>
! <route> ... </route>
!</start>
!
!<start name="platform_drv" >
! <resource name="RAM" quantum="3M" constrain_phys="yes"/>
! <provides> <service name="Platform"/> </provides>
! <route>
! <service name="ROM">
! <if-arg key="label" value="acpi"/> <child name="acpi_report_rom"/>
! </service>
! ...
! </route>
! <config>
! <policy label="ps2_drv"> <device name="PS2"/> </policy>
! <policy label="nic_drv"> <pci class="ETHERNET"/> </policy>
! <policy label="fb_drv"> <pci class="VGA"/> </policy>
! <policy label="wifi_drv"> <pci class="WIFI"/> </policy>
! <policy label="usb_drv"> <pci class="USB"/> </policy>
! <policy label="ahci_drv"> <pci class="AHCI"/> </policy>
! <policy label="audio_drv"> <pci class="AUDIO"/> <pci class="HDAUDIO"/> </policy>
! </config>
!</start>
In order to unify and simplify the writing of run scripts, we added the
commonly used platform configuration to the file
_repos/base/run/platform_drv.inc_. This file may be included by any test run
script in order to setup a default platform driver configuration.
In addition, the snippet provides the following functions:
'append_platform_drv_build_components', 'append_platform_drv_config' and
'append_platform_drv_boot_modules'. The functions add necessary information to
the 'build_components', 'config' and 'boot_modules' run variables. The
_platform_drv.inc_ also contains the distinction between various ARM/x86
platforms and includes the necessary pieces. Hence, run scripts are largely
relieved from platform-specific peculiarities.
The body of an example run script looks like this:
! set build_components { ... }
!
! source ${genode_dir}/repos/base/run/platform_drv.inc
! append_platform_drv_build_components
!
! build $build_components
!
! create_boot_directory
!
! set config { ... }
!
! append_platform_drv_config
!
! append config { ... }
!
! install_config $config
!
! append_platform_drv_boot_modules
!
! build_boot_image $boot_modules
!
! run_genode_until ...
BCM57cxx network cards
======================
During Hack'n Hike 2015, we had access to a server that featured a Broadcom
network card. Therefore Guido Witmond performed the first steps to enable
Broadcom's BCM 57cxx cards. With this preliminary work in place, we were
quickly able to perform the additional steps required to add BCM 57cxx support
to Genode.
VESA driver refinements
=======================
The VESA driver now reports the frame buffer's line width instead of the
visible width to the client. This fixes a possible distortion if these widths
differ, at the cost that content in the right-most area might be invisible in
such cases.
VirtualBox
##########
Policy-based mouse pointer
==========================
In the previous release, we implemented support for the transparent
integration of the guest mouse pointer with nitpicker via the VirtualBox guest
additions and the vbox_pointer component, which is capable of rendering
guest-provided mouse-pointer shapes. Now, we extended vbox_pointer by a
policy-based configuration that allows the selection of ROMs containing the
actual mouse shape based on the nitpicker session label or domain. With this
feature in place, it is possible to integrate several VirtualBox instances as
well as dedicated pointer shapes for specific components. To see the improved
vbox_pointer in action give _run/vbox_pointer_ a shot.
Dynamic adaptation to screen size changes
=========================================
VirtualBox now notifies the guest operating system about screen-size changes
(for example if the user resizes a window, which shows the guest frame
buffer). The VirtualBox guest additions can use this information to adapt the
guest frame buffer to the new size.
SMP support
===========
Guest operating systems can now use multiple virtual CPUs, which are mapped to
multiple host CPUs. The number of virtual CPUs can be configured in the
'.vbox' file.
Preliminary audio support
=========================
At some point, the use of VirtualBox as a stop-gap solution for using Genode
as everyday OS raises the need to handle audio. With this release, we address
this matter by enabling preliminary audio support in our VirtualBox port. A
back end that uses the audio-out and audio-in sessions to playback and record
sound samples has been added. It disguises itself as the OSS back end that is
already used by vanilla VirtualBox. Since Genode pretends to be FreeBSD in the
eyes of VirtualBox (because Genode's libc is based on FreeBSD's libc), the
provisioning of an implementation of the OSS back end as used on FreeBSD host
systems is the most natural approach. The audio support is complemented by
adding the necessary device models for the virtual HDA as well as the AC97
devices to our VirtualBox port.
For now, it is vital to have the guest OS configure the virtual device in a
way that considers the current implementation. For example, we cannot
guarantee distortion-free playback or recording if the guest OS uses a period
that is too short, typically 10ms or less. There are also remaining issues
with the mixing/filtering code in VirtualBox. Therefore, we bypass it to
achieve better audio quality. As a consequence, the device model of the VM has
to use the same sample rate as is used by the audio-out and audio-in sessions
(44.1kHz).
Enabling audio support is done be adding
! <AudioAdapter controller="HDA" driver="OSS" enabled="true"/>
to the .vbox file manually or configuring the VM accordingly by using the GUI.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Bender chain loader on base-hw x86_64
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Intel platforms, we use the Bender chain loader from the
[https://github.com/alex-ab/morbo - Morbo multiboot suite] to detect available
COM ports of PCI plug-in cards, the AMT SOL device, or as fall back the
default comport 1. The loader stores the I/O port information of the detected
cards into the BIOS data area (BDA), from where it is retrieved by core on
boot and subsequently used for logging. With this release, we added the BDA
parsing to base-hw on x86-64 and enabled the feature in the run tool. As a
prerequisite, we had to fix an issue in bender triggered by the loading of
only one (large) multi-boot kernel. Consequently, its binary in
_tool/boot/bender_ was updated.
Revised page-table handling
~~~~~~~~~~~~~~~~~~~~~~~~~~~
One of the main advantages of the base-hw platform is that the memory trading
concept of Genode is universally applied even with regard to kernel objects.
For instance, whenever a component wants to create a thread, it pays for the
thread's stack, UTCB, and for the corresponding kernel object. The same
applies to objects needed to manage the virtual address space of a component
with the single exception of page tables.
Normally, when the quota, which was donated by a component to a specific
service, runs out, the component receives an exception the next time it tries
to invoke the service. The component can respond by upgrading the respective
session quota. However, in the context of page-fault resolution, this is
particularly difficult to do. The allocation and thereby the shortage of
memory becomes evident only when the client produces a page fault. Therefore,
there is no way to inform the component to upgrade its session quota before
resolving the fault.
Instead of designing a sophisticated protocol between core and the other
components to solve this problem, we decided to simplify the current
page-fault resolution by using a static set of page-tables per component.
Formerly, page tables were dynamically allocated from core's memory allocator.
Now, an array of page tables gets allocated during construction of a
protection domain. When a component runs out of page tables, all of its
mappings get flushed, and the page tables are populated from scratch. This
change greatly simplifies the page-table handling inside of base-hw.
Dynamic interrupt mode setting on x86_64
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On x86-based hardware, user-level device drivers have become able to specify
the trigger mode and polarity of the interrupts when requesting an IRQ
session. On ARM, those session parameters are ignored. This change enables the
x86_64 platform to support devices, which use arbitrary trigger modes and
polarity settings, e.g., AHCI on QEMU and real hardware.
Fiasco.OC
=========
Genode's device-driver support when using the Fiasco.OC kernel as base
platform received an upgrade.
First, principle support for the Raspberry Pi was added. To make this platform
useful in practice, a working USB driver is important. I.e., the network
interface is connected via USB. Hence the USB driver got enabled for
Fiasco.OC, too. As a result, Genode's software stack can now be used on the
Raspberry Pi by using either our custom base-hw kernel or Fiasco.OC.
Second, support for the Odroid-X2 platform using the Exynos4412 SoC was added,
which includes the drivers for clock management (CMU), power management
(PMU) as well as USB.
Thanks to Reinier Millo Sánchez and Alexy Gallardo Segura for having
contributed this line of work.
Removal of deprecated features
##############################
We dropped the support for the *ARM Versatile Express* board from the Genode
source tree to relieve our automated testing infrastructure from supporting a
platform that remained unused for more than two years.
The device driver environment kit (DDE Kit) was originally intended as a
common API among the execution environments of ported user-level device
drivers. However, over the course of the past years, we found that this
approach could not fulfill its promise while introducing a number of new
problems. We reported our experiences in the release notes of versions
[http://genode.org/documentation/release-notes/12.05#Re-approaching_the_Linux_device-driver_environment - 12.05] and
[http://genode.org/documentation/release-notes/14.11#Roundup - 14.11].
To be able to remove the DDE-Kit API, we reworked the USB driver, our port of
the Linux TCP/IP stack, and the wireless driver accordingly.

1429
doc/release_notes-15-11.txt
View File

File diff suppressed because it is too large Load Diff

652
doc/release_notes-16-02.txt
View File

@@ -1,652 +0,0 @@
===============================================
Release notes for the Genode OS Framework 16.02
===============================================
Genode Labs
With version 16.02, we add RISC-V to Genode's supported CPU architectures,
enable the secure pass-through of individual USB devices to virtual machines,
and update the support for the Muen and seL4 kernels.
Trustworthy hardware becomes an increasingly pressing problem. With each new
generation of today's commodity hardware comes a dramatic increase of
complexity, the addition of proprietary companion processors, and opaque
firmware blobs. Even with a perfectly secure operating system, the user's
privacy and security remains at risk as there is no way to assess the
trustworthiness of our underlying hardware. RISC-V is a new hardware
architecture that tries to overcome this problem by the means of open source
and transparency. It is designed to scale from micro controllers to
general-purpose computers, and to be both synthesizable as FPGA softcores and
implementable in ASICs. The prospect of a scalable and trustworthy open-source
hardware platform motivated us to add RISC-V to Genode's supported CPU
architectures. Section [New support for the RISC-V CPU architecture] gives a
brief overview of this line of work.
Thanks to the growing number of our regular developers using Genode as day to
day OS, we create a natural incentive to address typical desktop-OS work
flows. In particular, the new version comes with the ability to assign
individual USB devices to VirtualBox instances. Conceptually, this looks like
a relatively straight-forward feature. But as discussed in Section
[Assignment of USB devices to virtual machines], we had to overcome a number of
challenging problems caused by the inherently dynamic nature of USB-device
hot-plugging. Also on the account of day-to-day computing, the GUI stack
received welcomed usability improvements like keyboard shortcuts for certain
window-management operations.
With respect to Genode's underlying base platforms, we are happy to announce
the updates of the Muen and seL4 kernels. The Muen separation kernel received
an update to version 0.7, which accommodates Genode's regular work flows (via
run scripts) much better than the previous version. As described in Section
[Muen separation kernel], this change clears the way to subject Muen to
Genode's regular automated tests. The seL4 kernel represents an exciting
playground as a future base platform for Genode. We have updated the kernel to
version 2.1, which prompted us to fundamentally revisit the low-level resource
management of Genode on this kernel. A summary of this undertaking is presented
in Section [seL4 version 2.1].
According to the [http:/about/road-map - road map], we originally planned to
revise the framework API in this release. Even though this topic is
[https://github.com/genodelabs/genode/issues/1832 - very actively pursued], we
decided to not rush it. We find it important to provide a smooth migration path
from the old API to the new one. Determining the best path is actually trickier
than revising the API, though. To let our decisions settle a bit, we postpone
the transition to the upcoming release.
Assignment of USB devices to virtual machines
#############################################
As a migration strategy for running Genode on a daily basis, using VirtualBox
to execute a feature-rich OS is vital. In release
[http://genode.org/documentation/release-notes/15.05#USB-device_pass-through_support - 15.05],
we added USB pass-through support to VirtualBox by enabling its integrated USB
proxy service. Since we use the open-source edition of VirtualBox, we were
merely able to use the OHCI device model and were therefore limited to using
USB 1.x devices in low and full speed mode only. To make matters worse, when
using the OHCI controller model, it is difficult if not impossible to access
USB mass-storage devices. Usually, VirtualBox facilitates the EHCI or xHCI
device models for the pass-through of storage devices. Unfortunately, those
models are only available as a proprietary extension, which cannot be used by
our VirtualBox port.
Having support for the pass-through of high-speed and super-speed USB devices
is a must in such controller models. Therefore, we either have to implement
these models ourselves or port existing ones from another VMM or emulator to
fill the gap. We went for porting existing models first because device-model
development from scratch could end up being time consuming if we want to
guarantee them to work with a variety of different OS drivers.
QEMU xHCI device model
----------------------
QEMU features a NEC xHCI (UPD720200) device model that works well with Windows
guests. For this reason, we decided to give porting this device model a shot.
We applied the DDE approach and started by creating a QEMU emulation
environment so that only the bare minimum amount of source code needed to be
taken from the QEMU sources. It came down to a handful of source files, mainly
the USB core and the xHCI device model files. We iteratively extended the
emulation environment until the QEMU sources compiled and linked fine. One
particular cumbersome issue we had to overcome was the emulation of the QEMU
Object Model. Since QEMU is written in C, it uses its own object model to
implement inheritance. This object model is used throughout QEMU. We took the
easy way out and just used a C++ wrapper class that contains all QEMU objects
that are used in the USB subsystem.
The next step was to develop a USB host device model. This model connects a
USB device attached to Genode's USB host-controller driver to the xHCI device
model. Lucky for us, QEMU already contains a USB host device model that uses
libusb, which we could use as blueprint. We implemented a USB host device that
leverages Genode's custom USB session interface. This host device reacts to a
USB device report coming from another component such as the host-controller
driver. It tries to claim all devices it finds in that report and then creates
a QEMU USB device for each of them that is attached to the xHCI device model.
The xHCI device model needs infrastructure that normally is provided by QEMU
itself such as a timer queue and PCI device handling. We introduced a QEMU
USB controller interface _repos/libports/include/qemu/usb.h_ whose back-end
library interface has to be implemented by a component, i.e. the VMM, that
wants to use the library.
In the end, this work resulted in a small library that contains the xHCI
device model and works in a standalone way. All required resources have to be
provided by the component using the library. This makes it easy to integrate
the library in different VMMs because the user of the library is not forced to
employ the library in a certain way but free to use it any way he chooses.
xHCI device model wrapper in VirtualBox
---------------------------------------
We implemented an xHCI device model _repos/port/src/virtualbox/devxhci.cc_ in
VirtualBox that merely wraps the QEMU USB library and provides the back-end
functionality required by the library to glue QEMU's xHCI device model to
VirtualBox. For now, this device is always part of a VM because there is
currently no way to disable it from within the VirtualBox configuration
front end. Therefore, it is necessary to always give VirtualBox access to a
_usb_devices_ ROM module.
We removed the afore mentioned USB proxy service from our VirtualBox port
because it became redundant with the advent of our xHCI device model.
USB device report filter
------------------------
With the xHCI support in VirtualBox in place, we had to come up with a
mechanism to select, which USB devices it may access. Since USB devices are
usually hot-plugged by the user of the system, we need to be able to configure
the access permissions dynamically at run-time. On this account, we created a
component that intercepts the report from the USB host-controller driver. On
the one hand, this USB device report-filter component screens the device
report coming from the USB host-controller driver by checking each reported
device against a given white list of devices. Only approved devices are
reported to a consumer of the report, i.e. VirtualBox. On the other hand, this
component generates a new configuration for the USB host-controller driver.
The configuration has to be changed each time the filter component finds a
suitable device because the driver will hand out access to a given device to a
client only if there is a valid policy. As we do not know in advance, which
devices might be plugged in, this policy must be maintained dynamically. The
report filter will send the device report only if the host-controller driver
has changed its configuration. This ensures that a matching policy will be in
effect at the time when the client component tries to access the device.
The configuration of the report-filter component can also be changed at run
time.
See _repos/os/src/app/usb_report_filter/README_ for more details on how the
USB device report filter may be configured.
Example configuration
---------------------
The following figure illustrates the interplay and configuration of the
involved components:
[image qemu_xhci]
When the user plugs in a USB device, the USB host-controller driver generates
a device report that is consumed by the USB device report-filter component
(1). The filter component then examines the report and checks if it contains a
device it should report to its report consumer. It then reconfigures the
host-controller driver (2). Afterwards it sends a report to its consumer (3).
The consumer, in this case a VMM, then accesses the USB device (4).
New support for the RISC-V CPU architecture
###########################################
We became aware of [http://riscv.org - RISC-V] when attending several talks
about the project at [https://fosdem.org - FOSDEM] in 2015. RISC-V aims to be
an open-source hardware architecture and is now complemented by many projects
that target the release of real hardware or ASICs (for example,
[http://lowrisc.org - the LowRISC project]). We have experience with various
major CPU architectures and many systems on a chip and, therefore, embrace a
sharp eye on certain platform properties. Intel's ME and ARM's Trustzone
practically lock out operating systems of certain hardware and firmware
features. The true nature of these mechanisms becomes increasingly dubious,
especially when trying to build a secure open-source operating system. Intel's
AMT technology for instance comes with a complete TCP/IP stack that intercepts
packets from the integrated NIC and a VNC server that can magically expose a
mouse and a keyboard at the USB controller. If you are interested in more
details about this topic
[http://blog.invisiblethings.org/papers/2015/x86_harmful.pdf - Intel x86 considered harmful]
by Joanna Rutkowska is a very good read. We decided to have a deeper look at
the RISC-V architecture as an alternative open hardware platform. Especially,
since the LowRISC project promises a completely open system on chip, including
the peripherals.
RISC-V comes with a lot of optional features, so it can cover a large field of
applications reaching from simple I/O processors to general-purpose computing.
For example, there are 64 and 32 bit ISA (instruction set architecture)
versions, three page table formats with the option to omit paging at all, up
to four privilege modes, and a minimal integer core ISA (I). Everything else,
like multiplication and division (M), atomic instructions (A), and floating
point support (F) are subject to ISA extensions and are completely optional
for a specific hardware implementation.
For Genode, we chose to add the RISC-V support to our custom _base-hw_ kernel.
Since Genode may be used as a general purpose OS, we implemented the kernel
using the 64 bit RISC-V version, the Sv39 three-level page table format, and
the so-called general-purpose extension (G), which is the abbreviation for the
IAMF extensions. The current implementation provides the kernel and the
necessary adaptations of the user level part of core.
For testing, we used the RISC-V instruction emulator called
[https://github.com/riscv/riscv-isa-sim - Spike]. There also exists a RISC-V
implementation for various Zynq FPGAs. Genode's Zynq board support has kindly
been added and contributed by Mark Vels.
In the current state, basic Genode applications including core, init, and
components that use shared libraries can be executed on top of our RISC-V
port. We did not enable the libc and postponed further activity as the
platform currently does not specify the interaction with peripherals.
Steps to test Genode on RISC-V
------------------------------
# Building the instruction emulator
! # download the front end server
! git clone https://github.com/ssumpf/riscv-fesvr.git
!
! # build the front end server
! cd riscv-fesvr
! mkdir build
! cd build
! export RISCV=<installation path>
! ../configure --prefix=$RISCV
! (sudo) make install
!
! # download the instruction emulator
! cd ../../
! git clone https://github.com/ssumpf/riscv-isa-sim.git
! cd riscv-isa-sim
!
! # build the emulator
! mkdir build
! cd build
! ../configure --prefix=$RISCV --with-fesvr=$RISCV
! (sudo) make install
!
! # add $RISCV/bin to path
! export PATH=$RISCV/bin:$PATH
# Building Genode and running a test scenario
! # download Genode
! cd ../../
! git clone https://github.com/genodelabs/genode.git
!
! # build the Genode tool chain
! cd genode
! ./tool/tool_chain riscv
!
! # create RISC-V build directory
! ./tool/create_builddir hw_riscv
! cd build/hw_riscv
!
! # build and execute the printf run script
! make run/printf
GUI stack usability improvements
################################
Motivated by the daily use of Genode as desktop OS by an increasingly number
of developers, the window-layouter component of the
[http://genode.org/documentation/release-notes/15.11#GUI_stack - GUI stack]
received welcomed usability improvements.
Configurable window placement
-----------------------------
The policy of the window layouter can be adjusted via its configuration. For
a given window label, the window's initial position and its maximized state
can be defined as follows:
! <config>
! <policy label="mupdf" maximized="yes"/>
! <policy label="nit_fb" xpos="50" ypos="50"/>
! </config>
Keyboard shortcuts
------------------
The window layouter has become able to respond to key sequences. However,
normally, the layouter is not a regular nitpicker client but receives only
those input events that refer to the window decorations. It never owns the
keyboard focus. In order to propagate global key sequences to the layouter,
nitpicker must be explicitly configured to direct key sequences initiated with
certain keys to the decorator. For example, the following nitpicker
configuration routes key sequences starting with the left windows key to the
decorator. The window manager, in turn, forwards those events to the layouter.
! <start name="nitpicker">
! ...
! <config>
! ...
! <global-key name="KEY_LEFTMETA" label="wm -> decorator" />
! ...
! </config>
! ...
! </start>
The response of the window layouter to key sequences can be expressed in the
layouter configuration as follows:
! <config>
! <press key="KEY_LEFTMETA">
! <press key="KEY_TAB" action="next_window">
! <release key="KEY_TAB">
! <release key="KEY_LEFTMETA" action="raise_window"/>
! </release>
! </press>
! <press key="KEY_LEFTSHIFT">
! <press key="KEY_TAB" action="prev_window">
! <release key="KEY_TAB">
! <release key="KEY_LEFTMETA" action="raise_window"/>
! </release>
! </press>
! </press>
! <press key="KEY_ENTER" action="toggle_fullscreen"/>
! </press>
! </config>
Each '<press>' node defines the policy when the specified 'key' is pressed.
It can be equipped with an 'action' attribute that triggers a window action.
The supported window actions are:
:next_window: Focus the next window in the focus history.
:prev_window: Focus the previous window in the focus history.
:raise_window: Bring the focused window to the front.
:toggle_fullscreen: Maximize/unmaximize the focused window.
By nesting '<press>' nodes, actions can be tied to key sequences. In the
example above, the 'next_window' action is executed only if TAB is pressed
while the left windows-key is kept pressed. Furthermore, key sequences can
contain specific release events. In the example above, the release of the left
windows key brings the focused window to front, but only if TAB was pressed
before.
Device drivers
##############
USB host-controller driver enhancements
=======================================
The _usb_drv_ component now solely uses a policy to grant other components
access to USB devices exposed by its raw interface (USB session). On the basis
of the 'label' attribute, it will choose a pre-configured device that is
identified by either the 'bus' and 'dev' or the 'vendor' and 'product'
attribute tuple. To accommodate policy decisions made at run time, the USB
driver is now able to reload its configuration on demand. The USB device
report now contains a 'bus' and a 'dev' attribute as well in order to identify
a USB device more precisely. In addition to that, there is also a generated
'label' attribute in form of 'usb-<bus>-<dev>' that may be used to form
policies while configuring the system dynamically, e.g., when using the
_usb_report_filter_ component.
USB mass-storage driver
=======================
Up to now, access to USB storage devices was provided by the USB
host-controller driver only. However, its ability to do so is limited. E.g.,
it only supports one storage device and the storage device cannot be changed
at run-time. With this release we add a USB mass-storage driver that supports
UMS bulk-only devices that use the SCSI Block Commands set (direct-access).
This is still most common for USB sticks. Devices using different command
sets, e.g SD/HC devices or some external disc drives, will not work properly
if at all. The driver uses the USB session interface to access the USB device
and provides its service as block session to its client.
This component is part of the first step providing the ability to mount and
use USB sticks dynamically when using Genode as a general purpose OS. In the
future, the _usb_drv_ component should solely be the host-controller driver
while other tasks are handled by dedicated USB driver components such as this
one.
Audio output on Linux
=====================
The audio-out driver for Linux was modernized by replacing its multi-threaded
architecture by an event-driven architecture using Genode's server API. In
addition, the playback is now driven by a timer. For now it is a periodic
timer that triggers every 11 ms which is roughly the current audio-out period.
The driver now also behaves like the other BSD-based audio-out driver, i.e.,
it always advances the play pointer. That is vital for the audio-out stack
above the driver to work properly (e.g., the mixer).
Libraries and applications
##########################
New Genode-world repository
===========================
With a growing number of users and contributors comes the desire to bring more
and more existing software to Genode. Most of such libraries and applications,
however, are outside of the scope of Genode as an OS framework. In contrast to
device drivers, protocol stacks, and low-level OS services, which we subject
to our regular automated tests, most 3rd-party software is pretty independent
from Genode. The attempt to integrate the growing pool of such diverse
software into the main repository does not scale.
For this reason, we introduce the new
[https://github.com/genodelabs/genode-world - Genode World] repository, which
is the designated place for hosting ported applications, libraries, and games.
To use it, you first need to obtain a clone of Genode:
! git clone https://github.com/genodelabs/genode.git genode
Now, clone the _genode-world.git_ repository to _genode/repos/world:_
! git clone https://github.com/genodelabs/genode-world.git genode/repos/world
By placing the _world_ repository under the _repos/_ directory, Genode's tools
will automatically incorporate the ports provided by the _world_ repository.
For building software of the _world_ repository, the build-directory
configuration _etc/build.conf_ must be extended with the following line:
! REPOSITORIES += $(GENODE_DIR)/repos/world
*Word of caution*
In contrast to the components found in the mainline Genode repository, the
components within the _world_ repository are not subjected to the regular
quality-assurance measures of Genode Labs. Hence, problems are to be expected.
If you encounter bugs, build problems, or stability issues, please report them
to the [https://github.com/genodelabs/genode-world/issues - issue tracker] or
the [http://genode.org/community/mailing-lists - mailing list].
Updated 3rd-party software
==========================
The following 3rd-party code packages of the _ports_ and _libports_
repositories have been ported or updated:
* Lynx 2.8.8rel.2 (noux package)
* OpenSSH 7.1p1 (noux package)
* tar-1.27 (noux package)
* libssh 0.7.2
* Lighttpd 1.4.38
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Within the last months, the initialization code of our custom kernel got
re-arranged to simplify the addition of new architectures, e.g., the RISC-V
port (Section [New support for the RISC-V CPU architecture]) while also making
its implementation leaner. A positive side effect of this work was the
generalization of multi-processor and L2-cache support for ARM's Cortex-A9
CPUs. For instance, the Wandboard (Freescale i.MX6 SoC) is now driven with all
four cores, and its memory can be accessed with full speed.
Besides those feature additions, we fixed an extremely rare and tricky race
condition in the implementation of the kernel-protected capabilities,
introduced in release 15.05. A capability's lifetime within a component is
tracked by a reference-counting like mechanism that is under control of the
component itself. When the kernel transfered a capability to a component, and
the very same capability was deleted within the component simultaneously, the
received capability was marked as invalid, which led to diverse, sporadic
faults. This deficit in the capabilities reference-counting is solved with the
current release.
Muen separation kernel
======================
Build integration
-----------------
Building Genode scenarios running on top of the
[http://muen.sk - Muen separation kernel] has been greatly simplified by
properly integrating the Muen system build process into the Genode build system.
As described in the
[http://genode.org/documentation/release-notes/15.08#Genode_on_top_of_the_Muen_Separation_Kernel - 15.08 release notes],
the architecture with Muen is different since the entire hw_x86_64_muen Genode
system runs as a guest VM on top of the separation kernel. This means that the
Genode base-hw image must itself be packaged into the final Muen system image
as an additional step after the Genode system build.
The packaging process of a Muen system image is performed by the new
_image/muen_ run-tool plugin, which processes the following RUN_OPT parameters.
:--image-muen-external-build:
Muen system is built automatically or externally
:--image-muen-system:
Muen system policy
:--image-muen-components:
Muen system components required for the given system policy
:--image-muen-hardware:
Muen target hardware platform
:--image-muen-gnat-path:
Path to GNAT toolchain
:--image-muen-spark-path:
Path to SPARK toolchain
The options are automatically added to the _etc/build.conf_ file for the
hw_x86_64_muen base-hw platform. The
[http://genode.org/documentation/platforms/muen - documentation] has been
updated to reflect the new, simplified build process.
A port file was added to facilitate the download of the Muen sources v0.7 and
to check the required dependencies.
Using the new _image/muen_ script in combination with iPXE allows to run the
Genode test suite via the autopilot tool.
MSI support
-----------
Muen employs Intel VT-d interrupt remapping (IR) besides DMA remapping for
secure device assignment. As a consequence, PCI devices using Message Signaled
Interrupts (MSI) must be programmed to trigger requests in remappable format
(see Intel VT-d specification, Section 5.1.2.2 for further details).
To enable the use of MSIs with the base-hw kernel, a platform-specific
function has been introduced that returns the necessary MSI parameters for a
given PCI device. If either the platform or the specific device does not
support MSI, the function returns false.
On hw_x86_64_muen, the function consults the Muen subject info page to supply
the appropriate information to the IRQ session. This allows Genode device
drivers to transparently use MSIs for passed-through PCI devices.
seL4 version 2.1
================
By the end of 2015, the [http://sel4.systems/ - seL4 kernel] version 2.0 was
published. With the current release, we update Genode's preliminary support
for this kernel from the experimental branch of one year ago to the master
branch of version 2.1. Note that this line of work is still considered as an
exploration. As of now, there is still a way to go until we can leverage seL4
as a fully featured base platform. Under the hood of Genode, the transition to
the version 2.1 master branch had the following implications.
In contrast to the experimental branch, the seL4 master branch has no way to
manually define the allocation of kernel objects within untyped memory ranges.
Instead, the kernel maintains a built-in allocation policy. This policy rules
out the deallocation of once-used parts of untyped memory. The only way to
reuse memory is to revoke the entire untyped memory range. Consequently, we
cannot share a large untyped memory range for kernel objects of different
protection domains. In order to reuse memory at a reasonably fine granularity,
we need to split the initial untyped memory ranges into small chunks that can
be individually revoked. Those chunks are called "untyped pages". An untyped
page is a 4 KiB untyped memory region.
The bootstrapping of core has to employ a two-stage allocation approach now.
For creating the initial kernel objects for core, which remain static during
the entire lifetime of the system, kernel objects are created directly out of
the initial untyped memory regions as reported by the kernel. The so-called
"initial untyped pool" keeps track of the consumption of those untyped memory
ranges by mimicking the kernel's internal allocation policy. Kernel objects
created this way can be of any size. For example the CNode, which is used to
store page-frame capabilities is 16 MiB in size. Also, core's CSpace uses a
relatively large CNode.
After the initial setup phase, all remaining untyped memory is turned into
untyped pages. From this point on, newly created kernel objects cannot exceed
4 KiB in size because one kernel object cannot span multiple untyped memory
regions. The capability selectors for untyped pages are organized similarly to
those of page-frame capabilities. There is a new 2nd-level CNode
(UNTYPED_CORE_CNODE) that is dimensioned according to the maximum amount of
physical memory (1M entries, each entry representing 4 KiB). The CNode is
organized such that an index into the CNode directly corresponds to the
physical frame number of the underlying memory. This way, we can easily
determine an untyped page selector for any physical addresses, i.e., for
revoking the kernel objects allocated at a specific physical page. The
downside is the need for another 16 MiB chunk of meta data. Also, we need to
keep in mind that this approach won't scale to 64-bit systems. We will
eventually need to replace the PHYS_CORE_CNODE and UNTYPED_CORE_CNODE by CNode
hierarchies to model a sparsely populated CNode. The following figure
illustrates the layout of core's capability space.
[image sel4_core_cspace_master]
Organization of core's capability space on seL4
For each protection domain, core maintains a so-called VM CSpace that holds
capability selectors for page frames and page tables. The size constraint of
kernel objects has the immediate implication that the VM CSpaces of protection
domains must be organized via several levels of CNodes. I.e., as the top-level
CNode of core has a size of 2^12, the remaining 20 PD-specific CSpace address
bits are organized as a 2nd-level 2^4 padding CNode, a 3rd-level 2^8 CNode,
and several 4th-level 2^8 leaf CNodes. The latter contain the actual selectors
for the page tables and page-table entries of the respective PD.
As another slight difference from the experimental branch, the master branch
requires the explicit assignment of page directories to an ASID pool.
Functionality-wise the update to version 2.1 brings no changes. The
preliminary support is still limited to Genode's most fundamental mechanisms
like the bootstrapping, the creation of protection domains, the execution of
threads, and inter-component communication. User-level device drivers are not
supported yet. Such functional improvements are scheduled for Genode 16.08.
Linux
=====
We started to experience crashes of our dynamic linker (ldso) when using
Genode's _base-linux_ platform on recent Linux kernels. Ldso is primarily a
shared object, which is linked to dynamic binaries. But ldso is also an
executable, which, once started loads the dynamically-linked binary along with
all shared libraries required by the binary. Up to now, ldso had to be loaded
at a link address defined at compilation time, which we enforced through
linker-script magic. Unfortunately, this does not work any longer on recent
Linux versions. The kernel notices that ldso is a shared object and loads it
at an arbitrary (randomized) address, which ultimately results in a
segmentation fault during ldso initialization. We found a fix for this issue
by marking ldso as an executable in the ELF header. But since ldso is linked
to all dynamic binaries (it contains Genode's base libraries) the GNU linker
then refused to link because ldso was not marked as a shared object.
Therefore, we decided to implement true self relocation within ldso. This
feature only works on Genode's base-linux platform as it requires some
symbol-address magic.

1315
doc/release_notes-16-05.txt
View File

File diff suppressed because it is too large Load Diff

1126
doc/release_notes-16-08.txt
View File

File diff suppressed because it is too large Load Diff

729
doc/release_notes-16-11.txt
View File

@@ -1,729 +0,0 @@
===============================================
Release notes for the Genode OS Framework 16.11
===============================================
Genode Labs
In contrast to most parts of the framework, the fundamental low-level
protocols, which define the interaction between parent and child components
have remained unchanged since the very first Genode version. From this
interplay, the entire architecture follows. That said, certain initial design
choices were not perfect. They partially resulted from limitations of the
kernels we used during Genode's early years and from our pre-occupation with a
certain style of programming. Over the years, the drawbacks inherent in our
original design became more and more clear and we drafted rough plans to
overcome them. However, reworking the fundamental protocols of a system that
already accommodates hundreds of component implementations cannot be taken
lightly. Because of this discomfort, we repeatedly deferred the topic -
until now. With the rapidly growing workloads carried by Genode, we
deliberately decided to address long-standing deficiencies rather than adding
the features we originally planned according to the
[https://genode.org/about/road-map - road map].
Section [Asynchronous parent-child interactions] presents the reworking of
Genode's component interplay at the lowest level. With this change in place,
we feel much more comfortable to scale up our workloads in the upcoming
releases.
Functionality-wise, the most prominent topic of the current release is the
vastly improved NIC-routing component. Since we introduced the first version
of the NIC router in the previous release, we took an iterative approach to
shape the component according to its most prominent use cases. Section
[Further improved virtual networking] summarizes the changes and the
motivation behind them.
Even though we added support for seL4 in the previous release, the NOVA
hypervisor is still our go-to kernel for x86-based hardware because of its
feature set. For this reason, we continuously improve this kernel and the
NOVA-specific components like VirtualBox. Section [NOVA hypervisor] covers
the introduction of an asynchronous map operation to NOVA.
Further topics of the current release range from added smart-card support,
over a new timeout API, to a VFS-based time-based password generator. With
respect to the road map, we postponed most topics originally planned. In
particular, we intended to enable the use of Genode on top of Xen by following
the footsteps of the existing Muen support - using our custom
base-hw kernel within a Xen DomU domain. However, before proceeding this
route, we decided to modernize the kernel design, in particular with respect
to bootstrapping and address-space management. Some parts of this line of work
are already present in the current release, for example the unification of the
boot-module handling as explained in Section
[Unified handling of boot modules].
Asynchronous parent-child interactions
######################################
When Genode was born in 2006, the L4 microkernels of the time universally
lacked an asynchronous inter-process-communication (IPC) mechanism.
Consequently, we designed the first version of Genode with the presumption
that components had to interact solely synchronously. To us, this seemed to be
the "right" way because the synchronous low-footprint IPC was presumably the
key for L4's good performance. It felt natural to leverage this benefit to the
maximum extent possible.
To illustrate the implications of this line of thinking for Genode, let's take
a look at a simple scenario where a parent component hosts two children and one
child provides a service to the other child.
[image simple_scenario]
During the creation of a session, the kernel's IPC mechanism serves three
purposes. First, it is used to communicate information between different
protection domains, in this case the parent, the client, and the server.
Second, it implicitly dictates the flow of control between the involved
parties because the caller blocks until the callee replies to the IPC call.
Third, the IPC is the mechanism to delegate authority (like the authority to
access the server's session object) between protection domains. The latter is
realized with the kernel's ability to carry capabilities as IPC message
payload. If this sounds a bit too abstract, please consider reviewing Section
3.1. "Capability-based security" of the
[https://genode.org/documentation/genode-foundations-16-05.pdf - Genode Foundations].
Using solely a synchronous IPC mechanism, the sequence of establishing a
session in the given scenario is as follows. In the context of Genode,
we usually refer to synchronous IPC as RPC (remote procedure call).
[image sync_session_seq]
The sequence looks straightforward:
# The client issues an RPC call to its parent, requesting a session for a
service of the given type while also passing a number of session-construction
arguments along with the request.
# Given the service name as provided with the session request, the parent
determines the server to ask for a new session. It requests a session
on behalf of the client by performing an RPC call to the server's prior
registered "root" capability. This capability refers to an interface for
creating and closing sessions.
# The server responds to the invocation of its root interface by creating
a new session object along with a session capability.
Whereas the session object is local to the server, the corresponding
session capability can be passed (delegated) to other components.
Each component in possession of the session capability is able to interact
with the server's corresponding session object via RPC calls.
The server returns the session capability to the parent as the result of the
parent's RPC call.
# The parent forwards the session capability to the client as the result of
the client's original RPC call.
Even though the simplicity of this protocol seems nice, it has inherent
limitations:
First, as the parent performs a synchronous RPC call to the server on behalf
of the client, it must trust the server to eventually respond to the RPC call.
If the server doesn't, the parent may block forever. In contrast to the client
that actually uses the service and thereby relies on the liveliness of the
server, the parent should not need to trust the server to be responsive. To
deal with the risk of an unresponsive server, Genode's existing runtime
environments (like the init component), maintain a dedicated thread for each
child. The session requests originating from a child are handled by the
corresponding parent-local child thread. In the worst case - if the server
fails to respond - only a single child thread stays blocked but the other
parts of the runtime environment remain unaffected. Consequently, runtime
environments have to be multi-threaded components. This, in turn, comes at the
cost of added complexity, in particular the need for error-prone inter-thread
synchronization.
Second, the approach keeps the parent's state implicitly stored in the stacks
of the parent's threads. This becomes a problem in dynamic runtime
environments that need to kill subsystems at arbitrary times. E.g., imagine
the situation where the client component is to be destroyed while the parent's
call to the server's root interface is still pending. The safe destruction of
the child - including its associated parent-local child thread - requires the
parent to abort the RPC call, which is a complex and - again - error-prone
operation.
Third, even though not inherent to synchronous RPC, Genode's original design
facilitated the use of a session capability as argument for requesting the
parent to close a specific session. However, the use of capabilities as
re-identifiable tokens is not well supported by most kernels, including seL4
([http://sel4.systems/pipermail/devel/2014-November/000114.html - discussion]
on the seL4 mailing list).
Asynchronous communication throughout Genode
--------------------------------------------
In 2008, we acknowledged the sole reliance on synchronous RPC as too limiting
and introduced an
[https://genode.org/documentation/release-notes/8.11#Asynchronous_notifications - API for asynchronous notifications].
On the traditional L4 kernels, we implemented the API by using Genode's
core component as a proxy for signal delivery. The use of asynchronous
notifications soon became natural and wide-spread throughout Genode. Today,
most session interfaces combine three forms of inter-component communication,
namely synchronous RPC calls, asynchronous notifications, and shared memory.
The new Genode API introduced in
[https://genode.org/documentation/release-notes/16.05#The_great_API_renovation - version 16.05]
further cultivated the modeling of Genode components as single-threaded state
machines instead of multi-threaded programs.
Still, until now, the most fundamental mechanism of Genode - the protocol
between parent and child components - has remained synchronous. The reasons
are twofold. First, our workaround for realizing runtime environments in a
multi-threaded way worked too well. So we were not constantly bothered by this
design problem. Second and more importantly, redesigning the fundamental
mechanism of the framework while not breaking the more than 300 existing
components is quite scary. But in anticipation of the rapidly scaling
workloads imposed on Genode, we had to take on the problem sooner or later.
We figured that now - with the modernized framework API in place - it's the
right time. From redesigning the interplay of parent and child components, we
will become able to create single-threaded runtime environments that behave
completely deterministically while consuming less resources than
multi-threaded programs. By the explicit enumeration of possible states, we
greatly ease the validation/evaluation of such crucial components.
New session-creation procedure
------------------------------
Following the asynchronous approach, the sequence of creating a session now
looks as follows:
[image async_session_seq]
The dotted lines are asynchronous notifications, which have fire-and-forget
semantics. A component that triggers a signal does not block.
The following points are worth noting:
* Sessions are identified via IDs, which are plain numbers as opposed to
capabilities. The IDs as seen by the client and server belong to different
ID name spaces.
IDs of sessions requested by the client are allocated by the client. IDs
of sessions requested at the server are allocated by the parent.
* The parent does no longer need to perform RPC calls to any of its children.
Hence, the need for multiple threads in runtime environments disappears.
* Each activation of the parent merely applies a state change of the session's
meta data structures maintained at the parent, which capture the entire
state of session requests. There is no hidden state stored on the parent's
stack.
* The information about pending session requests is communicated from the
parent to the server via a ROM session. At startup, the server requests
a ROM session for the ROM module "session_requests" from its parent. The
parent implements this ROM session locally. Since ROM sessions support
versions, the parent can post version updates of the "session_requests"
ROM with the regular mechanisms already present in Genode.
* The involved parties can potentially run in parallel.
Outcome and current state
-------------------------
Intuitively, the sequence of steps required to establish a session has
become more complicated. However, for the users of the framework, the entire
procedure is completely transparent. With a few tricks, we were actually able
to implement this fundamental change while keeping almost all existing
components untouched. One trick is the introduction of a server-local proxy
mechanism, which translates the requests obtained from the "session_requests"
ROM to component-local RPC calls on the server's root interface. So from the
perspective of an existing server component, a session request still looks
like a synchronous RPC request from the outside. Of course, the proxy is meant
as an intermediate solution until we have crafted a convenient front-end API
for the asynchronous mode of operation.
Even though the biggest share of components remains unaffected by the change,
this is not true for all components. In particular, runtime environments had
to be reworked, in some cases quite fundamentally. These include core, init,
noux, the loader, GDB monitor, launcher, CLI monitor, and the platform driver.
The change does not only affect the interplay between components but also
required a reconsideration of the child-creation procedure.
Besides the architectural improvement, this line of work had two welcome
effects.
First, in contrast to the original design, which relied on capabilities as
re-identifiable tokens, the new version greatly alleviates the need for
re-identifying capabilities on seL4. So we are able to eliminate a
long-standing problem with Genode on this kernel.
Second, the work called for new data structures for the safe interaction with
ID spaces (_base/id_space.h_) and object registries (_base/registry.h_). Those
data structures will possibly be useful in a lot of places that currently use
plain (and fairly unsafe) AVL trees or lists.
At the API level, the change is almost transparent to regular components,
except for two details. The upgrading of session quota is no longer
possible by a mere RPC call to the parent. Instead, 'Connection' objects
received a new 'upgrade_ram' method that must be used instead. Speaking
of 'Connection' objects, we had to remove the (fairly obscure) 'KEEP_OPEN'
feature, which is conceptually incompatible with the new design.
Further improved virtual networking
###################################
The
[https://genode.org/documentation/release-notes/16.08#Virtual_networking_and_support_for_TOR - previous release]
introduced the NIC router - a component that individually routes IP
packets between multiple NIC sessions, translates between different IP
subnets, and also supports port forwarding and NAT. For the first version of
the NIC router, we focused on the technical realization. Now, besides
some optimization and restructuring, we took the chance to polish the
configuration interface of the component. The goal was to make the interface
more intuitive and reduce pitfalls to a minimum. Roughly speaking, the
handling of the NIC router became more tailored to its/our typical use cases.
Let's create a practical setup to explain the changes in detail. Assume that
there are two virtual subnets 192.168.1.0/24 and 192.168.2.0/24 within our
Genode system. They connect as Virtnet A and B to the router. The standard
gateway of the virtual networks is the NIC router with IP 192.168.*.1 . The
router's uplink, on the other hand, is connected to the NIC driver. It
interfaces the machine with our real-world home network 10.0.2.0/24. The home
network is connected to the internet through its standard gateway 10.0.2.1.
[image nic_router_basic]
The basic router configuration for this setup without any routing rules would
be as follows:
! <policy label_prefix="virtnet_a" domain="virtnet_a" />
! <policy label_prefix="virtnet_b" domain="virtnet_b" />
!
! <domain name="uplink" interface="10.0.2.55/24" gateway="10.0.2.1" />
! <domain name="virtnet_a" interface="192.168.1.1/24" />
! <domain name="virtnet_b" interface="192.168.2.1/24" />
The first thing to notice is the changed usage of the policy tag. Previously,
the policy label - normally solely designated to correlate sessions with
configuration domains - was misused also as unique peer identifier in the
routing rules. This approach disregarded advanced label-matching techniques
such as the 'label_prefix' used above. Now, the whole NIC-router-specific
enhancement of the policy tag moved to the new '<domain>' tag, leaving the
policy tag only with its original purpose to select policies. Note that even
if this modification gives the impression, the router is not yet capable of
handling multiple NIC sessions at one domain at a time.
In the domain tag, the 'interface' attribute replaces the old policy attribute
named 'src'. That means, it tells the router which IP identity to use when
talking as itself to the domain. But in addition to that, the 'interface'
attribute also defines which subnet this identity and the domain belong to.
This reflects a basic decision we made during the reworking process: The new
NIC router is aware of subnets. Sessions of the same subnet have the same
configuration domain. We came to this conclusion as it solves some fundamental
problems with the old version. First, the equivalence of domain and subnet
enables us to link a default gateway to a subnet by adding the 'gateway'
attribute to the domain tag. In our example, this is done in the uplink
domain. The 'gateway' attribute is optional for a domain and replaces the
former 'via' attributes of the different routing rules. It is more efficient
and natural to have this value set only once at the corresponding subnet than
having it scattered all over the routing rules of the remote domains as done
before. If a domain has no default gateway, it drops all packets with a
foreign recipient.
The second advantage of a domain being equivalent to a subnet is that handling
ARP broadcasts becomes easy. It can be excluded that such ARP broadcasts
concern sessions outside the source domain anymore. And as sessions in the
same domain are not distinguishable to the routing, the broadcast can be sent
to all of them without breaking any rules.
Now, let's enhance our example by some routing rules. One pretty complicated
thing to do with the old NIC router was port forwarding. You had to combine
different routing rules, explicitly enable the back routing at the remote
side, and take care that NAT was applied - a lot of opportunities for
mistakes. With the new version, it became easier. Let's assume we have an HTTP
server in Virtnet A and an NTP server in Virtnet B. We want the NIC router to
act as proxy for their services in our home network.
[image nic_router_servers]
In order to achieve this, the uplink domain must be enhanced by two rules:
! <policy label_prefix="virtnet_a" domain="virtnet_a" />
! <policy label_prefix="virtnet_b" domain="virtnet_b" />
!
! <domain name="uplink" interface="10.0.2.55/24" gateway="10.0.2.1" />
! <tcp-forward port="443" domain="virtnet_a" to="192.168.1.2" />
! <udp-forward port="123" domain="virtnet_b" to="192.168.2.2" />
! </domain>
!
! <domain name="virtnet_a" interface="192.168.1.1/24" />
! <domain name="virtnet_b" interface="192.168.2.1/24" />
The TCP forwarding rule for port 443 (HTTP+TLS/SSL) redirects to IP address
192.168.1.2 in Virtnet A and the UDP forwarding rule for port 123 (NTP)
redirects to IP address 192.168.2.2 in Virtnet B. The Virtnet domains remain
empty as the router keeps track of the redirected transfers and routes back
reply packets automatically. Also automatically, the router applies NAT for the
server as it is in the nature of port forwarding.
Next, we add some clients to Virtnet B that like to talk to our home network
and the internet. We want them to be hidden via NAT when they do so. For
internet communication, they shall furthermore be limited to HTTP+TLS/SSL and
IMAP+TLS/SSL.
[image nic_router_client]
This is what the router configuration looks now:
! <policy label_prefix="virtnet_a" domain="virtnet_a" />
! <policy label_prefix="virtnet_b" domain="virtnet_b" />
!
! <domain name="uplink" interface="10.0.2.55/24" gateway="10.0.2.1" />
! <tcp-forward port="443" domain="virtnet_a" to="192.168.1.2" />
! <udp-forward port="123" domain="virtnet_b" to="192.168.2.2" />
! <nat domain="virtnet_b" tcp-ports="1000" udp-ports="1000">
! </domain>
!
! <domain name="virtnet_a" interface="192.168.1.1/24" />
! <domain name="virtnet_b" interface="192.168.2.1/24" >
! <tcp dst="10.0.2.0/24"> <permit-any domain="uplink" /> </tcp>
! <udp dst="10.0.2.0/24"> <permit-any domain="uplink" /> </udp>
! <tcp dst="0.0.0.0/0">
! <permit port="443" domain="uplink" />
! <permit port="993" domain="uplink" />
! </tcp>
! </domain>
There are several new tag types. One of them is the NAT configuration for
Virtnet B in the uplink domain. In contrast to the former NIC-router version
where NAT settings were part of the source domain, NAT is now configured in
the target domain with a sub-tag for each source. This has the advantage
of supporting heterogeneous NAT configurations for a packet source depending
on which domain it talks to. Besides, it is more intuitive to read. Apart from
that, the NAT settings haven't changed.
Furthermore, there are the new TCP and UDP tags in the Virtnet-B domain. The
first two of them have a 'permit-any' sub-tag. With this combination, we open
all ports to IP addresses of the 10.0.2.0/24 subnet, our home network, and
route them to the uplink domain. TCP packets that don't match these first two
rules may fall back to the third. This TCP rule doesn't have all ports opened
but only 443 (HTTP+TLS/SSL) and 993 (IMAP+TLS/SSL). Both ports are again bound
to the uplink domain. As the IP filter 0.0.0.0/0 of the surrounding rule isn't
restrictive, we now also route packets to a foreign destination. The NIC
router redirects such packets to the default gateway of our home network.
Compared to the old router version where IP and UDP/TCP routing had to be
combined for this purpose, the new TCP and UDP rules with their
port-permission sub-rules have some notable advantages. Like port-forwarding
rules, TCP and UDP rules always imply link-state tracking in order to route
back reply packets automatically. This can be seen also in our example as no
further routing rules had to be added to the uplink domain. This aspect is
clear from the outermost rule and not dependent on sub-rules anymore.
Furthermore, the strict separation of UDP and TCP routing prevents
configuration faults and increases readability. Last but not least, the
'permit-any' rule allows something new. Opening all ports for an address range
was previously only possible without link-state tracking as it could be
expressed only on the IP level.
At this point, we have thoroughly discussed the layer-3 routing abilities of
the new NIC router and our focus has indeed moved more into this direction.
Even though IP routing is still available, we found that it should be more
clearly separated from the rest. To illustrate this feature, we enhance our
example again. We want the Virtnets to be allowed to communicate to each other
without any restrictions. For that purpose, we add two more rules to the
router configuration:
! <policy label_prefix="virtnet_a" domain="virtnet_a" />
! <policy label_prefix="virtnet_b" domain="virtnet_b" />
!
! <domain name="uplink" interface="10.0.2.55/24" gateway="10.0.2.1" />
! <tcp-forward port="443" domain="virtnet_a" to="192.168.1.2" />
! <udp-forward port="123" domain="virtnet_b" to="192.168.2.2" />
! <nat domain="virtnet_b" tcp-ports="1000" udp-ports="1000">
! </domain>
!
! <domain name="virtnet_a" interface="192.168.1.1/24" />
! <ip dst="192.168.2.0/24" domain="virtnet_b"/>
! </domain>
!
! <domain name="virtnet_b" interface="192.168.2.1/24" >
! <tcp dst="10.0.2.0/24"> <permit-any domain="uplink" /> </tcp>
! <udp dst="10.0.2.0/24"> <permit-any domain="uplink" /> </udp>
! <tcp dst="0.0.0.0/0">
! <permit port="443" domain="uplink" />
! <permit port="993" domain="uplink" />
! </tcp>
! <ip dst="192.168.1.0/24" domain="virtnet_a"/>
! </domain>
As you can see, each of the new IP rules in the Virtnet domains match the
addresses of the opposite subnet and route to the corresponding domain. As
mentioned, the new IP rules and UDP/TCP rules are not combined anymore to
clearly distinguish IP routing from layer-3 routing because this decision has
far-reaching effects. First, in contrast to UDP and TCP routing, IP routing is
stateless. Thus, for each IP routing rule one has to be sure to have a
back-routing rule at the remote domain or else bidirectional communication
won't happen. And second, NAT does not apply to IP-routed packets. So, if
you're not aware of such packets, you may unintentionally reveal information
about a private network.
For more details on the new NIC router, you may refer to the comprehensive
documentation in the _repos/os/src/server/nic_router/README_ file and the
basic NIC-router test at _libports/run/nic_router.run_ .
Base framework
##############
Improved RPC mechanism
======================
Since we introduced Genode's current API for synchronous RPCs in
[https://genode.org/documentation/release-notes/11.05#New_API_for_type-safe_inter-process_communication - version 11.05],
inter-component communication within Genode has become almost a child's play.
The RPC framework leverages the C++ type system and templates to a great
effect. In contrast to the traditional use of IDL compilers, the interaction
with RPC objects provided by other components is robust and natural because
no language boundaries need to be crossed.
Still, a few differences between RPC calls and regular function calls remain.
In particular, there exist a few restrictions with regard to the types of
RPC function arguments. Those types did not just need to be POD (plain old
data) types but they had to be default-constructible, too. Whereas the former
restriction still applies (non-POD objects that include references or
vtables cannot be used as arguments), the latter limitation has been lifted
now. Generally, non-default-constructible types are a way to attain
simpler code because the special case of an "invalid" object does not need
to be considered. I.e., values of such types can be kept as constants as
opposed to variables. If an object exists (as equivalent to successful
instantiation), it is valid. With the improved RPC mechanism, the RPC
framework does no longer stay in the way in this respect.
Thanks to Edgard Schmidt for this welcome contribution!
Unification and tightening of session labels
============================================
In Genode, each session requested by a client component is labeled according
to the components that intermediate the session request. The client can
optionally specify a label of choice along with the session request. Its
parent prefixes the client-provided label by a label of its own. If the
session request is further passed to the parent's parent, the grandparent
prepends its own label. This works recursively. Consequently, the final label
as seen by the server is the product of the labeling policies of all
components on the route of the session request.
The label is used for two purposes. First, the server uses the label as
a key for a server-side policy selection. E.g., depending on the session label
received by the disk-partition server, the server decides which partition to
hand out to the client. Second, the label is used by intermediate components
to take session-routing decisions. E.g., based on the label of a file-system
session request, a parent component may route the request to one of several
file-system servers.
Originally, Genode did not impose a specific way of how labels are formed.
It was up to each intermediate component to filter the label of a session
request in any way desired. However, in practice, this freedom remained unused
and the very simple successive prefixing of labels prevails in all our use
cases. Each intermediate node concatenates its own label in front of the label
supplied by the originator of the session request. The different parts of the
label are separated with the character sequence '" -> "'. Some corner cases
were handles specially for aesthetic reasons. For example, if a client
provided no label, the parent would skip the pending separator. That said,
since each intermediate component had to provide the labeling policy, not all
components were consistent in these respects. Since we found no use for
arbitrary labeling policies, we decided to make the only prominent way of
session labeling mandatory for all intermediate components. We thereby removed
the aesthetically motivated corner cases and possible ambiguities. I.e., with
the original policy, it was not possible to distinguish a unlabeled session
requested by a client from a labeled session requested by the client's parent.
As a consequence, the stricter labeling must now be considered wherever
a precise label was specified as a key for a session route or a server-side
policy selection. The simplest way to adapt those cases is to use a
'label_prefix' instead of the 'label' attribute. Alternatively, the
'label' attribute may used by appending '" -> "' (note the whitespace).
Transition to new framework API
===============================
Since we fundamentally revised Genode's API in
[http://genode.org/documentation/release-notes/16.05#The_great_API_renovation - version 16.05],
we gradually adapt our existing components. Given that Genode comes with
over 300 components, this is no small feat. But with 30 percent of the
components converted, we already made substantial progress.
In some respects, the conversion is actually nearly complete. In particular,
the move away from format-string-based text output to our new type-safe output
facility has been applied to almost all components now. The former 'PDBG'
macro that is quite useful for temporary debug messages has been replaced with
a new version that must be manually included via the _base/debug.h_ header
file. Like the regular log functions, the new PDBG facility uses the type-safe
text-output facility.
Minor API adjustments
---------------------
While applying Genode's new API, we refined the API in the following respects:
We added a dedicated 'String' constructor overload to better accommodate
string literals. This overload covers the common case for initializing a
string from a literal without employing the 'Output' mechanism. This way, such
strings can by constructed without calling virtual functions, which in turn
makes the 'String' usable during the self-relocation phase of the dynamic
linker.
Up till now, several Genode components still rely on the use of 'snprintf'
whenever strings must be assembled out of smaller pieces. As we like to shun
format strings from Genode altogether, we needed an alternative mechanism.
Since we introduced the new type-safe text-output facilities in Genode 16.05,
there is an obvious solution: Let the 'String' constructor accept an arbitrary
list of arguments, which are turned into their respective textual
representation and appear concatenated in the resulting string. Consequently,
strings can be assembled with the same flexibility as log output. For the
construction of 'String' objects from character buffers of a known size, the
'Cstring' utility can be used, which takes a 'char const *' and an optional
length as arguments.
Several low-level types received support for the new output facilities, e.g.,
'Xml_node' or the network-related headers in _os/net/_.
In anticipation of the forthcoming package-management infrastructure, we try
to unify Genode's executable binaries across kernels and architectures
wherever reasonable. Of course, the latter is not possible with respect to the
used instructions. But unifying symbol information is deemed worthwhile. For
this reason, we changed the 'Genode::size_t' type to be always defined as an
'unsigned' 'long'. This is in contrast to GCC's built-in '__SIZE_TYPE__',
which is defined as 'unsigned int' on 32-bit architectures but 'unsigned long'
on 64-bit architectures.
OS-level infrastructure and device drivers
##########################################
New timeout-handing API
=======================
The new timeout API offers tools for easily multiplexing a single time
source among different timeouts. In general, the time source can be
implemented individually but we expect that the most prominent use case will
be the multiplexing of timer sessions. Thus, the timeout library also provides
a convenience tool for this use case. A library-usage example can be found
under _os/src/test/timeout_. If you're interested in implementing
your own time source, you can find an example at _os/include/os/timer.h_ .
Support for smart cards
=======================
We ported the [http://pcsclite.alioth.debian.org/pcsclite.html - PC/SC Lite]
library to Genode, which provides a commonly used API for communicating with
smart cards. It supports USB smart card readers, using the
[http://pcsclite.alioth.debian.org/ccid.html - CCID] library as driver.
The CCID driver itself requires [http://libusb.info - libusb] to access the
USB device.
Vanilla PC/SC Lite is structured as a client-server architecture, consisting
of the 'pcscd' daemon, which runs on a privileged user account and manages all
card reader devices, and one or more non-privileged client applications, which
communicate with pcscd to access the card readers. On Genode, pcscd's role as
privileged device manager is not really needed, since the devices can also be
managed using Genode's configuration mechanisms. For this reason, we merged
the part of pcscd which implements the API with the pcsc-lite client library.
In the current state, a Genode application using PC/SC Lite can access a single
card reader device, which is selected using its USB product ID and vendor ID in
the application's configuration and in the policy of the USB driver.
More configuration details can be found in the README files of the PC/SC Lite,
CCID, and libusb libraries in the libports repository and in the accompanying
_smartcard.run_ script.
Libraries and applications
##########################
Time-based password generation
==============================
A time-based one-time password authentication client that adheres to the
Google Authenticator standard has been introduced into the
[https://github.com/genodelabs/genode-world - world repository].
Single use, time-based passwords are commonly used as an additional
authentication step for web-based services. In this scheme, a user generates
and presents a six digit passcode to a service generated using a shared secret
and a timestamp. This short passcode length makes manual entry convenient so
that the shared secret may be stored on a separate device than the service
client, such as a smartphone, layering the security properties of both
devices.
The 'gtotp' VFS plugin provides these passcodes by embedding the generator as
a special file in the file-system layer of a component. This approach provides
readily available passcodes for programmatic and manual use without enlarging
the code base to encompass a GUI, command-line, or networked interface.
At the time of this release, the common use case is to manually retrieve codes
for clients running in VirtualBox by reading special files with an isolated
instance of the Noux runtime. Storing the shared secret on the same device
contradicts the recommendations of the standard but the trade-off is that the
software stack required to host the shared secret is significantly smaller
than that found on a mobile device.
Random number generator testing
===============================
No random number generator can be proved to be good, but empirical statistical
tests can prove that some are bad. A port of the TestU01 RNG test suite is
provided in the world repository. The TestU01 batteries give independent
assurance of the fitness of Genode's CPU jitter based RNG and are available
for testing future physical and non-phyical RNGs.
VirtualBox on top on the NOVA hypervisor
########################################
Both VirtualBox-based virtual machine monitors on Genode got updated to the
latest revision as provided by Oracle, namely 4.3.40 and 5.1.10 - mainly to
stay close to the upstream versions.
Platforms
#########
Unified handling of boot modules
================================
Until now, the way of passing boot modules from the boot procedure to the core
component, which core provides as ROM modules, varied from platform to
platform. Either we used a multiboot-compliant bootloader that accepts
multiple modules, or the platform provided some specific way of linking binary
modules together with the kernel, e.g., the Elfweaver tool of OKL4.
By unifying the boot-module handover, we further reduce platform specific core
code. Thereby, maintenance costs are decreased, and code analysis becomes
easier. With this new solution, when issuing to build the core component:
! make core
within the build system, only a core library gets built. Not until all
binaries needed by a run-script are available, a final image is linked
together using the core library and all additional binaries. The core
component now can access its ROM modules directly via addresses contained in
its binary. As a side effect of this change, there is no core binary in the
'bin' or 'core' directory of the corresponding build directory available
anymore. Instead, you will find the core binary with no ROM modules, but
including debug information under 'var/run/*.core' within your build
directory. The concrete name depends on the name of the run-script.
The new approach is used on all platforms except Linux where the ROM modules
still need to be accessed via the file-system.
NOVA hypervisor
===============
We extended the kernel to support the asynchronous delegation of kernel
resources. Up to now, resources could only be delegated during RPC or during
the initial protection-domain construction. With this extension, the
construction and setup of new protection domains, threads, and especially
virtual CPUs for the VirtualBox VMM became more straightforward and several
quirks inside the 'core' component could be dropped. The added kernel syscall
expects the NOVA-kernel capabilities of the source and target protection
domains, which effectively renders the operation solely available to 'core' -
as only holder of the NOVA protection domain capabilities.
Additionally, we changed the CPU ID enumeration in Genode/NOVA to a
predictable order. The lower CPU IDs used via the Genode 'Cpu_session'
interface now correspond to the first hyper-thread of all physical CPU cores.
For example, on a quad-core machine with hyper-threading enabled Genode's CPU
IDs 0-3 refer to the first hyper-threads of all physical cores and IDs 4-7 to
the second hyper-threads.

1033
doc/release_notes-17-05.txt
View File

File diff suppressed because it is too large Load Diff

651
doc/release_notes-17-08.txt
View File

@@ -1,651 +0,0 @@
===============================================
Release notes for the Genode OS Framework 17.08
===============================================
Genode Labs
The flagship feature of Genode 17.08 has been in the works for more than a
year: The support for hardware-accelerated graphics on Intel Gen-8 GPUs. This
is an especially challenging topic because it is riddled with terminology,
involves highly complex software stacks, carries a twisted history with it,
and remains to be a moving target. It took up a lot of patience to build up a
profound understanding of the existing driver architectures and the mechanisms
offered by modern graphics hardware. On the other hand, with the proliferation
of hardware-based sandboxing features like virtual GPU memory and hardware
contexts, we found that now is the perfect time for a clean-slate design of a
microkernelized GPU driver.
Section [Hardware-accelerated graphics for Intel Gen-8 GPUs] introduces this
work, which includes our new GPU multiplexer as well as the integration with
the client-side Mesa protocol stack.
The second focus of the current release is the extension of Genode's supported
base platforms. Most prominently, we upgrade the seL4 kernel to version 6.0
while extending the architecture support from 32-bit x86 to ARM and 64-bit
x86 (Section [The seL4 kernel on ARM and 64-bit x86 hardware]). To bring
Genode closer to cloud-computing scenarios, we added basic support for
executing Genode scenarios as Xen DomU domains (Section [Genode as Xen DomU]).
Furthermore, the Muen separation kernel has been updated to a current version.
As a cross-kernel effort, there is work under way to boot Genode-based
systems via UEFI, currently addressing the NOVA, base-hw, and seL4 kernels.
Among the many other functional additions are a new VFS plugin for accessing
FAT file systems, new components like _sequence_ and _fs_report_ that aid new
system compositions, and our evolving custom package-management
infrastructure.
Hardware-accelerated graphics for Intel Gen-8 GPUs
##################################################
The ability to leverage hardware-accelerated graphics is generally taken for
granted in modern commodity operating systems. The user experience of
modern desktop environments, web-browser performance, and obviously games
depend on it. On the other hand, the benefit of hardware-accelerated graphics
comes at the expense of tremendous added complexity in the lower software
stack, in particular in system components that need to be ultimately trusted.
For example, with circa 100 thousand lines of code, the Intel GPU driver in
the Linux kernel is an order of magnitude more complex than a complete modern
microkernel. In a monolithic-kernel-based system, this complexity is
generally neglected because the kernel is complex anyway. But in
microkernel-based scenarios optimized for a trusted computing base in the
order of a few ten thousand lines of code, it becomes unacceptable.
Fortunately, recent generations of graphics hardware provide a number of
hardware features that promise to solve this conflict, which prompted us to
investigate the use of these features for Genode.
During this year's Hack'n'Hike event, we ported the ioquake3 engine to Genode.
As preliminary requirement, we had to resurrect OpenGL support in our aging
graphics stack and enable support for current Intel HD Graphics devices (IGD).
We started by updating Mesa from the old 7.8.x to a more recent 11.2.2 release.
Since we focused mainly on supporting Intel devices, we dropped support for the
Gallium back end as Intel still uses the old DRI infrastructure. This decision,
however, also influenced the choice of the software rendering back end. Rather
than retaining the softpipe implementation, we now use swrast. In addition, we
changed the available OpenGL implementation from OpenGL ES 2.x to the fully
fledged OpenGL 4.5 profile, including the corresponding shader language
version. As with the previous Mesa port, EGL serves as front end API for
system integration and loads a DRI back-end driver (i965 or swrast). EGL
always requests the back-end driver 'egl_drv.lib.so' in form of a shared
object. Genode's relabeling features are used to select the proper back end
via a route configuration. The following snippet illustrates such a
configuration for software rendering:
! <start name="gears" caps="200">
! <resource name="RAM" quantum="32M"/>"
! <route>
! <service name="ROM" label="egl_drv.lib.so">
! <parent label="egl_swrast.lib.so"/>
! </service>
! <any-service> <parent/> <any-child/> </any-service>
! </route>
! </start>
With the graphics-stack front end in place, it was time to take care of the
GPU driver. In our case this meant implementing the DRM interface in our
ported version of the Intel i915 DRM driver. Up to now, this driver was solely
used for mode setting while we completely omitted supporting the render
engine.
[image mesa_genode]
With this new and adapted software stack, we successfully could play ioquake3
on top of Genode with a reasonable performance in 1080p on a Thinkpad X250.
During this work, we gathered valuable insights into the architecture of a
modern 3D-graphics software stack as well as into recent Intel HD Graphics
hardware. We found that the Intel-specific Mesa driver itself is far more
complex than its kernel counter part. The DRM driver is mainly concerned with
resource and execution management whereas the Mesa driver programs the GPU.
For example, amongst others, Mesa compiles the OpenGL shaders into a
GPU-specific machine code that is passed on to the kernel for execution.
While inspecting the DRM driver, it became obvious that one of the reasons for
its complexity is the need to support a variety of different HD Graphics
generations as well as different features driven by software-usage patterns.
For our security related use cases, it is important to offer a clear isolation
and separation mechanism per client. Hardware features provided by modern
Intel GPUs like per-process graphics translation tables (PPGTT) and hardware
contexts that are unique for each client make it possible to fulfill these
requirements.
By focusing on this particular feature set and thus limiting the supported
hardware generations, the development of a maintainable GPU multiplexer for
Genode became feasible. After all, we strive to keep all Genode components as
low complex as possible, especially resource multiplexers like such a GPU
multiplexer.
[image intel_gpu_drv]
This image shows multiple GPU-session clients and the resources they are
using. The fence registers as well as the aperture is partitioned between
them, the PPGTT is backed by the system memory, and the contexts are located
in disjoint GGTT regions.
Within four months, we implemented an experimental GPU multiplexer for Intel
HD Graphics Gen8 (Broadwell class) devices. We started out defining a GPU
session interface that is sufficient to implement the API used by the DRM
library. For each session, the driver creates a context consisting of a
hardware context, a set of page tables (PPGTT), and a part of the aperture.
The client may use the session to allocate and map memory buffers used by the
GPU. Each buffer is always eagerly mapped 1:1 into the PPGTT by using the
local virtual address of the client. Special memory buffers like an image
buffer are additionally mapped through the aperture to make use of the
hardware-provided de-tiling mechanism. As is essential in Genode components,
the client must donate all resources that the driver might need to fulfill the
request, i.e., quota for memory and capability allocations. Clients may
request the execution of their workload by submitting an execution buffer. The
GPU multiplexer will then enqueue the request and schedule all pending
requests sequentially. Once the request is completed, the client is notified
via a completion signal.
[image multi_gl]
Example scenario of multiple OpenGL programs that use the new GPU multiplexer
for hardware-accelerated rendering.
We consider this first version of the GPU driver as experimental. As of now,
it only manages the render engine of the GPU. Mode-setting or rather display
handling must be performed by another component. Currently, the VESA driver is
used for this purpose. It also lacks any power-management functionality and
permanently keeps the GPU awake. Both limitations will be addressed in future
releases and support for Gen9+ (Skylake) and newer devices might be added.
In its current incarnation, the GPU multiplexer component consists of about
4,200 lines of code whereas the Mesa DRI i965 driver complements the driver at
the client side with about 78,000 lines of code.
The seL4 kernel on ARM and 64-bit x86 hardware
##############################################
With the 16.08 release, we brought the seL4 support to a level to be
considered being on par with the other supported kernels. At the time,
Genode's use of seL4 was limited to 32-bit x86 platforms.
In the current release, we extend the platform support to ARM and 64-bit x86.
We started this line of work with an incremental kernel upgrade from version
3.2.0 to 5.2.0 and finally to seL4 6.0. Through these upgrades, we were able
to drop several Genode-specific seL4 patches, which were required in the 16.08
release. One major improvement of version 6.0 compared to earlier versions is
the handling of device-memory announcements by the kernel to Genode's roottask
_core_.
With the kernel update in place, we inspected the x86-specific part thoroughly
while splitting and separating it properly into architecture-agnostic and
architecture-dependent parts. Upon this work, we added the
architecture-specific counterparts for x86_64 and ARM. One major work item was
to make the page-table handling in Genode's core aware and generic enough to
handle the different page-table sizes of the three architectures.
For the ARM support, we decided to enable the i.MX6 FreeScale based SoC,
namely the Wandboard Quad board. Since the seL4 kernel interface provides no
timeout support, we revived a user-level timer driver that we originally
developed for our custom base-hw kernel: The so-called EPIT timer, which is
part of most i.MX SoCs.
We finished the essential work for the mentioned three platforms in
less time than expected and, thereby, had spare time to address additional
features.
First, we enabled multiprocessor support for Genode/seL4 on x86 and
thread-priority support for all seL4 platforms. Additionally, we were able to
utilize the seL4 benchmark interface for Genode's trace infrastructure in
order to obtain utilization information about threads and CPUs. The Genode
components _top_ (text-based) and _cpu_load_monitor_ (graphical) are now
usable on Genode/seL4.
Finally, as we are currently exploring the support for booting various kernels
via UEFI on x86, we took the chance to investigate the steps needed to boot
seL4 via UEFI. UEFI firmware does not always provide a compatibility support
module (CSM) for legacy BIOS boot support. Hence, we extended the seL4 kernel
for Genode according to the Multiboot2 specification, which enables us to
start Genode/seL4 together with GRUB2 - as an UEFI capable bootloader - on
machines missing CSM support.
Base framework and OS-level infrastructure
##########################################
Simplified IOMMU handling
=========================
When IOMMUs are used on x86, all host memory targeted via direct memory
accesses (DMA) by devices must eagerly be registered in the respective I/O
page table of the device. Up to now, Genode supports IOMMUs on NOVA only. On
this kernel, a device protection domain is represented as a regular protection
domain with its virtual memory layout being used for both the CPU's MMU and
the device. Traditionally, mappings into such virtual memory spaces are
inserted on demand as responses to page faults. However, as there are no page
faults for DMA transactions, DMA buffers must always be eagerly mapped. The
so-called device PD hid this gap for NOVA. In anticipation of adding IOMMU
support for more kernels, we desired to generalize the device-PD mechanism by
introducing an explicit way to trigger the insertion of DMA memory into the
proper page tables.
We extended the PD-session interface by a 'map' function, which takes a
virtual memory region of the PD's virtual address space as argument. The page
frames of the previously attached dataspaces are added eagerly by core to the
IOMMU page-tables. With this explicit 'map' support, we were able to replace
the Genode/NOVA-specific device-PD implementation with a generic one, which
will easily accommodate other kernels in the future.
New report server for capturing reports to files
================================================
The report session is a simple mechanism for components to publish structured
data without the complexity of a file-system layer. In the simplest case, a
client component will produce a report and communicate it directly to a
component acting as a server. The disadvantage is that the report client
becomes reliant on the liveliness and presence of the consumer component. So
in the more robust case, the _report_rom_ component acts as the server hosting
the report service as well as a ROM service for components consuming reports.
The _report_rom_ server permits ROM access only to clients matching an
explicit configuration policy. This is good for security but opaque to a user.
Reports can only be read where an explicit policy is in place and only a
single report session can report to an active ROM session.
The new _fs_report_ component is a friendlier and more flexible report server.
Reports are written to a file system using a file and directory hierarchy that
expresses session routing. This allows for intuitive report inspection and
injection via a file system. When used with the _ram_fs_ and _fs_rom_ servers,
it can also replicate the functionality of _report_rom_.
New runtime environment for starting components sequentially
============================================================
The _init_ component is a prime example of software with an emphasis on
function over features. It is the fundamental building block for combining
components yet its behavior is simple and without heuristics. Like other
contemporary init managers, it starts components in parallel, but to a more
extreme degree in that it has no concept of "runlevels" or "targets", all
components are started as soon as possible. The concrete sequence of execution
is instead determined by when server components make service announcements and
how quickly they respond to client requests.
In some cases, the execution of one component must not occur until the
execution of another component ends, be it that the first produces output that
is consumed by the second, or that the two contend for a service that cannot
be multiplexed. Init contains no provisions to enforce ordering. But we are
free to define new behaviors in other management components.
The solution to the problem of ordering is the _sequence_ component. Sequence
walks a list of children and executes them in order, one at a time. With only
one child active, there is no need for any local resource or routing
management. By applying the same session label transformations as init,
external routing and policy handling are unchanged.
An example of ordering a producer and consumer within an init configuration
follows:
! <start name="sequence">
! <resource name="RAM" quantum="128M"/>
! <config>
! <start name="producer">
! <config .. />
! </start>
! <start name="consumer">
! <config .. />
! </start>
! </config>
! <route>
! <service name="LOG" label_prefix="producer">
! <child name="log_a"/> </service>
! <service name="LOG" label_prefix="consumer">
! <child name="log_b"/> </service>
! <any-service> <parent/> <any-child/> </any-service>
! </route>
! </start>
Support for boot-time initialized frame buffer
==============================================
UEFI-based systems do not carry along legacy BIOS infrastructure, on which
our generic VESA driver depends. Hence, when booting via UEFI, one has to use
either a hardware-specific driver like our Intel-FB driver or - alternatively -
facilitate generic UEFI mechanisms.
Instead of booting in VGA text mode and leaving the switch to a graphics mode
(via real-mode SVGA BIOS subroutines) to the booted OS, UEFI employs the
so-called graphics output protocol as a means to setup a reasonable default
graphics mode prior booting the operating system. In order to produce
graphical output, the operating system merely has to know the physical address
and layout of the frame buffer. Genode's core exposes this information as the
_platform_info_ ROM module. The new _fb_boot_drv_ driver picks up this
information to provide a Genode framebuffer session interface. Hence, on
UEFI-based systems, it can be used as a drop-in replacement for the VESA
driver. In contrast to the VESA driver, however, it is not able to switch
graphics modes at runtime.
The new component is located at _os/src/drivers/framebuffer/boot/_. Thanks
to Johannes Kliemann for this contribution.
Extended non-blocking operation of the VFS
==========================================
In
[https://genode.org/documentation/release-notes/17.02#VFS_support_for_asynchronous_I_O_and_reconfiguration - version 17.02],
we added support for non-blocking reads from the VFS in the form of the
'read_ready()', 'queue_read()', and 'complete_read()' functions. Since then,
it has become obvious that blocking within the VFS is not only problematic in
the VFS server itself when multiple clients are connected, but also when the
VFS is deployed in a multi-threaded environment and a VFS plugin needs to
reliably wait for I/O-completion signals.
For this reason, we reworked the interface of the VFS even more towards
non-blocking operation and adapted the existing users of the VFS accordingly.
The most important changes are:
* Directories are now created and opened with the 'opendir()' function and
the directory entries are read with the 'queue_read()' and 'complete_read()'
functions.
* Symbolic links are now created and opened with the 'openlink()' function and
the link target is read with the 'queue_read()' and 'complete_read()'
functions and written with the 'write()' function.
* The 'write()' function does not wait for signals anymore. This can have the
effect that data written by a VFS library user has not been processed by a
file-system server when the library user asks for the size of the file or
closes it (both done with RPC functions at the file-system server). For this
reason, a user of the VFS library should request synchronization before
calling 'stat()' or 'close()'. To make sure that a file-system server has
processed all write request packets that a client submitted prior the
synchronization request, synchronization is now requested at the file-system
server with a synchronization packet instead of an RPC function. Because of
this change, the synchronization interface of the VFS library has been split
into the 'queue_sync()' and 'complete_sync()' functions.
Making block sessions read-only by default
==========================================
Genode server components are expected to apply the safest and strictest
behavior when exposing cross-component state or persistent data. In practice
block and file-system servers only allow access to clients with explicitly
configured local policies. The file-system servers enforce an additional
provision that sessions are implicitly read-only unless overridden by policy.
This release introduces a similar restriction to the AHCI driver and partition
multiplexer. Clients of these servers require an affirmative 'writeable'
attribute on policies to permit the writing of blocks. Write permission at
these servers may also be revoked by components that forward block-session
requests by placing 'writeable="no"' into session-request arguments.
All users of _ahci_drv_ and _part_blk_ are advised that this change may break
existing configurations without explicit 'writeable' policies.
Refined time handling
=====================
Release 17.05 introduced a
[https://genode.org/documentation/release-notes/17.05#New_API_for_user-level_timing - new API for user-level timing]
named _timeout framework_. Together with this new framework came a
comprehensive test that stresses all aspects of the interface. During the past
few months, this test has turned out to be an enrichment for Genode far beyond
its original scope. As the test significantly raised the standards in
user-level timing, it also sharpened our view on the measurement precision of
various timer drivers and timestamps, which act as input for the framework.
This revealed several problems previously unidentified. For instance, we
improved the accuracy and stability of the time values provided by the drivers
for the Raspberry-Pi timer, the Cortex-A9 timer, the PIT, and the LAPIC. We
also were able to further optimize the calibration of the TSC in the NOVA
kernel.
Additionally, the test also helped us to refine the timeout framework itself.
The initial calibration of the framework - that previously took about 1.5
seconds - is now performed much quicker. This makes microseconds-precise time
available immediately after the timer connection switched to the modern
fine-grained mode of operation, which is a prerequisite for hardware drivers
that need such precision during their early initialization phase. The
calculations inside the framework also became more flexible to better fit the
characteristics of all the hardware and kernels Genode supports.
Finally, we were able to extend the application of the timeout framework. Most
notably, our C runtime uses it as timing source to the benefit of all
libc-using components. Another noteworthy case is the USB driver on the
Raspberry Pi. It previously couldn't rely on the default Genode timing but
required a local hardware timer to reach the precision that the host
controller expected from software. With the timeout framework, this workaround
could be removed from the driver.
FatFS-based VFS plugin
======================
Genode has supported VFAT file-systems since the 9.11 release when the
[http://elm-chan.org/fsw/ff/00index_e.html - FatFS] library was first ported.
The 11.08 release fit the library into the libc plugin architecture and
in 12.08 FatFS was used in the _ffat_fs_ file-system server. Now, the 17.08
release revisits FatFS to mold the library into the newer and more flexible
VFS plugin system. The _vfs_fatfs_ plugin may be fitted into the VFS server or
used directly by arbitrary components linked to the VFS library. As the
collection of VFS plugins in combination with the VFS file-system server has a
lower net maintenance cost than multiple file-system servers, the _ffat_fs_
server will be retired in a future release.
Enhanced GUI primitives
=======================
Even though we consider Qt5 as the go-to solution for creating advanced
graphical user interfaces on top of Genode, we also continue to explore an
alternative approach that facilitates Genode's component architecture to an
extreme degree. The so-called menu-view component takes an XML description of
a dialog as input and produces rendered pixels as output. It also gives
feedback to user input such as the hovered widget at a given pointer position.
The menu view does not implement any application logic but is meant to be
embedded as a child component into the actual application. This approach
relieves the application from the complexity (and potential bugs) of widget
rendering. It also reinforces a rigid separation of a view and its underlying
data model.
The menu view was first introduced in
[https://genode.org/documentation/release-notes/14.11#New_menu_view_application - version 14.11].
The current release improves it in the following ways:
* The new '<float>' widget aligns a child widget within a
larger parent widget by specifying the boolean attributes 'north', 'south',
'east', and 'west'. If none is specified, the child is centered. If opposite
attributes are specified, the child is stretched.
* A new '<depgraph>' widget arranges child widgets in the form of a
dependency graph, which will be the cornerstone for Genode's upcoming
interactive component-composition feature. As a prerequisite for
implementing the depgraph widget, Genode's set of basic graphical primitives
received new operations for drawing sub-pixel-accurate anti-aliased lines
and bezier curves.
* All geometric changes of the widget layout are animated now. This includes
structural changes of the new '<depgraph>' widget.
[image depgraph]
The menu-view component is illustrated by the run script at
_gems/run/menu_view.run_.
C runtime
=========
The growing number of ported applications used on Genode is accompanied by the
requirement of extensive POSIX compatibility of our C runtime. Therefore, we
enhanced our implementation by half a dozen features (e.g., O_ACCMODE
tracking) during the past release cycle. We thank the contributors of patches
and test cases and will continue our efforts to accommodate more ported
open-source components in the future.
Libraries and applications
##########################
Mesa adjustments
================
The Mesa update required the adaption of all components that use OpenGL.
In particular that means the Qt5 framework. Furthermore, we also enabled
OpenGL support in our SDL1 port.
As playground, there are a few OpenGL examples. The demos are located under
_repos/libports/src/test/mesa_demos_, which use the EGLUT bindings. There
are also some SDL based examples in the world repository under
_repos/world/src/test/sdl_opengl_.
Package management
==================
The previous release featured the initial version of Genode's
[https://genode.org/documentation/release-notes/17.05#Package_management - custom package-management tools].
Since then, we continued this line of work in three directions.
First, we refined the depot tools and the integration of the depot with our
custom work-flow ("run") tool. One important refinement is a simplification of
the depot's directory layout for library binaries. We found that the initial
version implied unwelcome complexities down the road. Instead of placing
library binaries in a directory named after their API, they are now placed
directly in the architecture directory along with regular binaries.
Second, driven by the proliferated use of the depot by more and more run
scripts, we enhanced the depot with new depot recipes as needed.
Third, we took the first steps to use the depot on-target. The experimentation
with on-target depots is eased by the new 'create_tar_from_depot_binaries'
function of the run tool, which allows one to assemble a new depot in the form
of a tar archive out of a subset of packages. Furthermore, the new
_depot_query_ component is able to scan an on-target depot for runtime
descriptions and returns all the information needed to start a subsystem based
on the depot content. The concept is exemplified by the new
_gems/run/depot_deploy.run_ script, which executes the "fs_report" test case
supplied via a depot package.
Platforms
#########
Genode as Xen DomU
==================
We want to widen the application scope of Genode by enabling users to easily
deploy Genode scenarios on Xen-based cloud platforms.
As a first step towards this goal, we enhanced our run tool to support running
Genode scenarios as a local Xen DomU, managed from within the Genode build
system on Linux running as Xen Dom0.
The Xen DomU runs in HVM mode (full virtualization) and loads Genode from an
ISO image. Serial log output is printed to the text console and graphical
output is shown in an SDL window.
To use this new target platform, the following run options should be defined in
the 'build/x86_*/etc/build.conf' file:
! RUN_OPT = --include boot_dir/$(KERNEL)
! RUN_OPT += --include image/iso
! RUN_OPT += --include power_on/xen
! RUN_OPT += --include log/xen
! RUN_OPT += --include power_off/xen
The Xen DomU is managed using the 'xl' command line tool and it is possible to
add configuration options in the 'xen_args' variable of a run script. Common
options are:
* Disabling the graphical output:
! append xen_args { sdl="0" }
* Configuring a network device:
! append xen_args { vif=\["model=e1000,mac=02:00:00:00:01:01,bridge=xenbr0"\] }
* Configuring USB input devices:
! append xen_args { usbdevice=\["mouse","keyboard"\] }
Note that the 'xl' tool requires super-user permissions. Interactive
password input can be complicated in combination with 'expect' and is not
practical for automated tests. For this reason, the current implementation
assumes that no password input is needed when running 'sudo xl', which can
be achieved by creating a file '/etc/sudoers.d/xl' with the content
! user ALL=(root) NOPASSWD: /usr/sbin/xl
where 'user' is the Linux user name.
Execution on bare hardware (base-hw)
====================================
UEFI support
------------
Analogously to our work on the seL4 and NOVA kernels in this release, we
extended our base-hw kernel to become a Multiboot2 compliant kernel. When used
together with GRUB2, it can be started on x86 UEFI machines missing legacy
BIOS support (i.e., CSM).
RISC-V
------
With Genode version 17.05, we updated base-hw's RISC-V support to privileged
ISA revision 1.9.1. Unfortunately, this implied that dynamic linking was not
supported on the RISC-V architecture anymore. Since dynamic linking is now
required for almost all Genode applications by default, this became a severe
limitation. Therefore, we revisited our RISC-V implementation - in particular
the kernel entry code - to lift the limitation of being able to execute only
statically linked binaries.
Additionally, we integrated the Berkeley Boot Loader (BBL), which bootstraps
the system and implements the machine mode, more closely into our build
infrastructure. We also added a new timer implementation to base-hw by using
the _set timeout SBI_ call of BBL.
What still remains missing is proper FPU support. While we are building the
Genode tool chain with soft float support, we still encounter occasions where
FPU code is generated, which in turn triggers compile time errors. We will
have to investigate this behavior more thoroughly, but ultimately we want to
add FPU support for RISC-V to our kernel and enable hardware floating point in
the tool chain.
Muen separation kernel
======================
Besides updating the Muen port to the latest kernel version as of end of June,
Muen has been added to Genode's automated testing infrastructure. This
includes the revived support for VirtualBox 4 on top of this kernel.
NOVA microhypervisor
====================
The current release extends NOVA to become a Multiboot2 compliant kernel. Used
together with GRUB2, NOVA can now be started on x86 UEFI machines missing
legacy BIOS support (called CSM).
GRUB2 provides the initial ACPI RSDP (Root System Description Pointer) to a
Multiboot2 kernel. The RSDP contains vital information required to bootstrap
the kernel and the operating system in general on today's x86 machines. To
make this information available to the user-level ACPI and ACPICA drivers, the
kernel propagates the RSDP to Genode's core, which - in turn - exposes it to
the user land as part of the _platform_info_ ROM module.
In order to ease the setup of an UEFI bootable image, we added a new image
module to our run-tool infrastructure. The run option 'image/uefi' can be used
instead of 'image/iso' in order to create a raw image that contains a EFI
system partition in a GUID partition table (GPT). The image is equipped by the
new 'image/uefi' module with the GRUB2 boot loader, a GRUB2 configuration, and
the corresponding Genode run scenario. The final image can be copied with 'dd'
to a bootable USB stick. Additionally, we added support to boot such an image
on Qemu leveraging [http://www.tianocore.org - TianoCore's] UEFI firmware.
As a side project, minor virtualization support for AMD has been added to
Virtualbox 4 and to the NOVA kernel on Genode. This enables us to run a 32-bit
Windows 7 VM on a 32-bit Genode/NOVA host on an (oldish) AMD Phenom II X4 test
machine.

898
doc/release_notes-18-02.txt
View File

@@ -1,898 +0,0 @@
===============================================
Release notes for the Genode OS Framework 18.02
===============================================
Genode Labs
After being developed for over a decade, Genode remained a mystery for many
people who looked at the project from a distance as it does not seem to fit
any established category of software. In 2018 - declared as the Year of Sculpt
on our [https://genode.org/about/road-map - roadmap] - this will hopefully
change. Genode 18.02 features the first revision of Sculpt, which is a
Genode-based operating system for general-purpose computing. After being used
as day-to-day OS by the entire team of Genode Labs for several months, we feel
that the time is right to share the system with a broader audience (Section
[Sculpt for Early Adopters]).
One fundamental feature of Sculpt is the ability to install and deploy
software from within the running operating system, which is universally
expected from any modern general-purpose OS. Section
[On-target package installation and deployment] presents Genode's unique
take on the topic of software installation and deployment.
Besides Sculpt, the current release has no shortage of other improvements.
Genode's growing arsenal of 3rd-party software received profound updates and
additions, including VirtualBox, Muen, seL4, several GNU packages, and
libraries. Also the user-level networking stack - including the Linux-based
LxIP stack and our custom NIC-router component - received a lot of attention.
Thanks to the added network driver for i.MX-based hardware, this networking
infrastructure becomes usable on embedded platforms based on this SoC.
Furthermore, the current release continues the cultivation of the Nim
programming language for Genode components.
Sculpt for Early Adopters
#########################
The current release features the first revision of Sculpt, which is a
Genode-based operating system for general-purpose computing. This initial
version is called Sculpt for Early Adopters (EA). Its target audience are
enthusiasts who are already familiar with Genode and are eager to use a
Genode-based operating system on their machines. As outlined on the
[https://genode.org/about/road-map - roadmap], later versions will become
increasingly approachable.
[image sculpt_overview]
Please refer to the official
[https://genode.org/documentation/articles/sculpt-ea - Sculpt documentation]
to step right into the adventure.
On-target package installation and deployment
#############################################
In May last year, we introduced the package-management concept for Genode to
pursue two goals. First, to overcome the naturally limited scalability of
composing Genode systems solely from source. This limit became evident in
complex system scenarios that incorporate a huge amount of 3rd party software.
Thanks to the introduced _depot_ concept and its integration in Genode's
workflow - in particular the run tool - the work of system integration became
much more structured (by caring about packages instead of individual build
targets), robust (by avoiding conditions in run scripts), and quick (by the
accelerated test cycle when using pre-built packages).
The second goal is the ability to update and extend a running Genode system on
the fly. We are happy to have reached this goal with the current release. As
exemplified by the Sculpt scenario, packages cannot only be used as building
blocks for system images but also as subsystems dynamically installed and
deployed on target. Even though installation and deployment are closely
related topics, both involve distinct challenges, which allow Genode to shine.
Installation / update
=====================
In traditional operating systems, the installation and update of system
software is the job of privileged programs. For example, a package manager in
a GNU/Linux system is typically executed with root privileges. This is
troublesome because the functionality of such a program is extremely complex.
In particular it is exposed to the network and has to parse content
originating from potentially untrusted parties. Therefore, potential software
vulnerabilities should be expected. However, in modern OSes, these programs
are just assumed to behave correctly. If this overly optimistic assumption
doesn't hold, the entire system is at risk.
Genode helps us to mitigate this problem by modelling each installation step
as a distinct component composition where each component has a well-defined
and extremely narrow role. The installation is an iterative sequence that
is orchestrated by the so-called download-manager component
(Figure [depot_download]).
[image depot_download]
Initially, the download manager receives a list of content to be installed
into the local depot, which is stored on the file system. The depot may
already be populated with (portions of) this content. In the first step, the
download manager must determine the parts that are missing. To do that, it
does not access the file system directly but instead hands over this task to a
disposable helper component called _depot-query_ that is spawned within a
dynamic init instance. This indirection has two benefits. First, the download
manager is not bothered with the complexity of accessing the file system. It
does not even have any notion of files. Second, the download manager is
effectively shielded from the file system. Should the file system misbehave,
the liveliness of the download manager remains unaffected.
[image depot_download_query_deps]
The depot-query component reports its findings to a report session. The report
eventually reaches the download manager as an updated ROM module. Given the
list of missing content, the download manager has to determine the information
of where to obtain the content from and the public key of the content creator.
This information is contained within the depot. So the download manager issues
another request to the depot-query component in order to obtain it.
[image depot_download_query_url]
Once the depot-query component has responded, the download manager knows what
content to get, where to get it, and how to verify it. To download the
content, it changes the dynamic init instance as follows.
[image depot_download_fetch]
The depot-query component is now gone. Actually, the entire depot has moved
out of sight. Instead, a fresh _fetchurl_ component is spawned. This component
is connected to the network as well as the writeable download directory
_public/_. Internally, fetchurl employs a complex software stack, which
includes the C runtime, curl, libssl, and libssh. Hence, we expect this
component to be vulnerable. Since it is facing the network, we assume that
vulnerabilities are exploitable. In the worst case where the component is
completely in the hands of an attacker, it may write wrong content into the
_public/_ location. But compared to executing curl or wget as root on a
traditional Unix system, the reach of an attack is quite limited. For example,
the mere existence of the download manager remains completely out of view of
fetchurl. However, the content of _public/_ must not be trusted. To reinforce
trust in the downloaded content, the content is accompanied with cryptographic
signatures created by the content creator. Before we touch the content, we
first check its authenticity. To perform this verification step, the download
manager reshapes the dynamic init instance as follows.
[image depot_download_verify]
Note that fetchurl exists no more and network connectivity is cut, effectively
disposing any form of malware that might have infected fetchurl. Next a new
_verify_ component enters the picture. It is configured with a list of content
to check, the signatures of the content, and the public key of the content's
presumed creator. Since it accesses the _public/_ location exclusively, it is
not prone to any potential time-of-check to time-of-use problems during the
verification. Under the hood, the _verify_ component employs a hugely complex
implementation based on GnuPG. It would be naive to fully trust this code.
However, when embedded in our scenario, the reach of a bug is limited because
the verify component has no access to any mutable system state. It could
merely give the wrong answer (which is of course bad but there is no way we
can magically solve this).
Knowing that the downloaded content is indeed the same content as intended
by the creator, it is time for extraction. For this step, the download
manager - again - reshapes the dynamic init instance:
[image depot_download_extract]
This time, both the _public/_ location as well as the trusted _depot/_ are
visible and a new _extract_ component is spawned. As the depot may host
content from multiple sources, which potentially distrust each other, the
content of each content provider resides in a dedicated subdirectory within
the depot. Instead of handing over access to the entire depot to the extract
tool, we mediate the file-system access via a _chroot_ component that limits
the view to the depot-provider's respective subdirectory. In the worst case
where a misbehaving content provider delivers a forged (but correctly signed)
archive to exploit a vulnerability of the extract component, the reach of the
attack remains limited to the content provider's space within the depot.
After the extraction step has completed, the depot is populated with the new
content, which may - in turn - include new dependency information. At this
point, the download manager starts a new iteration. This iterative process
terminates as soon as the depot-query component signals that no content of
the software installation is missing.
The bottom line here is that we are able to use complex and useful software
like curl, libarchive, liblzma, and GnuPG while largely distrusting it. In
contrast to this software that sums up to hundreds of thousand lines of code,
the download manager comprises less than 1000 lines of code. The software
installation procedure described above is implemented by the 'depot_download'
subsystem hosted in the gems repository and illustrated by an equally named
run script. It also forms the basis of the install/update mechanism of the
Sculpt scenario.
Deployment
==========
Once software has entered the system in the form of depot content, the
remaining question is how to turn this content into running subsystems. The
answer is given by the following illustration.
[image sculpt_deploy_runtime]
Like for the installation process described above, the scenario employs a
dynamic init instance that is accompanied by an orchestrating component. The
latter is called _depot-deploy_. The depot-deploy component queries
information from the depot using the same depot-query component that was used
during the installation. Based on the returned _blueprint_ information for the
to-be-deployed subsystems, it generates the configuration for the dynamic init
instance. The subsystems hosted within this init instance access the depot
content via mere ROM sessions as provided by the FS-ROM component. This makes
the use of the depot transparent to the hosted subsystems.
The depot-deploy component is located in the gems repository and accompanied
by a same-named run script. More importantly, it is featured in the deploy
runtime of the Sculpt system.
Base framework and OS-level infrastructure
##########################################
Increased default warning level
===============================
For building Genode components written in C++, the compiler flags -Wextra,
-Weffc++, and -Werror are now enabled in addition to -Wall by default.
If this strict warning level is inapplicable for a given component or
library, it is possible to explicitly disable the strictness in the
respective build-description file by adding the following line:
! CC_CXX_WARN_STRICT =
We adjusted almost all the code of the base, base-<kernel>, os, and demo
repositories to comply with this new warning level. For most components
hosted in the higher-level repositories (libports, ports, dde_*, gems),
the strictness is disabled as of now and will be enabled component-wise
wherever feasible.
While adjusting our code base, we identified the following patterns worth
mentioning:
* A class with virtual functions can no longer publicly inherit base
classes without a vtable. The inherited object may either be moved
to a member variable, or inherited privately. The latter would be
used for classes that inherit 'List::Element' or 'Avl_node'. In order
to enable the 'List' and 'Avl_tree' to access the meta data, the
'List' must become a friend.
* Instead of adding a virtual destructor to abstract base classes,
we inherit the new 'Interface' class, which contains a virtual
destructor. This way, single-line abstract base classes can stay
as compact as they are. The 'Interface' utility resides in
_base/include/util/interface.h_.
* With the new warning level, all member variables must be explicitly
initialized. Basic types may be initialized with '='. All other types
are initialized with braces '{ ... }' or as class initializers. If
basic types and non-basic types appear in a row, it is nice to only
use the brace syntax (also for basic types) and align the braces.
* If a class contains pointers as members, it must now also provide a
copy constructor and assignment operator. In most cases, one
would make them private, effectively disallowing the objects to be
copied. Unfortunately, this warning cannot be fixed by inheriting
our existing 'Noncopyable' class (the compiler fails to detect that
the inheriting class cannot be copied and still gives the error).
For now, we have to manually add declarations for both the copy
constructor and the assignment operator as private class members.
Those declarations should be prepended with a comment like this:
! /*
! * Noncopyable
! */
! Thread(Thread const &);
! Thread &operator = (Thread const &);
In the future, we plan to revisit these occurrences and try to replace
the pointers with references. In the presence of at least one
reference member, the compiler would no longer implicitly generate
a copy constructor. So we could remove the manual declaration.
The following caveats are expected, even if you disable the strictness
in your component:
* If your component has a class called 'Interface', it may collide with
the new 'Genode::Interface' class. You may have to disambiguate the
names.
* The 'Genode::Rpc_client' is no longer a 'Genode::Capability'. Hence,
classes inherited from 'Genode::Rpc_client' cannot refer to a
'Capability' but must refer to 'Genode::Capability'.
* The 'Surface' class is no longer copyable, which led to API
changes of users of this class. E.g., the 'Nitpicker_buffer'
utility does no longer offer accessors for the contained surfaces
but a new 'apply_to_surface' method that takes a lambda function as
argument.
Init
====
Init selects session routes based on the requested service and the client's
label. The latter can be matched as 'label' (exact match), 'label_prefix', or
'label_suffix' (either end of the label matches). With the new version, these
options are complemented with an additional 'label_last' attribute that covers
the prominent case where the last part of the label identifies a requested
resource at the server. A typical example is the routing of a ROM session
based on the name of the requested ROM module.
Reflecting the core log to the application level
================================================
Core records now log messages in a ring buffer and exports this
memory as ROM named 'core_log'. User applications may monitor this ring buffer
and present or transfer the content as appropriate. The example component in
_repos/os/src/app/log_core_ transforms the content into normal log
messages, which may be routed to graphical terminals or stored on
file systems, e.g. by using the fs_log server.
NIC-router improvements
=======================
During the past three months, the NIC router has received several improvements
that were mainly inspired by our daily experience with the component as part
of our Sculpt based working environments.
The most notable new feature is the support for multiple NIC sessions at one
domain. If multiple NIC-session clients connect to one domain, the NIC router
acts as a simple hub between them. I.e., for every packet that is routed to
the domain, each connected session receives a copy of the packet. The same
applies for domain-local packets, meaning packets that target an IP address
inside the IP subnet of the domain they came from. This domain-local
forwarding applies before considering any other routing rules. So, in other
words, it is not possible to route such traffic to another domain.
Furthermore, the logging features of the NIC router were improved. First, the
router is now capable of periodically sending a report via Genode's report
session. This can be activated by adding the new '<report>' node to the router
configuration:
! <config>
! <report interval_sec="5" bytes="yes" config="yes">
! ...
! </config>
So far, the report provides per-domain information about the amount of sent
and received data ('bytes' attribute) and the current IPv4 configuration like
IP address, subnet mask, and gateway address ('config' attribute).
Second, there is a new verbosity option in the '<config>' node:
! <config verbose_domain_state="yes">
When this option is set, the NIC router will output a short message to the log
for each general state change of a domain. Currently, this includes the
IP-configuration state (IP address, subnet mask, gateway address) and the
number of connected NIC sessions. This is a useful addition because the
purpose of the regular verbose option is to give a very deep insight into
almost every activity of the router, which is vital for debugging
sophisticated problems but normally floods the log. Therefore, the regular
verbose option is not viable for complex setups like a Sculpt desktop
environment. In such a context, the new domain-state verbosity is pretty
discreet but already gives a good hint on why, for instance, packets get
dropped despite the routing rules being correct.
Last but not least, the timeout configuration of the NIC router has been
reworked and now allows for a much more precise adaption to the network
environment. The former 'rtt_sec' attribute of the '<config>' node has been
replaced by the following new attributes (default values shown):
! <config dhcp_discover_timeout_sec="10"
! dhcp_request_timeout_sec="10"
! dhcp_offer_timeout_sec="10"
! udp_idle_timeout_sec="30"
! tcp_idle_timeout_sec="600"
! tcp_max_segm_lifetime_sec="30">
Details about the new attributes can be found in the
_os/src/server/nic_router/README_ file. The default values should be
appropriate for the common use case so that specifying them is normally not
necessary.
New watch mechanism for file-system session
===========================================
The file-system session already provided a way for watching files or
directories for changes. However, the original mechanism was arguably hard to
use. In addition to opening the to-be-watched file-system node, the client had
to submit a so-called content-changed request into the session's request
queue. In turn, the server delivered the change notification by acknowledging
this request.
The new mechanism is much less bureaucratic. A file or directory can be
watched by opening a watch handle rather than submitting a 'CONTENT_CHANGED'
packet to the server. Whenever a change happens at a node with an open watch
handle, a CONTENT_CHANGED packet will be sent from the server to the client.
This serializes the registration with other handle operations and separates
I/O handle state from notification handle state.
C runtime
=========
We changed libc's handling of 'clock_gettime' to be explicitly configurable
rather than relying on built-in heuristics. With the new version, the libc
opens a timer session as a time source only if the 'rtc' attribute of the
'<libc>' configuration node is defined. If not configured, 'clock_gettime'
returns 0.
This change may require the adjustment of components that implicitly rely on
the libc as time source. To enable such a component to use relative time
(based on a timer session) but no wall-clock time, one can manually provide a
pseudo real-time clock value as follows:
! <vfs>
! <dir name="dev">
! <log/> <null/> <inline name="rtc">2000-01-01 00:00</inline>
! </dir>
! </vfs>
! <libc stdout="/dev/log" stderr="/dev/log" rtc="/dev/rtc"/>
GUI stack and terminal improvements
===================================
Nit-FB improvements
-------------------
The nit_fb component provides a framebuffer and input service while using the
nitpicker GUI server as back end. The new version adds the 'initial_width' and
'initial_height' attributes, which accommodate the use case where nit_fb is
used in a dynamic fashion like as a client of a window system. Here, the
initial dimensions define the initial window size but - in contrast to the
existing 'width' and 'height' attributes - the actual size can change
afterwards.
Terminal resizing
-----------------
The terminal-session interface gained the ability to propagate resize events
from the server to the client. The new version of the graphical terminal uses
this mechanism to support window resizing as well as dynamically changing the
font size. At the client side, noux has become able to reflect terminal-size
changes to noux applications. Applications based on ncurses (e.g., vim) are
able to gracefully respond to such changes now.
Using chroot to enforce read-only file-system access
====================================================
By placing a chroot component in-between a file-system client and server, the
client's view on the file system can be limited to a specific directory. With
the current release, chroot can additionally be used to restrict a writeable
file-system session to become read-only. This is accomplished by the new
'writeable' attribute of chroot's policy nodes. By default, it is set to "no".
API changes
===========
Noncopyable AVL node/tree
-------------------------
Copying an AVL node generally violates the integrity of the corresponding
tree. To rule out subtle bugs where AVL nodes are accidentally copied, AVL
nodes are no longer copyable.
New 'Buffered_xml' utility
--------------------------
The 'Buffered_xml' utility located at _os/buffered_xml.h_ simplifies the
implementation of dynamically reconfigurable components that need to keep a
verbatim copy of certain parts of their configuration during configuration
updates.
New 'List_model' utility
------------------------
More and more components respond to dynamic configuration updates. For most
components, such updates are quite simple: replace an old internal state by a
new one. But in cases like init, menu_view, or window decorator, a
differential update is in order. Until now, each of these components employed
custom code for this task. As this code is not trivial, a common solution is
preferable. This solution comes in the form of the new 'List_model' utility
located at _base/include/util/list_model.h_. It introduces a light-weight
formalism to feed a component-internal data model from an externally-provided
XML structure.
Dynamically expandable reporter utility
---------------------------------------
In many cases, components that generate reports don't explicitly handle the
situation where the default buffer size of 4096 bytes is exceeded by the
report. This problem is easy to miss because reports are often small at
testing time but become larger when deployed in complex scenarios. In most
cases, the best way to handle an 'Xml_generator::Buffer_exceeded' exception is
upgrading the report session. The new 'Expanding_reporter' that accompanies
the original 'Reporter' in _os/reporter.h_ eases the handling of this common
case.
Languages and runtime environments
##################################
Nim programming language
========================
A new Nim library for constructing Genode servers is now available in the
World repository. This module provides utilities for the asynchronous
session-creation procedure introduced in the
[https://genode.org/documentation/release-notes/16.11#New_session-creation_procedure - 16.11]
release. Some introductory code snippets are provided here for the
adventurous.
An example of server creation using the 'genodeservers' module:
! import romclient, genodeservers
!
! var
! sessionsRom = newRomClient "session_requests"
! # synchronously open a ROM client to the parent
! romContent = sessionsRom.stream.readAll()
! # copy the ROM content to a heap string
! requestsParser = initSessionRequestsParser(romContent)
! # a state machine for parsing 'session_requests' XML
!
! for id, service, label in requestsParser.create:
! # the `create` iterator provider for the parser
! # hides the details of parsing the XML data
! discard txBufSize = requestsParser.argInt "tx_buf_size"
! # extract typed session arguments from the current parser state
! discard label.lastLabelElement()
! # label handling utilities are provided
! if service == "MyService":
! myCreateSessionProc(id, label)
!
This module streamlines the handling of session metadata, but the developer
must still provide hand-crafted wrappers over the C++ methods for managing
RPC objects and passing session capabilities to the parent. Most notoriously
a global pointer symbol, `genodeEnv`, is used to expose the component
environment object. In the future, this will be replaced by a typed object
passed from runtime to an application entry procedure.
! type MySessionCapability {.
! importcpp: "My_session::Session_capability",
! header: "my_session/capability.h".}
! # import a capability type
!
! type MyNativeSessionBase {.
! importcpp: "My_session::Session_rpc_object",
! header: "my_session/rpc_object.h".}
! # import C++ session RPC object
!
! type MyNativeSession = Constructible[MyNativeSessionBase]
! # apply the C++ Constructible template to defer calling
! # the object constructor
!
! proc construct(cppObj: MyNativeSession) {.
! importcpp: "#.construct(*genodeEnv)".}
! # call the C++ constructor, passing the global Genode::Env
!
! proc manage(cppObj: MyNativeSession): MySessionCapability {.
! importcpp: "genodeEnv->ep().manage(*#)".}
! # call a method from the gobal Env, dereferencing
! # thru the Constructible template
!
! type MyNimSessionObj = ref object
! cppImpl: MyNativeSession
! cap: MySessionCapability
! id: SessionId
! # C++ RPC objects are best kept in native
! # reference-counted Nim objects
!
! proc manage(obj: MyNimSessionObj) =
! obj.cppImpl.construct() # call our wrapped constructor
! GC_ref(obj)
! # manually increase the reference count on our session
! # object to prevent the component entrypoint from
! # referencing an RPC object that has been lost and
! # freed from the heap
! obj.cap = obj.cppImpl.manage() # store our capability
!
! proc myCreateSessionProc(id: SessionId): MyNimSessionObj =
! result = new MyNimSessionObj
! # create our object on the heap
! result.manage()
! # construct and manage our RPC object
! result.id = id
! # store the session id from our parent
Procedures for calling Nim code from an RPC object, dissolving
and destructing RPC objects, and managing the session lifetime
are exercises left to the reader.
Updated VirtualBox
==================
Our VirtualBox port got updated from version 5.1.22 to version 5.1.32 in order
to leverage the security updates and improved audio support. Additionally the
boot time of Linux guests got improved by adjusting our custom virtualization
back end.
Libraries and applications
##########################
New trace-logging component
===========================
The new trace-logger component can be used to easily gather, process, and
export different types of tracing data. Furthermore, it marks the next step
towards a user framework that makes access to Genode's manifold tracing
abilities
([https://genode.org/documentation/release-notes/13.08#Light-weight_event_tracing - 13.08],
[https://genode.org/documentation/release-notes/13.11#Improved_event_tracing - 13.11],
[https://genode.org/documentation/release-notes/15.08#Enhanced_tracing_facilities - 15.08])
intuitive and convenient.
The component can filter the available tracing subjects according to session
label policies and thread names. The processing of the tracing data can then
be configured for each selected subject individually, for groups of subjects,
or for all subjects together. The resulting data is exported as log output.
This is an example configuration of the trace logger, which shows the default
value for each attribute (except policy.thread and policy.label):
! <config verbose="no"
! session_ram="10M"
! session_arg_buffer="4K"
! session_parent_levels="0"
! period_sec="5"
! activity="no"
! affinity="no"
! default_policy="null"
! default_buffer="4K">
!
! <policy label="init -> timer" />
! <policy label_suffix=" -> ram_fs" />
! <policy label_prefix="init -> encryption -> "
! thread="worker"
! policy="null"
! buffer="4K" />
! </config>
The most important features so far when it comes to processing the traced
data are:
* Trace CPU activity and affinity ('activity' and 'affinity' attribute),
* Install individual policies for the creation of further tracing data
('policy' attributes) for instance, 'rpc_name' for a log of issued RPC calls),
* Dimensioning the subject-local trace buffers and the frequency of Trace Logger
data examination ('buffer' and 'period' attributes), and
* Configure the session to the Tracing server ('session' attributes).
A comprehensive documentation of the trace-logger component can be found in
_os/src/app/trace_logger/README_. An example of how to use the component is
given through the run script _os/run/trace_logger.run_.
New component for extracting archives
=====================================
The new 'extract' component located at _libports/src/app/extract_ extracts
the content of an arbitrary number of tar.xz archives according to its
configuration. It is used by the depot-download subsystem described in
Section [On-target package installation and deployment]. The component
is accompanied by the run script _libports/run/extract.run_ that illustrates
its use.
New signature-checking tool based on GnuPG
==========================================
The on-target installation of software packages requires a way to verify
cryptographic signatures of downloaded content within a Genode system.
The new 'verify' component located at _ports/src/app/verify_ facilitates the
code of GnuPG to verify detached OpenPGP signatures against public keys.
Since GnuPG depends on libgcrypt and libgpg-error, ports of those libraries
were added to the libports repository. The component comes with the run
script _ports/run/verify.run_ that demonstrates its usage.
Fetchurl component for downloading files
========================================
Fetchurl is a component for downloading files from the network, based
on the curl library. It used to reside in the genode-world repository.
Since it has become a mandatory part of Genode's on-target software
installation mechanism, we have moved it to the _libports_ repository now.
Besides this relocation, fetchurl received a welcome modernization. In
particular, the new version uses the modern socket-fs infrastructure of
the libc instead of relying on the deprecated libc_lwip plugin as a hard-wired
dependency.
New interactive FLIF image viewer
=================================
A simple image viewing application for the FLIF lossless image format was
written from scratch using the FLIF reference decoder library. The viewer can
be used to interactively view a directory of images and supports animation of
GIF-like FLIF files.
Ported 3rd-party software
=========================
With the current release, the following 3rd-party software becomes available
on Genode:
:[https://www.libarchive.org/ - libarchive]: is a library for uncompressing
and extracting various archive formats. It nicely wraps format-specific
libraries like zlib behind a unified and easy-to-use API. The port can
be found in the _libports_ repository.
:[https://lz4.github.io/lz4/ - lz4] and [https://tukaani.org/xz/ - liblzma]:
implement modern compression algorithms as supported by libarchive.
Thanks to Ben Larson for contributing the port of these libraries.
:[https://www.tcl.tk/ - Tcl]: is used as scripting language for various
Genode tools. With the new 'check_abi' tool described in Section
[Automated ABI consistency checks], the Tcl shell 'tclsh' has become
a dependency of the build system. Therefore, we made 'tclsh' available as
noux package. Note, however, that this port comprises solely the
functionality needed for simple scripting.
:[http://flif.info/ - FLIF]: is a library for the Free Lossless Image
Format. Thanks to Emery Hemingway for making it available in the
genode-world repository.
:[https://github.com/json-c/json-c/wiki - JSON-C]:
is a library for processing JSON-formatted data. Thanks to
Johannes Kliemann for contributing the port to the genode-world
repository.
:[https://www.nlnetlabs.nl/projects/ldns/ - Drill (ldns)]:
provides a utility for DNS testing. Thanks to Emery Hemingway for adding it
to the genode-world repository as a side activity of improving Genode's
network stack.
Updated packages for the Noux runtime environment
=================================================
The current release updates the following noux packages: less (version 487),
grep (version 3.1), coreutils (version 8.29), tar (version 1.30), findutils
(version 4.6), which (version 2.21), sed (version 4.4), and bash (version
4.4.18). Thanks to Hinnerk van Bruinehsen for this welcome contribution.
Device drivers
##############
Ethernet-driver for i.MX-based Wandboard
========================================
The current release contains a port of the Linux kernel driver for the
Ethernet card family originally produced by Freescale. We followed our
established approach to tailor an independent device-driver environment (DDE)
for the specific driver. To profit from synergies with the existing drivers of
the _dde_linux_ repository, we took the Linux kernel 4.4.3 as reference.
For now the current version is limited to support the Wandboard Quad as this
is the i.MX-based board that is nightly tested by our infrastructure. The
support of other boards using the same IP core is planned for future releases.
The driver can be found in _dde_linux/src/drivers/nic/fec_. To test the driver,
no further configuration is needed and you can have a look at one of the
automatic network tests, like _lwip.run_, as a reference.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Thanks to Johannes Schlatow from the TU Braunschweig, the support of the
Zynq-7000 boards by our base-hw kernel got extended. It is now possible to use
all CPU cores instead of only the primary one.
Updated Muen separation kernel
==============================
The Muen SK port has been updated to the latest development version 0.9. The
most notable features and improvements are the Crash Audit facility and support
for MirageOS/Solo5 subjects which may be executed alongside Genode/base-hw.
Thanks to this feature, the Muen project has reached a milestone by
self-hosting the [https://muen.sk] website on a Muen system. Currently, the
network driver is provided by a Linux subject but with some work it should be
possible to replace it with a Genode/base-hw nic_drv in the future.
Further details regarding Muen v0.9 can be found in the project's release
notes [https://groups.google.com/forum/#!topic/muen-dev/FPL9sc4yaBE].
Updated seL4 kernel
===================
Our remaining patches regarding UEFI framebuffer support got integrated into
the upstream codebase of the seL4 kernel. Hence, we updated our seL4 port to
the upstream version containing our patches.
Build system and tools
######################
Package management
==================
The package-management tools introduced last year have become a vital part
of Genode's workflow.
:Package management documentation:
[https://genode.org/documentation/developer-resources/package_management]
Prompted by the development of the on-target installation and deployment
mechanism featured in the current release, the tools received the following
refinements:
:Use of tar.xz as archive format: This change significantly reduces the size of
published depot content compared to the previously used tar.gz format.
:Subdirectories for archive versions:
In the original version of the depot layout, archives were named as
'<archive-name>-<version>'. Hence, the depot - in particular the download
location - had directories that grew in two dimensions. First, when new
archives were added. Second, when new versions of existing archives were
added (usually corresponding to Genode's release cycle). In the mid-term,
this would have resulted in a huge number of directory entries, e.g., in the
_src/_ subdirectory. To avoid this problem, the new version uses the scheme
'<archive-name>/<version>' instead. This way, at the _src/_ level, each
archive has one subdirectory (the number of subdirectories corresponds to
the number of archives). Inside the subdirectory, there is one entry per
version.
:Controlled rebuild of binary archives:
When calling the depot/create tool for a binary archive with 'FORCE=1', the
underlying source archives are re-extracted and the binary archive is
rebuilt. This is usually done after local changes in the source tree to
apply version updates to depot archives as needed. However, the implicit
rebuild is superfluous whenever the source-version remains the same. This is
particular inconvenient when re-creating pkg archives that refer to a large
number of src archives. Here, all binaries referenced by the pkg archive are
rebuilt each time. The new 'REBUILD' argument allows the user to skip
superfluous rebuilds in such situations. Normally, 'FORCE=1' implies
'REBUILD=1'. However, by explicitly specifying 'REBUILD=', existing binary
archives whose versions remain unchanged are kept instead of being rebuilt.
Offline validation of XML configurations
========================================
The _tool/run_ tool now automatically checks configurations against
target-specific XML schemes. Each component may define a configuration
scheme-file in its _target.mk_ file as follows:
! CONFIG_XSD = my_config.xsd
When the run tool checks the configuration of an instance of Genode's init
component, it additionally iterates through all start nodes of this
configuration. For each start node, it checks whether the according component
provides a configuration-scheme file and, if so, applies it to the
configuration inside the start node. This is done recursively. I.e., also the
child configurations of a sub-init of a sub-init ... of the top-level init
are covered this way.
Whenever the run tool detects an error in one of the checked configurations,
it stops and points out the location of the error. By now, there exist
configuration schemes for the init, the NIC router, and the trace logger
components. Our intention is that every component that interprets its
configuration will eventually be accompanied by such a scheme - not only to
validate actual configuration input but also to serve as documentation for
users of the component.
Automated ABI consistency checks
================================
In [https://genode.org/documentation/release-notes/17.02#Genode_Application_Binary_Interface - version 17.02],
we introduced a kernel-agnostic ABI, which ultimately paved the ground for
Genode's package management. For the time being, the ABI is not set in stone.
It is expected to evolve for some time until it hopefully approaches ABI
stability in the mid term. Whenever Genode's API changes, the ABI may be
affected. For example, symbol sizes may grow. Until now, side effects on the
ABI had to be curated manually. In practice, however, such side effects are
too easy to miss. Therefore, the current release adds a mandatory ABI checking
step to the build process. A new _tool/check_abi_ tool is invoked whenever a
shared object is built. It reports flaws in the ABI definition (such as
duplicated symbols) as well as inconsistencies between a shared object and its
ABI.

766
doc/release_notes-18-05.txt
View File

@@ -1,766 +0,0 @@
===============================================
Release notes for the Genode OS Framework 18.05
===============================================
Genode Labs
The driver behind the release 18.05 is the rapid evolution of the Sculpt
general-purpose OS. Following the initial version from February, which was
targeted at early adopters, the new Sculpt for The Curious (TC) introduces a
much more welcoming and empowering user experience (Section
[Sculpt for The Curious]).
It goes without saying that the interactive and dynamic nature of the Sculpt
scenario puts a lot more pressure on Genode's components compared to static
workloads. For example, Sculpt calls for the dynamic adjustment of user-level
network routing, the dynamic detection and management of partitions and file
systems, the support of USB storage devices from diverse vendors, and a way to
adapt the visual appearance to a great variety of screen resolutions. Most
improvements described below are our responses to these challenges.
That said, the release is not short of new features either. E.g., it features
the initial port of OpenJDK's HotSpot VM for executing Java programs on Genode
directly (Section [Java language support]), improves the support for the NXP
i.MX family of SoCs (Section [NXP i.MX SoC]), and enhances the VFS with new
plugins for copy-on-write and the auditing of file accesses
(Section [New VFS plugins]).
The release is complemented by the annual update of the Genode Foundations
book (PDF), which covers the fundamentals of the framework in great detail
(Section [New revision of the Genode Foundations book]).
Sculpt for The Curious
######################
With Sculpt for The Curious (TC), Genode 18.05 features the second revision of
the Sculpt general-purpose OS. Compared to the initial version for Early
Adopters (EA), it features a new interactive system-management component that
streamlines common tasks like the management of storage devices, or
configuring the network connectivity. The highlights of the new version of the
base system image are:
* Live-customization of almost all aspects of the system,
* The ability to install and run software in memory only,
* Hotplugging of USB storage devices,
* New support for NVMe storage devices in addition to SATA disks,
* Interactive network configuration including Wifi connectivity,
* Interactive management and inspection of storage devices and partitions,
* The option to host a complete and customized Sculpt installation on a
USB stick,
* Automated on-demand installation of software packages with visual feedback,
* Scalable fonts that are automatically adjusted to the screen resolution, and
* UEFI boot supported by default.
The base image is extensible by downloadable software packages that may
originate from different sources, safeguarded by cryptographic signatures.
It contains several example subsystems as a starting point:
* Basic GUI components like a window manager, a scalable backdrop, a
font server, and a simple software-rendering demo,
* A light-weight noux runtime for executing command-line-based software
such as GNU coreutils, bash, and vim.
* A package for downloading the installer and a suitable virtual-machine
configuration for Debian Linux,
* VirtualBox running Debian Linux,
* An example for running libretro-based games,
* A disposable VM that runs Firefox on TinyCore Linux, executed either in
VirtualBox or the light-weight Seoul virtual-machine monitor,
* A Qt5-based text editor.
Please refer to the updated
[https://genode.org/documentation/articles/sculpt-tc - Sculpt documentation]
to explore Sculpt TC.
The Sculpt version included with the current release requires the user to
build a boot image by hand. Following the steps described in the
documentation, this procedure takes a few minutes. We plan to provide
downloadable boot images a few weeks down the road once Sculpt TC received
intensive day-to-day testing by the early adopters. Your feedback is very
welcome!
New revision of the Genode Foundations book
###########################################
The "Genode Foundations" book received its annual revision, which reflects
the evolution of the framework over the past year. Specifically, the changes
since the last year's edition are:
: <div class="visualClear"><!-- --></div>
: <p>
: <div style="clear: both; float: left; margin-right:20px;">
: <a class="internal-link" href="https://genode.org">
: <img class="image-inline" src="http://genode.org/documentation/genode-foundations-title.png">
: </a>
: </div>
: </p>
* Changed boot-loader infrastructure on PC hardware
* Package management
* Structural changes of Genode's custom base-hw kernel
* API improvements: Unicode handling, support for XML-based data models,
timeout-handling API
: <div class="visualClear"><!-- --></div>
To examine the changes in detail, please refer to the book's
[https://github.com/nfeske/genode-manual/commits/master - revision history].
Storage infrastructure
######################
VFS library and plugin interface
================================
The VFS (Virtual-File-System) library was expanded to meet new requirements
for the Sculpt scenario. The traditional file-system medium for component
state and configuration sculpting is the *ram_fs* server, but with the
limitation that files stored in the server are ephemeral. Any changes to
the initial state are lost when a system is shut down or the *ram_fs* server
is restarted. Now that persistent storage is usually served by a VFS plugin
hosted by the VFS server, it was a natural progression to introduce a means
for indicating VFS changes with 'File_system' session notifications. To this
end the VFS server was amended to send session notifications, and notification
support was added to the Rump and FatFs VFS plugins, allowing Ext2 and FAT
file-systems to host dynamic component state and configuration information.
Using the VFS for serving font data produced from files stored in the VFS made
it practical to allow VFS plugins to introspect the file system. Plugins now
have the means to access arbitrary paths from the file-system root or they may
host and expose their own internal file systems.
While the core of the VFS library is small compared to contemporaries in other
operating systems, the moment came to promote the VFS from a static to a
shared library. Components that use the C runtime have always loaded the VFS
dynamically as a subsystem of _libc.lib.so_, but native components carried the
bulk of its implementation. The VFS library is now provided as a shared
library and is included with the front-end server in the _src/vfs_ depot
archive. This change affects components that have been rebuilt against the
shared library but do not have their ROM policies updated to allow access to
the _vfs.lib.so_ ROM.
New VFS plugins
===============
File-system introspection has made two additional plugins possible, the *audit*
and *cow* plugins.
The *audit* plugin logs VFS paths as they are accessed to a dedicated LOG
session. This is useful for finding the files required by third-party
components without relying on documentation or auditing source code.
The *cow* plugin emulates copy-on-write behavior by copying the contents of
files lying in a read-only path to a read-write path as they are opened. This
plugin is considered a proof-of-concept and under-performing, but opens a way
of experimenting with seeding user-managed file-systems from immutable
file-system archives.
Plugins of this kind are most appropriately instantiated in the VFS server
with policies to restrict the intended components into paths provided by the
plugins. This prevents a component from escaping the effect of the plugin. An
example of "auditing" a libc component follows:
! <start name="audit_fs">
! <binary name="vfs"/>
! <config>
! <vfs>
! <dir name="data"> <!-- source files -->
! <tar "data.tar"/>
! <ram/>
! </dir>
! <dir name="audit"> <!-- virtual path that captures /data -->
! <audit path="/data"/>
! </dir>
! </vfs>
! <!-- route into virtual audit path -->
! <policy label_suffix="audit" root="/audit" writeable="yes"/>
! </config>
! </start>
!
! <start name="app">
! <config>
! <libc stdout="/log" stderr="/log"/>
! <vfs>
! <log/>
! <fs label="audit"/>
! </vfs>
! </config>
! </start>
Improved disk-partition discovery and access
============================================
The 'part_blk' component, which parses the partition table on a block device
and provides access to each partition through a block session, was extended to
make it easier to implement a management component on top of it. It now
features additional attributes in its report. For one the block size of each
partition as well as the type of the file system on the partition are
reported. The file system probing implementation is minimal and only contains
file systems that are commonly used on Genode systems, i.e., FAT32 and Ext2.
Furthermore, on GPT formatted disks, each partition has an 'expandable'
attribute that contains the number of blocks by which the partition can be
grown. The following exemplary report illustrates the adjustments:
!<partitions type="gpt" total_blocks="500118192" gpt_total="500118125" gpt_used="302254080">
! <partition number="1" name="BIOS boot partition"
! type="21686148-6449-6e6f-744e-656564454649" guid="db0701aa-02ae-474d-92d0-82738bfce5d2"
! start="2048" length="2048" block_size="512"/>
! <partition number="2" name="EFI System"
! type="c12a7328-f81f-11d2-ba4b-00a0c93ec93b" guid="74e43226-2afb-4575-bdda-83bf72f5a6e7"
! start="4096" length="262144" block_size="512" file_system="FAT32"/>
! <partition number="3" name="GENODE"
! type="0fc63daf-8483-4772-8e79-3d69d8477de4" guid="a950091d-87ba-4800-85bf-7b6a58abe6d5"
! start="235147264" length="67108864" block_size="512" file_system="Ext2"
! expandable="197862064"/>
!</partitions>
The heuristics of how the component probes the partition table were also
loosened. Instead of explicitly enabling support for GPT, the component will
now always try to parse the MBR as well as the GPT. It will bail out if both
are considered valid since using GPT/MBR hybrid tables is not supported and it
should be up to the user to make an educated decision. In cases where there is
no partition table, a 'partitions' report of 'type="disk"' will be generated
in which the complete disk is presented as partition number '0'. This is
needed as compatibility fallback for Sculpt EA installations.
Creating and modifying GUID partition tables
============================================
Part of the enhancements of Sculpt TC is the ability to manipulate the block
device used by Sculpt. We implemented a component called 'gpt_write', which
can create and modify a GPT and its entries. It considers alignment
constraints to make better use of 512e devices. It will, however, not perform
any boundary checking. It does not handle overlapping partitions and only when
applying a partition, it makes sure that the partition will fit. The following
configuration illustrates its operation:
!<start name="gpt_write">
! <resource name="RAM" quantum="2M"/>
! <config verbose="yes" initialize="yes" align="4K">
! <actions>
! <add entry="1" type="BIOS" label="GRUB BIOS" start="2048" size="1M"/>
! <add entry="2" type="EFI" label="EFI System" start="4096" size="16M"/>
! <add entry="3" type="Linux" label="GENODE" start="36864" size="128M"/>
! <add type="BDP" label="FAT32 Data" size="max"/>
! <delete entry="1"/>
! <delete label="FAT32 Data"/>
! <modify label="GENODE" new_label="GENODE*" new_size="max"/>
! </actions>
! </config>
!</start>
Please read _repos/gems/src/app/gpt_write/README_ for more detailed information
on how to use the component and feel free to check out the run script
_repos/gems/run/gpt_write.run_.
User-level networking
#####################
NIC router
==========
The NIC router has received major improvements that were mainly motivated by
our daily experience with the Sculpt scenario where the router serves as NAPT
component in front of the virtual machines that host our work OS's. In this
role, it is subject to a permanent load driven by real-world tasks.
Furthermore, it has to have a user interface that makes it a pleasant
experience to deploy in a dynamic environment. This led to our primary goal:
We had to overcome the need to restart the NIC router, and thereby all
components that depend on it, whenever its configuration changes and while
doing so, not to interrupt the communication of its client unnecessarily.
We managed to make the NIC router fully re-configurable at runtime in a way
that it always tries to keep as much state information as possible throughout
the process. This means that network communication going through the NIC
router is not affected by a configuration update unless the configuration
change affects parts that were involved in an existing communication channel.
One prerequisite for this feature was that NIC session clients can connect at
any time to the NIC router regardless of whether there is a matching domain
for the session or not. As long as a session has no domain, the NIC router
does not send any packet to it and drops all packets coming from it. But, at
least, the session and the corresponding client component stay alive, even if
their already assigned domain disappears with a new configuration.
At the uplink, in contrast, the lifetime of the session remains bound to the
lifetime of the domain. The uplink domain-tag received a new attribute
named 'label' (only considered at the domain-tag of the uplink). It denotes
the label of the uplink session. With these two particularities of the uplink
domain, one can now easily switch between different NIC session servers. The
NIC router will close and request the corresponding NIC session with the
current 'label' value if the 'domain' node is removed/added or the label
changes. Thereby, the NIC router can now be used to dynamically switch between
network interfaces like wireless and wired adapters.
Furthermore, we improved the NIC router's ability to handle DNS server
information. Domains can wait for the DNS server info of the DHCP client of
another domain. This is done with the new attribute 'dns_server_from' in the
'<dhcp_server>' tag. Each time the DNS server info of the remote domain
changes, the DHCP server with the 'dns_server_from' attribute will toggle the
link state of each session at its domain. This can be used by clients as a
hint to request their DHCP info anew from the NIC router and thereby receive
the updated DNS server information.
When it comes to protocols, the most notable change is that the NIC router now
also supports routing and NAPT for ICMP. With the new '<icmp>' sub node of the
'<domain>' tag, ICMP routes to other domains can be created. Instead of ports,
the ICMP IDs are used for NAPT. Similar to the 'udp-ports' and 'tcp-ports'
attributes, the size of the ID space for each NAPT client is configured via
the new 'icmp-ids' attribute in the '<nat>' tag.
Last but not least, the following small features were also added to the NIC
router:
:Attribute 'verbose_packets' for the '<config>' and the '<domain>' node:
Toggles the logging of most important protocol header fields globally or
domain-locally. The 'verbose' attribute does not affect this kind of debug
output anymore.
:Report DNS server info:
If the 'config' attribute in the '<report>' node is enabled, the NIC router
will now also report the DNS server info for each domain.
:Attribute 'config_triggers' in the '<report>' node:
Toggles whether the NIC router immediately sends a report whenever the IPv4
configuration of a domain changes, regardless of any timeouts.
:IPv4 point-to-point support:
If a domain receives an IP configuration with a subnet mask of
255.255.255.255 it will switch to point-to-point IPv4 (requires a valid
gateway address at the domain).
:ICMP destination unreachable on non-routable packets:
The NIC router now responds with an ICMP "destination unreachable" packet to
packets that are not routable at an interface with a domain.
For more information, have a look at the _os/src/server/nic_router/README_
file. Examples can be found in the run scripts
_dde_linux/run/nic_router_uplinks.run_,
_libports/run/nic_router_dyn_config.run_, and _os/run/ping_nic_router.run_.
NIC dump
========
The output level of the NIC dump component can now be configured per protocol
by using the protocol names as attributes: 'eth', 'arp', 'ipv4', 'dhcp',
'udp', 'icmp', and 'tcp'.
The available debug levels are:
:no: Do not print out this protocol.
:name: Print only the protocol name.
:default: Print a short summary of the most important header values.
:all: Print all available header values.
Additionally, you can set a default debug level for protocols that are not
configured using the 'default' attribute.
For more information, please refer to _os/src/server/nic_dump/README_.
GUI stack
#########
With Sculpt becoming more and more end-user oriented, Genode's GUI stack came
into focus. It was time to reconsider several interim solutions that worked
well in the past but would not scale up to a modern general-purpose OS. Two
concrete examples are the support of scalable fonts and Unicode characters. In
the past, Genode used to restrict textual output to the Latin-1 character set
and employed pixel-based fonts only. The current release overcomes these
limitations by featuring completely new text-output facilities.
UTF-8 support and improved text rendering
=========================================
The UTF-8 text encoding overcomes the severely limited code-point range of the
ASCII and Latin-1 character sets by representing characters by a varying
number of bytes. Today, UTF-8 is generally considered as the standard encoding
for text. The new UTF-8 decoder at _os/util/utf8.h_ clears the path for
Genode's native GUI components to follow suit. The first beneficiary is
Genode's graphical terminal, which has become able to display Unicode
characters and pass user input as UTF-8-encoded data to its terminal-session
client.
Terminal enhancements
=====================
Speaking of the graphical terminal, the current incarnation got a welcome
overhaul. First, we reduced its complexity by removing obsolete features like
built-in keyboard-layout handling, which are no longer needed when combining
the terminal with our modern input-filter component. Furthermore, the terminal
has become dynamically resizeable, forwarding screen-size changes to the
terminal client. Should the client be a Noux runtime, such a change is
reflected to the running application as a SIG_WINCH signal. The application -
e.g., Vim - responds to the signal by requesting the new terminal size.
Finally, the terminal protocol was changed from 'linux' escape sequences to
'screen' escape sequences in the anticipation of making the terminal more
flexible in the future.
Text rendering
==============
Throughout Genode, many GUI components reused the text-output utilities
of the nitpicker GUI server. These utilities, however, relied on a simple
pixel font format. To make the text output more flexible, nitpicker's text
painter located at _nitpicker_gfx/text_painter.h_ has been replaced by a
completely new implementation that decouples the font format from the
glyph rendering and takes UTF-8 strings as input. In the process, the glyph
rendering got a lot more sophisticated, supporting horizontal sub-pixel
positioning and filtering.
Font-format support
===================
To remove the omnipresent use of fixed-size pixel fonts throughout Genode,
the following new components entered the picture:
First, the new 'ttf_font' library implements nitpicker's font interface by
using the TrueType renderer of the STB single-header library.
Second, the new 'vfs_ttf' VFS plugin uses the 'ttf_font' library to export a
rendered TrueType font as a virtual file system. The various font properties
as well as the actual glyph images become accessible as regular files. This
way, an application that needs to draw text can read the glyph data directly
from its VFS instead of depending on a font-rendering library.
Third, the new 'Vfs_font' utility located at _gems/include/gems/vfs_font.h_
implements nitpicker's font interface by obtaining the glyphs from the
component-local VFS. It is complemented by the 'Cached_font' utility, which
implements an LRU glyph cache.
With this infrastructure in place, several existing GUI components could
be updated, most prominently the graphical terminal and the menu-view
widget-rendering engine. By facilitating the VFS as interface for propagating
glyph data, components no longer need to manage fonts and their configuration
individually. They just access their VFS. When integrating the component into
a scenario, one can decide whether to mount a font-rendering library directly
at the component, or - alternatively - route a file-system session to a
central font server. The latter is just a regular VFS server with the fonts
mounted as pseudo file systems. Since the glyph renderer is a VFS plugin, it
could be replaced by another implementation in the future without touching any
component.
Modernized API for input-event processing
=========================================
Genode's input-session interface changed very little over the years. Even
though it received evolutionary enhancements from time to time, its design
resembled a traditional C-style interface from the medieval era. We found that
the interface left too much room for interpretation. In particular, the meta
data per event type was defined in a rather ad-hoc way, which raised
uncertainties. For example, is a button-press event accompanied with a
positional value or not? To remove these uncertainties, the current release
replaces the 'Input::Event', with a new implementation that facilitates a safe
way of accessing event meta data. Besides this design change, there is one
noteworthy semantic change as well. With the new interface, symbolic character
information are provided along with their corresponding press events rather
than as distinct events, which - according to our practical findings - greatly
simplifies the consumer side of the 'Input::Event' interface.
Improved keyboard-focus handling
================================
The nitpicker GUI server multiplexes one screen among multiple GUI clients in
a secure way. One aspect remained underdeveloped so far, which is the keyboard
focus handling. Nitpicker's 'Session:focus' call previously triggered a one-off
focus change at call time. This focus change did not pass the same code paths
as a focus change triggered by a "focus" ROM update, which led to
inconsistencies.
The new version changes the implementation of 'Session::focus' such that the
relationship of the caller and the focused session is preserved beyond the
call time. Whenever the calling session is focused in the future, the
specified session will receive the focus instead. So 'Session::focus' no
longer represents a single operation but propagates the information about the
inter-session relationship. This information is taken into account whenever
the focus is evaluated regardless of how the change is triggered. This makes
the focus handling in scenarios like the window manager more robust.
Device drivers
##############
NVMe storage devices
====================
Since NVMe devices have become common in contemporary systems, it is time to
provide a driver for such devices on Genode. With this release, we introduce a
component that is able to drive consumer-grade NVMe storage devices, i.e.,
there is no support for namespace management or other enterprise-grade
features. For now, to keep things simple, the driver uses the device in an
old-fashioned way and uses only one I/O queue with at most 128 entries. That
is to say it does not exploit the parallelism necessary to unlock the full
potential of NVMe storage. Nonetheless, it performs well. The following
snippet illustrates its configuration:
!<start name="nvme_drv">
! <resource name="ram" quantum="8M"/>
! <provides><service name="Block"/></provides>
! <config>
! <report namespace="yes"/>
! <policy label_prefix="client1" writeable="yes"/>
! </config>
!</start>
The component will generate a report, which contains all active namespaces, if
reporting is enabled by setting the 'namespace' attribute of the '<report>'
node to 'yes'. A report may look like the following example:
!<controller model="QEMU NVMe Ctrl" serial="FNRD">
! <namespace id="1" block_count="32768" block_size="512"/>
!</controller>
For an example on how to integrate this component, please have a look at the
_repos/os/run/nvme.run_ script.
While implementing the NVMe driver, a new component for testing block-sessions
was used. In contrast to the already existing 'blk_bench' and 'blk_cli'
components, it features a variety of different test patterns, which can be
selected in its configuration and can be used to test a block component more
thoroughly. For more information please refer to
_repos/os/src/app/block_tester/README_
NXP i.MX SoC
============
We extended the Linux kernel driver port for Ethernet cards found in NXP i.MX
SoC, which was introduced in the previous release. Now does it not only
support i.MX6Q SoC based boards like the Wandboard, but the i.MX53 and i.MX6SX
SoC as well. The new driver was successfully tested with the i.MX53 Quick
Start Board and the Nitrogen6 SOLOX. The latter board even contains two
Ethernet cards. But due to technical limitations of the board design, the same
driver instance has to be used for both cards. Currently, the driver is
tweaked to run on different boards via its configuration ROM. When no
configuration is provided, it appropriates the values for successfully
executing on the Wandboard. The following is an example configuration for the
i.MX53:
! <config>
! <card name="fec0" type="fsl,imx25-fec" mii="rmii" irq="87" mmio="0x63fec000"/>
! </config>
As a side effect of enabling networking on the Nitrogen6 SOLOX, support for
GPIO based signals has been added to the framework too. The existing GPIO
driver for i.MX53 SoC got extended to additionally support the i.MX6 family.
There are some known limitations when using different drivers like Ethernet
and SD-card drivers on the Wandboard right now. At the moment, those drivers
adjust clock parameters and I/O pin configurations independently from each
other, which can lead to inconsistencies. We plan to address those issues with
the implementation of a platform driver for the i.MX6 SoC family.
Improved USB-storage driver
===========================
We improved the stability of the USB-storage driver (usb_block_drv) and
made it compatible with a lot more devices as the driver has become a pivotal
ingredient of the Sculpt scenario. Due to the changes, the way the driver
operates has changed. On the one hand, now it first tries to use 10-byte
Command Descriptor Blocks (CDB) in its SCSI layer and will only switch to
16-byte CDBs when it encounters a device whose blocks cannot be completely
accessed via the former descriptor size. On the other hand, because some
tested devices stopped working after issuing a USB device reset, the reset was
made optional. By setting the 'reset_device' attribute in the '<config>' node
to 'yes', the driver is instructed to perform the USB device reset.
Libraries and applications
##########################
Packaged Qt5 framework
======================
We created package recipes for all previously ported Qt5 libraries and their
dependencies and adapted the run scripts accordingly. Please note that the
host tools needed for building Qt applications (moc, rcc, uic) are not built
automatically anymore, but need to be built and installed manually with the
new 'tool/tool_chain_qt5' script.
Java language support
=====================
Over the course of the past year, we started to look into Java support for
Genode with the ultimate goal of porting an existing Java Virtual Machine
(JVM), which translates and executes Java byte code, to Genode. After
investigating possible JVM candidates, it became obvious that
[http://openjdk.java.net - OpenJDK] is the only viable option when looking for
a functional, maintained, feature complete, and open-source Java SDK.
Therefore, we decided upon OpenJDK version 9 and started to port OpenJDK's
HotSpot virtual machine.
In the first step, we followed the approach to enable HotSpot's internal
Just-in Time (JIT) compiler, which translates byte code into machine code and
is the option with the most to offer performance wise. But we also wanted
support for ARM platforms and soon realized, there was almost no JIT compiler
support for ARM other than for Linux. The Linux version is deeply integrated
into the Linux system libraries (e.g., glibc), which makes it very hard to
bring the compiler onto Genode. For example, Genode uses FreeBSD's libc and
that would now have to offer glibc semantics.
After additional research, we found the so-called interpreter version of the
HotSpot VM. This version does not compile byte code, but interprets and
emulates the code at runtime. It is of course slower than the JIT compiler
version, but also machine-architecture independent, so the same HotSpot VM can
be compiled for x86 and ARM platforms. With the JVM running on Genode, we
added networking and file-system access support via Genode's VFS layer. Note,
there is no graphical toolkit support as of now, but most standard library
classes should work. Also, the byte code has to be compiled on a different
host system (e.g., Linux, *BSD) as of now, since we did not bring the Java
compiler to Genode.
To give Java a spin, a run script can be found under _ports/run/java.run_.
Ada language support
====================
Support for components and libraries written in the Ada/SPARK programming
language experienced a rework with the final goal of seamless integration with
the base framework. We added a new _ada_ library, which contains a (currently
minimal) runtime taken from the sources of our GCC port and thus is always
consistent with the tool chain in use. It is built as a shared library
_ada.lib.so_ that needs to be added to the list of boot modules.
The example in _libports/src/test/ada_ showcases the implementation of an Ada
component using a custom library _test-ada_, which is also implemented in Ada.
Seoul VMM on NOVA
=================
The Seoul/Vancouver VMM - introduced to Genode with release 11.11 - received
some renovations to be able to run recent Linux VMs. Namely the output of the
guest during early boot is now visible and the network models got revised.
Additionally, the Seoul VMM has been packaged and can be used in Sculpt.
Ported software
===============
The [https://dnsprivacy.org/wiki/display/DP/DNS+Privacy+Daemon+-+Stubby - Stubby]
DNS daemon has been ported to begin experimentations with DNS as a native
service. There is a tendency for DNS configuration frameworks to diverge
between operating systems and releases, an inconvenience that is magnified
when maintaining virtual machines. Name-server configuration via DHCP has been
the only constant, so hosting DNS natively and configuring virtual-machines
with the *nic_router* DHCP server presents itself as a viable solution to the
guest resolver quagmire. Expect DNS services in later Sculpt releases.
Platforms
#########
Accessing PCI via ECAM/MMCONF
=============================
The platform driver on x86 is trusted with guarding access to PCI
devices. Up to now, I/O ports have been used to configure the PCI subsystem.
On modern x86 architectures, PCI devices can be configured by using Memory
Mapped I/O (MMIO). This method was introduced with PCI Express and is called
Enhanced Configuration Access Mechanism (ECAM). For Each PCI device a separate
4 KiB MMIO page exists to serves as the configuration interface between OS and
PCI device.
The exact location of all the 4K MMIO pages of the PCI devices is machine
specific and must be determined during the bootstrap phase. The ACPI driver on
Genode is in charge of this procedure and reports the location of the
ECAM/MMCONF region to the platform driver via the 'acpi' ROM.
Besides using a modern PCI interface, switching to ECAM/MMCON served to ease
the execution of Genode/hw on top of the Muen separation kernel.
Kernel-agnostic platform-information handling
=============================================
Up to now, special kernel-specific information was propagated to components
such as Virtualbox, the Seoul VMM, and the timer by reusing the
kernel-provided data structures. For Genode/NOVA, the hypervisor info page
(HIP) was exported as an ordinary Genode ROM. With the rise of Sculpt and the
packaging of components in a - as far as possible - kernel-independent way,
the propagation of kernel-specific information became a stumbling block.
With this release we abandon the 'hypervisor_info_page' ROM of Genode/NOVA and
replace it with a Genode ROM called 'platform_info'. The 'platform_info' ROM
is planned to contain solely information about the host hardware, which may
not be gathered otherwise by Genode components. In the current state it
contains information required by VMMs, namely whether AMD SVM or Intel VMX is
available and usable. Additionally, the ROM contains information about the
frequency of the time stamp counter.
Updated seL4 kernel to version 9.0.1
====================================
Thanks to Hinnerk van Bruinehsen, the seL4 version used by Genode has been
updated to 9.0.1.
Updated Muen separation kernel
==============================
With the addition of memory-mapped access to the PCI config-space in Genode,
base-hw subjects on Muen now only see the effectively assigned physical
devices. This makes it possible to run Genode in parallel with other subjects
and to pass-through different PCI devices for each instance.
The Muen update also brings a much simplified subject info structure plus some
tweaks to the Muen system policy XML format to facilitate easier integration
of new hardware platform specifications.
Build system and tools
######################
Validating 3rd-party code downloads via SHA256
==============================================
This release removes support for verifying source code of third-party ports
with the SHA1 hash algorithm. Last year, SHA1 was banished as a credible
cryptographic hash function after the demonstration of a full collision
attack. Since the
[https://genode.org/documentation/release-notes/14.05 - 14.05 release],
port files have been verified using SHA1, this release replaces all file
digests with SHA256 digests. Any port definitions maintained in external
repositories are required to make these replacements as well. No collisions
have been discovered against source code archives but nonetheless there is an
obligation to widen our margin of safety.
Creating GPT-based disk images by default
=========================================
Up to now Genode's run tool was able to create x86 bootable images in three
flavours:
* Either as ISO bootable by BIOS legacy - 'image/iso', or as
* GPT partitioned disk image only bootable by UEFI - 'image/uefi', or as
* MBR partitioned disk image only bootable by BIOS legacy - 'image/disk'.
With Sculpt came the demand to have a single image type that is in principle
bootable by both UEFI and BIOS legacy. Additionally with Sculpt, we began to
prefer working with GPT partitioned devices.
In the light of the new demands, we changed the 'image/disk' run tool support
to create a GPT partitioned disk image bootable by a legacy BIOS and by UEFI.

1075
doc/release_notes-18-08.txt
View File

File diff suppressed because it is too large Load Diff

830
doc/release_notes/08-11.txt Normal file
View File

@@ -0,0 +1,830 @@
==============================================
Release notes for the Genode OS Framework 8.11
==============================================
Genode Labs
Summary
#######
This document presents the new features and major changes introduced
in version 8.11 of the Genode OS Framework. It is geared towards
people interested in closely following the progress of the Genode
project and to developers who want to adopt their software to our
mainline development. The document aggregates important fragments
of the updated documentation such that you won't need to scan existing
documents for the new bits. Furthermore, it attempts to provide our
rationale behind the taken design decisions.
The general theme for the release 8.11 is enabling the use of the
Genode OS framework for real-world applications. Because we regard
the presence of device drivers and a way to reuse existing library
code as fundamental prerequisites for achieving this goal, the major
new additions are an API for device drivers written in C, an API for
handling asynchronous notifications, and a C runtime. Other noteworthy
improvements are the typification of capabilities at the C++-language
level, a way for receiving and handling application faults, the
introduction of managed dataspaces, and a new API for scheduling
timed events.
Base framework
##############
This section documents the new features and changes affecting the
'base' repository, in particular the base API.
New features
============
Connection handling
~~~~~~~~~~~~~~~~~~~
The interaction of a client with a server involves the definition of
session-construction arguments, the request of the session creation via
its parent, the initialization of the matching RPC-client stub code
with the received session capability, the actual use of the session
interface, and the closure of the session. A typical procedure of
using a service looks like this:
!#include <rom_session/client.h>
!...
!
!/* construct session-argument string and create session */
!char *args = "filename=config, ram_quota=4K");
!Capability session_cap = env()->parent()->session("ROM", args);
!
!/* initialize RPC stub code */
!Rom_session_client rsc(session_cap);
!
!/* invoke remote procedures, 'dataspace' is a RPC function */
!Capability ds_csp = rsc.dataspace();
!...
!
!/* call parent to close the session */
!env()->parent()->close(session_cap);
Even though this procedure does not seem to be overly complicated,
is has raised the following questions and criticism:
* The quota-donation argument is specific for each server. Most services
use client-donated RAM quota only for holding little meta data and,
thus, are happy with a donation of 4KB. Other services maintain larger
client-specific state and require higher RAM-quota donations. The
developer of a client has to be aware about the quota requirements for
each service used by his application.
* There exists no formalism for documenting session arguments.
* Because session arguments are passed to the 'session'-call as a plain
string, there are no syntax checks for the assembled string performed
at compile time. For example, a missing comma would go undetected until
a runtime test is performed.
* There are multiple lines of client code needed to open a session to
a service and the session capability must be maintained manually for
closing the session later on.
The new 'Connection' template provides a way to greatly simplify the
handling of session arguments, session creation, and destruction on the
client side. By implementing a service-specific connection class
inherited from 'Connection', session arguments become plain constructor
arguments, session functions can be called directly on the 'Connection'
object, and the session gets properly closed when destructing the
'Connection'. By convention, the 'Connection' class corresponding to a
service resides in a file called 'connection.h' in the directory of the
service's RPC interface. For each service, a corresponding 'Connection'
class becomes the natural place where session arguments and quota
donations are documented. With this new mechanism in place, the example
above becomes as simple as:
!#include <rom_session/connection.h>
!...
!
!/* create connection to the ROM service */
!Rom_connection rom("config");
!
!/* invoke remote procedure */
!Capability ds_csp = rom.dataspace();
[https://genode.org/documentation/api/base_index#Connecting_to_services - See the API documentation for the connection template...]
Typed capabilities
~~~~~~~~~~~~~~~~~~
A plain 'Capability' is an untyped reference to a remote object of any
type. For example, a capability can reference a thread object or a
session to a service. It is loosely similar to a C void pointer, for which
the programmer maintains the knowledge about which data type is actually
referenced. To facilitate the type-safe use of RPC interfaces at the C++
language level, we introduced a template for creating specialized
capability types ('Typed_capability' in 'base/typed_capability.h') and
the convention that each RPC interface declares a dedicated capability
type. Note that type-safety is not maintained across RPC interfaces. As
illustrated in Figure [layered_ipc], typification is done at the
object-framework level on the server side and via in the 'Connection'
classes at the client side.
[image layered_ipc]
From the application-developer's perspective, working with capabilities
has now become type-safe, making the produced code more readable and robust.
[https://genode.org/documentation/api/base_index#Capability_representation - See the updated API documentation for the capability representation...]
Fifo data structure
~~~~~~~~~~~~~~~~~~~
Because the 'List' data type inserts new list elements at the list head,
it cannot be used for implementing wait queues requiring first-in
first-out semantics. For such use cases, we introduced a dedicated
'Fifo' template. The main motivation for introducing 'Fifo' into the
base API is the new semaphore described below.
[https://genode.org/documentation/api/base_index#Structured_data_types - See the new API documentation for the fifo template...]
Semaphore
~~~~~~~~~
Alongside lock-based mutual exclusion of entering critical sections,
organizing threads in a producer-consumer relationship via a semaphore
is a common design pattern for thread synchronization. Prior versions
of Genode provided a preliminary semaphore implementation as part of
the 'os' repository. This implementation, however, supported only one
consumer thread (caller of the semaphore's 'down' function). We have
now enhanced our implementation to support multiple consumer threads
and added the semaphore to Genode's official base API. We have made
the wake-up policy in the presence of multiple consumers configurable
via a template argument. The default policy is first-in-first-out.
[https://genode.org/documentation/api/base_index#Synchronization - See the new API documentation for the semaphore...]
Thanks to Christian Prochaska for his valuable contributions to the new
semaphore design.
Asynchronous notifications
~~~~~~~~~~~~~~~~~~~~~~~~~~
Inter-process communication via remote procedure calls requires both
communication partners to operate in a synchronous fashion. The caller
of an RPC blocks as long as the RPC is not answered by the called
server. In order to receive the call, the server has to explicitly
wait for incoming messages. There are a number of situations where
synchronous communication is not suited.
For example, a GUI server wants to deliver a notification to one of its
clients about new input events being available. It does not want to
block on a RPC to one specific client because it has work to do for
other clients. Instead, the GUI server wants to deliver this
_notification_ with _fire-and-forget_ semantics and continue with
its operation immediately, regardless of whether the client received
the notification or not. The client, in turn, does not want to poll
for new input events at the GUI server but it wants to be _waken_up_
when something interesting happens. Another example is a block-device
driver that accepts many requests for read/write operations at once.
The operations may be processed out of order and may take a long time.
When having only synchronous communication available, the client and
the block device driver would have to employ one distinct thread for
each request, which is complicated and a waste of resources. Instead,
the block device driver just wants to acknowledge the completeness of
an operation _asynchronously_.
Because there are many more use cases for asynchronous inter-process
communication, we introduced a new signalling framework that complements
the existing synchronous RPC mode of communication with an interface for
issuing and receiving asynchronous notifications. It defines interfaces
for signal transmitters and signal receivers. A signal receiver can
receive signals from multiple sources, whereas the sources of incoming
signals are clearly distinguishable. One or multiple threads can either
poll or block for incoming signals. Each signal receiver is addressable
via a capability. The signal transmitter provides fire-and-forget
semantics for submitting signals to exactly one signal receiver. Signals
are communicated in a reliable fashion, which means that the exact number
of signals submitted to a signal transmitter is communicated to the
corresponding signal receiver. If notifications are generated at a higher
rate than as they can be processed at the receiver, the transmitter
counts the notifications and delivers the total amount with the next
signal transmission. This way, the total number of notifications gets
properly communicated to the receiver even if the receiver is not highly
responsive. Notifications do not carry any payload because this payload
would have to be queued at the transmitter.
[image signals]
Image [signals] illustrates the roles of signaller thread,
transmitter, receiver, and signal-handler thread.
[https://genode.org/documentation/api/base_index#Asynchronous_notifications - See the new API documentation for asynchronous notifications...]
The current generic implementation of the signalling API employs one
thread at each transmitter and one thread at each receiver. Because
the used threads are pretty heavy weight with regard to resource usage,
ports of Genode should replace this implementation with platform-
specific variants, for example by using inter-process semaphores or
native kernel support for signals.
Region-manager faults
~~~~~~~~~~~~~~~~~~~~~
In Genode, region-manager (RM) sessions are used to manage the
address-space layout for processes. A RM session is an address-space
layout that can be populated by attaching (portions of) dataspaces to
(regions of) the RM session. Normally, the RM session of a process is
first configured by the parent when decoding the process' ELF binary.
During the lifetime of the process, the process itself may attach
further dataspaces to its RM session to access the dataspace's content.
Core as the provider of the RM service uses this information for
resolving page faults raised by the process. In prior versions of
Genode, core ignored unresolvable page faults, printed a debug message
and halted the page-faulted thread. However, this condition may be of
interest, in particular to the process' parent for reacting on the
condition of a crashed child process. Therefore, we enhanced the RM
interface by a fault-handling mechanism. For each RM session, a fault
handler can be installed by registering a signal receiver capability.
If an unresolvable page fault occurs, core delivers a signal to the
registered fault handler. The fault handler, in turn, can request the
actual state of the RM session (page-fault address) and react upon
the fault. One possible reaction is attaching a new dataspace at the
fault address and thereby implicitly resolving the fault. If core
detects that a fault is resolved this way, it resumes the operation
of the faulted thread.
This mechanism works analogously to how page faults are handled by
CPUs, but on a more abstract level. A (n-level) page table corresponds
to a RM session, a page-table entry corresponds to a dataspace-
attachment, the RM-fault handler corresponds to a page-fault
exception handler, and the resolution of page-faults (RM fault)
follows the same basic scheme:
# Application accesses memory address with no valid page-table-entry
(RM fault)
# CPU generates page-fault exception (core delivers signal to fault
handler)
# Kernel reads exception-stack frame or special register to determine
fault address (RM-fault handler reads RM state)
# Kernel adds a valid page-table entry and returns from exception
(RM-fault handler attaches dataspace to RM session, core resumes
faulted thread)
The RM-fault mechanism is not only useful for detecting crashing child
processes but it enables a straight-forward implementation of growing
stacks and heap transparently for a child process. An example for
using RM-faults is provided at 'base/src/test/rm_fault'.
Note that this mechanism is only available on platforms on which core
resolves page faults. This is the case for kernels of the L4 family.
On Linux however, the Linux kernel resolves page faults and suspends
processes performing unresolvable memory accesses (segmentation fault).
Managed dataspaces (experimental)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The RM-fault mechanism clears the way for an exciting new feature
of Genode 8.11 called managed dataspaces. In prior versions of Genode,
each dataspace referred to a contiguous area of physical memory (or
memory-mapped I/O) obtained by one of core's RAM, ROM, or IO_MEM
services, hence we call them physical dataspaces. We have now added
a second type of dataspaces called managed dataspaces. In contrast
to a physical dataspace, a managed dataspace is backed by the content
described by an RM session. In fact, each RM session can be used as
dataspace and can thereby be attached to other RM sessions.
Combined with the RM fault mechanism described above, managed
dataspaces enable a new realm of applications such as dataspaces
entirely managed by user-level services, copy-on-write dataspaces,
non-contiguous large memory dataspaces that are immune to physical
memory fragmentation, process-local RM fault handlers (e.g., managing
the own thread-stack area as a sub-RM-session), and sparsely populated
dataspaces.
Current limitations
-------------------
Currently, managed dataspaces still have two major limitations. First,
this mechanism allows for creating cycles of RM sessions. Core must
detect such cycles during page-fault resolution. Although, a design for
an appropriate algorithm exists, cycle-detection is not yet implemented.
The missing cycle detection would enable a malicious process to force
core into an infinite loop. Second, RM faults are implemented using the
new signalling framework. With the current generic implementation, RM
sessions are far more resource-demanding than they should be. Once the
signalling framework is optimized for L4, RM sessions and thereby
managed dataspaces will become cheap. Until then, we do not recommend
to put this mechanism to heavy use.
Because of these current limitations, managed dataspaces are marked as
an experimental feature. When building Genode, experimental features are
disabled by default. To enable them, add a file called 'specs.conf'
with the following content to the 'etc/' subdirectory of your build
directory:
! SPECS += experimental
For an example of how to use the new mechanism to manage a part of a
process' own address space by itself, you may take a look at
'base/src/test/rm_nested'.
Changes
=======
Besides the addition of the new features described above, the following
parts of the base framework underwent changes worth describing.
Consistent use of typed capabilities and connection classes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We applied capability typification to all interfaces of Genode including
the base API and the interfaces defined in the 'os' repository. Figure
[base_cap_types] provides an overview about the capability types
provided by the base API.
[image base_cap_types]
Overview about the capability types provided by the base API
Furthermore, we have complemented all session interfaces with
appropriate 'Connection' classes taking service-specific session
arguments into account.
For session-interface classes, we introduced the convention to declare
the service name as part of the session-interface via a static member
function:
! static const char *service_name();
Allocator refinements
~~~~~~~~~~~~~~~~~~~~~
Throughout Genode, allocators are not only used for allocating memory
but also for managing address-space layouts and ranges of physical
resources such as I/O-port ranges or IRQ ranges. In these cases, the
address '0' may be a valid value. Consequently, this value cannot be
used to signal allocation errors as done in prior versions of Genode.
Furthermore, because managed dataspaces use the RM session interface to
define the dataspace layout, the address-'0' problem applies here as
well. We have now refined our allocator interfaces and the RM-session
interface to make them fit better for problems other than managing
virtual memory.
Misc changes
~~~~~~~~~~~~
We revised all interfaces to consistently use _exceptions_ to signal
error conditions rather than delivering error codes as return values.
This way, error codes become exception types that have a meaningful
name and, in contrast to global 'errno' definitions, an error exception
type can be defined local to the interface it applies to. Furthermore,
the use of exceptions allows for creating much cleaner looking interfaces.
Traditionally, we have provided our custom _printf_ implementation as C
symbol to make this function available from both C and C++ code. However,
we observed that we never called this function from C code and that the
'printf' symbol conflicts with the libc. Hence, we turned 'printf'
into a C++ symbol residing in the 'Genode' namespace.
Operating-system services and libraries
#######################################
This section documents the new features and changes affecting
the 'os' repository.
New Features
============
Device-driver framework for C device drivers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Genode's base API features everything needed to create user-level device
drivers. For example, the 'os' repository's PS/2 input driver and the
PCI bus driver are using Genode's C++ base API directly. However, most of
today's device drivers are written in C. To ease the reuse of existing
drivers on Genode, we have introduced a C API for device drivers into
Genode's 'os' repository. The API is called DDE kit (DDE is an acronym
for device-driver environment) and it is located at 'os/include/dde_kit'.
The DDE kit API is the result of long-year experiences with porting device
drivers from Linux and FreeBSD to custom OS environments. The following
references are the most significant contributions to the development of
the API.
;
Christian Helmuth created the initial version of the Linux device-driver
environment for L4. He describes his effort of reusing unmodified sound
drivers on the L4 platform in his thesis
[https://os.inf.tu-dresden.de/papers_ps/helmuth-diplom.pdf - Generische Portierung von Linux-Gerätetreibern auf die DROPS-Architektur].
;
Gerd Griessbach approached the problem of re-using Linux USB drivers
by following the DDE approach in his diploma thesis
[https://os.inf.tu-dresden.de/papers_ps/griessbach-diplom.pdf - USB for DROPS].
;
Marek Menzer adapted Linux DDE to Linux 2.6 and explored the DDE
approach for block-device drivers in his student research project
[https://os.inf.tu-dresden.de/papers_ps/menzer-beleg.pdf - Portierung des DROPS Device Driver Environment (DDE) für Linux 2.6 am Beispiel des IDE-Treibers ]
and his diploma thesis
[https://os.inf.tu-dresden.de/papers_ps/menzer-diplom.pdf - Entwicklung eines Blockgeräte-Frameworks für DROPS].
;
Thomas Friebel generalized the DDE approach and introduced the DDE kit
API to enable the re-use of device driver from other platforms than
Linux. In particular, he experimented with the block-device drivers of
FreeBSD in his diploma thesis
[https://os.inf.tu-dresden.de/papers_ps/friebel-diplom.pdf - Übertragung des Device-Driver-Environment-Ansatzes auf Subsysteme des BSD-Betriebssystemkerns].
;
Dirk Vogt successfully re-approached the port of USB device drivers
from the Linux kernel to L4 in his student research project
[https://os.inf.tu-dresden.de/papers_ps/beleg-vogt.pdf - USB for the L4 Environment].
The current incarnation of the DDE kit API provides the following
features:
* General infrastructure such as init calls, assertions, debug output
* Interrupt handling (attach, detach, disable, enable)
* Locks, semaphores
* Memory management (slabs, malloc)
* PCI access (find device, access device config space)
* Virtual page tables (translation between physical and virtual
addresses)
* Memory-mapped I/O, port I/O
* Multi-threading (create, exit, thread-local storage, sleep)
* Timers, jiffies
For Genode, we have created a complete reimplementation of the DDE kit
API from scratch by fully utilizing the existing Genode infrastructure
such as the available structured data types, core's I/O services,
the synchronization primitives, and the thread API.
[image dde_kit]
Figure [dde_kit] illustrates the role of DDE kit when re-using an
unmodified device driver taken from the Linux kernel. DDE kit translates
Genode's C++ base API to the DDE kit C API. The DDE kit API, in turn, is
used as back end by the Linux driver environment, which translates Linux
kernel interfaces to calls into DDE kit. With this translation in place,
an unmodified Linux device driver can be embedded into the Linux driver
environment. The device API is specific for a class of devices such as
NICs, block devices, or input devices. It can either be used directly as
a function interface by an application that is using the device driver
as a library, or it can be made accessible to external processes via an
RPC interface.
Limitations
-----------
The PCI sub system is not completely implemented, yet.
Alarm API providing a timed event scheduler
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The scheduling of timed events is a recurring pattern found in device
drivers, application frameworks such as Qt4 ('qeventdispatcher'), and
applications. Therefore, we have added a timed event scheduler to the
'os' repository. The new alarm API ('os/include/os/alarm.h') allows
for the scheduling of both one-shot alarms and periodic alarms.
Changes
=======
PS/2 input driver
~~~~~~~~~~~~~~~~~
The original PS/2 driver tried to switch the PS/2 keyboard to
scan-code set 2 and assumed that all modern keyboards support this
mode of operation. However, this assumption was wrong. We observed
that the legacy PS/2 support of some USB keyboards covers only the
emulated (xlate) scan-code set 1 mode. This is also case for the PS/2
emulation in VirtualBox. Therefore, we changed our PS/2 driver to
never touch the keyboard mode but to only detect the current mode
of operation. The driver has now to support both, scan-code set 1 and
scan-code set 2. This change comes along with a slightly more complex
state machine in the driver. Hence, we moved the state machine from
the IRQ handler to a distinct class and changed the control flow of
the driver to fetch only one value from the i8042 PS/2 controller
per received interrupt.
PCI bus driver
~~~~~~~~~~~~~~
Until now, Genode's PCI bus driver was only used for experimentation
purposes. With the forthcoming driver framework however, the PCI bus
driver will play a central role in the system. Therefore, we adapted
the interface of the PCI driver to these requirements. Specifically,
the scanning of the PCI bus can now be performed without constraining
the results by a specific vendor ID.
Nitpicker GUI server
~~~~~~~~~~~~~~~~~~~~
We improved the _output_latency_ of the Nitpicker GUI server by flushing
pixels eagerly and deferring the next periodically scheduled flush.
This change has a positive effect on the responsiveness of the GUI to
user input.
Misc changes
~~~~~~~~~~~~
Prior versions of the 'os' repository came with a custom 'os/include/base'
directory with interfaces extending the base API. To avoid confusion
between the 'base' repository and the 'os' repository, 'os'-local API
extensions are now located at 'os/include/os'. This way, the folder
prefix of include statements indicates well from which repository the
included header files comes from.
C runtime
#########
Most of existing libraries rely on the presence of a C library. For
making the reuse of this software on Genode possible, we have now
made a complete C library available for Genode. It comes as a separate
source-code repository called 'libc' and is based on the code of FreeBSD.
The original code is available at the official FreeBSD website.
:FreeBSD website:
[https://www.freebsd.org/developers/cvs.html]
Our libc port comprises the libraries 'gdtoa', 'gen', 'locale', 'stdio',
'stdlib', 'stdtime', 'string', and 'msun'. Currently, it supports the
x86 architecture. Support for other architectures is planned as future
addition. At the current stage, our back end is very basic and most of
its functions are dummy stubs. We used Christian Prochaska's forthcoming
Genode port of Qt4 as test case and successfully used the new libc as
foundation for building graphical Qt4 applications. We will further
extend the back end in correspondence to the growing feature set of the
Genode OS framework.
:Usage:
To use the libc in your application, just add 'libc' to the 'LIBS'
declaration in your build-description file. This declaration will make
the libc headers available for the include path of your target and link
the C library. When building, make sure that the 'libc' repository is
included in your build configuration ('etc/build.conf').
:Limitations:
The current version of the C library is not thread-safe. For most
string and math functions, this is not a problem (as these functions
do not modify global state) but be careful with using more complex
functions such as 'malloc' from multiple threads. Also, 'errno' may
become meaningless when calling libc functions from multiple threads.
We have left out the following files from the Genode port of the
FreeBSD libc: gdtoa 'strtodnrp.c' (gdtoa), 'getosreldate.c' (gen),
'strcoll.c', 'strxfrm.c', 'wcscoll.c', 'wcsxfrm.c' (string),
's_exp2l.c' ('msun').
The current back end is quite simplistic and it may help you to revisit
the current state of the implementation in the 'libc/src/lib/libc'
directory. If one of the functions in 'dummies.c' is called, you will
see the debug message:
! "<function-name> called, not yet implemented!"
However, some of the back-end function implemented in the other files
have dummy semantics but have to remain quiet because they are called
from low-level libc code.
Build infrastructure
####################
Build-directory creation tool
=============================
Because we think that each Genode developer benefits from knowing the
basics about the functioning of the build system, the manual creation of
build directories is described in Genode's getting-started document.
However, for regular developers, creating build directories becomes a
repetitive task. Hence, it should be automated. We have now added a
simple build-directory creation tool that creates pre-configured build
directories for some supported platforms. The tool is located at
'tool/builddir/create_builddir'. To print its usage information, just
execute the tool without arguments.
Improved linking of binary files
================================
For linking binary data, binary file have to be converted to object
files. Over the time, we have used different mechanisms for this
purpose. Originally, we used 'ld -r -b binary'. Unfortunately, these
linker options are not portable. Therefore, the mechanism was changed
to a 'hexdump' and 'sed' magic that generated a C array from binary data.
This solution however, is complicated and slow. Now, we have adopted
an idea of Ludwig Hähne to use the 'incbin' directive of the GNU
assembler, which is a very clean, flexible, and fast solution.
Lib-import mechanism
====================
Libraries often require specific include files to be available at the
default include search location. For example, users of a C library
expect 'stdio.h' to be available at the root of the include search
location. Placing the library's include files in the root of the
default search location would pollute the include name space for
all applications, regardless if they use the library or not. To
keep library-include files well separated from each other, we have
enhanced our build system by a new mechanism called lib-import.
For each library specified in the 'LIBS' declaration of a build
description file, the build system incorporates a corresponding
'import-<libname>.mk' file into the build process. Such as file
defines library-specific compiler options, in particular additional
include-search locations. The build system searches for lib-import
files in the 'lib/import/' subdirectories of all used repositories.
Using 'ar' for creating libraries
=================================
The previous versions of Genode relied on incremental linking ('ld -r')
for building libraries. This approach is convenient because the linker
resolves all cross-dependencies between libraries regardless of the
order of how libraries are specified at the linker's command line.
However, incremental linking prevents the linker from effectively
detecting dead code. In contrast, when linking '.a' files, the linker
detects unneeded object files. Traditionally, we have only linked our
own framework containing no dead code. This changed with the new 'libc'
support. When linking the 'libc', the presence of dead code becomes
the normal case rather than the exception. Consequently, our old
incremental-linking approach produced exceedingly large binaries
including all functions that come with the 'libc'. We have now adopted
the classic 'ar' mechanism for assembling libraries and use the linker's
'start-group' 'end-group' feature to resolve inter-library-dependencies.
This way, dead code gets eliminated at the granularity of object files.
In the future, we will possible look into the '-ffunction-sections' and
'-gc-sections' features of the GNU tool chain to further improve the
granularity to function level.
If your build-description files rely on custom rules referring to
'lib.o' files, these rules must be adapted to refer to 'lib.a' files
instead.
Misc changes
============
* Added sanity check for build-description files overriding 'INC_DIR'
instead of extending it.
* Restrict inclusion of dependency files to those that actually matter
when building libraries within 'var/libcache'. This change significantly
speeds up the build process in the presence of large libraries such as
Qt4 and libc.
* Added rule for building 'cpp' files analogously to the 'cc' rule.
Within Genode, we name all C++ implementation files with the 'cc'
suffix. However, Qt4 uses 'cpp' as file extension so we have to
support both.
* Build-description files do no longer need the declaration
'REQUIRES = genode'. Genode's include search locations are now
incorporated into the build process by default.
Applications
############
This section refers to the example applications contained in Genode's
'demo' repository.
We have enhanced the _Scout_widgets_ as used by the launchpad and the
Scout tutorial browser to perform all graphical output double-buffered,
which effectively eliminates drawing artifacts that could occur when
exposing intermediate drawing states via direct (unbuffered) output.
Furthermore, we have added a way to constrain the maximum size of
windows to perform pixel-buffer allocations on realistic window sizes.
Both launchpad and Scout can now start child applications. In Scout
this functionality is realized by special "execute" links. We have
generalized the underlying application logic for creating and
maintaining child processes between both applications and placed
the unification into a separate 'launchpad' library.
We have replaced the default document presented in Scout with an
_interactive_walk-through_guide_ explaining the basic features of Genode.
The document uses the new "execute" link facility to let the user start
a launchpad instance by clicking on a link.
Platform-specific changes
#########################
Genode used to define _fixed-width_integer_types_ in a file 'stdint.h'
placed in a directory corresponding to bit-width of the platform, for
example 'include/32bit/stdint.h'. When building for a 32bit platform,
the build system included the appropriate directory into the
include-search path and thereby made 'stdint.h' available at the root
of the include location. Unfortunately, this clashes with the 'stdint.h'
file that comes with the C library. To avoid conflict with libc header
files, we moved the definition of fixed-width integer types to
'32bit/base/fixed_stdint.h'.
For the L4/Fiasco version of Genode, there existed some x86-specific
header files that did not specifically depend on L4/Fiasco, for example
atomic operations. Because these files are not L4/Fiasco-specific and
may become handy for other platforms as well, we moved them to the
generic 'base' repository.
Linux 32bit
===========
:Dissolving Genode's dependency from the glibc:
The port of the C runtime to Genode posed an interesting challenge to
the Linux version of Genode. This version used to rely on certain
functions provided by the underlying glibc:
* For creating and destroying threads, we used to rely on POSIX threads
as provided by the 'pthread' library
* The lock implementation was based on the POSIX semaphore functions
'sem_init', 'sem_wait', and 'sem_post'
* Shared memory was realized by using files ('open', 'close',
'ftruncate') and the 'mmap' interface
* Starting and killing processes was implemented using 'fork' and 'kill'
* Inter-process communication used the glibc's socket functions
For our custom C runtime, we want to override the glibc functionality
with our own implementation. For example, we want to provide the 'mmap'
interface to a Genode application by implementing 'mmap' with
functions of our base API. On Linux, however, this base API, in turn,
used to rely on 'mmap'. This is just an example. The problem applies
also for the other categories mentioned above. We realized that we cannot
rely on the glibc on one hand but at the same time replace it by a custom
C runtime (in fact, we believe that such a thing is possible by using
awkward linker magic but we desire a clean solution). Consequently, we
have to remove the dependency of Genode from the glibc on Linux. Step
by step, we replaced the used glibc functions by custom Linux system-call
bindings. Each binding function has a prefix 'lx_' such that the symbol
won't collide with 'libc' symbols. The new bindings are located at the file
'base-linux/src/platform/linux_syscalls.h'. It consist of 20 functions,
most of them resembling the original interface ('socket', 'connect',
'bind', 'getsockname', 'recvfrom', 'write', 'close', 'open', 'fork',
'execve', 'mmap', 'ftruncate', 'unlink', 'tkill', 'nanosleep').
For other functions, we simplified the semantics for our use case
('sigaction', 'sigpending', 'sigsetmask', 'create_thread'). The most
noteworthy changes are the creation and destruction of threads by
directly using the 'clone' and 'tkill' system calls, and the lock
implementation. Because we cannot anymore rely on the convenience of
using futexes indirectly through the POSIX semaphore interface, we
have adopted the simple locking approach that we already use for the
L4/Fiasco version. This lock implementation is a simple sleeping
spinlock.
:Compromises:
The introduction of custom Linux system-call bindings for Genode has
several pros and cons. With this change, The Linux version of Genode is
not anymore easy to port to other POSIX platforms such as the Darwin
kernel. For each POSIX kernel used as Genode platform, a custom
implementation of our system-call bindings must be created. The
original POSIX variant could still be reanimated, but this version
would inherently lack support for Genode's C runtime, and thus would
have limited value. A positive side effect of this solution, however,
is that 'linux_syscalls.h' documents well the subset of the Linux'
kernel interface that we are actually using.
The replacement of POSIX semaphores with sleeping spinlocks decreases
locking performance quite significantly. In the contention case, the
wakeup from sleeping introduces a high latency of up to one millisecond.
Furthermore, fairness is not guaranteed and the spinning produces a bit
of system load. If this approach turns out to become a serious performance
bottleneck, we will consider creating custom bindings for Linux' futexes.
L4/Fiasco
=========
The concepts of _RM_faults_ and _managed_dataspaces_ as described in
Section [Base framework], had been implemented into the L4/Fiasco
version of core. Although the introduction of these concepts involved
only minimal changes at the API level, the required core-internal
changes had been quite invasive, affecting major parts of the pager
and RM-session implementations.
Prior versions of the L4/Fiasco version of core did not implement
the _cancel-blocking_mechanism_ as specified by the CPU-session API.
The missing implementation resulted in lock-ups when destructing a
thread that blocks for lock. With the new implementation based on
L4/Fiasco's inter-task ex-regs system call, such threads can now
be gracefully destructed.

460
doc/release_notes/09-02.txt Normal file
View File

@@ -0,0 +1,460 @@
==============================================
Release notes for the Genode OS Framework 9.02
==============================================
Genode Labs
Summary
#######
Whereas the focus of the previous release 8.11 was the refinement of
Genode's base API and the creation of the infrastructure needed to build
real-world applications, the release 9.02 is focused on functional
enhancements in two directions. The first direction is broadening the
number of possible base platforms for the framework. At present, most
microkernels bring along a custom user land, which is closely tied to the
particular kernel. Our vision is to establish Genode as a common ground for
developing applications, protocol stacks, and device drivers in such a way
that the software becomes easily portable among different kernels. This
release makes Genode available on the L4ka::Pistachio kernel. Hence,
software developed with the Genode API can now run unmodified on
Linux/x86, L4/Fiasco, and L4ka::Pistachio. In the second direction, we
are steadily advancing the functionality available on top of Genode. With
this release, we introduce a basic networking facility and support for
native Qt4 applications as major new features. Thanks to Genode's
portability, these features become automatically available on all
supported base platforms.
Our original plan for the release 9.02 also comprised the support of a
Linux-on-Genode (para-)virtualization solution. Initially, we intended to
make [https://os.inf.tu-dresden.de/L4/LinuxOnL4/ - L4Linux] available on
the L4/Fiasco version of Genode. However, we identified several downsides
with this approach. Apparently, the development of the officially available
version of L4/Fiasco has become slow and long-known issues remain unfixed.
L4Linux, however, is closely tied to L4/Fiasco and the L4 environment. For
us at Genode Labs, maintaining both a custom port of L4Linux for Genode
and L4/Fiasco by ourself in addition to developing Genode is unfeasible.
In contrast, the Pistachio kernel features more advanced options for
virtualization ([http://l4ka.org/projects/virtualization/afterburn/ - Afterburner]
and VT support) that we want to explore. Furthermore, there exists another
version of L4Linux called OKLinux for the OKL4 kernel developed at
[http://ok-labs.com - OK-Labs], which is very interesting as well.
Therefore, we decided against an ad-hoc solution and deferred this feature
to the next release. [https://genode.org/about/road-map - See our updated road map...]
Major new Features
##################
Genode on L4ka::Pistachio
=========================
From the very beginning, the base API of the Genode OS Framework was
designed for portability. We put a lot of effort into finding API
abstractions that are both implementable on a wide range of kernels and as
close to the hardware as possible to keep the abstraction overhead low. For
this reason, we developed the framework in parallel for the Linux kernel and
the L4/Fiasco kernel. To validate our claim that Genode is highly portable,
Julian Stecklina ported the framework to another member of the L4 family,
namely the [http://l4ka.org/projects/pistachio/ - L4ka::Pistachio kernel].
This high-performance kernel implements the latest official L4 API called
L4.x2 and has a number of advanced features such as multi-processor support
and virtualization support.
After Julian successfully created the first Pistachio version of Genode,
we successively refined his work and conducted further unifications among
the platform-dependent code for the different kernels. The result of this
effort is included in this release and comes in the form of the
'base-pistachio' source-code repository.
;Interesting technical notes:
* The IRQ handling on Pistachio is slightly different from L4/Fiasco.
On L4/Fiasco, an IRQ becomes unmasked only when the user-level IRQ
handler thread blocks for an IRQ by issuing an IPC call to the
kernel's corresponding IRQ thread. In contrast, Pistachio unmasks an
IRQ as soon as the user-level IRQ handler associates itself with an
IRQ. Thus, an IRQ message may occur not only when the user-level IRQ
handler blocks for any IRQ but anytime. In particular, IRQ messages
may interfere with the IRQ handler's IPC communication with other
threads. To ensure that IRQ messages do only occur when expecting
them, we lazily associate the IRQ handler thread to the IRQ the
first time we wait for an IRQ and issue an unmasking IPC call
subsequent times.
* Genode provides a mechanism for gracefully destructing threads that
are in a blocking state, for example waiting for an IPC message.
Such a thread may hold locks or other resources that would not
get properly freed when just killing the thread by force. Therefore,
Genode provides a way to issue the cancellation of a blocking
operation by another thread (e.g., the thread calling the destructor).
Once, a blocking operation got canceled, a C++ exception
('Blocking_canceled') is raised such the thread can fall back into
a defined state and then be destructed. On L4ka::Pistachio, we use
Pistachio's pager-exchange-registers feature in combination with
the user-defined UTCB handle for cancelling blocking operations and
detecting cancellations. The interesting code bits can be found in
'src/base/ipc/ipc.cc', 'src/base/lock/lock.cc',
'src/core/platform_thread.cc', and in the Pistachio-specific
timer-service implementation.
* During the refinement of the Pistachio version, we were able to further
generalize code that was previously specific for L4/Fiasco and
L4ka::Pistachio respectively. Now, the platform-specific code comprises
less than 3,000 lines of code (LOC) for L4/Pistachio, circa 2,000 LOC
for L4/Fiasco, and circa 1,000 LOC for Linux. Hence, we expect that
porting the framework to further kernels is possible at reasonable
engineering costs.
:Current limitations:
* The current version does not use superpages (4M mappings) because we
experienced problems with mapping 4K pages out of 4M pages. This is an
issue that we like to investigate further because using 4M mappings
would improve the boot time and reduce the kernel-memory usage.
* Currently, we use a simple sleeping spinlock for synchronization, which
is not optimal for several reasons. There are no fairness guarantees,
the spinning consumes CPU time, and threads that got blocked in the
contention case are woken up at the coarse granularity of the kernel's
timer tick, which is typically one millisecond.
* Nested RM sessions as introduced as an experimental feature in the
Genode release 8.11 are not yet supported.
:Further details:
You can find further technical details and usage instructions at this
dedicated [https://genode.org/documentation/platforms/pistachio - page].
Qt4 on Genode
=============
The minimalism of the Genode OS Framework with regard to its code
complexity raised the question of whether this framework is feasible
for hosting real-world applications and widely used runtime environments.
Christian Prochaska took the challenge to port Trolltech's Qt4 application
framework, which serves as the basis for the popular KDE desktop, to Genode.
Because Christian started his work more than a year ago at a time when no
C library was available on Genode, several intermediate steps were needed.
The first step was the integration of the Qt4 tools such as the meta-object
compiler (moc) and resource compiler properly into the our build systion.
With the tools in place, the Linux version of Genode came to an advantage.
In this environment, a Genode application is able to use glibc functionality.
So the problem of a missing C library could be deferred and Christian was
able to focus on interfacing Qt with the existing Genode services such as
the Nitpicker GUI sever. Next, the glibc dependencies were successively
replaced by custom implementations or simple dummy stubs. Thereby, all
needed functionalities such as timed semaphores and thread-local storage
had to be mapped to existing Genode API calls. Once, all glibc dependencies
had been dissolved, Qt could be compiled for the L4/Fiasco version.
Since a C library has become available in Genode 8.11, we were able to
replace Christian's intermediate stub codes with a real C library. We also
utilize recently added features of Genode such as its alarm framework to
simplify the Qt4 port. Furthermore, we were able to remove all
platform-specific bits such that the Qt4 port has now become completely
generic with regard to the underlying kernel. Qt4 can be executed on Linux,
L4/Fiasco, and L4ka::Pistachio without any changes. Figure [qt4_screenshot]
shows a screenshot of Qt's Tetrix example running side-by-side with native
Genode applications.
[image qt4_screenshot]
:Current state:
* The Qt4 port comes in the form of a source-code repository, which contains
all Qt source codes, and some example programs such as Tetrix. You can
download the Qt4 repository as a separate archive at the download page of
the Genode release 9.2. For the next release, we plan to separate the
Genode-specific parts from Qt original code and make the Genode-specific
parts a regular component of the Genode main line.
* The Qt4 port consists of Qt's Core library, GUI library, Script library,
XML library, and the UI tools library. Other libraries such as Webkit
are not ported yet.
* This first version of Qt4 on Genode is not to be considered as stable.
There are several known issues yet to be addressed. In particular,
the 'QEventDispatcher' is still work in progress and not fully stabilized.
* Because, we use to statically link programs, the binaries of Qt
applications are exceedingly large. For example the Tetrix binary is
100MB including debug information and 11MB in the stripped form. For
employing Qt on Genode at a larger scale, Genode should be enhanced with
shared-library support.
Networking
==========
With Genode 8.11, we introduced the Device-Driver-Environment Kit (DDE Kit)
API, which is a C API specifically designed for implementing and porting
device drivers written in plain C. We have now complemented DDE Kit with an
environment for executing Linux device drivers natively on Genode. This
library is called 'dde_linux26' and contained in our new 'linux_drivers'
source-code repository. The environment consists of several parts, which
correspond to the different sub systems of the Linux kernel 2.6, such as
'arch', 'drivers', 'kernel'.
The first class of device-drivers supported by DDE Linux 2.6 is networking.
At the current stage, the DDE Linux network library comprises general
network-device infrastructure as well as an exemplary driver for the PCnet32
network device.
Based on this library, we have created a basic TCP/IP test utilizing the
uIP stack, which uses the DDE Linux network library as back end. The test
program implements a basic web server displaying uIP packet statistics.
When executed on Qemu, you can use your host's web browser to connect to
the web server running on Genode:
For booting Genode on L4/Fiasco with the web-server demo, use a GRUB
entry in your 'menu.lst' file as follows.
! title Genode: DDE Linux 2.6 NET on L4/Fiasco
! kernel /fiasco/bootstrap -maxmem=64 -modaddr=0x02000000
! module /fiasco/fiasco -nokd -serial -serial_esc
! module /fiasco/sigma0
! module /genode/core
! module /genode/init
! module /config
! module /genode/timer
! module /genode/pci_drv
! module /genode/test-dde_linux26_net
The first four lines are L4/Fiasco specific. When using L4ka::Pistachio,
the 'menu.lst' entry looks like this:
! title Genode: DDE Linux 2.6 NET on L4/Pistachio
! kernel /pistachio/kickstart
! module /pistachio/x86-kernel
! module /pistachio/sigma0
! module /genode/core
! module /genode/init
! module /config
! module /genode/timer
! module /genode/pci_drv
! module /genode/test-dde_linux26_net
The web-server test requires the PCI bus driver and the timer service.
Therefore, the 'config' file for Genode's init should have following
content:
! <config>
! <start>
! <filename>timer</filename>
! <ram_quota>512K</ram_quota>
! </start>
! <start>
! <filename>pci_drv</filename>
! <ram_quota>512K</ram_quota>
! </start>
! <start>
! <filename>test-dde_linux26_net</filename>
! <ram_quota>16M</ram_quota>
! </start>
! </config>
Now, its time to create an ISO image from all files specified in
the GRUB configuration. For this, the new utility 'tool/create_iso'
becomes handy. The ISO image can then be booted on Qemu using the
following arguments:
! qemu -m 64 -serial stdio -no-kqemu -cdrom <iso-image> \
! -net nic,model=pcnet -net user -redir tcp:5555::80
The '-redir' argument tells qemu to redirect TCP connections with
localhost:5555 to the guest OS at port 80. After having booted
up Genode on Qemu, you can use your host's web browser to access
the web server:
! firefox http://localhost:5555
:Notes about using the TAP version:
* Preparations
* You must be permitted to sudo and have installed the tunctl
utility. Under Debian/Ubuntu execute
! sudo apt-get install uml-utilities
* Create TAP device
! TAPDEV=$(sudo tunctl -b -u $USER)
! sudo /sbin/ifconfig $TAPDEV 10.0.0.1
* setup DHCP server on $TAPDEV and 10.0.0.0/8
* Run qemu
! qemu -m 64 -serial stdio -no-kqemu -cdrom dde.iso \
! -net nic,model=pcnet \
! -net tap,ifname=$TAPDEV,script=no,downscript=no
* Ping
* Cleanup
* Stop DHCP server
* Remove TAP device
! sudo tunctl -d $TAPDEV
Operating-system services and libraries
#######################################
C Runtime
=========
We have replaced the 'malloc' implementation of the original FreeBSD C
library with a custom implementation, which relies on Genode's 'Heap' as
allocator. The FreeBSD libc reserves a default memory pool of 1MB, which
is no problem on FreeBSD because virtual memory is backed lazily with
physical pages on demand. On Genode however, we immediately account the
allocated memory, which implicates high quota requirements even for
applications that use little memory. In contrast, Genode's heap allocates
and accounts its backing store in relatively small chunks of a few KB.
Therefore, the quota accounting for applications is much more in line with
the actual memory usage. Furthermore, our custom 'malloc' implementation
has the additional benefit of being thread safe.
* Added i386-specific parts of gen lib, in particular longjmp, setjmp.
Device-Driver-Environment Kit
=============================
* The DDE Kit uses our alarm framework (located in the 'os' repository) now
rather than its own event-scheduler implementation formerly called 'Tick'.
* We refined the DDE Kit API and reduced the number of custom types. For
example, we removed the custom 'dde_kit_lock_t' and using
'struct dde_kit_lock' instead, and replaced 'dde_kit_thread_t' with
'struct dde_kit_thread'.
Because of the apparent stabilization of the DDE Kit API, we have now added
this API to Genode's official API reference.
[https://genode.org/documentation/api/dde_kit_index - See the documentation of the DDE Kit API...]
PS/2 input driver
=================
We improved the PS/2 keyboard driver by adding missing scan-code translations
for the scan code set 1, in particular the cursor keys.
Applications
############
Launchpad configuration
=======================
Launchpad is a graphical application for interactively starting and killing
programs. It is used for the default demonstration of Genode. By default,
launchpad displays a preconfigured list of programs and their respective
default memory quotas. The user can tweak the memory quota for each entry
with mouse and then start a program by clicking on its name. As an
alternative to using the default list, you can now define the list manually
by supplying a configuration to Launchpad. The following example tells
launchpad to display a list of two launcher entries:
!<config>
! <launcher>
! <filename>sdl_pathfind</filename>
! <ram_quota>10M</ram_quota>
! </launcher>
! <launcher>
! <filename>liquid_fb</filename>
! <ram_quota>10M</ram_quota>
! </launcher>
!</config>
To use this configuration for a Launchpad started via init, you can simply
insert the launchpad configuration into the '<start>' node of the launchpad
entry in init's 'config' file.
Platform-specific changes
#########################
L4/Fiasco
=========
* Raise 'Blocking_canceled' exceptions on canceled IPC calls
32-bit Linux
============
* We continued dissolving the dependency of Genode from the glibc by using
a custom 'getenv' implementation used during process creation.
* By default, we compile now with '-nostdinc' and explicitly specify
'/usr/include' as include search directory only when needed. Previously,
a Genode application, which included a host include file by mistake, has
not raised any compilation error when compiled for the Linux version of
Genode. Now, all Genode platforms behave equally with regard to include
search directories.
* We enforce using the actual compiler's C++ support libraries rather than
the default libraries installed on the host.
Tools and build infrastructure
##############################
Official tool chain
===================
At the download section of our website, we used to provide a crosstool-based
tool chain as pre-compiled binaries. Since we got several requests about
how to build such a tool chain from scratch, we created custom utility for
downloading, building, and installing the official Genode tool chain. You
can find the utility at 'tool/tool_chain'. For usage instructions, just
start 'tool_chain' without arguments. Because this utility is a plain script,
you can follow and verify each step that we use for creating the Genode tool
chain. Currently, this official tool chain is based on binutils 2.18 and
gcc 4.2.4.
As an alternative to installing the tool chain from source, we also
provide pre-compiled binaries at the download section of our website.
[https://genode.org/download/tool-chain - Visit our tool-chain download website...]
For the Linux version of Genode, we still use the host's default gcc
as tool chain. This way, we spare the hassle of downloading and installing
a custom tool chain for somebody who wants to give Genode a quick try.
With this is mind, we have fixes several small issues with gcc 4.3.2:
* Fixed dependency generation for gcc-4.3.2. Older version of gcc used to
append a '.o' dependency at the target of '.d'-files. However, gcc-4.3.2
seems to handle the option '-MT' differently, resulting in a rule that
contains only the '.d' as target. Now, we explicitly specify both the
'.o' file and the '.d' file as target. Consequently, on older gcc
versions, the '.o' file appears twice but that is no problem.
* Fixed assembler issue with the 'fnstsw' instruction in the C library.
This instruction does not accept eax but only ax as argument.
Build-directory creation tool
=============================
We added a rule for creating a pre-configured build directory for the
Pistachio version to our build-directory creation tool
('tool/builddir/create_builddir'). Furthermore, we changed the default
build configuration such that the official Genode tool chain is used for
L4/Fiasco and L4ka::Pistachio.
Build system
============
* Improved clean rule - visit each target directory only once
* Stop the build process on the first error by default, for continuing
the build process depite of an error, you can use the '-i' argument
of make.
* Compiler flags can now be set specific for compiling C and C++ sources.
This is needed because both variants allow different sets of warning
options. The new variables are called 'CC_CXX_OPT' and 'CC_C_OPT'.
ISO image creation tool
=======================
We have created a convenient front end for 'genisoimage', which we
use for testing Genode on Qemu. You can find this ISO-image-creation
tool at 'tool/create_iso'. For usage instructions, simply start the
tool without arguments.

585
doc/release_notes/09-05.txt Normal file
View File

@@ -0,0 +1,585 @@
==============================================
Release notes for the Genode OS Framework 9.05
==============================================
Genode Labs
Shortly after including support for the L4ka::Pistachio kernel in the
previous release, the Genode version 9.05 gives a further evidence of
Genode's high portability by making the framework available on top of
the OKL4 kernel. This kernel is a commercial-grade microkernel
developed by [http://ok-labs.com - Open Kernel Labs]. In Section
[Supporting the OKL4 kernel as new base platform], we elaborate on the
new base platform and summarize the experiences made during our porting
work.
The previous Genode release was accompanied by a source-code archive containing
the initial version of Qt4 for Genode. Our approach is to make the Qt4
framework available for building Genode applications running natively on the
microkernel rather than within a virtualization environment. As advertised in
our [https://genode.org/about/road-map - road map], we have now seamlessly
integrated the Qt4 framework into our mainline source tree. Furthermore, we
have adapted our port to the Qt4 version 4.5.1. Section [Integration of Qt4
into the mainline repository] gives a rough overview of the changes and an
introduction on how to use the Qt4 framework with Genode.
The third major feature of the release is the addition of preliminary USB
support. We have been able to port major parts of Linux' USB infrastructure
to Genode using the DDE Kit introduced in autumn 2008. Section [USB support]
presents an overview about the pursued design and the current state of
implementation.
Section [OKLinux on Genode] outlines our ongoing efforts of running Linux
as a node in Genode's process tree.
Supporting the OKL4 kernel as new base platform
###############################################
The OKL4 kernel developed by Open Kernel Labs started as a fork of the
L4ka::Pistachio kernel. Whereas L4ka::Pistachio remained true to the L4
x.2 specification, OKL4 was subject of major API changes geared towards high
performance on the ARM architecture. OKL4 earned much fame for executing a
user-level variant of Linux (OKLinux) on top the microkernel, which turned out
to be faster than executing Linux natively on the ARM9 architecture. Even
though OKL4 is primary targeted at the ARM architecture, we wanted to go for
the x86 variant because of two reasons. First, there exists the just mentioned
user-level port of Linux for OKL4, which looks like an attractive way to execute
Linux on Genode once Genode runs on OKL4. Second, we think that distributing
Genode in the form of ISO images bootable on plain PC hardware is the best
way to reach the OS community. Therefore, we decided to use OKL4 version 2.1 as
the base for our work. In contrast to later releases, this version supports
both x86 and ARM. The following section reviews the unique features of the
OKL4 kernel from our perspective.
OKL4 viewed from the angle of a Genode developer
================================================
On the kernel-API level, OKL4 has several interesting properties that had been
both welcome and challenging. We want to highlight the following points:
In contrast to prior L4 kernels, OKL4 has *removed wall-clock timeouts* from
the kernel interface. On L4, timeouts were used as arguments for for blocking
IPC operations serving two purposes. First, specifying IPC timeouts allowed the
IPC caller for regaining control over the blocking thread after the specified
time elapsed. This is considered as important in the case that the called
thread misbehaves and never answers the call. However, the problem of choosing
an appropriate timeout was never properly resolved. When dimensioning the
timeout too small, the called thread may miss the timeout just because it had
no chance to be selected by the scheduler in time. Such timeouts rely on the
presumption that there is low load on the system. On the other hand, when
dimensioning the timeout too high, the system will become sluggish when the
called thread misbehaves. For example, a simple GUI server may want to send
input events to its clients with a timeout to be robust against misbehaving
clients that never wait for events. When choosing a timeout too small, chances
are high that an event will occur at a time when the receiver is handling a
previous event. The timeout would trigger and the event would get lost. When
choosing the timeout too large, say 1 second, any misbehaving client could make
the GUI server freeze for 1 second. Therefore, timeouts for regaining control
over a blocked thread seem to be a bad idea. So we welcome their absence in
OKL4. The second use of timeouts is their use as user-level time source. On L4,
sleep is typically implemented as a blocking IPC with a timeout set to the
sleep value. For this purpose, a system built on top of OKL4 has to employ a
user level device driver accessing a timer device. In Genode, we already have a
timer service for this purpose. So we won't miss timeouts at all.
Classical L4 kernels provide two variants of *synchronous IPC*. So called long
IPC could copy any amount of memory from the sending to the receiving address
space. This is complicated operation because either communication partner may
specify communication buffers that contain unmapped pages. Hence, page faults
may occur during long-IPC operations. On L4, page faults, in turn, are handled
by the user land. Not until a user-level pager thread resolves the page fault
by establishing a mapping at the faulting address, the kernel can proceed the
IPC operation. This sounds pretty complicated, and it is. The second IPC
variant is called short IPC. It constrains the transferable payload to CPU
registers. Hence, these IPC operations should only be used for messages with a
payload of a maximum of 2 machine words. Because short IPCs are never touching
user-level memory pages, no page faults can occur.
On OKL4, there is only one IPC operation, which copies payload from the
sender's user-level thread-control block (UTCB) to the receiver's UTCB. An
UTCB is an always-mapped memory region. Hence no page faults can occur during
IPC operations. On Genode, the UTCB size of 256 bytes may become a limitation
when RPC messages get large. For example, session requests may include large
session-argument strings specifying session-constructor arguments. Current
services rely only on a few arguments so the size limitation is not an
apparent problem. But that may change for future services. Furthermore, in
contrast to L4 x.2, OKL4 does not allow for transferring payload other than
plain data. In particular, OKL4 does not support the transfer of memory
mappings via IPC. Removing memory mappings from the IPC operation is a very
good idea. On Genode, only roottask (core) establishes mappings and shared
memory is implemented as a user-level protocol (data spaces). There is no need
to allow arbitrary processes to establish memory mapping via IPC.
The *boot procedure* of OKL4 largely differs from other L4 kernels. This is
attributed to Open Kernel Labs' focus on embedded systems, which mostly rely on
single-image boot loading. OKL4 employs a tool (elfweaver) for creating a
bootable image from a bunch of files and an XML configuration file. Among the
declarations about which processes to be loaded and which policies to enforce,
the configuration file contains platform parameters such as the amount of
physical memory of the machine. This static approach to configure a system is
certainly useful for embedded systems but PC hardware uses to vary a lot. In
this case, evaluating boot-time memory descriptors would be the preferred
solution.
OKL4 introduces kernel support for *user-level synchronization*. Prior L4
kernels facilitated user-level synchronization through a combination of
synchronous IPC operations with either priorities or delayed preemption.
OKL4's mutexes can make the life in the user land much easier. However, we have
not looked into OKL4 mutexes yet.
There does not exist a recursive *map operation* as the source operand of the
map operation is a physical memory descriptor rather than a virtual address in
the mapper's address space. Consequently, this design eliminates the need for
having a recursive unmap operation and thereby, the need to maintain a mapping
data base in the kernel. This is cool because Genode keeps track of the
mappings at the user level anyway (within core). From our perspective, there is
no need to maintain mapping relationships in the kernel. Removing the mapping
database effectively discards a lot of much-discussed problems about how to
manage the mapping database in a clever way.
There exists *no root memory manager* (sigma0). Because the map operation
takes a physical memory descriptor as argument instead of a virtual address
in the mapper's address space. The mapper does not need to have the mapped
page locally mapped within its own address space. In fact, core (as the only
mapper in a Genode system) does only have very little memory mapped locally.
This design accelerates the boot time because there is no need to map each
physical page in core at startup as performed when running core on the other
L4 kernels.
These differences of OKL4 compared with the microkernels already supported
by Genode posed a number of interesting challenges and opportunities. We have
thoroughly documented the process in
[https://genode.org/documentation/articles/genode-on-okl4 - Bringing Genode to OKL4].
Usage
=====
For using Genode with OKL4, please refer to the following dedicated page:
:[https://genode.org/documentation/platforms/okl4 - Genode on the OKL4 microkernel]:
Site about building and using Genode with the OKL4 kernel.
Limitations of the current implementation
=========================================
The current implementation is able to execute the complete Genode demonstration
scenario on OKL4. This means, we can build and destroy arbitrary trees of
processes, have all the needed infrastructure in place to execute user-level
device drivers such as VESA and PS/2, perform inter-process communication
via RPC and shared memory, and have all basic framework API functions available.
We regard the current state as the first functional version. However, there are
the following points that need improvement and are subject to our future work.
:Incomplete timer driver:
On x86, the timer driver should program the PIT to dispatch sleep requests.
However, the I/O ports of the PIT can only by made available to one party in
the system (which naturally would be the timer driver). Unfortunately, there
are some VESA BIOSes around, which try using the PIT directly. The current
version of our VESA driver does not virtualize these accesses. It rather
tries to gain direct access to the I/O ports from core. This would not work
if the timer already uses this device resource. Our plan is to supplement
our VESA driver with a virtual PIT that uses the timer service as back end.
Then we can safely use the PIT by the timer driver.
:Signalling framework not yet implemented:
We have not yet implemented Genode's API for asynchronous notifications
in the OKL4 version. In fact, the goal of the initial version of the
OKL4 support was running the default demonstration scenario, which does
not rely on signals. The second and more technical reason is that we
consider exploiting OKL4's event mechanism for implementing the signalling
API but have not finalized the design. The generic implementation as used
on the other platforms cannot be used on OKL4 because this implementation
utilizes one helper thread per signal transmitter. Within core, each RM
session is a potential signal transmitter, which means that we need to
create a helper thread per process. Unfortunately, by default, OKL4
limits the number of threads within roottask (core) to only 8 threads,
which would impose a severe limit on the number of processes we could
run on OKL4.
:OKL4's kernel mutexes yet to be used:
We have not yet explored the use of mutexes provided by the OKL4 kernel
for implementing Genode synchronization APIs but we rather rely on a
yielding spin lock for now. This has a number of drawbacks such as high
wake-up latencies in the contention case (depending on the scheduling
time slice), no support for priorities, and no fairness. Although it
is a simple and robust solution to start with, we plan to facilitate
the OKL4 kernel feature with our upcoming work.
:Overly simplistic UTCB allocation:
Right now, we allocate a fixed amount of 32 UTCBs per address space and
thereby limit the maximum number of threads per process. In the future,
this limit should be made configurable.
:Managed dataspaces not yet supported:
The support of managed dataspaces relies on the signal API, which is
not yet available for OKL4.
:Message buffers are limited to 256 bytes:
Because OKL4 performs message-based inter-process communication by
copying data between the UTCBs of the communicating threads, the
UTCB size constaints the maximum message size. Therefore, message
must not exceed 256 bytes. This is not a huge problem for the currently
available Genode programs but we can imagine session argument-lists
to become larger in the future.
:Advanced thread functions are incomplete:
Thread functions such as querying registers of remote threads are not yet
implemented.
Integration of Qt4 into the mainline repository
###############################################
Qt4 is a tool kit for developing platform-independent applications. It
comprises a complete platform-abstraction layer and a rich GUI tool kit
widely used for commercial and open-source applications. It is particularly
known as the technical foundation of the KDE project. The previous Genode
release was accompanied by a snapshot of our initial port of Qt4 to Genode. For
the current release, we have turned this proof-of-concept implementation into a
properly integrated part of the Genode mainline development. This enables Qt4
applications to be executed natively on the full range of kernels supported by
Genode.
Usage
=====
We complemented Genode's source tree with the new 'qt4' source-code repository,
which contains the Genode-specific parts of the Qt4 framework. The most
portions for the Qt4 framework are used unmodified and thereby have not been
made part of the Genode source tree. Instead, we fetch the original Qt4 source
code from Trolltech's FTP server. This way, our source tree remains tidy and
neat.
For using Qt4 for your Genode applications, you first need to download and
prepare the original Qt4 source codes and build a few Qt4 tools such as the
meta-object compiler and the resource compiler. The makefile found in the
top-level directory of the 'qt4' repository automates this task:
! make prepare
To include the 'qt4' repository into the Genode build process, just add the
'qt4' directory to the 'REPOSITORIES' declaration of the 'etc/build.conf' file
within your build directory. Make sure that the repositories 'demo' and 'libc'
are included as well. The 'qt4' repository comes with a couple of demo applications.
The 'qt_launchpad' is especially interesting because it makes use of both the
Qt4 framework and the Genode framework in one application.
Features and limitations
========================
The Qt4 port comprises Qt's Core library, GUI library, Script library, XML
library, and the UI tools library.
For using Qt4 on the Linux version of Genode, we recommend using the Genode
tool chain rather than your host's tool chain. Qt4 makes use of a lot of libc
functionality, supplied by Genode's 'libc' repository. However, on Linux we
still link against your host's libc. This becomes a problem if your host
compiler's C++ support code references libc functionality that is not part of
Genode's libc. Thereby the linker will silently create references to glibc
symbols, making both libraries collide. So if using Qt4, we recommend using the
Genode tool chain:
:[https://genode.org/download/tool-chain]:
Information about downloading and using the Genode tool chain
USB support
###########
This release introduces the first fragments of USB support to Genode, taking
the USB human-interface device (HID) class as starting point. With this work,
we follow our approach of reusing unmodified Linux device drivers executed
within a device-driver environment called DDE Linux. In the previous release,
we already utilized this approach for realizing basic networking on Genode.
With this release, we complement DDE Linux with support required by USB
drivers. We are grateful for being able to base our implementation on the
excellent foundation laid by Dirk Vogt. He described his work in
[https://os.inf.tu-dresden.de/papers_ps/beleg-vogt.pdf - USB for the L4 environment].
For USB HID support, we added the Linux USB and input subsystems to the DDE
Linux 2.6 framework. Besides the 'dde_linux26/net.h' API for network drivers
added in Genode 9.02, the current version also includes APIs for input
('dde_linux26/input.h') and USB ('dde_linux26/usb.h'). We intend these
interfaces to mature towards generic driver-library APIs in the future. For
example, BSD-based drivers shall transparently provide the same functionality
as the current Linux drivers, which permits the simple reuse of driver server
implementations.
[image usb_current]
Image [usb_current] illustrates the current implementation of the USB-based
human-interface device (HID) driver. In this monolithic setup, all parts of the
USB stack and the device API are executed within one address space. These parts
are
* Input server glue code
* HID driver and input subsystem
* Core functions for management of USB request buffers (URBs),
attached devices, and registered drivers
* Host controller drivers for UHCI, OHCI, and EHCI
[image usb_aspired]
We regard this as an intermediate step towards our goal to decompose the USB
stack. Image [usb_aspired] shows our aspired design. In this design, the
USB server and one or more USB gadget drivers run in dedicated address spaces.
The USB server provides two interfaces called USB session interface and USB
device interface. A USB session interface corresponds to a virtual root hub,
from which USB devices can be queried. The client of the USB session interface
is usually an USB gadget driver that uses the USB device interface. Because
this interface is used for transferring the actual payload at a potentially
high bandwidth, it is based on shared memory and signals. The USB server
consists of the following components:
* USB server glue code
* Virtual USB device driver managing all attached devices
* Core functions including hardware hub management
* Host controller drivers
The USB server presents a virtual USB hub to each USB gadget driver. Such
a driver consists of:
* Device interface, e.g., input server glue code
* Gadget driver, e.g., HID driver and input subsystem
* Core functions
* Virtual host controller
* USB client glue code
The HID driver uses the USB session API to monitor ports of its virtual root
hub and submit URBs to attached devices. The session interface facilitates the
signalling framework for event notification and a shared-memory dataspace for
URB transmission.
The 'os' repository already contains the USB session and USB device interfaces.
However, the decomposition is not yet in a functional state.
:Current limitations:
The current monolithic implementation of the USB HID service can already be
used as a replacement of the PS/2 driver. However, both drivers cannot be used
at the same time, yet. To enable the use of both USB HID and PS/2, we plan to
create a further component that merges multiple streams of input events and
passes the result to the GUI server.
OKLinux on Genode
#################
According to our road map, we pursued the goal to run Linux as a node in
Genode's process tree. We explored two approaches:
:Reanimating the Afterburner project conducted by the [http://l4ka.org - L4Ka group]:
This approach is the result of the L4Ka groups's long-year experience with
manually supporting L4Linux on top of the L4ka::Pistachio kernel. Because of
the high costs of maintaining the paravirtualized Linux kernel, a
semiautomatic paravirtualization technique was created. According to the
impressive results presented in
[http://www.l4ka.org/l4ka/publ_2006_levasseur-ua_soft-layering.pdf - Pre-Virtualization: Soft Layering for Virtual Machines],
this approach is able to drastically reduce maintenance costs while retaining
good performance. Furthermore, the approach was applied not only to Linux
running on the L4 kernel but also for using Xen or Linux as underlying
host operating systems.
:Porting the OKL4-specific version of L4Linux to Genode:
Open Kernel Labs maintain a custom version of L4Linux that runs on OKL4. This
version is mostly referred to as OKLinux aka Wombat. Since Genode can now use OKL4
as base platform, the reuse of OKLinux in combination with Genode has become
a feasible option.
Both approaches have pros and cons. Whereas Afterburner is a intriguing
approach, this project seems to be stalled. It relies on a rather old tool
chain, and recent Linux features such as thread-local storage support are not
considered, yet. To pick up this solution for Genode will require us to fully
understand the mechanisms and the code. So we consider this as a mid-term
solution. In short term, running OKLinux on Genode is more feasible. We were
already able to create a prototype version of OKLinux running on Genode. This
version starts up the kernel including all Linux kernel threads, mounts the
boot partition supplied as memory image, and starts the init process. The
engineering costs had been rather low. We replaced the Iguana user land
libraries as originally used by Wombat by a Genode-specific reimplementation to
keep our manual adaptions of the Linux kernel code as small as possible.
Our custom reimplementation of the needed Iguana APIs consists of less than
1,000 lines of code (SLOC). The diff for our changes to the OKLinux kernel code
comprises less than 1,000 lines. We plan to make a snapshot of this prototype
publicly available soon.
Operating-system services and libraries
#######################################
Nitpicker GUI server
====================
We optimized the performance of the 'refresh' call, which updates all views of
a session, which display a given buffer portion. The new implementation restricts
the redraw operations to the fragment of each view that displays the specified
buffer portion. The performance improvement becomes most visible when updating
only small parts of a buffer.
USB session interface
=====================
Genode's emerging USB support introduces two new interfaces to the 'os' repository,
which are called USB session and USB device.
An _USB_session_ is a virtual USB bus with just one root hub with 'MAX_PORTS'
downstream ports. The client of such as session submits USB request blocks
(URBs) and is, in turn, informed about port changes on the root hub as well as
request completion. Connected USB devices can be referenced by USB device
capabilities and are associated with one port at the virtual root hub on
server side.
An _USB_device_ references a hardware device connected to a virtual USB bus's
root hub. The device capability enables the client to send USB request
blocks to the hardware device.
Input interface
===============
We updated the key codes of the input interface in response to recent changes
of Linux' 'dev/event' definitions.
VESA driver
===========
Until now, there existed different processes that tried to access the PCI bus
via I/O ports, in particular the VESA framebuffer driver and the PCI bus
driver.
Since core enforces that each I/O port can only be assigned exclusively to one
component in the system, multiple processes that need access to the same I/O
ports cannot run at the same time. For our default demonstration scenario, we
had been able to allow the VESA driver to use the PCI I/O ports because nobody
else needed them. However, our growing base of device drivers relies on the
PCI bus driver. To be able to use the VESA driver together with other drivers,
we virtualized the access to the PCI bus from within the VESA driver.
Our current PCI virtualization code is pretty limited. The VESA driver sees a
virtual PCI bus with only the VGA card attached. For now, we only allow reading
the PCI configuration space of this device, but not writing to it. Apparently,
this simple approach is sufficient to run the VESA BIOS of Qemu. However, other
VESA BIOS implementations may need further access to the PCI device's
configuration space. For example, for querying the size of a PCI resource,
write access to base address registers is required. In such an event, the VESA
driver will print a message about the missing virtualization feature:
! writing data register not supported
If you see such a message, we are very interested to see your log output such
that we can enhance our PCI virtualization code as needed. Please contact us!
Base framework
##############
In the process of bringing Genode to the OKL4 kernel, we have generalized much
of former platform-specific code:
* The initialization of C++ exception handling has now become part of the
generic 'cxx' library in the 'base' repository. All platforms except
Linux are using this generic library now.
* The 'server' library used to contain a platform-specific part that
implemented the 'manage' function of a 'Server_entrypoint'. The
generalized version of this library is now being used on all platforms
other than Linux.
* We unified core-internal interfaces and their implementations such as
'Dataspace_component', 'Cap_session_component', 'Rm_session_component',
and 'Irq_session_component'. The result has become part of the 'base'
repository.
* On OKL4, threads need to execute small startup code for querying their
own thread IDs. Therefore, we have extended the 'Thread_base' interface
with a platform-specific hook function called '_thread_bootstrap'.
* The types defined in 'base/native_types.h' had been complemented by a
new 'Native_thread_id' type. This type is exclusively used by core and the
framework libraries. For using the Genode API, this type is meaningless.
* For the 64bit support, we slightly refined the interfaces of some utility
template functions in 'util/misc_math.h'. Furthermore, parts of the generic
marshalling code of the IPC framework needed refinement, but no API changes
were needed.
Linux-specific changes
######################
Adaptation to 64 bit
====================
Because most Genode developers tend to work with the Linux version of Genode,
supporting 64-bit Linux becomes increasingly important. With the current release,
we start to officially support 64-bit Linux as base platform. This comes
along with the following changes:
* We replaced the 'spec-x86.mk' file with new 'spec-x86_32.mk' and 'spec-x86_64.mk'
files. The default version of 'base-linux/etc/specs.conf' automatically
chooses the right spec file according to the output of 'uname -m'. Therefore,
output of the build processes matches your host architecture. This behaviour
can be changed by placing a customized 'spec.conf' file in your build directory's
'etc/' subdirectory.
* We added type definitions for 64-bit-specific fixed-size integers in the form
of a 64-bit-specific 'fixed_stdint.h' file.
* Because using 64 bit instead of 32 bit changes the payload size of RPC
messages, we had to adjust several message buffers such as 'Ram_session_client'
and 'Input::Session_client', and adapted the used stack sizes.
* Towards the goal of completely dissolving Genode's dependency on the Linux' glibc,
we implemented custom system-call bindings. Apparently, Linux' syscall interface
differs between 32 bit and 64 bit. For example, the 32-bit version handles
all socket-related calls via a compound 'socketcall' whereas the 64-bit
version uses distinct syscalls. Another difference is the handling of the
'mmap' syscall and different behaviour of 'tkill'. The latter problem was
resolved by replacing 'tkill' with 'tgkill' and setting the thread-group
argument of the corresponding PID. Therefore, a 'Native_thread_id' on Linux
now comprises both the TID and the PID.
* The 'Platform_env' on Linux contains a local implementation of the 'Rm_session'
interface, which uses 'mmap' to attach dataspaces to the process' address
space and maintains the region list in the form of a static array. This array
was dimensioned to 256 entries, which constrained the maximum amount of
usable memory when allocating a lot of small blocks via Genode's heap. Since
the heap allocates backing store at the granularity of only 16KB, the worst
case for reaching this limit was about 4MB. This was OK for our simple test
applications. But for using Qt4, in particular on 64 bit, this has become a
serious limitation. For now, we increased the region limit to 4096 and plan
to replace the static array with a dynamically growing data structure.
Furthermore, we made the heap granularity depend on the actual machine-word
size. Therefore, the heap allocates its backing store in 32KB blocks when
running on 64 bit.
Debugging hooks
===============
On Linux, we use gdb for debugging Genode. This is feasible as long as the
targeted process is running. However, during low-level debugging, we had the
recurring problem of a thread dying shortly after starting up. So we added a hook
for halting a thread at the startup in order to be able to attach gdb to the
thread before it dies. This simple hook lets the thread wait for a key press by
directly calling the 'read' syscall. We also added a simple debug facility for
printing debug messages bypassing Genode's LOG service by calling the 'write'
syscall directly. Both hooks are now part of the Linux version of the 'env'
library (see 'base-linux/src/base/env/debug.cc'). Note that these hooks are not
part of the Genode API. There exists no header file.

0
doc/release_notes-09-08.txt → doc/release_notes/09-08.txt
View File

1016
doc/release_notes/09-11.txt Normal file
View File

File diff suppressed because it is too large Load Diff

1224
doc/release_notes/10-02.txt Normal file
View File

File diff suppressed because it is too large Load Diff

1211
doc/release_notes/10-05.txt Normal file
View File

File diff suppressed because it is too large Load Diff

0
doc/release_notes-10-08.txt → doc/release_notes/10-08.txt
View File

871
doc/release_notes/10-11.txt Normal file
View File

@@ -0,0 +1,871 @@
===============================================
Release notes for the Genode OS Framework 10.11
===============================================
Genode Labs
During the past three months, the Genode project was primarily driven by our
desire to create a bigger picture out of the rich set of components that we
introduced over time, in particular over the last year. Looking back at the
progress made since mid 2009, there were many functional additions to the
framework, waiting to get combined. To name a few, we added support for
networking, audio output, real-time priorities, mandatory access control,
USB, ATAPI block devices, Python, hardware-accelerated 3D graphics, Qt4,
the WebKit-based Arora browser, and the paravirtualized OKLinux kernel.
So many wonderful toys waiting to get played with. This is how the idea of
creating [https://genode.org/download/live-cds - the new Genode Live CD] was
born. In the past, Genode was mostly used in settings with a relatively static
configuration consisting of several components orchestrated to fulfill
a few special-purpose functions. Now, the time has come for the next step,
creating one dynamic setup that allows for the selection of different subsystems
at runtime rather than at boot time.
This step is challenging in several ways. First, the processes that form
the base system have to run during the entire time of all demo setups. If
any of those processes contained stability problems or leaked memory, it would
subvert the complete system. Second, the components of all subsystems combined
are far too complex to be loaded into memory at boot time. This would not
only take too long but would consume a lot of RAM. Instead, those components
and their data had to be fetched from disk (CDROM) on demand. Third, because
multiple demo subsystems can be active at a time, low-level resources such as
networking and audio output must be multiplexed to prevent different
subsystems from interfering with each other. Finally, we had to create a
single boot and configuration concept that is able to align the needs of all
demos, yet staying manageable.
Alongside these challenges, we came up with a lot of ideas about how Genode's
components could be composed in new creative ways. Some of these ideas such
as the browser-plugin concept and the http-based block server made it onto
the Live CD. So for producing the Live CD, we not only faced the said
technical challenges but also invested substantial development effort in new
components, which contributed to our overall goal. Two weeks ago, we released
the Live CD. This release-notes document is the story about how we got there.
To keep ourself focused on the mission described above, we deferred the
original roadmap goal for this release, which was the creation of a Unix-like
runtime environment to enable compiling Genode on Genode. This will be the
primary goal for the next release.
Execution environment for gPXE drivers
######################################
Up to now, DDE Linux provided Genode with drivers for hardware devices
ranging from USB HID to WLAN. In preparation of the live CD, we
noticed the demand for support of a broader selection of ethernet
devices. Intel's e1000 PCI and PCIe cards seemed to mark the bottom
line of what we had to support. The major advantage of NIC drivers
from Linux is their optimization for maximum performance. This emerges
a major downside if DDE Linux comes into play: We have to provide all
the nifty interfaces used by the driver in our emulation framework. To
achieve our short-term goal of a great live CD experience, we had to
walk a different path.
[https://gpxe.org/ - gPXE] is a lovely network boot loader / open-source
PXE ROM project and the successor of the famous Etherboot
implementation. Besides support for DNS, HTTP, iSCSI and AoE, gPXE
includes dozens of NIC drivers and applies a plain driver framework.
As we were also itching to evaluate DDE kit and the DDE approach at
large with this special _donator OS_, we went for implementing the
device-driver environment for gPXE (DDE gPXE).
The current version provides drivers for e1000, e1000e, and pcnet
devices. The emulation framework comprises just about 600 lines of
code compared to more than 22,000 LOC reused unmodified from gPXE.
Benchmarks with the PCNet32 driver showed that DDE gPXE's performance
is comparable to DDE Linux.
The gPXE driver environment comes in the form of the new 'dde_gpxe'
repository. For building DDE gPXE, you first need to download and patch
the original sources. The top-level makefile of this repository automates
this task. Just issue:
! make prepare
Now, you need to include the DDE gPXE repository into your Genode
build process. Just add the path to this directory to the
'REPOSITORIES' declaration of the 'etc/build.conf' file within your
build directory, for example
! REPOSITORIES += $(GENODE_DIR)/dde_gpxe
After successful build the DDE gPXE based ethernet driver is located
at 'bin/gpxe_nic_drv'.
On-demand paging
################
In the [https://genode.org/documentation/release-notes/8.11#section-8 - release 8.11],
we laid the foundation for implementing user-level dataspace managers.
But so far, the facility remained largely unused except for managing thread
contexts. This changed with this release.
So what is a user-level dataspace manager and who needs it? In short,
Genode's memory management is based on dataspaces. A dataspace is a
container for memory. Normally, it is created via core's RAM or ROM
services. The RAM service hands out dataspaces containing contiguous
physical memory. After allocating such a RAM dataspace, the creator can
attach the dataspace to its own address space to access the dataspace
content. In addition, it can pass a dataspace reference (called dataspace
capability) to other processes, which, in turn, can than attach the same
dataspace to their local address space, thereby establishing shared memory.
Similarly, core's ROM service hands out boot-time binary data as dataspaces.
For the most use cases of Genode so far, these two core services were the
only dataspace providers needed. However, there are use cases that require
more sophisticated memory management. For example, to implement swapping,
the content of a dataspace must be transferred to disk in a way that
is transparent to the users of the dataspace. In monolithic kernels, such
functionality is implemented in the kernel. But on a multi-server OS
such as Genode, this is no option. Implementing such a feature into
core would increase the trusted computing base of all applications
including those who do not need swapping. Core would need a hard-disk
driver, effectively subverting the Genode concept. Other examples for
advanced memory-management facilities are copy-on-write memory and
non-contiguous memory - complexity we wish to avoid at the root of the
process tree. Instead of implementing such memory management facilities
by itself, core provides a mechanism to let any process manage dataspaces.
This technique is also called user-level page-fault handling.
For the Live CD, we decided to give Genode's user-level page-fault handling
facility a go. The incentive was accessing files stored on CDROM in an
elegant way. We wanted to make the CDROM access completely transparent to
the applications. An application should be able to use a ROM session as
if the file was stored at core's ROM service. But instead of being
provided by core, the session request would be delegated to an
alternative ROM service implementation that reads the data from disk
as needed. Some of the files stored in the CDROM are large. For example,
the disk image that we use for the Linux demo is 160MB. So reading
this file at once and keeping it in memory is not an option. Instead, only
those parts of the file should be read from disk, which are actually
needed. To uphold the illusion of dealing with plain ROM files for
the client, we need to employ on-demand-paging in the CDROM server.
Here is how it works.
# The dataspace manager creates an empty managed dataspace. Core
already provides a tool for managing address spaces called
region manager (RM service). A RM session is an address space,
to which dataspaces can be attached. This is exactly what is
needed for a managed dataspace. So a dataspace manager uses the
same core service to define the layout of a managed dataspace
as is used to manage the address space of a process. In fact,
any RM session can be converted into a managed dataspace.
! enum { MANAGED_DS_SIZE = 64*1024*1024 };
! Rm_connection rm(0, MANAGED_DS_SIZE);
This code creates a RM session with the size of 64MB. This is an empty
address space. A dataspace capability that corresponds to this address
space can then be requested via
! Dataspace_capability ds = rm.dataspace();
# The dataspace capability can be passed to a client, which may
attach the dataspace to its local address space. Because the
managed dataspace is not populated by any backing store, however,
an access would trigger a page fault, halting the execution of
the client. Here, the page-fault protocol comes into play.
# The dataspace manager registers itself for receiving a signal each time
a fault occurs:
! Signal_receiver rec;
! Signal_context client;
! Signal_context_capability sig_cap = rec.manage(client);
! rm.fault_handler(sig_cap);
When an empty part of the managed dataspace is accessed by any
process, a signal is delivered. The dataspace manager can then
retrieve the fault information (access type, fault address) and
dispatch the page fault by attaching a real dataspace at the
fault address of the managed dataspace. In a simple case, the code
looks as follows:
! while (true) {
! Signal signal = rec.wait_for_signal();
! for (int i = 0; i < signal.num(); i++) {
! Rm_session::State state = rm.state();
! ds = alloc_backing_store_dataspace(PAGE_SIZE);
! rm.attach_at(ds, state.addr & PAGE_MASK);
! }
! }
This simple page-fault handler would lazily allocate a page of
backing store memory each time a fault occurs. When the backing
store is attached to the managed dataspace, core will automatically
wake up the faulted client.
# The example above has the problem that the dataspace manager has
to pay for the backing store that is indirectly used by the client.
To prevent the client from exhausting the dataspace manager's memory,
the dataspace manager may choose to use a limited pool of backing
store only. If this pool is exceeded, the dataspace manager can reuse
an already used backing-store block by first revoking it from its
current managed dataspace:
! rm.detach(addr);
This will flush all mappings referring to the specified address
from all users of the managed dataspace. The next time, this
address region is accessed, a new signal will be delivered.
This page-fault protocol has the following unique properties. First,
because core is used as a broker between client and dataspace manager, the
dataspace manager remains completely unaware of the identity of its client.
It does not even need to possess the communication right to the client. In
contrast, all other user-level page-fault protocols that we are aware of
require direct communication between client and dataspace manager. Second,
because dataspaces are used as first-level objects to resolve page faults,
page faults can be handed at an arbitrary granularity (of course, a multiple
of the physical page size). For example, a dataspace manager may decide to
attach backing-store dataspaces of 64K to the managed dataspace. So the
overhead produced by user-level page-fault handler can be traded for the
page-fault granularity. But most importantly, the API is the same across
all kernels that support user-level page fault handling. Thus the low-level
page-fault handling code becomes inherently portable.
Having said that, we have completed the implementation of the described
core mechanisms, in particular the 'detach' facility, for OKL4. The ISO9660
driver as featured on the Live CD implements the 'ROM' interface and
reads the contents of those files from CDROM on demand. It uses a
fixed pool of backing store, operates at a page-fault granularity of
64KB, and implements a simple fifo replacement strategy.
Base framework
##############
There had been only a few changes to the base framework described as
follows.
We unified the core-specific console implementation among all
base platforms and added synchronization of 'vprintf' calls.
The kernel-specific code resides now in the respective
'base-<platform>/src/base/console/core_console.h' files.
We removed the argument-less constructor from 'Allocator_avl_tpl'.
This constructor created an allocator that uses itself for
meta-data allocation, which is the usual case when creating
local memory allocators. However, on Genode, this code is typically
used to build non-memory allocators such as address-space regions.
For these use cases, the default policy is dangerous. Hence, we
decided to remove the default policy.
The 'printf' helper macros have been unified and simplified. The
available macros are 'PINF' for status information, 'PWRN' for warnings,
'PLOG' for log messages, and 'PERR' for errors. By default, the message
types are colored differently to make them easily distinguishable.
In addition to normal messages, there is the 'PDBG' for debugging
purposes. It remains to be the only macro that prints the function name
as message prefix and is meant for temporary messages, to be removed
before finalizing the code.
Genode's on-demand-paging mechanism relies on the signalling framework.
Each managed dataspace is assigned to a distinct signal context.
Hence, signal contexts need to be created and disposed alongside
with managed dataspaces. We complemented the signalling framework
with a 'dissolve' function to enable the destruction of signal
contexts.
Operating-system services and libraries
#######################################
Finished transition to new init concept
=======================================
With the release 10.05, we introduced the
[https://genode.org/documentation/release-notes/10.05#section-0 - current configuration concept of init].
This concept supports mandatory access control and provides flexible
ways for defining client-server relationships. Until now, we maintained
the old init concept. With the current release, the transition to the
new concept is finished and we removed the traditional init.
We retained the support for loading configurations for individual subsystems
from different files but adopted the syntax to the use of attributes.
Instead of
! <configfile>subsystem.config</configfile>
the new syntax is
! <configfile name="subsystem.config"/>
Virtual network bridge (Proxy ARP)
==================================
Since we originally added networking support to Genode, only one program could
use the networking facilities at a time. In the simplest form, such a program
included the network driver, protocol stack, and the actual application. For
example, the uIP stack featured with release 9.02 followed this approach.
In release 9.11 we added the 'Nic_session' interface to decouple the network
driver from the TCP/IP protocol stack. But the 1-to-1 relation between
application and network interface remained. With the current release, we
introduce the 'nic_bridge' server, which is able to multiplex the 'Nic_session'
interface.
The implementation is roughly based on the proxy ARP RFC 1027. At startup, the
'nic_bridge' creates a 'Nic_session' to the real network driver and, in turn,
announces a 'Nic' service at its parent. But in contrast to a network driver
implementing this interface, 'nic_bridge' supports an arbitrary number of
'Nic_sessions' to be opened. From the client's perspective, such a session
looks like a real network adaptor.
This way, it has become possible to run multiple TCP/IP stacks in
parallel, each obtaining a distinct IP address via DHCP. For example,
is has become possible to run multiple paravirtualized Linux kernels
alongside an lwIP-based web browser, each accessing the network via a
distinct IP address.
As a side effect for developing the 'nic_bridge', we created a set
of utilities for implementing network protocols. The utilities are
located at 'os/include/net' and comprise protocol definitions for
ethernet, IPv4, UDP, ARP, and DHCP.
Nitpicker GUI server
====================
Our work on the Live CD motivated several improvements of the Nitpicker
GUI server.
Alpha blending
~~~~~~~~~~~~~~
In addition to nitpicker's plain pixel buffer interface that is compatible
with a normal framebuffer session, each nitpicker session can now have
an optional alpha channel as well as an corresponding input-mask channel
associated. Both the alpha channel and the input mask are contained in the
same dataspace as the pixel buffer. The pixel buffer is followed by
the 8-bit alpha values, which are then followed by the input-mask values.
This way, the presence of an alpha channel does not interfere with the
actual pixel format. Each 8-bit input mask value specifies the user-input
policy for the respective pixel. If the value is zero, user input
referring to the pixel is not handled by the client but "falls through"
the view that is visible in the background of the pixel. This is typically
the case for drop shadows. If the input-mask value is '1', the input
is handled by the client.
With the input-mask mechanism in place, we no longer have a definitive
assignment of each pixel to a single client anymore. In principle, an
invisible client is able to track mouse movements by creating a full-screen
view with all alpha values set to '0' and all input-mask values set to '1'.
Once, the user clicks on this invisible view, the user input gets routed
to the invisible client instead of the actually visible view. This
security risk can be addressed at two levels:
* In X-Ray mode, nitpicker completely disables alpha blending
and the input-mask mechanism such that the user can identify the
client that is responsible for each pixel on screen.
* The use of the alpha channel is a session argument, which is specified
by nitpicker clients at session-creation time. Consequently, this
session argument is subjected to the policy of all processes involved
with routing the session request to nitpicker. Such a policy may permit
the use of an alpha channel only for trusted applications.
_Caution:_ The use of alpha channels implies read operations from
the frame buffer. On typical PC graphics hardware, such operations are
extremely slow. For this reason, the VESA driver should operate in
buffered mode when using alpha blending in Nitpicker.
Tinted views in X-Ray mode
~~~~~~~~~~~~~~~~~~~~~~~~~~
We added support for tinting individual clients or groups of clients
with different colors based on their label as reported at session-creation
time. By using session colors, nitpicker assists the user to tell apart
different security domains without reading textual information. In
addition to the tinting effect, the title bar presents the session
color of the currently focused session.
The following nitpicker configuration tints all views of the launchpad
subsystem in blue except for those views that belong to the testnit
child of launchpad. Those are tinted red.
! <config>
! <policy label="launchpad" color="#0000ff"/>
! <policy label="launchpad -> testnit" color="#ff0000"/>
! </config>
Misc Nitpicker changes
~~~~~~~~~~~~~~~~~~~~~~
We introduced a so-called 'stay-top' session argument, which declares
that views created via this session should stay on top of other views.
This function is useful for menus that should always remain accessible
or banner images as used for Live CD.
Nitpicker's reserved region at the top of the screen used to cover up
the screen area as seen by the clients. We have now excluded this area
from the coordinate system of the clients.
We implemented the 'kill' mode that can be activated by the 'kill' key.
(typically this is the 'Print Screen' key) This feature allows the user
to select a client to be removed from the GUI. The client is not
actually killed but only locked out. The 'kill' mode is meant as an
emergency brake if an application behaves in ways not wanted by the
user.
ISO9660 server
==============
As outlined in Section [On-demand paging], we revisited the ISO9660 server
to implement on-demand-paged dataspaces. It is the first real-world
use case for Genode's user-level page-fault protocol. The memory pool
to be used as backing store for managed dataspaces is dimensioned according
to the RAM assigned to the iso9660 server. The server divides this backing
store into blocks of 64KB and assigns those blocks to the managed dataspaces
in a fifo fashion. We found that using a granularity of 64KB improved the
performance over smaller block sizes because this way, we profit from reading
data ahead for each block request. This is particularly beneficial for
CDROM drives because of their extremely long seek times.
Audio mixer
===========
We added a new *channel synchronization* facility to the 'Audio_out_session'
interface. An 'Audio_out_session' refers to a single channel. For stereo
playback, two sessions must be created. At session-creation time, the
client can provide a hint about the channel type such as "front-left" as
session-construction argument. This design principally allows for supporting
setups with an arbitrary amount of channels. However, those channels must
be synchronized. For this reason, we introduced the 'sync_session' function
to the 'Audio_out_session' interface. It takes the session capability of
another 'Audio_out_session' as argument. The specified session is then
used as synchronization reference.
To reduce the latency when stopping audio replay, we introduced a new *flush*
function to the 'Audio_out_session' interface. By calling this function,
a client can express that it is willing to discard all audio data already
submitted to the mixer.
Furthermore, we improved the audio mixer to support both long-running
streams of audio and sporadic sounds. For the latter use case, low latency
is particularly critical. In this regard, the current implementation is a
vast improvement over the initial version. However, orchestrating the
mixer with audio drivers as well as with different clients (in particular
ALSA programs running on a paravirtualized Linux) is not trivial. In the
process, we learned a lot, which will eventually prompt us to further
optimize the current solution.
Nitpicker-based virtual Framebuffer
===================================
To support the browser-plugin demo, we introduced 'nit_fb', which is a
framebuffer service that uses the nitpicker GUI server as back end. It
is similar to the liquid framebuffer as featured in the 'demo' repository
but in contrast to liquid framebuffer, 'nit_fb' is non-interactive.
It has a fixed screen position and size. Furthermore, it does not
virtualize the framebuffer but passes through the framebuffer portion of
the nitpicker session, yielding better performance and lower latency.
If instantiated multiple times, 'nit_fb' can be used to statically arrange
multiple virtual frame buffers on one physical screen. The size and screen
position of each 'nit_fb' instance can be defined via Genode's configuration
mechanism using the following attributes of the 'nit_fb' config node:
! <config xpos="100" ypos="150"
! width="300" height="200"
! refresh_rate="25"/>
If 'refresh_rate' isn't set, the server will not trigger any refresh
operations by itself.
On the Live CD, each browser plugin instantiates a separate instance of
'nit_fb' to present the plugin's content on screen. In this case, the
view position is not fixed because the view is further virtualized by the
loader, which imposes its policy onto 'nit_fb' - Genode's nested
policies at work!
TAR ROM service
===============
For large setups, listing individual files as boot modules in single-image
creation tools (e.g., elfweaver) or multiboot boot loaders can be
cumbersome, especially when many data files or shared libraries are
involved. To facilitate the grouping of files, 'tar_rom' is an
implementation of the 'ROM' interface that operates on a 'tar' file.
The name of the TAR archive must be specified via the 'name' attribute of
an 'archive' tag, for example:
! <config>
! <archive name="archive.tar"/>
! </config>
The backing store for the dataspaces exported via ROM sessions is accounted
on the 'rom_tar' service (not on its clients) to make the use of 'rom_tar'
transparent to the regular users of core's ROM service. Hence, this service
must not be used by multiple clients that do not trust each other.
Typically, 'tar_rom' is instantiated per client.
The Live CD uses the 'tar_rom' service for the browser demo. Each plugin
is fetched from the web as a tar file containing the config file of the
plugin subsystem as well as supplemental binary files that are provided
to the plugin subsystem as ROM files. This way, a plugin can carry along
multiple components and data that form a complete Genode subsystem.
DDE Kit
=======
The DDE kit underwent slight modifications since the previous release.
It now provides 64-bit integer types and a revised virtual PCI bus
implementation.
Device drivers
##############
PCI bus
=======
Genode was tested on several hardware platforms in preparation of the
current release. This revealed some deficiencies with the PCI bus
driver implementation. The revised driver now efficiently supports
platforms with many PCI busses (as PCIe demands) and correctly handles
multi-function devices.
VESA framebuffer
================
We updated the configuration syntax of the VESA driver to better match
the style of new init syntax, preferring the use of attributes rather than
XML sub nodes. Please refer to the updated documentation at
'os/src/drivers/framebuffer/vesa/README'.
:Buffered output:
To accommodate framebuffer clients that need to read from the frame buffer,
in particular the nitpicker GUI server operating with alpha channels, we
introduced a buffered mode to the VESA driver. If enabled, the VESA driver
will hand out a plain memory dataspace to the client rather than the
physical framebuffer. Each time, the client issues as 'refresh' operation
on the framebuffer-session interface, the VESA driver copies the corresponding
screen region from the client-side virtual framebuffer to the physical
framebuffer. Note that the VESA driver will require additional RAM quota
to allocate the client buffer. If the quota is insufficient, the driver will
fall back to non-buffered output.
:Preinitialized video modes:
As an alternative to letting the VESA driver set up a screen mode, the
driver has become able to reuse an already initialized mode, which is useful
if the VESA mode is already initialized by the boot loader. If the screen
is initialized that way, the 'preinit' attribute of the 'config' node can
be set to '"yes"' to prevent the driver from changing the mode. This way,
the driver will just query the current mode and make the already
initialized framebuffer available to its client.
Audio
=====
We observed certain hardware platforms (in particular VirtualBox) to
behave strangely after ALSA buffer-underrun conditions. It seems that the
VirtualBox audio driver plays actually more frames than requested by
ALSA's 'writei' function, resulting in recurring replay of data that
was in the buffer at underrun time. As a work-around for this problem,
we zero-out the sound-hardware buffer in the condition of an ALSA buffer
underrun. This way, the recurring replay is still there, but it is
replaying silence.
To improve the support for sporadic audio output, we added a check for the PCM
state for buffer underruns prior issuing the actual playback. In the event of
an underrun, we re-prepare the sound card before starting the playback.
Furthermore, we implemented the new flush and channel-synchronization
abilities of the 'Audio_out_session' interface for the DDE Linux driver.
Paravirtualized Linux
#####################
To support the demo scenarios that showcase the paravirtualized Linux kernel,
we enhanced our custom stub drivers of the OKLinux kernel. Thereby, we have
reached a high level of integration of OKLinux with native Genode services,
including audio output, block devices, framebuffer output, seamless integration
with the Nitpicker GUI, and networking. All stub drivers are compiled in by
default and are ready to use by specifying a device configuration in the config
node for the Linux kernel. This way, one Linux kernel image can be easily used
in different scenarios.
:Integration with the Nitpicker GUI:
We enhanced our fbdev stub driver with a mechanism to merge view reposition
events. If a X11 window is moved, a lot of subsequent events of this type are
generated. Using the new optimization, only the most recent state gets
reported to Nitpicker, making the X11 GUI more responsive.
:UnionFS:
As we noticed that unionfs is required by all our Linux scenarios, we decided
to include and enable the patch by default.
:Network support:
With the introduction of the 'nic_bridge', multiple networking stacks can run
on Genode at the same time, which paves the way for new use cases. We have now
added a stub driver using Genode's 'Nic_session' interface to make the new
facility available to Linux.
:Audio output:
We adapted the ALSA stub driver to the changes of the 'Audio_out_session'
interface, using the new channel synchronization and flush functions.
Thereby, we optimized the stub driver to keep latency and seek times of
Linux userland applications reasonably low.
:Removed ROM file driver:
With the addition of the 'Block_session' stub driver, the original ROM file
driver is no longer required. So we removed the stub. For using ROM files as
disk images for Linux, there is the 'rom_loopdev' server, which provides a
block session that operates on a ROM file.
:Asynchronous block interface:
To improve performance, we changed the block stub driver to facilitate the
asynchronous mode of operation as provided by the 'Block_session' interface.
This way, multiple block requests can be issued at once, thereby shadowing
the round trip times for individual requests.
Protocol stacks and libraries
#############################
Gallium3D / Intel GEM
=====================
We improved the cache handling of our DRM emulation code (implementing
'drm_clflush_pages') and our EGL driver, thereby fixing caching
artifacts on i945 GPUs. Furthermore, we added a temporary work-around
for the currently dysfunctional sequence-number tracking with i945 GPUs.
On this chipset, issuing the 'MI_STORE_DWORD_INDEX' GPU command used
for tracking sequence numbers apparently halts the processing the command
stream. This condition is normally handled by an interrupt. However,
we have not enabled interrupts yet.
To prepare the future support for more Gallium drivers than i915, we
implemented a driver-selection facility in the EGL driver. The code
scans the PCI bus for a supported GPU and returns the name of the
corresponding driver library. If no driver library could be found,
the EGL driver falls back to softpipe rendering.
lwIP
====
We revised our port of the lwIP TCP/IP stack, and thereby improved its
stability and performance.
* The lwIP library is now built as shared object, following the convention
for libraries contained in the 'libports' repository.
* By default (when using the 'libc_lwip_nic_dhcp' library), lwIP will
issue a DHCP request at startup. If this request times out, the loopback
device is set as default.
* If there is no 'Nic' service available, the lwIP stack will fall back to
the loopback device.
* We increased the default number of PCBs in lwIP to 64.
* We removed a corner case of the timed semaphore that could occur
when a timeout was triggered at the same time ,'up' was called.
In this case, the semaphore was unblocked but the timeout condition
was not reflected at the caller of 'down'. However, the lwIP code
relies on detecting those timeouts.
Qt4
====
We implemented a custom *nitpicker plugin widget*, which allows for the
seamless integration of arbitrary nitpicker clients into a Qt4 application.
The primary use case is the browser plugin mechanism presented at
the Live CD. In principle, the 'QNitpickerViewWidget' allows for creating
mash-up allocations consisting of multiple native Genode programs. As shown
by the browser plugin demo, a Qt4 application can even integrate other
programs that run isolated from the Qt4 application, and thereby depend on
on a significantly less complex trusted computing base than the Qt4
application itself.
[image nitpicker_plugin]
The image above illustrates the use of the 'QNitpickerViewWidget' in the
scenario presented on the Live CD. The browser obtains the Nitpicker view to be
embedded into the website from the loader service, which virtualizes the
Nitpicker session interface for the loaded plugin subsystem. The browser then
tells the loader about where to present the plugin view on screen. But it has
neither control over the plugin's execution nor can it observe any user
interaction with the plugin.
New Gems repository with HTTP-based block server
################################################
To give the web-browser demo of our Live CD a special twist, and to show off
the possibilities of a real multi-server OS, we decided to implement the
somewhat crazy idea of letting a Linux OS run on a disk image fetched at
runtime from a web server. This way, the Linux OS would start right away and
disk blocks would be streamed over the network as needed. Implementing this
idea was especially attractive because such a feature would be extremely hard
to implement on a classical OS but is a breeze to realize on Genode where all
device drivers and protocol stacks are running as distinct user-level
components. The following figure illustrates the idea:
[image http_block]
The block stub driver of the Linux kernel gets connected to a special block
driver called 'http_block', which does not access a real block device but
rather uses TCP/IP and HTTP to fetch disk blocks from a web server.
Because the 'http_block' server is both user of high-level functionality (the
lwIP stack) and provider of a low-level interface ('Block_session'), the
program does not fit well into one of the existing source-code repositories.
The 'os' repository, which is normally hosting servers for low-level interfaces
is the wrong place for 'http_block' because this program would make the 'os'
repository depend on the higher-level 'libports' repository where the 'lwip'
stack is located. On the other hand, placing 'http_block' into the 'libports'
repository is also wrong because the program is not a ported library. It merely
uses libraries provided by 'libports'. In the future, we expect that native
Genode components that use both low-level and high-level repositories will
become rather the norm than an exception. Therefore, we introduced a new
repository called 'gems' for hosting such programs.
Tools
#####
Automated coding-style checker
==============================
As Genode's code base grows and new developers start to get involved,
we noticed recurring questions regarding coding style. There is a
[https://genode.org/documentation/developer-resources/coding_style - document]
describing our coding style but for people just starting to get involved,
adhering all the rules can become tedious. However, we stress the importance
of a consistent coding style for the project. Not only does a consistent style
make the framework more approachable for users, but it also eases the work
of all regular developers, who can feel right at home at any part of
the code.
To avoid wasting precious developer time with coding-style fixes, we
have created a tool for the automated checking and (if possible) fixing
the adherence of source code to Genode's coding style. The tool is
located at 'tool/beautify'. It takes a source file as argument and
reports coding-style violations. The checks are fairly elaborative:
* Placement of braces and parenthesis
* Indentation and alignment, trailing spaces
* Vertical spacing (e.g., between member functions, above comments)
* Naming of member variables and functions (e.g., private members start with '_')
* Use of upper and lower case
* Presence of a file header with the mandatory fields
* Policy for function-header comments (comment at declaration, not
at implementation)
* Style of single-line comments, function-header comments, multi-line comments
The user of 'beautify' may opt to let the tool fix most of the violations
automatically by specifying the command line arguments '-fix' and '-write'.
With only the '-fix' argument, the tool will output the fixed version of
the code via stdout. By specifying the '-write' argument, the changes will
be written back to the original file. In any case, we strongly recommend
to manually inspect all changes made by the tool.
Under the hood, the tool consists of two parts. A custom C++ parser called
'parse_cxx' reads the source code and converts it to a syntax tree. In the
syntax tree, all formating information such as whitespaces are preserved.
The C++ parser is a separate command-line tool, which we also use for
other purposes (e.g., generating the API documentation at the website).
The actual 'beautify' tool calls 'parse_cxx', and applies its checks and
fixes to the output of 'parse_cxx'. For this reason, both tools have to
reside in the same directory.
Platform-specific changes
#########################
OKL4
====
:Added support for shared interrupts:
The Genode Live CD operates on a large number of devices that trigger
interrupts (USB, keyboard, mouse, ATAPI, timer, network). On most
platforms, the chances are extremely high that some of them use
the same IRQ line. Therefore, we enhanced core's IRQ service to
allow multiple clients to request the same IRQ. If the interrupt occurs,
all clients referring to this interrupt are notified. The interrupt
gets cleared after all of those clients responded. Even though, we regard
PIC interrupts as a legacy, the support of shared interrupts enables
us to use OKL4 with such complex usage scenarios.
:Revised page-fault handling:
If a page fault occurs, the OKL4 kernel delivers a message to the page-fault
handler. The message contains the page-fault address and type as well as the
space ID where the fault happened. However, the identity of the faulting
thread is not delivered. Instead, the sender ID of the page fault message
contains the KTCB index of the faulting thread, which is only meaningful
within the kernel. This KTCB index is used as a reply token for answering the
page fault message. We wondered about why OKL4 choose to deliver the KTCB
index rather then the global thread ID as done for plain IPC messages. The
only reasonable answer is that by using the KTCB index directly in OKL4's
page-fault protocol, one lookup from the userland-defined thread ID to the
KTCB index can be avoided. However, this comes at the cost of losing the
identity of the faulting thread. We used to take the space ID as a key for
the fault context within core. However, with Genode's user-level page-fault
mechanism, this simplification does not suffice anymore. We have to know the
faulting thread as a page fault may not be answered immediately but at a
later time. During that time, the page-fault state has to be stored at core's
representation of the faulting thread. Our solution is reverting OKL4's
page-fault protocol to operate with global thread IDs only and to never make
kernel-internal KTCB indices visible at the user land. You can find the patch
for the OKL4 kernel at 'base-okl4/patches/reply_tid.patch'.
:Reboot via kernel debugger:
We fixed the reboot code of OKL4's kernel debugger to improve our work
flow. The patch can be found at 'base-okl4/patches/kdb_reboot.patch'.
:Relieved conflict with libc 'limits.h':
For some reason, the OKL4 kernel bindings provide definitions
normally found in libc headers. This circumstance ultimately leads
to trouble when combining OKL4 with a real C runtime. We have
relieved the problem with the patch 'base-okl4/patches/char_bit.patch'.
:Exception handling:
We added a diagnostic message to core that reports about exceptions
such as division by zero.
Pistachio
=========
Our revised syscall bindings for supporting position-independent code
on L4ka::Pistachio have been integrated into the mainline development
of the kernel. Therefore, the patch is not needed anymore when using
a kernel revision newer than 'r791:0d25c1f65a3a'.
Linux
=====
On Linux, we let the kernel manage all virtual address spaces for us,
except for the thread-context area. Because the kernel does not know
about the special meaning of the thread-context area, it may choose to
use this part of the virtual address space as target for 'mmap'. This
may lead to memory corruption. Fortunately, there is a way to tell the
kernel about virtual address regions that should be reserved. The
trick is to pre-populate the said region with anonymous memory using
the 'mmap' arguments 'MAP_PRIVATE', 'MAP_FIXED', 'MAP_ANONYMOUS', and
'PROT_NONE'. The kernel will still accept a fixed-address mapping
within such a reserved region (overmap) but won't consider using the
region by itself. The reservation must be done at the startup of each
process and each time when detaching a dataspace from the thread
context area. For the process startup, we use the hook function
'main_thread_bootstrap' in 'src/platform/_main_helper.h'. For reverting
detached dataspaces to a reserved region within the context area, we
added as special case to 'src/base/env/rm_session_mmap.cc'.
For hybrid programs (Genode processes that link against native
shared libraries of the Linux system), which are loaded by the dynamic
linker of Linux, we must further prevent the dynamic linker from
populating the thread-context area. This is achieved by adding a
special program segment at the linking stage of all elf binaries.

876
doc/release_notes/11-02.txt Normal file
View File

@@ -0,0 +1,876 @@
===============================================
Release notes for the Genode OS Framework 11.02
===============================================
Genode Labs
One year ago, the release 10.02 was our break-through with regard to the support
of multiple kernels as base platform for Genode. With the added support for
the NOVA hypervisor and the Codezero kernel, Genode applications could be executed
on 6 different kernels. With the current release, we take our commitment to
kernel platform support even further. With the added support for the Fiasco.OC
kernel, we make Genode available on one of the most feature-rich modern microkernels.
Additionally, we entered the realms of kernel design with our new platform support
for the Xilinx MicroBlaze architecture. This platform support comes in the shape
of a custom kernel specifically targeted to the MicroBlaze CPU architecture.
Furthermore, we updated our support for the NOVA Hypervisor to the bleeding-edge
version 0.3, which has been released earlier this month.
With the current support for 8 different kernel platforms (L4/Fiasco, Linux,
L4ka::Pistachio, OKL4, NOVA, Codezero, Fiasco.OC, and native MicroBlaze), testing
and integrating application scenarios across all platforms becomes increasingly
challenging. Therefore, we introduce a new framework for automating such tasks.
Thanks to the tight integration of the automation tool with Genode's build system,
going back and forth between different kernels becomes an almost seamless
experience.
Functionality-wise, the release carries on our vision to create a highly secure
yet easy to use general-purpose operating system. Because the Genode framework
is developed on Linux using the wonderful GNU tools, we consider the
availability of the GNU user land on Genode as crucial for using the system by
ourself. This motivation drives the creation of a custom execution environment
for GNU software on top of Genode. With the current release, we are proud to
present the first pieces of this execution environment. Even though not fully
functional yet, it clearly shows the direction of where we are heading.
Support for Fiasco.OC
#####################
The OC in the name of the Fiasco.OC kernel stands for "object capability", hinting
at the most significant feature that sets current-generation microkernels such as
NOVA, seL4, and Fiasco.OC apart from their predecessors. Whereas previous L4 kernels
succeeded in protecting subsystems from each other, the new generation of kernels
is geared towards strict security policies. Traditionally, two protection domains
were able to communicate with each other if they both agreed. Communication partners
were typically globally known via their respective thread/task IDs. Obviously, this
policy is not able to guarantee the separation of subsystems. If two subsystems
conspire, they could always share information. Object-capability-based kernels
are taking the separation much further by prohibiting any communication between
protection domains by default. Two protection domains can communicate only if
a common acquaintance of both agrees. This default-deny policy facilitates the
creation of least-privilege security policies. From the ground up, Genode has
been designed as a capability-based system which is naturally capable of leveraging
kernel-based object-capability support if present. After NOVA, Fiasc.OC is the
second of Genode's base platforms that provides this feature.
Apart from being a capability-based kernel, Fiasco.OC has a number of compelling
features such as thorough support for ARM platforms and the x86 32/64 bit
architectures. It supports SMP, hardware virtualization, and provides special
optimizations for running paravirtualized operating systems.
Technically, Fiasco.OC is the successor of the L4/Fiasco kernel developed by
the OS group of the TU-Dresden. However, the kernel interface of Fiasco.OC has
not much in common with L4/Fiasco. Some heritages are still there (e.g., IPC
timeouts) but the kernel API has evolved to a fully object-oriented model.
:Thanks:
We are indebted to the main developer of Fiasco.OC Alexander Warg for being
very reponsive to our inquiries while doing the porting work. Thanks to his
support, the adaptation of Genode to this kernel has been an almost smooth
ride.
Prerequisites
=============
You need GNU C & C++ Compilers, GNU Binutils, GNU Make, and Perl to use the
Fiasco.OC build system. On Debian/Ubuntu systems, you have to install the
following packages:
! apt-get install make gawk g++ binutils pkg-config subversion
Moreover, you need to download and install the tool-chain used by Genode. Have
a look at this page:
:[https://genode.org/download/tool-chain]:
Genode tool-chain
Downloading and building Fiasco.OC
==================================
Checkout the Fiasco.OC sources and tool-chain to an appropriated directory:
! export REPOMGR_SVN_REV=27
! svn cat https://svn.tudos.org/repos/oc/tudos/trunk/repomgr |\
! perl - init https://svn.tudos.org/repos/oc/tudos fiasco l4re
Building the kernel
~~~~~~~~~~~~~~~~~~~
Create the build directory for the kernel:
! cd <path_to_fiasco_src_dir>/src/kernel/fiasco
! make BUILDDIR=<path_to_kernel_build_dir>
Go to the build directory, configure the kernel:
! cd mybuild
! make config
This will launch the configuration menu. Here you can configure your kernel.
The default config is just fine to test the Genode port. It will build a
uniprocessor IA32 kernel with debugging features enabled. You can exit the menu and
save the configuration by simply typing 'x'.
Now, build Fiasco.OC by invoking:
! make
Building necessary tools
~~~~~~~~~~~~~~~~~~~~~~~~
To practically use Fiasco.OC, you need in addition to the kernel a tool to
bootstrap it, and the initial pager of the system, namely 'sigma0'. Both tools
can be found in the L4 runtime environment's base directory. Outgoing from
the directory where you checked out the sources, you have to change to the
following directory:
! cd <path_to_fiasco_src_dir>/src/l4
Create another build directory:
! make B=<path_to_l4re_build_dir>
Again, you might want to tweak the configuration:
! make O=<path_to_l4re_build_dir> config
Finally, build the tools:
! make O=<path_to_l4re_build_dir>
Building the Fiasco.OC version of Genode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The Fiasco.OC version of Genode is available at the Genode public subversion repository:
:https://genode.org/download/subversion-repository:
Information about accessing the Genode public subversion repository
Go to a directory where you want the Genode/Fiasco.OC build directory to remain. Use
the helper script in the 'tool/builddir' directory of the Genode source tree to
create the initial build environment. You need to state the absolute path to the
build directory of the L4 runtime environment as 'L4_DIR', as it contains the kernel
bindings needed by the Genode port.
! <path_to_genode_src_dir>/tool/builddir/create_builddir foc_x86_32 \
! L4_DIR=<path_to_l4re_build_dir> \
! GENODE_DIR=<path_to_genode_src_dir> \
! BUILD_DIR=<path_to_genode_build_dir>
Now, go to the newly created build directory and type make.
! cd <path_to_genode_build_dir>
! make
Booting Genode on top of Fiasco.OC
==================================
Example GRUB configuration entry:
! timeout 0
! default 0
!
! title Genode on Fiasco.OC
! kernel /bootstrap -modaddr=0x01100000
! module /fiasco -serial_esc
! module /sigma0
! module /core
! module /init
! module /config
! module /pci_drv
! module /vesa_drv
! module /ps2_drv
! module /timer
! module /nitpicker
! module /launchpad
! module /liquid_fb
! module /scout
! module /testnit
! module /nitlog
For an example of a matching Genode 'config' file, please take a look
at 'os/config/demo'.
The Genode binaries are located in '<path_to_genode_build_dir>/bin',
the 'fiasco' kernel in '<path_to_kernel_build_dir>'. Assuming you compiled
for x86/586 (the default), you can find the 'bootstrap' binary in
'bin/x86_586' and 'sigma0' in 'bin/x86_586/l4f' within the
'<path_to_l4re_build_dir>' directory.
Current state
=============
The adaptation of Genode to Fiasco.OC covers most parts of the Genode API
including advanced semantics such as cancelable locks and support for
real-time priorities. So far, it has been tested on the x86 architecture.
Because 'base-foc' does not contain x86-specific code, we expect no major
roadblocks for running Genode on Fiasco.OC on ARM. However, we have not
exercised tests in this regard.
As of today, there exist the following limitations of the Fiasco.OC support:
* The dynamic linker is not yet adapted to Fiasco.OC. Special care must
be taken for handling the parent capability for dynamically loaded
programs. We have already covered this issue for the NOVA version but
the adaptation to Fiasco.OC remains yet to be done.
* The destruction of sub systems is not yet fully stable. Because Genode
forms a more dynamic workload than the original userland accompanied with
the kernel, the usage pattern of the kernel API triggers different
effects. We are working with the Fiasco.OC developers to remedy this
issue.
* The signalling framework is not yet supported. A design exist but it is
not implemented yet.
We believe however that none of these limitations are a significant hurdle for
starting to use Genode with this kernel. Please expect this issues to be
resolved with the upcoming Genode release.
Technical details about 'base-foc'
==================================
The following technical bits are worth noting when exploring the use of
Genode with the 'base-foc' platform.
* The timer implementation uses a one thread-per-client mode of operation.
We use IPC timeouts as time source. Hence, the timer driver is hardware
independent and should work out of the box on all hardware platforms
supported by Fiasco.OC.
* Each 'Server_object' of Genode corresponds to a so-called IPC gate,
which is the Fiasco.OC kernel object used for capability invocation.
Therefore, protection and object integrity is provided at the fine
granularity of single 'Server_objects'. This is in line with our
support for NOVA's implementation of capability-based security.
* In contrast to the lock implementation that we used with the original
L4/Fiasco kernel, the 'base-foc' lock is a fully-featured Genode lock
with support for lock cancellation and blocking. For blocking and
waking up lock applicants, we use Fiasco.OC's IRQ objects.
* The allocator used for managing process-local capability selectors
does not yet support the reuse of capability selectors.
Further Information
===================
:genode/tool/builddir/README:
Reference manual for the 'create_builddir' script
:[https://os.inf.tu-dresden.de/fiasco]:
Official website for the Fiasco.OC microkernel.
Noux - an execution environment for the GNU userland
####################################################
Even though Genode is currently mainly geared to the classical special-purpose
application domains for microkernel-based systems, the main property that sets
Genode apart from traditional systems is the thorough support for dynamic
workloads and the powerful mechanisms for handling hardware resources and
security policies in highly dynamic setting. We are convinced that Genode's
architecture scales far beyond static special-purpose domains and believe in
the feasibility of Genode as a solid foundation for a fully-fledged general
purpose operating system. Internally at Genode Labs, we set up the ultimate
goal to switch from Linux to Genode for our day-to-day work. We identified
several functionalities that we could not live without and systematically try
to bring those features to Genode. Of course, the most fundamental programs
are the tools needed to develop and build Genode. Currently we are developing
on Linux and enjoy using the GNU userland.
Consequently, we require a solution for using this rich tool set on Genode.
The straight-forward way for making these tools available on Genode would be
running them within a virtualized Linux instance (e.g., using OKLinux on OKL4).
However, this approach would defeat our actual goal to create a highly secure
yet easy to use working environment because adding Linux to the picture would
involve administering the virtualized Linux system. We would prefer a native
solution that makes the overall system less, not more, complicated. This way
the idea for a native execution environment for the GNU userland on Genode
was born. The implementation is called Noux and the first bits of code are
featured in the 'ports' repository. Noux consists of two parts, a build
environment for compiling GNU programs such that they can be run as Genode
processes and an execution environment that provides the classical UNIX
functionality to these programs.
Noux build environment
======================
From our experience, porting existing UNIX applications to a non-UNIX system
tends to be a task of manual and time-consuming labour. One has to loosely
understand the build system and the relationship of the involved source codes,
implement dummy functions for unresolved references, and develop custom glue
code that interfaces the ported application to the actual system. Taking the
shortcut of changing the original code has to be avoided at any cost because
this produces recurring costs in the future when updating the application. In
short, this long-winding process does not scale. For porting a tool set such as
the GNU userland consisting of far more than a three-digit number of individual
programs, this manual approach becomes unfeasible. Therefore, we have created
a build environment that facilitates the use of the original procedure of
invoking './configure && make'. The challenge is to supply configure with
the right arguments and environment variables ('CFLAGS' and the like) such that
the package is configured against the Genode environment. The following
considerations must be taken:
* Configure must not detect any global headers (e.g., '/usr/include/')
or libraries (e.g., '/usr/lib/'). This can be achieved by the '-nostdinc' and
'-nostdlib' options
* Configure has to use the same include-search paths as used for compiling
normal libc-using Genode programs
* Configure must use the Genode tool chain
* The final linking stage must use the Genode linker script, the Genode
base libraries, and other Genode-specific linker arguments.
Thanks to the power of the GNU build system, all this can be achieved by
supplying arguments to './configure' and indirectly to the 'make' process via
environment variables. The new Noux build environment takes care of these
precautions. It comes in the form of the 'ports/mk/noux.mk' file which enables
the seamless integration of GNU packages with the Genode build system. To
compile a GNU package, the manual steps needed are reduced to the creation of a
'target.mk' file representing the package. This 'target.mk' defines the name
of the package (by default, the basename of the 'target.mk' enclosing directory
is assumed) and the location of the source package. With this approach, we
managed to build 'coreutils' (over 100 small UNIX utilities such as 'ls', 'cp',
'sort'), 'binutils' (GNU linker, assembler, object-file tools), 'findutils'
('find', 'xargs'), 'bash', 'dash', GNU make, and finally the GNU compiler
collection including 'g++'. The resulting binaries are ready to be executed as
native Genode processes. However, without the right environment that presents
the program the needed UNIX functionality, those programs won't do much.
This leads us to the Noux execution environment.
Noux execution environment
==========================
The Noux execution environment plays the role of a UNIX kernel for programs
built via the Noux build environment. In contrast to a real kernel, the Noux
environment is a plain Genode user-level process that plays the role of being
the parent of one or multiple Noux processes. In addition of providing the
'Genode::Parent' interface, Noux also provides a locally implemented service called
'Noux::Session' that offers UNIX-like system-calls via an RPC interface. Each
hosted program is linked against a special Noux libc plugin that catches all
libc calls that would normally result in a system call. It then transparently
forwards this function call to the 'Noux::Session' interface.
Currently the Noux execution environment implements the following
system calls: 'getcwd', 'write', 'stat', 'fstat', 'fcntl', 'open',
'close', 'dirent', 'fchdir', 'read', and 'execve'.
The execution environment submits arguments (argc, argv, environment) to the
hosted program, manages its current working directory and receives its exit
code. File operations are targeted to a custom VFS infrastructure, which
principally allows a flexible configuration of the virtual file system visible
to the hosted programs. At the current stage, Noux supports mounting plain tar
archives obtained from core's ROM service as read-only file system. On startup,
the Noux environment starts one process (the init process) and connects the
file descriptor 1 (stdout) to Genode's LOG service.
State of the implementation
~~~~~~~~~~~~~~~~~~~~~~~~~~~
The infrastructure implemented so far already allows the execution of many simple
UNIX tools such as 'ls -lRa', 'echo', 'seq', 'find'. The 'execve' system call
is implemented such that a new process is started that inherits the file
descriptors and the PID of the calling process. This allows using the exec
functionality of the 'bash' shell. However, because 'fork' is not implemented
yet, there is currently no way to start multiple programs hosted in a single
Noux execution environment.
As of today, the Noux environment is not considered to be usable for practical
purposes. However, it clearly shows the feasibility of the path we are walking.
With the foundation laid, we are looking forward to expanding Noux to a capable
solution for running our beloved GNU userland tools on Genode.
Vision
~~~~~~
The most significant intermediate result of pursuing the development of Noux is
the realization that such an environment is not exceedingly complex. Because of
the combination with Genode, we only need to provide a comfortable runtime as
expected by user processes but we can leave much of intricate parts of UNIX out
of the picture. For example, because we handle device drivers on Genode, we do
not need to consider device-user interaction in Noux. As another example,
because the problem of bootstrapping the OS is already solved by Genode, there
is no need to run an 'init' process within Noux. Our vision foresees that Noux
runtimes are to be created on demand for individual tasks such as editing a
file (starting a custom Noux instance containing only the file to edit and the
text editor), compiling source code (starting a custom Noux instance with only
the source code and the build tools). Because Noux is so simple, we expect the
runtime overhead of starting a Noux instance to be not more than the time
needed to spawn a shell in a normal UNIX-like system.
Test drive
~~~~~~~~~~
To give Noux a spin, we recommend using Linux as base platform as this is
the platform we use for developing it. First, you will need to download the
source code of the GNU packages. From within the 'ports' repository,
use the following command:
! make prepare PKG=coreutils
This command will download the source code of the GNU coreutils. You may
also like to give the other packages a try. To see what is available,
just call 'make' without any argument.
Create a build directory (e.g., using tool/builddir/create_builddir).
Change to the build directory and issue the command
! make run/noux
This command will execute the run script provided at 'ports/run/noux.run'.
First it builds core, init, and coreutils. Then it creates a tar archive
containing the installed coreutils. Finally, it starts the Noux environment on
Genode. Noux then mounts the TAR archive as file system and executes 'ls -laR',
showing the directory tree.
Approaching platform support for Xilinx MicroBlaze
##################################################
With the release 11.02, we are excited to include the first version of our
custom platform support for the Xilinx MicroBlaze CPU architecture. MicroBlaze
is a so-called softcore CPU, which is commonly used as part of FPGA-based
System-on-Chip designs. At Genode Labs, we are regularly using this IP core,
in particular for our Genode FPGA Graphics Project, which is a GUI software stack
and a set of IP cores for implementing fully-fledged windowed GUIs on FPGAs:
:Website of the Genode FPGA Graphics Project:
[https://genode-labs.com/products/fpga-graphics]
Ever since we first released the Genode FPGA project, we envisioned to combine
it with the Genode OS Framework. In Spring 2010, Martin Stein joined our team
at Genode Labs and accepted the challenge to bring the Genode OS Framework to
the realms of FPGA-based SoCs. Technically, this implies porting the framework
to the MicroBlaze CPU architecture. In contrast to most softcore CPUs such as
the popular Lattice Mico32, the MicroBlaze features a MMU, which is a fundamental
requirement for implementing a microkernel-based system. Architecturally-wise
MicroBlaze is a RISC CPU similar to MIPS. Many system parameters of the CPU
(caches, certain arithmetic and shift instructions) can be parametrized at
synthesizing time of the SoC. We found that the relatively simple architecture
of this CPU provides a perfect playground for pursuing some of our ideas about
kernel design that go beyond the scope of current microkernels. So instead of
adding MicroBlaze support into one of the existing microkernels already
supported by Genode, we went for a new kernel design. Deviating from the typical
microkernel, which is a self-sufficient program running in kernel mode that
executes user-level processes on top, our design regards the kernel as a part of
Genode's core. It is not a separate program but a library that implements the
glue between user-level core and the raw CPU. Specifically, it provides the
entrypoint for hardware exceptions, a thread scheduler, an IPC mechanism, and
functions to manipulate virtual address spaces (loading and flushing entries
from the CPU's software-loaded TLB). It does not manage any physical memory
resources or the relationship between processes. This is the job of core.
From the kernel-developer's point of view, the kernel part can be summarized as
follows:
* The kernel provides user-level threads that are scheduled in a round-robin
fashion.
* Threads can communicate via synchronous IPC.
* There is a mechanism for blocking and waking up threads. This mechanism
can be used by Genode to implement locking as well as asynchronous
inter-process communication.
* There is a single kernel thread, which never blocks in the kernel code paths.
So the kernel acts as a state machine. Naturally, there is no concurrency in the
execution paths traversed in kernel mode, vastly simplifying these code parts.
However, all code paths are extremely short and bounded with regard to
execution time. Hence, we expect the interference with interrupt latencies
to be low.
* The IPC operation transfers payload between UTCBs only. Each thread has a
so-called user-level thread control block which is mapped transparently by
the kernel. Because of this mapping, user-level page faults cannot occur
during IPC transfers.
* There is no mapping database. Virtual address spaces are manipulated by
loading and flushing physical TLB entries. There is no caching of mappings
done in the kernel. All higher-level information about the interrelationship
of memory and processes is managed by the user-level core.
* Core runs in user mode, mapped 1-to-1 from the physical address space
except for its virtual thread-context area.
* The kernel paths are executed in physical address space (MicroBlaze).
Because both kernel code and user-level core code are observing the same
address-space layout, both worlds appear to run within a single address
space.
* User processes can use the entire virtual address space (4G) except for a
helper page for invoking syscalls and a page containing atomic operations.
There is no reservation used for the kernel.
* The MicroBlaze architecture lacks an atomic compare-and-swap instruction. On
user-level, this functionality is emulated via delayed preemption. A kernel-
provided page holds the sequence of operations to be executed atomically and
prevents (actually delays) the preemption of a thread that is currently
executing instructions at that page.
* The MicroBlaze MMU supports several different page sizes (1K up to 16MB).
Genode fully supports this feature for page sizes >= 4K. This way, the TLB
footprint can be minimized by choosing sensible alignments of memory
objects.
Current state
=============
The MicroBlaze platform support resides in the 'base-mb' repository. At the
current stage, core is able to successfully start multiple nested instances of
the init process. Most of the critical kernel functionality is working. This
includes inter-process communication, address-space creation, multi-threading,
thread synchronization, page-fault handling, and TLB eviction.
This simple scenario already illustrates the vast advantage of
using different page sizes supported by the MicroBlaze CPU. If using
4KB pages only, a scenario with three nested init processes produces more than
300.000 page faults. There is an extremely high pressure on the TLB, which
only contains 64 entries. Those entries are constantly evicted so that
threshing effects are likely to occur. By making use of flexible page
sizes (4K, 16K, 64K, 256K, 1M, 4M, 16M), the number of page faults gets
slashed to only 1.800, speeding up the boot time by factor 10.
Currently, there is no restriction of IPC communication rights. Threads are
addressed using their global thread IDs (in fact, using their respective
indices in the KTCB array). For the future, we are planning to add
capabilty-based delegation of communication rights.
Building and using Genode on MicroBlaze
=======================================
For building Genode for the MicroBlaze platform, you need the MicroBlaze
tool chain as it comes with the Xilinx EDK. The tool chain is typically
prefixed with 'mb-'. Please make sure that the tool chain's 'bin/' directory
is included in your 'PATH' environment variable.
For building and starting Genode on MicroBlaze, you first need to create
a build directory using the build-directory creation tool:
! tool/builddir/create_builddir microblaze \
! BUILD_DIR=</path/to/build/dir> \
! GENODE_DIR=</path/to/genode/dir>
The 'base-mb' repository comes with support for Genode's run tool. In order to
use it, you will first need to declare the location of your qemu binary using
the 'QEMU=/path/to/qemu' variable in the '<build-dir>/etc/microblaze.conf'
file. Then you will be able to start an example scenario by issuing the
following command from within your build directory:
! make run/nested_init
Thereby, the 'run' tool will attempt to start core using the microblaze version
of qemu.
You can also find a simple hello-world example at 'base-mb/src/test/hello'.
The corresponding run script is located at 'base-mb/run/hello.run'. You can
execute it via 'make run/hello' from the build directory.
Note that currently, all boot modules are linked against the core binary.
To change the boot modules, the file 'base-mb/src/core/boot_modules.s' must
be modified.
For reference, we are using the following tools:
* mb-g++ (GCC) 4.1.1 20060524 (Xilinx 11.2 Build EDK_LS2.2
20 Apr 2009 Xilinx 11.2 Build EDK_LS2.2 23 Apr 2009)
* GNU ld version 2.16 Xilinx 11.2 Build EDK_LS2.2 23 Apr 2009
* GNU assembler 2.16 Xilinx 11.2 Build EDK_LS2.2 23 Apr 2009
* QEMU emulator version 0.14.50, Copyright (c) 2003-2008 Fabrice Bellard
Petalogix linux reference design targeting Xilinx Spartan 3ADSP-1800 boards.
Supporting the NOVA hypervisor version 0.3
##########################################
NOVA is a so called microhypervisor - a modern capability-based microkernel
with special support for hardware-based virtualization and IOMMUs. Since we
incorporated the initial support for the NOVA hypervisor in Genode one year
ago, this kernel underwent multiple revisions. The latest version was released
earlier this month. To our delight, much of the features that we missed from
the initial release had been implemented during the course of the last year. We
are especially happy about the fully functional 'revoke' system call and the
support for remote kernel-object creation.
With the Genode release 11.02, we officially support the latest NOVA version.
The update of Genode to the new version required two steps. First, because many
details of the kernel interface were changed between version 0.1 and version
0.3, we had to revisit our syscall bindings and adapting our code to changed
kernel semantics. Second, we filled our 'base-nova' code related to object
destruction and unmapping with life to benefit from NOVA's 'revoke' system
call. Consequently, we are now able to run the complete Genode software stack
including the dynamic linker on NOVA.
Note that for using Genode on NOVA, you will need to apply a small patch to the
NOVA source code. This patch enables the re-use of user-level thread control
blocks in the kernel. The patch can be found at 'base-nova/patches/utcb.patch'.
When executing NOVA on qemu, please specify the '-cpu coreduo' argument to the
qemu command line. When using Genode 'run' tool, you may assign this argument
to the 'QEMU_OPT' variable in '<build-dir>/etc/build.conf'.
:Thanks:
We are grateful for the ongoing very pleasant collaboration with Udo Steinberg
who is the driving force behind NOVA. Thanks for the ultra-fast responses to our
questions and for considering our suggestions regarding the feature set of
NOVA's kernel interface!
Base framework
##############
Upgrading existing sessions
===========================
Genode enables a client of a service to lend parts of its own resources to
the service when opening a session. This way, servers do not need to allocate
own resources on behalf of their clients and become inherently robust against
resource-exhaustion-based denial-of-service attacks.
However, there are cases when the client can not decide about the amount of
resources to lend at session-creation time. In such cases, we used to devise an
overly generous client policy. Now, we have added a new 'upgrade' function to
the 'Parent' and 'Root' interfaces that enables a client to upgrade the
resources of an existing session.
For the 'env()->rm_session()' and 'env()->ram_session()' of processes using
the Genode 'env' library, we implemented a transparent quota upgrade that kicks in
in the event of an exceeded metadata backing store.
Comprehensive accounting of core resources
==========================================
We changed all services of core to limit their respective resource usage
specifically for each individual session. For example, the number of dataspaces
that can be handled by a particular region-manager (RM) session depends on the
resource donation attached to the session. To implement this accounting scheme
throughout core, we added a generic 'Allocator_guard' utility to
'base/include/'. We recommend using this utility when implementing resource
multiplexers, in particular multi-level services. Thanks to this change in
core, the need for a slack memory reservation in core has vanished.
Various changes
===============
The remaining parts of the base API underwent no fundamental revision. The
changes are summarized as follows.
:C++ Support:
We removed 'libgcc' from our C++ support library ('cxx') and link
it to each individual final target and shared library instead. This change alleviates
the need to abuse the 'KEEP_SYMBOLS' mechanism that we used in 'cxx' to
keep libc-dependencies of GCC's support libraries local to the 'cxx'
library. Besides the benefit of reducing heuristics, this change improves
the compatibility with recent cross-compiling tool chains.
Furthermore, we added 'realloc' to the local libc support of the 'cxx'
library because recent ARM tool chains tend to use this function.
:Argument handling for 'main()':
We added a hook to the startup code to enable the implementation of
custom facilities for passing arguments to the main function. The
hook uses the global variables 'genode_argc' and 'genode_argv'.
:Child-exit policy hook:
We enhanced the 'Child_policy' with a new policy interface that allows
a simplified implementation of policies related to program termination.
:Changed API of 'Range_allocator':
We changed the return value of 'alloc_addr' to distinguish different error
conditions. Note that the boolean meaning of the return value is inverted.
Please check your uses of 'alloc_addr'!
Operating-system services and libraries
#######################################
C Runtime
=========
In conjunction with our work on Noux, we improved Genode's C runtime at many
places. First, we added libstdtime and some previously missing bits of libgdtoa
to the libc. These additions largely alleviate the need for dummy stubs, in
particular time-related functions. Second, we added the following functions to
our libc plugin interface: 'dup2', 'fchdir', 'fcntl', 'fstat', 'stat', and
'write'. This enables the creation of advanced libc plugins simulating a whole
file system as done with Noux. Still, there are a number of dummy stubs found
at 'libc/src/lib/libc/dummy.cc'. However, those stubs are now all defined as
weak symbols such that they can be overridden by libc plugins. Finally, we have
replaced the original 'exit' implementation that comes with the libc with a
Genode-specific version. The new version reports the exit code of the
application to the parent process via an 'Parent::exit()' RPC call.
Until now, Genode's libc magically handled output to stdout and stderr by
printing messages via Genode's LOG interface. We have now replaced this
hard-wired interface by an optional libc plugin called 'libc_log'. If present, write
operations to stdout are caught at the libc plugin interface and delegated to
the plugin, which implements the output to the LOG interface. If you have an
application using Genode's libc, you might consider adding the 'libc_log'
library to your 'target.mk' file.
Support for big numbers by the means of libgmp and libmpfr
==========================================================
We have now include both the GNU Multiple Precision Arithmetic Library and
(GMP) and MPFR to the 'ports' repository. This work was specifically motivated
by our port of GCC to Genode as GCC version 4.4.5 requires both libraries.
Because we intend to use those libraries primarily on x86_32, the current port
covers only this architecture. However, expanding the port to
further CPU architectures should be straight-forward if needed.
Furthermore, you can now also find GCC's 'longlong.h' header at
'libports/include/gcc'.
Qt4 updated to version 4.7.1
############################
The current release bumps the supported Qt4 version from 4.6.2 to 4.7.1 and the
Arora web browser (located at the ports repository) from version 0.10.2 to
version 0.11. Of course, we updated our custom additions such as our custom
Nitpicker plugin widget that enables the seamless integration of native
Nitpicker GUI clients into Qt4 applications to work with the new Qt4 version.
Tools
#####
Tool chain update to GCC 4.4.5 and Binutils 2.21
================================================
We upgraded the official Genode tool chain from gcc 4.2.4 to gcc 4.4.5. Please
update your tool chain by downloading the new binary archive (available for x86_32)
or building the tool chain from source using our 'tool/tool_chain' utility.
New support for automated integration and testing
=================================================
With the growing number of supported base platforms, the integration and testing
of Genode application scenarios across all kernels becomes
increasingly challenging. Each kernel has a different boot mechanism and
specific requirements such as the module order of multiboot modules (Fiasco's
bootstrap, Pistachio's sigma0 and kickstart), kernel parameters, or the
invocation of a single-image creation tool (OKL4's elfweaver). To make our
life supporting all those platforms easier, we have created a tool called
'run', which is tightly integrated within Genode's build system. In short 'run'
gathers the intrinsics in the form of a 'run/env' file specific for the
platform used by the current build directory from the respective
'base-<platform>' repository. It then executes a so-called run script, which
contains all steps needed to configure, build, and integrate an application
scenario. For example, a typical run script for building and running a test
case resides in a file called '<any-repository>/run/<run-script-name>.run' and
looks as follows:
! build "core init test/exception"
! create_boot_directory
! install_config {
! <config>
! <parent-provides>
! <!--<service name="ROM"/>-->
! <service name="LOG"/>
! </parent-provides>
! <default-route>
! <any-service> <parent/> </any-service>
! </default-route>
! <start name="test-exception">
! <resource name="RAM" quantum="1M"/>
! </start>
! </config>
! }
! build_boot_image "core init test-exception"
! append qemu_args "-nographic -m 64"
! run_genode_until {.*Exception \(label 0xffb0\) occured.*} 10
First, the build system is instructed to create the targets specified as argument
for the 'build' function. Next, for the integration part, a new boot directory is
created. On most kernel platform, the respective location of the boot directory
is '<build-dir>/var/run/<run-script-name>'. Initially, this directory is empty.
It gets populated with a 'config' file specified as argument of the 'install_config'
command, and by the boot modules specified at the 'build_boot_image' command.
Now that the integration is complete, the scenario is executed via the
'run_genode_until' command. This command takes a regular expression as
argument, which determines the successful termination of the test case. The
second argument is a timeout (is seconds). In the example, the test case will
fail if its output does not match the regular expression within the execution
time of 10 seconds.
The command 'append qemu_args' specifies run-script-specific qemu arguments in
the case that qemu is used to execute the scenario. This is the case for most
kernel platforms (except for Linux where core gets executed directly on the host).
Additional build-directory-specific qemu arguments can be specified in the
'etc/build.conf' file by defining the 'QEMU_OPT' variable. For example, to
prevent KVM being used on Ubuntu Linux, specify:
! QEMU_OPT = -no-kvm
To execute the run script from with build directory, you need to have Expect
installed. Typically, the Linux package is called 'expect'. Simply issue
the following command from within your build directory:
! make run/<run-script>
Note that you will need to have a GRUB 'stage2_eltorito' binary available
at '<genode-dir>/tool/grub' on base platforms that use an ISO image as boot
stategy.
Because the whole chain of actions, building, integrating, executing, and
validating an application scenario is now at the fingertips of issuing a
single command with no kernel-specific considerations needed, it has never
been easier to run the same scenario on a wide range of different kernels.
Please find further instructive examples at 'os/run/'. The 'ldso' run
script executes the test of the dynamic linker. It is completely generic.
The 'demo' run script starts Genode's default demo scenario and shows how
platform-specific considerations (e.g., which device drivers to use) can be
taken into account.
We found that the 'run' tool significantly boosted our productivity not
only for testing purposes but also for accelerating the development-test
cycle during our day-to-day work.
:Technical notes:
The 'run' tool uses Expect as automation tool. Expect is a Tcl interpreter,
which is accompanied by special functionality for automating interactive
command-line applications. Technically, a run script is an Expect script
which gets included by the 'tool/run' script. For the reference of
run-specific functions, please revise the documentation in the 'tool/run'
script. Because each run script is actual Expect source code, it is possible
to use all Tcl and Expect scripting features in a run script.
In particular, a run script may issue shell commands using Tcl's 'exec'
function. This way, even complex integration tasks can be accomplished.
For example, the integration of the Genode Live CD was done via a single
run script.
Build system
============
To facilitate the integration of 3rd-party build systems into the Genode build
process, we added support for pseudo targets that do not require any 'SRC'
declaration. Such 'target.mk' may contain custom rules that will be executed
when the target is revisited by the build system. The bindings are as follows:
! build_3rd_party:
! ...custom commands...
!
! $(TARGET): build_3rd_party
!
! clean_3rd_party:
! ...custom commands...
!
! clean_prg_objects: clean_3rd_party:

1289
doc/release_notes/11-05.txt Normal file
View File

File diff suppressed because it is too large Load Diff

703
doc/release_notes/11-08.txt Normal file
View File

@@ -0,0 +1,703 @@
===============================================
Release notes for the Genode OS Framework 11.08
===============================================
Genode Labs
One of Genode's most distinctive properties is its support for various
different kernels as base platforms. Each of the 8 currently supported kernels
differs with regard to features, security, hardware support, complexity, and
resource management. Even though different applications call for different
kernel properties, through Genode, those properties can be leveraged using a
unified API. The growing number of supported base platforms, however, poses two
challenges, which are the comprehension of the large diversity of tools and
boot concepts, and capturing of the semantic differences of all the kernels.
With the version 11.08, the framework mitigates the former challenge by
introducing a unified way to download, build, and use each of the
kernels with Genode's user-level infrastructure. The new tools empower users of
the framework to instantly change the underlying kernel without the need to know
the peculiarities of the respective kernels. Using microkernels has never been
easier.
The second challenge of translating each kernel's specific behaviour to the
framework's unified API longs for an automated testing infrastructure that
systematically exercises all the various facets of the API on all base
platforms. The new version introduces the tooling support especially designed
for conducting such quality-assurance measures. These tools largely remove the
burden of manual testing while helping us to uphold the stability and quality
of the framework as it grows in terms of functional complexity and number of
base platforms.
Speaking of functional enhancements, the work on version 11.08 was focused
on our block-device infrastructure and ARM support. The block-device-related
work is primarily motivated by our fundamental goal to scale Genode to a
general-purpose computing platform. The additions comprise new drivers for
SD-cards, IDE, SATA, USB storage as well as a new partition server. All those
components provide Genode's generic block interface, which is meant to be used
as back end for file systems. On file-system level, a new libc plugin utilizes
libffat to enable the straight-forward use of VFAT partitions by libc-using
programs.
The current release comes with far-reaching improvements with respect to
ARM-based platforms. The paravirtualized L4Linux kernel has been updated to
Linux version 2.6.39 running on both x86_32 and ARM. Also, Qt4 including Webkit
has become functional on ARMv6-based platforms.
Among the further improvements are many new examples in the form of
ready-to-use run scripts as well as a comprehensive documentation update.
Originally, we had planned to complement the Noux runtime environment to
support interactive command-line applications by the time of the current
release. However, we realized that the current users of the framework would
value the new streamlined tooling support, the enhanced documentation, and the
new quality-assurance infrastructure over such a functional addition. Hence, we
prioritized the topics accordingly. Even though you will find the first bits of
interactive GNU application support in this release, we deferred working on
this topic in full steam to the upcoming version 11.11.
Blurring the boundaries between different kernels
#################################################
Before the Genode project was born, each microkernel carried along its own
userland. For example, the L4/Fiasco kernel came with the L4 environment, the
OKL4 kernel came with Iguana, or the L4ka::Pistachio kernel came with a small
set of example components. Those user-level counterparts of the kernel
complemented their respective kernels with a runtime for user-level
applications and components while exposing significant parts of the kernel
interface at API level. Consequently, most if not all applications developed
against these APIs were tied to a particular kernel. On the one hand, this
approach enabled developers to fine-tune their programs using kernel-specific
features. On the other hand, much effort was wasted by duplicating other
people's work. Eventually, all of the mentioned userlands stayed limited to
special purposes - for the most part the purposes of operating-systems
researchers. Consequently, none of the microkernels gained much attention in
general-purpose computing. Another consequence of the highly fragmented
microkernel community was the lack of a common ground to compare different
kernels in an unbiased way because each userland provided a different set of
components and libraries.
Different application areas call for different kernel features such as
security mechanisms, scheduling, resource management, and hardware support.
Naturally, each kernel exhibits a specific profile of these parameters
depending on its primary purpose. If one microkernel attempted to accommodate
too many features, it would certainly sacrifice the fundamental idea of being
minimally complex. Consequently, kernels happen to be vastly different. During
the past three years, however, Genode has demonstrated that one carefully
crafted API can target highly diverse kernels, and thereby enables users of
the framework to select the kernel that fits best with the requirements
dictated by each application scenario individually. For us Genode developers,
it was extremely gratifying to see that kernels as different as Linux and NOVA
can be reconciled at the programming-interface level. Still, each kernel comes
with different tools, configuration mechanisms, and boot concepts. Even though
Genode programs can be developed in a kernel-independent way, the deployment of
such programs still required profound insights into the peculiarities of the
respective kernel.
With the current release, we introduce a fundamentally new way of using
different microkernels by unifying the procedures of downloading and building
kernels as well as integrating and running Genode programs with each of them.
Existing Genode application scenarios can be ported between kernels in an
instant without the need for deep insights into the kernel's technicalities. As
a teaser, consider the following commands for building and running Genode's
graphical demo scenario on the OKL4 microkernel:
! # check out Genode
! svn co https://genode.svn.sourceforge.net/svnroot/genode/trunk genode
!
! # download the kernel, e.g., OKL4
! make -C genode/base-okl4 prepare
!
! # create Genode build directory
! genode/tool/create_builddir \
! okl4_x86 BUILD_DIR=build
!
! # build everything and execute the interactive demo
! make -C build run/demo
The same principle steps can be used for any of the OKL4, NOVA,
L4/Fiasco, Fiasco.OC, L4ka::Pistachio, or Codezero kernels. You should
nevertheless consult the documentation at 'base-<platform>/doc/' before
starting to use a specific kernel because some base platforms require
the installation of additional tools.
Under the hood, this seamless way of dealing with different kernels is made
possible by the following considerations:
:Repository preparation:
Each kernel comes from a different source such as a Git/SVN/Mercurial
repository or a packaged archive. Some kernels require additional patches. For
example, OKL4 needs to be patched to overcome problems with modern tool chains.
Now, each 'base-<platform>' repository hosts a 'Makefile' that automates the
download and patch procedure. To download the source code of a kernel,
issue 'make prepare' from within the kernel's 'base-<platform>' directory. The
3rd-party source code will be located at 'base-<platform>/contrib/'.
:Building the kernel:
Each kernel has a different approach when it comes to configuration and
compilation. For example, NOVA comes with a simple 'Makefile', OKL4 relies on a
complex SCons-based build system, L4ka::Pistachio uses CML2 and autoconf (for
the userland tools). Furthermore, some kernels require the setting of specific
configuration values. We have streamlined all these procedures into the Genode
build process by the means of a 'kernel' pseudo target and a 'platform' pseudo
library. The kernel can be compiled directly from the Genode build directory by
issuing 'make kernel'. The 'platform' pseudo library takes care of making the
kernel headers available to Genode. For some kernels such as OKL4 and NOVA, we
replaced the original build mechanism with a Genode target. For other kernels
such as L4ka::Pistachio or Fiasco.OC, we invoke the kernel's build system.
:Genode build directory:
Genode build directories are created via the 'tool/create_builddir' tool.
This tool used to require certain kernel-specific arguments such as the
location of the kernel source tree. Thanks to the unified way of preparing
kernels, the need for such arguments has vanished. Now, the only remaining
arguments to 'create_builddir' are the actual platform and the location
of the build directory to create.
:System integration and booting:
As diverse the build systems of the kernels are, so are the boot concepts. Some
kernels rely on a multiboot-compliant boot loader whereas others have special
tools for creating boot images. Thankfully, Genode's run concept allows us to
hide the peculiarities of booting behind a neat and easy-to-use facade. For
each platform we have crafted a dedicated run environment located at
'base-<platform>/run/env', which contains the rules for system integration and
booting. Therefore, one and the same run script can be used to build and
execute one application scenario across various different kernels. For an
illustrative example, the 'os/src/run/demo.run' script can be executed on all
base platforms (except for base-mb) by issuing 'make run/demo' from within the
build directory.
Emerging block-device infrastructure
####################################
Since version 10.08, Genode is equipped with a block-session interface. Its
primary use cases so far were the supply of the paravirtualized OKLinux kernel
with backing store, and the access of the content of a bootable Live CD.
However, for our mission to use Genode as general-purpose computing platform,
disk device access is crucial. Therefore, we dedicated our attention to
various aspects of Genode's block-device infrastructure, reaching from
programming APIs for block drivers, over partition handling, to file-system
access.
:Block session interface:
The glue that holds all block-device-related components together is the generic
block interface 'os/include/block_session'. It is based on the framework's
packet-stream facility, which allows the communication of bulk data via shared
memory and a data-flow protocol using asynchronous notifications. The interface
supports arbitrary allocation schemes and the use of multiple outstanding
requests. Hence, it is generally suited for scatter-gather DMA and the use of
command queuing as offered by the firmware of modern block-device controllers.
(albeit the current drivers do not exploit this potential yet)
:Block component framework:
Our observation that components implementing the block session interface share
similar code patterns prompted us to design a framework API for implementing
this family of components. The set of classes located at 'os/include/block'
facilitate the separation of device-specific code from application logic.
Whereas 'component.h' provides the application logic needed to implement the
block service, the 'driver.h' is an abstract interface to be implemented by the
actual device driver. This new infrastructure significantly reduces code
duplication among new block-device drivers.
:Device-driver implementations:
The new block-device drivers introduced with the current release address
common types of block devices:
* By adding ATA read/write support to the ATAPI driver ('os/src/drivers/atapi'),
this driver can be used to access IDE disks now.
* The new fully-functional SD-card driver ('os/src/drivers/sdcard') enables the
use of SD-cards connected via the PL180 controller.
* The USB storage driver ('linux_drivers/src/drivers/usb') has been adapted
to the block-session interface and can be used on PC hardware.
* The new AHCI driver ('os/src/drivers/ahci') enables the access of disks
connected via SATA on PC hardware.
Because all drivers are providing the generic block-session interfaces, they
can be arbitrarily combined with components that use this interface as back
end, for example, the partition server and file systems.
:Partition manager as resource multiplexer:
The new partition manager ('os/src/server/part_blk') multiplexes one back-end
block session to multiple block sessions, each accessing a different partition.
Its natural role is being "plugged" between a block-device driver and a file
system.
:File-system access:
Even though a session interface for file systems does not exist yet, we
enabled the use of VFAT partitions through a libc plugin. This libc plugin uses
the ffat library to access files stored on a block device. An
application using this plugin can be directly connected to a block session.
New documentation
#################
The new way of dealing with different kernels motivated us to revisit and
complement our exiting documentation. The following documents are new or
have received considerable attention:
:[https://genode.org/documentation/developer-resources/getting_started - Getting started]:
The revised guide of how to explore Genode provides a quick way to
test drive Genode's graphical demo scenario with a kernel of your
choice and gives pointers to documents needed to proceed your
exploration.
:[https://genode.org/documentation/developer-resources/build_system - Build system manual]:
The new build-system manual explains the concepts behind Genode's
build system, provides guidance with creating custom programs and
libraries, and covers the tool support for the automated integration
and testing of application scenarios.
:[https://genode.org/documentation/components - Components overview]:
The new components-overview document explains the categorization of
Genode's components and lists all components that come with the framework.
:[https://genode.org/documentation/developer-resources/init - Configuration of the init process]:
The document describes Genode's configuration concept, the routing of
service requests, and the expression of mandatory access-control policies.
:[https://genode.org/community/wiki - Wiki]:
The platform-specific Wiki pages for L4/Fiasco, L4ka::Pistachio, NOVA,
Codezero, Fiasco.OC, and OKL4 have been updated to reflect the new flows of
working with the respective base platforms.
Base framework
##############
The RPC API for performing procedure calls across process boundaries
introduced with the version 11.05 was the most significant API change
in Genode's history. To make the transition from the old client-server
API to the new RPC API as smooth as possible, we temporarily upheld
compatibility to the old API. Now, the time has come to put the old
API at rest. The changes that are visible at API level are as follows:
* The old client-server API in the form of 'base/server.h' is no more.
The functionality of the original classes 'Server_entrypoint' and
'Server_activation' is contained in the 'Rpc_entrypoint' class provided
via 'base/rpc_server.h'.
* When introducing the RPC API, we intentionally left the actual session
interfaces as unmodified as possible to proof the versatility of the new
facility. However, it became apparent that some of the original interfaces
could profit from using a less C-ish style. For example, some interfaces used
to pass null-terminated strings as 'char const *' rather than via a dedicated
type. The methodology of using the new RPC API while leaving the original
interfaces intact was to implement such old-style functions as wrappers
around new-style RPC functions. These wrappers were contained in
'rpc_object.h' files, e.g. for 'linux_dataspace', 'parent', 'root',
'signal_session', 'cpu_session'. Now, we have taken the chance to modernise
the API by disposing said wrappers. Thereby, the need for 'rpc_object.h'
files has (almost) vanished.
* The remaining users of the old client-server API have been adapted to the
new RPC API, most prominently, the packet-stream-related interfaces such as
'block_session', 'nic_session', and 'audio_session'.
* We removed 'Typed_capability' and the second argument of the 'Capability'
template. The latter was an artifact that was only used to support the
transition from the old to the new API.
* The 'ipc_client' has no longer an 'operator int'. The result of an IPC can
be requested via the 'result' function.
* We refined the accessors of 'Rpc_in_buffer' in 'base/rpc_args.h'. The
'addr()' has been renamed to 'base()', 'is_valid_string()' considers the
buffer's capacity, and the new 'string()' function is guaranteed to return a
null-terminated string.
* We introduced a new 'Rm_session::Local_addr' class, which serves two
purposes. It allows the transfer of the bit representation of pointers across
RPC calls and effectively removes the need for casting the return type of
'Rm_session::attach' to the type needed at the caller side.
* The 'Connection' class template has been simplified, taking the session
interface as template argument (rather than the capability type). This change
simplified the 'Connection' classes of most session interfaces.
* The never-used return value of 'Parent::announce' has been removed. From the
child's perspective, an announcement always succeeds. The way of how the
announcement is treated is entirely up to the parent. The client should never
act differently depending on the parent's policy anyway.
* The new 'Thread_base::cap()' accessor function allows obtaining the thread's
capability as used for the argument to CPU-session operations.
Operating-system services and libraries
#######################################
Dynamic linker
==============
As a follow-up to the major revision of the dynamic linker that was featured
with the previous release, we addressed several corner cases related to
exception handling and improved the handling of global symbols.
The dynamic linker used to resolve requests for global symbols by handing out
its own symbols if present. However, in some cases, this behaviour is
undesired. For example, the dynamic linker contains a small set of libc
emulation functions specifically for the ported linker code. In the presence of
the real libc, however, these symbols should never be considered at all. To
avoid such ambiguities during symbol resolution, the set of symbols to be
exported is now explicitly declared by the white-list contained in the
'os/src/lib/ldso/symbol.map' file.
We changed the linkage of the C++ support library ('cxx') against dynamic
binaries to be consistent with the other base libraries. Originally, the 'cxx'
library was linked to both the dynamic linker and the dynamic binary, which
resulted in subtle problems caused by the duplication of cxx-internal data
structures. By linking 'cxx' only to the dynamic linker and exporting the
'__cxa' ABI as global symbols, these issues have been resolved. As a positive
side effect, this change reduces the size of dynamic binaries.
C++ exception handling in the presence of shared libraries turned out to be
more challenging than we originally anticipated. For example, the
'_Unwind_Resume' symbol is exported by the compiler's 'libsupc++' as a hidden
global symbol, which can only be resolved when linking this library to the
binary but is not seen by the dynamic linker. This was the actual reason of why
we used to link 'cxx' against both dynamic binaries and shared libraries
causing the problem mentioned in the previous paragraph. Normally, this problem
is addressed by a shared library called 'libgcc_s.so' that comes with the
compiler. However, this library depends on glibc, which prevents us from using
it on Genode. Our solution is renaming the hidden global symbol using a
'_cxx__' prefix and introducing a non-hidden global wrapper function
('__cxx__Unwind_Resume' in 'unwind.cc'), which is resolved at runtime by the
dynamic linker.
Another corner case we identified is throwing exceptions from within the
dynamic linker. In contrast to the original FreeBSD version of the dynamic
linker, which is a plain C program that can never throw a C++ exception,
Genode's version relies on C++ code that makes use of exceptions. To support
C++ exceptions from within the dynamic linker, we have to relocate the
linkers's global symbols again after having loaded the dynamic binary. This
way, type information that is also present within the dynamic binary becomes
relocated to the correct positions.
Block partition server
======================
The new block-partition server uses Genode's block-session interfaces as both
front and back end, leading to the most common use case where this server will
reside between a block driver and a higher level component like a file-system
server.
At startup, the partition server will try to parse the master boot record (MBR)
of its back-end block session. If no partition table is found, the whole block
device is exported as partition '0'. In the other case, the MBR and possible
extended boot records (EBRs) are parsed and offered as separate block sessions
to the front-end clients. The four primary partitions will receive partition
numbers '1' to '4' whereas the first logical partition will be assigned to '5'.
The policy of which partition is exposed to which client can be expressed
in the config supplied to the 'part_blk' server. Please refer to the
documentation at 'os/src/server/part_blk/README' for further details. As an
illustration of the practical use of the 'part_blk' server, you can find a run
script at 'os/run/part_blk.run'.
Skeleton of text terminal
=========================
As part of the ongoing work towards using interactive text-based GNU software
on Genode, we created the first bits of the infrastructure required for
pursuing this quest:
The new terminal-session interface at 'os/include/terminal_session/' is the
designated interface to be implemented by terminal programs.
After investigating the pros and cons of various terminal protocols and
terminal emulators, we settled for implementing a custom terminal emulator
implementing the Linux termcap. This termcap offers a reasonable small set of
commands while providing all essential features such as function-key support
and mouse support. Thanks to Peter Persson for pointing us to the right
direction! The preliminary code for parsing the escape sequences for the Linux
termcap is located at 'gems/include/terminal/'.
We have created a simplistic terminal service that implements the
terminal-session interface using a built-in font. Please note that the
implementation at 'gems/src/server/terminal/' is at an early stage. It is
accompanied by a simple echo program located at 'gems/src/test/terminal_echo'.
Device drivers
##############
USB HID and USB storage
=======================
We replaced the former DDE-Linux-based USB-related driver libraries (at the
'linux_drivers/' repository) by a single USB driver server that offers the
'Input' and 'Block' services. This enables us to use both USB HID and USB
storage at the same time. The new USB driver is located at
'linux_drivers/src/drivers/usb/'.
For using the USB driver as input service (supporting USB HID), add the
'<hid/>' tag to the 'usb_drv' configuration. Analogously, for using the driver
as block service, add the '<storage/>' tag. Both tags can be combined.
For testing the USB stack, the 'linux_drivers' repository comes with the run
scripts 'usb_hid.run' and 'usb_storage.run'.
ATA read/write support
======================
The ATAPI driver has been extended to support IDE block devices for both
read and write transactions. To use the new facility, supply 'ata="yes"'
as XML attribute to the config node of 'atapi_drv'. Please note that this
driver was primarily tested on Qemu. Use it with caution.
SATA driver
===========
The new SATA driver at 'os/src/drivers/ahci/' implements the block-driver
API ('os/include/block'), thus exposing the block-session interface as
front-end. AHCI depends on Genode's PCI driver as well as the timer server. For
a usage example see: 'os/run/ahci.run'.
Limitations and known issues
----------------------------
Currently, the server scans the PCI bus at startup and retrieves the first available
AHCI controller, scans the controller ports and uses the first non-ATAPI port
where a device is present.
On real hardware and on kernels taking advantage of I/O APICs (namely NOVA and
Fiasco.OC) we still lack support for ACPI parsing and thus for interrupts,
leading to a non-working driver.
SD-card driver
==============
The first fragments of our SD-card driver that we introduced with the previous
release have been complemented. The new SD-card driver located at
'os/src/drivers/sd_card/' implements the block-session interface by using
MMC/SD-cards and the PL180 controller as back end. Currently the driver
supports single-capacity SD cards. Therefore, the block file for Qemu should
not exceed 512 MB. Because the driver provides the generic block-session
interface, it can be combined with the new 'libc_ffat' plugin in a
straight-forward way. To give the driver a quick spin, you may give the
'libports/run/libc_ffat.run' script on the 'foc_pbxa9' platform a try.
ARM Realview PL011 UART driver
==============================
The new PL011 UART driver at 'os/src/drivers/uart/' implements the LOG session
interface using the PL011 device. Up to 4 UARTs are supported. The assignment
of UARTs to clients can be defined via a policy supplied to the driver's config
node. For further information, please refer to the README file within the
'uart' directory.
Libraries and applications
##########################
Hello tutorial
==============
The 'hello_tutorial/' repository contains a step-by-step guide for building
a simple client-server scenario. The tutorial has been rewritten for the new
RPC API and is now complemented by a run script for testing the final scenario
on various base platforms.
C and C++ runtimes
==================
:Support for standard C++ headers:
Triggered by public demand for using standard C++ headers for Genode applications,
we introduced a generally usable solution in the form of the 'stdcxx' library
to the 'libc' repository. The new 'stdcxx' library is not a real library. (you
will find the corresponding 'lib/mk/stdcxx.mk' file empty) However, it comes
with a 'lib/import/import-stdcxx.mk' file that adds the compiler's C++ includes
to the default include-search path for any target that has 'stdcxx' listed in
its 'LIBS' declaration.
:Libc back end for accessing VFAT partitions:
The new 'libc_ffat' libc plugin uses a block session via the ffat library. It
can be used by a Genode application to access a VFAT file system via the libc
file API. The file-system access is performed via the 'ffat' library. To
download this library and integrate it with Genode, change to the 'libports'
repository and issue the following command:
! make prepare PKG=ffat
For an example of how to use the libc-ffat plugin, please refer to the run
script 'libports/run/libc_ffat.run'. The source code of the test program can be
found at 'libports/src/test/libc_ffat/'.
Qt4
===
Qt4 version 4.7.1 has been enabled on ARMv6-based platforms, i.e., PBX-A9 on
Fiasco.OC. The support comprises the entire Qt4 framework including qt_webcore
(Webkit).
L4Linux
=======
L4Linux enables the use of one or multiple instances of Linux-based operating
systems as subsystems running on the Fiasco.OC kernel. The Genode version of
L4Linux has seen the following improvements:
:Kernel version: has been updated to Linux 2.6.39.
:ARM support: The L4Linux kernel can be used on ARM-based platforms now.
The PBX-A9 platform is supported via the 'l4linux.run' script as found
at 'ports-foc/run/'. Please find more information at 'ports-foc/README'.
:Genode-specific stub drivers outside the kernel tree:
The stub drivers that enable the use of Genode's services as virtual
devices for L4Linux have been moved outside the kernel patch, which
makes them much easier to maintain. These stub drivers are located
under 'ports-foc/src/drivers/'.
Platform support
################
All base platforms are now handled in a unified fashion. Downloading 3rd-party
source code is performed using the 'prepare' rule of the 'Makefile' provided by
the respective kernel's 'base-<platform>' repository. Once, the platform's base
repository is prepared, the kernel can be built directly from the Genode
build directory using 'make kernel'. All base platforms are now supported by
Genode's run mechanism that automates the tasks of system integration and
testing. For more details about each specific kernel, please revisit the
updated documentation within the respective 'base-<platform>/doc/' directory.
:L4/Fiasco:
The kernel has been updated to revision 472, enabling the use of recent
GNU tool chains.
:Fiasco.OC:
The kernel as been updated to revision 36, which remedies stability problems
related to interaction of the IPC path with thread destruction. The new version
improves the stability of highly dynamic workloads that involve the frequent
creation and destruction of subsystems. However, we experienced the new kernel
version to behave instable on the x86_64 architecture. If you depend on x86_64,
we recommend to temporarily stick with Genode 11.05 and Fiasco.OC revision 31.
:L4ka::Pistachio:
The kernel has been updated to revision 803, enabling the use of recent
versions of binutils.
:OKL4:
OKL4v2 is showing its age. Apparently, the use of the original distribution
requires tools (i.e., python 2.4) that do not ship with current Linux
distributions anymore. This makes it increasingly difficult to use this kernel.
Still, we find ourselves frequently using it for our day-to-day development. To
streamline the use of OKL4v2, we have now incorporated the kernel compilation
into the Genode build system and thereby weakened the kernel's dependency on
ancient tools. However, we decided to drop support for OKL4/ARM for now. We
figured that the supported GTA01 platform is hardly used anymore and hard to
test because it is unsupported by Qemu. Newer ARM platforms are supported by
other kernels anyway.
:Codezero:
Even though B-Labs apparently abandoned the idea of developing the Codezero
kernel in the open, we adapted Genode to the kernel's most recent Open-Source
version that is still available at the official Git repository. Furthermore,
the kernel is now fully supported by Genode's new 'make prepare' procedure and
run environment. Therefore, run scripts such as 'run/demo' can now easily be
executed on Codezero without the need to manually configure the kernel.
Note that, for now, we have disabled Codezero's capabilities because they do
not allow the assignment of device resources. Consequently, 'sys_map' fails for
MMIO regions when performing the capability check (calling 'cap_map_check').
Furthermore, the current version of the kernel requires a workaround for a
current limitation regarding the definition of a thread's pager. At some point,
Codezero abandoned the facility to define the pager for a given thread via the
exregs system call. Instead, the kernel hard-wires the creator of the thread as
the thread's pager. This is conflicting with Genode's way of creating and
paging threads. In the current version of Genode for this kernel, all threads
are paged by one thread (thread 3 happens to be the global pager) within core.
As a workaround to Codezero's current limitation, we define thread 3 to be the
pager of all threads. The patch of the upstream code is automatically being
applied by the 'make prepare' mechanism.
Build system and tools
######################
In addition to the major change with respect to the integration of the various
base platforms, Genode's tool support received the following incremental
improvements:
Build system
============
:Simplification of 'create_builddir' tool:
The 'create_builddir' tool has been relocated from
'tool/builddir/create_builddir' to 'tool/create_builddir' to make it more
readily accessible. Furthermore, we simplified the usage of the tool by
removing the mandatory 'GENODE_DIR' argument. If not explicitly specified, the
tool deduces 'GENODE_DIR' from the its known location within the Genode source
tree.
:Booting from USB sticks:
For most x86-based base platforms, their respective run environments execute
Genode from an ISO image via Qemu. Naturally, such an ISO image can be burned
onto a CD-ROM to be used to boot a real machine. However, booting from CD-ROM
is slow and optical drives are becoming scarce. Therefore we changed the
procedure of creating ISO images to support writing the resulting images to a
USB stick. Under the hood, the boot mechanism chain-loads GRUB via ISOLinux.
The files to implement the boot concept are located at 'tool/boot/'.
:Support for source files in target sub directories:
Until now, the 'SRC_*' declarations in target description files contained
a list of plain file names. The location of the files within the directory
tree had to be defined via 'vpath'. This led to inconveniences when building
3rd-party code that contains files with the same name at different subdirectories.
To resolve such an ambiguity, the target had to be decomposed into multiple
libraries each building a different set of subdirectories. To make the
build system more convenient to use, we have now added support for specifying
source codes with a relative pathname. For example, instead of using
! SRC_CC = main.cc addon.cc
! vpath addon.cc $(PRG_DIR)/contrib
we can now use
! SRC_CC = main.cc contrib/addon.cc
Automated testing across multiple kernels
=========================================
To execute one or multiple test cases on more than one base platform, we
introduced a dedicated tool located at 'tool/autopilot'. Its primary purpose is
the nightly execution of test cases. The tool takes a list of platforms and a
list of run scripts as arguments and executes each run script on each platform.
The build directory for each platform is created at
'/tmp/autopilot.<username>/<platform>' and the output of each run script is
written to a file called '<platform>.<run-script>.log'. On stderr, autopilot
prints the statistics about whether or not each run script executed
successfully on each platform. If at least one run script failed, autopilot
returns a non-zero exit code, which makes it straight forward to include
autopilot into an automated build-and-test environment.

1008
doc/release_notes/11-11.txt Normal file
View File

File diff suppressed because it is too large Load Diff

862
doc/release_notes/12-02.txt Normal file
View File

@@ -0,0 +1,862 @@
===============================================
Release notes for the Genode OS Framework 12.02
===============================================
Genode Labs
The release of Genode 12.02 marks an exciting point in the history of the
project as it is the first version developed in the open rather than within the
chambers of Genode Labs. Thereby, we have embraced GitHub as central facility
for discussion and source-code management. This change has benefits for users
and developers of the framework alike. For users, it has become possible to get
hold of the latest developments using the official 'genodelabs/master' branch and
get involved with discussing the current activities. For regular Genode
developers, the public Git repository replaces a former mix of public
Subversion and company-internal Mercurial repositories, making life much
easier. In Section [Liberation of the development process], we outline the
motivation behind this change and give pointers to the new resources.
The major new additions to the base system are a new framework API for accessing
memory-mapped I/O resources, special support for using Genode as user-level
component framework on Linux, and API support for the reuse of existing
components in the form of sandboxed libraries. These changes are accompanied
with new device-driver infrastructure such as the first version of a device
driver manager and a new ACPI parser.
Feature-wise, the current release takes the first steps towards the goal of the
[https://genode.org/about/road-map - Roadmap for 2012], turning Genode into a
general-purpose OS ready for everyday use by its developers. According to the
roadmap, we enhanced the Noux runtime with fork semantics so that we can run
command-line based GNU programs such as the bash shell and coreutils unmodified
and natively on various microkernels. Furthermore, the library infrastructure
has been enhanced by porting and updating libraries such as Qt 4.7.4 and the
MuPDF PDF rendering engine.
Liberation of the development process
#####################################
In summer 2011, we started a discussion within Genode Labs about changing the
mode of how Genode is developed. Until then, most design discussions and the
actual development work took place locally at the company. At quarterly
intervals, we used to publish our work in the form of official Genode
releases. This way of development seemed to work quite well for us, we were
satisfied about the pace of development, and with each release, our project got
more recognition.
However, the excellent book [https://producingoss.com/ - Producing Open Source Software]
made us realize that even though we released our work under an Open-Source
license, our development process was actually far from being open and may have
discouraged participation of people outside the inner circle of developers.
Because we believe that the framework has reached a state where it becomes
interesting for a wider audience of users and developers, the idea was born
to liberate the project from its closed fashion of development.
In the beginning of December, the vague idea has become a plan. So we finally
brought the topic to our mailing list
([https://genode.org/news/steps-towards-an-open-development-process - Steps towards an open development process]).
We decided to take the release cycle for Genode 12.02 as the opportunity to put
our plan to practice. The central element of this endeavour was moving the
project over to GitHub and adapt our workflows and tooling support accordingly.
First, we started to embrace GitHub's issue tracker for the management of
working topics:
:[https://github.com/genodelabs/genode/issues]: Issue Tracker
The most significant step was leaving our Genode-Labs-internal code
repositories behind and starting a completely public Git repository instead:
:[https://github.com/genodelabs]: Genode Labs at GitHub
With the code repository going public, the way was cleared to opening up design
discussions as well. Instead of having such discussions internally at Genode
Labs, we try to increasingly take them to our mailing list and issue tracker.
With this new way of development, we hope to make the project much more
approachable for people who want to get involved and let Genode reach far out
beyond the reach of our little company.
The changes mentioned above are actually just the tip of the iceberg. For
example, the transition phase required us to rethink the way the project
website is maintained. From now on, almost all of the content of genode.org
comes directly from the project's Git repository. So maintaining website
content is done in the same coherent and transparent way as working on Genode's
code base. So we could finally put the old Wiki to rest. In the process, we
largely revisited the existing content. For example, we rewrote the
[https://genode.org/community/contributions - contributions] document in a
tutorial-like style and incorporated several practical hints, in particular
related to the recommended use of Git.
Although it is probably too early to judge the outcome of our transition, we
are excited how smooth this massive change went. We attribute this pleasant
experience mostly to the excellent GitHub hosting platform, which instantly
ignited a spirit of open collaboration among us. We are excited to see new
people approaching us and showing their interest for teaming up, and we are
curious about where this new model of development will take Genode in the
future.
Base framework, low-level OS infrastructure
###########################################
RPC framework refinements
=========================
Until now, the RPC framework did not support const RPC functions. Rather than
being a limitation inherent to the concept, const RPC functions plainly did not
exist. So supporting them was not deemed too important. However, there are uses
of RPC interfaces that would benefit from a way to declare an RPC function as
const. Candidates are functions like 'Framebuffer::Session::mode()' and
'Input::Session::is_pending()'.
With the current version, we clear the way towards declaring such functions as
const. Even though the change is pretty straight-forward, the thorough support
for const-qualified RPC functions would double the number of overloads for the
'call_member' function template (in 'base/include/util/meta.h'). For this
reason, as of now, the support of const functions is limited to typical getter
functions with no arguments. This appears to be the most common use of such
functions.
API support for enslaving services
==================================
While evolving and using the framework, we always keep an eye on recurring
patterns of how its API is used. Once such a pattern becomes obvious, we try
to take a step back, generalize the observed pattern, and come up with a new
building block that unifies the former repetitive code fragments.
One of those patterns that was far from obvious when we designed Genode years
ago is the use of a service running as child of its own client. At the first
glance, this idea seems counter-intuitive because normally, services are
understood as components that operate independently and protected from their
(untrusted) clients. But there is a class of problems where this approach
becomes extremely useful: The reuse of protocol implementations as a
library-like building block. Most services are actually protocol stacks that
translate a low-level protocol to a more abstract API. For example, a block
device driver translates a specific device API to the generic 'Block_session'
interface. Or the 'iso9660' service translates the 'Block_session' interface to
the 'Rom_session' interface by parsing the ISO9660 file system. Or similarly,
the 'tar_rom' service parses the tar file format to make its content available
via the 'Rom_session' interface.
If a particular functionality is needed by multiple programs, it is common
practice to move this functionality into a dedicated library to avoid the
duplication of the same code at many places. For example, if a program would
need to parse a tar archive, it might be tempting to move the tar-parsing code
from the 'tar_rom' service into a dedicated library, which can then be used by
both the 'tar_rom' service and the new program. An alternative approach is to
just re-use the 'tar_rom' service as a black box and treat it like it was a
library. That is, start the 'tar_rom' service as a child process, supply the
resources the component needs to operate and, in turn, use its API (now in the
form of an RPC interface) to get work done. Because the service is started as a
mere tool at the discretion of its client, we call it *slave*. It turns out
that this idea works exceedingly well in many cases. In a way, it resembles the
Unix philosophy to solve complex problems by combining many small tools each
with a specific purpose. In contrast to the use of libraries, the reuse of
entire components has benefits with regard to fault isolation. Because the
reused functionality is sandboxed within a distinct process, the environment
exposed to this code can be tailored to a rigid subset of the host program's
environment. In the event of a fault within the reused component, the reach of
problem is therefore limited.
On the other hand, we observed that even though this idea works as intended,
implementing the idea for a particular use case involved a good deal of
boiler-plate code where most of this code is needed to define the slave's
environment and resources. Hence, we reviewed the existing occurrences of the
slave pattern and condensed their common concerns into the 'Slave_policy' and
'Slave' classes residing in 'os/include/os/slave.h'. The 'Slave' class is meant
to be used as is. It is merely a convenience wrapper for a child process and
its basic resources. The 'Slave_policy' is meant as a hook for service-specific
customizations. The best showcase is the new 'd3m' component located at
'gems/src/server/d3m'. D3m extensively uses the slave pattern by instantiating
and destroying drivers and file-system instances on-the-fly. A further instance
of this pattern can be found in the new ACPI driver introduced with the current
release.
Support for resizable framebuffers
==================================
The framebuffer-session interface has remained largely untouched since the
original release of Genode in 2008. Back then, we were used to rely on C-style
out parameters in RPC functions. The current RPC framework, however, promotes
the use of a more object-oriented style. So the time has come to revisit the
framebuffer session interface. Instead of using C-style out parameters, the new
'mode()' RPC call returns the mode information as an object of type 'Mode'.
Consequently, mode-specific functions such as 'bytes_per_pixel()' have been
moved to the new 'Framebuffer::Mode' class. The former 'info()' function is
gone.
In addition to the overhaul of the RPC interface, we introduced basic support
for resizable framebuffers. The new 'mode_sigh()' function enables a client to
register a signal handler at the framebuffer session. This signal handler gets
notified in the event of server-side mode changes. Via the new 'release()'
function, the client is able to acknowledge a mode change. By calling it, the
client tells the framebuffer service that it no longer uses the original
framebuffer dataspace. So the server can replace it by a new one. After having
called 'release()', the client can obtain the dataspace for the new mode by
calling 'dataspace()' again.
MMIO access framework
=====================
As the arsenal of native device drivers for Genode grows, we are observing
an increased demand to formalize the style of how drivers are written to
foster code consistency. One particular cause of inconsistency used to be
the way of how memory-mapped I/O registers are accessed. C++ has poor support
for defining bit-accurate register layouts in memory. Therefore, driver code
usually carries along a custom set of convenience functions for reading and
writing registers of different widths as well as a list of bit definitions in
the form of enum values or '#define' statements. To access parts of a register,
the usual pattern is similar to the following example (taken from the pl011
UART driver:
! enum {
! UARTCR = 0x030, /* control register */
! UARTCR_UARTEN = 0x0001, /* enable bit in control register */
! ...
! }
! ...
!
! /* enable UART */
! _write_reg(UARTCR, _read_reg(UARTCR) | UARTCR_UARTEN);
This example showcases two inconveniences: The way the register layout is
expressed and the manual labour needed to access parts of registers. In the
general case, a driver needs to also consider 'MASK' and 'SHIFT' values to
implement access to partial registers properly. This is not just inconvenient
but also error prone. For lazy programmers as ourselves, it's just too easy to
overwrite "undefined" bits in a register instead of explicitly masking the
access to the actually targeted bits. Consequently, the driver may work fine
with the current generation of devices but break with the next generation.
So the idea was born to introduce an easy-to-use formalism for this problem. We
had two goals: First, we wanted to cleanly separate the declaration of register
layouts from the program logic of the driver. The actual driver program should
be free from any intrinsics in the form of bit-masking operations. Second, we
wanted to promote uncluttered driver code that uses names (i.e., in the form of
type names) rather than values to express its operations. The latter goal is
actually achieved by the example above by the use of enum values, but this is
only achieved through discipline. We would prefer to have an API that
facilitates the use of proper names as the most convenient way to express an
operation.
The resulting MMIO API comes in the form of two new header files located at
'base/include/util/register.h' and 'base/include/util/mmio.h'.
Register declarations
~~~~~~~~~~~~~~~~~~~~~
The class templates found in 'util/register.h' provide a means to express
register layouts using C++ types. In a way, these templates make up for
C++'s missing facility to define accurate bitfields. Let's take a look at
a simple example of the 'Register' class template that can be used to define
a register as well as a bitfield within this register:
! struct Vaporizer : Register<16>
! {
! struct Enable : Bitfield<2,1> { };
! struct State : Bitfield<3,3> {
! enum{ SOLID = 1, LIQUID = 2, GASSY = 3 };
! };
!
! static void write (access_t value);
! static access_t read ();
! };
In the example, 'Vaporizer' is a 16-bit register, which is expressed via the
'Register' template argument. The 'Register' class template allows for
accessing register content at a finer granularity than the whole register
width. To give a specific part of the register a name, the 'Register::Bitfield'
class template is used. It describes a bit region within the range of the
compound register. The bit 2 corresponds to true if the device is enabled and
bits 3 to 5 encode the 'State'. To access the actual register, the methods
'read()' and 'write()' must be provided as backend, which performs the access
of the whole register. Once defined, the 'Vaporizer' offers a handy way to
access the individual parts of the register, for example:
! /* read the whole register content */
! Vaporizer::access_t r = Vaporizer::read();
!
! /* clear a bit field */
! Vaporizer::Enable::clear(r);
!
! /* read a bit field value */
! unsigned old_state = Vaporizer::State::get(r);
!
! /* assign new bit field value */
! Vaporizer::State::set(r, Vaporizer::State::LIQUID);
!
! /* write whole register */
! Vaporizer::write(r);
Memory-mapped I/O
~~~~~~~~~~~~~~~~~
The utilities provided by 'util/mmio.h' use the 'Register' template class as
a building block to provide easy-to-use access to memory-mapped I/O registers.
The 'Mmio' class represents a memory-mapped I/O region taking its local base
address as constructor argument. Let's take a simple example to see how it is
supposed to be used:
! class Timer : Mmio
! {
! struct Value : Register<0x0, 32> { };
! struct Control : Register<0x4, 8> {
! struct Enable : Bitfield<0,1> { };
! struct Irq : Bitfield<3,1> { };
! struct Method : Bitfield<1,2>
! {
! enum { ONCE = 1, RELOAD = 2, CYCLE = 3 };
! };
! };
!
! public:
!
! Timer(addr_t base) : Mmio(base) { }
!
! void enable();
! void set_timeout(Value::access_t duration);
! bool irq_raised();
! };
The memory-mapped timer device consists of two registers: The 32-bit 'Value'
register and the 8-bit 'Control' register. They are located at the MMIO offsets
0x0 and 0x4, respectively. Some parts of the 'Control' register have specific
meanings as expressed by the 'Bitfield' definitions within the 'Control'
struct.
Using these declarations, accessing the individual bits becomes almost a
verbatim description of how the device is used. For example:
! void enable()
! {
! /* access an individual bitfield */
! write<Control::Enable>(true);
! }
!
! void set_timeout(Value::access_t duration)
! {
! /* write complete content of a register */
! write<Value>(duration);
!
! /* write all bitfields as one transaction */
! write<Control>(Control::Enable::bits(1) |
! Control::Method::bits(Control::Method::ONCE) |
! Control::Irq::bits(0));
! }
!
! bool irq_raised()
! {
! return read<Control::Irq>();
! }
In addition to those basic facilities, further noteworthy features of the new
API are the support for register arrays and the graceful overflow handling
with respect to register and bitfield boundaries.
C Runtime
=========
We extended our FreeBSD-based C runtime to accommodate the needs of the Noux
runtime environment and our port of the MuPDF application.
* The dummy implementation of '_ioctl()' has been removed. This function is
called internally within the libc, i.e., by 'tcgetattr()'. For running
libreadline in Noux, we need to hook into those ioctl operations via the
libc plugin interface.
* The 'libc/regex' and 'libc/compat' modules have been added to the libc.
These libraries are needed by the forthcoming port of Slashem to Noux.
* We added a default implementation of 'chdir()'. It relies on the sequence of
'open()', 'fchdir()', 'close()'.
* The new libc plugin 'libc_rom' enables the use of libc file I/O functions
to access ROM files as provided by Genode ROM session.
* We changed the libc dummy implementations to always return ENOSYS. Prior
this change, 'errno' used to remain untouched by those functions causing
indeterministic behaviour of code that calls those functions, observes the
error return value (as returned by most dummies), and evaluates the error
condition reported by errno.
* If using the libc for Noux programs, the default implementations of
time-related functions such as 'gettimeofday()' cannot be used because they
rely on a dedicated timeout-scheduler thread. Noux programs, however, are
expected to contain only the main thread. By turning the functions into weak
symbols, we enabled the noux libc-plugin to provide custom implementations.
DDE Kit
=======
Linux DDE used to implement Linux spin locks based on 'dde_kit_lock'. This
works fine if a spin lock is initialized only once and used from then on. But
if spin locks are initialized on-the-fly at a high rate, each initialization
causes the allocation of a new 'dde_kit_lock'. Because in contrast to normal
locks, spinlocks cannot be explicitly destroyed, the spin-lock emulating locks
are never freed. To solve the leakage of locks, we complemented DDE Kit with
the new 'os/include/dde_kit/spin_lock.h' API providing the semantics as
expected by Linux drivers.
Libraries and applications
##########################
New and updated libraries
=========================
:Qt4 updated to version 4.7.4:
We updated Qt4 from version 4.7.1 to version 4.7.4. For the most part, the
update contains bug fixes as detailed in the release notes for the versions
[https://qt.nokia.com/products/changes/changes-4.7.2 - 4.7.2],
[https://qt.nokia.com/products/changes/changes-4.7.3 - 4.7.3], and
[https://labs.qt.nokia.com/2011/09/01/qt-4-7-4-released - 4.7.4].
:Update of zlib to version 1.2.6:
Because zlib 1.2.5 is no more available at zlib.net, we bumped the zlib
version to 1.2.6.
:New ports of openjpeg, jbig2dec, and mupdf:
MuPDF is a fast and versatile PDF rendering library with only a few
dependencies. It depends on openjpeg (JPEG2000 codec) and jbig2dec (b/w image
compression library). With the current version, we integrated those libraries
alongside the MuPDF library to the 'libports' repository.
GDB monitor refinements and automated test
==========================================
We improved the support for GDB-based user-level debugging as introduced with
the previous release.
For the x86 architecture, we fixed a corner-case problem with using the
two-byte 'INT 0' instruction for breakpoints. The fix changes the breakpoint
instruction to the single-byte 'HLT'. 'HLT' is a privileged instruction and
triggers an exception when executed in user mode.
The new 'gdb_monitor_interactive.run' script extends the original
'gdb_monitor.run' script with a startup sequence that automates the
initial break-in at the 'main()' function of a dynamically linked binary.
The revised 'gdb_monitor.run' script has been improved to become a full
automated test case for GDB functionalities. It exercises the following
features (currently on Fiasco.OC only):
* Breakpoint in 'main()'
* Breakpoint in a shared-library function
* Stack trace when not in a syscall
* Thread info
* Single stepping
* Handling of segmentation-fault exception
* Stack trace when in a syscall
PDF viewer
==========
According to our road map for 2012, we pursued the port of an existing PDF
viewer as native application to Genode.
We first looked at the [https://poppler.freedesktop.org - libpoppler],
which seems to be the most popular PDF rendering engine in the world of
freedesktop.org. To get a grasp on what the porting effort of this engine may
be, we looked at projects using this library as well as the library source
code. By examining applications such as the light-weight epdfview
application, we observed that libpoppler's primary design goal is to integrate
well with existing freedesktop.org infrastructure rather than to reimplement
functionality that is provided by another library. For example, fontconfig is
used to obtain font information and Cairo is used as rendering backend. In the
context of freedesktop.org, this makes perfect sense. But in our context,
porting libpoppler would require us to port all this infrastructure to Genode
as well. To illustrate the order of magnitude of the effort needed, epdfview
depends on 65 shared libraries. Of course, at some point in the future, we will
be faced to carry out this porting work. But for the immediate goal to have a
PDF rendering engine available on Genode, it seems overly involved. Another
criterion to evaluate the feasibility of integrating libpoppler with Genode is
its API. By glancing at the API, it seems to be extremely feature rich and
complex - certainly not a thing to conquer in one evening with a glass of wine.
The Qt4 backend of the library comprises circa 8000 lines of code. This value
can be taken as a vague hint at the amount of work needed to create a custom
backend, i.e., for Genode's framebuffer-session interface.
Fortunately for us, there exists an alternative PDF rendering engine named
MuPDF. The concept of MuPDF is quite the opposite of that of libpoppler.
MuPDF tries to be as self-sufficient as possible in order to be suitable
for embedded systems without comprehensive OS infrastructure. It comes with a
custom vector-graphics library (instead of using an existing library such as
Cairo) and it even has statically linked-in all font information needed to
display PDF files that come with no embedded fonts. That said, it does not
try to reinvent the wheel in every regard. For example, it relies on
common libraries such as zlib, libpng, jpeg, freetype, and openjpeg. Most
of them are already available on Genode. And the remaining libraries are rather
free-standing and easy to port. To illustrate the low degree of dependencies,
the MuPDF application on GNU/Linux depends on only 15 shared libraries. The
best thing about MuPDF from our perspective however, is its lean and clean API,
and the wonderfully simple example application. Thanks to this example, it was
a breeze to integrate the MuPDF engine with Genode's native framebuffer-session
and input-session interfaces. The effort needed for this integration work lies
in the order of less than 300 lines of code.
At the current stage, the MuPDF rendering engine successfully runs on Genode
in the form of a simple interactive test program, which can be started
via the 'libports/run/mupdf' run script. The program supports the basic key
handling required to browse through a multi-page PDF document
(page-up or enter -> next page, page-down or backspace -> previous page).
Improved terminal performance
=============================
The terminal component used to make all intermediate states visible to the
framebuffer in a fully synchronous fashion. This is an unfortunate behaviour
when scrolling through large text outputs. By decoupling the conversion of the
terminal state to pixels from the 'Terminal::write()' RPC function,
intermediate terminal states produced by sub sequential write operations do not
end up on screen one by one but only the final state becomes visible. This
improvement drastically improves the speed in situations with a lot of
intermediate states.
Noux support for fork semantics
===============================
Genode proclaims to be a framework out of which operating systems can be built.
There is no better way of putting this claim to the test than to use the
framework for building a Unix-like OS. This is the role of the Noux runtime
environment.
During the past releases, Noux evolved into a runtime environment that is able
to execute complex command-line-based GNU software such as VIM with no
modification. However, we cannot talk of Unix without talking about its
fundamental concept embodied in the form of the 'fork()' system call. We did
not entirely dismiss the idea of implementing 'fork()' into Noux but up to now,
it was something that we willingly overlooked. However, the primary goal of
Noux is to run the GNU userland natively on Genode. This includes a good deal
of programs that rely on fork semantics. We could either try to change all the
programs to use a Genode-specific way of starting programs or bite in the
bullet and implement fork. With the current release, we did the latter.
The biggest challenge of implementing fork was to find a solution that is not
tied to one kernel but one that works across all the different base platforms.
The principle problem of starting a new process in a platform-independent
manner is already solved by Genode in the form of the 'Process' API. But this
startup procedure is entirely different from the semantics of fork. The key to
the solution was Genode's natural ability to virtualize the access to low-level
platform resources. To implement fork semantics, all Noux has to do is to
provide local implementations of core's RAM, RM, and CPU session interfaces.
The custom implementation of the CPU session interface is used to tweak the
startup procedure as performed by the 'Process' class. Normally, processes
start execution immediately at creation time at the ELF entry point. For
implementing fork semantics, however, this default behavior does not work.
Instead, we need to defer the start of the main thread until we have finished
copying the address space of the forking process. Furthermore, we need to start
the main thread at a custom trampoline function rather than at the ELF entry
point. Those customizations are possible by wrapping core's CPU service.
The custom implementation of the RAM session interface provides a pool of RAM
shared by Noux and all Noux processes. The use of a shared pool alleviates the
need to assign RAM quota to individual Noux processes. Furthermore, the custom
implementation is needed to get hold of the RAM dataspaces allocated by each
Noux process. When forking a process, the acquired information is used to
create a shadow copy of the forking address space.
Finally, a custom RM service implementation is used for recording all RM
regions attached to the region-manager session of a Noux process. Using the
recorded information, the address-space layout can then be replayed onto a new
process created via fork.
With the virtualized platform resources in place, the only puzzle piece that
is missing is the bootstrapping of the new process. When its main thread is
started, it has an identical address-space content as the forking process but
it has to talk to a different parent entrypoint and a different Noux session.
The procedure of re-establishing the relationship of the new process to its
parent is achieved via a small trampoline function that re-initializes the
process environment and then branches to the original forking point via
setjmp/longjmp. This mechanism is implemented in the libc_noux plugin.
As the immediate result of this work, a simple fork test can be executed across
all base platforms except for Linux (Linux is not supported yet). The test
program is located at 'ports/src/test/noux_fork' and can be started with the
'ports/run/noux_fork.run' script.
Furthermore, as a slightly more exciting example, there is a run script for
running a bash shell on a tar file system that contains coreutils. By starting
the 'ports/run/noux_bash.run' script, you get presented an interactive bash
shell. The shell is displayed via the terminal service and accepts user input.
It allows you to start one of the coreutils programs such as ls or cat. Please
note that the current state is still largely untested, incomplete, and flaky.
But considering that Noux is comprised of less than 2500 lines of code, we are
quite surprised of how far one can get with such little effort.
Device drivers
##############
Driver improvements to accommodate dynamic (re-)loading
=======================================================
To support the dynamic probing of devices as performed by the new d3m
component, the ATAPI and USB device drivers have been enhanced to support the
subsequent closing and re-opening of sessions.
First bits of the d3m device-driver manager
===========================================
The abbreviation d3m stands for demo device-driver manager. It is our current
solution for the automated loading of suitable drivers as needed for running
Genode from a Live CD or USB stick. Because of the current narrow focus of d3m,
it is not a generic driver-management solution but a first step in this
direction. We hope that in the long run, d3m will evolve to become a generic
driver-management facility so that we can remove one of the "D"s from its name.
In the current form d3m solves the problems of merging input-event streams,
selecting the boot device, and dealing with failing network drivers.
When using the live CD, we expect user input to come from USB HID devices or
from a PS/2 mouse and keyboard. The live system should be operational if at
least one of those sources of input is available. In the presence of multiple
sources, we want to accumulate the events of all of them.
The live CD should come in the form of a single ISO image that can be burned onto a CDROM or
alternatively copied to an USB stick. The live system should boot fine in both
cases. The first boot stage is accommodated by syslinux and the GRUB boot
loader using BIOS functions. But once Genode takes over control, it needs to
figure out on its own from where to fetch data. A priori, there is no way to
guess whether the ATAPI driver or the USB storage driver should be used.
[image d3m_what_next]
Therefore, d3m implements a probing mechanism that starts each of the drivers,
probes for the presence of a particular file on an iso9660 file system.
[image d3m_probing]
Once d3m observes a drivers that is able to successfully access the magic file,
it keeps the driver and announces the driver's service to its own parent. For
the system outside of d3m, the probing procedure is completely transparent. D3m
appears to be just a service that always provides the valid block session for
the boot medium.
[image d3m_ready]
The network device drivers that we ported from the iPXE project cover the
range of most common wired network adaptors, in particular the E1000 family.
But we cannot presume that a computer running the live system comes equipped
with one of the supported devices. If no supported network card could be
detected the driver would simply fail. Applications requesting a NIC session
would block until a NIC service becomes available, which won't happen. To
prevent this situation, d3m wraps the NIC driver and provides a dummy NIC
service in the event the drivers fails. This way, the client application won't
block infinitely but receive an error on the first attempt to use the NIC.
ACPI support
============
To accommodate kernels like Fiasco.OC or NOVA that take advantage of x86's
APIC, we have introduced a simple ACPI parser located at 'os/src/drivers/acpi'.
The server traverses the ACPI tables and sets the interrupt line of devices
within the PCI config space to the GSIs found in the ACPI tables. Internally it
uses Genode's existing PCI driver as a child process for performing PCI access
and, in turn, announces a PCI service itself.
For obtaining the IRQ routing information from the ACPI tables without
employing a full-blown ACPI interpreter, the ACPI driver uses an ingenious
technique invented by Bernhard Kauer, which is described in the following
paper:
:[https://os.inf.tu-dresden.de/papers_ps/tr-atare-2009.pdf - ATARE - ACPI Tables and Regular Expressions]:
_TU Dresden technical report TUD-FI09-09, Dresden, Germany, August 2009_
:Usage:
Start the 'acpi_drv' in your Genode environment. Do not start the 'pci_drv'
since this will be used as a slave of the 'acpi_drv'. You still must load the
'pci_drv' in your boot loader. To integrate the ACPI driver into your boot
configuration, you may take the following snippet as reference:
!<start name="acpi">
! <resource name="RAM" quantum="2M"/>
! <binary name="acpi_drv"/>
! <provides><service name="PCI"/></provides>
! <route>
! <service name="ROM"> <parent/> </service>
! <any-service> <any-child/> <parent/> </any-service>
! </route>
!</start>
:Limitations and known issues:
Currently there is no interface to set the interrupt mode for core's IRQ
sessions (e.g., level or edge triggered). This is required by Fiasco.OCs kernel
interface. We regard this as future work.
Platform support
################
Fiasco.OC microkernel
=====================
The support for the Fiasco.OC base platform is still lacking proper handling
for releasing resources such as kernel capabilities. Although this is a known
issue, we underestimated the reach of the problem when Genode's signal API is
used. Each emitted signal happens to consume one kernel capability within core,
ultimately leading to a resource outage when executing signal-intensive code
such as the packet-stream interface. The current release comes with an interim
solution. To remedy the acute problem, we extended the 'Capability_allocator'
class with the ability to register the global ID of a Genode capability so
that the ID gets associated with a process-local kernel capability. Whenever
a Genode capability gets unmarshalled from an IPC message, the
capability-allocator is asked, with the global ID as key, whether the
kernel-cap already exists. This significantly reduces the waste of
kernel-capability slots.
To circumvent problems of having one and the same ID for different kernel
objects, the following problems had to be solved:
* Replace pseudo IDs with unique ones from core's badge allocator
* When freeing a session object, free the global ID _after_ unmapping
the kernel object, otherwise the global ID might get re-used in some
process and the registry will find a valid but wrong capability
for the ID
Because core aggregates all capabilities of all different processes, its
capability registry needs much more memory compared to a regular process.
By parametrizing capability allocators differently for core and non-core
processes, the global memory overhead for capability registries is kept
at a reasonable level.
Please note that this solution is meant as an interim fix until we have
resolved the root of the problem, which is the proper tracking and releasing
of capability selectors.
Linux
=====
Linux is one of the two original base platforms of Genode. The original
intension behind supporting Linux besides a microkernel was to facilitate
portability of the API design and to have a convenient testing environment for
platform-independent code. Running Genode in the form of a bunch of plain Linux
processes has become an invaluable feature for our fast-paced development.
To our delight, we lately discovered that the use of running Genode on Linux
can actually go far beyond this original incentive. Apparently, on Linux, the
framework represents an equally powerful component framework as on the other
platforms. Hence, Genode has the potential to become an attractive option for
creating complex component-based user-level software on Linux.
For this intended use, however, the framework has to fulfill the following
additional requirements:
* Developers on Linux expect that their components integrate seamlessly with
their existing library infrastructure including all shared libraries
installed on Linux.
* The use of a custom tool chain is hard to justify to developers who regard
Genode merely as an application framework. Hence, a way to use the normal
tool chain as installed on the Linux host system is desired.
* Application developers are accustomed with using GDB for debugging and expect
that GDB can be attached to an arbitrary Genode program in an intuitive way.
Genode's original support for Linux as base platform did not meet those
expectations. Because Genode's libc would ultimately collide with the Linux
glibc, Genode is built with no glibc dependency at all. It talks to the kernel
directly using custom kernel bindings. In particular, Genode threads are created
directly via the 'clone()' system call and thread-local storage (TLS) is managed
in the same way as for the other base platforms. This has two implications.
First, because Genode's TLS mechanism is different than the Linux TLS
mechanism, Genode cannot be built with the normal host tool chain. This
compiler would generate code that would simply break on the first attempt to
use TLS. We solved this problem with our custom tool chain, which is configured
for Genode's needs. The second implication is that GDB is not able to handle
threads created differently than those created via the pthread library. Even
though GDB can be attached to each thread individually, the debugger would not
correctly handle a multi-threaded Genode process as a multi-threaded Linux
program. With regard to the use of Linux shared libraries, Genode used to
support a few special programs that used both the Genode API and Linux
libraries. Those programs (called hybrid Linux/Genode programs) were typically
pseudo device drivers that translate a Linux API to a Genode service. For
example, there exists a framebuffer service that uses libSDL as back end.
Because those programs were a rarity, the support by the build system for such
hybrid targets was rather poor.
Fortunately, all the problems outlined above could be remedied pretty easily.
It turns out that our custom libc is simply not relevant when Genode is
used as plain application framework on Linux. For this intended use, we always
want to use the host's normal libc. This way, the sole reason for using plain
system calls instead of the pthread library vanishes, which, in turn,
alleviates the need for a custom tool chain. Genode threads are then simply
pthreads compatible with the TLS code as emitted by the host compiler and
perfectly recognised by GDB. With the surprisingly little effort of creating a
new implementation of Genode's thread API to target pthreads instead of using
syscalls, we managed to provide a decent level of support for using Genode as
user-level component framework on Linux.
These technical changes alone, however, are not sufficient to make Genode
practical for real-world use. As stated above, the few hybrid Linux/Genode
programs used to be regarded as some leprous artifacts. When using Genode as
Linux application framework, however, this kind of programs are becoming the
norm rather than an exception. For this reason, we introduced new support for
such hybrid programs into the build system. By listing the 'lx_hybrid'
library in the 'LIBS' declaration of a target, this target magically becomes a
hybrid Linux/Genode program. It gets linked to the glibc and uses pthreads
instead of direct syscalls. Furthermore, host libraries can be linked to the
program by stating their respective names in the 'LX_LIBS' variable. For an
example, please refer to the libSDL-based framebuffer at
'os/src/drivers/framebuffer/sdl/target.mk'.
To enforce the build of all targets as hybrid Linux/Genode programs, the build
system features the 'alyways_hybrid' 'SPEC' value. To make it easy to create a
build directory with all targets forced to be hybrid, we have added the new
'lx_hybrid_x86' platform to the 'create_builddir' tool.
OKL4
====
:Sending an invalid-opcode exception IPC on OKL4:
When an invalid opcode gets executed, OKL4 switches to the kernel debugger
console instead of sending an exception IPC to the userland. We fixed this
problem by removing the code that invokes the debugger console from the kernel.
:Hard-wire OKL4 elfweaver to Python 2:
Recent Linux distributions use Python version 3 by default. But OKL4's
elfweaver is not compatible with this version of Python. For this reason, we
introduced a patch for pinning the Python version used by elfweaver to
version 2.
Both patches get automatically applied when preparing the 'base-okl4'
repository via 'make prepare'.
Build system and tools
######################
:Facility for explicitly building all libraries:
During its normal operation, the build system creates libraries as mere side
effects of building targets. There is no way to explicitly trigger the build of
libraries only. However, in some circumstances (for example for testing the
thorough build of all libraries), a mechanism for explicitly building libraries
would be convenient. Hence we introduced this feature in the form of the pseudo
target located at 'base/src/lib/target.mk'. By issuing 'make lib' in a build
directory, this target triggers the build of all libraries available for the
given platform.

1029
doc/release_notes/12-05.txt Normal file
View File

File diff suppressed because it is too large Load Diff

665
doc/release_notes/12-08.txt Normal file
View File

@@ -0,0 +1,665 @@
===============================================
Release notes for the Genode OS Framework 12.08
===============================================
Genode Labs
With Genode 12.08, the project focused on platform support. It enters the world
of OMAP4-based ARM platforms, revived and vastly enhanced the support for the
NOVA hypervisor, and becomes able to run directly on ARM platforms without the
need for an underlying kernel.
The new 'base-hw' platform is a deviation from Genode's traditional approach to
complement existing kernels with user-land infrastructure. It completely leaves
the separate kernel out of the picture and thereby dwarfs the base line of the
trusted computing base of Genode-based systems to approximately the half. The
new base platform is described in Section [Genode on naked ARM hardware].
Speaking of base platforms, we are happy to have promoted the NOVA hypervisor
to a first-class citizen among the base platforms. During the last months, this
kernel underwent fundamental changes regarding its mode of development and its
feature set. This prompted us to vastly improve Genode's support for this
platform and leverage its unique features. If considering the use of Genode on
x86-based hardware, NOVA has become a very attractive foundation. Section
[Embracing the NOVA Hypervisor] describes the NOVA-specific changes.
The improvement of platform support with the current release does not entail
the base platforms only but extends to profound additions of device drivers, in
particular for the ARM-based OMAP4 SoC as used on the popular Pandaboard. We
are proud to announce the availability of device drivers for HDMI output,
SD-card, USB HID, and networking for this platform.
Beyond the low-level platform improvements, the new version comes with several
new services, optimizations of existing components, and new ported libraries.
In particular, the Noux runtime has reached a point where we can principally
execute serious networking applications such as the Lynx web browser natively
on Genode. Another example is the new FFAT-based file-system service, which
makes persistent storage available via Genode's file-system interface. By
combining this new service with existing components such as the partition
service, Noux, or the file-system plugin of the libc, a lot of new application
scenarios become available. Thanks to these new components, the framework has
become able to perform on-target debugging via GDB running in Noux, or host
the genode.org website via the lighttpd web server,
:What about the road map?:
Those of you who track the milestones laid out in our [https://genode.org/about/road-map - road map]
may wonder how Genode 12.08 relates to the stated goals. In fact, several
points of the road map haven't received the attention as originally planned.
As an explanation, let us quote the paragraph right atop of the road-map page:
"The road map is not fixed. If there is commercial interest of pushing the
Genode technology to a certain direction, we are willing to revisit our plans."
Well, this is what happened. So we traded the work on the tiled window manager,
the Intel wireless driver, and SMP support for the work on the platform topics
outlined above. Nevertheless, we stick to our overall plan to turn Genode into
a general-purpose OS that is fit for use by its developers by the end of the
year. If looking closely at the additions that come with the current release,
it will become apparent how well they fit into the big picture.
Genode on naked ARM hardware
############################
One of Genode's most distinguishing properties is the ability to use the framework
on top of a range of different kernels. This way, users of the framework
benefit from the wide variety of features provided by those kernels while
only dealing with a single API and configuration concept. For example, we
frequently find ourselves using the Linux kernel as base platform while
developing services, interfaces, and protocol stacks. By being able to start
Genode as a regular program, we effectively eliminate the reboot-time for each
test run and enjoy using commodity debugging and profiling tools. On the other
hand, if high security is a concern, NOVA and Fiasco.OC provide
capability-based security at kernel-level. So the use of one of those kernels
is desirable. Genode allows for switching between those vastly different
kernels almost seamlessly.
In general, a Genode system consists of a kernel, Genode's core, and the
largely generic components on top of core. Core abstracts away the
peculiarities of the respective kernel and provides a unified API to the
components on top. From the application's point of view both kernel and core
are always at the root of the process tree and thereby are a inherent part of
the application's trusted computing base (TCB). The distinction of both
programs is almost superficial.
Since both the kernel and core must be ultimately trusted, the complexity of
both programs is critical for each Genode-based system. On our quest for
minimizing the TCB complexity so far, however, we did not question the role of
the kernel as an inherent part of the TCB and focused our attention to the
software stack on top. However, with more and more kernels entering the
picture, we identified that there is typically a considerable overlap in
functionality between kernel and core. For example, both need to know about
address spaces and their relationship to physical memory objects. Most kernels
keep track of memory mappings in an in-kernel database. Core also needs to keep
track of this information. Consequently, we found several information
replicated without a clear benefit. With this comes a seemingly significant
redundancy of code for data structures, allocators, and utility functions.
Furthermore, there exists a class of problems that must be solved by the kernel
and core alike. In particular the resource management of dynamically allocated
in-kernel objects respectively in-core objects. Whereas core uses Genode's
resource-trading concept to solve this problem, most kernels lack a good
solution for the management of in-kernel resources and are consequently prone
to resource exhaustion problems.
Out of these observations, the idea was born to explore the opportunities of
merging both programs into one and thereby eliminating the redundancies. Our
first attempt to go into this direction was the 'base-mb' platform, which
enabled us to run Genode on the Xilinx MicroBlaze softcore CPU. With this
experiment, we gained confidence that the approach is generally feasible. So we
took on the challenge to implement the idea of a hybrid kernel/core on a more
complex architecture namely ARM Cortex-A9.
The 'base-hw' platform introduced with the current release is the intermediate
result of our experiment. With this base platform, core plays the role of core
and the kernel within one program. A few code paths that require execution in
privileged mode are executed in kernel mode whereas most code paths are
executed in user mode. Both user mode code and kernel mode code run in the same
address space. The kernel portion merely provides a few basic mechanisms
without performing complex operations such as dynamic memory allocations. For
example, if core is requested to create a new thread via core's CPU session
interface, the user-level code within core allocates a KTCB (kernel thread
control block) and UTCB (user-level thread-control block) from the physical
memory allocator and passes both physical addresses to the kernel function that
spawns the actual thread. This way, we can employ Genode's resource-trading
concept for managing typical kernel resources.
The experiment turned out to be a great success. Traditionally, we would account
at least 10,000 lines of code (LOC) for the kernel. Most kernels are actually
much larger than that. Core comes at a complexity of another 10,000 LOC. So
both kernel and core make up a base line of TCB complexity of more than 20,000
LOC. By co-locating core with the kernel, we end up with a program of just
about 13,000 LOC. The vast reduction of TCB complexity compared to having
kernel and core as separate programs strikes us.
The 'base-hw' version of core supports the complete Genode API covering support
for user-level device drivers, synchronous RPCs, asynchronous notifications,
shared memory, and managed dataspaces. It is thereby able to execute the
sophisticated Genode scenarios on top including the GUI, the dynamic linker,
and user-level device drivers. That said, we regard the current version still
as work in progress. We successfully use it as an experimentation platform for
ongoing research activities (i.e., for exploring ARM TrustZone) but some
important features such as capability-based security are not yet implemented.
:Using the base-hw platform:
The new base platform is fully integrated with Genode's build system.
When listing the supported base platforms via the 'tool/create_builddir' tool,
you will see the new 'hw_panda_a2', 'hw_vea9x4', 'hw_pbxa9' choices of
build-directory templates. The latter platform enables you to run a
'base-hw' Genode system on Qemu.
[https://genode.org/documentation/platforms/hw - Learn more about using the new base-hw platform...]
For running Genode directly on the Pandaboard, please refer to the
[https://genode.org/documentation/platforms/hw_panda_a2 - Pandaboard-specific documentation...]
Embracing the NOVA Hypervisor
#############################
NOVA is a so-called microhypervisor for the x86 architecture. It combines the
principles of microkernels with capability-based security and hardware-assisted
virtualization. Among the various base platforms supported by Genode, NOVA's
kernel interface stands out for being extremely minimalistic and orthogonal,
even by microkernel standards.
Genode has supported NOVA as base platform since 2010. But because we used NOVA
solely for sporadic research activities and NOVA's lack of a regular release
schedule, the framework's platform support received only little attention. This
has changed now. NOVA's main developer Udo Steinberg moved from TU Dresden to
Intel Labs where he leads the development of NOVA as a true Open-Source
project. In fact, the code is now being hosted at GitHub:
:[https://github.com/IntelLabs/NOVA]:
NOVA hypervisor at GitHub
Since its move to GitHub, the hypervisor has already seen significant
improvements. The repository is continuously updated, which enables us to stay
in a close feedback loop with the NOVA developers. This change of how NOVA's
development is conducted ignited our renewed interest in promoting this
platform to a first-level citizen of our framework. The first noteworthy
improvement is the recently added 64-bit support of NOVA. We enabled Genode to
work with both variants of the kernel - 32 bit and 64 bit.
But this was just the first step. The second major change addresses the
allocation of kernel resources. Early versions of the hypervisor allowed each
process to create kernel objects and thereby indirectly consume the limited
memory resources of the kernel. This is perfectly fine for a research project
but it becomes a potential denial-of-service problem in real-world use cases.
For this reason, newer versions introduced the ability to retain the allocation
of kernel objects within a trusted component only. In the Genode world, this
component is naturally core. Even though NOVA still lacks a flexible concept for
kernel-resource management as of now, Genode has become able to use NOVA
without suffering the inherent resource management limitation. This is achieved
because core is able to arbitrate the allocation of kernel resources.
The third fundamental change is the abolishment of the last traces of global
names in a NOVA-based Genode system. There are no thread IDs, object IDs, or
any other kind of globally meaningful names. Each process has a local view on
(a small part of) the system only. If a process interacts with another process,
the kernel translates the references to remote objects from one namespace to
the other. The security implications are eminent. First, a process can only
interact with or refer to objects for which it has a name, which vastly reduces
problems of ambient authority. Second, because the kernel translates names, it
becomes impossible to forge object identities. If a process tried to pass a
forged object reference to another process, the translation would simply fail,
rendering the attack ineffective.
The described changes do not come without issues, though. To make the NOVA
kernel fit with Genode's requirements, minor patches of the hypervisor are
needed. The patches are located at 'base-nova/patches/'. However, those patches
are meant as interim solutions until we find mechanisms that fit well with the
design of the hypervisor and also fulfil our requirements.
So far, we greatly enjoyed the revived collaboration with the NOVA developers
and congratulate Udo Steinberg for the new mode of development of the
hypervisor.
Base framework
##############
In the following, we describe changes of the base API that may affect users of
the framework.
:Allocation of DMA buffers:
We extended the RAM session interface with the ability to allocate DMA buffers.
The client specifies the type of RAM dataspace to allocate via the new 'cached'
argument of the 'Ram_session::alloc()' function. By default, 'cached' is true,
which corresponds to the common case and the original behavior. When setting
'cached' to 'false', core takes the precautions needed to register the memory
as uncached in the page table of each process that has the dataspace attached.
Currently, the support for allocating DMA buffers is implemented for Fiasco.OC
only. On x86 platforms, it is generally not needed. But on platforms with more
relaxed cache coherence (such as ARM), user-level device drivers should always
use uncacheable memory for DMA transactions.
:MMIO framework improvements:
As we find ourselves increasingly using the 'Register' and 'Mmio' templates
provided by 'util/register.h' and 'util/mmio.h' for dealing with memory-mapped
devices, we extended the utilities with support for 64-bit registers and a new
interface for polling bit states. The latter functionality is provided by the
new 'wait_for' function template. To decouple the MMIO-related utility code
from an actual timer facility, the function takes a so-called 'delayer' functor
as argument. This way the user of the MMIO framework is able to pick a timer
facility that fits best with the device.
:New 'memcpy' implementation:
The memory-copy functions provided by 'util/string.h' are extremely simple
and arguably slow, particularly on platforms where byte-wise copy operations
are not supported by the CPU (i.e., ARM). Hence, we have added a CPU-specific
memcpy function ('memcpy_cpu') to 'cpu/string.h', which enables us to
provide optimized implementations. So far, we did so for the ARM architecture.
Low-level OS infrastructure
###########################
FFat-based file-system service
==============================
With the previous release, we introduced Genode's file-system interface
accompanied with a simple in-memory file-system service. With the addition of
'ffat_fs', the current release adds the first persistent file system to the
framework. The service is located at 'libports/src/server/ffat_fs'. It uses
Genode's 'Block::Session' interface as back end. Therefore, it can be combined
with any of Genode's block-device drivers and the partition service called
'part_blk'. To see the new 'ffat_fs' service in action, please refer to the new
'libports/run/libc_ffat_fs.run' script.
On the course of our work on the 'ffat_fs' service, we enabled support for long
file names in libffat and added 'lseek' support to the 'libc_ffat' plugin.
TAR-based file-system service
=============================
The new 'tar_fs' service located at 'os/src/server/tar_fs' provides a read-only
file-system session interface by reading data from a TAR archive, which, in
turn, is fetched from a ROM service. By combining 'tar_fs' with the 'libc_fs'
plugin, it becomes easy to provide customized pseudo file systems to individual
Genode programs. For example, one instance of 'tar_fs' containing a static
website and a web-server configuration can be provided as file system to a web
server. The configuration is similar to the patterns known from the 'tar_rom'
and 'ram_fs' servers:
! <config>
! <archive name="tar_archive.tar" />
! <policy label="label_of_client" root="/rootdir/for/client" />
! </config>
The policy node allows for assigning different parts of one TAR archive to
different clients. For a practical usage example of 'tar_fs', please refer to
the 'libports/run/libc_fs_tar_fs.run' script.
Terminal improvements
=====================
Our work on running a growing number of command-line-based Unix programs via
Noux prompted us to improve our terminal implementation as needed. To ease
debugging for terminal colors, we changed the previous default color scheme to
fully saturated combinations of red, green, and blue. Albeit this looks quite
painful on the eyes, it is easier to spot wrong colors when using a program
that uses ncurses, for example Lynx. Furthermore, we added the handling of
sgr0 and sgr escape sequences and thereby enabled Lynx to become almost
usable when running within Noux.
Terminal cross-link service
===========================
The 'Terminal::Session' interface gets increasingly popular within Genode.
It is used by the UART drivers, the graphical terminal, GDB monitor, the TCP
terminal, and Noux. For most of these programs, their respective client or
server role is quite clear but we find ourselves tempted to combine components
in unusual ways. For example, to let Noux run an instance of GDB, which operates
on a terminal via a virtual character device. For remote debugging, GDB plays
the role of a terminal client and the UART driver plays the role of the server.
But when running GDB monitor on the same machine, GDB monitor will also
expect to play the role of the client. In order to connect GDB monitor
to a local instance of GDB, both of them being terminal clients, we need an
adapter component. This is where the new terminal cross-link service enters
the picture. It plays the role of a terminal server between exactly two
clients. The output of one client ends up as input to the other and vice
versa. Data sent to the server gets stored in a buffer of 4096 bytes (one
buffer per client). As long as the data to be written fits into the buffer, the
'write()' call returns immediately. If no more data fits into the buffer, the
'write()' call blocks until the other client has consumed some of the data from
the buffer via the 'read()' call. The 'read()' call never blocks. A signal
receiver can be used to block until new data is ready for reading.
The new terminal crosslink can be tested via the 'os/run/terminal_crosslink.run'
script. It is also used for the just mentioned on-target debugging scenario
demonstrated by the 'ports/run/noux_gdb.run' script.
DMA-aware and optimized packet streams
======================================
Motivated by our work on OMAP4 platform support, we introduced API extensions
for handling of DMA buffers to the following interfaces:
:'Attached_ram_dataspace':
The convenience utility for allocating and locally mapping a RAM dataspace
has been enhanced with the 'cached' constructor argument, which is true
by default. When using 'Attached_ram_dataspace' for allocating DMA buffers,
this argument should be set to false.
:Block and network packet stream:
The 'Block::Session' and 'Nic::Session' interfaces use Genode's packet stream
facility for transferring bulk payload between processes. A packet stream
combines shared memory with asynchronous notifications and thereby facilitates
the use of batched packet processing. To principally enable zero-copy semantics
for device drivers, the packet-stream buffer is now explicitly allocated as DMA
buffer. This clears the way to let the SD-card driver direct DMA transactions
right into the packet stream buffer. Consequently, when attaching the SD-card
driver directly to a file system, there is no copy of payload needed.
The 'Nic::Session' interface has further been improved by using a fast
bitmap allocator for allocations within the packet-stream buffer. This is
possible because networking packets have the MTU size as an upper limit.
In contrast to the 'Block::Session' interface where requests are relatively
large, 'Nic::Session' packets are tiny, and thus, greatly benefit from the
optimized allocator.
Libraries and applications
##########################
C runtime
=========
:File I/O:
We complemented our C runtime with support for the 'pread', 'pwrite', 'readv',
and 'writev' functions. The 'pread' and 'pwrite' functions are shortcuts for
randomly accessing different parts of a file. Under the hood, the functions are
implemented via 'lseek' and 'read/write'. To provide the atomicity of the
functions, a lock guard prevents the parallel execution of either or both
functions if called concurrently by multiple threads. The 'readv' and 'writev'
functions principally enable the chaining of multiple I/O requests.
Furthermore, we added 'ftruncate', 'poll', and basic support for (read-only)
mmapped files to the C runtime.
:Libc RPC framework headers:
Certain RPC headers of the libc are needed for compiling 'getaddrinfo.c'.
Unfortunately that means we have to generate a few header files, which we do
when we prepare the libc.
New and updated 3rd-party libraries
===================================
:Expat:
[https://expat.sourceforge.net - Expat] is an XML parsing library. The port of
this library was motivated by our goal to use the GNU debugger for on-target
debugging. GDB depends on this library.
:MPC and GMP:
We complemented our existing port of the
[https://gmplib.org - GNU multiple precision arithmetic library (libgmp)] with
support for the x86_64 and ARM architectures. This change combined with the
port of the [http://www.multiprecision.org/index.php?prog=mpc - MPC library]
enables us to build the Genode tool chain for these architectures.
:OpenSSL:
Our port of OpenSSL has been updated to version 1.0.1c. Because libcrypto
provides certain optimized assembler functions, which unfortunately are not
expressed with position-independent code, we removed this assembler code and
build libcrypto with '-DOPENSSL_NO_ASM'. Because the assembler code is not
needed anymore, its generation is also removed from 'openssl.mk'.
:Light-weight IP stack (lwIP):
We enabled the lwIP TCP/IP stack for 64-bit machines and updated the library to
version 1.4.1-rc1. With the new version, the call of 'lwip_loopback_init' is
not needed anymore because lwIP always creates a loopback device. Hence, we
will be able to remove the 'libc_lwip_loopback' in the future. For now, we keep
it around so we currently do not need to update the other targets that depend
on it.
:PCRE:
[https://www.pcre.org/ - PCRE] is a library for parsing regular rexpressions. We
require this library for our ongoing work on porting the lighttpd webserver.
Lighttpd web server
===================
The [https://www.lighttpd.net/ - Lighttpd] web server has been added to the
'ports' repository. The port runs as a native Genode application and ultimately
clears the way to hosting the genode.org website on Genode. To test drive this
scenario, please give the 'ports/run/genode_org.run' script a try.
At the current stage, the port is still quite limited. For example, it does not
make use of non-blocking sockets yet. But the 'genode_org.run' run script
already showcases very well how simple a Genode-based web-server appliance can
look like.
Device drivers
##############
OMAP4 platform drivers
======================
:HDMI output:
The new HDMI driver at 'os/src/drivers/framebuffer/omap4' implements Genode's
'Framebuffer::Session' interface by using the HDMI output of OMAP4. The current
version sets up a fixed XGA screen mode of 1024x768 with the RGB565 pixel
format.
:SD-card:
The new SD card driver at 'os/src/drivers/sd_card/omap4' allows the use of a
HDSD card with the Pandaboard as block service. The driver can be tested using
the 'os/run/sd_card.run' script. Because it implements the generic
'Block::Session' interface, it can be combined with a variety of other
components such as 'part_blk' (for accessing individual partitions) or
'ffat_fs' for accessing a VFAT file system on the SD card.
The driver uses the master DMA facility of the OMAP4 SD-card controller, which
yields to good performance at low CPU utilization. The throughput matches (and
in some cases outperforms) the Linux kernel driver. In the current version,
both modes of operation PIO and DMA are functional. However, PIO mode is
retained for benchmarking purposes only and will possibly be removed to further
simplify the driver.
:USB HID:
The OMAP4-based Pandaboard relies on USB for attaching input devices.
Therefore, we need a complete USB stack to enable the interactive use of the
board. Instead of implementing a USB driver from scratch, we built upon the USB
driver introduced with the Genode release 12.05. This driver was ported from the
Linux kernel.
:Networking:
The Pandaboard realizes network connectivity via the SMSC95xx chip attached to
the USB controller. Therefore, we enhanced our USB driver with support for USB
net and the smsc95xx driver. In addition to enabling the actual device-driver
functionality, the USB stack has received much attention concerning performance
optimizations. To speed up the allocation of SKBs, we replaced the former
AVL-tree based allocator with a fast bitmap allocator. For anonymous
allocations, we introduced a slab-based allocator. Furthermore, we introduced
the distinction between memory objects that are subjected to DMA operations
from non-DMA memory objects. The most profound conceptual optimization is the
use of transmit bursts by the driver. The Linux kernel, which our driver
originates from, does not provide an API for transmitting multiple packets as a
burst. For our driver, however, this optimization opportunity opened up thanks
to Genode's packet stream interface, which naturally facilitates the batching
of networking packets. So the driver has all the information needed to create
burst transactions.
USB driver
==========
By testing our new USB driver on a variety of real PC hardware, we discovered
several shortcomings, which we resolved. In particular, we added support for
more than one UHCI controller, make sure that the 'PIRQ' bit in the legacy
support register (PCI config space) of the UHCI controller is enabled and that
the 'Trap on IRQ' bit is disabled.
With those modifications in place, the USB driver works reliably on the tested
platforms.
Runtime environments
####################
Noux
====
Noux enables the easy reuse of unmodified GNU software on Genode by providing
a Unix-like kernel interface as user-level service. Because Noux is pivotal for
our plan to use Genode for productive work, we significantly enhanced and
complemented its feature set.
:Noux on ARM and x86_64:
For keeping the scope of the development manageable, the initial version of
Noux was tied to the x86_32 platform. This was not a principal limitation of
the approach but rather an artificial restriction to keep us focused on
functionality first. Now that Noux reaches a usable state, we desire to use it
on platforms other than x86_32. The current release enables Noux for the 64-bit
x86 and ARM architectures.
The level of support is pretty far-reaching and even includes the building and
execution of the Genode tool chain on those platforms. In the process of
enabling these platforms, we updated the Noux package for GCC to version 4.6.1,
which matches the version of the current Genode tool chain.
:Terminal file system:
Noux supports the concept of stacked file systems. The virtual file system
is defined at the start of a Noux instance driven by the static Noux
configuration. This way, arbitrary directory structures can be composed out
of file-system sessions and TAR archives. The VFS concept allows for the
easy addition of new file system types. To allow programs running in a Noux
instance to communicate over a dedicated terminal session, we added a new
file-system type that corresponds to a virtual character device node attached
to a terminal session.
:GDB running in the Noux environment:
With the terminal file system in place, we are ready to execute GDB within
Noux and let it talk to a GDB monitor instance over the terminal session
interface. From GDB's point of view, the setup looks like a remote debugging
session. But in reality both the debugging target and GDB reside in different
subtrees of the same Genode system.
:Executing shell scripts:
By inspecting the program specified to the execve system call, Noux has become
able to spawn scripts that use the '#!' syntax. If such a file is detected, it
executes the specified interpreter instead and passes the arguments specified
after the '#!' marker, followed by command-line arguments.
:Networking support:
Our work on porting various networking tools to Noux triggers us to steadily
improve the networking support introduced with Genode 12.05. In particular, we
added proper support for DNS resolving, which enables us to execute the
command-line based Lynx web browser within Noux.
:User information:
Because there are certain programs, which need the information that is stored
in 'struct passwd', we introduced configurable user information support to
Noux. One can set the user information via the '<user>' node in the Noux
config:
! <config>
! <user name="baron" uid="1" gid="1">
! <shell name="/bin/bash" />
! <home name="/home" />
! </user>
! ...
! </config>
When '<user>' is not specified, default values are used. Currently these
are 'root', 0, 0, '/bin/bash', '/'. Note that this is just a single user
implementation because each Noux instance has only one user or rather one
identity and there will be no complete multi-user support in Noux. If you need
different users, just start new Noux instances for each of them.
:New '/dev/null' and '/dev/zero' pseudo devices:
These device are mandatory for most programs (well, at least null is required
to be present for a POSIX compliant OS, which Noux is actually not). But for
proper shell-script support we will need them anyway. Under the hood, both
pseudo devices are implemented as individual file-systems and facilitate Noux's
support for stacked file systems. The following example configuration snippet
creates the pseudo devices under the '/dev' directory.
! <config>
! <fstab>
! <dir name="dev" >
! <null /> <zero />
! </dir>
! ...
! <fstab>
! ...
! </config>
Vancouver
=========
The comprehensive rework of the NOVA base platform affected the Genode version
of the Vancouver virtual machine monitor as this program used to speak directly
to the NOVA kernel. Since no kernel objects can be created outside of core
anymore, the Vancouver port had to be adjusted to solely use Genode interfaces.
L4Linux
=======
To improve the stability and performance of L4Linux on OMAP4 platforms, we
reworked parts of the Genode-specific stub drivers, in particular the
networking code. Among the improvements are the use of a high-performance
allocator for networking packets, improved IRQ safety of IPC calls (to
the Genode world), and tweaks of the TCP rmem and wmem buffer sizes to
achieve good TCP performance when running Linux with little memory.
Furthermore, we added two ready-to-use run scripts residing within
'ports-foc/run' as examples for executing L4Linux on the OMAP4-based
Pandaboard. The 'linux_panda.run' script is meant as a blue print for
experimentation. It integrates one instance of L4Linux with the native SD-card
driver, the HDMI driver, and the USB HID input driver. The
'two_linux_panda.run' script is a more elaborative example that executes two
instances of L4Linux, a block-device test, and a simple web server. Each of
the L4Linux instances accesses a different SD-card partition whereas the
block-device test operates on a third partition.

735
doc/release_notes/12-11.txt Normal file
View File

@@ -0,0 +1,735 @@
===============================================
Release notes for the Genode OS Framework 12.11
===============================================
Genode Labs
The central theme of version 12.11 of the Genode OS Framework is
self-hosting Genode on Genode. With self-hosting, we understand the execution of the
entire Genode build system within the Genode environment. There are two motivations
for pursing this line of work. First, it is a fundamental prerequisite for the
Genode developers to move towards using Genode as a day-to-day OS. Of course,
this prerequisite could be realized using one of the available virtualization
solutions. For example, we could run L4Linux on top of Genode on the Fiasco.OC
kernel and use the Genode build system from within an L4Linux instance.
However, this defeats the primary incentive behind Genode to reduce system
complexity. By having both Genode and L4Linux in the picture, we would indeed
increase the overall complexity in configuring, maintaining, and using the
system. Therefore, we would largely prefer to remove the complex Linux user
land from the picture. The second motivation is to prove that the framework and
underlying base platforms are suited and stable enough for real-world use.
If the system is not able to handle a workload like the build system,
there is little point in arguing about the added value of having a
microkernel-based system over current commodity OSes such as GNU/Linux.
We are happy to have reached the state where we can execute the unmodified
Genode build system directly on Genode running on a microkernel. As the
build system is based on GNU utilities and the GNU compiler collection,
significant effort went into the glue between those tools and the Genode API.
Section [Building Genode on Genode] provides insights into the way we achieved
the goal and the current state of affairs.
Along with the work on bringing the build system to Genode came numerous
stability improvements and optimizations all over the place, reaching from the
respective kernels, over the C runtime, the file-system implementations, memory
allocators, up to the actual programs the tool chain is composed of. Speaking
of the tool chain, the official Genode tool chain has been updated from GCC
version 4.6.1 to version 4.7.2. Thereby, all 3rd-party code packages were
subjected to testing and fixing activities.
For running the build system, the project currently focuses on NOVA and
Fiasco.OC as base platforms. However, our custom kernel platform for the ARM
architecture has also received significant improvements. With added support for
Freescale i.MX and Texas Instruments OMAP4, this platform proved to be very
well adaptable to new SoCs whereas new cache handling brings welcome
performance improvements. Furthermore, we have added experimental support for
ARM TrustZone technology, which principally enables the execution of Genode in
the so-called secure world of TrustZone while executing Linux in the so-called
normal world.
As we discovered the increasing interest in using Genode as a middleware
solution on Linux, we largely revisited the support for this kernel platform
and discovered amazing new ways to align the concept of Genode with the
mechanisms provided by the Linux kernel. Section [Linux] provides a summary
of the new approaches taken for supporting this platform.
Functionality-wise, the new version introduces support for audio drivers of
the Open Sound System, a new OMAP4 GPIO driver, improvements of the graphical
terminal, and the initial port of an SSH client.
Building Genode on Genode
#########################
On the Genode developer's way towards using Genode as a day-to-day OS, the
ability to execute the Genode build system within the Genode environment is a
pivotal step - a step that is highly challenging because the build system is
based on the tight interplay of many GNU programs. Among those
programs are GNU make, coreutils, findutils, binutils, gcc, and bash. Even
though there is a large track record of individual programs and libraries ported
to the environment, those programs used to be self-sustaining applications that
require only little interaction with other programs. In contrast, the build
system relies on many utilities working together using mechanisms such as
files, pipes, output redirection, and execve. The Genode base system does not
come with any of those mechanisms let alone the subtle semantics of the POSIX
interface as expected by those utilities. Being true to microkernel principles,
Genode's API has a far lower abstraction level and is much more rigid in scope.
To fill the gap between the requirements of the build system and the bare
Genode mechanisms, the Noux runtime environment was created. Noux is a Genode
process that acts like a Unix kernel. When started, it creates a child process,
which plays a similar role as the init process of Unix. This process communicates
via RPC messages to Noux. Using those messages, the process can perform all the
operations normally provided by a classical Unix kernel. When executed under
Noux, a process can even invoke functionalities such as fork and execve, which
would normally contradict with Genode's principles of resource management.
Over the course of the past year, more and more programs have been ported to
the Noux environment. Thereby, the semantics provided by Noux have been
successively refined so that those program behave as expected. This was an
iterative process. For example, at the beginning, Noux did not consider the
differences between 'lstat' and 'stat' as they did not matter for the first
batch of GNU programs ported to Noux. As soon as the programs got more
sophisticated, such shortcuts had to be replaced by the correct semantics. The
Genode build system is by far the most complex scenario exposed to Noux so far.
It revealed many shortcomings by both functionality implemented in Noux or the
C runtime as well as the underlying base platforms. So it proved to be a great
testing ground for analysing and improving those platform details. Therefore,
the secondary effects of self-hosting Genode on Genode in terms of stability
turned out to be extremely valuable.
The release comes with two ready-to-use run scripts for building bootable
system images that are able to execute the Genode tool chain, one for targeting
NOVA and one for targeting Fiasco.OC. Those run scripts are located at
'ports/run/' and called 'noux_tool_chain_nova.run' and 'noux_tool_chain_foc.run'
respectively. Each of those run scripts can be executed on either of those base
platforms. For example, by executing 'noux_tool_chain_nova' on Fiasco.OC, the
image will run Genode on Fiasco.OC and the tool chain will build binaries for
NOVA. When started, a build directory will be created at '/home/build'.
The Genode source code is located at '/genode'. In the '/bin' directory,
there are all the GNU programs needed to execute the tool chain. For
taking a look into the source code, 'vim' is available. To build core,
change to the build directory '/home/build' and issue 'make core'.
On Fiasco.OC, the complete Genode demo scenario can be compiled. On NOVA, the
incomplete life-time management of kernel objects will still result in an
out-of-memory error of the kernel. This kernel issue is currently being worked
on. Executing the tool chain on either of those platforms is still relatively
slow as extensive trace output is being generated and no actions have been taken to
optimize the performance so far. There are many opportunities for such
optimizations, which will be taken on as the next step.
Base framework
##############
Genode's base framework has received new support for extending session
interfaces and gained improvements with regard to interrupt handling on the x86
platform. At the API level, there are minor changes related to the CPU session
and 'Range_allocator' interfaces.
Support for specializing session interfaces
===========================================
With increasingly sophisticated application scenarios comes the desire to
extend Genode's existing session interface with new functionality. For example,
the 'Terminal::Session' interface covers plain read and write operations. It is
implemented by services such as a graphical terminal, the telnet-like TCP
terminal, or UART drivers. However, for the latter category, the breadth of the
interface is severely limited as UART drivers tend to supplement the read / write
interface with additional control functions, e.g., for setting the baud rate.
One way to go would be to extend the existing 'Terminal::Session' interface
with those control functions. However, these functions would be meaningless for
most implementations. Some of those other implementations may even desire their
own share of additions. In the longer term, this approach might successively
broaden the interface and each implementation will cover a subset only.
Because Genode aspires to keep interfaces as low-complex as possible while, at
the same time, it wants to accommodate the growing sophistication of usage
scenarios, we need a solution that scales. The solution turns out to be
strikingly simple. The RPC framework already supports the inheritance of RPC
interfaces. So it is possible to model the problem such that a new
'Uart::Session' interface derived from the existing 'Terminal::Session' will
be the host of UART-specific functionality. The only piece missing is the
propagation of both 'Uart' and 'Terminal' through the parent interface while
announcing the service. To spare the work of manually announcing the chain of
inherited interfaces from the implementor, the 'Parent::announce()' function has
been enhanced to automatically announce all service types implemented by the
announced interface. This way, a UART driver will always announce a "Uart"
and a "Terminal" service.
Improved interrupt handling
===========================
To accommodate modern x86 platforms, the session arguments of core's IRQ
service have been supplemented with the IRQ mode. There are two degrees of
freedom, namely the trigger (level / edge) and polarity (high / low). Thanks to
this addition, device drivers have become able to supply their knowledge of
devices to core.
In system scenarios with many peripherals, in particular when using the USB
driver, IRQ lines are shared between devices. Until now, Genode supported
shared interrupts for the OKL4 base platform only. To also cover the other
x86 kernels, we have generalized the interrupt sharing code and enabled this
feature on Fiasco.OC and NOVA.
Revised CPU session interface
=============================
We revisited the CPU session interface, removed no-longer used functions and
added support for assigning threads to CPUs.
The original CPU session interface contained functions for iterating through
the threads of a session. This interface was originally motivated by an
experimental statistical profiling tool that was developed at an early stage of
Genode. In the meanwhile, we discovered that the virtualization of the CPU
session interface is much more elegant to cover this use case than the
thread-iterator interface. Because the iteration has no transactional
semantics, it was unsafe to use it anyway.
To enable the use of multiple CPUs on multi-processor systems, the CPU
session interface has been enhanced with two functions, namely 'affinity'
and 'num_cpus'. The interface extension principally allows the assignment of
individual threads to CPUs. It is currently implemented on Fiasco.OC only.
On all other base platforms, 'num_cpus' returns one CPU. Note that on
the Linux platform, multiple CPUs will be used transparently.
The 'Cpu_session::state' function has been split into two functions, one
for retrieving information and one for propagating state information. The
prior interface was less explicit about the semantics of the 'state' function
as it took a non-const pointer to a 'Thread_state' object as argument.
Platform-tailored protection domains
====================================
Genode tries to provide a uniform API across all the different base platforms.
Yet, it also strives to make genuine platform features available to the
users of the framework. Examples for such features are the virtualization
support of the NOVA hypervisor or the special support for paravirtualizing
Linux on Fiasco.OC. Another example is the security model as found on the Linux
platform. Even though the security mechanisms of plain Linux are not as strong
as Genode's capability concept on a conceptual level, we still want to leverage
the available facilities such as user IDs and chroot as far as possible.
Consequently, we need a way to assign platform-specific properties to PD
sessions. With the new 'Native_pd_args' type introduced into
'base/native_types.h', there is now a way to express those platform-specific
concerns. This type is now used at all the places that deal with the creation
of protection domains such as 'Process', 'Child', and the loader.
Revised 'Range_allocator' interface
===================================
The handling of allocation errors has been refined in order to distinguish
different error conditions, in particular out-of-metadata and out-of-memory
conditions. The user of the allocator might want to handle both cases
differently. Hence we return an 'Alloc_return' value as result. In prior
versions, this type was just an enum value. With the new version, the type has
been changed to a class. This makes the differentiation of error conditions at
the caller side more robust because, in contrast to enum values, typed objects
don't get implicitly converted to bool values.
Low-level OS infrastructure
###########################
New UART session interface
==========================
To accommodate UART specific extensions of the 'Terminal::Session' interface,
in particular setting the baud rate, we introduced the new 'Uart::Session'
interface and changed the existing UART drivers to implement this
interface instead of the 'Terminal::Session' interface. Because 'Uart::Session'
inherits the 'Terminal::Session' interface, 'Uart' services announce both
"Uart" and "Terminal" at their parent.
New GPIO session interface
==========================
Embedded SoCs such as OMAP4 provide many general-purpose I/O pins, which can be
used for different purposes depending on the board where they are soldered on.
For example, the Pandaboard uses such GPIO pins to detect the presence of a
HDMI plug or control the power supply for the USB. If only one driver deals
with GPIO pins, the GPIO programming can reside in the driver. However, if
multiple drivers are used, the GPIO device resources cannot be handed out to
more than one driver. This scenario calls for the creation of a GPIO driver as
a separate component, which intermediates (and potentially multiplexes) the
access to the physical GPIO pins. The new 'Gpio::Session' interface allows one
or multiple clients to configure I/O pins, request states, as well as to
register for events happening on the pins.
Terminal
========
The graphical terminal has been enhanced with support for different built-in
font sizes and background-color handling.
In addition to those functional changes, the implementation has been decomposed
into several parts that thereby became reusable. Those parts comprise the
handling of key mappings, decoding the VT character stream, and the handling of
the character array. These functionalities are now available at
'gems/include/terminal'.
Libraries and applications
##########################
C runtime
=========
:Allocator optimized for small-object allocations:
To optimize the performance of workloads that depend on a large number of small
dynamic memory allocations, in particular the lwIP TCP/IP stack, we replaced
the memory allocator of the libc with a more sophisticated strategy. Until now,
the libc used 'Genode::Heap' as allocator. This implementation is an
AVL-tree-based best-fit allocator that is optimized for low code complexity
rather than performance for small allocations. The observation of the allocator
usage pattern of lwIP prompted us to replace the original libc malloc/free with
a version that uses slab allocators for small objects and relies on the
'Genode::Heap' for large objects only.
:Symbolic links:
Because part of our ongoing refinements of the Noux runtime is the provision of
symbolic links, support for symbolic links was added in the libc, libc plugins,
and file system servers.
lwIP
====
We updated the light-weight IP stack to version STABLE-1.4.1. Additionally,
the following optimizations were conducted to improve its performance and
robustness.
We reduced the maximum segment lifetime from one minute to one second to avoid
queuing up PCBs in TIME-WAIT state. This is the state, PCBs end up after
closing a TCP connection socket at the server side. The number of PCBs in this
state is apparently not limited by the value of 'MEMP_NUM_TCP_PCB'. One
allocation costs around 160 bytes. If clients connect to the server at a high
rate, those allocations accumulate quickly and thereby may exhaust the memory
of the server. By reducing the segment lifetime, PCBs in TIME-WAIT state are
cleaned up from the 'tcp_tw_pcbs' queue in a more timely fashion (by
'tcp_slowtmr()').
To prevent the TCP/IP stack from artificially throttling TCP throughput,
we adjusted lwIP's TCP_SND_BUF size.
From our work on optimizing the NIC stub-code performance of L4Linux as
described [https://genode.org/documentation/articles/pandaboard - here],
we learned that the use of a NIC-specific packet allocator for the
packet-stream interface is beneficial. At the lwIP back end, we still relied on
the original general-purpose allocator. Hence, we improved the lwIP back-end
code by using the bitmap-based 'Nic::Packet_allocator' allocator instead.
Standard C++ library
====================
Genode used to rely on the standard C++ library that comes with the tool chain.
However, this mechanism was prone to inconsistencies of the types defined in
the header files used at compile time of the tool chain and the types provided
by our libc. By building the C++ standard library as part of the Genode build
process, such inconsistencies cannot happen anymore. The current version of the
C++ standard library corresponds to GCC 4.7.2.
Note that the patch changes the meaning of the 'stdcxx' library for users that
happened to rely on 'stdcxx' for hybrid Linux/Genode applications. For such
uses, the original mechanism is still available, in the renamed form of
'toolchain_stdcxx'.
Device drivers
##############
Open Sound System
=================
Genode tries to re-use existing device drivers as much as possible using an
approach called device-driver environment (DDE). A DDE is a library that
emulates the environment of the original driver by translating device accesses
to the Genode API. There are many success stories of drivers successfully ported
to the framework this way. For example, using DDE-Linux, we are able to use the
Linux USB stack. Using DDE-ipxe, we are able to use iPXE networking drivers.
With Genode 12.11 we extend our arsenal of DDEs with DDE-OSS, which is a
device-driver environment for the audio drivers of the Open Sound System (OSS).
:Website of the Open Sound System:
[http://www.4front-tech.com]
The new 'dde_oss' contains all the pieces needed to use Intel HDA, AC97, and
ES1370 audio cards on Genode. On first use, the 3rd-party code can be
downloaded by issuing 'make prepare' from within the 'dde_oss' source-code
repository. Also, you need to make sure to add the 'dde_oss' repository to your
'REPOSITORIES' variable in 'etc/build.conf'.
An OSS demo configuration can be found under 'run/oss.run' and can be started
via 'make run/oss' from a Genode build directory. Be sure to adjust the
'filename' tag of the 'audio0' program. The file has to reside under
'<build-dir>/bin/'. The file format is header-less two-channel float-32 at
44100 Hz. You may use the 'sox' utility to create these audio files:
! sox -c 2 -r 44100 foo.mp3 foo.f32
OMAP4 GPIO driver
=================
The new OMAP4 GPIO driver is the first implementation of the just introduced
'Gpio::Session' interface. The driver supports two ways of interacting
with GPIO pins, by providing a static configuration, or by interacting with a
session interface at runtime. An example for a static configuration looks as
follows:
! <config>
! <gpio num="121" mode="I"/>
! <gpio num="7" mode="O" value="0"/>
! <gpio num="8" mode="O" value="0"/>
! </config>
The driver is located at 'os/src/drivers/gpio/omap4'. As reference for using
the driver, please refer to the 'os/run/gpio_drv.run' script.
Thanks to Ivan Loskutov of Ksys-Labs for contributing the session interface
and the driver!
iPXE networking drivers
=======================
We updated our device-driver environment for iPXE networking drivers to a
recent git revision and enabled support for the x86_64 architecture.
Currently, the driver covers Intel gigabit ethernet (e1000, e1000e, igb),
Intel eepro100, and Realtek 8139/8169.
Runtime environments
####################
Noux
====
The Noux runtime environment has received plenty of love thanks to the
aspiration to execute the Genode build system.
:Time:
The build system uses GNU make, which depends on time stamps of files. We do
not necessarily need a real clock. A monotonic increasing virtual time is
enough. To provide such a virtual time, the libc was enhanced with basic
support for functions like 'gettimeofday', 'clock_gettime', and 'utimes'. As
there is currently no interface to obtain the real-world time in Genode, Noux
simulates a pseudo real-time clock using a jiffies-counting thread. This
limited degree of support for time is apparently sufficient to trick tools like
ping, find, and make into working as desired.
:Improved networking support:
The Noux/net version of Noux extends the Noux runtime with the BSD-socket
interface by using the lwIP stack. This version of Noux multiplexes the
BSD-socket interface of lwIP to multiple Noux programs, each having a different
socket-descriptor name space and the principal ability to use blocking calls
such as 'select'. The code for multiplexing the lwIP stack among multiple Noux
processes has been improved to cover corner cases exposed by sophisticated
network clients, i.e., openssh.
:Directory cache for the TAR file system:
The original version of the TAR file system required a search in all TAR
records for each file lookup. This takes a long time when composing a large
directory tree out of multiple TAR archives stacked together. This is the case
for the Genode build-system scenario where we have all the files of the GNU
tools as well as the Genode source tree. Searching through thousands of records
for each call of 'stat' quickly becomes a scalability issue. Therefore, we
introduced a TAR indexing mechanism that scans each TAR file only once at the
startup of Noux and generates a tree structure representing the directory
layout. Looking up files using this index is quick.
:New packages:
With Genode-12.11, new 3rd-party packages have become available, namely
OpenSSH, the 'which' command, and all tool-chain components in their current
version. OpenSSH is still at an experimental stage. The run script at
'ports/run/noux_net_openssh_interactive.run' demonstrates how SSH can be used
to login into a remote machine.
:New pseudo file systems:
The new 'stdio' and 'random' file systems are intended to represent the pseudo
devices '/dev/random' and '/dev/tty' on Noux. Both are needed to run OpenSSH.
Note that the 'Arc4random' class, on which the random file system is based on,
currently _does not collect enough_ random bytes! It should not be used for
security-critical applications.
L4Linux
=======
The paravirtualized L4Linux kernel for the Fiasco.OC platform was updated to
SVN revision 25, which matches the Fiasco.OC SVN revision 40. We further
improved the integration of L4Linux with Genode by optimizing the stub drivers
for block devices and networking, and added principal support for running
L4Linux on SMP platforms.
Platforms
#########
NOVA
====
Genode follows the steady development of the NOVA microhypervisor very closely.
The kernel used by the framework corresponds to the current state of the master
branch of IntelLabs/NOVA.
:Improvements towards GDB support:
The NOVA-specific implementation of the CPU session interface has been improved
to accommodate the requirements posed by GDB. In particular, the 'pause',
'resume', 'state', and 'single_step' functions have been implemented. Those
functions can be used to manipulate the execution and register state of
threads. Under the hood, NOVA's 'recall' feature is used to implement these
mechanisms. By issuing a 'recall' for a given thread, the targeted thread is
forced into an exception. In the exception, the current state of the thread can
be obtained and its execution can be halted/paused.
:Maximizing contiguous virtual space:
To enable the Vancouver virtual machine monitor to hand out large amounts of
guest memory, we optimized core's virtual address space to retain large and
naturally aligned contiguous memory regions. For non-core processes, the
thread-context area that contains the stacks of Genode threads has been moved
to the end of the available virtual address space.
:Life-time management of kernel resources:
We improved the life-time management of kernel resources, in particular
capabilities, within Genode. Still the management of such kernel resources
is not on par with the Fiasco.OC version, partially because of missing
kernel functionality. This is an ongoing topic that is being worked on.
:Using the BIOS data area (BDA) to get serial I/O ports on x86:
If the I/O ports for the comport are non default (default is 0x3f8), we had to
specify manually the correct I/O ports in the source code. To avoid the need
for source-code modifications when changing test machines, we changed the core
console to read the BDA and use the first serial interface that is available.
If no serial interface is available, no device configuration will be
undertaken. The BDA can be populated via a multi-boot chain loader. Bender is
such a chain loader that can detect serial ports accessible via PCI and writes
the I/O ports to the Bios Data area (BDA). These values get then picked up by
core.
Fiasco.OC
=========
The Fiasco.OC kernel has been updated to the SVN revision 40. The update improves
SMP support and comes with various bug fixes. There is no noteworthy change
with regard to the kernel interface. We extended the number of supported
Fiasco.OC-based platforms for Genode by including the Freescale i.MX53.
To enable the use of multiple CPUs by Genode processes, the CPU session
interface has been enhanced to support configuring the affinity of threads with
CPUs. We changed the default kernel configuration for x86 and ARM to
enable SMP support and adapted L4Linux to use the new interface.
Execution on bare hardware (base-hw)
====================================
The development of our custom platform for executing Genode directly on bare
hardware with no kernel underneath went full steam ahead during the release
cycle.
:Pandaboard:
The in-kernel drivers needed to accommodate the Pandaboard, more specifically
the timer and interrupt controller, are now supported. So the Pandaboard can be
used with both 'base-hw' and 'base-foc'. Also, the higher-level platform
drivers for USB, HDMI, and SD-card that were introduced with the previous
release, are equally functional on both platforms.
:Freescale i.MX31:
We added principal support for the Freescale i.MX line of SoCs taking the
ARMv6-based i.MX31 as starting point. As of now, the degree of support is
limited to the devices needed by the kernel to operate. Pure software-based
scenarios are able to work, i.e., the nested init run script executes
successfully.
:TrustZone support:
The new VM session interface of core provides a way to execute software
in the normal world of a TrustZone system whereas Genode runs in the secure
world. From Genode's point of view, the normal world looks like a virtual
machine. Each time, the normal world produces a fault or issues a secure
monitor call, control gets transferred to the virtual machine monitor, which is
a normal user-level Genode process. The base-hw kernel has been enhanced to
perform world switches between the secure and normal world and with the ability
to handle fast interrupts (FIQs) in addition to normal interrupts. The latter
extension is needed to assign a subset of devices to either of both worlds.
Currently, the only TrustZone capable platform is the ARM CoreTile Express
CA9x4 for the Versatile Express board. For a virtual machine working properly
on top, some platform resources must be reserved. Therefore, there exist two
flavours of this platform now, one with the 'trustzone' spec-variable enabled
and one without. If 'trustzone' is specified, most platform resources (DDR-RAM,
and most IRQs) are reserved for the normal world and not available to the
secure Genode world.
:Memory attributes and caching:
We successively activated various levels of caching and improved the handling
of caching attributes propagated into the page tables. These changes resulted
in a significant boost in performance on non-emulated platforms.
Linux
=====
The Linux version of Genode was originally meant as a vehicle for rapid
development. It allows the framework components including core to be executed
as plain Linux processes. But in contrast to normal Linux programs, which
use the glibc, Genode's components interact with the kernel directly without
any C runtime other than what comes with Genode. We use the Linux version on a
regular basis to implement platform-agnostic functionality and protocols. Most
of Genode's code (except for device drivers) falls in this category. Because
the Linux version was meant as a mere tool, however, we haven't put much
thought into the principle way to implementing Genode's security concept on
this platform. Threads used to communicate over globally accessible Unix-domain
sockets and memory objects were represented as globally accessible files within
'/tmp'.
That said, even though Linux was not meant as a primary platform for Genode in
the first place, Genode can bring additional value to Linux. When considering
the implementation of a component-based system on Linux, there are several
possible approaches to take. For example, components may use DBus to
communicate, or components could pick from the manifold Unix mechanisms such as
named pipes, files, sysv-shared memory, signals, and others. Unfortunately
those mechanisms are not orthogonal and most of them live in the global name
space of the virtual file system. Whereas those mechanisms are principally able
to let processes communicate, questions about how processes get to know each
other, access-control policy, synchronization of the startup of processes are
left to the developer.
Genode, on the other hand, does provide an API for letting components
communicate but also answers those tricky questions concerning the composition
of components. This makes Genode an interesting option to build component based
applications, even on Linux. However, when used in such a context, the
limitations of the original Linux support need resolutions. Therefore, the
current release comes with a largely revised platform support for the Linux
base platform.
The changes can be summarized as follows:
:Using file descriptors as communication addresses:
Genode's synchronous RPC framework was using Unix domain sockets. Each RPC
entrypoint was represented by a pair of named files, one for sending and one
for receiving messages. In the new version, inter-process communication is
performed via file descriptors only.
:Transfer of communication rights via RPC only:
Capabilities used to be represented as a pair of the destination thread ID and
a global object ID. The thread ID has been replaced by a file descriptor that
points to the corresponding RPC entrypoint. When capabilities are transferred
as RPC arguments, those file descriptors are transferred via SCM rights
messages. This is in line with Genode's way of capability-based delegation of
access rights.
:Core-only creation of communication channels:
Communication channels used to be created locally by each process. The naming
of those channels was a mere convention. In contrast, now, communication
channels are created by core only and do not reside on the Linux virtual file
system. When creating an RPC entrypoint, core creates a socket pair and hands
out both ends to the creator of the entrypoint.
:Restricted access to memory objects:
Access to dataspace content was performed by mmap'ing a file. For a given
dataspace, the file name could be requested at core via a Linux-specific RPC
call. Now, core holds the file descriptors of all dataspaces, which are
actually unlinked files. A process that is in possession of a dataspace
capability can request the file descriptor for the content from core and mmap
the file locally. This way, access to memory objects is subjected to the
delegation of dataspace capabilities.
:Core-local process creation:
Genode used to create new processes by directly forking from the respective
Genode parent using the process library. The forking process created a PD
session at core merely for propagating the PID of the new process into core
(for later destruction). This traditional mechanism has the following
disadvantages:
First, the PID reported by the creating process to core cannot easily be
validated by core. Therefore core has to trust the PD client to not specify a
PID of an existing process, which would happen to be killed once the PD session
gets destructed. Second, there is no way for a Genode process to detect the
failure of any of its grandchildren. The immediate parent of a faulting process
could use the SIGCHLD-and-waitpid mechanism to observe its children but this
mechanism does not work transitively.
By performing the process creation exclusively within core, all Genode
processes become immediate child processes of core. Hence, core can respond to
failures of any of those processes and reflect such conditions via core's
session interfaces. Furthermore, the PID associated to a PD session is locally
known within core and cannot be forged anymore. In fact, there is actually no
need at all to make processes aware of any PIDs of other processes.
:Handling of chroot, user IDs, and group IDs:
With the move of the process creation into core, the original chroot trampoline
mechanism implemented in 'os/src/app/chroot' does not work anymore. A process
could simply escape the chroot environment by spawning a new process via core's
PD service. Therefore, chroot support has been integrated into core and the
chroot policy becomes a mandatory part of the process creation. For each process
created by core, core checks for a 'root' argument of the PD session. If a path
is present, core takes the precautions needed to execute the new process in the
specified chroot environment.
This conceptual change implies minor changes with respect to the Genode API and
the configuration of the init process. The API changes are the enhancement of
the 'Genode::Child' and 'Genode::Process' constructors to take the root path as
argument. Init supports the specification of a chroot per process by specifying
the new 'root' attribute to the '<start>' node of the process. In line with
these changes, the 'Loader::Session::start' function has been enhanced with the
additional (optional) PD argument.
In line with how the chroot path can be propagated into core, core has become
able to assign customized UIDs and GIDs to individual Genode processes or whole
Genode subsystems. The new 'base-linux/run/lx_uid.run' script contains an
example of how to use the feature.
Build system and tools
######################
The current release comes with a new tool chain based on GCC 4.7.2 and binutils
2.22. The tool-chain upgrade involved adapting the Genode code base and fixing
various issues in 3rd-party software. To obtain the new tool chain, please
refer to the tool-chain website:
:Genode tool chain:
[https://genode.org/download/tool-chain]

941
doc/release_notes/13-02.txt Normal file
View File

@@ -0,0 +1,941 @@
===============================================
Release notes for the Genode OS Framework 13.02
===============================================
Genode Labs
Traditionally, the February release of Genode is focused on platform support.
The version 13.02 follows this tradition by vastly improving Genode for the
NOVA base platform and the extending the range of ARM SoCs supported by
both our custom kernel platform and the Fiasco.OC kernel.
The NOVA-specific improvements concern three major topics, namely the added
support for running dynamic workloads on this kernel, the use of IOMMUs, and
the profound integration of the Vancouver virtual machine monitor with the
Genode environment. The latter point is particularly exciting to us because
this substantial work is the first contribution by Intel Labs to the Genode
code base. Thanks to Udo Steinberg and Markus Partheymüller for making that
possible.
Beyond the x86 architecture, the new version comes with principal support for
the ARM Cortex-A15-based Exynos 5250 SoC and the Freescale i.MX53 SoC. The
Exynos 5250 SoC has been enabled for our custom kernel as well as for the
Fiasco.OC kernel. The most significant functional improvements are a new
facility to detect faulting processes and a new mechanism for file-system
notifications.
Besides those added functionalities, the release cycle was taken as an
opportunity to revisit several aspects under the hood of the framework. A few
examples are the reworked synchronization primitives, the simplified base
library structure, the completely redesigned audio-output interface, and a
modernized timer interface.
DMA protection via IOMMU
########################
Direct memory access (DMA) of devices is universally considered as the Achilles
heel of microkernel-based operating systems. The most compelling argument in
favour of using microkernels is that by encapsulating each system component
within a dedicated user-level address space, the system as a whole becomes more
robust and secure compared to a monolithic operating-system kernel. In the
event that one component fails due to a bug or an attack, other components
remain unaffected. The prime example for such buggy components are device
drivers. By empirical evidence, those remain the most prominent trouble makers
in today's operating systems. Unfortunately however, most commodity hardware
used to render this nice argumentation moot because it left one giant loophole
open, namely bus-master DMA.
Via bus-master DMA, a device attached to the system bus is able to directly
access the RAM without involving the CPU. This mechanism is crucial for all
devices that process large amounts of data such as network adapters, disk
controllers, or USB controllers. Because those devices can issue bus requests
targeting the RAM directly and not involving the CPU altogether, such requests
are naturally not subjected by the virtual-memory mechanism implemented in the
CPU in the form of an MMU. From the device's point of view there is just
physical memory. Hence, if a driver sets up a DMA transaction, let's say a disk
driver wants to read a block from the disk, the driver tells the device about
the address and size of a physical-memory buffer where the it wants to receive
the data. If the driver lives in a user-level process, as is the case for a
microkernel-based system, it still needs to know the physical address to
program the device correctly. Unfortunately, there is nothing to prevent the
driver from specifying any physical address to the device. Consequently, a
malicious driver could misuse the device to read and manipulate all parts of
the memory, including the kernel.
[image no_iommu]
Traditional machine without IOMMU. Direct memory accesses issued by the
disk controller are not subjected to the MMU. The disk controller can
access the entity of memory present in the system.
So - does this loop hole render the micro-kernel approach useless? Of course not.
Putting each driver in a dedicated address space is still beneficial in two
ways. First, classes of bugs that are unrelated to DMA remain confined in the
driver's address space. In practice most driver issues arise from issues like
memory leaks, synchronization problems, deadlocks, flawed driver logic, wrong
state machines, or incorrect device-initialization sequences. For those classes
of problems, the microkernel argument still applies. Second, executing a driver
largely isolated from other operating-system code minimizes the attack surface
of the driver. If the driver interface is rigidly small and well-defined, it is
hard to compromise the driver by exploiting its interface.
Still the DMA issue remains to be addressed. Fortunately, modern PC hardware
has closed the bus-master-DMA loophole by incorporating a so-called IOMMU into
the system. As depicted in the following figure, the IOMMU sits between the RAM
and the system bus where the devices are attached to. So each DMA request has
to pass the IOMMU, which is not only able to arbitrate the access of DMA
requests to the RAM but also able to virtualize the address space per device.
Similar to how a MMU confines each process running on the CPU within a distinct
virtual address space, the IOMMU is able to confine each device within a
dedicated virtual address space. To tell the different devices apart, the IOMMU
uses the PCI device's bus-device-function triplet as unique identification.
[image iommu]
An IOMMU arbitrates and virtualizes DMA accesses issued by a device to the
RAM. Only if a valid IOMMU mapping exists for a given DMA access, the memory
access is performed.
Of the microkernels supported by Genode, NOVA is the first kernel that supports
the IOMMU. NOVAs interface to the IOMMU is quite elegant. The kernel simply
applies a subset of the (MMU) address space of a process (aka protection domain
in NOVA speak) to the (IOMMU) address space of a device. So the device's
address space can be managed in the same way as we normally manage the address
space of a process. The only missing link is the assignment of device address
spaces to process address spaces. This link is provided by the dedicated system
call "assign_pci" that takes a process identifier and a device identifier as
arguments. Of course, both arguments must be subjected to a security policy.
Otherwise, any process could assign any device to any other process. To enforce
security, the process identifier is a capability to the respective protection
domain and the device identifier is a virtual address where the extended PCI
configuration space of the device is mapped in the specified protection domain.
Only if a user-level device driver got access to the extended PCI configuration
space of the device, it is able to get the assignment in place.
To make NOVA's IOMMU support available to Genode programs, we enhanced the
ACPI/PCI driver with the ability to hand out the extended PCI configuration
space of a device and added a NOVA-specific extension to the PD session
interface. The new 'assign_pci' function allows the assignment of a PCI device
to the protection domain.
[image iommu_aware]
NOVAs management of the IOMMU address spaces facilities the use of
driver-local virtual addresses as DMA addresses.
Even though these mechanisms combined principally
suffice to let drivers operate with the IOMMU enabled, in practice, the
situation is a bit more complicated. Because NOVA uses the same
virtual-to-physical mappings for the device as it uses for the process, the DMA
addresses the driver needs to supply to the device must be virtual addresses
rather than physical addresses. Consequently, to be able to make a device
driver usable on systems without IOMMU as well as on systems with IOMMU, the
driver needs to be IOMMU-aware and distinguish both cases. This is an
unfortunate consequence of the otherwise elegant mechanism provided by NOVA. To
relieve the device drivers from caring about both cases, we came up with a
solution that preserves the original device interface, which expects physical
addresses. The solution comes in the form of so called device PDs. A device PD
represents the address space of a device as a Genode process. Its sole purpose
is to hold mappings of DMA buffers that are accessible by the associated
device. By using one-to-one physical-to-virtual mappings for those buffers
within the device PD, each device PD contains a subset of the physical address
space. The ACPI/PCI server performs the assignment of device PDs to PCI
devices. If a device driver intends to use DMA, it asks the ACPI/PCI driver for
a new DMA buffer. The ACPI/PCI driver allocates a RAM dataspace at core,
attaches it to the device PD using the dataspace's physical address as virtual
address, and hands out the dataspace capability to the driver. If the driver
requests the physical address of the dataspace, the returned address will be a
valid virtual address in the associated device PD. From this design follows
that a device driver must allocate DMA buffers at the ACPI/PCI server (while
specifying the PCI device the buffer is intended for) instead of using core's
RAM service to allocate buffers anonymously. The current implementation of the
ACPI/PCI server assigns all PCI devices to only one device PD. However, the
design devises a natural way to partition devices into different device PDs.
[image iommu_agnostic]
By modelling a device address space as a dedicated process (device PD),
the traditional way of programming DMA transactions can be maintained,
even with the IOMMU enabled.
Because the changed way of how DMA buffers are allocated, our existing drivers
such as the AHCI disk driver, the OSS sound driver, the iPXE network driver,
and the USB driver had to be slightly modified. We also extended DDE Kit with
the new 'dde_kit_pci_alloc_dma_buffer' function for allocating DMA buffers.
With those changes, the complete Genode user land can be used on systems with
IOMMU enabled. Hence, we switched on the IOMMU on NOVA by default.
Full virtualization on NOVA/x86
###############################
Vancouver is a x86 virtual machine monitor that is designed to run as
user-level process on top of the NOVA hypervisor. In
[https://genode.org/documentation/release-notes/11.11#Faithful_x86_PC_Virtualization_enabled_by_the_Vancouver_VMM - Genode version 11.11],
we introduced the preliminary adaptation of Vancouver to Genode. This version
was meant as a mere proof of concept, which allowed the bootup of small Guest
OSes (such as Fiasco.OC or Pistachio) inside the VMM. However, it did not
support any glue code to Genode's session interface, which limited the
usefulness of this virtualization solution at that point. We had planned to
continue the integration of Vancouver with Genode once we observed public
demand.
The move of NOVA's development to Intel Labs apparently created this demand.
It is undeniable that combining the rich user land provided by Genode with the
capabilities of the Vancouver VMM poses an attractive work load for NOVA. So
the stalled line of the integration work of Vancouver with Genode was picked up
within Intel Labs, more specifically by Markus Partheymüller. We are delighted
to be able to merge the outcome of this undertaking into the mainline Genode
development. Thanks to Intel Labs and Markus in particular for this substantial
contribution!
The features added to the new version of Vancouver for Genode are as follows:
:VMX support:
Our initial version supported AMD's SVM technology only because this was
readily supported by Qemu. With the added support for Intel VMX, Vancouver
has become able to operate on both Intel and AMD processors with hardware
virtualization support.
:Timer support:
With added support for timer interrupts, the VMM has become able to
boot a complete Linux system.
:Console support:
With this addition, the guest VM can be provided with a frame buffer and
keyboard input.
For the frame-buffer size in Vancouver, the configuration value in the
machine XML node is used. It is possible to map the corresponding memory
area directly to the guest regardless if it comes from nitpicker, a virtual
frame buffer, or the VESA driver. The guest is provided with two modes (text
mode 3 and graphics mode 0x114 (0x314 in Linux).
Pressing LWIN+END while a VM has focus resets the virtual machine. Also,
RESET and DEBUG key presses will not be forwarded to the VM anymore.
It is possible to dump a VM's state by pressing LWIN+INS keys.
The text console is able to detect idle mode, unmaps the buffer from the
guest and stops interpreting. Upon the next page fault in this area, it
resumes operation again. The code uses a simple checksum mechanism instead
of a large buffer and 'memcmp' to detect an idle text console. False
positives don't matter very much.
:Network support:
The VMM has become able to use the Intel 82576 device model from the NUL
user land to give VMs access to the network via Genode's NIC bridge service
or a NIC driver.
:Disk support:
The VMM can now assign block devices to guests using Genode's block-session
interface. The machine has to be configured to use a specified drive, which
could be theoretically routed to different partitions or services via policy
definitions. Currently the USB driver only supports one device. Genode's AHCI
driver is untested.
:Real-time clock:
By using the new RTC session interface, Vancouver is able to provide the
wall-clock time to guest OSes.
To explore the new version of the Vancouver VMM, there exists a ready-to-use
run script at 'ports/run/vancouver.run'. Only the guest OS binaries such as a
Linux kernel image and a RAM disk must be manually supplied in the
'<build-dir>/bin' directory. The run script is able to start one or multiple
instances of the VMM using the graphical launchpad.
Low-latency audio output
########################
In version 10.05, we introduced an interface for the playback of audio data
along with an audio mixer component and ALSA-based sound drivers ported from
the Linux kernel. The original 'Audio_out' session interface was based on
Genode's packet stream facility, which allows the communication of bulk data
across address spaces via a combination of shared memory and signals. Whereas
shared memory is used to transfer the payload in an efficient manner without
the need to copy data via the kernel, signals are used to manage the data flow
between the information source and sink.
[image packet_stream]
Figure [packet_stream] displays the life cycle of a packet of information
transferred from the source to the sink. The original intent behind the
packet-stream facility was the transmission of networking packets and blocks
of block devices. At the time when we first introduced the 'Audio_out'
interface, the packet stream seemed like a good fit for audio, too. However, in
the meanwhile, we came to the conclusion that this is not the case when trying
to accommodate streamed audio data and sporadic audio output at the same time.
For the output of streamed audio data, a codec typically decodes a relatively
large portion of an audio stream and submits the sample data to the mixer. The
mixer, in turn, mixes the samples of multiple sources and forwards the result
to the audio driver. Each of those components the codec, the mixer, and the
audio driver live in a separate process. By using large buffer sizes between
them, the context-switching overhead is hardly a concern. Also, the driver can
submit large buffers of sample data to the sound device without any further
intervention needed.
In contrast, sporadic sounds are used to inform the user about an immediate
event. It is ultimately expected that such sounds are played back without much
latency. Otherwise the interactive experience (e.g., of games) would suffer.
Hence, using large buffers between the audio source, the mixer, and the driver
is not an option. By using the packet stream concept, we have to settle on a
specific buffer size. A too small buffer increases CPU load caused by many
context switches and the driver, which has to feed small chunks of sample data
to the sound device. A too large buffer, however, makes sporadic sounds at low
latencies impossible. We figured out that the necessity to find a sweet spot
for picking a buffer size is a severe drawback. This observation triggered us
to replace the packet-stream-based communication mechanism of the 'Audio_out'
session interface by a new solution that we specifically designed to
accommodate both corner cases of audio output.
[image audio_out]
Similarly to the packet-stream mechanism, the new interface is based on a
combination of shared memory and signals. However, we dropped the notion of
ownership of packets. When using the packet-stream protocol depicted as above,
either the source or the sink is in charge of handling a given packet at a
given time, not both. At the points 1, 2, and 4, the packet is owned by the
source. At the points 3 and 4, the packet is owned by the sink. By putting a
packet descriptor in the submit queue or acknowledgement queue, there is a
handover of responsibility. The new interface weakens this notion of ownership
by letting the source update once submitted audio frames even after submitting
them. If there are solely continuous streams of audio arriving at the mixer,
the mixer can mix those large batches of audio samples at once and pass the
result to the driver.
[image mixer_streaming]
The mixer processes incoming data from multiple streaming sources as batches.
Now, if a sporadic sound comes in, the mixer checks the
current output position reported by the audio driver, and re-mixes those
portions that haven't been played back yet by incorporating the sporadic sound.
So the buffer consumed by the driver gets updated with new data.
[image mixer_sporadic]
A sporadic occuring sound prompts the mixer to remix packets that are
already submitted in the output queue.
Besides changing the way of how packets are populated with data, the second
major change is turning the interface into a time-triggered concept. The
driver produces periodic signals that indicate the completeness of a
played-back audio packet. This signal triggers the mixer to become active,
which in turn serves as a time base for its clients. The current playback
position is denoted alongside the sample data as a field in the memory buffer
shared between source and sink.
The new 'Audio_out' interface has the potential to align the requirements of
both streamed audio with those of sporadic sounds. As a side benefit, the now
domain-specific interface has become simpler than the original packet-stream
based solution. This becomes nowhere as evident as in the implementation of the
mixer, which has become much simpler (30% less code). The interface change
is accompanied with updates of components related to audio output, in
particular the OSS sound drivers contained in 'dde_oss', the ALSA audio driver
for Linux, the avplay media player, and the libSDL audio back-end.
Base framework
##############
Signalling API improvements
===========================
The signalling API provided by 'base/signal.h' is fairly low level. For
employing the provided mechanism by application software, we used to craft
additional glue code that translates incoming signals to C++ method
invocations. Because the pattern turned out to be not only useful but a good
practice, we added the so called 'Signal_dispatcher' class template to the
signalling API.
In addition to being a 'Signal_context', a 'Signal_dispatcher' associates a
member function with the signal context. It is intended to be used as a member
variable of the class that handles incoming signals of a certain type. The
constructor takes a pointer-to-member to the signal handling function as
argument. If a signal is received at the common signal reception code, this
function will be invoked by calling 'Signal_dispatcher_base::dispatch'. This
pattern can be observed in the implementation of RAM file system
('os/src/server/ram_fs').
Under the hood, the signalling implementation received a major improvement with
regard to the life-time management of signal contexts. Based on the observation
that 'Signal' objects are often referring to non-trivial objects derived from
'Signal_context', it is important to defer the destruction of such objects to a
point when no signal referring to the context is in flight anymore. We solved
this problem by modelling 'Signal' type as a shared pointer that operates on a
reference counter embedded in the corresponding 'Signal_context'. Based on
reference counter, the 'Signal_receiver::dissolve()' function does not return
as long as the signal context to be dissolved is still referenced by one or
more 'Signal' objects.
Trimmed and unified framework API
=================================
A though-provoking
[https://sourceforge.net/mailarchive/forum.php?thread_name=CAGQ-%3Dq27%2B_UooBiJmz9RdTE1gDmVcg9v0w-8TNgEH5fzHYiA%2BQ%40mail.gmail.com&forum_name=genode-main - posting]
on our mailing list prompted us to explore the idea to make shared libraries
and dynamically linked executables binary compatible among different kernels.
This sounds a bit crazy at first but it is not downright infeasible.
As a baby step into this direction, we unified several public headers of the
Genode API and tried to make headers private to the framework where possible.
The latter is the case for the 'base/platform_env.h' header, which is actually
not part of the generic Genode API. Hence, it was moved to the
framework-internal 'src/base/env'. Another step was the removal of
platform-specific types that are not necessarily platform-dependent. We could
remove the 'Native_lock' type without any problems. Also, we were able to unify
the IPC API, which was formerly split into the two parts 'base/ipc_generic.h'
and 'base/ipc.h' respectively. Whereas 'base/ipc_generic.h' was shared among
all platforms, the 'base/ipc.h' header used to contain platform-specific IPC
marshalling and unmarshalling code. But by moving this code from the header to
the corresponding (platform-specific) IPC library, we were able to discard the
content of 'base/ipc.h' altogether. Consequently, the former
'base/ipc_generic.h' could be renamed to 'base/ipc.h'. These changes imply no
changes at the API level.
Simplified structure of base libraries
======================================
The Genode base API used to come in the form of many small libraries, each
covering a dedicated topic. Those libraries were 'allocator_avl', 'avl_tree',
'console', 'env', 'cxx', 'elf', 'env', 'heap', 'server', 'signal', 'slab',
'thread', 'ipc', and 'lock'. The term "library" for those bits of code was
hardly justified as most of the libraries consisted of only a few .cc files.
Still the build system had to check for their inter-dependencies on each run of
the build process. Furthermore, for Genode developers, specifying the list of
base libraries in their 'target.mk' files tended to be an inconvenience. Also,
the number of libraries and their roles (core only, non-core only, shared by
both core and non-core) were not easy to capture. Hence, we simplified the way
of how those base libraries are organized. They have been reduced to the
following few libraries:
* 'cxx.mk' contains the C++ support library
* 'startup.mk' contains the startup code for normal Genode processes
On some platform, core is able to use the library as well.
* 'base-common.mk' contains the parts of the base library that are
identical by core and non-core processes.
* 'base.mk' contains the complete base API implementation for non-core
processes
Consequently, the 'LIBS' declaration in 'target.mk' files becomes simpler as
well. In the normal case, only the 'base' library must be mentioned.
New fault-detection mechanism
=============================
Until now, it was hardly possible for a parent process to respond to crashes of
child processes in a meaningful way. If a child process crashed, the parent
would normally just not take notice. Even though some special use cases such as
GDB monitor could be accommodated by the existing
'Cpu_session::exception_handler' facility, this mechanism requires the
virtualization of the 'Cpu_session interface' because an exception handler used
to refer to an individual thread rather than the whole process. For ordinary
parents, this mechanism is too cumbersome to use. However, there are several
situations where a parent process would like to actively respond to crashing
children. For example, the parent might like to restart a crashed component
automatically, or enter a special failsafe mode.
To ease the implementation of such scenarios, we enhanced the existing
'Cpu_session::exception_handler' mechanism with the provision of a
default signal handler that is used if no thread-specific handler is installed.
The default signal handler can be set by specifying an invalid thread
capability and a valid signal-context capability. So for registering a signal
handler to all threads of a process, no virtualization of the 'Cpu_session'
interface is needed anymore. The new mechanism is best illustrated by the
'os/run/failsafe.run' script, which creates a system that repeatedly spawns a
crashing child process.
Reworked synchronization primitives
===================================
We reworked the framework-internal lock interface in order to be able to use
the 'futex' syscall on the Linux base platform. Previously, the lock
implementation on Linux was based on Unix signals. In the contention case, the
applicant for a contended lock would issue a blocking system call, which gets
canceled by the occurrence of a signal. We used 'nanosleep' for this purpose.
Once the current owner of the lock releases the lock, it sends a signal to the
next applicant of the lock. Because signals are buffered by the kernel, they
are guaranteed to be received by the targeted thread. However, in situations
with much lock contention, we observed the case where the signal was delivered
just before the to-be-blocked thread could enter the 'nanosleep' syscall. In
this case, the signal was not delivered at the next entrance into the kernel
(when entering 'nanosleep') but earlier. Even though the signal handler was
invoked, we found no elegant way to handle the signal such that the subsequent
'nanosleep' call would get skipped. So we decided to leave our signal-based
solution behind and went for the mainstream 'futex' mechanism instead.
Using this mechanism required us to re-design the internal lock API, which was
originally designed with the notion of thread IDs. The 'Native_thread_id' type,
which was previously used in the lock-internal 'Applicant' class to identify a
thread to be woken up, was not suitable anymore for implementing this change.
Hence, we replaced it with the 'Thread_base*' type, which also has the positive
effect of making the public 'base/cancelable_lock.h' header file
platform-independent.
In addition to reworking the basic locking primitives, we changed the
'Object_pool' data structure to become safer to use. The former 'obj_by_*'
functions have been replaced by 'lookup_and_lock' that looks up an object and
locks it in one atomic operation. Additionally, the case that an object may
already be in destruction is handled gracefully. In this case, the lookup will
return that the object is not available anymore.
Low-level OS infrastructure
###########################
Notification mechanism for the file-system interface
====================================================
To support dynamic system scenarios, we extended Genode's file-system interface
with the ability to monitor changes of files or directories, similar to the
inotify mechanism on Linux but simpler. The new 'File_system::sigh' function
can be used to install a signal handler for an open file node. When a node is
closed after a write operation, a prior registered signal handler for this file
gets notified. Signal handlers can also be installed for directories. In this
case, the signal handler gets informed about changes of immediate nodes hosted
in the directory, i.e., the addition, renaming, or removal of nodes.
The 'ram_fs' server has been enhanced to support the new interface. So any file
or directory change can now be observed by 'ram_fs' clients.
New adapter from file-system to ROM session interface
=====================================================
The new 'fs_rom' server translates the 'File_system' session interface to the
'ROM' session' interface. Each request for a ROM file is handled by looking
up an equally named file on the file system. If no such file can be found,
then the server will monitor the file system for the creation of the
corresponding file. Furthermore, the server reflects file changes as signals
to the ROM session.
There currently exist two limitations: First, symbolic links are not handled.
Second, the server needs to allocate RAM for each requested file. The RAM is
always allocated from the RAM session of the server. Thereby, the RAM quota
consumed by the server depends on the client requests and the size of the
requested files. Therefore, one instance of the server should not be used by
untrusted clients and trusted clients at the same time. In such situations,
multiple instances of the server could be used.
The most interesting feature of the 'fs_rom' server is the propagation of
file-system changes as ROM module changes. This clears the way to use this
service to supply dynamic configurations to Genode programs.
Dynamic re-configuration of the init process
============================================
The init process has become able to respond to configuration changes by
restarting the scenario using the new configuration. To make this feature
useful in practice, init must not fail under any circumstances. Even on
conditions that were considered previously as fatal and led to the abort of
init (such as ambiguous names of the children or misconfiguration in general),
init must stay alive and responsive to configuration changes.
With this change, the init process is one of the first use cases of the dynamic
configuration feature enabled via the 'fs_rom' service and the new file-system
notifications. By supplying the configuration of an init instance via the
'fs_rom' and 'ram_fs' services, the configuration of this instance gets fetched
from a file of the 'ram_fs' service. Each time, this file is changed, for
example via VIM running within a Noux runtime environment, the init process
re-evaluates its configuration.
In addition to the support for dynamic re-configurations, we simplified the use
of conditional session routing, namely the '<if-args>' mechanism. When matching
the 'label' session argument using '<if-args>' in a routing table, we can omit
the child name prefix because it is always the same for all sessions
originating from the child anyway. By handling the matching of session labels
as a special case, the expression of label-specific routing
becomes more intuitive.
Timer interface turned into asynchronous mode of operation
==========================================================
The 'msleep' function of 'Timer::Session' interface is one of the last relics
of blocking RPC interfaces present in Genode. As we try to part away from
blocking RPC calls inside servers and as a means to unify the timer
implementation across the many different platforms supported by Genode, we
changed the interface to an asynchronous mode of operation.
Synchronous blocking RPC interfaces turned out to be constant sources of
trouble and code complexity. E.g., a timer client that also wants to respond to
non-timer events was forced to be a multi-threaded process. Now, the blocking
'msleep' call has been replaced by a mechanism for programming timeouts and
receiving wakeup signals in an asynchronous fashion. Thereby signals
originating from the timer can be handled, along with signals from other signal
sources, by a single thread. Once a timer client has registered a signal
handler using the 'Timer::sigh' function, it can program timeouts using the
functions 'trigger_once' and 'trigger_periodic', which take an amount of
microseconds as argument. For maintaining compatibility and convenience, the
interface still contains the virtual 'msleep' function. However, it is not an
RPC function anymore but a mere client-side wrapper around the 'sigh' and
'trigger_once' functions. For use cases where sleeping at the granularity of
milliseconds is too coarse (such as udelay calls by device drivers), we added
a new 'usleep' call, which takes a number of microseconds as argument.
As a nice side effect of the interface changes, the platform-specific
implementations could be vastly unified. On NOVA and Fiasco.OC, the need to use
one thread per client has vanished. As a further simplification, we changed the
timer to use the build system's library-selection mechanism instead of
providing many timer targets with different 'REQUIRES' declarations. This
reduces the noise of the build system. For all platforms, the target at
'os/src/drivers/timer' is built. The target, in turn, depends on a 'timer'
library, which is platform-specific. The various library description files are
located under 'os/lib/mk/<platform>'. The common bits are contained in
'os/lib/mk/timer.inc'.
Since the 'msleep' call is still available from the client's perspective,
the change of the timer interface does not imply an API incompatibility.
However, it provides the opportunity to simplify clients in cases that required
the maintenance of a separate thread for the sole purpose of
periodic signal generation.
Loader
======
The loader is a service that enables its clients to dynamically create Genode
subsystems. Leveraging the new fault-detection support described in section
[New fault-detection mechanism], we enabled loader clients to respond to
failures that occur inside the spawned subsystem. This is useful for scenarios
where subsystems should be automatically restarted or in situations where the
system should enter a designated failsafe mode once an unexpected fault
happens.
The loader provides this feature by installing an optional client-provided
fault handler as default CPU exception handler and a RM fault handler for all
CPU and RM sessions of the loaded subsystem. This way, the failure of any
process within the subsystem gets reflected to the loader client as a signal.
The new 'os/run/failsafe.run' test illustrate this mechanism. It covers two
cases related to the loader, which are faults produced by the immediate child
of the loader and faults produced by indirect children.
Focus events for the nitpicker GUI server
=========================================
To enable a way for applications to provide visual feedback to changed keyboard
focus, we added a new 'FOCUS' event type to the 'Input::Event' structure. To
encode whether the focus was entered or left, the former 'keycode' member is
used (value 0 for leaving, value 1 for entering). Because 'keycode' is
misleading in this context, the former 'Input::Event::keycode' function was
renamed to 'Input::Event::code'. The nitpicker GUI server has been adapted to
deliver focus events to its clients.
NIC bridge with support for static IP configuration
===================================================
NIC bridge is a service that presents one physical network adaptor as many
virtual network adaptors to its clients. Up to now, it required each client
to obtain an IP address from a DHCP server at the physical network. However,
there are situations where the use of static IPs for virtual NICs is useful.
For example, when using the NIC bridge to create a virtual network between
the lighttpd web server and the Arora web browser, both running as Genode
processes without real network connectivity.
The static IP can be configured per client of the NIC bridge using a '<policy>'
node of the configuration. For example, the following policy assigns a static
address to a client with the session label "lighttpd".
!<start name="nic_bridge">
! ...
! <config>
! <policy label="lighttpd" ip_addr="10.0.2.55"/>
! </config>
!</start>
Of course, the client needs to configure its TCP/IP stack to use the assigned
IP address. This can be done via configuration arguments examined by the
'lwip_nic_dhcp' libc plugin. For the given example, the configuration for the
lighttpd process would look as follows.
!<start name="lighttpd">
! <config>
! <interface ip_addr="10.0.2.55"
! netmask="255.255.255.0"
! gateway="10.0.2.1"/>
! </config>
!</start>
Libraries and applications
##########################
New terminal multiplexer
========================
The new 'terminal_mux' server located at 'gems/src/server/terminal_mux' is able
to provide multiple terminal sessions over one terminal-client session. The
user can switch between the different sessions using a keyboard shortcut, which
brings up an ncurses-based menu.
The terminal sessions provided by terminal_mux implement (a subset of) the
Linux terminal capabilities. By implementing those capabilities, the server
is interchangeable with the graphical terminal ('gems/src/server/terminal').
The terminal session used by the server is expected to by VT102 compliant.
This way, terminal_mux can be connected via an UART driver with terminal
programs such as minicom, which typically implement VT102 rather than the Linux
terminal capabilities.
When started, terminal_mux displays a menu with a list of currently present
terminal sessions. The first line presents status information, in particular
the label of the currently visible session. A terminal session can be selected
by using the cursor keys and pressing return. Once selected, the user is able
to interact with the corresponding terminal session. Returning to the menu is
possible at any time by pressing control-x.
For trying out the new terminal_mux component, the 'gems/run/termina_mux.run'
script sets up a system with three terminal sessions, two instances of Noux
executing VIM and a terminal_log service that shows the log output of both Noux
instances.
New ported 3rd-party libraries
==============================
To support our forthcoming port of Git to the Noux runtime environment, we
have made the following libraries available via the libports repository:
* libssh-0.5.4
* curl-7.29.0 (for now the port is x86_* only because it depends on libcrypto,
which is currently not tested on ARM)
* iconv-1.14
Device drivers
##############
Besides the changes concerning the use of IOMMUs, the following device driver
have received improvements:
:UART drivers:
The OMAP4 platform support has been extended by a new UART driver, which
enables the use of up to 4 UART interfaces. The new driver is located at
'os/src/drivers/uart/omap4'.
All UART drivers implement the 'Terminal::Session' interface, which
provides read/write functionality accompanied by a function to determine
the terminal size. The generic UART driver code shared among the various
implementations has been enhanced to support the detection of the terminal
size using a protocol of escape sequences. This feature can be enabled by
including the attribute 'detect_size="yes"' in the policy of a UART client.
This is useful for combining UART drivers with the new 'terminal_mux'
server.
:ACPI support for 64-bit machines:
In addition to IOMMU-related modifications, the ACPI driver has been enhanced
to support 64-bit machines and MCFG table parsing has been added.
:PCI support for IOMMUs:
With the added support of IOMMUs, the 'Pci::Session' interface has been
complemented with a way to obtain the extended PCI configuration space in the
form of a 'Genode::Dataspace'. Also, the interface provides a way to allocate
DMA buffers for a given PCI device. Device drivers that are meant to be used
on system with and without IOMMUs should use this interface rather than
core's RAM session interface to allocate DMA buffers.
:Real-time clock on x86:
Up to now, the x86 real-time clock driver served as a mere example for
accessing I/O ports on x86 machines but the driver did not expose any service
interface. With the newly added 'os/include/rtc_session' interface and the
added support of this interface in the RTC driver, Genode programs have now
become able to read the real-time clock. Currently, the interface is used by
the Vancouver VMM.
:USB driver restructured, support for Arndale board added:
While adding support for the Exynos-5-based Arndale board, we took the
chance to restructure the driver to improve portability to new
platforms. The most part of the driver has become a library, which is
built in a platform-specific way. The build system automatically selects
the library that fits for the platform as set up for the build directory.
Platforms
#########
NOVA
====
The NOVA base platform received major improvements that address the kernel
as well as Genode's NOVA-specific code. We pursued two goals with this line
of work. The first goal was the use of NOVA in highly dynamic settings, which
was not possible before, mainly due to lacking kernel features. The second
goal was the use of IOMMUs.
NOVA is ultimately designed for accommodating dynamic workloads on top of the
kernel. But we found that the implementation of crucial functionality was
missing. In particular, the kernel lacked the ability to destroy all kinds of
kernel objects and to reuse memory of kernel objects that had been destroyed.
Consequently, when successively creating and destroying kernel objects such as
threads and protection domains, the kernel would eventually run out of memory.
This issue became a show stopper for running the Genode tool chain on NOVA
because this scenario spawns and destroys hundreds of processes. For this
reason, we complemented the kernel with the missing functionality. This step
involved substantial changes in the kernel code. So our approach of using the
upstream kernel and applying a hand full of custom patches started to show its
limitations.
To streamline our work flow and to track the upstream kernel in a structured
way, we decided to fork NOVA's Git repository and maintain our patches in our
fork. For each upstream kernel revision that involves kernel ABI changes, we
create a separate branch called "r<number>". This branch corresponds to the
upstream kernel with our series of custom patches applied (actually rebased) on
top. This way, our additions to the upstream kernel are well documented. The
'make prepare' mechanism in the base-nova repository automates the task of
checking out the right branch. So from the Genode user's point of view, this
change is transparent.
The highly dynamic application scenarios executed on NOVA triggered several
synchronization issues in Genode's core process that had not been present on
other base platforms. The reason for those issues to occur specifically on NOVA
lies in the concurrent page fault handling as employed on this base platform.
For all classical L4-like kernels and Fiasco.OC, we use one global pager thread
to resolve all page faults that occur in the whole Genode system. In contrast,
on NOVA we use one pager thread per user thread. Consequently, proper
fine-grained synchronization between those pager threads and the other parts of
core is mandated. Even though the immediate beneficiary of these changes is the
NOVA platform, many of the improvements refer to generic code. This paves the
ground for scaling the page-fault handling on other base platforms (such as
Fiasco.OC) to multiple threads. With these improvements in place, we are able
to successfully execute the 'noux_tool_chain_nova' scenario on the NOVA kernel
and build Genode's core on NOVA. That said, however, not all issues are covered
yet. So there is still a way left to go to turn base-nova into a base platform
that is suitable for highly dynamic scenarios.
The second goal was the use of NOVA's IOMMU support on Genode. This topic is
covered in detail in section [DMA protection via IOMMU].
To be able to use and debug Genode on NOVA on modern machines that lack legacy
comports, we either use UART PCI cards or the Intel's Active Management
Technology (AMT) mechanism. In both cases, the I/O ports to access the serial
interfaces differ from the legacy comports. To avoid the need for adjusting the
I/O port base addresses per platform, we started using the chain-boot-loader
called "bender" developed by the Operating Systems Group of TU Dresden,
Germany. This boot loader is started prior the kernel, searches the PCI bus for
the first suitable device and registers the corresponding I/O port base address
at the bios data area (BDA). Genode's core, in turn, picks the I/O port base
address up from the BDA and uses the registered i8250 serial controller for its
LOG service.
Execution on bare hardware (base-hw)
====================================
The base-hw platform enables the use of Genode on ARM-based hardware without
the need for a 3rd-party kernel.
With the new release, the range of supported ARM-based hardware has been
extended to cover the following additional platforms. With the previous
release, we introduced the support for Freescale i.MX family of SoC, starting
with i.MX31. The current release adds support for the i.MX53 SoC and adds
a user-level timer driver for this platform. With the Samsung Exynos 5, the
first Cortex-A15-based SoC has entered the list of supported SoCs. Thanks to
this addition, Genode has become able to run on the
[http://www.arndaleboard.org - Howchip Arndale board]. At the current state,
core and multiple instances of init can be executed but drivers for peripherals
are largely missing. Those will be covered by our ongoing work with this SoC.
The added platforms are readily available via the 'create_builddir' tool.
To make base-hw practically usable on real hardware (i.e., the Pandaboard),
support for caches has been implemented. Furthermore, the implementation of the
signalling API underwent a redesign, which leverage the opportunities that
arise with tailoring a kernel specifically to the Genode API. As a side-benefit
of this endeavour, we could unify the 'base/signal.h' header with the generic
version and thereby took another step towards the unification of the Genode
headers across different kernels.
Microblaze platform removed
===========================
The 'base-mb' platform has been removed because it is no longer maintained.
This platform enabled Genode to run directly on the Xilinx Microblaze softcore
CPU. For supporting the Microblaze CPU architecture in the future, we might
consider integrating support for this architecture into base-hw. Currently
though, there does not seem to be any demand for it.
Fiasco.OC forked, support for Exynos 5 SoC added
================================================
In the last release cycle, we went beyond just using the Fiasco.OC kernel and
started to engage with the kernel code more intensively. To avoid that the
management of a growing number of kernel patches goes out of hand, we forked
the Fiasco.OC kernel and conduct our development in our Fiasco.OC Git
repository. When using the 'make prepare' mechanism in the 'base-foc'
repository, the new Git repository will be used automatically. There exists a
dedicated branch for each upstream SVN revision that we use. We started with
updating Fiasco.OC to the current revision 47. Hence, the current branch used
by Genode is named "r47". The branch contains the unmodified state of the
upstream SVN repository with our modifications appearing as individual commits
on top. This makes it easy to keep track of the Genode-specific modifications.
Please note that the update to Fiasco.OC requires minor adaptations inside
the 'ports-foc' repository. So for using L4Linux, "make prepare" must be
issued in both repositories 'base-foc' and 'ports-foc'.
Speaking of engaging with the kernel code, the most profound improvement is
the support for the Samsung Exynos-5-based Arndale board that we added to the
kernel. This goes hand in hand with the addition of this platform to Genode.
For creating a build directory targeting the Arndale board, just specify
"foc_arndale" to the 'create_builddir' tool. At the time of the release,
several basic scenarios including the timer driver and the USB driver are
working. Also, both Cortex-A15 CPUs of the Exynos 5 SoC are operational.
However, drivers for most of the peripherals of the Exynos-5 SoC are missing,
which limits the current scope of Genode on this platform.
Linux
=====
Since the base-linux platform became used for more than a mere development
vehicle, we are revisiting several aspects of this base platform. In the last
release, we changed the synchronous inter-process-communication mechanism to
the use of SCM rights. For the current release, it was time to have a closer
look at the memory management within core. The Linux version of core used a
part of the BSS to simulate access to physical memory. All dataspaces would
refer to a portion of 'some_mem'. So each time when core would access the
dataspace contents, it would access its local BSS. For all processes outside of
core, dataspaces were represented as files. We have now removed the distinction
between core and non-core processes. Now, core uses the same 'Rm_session_mmap'
implementation as regular processes. This way, the 'some_mem' could be
abandoned. We still use a BSS variable for allocating core-local meta data
though. The major benefit of this change is the removal of the artificial
quota restriction that was imposed by the predefined size of the 'some_mem'
array. Now, the Linux base platform can use as much memory as it likes. Because
the Linux kernel implements virtual memory, we are not bound by the physical
memory. Hence, the available quota assigned to the init process is almost
without bounds.
To implement the fault-detection mechanism described in section
[New fault-detection mechanism] on Linux, we let core catch SIGCHLD signals of
all Genode processes. If such a signal occurs, core determines the process that
produced the signal by using 'wait_pid', looks up the CPU session that belongs
to the process and delivers an exception signal to the registered exception
handler. This way, abnormal terminations of Genode processes are reflected to
the Genode API in a clean way and Genode processes become able to respond to
terminating Genode child processes.
OKL4
====
The audio stub driver has been removed from OKLinux. Because of the changed
'Audio_out::Session' interface, we needed to decide on whether to adapt the
OKLinux stub driver to the changed interface or to remove the stub driver.
Given the fact that OKLinux is not actively used, we decided for the latter.

943
doc/release_notes/13-05.txt Normal file
View File

@@ -0,0 +1,943 @@
===============================================
Release notes for the Genode OS Framework 13.05
===============================================
Genode Labs
With Genode 13.05, we have diverged quite a bit from the feature-laden plans
laid out in our [https://genode.org/about/road-map road map] as we realized
that consolidating and optimizing the current feature set will have a more
sustainable effect than functional enhancements at this point. In particular,
we addressed the problem that the ever growing diversity of platforms imposes
on the quality and coverage of testing. We also desired to extend our
systematic testing efforts to real hardware platforms, and to have a mechanism
for detecting performance regressions. Section
[Automated quality-assurance testing] details how we approached these
challenges, and how we went on analyzing Genode's network performance in
particular.
That said, we haven't completely restrained ourself from implementing new
features. Closely related to test automation but very useful in other
situations, we improved the terminal infrastructure in order to enable the
interactive use of dynamic system scenarios in headless situations. Section
[Terminal infrastructure] introduces a new command-line interface for managing
Genode subsystems.
With regard to platform support, the current release follows up on the
hardware support added in the previous releases. For Samsung Exynos-5-based
platforms, drivers for USB-3, fast-ethernet networking, gigabit networking,
eMMC, and SATA have been added. For Freescale i.MX53-based devices, new
drivers for display, touchscreen, and GPIO have become available. The
OMAP4 display driver has been enhanced to cover both LCD displays and HDMI.
Our custom base-hw kernel has been enabled on the Raspberry Pi
board. Finally, Linux/ARM was added to accompany Linux/x86 as a fully usable
Genode base platform.
Automated quality-assurance testing
###################################
One of the greatest challenges of the Genode OS Framework is preventing
regressions in the face of the growing number of supported platforms.
The challenge stems from the fact that the space of Genode scenarios grow
two-dimensional. On one axis, the software stack on top of Genode gets more
and more complex, which calls for contiguous testing. On the other axis, there
is a growing number of kernel and hardware platforms to support.
In principle, there are even more dimensions, for example the diversity
of tool chains or the diversity of the OS used on the development machine.
Luckily, the problem of tool-chain diversity could be mitigated with the
introduction of the Genode tool chain since version 11.11, which was a huge
relief. However, the mentioned two dimensions cannot be avoided. Because
manual testing of manifold scenarios of component compositions on top of many
different kernels became infeasible, we automated the task of building and
testing years ago.
The automated builder checks out the staging branch of Genode, prepares
the repositories that integrate 3rd-party code, and builds the software
for 12 different kernel/platform combinations. Not all 3rd-party software
packages are built for each combination though. But we make sure that each
piece of software is exposed to different combinations of CPU architectures
and kernels.
The build test is accompanied with automated runtime tests of various
run scripts on Qemu. Each run script listed in 'tool/autopilot.lst' is
executed on each kernel using the autopilot tool. The tests range from
stimulating low-level mechanisms (such as signal, timer, and ldso) to complex
scenarios (such as testing networking with L4Linux, or running Noux).
Both build and runtime tests are executed daily. If any of the
tests fail, the Genode developers receive a notification email.
Once all tests are passed, the staging branch can be merged into the master
branch. This way, we spare the users of Genode to deal with intermediate
problems introduced in the staging branch.
The build and runtime tests have become a fundamental tool for our
development work. With the growing variety of real hardware
(as opposed to hardware emulated via Qemu), however, our existing solution
was falling short. Even though our tests confirm that Genode is running
happily on Qemu, they won't help us to detect regressions in our device
drivers for non-Qemu hardware such as Pandaboard, Arndale, or modern PC
hardware. Furthermore, we are increasingly focussing on performance
considerations. In order to be a viable OS platform, Genode does not only need
to be able to do networking, but networking performance must be on par with
mainstream OSes. This raises the new challenge to extend our
continuous-testing tools to become continuous-benchmarking tools. The ultimate
goal is to monitor the performance of Genode on real hardware over long
periods of development.
In this release cycle, we attacked this problem in two steps. First, we
enabled Genode's run tool to target not only Qemu but real hardware, with the
premise that existing run scripts must not be changed. The second step is the
creation of new run scripts that perform benchmarks in an automated fashion.
By aggregating the results of this automatically executed benchmarks, we can
correlate performance effects with commits in our code repository.
Targeting real hardware via the run tool
========================================
In the following, we briefly describe the procedure to execute run scripts
on native hardware, for both Intel-based x86 machines and ARM-based platforms.
TFTP boot x86
~~~~~~~~~~~~~
The following description uses NOVA as an example to illustrate the usage.
Other base platforms are supported as well and can be configured analogously.
[https://os.inf.tu-dresden.de/~us15/pulsar/ - Pulsar] is a tiny boot loader
that uses PXE to fetch boot images via TFTP over the network. On the x86
architecture, Genode supports the automatic generation of Pulsar configuration
files, which can be placed directly onto a TFTP server. Genode can be booted
via Pulsar using the following steps:
* On the x86 test machine, enable "PXE boot feature" in the BIOS.
* When booting, the machine will look for a DHCP server announcing a TFTP server.
So you need to make sure to have both the DHCP server and the TFTP server
configured such that the 'pulsar' binary will be loaded as PXE binary.
* After the PXE BIOS of the test machine has loaded and started the pulsar
binary, Pulsar will look on the TFTP server for a file called
'config-XX-XX-XX-XX-XX-XX', where the sequence of 'XX' corresponds to the
MAC address of the test machine.
For example, if the MAC of the network card is 01:02:03:04:05:06, Pulsar
would request a file called 'config-01-02-03-04-05-06'.
* Using this configuration file, we direct Pulsar to the configuration
generated by the run tool. I.e., it should look as follows
! root /tftpboot/nova
! config config-00-00-00-00-00-00
The lines above tell pulsar to load another config file, which contains the
actual configuration. To instruct the run script to actually generate the
'config-00-00-00-00-00-00' file, set the following environment variables in
your shell prior executing the run script:
! export PXE_TFTP_DIR_BASE=/tftpboot
! export PXE_TFTP_DIR_OFFSET=/nova
The two-staged configuration of Pulsar may look overly complicated at first
sight but has the benefit that the run tool does not need to know the MAC
address of the test machine in order to generate the Pulsar configuration
file.
* Create a symbolic link '/tftpboot/nova' pointing to the corresponding
Genode build directory.
* The next time 'make run/printf' is invoked,
the run script will generate the 'config-00-00-00-00-00-00' in
'/tftpboot/nova'.
* When rebooting the test machine, it will load and start the printf test.
TFTP boot using U-Boot
~~~~~~~~~~~~~~~~~~~~~~
Configure your U-Boot boot loader to load the images via TFTP.
The remainder of the procedure is similar to the description for x86 above.
On ARM platforms, the run tool automatically generates the uBoot image and
creates a symbolic link into the TFTP directory.
* Pandaboard:
! export PXE_TFTP_DIR_BASE=/tftpboot
! export PXE_TFTP_DIR_OFFSET=/panda
! ln -s <genode-build-dir> /tftpboot/panda
! RUN_OPT="--target uboot" make run/printf
* Arndale board:
! export PXE_TFTP_DIR_BASE=/tftpboot
! export PXE_TFTP_DIR_OFFSET=/arndale
! ln -s <genode-build-dir> /tftpboot/panda
! RUN_OPT="--target uboot" make run/printf
Output and reset with Intel's AMT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Most modern x86-based machines lack a COM port, which is normally used for
kernel debug messages as well as LOG messages printed by Genode's core.
However, Intel's Advanced Management Technology (AMT) can be used to obtain
the serial output of the test machine and to reset the test machine. To use
AMT with Genode's run tool, install the 'amtterm' package (version 1.3 is
known to work well) and set the following environment variables, specifying
the IP address of the test machine and the AMT password.
! export AMT_TEST_MACHINE_IP=XXX.XXX.XXX.XXX
! export AMT_TEST_MACHINE_PWD=XXXXXXXXX
Via setting the RUN_OPT environment variable, we instruct the run tool to use
AMT instead of Qemu. The following command will reset the test machine, the test
machine will load the binaries of the printf run script via PXE, and we will be
able to see the serial output of the test machine through Intel's AMT Serial
Over Line (SOL),
! RUN_OPT="--target amt" make run/printf
Output via a COM port (UART)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If the x86 test machine, Pandaboard or Arndale test board is connected
via UART, the run tool can use a specified command to interact with it.
For example, if the UART interface of the test machine is connected directly
to the host machine at /dev/ttyUSB3, and the picocom tool is available,
the following command can be used to establish a connection:
! RUN_OPT="--target serial --serial-cmd \"picocom -b 115200 /dev/ttyUSB3\"" make run/printf
Alternatively, if the board is connected to some remote machine, which exports
the corresponding serial line via TCP/IP, the socat tool can be used for
communicating with the remote test machine:
! RUN_OPT="--target serial --serial-cmd \"socat - tcp:10.0.0.1:2000\"" make run/printf
Reset via a IP power plug NETIO-230B from Koukaam
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
At Genode Labs, we use a NETIO-230B power plug to automate power-cycling ARM
boards. This power plug can be controlled over the network. For example, if
the Pandaboard is connected to power port 3, the following command will
automatically turn on the board when the run script is started:
! RUN_OPT="--target uboot --target reset --reset-port 2 --reset-ip 10.0.0.1 --reset-user admin --reset-passwd secret" make run/printf
The '--target reset' option can be combined with '--target uboot' to
instruct the run tool to boot via TFTP (as described above) and take care
of power cycling. When the run script has finished, the specified port
will be automatically switched off by the run tool.
Of course, the IP address settings, as well as the actual user name and
password, to access the NETIO-230B power plug, have to be adjusted accordingly.
Automated benchmarking
======================
With the '--target' features added to the run tool, the road is paved to
obtain benchmark results in an automated fashion. Currently, we are most
interested in exploring the network-performance characteristics of Genode.
Network performance can be explored at different levels. We started with
looking at raw driver performance, then looked at the overhead of separating
the network application from the device driver (and thereby introducing
inter-process communication overhead), and finally explored the effects
of the TCP/IP stack.
For pursuing the packet-level performance measurements, we crafted a library
called 'net-stat', which contains the application logic of a low-level
benchmark operating at network-packet level. This library has been
successively incorporated into the 'dde_ipxe' NIC driver and the 'usb_drv'
(NIC driver via ethernet-over-USB) to measure the raw driver performance
without any microkernel overhead or TCP/IP protocol overhead.
To see the influence of the inter-process communication, namely the
packet-stream interface employed by Genode's NIC-session interface,
we implanted the same net-stat library into a NIC-session client. This
experiment enables us to compare the operation of the NIC driver
with the operation of a NIC driver separated from the NIC
application.
The raw networking tests can be executed automatically using the set
of 'network_test_nic*.run' scripts located at 'os/run'.
The scenario sends raw ethernet packets from the host machine to the
target machine. Three tests are provided:
The 'network_test_nic_raw.run' test measures the net-stat-instrumented driver
(of usb_drv and net_drv respectively) to observe the raw receive performance.
The 'network_test_nic_raw_client.run' test implements the benchmark in a
NIC-session client connected to the NIC driver running as a separate
component whereas the NIC driver is not instrumented.
The 'network_test_nic_raw_bridge_client.run' test further adds a NIC bridge
in-between the driver and the NIC-session client.
In addition to analyzing the performance on a low level, we investigated
the effects of TCP/IP for the application performance. This topic is
covered in more detail in Section [TCP/IP performance].
Terminal infrastructure
#######################
Closely related to the quality-assurance measures detailed in the previous
section, there is the arising need to interact with increasingly complex system
scenarios in headless settings. In particular when executing tests remotely on a
development board, manual user-interaction via a GUI
becomes impractical. We vastly prefer a low-bandwidth textual interface
in such situations. But how should a textual user interface for dynamic
systems comprised of many components look like? This is particularly difficult
because most development boards are equipped with merely a single UART
connector.
On a normal Genode system, the UART connector is typically used
by the kernel debugger to print debugging output, or for the interactive
use of a debugger. This leaves no interface for interacting with Genode
components. So how can we expose complex scenarios, such as concurrently
running several instances of Genode subsystems, to the user?
Our solution consists of three parts: A pseudo UART driver for Genode
that uses the kernel debugger as back end, a terminal-multiplexing
facility running on the reference platform, and a command-line based
tool for interacting with Genode. By combining those, the user
can interact with the kernel debugger, a Genode command line, and the
consoles of executed Linux instances over a single serial connection.
The pseudo UART driver called kdb_uart_drv is a Genode service that
implements the 'Uart::Session' interface. Therefore, it can be combined
with all components that use the 'Uart::Session' or the 'Terminal::Session'
interfaces, for example the Noux runtime environment, the terminal_log
service (for displaying LOG messages via the terminal interface), L4Linux, or
programs linked against the 'libc_terminal' plugin. The kdb_uart_drv component
is located at 'os/src/drivers/uart/kdb'. It does not access a real UART device
but rather uses the user-level bindings of the kernel debugger to indirectly
read and write data over the UART interface.
[image kdb_uart_drv 65%]
The kdb_uart_drv driver used for sharing one UART among the kernel
debugger, core's LOG service, and a terminal client application
running on Genode.
Figure [kdb_uart_drv] illustrates the relationship between the kernel
debugger, core's LOG service, and kdb_uart_drv. Because write operations
target the kernel debugger directly, core's LOG service gets bypassed. Output
written to the kdb_uart_drv will directly appear at the terminal program of
the host system. Because kdb_uart_drv has
direct access to the host terminal, it can leverage all facilities of the host
terminal, in particular various escape sequences for terminal manipulations.
For reading from the kernel debugger, there is no way to block for UART input.
Hence, the kdb_uart_drv periodically polls for new input with a period of 20
milliseconds. If new input is available, the driver reads as many characters as
available at once. So the runtime overhead of polling is negligible. To test
kdb_uart_drv as individual component, there is a run script provided at
'os/run/kdb_uart_drv.run'.
Thanks to kdb_uart_drv, both the kernel debugger and Genode can
share one single UART connection. So we have a principal way to let the user
interact with a Genode component that uses the 'Terminal::Session' interface.
However, typical system scenarios should accommodate not just a single program
but multiple Linux instances and native Genode applications simultaneously,
each requiring a dedicated 'Terminal::Session'. Hence, we need a way to
multiplex the 'Terminal::Session' interface between those clients. Our
multiplexing solution comes in the form of a component called terminal_mux,
which we just introduced in the
[https://genode.org/documentation/release-notes/13.02#New_terminal_multiplexer - previous release].
It uses a single terminal connection to implement a text-based user interface
to multiple virtual terminal consoles.
[image terminal_mux 40%]
Operation of the terminal_mux service.
Figure [terminal_mux] depicts the basic functioning of this component. For
terminal_mux clients, the service implements the Linux terminal capabilities.
For doing that, it shares large parts of the implementation of the existing
Genode terminal program. For each client, terminal_mux renders the client
output into a client-specific text-screen buffer. So any number of clients can
perform output on terminal_mux concurrently. According to the selection by
the user, terminal_mux periodically translates one client buffer (the
foreground buffer) to escape sequences as understood by the host terminal. This
translation is performed using the ncurses library. The user can pick the
foreground buffer using an interactive menu that can be activated via the
keyboard shortcut _Control-x_.
By combining kdb_uart_drv with terminal_mux, we created a flexible way
to let the user interact with many Genode applications. The last part
missing for a real dynamic system is a text-based command interface to
start and stop Genode subsystems. This functionality is provided by the
new cli_monitor component located at 'os/src/app/cli_monitor'.
It uses the 'Terminal::Session' interface to present a simple interactive
command line with commands for starting and stopping Genode subsystems,
entering the kernel debugger, and showing status information. It provides
tab completion and inline help to make it easily explorable. The cli_monitor
component is integrated in the scenario of the 'terminal_mux.run' script
mentioned above. Because cli_command is a 'Terminal::Session' client, it can
be interfaced with terminal_mux. This composition is illustrated by Figure
[uart_overview].
[image uart_overview 100%]
Overview of the terminal infrastructure as employed in the
demonstration scenario.
Note that in some situations, e.g., when killing subsystems, the kernel, core,
or the init process may print LOG messages. Because those messages are
naturally not routed through terminal_log, they will interfere with the
operation of terminal_mux and thereby result in visible inconsistencies.
Pressing _Control-x_ will clear such artifacts. This will bring up the
terminal_mux menu, which implicitly triggers the redraw of the entire terminal.
Base framework
##############
The current release comes with incremental improvements of the MMIO framework
API and a new utility to ease the synchronized accesses to otherwise
unsynchronized class interfaces.
:MMIO framework improvements:
For native Genode device drivers, we consistently use our
[https://genode.org/documentation/release-notes/12.02#MMIO_access_framework - MMIO framework API].
These utilities help us to safeguard the access to individual bit fields of
memory-mapped device registers and cleanly separate the declaration of device
registers from the driver logic. During the increased use of the API, we
observe that the 'Genode::Mmio' class template operates mostly on addresses
that belong to dataspaces provided by core's IO_MEM service. Those dataspaces
are typically obtained via the 'Attached_io_mem_dataspace' convenience class,
which requests the dataspace and attaches it to the local address
space at once. To further reduce repetitive code, we introduced the new
'Attached_mmio' class (located at 'os/attached_mmio.h'), which handles the
common case of making the content of a IO_MEM dataspace available through
register definitions using the 'Mmio' utility. Furthermore, the MMIO framework
API has been enhanced with a variant of the 'Mmio::wait_for()' function that
waits for whole register values rather than bits.
:Synchronized interfaces:
Most Genode programs are multi-threaded, which makes the proper use of locks
inevitable. For most data structures, Genode does not implicitly manage the
locking but expects the user of the data structures to know what he is doing.
This way, we can avoid the locking overhead if a data structure is known to be
accessed by a single thread only. If accessed by multiple threads, we usually
wrap such data structures within an accessor interface that takes care of the
locking. For example, for the 'Allocator' interface, there exists a
corresponding 'Synchronized_allocator' interface wrapper. This technique works
well as long as the number of interfaces is low -- as is the case for Genode's
base API. However, as the wrapper code is for the most part pretty dumb, we'd
like to avoid it. Also, when using the Genode API to implement programs on
top, we do not anticipate manually creating such accessor wrappers. To ease
the creation of synchronized interfaces, we introduced the new
'Synced_interface' class template. It takes a pointer to an existing interface
and a lock as arguments. An instance of a 'Synced_interface' provides
synchronized access to the wrapped interface functions via the 'operator ()'.
Because the 'Synced_interface' does not provide any means to obtain the
unsynchronized version of the interface, once wrapped, the interface cannot be
misused by subsystems that get handed over a reference to a
'Synced_interface'. To see how to employ this utility, please have a look of
how we realize the synchronization within the Vancouver VMM (in particular,
the access to the motherboard).
Low-level OS infrastructure
###########################
TCP/IP performance
==================
On the course of the automated benchmarking described in Section
[Automated quality-assurance testing], we conducted the following steps
to enable benchmarks and to improve performance at the TCP/IP level.
At application level, we desire to compare our network performance with the
performance on GNU/Linux using commodity benchmarks. For this reason, netperf
has been ported to run as native Genode using the lwIP stack. This benchmark
allows us to systematically compare our results with those achieved by Linux.
The port of netperf is available in the ports repository.
In addition to running a commodity benchmark, we pursue synthetic benchmarks
that model the behaviour of typical application scenarios, for example, a
web server that receive many small requests. This is where the added
'test-ping_client' and 'test-ping_server' tests come into play. The test
is located at 'libports/src/test/lwip/pingpong'. It is used by the
series of 'network_test_*.run' scripts located at 'libports/run'. The
run scripts exercise the test in various scenarios and thereby allow us to
systematically explore the impact of the libc and NIC bridge on the
application performance.
# Using raw lwIP without the libc
# Like the first test, but with an instance of the NIC bridge in between the
test program and the driver.
# Using lwIP with the libc socket bindings
# Like the third test, but with NIC bridge added
To keep track of the lwIP development more closely, we switched to the
Git version of lwIP instead of using a source snapshot.
Furthermore, we incorporated "window scaling" support (RFC 1323) into our
version of lwIP as we identify the TCP window size as a limiting factor
of the TCP throughput achieved via lwIP.
C runtime
=========
We added support for "resolv" functionality to the libc_lwip_nic_dhcp plugin.
Normally, a file called 'resolv.conf' is expected to be located at '/etc'.
On Genode, however, we don't have a global file system, which makes this
way of configuration cumbersome. To ease the provision of a simple default
'resolv.conf' configuration, the plugin hands out the file as a virtual file.
The configuration automatically provides the DNS server address acquired by
lwIP via DHCP. If, for some reason, this policy is not desired, the feature
can be disabled via:
! <libc resolv="no" />
*Note that the configuration of the C runtime has changed*
To foster consistency of the libc configuration, we moved the static
network "interface" attributes into the 'libc' XML node. A new configuration
of static networking would look as follows:
! <libc ip_addr="..." netmask="..." gateway="..." />
Terminal
========
Genode's custom terminal implementation has been improved to better handle
widely used escape sequences.
The new version is able to handle two-argument SGR commands with
attribute/color arguments in any order, and supports the ED, EL0, and
CUB commands.
Because the terminal classes do not rely on any 3rd-party code, they
have been moved to the os repository at 'os/include/terminal'. This way,
we can use those classes by other components of the os repository such
as the new CLI monitor.
FS-LOG service
==============
Using the new FS-LOG service residing at 'libports/os/src/server/fs_log', log
messages of different processes can be redirected to files on a file-system
service. The assignment of processes to files can be expressed in the
configuration as follows:
! <start name="fs_log">
! <resource name="RAM" quantum="2M"/>
! <provides><service name="LOG"/></provides>
! <config>
! <policy label="noux" file="/noux.log" />
! <policy label="noux ->" file="/noux_process.log" />
! </config>
! </start>
In this example, all messages originating from the noux process are directed
to the file '/noux.log'. All messages originating from children of the noux
process end up in the file '/noux_process.log'.
Liquid FB
=========
Liquid FB is a virtual framebuffer service that uses the nitpicker GUI
server as back end. The virtual framebuffer is presented as a movable
window with a title bar. Until now, we used it primarily for demonstration
purposes, i.e., it is part of Genode's default demo scenario.
Thanks to our forthcoming adaptation of Qt5 to Genode, which requires a
very similar solution to interface Qt5's platform-abstraction layer (QPA) to
Genode, liquid FB got in the spotlight of this release.
First, we took the chance to update its configuration parameters to
become more consistent with similar services such as nit_fb. As liquid_fb was
originally conceived at a time when Genode's XML parser did not support
XML attributes, its configuration syntax used to be a bit arcane. This
has changed now. Apart from this cosmetic refinement, there are two prominent
new features: Support for resizing the framebuffer window with the
mouse and support for dynamic reconfiguration of the virtual framebuffer
via Genode's configuration mechanism.
When the liquid FB window gets resized by the user, the virtual framebuffer
emits a mode-changed signal to its client, which, in turn can handle the
event by re-acquiring the frame-buffer dataspace.
The added support for dynamic reconfiguration allows for changing the
properties of a liquid FB instance via Genode's configuration mechanism.
For example, the window position and size can be manipulated this way.
Furthermore, two new configuration options have been added. The
'resize_handle' option shows or hides the resize handle widget at the
lower-right window corner (by default, it is hidden). The 'decoration' option
defines whether window decorations should be visible (default is yes). Both
options can have the values "on" or "off".
3rd-party libraries
###################
The following 3rd-party libraries have been added or updated:
* To complement libSDL, we have added ports of SDL_ttf, SDL_image,
SDL_image, SDL_mixer, and SDL_loadso. Those additions to libSDL
are used by popular libSDL-based applications such as Tuxpaint.
They are now available at the libports repository.
* GNU FriBidi 0.19.5 added to the libports repository
* Qt4 updated to version 4.8.4
* zlib updated to version 1.2.8
Device drivers
##############
Unified driver names
====================
The growing diversity of supported hardware platforms calls for improved
conventions of how to name device drivers. Otherwise, run scripts that are
meant to support a wide range of platforms will eventually become more
and more complicated due to platform-dependent conditional configuration
snippets. For example, the default framebuffer drivers of the respective
platforms used to be called "vesa_drv" (for x86), "omap4_fb_drv", or "pl11x_drv".
In order to support the different platforms, run scripts that were otherwise
platform-agnostic had to explicitly deal with those differences.
To solve this issue, we introduced a generic SPEC values for device types, for
which a default driver is expected to exist. If a platform features a
framebuffer driver, it includes the SPEC value "framebuffer". On each
platform, the default driver for the respective device has the same name. So
each of "vesa_drv", "pl11x_drv", and "omap4_fb_drv" had been renamed to
"fb_drv". This is possible because the use of those drivers is mutually
exclusive.
The same convention has been applied to GPIO drivers as well. The
corresponding SPEC value is called "gpio". The driver binaries are called
"gpio_drv".
ATAPI
=====
LBA48 support has been added to the ATAPI driver. Thanks to Ivan Loskutov!
KDB UART driver for L4/Fiasco and Fiasco.OC
===========================================
The new KDB UART driver at 'os/src/drivera/uart/kdb' uses the kernel debugger
console as backend for input and output. This is useful in the case that only
one UART is available as described in Section [Terminal infrastructure].
Examples for using the kdb_uart_drv are available in the form of the run scripts
'ports-foc/run/l4linux.run' and 'os/run/kdb_uart_drv.run'.
Revised GPIO session interface
==============================
The original design of the GPIO session interface enabled the client of a
single session to interact with any number GPIO pins. Each function of the
interface took a GPIO number as first argument, which addressed the GPIO pin.
To simplify the interface and to enable fine-grained GPIO-assignment policies,
the interface has been changed to provide access to a single GPIO pin per
session only. At session creation time, the client specifies a single GPIO
pin, to which the session refers. This information can be evaluated for the
session routing. So access-control policies can be easily implemented per GPIO
pin. The server stores the pin as part of the session context and implicitly
uses the pin for operations on the session interface.
Furthermore, a generic driver interface for GPIO-class-device drivers
has been introduced. The new interface at 'os/include/gpio' alleviates the
need to implement the boilerplate code to interface the driver with Genode.
The existing GPIO drivers for OMAP4 and i.MX53 are the first beneficiaries of
these changes.
Exynos 5 SoC
============
After principally enabling the Exynos 5 SoC platform in the previous
release, we moved on with extending the device-driver coverage of this SoC. In
particular, we addressed USB networking, XHCI (USB-3), Gigabit networking over
USB-3, eMMC, and SATA.
The development of those device drivers follows our rationale that guided our
[https://genode.org/documentation/articles/pandaboard - previous work on the OMAP4 platform].
For the USB driver, we employed the device-driver-environment (DDE) approach
for reusing the Linux USB stack and the host controller drivers. In contrast,
the eMMC and SATA drivers are built as genuine Genode drivers with no
3rd-party code used.
Technically, the addition of Exynos-5 support to our USB driver was
an evolutionary step. It required us to add the corresponding EHCI
controller and to supply a few additions to the device-driver
environment. To simplify the driver, we decided to let the driver
rely on the platform initialization as performed by the U-Boot boot
loader. Since the initialization is performed during the boot process
already, there is no need to do this work twice. Because the platforms
supported by the USB driver become more and more diverse, we re-organized the
internal structure of the 'dde_linux' repository to keep those platforms well
separated. Furthermore, we reworked the memory management of the USB driver to
improve the utilization of the available RAM. The new solution employs Genode's
concept of managed dataspaces to manage a part of the local address-space
layout manually. This helps us to implement a fast translation of driver-local
virtual addresses to physical addresses as needed for issuing DMA requests.
The eMMC driver builds upon our protocol implementation for the SD-card
protocol, which was originally developed for the OMAP4 SD-card driver.
Because we kept the SD-card protocol implementation well separated
from the host-controller driver, it was possible to leverage parts of our
existing work for the eMMC driver. Because the eMMC protocol is an extension
of the SD-card protocol, however, we needed to enhance the protocol
implementation accordingly. The extension comprises support for the
MMC_SEND_EXT_CSD, MMC_SEND_OP_COND, and STOP_TRANSMISSION commands as well as
the MMC detection. The host controller driver was implemented from scratch
with the help of I/O access traces gathered from instrumenting the U-Boot boot
loader and the Linux kernel. The driver operates the eMMC in high-speed, 8-bit
mode at 52 MHz using DMA. The implementation can be found at
'os/src/drivers/sd_card/exynos5'.
The initial version of our new SATA driver for Exynos 5 has been implemented
from the ground up. Even though it is at an early stage, it has been
successfully tested with a UDMA-133 disk, e.g., our generic block test
is passed and the disk can be attached as a block device to an instance of
L4Linux.
Freescale i.MX SoC
==================
The support for the Freescale i.MX53 SoC has been extended by a number of
devices. All drivers reside in the os repository under the 'os/src/drivers'
subdirectory.
The general-purpose I/O (GPIO) driver located at 'gpio/imx53' implements the
revised GPIO-session interface.
The i.MX53 input driver provides support for the input devices featured on the
i.MX53 SABRE tablet. The tablet uses an Egalaxy touchscreen and Freescale's
MPR121 capacitative touch buttons. Both are supported by the new driver. The
driver is located at 'input/imx53'.
The new framebuffer driver for the i.MX53 quick-start board (QSB) as well as
the SABRE tablet comes with special support for using the
hardware overlay feature provided by the i.MX53 image processing unit (IPU)
Access to the overlay is implemented via an IPU-specific extension
of the framebuffer-session interface. To combine the driver well with
nitpicker using alpha-channels, optional support for double-buffering
is provided. The driver is located at 'framebuffer/imx53'.
As an abstraction of platform features that need to be accessed by
multiple drivers, a so-called platform driver has been introduced.
The platform driver safeguards the access to global resources such
as clocks and system-configuration bits. It can be found at 'platform/imx53'.
OMAP4 SoC
=========
The OMAP4 framebuffer driver used to support HDMI only, which was used
for connecting a display to the Pandaboard. To make the driver usable on
phones and tablets, the driver has been enhanced to support LCD output. Thanks
to Alexander Tarasikov for the patch and the insightful story about
[https://allsoftwaresucks.blogspot.com/2013/05/porting-genode-to-commercial-hardware.html - porting Genode to the B&N Nook HD+ tablet]!
USB
===
The USB driver of the 'dde_linux' repository has received substantial
improvements both feature-wise and under the hood.
First and foremost, the Linux device-driver environment, on which the
driver is based on, has been updated from kernel version 3.2 to version
3.9 as the latter version includes drivers for recent host controllers
such as DWC3 out of the box.
DWC3 is the host controller employed on the Exynos-5-based
Arndale platform for USB 3. We added the support needed to operate this
controller in XHCI mode and added support for Gigabit networking through
the ASIX AX88179 Gigabit-Ethernet Adapter as well as USB storage support.
Apart from extending the device-driver coverage, we revised the driver
internally. The back-end allocators for DMA buffers and normal memory have been
rewritten to allocate RAM more sparingly. Furthermore, we enabled the USB
driver for 64-bit x86 machines and improved the support for HID keyboards,
including the application of quirks to cherry keyboards.
*Note the change of the USB configuration*
With the addition of XHCI, the USB driver supports a growing number
of host controllers. In some situations, it is desirable to constrain the
driver to a subset of controllers only. For example, on the Arndale platform,
we desire to use a dedicated USB stack for XHCI, which operates completely
independent from the USB stack accessing USB-2. This way, gigabit networking
over USB-3 won't interfere with the operation of USB-2. To make this
possible, we added new configuration options to the USB driver.
With the new scheme, host controllers must be explicitly enabled in the
configuration. Supported config attributes are: 'uhci', 'ehci', and 'xhci'.
For example, a configuration snippet to enable UHCI and EHCI looks as
follows:
! <config uhci="yes" ehci="yes">
Updated iPXE device-driver environment
======================================
The iPXE device-driver environment was update to the most recent
iPXE upstream Git version in order to benefit from upstream improvements
of the Intel E1000 NIC driver.
Runtime environments
####################
Vancouver VMM on NOVA
=====================
Vancouver is the user-level virtual-machine monitor that accompanies the
NOVA hypervisor for hosting unmodified guest operating systems.
The most active line of development is led by Julian Stecklina at TU Dresden
via a fork called Seoul. In contrast to the original version of Vancouver,
this fork is open for outside contributions. Hence, it represents an ideal
platform for those parties with a stake in Vancouver to collaborate, i.e.,
the NUL userland, the NOVA runtime environment of TUD, and Genode.
In the current state of the transition, the Hip structure from Genode
is reused. String functions, which were formerly taken from NUL are now
provided by a stripped-down version of the C library called
'seoul_libc_support'. The nul/config.h is replaced by just using a constant
value in the one place where the file was needed.
The Genode-specific back ends of Vancouver, as largely introduced with the
previous Genode release, have been improved in several respects:
* CPUID 0x40000000: This instruction is issued by Linux when the KVM
guest support is compiled in. We have to return deterministic values to let
the Linux kernel survive.
* Replaced busy thread startup synchronization by proper locking.
* New locking scheme: We replaced the error-prone manual locking with the
use of the freshly introduced 'Synced_interface' for the motherboard and the
VCPU dispatcher. Also, all globally visible locks have been removed. They are
explicitly passed to subsystems only when needed.
* Improved PS/2 mouse back-end:
The previous version of the PS/2 mouse back end managed mouse-motion
events in a strange way, effectively throwing away most information
about the motion vector. Furthermore, the tracking of the mouse-button
states were missing. So drag-and-drop in a guest OS won't work. The new
version fixes those issues. For the transformation of input events to
PS/2 packets, the 'Genode::Register' facility is used, which greatly
simplifies the code.
L4Linux on Fiasco.OC
====================
We improved the memory management of L4Linux on Genode in two ways.
The first improvement is concerned about the upper limit of memory per Linux
instance. The corresponding discussion can be found at
[https://github.com/genodelabs/genode/issues/414 - issue #414].
We changed our L4Re emulation library to match the semantics of the original
L4Re more closely. Furthermore, we removed a heuristic in the L4Linux kernel,
which assumed that all kernel-local addresses above 0x8000000 refer to device
resources. In our version of L4Linux, there exist no MMIO resources. In
contrary, the virtual addresses above this addresses are used for normal
memory. By removing this artificial restriction with regard to the virtual
memory layout of the L4Linux kernel, we can host a larger kernel memory area.
The second improvement is concerned with the allocation of L4Linux
memory at Genode's core. Until now, L4Linux used to allocate its memory
as one contiguous RAM dataspace at core's RAM service. Core tries to
naturally align the allocation to improve the likelihood for large-page
mappings. So a dataspace is likely to be physically located at a
power-of-two boundary larger or equal than the dataspace size. For example,
the allocation of a 100 MiB RAM dataspace for a Linux instance will
be located at a 128 MiB boundary. If multiple of such allocations happen
sub-sequentially, this allocation strategy results in 28 MiB gaps between
100 MiB dataspaces. This memory cannot be used for large contiguous
allocations anymore. So even if the available memory capacity is far
larger than 100 MiB, an allocation of a 100 MiB block may fail.
To relieve this problem, we weakened the requirement for contiguous memory
by assembling L4Linux memory from multiple chunks of small dataspaces.
For example, by using a chunk size of 16 MiB, core's best-fit allocator
will have a better chance to find a more suited position for allocation
when aligning the block to a 16 MiB boundary compared to the allocation
of a larger block. Furthermore, slack memory can be used more efficiently
because smaller gaps (such as a 20 MiB gap) remain to be usable for L4Linux.
The discussion of this topic and the individual patch can be found at
[https://github.com/genodelabs/genode/issues/695 - issue #695].
Furthermore, the L4Linux block driver has been improved to support large
partitions.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Raspberry Pi
~~~~~~~~~~~~
Principal support for the Raspberry Pi platform has been added to the base-hw
kernel. The popular Raspberry Pi board is based on an ARMv6 Broadcom BCM2835
SoC. The current scope of the platform support comprises:
* IRQ controller driver: Because the interrupt controller uses a cascade of
registers, we settled on the following IRQ enumeration scheme.
IRQ numbers 0..7 refer to the basic IRQs.
IRQ numbers 8..39 refer to GPU IRQs 0..31.
IRQ numbers 40..71 refer to GPU IRQs 32..63.
* The kernel employs the so-called system timer for the preemptive scheduling.
* Core's LOG messages are printed over the PL011-based UART.
* The user-level timer driver uses the so-called ARM timer, which is a
slightly modified SP804 timer device.
Up to this point, a few device driver are missing to use Genode on the
Raspberry Pi in practice, most notably USB.
To build and run Genode on the Raspberry Pi, create a new build directory
via the 'create_builddir' tool, specifying 'hw_rpi' as platform.
User-level timer driver for Arndale platform
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
By adding our new Exynos 5250 PWM timer driver, the base-hw kernel can now
be used for executing meaningful scenarios on the Arndale board including
the USB stack and networking.
Linux
=====
Until now, Genode on Linux supported x86-based platforms only.
The newly added 'linux_arm' platform clears the way to run Genode directly on
Linux-based ARM platforms. Genode's entire software stack is supported,
including the dynamic linker, graphical applications, and Qt4.
As a known limitations, the libc 'setjmp()'/'longjmp()' doesn't currently
save/restore floating point registers.
Build system and tools
######################
The run tool has been enhanced as detailed in Section
[Automated quality-assurance testing].

995
doc/release_notes/13-08.txt Normal file
View File

@@ -0,0 +1,995 @@
===============================================
Release notes for the Genode OS Framework 13.08
===============================================
Genode Labs
The release of version 13.08 marks the 5th anniversary of the Genode OS
framework. We celebrate this anniversary with the addition of three major
features that we have much longed for, namely the port of Qt5 to Genode,
profound multi-processor support, and a light-weight event tracing
framework. Additionally, the new version comes with new device drivers for
SATA 3.0 and power management for the Exynos-5 SoC, improved virtualization
support on NOVA on x86, updated kernels, and integrity checks for
downloaded 3rd-party source code.
Over the course of the past five years, Genode's development was primarily
motivated by adding and cultivating features to make the framework fit for as
many application areas as possible. Now that we have a critical mass of
features, the focus on mere functionality does not suffice anymore. The
question of what Genode can do ultimately turns into the question of how well
Genode can do something: How stable is a certain workload? How does networking
perform? How does it scale to multi-processor systems? Because we are lacking
concise answers to these kind of questions, we have to investigate.
When talking about stability, our recently introduced automated testing
infrastructure makes us more confident than ever. Each night, over 200
automated tests are performed, covering various kernels and several hardware
platforms. All those tests are publicly available in the form of so-called run
scripts and are under continues development.
Regarding performance investigations, recently we have begun to benchmark
application performance focusing on network throughput. Interestingly, our
measurements reveal significant differences between the used kernels, but
also shortcomings in our software stack. For example, currently we see that
our version of lwIP performs poorly with gigabit networking. To thoroughly
investigate such performance issues, the current version adds support
for tracing the behaviour of Genode components. This will allow us to get a
profound understanding of all inter-component interaction that are on the
critical path for the performance of complex application-level workloads.
Thanks to the Genode architecture, we could come up with a strikingly simple,
yet powerful design for a tracing facility. Section
[Light-weight event tracing] explains how it works.
When it comes to multi-processor scalability, we used to shy away from such
inquiries because, honestly, we haven't paid much consideration to it. This
view has changed by now. With the current release, we implemented the
management of CPU affinities right into the heart of the framework, i.e.,
Genode's session concept. Additionally, we cracked a damn hard nut by enabling
Genode to use multiple CPUs on the NOVA hypervisor. This kernel is by far the
most advanced Open-Source microkernel for the x86 architecture. However,
NOVA's MP model seemed to inherently contradict with the API design of Genode.
Fortunately, we found a fairly elegant way to go forward and we're able to
tame the beast. Section [Enhanced multi-processor support] goes into more
detail.
Functionality-wise, we always considered the availability of Qt on Genode as a
big asset. With the current release, we are happy to announce that we finally
made the switch from Qt4 to Qt5. Section [Qt5 available on all kernels] gives
insights into the challenges that we faced during porting work.
In addition to those highlights, the new version comes with improvements all
over the place. To name a few, there are improved support for POSIX threads,
updated device drivers, an updated version of the Fiasco.OC kernel and
L4Linux, and new device drivers for Exynos-5. Finally, the problem of
verifying the integrity of downloaded 3rd-party source codes has been
addressed.
Qt5 available on all kernels
############################
Since its integration with Genode version 9.02, Qt4 is regarded as
one of the most prominent features of the framework. For users, combining Qt
with Genode makes the world of sophisticated GUI-based end-user applications
available on various microkernels. For Genode developers, Qt represents by far
the most complex work load natively executed on top of the framework API,
thereby stressing the underlying system in any way imaginable. We have been
keeping an eye on Qt version 5 for a while and highly anticipate the direction
where Qt is heading. We think that the time is right to leave Qt4 behind to
embrace Qt5 for Genode.
For the time being, both Qt4 and Qt5 are available for Genode, but Qt4 is
declared as deprecated and will be removed with the upcoming version 13.11.
Since Qt5 is almost API compatible to Qt4, the migration path is relatively
smooth. So we recommend to move your applications over to Qt5 during the
next release cycle.
In the following, we briefly describe the challenges we faced while adding Qt5
support to Genode, point you to the place where to find Qt5 in the source
tree, and give quick-start instructions for getting a Qt5 application
scenario running.
We found that the biggest architectural difference between version 4 and
version 5 is the use of the so-called Qt Platform Abstraction (QPA) interface,
which replaces the former Qt Window System (QWS).
Moving from QWS to QPA
======================
With Qt4, we relied on QWS
to perform the window handling. A Qt4 application used to create a session to
Genode's GUI server (called nitpicker) and applied its graphical output onto a
virtual framebuffer. The virtual framebuffer was not visible per se. To make
portions of the virtual framebuffer visible on screen, the application had to
create so-called nitpicker views. A view is a rectangular area of the physical
screen that displays a (portion of) the virtual framebuffer. The position,
size, and stacking order of views is managed by the application. For each Qt
window, the application would simply create a corresponding nitpicker view and
maintain the consistency of the view with the geometry of the Qt window. Even
though each Qt application seemingly operated as a full-screen application
with the windows managed by the application-local QWS, the use of nitpicker
views still allowed the integration of any number of Qt applications into one
windowed environment.
With the advent of compositing window managers, the typical way of how an
application interacts with the window system of the OS changed. Whereas
old-generation GUIs relied on a tight interplay of the application with the
window server in order to re-generate newly exposed window regions whenever
needed (e.g., revealing the window content previously covered by another
window), the modern model of a GUI server keeps all pixels of all windows in
memory regardless of whether the window is visible or covered by other
windows. The use of one pixel buffer per window seems wasteful with respect to
memory usage when many large windows are overlapping each other. On the other
hand, this technique largely simplifies GUI servers and makes the
implementation of fancy effects, like translucent windows, straight forward.
Since memory is cheap, the Qt developers abandoned the old method and fully
embraced the buffer-per-window approach by the means of QPA.
For Genode, we faced the challenge that we don't have a window server
in the usual sense. With nitpicker, we have a GUI server, but with a more
radical design. In particular, nitpicker leaves the management of window
geometries and stacking to the client. In contrast, QPA expects the window
system to provide both means for a user to interactively change the window
layout and a way for an application to define the properties (such as the
geometry, title, and visibility) of its windows.
The obviously missing piece was the software component that deals with window
controls. Fortunately, we already have a bunch of native nitpicker applications
that come with client-side window controls, in particular the so-called liquid
framebuffer (liquid_fb). This nitpicker client presents a virtual framebuffer
in form of a proper window on screen and, in turn, provides a framebuffer and
input service. These services can be used by other Genode processes, for
example, another nested instance of nitpicker.
This way, liquid_fb lends itself to be the interface between the nitpicker
GUI server and QPA.
For each QPA window, the application creates a new liquid_fb instance as a
child process. The liquid_fb instance will request a dedicated nitpicker
session, which gets routed through the application towards the parent of the
application, which eventually routes the request to the nitpicker GUI server.
Finally, the liquid_fb instance announces its input and framebuffer services
to its parent, which happens to be the application. Now, the application is
able to use those services in order to access the window. Because the
liquid_fb instances are children of the application, the application can
impose control over those. In particular, it can update the liquid_fb
configuration including the window geometry and title at any time. Thanks to
Genode's dynamic reconfiguration mechanism, the liquid_fb instances are able
to promptly respond to such reconfigurations.
Combined, those mechanisms give the application a way to receive user input
(via the input services provided by the liquid_fb instances), perform
graphical output (via the virtual framebuffers provided by the liquid_fb
instances), and define window properties (by dynamically changing the
respective liquid_fb configurations). At the same time, the user can use
liquid_fb's window controls to move, stack, and resize application windows as
expected.
[image qt5_screenshot]
Steps of porting Qt5
====================
Besides the switch to QPA, the second major change was related to the build
system. For the porting work, we use a Linux host system to obtain the
starting point for the build rules. The Qt4 build system would initially
generate all Makefiles, which could be inspected and processed at once. In
contrast, Qt5 generates Makefiles during the build process whenever needed.
When having configured Qt for Genode, however, the build on Linux will
ultimately fail. So the much-desired intermediate Makefiles won't be created.
The solution was to have 'configure' invoke 'qmake -r' instead of 'qmake'.
This way, qmake project files will be processed recursively. A few additional
tweaks were needed to avoid qmake from backing out because of missing
dependencies (qt5_configuration.patch). To enable the build of the Qt tools
out of tree, qmake-specific files had to be slightly adapted
(qt5_tools.patch). Furthermore, qtwebkit turned out to use code-generation
tools quite extensively during the build process. On Genode, we perform this
step during the 'make prepare' phase when downloading and integrating the Qt
source code with the Genode source tree.
For building Qt5 on Genode, we hit two problems. First, qtwebkit depends on
the ICU (International Components for Unicode) library, which was promptly
ported and can be found in the libports repository. Second, qtwebkit
apparently dropped the support of the 'QThread' API in favor of
POSIX-thread support only. For this reason, we had to extend the coverage
of Genode's pthread library to fulfill the needs of qtwebkit.
Once built, we entered the territory of debugging problems at runtime.
* We hit a memory-corruption problem caused by an assumption of 'QArrayData'
with regard to the alignment of memory allocated via malloc. As a
work-around, we weakened the assumptions to 4-byte alignment
(qt5_qarraydata.patch).
* Page faults in QWidgetAnimator caused by
use-after-free problems. Those could be alleviated by adding pointer
checks (qt5_qwidgetanimator.patch).
* Page faults caused by the slot function 'QWidgetWindow::updateObjectName()'
with a 'this' pointer of an incompatible type 'QDesktopWidget*'.
As a workaround, we avoid this condition by delegating the
'QWidgetWindow::event()' that happened to trigger the slot method to
'QWindow' (base class of 'QWidgetWindow') rather than to a
'QDesktopWidget' member (qt5_qwidgetwindow.patch).
* We observed that Arora presented web sites incomplete, or including HTTP
headers. During the evaluation of HTTP data, a signal was sent to another
thread, which activated a "user provided download buffer" for optimization
purposes. On Linux, the receiving thread was immediately scheduled and
everything went fine. However, on some kernels used by Genode, scheduling
is different, so that the original thread continued to execute a bit longer,
ultimately triggering a race condition. As a workaround, we disabled the
"user provided download buffer" optimization.
* Page faults in the JavaScript engine of Webkit. The JavaScript
'RegExp.exec()' function returned invalid string objects. We worked around
this issue by deactivating the JIT compiler for the processing of
regular expressions (ENABLE_YARR_JIT).
The current state of the Qt5 port is fairly complete. It covers the core, gui,
jscore, network, script, scriptclassic, sql, ui, webcore, webkit, widgets,
wtf, and xml modules. That said, there are a few known limitations and
differences compared to Qt4. First, the use of one liquid_fb instance per
window consumes more memory compared to the use of QWS in Qt4. Furthermore,
external window movements are not recognized by our QPA implementation yet.
This can cause popup menus to appear at unexpected positions. Key repeat is
not yet handled. The 'QNitpickerViewWidget' is not yet adapted to Qt5. For this
reason, qt_avplay is not working yet.
Test drive
==========
Unlike Qt4, which was hosted in the dedicated 'qt4' repository, Qt5 is
integrated in the libports repository. It can be downloaded and integrated
into the Genode build system by issuing 'make prepare' from within the
libports repository. The Qt5 versions of the known Qt examples are located at
libports/src/app/qt5. Ready-to-use run scripts for those examples are available
at libports/run.
Migration away from Qt4 to Qt5
==============================
The support for Qt4 for Genode has been declared as deprecated. By default,
it's use is inhibited to avoid name aliasing problems between both versions.
Any attempt to build a qt4-based target will result in a message:
!Skip target app/qt_launchpad because it requires qt4_deprecated
To re-enable the use of Qt4, the SPEC value qt4_deprecated must be defined
manually for the build directory:
!echo "SPECS += qt4_deprecated" >> etc/specs.conf
We will keep the qt4 repository in the source tree during the current
release cycle. It will be removed with version 13.11.
Light-weight event tracing
##########################
With Genode application scenarios getting increasingly sophisticated,
the need for thorough performance analysis has come into spotlight.
Such scenarios entail the interaction of many components.
For example, with our recent work on optimizing network performance, we
have to consider several possible attack points:
* Device driver: Is the device operating in the proper mode? Are there
CPU-intensive operations such as allocations within the critical path?
* Interface to the device driver: How frequent are context switches between
client and device driver? Is the interface designed appropriately for
the access patterns?
* TCP/IP stack: How does the data flow from the raw packet level to the
socket level? How dominant are synchronization costs between the involved
threads? Are there costly in-band operations performed, e.g., dynamic
memory allocations per packet?
* C runtime: How does integration of the TCP/IP stack with the C runtime
work, for example how does the socket API interfere with timeout
handling during 'select' calls?
* Networking application
* Timer server: How often is the timer consulted by the involved components?
What is the granularity of timeouts and thereby the associated costs for
handling them?
* Interaction with core: What is the profile of the component's interaction
with core's low-level services?
This example is just an illustration. Most real-world performance-critical
scenarios have a similar or even larger scope. With our traditional
tools, it is hardly possible to gather a holistic view of the scenario. Hence,
finding performance bottlenecks tends to be a series of hit-and-miss
experiments, which is a tiresome and costly endeavour.
To overcome this situation, we need the ability to gather traces of component
interactions. Therefore, we started investigating the design of a tracing
facility for Genode one year ago while posing the following requirements:
* Negligible impact on the performance, no side effects:
For example, performing a system call per traced event
is out of question because this would severely influence the flow of
control (as the system call may trigger the kernel to take a scheduling
decision) and the execution time of the traced code, not to speak of
the TLB and cache footprint.
* Kernel independence: We want to use the same tracing facility across
all supported base platforms.
* Accountability of resources: It must be clearly defined where the
resources for trace buffers come from. Ideally, the tracing tool should be
able to dimension the buffers according to its needs and, in turn, pay for
the buffers.
* Suitable level of abstraction: Only if the trace contains information at
the right level of abstraction, it can be interpreted for large scenarios.
A counter example is the in-kernel trace buffer of the Fiasco.OC kernel,
which logs kernel-internal object names and a few message words when tracing
IPC messages, but makes it almost impossible to map this low-level
information to the abstraction of the control flow of RPC calls. In
contrast, we'd like to capture the names of invoked RPC calls (which is an
abstraction level the kernel is not aware of). This requirement implies the
need to have useful trace points generated automatically. Ideally those
trace points should cover all interactions of a component with the outside
world.
* (Re-)definition of tracing policies at runtime: The
question of which information to gather when a trace point is passed
should not be solely defined at compile time. Instead of changing static
instrumentations in the code, we'd prefer to have a way to configure
the level of detail and possible conditions for capturing events at runtime,
similar to dtrace. This way, a series of different hypotheses could be
tested by just changing the tracing policy instead of re-building and
rebooting the entire scenario.
* Straight-forward implementation: We found that most existing tracing
solutions are complicated. For example, dtrace comes with a virtual
machine for the sandboxed interpretation of policy code. Another typical
source of complexity is the synchronization of trace-buffer accesses.
Because for Genode, low TCB complexity is of utmost importance, the
simplicity of the implementation is the prerequisite to make it an
integral part of the base system.
* Support for both online and offline analysis of traces.
We are happy to report to have come up with a design that meets all those
requirements thanks to the architecture of Genode. In the following, we
present the key aspects of the design.
The tracing facility comes in the form of a new TRACE service implemented
in core. Using this service, a TRACE client can gather information about
available tracing subjects (existing or no-longer existing threads),
define trace buffers and policies and assign those to tracing subjects,
obtain access to trace-buffer contents, and control the tracing state
of tracing subjects. When a new thread is created via a CPU session, the
thread gets registered at a global registry of potential tracing sources. Each
TRACE service manages a session-local registry of so-called trace subjects.
When requested by the TRACE client, it queries new tracing sources from the
source registry and obtains references to the corresponding threads. This way,
the TRACE session becomes able to control the thread's tracing state.
To keep the tracing overhead as low as possible, we assign a separate trace
buffer to each individually traced thread. The trace buffer is a shared memory
block mapped to the virtual address space of the thread's process. Capturing
an event comes down to a write operation into the thread-local buffer. Because
of the use of shared memory for the trace buffer, no system call is needed and
because the buffer is local to the traced thread, there is no need for
synchronizing the access to the buffer. When no tracing is active, a thread
has no trace buffer. The buffer gets installed only when tracing is started.
The buffer is not installed magically from the outside of the traced process
but from the traced thread itself when passing a trace point. To detect
whether to install a new trace buffer, there exists a so-called trace-control
dataspace shared between the traced process and its CPU service. This
dataspace contains control bits for each thread created via the CPU session.
The control bits are evaluated each time a trace point is passed by the
thread. When the thread detects a change of the tracing state, it actively
requests the new trace buffer from the CPU session and installs it into its
address space. The same technique is used for loading the code for tracing
policies into the traced process. The traced thread actively checks for
policy-version updates by evaluating the trace-control bits. If an update is
detected, the new policy code is requested from the CPU session. The policy
code comes in the form of position-independent code, which gets mapped into
the traced thread's address space by the traced thread itself. Once mapped,
a trace point will call the policy code. When called, the policy
module code returns the data to be captured into the trace buffer. The
relationship between the trace monitor (the client of TRACE service), core's
TRACE service, core's CPU service, and the traced process is depicted in
Figure [trace_control].
[image trace_control]
There is one trace-control dataspace per CPU session, which gets accounted
to the CPU session's quota. The resources needed for the
trace-buffer dataspaces and the policy dataspaces are paid-for by the
TRACE client. On session creation, the TRACE client can specify the amount
of its own RAM quota to be donated to the TRACE service in core. This
enables the TRACE client to define trace buffers and policies of arbitrary
sizes, limited only by its own RAM quota.
In Genode, the interaction of a process with its outside world is
characterized by its use of inter-process communication, namely synchronous
RPC, signals, and access to shared memory. For the former two types of
inter-process communication, Genode generates trace points automatically. RPC
clients generate trace points when an RPC call is issued and when a call
returned. RPC servers generate trace points in the RPC dispatcher, capturing
incoming RPC requests as well as RPC replies. Thanks to Genode's RPC
framework, we are able to capture the names of the RPC functions in the RPC
trace points. This information is obtained from the declarations of the RPC
interfaces. For signals, trace points are generated for submitting and
receiving signals. Those trace points form a useful base line for gathering
tracing data. In addition, manual trace points can be inserted into the code.
State of the implementation
===========================
The implementation of Genode's tracing facility is surprisingly low complex.
The addition to the base system (core as well as the base library) are
merely 1500 lines of code. The mechanism works across all base platforms.
Because the TRACE client provides the policy code and trace buffer to the
traced thread, the TRACE client imposes ultimate control over the traced
thread. In contrast to dtrace, which sandboxes the trace policy, we express
the policy module in the form of code executed in the context of the traced
thread. However, in contrast to dtrace, such code is never loaded into a large
monolithic kernel, but solely into the individually traced processes. So the
risk of a misbehaving policy is constrained to the traced process.
In the current form, the TRACE service of core should be considered as a
privileged service because the trace-subject namespace of each session
contains all threads of the system. Therefore, TRACE sessions should be routed
only for trusted processes. In the future, we plan to constrain the
namespaces for tracing subjects per TRACE session.
The TRACE session interface is located at base/include/trace_session/.
A simple example for using the service is available at os/src/test/trace/
and is accompanied with the run script os/run/trace.run. The test
demonstrates the TRACE session interface by gathering a trace of a thread
running locally in its address space.
Enhanced multi-processor support
################################
Multi-processor (MP) support is one of those features that most users take for
granted. MP systems are so ubiquitous, even on mobile platforms, that a
limitation to utilizing a single CPU only is almost a fallacy.
That said, MP support in operating systems is hard to get right. For this
reason, we successively deferred the topic on the agenda of Genode's
road map.
For some base platforms such as the Linux or Codezero kernels, Genode
always used to support SMP because the kernel would manage the affinity
of threads to CPU cores transparently to the user-level process. So on
these kernels, there was no need to add special support into the framework.
However, on most microkernels, the situation is vastly different. The
developers of such kernels try hard to avoid complexity in the kernel and
rightfully argue that in-kernel affinity management would contribute to kernel
complexity. Another argument is that, in contrast to monolithic kernels that
have a global view on the system and an "understanding" of the concerns of the
user processes, a microkernel is pretty clueless when it comes to the roles
and behaviours of individual user-level threads. Not knowing whether a thread
works as a device driver, an interactive program, or a batch process, the
microkernel is not in the position to form a reasonably useful model of the
world, onto which it could intelligently apply scheduling and affinity
heuristics. In fact, from the perspective of a microkernel, each thread does
nothing else than sending and receiving messages, and causing page faults.
For these reasons, microkernel developers tend to provide the bootstrapping
procedure for the physical CPUs and a basic mechanism to assign
threads to CPUs but push the rest of the problem to the user space, i.e.,
Genode. The most straight-forward way would make all physical CPUs visible
to all processes and require the user or system integrator to assign
physical CPUs when a thread is created. However, on the recursively
structured Genode system, we virtualize resources at each level, which
calls for a different approach. Section [Management of CPU affinities]
explains our solution.
When it comes to inter-process communication on MP systems, there is a
certain diversity among the kernel designs. Some kernels allow the user land
to use synchronous IPC between arbitrary threads, regardless of whether both
communication partners reside on the same CPU or on two different CPUs. This
convenient model is provided by Fiasco.OC. However, other kernels do not offer
any synchronous IPC mechanism across CPU cores at all, NOVA being a poster
child of this school of thought. If a user land is specifically designed for
a particular kernel, those peculiarities can be just delegated to the
application developers. For example, the NOVA user land called NUL is designed
such that a recipient of IPC messages spawns a dedicated thread on each
physical CPU. In contrast, Genode is meant to provide a unified API that
works well across various different kernels. To go forward, we had four
options:
# Not fully supporting the entity of API semantics across all base platforms.
For example, we could stick with the RPC API for synchronous communication
between threads. Programs just would happen to fail on some base platforms
when the called server resides on a different CPU. This would effectively
push the problem to the system integrator. The downside would be the
sacrifice of Genode's nice feature that a program developed
on one kernel usually works well on other kernels without any changes.
# Impose the semantics provided by the most restrictive kernel onto all
users of the Genode API. Whereas this approach would facilitate that
programs behave consistently across all base platforms, the restrictions
would be artificially imposed onto all Genode users, in particular the
users of kernels with less restrictions. Of course, we don't change
the Genode API lightheartedly, which attributes to our hesitance to go
into this direction.
# Hiding kernel restrictions behind the Genode API. This approach could come
in many different shapes. For example, Genode could transparently spawn a
thread on each CPU when a single RPC entrypoint gets created, following
the model of NUL. Or Genode could emulate synchronous IPC using the core
process as a proxy.
# Adapting the kernel to the requirements of Genode. That is, persuading
kernel developers to implement the features we find convenient, i.e., adding
a cross-CPU IPC feature to NOVA. History shows that our track record in
doing that is not stellar.
Because each of those options is like opening a different can of worms, we
used to defer the solution of the problem. Fortunately, however, we finally
kicked off a series of practical experiments, which led to a fairly elegant
solution, which is detailed in Section
[Adding multi-processor support to Genode on NOVA].
Management of CPU affinities
============================
In line with our experience of supporting
[https://genode.org/documentation/release-notes/10.02#Real-time_priorities - real-time priorities]
in version 10.02, we were seeking a way to express CPU affinities such that
Genode's recursive nature gets preserved and facilitated. Dealing with
physical CPU numbers would contradict with this mission. Our solution
is based on the observation that most MP systems have topologies that can
be represented on a two-dimensional coordinate system. CPU nodes close
to each other are expected to have closer relationship than distant nodes.
In a large-scale MP system, it is natural to assign clusters of closely
related nodes to a given workload. Genode's architecture is based on the
idea to recursively virtualize resources and thereby lends itself to the
idea to apply this successive virtualization to the problem of clustering
CPU nodes.
In our solution, each process has a process-local view on a so-called affinity
space, which is a two-dimensional coordinate space. If the process creates a
new subsystem, it can assign a portion of its own affinity space to the new
subsystem by imposing a rectangular affinity location to the subsystem's CPU
session. Figure [affinity_space] illustrates the idea.
[image affinity_space]
Following from the expression of affinities as a rectangular location within a
process-local affinity space, the assignment of subsystems to CPU nodes
consists of two parts, the definition of the affinity space dimensions as used
for the process and the association of sub systems with affinity locations
(relative to the affinity space). For the init process, the affinity space is
configured as a sub node of the config node. For example, the following
declaration describes an affinity space of 4x2:
! <config>
! ...
! <affinity-space width="4" height="2" />
! ...
! </config>
Subsystems can be constrained to parts of the affinity space using the
'<affinity>' sub node of a '<start>' entry:
! <config>
! ...
! <start name="loader">
! <affinity xpos="0" ypos="1" width="2" height="1" />
! ...
! </start>
! ...
! </config>
As illustrated by this example, the numbers used in the declarations for this
instance of the init process are not directly related to physical CPUs. If
the machine has just two cores, init's affinity space would be mapped
to the range [0,1] of physical CPUs. However, in a machine
with 16x16 CPUs, the loader would obtain 8x8 CPUs with the upper-left
CPU at position (4,0). Once a CPU session got created, the CPU client can
request the physical affinity space that was assigned to the CPU session
via the 'Cpu_session::affinity()' function. Threads of this CPU session
can be assigned to those physical CPUs via the 'Cpu_session::affinity()'
function, specifying a location relative to the CPU-session's affinity space.
Adding multi-processor support to Genode on NOVA
================================================
The NOVA kernel has been supporting MP systems for a long time. However
Genode did not leverage this capability until now. The main reason was that
the kernel does not provide - intentionally by the kernel developer - the
possibility to perform synchronous IPC between threads residing on different
CPUs.
To cope with this situation, Genode servers and clients would need to make
sure to have at least one thread on a common CPU in order to communicate.
Additionally, shared memory and semaphores could be used to communicate across
CPU cores. Both options would require rather fundamental changes to the Genode
base framework and the API. An exploration of this direction should in any
case be pursued in evolutionary steps rather than as one big change, also
taking into account that other kernels do not impose such hard requirements on
inter-CPU communication. To tackle the challenge, we conducted a series of
experiments to add some kind of cross-CPU IPC support to Genode/NOVA.
As a general implication of the missing inter-CPU IPC, messages between
communication partners that use disjoint CPUs must take an indirection through
a proxy process that has threads running on both CPUs involved. The sender
would send the message to a proxy thread on its local CPU, the proxy process
would transfer the message locally to the CPU of the receiver by using
process-local communication, and the proxy thread on the receiving CPU would
deliver the message to the actual destination. We came up with three options
to implement this idea prototypically:
# Core plays the role of the proxy because it naturally has access to all
CPUs and emulates cross-CPU IPC using the thread abstractions of the
Genode API.
# Core plays the role of the proxy but uses NOVA system calls directly
rather than Genode's thread abstraction.
# The NOVA kernel acts as the proxy and emulates cross-CPU IPC directly
in the kernel.
After having implemented the first prototypes, we reached the following
conclusions.
For options 1 and 2 where core provides this service: If a client can not
issue a local CPU IPC, it asks core - actually the pager of the client
thread - to perform the IPC request. Core then spawns or reuses a proxy thread
on the target CPU and performs the actual IPC on behalf of the client. Option
1 and 2 only differ in respect to code size and the question to whom to
account the required resources - since a proxy thread needs a stack and some
capability selectors.
As one big issue for option 1 and 2, we found that in order to delegate
capabilities during the cross-CPU IPC, core has to receive capability mappings
to delegate them to the target thread. However, core has no means to know
whether the capabilities must be maintained in core or not. If a capability is
already present in the target process, the kernel would just translate the
capability to the target's capability name space. So core wouldn't need to
keep it. In the other case where the target receives a prior unknown
capability, the kernel creates a new mapping. Because the mapping gets
established by the proxy in core, core must not free the capability.
Otherwise, the mapping would disappear in the target process. This means that
the use of core as a proxy ultimately leads to leaking kernel resources
because core needs to keep all transferred capabilities, just for the case a
new mapping got established.
For option 3, the same general functionality as for option 1 and 2 is
implemented in the kernel instead of core. If a local CPU IPC call
fails because of a BAD_CPU kernel error code, the cross-CPU IPC extension
will be used. The kernel extension creates - similar as to option 1 and 2 - a
semaphore (SM), a thread (EC), and a scheduling context (SC) on the remote
CPU and lets it run on behalf of the caller thread. The caller thread
gets suspended by blocking on the created semaphore until the remote EC has
finished the IPC. The remote proxy EC reuses the UTCB of the suspended caller
thread as is and issues the IPC call. When the proxy EC returns, it wakes up
the caller via the semaphore. Finally, the proxy EC and SC de-schedule
themselves and the resources get to be destroyed later on by the kernel's RCU
mechanism. Finally, when the caller thread got woken up, it takes care to
initiate the deconstruction of the semaphore.
The main advantage of option 3 compared to options 1 and 2 is that we don't
have to keep and track the capability delegations during a cross-CPU IPC.
Furthermore, we do not have potentially up to two additional address space
switches per cross-CPU IPC (from client to core and core to the server).
Additionally, the UTCB of the caller is reused by the proxy EC and does not
need to be allocated separately as for option 1 and 2.
For these reasons, we decided to go for the third option. From Genode's API
point of view, the use of cross-CPU IPC is completely transparent. Combined
with the affinity management described in the previous section, Genode/NOVA
just works on MP systems.
As a simple example for using Genode on MP systems, there is a ready-to-use
run script available at base/run/affinity.run.
Base framework
##############
Affinity propagation in parent, root, and RPC entrypoint interfaces
===================================================================
To support the propagation of CPU affinities with session requests, the
parent and root interfaces had to be changed. The 'Parent::Session'
and 'Root::Session' take the affinity of the session as a new argument.
The affinity argument contains both the dimensions of the affinity space
used by the session and the session's designated affinity location within
the space. The corresponding type definitions can be found at
base/affinity.h.
Normally, the 'Parent::Session' function is not used directly but indirectly
through the construction of a so-called connection object, which represents an
open session. For each session type there is a corresponding connection type,
which takes care of assembling the session-argument string by using the
'Connection::session()' convenience function. To maintain API compatibility,
we kept the signature of the existing 'Connection::session()' function using a
default affinity and added a new overload that takes the affinity as
additional argument. Currently, this overload is used in
cpu_session/connection.h.
For expressing the affinities of RPC entrypoints to CPUs within the affinity
space of the server process, the 'Rpc_entrypoint' takes the desired affinity
location of the entrypoint as additional argument. For upholding API
compatibility, the affinity argument is optional.
CPU session interface
=====================
The CPU session interface underwent changes to accommodate the new event
tracing infrastructure and the CPU affinity management.
Originally the 'Cpu_session::num_cpus()' function could be used to
determine the number of CPUs available to the session. This function
has been replaced by the new 'affinity_space' function, which returns the
bounds of the CPU session's physical affinity space. In the simplest case of
an SMP machine, the affinity space is one-dimensional where the width
corresponds to the number of CPUs. The 'affinity' function, which is used to
bind a thread to a specified CPU, has been changed to take an affinity
location as argument. This way, the caller can principally express the
affiliation of the thread with multiple CPUs to guide load-balancing in a CPU
service.
New TRACE session interface
===========================
The new event tracing mechanism as described in Section
[Light-weight event tracing] is exposed to Genode processes in the form
of the TRACE service provided by core. The new session interface
is located under base/include/trace_session/. In addition to the new session
interface, the CPU session interface has been extended with functions for
obtaining the trace-control dataspace for the session as well as the trace
buffer and trace policy for a given thread.
Low-level OS infrastructure
###########################
Event-driven operation of NIC bridge
====================================
The NIC bridge component multiplexes one physical network device among
multiple clients. It enables us to multiplex networking on the network-packet
level rather than the socket level and thereby take TCP/IP out of the
critical software stack for isolating network applications. As it
represents an indirection in the flow of all networking packets, its
performance is important.
The original version of NIC bridge was heavily multi-threaded. In addition to
the main thread, a timer thread, and a thread for interacting with the NIC
driver, it employed one dedicated thread per client. By merging those flows of
control into a single thread, we were able to significantly reduce the number
of context switches and improve data locality. These changes reduced the
impact of the NIC bridge on the packet throughput from 25% to 10%.
Improved POSIX thread support
=============================
To accommodate qtwebkit, we had to extend Genode's pthread library with
working implementations of condition variables, mutexes, and thread-local
storage. The implemented functions are attr_init, attr_destroy, attr_getstack,
attr_get_np, equal, mutex_attr, mutexattr_init, mutexattr_destroy,
mutexattr_settype, mutex_init, mutex_destroy, mutex_lock, mutex_unlock,
cond_init, cond_timedwait, cond_wait, cond_signal, cond_broadcast, key_create,
setspecific, and getspecific.
Device drivers
##############
SATA 3.0 on Exynos 5
====================
The previous release featured the initial version of our SATA 3.0 driver for
the Exynos 5 platform. This driver located at os/src/drivers/ahci/exynos5 has
reached a fully functional state by now. It supports UDMA-133 with up to
6 GBit/s.
For driver development, we set the goal to reach a performance equal to
the Linux kernel. To achieve that goal, we had to make sure to
operate the controller and the disks in the same ways as Linux does.
For this reason, we modeled our driver closely after the behaviour of the
Linux driver. That is, we gathered traces of I/O transactions to determine the
initialization steps and the request patterns that Linux performs to access
the device, and used those behavioral traces as a guide for our
implementation. Through step-by-step analysis of the traces, we not only
succeeded to operate the device in the proper modes, but we also found
opportunities for further optimization, in particular regarding the error
recovery implementation.
This approach turned out to be successful. We measured that our driver
generally operates as fast (and in some cases even significantly faster)
than the Linux driver on solid-state disks as well as on hard disks.
Dynamic CPU frequency scaling for Exynos 5
==========================================
As the Samsung Exynos-5 SoC is primarily targeted at mobile platforms,
power management is an inherent concern. Until now, Genode did not pay
much attention to power management though. For example, we completely
left out the topic from the scope of the OMAP4 support. With the current
release, we took the first steps towards proper power management on ARM-based
platforms in general, and the Exynos-5-based Arndale platform in particular.
First, we introduced a general interface to regulate clocks and voltages.
Priorly, each driver did its own part of configuring clock and power control
registers. The more device drivers were developed, the higher were the chances
that they interfere when accessing those clock, or power units.
The newly introduced "Regulator" interface provides the possibility to enable
or disable, and to set or get the level of a regulator. A regulator might be
a clock for a specific device (such as a CPU) or a voltage regulator.
For the Arndale board, an exemplary implementation of the regulator interface
exists in the form of the platform driver. It can be found at
os/src/drivers/platform/arndale. Currently, the driver implements
clock regulators for the CPU, the USB 2.0 and USB 3.0 host controller, the
eMMC controller, and the SATA controller. Moreover, it provides power
regulators for SATA, USB 2.0, and USB 3.0 host controllers. The selection of
regulators is dependent on the availability of drivers for the platform.
Otherwise it wouldn't be possible to test that clock and power state doesn't
affect the device.
Apart from providing regulators needed by certain device drivers, we
implemented a clock regulator for the CPU that allows changing the CPU
frequency dynamically and thereby giving the opportunity to scale down
voltage and power consumption. The possible values range from 200 MHz to
1.7 GHz whereby the last value isn't recommended and might provoke system
crashes due to overheating. When using Genode's platform driver for Arndale
it sets CPU clock speed to 1.6 GHz by default. When reducing
the clock speed to the lowest level, we observed a power consumption
reduction of approximately 3 Watt. Besides reducing dynamic power consumption
by regulating the CPU clock frequency, we also explored the gating of the clock
management and power management to further reduce power consumption.
With the CPU frequency scaling in place, we started to close all clock gates
not currently in use. When the platform driver for the Arndale board gets
initialized, it closes everything. If a device driver enables its clock
regulator, all necessary clock gates for the device's clock are opened. This
action saves about 0.7 Watt. The initial closing of all unnecessary power
gates was much more effective. Again, everything not essential for the working
of the kernel is disabled on startup. When a driver enables its power
regulator, all necessary power gates for the device are opened. Closing all
power gates saves about 2.6 Watt.
If we consider all measures taken to save power, we were able to reduce power
consumption to about 59% without performance degradation. When measuring power
consumption after boot up, setting the CPU clock to 1.6 GHz, and fully load
both CPU cores without the described changes, we measured about 8 Watt. With
the described power saving provisions enabled, we measured about 4.7 Watt.
When further reducing the CPU clock frequency to 200 MHz, only 1.7 Watt were
measured.
VESA driver moved to libports
=============================
The VESA framebuffer driver executes the initialization code located
in VESA BIOS of the graphics card. As the BIOS code is for real mode,
the driver uses the x86emu library from X11 as emulation environment.
We updated x86emu to version 1.20 and moved the driver from the 'os'
repository to the 'libports' repository as the library is third-party
code. Therefore, if you want to use the driver, the 'libports'
repository has to be prepared
('make -C <genode-dir>/libports prepare PKG=x86emu') and enabled in
your 'etc/build.conf'.
Runtime environments
####################
Seoul (aka Vancouver) VMM on NOVA
=================================
Since we repeatedly received requests for using the Seoul respectively
Vancouver VMM on NOVA, we improved the support for this virtualization
solution on Genode. Seoul now supports booting from raw hard disk images
provided via Genode's block session interface. Whether this image is actually
a file located in memory, or it is coming directly from the hard disk, or just
from a partition of the hard disk using Genode's part_blk service, is
completely transparent thanks to Genode's architecture.
Additionally, we split up the one large Vancouver run script into several
smaller Seoul run scripts for easier usage - e.g. one for disk, one for
network testing, one for automated testing, and one we call "fancy". The
latter resembles the former vancouver.run script using Genode's GUI to let the
user start VMs interactively. The run scripts prefixed with 'seoul-' can be
found at ports/run. For the fancy and network scripts, ready-to-use VM images
are provided. Those images are downloaded automatically when executing the run
script for the first time.
L4Linux on Fiasco.OC
====================
L4Linux has been updated from version 3.5.0 to Linux kernel version 3.9.0 thus
providing support for contemporary user lands running on top of L4Linux on both
x86 (32bit) and ARM platforms.
Noux runtime for Unix software
==============================
Noux is our way to use the GNU software stack natively on Genode. To improve
its performance, we revisited the address-space management of the runtime to
avoid redundant revocations of memory mappings when Noux processes are cleaned
up.
Furthermore, we complemented the support for the Genode tool chain to
cover GNU sed and GNU grep as well. Both packages are available at the ports
repository.
Platforms
#########
Fiasco.OC updated to revision r56
=================================
Fiasco.OC and the required L4RE parts have been updated to the current SVN
revision (r56). For us, the major new feature is the support of Exynos SOCs in
the mainline version of Fiasco.OC (www.tudos.org). Therefore Genode's
implementation of the Exynos5250 platform could be abandoned leading to less
maintenance overhead of Genode on Fiasco.OC.
Furthermore, Genode's multi-processor support for this kernel has been
improved so that Fiasco.OC users benefit from the additions described in
Section [Enhanced multi-processor support].
NOVA updated
============
In the process of our work on the multi-processor support on NOVA, we updated
the kernel to the current upstream version. Additionally, our customized branch
(called r3) comes with the added cross-CPU IPC system call and improvements
regarding the release of kernel resources.
Integrity checks for downloaded 3rd-party software
##################################################
Even though Genode supports a large variety of 3rd-party software, its
source-code repository contains hardly any 3rd-party source code. Whenever
3rd-party source code is needed, Genode provides automated tools for
downloading the code and integrating it with the Genode environment. As of
now, there exists support for circa 70 software packages, including the
tool chain, various kernels, libraries, drivers, and a few applications. Of
those packages, the code for 13 packages comes directly from their respective
Git repositories. The remaining 57 packages are downloaded in the form of tar
archives from public servers via HTTP or FTP. Whereas we are confident with
the integrity of the code that comes from Git repositories, we are less so
about the archives downloaded from HTTP or FTP servers.
Fortunately, most Open-Source projects provide signature files that allow
the user to verify the origin of the archive. For example, archives of
GNU software are signed with the private key of the GNU project. So the
integrity of the archive can be tested with the corresponding public key.
We used to ignore the signature files for many years but
this has changed now. If there is a signature file available for a package,
the package gets verified right after downloading. If only a hash-sum file
is provided, we check it against a known-good hash sum.
The solution required three steps, the creation of tools for validating
signatures and hashes, the integration of those tools into Genode's
infrastructure for downloading the 3rd-party code, and the definition of
verification rules for the individual packages.
First, new tools for downloading and validating hash sums and signatures were
added in the form of the shell scripts download_hashver (verify hash sum) and
download_sigver (verify signature) found at the tool/ directory. Under the
hood, download_sigver uses GNU GPG, and download_hashver uses the tools
md5sum, sha1sum, and sha256sum provided by coreutils.
Second, hooks for invoking the verification tools were added to the
tool-chain build script as well as the ports and the libports repositories.
The third and the most elaborative step, was going through all the packages,
looking for publicly available signature files, and adding corresponding
package rules. As of now, this manual process has been carried out for 30
packages, thereby covering the half of the archives.
Thanks to Stephan Mueller for pushing us into the right direction, kicking off
the work on this valuable feature, and for the manual labour of revisiting all
the 3rd-party packages!

0
doc/release_notes-13-11.txt → doc/release_notes/13-11.txt
View File

791
doc/release_notes/14-02.txt Normal file
View File

@@ -0,0 +1,791 @@
===============================================
Release notes for the Genode OS Framework 14.02
===============================================
Genode Labs
During the release cycle of version 14.02, our development has been focused on
storage and virtualization. It goes without saying that proper support for
block-device access and file systems is fundamental for the use of
Genode as general-purpose OS. Virtualization is relevant as well because
it bridges the gap between the functionality we need and the features
natively available on Genode today.
Our work on the storage topic involved changes of the block-driver APIs to an
asynchronous mode of operation, overhauling most of the existing block-level
components, as well as the creation of new block services, most importantly a
block cache. At file-system level, we continued our line of work on FUSE-based
file systems, adding support for NTFS-3g. A new highlight, however, is a new
file-system service that makes the file systems of the NetBSD kernel available
to Genode. This is made possible by using rump kernels as described in Section
[NetBSD file systems using rump kernels].
Virtualization on Genode has a long history, starting with the original
support of OKLinux on the OKL4 kernel (OKLinux is no longer supported), over
the support of L4Linux on top of the Fiasco.OC kernel, to the support of the
Vancouver VMM on top of NOVA. However, whereas each of those variants has
different technical merits, all of them were developed in the context of
university research projects and were never exposed to real-world scenarios.
We were longing for a solution that meets the general expectations from a
virtualization product, namely the support for a wide range of guest OSes,
guest-host integration features, ease of use, and an active development.
VirtualBox is one of the most popular commodity virtualization products as of
today. With the current release, we are happy to announce the availability of
VirtualBox on top of Genode/NOVA. Section
[VirtualBox on top of the NOVA microhypervisor] gives insights into the
background of this development, the technical challenges we had to overcome,
and the current state of the implementation.
In addition to addressing storage and virtualization, the current release
comes with a new pseudo file system called trace_fs that allows the
interactive use of Genode's tracing facilities via Unix commands,
a profound unification of the various graphics back ends used throughout
the framework, a new facility for propagating status reports, and
improvements of the Noux runtime for executing Unix software on Genode.
VirtualBox on top of the NOVA microhypervisor
#############################################
Virtualization is an important topic for Genode for two distinct reasons.
It is repeatedly requested by users of the framework who consider
Genode as a microkernel-based hosting platform for virtual machines,
and it provides a smooth migration path from using Linux-based systems
towards using Genode as day-to-day OS.
Why do people consider Genode as a hosting platform for virtual machines
if there is an abundance of mature virtualization solutions on the market?
What all existing popular solutions have in common is the staggering complexity
of their respective trusted-computing base (TCB). The user of a virtual
machine on a commodity hosting platform has to trust millions of lines of
code. For example, with Xen, the TCB comprises the hypervisor and the Linux
system running as DOM0. For security-sensitive application areas, it is
almost painful to trust such a complex foundation. In contrast, the TCB of a
hosting platform based on Genode/NOVA is two orders of magnitude less complex.
Lowering the complexity reduces the likelihood for vulnerabilities and thereby
mitigates the attack surface of the system. It also enables the assessment of
security properties by thorough evaluation or even formal verification. In the
light of the large-scale privacy issues of today, the desire for systems that
are resilient against malware and zero-day exploits has never been higher.
Microkernel-based operating systems promise a solution. Virtualization enables
compatibility to existing software. Combining both seems natural. This is what
Genode/NOVA stands for.
From the perspective of us Genode developers who are in the process of
migrating from Linux-based OSes to Genode as day-to-day OS, we consider
virtualization as a stop-gap solution for all those applications that
do not exist natively on Genode, yet. Virtualization makes our transition
an evolutionary process.
Until now, NOVA was typically accompanied with a co-developed virtual machine
monitor called Seoul (formerly called Vancouver), which is executed as a
regular user-level process on top of NOVA. In contrast to conventional wisdom
about the performance of microkernel-based systems, the Seoul VMM on top of
NOVA is extremely fast, actually faster then most (if not all) commonly used
virtualization solutions. However, originating from a research project, Seoul
is quite challenging to use and not as mature as commodity VMMs that were
developed as real-world products. For example, there is a good chance that an
attempt to boot an arbitrary version of a modern Linux distribution might just
fail. In our experience, it takes a few days to investigate the issues, modify
the guest OS configuration, and tweak the VMM here and there, to run the OS
inside the Seoul VMM. That is certainly not a show stopper in appliance-like
scenarios, but it rules out Seoul as a general solution. Running Windows
OS as guest is not supported at all, which further reduces the application
areas of Seoul. With this in mind, it is unrealistic to propose the use
of Genode/NOVA as an alternative for popular VM hosting solutions.
Out of this realization, the idea was born to combine NOVA's virtualization
interface with a time-tested and fully-featured commodity VMM. Out of the
available Open-Source virtualization solutions, we decided to take a closer
look at VirtualBox, which attracted us for several reasons: First, it is
portable, supporting various host OSes such as Solaris, Windows OS, Linux,
and Mac OS X. Second, it has all the guest-integration features we could
wish for. There are extensive so-called guest additions for popular guest
OSes that vastly improve the guest-OS performance and allow a tight
integration with the host OS using shared folders or a shared clipboard.
Third, it comes with sophisticated device models that support all
important popular guest OSes. And finally, it is actively developed and
commercially supported.
However, moving VirtualBox over to NOVA presented us with a number of
problems. As a precondition, we needed to gain a profound understanding
of the VirtualBox architecture and the code base. To illustrate the challenge,
the source-code distribution of VirtualBox comprises 2.8 million lines of
code. This code contains build tools, the VMM, management tools, several
3rd-party libraries, middleware, the guest additions, and tests. The pieces
that are relevant for the actual VMM amount to 700 thousand lines. By
reviewing the architecture, we found that the part of VirtualBox that
implements the hypervisor functionality (the world switch) runs in the
kernel of the host OS (it is loaded on demand by the user-level VM process
through the _/dev/vboxdrv_ interface into the host OS kernel). It is
appropriately named VMMR0. Once installed into the host OS kernel, it
takes over the control over the machine. To put it blatantly simple, it runs
"underneath" the host OS. The VMMR0 code is kernel agnostic, which explains
the good portability of VirtualBox across various host OSes. Porting
VirtualBox to a new host OS comes down to finding a hook for installing the
VMMR0 code into the host OS kernel and adapting the VirtualBox runtime API
to the new host OS.
In the context of microkernel-based systems, however, it becomes clear that
this classical approach of porting VirtualBox would subvert the microkernel
architecture. Not only would we need to punch a hole into NOVA for loading
additional kernel code, but also the VMMR0 code would inflate the amount of
code executed in privileged mode by more than factor 20. Both implications
are gross violations of the microkernel principle. Consequently, we needed to
find a different way to marry NOVA with VirtualBox.
Our solution was the creation of a drop-in replacement of the VMMR0 code that
runs solely at user level and interacts with NOVA's virtualization
interface. Our VMMR0 emulation code is co-located with the VirtualBox
VM process. Architecturally, the resulting solution is identical to the
use of Seoul on top of NOVA. There is one VM process per virtual machine,
and each VM process is isolated from others by the NOVA kernel. In
addition to creating the VMMR0 emulation code, we needed to replace some parts
of the VirtualBox VMMR3 code with custom implementations because they
overlapped with functionality provided by NOVA's virtualization interface,
in particular the provisioning of guest-physical memory. Finally, we needed
to interface the VM process with Genode's API to let the VM process
interact with Genode's input, file-system, and framebuffer services.
The result of this undertaking is available at the _ports_ repository.
VirtualBox can be downloaded and integrated with Genode via the following
command issued from within the repository:
! make prepare PKG=virtualbox
To illustrate the integration of VirtualBox into a Genode system, there
is run script located at _ports/run/virtualbox.run_. It expects a
bootable ISO image containing a guest OS at _<build-dir>/bin/test.iso_.
The configuration of the VirtualBox process is as simple as
! <config>
! <image type="iso" file="/iso/test.iso" />
! </config>
VirtualBox will try to obtain the specified ISO file via a file-system
session. Furthermore, it will open a framebuffer session and an input session.
The memory assigned to the guest OS depends on the RAM quota assigned to the
VirtualBox process. Booting a guest OS stored in a VDI file is supported. The
image type must be changed to "vdi" accordingly.
Please note that this first version of VirtualBox is far from being complete
as it lacks many features (SMP, guest-addition support, networking), is not
optimized, and must be considered as experimental. However, we could
successfully run GNU/Linux, Android, Windows XP, Windows 7, HelenOS, Minix-3,
GNU Hurd, and of course Genode inside VirtualBox.
One point we are pretty excited about is that the porting effort to
Genode/NOVA did not require any change of Genode. From Genode's point of
view, VirtualBox is just an ordinary leaf node of the process tree, which
can happily co-exist with other processes - even if it is the Seoul VMM.
[image seoul-vbox-win7-tinycore]
In the screenshot above, VirtualBox is running besides the Seoul VMM on top of
Genode/NOVA. Seoul executes Tinycore Linux as guest OS. VirtualBox executes MS
Windows 7. Both VMMs are using hardware virtualization (VT-x) but are plain
user-level programs with no special privileges.
NetBSD file systems using rump kernels
######################################
In the previous release, we made FUSE-based file systems available to Genode
via a custom implementation of the FUSE API. Even though this step made
several popular file systems available, we found that the file systems most
important to us (such as ext) are actually not well supported by FUSE. For
example, write support on ext2 is declared as an experimental feature. In
hindsight it is clear why: FUSE is primarily being used for accessing file
systems not found in the Linux kernel. So it shines with supporting NTFS
but less so with file systems that are well supported by the Linux kernel.
Coincidentally, when we came to this realization, we stumbled upon the
wonderful work of Antti Kantee on so-called rump kernels:
:[https://wiki.netbsd.org/rumpkernel/]:
Rump kernel Wiki
The motivation behind the rump kernels was the development of
NetBSD kernel subsystems (referred to as "drivers") in the NetBSD user land.
Such subsystems like file systems, device drivers, or the TCP/IP stack are
linked against a stripped-down version of the NetBSD kernel that can be
executed in user mode and uses a fairly small "hypercall" interface to
interact with the outside world. A rump kernel contains everything needed to
execute NetBSD kernel subsystems but hardly anything else. In particular, it
does not support the execution of programs on top. From our perspective,
having crafted device-driver environments (DDEs) for Linux, iPXE, and OSS over
the years, a rump kernel sounded pretty much like a DDE for NetBSD. So we
started exploring rump kernels with the immediate goal of making time-tested
NetBSD file systems available to Genode.
To our delight, the integration of rump kernels into the Genode system went
fairly smooth. The most difficult part was the integration of the NetBSD build
infrastructure with Genode's build system. The glue between rump kernels and
Genode is less than 3,000 lines of code. This code enables us to reuse all
NetBSD file systems on Genode. A rump kernel instance that contains several
file systems such as ext2, iso9660, msdos, and ffs takes about 8 MiB of memory
when executed on Genode.
The support for rump kernels comes in the form of the dedicated _dde_rump_
repository. For downloading and integrating the required NetBSD source code,
the repository contains a Makefile providing the usual 'make prepare'
mechanism. To build the file-system server, make sure to add the _dde_rump_
repository to the 'REPOSITORIES' declaration of your _etc/build.conf_ file
within your build directory. The server then can be built via
! make server/rump_fs
There is a run script located at _dde_rump/run/rump_ext2.run_ to execute
a simple test scenario:
! make run/rump_ext2
The server can be configured as follows:
!<start name="rump_fs">
! <resource name="RAM" quantum="8M" />
! <provides><service name="File_system"/></provides>
! <config fs="ext2fs"><policy label="" root="/" writeable="yes"/></config>
!</start>
On startup, it requests a service that provides a block session. If
there is more than one block session in the system, the block session must be
routed to the right block-session server. The value of the _fs_ attribute of
the '<config>' node can be one of the following: _ext2fs_ for EXT2, _cd9660_ for
ISO-9660, or _msdos_ for FAT file-system support. _root_ defines the directory
of the file system as seen as root directory by the client. The server hands
most of its RAM quota to the rump kernel. This means the larger the quota is,
the larger the internal block caches of the rump kernel will be.
Base framework
##############
The base API has not underwent major changes apart from the addition of
a few new utilities and minor refinements. Under the hood, however, the inner
workings of the framework received much attention, including an extensive
unification of the startup code and stack management.
New 'construct_at' utility
==========================
A new utility located at 'base/include/util/construct_at.h' allows for the
manual placement of objects without the need to have a global placement new
operation nor the need for type-specific new operators.
New utility for managing volatile objects
=========================================
Throughout Genode, we maintain a programming style that largely avoids dynamic
memory allocations. For the most part, higher-level objects aggregate
lower-level objects as class members. For example, the nitpicker GUI server
is actually a compound of such aggregations (see
[https://github.com/genodelabs/genode/blob/master/os/src/server/nitpicker/main.cc#L803 - Nitpicker::Main]).
This functional programming style leads to robust programs but it poses a
problem for programs that are expected to adopt their behaviour at runtime.
For the example of nitpicker, the graphics back end of the GUI server takes
the size of the screen as constructor argument. If the screen size changes,
the once constructed graphics back end becomes inconsistent with the new
screen size. We desire a way to selectively replace an aggregated object by a
new version with updated constructor arguments. The new utilities found in
'os/include/util/volatile_object.h' solve this problem. A so-called
'Volatile_object' wraps an object of the type specified as template argument.
In contrast of a regular object, a 'Volatile_object' can be re-constructed any
number of times by calling 'construct' with the constructor arguments. It is
accompanied with a so-called 'Lazy_volatile_object', which remains
unconstructed until 'construct' is called the first time.
Changed interface of 'Signal_rpc_member'
========================================
We unified the 'Signal_rpc_member' interface to be more consistent with the
'Signal_rpc_dispatcher'. The new version takes an entrypoint as argument and
cares for dissolving itself from the entrypoint when destructed.
Filename as default label for ROM connections
=============================================
Since the first version of Genode, ROM services used to rely on a "filename"
provided as session argument. In the meanwhile, we established the use of the
session label to select routing policies as well as server-side policies.
Strictly speaking, the name of a ROM module is used as a key to a server-side
policy of ROM services. So why not to use the session label to express the
key as we do with other services? By assigning the file name as label for ROM
sessions, we may become able to remove the filename argument in the future by
just interpreting the last part of the label as filename. By keeping only the
label, we won't need to consider conditional routing (via '<if-arg>') based on
session arguments other than the label anymore, which would simplify Genode
configurations in the long run. This change is transparent at API level but
may be taken into consideration when configuring Genode systems.
New 'Genode::Deallocator' interface
===================================
By splitting the new 'Genode::Deallocator' interface from the former
'Genode::Allocator' interface, we become able to restrict the accessible
operations for code that is only supposed to release memory, but not
perform any allocations.
Closely related to the allocator interface, we introduced variants of the
'new' operator that take a reference (as opposed to a pointer) to a
'Genode::Allocator' as argument.
Unified main-stack management and startup code among all platforms
==================================================================
In contrast to the stacks of regular threads, which are located within a
dedicated virtual-address region called thread-context area, the stack of
the main thread of a Genode program used to be located within the BSS
segment. If the stack of a normal thread overflows, the program produces
an unresolvable page fault, which can be easily debugged. However,
an overflowing main stack would silently corrupt the BSS segment. With
the current release, we finally resolved this long-standing problem by
moving the main stack to the context area, too. The tricky part was that
the context area is created by the main thread. So we hit a hen-and-egg
problem. We overcame this problem by splitting the process startup
into two stages, both called from the crt0 assembly code. The first
stage runs on a small stack within the BSS and has the sole purpose
of creating the context area and a thread object for the main thread.
This code path (and thereby the stack usage) is the same for all programs.
So we can safely dimension the stage-1 stack. Once the first stage
returns to the crt0 assembly code, the stack pointer is loaded with the
stack that is now located within the context area. Equipped with the
new stack, the actual startup code ('_main') including the global
constructors of the program is executed.
This change paved the ground for several further code unifications and
simplifications, in particular related to the dynamic linker.
Low-level OS infrastructure
###########################
Revised block-driver framework
==============================
Whereas Genode's block-session interface was designed to work asynchronously
and supports the out-of-order processing of requests, those capabilities
remained unused by the existing block services as those services used to
operate synchronously to keep their implementation simple. However, this
simplicity came at the prize of two disadvantages: First, it prevented us
to fully utilize native command queuing of modern disk controllers. Second,
when chaining components such as a block driver, the part_blk server, and
a file system, latencies accumulated along the chain of services. This
hurts the performance of random access patterns.
To overcome this limitation, we changed the block-component framework to work
asynchronously and to facilitate the recently introduced server API.
Consequently, all users of the API underwent an update. The affected
components are rom_loopdev, atapi_drv, fb_block_adapter, http_block, usb_drv,
and part_blk. For some components, in particular part_blk, this step led to a
complete redesign.
Besides the change of the block-component framework, the block-session
interface got extended to support logical block addresses greater than
32bit (LBA48). Thereby, the block component framework can now support
devices that exceed 2 TiB in size.
Block cache
===========
The provisioning of a block cache was one of the primary motivations behind the
[https://genode.org/documentation/release-notes/13.11#Dynamic_resource_balancing - dynamic resource balancing]
concept that was introduced in Genode 13.11. We are now introducing the first
version of such a cache.
The new block cache component located at _os/src/server/blk_cache/_ is both
a block-session client as well as a block-session server serving a single
client. It is meant to sit between a block-device driver and a file-system
server. When accessing the block device, it issues requests at a granularity
of 4K and thereby implicitly reads ahead whenever a client requests a smaller
amount of blocks. Blocks obtained from the device or written by the client
are kept in memory. If memory becomes scarce, the block cache first tries
to request further memory resources from its parent. If the request
gets denied, the cache evicts blocks from memory to the block device following
a least-recently-used replacement strategy. As of now, the block cache supports
dynamic resource requests to grow on demand but support for handling yield
requests is not yet implemented. So memory once handed out to the block cache
cannot be regained. Adding support for yielding memory on demand will be
complemented in the next version.
To see how to integrate the block cache in a Genode scenario, there is a
ready-to-use run script available at _os/run/blk_cache.run_.
File-system infrastructure
==========================
In addition to the integration of NetBSD's file systems, there are
file-system-related improvements all over the place.
First, the 'File_system::Session' interface has been extended with a 'sync'
RPC function. This function allows the client of a file system to force
the file system to write back its internal caches.
Second, we extended the FUSE implementation introduced with the previous
release.
Since file systems tend to have a built-in caching mechanism, we need to
sync these caches at the end of a session when using the fuse_fs server.
Therefore, each FUSE file system port has to implement a 'Fuse::sync_fs()'
function that executes the necessary actions if requested. Further
improvements are related to the handling of symbolic links and error
handling. Finally, we added a libc plugin for accessing NTFS file systems
via the ntfs-3g library.
Third, we complemented the family of FUSE-based libc plugins with a family of
FUSE-based file-system servers. To utilize a FUSE file system, there is a
dedicated binary (e.g., _os/src/server/fuse_fs/ext2_) for each FUSE
file-system server.
Note that write support is possible but considered to be experimental at this
point. For now, using it is not recommended.
To use the ext2_fuse_fs server in Noux, the following configuration snippet
may be used:
! <start name="ext2_fuse_fs">
! <resource name="RAM" quantum="8M"/>
! <provides> <service name="File_system"/> </provides>
! <config>
! <policy label="noux -> fuse" root="/" writeable="no" />
! </config>
! </start>
Finally, the libc file-system plugin has been extended to support 'unlink'.
Trace file system
=================
The new _trace_fs_ server provides access to a trace session by providing a
file-system session as front end. Combined with Noux, it allows for the
interactive exploration and tracing of Genode's process tree using
traditional Unix tools.
Each trace subject is represented by a directory ('thread_name.subject') that
contains specific files, which are used to control the tracing process of the
thread as well as storing the content of its trace buffer:
:'enable': The tracing of a thread is activated if there is a valid policy
installed and the intend to trace the subject was made clear by writing '1'
to the 'enable' file. The tracing of a thread may be deactivated by writing a
'0' to this file.
:'policy': A policy may be changed by overwriting the currently used one in the
'policy' file. In this case, the old policy is replaced by the new one and
automatically used by the framework.
:'buffer_size': Writing a value to the 'buffer_size' file changes the size of
the trace buffer. This value is evaluated only when reactivating the tracing
of the thread.
:'events': The trace-buffer contents may be accessed by reading from the
'events' file. New trace events are appended to this file.
:'active': Reading the file will return whether the tracing is active (1) or
not (0).
:'cleanup': Nodes of untraced subjects are kept as long as they do not change
their tracing state to dead. Dead untraced nodes are automatically removed
from the file system. Subjects that were traced before and are now untraced
can be removed by writing '1' to the 'cleanup' file.
To use the trace_fs, a configuration similar to the following may be used:
! <start name="trace_fs">
! <resource name="RAM" quantum="128M"/>
! <provides><service name="File_system"/></provides>
! <config>
! <policy label="noux -> trace"
! interval="1000"
! subject_limit="512"
! trace_quota="64M" />
! </config>
! </start>
:'interval': sets the period the Trace_session is polled. The
time is given in milliseconds.
:'subject_limit': specifies how many trace subjects should by acquired at
max when the Trace_session is polled.
:'trace_quota': is the amount of quota the trace_fs should use for the
Trace_session connection. The remaining amount of RAM quota will be used
for the actual nodes of the file system and the 'policy' as well as the
'events' files.
In addition, there are 'buffer_size' and 'buffer_size_limit' that define
the initial and the upper limit of the size of a trace buffer.
A ready-to-use run script can by found in 'ports/run/noux_trace_fs.run'.
Unified interfaces for graphics
===============================
Genode comes with several programs that perform software-based graphics
operations. A few noteworthy examples are the nitpicker GUI server,
the launchpad, the scout tutorial browser, or the terminal. Most of those
programs were equipped with their custom graphics back end. In some
cases such as the terminal, nitpicker's graphics back end was re-used.
But this back end is severely limited because its sole purpose is the
accommodation of the minimalistic (almost invisible) nitpicker GUI server.
The ongoing work on Genode's new user interface involves the creation of
new components that rely on a graphics back end. Instead of further
diversifying the zoo of graphics back ends, we took the intermediate step
to consolidate the existing back ends into one unified concept such that
application-specific graphics back ends can be created and extended using
modular building blocks. The new versions of nitpicker, scout, launchpad,
liquid_fb, nitlog, and terminal have been changed to use the new common
interfaces:
:os/include/util/geometry.h: Basic data structures and operations needed
for 2D graphics.
:os/include/util/color.h: Common color representation and utilities.
:os/include/os/pixel_rgba.h: Class template for representing a pixel.
:os/include/os/pixel_rgb565.h: Template specializations for RGB565 pixels.
:os/include/os/surface.h: Target surface, onto which graphics operations
can be applied.
:os/include/os/texture.h: Source texture for graphics operations that
transfer 2D pixel data to a surface.
The former _os/include/nitpicker_gfx/_ directory is almost deserted. The only
remainders are functors for the few graphics operations actually required by
nitpicker. For the scout widgets, the corresponding functors have become
available at the public headers at _demo/include/scout_gfx/_.
Because the scout widget set is used by at least three programs and will
most certainly play a role in new GUI components, we undertook a major
cleanup of the parts worth reusing. The result can be found at
_demo/include/scout/_.
New session interface for status reporting
==========================================
Genode has a uniform way of how configuration information is passed from
parents to children within the process tree by the means of "config" ROM
modules. Using this mechanism, a parent is able to steer the behaviour of
its children, not just at their start time but also during runtime.
Until now, however, there was no counterpart to the config mechanism, which
would allow a child to propagate runtime information to its parent. There
are many use cases for such a mechanism. For example, a bus-controller driver
might want to propagate a list of devices attached to the bus. When a new
device gets plugged in, this list should be updated to let the parent
take the new device resource into consideration. Another use case would be the
propagation of status information such as the feature set of a plugin.
Taken to the extreme, a process might expose its entire internal state to its
parent in order to allow the parent to kill and restart the process, and
feed the saved state back to the new process instance.
To cover these use cases, we introduced the new report-session interface. When
a client opens a report session, it transfers a part of its RAM quota to the
report server. In return, the report server hands out a dataspace dimensioned
according to the donated quota. Upon reception of the dataspace, the client
can write its status reports into the dataspace and inform the server about
the update via the 'submit' function. In addition to the mere reporting of
status information, the report-session interface is designed to allow the
server to respond to reports. For example, if the report mechanism is used to
implement a desktop notification facility, the user may interactively respond
to an incoming notification. This response can be reflected to the originator
of the notification via the 'response_sigh' and 'obtain_response' functions.
The new _report_rom_ component is both a report service and a ROM service. It
reflects incoming reports as ROM modules. The ROM modules are named
after the label of the corresponding report session.
Configuration
-------------
The report-ROM server hands out ROM modules only if explicitly permitted by a
configured policy. For example:
! <config>
! <rom>
! <policy label="decorator -> pointer" report="nitpicker -> pointer"/>
! <policy ... />
! ...
! </rom>
! </config>
The label of an incoming ROM session is matched against the 'label' attribute
of all '<policy>' nodes. If the session label matches a policy label, the
client obtains the data from the report client with the label specified in the
'report' attribute. In the example above, the nitpicker GUI server sends
reports about the pointer position to the report-ROM service. Those reports
are handed out to a window decorator (labeled "decorator") as ROM module.
XML generator utility
=====================
With the new report-session interface in place, comes the increased
need to produce XML data. The new XML generator utility located at
_os/include/util/xml_generator.h_ makes this extremely easy, thanks to
C++11 language features. For an example application, refer to
_os/src/test/xml_generator/_ and the corresponding run script at
_os/run/xml_generator.run_.
Dynamic ROM service for automated testing
=========================================
The new _dynamic_rom_ service provides ROM modules that change during the
lifetime of a ROM session according to a timeline. The main purpose of this
service is the automated testing of programs that are able to respond to ROM
module changes, for example configuration changes.
The configuration of the dynamic ROM server contains a '<rom>' sub node per
ROM module provided by the service. Each '<rom>' node hosts a 'name' attribute
and contains a sequence of sub nodes that define the timeline of the ROM
module. The possible sub nodes are:
:'<inline>': The content of the '<inline>' node is assigned to the content
of the ROM module.
:'<sleep>': Sleeps a number of milliseconds as specified via the 'milliseconds'
attribute.
:'<empty>': Removes the ROM module.
At the end of the timeline, it re-starts at the beginning.
Nitpicker GUI server
====================
The nitpicker GUI server has been enhanced to support dynamic screen
resizing. This is needed to let nitpicker respond to screen-resolution
changes, or when using a nested version of nitpicker within a resizable
virtual framebuffer window.
To accommodate Genode's upcoming user-interface concept, we introduced the
notion of a parent-child relationship between nitpicker views. If an existing
view is specified as parent at construction time of a new view, the parent
view's position is taken as the origin of the child view's coordinate space.
This allows for the grouping of views, which can be atomically repositioned by
moving their common parent view. Another use case is the handling of popup
menus in Qt5, which can now be positioned relative to their corresponding
top-level window. The relative position is maintained transparently to Qt when
the top-level window gets repositioned.
Libraries and applications
##########################
Noux runtime for executing Unix software
========================================
Noux plays an increasingly important role for Genode as it allows the use
of the GNU software stack. Even though it already supported a variety of
packages including bash, gcc, binutils, coreutils, make, and vim, some
programs were still limited by Noux' not fully complete POSIX semantics,
in particular with regard to signal handling. For example, it was not
possible to cancel the execution of a long-running process via Control-C.
To overcome those limitations, we enhanced Noux by adding the _kill_ syscall,
reworking the _wait_ and _execve_ syscalls, as well as adding
signal-dispatching code to the Noux libc. Special attention had to be paid to
the preservation of pending signals during the process creation via _fork_ and
_execve_.
The current implementation delivers signals each time a Noux syscall
returns. Signal handlers are executed as part of the normal control flow. This
is in contrast to traditional Unix implementations, which allow the
asynchronous invocation of signal handlers out of band with the regular
program flow. The obvious downside of our solution is that a program that got
stuck in a busy loop (and thereby not issuing any system calls) won't respond
to signals. However, as we regard the Unix interface just as a runtime and not
as the glue that holds the system together, we think that this compromise is
justified to keep the implementation simple and kernel-agnostic. In the worst
case, if a Noux process gets stuck because of such a bug, we certainly can
live with the inconvenience of restarting the corresponding Noux subsystem.
To complement our current activities on the block and file-system levels,
the e2fsprogs-v1.42.9 package as been ported to Noux. To allow the
block-device utilities to operate on Genode's block sessions, we added a new
"block" file system to Noux. Such a block file system can be mounted using a
'<block>' node within the '<fstab>'. By specifying a label attribute, each
block session request can be routed to the proper block session provider:
! <fstab>
! ...
! <dir name="dev">
! <block name="blkdev0" label="block_session_0" />
! </dir>
! ...
! </fstab>
In addition to this file system, support for the DIOCGMEDIASIZE ioctl
request was added. This request is used by FreeBSD and therefore by our
libc to query the size of the block device in bytes.
Qt5 refinements
===============
Our port of Qt5 used to rely on custom versions of synchronization
primitives such as 'QWaitCondition' and 'QMutex'. However, since most of the
usual pthread synchronization functions as relied on by Qt5's regular POSIX
back end have been added to Genode's pthread library by now, we could replace
our custom implementations by Qt5's POSIX version.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
The development of our base-hw kernel platform during this release cycle was
primarily geared towards adding multi-processor support. However, as we
haven't exposed the code to thorough testing yet, we deferred the integration
of this feature for the current release.
We increased the number of usable ARM platforms by adding basic support for
the ODROID XU board.
NOVA microhypervisor
====================
The port of VirtualBox to Genode prompted us to improve the NOVA platform in
the following respects.
NOVA used to omit the propagation of the FPU state of the guest OS to the
virtual machine monitor (VMM) during the world switch between the guest OS and
the VMM. With the Vancouver VMM, which is traditionally used on NOVA, this
omission did not pose any problem because Vancouver would never touch the FPU
state of the guest. So the FPU context of the guest was always preserved
throughout the handling of virtualization events. However, in contrast to the
Vancouver VMM, VirtualBox relies on the propagation of the FPU state between
the guest running in VT-X non-root mode and the guest running within the
VirtualBox recompiler. Without properly propagating the FPU state between both
virtualization back ends, both the guest OS in non-root mode and VirtualBox's
recompiler would corrupt each other's FPU state. After first implementing an
interim solution in our custom version of the kernel, the missing FPU context
propagation had been implemented in the upstream version of NOVA as well.
In contrast to most kernels, NOVA did not allow a thread to yield its current
time slice to another thread. The only way to yield CPU time was to block on
a semaphore or to perform an RPC call. Unfortunately both of those instruments
require the time-receiving threads to explicitly unblock the yielding thread
(by releasing the semaphore or replying to the RPC call). However, there are
situations where the progress of a thread may depend on an external
condition or a side effect produced by another (unknown) thread. One
particular example is the spin lock used to protect (an extremely short)
critical section of Genode's lock metadata. Apparently VirtualBox presented
us with several more use cases for thread-yield semantics. Therefore, we
decided to extend NOVA's kernel interface with a new 'YIELD' opcode to the
'ec_control' system call.

1123
doc/release_notes/14-05.txt Normal file
View File

File diff suppressed because it is too large Load Diff

1136
doc/release_notes/14-08.txt Normal file
View File

File diff suppressed because it is too large Load Diff

1047
doc/release_notes/14-11.txt Normal file
View File

File diff suppressed because it is too large Load Diff

899
doc/release_notes/15-02.txt Normal file
View File

@@ -0,0 +1,899 @@
===============================================
Release notes for the Genode OS Framework 15.02
===============================================
Genode Labs
Genode's [https://genode.org/about/road-map - roadmap] for this year puts a
strong emphasis on the consolidation and cultivation of the existing feature
set. With the first release of the year, version 15.02 pays tribute to this
mission by stepping up to extensive and systematic automated testing. As
a precondition for scaling up Genode's test infrastructure, the release
features a highly modular tool kit for exercising system scenarios on a growing zoo
of test machines. Section [Modular tool kit for automated testing] explains
the new tools in detail. In the spirit of improving the existing feature
set, Genode 15.02 vastly improves the performance and stability of our version of
VirtualBox running on the NOVA microhypervisor, solves long-standing shortcomings
of memory management on machines with a lot of RAM, addresses NOVA-related
scalability limitations, stabilizes our Rump-kernel-based file-system server,
and refines the configuration interface of the Intel wireless driver.
As the most significant new feature, the new version introduces virtualization
support for ARM to our custom base-hw kernel. Section [Virtualization on ARM]
outlines the design and implementation of this feature, which was greatly
inspired by NOVA's virtualization architecture and has been developed over the
time span of more than a year.
With respect to platform support, we are happy to accommodate the upcoming
USB-Armory board, which is a computer in the form factor of a USB
stick especially geared towards security applications. Section
[Support for the USB-Armory board] covers the background and the current
state of this line of work.
Virtualization on ARM
#####################
The ARMv7 architecture of recent processors like Cortex-A7, Cortex-A15, or
Cortex-A17 CPUs support hardware extensions to facilitate virtualization of
guest operating systems. With the current release, we enable the use of these
virtualization extensions in our custom base-hw kernel when running on the
Cortex-A15-based Arndale board.
While integrating ARM's virtualization extension, we aimed to strictly follow
microkernel-construction principles. The primary design is inspired by the
[https://hypervisor.org/ - NOVA OS Virtualization Architecture]. It is based on a
microhypervisor that provides essential microkernel mechanisms along with
basic primitives to switch between virtual machines (VMs). On top of the
microhypervisor, classical OS services are implemented as
ordinary, unprivileged user-level components. Those services can be used by other
applications. Services may be shared between applications or instantiated
separately, according to security and safety needs. Correspondingly,
following the NOVA principles, each VM has its own associated virtual-machine
monitor (VMM) that runs as an unprivileged user-level component. VMM implementations
can range from simple ones that just emulate primary device requirements to highly
complex monitors including sophisticated device models, like VirtualBox. The
NOVA approach allows to decouple the TCB complexity of one VM with respect to
another, as well as with respect to all components not related to
virtualization at all.
Along those lines, we extended the base-hw kernel/core conglomerate with API
extensions that enable user-level VMM components to create and control virtual
machines.
Design
======
The ARM virtualization extensions are based on the so-called security
extensions, commonly known as
[https://genode.org/documentation/articles/trustzone - TrustZone].
The ARM designers did not follow the
Intel approach to split the CPU into a "root" and a "guest" world while having all prior
existing CPU modes available in both worlds. Instead, ARM added a new privilege level
to the non-secure side of TrustZone that sits underneath the ordinary kernel
and userland privilege levels. It is subjected to a hypervisor-like kernel. All
instructions used to prepare a VM's environment have to be executed in this so
called "hyp" mode. In hyp mode, some instructions
differ from their regular behaviour on the kernel-privilege level.
For this reason, prior-existing kernel code cannot simply be reused in
hyp mode without modifications.
The base-hw kernel is meant to execute Genode's core component on bare hardware.
Core, which is an ordinary user-level component, is
linked together with a slim kernel library that is executed in privileged kernel
mode. To enable ARM hardware virtualization, we pushed this approach
even further by executing core in three different privilege levels. Thereby,
core shares the same view on hardware resources and virtual memory across all
levels. A code path is executed on a higher privilege level only if the code
would fail to execute on a lower privilege level.
Following this approach, we were able to keep most of the existing kernel code
with no modifications.
[image avirt_overview]
Genode's ARM kernel (core) runs across all privilege levels
The hypervisor part of core is solely responsible to switch between VMs and the
host system. Therefore, it needs to load/store additional CPU state that
normally remains untouched during context switches of ordinary tasks. It also needs to
configure the VM's guest-physical to host-physical memory translations. Moreover, the
virtualization extensions of the ARMv7 architecture are not related to the CPU
cores only. The interrupt controller and the CPU-local timers are also
virtualization-aware. Therefore, the hypervisor has to load/store state specific
to those devices, too. Nevertheless, the hypervisor merely reloads those
devices. It does not interpret their state.
In contrast to the low-complexity hypervisor, a user-level VMM can be complex
without putting the system's security at risk. It contains potentially complex
device-emulation code and assigns hardware resources such as memory and
interrupts to the VM. The VMM is an ordinary user-level component running
unprivileged. Of course, as a plain user-level component, it is not able to
directly access hardware resources. Hence an interface between VMMs and the
kernel is needed to share the state of a virtual machine. In the past, we faced a similar
problem when building a VMM for our former TrustZone experiments. It was natural
to build upon the available solution and to extend it where necessary. Core
provides a so-called VM service. Each VM corresponds to a session of this
service. The session provides the following extended interface:
:CPU state:
The CPU-state function returns a dataspace containing the virtual machine's
state. The state is initialized by the VMM before bootstrapping the VM, gets updated
by the hypervisor whenever it switches away from the VM, and can be used by
the VMM to interpret the behavior of the guest OS. Moreover, the CPU state can be
updated after the virtual machine monitor emulated instructions
for the VM.
:Exception handler:
The second function is used to register a signal handler that gets informed
whenever the VM produces a virtualization fault.
:Run:
The run function starts or resumes the execution of the VM.
:Pause:
The pause function removes the VM from the kernel's scheduler.
:Attach:
This function attaches a given RAM dataspace to a designated area of the
guest-physical address space.
:Detach:
The detach function invalidates a designated area of the guest-physical
address space.
:Attach_pic: Tells the hypervisor to attach the CPU's virtual interface of the
virtualization-aware interrupt controller to a designated area of the
guest-physical address space.
Implementation
==============
By strictly following the micro-kernel construction principles when integrating the
hypervisor into the base-hw kernel, we reached a minimally invasive solution. In
doing so, we took the time to separate TrustZone-specific code that was formerly
an inherent part of the kernel on ARMv7 platforms. Now, TrustZone- and
virtualization-specific aspects are incorporated into the kernel only if
actually used. The change in complexity of the whole core component expressed in
lines of code is shown in the table below. As can be seen, the additional code in
the root of the trusted computing base when using virtualization is about 700-800
LOC.
Platform | with TrustZone, no VT | TrustZone/VT optional
-----------------------------------------------------------------
hw_arndale | 17970 LOC | 18730 LOC
----------------------------------------------------------------
hw_imx53_qsb | 17900 LOC | 17760 LOC
----------------------------------------------------------------
hw_imx53_qsb_tz | 18260 LOC | 18320 LOC
----------------------------------------------------------------
hw_rpi | 17500 LOC | 17430 LOC
----------------------------------------------------------------
hw_panda | 18040 LOC | 17880 LOC
----------------------------------------------------------------
hw_odroid_xu | 17980 LOC | 18050 LOC
Besides the VM world switch, we enabled support for the so-called "large
physical address extension" (LPAE), which is obligatory when using
virtualization. It allows for addressing a 40-bit instead of only 32-bit physical
address space. Moreover, to execute in hypervisor mode, the bootstrap code of
the kernel had to be set up properly. Hence, when booting on the Arndale board,
the kernel now prepares the non-secure TrustZone world first, and finally leaves the
secure world forever.
To test and showcase the ARM virtualization features integrated in base-hw, we
implemented a minimal, exemplary VMM. It can be found in
_repos/os/src/server/vmm_. The VMM emulates a simplified variant of ARM's
Versatile Express Cortex-A15 development platform. Currently, it only comprises
support for the CPU, the timer, the interrupt controller, and a UART device. It is
written in 1100 lines of C++ in addition to the base Genode libraries. The VMM
is able to boot a vanilla Linux kernel compiled with a slightly modified
standard configuration (no-SMP), and a device tree description stripped down to
the devices provided by the VMM. This release includes an automated run test that
executes the Linux kernel on top of the VMM on Genode. It can be started via:
! make run/vmm
[image avirt_screen]
Three Linux serial consoles running in parallel on top of Genode
Modular tool kit for automated testing
######################################
In
[https://genode.org/documentation/release-notes/13.05#Automated_quality-assurance_testing - Genode version 13.05],
we already introduced comprehensive support for the automated testing of
Genode scenarios. Since then, Genode Labs has significantly widened the scope
of its internal test infrastructure, both in terms of the coverage of the test
scenarios as well as the variety of the used hardware platforms.
The centerpiece of our test infrastructure is the so-called run tool. Steered
by a script (run script), it performs all the steps necessary to test drive
a Genode system scenario. Those steps are:
# *Building* the components of a scenario
# *Configuration* of the init component
# Assembly of the *boot directory*
# Creation of the *boot image*
# *Powering-on* the test machine
# *Loading* of the boot image
# Capturing the *LOG output*
# *Validation* of the scenario behavior
# *Powering-off* the test machine
Each of those steps depends on various parameters such as the
used kernel, the hardware platform used to run the scenario, the
way the test hardware is connected to the test infrastructure
(e.g., UART, AMT, JTAG, network), the way the test hardware is powered or
reseted, or the way of how the scenario is loaded into the test hardware.
Naturally, to accommodate the growing variety of combinations of those
parameters, the complexity of the run tool increased over time.
This growth of complexity prompted us to eventually turn the run tool into a
highly modular and extensible tool kit.
Originally, the run tool consisted of built-in rules that could be
extended and tweaked by a kernel-specific supplement called run environment.
The execution of a run script used to depend on the policies built into
the run tool, the used run environment, and optional configuration
parameters (run opts).
The new run tool kit replaces most of the formerly built-in policies by the
ability to select and configure different modules for the various steps.
The selection and configuration of the modules is expressed in the run-tool
configuration. There exist the following types of modules:
:boot-dir modules:
These modules contain the functionality to populate the boot directory
and are specific to each kernel. It is mandatory to always include the
module corresponding to the used kernel.
_(the available modules are: linux, hw, okl4, fiasco, pistachio, nova,_
_codezero, foc)_
:image modules:
These modules are used to wrap up all components used by the run script
in a specific format and thereby prepare them for execution.
Depending on the used kernel, different formats can be used. With these
modules, the creation of ISO and disk images is also handled.
_(the available modules are: uboot, disk, iso)_
:load modules:
These modules handle the way the components are transfered to the
target system. Depending on the used kernel there are various options
to pass on the components. For example, loading from TFTP or via JTAG is handled
by the modules of this category.
_(the available modules are: tftp, jtag, fastboot)_
:log modules:
These modules handle how the output of a currently executed run script
is captured.
_(the available modules are: qemu, linux, serial, amt)_
:power_on modules:
These modules are used for bringing the target system into a defined
state, e.g., by starting or rebooting the system.
_(the available modules are: qemu, linux, softreset, powerplug, amt)_
:power_off modules:
These modules are used for turning the target system off after the
execution of a run script.
_(the available modules are: powerplug)_
When executing a run script, only one module of each category must be used.
Each module has the form of a script snippet located under the
_tool/run/<step>/_
directory where _<step>_ is a subdirectory named after the module type.
Further instructions about the use of each module (e.g., additional
configuration arguments) can be found in the form of comments inside the
respective script snippets.
Thanks to this modular structure,
the extension of the tool kit comes down to adding a file at the corresponding
module-type subdirectory. This way, custom work flows (such as tunneling JTAG
over SSH) can be accommodated fairly easily.
Usage examples
==============
To execute a run script, a combination of modules may be used. The combination
is controlled via the RUN_OPT variable used by the build framework. Here are a
few common exemplary combinations:
Executing NOVA in Qemu:
!RUN_OPT = --include boot_dir/nova \
! --include power_on/qemu --include log/qemu --include image/iso
Executing NOVA on a real x86 machine using AMT for resetting the target system
and for capturing the serial output while loading the files via TFTP:
!RUN_OPT = --include boot_dir/nova \
! --include power_on/amt --power-on-amt-host 10.23.42.13 \
! --power-on-amt-password 'foo!' \
! --include load/tftp --load-tftp-base-dir /var/lib/tftpboot \
! --load-tftp-offset-dir /x86 \
! --include log/amt --log-amt-host 10.23.42.13 \
! --log-amt-password 'foo!'
Executing Fiasco.OC on a real x86 machine using AMT for resetting, USB serial
for output while loading the files via TFTP:
!RUN_OPT = --include boot_dir/foc \
! --include power_on/amt --amt-host 10.23.42.13 --amt-password 'foo!' \
! --include load/tftp --tftp-base-dir /var/lib/tftpboot \
! --tftp-offset-dir /x86 \
! --include log/serial --log-serial-cmd 'picocom -b 115200 /dev/ttyUSB0'
Executing base-hw on a Raspberry Pi using powerplug to reset the hardware,
JTAG to load the image and USB serial to capture the output:
!RUN_OPT = --include boot_dir/hw \
! --include power_on/powerplug --power-on-powerplug-ip 10.23.42.5 \
! --power-on-powerplug-user admin \
! --power-on-powerplug-password secret \
! --power-on-powerplug-port 1
! --include power_off/powerplug --power-off-powerplug-ip 10.23.42.5 \
! --power-off-powerplug-user admin \
! --power-off-powerplug-password secret \
! --power-off-powerplug-port 1
! --include load/jtag \
! --load-jtag-debugger /usr/share/openocd/scripts/interface/flyswatter2.cfg \
! --load-jtag-board /usr/share/openocd/scripts/interface/raspberrypi.cfg \
! --include log/serial --log-serial-cmd 'picocom -b 115200 /dev/ttyUSB0'
After the run script was executed successfully, the run tool will print the
string 'Run script execution successful.". This message can be used to check
for the successful completion of the run script when doing automated testing.
Meaningful default behaviour
============================
To maintain the ease of use of creating and using a build directory, the
'create_builddir' tool equips a freshly created build directory with a meaningful
default configuration that depends on the selected platform. For example, if
creating a build directory for the Linux base platform, RUN_OPT
is initially defined as
! RUN_OPT = --include boot_dir/linux \
! --include power_on/linux --include log/linux
Low-level OS infrastructure
###########################
Improved management of physical memory
======================================
On machines with a lot of memory, there exist constraints with regard to
the physical address ranges of memory:
* On platforms with a non-uniform memory architecture, subsystems should
preferably use memory that is local to the CPU cores the subsystem is using.
Otherwise the performance is impeded by costly memory accesses to
the memory of remote computing nodes.
* Unless an IOMMU is used, device drivers program physical addresses
into device registers to perform DMA operations. Legacy devices such as
USB UHCI controllers expect a 32-bit address. Consequently, the memory
used as DMA buffers for those devices must not be allocated above 4 GiB.
* When using an IOMMU on NOVA, Genode represents the address space
accessible by devices (by the means of DMA) using a so-called device PD
([https://genode.org/documentation/release-notes/13.02#DMA_protection_via_IOMMU]).
DMA transactions originating from PCI devices are subjected to the virtual
address space of the device PD.
All DMA buffers are identity-mapped with their physical addresses within
the device PD. On 32-bit systems with more than 3 GiB of memory, this
creates a problem. Because the device PD is a regular user-level component, the
upper 1 GiB of its virtual address space is preserved for the kernel. Since
no user-level memory objects can be attached to this
area, the physical address range to be used for DMA buffers is limited
to the lower 3 GiB.
Up to now, Genode components had no way to influence the allocation of
memory with respect to physical address ranges. To solve the problems outlined
above, we extended core's RAM services to take allocation constraints
as session arguments when a RAM session is created. All dataspaces created
from such a session are subjected to the specified constraints. In particular,
this change enables the AHCI/PCI driver to allocate DMA buffers at suitable
physical address ranges.
This innocent looking feature to constrain RAM allocations raises a problem
though: If any component is able to constrain RAM allocations in
arbitrary ways, it would become able to scan the physical address space for
allocated memory by successively opening RAM sessions with the constraints set
to an individual page and observe whether an allocation succeeds or not. Two
conspiring components could use this information to construct a covert storage
channel.
To prevent such an abuse, the init component filters out allocations
constrains from RAM-session requests unless explicitly permitted. The
permission is granted by supplementing the RAM resource assignment of
a component with a new 'constrain_phys' attribute. For example:
! <resource name="RAM" quantum="3M" constrain_phys="yes"/>
Init component
==============
Most of Genode's example scenarios in the form of run scripts support
different platforms. However, as the platform details vary, the run scripts
have to tweak the configuration of the init component according to the
features of the platform.
For example, when declaring an explicit route to a framebuffer driver named
"fb_drv", the run script won't work on Linux because on this platform, the
framebuffer driver is called "fb_sdl".
Another example is the role of the USB driver. Depending on the platform, the
USB driver is an input driver, a block driver, a networking driver, or a
combination of those.
Consequently, run scripts with support
for a great variety of platforms tend to become convoluted with
platform-specific conditionals.
To counter this problem, we enhanced init to support aliases for component
names. By defining the following aliases in the init configuration
! <alias name="nic_drv" child="usb_drv"/>
! <alias name="input_drv" child="usb_drv"/>
! <alias name="block_drv" child="usb_drv"/>
the USB driver becomes reachable for session requests routed to either "usb_drv",
"nic_drv", "input_drv", and "block_drv". Consequently, the routing
configuration of components that use either of those drivers does no longer
depend on any platform-intrinsic knowledge.
RTC session interface
=====================
Until now, the RTC session interface used an integer to return the current
time. Although this is preferable when performing time-related
calculations, a structured representation is more convenient to use, i.e., if
the whole purpose is showing the current time. This interface change is only
visible to components that use the RTC session directly.
Since the current OS API of Genode lacks time-related functions, most users
end up using the libc, which already converts the structured time stamp
internally, or provide their own time related functions.
Update of rump-kernel-based file systems
========================================
We updated the rump-kernel support to a newer rump-kernel version (as of mid of
January 2015). This way, Genode is able to benefit from upstream stability
improvements related to the memory management. Furthermore, we revised the
Genode backend to allow the rump_fs server to cope well with a large amount of
memory assigned to it. The latter is useful to utilize the block cache of the
NetBSD kernel.
Libraries and applications
##########################
As a stepping stone in the
[https://github.com/genodelabs/genode/issues/1399 - forthcoming community effort]
to bring the Nix package manager to Genode, ports of libbz2 and sqlite have
been added to the _repos/libports/_ repository.
Runtime environments
####################
VirtualBox on NOVA
==================
Whereas our previous efforts to run VirtualBox on Genode/NOVA were mostly
concerned with enabling principal functionality and with the addition of
features, we took the release cycle of Genode 15.02 as a chance to focus
on performance and stability improvements.
:Performance:
Our goal with VirtualBox on NOVA is to achieve a user experience
comparable to running VirtualBox on Linux. Our initial port of VirtualBox used
to cut a lot of corners with regards to performance and timing accuracy
because we had to concentrate on more fundamental issues of the porting
work first. Now, with the feature set settled, it was time to revisit
and solidify our interim solutions.
The first category of performance improvements is the handling of timing,
and virtual guest time in particular. In our original version,
we could observe a substantial drift of the guest time compared to the host time.
The drift is not merely inconvenient but may even irritate the guest OS
because it violates its assumptions about the behaviour of certain virtual devices.
The drift was caused by basing the timing on a simple jiffies counter
that was incremented by a thread after sleeping for a fixed period. Even
though the thread almost never executes, there is still a chance that it gets
preempted by the kernel and resumed only after the time slices of
concurrently running threads have elapsed. This can take tens of milliseconds.
During this time, the jiffies counter remains unchanged. We could
significantly reduce the drift by basing the timing on absolute time values
requested from the timer driver. Depending on the used guest OS, however,
there is still a residual inaccuracy left, which is subject to ongoing
investigations.
The second type of improvements is related to the handling of virtual
interrupts. In its original habitat, VirtualBox relies on so-called
external-interrupt virtualization events. If a host interrupt occurs while the
virtual machine is active, the virtualization event is forwarded by the
VirtualBox hypervisor to the virtual machine monitor (VMM).
On NOVA, however, the kernel does not propagate this
condition to the user-level VMM because the occurrence of host interrupts should
be of no matter to the VMM. In the event of a host interrupt, NOVA takes
a normal scheduling decision (eventually activating the user-level device driver
the interrupt belongs to) and leaves the virtual CPU (vCPU) in a runnable
state - to be rescheduled later. Once the interrupt is handled, the vCPU gets
resumed. The VMM remains out of the loop. Because the update of the VirtualBox
device models ultimately relies on the delivery of external-interrupt
virtualization events, the lack of this kind of event introduced huge delays
with respect to the update of device models and the injection of virtual
interrupts. We solved this problem by exploiting a VirtualBox-internal
mechanism called POKE. By setting the so-called POKE flag, an I/O thread is
able to express its wish to force the virtual machine into the VMM. We only
needed to find the right spots to set the POKE flag.
Another performance-related optimization is the caching of RTC time
information inside VirtualBox. The original version of the gettimeofday
function used by VirtualBox contacted the RTC server for obtaining the
wall-clock time on each call. After the update to VirtualBox 4.3, the rate of those
calls increased significantly. To reduce the costs of these calls, our
new version of gettimeofday combines infrequent calls to the RTC driver
with a component-local time source based on the jiffies mechanism mentioned above.
With these optimizations in place,
simple benchmarks like measuring the boot time of Window 7 or the time of
compiling Genode within a Debian VM suggest that our version of VirtualBox
has reached a performance that is roughly on par with the Linux version.
:Stability:
Since the upgrade to VirtualBox 4.3.16 in release 14.11, we fixed several
regression issues caused by the upgrade. Beside that, we completed the
support to route serial output of guests to Genode, lifted the restriction
to use just one fixed VESA mode, and enabled support for 32-bit Windows 8
guests on 64-bit Genode/NOVA. The 64-bit host restriction stems from
the fact that Windows 8 requires support for the non-executable bit (NX)
feature of page tables. The 32-bit version of the NOVA kernel does not leverage
the physical address extension (PAE) feature, which is a pre-requisite for
using NX on 32-bit.
In the course of the adaptation, our port of VirtualBox now evaluates the
PAE and HardwareVirtExUX XML tags of .vbox files:
!<VirtualBox xmlns=...>
! <Machine uuid=...>
! <Hardware ..>
! <CPU ...>
! <HardwareVirtExUX enabled="true"/>
! <PAE enabled="true"/>
! ...
The PAE tag specifies whether to report PAE capabilities to the guest
or not. The HardwareVirtExUx tag is used by our port to decide whether to stay
for non-paged x86 modes in Virtualbox's recompiler (REM) or not. Until now, we used REM
to emulate execution when the guest was running in real mode and protected mode
with paging disabled. However, newer Intel machines support the unrestricted guest
feature, which makes the usage of REM in non-paged modes not strictly
necessary anymore. Setting the HardwareVirtExUx tag to false accommodates
older machines with no support for the unrestricted-guest feature.
Device drivers
##############
iPXE-based network drivers
==========================
We enabled and tested the driver with Intel I218-LM and I218-V PCI devices.
Intel wireless stack
====================
In this release, several small issues regarding the wireless stack are fixed.
From now on, the driver only probes devices on the PCI bus that correspond to
the PCI_CLASS_NETWORK_OTHER device class. Prior to that, the driver probed all
devices attached to the bus resulting in problems with other devices, e.g.
the GPU, when accessing their extended PCI config space.
Since the driver uses cooperative scheduling internally, it must never block
or, in case it blocks, must schedule another task. Various sleep functions
lacked this scheduling call and are now fixed. Furthermore, a bug in the timer
implementation has been corrected, which caused the scheduling of wrong timeouts.
In addition to these fixes, patches for enabling the support for
Intel 7260 cards were incorporated.
Up to now, the configuration of the wireless driver was rather inconvenient because
it did not export any information to the system. The driver now creates two
distinct reports to communicate its state and information about the wireless
infrastructure to other components. The first one is a list of all available
access points. The following exemplary report shows its structure:
!<wlan_accesspoints>
! <accesspoint ssid="skynet" bssid="00:01:02:03:04:05" quality="40"/>
! <accesspoint ssid="foobar" bssid="01:02:03:04:05:06" quality="70" protection="WPA-PSK"/>
! <accesspoint ssid="foobar" bssid="01:02:03:04:05:07" quality="10" protection="WPA-PSK"/>
!</wlan_accesspoints>
Each '<accesspoint>' node has attributes that contain the SSID and the BSSID
of the access point as well as the link quality (signal strength). These
attributes are mandatory. If the network is protected, the node will also
have an attribute describing the type of protection in addition.
The second report provides information about the state of the connection
with the currently associated access point:
!<wlan_state>
! <accesspoint ssid="foobar" bssid="01:02:03:04:05:06" quality="70"
! protection="WPA-PSK" state="connected"/>
!</wlan_state>
Valid state values are 'connected', 'disconnected', 'connecting' and
'disconnecting'.
The driver obtains its configuration via a ROM module. This ROM
module contains the selected access point and can be updated during runtime.
To connect to an access point, a configuration like the following is used:
!<selected_accesspoint ssid="foobar" bssid="01:02:03:04:05:06"
! protection="WPA-PSK" psk="foobar123!"/>
To disconnect from an access point, an empty configuration can be set:
!<selected_accesspoint/>
For now, the prevalent WPA/WPA2 protection using a pre-shared key is supported.
Improved UART driver for Exynos5
================================
The UART driver for the Exynos5 SoC has been enhanced by enabling the RX
channel. This improvement was motivated by automated tests, where a run script
needs to interact with some component via a terminal connection.
Touchscreen support
===================
We enabled support of Wacom USB touchscreen devices via dde_linux - a port of
Linux USB driver to Genode. In order to make touchscreen coordinates
usable by Genode's input services, they must be calibrated
to screen-absolute coordinates. The screen resolution is not determined
automatically by the USB driver. It can, however, be configured as a sub
node of the '<hid>' XML tag of the USB driver's configuration:
!<start name="usb_drv">
! ...
! <config uhci=... ohci=... xhci=...>
! <hid>
! <screen width="1024" height="768"/>
! </hid>
! ...
USB session interface
=====================
We enhanced our USB driver with the support of remote USB sessions. This
feature makes it possible to implement USB-device drivers outside the USB
server using a native Genode API. The new USB session can be found under
_repos/os/include/usb_session_ and can be used to communicate with the USB
server, which merely acts as a host controller and HUB driver in this scenario.
Under _repos/os/include/usb_, there are a number of convenience
and wrapper functions that operate directly on top of a USB session. These
functions are meant to ease the task of USB-device-driver programming by hiding
most of the USB session management, like packet-stream handling.
We also added a USB terminal server, which exposes a Genode terminal session to
its clients and drives the popular PL2303 USB to UART adapters using the new
USB-session interface.
A practical use case for this component is the transmission of logging data on
systems where neither UART, AMT, nor JTAG are available. A run script
showcasing this feature can be found at _repos/dde_linux/run/usb_terminal.run_.
RTC proxy driver for Linux
==========================
There are a handful of run scripts that depend on the RTC service. So far,
it was not possible to run these tests on Linux due to the lack of an RTC
driver on this platform. To address this problem, we created a proxy driver
that uses the time() system call to provide a
reasonable base period on Linux.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Support for the USB-Armory board
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
With [https://www.crowdsupply.com/inverse-path/usb-armory - USB Armory],
there is an intriguing hardware platform for Genode on the horizon.
In short, USB Armory is a computer in the form factor of a USB
stick. It is meant for security applications such as VPNs,
authentication tokens, and encrypted storage. It is based on the
FreeScale i.MX53 SoC, which is well supported by Genode, i.e.,
Genode can be used as secure-world OS besides Linux running in the
normal world.
Apart from introducing a novel form factor, this project is
interesting because it strives to be an 100% open platform, which
includes hardware, software, and firmware. This motivated us to
bring Genode to this platform.
The underlying idea is to facilitate
[https://genode.org/documentation/articles/trustzone - ARM TrustZone] to
use Genode as a companion to a Linux-based OS on the platform.
Whereas Linux would run in the normal world of TrustZone, Genode runs
in the secure world. With Linux, the normal world will control the
communication over USB and provide a familiar environment to implement
USB-Armory applications. However, security-critical functions and data like
cryptographic keys will reside exclusively in the secure world. Even in
the event that Linux gets compromised, the credentials of the user
will stay protected.
The support of the USB Armory platform was added in two steps:
First, we enabled our base-hw kernel to run as TrustZone monitor with
Genode on the "secure side". Since the USB Armory is based on the
FreeScale i.MX53 SoC, which Genode already supported, this step went
relatively straight-forward.
Second, we enabled a recent version of the Linux kernel (3.18) to run in the
normal world. The normal world is supervised by a user-level Genode component
called tz_vmm (TrustZone Virtual Machine Monitor). The tz_vmm is, among
others, responsible for providing startup and hardware information to the
non-secure guest. The Linux kernel version we used previously as TrustZone
guest on i.MX53 boards expected this information to be communicated via
so-called ATAGs. The new version, however, expects this to be done via a
device tree blob. As a consequence, the tz_vmm had to be adapted to properly
load this blob into the non-secure RAM. The original USB-Armory device tree
was modified to blind out the RAM regions that get protected by the TrustZone
hardware. This way, Linux won't attempt to access them. Furthermore,
to keep basic user interaction simple, our device tree tells Linux to use the
same non-secure UART as Genode for console I/O.
The kernel itself received some modifications, for two reasons. First,
we don't want Linux to rely on resources that are protected to keep
the secure world secure. This is why the driver for the interrupt controller
that originally made use of the TrustZone interrupt configuration, had to be
adapted. Second, to prevent Linux from disturbing Genode activities, we
disabled most of the dynamic clock and power management as it may sporadically
gear down or even disable hardware that Genode relies on. Furthermore, we
disabled the Linux drivers for I2C interfaces and the GPIO configuration as
these are reserved for Genode.
IPC helping
~~~~~~~~~~~
In traditional L4 microkernels, scheduling parameters (like time-slice
length and priority) used to be bound to threads. Usually, those parameters
are defined at thread creation time. The initial version
of base-hw followed this traditional approach. However, it has a few problems:
* For most threads, the proper *choice of scheduling parameters* is very
difficult if not impossible. For example, the CPU-time demands of a
server thread may depend on the usage patterns of its clients. Most
theoretical work in the domain of scheduling presumes the knowledge of
job lengths in advance of computing a schedule. But in practice and in
particular in general-purpose computing, job lengths are hardly known a priori.
As a consequence, in most scenarios, scheduling parameters are
set to default values.
* With each thread being represented as an independent schedulable entity,
the kernel has to take a scheduling decision each time a thread performs an
IPC call because the calling thread gets blocked and the called thread
may get unblocked. In a microkernel-based system, those events occur at a
much higher rate than the duration of typical time slices, which puts the
scheduler in a *performance-critical* position.
* Regarding IPC calls, a synchronous flow of control along IPC call chains is
desired. Ideally, an IPC call should have the same characteristics as
a function call with respect to scheduling. When a client thread performs an
IPC call, it expects the server to immediately become active to
handle the request. But if the kernel treats each thread independently,
it may pick any other thread and thereby introduce *high latencies* into
IPC operations.
To counter those problems, the NOVA microhypervisor introduced a new approach
that decouples scheduling parameters from threads. Instead of selecting
threads for execution, the scheduler selects so-called scheduling contexts.
For a selected scheduling context, the kernel dynamically determines a
thread to execute by taking IPC relationships into account. When a thread
performs an IPC, the thread's scheduling context will be used to execute
the called server. In principle, a server does not need CPU time on its own
but always works with CPU resources provided by clients.
The new version of the base-hw kernel adapts NOVA's approach with slight
modifications. Each thread owns exactly one scheduling context for its entire
lifetime. However, by the means of "helping" during an IPC call, the caller
lends its scheduling context to the callee. Even if the callee is still busy
and cannot handle the IPC request right away, the caller helps because it
wants the callee to become available for its request as soon as
possible. Consequently, a thread has potentially many scheduling contexts at
its disposal, its own scheduling context plus all scheduling contexts
provisioned by helpers. This works transitively.
Purged outdated platforms
~~~~~~~~~~~~~~~~~~~~~~~~~
We removed the support for two stale platforms that remained unused for
more than a year, namely FreeScale i.MX31 and the TrustZone variant
of the Coretile Versatile Express board.
NOVA
====
On Genode/NOVA, we used to employ one pager thread in core for each thread
in the system. We were forced to do so because not every page
fault can be resolved immediately. In some situations, core asynchronously
propagates the fault to an external component for the resolution.
In the meantime, the
pager thread leaves the page fault unanswered. Unfortunately, the kernel
provides no mechanism to support this scenario besides just blocking the
pager thread using a semaphore. This, in turn, means that the pager thread is not
available for other page-fault requests. Ultimately, we had to setup a
dedicated pager per thread.
This implementation has the downside of "wasting" memory for a lot of
pager threads. Moreover, it becomes a denial-of-service vector as soon as more
threads get created than core can accommodate. The number of threads is
limited per address space - also for core - by the size of Genode's context
area, which typically means 256 threads.
To avoid the downsides mentioned, we extended the NOVA IPC reply syscall to
specify an optional semaphore capability. The NOVA kernel validates the
capability and blocks the faulting thread in the semaphore. The faulted thread
remains blocked even after the pager has replied to the fault message. But
the pager immediately becomes available for other
page-fault requests. With this change, it suffices to maintain only one pager
thread per CPU for all client threads.
The benefits are manifold. First, the base-nova implementation converges more
closely to other Genode base platforms. Second, core can not run out of threads
anymore as the number of threads in core is fixed for a given setup. And the
third benefit is that the helping mechanism of NOVA can be leveraged for
concurrently faulting threads.
Build system and tools
######################
Tools for convenient handling of port contrib directories
=========================================================
We supplemented our tools for the ports mechanism with two convenient
scripts:
:_tool/ports/shortcut_:
Creates a symbolic link from _contrib/<port-name>-<hash>_ to
_contrib/<port-name>_. This is useful when working on the third-party
code contained in the _contrib_ directory.
:_tool/ports/current_:
Prints the current contrib directory of a port. When switching
branches back and forth, the hash of the used port might change.
The script provides a shortcut to looking up the hash file for a
specific port within the repositories and printing its content.

1216
doc/release_notes/15-05.txt Normal file
View File

File diff suppressed because it is too large Load Diff

791
doc/release_notes/15-08.txt Normal file
View File

@@ -0,0 +1,791 @@
===============================================
Release notes for the Genode OS Framework 15.08
===============================================
Genode Labs
The version 15.08 marks the beginning of Genode as day-to-day OS as one of the
project's core developers switched to using Genode/NOVA on his machine,
stressing the OS infrastructure we created over the course of the last seven
years. Thanks to components like VirtualBox, the Noux runtime for GNU software,
the Linux wireless stack and Rump-kernel-based file systems, the transition
went actually much smoother than expected. So other members of the team plan
to follow soon. Section [Genode as day-to-day operating system] gives an
overview of the taken approach. Genode's use as general-purpose OS provided
the incentive for most of the improvements featured by the current release,
starting with the addressing of the long-standing kernel-memory management
deficiencies of the NOVA kernel (Section [NOVA kernel-resource management]),
over enhancements of Genode's tracing and file-system facilities, to vast
improvements of the guest-host integration of VirtualBox when running on
Genode.
The release is accompanied with a second line of work led by our friends
at Codelabs: Enabling Genode to run on top of their Muen separation
kernel as described in Section [Genode on top of the Muen Separation Kernel].
Muen is a low-complexity kernel for the 64-bit x86 architecture that
statically partitions the machine into multiple domains. In contrast to
microkernels like the ones already supported by Genode, the assignment
of physical resources (such as memory, CPU time, and devices) happens at
system-integration time. Since an isolation kernel does not have to deal
with dynamic resource management at runtime, it is less complex than
a general-purpose microkernel. This makes it relatively easy to reason about
its strong isolation properties, which, in turn, makes it attractive for
high-assurance computing. With Genode being able to run within a Muen
domain, the rich component infrastructure of Genode can be combined with
the strong isolation guarantees of Muen.
Genode on top of the Muen Separation Kernel
###########################################
_This section was written by Adrian-Ken Rueegsegger and Reto Buerki who_
_conducted the described line of work independent from Genode Labs._
After completing our x86_64 port of the Genode base-hw kernel, which was
featured in the
[https://genode.org/documentation/release-notes/15.05#Principal_support_for_the_64-bit_x86_architecture - previous release (15.05)],
we immediately started working on our main goal: running a Genode system as
guest on the Muen Separation Kernel (SK). This would enable the Muen platform
to benefit from the rich ecosystem of Genode.
For those who have not read the 15.05 Genode release notes, [https://muen.sk - Muen]
is an Open-Source microkernel, which uses the [https://spark-2014.org/ - SPARK]
programming language to enable light-weight formal methods for high assurance.
The 64-bit x86 kernel, currently consisting of a little over 5'000 LOC, makes
extensive use of the latest Intel virtualization features and has been formally
proven to contain no runtime errors at the source-code level.
The new 'hw_x86_64_muen' platform, as the name implies, extends the 'hw_x86_64'
base-hw kernel by replacing the PIC and timer drivers with paravirtualized
variants.
In contrast to other kernels supported by Genode, the architecture with Muen is
different in the sense that the entire 'hw_x86_64_muen' Genode system runs as
guest VM in VMX non-root mode on the SK. From the perspective of Muen, Genode
is executed on top of the kernel like any other guest OS without special
privileges.
[image muen_system_overview]
Genode running on top of the Muen Separation Kernel alongside other subjects
This loose coupling of Muen and Genode base-hw enables the robust combination
of a static, low-complexity SK with a feature-rich and extensive OS framework.
The result is a flexible platform for the construction of component-based
high-assurance systems.
People interested in giving the 'hw_x86_64_muen' platform a spin can find a
small tutorial at _repos/base-hw/doc/x86_64_muen.txt_.
NOVA kernel-resource management
###############################
For several years, the NOVA kernel has served as Genode's primary base
platform on x86. The main reasons for this choice are: the kernel provides -
among the supported x86 kernels - the richest feature set like the support of
IOMMUs, virtualization, and SMP. It also offers a clean design and a stable
kernel interface. The available kernel-interface specification and the
readable and modern source base are a pleasure to work with. Hence, Genode
Labs is able to fully commit to the maintenance and further evolution of this
kernel.
Nevertheless, since the beginning, the vanilla kernel lacks one essential
feature to reliably host Genode as user-land, namely the proper management of
the memory used by the kernel itself (in short kernel-memory management). In
the past, we already extended the kernel to free up kernel resources when
destroying kernel objects, e.g., protection domains and page-tables, threads,
semaphores, and portals. Still, on Genode/NOVA, a component may trigger
arbitrary kernel-memory consumption during RPC by delegating memory,
capabilities, or by creating other components via Genode's core component. If
the kernel memory gets depleted, the kernel panics with an "Out of memory"
message and the entire Genode scenario stops.
In principal, the consumption of kernel memory can be deliberately provoked by
a misbehaving (greedy) component. But also during the regular day-to-day usage
of Genode, can such a situation occur when the system is used in a highly
dynamic fashion. For example, compiling and linking source code within the
noux environment constantly creates and destroys protection domains, threads,
and memory mappings. Our nightly test of compiling Genode within noux triggers
this condition every once in a while.
The main issue here is that the consumption of kernel memory is not accounted
by Genode. The kernel interface does not support such a feature. Kernels like
seL4 as well as Genode's custom base-hw kernel show how this problem can be
solved.
To improve the current situation - where the overall kernel memory is a fixed
amount - we extended NOVA in the following ways: First, the NOVA kernel
accounts any kernel memory consumption per protection domain. Second, each
process has a limited amount of kernel-memory quota it can use. Last, the
kernel detects when the quota limit of a protection domain is reached.
If the third condition occurs, the kernel stops the offending thread and
(optionally) notifies a handler thread. This so called out-of-memory (OOM)
handler thread receives information about the current situation and may
respond to it in the following ways:
* Stop the thread of the depleted protection domain, or
* Transfer kernel-memory quota between protection domains (upgrading the limit
if desired), or
* Free up kernel memory if possible, e.g., revoke memory delegations, which
can be re-created.
We implemented the steps above inside the NOVA kernel and extended Genode's
core component to handle such OOM situations. All system calls beside the IPC
call/reply may now return an error code upon depletion of the quota. Most of
these system calls can solely be performed by core and are handled inside
core's NOVA-specific platform code.
In the case of IPC call/reply operations, we desired to handle OOM cases
transparently to Genode user-level components. Therefore, each thread in
Genode/NOVA now gets constructed with an OOM IPC portal attached. This portal
is served by the pager thread in core and is traversed on OOM occurrences
during IPC operations. If a pager thread receives such an OOM IPC, it decodes
the involved IPC sender and IPC receiver and locates the appropriate
core-internal paging objects. The currently implemented out-of-memory policy
tries to upgrade the quota. If this is not possible, an attempt to revoke
memory mappings from the OOM-causing protection domain is made. This
implicitly frees-up some kernel memory (e.g., mapping nodes). If none of the
responses suffices, the handler stops the OOM-causing thread and writes a
message to the system log.
The current policy implementation constitutes a rather rough heuristic, which
may not suffice under all circumstances. In the future, we would like to
specify a distinct policy per component, e.g. depending on prior known memory
usage patterns. For example, some components follow well-known usage patterns
and therefore a fixed upper quota limit can be specified. Other components are
highly dynamic and desire quota upgrades on demand. There are many more
combinations imaginable.
Our current plan is to collect more experience over the next months with this
new kernel mechanism. Based on our observations, we may externalize such
policy decisions and possibly make them configurable per component.
The current implementation however, already avoids the situation that the
kernel goes out of service if a single component misbehaves
kernel-memory-wise.
Genode as day-to-day operating system
#####################################
At the beginning of June, Genode reached the probably most symbolic milestone
in the project's history: Norman - one of the core developers - replaced his
Linux-based working environment with a Genode-based system. This system is
composed of the following ingredients:
[image turmvilla_scenario]
The machine used is a Lenovo Thinkpad X201. We settled on this five-year-old
machine for several reasons. First, it is a very solid platform with a nice
form factor. Second, it features Intel's AMT (Active Management Technology),
which is handy to obtain low-level system logs in the case something goes
wrong. Third, refurbished machines of this type can be obtained for as little
as 200 EUR. Finally, an older machine reinforces the need for good performance
of the operating system. So it creates a natural incentive for Norman to find
and address performance bottlenecks.
Our modified version of the NOVA microhypervisor is the used kernel.
The user interface is based on our custom GUI stack including the nitpicker
GUI server as well as the window manager and its companion components
(decorator, layouter, pointer) we introduced in
[https://genode.org/documentation/release-notes/14.08#New_GUI_architecture - version 14.08].
The display is driven by the VESA driver. User input is handled by the PS/2
driver for handling the laptop keyboard and trackpoint, and the USB driver for
handling an externally connected keyboard and mouse.
Network connectivity is provided by our port of the Intel Wireless stack that
we introduced with the version
[https://genode.org/documentation/release-notes/14.11#Intel_wireless_stack - 14.11].
Our custom AHCI driver provides access to the physical hard disk. File-system
access is provided by our
[https://genode.org/documentation/release-notes/14.02#NetBSD_file_systems_using_rump_kernels - Rump-kernel-based file-system server].
A simple Genode shell called CLI monitor allows the user to start and kill
subsystems dynamically. Initially, the two most important subsystems are
VirtualBox and Noux.
VirtualBox executes a GNU/Linux-based guest OS that we refer to as "rich OS".
The rich OS serves as a migration path from GNU/Linux to Genode. It is used
for all tasks that cannot be accomplished directly on Genode yet. At the
beginning of the transition, the daily routine still very much depends on the
rich OS. By moving more and more functionality over to the Genode world, we
will eventually be able to make the rich OS obsolete step by step. Thanks to
VirtualBox' excellent host-guest-integration features, the VirtualBox window
can be dynamically resized and the guest mouse cursor integrates seamlessly
with Genode's pointer. VirtualBox is directly connected to the wireless
network driver. So common applications like Firefox can be used.
The noux runtime allows us to use command-line-based GNU software directly on
Genode. Coreutils and Bash are used for managing files. Vim is used for
editing files. Unlike the rich OS, the noux environment has access to the
Genode partition of the hard disk. In particular, it can be used to update the
Genode system. It has access to a number of pseudo files that contain status
information of the underlying components, e.g., the list of wireless access
points. Furthermore, it has limited access to the configuration interfaces of
the base components. For example, it can point the wireless driver to the
access point to use, or change the configuration of the nitpicker GUI server
at runtime.
As a bridge between the rich OS and the Genode world, we combine VirtualBox'
shared-folder mechanism with Genode's VFS infrastructure. The shared folder is
represented by a dedicated instance of a RAM file system, which is mounted in
both the VFS of VirtualBox and the VFS of noux.
As evidenced by Norman's use since June, the described system setup is
sufficient to be productive. So other members of the Genode team plan to
follow in his footsteps soon. At the same time, the continued use of the
system from day to day revealed a number of shortcomings, performance
limitations, and rough edges, which we eventually eliminated. It goes without
saying that this is an ongoing effort. Eating our own dog food forces us to
address the right issues to make the daily life more comfortable.
Feature-wise the switch to Genode motivated three developments, namely the
enhancement of Genode's CLI monitor, the improvement of the window manager,
and the creation of a CPU-load monitoring tool.
Interactive management of subsystem configurations
==================================================
The original version of CLI monitor obtained the configuration data of its
subsystems at start time via the Genode::config mechanism. But for managing
complex scenarios, the config node becomes very complex. Hence, it is
preferable to have a distinct file for each subsystem configuration.
The new version of CLI monitor scans the directory '/subsystems' for files
ending with ".subsystem". Each file has the same syntax as the formerly used
subsystem nodes. This change has the welcome implication that subsystem
configurations can be changed during the runtime of the CLI monitor, e.g., by
using a concurrently running instance of noux with access to the _subsystems/_
directory. This procedure has become an essential part of the daily work flow
as it enables the interactive evolution of the Genode system.
Window-management improvements
==============================
To make the window manager more flexible while reducing its complexity at the
same time, we removed the formerly built-in policy hosting the decorator and
layout components as children of the window manager. Those components are no
longer child components but siblings. The relationship of the components is
now solely expressed by the configuration of their common parent, i.e., init.
This change clears the way to dynamically replace those components during
runtime (e.g., switching between different decorators).
To improve the usability of the windowed GUI, we enabled the layouter to
raise windows on click and to let the keyboard focus follow the pointer.
Furthermore, the window manager, the decorator, and the floating window
layouter became able to propagate the usage of an alpha channel from the
client application to the decorator. This way, the decorator can paint the
decoration elements behind the affected windows, which would otherwise be
skipped. Consequently, partially transparent windows can be properly displayed.
CPU-load monitoring
===================
During daily system use, we started to wish to know in detail where the CPU
cycles are spent. For example, the access of a file by the rich OS involves
several components, including the guest OS itself, VirtualBox, rump_fs (file
system), part_blk (partition access), ahci_drv (SATA device access), core, and
NOVA. Investigating performance issues requires a holistic view of all those
components. For this reason, we enhanced our existing tracing infrastructure
(Section [Enhanced tracing facilities]) to allow the creation of CPU-load
monitoring tools. The first tool in this category is the graphical CPU-load
monitor located at _gems/app/cpu_load_display/_, which displays a timeline of
the CPU load where each thread is depicted with a different color. Thanks to
this tool, we have become able to explore performance issues in an interactive
way. In particular, it helped us to identify and resolve a long-standing
inaccuracy problem in our low-level timer service.
Base framework and low-level OS infrastructure
##############################################
Improved audio support
======================
In the previous release, we replaced our old audio driver with a new one that
provided the same audio-out session interface. Complementing the audio-out
session, we are now introducing a new audio-in session interface that can be
used to record audio frames. It is modeled after the audio-out interface in
the way how it handles the communication between the client and the server. It
uses shared memory in the form of the Audio_in::Stream to transport the frames
between the components. A server component captures frames and puts them into
a packet queue, which is embedded in the Audio_in::Stream. The server
allocates packets from this queue to store the recorded audio frames. If the
queue is already full, the server will override already allocated packets and
will notify the client by submitting an 'overrun' signal. The client has to
cope with this situation, e.g., by consuming packets more frequently. A client
can install a signal handler to respond to a progress signal, which is sent by
the server when a new Audio_in::Packet has been submitted to the packet queue.
For now, all audio-in server components only support one channel (left)
although the audio-in session interface principally supports multiple
channels.
The _dde_bsd_ audio_drv is the first and currently only audio driver component
that was extended to provide the audio-in session. To express this fact, the
driver was renamed from _audio_out_drv_ to _audio_drv_. In contrast to its
playback functionality, which is enabled by default, recording has to be
enabled explicitly by setting the configuration attribute 'recording' to
'yes'. If the need arises, playback may be disabled by setting 'playback' to
'no'. In addition, it is now possible to configure the driver by adjusting the
mixer in the driver's configuration node. For the time being, the interface as
employed by the original OpenBSD mixer utility is used.
The following snippet shows how to enable and configure recording on a
Thinkpad X220 where the headset instead of the internal microphone is used as
source:
! <start name="audio_drv">
! <resource name="RAM" quantum="8M"/>
! <provides>
! <service name="Audio_out"/>
! <service name="Audio_in"/>
! </provides>
! <config recording="yes">
! <mixer field="outputs.master" value="255"/>
! <mixer field="record.adc-0:1_source" value="sel2"/>
! <mixer field="record.adc-0:1" value="255"/>
! </config>
! </start>
In addition to selecting the recording source, the playback as well as the
recording volume are raised to the maximum. Information about all available
mixers and settings in general may be obtained by specifying the 'verbose'
attribute in the config node.
The enriched driver is accompanied by a simple monitor application, which
directly plays back all recorded audio frames and shows how to use the
audio-in session. It can be tested by executing the
_repos/dde_bsd/run/audio_in.run_ run script.
There are also changes to the audio-out session itself. The length of a period
was reduced from 2048 to 512 samples to accommodate for a lower latency when
mixing audio-out packets. A method for invalidating all packets in the queue
was also added.
File-system infrastructure
==========================
Unlike traditional operating systems that rely on a global name space for
files, each Genode component has a distinct view on files. Many low-level
components do not even have the notion of files. Whereas traditional operating
systems rely on a virtual file system (VFS) implemented in the OS kernel,
Genode's VFS has the form of a library that can optionally be linked to a
component. The implementation of this library originated from the noux runtime
introduced in version
[https://genode.org/documentation/release-notes/11.02#Noux_-_an_execution_environment_for_the_GNU_userland - 11.02],
and was later integrated into our C runtime in version
[https://genode.org/documentation/release-notes/14.05#Per-process_virtual_file_systems - 14.05].
With the current release, we take the VFS a step further by making it
available to components without a C runtime. Thereby, low-complexity
security-sensitive components such as CLI monitor become able to benefit from
the powerful VFS infrastructure.
The VFS itself received a welcome improvement in the form of private RAM file
systems. A need for process-local storage motivated a conversion of the
existing ram_fs server component to an embeddable VFS file system. This
addition to the set of VFS plugins enables components to use temporary file
systems without relying on the resources of an external component.
Unified networking components
=============================
Having had a good experience with our Block::Driver implementation, which
wraps the block-session interface and takes care of the packet-stream
handling, thus easing the implementation of driver and other block components,
we observed that this approach did not provide enough flexibility for
NIC-session servers. For example, NIC servers are bi-directional and when a
network packet arrives the server has to make sure that there are enough
resources available to dispatch the network packet to the client. This has to
be done because the server must never block, e.g., by waiting for allocations
to succeed or for an empty spot in the packet queue of a client. Therefore,
such a non-blocking NIC server needs to validate all preconditions for
dispatching the packet in advance and, if they cannot be met, drop the network
packet.
In order to implement this kind of behavior, NIC-session servers must have
direct access to the actual NIC session. For this reason, we removed the
Nic::Driver interface from Genode and added a Nic::Session_component that
offers common basic packet-stream-signal dispatch functionality. Servers may
now inherit from this component and implement their own policy.
We adjusted all servers that implement NIC sessions to the new interface
(dde_ipxe, wifi, usb, nic_bridge, OpenVPN, ...), and thereby unified all
networking components within Genode.
Enhanced tracing facilities
===========================
Recent Genode-based system scenarios like the one described in Section
[Genode as day-to-day operating system] consist of dozens of components that
interact with each other. For reasoning about the behaviour of such scenarios
and identifying effective optimization vectors, tools for gathering a holistic
view of the system are highly desired.
With the introduction of our light-weight
[https://genode.org/documentation/release-notes/13.08#Light-weight_event_tracing - event-tracing facility]
in version 13.08, we laid the foundation for such tools. The current release
extends core's TRACE service with the ability to obtain statistics about CPU
utilization. More specifically, it enables clients of core's TRACE service to
obtain the execution times of trace subjects (i.e., threads). The execution
time is delivered as part of the 'Subject_info' structure. In addition to the
execution time, the structure delivers the information about the affinity of
the subject with a physical CPU.
At the current stage, the feature is available solely on NOVA since this is
our kernel of choice for using Genode as our day-to-day OS. On all other base
platforms, the returned execution times are 0. To give a complete picture of
the system's threads, the kernel's idle threads (one per CPU) are featured as
trace subjects as well. Of course, idle threads cannot be traced but their
corresponding trace subjects allow TRACE clients to obtain the idle time of
each CPU.
By obtaining the trace-subject information in periodic intervals, a TRACE
client is able to gather statistics about the CPU utilization attributed to
the individual threads present (or no longer present) in the system. One
instance of such a tool is the new trace-subject reporter located at
_os/src/app/trace_subject_reporter_. It acts as a TRACE client, which delivers
the gathered trace-subject information in the form of XML-formatted data to a
report session. This information, in turn, can be consumed by a separate
component that analyses the data. In contrast to the low-complexity
trace-subject reporter, which requires access to the privileged TRACE services
of core, the (potentially complex) analysing component does not require access
to core's TRACE service. So it isn't as critical as the trace-subject monitor.
The first representative of a consumer of trace-subject reports is the
CPU-load display mentioned in Section [CPU-load monitoring] and depicted in
Figure [nano3d].
In addition to the CPU-monitoring additions, the tracing facilities received
minor refinements. Up to now, it was not possible to trace threads that use a
CPU session other than the component's initial one. A specific example is
VirtualBox, which employs several CPU sessions, one for each priority. This
problem has been solved by associating the event logger of each thread with
its actual CPU session. Consequently, the tracing mechanism has become able to
trace VirtualBox, which is pivotal for our further optimizations.
Low-complexity software rendering functions
===========================================
Our ambition to use Genode as our day-to-day OS raises the need for custom
graphical applications. Granted, it is principally possible to base such
applications on Qt5, which is readily available to native Genode components.
However, for certain applications like status displays, we prefer to avoid the
dependency on an overly complex GUI tool kit. To accommodate such
applications, Genode hosts a small collection of low-complexity graphics
functions called painters. All of Genode's low-complexity graphical components
such as nitpicker, launchpad, window decorator, or the terminal are based on
this infrastructure.
With the current release, we extend the collection with two new painters
located at _gems/include/polygon_gfx_. Both draw convex polygons with an
arbitrary number of points. The shaded-polygon painter interpolates the color
and alpha values whereas the textured-polygon painter applies a texture to the
polygon. The painters are accompanied by simplistic 3D routines located at
_gems/include/nano3d/_ and a corresponding example (_gems/run/nano3d.run_).
[image nano3d]
With the nano3d demo and our new CPU load display, the screenshot above shows
two applications that make use of the new graphics operations.
Device drivers
##############
Completing the transition to the new platform driver
====================================================
Until now, the platform driver on x86-based machines was formed by the ACPI
and PCI drivers. The ACPI driver originally executed the PCI driver as a slave
(child) service. The ACPI driver parsed the ACPI tables and provided the
relevant information as configuration during the PCI-driver startup. We
changed this close coupling to the more modern and commonly used
[https://genode.org/documentation/release-notes/14.02#New_session_interface_for_status_reporting - report_rom mechanism].
When the new ACPI driver finishes the ACPI table parsing, it provides the
information via a report to any interested and registered components. The
report contains among other the IRQ re-routing information. The PCI driver is
a component, which - according to its session routing configuration - plays
the role of a consumer of the ACPI report.
With this change of interaction of ACPI and PCI driver, the policy for devices
must be configured solely at the PCI driver and not at the ACPI driver. The
syntax, however, stayed the same as introduced with release 15.05.
Finally, the PCI driver 'pci_drv' got renamed to 'platform_drv' as already
used on most ARM platforms. All files and session interfaces containing
PCI/pci in the names were renamed to Platform/platform. The x86 platform
interfaces moved to _repos/os/include/platform/x86/_ and the implementation of
the platform driver to _repos/os/src/drivers/platform/x86/_.
An example x86 platform configuration snippet looks like this:
!<start name="acpi_drv" >
! <resource .../>
! <route>
! ...
! <service name="Report"> <child name="acpi_report_rom"/> </service>
! </route>
!</start>
!
!<start name="acpi_report_rom" >
! <binary name="report_rom"/>
! <resource .../>
! <provides> <service name="ROM" /> <service name="Report" /> </provides>
! <config>
! <rom> <policy label="platform_drv -> acpi" report="acpi_drv -> acpi"/> </rom>
! </config>
! <route> ... </route>
!</start>
!
!<start name="platform_drv" >
! <resource name="RAM" quantum="3M" constrain_phys="yes"/>
! <provides> <service name="Platform"/> </provides>
! <route>
! <service name="ROM">
! <if-arg key="label" value="acpi"/> <child name="acpi_report_rom"/>
! </service>
! ...
! </route>
! <config>
! <policy label="ps2_drv"> <device name="PS2"/> </policy>
! <policy label="nic_drv"> <pci class="ETHERNET"/> </policy>
! <policy label="fb_drv"> <pci class="VGA"/> </policy>
! <policy label="wifi_drv"> <pci class="WIFI"/> </policy>
! <policy label="usb_drv"> <pci class="USB"/> </policy>
! <policy label="ahci_drv"> <pci class="AHCI"/> </policy>
! <policy label="audio_drv"> <pci class="AUDIO"/> <pci class="HDAUDIO"/> </policy>
! </config>
!</start>
In order to unify and simplify the writing of run scripts, we added the
commonly used platform configuration to the file
_repos/base/run/platform_drv.inc_. This file may be included by any test run
script in order to setup a default platform driver configuration.
In addition, the snippet provides the following functions:
'append_platform_drv_build_components', 'append_platform_drv_config' and
'append_platform_drv_boot_modules'. The functions add necessary information to
the 'build_components', 'config' and 'boot_modules' run variables. The
_platform_drv.inc_ also contains the distinction between various ARM/x86
platforms and includes the necessary pieces. Hence, run scripts are largely
relieved from platform-specific peculiarities.
The body of an example run script looks like this:
! set build_components { ... }
!
! source ${genode_dir}/repos/base/run/platform_drv.inc
! append_platform_drv_build_components
!
! build $build_components
!
! create_boot_directory
!
! set config { ... }
!
! append_platform_drv_config
!
! append config { ... }
!
! install_config $config
!
! append_platform_drv_boot_modules
!
! build_boot_image $boot_modules
!
! run_genode_until ...
BCM57cxx network cards
======================
During Hack'n Hike 2015, we had access to a server that featured a Broadcom
network card. Therefore Guido Witmond performed the first steps to enable
Broadcom's BCM 57cxx cards. With this preliminary work in place, we were
quickly able to perform the additional steps required to add BCM 57cxx support
to Genode.
VESA driver refinements
=======================
The VESA driver now reports the frame buffer's line width instead of the
visible width to the client. This fixes a possible distortion if these widths
differ, at the cost that content in the right-most area might be invisible in
such cases.
VirtualBox
##########
Policy-based mouse pointer
==========================
In the previous release, we implemented support for the transparent
integration of the guest mouse pointer with nitpicker via the VirtualBox guest
additions and the vbox_pointer component, which is capable of rendering
guest-provided mouse-pointer shapes. Now, we extended vbox_pointer by a
policy-based configuration that allows the selection of ROMs containing the
actual mouse shape based on the nitpicker session label or domain. With this
feature in place, it is possible to integrate several VirtualBox instances as
well as dedicated pointer shapes for specific components. To see the improved
vbox_pointer in action give _run/vbox_pointer_ a shot.
Dynamic adaptation to screen size changes
=========================================
VirtualBox now notifies the guest operating system about screen-size changes
(for example if the user resizes a window, which shows the guest frame
buffer). The VirtualBox guest additions can use this information to adapt the
guest frame buffer to the new size.
SMP support
===========
Guest operating systems can now use multiple virtual CPUs, which are mapped to
multiple host CPUs. The number of virtual CPUs can be configured in the
'.vbox' file.
Preliminary audio support
=========================
At some point, the use of VirtualBox as a stop-gap solution for using Genode
as everyday OS raises the need to handle audio. With this release, we address
this matter by enabling preliminary audio support in our VirtualBox port. A
back end that uses the audio-out and audio-in sessions to playback and record
sound samples has been added. It disguises itself as the OSS back end that is
already used by vanilla VirtualBox. Since Genode pretends to be FreeBSD in the
eyes of VirtualBox (because Genode's libc is based on FreeBSD's libc), the
provisioning of an implementation of the OSS back end as used on FreeBSD host
systems is the most natural approach. The audio support is complemented by
adding the necessary device models for the virtual HDA as well as the AC97
devices to our VirtualBox port.
For now, it is vital to have the guest OS configure the virtual device in a
way that considers the current implementation. For example, we cannot
guarantee distortion-free playback or recording if the guest OS uses a period
that is too short, typically 10ms or less. There are also remaining issues
with the mixing/filtering code in VirtualBox. Therefore, we bypass it to
achieve better audio quality. As a consequence, the device model of the VM has
to use the same sample rate as is used by the audio-out and audio-in sessions
(44.1kHz).
Enabling audio support is done be adding
! <AudioAdapter controller="HDA" driver="OSS" enabled="true"/>
to the .vbox file manually or configuring the VM accordingly by using the GUI.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Bender chain loader on base-hw x86_64
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Intel platforms, we use the Bender chain loader from the
[https://github.com/alex-ab/morbo - Morbo multiboot suite] to detect available
COM ports of PCI plug-in cards, the AMT SOL device, or as fall back the
default comport 1. The loader stores the I/O port information of the detected
cards into the BIOS data area (BDA), from where it is retrieved by core on
boot and subsequently used for logging. With this release, we added the BDA
parsing to base-hw on x86-64 and enabled the feature in the run tool. As a
prerequisite, we had to fix an issue in bender triggered by the loading of
only one (large) multi-boot kernel. Consequently, its binary in
_tool/boot/bender_ was updated.
Revised page-table handling
~~~~~~~~~~~~~~~~~~~~~~~~~~~
One of the main advantages of the base-hw platform is that the memory trading
concept of Genode is universally applied even with regard to kernel objects.
For instance, whenever a component wants to create a thread, it pays for the
thread's stack, UTCB, and for the corresponding kernel object. The same
applies to objects needed to manage the virtual address space of a component
with the single exception of page tables.
Normally, when the quota, which was donated by a component to a specific
service, runs out, the component receives an exception the next time it tries
to invoke the service. The component can respond by upgrading the respective
session quota. However, in the context of page-fault resolution, this is
particularly difficult to do. The allocation and thereby the shortage of
memory becomes evident only when the client produces a page fault. Therefore,
there is no way to inform the component to upgrade its session quota before
resolving the fault.
Instead of designing a sophisticated protocol between core and the other
components to solve this problem, we decided to simplify the current
page-fault resolution by using a static set of page-tables per component.
Formerly, page tables were dynamically allocated from core's memory allocator.
Now, an array of page tables gets allocated during construction of a
protection domain. When a component runs out of page tables, all of its
mappings get flushed, and the page tables are populated from scratch. This
change greatly simplifies the page-table handling inside of base-hw.
Dynamic interrupt mode setting on x86_64
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On x86-based hardware, user-level device drivers have become able to specify
the trigger mode and polarity of the interrupts when requesting an IRQ
session. On ARM, those session parameters are ignored. This change enables the
x86_64 platform to support devices, which use arbitrary trigger modes and
polarity settings, e.g., AHCI on QEMU and real hardware.
Fiasco.OC
=========
Genode's device-driver support when using the Fiasco.OC kernel as base
platform received an upgrade.
First, principle support for the Raspberry Pi was added. To make this platform
useful in practice, a working USB driver is important. I.e., the network
interface is connected via USB. Hence the USB driver got enabled for
Fiasco.OC, too. As a result, Genode's software stack can now be used on the
Raspberry Pi by using either our custom base-hw kernel or Fiasco.OC.
Second, support for the Odroid-X2 platform using the Exynos4412 SoC was added,
which includes the drivers for clock management (CMU), power management
(PMU) as well as USB.
Thanks to Reinier Millo Sánchez and Alexy Gallardo Segura for having
contributed this line of work.
Removal of deprecated features
##############################
We dropped the support for the *ARM Versatile Express* board from the Genode
source tree to relieve our automated testing infrastructure from supporting a
platform that remained unused for more than two years.
The device driver environment kit (DDE Kit) was originally intended as a
common API among the execution environments of ported user-level device
drivers. However, over the course of the past years, we found that this
approach could not fulfill its promise while introducing a number of new
problems. We reported our experiences in the release notes of versions
[https://genode.org/documentation/release-notes/12.05#Re-approaching_the_Linux_device-driver_environment - 12.05] and
[https://genode.org/documentation/release-notes/14.11#Roundup - 14.11].
To be able to remove the DDE-Kit API, we reworked the USB driver, our port of
the Linux TCP/IP stack, and the wireless driver accordingly.

1429
doc/release_notes/15-11.txt Normal file
View File

File diff suppressed because it is too large Load Diff

652
doc/release_notes/16-02.txt Normal file
View File

@@ -0,0 +1,652 @@
===============================================
Release notes for the Genode OS Framework 16.02
===============================================
Genode Labs
With version 16.02, we add RISC-V to Genode's supported CPU architectures,
enable the secure pass-through of individual USB devices to virtual machines,
and update the support for the Muen and seL4 kernels.
Trustworthy hardware becomes an increasingly pressing problem. With each new
generation of today's commodity hardware comes a dramatic increase of
complexity, the addition of proprietary companion processors, and opaque
firmware blobs. Even with a perfectly secure operating system, the user's
privacy and security remains at risk as there is no way to assess the
trustworthiness of our underlying hardware. RISC-V is a new hardware
architecture that tries to overcome this problem by the means of open source
and transparency. It is designed to scale from micro controllers to
general-purpose computers, and to be both synthesizable as FPGA softcores and
implementable in ASICs. The prospect of a scalable and trustworthy open-source
hardware platform motivated us to add RISC-V to Genode's supported CPU
architectures. Section [New support for the RISC-V CPU architecture] gives a
brief overview of this line of work.
Thanks to the growing number of our regular developers using Genode as day to
day OS, we create a natural incentive to address typical desktop-OS work
flows. In particular, the new version comes with the ability to assign
individual USB devices to VirtualBox instances. Conceptually, this looks like
a relatively straight-forward feature. But as discussed in Section
[Assignment of USB devices to virtual machines], we had to overcome a number of
challenging problems caused by the inherently dynamic nature of USB-device
hot-plugging. Also on the account of day-to-day computing, the GUI stack
received welcomed usability improvements like keyboard shortcuts for certain
window-management operations.
With respect to Genode's underlying base platforms, we are happy to announce
the updates of the Muen and seL4 kernels. The Muen separation kernel received
an update to version 0.7, which accommodates Genode's regular work flows (via
run scripts) much better than the previous version. As described in Section
[Muen separation kernel], this change clears the way to subject Muen to
Genode's regular automated tests. The seL4 kernel represents an exciting
playground as a future base platform for Genode. We have updated the kernel to
version 2.1, which prompted us to fundamentally revisit the low-level resource
management of Genode on this kernel. A summary of this undertaking is presented
in Section [seL4 version 2.1].
According to the [https://genode.org/about/road-map - road map], we originally planned to
revise the framework API in this release. Even though this topic is
[https://github.com/genodelabs/genode/issues/1832 - very actively pursued], we
decided to not rush it. We find it important to provide a smooth migration path
from the old API to the new one. Determining the best path is actually trickier
than revising the API, though. To let our decisions settle a bit, we postpone
the transition to the upcoming release.
Assignment of USB devices to virtual machines
#############################################
As a migration strategy for running Genode on a daily basis, using VirtualBox
to execute a feature-rich OS is vital. In release
[https://genode.org/documentation/release-notes/15.05#USB-device_pass-through_support - 15.05],
we added USB pass-through support to VirtualBox by enabling its integrated USB
proxy service. Since we use the open-source edition of VirtualBox, we were
merely able to use the OHCI device model and were therefore limited to using
USB 1.x devices in low and full speed mode only. To make matters worse, when
using the OHCI controller model, it is difficult if not impossible to access
USB mass-storage devices. Usually, VirtualBox facilitates the EHCI or xHCI
device models for the pass-through of storage devices. Unfortunately, those
models are only available as a proprietary extension, which cannot be used by
our VirtualBox port.
Having support for the pass-through of high-speed and super-speed USB devices
is a must in such controller models. Therefore, we either have to implement
these models ourselves or port existing ones from another VMM or emulator to
fill the gap. We went for porting existing models first because device-model
development from scratch could end up being time consuming if we want to
guarantee them to work with a variety of different OS drivers.
QEMU xHCI device model
----------------------
QEMU features a NEC xHCI (UPD720200) device model that works well with Windows
guests. For this reason, we decided to give porting this device model a shot.
We applied the DDE approach and started by creating a QEMU emulation
environment so that only the bare minimum amount of source code needed to be
taken from the QEMU sources. It came down to a handful of source files, mainly
the USB core and the xHCI device model files. We iteratively extended the
emulation environment until the QEMU sources compiled and linked fine. One
particular cumbersome issue we had to overcome was the emulation of the QEMU
Object Model. Since QEMU is written in C, it uses its own object model to
implement inheritance. This object model is used throughout QEMU. We took the
easy way out and just used a C++ wrapper class that contains all QEMU objects
that are used in the USB subsystem.
The next step was to develop a USB host device model. This model connects a
USB device attached to Genode's USB host-controller driver to the xHCI device
model. Lucky for us, QEMU already contains a USB host device model that uses
libusb, which we could use as blueprint. We implemented a USB host device that
leverages Genode's custom USB session interface. This host device reacts to a
USB device report coming from another component such as the host-controller
driver. It tries to claim all devices it finds in that report and then creates
a QEMU USB device for each of them that is attached to the xHCI device model.
The xHCI device model needs infrastructure that normally is provided by QEMU
itself such as a timer queue and PCI device handling. We introduced a QEMU
USB controller interface _repos/libports/include/qemu/usb.h_ whose back-end
library interface has to be implemented by a component, i.e. the VMM, that
wants to use the library.
In the end, this work resulted in a small library that contains the xHCI
device model and works in a standalone way. All required resources have to be
provided by the component using the library. This makes it easy to integrate
the library in different VMMs because the user of the library is not forced to
employ the library in a certain way but free to use it any way he chooses.
xHCI device model wrapper in VirtualBox
---------------------------------------
We implemented an xHCI device model _repos/port/src/virtualbox/devxhci.cc_ in
VirtualBox that merely wraps the QEMU USB library and provides the back-end
functionality required by the library to glue QEMU's xHCI device model to
VirtualBox. For now, this device is always part of a VM because there is
currently no way to disable it from within the VirtualBox configuration
front end. Therefore, it is necessary to always give VirtualBox access to a
_usb_devices_ ROM module.
We removed the afore mentioned USB proxy service from our VirtualBox port
because it became redundant with the advent of our xHCI device model.
USB device report filter
------------------------
With the xHCI support in VirtualBox in place, we had to come up with a
mechanism to select, which USB devices it may access. Since USB devices are
usually hot-plugged by the user of the system, we need to be able to configure
the access permissions dynamically at run-time. On this account, we created a
component that intercepts the report from the USB host-controller driver. On
the one hand, this USB device report-filter component screens the device
report coming from the USB host-controller driver by checking each reported
device against a given white list of devices. Only approved devices are
reported to a consumer of the report, i.e. VirtualBox. On the other hand, this
component generates a new configuration for the USB host-controller driver.
The configuration has to be changed each time the filter component finds a
suitable device because the driver will hand out access to a given device to a
client only if there is a valid policy. As we do not know in advance, which
devices might be plugged in, this policy must be maintained dynamically. The
report filter will send the device report only if the host-controller driver
has changed its configuration. This ensures that a matching policy will be in
effect at the time when the client component tries to access the device.
The configuration of the report-filter component can also be changed at run
time.
See _repos/os/src/app/usb_report_filter/README_ for more details on how the
USB device report filter may be configured.
Example configuration
---------------------
The following figure illustrates the interplay and configuration of the
involved components:
[image qemu_xhci]
When the user plugs in a USB device, the USB host-controller driver generates
a device report that is consumed by the USB device report-filter component
(1). The filter component then examines the report and checks if it contains a
device it should report to its report consumer. It then reconfigures the
host-controller driver (2). Afterwards it sends a report to its consumer (3).
The consumer, in this case a VMM, then accesses the USB device (4).
New support for the RISC-V CPU architecture
###########################################
We became aware of [https://riscv.org - RISC-V] when attending several talks
about the project at [https://fosdem.org - FOSDEM] in 2015. RISC-V aims to be
an open-source hardware architecture and is now complemented by many projects
that target the release of real hardware or ASICs (for example,
[https://www.lowrisc.org - the LowRISC project]). We have experience with various
major CPU architectures and many systems on a chip and, therefore, embrace a
sharp eye on certain platform properties. Intel's ME and ARM's Trustzone
practically lock out operating systems of certain hardware and firmware
features. The true nature of these mechanisms becomes increasingly dubious,
especially when trying to build a secure open-source operating system. Intel's
AMT technology for instance comes with a complete TCP/IP stack that intercepts
packets from the integrated NIC and a VNC server that can magically expose a
mouse and a keyboard at the USB controller. If you are interested in more
details about this topic
[https://blog.invisiblethings.org/papers/2015/x86_harmful.pdf - Intel x86 considered harmful]
by Joanna Rutkowska is a very good read. We decided to have a deeper look at
the RISC-V architecture as an alternative open hardware platform. Especially,
since the LowRISC project promises a completely open system on chip, including
the peripherals.
RISC-V comes with a lot of optional features, so it can cover a large field of
applications reaching from simple I/O processors to general-purpose computing.
For example, there are 64 and 32 bit ISA (instruction set architecture)
versions, three page table formats with the option to omit paging at all, up
to four privilege modes, and a minimal integer core ISA (I). Everything else,
like multiplication and division (M), atomic instructions (A), and floating
point support (F) are subject to ISA extensions and are completely optional
for a specific hardware implementation.
For Genode, we chose to add the RISC-V support to our custom _base-hw_ kernel.
Since Genode may be used as a general purpose OS, we implemented the kernel
using the 64 bit RISC-V version, the Sv39 three-level page table format, and
the so-called general-purpose extension (G), which is the abbreviation for the
IAMF extensions. The current implementation provides the kernel and the
necessary adaptations of the user level part of core.
For testing, we used the RISC-V instruction emulator called
[https://github.com/riscv/riscv-isa-sim - Spike]. There also exists a RISC-V
implementation for various Zynq FPGAs. Genode's Zynq board support has kindly
been added and contributed by Mark Vels.
In the current state, basic Genode applications including core, init, and
components that use shared libraries can be executed on top of our RISC-V
port. We did not enable the libc and postponed further activity as the
platform currently does not specify the interaction with peripherals.
Steps to test Genode on RISC-V
------------------------------
# Building the instruction emulator
! # download the front end server
! git clone https://github.com/ssumpf/riscv-fesvr.git
!
! # build the front end server
! cd riscv-fesvr
! mkdir build
! cd build
! export RISCV=<installation path>
! ../configure --prefix=$RISCV
! (sudo) make install
!
! # download the instruction emulator
! cd ../../
! git clone https://github.com/ssumpf/riscv-isa-sim.git
! cd riscv-isa-sim
!
! # build the emulator
! mkdir build
! cd build
! ../configure --prefix=$RISCV --with-fesvr=$RISCV
! (sudo) make install
!
! # add $RISCV/bin to path
! export PATH=$RISCV/bin:$PATH
# Building Genode and running a test scenario
! # download Genode
! cd ../../
! git clone https://github.com/genodelabs/genode.git
!
! # build the Genode tool chain
! cd genode
! ./tool/tool_chain riscv
!
! # create RISC-V build directory
! ./tool/create_builddir hw_riscv
! cd build/hw_riscv
!
! # build and execute the printf run script
! make run/printf
GUI stack usability improvements
################################
Motivated by the daily use of Genode as desktop OS by an increasingly number
of developers, the window-layouter component of the
[https://genode.org/documentation/release-notes/15.11#GUI_stack - GUI stack]
received welcomed usability improvements.
Configurable window placement
-----------------------------
The policy of the window layouter can be adjusted via its configuration. For
a given window label, the window's initial position and its maximized state
can be defined as follows:
! <config>
! <policy label="mupdf" maximized="yes"/>
! <policy label="nit_fb" xpos="50" ypos="50"/>
! </config>
Keyboard shortcuts
------------------
The window layouter has become able to respond to key sequences. However,
normally, the layouter is not a regular nitpicker client but receives only
those input events that refer to the window decorations. It never owns the
keyboard focus. In order to propagate global key sequences to the layouter,
nitpicker must be explicitly configured to direct key sequences initiated with
certain keys to the decorator. For example, the following nitpicker
configuration routes key sequences starting with the left windows key to the
decorator. The window manager, in turn, forwards those events to the layouter.
! <start name="nitpicker">
! ...
! <config>
! ...
! <global-key name="KEY_LEFTMETA" label="wm -> decorator" />
! ...
! </config>
! ...
! </start>
The response of the window layouter to key sequences can be expressed in the
layouter configuration as follows:
! <config>
! <press key="KEY_LEFTMETA">
! <press key="KEY_TAB" action="next_window">
! <release key="KEY_TAB">
! <release key="KEY_LEFTMETA" action="raise_window"/>
! </release>
! </press>
! <press key="KEY_LEFTSHIFT">
! <press key="KEY_TAB" action="prev_window">
! <release key="KEY_TAB">
! <release key="KEY_LEFTMETA" action="raise_window"/>
! </release>
! </press>
! </press>
! <press key="KEY_ENTER" action="toggle_fullscreen"/>
! </press>
! </config>
Each '<press>' node defines the policy when the specified 'key' is pressed.
It can be equipped with an 'action' attribute that triggers a window action.
The supported window actions are:
:next_window: Focus the next window in the focus history.
:prev_window: Focus the previous window in the focus history.
:raise_window: Bring the focused window to the front.
:toggle_fullscreen: Maximize/unmaximize the focused window.
By nesting '<press>' nodes, actions can be tied to key sequences. In the
example above, the 'next_window' action is executed only if TAB is pressed
while the left windows-key is kept pressed. Furthermore, key sequences can
contain specific release events. In the example above, the release of the left
windows key brings the focused window to front, but only if TAB was pressed
before.
Device drivers
##############
USB host-controller driver enhancements
=======================================
The _usb_drv_ component now solely uses a policy to grant other components
access to USB devices exposed by its raw interface (USB session). On the basis
of the 'label' attribute, it will choose a pre-configured device that is
identified by either the 'bus' and 'dev' or the 'vendor' and 'product'
attribute tuple. To accommodate policy decisions made at run time, the USB
driver is now able to reload its configuration on demand. The USB device
report now contains a 'bus' and a 'dev' attribute as well in order to identify
a USB device more precisely. In addition to that, there is also a generated
'label' attribute in form of 'usb-<bus>-<dev>' that may be used to form
policies while configuring the system dynamically, e.g., when using the
_usb_report_filter_ component.
USB mass-storage driver
=======================
Up to now, access to USB storage devices was provided by the USB
host-controller driver only. However, its ability to do so is limited. E.g.,
it only supports one storage device and the storage device cannot be changed
at run-time. With this release we add a USB mass-storage driver that supports
UMS bulk-only devices that use the SCSI Block Commands set (direct-access).
This is still most common for USB sticks. Devices using different command
sets, e.g SD/HC devices or some external disc drives, will not work properly
if at all. The driver uses the USB session interface to access the USB device
and provides its service as block session to its client.
This component is part of the first step providing the ability to mount and
use USB sticks dynamically when using Genode as a general purpose OS. In the
future, the _usb_drv_ component should solely be the host-controller driver
while other tasks are handled by dedicated USB driver components such as this
one.
Audio output on Linux
=====================
The audio-out driver for Linux was modernized by replacing its multi-threaded
architecture by an event-driven architecture using Genode's server API. In
addition, the playback is now driven by a timer. For now it is a periodic
timer that triggers every 11 ms which is roughly the current audio-out period.
The driver now also behaves like the other BSD-based audio-out driver, i.e.,
it always advances the play pointer. That is vital for the audio-out stack
above the driver to work properly (e.g., the mixer).
Libraries and applications
##########################
New Genode-world repository
===========================
With a growing number of users and contributors comes the desire to bring more
and more existing software to Genode. Most of such libraries and applications,
however, are outside of the scope of Genode as an OS framework. In contrast to
device drivers, protocol stacks, and low-level OS services, which we subject
to our regular automated tests, most 3rd-party software is pretty independent
from Genode. The attempt to integrate the growing pool of such diverse
software into the main repository does not scale.
For this reason, we introduce the new
[https://github.com/genodelabs/genode-world - Genode World] repository, which
is the designated place for hosting ported applications, libraries, and games.
To use it, you first need to obtain a clone of Genode:
! git clone https://github.com/genodelabs/genode.git genode
Now, clone the _genode-world.git_ repository to _genode/repos/world:_
! git clone https://github.com/genodelabs/genode-world.git genode/repos/world
By placing the _world_ repository under the _repos/_ directory, Genode's tools
will automatically incorporate the ports provided by the _world_ repository.
For building software of the _world_ repository, the build-directory
configuration _etc/build.conf_ must be extended with the following line:
! REPOSITORIES += $(GENODE_DIR)/repos/world
*Word of caution*
In contrast to the components found in the mainline Genode repository, the
components within the _world_ repository are not subjected to the regular
quality-assurance measures of Genode Labs. Hence, problems are to be expected.
If you encounter bugs, build problems, or stability issues, please report them
to the [https://github.com/genodelabs/genode-world/issues - issue tracker] or
the [https://genode.org/community/mailing-lists - mailing list].
Updated 3rd-party software
==========================
The following 3rd-party code packages of the _ports_ and _libports_
repositories have been ported or updated:
* Lynx 2.8.8rel.2 (noux package)
* OpenSSH 7.1p1 (noux package)
* tar-1.27 (noux package)
* libssh 0.7.2
* Lighttpd 1.4.38
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Within the last months, the initialization code of our custom kernel got
re-arranged to simplify the addition of new architectures, e.g., the RISC-V
port (Section [New support for the RISC-V CPU architecture]) while also making
its implementation leaner. A positive side effect of this work was the
generalization of multi-processor and L2-cache support for ARM's Cortex-A9
CPUs. For instance, the Wandboard (Freescale i.MX6 SoC) is now driven with all
four cores, and its memory can be accessed with full speed.
Besides those feature additions, we fixed an extremely rare and tricky race
condition in the implementation of the kernel-protected capabilities,
introduced in release 15.05. A capability's lifetime within a component is
tracked by a reference-counting like mechanism that is under control of the
component itself. When the kernel transfered a capability to a component, and
the very same capability was deleted within the component simultaneously, the
received capability was marked as invalid, which led to diverse, sporadic
faults. This deficit in the capabilities reference-counting is solved with the
current release.
Muen separation kernel
======================
Build integration
-----------------
Building Genode scenarios running on top of the
[https://muen.sk - Muen separation kernel] has been greatly simplified by
properly integrating the Muen system build process into the Genode build system.
As described in the
[https://genode.org/documentation/release-notes/15.08#Genode_on_top_of_the_Muen_Separation_Kernel - 15.08 release notes],
the architecture with Muen is different since the entire hw_x86_64_muen Genode
system runs as a guest VM on top of the separation kernel. This means that the
Genode base-hw image must itself be packaged into the final Muen system image
as an additional step after the Genode system build.
The packaging process of a Muen system image is performed by the new
_image/muen_ run-tool plugin, which processes the following RUN_OPT parameters.
:--image-muen-external-build:
Muen system is built automatically or externally
:--image-muen-system:
Muen system policy
:--image-muen-components:
Muen system components required for the given system policy
:--image-muen-hardware:
Muen target hardware platform
:--image-muen-gnat-path:
Path to GNAT toolchain
:--image-muen-spark-path:
Path to SPARK toolchain
The options are automatically added to the _etc/build.conf_ file for the
hw_x86_64_muen base-hw platform. The
[https://genode.org/documentation/platforms/muen - documentation] has been
updated to reflect the new, simplified build process.
A port file was added to facilitate the download of the Muen sources v0.7 and
to check the required dependencies.
Using the new _image/muen_ script in combination with iPXE allows to run the
Genode test suite via the autopilot tool.
MSI support
-----------
Muen employs Intel VT-d interrupt remapping (IR) besides DMA remapping for
secure device assignment. As a consequence, PCI devices using Message Signaled
Interrupts (MSI) must be programmed to trigger requests in remappable format
(see Intel VT-d specification, Section 5.1.2.2 for further details).
To enable the use of MSIs with the base-hw kernel, a platform-specific
function has been introduced that returns the necessary MSI parameters for a
given PCI device. If either the platform or the specific device does not
support MSI, the function returns false.
On hw_x86_64_muen, the function consults the Muen subject info page to supply
the appropriate information to the IRQ session. This allows Genode device
drivers to transparently use MSIs for passed-through PCI devices.
seL4 version 2.1
================
By the end of 2015, the [https://sel4.systems/ - seL4 kernel] version 2.0 was
published. With the current release, we update Genode's preliminary support
for this kernel from the experimental branch of one year ago to the master
branch of version 2.1. Note that this line of work is still considered as an
exploration. As of now, there is still a way to go until we can leverage seL4
as a fully featured base platform. Under the hood of Genode, the transition to
the version 2.1 master branch had the following implications.
In contrast to the experimental branch, the seL4 master branch has no way to
manually define the allocation of kernel objects within untyped memory ranges.
Instead, the kernel maintains a built-in allocation policy. This policy rules
out the deallocation of once-used parts of untyped memory. The only way to
reuse memory is to revoke the entire untyped memory range. Consequently, we
cannot share a large untyped memory range for kernel objects of different
protection domains. In order to reuse memory at a reasonably fine granularity,
we need to split the initial untyped memory ranges into small chunks that can
be individually revoked. Those chunks are called "untyped pages". An untyped
page is a 4 KiB untyped memory region.
The bootstrapping of core has to employ a two-stage allocation approach now.
For creating the initial kernel objects for core, which remain static during
the entire lifetime of the system, kernel objects are created directly out of
the initial untyped memory regions as reported by the kernel. The so-called
"initial untyped pool" keeps track of the consumption of those untyped memory
ranges by mimicking the kernel's internal allocation policy. Kernel objects
created this way can be of any size. For example the CNode, which is used to
store page-frame capabilities is 16 MiB in size. Also, core's CSpace uses a
relatively large CNode.
After the initial setup phase, all remaining untyped memory is turned into
untyped pages. From this point on, newly created kernel objects cannot exceed
4 KiB in size because one kernel object cannot span multiple untyped memory
regions. The capability selectors for untyped pages are organized similarly to
those of page-frame capabilities. There is a new 2nd-level CNode
(UNTYPED_CORE_CNODE) that is dimensioned according to the maximum amount of
physical memory (1M entries, each entry representing 4 KiB). The CNode is
organized such that an index into the CNode directly corresponds to the
physical frame number of the underlying memory. This way, we can easily
determine an untyped page selector for any physical addresses, i.e., for
revoking the kernel objects allocated at a specific physical page. The
downside is the need for another 16 MiB chunk of meta data. Also, we need to
keep in mind that this approach won't scale to 64-bit systems. We will
eventually need to replace the PHYS_CORE_CNODE and UNTYPED_CORE_CNODE by CNode
hierarchies to model a sparsely populated CNode. The following figure
illustrates the layout of core's capability space.
[image sel4_core_cspace_master]
Organization of core's capability space on seL4
For each protection domain, core maintains a so-called VM CSpace that holds
capability selectors for page frames and page tables. The size constraint of
kernel objects has the immediate implication that the VM CSpaces of protection
domains must be organized via several levels of CNodes. I.e., as the top-level
CNode of core has a size of 2^12, the remaining 20 PD-specific CSpace address
bits are organized as a 2nd-level 2^4 padding CNode, a 3rd-level 2^8 CNode,
and several 4th-level 2^8 leaf CNodes. The latter contain the actual selectors
for the page tables and page-table entries of the respective PD.
As another slight difference from the experimental branch, the master branch
requires the explicit assignment of page directories to an ASID pool.
Functionality-wise the update to version 2.1 brings no changes. The
preliminary support is still limited to Genode's most fundamental mechanisms
like the bootstrapping, the creation of protection domains, the execution of
threads, and inter-component communication. User-level device drivers are not
supported yet. Such functional improvements are scheduled for Genode 16.08.
Linux
=====
We started to experience crashes of our dynamic linker (ldso) when using
Genode's _base-linux_ platform on recent Linux kernels. Ldso is primarily a
shared object, which is linked to dynamic binaries. But ldso is also an
executable, which, once started loads the dynamically-linked binary along with
all shared libraries required by the binary. Up to now, ldso had to be loaded
at a link address defined at compilation time, which we enforced through
linker-script magic. Unfortunately, this does not work any longer on recent
Linux versions. The kernel notices that ldso is a shared object and loads it
at an arbitrary (randomized) address, which ultimately results in a
segmentation fault during ldso initialization. We found a fix for this issue
by marking ldso as an executable in the ELF header. But since ldso is linked
to all dynamic binaries (it contains Genode's base libraries) the GNU linker
then refused to link because ldso was not marked as a shared object.
Therefore, we decided to implement true self relocation within ldso. This
feature only works on Genode's base-linux platform as it requires some
symbol-address magic.

1315
doc/release_notes/16-05.txt Normal file
View File

File diff suppressed because it is too large Load Diff

1126
doc/release_notes/16-08.txt Normal file
View File

File diff suppressed because it is too large Load Diff

729
doc/release_notes/16-11.txt Normal file
View File

@@ -0,0 +1,729 @@
===============================================
Release notes for the Genode OS Framework 16.11
===============================================
Genode Labs
In contrast to most parts of the framework, the fundamental low-level
protocols, which define the interaction between parent and child components
have remained unchanged since the very first Genode version. From this
interplay, the entire architecture follows. That said, certain initial design
choices were not perfect. They partially resulted from limitations of the
kernels we used during Genode's early years and from our pre-occupation with a
certain style of programming. Over the years, the drawbacks inherent in our
original design became more and more clear and we drafted rough plans to
overcome them. However, reworking the fundamental protocols of a system that
already accommodates hundreds of component implementations cannot be taken
lightly. Because of this discomfort, we repeatedly deferred the topic -
until now. With the rapidly growing workloads carried by Genode, we
deliberately decided to address long-standing deficiencies rather than adding
the features we originally planned according to the
[https://genode.org/about/road-map - road map].
Section [Asynchronous parent-child interactions] presents the reworking of
Genode's component interplay at the lowest level. With this change in place,
we feel much more comfortable to scale up our workloads in the upcoming
releases.
Functionality-wise, the most prominent topic of the current release is the
vastly improved NIC-routing component. Since we introduced the first version
of the NIC router in the previous release, we took an iterative approach to
shape the component according to its most prominent use cases. Section
[Further improved virtual networking] summarizes the changes and the
motivation behind them.
Even though we added support for seL4 in the previous release, the NOVA
hypervisor is still our go-to kernel for x86-based hardware because of its
feature set. For this reason, we continuously improve this kernel and the
NOVA-specific components like VirtualBox. Section [NOVA hypervisor] covers
the introduction of an asynchronous map operation to NOVA.
Further topics of the current release range from added smart-card support,
over a new timeout API, to a VFS-based time-based password generator. With
respect to the road map, we postponed most topics originally planned. In
particular, we intended to enable the use of Genode on top of Xen by following
the footsteps of the existing Muen support - using our custom
base-hw kernel within a Xen DomU domain. However, before proceeding this
route, we decided to modernize the kernel design, in particular with respect
to bootstrapping and address-space management. Some parts of this line of work
are already present in the current release, for example the unification of the
boot-module handling as explained in Section
[Unified handling of boot modules].
Asynchronous parent-child interactions
######################################
When Genode was born in 2006, the L4 microkernels of the time universally
lacked an asynchronous inter-process-communication (IPC) mechanism.
Consequently, we designed the first version of Genode with the presumption
that components had to interact solely synchronously. To us, this seemed to be
the "right" way because the synchronous low-footprint IPC was presumably the
key for L4's good performance. It felt natural to leverage this benefit to the
maximum extent possible.
To illustrate the implications of this line of thinking for Genode, let's take
a look at a simple scenario where a parent component hosts two children and one
child provides a service to the other child.
[image simple_scenario]
During the creation of a session, the kernel's IPC mechanism serves three
purposes. First, it is used to communicate information between different
protection domains, in this case the parent, the client, and the server.
Second, it implicitly dictates the flow of control between the involved
parties because the caller blocks until the callee replies to the IPC call.
Third, the IPC is the mechanism to delegate authority (like the authority to
access the server's session object) between protection domains. The latter is
realized with the kernel's ability to carry capabilities as IPC message
payload. If this sounds a bit too abstract, please consider reviewing Section
3.1. "Capability-based security" of the
[https://genode.org/documentation/genode-foundations-16-05.pdf - Genode Foundations].
Using solely a synchronous IPC mechanism, the sequence of establishing a
session in the given scenario is as follows. In the context of Genode,
we usually refer to synchronous IPC as RPC (remote procedure call).
[image sync_session_seq]
The sequence looks straightforward:
# The client issues an RPC call to its parent, requesting a session for a
service of the given type while also passing a number of session-construction
arguments along with the request.
# Given the service name as provided with the session request, the parent
determines the server to ask for a new session. It requests a session
on behalf of the client by performing an RPC call to the server's prior
registered "root" capability. This capability refers to an interface for
creating and closing sessions.
# The server responds to the invocation of its root interface by creating
a new session object along with a session capability.
Whereas the session object is local to the server, the corresponding
session capability can be passed (delegated) to other components.
Each component in possession of the session capability is able to interact
with the server's corresponding session object via RPC calls.
The server returns the session capability to the parent as the result of the
parent's RPC call.
# The parent forwards the session capability to the client as the result of
the client's original RPC call.
Even though the simplicity of this protocol seems nice, it has inherent
limitations:
First, as the parent performs a synchronous RPC call to the server on behalf
of the client, it must trust the server to eventually respond to the RPC call.
If the server doesn't, the parent may block forever. In contrast to the client
that actually uses the service and thereby relies on the liveliness of the
server, the parent should not need to trust the server to be responsive. To
deal with the risk of an unresponsive server, Genode's existing runtime
environments (like the init component), maintain a dedicated thread for each
child. The session requests originating from a child are handled by the
corresponding parent-local child thread. In the worst case - if the server
fails to respond - only a single child thread stays blocked but the other
parts of the runtime environment remain unaffected. Consequently, runtime
environments have to be multi-threaded components. This, in turn, comes at the
cost of added complexity, in particular the need for error-prone inter-thread
synchronization.
Second, the approach keeps the parent's state implicitly stored in the stacks
of the parent's threads. This becomes a problem in dynamic runtime
environments that need to kill subsystems at arbitrary times. E.g., imagine
the situation where the client component is to be destroyed while the parent's
call to the server's root interface is still pending. The safe destruction of
the child - including its associated parent-local child thread - requires the
parent to abort the RPC call, which is a complex and - again - error-prone
operation.
Third, even though not inherent to synchronous RPC, Genode's original design
facilitated the use of a session capability as argument for requesting the
parent to close a specific session. However, the use of capabilities as
re-identifiable tokens is not well supported by most kernels, including seL4
([https://sel4.systems/pipermail/devel/2014-November/000114.html - discussion]
on the seL4 mailing list).
Asynchronous communication throughout Genode
--------------------------------------------
In 2008, we acknowledged the sole reliance on synchronous RPC as too limiting
and introduced an
[https://genode.org/documentation/release-notes/8.11#Asynchronous_notifications - API for asynchronous notifications].
On the traditional L4 kernels, we implemented the API by using Genode's
core component as a proxy for signal delivery. The use of asynchronous
notifications soon became natural and wide-spread throughout Genode. Today,
most session interfaces combine three forms of inter-component communication,
namely synchronous RPC calls, asynchronous notifications, and shared memory.
The new Genode API introduced in
[https://genode.org/documentation/release-notes/16.05#The_great_API_renovation - version 16.05]
further cultivated the modeling of Genode components as single-threaded state
machines instead of multi-threaded programs.
Still, until now, the most fundamental mechanism of Genode - the protocol
between parent and child components - has remained synchronous. The reasons
are twofold. First, our workaround for realizing runtime environments in a
multi-threaded way worked too well. So we were not constantly bothered by this
design problem. Second and more importantly, redesigning the fundamental
mechanism of the framework while not breaking the more than 300 existing
components is quite scary. But in anticipation of the rapidly scaling
workloads imposed on Genode, we had to take on the problem sooner or later.
We figured that now - with the modernized framework API in place - it's the
right time. From redesigning the interplay of parent and child components, we
will become able to create single-threaded runtime environments that behave
completely deterministically while consuming less resources than
multi-threaded programs. By the explicit enumeration of possible states, we
greatly ease the validation/evaluation of such crucial components.
New session-creation procedure
------------------------------
Following the asynchronous approach, the sequence of creating a session now
looks as follows:
[image async_session_seq]
The dotted lines are asynchronous notifications, which have fire-and-forget
semantics. A component that triggers a signal does not block.
The following points are worth noting:
* Sessions are identified via IDs, which are plain numbers as opposed to
capabilities. The IDs as seen by the client and server belong to different
ID name spaces.
IDs of sessions requested by the client are allocated by the client. IDs
of sessions requested at the server are allocated by the parent.
* The parent does no longer need to perform RPC calls to any of its children.
Hence, the need for multiple threads in runtime environments disappears.
* Each activation of the parent merely applies a state change of the session's
meta data structures maintained at the parent, which capture the entire
state of session requests. There is no hidden state stored on the parent's
stack.
* The information about pending session requests is communicated from the
parent to the server via a ROM session. At startup, the server requests
a ROM session for the ROM module "session_requests" from its parent. The
parent implements this ROM session locally. Since ROM sessions support
versions, the parent can post version updates of the "session_requests"
ROM with the regular mechanisms already present in Genode.
* The involved parties can potentially run in parallel.
Outcome and current state
-------------------------
Intuitively, the sequence of steps required to establish a session has
become more complicated. However, for the users of the framework, the entire
procedure is completely transparent. With a few tricks, we were actually able
to implement this fundamental change while keeping almost all existing
components untouched. One trick is the introduction of a server-local proxy
mechanism, which translates the requests obtained from the "session_requests"
ROM to component-local RPC calls on the server's root interface. So from the
perspective of an existing server component, a session request still looks
like a synchronous RPC request from the outside. Of course, the proxy is meant
as an intermediate solution until we have crafted a convenient front-end API
for the asynchronous mode of operation.
Even though the biggest share of components remains unaffected by the change,
this is not true for all components. In particular, runtime environments had
to be reworked, in some cases quite fundamentally. These include core, init,
noux, the loader, GDB monitor, launcher, CLI monitor, and the platform driver.
The change does not only affect the interplay between components but also
required a reconsideration of the child-creation procedure.
Besides the architectural improvement, this line of work had two welcome
effects.
First, in contrast to the original design, which relied on capabilities as
re-identifiable tokens, the new version greatly alleviates the need for
re-identifying capabilities on seL4. So we are able to eliminate a
long-standing problem with Genode on this kernel.
Second, the work called for new data structures for the safe interaction with
ID spaces (_base/id_space.h_) and object registries (_base/registry.h_). Those
data structures will possibly be useful in a lot of places that currently use
plain (and fairly unsafe) AVL trees or lists.
At the API level, the change is almost transparent to regular components,
except for two details. The upgrading of session quota is no longer
possible by a mere RPC call to the parent. Instead, 'Connection' objects
received a new 'upgrade_ram' method that must be used instead. Speaking
of 'Connection' objects, we had to remove the (fairly obscure) 'KEEP_OPEN'
feature, which is conceptually incompatible with the new design.
Further improved virtual networking
###################################
The
[https://genode.org/documentation/release-notes/16.08#Virtual_networking_and_support_for_TOR - previous release]
introduced the NIC router - a component that individually routes IP
packets between multiple NIC sessions, translates between different IP
subnets, and also supports port forwarding and NAT. For the first version of
the NIC router, we focused on the technical realization. Now, besides
some optimization and restructuring, we took the chance to polish the
configuration interface of the component. The goal was to make the interface
more intuitive and reduce pitfalls to a minimum. Roughly speaking, the
handling of the NIC router became more tailored to its/our typical use cases.
Let's create a practical setup to explain the changes in detail. Assume that
there are two virtual subnets 192.168.1.0/24 and 192.168.2.0/24 within our
Genode system. They connect as Virtnet A and B to the router. The standard
gateway of the virtual networks is the NIC router with IP 192.168.*.1 . The
router's uplink, on the other hand, is connected to the NIC driver. It
interfaces the machine with our real-world home network 10.0.2.0/24. The home
network is connected to the internet through its standard gateway 10.0.2.1.
[image nic_router_basic]
The basic router configuration for this setup without any routing rules would
be as follows:
! <policy label_prefix="virtnet_a" domain="virtnet_a" />
! <policy label_prefix="virtnet_b" domain="virtnet_b" />
!
! <domain name="uplink" interface="10.0.2.55/24" gateway="10.0.2.1" />
! <domain name="virtnet_a" interface="192.168.1.1/24" />
! <domain name="virtnet_b" interface="192.168.2.1/24" />
The first thing to notice is the changed usage of the policy tag. Previously,
the policy label - normally solely designated to correlate sessions with
configuration domains - was misused also as unique peer identifier in the
routing rules. This approach disregarded advanced label-matching techniques
such as the 'label_prefix' used above. Now, the whole NIC-router-specific
enhancement of the policy tag moved to the new '<domain>' tag, leaving the
policy tag only with its original purpose to select policies. Note that even
if this modification gives the impression, the router is not yet capable of
handling multiple NIC sessions at one domain at a time.
In the domain tag, the 'interface' attribute replaces the old policy attribute
named 'src'. That means, it tells the router which IP identity to use when
talking as itself to the domain. But in addition to that, the 'interface'
attribute also defines which subnet this identity and the domain belong to.
This reflects a basic decision we made during the reworking process: The new
NIC router is aware of subnets. Sessions of the same subnet have the same
configuration domain. We came to this conclusion as it solves some fundamental
problems with the old version. First, the equivalence of domain and subnet
enables us to link a default gateway to a subnet by adding the 'gateway'
attribute to the domain tag. In our example, this is done in the uplink
domain. The 'gateway' attribute is optional for a domain and replaces the
former 'via' attributes of the different routing rules. It is more efficient
and natural to have this value set only once at the corresponding subnet than
having it scattered all over the routing rules of the remote domains as done
before. If a domain has no default gateway, it drops all packets with a
foreign recipient.
The second advantage of a domain being equivalent to a subnet is that handling
ARP broadcasts becomes easy. It can be excluded that such ARP broadcasts
concern sessions outside the source domain anymore. And as sessions in the
same domain are not distinguishable to the routing, the broadcast can be sent
to all of them without breaking any rules.
Now, let's enhance our example by some routing rules. One pretty complicated
thing to do with the old NIC router was port forwarding. You had to combine
different routing rules, explicitly enable the back routing at the remote
side, and take care that NAT was applied - a lot of opportunities for
mistakes. With the new version, it became easier. Let's assume we have an HTTP
server in Virtnet A and an NTP server in Virtnet B. We want the NIC router to
act as proxy for their services in our home network.
[image nic_router_servers]
In order to achieve this, the uplink domain must be enhanced by two rules:
! <policy label_prefix="virtnet_a" domain="virtnet_a" />
! <policy label_prefix="virtnet_b" domain="virtnet_b" />
!
! <domain name="uplink" interface="10.0.2.55/24" gateway="10.0.2.1" />
! <tcp-forward port="443" domain="virtnet_a" to="192.168.1.2" />
! <udp-forward port="123" domain="virtnet_b" to="192.168.2.2" />
! </domain>
!
! <domain name="virtnet_a" interface="192.168.1.1/24" />
! <domain name="virtnet_b" interface="192.168.2.1/24" />
The TCP forwarding rule for port 443 (HTTP+TLS/SSL) redirects to IP address
192.168.1.2 in Virtnet A and the UDP forwarding rule for port 123 (NTP)
redirects to IP address 192.168.2.2 in Virtnet B. The Virtnet domains remain
empty as the router keeps track of the redirected transfers and routes back
reply packets automatically. Also automatically, the router applies NAT for the
server as it is in the nature of port forwarding.
Next, we add some clients to Virtnet B that like to talk to our home network
and the internet. We want them to be hidden via NAT when they do so. For
internet communication, they shall furthermore be limited to HTTP+TLS/SSL and
IMAP+TLS/SSL.
[image nic_router_client]
This is what the router configuration looks now:
! <policy label_prefix="virtnet_a" domain="virtnet_a" />
! <policy label_prefix="virtnet_b" domain="virtnet_b" />
!
! <domain name="uplink" interface="10.0.2.55/24" gateway="10.0.2.1" />
! <tcp-forward port="443" domain="virtnet_a" to="192.168.1.2" />
! <udp-forward port="123" domain="virtnet_b" to="192.168.2.2" />
! <nat domain="virtnet_b" tcp-ports="1000" udp-ports="1000">
! </domain>
!
! <domain name="virtnet_a" interface="192.168.1.1/24" />
! <domain name="virtnet_b" interface="192.168.2.1/24" >
! <tcp dst="10.0.2.0/24"> <permit-any domain="uplink" /> </tcp>
! <udp dst="10.0.2.0/24"> <permit-any domain="uplink" /> </udp>
! <tcp dst="0.0.0.0/0">
! <permit port="443" domain="uplink" />
! <permit port="993" domain="uplink" />
! </tcp>
! </domain>
There are several new tag types. One of them is the NAT configuration for
Virtnet B in the uplink domain. In contrast to the former NIC-router version
where NAT settings were part of the source domain, NAT is now configured in
the target domain with a sub-tag for each source. This has the advantage
of supporting heterogeneous NAT configurations for a packet source depending
on which domain it talks to. Besides, it is more intuitive to read. Apart from
that, the NAT settings haven't changed.
Furthermore, there are the new TCP and UDP tags in the Virtnet-B domain. The
first two of them have a 'permit-any' sub-tag. With this combination, we open
all ports to IP addresses of the 10.0.2.0/24 subnet, our home network, and
route them to the uplink domain. TCP packets that don't match these first two
rules may fall back to the third. This TCP rule doesn't have all ports opened
but only 443 (HTTP+TLS/SSL) and 993 (IMAP+TLS/SSL). Both ports are again bound
to the uplink domain. As the IP filter 0.0.0.0/0 of the surrounding rule isn't
restrictive, we now also route packets to a foreign destination. The NIC
router redirects such packets to the default gateway of our home network.
Compared to the old router version where IP and UDP/TCP routing had to be
combined for this purpose, the new TCP and UDP rules with their
port-permission sub-rules have some notable advantages. Like port-forwarding
rules, TCP and UDP rules always imply link-state tracking in order to route
back reply packets automatically. This can be seen also in our example as no
further routing rules had to be added to the uplink domain. This aspect is
clear from the outermost rule and not dependent on sub-rules anymore.
Furthermore, the strict separation of UDP and TCP routing prevents
configuration faults and increases readability. Last but not least, the
'permit-any' rule allows something new. Opening all ports for an address range
was previously only possible without link-state tracking as it could be
expressed only on the IP level.
At this point, we have thoroughly discussed the layer-3 routing abilities of
the new NIC router and our focus has indeed moved more into this direction.
Even though IP routing is still available, we found that it should be more
clearly separated from the rest. To illustrate this feature, we enhance our
example again. We want the Virtnets to be allowed to communicate to each other
without any restrictions. For that purpose, we add two more rules to the
router configuration:
! <policy label_prefix="virtnet_a" domain="virtnet_a" />
! <policy label_prefix="virtnet_b" domain="virtnet_b" />
!
! <domain name="uplink" interface="10.0.2.55/24" gateway="10.0.2.1" />
! <tcp-forward port="443" domain="virtnet_a" to="192.168.1.2" />
! <udp-forward port="123" domain="virtnet_b" to="192.168.2.2" />
! <nat domain="virtnet_b" tcp-ports="1000" udp-ports="1000">
! </domain>
!
! <domain name="virtnet_a" interface="192.168.1.1/24" />
! <ip dst="192.168.2.0/24" domain="virtnet_b"/>
! </domain>
!
! <domain name="virtnet_b" interface="192.168.2.1/24" >
! <tcp dst="10.0.2.0/24"> <permit-any domain="uplink" /> </tcp>
! <udp dst="10.0.2.0/24"> <permit-any domain="uplink" /> </udp>
! <tcp dst="0.0.0.0/0">
! <permit port="443" domain="uplink" />
! <permit port="993" domain="uplink" />
! </tcp>
! <ip dst="192.168.1.0/24" domain="virtnet_a"/>
! </domain>
As you can see, each of the new IP rules in the Virtnet domains match the
addresses of the opposite subnet and route to the corresponding domain. As
mentioned, the new IP rules and UDP/TCP rules are not combined anymore to
clearly distinguish IP routing from layer-3 routing because this decision has
far-reaching effects. First, in contrast to UDP and TCP routing, IP routing is
stateless. Thus, for each IP routing rule one has to be sure to have a
back-routing rule at the remote domain or else bidirectional communication
won't happen. And second, NAT does not apply to IP-routed packets. So, if
you're not aware of such packets, you may unintentionally reveal information
about a private network.
For more details on the new NIC router, you may refer to the comprehensive
documentation in the _repos/os/src/server/nic_router/README_ file and the
basic NIC-router test at _libports/run/nic_router.run_ .
Base framework
##############
Improved RPC mechanism
======================
Since we introduced Genode's current API for synchronous RPCs in
[https://genode.org/documentation/release-notes/11.05#New_API_for_type-safe_inter-process_communication - version 11.05],
inter-component communication within Genode has become almost a child's play.
The RPC framework leverages the C++ type system and templates to a great
effect. In contrast to the traditional use of IDL compilers, the interaction
with RPC objects provided by other components is robust and natural because
no language boundaries need to be crossed.
Still, a few differences between RPC calls and regular function calls remain.
In particular, there exist a few restrictions with regard to the types of
RPC function arguments. Those types did not just need to be POD (plain old
data) types but they had to be default-constructible, too. Whereas the former
restriction still applies (non-POD objects that include references or
vtables cannot be used as arguments), the latter limitation has been lifted
now. Generally, non-default-constructible types are a way to attain
simpler code because the special case of an "invalid" object does not need
to be considered. I.e., values of such types can be kept as constants as
opposed to variables. If an object exists (as equivalent to successful
instantiation), it is valid. With the improved RPC mechanism, the RPC
framework does no longer stay in the way in this respect.
Thanks to Edgard Schmidt for this welcome contribution!
Unification and tightening of session labels
============================================
In Genode, each session requested by a client component is labeled according
to the components that intermediate the session request. The client can
optionally specify a label of choice along with the session request. Its
parent prefixes the client-provided label by a label of its own. If the
session request is further passed to the parent's parent, the grandparent
prepends its own label. This works recursively. Consequently, the final label
as seen by the server is the product of the labeling policies of all
components on the route of the session request.
The label is used for two purposes. First, the server uses the label as
a key for a server-side policy selection. E.g., depending on the session label
received by the disk-partition server, the server decides which partition to
hand out to the client. Second, the label is used by intermediate components
to take session-routing decisions. E.g., based on the label of a file-system
session request, a parent component may route the request to one of several
file-system servers.
Originally, Genode did not impose a specific way of how labels are formed.
It was up to each intermediate component to filter the label of a session
request in any way desired. However, in practice, this freedom remained unused
and the very simple successive prefixing of labels prevails in all our use
cases. Each intermediate node concatenates its own label in front of the label
supplied by the originator of the session request. The different parts of the
label are separated with the character sequence '" -> "'. Some corner cases
were handles specially for aesthetic reasons. For example, if a client
provided no label, the parent would skip the pending separator. That said,
since each intermediate component had to provide the labeling policy, not all
components were consistent in these respects. Since we found no use for
arbitrary labeling policies, we decided to make the only prominent way of
session labeling mandatory for all intermediate components. We thereby removed
the aesthetically motivated corner cases and possible ambiguities. I.e., with
the original policy, it was not possible to distinguish a unlabeled session
requested by a client from a labeled session requested by the client's parent.
As a consequence, the stricter labeling must now be considered wherever
a precise label was specified as a key for a session route or a server-side
policy selection. The simplest way to adapt those cases is to use a
'label_prefix' instead of the 'label' attribute. Alternatively, the
'label' attribute may used by appending '" -> "' (note the whitespace).
Transition to new framework API
===============================
Since we fundamentally revised Genode's API in
[https://genode.org/documentation/release-notes/16.05#The_great_API_renovation - version 16.05],
we gradually adapt our existing components. Given that Genode comes with
over 300 components, this is no small feat. But with 30 percent of the
components converted, we already made substantial progress.
In some respects, the conversion is actually nearly complete. In particular,
the move away from format-string-based text output to our new type-safe output
facility has been applied to almost all components now. The former 'PDBG'
macro that is quite useful for temporary debug messages has been replaced with
a new version that must be manually included via the _base/debug.h_ header
file. Like the regular log functions, the new PDBG facility uses the type-safe
text-output facility.
Minor API adjustments
---------------------
While applying Genode's new API, we refined the API in the following respects:
We added a dedicated 'String' constructor overload to better accommodate
string literals. This overload covers the common case for initializing a
string from a literal without employing the 'Output' mechanism. This way, such
strings can by constructed without calling virtual functions, which in turn
makes the 'String' usable during the self-relocation phase of the dynamic
linker.
Up till now, several Genode components still rely on the use of 'snprintf'
whenever strings must be assembled out of smaller pieces. As we like to shun
format strings from Genode altogether, we needed an alternative mechanism.
Since we introduced the new type-safe text-output facilities in Genode 16.05,
there is an obvious solution: Let the 'String' constructor accept an arbitrary
list of arguments, which are turned into their respective textual
representation and appear concatenated in the resulting string. Consequently,
strings can be assembled with the same flexibility as log output. For the
construction of 'String' objects from character buffers of a known size, the
'Cstring' utility can be used, which takes a 'char const *' and an optional
length as arguments.
Several low-level types received support for the new output facilities, e.g.,
'Xml_node' or the network-related headers in _os/net/_.
In anticipation of the forthcoming package-management infrastructure, we try
to unify Genode's executable binaries across kernels and architectures
wherever reasonable. Of course, the latter is not possible with respect to the
used instructions. But unifying symbol information is deemed worthwhile. For
this reason, we changed the 'Genode::size_t' type to be always defined as an
'unsigned' 'long'. This is in contrast to GCC's built-in '__SIZE_TYPE__',
which is defined as 'unsigned int' on 32-bit architectures but 'unsigned long'
on 64-bit architectures.
OS-level infrastructure and device drivers
##########################################
New timeout-handing API
=======================
The new timeout API offers tools for easily multiplexing a single time
source among different timeouts. In general, the time source can be
implemented individually but we expect that the most prominent use case will
be the multiplexing of timer sessions. Thus, the timeout library also provides
a convenience tool for this use case. A library-usage example can be found
under _os/src/test/timeout_. If you're interested in implementing
your own time source, you can find an example at _os/include/os/timer.h_ .
Support for smart cards
=======================
We ported the [https://pcsclite.alioth.debian.org/pcsclite.html - PC/SC Lite]
library to Genode, which provides a commonly used API for communicating with
smart cards. It supports USB smart card readers, using the
[https://pcsclite.alioth.debian.org/ccid.html - CCID] library as driver.
The CCID driver itself requires [https://libusb.info - libusb] to access the
USB device.
Vanilla PC/SC Lite is structured as a client-server architecture, consisting
of the 'pcscd' daemon, which runs on a privileged user account and manages all
card reader devices, and one or more non-privileged client applications, which
communicate with pcscd to access the card readers. On Genode, pcscd's role as
privileged device manager is not really needed, since the devices can also be
managed using Genode's configuration mechanisms. For this reason, we merged
the part of pcscd which implements the API with the pcsc-lite client library.
In the current state, a Genode application using PC/SC Lite can access a single
card reader device, which is selected using its USB product ID and vendor ID in
the application's configuration and in the policy of the USB driver.
More configuration details can be found in the README files of the PC/SC Lite,
CCID, and libusb libraries in the libports repository and in the accompanying
_smartcard.run_ script.
Libraries and applications
##########################
Time-based password generation
==============================
A time-based one-time password authentication client that adheres to the
Google Authenticator standard has been introduced into the
[https://github.com/genodelabs/genode-world - world repository].
Single use, time-based passwords are commonly used as an additional
authentication step for web-based services. In this scheme, a user generates
and presents a six digit passcode to a service generated using a shared secret
and a timestamp. This short passcode length makes manual entry convenient so
that the shared secret may be stored on a separate device than the service
client, such as a smartphone, layering the security properties of both
devices.
The 'gtotp' VFS plugin provides these passcodes by embedding the generator as
a special file in the file-system layer of a component. This approach provides
readily available passcodes for programmatic and manual use without enlarging
the code base to encompass a GUI, command-line, or networked interface.
At the time of this release, the common use case is to manually retrieve codes
for clients running in VirtualBox by reading special files with an isolated
instance of the Noux runtime. Storing the shared secret on the same device
contradicts the recommendations of the standard but the trade-off is that the
software stack required to host the shared secret is significantly smaller
than that found on a mobile device.
Random number generator testing
===============================
No random number generator can be proved to be good, but empirical statistical
tests can prove that some are bad. A port of the TestU01 RNG test suite is
provided in the world repository. The TestU01 batteries give independent
assurance of the fitness of Genode's CPU jitter based RNG and are available
for testing future physical and non-phyical RNGs.
VirtualBox on top on the NOVA hypervisor
########################################
Both VirtualBox-based virtual machine monitors on Genode got updated to the
latest revision as provided by Oracle, namely 4.3.40 and 5.1.10 - mainly to
stay close to the upstream versions.
Platforms
#########
Unified handling of boot modules
================================
Until now, the way of passing boot modules from the boot procedure to the core
component, which core provides as ROM modules, varied from platform to
platform. Either we used a multiboot-compliant bootloader that accepts
multiple modules, or the platform provided some specific way of linking binary
modules together with the kernel, e.g., the Elfweaver tool of OKL4.
By unifying the boot-module handover, we further reduce platform specific core
code. Thereby, maintenance costs are decreased, and code analysis becomes
easier. With this new solution, when issuing to build the core component:
! make core
within the build system, only a core library gets built. Not until all
binaries needed by a run-script are available, a final image is linked
together using the core library and all additional binaries. The core
component now can access its ROM modules directly via addresses contained in
its binary. As a side effect of this change, there is no core binary in the
'bin' or 'core' directory of the corresponding build directory available
anymore. Instead, you will find the core binary with no ROM modules, but
including debug information under 'var/run/*.core' within your build
directory. The concrete name depends on the name of the run-script.
The new approach is used on all platforms except Linux where the ROM modules
still need to be accessed via the file-system.
NOVA hypervisor
===============
We extended the kernel to support the asynchronous delegation of kernel
resources. Up to now, resources could only be delegated during RPC or during
the initial protection-domain construction. With this extension, the
construction and setup of new protection domains, threads, and especially
virtual CPUs for the VirtualBox VMM became more straightforward and several
quirks inside the 'core' component could be dropped. The added kernel syscall
expects the NOVA-kernel capabilities of the source and target protection
domains, which effectively renders the operation solely available to 'core' -
as only holder of the NOVA protection domain capabilities.
Additionally, we changed the CPU ID enumeration in Genode/NOVA to a
predictable order. The lower CPU IDs used via the Genode 'Cpu_session'
interface now correspond to the first hyper-thread of all physical CPU cores.
For example, on a quad-core machine with hyper-threading enabled Genode's CPU
IDs 0-3 refer to the first hyper-threads of all physical cores and IDs 4-7 to
the second hyper-threads.

0
doc/release_notes-17-02.txt → doc/release_notes/17-02.txt
View File

1033
doc/release_notes/17-05.txt Normal file
View File

File diff suppressed because it is too large Load Diff

651
doc/release_notes/17-08.txt Normal file
View File

@@ -0,0 +1,651 @@
===============================================
Release notes for the Genode OS Framework 17.08
===============================================
Genode Labs
The flagship feature of Genode 17.08 has been in the works for more than a
year: The support for hardware-accelerated graphics on Intel Gen-8 GPUs. This
is an especially challenging topic because it is riddled with terminology,
involves highly complex software stacks, carries a twisted history with it,
and remains to be a moving target. It took up a lot of patience to build up a
profound understanding of the existing driver architectures and the mechanisms
offered by modern graphics hardware. On the other hand, with the proliferation
of hardware-based sandboxing features like virtual GPU memory and hardware
contexts, we found that now is the perfect time for a clean-slate design of a
microkernelized GPU driver.
Section [Hardware-accelerated graphics for Intel Gen-8 GPUs] introduces this
work, which includes our new GPU multiplexer as well as the integration with
the client-side Mesa protocol stack.
The second focus of the current release is the extension of Genode's supported
base platforms. Most prominently, we upgrade the seL4 kernel to version 6.0
while extending the architecture support from 32-bit x86 to ARM and 64-bit
x86 (Section [The seL4 kernel on ARM and 64-bit x86 hardware]). To bring
Genode closer to cloud-computing scenarios, we added basic support for
executing Genode scenarios as Xen DomU domains (Section [Genode as Xen DomU]).
Furthermore, the Muen separation kernel has been updated to a current version.
As a cross-kernel effort, there is work under way to boot Genode-based
systems via UEFI, currently addressing the NOVA, base-hw, and seL4 kernels.
Among the many other functional additions are a new VFS plugin for accessing
FAT file systems, new components like _sequence_ and _fs_report_ that aid new
system compositions, and our evolving custom package-management
infrastructure.
Hardware-accelerated graphics for Intel Gen-8 GPUs
##################################################
The ability to leverage hardware-accelerated graphics is generally taken for
granted in modern commodity operating systems. The user experience of
modern desktop environments, web-browser performance, and obviously games
depend on it. On the other hand, the benefit of hardware-accelerated graphics
comes at the expense of tremendous added complexity in the lower software
stack, in particular in system components that need to be ultimately trusted.
For example, with circa 100 thousand lines of code, the Intel GPU driver in
the Linux kernel is an order of magnitude more complex than a complete modern
microkernel. In a monolithic-kernel-based system, this complexity is
generally neglected because the kernel is complex anyway. But in
microkernel-based scenarios optimized for a trusted computing base in the
order of a few ten thousand lines of code, it becomes unacceptable.
Fortunately, recent generations of graphics hardware provide a number of
hardware features that promise to solve this conflict, which prompted us to
investigate the use of these features for Genode.
During this year's Hack'n'Hike event, we ported the ioquake3 engine to Genode.
As preliminary requirement, we had to resurrect OpenGL support in our aging
graphics stack and enable support for current Intel HD Graphics devices (IGD).
We started by updating Mesa from the old 7.8.x to a more recent 11.2.2 release.
Since we focused mainly on supporting Intel devices, we dropped support for the
Gallium back end as Intel still uses the old DRI infrastructure. This decision,
however, also influenced the choice of the software rendering back end. Rather
than retaining the softpipe implementation, we now use swrast. In addition, we
changed the available OpenGL implementation from OpenGL ES 2.x to the fully
fledged OpenGL 4.5 profile, including the corresponding shader language
version. As with the previous Mesa port, EGL serves as front end API for
system integration and loads a DRI back-end driver (i965 or swrast). EGL
always requests the back-end driver 'egl_drv.lib.so' in form of a shared
object. Genode's relabeling features are used to select the proper back end
via a route configuration. The following snippet illustrates such a
configuration for software rendering:
! <start name="gears" caps="200">
! <resource name="RAM" quantum="32M"/>"
! <route>
! <service name="ROM" label="egl_drv.lib.so">
! <parent label="egl_swrast.lib.so"/>
! </service>
! <any-service> <parent/> <any-child/> </any-service>
! </route>
! </start>
With the graphics-stack front end in place, it was time to take care of the
GPU driver. In our case this meant implementing the DRM interface in our
ported version of the Intel i915 DRM driver. Up to now, this driver was solely
used for mode setting while we completely omitted supporting the render
engine.
[image mesa_genode]
With this new and adapted software stack, we successfully could play ioquake3
on top of Genode with a reasonable performance in 1080p on a Thinkpad X250.
During this work, we gathered valuable insights into the architecture of a
modern 3D-graphics software stack as well as into recent Intel HD Graphics
hardware. We found that the Intel-specific Mesa driver itself is far more
complex than its kernel counter part. The DRM driver is mainly concerned with
resource and execution management whereas the Mesa driver programs the GPU.
For example, amongst others, Mesa compiles the OpenGL shaders into a
GPU-specific machine code that is passed on to the kernel for execution.
While inspecting the DRM driver, it became obvious that one of the reasons for
its complexity is the need to support a variety of different HD Graphics
generations as well as different features driven by software-usage patterns.
For our security related use cases, it is important to offer a clear isolation
and separation mechanism per client. Hardware features provided by modern
Intel GPUs like per-process graphics translation tables (PPGTT) and hardware
contexts that are unique for each client make it possible to fulfill these
requirements.
By focusing on this particular feature set and thus limiting the supported
hardware generations, the development of a maintainable GPU multiplexer for
Genode became feasible. After all, we strive to keep all Genode components as
low complex as possible, especially resource multiplexers like such a GPU
multiplexer.
[image intel_gpu_drv]
This image shows multiple GPU-session clients and the resources they are
using. The fence registers as well as the aperture is partitioned between
them, the PPGTT is backed by the system memory, and the contexts are located
in disjoint GGTT regions.
Within four months, we implemented an experimental GPU multiplexer for Intel
HD Graphics Gen8 (Broadwell class) devices. We started out defining a GPU
session interface that is sufficient to implement the API used by the DRM
library. For each session, the driver creates a context consisting of a
hardware context, a set of page tables (PPGTT), and a part of the aperture.
The client may use the session to allocate and map memory buffers used by the
GPU. Each buffer is always eagerly mapped 1:1 into the PPGTT by using the
local virtual address of the client. Special memory buffers like an image
buffer are additionally mapped through the aperture to make use of the
hardware-provided de-tiling mechanism. As is essential in Genode components,
the client must donate all resources that the driver might need to fulfill the
request, i.e., quota for memory and capability allocations. Clients may
request the execution of their workload by submitting an execution buffer. The
GPU multiplexer will then enqueue the request and schedule all pending
requests sequentially. Once the request is completed, the client is notified
via a completion signal.
[image multi_gl]
Example scenario of multiple OpenGL programs that use the new GPU multiplexer
for hardware-accelerated rendering.
We consider this first version of the GPU driver as experimental. As of now,
it only manages the render engine of the GPU. Mode-setting or rather display
handling must be performed by another component. Currently, the VESA driver is
used for this purpose. It also lacks any power-management functionality and
permanently keeps the GPU awake. Both limitations will be addressed in future
releases and support for Gen9+ (Skylake) and newer devices might be added.
In its current incarnation, the GPU multiplexer component consists of about
4,200 lines of code whereas the Mesa DRI i965 driver complements the driver at
the client side with about 78,000 lines of code.
The seL4 kernel on ARM and 64-bit x86 hardware
##############################################
With the 16.08 release, we brought the seL4 support to a level to be
considered being on par with the other supported kernels. At the time,
Genode's use of seL4 was limited to 32-bit x86 platforms.
In the current release, we extend the platform support to ARM and 64-bit x86.
We started this line of work with an incremental kernel upgrade from version
3.2.0 to 5.2.0 and finally to seL4 6.0. Through these upgrades, we were able
to drop several Genode-specific seL4 patches, which were required in the 16.08
release. One major improvement of version 6.0 compared to earlier versions is
the handling of device-memory announcements by the kernel to Genode's roottask
_core_.
With the kernel update in place, we inspected the x86-specific part thoroughly
while splitting and separating it properly into architecture-agnostic and
architecture-dependent parts. Upon this work, we added the
architecture-specific counterparts for x86_64 and ARM. One major work item was
to make the page-table handling in Genode's core aware and generic enough to
handle the different page-table sizes of the three architectures.
For the ARM support, we decided to enable the i.MX6 FreeScale based SoC,
namely the Wandboard Quad board. Since the seL4 kernel interface provides no
timeout support, we revived a user-level timer driver that we originally
developed for our custom base-hw kernel: The so-called EPIT timer, which is
part of most i.MX SoCs.
We finished the essential work for the mentioned three platforms in
less time than expected and, thereby, had spare time to address additional
features.
First, we enabled multiprocessor support for Genode/seL4 on x86 and
thread-priority support for all seL4 platforms. Additionally, we were able to
utilize the seL4 benchmark interface for Genode's trace infrastructure in
order to obtain utilization information about threads and CPUs. The Genode
components _top_ (text-based) and _cpu_load_monitor_ (graphical) are now
usable on Genode/seL4.
Finally, as we are currently exploring the support for booting various kernels
via UEFI on x86, we took the chance to investigate the steps needed to boot
seL4 via UEFI. UEFI firmware does not always provide a compatibility support
module (CSM) for legacy BIOS boot support. Hence, we extended the seL4 kernel
for Genode according to the Multiboot2 specification, which enables us to
start Genode/seL4 together with GRUB2 - as an UEFI capable bootloader - on
machines missing CSM support.
Base framework and OS-level infrastructure
##########################################
Simplified IOMMU handling
=========================
When IOMMUs are used on x86, all host memory targeted via direct memory
accesses (DMA) by devices must eagerly be registered in the respective I/O
page table of the device. Up to now, Genode supports IOMMUs on NOVA only. On
this kernel, a device protection domain is represented as a regular protection
domain with its virtual memory layout being used for both the CPU's MMU and
the device. Traditionally, mappings into such virtual memory spaces are
inserted on demand as responses to page faults. However, as there are no page
faults for DMA transactions, DMA buffers must always be eagerly mapped. The
so-called device PD hid this gap for NOVA. In anticipation of adding IOMMU
support for more kernels, we desired to generalize the device-PD mechanism by
introducing an explicit way to trigger the insertion of DMA memory into the
proper page tables.
We extended the PD-session interface by a 'map' function, which takes a
virtual memory region of the PD's virtual address space as argument. The page
frames of the previously attached dataspaces are added eagerly by core to the
IOMMU page-tables. With this explicit 'map' support, we were able to replace
the Genode/NOVA-specific device-PD implementation with a generic one, which
will easily accommodate other kernels in the future.
New report server for capturing reports to files
================================================
The report session is a simple mechanism for components to publish structured
data without the complexity of a file-system layer. In the simplest case, a
client component will produce a report and communicate it directly to a
component acting as a server. The disadvantage is that the report client
becomes reliant on the liveliness and presence of the consumer component. So
in the more robust case, the _report_rom_ component acts as the server hosting
the report service as well as a ROM service for components consuming reports.
The _report_rom_ server permits ROM access only to clients matching an
explicit configuration policy. This is good for security but opaque to a user.
Reports can only be read where an explicit policy is in place and only a
single report session can report to an active ROM session.
The new _fs_report_ component is a friendlier and more flexible report server.
Reports are written to a file system using a file and directory hierarchy that
expresses session routing. This allows for intuitive report inspection and
injection via a file system. When used with the _ram_fs_ and _fs_rom_ servers,
it can also replicate the functionality of _report_rom_.
New runtime environment for starting components sequentially
============================================================
The _init_ component is a prime example of software with an emphasis on
function over features. It is the fundamental building block for combining
components yet its behavior is simple and without heuristics. Like other
contemporary init managers, it starts components in parallel, but to a more
extreme degree in that it has no concept of "runlevels" or "targets", all
components are started as soon as possible. The concrete sequence of execution
is instead determined by when server components make service announcements and
how quickly they respond to client requests.
In some cases, the execution of one component must not occur until the
execution of another component ends, be it that the first produces output that
is consumed by the second, or that the two contend for a service that cannot
be multiplexed. Init contains no provisions to enforce ordering. But we are
free to define new behaviors in other management components.
The solution to the problem of ordering is the _sequence_ component. Sequence
walks a list of children and executes them in order, one at a time. With only
one child active, there is no need for any local resource or routing
management. By applying the same session label transformations as init,
external routing and policy handling are unchanged.
An example of ordering a producer and consumer within an init configuration
follows:
! <start name="sequence">
! <resource name="RAM" quantum="128M"/>
! <config>
! <start name="producer">
! <config .. />
! </start>
! <start name="consumer">
! <config .. />
! </start>
! </config>
! <route>
! <service name="LOG" label_prefix="producer">
! <child name="log_a"/> </service>
! <service name="LOG" label_prefix="consumer">
! <child name="log_b"/> </service>
! <any-service> <parent/> <any-child/> </any-service>
! </route>
! </start>
Support for boot-time initialized frame buffer
==============================================
UEFI-based systems do not carry along legacy BIOS infrastructure, on which
our generic VESA driver depends. Hence, when booting via UEFI, one has to use
either a hardware-specific driver like our Intel-FB driver or - alternatively -
facilitate generic UEFI mechanisms.
Instead of booting in VGA text mode and leaving the switch to a graphics mode
(via real-mode SVGA BIOS subroutines) to the booted OS, UEFI employs the
so-called graphics output protocol as a means to setup a reasonable default
graphics mode prior booting the operating system. In order to produce
graphical output, the operating system merely has to know the physical address
and layout of the frame buffer. Genode's core exposes this information as the
_platform_info_ ROM module. The new _fb_boot_drv_ driver picks up this
information to provide a Genode framebuffer session interface. Hence, on
UEFI-based systems, it can be used as a drop-in replacement for the VESA
driver. In contrast to the VESA driver, however, it is not able to switch
graphics modes at runtime.
The new component is located at _os/src/drivers/framebuffer/boot/_. Thanks
to Johannes Kliemann for this contribution.
Extended non-blocking operation of the VFS
==========================================
In
[https://genode.org/documentation/release-notes/17.02#VFS_support_for_asynchronous_I_O_and_reconfiguration - version 17.02],
we added support for non-blocking reads from the VFS in the form of the
'read_ready()', 'queue_read()', and 'complete_read()' functions. Since then,
it has become obvious that blocking within the VFS is not only problematic in
the VFS server itself when multiple clients are connected, but also when the
VFS is deployed in a multi-threaded environment and a VFS plugin needs to
reliably wait for I/O-completion signals.
For this reason, we reworked the interface of the VFS even more towards
non-blocking operation and adapted the existing users of the VFS accordingly.
The most important changes are:
* Directories are now created and opened with the 'opendir()' function and
the directory entries are read with the 'queue_read()' and 'complete_read()'
functions.
* Symbolic links are now created and opened with the 'openlink()' function and
the link target is read with the 'queue_read()' and 'complete_read()'
functions and written with the 'write()' function.
* The 'write()' function does not wait for signals anymore. This can have the
effect that data written by a VFS library user has not been processed by a
file-system server when the library user asks for the size of the file or
closes it (both done with RPC functions at the file-system server). For this
reason, a user of the VFS library should request synchronization before
calling 'stat()' or 'close()'. To make sure that a file-system server has
processed all write request packets that a client submitted prior the
synchronization request, synchronization is now requested at the file-system
server with a synchronization packet instead of an RPC function. Because of
this change, the synchronization interface of the VFS library has been split
into the 'queue_sync()' and 'complete_sync()' functions.
Making block sessions read-only by default
==========================================
Genode server components are expected to apply the safest and strictest
behavior when exposing cross-component state or persistent data. In practice
block and file-system servers only allow access to clients with explicitly
configured local policies. The file-system servers enforce an additional
provision that sessions are implicitly read-only unless overridden by policy.
This release introduces a similar restriction to the AHCI driver and partition
multiplexer. Clients of these servers require an affirmative 'writeable'
attribute on policies to permit the writing of blocks. Write permission at
these servers may also be revoked by components that forward block-session
requests by placing 'writeable="no"' into session-request arguments.
All users of _ahci_drv_ and _part_blk_ are advised that this change may break
existing configurations without explicit 'writeable' policies.
Refined time handling
=====================
Release 17.05 introduced a
[https://genode.org/documentation/release-notes/17.05#New_API_for_user-level_timing - new API for user-level timing]
named _timeout framework_. Together with this new framework came a
comprehensive test that stresses all aspects of the interface. During the past
few months, this test has turned out to be an enrichment for Genode far beyond
its original scope. As the test significantly raised the standards in
user-level timing, it also sharpened our view on the measurement precision of
various timer drivers and timestamps, which act as input for the framework.
This revealed several problems previously unidentified. For instance, we
improved the accuracy and stability of the time values provided by the drivers
for the Raspberry-Pi timer, the Cortex-A9 timer, the PIT, and the LAPIC. We
also were able to further optimize the calibration of the TSC in the NOVA
kernel.
Additionally, the test also helped us to refine the timeout framework itself.
The initial calibration of the framework - that previously took about 1.5
seconds - is now performed much quicker. This makes microseconds-precise time
available immediately after the timer connection switched to the modern
fine-grained mode of operation, which is a prerequisite for hardware drivers
that need such precision during their early initialization phase. The
calculations inside the framework also became more flexible to better fit the
characteristics of all the hardware and kernels Genode supports.
Finally, we were able to extend the application of the timeout framework. Most
notably, our C runtime uses it as timing source to the benefit of all
libc-using components. Another noteworthy case is the USB driver on the
Raspberry Pi. It previously couldn't rely on the default Genode timing but
required a local hardware timer to reach the precision that the host
controller expected from software. With the timeout framework, this workaround
could be removed from the driver.
FatFS-based VFS plugin
======================
Genode has supported VFAT file-systems since the 9.11 release when the
[http://elm-chan.org/fsw/ff/00index_e.html - FatFS] library was first ported.
The 11.08 release fit the library into the libc plugin architecture and
in 12.08 FatFS was used in the _ffat_fs_ file-system server. Now, the 17.08
release revisits FatFS to mold the library into the newer and more flexible
VFS plugin system. The _vfs_fatfs_ plugin may be fitted into the VFS server or
used directly by arbitrary components linked to the VFS library. As the
collection of VFS plugins in combination with the VFS file-system server has a
lower net maintenance cost than multiple file-system servers, the _ffat_fs_
server will be retired in a future release.
Enhanced GUI primitives
=======================
Even though we consider Qt5 as the go-to solution for creating advanced
graphical user interfaces on top of Genode, we also continue to explore an
alternative approach that facilitates Genode's component architecture to an
extreme degree. The so-called menu-view component takes an XML description of
a dialog as input and produces rendered pixels as output. It also gives
feedback to user input such as the hovered widget at a given pointer position.
The menu view does not implement any application logic but is meant to be
embedded as a child component into the actual application. This approach
relieves the application from the complexity (and potential bugs) of widget
rendering. It also reinforces a rigid separation of a view and its underlying
data model.
The menu view was first introduced in
[https://genode.org/documentation/release-notes/14.11#New_menu_view_application - version 14.11].
The current release improves it in the following ways:
* The new '<float>' widget aligns a child widget within a
larger parent widget by specifying the boolean attributes 'north', 'south',
'east', and 'west'. If none is specified, the child is centered. If opposite
attributes are specified, the child is stretched.
* A new '<depgraph>' widget arranges child widgets in the form of a
dependency graph, which will be the cornerstone for Genode's upcoming
interactive component-composition feature. As a prerequisite for
implementing the depgraph widget, Genode's set of basic graphical primitives
received new operations for drawing sub-pixel-accurate anti-aliased lines
and bezier curves.
* All geometric changes of the widget layout are animated now. This includes
structural changes of the new '<depgraph>' widget.
[image depgraph]
The menu-view component is illustrated by the run script at
_gems/run/menu_view.run_.
C runtime
=========
The growing number of ported applications used on Genode is accompanied by the
requirement of extensive POSIX compatibility of our C runtime. Therefore, we
enhanced our implementation by half a dozen features (e.g., O_ACCMODE
tracking) during the past release cycle. We thank the contributors of patches
and test cases and will continue our efforts to accommodate more ported
open-source components in the future.
Libraries and applications
##########################
Mesa adjustments
================
The Mesa update required the adaption of all components that use OpenGL.
In particular that means the Qt5 framework. Furthermore, we also enabled
OpenGL support in our SDL1 port.
As playground, there are a few OpenGL examples. The demos are located under
_repos/libports/src/test/mesa_demos_, which use the EGLUT bindings. There
are also some SDL based examples in the world repository under
_repos/world/src/test/sdl_opengl_.
Package management
==================
The previous release featured the initial version of Genode's
[https://genode.org/documentation/release-notes/17.05#Package_management - custom package-management tools].
Since then, we continued this line of work in three directions.
First, we refined the depot tools and the integration of the depot with our
custom work-flow ("run") tool. One important refinement is a simplification of
the depot's directory layout for library binaries. We found that the initial
version implied unwelcome complexities down the road. Instead of placing
library binaries in a directory named after their API, they are now placed
directly in the architecture directory along with regular binaries.
Second, driven by the proliferated use of the depot by more and more run
scripts, we enhanced the depot with new depot recipes as needed.
Third, we took the first steps to use the depot on-target. The experimentation
with on-target depots is eased by the new 'create_tar_from_depot_binaries'
function of the run tool, which allows one to assemble a new depot in the form
of a tar archive out of a subset of packages. Furthermore, the new
_depot_query_ component is able to scan an on-target depot for runtime
descriptions and returns all the information needed to start a subsystem based
on the depot content. The concept is exemplified by the new
_gems/run/depot_deploy.run_ script, which executes the "fs_report" test case
supplied via a depot package.
Platforms
#########
Genode as Xen DomU
==================
We want to widen the application scope of Genode by enabling users to easily
deploy Genode scenarios on Xen-based cloud platforms.
As a first step towards this goal, we enhanced our run tool to support running
Genode scenarios as a local Xen DomU, managed from within the Genode build
system on Linux running as Xen Dom0.
The Xen DomU runs in HVM mode (full virtualization) and loads Genode from an
ISO image. Serial log output is printed to the text console and graphical
output is shown in an SDL window.
To use this new target platform, the following run options should be defined in
the 'build/x86_*/etc/build.conf' file:
! RUN_OPT = --include boot_dir/$(KERNEL)
! RUN_OPT += --include image/iso
! RUN_OPT += --include power_on/xen
! RUN_OPT += --include log/xen
! RUN_OPT += --include power_off/xen
The Xen DomU is managed using the 'xl' command line tool and it is possible to
add configuration options in the 'xen_args' variable of a run script. Common
options are:
* Disabling the graphical output:
! append xen_args { sdl="0" }
* Configuring a network device:
! append xen_args { vif=\["model=e1000,mac=02:00:00:00:01:01,bridge=xenbr0"\] }
* Configuring USB input devices:
! append xen_args { usbdevice=\["mouse","keyboard"\] }
Note that the 'xl' tool requires super-user permissions. Interactive
password input can be complicated in combination with 'expect' and is not
practical for automated tests. For this reason, the current implementation
assumes that no password input is needed when running 'sudo xl', which can
be achieved by creating a file '/etc/sudoers.d/xl' with the content
! user ALL=(root) NOPASSWD: /usr/sbin/xl
where 'user' is the Linux user name.
Execution on bare hardware (base-hw)
====================================
UEFI support
------------
Analogously to our work on the seL4 and NOVA kernels in this release, we
extended our base-hw kernel to become a Multiboot2 compliant kernel. When used
together with GRUB2, it can be started on x86 UEFI machines missing legacy
BIOS support (i.e., CSM).
RISC-V
------
With Genode version 17.05, we updated base-hw's RISC-V support to privileged
ISA revision 1.9.1. Unfortunately, this implied that dynamic linking was not
supported on the RISC-V architecture anymore. Since dynamic linking is now
required for almost all Genode applications by default, this became a severe
limitation. Therefore, we revisited our RISC-V implementation - in particular
the kernel entry code - to lift the limitation of being able to execute only
statically linked binaries.
Additionally, we integrated the Berkeley Boot Loader (BBL), which bootstraps
the system and implements the machine mode, more closely into our build
infrastructure. We also added a new timer implementation to base-hw by using
the _set timeout SBI_ call of BBL.
What still remains missing is proper FPU support. While we are building the
Genode tool chain with soft float support, we still encounter occasions where
FPU code is generated, which in turn triggers compile time errors. We will
have to investigate this behavior more thoroughly, but ultimately we want to
add FPU support for RISC-V to our kernel and enable hardware floating point in
the tool chain.
Muen separation kernel
======================
Besides updating the Muen port to the latest kernel version as of end of June,
Muen has been added to Genode's automated testing infrastructure. This
includes the revived support for VirtualBox 4 on top of this kernel.
NOVA microhypervisor
====================
The current release extends NOVA to become a Multiboot2 compliant kernel. Used
together with GRUB2, NOVA can now be started on x86 UEFI machines missing
legacy BIOS support (called CSM).
GRUB2 provides the initial ACPI RSDP (Root System Description Pointer) to a
Multiboot2 kernel. The RSDP contains vital information required to bootstrap
the kernel and the operating system in general on today's x86 machines. To
make this information available to the user-level ACPI and ACPICA drivers, the
kernel propagates the RSDP to Genode's core, which - in turn - exposes it to
the user land as part of the _platform_info_ ROM module.
In order to ease the setup of an UEFI bootable image, we added a new image
module to our run-tool infrastructure. The run option 'image/uefi' can be used
instead of 'image/iso' in order to create a raw image that contains a EFI
system partition in a GUID partition table (GPT). The image is equipped by the
new 'image/uefi' module with the GRUB2 boot loader, a GRUB2 configuration, and
the corresponding Genode run scenario. The final image can be copied with 'dd'
to a bootable USB stick. Additionally, we added support to boot such an image
on Qemu leveraging [https://www.tianocore.org - TianoCore's] UEFI firmware.
As a side project, minor virtualization support for AMD has been added to
Virtualbox 4 and to the NOVA kernel on Genode. This enables us to run a 32-bit
Windows 7 VM on a 32-bit Genode/NOVA host on an (oldish) AMD Phenom II X4 test
machine.

0
doc/release_notes-17-11.txt → doc/release_notes/17-11.txt
View File

898
doc/release_notes/18-02.txt Normal file
View File

@@ -0,0 +1,898 @@
===============================================
Release notes for the Genode OS Framework 18.02
===============================================
Genode Labs
After being developed for over a decade, Genode remained a mystery for many
people who looked at the project from a distance as it does not seem to fit
any established category of software. In 2018 - declared as the Year of Sculpt
on our [https://genode.org/about/road-map - roadmap] - this will hopefully
change. Genode 18.02 features the first revision of Sculpt, which is a
Genode-based operating system for general-purpose computing. After being used
as day-to-day OS by the entire team of Genode Labs for several months, we feel
that the time is right to share the system with a broader audience (Section
[Sculpt for Early Adopters]).
One fundamental feature of Sculpt is the ability to install and deploy
software from within the running operating system, which is universally
expected from any modern general-purpose OS. Section
[On-target package installation and deployment] presents Genode's unique
take on the topic of software installation and deployment.
Besides Sculpt, the current release has no shortage of other improvements.
Genode's growing arsenal of 3rd-party software received profound updates and
additions, including VirtualBox, Muen, seL4, several GNU packages, and
libraries. Also the user-level networking stack - including the Linux-based
LxIP stack and our custom NIC-router component - received a lot of attention.
Thanks to the added network driver for i.MX-based hardware, this networking
infrastructure becomes usable on embedded platforms based on this SoC.
Furthermore, the current release continues the cultivation of the Nim
programming language for Genode components.
Sculpt for Early Adopters
#########################
The current release features the first revision of Sculpt, which is a
Genode-based operating system for general-purpose computing. This initial
version is called Sculpt for Early Adopters (EA). Its target audience are
enthusiasts who are already familiar with Genode and are eager to use a
Genode-based operating system on their machines. As outlined on the
[https://genode.org/about/road-map - roadmap], later versions will become
increasingly approachable.
[image sculpt_overview]
Please refer to the official
[https://genode.org/documentation/articles/sculpt-ea - Sculpt documentation]
to step right into the adventure.
On-target package installation and deployment
#############################################
In May last year, we introduced the package-management concept for Genode to
pursue two goals. First, to overcome the naturally limited scalability of
composing Genode systems solely from source. This limit became evident in
complex system scenarios that incorporate a huge amount of 3rd party software.
Thanks to the introduced _depot_ concept and its integration in Genode's
workflow - in particular the run tool - the work of system integration became
much more structured (by caring about packages instead of individual build
targets), robust (by avoiding conditions in run scripts), and quick (by the
accelerated test cycle when using pre-built packages).
The second goal is the ability to update and extend a running Genode system on
the fly. We are happy to have reached this goal with the current release. As
exemplified by the Sculpt scenario, packages cannot only be used as building
blocks for system images but also as subsystems dynamically installed and
deployed on target. Even though installation and deployment are closely
related topics, both involve distinct challenges, which allow Genode to shine.
Installation / update
=====================
In traditional operating systems, the installation and update of system
software is the job of privileged programs. For example, a package manager in
a GNU/Linux system is typically executed with root privileges. This is
troublesome because the functionality of such a program is extremely complex.
In particular it is exposed to the network and has to parse content
originating from potentially untrusted parties. Therefore, potential software
vulnerabilities should be expected. However, in modern OSes, these programs
are just assumed to behave correctly. If this overly optimistic assumption
doesn't hold, the entire system is at risk.
Genode helps us to mitigate this problem by modelling each installation step
as a distinct component composition where each component has a well-defined
and extremely narrow role. The installation is an iterative sequence that
is orchestrated by the so-called download-manager component
(Figure [depot_download]).
[image depot_download]
Initially, the download manager receives a list of content to be installed
into the local depot, which is stored on the file system. The depot may
already be populated with (portions of) this content. In the first step, the
download manager must determine the parts that are missing. To do that, it
does not access the file system directly but instead hands over this task to a
disposable helper component called _depot-query_ that is spawned within a
dynamic init instance. This indirection has two benefits. First, the download
manager is not bothered with the complexity of accessing the file system. It
does not even have any notion of files. Second, the download manager is
effectively shielded from the file system. Should the file system misbehave,
the liveliness of the download manager remains unaffected.
[image depot_download_query_deps]
The depot-query component reports its findings to a report session. The report
eventually reaches the download manager as an updated ROM module. Given the
list of missing content, the download manager has to determine the information
of where to obtain the content from and the public key of the content creator.
This information is contained within the depot. So the download manager issues
another request to the depot-query component in order to obtain it.
[image depot_download_query_url]
Once the depot-query component has responded, the download manager knows what
content to get, where to get it, and how to verify it. To download the
content, it changes the dynamic init instance as follows.
[image depot_download_fetch]
The depot-query component is now gone. Actually, the entire depot has moved
out of sight. Instead, a fresh _fetchurl_ component is spawned. This component
is connected to the network as well as the writeable download directory
_public/_. Internally, fetchurl employs a complex software stack, which
includes the C runtime, curl, libssl, and libssh. Hence, we expect this
component to be vulnerable. Since it is facing the network, we assume that
vulnerabilities are exploitable. In the worst case where the component is
completely in the hands of an attacker, it may write wrong content into the
_public/_ location. But compared to executing curl or wget as root on a
traditional Unix system, the reach of an attack is quite limited. For example,
the mere existence of the download manager remains completely out of view of
fetchurl. However, the content of _public/_ must not be trusted. To reinforce
trust in the downloaded content, the content is accompanied with cryptographic
signatures created by the content creator. Before we touch the content, we
first check its authenticity. To perform this verification step, the download
manager reshapes the dynamic init instance as follows.
[image depot_download_verify]
Note that fetchurl exists no more and network connectivity is cut, effectively
disposing any form of malware that might have infected fetchurl. Next a new
_verify_ component enters the picture. It is configured with a list of content
to check, the signatures of the content, and the public key of the content's
presumed creator. Since it accesses the _public/_ location exclusively, it is
not prone to any potential time-of-check to time-of-use problems during the
verification. Under the hood, the _verify_ component employs a hugely complex
implementation based on GnuPG. It would be naive to fully trust this code.
However, when embedded in our scenario, the reach of a bug is limited because
the verify component has no access to any mutable system state. It could
merely give the wrong answer (which is of course bad but there is no way we
can magically solve this).
Knowing that the downloaded content is indeed the same content as intended
by the creator, it is time for extraction. For this step, the download
manager - again - reshapes the dynamic init instance:
[image depot_download_extract]
This time, both the _public/_ location as well as the trusted _depot/_ are
visible and a new _extract_ component is spawned. As the depot may host
content from multiple sources, which potentially distrust each other, the
content of each content provider resides in a dedicated subdirectory within
the depot. Instead of handing over access to the entire depot to the extract
tool, we mediate the file-system access via a _chroot_ component that limits
the view to the depot-provider's respective subdirectory. In the worst case
where a misbehaving content provider delivers a forged (but correctly signed)
archive to exploit a vulnerability of the extract component, the reach of the
attack remains limited to the content provider's space within the depot.
After the extraction step has completed, the depot is populated with the new
content, which may - in turn - include new dependency information. At this
point, the download manager starts a new iteration. This iterative process
terminates as soon as the depot-query component signals that no content of
the software installation is missing.
The bottom line here is that we are able to use complex and useful software
like curl, libarchive, liblzma, and GnuPG while largely distrusting it. In
contrast to this software that sums up to hundreds of thousand lines of code,
the download manager comprises less than 1000 lines of code. The software
installation procedure described above is implemented by the 'depot_download'
subsystem hosted in the gems repository and illustrated by an equally named
run script. It also forms the basis of the install/update mechanism of the
Sculpt scenario.
Deployment
==========
Once software has entered the system in the form of depot content, the
remaining question is how to turn this content into running subsystems. The
answer is given by the following illustration.
[image sculpt_deploy_runtime]
Like for the installation process described above, the scenario employs a
dynamic init instance that is accompanied by an orchestrating component. The
latter is called _depot-deploy_. The depot-deploy component queries
information from the depot using the same depot-query component that was used
during the installation. Based on the returned _blueprint_ information for the
to-be-deployed subsystems, it generates the configuration for the dynamic init
instance. The subsystems hosted within this init instance access the depot
content via mere ROM sessions as provided by the FS-ROM component. This makes
the use of the depot transparent to the hosted subsystems.
The depot-deploy component is located in the gems repository and accompanied
by a same-named run script. More importantly, it is featured in the deploy
runtime of the Sculpt system.
Base framework and OS-level infrastructure
##########################################
Increased default warning level
===============================
For building Genode components written in C++, the compiler flags -Wextra,
-Weffc++, and -Werror are now enabled in addition to -Wall by default.
If this strict warning level is inapplicable for a given component or
library, it is possible to explicitly disable the strictness in the
respective build-description file by adding the following line:
! CC_CXX_WARN_STRICT =
We adjusted almost all the code of the base, base-<kernel>, os, and demo
repositories to comply with this new warning level. For most components
hosted in the higher-level repositories (libports, ports, dde_*, gems),
the strictness is disabled as of now and will be enabled component-wise
wherever feasible.
While adjusting our code base, we identified the following patterns worth
mentioning:
* A class with virtual functions can no longer publicly inherit base
classes without a vtable. The inherited object may either be moved
to a member variable, or inherited privately. The latter would be
used for classes that inherit 'List::Element' or 'Avl_node'. In order
to enable the 'List' and 'Avl_tree' to access the meta data, the
'List' must become a friend.
* Instead of adding a virtual destructor to abstract base classes,
we inherit the new 'Interface' class, which contains a virtual
destructor. This way, single-line abstract base classes can stay
as compact as they are. The 'Interface' utility resides in
_base/include/util/interface.h_.
* With the new warning level, all member variables must be explicitly
initialized. Basic types may be initialized with '='. All other types
are initialized with braces '{ ... }' or as class initializers. If
basic types and non-basic types appear in a row, it is nice to only
use the brace syntax (also for basic types) and align the braces.
* If a class contains pointers as members, it must now also provide a
copy constructor and assignment operator. In most cases, one
would make them private, effectively disallowing the objects to be
copied. Unfortunately, this warning cannot be fixed by inheriting
our existing 'Noncopyable' class (the compiler fails to detect that
the inheriting class cannot be copied and still gives the error).
For now, we have to manually add declarations for both the copy
constructor and the assignment operator as private class members.
Those declarations should be prepended with a comment like this:
! /*
! * Noncopyable
! */
! Thread(Thread const &);
! Thread &operator = (Thread const &);
In the future, we plan to revisit these occurrences and try to replace
the pointers with references. In the presence of at least one
reference member, the compiler would no longer implicitly generate
a copy constructor. So we could remove the manual declaration.
The following caveats are expected, even if you disable the strictness
in your component:
* If your component has a class called 'Interface', it may collide with
the new 'Genode::Interface' class. You may have to disambiguate the
names.
* The 'Genode::Rpc_client' is no longer a 'Genode::Capability'. Hence,
classes inherited from 'Genode::Rpc_client' cannot refer to a
'Capability' but must refer to 'Genode::Capability'.
* The 'Surface' class is no longer copyable, which led to API
changes of users of this class. E.g., the 'Nitpicker_buffer'
utility does no longer offer accessors for the contained surfaces
but a new 'apply_to_surface' method that takes a lambda function as
argument.
Init
====
Init selects session routes based on the requested service and the client's
label. The latter can be matched as 'label' (exact match), 'label_prefix', or
'label_suffix' (either end of the label matches). With the new version, these
options are complemented with an additional 'label_last' attribute that covers
the prominent case where the last part of the label identifies a requested
resource at the server. A typical example is the routing of a ROM session
based on the name of the requested ROM module.
Reflecting the core log to the application level
================================================
Core records now log messages in a ring buffer and exports this
memory as ROM named 'core_log'. User applications may monitor this ring buffer
and present or transfer the content as appropriate. The example component in
_repos/os/src/app/log_core_ transforms the content into normal log
messages, which may be routed to graphical terminals or stored on
file systems, e.g. by using the fs_log server.
NIC-router improvements
=======================
During the past three months, the NIC router has received several improvements
that were mainly inspired by our daily experience with the component as part
of our Sculpt based working environments.
The most notable new feature is the support for multiple NIC sessions at one
domain. If multiple NIC-session clients connect to one domain, the NIC router
acts as a simple hub between them. I.e., for every packet that is routed to
the domain, each connected session receives a copy of the packet. The same
applies for domain-local packets, meaning packets that target an IP address
inside the IP subnet of the domain they came from. This domain-local
forwarding applies before considering any other routing rules. So, in other
words, it is not possible to route such traffic to another domain.
Furthermore, the logging features of the NIC router were improved. First, the
router is now capable of periodically sending a report via Genode's report
session. This can be activated by adding the new '<report>' node to the router
configuration:
! <config>
! <report interval_sec="5" bytes="yes" config="yes">
! ...
! </config>
So far, the report provides per-domain information about the amount of sent
and received data ('bytes' attribute) and the current IPv4 configuration like
IP address, subnet mask, and gateway address ('config' attribute).
Second, there is a new verbosity option in the '<config>' node:
! <config verbose_domain_state="yes">
When this option is set, the NIC router will output a short message to the log
for each general state change of a domain. Currently, this includes the
IP-configuration state (IP address, subnet mask, gateway address) and the
number of connected NIC sessions. This is a useful addition because the
purpose of the regular verbose option is to give a very deep insight into
almost every activity of the router, which is vital for debugging
sophisticated problems but normally floods the log. Therefore, the regular
verbose option is not viable for complex setups like a Sculpt desktop
environment. In such a context, the new domain-state verbosity is pretty
discreet but already gives a good hint on why, for instance, packets get
dropped despite the routing rules being correct.
Last but not least, the timeout configuration of the NIC router has been
reworked and now allows for a much more precise adaption to the network
environment. The former 'rtt_sec' attribute of the '<config>' node has been
replaced by the following new attributes (default values shown):
! <config dhcp_discover_timeout_sec="10"
! dhcp_request_timeout_sec="10"
! dhcp_offer_timeout_sec="10"
! udp_idle_timeout_sec="30"
! tcp_idle_timeout_sec="600"
! tcp_max_segm_lifetime_sec="30">
Details about the new attributes can be found in the
_os/src/server/nic_router/README_ file. The default values should be
appropriate for the common use case so that specifying them is normally not
necessary.
New watch mechanism for file-system session
===========================================
The file-system session already provided a way for watching files or
directories for changes. However, the original mechanism was arguably hard to
use. In addition to opening the to-be-watched file-system node, the client had
to submit a so-called content-changed request into the session's request
queue. In turn, the server delivered the change notification by acknowledging
this request.
The new mechanism is much less bureaucratic. A file or directory can be
watched by opening a watch handle rather than submitting a 'CONTENT_CHANGED'
packet to the server. Whenever a change happens at a node with an open watch
handle, a CONTENT_CHANGED packet will be sent from the server to the client.
This serializes the registration with other handle operations and separates
I/O handle state from notification handle state.
C runtime
=========
We changed libc's handling of 'clock_gettime' to be explicitly configurable
rather than relying on built-in heuristics. With the new version, the libc
opens a timer session as a time source only if the 'rtc' attribute of the
'<libc>' configuration node is defined. If not configured, 'clock_gettime'
returns 0.
This change may require the adjustment of components that implicitly rely on
the libc as time source. To enable such a component to use relative time
(based on a timer session) but no wall-clock time, one can manually provide a
pseudo real-time clock value as follows:
! <vfs>
! <dir name="dev">
! <log/> <null/> <inline name="rtc">2000-01-01 00:00</inline>
! </dir>
! </vfs>
! <libc stdout="/dev/log" stderr="/dev/log" rtc="/dev/rtc"/>
GUI stack and terminal improvements
===================================
Nit-FB improvements
-------------------
The nit_fb component provides a framebuffer and input service while using the
nitpicker GUI server as back end. The new version adds the 'initial_width' and
'initial_height' attributes, which accommodate the use case where nit_fb is
used in a dynamic fashion like as a client of a window system. Here, the
initial dimensions define the initial window size but - in contrast to the
existing 'width' and 'height' attributes - the actual size can change
afterwards.
Terminal resizing
-----------------
The terminal-session interface gained the ability to propagate resize events
from the server to the client. The new version of the graphical terminal uses
this mechanism to support window resizing as well as dynamically changing the
font size. At the client side, noux has become able to reflect terminal-size
changes to noux applications. Applications based on ncurses (e.g., vim) are
able to gracefully respond to such changes now.
Using chroot to enforce read-only file-system access
====================================================
By placing a chroot component in-between a file-system client and server, the
client's view on the file system can be limited to a specific directory. With
the current release, chroot can additionally be used to restrict a writeable
file-system session to become read-only. This is accomplished by the new
'writeable' attribute of chroot's policy nodes. By default, it is set to "no".
API changes
===========
Noncopyable AVL node/tree
-------------------------
Copying an AVL node generally violates the integrity of the corresponding
tree. To rule out subtle bugs where AVL nodes are accidentally copied, AVL
nodes are no longer copyable.
New 'Buffered_xml' utility
--------------------------
The 'Buffered_xml' utility located at _os/buffered_xml.h_ simplifies the
implementation of dynamically reconfigurable components that need to keep a
verbatim copy of certain parts of their configuration during configuration
updates.
New 'List_model' utility
------------------------
More and more components respond to dynamic configuration updates. For most
components, such updates are quite simple: replace an old internal state by a
new one. But in cases like init, menu_view, or window decorator, a
differential update is in order. Until now, each of these components employed
custom code for this task. As this code is not trivial, a common solution is
preferable. This solution comes in the form of the new 'List_model' utility
located at _base/include/util/list_model.h_. It introduces a light-weight
formalism to feed a component-internal data model from an externally-provided
XML structure.
Dynamically expandable reporter utility
---------------------------------------
In many cases, components that generate reports don't explicitly handle the
situation where the default buffer size of 4096 bytes is exceeded by the
report. This problem is easy to miss because reports are often small at
testing time but become larger when deployed in complex scenarios. In most
cases, the best way to handle an 'Xml_generator::Buffer_exceeded' exception is
upgrading the report session. The new 'Expanding_reporter' that accompanies
the original 'Reporter' in _os/reporter.h_ eases the handling of this common
case.
Languages and runtime environments
##################################
Nim programming language
========================
A new Nim library for constructing Genode servers is now available in the
World repository. This module provides utilities for the asynchronous
session-creation procedure introduced in the
[https://genode.org/documentation/release-notes/16.11#New_session-creation_procedure - 16.11]
release. Some introductory code snippets are provided here for the
adventurous.
An example of server creation using the 'genodeservers' module:
! import romclient, genodeservers
!
! var
! sessionsRom = newRomClient "session_requests"
! # synchronously open a ROM client to the parent
! romContent = sessionsRom.stream.readAll()
! # copy the ROM content to a heap string
! requestsParser = initSessionRequestsParser(romContent)
! # a state machine for parsing 'session_requests' XML
!
! for id, service, label in requestsParser.create:
! # the `create` iterator provider for the parser
! # hides the details of parsing the XML data
! discard txBufSize = requestsParser.argInt "tx_buf_size"
! # extract typed session arguments from the current parser state
! discard label.lastLabelElement()
! # label handling utilities are provided
! if service == "MyService":
! myCreateSessionProc(id, label)
!
This module streamlines the handling of session metadata, but the developer
must still provide hand-crafted wrappers over the C++ methods for managing
RPC objects and passing session capabilities to the parent. Most notoriously
a global pointer symbol, `genodeEnv`, is used to expose the component
environment object. In the future, this will be replaced by a typed object
passed from runtime to an application entry procedure.
! type MySessionCapability {.
! importcpp: "My_session::Session_capability",
! header: "my_session/capability.h".}
! # import a capability type
!
! type MyNativeSessionBase {.
! importcpp: "My_session::Session_rpc_object",
! header: "my_session/rpc_object.h".}
! # import C++ session RPC object
!
! type MyNativeSession = Constructible[MyNativeSessionBase]
! # apply the C++ Constructible template to defer calling
! # the object constructor
!
! proc construct(cppObj: MyNativeSession) {.
! importcpp: "#.construct(*genodeEnv)".}
! # call the C++ constructor, passing the global Genode::Env
!
! proc manage(cppObj: MyNativeSession): MySessionCapability {.
! importcpp: "genodeEnv->ep().manage(*#)".}
! # call a method from the gobal Env, dereferencing
! # thru the Constructible template
!
! type MyNimSessionObj = ref object
! cppImpl: MyNativeSession
! cap: MySessionCapability
! id: SessionId
! # C++ RPC objects are best kept in native
! # reference-counted Nim objects
!
! proc manage(obj: MyNimSessionObj) =
! obj.cppImpl.construct() # call our wrapped constructor
! GC_ref(obj)
! # manually increase the reference count on our session
! # object to prevent the component entrypoint from
! # referencing an RPC object that has been lost and
! # freed from the heap
! obj.cap = obj.cppImpl.manage() # store our capability
!
! proc myCreateSessionProc(id: SessionId): MyNimSessionObj =
! result = new MyNimSessionObj
! # create our object on the heap
! result.manage()
! # construct and manage our RPC object
! result.id = id
! # store the session id from our parent
Procedures for calling Nim code from an RPC object, dissolving
and destructing RPC objects, and managing the session lifetime
are exercises left to the reader.
Updated VirtualBox
==================
Our VirtualBox port got updated from version 5.1.22 to version 5.1.32 in order
to leverage the security updates and improved audio support. Additionally the
boot time of Linux guests got improved by adjusting our custom virtualization
back end.
Libraries and applications
##########################
New trace-logging component
===========================
The new trace-logger component can be used to easily gather, process, and
export different types of tracing data. Furthermore, it marks the next step
towards a user framework that makes access to Genode's manifold tracing
abilities
([https://genode.org/documentation/release-notes/13.08#Light-weight_event_tracing - 13.08],
[https://genode.org/documentation/release-notes/13.11#Improved_event_tracing - 13.11],
[https://genode.org/documentation/release-notes/15.08#Enhanced_tracing_facilities - 15.08])
intuitive and convenient.
The component can filter the available tracing subjects according to session
label policies and thread names. The processing of the tracing data can then
be configured for each selected subject individually, for groups of subjects,
or for all subjects together. The resulting data is exported as log output.
This is an example configuration of the trace logger, which shows the default
value for each attribute (except policy.thread and policy.label):
! <config verbose="no"
! session_ram="10M"
! session_arg_buffer="4K"
! session_parent_levels="0"
! period_sec="5"
! activity="no"
! affinity="no"
! default_policy="null"
! default_buffer="4K">
!
! <policy label="init -> timer" />
! <policy label_suffix=" -> ram_fs" />
! <policy label_prefix="init -> encryption -> "
! thread="worker"
! policy="null"
! buffer="4K" />
! </config>
The most important features so far when it comes to processing the traced
data are:
* Trace CPU activity and affinity ('activity' and 'affinity' attribute),
* Install individual policies for the creation of further tracing data
('policy' attributes) for instance, 'rpc_name' for a log of issued RPC calls),
* Dimensioning the subject-local trace buffers and the frequency of Trace Logger
data examination ('buffer' and 'period' attributes), and
* Configure the session to the Tracing server ('session' attributes).
A comprehensive documentation of the trace-logger component can be found in
_os/src/app/trace_logger/README_. An example of how to use the component is
given through the run script _os/run/trace_logger.run_.
New component for extracting archives
=====================================
The new 'extract' component located at _libports/src/app/extract_ extracts
the content of an arbitrary number of tar.xz archives according to its
configuration. It is used by the depot-download subsystem described in
Section [On-target package installation and deployment]. The component
is accompanied by the run script _libports/run/extract.run_ that illustrates
its use.
New signature-checking tool based on GnuPG
==========================================
The on-target installation of software packages requires a way to verify
cryptographic signatures of downloaded content within a Genode system.
The new 'verify' component located at _ports/src/app/verify_ facilitates the
code of GnuPG to verify detached OpenPGP signatures against public keys.
Since GnuPG depends on libgcrypt and libgpg-error, ports of those libraries
were added to the libports repository. The component comes with the run
script _ports/run/verify.run_ that demonstrates its usage.
Fetchurl component for downloading files
========================================
Fetchurl is a component for downloading files from the network, based
on the curl library. It used to reside in the genode-world repository.
Since it has become a mandatory part of Genode's on-target software
installation mechanism, we have moved it to the _libports_ repository now.
Besides this relocation, fetchurl received a welcome modernization. In
particular, the new version uses the modern socket-fs infrastructure of
the libc instead of relying on the deprecated libc_lwip plugin as a hard-wired
dependency.
New interactive FLIF image viewer
=================================
A simple image viewing application for the FLIF lossless image format was
written from scratch using the FLIF reference decoder library. The viewer can
be used to interactively view a directory of images and supports animation of
GIF-like FLIF files.
Ported 3rd-party software
=========================
With the current release, the following 3rd-party software becomes available
on Genode:
:[https://www.libarchive.org/ - libarchive]: is a library for uncompressing
and extracting various archive formats. It nicely wraps format-specific
libraries like zlib behind a unified and easy-to-use API. The port can
be found in the _libports_ repository.
:[https://lz4.github.io/lz4/ - lz4] and [https://tukaani.org/xz/ - liblzma]:
implement modern compression algorithms as supported by libarchive.
Thanks to Ben Larson for contributing the port of these libraries.
:[https://www.tcl.tk/ - Tcl]: is used as scripting language for various
Genode tools. With the new 'check_abi' tool described in Section
[Automated ABI consistency checks], the Tcl shell 'tclsh' has become
a dependency of the build system. Therefore, we made 'tclsh' available as
noux package. Note, however, that this port comprises solely the
functionality needed for simple scripting.
:[https://flif.info/ - FLIF]: is a library for the Free Lossless Image
Format. Thanks to Emery Hemingway for making it available in the
genode-world repository.
:[https://github.com/json-c/json-c/wiki - JSON-C]:
is a library for processing JSON-formatted data. Thanks to
Johannes Kliemann for contributing the port to the genode-world
repository.
:[https://www.nlnetlabs.nl/projects/ldns/ - Drill (ldns)]:
provides a utility for DNS testing. Thanks to Emery Hemingway for adding it
to the genode-world repository as a side activity of improving Genode's
network stack.
Updated packages for the Noux runtime environment
=================================================
The current release updates the following noux packages: less (version 487),
grep (version 3.1), coreutils (version 8.29), tar (version 1.30), findutils
(version 4.6), which (version 2.21), sed (version 4.4), and bash (version
4.4.18). Thanks to Hinnerk van Bruinehsen for this welcome contribution.
Device drivers
##############
Ethernet-driver for i.MX-based Wandboard
========================================
The current release contains a port of the Linux kernel driver for the
Ethernet card family originally produced by Freescale. We followed our
established approach to tailor an independent device-driver environment (DDE)
for the specific driver. To profit from synergies with the existing drivers of
the _dde_linux_ repository, we took the Linux kernel 4.4.3 as reference.
For now the current version is limited to support the Wandboard Quad as this
is the i.MX-based board that is nightly tested by our infrastructure. The
support of other boards using the same IP core is planned for future releases.
The driver can be found in _dde_linux/src/drivers/nic/fec_. To test the driver,
no further configuration is needed and you can have a look at one of the
automatic network tests, like _lwip.run_, as a reference.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Thanks to Johannes Schlatow from the TU Braunschweig, the support of the
Zynq-7000 boards by our base-hw kernel got extended. It is now possible to use
all CPU cores instead of only the primary one.
Updated Muen separation kernel
==============================
The Muen SK port has been updated to the latest development version 0.9. The
most notable features and improvements are the Crash Audit facility and support
for MirageOS/Solo5 subjects which may be executed alongside Genode/base-hw.
Thanks to this feature, the Muen project has reached a milestone by
self-hosting the [https://muen.sk] website on a Muen system. Currently, the
network driver is provided by a Linux subject but with some work it should be
possible to replace it with a Genode/base-hw nic_drv in the future.
Further details regarding Muen v0.9 can be found in the project's release
notes [https://groups.google.com/forum/#!topic/muen-dev/FPL9sc4yaBE].
Updated seL4 kernel
===================
Our remaining patches regarding UEFI framebuffer support got integrated into
the upstream codebase of the seL4 kernel. Hence, we updated our seL4 port to
the upstream version containing our patches.
Build system and tools
######################
Package management
==================
The package-management tools introduced last year have become a vital part
of Genode's workflow.
:Package management documentation:
[https://genode.org/documentation/developer-resources/package_management]
Prompted by the development of the on-target installation and deployment
mechanism featured in the current release, the tools received the following
refinements:
:Use of tar.xz as archive format: This change significantly reduces the size of
published depot content compared to the previously used tar.gz format.
:Subdirectories for archive versions:
In the original version of the depot layout, archives were named as
'<archive-name>-<version>'. Hence, the depot - in particular the download
location - had directories that grew in two dimensions. First, when new
archives were added. Second, when new versions of existing archives were
added (usually corresponding to Genode's release cycle). In the mid-term,
this would have resulted in a huge number of directory entries, e.g., in the
_src/_ subdirectory. To avoid this problem, the new version uses the scheme
'<archive-name>/<version>' instead. This way, at the _src/_ level, each
archive has one subdirectory (the number of subdirectories corresponds to
the number of archives). Inside the subdirectory, there is one entry per
version.
:Controlled rebuild of binary archives:
When calling the depot/create tool for a binary archive with 'FORCE=1', the
underlying source archives are re-extracted and the binary archive is
rebuilt. This is usually done after local changes in the source tree to
apply version updates to depot archives as needed. However, the implicit
rebuild is superfluous whenever the source-version remains the same. This is
particular inconvenient when re-creating pkg archives that refer to a large
number of src archives. Here, all binaries referenced by the pkg archive are
rebuilt each time. The new 'REBUILD' argument allows the user to skip
superfluous rebuilds in such situations. Normally, 'FORCE=1' implies
'REBUILD=1'. However, by explicitly specifying 'REBUILD=', existing binary
archives whose versions remain unchanged are kept instead of being rebuilt.
Offline validation of XML configurations
========================================
The _tool/run_ tool now automatically checks configurations against
target-specific XML schemes. Each component may define a configuration
scheme-file in its _target.mk_ file as follows:
! CONFIG_XSD = my_config.xsd
When the run tool checks the configuration of an instance of Genode's init
component, it additionally iterates through all start nodes of this
configuration. For each start node, it checks whether the according component
provides a configuration-scheme file and, if so, applies it to the
configuration inside the start node. This is done recursively. I.e., also the
child configurations of a sub-init of a sub-init ... of the top-level init
are covered this way.
Whenever the run tool detects an error in one of the checked configurations,
it stops and points out the location of the error. By now, there exist
configuration schemes for the init, the NIC router, and the trace logger
components. Our intention is that every component that interprets its
configuration will eventually be accompanied by such a scheme - not only to
validate actual configuration input but also to serve as documentation for
users of the component.
Automated ABI consistency checks
================================
In [https://genode.org/documentation/release-notes/17.02#Genode_Application_Binary_Interface - version 17.02],
we introduced a kernel-agnostic ABI, which ultimately paved the ground for
Genode's package management. For the time being, the ABI is not set in stone.
It is expected to evolve for some time until it hopefully approaches ABI
stability in the mid term. Whenever Genode's API changes, the ABI may be
affected. For example, symbol sizes may grow. Until now, side effects on the
ABI had to be curated manually. In practice, however, such side effects are
too easy to miss. Therefore, the current release adds a mandatory ABI checking
step to the build process. A new _tool/check_abi_ tool is invoked whenever a
shared object is built. It reports flaws in the ABI definition (such as
duplicated symbols) as well as inconsistencies between a shared object and its
ABI.

766
doc/release_notes/18-05.txt Normal file
View File

@@ -0,0 +1,766 @@
===============================================
Release notes for the Genode OS Framework 18.05
===============================================
Genode Labs
The driver behind the release 18.05 is the rapid evolution of the Sculpt
general-purpose OS. Following the initial version from February, which was
targeted at early adopters, the new Sculpt for The Curious (TC) introduces a
much more welcoming and empowering user experience (Section
[Sculpt for The Curious]).
It goes without saying that the interactive and dynamic nature of the Sculpt
scenario puts a lot more pressure on Genode's components compared to static
workloads. For example, Sculpt calls for the dynamic adjustment of user-level
network routing, the dynamic detection and management of partitions and file
systems, the support of USB storage devices from diverse vendors, and a way to
adapt the visual appearance to a great variety of screen resolutions. Most
improvements described below are our responses to these challenges.
That said, the release is not short of new features either. E.g., it features
the initial port of OpenJDK's HotSpot VM for executing Java programs on Genode
directly (Section [Java language support]), improves the support for the NXP
i.MX family of SoCs (Section [NXP i.MX SoC]), and enhances the VFS with new
plugins for copy-on-write and the auditing of file accesses
(Section [New VFS plugins]).
The release is complemented by the annual update of the Genode Foundations
book (PDF), which covers the fundamentals of the framework in great detail
(Section [New revision of the Genode Foundations book]).
Sculpt for The Curious
######################
With Sculpt for The Curious (TC), Genode 18.05 features the second revision of
the Sculpt general-purpose OS. Compared to the initial version for Early
Adopters (EA), it features a new interactive system-management component that
streamlines common tasks like the management of storage devices, or
configuring the network connectivity. The highlights of the new version of the
base system image are:
* Live-customization of almost all aspects of the system,
* The ability to install and run software in memory only,
* Hotplugging of USB storage devices,
* New support for NVMe storage devices in addition to SATA disks,
* Interactive network configuration including Wifi connectivity,
* Interactive management and inspection of storage devices and partitions,
* The option to host a complete and customized Sculpt installation on a
USB stick,
* Automated on-demand installation of software packages with visual feedback,
* Scalable fonts that are automatically adjusted to the screen resolution, and
* UEFI boot supported by default.
The base image is extensible by downloadable software packages that may
originate from different sources, safeguarded by cryptographic signatures.
It contains several example subsystems as a starting point:
* Basic GUI components like a window manager, a scalable backdrop, a
font server, and a simple software-rendering demo,
* A light-weight noux runtime for executing command-line-based software
such as GNU coreutils, bash, and vim.
* A package for downloading the installer and a suitable virtual-machine
configuration for Debian Linux,
* VirtualBox running Debian Linux,
* An example for running libretro-based games,
* A disposable VM that runs Firefox on TinyCore Linux, executed either in
VirtualBox or the light-weight Seoul virtual-machine monitor,
* A Qt5-based text editor.
Please refer to the updated
[https://genode.org/documentation/articles/sculpt-tc - Sculpt documentation]
to explore Sculpt TC.
The Sculpt version included with the current release requires the user to
build a boot image by hand. Following the steps described in the
documentation, this procedure takes a few minutes. We plan to provide
downloadable boot images a few weeks down the road once Sculpt TC received
intensive day-to-day testing by the early adopters. Your feedback is very
welcome!
New revision of the Genode Foundations book
###########################################
The "Genode Foundations" book received its annual revision, which reflects
the evolution of the framework over the past year. Specifically, the changes
since the last year's edition are:
: <div class="visualClear"><!-- --></div>
: <p>
: <div style="clear: both; float: left; margin-right:20px;">
: <a class="internal-link" href="https://genode.org">
: <img class="image-inline" src="https://genode.org/documentation/genode-foundations-title.png">
: </a>
: </div>
: </p>
* Changed boot-loader infrastructure on PC hardware
* Package management
* Structural changes of Genode's custom base-hw kernel
* API improvements: Unicode handling, support for XML-based data models,
timeout-handling API
: <div class="visualClear"><!-- --></div>
To examine the changes in detail, please refer to the book's
[https://github.com/nfeske/genode-manual/commits/master - revision history].
Storage infrastructure
######################
VFS library and plugin interface
================================
The VFS (Virtual-File-System) library was expanded to meet new requirements
for the Sculpt scenario. The traditional file-system medium for component
state and configuration sculpting is the *ram_fs* server, but with the
limitation that files stored in the server are ephemeral. Any changes to
the initial state are lost when a system is shut down or the *ram_fs* server
is restarted. Now that persistent storage is usually served by a VFS plugin
hosted by the VFS server, it was a natural progression to introduce a means
for indicating VFS changes with 'File_system' session notifications. To this
end the VFS server was amended to send session notifications, and notification
support was added to the Rump and FatFs VFS plugins, allowing Ext2 and FAT
file-systems to host dynamic component state and configuration information.
Using the VFS for serving font data produced from files stored in the VFS made
it practical to allow VFS plugins to introspect the file system. Plugins now
have the means to access arbitrary paths from the file-system root or they may
host and expose their own internal file systems.
While the core of the VFS library is small compared to contemporaries in other
operating systems, the moment came to promote the VFS from a static to a
shared library. Components that use the C runtime have always loaded the VFS
dynamically as a subsystem of _libc.lib.so_, but native components carried the
bulk of its implementation. The VFS library is now provided as a shared
library and is included with the front-end server in the _src/vfs_ depot
archive. This change affects components that have been rebuilt against the
shared library but do not have their ROM policies updated to allow access to
the _vfs.lib.so_ ROM.
New VFS plugins
===============
File-system introspection has made two additional plugins possible, the *audit*
and *cow* plugins.
The *audit* plugin logs VFS paths as they are accessed to a dedicated LOG
session. This is useful for finding the files required by third-party
components without relying on documentation or auditing source code.
The *cow* plugin emulates copy-on-write behavior by copying the contents of
files lying in a read-only path to a read-write path as they are opened. This
plugin is considered a proof-of-concept and under-performing, but opens a way
of experimenting with seeding user-managed file-systems from immutable
file-system archives.
Plugins of this kind are most appropriately instantiated in the VFS server
with policies to restrict the intended components into paths provided by the
plugins. This prevents a component from escaping the effect of the plugin. An
example of "auditing" a libc component follows:
! <start name="audit_fs">
! <binary name="vfs"/>
! <config>
! <vfs>
! <dir name="data"> <!-- source files -->
! <tar "data.tar"/>
! <ram/>
! </dir>
! <dir name="audit"> <!-- virtual path that captures /data -->
! <audit path="/data"/>
! </dir>
! </vfs>
! <!-- route into virtual audit path -->
! <policy label_suffix="audit" root="/audit" writeable="yes"/>
! </config>
! </start>
!
! <start name="app">
! <config>
! <libc stdout="/log" stderr="/log"/>
! <vfs>
! <log/>
! <fs label="audit"/>
! </vfs>
! </config>
! </start>
Improved disk-partition discovery and access
============================================
The 'part_blk' component, which parses the partition table on a block device
and provides access to each partition through a block session, was extended to
make it easier to implement a management component on top of it. It now
features additional attributes in its report. For one the block size of each
partition as well as the type of the file system on the partition are
reported. The file system probing implementation is minimal and only contains
file systems that are commonly used on Genode systems, i.e., FAT32 and Ext2.
Furthermore, on GPT formatted disks, each partition has an 'expandable'
attribute that contains the number of blocks by which the partition can be
grown. The following exemplary report illustrates the adjustments:
!<partitions type="gpt" total_blocks="500118192" gpt_total="500118125" gpt_used="302254080">
! <partition number="1" name="BIOS boot partition"
! type="21686148-6449-6e6f-744e-656564454649" guid="db0701aa-02ae-474d-92d0-82738bfce5d2"
! start="2048" length="2048" block_size="512"/>
! <partition number="2" name="EFI System"
! type="c12a7328-f81f-11d2-ba4b-00a0c93ec93b" guid="74e43226-2afb-4575-bdda-83bf72f5a6e7"
! start="4096" length="262144" block_size="512" file_system="FAT32"/>
! <partition number="3" name="GENODE"
! type="0fc63daf-8483-4772-8e79-3d69d8477de4" guid="a950091d-87ba-4800-85bf-7b6a58abe6d5"
! start="235147264" length="67108864" block_size="512" file_system="Ext2"
! expandable="197862064"/>
!</partitions>
The heuristics of how the component probes the partition table were also
loosened. Instead of explicitly enabling support for GPT, the component will
now always try to parse the MBR as well as the GPT. It will bail out if both
are considered valid since using GPT/MBR hybrid tables is not supported and it
should be up to the user to make an educated decision. In cases where there is
no partition table, a 'partitions' report of 'type="disk"' will be generated
in which the complete disk is presented as partition number '0'. This is
needed as compatibility fallback for Sculpt EA installations.
Creating and modifying GUID partition tables
============================================
Part of the enhancements of Sculpt TC is the ability to manipulate the block
device used by Sculpt. We implemented a component called 'gpt_write', which
can create and modify a GPT and its entries. It considers alignment
constraints to make better use of 512e devices. It will, however, not perform
any boundary checking. It does not handle overlapping partitions and only when
applying a partition, it makes sure that the partition will fit. The following
configuration illustrates its operation:
!<start name="gpt_write">
! <resource name="RAM" quantum="2M"/>
! <config verbose="yes" initialize="yes" align="4K">
! <actions>
! <add entry="1" type="BIOS" label="GRUB BIOS" start="2048" size="1M"/>
! <add entry="2" type="EFI" label="EFI System" start="4096" size="16M"/>
! <add entry="3" type="Linux" label="GENODE" start="36864" size="128M"/>
! <add type="BDP" label="FAT32 Data" size="max"/>
! <delete entry="1"/>
! <delete label="FAT32 Data"/>
! <modify label="GENODE" new_label="GENODE*" new_size="max"/>
! </actions>
! </config>
!</start>
Please read _repos/gems/src/app/gpt_write/README_ for more detailed information
on how to use the component and feel free to check out the run script
_repos/gems/run/gpt_write.run_.
User-level networking
#####################
NIC router
==========
The NIC router has received major improvements that were mainly motivated by
our daily experience with the Sculpt scenario where the router serves as NAPT
component in front of the virtual machines that host our work OS's. In this
role, it is subject to a permanent load driven by real-world tasks.
Furthermore, it has to have a user interface that makes it a pleasant
experience to deploy in a dynamic environment. This led to our primary goal:
We had to overcome the need to restart the NIC router, and thereby all
components that depend on it, whenever its configuration changes and while
doing so, not to interrupt the communication of its client unnecessarily.
We managed to make the NIC router fully re-configurable at runtime in a way
that it always tries to keep as much state information as possible throughout
the process. This means that network communication going through the NIC
router is not affected by a configuration update unless the configuration
change affects parts that were involved in an existing communication channel.
One prerequisite for this feature was that NIC session clients can connect at
any time to the NIC router regardless of whether there is a matching domain
for the session or not. As long as a session has no domain, the NIC router
does not send any packet to it and drops all packets coming from it. But, at
least, the session and the corresponding client component stay alive, even if
their already assigned domain disappears with a new configuration.
At the uplink, in contrast, the lifetime of the session remains bound to the
lifetime of the domain. The uplink domain-tag received a new attribute
named 'label' (only considered at the domain-tag of the uplink). It denotes
the label of the uplink session. With these two particularities of the uplink
domain, one can now easily switch between different NIC session servers. The
NIC router will close and request the corresponding NIC session with the
current 'label' value if the 'domain' node is removed/added or the label
changes. Thereby, the NIC router can now be used to dynamically switch between
network interfaces like wireless and wired adapters.
Furthermore, we improved the NIC router's ability to handle DNS server
information. Domains can wait for the DNS server info of the DHCP client of
another domain. This is done with the new attribute 'dns_server_from' in the
'<dhcp_server>' tag. Each time the DNS server info of the remote domain
changes, the DHCP server with the 'dns_server_from' attribute will toggle the
link state of each session at its domain. This can be used by clients as a
hint to request their DHCP info anew from the NIC router and thereby receive
the updated DNS server information.
When it comes to protocols, the most notable change is that the NIC router now
also supports routing and NAPT for ICMP. With the new '<icmp>' sub node of the
'<domain>' tag, ICMP routes to other domains can be created. Instead of ports,
the ICMP IDs are used for NAPT. Similar to the 'udp-ports' and 'tcp-ports'
attributes, the size of the ID space for each NAPT client is configured via
the new 'icmp-ids' attribute in the '<nat>' tag.
Last but not least, the following small features were also added to the NIC
router:
:Attribute 'verbose_packets' for the '<config>' and the '<domain>' node:
Toggles the logging of most important protocol header fields globally or
domain-locally. The 'verbose' attribute does not affect this kind of debug
output anymore.
:Report DNS server info:
If the 'config' attribute in the '<report>' node is enabled, the NIC router
will now also report the DNS server info for each domain.
:Attribute 'config_triggers' in the '<report>' node:
Toggles whether the NIC router immediately sends a report whenever the IPv4
configuration of a domain changes, regardless of any timeouts.
:IPv4 point-to-point support:
If a domain receives an IP configuration with a subnet mask of
255.255.255.255 it will switch to point-to-point IPv4 (requires a valid
gateway address at the domain).
:ICMP destination unreachable on non-routable packets:
The NIC router now responds with an ICMP "destination unreachable" packet to
packets that are not routable at an interface with a domain.
For more information, have a look at the _os/src/server/nic_router/README_
file. Examples can be found in the run scripts
_dde_linux/run/nic_router_uplinks.run_,
_libports/run/nic_router_dyn_config.run_, and _os/run/ping_nic_router.run_.
NIC dump
========
The output level of the NIC dump component can now be configured per protocol
by using the protocol names as attributes: 'eth', 'arp', 'ipv4', 'dhcp',
'udp', 'icmp', and 'tcp'.
The available debug levels are:
:no: Do not print out this protocol.
:name: Print only the protocol name.
:default: Print a short summary of the most important header values.
:all: Print all available header values.
Additionally, you can set a default debug level for protocols that are not
configured using the 'default' attribute.
For more information, please refer to _os/src/server/nic_dump/README_.
GUI stack
#########
With Sculpt becoming more and more end-user oriented, Genode's GUI stack came
into focus. It was time to reconsider several interim solutions that worked
well in the past but would not scale up to a modern general-purpose OS. Two
concrete examples are the support of scalable fonts and Unicode characters. In
the past, Genode used to restrict textual output to the Latin-1 character set
and employed pixel-based fonts only. The current release overcomes these
limitations by featuring completely new text-output facilities.
UTF-8 support and improved text rendering
=========================================
The UTF-8 text encoding overcomes the severely limited code-point range of the
ASCII and Latin-1 character sets by representing characters by a varying
number of bytes. Today, UTF-8 is generally considered as the standard encoding
for text. The new UTF-8 decoder at _os/util/utf8.h_ clears the path for
Genode's native GUI components to follow suit. The first beneficiary is
Genode's graphical terminal, which has become able to display Unicode
characters and pass user input as UTF-8-encoded data to its terminal-session
client.
Terminal enhancements
=====================
Speaking of the graphical terminal, the current incarnation got a welcome
overhaul. First, we reduced its complexity by removing obsolete features like
built-in keyboard-layout handling, which are no longer needed when combining
the terminal with our modern input-filter component. Furthermore, the terminal
has become dynamically resizeable, forwarding screen-size changes to the
terminal client. Should the client be a Noux runtime, such a change is
reflected to the running application as a SIG_WINCH signal. The application -
e.g., Vim - responds to the signal by requesting the new terminal size.
Finally, the terminal protocol was changed from 'linux' escape sequences to
'screen' escape sequences in the anticipation of making the terminal more
flexible in the future.
Text rendering
==============
Throughout Genode, many GUI components reused the text-output utilities
of the nitpicker GUI server. These utilities, however, relied on a simple
pixel font format. To make the text output more flexible, nitpicker's text
painter located at _nitpicker_gfx/text_painter.h_ has been replaced by a
completely new implementation that decouples the font format from the
glyph rendering and takes UTF-8 strings as input. In the process, the glyph
rendering got a lot more sophisticated, supporting horizontal sub-pixel
positioning and filtering.
Font-format support
===================
To remove the omnipresent use of fixed-size pixel fonts throughout Genode,
the following new components entered the picture:
First, the new 'ttf_font' library implements nitpicker's font interface by
using the TrueType renderer of the STB single-header library.
Second, the new 'vfs_ttf' VFS plugin uses the 'ttf_font' library to export a
rendered TrueType font as a virtual file system. The various font properties
as well as the actual glyph images become accessible as regular files. This
way, an application that needs to draw text can read the glyph data directly
from its VFS instead of depending on a font-rendering library.
Third, the new 'Vfs_font' utility located at _gems/include/gems/vfs_font.h_
implements nitpicker's font interface by obtaining the glyphs from the
component-local VFS. It is complemented by the 'Cached_font' utility, which
implements an LRU glyph cache.
With this infrastructure in place, several existing GUI components could
be updated, most prominently the graphical terminal and the menu-view
widget-rendering engine. By facilitating the VFS as interface for propagating
glyph data, components no longer need to manage fonts and their configuration
individually. They just access their VFS. When integrating the component into
a scenario, one can decide whether to mount a font-rendering library directly
at the component, or - alternatively - route a file-system session to a
central font server. The latter is just a regular VFS server with the fonts
mounted as pseudo file systems. Since the glyph renderer is a VFS plugin, it
could be replaced by another implementation in the future without touching any
component.
Modernized API for input-event processing
=========================================
Genode's input-session interface changed very little over the years. Even
though it received evolutionary enhancements from time to time, its design
resembled a traditional C-style interface from the medieval era. We found that
the interface left too much room for interpretation. In particular, the meta
data per event type was defined in a rather ad-hoc way, which raised
uncertainties. For example, is a button-press event accompanied with a
positional value or not? To remove these uncertainties, the current release
replaces the 'Input::Event', with a new implementation that facilitates a safe
way of accessing event meta data. Besides this design change, there is one
noteworthy semantic change as well. With the new interface, symbolic character
information are provided along with their corresponding press events rather
than as distinct events, which - according to our practical findings - greatly
simplifies the consumer side of the 'Input::Event' interface.
Improved keyboard-focus handling
================================
The nitpicker GUI server multiplexes one screen among multiple GUI clients in
a secure way. One aspect remained underdeveloped so far, which is the keyboard
focus handling. Nitpicker's 'Session:focus' call previously triggered a one-off
focus change at call time. This focus change did not pass the same code paths
as a focus change triggered by a "focus" ROM update, which led to
inconsistencies.
The new version changes the implementation of 'Session::focus' such that the
relationship of the caller and the focused session is preserved beyond the
call time. Whenever the calling session is focused in the future, the
specified session will receive the focus instead. So 'Session::focus' no
longer represents a single operation but propagates the information about the
inter-session relationship. This information is taken into account whenever
the focus is evaluated regardless of how the change is triggered. This makes
the focus handling in scenarios like the window manager more robust.
Device drivers
##############
NVMe storage devices
====================
Since NVMe devices have become common in contemporary systems, it is time to
provide a driver for such devices on Genode. With this release, we introduce a
component that is able to drive consumer-grade NVMe storage devices, i.e.,
there is no support for namespace management or other enterprise-grade
features. For now, to keep things simple, the driver uses the device in an
old-fashioned way and uses only one I/O queue with at most 128 entries. That
is to say it does not exploit the parallelism necessary to unlock the full
potential of NVMe storage. Nonetheless, it performs well. The following
snippet illustrates its configuration:
!<start name="nvme_drv">
! <resource name="ram" quantum="8M"/>
! <provides><service name="Block"/></provides>
! <config>
! <report namespace="yes"/>
! <policy label_prefix="client1" writeable="yes"/>
! </config>
!</start>
The component will generate a report, which contains all active namespaces, if
reporting is enabled by setting the 'namespace' attribute of the '<report>'
node to 'yes'. A report may look like the following example:
!<controller model="QEMU NVMe Ctrl" serial="FNRD">
! <namespace id="1" block_count="32768" block_size="512"/>
!</controller>
For an example on how to integrate this component, please have a look at the
_repos/os/run/nvme.run_ script.
While implementing the NVMe driver, a new component for testing block-sessions
was used. In contrast to the already existing 'blk_bench' and 'blk_cli'
components, it features a variety of different test patterns, which can be
selected in its configuration and can be used to test a block component more
thoroughly. For more information please refer to
_repos/os/src/app/block_tester/README_
NXP i.MX SoC
============
We extended the Linux kernel driver port for Ethernet cards found in NXP i.MX
SoC, which was introduced in the previous release. Now does it not only
support i.MX6Q SoC based boards like the Wandboard, but the i.MX53 and i.MX6SX
SoC as well. The new driver was successfully tested with the i.MX53 Quick
Start Board and the Nitrogen6 SOLOX. The latter board even contains two
Ethernet cards. But due to technical limitations of the board design, the same
driver instance has to be used for both cards. Currently, the driver is
tweaked to run on different boards via its configuration ROM. When no
configuration is provided, it appropriates the values for successfully
executing on the Wandboard. The following is an example configuration for the
i.MX53:
! <config>
! <card name="fec0" type="fsl,imx25-fec" mii="rmii" irq="87" mmio="0x63fec000"/>
! </config>
As a side effect of enabling networking on the Nitrogen6 SOLOX, support for
GPIO based signals has been added to the framework too. The existing GPIO
driver for i.MX53 SoC got extended to additionally support the i.MX6 family.
There are some known limitations when using different drivers like Ethernet
and SD-card drivers on the Wandboard right now. At the moment, those drivers
adjust clock parameters and I/O pin configurations independently from each
other, which can lead to inconsistencies. We plan to address those issues with
the implementation of a platform driver for the i.MX6 SoC family.
Improved USB-storage driver
===========================
We improved the stability of the USB-storage driver (usb_block_drv) and
made it compatible with a lot more devices as the driver has become a pivotal
ingredient of the Sculpt scenario. Due to the changes, the way the driver
operates has changed. On the one hand, now it first tries to use 10-byte
Command Descriptor Blocks (CDB) in its SCSI layer and will only switch to
16-byte CDBs when it encounters a device whose blocks cannot be completely
accessed via the former descriptor size. On the other hand, because some
tested devices stopped working after issuing a USB device reset, the reset was
made optional. By setting the 'reset_device' attribute in the '<config>' node
to 'yes', the driver is instructed to perform the USB device reset.
Libraries and applications
##########################
Packaged Qt5 framework
======================
We created package recipes for all previously ported Qt5 libraries and their
dependencies and adapted the run scripts accordingly. Please note that the
host tools needed for building Qt applications (moc, rcc, uic) are not built
automatically anymore, but need to be built and installed manually with the
new 'tool/tool_chain_qt5' script.
Java language support
=====================
Over the course of the past year, we started to look into Java support for
Genode with the ultimate goal of porting an existing Java Virtual Machine
(JVM), which translates and executes Java byte code, to Genode. After
investigating possible JVM candidates, it became obvious that
[https://openjdk.java.net - OpenJDK] is the only viable option when looking for
a functional, maintained, feature complete, and open-source Java SDK.
Therefore, we decided upon OpenJDK version 9 and started to port OpenJDK's
HotSpot virtual machine.
In the first step, we followed the approach to enable HotSpot's internal
Just-in Time (JIT) compiler, which translates byte code into machine code and
is the option with the most to offer performance wise. But we also wanted
support for ARM platforms and soon realized, there was almost no JIT compiler
support for ARM other than for Linux. The Linux version is deeply integrated
into the Linux system libraries (e.g., glibc), which makes it very hard to
bring the compiler onto Genode. For example, Genode uses FreeBSD's libc and
that would now have to offer glibc semantics.
After additional research, we found the so-called interpreter version of the
HotSpot VM. This version does not compile byte code, but interprets and
emulates the code at runtime. It is of course slower than the JIT compiler
version, but also machine-architecture independent, so the same HotSpot VM can
be compiled for x86 and ARM platforms. With the JVM running on Genode, we
added networking and file-system access support via Genode's VFS layer. Note,
there is no graphical toolkit support as of now, but most standard library
classes should work. Also, the byte code has to be compiled on a different
host system (e.g., Linux, *BSD) as of now, since we did not bring the Java
compiler to Genode.
To give Java a spin, a run script can be found under _ports/run/java.run_.
Ada language support
====================
Support for components and libraries written in the Ada/SPARK programming
language experienced a rework with the final goal of seamless integration with
the base framework. We added a new _ada_ library, which contains a (currently
minimal) runtime taken from the sources of our GCC port and thus is always
consistent with the tool chain in use. It is built as a shared library
_ada.lib.so_ that needs to be added to the list of boot modules.
The example in _libports/src/test/ada_ showcases the implementation of an Ada
component using a custom library _test-ada_, which is also implemented in Ada.
Seoul VMM on NOVA
=================
The Seoul/Vancouver VMM - introduced to Genode with release 11.11 - received
some renovations to be able to run recent Linux VMs. Namely the output of the
guest during early boot is now visible and the network models got revised.
Additionally, the Seoul VMM has been packaged and can be used in Sculpt.
Ported software
===============
The [https://dnsprivacy.org/wiki/display/DP/DNS+Privacy+Daemon+-+Stubby - Stubby]
DNS daemon has been ported to begin experimentations with DNS as a native
service. There is a tendency for DNS configuration frameworks to diverge
between operating systems and releases, an inconvenience that is magnified
when maintaining virtual machines. Name-server configuration via DHCP has been
the only constant, so hosting DNS natively and configuring virtual-machines
with the *nic_router* DHCP server presents itself as a viable solution to the
guest resolver quagmire. Expect DNS services in later Sculpt releases.
Platforms
#########
Accessing PCI via ECAM/MMCONF
=============================
The platform driver on x86 is trusted with guarding access to PCI
devices. Up to now, I/O ports have been used to configure the PCI subsystem.
On modern x86 architectures, PCI devices can be configured by using Memory
Mapped I/O (MMIO). This method was introduced with PCI Express and is called
Enhanced Configuration Access Mechanism (ECAM). For Each PCI device a separate
4 KiB MMIO page exists to serves as the configuration interface between OS and
PCI device.
The exact location of all the 4K MMIO pages of the PCI devices is machine
specific and must be determined during the bootstrap phase. The ACPI driver on
Genode is in charge of this procedure and reports the location of the
ECAM/MMCONF region to the platform driver via the 'acpi' ROM.
Besides using a modern PCI interface, switching to ECAM/MMCON served to ease
the execution of Genode/hw on top of the Muen separation kernel.
Kernel-agnostic platform-information handling
=============================================
Up to now, special kernel-specific information was propagated to components
such as Virtualbox, the Seoul VMM, and the timer by reusing the
kernel-provided data structures. For Genode/NOVA, the hypervisor info page
(HIP) was exported as an ordinary Genode ROM. With the rise of Sculpt and the
packaging of components in a - as far as possible - kernel-independent way,
the propagation of kernel-specific information became a stumbling block.
With this release we abandon the 'hypervisor_info_page' ROM of Genode/NOVA and
replace it with a Genode ROM called 'platform_info'. The 'platform_info' ROM
is planned to contain solely information about the host hardware, which may
not be gathered otherwise by Genode components. In the current state it
contains information required by VMMs, namely whether AMD SVM or Intel VMX is
available and usable. Additionally, the ROM contains information about the
frequency of the time stamp counter.
Updated seL4 kernel to version 9.0.1
====================================
Thanks to Hinnerk van Bruinehsen, the seL4 version used by Genode has been
updated to 9.0.1.
Updated Muen separation kernel
==============================
With the addition of memory-mapped access to the PCI config-space in Genode,
base-hw subjects on Muen now only see the effectively assigned physical
devices. This makes it possible to run Genode in parallel with other subjects
and to pass-through different PCI devices for each instance.
The Muen update also brings a much simplified subject info structure plus some
tweaks to the Muen system policy XML format to facilitate easier integration
of new hardware platform specifications.
Build system and tools
######################
Validating 3rd-party code downloads via SHA256
==============================================
This release removes support for verifying source code of third-party ports
with the SHA1 hash algorithm. Last year, SHA1 was banished as a credible
cryptographic hash function after the demonstration of a full collision
attack. Since the
[https://genode.org/documentation/release-notes/14.05 - 14.05 release],
port files have been verified using SHA1, this release replaces all file
digests with SHA256 digests. Any port definitions maintained in external
repositories are required to make these replacements as well. No collisions
have been discovered against source code archives but nonetheless there is an
obligation to widen our margin of safety.
Creating GPT-based disk images by default
=========================================
Up to now Genode's run tool was able to create x86 bootable images in three
flavours:
* Either as ISO bootable by BIOS legacy - 'image/iso', or as
* GPT partitioned disk image only bootable by UEFI - 'image/uefi', or as
* MBR partitioned disk image only bootable by BIOS legacy - 'image/disk'.
With Sculpt came the demand to have a single image type that is in principle
bootable by both UEFI and BIOS legacy. Additionally with Sculpt, we began to
prefer working with GPT partitioned devices.
In the light of the new demands, we changed the 'image/disk' run tool support
to create a GPT partitioned disk image bootable by a legacy BIOS and by UEFI.

1075
doc/release_notes/18-08.txt Normal file
View File

File diff suppressed because it is too large Load Diff

1107
doc/release_notes/18-11.txt Normal file
View File

File diff suppressed because it is too large Load Diff

836
doc/release_notes/19-02.txt Normal file
View File

@@ -0,0 +1,836 @@
===============================================
Release notes for the Genode OS Framework 19.02
===============================================
Genode Labs
In our [https://genode.org/about/road-map - road map] for 2019, we stated
our goal to make Genode more relevant and appealing for a broader community.
The current release takes a big leap towards that goal: It opens up Sculpt
OS for 3rd-party software providers, introduces a federated blogging
platform about Genode-related topics, and makes the world's
[https://www.tiobe.com/tiobe-index/ - most popular] programming language
(Java) available to Genode users.
With the 4th stage of the evolution of Sculpt OS - themed as "community
experience" (CE) - Genode's custom general-purpose OS introduces a novel and
simple way for users to discover software, and for software providers to
announce software. There is no middle man like an app store or a distribution!
We hope that this federated model of software provisioning and deployment will
have a vitalizing effect on the community around Genode. On a practical level,
the interactivity of the new version is a playful and fun experience. Section
[Sculpt OS as a community experience (CE)] gives a rough overview about
*Sculpt CE*. A ready-to-use disk image will be released mid of March.
When speaking of "software providers", we don't think of anonymous
repositories. Software - and Free Software in particular - is developed and
provided by individuals after all. So in our federated way of software
distribution, we want to highlight this individuality. For this reason, we
launched a new blogging platform for Genode-related stories called
[https://genodians.org - Genodians.org], which gives everyone who is
enthusiastic about Genode - users and developers alike - a platform to express
ideas, announce software, or share practical tips and tricks. As explained in
the [https://genodians.org/nfeske/2019-01-07-welcome - initial posting],
*Genodians.org* is - in the spirit of Sculpt's software distribution model -
also organized in a federated way. The content is hosted and remains
completely under control by the respective authors. The website merely
aggregates and presents the content. It goes without saying that Genodians.org
is based on Genode. The system image that contains the entire web appliance -
from the kernel over the content management to the web server - is contained
in a 8 MiB disk image.
Section [Genodians.org as a showcase of a Genode-based web appliance] goes
into more detail.
We identified the support of popular programming languages as one key aspect
to attract a broader community. With the current release, our port of
*OpenJDK* (Section [Java]) for both 64-bit x86 and 32-bit ARM has reached a
mature state suitable for the creation of web services. This paves the way
towards quite exciting new system creations, like the *Boot2Java* system
presented in Section [Showcase of a Java-based network appliance].
On the account of programming-language support, we are happy to announce
a vastly improved runtime support for *Ada* and *SPARK* (Section
[Ada and SPARK] as well as the initial support for OCaml (Section [OCaml]).
Besides the work on shiny features, the current release cycle included a
profound *spring cleanup* described in Section
[Base framework and OS-level infrastructure].
It thereby finalizes the huge API modernization initiated almost three years
ago.
Sculpt OS as a community experience (CE)
########################################
When we laid out the road map for Sculpt OS a little more than one year ago,
we envisioned four stages of development. Sculpt EA was geared towards
die-hard early adopters facilitating the live editing of the system using a
text editor. Sculpt TC was targeted at "the curious" and included a graphical
user interface for common administrative tasks. Sculpt VC introduced the
visual composition of the system using an interactive graph. Even though
Sculpt already allowed for the installation of components from different
software providers at this stage, it offered no simple means to discover
software and the installation still required the manual editing of "launcher"
text files.
The final - community experience - stage of the plan ought to foster the
federated provisioning of software. Software providers should have the ability
to announce new packages. Conversely, users should be able to "subscribe" to
such announcements, similar how one would subscribe to an RSS feed. Once
software is discovered in this way, it should take only a few clicks to
install and integrate it into the running Sculpt system. No command-line
interface should stand in the way of discovery.
Sculpt CE is the realization of this idea.
For Sculpt users, the discovery starts at the '+' menu. In contrast to Sculpt
VC where the menu offered a rather long list of components to choose from, the
menu now offers nothing to be excited about.
[image sculpt_ce_menu]
However, at the very bottom, there is a new menu entry "Depot ...", which
leads to a sub menu with a list of enabled software providers. If connected to
the network, it also shows an entry named "Selection ...".
[image sculpt_ce_menu_depot]
The "Selection" sub menu allows the user to enable software providers.
[image sculpt_ce_selection]
When clicking on one of the checkboxes, the package index of the provider is
downloaded and the checkbox is highlighted. In the example, "nfeske" is
already enabled.
By clicking on the top-left triangle, one can always go one menu up. Back in
the depot menu, selecting a software provider will now display the index of
available packages. When selecting a new package, it can be installed via a
single click:
[image sculpt_ce_install]
Once the package is installed, the menu takes the user to a dialog for
integrating the package as a new component into the running Sculpt system. For
each resource required by the package, the user can decide how to connect it.
For example, the file system to be mounted at _/config/_ of a fresh
noux-system instance can be routed to any file system service the user wishes.
The noux-system won't know which file system is selected. It will just work
with it.
[image sculpt_ce_routes]
As soon as all routes are defined, the dialog presents a button for adding the
component to the system.
Using this interactive work flow, the discovery of software and its
integration becomes a rather playful process.
To make a once composed system permanent, one can use the inspect window to
copy the _/config/managed/deploy_ file to the _config/19.02/_ directory of your
Genode partition. This way, the deployment configuration will take effect
immediately at boot time.
Genode 19.02 comes with Sculpt CE included. As usual, we will take a bit
of time following the release for thorough testing and refining before
announcing an updated disk image mid March. We would greatly appreciate
your feedback during this phase! For baking a fresh Sculpt image, please
refer to the
[https://genode.org/documentation/articles/sculpt-vc - documentation]
of Sculpt VC, but using the Git tag '19.02' instead of 'sculpt_vc'.
Announcing software packages
----------------------------
The community experience of Sculpt CE will ultimately depend on the
participation of software providers. To become listed in the selection dialog
mentioned above, you may consider including your public key and download
location at the Genode repository at
[https://github.com/genodelabs/genode/tree/master/depot - /depot/].
For the announcement of packages, a software provider can publish so-called
"index" files for a particular Sculpt version in the software provider's
depot. E.g., the index of packages supported on Sculpt 19.02 would be located
at _index/19.02_ within the depot. Like any other depot content, index files
are digitally signed. An example index for the software provider "nfeske"
would look as follows:
! <index>
! <index name="GUI">
! <pkg path="nfeske/pkg/sticks_blue_backdrop/2019-02-22"
! info="default desktop background"/>
! <pkg path="nfeske/pkg/themed_wm/2019-02-26"
! info="ready-to-use window manager"/>
! </index>
! <pkg path="nfeske/pkg/nano3d/2019-02-22"
! info="simple software-rendering demo"/>
! </index>
Each '<pkg>' node refers to a package with a concrete version and a short
description. By nesting '<index>' nodes, software categories can be defined.
Index files can be published like any other depot content using the
_depot/publish_ tool, which takes care about compressing and digitally signing
the published information:
! ./tool/depot/publish nfeske/index/19.02
To ease the updating of the index with current package versions, the
_sculpt.run_ script creates the _depot/index/<version>_ file from the input
found in _repos/gems/run/sculpt/index_. The latter file is void of any version
numbers.
For any questions about the process, please consult the Genode
[https://genode.org/community/mailing-lists - mailing list].
Showcase of a Java-based network appliance
##########################################
With OpenJDK in good shape (Section [Java]), we have created a Genode scenario
that demonstrates JVM's ability to execute well on embedded hardware. As
target platform, we choose an ARM (i.MX 6) based SoC with two integrated
network interface controllers. In the scenario, the Genode system boots
directly into a Java application, which in turn spawns two HTTP server
instances where each instance communicates through a dedicated NIC. Both
server instances run as one Java program.
[image java_nic_filter]
The Java application (a JAR file) is loaded from the board's SD card, and
therefore, can easily be replaced. If you are interested in Java and ARM SoC,
the full details and a step by step instruction can be found at our
[https://genodians.org/ssumpf/2019-02-27-java-19-02 - Genodians.org] site.
Genodians.org as a showcase of a Genode-based web appliance
###########################################################
[https://genodians.org - Genodians.org] is our take on a federated blogging
platform about Genode-related topics. It does not host the actual content
but rather aggregates content hosted elsewhere.
[image genodians]
The site periodically fetches content in the form of zip archives containing
raw text and PNG images, extracts the archives, and transforms the text into
HTML using a custom static site generator. The result of the transformation
process is served by the lighttpd web server.
This system is based on Genode and can be found here:
:Genodians.org repository:
[https://github.com/genodelabs/genodians.org]
A few noteworthy technical points about the site:
* The periodic process of downloading, extracting, and transforming content
is realized via the 'fetchurl', 'extract', and 'sequence' components.
* The textual content uses the markup of the
[https://github.com/nfeske/gosh - GOSH]
text processing tool, which is similar to
[https://daringfireball.net/projects/markdown/ - Markdown].
* The custom static site generator consists of a plain makefile and a few GOSH
style files. The makefile is executed within a noux instance, which is
Genode's custom Unix runtime environment. It is re-spawned for each
iteration. As GOSH is written in Tcl, the tclsh is used within the noux
environment.
* The lighttpd web server uses a statically supplied SSL certificate.
* The most time-consuming part of creating Genodians.org was the CSS
definition.
* The entire system image is about *8 MiB* in size. It includes the following
ingredients:
* Kernel (e.g., NOVA),
* Basic Genode components such as init, NIC router, the VFS server,
* Network driver (based on iPXE),
* Linux TCP/IP stack as a library,
* Curl, libssh, libssl, libcrypto, lighttpd
(for downloading and serving content),
* libarchive, zlib, liblzma (for extracting the downloaded content)
* Noux, coreutils (stripped down), bash, GNU make, and tclsh (for
transforming text into HTML)
More information about the site's inner working will be posted in a series
of articles - guess where? - at Genodians.org!
Base framework and OS-level infrastructure
##########################################
Removal of deprecated APIs
==========================
Almost three years ago -
with [https://genode.org/documentation/release-notes/16.05 - version 16.05] -
we started the transition to Genode's modern API. One year later - in
[https://genode.org/documentation/release-notes/17.05#Completed_component_transition_to_the_modern_API - version 17.05],
we announced the completion of this transition but we retained the deprecated
APIs to accommodate Genode users that picked up the new API only gradually.
With the current release, we finally drop the deprecated APIs along with a
couple of other legacies:
* The _base/timed_semaphore.h_ has been removed. In hindsight, officially
providing this utility was a big mistake because it lured developers into
a wrong direction. In fact, we found that there is no legitimate use of it
when a component is designed in a clean way. If a component relies on a
timed semaphore, it should better be redesigned. There are two noteworthy
places where a timed semaphore is still used as a band-aid solution:
the 'pthread_cond_wait' implementation of the libc, and DDE for rump
kernels. Those places now host a private copy of the timed semaphore, but
should ultimately be reworked.
* The header _base/printf.h_ has been removed along with the log back end
for 'printf'. The 'Console' with the format-string parser is still there
along with 'snprintf.h' because the latter is still used at a few places,
most prominently the 'Connection' classes.
* The notion of a 'Ram_session' exists no more since the former RAM-session
interface was merged into the PD-session interface in
[https://genode.org/documentation/release-notes/16.05#Consolidation_of_core_s_SIGNAL__CAP__RM__and_PD_services - version 16.05].
Still, the types were preserved (by typedefs to 'Pd_session') to keep up
API compatibility. Those last traces of 'Ram_session' are gone now. The
'Env::ram()' accessor returns the 'Ram_allocator' interface, which is a
subset of the 'Pd_session' interface.
* The use of the global 'Genode::env()' accessor function is not possible
anymore.
* The old 'Child_policy::resolve_session_request' interface that returned
a 'Service' instead of a 'Route' has been removed.
* Boolean accessor methods are no longer prefixed with 'is_'. E.g., instead
of 'is_valid()', use 'valid()'.
* All connection constructors need the 'Env' as argument.
* The 'Reporter' constructor needs an 'Env' argument now because the
reporter creates a report connection.
* The old notion of 'Signal_dispatcher' is gone. For receiving
asynchronous notifications, the 'Signal_handler' interface must be used.
* Transitional headers like _os/server.h_, _cap_session/_,
_volatile_object.h_, _os/attached*_dataspace.h_, and
_signal_rpc_dispatcher.h_ have been removed.
* The distinction between 'Thread_state' and 'Thread_state_base' does
not exist anymore. Only 'Thread_state' prevails.
* The header _cpu_thread/capability.h_ along with the type definition of
'Cpu_thread_capability' has been removed. Use the type
'Thread_capability' defined in cpu_session/cpu_session.h instead.
* The _os/ram_session_guard.h_ has been removed.
Use 'Constrained_ram_allocator' provided by _base/ram_allocator.h_ instead.
Source-tree reorganization
==========================
Timer moved from os to base repository
--------------------------------------
Traditionally, the user-level timer was hosted at the os repository at
_drivers/timer/_. However, since the timer and timeout handling have become
part of the base library (the dynamic linker), the component naturally belongs
to the base repository. It is now located at _base/src/timer/_ and
_base-<kernel>/src/timer/_ respectively.
Note that this change affects include paths for the former
_include/os/timer/_, _include/os/alarm.h_, and _include/os/duration.h_
headers. Those can now be found in _include/base/_.
Consistent naming of block components
-------------------------------------
Regarding the naming of files and APIs, Genode follows the convention of
avoiding abbreviations. Most components follow this convention, with the block
servers being the exception to the rule. Those were named "part_blk" or
"rom_blk". With the current release, we removed this inconsistency by renaming
those offenders, changing "blk" to "block". Closely related, abbreviations
like "cli" and "srv" have been replaced by "client" and "server".
Improved API safety
===================
XML-parsing API
---------------
Genode consistently uses XML for component configurations and for reports
generated by components. The latter are often consumed by other components.
This puts the XML parser into a prominent position. Genode's XML parser
comes in the form of the 'Xml_node' class. Since it was introduced before the
age of modern C++, it offers several risky "C-ish" methods, in particular
accessors that return pointers, raising memory-safety concerns. To promote a
safe programming style, the following parts of the interface are subject to
change now:
* The 'Xml_node::addr', 'Xml_node::content_addr', and 'Xml_node::content_base'
accessors will be removed because it is all too easy to store the returned
pointers and forget about the lifetime of the originating 'Xml_node' object.
Fortunately, in practice, those methods are rarely used because information
is typically represented in attributes, not as node content. However, to
still support the access to the raw content, the new
'Xml_node::with_raw_node', 'Xml_node::with_raw_content', and
'Xml_attribute::with_raw_value' methods call a functor taking the raw byte
buffer and size as arguments. This way, the lifetime of the pointer is
naturally bound to the scope of the functor.
* The new 'with_sub_node' method calls a functor with the specified
sub node as argument and thereby reduces the need for the traditional
'Xml_node::sub_node' method, which returns an 'Xml_node'.
The latter is risky because an 'Xml_node' contains a pointer to the actual
data. In contrast, the lifetime of the 'Xml_node' processed via the new
'Xml_node::with_sub_node' is naturally bound to the scope of the passed
functor.
* The 'Xml_attribute::value' and 'Xml_node::value' methods now take an
argument of type 'T &out' instead of 'T *out', which eliminates the
uncertainty of a possible nullptr argument.
The original interface will still be available for a while but it will
eventually be removed.
Simplified session-policy handling
----------------------------------
Most server components make use of the 'Session_policy' utility for selecting
a client policy depending on the session label. However, the pattern of its
use remained a bit inconsistent across components, in particular the handling
of the case where no policy could be found. Some components outright
denied the session where others implemented a fallback to a built-in default
policy. This inconsistency is now removed.
Following the principle of deny-by-default the absence of a matching policy
denies the session request. Since the 'Session_policy::No_policy_defined'
exception is a typedef to the 'Genode::Service_denied' exception, a server
does not need to explicitly handle it, which simplifies the implementation.
With this change, the 'No_policy_defined' case is always an error case. Hence,
the new version of 'Session_policy' prints a error message, which relieves the
server developer from implementing diagnostics in the server code.
As a consequence of this change, scenarios that used to rely on the
policy-fallback approach of some servers - notably the NIC bridge, window
manager, the window decorators - need a slight adaption: To enable the
fallback to a default policy, a '<default-policy>' node must be explicitly
specified in the server's configuration.
Removed pointers from Genode::Fifo interface
--------------------------------------------
To make the use of the 'Genode::Fifo' data structure more safe, all methods
that return pointers have been replaced by methods that call a functor with
a reference as argument.
New server-side block-request stream API
----------------------------------------
The current block-component API (_os/include/block/_) was designed at a time
long before Genode's modern component API was introduced. Back then, the use
of blocking calls was prevalent. Today, we design components to work
asynchronously. The original block-server API was successively enhanced to
allow the implementation of asynchronous block servers but it remained "upside
down" while growing more complicated than it should be.
To simplify the implementation and verification of block servers, the current
release introduces a modern API that will eventually replace the original
block-component API. The new API is called *block-request stream* and can be
found at
[https://github.com/genodelabs/genode/blob/master/repos/os/include/block/request_stream.h - os/include/block/request_stream.h].
It is designed with the following considerations:
* It anticipates the asynchronous operation of block servers by default.
Using the new API, such servers - in particular block-device drivers - can
be implemented as state machines triggered by client requests and device
interrupts.
* It reinforces the memory safety of the server code by not returning any
pointers or references.
* It relieves the server developers from handling special cases (like a
congested acknowledgement queue) while being flexible enough to accommodate
different categories of components like drivers, resource multiplexers
(part_block), and bump-in-the-wire components in a natural way.
* It naturally supports the batching of requests as well as zero-copy
(device DMA directly into the client's communication buffer).
The use of the new API is illustrated by the artificial test at
[https://github.com/genodelabs/genode/blob/master/repos/os/src/test/block_request_stream/main.cc - os/src/test/block_request_stream/].
We plan to successively migrate all existing block servers to this new API
and will remove the traditional block-component API eventually.
GUI stack
=========
Motivated by our work on Sculpt as described in Section
[Sculpt OS as a community experience (CE)], Genode's GUI stack received
the following improvements:
Window management
~~~~~~~~~~~~~~~~~
To improve the visual appearance of Sculpt's administrative GUI, the *themed*
*decorator* was enhanced with an option to disable decorations based on the
window label. By setting the 'decoration' attribute of a '<policy>' node to
"no", matching windows appear without any border, which is desirable for
Sculpt's component graph.
Furthermore, the themed decorator accepts the '<policy>' attribute
'motion=<number>'. The default value is 0. If a value higher than 0 is
specified, window-geometry changes are applied as an animation where the
<number> denotes the number of animation steps. This feature is used for
smoothing the placement of Sculpt's component graph.
The *motif* *decorator* - which is a nice alternative to the themed decorator -
is now giving visual feedback to mouse clicks. For example, while the user
drags a window by clicking on the window title, the title bar appears as
pressed, which creates an improved sense of responsiveness.
Unified shape-report routing
----------------------------
Applications propagate custom mouse-cursor shapes by issuing "shape" reports
towards the pointer component. The pointer component correlates the session
labels of the incoming shape reports with the label of nitpicker's currently
hovered client. Hence, an application's shape report session should take the
same route as its nitpicker session.
The presence of the window manager as an intermediary of the nitpicker session
breaks this rule. Consequently, rather awkward label-rewriting magic was
required when routing shape reports originating from windowed applications. To
make the shape reporting more natural, the window manager has become able to
proxy shape reports on behalf of its clients. When routing a shape through the
window manager, the labels of both the nitpicker sessions as well as the shape
report sessions contain the intermediary "wm ->" part.
Unicode support for the graphical terminal
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The terminal has been changed to represent characters as 16-bit codepoints
internally. It thereby became able to display a much larger variety of
international characters when using a suitable font. Unicodes are sent through
the terminal session via bursts of UTF-8 bytes now.
As a collateral change, the generic 'Codepoint' class became printable via
Genode's 'Output' interface. This way, a codepoint can easily be serialized
into UTF-8 bytes.
Programming languages
#####################
Ada and SPARK
=============
_The work described within this section was contributed by_
_[https://componolit.com - Componolit]. Thanks to Alexander Senier_
_and Johannes Kliemann for the fantastic collaboration!_
The integration of SPARK/Ada programs was improved greatly and the Ada runtime
was renamed to 'spark'. This emphasizes its main purpose: trusted components
that can be formally verified. A more feature-rich runtime based on 'libc' was
added to genode-world under the name 'ada'.
Build system integration
------------------------
Up until now, the (deprecated) 'gnatmake' tool was used to build Ada object
files. This was unfavorable for two reasons: First, multiple invocations of
gnatmake would sporadically corrupt the compiler-generated Ada linker files
('.ali') and break parallel builds. Second, Ada dependency information did not
get propagated into Genode's build system such that certain source changes
failed to trigger a rebuild.
For these reasons the 'gnatmake' tool has been dropped in favor of regular
compiler calls as done for all other languages. To facilitate consistent
rebuilds on source-code changes, the
[https://github.com/Componolit/ali2dep - ali2dep] tool was created. From an
'.ali' file, it produces '.d' files suitable for direct inclusion into
Genode's build system.
Until the integration into the toolchain, 'ali2dep' support needs to be
enabled through the 'CUSTOM_ALI2DEP' variable (absolute path or command name
if the command is in 'PATH'). By default, a warning about the absence of the
tool is emitted and dependency information is not generated. You can add this
variable to the 'etc/tools.conf' of your build directory as follows:
! CUSTOM_ALI2DEP = /path/to/ali2dep
Elaboration code
----------------
Previously, Ada programs that required elaboration code to be run were
unsupported on Genode. With this release, Ada programs are bound using
'gnatbind', which results in elaboration code being generated. An additional
benefit comes in the form of proper error messages at compile time if, for
example, source code is missing or outdated.
To run an Ada main program with elaboration, calls to 'adainit()' and
'adafinal()', generated by the binder, need to be added to your
component-construction code:
! extern "C" void _ada_main(void);
! extern "C" void adainit();
! extern "C" void adafinal();
!
! void Component::construct(Genode::Env &env)
! {
! adainit();
! _ada_main();
! adafinal();
! env.parent().exit(0);
! }
Note, that the name of the Ada main program ('_ada_main()') in this example
depends on the name of your main procedure.
Debug output
------------
Support for 'GNAT.IO' was added to the runtime. 'GNAT.IO' is a stripped-down
text I/O facility, which we map to a terminal session on Genode. Only output
is supported at the moment. A pointer to a terminal session has to be provided
to the ADA runtime in order to use 'GNAT.IO'. Please refer to
_repos/libports/src/test/gnatio/_ as an example.
Unit testing
------------
To facilitate test-driven development of Ada components, the
[https://www.adacore.com/documentation/aunit-cookbook - AUnit] unit testing
framework was ported to Genode. It is located in the genode-world repository
whereas a usage example can be found at 'src/test/aunit'. Note that the full
Ada runtime ('ada') is required as the framework uses features not present in
the SPARK runtime.
Java
====
Since Genode release
[https://genode.org/documentation/release-notes/18.11#Java_language_runtime - 18.11],
we continued our effort to enable the just-in-time (JIT) compiler for OpenJDK.
We are happy to announce that we were able to achieve this goal. The JIT
compiler is now enabled as default for both x86 (64 bit) and ARM (32 bit),
which significantly improves the performance of the Java virtual machine for
these architectures.
Additionally, we even further improved JVM's performance by taking advantage
of more aggressive compiler optimizations. This triggered some unidentified
bugs and led to a greatly enhanced stability of OpenJDK.
For a small hello world example we offer a simple run script:
! make run/java
For a more complex scenario please refer to Section
[Showcase of a Java-based network appliance].
Nim
===
Support for Nim has been removed from the Genode build system to encourage
building Nim components out-of-tree using the Nimble package manager.
The compatibility of the base system with the Nim runtime will be monitored
and improved regardless of this change.
OCaml
=====
A proof-of-concept port of the OCaml bytecode interpreter has been placed
in the world repository. This allows simple programs that are compiled to the
bytecode instructions to be executed and is a first step in supporting the
OCaml language. This interpreter does not yet include the standard library.
Therefore only language primitives are available. Porting the standard library
appears to simply be a matter of defining a build process. The greatest
hurdle to porting existing OCaml applications would seem to be finding a path
into the workflow of the OPAM package tooling.
Libraries and applications
##########################
New utility for taking screenshots
==================================
The flif_caputure screenshot utility has been added to the world repository.
This utility implements a framebuffer and input service which it proxies to
its parent. When the utility observes the *PrtSc* key, it captures the content
of the framebuffer and writes it to the file system in the FLIF image format,
FLIF being chosen primarily for its uncomplicated API. To use the utility in
practice, the common case would be to run it as a client of the default window
manager with a second window manager stack running as a client of the capture
utility. This allows capture behavior and scope to remain simple and explicit.
The background story behind the tool is covered by a dedicated
[https://genodians.org/ehmry/2019-01-31-flif_capture - posting] at Genodians.org.
Growing use of the Genode-World repository
==========================================
The [https://github.com/genodelabs/genode-world - Genode-World] repository
contains software that is not strictly part of the official Genode OS
framework but rather supplemental. In particular, it contains ports of
3rd-party software to Genode. The pool of ported software is steadily growing,
which moves the world repository more and more into the spotlight of Genode
users. For example, among a variety of games and experimental components, our
port of OpenJDK ([Java]) is also hosted there. To acknowledge the growing
importance of the world repository, we added the building of all world depot
archives to our nightly build tests.
With this baseline of quality assurance in place, it was a good time to move
supplemental software such as Dosbox, FUSE, libav, libSDL, and its companion
libraries from the Genode repository to the Genode-World repository. Speaking
of libSDL, as part of the curation of the world content, the library back end
of libSDL has been changed to the direct use of the nitpicker session
interface instead of the lower-level framebuffer and input interfaces. Thanks
to this change, many scenarios could be greatly simplified, in particular
packages designated for the use in Sculpt OS.
Updated or removed 3rd-party software
=====================================
The following 3rd-party software received an update:
* OpenSSL updated to version 1.0.2q with 'SSL_CONF_*' enabled,
as needed by lighttpd's mod_openssl.
* Lighttpd updated to version 1.4.52, with TLS enabled.
* libpng updated to version 1.6.36
* jbig2dec updated to version 0.15
Removal of VirtualBox 4
-----------------------
All regular users of Genode's VirtualBox port migrated to version 5 a long
time ago. Version 4 was still maintained because this is the only version
supported on the Muen separation kernel. However, since the focus of the Muen
developers recently moved towards the use of nested virtualization, our
version of VirtualBox 4 is no longer vital for Muen. So we could remove it.
Platforms
#########
Board support for i.MX6 Quad Sabrelite and Nitrogen6 SoloX
==========================================================
In 2018, we extended our device-driver support for NXP i.MX 5 and 6 based ARM
platforms. This year, we continue our commitment to this platform with
additional support for the NXP reference board i.MX6 Quad Sabrelite and the
Nitrogen6 SoloX, which features dual Gigabit Ethernet.
Resizeable virtual framebuffer on Linux
=======================================
When executing Genode on Linux, we use a libSDL-based pseudo driver for
running graphical scenarios. This *fb_sdl* server has now been enhanced with
the ability to resize the SDL window. Such a resize event is translated into
Genode's framebuffer resize protocol. Thereby, the resizing of the host window
looks like a mode change to Genode's GUI stack. This greatly eases the testing
of the mode-change handling of components like the nitpicker GUI server.
Tooling and build system
########################
Enforced 'override' annotations
-------------------------------
The '-Wsuggest-override' warning complains about implementations of virtual
functions that lack the override keyword. If the implementation of a virtual
method is marked as 'override' the compiler checks for a matching virtual
method in the base class. If there is no such method, the implementation
unexpectedly diverged from the interface.
An interface may change over time, which is especially troublesome when it
contains a default implementation of a virtual method. Without 'override'
annotations, the compiler will silently add the outdated implementation of the
derived class as an overload of the interface's default implementation, which
introduces a subtle but potentially serious bug.
To rule out such bugs in the future, we made 'override' annotations mandatory
when using the default strict warning level.
Generalized use of the depot by run scripts
-------------------------------------------
With more and more run scripts using archives out of Genode's depot, we gained
experience with typical usage patterns. In particular, we found the universal
use of the configurable depot user preferable over the hard-wiring of
'genodelabs'. Hence the 'depot_user' function has become a built-in function
of the run tool.
Repeat mode for the depot-autopilot test environment
----------------------------------------------------
The _depot_autopilot.run_ script now supports the environment variable
'TEST_REPEAT' with the possible values 'until_forever' and 'until_failed'.
When specified, the tests are run in a loop. The latter argument is
particularly useful for reproducing sporadic errors.
New run script to execute a single test w/o the depot
-----------------------------------------------------
The new _os/run/test.run_ script allows for the quick execution of an
individual test from the build directory while side-stepping the depot.
It expects a 'PKG' variable specifying the test package. E.g.,
! make run/test KERNEL=nova PKG=test-xml_node
Note that it does not cover all test packages right now.
The tool and its limitations are explained in more detail by a dedicated
[https://genodians.org/nfeske/2019-02-05-shortcut-for-testing - posting]
at Genodians.org.
Support for undefined-behavior sanitizer
----------------------------------------
The
[https://developers.redhat.com/blog/2014/10/16/gcc-undefined-behavior-sanitizer-ubsan/ - UndefinedBehaviorSanitizer]
(UBSan), also described in the
[https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html - GCC documentation],
can detect undefined behavior while a program is running.
With the current release, this diagnostic feature of GCC becomes available for
analyzing Genode components. By specifying 'SANITIZE_UNDEFINED = yes' in a
'target.mk' file, the '-fsanitize=undefined' compiler flag is enabled and the
program is linked with libubsan and libsanitizer_common. The program has to
call 'env.exec_static_constructors()' and 'sanitizer_init(env)' upon startup
to initialize the sanitizer libraries. Whenever undefined behavior is detected
while the program is running, a "runtime error:" message including the source
code location of the error is printed to the log.

886
doc/release_notes/19-05.txt Normal file
View File

@@ -0,0 +1,886 @@
===============================================
Release notes for the Genode OS Framework 19.05
===============================================
Genode Labs
The Genode release 19.05 is primarily focused on platform support.
It adds compatibility with the 64-bit ARM architecture (AARCH64),
comes with improvements of the various kernels targeted by the framework,
and extends the list of supported hardware. The increased diversity of base
platforms calls for unifications to keep the hardware and kernel landscape
manageable.
On that account, Genode uses one reference tool chain across all kernels
and CPU architectures. The current release upgrades this tool chain to
*GCC 8.3* with C++17 enabled by default
(Section [Tool chain based on GCC 8.3.0 and binutils 2.32]).
To increase the velocity of Genode system scenarios across different boards
of a given CPU architecture, the release introduces the notion
of *board and kernel-agnostic build directories* presented in Section
[Unified build directories for ARM]. Once built for one particular
CPU architecture, the same binaries can be deployed at any supported board or
kernel without recompilation. This vastly accelerates the workflow when
targeting multiple boards and emulators at once.
As another major unification effort, the current release introduces a new
*kernel-agnostic virtualization* interface. Up until now, virtualization
used to be inherently tied to a specific kernel. Thanks to the new interface,
however, one virtual machine monitor implementation can be combined with
kernels as different as NOVA, seL4, or Fiasco.OC. No recompilation needed.
As outlined in Section [Kernel-agnostic virtual-machine monitors], Genode
has now become able to run the Seoul VMM on all those kernels, while
VirtualBox is planned to follow.
On our [https://genode.org/about/road-map - road map], we originally
planned several user-facing features related to Sculpt OS. However, in the
light of the major platform efforts, we decided to defer those topics instead
of rushing them.
That said, the release is not without new features. For example, our port
of *OpenJDK* has become able to host the Spring framework and the Tomcat web
server, there are welcome improvements of the *package-management tooling*,
and we added new options for user-level networking.
Finally, version 19.05 is accompanied with the annual revision of the *Genode*
*Foundations book* (Section [New revision of the Genode Foundations book]),
which is now available as an online version in addition to the regular PDF
document.
Kernel-agnostic virtual-machine monitors
########################################
Since the introduction of Genode's
[https://genode.org/documentation/release-notes/17.02#Genode_Application_Binary_Interface - Application Binary Interface]
in the 17.02 release,
Genode components can be assembled once for a given hardware platform and
executed without further adjustments on all the supported kernels. However, at
that time, the supported virtual machine monitors - a port of VirtualBox 4 & 5,
Seoul, and our
[https://genode.org/documentation/articles/arm_virtualization - custom VMM] -
remained kernel specific.
Of course, last remaining bastions tempt to be taken! So last year, we started
the venture to unify our virtualization interface across different kernels.
Starting point was the already existing Genode VM interface of our custom VMM
on ARM. We took it and extended the interface with caution to the x86 world.
Having an eye on the requirements of our already supported VMMs on NOVA(x86),
namely VirtualBox and Seoul, the VM interface got extended with missing
features like multiple vCPU support and specific VM handlers per vCPU.
In parallel, we started to investigate the other x86 microkernels with regard
to hardware-assisted virtualization features, namely seL4 and Fiasco.OC.
Over several weeks, we iteratively extended the interface. On the one hand
we familiarized ourself with the kernel interfaces of seL4 & Fiasco.OC while
on the other hand considered known requirements of the NOVA microhypervisor.
Additionally, we kept our custom VMM for ARM still compatible with the new VM
interface.
During this time, it became apparent that the control flow on a VM resume/pause
and a VM event(exit) are different between seL4/Fiasco.OC and NOVA/base-hw.
For seL4 and Fiasco.OC, a VM is resumed by making a blocking syscall on the
kernel. On a VM event, the blocking syscall would return. Logically, on both
kernels the VMM 'calls' into the VM.
On base-hw and NOVA, it is the other way around. Whenever a VM causes a VM
event, the kernels set up either an asynchronous notification (base-hw) or a
synchronous IPC call (NOVA) to the VMM. In both cases the VMM executes a prior
registered VM event handler as response.
Upon return of the VM event handler, the kernel resumes the VM. Logically, on
NOVA and base-hw the VM 'calls' into the VMM. The following two figures
contrast the different flows of control between a user-level virtual machine
monitor and the respective kernels.
[image vm_seq_foc_sel4]
Control flow of handling virtualization events on Fiasco.OC and seL4
[image vm_seq_nova_hw]
Control flow of handling virtualization events on NOVA and the base-hw kernel
Hiding this differences behind a common VM interface was the challenge we were
faced, accepted, and won. Finally, at one point in December we had all 3
x86 kernels running with a test VMM - without re-compilation. The toy VMM
(vmm_x86.run) runs multiple vCPUs on multiple physical CPUs and tests several
VM events/exits.
After this major breakthrough, we spent the days left before Christmas to
adjust the Seoul VMM to the new VM interface, freeing it from the ties to the
NOVA kernel. The choice to start with Seoul stems from the fact that it is -
compared to VirtualBox - much smaller and therefore easier to debug if things
go wrong in the beginning. After one week, the Seoul VMM became in principle
kernel independent and worked again on NOVA. After some more days, it started
to hobble on seL4 and Fiasco.OC as well.
With the New Year, VirtualBox was the next target where all NOVA kernel
specific calls were replaced with the new Genode VM interface. Mid of January,
the work showed first results by having a prototype running simple VMs on NOVA
again. At this point, it became apparent that this venture is not anymore an
adventure. All the findings and technical details so far got condensed to a
[https://fosdem.org/2019/schedule/event/microkernel_virtualization - presentation]
given and recorded at the [https://fosdem.org/2019 - FOSDEM 2019] in Brussels
in February in the
[https://fosdem.org/2019/schedule/track/microkernels_and_component_based_os - Microkernel and Component based OS]
developer room.
At this point, we started transforming our prototype for the 4 kernels into a
clean solution to be featured in Genode 19.05. Eventually, the kernel-agnostic
Seoul VMM runnable on seL4, Fiasco.OC, and NOVA entered Genode master. In the
Genodians article
[https://genodians.org/alex-ab/2019-05-09-seoul-vmm - Seoul VMM and the new VM interface],
we conserved the current state and a few performance measurements.
Shortly before this release, the kernel-agnostic VirtualBox VMM version on
Genode/NOVA got ready. The kernel-agnostic version is in principle capable to
run Linux VMs and Windows 7/10 VMs on Genode/NOVA. Currently, this version
must still be considered as experimental and does not run on seL4 or
Fiasco.OC.
Because of the experimental nature of the kernel-agnostic VirtualBox VMM
version, we decided to keep the kernel-specific version for NOVA for the
moment. This gives us time to test and improve the kernel-agnostic version. It
also allows us to compare both versions to each other.
If time and interest permits, we will consider bringing the virtualization
support on Genode/seL4 and Genode/Fiasco.OC on par with Genode/NOVA.
When building VirtualBox with Genode 19.05,
you will find both the 'virtualbox5-nova' and the new 'virtualbox5' binaries
in the build directory. The former relies on NOVA's kernel interface whereas
the latter uses Genode's kernel-agnostic VM interface. Nightly tested run
scenarios with the new VM interface are named 'vbox5_vm*.run' and can be found
in the 'repos/ports/run' directory.
Broadened CPU architecture support and updated tool chain
#########################################################
With the major update of Genode's tool chain and library infrastructure in
tandem, the framework gains a consistent architecture support across x86-32,
x86-64, ARM-32, RISC-V, and the newly added AARCH64. This includes the tool
chain (Section [Tool chain based on GCC 8.3.0 and binutils 2.32]), the base
framework, the dynamic linker, and the C runtime
(Section [Updated dynamic linker and C runtime]).
Together with this update, we took the chance to wrap up our long-time move
away from board-specific build directories to one generic build directory
shared by multiple kernels and boards for a given CPU architecture
(Section [Unified build directories for ARM]).
Tool chain based on GCC 8.3.0 and binutils 2.32
===============================================
Genode uses a tailored tool chain based on GCC and binutils that is used
across all supported kernels and architectures. Since the previous tool-chain
update in version
[https://genode.org/documentation/release-notes/17.05#Tool_chain - 17.05],
we relied on GCC 6.3. After two years, it was time for an update, motivated by
three major reasons. First, the C++17 standard is common-place now. We Genode
developers anticipate the improvements that come with it. Second, RISC-V and
AARCH64 are now supported by mainline GCC. Up till now, we had to maintain a
custom patch set for Genode's RISC-V support. AARCH64 was not supported yet.
Third, our increasing engagement with SPARK depends on recent improvements of
the Ada compiler that is part of GCC.
With Genode 19.05, the tool chain is now based on binutils version 2.32, GCC
version 8.3.0, GDB version 8.2.1, gcov version 8.3.0, standard C++ library
version 8.3.0.
The tool chain supports x86 (32 and 64 bit), ARM, AARCH64, and RISC-V.
For C++ code, the C++17 standard is enabled by default.
The update of the tool chain provided a perfect opportunity to replace the
former use of gnatmake with a much more natural integration of Ada in Genode's
build system, using a custom ali2dep dependency-extraction tool developed
by [https://github.com/Componolit/ali2dep - Componolit].
In contrast to the previous versions, we switched to a versioned installation
directory for the new tool chain. By default, it is now installed to
_/usr/local/genode/tool/19.05/_. This eases the use of different tool-chain
versions for different development branches.
:Tool-chain installation:
[https://genode.org/download/tool-chain]
Caveats
-------
The tool-chain update required a number of adaptations throughout the source
tree, and may affect Genode users too:
* The silent fall-though within switch statements must now be replaced
by an explicit annotation of the form
! [[fallthrough]]
* The 'register' keyword is no longer valid with C++17. Hence, it must
be removed from the code.
* Types marked as 'Noncopyable' can no longer have an implicit default
constructor. A default constructor must be provided manually.
Updated dynamic linker and C runtime
====================================
The tool-chain update is accompanied with a major update of the dynamic linker
and the C runtime to cover both the AARCH64 and RISC-V architectures in
addition to the traditional x86 and ARM architectures.
FreeBSD 12 supports AARCH64 and RISC-V. Hence, by updating our C runtime to
this version, Genode's libc support extends to those architectures now.
Until now, Genode's dynamic linker supported only the eager binding of symbols
at loading time on the *RISC-V* architecture. With the current version, we
lifted this limitation in favor of lazy binding as used on all other CPU
architectures.
Unified build directories for ARM
=================================
In version
[https://genode.org/documentation/release-notes/17.02#Genode_Application_Binary_Interface - 17.02],
we introduced unified build directories for x86, which allow us to build and
run Genode scenarios on various kernels while using only one build directory.
This concept leverages Genode's cross-kernel binary compatibility to make
the switch from one kernel to another - like developing on base-linux and
deploying on base-nova - a seamless experience.
On ARM, this concept was held back by a third dimension. The
system-integration step does not only depend on the CPU architecture and
the kernel but also on the used board. Our traditional approach was the
use of one build directory per board. Granted, within such a build directory,
one could easily switch between different kernels like Fiasco.OC and seL4.
But on ARM, we find an extreme proliferation of different board
configurations, which share the same CPU architecture but demand different
integration steps. This ensues large redundancies among different build
directories. Switching from one board to another - even when most binaries
happen to be exactly the same - requires an additional rebuilding effort.
With version 19.05, we took the chance to generalize the unified build
directory concept to support multiple different boards per build directory,
greatly reducing the friction when switching kernels and boards for a given
CPU architecture (like ARMv7a). This change has the following implications:
* Drivers no longer depend on the SPEC values as configured for a build
directory.
* All *binaries* are now *named unambiguously*. For example, the USB drivers
for the Panda (OMAP) and Arndale (Exynos) boards were formerly called
'usb_drv' but were different programs. They just never happened to
appear in the same build directory. In the new version, they are named
'panda_usb_drv' and 'arndale_usb_drv' respectively and can thereby
peacefully co-exist within the same 'armv7a' build directory.
Note that this binary renaming will likely affect existing run scripts.
* Include paths no longer hide the board details, which makes the included
code much more easy to follow.
* Run scripts need to pick the right binary, depending on the used board.
Since the board is no longer tied to a build directory, the selection
of the used board has become a build-time variable 'BOARD' following
the successful pattern of how we specify the targeted 'KERNEL'.
To avoid the pollution of run scripts with difficult conditions, we wrap
the drivers needed for a particular board and use case into so-called
_drivers_ packages. Such a package can be instantiated within a generic
scenario using a nested init instance. The details about the drivers and
how they access the hardware remain nicely hidden inside this building block.
Currently, there exist _drivers_ packages for two distinct use cases:
:drivers_interactive pkgs: contain all drivers needed for simple
interactive scenarios, including graphical output and user input.
:drivers_nic pkgs: contain the drivers needed for communication over the
network.
Whenever a run script fits one of these use cases, it can rely on the
corresponding ready-to-use drivers packages via:
! import_from_depot [depot_user]/src/[base_src] \
! [depot_user]/pkg/[drivers_nic_pkg] \
! ...
With the drivers package incorporated, the drivers subsystem can be
instantiated as follows (note the absence of any board or kernel-specific
details):
! <start name="drivers" caps="1000">
! <resource name="RAM" quantum="32M" constrain_phys="yes"/>
! <binary name="init"/>
! <route>
! <service name="ROM" label="config">
! <parent label="drivers.config"/> </service>
! <service name="Timer"> <child name="timer"/> </service>
! <any-service> <parent/> </any-service>
! </route>
! <provides> <service name="Nic"/> </provides>
! </start>
Using the 'BOARD' build variable
--------------------------------
The new 'BOARD' variable selects the board to use. It can be specified either
as a 'make' command-line argument (or environment variable), or defined in the
build-directory configuration (_etc/build.conf_). The following boards are
available:
:arm_v6: rpi
:arm_v7a: arndale, imx53_qsb, imx53_qsb_tz, imx6q_sabrelite, imx7d_sabre,
nit6_solox, odroid_x2, odroid_xu, panda, pbxa9, usb_armory,
wand_quad, zynq_qemu
:arm_v8a: rpi3
:x86_64: pc, linux, muen
:x86_32: pc, linux
:riscv: spike
Please note, when running Genode on Linux or the Muen separation kernel -
although it is run on common x86 PC hardware - we treat both runtime
environments as separate "boards" because their device driver environments
are fundamentally different.
New revision of the Genode Foundations book
###########################################
The "Genode Foundations" book received its annual update, which is actually
rather a refinement than a revision. The noteworthy additions and changes are:
: <div class="visualClear"><!-- --></div>
: <p>
: <div style="clear: both; float: left; margin-right:20px;">
: <a class="internal-link" href="https://genode.org">
: <img class="image-inline" src="https://genode.org/documentation/genode-foundations-title.png">
: </a>
: </div>
: </p>
* Component health monitoring
* Static code analysis
* Documentation of --depot-user and --depot-auto-update
* Minor adjustments in the under-the-hood chapter
* Changes of the build system
* Updated tool requirements
* Updated API reference
: <div class="visualClear"><!-- --></div>
To examine the changes in detail, please refer to the book's
[https://github.com/nfeske/genode-manual/commits/master - revision history].
New online version of the book
------------------------------
We are happy to announce that the Genode Foundations book is now available
as an online version in addition to the regular PDF version.
:Browse the Genode Foundations book online:
[https://genode.org/documentation/genode-foundations/19.05/index.html]
Thanks a lot to Edgard Schmidt for creating the tooling for the HTML version
of the book!
Base framework and OS-level infrastructure
##########################################
Modernized block-storage interfaces
===================================
With the current release, we revisited Genode's interfaces for accessing
block devices to ease the implementation of asynchronous I/O, to accommodate
zero-copy block drivers, and to support trim and sync operations.
Revised RPC interface
---------------------
The 'Block::Session' RPC interface remained untouched for a long time.
We have now rectified long-standing deficiencies.
First, *sync requests* used to be handled as synchronous RPCs. This is bad
for components like part_block that multiplex one block device for multiple
clients. One long-taking sync request of one client could stall the I/O for
all other clients. The new version handles sync requests as asynchronous
block-request packets instead.
Second, the new version allows a server to dictate the *alignment* of
block-request payload. This way, a driver becomes able to use the payload
buffer shared between client and server directly for DMA transfers while
respecting the device's buffer-alignment constraints.
Third, we added support for *trim* as an asynchronous block operation.
However, as of now, this operation is ignored by all servers.
Fourth, each block operation can now be accompanied with a client-defined
request tag independent from the other parameters of the operation. The tag
allows a block-session client to uniquely correlate acknowledgments with
outstanding requests. Until now, this was possible for read and write
operations by taking the value of the request's packet-stream offset. However,
sync and trim requests do not carry any packet-stream payload and thereby lack
meaningful and unique offset values. By introducing the notion of a tag, we
can support multiple outstanding requests of any type and don't need to
overload the meaning of the offset value.
New client-side API
-------------------
We have now equipped the 'Block::Connection' with a framework API for the
implementation of robust block-session clients that perform block I/O in an
asynchronous fashion.
An application-defined 'JOB' type, inherited from 'Connection::Job',
encapsulates the application's context information associated with a block
operation.
The life cycle of the jobs is implemented by the 'Connection' and driven by
the application's invocation of 'Connection::update_jobs'. The 'update_jobs'
mechanism takes three hook functions as arguments, which implement the
applications-defined policy for producing and consuming data, and for the
completion of jobs.
We plan to gradually move the existing block clients to the new API to benefit
from the latency-hiding effects of asynchronous I/O. The first updated client
is the _block_tester_ component located at _os/src/app/block_tester/_, which
received a number of new features like the choice of the batch size. Please
refer to the accompanied README for a detailed description of the
block-tester.
Unified types for time values
=============================
[https://genode.org/documentation/release-notes/17.05#New_API_for_user-level_timing - Two years ago],
we introduced the so-called timeout framework to provide a general solution
for requirements unmet by the bare timer-session interface - most notably
timer-session multiplexing amongst multiple timeouts, and microseconds
accuracy. Up to this day, the timeout framework has proved itself many times
in both real-life appliances and artificial tests and has become the standard
front end for timing in Genode applications.
With this release, we solved one of the few remaining limitations with the
framework by enabling timeouts of up to 2^64 microseconds (> 500000 years)
across all supported architectures. In order to achieve this, we replaced the
former machine-word-wide types used for plain time values by unsigned 64-bit
integers. We did this not only inside the timeout framework but also to almost
all code in the basic Genode repositories that uses the framework.
By doing so, we also paved the way for a second step, in which we are planning
to replace plain time values as far as possible with the abstract 'Duration'
type. With this type in place, the user wouldn't have to worry anymore about
any plain-integer implications when calculating with time values.
Support for chained EBR partitions
==================================
Having an active community around Sculpt leads to bugfixes in unexpected
places. By now we prefer to use a GPT rather than an MBR based partition table
and although we test 'part_block', the component that parses the tables, on
regular basis, the handling of chained EBR's was flawed. Community member
[https://genodians.org/valerius/index - Valery Sedletski] who relies on such a
setup encountered this flaw and provided a bug report, which enabled us to
quickly reproduce and fix the problem.
IP forwarding with port redirection
===================================
The NIC router can now be used to redirect to individual destination ports on
port-forwarding. To express the redirection, the new 'to_port' attribute can
be added to '<tcp-forward>' and '<udp-forward>' rules in the NIC router
configuration. If the new attribute isn't added, the rules behave as usual and
forward with an unaltered destination port.
Libraries, languages, and applications
######################################
Ada/SPARK runtime and SPARK-based cryptography
==============================================
The SPARK runtime has been updated to GCC 8.3. SPARK components do not require
'Genode::Env' or a terminal session anymore. Debug messages can still be
printed using 'GNAT.IO', which uses 'Genode::log' and 'Genode::error'
internally now.
Threading support, which was never fully implemented, has been removed to
further simplify the runtime. This simplification allowed us to prove absence
of runtime errors for the secondary stack allocator and other parts of the
runtime.
[https://github.com/Componolit/libsparkcrypto.git - Libsparkcrypto] is a
library of common cryptographic algorithms implemented in SPARK. It is
free-standing and has a very small footprint. The port of libsparkcrypto for
Genode has been added to the libports repository. Thanks to Alexander Senier
and Johannes Kliemann of [https://componolit.com - Componolit] for maintaining
the Ada/SPARK runtime and libsparkcrypto.
To accommodate the use case of block encryption, we added the small wrapper
library 'aes_cbc_4k' around libsparkcrypto that provides a simple C++
interface for the en/decryption of 4 KiB data blocks. It uses AES-CBC while
incorporating the block number and the private key as salt values.
Improved resilience of the sequence tool
========================================
We have a simple component that starts other components sequentially. It
will exit whenever one of those components has exited with an error.
However, this component is used by our [https://genodians.org - Genodians]
appliance where it controls the content-update mechanism. Since updating
involves fetching content via HTTP/S depending on external events, e.g.,
the remote site is not reachable, the sequence tool might exit. In a long
running appliance, this is obviously not a useful action where no one is
in place to restart the sequence tool. Rather than increasing the overall
complexity of the appliance by introducing such a management component, we
added a _keep-going_ feature to the sequence tool that will instruct it
to carry on even if one of the started components has failed.
Please look at _repos/os/src/app/sequence/README_ for instructions on
how to use the feature.
NIC-bus server for private LANs
===============================
The 'nic_bus' server was added to the world repository as an alternative
to the 'nic_router' and 'nic_bridge' components. The name may be a slight
misnomer, but this component acts neither as a hub, switch, or router.
The 'nic_bus' implements unicast and multicast Ethernet packet switching
between sessions, but drops any unicast packet not destined for a session
attached to the bus. This is in opposition to the behavior of a typical
Ethernet switch and is intending to create simple, software-defined
local-area-networks for native components as well as virtual machines.
In practice the component has been used for attaching VMs to the
[https://yggdrasil-network.github.io/ - Yggdrasil] overlay network via
a bus-local IPv6 prefix.
Distributed Genode
==================
In
[https://genode.org/documentation/release-notes/16.08#Network-transparent_ROM_sessions_to_a_remote_Genode_system - 16.08],
we initially released the _remote_rom_ components that act as communication
proxies. A communication proxy transparently relays a particular service to
another Genode system. As the name suggests, the remote_rom relays ROM
sessions.
Originally implemented as a proof of concept using bare IP packets, broadcast
MACs and static configuration of IP addresses, we added several improvements
to allow a more general use. First, we adopted the size-guard idea for packet
construction and processing from the NIC router. Furthermore, we adopted the
single-threaded implementation style that was already established in other NIC
components. Thanks to Edgard Schmidt for this contribution. Second, we
implemented ARP requests to eliminate broadcasting. Third, we moved from bare
IP packets to UDP/IP and implemented a go-back-N ARQ strategy in order to
reliably transmit larger ROM dataspaces.
As the remote_rom proved valuable for distributing functionality across
multiple Genode devices, we also applied this concept to the LOG session in
order to transmit LOG output from a headless Genode device to a
[https://genode.org/download/sculpt - Sculpt] system for instance. The udp_log
component provides a LOG service and sends the LOG messages as UDP packets to
another machine. The log_udp reverses this process by receiving these UDP
packets and forwarding the messages to a LOG service. An example can be found
in the world repository at _run/udp_log.run_ and _run/log_udp.run_.
Seoul and VirtualBox virtual machine monitors
=============================================
Besides the conversion of the Genode back end of Seoul to the new VM
interface, we added mouse-wheel support to the PS/2 model and changed the VMM
to request a single GUI/nitpicker session rather than distinct framebufer and
input sessions.
Similar to the Seoul VMM, the VirtualBox VMM was adjusted to the new VM
interface and now uses the GUI/nitpicker session. The original kernel-specific
VirtualBox version tied to the NOVA kernel is still available. Both versions
can be used simultaneously.
Use of Nim decoupled from Genode build system
=============================================
With this release, all integration with Nim tooling has been removed from the
Genode build system as a result of maturing support for additional languages
via Genode SDKs. Building Nim components independently of the Genode source
tree has the benefit of smaller upstream checkouts and faster build times, and
has yielded components such as the
[https://genodians.org/ehmry/2019-03-22-depot_9P - 9P server] used in some
Sculpt developer workflows. An example of an independent build system for Nim
components is
[https://genodians.org/ehmry/2019-04-27-nim_packaging - documented on the Genodians blog].
OpenJDK improvements
====================
Within the 19.05 release cycle, we further improved Genode's OpenJDK support
by enabling additional networking infrastructure required by the
[https://spring.io - Spring Framework]. The improvements especially concern
support for SSL connections, which enabled us to successfully execute an embedded
[https://docs.spring.io/spring-boot/docs/current/reference/html/howto-embedded-web-servers.html - Tomcat]
server natively on Genode x86 and ARMv7 platforms using the same JAR archive.
This line of work continues our Java for embedded systems effort as described in
our [https://genodians.org/ssumpf/2019-02-27-java-19-02 - Boot2Java] article.
Having these features in place, our Java efforts will continue in the direction
of Java Swing and the support of input devices in the future, with the ultimate
goal of seamless Java application integration into
[https://genode.org/download/sculpt - Sculpt OS].
Device drivers
##############
Improved Zynq board support
===========================
The initial support of the Xilinx Zynq-7000 SoC was added to our custom kernel
in 15.11. Since then, the support of this hardware has been incrementally
extended. The definitions of memory maps, frequencies, and RAM sizes for
different Zynq-based boards are found in the world repository.
One of the major additions in this release is the initialization of the L2
cache. In this context, we also added a simple cache benchmark at
_repos/os/run/cache.run_ that measures the access times for memory regions of
different size and thereby reveals the number of cache levels and their sizes.
With the latest improvements of the network driver in 18.11, a zero-copy
approach was introduced as an effort to eliminate bottlenecks in the driver's
performance. However, this modification also introduced a kernel dependency of
the driver in order to flush packet-buffer memory from the cache before handing
it over to the DMA-controller. With this release, we moved back to using
uncached dataspaces in order to eliminate the cache flushes and the kernel
dependency. Interestingly, we could not recognize a significant impact on the
driver's performance, which confirms the presumption that flushing the cache
nullifies the gain from using cached dataspaces.
In order to enable the continuous operation of the network driver, we extended
the driver-internal error handling that is necessary to recover the network
driver in certain situations.
_Thanks to Johannes Schlatow for contributing and maintaining Genode's Zynq support!_
Updated Intel network drivers
=============================
As a result of recurring issues with modern Intel i219 laptop NICs, we
updated the driver sources for Intel chipsets to the latest upstream
iPXE version. This update also enables all NIC variants, which were
missing from our manually maintained PCI ID whitelist before.
New drivers-nic and drivers-interactive depot packages
======================================================
As already described in section [Unified build directories for ARM],
_drivers_nic_ packages nicely hide the driver configuration internals needed for
a specific board to communicate over the network. Until now there was only one
package available for x86 based PCs. Now, additional _drivers_nic_ packages
are available for:
:boards: imx53_qsb imx6q_sabrelite linux muen pbxa9 rpi zynq
Beside the formerly available _drivers_interactive_ packages for linux, pbxa9
and pc, there are now additional ones for the following:
:boards: imx53_qsb rpi muen
Platforms
#########
For most kernel environments, the core component provides a ROM module named
'platform_info', which comprises information provided by either the kernel or
the bootloader. The information entails e.g., the TSC clock frequency and
framebuffer dimensions. Most of the information is of interest for special
device driver components only.
Over the time, there was an increasing need to incorporate the information
about which kernel Genode runs on top of. Thereby, special test components,
like depot_autopilot could use the information to, e.g., skip certain tests
on kernels known to not support them. Moreover, there are rare corner-cases
where kernels behave differently, for instance, interrupts are enumerated
differently on certain ARM platforms. Rather than maintaining multiple driver
binaries with different names depending on specific kernels, the
'platform_info' ROM module can now be used to differentiate between kernels
when necessary.
Execution on bare hardware (base-hw)
====================================
This release comes with fundamental optimizations and corrections for
executing Genode on bare hardware when using the core component as the actual
kernel.
In the past, we could observe some serious peculiarities regarding the timing
behavior on the hw kernel. After a careful review, we identified the obstacles
that led to time drifts on several platforms and to quite different runtime
execution.
First and foremost, we limited the CPU-load wasted by the kernel, which
unnecessarily made new scheduling decisions quite often. When the hw kernel
was started as an experiment, there was less focus on performance, but more on
simplicity. Instead of caring about state changes that make a scheduling
decision necessary, the scheduler was asked for the next execution context
unconditionally, whenever the kernel was entered. Now, the scheduler gets
invoked only whenever an execution context gets blocked, or unblocked, or if
the kernel's timer fires due to a timeout. This dramatically influences the
CPU-load caused by the hw kernel in a positive way.
The timing accuracy got increased by reworking most hardware timer drivers
used in the kernel to let the timer never stop counting. Moreover, we limit
the scope in between reading the clock and adjusting the next timeout to a
minimum. The whole internal time representation got widened to 64-bit.
In some rare use cases, we could observe components that do I/O polling, and
thereby actively ask for pending signals, to starve. The reason was a gap in
the hw kernel's syscall API. Beside the ability to wait for signals, the
base-library offers the ability to check for pending signals without blocking
in the case of no available signals. The equivalent call in the kernel was
still missing, and is now present and integrated in the base-library of
base-hw.
ARM architecture
----------------
With this release, we add the i.MX 7 Dual SABRE reference board to the rich
hardware zoo Genode runs directly on top of. This includes the use of the
virtualization extensions available on this platform.
Apart from the new board support, several optimizations were added
specifically for the ARM architecture. Several unnecessary cache maintenance
operations were eliminated, which resided in the code base since the time when
the kernel used a separate address-space only. Moreover, the kernel-lock -
used when several execution contexts on different CPU-cores try to enter the
kernel - does not spin anymore. Instead, the CPU goes into a sleep-state to
save energy. As a side-effect, multi-core scenarios become usable when
executed in Qemu.
X86 architecture
----------------
Since the newly used compiler version makes aggressive use of FPU instructions
including the core component, the kernel itself makes use of FPU registers and
state. Therefore, lazy FPU switching becomes a no go for base-hw. Although, we
incorporated eager FPU switching into the ARM-specific part of the hw kernel
already, the x86 version was still missing it. Now, the FPU context of a thread
gets saved and restored on every kernel entry and exit on x86 too.
Updated Muen separation kernel
==============================
The Muen port has been updated to the latest development version, which comes
with many improvements under the hood. Most notably this version of Muen brings
support for Linux SMP subjects, GNAT Community 2018 toolchain support as well
as much improved build speed, which is most noticeable during autopilot runs.
Additionally, the debug server buffer size in the Genode system policy has been
increased to avoid potential message loss in case of rapid successive logging.
_Thanks to Adrian-Ken Rueegsegger of [https://codelabs.ch - Codelabs] for_
_this welcome contribution!_
NOVA microhypervisor
====================
The kernel got updated due to the tool-chain update to GNU G++ 8.3.0.
Additionally, several issues reported by Julian Stecklina regarding FPU and
page-table synchronization got addressed. The kernel memory allocation at boot
time got more flexible to address target machines with fragmented physical
memory. Additionally, the vTLB implementation is no longer used on AMD
machines whenever nested paging is available.
seL4 microkernel
================
With this release, we extend the variety of hardware to run Genode on top of
the seL4 kernel with NXP's i.MX 7 Dual SABRE reference board. To do so, we had
to update the seL4 tools used to craft a bootable ELF image to a state that is
consistent with the currently supported seL4 kernel version 9.0.1.
As a side-effect of this development work, the General Purpose Timer (GPT) used
in the i.MX series can now be used as a timer service component.
Fiasco.OC microkernel
=====================
As with base-hw and seL4, we add the i.MX 7 Dual SABRE reference board to the
list of working hardware for Genode running on top of the Fiasco.OC
microkernel. Moreover, with Fiasco.OC it is now possible to take the first
steps using Genode on the ARM 64-bit architecture. Therefore, we add Raspberry
Pi 3 as a candidate board to be used with Genode/Fiasco.OC. Currently, only
basic tests without peripheral dependencies are supported.
Tooling and build system
########################
Improved handling of missing ports
==================================
The depot tools _tool/depot/create_ and _tool/depot/extract_ now detect and
report all missing third-party sources - called ports - for a given set of
archives at once. Additionally, the user can tell the tools to download and
prepare such missing ports automatically by setting the argument
'PREPARE_PORTS=1'. Please be aware that doing so may cause downloads and
file operations in your _contrib/_ directory without further interaction.
These features make building archives with dependencies to many ports more
enjoyable. If you merely need a list of ports that are missing for your
archives, you can use the new tool _tool/depot/missing_ports_.
For more details you may read the
[https://genodians.org/m-stein/2019-05-21-depot-missing-ports - article on genodians.org].
Automated depot management
==========================
When using the 'import_from_depot' mechanism of the run tool, one frequently
encounters a situation where the depot lacks a particular archive. Whenever
the run tool detects such a situation, it prompts the user to manually curate
the depot content via the _tool/depot/create_ tool. The need for such manual
steps negatively interferes with the development workflow. The right manual
steps are sometimes not straight-forward to find, in particular after
switching between Git branches.
To relieve the developer from this uncreative manual labor, we extended the
run tool with the option '--depot-auto-update' for managing the depot
automatically according to the needs of the executed run script. To enable
this option, use the following line in the build configuration:
! RUN_OPT += --depot-auto-update
If enabled, the run tool automatically invokes the right depot-management
commands to populate the depot with the required archives, and to ensure the
consistency of the depot content with the current version of the source tree.
The feature comes at the price of a delay when executing the run script
because the consistency check involves the extraction of all used source
archives from the source tree. In regular run scripts, this delay is barely
noticeable. Only when working with a run script of a large system, it may be
better to leave the depot auto update disabled.
Please note that the use of the automated depot update may result in version
updates of the corresponding depot recipes in the source tree (recipe hash
files). It is a good practice to review and commit those hash files once the
local changes in the source tree have reached a good shape.

681
doc/release_notes/19-08.txt Normal file
View File

@@ -0,0 +1,681 @@
===============================================
Release notes for the Genode OS Framework 19.08
===============================================
Genode Labs
The stated theme of this year's [https://genode.org/about/road-map - road map]
is "bridging worlds", which expresses our ambition to smoothen the practical
use of Genode-based systems such as Sculpt OS. The current release pays
tribute to this ambition by addressing a great number of practical concerns:
How to accommodate the staggering variety of keyboard layouts out there?
(Section [Flexible keyboard layouts])
How can the system gracefully respond when confronted with exotic USB devices?
(Section [Storage-stack improvements])
How to set the system time from within the system? How does SNTP fit in here?
(Section [General system time concept])
How to approach the remote administration of the system?
(Section [Enhanced SSH terminal])
How to copy and paste text securely between mutually distrusting subsystems?
(Section [Clipboard])
Or how to overcome the captive portal of a Hotel WiFi with Sculpt OS?
(Section [Disposable VM for handling captive portals])
By providing answers to those questions, we believe to make Genode - and Sculpt
OS in particular - generally more useful.
As another take on "bridging worlds", we continue our effort to bring the rich
Sculpt OS software stack to the 64-bit ARM world, in particular to our most
loved SoC family, namely NXP i.MX. Section [64-bit ARM and NXP i.MX8] reports
on our progress in this direction.
Under the hood, there are a few exciting developments that will greatly reduce
the effort of running existing software on Genode. In particular, Genode's
(entirely optional) C runtime has gained the ability to emulate the
traditional execve and fork mechanisms.
(Section [Consolidation of the C runtime and Noux]) This will eventually
alleviate the need for our present noux runtime environment to the benefits of
better performance and increased flexibility.
Further highlights of Genode 19.08 are a major update of Qt5 to version 5.13
(Section [Updated Qt5]) and the continuation of our kernel-agnostic
virtualization story (Section [Virtualization]).
Flexible keyboard layouts
#########################
Genode is used worldwide in a multilingual context beyond Germany and common
technical realms of English. Therefore, we had to address localized
keyboard-input handling for quite some time now and introduced the
_input-filter_ component in
[https://genode.org/documentation/release-notes/17.02#Input-event_filter - 17.02].
The component merges input streams and applies several forms of input
transformations, in particular the application of keyboard layouts to
supplement the input-event stream with character events.
But as we are by no means localization experts, our solution, while performing
a solid job for selected layouts, also had some quirks and rough edges when it
came to French or even Swiss German. First, our oversimplified notion of
[https://en.wikipedia.org/wiki/Caps_Lock - Caps Lock] as _just a pressed Shift_
_key_ is plain wrong but part of all our character-generator configurations.
We just missed this drawback because none of our developers uses Caps Lock
regularly. Further, US English and Germany layouts work very well without
[https://en.wikipedia.org/wiki/Dead_key - dead keys], but crossing any German
border (except the Austrian) is impossible without support for key sequences
composing special characters. The French keyboard layout in Genode tried to
alleviate the lack of compose sequences by adding an additional Circumflex
modifier and character mapping, which unfortunately is not standard.
[image keyboard_stack]
Beginning at this state of affairs, we researched common practice in
international keyboard-input handling, sought a quasi-standard source for
layout configurations, and addressed the drawbacks mentioned before. During
our research we found out that no current implementation is void of critique
and, therefore, decided to look more into X11/XKB as our open-source
quasi-standard solution, but always had an eye on the proprietary world.
The handling of key events in X11/XKB happens on three layers.
:Key codes: On the key-code layer, the device driver programs the
keyboard and generates a stream of key-code (i.e., scan-code)
events, which represent the physical location of the actual key on
the keyboard.
:Key symbols: These key codes are mapped to key symbols, which
represent the label imprinted on the key. So, the key code producing
US English _Q_ (QWERTY keyboard) generates _A_ on a French keyboard
(AZERTY). Modifiers like Shift, AltGr, and Caps Lock are included in
the key-symbol mapping. Additionally, some layouts map key codes to
dead key symbols, which start the before-mentioned compose
sequences. Key repeat is also implemented as key-symbol repeat
actually.
:Characters: On top of this stack, the key symbols are mapped to
characters represented as Unicode codepoints or UTF-8 strings.
The procedure obviously includes key symbols that have no character
representation (e.g. Control and Alt). Key symbols forming a valid compose
sequence generate characters on this level (e.g., dead-key circumflex plus
e generates ê).
We limited our research to Western keyboard-input handling and only had a
blink into the direction of Chinese-Japanese-Korean (CJK) and advanced input
methods (IM). This simplification is supported by the fact that CJK can also
be based on the mechanisms mentioned with some limitations only. Nevertheless,
we do not expect to never touch this topic again.
After doing our homework of keyboard-input handling, we worked on squeezing
all available layout information out of X11/XKB, which resulted in a small
tool residing in _tool/xkb2ifcfg_. For those wondering, the name is just a
silly acronym for _XKB to input-filter_ _configuration_ that pays tribute to
the boringness of this task. After building the tool by a run of 'make' in the
tool path, it can be used as follows. Please make sure you have libxkbcommon
development packages installed beforehand.
! xkb2ifcfg generate <layout> <variant> <locale>
!
! xkb2ifcfg generate us euro en_US.UTF-8
! xkb2ifcfg generate de nodeadkeys de_DE.UTF-8
If the parameter combination is available, xkb2ifcfg prints a input-filer
chargen configuration for the selected layout to standard output. Valid
'layout' and 'variant' options can be figured out from the LAYOUTS section in
'man 7 xkeyboard-config', where 'variant' strings are depicted in parentheses
after the layout (e.g., 'us(euro)'). The 'locale' option has the standard
locale syntax (see /usr/share/i18n/locales). The tool needs all three
parameters to gather the correct key-map and compose-sequence information. The
generated chargen configurations include '<map>' and '<key>' nodes
corresponding to significant modifier states and '<sequence>' nodes (described
later). For simplicity of the generator, the '<key>' nodes always use the
'code="..."' attribute, but also have a comment with the UTF-8 character
appended.
! <key name="KEY_MINUS" code="0x00df"/> <!-- ß -->
Last, we addressed the improvement of the input-filter character generator and
the actual chargen configuration files in Genode. Therefore, we specified the
modifier configuration assumed by the standard chargen files as '<mod1>'
corresponds to Shift, '<mod2>' to Control, '<mod3>' to AltGr, and '<mod4>' to
Caps Lock.
! <mod1> <key name="KEY_LEFTSHIFT"/> <key name="KEY_RIGHTSHIFT"/> </mod1>
! <mod2> <key name="KEY_LEFTCTRL"/> <key name="KEY_RIGHTCTRL"/> </mod2>
! <mod3> <key name="KEY_RIGHTALT"/> </mod3> <!-- AltGr -->
! <mod4> <rom name="capslock"/> </mod4>
As outlined above, the '<key>' nodes generated by xkb2ifcfg always use the
'code' attribute for the Unicode codepoint. Because of this and because UTF-8
also refers to codepoints, we deprecated the 'b0/b1/b2/b3' attributes for
character definition with this release.
The chargen is also extended by the '<sequence>' configuration node. A
sequence node permits the definition of dead-key/composing character
sequences. With such sequences, the character is not generated instantly on
key press but only after the sequence is completed. If an unfinished sequence
can't be completed due to an unmatched character, the sequence is aborted and
no character is generated. We support sequences of up to four characters at
the moment.
For example, the French AZERTY
[https://docs.microsoft.com/en-us/globalization/keyboards/kbdfr.html - keyboard layout]
has a dead key for Circumflex Accent _^_ right of the _P_ key (which is
bracket left _[_ on US keyboards). When Circumflex is pressed no visible
character should be generated instantly but the accent must be combined with a
follow-up character (e.g., Circumflex plus _a_ generates _â_).
Dead keys can be defined in the '<key>' nodes of any '<map>' by using
codepoints not used for direct output, for example, Combining Diacritical
Marks beginning at U+0300. The French Circumflex example can be configured
like follows.
! <mod1>
! <key name="KEY_LEFTSHIFT"/> <key name="KEY_RIGHTSHIFT"/>
! </mod1>
! <map>
! <key name="KEY_Q" code="0x0061"/> <!-- a -->
! <key name="KEY_LEFTBRACE" code="0x0302"/> <!-- dead_circumflex -->
! </map>
! <map mod1="true">
! <key name="KEY_Q" code="0x0041"/> <!-- A -->
! </map>
! <sequence first="0x0302" second="0x0061" code="0x00e2"/> <!-- â -->
! <sequence first="0x0302" second="0x0041" code="0x00c2"/> <!-- Â -->
Fortunately, the configuration is automatically generated by xkb2ifcfg, but
admittedly quite extensive. Therefore, we manually amended the chargen
configurations before adding them to Genode, which also gave us the chance to
apply some adjustments like follows for AltGr in Swiss German.
! <map mod1="false" mod2="false" mod3="true" mod4="false">
! <key name="KEY_1" code="0x00a6"/> <!-- ¦ (*) -->
! <key name="KEY_4" code="0x00b0"/> <!-- ° (*) -->
! <key name="KEY_5" code="0x00a7"/> <!-- § (*) -->
! </map>
Beside the advanced input methods mentioned before, there are still loose ends
we are going to address in the upcoming releases. For example, the current key
handling in our Qt5 back end maps localized key symbols incorrectly (think
AZERTY vs. QWERTY) in combination with shortcuts like Ctrl-A.
64-bit ARM and NXP i.MX8
########################
64-bit ARM support in our custom base-hw kernel
-----------------------------------------------
By introducing rudimentary Raspberry Pi 3 support on top of the Fiasco.OC
kernel in the previous release, the first ARM 64-bit support has entered the
Genode OS framework. We continued pursuing the ARM 64-bit path and introduce
support for Raspberry Pi 3 as well as the i.MX8 evaluation kit (EVK), this
time using our own base-hw kernel.
Noteworthy additions in the base-hw kernel are support for the AARCH64 system
level architecture, and the use of the modern GIC v3 interrupt controller on
top of the i.MX8 EVK board. In comparison to the GICv2, GICv3 adds support for
more than eight CPUs, more than 1020 interrupt IDs, and offers fast register
access to the CPU interface, instead of memory-mapped I/O access. Minor
changes had to be made to the page-table implementation of ARMv7 with Large
Physical Address Extension (LPAE) to re-use it for ARMv8. Moreover, the
internal kernel API for TLB maintenance needed to be changed slightly for all
ARM platforms.
We expanded our regular testing infrastructure with two AARCH64 platforms,
namely Raspberry Pi 3 via Qemu and the NXP i.MX8 EVK board as physical
hardware. Both platforms are driven with a single CPU core only at the moment.
Network driver for i.MX7 and i.MX8
----------------------------------
We updated the 'fec' network driver to version 4.16.3, which adds support for
i.MX7 and i.MX8 SoCs. This makes i.MX8 a viable platform for Genode-based
networking scenarios.
Enhanced packaging and test infrastructure for ARMv8
----------------------------------------------------
Besides the improved base-hw kernel, we enabled additional infrastructure for
ARMv8 platforms. For example, noux packages - like _coreutils_, _bash_ - are
now available, the standard C++ library is in place, and support for Genode's
port of the Linux TCP/IP stack is enabled.
Additionally, ARMv8 is now regularly tested within our nightly
_depot_autopilot_ runs.
Base framework and OS-level infrastructure
##########################################
Tracing
=======
Support for fast tracing has been built into Genode for a long time. However,
the stakes to take advantage of this feature remained high because convenience
functions were not in place. With the current release, we added the support
for easy trace setups through a VFS plugin. The plugin is called _vfs_trace_
and can be mounted into a Genode component as follows:
!<config>
! <vfs>
! <trace ram=32MB/>
! </vfs>
!</config>
This configuration will create a trace file system at the root of the VFS. The
_ram_ attribute is mandatory and determines the maximum size of all trace
buffers. The file system forms a recursive directory structure that represents
the parent/child relationship of running components, whereas the leaf
directories represent single threads within a component. Each leaf directory
currently contains three files:
:'enable': Start or stop the tracing of a thread by writing "true" or "false"
into the file.
:'buffer_size': Allows for the configuration of the trace-buffer size for the
thread in the usual Genode format (e.g. 5M, 512K, 1024).
:'trace_buffer': This read-only file contains the current content of the trace
buffer. Each trace entry can only be read once, after that only new entries
appear. "tail -f" can also be used to display continuous output.
As an example, tracing is started by writing _true_ to the _enable_ file:
! echo "true" > enable
The trace buffer can then be displayed using Unix tools like _tail_
! tail -f trace_buffer
which provides a continuous output.
Additionally, we have added the _trace_ function to _base/log.h_ that
facilitates identical functionality as _Genode::log_
! Genode::trace("Tracepoint value: ", value);
In order to enable tracing, the parent must provide the "TRACE" service. For a
real world example on Sculpt OS, please refer to this
[https://genodians.org/ssumpf/2019-06-18-trace_fs - Genodians article].
With the _vfs_trace_ plugin in place, we removed the outdated _trace_fs_.
Consolidation of the C runtime and Noux
=======================================
On our [https://genode.org/about/road-map#August_-_Release_19.08 - road map],
we vaguely hinted at our plan for the "consolidation" of the noux runtime,
which is actually meant as a polite way of announcing that we are going to
remove it. We introduced the
[https://genode.org/documentation/release-notes/11.02#Noux_-_an_execution_environment_for_the_GNU_userland - Noux runtime]
in 2011 as a way to execute command-line-based GNU software directly on
Genode. It has served us well over the years and is - in fact - a crucial
ingredient of Sculpt OS and other system scenarios such as the Genodians.org
web server. Noux supplements Genode with two valuable assets, namely a
flexible and expandable virtual file system (VFS) layer, and the
implementation of the
[https://genode.org/documentation/release-notes/12.02#Noux_support_for_fork_semantics - Unix way]
to spawn applications ('fork' and 'execve').
In the
[https://genode.org/documentation/release-notes/17.02#Enhanced_VFS_infrastructure - meantime],
noux' VFS implementation has become independent from the noux runtime and is
now prominently employed by Genode's C runtime and the VFS server component.
Genode's C runtime became more and more complete, alleviating the use of noux
as POSIX compatibility layer except for programs that depended on a working
implementation of 'fork' and 'execve'.
The current release fills this remaining gap in Genode's C runtime by
providing 'fork', 'execve', and cousins such as 'wait4' and 'getpid' as
regular parts of the libc. This will eventually make noux redundant.
Note that this change does *NOT* make Genode reliant on POSIX. The C runtime
including the Unix features are entirely optional.
As one stepping stone of this undertaking, noux applications, which previously
had to be compiled for noux, have become binary compatible with the regular C
runtime. So one can execute programs like 'bash' directly as a Genode
component without any friction.
There are a few collateral improvements of Genode's dynamic linker and the C
runtime on the account of the new 'fork' and 'execve' implementation. E.g., in
addition to the already supported 'stdin', 'stdout', and 'stderr'
configuration, the libc can be instructed to initialize arbitrary file
descriptors as follows:
! <config>
! ...
! <libc ...>
! <fd id="3" path="/dev/log" writeable="yes" readable="no" seek="10"/>
! ...
! </libc>
! </config>
The libc-based implementation of 'fork' and 'execve' can be tried out via
the new _ports/run/bash.run_ script. Note that there are still a number of
limitations such as the lack of signal and ioctl handling. Pipes are not
supported, and shebangs ('#!') are not interpreted yet. That said, once those
missing pieces come into place, we can fade out the use of noux within Genode.
General system time concept
===========================
Briefly speaking, up to now there has been no notion of an overall concept of
system time in Genode. Components that need to have access to some kind of
real time are either configured locally, e.g., libc-based components access a
configured "device" (/dev/rtc), which just might be an inline file system
containing an artificial timestamp or the VFS RTC plugin, while other
components query some RTC session directly. Most of the time, this session is
provided by the 'rtc_drv' on x86 machines, which is somewhat costly as reading
the RTC via I/O ports takes time and is therefore done scarcely. For example,
the libc will query an RTC source only once and uses this initial value to
interpolate the current time. However, for executing long-running components,
it will be necessary to adjust the clock to compensate for any occurring clock
drift or to correct a misconfigured clock in general. In addition it is
desirable to be able to use a remote time source, e.g., an NTP-server, to
synchronize the system time.
To address this, we came up with the following concept:
[image system_rtc]
The new "System RTC" component, located at
_repos/libports/src/server/system_rtc_, acts as proxy for the RTC service in
front of the actual RTC driver. It uses the driver to get the initial RTC
value and then uses a timer session (via the timeout framework) to locally
interpolate the time. In contrast to querying the RTC driver, querying the
System RTC is fast.
The RTC driver and the System RTC are bundled up together in the new
_drivers-rtc-pc_ package. The runtime of this package requests two ROM modules
used to update the RTC value. The first one, named 'system_set_rtc', is used
to update the proxy component while the second one, called 'hw_set_rtc', is
used by the RTC driver to write the value into the battery-backed RTC. A
separate component, potentially accessing a remote time source, may generate
these ROMs to adjust the time in the package's runtime.
The new native *SNTP* client at _repos/libports/src/app/sntp_client_ is such a
component. It periodically requests the current time from a given SNTP server
and generates a report. The report produced by the component contains the time
as UTC/GMT. Depending on the system policy, it can be used to update the time
of the System RTC and/or instruct the driver to set the RTC value.
To propagate such changes to RTC values, the RTC session was enhanced by the
new 'set' signal. A client of the session can install a signal handler to
adapt its own time when necessary. Based on this, the time back end of the
libc was changed to instantiate a watch handler for the RTC device, which,
when triggered, will cause the libc to re-read the RTC value.
This constellation should, under normal operation, allow for second to
sub-second granularity updates of the overall system time and avoid drifting
away from network time.
Accessing SMBIOS tables
=======================
The System Management BIOS (SMBIOS) is a specification that allows for reading
management information produced by the BIOS of a system as a collection of
data structures in memory. It has the potential to eliminate the need for the
operating system to probe hardware for discovering present devices and their
characteristics. Nowadays, the SMBIOS specification is implemented widely in
PC systems, which includes modern UEFI systems as well. The data structures
are referred to as _tables_ or _records_ by public documentation.
The new native SMBIOS decoder at _os/src/app/smbios_decoder_ can be used on
x86 to parse SMBIOS tables and report gathered information in a human-readable
way. Besides general table information like number and size of structures,
etc., the component supports complete parsing of SMBIOS structures of types
"BIOS", "System", and "Baseboard".
The component is free from any code for acquiring an SMBIOS table through
means like the bootloader or BIOS information. It expects a table to be
present through a regular Genode ROM session with a 'smbios_table' label. This
way, the underlying system is required to find, select, and save the raw table
on startup and create a ROM module out of it. This is currently achieved on
NOVA and base-hw through an interplay of kernel, the core component, and the
ACPI driver and was tested for legacy BIOSes as well as UEFI systems.
Clipboard
=========
Genode introduced a principle copy-and-paste mechanism already
[https://genode.org/documentation/release-notes/15.11#Copy_and_paste - four years ago].
However, originally created as a part of a tech demo, the mechanism remained
unused in our day to day Genode work. This changed now. We took the
integration of copy-and-paste support in Sculpt OS as an opportunity to revive
and refine the existing mechanism and supplement it with the features needed
to make it practical for daily use. We believe that the result aligns ease of
use nicely with security. The concept is described in a
[https://genodians.org/nfeske/2019-07-03-copy-paste - dedicated article]
at Genodians.org.
On a technical level, the existing clipboard component has received a new
option that allows for dynamic information-flow policies based on user
interactivity (keyboard focus, activity). When setting the config attribute
'match_labels="yes"', the clipboard performs plausibility checks for copy and
paste operations against the focus of the Nitpicker GUI server. All aspects of
the clipboard policy - including information-flow policies - have become
reconfigurable.
To make window-manager clients compatible with the clipboard's dynamic policy,
the window manager got enhanced with the ability to proxy the interaction with
the clipboard. GUI clients in turn - in particular the graphical *terminal* -
became able to interact with the clipboard. With the '<config>' attribute
'copy="yes"' specified, the terminal allows the user to select text to be
reported to a "clipboard" report. The selection mode is activated by holding
the left shift key. While the selection mode is active, the text position
under the mouse pointer is highlighted and the user can select text via the
left mouse button. Upon release of the mouse button, the selection is
reported. Vice versa, with the '<config>' attribute 'paste="yes"' specified,
the terminal allows the user to paste the content of a "clipboard" ROM session
to the terminal client by pressing the middle mouse button.
Finally, we integrated those new abilities into Sculpt OS and into several
installable packages, including virtual machines, the noux-system package,
and graphical Qt5-based applications.
Enhanced SSH terminal
=====================
This release paves the way for remotely managing Genode devices over SSH.
Until now, only interactive SSH sessions were supported. It is now possible to
execute commands from a remote SSH client. E.g., 'ssh noux@localhost -p 5555
"ls -hal /bin/"'. For non-interactive sessions, ssh_terminal requires a helper
component. This component is responsible to create the environment for the
command to run in. You can find an example for such a component at
_gems/src/test/exec_terminal_. It starts noux in a sub init and executes the
provided command inside of it. The new _ssh_exec_channel.run_ script gives a
demonstration on how this feature can be used.
This work is a contribution by Sid Hussmann of
[https://gapfruit.com - Gapfruit]. Thanks for this great new feature!
Storage-stack improvements
==========================
The desire of one Genode developer to exchange data via Iomega ZIP drives
between an Atari Falcon and Sculpt OS called for a number of small
improvements across several components of the storage stack.
First, the USB-block driver has been changed to exit on an initialization
failure instead of waiting for another (supported) device. This change enables
the Sculpt manager to detect such conditions and release the USB device
hardware by removing the driver component. Such a failed initialization may
happen with exotic USB-storage devices such as ZIP drives. With the device
released, however, it can be assigned to a virtual machine to access it using
a guest OS with a broader support of devices.
Second, the USB-block driver received new support for issuing the SCSI
START-STOP command at initialization time, thereby overcoming the ZIP-drive
initialization failure.
Third, we enhanced the part-block component with the ability to parse AHDI
partition schemes and detect the GEMDOS variant of FAT as used by Atari TOS.
Fourth, we enabled the Rump VFS plugin to access GEMDOS file systems. The
GEMDOS variant is readily supported by NetBSD's "msdos" file-system driver.
However, it must explicitly be enabled by a mount flag. Hence, we added the
principle ability for passing mount flags to NetBSD file-system drivers and
enabled the MSDOSFSMNT_GEMDOSFS flag based on the VFS plugin's config
attribute 'gemdos="yes"'.
With these changes in place, data can now be exchanged directly between
Atari-formatted disks and Sculpt OS. That said, advanced use cases such as
media changes at runtime are not covered yet.
Updated Ada/SPARK runtime
=========================
Genode's Ada/SPARK runtime is developed and maintained by
[https://componolit.com - Componolit]. Thanks for this excellent
collaboration!
The updated Componolit Ada runtime 1.1.0 increases the proof coverage and
cleans up the source-code structure. SPARK mode is now enabled wherever
possible and unneeded abstractions have been removed. Furthermore, the 64-bit
addition and subtraction have been proven to be free of runtime errors.
As a new feature, the runtime now supports the use of inline assembly in Ada.
The removal of unneeded features such as the incomplete threading support for
the secondary stack has greatly reduced the runtime's complexity while keeping
the current functionality available. Also GNAT.IO has been removed as its
implementation was incomplete and complex. A simpler replacement has been
introduced with 'Componolit.Runtime.Debug'.
Unrelated to Genode, the runtime now supports [https://muen.sk/ - Muen] and
the API/ABI of the runtime has been separated from the GNAT ABI.
Libraries and applications
##########################
Updated Qt5
===========
We updated our Qt5 port to the latest upstream version 5.13.0. Before
preparing the 'qt5' port, please make sure to build and install the updated
Qt5 host tools with the 'tool/tool_chain_qt5' script.
Virtualization
==============
As follow-up of our work on the
[https://genode.org/documentation/release-notes/19.05#Kernel-agnostic_virtual-machine_monitors - kernel agnostic virtual-machine monitor interface]
on x86, we added principle support to run our port of VirtualBox on
Genode/Fiasco.OC. We write _principle_ support, since we managed to get the
VMM running with Fiasco.OC, but unfortunately not all features required by the
VMM are available using the Fiasco.OC kernel, e.g., guest FPU registers, PDPTE
registers, and task-priority support. In practice this means that the VMs with
Windows and Linux come up to a certain point but will fail later whenever the
guest state runs out of synchronization between VMM and hardware. In contrast,
the Seoul VMM runs fine on Fiasco.OC since it does not depend on the mentioned
missing features.
Our main working items have been the completion of transfer of the available
guest registers and control flow synchronization improvements between VMM and
Fiasco.OC kernel. Additionally, the usage of priorities for VirtualBox's
pthreads in the VMM had to be disabled. Finally, some tests for VirtualBox
with Genode/Fiasco.OC are enabled for nightly regular testing now.
As a side topic, we added support for using the VirtualBox
[https://forums.virtualbox.org/viewtopic.php?f=2&t=82299&start=15 - CPU profile]
feature, which allows for presenting a different CPUID to the VM than the one
of the real CPU. This can help when running Windows 7 on a Kaby Lake or newer
CPU, which are considered _unsupported hardware_ and reason enough not to
receive security updates from Microsoft. The feature can be used on Genode by
adding the 'CpuProfile' attribute to the '<CPU>' XML node in the .vbox file,
like:
! <CPU CpuProfile="Intel Core i7-5600U">
Disposable VM for handling captive portals
==========================================
It is common that WiFi networks require the user to interact with a specific
web page before gaining access to full network functionality. Such captive
portal pages are completely individual to the accessed network and not limited
in the use of common web techniques. Therefore, their handling is best be done
using a fully-featured web browser like Mozilla Firefox.
This is where, in a Genode-based desktop system like Sculpt, a disposable VM
for hosting a minimal browser setup becomes desirable. Its goal is to unlock a
network for the native Genode surroundings with as little inconvenience as
possible just to be thrown away afterwards without any side effects on the
system.
Now, one could use the Firefox appliance VM of Sculpt (see the
[https://genode.org/documentation/release-notes/18.05 - release notes] or the
[https://genodians.org/alex-ab/2019-03-06-disposal-browser-vm - Genodians article])
for this. But this VM aims for a long-term browsing experience which, in the
context of mere captive-portal handling, brings some drawbacks like a much
higher RAM consumption or the required sessions for USB detection and shared
folders.
Furthermore, in the captive portal VM, there's no need for managing windows or
browser tabs. The one browser tab needed can always be shown in fullscreen. It
is also unnecessary for the browser to maintain a content cache or remember
user data. This can reduce resource consumption.
[image captive_portal_vm]
The VM we came up with is provided as package for Sculpt by Martin Stein
(depot user 'mstein'). You'll possibly need to manually add Martin's
[https://github.com/genodelabs/genode/tree/master/depot/mstein - depot key and download location]
to your Sculpt depot directory. After enabling this user, the captive portal
VM can be found in the Sculpt menu under "Depot -> mstein -> Virtual
Machines -> vbox5-nova-captive-portal".
The VM is based on a TinyCore 10 Linux with Xserver, i3 WM, and a tailored
Firefox browser. The package runtime doesn't need access to your file system,
it merely loads some ROMs into a RAM FS, so, it will completely forget any
changes made during a session. Therefore, it's also safe to simply remove an
instance via the Leitzentrale component-view once you don't need it anymore.
The guest additions are also included to make the VM window resizable.
Build system and tools
######################
At Genode Labs, we have used _tool/autopilot_ for the steering of tests in our
Continuous Integration workflow for almost a decade now. This implied various
improvements over the years and with the completion of our work on
[https://genode.org/documentation/release-notes/19.05#Unified_build_directories_for_ARM - unified build directories]
it was time to amend this handy tool once again. Unified build directories
support building all components for one CPU architecture in one directory
saving the build server from the redundant work we previously had with
board-specific directories. With the new notion of boards during builds, the
definition of the target platform when integrating Genode system scenarios is
now a triplet of _CPU architecture_, _board_, and _kernel_. This is reflected
in the new '-t <architecture-board-kernel>' command line option, which
instructs autopilot to generate a build directory for _architecture_ and
execute tests for the _board-kernel_ combination.
! autopilot -t x86_64-pc-sel4 -t x86_64-pc-nova -r log
The known options for '-k kernel' and '-p platform' are still supported with
the small change that the platform must now be defined as
_architecture-board_.
! autopilot -p x86_64-pc -k sel4 -k nova -r log
Autopilot now also documents the hidden feature to propagate custom 'RUN_OPTs'
via the 'RUN_OPT_AUTOPILOT' environment variable to the run tool executed.
Besides that, the tool always appends 'RUN_OPT' with '--autopilot'.
! RUN_OPT_AUTOPILOT="--depot-dir /data/depot" autopilot ...

815
doc/release_notes/19-11.txt Normal file
View File

@@ -0,0 +1,815 @@
===============================================
Release notes for the Genode OS Framework 19.11
===============================================
Genode Labs
On our [https://genode.org/about/road-map - road map] for this year, we stated
"bridging worlds" as our guiding theme of 2019. The current release pays
tribute to this ambition on several accounts.
First, acknowledging the role of POSIX in the real world outside the heavens
of Genode, the release vastly improves our (optional) C runtime with respect
to the emulation of POSIX signals, execve, and ioctl calls. With the line of
work described in Section [C runtime with improved POSIX compatibility], we
hope to greatly reduce the friction when porting and hosting existing
application software directly on Genode.
Second, we identified the process of porting or developing application
software worth improving. Our existing tools were primarily geared to
operating-system development, not application development. Application
developers demand different work flows and tools, including the freedom to use
a build system of their choice.
Section [New tooling for bridging existing build systems with Genode]
introduces our new take on this productivity issue.
Third, in cases where the porting of software to Genode is considered
infeasible, virtualization comes to the rescue. With the current release, a
new virtual machine monitor for the 64-bit ARM architecture enters the
framework. It is presented in Section [Virtualization of 64-bit ARM platforms].
As another goal for 2019, we envisioned a solution for block-level device
encryption, which is a highly anticipated feature among Genode users. We are
proud to present the preliminary result of our year-long development in
Section [Preliminary block-device encrypter].
Preliminary block-device encrypter
##################################
Over the past year, we worked on implementing a block-device encryption
component that makes use of the
[https://en.wikipedia.org/wiki/SPARK_(programming_language) - SPARK]
programming language for its core logic. In contrast to common
block-device encryption techniques where normally is little done besides
the encryption of the on-disk blocks, the _consistent block encrypter (CBE)_
aims for more. It combines multiple techniques to ensure integrity -
the detection of unauthorized modifications of the block-device -
and robustness against data loss. Robustness is achieved by keeping snapshots
of old states of the device that remain unaffected by the further operation of
the device. A copy-on-write mechanism (only the differential changes to the
last snapshot are stored) is employed to maintain this snapshot history with
low overhead. To be able to access all states of the device in the same manner,
some kind of translation from virtual blocks to blocks on the device is needed.
Hash-trees, where each node contains the hash of its sub-nodes, combine the
aspect of translating blocks and ensuring their integrity in an elegant way.
During the tree traversal, the computed hash of each node can be easily checked
against the hash stored in the parent node.
The CBE does not perform any cryptography by itself but delegates
cryptographic operations to another entity. It neither knows nor cares about
the used algorithm. Of all the nodes in the virtual block device (VBD), only
the leaf nodes, which contain the data, are encrypted. All other nodes, which
only contain meta-data, are stored unencrypted.
Design
------
As depicted in Figure [cbe_trees], all information describing the various
parts of the CBE is stored in the superblock. The referenced VBD is a set of
several hash trees, each representing a certain device state including the
current working state. Only the tree of the current working state is used to
write data to the block device. All other trees represent snapshots of older
states and are immutable. Each stored device state has a generation number
that provides the chronological order of the states.
As you can see, in the depicted situation, there exist four device states - the
snapshot with generation 3 is the oldest, followed by two newer snapshots and
generation 6 that marks the working state of the virtual device. The tree with
generation 6 is the current working tree. Each tree contains all changes done
to the VBD since the previous generation (for generation 6 the red nodes). All
parts of a tree that didn't change since the previous generation are references
into older trees (for generation 6 the gray nodes). Note that in the picture,
nodes that are not relevant for generation 6 are omitted to keep it manageable.
The actual data blocks of the virtual device are the leaf nodes of the trees,
shown as squares.
[image cbe_trees]
Whenever a block request from the client would override data blocks in
generation 6 that are still referenced from an older generation, new blocks for
storing the changes are needed. Here is where the so-called _Free Tree_ enters
the picture. This tree contains and manages the spare blocks. Spare blocks are
a certain amount of blocks that the CBE has in addition to the number of blocks
needed for initializing the virtual device. So, after having initialized a
virtual device, they remain unused and are only referenced by the Free Tree.
Therefore, in case the VBD needs new blocks, it consults the Free Tree (red
arrow).
In the depicted situation, writing the first data block (red square) would
require allocating 4 new blocks as all nodes in the branch leading to the
corresponding leaf node - including the leaf node itself - have to be written.
In contrast, writing the second data block would require allocating only one
new block as the inner nodes (red circles) now already exist. Subsequent write
requests affecting only the new blocks will not trigger further block
allocations because they still belong to the current generation and will be
changed in-place. To make them immutable we have to create a new snapshot.
The blocks in generation 5 that were replaced by the change to generation 6
(blue nodes) are not needed for the working state of the virtual device
anymore. They are therefore, in exchange for the allocated blocks, added to
the Free Tree. But don't be fooled by the name, they are not free for
allocation yet, but marked as "reserved" only. This means, they are
potentially still part of a snapshot (as is the case in our example) but the
Free Tree shall keep checking, because once all snapshots that referenced the
blue blocks have disappeared, they become free blocks and can be allocated
again.
To create a new snapshot, we first have to make all changes done to the VBDs
working state as well as the Free Tree persistent by writing all corresponding
blocks to the block-device. After that, the new superblock state is written to
the block-device. To safeguard this operation, the CBE always maintains several
older states of the superblock on the block device. In case writing the new
state of the superblock fails, the CBE could fall back to the last state that,
in our example, would contain only generations 3, 4, and 5. Finally, the
current generation of the superblock in RAM is incremented by one (in the
example to generation 7). Thereby, generation 6 becomes immutable.
A question that remains is when to create snapshots. Triggering a snapshot
according to some heuristics inside the CBE might result in unnecessary
overhead. For instance, the inner nodes of the tree change frequently during a
sequential operation. We might not want them to be re-allocated all the time.
Therefore, the creation of a snapshot must be triggered explicitly from the
outside world. This way, we can accommodate different strategies, for
instance, client-triggered, time-based, or based on the amount of data
written.
When creating a snapshot, it can be specified whether it shall be disposable
or persistent. A disposable snapshot will be removed automatically by the CBE
in two situations, either
* When there are not enough usable nodes in the Free Tree left to
satisfy a write request, or
* When creating a new snapshot and all slots in the superblock that might
reference snapshots are already occupied.
A persistent snapshot, or quarantine snapshot, on the other hand will never be
removed automatically. Its removal must be requested explicitly.
During initialization, the CBE selects the most recent superblock and reads the
last generation value from it. The current generation (or working state
generation) is then set to the value incremented by one. Since all old blocks,
that are still referenced by a snapshot, are never changed again, overall
consistency is guaranteed for every generation whose superblock was stored
safely on disk.
Implementation
--------------
Although we aimed for a SPARK implementation of the CBE, we saw several
obstacles with developing it in SPARK right from the beginning. These obstacles
mainly came from the fact that none of us was experienced in designing
complex software in SPARK. So we started by conducting a rapid design-space
exploration using our mother tongue (C++) while using only language features
that can be mapped 1:1 to SPARK concepts. Additionally, we applied a clear
design methodology that allowed us to keep implementation-to-test cycles
small and perform a seamless and gradual translation into SPARK:
* _Control flow_
The core logic of the CBE is a big state machine that doesn't block. On each
external event, the state machine gets poked to update itself accordingly.
C++ can call SPARK but SPARK never calls C++. The SPARK code therefore
evolves as self-contained library.
* _Modularity_
The complex state machine of the CBE as a whole is split-up into smaller
manageable sub-state-machines, working independently from each other. These
modules don't call each other directly. Instead, an additional superior
module handles the interplay. This is done by constantly iterating over all
modules with the following procedure until no further progress can be made:
# Try to enter requests of other modules into the current one
# Poke the state machine of the current module
# The current module may have generated requests - Try to enter them into
the targeted modules
# The current module may have finished requests - Acknowledge them at the
modules they came from
Each module is represented through a class (C++) respectively a package with
a private state record (SPARK).
* _No global state_
There are no static (C++) or package (SPARK) variables. All state is kept in
members of objects (C++) respectively records (SPARK). All packages are pure
and sub-programs have no side-effects. Therefore, memory management and
communication with other components is left to OS glue-code outside the
core logic.
This approach worked out well. Module by module, we were able to translate the
C++ prototype to SPARK without long untested phases, rendering all regression
bugs manageable. In Genode, the CBE library is currently integrated through
the CBE-VFS plugin. Figure [cbe_modules] depicts its current structure and the
integration via VFS plugin.
[image cbe_modules]
The green and blue boxes each represent an Ada/SPARK package. The translation
to SPARK started at the bottom of the picture moving up to the more abstract
levels until it reached the Library module. This is the module that handles
the interplay of all other modules. Its interface is the front end of the CBE
library. So, all green packages are now completely written in SPARK and
together form the CBE library. Positioned above, the CXX library in blue is
brought in by a separate library and exports the CBE interface to C++. This
way, the CBE can also be used in other environments including pure SPARK
programs. The CXX Library package is not written in SPARK but Ada and performs
all the conversions and checks required to meet the preconditions set by the
SPARK packages below.
At the C++ side, we have the VFS plugin. Even at this level, the already
mentioned procedure applies: The plugin continuously tries to enter requests
coming from the VFS client (above) into the CBE (below), pokes the CBE state
machine, and puts thereby generated block/crypto requests of the CBE into the
corresponding back-ends (left). This process is repeated until there is no
further progress without waiting for an external event.
Current state
-------------
In its current state, the CBE library is still pretty much in flux and is not
meant for productive use.
As the Free Tree does not employ copy-on-write semantics for its meta-data, a
crash, software- or hardware-wise, will corrupt the tree structure and renders
the CBE unusable on the next start.
This issue is subject to ongoing work. That being said, there are
components that, besides being used for testing, show how the interface of the
CBE library lends itself to be integrated in components in different ways. At
the moment, there are two components making use of the CBE library as
block-device provider.
The first one is the aforementioned CBE-VFS plugin. Besides r/w access to the
working tree and r/o access to all persistent snapshots, it also provides a
management interface where persistent snapshots can be created or discarded.
Its current layout is illustrated by Figure [cbe_vfs]. The VFS plugin
generates three top directories in its root directory. The first one is the
_control_ directory. It contains several pseudo files for managing the CBE:
[image cbe_vfs]
:'key': set a key by writing a string into the file.
:'create_snapshot': writing 'true' to this file will attempt to create
a new snapshot. (Eventually the snapshot will
appear in the 'snapshots' directory if it could be
created successfully.)
:'discard_snapshot': writing a snapshot ID into this file will discard
the snapshot
The second is the 'current' directory. It gives access to the current
working tree of the CBE and contains the following file:
:'data': this file represents the virtual block device and gives
read and write access to the data stored by the CBE.
The third and last is the 'snapshots' directory. For each persistent snapshot,
there is a sub-directory named after the ID of the snapshot. This directory,
like the 'current' directory, contains a 'data' file. This file, however,
gives only read access to the data belonging to the snapshot.
The CBE-VFS plugin itself uses the VFS to access the underlying block device.
It utilizes the file specified in its configuration. Here is a '<vfs>'
snippet that shows a configured CBE-VFS plugin where the block device is
provided by the block VFS plugin.
! <vfs>
! <dir name="dev">
! <block name="block"/>
! <cbe name="cbe" block="/dev/block"/>
! </dir>
! </vfs>
An exemplary ready-to-use run script can be found in the CBE repository
at _run/cbe_vfs_snaps.run_. This run script uses a bash script to
automatically perform a few operations on the CBE using the VFS plugin.
Afterwards it will drop the user into a shell where further operations
can be performed manually, e.g.:
! dd if=/dev/zero of=/dev/cbe/current/data bs=4K
The second component is the CBE server. In contrast to the CBE-VFS plugin,
it is just a simple block-session proxy component that uses a block connection
as back end to access a block-device. It provides a front-end block session to
its client, creates disposable snapshots every few seconds, and uses the
'External_Crypto' library to encrypt the data blocks using AES-CBC-ESSIV. The
used key is a plain passphrase. The following snippet illustrates its
configuration:
! <start name="cbe">
! <resource name="RAM" quantum="4M"/>
! <provides><service name="Block"/></provides>
! <config sync_interval="5" passphrase="All your base are belong to us"/>
! </start>
The _run/cbe.run_ run script in the CBE repository showcases the use of the
CBE server.
Both run scripts will create the initial CBE state in a RAM-backed
block device that is then accessed by the CBE server or the CBE-VFS
plugin.
The run-script and the code itself can be found on the
[https://github.com/cnuke/cbe/tree/cbe_19.11 - cbe/cbe_19.11] branch on
GitHub. If you intend to try it out, you have to checkout
the corresponding
[https://github.com/cnuke/genode/tree/cbe_19.11 - genode/cbe_19.11]
branch in the Genode repository as well.
Future plans
------------
Besides addressing the current shortcomings and getting the CBE library
production-ready so that it can be used in Sculpt, there are still
a few features that are currently unimplemented. For one we would like
to add support for making it possible to resize the VBD as well as the
Free Tree. For now the geometry is fixed at initialization time and cannot
be changed afterwards. Furthermore, we would like to enable re-keying,
i.e., changing the used cryptographic key and re-encrypting the tree
set of the VBD afterwards. In addition to implementing those features, the
overall tooling for the CBE needs to be improved. E.g., there is currently
no proper initialization component. For now, we rely on a component
that was built merely as a test vehicle to generate the initial trees.
Virtualization of 64-bit ARM platforms
######################################
Genode has a long history regarding support of all kinds of
virtualization-related techniques including
[https://genode.org/documentation/release-notes/9.11#Paravirtualized_Linux_on_Genode_OKL4 - para-virtualization],
[https://genode.org/documentation/articles/trustzone - TrustZone],
hardware-assisted virtualization on
[https://genode.org/documentation/articles/arm_virtualization - ARM],
[https://genode.org/documentation/release-notes/13.02#Full_virtualization_on_NOVA_x86 - x86],
up to the full virtualization stack of
[https://genode.org/documentation/release-notes/14.02#VirtualBox_on_top_of_the_NOVA_microhypervisor - VirtualBox].
We regard those techniques as welcome stop-gap solutions for using non-trivial
existing software stacks on top of Genode's clean-slate OS architecture. The
[https://genode.org/documentation/release-notes/19.05#Kernel-agnostic_virtual-machine_monitors - recent]
introduction of a kernel-agnostic interface to control virtual machines (VM)
ushered a new level for the construction respectively porting of
virtual-machine monitors (VMM). By introducing a new ARMv8-compliant VMM
developed from scratch, we continue this line of work.
The new VMM builds upon our existing proof-of-concept (PoC) implementation for
ARMv7 as introduced in release
[https://genode.org/documentation/release-notes/15.02#Virtualization_on_ARM - 15.02].
In contrast to the former PoC implementation, however, it aims to be complete
to a greater extent. Currently, it comprises device models for the following
virtual hardware:
* RAM
* System Bus
* CPU
* Generic Interrupt Controller v2 and v3
* Generic Timer
* PL011 UART (limited)
* Pass-through devices
The VMM is able to load diverse 64-bit Linux kernels including
Device-Tree-Binary (DTB) and Initramfs. Currently, the implementation uses a
fixed memory layout for the guest-physical memory view, which needs to be
reflected by the DTB used by the guest OS. An example device-tree source file
can be found at _repos/os/src/server/vmm/spec/arm_v8/virt.dts_. The actual VMM
is located in the same directory.
Although support for multi-core VMs is already considered internally, it is
not yet finished. Further outstanding features that are already in development
are Virtio device model support for networking and console. As the first - and
by now only - back end, we tied the VMM to the ARMv8 broadened Kernel-agnostic
VM-session interface as implemented by Genode's custom base-hw kernel. As a
side effect of this work, we consolidated the generic VM session interface
slightly. The RPC call to create a new virtual-CPU now returns an identifier
for identification.
The VMM has a strict dependency on ARM's hardware virtualization support
(EL2), which comprises extensions for the ARMv8-A CPU, ARM's generic timer,
and ARM's GIC. This rules out the Raspberry Pi 3 board as a base platform
because it does not include a GIC but a custom interrupt-controller without
hardware-assisted virtualization of interrupts. To give the new VMM a try, we
recommend using the run script _repos/os/run/vmm_arm.run_ as a starting point
for executing the VMM on top of the i.MX8 Evaluation Kit board.
New tooling for bridging existing build systems with Genode
###########################################################
Genode's development tools are powerful and intimidating at the same time.
Being designed from the perspective of a whole-systems developer, they put
emphasis on the modularity of the code base (separating concerns like
different kernels or system abstraction levels), transitive dependency
tracking between libraries, scripting of a wide variety of system-integration
tasks, and the continuous integration of complete Genode-based
operating-system scenarios. Those tools are a two-edged sword though.
On the one hand, the tools are key for the productivity of seasoned Genode
developers once the potential of the tools is fully understood and leveraged.
For example, during the development of Sculpt OS, we are able to
change an arbitrary line of code in any system component and can test-drive
the resulting Sculpt system on real hardware within a couple of seconds.
As another example, the almost seamless switching from one OS kernel to
another has become a daily routine that we just take for granted without
even thinking about it.
On the other hand, the sophistication of the tools stands in the way of
application developers who are focused on a particular component instead
of the holistic Genode system. In this case, the powerful system-integration
features remain unused but the complexity of the tools and the build system
prevails. Speaking of build systems, this topic is ripe of emotions
anyway. _Developers use to hate build systems._ Forcing Genode's build
system down the throats of application developers is probably not the best
idea to make Genode popular.
This line of thoughts prompted us to re-approach the tooling for Genode from
the perspective of an application developer. The intermediate result is a new
tool called Goa:
:Goa project at GitHub:
[https://github.com/nfeske/goa]
Unlike Genode's regular tools, Goa's work flow is project-centered. A project
is a directory that may contain source code, data, instructions how to
download source codes from a 3rd party, descriptions of system scenarios, or
combinations thereof. Goa is independent from Genode's regular build system.
It combines Genode's package management (depot) with commodity build systems
such a CMake. In addition to building and test-driving application software
directly on a Linux-based development system, Goa is able to aid the process
of exporting and packaging the software in the format expected by Genode
systems like Sculpt OS.
At the current stage, Goa should be considered as work in progress. It's a new
approach and its success is anything but proven. That said, if you are
interested in developing or porting application software for Genode, your
feedback would be especially valuable. As a starting point, you may find the
following introductory article helpful:
:Goa - streamlining the development of Genode applications:
[https://genodians.org/nfeske/2019-11-25-goa]
Base framework and OS-level infrastructure
##########################################
File-system session
===================
The file-system session interface received a much anticipated update.
Writing modification times
--------------------------
The new operation WRITE_TIMESTAMP allows a client to update the modification
time of a file-system node. The time is defined by the client to keep
file-system servers free from time-related concerns. The VFS server implements
the operation by forwarding it to the VFS plugin interface. At present, this
new interface is implemented by the rump VFS plugin to store modification
times on EXT2 file systems.
Enhanced file-status info
-------------------------
The status of a file-system node as returned by the 'File_system::Status'
operation has been revisited. First, we replaced the fairly opaque "mode" bits -
which were an ad-hoc attempt to stay compatible with Unix - with the explicit
notion of 'readable', 'writeable', and 'executable' attributes. We completely
dropped the notion of users and groups. Second, we added the distinction
between *continuous* and *transactional* files to allow for the robust
implementation of continuous write operations across component boundaries. A
continuous file can be written-to via a sequence of arbitrarily sized chunks
of data. For such files, a client can split a large write operation into any
number of smaller operations in accordance to the size of the used I/O
buffers. In contrast, a write to a transactional file is regarded as a
distinct operation. The canonical example of a transactional file is a
socket-control pseudo file.
Virtual file-system infrastructure
==================================
First fragments of a front-end API
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The VFS is mostly used indirectly via the C runtime. However, it is also
useful for a few components that use the Genode API directly without any
libc. To accommodate such users of the VFS, we introduced the front-end
API at _os/vfs.h_ that covers a variety of current use cases. Currently, those
use cases revolve around the watching, reading, and parsing of files and
file-system structures - as performed by Sculpt's deployment mechanism.
Writing to files is not covered.
Improved file-watching support
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
All pseudo files that use the VFS-internal 'Readonly_value_file_system'
utility have become able to deliver watch notifications. This change enables
VFS clients to respond to VFS-plugin events (think of terminal resize)
dynamically.
Speaking of the *terminal VFS plugin*, the current release enhances the plugin
in several respects. First, it now delivers status information such as the
terminal size via pseudo files. Second, we equipped the VFS terminal file
system with the ability to detect user interrupts in the incoming data stream,
and propagate this information via the new pseudo file '.terminal/interrupts'.
Each time, the user presses control-c in the terminal, the value stored in
this pseudo file is increased. Thereby, a VFS client can watch this file to
get notified about the occurrences of user interrupts.
VFS plugin for emulating POSIX pipes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We added a new VFS plugin for emulating POSIX pipes. The new plugin creates
pipes between pairs of VFS handles. It replaces the deprecated libc_pipe
plugin. In contrast to the libc_pipe plugin, which was limited to pipes within
one component, the new VFS plugin can also be used to establish pipes between
different components by mounting the plugin at a shared VFS server.
C runtime with improved POSIX compatibility
===========================================
Within Genode, we used to think of POSIX as a legacy that is best avoided.
In fact, the foundational components of the framework do not depend on a
C runtime at all. However, higher up the software stack - at the latest when
3rd-party libraries enter the picture - a working C runtime is unavoidable. In
this statement, the term "working" is rather muddy though. Since we have never
fully embraced POSIX, we were content with cutting corners here and there. For
example, given Genode's architecture, supporting 'fork' and 'execve' seemed
totally out of question because those mechanisms would go against the grain of
Genode.
However, our growing aspiration to bridge the gap between existing popular
applications and Genode made us re-evaluate our stance towards POSIX.
All technical criticism aside, POSIX is immensely useful because it is
a universally accepted stable interface. To dissolve friction between
Genode and popular application software, we have to satisfy the application's
expectations. This ignited a series of developments, in particular
the added support for 'fork' and 'execve' - of all things - in
[https://genode.org/documentation/release-notes/19.08#Consolidation_of_the_C_runtime_and_Noux - Genode 19.08],
which was nothing short of surprising, even to us.
The current release continues this line of development and brings the
following improvements.
Execve
------
The libc's 'execve' implementation got enhanced to evaluate the path of the
executable binary according to the information found on the VFS, in particular
by traversing directories and following symbolic links. This enables the libc
to execute files stored at sub directories of the file system.
Furthermore, 'execve' received handling for *executing shell scripts* by
parsing the shebang marker at the beginning of the executable file. This way,
the 'execve' mechanism of the libc reaches parity with the feature set of the
Noux runtime that we traditionally used to host Unix software on top of
Genode.
Modification-time handling
--------------------------
By default, the libc uses the just added facility for updating the timestamp
of file-system nodes when closing a written-to file, which clears the path
towards using tools like 'make' that rely on file-modifications times.
The libc's mechanism can be explicitly disabled by specifying
! <libc update_mtime="no"...>
This is useful for applications that have no legitimate access to a time
source.
Emulation of 'ioctl' operations via pseudo files
------------------------------------------------
With the current release, we introduce a new scheme of handling ioctl
operations, which maps 'ioctl' calls to pseudo-file accesses, similar to how
the libc already maps socket calls to socket-fs operations.
A device file can be accompanied with a (hidden) directory that is named after
the device file and hosts pseudo files for triggering the various device
operations. For example, for accessing a terminal, the directory structure
looks like this:
! /dev/terminal
! /dev/.terminal/info
! /dev/.terminal/rows
! /dev/.terminal/columns
! /dev/.terminal/interrupts
The 'info' file contains device information in XML format. The type of the XML
node corresponds to the device type. Whenever the libc receives a TIOCGWINSZ
ioctl for _/dev/terminal_, it reads the content of _/dev/.terminal/info_ to
obtain the terminal-size information. In this case, the _info_ file looks as
follows:
! <terminal rows="25" columns="80/>
Following this scheme, VFS plugins can support ioctl operations by providing
an ioctl directory in addition to the actual device file.
Emulation of POSIX signals
--------------------------
Even though there is no notion of POSIX signals at the Genode level, we
can reasonably emulate certain POSIX signals at the libc level. The current
release introduces the first bunch of such emulated signals:
:SIGWINCH: If 'stdout' is connected to a terminal, the libc watches the
terminal's ioctl pseudo file _.terminal/info_. Whenever the terminal
size changes, the POSIX signal SIGWINCH is delivered to the application.
With this improvement, Vim becomes able to dynamically adjust itself
to changed window dimensions when started as a native Genode component
(w/o the Noux runtime environment).
:SIGINT: If 'stdin' is connected to a terminal, the libc watches the
terminal's pseudo file _.terminal/interrupts_. Since, the terminal VFS
plugin modifies the file for each occurred user interrupt (control-c),
the libc is able to reflect such an event as SIGINT signal to the
application.
:Process-local signal delivery: The libc's implementation of 'kill' got
enhanced with the ability to submit signals to the local process.
Support for arbitrarily large write operations
----------------------------------------------
The number of bytes written by a single 'write' call used to be constrained by
the file's underlying I/O buffer size. Even though our libc correctly returned
this information to the application, we found that real-world applications
rarely check the return value of 'write' because partial writes do usually not
occur on popular POSIX systems. Thanks to the added distinction between
continuous and transactional files as described in Section
[File-system session], we became able to improve the libc's write operation to
iterate on partial writes to continuous files until the original write count
is reached. The split of large write operations into small partial writes as
dictated by the VFS infrastructure becomes invisible to the libc-using
application.
Input-event handling
====================
In Genode 19.08, we undertook a comprehensive rework of our keyboard-event
handling in the light of localization and also promised to tie up remaining
loose ends soon.
First, we again dived into our character generators for a thorough check of
our stack of keyboards and fixed remaining inconsistencies in French and
German layouts. En passant, we also increased the default RAM quotas for
the input filter to 1280K in our recipes to cope with the increased
layout-configuration sizes in corner cases.
Next - and more importantly - we subdued the monsters lurking in our Qt5
keyboard back end and enabled transparent support for system-wide keyboard
layout configuration for Qt5 components. One important change during this work
was to move the handling of control key sequences into the clients. For
example, the graphical terminal and Qt5 interpret key events in combination
with the CTRL modifier based on characters and, thus, support CTRL-A with
AZERTY and QWERTY layouts correctly. As a result we removed all CTRL modifier
(mod2) configurations from our character-generator configurations.
Finally we'd like to point out one important change of our rework that
repeatedly led to surprises: For keys without character mappings the reworked
character-generator mechanism emits invalid codepoints in contrast to
codepoints with value 0. For that reason, components interpreting character
events should check 'Codepoint::valid()' to prevent the processing of invalid
characters (and not the frequent pattern of 'codepoint.value != 0').
NIC router
==========
The NIC router has received the ability to report the link state of its NIC
interfaces (downlinks and uplinks). To control this mechanism, there are two
new boolean attributes 'link_state' and 'link_state_triggers' in the <report>
tag of the NIC router configuration. If the former is set to "true", the report
will contain the current link state for each interface:
! <domain name="domain1">
! <interface label="uplink1" link_state="false"/>
! <interface label="downlink1" link_state="true"/>
! </domain>
! <domain name="domain2">
! <interface label="downlink2" link_state="true"/>
! </domain>
The second attribute decides whether to trigger a report update each time the
link state of an interface changes. By default, both attributes are set to
"false".
Device drivers
==============
Platform driver on x86
~~~~~~~~~~~~~~~~~~~~~~
During our enablement of Genode on a
[https://genodians.org/chelmuth/2019-10-21-sculpt-elitebook - recent notebook],
we spotted some PC platform shortcomings, we address with this release. Most
prominently we added support for
[https://en.wikipedia.org/wiki/PCI_configuration_space#Bus_enumeration - 64-bit PCI base address registers]
to the x86 platform driver. This allows the use of PCI devices that are
assigned to physical I/O-memory regions beyond 4 GiB by the boot firmware.
Wireless driver
~~~~~~~~~~~~~~~
We added the firmware images for the 5000 and 9000 series of Intel wireless
devices to the firmware white-list in the _wifi_drv_ component. Such devices
as 5100AGN, 5300AGN and 5350AGN as well as 9461, 9462 and 9560 should now be
usable on Genode.
Libraries and applications
##########################
VirtualBox improvements
=======================
The GUI handling of our VirtualBox port got improved to react on window-size
changes more instantly. The effect is that an interactive adjustment of the
window size, e.g., on Sculpt, becomes quickly visible to the user. Still, the
VM may take some time to adjust to the resolution change, which ultimately
depends on the behavior of the driver of the VirtualBox guest additions.
Updated 3rd-party software
==========================
With the addition of the 64-bit ARM architecture (AARCH64) with the
[https://genode.org/documentation/release-notes/19.05#Broadened_CPU_architecture_support_and_updated_tool_chain - 19.05]
release, it became necessary to update the libraries the Genode tool chain
(gcc) depends on in order to support AARCH64 properly. This concerns the GNU
multi precision arithmetic library (gmp), which has been updated from version
4.3.2 to 6.1.2, as well as the libraries that depend on it: Multi precision
floating point (mpfr) and multi precision complex arithmetic (mpc). All those
old versions did not offer support for the AARCH64 architecture, which is a
requirement to make Genode self hosting. Targets for building binutils and GCC
within Genode for AARCH64 are in place, GNU make is in place, and even code
coverage (gcov) has been added. This work puts AARCH64 in line with other
supported CPU architectures and emphasizes our interest in the ARM 64-bit
architecture.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
With the previous release, Genode's base-hw kernel got extended to support the
ARMv8-A architecture in principle. The first hardware supported was the
Raspberry Pi 3 as well as the i.MX8 evaluation kit (EVK). But only a single
CPU-core was usable at that time. Now, we lifted this limitation. On both
boards, all four CPU-cores are available henceforth.
Removed components
##################
The current release removes the following components:
:gems/src/app/launcher:
The graphical launcher remained unused for a few years now. It is not
suitable for systems as flexible as Sculpt OS.
:os/src/app/cli_monitor:
CLI monitor was a runtime environment with a custom command-line interface
to start and stop subsystems. It was part of the user interface of our
first take on a Genode-based desktop OS called
[https://genode.org/documentation/release-notes/15.11#Genode_as_desktop_OS - Turmvilla].
Nowadays, we use standard command-line tools like Vim to edit init
configurations dynamically, which is more flexible and - at the same time -
alleviates the need for a custom CLI. The CLI-monitor component was too
limited for use cases like Sculpt anyway.
Along with the CLI monitor, we removed the ancient (and untested for long
time) _terminal_mux.run_ script, which was the only remaining user of the CLI
monitor.
:fatfs_fs, rump_fs, and libc_fatfs plugin:
The stand-alone file-system servers fatfs_fs and rump_fs as well as the
fatfs libc plugin have been superseded by the fatfs and rump VFS plugins.
The stand-alone servers can be replaced by using the VFS server plus the
corresponding VFS plugin as a drop-in replacement.

814
doc/release_notes/20-02.txt Normal file
View File

@@ -0,0 +1,814 @@
===============================================
Release notes for the Genode OS Framework 20.02
===============================================
Genode Labs
This year's [https://genode.org/about/road-map - road map] is all about making
Genode and Sculpt OS more approachable. It turns out that the first release of
the year already pays tribute to that goal. First, it equips Sculpt OS with a
much more logical and welcoming graphical user interface
(Section [Redesign of the administrative user interface of Sculpt OS]).
Second, it greatly reduces the friction when hosting existing applications on
Genode by smoothening several rough edges with respect to POSIX compatibility,
and by generally improving performance.
Most topics of the release are closely related to Sculpt. The biggest
break-though is certainly the ability of running Sculpt OS on 64-bit ARM
hardware (Section [Sculpt OS on 64-bit ARM i.MX8 hardware]) along with our
custom virtual machine monitor (VMM). On PC hardware, Sculpt users can enjoy
an updated audio driver and optimizations of the Seoul VMM. Furthermore,
Sculpt's window manager received the much anticipated ability to use virtual
desktops.
At the framework-API level, the most significant changes are the introduction
of dedicated types for inter-thread synchronization patterns
(Section [Base-framework refinements]) and a new library for
bringing the benefits of the Genode architecture to the application level
(Section [New sandbox library based on the init component]).
Redesign of the administrative user interface of Sculpt OS
##########################################################
On our [https://genode.org/about/road-map - road map] for 2020, we stated
the reducing of the barrier of entry as our main concern of the year.
We highlighted the ease of use of Sculpt OS as one particular work area.
Removing Unix from the picture
------------------------------
Until now, Sculpt's administrative user interface - lyrically called
Leitzentrale - employed a small Unix runtime and the Vim editor as utility for
basic file operations and for the tweaking of configurations. Even though this
was a practical intermediate solution, we have to face the fact that not
everyone loves the Unix command-line interface as much as we do. Quite the
opposite, actually. When presenting Sculpt, we can clearly sense that people
with a non-Unix background are put off by it. The audience generally loves the
runtime graph, visual cues, and discoverability. Furthermore, command-line
interfaces are (albeit wrongly) perceived as archaic and impenetrable relics
by many computer users who are otherwise perfectly happy with the notion of
files and directories. We identified that file-manipulation tasks performed in
the Leitzentrale are rare and simple. Relying on Unix for those basic tasks is
like taking a sledgehammer to crack a nut. On average, the Leitzentrale is
used in just a few moments a day for basic things like browsing a file-system
hierarchy, glimpsing at the reports stored on the report file system, deleting
or copying a file or two, or tweaking a configuration file. With a Unix shell
presenting one barrier, Vim is certainly an even higher one. Familiarity with
Vim should definitely not be a prerequisite for using an operating system.
Following this reasoning, we decided to swap out the command-line interface
and Vim by a simple GUI-based file browser and a notepad-like editor, which do
not require any learning curve.
Note that even once the Unix command-line interface is removed from Sculpt's
Leitzentrale, advanced users will still be able to manipulate Sculpt's config
file system via a Unix runtime deployed as a regular component, similar to the
use of the noux-system package we have today.
New user-interface layout
-------------------------
The move away from the command-line interface goes hand in hand with the
redesign of the overall user-interface layout. A new panel at the top of the
screen contains two centered tabs for switching between the runtime graph and
the file-system browser.
[image sculpt_20.02_panel]
The storage-management functionality has been moved from the former storage
dialog into the respective nodes of the runtime graph. E.g., to format a block
device, the user can now select a USB or storage node of the graph to get a
menu of block-device-level operations.
[image sculpt_20.02_storage]
The network-management is now located at a drop-down menu that can be toggled
via a button at the right side of the panel.
[image sculpt_20.02_network]
A new button on the left side of the panel allows the user to toggle a
drop-down menu for GUI settings. At the current time, there is only the option
to adjust the font size. In the future, the dialog will give easy access to
the screen-resolution options and the keyboard layout.
The log-message view is now hidden in another drop-down menu that can be
toggled via a panel button. So when starting the system, the user is greeted
with only the runtime graph, which is a much nicer and cleaner looking
experience.
Informative or diagnostic messages are displayed in the left-bottom corner of
the screen.
[image sculpt_20.02_message]
The "Files" tab of the panel switches the main screen area to a simple file
browser that lists all file systems available. By toggling one of the
file-system buttons, the directory hierarchy can be browsed. When hovering
a file, an "Edit" or "View" button appears, which can be used to open
the file in a text area that appears on the right side of the file browser.
The editor supports the usual notepad-like motions, operations, and
shortcuts (control-c for copy, control-v for paste, control-s for save).
[image sculpt_20.02_editor]
Half-way there
--------------
With the current release, one can already accomplish a lot without having to
resort to a command-line interface: connecting to the network, managing
storage devices, installing and deploying software, inspecting the system
state, and tweaking configurations.
There are still a few gaps though. In particular the file browser does
not yet support file operations like the copying, renaming, or removal of
files. For these tasks, the current version of Sculpt still features the
Unix-based inspect window, which can be accessed by toggling the "Inspect"
button inside the USB or storage dialog. Once selected, the panel presents an
"Inspect" tab that features the familiar Unix shell and Vim. Note, however,
that we keep the inspect window only as an interim solution. It will
eventually be removed. As with every new feature, there are still rough edges
to be expected in the editor and file browser, e.g., the editing of files with
long lines or the browsing of directories with many entries is not
appropriately covered yet.
To see the current new version of Sculpt OS in action, you may find the
following presentation entertaining.
:Live demonstration of Sculpt OS at FOSDEM 2020:
[https://fosdem.org/2020/schedule/event/uk_sculpt/]
The new version 20.02 of Sculpt OS is part of this release and can be built
from source and used right now. Several Genode developers already provide
ready-to-use packages for the new version. The software depots by alex-ab,
cnuke, skalk are worth exploring. A downloadable system image along with an
updated manual will be released shortly.
Sculpt OS on 64-bit ARM i.MX8 hardware
######################################
Within the past two releases, big steps were taken to support ARMv8 hardware in
the Genode OS framework. After implementing basic support for Raspberry Pi 3,
and the i.MX 8M Evaluation Kit, the network card was enabled for the latter.
Moreover, we updated the Linux TCP/IP, and C library ports, as well as
the Noux environment to support the architecture. Finally, with the latest
releases, a new ARMv8-compliant virtual-machine monitor for the base-hw kernel
entered the framework.
The rapid achievements motivated us to strive for a more ambitious scenario to
run on top of the currently focused ARMv8 hardware platform. So why not using
Sculpt OS on the i.MX 8M System-on-Chip?
Persistent storage
==================
There were several challenges to cope with initially. First, persistent
storage was needed. Luckily, the Genode OS framework contained already an
SD-card driver implementation for the i.MX series. The driver was written for
Genode from scratch and initially supported the i.MX53 SoC only. From then, it
got extended repeatedly to drive the SD-card controller of several i.MX6 and
i.MX7 platforms. Therefore, it was not a big issue to support the new hardware
too. However, when we later used it in Sculpt, it turned out that the driver
has some low-latency requirements. If those were not met, it got stuck. This
was the time where the CPU-quota mechanism came in handy in a real-world
scenario. It helped to let the interrupt handler of the driver be scheduled in
time, and thereby let the driver run stable.
Having a working block device is one part, but it is of little use without a
file system. In Sculpt OS, the NetBSD rump kernel's ext2 file-system is
typically used to host the depot package system and for keeping configuration
files persistent. Unfortunately, the version of NetBSD as used in Genode's
rump kernel port does not contain the ARMv8 architecture. Of course, we could
have upgraded the rump kernel as a whole. But this software stack is quite
complex with a lot of threads reproducing a sophisticated state machine. It
took some time in the past to meet its required semantics. Therefore,
backporting some header definitions and a few architecture-dependent functions
seemed more attractive. Luckily, it turned out to be the right decision, and
after a day of backporting work, the file system could run on ARMv8.
Display engine
==============
One of the more challenging tasks was certainly the enabling of the Display
Controller Subsystem (DCSS) of the i.MX 8M SoC. Originally, we hoped to profit
from our experiences with the Image Processing Unit (IPU), the display engine
of former i.MX SoCs. But as it turned out, the DCSS is a completely new
design, and has not much in common with the IPU. When first writing a driver
for the IPU of the i.MX53, we were surprised by the complexity and flexibility
of this piece of hardware. Back then, it took months to get something
meaningful working. To not lose too much time by re-implementing a driver from
scratch, we decided to take the DDE Linux approach, which worked out pretty
fast. The resulting driver should provide the same flexibility like the Linux
original one. However, as the i.MX 8M EVK board provides a HDMI connector
only, we did not test more than that. The configuration of the driver is
analogous to the Intel framebuffer driver, and looks like the following:
! <config>
! <connector name="HDMI-A-1" width="1920" height="1080" hz="60" enabled="true"/>
! </config>
Later, when using the driver in practice within the Sculpt OS, we could
experience a slightly sluggish behaviour, which was due to a missing
architectural back end of the blitting library of Genode. After tweaking this
too, the graphical user interface experience was good.
USB and Input
=============
The last missing I/O device to run Sculpt OS on the ARMv8 was something for
user generated input. Therefore, the existent USB host controller driver for
the i.MX series got updated. The only roadblock here was the powering of the
device. As there is no platform driver for the target hardware yet, which
would manage power and clocks, the hardware either has to be pre-configured
correctly, or the driver has to enable it on its own. Ethernet card, SD-card,
and the display engine were all already powered by the bootloader, but not
USB. In contrast to the first devices, the u-boot bootloader turns off USB
explicitly as soon as it starts the OS. As an interim solution, we patched
u-boot to not turn off the USB host controller, and enforced u-boot to
initialize the powering in our boot scripts. Therefore, if one wants to use
USB on the i.MX 8M EVK, make sure to take our modified version. As a
convenient solution, you can use the 'uboot' port within the base repository.
Just issue the following command in the Genode directory:
! tool/ports/prepare_port uboot
Finally, you have to copy u-boot to the SD-card as root user:
! dd if=`tool/ports/current uboot`/imx8q_evk/imx-mkimage/iMX8M/flash.bin \
! of=/dev/sd<?> bs=1k seek=33 conv=fsync
Of course, you have to replace 'sd<?>' with the correct device node of your
attached SD-card.
After enabling the USB host controller driver, we could successfully re-use the
USB HID client driver to drive keyboard and mouse connected to the board. As a
nice side-effect, the list of possible storage devices got extended with USB
mass storage too by adding the USB block client driver.
Missing libraries
=================
Finally, when building the necessary and optional packages for Sculpt OS, we
stumbled across several libraries that needed to be adapted to compile and
link for ARMv8 too. Mostly, the inclusion of some other compilation units and
headers was sufficient. The related libraries are: libssl, libcrypto, libpng,
and Mesa. With the latter two, it is now even possible to execute Qt5
components on the target hardware.
Apart from all the new driver components and extended libraries, the Sculpt
manager had to be slightly modified to execute on the i.MX 8M hardware. In its
original form it is inherently dependent on x86 drivers, as it for example
generates configurations for some of those drivers. For the time being, the
changes to the Sculpt manager are not yet part of the official release.
Nevertheless, you can produce a Sculpt OS image to be run on an i.MX 8M EVK
board by using the following
[https://github.com/skalk/genode/commits/sculpt_20.02_imx8q_evk - topic branch].
Alternatively, you can also have a look at Sculpt OS on ARMv8 hardware by
following the video recordings of the following talk at FOSDEM 2020.
:Live demonstration of Sculpt OS on i.MX 8M EVK at FOSDEM 2020:
[https://fosdem.org/2020/schedule/event/uk_genode_armv8/]
Base framework and OS-level infrastructure
##########################################
New sandbox library based on the init component
===============================================
The init component is Genode's canonical mechanism for the composition of
components. This role was further amplified when init became
[https://genode.org/documentation/release-notes/17.02#Dynamically_reconfigurable_init_component - dynamically reconfigurable].
The latter change cleared the ground for system scenarios like Sculpt OS, the
on-target deployment of packages, and dynamic device discovery. One typical
pattern found in such scenarios is one dynamically configured instance of init
accompanied by a controlling component that is usually called "manager". The
manager would consume reports of the subsystem hosted within the dynamic init,
and adjust the init configuration according to a domain-specific policy. Such
a configuration change, in turn, may trigger new reports, which effectively
turns this setting into a feedback control loop.
Whereas this established pattern is suitable for many scenarios, it is not
always natural. In particular if the manager does not only need to
manage a subsystem but also wants to intercept a service used by the
subsystem, the roles are no longer clear-cut. A practical example is a
GUI application that employs the menu-view component for the GUI rendering
while processing keyboard events locally. This application would need to
intercept the menu-view's GUI session to obtain the stream of user input
events. For such an application, the most natural approach would be the
co-location of the init functionality with the application logic into a
single all-encompassing component.
To accommodate such scenarios where a domain-specific management component is
tightly coupled with a dynamic subsystem, we extracted the child-management
functionality from the init component into a new library called "sandbox". The
library API is located at
[https://github.com/genodelabs/genode/blob/master/repos/os/include/os/sandbox.h - os/include/os/sandbox.h].
In addition to the hosting of components, the sandbox API also allows for the
interaction with the sandboxed children by providing locally implemented
services. The latter mechanism is illustrated by a new test available at
_os/src/test/sandbox_.
POSIX compatibility improvements
================================
During the release cycle of Genode 20.02, we continued our mission to host
POSIX software effortlessly as Genode components. In particular, we followed
up the line of work pursued with the two previous releases
[https://genode.org/documentation/release-notes/19.08#Consolidation_of_the_C_runtime_and_Noux - 19.08] and
[https://genode.org/documentation/release-notes/19.11#C_runtime_with_improved_POSIX_compatibility - 19.11]
with respect to the traditional Unix mechanisms fork, execve, and pipes.
After covering several edge cases - cloexec, file-descriptor lifetimes,
line-buffer handling, vfork, just to name a few - as needed by programs like
make, bash, and tclsh, we eventually reached a state where the website
generator of [https://genodians.org] works without the need for the now
deprecated Noux runtime.
For years we have been running complex software stacks like the Qt-based web
browser on top of our C runtime but not without carefully placed tweaks and
occasional patches. With the current release, we address the area of
concurrency and introduce a thorough reimplementation of the synchronization
primitives namely POSIX mutexes and condition variables as well as semaphores.
We also reaped the fruit of our labor by replacing our custom Qt thread back
end by the standard POSIX-thread based implementation. Further, we reduced the
number of threads in Qt applications by moving the QPA event handling to the
component entrypoint and removing the timed-semaphore utility from LibC.
Beyond Qt, we also address synchronization issues revealed by running a
third-party port of [https://grpc.io/ - gRPC] in our network back ends and
amended thread-local errno in the C runtime. Finally, our POSIX thread
implementation supports cleanup handlers now.
Base-framework refinements
==========================
Replacing the 'Lock' type by new 'Mutex' and 'Blockade' types
-------------------------------------------------------------
Up to now, Genode's lock implementation supports mainly two flavours of usage.
On the one hand, it is used to protect critical sections where the lock is
initialized as unlocked. In the contention case, the lock holder is supposed
to release the critical section. On the other hand, the lock is used as
blockade to synchronize startup between various executions of threads. Here
the lock is initialized as locked during instantiation whereby the thread that
releases the lock is not necessarily the same thread as the creator of the
lock.
We decided to make the two usage patterns more obvious by introducing two
separate classes, called 'Mutex' and 'Blockade'. The reasons are twofold.
First, during code review, the usage pattern at hand becomes more obvious.
Second, by codifying the programmer's intent behind the use of a
synchronization primitive, Genode becomes able to perform additional checks,
and diagnose certain dead-lock situations and other usage errors on the spot.
The separation got introduced shortly before this release. Up to now, it is
only used in 'Genode::Thread', 'Genode::Heap', and 'Genode::Registry'. The
plan is to cultivate the usage across all Genode sources over the next
releases and to ultimately remove the 'Genode::Lock' from the public API.
The 'Mutex' class is more restrictive compared to the 'Lock' class.
* At initialization time, it is always unlocked.
* To enter and leave a critical section the methods 'acquire()' and
'release()' are used.
* A 'Mutex::Guard' is provided, which will 'acquire()' a mutex at
construction time and release it automatically at destruction time of
the guard.
* No thread is permitted to lock twice. The code will generate a warning if
a dead-lock is detected.
* Only the lock holder is permitted to release the mutex. The code will
generate a warning and will not release the mutex if this rule is violated.
! Genode::Mutex mutex;
! mutex.acquire();
! mutex.release();
!
! {
! Genode::Mutex::Guard guard(mutex) /* acquire() during construction */
! } /* release() on guard object destruction */
!
! Genode::Mutex::Guard guard(mutex);
! mutex.acquire(); /* <-- Will cause a warning about the dead-lock */
The 'Blockade' class is always initialized as locked and provides the methods
'block()' and 'wakeup()'. Beside the initialization aspect, the 'Blockade'
behaves up to now like the 'Genode::Lock' implementation.
! Genode::Blockade blockade;
!
! /* step */ /* thread A */ /* thread B */
! 0: -start thread B-
! 1: ... -startup-
! 2: blockade.block(); ...
! 3: -sleep- ...
! 4: -sleep- blockade.wakeup();
! 5: ... ...
Performance optimization of the XML parser
------------------------------------------
Genode's XML parser used to rely on C++ exceptions while parsing, which is an
almost historic artifact inherited from the initial implementation. The
performance penalties of exceptions in the rare use of XML was acceptable
back when we started. But modern Genode systems like Sculpt OS rely on the
dynamic processing of XML like a back bone. The overhead became particularly
apparent when executing [Sculpt OS on 64-bit ARM i.MX8 hardware]. Prompted by
this observation, we reworked the code such that exceptions are no longer
thrown in any hot code path. The public interface of 'Xml_node' remains
unchanged.
New polling variant for register framework
------------------------------------------
Genode's register framework has offered a 'wait_for' method for a long time.
This function sleeps for a certain amount of microseconds and checks if one or
more given conditions become true. The number of attempts to sleep and check
the conditions must also be specified. In case the conditions are not met
after these attempts, a polling timeout exception is thrown. The function
simply returns in case of success. With the current Genode release, we have
added a 'wait_for_any' method with almost the same semantics but instead of
waiting for all conditions to become true, it returns if any condition is
met, and thus, implements a logical OR.
Migration to modern block-device API
====================================
With release 19.02, Genode introduced two new APIs for block-session handling.
The client side of a block session now uses the job API in order to send block
requests to the server, which in turn receives those jobs as requests through
the Request API. These two APIs replace Genode's 'Block::Driver' and
'Block::Session_component' implementations that used the packet stream API
directly, which turned out to be error prone for block session implementations.
Instead, these new APIs wrap the packet stream handling in a controlled
manner while handling all corner cases and even the overcommit of packets.
With the current release, we have adapted Genode's AHCI driver and partition
manager to these new interfaces, with the plan to adjust all block session
clients/servers to the new APIs with Genode release 20.05.
During this line of work, the AHCI driver received a major cleanup. For
example, dynamic memory allocations were removed, the whole initialization
state machine has been removed, ATAPI support for Qemu has been re-enabled,
and Exynos5 AHCI support is gone - since the platform is outdated and not
supported by Genode any more.
Updated audio driver based on OpenBSD 6.6
=========================================
In this release, we updated the 3rd-party sources of the audio driver component
to OpenBSD 6.6 and adapted the emulation glue code. While doing so, we fixed
a bug regarding the 'delay()' implementation where the function expects
microseconds but was given milliseconds. This led to a increased start-up
time of the component. We also fixed the logging back end that accidentally
was rendered silent and brought in the 'printf' back end from DDE Linux to
be able to produce better formatted LOG messages in the future.
Until now the component only supported HDA and EAP (ES1370 PCI) devices. The
first is primarily intended to be used with real hardware whereas the latter
was used during the initial porting effort in Qemu. That being said, the EAP
driver apparently also works on hardware according to community feedback.
Since the HDA driver does not work when used in VirtualBox and users expressed
the desire to also use audio when running in a VM, we enabled another driver,
for which a device-model in VirtualBox exists: the AC97 ICH. As it turned out,
using this driver, we can produce audio, albeit the quality is far from
usable. Nevertheless, with the driver enabled, interested parties are free to
investigate the cause for the current issues.
All in all, this update is solely a catch up effort to stay more
up-to-date with the upstream changes and to pull in HDA quirks for more
recent systems. More interesting changes to the driver component, like
reworking the OpenBSD kernel emulation layer and bringing support for USB
audio devices, are scheduled for future releases.
Support for unlabeled LOG output
================================
In situations where a Genode system is remotely controlled and monitored,
it is useful to allow a special component to produce log output with no
Genode label applied. This way, such a component can produce log data in
a format that is immediately suitable for a controller. This feature can be
enabled for a component by rewriting the label of the component's LOG session
to "unlabeled".
! <route>
! <service name="LOG"> <parent label="unlabeled"/> </service>
! ...
! </route>
Libraries and applications
##########################
Custom virtual machine monitor on ARM
=====================================
The ARMv8-compliant virtual-machine monitor introduced in the previous release
19.11 now contains new device models to enable the interaction with a
virtual-machine via network and terminal services. The new virtual ethernet
card and console implementations are compliant to the virtualization standard
VIRTIO 1.1.
Currently, the VMM cannot be configured to contain specific devices. It is
hard-wired to provide exactly:
* One virtual ethernet card that connects to Genode's "Nic" service,
* A VIRTIO console that opens up a session to the "Terminal" service using the
label "console", and
* The traditional PL011 serial device model, which connects to a
"Terminal" service too but uses the label "earlycon"
Seoul VMM
=========
During the usage of Seoul on Sculpt, it became apparent that the Seoul VMM
caused a constant CPU load even when the guest VM was idling. After some
investigation it became clear that having a fixed rate to synchronize the
guest graphic memory with the Genode GUI service was the main reason for the
constant load. With this release, we added the feature to dynamically adjust
the GUI refresh rate depending on the rate of user interactivity.
Additionally, if all virtual CPUs go to idle state, the GUI refresh is stopped
completely. With these measures, the overall CPU load could be reduced
noticeably.
TCP terminal
============
The TCP terminal is a long-living component in the Genode OS framework since
release 11.11. It can be used, e.g., to connect to a headless Genode system
via telnet. Until now, it always listened to incoming network connections at
configured ports. The port had to be configured for each terminal session
client.
The TCP terminal got extended to either listen to incoming network
connections, or to directly connect to another network server, dependent on
the policy defined for the corresponding terminal client. The following
example configuration illustrates the differences:
! <config>
! <policy label="client" ip="10.0.0.5" port="1234"/>
! <policy label="another_client" port="4567"/>
! </config>
If only a port is described in the policy, the TCP terminal will listen on
that port for incoming connections. If an IP address is provided additionally,
it connects to the IP address using the given port.
Virtual desktops
================
Genode's GUI stack enables a high degree of flexibility. Beside the fundamental
nitpicker component, responsible for basically multiplexing input events and
framebuffer content, there is the window-manager component, and example
implementations of a window-layouter, and decorator. The interplay of the
latter three allows a window management that scales from simple to rich and
sophisticated without lowering its security properties. For a brief description
of its architecture, please refer to the release notes of
[https://genode.org/documentation/release-notes/14.08 - 14.08].
In this architecture, the window layouter is responsible for the arrangement
of the different windows. It exports a data model of the window layout.
Although, the example implementation of the window layouter introduced in
14.08 was simple, it already contained a notion of having different virtual
screens and screen sections, beside the actual window placements. However,
until now there was no use-case of switching dynamically between different
virtual screens respectively window sets related to them.
While using more and more different graphical components within Sculpt, the
window layouter in its initial form hit a limit. Although it already allowed to
switch in-between different windows via configured key-combinations, it became
inconvenient when having more than a handful windows hiding each other.
Therefore, the window layouter now got extended to allow switching dynamically
in between several pre-defined virtual screens. For the time being, one has to
assign a new window to a screen in the rule-set of the window layouter
initially by hand. Defining the currently visible screen can either be done by
editing the rule-set, or by using pre-configured key-combinations.
The new default configuration of the window layouter as exported by its
corresponding depot package looks like the following:
! <config rules="rom">
! <rules>
! <screen name="screen_1"/>
! <screen name="screen_2"/>
! <screen name="screen_3"/>
! <screen name="screen_4"/>
! <screen name="screen_5"/>
! <screen name="screen_6"/>
! <screen name="screen_7"/>
! <screen name="screen_8"/>
! <screen name="screen_9"/>
! <screen name="screen_0"/>
! <assign label_prefix="" target="screen_1" xpos="any" ypos="any"/>
! </rules>
!
! <press key="KEY_SCREEN">
! <press key="KEY_ENTER" action="toggle_fullscreen"/>
! <press key="KEY_1" action="screen" target="screen_1"/>
! <press key="KEY_2" action="screen" target="screen_2"/>
! <press key="KEY_3" action="screen" target="screen_3"/>
! <press key="KEY_4" action="screen" target="screen_4"/>
! <press key="KEY_5" action="screen" target="screen_5"/>
! <press key="KEY_6" action="screen" target="screen_6"/>
! <press key="KEY_7" action="screen" target="screen_7"/>
! <press key="KEY_8" action="screen" target="screen_8"/>
! <press key="KEY_9" action="screen" target="screen_9"/>
! <press key="KEY_0" action="screen" target="screen_0"/>
! ...
As can be seen, individual keys are assigned to switch to a specific virtual
screen. By default ten screens are defined that are accessible via the number
keys. The first screen definition in the rules configuration marks the
currently visible screen.
Menu-view widget renderer
=========================
The line of work described in Section
[Redesign of the administrative user interface of Sculpt OS] called for
the enhancement of Genode's GUI-rendering component. This component - named
menu view - was
[https://genode.org/documentation/release-notes/14.11#New_menu_view_application - originally introduced in Genode 14.11]
for the rendering of the relatively simple menus of an application launcher.
Its software design largely deviates from the beaten track of established
widget toolkits, which come in the form of client-side libraries. The
menu view is not a complete toolkit but solely a dialog renderer sandboxed
in a dedicated component. This design reinforces the strict separation of the
view from the application logic, fosters screen-resolution independence, and -
most importantly - keeps the complexity of pixel processing out of the
application program. Because of the latter, it lends itself to the
implementation of security-sensitive interactive applications.
It would certainly be misguiding to tout our menu-view as feature competitive
with existing toolkits. We certainly won't recommend using it over Qt in
general. But Sculpt's custom administrative user interface "Leitzentrale"
presented us with the perfect playground to explore and grow the potential of
our novel approach.
In contrast to the previous iteration of the Leitzentrale GUI, which relied on
a small Unix runtime and Vim for editing text files, the new version ought to
feature a simple text editor integrated in the GUI. A text editor requires
a much tighter interplay between the view and the actual program logic
compared to an application with just a bunch of buttons. Think about cursor
handling, scrolling text, displaying textual selections, or placing a text
cursor with the mouse. On the course of the work towards the text-area
component featured in the new Leitzentrale, the menu view received the
following improvements:
:Text-cursor support:
The label widget gained the ability to display one or multiple text cursors,
as illustrated by the following example:
! <label text="...">
! <cursor at="10"/>
! </label>
For the display of multiple cursors, each cursor must feature a distinctive
'name' attribute.
:Character position featured in the hover report:
The hovering information provided by the menu view used to be at the
granularity of widgets, which is insufficient for placing a text cursor with
the mouse. Hence, the information of a hovered label additionally provides
the character position within the label now.
:Unquoting label text attribute values:
The text displayed in label widgets is provided by a 'text' attribute value,
which raises the question of how to present '"' characters on the GUI. With
the new version, the attribute value can contain XML-quoted characters,
specifically "&quot;".
:Support for displaying text selections:
Similarly to the way of how a <cursor> can be defined for a <label>
widget, a selection can now be expressed as follows:
! <label ...>
! <selection at="2" length="12"/>
! </label>
:Support of multiple '<float>' widgets within a '<frame>':
We refined the hover reporting of <float> widgets such that a float widget
never responds to hovering unless a child is hovered. This way, it becomes
possible to stack multiple float widgets within one frame and still reach
all child widgets. This is useful for aligning multiple widgets within one
screen area independently from each other. For example, for left-aligning,
centering, and right-aligning the elements of a panel.
:Enforcing the minimum size of a label:
The new '<label min_ex="..">' attribute can be used to enforce a minimum
width in the unit of the size of the character 'x'. In the absence of a
'text' attribute, the minimum height of a label is implicitly set to 0. The
combination of both changes makes the label usable as a horizontal spacer.
:Basic support for styling labels:
The new version allows for the customization of the text color and alpha
value of the label widget by the means of a style-definition file. The
mechanism is exemplified with the new "invisible" label style that sets the
alpha value to zero.
With these few incremental changes in place, the menu-view widget renderer
becomes usable as the basis of the simple text editor used in Sculpt's new user
interface.
Self-hosting the tool chain on 64-bit ARM
=========================================
With our ongoing ARM 64-bit effort, we have successfully updated Genode's tool
chain with release
[https://genode.org/documentation/release-notes/19.05#Broadened_CPU_architecture_support_and_updated_tool_chain - 19.05].
With the current release, we have additionally managed to make Genode's tool
chain self hosting on ARM 64-bit, which means the tool chain can compile
source code on ARM 64-bit directly.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
The generic code base of the base-hw kernel underwent several cosmetic changes
to reduce or eliminate the application of certain problematic constructs like
too much inheritance, pointers, and dynamic casts. Those changes were
motivated to ease the translation of several kernel parts to the Ada/SPARK
language in the context of the Spunky project. For more information regarding
this experiment to write a Genode kernel in Ada/SPARK, please have a look at
the recent [https://genodians.org/m-stein/index - genodians.org article series]
of Martin Stein or listen to his recent
[https://video.fosdem.org/2020/AW1.125/ada_spunky.mp4 - FOSDEM talk].
Moreover, the IPC path implementation got simplified to lower the overhead
costs introduced by the transfer of capabilities. Together with the mentioned
Spunky cleanup efforts, this change measurably improved IPC performance.
The base-hw kernel now exports time consumption of individual threads via
the trace service analogously to the implementation for NOVA. Thereby, it
becomes possible to use for instance the top component within the Sculpt OS
also on this kernel.
Until now, support for the Raspberry Pi 3 was limited to Qemu emulation only.
Thanks to a contribution of Tomasz Gajewski, it is now possible to execute
Genode on all four CPUs of the actual hardware concurrently.
Execution on Linux
==================
Traditionally, the Linux version of Genode serves us as very handy development
vehicle but it was never intended as an actual target platform. On Linux,
Genode is usually executed as a multi-process application on top of a regular
GNU/Linux desktop distribution by specifying 'KERNEL=linux' and 'BOARD=linux'
to the run tool.
However, thanks to the work of Johannes Kliemann, Genode has become able to
run on a bare-bone Linux kernel without any other user land.
We blatantly used to refer to this idea as the
[https://genode.org/about/challenges#Platforms - microkernelization of Linux].
Johannes picked up the idea, supplemented Genode's core with the services
needed for user-level device drivers (IRQ, IOMEM, IOPORT) and supplemented
the tooling for the integration of Genode scenarios into a bootable initrd
image. This target of execution can be addressed by specifying 'KERNEL=linux'
and 'BOARD=pc' to the run tool now. If specified, the run tool will produce a
bootable Linux system image for the given run script and run it in Qemu.
That said, as this line of work is still considered as an experimental
playground - not for productive use - the work flow is not entirely automated.
In particular, one needs to prepare a suitable
[https://github.com/jklmnn/linux/commits/genode - Linux kernel] manually.
If you are interested in the topic, please refer to the background information
given in the [https://github.com/genodelabs/genode/pull/2829 - issue tracker].

1168
doc/release_notes/20-05.txt Normal file
View File

File diff suppressed because it is too large Load Diff

1015
doc/release_notes/20-08.txt Normal file
View File

File diff suppressed because it is too large Load Diff

629
doc/release_notes/20-11.txt Normal file
View File

@@ -0,0 +1,629 @@
===============================================
Release notes for the Genode OS Framework 20.11
===============================================
Genode Labs
With Genode 20.11, we focused on the scalability of real-world application
workloads, and nurtured Genode's support for 64-bit ARM hardware. We thereby
follow the overarching goal to run highly sophisticated Genode-based systems
on devices of various form factors.
When speaking of real-world workloads, we acknowledge that we cannot always
know the exact behavior of applications. The system must deal gracefully with
many unknowns: The roles and CPU intensity of threads, the interplay of
application code with I/O, memory-pressure situations, or the sudden fragility
of otherwise very useful code. The worst case must always be anticipated. In
traditional operating systems, this implies that the OS kernel needs to be
aware of certain behavioral patterns of the applications, and has to take
decisions based on heuristics. Think of CPU scheduling, load balancing among
CPU cores, driving power-saving features of the hardware, memory swapping,
caching, and responding to near-fatal situations like OOM.
Genode allows us to move such complex heuristics outside the kernel into
dedicated components. Our new CPU balancer described in Section
[CPU-load balancing] is a living poster child of our approach. With this
optional component, a part of a Genode system can be subjected to a CPU-load
balancing policy of arbitrary complexity without affecting the quality of
service of unrelated components, and without polluting the OS kernel with
complexity.
A second aspect of real-world workloads is that they are usually *not*
designed for Genode. To accommodate the wealth of time tested applications, we
need to bridge the massive gap between APIs of olde (think of POSIX) and
Genode's clean-slate interfaces.
Section [Streamlined ioctl handling in the C runtime / VFS] shows how the
current release leverages our novel VFS concept for the emulation of
traditional ioctl-based interfaces. So useful existing applications come to
live without compromising the architectural benefits of Genode.
Platform-wise, the new release continues our mission to host Genode-based
systems such as [https://genode.org/download/sculpt - Sculpt OS] on 64-bit
ARM hardware. This work entails intensive development of device drivers and
the overall driver architecture.
Section [Sculpt OS on 64-bit ARM hardware (i.MX8 EVK)] reports on the
achievement of bringing Sculpt to 64-bit i.MX8 hardware. This line of work
goes almost hand in hand with the improvements of our custom virtual machine
monitor for ARM as outlined in Section [Multicore virtualization on ARM].
CPU-load balancing
##################
Migrating load over CPUs may be desirable in dynamic scenarios, where the
workload is not known in advance or too complex. For example, in case of POSIX
software ported to Genode, amount and roles of threads and processes can
generally not planned for. With the current release, we add an optional CPU
service designated for such dynamic scenarios. The new component called
[https://genodians.org/alex-ab/2020-11-16-cpu-balancer - CPU balancer] is able
to monitor threads and their utilization behaviour. Depending on configured
policies, the balancer can instruct Genode's core via the CPU session
interface to migrate threads between CPUs.
[image cpu_balancer]
The CPU balancer intercepts the interaction of a Genode subsystem
(workload) with core's low-level CPU service.
This feature requires a kernel that supports thread migration, which are
Fiasco.OC, seL4, and to some degree the NOVA kernel. For the NOVA kernel,
solely threads with an attached scheduling context can be migrated, which are
'Genode::Thread' and POSIX pthread instances. Genode's entrypoint and virtual
CPU instances are not supported.
The feature can be tested by the scenario located at _repos/os/run/cpu_balancer.run_.
Further information regarding policy configuration, a demo integration into
Sculpt 20.08, and a screencast video are available as a dedicated
[https://genodians.org/alex-ab/2020-11-16-cpu-balancer - CPU balancer]
article.
Sculpt OS on 64-bit ARM hardware (i.MX8 EVK)
############################################
Within the last year, a lot of effort was put into Genode's support for ARM
64-bit hardware. A consequent next step was to port Sculpt OS to the i.MX8 EVK
board, which we have used so far as reference platform. With the current
release, we proudly present the first incarnation of Sculpt OS for this board.
In contrast to the original x86 PC variant, this first ARM version ships with
a static set of devices inside the drivers subsystem. No device manager
component probes for the used hardware and starts drivers on demand. Instead,
the set of drivers defined in the _drivers_managed-imx8q_evk_ package enables
USB HID devices to make use of mouse and keyboard peripherals connected to the
board. It drives the SD-card, which can be used as storage back end for
Genode's depot package management. Finally, it contains drivers to manage the
display engine and the platform's device resources.
With Sculpt OS for ARM 64-bit, we not only aim for classical desktop/notebook
systems - like on x86 - but also for embedded consumer hardware like phones
and tablets. In order to leverage this goal, we enabled support for
[https://www.nxp.com/design/development-boards/i-mx-evaluation-and-development-boards/i-mx-8-series-accessory-boards:i.MX8-ACCESSORY-BOARDS - NXP's MX8_DSI_OLED1]
display on the i.MX8 platform on Genode. The panel features an OLED display as
well as a Synaptics RMI4 compliant touch screen.
Genode's i.MX8 display driver that we released with version
[https://genode.org/documentation/release-notes/20.02#Display_engine - 20.02]
supported HDMI devices only, whereas the OLED display is connected via
[https://www.mipi.org/specifications/dsi - MIPI DSI] to the SoC. Therefore, we
extended the display driver by the MIPI DSI infrastructure as well as the
actual driver for the OLED display. This endeavor turned out to be a very
rocky one, which we have documented in detail on our
[https://genodians.org/ssumpf/2020-09-30-mipi_touch - Genodians] website.
[image imx8_oled]
The administrative user interface of Sculpt OS responds to touch input.
In order to enable the touch screen device, we implemented a new Genode
component from scratch. The touch screen is connected via an I2C bus to the
SoC where data can be sent to and received from. At the moment, the I2C
implementation is hidden within the driver but as more devices require I2C
access, it will eventually become a standalone component. Interrupts are
delivered via GPIO pins from the touch screen to the SoC, which made it
necessary to enable i.MX8 support within Genode's generic i.MX GPIO driver. We
took this as an opportunity to streamline, cleanup, and make the driver more
robust. Additionally, all driver components now take advantage of the new
platform driver API for ARM that has been introduced with release
[https://genode.org/documentation/release-notes/20.05#New_platform_driver_for_the_ARM_universe - 20.05].
In its current incarnation, the driver for the display management is not able
to switch in between HDMI or MIPI-DSI connected displays dynamically.
Therefore, the display to be used in Sculpt has to be configured in the
framebuffer configuration manually. By default the HDMI connector is used.
Beyond the driver subsystem, there are few components dependent on the actual
hardware, which is why the look & feel of the Sculpt desktop does not actually
differ from the x86 PC version, with the following exceptions:
When you select the network configuration dialog, you'll have no "Wifi" option
because of the missing hardware. However, the "Wired" option allows you to
start the corresponding driver for the i.MX FEC Ethernet device. The second
difference to the Sculpt OS x86 PC variant is the absence of a virtual machine
solution at the moment. Although Genode comprises a mature
virtual-machine-monitor solution for ARM - see
Section [Multicore virtualization on ARM] - it still lacks a reasonable
storage back end. Therefore, we left virtualization out of the picture for
now. Lastly, there is no possibility to use USB block devices, because the
required management component - a driver manager for i.MX8 - does not exist
yet. We plan to bridge these remaining few gaps compared to the x86 version
with the upcoming Genode releases.
To give Sculpt a try on the i.MX8 EVK board, you have to start the well-known
Sculpt run-script as usual, but for the base-hw kernel. For example:
! tool/create_builddir arm_v8a
! cd build/arm_v8a
! make run/sculpt KERNEL=hw BOARD=imx8q_evk
Under the hood, the run script requests a sculpt-<board> specific package from
the depot package system. Currently, _sculpt-pc_ and _sculpt-imx8q_evk_ are
available.
Multicore virtualization on ARM
###############################
The written-from-scratch virtualization solution for Genode on ARMv8 entered the
picture exactly one year ago with
[https://genode.org/documentation/release-notes/19.11#Virtualization_of_64-bit_ARM_platforms - release 19.11].
Since then, a couple of improvements and validations have been incorporated
into it. Support for VirtIO network and console models had been added.
Moreover, it got streamlined with our prior existing ARMv7 hypervisor and
virtual-machine monitor (VMM). But although the architecture of the VMM was
designed from the very beginning with more than one virtual-CPU (VCPU) in
mind, running a VM on multiple cores had not been addressed nor tested.
With this release, we enhance the virtualization support of the base-hw
kernel, acting as the ARM hypervisor, to support multicore virtual machines.
The VMM implementation got extended to start an entrypoint for each VCPU owned
by a VM. The affinities of those entrypoints are configured to distribute over
all physical CPUs available to the VMM. The affinity of an entrypoint that
handles events of a VCPU is automatically used as the affinity of the VCPU
itself. Whenever a VCPU exit needs to be handled, this is delegated to the VMM
entrypoint running on the same CPU. Once the VMM's entrypoint successfully
handled the exit reason, it resumes the VCPU.
Formerly, the control to start or stop a VCPU was implemented by core's VM
service that runs on the first CPU. But that implied that all different VMM
entrypoints running on distinct CPUs would have needed to frequently call
core's service entrypoint on the first CPU, inducing costly cross-CPU
communication. This is amplified by the fact that core's entrypoint uses a
system call to instruct the kernel's internal scheduler of the corresponding
target CPU, which again would potentially target a remote CPU. For simplifying
the implementation and for improving performance, we slightly extended the
VM-session interface to return a kernel-specific capability addressing a VCPU
directly. With this capability, a VMM's entrypoint is able to directly call
the kernel to start or stop a VCPU instead of using the indirection over core.
However, the detail whether the kernel is called directly or not is hidden
behind the VM session client API and transparent to the user.
Base framework and OS-level infrastructure
##########################################
C runtime
=========
We improved the support for aligned memory allocations to fix sporadic memory
leaks, which occurred with our port of the Falkon web browser. One relevant
change is the implementation of the 'posix_memalign()' function, another
change is that the address alignment of anonymous 'mmap()' allocations is now
configurable like follows:
! <config>
! <libc>
! <mmap align_log2="21"/>
! </libc>
! </config>
Standard C++ library
====================
Even though Genode uses C++ as its primary programming language, we do not
rely on or make use of any C++ standard library within the Genode OS
framework. However, since a C++ STL is a vital part of application programming
with C++, we provide one for applications built on top of the base framework;
in particular the GNU C++ STL library (_libstdc++_). It is treated as a
regular 3rd party library and its functionality is extended on demand. This
approach worked well enough to even enable larger C++-based software like Qt5
and Chromium's Blink engine (as part of QtWebEngine) to run on Genode. That
being said, for developers using _libstdc++_ on Genode, it is not immediately
clear, which features are supported and which are not.
Fortunately, _libstdc++_ includes a testsuite that - as the name suggests -
allows for testing the range of functionality of the library on a given
platform. So we turned to it to establish a base line of supported features.
We were particularly interested in how our port behaves when C++17 is
requested. It goes without saying that this only includes the aspects, which
are specifically probed by the testsuite. Rather than adding thorough Genode
support to the testsuite, we opted for providing an
[https://github.com/cnuke/genode-libstdcxx-testsuite/ - environment] that
mimics the common 'unix' target and allows us to execute the testsuite on
the Linux version of Genode via a regular Linux host OS. It uses the Genode
tool chain to compile the tests and spawns a Genode base-linux system to
execute them.
Executing the testsuite was an iterative process because in the beginning, we
encountered many falsely failed tests. On one hand, most of them were due to
the way C++ is applied in Genode or rather how our build system works
internally. For one, _libsupc++_ on Genode is part of the _cxx_ library. This
library in turn is part of _ldso.lib.so_, the dynamic linker that provides
the base API. As the build system uses stub libraries generated from 'symbol'
files containing the ABI of a given shared object, each missing symbol must
be made available. Otherwise the linking step is going to fail complaining
about undefined references because components use these stub libraries
during compilation. On the other hand, we had to get cozy with the testsuite's
underlying test framework in order to get our test environment straight.
In case of the testsuite, there were a lot of symbols missing because we did
not encounter them so far in our workloads, and thus, were not part of the
symbols file. After all, templates will always generate specific symbols that
are difficult to foresee. Besides that, we lacked support for aligned 'new'
and 'delete' operators. With these adaptions in place, we were able to
successfully execute the testsuite.
In the end, the results paint a good picture. The current short-comings boil
down to
* Support for the *stdc++fs* library is not available as the library is
not ported yet.
* Proper *locale* support in the 'libc' as well as 'stdc++' is not available.
* Support for parallel operations with *openmp* is not available.
* Various subsystems ('std::thread', 'std::random_device', numerics library)
need further attention for proper functionality. This is most prominent
for the failing execution tests where sometimes the threads appear to
get stuck.
These findings are documented at issue
[https://github.com/genodelabs/genode/issues/3925 - 3925].
Consistent Block Encrypter (CBE)
================================
The CBE is a library for the management of encrypted block-devices that is
entirely written in SPARK. It was first announced and integrated with
[https://genode.org/documentation/release-notes/19.11#Preliminary_block-device_encrypter - Genode 19.11],
reached feature-completeness with
[https://genode.org/documentation/release-notes/20.05#Feature-completeness_of_the_consistent_block_encrypter - Genode 20.05],
and has received a highly modular back-end system with version
[https://genode.org/documentation/release-notes/20.08#Consistent_Block_Encrypter - 20.08].
For this release, we thoroughly streamlined the CBE repository, added enhanced
automated quality assurance, and switched to another default encryption
back end.
Repository restructuring
------------------------
Generally speaking, the [https://github.com/m-stein/cbe - CBE repository] has
been freed from everything that is not either part of the SPARK-based core
logic (cbe, cbe_common, and the hashing algorithm), the essential SPARK-based
tooling (initialization, checking), or the Ada-based C++ bindings (*_cxx
libraries). The whole Genode-specific integration, testing, and packaging
moved to Genode's 'gems' repository and the former Genode sub-repository 'cbe'
was replaced by the new CBE port _gems/ports/cbe.port_. We also took the
opportunity to remove many unused remnants of earlier development stages and
to drastically simplify the ecosystem of CBE-related packages.
We hope that this allows for certain characteristics of the CBE project, like
its strong OS-independence or a completely "flow-mode"-provable core logic to
become more clear, while at the same time, the Genode-specific accessories can
benefit from being part of Genode's mainline development.
Automated testing, benchmarking, and proving
--------------------------------------------
The CBE tester is a scriptable environment meant for testing all aspects of
the CBE library and its basic tooling. Through its XML command interface, one
can not only access and validate data of CBE devices but also initialize them,
check their consistency, analyze their meta data, execute performance
benchmarks, manage device snapshots, perform online re-keying or online
re-dimensioning of devices, and, last but not least, manage the required Trust
Anchors.
Before this release, the CBE tester was a mere patchwork solution and many of
the above mentioned features were limited or even missing. For instance block
access was issued only in a synchronous fashion, the Trust-Anchor was managed
implicitly, and validating read data wasn't possible. Besides adding the
missing features, we also reworked the component entirely to follow a clean
and comprehensible implementation concept. The new CBE tester comes together
with the run script _gems/run/cbe_tester.run_ that shall serve as both a
demonstration how to use the tester and an extensive automated test and
benchmark for the CBE.
Furthermore, we created the CBE-specific autopilot tool _tool/cbe_autopilot_
that is meant to establish a common reference for the quality of CBE releases
as well as for their integration in Genode. Running the tool without arguments
will give instructions how to use it. In a nutshell, when running
'tool/cbe_autopilot basics', the tool will GNAT-prove what is expected to be
provable, run all CBE-related run scripts expected to work, and build all
CBE-related packages (existing build and depot directories are not touched in
this process). The idea is to make the successful execution of the test
mandatory before advancing the master branch of the CBE repository or
releasing a new version of the integration in Genode. A handy side-feature of
the tool is that one can run 'tool/cbe_autopilot prove' to do only the
GNAT-proving part. With 'tool/cbe_autopilot clean' finally, the tool cleans up
all of its artifacts.
Libcrypto back end for block encryption
---------------------------------------
The introduction of VFS plugins for CBE back ends in the previous Genode
release made it much easier to interchange concrete implementations. This
motivated us to play around a bit in our endeavour of optimizing execution
time. It turned out that especially the choice of the block-encryption back
end has a significant impact on the overall performance of CBE block
operations. It furthermore seemed that especially the 'libsparkcrypto'
library, our former default for block encryption, prioritizes other qualities
over performance.
That said, in general, we want to enable an informed user to decide for him-
or herself which qualities one prefers in such an algorithm. The VFS plugin
mechanism pays tribute to this. And it also seems very natural to us to
combine a SPARK-based block-device management with a SPARK-based encryption
back-end like 'libsparkcrypto'. But for our default use case, we came to the
conclusion that the 'libcrypto' library might be a better choice.
Streamlined ioctl handling in the C runtime / VFS
=================================================
The Genode release
[https://genode.org/documentation/release-notes/19.11#C_runtime_with_improved_POSIX_compatibility - 19.11]
introduced the emulation of ioctl operations via pseudo files. This feature
was first used by the Terminal. With the current release, we further employ
this mechanism for additional ioctl operations, like the block-device related
I/O controls, as the long-term plan is to remove the notion of ioctl's from
the 'Vfs::File_io_services' API all-together.
We therefore equipped the block VFS-plugin with a compound directory hosting
the pseudo files for triggering device operations:
:info: This file contains the device information structured as 'block'
XML node having 'size' and 'count' attributes providing the used block size
as well as the total number of blocks.
:block_count: contains the total number of blocks.
:block_size: contains the size of one block in bytes.
Furthermore, we split the existing 'ioctl' handling method in the libc into
specific ones for dealing with terminals and block devices because at some
point more different groups of I/O controls are to follow.
The first one to follow is the 'SNDCTL' group. This group deals with audio
devices and corresponds to the standard set by the OpenSoundSystem (OSS)
specification years ago. In the same vein as the terminal and block I/O
controls, the sound controls are implemented via property files.
The controls currently implemented are the ones used by the OSS-output plugin
of [https://cmus.github.io/ - cmus], the driving factor behind the
implementation, which uses the (obsolete) version 3 API.
At the moment, it is not possible to set or rather change any parameters. In
case the requested setting differs from the parameters of the underlying
audio-out session - in contrast to the suggestion in the OSS manual - we do
not silently adjust the parameters returned to the callee but let the I/O
control operation fail.
The following list contains the currently handled SNDCTL I/O controls:
:SNDCTL_DSP_CHANNELS: sets the number of channels. We return the available
channels here and return ENOTSUP if it differs from the requested number of
channels.
:SNDCTL_DSP_GETOSPACE: returns the amount of playback data that can be written
without blocking. For now it amounts the space left in the stream buffer of
the audio-out session.
:SNDCTL_DSP_POST: forces playback to start. We do nothing and return success.
:SNDCTL_DSP_RESET: is supposed to reset the device when it is active before
any parameters are changed. We do nothing and return success.
:SNDCTL_DSP_SAMPLESIZE: sets the sample size. We return the sample size of the
underlying audio-out session and return ENOTSUP if it differs from the
requested format.
:SNDCTL_DSP_SETFRAGMENT: sets the buffer size hint. We ignore the hint and
return success.
:SNDCTL_DSP_SPEED: sets the sample rate. For now, we always return the rate of
the underlying audio out session and return ENOTSUP if it differs from the
requested one.
The libc extension is accompanied by an OSS VFS plugin that gives access to an
audio-out session by roughly implementing an OSS pseudo-device. It merely
wraps the session and does not provide any form of resampling or re-coding of
the audio stream.
[image cmus]
Image [cmus] depicts how the various pieces work together in a real-world
scenario. The interplay of the extended libc with the OSS VFS plugin allows
for listening to MP3s - for the time being the format is restricted to
44.1kHz/16bit - on Sculpt using the [https://cmus.github.io/ - cmus]
audio player.
The current state serves as a starting point for further implementing the OSS
API to cover more use cases, especially with ported POSIX software like
VirtualBox and Qt5 or even as SDL2 audio back end. While showing its age, OSS
is still supported by the majority of middle ware and makes for a decent
experimentation target.
Device drivers
##############
VirtIO support
==============
Thanks to the remarkable contribution by Piotr Tworek, the Genode OS framework
has become able to drive VirtIO network devices.
He did not only provide a single VirtIO network driver but a framework to
easily add more VirtIO driver classes in the future. Either the devices are
connected as PCI devices or directly as platform devices with fixed
memory-mapped I/O addresses. The framework supports both and abstracts away
from the concrete connection type.
The VirtIO network driver enables networking for Genode when using the
'virt_qemu' board on either the ARMv7a or ARMv8a architecture. However, the
VirtIO device configuration on Qemu is dynamic. The order and presence of
different command line switches affect the bus address and interrupt
assignment of each device. To make the use of Genode with Qemu robust in
changing environments, a tiny helper component was supplemented. This
component named 'virtdev_rom' probes the memory-mapped I/O areas of the system
bus and detects available and known VirtIO devices. The results are provided
in the form of a configuration that can be consumed by the platform driver to
assign the correct device resources to the corresponding VirtIO driver.
The VirtIO network driver in action, as well as the interplay of the platform
driver and the 'virtdev_rom' component can be observed when using the
'drivers_nic-virt_qemu' package.
Improved support for OpenBSD audio drivers
==========================================
So far, the supported drivers exclusively used PCI as transport bus and for
practical reasons, the emulation environment was tied to it. The bus handling
has now moved into its own compilation unit to make future addition of drivers
that employ other transport buses easier. On the same account, the component
got renamed to 'pci_audio_drv' to reflect its bus connection.
While at it, the execution flow of the component got adapted. The kernel code
should have been executed within the context of the main task like it is done
in the DDE Linux drivers. The initial port of the HDA driver, however, called
the code directly from within the session as there was no immediate reason to
use a task context because suspending the execution was not needed. When using
USB devices, that is no longer possible as we have to suspend the execution
during the execution of the kernel code. So we pass in the audio data and
schedule the emulated BSD kernel code.
The above mentioned changes are mostly preliminary clean-up work for the
upcoming support of USB audio devices.
Furthermore, we implemented timeout handling in the driver and use Genode's
timeout framework API to schedule timeouts and for providing the current time.
For now there is only one timeout - the unsolicited Azalia codec event - and
therefore the timeout queue consists of solely one timeout object. Those
events are important for detecting plugged in headphones.
Supporting headphones was further refined by accounting for the situation
where the driver is started while headphones are already plugged in and the
mixer needs to be configured accordingly. In particular, on the Fujitsu S938
the driver lacked the proper quirk for switching between the internal and
external microphone.
In addition to the changes made to the audio driver component, the behaviour
of the audio mixer was adjusted with regard to handling the configuration
of a new session. The mixer now applies the settings already stored in its
configuration to new sessions instead of only reporting them. In case of
Sculpt, where an existing launcher already contains a valid configuration,
that allows for setting the volume levels appropriately for known sessions
prior to establishing the connection.
Retiring the monolithic USB driver
==================================
With [https://genode.org/documentation/release-notes/18.08#Decomposed_USB_stack - release 18.08],
a componentized USB stack got introduced next to our time-tested monolithic
USB driver. With the current release, the driver manager as used by Sculpt OS
switched to use the new USB stack in order to benefit from the de-composition
and from more supported USB devices. The monolithic driver was still based on
an older DDE-Linux revision compared to the componentized version. This step
paves the ground to retire the monolithic USB driver with the next Genode
release and will improve the number of supported USB devices with the upcoming
Sculpt OS release.
Platforms
#########
Hardware P-State support on PC hardware
=======================================
Intel CPUs feature Speed Shift respectively Hardware P-State (HWP)
functionality in order to balance CPU frequency and voltage for performance
and power efficiency. Up to now, the UEFI firmware of the notebooks we worked
with selected or made an option selectable in the UEFI configuration to
specify the desired behaviour, e.g. optimize for performance or power
efficiency.
With a recent Lenovo notebook, however, we faced the issue that either the fan
would run for too long after some load and/or the performance of the CPUs
regressed. Finding a well working sweet spot
[https://github.com/genodelabs/genode/issues/3871 - seems hard].
This experience prompted us to investigate how the Intel HWP feature can be
set and configured. After some experiments, we achieved to reduce the fan
noise and received better performance by tweaking the Intel HWP settings.
However, changing the Intel HWP settings requires access to the privileged
mode on all available CPUs. Since Genode supports several kernels, a solution
would require us to modify all kernels or the feature would remain solely
available to one kernel. We went for a different approach.
On x86, we use the tools from the
[https://genode.org/documentation/release-notes/18.08#New_Intel_Microcode_update_mechanism - Morbo project],
e.g., bender and microcode, to run code before the kernels are booted. The
jobs of the tools are to scan, enable, or apply changes to the CPUs and
chipset, which are not required to change during runtime. We came to the
conclusion that the named bootstrap tools are good places to apply such
one-time Intel HWP settings for the moment.
During the course of adding the Intel HWP functionality, we merged the
microcode functionality into the bender tool and made it configurable via the
boot options 'microcode' and 'intel_hwp'. A typical generated grub2
configuration by using both options would look like this:
| insmod multiboot2
| insmod gzio
| multiboot2 /boot/bender bender microcode intel_hwp
| module2 /boot/micro.code micro.code
| module2 /boot/hypervisor hypervisor ...
| module2 /boot/image.elf.gz image.elf ...
When using the NOVA kernel and Genode's _run_ tool for booting respectively
disk-image creation, one may use the existing 'options_bender' variable in
_tool/run/boot/nova_. The microcode option is added by setting the
'apply_microcode' flag in the same file. The 'intel_hwp' option, at the other
hand, can simply be appended to 'options_bender'. On startup, bender will print
the applied HWP settings for each core to the serial output if the
'intel_hwp' option was set. The new feature will try to set Intel HWP to
'PERFORMANCE' mode, the mode for which we observed the best results.
NOVA microhypervisor
====================
The IO-MMU is a hardware feature to protect operating systems, e.g., Genode,
against misbehaving devices and/or corresponding device drivers. The feature
is supported on x86 since the
[https://genode.org/documentation/release-notes/13.02#DMA_protection_via_IOMMU - 13.02 release]
and described in the release notes. Up to now, this feature is solely
supported for Intel hardware, in particular CPUs and chipsets supporting Intel
VT-d.
With the current release, we add support for AMD's IO-MMU variant to the
Genode framework for the NOVA kernel - being the first one out of the
supported microkernels. Being conceptionally equivalent, the actual
implementation for AMD differs from Intel unsurprisingly. In order to add the
support, a new IO-MMU interface abstraction for accommodating both versions -
Intel and AMD - has been added to the NOVA kernel. Further, the discovery of
the available AMD IO-MMUs required the traversal of different ACPI tables than
for Intel and another page table format for the IO-MMU had to be added. On the
Genode framework side, only very few changes were necessary, namely the
detection of the IO-MMU feature by parsing the ACPI tables in Genode's ACPI
driver as well as the ported Intel ACPICA component.
The change has been already successfully tested on various Ryzen desktops and
notebooks on a backported Sculpt 20.08 branch.

Some files were not shown because too many files have changed in this diff Show More